gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-23 20:57:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: gsadmin/spinifex:v1.20.0
Branches Tags
gsadmin/spinifex:main gsadmin/spinifex:dev gsadmin/spinifex:feat/ebs-export-descriptor gsadmin/spinifex:feat/oracle-linux gsadmin/spinifex:feat/br-imds-flow-reconcile gsadmin/spinifex:feat/adr-0005-iam-enforcement-projection gsadmin/spinifex:feat/churn-wal-retention-visibility
gsadmin/spinifex:v1.22.0-98-g6daa34004-dev gsadmin/spinifex:v1.22.0-97-gb4d4fa65f-dev gsadmin/spinifex:v1.22.0-96-g2e2465dd2-dev gsadmin/spinifex:v1.22.0-95-g5d2c3e749-dev gsadmin/spinifex:v1.22.0-94-gfb0f8e929-dev gsadmin/spinifex:v1.22.0 gsadmin/spinifex:v1.21.0 gsadmin/spinifex:v1.20.0 gsadmin/spinifex:v1.19.0 gsadmin/spinifex:v1.18.0 gsadmin/spinifex:v1.17.0 gsadmin/spinifex:pre-dev-merge-backup gsadmin/spinifex:v1.16.0 gsadmin/spinifex:v1.15.0 gsadmin/spinifex:v1.14.0 gsadmin/spinifex:v1.13.0 gsadmin/spinifex:v1.12.1 gsadmin/spinifex:v1.12.0 gsadmin/spinifex:v1.11.0 gsadmin/spinifex:v1.10.0 gsadmin/spinifex:v1.9.0 gsadmin/spinifex:v1.8.0 gsadmin/spinifex:v1.7.0 gsadmin/spinifex:v1.6.1 gsadmin/spinifex:v1.6.0 gsadmin/spinifex:v1.5.0 gsadmin/spinifex:v1.4.0 gsadmin/spinifex:v1.3.0 gsadmin/spinifex:v1.2.0 gsadmin/spinifex:v1.1.0 gsadmin/spinifex:v1.0.2 gsadmin/spinifex:v1.0.1 gsadmin/spinifex:v1.0.0-hotfix gsadmin/spinifex:v1.0.0 gsadmin/spinifex:v0.5.0 gsadmin/spinifex:v0.4.2 gsadmin/spinifex:v0.4.1 gsadmin/spinifex:v0.4.0 gsadmin/spinifex:v0.3.0 gsadmin/spinifex:v0.2.4 gsadmin/spinifex:v0.2.3 gsadmin/spinifex:v0.2.2 gsadmin/spinifex:v0.2.1 gsadmin/spinifex:v0.2.0 gsadmin/spinifex:v0.1.0
...
compare: gsadmin/spinifex:v1.21.0
Branches Tags
gsadmin/spinifex:dev gsadmin/spinifex:feat/ebs-export-descriptor gsadmin/spinifex:feat/oracle-linux gsadmin/spinifex:feat/br-imds-flow-reconcile gsadmin/spinifex:main gsadmin/spinifex:feat/adr-0005-iam-enforcement-projection gsadmin/spinifex:feat/churn-wal-retention-visibility
gsadmin/spinifex:v1.22.0-98-g6daa34004-dev gsadmin/spinifex:v1.22.0-97-gb4d4fa65f-dev gsadmin/spinifex:v1.22.0-96-g2e2465dd2-dev gsadmin/spinifex:v1.22.0-95-g5d2c3e749-dev gsadmin/spinifex:v1.22.0-94-gfb0f8e929-dev gsadmin/spinifex:v1.22.0 gsadmin/spinifex:v1.21.0 gsadmin/spinifex:v1.20.0 gsadmin/spinifex:v1.19.0 gsadmin/spinifex:v1.18.0 gsadmin/spinifex:v1.17.0 gsadmin/spinifex:pre-dev-merge-backup gsadmin/spinifex:v1.16.0 gsadmin/spinifex:v1.15.0 gsadmin/spinifex:v1.14.0 gsadmin/spinifex:v1.13.0 gsadmin/spinifex:v1.12.1 gsadmin/spinifex:v1.12.0 gsadmin/spinifex:v1.11.0 gsadmin/spinifex:v1.10.0 gsadmin/spinifex:v1.9.0 gsadmin/spinifex:v1.8.0 gsadmin/spinifex:v1.7.0 gsadmin/spinifex:v1.6.1 gsadmin/spinifex:v1.6.0 gsadmin/spinifex:v1.5.0 gsadmin/spinifex:v1.4.0 gsadmin/spinifex:v1.3.0 gsadmin/spinifex:v1.2.0 gsadmin/spinifex:v1.1.0 gsadmin/spinifex:v1.0.2 gsadmin/spinifex:v1.0.1 gsadmin/spinifex:v1.0.0-hotfix gsadmin/spinifex:v1.0.0 gsadmin/spinifex:v0.5.0 gsadmin/spinifex:v0.4.2 gsadmin/spinifex:v0.4.1 gsadmin/spinifex:v0.4.0 gsadmin/spinifex:v0.3.0 gsadmin/spinifex:v0.2.4 gsadmin/spinifex:v0.2.3 gsadmin/spinifex:v0.2.2 gsadmin/spinifex:v0.2.1 gsadmin/spinifex:v0.2.0 gsadmin/spinifex:v0.1.0

101 Commits
v1.20.0 ... v1.21.0

Author SHA1 Message Date
Julian Sommer dd9c9666f2 release: update go.mod for v1.21.0 2026-09-29 10:10:36 +10:00
Julian Sommer ae2fbef021 fix: fail the capabilities benchmark on a nil response instead of panicking 
benchCapabilities read resp.Capabilities after checking only err, so a
provider returning (nil, nil) panicked the benchmark rather than failing it.
 2026-09-29 09:50:54 +10:00
Ben Duncan fe4ce999ed fix: wait for the capabilities handlers instead of reading the count once (#1127) 
TestServeQueueGroupBalancesSharedSubjects read both workers' handler counts
immediately after waiting for the mount to land on both of them, on the
argument that ordering on the shared connection meant the earlier capabilities
message must already have been delivered to everyone getting it.

That ordering does not exist. Each async subscription dispatches on its own
goroutine, so a worker's mount handler and its capabilities handler race, and
the mount having landed says nothing about whether the capabilities message has
been counted yet. Race Detection caught it: publishes reached 2 while
capabilities was still 1.

Wait for the expected count and then confirm it does not move. Both halves
matter — how many handlers a message reaches is the server's decision, so the
count converges from below and cannot overshoot, but a broken queue group
delivering to both members would pass through the right value on its way to the
wrong one.
 2026-09-28 18:48:27 -05:00
Julian Sommer 39759e1ed2 fix: report SupportedNetworkTypes and SubnetOutpost on RDS subnet groups (#1132) 2026-09-29 09:35:11 +10:00
Ben Duncan 8dad0ab9ab fix: wait for JetStream quorum before deciding viperblock has no storage (#1129) 
* fix: wait for JetStream quorum before deciding viperblock has no storage

On a cold multi-node start every service comes up at once against a cluster
that has not elected a meta leader yet. IAM retries that window; viperblock did
not. Its lease and dirty bucket opens are both on the path to the daemon
running at all, so the first refusal took the process down with "volume lease
bucket: context deadline exceeded" and systemd crash-looped it into a failed
bootstrap.

The start path now opens both through kvstore.OpenWithRetry, which is what vpcd
and IMDS already use for the same window. The plain constructors are untouched:
they are what the replica-count and cluster-size tests assert fail closed, and
an operator tool reading a refusal as a hang would be worse than the crash.

OpenWithRetry now stops on an undeclared cluster size, matching the IAM and
kvlease loops. Waiting cannot declare one, so retrying it would turn a clear
configuration fault into ninety seconds of silence followed by the same error.

* test: mark the viperblock bucket startup test as in-package

test-package-check requires new test files to be external unless the file says
why it cannot be. The start-path openers and the constructors beneath them are
both unexported, and which of the two waits is the whole subject.
 2026-09-28 18:34:09 -05:00
Ben Duncan b73ab7cfb3 fix: assert the takeover against the volume that was actually left unsealed (#1130) 
* fix: assert the takeover against the volume that was actually left unsealed

TestCrossNodeStartAfterFailedSealTakesOverLoudly picked its subject in advance
— the attached pattern volume — and treated the stop returning an error as
proof that volume's seal had failed. It is not. The unmount is a NATS request,
and the caller giving up says nothing about the seal behind it; local state
kept with no seal receipt is equally what an in-flight seal looks like.

In the nightly the test then thawed predastore, the pattern volume's seal
finished against a backend that was answering again, and the marker cleared. By
the time the volume moved there were no unsealed writes, so viperblock was
right to say nothing and the test failed it as a silent loss.

The product was loud where it mattered. The instance's root volume has to read
its AMI snapshot to seal, which the freeze does break, so its seal genuinely
failed and the node it landed on logged the takeover naming the node that had
been holding the writes. A blank attached volume has no such read and can seal
straight through the outage.

So the test now reads which volumes the cluster records as unsealed, after the
thaw has settled rather than before, and asserts the takeover against those. If
nothing was left behind it skips: the fault injection did not reproduce, and
asserting a warning there would be asserting a wrong one.

* fix: publish the instance cache resync mark with its readiness, not after it

installSync set ready under c.mu and markResynced ran after the unlock, so
there was a window where the cache was ready and serving records while
resyncAge reported that no sync had ever succeeded. A caller that observes a
ready cache and then asks how stale it is got "never", which reads as a cache
that cannot be trusted, and the resync-age metric was skipped for as long as
the window lasted.

The mark now goes up under the same lock that publishes readiness, so any
reader that takes the lock and sees a ready cache sees the mark too. Both
callers dropped their trailing markResynced, which installSync now does.

Widened under the race detector, which is where TestPeriodicResync_Undecodable
RecordMarksDegraded had been failing intermittently: it reads resyncAge
straight after waiting for readiness.
 2026-09-28 18:33:54 -05:00
Julian Sommer 8746b9e79d fix: page every RDS Describe call by MaxRecords and Marker (#1131) 2026-09-29 09:16:39 +10:00
Ben Duncan 8ab96b29a1 fix: replicate every JetStream KV bucket across the cluster (#1117) 
* fix: replicate every JetStream KV bucket across the cluster

A KV bucket created without a replica count gets one replica, and one
replica lives on one JetStream-chosen server recorded in no config.
Losing, restarting or partitioning that server makes the bucket return
"no responders available" from every node at once, so a single-node event
becomes a cluster-wide outage. The reconciler leader-lease buckets were
the worst case: acquiring a lease is a write, so a lease bucket without a
quorum cannot be acquired by anyone, and every reconciler sharing it stops
cluster-wide at exactly the moment a node has failed.

The cluster's node count is now declared once, in config.LoadConfig, and
read by the one helper every bucket creation goes through. Every process
that creates a bucket loads the cluster config first, so no service can
forget to pass it and no signature has to carry it. An undeclared size is
an error at creation rather than a silent fall back to one replica.

Existing buckets are raised rather than left behind: a service opening one
raises it, the daemon sweeps them all after formation, and
"spx admin kv replicas --repair" does it on demand. Raising reads the live
stream config and changes only the replica count, and never lowers one.

Removes the ~30 replicas/clusterSize parameters that only forwarded a
number, and JetStreamManager.UpdateReplicas, which set len(nodes) with no
clamp and would have failed outright on a cluster of six or more.

Proved by a repo-wide invariant test pinning bucket creation to the one
helper, a whole-cluster sweep in every multi-node nightly cell, and a new
kvquorum suite (cell 33) that takes away the node leading the most buckets
and requires the survivors to keep answering for all of them.

* chore: name cell 33 in the nightly dispatch input

* fix: a KV bucket that cannot be raised must still open

The first version of the replica fix treated a raise it could not place as a
hard error, and on a real three-node cluster that turned the fix into the
outage it prevents. A node whose config named a fourth server wanted four
replicas, NATS answered "no suitable peers for placement" for three, and the
daemon retried for its full five-minute budget without finishing bootstrap.
The bucket was present, quorate and serving on three nodes throughout. A
config naming a node before that node serves is the ordinary middle of
growing a cluster, so that is a state to tolerate rather than refuse.

The line is now drawn between the two cases instead of around both. Creating
a bucket below the cluster's replica count still fails, because that is the
defect and a fallback there reintroduces it where it is hardest to notice.
Failing to raise an existing bucket warns and hands the bucket back, and the
raise is retried on every open besides the daemon sweep and --repair. This
matters most in kvlease, which runs on every reconcile tick: refusing there
would stop every reconciler sharing a lease bucket that was working.

getOrCreateBucket now attaches before it creates. A create asking for more
replicas than the cluster can place is refused before the server looks at the
name, so with create-first "this bucket already exists" was an inference from
which error came back, and the wrong one comes back on a single-node server.

RaiseAllBucketReplicas attempts every bucket and joins the failures, and
--repair keeps going too. Stopping at the first would leave the rest of a
cluster short because one bucket could not be placed.

Also adds nats.ServerName/NodeFromServerName beside the template that renders
server_name, with a test tying them together. The kvquorum suite had mapped
stream peers to nodes by comparing them to node names, which they are not:
they are the node name behind a "spinifex-nats-" prefix, so nothing matched
and the node-loss test could not choose a victim.

* test: wait for the right NATS peer count after a node rejoins

/routez reports a node's routes to its peers, not its own membership, so a
fully formed three-node cluster is two peers per node. The kvquorum suite
waited for three and spent its whole five-minute rejoin budget on a cluster
that had already reformed. Every other caller passes len(Nodes)-1; the
helper's doc comment now says so.

The assertion the test exists for had already passed: node1 was leading 18 of
47 buckets when it was taken away, and node2 and node3 each still answered for
all 47.

* fix: replicate KV buckets across the largest odd count a cluster can place

A bucket was replicated across min(nodes, 5) servers, which stores a fourth
full copy on a four-node cluster and still survives exactly one loss — the
same tolerance R3 gives for three quarters of the write amplification. A Raft
majority needs an odd count, so the rule becomes the largest odd count the
cluster can place, capped at five: 1, 3, 3, 5, 5. It is one expression with two
entry points, so no caller can hold a stale copy of the policy and keep passing
after the policy changes.

Three retry loops could not tell a configuration fault from a cluster that had
yet to form. An undeclared cluster size never becomes declared by waiting, so
each of them spent its whole budget on one and then reported quorum as the
cause. They now stop on a permanent error and say what is actually wrong. This
is what the integration suite was hitting: it builds its gateway by hand and
never loads a cluster config, so the size was never declared and the IAM retry
sat in its sleep until the test timed out. Both ends are fixed.

The audit reported configuration alone. A bucket at three replicas with one
peer offline and one lagging is one failure from unavailable and read as ok.
BucketReport now carries the leader and a count of the peers JetStream itself
calls current and reachable, quorum is measured against that count, and
spx admin kv replicas exits 0, 1 or 2 so a deployment gate can tell a problem
--repair fixes from one needing a human.

Declaration is write-once: daemon.Start refuses a conflicting redeclaration
because it is about to create buckets, and it declares before JetStream init
rather than after it. config.LoadConfig warns instead, because it also reads
config files that are not this process's own cluster.

spx admin init --nodes 2 is refused — no replica count on two nodes both
survives a loss and accepts writes — and single-node replication is asserted in
the single e2e suite, so it runs in every nightly host permutation.

* fix: withhold the vLLM serving image until its artifact is published

The catalog offers ubuntu-26.04-vllm-serving-x86_64, and the qcow2 it names
404s on iso.mulgadc.com, so spx admin images import would fail for every
operator who tried it. Its GPU siblings are published; this one never was.

The entry moves to WithheldImages rather than being deleted, because the entry
is the definition — the URL, the checksum and the two tags resolveServingAMI
filters on. Publishing the artifact is then a move back, not a retype.
Bedrock already handles the image not being registered: resolveServingAMI
returns ErrServingAMINotFound, which is the honest answer while there is
nothing to launch.

The bedrock tag contract is still pinned, now against the map the definition
lives in, and it additionally asserts the entry is not offered for import. The
e2e reachability walk covers AvailableImages alone, so a withheld entry is out
of it by construction.

* fix: drop the cluster size from the call sites dev added

Two constructors on dev still pass a replica count that no longer exists as a
parameter: the cluster size is a declared process fact, so EKSServiceDeps has
no ClusterSize field and NewIAMServiceImpl takes three arguments. The merge is
textually clean and breaks the build, which is the shape of collision a
signature removal always has.

* docs: explain what an operator sees when a cluster server fails

The replication section said how many copies a bucket has and stopped there,
which leaves the question an operator actually has during an incident: one
server is gone, what still works. It now covers the majority side carrying on,
an isolated server refusing changes on purpose and why that is the behaviour
you want, guests continuing to run while anything that has to be recorded does
not, the risk of reading state from a server that is cut off, and a returning
server catching up by itself.

The surrounding prose reads from the operator's side rather than the
implementation's: what you see and what to do, not which consensus protocol is
underneath.
 2026-09-28 12:34:55 -05:00
Julian Sommer ecef5e4d2f feat: CONTRIBUTING.md 2026-09-28 16:27:01 +10:00
Julian Sommer 772c7657d8 fix: return AWS's per-action field sets for IAM roles, policies, users and account summary 
CreateRole, GetRole, ListRoles and instance-profile roles each get their own projection; GetRole carries an empty RoleLastUsed. Policies carry UpdateDate and PermissionsBoundaryUsageCount, CreatePolicy drops Description and ListPolicies drops Tags. CreateUser echoes Tags, list instance-profile actions drop profile Tags, and GetAccountSummary gains AWS's seven missing keys and loses SigningCertificatesPerUser.
 2026-09-28 16:12:16 +10:00
Julian Sommer 2898181837 fix: test packages 2026-09-28 16:01:49 +10:00
Julian Sommer b784483a03 fix: return ValidationError for IAM name, path and enum violations 
AWS answers a bad name charset, a path not bounded by slashes and an
enum violation such as UpdateAccessKey's status with ValidationError.
Length and tag violations keep InvalidInput until AWS's code for them
is captured.
 2026-09-28 15:59:51 +10:00
Julian Sommer 866392e680 fix: honour MaxItems and Marker on IAM list actions (#1125) 2026-09-28 15:53:41 +10:00
Julian Sommer 0fd0e2ba15 fix: apply PathPrefix, Scope and OnlyAttached in IAM ListPolicies 2026-09-28 15:36:22 +10:00
Julian Sommer 377c573bf6 feat: add IAM managed policy versions (#1124) 2026-09-28 15:28:20 +10:00
Julian Sommer 1426cdd072 fix: URL-encode IAM policy documents in Query API responses (#1123) 
AWS percent-encodes every policy document it returns. Spinifex returned
raw JSON, so a client decoding once read '+' as a space and '%41' as 'A',
giving perpetual Terraform drift, and a lone '%' failed the read.

Encode in the gateway wrappers for CreateRole, GetRole, ListRoles,
GetRolePolicy, GetUserPolicy, GetGroupPolicy, GetPolicyVersion and the
instance-profile reads. The service layer keeps raw JSON because STS and
spx admin parse it in-process. The web UI now decodes role inline
policies like every other document.
 2026-09-28 14:31:16 +10:00
Julian Sommer 2d2d2f27ed feat: model-driven request conformance in the integration suite (#1122) 2026-09-28 14:14:15 +10:00
Julian Sommer 0c3403889c feat: load AWS conformance models from api-models-aws (#1121) 2026-09-28 12:21:05 +10:00
Julian Sommer d8cf17bf6e fix: return validation messages that name the failing parameter (#1120) 2026-09-28 12:05:22 +10:00
Julian Sommer 7b3ebb2ab4 fix(gateway): answer DryRunOperation for every EC2 action that models DryRun (#1119) 2026-09-28 11:30:51 +10:00
Julian Sommer 83f351bd6d fix: return AWS's messages for AssumeRole and GetAccessKeyInfo errors (#1118) 2026-09-28 11:18:17 +10:00
Julian Sommer c473365f1a refactor: move new test files to external test packages 
Tests added since main used in-package test packages, failing the
test-package check. Convert them to <pkg>_test with export_test.go
hooks; the ecs-agent main package keeps an in-package marker.
 2026-09-28 10:54:58 +10:00
Julian Sommer 94488dc399 fix(gateway): don't count body-read failures toward the auth lockout 
A request whose body cannot be read to hash failed through the catch-all
branch, logging a malformed envelope, returning IncompleteSignature and
recording an anonymous lockout strike, so ten dropped connections locked
out an IP. Match sigv4.ErrReadingBody and answer InternalError without
recording a failure. Bumps bluebottle for the new sentinel.
 2026-09-28 10:49:28 +10:00
Julian Sommer 9bd1860622 chore: update pnpm 2026-09-28 09:45:16 +10:00
Julian Sommer 4abd2fd984 fix: delete crap tests 2026-09-28 09:41:00 +10:00
dependabot[bot] b3d1e859c4 chore: bump the npm-dependencies group (#1113) 2026-09-28 09:03:05 +10:00
Ben Duncan 021f439bab feat: recover instances automatically from a node that stopped heartbeating (#1111) 
* feat: recover instances automatically from a node that stopped heartbeating

A node that dies hard takes its guests with it, and nothing brings them back
until an operator brings that node back. Every daemon now runs the same small
reconciler: it reads the instance records, selects desired-running instances
whose owner is stale, and relaunches them here.

The claim is a CAS on Status.LastNode and that is the whole of the
coordination. Several survivors race one instance, exactly one revision wins,
and the losers find a record naming a live owner and stop considering it. A
claimant that dies mid-launch leaves the record naming a node that is itself
stale, which is the same candidate the next pass derives again, so there is no
lease to expire and nothing to reconcile by hand.

Ownership is not permission to write. The volume leases still decide whether the
guest can move, and VolumeMounter.Mount runs before startQEMU, so an attempt
that loses the volume race has touched no tap, no OVS port and no OVN binding.

A storage failure is not a host failure and wants the opposite response. A guest
paused by werror=stop publishes Health.IOErrorSince, and a record carrying it is
refused: the volumes are the same objects in the same store from anywhere, so
relaunching moves the pause rather than ending it. A node whose own object store
is not answering stands down entirely, which separates a store broken here from
a store broken everywhere without either having to be diagnosed. A launch that
does not complete backs this node off that instance, doubling to a cap, because
nothing on the mount path can yet say which of the two it hit.

Recovery is off by default and inert below three nodes. It also waits until it
has seen its peers live, or five minutes, so the first node back from a
coordinated shutdown does not recover guests from nodes that are still booting.

* test: prove instance recovery moves a host failure and refuses a storage failure

The e2e half of automatic instance recovery, asserted from both sides because a reconciler that cannot tell the two failures apart passes one and fails the other. InstanceAutoRecovery takes a node down and requires the guest back on exactly one survivor, identified by the qemu process rather than by what the API reports. InstanceRecoveryRefusesStorageFault stops predastore cluster-wide first, so the guest is paused by werror=stop when its node goes away, and requires that nothing moves.

Also fixes the settle window, which re-evaluated every pass and so held a daemon back for its first five minutes over exactly the failure the reconciler exists to handle. It now latches once every peer has been seen live.

harness.InstanceHostingNodes replaces the single-node lookup so a caller can assert there is exactly one host rather than assume it, and skip a node it has deliberately taken down.

* fix: make automatic instance recovery a cluster-wide setting

Recovery was read from the per-node Config, so it had to be written three times on a three-node cluster, once inside each [nodes.<name>] table. A top-level [recovery] table — the obvious thing to write, and what the e2e suite writes — was parsed into nothing and the daemon reported recovery disabled on every node with the setting plainly in the file.

Cluster-wide is also the honest shape. A node with recovery off is not opting out of anything: recovery is what survivors do to a dead node's instances, so the setting belongs where there is one of it.

* test: drive real I/O into the storage-fault case, and freeze the store rather than stop it

Measured on a three-node cluster: predastore stopped cluster-wide for twelve minutes against idle guests paused none of them. Viperblock serves what it has cached and an idle guest issues nothing that reaches the object store, so the test asserted that nothing moved in a run where nothing was ever faulted.

The victim now launches with cloud-init user-data that writes to its root volume continuously. User-data rather than SSH because a routed-NAT cell gives guests no public address, so a test that had to log in could not run on half of them.

The fault is SIGSTOP rather than a unit stop, for the reason the storagefault suite already gives: a frozen process holds its connections open and answers nothing. Under that fault and that load the first BLOCK_IO_ERROR arrived 4m47s later, so the pause budget is now ten minutes rather than three.

* test: give instance recovery its own suite and its own nightly cell

Both tests take a node away and one of them freezes predastore cluster-wide, so anything sharing the environment reports that outage as its own failure — the same argument storagefault already makes for being its own package. They are also slow by nature: a dead owner has to be seen stale twice, a guest has to boot, and the storage case waits for a pause measured at 4m47s and then holds four minutes to prove nothing moved. Roughly twenty minutes on top of a permutation cell budgeted at thirty-five.

Cell 31 is three-node and static, matching storagefault's, with a 75-minute job budget over a 60-minute suite timeout. The suite is eligible on multi-node environments and deliberately not in the nightly permutation set.

* fix: give a guest long enough to open a volume the object store is slow to serve

QEMU opens every drive before its monitor answers a greeting, so the QMP
greeting deadline is really a deadline on opening the guest's volumes. At 30s
it was shorter than a cold viperblock open costs on a cluster that has lost a
node, where every object read is served degraded: measured at 35s with one node
of three away. The launch killed the guest five seconds short of ready and then
did it again on every retry, which is a recovery that can never complete.

A network-backed drive now gets its own deadline for the same reason VFIO
already had one — synchronous work before the monitor answers — and the
instancerecovery suite is what found it.

Also make the suite's predastore thaw best effort per node. It runs after a
node has been taken away, where there is no process to signal, and a node it
cannot reach must not stop it restoring the rest.

* docs: name the two newest cells in the dispatch input

The cells input is the operator's index of what a number means, and it stopped
at 29 while 30 and 31 were added.

* test: prove the guest is writing before freezing the store in the recovery suite

The storage-fault test froze the object store as soon as RunInstances reported
running, which is when QEMU has launched rather than when the guest has booted.
On the nightly cluster the freeze landed while the guest was still in UEFI, so
the only drive it could reach was the EFI varstore — and pflash has no
werror=stop to pause on, so the guest never paused and the test reported a
product failure it had caused itself.

The load driver now prints to the serial console after each pass it has written
and synced, and the test waits for that before freezing. The console is read
through GetConsoleOutput, so this still needs no login and still runs on cells
whose guests have no public address.

* feat: recover instances automatically with nothing to configure

The switch is gone. An instance whose host has failed is down whether or
not recovery is enabled, so the only outcome a disabled reconciler can
produce is leaving it down — which is the state recovery exists to end,
not a safer one to fall back to.

Against that, the switch cost a cluster that can be configured into
never recovering anything, with no symptom until a node dies. This
feature already hit that once: a top-level table parsed into nothing and
every node reported recovery off with the setting plainly in the file.

What makes recovery inert is now only facts about the cluster — no
cluster config, no JetStream, or fewer than three nodes — and each says
in the log which one applied. The e2e no longer writes config or
restarts daemons: it waits for every node to report the reconciler
active on a cluster nothing configured, so reintroducing a switch fails
there as well as in the unit tests.

* feat: stop retrying a recovery that is never going to work

Every failed attempt used to go to the same exponential backoff and stay
there. An instance nothing could launch was retried every fifteen
minutes forever, still observed-running on a node that is gone, with no
account of it anywhere the customer or an operator would look.

A failure is now classified and counted. Two classes are read from what
the launch actually reports: the node it was taken from still holds a
volume lease, which clears on that lease's clock and is retried on a
faster one, and this node has no room, which another survivor may. There
is no terminal class, because nothing on this path can prove one — the
launch collapses everything else into ServerInternal, so a verdict of
"never going to work" would be a guess, and the guess that abandons a
recoverable instance is the expensive one.

The budget bounds the unknown case instead. Every class counts against
it, since a class that did not would be the old unbounded retry under a
narrower condition. Exhausting it leaves the instance where AWS leaves
one whose host failed and could not be brought back: stopped, carrying a
StateReason that names the cause and the last error, one StartInstances
from running again. Writing the desired state down is also what makes
one node's decision cluster-wide, because selection reads desired state
— the same property the claim already has.

A record another node has since claimed is never touched, so a node
exhausting its attempts cannot stop a guest that is now running
elsewhere.

* fix: give a guest the time for every cost it pays before its monitor answers

A GPU instance booting from a network-backed root volume pays both
costs: QEMU opens every drive across the object store and then pins the
whole of guest RAM, one after the other, all before the monitor answers.
The deadline took whichever was larger, so such a guest was given less
time than it needs.

That the two floors are both 180s is why nothing has been seen yet — the
VFIO branch has always returned at least the network-drive deadline. It
is the same defect either way, and the symptom when it lands is a GPU
instance SIGKILLed on every attempt while its plain sibling launches,
which reads as a GPU fault rather than as a deadline.

The costs are now added, the plain QEMU startup is counted once rather
than replaced, and the cap covers both floors together so it cannot clip
a guest before either cost has been waited out.

* test: assert the AWS-visible arc of a host failure end to end

A customer cannot see which host an instance is on, and AWS never tells
them. What they can see is DescribeInstanceStatus, so that is where a
recovery has to be provable, and the suite never looked at it.

Three frames are now required, in order: running with system-status ok
before anything is broken, system-status impaired while the owning node
is gone, and ok again once a survivor is running it. The middle frame is
the one worth having — the window opens when the owner's heartbeat goes
stale, which is strictly before any survivor may claim, and closes when
the relaunch registers, so it is bounded rather than lucky. Reporting ok
throughout would be the more comfortable lie.

The availability zone is compared across the first and last frames as
well. A recovery that crossed one would break every placement promise
made to the customer, and nothing else in the suite would have noticed.

* test: prove a half-healthy node fences itself and is never a second writer

The two existing recovery cases take a node away by stopping its services,
which is the failure this feature handles least interestingly: a node that is
gone has no opinion about who owns its volumes. The one that matters is the node
that is still up, still running QEMU, still answering its own local clients, and
simply cannot reach consensus. That is the state in which two nodes can each
believe they own a volume, and the only thing between that and a corrupted disk
is that the partitioned node gives up first, on a clock it can evaluate without
reaching anything.

So this drops NATS on 4222 and 4248 between one node and its peers and nothing
else, then asserts the ordering the safety argument rests on rather than the
outcome it produces. A watch samples every node every three seconds for the
whole run, including the heal, and fails on any single sample naming two hosts.
Two guests writing one volume through one object store is data loss already
under way, not a race that resolves, and no point-in-time assertion can see one
that appeared between its own samples.

The injector takes its own nft table rather than touching inet spinifex_filter,
because a test that flushed the node's real firewall would leave it either
unfirewalled or misfirewalled with no sign of which, and a separate table at
priority -300 heals in one atomic delete. A transient systemd timer removes it
independently, so a cancelled workflow does not leave a node cut off from its
cluster until somebody notices. It proves SSH still works and proves a peer port
is genuinely unreachable before anything is asserted, since a rule set that
installed cleanly and dropped nothing would read as the system tolerating a
partition.

The suite budget goes to 90m and cell 31 to 105m to fit the third case.

* fix: move an instance's boot state, volumes, logical port and address with it

A record-driven launch assumes the node reading the record is the node that wrote
it. On a daemon restart that holds. On a recovery onto a survivor it does not, and
five things follow from it — one that stops a moved instance booting at all, and
four that stop it running anywhere.

A load balancer never comes back from a host failure. Its boot configuration is a
set of files under the runtime directory and the record holds their paths, so the
survivor restores paths that exist only on the node that died. The hook that
rebuilds them was reachable from Restore alone, and the cross-node path goes
straight to vmMgr.Run — so QEMU is handed a netcfg, an agent env and a CA cert
that are not there. The gateway URL baked into that env names the creating node's
management bridge as well, which the rebuild corrects because it rebuilds from
whichever node is running it. Firing the hook on both paths fixes both, and a
rebuild that fails now refuses the launch: a guest that boots without its network
configuration is worse than one that does not boot, because the records would say
it recovered.

The export outlives the guest and holds the volume lease. A returning node is
healthy, so it renews that lease indefinitely and the new owner never acquires:
the guest is stopped here and refused there, recovery spends its attempt budget on
a volume that was never going to be free, and gives up on an instance that was
never unrecoverable. So the export is given up too, over a new node-addressed
request that tears it down exactly as the fence does. Deliberately not an unmount
— an unmount seals, this node's block map is the stale one, and sealing publishes
it over the winner's. Unlike the fence it deletes the lease key rather than
leaving it to expire, because the lease is still this node's to hand back.

The tap outlives the guest, and the tap is what claims the instance's OVN logical
port. The binding names one chassis, so two nodes offering the same iface-id do
not split the traffic — they contend, and the address answers from whichever won
last. It also blocks the layer above: the external IP's host ingress is pruned on
the evidence that the owning guest sits on another chassis, and that stays false
while this node is still claiming the port.

The management address outlives the guest as a cluster-wide reservation, and the
release was unconditional. br-mgmt is one flat L2 segment, so the node an instance
left was freeing an address still in use and offering it to the next instance
anywhere. Only the node an entry names may release it now, with the takeover
recorded on arrival so the node actually running the instance can still give the
address back. An entry written before the node field existed names nobody and
stays releasable, because there is no other holder to collide with.

The fence path already cleaned up the tap, so the release fix repairs that path
too.

* test: prove a load balancer keeps its name, its address and its traffic when its host dies

Takes away the node running an internet-facing ALB and asserts the four things a
customer of one actually holds: the name keeps resolving, the address behind it
does not move, requests through the name are answered again without anybody being
called, and the backends are healthy again from the recovered load balancer's
point of view.

The backends are placed around the load balancer rather than beside it. A load
balancer lands wherever a node answers its queue group, so its node is discovered
and the backends are relaunched until both are elsewhere; one on the node about to
be taken away would be recovered too, and the run could not then tell an ALB that
came back serving from one that came back with nothing to serve.

DNS is asked of a node's own northstar over the wire, and the HTTP probes resolve
through the same server, so the claim under test is what this cluster answers for
a name rather than what the machine running the tests was configured to forward
to. A watch samples every surviving node every five seconds for the whole run,
because an answer that went missing or pointed elsewhere for one reconcile pass is
the outage, and no single sample reaches it.

The returning node then has to give up what it no longer owns: no second copy of
the guest, and no tap still claiming the load balancer's logical port, since OVN
binds a port to one chassis and a second claimant contends for it rather than
splitting the traffic. The last assertion is that the name still serves with that
node back in the cluster.

* docs: name the new cell in the nightly's cell list

* test: stop the disk-load guest from filling the console it is judged by

The load generator announced every write, once a second, for as long as the
test ran. The console is a ring buffer this suite reads back, so an hour of
that pushes out everything explaining a failure, including the kernel's own
account of the I/O error the test exists to produce. Announce twenty times,
then keep writing quietly.

* docs: say what the OCI guide has been proven on, and what a host failure does

The guide read as a support statement for OCI rather than as an account of one
tested path, promised seven steps and then gave eight, and said nothing about
what happens to a guest or its public address when the node under it fails.
State the evidence, state that the nightly guards the datapath and not the
cloud, and give the failure behaviour as a table with the reason codes a
customer will actually see.

Also say plainly what the volume lease is: one running guest and a bounded,
logged window, not a write barrier the object store enforces. Someone sizing a
workload against this should read that before they read the table.

* docs: say why the mount classifier refuses to retry a lease another node holds

It reads as an omission — a lease held elsewhere is the most transient refusal
there is. It is not. This classifier drives the restore path, where the holder
is a node that is running the guest, so retrying would relaunch against a live
owner. Recovery wants the opposite and reads the refusal itself.

* feat: claim a recovered guest's public address in line, and verify it arrived

A recovered instance was reported running before anything moved its public
address off the node that died, so for up to fifteen seconds the records said
recovered and the address answered nothing. The recovery path now runs the
affinity pass itself, which is idempotent and is the same pass the ticker runs.

Two things the pass could not do before. It abandoned every remaining address
on the first API failure, though a throttle or a denial is per-object and the
guests behind the others are just as dark. And it trusted the move's own
response, which is OCI agreeing to the request rather than OCI having done it,
so an accepted move that never landed read as success. It now re-reads what OCI
delivers here and reports the difference.

An address that will not arrive makes the guest a system-status failure with the
time it went dark, the same shape AWS uses for a host-level fault. The guest
passes every check it can be asked about itself, which is exactly why reporting
it healthy was the worst available answer.

* feat: give a recovery attempt one trace across all three of its phases

The volumes opening, QEMU starting and its monitor answering already trace
themselves, but nothing tied them to the attempt that asked for them: three
orphan traces per recovery, no way to ask which phase a slow one spent its time
in, and nothing naming the node the guest came from. A span over the attempt
makes them its children, and the reconciler's own log lines carry its trace ID
because the handler stamps it from the context.

A failed attempt records the fault and the attempt count on the span, so the
budget running out is readable as a sequence rather than as twelve unrelated
warnings.

* fix: finish a fence once its writer finally exits

A kill that did not take effect was logged, counted and dropped. The export
stayed registered and the lease stayed held, which is correct while the writer
lives and wrong forever after it dies: a lease a healthy node keeps renewing
never expires, so the volume could not start anywhere.

The failure is now re-reported every 30 s until it resolves, and the teardown
completes as soon as the writer is confirmed gone. Only a delivered kill is
waited on — a refused one says nothing about what became of the writer.

* test: stop judging a detached stop or terminate on a transient state

Both tests read the instance state immediately after the ack, but the transition
runs on its own goroutine, so the read races it in both directions: too early
sees running, too late sees the final state. The race gate caught the first of
those on a loaded runner.

* fix: replicate the volume lease and dirty buckets across the cluster

Both were created with the default single replica, so both lived on one node.
That node holds the answer to "who may write this volume" and the marker every
mount reads, and losing it takes volume leasing and mounting down cluster-wide.
A partitioned node was fenced correctly and then no survivor could relaunch its
guest: every attempt failed on "read dirty mark: no responders available", for
ten minutes, until the budget ran out.

The replica count is now the cluster's node count, as the gateway's own buckets
already do. Existing buckets are raised in place rather than requiring a wipe.

The e2e diagnostics that should have said this are fixed too. The fence and the
surrender are logged by the storage service, not the daemon, so the assertions
read the wrong journal; and Eventuallyf evaluates its message at the call site,
so a ten-minute wait was explained by a log that stopped before it began.
 2026-09-27 15:56:09 -05:00
dependabot[bot] b7b1ae562a chore: bump the go-dependencies group with 2 updates (#1112) 
Bumps the go-dependencies group with 2 updates: [github.com/nats-io/nats-server/v2](https://github.com/nats-io/nats-server) and [github.com/nats-io/nats.go](https://github.com/nats-io/nats.go).


Updates `github.com/nats-io/nats-server/v2` from 2.14.7 to 2.15.0
- [Release notes](https://github.com/nats-io/nats-server/releases)
- [Changelog](https://github.com/nats-io/nats-server/blob/main/RELEASES.md)
- [Commits](https://github.com/nats-io/nats-server/compare/v2.14.7...v2.15.0)

Updates `github.com/nats-io/nats.go` from 1.53.1 to 1.54.0
- [Release notes](https://github.com/nats-io/nats.go/releases)
- [Commits](https://github.com/nats-io/nats.go/compare/v1.53.1...v1.54.0)

---
updated-dependencies:
- dependency-name: github.com/nats-io/nats-server/v2
  dependency-version: 2.15.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: github.com/nats-io/nats.go
  dependency-version: 1.54.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-09-27 10:38:42 -05:00
Ben Duncan 20b3eae728 fix: record the OCI SDK's own dependencies in go.sum (#1115) 
spinifex/cloud/oci imports oci-go-sdk directly, but go.mod listed it as indirect
and go.sum was missing three of the modules it needs — sony/gobreaker/v2,
gofrs/flock and youmark/pkcs8. The module therefore only type-checked inside the
go.work workspace, whose go.work.sum happened to carry them, so a standalone
checkout failed to build and the linter reported it as a typecheck error against
an unrelated file.
 2026-09-27 10:38:00 -05:00
Ben Duncan fcda031117 test(e2e): make the workbook teardown a gate, not a warning (#1110) 
* test(e2e): make the workbook teardown a gate, not a warning

A customer who cannot remove what they created is as stuck as one who could
not create it, so the teardown is part of what a workbook proves. It logged
WARN and passed the suite regardless, and a bucket that could never be emptied
went unnoticed for weeks behind exactly that line.

The failure it hides is a wedge rather than an error: the AWS provider retries
DeleteBucket against a bucket it cannot empty indefinitely. So the run is
bounded by DESTROY_TIMEOUT, defaulting to 900s, and the last forty lines of its
output are printed when it fails. The cleanup after a failed apply is bounded
the same way for the same reason.

* test(e2e): print a workbook's PASS after its teardown, not before

The verdict now depends on the teardown, so a workbook that built everything
correctly and then could not delete it printed PASS and FAIL for the same run.
 2026-09-27 10:37:19 -05:00
Ben Duncan cacd2c2c85 fix: re-read a rejected volume lease renewal before fencing the guest (#1116) 
A renewal that times out is treated as transient and keeps the lease, correctly,
but the server may have applied it anyway. It is then a revision ahead of what
the holder recorded, so every later conditional update is refused for that
reason alone — and a refusal was read as a peer having taken the volume. The
export was fenced, nbdkit killed and the guest lost its disk, with the log
naming a winner of "unknown" because no peer ever existed. A ten-second
JetStream stall destroyed four running guests this way.

A rejected update now re-reads the entry. One that still carries this lease's
owner and generation is this node's own write coming back, so the revision is
adopted and the lease continues; anything else, or an entry that has gone, is a
real loss. A re-read that cannot be answered keeps the lease without extending
it, because local validity is the bound that depends on reading nothing.

The rule is stated in its own test: a revision mismatch may not fence before a
re-read says the entry is no longer ours, and lapsed local validity always
fences whether or not a winner can be named. The weaker rule — that an
unnameable winner never fences — would leave a partitioned node writing for as
long as it stayed partitioned.
 2026-09-27 10:31:20 -05:00
Ben Duncan adb3d9876d fix: read a tagged bucket's tags through S3 instead of S3 Control (#1109) 
The AWS provider reads a bucket's tags with `ListTagsForResource` on S3 Control when it knows the account ID. S3 Control puts the account in the hostname — `https://{account}.{endpoint}` — so against an IP-addressed endpoint the request fails a DNS lookup before a packet leaves, and no endpoint configuration can help. A tagged bucket therefore fails its first read after being created.

`skip_requesting_account_id` alone does not fix it: `aws-sdk-go-base` takes the account ID from the STS `GetCallerIdentity` credentials check before it consults that flag, so `skip_credentials_validation` is needed alongside it. With no account ID the provider falls back to `GetBucketTagging`, which predastore serves.

`s3-webapp` is the only workbook that creates a bucket, and it uses no `aws_caller_identity` and references no other security group, so a blank account ID costs it nothing.

The bucket also gets `force_destroy`, since the webapp uploads into it and a destroy of a used bucket fails `BucketNotEmpty` without it.

The e2e harness now asserts the bucket carries the tags Terraform set, which is what catches this regressing, and no longer aborts the remaining workbooks on a failure — `s3-webapp` failing says nothing about whether RDS works, and four nights of `rds-quickstart` coverage were lost that way. `STOP_ON_FAIL=1` restores the early abort.

Every workbook's provider versions are pinned, so a run reproduces rather than picking up whatever the registry published that morning.
 2026-09-26 16:11:00 -05:00
Ben Duncan 7163f7d62d feat(oci): run Spinifex on Oracle Cloud, and fix the public-address and NAT gateway lifecycle (#1108) 
* feat: ship uninstall-spx.sh and fix the OCI bring-up path

Adds scripts/uninstall-spx.sh, the inverse of setup.sh: it removes the
software where node-reset.sh clears state and leaves the node installed. An
operator who installed with `curl ... | bash` had no teardown path at all.
State teardown is delegated to node-reset.sh rather than reimplemented, and
setup.sh now ships both to /usr/local/share/spinifex so they are reachable
without a checkout.

It refuses on a multi-node cluster and --yes does not override that: a flag
meaning "don't ask me" must not also mean "ignore a safety property".
/var/lib/spinifex and every apt package are kept unless asked for, and both
opt-ins confirm separately even under --yes.

Proved by a full uninstall/reinstall round trip on dev-oci, which found two
bugs neither prod nor dev-prod can reach, because both are ISO-installed:

node-reset.sh exited 2, silently, at its last step and after the wipe.
setup.sh creates the spinifex *group* and the spinifex-* service users but
never a *user* named spinifex, so `getent passwd spinifex` exits 2 and
pipefail carried it into set -e.

The uninstaller broke openvswitch-switch on its first run by removing the
socket-perms helper while leaving the drop-in referencing it, failing the
unit at step EXEC with status 203. Fixed on both sides: the uninstaller
removes drop-ins it wrote into other units' namespaces and restarts anything
it failed, and setup-ovn.sh writes ExecStartPost=- so a missing helper can
never stop another package's daemon.

setup-ovn.sh also learns the shape OCI needs: when --wan-bridge names a Linux
bridge it links it to OVS with a veth pair instead of enslaving the NIC, so
the VNIC keeps its MAC identity and netplan keeps owning the addresses. It
now fails on OVS datapath errors instead of reporting success, and the
--management demotion guard keys on /etc/default/ovn-central rather than on
database presence, which the ovn-central package creates on every node.

Tests: scripts/uninstall-spx_test.sh, wired into make test-build-scripts
along with shellcheck over the four scripts that ship to every node.

* feat: allocate external IPs from the Oracle Cloud API

An OCI VNIC drops any packet whose source is not a registered private IP
object on it, so an external pool cannot be range math there — the address
has to be created through the provider before it exists at all. Add
source = "oci" and the allocator behind it.

One allocation is two OCI objects: a secondary private IP that rides the
wire, and a RESERVED public IP attached to it that the internet reaches.
Allocate returns the public address because that is what AWS semantics are
about, and records the pair so the host plumbing and release path can find
the private half.

The OCI objects are created before the binding is written, and dropped
after it on release — both orders chosen so a crash leaves a discoverable
leak rather than a record that lies. Reconcile squares the two on startup
and is required, not optional: a leaked reserved public IP bills and holds
one of the 50 per-region slots with nothing referencing it. It never
touches an object without our display-name prefix, because the reference
host carries operator-created secondaries on the same VNIC.

Resolve the VNIC from an interface name by MAC through IMDS rather than by
Oracle's VNIC ordering, which is not a documented contract.

Generalise ExternalIPAM.EnableDHCP into InstallAllocator so the OCI wiring
is not a second single-purpose installer.

* fix: refuse source = "oci" outside routed mode, and cover the SDK path

An OCI VNIC accepts exactly one MAC, its own, and delivers every inbound
frame to it regardless of which of its addresses the packet is for.
Measured on the reference host: an internet source reaches 10.200.0.144 as
router-MAC to VNIC-MAC, the public address never appears on the wire at
all, and every off-subnet destination resolves to the same next hop.

Pool mode puts a per-VPC gateway LRP MAC and a per-rule external NAT MAC on
the uplink and ARPs for the address, which policy.go:50 justifies with "the
Linux bridge learns the per-rule external MAC on the veth port like any
other MAC". An OCI VNIC does not learn MACs; it accepts one. So a pool-mode
guest's egress is dropped and its ingress never reaches OVN. Routed mode
keeps the address on the host, which already wears the VNIC's MAC.

Pinning every LRP to that MAC does not rescue pool mode: they share one
external logical switch and OVN resolves inbound frames by an L2 lookup on
it, so duplicate MACs work for one VPC and silently misroute past that.

The refusal names the MAC reason, because a refusal that only says "use nat
mode" reads as arbitrary and gets worked around.

Also test the SDK request shapes against a local endpoint. Two of them are
wrong in ways reading does not catch: pagination stopping early reports a
partial VNIC to reconcile, which then drops live bindings, and a nil
privateIpId on detach is omitted from the body entirely, so OCI reads the
update as a no-op, returns 200, and leaves the address attached and billing.

* fix: install host firewall rules where a distro's catch-all reject cannot precede them

Oracle's Ubuntu cloud image ships a catch-all REJECT in both INPUT and FORWARD.
Two Spinifex datapaths were silently dead behind it on any such host.

EnsureNATEgressRules probed with -C and appended with -A, so its ACCEPT rules
landed after the REJECT and never matched: 24113 packets on the reject, 0 on
both accepts, and guests had no egress. The comment named the case it did not
handle, anticipating a FORWARD policy of DROP — which an append does survive —
rather than an explicit REJECT rule, which it cannot. Insert at the head, and
drain existing copies first, because a rule an earlier build appended still
satisfies -C and would otherwise stay stranded across an upgrade.

Nothing opened INPUT for the IMDS endpoints at all. The responder binds .254
and .253 per instance on the ime- devices, so guest requests are locally
delivered and meet INPUT. Requests arrived and no reply ever left; cloud-init
spent 490s timing out and reported the endpoint as disabled. With one wildcarded
accept the same image boots in 18.6s and finds its datasource.

The natuplink e2e check had the same blind spot the code did, asserting -C and
therefore presence but never position. It now asserts the accepts precede the
first reject.

* feat: OCI address pairs, and an IMDS remap so the host keeps its own metadata

An OCI external address is two objects, not one: a RESERVED public IP that
OCI 1:1-NATs upstream and never puts on the wire, and a secondary private IP
on the VNIC that the wire actually carries. Every AWS-facing record holds the
public half; every datapath object must hold the private one.

The mapping goes in policy.natManager and nowhere else. AddEIP, DeleteEIP,
PruneOrphanEIPs and the HostEIPBinder all converge there, and reconcile
rebuilds the rule set from AWS-facing intent on every pass — so a correctly
mapped rule installed further upstream is swept on the next reconcile and
replaced with one matching an address that never arrives. The seam is a
DatapathResolver defaulting to identity, so every non-OCI deployment is
unchanged. A resolve failure is fatal: falling back to the public address
would install a rule that reads correctly in ovn-nbctl and blackholes.

ocinet.Lookup is the OCI side — read-only, KV-backed, no client and no
credentials, because vpcd is a different process from the daemon that owns
the allocator and the bindings record is all either would consult.

Two live bugs the in-memory test store was hiding, both found by making the
fake report ErrNotFound the way the real store does:

  - KVStore.Mutate could not create, so every pool failed its own first
    allocation after OCI had already created and billed the pair.
  - Reconcile treated a missing record as a failure, aborting the one pass
    that exists to find an object created before its binding was written.

Guest IMDS must stay on 169.254.169.254, which is also where a cloud host's
own metadata service lives — the first guest launch made it a local address
and took the host's metadata and DNS with it. The datapath is OpenFlow
steered, so the guest-facing addresses stay put and only the host-side
endpoint address moves, with a PREROUTING DNAT on ime-+ translating and
conntrack un-NATing the reply. Off by default; identity when unset.

The IMDS firewall rules move to vpcd. They are wildcarded over every
endpoint rather than per-launch work, and spinifex-daemon holds no iptables
grant and no CAP_NET_ADMIN by design, so attempting them aborted every
launch.

* fix: name the real constructor in the IMDS bind-address error, and mark the NAT datapath test in-package

The error string pointed at DefaultHostBindAddrs, which does not exist — a
caller following it would have looked for a symbol that was never written.

nat_datapath_test.go reuses nat_test.go's in-package fixtures, so it carries
the //test:in-package marker the package check asks for.

* docs: index what each e2e suite proves

Written because a datapath comment was read as an end-to-end behavioural claim
and the suite that already answered it was found only after the wrong answer
had been given. TestVPCEgressPaths/NATBaseline is the authority on private
subnet egress, and nothing pointed at it.

* fix: source-route an EIP's reply out the VNIC that holds it

An EIP's reply carries the EIP as its source. On a host that pins its
uplink with a policy rule, the reply took the main table instead and left
by the wrong interface — which a cloud VNIC drops, because the address is
registered on a different one. Ingress reached the guest and nothing came
back.

Every bind now mirrors the uplink's rule onto the EIP. The selector is the
rule rather than the interface: both OCI VNICs sit in one subnet, so "the
interface owning the EIP's subnet" names two of them, and two candidate
rules is a refusal rather than a guess. A host with no source routing gets
no rule and no change.

* docs: name the setup-ovn recipe routed mode actually needs on OCI

The OCI guide told operators to run `--wan-bridge=br-wan`, which is
mutually exclusive with `--nat-uplink` and deletes the spx-nat transit
veth routed mode runs on. The node then refuses to bring the uplink up,
with an error naming the flag the guide should have used.

Several messages also claimed routed mode is outbound-only with no
public IPs. That stopped being true when the routed EIP path landed, and
it reads as "what you are about to do cannot work" during the one init
the OCI path documents.

* feat: let routed NAT form a multi-node cluster, and carry every pool through formation

Lifts the --nodes >= 2 refusal on --external-mode=nat. The refusal carried no
technical justification -- it was a v1 scope decision -- but routed NAT is the
only datapath a cloud VNIC can carry, so "routed NAT is single-node" had become
"Spinifex on a cloud host is single-node".

It is not fixed by lifting it. Measured on three OCI nodes: the cluster forms,
all chassis register and guest-to-guest traffic runs at 7.1-7.7 Gbit/s, but
every node claims every EIP, nodes disagree on the VPC's gateway LRP, and only
the gateway chassis reaches its own guests. init now says so on stderr rather
than refusing, because the refusal made the defects unreachable and therefore
unfixable. Distributing the EIP datapath is the actual fix.

Lifting it exposed a second defect that stops formation outright. NetworkConfig
held nine flattened Pool* scalars and the init node set PoolName = "wan"
unconditionally; routed NAT renders two pools and the one that matters --
nat-transit, carrying the gateway-LRP range -- was dropped. Each joiner got a
single pool named "wan" holding the transit gateway and the detected WAN prefix,
and the leader re-rendered its own config from the same struct, so all three
nodes were wrong. Pools is now []admin.PoolData in render order, and PoolData
gains json tags because it is on the wire now rather than a template input.

* feat(install-node): add --external-mode so a cluster can be formed in routed NAT

install-node.sh could not express routed NAT at all. It required
--external-pool/--external-gateway/--external-prefix-len and passed
--external-mode=pool to every spx admin init unconditionally, so the only
supported way to form a routed-NAT cluster was by hand.

--external-mode nat refuses the three pool flags rather than ignoring them, and
threads --nat-uplink into setup-ovn.sh -- without it the WAN interface check
rejects a physical NIC during formation, which is correct in pool mode and
wrong in routed mode, where nothing is bridged.

* fix(ocinet): stop each node deleting its peers' OCI addresses on startup

Reconcile compares the binding record against the private IPs on one VNIC. The
record is one key per pool and therefore cluster-wide, so on more than one node
every peer's binding looks exactly like a binding whose objects OCI has lost.
Each node dropped them, and direction 1 on the owning node then collected the
now-unclaimed objects as leaks -- deleting the reserved public IP and private IP
of a running instance.

Observed on the three-node OCI cluster: node2's startup pass logged "collected a
leaked address" for an address it had allocated ten minutes earlier and "dropped
a stale binding" for one living on node3's VNIC.

The damage did not stop at the OCI objects. With the binding gone,
Lookup.DatapathIP returns the public address unchanged, so the OVN
dnat_and_snat row carries the public address instead of the private half. Guest
egress then leaves the VNIC sourced from a reserved public IP, which OCI drops
because it enforces the registered source per VNIC -- and EnsureEIPSourceRoute
no-ops too, since a public address shares no subnet with any host address. One
destroyed binding takes out both directions.

Binding.VNICID already exists for exactly this ("how a node that comes back from
the dead tells mine from moved on without me") and Reconcile never consulted it.
A binding is now only dropped when it names this node's VNIC; one naming no VNIC
cannot be attributed and is left alone rather than guessed at.

* fix(host): find the EIP's source-routing table when the uplink is held as a /32

uplinkSourceRuleFor identified the interface owning an EIP's subnet by finding
a host address configured with a prefix that contains it. On OCI both VNICs sit
in one subnet, so br-wan has to hold its address as a /32 -- a /23 there puts
the subnet's on-link route in the main table at metric 0 and takes the whole
subnet off the primary VNIC. A /32 contains nothing but itself, so no rule was
ever found, no source rule was installed, and the reply left by the primary
VNIC where OCI dropped it for carrying an unregistered source.

The address is not the only record of who owns the subnet. The rule names a
table, and that table holds the on-link route. Consulted as a fallback, only
when the address prefixes matched nobody, so a host that already resolved this
the direct way keeps its existing answer -- prod and dev-prod are unaffected.

Only scope-link routes count. A default route covers every address and would
make every source-routing table an equally good answer.

* docs(oci): correct the multi-node story and name the oci_vnic_id constraint

The nat-mode warning said EIPs and default egress were both broken on more
than one node. Default egress is fine: delete a guest's dnat_and_snat row and
it reaches 1.1.1.1 in 2.3 ms from a guest on any of three nodes. Only the
public-IP path was broken, and the three defects behind it are now fixed.

What remains is not a bug but the shape of the datapath: routed NAT centralises
the EIP path on the VPC's gateway chassis, ext-shared is a localnet so the other
nodes cannot reach it, and OCI drops any frame whose source is not registered on
the VNIC it leaves by. So every node's pool must pin oci_vnic_id to the gateway
chassis's VNIC rather than resolving its own. Say so in §5.3, where an operator
writing the pool will read it, and in §9, where the cost is that a gateway
change kills every public IP until the pools are repointed.

The single-node cap itself is gone from the doc: a three-node cluster has been
formed and measured, and the warning proceeds rather than refusing.

* feat(nat): distribute the routed-NAT EIP path so it works on more than one node

Routed NAT processed every EIP on the VPC's gateway chassis, and ext-shared is
a localnet, so the transit segment a /32 route pointed into existed only on
that one node. Measured on three nodes: all three claimed all three EIPs, the
nodes disagreed on the gateway LRP, and only guests on the gateway chassis were
reachable.

The mechanism to fix it is not new. Pool mode already distributes EIPs by
putting external_mac and logical_port on the dnat_and_snat row and letting
OVN's ARP responder answer for the external IP on the external segment. Routed
mode has an external segment too, the spx-nat veth pair, and it being per node
is the distribution: only the node running the instance can answer ARP on it.

So a routed rule is now distributed on the same terms as a pool one, and its
host route becomes on-link on the transit veth instead of via a gateway LRP
that may be on another chassis. bindHostEIP stops demanding that address for a
distributed rule, which is what previously failed the bind on every node but
one. vpcd plumbs only the ENIs local OVS reports as its own, asked of OVS
rather than the Southbound DB because the question is about this host.

The gateway LRP keeps its transit-pool address. Default egress routes through
it and works across three nodes today, so making it link-local as pool mode
does would break a working path to fix one distribution has already fixed.

Pool and static are unmoved, and a test asserts that against the same input
rather than leaving it to inspection. A rule with no MAC stays centralised, so
a cluster part-way through an upgrade is never in a state neither shape handles.

Beads: mulga-opyl7, mulga-la0pe

* fix(nat): make routed-EIP host plumbing work on every node

Three guests on three OCI nodes, one guest per node: only one was
reachable. Two defects, both invisible on a single node.

The transit veth took a random MAC per node, but OVN holds one MAC
binding for the transit nexthop across the whole cluster — so every
chassis addressed its egress to whichever node seeded the binding, and
the other kernels dropped it at L2. The pair is node-local, so one
locally-administered address everywhere is unambiguous; setup-ovn.sh
pins it at creation and in the netdev unit, and EnsureUplinkPort repairs
a veth made before that.

Host routes and proxy-ARP also rode the leader-gated reconcile pass, so
a node that never won the lease never plumbed its own guests' EIPs.
BindHostEIPs does that half on its own loop on every node; Bind already
leaves alone an EIP whose ENI is elsewhere, so two nodes never contend.
An unresolvable address skips the sweep rather than shrinking it.

Pool and static are untouched: the new pass is a no-op outside routed
mode, and a test asserts it.

Also adds --teardown and --dry-run to setup-ovn.sh, for re-running it on
a node that is already tangled. It deliberately spares the NB/SB
databases, /etc/spinifex, the system-id and the packages, and refuses to
del-br a Linux bridge.

* fix(nat): assert the transit MAC where vpcd actually runs it

EnsureUplinkPort is never called in production — host.Wiring is
constructed nowhere — so the repair landed in a path the deployed
daemon does not take, and three nodes came back with three different
MACs. vpcd re-asserts it beside the NAT egress rules instead, which is
the routed-NAT branch that does run on every start.

* fix(vpcd): make uplink-mode detection say what it saw, and check it

detectBridgeMode logged only the branch it took, so a node detected as
veth when it was wired for nat looked identical to one that was right.
It now reports both probes every time, errors when a node carries both
uplink veths (set up twice, and --teardown is the way out), and errors
when external_mode and the wiring disagree — the datapath follows the
wiring, so the config silently loses that argument. An explicitly
configured bridge_mode gets the same cross-check, since nothing else
ever compared it against the wiring.

The vpc-networking doc now opens nat with "start with pool unless one
of these applies" and gains a section on nat across more than one node;
the admin warning for --external-mode=nat on two or more nodes says
what was verified by hand and what is still uncovered.

* test(nat): prove routed NAT delivers each node's own guests, on a cluster

natuplink ran single-node only, which is exactly where the two defects
that broke three nodes are invisible. It now admits ModeMultinode, and
phase 9 takes a second shape there: a guest per node in a spread
placement group, and for each one — its public IP plumbed by the node
running it and by no other node, the route on-link with proxy-ARP
answered there, a dnat_and_snat row carrying external_mac and
logical_port, a TCP handshake in from another node, and the guest's own
console verdict for egress. Every node's transit veth is checked
against the one cluster-wide MAC before any guest boots.

Single-node phase 9 gains the same row and on-link assertions, so the
shape is held on both topologies.

Two gates come from what three OCI nodes actually do: every node holds
a gateway_chassis row for every VPC (the active one is the highest
priority) and every node installs the VPC ingress route, so only phase
7 needs gating — it runs where the gateway port and the guest are both
local, and logs which one is missing otherwise.

Cell 20 is real-multi x nat with its own nightly step, leaving the
permutation cells' suite set alone.

* test(nat): run the VPC egress paths in nat mode, not pool mode only

The single suite gated TestVPCEgressPaths and TestWANEgressSMTPBlock on
external_mode = "pool", with a comment saying nat clusters have no
EIP/NAT-GW support. They have had EIPs since the routed path landed, so
the gate now asks whether the cluster can allocate public addresses at
all — pool, or nat with a non-transit pool configured.

The nat-single cell runs TestVPCEgressPaths beside natuplink, which is
the only nightly coverage of a private subnet's isolation and its NAT
gateway in routed mode. natuplink never creates a private subnet.

* docs(config): list nat among external_mode values

* fix(ci): renumber the multi-node nat cell off the bare-metal job

Cell 20 is already the bare-metal single-node job, gated on the same
cells input but declared in the workflow rather than e2e-cells.json.
A matrix entry with the same number runs two unrelated jobs under one
name, and naming 20 in a dispatch selects both.

* feat(images): add Oracle Linux 10.1 and 9.8, and pin their catalog digests

Oracle publishes its qcow2 checksums inline on an HTML page and ships no
sums file, so the four new entries carry the expected digest in the catalog
rather than a URL to fetch one from. Images gains ChecksumDigest for that,
and the catalog import path verifies against it and says which of the two
sources it used, so a recorded digest never reads as a fetched sums file.

Pinning is safe here and only here: each URL names an immutable build
(b291, b293, b178, b182) and never moves. aarch64 has a -kvm-cloud- build
alongside the plain -kvm- one; the cloud build is the cloud-init image and
is the one that matches x86_64's -kvm-.

TestImageCatalogReachable HEADs every URL in the catalog from the general
single-node suite. These artifacts live on hosts nobody here controls, so
an upstream that retires a build breaks `spx admin images import` for a
customer and nothing else in the suite would notice. It also asserts that
every entry sets exactly one checksum source, and that no pinned entry
points at a "latest" URL whose bytes would change under it.

* fix(formation): carry bridge_mode onto the leader and every joiner

applyNetworkConfig copied external_mode, external_interface and the pools
out of the formation NetworkConfig but never bridge_mode, so a multi-node
cluster wrote a spinifex.toml without the key while single-node init wrote
it. Every node then fell back to autodetecting a mode that init had pinned.

Nightly cell 30 caught it: TestNATUplink failed at the config phase on a
spinifex.toml that did not contain bridge_mode = "nat". Deriving it from
the mode rather than adding a NetworkConfig field keeps the leader and the
joiners on the same rule single-node init already uses.

* feat(ui): show how to ssh into an instance, and which user to use

An instance detail page gave the addresses and the key pair name but never
the login user, which is the one part of the command nobody can derive. It
belongs to the AMI, not to Spinifex: nothing in the launch path creates or
renames an account, so stock cloud-init applies the distro's own
default_user. Oracle Linux is cloud-user, Debian is debian, Ubuntu is
ubuntu, and guessing from the distro name gets two of those three wrong.

The card resolves the user from the AMI's name and description, and an
unrecognised image yields a placeholder rather than a guess, because a
wrong username in a copy-paste command is worse than an honest gap. A
private-only instance is called out as reachable from inside the VPC only,
rather than presented as a command that will simply time out.

* fix(nat): plumb host state for a NAT gateway's public address

AddNATGateway installed the OVN snat row and stopped there. On a routed-NAT
node the host is what delivers a public address, so an egressing private guest
left with a source the host held no route back to, and the masquerade rule —
which matches the transit 100.127.0.0/24 alone — never saw the packet. The
gateway's address needs the same /32 route into OVN, proxy-ARP and FORWARD
accepts an EIP gets.

hostBindSpecs unions floatingIPSpecs with one deduped spec per NAT gateway
address and feeds BindHostEIPs. It is kept apart from floatingIPSpecs
deliberately: pruneOrphanEIPs and addLive read that set as the complete list of
addresses allowed to hold a dnat_and_snat, and a gateway holds an snat, so
widening it would stop the sweep reclaiming a stale row on an address later
reused as a gateway's.

AddNATGateway also binds directly, on both the create and idempotent-skip
paths, so a fresh gateway is not dark for up to 30s waiting on the host EIP
pass — and so a reboot that keeps the OVN row still replays the volatile half.

Documents NAT gateways in nat mode in vpc-networking, and says at the head of
that section that nat is a supported mode covered by nightly cells 19 and 30.

* docs(oci): document Oracle Linux as a guest image, and that its user is cloud-user

Four catalog entries, all UEFI. The login user is the one that catches
people: opc is the default on the images Oracle publishes into OCI itself,
while these are the generic KVM cloud images from yum.oracle.com, whose
cloud-init default_user is cloud-user. Verified on 10.1 and 9.8 — opc, oracle
and ec2-user are all refused.

Also records why these four pin a checksum digest inline (Oracle ships no sums
file, and each URL names an immutable build), why arm64 takes the -kvm-cloud-
build, and that SELinux ships enforcing.

* fix(reconcile): stop reading an unreadable intent as an empty one

A wedged JetStream KV returned no keys for the ENI bucket while writes were
still landing. loadPorts mapped that to an empty port map, pruneOrphanPorts
read every guest LSP as an orphan against it, and every guest on the node lost
DHCP and IMDS while the API kept answering and the instances stayed "running".
The existing re-read guard did not help: it re-reads through the same path.

Three defences, because no single one covers every shape of the fault.

A failed kv.Get now fails the pass instead of dropping the record. Dropping one
VPC narrows localVPCs, which filters out every ENI in it, which the sweep reads
as permission to delete every guest port in that VPC — one failed read, same
outcome, and it needs no exotic KV state. Unmarshal failures stay skips: that
is a poison record rather than an unhealthy store, and failing on it would
wedge reconcile forever.

The port sweep now asks local OVS which taps are plugged in here and never
deletes a port that has one. A running guest's port cannot be an orphan
whatever intent says, and OVS is the one liveness signal that does not come
through the KV read path the sweep is otherwise entirely driven by. Wired from
vpcd; unwired callers fall back to the guard below. An unreadable OVS skips the
sweep rather than deciding on intent alone.

Each sweep also refuses outright when the keep set it would delete against
holds nothing at all — no ENIs, no SGs, or no ports and no external IPs. That
is the case local taps cannot cover, because the drift sweep is leader-gated
and lists NB cluster-wide, so the leader holds no tap for another node's guest.
The cost is real: a genuinely orphaned row now survives until one live record
of its class exists. Leaking a stale row is the recoverable half.

Three existing prune tests drove their fixtures to a wholly empty intent, which
is now the refusal case; they keep one live record of the class so they still
prove the prune. The Get-failure path has no unit test — there is no
jetstream.JetStream fake in the tree and an embedded server cannot be made to
list a key it will then refuse.

* docs(oci): separate the two address quotas, and the principal that can read them

The regional reserved-public-IP ceiling is tenancy-wide and does not grow with
node count, so a cluster hits it before the per-VNIC 64, which does. Records
the commands that read both, and that a detached reserved public IP still
bills.

Reading limits or usage needs a tenancy-level grant, which cannot be
compartment-scoped the way the node's policy is. That is a second, read-only
principal kept off every node, not an addition to SpinifexOperators.

* docs(oci): give quotas their own troubleshooting section, and say we enforce none

Spinifex hardcodes no address ceiling: it attempts the allocation and maps
OCI's own refusal to InsufficientAddressCapacity. So a quota is raised with
Oracle and nothing here is reconfigured or restarted — which is what an
operator needs to know before it bites, not after.

Separates the two quotas by how they scale: the regional reserved-public-IP
default is tenancy-wide and raisable by ticket, the per-VNIC 64 is per node and
is not.

* fix(oci): follow a guest to its new node when its EIP moves

OCI delivers a public address to whichever VNIC holds the private IP behind
it, and nothing ever moved that private IP. Binding.VNICID was written once at
allocate time, as the VNIC of whichever queue-group daemon happened to serve
the AWS request, and never updated — so an address landed on the right node
only by chance, and a guest that stopped and started elsewhere went dark while
every OVN row, host route and firewall rule on the new node was correct.

ClaimLocalAddresses closes the gap from the node that should hold the address.
Local OVS decides ownership, the same authority the host EIP binder already
trusts for the same question, and UpdatePrivateIp with a new VnicId does the
move in one server-side call. The private IP OCID survives it, so the reserved
public IP attached to it follows without being touched and the customer's
address never changes.

A loop per node, not leader-gated: the question is about this host's guests,
and a node that never wins the reconcile lease would otherwise never claim the
addresses of the instances it is running.

* fix(oci): record which ENI an EIP is associated with

An EIP is allocated bare and attached by a later AssociateAddress, so the OCI
binding written at allocate time named no ENI and nothing ever filled it in.
The address then stayed on whichever node happened to serve the allocation:
the affinity pass could follow an auto-assigned address, which is allocated
with its ENI, but never an EIP — the case customers actually use.

BindOwner is on an optional OwnerTracker interface rather than Allocator,
because only OCI has a notion of an address being delivered to one node. A
static or DHCP address is on the wire everywhere its node can reach, those
allocators do not implement it, and ExternalIPAM treats a pool that does not
as a success with nothing to do.

The same gap made the release owner guard inert for EIPs: it compares against
a binding ENI that was always empty, so a stale teardown could free an address
already handed to another instance.

* fix(ec2): report the address an instance is actually reachable on

Three faults kept DescribeInstances answering with the address launch assigned,
whatever happened afterwards.

The launch-time stamp lives on the stored ec2.Instance, and the stopped path
only declined to add an address rather than removing one, so a stopped instance
went on advertising an address it had released and that may already belong to
someone else. The projection now clears it, and the NIC association with it,
working on its copy so a restart can still report the address.

The running path filled the public address only when it was nil, which the
stamp never is, so an EIP could not be reported however correctly it was
recorded. The runtime address now wins.

And nothing told the rest of the system about an association: AssociateAddress
wrote neither the ENI record, which is what IMDS and the network reconcile
read, nor the instance record, which is what a describe answers from. The ENI
write is direct; the instance write is a fan-out, because the queue-group
worker that serves AssociateAddress is rarely the node running the instance.

* fix(net): take the host EIP plumbing off a node its guest has left

Bind refuses an EIP whose ENI is on another chassis; the prune beside it kept
anything still wanted cluster-wide. A guest that moved therefore left the old
node holding the /32 route into OVN and the proxy-ARP entry on the uplink, with
nothing that would ever remove them. On a shared segment that node goes on
answering ARP for an address another node is now serving.

The two halves now ask the same question by the same authority, in the loop
that already runs per node. An unreadable local OVS is not a yes: tearing down
a live address on a guess would fire everywhere at once, so it keeps the
binding and the next pass tries again. A binder with no chassis test behaves
exactly as before, and a centralised EIP is exempt because it hairpins through
the gateway chassis and every node plumbs it legitimately.

* fix(net): refuse to free an Elastic IP on behalf of a departing interface

Releasing an instance's public address on teardown releases it by address,
scoped to the ENI holding it. Since an association now writes the EIP onto the
instance record, that address is the EIP — so terminating an instance gave a
customer's Elastic IP back to the pool while describe-addresses still reported
it as theirs, free to be handed to another guest.

The existing ownership guard cannot catch this. It only fires when the stored
allocation already names an ENI, and an EIP is allocated bare and associated
later, so the stored ENI is empty and the guard passes straight through.

The distinction that holds is who is asking. ReleaseAddress is the address's own
owner and names no interface; instance teardown names the interface it is tearing
down. So a release that names an owner may free an auto-assigned address and
never an EIP, which is enforced in each of the three allocators.

DHCP needed the scope carried on the wire, since it releases by address and
resolves the lease afterwards. Absent on an older caller, which reads as false
and keeps the previous behaviour.

Worst on the two providers where it is not recoverable: on DHCP the lease is the
only thing holding the address and the server promises nothing about handing it
back, and on OCI the address is a reserved public IP object that the release
destroys outright.

* test(e2e): prove an Elastic IP survives a stop and start

Instance teardown releases the public address by value, and since the
association writes the EIP onto the instance record, that release aims
straight at the customer's Elastic IP. The allocator guards now refuse it,
but nothing end to end proved the address comes back on a restart.

The step goes inside EIPFlip, which already runs last on the shared public
guest for the same reason a stop is safe there: the address the guest
returns on is the allocation's, not the pool's. It runs on the static
cells and the DHCP ones alike, and on DHCP it doubles as the only outside
view of the lease still renewing while the guest is down.

* feat: release an auto-assigned public address on stop, take a new one on start

AWS lends an instance a public address for as long as it runs and gives it a
different one on the next start. Spinifex kept the address across a stop, which
holds a pool slot for a stopped instance and, on a DHCP pool, keeps renewing a
lease nothing is using.

A stop now offers the address back through the instance cleaner, which refuses
anything an Elastic IP is behind — the EIP association record answers that, the
same question an ENI delete already asks — and records on the VM that the start
owes it a new one. A host drain keeps the address: Restore relaunches the guest
on the next boot, and changing its address under it is an outage nobody asked
for.

The start allocates before the launch, so an exhausted pool refuses the start
rather than booting a guest the customer cannot reach, adopts an Elastic IP
associated while the instance was stopped instead of assigning a second address
beside it, and hands the address back if the launch then fails.

e2e: a StopStartAddress stage on the public guest asserts a stopped instance
reports no address and comes back on a different one from a static pool, and a
StopStartKeepsTheEIP step asserts the opposite for an Elastic IP.

* docs: restructure the OCI guide around two deployment paths, with diagrams

The guide had grown into a numbered wall with no shape: no frontmatter, no
contents, and a reader with an OCI account had no way to tell where to start.
It now follows the multi-node install template — overview, prerequisites, then
the deployment path — and splits that path in two: Terraform for one node or
three, which is the preferred route, and a manual single-node walkthrough for
an instance that already exists. Both meet at one shared configuration section
rather than repeating it.

Two ASCII diagrams carry what the prose could not: one OCI instance with the
whole AWS surface inside it, and three nodes with the same surface distributed,
showing which plane each VNIC carries and how an address follows a guest that
moves.

The limits section was materially out of date and said so in the places that
mattered most. Multi-node is no longer experimental, public addresses are no
longer pinned to one node — each allocates on its own VNIC and the affinity
pass moves the private half to wherever the guest is — and an auto-assigned
address is now returned to OCI on stop, so a stopped fleet stops paying for
addresses. What changed is recorded rather than quietly edited.

The Terraform configuration moves in from the monorepo as
scripts/terraform/oci-spx, so the guide points at something in this repository.

Also marks the two new lifecycle test files in-package, with the reason each
one needs to be: both substitute unexported dependencies.

* docs: rework the OCI guide — SVG diagrams, Terraform variables, iSCSI and br-wan

Replaces the two ASCII diagrams with SVGs in the house style, one for a
single node and one for three. Renames the two deployment sections to
Terraform deployment and Manual deployment, documents what the Terraform
builds per node and every variable worth setting, and explains that the
data volume is an iSCSI attachment rather than a disk.

The manual path now covers the iSCSI login, the fstab entry and the
netplan source routing that Terraform writes, and states what br-wan is
for on a routed-NAT node: it names the VNIC for the OCI allocator and
holds the registered addresses, but is never bridged into OVS.

Opens with what Spinifex is and why OCI is worth running it on.

* fix: make gateway convergence work on a multi-node routed-NAT cluster

Three defects each stopped a multi-node cluster from ever attaching an
internet gateway, and all three produced the same uninformative symptom:
`terraform apply` sitting on `aws_internet_gateway.igw: Still creating...`
forever while every service reported healthy.

ovn-northd was given only its local OVSDB socket as a client connection.
A RAFT follower does not forward writes -- it answers "not cluster leader,
trying another server" -- so with one entry in the list, northd stops
translating Northbound intent into Southbound flows the moment database
leadership moves to another node. Measured on the OCI cluster:
SB_Global.nb_cfg seventeen transactions behind NB_Global.nb_cfg, with no
chassisredirect port for any VPC created after leadership moved.
setup-ovn.sh wrote the member list only when the caller passed
--ovn-remote, which only install-node.sh did; it now derives the list from
the live RAFT membership otherwise, and says so when it finds a single
member, because that node needs a second pass once the others join.

The gateway datapath probe ran wherever the reconcile lease landed rather
than on the chassis owning the port. Both probes leave from the local
uplink, and under routed NAT the transit subnet exists identically on every
node, so a remote probe reported a healthy gateway as dead and aimed a
repair at a datapath the node does not own. GatewayPortLocal gates it.

On OCI the probe then ARPed the tenant-visible public IP, which the cloud
NATs and which appears on no interface anywhere. The reconciler now maps
the address through the same DatapathResolver the NAT rules are built from.

Also adds DescribeVpcEndpoints, which the AWS provider calls on every VPC
delete to sweep GuardDuty-managed endpoints; an InvalidAction there failed
the destroy and stranded the VPC.

The OCI guide gains a Terraform section recording which shipped workbooks
were validated against a three-node OCI cluster and which are still open.

* docs: record the validated Terraform workbook results for the three-node OCI cluster

Six of the seven workbooks run against the OCI cluster pass, including the
three the section exists to prove: RDS, ECS and EKS. Records nginx-alb and the
clean eks-quickstart re-run, and names the two lifecycle gaps that bit during
the run so an operator recognises them rather than reading them as a broken
workbook.

* fix: route the NAT gateway's SNAT source through the datapath resolver

AddNATGateway was the one NAT path that did not map its external address to
the one that rides the wire. Where the public half of an allocation is NAT'd by
the cloud it is on no interface anywhere, so every private-subnet guest egressed
with a source that was dropped, while the gateway reported available and DNS
kept working because the VPC resolver is host-local.

The unmapped address also failed the host EIP sweep, which works in resolved
addresses, so the bind was pruned as foreign on the next pass and reinstalled on
the one after. The public half now survives as an external ID instead.

Also restructures the OCI guide around numbered steps through to running
Terraform, and moves the cloud-init and metadata sections into troubleshooting
where they are read when something is wrong rather than followed when it is not.

* fix: place a NAT gateway's OCI address on the node that forwards for it

OCI drops a packet whose source is not a registered private IP on the VNIC it
leaves from, so a NAT gateway's SNAT source has to be on the gateway chassis.
Two independent placements decided those and nothing reconciled them: the
address was created by whichever worker served AllocateAddress, while the
chassis is a fixed ordering the reconcile pass asserts. On a three-node cluster
that made private-subnet egress a coin flip, and the same Terraform passed or
died depending on which node answered.

The affinity pass already solves this for guests, by asking local OVS where the
tap is. A NAT gateway has no tap, and it skipped anything with no ENI — which
is exactly and only this address. The equivalent local fact is the VPC's
chassisredirect binding, so ownership now has two answers rather than one, and
an address that is neither a guest's nor a gateway's is still left alone.

The VPC is recorded from the reconcile pass rather than from CreateNatGateway,
because the answer is the gateway chassis and reconcile is what decides it.

* fix: plumb a NAT gateway's host ingress only on its gateway chassis

A NAT gateway's address has no ENI, so neither the bind's locality test nor
the prune's could ever fire for it and every node installed its /32 route into
OVN. Latent on bare metal, where pool addresses are disjoint from the node
subnet; on a cloud that hands out the address from the nodes' own subnet it
outranks the connected route and swallows node-to-node traffic.

Give that address the same second authority the ocinet affinity pass has: the
VPC's chassisredirect binding, read from the local ovn-controller. An
unanswerable question still leaves the binding alone, so a single node whose
gateway port is not yet claimed behaves exactly as before.

The orphan EIP prune built its live set from ENI and user EIPs alone while also
sweeping host ingress, so every drift pass tore down the ingress of every live
NAT gateway and left it dark until the next host EIP tick.

* fix: ask the gateway chassis only about a NAT gateway's address

gatewayElsewhere skipped the NAT gateway marker it was named for and answered
for every spec, so on a node that is not the VPC's gateway chassis every EIP —
including a guest's, whose locality is its own port — was left unplumbed. Two
of three nodes bound nothing.

* test: assert the two private-subnet EKS workbooks in the tofu harness

eks-https-ingress and eks-gitops-argocd put their workers in private subnets
behind a NAT gateway, so a worker only registers Ready if its egress to the
cluster endpoint and to ECR survives that path. That makes them the workbooks
that catch a NAT gateway fault, and neither had an assertion, so neither could
be run at all — run_workbook calls assert_<name> by construction.

Both share eks-quickstart's readiness check, extracted unchanged. They also
name the worker size node_instance_type rather than instance_type, and tofu
rejects a -var for an undeclared variable outright, so the name is read from
the config instead of kept in a list here.

* test: count agents, not every node, when asserting EKS workers Ready

The control plane is three K3s servers, so an unfiltered node count reaches a
one-worker cluster's desired size before any worker joins — the assertion
passed on exactly the failure it exists to catch, reporting "4/1 workers
Ready". Filtering on the control-plane role label makes it 1/1, which is the
number that says a node-group worker registered.

Matters most for the two private-subnet workbooks, where the worker joining is
the whole proof that egress through the NAT gateway works.

* docs: record the two private-subnet EKS workbooks passing on OCI
 2026-09-26 14:31:07 -05:00
Julian Sommer 6d4a66be90 fix(sts): reject session policies and tags as unsupported, and echo SourceIdentity (#1105) 2026-09-24 16:31:03 +10:00
Julian Sommer cb83f9b193 fix(bedrock): GetFoundationModel answers an unknown model with ValidationException (#1104) 2026-09-24 16:22:33 +10:00
Julian Sommer 372991c304 feat: implement EC2 ModifySecurityGroupRules (#1103) 2026-09-24 15:20:57 +10:00
Julian Sommer 28963e28af feat: verify local image imports with --checksum and record AMI source digests (#1102) 2026-09-23 15:15:23 +10:00
Julian Sommer 85dd1dcd75 docs: stop skipping the account ID lookup in Terraform provider blocks (#1101) 2026-09-23 14:13:04 +10:00
Julian Sommer da2e6c5c7b chore: update coverage 2026-09-23 14:10:35 +10:00
Julian Sommer 71059834e7 feat(gpu): multi-GPU instance types (#1099) 2026-09-23 09:02:09 +10:00
Josh Siviour 5c5a2bc0b7 fix: omit the user data attribute value when an instance has none (#1097) 
DescribeInstanceAttribute always returned a Value for userData, an empty string when the instance was launched without any. AWS returns the attribute with no Value at all in that case.

The Terraform provider reads user_data only when Value is non-nil, so an empty string made it record the hash of the empty string in state and propose a change on every plan for an instance whose configuration names no user data.
 2026-09-21 15:15:27 +10:00
Josh Siviour 259b737f7a fix: size each EBS provider deadline to the work its subject names (#1096) 
One 30s timeout bounded every ebs.provider.v1.* request. capabilities, volume.describe and volume.list answer from state already held; volume.create clones an AMI snapshot. The cheap calls set the budget and the clone inherited it.

On a freshly restarted cluster the first clone of an AMI does not finish inside that budget, so RunInstances took a context deadline for a preparation failure and terminated the instance, taking its ENI and volumes with it. A retry of the identical request succeeded once the snapshot had been read once.

The subjects that move or reference data now carry 180s, matching unmountSealTimeout, the existing budget here for an EBS operation the caller cannot bound. The classification lives with the subject constants, so a caller constructing a provider cannot reintroduce one flat value, and a new subject stays fast until someone classifies it.

Retrying a timed-out create is deliberately not attempted: the responder may still be running one, and that needs idempotency this change does not establish.
 2026-09-21 14:16:51 +10:00
Julian Sommer 0ae2826cca chore: update pnpm 2026-09-21 10:54:11 +10:00
Josh Siviour 2dee58d749 fix: default the root block device to the name the AMI declares (#1095) 
RunInstances defaulted the root device to /dev/vda, the path the guest sees, so an instance launched without a block device mapping reported a DeviceName that contradicted its own RootDeviceName of /dev/sda1.

ECS capacity and EKS nodegroups named the same guest path explicitly, so every internally launched instance carried it too. All three now name ebsmetadata.RootDeviceName, which is what AWS returns and what the guest already resolves through its own device links.
 2026-09-21 10:12:45 +10:00
Julian Sommer b53ac2d148 chore: frontend deps 2026-09-21 09:50:41 +10:00
dependabot[bot] 7cc8f59a7d chore: bump the go-dependencies group with 5 updates (#1094) 2026-09-21 09:01:03 +10:00
Josh Siviour 4fc99a5e3e fix: report the block device name the caller asked for (#1093) 
BlockDeviceMappings[].DeviceName was overwritten with the path the guest kernel assigns, on every launch, start and hot-attach. The name is ForceNew on aws_instance.ebs_block_device and root_block_device, so a value that never reads back as it was sent is not drift to reconcile: every plan destroyed and recreated the instance and everything on its volumes, and a stock terraform-aws-modules/ec2-instance configuration never converged.

The convention it served — lsblk inside the guest showing the same names as DescribeInstances — is not one AWS keeps either. On Nitro the API reports /dev/sda1 while the guest shows /dev/nvme0n1p1, and IMDS block-device-mapping reports the API name too. The API name is a label, and the wire contract owes the caller the label it supplied.

UpdateGuestDeviceNames becomes LogGuestDeviceMap: the guest paths are still discovered and logged for an operator correlating a volume against what the guest sees, and no longer recorded as the device the caller asked for. AttachVolume records the API name it already returns and already persists to volume metadata. IMDS projects the same field and so becomes AWS-shaped with it.
 2026-09-21 08:37:34 +10:00
Josh Siviour 8b5e23da70 fix: return the OIDC provider URL with the scheme it was created with (#1092) 
The Terraform AWS provider compares the stored url against the configured one through suppressOpenIDURL, which parses the stored value and forces the https scheme onto it. An issuer carrying a port is not parseable without a scheme — Go reads the host as one and rejects the rest — so the parse fails, the suppressor returns false, and the ForceNew url attribute replaces the provider on every plan. Every issuer this platform mints carries a port, so the AWS-shaped stripped form is the one that cannot round-trip here. The ARN keeps the stripped host-and-path form, which is what AWS uses and what issuerFromOIDCProviderARN reverses.
 2026-09-21 08:37:14 +10:00
Josh Siviour c2332746f3 fix: retry a recovery-failed instance and stop calling a dead record running (#1091) 
An instance whose relaunch fails during daemon-startup recovery goes to
StateError, and nothing ever retried it: every later restart read the same
StateError and skipped it again, permanently. The usual cause is a dependency
that was not ready at boot, which is over seconds later, so the instance stayed
down for a condition that had already cleared.

The manager already bounds restarts for a crashing instance, through
Health.CrashCount and Health.FirstCrashTime against MaxRestartsInWindow. A
recovery failure wrote none of it, so it was invisible to the only budget there
is. MarkRecoveryFailed now counts one, the window's two rules move into
recordCrash and rollRestartWindow, and MaybeRestart is rewritten over them with
the same constants and comparisons it had.

With a failure counted, restore can retry: an instance in StateError with budget
left returns to Pending and goes through the existing relaunch, including its
mount-not-ready backoff. With the budget spent it stays in StateError for the
operator and is only announced to the recovering hook, as before. The window
expires, so a later restart tries again rather than writing the instance off.

StartStoppedInstance is the operator's remedy and it was refusing to work. The
shared stopped store never holds a StateError record, so the load missed, and
the fallback treated the presence of the dead local record as proof the instance
was already running. It reads the record's state now. StateError is started
locally through StartInstance, which already accepts it; terminated answers
InvalidInstanceID.NotFound; everything else keeps IncorrectInstanceState,
because a claim inserts the instance before it launches, so a local record in a
startable state is a start another caller is part-way through.
 2026-09-19 13:02:26 +10:00
Josh Siviour a74052fd6f fix: keep the ecs-agent dialling containerd and brake a deployment that cannot start (#1090) 
The agent resolved the container runtime once, in New. A containerd socket that was not accepting during the first second of boot left the agent runtime-less for the life of the process, and nothing retried. It registered and heartbeated normally, so the scheduler saw a healthy instance and kept placing tasks that died immediately at the nil runtime check.

The runtime now lives behind a gate that a watch loop keeps current: it dials while the gate is empty, and probes with a container list while it is full, clearing the gate when the probe fails so the next tick re-dials. That covers a socket that is not ready yet at boot and a containerd restarted underneath a running agent alike, and everything that used the puller or the runner reads through the gate, so a runtime arriving late is picked up without a restart.

An agent with no runtime advertises no capacity. Registration and the heartbeat report zero CPU and zero memory while the gate is empty, so placement skips the host, and the control plane reads that pair of zeros as an involuntary drain: the instance becomes DRAINING with agentConnected false and zero registered resources, which is visible through DescribeContainerInstances rather than only in the node's log. Capacity returning clears it through the branch a reaped instance already used. A host with no CPU cannot exist, so the zero needs no new wire field. An operator drain is no longer cleared by a bus registration either, which it previously was.

A deployment whose tasks keep failing to start now backs off. Deployment gains NextLaunchAt, and past the failure count the circuit breaker already tracks, the reconciler widens the gap between launch passes up to a cap. It is keyed on that counter but applies whether or not the breaker is enabled, because AWS leaves it off by default and an unbraked service relaunches a task that cannot start on every pass, forever.
 2026-09-19 12:47:55 +10:00
Josh Siviour 32bcada227 fix: make an error-state instance terminable and stop reporting a refusal as terminated (#1089) 
An instance left in StateError could not be terminated through any path, and the gateway reported success anyway, so a Terraform destroy waited on a record the platform had already claimed was gone.

classifyRestoredInstances skipped a StateError instance with the comment that the operator must retry or terminate it, but continued before firing OnInstanceRecovering, which is what binds ec2.cmd.<id>. The topic the terminate travels on therefore had no responder and the stated remedy was unreachable.

With no responder there, TerminateInstances fell back to the ec2.terminate topic, whose handler reads the stopped KV bucket only and answered InvalidInstanceID.NotFound for a record held in the active registry. The gateway discarded that payload without reading its code and took the idempotent-absence branch, returning terminated for a record it had not removed.

The StateError branch now announces the instance to the recovering hook before skipping relaunch, and the gateway reads the refusal: only a genuine absence falls through to the idempotent branch, and any other code is logged and returned.
 2026-09-19 12:47:34 +10:00
Josh Siviour db5e446338 feat: create the volumes a launch names beyond the root device (#1088) 
RunInstances created exactly one volume, the root, plus the EFI variable store, and refused a request naming any other EBS device. An instance could not be launched with a data volume attached, so terraform-aws-modules/ec2-instance with an ebs_block_device failed the apply and the workaround was CreateVolume plus AttachVolume afterwards.

The launcher was never the constraint: buildDrives already has a data-volume branch and already allocates a hot-plug port for a volume attached at RunInstances time. The control plane now creates a volume for every EBS mapping through the volume service, which is where CreateVolume's validation and metadata write already live, and records each as a non-boot request at the device name the caller named.

A data volume is created empty, so a mapping naming a snapshot or naming no size is still refused by name rather than served as something it did not ask for. A launch that fails partway deletes the volumes it created.
 2026-09-19 12:47:14 +10:00
Josh Siviour 2ee22e0f66 fix: honour the status filter on ListContainerInstances (#1087) 
The filter was never read, so ACTIVE, DRAINING and no filter at all returned every registration in the cluster. A caller could not ask which container instances can take work without describing every one of them, and a cluster carrying a drained node read as if every node were live.

The status a record holds is the one DescribeContainerInstances reports, so a filtered list and a describe of what it returns agree. REGISTERING, DEREGISTERING and REGISTRATION_FAILED are accepted and match nothing, because no record here ever holds them; a status outside the enum is refused.
 2026-09-19 12:46:05 +10:00
Josh Siviour eb365eca7b fix: pin predastore to the object sub-resource guard (#1086) 
An unserved object sub-resource fell through to PutObject, GetObject or DeleteObject, so a write to one stored the object itself. The pin also carries the meta leader health check fix and the router's single query parse.
 2026-09-19 08:41:14 +10:00
Ben Duncan 2ba8f5e4ab docs: separate the install guides from the reference material they had absorbed 
Both install guides had grown sections that were not install steps, which is what made the multi-node one hard to follow. Each now ends on a step that hands off to Setting Up Your Cluster, and the reference material moves under Administration where it can be read on its own.

- Single-node and multi-node both end on a "Set Up Your Cluster" step rather than a trailing sentence.
- The multi-node cluster-state check via `spx admin kv digest` is gone. Verification is now every node Ready, capacity pooled, and the AWS API answering.
- "How Multi-Node Storage Works" becomes docs/admin/predastore-distributed-storage, expanded with the host and node model, the write and read paths, degraded writes, hinted handoff, repair, encryption at rest, and what each class of failure costs.
- "Firewall and Cluster Membership" becomes docs/admin/host-firewall, expanded with the full public, cluster and encap port groups, how to narrow SSH, and which files under /etc/spinifex/firewall survive an upgrade.
- Setting Up Your Cluster gains a closing step that arms the firewall and narrows SSH, so the securing work lands after a cluster that works rather than during the install.

install-usb and vpc-networking followed the two moved sections to their new paths.
 2026-09-18 10:58:25 -05:00
Josh Siviour e035842503 fix: report the root volume as it was requested (#1082) 
Three omissions left the launched root volume reading back as something the caller never asked for.

DescribeInstances returned a null RootDeviceName and RootDeviceType while DescribeImages populated both for the same AMI, so the two answers disagreed about the same image. The launch path now projects both onto the instance.

The root volume's EBS request carried no device name, so the launcher wrote an empty string to the volume's metadata and DescribeVolumes reported an attached volume with an empty Attachments[].Device. The name parseVolumeParams already resolved now travels with the request.

A volume-scoped TagSpecification was discarded: only instance-scoped tags were read, and the comment claimed volume specs were handled elsewhere, which nothing did. They are now written to the root volume's metadata document.

BlockDeviceMappings[].DeviceName still reports the guest path rather than the requested name. That rewrite is deliberate and is left alone here.
 2026-09-18 17:11:04 +10:00
Josh Siviour 0b93b1a957 fix: route the EKS update reads so they refuse instead of reading as a typo (#1084) 
* fix: route the EKS update reads so they refuse instead of reading as a typo

UpdateClusterVersion was registered at POST /clusters/{name}/update-version, which is the nodegroup pattern and not a path the SDK ever calls; AWS puts it at POST /clusters/{name}/updates. The refusal written for it was therefore unreachable, and the router answered InvalidAction, which tells a caller they typed something that does not exist.

ListUpdates and DescribeUpdate had no route at all and answered the same way. Both now refuse with NotImplemented at the gateway rather than through a service method: the update surface they would read is absent, so a round-trip to reach a constant refusal is surface for its own sake.

The absence of the cluster-level update surface is unchanged. This only stops the platform blaming the caller's spelling for its own routing.

* test: account for the two EKS update reads in the cross-tenant table

Both name their cluster by a path segment that carries no account segment, so
neither can name another tenant's resource and neither needs a case of its own.
 2026-09-18 17:10:48 +10:00
Josh Siviour 664db686e1 fix: ack an ECS stop directive only after the reap reports STOPPED (#1085) 
The agent acked a stop in the same loop iteration that launched the reap, so
the directive was deleted before the reap had done anything. When the reap
could not run — no container runtime, or a failing List — nothing re-posted it,
and the task held its ENI, its capacity reservation and its target-group
registrations forever. The in-memory `stopping` map made that permanent: it was
set at dispatch and never cleared, so the agent would not re-dispatch even if
the directive did come back.

A stop ledger now separates a reap in flight from a reap that reported STOPPED.
Only the second is acked, so a failed reap leaves the directive in the inbox to
be redelivered and retried, while the assign loop keeps reading both halves and
so still refuses to start a task that is stopping or stopped.
 2026-09-18 15:47:33 +10:00
Josh Siviour e2db95d746 fix: persist security group rule tags and report -1 ports for all-protocol rules (#1083) 
* fix: persist security group rule tags and report -1 ports for all-protocol rules

A security group rule accepted a security-group-rule TagSpecification and kept
none of it, so DescribeSecurityGroupRules answered with no tags and every
subsequent plan offered to add them again. SGRule now carries a Tags map,
populated on both authorize paths and projected into the central tag store, and
DescribeSecurityGroupRules evaluates tag: and tag-key filters against it instead
of excluding every rule.

CreateTags and DeleteTags on an sgr- id were a silent no-op because
updateRecordTags routed only subnet-, vpc-, sg- and eni-. It now finds the rule
inside its parent group's record and mutates it there, and a revoked rule's
central entry is cleared with it.

An all-protocol rule reported the ports it was stored with, 0 and 0, where AWS
reports -1 and -1, which is the same never-converging plan by another route.

* fix: recognise sgr- as a security-group-rule resource id

DescribeTags reported ResourceType "unknown" for a security group rule, so a
resource-type filter naming security-group-rule matched none of them, and the
authorizer could build no ARN for a rule id and fell back to the any-resource
wildcard. Both read the same prefix table, which had no entry for sgr-.
 2026-09-18 15:47:16 +10:00
Josh Siviour ccc7478f19 fix: select the root block device mapping by name and refuse the rest (#1081) 
RunInstances read BlockDeviceMappings[0] and nothing else, so a request carrying a root mapping and a data mapping created one volume sized from whichever happened to be first, and the first mapping became the root whatever device it named. A module emitting the data mapping first produced a root volume of the data volume's size and no data volume at all, with no error.

The root is now the mapping naming the AMI root device, with position deciding only when nothing names it, so the single /dev/vda mapping ECS and EKS launch with still works. Any further EBS mapping is refused in ValidateRunInstancesInput with InvalidParameterValue naming BlockDeviceMappings and the devices it cannot attach, rather than being discarded silently. Ephemeral mappings carry no volume and are ignored.

Creating the additional volumes at launch remains unimplemented; this makes its absence visible.
 2026-09-18 15:47:05 +10:00
Josh Siviour f536b1a0cb chore: pin predastore to the revision that serves bucket versioning (#1080) 
The deploy builds with GOWORK=off, so spx links the pin rather than the sibling
checkout, and the versioning handlers are absent from every deployed binary
until it moves.
 2026-09-18 09:28:35 +10:00
Ben Duncan deff3a1e1e chore: dispatch the E2E Nightly from Cloudflare instead of a GitHub schedule (#1076) 
GitHub registers a schedule: only from a repository's default branch, so this cron could drive exactly one branch and always executed main's copy of this file whatever it tested. main was never exercised by the nightly, and a fix to the workflow itself could not be proved until it had already shipped — which is how cell-24's unguarded pkill stayed live on the schedule after its fix merged to dev.

A Cloudflare Worker now dispatches this workflow once on dev and once on main. A dispatch takes both the workflow file and the source under test from its ref, so GIT_BRANCH is simply github.ref_name and every schedule-only expression goes with the cron.
 2026-09-17 15:17:29 -05:00
Ben Duncan 2039c05c17 fix(ci): take the host benchmark lock before killing predastore nodes (#1065) 
cell-24 clears leftover predastore nodes with `pkill -KILL -f '/bin/s3d
-config'`, which matches every predastore node on the runner and not only the
ones this run left behind. The predastore repository runs the same harnesses on
the same machine, so "still running" does not mean "stale" — on 2026-09-16 this
step killed a predastore benchmark 32 seconds into its workload, and the run
failed with 61 connection-refused errors that look nothing like their cause.

Both kills now run under the host benchmark lock the harnesses themselves take,
so anything alive while it is held really is a leftover. The second one needs it
as much as the first: by the time cleanup runs the harness has released the
lock, so another repository's benchmark can already be up.
 2026-09-17 10:59:39 -05:00
Josh Siviour 451df1a8cd docs: publish AWS operation coverage inventory (#1075) 2026-09-17 21:33:53 +10:00
Josh Siviour b26478bd06 chore: pin predastore to the bucket sub-resource dispatch fix (#1074) 
Every deployed binary links whatever go.mod pins, so the routing fix for bucket sub-resources is absent from the gateway until this moves. Without it a provider-level default_tags block still leaves no bucket creatable by Terraform.
 2026-09-17 20:55:08 +10:00
Josh Siviour c9460628fb fix(ec2): clear a resource's central tags when it is deleted (#1073) 
DescribeTags kept answering for resources that no longer exist: the central tag store's DeleteAllTags had one production caller, the instance terminate path, so every other resource type left its entry behind forever.

CentralTagWriter becomes CentralTagStore and gains DeleteAllTags, which TagsServiceImpl already implemented, so the daemon wiring is a rename rather than new plumbing. clearRecordTags mirrors projectRecordTags: a nil store is a no-op and a failure is logged rather than returned.

The clear runs after the record delete has succeeded, never before, so a delete that fails leaves a live resource with its tags intact. deleteNetworkInterface covers both the public delete and the forced instance teardown, and deleteSecurityGroupInternal covers the default group that only the DeleteVpc cascade ever removes.
 2026-09-17 15:18:31 +10:00
Josh Siviour e88691e64d feat(imds): serve the cluster services domain, not amazonaws.com (#1072) 
An AWS SDK's default endpoint resolution builds
{service}.{region}.{services/domain}. IMDS previously answered that
leaf with the literal amazonaws.com, so a workload that forgot its
endpoint override silently reached real AWS instead of failing.

Now that service-endpoint DNS resolves cluster-side, serve the
cluster's configured AWS.ServicesDomain instead, threaded from
config through vpcd.Config into NewIMDSServiceImpl. An empty value
fails closed with a 404 rather than ever falling back to
amazonaws.com. services/partition is unchanged: it still returns
aws, since AWS SDKs carry hardcoded partition tables.
 2026-09-17 14:33:11 +10:00
Josh Siviour c674e4b5ce fix(ec2): project creation-time tags into the central tag store (#1063) 
A tag set at creation time through TagSpecifications was written only onto the resource's own KV record. DescribeTags reads a separate central index at tags/{accountID}/{resourceID}.json, which only CreateTags writes, so a resource born with tags described as having none until something tagged it again.

Terraform finds this immediately: terraform-aws-modules/eks reads a cluster security group's tags back after creating it, and aws_ec2_tag fails against a resource whose tags the tag API cannot see.

The four VPC create paths now project their stored tags into the central store through an optional CentralTagWriter, injected after construction like the external IPAM. A nil writer leaves create behaviour unchanged, and a write failure is logged rather than failing the create — the resource exists either way, and failing a create over a tag index would be worse than the divergence it repairs.

Only vpc, subnet, security group and network interface are covered here. Image, volume, snapshot, route table, internet gateway, egress-only internet gateway, elastic IP, NAT gateway, key pair and launch template have the same split and are follow-up work.
 2026-09-17 14:28:29 +10:00
Josh Siviour bef0780b10 fix(ecs): make awsvpc ENI slots a placement resource (#1064) 
* fix(ecs): make awsvpc ENI slots a placement resource

A container instance carries a finite number of awsvpc task ENIs — the hot-plug PCIe root ports the daemon pre-allocates from the instance type's cap — and nothing outside the daemon knew the limit existed. The scheduler's capacity model was CPU, memory and GPU, so every over-capacity task was placed, created an ENI, failed to hot-plug it and rolled it back. A service asking for more tasks than the cluster has slots churned ENI create/delete indefinitely instead of reporting a placement failure once. On a t3.micro the third task and everything after it failed with "attach task ENI eni-… -> i-…: ServerInternal".

The instance record now carries the instance type, reported the way AWS carries it, as the ecs.instance-type container-instance attribute; the ENI cap is derived from it rather than stored, so a corrected limit table applies on the next read. An instance whose type has not been reported is not gated at all: a recognised type always has at least one slot, so zero can only mean unknown, and refusing every awsvpc placement against an older agent would turn a mixed-version cluster into an outage.

Both placement paths gate on it. A task declined for want of an interface is reported as RESOURCE:ENI, spelled as AWS spells it, and separately from a plain capacity refusal. The caller-visible detail is a fixed sentence; the wrapped error keeps its identifiers and server error codes in the log, where someone can act on them.

DescribeInstanceTypes now reports NetworkInfo.MaximumNetworkInterfaces, from the same table the slot allocator enforces, so a caller sizing an autoscaling group has something to size against. Fields this stack does not implement are left unset rather than guessed.

Finally, the daemon records the slot it actually allocated as the ENI's device index. The requested index was stored verbatim, so every task ENI on an instance reported index 1 while landing on a different PCIe slot; on AWS a device index is unique per instance.

* fix(ecs): send an ENI capacity refusal as the reason alone

AWS returns RESOURCE:* placement failures with no detail. Carrying a sentence
there gives a caller a second string to parse that says nothing the reason does
not already say.
 2026-09-17 13:54:11 +10:00
Josh Siviour 86014ac08a fix(ec2): serialise tag writes across the cluster (#1070) 
CreateTags is a read-modify-write of one resource's whole tag map. It was
serialised by a sync.Mutex on the service struct, which exists once per
process, while the handler answers on the spinifex-workers queue group and so
runs on whichever node NATS picks. Two nodes read the same map, each added its
own key, and the later write won. Every call returned success.

Six parallel writers against one security group on a three-node cluster left
four tags. On a single node the mutex hides it entirely, which is why no test
caught it.

The tag index moves from one S3 object per resource to a JetStream KV bucket,
where the merge runs inside kvutil.Update and commits at the revision it read.
A writer that loses the race re-reads the winner's keys and adds its own on
top. S3 could not host the fix: predastore's PutObject has no If-Match, so
there is nothing to compare and swap against.

Tags written before the move are not migrated in a batch. A batch that
half-failed would leave a resource looking untagged, and the next write would
commit that as the truth. Instead the CAS seeds a missing entry from the old
object inside the attempt, so a resource migrates on its first write and a
failure to read the old object fails that one call. DescribeTags reads the
union of both key sets, preferring the bucket, and DeleteAllTags clears both so
old tags cannot reappear through the fallback.

The process-local mutex is removed rather than left beside the CAS.

The regression test drives six separate service values over one bucket,
because that is what makes it able to fail: with the old code every caller
inside one process was already serialised, and only a second node could lose an
update.
 2026-09-17 13:53:47 +10:00
Josh Siviour fd22c518bf fix: bound the KV bucket listing so a stuck DNS reconcile cannot wedge the cluster (#1071) 
kvutil.BucketNames ranged the KeyValueStoreNames lister channel using whatever context the caller supplied, and the EKS desired-set builder supplies one that lives as long as the daemon. A lost stream-names reply therefore left the channel open with nothing ever sent on it and parked the caller for the life of the process. kvutil.Keys, immediately below it, already bounds exactly this case and says why; BucketNames never got the same treatment.

The DNS reconcile made that hang cluster-wide. A pass takes the reconcile lease before computing its batch and releases it on a deferred call, and the lease renews on a background timer for as long as the holder lives, so a pass that never returns holds it forever and the other nodes never take over. Observed on a three-node cluster: the zone object stopped being written after a rolling deploy, and every name published afterwards stayed NXDOMAIN until the holder was restarted.

The listing now carries its own deadline and reports a timeout as an error, never as a short list, because callers prune on absence. The pass bounds the batch computation as well, so any other deadline-free call reached from the desired-set builders costs a cycle rather than the cluster. Only the read side is bounded: it has no side effects, so abandoning it is safe, while the apply stays inline under the lease.
 2026-09-17 13:53:23 +10:00
Josh Siviour 4a9dda0b99 fix(ecs): honour ListTasks filters and count instance tasks by status (#1062) 
ListTasks read only the cluster and returned every task record under it, discarding DesiredStatus, Family, ServiceName, StartedBy and ContainerInstance. On a cluster with one running task and a hundred stopped ones, both status filters returned the same hundred-and-one ARNs.

A taskFilter now carries the input's matchable fields and is applied per record. An omitted desiredStatus means RUNNING, as it does on AWS, so an unfiltered call no longer hands back everything the cluster has ever run.

recordTaskState moves a self-exited task's desired status to STOPPED alongside its last status. Without that the filter would be honoured and still wrong: a task that died on its own would answer the RUNNING filter for as long as its record survived.

Container-instance counts came from len(PlacedTasks), which is the set of tasks still owing capacity rather than a status tally, so a PENDING task counted as running and a task whose release did not complete counted as running forever. PendingTasksCount was never set at all. Both are now tallied from the task records, the way the cluster-level counts already were. DeregisterContainerInstance passes zero deliberately — the instance is gone, and counting there would report the tasks it just force-stopped.

launchType is not filtered because TaskRecord has no launch-type field to match on, and maxResults/nextToken are left alone because no List call in this package paginates; adding a token scheme to one would leave the other four inconsistent.
 2026-09-17 13:48:59 +10:00
Josh Siviour c10b6732d5 feat: publish AWS service endpoint names in DNS and match gateway cert SANs (#1068) 
* feat: publish AWS service endpoint names in DNS and match gateway cert SANs

ec2.<region>.spinifex.internal and its sibling AWS service names (sts,
elasticloadbalancing, ecs, eks, acm) NXDOMAINed because the internal
suffix was only ever a cert SAN and Host-header match, never a
northstar zone. BootstrapBaseZone now seeds it as a third zone, guarded
three-way so a suffix colliding with the default or internal domain
seeds no duplicate.

The reconciler publishes each name as a set-valued A record spanning
every cluster node's own gateway address, since a service endpoint's
natural target is every node rather than one resource address that
answers the same from anywhere. ConfigArr.SetRecordSet backs this in
northstar; UpsertRecord's single-record semantics are untouched for
EC2/ELB/EKS/RDS. A non-IP gateway host is skipped rather than written
as a malformed record, and prune authority for the new class is scoped
to its own zone so it can never act on the base or private zone.

admin.AWSGWServiceDNSNames and handlers/dns.ServiceEndpointNames both
consume config.AWSGWServiceNames, so the cert SANs and the DNS records
they must match can never drift apart.

Depends on github.com/mulgadc/northstar#31 (ConfigArr.SetRecordSet);
go.mod pins that branch's commit until it merges and tags.

* refactor(config): rename AWS.InternalSuffix to AWS.ServicesDomain

ServicesDomain names the value IMDS serves at
/latest/meta-data/services/domain, the slot an AWS SDK substitutes
into {service}.{region}.{domain} for default endpoint resolution.
The old name read as a sibling of Northstar's compute.internal while
meaning something entirely different, which invited confusion.

The default changes from spinifex.internal to services.internal, role-
named like compute.internal (per-instance hostnames) now that
services.internal holds service endpoints instead.

Deployed clusters have internal_suffix baked into spinifex.toml and
their gateway cert SANs built from it, so LoadConfig now reads
services_domain, falls back to the legacy internal_suffix key when
that is unset, and only then applies the new default. Every consumer
of the field and the renamed DefaultAWSServicesDomain constant is
updated accordingly, including the spx admin init template and the
e2e test harness's own config loader.

* chore: pin northstar to the merged set-valued write path
 2026-09-17 13:46:33 +10:00
Tom Newton c4b380b2ed chore: update cisco docs 2026-09-17 13:28:30 +10:00
Josh Siviour ef9bfb4c1d fix(gateway): resolve DescribeRegions endpoint from the gateway's advertised host (#1066) 
* fix(gateway): resolve DescribeRegions endpoint from the gateway's advertised host

DescribeRegions hardcoded https://localhost:9999 as the Region's endpoint
regardless of where the gateway actually runs, so a workload doing endpoint
discovery through this API was pointed at its own loopback instead of a
reachable address. The endpoint is now resolved by the caller from
GatewayConfig.RegistryHost/RegistryPort, the same advertised host:port ECR
already uses (concrete AWSGW bind host, else AdvertiseIP), and passed into
DescribeRegions rather than invented there. Falls back to localhost:9999
only when no concrete host is configured, which is also the only case where
that value is actually correct.

Closes mulga-0bqih.

* style: trim DescribeRegions endpoint comments
 2026-09-17 13:28:08 +10:00
Josh Siviour 969163c114 feat(imds): serve block-device-mapping from instance block device data (#1067) 
* feat(imds): serve block-device-mapping from instance block device data

The aws-ebs-csi-driver's IMDS client fails outright on this 404,
aborting IMDSInstanceInfo before it reaches the Kubernetes fallback the
driver otherwise uses. Serve ami, root, and one ebsN per attached EBS
volume from the instance's existing BlockDeviceMappings, root device
first, matching how the driver counts "ebs" occurrences in the listing.

* style: trim block-device-mapping comments
 2026-09-17 13:27:46 +10:00
Julian Sommer ff41bcf369 fix: read only the named documents on explicit-ID describes (#1069) 2026-09-17 12:29:35 +10:00
Julian Sommer 5991b92b06 chore: concurrent preflight 2026-09-17 11:31:42 +10:00
Julian Sommer 5ff54ea110 refactor: route the last hand-rolled KV codecs and CAS loops through kvstore (#1061) 2026-09-17 10:45:11 +10:00
Julian Sommer 3a51646e1c chore: disable wastedassign linter 2026-09-17 10:42:18 +10:00
Josh Siviour 809ec5f1fe fix(vpc): accept security-group rules carrying an IPv6 range (#1060) 
AuthorizeSecurityGroupIngress and AuthorizeSecurityGroupEgress rejected any
IpPermission carrying an Ipv6Ranges entry. The security-group module defaults
egress_ipv6_cidr_blocks to ::/0 and the provider revokes the group's own
default egress before authorizing the replacement, so the rejection left the
group with no egress rules at all, which is treated as deny.

SGRule gains CidrIpv6. The rule is stored, deduped, revoked and projected like
any other, and the two policy.Rule adapters drop it before the ACL builder,
which is IPv4-only. No interface here has an IPv6 address, so the rule matches
nothing - which is what an IPv6 rule does on AWS in a VPC with no IPv6 CIDR.

Revoking an IPv6 rule that does not exist now returns InvalidPermission.NotFound
rather than succeeding as a no-op. The no-op existed only because authorize
refused to store the rule, so a caller's state could never match.

Closes mulga-6bu36.
 2026-09-17 10:33:43 +10:00
Josh Siviour ead2f2fe6c fix(ecs): keep a stopped task's ENI attachment after its release (#1059) 
DescribeTasks returned an empty attachments array for every STOPPED awsvpc task. On env19, 134 of 135 tasks in a cluster reported attachments: [], and only the one PENDING task carried an ElasticNetworkInterface block. On AWS a stopped task keeps its attachment for the retention window, with status DELETED and the networkInterfaceId still readable, because it is part of the forensic record of a task that has stopped.

taskToAWS gates the attachment on ENIID being set, and the ENI release retry had made a successful release clear that field so a non-empty ENIID would mean "still owed" and the sweep could retry from it. The signal was right; carrying it in the emptiness of ENIID was not, because the same field is the projection's only source.

The state moves to an ENIReleased flag of its own. ENIID stays put, so the attachment survives as long as the record does, and the sweep's owed test, its refusal to prune, and the placement rollback all read the flag instead. A released record ages out on retention exactly as it did before the retry landed. The attachment reports DELETED once released rather than echoing the task's own status.

This also restores the evidence path the ENI-leak probe depends on: it follows a stopped task to its ENI, and with no attachment to follow it had been reporting skipped on every run.

Closes mulga-5bnx3.
 2026-09-17 07:56:45 +10:00
Josh Siviour 0fd6848fbd fix(ecs): retry an owed awsvpc task ENI release (#1058) 
Releasing an awsvpc task ENI was best effort: reclaimTaskENI logged a failure and returned, and nothing ever tried again. Its own comment said the ENI fields stayed on the record for a later retry, but no later retry existed, and the stopped-task sweep pruned the record an hour later without looking at those fields — deleting the only pointer to the leaked interface. An orphaned task ENI still references its security group, so DeleteSecurityGroup fails with DependencyViolation and no awsvpc stack can be destroyed without deleting the ENIs by hand.

A non-empty ENIID on a stopped record now means exactly one thing: this ENI is still owed. reclaimTaskENI clears the identity when Release succeeds and leaves it when it does not, forceStopTask reclaims before its single write so the cleared fields land in it, and the stopped-task sweep gains the job of paying the debt off — retrying with a backoff that doubles from 30s to a 15m ceiling, capped at twenty attempts per pass because Release is a 30s NATS round trip and a bucket full of them would stall the sweep for the whole fleet.

A record that still owes an ENI is never pruned, however far past retention it sits. It is the only pointer to the interface, so dropping it converts a recoverable leak into a permanent one.

The placement rollback had the same hole from the other end: an attach failure cleared the ENI fields unconditionally and RunTask never persisted the record, so a failed release there leaked with no trace at all. It now clears only when the release succeeded, and otherwise persists the record as STOPPED so the sweep owns it like any other.

Closes mulga-1ysjx.
 2026-09-16 23:46:21 +10:00
Josh Siviour 208a108c27 fix(ec2): render image CreationDate as UTC (#1057) 
An AMI imported on a node whose timezone is not UTC was stamped with local wall-clock time carrying a Z suffix, so on a +1000 node an 11:42Z import reported 21:42Z — ten hours in the future. The layout "2006-01-02T15:04:05.000Z" treats the trailing Z as a literal character rather than a zone directive, so Format printed whatever zone the value carried and then asserted UTC regardless.

DescribeImages now converts before formatting, which also corrects records already stored, since the stored time.Time still carries its original zone. The four sites that stamp a new image now store time.Now().UTC() as well, so the value is right on both sides of the round trip.

Terraform's most_recent sorts on this field, so before this an AMI imported on a +1000 node outranked a genuinely newer one imported on a UTC node for ten hours.

Closes mulga-il7w1.
 2026-09-16 23:45:40 +10:00
Josh Siviour e2ca21a18d feat: report availabilityZoneRebalancing on ECS services (#1055) 
DescribeServices returned no availabilityZoneRebalancing field, so the terraform provider read the computed attribute as empty and planned an in-place update to DISABLED on every plan, including one taken immediately after a clean apply.

The pinned aws-sdk-go v1.55.8 is the last v1 release and its ECS shapes predate the field, so it can be neither parsed from an input nor set on an output through the SDK types. Embedding ecs.Service in a wrapper does not work either: jsonutil nests an embedded struct rather than flattening it, so the service arrives under its own key.

The field is read off the raw request body and written onto the encoded response instead. checkAZRebalancing refuses anything but DISABLED with InvalidParameterException, because nothing here rebalances tasks across zones and a silently dropped ENABLED would report a service as rebalancing when nothing does. addSDKGapFields runs from WriteJSONResponse so every service-carrying action is covered from one place, decodes with UseNumber to keep the epoch-second times jsonutil emitted, and leaves a value already present alone.

Closes mulga-t239t.
 2026-09-16 23:44:52 +10:00
Josh Siviour ca5ff7695e fix(ecs-agent): allocate a console for a pseudoTerminal container (#1054) 
oci.WithTTY put Terminal in the OCI spec while the task was created with a cio
creator that asked for no terminal, so the shim made no console socket and runc
refused to start the container at all. The terminal flag now reaches the FIFO
set, and a terminal carries no separate stderr stream.
 2026-09-16 23:44:31 +10:00
Josh Siviour 1ab61b0dd5 fix(ecs): enforce or refuse container runtime fields (TFC-0040) (#1053) 
* fix(ecs): enforce or refuse container runtime fields (TFC-0040)

user, mountPoints, volumesFrom, readonlyRootFilesystem, privileged,
pseudoTerminal, interactive, linuxParameters, systemControls,
startTimeout and stopTimeout registered successfully but were never
stored, sent to the agent, enforced, or echoed back on describe.

Store and echo the fields the agent can enforce (user, readonly root
filesystem, privileged, pseudo terminal, interactive, systemControls,
capability add/drop, start/stop timeout), apply them through containerd
via OCI spec options and a graceful Stop, and refuse at
RegisterTaskDefinition the fields the agent cannot honor (mountPoints,
volumesFrom, linuxParameters.devices/initProcessEnabled/
sharedMemorySize/tmpfs). Refusal only triggers on a value that asks
for the restriction; empty lists and false/unset flags are accepted
and echoed as before.

* test: move the bus wire-contract test to an external package

* fix(ecs-agent): release an interactive container's stdin so its copier exits

* fix(ecs): return a caller-supplied empty collection instead of omitting it
 2026-09-16 21:23:06 +10:00
Josh Siviour 6c86dd5f34 fix(ec2): validate and honour a requested private IP address (#1050) 
CreateNetworkInterface accepted a caller-supplied PrivateIpAddress
verbatim, with no subnet-range check and no IPAM reservation, so a
direct call could collide with a concurrent AllocateIP handing out the
same address. Add IPAM.ClaimIP, a CAS-guarded sibling of AllocateIP
that validates the address against the subnet and rejects it if
already allocated, and wire it into CreateNetworkInterface with the
matching AWS error codes.

RunInstances --private-ip-address silently returned a different,
pool-assigned address instead of honouring the request. Forward
PrivateIpAddress (including the NetworkInterfaces[].PrivateIpAddresses
primary entry) into the auto-created ENI's CreateNetworkInterfaceInput,
and reject a pinned address on a launch with MaxCount > 1 since AWS
only allows it for a single instance.
 2026-09-16 21:22:26 +10:00
Josh Siviour d632b607ff fix: decompress single-file zst/bz2/xz/gz disk images (#1048) 
* fix: decompress single-file zst/bz2/xz/gz disk images into tmpdir

ExtractDiskImageFromFile only handled tar archives and a broken plain
.xz branch that ran xz -dk beside the source and then tried to parse
tar chatter for a path, so it silently returned an empty string. Talos
(.raw.zst, .raw.xz), Fedora CoreOS (.qcow2.xz, .raw.xz) and Flatcar
(.img.bz2) publish only single-file compressed artifacts, so none of
them could be imported.

Add streaming decompression for .zst, .bz2, .xz and .gz that writes
into the caller's tmpdir and returns that path directly, instead of
inferring it from tar output. A decompressed .qcow2 payload now flows
through the existing qcow2-to-raw qemu-img conversion via a shared
finalizeDiskImage helper. zstd uses klauspost/compress, already a
transitive dependency via containerd, for a pure-Go streaming decoder.
bz2 and gzip use the standard library. xz shells out to the xz binary
since no pure-Go xz decoder is already a dependency, and the existing
.tar.xz branch already requires that binary on the host via tar's -J
flag, so this adds no new host requirement.

* fix: write sparse holes when decompressing single-file disk images

Verified live on env13: a Talos raw.zst imported successfully but the
decompressed output was written literally rather than sparsely, an
11 GiB file on disk for content that decompresses to 183 MiB with
xz -dc | dd conv=sparse. This is a regression relative to the
qcow2-to-raw path beside it, which already writes sparse via
qemu-img convert.

decompressToTemp now reads in 1 MiB blocks, seeks past all-zero
blocks instead of writing them, and truncates to the final size so a
trailing zero run is still recorded as a hole. 1 MiB amortises the
read/seek syscall overhead over a multi-GiB image while still being
small enough to catch the large contiguous zero runs that dominate a
sparse raw disk; a zero run shorter than one block is written
literally, which is the safe direction for a false negative to fail
in.

zstd, bzip2 and gzip all route through decompressToTemp, so all three
gain sparse output from this change. The plain-.xz path shells out to
the xz binary with output piped straight to a file and does not go
through decompressToTemp, so it is unaffected by this fix and remains
a known follow-up.

* fix: make plain-.xz decompression sparse too

decompressXz shelled its output straight to a file, bypassing the
sparse-hole handling just added to decompressToTemp for zstd, bzip2
and gzip. .raw.xz is the format Talos and Flatcar actually publish
alongside .zst, so this path is the one most likely hit in practice,
and left dense the exact disk-size regression this fix targets still
fires for it.

exec.Cmd already runs its own copier goroutine when Stdout is not an
*os.File, and Wait joins it. decompressXz now hands the xz process an
io.Pipe write end and reuses decompressToTemp on the read end, so
plain .xz gets identical sparse-hole handling to the other codecs. A
separate goroutine closes the pipe with the process's exit error once
Wait returns, so a read past the end of a failed decompression
surfaces that error, stderr included, instead of a bare EOF; the
error path also closes the read end explicitly so a stuck copier
cannot wedge Wait forever if decompressToTemp gives up early.
 2026-09-16 16:54:51 +10:00
Josh Siviour 68eeff069e feat: project EKS access policies into Kubernetes RBAC (#1049) 
AssociateAccessPolicy recorded a policy association but never granted
it in the cluster. Bake static ClusterRoleBindings for the mulga:
groups into the eks-node image, project a principal's cluster-scoped
associated policies onto those groups in the TokenReview webhook, and
reject namespace scope at association time since there is no
namespace-scoped RBAC writer.
 2026-09-16 16:45:24 +10:00
Julian Sommer 0c83f7f954 refactor: use strings.CutPrefix/CutSuffix for guard-then-trim pairs 2026-09-16 16:18:33 +10:00
Josh Siviour 4b35de1409 feat(iam): evaluate trust-policy conditions, accept aws:SourceAccount (#1051) 
* feat(iam): evaluate trust-policy conditions and accept aws:SourceAccount

evalTrustPolicy read Effect, Action and Principal but never Condition, so a
stored aws:SourceAccount clause was a control nothing enforced. Give it a
condition arm evaluated in both the Deny and Allow passes, following the
StringEquals-only, fail-closed-on-unknown-key shape
evalTrustPolicyForWebIdentity already uses. The value is the caller ARN's own
account, derived server-side rather than caller-asserted.

Relax validateWebIdentityCondition from a single-action gate to a per-action
allowlist: sts:AssumeRoleWithWebIdentity keeps its existing IRSA keys, and
sts:AssumeRole now accepts aws:SourceAccount under StringEquals. Everything
else still fails at write time, naming the operator, key and action rather
than a bare MalformedPolicyDocument, and CreateRole/UpdateAssumeRolePolicy now
carry that reason to the caller instead of swallowing it into slog.Debug.

Extend the identity-policy refusal at validateStatementRestrictions with the
reason for aws:SourceArn and aws:RequestTag/*, the two keys a companion change
records as divergences in rfc0-7_aws-divergence.md.

* test(iam): fix trust-policy reason tests after aws:SourceAccount now valid

PR #1046 merged into dev with fixtures asserting aws:SourceAccount plus
StringEquals on sts:AssumeRole is refused. Part B of mulga-yxd9p makes
that combination valid, so the fixtures now exercise an unsupported
action and an unsupported key instead.

* docs: drop bead and plan-decision references from test comments
 2026-09-16 15:34:23 +10:00
Julian Sommer 7474fa2073 chore: use native loadx509KeyPair 2026-09-16 15:33:57 +10:00
Julian Sommer 95035a1754 chore: use go-toml instead of handparsed 2026-09-16 15:31:58 +10:00
Julian Sommer 9bc3ea8658 refactor: use crypto/ssh for key gen 2026-09-16 15:24:45 +10:00
Julian Sommer 2dfc9e1efb refactor: route the REST-JSON gateways through chi instead of regex tables 2026-09-16 15:15:16 +10:00
Josh Siviour 578185f533 feat: supply the aws:PrincipalType condition key at the AWS gateway door (#1052) 
Bump bluebottle for the new key. requestConditionKeys now maps this
door's internal principal-type spelling to the canonical AWS value
through the bluebottle constants, and omits the key for an
unrecognized type rather than supplying an empty string.
 2026-09-16 14:36:46 +10:00
Josh Siviour 4eee02fcfd fix: carry the trust-policy validation reason to the caller (#1046) 
ValidateTrustPolicyDocument builds precise errors naming the statement index
and the offending element, and both callers discarded them into slog.Debug
and returned a bare MalformedPolicyDocument. Debug is below the default
level, so the reason reached nobody: not the caller, and not the operator
reading the gateway log.

The contrast with the identity-policy path was stark. CreatePolicy names the
operator and the key exactly; CreateRole answered only 'The policy document
is malformed.', so an operator could not tell whether Principal, Action, a
size limit or a Condition block was at fault. The trust gate is the one the
terraform ECS module hits, which is what made this worth fixing on its own.

Both sites now use awserrors.Errorf, the same mechanism the identity path
uses. The separate Debug line is dropped rather than duplicated: the gateway
already logs a returned error at Error level, so one message now serves both
the client and the operator.
 2026-09-16 14:12:09 +10:00
Josh Siviour 062880687d feat: return ECS service events, and project the service and deployment timestamps (#1047) 
* feat(ecs): populate service event ring on DescribeServices

Events was always projected as null since ServiceRecord carried no event
history. Add a capped, newest-first event ring populated at the rollout
transitions the reconciler already computes: steady state reached, a
circuit-breaker trip, and a placement shortfall it cannot resolve this pass.
Emission is edge-triggered on the state change already guarding each site, so
the repeated reconcile loop does not refill the ring with one repeated
message every pass.

* feat(ecs): project createdAt on services and deployments

CreatedAt was already computed and persisted on ServiceRecord and Deployment
at creation time, but serviceToAWS never echoed it, so DescribeServices could
never answer which build wrote a given record. Project ServiceRecord.CreatedAt
and Deployment.CreatedAt/UpdatedAt, omitting the key entirely for a legacy
record whose timestamp predates the field rather than emitting the 1970 epoch.
 2026-09-16 14:01:53 +10:00
Josh Siviour bb8c639c82 fix(eks): accept API_AND_CONFIG_MAP and always return nodegroup health (#1045) 
CreateCluster refused accessConfig.authenticationMode = API_AND_CONFIG_MAP,
which is both AWS's default since 1.23 and the terraform-aws-eks module
default, and named no parameter in the refusal. The mode is now stored on
the cluster record and echoed by DescribeCluster, so a caller does not see
a permanent access_config diff on every subsequent plan; bare CONFIG_MAP
still fails, since Spinifex has no aws-auth ConfigMap path. The refusal
now names accessConfig.authenticationMode via awserrors.Errorf.

nodegroupRecordToAWS also only set Health when a node group had a status
reason, so a healthy node group described back with health: null instead
of the empty-issues object AWS always returns. Health is now always
constructed with a non-nil issues slice.
 2026-09-16 14:01:29 +10:00
Josh Siviour de295b3f15 fix: return InvalidAction and the correct envelope for an unserved service (#1044) 
The unserved-service gate ran before signature verification but answered
SignatureDoesNotMatch, falsely blaming credentials that were never checked.
It now returns InvalidAction, the same code an unimplemented action on a
served service already gets, so the behaviour is consistent rather than
novel.

The envelope was also always XML, because jsonErrorService only lists
scopes the gateway serves and an unserved scope is by definition absent
from it. A JSON-protocol caller such as sqs or dynamodb therefore could
not deserialize the body and never saw the true error. writeSigV4Error now
falls back to the request's own X-Amz-Target and Content-Type headers to
pick JSON when the scope is not in that list, leaving every served scope's
envelope selection unchanged.
 2026-09-16 14:01:13 +10:00
1127 changed files with 55276 additions and 9906 deletions
Expand all files Collapse all files
.github/assets/diagrams/oci-single-node.svg
+80
View File
@@ -0,0 +1,80 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1020" height="836" viewBox="0 0 1020 836" role="img" aria-label="Spinifex on a single OCI instance: the instance sits inside the VCN public subnet 10.200.0.0/23; its primary VNIC carries the default route and the cluster mesh, its secondary VNIC is the br-wan bridge holding registered secondary private IPs paired with reserved public IPs; guests reach them through the spx-nat transit veth, and an iSCSI block volume backs /var/lib/spinifex.">
<defs><linearGradient id="nats" x1="0" y1="0" x2="1" y2="0"><stop offset="0" stop-color="#F5652E"/><stop offset="1" stop-color="#C63C0E"/></linearGradient></defs>
<rect x="0" y="0" width="1020" height="836" rx="20" fill="#14181D" stroke="#2A2F36" stroke-width="1"/>
<text x="28" y="42" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#EE4B12" font-weight="600" text-anchor="start" letter-spacing="2.5">SPINIFEX ON OCI — SINGLE NODE</text>
<text x="992" y="42" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#9AA1A8" font-weight="400" text-anchor="end">one instance · routed NAT · real public addresses</text>
<text x="28" y="72" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="23" fill="#F5F3EE" font-weight="800" text-anchor="start">One OCI instance, single Spinifex node</text>
<rect x="28" y="92" width="964" height="76" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="48" y="122" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">INTERNET</text>
<text x="972" y="122" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">AWS SDKs, SSH and guest egress all arrive here</text>
<rect x="46" y="126" width="272.7" height="22" rx="7" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="182.3" y="140.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">aws-cli / Terraform / boto3 → :9999</text>
<rect x="327.7" y="126" width="177.2" height="22" rx="7" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="416.3" y="140.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">ssh → guest public IP</text>
<rect x="513.92" y="126" width="218.1" height="22" rx="7" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="623.0" y="140.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">guest egress ← same address</text>
<line x1="510.0" y1="170" x2="510.0" y2="186" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
<rect x="28" y="188" width="964" height="582" rx="16" fill="#EE4B12" fill-opacity="0.022" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.45" stroke-dasharray="6 5"/>
<text x="50" y="218" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#EE4B12" font-weight="600" text-anchor="start" letter-spacing="2">OCI VCN 10.200.0.0/16 · PUBLIC SUBNET 10.200.0.0/23</text>
<text x="970" y="218" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="end">virtual router 10.200.0.1 · internet gateway</text>
<rect x="48" y="232" width="924" height="520" rx="14" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
<text x="68" y="266" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="16" fill="#F5F3EE" font-weight="800" text-anchor="start">BM.Standard.E2.64</text>
<text x="260" y="266" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="400" text-anchor="start">spinifex-node-01 · FAULT-DOMAIN-1</text>
<text x="952" y="266" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">one host = one Spinifex region</text>
<line x1="68" y1="282" x2="952" y2="282" stroke="#EE4B12" stroke-opacity="0.25" stroke-width="1"/>
<rect x="68" y="298" width="434.0" height="108" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="88" y="328" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">VNIC 0 — PRIMARY</text>
<text x="482.0" y="328" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">enp0s9</text>
<text x="88" y="356" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">10.200.0.252/23 · holds the default route</text>
<text x="88" y="378" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">advertise IP · ovn-encap-ip · NATS · API :9999</text>
<rect x="518.0" y="298" width="434.0" height="108" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="538.0" y="328" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">VNIC 1 — SECONDARY</text>
<text x="932.0" y="328" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">br-wan (Linux bridge)</text>
<text x="538.0" y="356" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#F3F1EC" font-weight="400" text-anchor="start">10.200.1.233/32 + ip rule → table 200</text>
<text x="538.0" y="378" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">secondary private IPs ⇄ reserved public IPs</text>
<line x1="735.0" y1="406" x2="735.0" y2="422" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
<rect x="68" y="424" width="884" height="40" rx="11" fill="#1E242B" fill-opacity="1.0" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.45"/>
<text x="88" y="449" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#EE4B12" font-weight="600" text-anchor="start">spx-nat-host ⇄ spx-nat-ovs</text>
<text x="296" y="449" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#9AA1A8" font-weight="400" text-anchor="start">transit 100.127.0.0/24 · nft chain spinifex-nat-egress masquerades out br-wan</text>
<line x1="510.0" y1="464" x2="510.0" y2="480" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
<rect x="68" y="482" width="884" height="100" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="88" y="512" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">SPINIFEX CONTROL PLANE</text>
<text x="932" y="512" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">every service is one binary: spx service &lt;name&gt;</text>
<rect x="86" y="530" width="109.0" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="140.5" y="544.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">awsgw :9999</text>
<rect x="204.01999999999998" y="530" width="74.9" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="241.5" y="544.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">daemon</text>
<rect x="287.94" y="530" width="61.3" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="318.6" y="544.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">vpcd</text>
<rect x="358.22" y="530" width="61.3" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="388.9" y="544.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">NATS</text>
<rect x="428.5" y="530" width="102.2" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="479.6" y="544.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">predastore</text>
<rect x="539.7" y="530" width="102.2" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="590.8" y="544.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">viperblock</text>
<rect x="650.9000000000001" y="530" width="95.4" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="698.6" y="544.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">northstar</text>
<rect x="755.2800000000001" y="530" width="88.6" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="799.6" y="544.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">ui :3000</text>
<text x="86" y="568" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">SigV4 in, NATS in the middle, QEMU/KVM · OVN/OVS · Predastore · Viperblock underneath</text>
<line x1="248" y1="582" x2="248" y2="598" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
<line x1="772" y1="582" x2="772" y2="598" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.4" stroke-dasharray="2 4"/>
<rect x="68" y="600" width="532" height="130" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="88" y="630" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">TENANT VPCs — OVN br-int</text>
<text x="580" y="630" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">geneve-local</text>
<rect x="86" y="644" width="496" height="34" rx="9" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="100" y="666" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">vpc-a 10.0.0.0/16</text>
<text x="310" y="666" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#9AA1A8" font-weight="400" text-anchor="start">i-… 10.0.1.4 → 150.230.13.131</text>
<rect x="86" y="686" width="496" height="34" rx="9" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="100" y="708" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">vpc-b 172.31.0.0/16</text>
<text x="310" y="708" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#9AA1A8" font-weight="400" text-anchor="start">i-… 172.31.4.9 · private, egress only</text>
<rect x="616" y="600" width="336" height="130" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="636" y="630" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">BLOCK VOLUME — iSCSI</text>
<text x="932" y="630" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">256 GB+</text>
<text x="636" y="658" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">169.254.2.2:3260 → /dev/sdb</text>
<text x="636" y="680" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">/var/lib/spinifex</text>
<text x="636" y="700" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="11.5" fill="#5B636C" font-weight="400" text-anchor="start">mounted by UUID, defaults,_netdev,nofail</text>
<text x="636" y="718" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="11.5" fill="#5B636C" font-weight="400" text-anchor="start">viperblock · predastore · JetStream</text>
<text x="28" y="796" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">A guest's public address is a real OCI reserved public IP, attached to a secondary private IP on br-wan.</text>
<text x="28" y="814" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">Inbound and outbound are therefore the same address, and OCI's per-VNIC source check passes.</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 15 KiB

.github/assets/diagrams/oci-three-node.svg
+116
View File
@@ -0,0 +1,116 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1020" height="796" viewBox="0 0 1020 796" role="img" aria-label="Spinifex across three OCI instances: all three sit inside the VCN public subnet 10.200.0.0/23, one per fault domain. Each carries a primary VNIC for the cluster mesh and a br-wan secondary VNIC holding its reserved public addresses, its own iSCSI block volume, and its share of the guests. Geneve, NATS JetStream, predastore erasure coding and the OVN raft join them east-west.">
<defs><linearGradient id="nats" x1="0" y1="0" x2="1" y2="0"><stop offset="0" stop-color="#F5652E"/><stop offset="1" stop-color="#C63C0E"/></linearGradient></defs>
<rect x="0" y="0" width="1020" height="796" rx="20" fill="#14181D" stroke="#2A2F36" stroke-width="1"/>
<text x="28" y="42" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#EE4B12" font-weight="600" text-anchor="start" letter-spacing="2.5">SPINIFEX ON OCI — THREE NODES</text>
<text x="992" y="42" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#9AA1A8" font-weight="400" text-anchor="end">three fault domains · one region · one API endpoint</text>
<text x="28" y="72" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="23" fill="#F5F3EE" font-weight="800" text-anchor="start">Three OCI instances, redundant multi-node install</text>
<rect x="28" y="92" width="964" height="76" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="48" y="122" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">INTERNET</text>
<text x="972" y="122" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">one endpoint, whichever node answers</text>
<rect x="46" y="126" width="272.7" height="22" rx="7" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="182.3" y="140.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">aws-cli / Terraform / boto3 → :9999</text>
<rect x="327.7" y="126" width="177.2" height="22" rx="7" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="416.3" y="140.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">ssh → guest public IP</text>
<rect x="513.92" y="126" width="218.1" height="22" rx="7" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="623.0" y="140.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#9AA1A8" font-weight="400" text-anchor="middle">guest egress ← same address</text>
<line x1="510.0" y1="170" x2="510.0" y2="186" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.55" stroke-dasharray="2 4"/>
<rect x="28" y="188" width="964" height="464" rx="16" fill="#EE4B12" fill-opacity="0.022" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.45" stroke-dasharray="6 5"/>
<text x="50" y="218" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#EE4B12" font-weight="600" text-anchor="start" letter-spacing="2">OCI VCN 10.200.0.0/16 · PUBLIC SUBNET 10.200.0.0/23</text>
<text x="970" y="218" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="end">virtual router 10.200.0.1 · internet gateway</text>
<rect x="48.0" y="232" width="297.3" height="272" rx="13" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
<text x="66.0" y="262" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="800" text-anchor="start">spinifex-node-01</text>
<text x="327.3333333333333" y="262" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="400" text-anchor="end" fill-opacity="0.85">FAULT-DOMAIN-1</text>
<text x="66.0" y="280" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="start">BM.Standard.E2.64</text>
<line x1="66.0" y1="292" x2="327.3333333333333" y2="292" stroke="#EE4B12" stroke-opacity="0.25" stroke-width="1"/>
<text x="66.0" y="314" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">VNIC 0 10.200.0.252</text>
<text x="66.0" y="330" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="11" fill="#5B636C" font-weight="400" text-anchor="start">mesh · advertise · encap</text>
<text x="66.0" y="352" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">VNIC 1 br-wan 10.200.1.233/32</text>
<text x="66.0" y="368" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="11" fill="#5B636C" font-weight="400" text-anchor="start">public IP pairs · table 200</text>
<rect x="66.0" y="382" width="51.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="91.5" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">awsgw</text>
<rect x="123.0" y="382" width="57.2" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="151.6" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">daemon</text>
<rect x="186.2" y="382" width="44.8" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="208.6" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">vpcd</text>
<rect x="66.0" y="408" width="44.8" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="88.4" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">nats</text>
<rect x="116.8" y="408" width="82.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="157.8" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">predastore</text>
<rect x="204.8" y="408" width="82.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="245.8" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">viperblock</text>
<rect x="66.0" y="438" width="261.3" height="24" rx="8" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="76.0" y="454" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">i-23d2… 10.0.1.4 → 150.230.13.131</text>
<rect x="66.0" y="466" width="261.3" height="24" rx="8" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="76.0" y="482" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">iSCSI /dev/sdb → /var/lib/spinifex</text>
<line x1="345.3333333333333" y1="364" x2="361.3333333333333" y2="364" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.5" stroke-dasharray="2 4"/>
<rect x="361.3333333333333" y="232" width="297.3" height="272" rx="13" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
<text x="379.3333333333333" y="262" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="800" text-anchor="start">spinifex-node-02</text>
<text x="640.6666666666666" y="262" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="400" text-anchor="end" fill-opacity="0.85">FAULT-DOMAIN-2</text>
<text x="379.3333333333333" y="280" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="start">BM.Standard.E2.64</text>
<line x1="379.3333333333333" y1="292" x2="640.6666666666666" y2="292" stroke="#EE4B12" stroke-opacity="0.25" stroke-width="1"/>
<text x="379.3333333333333" y="314" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">VNIC 0 10.200.0.118</text>
<text x="379.3333333333333" y="330" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="11" fill="#5B636C" font-weight="400" text-anchor="start">mesh · advertise · encap</text>
<text x="379.3333333333333" y="352" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">VNIC 1 br-wan 10.200.1.96/32</text>
<text x="379.3333333333333" y="368" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="11" fill="#5B636C" font-weight="400" text-anchor="start">public IP pairs · table 200</text>
<rect x="379.3333333333333" y="382" width="51.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="404.8" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">awsgw</text>
<rect x="436.3333333333333" y="382" width="57.2" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="464.9" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">daemon</text>
<rect x="499.5333333333333" y="382" width="44.8" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="521.9" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">vpcd</text>
<rect x="379.3333333333333" y="408" width="44.8" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="401.7" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">nats</text>
<rect x="430.1333333333333" y="408" width="82.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="471.1" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">predastore</text>
<rect x="518.1333333333333" y="408" width="82.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="559.1" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">viperblock</text>
<rect x="379.3333333333333" y="438" width="261.3" height="24" rx="8" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="389.3333333333333" y="454" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">i-6b91… 10.0.1.9 → 137.23.9.125</text>
<rect x="379.3333333333333" y="466" width="261.3" height="24" rx="8" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="389.3333333333333" y="482" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">iSCSI /dev/sdb → /var/lib/spinifex</text>
<line x1="658.6666666666666" y1="364" x2="674.6666666666666" y2="364" stroke="#EE4B12" stroke-width="1.3" stroke-opacity="0.5" stroke-dasharray="2 4"/>
<rect x="674.6666666666666" y="232" width="297.3" height="272" rx="13" fill="#EE4B12" fill-opacity="0.05" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.5"/>
<text x="692.6666666666666" y="262" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="14" fill="#F5F3EE" font-weight="800" text-anchor="start">spinifex-node-03</text>
<text x="954.0" y="262" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#EE4B12" font-weight="400" text-anchor="end" fill-opacity="0.85">FAULT-DOMAIN-3</text>
<text x="692.6666666666666" y="280" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="start">BM.Standard.E2.64</text>
<line x1="692.6666666666666" y1="292" x2="954.0" y2="292" stroke="#EE4B12" stroke-opacity="0.25" stroke-width="1"/>
<text x="692.6666666666666" y="314" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">VNIC 0 10.200.0.41</text>
<text x="692.6666666666666" y="330" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="11" fill="#5B636C" font-weight="400" text-anchor="start">mesh · advertise · encap</text>
<text x="692.6666666666666" y="352" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11.5" fill="#F3F1EC" font-weight="400" text-anchor="start">VNIC 1 br-wan 10.200.1.12/32</text>
<text x="692.6666666666666" y="368" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="11" fill="#5B636C" font-weight="400" text-anchor="start">public IP pairs · table 200</text>
<rect x="692.6666666666666" y="382" width="51.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="718.2" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">awsgw</text>
<rect x="749.6666666666666" y="382" width="57.2" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="778.3" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">daemon</text>
<rect x="812.8666666666667" y="382" width="44.8" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="835.3" y="394.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">vpcd</text>
<rect x="692.6666666666666" y="408" width="44.8" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="715.1" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">nats</text>
<rect x="743.4666666666666" y="408" width="82.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="784.5" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">predastore</text>
<rect x="831.4666666666666" y="408" width="82.0" height="20" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="872.5" y="420.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">viperblock</text>
<rect x="692.6666666666666" y="438" width="261.3" height="24" rx="8" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="702.6666666666666" y="454" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">i-0ae4… 10.0.2.6 → 150.230.4.88</text>
<rect x="692.6666666666666" y="466" width="261.3" height="24" rx="8" fill="none" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="702.6666666666666" y="482" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#9AA1A8" font-weight="400" text-anchor="start">iSCSI /dev/sdb → /var/lib/spinifex</text>
<rect x="48" y="524" width="924" height="104" rx="13" fill="#181D23" fill-opacity="1.0" stroke="#2A2F36" stroke-width="1" stroke-opacity="1.0"/>
<text x="68" y="554" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="12" fill="#9AA1A8" font-weight="700" text-anchor="start" letter-spacing="1.5">EAST–WEST, INSIDE THE VCN</text>
<text x="952" y="554" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="10.5" fill="#5B636C" font-weight="400" text-anchor="end">no public address is involved</text>
<rect x="66" y="574" width="136.3" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="134.2" y="588.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">Geneve UDP 6081</text>
<rect x="211.3" y="574" width="149.9" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="286.3" y="588.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">NATS JetStream R3</text>
<rect x="370.24" y="574" width="156.8" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="448.6" y="588.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">predastore RS(2,1)</text>
<rect x="536.0" y="574" width="163.6" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="617.8" y="588.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">OVN Northbound raft</text>
<rect x="708.58" y="574" width="68.1" height="22" rx="7" fill="#EE4B12" fill-opacity="0.06" stroke="#EE4B12" stroke-width="1" stroke-opacity="0.5"/>
<text x="742.6" y="588.8" font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace" font-size="11" fill="#F3F1EC" font-weight="400" text-anchor="middle" fill-opacity="0.9">OVSDB</text>
<text x="66" y="612" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">Guest-to-guest traffic rides Geneve between the nodes' primary VNICs — it never leaves the subnet and never touches br-wan.</text>
<rect x="28" y="668" width="964" height="46" rx="12" fill="#EE4B12" fill-opacity="0.07" stroke="#EE4B12" stroke-width="1.4" stroke-opacity="0.55"/>
<text x="48" y="688" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="13" fill="#F5F3EE" font-weight="700" text-anchor="start">A guest that moves takes its address with it.</text>
<text x="48" y="705" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#9AA1A8" font-weight="400" text-anchor="start">The reserved public IP follows the secondary private IP to whichever node's br-wan the guest lands on — the tenant sees no change.</text>
<text x="28" y="756" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">Three nodes is the smallest cluster that tolerates losing one: NATS JetStream keeps quorum, predastore RS(2,1)</text>
<text x="28" y="774" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif" font-size="12" fill="#5B636C" font-weight="400" text-anchor="start">keeps the data readable, and the OVN Northbound raft keeps a leader. One node per fault domain is what makes that real.</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 21 KiB

.github/assets/images/cisco-ucs-platform-benchmark/guest-storage-throughput.svg
+44
View File
@@ -0,0 +1,44 @@
<svg xmlns="http://www.w3.org/2000/svg" width="760" height="470" viewBox="0 0 760 470" font-family="-apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif">
<rect width="760" height="470" fill="#ffffff"/>
<text x="74" y="30" font-size="19" font-weight="700" fill="#1f2937">Guest block storage throughput (Viperblock)</text>
<text x="74" y="49" font-size="12.5" fill="#6b7280">fio, 4 jobs, iodepth 32, direct I/O, per-guest sequential runs — Spinifex v1.20.0</text>
<line x1="74" y1="386.0" x2="734" y2="386.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="390.0" font-size="12" text-anchor="end" fill="#6b7280">0</text>
<line x1="74" y1="305.0" x2="734" y2="305.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="309.0" font-size="12" text-anchor="end" fill="#6b7280">100</text>
<line x1="74" y1="224.0" x2="734" y2="224.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="228.0" font-size="12" text-anchor="end" fill="#6b7280">200</text>
<line x1="74" y1="143.0" x2="734" y2="143.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="147.0" font-size="12" text-anchor="end" fill="#6b7280">300</text>
<line x1="74" y1="62.0" x2="734" y2="62.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="66.0" font-size="12" text-anchor="end" fill="#6b7280">400</text>
<text x="22" y="224.0" font-size="12.5" fill="#6b7280" transform="rotate(-90 22 224.0)" text-anchor="middle">MiB/s</text>
<rect x="111.4" y="81.4" width="45.4" height="304.6" rx="2.5" fill="#2563eb"/>
<text x="134.1" y="75.4" font-size="11" text-anchor="middle" fill="#1f2937">376</text>
<rect x="159.8" y="198.1" width="45.4" height="187.9" rx="2.5" fill="#0d9488"/>
<text x="182.5" y="192.1" font-size="11" text-anchor="middle" fill="#1f2937">232</text>
<rect x="208.2" y="299.3" width="45.4" height="86.7" rx="2.5" fill="#f59e0b"/>
<text x="230.9" y="293.3" font-size="11" text-anchor="middle" fill="#1f2937">107</text>
<text x="184.0" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">GPU</text>
<rect x="331.4" y="85.5" width="45.4" height="300.5" rx="2.5" fill="#2563eb"/>
<text x="354.1" y="79.5" font-size="11" text-anchor="middle" fill="#1f2937">371</text>
<rect x="379.8" y="202.9" width="45.4" height="183.1" rx="2.5" fill="#0d9488"/>
<text x="402.5" y="196.9" font-size="11" text-anchor="middle" fill="#1f2937">226</text>
<rect x="428.2" y="321.2" width="45.4" height="64.8" rx="2.5" fill="#f59e0b"/>
<text x="450.9" y="315.2" font-size="11" text-anchor="middle" fill="#1f2937">80</text>
<text x="404.0" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">CPU1</text>
<rect x="551.4" y="84.7" width="45.4" height="301.3" rx="2.5" fill="#2563eb"/>
<text x="574.1" y="78.7" font-size="11" text-anchor="middle" fill="#1f2937">372</text>
<rect x="599.8" y="201.3" width="45.4" height="184.7" rx="2.5" fill="#0d9488"/>
<text x="622.5" y="195.3" font-size="11" text-anchor="middle" fill="#1f2937">228</text>
<rect x="648.2" y="321.2" width="45.4" height="64.8" rx="2.5" fill="#f59e0b"/>
<text x="670.9" y="315.2" font-size="11" text-anchor="middle" fill="#1f2937">80</text>
<text x="624.0" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">CPU2</text>
<line x1="74" y1="386" x2="734" y2="386" stroke="#1f2937" stroke-width="1.5"/>
<rect x="74" y="444" width="13" height="13" rx="2.5" fill="#2563eb"/>
<text x="93" y="455" font-size="12.5" fill="#1f2937">128K random read</text>
<rect x="216.8" y="444" width="13" height="13" rx="2.5" fill="#0d9488"/>
<text x="235.8" y="455" font-size="12.5" fill="#1f2937">128K mixed read</text>
<rect x="352.3" y="444" width="13" height="13" rx="2.5" fill="#f59e0b"/>
<text x="371.3" y="455" font-size="12.5" fill="#1f2937">16K random read</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 3.7 KiB

.github/assets/images/cisco-ucs-platform-benchmark/network-goodput-vs-streams.svg
+35
View File
@@ -0,0 +1,35 @@
<svg xmlns="http://www.w3.org/2000/svg" width="760" height="470" viewBox="0 0 760 470" font-family="-apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif">
<rect width="760" height="470" fill="#ffffff"/>
<text x="74" y="30" font-size="19" font-weight="700" fill="#1f2937">Network throughput: overlay vs physical fabric</text>
<text x="74" y="49" font-size="12.5" fill="#6b7280">iperf3 TCP receiver goodput, median across node pairs — Spinifex v1.20.0</text>
<line x1="74" y1="386.0" x2="734" y2="386.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="390.0" font-size="12" text-anchor="end" fill="#6b7280">20</text>
<line x1="74" y1="62.0" x2="734" y2="62.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="66.0" font-size="12" text-anchor="end" fill="#6b7280">25</text>
<text x="22" y="224.0" font-size="12.5" fill="#6b7280" transform="rotate(-90 22 224.0)" text-anchor="middle">Gbit/s</text>
<text x="140.0" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">1</text>
<text x="272.0" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">2</text>
<text x="404.0" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">4</text>
<text x="536.0" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">8</text>
<text x="668.0" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">16</text>
<path d="M140.0,164.4 L272.0,160.5 L404.0,160.5 L536.0,160.5 L668.0,160.5" fill="none" stroke="#2563eb" stroke-width="3"/>
<circle cx="140.0" cy="164.4" r="4.5" fill="#fff" stroke="#2563eb" stroke-width="2.5"/>
<circle cx="272.0" cy="160.5" r="4.5" fill="#fff" stroke="#2563eb" stroke-width="2.5"/>
<circle cx="404.0" cy="160.5" r="4.5" fill="#fff" stroke="#2563eb" stroke-width="2.5"/>
<circle cx="536.0" cy="160.5" r="4.5" fill="#fff" stroke="#2563eb" stroke-width="2.5"/>
<circle cx="668.0" cy="160.5" r="4.5" fill="#fff" stroke="#2563eb" stroke-width="2.5"/>
<text x="664.0" y="149.5" font-size="11.5" text-anchor="end" font-weight="600" fill="#2563eb">23.48</text>
<path d="M140.0,255.1 L272.0,222.1 L404.0,222.1 L536.0,222.1 L668.0,228.5" fill="none" stroke="#0d9488" stroke-width="3"/>
<circle cx="140.0" cy="255.1" r="4.5" fill="#fff" stroke="#0d9488" stroke-width="2.5"/>
<circle cx="272.0" cy="222.1" r="4.5" fill="#fff" stroke="#0d9488" stroke-width="2.5"/>
<circle cx="404.0" cy="222.1" r="4.5" fill="#fff" stroke="#0d9488" stroke-width="2.5"/>
<circle cx="536.0" cy="222.1" r="4.5" fill="#fff" stroke="#0d9488" stroke-width="2.5"/>
<circle cx="668.0" cy="228.5" r="4.5" fill="#fff" stroke="#0d9488" stroke-width="2.5"/>
<text x="664.0" y="217.5" font-size="11.5" text-anchor="end" font-weight="600" fill="#0d9488">22.43</text>
<line x1="74" y1="386" x2="734" y2="386" stroke="#1f2937" stroke-width="1.5"/>
<text x="404.0" y="430.0" font-size="12.5" text-anchor="middle" fill="#6b7280">parallel TCP streams</text>
<rect x="74" y="444" width="13" height="13" rx="2.5" fill="#2563eb"/>
<text x="93" y="455" font-size="12.5" fill="#1f2937">Host physical fabric (25GbE)</text>
<rect x="304.4" y="444" width="13" height="13" rx="2.5" fill="#0d9488"/>
<text x="323.4" y="455" font-size="12.5" fill="#1f2937">Guest overlay (OVN)</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 3.2 KiB

.github/assets/images/cisco-ucs-platform-benchmark/predastore-s3-read-by-client.svg
+39
View File
@@ -0,0 +1,39 @@
<svg xmlns="http://www.w3.org/2000/svg" width="760" height="470" viewBox="0 0 760 470" font-family="-apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif">
<rect width="760" height="470" fill="#ffffff"/>
<text x="74" y="30" font-size="19" font-weight="700" fill="#1f2937">Predastore S3 read throughput by client</text>
<text x="74" y="49" font-size="12.5" fill="#6b7280">1 GiB objects, same presigned-curl client on every path — Spinifex v1.20.0</text>
<line x1="74" y1="386.0" x2="734" y2="386.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="390.0" font-size="12" text-anchor="end" fill="#6b7280">0</text>
<line x1="74" y1="278.0" x2="734" y2="278.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="282.0" font-size="12" text-anchor="end" fill="#6b7280">200</text>
<line x1="74" y1="170.0" x2="734" y2="170.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="174.0" font-size="12" text-anchor="end" fill="#6b7280">400</text>
<line x1="74" y1="62.0" x2="734" y2="62.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="66.0" font-size="12" text-anchor="end" fill="#6b7280">600</text>
<text x="22" y="224.0" font-size="12.5" fill="#6b7280" transform="rotate(-90 22 224.0)" text-anchor="middle">MiB/s</text>
<rect x="102.0" y="202.4" width="51.5" height="183.6" rx="2.5" fill="#2563eb"/>
<text x="127.8" y="196.4" font-size="11" text-anchor="middle" fill="#1f2937">340</text>
<rect x="156.5" y="136.5" width="51.5" height="249.5" rx="2.5" fill="#f59e0b"/>
<text x="182.2" y="130.5" font-size="11" text-anchor="middle" fill="#1f2937">462</text>
<text x="156.5" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">Host</text>
<rect x="267.1" y="170.5" width="51.5" height="215.5" rx="2.5" fill="#2563eb"/>
<text x="292.8" y="164.5" font-size="11" text-anchor="middle" fill="#1f2937">399</text>
<rect x="321.5" y="174.9" width="51.5" height="211.1" rx="2.5" fill="#f59e0b"/>
<text x="347.2" y="168.9" font-size="11" text-anchor="middle" fill="#1f2937">391</text>
<text x="321.5" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">GPU guest</text>
<rect x="432.1" y="102.5" width="51.5" height="283.5" rx="2.5" fill="#2563eb"/>
<text x="457.8" y="96.5" font-size="11" text-anchor="middle" fill="#1f2937">525</text>
<rect x="486.5" y="197.0" width="51.5" height="189.0" rx="2.5" fill="#f59e0b"/>
<text x="512.2" y="191.0" font-size="11" text-anchor="middle" fill="#1f2937">350</text>
<text x="486.5" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">CPU1 guest</text>
<rect x="597.0" y="106.8" width="51.5" height="279.2" rx="2.5" fill="#2563eb"/>
<text x="622.8" y="100.8" font-size="11" text-anchor="middle" fill="#1f2937">517</text>
<rect x="651.5" y="196.5" width="51.5" height="189.5" rx="2.5" fill="#f59e0b"/>
<text x="677.2" y="190.5" font-size="11" text-anchor="middle" fill="#1f2937">351</text>
<text x="651.5" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">CPU2 guest</text>
<line x1="74" y1="386" x2="734" y2="386" stroke="#1f2937" stroke-width="1.5"/>
<rect x="74" y="444" width="13" height="13" rx="2.5" fill="#2563eb"/>
<text x="93" y="455" font-size="12.5" fill="#1f2937">Single-stream GET</text>
<rect x="224.1" y="444" width="13" height="13" rx="2.5" fill="#f59e0b"/>
<text x="243.1" y="455" font-size="12.5" fill="#1f2937">Chunked/concurrent GET</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 3.3 KiB

.github/e2e-cells.json
+47
View File
@@ -143,5 +143,52 @@
"extra": "",
"suite": "storagefault",
"timeout_minutes": 100
},
{
"cell": 30,
"install": "binary",
"topo": "real-multi",
"net": "nat",
"host": "debian-13",
"arch": "x86_64",
"runner": "ci-multi",
"extra": "nat-multi",
"suite": ""
},
{
"cell": 31,
"install": "binary",
"topo": "real-multi",
"net": "static",
"host": "debian-13",
"arch": "x86_64",
"runner": "ci-multi",
"extra": "",
"suite": "instancerecovery",
"timeout_minutes": 105
},
{
"cell": 32,
"install": "binary",
"topo": "real-multi",
"net": "static",
"host": "debian-13",
"arch": "x86_64",
"runner": "ci-multi",
"extra": "",
"suite": "lbrecovery",
"timeout_minutes": 90
},
{
"cell": 33,
"install": "binary",
"topo": "real-multi",
"net": "static",
"host": "debian-13",
"arch": "x86_64",
"runner": "ci-multi",
"extra": "",
"suite": "kvquorum",
"timeout_minutes": 55
}
]
.github/scripts/suite-sets.sh
+39 -1
View File
@@ -40,7 +40,24 @@
E2E_SUITES_SINGLE="single iam cert eks ecs storagegrowth partialblock rds quota storagefault"
# Suites runnable against a multi-node environment.
E2E_SUITES_MULTI="multinode lb cert quota storagefault"
#
# instancerecovery is multi-only and needs three: recovery is inert below that,
# because there is no survivor to move a guest to that is not the node that
# failed. Like storagefault it takes a node away and freezes a cluster-wide
# service, so it runs last and owns its environment.
#
# lbrecovery is multi-only and needs three for two reasons rather than one: a
# survivor to move the load balancer to, and a node for its backends that is not
# the load balancer's, so taking that node away does not take the backends with
# it. It runs beside instancerecovery at the end because it also takes a node
# away and owns its environment while it does.
#
# kvquorum is multi-only and needs three, for the same reason as those two: a
# surviving majority. It is what proves the control plane's own state is
# replicated rather than sitting on one node, so it runs before the pair that
# take a node away — a cluster whose buckets are already wrong should report
# that, not report a recovery failing.
E2E_SUITES_MULTI="multinode lb cert quota kvquorum lbrecovery instancerecovery storagefault"
# grep -xE alternation form, for narrowing a requested set to the eligible one.
E2E_SUITES_SINGLE_RE="$(printf '%s' "${E2E_SUITES_SINGLE}" | tr ' ' '|')"
@@ -64,6 +81,10 @@ E2E_SUITES_MULTI_RE="$(printf '%s' "${E2E_SUITES_MULTI}" | tr ' ' '|')"
# after a failed seal do not exist on one node, and they are the assertions
# worth a nightly.
E2E_SUITES_NIGHTLY_SINGLE="single cert iam"
#
# The multi-node nightly subset carries kvquorum's read-only half through
# `multinode`, which sweeps every KV bucket rather than the three the daemon
# owns. The node-loss half is too slow for a 35-minute cell and has its own.
E2E_SUITES_NIGHTLY_MULTI="multinode cert lb"
# How long a suite gets, for the same reason the lists above live here: the
@@ -88,6 +109,23 @@ e2e_suite_timeout() {
# SIGSTOPped for seven minutes to prove a guest survives it. A full pass on
# dev-prod is ~63 minutes, so the 30m default truncated it mid-suite.
storagefault) echo "90m" ;;
# instancerecovery is almost entirely waiting, and none of it is avoidable:
# a dead owner has to be seen stale twice, a guest has to boot, and the
# storage case waits for a pause measured at 4m47s and then holds four
# minutes to prove nothing moved. Three tests, each taking a node away and
# bringing it back, and the partition case adds a lease validity, a lease
# TTL and a boot on top of everything the other two wait for.
instancerecovery) echo "90m" ;;
# lbrecovery pays for a load balancer twice: once to stand one up with two
# backends placed around it, and once for the recovery itself — a stale
# owner, a lease, a boot, HAProxy's own start, and its health checks against
# the backends before it forwards anything. Then the node comes back and has
# to be shown not to answer for an address it no longer holds.
lbrecovery) echo "75m" ;;
# kvquorum audits every bucket in seconds and then spends the rest of its
# budget on a node stopping and starting: seven units down, a raft election,
# and the whole service stack back up with every stream caught up.
kvquorum) echo "40m" ;;
*) echo "30m" ;;
esac
}
.github/workflows/e2e-nightly.yml
+86 -25
View File
@@ -1,18 +1,16 @@
name: E2E Nightly
# Triggered nightly by a Cloudflare Worker that dispatches this workflow once on
# `dev` and once on `main`, rather than by a `schedule:` here. GitHub registers a
# schedule only from the default branch, so a cron in this file can never run
# `main` and `dev` as separate nightlies, and it would always execute `main`'s
# copy of this file — leaving a fix to the workflow itself untestable until it
# had already shipped. A dispatch takes both the file and the source from its ref.
on:
schedule:
# 14:00 UTC is midnight AEST (01:00 AEDT — cron has no DST), when the
# hypervisors are idle. Every day, not weeknights: a weekend gap is two runs
# of evidence lost on a suite whose remaining failures are a rate, not a bug.
# :17 rather than :00 because GitHub queues scheduled runs behind every other
# top-of-hour cron and drops them under load, which is why 2026-09-14 never
# fired and why every run in that window started 3h15m-3h45m late.
- cron: "17 14 * * *"
workflow_dispatch:
inputs:
cells:
description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection"
description: "Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes, 33 KV bucket quorum under node loss on three nodes"
required: false
permissions:
@@ -33,7 +31,7 @@ concurrency:
cancel-in-progress: false
env:
GIT_BRANCH: ${{ github.event_name == 'schedule' && 'dev' || github.ref_name }}
GIT_BRANCH: ${{ github.ref_name }}
UBUNTU_GOLD_IMAGE: /mnt/disk1/gold-images/spinifex-image-builder-ubuntu-26.04.qcow2
SPINIFEX_OTLP_ENDPOINT: ${{ vars.SPINIFEX_OTLP_ENDPOINT || 'http://192.168.1.13:8200' }}
SPINIFEX_OTEL_ATTRS: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }}
@@ -280,8 +278,31 @@ jobs:
rc=\${PIPESTATUS[0]}; echo '::endgroup::'; \
if [ \$rc -ne 0 ]; then echo 'natuplink: FAILED' >> '${ARTIFACT_DIR}/suite-status.txt'; fi; exit \$rc"
# natuplink proves a guest with a public IP reaches the internet. The two
# surfaces it never touches are a private subnet's isolation and its NAT
# gateway, and TestVPCEgressPaths is the authority on both — it skipped in
# nat mode until the fixture learned that routed NAT carries EIPs too.
- name: Run VPC Egress Paths in NAT mode
if: matrix.extra == 'nat-single'
working-directory: mulga/scripts/tofu-cluster
run: |
INVENTORY=$(tofu output -json inventory)
WAN_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
SSH_KEY="${SSH_KEY/#\~/$HOME}"
ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${WAN_IP} \
"set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; \
echo '::group::suite single/TestVPCEgressPaths'; \
date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-vpcegress-nat.start'; \
GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
ARTIFACT_DIR='${ARTIFACT_DIR}' \
./single.test -test.v -test.timeout=30m -test.run '^TestVPCEgressPaths\$' \
2>&1 | tee '${ARTIFACT_DIR}/test-vpcegress-nat.log'; \
rc=\${PIPESTATUS[0]}; echo '::endgroup::'; \
if [ \$rc -ne 0 ]; then echo 'vpcegress-nat: FAILED' >> '${ARTIFACT_DIR}/suite-status.txt'; fi; exit \$rc"
- name: Run Multi Node E2E
if: matrix.topo == 'real-multi' && matrix.suite == ''
if: matrix.topo == 'real-multi' && matrix.suite == '' && matrix.extra == ''
working-directory: mulga/scripts/tofu-cluster
run: |
set -euo pipefail
@@ -316,6 +337,40 @@ jobs:
if [ \$rc -ne 0 ]; then echo \"\$SUITE: FAILED\" >> '${ARTIFACT_DIR}/suite-status.txt'; FAIL=1; fi; \
done; exit \$FAIL"
# The same suite as the nat-single cell, run against three nodes: there it
# proves routed NAT works at all, here that each node delivers its own
# guests' public IPs and no others — which one node cannot show.
- name: Run NAT Uplink Multi E2E
if: matrix.extra == 'nat-multi'
working-directory: mulga/scripts/tofu-cluster
run: |
set -euo pipefail
INVENTORY=$(tofu output -json inventory)
NODE1_IP=$(echo "$INVENTORY" | jq -r '.nodes[0].wan_ip')
SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
SSH_KEY="${SSH_KEY/#\~/$HOME}"
NODES_CSV=""
NODE_COUNT=$(echo "$INVENTORY" | jq '.nodes | length')
for i in $(seq 0 $((NODE_COUNT - 1))); do
ip=$(echo "$INVENTORY" | jq -r ".nodes[$i].wan_ip")
NODES_CSV="${NODES_CSV:+$NODES_CSV,}$ip"
done
ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" tf-user@${NODE1_IP} \
"set -u; mkdir -p '${ARTIFACT_DIR}'; cd \$HOME/e2e-tests/_bin; \
if [ ! -f natuplink.test ]; then echo 'natuplink.test not in tests tar (branch predates suite) — skipping'; exit 0; fi; \
echo '::group::suite natuplink'; \
date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-natuplink.start'; \
GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=multinode \
SPINIFEX_NODES='${NODES_CSV}' SPINIFEX_NODE_IPS='${NODES_CSV}' \
SPINIFEX_SSH_USER=tf-user SPINIFEX_SSH_KEY=\$HOME/.ssh/tf-user-ap-southeast-2 \
ARTIFACT_DIR='${ARTIFACT_DIR}' \
./natuplink.test -test.v -test.timeout=30m \
2>&1 | tee '${ARTIFACT_DIR}/test-natuplink.log'; \
rc=\${PIPESTATUS[0]}; echo '::endgroup::'; \
if [ \$rc -ne 0 ]; then echo 'natuplink: FAILED' >> '${ARTIFACT_DIR}/suite-status.txt'; fi; exit \$rc"
# The dedicated suite cells only. Resolved once here rather than in each
# step below, which would repeat the same three lines five times.
- name: Resolve VM SSH details
@@ -822,12 +877,11 @@ jobs:
# Bare-metal single-node suite. Doesn't fit the tofu matrix shape, so it
# runs as a sibling job via the reusable e2e-baremetal workflow. Gated like a
# matrix cell ("cell 20"): every nightly cron, or dispatch with 20 selected
# (blank cells = all). Tests `dev` on schedule to match GIT_BRANCH above.
# matrix cell ("cell 20"): blank cells is the whole nightly, or name 20 to run
# it alone. A reusable call needs its own ref, since it cannot read GIT_BRANCH.
baremetal:
name: cell-20 (baremetal/pxe/real-hw)
if: >-
github.event_name == 'schedule' ||
inputs.cells == '' ||
contains(format(',{0},', inputs.cells), ',20,')
permissions:
@@ -835,7 +889,7 @@ jobs:
statuses: write
uses: ./.github/workflows/e2e-baremetal.yml
with:
ref: ${{ github.event_name == 'schedule' && 'dev' || github.ref_name }}
ref: ${{ github.ref_name }}
# A reusable-workflow call gets none of this workflow's context (env:
# above, matrix.cell) for free — cell-20 needs its own attrs string.
# There's no matrix leg here, just the one fixed baremetal cell.
@@ -855,7 +909,6 @@ jobs:
needs: baremetal
if: >-
!cancelled() && (
github.event_name == 'schedule' ||
inputs.cells == '' ||
contains(format(',{0},', inputs.cells), ',28,')
)
@@ -864,7 +917,7 @@ jobs:
statuses: write
uses: ./.github/workflows/e2e-baremetal.yml
with:
ref: ${{ github.event_name == 'schedule' && 'dev' || github.ref_name }}
ref: ${{ github.ref_name }}
cell: "28"
suites: diskperf
otel-attrs: ci.run_id=${{ github.run_id }},ci.workflow=${{ github.workflow }},ci.cell=28,ci.matrix=baremetal/pxe/diskperf
@@ -881,7 +934,6 @@ jobs:
iso-single:
name: cell-21 (iso/dhcp/single)
if: >-
github.event_name == 'schedule' ||
inputs.cells == '' ||
contains(format(',{0},', inputs.cells), ',21,')
runs-on: [self-hosted, ci-single]
@@ -1136,7 +1188,6 @@ jobs:
needs: [iso-single]
if: >-
!cancelled() && (
github.event_name == 'schedule' ||
inputs.cells == '' ||
contains(format(',{0},', inputs.cells), ',22,') )
runs-on: [self-hosted, ci-multi]
@@ -1298,7 +1349,6 @@ jobs:
needs: [matrix, baremetal, iso-single, iso-multi]
if: >-
!cancelled() && (
github.event_name == 'schedule' ||
inputs.cells == '' ||
contains(format(',{0},', inputs.cells), ',23,') )
runs-on: [self-hosted, ci-single]
@@ -1329,7 +1379,7 @@ jobs:
- name: Resolve sibling refs
id: siblings
env:
BRANCH: ${{ github.event_name == 'schedule' && 'dev' || github.ref_name }}
BRANCH: ${{ github.ref_name }}
run: bash .github/resolve-sibling-refs.sh
working-directory: spinifex
@@ -1410,7 +1460,6 @@ jobs:
needs: [matrix, baremetal, iso-single, iso-multi, performance]
if: >-
!cancelled() && (
github.event_name == 'schedule' ||
inputs.cells == '' ||
contains(format(',{0},', inputs.cells), ',24,') )
runs-on: [self-hosted, ci-single]
@@ -1454,7 +1503,7 @@ jobs:
- name: Resolve sibling refs
id: siblings
env:
BRANCH: ${{ github.event_name == 'schedule' && 'dev' || github.ref_name }}
BRANCH: ${{ github.ref_name }}
run: bash .github/resolve-sibling-refs.sh
working-directory: spinifex
@@ -1493,8 +1542,16 @@ jobs:
# Leftovers from a run killed before its cleanup trap: a surviving node
# holds its ports and fails this run somewhere unrelated.
#
# Under the host benchmark lock, because this pattern matches every
# predastore node on the runner and the predastore repository runs the
# same harnesses on the same machine. Holding the lock is what makes
# "still running" mean "stale" rather than "somebody else's benchmark":
# on 2026-09-16 this step killed a predastore run 32s into its workload.
- name: Clear stale nodes
run: pkill -KILL -f '/bin/s3d -config' || true
run: |
flock -w 3600 "${PREDA_BENCH_LOCK:-/var/lock/predastore-bench.lock}" \
pkill -KILL -f '/bin/s3d -config' || true
- name: Run fault injection
working-directory: predastore
@@ -1514,10 +1571,14 @@ jobs:
if-no-files-found: warn
retention-days: 90
# Under the lock for the same reason as "Clear stale nodes": by the time
# this runs the harness has released it, so another repository's benchmark
# can already have a cluster of its own up.
- name: Clean up
if: always()
run: |
pkill -KILL -f '/bin/s3d -config' || true
flock -w 3600 "${PREDA_BENCH_LOCK:-/var/lock/predastore-bench.lock}" \
pkill -KILL -f '/bin/s3d -config' || true
rm -rf "$STRESS_RESULTS_ROOT"
# Reading a nightly meant opening every cell job, and the Actions UI does not
.golangci.yml
+1 -2
View File
@@ -56,7 +56,6 @@ linters:
- unconvert # removes unnecessary type conversions
- usestdlibvars # detects hardcoded values that should use stdlib constants
- usetesting # catches os.TempDir() etc. in tests — should use t.TempDir()
- wastedassign # finds assignments to variables that are never subsequently read
- whitespace # detects leading and trailing whitespace in functions
settings:
@@ -151,7 +150,7 @@ linters:
- -ST1020 # godoc on exported types — not enforcing yet
- -ST1021 # godoc on exported type groups — not enforcing yet
- -ST1022 # godoc on exported vars/consts — not enforcing yet
- -SA4006 # value overwritten — ineffassign + wastedassign cover this better
- -SA4006 # value overwritten — ineffassign catches strictly more of these
- -SA9005 # marshaling structs with no exported fields — intended {} for empty API-response types
testifylint:
CONTRIBUTING.md
+96
View File
@@ -0,0 +1,96 @@
# How to Contribute
We welcome contributions to Spinifex!
Report security vulnerabilities privately through [SECURITY.md](SECURITY.md), not as a public issue.
**AWS parity is the design rule.** When Spinifex's behaviour is in question, the answer is what AWS does. If you believe Spinifex diverges from AWS, state what AWS does and show it with a test or a reproduction against a running cluster.
## Understanding the Codebase
Every AWS call enters through a SigV4-authenticated gateway, is published to NATS, and is answered by the daemon that claims it. Start with these:
| Doc | Covers |
|---|---|
| [`docs/DESIGN.md`](docs/DESIGN.md) | Architecture, request flow, daemons, storage integration, file layout. |
| [`docs/COMMANDS.md`](docs/COMMANDS.md) | The `spx admin` CLI and the systemd units it manages. |
| [`docs/coverage/`](docs/coverage/README.md) | Which AWS API operations each service implements. |
| [`scripts/README.md`](scripts/README.md) | Install, dev-environment, and image-build scripts. |
User-facing docs are published at [docs.mulgadc.com](https://docs.mulgadc.com).
## Development Setup
These steps build Spinifex from source and run it as a single-node cluster, following the [Source Install](https://docs.mulgadc.com/docs/install-source) guide.
### Prerequisites
- **Ubuntu 26.04** or **Debian 13**, on x86_64 or aarch64
- A host with hardware virtualisation (KVM) and `sudo` access
- The host's WAN interface enslaved to a Linux bridge named `br-wan`, with the host IP, default route and DHCP on the bridge rather than the bare NIC (see [VPC Networking → Bridge Setup](https://docs.mulgadc.com/docs/vpc-networking#bridge-setup-physical-network-wiring))
Check the bridge before continuing:
```bash
ip -br link show br-wan # bridge exists and is UP
ip route # default route's dev is br-wan
```
### Install and Run
```bash
mkdir -p ~/Development/mulga && cd ~/Development/mulga
git clone https://github.com/mulgadc/spinifex.git
sudo make -C spinifex quickinstall # system packages, Go 1.27, AWS CLI v2
export PATH=$PATH:/usr/local/go/bin
cd spinifex
./scripts/clone-deps.sh # sibling repos: viperblock, predastore, northstar, bluebottle
./scripts/dev-install.sh # build, install, init a one-node cluster, start services
AWS_PROFILE=spinifex aws ec2 describe-instance-types
```
A list of instance types means the cluster is up.
`make preflight` also needs `shellcheck` and `golangci-lint` (CI pins v2.13.0):
```bash
sudo apt-get install -y shellcheck
curl -sSfL https://golangci-lint.run/install.sh | sh -s -- -b "$(go env GOPATH)/bin" v2.13.0
```
The web console additionally needs Node.js 24 and `pnpm`.
## Development Workflow
| Command | Use it to |
|---|---|
| `make deploy` | Rebuild, install `spx`, and restart `spinifex.target` after a code change. |
| `make reinstall` | Re-run `dev-install.sh` after changing systemd units, helper scripts, or logrotate config. |
| `./scripts/reset-dev-env.sh` | Wipe all cluster state and reinstall from scratch, keeping your network topology. |
| `make test` | Run unit tests (`make test-race` for the race detector). |
| `make test-integration` | Run the real gateway router against embedded NATS JetStream. |
| `make preflight` | Run the pre-commit gate: lint, `govulncheck`, coverage, and manifest and script checks. Must pass. |
Follow the services with `journalctl -u 'spinifex-*' -f`.
The web console lives in `spinifex/services/spinifexui/frontend/` (`pnpm install`, `pnpm dev`, `pnpm lint`, `pnpm test`). `spx` embeds its checked-in `dist/`, so run `make build-ui` and commit the rebuilt `dist/` with your change.
## Creating a Pull Request
1. Fork [mulgadc/spinifex](https://github.com/mulgadc/spinifex) and clone your fork into the same parent directory as the sibling repositories, branching off `dev`:
```bash
cd ~/Development/mulga
git clone git@github.com:YOUR_USERNAME/spinifex.git && cd spinifex
git remote add upstream git@github.com:mulgadc/spinifex.git
git fetch upstream && git checkout -b feat/my-feature upstream/dev
```
2. Make your change, then verify it:
- `make preflight` passes.
- `go mod tidy` leaves only intended changes in `go.mod` and `go.sum`, with none of the `clone-deps.sh` `replace` lines.
- Runtime changes are exercised on your dev cluster with `make deploy`.
3. Commit using [Conventional Commits](https://www.conventionalcommits.org/) (`feat:`, `fix:`, `refactor:`, `docs:`, `chore:`), push, and open a pull request against `dev`, filling in the template.
Makefile
+25 -6
View File
@@ -149,21 +149,38 @@ install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share
# the pre-commit gate: manifest checks, lint, vuln, and the unit and e2e-harness tiers.
# integration and race tests skipped to keep quick, they run in CI.
#
# -j2 overlaps the only two heavy gates, lint and test-cover: alone each leaves
# most of the machine idle, so pairing them cuts a cold run by about a quarter.
# -Otarget keeps each gate's output in one block instead of interleaving them.
preflight:
@$(MAKE) --no-print-directory QUIET=1 manifest-check manifest-lint lint govulncheck test-cover diff-coverage test-package-check test-harness test-build-scripts
@$(MAKE) --no-print-directory -j2 -Otarget QUIET=1 manifest-check manifest-lint lint govulncheck test-cover diff-coverage test-package-check test-harness test-build-scripts
@echo -e "\n ✅ Preflight passed — safe to commit."
# Shell suites + shellcheck for build/scripts/, the systemd-unit helpers that
# ship on every node (unlike scripts/images/, kept in preflight: a wrong
# restart decision here is an outage, not asset churn).
#
# NODE_SCRIPTS are the four that setup.sh installs into
# /usr/local/share/spinifex on every node. They are covered here for the same
# reason: between them they install the software, wire OVN, erase the state and
# remove the install, so a shellcheck warning in one is an outage or a wrong
# deletion, not a style note.
NODE_SCRIPTS := scripts/setup.sh scripts/setup-ovn.sh scripts/node-reset.sh scripts/uninstall-spx.sh
test-build-scripts:
@echo -e "\n....Running build/scripts/**/*_test.sh...."
@for t in $$(find build/scripts -name '*_test.sh' | sort); do \
echo "-- $$t"; \
sh "$$t" || exit 1; \
done
@echo -e "\n....Running shellcheck over build/scripts/**/*.sh...."
shellcheck -S warning $$(find build/scripts -name '*.sh' | sort)
@echo -e "\n....Running scripts/*_test.sh...."
@for t in $$(find scripts -maxdepth 1 -name '*_test.sh' | sort); do \
echo "-- $$t"; \
sh "$$t" || exit 1; \
done
@echo -e "\n....Running shellcheck over build/scripts/**/*.sh + the node scripts...."
shellcheck -S warning $$(find build/scripts -name '*.sh' | sort) $(NODE_SCRIPTS)
@echo " test-build-scripts ok"
# E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
@@ -185,16 +202,18 @@ AWS_MODEL_OPERATION_COVERAGE_DIR ?= $(CURDIR)/docs/coverage
# -count=1 is load-bearing: the conformance report is written by the test binary,
# so a cached pass skips the run, leaves no report, and the cat below fails the
# target. It also keeps the conformance gate honest — a cached result would mean
# the check never ran against this commit.
# the check never ran against this commit. The generated request sweep adds
# about 15s, most of it EC2 fan-outs waiting out their window for instances
# no daemon-lite owns.
test-integration:
@echo -e "\n....Running in-process integration tests...."
$(_Q)LOG_IGNORE=1 AWS_MODEL_CONFORMANCE_MODE=$(AWS_MODEL_CONFORMANCE_MODE) AWS_MODEL_CONFORMANCE_REPORT=$(AWS_MODEL_CONFORMANCE_REPORT) go test -count=1 -tags=integration -timeout 60s ./tests/integration/... $(_RACEQ)
$(_Q)LOG_IGNORE=1 AWS_MODEL_CONFORMANCE_MODE=$(AWS_MODEL_CONFORMANCE_MODE) AWS_MODEL_CONFORMANCE_REPORT=$(AWS_MODEL_CONFORMANCE_REPORT) go test -count=1 -tags=integration -timeout 120s ./tests/integration/... $(_RACEQ)
@cat $(AWS_MODEL_CONFORMANCE_REPORT)
@$(MAKE) --no-print-directory generate-aws-model-coverage
@echo "AWS operation coverage: $(AWS_MODEL_OPERATION_COVERAGE_DIR)"
generate-aws-model-coverage:
@go run ./cmd/aws-model-coverage -out $(AWS_MODEL_OPERATION_COVERAGE_DIR)
@go run ./cmd/aws-model-coverage -out $(AWS_MODEL_OPERATION_COVERAGE_DIR) -json $(AWS_MODEL_OPERATION_COVERAGE_DIR)/coverage.json
aws-model-coverage: generate-aws-model-coverage
@go run ./cmd/aws-model-coverage
cmd/aws-model-coverage/main.go
+11 -3
View File
@@ -14,18 +14,26 @@ import (
func main() {
outputDir := flag.String("out", "", "write one publishable page per service into this directory")
jsonOutput := flag.String("json", "", "write a machine-readable coverage inventory to this file")
flag.Parse()
coverages, err := compareAll()
if err != nil {
fail(err)
}
if *outputDir == "" {
if *outputDir == "" && *jsonOutput == "" {
fmt.Print(awsmodel.RenderCoverageSummary(coverages))
return
}
if err := awsmodel.WritePages(*outputDir, coverages); err != nil {
fail(err)
if *outputDir != "" {
if err := awsmodel.WritePages(*outputDir, coverages); err != nil {
fail(err)
}
}
if *jsonOutput != "" {
if err := awsmodel.WriteCoverageJSON(*jsonOutput, coverages); err != nil {
fail(err)
}
}
}
cmd/ecs-agent/agent.go
+38 -22
View File
@@ -30,10 +30,13 @@ type Agent struct {
cfg config
id identity
cp controlPlane
puller ctrruntime.ImagePuller
runner ctrruntime.Runner
resolver ctrruntime.Resolver
// rt holds the container runtime, which is resolved by a loop rather than
// once at boot. Empty until the containerd socket answers, and emptied again
// if it stops answering.
rt *runtimeGate
reg *registrar
hb *heartbeater
@@ -48,26 +51,34 @@ type Agent struct {
// nvidia-smi discovery.
gpu *gpuLedger
// dialRuntime connects to the container runtime. nil in unit tests, which
// drive the gate directly.
dialRuntime func() (ctrruntime.Runtime, error)
closers []func() error
}
// newAgent assembles an Agent from already-built seams. Tests use this directly
// with fakes; New builds the production seams and delegates here. runner may be
// nil when containerd is unavailable; the assign path then reports the task
// STOPPED rather than crashing.
func newAgent(cfg config, id identity, cp controlPlane, puller ctrruntime.ImagePuller, runner ctrruntime.Runner, resolver ctrruntime.Resolver) *Agent {
return &Agent{
// with fakes; New builds the production seams and delegates here. rt may be nil
// when containerd is unavailable; the assign path then reports the task STOPPED
// rather than crashing, and the watch loop keeps dialling for it.
func newAgent(cfg config, id identity, cp controlPlane, rt ctrruntime.Runtime, resolver ctrruntime.Resolver) *Agent {
gate := newRuntimeGate(rt)
a := &Agent{
cfg: cfg,
id: id,
cp: cp,
puller: puller,
runner: runner,
resolver: resolver,
reg: newRegistrar(cp, id),
hb: newHeartbeater(cp, id, cfg.Heartbeat),
rt: gate,
netns: newTaskNetns(execNetRunner{}),
gpu: newGPULedger(id.Capacity.GPUIDs),
}
// Registration reports the host's capacity only while a runtime is there to
// use it, so an agent that cannot run a container advertises none and takes
// no placement.
a.reg = newRegistrar(cp, id, gate.available)
a.hb = newHeartbeater(cp, id, cfg.Heartbeat, gate.available)
return a
}
// New builds an Agent from config: it resolves the host identity from IMDS,
@@ -89,6 +100,7 @@ func New(cfg config) (*Agent, error) {
ClusterName: cfg.ClusterName,
InstanceID: meta.InstanceID,
AZ: meta.AZ,
InstanceType: meta.InstanceType,
Hostname: host,
Capacity: detectCapacity(discoverNvidiaGPUs(execCommandRunner)),
AgentVersion: version,
@@ -97,13 +109,12 @@ func New(cfg config) (*Agent, error) {
creds := credentials.NewIMDSProvider(imdsClient, cfg.IMDSBase)
resolver := newLazyECRResolver(creds, cfg.Region, cfg.GatewayURL, cfg.GatewayCA)
var puller ctrruntime.ImagePuller
var runner ctrruntime.Runner
if rt, perr := ctrruntime.New(cfg.ContainerdSocket); perr != nil {
slog.Warn("ecs-agent: containerd unavailable at boot, image pulls disabled", "err", perr)
} else {
puller = rt
runner = rt
// A dial that fails here is not fatal and not final: the node starts the
// container daemon and the agent back to back, so the socket is often not
// accepting yet. Run's watch loop keeps dialling.
rt, perr := ctrruntime.New(cfg.ContainerdSocket)
if perr != nil {
slog.Warn("ecs-agent: containerd unavailable at boot, will keep retrying", "err", perr)
}
cp, err := newGatewayControlPlane(cfg, creds)
@@ -111,12 +122,10 @@ func New(cfg config) (*Agent, error) {
return nil, fmt.Errorf("build gateway control-plane: %w", err)
}
a := newAgent(cfg, id, cp, puller, runner, resolver)
a := newAgent(cfg, id, cp, rt, resolver)
a.cred = newCredEndpoint(creds, cfg.Region, cfg.GatewayURL, cfg.GatewayCA,
cfg.CredEndpointIP, cfg.CredEndpointPort, execNetRunner{})
if puller != nil {
a.closers = append(a.closers, puller.Close)
}
a.dialRuntime = func() (ctrruntime.Runtime, error) { return ctrruntime.New(cfg.ContainerdSocket) }
return a, nil
}
@@ -137,6 +146,12 @@ func (a *Agent) Run(ctx context.Context) error {
go a.hb.Run(ctx)
// Keeps the runtime current for the life of the process: dials while there
// is none, and drops one that stops answering.
if a.dialRuntime != nil {
go a.rt.watch(ctx, a.dialRuntime, a.cfg.RuntimeRetry)
}
// Re-adopt containers still running from before this restart, then poll with
// their tasks pre-seeded so re-delivered assignments are acked, not re-run.
go a.pollAssignments(ctx, a.reconcile(ctx))
@@ -156,6 +171,7 @@ func (a *Agent) Stop() error {
firstErr = err
}
}
a.rt.clear()
return firstErr
}
cmd/ecs-agent/agent_test.go
+1 -1
View File
@@ -12,7 +12,7 @@ func TestAgent_RunRegistersThenStopsOnContext(t *testing.T) {
cp := &fakeCP{}
cfg := config{Heartbeat: 5 * time.Millisecond, PollInterval: 5 * time.Millisecond}
puller := &ctrruntime.FakePuller{}
a := newAgent(cfg, testIdentity(), cp, puller, puller, nil)
a := newAgent(cfg, testIdentity(), cp, puller, nil)
a.closers = append(a.closers, puller.Close)
ctx, cancel := context.WithCancel(context.Background())
cmd/ecs-agent/config.go
+7
View File
@@ -33,6 +33,7 @@ type config struct {
ContainerdSocket string
Heartbeat time.Duration
PollInterval time.Duration
RuntimeRetry time.Duration
CredEndpointIP string
CredEndpointPort int
}
@@ -75,6 +76,12 @@ func loadConfig(envFile string) config {
cfg.PollInterval = d
}
}
cfg.RuntimeRetry = defaultRuntimeRetry
if v := get("ECS_RUNTIME_RETRY"); v != "" {
if d, err := time.ParseDuration(v); err == nil && d > 0 {
cfg.RuntimeRetry = d
}
}
cfg.CredEndpointIP = get("ECS_CRED_ENDPOINT_IP")
if cfg.CredEndpointIP == "" {
cfg.CredEndpointIP = defaultCredEndpointIP
cmd/ecs-agent/controlplane.go
+4
View File
@@ -76,6 +76,10 @@ func (g *gatewayControlPlane) Register(id identity) error {
{Name: aws.String("MEMORY"), Type: aws.String("INTEGER"), IntegerValue: aws.Int64(int64(id.Capacity.MemoryMiB))},
},
VersionInfo: &ecs.VersionInfo{AgentVersion: aws.String(id.AgentVersion)},
// The built-in container-instance attributes AWS's own agent reports.
// The scheduler sizes the host's awsvpc ENI capacity from the instance
// type, so a registration without one is not ENI-gated.
Attributes: instanceAttributes(id),
}
if len(id.Capacity.GPUIDs) > 0 {
in.TotalResources = append(in.TotalResources, &ecs.Resource{
cmd/ecs-agent/heartbeat.go
+8 -3
View File
@@ -14,18 +14,23 @@ type heartbeater struct {
cp controlPlane
id identity
interval time.Duration
// runtimeReady reports whether a container runtime is resolved, so a beat
// advertises capacity only while there is something to run it with. nil
// means always ready.
runtimeReady func() bool
}
func newHeartbeater(cp controlPlane, id identity, interval time.Duration) *heartbeater {
func newHeartbeater(cp controlPlane, id identity, interval time.Duration, runtimeReady func() bool) *heartbeater {
if interval <= 0 {
interval = defaultHeartbeat
}
return &heartbeater{cp: cp, id: id, interval: interval}
return &heartbeater{cp: cp, id: id, interval: interval, runtimeReady: runtimeReady}
}
// beat re-registers the instance once, refreshing its LastSeen.
func (h *heartbeater) beat() error {
return h.cp.Register(h.id)
ready := h.runtimeReady == nil || h.runtimeReady()
return h.cp.Register(advertisedIdentity(h.id, ready))
}
// Run beats every interval until ctx is cancelled. A failed beat is logged and
cmd/ecs-agent/metadata.go
+11 -6
View File
@@ -13,9 +13,10 @@ import (
// instanceMetadata is the IMDS-derived identity of the host the agent runs on.
type instanceMetadata struct {
AccountID string
InstanceID string
AZ string
AccountID string
InstanceID string
AZ string
InstanceType string
}
// fetchInstanceMetadata reads the instance-id, AZ and account-id from IMDSv2
@@ -39,10 +40,14 @@ func fetchInstanceMetadata(client *http.Client, base string) (instanceMetadata,
return instanceMetadata{}, fmt.Errorf("identity document: %w", err)
}
// The instance type comes off the identity document already in hand rather
// than a further meta-data read; the scheduler sizes the host's awsvpc ENI
// capacity from it.
return instanceMetadata{
AccountID: doc.AccountID,
InstanceID: strings.TrimSpace(instanceID),
AZ: strings.TrimSpace(az),
AccountID: doc.AccountID,
InstanceID: strings.TrimSpace(instanceID),
AZ: strings.TrimSpace(az),
InstanceType: strings.TrimSpace(doc.InstanceType),
}, nil
}
cmd/ecs-agent/reconcile.go
+8 -3
View File
@@ -24,10 +24,11 @@ import (
// agent falls back to its pre-reconcile behaviour (poll + run).
func (a *Agent) reconcile(ctx context.Context) map[string]bool {
adopted := map[string]bool{}
if a.runner == nil {
runner := a.rt.get()
if runner == nil {
return adopted
}
containers, err := a.runner.List(ctx)
containers, err := runner.List(ctx)
if err != nil {
slog.Warn("ecs-agent: reconcile list failed", "err", err)
return adopted
@@ -96,9 +97,13 @@ func (a *Agent) reconcile(ctx context.Context) map[string]bool {
// is STOPPED, so the scheduler stops counting a task this instance is not
// running and its service can place a replacement.
func (a *Agent) reportStoppedTask(ctx context.Context, as *bus.Assign, dead []ctrruntime.Container) {
runner := a.rt.get()
statuses := make([]bus.ContainerStatus, 0, len(dead))
for _, c := range dead {
if err := a.runner.Remove(ctx, c.ID); err != nil {
if runner == nil {
break
}
if err := runner.Remove(ctx, c.ID); err != nil {
slog.Warn("ecs-agent: reconcile remove failed", "task", as.TaskID, "container", c.ID, "err", err)
}
statuses = append(statuses, bus.ContainerStatus{
cmd/ecs-agent/reconcile_test.go
+7 -7
View File
@@ -55,7 +55,7 @@ func TestReconcile_AdoptsRunningLabeledContainer(t *testing.T) {
adoptedContainer("t-001", "web", "default", "t-001", testTaskRole, "52:54:00:de:ad:01", true),
},
}
a := newAgent(config{}, testIdentity(), cp, rt, rt, nil)
a := newAgent(config{}, testIdentity(), cp, rt, nil)
a.cred = newCredEndpoint(nil, "us-east-1", "https://gw", "", "127.0.0.1", 0, nil)
adopted := a.reconcile(context.Background())
@@ -84,7 +84,7 @@ func TestReconcile_ReportsTaskWhoseContainerDied(t *testing.T) {
adoptedContainer("t-dead", "web", "default", "t-dead", testTaskRole, "52:54:00:de:ad:02", false),
},
}
a := newAgent(config{}, testIdentity(), cp, rt, rt, nil)
a := newAgent(config{}, testIdentity(), cp, rt, nil)
a.cred = newCredEndpoint(nil, "us-east-1", "https://gw", "", "127.0.0.1", 0, nil)
adopted := a.reconcile(context.Background())
@@ -116,7 +116,7 @@ func TestReconcile_PartiallyDeadTaskStaysRunning(t *testing.T) {
adoptedContainer("t-mixed", "sidecar", "default", "t-mixed", testTaskRole, "52:54:00:de:ad:03", false),
},
}
a := newAgent(config{}, testIdentity(), cp, rt, rt, nil)
a := newAgent(config{}, testIdentity(), cp, rt, nil)
a.cred = newCredEndpoint(nil, "us-east-1", "https://gw", "", "127.0.0.1", 0, nil)
a.reconcile(context.Background())
@@ -145,7 +145,7 @@ func TestPollAssignments_SeededTaskNotRerun(t *testing.T) {
nil,
}}
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, rt, nil)
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, nil)
ctx, cancel := context.WithCancel(context.Background())
go a.pollAssignments(ctx, map[string]bool{"t-001": true})
@@ -197,7 +197,7 @@ func TestReconcile_IgnoresOtherClusterContainers(t *testing.T) {
adoptedContainer("t-ok", "web", "default", "", "", "", true),
},
}
a := newAgent(config{}, testIdentity(), cp, rt, rt, nil)
a := newAgent(config{}, testIdentity(), cp, rt, nil)
adopted := a.reconcile(context.Background())
@@ -213,13 +213,13 @@ func TestReconcile_IgnoresOtherClusterContainers(t *testing.T) {
// A nil runner or a List error degrades to an empty set without panicking.
func TestReconcile_DegradesGracefully(t *testing.T) {
noRunner := newAgent(config{}, testIdentity(), &fakeCP{}, nil, nil, nil)
noRunner := newAgent(config{}, testIdentity(), &fakeCP{}, nil, nil)
if got := noRunner.reconcile(context.Background()); len(got) != 0 {
t.Errorf("nil runner: adopted = %v, want empty", got)
}
rt := &ctrruntime.FakePuller{ListErr: errors.New("boom")}
listErr := newAgent(config{}, testIdentity(), &fakeCP{}, rt, rt, nil)
listErr := newAgent(config{}, testIdentity(), &fakeCP{}, rt, nil)
if got := listErr.reconcile(context.Background()); len(got) != 0 {
t.Errorf("list error: adopted = %v, want empty", got)
}
cmd/ecs-agent/registration.go
+42 -3
View File
@@ -3,6 +3,9 @@ package main
import (
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/ecs"
handlers_ecs "github.com/mulgadc/spinifex/spinifex/handlers/ecs"
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
)
@@ -14,25 +17,61 @@ type identity struct {
ClusterName string
InstanceID string
AZ string
InstanceType string
Hostname string
Capacity bus.InstanceCapacity
AgentVersion string
}
// instanceAttributes builds the built-in container-instance attributes from the
// host's identity. Only attributes whose value was actually discovered are
// reported, so an empty one is absent rather than present and blank.
func instanceAttributes(id identity) []*ecs.Attribute {
var attrs []*ecs.Attribute
add := func(name, value string) {
if value == "" {
return
}
attrs = append(attrs, &ecs.Attribute{Name: aws.String(name), Value: aws.String(value)})
}
add(handlers_ecs.AttrInstanceType, id.InstanceType)
add(handlers_ecs.AttrAvailabilityZone, id.AZ)
return attrs
}
// advertisedIdentity is the identity to register with, given whether a
// container runtime is available. A host with no runtime can run nothing, so it
// advertises no capacity: the scheduler places nothing on it, and the control
// plane reads the zero as an involuntary drain. The identity is otherwise
// unchanged, so the instance stays visible rather than disappearing.
func advertisedIdentity(id identity, runtimeReady bool) identity {
if runtimeReady {
return id
}
id.Capacity = bus.InstanceCapacity{}
return id
}
// registrar registers the host as a container instance through the gateway.
type registrar struct {
cp controlPlane
id identity
// runtimeReady reports whether a container runtime is resolved. nil means
// always ready, which is what a test with no gate wants.
runtimeReady func() bool
}
func newRegistrar(cp controlPlane, id identity) *registrar {
return &registrar{cp: cp, id: id}
func newRegistrar(cp controlPlane, id identity, runtimeReady func() bool) *registrar {
return &registrar{cp: cp, id: id, runtimeReady: runtimeReady}
}
// ready reports the runtime state, defaulting to ready when no gate is wired.
func (r *registrar) ready() bool { return r.runtimeReady == nil || r.runtimeReady() }
// Register calls the gateway's RegisterContainerInstance. The scheduler records
// (or refreshes) the container instance on receipt.
func (r *registrar) Register() error {
if err := r.cp.Register(r.id); err != nil {
if err := r.cp.Register(advertisedIdentity(r.id, r.ready())); err != nil {
return fmt.Errorf("register instance %s: %w", r.id.InstanceID, err)
}
return nil
cmd/ecs-agent/registration_test.go
+76 -14
View File
@@ -3,6 +3,7 @@ package main
import (
"context"
"errors"
"slices"
"sync"
"testing"
"time"
@@ -17,14 +18,16 @@ import (
type fakeCP struct {
mu sync.Mutex
registers int
states []bus.TaskState
gpuReports []gpuReport
pollAcks [][]string
pollStopAcks [][]string
pollReplies [][]bus.Assign
stopReplies [][]bus.StopDirective
pollCalls int
registers int
lastRegistered identity
states []bus.TaskState
gpuReports []gpuReport
pollAcks [][]string
pollStopAcks [][]string
pollReplies [][]bus.Assign
stopReplies [][]bus.StopDirective
stopPending []bus.StopDirective
pollCalls int
registerErr bool
submitErr bool
@@ -39,16 +42,24 @@ type gpuReport struct {
var _ controlPlane = (*fakeCP)(nil)
func (f *fakeCP) Register(_ identity) error {
func (f *fakeCP) Register(id identity) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.registerErr {
return errors.New("register boom")
}
f.registers++
f.lastRegistered = id
return nil
}
// registered returns the identity of the most recent Register call.
func (f *fakeCP) registered() identity {
f.mu.Lock()
defer f.mu.Unlock()
return f.lastRegistered
}
func (f *fakeCP) SubmitTaskState(st bus.TaskState) error {
f.mu.Lock()
defer f.mu.Unlock()
@@ -72,6 +83,12 @@ func (f *fakeCP) PollAssignments(_, _ string, ackAssigns, ackStops []string) ([]
if f.pollCalls < len(f.stopReplies) {
stops = f.stopReplies[f.pollCalls]
}
// stopPending is the durable inbox the gateway keeps: an acked directive is
// deleted, and everything still in it is redelivered on every poll.
f.stopPending = slices.DeleteFunc(f.stopPending, func(sd bus.StopDirective) bool {
return slices.Contains(ackStops, sd.TaskID)
})
stops = slices.Concat(stops, f.stopPending)
f.pollCalls++
return out, stops, nil
}
@@ -142,7 +159,7 @@ func testIdentityWithGPUs(uuids ...string) identity {
func TestRegistrar_Register(t *testing.T) {
cp := &fakeCP{}
if err := newRegistrar(cp, testIdentity()).Register(); err != nil {
if err := newRegistrar(cp, testIdentity(), nil).Register(); err != nil {
t.Fatalf("Register: %v", err)
}
if cp.registerCount() != 1 {
@@ -152,14 +169,14 @@ func TestRegistrar_Register(t *testing.T) {
func TestRegistrar_RegisterError(t *testing.T) {
cp := &fakeCP{registerErr: true}
if err := newRegistrar(cp, testIdentity()).Register(); err == nil {
if err := newRegistrar(cp, testIdentity(), nil).Register(); err == nil {
t.Fatal("expected register error")
}
}
func TestHeartbeater_Beat(t *testing.T) {
cp := &fakeCP{}
if err := newHeartbeater(cp, testIdentity(), time.Second).beat(); err != nil {
if err := newHeartbeater(cp, testIdentity(), time.Second, nil).beat(); err != nil {
t.Fatalf("beat: %v", err)
}
if cp.registerCount() != 1 {
@@ -168,7 +185,7 @@ func TestHeartbeater_Beat(t *testing.T) {
}
func TestHeartbeater_DefaultInterval(t *testing.T) {
h := newHeartbeater(&fakeCP{}, testIdentity(), 0)
h := newHeartbeater(&fakeCP{}, testIdentity(), 0, nil)
if h.interval != defaultHeartbeat {
t.Errorf("interval = %v, want default", h.interval)
}
@@ -176,7 +193,7 @@ func TestHeartbeater_DefaultInterval(t *testing.T) {
func TestHeartbeater_RunStopsOnContext(t *testing.T) {
cp := &fakeCP{}
h := newHeartbeater(cp, testIdentity(), 5*time.Millisecond)
h := newHeartbeater(cp, testIdentity(), 5*time.Millisecond, nil)
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
@@ -193,3 +210,48 @@ func TestHeartbeater_RunStopsOnContext(t *testing.T) {
t.Error("expected at least one re-register before cancel")
}
}
// TestRegistrar_NoRuntimeAdvertisesNoCapacity is the placement half of the boot
// bug: an agent with no container runtime kept registering its full CPU and
// memory, so the scheduler placed tasks it could only fail to start. Capacity is
// advertised only while a runtime is there to use it.
func TestRegistrar_NoRuntimeAdvertisesNoCapacity(t *testing.T) {
cp := &fakeCP{}
ready := false
r := newRegistrar(cp, testIdentity(), func() bool { return ready })
if err := r.Register(); err != nil {
t.Fatalf("register: %v", err)
}
if got := cp.registered().Capacity; got.CPU != 0 || got.MemoryMiB != 0 {
t.Fatalf("advertised capacity with no runtime: cpu=%d memory=%d", got.CPU, got.MemoryMiB)
}
ready = true
if err := r.Register(); err != nil {
t.Fatalf("register: %v", err)
}
want := testIdentity().Capacity
if got := cp.registered().Capacity; got.CPU != want.CPU || got.MemoryMiB != want.MemoryMiB {
t.Fatalf("capacity not restored once the runtime resolved: cpu=%d memory=%d", got.CPU, got.MemoryMiB)
}
}
// TestHeartbeater_NoRuntimeAdvertisesNoCapacity is the same rule on the beat,
// which is how a runtime lost after boot reaches the control plane.
func TestHeartbeater_NoRuntimeAdvertisesNoCapacity(t *testing.T) {
cp := &fakeCP{}
h := newHeartbeater(cp, testIdentity(), time.Second, func() bool { return false })
if err := h.beat(); err != nil {
t.Fatalf("beat: %v", err)
}
if got := cp.registered().Capacity; got.CPU != 0 || got.MemoryMiB != 0 {
t.Fatalf("beat advertised capacity with no runtime: cpu=%d memory=%d", got.CPU, got.MemoryMiB)
}
// The instance still identifies itself, so it stays visible as drained rather
// than vanishing from the cluster.
if cp.registered().InstanceID != testIdentity().InstanceID {
t.Error("beat dropped the instance identity")
}
}
cmd/ecs-agent/runtime/containerd.go
+6
View File
@@ -3,6 +3,7 @@ package runtime
import (
"context"
"fmt"
"sync"
"time"
containerd "github.com/containerd/containerd/v2/client"
@@ -22,6 +23,11 @@ const servingProbeTimeout = 3 * time.Second
// containerdPuller drives a real containerd daemon over its unix socket.
type containerdPuller struct {
client *containerd.Client
// stdinDone holds one channel per interactive container, closed when that
// container is stopped or removed so its stdin reader returns EOF.
mu sync.Mutex
stdinDone map[string]chan struct{}
}
var _ ImagePuller = (*containerdPuller)(nil)
cmd/ecs-agent/runtime/containerd_run.go
+155 -1
View File
@@ -3,9 +3,14 @@ package runtime
import (
"context"
"fmt"
"io"
"sort"
"strings"
"syscall"
"time"
containerd "github.com/containerd/containerd/v2/client"
"github.com/containerd/containerd/v2/core/containers"
"github.com/containerd/containerd/v2/pkg/cdi"
"github.com/containerd/containerd/v2/pkg/cio"
"github.com/containerd/containerd/v2/pkg/namespaces"
@@ -41,6 +46,106 @@ func cdiSpecOpts(gpuIDs []string) []oci.SpecOpts {
return []oci.SpecOpts{cdi.WithCDIDevices(devices...)}
}
// ociCapName renders an ECS Linux capability (Docker's bare name, e.g.
// "SYS_ADMIN") as the OCI runtime-spec name ("CAP_SYS_ADMIN") capabilities.Add/
// Drop expect.
func ociCapName(name string) string {
if strings.HasPrefix(name, "CAP_") {
return name
}
return "CAP_" + name
}
// ociCapNames maps a list of ECS capability names to their OCI form.
func ociCapNames(names []string) []string {
if len(names) == 0 {
return nil
}
out := make([]string, len(names))
for i, n := range names {
out[i] = ociCapName(n)
}
return out
}
// withSysctls sets Linux.Sysctl on the OCI spec from systemControls namespace/
// value pairs. containerd has no built-in SpecOpts for sysctls.
func withSysctls(ctls []SystemControl) oci.SpecOpts {
return func(_ context.Context, _ oci.Client, _ *containers.Container, s *oci.Spec) error {
if len(ctls) == 0 {
return nil
}
if s.Linux == nil {
s.Linux = &specs.Linux{}
}
if s.Linux.Sysctl == nil {
s.Linux.Sysctl = map[string]string{}
}
for _, c := range ctls {
s.Linux.Sysctl[c.Namespace] = c.Value
}
return nil
}
}
// heldStdin backs an interactive container's stdin: reads block instead of
// hitting an immediate EOF, which is the "stdin held open" behaviour
// interactive=true asks for, until done closes and the read reports EOF.
type heldStdin struct{ done <-chan struct{} }
func (h heldStdin) Read([]byte) (int, error) {
<-h.done
return 0, io.EOF
}
// ioStreamOpts describes the task IO for a container. A terminal container needs
// a terminal FIFO set: oci.WithTTY alone puts Terminal in the OCI spec while the
// shim creates no console socket, and runc refuses that pair outright. A
// terminal carries no separate stderr, so that stream stays nil and the shim
// merges it into the console.
func ioStreamOpts(stdin io.Reader, terminal bool) []cio.Opt {
if terminal {
return []cio.Opt{cio.WithStreams(stdin, io.Discard, nil), cio.WithTerminal}
}
return []cio.Opt{cio.WithStreams(stdin, io.Discard, io.Discard)}
}
// containerIOCreator returns cio.NullIO for a container needing neither stdin
// nor a terminal, else a creator carrying whichever it asked for. An interactive
// container's stdin is held open until releaseStdin; containerd copies stdin on
// a goroutine that exits only once the read returns, so a reader that never
// returned would park it for the life of the agent. A terminal container that is
// not interactive gets no stdin stream at all, which is what a nil reader means.
func (p *containerdPuller) containerIOCreator(containerID string, interactive, terminal bool) cio.Creator {
if !interactive && !terminal {
return cio.NullIO
}
var stdin io.Reader
if interactive {
done := make(chan struct{})
p.mu.Lock()
if p.stdinDone == nil {
p.stdinDone = make(map[string]chan struct{})
}
p.stdinDone[containerID] = done
p.mu.Unlock()
stdin = heldStdin{done: done}
}
return cio.NewCreator(ioStreamOpts(stdin, terminal)...)
}
// releaseStdin closes a held stdin so containerd's copier goroutine exits. Safe
// for a container that never held one, and safe to call more than once.
func (p *containerdPuller) releaseStdin(containerID string) {
p.mu.Lock()
done, ok := p.stdinDone[containerID]
delete(p.stdinDone, containerID)
p.mu.Unlock()
if ok {
close(done)
}
}
var _ Runner = (*containerdPuller)(nil)
// Run creates a container from an already-pulled image and starts its task. The
@@ -81,6 +186,26 @@ func (p *containerdPuller) Run(ctx context.Context, id string, spec RunSpec) (st
specOpts = append(specOpts, oci.WithEnv(envSlice(spec.Env)))
}
specOpts = append(specOpts, cdiSpecOpts(spec.GPUIDs)...)
if spec.User != "" {
specOpts = append(specOpts, oci.WithUser(spec.User))
}
if spec.ReadonlyRootFilesystem != nil && *spec.ReadonlyRootFilesystem {
specOpts = append(specOpts, oci.WithRootFSReadonly())
}
if spec.Privileged != nil && *spec.Privileged {
specOpts = append(specOpts, oci.WithPrivileged)
}
terminal := spec.PseudoTerminal != nil && *spec.PseudoTerminal
if terminal {
specOpts = append(specOpts, oci.WithTTY)
}
specOpts = append(specOpts, withSysctls(spec.SystemControls))
if len(spec.CapAdd) > 0 {
specOpts = append(specOpts, oci.WithAddedCapabilities(ociCapNames(spec.CapAdd)))
}
if len(spec.CapDrop) > 0 {
specOpts = append(specOpts, oci.WithDroppedCapabilities(ociCapNames(spec.CapDrop)))
}
container, err := p.client.NewContainer(ctx, id,
containerd.WithNewSnapshot(id+"-snapshot", image),
@@ -91,12 +216,15 @@ func (p *containerdPuller) Run(ctx context.Context, id string, spec RunSpec) (st
return "", fmt.Errorf("create container %s: %w", id, err)
}
task, err := container.NewTask(ctx, cio.NullIO)
interactive := spec.Interactive != nil && *spec.Interactive
task, err := container.NewTask(ctx, p.containerIOCreator(id, interactive, terminal))
if err != nil {
p.releaseStdin(id)
_ = container.Delete(ctx, containerd.WithSnapshotCleanup)
return "", fmt.Errorf("create task %s: %w", id, err)
}
if err := task.Start(ctx); err != nil {
p.releaseStdin(id)
_, _ = task.Delete(ctx)
_ = container.Delete(ctx, containerd.WithSnapshotCleanup)
return "", fmt.Errorf("start task %s: %w", id, err)
@@ -148,9 +276,35 @@ func (p *containerdPuller) List(ctx context.Context) ([]Container, error) {
return out, nil
}
// Stop enforces a container's stopTimeout: SIGTERM the task, wait up to
// timeout for it to exit on its own, then fall through to Remove's forced
// kill+delete regardless of whether it exited in time. A container with no
// live task (already gone) or that fails to accept the signal goes straight to
// Remove, matching Remove's own already-gone tolerance.
func (p *containerdPuller) Stop(ctx context.Context, containerID string, timeout time.Duration) error {
ctx = namespaces.WithNamespace(ctx, ecsNamespace)
container, err := p.client.LoadContainer(ctx, containerID)
if err != nil {
return nil //nolint:nilerr // load failure means the container is already gone
}
task, terr := container.Task(ctx, nil)
if terr != nil {
return p.Remove(ctx, containerID)
}
statusC, werr := task.Wait(ctx)
if werr == nil && task.Kill(ctx, syscall.SIGTERM) == nil {
select {
case <-statusC:
case <-time.After(timeout):
}
}
return p.Remove(ctx, containerID)
}
// Remove kills and deletes the container's task, then the container + snapshot.
func (p *containerdPuller) Remove(ctx context.Context, containerID string) error {
ctx = namespaces.WithNamespace(ctx, ecsNamespace)
p.releaseStdin(containerID)
container, err := p.client.LoadContainer(ctx, containerID)
if err != nil {
return nil //nolint:nilerr // load failure means the container is already gone
cmd/ecs-agent/runtime/containerd_run_test.go
+170
View File
@@ -1,8 +1,15 @@
package runtime
import (
"context"
"errors"
"io"
"reflect"
"testing"
"time"
"github.com/containerd/containerd/v2/pkg/cio"
specs "github.com/opencontainers/runtime-spec/specs-go"
)
// cdiDeviceNames maps a container's pinned GPU UUIDs to CDI device names
@@ -45,3 +52,166 @@ func TestCDISpecOpts_Wiring(t *testing.T) {
t.Error("GPU container: spec opt is nil")
}
}
// TestOCICapNames verifies ECS's bare capability names (Docker convention,
// e.g. "SYS_PTRACE") are prefixed to the OCI runtime-spec form the process
// capability sets expect, and a name already carrying the prefix passes
// through unchanged.
func TestOCICapNames(t *testing.T) {
cases := []struct {
name string
in []string
want []string
}{
{name: "empty", in: nil, want: nil},
{name: "bare", in: []string{"SYS_PTRACE"}, want: []string{"CAP_SYS_PTRACE"}},
{name: "already prefixed", in: []string{"CAP_NET_ADMIN"}, want: []string{"CAP_NET_ADMIN"}},
{name: "mixed", in: []string{"SYS_PTRACE", "CAP_NET_ADMIN"}, want: []string{"CAP_SYS_PTRACE", "CAP_NET_ADMIN"}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := ociCapNames(tc.in)
if !reflect.DeepEqual(got, tc.want) {
t.Errorf("ociCapNames(%v) = %v, want %v", tc.in, got, tc.want)
}
})
}
}
// TestWithSysctls verifies systemControls namespace/value pairs land on
// Linux.Sysctl, and an empty list leaves the spec's Linux section untouched
// rather than allocating an empty map (the "requests nothing" case).
func TestWithSysctls(t *testing.T) {
s := &specs.Spec{}
if err := withSysctls(nil)(context.Background(), nil, nil, s); err != nil {
t.Fatalf("empty ctls: %v", err)
}
if s.Linux != nil {
t.Errorf("empty ctls: want no Linux section, got %+v", s.Linux)
}
ctls := []SystemControl{{Namespace: "net.core.somaxconn", Value: "1024"}}
if err := withSysctls(ctls)(context.Background(), nil, nil, s); err != nil {
t.Fatalf("with ctls: %v", err)
}
if s.Linux == nil || s.Linux.Sysctl["net.core.somaxconn"] != "1024" {
t.Errorf("want sysctl net.core.somaxconn=1024, got %+v", s.Linux)
}
}
// An interactive container's stdin read blocks until the container is released,
// then reports EOF so containerd's copier goroutine exits instead of parking
// for the life of the agent.
func TestHeldStdin_ReadsEOFOnceReleased(t *testing.T) {
p := &containerdPuller{}
p.containerIOCreator("c1", true, false)
p.mu.Lock()
done := p.stdinDone["c1"]
p.mu.Unlock()
if done == nil {
t.Fatal("interactive container registered no stdin channel")
}
read := make(chan error, 1)
go func() {
_, err := heldStdin{done: done}.Read(make([]byte, 1))
read <- err
}()
select {
case err := <-read:
t.Fatalf("read returned before release: %v", err)
case <-time.After(50 * time.Millisecond):
}
p.releaseStdin("c1")
select {
case err := <-read:
if !errors.Is(err, io.EOF) {
t.Fatalf("want io.EOF after release, got %v", err)
}
case <-time.After(time.Second):
t.Fatal("read did not return after release")
}
}
// A non-interactive container uses cio.NullIO and registers nothing to release.
func TestContainerIOCreator_NonInteractiveRegistersNothing(t *testing.T) {
p := &containerdPuller{}
p.containerIOCreator("c1", false, false)
p.mu.Lock()
defer p.mu.Unlock()
if len(p.stdinDone) != 0 {
t.Fatalf("want no registered stdin channels, got %d", len(p.stdinDone))
}
}
// applyIOOpts resolves the cio options onto a Streams the way cio.NewCreator
// does, which is the only way to observe what the creator was built with.
func applyIOOpts(opts []cio.Opt) cio.Streams {
var s cio.Streams
for _, o := range opts {
o(&s)
}
return s
}
// A terminal container needs the terminal flag set on its FIFO set, else the
// shim creates no console socket and runc refuses to start the container at all.
// The terminal carries no separate stderr stream.
func TestIOStreamOpts_TerminalRequestsAConsole(t *testing.T) {
s := applyIOOpts(ioStreamOpts(nil, true))
if !s.Terminal {
t.Error("terminal container: want Terminal set")
}
if s.Stderr != nil {
t.Errorf("terminal container: want no stderr stream, got %#v", s.Stderr)
}
if s.Stdout == nil {
t.Error("terminal container: want a stdout stream")
}
}
// A container with stdin but no terminal keeps both output streams and asks for
// no console.
func TestIOStreamOpts_NonTerminalKeepsStderr(t *testing.T) {
s := applyIOOpts(ioStreamOpts(heldStdin{}, false))
if s.Terminal {
t.Error("non-terminal container: want Terminal unset")
}
if s.Stderr == nil {
t.Error("non-terminal container: want a stderr stream")
}
if s.Stdin == nil {
t.Error("non-terminal interactive container: want a stdin stream")
}
}
// A terminal container that is not interactive still gets a creator rather than
// cio.NullIO, and registers no stdin to release.
func TestContainerIOCreator_TerminalWithoutInteractive(t *testing.T) {
p := &containerdPuller{}
if c := p.containerIOCreator("c1", false, true); c == nil {
t.Fatal("terminal container: want a creator")
}
p.mu.Lock()
defer p.mu.Unlock()
if len(p.stdinDone) != 0 {
t.Fatalf("want no registered stdin channels, got %d", len(p.stdinDone))
}
}
// Remove calls releaseStdin unconditionally and Stop falls through to Remove,
// so an unknown container and a second release must both be no-ops.
func TestReleaseStdin_UnknownAndRepeatedAreSafe(t *testing.T) {
p := &containerdPuller{}
p.releaseStdin("never-registered")
p.containerIOCreator("c1", true, false)
p.releaseStdin("c1")
p.releaseStdin("c1")
}
cmd/ecs-agent/runtime/runtime.go
+33 -4
View File
@@ -4,7 +4,10 @@
// scheduler-only build run against a fake.
package runtime
import "context"
import (
"context"
"time"
)
// PullSpec describes one image to pull.
type PullSpec struct {
@@ -35,6 +38,12 @@ type ImagePuller interface {
Close() error
}
// SystemControl is a sysctl namespace/value pair applied to the OCI spec.
type SystemControl struct {
Namespace string
Value string
}
// RunSpec describes a single container the agent must create and start. v1 runs
// the container in the host network namespace (bridge/CNI task networking lands
// in a later sprint); Labels carry the mulga.ecs.* task identity for the reboot
@@ -54,6 +63,22 @@ type RunSpec struct {
// per GPU requested. Empty for non-GPU containers or if pinning fell short.
// The runner injects these as CDI devices into the OCI spec.
GPUIDs []string
// User is the OCI process user (oci.WithUser). Empty runs as the image
// default.
User string
// ReadonlyRootFilesystem, Privileged, PseudoTerminal and Interactive are
// pointers so the runner can tell "not requested" (nil) from an explicit
// false, which still needs echoing but applies nothing.
ReadonlyRootFilesystem *bool
Privileged *bool
PseudoTerminal *bool
Interactive *bool
// SystemControls are sysctls set on the OCI spec's Linux.Sysctl map.
SystemControls []SystemControl
// CapAdd / CapDrop are Linux capabilities added/dropped from the OCI
// spec's process capability sets (linuxParameters.capabilities).
CapAdd []string
CapDrop []string
}
// RunStatus is a finished container's outcome.
@@ -71,12 +96,16 @@ type Container struct {
}
// Runner creates and starts containers from already-pulled images. id is a
// caller-unique container ID; Wait blocks until the container exits; Remove
// tears down the container + its task; List enumerates known containers so the
// reboot reconciler can re-adopt the ones still running.
// caller-unique container ID; Wait blocks until the container exits; Stop
// requests a graceful exit (SIGTERM, then a forced kill+remove once timeout
// elapses) — the agent's enforcement of a container's stopTimeout; Remove
// tears down the container + its task immediately, with no grace period; List
// enumerates known containers so the reboot reconciler can re-adopt the ones
// still running.
type Runner interface {
Run(ctx context.Context, id string, spec RunSpec) (containerID string, err error)
Wait(ctx context.Context, containerID string) (RunStatus, error)
Stop(ctx context.Context, containerID string, timeout time.Duration) error
Remove(ctx context.Context, containerID string) error
List(ctx context.Context) ([]Container, error)
}
cmd/ecs-agent/runtime/runtime_fake.go
+46 -3
View File
@@ -4,6 +4,7 @@ import (
"context"
"fmt"
"sync"
"time"
)
// FakePuller is an in-memory ImagePuller for tests and the scheduler-only build.
@@ -26,11 +27,21 @@ type FakePuller struct {
Removed []string
OnRun func(string, RunSpec) (string, error)
// List bookkeeping: Containers is replayed by List; ListErr forces a failure;
// Listed records that List was called.
// Stop bookkeeping: StopTimeouts pairs the container ID with the timeout it
// was stopped with, in call order, so a test can assert both the target and
// the enforced stopTimeout without a fake containerd.
Stopped []string
StopTimeouts []time.Duration
StopErr error
// List bookkeeping: Containers is replayed by List; ListErr forces a failure,
// limited to the first ListErrFor calls when that is non-zero, so a test can
// model a runtime that recovers. Listed and ListCalls record the calls.
Containers []Container
ListErr error
ListErrFor int
Listed bool
ListCalls int
}
var (
@@ -98,6 +109,15 @@ func (f *FakePuller) Wait(_ context.Context, containerID string) (RunStatus, err
return RunStatus{ExitCode: f.WaitCode}, f.WaitErr
}
// Stop records the container ID and the timeout it was stopped with.
func (f *FakePuller) Stop(_ context.Context, containerID string, timeout time.Duration) error {
f.mu.Lock()
defer f.mu.Unlock()
f.Stopped = append(f.Stopped, containerID)
f.StopTimeouts = append(f.StopTimeouts, timeout)
return f.StopErr
}
// Remove records the container ID.
func (f *FakePuller) Remove(_ context.Context, containerID string) error {
f.mu.Lock()
@@ -113,13 +133,36 @@ func (f *FakePuller) Waits() []string {
return append([]string(nil), f.Waited...)
}
// RunCalls returns a copy of the specs passed to Run, for assertions made while
// the agent's goroutines may still be live.
func (f *FakePuller) RunCalls() []RunSpec {
f.mu.Lock()
defer f.mu.Unlock()
return append([]RunSpec(nil), f.Runs...)
}
// Reaped returns a copy of the container IDs passed to Stop.
func (f *FakePuller) Reaped() []string {
f.mu.Lock()
defer f.mu.Unlock()
return append([]string(nil), f.Stopped...)
}
// List replays the programmed container set (or error).
func (f *FakePuller) List(_ context.Context) ([]Container, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.Listed = true
if f.ListErr != nil {
f.ListCalls++
if f.ListErr != nil && (f.ListErrFor == 0 || f.ListCalls <= f.ListErrFor) {
return nil, f.ListErr
}
return f.Containers, nil
}
// Lists reports how many times List was called.
func (f *FakePuller) Lists() int {
f.mu.Lock()
defer f.mu.Unlock()
return f.ListCalls
}
cmd/ecs-agent/runtimegate.go
+116
View File
@@ -0,0 +1,116 @@
package main
import (
"context"
"log/slog"
"sync"
"time"
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
)
// runtimeProbeTimeout bounds the liveness probe against a runtime the gate is
// already holding, so a wedged socket cannot stall the watch loop.
const runtimeProbeTimeout = 5 * time.Second
// defaultRuntimeRetry is how often the gate dials a runtime it does not have,
// and probes one it does. The node starts containerd and the agent back to
// back, so the first few dials are expected to fail.
const defaultRuntimeRetry = 10 * time.Second
// runtimeGate holds the container runtime the agent has resolved, or nothing
// when the socket has not answered. The runtime is resolved by a loop rather
// than once at boot: the node's service manager reports the container daemon
// started when the process spawned, not when its socket began accepting, so a
// dial that fails in the first second of boot is a transient condition.
//
// Every read of the runtime goes through the gate, so a runtime that arrives
// after boot is picked up on the next task without restarting the agent.
type runtimeGate struct {
mu sync.RWMutex
rt ctrruntime.Runtime
}
// newRuntimeGate returns a gate holding rt, which may be nil.
func newRuntimeGate(rt ctrruntime.Runtime) *runtimeGate {
return &runtimeGate{rt: rt}
}
// get returns the current runtime, or nil when none is resolved.
func (g *runtimeGate) get() ctrruntime.Runtime {
g.mu.RLock()
defer g.mu.RUnlock()
return g.rt
}
// available reports whether a runtime is resolved. It is what registration
// consults to decide whether the host has any capacity to advertise.
func (g *runtimeGate) available() bool { return g.get() != nil }
// set replaces the held runtime, closing the one it displaces.
func (g *runtimeGate) set(rt ctrruntime.Runtime) {
g.mu.Lock()
old := g.rt
g.rt = rt
g.mu.Unlock()
if old != nil && old != rt {
_ = old.Close()
}
}
// clear drops and closes the held runtime, returning whether it held one.
func (g *runtimeGate) clear() bool {
g.mu.Lock()
old := g.rt
g.rt = nil
g.mu.Unlock()
if old == nil {
return false
}
_ = old.Close()
return true
}
// watch keeps the gate current until ctx is cancelled: it dials while the gate
// is empty, and probes while it is full, clearing the gate when the probe fails
// so the next tick re-dials. A containerd stopped and restarted underneath a
// running agent is therefore picked up the same way a slow boot is.
func (g *runtimeGate) watch(ctx context.Context, dial func() (ctrruntime.Runtime, error), interval time.Duration) {
if interval <= 0 {
interval = defaultRuntimeRetry
}
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
g.step(ctx, dial)
}
}
}
// step performs one pass of the watch loop: dial when empty, probe when full.
func (g *runtimeGate) step(ctx context.Context, dial func() (ctrruntime.Runtime, error)) {
if rt := g.get(); rt != nil {
probeCtx, cancel := context.WithTimeout(ctx, runtimeProbeTimeout)
// A container list is what reconcile already asks the runtime at boot and
// the cheapest thing it answers, so it doubles as the liveness probe.
_, err := rt.List(probeCtx)
cancel()
if err != nil && ctx.Err() == nil {
slog.Warn("ecs-agent: container runtime stopped answering, will re-dial", "err", err)
g.clear()
}
return
}
rt, err := dial()
if err != nil {
slog.Debug("ecs-agent: container runtime still unavailable", "err", err)
return
}
slog.Info("ecs-agent: container runtime available")
g.set(rt)
}
cmd/ecs-agent/runtimegate_test.go
+124
View File
@@ -0,0 +1,124 @@
//test:in-package — package main cannot be imported by an external test package.
package main
import (
"context"
"errors"
"testing"
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
)
// TestRuntimeGate_DialsUntilTheSocketAnswers is the boot case the agent used to
// lose: the node starts containerd and the agent together, so the first dials
// fail. Resolving once left the agent runtime-less for the life of the process.
func TestRuntimeGate_DialsUntilTheSocketAnswers(t *testing.T) {
g := newRuntimeGate(nil)
if g.available() {
t.Fatal("empty gate reported a runtime")
}
rt := &ctrruntime.FakePuller{}
dials := 0
dial := func() (ctrruntime.Runtime, error) {
dials++
if dials < 3 {
return nil, errors.New("dial containerd: no such file or directory")
}
return rt, nil
}
for range 2 {
g.step(t.Context(), dial)
if g.available() {
t.Fatalf("gate resolved a runtime after %d failed dials", dials)
}
}
g.step(t.Context(), dial)
if g.get() != rt {
t.Fatalf("gate did not pick up the runtime once the socket answered (dials=%d)", dials)
}
}
// TestRuntimeGate_DropsARuntimeThatStopsAnswering covers a containerd restarted
// underneath a live agent: the held runtime stops answering, so the gate clears
// it and the next pass re-dials rather than handing out a dead client forever.
func TestRuntimeGate_DropsARuntimeThatStopsAnswering(t *testing.T) {
dead := &ctrruntime.FakePuller{ListErr: errors.New("connection refused")}
fresh := &ctrruntime.FakePuller{}
g := newRuntimeGate(dead)
dial := func() (ctrruntime.Runtime, error) { return fresh, nil }
g.step(t.Context(), dial)
if g.available() {
t.Fatal("gate kept a runtime whose probe failed")
}
if !dead.Closed {
t.Error("displaced runtime was not closed")
}
g.step(t.Context(), dial)
if g.get() != fresh {
t.Fatal("gate did not re-dial after clearing")
}
}
// TestRuntimeGate_KeepsAHealthyRuntime guards the other direction: a probe that
// succeeds must not churn the runtime, or every pass would drop a working client.
func TestRuntimeGate_KeepsAHealthyRuntime(t *testing.T) {
rt := &ctrruntime.FakePuller{}
g := newRuntimeGate(rt)
dial := func() (ctrruntime.Runtime, error) {
t.Error("dialled while holding a healthy runtime")
return nil, errors.New("unexpected dial")
}
g.step(t.Context(), dial)
if g.get() != rt {
t.Fatal("gate dropped a runtime whose probe succeeded")
}
if rt.Closed {
t.Error("healthy runtime was closed")
}
}
// TestRuntimeGate_ClearIsIdempotent keeps Stop safe to call on an agent that
// never resolved a runtime.
func TestRuntimeGate_ClearIsIdempotent(t *testing.T) {
g := newRuntimeGate(nil)
if g.clear() {
t.Error("clear reported dropping a runtime the gate never held")
}
rt := &ctrruntime.FakePuller{}
g.set(rt)
if !g.clear() {
t.Error("clear did not report dropping the held runtime")
}
if !rt.Closed {
t.Error("cleared runtime was not closed")
}
if g.clear() {
t.Error("second clear reported dropping a runtime")
}
}
// TestRuntimeGate_WatchStopsWithContext keeps the watch loop from outliving the
// agent it belongs to.
func TestRuntimeGate_WatchStopsWithContext(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
g := newRuntimeGate(nil)
done := make(chan struct{})
go func() {
g.watch(ctx, func() (ctrruntime.Runtime, error) { return nil, errors.New("no socket") }, 0)
close(done)
}()
cancel()
<-done
}
cmd/ecs-agent/taskrun.go
+159 -30
View File
@@ -5,6 +5,8 @@ import (
"fmt"
"log/slog"
"maps"
"strconv"
"sync"
"time"
ctrruntime "github.com/mulgadc/spinifex/cmd/ecs-agent/runtime"
@@ -12,6 +14,10 @@ import (
"github.com/mulgadc/spinifex/spinifex/handlers/ecs/bus"
)
// defaultStopTimeout is applied when a container carries no explicit
// stopTimeout, matching the EC2 launch-type default ECS itself uses.
const defaultStopTimeout = 30 * time.Second
// pollAssignments drains the instance's assignment inbox on a ticker until ctx is
// cancelled. Each assign is dispatched to runTask exactly once; the taskIDs seen
// on a poll are acked on the next poll so the gateway can drop them — a crash
@@ -23,7 +29,7 @@ func (a *Agent) pollAssignments(ctx context.Context, dispatched map[string]bool)
ticker := time.NewTicker(a.cfg.PollInterval)
defer ticker.Stop()
stopping := map[string]bool{}
reaps := newStopLedger()
var ackAssigns, ackStops []string
for {
select {
@@ -40,17 +46,18 @@ func (a *Agent) pollAssignments(ctx context.Context, dispatched map[string]bool)
ackStops = ackStops[:0]
for i := range stops {
sd := stops[i]
if !stopping[sd.TaskID] {
stopping[sd.TaskID] = true
switch {
case reaps.done(sd.TaskID):
ackStops = append(ackStops, sd.TaskID)
case reaps.begin(sd.TaskID):
slog.Info("ecs-agent: task stop requested", "task", sd.TaskID, "reason", sd.Reason)
go a.stopTask(ctx, sd)
go func() { reaps.end(sd.TaskID, a.stopTask(ctx, sd)) }()
}
ackStops = append(ackStops, sd.TaskID)
}
ackAssigns = ackAssigns[:0]
for i := range assigns {
as := assigns[i]
if !dispatched[as.TaskID] && !stopping[as.TaskID] {
if !dispatched[as.TaskID] && !reaps.seen(as.TaskID) {
dispatched[as.TaskID] = true
slog.Info("ecs-agent: task assigned", "task", as.TaskID, "containers", len(as.Containers))
go a.runTask(ctx, &as)
@@ -61,27 +68,83 @@ func (a *Agent) pollAssignments(ctx context.Context, dispatched map[string]bool)
}
}
// stopTask reaps a task's containers on a control-plane stop directive: it lists
// the runtime's containers, removes the ones labeled with this task (kill +
// delete, releasing host ports/netns), then reports the task STOPPED with the
// directive's reason. Idempotent — a task with no live containers still reports
// STOPPED so the scheduler releases its capacity.
func (a *Agent) stopTask(ctx context.Context, sd bus.StopDirective) {
if a.runner == nil {
return
// stopLedger tracks reaps in flight and reaps that reported STOPPED. The two
// are separate because only the second may be acked: acking the first deletes a
// directive that was never performed, and nothing re-posts it.
type stopLedger struct {
mu sync.Mutex
inflight map[string]bool
stopped map[string]bool
}
func newStopLedger() *stopLedger {
return &stopLedger{inflight: map[string]bool{}, stopped: map[string]bool{}}
}
// begin claims a task for a reap, reporting false when one is already running.
func (l *stopLedger) begin(taskID string) bool {
l.mu.Lock()
defer l.mu.Unlock()
if l.inflight[taskID] {
return false
}
containers, err := a.runner.List(ctx)
l.inflight[taskID] = true
return true
}
// end releases the claim, recording the task as stopped only if the reap got
// that far. A failed reap leaves nothing behind, so the next poll redelivers
// the directive and dispatches it again.
func (l *stopLedger) end(taskID string, reported bool) {
l.mu.Lock()
defer l.mu.Unlock()
delete(l.inflight, taskID)
if reported {
l.stopped[taskID] = true
}
}
// done reports whether the task's reap reported STOPPED, which is the only
// condition under which its directive may be acked.
func (l *stopLedger) done(taskID string) bool {
l.mu.Lock()
defer l.mu.Unlock()
return l.stopped[taskID]
}
// seen reports whether a stop for the task is in flight or complete, so an
// assignment arriving alongside its own stop is never started.
func (l *stopLedger) seen(taskID string) bool {
l.mu.Lock()
defer l.mu.Unlock()
return l.inflight[taskID] || l.stopped[taskID]
}
// stopTask reaps a task's containers on a control-plane stop directive: it
// lists the runtime's containers, gracefully stops (SIGTERM, wait up to the
// container's stopTimeout, then force kill + delete) the ones labeled with
// this task, then reports the task STOPPED with the directive's reason.
// Idempotent — a task with no live containers still reports STOPPED so the
// scheduler releases its capacity. Reports whether it got as far as saying
// STOPPED: a caller may only ack the directive when it did.
func (a *Agent) stopTask(ctx context.Context, sd bus.StopDirective) bool {
runner := a.rt.get()
if runner == nil {
slog.Warn("ecs-agent: stop deferred, no container runtime", "task", sd.TaskID)
return false
}
containers, err := runner.List(ctx)
if err != nil {
slog.Warn("ecs-agent: stop list failed", "task", sd.TaskID, "err", err)
return
return false
}
statuses := make([]bus.ContainerStatus, 0)
for _, c := range containers {
if c.Labels[labelTaskID] != sd.TaskID {
continue
}
if rerr := a.runner.Remove(ctx, c.ID); rerr != nil {
slog.Warn("ecs-agent: stop remove failed", "task", sd.TaskID, "container", c.ID, "err", rerr)
if rerr := runner.Stop(ctx, c.ID, containerStopTimeout(c.Labels)); rerr != nil {
slog.Warn("ecs-agent: stop failed", "task", sd.TaskID, "container", c.ID, "err", rerr)
}
statuses = append(statuses, bus.ContainerStatus{
Name: c.Labels[labelContainerName], Status: bus.TaskStatusStopped, ContainerID: c.ID,
@@ -93,13 +156,15 @@ func (a *Agent) stopTask(ctx context.Context, sd bus.StopDirective) {
}
a.reportTaskState(&bus.Assign{TaskID: sd.TaskID}, bus.TaskStatusStopped, reason, statuses)
slog.Info("ecs-agent: task reaped", "task", sd.TaskID, "containers", len(statuses))
return true
}
// runTask pulls each container image, starts the containers, and reports task
// state on the bus. A missing runtime or any per-container failure reports the
// task STOPPED with a reason; success reports RUNNING and waits for exit.
func (a *Agent) runTask(ctx context.Context, as *bus.Assign) {
if a.puller == nil || a.runner == nil {
rt := a.rt.get()
if rt == nil {
a.reportTaskState(as, bus.TaskStatusStopped, "containerd unavailable on agent", nil)
return
}
@@ -117,7 +182,12 @@ func (a *Agent) runTask(ctx context.Context, as *bus.Assign) {
statuses := make([]bus.ContainerStatus, 0, len(as.Containers))
for _, c := range as.Containers {
if _, err := a.puller.Pull(ctx, ctrruntime.PullSpec{Ref: c.Image}, resolver); err != nil {
// startTimeout bounds pull-through-start as a unit: real ECS measures a
// container's time-to-RUNNING from the same starting point.
runCtx, cancel := containerStartCtx(ctx, c.StartTimeout)
if _, err := rt.Pull(runCtx, ctrruntime.PullSpec{Ref: c.Image}, resolver); err != nil {
cancel()
slog.Error("ecs-agent: pull failed", "task", as.TaskID, "image", c.Image, "err", err)
a.teardownTaskNetns(as)
a.reportTaskState(as, bus.TaskStatusStopped, "image pull failed: "+err.Error(), statuses)
@@ -126,16 +196,29 @@ func (a *Agent) runTask(ctx context.Context, as *bus.Assign) {
cid := containerID(as.TaskID, c.Name)
gpuIDs := a.pinContainerGPUs(as.TaskID, c.Name, c.GPU)
spec := ctrruntime.RunSpec{
Image: c.Image,
Command: c.Command,
Env: withCredEnv(c.Environment, credID),
Labels: taskLabels(as, c.Name),
NetnsPath: netnsPath,
GPU: c.GPU,
GPUIDs: gpuIDs,
labels := taskLabels(as, c.Name)
if c.StopTimeout != nil {
labels[labelStopTimeout] = strconv.FormatInt(*c.StopTimeout, 10)
}
id, err := a.runner.Run(ctx, cid, spec)
spec := ctrruntime.RunSpec{
Image: c.Image,
Command: c.Command,
Env: withCredEnv(c.Environment, credID),
Labels: labels,
NetnsPath: netnsPath,
GPU: c.GPU,
GPUIDs: gpuIDs,
User: c.User,
ReadonlyRootFilesystem: c.ReadonlyRootFilesystem,
Privileged: c.Privileged,
PseudoTerminal: c.PseudoTerminal,
Interactive: c.Interactive,
SystemControls: toRuntimeSystemControls(c.SystemControls),
CapAdd: c.CapAdd,
CapDrop: c.CapDrop,
}
id, err := rt.Run(runCtx, cid, spec)
cancel()
if err != nil {
slog.Error("ecs-agent: run failed", "task", as.TaskID, "container", c.Name, "err", err)
a.teardownTaskNetns(as)
@@ -155,7 +238,12 @@ func (a *Agent) runTask(ctx context.Context, as *bus.Assign) {
// waitContainer blocks until a container exits, then reports the task STOPPED
// with the exit code. v1 stops the whole task when its first container exits.
func (a *Agent) waitContainer(ctx context.Context, as *bus.Assign, name, containerID string) {
status, err := a.runner.Wait(ctx, containerID)
runner := a.rt.get()
if runner == nil {
slog.Warn("ecs-agent: cannot wait on container, no container runtime", "task", as.TaskID, "container", name)
return
}
status, err := runner.Wait(ctx, containerID)
if err != nil {
slog.Warn("ecs-agent: wait container failed", "task", as.TaskID, "container", name, "err", err)
return
@@ -240,6 +328,44 @@ func withCredEnv(env map[string]string, credID string) map[string]string {
return out
}
// containerStartCtx bounds pull+run by a container's startTimeout, if set. A
// nil startTimeout (the common case) returns ctx unchanged with a no-op
// cancel, matching today's unbounded behaviour.
func containerStartCtx(ctx context.Context, startTimeout *int64) (context.Context, context.CancelFunc) {
if startTimeout == nil {
return ctx, func() {}
}
return context.WithTimeout(ctx, time.Duration(*startTimeout)*time.Second)
}
// containerStopTimeout reads the stopTimeout stamped on a container's labels
// by runTask, falling back to defaultStopTimeout when absent or unparseable.
func containerStopTimeout(labels map[string]string) time.Duration {
v, ok := labels[labelStopTimeout]
if !ok {
return defaultStopTimeout
}
secs, err := strconv.ParseInt(v, 10, 64)
if err != nil {
return defaultStopTimeout
}
return time.Duration(secs) * time.Second
}
// toRuntimeSystemControls maps the assign's sysctl pairs to the runtime
// package's own type, keeping cmd/ecs-agent/runtime decoupled from the ecs
// handlers' bus package.
func toRuntimeSystemControls(in []bus.SystemControl) []ctrruntime.SystemControl {
if len(in) == 0 {
return nil
}
out := make([]ctrruntime.SystemControl, len(in))
for i, sc := range in {
out[i] = ctrruntime.SystemControl{Namespace: sc.Namespace, Value: sc.Value}
}
return out
}
// pinContainerGPUs reserves n device UUIDs for a container from the local
// ledger, before the container is started so the runner can inject them as
// CDI devices. A short ledger (fewer free devices than requested, e.g.
@@ -317,6 +443,9 @@ const (
labelCredID = "mulga.ecs.credID"
labelTaskRoleARN = "mulga.ecs.taskRoleArn"
labelENIMac = "mulga.ecs.eniMac"
// labelStopTimeout carries a container's stopTimeout (seconds) so stopTask
// can read it back from List's labels; it has no assign to consult there.
labelStopTimeout = "mulga.ecs.stopTimeout"
)
// taskLabels are the mulga.ecs.* labels stamped on a container. The cred/role/MAC
cmd/ecs-agent/taskrun_test.go
+282 -14
View File
@@ -3,6 +3,7 @@ package main
import (
"context"
"errors"
"slices"
"testing"
"time"
@@ -24,13 +25,13 @@ func testAssign() *bus.Assign {
}
func newRunAgent(cp controlPlane, rt *ctrruntime.FakePuller) *Agent {
return newAgent(config{}, testIdentity(), cp, rt, rt, nil)
return newAgent(config{}, testIdentity(), cp, rt, nil)
}
// runTask with no runtime reports the task STOPPED instead of crashing.
func TestRunTask_NoRuntimeReportsStopped(t *testing.T) {
cp := &fakeCP{}
a := newAgent(config{}, testIdentity(), cp, nil, nil, nil)
a := newAgent(config{}, testIdentity(), cp, nil, nil)
a.runTask(context.Background(), testAssign())
st := cp.taskStates()
@@ -70,6 +71,157 @@ func TestRunTask_PullRunReportsRunning(t *testing.T) {
}
}
// Each enforced field maps onto RunSpec exactly, asserted through the fake
// runtime — the mapping this change exists to wire, without a real containerd.
func TestRunTask_RuntimeFieldsCarriedToRunSpec(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
a := newRunAgent(cp, rt)
as := testAssign()
ro := true
priv := false
tty := true
interactive := true
as.Containers[0].User = "1000:1000"
as.Containers[0].ReadonlyRootFilesystem = &ro
as.Containers[0].Privileged = &priv
as.Containers[0].PseudoTerminal = &tty
as.Containers[0].Interactive = &interactive
as.Containers[0].SystemControls = []bus.SystemControl{{Namespace: "net.core.somaxconn", Value: "1024"}}
as.Containers[0].CapAdd = []string{"SYS_PTRACE"}
as.Containers[0].CapDrop = []string{"NET_RAW"}
a.runTask(context.Background(), as)
if len(rt.Runs) != 1 {
t.Fatalf("expected one run, got %+v", rt.Runs)
}
got := rt.Runs[0]
if got.User != "1000:1000" {
t.Errorf("User = %q, want 1000:1000", got.User)
}
if got.ReadonlyRootFilesystem == nil || !*got.ReadonlyRootFilesystem {
t.Errorf("ReadonlyRootFilesystem = %v, want true", got.ReadonlyRootFilesystem)
}
if got.Privileged == nil || *got.Privileged {
t.Errorf("Privileged = %v, want false (present, not applied)", got.Privileged)
}
if got.PseudoTerminal == nil || !*got.PseudoTerminal {
t.Errorf("PseudoTerminal = %v, want true", got.PseudoTerminal)
}
if got.Interactive == nil || !*got.Interactive {
t.Errorf("Interactive = %v, want true", got.Interactive)
}
if len(got.SystemControls) != 1 || got.SystemControls[0].Namespace != "net.core.somaxconn" {
t.Errorf("SystemControls = %+v, want one net.core.somaxconn entry", got.SystemControls)
}
if len(got.CapAdd) != 1 || got.CapAdd[0] != "SYS_PTRACE" {
t.Errorf("CapAdd = %+v, want [SYS_PTRACE]", got.CapAdd)
}
if len(got.CapDrop) != 1 || got.CapDrop[0] != "NET_RAW" {
t.Errorf("CapDrop = %+v, want [NET_RAW]", got.CapDrop)
}
}
// An assign carrying none of the new fields leaves RunSpec's pointer fields
// nil rather than defaulting to a false indistinguishable from "explicitly
// not requested".
func TestRunTask_NilRuntimeFieldsLeaveRunSpecUnset(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
a := newRunAgent(cp, rt)
a.runTask(context.Background(), testAssign())
if len(rt.Runs) != 1 {
t.Fatalf("expected one run, got %+v", rt.Runs)
}
got := rt.Runs[0]
if got.ReadonlyRootFilesystem != nil || got.Privileged != nil || got.PseudoTerminal != nil || got.Interactive != nil {
t.Errorf("want all runtime-field pointers nil, got %+v", got)
}
if len(got.SystemControls) != 0 || len(got.CapAdd) != 0 || len(got.CapDrop) != 0 {
t.Errorf("want no sysctls/capabilities, got %+v", got)
}
}
// TestRunTask_StampsStopTimeoutLabel verifies runTask stamps a container's
// stopTimeout as a label — the only way stopTask (which only has List's
// labels, not the original assign) can recover it later.
func TestRunTask_StampsStopTimeoutLabel(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
a := newRunAgent(cp, rt)
as := testAssign()
stop := int64(45)
as.Containers[0].StopTimeout = &stop
a.runTask(context.Background(), as)
if len(rt.Runs) != 1 {
t.Fatalf("expected one run, got %+v", rt.Runs)
}
if got := rt.Runs[0].Labels[labelStopTimeout]; got != "45" {
t.Errorf("stopTimeout label = %q, want 45", got)
}
}
// TestRunTask_NoStopTimeoutLabelWhenUnset verifies a container with no
// stopTimeout carries no label, so stopTask falls back to the default rather
// than reading a stray "0".
func TestRunTask_NoStopTimeoutLabelWhenUnset(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
a := newRunAgent(cp, rt)
a.runTask(context.Background(), testAssign())
if len(rt.Runs) != 1 {
t.Fatalf("expected one run, got %+v", rt.Runs)
}
if _, ok := rt.Runs[0].Labels[labelStopTimeout]; ok {
t.Errorf("want no stopTimeout label, got %+v", rt.Runs[0].Labels)
}
}
// TestStopTask_UsesContainerStopTimeout verifies that stopTask reads a
// container's stopTimeout label and calls the runner's graceful Stop (SIGTERM
// + bounded wait) with it, instead of the hard kill+delete used before this
// fix.
func TestStopTask_UsesContainerStopTimeout(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{Containers: []ctrruntime.Container{{
ID: "t-001-web",
Labels: map[string]string{
labelTaskID: "t-001", labelContainerName: "web", labelStopTimeout: "45",
},
}}}
a := newRunAgent(cp, rt)
a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "bye"})
if len(rt.Stopped) != 1 || rt.Stopped[0] != "t-001-web" {
t.Fatalf("want t-001-web stopped, got %+v", rt.Stopped)
}
if len(rt.StopTimeouts) != 1 || rt.StopTimeouts[0] != 45*time.Second {
t.Fatalf("want stopTimeout 45s, got %+v", rt.StopTimeouts)
}
}
// TestStopTask_DefaultStopTimeoutWhenUnset verifies a container with no
// stopTimeout label gets the EC2 launch-type default rather than an unbounded
// or zero grace period.
func TestStopTask_DefaultStopTimeoutWhenUnset(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{Containers: []ctrruntime.Container{{
ID: "t-001-web",
Labels: map[string]string{labelTaskID: "t-001", labelContainerName: "web"},
}}}
a := newRunAgent(cp, rt)
a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "bye"})
if len(rt.StopTimeouts) != 1 || rt.StopTimeouts[0] != defaultStopTimeout {
t.Fatalf("want default stopTimeout %v, got %+v", defaultStopTimeout, rt.StopTimeouts)
}
}
// TestRunTask_GPUCarriedToRunSpec verifies that the AssignContainer's GPU count
// reaches the runtime RunSpec unchanged; a
// container with no GPU request keeps RunSpec.GPU at zero (regression).
@@ -100,7 +252,7 @@ func TestRunTask_GPUCarriedToRunSpec(t *testing.T) {
func TestRunTask_GPUPinsAndReportsUUIDs(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
a := newAgent(config{}, testIdentityWithGPUs("GPU-aaa", "GPU-bbb"), cp, rt, rt, nil)
a := newAgent(config{}, testIdentityWithGPUs("GPU-aaa", "GPU-bbb"), cp, rt, nil)
as := testAssign()
as.Containers = append(as.Containers, bus.AssignContainer{
Name: "trainer", Image: "registry/trainer:1", GPU: 1,
@@ -144,7 +296,7 @@ func TestRunTask_GPUPinsAndReportsUUIDs(t *testing.T) {
func TestRunTask_GPUIDsReachRunSpec(t *testing.T) {
cp := &fakeCP{}
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
a := newAgent(config{}, testIdentityWithGPUs("GPU-aaa", "GPU-bbb"), cp, rt, rt, nil)
a := newAgent(config{}, testIdentityWithGPUs("GPU-aaa", "GPU-bbb"), cp, rt, nil)
as := testAssign()
as.Containers = append(as.Containers, bus.AssignContainer{
Name: "trainer", Image: "registry/trainer:1", GPU: 1,
@@ -209,7 +361,7 @@ func TestStopTask_ReleasesGPUsBackToLedger(t *testing.T) {
"mulga.ecs.taskID": "t-001", "mulga.ecs.containerName": "trainer",
}},
}}
a := newAgent(config{}, testIdentityWithGPUs("GPU-aaa"), cp, rt, rt, nil)
a := newAgent(config{}, testIdentityWithGPUs("GPU-aaa"), cp, rt, nil)
if _, err := a.gpu.Pin(gpuKey("t-001", "trainer"), 1); err != nil {
t.Fatalf("seed pin: %v", err)
}
@@ -287,7 +439,7 @@ func TestPollAssignments_DispatchesOnceAndAcks(t *testing.T) {
nil, // poll 3: gateway dropped it after the ack
}}
rt := &ctrruntime.FakePuller{WaitErr: errors.New("blocked")}
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, rt, nil)
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, nil)
ctx, cancel := context.WithCancel(context.Background())
go a.pollAssignments(ctx, map[string]bool{})
@@ -342,8 +494,8 @@ func TestStopTask_ReapsLabeledContainersAndReportsStopped(t *testing.T) {
a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "bye"})
if len(rt.Removed) != 1 || rt.Removed[0] != "t-001-web" {
t.Fatalf("want only t-001-web removed, got %+v", rt.Removed)
if len(rt.Stopped) != 1 || rt.Stopped[0] != "t-001-web" {
t.Fatalf("want only t-001-web stopped, got %+v", rt.Stopped)
}
st := cp.taskStates()
if len(st) != 1 || st[0].LastStatus != bus.TaskStatusStopped {
@@ -381,12 +533,12 @@ func TestPollAssignments_StopReapsAndSuppressesAssign(t *testing.T) {
as := testAssign()
cp := &fakeCP{
pollReplies: [][]bus.Assign{{*as}, {*as}, nil},
stopReplies: [][]bus.StopDirective{{{TaskID: as.TaskID, Reason: "bye"}}},
stopPending: []bus.StopDirective{{TaskID: as.TaskID, Reason: "bye"}},
}
rt := &ctrruntime.FakePuller{Containers: []ctrruntime.Container{
{ID: "t-001-web", Running: true, Labels: map[string]string{"mulga.ecs.taskID": "t-001"}},
}}
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, rt, nil)
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, nil)
ctx, cancel := context.WithCancel(context.Background())
go a.pollAssignments(ctx, map[string]bool{})
@@ -425,10 +577,126 @@ func TestPollAssignments_StopReapsAndSuppressesAssign(t *testing.T) {
}
}
// countStopAcks reports how many polls carried taskID in their stop-ack list.
func countStopAcks(acks [][]string, taskID string) int {
n := 0
for _, ack := range acks {
if slices.Contains(ack, taskID) {
n++
}
}
return n
}
// A reap that cannot run is never acked, so the directive stays in the gateway's
// inbox and is redelivered and retried until the runtime comes back.
func TestPollAssignments_UnperformableStopIsNotAckedAndRetried(t *testing.T) {
cp := &fakeCP{stopPending: []bus.StopDirective{{TaskID: "t-001", Reason: "bye"}}}
rt := &ctrruntime.FakePuller{ListErr: errors.New("containerd is down")}
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, nil)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go a.pollAssignments(ctx, map[string]bool{})
deadline := time.After(time.Second)
for rt.Lists() < 3 {
select {
case <-deadline:
t.Fatalf("failed reap was not retried; List calls=%d", rt.Lists())
case <-time.After(2 * time.Millisecond):
}
}
cancel()
if n := countStopAcks(cp.stopAcks(), "t-001"); n != 0 {
t.Errorf("a reap that never ran was acked %d times; stopAcks=%v", n, cp.stopAcks())
}
if stopped := countStatus(cp.taskStates(), "t-001", bus.TaskStatusStopped); stopped != 0 {
t.Errorf("a reap that never ran reported STOPPED %d times", stopped)
}
}
// Once the runtime recovers, the redelivered directive is reaped, reported
// STOPPED and acked exactly once, which deletes it from the inbox.
func TestPollAssignments_RecoveredStopIsAckedExactlyOnce(t *testing.T) {
cp := &fakeCP{stopPending: []bus.StopDirective{{TaskID: "t-001", Reason: "bye"}}}
rt := &ctrruntime.FakePuller{
ListErr: errors.New("containerd is down"),
ListErrFor: 1,
Containers: []ctrruntime.Container{
{ID: "t-001-web", Running: true, Labels: map[string]string{"mulga.ecs.taskID": "t-001"}},
},
}
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, nil)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go a.pollAssignments(ctx, map[string]bool{})
deadline := time.After(2 * time.Second)
for countStopAcks(cp.stopAcks(), "t-001") == 0 {
select {
case <-deadline:
t.Fatalf("recovered reap was never acked; states=%+v stopAcks=%v", cp.taskStates(), cp.stopAcks())
case <-time.After(2 * time.Millisecond):
}
}
time.Sleep(30 * time.Millisecond)
cancel()
if n := countStopAcks(cp.stopAcks(), "t-001"); n != 1 {
t.Errorf("stop acked %d times, want exactly 1; stopAcks=%v", n, cp.stopAcks())
}
if stopped := countStatus(cp.taskStates(), "t-001", bus.TaskStatusStopped); stopped != 1 {
t.Errorf("task reported STOPPED %d times, want exactly 1", stopped)
}
if reaped := rt.Reaped(); len(reaped) != 1 || reaped[0] != "t-001-web" {
t.Errorf("want t-001-web reaped once, got %+v", reaped)
}
}
// A task whose reap completed is never started again by an assign the gateway
// is still redelivering.
func TestPollAssignments_StoppedTaskIsNotRerunByRedeliveredAssign(t *testing.T) {
as := testAssign()
cp := &fakeCP{
pollReplies: [][]bus.Assign{{*as}, {*as}, {*as}, {*as}, {*as}, {*as}},
stopPending: []bus.StopDirective{{TaskID: as.TaskID, Reason: "bye"}},
}
rt := &ctrruntime.FakePuller{Containers: []ctrruntime.Container{
{ID: "t-001-web", Running: true, Labels: map[string]string{"mulga.ecs.taskID": "t-001"}},
}}
a := newAgent(config{PollInterval: 5 * time.Millisecond}, testIdentity(), cp, rt, nil)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go a.pollAssignments(ctx, map[string]bool{})
deadline := time.After(time.Second)
for countStopAcks(cp.stopAcks(), as.TaskID) == 0 {
select {
case <-deadline:
t.Fatalf("stop never completed; states=%+v", cp.taskStates())
case <-time.After(2 * time.Millisecond):
}
}
// Let the remaining scripted assigns arrive after the reap has finished.
time.Sleep(40 * time.Millisecond)
cancel()
if runs := rt.RunCalls(); len(runs) != 0 {
t.Errorf("stopped task was started by a redelivered assign: %+v", runs)
}
if running := countStatus(cp.taskStates(), as.TaskID, bus.TaskStatusRunning); running != 0 {
t.Errorf("stopped task reported RUNNING %d times", running)
}
}
// stopTask with no runtime is a no-op: nothing to reap, no state reported.
func TestStopTask_NilRunnerNoOp(t *testing.T) {
cp := &fakeCP{}
a := newAgent(config{}, testIdentity(), cp, nil, nil, nil)
a := newAgent(config{}, testIdentity(), cp, nil, nil)
a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001", Reason: "x"})
if len(cp.taskStates()) != 0 {
t.Fatalf("nil runner should report no state, got %+v", cp.taskStates())
@@ -459,8 +727,8 @@ func TestStopTask_EmptyReasonReapsWithDefault(t *testing.T) {
a := newRunAgent(cp, rt)
a.stopTask(context.Background(), bus.StopDirective{TaskID: "t-001"})
if len(rt.Removed) != 1 || rt.Removed[0] != "t-001-web" {
t.Fatalf("want container t-001-web reaped, got %+v", rt.Removed)
if len(rt.Stopped) != 1 || rt.Stopped[0] != "t-001-web" {
t.Fatalf("want container t-001-web reaped, got %+v", rt.Stopped)
}
st := cp.taskStates()
if len(st) != 1 || st[0].LastStatus != bus.TaskStatusStopped {
@@ -539,7 +807,7 @@ func (sentinelResolver) Authorize(context.Context, string) (string, string, stri
// exec-role resolver is built.
func TestPullResolver(t *testing.T) {
fake := sentinelResolver{}
a := newAgent(config{Region: "us-east-1"}, testIdentity(), &fakeCP{}, nil, nil, fake)
a := newAgent(config{Region: "us-east-1"}, testIdentity(), &fakeCP{}, nil, fake)
if got := a.pullResolver(&bus.Assign{TaskID: "t1"}); got != fake {
t.Fatalf("no execution role: want instance resolver fallback")
cmd/eks-token-webhook/cert.go
+4 -16
View File
@@ -33,12 +33,12 @@ func serverTLSConfig(cert tls.Certificate) *tls.Config {
// valid even if the webhook process is restarted. Returns the parsed cert and
// the certificate PEM (for embedding as the kubeconfig CA bundle).
func ensureServingCert(certPath, keyPath string) (tls.Certificate, []byte, error) {
if certPEM, keyPEM, ok := readCertPair(certPath, keyPath); ok {
cert, err := tls.X509KeyPair(certPEM, keyPEM)
if err == nil {
// Fall through to regenerate on a missing/corrupt/incompatible persisted
// pair. The cert PEM is re-read because tls.Certificate keeps only DER.
if cert, err := tls.LoadX509KeyPair(certPath, keyPath); err == nil {
if certPEM, err := os.ReadFile(certPath); err == nil {
return cert, certPEM, nil
}
// Fall through to regenerate on a corrupt/incompatible persisted pair.
}
certPEM, keyPEM, err := generateSelfSigned()
@@ -58,18 +58,6 @@ func ensureServingCert(certPath, keyPath string) (tls.Certificate, []byte, error
return cert, certPEM, nil
}
func readCertPair(certPath, keyPath string) (certPEM, keyPEM []byte, ok bool) {
c, err := os.ReadFile(certPath)
if err != nil {
return nil, nil, false
}
k, err := os.ReadFile(keyPath)
if err != nil {
return nil, nil, false
}
return c, k, true
}
func generateSelfSigned() (certPEM, keyPEM []byte, err error) {
priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
cmd/spinifex/cmd/admin.go
+260 -50
View File
@@ -30,6 +30,7 @@ import (
"github.com/mulgadc/bluebottle/pkg/masterkey"
"github.com/mulgadc/bluebottle/pkg/safecast"
"github.com/mulgadc/spinifex/spinifex/admin"
"github.com/mulgadc/spinifex/spinifex/clustersize"
"github.com/mulgadc/spinifex/spinifex/config"
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
@@ -200,6 +201,15 @@ takes effect immediately. Prompts for confirmation unless --yes is passed.`,
Run: runimagesPromoteCmd,
}
var imagesDescribeCmd = &cobra.Command{
Use: "describe",
Short: "Show an AMI's metadata and source digest",
Long: `Print an AMI's control-plane metadata, including the digest of the
artifact it was imported from and who verified it. The digest describes the
imported source file, not the volume's current contents.`,
Run: runimagesDescribeCmd,
}
var volumesCmd = &cobra.Command{
Use: "volumes",
Short: "Inspect block storage volumes",
@@ -294,7 +304,7 @@ spx admin images import --name debian-13-x86_64
spx admin images list
// Manually import a path
spx admin images import --file /path/to/image --distro debian --version 13 --arch x86_64
spx admin images import --file /path/to/image --checksum /path/to/SHA512SUMS --distro debian --version 13 --arch x86_64
-> x <-
*/
@@ -328,6 +338,7 @@ func init() {
imagesCmd.AddCommand(imagesListCmd)
imagesCmd.AddCommand(imagesRemoveCmd)
imagesCmd.AddCommand(imagesPromoteCmd)
imagesCmd.AddCommand(imagesDescribeCmd)
adminCmd.AddCommand(volumesCmd)
volumesCmd.AddCommand(volumesOrphansCmd)
@@ -362,7 +373,7 @@ func init() {
adminInitCmd.Flags().String("region", "ap-southeast-2", "Mulga region to create")
adminInitCmd.Flags().String("az", "ap-southeast-2a", "Mulga AZ to create")
adminInitCmd.Flags().String("node", "node1", "Node name, increment for additional nodes (default, node1)")
adminInitCmd.Flags().Int("nodes", 3, "Number of nodes to expect for cluster")
adminInitCmd.Flags().Int("nodes", 3, "Number of nodes to expect for cluster (1, or 3 and up)")
adminInitCmd.Flags().String("host", "", "Leader node to join (if not specified, tries multicast discovery)")
adminInitCmd.Flags().Int("port", 4432, "Port to bind cluster services on")
adminInitCmd.Flags().String("bind", "0.0.0.0", "IP address to bind services to (e.g., 10.11.12.1 for multi-node). Default 0.0.0.0 listens on all interfaces.")
@@ -427,6 +438,7 @@ func init() {
imagesImportCmd.Flags().StringSlice("tag", nil, "Tag to apply to the imported AMI as key=value (repeatable; e.g. --tag spinifex:managed-by=elbv2)")
imagesImportCmd.Flags().Bool("force", false, "Force command execution (overwrites existing files)")
imagesImportCmd.Flags().Bool("skip-verify", false, "Skip catalog-image checksum verification (INSECURE; operator assumes integrity responsibility)")
imagesImportCmd.Flags().String("checksum", "", "Sums file to verify a --file image against (SHA256SUMS, SHA512SUMS, <image>.sha256/.sha512)")
imagesRemoveCmd.Flags().String("image-id", "", "AMI ID to remove (required)")
imagesRemoveCmd.Flags().Bool("force", false, "Bypass dependency, ownership and config-corrupt checks (salvage mode)")
@@ -436,6 +448,9 @@ func init() {
imagesPromoteCmd.Flags().String("image-id", "", "AMI ID to promote to system image (required)")
imagesPromoteCmd.Flags().Bool("yes", false, "Skip interactive confirmation prompt")
_ = imagesPromoteCmd.MarkFlagRequired("image-id")
imagesDescribeCmd.Flags().String("image-id", "", "AMI ID to describe (required)")
_ = imagesDescribeCmd.MarkFlagRequired("image-id")
}
const bytesPerGiB = 1024 * 1024 * 1024
@@ -464,6 +479,108 @@ func amiVolumeSizeGiB(sizeBytes int64) uint64 {
return safecast.Int64ToUint64((sizeBytes + bytesPerGiB - 1) / bytesPerGiB)
}
// validateChecksumFlags rejects --checksum combinations that would otherwise be silently ignored.
// An explicitly empty --checksum (e.g. an unset shell variable) must not fall through to an unverified import.
func validateChecksumFlags(checksumPath string, checksumSet bool, localFile string, skipVerify bool) error {
if !checksumSet {
return nil
}
if strings.TrimSpace(checksumPath) == "" {
return errors.New("--checksum requires a sums file path")
}
if localFile == "" {
return errors.New("--checksum requires --file")
}
if skipVerify {
return errors.New("--checksum and --skip-verify are mutually exclusive")
}
return nil
}
// catalogPinnedDigestSource records that the expected digest came from this
// binary's catalog, so a recorded digest never reads as a fetched sums file.
const catalogPinnedDigestSource = "catalog:pinned"
// importSource is the file an import is about to extract, and what it is to be verified against.
type importSource struct {
imageFile, imageName, localFile string
image utils.Images
checksumPath, checksumAlgo, expectedDigest string
skipVerify bool
}
// resolveDigest verifies or hashes the source before extract and returns the digest to record.
// Upstream digests are of the artifact as supplied. Failures are reported to errOut before
// returning; the file is left in place, and a catalog download recovers with --force.
func (src importSource) resolveDigest(out, errOut io.Writer) (ebsmetadata.ImageDigest, error) {
digest := ebsmetadata.ImageDigest{Filename: filepath.Base(src.imageFile)}
switch sourceDigestMode(src.localFile, src.checksumPath, src.skipVerify) {
case ebsmetadata.DigestCatalog:
if src.image.ChecksumType == "" || (src.image.Checksum == "" && src.image.ChecksumDigest == "") {
fmt.Fprintf(errOut, "Catalog entry %q is missing Checksum/ChecksumDigest/ChecksumType; refusing import.\n", src.imageName)
return digest, errors.New("catalog entry has no checksum")
}
// A pinned digest is checked against this file rather than a sums file
// fetched from the same host that served the image.
if src.image.ChecksumDigest != "" {
actual, err := utils.VerifyImageDigest(src.imageFile, src.image.ChecksumType, src.image.ChecksumDigest)
if err != nil {
printChecksumError(errOut, src.imageFile, src.imageName, src.image, err)
return digest, err
}
fmt.Fprintf(out, "✅ Verified image checksum (%s) against the pinned catalog digest\n", src.image.ChecksumType)
digest.Algorithm, digest.Value, digest.Source = src.image.ChecksumType, actual, catalogPinnedDigestSource
digest.Verification = ebsmetadata.DigestCatalog
return digest, nil
}
actual, err := utils.VerifyImageChecksum(src.imageFile, src.image.Checksum, src.image.ChecksumType)
if err != nil {
printChecksumError(errOut, src.imageFile, src.imageName, src.image, err)
return digest, err
}
fmt.Fprintf(out, "✅ Verified image checksum (%s)\n", src.image.ChecksumType)
digest.Algorithm, digest.Value, digest.Source = src.image.ChecksumType, actual, src.image.Checksum
digest.Verification = ebsmetadata.DigestCatalog
case ebsmetadata.DigestOperator:
actual, err := utils.VerifyImageDigest(src.localFile, src.checksumAlgo, src.expectedDigest)
if err != nil {
fmt.Fprintf(errOut, "Image integrity verification failed: %v\n", err)
fmt.Fprintf(errOut, " file: %s\n", src.localFile)
fmt.Fprintf(errOut, " checksum: %s\n", src.checksumPath)
return digest, err
}
fmt.Fprintf(out, "✅ Verified image checksum (%s) against %s\n", src.checksumAlgo, src.checksumPath)
digest.Algorithm, digest.Value, digest.Source = src.checksumAlgo, actual, filepath.Base(src.checksumPath)
digest.Verification = ebsmetadata.DigestOperator
default:
if src.localFile != "" {
fmt.Fprintf(errOut, "Importing %s without checksum validation (pass --checksum <sums-file> to verify)\n", src.localFile)
} else {
fmt.Fprintf(errOut, "⚠️ --skip-verify set: checksum verification skipped for %s\n", src.imageName)
}
actual, err := utils.HashImageFile(src.imageFile, "sha256")
if err != nil {
fmt.Fprintf(errOut, "Could not hash image: %v\n", err)
return digest, err
}
digest.Algorithm, digest.Value = "sha256", actual
digest.Verification = ebsmetadata.DigestUnverified
}
return digest, nil
}
// sourceDigestMode picks how an import's source digest is obtained.
func sourceDigestMode(localFile, checksumPath string, skipVerify bool) string {
switch {
case localFile != "" && checksumPath != "":
return ebsmetadata.DigestOperator
case localFile != "" || skipVerify:
return ebsmetadata.DigestUnverified
default:
return ebsmetadata.DigestCatalog
}
}
func runimagesImportCmd(cmd *cobra.Command, args []string) {
var image utils.Images
@@ -503,6 +620,20 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
os.Exit(1)
}
checksumPath, _ := cmd.Flags().GetString("checksum")
if err := validateChecksumFlags(checksumPath, cmd.Flags().Changed("checksum"), localFile, skipVerify); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
var checksumAlgo, expectedDigest string
if checksumPath != "" {
checksumAlgo, expectedDigest, err = utils.ReadExpectedDigest(checksumPath, filepath.Base(localFile))
if err != nil {
fmt.Fprintf(os.Stderr, "Could not read checksum from %s: %v\n", checksumPath, err)
os.Exit(1)
}
}
if imageName != "" {
var exists bool
image, exists = utils.AvailableImages[imageName]
@@ -589,8 +720,8 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
os.Exit(1)
}
// Next, if the file is selected to download, fetch it, extract disk image, and save to path
if imageName != "" && localFile == "" {
// Next, if the file is selected to download, fetch it and save to path
if localFile == "" {
// Download the file to the image path
filename := path.Base(image.URL)
imageFile = fmt.Sprintf("%s/%s", imagePath, filename)
@@ -606,24 +737,20 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
os.Exit(1)
}
}
}
// Verify before extract: the catalog digest is of the artifact as it
// sits on the mirror (.tar.xz/.img/.raw). Also catches a poisoned
// cache on the re-run path — failure leaves the file on disk so the
// operator can inspect; recover with --force.
if skipVerify {
fmt.Fprintf(os.Stderr, "⚠️ --skip-verify set: checksum verification skipped for %s\n", imageName)
} else {
if image.Checksum == "" || image.ChecksumType == "" {
fmt.Fprintf(os.Stderr, "Catalog entry %q is missing Checksum/ChecksumType; refusing import.\n", imageName)
os.Exit(1)
}
if err := utils.VerifyImageChecksum(imageFile, image.Checksum, image.ChecksumType); err != nil {
printChecksumError(os.Stderr, imageFile, imageName, image, err)
os.Exit(1)
}
fmt.Printf("✅ Verified image checksum (%s)\n", image.ChecksumType)
}
sourceDigest, err := importSource{
imageFile: imageFile,
imageName: imageName,
localFile: localFile,
image: image,
checksumPath: checksumPath,
checksumAlgo: checksumAlgo,
expectedDigest: expectedDigest,
skipVerify: skipVerify,
}.resolveDigest(os.Stdout, os.Stderr)
if err != nil {
os.Exit(1)
}
// Next, validate if the image is raw, tar, gz, xv, etc. We need to upload the raw image
@@ -660,7 +787,7 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
Description: fmt.Sprintf("%s cloud image prepared for Spinifex", amiName),
Architecture: image.Arch,
PlatformDetails: image.Platform,
CreationDate: time.Now(),
CreationDate: time.Now().UTC(),
RootDeviceType: "ebs",
Virtualization: "hvm",
ImageOwnerAlias: "system",
@@ -669,6 +796,7 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
BootMode: image.BootMode,
Distro: image.Distro,
DistroFamily: utils.DistroFamily(image.Distro),
SourceDigest: &sourceDigest,
}
// Copy catalog-provided tags (e.g. spinifex:managed-by for system AMIs)
@@ -739,6 +867,7 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
}
fmt.Printf("✅ Image import complete. Image-ID (AMI): %s\n", volumeId)
fmt.Printf("✅ Source digest: %s:%s (%s)\n", sourceDigest.Algorithm, sourceDigest.Value, sourceDigest.Verification)
}
// registerImportedAMISnapshot writes the EC2 control plane's snapshot document
@@ -1001,6 +1130,62 @@ func runimagesPromoteCmd(cmd *cobra.Command, args []string) {
fmt.Printf("✅ Promoted %s to system image (owner: %s).\n", imageID, admin.SystemOwnerAlias)
}
func runimagesDescribeCmd(cmd *cobra.Command, args []string) {
imageID, _ := cmd.Flags().GetString("image-id")
cfgFile, _ := cmd.Flags().GetString("config")
if cfgFile == "" {
cfgFile = DefaultConfigFile()
}
appConfig, err := config.LoadConfig(cfgFile)
if err != nil {
fmt.Fprintln(os.Stderr, "Error loading config file:", err)
os.Exit(1)
}
node := appConfig.Nodes[appConfig.Node]
store := objectstore.NewS3ObjectStoreFromConfig(
node.Predastore.Host,
node.Predastore.Region,
node.Predastore.AccessKey,
node.Predastore.SecretKey,
)
meta, err := admin.GetAMIMetadata(store, node.Predastore.Bucket, imageID)
if err != nil {
fmt.Fprintln(os.Stderr, "Failed to describe AMI:", err)
os.Exit(1)
}
printAMIDescription(os.Stdout, meta)
}
// printAMIDescription writes an AMI's metadata and source digest for an operator.
func printAMIDescription(w io.Writer, meta ebsmetadata.AMI) {
fmt.Fprintf(w, "Image ID: %s\n", meta.ImageID)
fmt.Fprintf(w, "Name: %s\n", meta.Name)
fmt.Fprintf(w, "Owner: %s\n", meta.ImageOwnerAlias)
if !meta.CreationDate.IsZero() {
fmt.Fprintf(w, "Created: %s\n", meta.CreationDate.UTC().Format("2006-01-02T15:04:05Z"))
}
fmt.Fprintf(w, "Boot mode: %s\n", meta.BootMode)
fmt.Fprintf(w, "Architecture: %s\n", meta.Architecture)
d := meta.SourceDigest
if d == nil {
fmt.Fprintln(w, "Source digest: not recorded (imported before digest recording)")
return
}
fmt.Fprintf(w, "Source digest: %s:%s\n", d.Algorithm, d.Value)
if d.Source != "" {
fmt.Fprintf(w, "Verification: %s (%s)\n", d.Verification, d.Source)
} else {
fmt.Fprintf(w, "Verification: %s\n", d.Verification)
}
fmt.Fprintf(w, "Source file: %s\n", d.Filename)
}
// List remote images available.
func runimagesListCmd(cmd *cobra.Command, args []string) {
//fmt.Println(availableImages)
@@ -1028,6 +1213,26 @@ func runimagesListCmd(cmd *cobra.Command, args []string) {
pterm.Println("spx admin images import --name <image-name>")
}
// validateNodeCount accepts the cluster sizes Spinifex supports: one server, or
// three and up.
//
// Two is rejected rather than allowed and warned about. On two servers OVN runs
// standalone, the storage metadata quorum has no majority to lose, and the
// control plane's own KV state has no replica count that works — one replica
// disappears with its node, and two stop accepting writes when either goes. A
// cluster that cannot survive the event it was built for is worth refusing at the
// point someone asks for it.
func validateNodeCount(nodes int) error {
if nodes < 1 {
return fmt.Errorf("a cluster needs at least one node, got %d", nodes)
}
if nodes > 1 && nodes < clustersize.MinHANodes {
return fmt.Errorf("%d nodes is not a supported cluster size: use 1 for a single server, or %d and up for a cluster that can survive losing one",
nodes, clustersize.MinHANodes)
}
return nil
}
// TODO: Move all logic to a module, use minimal application logic in viper commands.
func runAdminInit(cmd *cobra.Command, args []string) {
if os.Getuid() != 0 {
@@ -1059,6 +1264,15 @@ func runAdminInit(cmd *cobra.Command, args []string) {
clusterName, _ := cmd.Flags().GetString("cluster-name")
services, _ := cmd.Flags().GetStringSlice("services")
// Rejected before anything is written, because a two-node cluster has no
// correct answer rather than a worse one: no layer of the stack has a
// majority to lose, and no replica count for its state both survives a node
// going away and still accepts writes.
if err := validateNodeCount(nodes); err != nil {
fmt.Fprintf(os.Stderr, "--nodes: %v\n", err)
os.Exit(1)
}
// Optional operator email — validated up-front so a bad address fails
// before we touch any config state. Empty is allowed here; reset / repeat
// inits on a box that already has an email in /etc/spinifex/spinifex.toml
@@ -1116,7 +1330,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
detected, err := admin.DetectNetwork()
if err != nil {
fmt.Fprintf(os.Stderr, "⚠️ Network auto-detection failed: %v\n", err)
fmt.Fprintf(os.Stderr, " Use --external-mode=nat for outbound-only VMs on a non-bridgeable uplink, or specify --external-* flags manually.\n")
fmt.Fprintf(os.Stderr, " Use --external-mode=nat on a non-bridgeable uplink, or specify --external-* flags manually.\n")
} else {
detectedNet = detected
@@ -1154,7 +1368,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
if externalMode == "" && !cmd.Flags().Changed("external-mode") {
if isNonBridgeableUplink(detected.WAN.Name) {
fmt.Fprintf(os.Stderr, "\n❌ Detected WAN interface %s cannot be bridged (WiFi/cellular/PPP).\n", detected.WAN.Name)
fmt.Fprintf(os.Stderr, " Use routed NAT mode instead (outbound-only VM networking):\n")
fmt.Fprintf(os.Stderr, " Use routed NAT mode instead:\n")
fmt.Fprintf(os.Stderr, " ./scripts/setup-ovn.sh --management --nat-uplink\n")
fmt.Fprintf(os.Stderr, " spx admin init --external-mode=nat\n")
os.Exit(1)
@@ -1176,9 +1390,14 @@ func runAdminInit(cmd *cobra.Command, args []string) {
// the transit segment claims externalGateway below.
natPublicGateway := externalGateway
if externalMode == "nat" {
// Verified on three OCI nodes 2026-09-26: each node delivers its own
// instances' public IPs, inbound and outbound. Still flagged because no
// nightly cell covers multi-node nat yet.
if nodes >= 2 {
fmt.Fprintf(os.Stderr, "❌ Error: --external-mode=nat is single-node only (v1); use --nodes=1\n")
os.Exit(1)
fmt.Fprintf(os.Stderr, "⚠️ --external-mode=nat on %d nodes is NEW.\n", nodes)
fmt.Fprintf(os.Stderr, " Public IPs are delivered by the node running the instance; default egress stays on the VPC's gateway chassis.\n")
fmt.Fprintf(os.Stderr, " Verified by hand on three nodes, but not yet covered by a nightly cell.\n")
fmt.Fprintf(os.Stderr, " Check with: ip route show dev spx-nat-host — exactly one node should hold each public address.\n")
}
if !natPublicPool && (externalBindBridge != "" || gatewayIP != "") {
fmt.Fprintf(os.Stderr, "❌ Error: --external-bind-bridge/--gateway-ip require --external-pool or --external-source in --external-mode=nat\n")
@@ -1439,7 +1658,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
}
// Generate SSL certificates (with bind IP in SANs for multi-node support)
certPath := admin.GenerateCertificatesIfNeeded(configDir, force, bindIP, region, config.DefaultAWSInternalSuffix)
certPath := admin.GenerateCertificatesIfNeeded(configDir, force, bindIP, region, config.DefaultAWSServicesDomain)
// Generate per-node IPsec peer cert when cluster-wide IPsec is enabled
// (default true). Reuses the cluster CA — no intermediate strongSwan PKI.
@@ -1508,14 +1727,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
bootstrapSubnetId := utils.GenerateResourceID("subnet")
bootstrapIgwId := utils.GenerateResourceID("igw")
networkConfig.ExternalMode = externalMode
networkConfig.PoolName = "wan"
networkConfig.PoolSource = externalSource
networkConfig.PoolBindBridge = externalBindBridge
networkConfig.PoolStart = poolStart
networkConfig.PoolEnd = poolEnd
networkConfig.PoolGateway = externalGateway
networkConfig.PoolGatewayIP = gatewayIP
networkConfig.PoolPrefixLen = externalPrefixLen
networkConfig.Pools = externalPools
networkConfig.PoolDNSServers = dnsServers
networkConfig.BootstrapAccountId = admin.DefaultAccountID()
networkConfig.BootstrapVpcId = bootstrapVpcId
@@ -1668,7 +1880,7 @@ func runAdminInit(cmd *cobra.Command, args []string) {
fmt.Printf(" Public pool: dhcp via %s\n", externalBindBridge)
}
} else {
fmt.Printf("\n📡 External networking: nat (routed, outbound-only — no public IPs/EIPs)\n")
fmt.Printf("\n📡 External networking: nat (routed; no public pool configured yet)\n")
}
fmt.Printf(" Transit: %s via %s (host masquerades out any uplink)\n", host.NATTransitCIDR, host.NATTransitHostEnd)
fmt.Printf(" Host setup: ./scripts/setup-ovn.sh --nat-uplink (run before starting services)\n")
@@ -2423,7 +2635,7 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
}
// Generate server cert signed by CA with this node's bind IP
if err := admin.GenerateServerCertOnly(configDir, bindIP, region, config.DefaultAWSInternalSuffix); err != nil {
if err := admin.GenerateServerCertOnly(configDir, bindIP, region, config.DefaultAWSServicesDomain); err != nil {
fmt.Fprintf(os.Stderr, "Error generating server certificate: %v\n", err)
os.Exit(1)
}
@@ -2642,7 +2854,7 @@ func initIAMServiceFromConfig() (*handlers_iam.IAMServiceImpl, *config.ClusterCo
// Background: this runs at CLI top level, where there is no request to
// inherit a deadline from and the process exits after the one command.
svc, err := handlers_iam.NewIAMServiceImpl(context.Background(), nc, masterKey, len(cfg.Nodes))
svc, err := handlers_iam.NewIAMServiceImpl(context.Background(), nc, masterKey)
if err != nil {
nc.Close()
return nil, nil, nil, nil, fmt.Errorf("init IAM service: %w", err)
@@ -3032,18 +3244,11 @@ func applyNetworkConfig(settings *admin.ConfigSettings, nc *formation.NetworkCon
settings.IPSecEnabled = nc.IPSecEnabled
settings.ExternalMode = nc.ExternalMode
settings.PoolDNSServers = nc.PoolDNSServers
// Derived from the mode rather than carried in NetworkConfig, so the leader
// and every joiner pin the same value from the same rule as single-node init.
settings.BridgeMode = bridgeModeFor(nc.ExternalMode)
if nc.ExternalMode != "" {
settings.Pools = []admin.PoolData{{
Name: nc.PoolName,
Source: nc.PoolSource,
BindBridge: nc.PoolBindBridge,
Start: nc.PoolStart,
End: nc.PoolEnd,
Gateway: nc.PoolGateway,
GatewayIP: nc.PoolGatewayIP,
PrefixLen: nc.PoolPrefixLen,
DNSServers: nc.PoolDNSServers,
}}
settings.Pools = nc.Pools
}
settings.BootstrapAccountId = nc.BootstrapAccountId
@@ -3744,7 +3949,12 @@ func resolveIfaceIP(iface string) string {
func printChecksumError(w io.Writer, imageFile, imageName string, image utils.Images, err error) {
fmt.Fprintf(w, "Image integrity verification failed: %v\n", err)
fmt.Fprintf(w, " file: %s\n", imageFile)
fmt.Fprintf(w, " source: %s\n", image.Checksum)
if image.ChecksumDigest != "" {
fmt.Fprintf(w, " source: %s (%s)\n", catalogPinnedDigestSource, image.ChecksumDigest)
fmt.Fprintf(w, " url: %s\n", image.URL)
} else {
fmt.Fprintf(w, " source: %s\n", image.Checksum)
}
fmt.Fprintln(w)
fmt.Fprintln(w, "The cached file was left in place. To re-download and retry:")
fmt.Fprintf(w, " spx admin images import --name %s --force\n", imageName)
cmd/spinifex/cmd/admin_kv_replicas.go
+201
View File
@@ -0,0 +1,201 @@
package cmd
import (
"context"
"encoding/json"
"fmt"
"io"
"os"
"strings"
"text/tabwriter"
"github.com/mulgadc/spinifex/spinifex/kvutil"
"github.com/nats-io/nats.go/jetstream"
"github.com/spf13/cobra"
)
var kvReplicasCmd = &cobra.Command{
Use: "replicas",
Short: "Report how many nodes each KV bucket is replicated across",
Long: `Print one line per JetStream KV bucket: how many replicas it has, how many the
cluster's size entitles it to, and which nodes currently hold it.
A bucket with fewer replicas than the cluster has nodes lives on fewer nodes
than exist, and every node depends on those. Losing one of them makes the
bucket unreadable and unwritable from every node at once, which is a
cluster-wide outage caused by a single-node event.
A bucket also has to be able to serve, which is a different question from how
many replicas it was configured for. ONLINE counts the replicas that are caught
up and reachable; a bucket whose ONLINE is not a majority of REPLICAS cannot
accept a write no matter what its replica count says.
--repair raises any under-replicated bucket to the cluster's replica count,
capped at JetStream's maximum of five. It changes nothing else about a bucket
and never lowers a replica count, so it is safe to run on a healthy cluster and
safe to run twice.
Exit status separates the two problems, so this can gate a deployment without
parsing the output:
0 every bucket is at the cluster's replica count and can serve
1 at least one bucket is under-replicated — --repair is the answer
2 at least one bucket has no quorum, or the report could not be produced —
a node needs looking at, and --repair will not help`,
Run: runKVReplicas,
}
func init() {
kvCmd.AddCommand(kvReplicasCmd)
kvReplicasCmd.Flags().Bool("repair", false, "raise every under-replicated bucket to the cluster's node count")
kvReplicasCmd.Flags().Bool("json", false, "emit the report as JSON")
}
func runKVReplicas(cmd *cobra.Command, _ []string) {
repair, _ := cmd.Flags().GetBool("repair")
asJSON, _ := cmd.Flags().GetBool("json")
// Anything that stops the report being produced exits unreachable rather
// than under-replicated: an empty answer is not a cluster with no buckets,
// and --repair is not the response to it.
_, nc, err := loadConfigAndConnect()
if err != nil {
fmt.Fprintf(os.Stderr, "connect to cluster: %v\n", err)
os.Exit(exitUnreachable)
}
defer nc.Close()
js, err := jetstream.New(nc)
if err != nil {
fmt.Fprintf(os.Stderr, "get JetStream context: %v\n", err)
os.Exit(exitUnreachable)
}
// Background: this runs at CLI top level, where there is no request to
// inherit a deadline from and the process exits after the one command.
ctx := context.Background()
reports, err := kvutil.AuditBucketReplicas(ctx, js)
if err != nil {
fmt.Fprintf(os.Stderr, "audit KV bucket replicas: %v\n", err)
os.Exit(exitUnreachable)
}
if repair {
repairReplicaReports(ctx, js, reports, os.Stderr)
}
if err := writeReplicaReport(os.Stdout, reports, asJSON); err != nil {
fmt.Fprintf(os.Stderr, "write report: %v\n", err)
os.Exit(exitUnreachable)
}
os.Exit(replicaReportExit(reports))
}
// Exit statuses, documented in the command's own help because a deployment gate
// reads them rather than the output.
const (
exitHealthy = 0
// exitUnderReplicated is the repairable problem: the buckets serve, but on
// fewer nodes than the cluster has.
exitUnderReplicated = 1
// exitUnreachable is the problem repair cannot touch — a bucket that cannot
// accept a write, or a report that could not be produced at all.
exitUnreachable = 2
)
// replicaReportExit is the exit status for a set of reports.
//
// No quorum outranks under-replicated because they need different responses: one
// is fixed by raising a count, the other by finding out why a node is not
// answering. A bucket that is both should send the operator to the node.
func replicaReportExit(reports []kvutil.BucketReport) int {
status := exitHealthy
for _, r := range reports {
if !r.HasQuorum() {
return exitUnreachable
}
if r.UnderReplicated() {
status = exitUnderReplicated
}
}
return status
}
// repairReplicaReports raises every under-replicated bucket and updates reports
// in place for the ones that were raised.
//
// One bucket that cannot be placed must not stop the others being repaired, so a
// failure is named on errOut and the sweep continues. The bucket it failed on
// stays UNDER-REPLICATED in the report, which is already what makes the exit
// status non-zero, so nothing is lost by not returning an error here.
func repairReplicaReports(ctx context.Context, js jetstream.JetStream, reports []kvutil.BucketReport, errOut io.Writer) {
for i, r := range reports {
if !r.UnderReplicated() {
continue
}
if err := kvutil.RaiseBucketReplicas(ctx, js, r.Bucket, r.Want); err != nil {
fmt.Fprintf(errOut, "repair %s: %v\n", r.Bucket, err)
continue
}
reports[i].Replicas = r.Want
}
}
// underReplicatedCount is how many buckets are below the cluster's replica count.
func underReplicatedCount(reports []kvutil.BucketReport) int {
under := 0
for _, r := range reports {
if r.UnderReplicated() {
under++
}
}
return under
}
// noQuorumCount is how many buckets cannot currently accept a write.
func noQuorumCount(reports []kvutil.BucketReport) int {
stuck := 0
for _, r := range reports {
if !r.HasQuorum() {
stuck++
}
}
return stuck
}
// replicaStatus is the one word for a bucket's condition.
//
// No quorum wins over under-replicated when both are true, matching the exit
// status: a bucket that cannot serve is the more urgent of the two and is not
// what --repair is for.
func replicaStatus(r kvutil.BucketReport) string {
switch {
case !r.HasQuorum():
return "NO-QUORUM"
case r.UnderReplicated():
return "UNDER-REPLICATED"
default:
return "ok"
}
}
// writeReplicaReport renders reports as a table, or as JSON when asJSON.
func writeReplicaReport(out io.Writer, reports []kvutil.BucketReport, asJSON bool) error {
if asJSON {
return json.NewEncoder(out).Encode(reports)
}
w := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0)
fmt.Fprintln(w, "BUCKET\tREPLICAS\tWANT\tONLINE\tSTATUS\tLEADER\tHELD BY")
for _, r := range reports {
fmt.Fprintf(w, "%s\t%d\t%d\t%d\t%s\t%s\t%s\n",
r.Bucket, r.Replicas, r.Want, r.Online, replicaStatus(r), r.Leader, strings.Join(r.Peers, ","))
}
if err := w.Flush(); err != nil {
return err
}
_, err := fmt.Fprintf(out, "\n%d buckets, %d under-replicated, %d without quorum\n",
len(reports), underReplicatedCount(reports), noQuorumCount(reports))
return err
}
cmd/spinifex/cmd/admin_kv_replicas_test.go
+133
View File
@@ -0,0 +1,133 @@
//test:in-package — it tests this command's unexported rendering and repair
// helpers, which exist only to be exercised without a live cluster.
package cmd
import (
"bytes"
"encoding/json"
"strings"
"testing"
"github.com/mulgadc/spinifex/spinifex/clustersize"
"github.com/mulgadc/spinifex/spinifex/kvutil"
"github.com/mulgadc/spinifex/spinifex/testutil"
"github.com/nats-io/nats.go/jetstream"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestWriteReplicaReport_NamesEveryUnderReplicatedBucket(t *testing.T) {
reports := []kvutil.BucketReport{
{Bucket: "healthy", Replicas: 3, Want: 3, Online: 3, Leader: "a", Peers: []string{"a", "b", "c"}},
{Bucket: "short", Replicas: 1, Want: 3, Online: 1, Leader: "a", Peers: []string{"a"}},
{Bucket: "stuck", Replicas: 3, Want: 3, Online: 1, Peers: []string{"a", "b", "c"}},
}
var out bytes.Buffer
require.NoError(t, writeReplicaReport(&out, reports, false))
got := out.String()
assert.Contains(t, got, "BUCKET")
assert.Contains(t, got, "HELD BY")
assert.Regexp(t, `healthy\s+3\s+3\s+3\s+ok\s+a\s+a,b,c`, got)
assert.Regexp(t, `short\s+1\s+3\s+1\s+UNDER-REPLICATED\s+a\s+a`, got)
// A bucket that cannot serve reads as NO-QUORUM even though its configured
// count is right, because raising a count is not the response to it.
assert.Regexp(t, `stuck\s+3\s+3\s+1\s+NO-QUORUM`, got)
assert.Contains(t, got, "3 buckets, 1 under-replicated, 1 without quorum")
}
// TestReplicaReportExit pins the three exit classes a deployment gate reads. They
// are separate because the responses are: a raise fixes an under-replicated
// bucket, and nothing the command can do fixes one that will not answer.
func TestReplicaReportExit(t *testing.T) {
healthy := kvutil.BucketReport{Bucket: "a", Replicas: 3, Want: 3, Online: 3}
short := kvutil.BucketReport{Bucket: "b", Replicas: 1, Want: 3, Online: 1}
stuck := kvutil.BucketReport{Bucket: "c", Replicas: 3, Want: 3, Online: 1}
assert.Equal(t, exitHealthy, replicaReportExit(nil))
assert.Equal(t, exitHealthy, replicaReportExit([]kvutil.BucketReport{healthy}))
assert.Equal(t, exitUnderReplicated, replicaReportExit([]kvutil.BucketReport{healthy, short}))
assert.Equal(t, exitUnreachable, replicaReportExit([]kvutil.BucketReport{healthy, stuck}))
// Both problems at once sends the operator to the node rather than to --repair.
assert.Equal(t, exitUnreachable, replicaReportExit([]kvutil.BucketReport{short, stuck}))
}
// TestWriteReplicaReport_JSONIsWhatTheE2ESuiteDecodes keeps the machine-readable
// form honest: the e2e suite and anything gating a deployment parse these field
// names, so a rename here breaks them silently.
func TestWriteReplicaReport_JSONIsWhatTheE2ESuiteDecodes(t *testing.T) {
var out bytes.Buffer
require.NoError(t, writeReplicaReport(&out, []kvutil.BucketReport{
{Bucket: "short", Replicas: 1, Want: 3, Cluster: "spinifex", Online: 1, Leader: "a", Peers: []string{"a"}},
}, true))
assert.Contains(t, out.String(), `"bucket":"short"`)
var decoded []kvutil.BucketReport
require.NoError(t, json.Unmarshal(out.Bytes(), &decoded))
require.Len(t, decoded, 1)
assert.Equal(t, "short", decoded[0].Bucket)
assert.Equal(t, 1, decoded[0].Replicas)
assert.Equal(t, 3, decoded[0].Want)
assert.Equal(t, 1, decoded[0].Online)
assert.Equal(t, "a", decoded[0].Leader)
assert.True(t, decoded[0].UnderReplicated())
assert.True(t, decoded[0].HasQuorum())
}
func TestUnderReplicatedCount(t *testing.T) {
assert.Equal(t, 0, underReplicatedCount(nil))
assert.Equal(t, 2, underReplicatedCount([]kvutil.BucketReport{
{Bucket: "a", Replicas: 1, Want: 3},
{Bucket: "b", Replicas: 3, Want: 3},
{Bucket: "c", Replicas: 2, Want: 3},
}))
}
// TestRepairReplicaReports_KeepsGoingPastABucketItCannotPlace is the behaviour
// that makes --repair usable mid-growth: the buckets it can raise are raised,
// and the one it cannot is named rather than aborting the run.
func TestRepairReplicaReports_KeepsGoingPastABucketItCannotPlace(t *testing.T) {
_, nc, js := testutil.StartTestJetStream(t)
defer nc.Close()
for _, b := range []string{"repair-alpha", "repair-beta"} {
_, err := js.CreateKeyValue(t.Context(), jetstream.KeyValueConfig{Bucket: b, History: 1})
require.NoError(t, err)
}
// One embedded server cannot hold two replicas, so every raise fails — and
// every bucket must still have been attempted and named.
reports := []kvutil.BucketReport{
{Bucket: "repair-alpha", Replicas: 1, Want: 2},
{Bucket: "repair-beta", Replicas: 1, Want: 2},
}
var errOut bytes.Buffer
repairReplicaReports(t.Context(), js, reports, &errOut)
assert.Contains(t, errOut.String(), "repair repair-alpha")
assert.Contains(t, errOut.String(), "repair repair-beta")
assert.Equal(t, 2, underReplicatedCount(reports), "a failed raise must not be reported as repaired")
}
// TestRepairReplicaReports_RaisesWhatItCan covers the success side against a real
// server, so the in-place report update is proved rather than assumed.
func TestRepairReplicaReports_RaisesWhatItCan(t *testing.T) {
_, nc, js := testutil.StartTestJetStream(t)
defer nc.Close()
clustersize.DeclareForTest(t, 1)
_, err := js.CreateKeyValue(t.Context(), jetstream.KeyValueConfig{Bucket: "repair-ok", History: 1})
require.NoError(t, err)
// Want 1 against a bucket already at 1: the raise is a no-op that must still
// leave the report reading as healthy rather than as repaired-and-wrong.
reports := []kvutil.BucketReport{{Bucket: "repair-ok", Replicas: 1, Want: 1}}
var errOut bytes.Buffer
repairReplicaReports(t.Context(), js, reports, &errOut)
assert.Empty(t, strings.TrimSpace(errOut.String()))
assert.Equal(t, 0, underReplicatedCount(reports))
}
cmd/spinifex/cmd/admin_node_count_test.go
+39
View File
@@ -0,0 +1,39 @@
//test:in-package — validateNodeCount is an unexported guard on this command's
// flags, and exporting it would widen the package's API for a test's convenience.
package cmd
import (
"strings"
"testing"
)
// TestValidateNodeCount pins the supported cluster sizes. Two is rejected rather
// than warned about: on two servers no layer of the stack has a majority to lose,
// and the control plane's own state has no replica count that both survives a node
// going away and still accepts writes.
func TestValidateNodeCount(t *testing.T) {
for _, nodes := range []int{1, 3, 4, 5, 9} {
if err := validateNodeCount(nodes); err != nil {
t.Errorf("validateNodeCount(%d) = %v, want nil", nodes, err)
}
}
for _, nodes := range []int{0, -1} {
if err := validateNodeCount(nodes); err == nil {
t.Errorf("validateNodeCount(%d) returned no error", nodes)
}
}
err := validateNodeCount(2)
if err == nil {
t.Fatal("validateNodeCount(2) returned no error")
}
// The message has to say what to do instead, because the operator typed a
// number and needs to know which one to type next.
for _, want := range []string{"2", "1", "3"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("error %q does not mention %s", err, want)
}
}
}
cmd/spinifex/cmd/admin_ochre.go
+8 -8
View File
@@ -854,7 +854,7 @@ func runOchreWeightsStage(cmd *cobra.Command, _ []string) {
ochreExit(1)
return
}
weightsStore := gateway_bedrock.NewWeightsStore(js, len(appConfig.Nodes))
weightsStore := gateway_bedrock.NewWeightsStore(js)
store := objectstore.NewS3ObjectStoreFromConfig(node.Predastore.Host, node.Predastore.Region, node.Predastore.AccessKey, node.Predastore.SecretKey)
provider := ebsprovider.NewNATSProvider(nc, imageImportTimeout)
@@ -930,7 +930,7 @@ func runOchreWeightsList(_ *cobra.Command, _ []string) {
ochreExit(1)
return
}
weightsStore := gateway_bedrock.NewWeightsStore(js, len(appConfig.Nodes))
weightsStore := gateway_bedrock.NewWeightsStore(js)
store := objectstore.NewS3ObjectStoreFromConfig(node.Predastore.Host, node.Predastore.Region, node.Predastore.AccessKey, node.Predastore.SecretKey)
checkSnapshotLive := newWeightsSnapshotChecker(store, node.Predastore.Bucket)
@@ -967,7 +967,7 @@ func removeWeights(ctx context.Context, weightsStore *gateway_bedrock.WeightsSto
func runOchreWeightsRemove(cmd *cobra.Command, _ []string) {
modelID, _ := cmd.Flags().GetString("model-id")
appConfig, nc, err := loadConfigAndConnectFn()
_, nc, err := loadConfigAndConnectFn()
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
ochreExit(1)
@@ -981,7 +981,7 @@ func runOchreWeightsRemove(cmd *cobra.Command, _ []string) {
ochreExit(1)
return
}
weightsStore := gateway_bedrock.NewWeightsStore(js, len(appConfig.Nodes))
weightsStore := gateway_bedrock.NewWeightsStore(js)
entry, err := removeWeights(context.Background(), weightsStore, modelID)
if err != nil {
@@ -1014,7 +1014,7 @@ func resolveHFToken(ctx context.Context, cmd *cobra.Command, cfg *config.Cluster
if err != nil {
return ""
}
credStore := gateway_bedrock.NewCredentialStore(js, masterKey, len(cfg.Nodes), nil)
credStore := gateway_bedrock.NewCredentialStore(js, masterKey, nil)
token, ok, err := credStore.Resolve(ctx, utils.GlobalAccountID, vendorHuggingFace)
if err != nil || !ok {
return ""
@@ -1088,7 +1088,7 @@ func ochreCredentialsStore() (*gateway_bedrock.CredentialStore, func(), error) {
nc.Close()
return nil, nil, fmt.Errorf("jetstream context: %w", err)
}
return gateway_bedrock.NewCredentialStore(js, masterKey, len(cfg.Nodes), nil), func() { nc.Close() }, nil
return gateway_bedrock.NewCredentialStore(js, masterKey, nil), func() { nc.Close() }, nil
}
func runOchreCredentialsSet(cmd *cobra.Command, _ []string) {
@@ -1186,7 +1186,7 @@ func init() {
// ochreAccessStore connects to the cluster and returns the grant store along
// with a cleanup that closes the connection.
func ochreAccessStore() (*gateway_bedrock.ModelAccessStore, func(), error) {
cfg, nc, err := loadConfigAndConnectFn()
_, nc, err := loadConfigAndConnectFn()
if err != nil {
return nil, nil, fmt.Errorf("connect to cluster: %w", err)
}
@@ -1195,7 +1195,7 @@ func ochreAccessStore() (*gateway_bedrock.ModelAccessStore, func(), error) {
nc.Close()
return nil, nil, fmt.Errorf("jetstream context: %w", err)
}
return gateway_bedrock.NewModelAccessStore(js, len(cfg.Nodes)), func() { nc.Close() }, nil
return gateway_bedrock.NewModelAccessStore(js), func() { nc.Close() }, nil
}
// ochreTargetModels resolves the --model-id / --all-models pair into the model
cmd/spinifex/cmd/admin_ochre_pull_test.go
+2 -2
View File
@@ -470,7 +470,7 @@ func TestResolveHFToken_FallsBackToStoredPlatformCredential(t *testing.T) {
require.NoError(t, os.MkdirAll(filepath.Join(baseDir, "config"), 0750))
require.NoError(t, handlers_iam.SaveMasterKey(filepath.Join(baseDir, "config", "master.key"), masterKey))
credStore := gateway_bedrock.NewCredentialStore(js, masterKey, 1, nil)
credStore := gateway_bedrock.NewCredentialStore(js, masterKey, nil)
require.NoError(t, credStore.PutCredential(context.Background(), utils.GlobalAccountID, vendorHuggingFace, "hf_stored_token"))
cmd := newOchreWeightsPullTestCmd(t, selfHostModelID, testHFRepo, "main", "", "")
@@ -621,7 +621,7 @@ func TestRunOchreCredentialsSet_RoundTripsAndDefaultsToPlatformAccount(t *testin
defer verifyConn.Close()
verifyJS := testutil.NewJetStream(t, verifyConn)
credStore := gateway_bedrock.NewCredentialStore(verifyJS, masterKey, 1, nil)
credStore := gateway_bedrock.NewCredentialStore(verifyJS, masterKey, nil)
token, ok, err := credStore.Resolve(context.Background(), utils.GlobalAccountID, vendorHuggingFace)
require.NoError(t, err)
require.True(t, ok)
cmd/spinifex/cmd/admin_ochre_test.go
+4 -4
View File
@@ -42,7 +42,7 @@ const (
func newWeightsStoreForTest(t *testing.T) *gateway_bedrock.WeightsStore {
t.Helper()
_, nc, _ := testutil.StartTestJetStream(t)
return gateway_bedrock.NewWeightsStore(testutil.NewJetStream(t, nc), 1)
return gateway_bedrock.NewWeightsStore(testutil.NewJetStream(t, nc))
}
// explodingObjectStore fails the test immediately if any of its methods is
@@ -775,7 +775,7 @@ func TestRunOchreWeightsList_PrintsStagedModels(t *testing.T) {
}
stubConnect(t, cfg, nc, nil)
seedStore := gateway_bedrock.NewWeightsStore(js, 1)
seedStore := gateway_bedrock.NewWeightsStore(js)
require.NoError(t, seedStore.PutWeights(context.Background(), selfHostModelID, "s3://models/llama-3.2-1b/", "snap-0001"))
out := captureStdout(t, func() { runOchreWeightsList(nil, nil) })
@@ -822,7 +822,7 @@ func TestRunOchreWeightsRemove_RoundTripPrintsSnapshotAndDropsEntry(t *testing.T
ns, nc, js := testutil.StartTestJetStream(t)
stubConnect(t, fakeClusterConfig(), nc, nil)
seedStore := gateway_bedrock.NewWeightsStore(js, 1)
seedStore := gateway_bedrock.NewWeightsStore(js)
require.NoError(t, seedStore.PutWeights(context.Background(), selfHostModelID, "s3://models/llama-3.2-1b/", "snap-0001"))
cmd := newOchreWeightsRemoveTestCmd(t, selfHostModelID)
@@ -835,7 +835,7 @@ func TestRunOchreWeightsRemove_RoundTripPrintsSnapshotAndDropsEntry(t *testing.T
verifyNC, err := nats.Connect(ns.ClientURL())
require.NoError(t, err)
t.Cleanup(func() { verifyNC.Close() })
verifyStore := gateway_bedrock.NewWeightsStore(testutil.NewJetStream(t, verifyNC), 1)
verifyStore := gateway_bedrock.NewWeightsStore(testutil.NewJetStream(t, verifyNC))
_, ok, err := verifyStore.GetWeights(context.Background(), selfHostModelID)
require.NoError(t, err)
cmd/spinifex/cmd/admin_test.go
+242 -6
View File
@@ -2,6 +2,8 @@ package cmd
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"net/netip"
@@ -9,11 +11,13 @@ import (
"path/filepath"
"strings"
"testing"
"time"
"github.com/mulgadc/northstar/pkg/backend"
nsconfig "github.com/mulgadc/northstar/pkg/config"
"github.com/mulgadc/spinifex/spinifex/admin"
"github.com/mulgadc/spinifex/spinifex/config"
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
"github.com/mulgadc/spinifex/spinifex/formation"
handlers_dns "github.com/mulgadc/spinifex/spinifex/handlers/dns"
handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
@@ -734,21 +738,45 @@ func TestSpinifexTomlTemplate_ExternalPoolStaticSource(t *testing.T) {
assert.Contains(t, content, `gateway = "192.168.1.1"`)
}
// Formation joiners must receive PoolBindBridge from the init node so the
// Formation joiners must receive BindBridge from the init node so the
// cluster-wide config reaches every chassis intact.
func TestApplyNetworkConfig_PropagatesPoolBindBridge(t *testing.T) {
settings := &admin.ConfigSettings{}
nc := &formation.NetworkConfig{
ExternalMode: "pool",
PoolName: "wan",
PoolSource: "dhcp",
PoolBindBridge: "br-wan",
PoolPrefixLen: 24,
ExternalMode: "pool",
Pools: []admin.PoolData{{
Name: "wan", Source: "dhcp", BindBridge: "br-wan", PrefixLen: 24,
}},
}
applyNetworkConfig(settings, nc)
require.Len(t, settings.Pools, 1)
assert.Equal(t, "dhcp", settings.Pools[0].Source)
assert.Equal(t, "br-wan", settings.Pools[0].BindBridge)
assert.Empty(t, settings.BridgeMode, "bridged modes stay auto-detected, only nat is pinned")
}
// Routed NAT renders two pools, and the transit one carries a gateway-LRP range
// that pool mode has no equivalent for. A joiner that received only the first,
// or lost the range, would allocate VPC gateway addresses from nowhere.
func TestApplyNetworkConfig_PropagatesEveryPool(t *testing.T) {
settings := &admin.ConfigSettings{}
nc := &formation.NetworkConfig{
ExternalMode: "nat",
Pools: []admin.PoolData{
{Name: "nat-transit", Gateway: "100.127.0.1", PrefixLen: 24,
GwLrpRangeStart: "100.127.0.16", GwLrpRangeEnd: "100.127.0.254"},
{Name: "wan", Source: "static", Start: "192.168.1.150", End: "192.168.1.200",
Gateway: "192.168.1.1", PrefixLen: 24},
},
}
applyNetworkConfig(settings, nc)
require.Len(t, settings.Pools, 2)
assert.Equal(t, "nat", settings.BridgeMode,
"a joiner that loses bridge_mode falls back to autodetection for a mode init pinned")
assert.Equal(t, "nat-transit", settings.Pools[0].Name)
assert.Equal(t, "100.127.0.16", settings.Pools[0].GwLrpRangeStart)
assert.Equal(t, "100.127.0.254", settings.Pools[0].GwLrpRangeEnd)
assert.Equal(t, "wan", settings.Pools[1].Name)
}
func renderSingleNodePredastore(t *testing.T, settings admin.ConfigSettings) string {
@@ -1288,6 +1316,214 @@ func TestAMIVolumeSizeGiB(t *testing.T) {
}
}
func TestValidateChecksumFlags(t *testing.T) {
tests := []struct {
name string
checksum string
unset bool
file string
skipVerify bool
wantErr string
}{
{name: "no checksum", unset: true, file: "img.raw"},
{name: "checksum with file", checksum: "SHA256SUMS", file: "img.raw"},
{name: "empty checksum value", checksum: "", file: "img.raw", wantErr: "--checksum requires a sums file path"},
{name: "blank checksum value", checksum: " ", file: "img.raw", wantErr: "--checksum requires a sums file path"},
{name: "checksum without file", checksum: "SHA256SUMS", wantErr: "--checksum requires --file"},
{name: "checksum with skip-verify", checksum: "SHA256SUMS", file: "img.raw", skipVerify: true,
wantErr: "mutually exclusive"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := validateChecksumFlags(tt.checksum, !tt.unset, tt.file, tt.skipVerify)
if tt.wantErr == "" {
assert.NoError(t, err)
return
}
require.Error(t, err)
assert.Contains(t, err.Error(), tt.wantErr)
})
}
}
func TestSourceDigestMode(t *testing.T) {
tests := []struct {
name string
localFile string
checksum string
skipVerify bool
want string
}{
{name: "file with checksum", localFile: "img.raw", checksum: "SHA256SUMS", want: ebsmetadata.DigestOperator},
{name: "catalog download verified", want: ebsmetadata.DigestCatalog},
{name: "catalog download skip-verify", skipVerify: true, want: ebsmetadata.DigestUnverified},
{name: "file without checksum", localFile: "img.raw", want: ebsmetadata.DigestUnverified},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.want, sourceDigestMode(tt.localFile, tt.checksum, tt.skipVerify))
})
}
}
func TestImportSourceResolveDigest(t *testing.T) {
imgBytes := []byte("import-source-fixture")
sum := sha256.Sum256(imgBytes)
imgHex := hex.EncodeToString(sum[:])
dir := t.TempDir()
img := filepath.Join(dir, "image.raw")
require.NoError(t, os.WriteFile(img, imgBytes, 0o600))
sums := filepath.Join(dir, "SHA256SUMS")
t.Run("operator verified", func(t *testing.T) {
var out, errOut bytes.Buffer
got, err := importSource{
imageFile: img, localFile: img,
checksumPath: sums, checksumAlgo: "sha256", expectedDigest: imgHex,
}.resolveDigest(&out, &errOut)
require.NoError(t, err)
assert.Equal(t, ebsmetadata.ImageDigest{
Algorithm: "sha256", Value: imgHex, Verification: ebsmetadata.DigestOperator,
Source: "SHA256SUMS", Filename: "image.raw",
}, got)
assert.Contains(t, out.String(), "Verified image checksum (sha256)")
assert.Empty(t, errOut.String())
})
t.Run("operator mismatch names the file and sums file", func(t *testing.T) {
var out, errOut bytes.Buffer
_, err := importSource{
imageFile: img, localFile: img,
checksumPath: sums, checksumAlgo: "sha256", expectedDigest: strings.Repeat("0", 64),
}.resolveDigest(&out, &errOut)
require.ErrorIs(t, err, utils.ErrChecksumMismatch)
assert.Contains(t, errOut.String(), "file: "+img)
assert.Contains(t, errOut.String(), "checksum: "+sums)
assert.Contains(t, errOut.String(), imgHex)
assert.FileExists(t, img, "the operator's file must be left in place")
})
t.Run("local file without checksum is recorded unverified", func(t *testing.T) {
var out, errOut bytes.Buffer
got, err := importSource{imageFile: img, localFile: img}.resolveDigest(&out, &errOut)
require.NoError(t, err)
assert.Equal(t, ebsmetadata.ImageDigest{
Algorithm: "sha256", Value: imgHex, Verification: ebsmetadata.DigestUnverified, Filename: "image.raw",
}, got)
assert.Equal(t, "Importing "+img+" without checksum validation (pass --checksum <sums-file> to verify)\n",
errOut.String())
})
t.Run("catalog skip-verify is recorded unverified", func(t *testing.T) {
var out, errOut bytes.Buffer
got, err := importSource{imageFile: img, imageName: "debian-13-x86_64", skipVerify: true}.resolveDigest(&out, &errOut)
require.NoError(t, err)
assert.Equal(t, ebsmetadata.DigestUnverified, got.Verification)
assert.Equal(t, imgHex, got.Value)
assert.Contains(t, errOut.String(), "--skip-verify set")
})
t.Run("catalog entry without checksum is refused", func(t *testing.T) {
var out, errOut bytes.Buffer
_, err := importSource{imageFile: img, imageName: "debian-13-x86_64"}.resolveDigest(&out, &errOut)
require.Error(t, err)
assert.Contains(t, errOut.String(), "missing Checksum/ChecksumDigest/ChecksumType")
})
// Oracle publishes its digests inline on a web page and ships no sums file,
// so a pinned digest is the only way its entries can import verified.
t.Run("catalog pinned digest verifies without fetching a sums file", func(t *testing.T) {
var out, errOut bytes.Buffer
got, err := importSource{
imageFile: img, imageName: "oracle-10.1-x86_64",
image: utils.Images{ChecksumType: "sha256", ChecksumDigest: imgHex},
}.resolveDigest(&out, &errOut)
require.NoError(t, err)
assert.Equal(t, ebsmetadata.ImageDigest{
Algorithm: "sha256", Value: imgHex, Source: catalogPinnedDigestSource,
Verification: ebsmetadata.DigestCatalog, Filename: "image.raw",
}, got)
assert.Contains(t, out.String(), "pinned catalog digest")
})
t.Run("catalog pinned digest mismatch is refused and names the url", func(t *testing.T) {
var out, errOut bytes.Buffer
wrong := strings.Repeat("a", 64)
_, err := importSource{
imageFile: img, imageName: "oracle-10.1-x86_64",
image: utils.Images{ChecksumType: "sha256", ChecksumDigest: wrong, URL: "https://example.invalid/ol.qcow2"},
}.resolveDigest(&out, &errOut)
require.ErrorIs(t, err, utils.ErrChecksumMismatch)
assert.Contains(t, errOut.String(), catalogPinnedDigestSource)
assert.Contains(t, errOut.String(), "https://example.invalid/ol.qcow2")
assert.FileExists(t, img, "the downloaded file must be left in place")
})
t.Run("unreadable image fails the unverified hash", func(t *testing.T) {
var out, errOut bytes.Buffer
missing := filepath.Join(dir, "missing.raw")
_, err := importSource{imageFile: missing, localFile: missing}.resolveDigest(&out, &errOut)
require.Error(t, err)
assert.Contains(t, errOut.String(), "Could not hash image")
})
}
func TestImagesDescribeCmd_FlagSchema(t *testing.T) {
imageIDFlag := imagesDescribeCmd.Flags().Lookup("image-id")
require.NotNil(t, imageIDFlag, "--image-id must be defined")
assert.Equal(t, []string{"true"}, imageIDFlag.Annotations[cobraRequiredAnnotation],
"--image-id must be marked required")
}
func TestPrintAMIDescription(t *testing.T) {
base := ebsmetadata.AMI{
ImageID: "ami-1",
Name: "ami-debian-13-x86_64",
ImageOwnerAlias: "system",
CreationDate: time.Date(2026, 9, 23, 1, 2, 3, 0, time.UTC),
BootMode: "uefi",
Architecture: "x86_64",
}
t.Run("digest recorded", func(t *testing.T) {
meta := base
meta.SourceDigest = &ebsmetadata.ImageDigest{
Algorithm: "sha512", Value: "abc123", Verification: ebsmetadata.DigestOperator,
Source: "SHA512SUMS", Filename: "debian-13-generic-amd64.tar.xz",
}
var buf bytes.Buffer
printAMIDescription(&buf, meta)
out := buf.String()
assert.Contains(t, out, "Image ID: ami-1")
assert.Contains(t, out, "Created: 2026-09-23T01:02:03Z")
assert.Contains(t, out, "Boot mode: uefi")
assert.Contains(t, out, "Source digest: sha512:abc123")
assert.Contains(t, out, "Verification: operator (SHA512SUMS)")
assert.Contains(t, out, "Source file: debian-13-generic-amd64.tar.xz")
})
t.Run("unverified digest has no source", func(t *testing.T) {
meta := base
meta.SourceDigest = &ebsmetadata.ImageDigest{
Algorithm: "sha256", Value: "def456", Verification: ebsmetadata.DigestUnverified,
Filename: "image.raw",
}
var buf bytes.Buffer
printAMIDescription(&buf, meta)
assert.Contains(t, buf.String(), "Verification: unverified\n")
})
t.Run("digest not recorded", func(t *testing.T) {
var buf bytes.Buffer
printAMIDescription(&buf, base)
out := buf.String()
assert.Contains(t, out, "Source digest: not recorded (imported before digest recording)")
assert.NotContains(t, out, "Verification:")
})
}
func TestCheckJoinPreconditions(t *testing.T) {
withFakeProc(t, nil)
noStore := filepath.Join(t.TempDir(), "nats", "jetstream")
cmd/spinifex/cmd/cluster.go
+1 -1
View File
@@ -167,7 +167,7 @@ func runClusterShutdown(cmd *cobra.Command, args []string) {
fmt.Printf("Timeout per phase: %s\n\n", timeout)
// Write cluster shutdown marker to KV
jsm, err := daemon.NewJetStreamManager(nc, 1)
jsm, err := daemon.NewJetStreamManager(nc)
if err != nil {
fmt.Fprintf(os.Stderr, "Warning: failed to create JetStream manager: %v\n", err)
} else {
cmd/spinifex/cmd/nats_server_name_test.go
+44
View File
@@ -0,0 +1,44 @@
//test:in-package — the embedded nats.conf template it checks is unexported,
// and exporting it would widen the package's API for a test's convenience.
package cmd
import (
"strings"
"testing"
natssvc "github.com/mulgadc/spinifex/spinifex/services/nats"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestNATSConfServerNameMatchesTheHelper ties the rendered server_name to
// nats.ServerName. JetStream reports stream peers by server name, so anything
// mapping a replica back to the node holding it reads the prefix from that
// helper — and a template that drifted from it would break the mapping silently,
// reporting a healthy cluster as one whose buckets name no node at all.
func TestNATSConfServerNameMatchesTheHelper(t *testing.T) {
var line string
for l := range strings.SplitSeq(natsConfTemplate, "\n") {
if strings.HasPrefix(strings.TrimSpace(l), "server_name:") {
line = strings.TrimSpace(l)
break
}
}
require.NotEmpty(t, line, "nats.conf no longer sets server_name")
got := strings.TrimSpace(strings.TrimPrefix(line, "server_name:"))
assert.Equal(t, natssvc.ServerName("{{.Node}}"), got)
node, ok := natssvc.NodeFromServerName(natssvc.ServerName("node2"))
assert.True(t, ok)
assert.Equal(t, "node2", node)
_, ok = natssvc.NodeFromServerName("node2")
assert.False(t, ok, "a bare node name is not a server name")
// The bare prefix is a name we could not have generated, and mapping it to the
// empty node would put a phantom peer into anything counting nodes.
_, ok = natssvc.NodeFromServerName(natssvc.ServerNamePrefix)
assert.False(t, ok, "the prefix with no node after it is not a server name")
}
cmd/spinifex/cmd/service.go
+3
View File
@@ -1025,12 +1025,15 @@ var vpcdStartCmd = &cobra.Command{
AZ: nodeConfig.AZ,
NorthstarBaseDomain: handlers_dns.ResolveBaseDomain(&nodeConfig),
NorthstarInternalDomain: handlers_dns.ResolveInternalDomain(&nodeConfig),
ServicesDomain: clusterConfig.AWS.ServicesDomain,
ResolverNameservers: handlers_dns.ResolverNameserverIPs(clusterConfig),
NATExemptCIDRs: clusterConfig.Network.NATExemptCIDRs,
IPSecEnabled: clusterConfig.Network.IPSecEnabled,
UnderlayMTU: clusterConfig.Network.UnderlayMTU,
BlockedWANPorts: clusterConfig.Network.ResolvedBlockedWANPorts(),
EgressBlockExemptVPCs: clusterConfig.Network.EgressBlockExemptVPCs,
IMDSHostMetaIP: clusterConfig.Network.IMDSHostMetaIP,
IMDSHostDNSIP: clusterConfig.Network.IMDSHostDNSIP,
})
if err != nil {
return fmt.Errorf("create vpcd service: %w", err)
cmd/spinifex/cmd/templates/spinifex.toml
+1 -1
View File
@@ -10,7 +10,7 @@ email = "{{.OperatorEmail}}"
[aws]
region = "{{.Region}}"
internal_suffix = "spinifex.internal"
services_domain = "services.internal"
[network]
ipsec_enabled = {{.IPSecEnabled}}
docs/COMMANDS.md
+2 -1
View File
@@ -166,8 +166,9 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin images import` | `--name`, `--file`, `--distro`, `--version`, `--arch`, `--platform`, `--boot-mode` (bios/uefi/uefi-preferred), `--tag`, `--force`, `--skip-verify`, `--ami-name` (override the registered `DescribeImages` name, which otherwise defaults to `ami-{distro}-{version}-{arch}`; used for locally-built appliances such as `--ami-name spinifex-eks-node`), `--tmp-dir` (default: `/tmp`) | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, verify the SHA-256/SHA-512 digest, and inherit `BootMode` from the catalog entry. `--boot-mode` overrides the catalog value when set. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports skip checksum verification (operator-supplied media is outside Spinifex's trust boundary, the skip is logged at INFO for audit) and require an explicit `--boot-mode` because there is no catalog metadata to inherit from. `--skip-verify` bypasses verification for catalog imports and emits a WARN slog + stderr notice; use only for debugging or when upstream mirrors are confirmed-broken. Import never modifies the image: the bytes written to storage are the bytes that were verified. A guest that needs to trust the deployment CA for in-guest SDK-over-TLS calls fetches it at runtime from `http://169.254.169.254/spinifex/ca.pem` (see the IMDS section), so a CA rotation reaches guests without re-importing anything. |
| `spx admin images import` | `--name`, `--file`, `--distro`, `--version`, `--arch`, `--platform`, `--boot-mode` (bios/uefi/uefi-preferred), `--tag`, `--force`, `--skip-verify`, `--checksum`, `--ami-name` (override the registered `DescribeImages` name, which otherwise defaults to `ami-{distro}-{version}-{arch}`; used for locally-built appliances such as `--ami-name spinifex-eks-node`), `--tmp-dir` (default: `/tmp`) | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, verify the SHA-256/SHA-512 digest, and inherit `BootMode` from the catalog entry. `--boot-mode` overrides the catalog value when set. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports require an explicit `--boot-mode` because there is no catalog metadata to inherit from, and are verified only when `--checksum <sums-file>` is passed. `--checksum` takes the sums file the image was published with (`SHA256SUMS`, `SHA512SUMS`, `<image>.sha256`/`.sha512`, coreutils, BSD or bare-digest format), looks up the entry for the image's basename, and infers sha256 or sha512 from the digest length. A mismatch exits 1 before anything is written to storage and leaves the file in place. `--checksum` proves only that the image matches the sums file supplied with it, not that either is authentic; verify the sums file's vendor signature separately. `--checksum` is refused without `--file` or with `--skip-verify`. A `--file` import without `--checksum` prints `Importing <file> without checksum validation` to stderr and proceeds; `--name --file` does not fall back to the catalog digest. Every import records the source artifact's digest on the AMI with who verified it (`operator`, `catalog` or `unverified`, the unverified cases hashed as sha256), prints it on success, and shows it in `spx admin images describe`. The digest is of the file as supplied, not of the stored volume. `--skip-verify` bypasses verification for catalog imports and prints a warning to stderr; use only for debugging or when upstream mirrors are confirmed-broken. Import never modifies the image: the bytes written to storage are the bytes that were verified. A guest that needs to trust the deployment CA for in-guest SDK-over-TLS calls fetches it at runtime from `http://169.254.169.254/spinifex/ca.pem` (see the IMDS section), so a CA rotation reaches guests without re-importing anything. |
| `spx admin images list` | — | Lists available OS images that can be imported or downloaded |
| `spx admin images describe` | `--image-id` (required) | Read-only. Prints the AMI's name, owner, creation date, boot mode and architecture, then its source digest: algorithm and value, verification (`operator` with the sums file name, `catalog` with the sums URL, or `unverified`), and the source file name. AMIs imported before digest recording print `Source digest: not recorded`. The digest is not exposed through `DescribeImages`. |
| `spx admin images promote` | `--image-id` (required), `--yes` | Reads `ami-<id>/config.json`, validates the AMI is account-owned, then rewrites `ImageOwnerAlias` to `"system"` in-place. No block data is copied. The change takes effect immediately — the AMI becomes visible to all accounts via `DescribeImages`. Prompts for confirmation (skipped with `--yes`). Already-system AMIs are refused. |
| `spx admin images remove` | `--image-id` (required), `--force`, `--yes` | Loads `ami-<id>/config.json`, walks transitive dependents — copied snapshots whose `VolumeID == imageID`, volumes whose `SnapshotID` references the internal `snap-ami-<id>` or any derived snap, and account AMIs created via `CopyImage` whose `SnapshotID` is a derived snap — then prompts (skipped with `--yes`) before deleting `ami-<id>/config.json` (the DescribeImages barrier) followed by the rest of `ami-<id>/` and `snap-ami-<id>/`. Account-owned AMIs are refused with a hint pointing at `aws ec2 deregister-image` + `aws ec2 delete-snapshot`. `--force` bypasses the dependency, ownership and config-corrupt checks for salvage of orphaned blocks. |
docs/DESIGN.md
+18 -24
View File
@@ -289,31 +289,25 @@ Configs are generated by `spx admin init` (leader) or `spx admin join` (follower
```
spinifex/
├── cmd/
│ ├── spinifex/ # spx CLI: service entry points and `spx admin`
│ ├── installer/ # Platform installer
│ └── ... # In-guest agents (lb, ecs, rds, eks helpers) and dev tools
├── spinifex/
│ ├── services/ # Long-running services (awsgw, spinifex, viperblockd,
│ │ # predastore, nats, spinifexui)
│ ├── gateway/ # AWS API surface
│ │ ├── gateway.go # chi router, SigV4 auth, throttling
│ │ ├── auth.go # SigV4 verification + IAM lookup
│ │ ├── ec2.go # EC2 action dispatcher
│ │ ├── ec2/ # Per-resource handlers (instance, volume, vpc, ...)
│ │ ├── elbv2/, iam/, policy/
│ ├── handlers/ # NATS-side service implementations (ec2, elbv2, iam, ...)
│ ├── daemon/
│ │ ├── daemon.go # Daemon entry, NATS subscription table
│ │ └── daemon_handlers_*.go # One file per resource family
│ ├── admin/ # Cluster bootstrap (init/join, master.key, CA, tokens)
│ ├── config/ # Configuration structures
│ ├── instancetypes/ # t3.*, m5.*, sys.* definitions
│ ├── lbagent/ # In-VM agent for ELB target health
│ ├── vm/ # VM instance representation
│ ├── vpcd/ # VPC daemon (OVN/OVS network programming)
│ └── qmp/ # QEMU Machine Protocol client
└── cmd/
├── spinifex/ # spx CLI
├── lb-agent/ # ELB target agent binary
├── installer/ # Platform installer
└── ... # ECS agent, ECR credential provider, EKS helper binaries
│ ├── services/ # Long-running services (awsgw, spinifex, nats, predastore, ...)
│ ├── gateway/ # AWS API surface: routing, SigV4 auth, per-service dispatch
│ ├── handlers/ # NATS-side service implementations (ec2, iam, eks, rds, ...)
│ ├── daemon/ # Per-node daemon and its NATS subscription table
│ ├── vm/, qmp/ # VM lifecycle and the QEMU Machine Protocol client
│ ├── vpcd/, network/ # VPC daemon and the layered OVN/OVS network stack
│ ├── admin/ # Cluster bootstrap (init/join, master.key, CA, tokens)
│ └── config/ # Configuration structures
├── internal/ # In-guest SigV4 clients and the AWS Smithy models
├── build/ # systemd units, node helper scripts, microVM build
├── scripts/ # Install, dev-environment, image-build scripts
└── tests/
├── integration/ # Gateway router against embedded NATS JetStream
└── e2e/ # Live-cluster suites (see docs/testing/e2e-coverage.md)
```
## Development
docs/README.md
+2
View File
@@ -4,6 +4,8 @@ This directory contains the Spinifex documentation. This is available in individ
The [AWS API coverage](coverage/README.md) pages are generated from Spinifex's dispatch tables and the pinned AWS API models, one page per service. Run `make aws-model-coverage` to regenerate them and print a summary. Only `pages.json` and the `intro.md` files under `coverage/` are hand-written; every `README.md` there is generated and will be overwritten.
The same run writes [`coverage/coverage.json`](coverage/coverage.json), a versioned machine-readable model-versus-dispatch inventory. It is generated and must not be edited. Its operation states are structural: `implemented` means a registered action has neither a stub nor an explicit refusal. It is not a behavioural compatibility claim; conformance evidence and capability profiles establish that separately.
A published page names the operations Spinifex serves, and nothing else. It carries no percentage, no unimplemented operation and no operation the platform will never offer: the same run writes those to [`coverage/internal-report.md`](coverage/internal-report.md), which no slug in the docs site's configuration points at, so it stays internal. A `notApplicable` declaration in `pages.json` therefore publishes nothing; it keeps a permanent absence out of the report's candidate work.
## Frontmatter
docs/admin/host-firewall/README.md
+233
View File
@@ -0,0 +1,233 @@
---
title: "Host Firewall"
seoTitle: "Host Firewall and Cluster Membership — Spinifex Docs"
description: "Arm the Spinifex host firewall, narrow SSH to your management networks, and learn how the peer-scoped cluster plane tracks membership changes on its own."
category: "Admin"
tags:
- security
- firewall
- nftables
- hardening
resources:
- title: "Network Connections"
url: "/docs/network-connections"
- title: "Multi-Node Install"
url: "/docs/install-multi-node"
- title: "Setting Up Your Cluster"
url: "/docs/setting-up-your-cluster"
- title: "Spinifex Repository"
url: "https://github.com/mulgadc/spinifex"
---
# Host Firewall
> Arm the host firewall, narrow SSH, and keep the cluster plane off the public internet.
## Table of Contents
- [Overview](#overview)
- [What the policy allows](#what-the-policy-allows)
- [Cluster membership](#cluster-membership)
- [Is it on?](#is-it-on)
- [Narrowing SSH access](#narrowing-ssh-access)
- [Turning it off and on around cluster changes](#turning-it-off-and-on-around-cluster-changes)
- [The files](#the-files)
- [Checking it](#checking-it)
- [Troubleshooting](#troubleshooting)
---
## Overview
Spinifex ships an optional host firewall — an nftables input policy that defaults to drop and divides the node's ports into two groups:
| Group | Ports | Who can reach them |
|---|---|---|
| **Public** | SSH, 443, 3000 (console), 8443 (S3), 9999 (AWS gateway), 53 (DNS) | anyone |
| **Internal** | OVN, NATS, formation, Geneve and the rest of the cluster plane | **cluster members only** |
The internal group is the point. Before this existed, OVN and NATS were reachable from the public internet on a WAN-facing node.
**"Cluster members" is not a list you maintain.** Each node works it out from the cluster it belongs to and rewrites its own rules whenever membership changes — you never edit the peer list by hand.
The policy governs inbound traffic only. Outbound is untouched, because the metadata reply path egresses under per-tap policy routing and a filter there would break it.
## Instructions
## What the Policy Allows
The public plane is open to any source, IPv4 only — an accept with no source scope would otherwise also serve on any global IPv6 address the node picks up from an upstream router advertisement:
| Ports | Serving |
|---|---|
| TCP 443 | Held for the TLS edge; no listener yet, so the rollout needs no firewall change |
| TCP 3000 | Web console |
| TCP 8443 | Predastore S3 gate |
| TCP 9999 | AWS gateway — also how every guest agent (load balancer, EKS, ECS, RDS) reaches its control plane, since they speak SigV4 rather than NATS |
| TCP and UDP 53 | Northstar authoritative DNS. Open on every node deliberately: a node serving no public zone still answers here, which is the accepted cost of not templating the policy per node |
SSH is public too, but scoped by a variable you own — see [Narrowing SSH access](#narrowing-ssh-access).
The cluster plane is peer-scoped. On a single-NIC node the planes collapse onto one address, which is why these rules match peer addresses rather than an interface or a CIDR:
| Ports | Serving |
|---|---|
| TCP 4222, 4248 | NATS client and cluster routes |
| TCP 4432 | Formation, then the daemon cluster manager |
| TCP and UDP 5300 | Internal DNS |
| TCP 6641, 6642 | OVN northbound and southbound |
| TCP 6643, 6644 | OVN database clustering |
| TCP 8660 | Predastore's admin listener |
| UDP 6660, 7660 | Object shards and metadata consensus |
| UDP 6081, 500, 4500, and ESP | Geneve, IKE, NAT-T and IPsec |
Two of those are peer-scoped for reasons worth knowing. Outside a formation window, 4432 is the daemon cluster manager, whose routes report node topology, service inventory and running instances to anyone who asks. And 8660 is Predastore's unauthenticated `/healthz` and `/readyz`, which name the peers a gate cannot reach.
**The formation port needs nothing from you.** `spx admin init` opens 4432 to any source for the length of the formation window and closes it again afterwards, because a node dialling in to join is not a peer yet. It is one slot rather than a list, so it cannot be grown into a general-purpose hole. The handshake behind it is TLS 1.3 with a bearer token.
Beyond those groups the policy accepts what the platform cannot work without: established and related connections, loopback, the ICMP types PMTUD needs under a Geneve overlay, broadcast DHCP replies for the uplink and external-pool leases, and guest traffic to the metadata and VPC DNS endpoints on `169.254.169.254` and `169.254.169.253`. The guest rules are scoped to those two addresses and their ports rather than accepting everything arriving on a guest port, which would expose every wildcard-bound host service to any tenant.
## Cluster Membership
The daemon resolves who the node's peers are from the cluster it belongs to, writes them into `peers.nft`, and reloads the policy. A node that has not resolved membership yet is left reachable rather than cut off from a cluster it cannot name: the apply helper fails without touching the ruleset when the peer file is missing.
One consequence matters in practice. **A node only recognises the members of the cluster it currently belongs to.** During formation the nodes do not yet know each other, so internal traffic between them is dropped — which is why forming or expanding a cluster starts by turning the policy off.
## Is It On?
| How the node was installed | Firewall |
|---|---|
| From the ISO | **on** |
| Binary installer (`curl \| bash`) or `setup.sh` | **off** |
| `setup.sh --firewall=on` | **on** |
The binary installer defaults to off deliberately: it runs on servers that already have an operating system and services on them, and switching on a default-deny policy uninvited could cut off something Spinifex knows nothing about. The ISO is an appliance on hardware we define, so there the policy is the product.
**For production, turn it on** — either at install time:
```bash
curl -fsSL https://install.mulgadc.com | bash -s -- --firewall=on
```
or afterwards, by setting it in `/etc/spinifex/spinifex.toml` and restarting the daemon:
```toml
[network]
firewall_enabled = true
```
Before you do, check what else the machine is serving. Anything listening on a port outside the public group above stops accepting new connections.
## Narrowing SSH Access
SSH is accepted from **every source** by default. That is deliberate — arming the policy should not lock an operator out of the node they are arming it from — but it is the first thing to change on a production cluster.
The sources live in `/etc/spinifex/firewall/custom.nft`, which the installer creates once and never rewrites, so an edit here survives upgrades, re-installs and node resets:
```
define trusted_ssh_peers = { 10.0.0.0/8, 192.168.1.0/24 }
```
Check the change before applying it, and check the policy file rather than this one — on its own `custom.nft` parses as a bare define and proves nothing, because the rule that uses the variable lives elsewhere:
```bash
sudo nft -c -f /etc/spinifex/firewall/spinifex.nft
sudo nft -f /etc/spinifex/firewall/spinifex.nft
```
Never leave the variable empty. An undefined or empty set fails the whole ruleset, not just its own rule.
The ports come from sshd's own configuration, re-read on every `setup.sh` run, so moving sshd to a non-standard port is picked up by an upgrade rather than needing an edit here. IPv4 only: sshd reached over IPv6 is not matched by this rule.
## Turning It Off and On Around Cluster Changes
Turn the firewall off while you form or expand a cluster, and on again once it is up:
```bash
# Off — before forming or expanding a cluster. Run on every node.
sudo /usr/local/lib/spinifex/spinifex-firewall-apply disable
# On — once the cluster is formed and verified. Run on every node.
sudo systemctl restart spinifex-daemon
```
Restarting the daemon is what re-arms it: the node rebuilds its peer list from the cluster it is now part of, reloads the rules, and re-enables the boot-time unit so the policy survives a reboot. It also happens on its own within five minutes if you would rather wait.
`disable` removes Spinifex's own table and the peer file and leaves every other nftables table alone. It is idempotent, and it is the recovery path — it runs even on a node with no policy installed, so it cannot leave you with a node you can neither arm nor disarm.
> [!WARNING]
> Do not flush the whole nftables ruleset to clear the policy. The VPC daemon writes SNAT and per-elastic-IP forward rules into the `ip nat` and `ip filter` tables and reinstalls them only when it starts, so `nft flush ruleset` breaks elastic IPs silently until the next `systemctl restart spinifex-vpcd`. Use `disable`, which only ever touches `inet spinifex_filter`.
## The Files
Everything lives under `/etc/spinifex/firewall/`. Which files are safe to edit follows from who writes them:
| File | Written by | Survives an upgrade |
|---|---|---|
| `spinifex.nft` | `setup.sh` — the policy itself | No. Edits are overwritten |
| `local.nft` | `setup.sh` — the sshd ports it detected | No. Refreshed on every run |
| `peers.nft` | The daemon — cluster and encap peer addresses | No. Rewritten on every membership change |
| `open-ports.nft` | The apply helper — the formation window only | No. Reset closed on install and upgrade |
| `custom.nft` | **You** — `trusted_ssh_peers` and anything else you add | **Yes.** Created once, never rewritten |
| `mode` | `setup.sh` — whether the install path armed the policy | No |
The daemon cannot write `/etc` itself, and a blanket `nft` grant would be root-equivalent, so changes go through `/usr/local/lib/spinifex/spinifex-firewall-apply` — a root helper that accepts a fixed set of verbs and re-validates every address handed to it.
## Checking It
```bash
sudo nft list table inet spinifex_filter
```
The peer list is an nft variable, expanded when the rules load, so it does not appear under a name of its own. Look instead at the `ip saddr { ... }` addresses on the cluster-plane rules — the ones accepting 4222, 6641, 6642 and the rest.
Every node's addresses should be there. On a multi-NIC node that means its WAN, LAN and VPC addresses, so expect several entries per node. A missing node means its cluster traffic is being dropped.
Dropped packets are logged, rate-limited to five a minute, so this tells you whether a connection problem is the firewall or something else:
```bash
sudo journalctl -k | grep 'spinifex-fw drop'
```
An empty log with a connection still failing means the policy is not the cause. A node with no table loaded at all is not armed:
```bash
systemctl is-enabled spinifex-firewall.service
```
## Troubleshooting
### A Join Hangs Instead of Failing
A hang rather than a quick refusal means packets are being dropped. Check the drop log on the node running init, and confirm the formation port opened — `spx admin init` prints `⚠️ Could not open port 4432 in the host firewall` when it could not:
```bash
sudo journalctl -k | grep 'spinifex-fw drop'
```
On ISO-installed nodes the usual cause is that each node's policy still trusts only itself, because that is the whole cluster as far as it knows. Turn the firewall off on **every** node, retry, and re-arm once the cluster is verified.
### A Node Is Missing from the Peer List
The daemon rewrites `peers.nft` from the cluster it belongs to, so a missing node means that node is not in the membership this one resolved. Confirm what the cluster thinks first, then restart the daemon to force a rebuild:
```bash
spx get nodes
sudo systemctl restart spinifex-daemon
```
### One of My Own Services Stopped Answering
Arming the policy closes every port outside the public group. Add what you need to `custom.nft`, which is the only file an upgrade leaves alone, and check the policy before applying it:
```bash
sudo nft -c -f /etc/spinifex/firewall/spinifex.nft
```
### Elastic IPs Stopped Working
Something flushed the whole nftables ruleset, taking the VPC daemon's SNAT and forward rules with it. They are reinstalled on start:
```bash
sudo systemctl restart spinifex-vpcd
```
docs/admin/predastore-distributed-storage/README.md
+250
View File
@@ -0,0 +1,250 @@
---
title: "Predastore Distributed Storage"
seoTitle: "How Predastore Distributed Storage Works — Spinifex Docs"
description: "How Spinifex stores objects across a cluster: gate, blob and meta nodes, Reed-Solomon erasure coding, placement, degraded writes, repair and encryption at rest."
category: "Admin"
tags:
- storage
- s3
- predastore
- erasure coding
- durability
resources:
- title: "Predastore Repository"
url: "https://github.com/mulgadc/predastore"
- title: "Viperblock (EBS)"
url: "https://github.com/mulgadc/viperblock"
- title: "Multi-Node Install"
url: "/docs/install-multi-node"
- title: "Network Connections"
url: "/docs/network-connections"
---
# Predastore Distributed Storage
> How Spinifex stores objects across a cluster, and what a server failure costs.
## Table of Contents
- [Overview](#overview)
- [Hosts and nodes](#hosts-and-nodes)
- [The configuration file](#the-configuration-file)
- [Erasure coding and durability](#erasure-coding-and-durability)
- [Placement](#placement)
- [The write path](#the-write-path)
- [The read path](#the-read-path)
- [Repair](#repair)
- [Encryption at rest](#encryption-at-rest)
- [What a server failure costs](#what-a-server-failure-costs)
- [Checking it](#checking-it)
- [Troubleshooting](#troubleshooting)
---
## Overview
Predastore is the S3-compatible object store underneath a Spinifex cluster, and almost everything durable ends up in it:
- **Machine images** — the AMIs instances launch from
- **Block storage** — EBS volume data and snapshots, written through [Viperblock](https://github.com/mulgadc/viperblock), which uses Predastore as its S3 backend
- **User data** — cloud-init configuration and system artifacts
- **DNS zone files** — the zones Northstar serves
- **Anything your own workloads put in a bucket** through the S3 API
This is background reading. You do not configure any of it by hand: `spx admin init` and `spx admin join` build the topology from the servers that actually form the cluster, and every machine gets the same file with its own host ID recorded in `spinifex.toml`.
## Instructions
## Hosts and Nodes
A cluster is two levels.
A **host** is one machine: one Predastore process owning that machine's data directory, TLS identity and encryption key. A **node** is a role pinned to a host, running inside that process on its own port. Every server in a Spinifex cluster is one host carrying three nodes:
| Role | Port | Purpose |
|---|---|---|
| `gate` | TCP 8443 | Serves the S3 API — SigV4, IAM, erasure coding, placement. Every server runs one, so any of them answers an S3 request. |
| `blob` | UDP 6660 | Holds erasure-coded object shards. One per machine. |
| `meta` | UDP 7660 | Member of the Raft quorum over global state — buckets and the object index. |
Ports have to be unique within a host but not across the cluster, so every machine uses the same three. That is why the port surface does not widen as the cluster grows — see [Network requirements](/docs/install-multi-node#network-requirements).
Which transport a pair of nodes uses follows from where they sit, not from a mode flag. Nodes on the same machine talk over an in-process pipe and open no socket at all; nodes on different machines talk over QUIC, TLS 1.3, authenticated by the cluster CA. A single-server install is therefore one process listening on 8443 and nothing else.
Nothing dials a gate, so a gate binds no cluster-plane listener. Its port is the S3 port and nothing more.
## The Configuration File
The whole cluster's storage topology lives in `/etc/spinifex/predastore/predastore.toml`, identical on every server apart from which host the local process is told to be.
Each server is one `[[host]]` block carrying three `[[host.node]]` blocks:
```toml
[[host]]
id = 1
bind_addr = "10.2.0.2" # cluster plane: raft and blob
addr = "10.2.0.2" # what the other hosts dial
data_dir = "/var/lib/spinifex/predastore"
tls_cert = "/etc/spinifex/server.pem"
tls_key = "/etc/spinifex/server.key"
encryption_key = "/etc/spinifex/predastore/encryption.key"
admin_port = 8660 # /healthz, /readyz, /repair
[[host.node]]
id = 1
role = "gate"
port = 8443
bind_addr = "0.0.0.0" # public plane; only a gate may set this
```
The two bind addresses are the point. `bind_addr` on the host is the cluster plane — Raft consensus and shard traffic listen there and nowhere else, at the address peers dial rather than a wildcard, so on a multi-homed machine replication never reaches the public interface. The gate binds the public plane separately, because S3 is meant to be reachable.
## Erasure Coding and Durability
The erasure code is cluster configuration, not a per-request choice, and it is chosen at formation from the number of servers — each machine contributes exactly one blob node:
| Servers | Code | What it survives |
|---|---|---|
| 1 | `RS(1,0)` | Nothing at this layer. One shard on one node; redundancy is the pool's job — ZFS raidz or a mirror under the data directory. |
| 2 | `RS(1,1)` | One server's shards, but the metadata quorum has no majority to lose. |
| 3 or more | `RS(2,1)` | Any one server's shards, with the metadata quorum intact. |
An object is encoded end to end into `K` data and `M` parity shards with no intermediate chunking. Each shard is roughly `⌈size / K⌉` bytes, and any `K` of the `K+M` shards rebuild the object.
`data + parity` must not exceed the number of blob nodes, because a stripe is spread over distinct nodes and a narrower cluster would fail every write. `RS(2,1)` surviving the loss of any one server is the reason [three servers is the recommended minimum](/docs/install-multi-node).
## Placement
Which nodes hold an object's shards comes from a consistent hash ring over the blob nodes, keyed by the object hash. `Nodes(hash, K+M)` returns the placement in shard order — shard `i` goes to the node at position `i` — so every gate in the cluster derives the same placement from the same object without consulting anything.
The gate then **records** that placement in the metadata plane rather than trusting the ring at read time. A ring that has since changed shape does not lose objects placed under the old one.
## The Write Path
The gate authenticates the request, resolves the bucket, hashes the object name, places it on the ring, encodes the body into data and parity shards, streams each shard to its blob node in parallel, and records the placement.
Three properties matter operationally:
- **The write is two-phase.** Each shard put makes the bytes durable and invisible. The placement record landing in Raft is the commit point, and the shards are published only afterwards — so an interrupted writer costs a round trip rather than the object. Shards are always durable before metadata: a crash between the two leaves shards nothing points at, rather than metadata pointing at data that was never written.
- **Every write mints an epoch**, stamped on every shard of the stripe and into the record. The record names one generation, and a shard either belongs to it or does not. That is what makes a half-completed overwrite a reconstruction rather than an object spliced from two generations.
- **One server down does not refuse writes.** The write is acknowledged once `K` shards are durable, and the response carries `X-Spx-Degraded-Write: <n>`. `K` is the floor rather than `K+1` because any `K` of the `K+M` reconstruct — at `RS(2,1)` one more buys no redundancy and would refuse three quarters of writes with a single server down.
A shard its owner will not take is written to the next node along the ring instead of being given up on — **hinted handoff** — and the response carries `X-Spx-Handoff: <n>`. The stripe is then complete and the object is as redundant as a full-width write; what is outstanding is only that the shards are not all where the record says. The holder is derived rather than recorded, so there is no hint to store, replicate or lose. Handoff is skipped when the cluster has no node to spare, since a second shard of one object on a node that already holds one is not redundancy.
If any node reported storage pressure the response also carries `X-Predastore-Pool-Pressure: nearfull`; a node that refused outright surfaces as `507 InsufficientStorage`.
## The Read Path
The gate reads the recorded placement, fetches the data shards, and joins them. Parity is never touched on the happy path. Only if the join fails does it refetch including parity and reconstruct, and a read that had to reconstruct says so in `X-Spx-Degraded: <n>` — a cost report, not an error.
Every shard read names the record's epoch, and a node holding another generation answers `epoch-mismatch`. **The gate counts that exactly as a missing shard**, so the read either rebuilds the object the record names or fails. It never returns plausible nonsense.
A shard its owner cannot serve is looked for on the handoff node before it is counted as failed, since a write that could not reach the owner will have put it there.
A `Range` request that lands entirely inside one data shard takes a fast path — a single ranged shard read — because Reed-Solomon splits data sequentially. Anything wider reconstructs the object and slices it.
## Repair
Degraded writes trade redundancy for availability, and repair is what pays that back. It restores shards a blob node owns but does not hold at the generation its object's record names, and it runs on every server with a local blob node.
Two things are worth knowing about it:
- **It carries no correctness weight.** A read compares each shard's epoch against the record it has already loaded and discards a stale one whether or not repair has ever run. What repair restores is redundancy — the number of further losses an object survives — and never the object.
- **It rebuilds onto the owner**, so it closes the gap only once the missing server returns. At `RS(2,1)` an object written while a server is down holds no parity for the duration of the outage. That is the accepted position: refusing writes meanwhile is the worse failure.
Every pass scans the placement records from the beginning, asks each holder which generation it has, and rebuilds the ones that disagree. A shard that was handed off is returned rather than recomputed — cheaper by a decode — and the holder's copy is released only after the owner's is published. If fewer than `K` peers hold the record's generation the rebuild fails and says so, rather than inventing a shard from a mixture of generations.
## Encryption at Rest
Every fragment is sealed independently under AES-256-GCM, with a deterministic nonce and additional data reconstructed at read time from the on-disk header plus per-data-directory state. There is no unencrypted code path: the process refuses to start without an encryption key.
| Property | What it means |
|---|---|
| **Confidentiality** | Disk-level access yields ciphertext and tags. Plaintext is never written. |
| **Authenticated integrity** | GCM is the sole integrity authority; there is no separate checksum. Corruption, tampering and a wrong key all surface as one integrity error. |
| **Position binding** | The additional data binds each fragment to its slot, so swapping fragments between objects fails the tag. |
| **Cross-directory defence** | A per-data-directory store ID enters the nonce, so splicing a fragment from one data directory into another fails the tag. |
The key is one 32-byte AES-256 key per cluster, and **every server must be given the same file** — fragments sealed under one key cannot be opened under another. `spx admin init` generates it and `spx admin join` distributes it, which is also why joining a server that has already been in service discards its own key and orphans everything sealed under it. The loader is fail-closed on permissions: a group- or other-readable key file is rejected outright, with no override.
## What a Server Failure Costs
| What fails | Reads | Writes |
|---|---|---|
| One gate (of three or more) | Unaffected — any gate answers for the cluster | Unaffected |
| One blob node, `RS(2,1)` | Served by reconstruction, reported in `X-Spx-Degraded` | Accepted at the `K` floor, reported in `X-Spx-Degraded-Write`, or placed complete via handoff |
| More than `M` blob nodes | Objects placed on them are lost | Refused |
| Metadata quorum | Serve whatever the surviving replicas last applied | Stopped entirely |
Reads try the cached Raft leader first, then every replica, and report not-found only once every replica has answered not-found — a replica that has not applied a key yet must not be mistaken for absence. Writes go through the leader, and a replica that cannot commit redirects the client to the one that can.
Replacing a server restores capacity, not the shards that were on the old one: repair rebuilds onto the owner it expects, so plan a replacement as a return of that node rather than as re-replication onto a new one.
## Checking It
Each host runs an unauthenticated admin listener on port 8660, bound to the cluster plane rather than the S3 address. It serves counters and readiness, never addresses or keys:
```bash
curl -s http://$(hostname -I | awk '{print $1}'):8660/readyz
curl -s http://$(hostname -I | awk '{print $1}'):8660/repair
```
`/readyz` reports the most recent probe cycle for the nodes in that process, and answers `503` until one has completed — reporting ready on no information would send traffic to a process that has never answered the question. `/healthz` answers for the process itself: reaching the handler is the whole check.
That the gate is serving is an S3 request like any other:
```bash
export AWS_PROFILE=spinifex
aws s3 ls
```
To see whether a write or read was degraded, ask for the response headers:
```bash
aws s3api put-object --bucket predastore --key probe --body /etc/hostname --debug 2>&1 | grep -i 'x-spx-\|x-predastore-'
```
And `spx get nodes` should list `predastore` in `SERVICES` on every node — a short service list on one server means something failed to start there.
## Troubleshooting
### S3 Answers but Every Write Fails
The gate is up and the metadata quorum is not. Check that a majority of servers are running `spinifex-predastore`:
```bash
spx get nodes
sudo systemctl is-active spinifex-predastore
```
Reads keep serving what the surviving replicas last applied, which is why this presents as a write-only outage.
### 507 InsufficientStorage, or `nearfull` on Every Response
A blob node is out of space, or close enough that compaction has been kicked. Free space in the data directory on each server:
```bash
df -h /var/lib/spinifex/predastore
```
Capacity here is per server, not pooled: a stripe needs its shard on a specific node, so one full server refuses writes for the objects the ring places on it.
### Reads Fail with an Integrity Error
GCM is the only integrity authority, so corruption, tampering and a wrong encryption key are indistinguishable. Confirm the key first, since it is by far the most likely of the three — every server must hold the identical file:
```bash
sudo sha256sum /etc/spinifex/predastore/encryption.key
```
Compare the digest across servers. A mismatch means that server was initialized separately rather than joined, and the fragments it sealed cannot be opened by the cluster.
### Degraded Reads That Never Clear
`X-Spx-Degraded` on reads long after an outage means repair has not restored the shards. Repair rebuilds onto the node that owns the shard, so it cannot finish while that node is absent:
```bash
curl -s http://$(hostname -I | awk '{print $1}'):8660/repair
```
`"enabled": false` means the sweep is off on that host. Otherwise bring the missing server back and let a pass complete — the sweep runs every five minutes by default.
docs/admin/setting-up-your-cluster/README.md
+57 -4
View File
@@ -1,7 +1,7 @@
---
title: "Setting Up Your Cluster"
seoTitle: "Set Up Your First Spinifex Cluster — Spinifex Docs"
description: "Import an AMI, create an SSH key pair and a VPC with a public subnet, then launch your first EC2 instance and connect to it on a fresh Spinifex cluster."
description: "Import an AMI and the RDS, ECS and EKS base images, create an SSH key pair and a VPC with a public subnet, then launch your first EC2 instance on a fresh Spinifex cluster."
category: "Admin"
tags:
- setup
@@ -17,24 +17,26 @@ resources:
# Setting Up Your Cluster
> Import an AMI, create a VPC with a public subnet, and launch your first instance.
> Import an AMI and the system base images, create a VPC with a public subnet, and launch your first instance.
## Table of Contents
- [1. Import an AMI](#1-import-an-ami)
- [Option C: Import the system base images](#option-c-import-the-system-base-images)
- [2. Create an SSH Key](#2-create-an-ssh-key)
- [3. Create a VPC and Public Subnet](#3-create-a-vpc-and-public-subnet)
- [4. Launch an Instance](#4-launch-an-instance)
- [5. Connect via SSH](#5-connect-via-ssh)
- [6. Managing Instances](#6-managing-instances)
- [7. Launching the Web UI](#7-launching-the-web-ui)
- [8. Secure the Install](#8-secure-the-install)
- [Troubleshooting](#troubleshooting)
---
## Overview
This guide walks through the first steps on a freshly installed cluster: importing an AMI, creating an SSH key and a VPC with a public subnet, launching an instance, and connecting to it.
This guide walks through the first steps on a freshly installed cluster: importing an AMI and the system base images RDS, ECS and EKS need, creating an SSH key and a VPC with a public subnet, launching an instance, and connecting to it.
## Instructions
@@ -72,7 +74,7 @@ spx admin images import --name ubuntu-26.04-x86_64
### Option B: Import a local image file
```bash
spx admin images import --file ~/images/ubuntu-26.04.img --distro ubuntu --version 26.04 --arch x86_64 --boot-mode uefi
spx admin images import --file ~/images/ubuntu-26.04-server-cloudimg-amd64.img --checksum ~/images/SHA256SUMS --distro ubuntu --version 26.04 --arch x86_64 --boot-mode uefi
```
Verify the import and note the AMI ID:
@@ -81,6 +83,41 @@ Verify the import and note the AMI ID:
AMI_ID=$(aws ec2 describe-images --query 'Images[0].ImageId' --output text)
```
### Option C: Import the system base images
**RDS, ECS and EKS each launch from an image of their own, and none of them is imported for you.** Until these are in the catalog, `CreateDBInstance`, `CreateCluster` and `CreateNodegroup` all fail at the point they go looking for a node image — so a cluster without them has those three services present in the API and unusable in practice. Import them once, now, and the install is functional from the start.
Run these one at a time rather than in parallel, so predastore is not taking four concurrent uploads:
```bash
for img in spinifex-rds-postgres spinifex-rds-mariadb spinifex-ecs-node spinifex-eks-node; do
sudo spx admin images import --name "$img" --config /etc/spinifex/spinifex.toml
done
```
| Image | Serves | Contents |
| --- | --- | --- |
| `spinifex-rds-postgres` | RDS, `--engine postgres` | Alpine 3.24.1 + PostgreSQL 18 + `rds-init` |
| `spinifex-rds-mariadb` | RDS, `--engine mariadb` | Alpine 3.24.1 + MariaDB 11.8 + `rds-init` |
| `spinifex-ecs-node` | ECS container instances | Alpine 3.21.7 + containerd + ecs-agent |
| `spinifex-eks-node` | EKS control plane and nodegroups | Alpine 3.21.7 + K3s v1.32.5 + eks-token-webhook |
All four boot **BIOS**, unlike the distro images, which boot UEFI. That is a property of how they are built and nothing you need to pass.
If you run GPU workloads, add `spinifex-ecs-node-gpu` and `spinifex-eks-node-gpu` as well — Ubuntu 26.04 with the NVIDIA driver, selected automatically for a GPU instance type. They boot UEFI and they are large, so import them only if you have GPUs.
> [!IMPORTANT]
> **These register under generic names, and that is correct.** `describe-images` shows `spinifex-eks-node` and `spinifex-ecs-node` both as `ami-alpine-3.21.7-x86_64`, and the two RDS images both as `ami-alpine-3.24.1-x86_64`. Nothing resolves them by name: ECS and EKS find theirs by the `spinifex:managed-by=ecs|eks` tag, and RDS by `spinifex:managed-by=rds` plus the `engine` and `engine-version` tags. **Verify the tags, not the names** — four rows with duplicate-looking names is what success looks like:
>
> ```bash
> aws ec2 describe-images \
> --filters 'Name=tag-key,Values=spinifex:managed-by' \
> --query 'Images[].[ImageId,State,Tags[?Key==`spinifex:managed-by`].Value|[0],Tags[?Key==`engine`].Value|[0]]' \
> --output text
> ```
`spx admin images list --config /etc/spinifex/spinifex.toml` on a node shows the whole catalog, including the images not imported here.
## 2. Create an SSH Key
### Option A: Import an existing key
@@ -340,6 +377,22 @@ At the Spinifex login screen, paste the **Access Key ID** and **Secret Access Ke
Once logged in, you have browser-based access to every Spinifex feature: launch and manage instances, attach EBS volumes, browse S3 buckets, configure VPCs and security groups, and manage IAM users and keys — all backed by the same AWS-compatible control plane the CLI uses.
## 8. Secure the Install
Your cluster works. Before it carries anything real, close the host down.
**Arm the host firewall.** A binary install leaves it off, because it lands on a machine that was already doing something. Armed, it keeps OVN, NATS and the rest of the cluster plane reachable by cluster members only, and everything outside the public group closed:
```toml
# /etc/spinifex/spinifex.toml
[network]
firewall_enabled = true
```
**Narrow SSH.** It is accepted from every source by default, so that arming the policy cannot lock you out of the node you are arming it from. Replace `0.0.0.0/0` with your management networks in `/etc/spinifex/firewall/custom.nft` — the one file an upgrade never overwrites.
[Host Firewall](/docs/host-firewall) covers both, what the policy allows, and how each node tracks cluster membership without a peer list you maintain.
## Additional Options
### Private Subnets (No Public IP)
docs/admin/spinifex-admin-cli/README.md
+1 -1
View File
@@ -109,7 +109,7 @@ spx admin images list
spx admin images import --name debian-13-arm64
```
Catalog imports verify the image against the catalog-declared SHA-256/SHA-512 digest before extraction. Use `--file` to import operator-supplied media (verification skipped — operator is responsible for integrity), or `--force` to re-download after a checksum mismatch.
Catalog imports verify the image against the catalog-declared SHA-256/SHA-512 digest before extraction. Use `--file` with `--checksum <sums-file>` to import operator-supplied media verified against the sums file it was published with, or `--force` to re-download after a checksum mismatch. A `--file` import without `--checksum` proceeds unverified and prints a notice. Every import records the source file's digest on the AMI; `spx admin images describe --image-id <ami>` shows it.
### EKS node image
docs/compute/gpu-passthrough/README.md
+15
View File
@@ -136,6 +136,21 @@ aws ec2 run-instances \
--key-name spinifex-key
```
### Multi-GPU Instances
The GPU count comes from the instance type, exactly as it does in EC2. AWS multi-GPU types pair eight GPUs with a very large host — `p4d.24xlarge` is 96 vCPU and 1152 GiB, — so an edge rig with eight cards and 256 GiB cannot admit them even though it has the GPUs. For those hosts Spinifex offers its own `gpu.*` family, which is not an EC2 family:
```bash
aws ec2 run-instances \
--image-id $GPU_AMI \
--instance-type gpu.8x4c \
--key-name spinifex-key
```
`gpu.<count>x<vcpu>c` is that many GPUs at that many vCPU each, so `gpu.8x4c` is eight GPUs on 32 vCPU and 128 GiB. Counts run in three tiers — `2c`, `4c` and `8c` — all at 4 GiB of memory per vCPU. The GPU model and VRAM reported in `DescribeInstanceTypes` are whatever the node discovered, so a `gpu.8x4c` on an 8×RTX 3090 box advertises eight RTX 3090s at 24 GiB each.
A node holding more than one distinct GPU model offers single-GPU types only, and logs each withheld type by name. Two devices count as the same model when their reported name and VRAM match, which keeps a mixed-SKU H100 node fully usable.
To verify the GPU is visible from inside the instance, SSH in and run:
```bash
docs/compute/vpc-networking/README.md
+30 -3
View File
@@ -161,6 +161,12 @@ Both support the same AWS features: public subnets, Elastic IPs, security groups
## `nat` — Shared SNAT (Simple)
**`nat` is a supported mode, not a fallback.** It is covered by the nightly e2e suite (cells 19 and 30, single- and multi-node) and it is the only mode a cloud-hosted node can run, so the OCI deployment depends on it. Read "specialised" below as "fewer deployments need it", never as "less finished".
**Start with `pool` unless one of the reasons below applies.** Pool mode is the default, is what most of this document describes, and is what the AWS feature set was built against. `nat` is the specialised mode: it exists for uplinks that cannot be bridged, and it reaches AWS parity through host-side plumbing rather than by putting VM addresses on the wire.
Reach for `nat` when the uplink refuses to carry anything but its own MAC and its own addresses — WiFi and WWAN, and **any cloud-hosted node**, where the VNIC drops a frame whose source address is not registered to it. On such a host `nat` is not a preference, it is the only mode that works at all; pool mode's localnet needs to announce VM addresses by ARP and a cloud VNIC will not carry them. See [oci-integration](../../oci-integration/README.md).
All VMs share a single external IP for outbound SNAT. By default there are no public IPs, no Elastic IPs, and no inbound from WAN — all subnets behave as private subnets with internet access. On routed-NAT nodes, adding a public pool restores full public IP parity (see below).
> **Limitation (routed-NAT v1):** System instances (ECS/EKS/load-balancer
@@ -244,6 +250,24 @@ gateway = "192.168.1.1"
prefix_len = 24
```
### NAT gateways in NAT mode
`CreateNatGateway` works here too, and draws its public address from the same public pool an EIP does. On the OVN side it is an `snat` row on the VPC router rewriting the private subnet's CIDR to that address — not the `dnat_and_snat` an EIP gets — so it is centralised on the VPC's gateway chassis by construction and there is no port or MAC to bind it to.
**That address needs the same host plumbing as an EIP**, and on a routed-NAT node it is the host that delivers it: the `/32` route steering it into OVN, the proxy-ARP entry answering for it on the uplink, and the FORWARD accepts. Without them an egressing private instance leaves with a source address the host holds no route back to, while the masquerade rule — which matches the transit `100.127.0.0/24` alone — never sees the packet. Spinifex binds it when the gateway is created and re-asserts it on the host EIP pass, so no operator step is involved; `ip route show dev spx-nat-host` should list the gateway's public address alongside any EIPs.
### NAT mode on more than one node
`spx admin init --external-mode=nat --nodes N` works for N > 1, and each public IP is delivered by the node running its instance rather than by one gateway node. Three things make that true, and all three are automatic:
- The `dnat_and_snat` row carries `external_mac` and `logical_port`, so OVN processes it on the instance's own chassis — the same mechanism pool mode has always used. It is safe here because `ext-shared` is a localnet, so the transit veth is node-local and only the node running the instance can answer for the address on it.
- Each node installs a host `/32` for its own instances' addresses and leaves the rest alone, asking local OVS which ENIs are plugged in here. Run `ip route show dev spx-nat-host` on each node: exactly one node should hold each public address.
- Every node's `spx-nat-host` carries the same MAC, `02:00:64:7f:00:01`. OVN keeps one MAC binding for the transit next hop across the whole cluster, so a per-node address would send every chassis's return traffic to whichever node wrote the binding last. `setup-ovn.sh` sets it and vpcd re-asserts it on every start.
Default egress — an instance with no public IP — is still centralised on the VPC's gateway chassis, which is unchanged and is what the gateway LRP's transit address is for.
**On a cloud host the address must also be registered on that node's VNIC.** The allocator runs per node and registers on the local VNIC, which is the right one exactly because the traffic now leaves there. `oci_vnic_iface` is the setting; `oci_vnic_id` is only needed to pin a pool to a specific VNIC and should not be set for this.
**Caveat — reaching an EIP from the spinifex host itself.** Host-sourced traffic enters OVN from the transit net, which is exempt from NAT (that is what makes the jumpbox pattern work) — so a host connection to an EIP that carries the transit source IP would skip DNAT. Spinifex stamps the EIP route with the uplink's LAN IP as source to avoid this, but if no uplink address can be determined, connect to the instance's **private IP** from the host instead. Other machines on the LAN are unaffected.
## Disabled (Empty/Omitted)
@@ -263,6 +287,9 @@ VPC networking is overlay-only. No external connectivity. Instances can only com
| DescribeInstances shows public IP | Yes | Yes | With public pool | No |
| Admin must reserve IP range | Yes | No | Only static pool | No |
| Needs router DHCP | No | Yes | Optional | No |
| Works on a non-bridgeable uplink | No | No | Yes | n/a |
| NAT gateways | Yes | Yes | With public pool | No |
| More than one node | Yes | Yes | Yes | Yes |
If you start with `nat` and later need public subnets: on a bridgeable uplink switch to `pool` and define a range (or use `source = "dhcp"`); on a routed-NAT node just add a public pool alongside `nat-transit` — no data migration needed.
@@ -460,8 +487,8 @@ external_mode = "pool" # "pool", "nat", or "" (disabled)
| Value | Behavior |
| -------------- | ----------------------------------------------------------------- |
| `"pool"` | Full public networking — public subnets, auto-assign, Elastic IPs |
| `"nat"` | Outbound-only SNAT — all VMs share one external IP |
| `"pool"` | **Default.** Full public networking — public subnets, auto-assign, Elastic IPs |
| `"nat"` | Outbound-only SNAT — all VMs share one external IP. Add a public pool for full parity. For uplinks that cannot be bridged, including every cloud host |
| `""` / omitted | Overlay-only — no external connectivity |
## IP Pools: network.external_pools
@@ -923,7 +950,7 @@ firewall_enabled = true
Before enabling it, check what else the machine is serving — anything listening
outside the public port group stops accepting new connections. See
[Firewall and Cluster Membership](/docs/install-multi-node#firewall-and-cluster-membership)
[Host Firewall](/docs/host-firewall)
for the full port policy and the cluster-formation steps.
### Verifying it
docs/containers/ecr/README.md
+2 -3
View File
@@ -146,12 +146,11 @@ Repositories can be managed as code with the standard `aws_ecr_repository` resou
```hcl
provider "aws" {
region = "ap-southeast-2"
skip_credentials_validation = true
skip_requesting_account_id = true
region = "ap-southeast-2"
endpoints {
ecr = "https://<your-spinifex-gateway>"
sts = "https://<your-spinifex-gateway>"
}
}
docs/coverage/README.md
+7 -7
View File
@@ -17,18 +17,18 @@ tags:
## Overview
The platform serves **381 operations** across the AWS APIs below. Every page names the operations its service implements from the pinned model, generated from the dispatch tables on each build rather than written by hand.
The platform serves **397 operations** across the AWS APIs below. Every page names the operations its service implements from the pinned model, generated from the dispatch tables on each build rather than written by hand.
| Service | Operations |
|---|---:|
| [ACM](/coverage/acm) | 9 |
| [EC2](/coverage/ec2) | 125 |
| [EC2](/coverage/ec2) | 127 |
| [ECR](/coverage/ecr) | 22 |
| [ECS](/coverage/ecs) | 31 |
| [EKS](/coverage/eks) | 34 |
| [ELBv2](/coverage/elbv2) | 34 |
| [IAM](/coverage/iam) | 76 |
| [EKS](/coverage/eks) | 36 |
| [ELBv2](/coverage/elbv2) | 36 |
| [IAM](/coverage/iam) | 79 |
| [RDS](/coverage/rds) | 26 |
| [S3](/coverage/s3) | 19 |
| [S3](/coverage/s3) | 26 |
| [STS](/coverage/sts) | 5 |
| **Total** | **381** |
| **Total** | **397** |
docs/coverage/coverage.json
+3808
View File
File diff suppressed because it is too large Load Diff
docs/coverage/ec2/README.md
+3 -1
View File
@@ -18,7 +18,7 @@ tags:
## Overview
Spinifex implements **125 operations** in the EC2 `2016-11-15` API model.
Spinifex implements **127 operations** in the EC2 `2016-11-15` API model.
### Spot Instances Are a Mock
@@ -110,6 +110,7 @@ Spot Instance Requests are a mock over the on-demand `RunInstances` path. A requ
| `DescribeVolumes` |
| `DescribeVolumesModifications` |
| `DescribeVpcAttribute` |
| `DescribeVpcEndpoints` |
| `DescribeVpcs` |
| `DetachInternetGateway` |
| `DetachNetworkInterface` |
@@ -132,6 +133,7 @@ Spot Instance Requests are a mock over the on-demand `RunInstances` path. A requ
| `ModifyInstanceMetadataOptions` |
| `ModifyLaunchTemplate` |
| `ModifyNetworkInterfaceAttribute` |
| `ModifySecurityGroupRules` |
| `ModifySubnetAttribute` |
| `ModifyVolume` |
| `ModifyVpcAttribute` |
docs/coverage/eks/README.md
+3 -1
View File
@@ -18,7 +18,7 @@ tags:
## Overview
Spinifex implements **34 operations** in the EKS `2017-11-01` API model.
Spinifex implements **36 operations** in the EKS `2017-11-01` API model.
### Operations
@@ -40,6 +40,7 @@ Spinifex implements **34 operations** in the EKS `2017-11-01` API model.
| `DescribeCluster` |
| `DescribeIdentityProviderConfig` |
| `DescribeNodegroup` |
| `DescribeUpdate` |
| `DisassociateAccessPolicy` |
| `DisassociateIdentityProviderConfig` |
| `ListAccessEntries` |
@@ -50,6 +51,7 @@ Spinifex implements **34 operations** in the EKS `2017-11-01` API model.
| `ListIdentityProviderConfigs` |
| `ListNodegroups` |
| `ListTagsForResource` |
| `ListUpdates` |
| `TagResource` |
| `UntagResource` |
| `UpdateAccessEntry` |
docs/coverage/elbv2/README.md
+3 -1
View File
@@ -18,7 +18,7 @@ tags:
## Overview
Spinifex implements **34 operations** in the ELBv2 `2015-12-01` API model.
Spinifex implements **36 operations** in the ELBv2 `2015-12-01` API model.
### Two data planes
@@ -40,6 +40,7 @@ The data plane is a system-managed load balancer VM, launched automatically duri
| `DeleteTargetGroup` |
| `DeregisterTargets` |
| `DescribeAccountLimits` |
| `DescribeListenerAttributes` |
| `DescribeListenerCertificates` |
| `DescribeListeners` |
| `DescribeLoadBalancerAttributes` |
@@ -51,6 +52,7 @@ The data plane is a system-managed load balancer VM, launched automatically duri
| `DescribeTargetGroups` |
| `DescribeTargetHealth` |
| `ModifyListener` |
| `ModifyListenerAttributes` |
| `ModifyLoadBalancerAttributes` |
| `ModifyRule` |
| `ModifyTargetGroup` |
docs/coverage/iam/README.md
+4 -1
View File
@@ -18,7 +18,7 @@ tags:
## Overview
Spinifex implements **76 operations** in the IAM `2010-05-08` API model.
Spinifex implements **79 operations** in the IAM `2010-05-08` API model.
### Scope
@@ -38,6 +38,7 @@ All IAM operations are account-scoped. The root user of an account bypasses poli
| `CreateInstanceProfile` |
| `CreateOpenIDConnectProvider` |
| `CreatePolicy` |
| `CreatePolicyVersion` |
| `CreateRole` |
| `CreateUser` |
| `DeleteAccessKey` |
@@ -46,6 +47,7 @@ All IAM operations are account-scoped. The root user of an account bypasses poli
| `DeleteInstanceProfile` |
| `DeleteOpenIDConnectProvider` |
| `DeletePolicy` |
| `DeletePolicyVersion` |
| `DeleteRole` |
| `DeleteRolePolicy` |
| `DeleteUser` |
@@ -91,6 +93,7 @@ All IAM operations are account-scoped. The root user of an account bypasses poli
| `PutUserPolicy` |
| `RemoveRoleFromInstanceProfile` |
| `RemoveUserFromGroup` |
| `SetDefaultPolicyVersion` |
| `TagInstanceProfile` |
| `TagOpenIDConnectProvider` |
| `TagPolicy` |
docs/coverage/internal-report.md
+307 -31
View File
@@ -4,29 +4,56 @@ Generated by `make aws-model-coverage`. This file is not published: it carries t
**Not implemented** is the candidate work. **Not applicable** is declared in `pages.json` and is not work: those operations describe something this platform will not have.
Pinned model source: `aws-sdk-go v1.55.8`.
Pinned model source: `github.com/aws/api-models-aws@9b1a49292744, 2026-09-25`.
## acm
`2015-12-08` — 9 of 15 modelled operations implemented (60.0%), 0 stubbed, 0 not applicable, 0 registered outside the pinned model.
`2015-12-08` — 9 of 40 modelled operations implemented (22.5%), 0 stubbed, 0 not applicable, 0 registered outside the pinned model.
### Not implemented
- `CreateAcmeDomainValidation`
- `CreateAcmeEndpoint`
- `CreateAcmeExternalAccountBinding`
- `DeleteAcmeDomainValidation`
- `DeleteAcmeEndpoint`
- `DeleteAcmeExternalAccountBinding`
- `DescribeAcmeAccount`
- `DescribeAcmeDomainValidation`
- `DescribeAcmeEndpoint`
- `DescribeAcmeExternalAccountBinding`
- `ExportCertificate`
- `GetAccountConfiguration`
- `GetAcmeExternalAccountBindingCredentials`
- `ListAcmeAccounts`
- `ListAcmeDomainValidations`
- `ListAcmeEndpoints`
- `ListAcmeExternalAccountBindings`
- `ListCertificateDomainValidations`
- `ListTagsForResource`
- `PutAccountConfiguration`
- `RenewCertificate`
- `ResendValidationEmail`
- `RevokeAcmeAccount`
- `RevokeAcmeExternalAccountBinding`
- `RevokeCertificate`
- `SearchCertificates`
- `TagResource`
- `UntagResource`
- `UpdateAcmeDomainValidation`
- `UpdateAcmeEndpoint`
- `UpdateCertificateOptions`
## ec2
`2016-11-15` — 125 of 625 modelled operations implemented (20.0%), 0 stubbed, 39 not applicable, 0 registered outside the pinned model.
`2016-11-15` — 127 of 804 modelled operations implemented (15.8%), 0 stubbed, 39 not applicable, 0 registered outside the pinned model.
### Not implemented
- `AcceptAddressTransfer`
- `AcceptCapacityReservationBillingOwnership`
- `AcceptReservedInstancesExchangeQuote`
- `AcceptTransitGatewayClientVpnAttachment`
- `AcceptTransitGatewayMulticastDomainAssociations`
- `AcceptTransitGatewayPeeringAttachment`
- `AcceptTransitGatewayVpcAttachment`
@@ -39,6 +66,8 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `AssignIpv6Addresses`
- `AssignPrivateIpAddresses`
- `AssignPrivateNatGatewayAddress`
- `AssociateApplicationStatusCheck`
- `AssociateCapacityReservationBillingOwner`
- `AssociateClientVpnTargetNetwork`
- `AssociateDhcpOptions`
- `AssociateEnclaveCertificateIamRole`
@@ -46,23 +75,34 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `AssociateIpamByoasn`
- `AssociateIpamResourceDiscovery`
- `AssociateNatGatewayAddress`
- `AssociateRouteServer`
- `AssociateSecurityGroupVpc`
- `AssociateSubnetCidrBlock`
- `AssociateTransitGatewayMulticastDomain`
- `AssociateTransitGatewayPolicyTable`
- `AssociateTransitGatewayRouteTable`
- `AssociateTrunkInterface`
- `AssociateVpcCidrBlock`
- `AttachImageWatermark`
- `AttachVerifiedAccessTrustProvider`
- `AttachVpnGateway`
- `AuthorizeClientVpnIngress`
- `BatchModifyIpamRoutingPolicyRegistrations`
- `BundleInstance`
- `CancelBundleTask`
- `CancelCapacityReservationFleets`
- `CancelConversionTask`
- `CancelDeclarativePoliciesReport`
- `CancelExportTask`
- `CancelImageLaunchPermission`
- `CancelImportTask`
- `CancelSpotFleetRequests`
- `CopyVolumes`
- `CreateApplicationStatusCheck`
- `CreateCapacityManagerDataExport`
- `CreateCapacityReservationBySplitting`
- `CreateCapacityReservationCancellationQuote`
- `CreateCapacityReservationDateChangeQuote`
- `CreateCapacityReservationFleet`
- `CreateClientVpnEndpoint`
- `CreateClientVpnRoute`
@@ -71,21 +111,32 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `CreateCustomerGateway`
- `CreateDefaultSubnet`
- `CreateDefaultVpc`
- `CreateDelegateMacVolumeOwnershipTask`
- `CreateDhcpOptions`
- `CreateFleet`
- `CreateFlowLogs`
- `CreateImageUsageReport`
- `CreateInstanceConnectEndpoint`
- `CreateInstanceEventWindow`
- `CreateInstanceExportTask`
- `CreateInterruptibleCapacityReservationAllocation`
- `CreateIpam`
- `CreateIpamExternalResourceVerificationToken`
- `CreateIpamInternetRegistryAssociation`
- `CreateIpamPolicy`
- `CreateIpamPool`
- `CreateIpamPrefixListResolver`
- `CreateIpamPrefixListResolverTarget`
- `CreateIpamResourceDiscovery`
- `CreateIpamRoutingPolicyRegistration`
- `CreateIpamScope`
- `CreateLocalGatewayRoute`
- `CreateLocalGatewayRouteTable`
- `CreateLocalGatewayRouteTableVirtualInterfaceGroupAssociation`
- `CreateLocalGatewayRouteTableVpcAssociation`
- `CreateLocalGatewayVirtualInterface`
- `CreateLocalGatewayVirtualInterfaceGroup`
- `CreateMacSystemIntegrityProtectionModificationTask`
- `CreateManagedPrefixList`
- `CreateNetworkAcl`
- `CreateNetworkAclEntry`
@@ -95,6 +146,11 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `CreatePublicIpv4Pool`
- `CreateReplaceRootVolumeTask`
- `CreateRestoreImageTask`
- `CreateRouteServer`
- `CreateRouteServerEndpoint`
- `CreateRouteServerPeer`
- `CreateSecondaryNetwork`
- `CreateSecondarySubnet`
- `CreateSnapshots`
- `CreateSpotDatafeedSubscription`
- `CreateStoreImageTask`
@@ -106,9 +162,12 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `CreateTransitGateway`
- `CreateTransitGatewayConnect`
- `CreateTransitGatewayConnectPeer`
- `CreateTransitGatewayMeteringPolicy`
- `CreateTransitGatewayMeteringPolicyEntry`
- `CreateTransitGatewayMulticastDomain`
- `CreateTransitGatewayPeeringAttachment`
- `CreateTransitGatewayPolicyTable`
- `CreateTransitGatewayPolicyTableEntry`
- `CreateTransitGatewayPrefixListReference`
- `CreateTransitGatewayRoute`
- `CreateTransitGatewayRouteTable`
@@ -118,13 +177,18 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `CreateVerifiedAccessGroup`
- `CreateVerifiedAccessInstance`
- `CreateVerifiedAccessTrustProvider`
- `CreateVpcBlockPublicAccessExclusion`
- `CreateVpcEncryptionControl`
- `CreateVpcEndpoint`
- `CreateVpcEndpointConnectionNotification`
- `CreateVpcEndpointServiceConfiguration`
- `CreateVpcPeeringConnection`
- `CreateVpnConcentrator`
- `CreateVpnConnection`
- `CreateVpnConnectionRoute`
- `CreateVpnGateway`
- `DeleteApplicationStatusCheck`
- `DeleteCapacityManagerDataExport`
- `DeleteClientVpnEndpoint`
- `DeleteClientVpnRoute`
- `DeleteCoipCidr`
@@ -133,17 +197,25 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `DeleteDhcpOptions`
- `DeleteFleets`
- `DeleteFlowLogs`
- `DeleteImageUsageReport`
- `DeleteInstanceConnectEndpoint`
- `DeleteInstanceEventWindow`
- `DeleteIpam`
- `DeleteIpamExternalResourceVerificationToken`
- `DeleteIpamInternetRegistryAssociation`
- `DeleteIpamPolicy`
- `DeleteIpamPool`
- `DeleteIpamPrefixListResolver`
- `DeleteIpamPrefixListResolverTarget`
- `DeleteIpamResourceDiscovery`
- `DeleteIpamRoutingPolicyRegistration`
- `DeleteIpamScope`
- `DeleteLocalGatewayRoute`
- `DeleteLocalGatewayRouteTable`
- `DeleteLocalGatewayRouteTableVirtualInterfaceGroupAssociation`
- `DeleteLocalGatewayRouteTableVpcAssociation`
- `DeleteLocalGatewayVirtualInterface`
- `DeleteLocalGatewayVirtualInterfaceGroup`
- `DeleteManagedPrefixList`
- `DeleteNetworkAcl`
- `DeleteNetworkAclEntry`
@@ -154,6 +226,11 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `DeleteNetworkInterfacePermission`
- `DeletePublicIpv4Pool`
- `DeleteQueuedReservedInstances`
- `DeleteRouteServer`
- `DeleteRouteServerEndpoint`
- `DeleteRouteServerPeer`
- `DeleteSecondaryNetwork`
- `DeleteSecondarySubnet`
- `DeleteSpotDatafeedSubscription`
- `DeleteSubnetCidrReservation`
- `DeleteTrafficMirrorFilter`
@@ -161,11 +238,15 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `DeleteTrafficMirrorSession`
- `DeleteTrafficMirrorTarget`
- `DeleteTransitGateway`
- `DeleteTransitGatewayClientVpnAttachment`
- `DeleteTransitGatewayConnect`
- `DeleteTransitGatewayConnectPeer`
- `DeleteTransitGatewayMeteringPolicy`
- `DeleteTransitGatewayMeteringPolicyEntry`
- `DeleteTransitGatewayMulticastDomain`
- `DeleteTransitGatewayPeeringAttachment`
- `DeleteTransitGatewayPolicyTable`
- `DeleteTransitGatewayPolicyTableEntry`
- `DeleteTransitGatewayPrefixListReference`
- `DeleteTransitGatewayRoute`
- `DeleteTransitGatewayRouteTable`
@@ -175,10 +256,13 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `DeleteVerifiedAccessGroup`
- `DeleteVerifiedAccessInstance`
- `DeleteVerifiedAccessTrustProvider`
- `DeleteVpcBlockPublicAccessExclusion`
- `DeleteVpcEncryptionControl`
- `DeleteVpcEndpointConnectionNotifications`
- `DeleteVpcEndpointServiceConfigurations`
- `DeleteVpcEndpoints`
- `DeleteVpcPeeringConnection`
- `DeleteVpnConcentrator`
- `DeleteVpnConnection`
- `DeleteVpnConnectionRoute`
- `DeleteVpnGateway`
@@ -189,11 +273,24 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `DeregisterInstanceEventNotificationAttributes`
- `DeregisterTransitGatewayMulticastGroupMembers`
- `DeregisterTransitGatewayMulticastGroupSources`
- `DescribeAccountVpcEncryptionControl`
- `DescribeAddressTransfers`
- `DescribeApplicationStatus`
- `DescribeApplicationStatusCheckAssociations`
- `DescribeApplicationStatusChecks`
- `DescribeBundleTasks`
- `DescribeByoipCidrs`
- `DescribeCapacityBlockExtensionHistory`
- `DescribeCapacityBlockExtensionOfferings`
- `DescribeCapacityBlockOfferings`
- `DescribeCapacityBlockStatus`
- `DescribeCapacityBlocks`
- `DescribeCapacityManagerDataExports`
- `DescribeCapacityReservationBillingRequests`
- `DescribeCapacityReservationCancellationQuotes`
- `DescribeCapacityReservationDateChangeQuotes`
- `DescribeCapacityReservationFleets`
- `DescribeCapacityReservationTopology`
- `DescribeClientVpnAuthorizationRules`
- `DescribeClientVpnConnections`
- `DescribeClientVpnEndpoints`
@@ -202,6 +299,7 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `DescribeCoipPools`
- `DescribeConversionTasks`
- `DescribeCustomerGateways`
- `DescribeDeclarativePoliciesReports`
- `DescribeDhcpOptions`
- `DescribeExportImageTasks`
- `DescribeExportTasks`
@@ -214,15 +312,26 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `DescribeHostReservationOfferings`
- `DescribeHostReservations`
- `DescribeHosts`
- `DescribeImageReferences`
- `DescribeImageUsageReportEntries`
- `DescribeImageUsageReports`
- `DescribeImportImageTasks`
- `DescribeImportSnapshotTasks`
- `DescribeInstanceConnectEndpoints`
- `DescribeInstanceEventNotificationAttributes`
- `DescribeInstanceEventWindows`
- `DescribeInstanceImageMetadata`
- `DescribeInstanceSqlHaHistoryStates`
- `DescribeInstanceSqlHaStates`
- `DescribeInstanceTopology`
- `DescribeIpamByoasn`
- `DescribeIpamExternalResourceVerificationTokens`
- `DescribeIpamInternetRegistryAssociations`
- `DescribeIpamPolicies`
- `DescribeIpamPoolAllocations`
- `DescribeIpamPools`
- `DescribeIpamPrefixListResolverTargets`
- `DescribeIpamPrefixListResolvers`
- `DescribeIpamResourceDiscoveries`
- `DescribeIpamResourceDiscoveryAssociations`
- `DescribeIpamScopes`
@@ -236,6 +345,7 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `DescribeLocalGateways`
- `DescribeLockedSnapshots`
- `DescribeMacHosts`
- `DescribeMacModificationTasks`
- `DescribeManagedPrefixLists`
- `DescribeNetworkAcls`
- `DescribeNetworkInsightsAccessScopeAnalyses`
@@ -244,15 +354,24 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `DescribeNetworkInsightsPaths`
- `DescribeNetworkInterfaceAttribute`
- `DescribeNetworkInterfacePermissions`
- `DescribeOutpostLags`
- `DescribePrefixLists`
- `DescribePublicIpv4Pools`
- `DescribeReplaceRootVolumeTasks`
- `DescribeReservedInstances`
- `DescribeReservedInstancesModifications`
- `DescribeReservedInstancesOfferings`
- `DescribeRouteServerEndpoints`
- `DescribeRouteServerPeers`
- `DescribeRouteServers`
- `DescribeScheduledInstanceAvailability`
- `DescribeScheduledInstances`
- `DescribeSecondaryInterfaces`
- `DescribeSecondaryNetworks`
- `DescribeSecondarySubnets`
- `DescribeSecurityGroupReferences`
- `DescribeSecurityGroupVpcAssociations`
- `DescribeServiceLinkVirtualInterfaces`
- `DescribeSnapshotAttribute`
- `DescribeSnapshotTierStatus`
- `DescribeSpotDatafeedSubscription`
@@ -268,6 +387,7 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `DescribeTransitGatewayAttachments`
- `DescribeTransitGatewayConnectPeers`
- `DescribeTransitGatewayConnects`
- `DescribeTransitGatewayMeteringPolicies`
- `DescribeTransitGatewayMulticastDomains`
- `DescribeTransitGatewayPeeringAttachments`
- `DescribeTransitGatewayPolicyTables`
@@ -282,34 +402,49 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `DescribeVerifiedAccessInstances`
- `DescribeVerifiedAccessTrustProviders`
- `DescribeVolumeAttribute`
- `DescribeVpcBlockPublicAccessExclusions`
- `DescribeVpcBlockPublicAccessOptions`
- `DescribeVpcEncryptionControls`
- `DescribeVpcEndpointAssociations`
- `DescribeVpcEndpointConnectionNotifications`
- `DescribeVpcEndpointConnections`
- `DescribeVpcEndpointServiceConfigurations`
- `DescribeVpcEndpointServicePermissions`
- `DescribeVpcEndpointServices`
- `DescribeVpcEndpoints`
- `DescribeVpcPeeringConnections`
- `DescribeVpnConcentrators`
- `DescribeVpnConnections`
- `DescribeVpnGateways`
- `DetachImageWatermark`
- `DetachVerifiedAccessTrustProvider`
- `DetachVpnGateway`
- `DisableAddressTransfer`
- `DisableAllowedImagesSettings`
- `DisableApplicationStatusCheckSuppression`
- `DisableCapacityManager`
- `DisableFastLaunch`
- `DisableFastSnapshotRestores`
- `DisableImage`
- `DisableImageBlockPublicAccess`
- `DisableImageDeprecation`
- `DisableImageDeregistrationProtection`
- `DisableInstanceSqlHaStandbyDetections`
- `DisableIpamOrganizationAdminAccount`
- `DisableIpamPolicy`
- `DisableRouteServerPropagation`
- `DisableSnapshotBlockPublicAccess`
- `DisableTransitGatewayRouteTablePropagation`
- `DisableVgwRoutePropagation`
- `DisassociateApplicationStatusCheck`
- `DisassociateCapacityReservationBillingOwner`
- `DisassociateClientVpnTargetNetwork`
- `DisassociateEnclaveCertificateIamRole`
- `DisassociateInstanceEventWindow`
- `DisassociateIpamByoasn`
- `DisassociateIpamResourceDiscovery`
- `DisassociateNatGatewayAddress`
- `DisassociateRouteServer`
- `DisassociateSecurityGroupVpc`
- `DisassociateSubnetCidrBlock`
- `DisassociateTransitGatewayMulticastDomain`
- `DisassociateTransitGatewayPolicyTable`
@@ -317,14 +452,21 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `DisassociateTrunkInterface`
- `DisassociateVpcCidrBlock`
- `EnableAddressTransfer`
- `EnableAllowedImagesSettings`
- `EnableApplicationStatusCheckSuppression`
- `EnableCapacityManager`
- `EnableFastLaunch`
- `EnableFastSnapshotRestores`
- `EnableImage`
- `EnableImageBlockPublicAccess`
- `EnableImageDeprecation`
- `EnableImageDeregistrationProtection`
- `EnableInstanceSqlHaStandbyDetections`
- `EnableIpamInternetRegistryAssociation`
- `EnableIpamOrganizationAdminAccount`
- `EnableIpamPolicy`
- `EnableReachabilityAnalyzerOrganizationSharing`
- `EnableRouteServerPropagation`
- `EnableSnapshotBlockPublicAccess`
- `EnableTransitGatewayRouteTablePropagation`
- `EnableVgwRoutePropagation`
@@ -333,16 +475,26 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `ExportClientVpnClientConfiguration`
- `ExportImage`
- `ExportTransitGatewayRoutes`
- `ExportVerifiedAccessInstanceClientConfiguration`
- `GetActiveVpnTunnelStatus`
- `GetAllowedImagesSettings`
- `GetAssociatedEnclaveCertificateIamRoles`
- `GetAssociatedIpv6PoolCidrs`
- `GetCapacityManagerAttributes`
- `GetCapacityManagerMetricData`
- `GetCapacityManagerMetricDimensions`
- `GetCapacityManagerMonitoredTagKeys`
- `GetCapacityReservationUsage`
- `GetCoipPoolUsage`
- `GetConsoleScreenshot`
- `GetDeclarativePoliciesReportSummary`
- `GetDefaultCreditSpecification`
- `GetEbsDefaultKmsKeyId`
- `GetEnabledIpamPolicy`
- `GetFlowLogsIntegrationTemplate`
- `GetGroupsForCapacityReservation`
- `GetHostReservationPurchasePreview`
- `GetImageAncestry`
- `GetImageBlockPublicAccessState`
- `GetInstanceMetadataDefaults`
- `GetInstanceTpmEkPub`
@@ -352,19 +504,36 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `GetIpamDiscoveredAccounts`
- `GetIpamDiscoveredPublicAddresses`
- `GetIpamDiscoveredResourceCidrs`
- `GetIpamDiscoveredRoutes`
- `GetIpamInternetRegistryAssociationAsns`
- `GetIpamInternetRegistryAssociationCidrs`
- `GetIpamPolicyAllocationRules`
- `GetIpamPolicyOrganizationTargets`
- `GetIpamPoolAllocations`
- `GetIpamPoolCidrs`
- `GetIpamPrefixListResolverRules`
- `GetIpamPrefixListResolverVersionEntries`
- `GetIpamPrefixListResolverVersions`
- `GetIpamResourceCidrs`
- `GetIpamRouteOriginAuthorizations`
- `GetIpamRouteProtectionFindings`
- `GetIpamRoutingPolicyRegistrationDeltas`
- `GetIpamRoutingPolicyRegistrations`
- `GetLaunchTemplateData`
- `GetManagedPrefixListAssociations`
- `GetManagedPrefixListEntries`
- `GetManagedResourceVisibility`
- `GetNetworkInsightsAccessScopeAnalysisFindings`
- `GetNetworkInsightsAccessScopeContent`
- `GetReservedInstancesExchangeQuote`
- `GetRouteServerAssociations`
- `GetRouteServerPropagations`
- `GetRouteServerRoutingDatabase`
- `GetSnapshotBlockPublicAccessState`
- `GetSpotPlacementScores`
- `GetSubnetCidrReservations`
- `GetTransitGatewayAttachmentPropagations`
- `GetTransitGatewayMeteringPolicyEntries`
- `GetTransitGatewayMulticastDomainAssociations`
- `GetTransitGatewayPolicyTableAssociations`
- `GetTransitGatewayPolicyTableEntries`
@@ -372,7 +541,9 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `GetTransitGatewayRouteTableAssociations`
- `GetTransitGatewayRouteTablePropagations`
- `GetVerifiedAccessEndpointPolicy`
- `GetVerifiedAccessEndpointTargets`
- `GetVerifiedAccessGroupPolicy`
- `GetVpcResourcesBlockingEncryptionEnforcement`
- `GetVpnConnectionDeviceSampleConfiguration`
- `GetVpnConnectionDeviceTypes`
- `GetVpnTunnelReplacementStatus`
@@ -383,8 +554,11 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `ImportVolume`
- `ListImagesInRecycleBin`
- `ListSnapshotsInRecycleBin`
- `ListVolumesInRecycleBin`
- `LockSnapshot`
- `ModifyAccountVpcEncryptionControl`
- `ModifyAddressAttribute`
- `ModifyApplicationStatusCheck`
- `ModifyAvailabilityZoneGroup`
- `ModifyCapacityReservation`
- `ModifyCapacityReservationFleet`
@@ -394,22 +568,32 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `ModifyFleet`
- `ModifyHosts`
- `ModifyInstanceCapacityReservationAttributes`
- `ModifyInstanceConnectEndpoint`
- `ModifyInstanceCpuOptions`
- `ModifyInstanceCreditSpecification`
- `ModifyInstanceEventStartTime`
- `ModifyInstanceEventWindow`
- `ModifyInstanceMaintenanceOptions`
- `ModifyInstanceMetadataDefaults`
- `ModifyInstanceNetworkPerformanceOptions`
- `ModifyInstancePlacement`
- `ModifyIpam`
- `ModifyIpamPolicyAllocationRules`
- `ModifyIpamPool`
- `ModifyIpamPoolAllocation`
- `ModifyIpamPrefixListResolver`
- `ModifyIpamPrefixListResolverTarget`
- `ModifyIpamResourceCidr`
- `ModifyIpamResourceDiscovery`
- `ModifyIpamRoutingPolicyRegistration`
- `ModifyIpamScope`
- `ModifyLocalGatewayRoute`
- `ModifyManagedPrefixList`
- `ModifyManagedResourceVisibility`
- `ModifyPrivateDnsNameOptions`
- `ModifyPublicIpDnsNameOptions`
- `ModifyReservedInstances`
- `ModifySecurityGroupRules`
- `ModifyRouteServer`
- `ModifySnapshotAttribute`
- `ModifySnapshotTier`
- `ModifySpotFleetRequest`
@@ -417,6 +601,8 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `ModifyTrafficMirrorFilterRule`
- `ModifyTrafficMirrorSession`
- `ModifyTransitGateway`
- `ModifyTransitGatewayMeteringPolicy`
- `ModifyTransitGatewayPolicyTableEntry`
- `ModifyTransitGatewayPrefixListReference`
- `ModifyTransitGatewayVpcAttachment`
- `ModifyVerifiedAccessEndpoint`
@@ -427,8 +613,12 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `ModifyVerifiedAccessInstanceLoggingConfiguration`
- `ModifyVerifiedAccessTrustProvider`
- `ModifyVolumeAttribute`
- `ModifyVpcBlockPublicAccessExclusion`
- `ModifyVpcBlockPublicAccessOptions`
- `ModifyVpcEncryptionControl`
- `ModifyVpcEndpoint`
- `ModifyVpcEndpointConnectionNotification`
- `ModifyVpcEndpointPayerResponsibility`
- `ModifyVpcEndpointServiceConfiguration`
- `ModifyVpcEndpointServicePayerResponsibility`
- `ModifyVpcEndpointServicePermissions`
@@ -439,17 +629,21 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `ModifyVpnTunnelCertificate`
- `ModifyVpnTunnelOptions`
- `MoveByoipCidrToIpam`
- `MoveCapacityReservationInstances`
- `ProvisionByoipCidr`
- `ProvisionIpamByoasn`
- `ProvisionIpamPoolCidr`
- `ProvisionPublicIpv4PoolCidr`
- `PurchaseCapacityBlock`
- `PurchaseCapacityBlockExtension`
- `PurchaseHostReservation`
- `PurchaseReservedInstancesOffering`
- `PurchaseScheduledInstances`
- `RegisterInstanceEventNotificationAttributes`
- `RegisterTransitGatewayMulticastGroupMembers`
- `RegisterTransitGatewayMulticastGroupSources`
- `RejectCapacityReservationBillingOwnership`
- `RejectTransitGatewayClientVpnAttachment`
- `RejectTransitGatewayMulticastDomainAssociations`
- `RejectTransitGatewayPeeringAttachment`
- `RejectTransitGatewayVpcAttachment`
@@ -457,6 +651,8 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `RejectVpcPeeringConnection`
- `ReleaseHosts`
- `ReleaseIpamPoolAllocation`
- `ReplaceImageCriteriaInAllowedImagesSettings`
- `ReplaceImageInstanceTypeSpecification`
- `ReplaceNetworkAclAssociation`
- `ReplaceNetworkAclEntry`
- `ReplaceTransitGatewayRoute`
@@ -471,12 +667,14 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `RestoreManagedPrefixListVersion`
- `RestoreSnapshotFromRecycleBin`
- `RestoreSnapshotTier`
- `RestoreVolumeFromRecycleBin`
- `RevokeClientVpnIngress`
- `RunScheduledInstances`
- `SearchLocalGatewayRoutes`
- `SearchTransitGatewayMulticastGroups`
- `SearchTransitGatewayRoutes`
- `SendDiagnosticInterrupt`
- `StartDeclarativePoliciesReport`
- `StartNetworkInsightsAccessScopeAnalysis`
- `StartNetworkInsightsAnalysis`
- `StartVpcEndpointServicePrivateDnsVerification`
@@ -485,6 +683,10 @@ Pinned model source: `aws-sdk-go v1.55.8`.
- `UnassignPrivateIpAddresses`
- `UnassignPrivateNatGatewayAddress`
- `UnlockSnapshot`
- `UpdateCapacityManagerMonitoredTagKeys`
- `UpdateCapacityManagerOrganizationsAccess`
- `UpdateInterruptibleCapacityReservationAllocation`
- `ValidateSecurityGroupQuotasForInterface`
- `WithdrawByoipCidr`
### Not applicable — do not build
@@ -560,7 +762,7 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
## ecr
`2015-09-21` — 22 of 47 modelled operations implemented (46.8%), 11 stubbed, 0 not applicable, 3 registered outside the pinned model.
`2015-09-21` — 22 of 58 modelled operations implemented (37.9%), 11 stubbed, 0 not applicable, 3 registered outside the pinned model.
### Not implemented
@@ -569,9 +771,20 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
- `DeletePullThroughCacheRule`
- `DeleteRegistryPolicy`
- `DeleteRepositoryCreationTemplate`
- `DeleteSigningConfiguration`
- `DeregisterPullTimeUpdateExclusion`
- `DescribeImageReplicationStatus`
- `DescribeImageSigningStatus`
- `DescribePullThroughCacheRules`
- `DescribeRepositoryCreationTemplates`
- `GetAccountSetting`
- `GetSigningConfiguration`
- `ListImageReferrers`
- `ListPullTimeUpdateExclusions`
- `PutAccountSetting`
- `PutSigningConfiguration`
- `RegisterPullTimeUpdateExclusion`
- `UpdateImageStorageClass`
- `UpdatePullThroughCacheRule`
- `UpdateRepositoryCreationTemplate`
- `ValidatePullThroughCacheRule`
@@ -607,27 +820,48 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
## ecs
`2014-11-13` — 31 of 56 modelled operations implemented (55.4%), 5 stubbed, 0 not applicable, 3 registered outside the pinned model.
`2014-11-13` — 31 of 77 modelled operations implemented (40.3%), 5 stubbed, 0 not applicable, 3 registered outside the pinned model.
### Not implemented
- `ContinueServiceDeployment`
- `CreateDaemon`
- `CreateExpressGatewayService`
- `CreateTaskSet`
- `DeleteAccountSetting`
- `DeleteAttributes`
- `DeleteDaemon`
- `DeleteDaemonTaskDefinition`
- `DeleteExpressGatewayService`
- `DeleteTaskDefinitions`
- `DeleteTaskSet`
- `DescribeDaemon`
- `DescribeDaemonDeployments`
- `DescribeDaemonRevisions`
- `DescribeDaemonTaskDefinition`
- `DescribeExpressGatewayService`
- `DescribeServiceDeployments`
- `DescribeServiceRevisions`
- `DescribeTaskSets`
- `DiscoverPollEndpoint`
- `ExecuteCommand`
- `GetTaskProtection`
- `ListAttributes`
- `ListDaemonDeployments`
- `ListDaemonTaskDefinitions`
- `ListDaemons`
- `ListServiceDeployments`
- `PutAccountSettingDefault`
- `PutAttributes`
- `RegisterDaemonTaskDefinition`
- `StopServiceDeployment`
- `SubmitAttachmentStateChanges`
- `SubmitContainerStateChange`
- `UpdateCapacityProvider`
- `UpdateClusterSettings`
- `UpdateContainerAgent`
- `UpdateDaemon`
- `UpdateExpressGatewayService`
- `UpdateServicePrimaryTaskSet`
- `UpdateTaskProtection`
- `UpdateTaskSet`
@@ -648,30 +882,42 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
## eks
`2017-11-01` — 34 of 56 modelled operations implemented (60.7%), 0 stubbed, 0 not applicable, 4 registered outside the pinned model.
`2017-11-01` — 36 of 70 modelled operations implemented (51.4%), 0 stubbed, 0 not applicable, 4 registered outside the pinned model.
### Not implemented
- `ActivateCertificateAuthority`
- `AssociateEncryptionConfig`
- `CancelUpdate`
- `CreateCapability`
- `CreateCertificateAuthority`
- `CreateEksAnywhereSubscription`
- `CreateFargateProfile`
- `CreatePodIdentityAssociation`
- `DeleteCapability`
- `DeleteCertificateAuthority`
- `DeleteEksAnywhereSubscription`
- `DeleteFargateProfile`
- `DeletePodIdentityAssociation`
- `DeregisterCluster`
- `DescribeAddonConfiguration`
- `DescribeCapability`
- `DescribeCertificateAuthority`
- `DescribeClusterVersions`
- `DescribeEksAnywhereSubscription`
- `DescribeFargateProfile`
- `DescribeInsight`
- `DescribeInsightsRefresh`
- `DescribePodIdentityAssociation`
- `DescribeUpdate`
- `ListCapabilities`
- `ListCertificateAuthorities`
- `ListEksAnywhereSubscriptions`
- `ListFargateProfiles`
- `ListInsights`
- `ListPodIdentityAssociations`
- `ListUpdates`
- `RegisterCluster`
- `StartInsightsRefresh`
- `UpdateCapability`
- `UpdateEksAnywhereSubscription`
- `UpdatePodIdentityAssociation`
@@ -684,7 +930,7 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
## elasticloadbalancingv2
`2015-12-01` — 34 of 46 modelled operations implemented (73.9%), 0 stubbed, 0 not applicable, 4 registered outside the pinned model.
`2015-12-01` — 36 of 51 modelled operations implemented (70.6%), 0 stubbed, 0 not applicable, 2 registered outside the pinned model.
### Not implemented
@@ -692,74 +938,92 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
- `CreateTrustStore`
- `DeleteSharedTrustStoreAssociation`
- `DeleteTrustStore`
- `DescribeCapacityReservation`
- `DescribeTrustStoreAssociations`
- `DescribeTrustStoreRevocations`
- `DescribeTrustStores`
- `GetResourcePolicy`
- `GetTrustStoreCaCertificatesBundle`
- `GetTrustStoreRevocationContent`
- `ModifyCapacityReservation`
- `ModifyIpPools`
- `ModifyTrustStore`
- `RemoveTrustStoreRevocations`
### Registered outside the pinned model
- `DescribeListenerAttributes`
- `GetLBConfig`
- `LBAgentHeartbeat`
- `ModifyListenerAttributes`
## iam
`2010-05-08` — 76 of 159 modelled operations implemented (47.8%), 0 stubbed, 19 not applicable, 0 registered outside the pinned model.
`2010-05-08` — 79 of 180 modelled operations implemented (43.9%), 0 stubbed, 19 not applicable, 0 registered outside the pinned model.
### Not implemented
- `AcceptDelegationRequest`
- `AcquireRole`
- `AddClientIDToOpenIDConnectProvider`
- `AssociateDelegationRequest`
- `ChangePassword`
- `CreateAccountAlias`
- `CreateDelegationRequest`
- `CreateLoginProfile`
- `CreatePolicyVersion`
- `CreateSAMLProvider`
- `CreateVirtualMFADevice`
- `DeactivateMFADevice`
- `DeleteAccountAlias`
- `DeleteAccountPasswordPolicy`
- `DeleteLoginProfile`
- `DeletePolicyVersion`
- `DeleteRolePermissionsBoundary`
- `DeleteSAMLProvider`
- `DeleteServerCertificate`
- `DeleteUserPermissionsBoundary`
- `DeleteVirtualMFADevice`
- `DisableOrganizationsRootCredentialsManagement`
- `DisableOrganizationsRootSessions`
- `DisableOutboundWebIdentityFederation`
- `EnableMFADevice`
- `EnableOrganizationsRootCredentialsManagement`
- `EnableOrganizationsRootSessions`
- `EnableOutboundWebIdentityFederation`
- `GenerateCredentialReport`
- `GenerateServiceLastAccessedDetails`
- `GetAccessKeyLastUsed`
- `GetAccountAuthorizationDetails`
- `GetAccountPasswordPolicy`
- `GetAccountProperties`
- `GetContextKeysForCustomPolicy`
- `GetContextKeysForPrincipalPolicy`
- `GetCredentialReport`
- `GetDelegationRequest`
- `GetHumanReadableSummary`
- `GetLoginProfile`
- `GetMFADevice`
- `GetOutboundWebIdentityFederationInfo`
- `GetRoleTemplateVersion`
- `GetSAMLProvider`
- `GetServerCertificate`
- `GetServiceLastAccessedDetails`
- `GetServiceLastAccessedDetailsWithEntities`
- `ListAccountAliases`
- `ListDelegationRequests`
- `ListMFADeviceTags`
- `ListMFADevices`
- `ListOrganizationsFeatures`
- `ListPoliciesGrantingServiceAccess`
- `ListSAMLProviderTags`
- `ListSAMLProviders`
- `ListServerCertificateTags`
- `ListServerCertificates`
- `ListVirtualMFADevices`
- `PutAccountProperties`
- `PutRolePermissionsBoundary`
- `PutUserPermissionsBoundary`
- `RejectDelegationRequest`
- `RemoveClientIDFromOpenIDConnectProvider`
- `ResyncMFADevice`
- `SetDefaultPolicyVersion`
- `SendDelegationToken`
- `SetSecurityTokenServicePreferences`
- `SimulateCustomPolicy`
- `SimulatePrincipalPolicy`
@@ -770,6 +1034,7 @@ spot-price: On owned hardware there is no spot-to-on-demand price differential,
- `UntagSAMLProvider`
- `UntagServerCertificate`
- `UpdateAccountPasswordPolicy`
- `UpdateDelegationRequest`
- `UpdateGroup`
- `UpdateLoginProfile`
- `UpdateOpenIDConnectProviderThumbprint`
@@ -814,7 +1079,7 @@ signing-certificates: X.509 signing certificates are an EC2-Classic SOAP credent
## rds
`2014-10-31` — 26 of 162 modelled operations implemented (16.0%), 0 stubbed, 4 not applicable, 5 registered outside the pinned model.
`2014-10-31` — 26 of 164 modelled operations implemented (15.9%), 0 stubbed, 4 not applicable, 5 registered outside the pinned model.
### Not implemented
@@ -870,6 +1135,7 @@ signing-certificates: X.509 signing certificates are an EC2-Classic SOAP credent
- `DescribeDBClusterSnapshotAttributes`
- `DescribeDBClusterSnapshots`
- `DescribeDBLogFiles`
- `DescribeDBMajorEngineVersions`
- `DescribeDBProxies`
- `DescribeDBProxyEndpoints`
- `DescribeDBProxyTargetGroups`
@@ -890,6 +1156,7 @@ signing-certificates: X.509 signing certificates are an EC2-Classic SOAP credent
- `DescribePendingMaintenanceActions`
- `DescribeReservedDBInstances`
- `DescribeReservedDBInstancesOfferings`
- `DescribeServerlessV2PlatformVersions`
- `DescribeSourceRegions`
- `DescribeTenantDatabases`
- `DescribeValidDBInstanceModifications`
@@ -973,45 +1240,48 @@ option-groups: Option groups configure engine add-ons for engines this platform
## s3
`2006-03-01` — 19 of 99 modelled operations implemented (19.2%), 0 stubbed, 6 not applicable, 0 registered outside the pinned model.
`2006-03-01` — 26 of 112 modelled operations implemented (23.2%), 0 stubbed, 6 not applicable, 0 registered outside the pinned model.
### Not implemented
- `CreateBucketMetadataConfiguration`
- `CreateBucketMetadataTableConfiguration`
- `DeleteBucketAnalyticsConfiguration`
- `DeleteBucketCors`
- `DeleteBucketEncryption`
- `DeleteBucketIntelligentTieringConfiguration`
- `DeleteBucketInventoryConfiguration`
- `DeleteBucketLifecycle`
- `DeleteBucketMetadataConfiguration`
- `DeleteBucketMetadataTableConfiguration`
- `DeleteBucketMetricsConfiguration`
- `DeleteBucketOwnershipControls`
- `DeleteBucketPolicy`
- `DeleteBucketReplication`
- `DeleteBucketTagging`
- `DeleteBucketWebsite`
- `DeleteObjectAnnotation`
- `DeleteObjectTagging`
- `DeletePublicAccessBlock`
- `GetBucketAbac`
- `GetBucketAcl`
- `GetBucketAnalyticsConfiguration`
- `GetBucketCors`
- `GetBucketEncryption`
- `GetBucketIntelligentTieringConfiguration`
- `GetBucketInventoryConfiguration`
- `GetBucketLifecycle`
- `GetBucketLifecycleConfiguration`
- `GetBucketLocation`
- `GetBucketLogging`
- `GetBucketMetadataConfiguration`
- `GetBucketMetadataTableConfiguration`
- `GetBucketMetricsConfiguration`
- `GetBucketNotification`
- `GetBucketNotificationConfiguration`
- `GetBucketOwnershipControls`
- `GetBucketPolicy`
- `GetBucketPolicyStatus`
- `GetBucketReplication`
- `GetBucketTagging`
- `GetBucketVersioning`
- `GetBucketWebsite`
- `GetObjectAcl`
- `GetObjectAnnotation`
- `GetObjectAttributes`
- `GetObjectLegalHold`
- `GetObjectLockConfiguration`
@@ -1024,33 +1294,36 @@ option-groups: Option groups configure engine add-ons for engines this platform
- `ListBucketInventoryConfigurations`
- `ListBucketMetricsConfigurations`
- `ListDirectoryBuckets`
- `ListObjectVersions`
- `ListObjectAnnotations`
- `PutBucketAbac`
- `PutBucketAcl`
- `PutBucketAnalyticsConfiguration`
- `PutBucketCors`
- `PutBucketEncryption`
- `PutBucketIntelligentTieringConfiguration`
- `PutBucketInventoryConfiguration`
- `PutBucketLifecycle`
- `PutBucketLifecycleConfiguration`
- `PutBucketLogging`
- `PutBucketMetricsConfiguration`
- `PutBucketNotification`
- `PutBucketNotificationConfiguration`
- `PutBucketOwnershipControls`
- `PutBucketPolicy`
- `PutBucketReplication`
- `PutBucketTagging`
- `PutBucketVersioning`
- `PutBucketWebsite`
- `PutObjectAcl`
- `PutObjectAnnotation`
- `PutObjectLegalHold`
- `PutObjectLockConfiguration`
- `PutObjectRetention`
- `PutObjectTagging`
- `PutPublicAccessBlock`
- `RenameObject`
- `RestoreObject`
- `SelectObjectContent`
- `UpdateBucketMetadataAnnotationTableConfiguration`
- `UpdateBucketMetadataInventoryTableConfiguration`
- `UpdateBucketMetadataJournalTableConfiguration`
- `UpdateObjectEncryption`
### Not applicable — do not build
@@ -1074,11 +1347,14 @@ request-payment: Requester Pays shifts transfer charges to the caller, which nee
## sts
`2011-06-15` — 5 of 8 modelled operations implemented (62.5%), 0 stubbed, 0 not applicable, 0 registered outside the pinned model.
`2011-06-15` — 5 of 11 modelled operations implemented (45.5%), 0 stubbed, 0 not applicable, 0 registered outside the pinned model.
### Not implemented
- `AssumeRoleWithSAML`
- `AssumeRoot`
- `DecodeAuthorizationMessage`
- `GetDelegatedAccessToken`
- `GetFederationToken`
- `GetWebIdentityToken`
docs/coverage/s3/README.md
+8 -1
View File
@@ -18,7 +18,7 @@ tags:
## Overview
Predastore implements **19 operations** in the S3 `2006-03-01` API model.
Predastore implements **26 operations** in the S3 `2006-03-01` API model.
### Predastore serves this endpoint
@@ -40,16 +40,23 @@ That behaviour is measured separately, against the `ceph/s3-tests` suite Ceph RG
| `CreateBucket` |
| `CreateMultipartUpload` |
| `DeleteBucket` |
| `DeleteBucketTagging` |
| `DeleteObject` |
| `DeleteObjects` |
| `GetBucketLocation` |
| `GetBucketTagging` |
| `GetBucketVersioning` |
| `GetObject` |
| `HeadBucket` |
| `HeadObject` |
| `ListBuckets` |
| `ListMultipartUploads` |
| `ListObjectVersions` |
| `ListObjects` |
| `ListObjectsV2` |
| `ListParts` |
| `PutBucketTagging` |
| `PutBucketVersioning` |
| `PutObject` |
| `UploadPart` |
| `UploadPartCopy` |
docs/hardware/cisco/llm-serving/README.md
-2
View File
@@ -22,8 +22,6 @@ resources:
url: "/hardware/cisco/platform-benchmark"
- title: "Vision pipeline (companion document)"
url: "/hardware/cisco/vision-pipeline"
- title: "Benchmark and workload materials (CISCO-refarch)"
url: "https://github.com/tomnewton-mulga/CISCO-refarch"
- title: "vLLM"
url: "https://docs.vllm.ai"
- title: "Qwen2.5-7B-Instruct"
docs/hardware/cisco/platform-benchmark/README.md
+31 -29
View File
@@ -22,8 +22,6 @@ resources:
url: "/docs/launching-instances"
- title: "GPU Passthrough"
url: "/docs/gpu-passthrough"
- title: "cisco-ucs-platform-benchmark (scripts and raw results)"
url: "https://github.com/tomnewton-mulga/CISCO-refarch"
sections:
- overview
- prerequisites
@@ -239,42 +237,46 @@ sequential repetitions per guest.
| Guest | 16K mixed read/write | 16K random read | 128K mixed read/write | 128K random read |
|---|---:|---:|---:|---:|
| GPU | 75 / 32 MiB/s | 75 MiB/s | 210 / 91 MiB/s | 332 MiB/s |
| CPU1 | 74 / 32 MiB/s | 76 MiB/s | 218 / 94 MiB/s | 333 MiB/s |
| CPU2 | 69 / 30 MiB/s | 107 MiB/s | 226 / 98 MiB/s | 405 MiB/s |
| GPU | 107 / 46 MiB/s | 107 MiB/s | 232 / 101 MiB/s | 376 MiB/s |
| CPU1 | 80 / 35 MiB/s | 80 MiB/s | 226 / 98 MiB/s | 371 MiB/s |
| CPU2 | 79 / 34 MiB/s | 80 MiB/s | 228 / 98 MiB/s | 372 MiB/s |
The 128K guest random-read numbers (332–405 MiB/s) sit between the [AWS EBS gp3](https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html)
baseline (125 MiB/s) and its provisioned maximum (1,000 MiB/s) — a meaningful result
given that gp3 baseline is what most AWS operators treat as the default floor for
general-purpose block storage. The 16K numbers (75–107 MiB/s) land near that baseline.
The gap to the raw NVMe host performance (the [KIOXIA CD8P](https://americas.kioxia.com/en-us/business/ssd/data-center-ssd/cd8p-r.html) delivers over 4,000 MiB/s
host-side at comparable queue depths) is not architectural — it is the current
single-queue virtio-blk attach path and unthreaded NBD backend, both of which have
a clear source-level fix. These numbers are the baseline to improve against in future
releases, not the ceiling.
<img src="../../../../.github/assets/images/cisco-ucs-platform-benchmark/guest-storage-throughput.svg" alt="Guest block storage throughput by block size and access pattern">
The 128K guest random-read numbers (371–376 MiB/s) sit between the [AWS EBS gp3](https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html)
baseline (125 MiB/s) and its provisioned maximum (1,000 MiB/s). The 16K numbers (79–107 MiB/s) land near that baseline.
128K mixed I/O improves throughput but carries a high p99 latency envelope (roughly
0.47–0.75 s); size queues and write patterns accordingly for latency-sensitive
0.45–0.80 s); size queues and write patterns accordingly for latency-sensitive
applications.
Predastore S3 validation used 1 GiB objects with three sequential and three
distributed-concurrent repetitions per host:
Predastore S3 validation used 1 GiB objects. The AWS CLI splits any download above 8 MiB into concurrent ranged GETs while a plain client issues a single streaming GET, so the sweep measures both patterns with the same client (presigned `curl`) on the host and on each guest — a single-stream PUT/GET, and a chunked/concurrent Range GET (8 MiB parts, 10 in flight) that mirrors the AWS CLI's default download path — with three repetitions each:
| Workload | Write, median | Read, median |
| Client | Single-stream read | Chunked/concurrent read |
|---|---:|---:|
| One host client | 121.9 MiB/s | 213.2 MiB/s |
| Three-host aggregate | 184.2 MiB/s | 381.7 MiB/s |
| Host | 340 MiB/s | 462 MiB/s |
| GPU guest | 399 MiB/s | 391 MiB/s |
| CPU1 guest | 525 MiB/s | 350 MiB/s |
| CPU2 guest | 517 MiB/s | 351 MiB/s |
The single-host sequential read (213 MiB/s) is a credible result for a single-node
S3-compatible store at low-to-moderate concurrency with 1 GiB objects — throughput at
this scale is typically network- and per-request-latency-bound rather than disk-bound,
and 213 MiB/s uses roughly 17% of the 25GbE storage fabric. The three-node distributed read (381.7 MiB/s, 1.79× single-host) confirms
Predastore is distributing reads across the cluster. The write scaling ratio (184.2 vs
121.9 MiB/s, 1.51×) is narrower than read, so worth tracking as a leading indicator of
backend contention as the cluster grows and write load increases.
<img src="../../../../.github/assets/images/cisco-ucs-platform-benchmark/predastore-s3-read-by-client.svg" alt="Predastore S3 read throughput by client, single-stream versus chunked/concurrent GET">
All sequential S3 checksum validations passed. Full methodology, raw fio JSON and S3
metrics are in the [benchmark repository](https://github.com/tomnewton-mulga/CISCO-refarch).
Single-stream writes land at 282–304 MiB/s across the host and all three guests. Under three simultaneous clients — one per physical node — the cluster sustains 640 MiB/s aggregate read and 538 MiB/s aggregate write from the host, and 665 / 515 MiB/s from the guests.
With the client held constant, single-stream host and guest reads fall in the same 340–525 MiB/s band, and aggregate read reaches 640–665 MiB/s across three concurrent clients. The chunked/concurrent Range-GET results vary by individual client rather than tracking host-versus-guest; that variation has no confirmed explanation yet and remains under investigation. All sequential S3 checksum validations passed.
### 6. Network benchmark
Network throughput was measured with iperf3 over TCP, with a 10-second run after a 3-second warmup, five repetitions at 1, 2, 4, 8 and 16 parallel streams, on two paths: host-to-host across the dedicated 25GbE storage fabric (VLAN 1336), and guest-to-guest across the OVN overlay that connects instances.
| Path | Single stream | 2–16 streams |
|---|---:|---:|
| Host physical fabric (25GbE) | 23.4 Gbit/s | 23.5 Gbit/s |
| Guest overlay (OVN) | 22.0 Gbit/s | 22.5 Gbit/s |
<img src="../../../../.github/assets/images/cisco-ucs-platform-benchmark/network-goodput-vs-streams.svg" alt="Network TCP goodput versus parallel stream count, host physical fabric versus guest overlay">
TCP goodput on a 25GbE link settles near 23.5 Gbit/s once framing and protocol overhead are removed, so the host path runs at effectively line rate with zero retransmissions. The guest overlay sustains 22.0 Gbit/s on a single stream, rising to 22.5 Gbit/s at two streams and holding flat through sixteen — 95–96% of the physical fabric across the full range.
## What this architecture unlocks
docs/hardware/cisco/vision-pipeline/README.md
-2
View File
@@ -14,8 +14,6 @@ tags:
- predastore
- terraform
resources:
- title: "Benchmark and workload materials (CISCO-refarch)"
url: "https://github.com/tomnewton-mulga/CISCO-refarch"
- title: "Multi-Node Install"
url: "/docs/install-multi-node"
- title: "VPC Networking"
docs/identity/imds/README.md
+5 -2
View File
@@ -130,15 +130,18 @@ Commonly used paths under `/latest/meta-data/`:
| `security-groups` | Security group names, one per line |
| `placement/availability-zone` | Availability zone |
| `placement/region` | Region (AZ with trailing letter stripped) |
| `services/domain`, `services/partition` | `amazonaws.com` / `aws` |
| `services/domain`, `services/partition` | The cluster's configured services domain (default `services.internal`) / `aws` |
| `public-keys/0/openssh-key` | Launch key pair's SSH public key; 404 if no key pair or the key was deleted |
| `iam/info` | Instance profile ARN and ID; 404 if no profile |
| `iam/security-credentials/<role>` | Temporary role credentials (see below) |
| `network/interfaces/macs/<mac>/...` | Primary interface subtree: `interface-id`, `owner-id`, `subnet-id`, `vpc-id`, `local-ipv4s`, `security-group-ids`, `subnet-ipv4-cidr-block`, `vpc-ipv4-cidr-block`, and more |
| `block-device-mapping/` | `ami` and `root` (both name the boot device) plus one `ebsN` per additional attached EBS volume, numbered from 1 in attachment order |
`services/domain` is the slot an AWS SDK's default endpoint resolution substitutes into `{service}.{region}.{domain}`. Real AWS varies this per partition: standard AWS returns `amazonaws.com`, China returns `amazonaws.com.cn`, and GovCloud differs again. Spinifex is effectively its own partition, so it returns its own configured domain here rather than `amazonaws.com` — an SDK left on default endpoint resolution now resolves a Spinifex-served name instead of silently reaching real AWS. `services/partition` still reports `aws`, since AWS SDKs carry hardcoded partition tables and a novel partition name would not match any of them. Note that this fixes **addressing**, not **trust**: a stock guest image does not trust the Spinifex CA, so a discovered HTTPS endpoint still needs `AWS_CA_BUNDLE` set or the CA installed.
The `network/interfaces/macs/` subtree covers the **primary interface only**; querying another MAC returns 404 (multi-ENI metadata is deferred).
Paths that intentionally return **404**: `tags/instance/*`, `block-device-mapping/*`, `placement/{group-name,partition-number,availability-zone-id,host-id}`, `instance-action`, and `spot/{instance-action,termination-time}` (404 is the correct "no interruption scheduled" answer for spot pollers).
Paths that intentionally return **404**: `tags/instance/*`, `placement/{group-name,partition-number,availability-zone-id,host-id}`, `instance-action`, and `spot/{instance-action,termination-time}` (404 is the correct "no interruption scheduled" answer for spot pollers).
## User Data
docs/install/install-airgapped/README.md
+15 -7
View File
@@ -72,14 +72,19 @@ curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" \
-o awscliv2.zip
```
Mirror the cloud image you intend to run as guest VMs:
Mirror the cloud image you intend to run as guest VMs, together with the sums file it was published with. Keep the upstream filename, because the import looks the image up in the sums file by name:
```bash
mkdir -p images
curl -fsSL "https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2" \
-o images/debian-13-amd64.qcow2
cd images
curl -fsSLO "https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2"
curl -fsSLO "https://cloud.debian.org/images/cloud/trixie/latest/SHA512SUMS"
sha512sum -c --ignore-missing SHA512SUMS
cd ..
```
The sums file only proves the image matches it; a sums file fetched over the same channel as the image can be tampered with alongside it. Where the vendor publishes a signature for the sums file (for example Ubuntu's `SHA256SUMS.gpg`), download it too and check it with `gpg --verify` against the vendor's signing key on this connected machine.
## Step 3. Assemble Transfer Media
```bash
@@ -88,7 +93,7 @@ cp spinifex-${TAG}-linux-${ARCH}.tar.gz /media/spinifex-deploy/tarball/
cp setup.sh /media/spinifex-deploy/
cp /var/cache/apt/archives/*.deb /media/spinifex-deploy/apt-packages/
cp awscliv2.zip /media/spinifex-deploy/aws/
cp images/*.qcow2 /media/spinifex-deploy/images/
cp images/*.qcow2 images/SHA512SUMS /media/spinifex-deploy/images/
```
## Step 4. Install on the Air-Gapped Target
@@ -139,13 +144,16 @@ sudo systemctl start spinifex.target
## Step 8. Import Cloud Images
Register the pre-staged image with Spinifex:
Register the pre-staged image with Spinifex, verifying it against the sums file carried across with it:
```bash
sudo spx admin images import --file /mnt/usb/images/debian-13-amd64.qcow2 \
sudo spx admin images import --file /mnt/usb/images/debian-13-genericcloud-amd64.qcow2 \
--checksum /mnt/usb/images/SHA512SUMS \
--distro debian --version 13 --arch x86_64 --boot-mode uefi
```
A digest mismatch exits before anything is written to storage. On success the import prints the verified digest, which `spx admin images describe --image-id <ami>` shows again later.
## Step 9. Verify
```bash
@@ -185,4 +193,4 @@ sudo INSTALL_SPINIFEX_TARBALL=/mnt/usb/tarball/spinifex-...tar.gz \
### Image Import Fails
`spx admin images import --file` requires the distro/version/arch/boot-mode flags so the image registers in the catalogue. If the import succeeds but `aws ec2 describe-images` returns empty, check `journalctl -u spinifex-daemon -f` for predastore upload errors.
`spx admin images import --file` requires the distro/version/arch/boot-mode flags so the image registers in the catalogue. A `checksum entry for image filename not found` error means the image was renamed after download; restore the upstream filename so it matches its line in the sums file. If the import succeeds but `aws ec2 describe-images` returns empty, check `journalctl -u spinifex-daemon -f` for predastore upload errors.
docs/install/install-multi-node/README.md
+140 -116
View File
@@ -30,13 +30,13 @@ resources:
- [Overview](#overview)
- [Cluster sizing](#cluster-sizing)
- [How the cluster's own state is replicated](#how-the-clusters-own-state-is-replicated)
- [Hardware](#hardware)
- [Network requirements](#network-requirements)
- [Prerequisites](#prerequisites)
- [Instructions](#instructions)
- [Verifying the cluster](#step-6-verify-the-cluster)
- [Firewall and cluster membership](#firewall-and-cluster-membership)
- [How multi-node storage works](#how-multi-node-storage-works)
- [Setting up your cluster](#step-7-set-up-your-cluster)
- [Troubleshooting](#troubleshooting)
---
@@ -66,11 +66,82 @@ A Spinifex cluster distributes services across multiple servers for high availab
| **VPC networking** — OVN | Control-plane databases run clustered, surviving the loss of any one node. |
| **Object storage** — Predastore (S3) | Objects are erasure coded `RS(2,1)`, surviving the loss of any one node's shards. |
| **Block storage** — Viperblock (EBS) | Volumes are stored in Predastore, so they inherit the same durability. |
| **Cluster state** — NATS JetStream | Every key-value bucket is replicated onto all three nodes, so the control plane keeps its own state through the loss of any one. |
On one or two servers none of that holds. OVN runs standalone on the first node, and the storage metadata quorum has no majority to lose. If that node goes down, running instances keep full networking — but nothing can *change*: no new VPCs, no launches, no security group edits. See [OVN control plane on multi-node clusters](/docs/vpc-networking#ovn-control-plane-on-multi-node-clusters).
Servers beyond the third run the full set of services — storage, gateway and networking agents — and add their capacity to the pool. What they do not do is join the OVN database cluster, which stays at three members, so write latency there stays flat as the cluster grows.
### How the cluster's own state is replicated
Guest data has an obvious home: volumes live in Viperblock, objects in Predastore, both erasure coded across the cluster. The cluster's own bookkeeping has a less obvious one, and it matters just as much. Every instance record, every VPC and subnet, the server roster, IAM users and access keys, quotas and DNS records are kept in **NATS JetStream key-value buckets** — Spinifex's control-plane database, replicated across the servers you installed rather than sitting on any one of them. A cluster that has lost its state has lost the ability to describe, launch, terminate or route anything, even while every guest is still running.
**A bucket is replicated across the largest odd number of nodes the cluster has, up to five.** That number is decided once, by the `--nodes` count you give `spx admin init` in Step 4, and every service on every node reads it back out of `/etc/spinifex/spinifex.toml` before it creates anything.
| Servers | Replicas per bucket | Quorum | Survives |
|---|---|---|---|
| 1 | 1 | 1 | nothing — a single node is a single point of failure by definition |
| 2 | — | — | rejected by `spx admin init`; see below |
| 3 | 3 | 2 | any 1 node lost |
| 4 | 3 | 2 | any 1 node lost |
| 5 | 5 | 3 | any 2 nodes lost |
| 10 | 5 | 3 | any 2 of the 5 nodes holding each bucket |
**The count is always odd, and that is why four servers replicate three ways rather than four.** Agreement is by majority, and an even number of copies does not buy you one: four copies cost a fourth write every time and still survive exactly one loss, the same as three. On four servers NATS spreads which three nodes hold each bucket, so all four carry their share and none of them is idle.
Five is JetStream's own ceiling on a stream's replica count, not a choice of ours. Past five servers the buckets stay at five and NATS spreads which five nodes hold each one, so the cluster keeps growing while the cost of a write does not.
**Two servers is refused rather than allowed with a warning.** There is no replica count that works on two: one disappears with the node holding it, and two stop accepting writes when either member goes, because neither survivor is a majority on its own. OVN and the storage metadata quorum have the same problem on two servers, so `spx admin init --nodes 2` fails and tells you to use one server or three.
#### Worked example — three servers
Say you ran Step 4 with `--nodes 3` on `node1`, `node2` and `node3`. Every bucket the cluster creates is held by all three servers, with one of them responsible for coordinating changes to it:
```
KV bucket "spinifex-instance-state" KV bucket "spinifex-vpcd-reconcile"
┌──────────┬──────────┬──────────┐ ┌──────────┬──────────┬──────────┐
│ node1 │ node2 │ node3 │ │ node1 │ node2 │ node3 │
│ leader │ follower │ follower │ │ follower │ leader │ follower │
└──────────┴──────────┴──────────┘ └──────────┴──────────┴──────────┘
quorum = 2 of 3 quorum = 2 of 3
```
Three things follow from that picture, and they are the whole reason three servers is the floor:
- **A change is not confirmed until most of the cluster has it.** When you launch an instance, the API does not report success until at least two of the three servers have written the record to disk. Losing a server afterwards cannot lose that instance.
- **Losing a server costs a pause, not the data.** If `node1` goes down, `node2` and `node3` are still a majority. They take over the buckets `node1` was responsible for — seconds on a healthy cluster — and the API keeps answering throughout. Nothing becomes unreadable, because nothing lived only on `node1`.
- **The cluster keeps repairing itself.** Spinifex runs background workers that fix up VPC state, enforce quotas and publish DNS. Only one server runs each at a time, and they claim that right by writing to a bucket. If those buckets were not replicated, a server failure would stop the repair work at precisely the moment there was repair work to do.
Contrast that with a bucket on **one** server. It lives wherever it happened to be created, which is not recorded in any config file you can inspect. Stopping, rebooting or losing that one server makes the bucket unreachable from *every* node at once — a single-server event becomes a cluster-wide outage, on a server nobody chose.
#### What you see when a server fails
Majority is not an implementation detail. It is what stops two halves of a damaged cluster from both believing they are in charge, and it is the reason the minimum is three servers rather than two.
**With one server down, the cluster carries on.** `node2` and `node3` are two of three. Instances launch, VPCs change, the API answers normally, and there is nothing to do but replace the failed server. What you have lost is the *next* failure: a second server down leaves no majority and the cluster stops accepting changes until one comes back.
**A server that is cut off stops accepting changes, on purpose.** If `node1` is not down but isolated — a failed switch, a misconfigured port — it is one server out of three, which is not a majority, so it will not record anything. An operator working on `node1` who tries to launch an instance gets a failure rather than a success. That is the behaviour you want: if `node1` accepted changes while `node2` and `node3` were also accepting them, you would end up with two different versions of the same cluster and no way to merge them. Refusing the isolated side is what makes the surviving side worth trusting.
Everything else on `node1` keeps running. Guests continue to execute and their volumes keep serving. What stops is anything that has to be recorded: launching, terminating, attaching a volume, changing a security group. Storage writes fail alongside it, so a guest on the isolated server stalls rather than quietly diverging from the rest of the cluster.
**Be careful reading state from an isolated server.** A server that is cut off can still answer `describe-instances` from its own copy of the data — frozen at the moment it lost contact, with nothing in the answer to say so. **Run operator commands from a server you know is with the majority**, and if one server's answers disagree with its peers', believe the peers.
**A returning server catches up on its own.** Bring `node1` back and it synchronises from the others automatically: no command to run, no restart, no repair step. (`spx admin kv replicas --repair` exists for a different situation — a bucket configured with too few copies — not a server that is simply behind.)
#### Growing an existing cluster
Adding a fourth server does not, on its own, re-replicate the buckets that already exist — the nodes that hold them are perfectly healthy, so nothing prompts a change. Spinifex raises them for you at the next service start: each service checks every bucket it opens against the cluster's current node count and raises any that are short. It never lowers one.
**A config that names a node before that node is serving is safe.** It is the ordinary middle of growing a cluster, and services start normally through it: a bucket that cannot be raised yet keeps working at the count it has, and the node logs `Could not raise KV bucket to the cluster's replica count`. The raise is retried every time a service opens the bucket, so it completes on its own once the new server is in the cluster.
To do it immediately rather than waiting for a restart, run the audit with `--repair` from any node:
```bash
sudo spx admin kv replicas --repair
```
It changes nothing about a bucket except the replica count, never lowers one, and is safe to run twice or on a healthy cluster.
### Hardware
Per server, for the three-node minimum:
@@ -103,14 +174,14 @@ Open between the three OVN database servers (servers 1 to 3) only:
|---|---|
| TCP 6643, 6644 | OVN database clustering |
Predastore uses the same three ports on every server, so the surface does not widen as the cluster grows. See [How multi-node storage works](#how-multi-node-storage-works).
Predastore uses the same three ports on every server, so the surface does not widen as the cluster grows. See [Predastore Distributed Storage](/docs/predastore-distributed-storage).
Those are the ports your **network** has to permit between the servers — switches, upstream firewalls, and anything else in the path.
Spinifex's own **host** firewall is separate, and already carries this policy. It ships armed on the ISO path and off on the binary installer path. Two things follow from that, both of which the ISO box in Step 2 acts on:
Spinifex's own **host** firewall is separate, and already carries this policy — see [Host Firewall](/docs/host-firewall). It ships armed on the ISO path and off on the binary installer path. Two things follow from that, both of which the ISO box in Step 2 acts on:
- **The formation port needs nothing from you.** `spx admin init` opens 4432 to any source for the length of the formation window and closes it again afterwards, because a node dialling in to join is not a peer yet. The handshake behind it is TLS 1.3 with a bearer token.
- **The rest of the cluster plane is peer-scoped**, and that includes the OVN database ports Step 3 uses. Nodes that do not yet know each other cannot reach them, which is why an ISO-installed node has its firewall taken down before Step 3 and re-armed after Step 6.
- **The rest of the cluster plane is peer-scoped**, and that includes the OVN database ports Step 3 uses. Nodes that do not yet know each other cannot reach them, which is why an ISO-installed node has its firewall taken down before Step 3 and re-armed once the cluster is verified.
## Prerequisites
@@ -172,7 +243,7 @@ Adding a fourth server or more? Export `SPINIFEX_NODE4` and so on alongside thes
>
> Each node's firewall currently trusts only itself, because that is the whole cluster as far as it knows, and the cluster plane is peer-scoped — so the OVN database connections Step 3 makes between servers are dropped. Stopping `spinifex.target` is not enough: the firewall lives in the kernel and outlives the services.
>
> Turn it back on after Step 6 — see [Firewall and cluster membership](#firewall-and-cluster-membership).
> Turn it back on once the cluster is verified — see [Host Firewall](/docs/host-firewall).
## Step 3. Set Up OVN Networking
@@ -253,6 +324,8 @@ sudo spx admin init --force \
`--nodes 3` is the number of servers init waits for. Set it to your total node count if you are building a larger cluster.
It is also what decides how many nodes the cluster's JetStream state is replicated across, so it is worth getting right rather than raising later — see [How the cluster's own state is replicated](#how-the-clusters-own-state-is-replicated). Every joining server takes the count from server 1, so it is chosen once, here. `--nodes 2` is refused: use 1 for a single server, or 3 and up for a cluster that can survive losing one.
IPsec encrypts the Geneve overlay between servers and is on by default. Joining servers take the setting from server 1, so it is chosen once, on init. On servers that share a trusted private link, `--ipsec=false` leaves the overlay unencrypted in exchange for considerably higher throughput between instances.
The init output displays the join command including the token:
@@ -361,7 +434,40 @@ The top table is per-node CPU, memory and GPU usage. The bottom table is what th
If capacity looks like a single server rather than the sum of your nodes, the others have not joined.
**3. The AWS API answers.**
**3. The cluster's own state is replicated across every node.**
```bash
sudo spx admin kv replicas
```
```
spinifex@node1:~$ sudo spx admin kv replicas
BUCKET REPLICAS WANT ONLINE STATUS LEADER HELD BY
spinifex-cluster-state 3 3 3 ok spinifex-nats-node1 spinifex-nats-node1,spinifex-nats-node2,spinifex-nats-node3
spinifex-dns-reconcile 3 3 3 ok spinifex-nats-node2 spinifex-nats-node2,spinifex-nats-node1,spinifex-nats-node3
spinifex-iam 3 3 3 ok spinifex-nats-node3 spinifex-nats-node3,spinifex-nats-node1,spinifex-nats-node2
spinifex-instance-state 3 3 3 ok spinifex-nats-node1 spinifex-nats-node1,spinifex-nats-node2,spinifex-nats-node3
spinifex-quota-reconcile 3 3 3 ok spinifex-nats-node2 spinifex-nats-node2,spinifex-nats-node3,spinifex-nats-node1
spinifex-terminated-instances 3 3 3 ok spinifex-nats-node3 spinifex-nats-node3,spinifex-nats-node2,spinifex-nats-node1
spinifex-vpcd-reconcile 3 3 3 ok spinifex-nats-node2 spinifex-nats-node2,spinifex-nats-node1,spinifex-nats-node3
47 buckets, 0 under-replicated, 0 without quorum
```
Server names are the node name behind a `spinifex-nats-` prefix, which is what NATS itself calls each server.
What to check:
- **`0 under-replicated, 0 without quorum`.** These are different problems. Under-replicated means a bucket lives on fewer nodes than depend on it — see [Buckets report as under-replicated](#buckets-report-as-under-replicated), and `--repair` is the fix. Without quorum means a bucket is not able to accept a write right now, whatever its replica count says, and no repair helps until you find out why a node is not answering.
- **`WANT` is the largest odd number of nodes you have**, capped at five — 3 on three or four servers, 5 on five or more. If it says 1 on a three-server cluster, the node the command ran on is not configured for three: check `--nodes` in Step 4 and the `[nodes.*]` sections of that host's `/etc/spinifex/spinifex.toml`.
- **`ONLINE` equals `REPLICAS`.** It counts the replicas that are caught up and reachable, so a bucket at `3` replicas with `ONLINE 2` is one node away from being unavailable and needs looking at before anything else is done to the cluster.
- **`LEADER` is spread across the cluster** rather than parked on one node, and `HELD BY` names as many distinct servers as `REPLICAS`.
Exit status is what a deployment gate should read rather than the output: `0` healthy, `1` something is under-replicated, `2` something has no quorum or the report could not be produced at all.
Run it on every node, not just one. It reads cluster-global Raft metadata, so all of them should give the same answer; one that disagrees is not fully part of the cluster.
**4. The AWS API answers.**
```bash
export AWS_PROFILE=spinifex
@@ -370,117 +476,13 @@ aws ec2 describe-instance-types
A list of instance types means the gateway, IAM and the cluster behind them are all working.
**4. Every server holds the same cluster state.**
**Congratulations! Your multi-server Spinifex cluster is installed.**
Services report Ready even over a cluster whose JetStream replicas disagree, so check the replicas themselves. On **every server**, take a digest of its local store:
## Step 7. Set Up Your Cluster
```bash
sudo spx admin kv digest --json --seqs > kv-digest-$(hostname).json
```
The cluster is running, but it holds nothing yet — no machine images, no networks, no instances.
Copy the files to one server and compare them:
```bash
spx admin kv compare kv-digest-*.json
```
```
compared 3 digests: node1@10:02:11 node2@10:02:14 node3@10:02:18
ok KV_spinifex-iam-users [node1,node2,node3]
identical, seqs 1-14
...
42 streams: 42 consistent, 0 diverged
```
The last line must report `0 diverged`, and the command exits non-zero otherwise. The digests are copied from live stores a few seconds apart, so a write landing between two copies can briefly show as a difference: take fresh digests and compare again before acting on one. A difference that persists means the cluster adopted a server's old store as a replica. Do not put that cluster in service — reset the servers and form it again.
A replica that is merely behind is not reported, and neither is a message that one server has already replaced with a newer write to the same key. `spx admin kv compare --help` lists exactly what counts as divergence.
**Congratulations! Your Spinifex cluster is installed.**
Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import an AMI, create a VPC, and launch your first instance.
## Firewall and Cluster Membership
Spinifex ships an optional host firewall. It divides the node's ports into two groups:
| Group | Ports | Who can reach them |
|---|---|---|
| **Public** | SSH, 443, 3000 (console), 8443 (S3), 9999 (AWS gateway), 53 (DNS) | anyone |
| **Internal** | OVN, NATS, formation, Geneve and the rest of the cluster plane | **cluster members only** |
The internal group is the point. Before this existed, OVN and NATS were reachable from the public internet on a WAN-facing node.
"Cluster members" is not a list you maintain. Each node works it out from the cluster it belongs to and rewrites its own rules whenever membership changes — you never edit the peer list by hand.
### Is it on?
| How the node was installed | Firewall |
|---|---|
| From the ISO | **on** |
| Binary installer (`curl \| bash`) or `setup.sh` | **off** |
| `setup.sh --firewall=on` | **on** |
The binary installer defaults to off deliberately: it runs on servers that already have an operating system and services on them, and switching on a default-deny policy uninvited could cut off something Spinifex knows nothing about. **For production, turn it on** — either at install time:
```bash
curl -fsSL https://install.mulgadc.com | bash -s -- --firewall=on
```
or afterwards, by setting it in `/etc/spinifex/spinifex.toml` and restarting the daemon:
```toml
[network]
firewall_enabled = true
```
Before you do, check what else the machine is serving. Anything listening on a port outside the public group above stops accepting new connections.
### Turning it off and on around cluster changes
A node only recognises the members of the cluster it currently belongs to, so during formation — when the nodes do not yet know each other — internal traffic between them is blocked. Turn the firewall off while you form the cluster, and on again once it is up:
```bash
# Off — before forming or expanding a cluster. Run on every node.
sudo /usr/local/lib/spinifex/spinifex-firewall-apply disable
# On — once the cluster is formed and verified. Run on every node.
sudo systemctl restart spinifex-daemon
```
Restarting the daemon is what re-arms it: the node rebuilds its peer list from the cluster it is now part of, reloads the rules, and re-enables the boot-time unit so the policy survives a reboot. It also happens on its own within five minutes if you would rather wait.
### Checking it
```bash
sudo nft list table inet spinifex_filter
```
The peer list is an nft variable, expanded when the rules load, so it does not appear under a name of its own. Look instead at the `ip saddr { ... }` addresses on the cluster-plane rules — the ones accepting 4222, 6641, 6642 and the rest.
Every node's addresses should be there. On a multi-NIC node that means its WAN, LAN and VPC addresses, so expect several entries per node. A missing node means its cluster traffic is being dropped.
Dropped packets are logged, rate-limited, so this tells you whether a connection problem is the firewall or something else:
```bash
sudo journalctl -k | grep 'spinifex-fw drop'
```
## How Multi-Node Storage Works
Background reading — you do not configure any of this by hand. `spx admin init` and `spx admin join` build the topology from the servers that actually form the cluster in Step 4, and each machine gets the same file with its own host ID recorded in `spinifex.toml`.
Predastore is configured for the whole cluster in `/etc/spinifex/predastore/predastore.toml`. Each server is one `[[host]]` — a single Predastore process owning that machine's data directory and TLS identity — carrying three nodes under `[[host.node]]`:
| Role | Port | Purpose |
|---|---|---|
| `gate` | TCP 8443 | Serves the S3 API. Every server runs one, so any of them answers an S3 request. |
| `blob` | UDP 6660 | Holds erasure-coded object shards. One per machine. |
| `meta` | UDP 7660 | Member of the Raft quorum over global state — buckets and the object index. |
Ports have to be unique within a host but not across the cluster, so every machine uses the same three. Blob and meta traffic between hosts runs over QUIC, authenticated by the cluster CA; nodes on the same machine talk over an in-process pipe and open no socket, which is why a single-server install listens on 8443 alone.
Reed-Solomon parameters are chosen from the cluster size, since each machine contributes exactly one blob node: two servers get `RS(1,1)`, three or more get `RS(2,1)`. `RS(2,1)` survives the loss of any one server's shards — another reason three nodes is the recommended minimum.
Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import an AMI, create an SSH key pair, create a VPC with a public subnet, and launch your first instance. It ends by arming the [host firewall](/docs/host-firewall), which is also where you re-arm it if you turned it off to form the cluster.
## Troubleshooting
@@ -500,7 +502,7 @@ If it hangs, check node 1's init output before blaming the firewall — `spx adm
sudo journalctl -k | grep 'spinifex-fw drop'
```
Turn the firewall off on **every** node and retry the join, then re-arm once the cluster is up — see [Firewall and cluster membership](#firewall-and-cluster-membership). The joining node retries for 20 minutes by default, so it is often still waiting while you fix this.
Turn the firewall off on **every** node and retry the join, then re-arm once the cluster is up — see [Host Firewall](/docs/host-firewall). The joining node retries for 20 minutes by default, so it is often still waiting while you fix this.
### Join Refuses: "this node is already initialized"
@@ -508,6 +510,28 @@ The node has its own cluster configuration — normal for anything installed fro
Safe on a freshly installed node. On one that has been in service it orphans every volume and fragment sealed under the old key, so check before forcing.
### Buckets Report as Under-Replicated
```bash
sudo spx admin kv replicas
```
A bucket showing `UNDER-REPLICATED` lives on fewer nodes than the cluster has, so losing the node holding it takes it down for the whole cluster. Two things put a bucket in that state, and the fix is the same for both:
```bash
sudo spx admin kv replicas --repair
```
The first cause is **a cluster that grew**: buckets created when there were three servers stay on three replicas until something raises them. Services do this themselves as they restart, and `--repair` does it now.
The second is **a node that was down when the bucket was created**, so the Raft group formed short. Bring the node back first — `--repair` on an incomplete cluster raises the configured count, but the group cannot place the extra replica until the node is there to hold it. Confirm with `spx get nodes` before repairing.
If `WANT` itself is wrong — 1 on a three-server cluster — the problem is not the buckets but that host's view of the cluster. Check it has all three `[nodes.*]` sections:
```bash
sudo grep -oP '^\[nodes\.\K[^.\]]+' /etc/spinifex/spinifex.toml | sort -u
```
### OVN Database Cluster Not Forming
```bash
docs/install/install-usb/README.md
+1 -1
View File
@@ -183,7 +183,7 @@ The node rewrites its peer list from the cluster it is now part of and re-arms i
sudo nft list table inet spinifex_filter
```
Check the `ip saddr { ... }` addresses on the cluster-plane rules — the peer list is an nft variable expanded at load time, so it has no name of its own in the output. The addresses of **all** your servers should appear. If one is missing, that server's cluster traffic will be blocked — see [Firewall and cluster membership](/docs/install-multi-node#firewall-and-cluster-membership).
Check the `ip saddr { ... }` addresses on the cluster-plane rules — the peer list is an nft variable expanded at load time, so it has no name of its own in the output. The addresses of **all** your servers should appear. If one is missing, that server's cluster traffic will be blocked — see [Host Firewall](/docs/host-firewall).
### Setup Complete
docs/install/install/README.md
+5 -1
View File
@@ -122,7 +122,11 @@ If this returns a list of available instance types, your installation is working
**Congratulations! Spinifex is installed.**
Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import an AMI, create a VPC, and launch your first instance.
## Step 6. Set Up Your Cluster
Spinifex is running, but it holds nothing yet — no machine images, no networks, no instances.
Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import an AMI, create an SSH key pair, create a VPC with a public subnet, and launch your first instance.
## Troubleshooting
docs/migration/moving-aws-workload/README.md
+26 -10
View File
@@ -74,10 +74,8 @@ provider "aws" {
s3 = "https://localhost:8443"
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
skip_metadata_api_check = true
skip_region_validation = true
}
```
@@ -91,16 +89,34 @@ aws s3 sync s3://local-bucket/ s3://cloud-bucket/ --source-region spinifex --reg
## Terraform Provider Errors
Ensure all four skip flags are set in your provider configuration:
Ensure these two skip flags are set in your provider configuration:
```hcl
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
skip_metadata_api_check = true
skip_region_validation = true
```
Without these, Terraform will try to validate credentials and metadata against real AWS endpoints.
Without them, Terraform queries the EC2 instance metadata service for credentials and rejects a region name AWS does not know.
Leave `skip_credentials_validation` and `skip_requesting_account_id` unset, and point the `sts` endpoint at Spinifex. The provider then learns the account ID from `GetCallerIdentity` as it does on AWS. With the account ID blank, a security group rule that references a group in the same account reads back as `<account>/sg-...` and never reaches a clean plan.
### Tagged S3 buckets need both flags set
There is one exception, and it applies to a provider block that manages an `aws_s3_bucket` with tags. AWS provider 6.x reads bucket tags through S3 Control, whose endpoint puts the account ID in the hostname — so the provider dials `https://000000000001.<your-endpoint>`, which cannot resolve when your endpoint is an IP address:
```
Error: listing tags for S3 (Simple Storage) Bucket (uploads): operation error S3 Control:
ListTagsForResource, ... dial tcp: lookup 000000000001.10.0.0.10: no such host
```
Set both flags in that provider block. With no account ID the provider reads tags through S3 `GetBucketTagging` instead, which Spinifex serves:
```hcl
skip_requesting_account_id = true
skip_credentials_validation = true
```
Both are needed — `skip_requesting_account_id` on its own has no effect, because the credentials check calls `sts:GetCallerIdentity` and supplies the account ID before the account lookup is ever skipped. If the same configuration also needs the account ID, for `aws_ecr_repository` or a security group rule that references another group, put the bucket in a second, aliased provider block rather than setting the flags globally.
## S3 Signature Errors
docs/oci-integration/README.md
+1033
View File
File diff suppressed because it is too large Load Diff
docs/security/network-connections/README.md
+2 -2
View File
@@ -149,7 +149,7 @@ Spinifex nodes initiate a small, fixed set of outbound connections.
**Update checks and metadata.** Spinifex does not check for updates and does not consume a cloud metadata service (`169.254.169.254` is served *by* the cluster to guest VMs). Node software updates come from the operator's OS package channel. The install-telemetry endpoint above is the only vendor-operated destination contacted by a node; closed-egress deployments should disable it and record the opt-out in the security plan.
**Air-gapped deployments.** The image URLs above are the only destinations needed for the standard image catalogue. Mirror them locally and use `spx admin images import --file` with pre-staged files. Telemetry must also be disabled. See [Air-Gapped Install](/docs/install-airgapped).
**Air-gapped deployments.** The image URLs above are the only destinations needed for the standard image catalogue. Mirror them locally with their sums files and use `spx admin images import --file <image> --checksum <sums-file>` with the pre-staged files. Telemetry must also be disabled. See [Air-Gapped Install](/docs/install-airgapped).
## 3. Cross-Node (Internal) Connections
@@ -226,7 +226,7 @@ Notes that make the difference between this working and locking you out:
- **`output` is deliberately untouched.** The metadata service's reply path egresses through per-instance policy routing; filtering `output` breaks it in ways that are hard to attribute.
- **On a single-NIC node the cluster and encap sets are the node's public addresses**, because the planes collapse. The rules are still correct, but they are no longer a boundary — an upstream ACL is the only real control there.
Port 4432 must be closed outside the bootstrap window; `spx admin join` opens it transiently. Outbound egress can be limited to the image-catalogue hostnames in [§2](#2-outbound-connections) plus the operator's OS package repositories; on air-gapped nodes, block all outbound HTTPS and use `spx admin images import --file`.
Port 4432 must be closed outside the bootstrap window; `spx admin join` opens it transiently. Outbound egress can be limited to the image-catalogue hostnames in [§2](#2-outbound-connections) plus the operator's OS package repositories; on air-gapped nodes, block all outbound HTTPS and use `spx admin images import --file <image> --checksum <sums-file>`.
## 5. Configuration Surface
docs/service-interfaces.yaml
+5
View File
@@ -86,6 +86,9 @@ services:
# Cluster coordination subjects.
- "ec2.ReserveClusterNode"
- "ec2.FinalizeClusterInstances"
# An EIP association reaches whichever node holds the instance, which is
# rarely the node that served the request (fan-out, no queue group).
- "ec2.eni.public-ip-changed"
- "spinifex.node.status"
- "spinifex.node.vms"
- "spinifex.nodes.discover"
@@ -168,6 +171,8 @@ services:
- "ebs.config.*"
# → predastore (S3 / AMI / keypair store)
- "s3.*"
# → every daemon, so the one holding the instance reports the new address
- "ec2.eni.public-ip-changed"
# IAM lifecycle event
- "iam.account.created"
# → the in-guest rds-agent: set-password and the like, published on the
docs/terraform-workbooks/bastion-private-subnet/main.tf
+5 -7
View File
@@ -30,15 +30,15 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.0"
version = "6.66.0"
}
tls = {
source = "hashicorp/tls"
version = ">= 4.0"
version = "4.4.1"
}
local = {
source = "hashicorp/local"
version = ">= 2.0"
version = "2.9.1"
}
}
}
@@ -76,10 +76,8 @@ provider "aws" {
sts = var.spinifex_endpoint
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
skip_metadata_api_check = true
skip_region_validation = true
}
# ---------------------------------------------------------------------------
docs/terraform-workbooks/ecs-quickstart/main.tf
+5 -7
View File
@@ -32,15 +32,15 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.40, < 6.0"
version = "5.100.0"
}
tls = {
source = "hashicorp/tls"
version = ">= 4.0"
version = "4.4.1"
}
local = {
source = "hashicorp/local"
version = ">= 2.0"
version = "2.9.1"
}
}
}
@@ -114,10 +114,8 @@ provider "aws" {
elasticloadbalancingv2 = var.spinifex_endpoint
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
skip_metadata_api_check = true
skip_region_validation = true
}
# ---------------------------------------------------------------------------
docs/terraform-workbooks/eks-gitops-argocd/main.tf
+4 -6
View File
@@ -30,11 +30,11 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.40, < 6.0"
version = "5.100.0"
}
tls = {
source = "hashicorp/tls"
version = ">= 4.0"
version = "4.4.1"
}
}
}
@@ -118,10 +118,8 @@ provider "aws" {
acm = var.spinifex_endpoint
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
skip_metadata_api_check = true
skip_region_validation = true
}
data "aws_availability_zones" "available" {
docs/terraform-workbooks/eks-gitops-argocd/workloads/main.tf
+4 -6
View File
@@ -25,11 +25,11 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.40, < 6.0"
version = "5.100.0"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = ">= 2.20"
version = "3.2.1"
}
}
}
@@ -80,10 +80,8 @@ provider "aws" {
eks = var.spinifex_endpoint
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
skip_metadata_api_check = true
skip_region_validation = true
}
data "terraform_remote_state" "infra" {
docs/terraform-workbooks/eks-https-ingress/main.tf
+4 -6
View File
@@ -33,11 +33,11 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.40, < 6.0"
version = "5.100.0"
}
tls = {
source = "hashicorp/tls"
version = ">= 4.0"
version = "4.4.1"
}
}
}
@@ -115,10 +115,8 @@ provider "aws" {
acm = var.spinifex_endpoint
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
skip_metadata_api_check = true
skip_region_validation = true
}
data "aws_availability_zones" "available" {
docs/terraform-workbooks/eks-https-ingress/workloads/main.tf
+4 -6
View File
@@ -22,11 +22,11 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.40, < 6.0"
version = "5.100.0"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = ">= 2.20"
version = "3.2.1"
}
}
}
@@ -63,10 +63,8 @@ provider "aws" {
eks = var.spinifex_endpoint
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
skip_metadata_api_check = true
skip_region_validation = true
}
# Cluster identity comes from the parent module's state; the live endpoint/CA
docs/terraform-workbooks/eks-quickstart/main.tf
+3 -5
View File
@@ -29,7 +29,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.40, < 6.0"
version = "5.100.0"
}
}
}
@@ -103,10 +103,8 @@ provider "aws" {
ecr = var.spinifex_endpoint
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
skip_metadata_api_check = true
skip_region_validation = true
}
# ---------------------------------------------------------------------------
docs/terraform-workbooks/eks-quickstart/workloads/main.tf
+4 -6
View File
@@ -17,11 +17,11 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.40, < 6.0"
version = "5.100.0"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = ">= 2.20"
version = "3.2.1"
}
}
}
@@ -54,10 +54,8 @@ provider "aws" {
eks = var.spinifex_endpoint
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
skip_metadata_api_check = true
skip_region_validation = true
}
# Cluster identity comes from the parent module's state; the live endpoint/CA
docs/terraform-workbooks/nginx-alb/main.tf
+6 -8
View File
@@ -35,15 +35,15 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.65.0, < 5.66.0"
version = "5.65.0"
}
tls = {
source = "hashicorp/tls"
version = ">= 4.0"
version = "4.4.1"
}
local = {
source = "hashicorp/local"
version = ">= 2.0"
version = "2.9.1"
}
}
}
@@ -58,7 +58,7 @@ variable "region" {
}
variable "instance_type" {
type = string
type = string
default = "t3.small"
}
@@ -82,10 +82,8 @@ provider "aws" {
elasticloadbalancingv2 = var.spinifex_endpoint
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
skip_metadata_api_check = true
skip_region_validation = true
}
# ---------------------------------------------------------------------------
docs/terraform-workbooks/nginx-webserver/main.tf
+5 -7
View File
@@ -19,15 +19,15 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.0"
version = "6.66.0"
}
tls = {
source = "hashicorp/tls"
version = ">= 4.0"
version = "4.4.1"
}
local = {
source = "hashicorp/local"
version = ">= 2.0"
version = "2.9.1"
}
}
}
@@ -65,10 +65,8 @@ provider "aws" {
sts = var.spinifex_endpoint
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
skip_metadata_api_check = true
skip_region_validation = true
}
# ---------------------------------------------------------------------------
docs/terraform-workbooks/rds-quickstart/main.tf
+5 -7
View File
@@ -38,15 +38,15 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.40, < 6.0"
version = "5.100.0"
}
tls = {
source = "hashicorp/tls"
version = ">= 4.0"
version = "4.4.1"
}
local = {
source = "hashicorp/local"
version = ">= 2.0"
version = "2.9.1"
}
}
}
@@ -135,10 +135,8 @@ provider "aws" {
rds = var.spinifex_endpoint
}
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
skip_region_validation = true
skip_metadata_api_check = true
skip_region_validation = true
}
# ---------------------------------------------------------------------------
docs/terraform-workbooks/s3-webapp/README.md
+6
View File
@@ -139,6 +139,12 @@ curl -k https://localhost:8443/
aws s3 ls --profile spinifex --endpoint-url https://localhost:8443
```
### `ListTagsForResource ... no such host` After the Bucket Is Created
The bucket was created and the failure is on the read-back of its tags. AWS provider 6.x reads bucket tags through S3 Control, whose endpoint carries the account ID in the hostname, so the provider dials `https://000000000001.<your-endpoint>` — a name that cannot resolve against an IP address.
This workbook's provider block already sets `skip_requesting_account_id` and `skip_credentials_validation`, which is what keeps the provider on S3 `GetBucketTagging`. If you copied the block into your own configuration, carry both across: `skip_requesting_account_id` on its own has no effect, because the credentials check supplies the account ID before the account lookup is skipped.
### Flask App Not Starting
SSH into the instance and check the service:
docs/terraform-workbooks/s3-webapp/main.tf
+19 -6
View File
@@ -46,15 +46,15 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.0"
version = "6.66.0"
}
tls = {
source = "hashicorp/tls"
version = ">= 4.0"
version = "4.4.1"
}
local = {
source = "hashicorp/local"
version = ">= 2.0"
version = "2.9.1"
}
}
}
@@ -124,10 +124,14 @@ provider "aws" {
s3_use_path_style = true
skip_credentials_validation = true
skip_metadata_api_check = true
skip_metadata_api_check = true
skip_region_validation = true
# Both, not either: an account ID sends the bucket tag read to S3 Control at
# https://{account}.{endpoint}, which cannot resolve against an IP, and the
# credentials check supplies one even when the account lookup is skipped.
skip_requesting_account_id = true
skip_region_validation = true
skip_credentials_validation = true
}
# ---------------------------------------------------------------------------
@@ -164,6 +168,15 @@ data "aws_ami" "ubuntu" {
resource "aws_s3_bucket" "uploads" {
bucket = var.bucket_name
# The webapp uploads into this bucket, so a destroy of a bucket that has been
# used fails BucketNotEmpty without this.
force_destroy = true
tags = {
Name = var.bucket_name
Example = "s3-webapp"
}
}
# ---------------------------------------------------------------------------
docs/testing/e2e-coverage.md
+132
View File
@@ -0,0 +1,132 @@
# What each E2E suite proves
**This index exists to answer "does anything already test that?" before someone reasons it out from the code.** Comments describe intent when written and can go stale; these suites run against a live cluster and are the current answer on behaviour.
Suites live in `tests/e2e/<name>/` and build to `tests/e2e/_bin/<name>.test`. Every file carries `//go:build e2e`, so a default build skips them. The sets that decide what runs where are in `.github/scripts/suite-sets.sh`, and the nightly permutation matrix is `.github/e2e-cells.json`.
| Set | Suites |
| --- | --- |
| `E2E_SUITES_SINGLE` | `single iam cert eks ecs storagegrowth partialblock rds quota storagefault` |
| `E2E_SUITES_MULTI` | `multinode lb cert quota kvquorum lbrecovery instancerecovery storagefault` |
| `E2E_SUITES_NIGHTLY_SINGLE` | `single cert iam` |
| `E2E_SUITES_NIGHTLY_MULTI` | `multinode cert lb` |
The nightly permutation sets are deliberately narrower than the full ones: those cells have a ~35 minute budget and exist to prove every install / network / host-OS combination boots and serves. `eks`, `ecs`, `rds`, `storagefault`, `instancerecovery`, `lbrecovery` and `kvquorum` each get a dedicated cell instead.
**`multinode` carries the KV replication sweep as well**, which is why the thing `kvquorum` proves is still checked in every multi-node permutation cell even though the suite itself is not. The read-only half costs a NATS connection and a stream listing; only the node-loss half needs a cell of its own.
## The suites
### `single` — the broad single-node EC2/EBS/VPC surface
The largest suite, 25 top-level tests. Notable entry points:
- **`TestVPCEgressPaths`** — **the authority on subnet egress semantics.** One VPC with a public and a private subnet, sharing two guests across four stages: `PublicSubnetEgress` (IGW egress works), `RouteBeforeSubnet` (a regression guard for an IGW route installed on the main route table before the subnet exists), the NAT Gateway rounds, and `EIPFlip` (associating an Elastic IP onto a running instance).
- **`NATBaseline` hard-fails if a private-subnet guest reaches the internet with no NAT Gateway.** `NATGatewayUp` then proves egress appears and `NATGatewayDown` that it disappears. `SPINIFEX_VPCEGRESS_NAT_ROUNDS` repeats the up/down cycle for a soak run.
- **This is AWS parity for private subnets.** It ran in pool mode only until 2026-09-26: the fixture gated it on `external_mode = "pool"`, on the stale premise that nat clusters have no EIPs. They do when a public pool is configured, so the gate is now "can this cluster allocate public addresses" and the nat-single cell runs this test beside `natuplink`, which never creates a private subnet.
- `TestSGPolicyDatapath`, `TestSGReachabilityPolicy` — security groups as an actual datapath, not just API state.
- `TestWANEgressSMTPBlock` — egress port blocking.
- `TestInstanceMetadata` — IMDS from inside a guest.
- `TestStopStart`, `TestEarlyRebootLiveness`, `TestGuestChurnDurability`, `TestSnapshotLifecycle`, `TestSnapshotBackedLaunch`, `TestCreateImage`, `TestVolumeLifecycle`, `TestSpotInstanceLifecycle`, `TestLaunchTemplateBoot`.
- `TestNegativeErrorPaths` — the `8a`–`8h` AWS error-shape cases (malformed AMI ID, invalid instance type, volume in use, detach-root forbidden, and so on).
- **`TestKVReplication`** — the single-node half of the KV replication policy, and it is not the multi-node assertion with a smaller number in it. On one node one replica per bucket is the correct answer rather than the defect, so what it proves is the opposite: that nothing declared a replica count this deployment cannot place, that `spx admin kv replicas` exits 0 and reports `WANT 1`, and that `--repair` run twice against a cluster with nothing to repair changes no bucket's replica count and leaves every bucket serving. The multi-node suites cannot cover any of that, because on three nodes the unplaceable count is the right one. It lives here rather than in its own suite so it runs in every single-node nightly cell, across all three host and network permutations.
### `natuplink` — routed NAT (`external_mode = "nat"`), one node or three
Runs **on** a node, shelling out to `ip`/`iptables`/`ovn-nbctl` locally. Skips unless `spinifex.toml` says `external_mode = "nat"`. Nine sequential phases: host wiring (transit veth, `ip_forward`, NAT egress rules), config, the EIP surface (enabled or disabled depending on whether a public pool exists), default-subnet public IP mapping, OVN gateway and SNAT on the transit net, instance egress **proven via the serial console**, host-to-guest Tier 1 ingress, a second VPC getting a unique transit gateway IP, and the EIP ingress lifecycle.
**Phase 9 has two shapes.** On one node it is the EIP lifecycle: allocate, associate, host delivery plumbing, a TCP handshake to the address, a vpcd restart that must replay the plumbing, and disassociate tearing it down. On a cluster it is the distributed lane instead — a guest on every node, each public IP plumbed by the node running its guest and by **no other**, every transit veth carrying the one cluster-wide MAC, each `dnat_and_snat` row carrying `external_mac` and `logical_port`, inbound proven by a handshake and outbound by each guest's own console. **That lane is the regression guard for the two defects that made routed NAT work on one node only**, and neither is visible to a single-node run.
Phases 7 and 8 are gated on a cluster by what the datapath can actually do: the transit segment is a localnet, so host-to-guest Tier 1 ingress only exists where the VPC's gateway router port and the guest are both on this host. The VPC ingress route itself is installed on every node, so phase 8 checks it unconditionally.
Two nightly cells run it: `nat-single` (cell 19) and `nat-multi` (cell 30). Cell 20 is the bare-metal job and is not in the permutation matrix at all, which is why the multi-node NAT cell is numbered past the end rather than filling a gap.
### `multinode` — behaviour that only exists on more than one node
`VPCSetup`, `SpansMultipleNodes`, `SpreadPlacement`, `EveryRunningInstanceReported`, `BastionSSH`, and its own NAT Gateway lane: `PreNATIsolation`, `NATGatewayInternet`, `NATCleanupOrdering`.
**`TestMultinodeJetStreamReplicas` is a whole-cluster sweep, not a named list, and it lives here on purpose.** It audits every KV bucket the cluster has and fails any that is replicated across fewer nodes than the cluster has, capped at JetStream's ceiling of five. Naming buckets would only ever cover the ones somebody remembered — a replica count is set where a bucket is created, and there is a bucket-creating call in most services — so the sweep is the assertion and the three daemon-owned buckets are named only so that a run enumerating nothing still fails. Because `multinode` is in `E2E_SUITES_NIGHTLY_MULTI`, this runs in every multi-node permutation cell, and no cell can be added that quietly opts out of it.
### `kvquorum` — the cluster's own state survives losing a node (multi-node, needs three)
**The control plane keeps its state in JetStream KV buckets, and this is the suite that proves those buckets are actually replicated.** A bucket on one replica lives on one JetStream-chosen server recorded in no config, so losing that node makes it return `nats: no responders available for request` from *every* node at once — a cluster-wide outage caused by a single-node event. Leader leases are the worst case: acquiring one is a write, so a lease bucket without a quorum cannot be acquired by anyone, and every reconciler sharing it stops cluster-wide at exactly the moment a node has failed and there is repair work to do.
Three entry points, in the order they run:
- **`TestKVBucketsAreReplicatedAcrossTheCluster`** is read-only and runs first, so a cluster that is already wrong is reported as wrong rather than as a failure to survive a node loss. It sweeps every bucket, then separately requires the leader-lease buckets to be among those checked — they were the worst case of the defect, and a regression reaching only them would otherwise be one line in a long list rather than the headline.
- **`TestKVReplicasCommandReportsHealthy`** runs `spx admin kv replicas --json` on **every** node, not one. The command reads cluster-global Raft metadata, so all nodes must give the same answer; a node that disagrees is reporting on a cluster it is not fully part of, which is worth knowing before a deployment is gated on its exit status.
- **`TestKVBucketsSurviveANodeLoss`** produces the fault rather than inspecting for it. It takes away **the node leading the most buckets** — the worst one to lose, and the one a test picking arbitrarily would usually miss — then requires every survivor to still answer for every bucket, brings the node back, and requires it to carry its share again.
**Configured, placed and working replicas are three different facts, and all three are checked.** A stream can carry the right replica count while its Raft group is short of peers, or hold the right peers while one of them is offline or still catching up, and either reads as healthy right up until the node it is really on goes away. The audit reports the leader and a count of peers NATS itself calls current and reachable, and the assertion is against that count rather than against the configured one. The last assertion is that no bucket came back with *fewer* replicas than it had: a cluster that "repaired" itself by lowering a count would pass every other assertion here and have quietly traded the durability this suite is about for a green run.
**Its own suite and its own cell, for lbrecovery's reasons.** It takes a node away, so anything sharing the environment reports that outage as its own failure, and the question it answers — "is our internal state actually replicated" — has to read as itself rather than as one more red subtest somewhere else. Three nodes is the floor: on two, a majority is two, so losing either leaves no quorum for any stream and there is nothing to assert beyond "Raft needs a majority", which is true of NATS and not a property of ours.
### `instancerecovery` — a guest whose node goes away (multi-node)
**Asserted from both sides, and the pair is the point.** `InstanceAutoRecovery` takes a node down and requires the guest to come back on exactly one survivor, identified by the qemu process rather than by what the API says, then brings the node back and requires it to drop the local copy instead of relaunching it. `InstanceRecoveryRefusesStorageFault` freezes predastore cluster-wide first, so the guest is paused by `werror=stop` when its node goes away, and requires that **nothing moves** — a store that refuses every node cannot be fixed by moving a guest, and the relaunch would cost it the request QEMU is holding. A reconciler that cannot tell a host failure from a storage failure passes the first and fails the second. Neither turns recovery on, because there is no setting for it: both wait for every node to report the reconciler active on a cluster nothing configured, so a build that reintroduced a switch fails here.
**`InstancePartitionRecovery` is the fault this feature exists for, and the only one of the three that produces it rather than approximating it.** It drops NATS on 4222 and 4248 between one node and its peers and nothing else, so that node keeps its services, keeps its QEMU and keeps answering its own local clients while it cannot reach consensus — half healthy, which is what a real network fault looks like and what a service stop cannot imitate. It then requires that the node gives up the volume on a clock it can evaluate without reaching anything, that a survivor takes the guest over, and that the rejoining node forgets its copy rather than relaunching it. **The assertion it exists for is the one no single sample can reach:** a watch samples every node every three seconds for the whole run and fails on any sample naming two hosts, because two guests writing one volume through one object store is data loss already under way rather than a race that resolves. The evidence for *why* the guest stopped is the surrender and the fence in the node's own journal — a guest killed for any other reason satisfies the process count and means the opposite.
**The partition gets its own nft table and a systemd deadman timer, for reasons worth keeping.** The node's real firewall is `inet spinifex_filter`, so a test that flushed or edited it would leave the node unfirewalled with no sign of which rules it lost; a separate table at priority -300 says exactly "these packets, dropped" and heals in one atomic delete. `systemd-run --on-active` removes it independently, so a cancelled workflow or a panicking test does not leave a node cut off from its cluster until somebody notices. It also proves SSH still works and proves a peer port is actually unreachable before asserting anything — a rule set that installed cleanly and dropped nothing would otherwise read as the system tolerating a partition.
**The storage-fault victim launches with cloud-init user-data that writes continuously, and that is load bearing.** An idle guest issues no I/O that reaches the object store, so a version of this test without a workload left three guests running happily through twelve minutes of dead predastore and proved nothing. Driving the load from user-data rather than over SSH is what lets it run on a routed-NAT cell, where guests have no public address. The pause itself takes minutes — 4m47s when measured — so the budget is generous on purpose.
**Its own suite and its own nightly cell, for storagefault's reasons.** It freezes a cluster-wide service and removes a node, so anything sharing the environment reports that outage as its own failure; and it is slow by nature, which is why it is not in a permutation cell budgeted at thirty-five minutes. Both behaviours were first proven by hand against a real hypervisor power-off on the three-node OCI cluster.
### `lbrecovery` — a load balancer whose node goes away (multi-node, needs three)
**The customer-visible half of host failure, asserted from outside the cluster rather than from the records.** `ALBSurvivesItsHostFailing` stands up an internet-facing ALB with two HTTP backends, discovers which node the load balancer's guest landed on, and takes that node away. It then requires the four things a customer of a load balancer actually holds: the name keeps resolving, the address behind it does not move, requests through the name are answered again within a bounded time, and the backends — which never moved — are healthy again from the recovered load balancer's point of view.
**The backends are placed around the load balancer, not beside it.** A load balancer is launched over a queue group and lands wherever a node answers, so its node is discovered and the backends are then launched and relaunched until both are somewhere else. A backend on the node about to be taken away would be recovered too, and the run could not then tell an ALB that came back serving from one that came back with nothing to serve.
**DNS is queried against a node's own northstar, not through the runner's resolver.** Go's resolver dials port 53 on a named node — which is open on every node deliberately, because northstar serves public names — so the assertion is about what this cluster answers rather than about what the machine running the tests was configured to forward to. The same resolver drives the HTTP probes, so traffic goes through the load balancer's name end to end and not through an address resolved once at the start. `dig` is deliberately not used: it is proven present in the guest image and nowhere else.
**The assertion no single sample can reach is the name.** A watch queries every surviving node every five seconds for the whole run, and fails on any sample where the record is missing, where a node refuses the query, or where it answers an address that is not the load balancer's. An answer naming another address is the worst of the three, because the traffic goes there; an empty answer outlives the fix for as long as anything cached it. An address that moves is asserted separately and before any traffic, since a passing probe against a renumbered address would hide it.
**Then the node comes back, and has to give up what it no longer owns.** The returning node must not relaunch its local copy, and must no longer carry the tap that claims the load balancer's logical port — OVN binds a port to one chassis, so a second claimant does not split the traffic, it contends for it and the address answers from whichever won last. That same stale tap is also what keeps the external address's host ingress from being pruned, since the prune waits on evidence the guest sits elsewhere. The last assertion is that the name still serves traffic with the node back in the cluster, which is the end-user form of the same question.
**`DescribeLoadBalancers` is required to keep reporting `active` throughout, and that is AWS parity rather than an oversight.** AWS never marks a load balancer degraded because a host under it failed: the state describes the customer's configuration, not our hardware. The honest signals about the outage are target health and the traffic itself, which is why both are asserted and the state is not trusted as evidence of anything.
**Its own suite and its own cell, for instancerecovery's reasons plus one.** It takes a node away, so anything sharing the environment reports that outage as its own failure; it needs its own VPC, public subnet and a backend placement no shared fixture describes; and a failure here has to read as "the load balancer did not survive its host" rather than as one more red subtest in a suite about load balancer features.
### `iam` — everything IAM and STS
Policy evaluation against live requests rather than unit-level: a deny applying to the next request without a new credential, resource-scoped grants naming one key pair, `aws:SourceIp` seeing the real client address, assumed-role sessions carrying role policies, and revocation stopping one key alone.
### `cert` — the cluster CA and TLS identity
CA installed and downloadable, fingerprint matches config, hostnames present in DNS SANs, OpenSSL verification, `curl` without a CA flag, and a shared issuer across nodes.
### `lb` — load balancers (multi-node)
Internal and internet-facing ALB and NLB, HTTPS listeners, listener rules, `ModifyListener`, and NLB UDP.
### `eks`, `ecs`, `ecr` — container services
`eks`: cluster create/delete, access entries, addon delivery, `GetToken`, kubeconfig artifacts, the EBS CSI volume path, and an IRSA pod wired with a projected SA token. `ecs`: clusters, container instances, the three network modes (`awsvpc`, `bridge`, `host`), task-role credentials fetched from the container credentials endpoint, a service behind an ELB, and ENI release on terminate. `ecr`: control-plane CRUD plus a real OCI data plane — docker login/build/push/pull round trips, multi-layer images, and raw blob HTTP.
### `rds` — managed databases
The create/describe/connect path against a live cluster: a DB VM that actually boots, TLS enforced with plaintext refused, clients connecting from the same and another subnet, class changes replacing the VM while keeping overrides, deferred parameter-group attachment windows, and delete-while-creating leaving nothing behind.
### Storage and durability
- **`storagefault`** — what a guest does when its storage backend stops answering, and what stop/start does when the seal failed. **The claim is about corruption, not availability.** SIGSTOPs a cluster-wide service, so it always runs last and in its own cell. Three-node, because the RS(2,1) shard floor and cross-node takeover after a failed seal do not exist on one node.
- **`partialblock`** — concurrent sub-block writes to the same block surviving a cold reread through NBD from a real guest.
- **`storagegrowth`** — volume growth and image pull. Requires `external_mode=pool`; skips on a nat env, which has no EIP and therefore no guest SSH.
- **`reboot`** — a slow guest waited for and keeping what it wrote, and a wedged guest reset once its budget expires. Proves a real reboot by requiring a *new* boot ID.
- **`diskperf`** — guest disk performance gate, pinned fio profile on a dedicated blank volume. Also requires `external_mode=pool` for guest SSH.
### Other
- **`quota`** — every enforced dimension refuses with the same `quotaExceeded` code; a launch is charged for every instance it asks for; one instance wider than the cap is refused.
- **`gpu`** — device visible in the guest, driver access, launch.
- **`bedrock`** — `ListFoundationModels`/`GetFoundationModel` against a running cluster. Wire-format parity is unit-tested, not here.
- **`ddil`** — Denied, Disrupted, Intermittent, Limited network scenarios.
- **`teardown`** — sweeps leftovers.
- **`harness`** — shared primitives, not a suite. `harness/vpc_egress.go` carries the VPC/subnet/IGW/NAT-gateway fixtures the egress scenarios build on.
- **`manifest-check`, `manifest-lint`** — manifest validation, not cluster behaviour.
## Keeping this honest
Add a row when a suite is added, and correct one when a suite's claim changes. An index that has drifted is worse than none, because it will be trusted. If a suite's package comment and this file disagree, check the test bodies — both are prose and either can be stale.
go.mod
+45 -49
View File
@@ -6,29 +6,31 @@ require (
github.com/aws/aws-sdk-go v1.55.8
github.com/aws/aws-sdk-go-v2 v1.47.0
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.20
github.com/aws/aws-sdk-go-v2/config v1.33.4
github.com/aws/aws-sdk-go-v2/credentials v1.20.4
github.com/aws/aws-sdk-go-v2/config v1.33.5
github.com/aws/aws-sdk-go-v2/credentials v1.20.5
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.0
github.com/cenkalti/backoff/v4 v4.3.0
github.com/charmbracelet/bubbles v1.0.0
github.com/charmbracelet/bubbletea v1.3.10
github.com/charmbracelet/lipgloss v1.1.0
github.com/containerd/containerd/v2 v2.3.5
github.com/containerd/containerd/v2 v2.4.0
github.com/distribution/reference v0.6.0
github.com/go-chi/chi/v5 v5.3.2
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/insomniacslk/dhcp v0.0.0-20260719225207-c76316d4aa82
github.com/jackc/pgx/v5 v5.11.0
github.com/klauspost/compress v1.20.0
github.com/klauspost/cpuid/v2 v2.4.0
github.com/miekg/dns v1.1.73
github.com/minio/highwayhash v1.0.4
github.com/mulgadc/bluebottle v1.20.0
github.com/mulgadc/northstar v1.20.0
github.com/mulgadc/predastore v1.20.0
github.com/mulgadc/viperblock v1.20.0
github.com/nats-io/nats-server/v2 v2.14.6
github.com/nats-io/nats.go v1.53.1
github.com/mulgadc/bluebottle v1.21.0
github.com/mulgadc/northstar v1.21.0
github.com/mulgadc/predastore v1.21.0
github.com/mulgadc/viperblock v1.21.0
github.com/nats-io/nats-server/v2 v2.15.0
github.com/nats-io/nats.go v1.54.0
github.com/opencontainers/runtime-spec v1.3.0
github.com/oracle/oci-go-sdk/v65 v65.126.0
github.com/ovn-kubernetes/libovsdb v0.8.1
github.com/pelletier/go-toml/v2 v2.4.3
github.com/pterm/pterm v0.12.83
@@ -56,23 +58,22 @@ require (
atomicgo.dev/keyboard v0.2.9 // indirect
atomicgo.dev/schedule v0.1.0 // indirect
github.com/Microsoft/go-winio v0.6.3-0.20251027160822-ad3df93bed29 // indirect
github.com/Microsoft/hcsshim v0.15.0-rc.1 // indirect
github.com/antithesishq/antithesis-sdk-go v0.7.2-default-no-op // indirect
github.com/armon/go-metrics v0.4.1 // indirect
github.com/Microsoft/hcsshim v0.15.0-rc.4 // indirect
github.com/antithesishq/antithesis-sdk-go v0.8.0-default-no-op // indirect
github.com/atotto/clipboard v0.1.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.3 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.3 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.3 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.113.0 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.113.3 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.10.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.38.0 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.50.0 // indirect
github.com/aws/smithy-go v1.28.1 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.51.0 // indirect
github.com/aws/smithy-go v1.28.2 // indirect
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/boltdb/bolt v1.3.1 // indirect
@@ -90,16 +91,17 @@ require (
github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
github.com/containerd/cgroups/v3 v3.1.3 // indirect
github.com/containerd/console v1.0.5 // indirect
github.com/containerd/containerd/api v1.11.1 // indirect
github.com/containerd/containerd/api v1.12.0 // indirect
github.com/containerd/continuity v0.5.0 // indirect
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/containerd/fifo v1.1.0 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/log v0.2.0 // indirect
github.com/containerd/log/otel v0.1.0 // indirect
github.com/containerd/platforms v1.0.0-rc.5 // indirect
github.com/containerd/plugin v1.1.0 // indirect
github.com/containerd/ttrpc v1.2.8 // indirect
github.com/containerd/typeurl/v2 v2.2.3 // indirect
github.com/containerd/ttrpc v1.2.9 // indirect
github.com/containerd/typeurl/v2 v2.3.0 // indirect
github.com/dgraph-io/badger/v4 v4.9.6 // indirect
github.com/dgraph-io/ristretto/v2 v2.4.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
@@ -114,8 +116,7 @@ require (
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.30.1 // indirect
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
github.com/gofrs/flock v0.10.0 // indirect
github.com/google/flatbuffers v25.12.19+incompatible // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/google/uuid v1.6.0 // indirect
@@ -123,37 +124,34 @@ require (
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect
github.com/hashicorp/go-hclog v1.6.3 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/hashicorp/go-metrics v0.5.4 // indirect
github.com/hashicorp/go-metrics v0.7.0 // indirect
github.com/hashicorp/go-msgpack/v2 v2.1.5 // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/hashicorp/raft v1.7.3 // indirect
github.com/hashicorp/raft-boltdb/v2 v2.3.1 // indirect
github.com/hashicorp/raft v1.8.0 // indirect
github.com/hashicorp/raft-boltdb/v2 v2.4.2 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/josharian/native v1.1.0 // indirect
github.com/klauspost/compress v1.19.2 // indirect
github.com/klauspost/reedsolomon v1.14.2 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/lithammer/fuzzysearch v1.1.8 // indirect
github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-colorable v0.1.15 // indirect
github.com/mattn/go-isatty v0.0.24 // indirect
github.com/mattn/go-localereader v0.0.1 // indirect
github.com/mattn/go-runewidth v0.0.21 // indirect
github.com/mdlayher/packet v1.1.2 // indirect
github.com/mdlayher/socket v0.5.1 // indirect
github.com/mdlayher/socket v0.6.0 // indirect
github.com/minio/crc64nvme v1.1.1 // indirect
github.com/moby/locker v1.0.1 // indirect
github.com/moby/sys/capability v0.4.0 // indirect
github.com/moby/sys/mountinfo v0.7.2 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
github.com/moby/sys/signal v0.7.1 // indirect
github.com/moby/sys/user v0.4.0 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/moby/sys/user v0.4.1 // indirect
github.com/moby/sys/userns v0.2.1 // indirect
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
github.com/muesli/cancelreader v0.2.2 // indirect
github.com/muesli/termenv v0.16.0 // indirect
@@ -163,17 +161,18 @@ require (
github.com/nats-io/nuid v1.0.1 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/opencontainers/runtime-tools v0.9.1-0.20251114084447-edf4cb3d2116 // indirect
github.com/pierrec/lz4/v4 v4.1.18 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/prometheus/client_golang v1.23.2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.5 // indirect
github.com/prometheus/procfs v0.20.1 // indirect
github.com/quic-go/quic-go v0.62.0 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_golang v1.24.1 // indirect
github.com/prometheus/client_model v0.6.3 // indirect
github.com/prometheus/common v0.71.0 // indirect
github.com/prometheus/procfs v0.21.1 // indirect
github.com/quic-go/quic-go v0.63.0 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/sagikazarmark/locafero v0.12.0 // indirect
github.com/sirupsen/logrus v1.9.4 // indirect
github.com/sirupsen/logrus v1.10.2 // indirect
github.com/sony/gobreaker/v2 v2.4.0 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
@@ -181,8 +180,8 @@ require (
github.com/tidwall/btree v1.8.1 // indirect
github.com/u-root/uio v0.0.0-20230220225925-ffce2a382923 // indirect
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
go.etcd.io/bbolt v1.4.3 // indirect
go.opencensus.io v0.24.0 // indirect
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect
go.etcd.io/bbolt v1.5.0 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/bridges/otelslog v0.20.1 // indirect
go.opentelemetry.io/contrib/instrumentation/runtime v0.71.0 // indirect
@@ -194,7 +193,6 @@ require (
go.opentelemetry.io/otel/sdk/log v0.22.0 // indirect
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
go.uber.org/nilaway v0.0.0-20260721205819-4a0f7653488a // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
golang.org/x/exp/typeparams v0.0.0-20260611194520-c48552f49976 // indirect
golang.org/x/mod v0.41.0 // indirect
golang.org/x/net v0.59.0 // indirect
@@ -202,14 +200,12 @@ require (
golang.org/x/term v0.46.0 // indirect
golang.org/x/text v0.42.0 // indirect
golang.org/x/vuln v1.6.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260825221802-da73d73af1c5 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260825221802-da73d73af1c5 // indirect
google.golang.org/grpc v1.83.2 // indirect
google.golang.org/protobuf v1.36.12 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
sigs.k8s.io/yaml v1.6.0 // indirect
tags.cncf.io/container-device-interface v1.1.0 // indirect
tags.cncf.io/container-device-interface/specs-go v1.1.0 // indirect
tags.cncf.io/container-device-interface v1.1.1 // indirect
tags.cncf.io/container-device-interface/specs-go v1.1.1 // indirect
)
// go.opencensus.io -> grpc@v1.33.2 drags in the pre-split monolithic genproto,
go.sum
+90 -2770
View File
File diff suppressed because it is too large Load Diff
internal/awsmodel/coverage.go
+98 -2
View File
@@ -1,7 +1,9 @@
package awsmodel
import (
"encoding/json"
"fmt"
"os"
"sort"
"strings"
)
@@ -33,6 +35,44 @@ type OperationCoverage struct {
Note string
}
// CoverageJSONSchemaVersion identifies the generated coverage.json schema.
// Increment it only for an incompatible change to the JSON document.
const CoverageJSONSchemaVersion = 1
// coverageJSON is the generated, machine-readable form of the operation
// inventory. It deliberately reports dispatch state only: an operation in
// implemented has a registered, non-stubbed and non-refused dispatcher, but
// that alone does not establish behavioural AWS compatibility.
type coverageJSON struct {
SchemaVersion int `json:"schema_version"`
ModelSource coverageJSONModelSource `json:"model_source"`
Services []coverageJSONService `json:"services"`
}
type coverageJSONModelSource struct {
Repository string `json:"repository"`
Commit string `json:"commit"`
CommitDate string `json:"commit_date"`
}
type coverageJSONService struct {
Service string `json:"service"`
APIVersion string `json:"api_version"`
Opaque bool `json:"opaque"`
Note string `json:"note"`
Operations coverageJSONOperations `json:"operations"`
}
type coverageJSONOperations struct {
Modelled []string `json:"modelled"`
Registered []string `json:"registered"`
Implemented []string `json:"implemented"`
Stubbed []string `json:"stubbed"`
Unsupported []string `json:"unsupported"`
Missing []string `json:"missing"`
Extra []string `json:"extra"`
}
// CompareOperations compares one embedded service model with an authoritative
// gateway dispatch inventory.
func CompareOperations(service Service, dispatch DispatchInventory) (OperationCoverage, error) {
@@ -127,7 +167,7 @@ func toSet(values []string) map[string]bool {
// terminal. The published pages carry the per-operation detail.
func RenderCoverageSummary(coverages []OperationCoverage) string {
var report strings.Builder
fmt.Fprintf(&report, "AWS model operation coverage (aws-sdk-go %s)\n\n", SourceSDKVersion)
fmt.Fprintf(&report, "AWS model operation coverage (%s)\n\n", ModelSourceDescription())
for _, coverage := range sortedCoverages(coverages) {
if coverage.Opaque {
fmt.Fprintf(&report, "%-24s %-12s not enumerable\n", coverage.Service, coverage.APIVersion)
@@ -140,6 +180,62 @@ func RenderCoverageSummary(coverages []OperationCoverage) string {
return report.String()
}
// RenderCoverageJSON renders a deterministic, versioned inventory suitable
// for tools. The operation lists describe only the generated model-versus-
// dispatch comparison; consumers must establish behavioural support from
// their own evidence rather than inferring it from dispatch registration.
func RenderCoverageJSON(coverages []OperationCoverage) (string, error) {
document := coverageJSON{
SchemaVersion: CoverageJSONSchemaVersion,
ModelSource: coverageJSONModelSource{
Repository: ModelRepository,
Commit: ModelCommit,
CommitDate: ModelCommitDate,
},
Services: make([]coverageJSONService, 0, len(coverages)),
}
for _, coverage := range sortedCoverages(coverages) {
document.Services = append(document.Services, coverageJSONService{
Service: string(coverage.Service),
APIVersion: coverage.APIVersion,
Opaque: coverage.Opaque,
Note: coverage.Note,
Operations: coverageJSONOperations{
Modelled: copiedOperations(coverage.Modelled),
Registered: copiedOperations(coverage.Registered),
Implemented: copiedOperations(coverage.Implemented),
Stubbed: copiedOperations(coverage.Stubbed),
Unsupported: copiedOperations(coverage.Unsupported),
Missing: copiedOperations(coverage.Missing),
Extra: copiedOperations(coverage.Extra),
},
})
}
contents, err := json.MarshalIndent(document, "", " ")
if err != nil {
return "", fmt.Errorf("awsmodel: marshal coverage JSON: %w", err)
}
return string(contents) + "\n", nil
}
// WriteCoverageJSON writes the generated machine-readable inventory to path.
func WriteCoverageJSON(path string, coverages []OperationCoverage) error {
contents, err := RenderCoverageJSON(coverages)
if err != nil {
return err
}
if err := os.WriteFile(path, []byte(contents), 0o600); err != nil {
return fmt.Errorf("awsmodel: write coverage JSON: %w", err)
}
return nil
}
// copiedOperations represents an empty operation set as [] rather than null,
// so consumers can use every array field without a special case.
func copiedOperations(operations []string) []string {
return append([]string{}, operations...)
}
// RenderInternalReport renders the full comparison, including the operations
// the published pages leave out. It is not a site page: the docs build renders
// only the slugs it is configured with, and this file is not one of them.
@@ -152,7 +248,7 @@ func RenderInternalReport(coverages []OperationCoverage, pages PageSet) string {
report.WriteString("# AWS Model Operation Coverage (internal)\n\n")
report.WriteString("Generated by `make aws-model-coverage`. This file is not published: it carries the counts and the unimplemented operations that the pages under this directory deliberately leave out.\n\n")
report.WriteString("**Not implemented** is the candidate work. **Not applicable** is declared in `pages.json` and is not work: those operations describe something this platform will not have.\n\n")
fmt.Fprintf(&report, "Pinned model source: `aws-sdk-go %s`.\n\n", SourceSDKVersion)
fmt.Fprintf(&report, "Pinned model source: `%s`.\n\n", ModelSourceDescription())
for _, coverage := range sortedCoverages(coverages) {
page := pages.Services[coverage.Service]
internal/awsmodel/coverage_json_test.go
+98
View File
@@ -0,0 +1,98 @@
package awsmodel_test
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/mulgadc/spinifex/internal/awsmodel"
)
func TestRenderCoverageJSONIsDeterministicAndPreservesDispatchStates(t *testing.T) {
sts, err := awsmodel.CompareOperations(awsmodel.STS, awsmodel.DispatchInventory{
Registered: []string{"AssumeRole", "GetCallerIdentity", "GetSessionToken"},
Stubbed: []string{"GetSessionToken"},
Unsupported: []string{"GetCallerIdentity"},
})
if err != nil {
t.Fatal(err)
}
iam, err := awsmodel.CompareOperations(awsmodel.IAM, awsmodel.DispatchInventory{})
if err != nil {
t.Fatal(err)
}
contents, err := awsmodel.RenderCoverageJSON([]awsmodel.OperationCoverage{sts, iam})
if err != nil {
t.Fatal(err)
}
var document struct {
SchemaVersion int `json:"schema_version"`
ModelSource struct {
Repository string `json:"repository"`
Commit string `json:"commit"`
CommitDate string `json:"commit_date"`
} `json:"model_source"`
Services []struct {
Service string `json:"service"`
Operations struct {
Implemented []string `json:"implemented"`
Stubbed []string `json:"stubbed"`
Unsupported []string `json:"unsupported"`
Registered []string `json:"registered"`
} `json:"operations"`
} `json:"services"`
}
if err := json.Unmarshal([]byte(contents), &document); err != nil {
t.Fatalf("generated JSON is invalid: %v\n%s", err, contents)
}
if document.SchemaVersion != awsmodel.CoverageJSONSchemaVersion {
t.Errorf("schema version = %d, want %d", document.SchemaVersion, awsmodel.CoverageJSONSchemaVersion)
}
if source := document.ModelSource; source.Repository != awsmodel.ModelRepository || source.Commit != awsmodel.ModelCommit || source.CommitDate != awsmodel.ModelCommitDate {
t.Errorf("model source = %+v, want %s at %s (%s)", source, awsmodel.ModelRepository, awsmodel.ModelCommit, awsmodel.ModelCommitDate)
}
if !strings.Contains(contents, `"registered": []`) {
t.Errorf("an empty operation set must be an array, not null:\n%s", contents)
}
if len(document.Services) != 2 || document.Services[0].Service != "iam" || document.Services[1].Service != "sts" {
t.Fatalf("services are not deterministic: %#v", document.Services)
}
states := document.Services[1].Operations
if got, want := strings.Join(states.Implemented, ","), "AssumeRole"; got != want {
t.Errorf("implemented = %q, want %q", got, want)
}
if got, want := strings.Join(states.Stubbed, ","), "GetSessionToken"; got != want {
t.Errorf("stubbed = %q, want %q", got, want)
}
if got, want := strings.Join(states.Unsupported, ","), "GetCallerIdentity"; got != want {
t.Errorf("unsupported = %q, want %q", got, want)
}
if got, want := strings.Join(states.Registered, ","), "AssumeRole,GetCallerIdentity,GetSessionToken"; got != want {
t.Errorf("registered = %q, want %q", got, want)
}
}
func TestWriteCoverageJSONWritesTheSameDocument(t *testing.T) {
coverage, err := awsmodel.CompareOperations(awsmodel.STS, awsmodel.DispatchInventory{Registered: []string{"AssumeRole"}})
if err != nil {
t.Fatal(err)
}
want, err := awsmodel.RenderCoverageJSON([]awsmodel.OperationCoverage{coverage})
if err != nil {
t.Fatal(err)
}
path := filepath.Join(t.TempDir(), "coverage.json")
if err := awsmodel.WriteCoverageJSON(path, []awsmodel.OperationCoverage{coverage}); err != nil {
t.Fatal(err)
}
got, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
if string(got) != want {
t.Errorf("written JSON differs from renderer:\n got: %s\nwant: %s", got, want)
}
}
internal/awsmodel/coverage_test.go
+4 -4
View File
@@ -17,8 +17,8 @@ func TestCompareOperations(t *testing.T) {
if len(coverage.Implemented) != 1 || coverage.Implemented[0] != "AssumeRole" {
t.Fatalf("implemented = %v, want [AssumeRole]", coverage.Implemented)
}
if len(coverage.Missing) != 5 {
t.Fatalf("missing count = %d, want 5: %v", len(coverage.Missing), coverage.Missing)
if len(coverage.Missing) != 8 {
t.Fatalf("missing count = %d, want 8: %v", len(coverage.Missing), coverage.Missing)
}
if len(coverage.Extra) != 1 || coverage.Extra[0] != "ExtraOperation" {
t.Fatalf("extra = %v, want [ExtraOperation]", coverage.Extra)
@@ -39,8 +39,8 @@ func TestRenderCoverageSummary(t *testing.T) {
}
report := RenderCoverageSummary([]OperationCoverage{coverage})
for _, want := range []string{
"aws-sdk-go " + SourceSDKVersion,
"1 of 8 implemented ( 12.5%)",
ModelSourceDescription(),
"1 of 11 implemented ( 9.1%)",
} {
if !strings.Contains(report, want) {
t.Errorf("summary does not contain %q:\n%s", want, report)
internal/awsmodel/ec2error.go
+2 -2
View File
@@ -85,8 +85,8 @@ var loadEC2ErrorCatalog = sync.OnceValues(func() (*ec2ErrorCatalog, error) {
// ValidateEC2ErrorResponse validates an EC2 Query API error envelope against
// the checked-in catalog curated from AWS's EC2 error-code reference. The AWS
// api-2.json model declares no operation errors, so this is intentionally a
// separate oracle.
// EC2 model declares no operation errors, so this is intentionally a separate
// oracle.
func ValidateEC2ErrorResponse(status int, body []byte) ([]Violation, error) {
root, err := parseXML(body)
if err != nil {
internal/awsmodel/embed.go
+13 -14
View File
@@ -2,22 +2,21 @@ package awsmodel
import "embed"
// modelFS contains only the curated EC2 error catalog. The large upstream
// api-2.json files are resolved from the pinned aws-sdk-go module cache at
// runtime; they are intentionally not source-controlled here.
// modelFS holds the Smithy service models, gzipped, from the api-models-aws
// commit named by ModelCommit, and the curated EC2 error catalog.
//
//go:embed models/ec2/error-codes.json
//go:embed models/*.json.gz models/ec2/error-codes.json
var modelFS embed.FS
var modelFiles = map[Service]string{
ACM: "models/apis/acm/2015-12-08/api-2.json",
EC2: "models/apis/ec2/2016-11-15/api-2.json",
ECR: "models/apis/ecr/2015-09-21/api-2.json",
ECS: "models/apis/ecs/2014-11-13/api-2.json",
EKS: "models/apis/eks/2017-11-01/api-2.json",
ElasticLoadBalancingV2: "models/apis/elasticloadbalancingv2/2015-12-01/api-2.json",
IAM: "models/apis/iam/2010-05-08/api-2.json",
RDS: "models/apis/rds/2014-10-31/api-2.json",
S3: "models/apis/s3/2006-03-01/api-2.json",
STS: "models/apis/sts/2011-06-15/api-2.json",
ACM: "models/acm-2015-12-08.json.gz",
EC2: "models/ec2-2016-11-15.json.gz",
ECR: "models/ecr-2015-09-21.json.gz",
ECS: "models/ecs-2014-11-13.json.gz",
EKS: "models/eks-2017-11-01.json.gz",
ElasticLoadBalancingV2: "models/elastic-load-balancing-v2-2015-12-01.json.gz",
IAM: "models/iam-2010-05-08.json.gz",
RDS: "models/rds-2014-10-31.json.gz",
S3: "models/s3-2006-03-01.json.gz",
STS: "models/sts-2011-06-15.json.gz",
}
internal/awsmodel/encode.go
+450
View File
@@ -0,0 +1,450 @@
package awsmodel
import (
"encoding/base64"
"encoding/json"
"fmt"
"maps"
"net/http"
"net/url"
"slices"
"strconv"
"strings"
"time"
"unicode"
)
// EncodedRequest is an operation input serialized for the service's protocol,
// ready to be addressed to an endpoint and signed.
type EncodedRequest struct {
Method string
// Path is escaped; it may carry a query string of its own.
Path string
Query url.Values
Header http.Header
Body []byte
}
// URL joins the request's path and query onto endpoint.
func (r *EncodedRequest) URL(endpoint string) string {
target := strings.TrimSuffix(endpoint, "/") + r.Path
if len(r.Query) == 0 {
return target
}
separator := "?"
if strings.Contains(r.Path, "?") {
separator = "&"
}
return target + separator + r.Query.Encode()
}
// EncodeRequest serializes input, as produced by GenerateRequests, for the
// service's protocol. S3's rest-xml is not supported.
func EncodeRequest(service Service, operationName string, input map[string]any) (*EncodedRequest, error) {
model, err := Load(service)
if err != nil {
return nil, err
}
operation, ok := model.Operation(operationName)
if !ok {
return nil, fmt.Errorf("awsmodel: %s operation %q is not modelled", service, operationName)
}
encoder := requestEncoder{model: model}
switch model.metadata.Protocol {
case "query", "ec2":
return encoder.query(operation, input)
case "json":
return encoder.json(operation, input)
case "rest-json":
return encoder.restJSON(operation, input)
default:
return nil, fmt.Errorf("awsmodel: encoding %s requests is not implemented", model.metadata.Protocol)
}
}
type requestEncoder struct {
model *Model
}
func (e requestEncoder) query(operation *Operation, input map[string]any) (*EncodedRequest, error) {
values := url.Values{
"Action": {operation.Name},
"Version": {e.model.metadata.APIVersion},
}
if operation.Input != nil {
if err := e.queryStructure(values, "", operation.Input.Shape, input); err != nil {
return nil, err
}
}
return &EncodedRequest{
Method: http.MethodPost,
Path: "/",
Header: http.Header{"Content-Type": {"application/x-www-form-urlencoded; charset=utf-8"}},
Body: []byte(values.Encode()),
}, nil
}
func (e requestEncoder) queryStructure(values url.Values, prefix, shapeName string, input map[string]any) error {
shape := e.model.shapes[shapeName]
for _, member := range slices.Sorted(maps.Keys(input)) {
ref, ok := shape.Members[member]
if !ok {
return fmt.Errorf("%s has no member %q", shapeName, member)
}
if err := e.queryValue(values, prefix+e.queryName(member, ref), ref, input[member]); err != nil {
return err
}
}
return nil
}
// queryName is the parameter name for a member: the ec2 protocol capitalises
// its XML name unless ec2QueryName gives one; awsQuery uses the XML name.
func (e requestEncoder) queryName(member string, ref ShapeRef) string {
if e.model.metadata.Protocol != "ec2" {
if ref.LocationName != "" {
return ref.LocationName
}
return member
}
if ref.QueryName != "" {
return ref.QueryName
}
name := member
if ref.LocationName != "" {
name = ref.LocationName
}
runes := []rune(name)
runes[0] = unicode.ToUpper(runes[0])
return string(runes)
}
func (e requestEncoder) queryValue(values url.Values, name string, ref ShapeRef, value any) error {
shape := e.model.shapes[ref.Shape]
switch shape.Type {
case "structure":
fields, ok := value.(map[string]any)
if !ok {
return fmt.Errorf("%s: want a structure, got %T", name, value)
}
return e.queryStructure(values, name+".", ref.Shape, fields)
case "list":
items, ok := value.([]any)
if !ok {
return fmt.Errorf("%s: want a list, got %T", name, value)
}
if len(items) == 0 {
// awsQuery sends an empty list as an empty parameter; ec2 cannot.
if e.model.metadata.Protocol != "ec2" {
values.Set(name, "")
}
return nil
}
itemPrefix := name + "."
if e.model.metadata.Protocol != "ec2" && !ref.Flattened && !shape.Flattened {
memberName := "member"
if shape.Member.LocationName != "" {
memberName = shape.Member.LocationName
}
itemPrefix += memberName + "."
}
for i, item := range items {
if err := e.queryValue(values, itemPrefix+strconv.Itoa(i+1), *shape.Member, item); err != nil {
return err
}
}
return nil
case "map":
entries, ok := value.(map[string]any)
if !ok {
return fmt.Errorf("%s: want a map, got %T", name, value)
}
entryPrefix := name + "."
if e.model.metadata.Protocol != "ec2" && !ref.Flattened && !shape.Flattened {
entryPrefix += "entry."
}
keyName, valueName := "key", "value"
if shape.Key.LocationName != "" {
keyName = shape.Key.LocationName
}
if shape.Value.LocationName != "" {
valueName = shape.Value.LocationName
}
for i, key := range slices.Sorted(maps.Keys(entries)) {
entry := entryPrefix + strconv.Itoa(i+1) + "."
values.Set(entry+keyName, key)
if err := e.queryValue(values, entry+valueName, *shape.Value, entries[key]); err != nil {
return err
}
}
return nil
default:
scalar, err := scalarString(value, ref, shape, "iso8601")
if err != nil {
return fmt.Errorf("%s: %w", name, err)
}
values.Set(name, scalar)
return nil
}
}
func (e requestEncoder) json(operation *Operation, input map[string]any) (*EncodedRequest, error) {
body := []byte("{}")
if operation.Input != nil {
document, err := e.jsonValue(*operation.Input, input)
if err != nil {
return nil, err
}
if body, err = json.Marshal(document); err != nil {
return nil, err
}
}
metadata := e.model.metadata
return &EncodedRequest{
Method: http.MethodPost,
Path: "/",
Header: http.Header{
"Content-Type": {"application/x-amz-json-" + metadata.JSONVersion},
"X-Amz-Target": {metadata.TargetPrefix + "." + operation.Name},
},
Body: body,
}, nil
}
// jsonValue converts a generated value into its JSON document form.
func (e requestEncoder) jsonValue(ref ShapeRef, value any) (any, error) {
shape := e.model.shapes[ref.Shape]
switch shape.Type {
case "structure":
fields, ok := value.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: want a structure, got %T", ref.Shape, value)
}
document := make(map[string]any, len(fields))
for member, field := range fields {
memberRef, ok := shape.Members[member]
if !ok {
return nil, fmt.Errorf("%s has no member %q", ref.Shape, member)
}
encoded, err := e.jsonValue(memberRef, field)
if err != nil {
return nil, err
}
document[member] = encoded
}
return document, nil
case "list":
items, ok := value.([]any)
if !ok {
return nil, fmt.Errorf("%s: want a list, got %T", ref.Shape, value)
}
document := make([]any, len(items))
for i, item := range items {
encoded, err := e.jsonValue(*shape.Member, item)
if err != nil {
return nil, err
}
document[i] = encoded
}
return document, nil
case "map":
entries, ok := value.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: want a map, got %T", ref.Shape, value)
}
document := make(map[string]any, len(entries))
for key, entry := range entries {
encoded, err := e.jsonValue(*shape.Value, entry)
if err != nil {
return nil, err
}
document[key] = encoded
}
return document, nil
case "timestamp":
stamp, ok := value.(time.Time)
if !ok {
return nil, fmt.Errorf("%s: want a timestamp, got %T", ref.Shape, value)
}
if format := timestampFormatFor(ref, shape, "unixTimestamp"); format != "unixTimestamp" {
return formatTimestamp(stamp, format), nil
}
return stamp.Unix(), nil
case "blob":
data, ok := value.([]byte)
if !ok {
return nil, fmt.Errorf("%s: want a blob, got %T", ref.Shape, value)
}
return base64.StdEncoding.EncodeToString(data), nil
default:
return value, nil
}
}
func (e requestEncoder) restJSON(operation *Operation, input map[string]any) (*EncodedRequest, error) {
request := &EncodedRequest{Method: operation.HTTP.Method, Query: url.Values{}, Header: http.Header{}}
path := operation.HTTP.RequestURI
if operation.Input == nil {
request.Path = path
return request, nil
}
shape := e.model.shapes[operation.Input.Shape]
body := map[string]any{}
for _, member := range slices.Sorted(maps.Keys(input)) {
ref, ok := shape.Members[member]
if !ok {
return nil, fmt.Errorf("%s has no member %q", operation.Input.Shape, member)
}
value := input[member]
target := e.model.shapes[ref.Shape]
switch ref.Location {
case "uri":
text, err := scalarString(value, ref, target, "iso8601")
if err != nil {
return nil, fmt.Errorf("%s: %w", member, err)
}
path = strings.Replace(path, "{"+member+"+}", escapeGreedyLabel(text), 1)
path = strings.Replace(path, "{"+member+"}", url.PathEscape(text), 1)
case "querystring":
if err := e.restQuery(request.Query, ref, target, value); err != nil {
return nil, fmt.Errorf("%s: %w", member, err)
}
case "header":
text, err := e.headerValue(ref, target, value)
if err != nil {
return nil, fmt.Errorf("%s: %w", member, err)
}
request.Header.Set(ref.LocationName, text)
case "headers":
entries, _ := value.(map[string]any)
for key, entry := range entries {
request.Header.Set(ref.LocationName+key, fmt.Sprint(entry))
}
default:
encoded, err := e.jsonValue(ref, value)
if err != nil {
return nil, err
}
name := member
if ref.LocationName != "" {
name = ref.LocationName
}
body[name] = encoded
}
}
request.Path = path
if payload := shape.Payload; payload != "" {
if document, ok := body[payload]; ok {
body = nil
encoded, err := json.Marshal(document)
if err != nil {
return nil, err
}
request.Body = encoded
}
}
if len(body) > 0 {
encoded, err := json.Marshal(body)
if err != nil {
return nil, err
}
request.Body = encoded
}
if request.Body != nil {
request.Header.Set("Content-Type", "application/json")
}
return request, nil
}
func (e requestEncoder) restQuery(query url.Values, ref ShapeRef, shape *Shape, value any) error {
switch shape.Type {
case "list":
items, _ := value.([]any)
for _, item := range items {
text, err := scalarString(item, *shape.Member, e.model.shapes[shape.Member.Shape], "iso8601")
if err != nil {
return err
}
query.Add(ref.LocationName, text)
}
return nil
case "map":
entries, _ := value.(map[string]any)
for key, entry := range entries {
query.Add(key, fmt.Sprint(entry))
}
return nil
default:
text, err := scalarString(value, ref, shape, "iso8601")
if err != nil {
return err
}
query.Set(ref.LocationName, text)
return nil
}
}
func (e requestEncoder) headerValue(ref ShapeRef, shape *Shape, value any) (string, error) {
if shape.Type == "list" {
items, _ := value.([]any)
texts := make([]string, len(items))
for i, item := range items {
text, err := scalarString(item, *shape.Member, e.model.shapes[shape.Member.Shape], "rfc822")
if err != nil {
return "", err
}
texts[i] = text
}
return strings.Join(texts, ","), nil
}
return scalarString(value, ref, shape, "rfc822")
}
// escapeGreedyLabel escapes a greedy label's segments but keeps its slashes.
func escapeGreedyLabel(value string) string {
segments := strings.Split(value, "/")
for i, segment := range segments {
segments[i] = url.PathEscape(segment)
}
return strings.Join(segments, "/")
}
func scalarString(value any, ref ShapeRef, shape *Shape, defaultTimestamp string) (string, error) {
switch value := value.(type) {
case string:
return value, nil
case bool:
return strconv.FormatBool(value), nil
case int64:
return strconv.FormatInt(value, 10), nil
case float64:
return strconv.FormatFloat(value, 'g', -1, 64), nil
case []byte:
return base64.StdEncoding.EncodeToString(value), nil
case time.Time:
return formatTimestamp(value, timestampFormatFor(ref, shape, defaultTimestamp)), nil
default:
return "", fmt.Errorf("unsupported scalar %T", value)
}
}
func timestampFormatFor(ref ShapeRef, shape *Shape, defaultFormat string) string {
if ref.TimestampFormat != "" {
return ref.TimestampFormat
}
if shape.TimestampFormat != "" {
return shape.TimestampFormat
}
return defaultFormat
}
func formatTimestamp(value time.Time, format string) string {
switch format {
case "rfc822":
return value.UTC().Format(http.TimeFormat)
case "unixTimestamp":
return strconv.FormatInt(value.Unix(), 10)
default:
return value.UTC().Format(time.RFC3339)
}
}
internal/awsmodel/encode_test.go
+124
View File
@@ -0,0 +1,124 @@
package awsmodel_test
import (
"encoding/json"
"net/http"
"net/url"
"testing"
"github.com/mulgadc/spinifex/internal/awsmodel"
"github.com/stretchr/testify/require"
)
func encodedForm(t *testing.T, request *awsmodel.EncodedRequest) url.Values {
t.Helper()
require.Equal(t, http.MethodPost, request.Method)
require.Equal(t, "/", request.Path)
form, err := url.ParseQuery(string(request.Body))
require.NoError(t, err)
return form
}
func TestEncodeRequestAWSQueryWrapsListMembers(t *testing.T) {
request, err := awsmodel.EncodeRequest(awsmodel.IAM, "CreateRole", map[string]any{
"RoleName": "app",
"AssumeRolePolicyDocument": "{}",
"MaxSessionDuration": int64(3600),
"Tags": []any{map[string]any{"Key": "team", "Value": "core"}},
})
require.NoError(t, err)
require.Equal(t, url.Values{
"Action": {"CreateRole"},
"Version": {"2010-05-08"},
"RoleName": {"app"},
"AssumeRolePolicyDocument": {"{}"},
"MaxSessionDuration": {"3600"},
"Tags.member.1.Key": {"team"},
"Tags.member.1.Value": {"core"},
}, encodedForm(t, request))
}
func TestEncodeRequestEC2UsesQueryNamesWithoutMemberWrapper(t *testing.T) {
request, err := awsmodel.EncodeRequest(awsmodel.EC2, "RunInstances", map[string]any{
"ImageId": "ami-0123456789abcdef0",
"MinCount": int64(1),
"MaxCount": int64(1),
"SecurityGroupIds": []any{"sg-0123456789abcdef0"},
"TagSpecifications": []any{map[string]any{
"ResourceType": "instance",
"Tags": []any{map[string]any{"Key": "Name", "Value": "web"}},
}},
})
require.NoError(t, err)
require.Equal(t, url.Values{
"Action": {"RunInstances"},
"Version": {"2016-11-15"},
"ImageId": {"ami-0123456789abcdef0"},
"MinCount": {"1"},
"MaxCount": {"1"},
"SecurityGroupId.1": {"sg-0123456789abcdef0"},
"TagSpecification.1.ResourceType": {"instance"},
"TagSpecification.1.Tag.1.Key": {"Name"},
"TagSpecification.1.Tag.1.Value": {"web"},
}, encodedForm(t, request))
}
func TestEncodeRequestJSONTargetsOperation(t *testing.T) {
tests := []struct {
service awsmodel.Service
operation string
target string
}{
{awsmodel.ECS, "ListClusters", "AmazonEC2ContainerServiceV20141113.ListClusters"},
{awsmodel.ECR, "DescribeRepositories", "AmazonEC2ContainerRegistry_V20150921.DescribeRepositories"},
{awsmodel.ACM, "ListCertificates", "CertificateManager.ListCertificates"},
}
for _, test := range tests {
t.Run(string(test.service), func(t *testing.T) {
request, err := awsmodel.EncodeRequest(test.service, test.operation, map[string]any{"maxResults": int64(5)})
if test.service == awsmodel.ACM {
request, err = awsmodel.EncodeRequest(test.service, test.operation, map[string]any{"MaxItems": int64(5)})
}
require.NoError(t, err)
require.Equal(t, http.MethodPost, request.Method)
require.Equal(t, test.target, request.Header.Get("X-Amz-Target"))
require.Equal(t, "application/x-amz-json-1.1", request.Header.Get("Content-Type"))
var body map[string]any
require.NoError(t, json.Unmarshal(request.Body, &body))
require.Len(t, body, 1)
})
}
}
func TestEncodeRequestRestJSONPlacesMembers(t *testing.T) {
describe, err := awsmodel.EncodeRequest(awsmodel.EKS, "DescribeCluster", map[string]any{"name": "prod cluster"})
require.NoError(t, err)
require.Equal(t, http.MethodGet, describe.Method)
require.Equal(t, "https://eks.example/clusters/prod%20cluster", describe.URL("https://eks.example"))
list, err := awsmodel.EncodeRequest(awsmodel.EKS, "ListClusters", map[string]any{"maxResults": int64(5), "include": []any{"all"}})
require.NoError(t, err)
require.Equal(t, url.Values{"maxResults": {"5"}, "include": {"all"}}, list.Query)
require.Empty(t, list.Body)
create, err := awsmodel.EncodeRequest(awsmodel.EKS, "CreateCluster", map[string]any{
"name": "prod",
"roleArn": "arn:aws:iam::123456789012:role/eks",
"resourcesVpcConfig": map[string]any{"subnetIds": []any{"subnet-1"}},
})
require.NoError(t, err)
require.Equal(t, http.MethodPost, create.Method)
require.Equal(t, "/clusters", create.Path)
require.Equal(t, "application/json", create.Header.Get("Content-Type"))
require.JSONEq(t, `{"name":"prod","roleArn":"arn:aws:iam::123456789012:role/eks","resourcesVpcConfig":{"subnetIds":["subnet-1"]}}`, string(create.Body))
}
func TestEncodeRequestRejectsUnmodelledMember(t *testing.T) {
_, err := awsmodel.EncodeRequest(awsmodel.IAM, "GetRole", map[string]any{"RoleName": "app", "Bogus": "x"})
require.ErrorContains(t, err, `no member "Bogus"`)
}
func TestEncodeRequestS3IsNotSupported(t *testing.T) {
_, err := awsmodel.EncodeRequest(awsmodel.S3, "ListBuckets", map[string]any{})
require.ErrorContains(t, err, "not implemented")
}
internal/awsmodel/error.go
+1 -1
View File
@@ -51,7 +51,7 @@ func isUnmodelledCommonError(service Service, code string) bool {
case IAM, ElasticLoadBalancingV2:
return isUnmodelledQueryError(code)
case STS:
// STS also uses these two runtime input errors even though api-2.json
// STS also uses these two runtime input errors even though the model
// does not declare them on AssumeRole/GetSessionToken.
return isUnmodelledQueryError(code) || code == "ValidationError" || code == "InvalidParameterValue"
case ECS:
internal/awsmodel/export_test.go
+8
View File
@@ -0,0 +1,8 @@
package awsmodel
// Hooks for the external test package.
var ParseSmithyModel = parseSmithyModel
func (m *Model) OperationErrorCodes(operation *Operation) []string {
return m.operationErrorCodes(operation)
}
internal/awsmodel/model.go
+83 -122
View File
@@ -1,26 +1,28 @@
// Package awsmodel loads the AWS api-2.json service definitions used by the
// conformance suite.
// Package awsmodel loads the AWS Smithy service models used by the
// conformance suite and translates them into the api-2.json shaped structs
// its validators read.
package awsmodel
import (
"encoding/json"
"bytes"
"compress/gzip"
"fmt"
"io"
"maps"
"os"
"os/exec"
"os/user"
"path/filepath"
"slices"
"sync"
)
// SourceSDKVersion is the aws-sdk-go release whose module-cache models are
// used by the conformance suite.
const SourceSDKVersion = "v1.55.8"
// ModelRepository is where the embedded models come from. The commit is
// recorded in model_source.go by scripts/sync-aws-models.sh.
const ModelRepository = "github.com/aws/api-models-aws"
const sourceSDKModule = "github.com/aws/aws-sdk-go"
// ModelSourceDescription names the pinned model source for reports.
func ModelSourceDescription() string {
return fmt.Sprintf("%s@%.12s, %s", ModelRepository, ModelCommit, ModelCommitDate)
}
// Service identifies an AWS service model loaded from the SDK module cache.
// Service identifies an AWS service model embedded in this package.
type Service string
const (
@@ -37,77 +39,79 @@ const (
)
// Metadata describes the service and wire protocol represented by a model.
// Protocol uses the api-2.json names: json, rest-json, rest-xml, query, ec2.
// JSONVersion and TargetPrefix are set only for the json protocol.
type Metadata struct {
APIVersion string `json:"apiVersion"`
EndpointPrefix string `json:"endpointPrefix"`
Protocol string `json:"protocol"`
Protocols []string `json:"protocols"`
ServiceAbbreviation string `json:"serviceAbbreviation"`
ServiceFullName string `json:"serviceFullName"`
ServiceID string `json:"serviceId"`
SignatureVersion string `json:"signatureVersion"`
UID string `json:"uid"`
APIVersion string
EndpointPrefix string
JSONVersion string
Protocol string
ServiceFullName string
ServiceID string
SigningName string
TargetPrefix string
}
// Operation describes an AWS API operation and the shapes used by its input,
// output and declared errors.
type Operation struct {
Name string `json:"name"`
HTTP HTTP `json:"http"`
Input *ShapeRef `json:"input"`
Output *ShapeRef `json:"output"`
Errors []ShapeRef `json:"errors"`
Name string
HTTP HTTP
Input *ShapeRef
Output *ShapeRef
Errors []ShapeRef
}
// HTTP describes an operation's HTTP binding.
type HTTP struct {
Method string `json:"method"`
RequestURI string `json:"requestUri"`
ResponseCode int `json:"responseCode"`
Method string
RequestURI string
ResponseCode int
}
// Shape describes a value in an AWS service model. Depending on Type, it can
// refer to structure members, a list member, or map keys and values.
type Shape struct {
Type string `json:"type"`
Required []string `json:"required"`
Members map[string]ShapeRef `json:"members"`
Member *ShapeRef `json:"member"`
Key *ShapeRef `json:"key"`
Value *ShapeRef `json:"value"`
Enum []string `json:"enum"`
Min *float64 `json:"min"`
Max *float64 `json:"max"`
Pattern string `json:"pattern"`
LocationName string `json:"locationName"`
TimestampFormat string `json:"timestampFormat"`
Payload string `json:"payload"`
Flattened bool `json:"flattened"`
Sensitive bool `json:"sensitive"`
Exception bool `json:"exception"`
Fault bool `json:"fault"`
Error *ErrorInfo `json:"error"`
Type string
Required []string
Members map[string]ShapeRef
Member *ShapeRef
Key *ShapeRef
Value *ShapeRef
Enum []string
Min *float64
Max *float64
Pattern string
LocationName string
TimestampFormat string
Payload string
Flattened bool
Sensitive bool
Union bool
Exception bool
Fault bool
Error *ErrorInfo
}
// ErrorInfo describes an error shape's code and HTTP classification.
type ErrorInfo struct {
Code string `json:"code"`
HTTPStatusCode int `json:"httpStatusCode"`
SenderFault bool `json:"senderFault"`
Code string
HTTPStatusCode int
SenderFault bool
}
// ShapeRef names another shape and carries any wire binding specific to the
// place where it is referenced.
type ShapeRef struct {
Shape string `json:"shape"`
ResultWrapper string `json:"resultWrapper"`
Location string `json:"location"`
LocationName string `json:"locationName"`
QueryName string `json:"queryName"`
TimestampFormat string `json:"timestampFormat"`
Flattened bool `json:"flattened"`
Streaming bool `json:"streaming"`
XMLAttribute bool `json:"xmlAttribute"`
Shape string
ResultWrapper string
Location string
LocationName string
QueryName string
TimestampFormat string
Flattened bool
Streaming bool
XMLAttribute bool
}
// Model is an indexed AWS service definition.
@@ -118,13 +122,6 @@ type Model struct {
shapes map[string]*Shape
}
type modelDocument struct {
Version string `json:"version"`
Metadata Metadata `json:"metadata"`
Operations map[string]*Operation `json:"operations"`
Shapes map[string]*Shape `json:"shapes"`
}
type loadResult struct {
model *Model
err error
@@ -132,31 +129,6 @@ type loadResult struct {
var modelCache sync.Map
var modelRoot = sync.OnceValues(resolveModelRoot)
func resolveModelRoot() (string, error) {
if root := os.Getenv("GOMODCACHE"); root != "" {
return filepath.Join(root, sourceSDKModule+"@"+SourceSDKVersion), nil
}
if output, err := exec.Command("go", "env", "GOMODCACHE").Output(); err == nil {
if root := string(output); root != "" {
return filepath.Join(stringTrimSpace(root), sourceSDKModule+"@"+SourceSDKVersion), nil
}
}
currentUser, err := user.Current()
if err != nil {
return "", fmt.Errorf("awsmodel: resolve Go module cache: %w", err)
}
return filepath.Join(currentUser.HomeDir, "go", "pkg", "mod", sourceSDKModule+"@"+SourceSDKVersion), nil
}
func stringTrimSpace(value string) string {
for len(value) > 0 && (value[len(value)-1] == '\n' || value[len(value)-1] == '\r' || value[len(value)-1] == ' ' || value[len(value)-1] == '\t') {
value = value[:len(value)-1]
}
return value
}
// Services returns the supported service identifiers in stable order.
func Services() []Service {
services := make([]Service, 0, len(modelFiles))
@@ -167,45 +139,21 @@ func Services() []Service {
return services
}
// Load parses and indexes the cached SDK model for service. Each service is
// Load parses and indexes the embedded model for service. Each service is
// parsed at most once and the resulting Model is safe for concurrent reads.
func Load(service Service) (*Model, error) {
path, ok := modelFiles[service]
file, ok := modelFiles[service]
if !ok {
return nil, fmt.Errorf("awsmodel: unsupported service %q", service)
}
loader, _ := modelCache.LoadOrStore(service, sync.OnceValue(func() loadResult {
root, err := modelRoot()
contents, err := readModel(file)
if err != nil {
return loadResult{err: err}
return loadResult{err: fmt.Errorf("awsmodel: read %s model: %w", service, err)}
}
contents, err := os.ReadFile(filepath.Join(root, path))
if err != nil {
return loadResult{err: fmt.Errorf("awsmodel: read cached %s model at %s: %w", service, filepath.Join(root, path), err)}
}
var document modelDocument
if err := json.Unmarshal(contents, &document); err != nil {
return loadResult{err: fmt.Errorf("awsmodel: parse %s model: %w", service, err)}
}
if document.Version != "2.0" {
return loadResult{err: fmt.Errorf("awsmodel: %s model version is %q, want %q", service, document.Version, "2.0")}
}
if len(document.Operations) == 0 || len(document.Shapes) == 0 {
return loadResult{err: fmt.Errorf("awsmodel: %s model has no operations or shapes", service)}
}
model := &Model{
service: service,
metadata: document.Metadata,
operations: document.Operations,
shapes: document.Shapes,
}
if err := model.validateReferences(); err != nil {
return loadResult{err: err}
}
return loadResult{model: model}
model, err := parseSmithyModel(service, contents)
return loadResult{model: model, err: err}
}))
load, ok := loader.(func() loadResult)
if !ok {
@@ -215,10 +163,23 @@ func Load(service Service) (*Model, error) {
return result.model, result.err
}
func readModel(file string) ([]byte, error) {
compressed, err := modelFS.ReadFile(file)
if err != nil {
return nil, err
}
reader, err := gzip.NewReader(bytes.NewReader(compressed))
if err != nil {
return nil, err
}
defer func() { _ = reader.Close() }()
return io.ReadAll(reader)
}
// Service returns the identifier used to load the model.
func (m *Model) Service() Service { return m.service }
// Metadata returns the service metadata from api-2.json.
// Metadata returns the service metadata.
func (m *Model) Metadata() Metadata { return m.metadata }
// Operation resolves an operation by its model key, such as DescribeInstances.

Some files were not shown because too many files have changed in this diff Show More