Enables the rule with functions: false, since hoisted declarations are safe, and defines CONDITION_FIELD_TEXT and the handoff message listener before their first use.
Turns on oxlint's eslint/complexity rule (max 20) and brings the 18 offending functions under the limit by extracting helpers, hooks and subcomponents with no behaviour change.
Shares the RDS subnet and parameter group option builders through rds-form-fields, and rebuilds dist.
* refactor: share the ECR registry address precedence in domains/ecr
* feat: expose every gateway's ECR registry host to the EKS service
* fix: skip wildcard peer hosts when listing ECR registry hosts
* feat: trust every gateway's ECR registry host on EKS workers
* docs: describe how EKS workers trust every gateway's ECR registry host
* docs: split the ECR gateway limits to one sentence per line
* refactor: format the registry URI host in domains/ecr so the daemon avoids the awsapi adapter
* docs: state that a refused gateway fails over at once and record the 3-node ECR measurements
* fix: add GatewayMTU capping the gateway MTU at the internet path MTU
* fix: set gateway_mtu on the VPC gateway router port so oversized packets get ICMP frag-needed
* fix: wire the gateway MTU into vpcd and cover reconcile convergence
* feat: add host MSS clamp tables for the veth and routed uplinks
* fix: shorten MSS clamp comments and log the periodic ensure at debug
* feat: run the host MSS clamp loop in vpcd
* fix: run nft under the ambient CAP_NET_ADMIN instead of sudo
* docs: describe the gateway MTU and the host MSS clamp for the internet path
Bound the zone reload wait with a test context, generate 2048-bit keys in
cert renew tests via the existing seam, replace 100ms negative waits with
deterministic checks, and run tests bound by production timeouts in parallel.
Ubuntu 26.04's openvswitch-switch declares Alias=openvswitch.service, so
enabling it leaves an /etc symlink that distro_ovs_unit read as a second,
EL-style umbrella unit. setup-ovn.sh then refused to pick one and every
ubuntu-26.04 bootstrap failed. A symlink back to openvswitch-switch.service
now counts as the Debian unit; a real or masked openvswitch.service still
does not.
ResourcesPerPage outside 1-100 and a malformed PaginationToken now return
InvalidParameterException. An unbounded page size let start+size overflow
negative and panic on the slice.
The test checked IPsec config and that AES-GCM SAs exist, and only warned when
a 5s idle capture saw no ESP, so a cluster sending plaintext Geneve passed.
It now launches an instance pair on distinct nodes, pings 200 x 1000 bytes
between them, and for each direction fails unless the outbound SA counters to
the peer's ovn-encap-ip rose by the payload, an outbound-only capture on the
interface routing to the peer saw ESP or UDP/4500 ESP, and saw no UDP/6081.
The distinct-node launch loop moves from the baseline cross-host test into a
shared helper. The test is no longer parallel since it now launches instances.
Vendor the route-53 2013-04-01 Smithy model at the pinned commit, add the
Route53 service, and pass route53awsapi.OperationInventory() to the
generator. coverage.json and docs/coverage/route53 now list the 13 served
operations; the Spinifex activation extensions stay out of the model count.
The generator now fails when a served service has no embedded model, so
the next one cannot drop out of the report silently. Request encoding
skips rest-xml by protocol, which covers Route 53 as well as S3.
ensure_clustered_db_storage() collected only databases in standalone
format, so --recreate-db returned early when the NB/SB databases were
already clustered and removed nothing. ovn-ctl then served the stale
database and silently ignored the RAFT flags, which is the failure the
function exists to prevent, reached from the other direction. Because
the old database carries the previous member list and server ID, the
node rejoined the earlier cluster instead of replacing it: re-forming a
production cluster brought the create node back as a follower, and
ovn-nbctl over the leader-only unix socket failed with "database
connection failed ()".
install-node.sh passes --recreate-db on every database node on every
run, so formation already intended a purge. Remove whatever NB/SB
state is present, standalone or clustered, and keep refusing a
standalone database when the flag is absent.
Two further gaps in the same path. systemctl stop returns once the unit
is inactive rather than once the processes are gone, so the purge now
waits for the database servers and escalates to SIGKILL, failing the
run if one survives rather than unlinking files under it. The pattern
is anchored on the OVN database directory so it can never reap the OVS
conf.db server. The lock files, the compute-node move-asides and an
earlier run's standalone backup are cleared alongside the databases.
* feat: add a dedicated SQS message store JetStream server separate from control-plane NATS
* test: cover the SQS store server start, validation and memory limit
HAProxy's short 'option httpchk GET <path>' form sends HTTP/1.0 with no Host header, which Envoy rejects with 426 (and 400 for HTTP/1.1 without Host), so Envoy-fronted targets never became healthy. Match the AWS request shape: GET <path> HTTP/1.1, Host set to the target IP (with the port only when it is not the scheme default), and User-Agent ELB-HealthChecker/2.0.
MemoryObjectStore.DeleteObject silently succeeded for an absent key while
Predastore answers NoSuchKey, so handlers that treated a missing key as fatal
passed unit tests and failed in production. The fake now returns
NoSuchKeyError, and delete paths that must be idempotent (DeleteSnapshot's
sweep and document delete, DeleteVolume, DeregisterImage, the admin image
removal sweep, and ECR upload and GC cleanup) treat it as already deleted.
* feat: let a registration select REST-JSON or JSON 1.0 query-compatible error envelopes
* feat: add a per-registration internal error code and decode envelopes through the AWS SDK v2
* feat: let a JSON 1.0 registration namespace the __type of its errors
* feat: match captured SQS error bytes on the JSON 1.0 envelope
* refactor: move gateway request audit into ingress/aws/audit
First generic-ingress extraction from the root gateway under ADR-0001.
requestaudit.go and slogRequestLogger move unchanged in behaviour to
package audit. The gateway's auth flow and dispatch populate the record
through the exported From and Record setters at the same call sites.
Record.Fields is exported for the SigV4-path tests that stay in gateway.
The migration-record deferral bullet is replaced by a move row.
* docs: cite #1285 in the request-audit move row
UpdateKVRecordTags now goes through kvutil.Update, which commits at the
revision it read and retries on conflict, instead of a plain put that
reverted any tag or non-tag write landing between its read and write.
Volume and AMI tag mirrors write ebsmetadata documents to the object
store, which has no conditional put; characterisation tests pin that
they still lose interleaved writes.
aws iam get-user sends no UserName and got MissingParameter. The gateway now
applies the AWS default before authorization: an IAM user is treated as naming
themselves, root gets the account root shape, and a role session gets
ValidationError.
Refs mulgadc/spinifex#1275
* refactor: move accountteardown to operator/accountlifecycle
Mechanical relocation under the amended ADR-0001 accountteardown row.
Package renamed to accountlifecycle; importers keep the accountteardown
alias. No API, NATS subject, KV access or behaviour change. Replacing
direct domain/adapter cleanup calls with owner capabilities stays
recorded as deferred debt.
* docs: cite #1277 in the accountteardown move row
* refactor: relocate gateway/{ecs,elbv2} to domains/{ecs,elbv2}/awsapi
Mechanical ADR-0001 relocation of the ECS and ELBv2 AWS action adapters
to their domain homes, as #1265 did for IAM, STS, EKS and RDS. APIs, NATS
subjects, KV buckets and behaviour are unchanged. The root gateway
dispatch still imports both packages, recorded as gateway (service
registration) debt; the quota live ELBv2 counter import moves with it.
* docs: cite spinifex#1273 in the migration record
The IAM, STS, RDS, EKS and ELBv2 relocations moved their packages out of
spinifex/handlers, which the S4, S6 and S7 network invariant scans still
walked, so those packages silently dropped out of the scans. Re-add their
new homes alongside domains/ecs. The checked rules are unchanged.
* refactor: relocate handlers/ecs to domains/ecs
Mechanical move per the ADR-0001 handlers/ecs row (files already
organised by resource). Package renamed handlers_ecs -> ecs; importers
alias ecsdomain. APIs, NATS subjects, storage and behaviour unchanged.
handlers is now a retired legacy root; ECS allowlist debt retargeted to
domains/ecs and R5 holds it to the interim daemon-import ban.
* docs: cite spinifex#1271 in the ECS relocation row
* fix: point network invariant scans at domains/ecs
The ADR-0007 S4, S6 and S7 scans walked spinifex/handlers, which no
longer exists after the ECS move. Scan domains/ecs, the same code.
Mechanical move permitted by the ADR-0001 lbagent amendment. The package
becomes elbv2 under agents/; importers alias it lbagent. APIs, on-guest
paths, NATS subjects and behaviour are unchanged. Q-30's data-plane
boundary is recorded as deferred debt in the migration record.
Also brings docs/DESIGN.md up to date with the current code: gateway
dispatch and registrations, EC2 handler and NATS request signatures,
RunInstances placement, daemon subscription form, volume mount subject,
DescribeInstances scatter-gather, and the package layout.
Mechanical whole-package move under the amended ADR-0001 daemon row. Package name, APIs, NATS subjects, storage and behaviour are unchanged; only import paths and path comments outside the package change.
The interim daemon -> gateway -> handlers layering guardrail, the ADR-0007 network invariant scans and the instance-state ratchet locate the daemon by directory, so they now match runtime/daemon. Q-28's ownership and deployment refactor is recorded as deferred debt.
* refactor: relocate gateway/bedrock into domains/ochre
Mechanical ADR-0001 whole-package relocation. The Ochre domain core,
stores, inference engine and Bedrock action layer join package ochre
with their package-private code and tests intact; no identifier, API,
NATS subject, KV bucket, metric, error string or behaviour changes.
Separating the action adapters into domains/ochre/awsapi is deferred
and recorded as debt.
* docs: cite spinifex#1266 in the migration record
* refactor: move IAM, STS, EKS and RDS AWS adapters to domains/<svc>/awsapi
Mechanical ADR-0001 relocation of gateway/{iam,sts,eks,rds} to
domains/{iam,sts,eks,rds}/awsapi. APIs, NATS subjects, storage and
behaviour are unchanged. The root gateway dispatch still imports each
package and is recorded as service-registration debt. gateway/bedrock
stays: ADR-0001 moves only its action adapters, and the package also
holds the Ochre domain core, stores and inference engine.
* docs: cite spinifex#1265 in the migration record
* refactor(eks): move handlers/eks to domains/eks/cluster
Mechanical ADR-0001 relocation of the remaining EKS implementation,
nodegroups included, alongside the access and addon packages. APIs, NATS
subjects, KV buckets and behaviour are unchanged; only the package name
changes from handlers_eks to cluster (imported as ekscluster). The
frozen-legacy entry is removed, allowlist entries now name
domains/eks/cluster, and the stale operator/cli entry is removed.
Nodegroup service extraction is recorded as deferred.
* docs: name the EKS cluster relocation PR in the migration record
* feat(vm): attach host block device exports via raw over host_device
EBSRequest gains optional BlockDevice and BlockSizeBytes. A single
classifier decides whether a request is an NBD export, a block export or
invalid, and buildDrives, AttachVolume, launch readiness, the QMP greeting
deadline and restore all use it. A block data volume is attached as a
sized raw node over a host_device file node with the same node name,
iothread and virtio-blk device as an NBD volume; boot and EFI volumes
reject block exports. NBD output is unchanged.
* feat(ebsprovider): replace PublishedVolume.NBDURI with a typed export descriptor
PublishedVolume now carries an Export of kind nbd or block, checked by VolumeExport.Validate.
The NATS client rejects an invalid export in a PublishVolume reply, and the server refuses to relay one, reporting it as internal.
SchemaVersion moves to 2.
viperblockd, qemunbdd and the memory provider publish nbd exports exactly as before.
Image import refuses a non-nbd export and still unpublishes.
Conformance requires a valid export, and the NBD-dialling suites fail explicitly on a non-nbd export.
* fix(ebsprovider): preserve v1 export compatibility
---------
Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>
* refactor(rds): move handlers/rds to domains/rds/instance
Mechanical ADR-0001 relocation of the remaining RDS instance resource to
its resource package, alongside engine, parametergroup and subnetgroup.
APIs, NATS subjects, KV buckets and behaviour are unchanged; only the
package name changes from handlers_rds to instance (imported as
rdsinstance). The frozen-legacy entry is removed and the allowlist
entries now name domains/rds/instance; the recorded debt remains.
* docs: name the RDS instance relocation PR in the migration record
* fix: commit launch template version deletes through the header revision so a concurrent default change cannot name a deleted version
* fix: keep version-delete rollbacks from leaving headerless bodies or a false default-version refusal
* fix: resolve a GetSessionToken session as its IAM user in presigned GetCallerIdentity, and reject an identity with no ARN
* fix: report the account ID as root's UserId for a root session in presigned GetCallerIdentity
* refactor(sts): move handlers/sts to domains/sts
Mechanical ADR-0001 relocation of the STS implementation. APIs, NATS
subjects, KV buckets and behaviour are unchanged; only the package name
changes from handlers_sts to sts (imported as stsdomain). The frozen-legacy
entry is removed and the allowlist entries now name domains/sts; the
per-consumer IAM capability debt remains.
* docs: name the STS relocation PR in the migration record
Bumps the docker-images group with 1 update in the /build directory: golang.
Bumps the docker-images group with 1 update in the /docs/terraform-workbooks/demo-app directory: distroless/static.
Bumps the docker-images group with 1 update in the /scripts directory: debian.
Updates `golang` from `5c3e9b4` to `972b411`
Updates `distroless/static` from `e2e927e` to `18f4cc2`
Updates `debian` from `a29215f` to `62bd423`
---
updated-dependencies:
- dependency-name: golang
dependency-version: 1.27.3-bookworm
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: docker-images
- dependency-name: distroless/static
dependency-version: nonroot
dependency-type: direct:production
dependency-group: docker-images
- dependency-name: debian
dependency-version: trixie-slim
dependency-type: direct:production
dependency-group: docker-images
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* refactor(iam): move handlers/iam to domains/iam
Relocate IAM's implementation and its test double to the ADR-0001 home
with git mv. The package is renamed from handlers_iam to iam and its
importers use the iamdomain alias. APIs, NATS subjects, KV buckets and
behaviour are unchanged.
The cross-domain allowlist entries now target domains/iam; the two
frozen-legacy entries and the operator/cli entry, which no longer breaks
the legacy-implementation rule, are removed. The migration record states
that per-consumer IAM capabilities remain debt but were not a
prerequisite to the move.
* docs: record the IAM move against its PR
Store the override record with a state field kept separate from the admin
wire type. Every set and clear is revision-guarded and refuses a sealed
record with Blocked, and every read of a sealed record, including
admission, returns Blocked. The read-failure fallback to configured limits
is unchanged.
Add Service.SealAccountOverrides, accepted only when IAM already holds the
account as TERMINATING. It creates the override bucket if needed after that
check, creates or updates the record as sealed, and succeeds only on a
leader-accepted write, returning its revision. awsgw serves it on
admission.v1.SealAccountQuotaOverride. Nothing calls it yet.
Replace the unwired delete-based CleanupAccount.
Joining a coded error with RetryAfter rendered the bare code on a second
line, so logged and stored reasons read "...: ServiceUnavailable
ServiceUnavailable". Add WithRetryAfter, which wraps the error and keeps
its text while ResolveRetryAfter and ResolveErrorDetail still resolve the
hint and the code. Use it for the seal-capability refusal and the
incomplete DescribeInstances and DescribeInstanceTypes fan-out errors.
Wire the EC2 account-settings seal into account termination as a
StageAccount reaper. Immediately before every seal attempt the reaper
probes each configured node's new read-only, per-node seal-capability
subject and requires exactly one identified reply per node at capability
version 1 or later. Missing, duplicate, wrong-identity, malformed, error,
older-version or unreachable responders send no seal and return a
retryable ServiceUnavailable, so the account stays TERMINATING until the
cluster is compatible.
The reaper completes only on a sealed reply for the account with a
non-zero revision; a fresh termination run seals again. Daemons validate
their node ID before subscribing, so an invalid ID serves no capability
and reads as incompatible.
Replace the unwired delete-based CleanupAccount with an owner-local sealed
state in the existing account-settings KV value. All settings mutations now
use revision-guarded create/update; a sealed record rejects every read and
mutation with Blocked and cannot be recreated. Records without a state
field remain active.
SealAccount CAS-writes the sealed value whether the key is present, absent
or tombstoned, and succeeds only on a leader-accepted write, returning its
revision. It first checks, through a read-only IAM account-status reader,
that the account is already TERMINATING; otherwise it refuses with
IncorrectState and writes nothing.
The daemon serves the internal subject ec2.v1.SealAccountSettings. Nothing
calls it yet.
Pin that account teardown removes the usage counter but leaves the
per-account quota override, so a reused account ID inherits raised or
Unlimited limits. Add an unwired, idempotent CleanupAccount on the quota
service: a present override key is removed by a revision-guarded delete
and reported deleted with its revision; a missing key is reported only as
observed-absent-at-scan.
Characterise that EC2 account settings survive today's account teardown,
and add an owner-side CleanupAccount in domains/ec2/account that deletes
one account's settings key.
A present key is removed with a revision-guarded delete, which the stream
leader orders regardless of which replica served the preceding read, and
is reported deleted with that revision. A key that a direct get does not
find is reported only as observed absent at scan time, because KV direct
gets can be answered by a lagging follower. Store errors and exhausted
revision conflicts are failed, never absent. The capability refuses empty,
malformed and global account IDs and never falls back to the global key.
Nothing calls it yet: no NATS subject, no accountteardown wiring.
The inventory and disclosure test helpers subscribed on one connection and
requested on another, so a request could reach the server before its SUBs
and fail with no responders under load.
A node that lost the election logged holder=<itself>, which reads as though it holds the lock. Lease lines now carry the local node as node, and a refused claim logs the node that owns the key, read from the Get the claim already makes, as holder at debug level.
TestEbsRequestWithTrace_ProducerConsumerLinked read the span recorder as
soon as UnmountOne returned, but the handler replies before its deferred
span.End runs, so under -race the recorder could hold only the producer
span. The handler now signals after its span ends and the test waits for
that signal instead of racing it.
DeleteSnapshot checked only volume documents, so a snapshot backing a
registered AMI deleted anyway and left the AMI pointing at nothing.
Add a strict, whole-cluster AMI read next to the existing volume-clone
check and refuse with InvalidSnapshot.InUse when a registered AMI
names the snapshot; an unreadable or undecodable AMI document fails
the delete closed with ServerInternal.
The e2e teardown sweep now deregisters images before deleting
snapshots, since a snapshot sweep that runs first would now be
blocked by its AMI.
Known residual, out of scope here: RegisterImage reading the snapshot
and DeleteSnapshot's clone/AMI checks are not atomic, so a concurrent
RegisterImage can still race a DeleteSnapshot that passed its checks.
* fix: report an unreadable volume document as InternalError, not NotFound
* fix: ForceDetachVolume refuses an unreadable volume record
ForceDetachVolume mapped every metadata read error to InvalidVolume.NotFound.
An unreadable (present but undecodable or unreconstructable) document now
returns InternalError and leaves the attachment untouched; a genuinely
absent document or another account's volume still returns NotFound.
This is the call spx admin account delete --force uses, so an unreadable
volume no longer reads as already detached.
Pin current behaviour only; no production change. The tests document why
owner cleanup of images and snapshots is not yet safe:
- a CreateImage snapshot does not pin its source volume;
- DeleteSnapshot ignores a registered AMI;
- a snapshot deleted inside the clone window strands the new volume;
- CreateImage of a retained instance needs its launch AMI;
- a snapshotted DeleteOnTermination volume survives terminate.
Add domains/ec2/accountcleanup, an unwired EC2-owned capability that
inventories key pairs, placement groups and launch templates for one
account, applies their owner delete rules, and proves each disposition
(deleted, blocked, failed, residue) by owner re-scan, never by Describe.
Cleanup deletes only the proven-safe subset: placement groups with no
members and key pairs that no non-terminated instance, open spot request
or live launch-template version references, plus unreachable
launch-template residue. Live launch templates are always blocked while
the EKS nodegroup reference is unresolved. Unknown or unreadable
reference state fails closed. Reference readers are injected; no action
adapter, NATS subject, IAM or accountteardown wiring is added.
Owner delete corrections shared by the public Delete* actions:
- DeleteKeyPair keeps metadata when the public-key delete fails, removes
leftover public-key objects by name, and fails closed on unreadable or
undecodable metadata.
- DeleteLaunchTemplate removes all version bodies and the name index
before the revision-guarded header delete, and returns InternalError
when a step fails.
- CreateLaunchTemplateVersion commits through the header revision, so a
concurrent delete leaves no headerless body.
- DeletePlacementGroup is revision-guarded against concurrent
reservations and maps store read errors to InternalError.
- Placement-group and launch-template deletes clear their tag-index
entries.
On Alpine 3.21 apk does not run the mkinitfs trigger inside the libguestfs
appliance, so apk upgrade left the old initramfs beside the new kernel. The
guest could not load virtio_blk and dropped to the emergency shell, which
stalled EKS cluster creation. Run mkinitfs for the installed kernel and fail
the build if the initramfs lacks its virtio_blk module.
build-system-image.sh never upgraded the base cloud image, so every Alpine
system AMI kept the kernel its base image was released with. Run apk upgrade
before installing manifest packages so rebuilt AMIs carry the branch's current
linux-virt (6.18.57-r0 on 3.24, 6.12.113-r0 on 3.21; CVE-2026-99999).
The ELBv2 microVM defaulted to Alpine 3.20, which is EOL and will never get a
fixed kernel. Default to 3.24.
Sidebar built from a typed NAV array with links rendered by the menu
button; one dashboard ResourceCard; signOut owns the clear-everything rule.
SystemImageRequired rechecks itself, wizards read mutation state, and
dialogs reset by unmounting. IAM policy hooks take a principal kind; ECS,
RDS, Nodes, S3 and ELBv2 share their duplicated tables and fields; EC2 CLI
builders use the flag helpers; CIDR fields validate with isValidCidr and
AWS prefix ranges.
Signed fetch: signedAdminFetch wraps signedProxyFetch, which resolves
credentials itself, removing 11 per-caller checks.
Tags: one tagSchema, toSdkTags and diffTags; the tags editor moves to
components/ on useForm and TagsFieldArray.
Utilities: arnResourceId, byName, sameSet and shellSingleQuote replace
22 local copies.
Components: DeleteResourceButton (13 EC2 detail pages), ResourceList
(23 list pages), ResourceDeleteDialog (RDS, ECR), SelectField,
TextField, VpcSelectField, WizardFailureBanner, EditorShell and a
mode-driven ListenerDialog.
Tests: test/render.tsx with renderMutationHook, mockRouter,
createFakeSdk and mockAwsClients; the RDS and ELBv2 hook flows move to
mutations/*-flow.test.tsx.
Deletes unreferenced components, hook fields, types and query options,
drops exports used only in their own file and redundant Outlet route
components, and removes tests that assert their own fakes, restate query
key literals or repeat vitest's clearMocks. Query-key tests become
invalidation-prefix behaviour tests, and the Go compression tests now
drive the real handler chain.
* test: characterise current tagging read, write and aggregation behaviour
* docs: record tagging capability discovery and narrowed ADR-0001 cut
* test: update ACM tagging characterisation after key preservation
* feat: add read-only TaggedLister capability in domains/tagging
* feat: add AWSGW tag index and ELBv2 TaggedLister adapters
* refactor: move GetResources aggregation to domains/tagging
* docs: close out the ADR-0001 tagging boundary
* docs: repoint tagging migration citations to the rebased commits
* refactor: move telemetry file naming to contract
* docs: record telemetry contract move
* refactor: move aws xml response helpers
* docs: record aws xml helper move
* refactor: move aws tag projections to foundation
* docs: record aws tag projection move
* refactor: move account key to state foundation
* docs: record account key state move
* refactor: move host privilege escalation to runtime
* docs: record host privilege move
* chore: resolve lint findings from package-boundary moves
* refactor: move process supervision helpers to runtime host
* docs: record process supervision move
* refactor: move nbd endpoint helpers to compute runtime
* docs: record nbd endpoint move
* refactor: move image catalogue to operator
* docs: record image catalogue move
* refactor: move resource id generation to aws foundation
* docs: record resource id move
* refactor: move predastore role to runtime roles
* docs: point service manifest at acm domain
* docs: record predastore role move
* refactor: move northstar role to runtime roles
* docs: record northstar role move
* refactor: move awsgw role to runtime roles
* docs: record awsgw role move
* refactor: move error payload helpers to aws errors
* docs: record error payload move
* refactor: move account identifiers to aws identifiers
* docs: record account identifier move
* refactor: move NATS transport to messaging foundation
* docs: record NATS transport move
* refactor: move system instance contract to EC2 domain
* docs: record system instance move
* refactor: move system VPC builder to network domain
* docs: record system VPC move
* refactor: move instance metadata service to EC2 domain
* docs: record guest metadata move and dependency-rule debt
* refactor: move DNS control plane to DNS domain
* docs: record DNS domain move
* refactor: move base zone bootstrap to DNS domain
* docs: record base zone bootstrap move
* refactor: replace SystemAccountID with GlobalAccountID
* docs: record SystemAccountID removal
* refactor: extract CA key-pair loader to foundation pki
* docs: record foundation pki extraction
* refactor: move tenant CA paths to ACM domain
* docs: record tenant CA path move
* refactor: move IPsec certificate handling to network host
* docs: record IPsec certificate move
* refactor: move Ochre serving lifecycle to the Ochre domain
handlers/bedrock becomes domains/ochre. Ochre is the platform service;
Bedrock names only its AWS API surface. Persisted and observable
identifiers keep their bedrock names.
* docs: record Ochre domain move
* refactor: move Ochre vector store to the Ochre domain
handlers/ochrevector becomes domains/ochre/vector. Callers import it as
ochrevector because vector is a common local name in the gateway.
* docs: record Ochre vector store move
* refactor: move Hugging Face Hub client to the Ochre domain
gateway/bedrock/hfhub becomes domains/ochre/hfhub. Its only caller is
the spx admin ochre weights pull.
* docs: record Hugging Face Hub client move and Ochre gateway debt
* refactor: move EC2 tag mutation helpers to the EC2 domain
utils/tagmirror.go becomes domains/ec2/tagmirror. It is EC2-wide tag
mutation behaviour, and the tags resource package cannot host it
without a resource-to-resource import cycle.
* docs: record EC2 tag mutation move and migrate split finding
* refactor: move network packages under the network domain
Move spinifex/network/* to spinifex/domains/network/* as one enclosing
move so the L0-L5 layer invariants keep classifying every package from
a single root. Package names and dependency direction are unchanged;
invariant scan roots, the RG-10 sudo allow-list path, CODEOWNERS and
the repo-root walk in the IMDS netns unit test follow the new path.
* docs: record network domain move
* refactor: move standalone EC2 resource packages to the EC2 domain
Move handlers/ec2/{account,key,launchtemplate,placementgroup,
spotinstance,volumestate} to domains/ec2/<resource>. Each package is
named after its directory and callers import it as ec2<resource>;
volumestate already carried its directory name.
* docs: record standalone EC2 resource moves
* refactor: move EC2 VPC networking packages to the EC2 domain
Move handlers/ec2/{vpc,eigw,eip,igw,natgw,routetable} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>. The reconcile drift-watch source scan now matches the
ec2<resource> bucket qualifiers; its rule is unchanged.
* docs: record EC2 VPC networking moves
* refactor: move EC2 compute and storage packages to the EC2 domain
Move handlers/ec2/{instance,volume,snapshot,image,tags} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>, which empties handlers/ec2. The daemon's e2e suite
selection now lists the moved EC2 resource paths in place of
handlers/ec2, so EC2-only changes select the daemon suites again.
* docs: record EC2 compute and storage moves
* refactor: move ECR credential provider to the EKS agent tree
Move the kubelet exec credential provider implementation from
cmd/ecr-credential-provider to agents/eks/credentialprovider, exposing
one Main(in, out) entry point. The binary keeps its name and FIPS boot
import as a thin main. Moving out of cmd/ brings the code under gosec;
the one G101 hit on the kubelet API version constant is annotated.
* docs: record ECR credential provider move
* refactor: move EKS gateway publish helper to the EKS agent tree
Move the eks-gateway-publish implementation to agents/eks/gatewaypublish
behind Run(ctx, Config) error. The binary keeps flag definitions,
environment defaults, parsing, its log prefix and exit code; each former
fatal site returns an error with the same text. The retry wait now also
honours context cancellation, which the binary never triggers.
* docs: record EKS gateway publish move
* refactor: move EKS gateway fetch helper to the EKS agent tree
Move the eks-gateway-fetch implementation to agents/eks/gatewayfetch
behind Run(ctx, Config) error, writing its TSV to Config.Out. The binary
keeps flag definitions, environment defaults, parsing, its log prefix and
exit code; each former fatal site returns an error with the same text.
* docs: record EKS gateway fetch move
* refactor: move EKS webhook cert helper to the EKS agent tree
cmd/eks-webhook-cert keeps flag parsing, the log prefix and the exit;
agents/eks/webhookcert owns Config and Run. Output, error text, file
modes and exit codes are unchanged.
* docs: record EKS webhook cert move
* refactor: move EKS konnectivity cert helper to the EKS agent tree
cmd/eks-konnectivity-cert keeps flag parsing, the log prefix and the
exit; agents/eks/konnectivitycert owns Config and Run. Output, error
text, file modes and exit codes are unchanged.
* docs: record EKS konnectivity cert move
* refactor: move EKS token webhook to the EKS agent tree
cmd/eks-token-webhook keeps the -addr flag, environment-derived
config, logging setup and exits; agents/eks/tokenwebhook owns Config
and Run. The handlers/eks TokenReview result type stays as recorded
temporary debt.
* docs: record EKS token webhook move
* refactor: move ECS agent runtime and credentials packages to the ECS agent tree
* docs: record ECS agent subpackage move
* refactor: move ECS agent implementation to the ECS agent tree
cmd/ecs-agent keeps process lifecycle: signal context, the separate New
and Run calls, both failure log lines, the exit and the ldflags-set
build version, now passed in through Config.AgentVersion.
agents/ecs/agent owns Config, LoadConfig and the agent.
* docs: record ECS agent move
* refactor: move RDS guest agent to agents/rds/agent
* docs: record RDS guest agent move
* refactor: move installer packages to operator/installer
* docs: record installer move
* refactor: move spx command tree to operator/cli
Moving the command tree out of cmd/ brings it under errcheck and gosec.
Flag and config wiring now fails loudly instead of silently dropping a
binding, listing commands fail when they cannot write their table, and
the formation server's unauthenticated TLS is annotated as a known gap.
* docs: record spx command tree move
* chore: accept known vulnerabilities with an expiry
The vulnerability scan failed preflight on a containerd advisory that dev
already carries and that is deferred to a separate dependency update. The
gate now passes only reachable findings listed with an expiry date, fails
on any new one or on an expired acceptance, and warns when an acceptance
is no longer needed.
* refactor: move EC2 account action adapters to domains/ec2/awsapi
* refactor: move EC2 capacityreservation action adapters to domains/ec2/awsapi
* refactor: move EC2 key action adapters to domains/ec2/awsapi
* refactor: move EC2 launchtemplate action adapters to domains/ec2/awsapi
* refactor: move EC2 placementgroup action adapters to domains/ec2/awsapi
* docs: record EC2 standalone action adapter moves
* refactor: move EC2 zone action adapters to domains/ec2/awsapi
* refactor: move EC2 vpc action adapters to domains/ec2/awsapi
* refactor: move EC2 eigw action adapters to domains/ec2/awsapi
* refactor: move EC2 eip action adapters to domains/ec2/awsapi
* refactor: move EC2 igw action adapters to domains/ec2/awsapi
* refactor: move EC2 natgw action adapters to domains/ec2/awsapi
* refactor: move EC2 routetable action adapters to domains/ec2/awsapi
* docs: record EC2 VPC networking action adapter moves
* refactor: move EC2 image action adapters to domains/ec2/awsapi
* refactor: move EC2 snapshot action adapters to domains/ec2/awsapi
* refactor: move EC2 volume action adapters to domains/ec2/awsapi
* refactor: move EC2 tags action adapters to domains/ec2/awsapi
* docs: record EC2 storage and tag action adapter moves
* refactor: move EC2 resource authorization scopes to domains/ec2/awsapi
* refactor: move EC2 idempotency adapter to domains/ec2/awsapi
* refactor: move EC2 instance action adapters to domains/ec2/awsapi
* refactor: move EC2 spotinstance action adapters to domains/ec2/awsapi
* docs: record EC2 instance action adapter moves
* refactor: move AWS REST-JSON router to ingress/aws/rest
* refactor: move SigV4 auth-failure rate limiter to ingress/aws/ratelimit
* docs: record AWS ingress helper moves and tooling conformance
* docs: correct historical lifecycle references
* test: characterize ECR control-plane dispatch before registration seam
* feat: add AWS ingress service registration contract
* docs: record ingress registration phases 0 and 1
* refactor: route registered AWS services through the ingress dispatch registry
The gateway now consumes ingress/aws/dispatch. A registered service is
accepted by the auth gate, resolves its own action label, renders errors in
its declared envelope and is dispatched with an authenticated Invocation
(account, region, policy authorisation and a lazy verified actor). Every
other service keeps the existing switch, now named as the legacy path.
The awsgw role wires an empty registry and refuses to start if a registered
name is also served by the legacy table. No service is registered yet, so
behaviour for every existing service is unchanged.
* docs: record ingress dispatch registry consumption
* refactor: register the ECR control plane from its domain
domains/ecr/awsapi now registers the "ecr" AWS JSON 1.1 surface with the
ingress dispatch registry: action resolution, resource scope, authorisation
through the invocation, the composed capabilities and the success writer all
live with the domain. The awsgw role is the only serving composition point;
the gateway no longer imports the ECR API package and ECR leaves the legacy
service table, JSON error table and dispatch switch. The OCI /v2 route is
unchanged.
Operation coverage reads ECR's declared inventory, which the registration
uses verbatim; coverage.json is byte-identical.
* docs: record ECR control-plane registration
* fix: compose the ECR control plane in the integration harness
The harness stopped wiring ECR's composed capabilities when the
authorization-token action was composed, so every integration test that
creates a repository or mints a token got ServerInternal. It now composes
ECR as the awsgw role does, sharing the OCI registry, meta store and token
issuer the /v2 data plane uses.
* docs: record ingress pipeline deferral
* refactor: extract AWS error envelope rendering to ingress
ingress/aws/envelope owns the EC2, IAM-style and S3 XML error envelopes, the
AWS JSON 1.1 error body and X-Amzn-Errortype value, the JSON-protocol header
detector and the two error content types. Each shape keeps its own builder.
The gateway error writers, ErrorHandler and the legacy per-service envelope
selection stay in gateway and call the new package; gateway/eks renders its
JSON errors through it directly. A 140-case status/header/body golden
comparison across services and error writers was identical before and after.
* docs: record error envelope extraction
* refactor: move AWS service selection into ingress dispatch
Selector answers served, action, error protocol, registered entry and
startup overlap. The gateway supplies its legacy tables through the
dispatch.Legacy interface; legacy dispatch and invocation building stay
in gateway.
* docs: record ingress service selection move
* test: characterize ACM control-plane dispatch before registration
Pins full-pipeline error envelopes, a successful action, target parsing
agreement between dispatch and telemetry, authorization before the NATS
call, 413 and 503 paths, throttle/trace/audit action and the inventory.
* refactor: register ACM control plane from its domain
domains/acm/awsapi registers the acm JSON 1.1 surface through the
ingress dispatch registry, wired by the awsgw role. The gateway drops
ACM_Request, the ACM action table and every legacy acm entry; the
import invariant now covers each registered domain.
* docs: record ACM registration
* docs: record ECS, EKS and RDS source inventory and first-resource ranking
* docs: record EKS control-plane secrets persisted in the cluster record
* chore: drop containerd advisory acceptance now that dev ships v2.4.1
* test: characterize RDS DB subnet group lifecycle before extraction
* docs: record RDS DB subnet group lifecycle contract
* refactor: extract the RDS DB subnet group lifecycle owner
domains/rds/subnetgroup owns the db-subnet-groups/{name} record, member
subnet validation and the delete in-use guard. handlers/rds keeps AWS
validation, projection and paging and binds two consumer-owned
capabilities: the owner's Dependants (instances naming the group) and the
instance side's subnetGroupPlacement. Stored shape and behaviour are
unchanged.
* docs: record RDS DB subnet group owner extraction
* docs: correct RDS subnet group tagging debt and conformance scope
* test: characterize ECS task definition lifecycle before extraction
* docs: record ECS task definition lifecycle contract
* refactor: extract ECS task definition owner into domains/ecs/taskdefinition
Move the task definition record, container types, keys, reference parsing,
ARN helpers and the register/deregister/resolve/list logic into a resource
owner package. Stored JSON field names and KV keys are unchanged, and
revision allocation keeps its existing non-atomic read-then-write.
The owner declares its own PortMapping and SystemControl persisted types so
it no longer depends on handlers/ecs/bus; the handler translates them onto
the assign payload. Task and service launches reach task definitions only
through a launch-side taskDefinitionResolver. AWS conversion, the guest
projection and tags stay in handlers/ecs, and gateway authz only switches to
the moved reference/ARN helpers.
List handlers/ecs and the new package under spinifex_daemon so ECS-only
changes select e2e-ecs.
* docs: record ECS task definition owner extraction
* test: characterize EKS access entry lifecycle before extraction
* docs: record EKS access entry lifecycle contract
* refactor: extract EKS access entry owner into domains/eks/access
Move the access entry record, associated policy and scope types, keys,
PrincipalARNHash, the policy catalogue and group mapping into a resource
owner package. Stored JSON field names and KV keys are unchanged.
Owner is the only writer: Create, SeedCreatorAdmin, Update, Associate,
Disassociate and Delete carry the existing logic verbatim, and the put,
compare-and-set and delete primitives are private. The handler keeps AWS
validation, the cluster existence check, error mapping and projection.
Token review still reads records through the exported Get, and the cluster
prefix purge still sweeps entry keys.
List handlers/eks and the new package under spinifex_daemon so EKS-only
changes select the EKS suites.
* docs: record EKS access entry owner extraction
* test: characterize RDS engine catalogue before extraction
* fix: return InvalidParameterValue for a missing RDS master password
ValidateMasterUserPassword built the empty-password error with a bare
errors.New carrying the code only in its text, which awserrors does not
resolve, so CreateDBInstance without MasterUserPassword returned HTTP 500
ServerInternal. Build it with awserrors.Errorf like the other branches so
the gateway returns 400 InvalidParameterValue, and cover the full gateway
to RDS service path with a regression test.
* docs: record RDS engine catalogue boundary and evidence
* refactor: extract RDS engine catalogue into domains/rds/engine
Move engine identity and validation, the per-engine parameter catalogues,
effective-parameter resolution, db.* instance classes and engine/orderable
selection into an engine package. Behaviour, error codes and ordering are
unchanged.
The package imports no EC2 package: instance-class memory comes from an
immutable Sizing value built from an engine-declared InstanceTypes
interface, backed by the fixed instance-type table in handlers/rds.
Effective parameters leave as an engine-owned Setting, converted to
Parameter in handlers/rds. AWS output building, paging, apply-method
mapping, storage limits and the gateway runnable check stay in the
adapter path.
The RDS agent declares its own EngineCatalog interface and types and
receives an engine-backed implementation from cmd/rds-agent, so agent
code does not import the catalogue. Cross-engine tests use the two real
engines instead of a registered fake.
* docs: record RDS engine catalogue owner move
* test: characterize RDS DB parameter group lifecycle before extraction
* docs: record RDS DB parameter group lifecycle contract
* refactor: extract RDS DB parameter group owner
Move the parameter group and override records, keys and listing into
domains/rds/parametergroup behind an Owner. Delete asks an owner-declared
Dependants interface; instance-side resolution reads through its own
parameterGroupValues interface. SetOverrides refuses an unknown group so
no override key is written without its group. Stored shape, keys, public
errors and propagation are unchanged.
* docs: record RDS DB parameter group owner extraction
* test: characterize EKS add-on lifecycle and guest wire
Pin the persisted record and manifest layout, create/update/delete and
status-report transitions, the ignored client token and report version,
the missing serviceAccountRoleArn PassRole check, and the exact internal
add-ons HTTP body and agent TSV output as they stand today.
* docs: record EKS add-on lifecycle contract
* fix: require iam:PassRole for EKS add-on service account role
CreateAddon and UpdateAddon now pass serviceAccountRoleArn through the
existing EKS PassedRoleARNs path, so the caller needs iam:PassRole on the
role for eks.amazonaws.com, matching CreateCluster and CreateNodegroup.
* docs: record EKS add-on PassRole check
* refactor: extract EKS add-on owner
Move the add-on catalogue, record and manifest keys, CAS, status
transitions and staging into domains/eks/addon behind an Owner that is
the sole writer. The handler keeps AWS validation, error mapping and
projection; the cluster reconciler delegates reports through the narrow
addonReports interface; node groups call EnsureGPUDevicePlugin. Stored
keys, JSON shape, wire types and lifecycle behaviour are unchanged.
* docs: record EKS add-on owner extraction
* refactor: extract EKS add-on guest wire contract
Move the add-on status subject, delivery phase, status report, staged
manifest and internal add-ons response into contracts/eks/v1 and use it
from the handler, gateway and gatewayfetch, replacing the agent's
duplicate. Wire bytes are unchanged, including the internal add-ons body
that the SDK renderer emits with Go field names; compatibility tests pin
the JSON, the guest printf form and the TSV.
* docs: record EKS add-on wire contract extraction
* docs: update EKS add-on inventory and remaining edges
* docs: draft EKS add-on lifecycle correction design for review
* chore: run contracts tests in the default test, coverage and race targets
* docs: apply review decisions to EKS add-on lifecycle design
* test: characterize EKS add-on v1 compatibility with added fields
Pin which owner writes drop unknown record and manifest fields, that
reads and deletes ignore or erase them, that one undecodable entry fails
a listing, and that v1 wire consumers ignore extra fields.
* docs: add EKS add-on executor, bundle, v1 compatibility and release inventories
* docs: draft EKS add-on v2 guest contract design for review
* docs: qualify historical architecture references
* docs: apply review decisions and ADR-0006 to EKS add-on v2 design
* fix(eks): bind internal report publishing to the caller's cluster
PublishInternal was authorised by IAM policy alone, with its resource
built from the body accountId and the URL cluster. The control-plane
role grants it on "*", so any control-plane VM session, any tenant
holding eks:*, or static system credentials could publish cluster
state, bootstrap and add-on reports for another cluster or account.
PublishInternal now passes the existing internal-route gate: the caller
must be a control-plane instance-role session that is a member of the
URL cluster in the body account. The gate and the handler decode the
body through one shared decoder, so the account checked is the account
the subject is built from. Every unauthorised target returns the same
AccessDenied, and invalid account or cluster names are treated as not
found. Valid publishes keep their subjects and payload bytes.
* fix(eks): bind webhook token review to the caller's cluster
WebhookTokenReview took its account from the request body and was
authorised by IAM policy alone, with the control-plane role granting it
on "*". A control-plane VM, any tenant holding eks:*, or static system
credentials could resolve a token against another cluster or account
and learn the Kubernetes groups it maps to there. Error responses also
distinguished absent accounts from absent clusters.
WebhookTokenReview now passes the same internal-route gate as
PublishInternal: a control-plane instance-role session that is a member
of the URL cluster in the body account. The gate and the handler share
one body decoder, each body-scoped internal route must declare its
reader, and every unauthorised target returns the same AccessDenied
without reaching STS verification. Same-cluster responses are unchanged.
* fix(eks): validate add-on serviceAccountRoleArn syntax
CreateAddon and UpdateAddon stored serviceAccountRoleArn without
checking it was an ARN, and the guest renders the value into the
add-on's manifests. A non-empty value must now be syntactically an IAM
role ARN (12-digit account, no region, role resource, IAM name and path
rules); anything else returns InvalidParameterException before any
record is written or manifest staged.
The syntax check is foundation/aws/arn.ValidateRoleARN. iam:PassRole at
the gateway remains the permission and existence check. Stored values
are not rewritten, and an UpdateAddon that omits the field still
succeeds on a record holding a legacy value.
* docs: record EKS internal-route and role-ARN security corrections
* docs: add EKS add-on release membership and activation case study
* fix: point UI build, CI and lint rules at moved package paths
* docs: record EKS add-on activation review decisions
* refactor: move quota admission to domains/admission/quota
Relocate the quota service, live counters and vCPU records from
handlers/quota to domains/admission/quota. Package renamed from
handlers_quota to quota; callers with a local quota identifier import it
as admissionquota. KV buckets, subjects and error codes are unchanged, and
the package's existing debt edges move with it unchanged.
* docs: reconcile package-boundary record for ADR-0001 close-out
Add the missing slice rows (the commits that dissolved spinifex/types, the
ECR repository projection, the stale-path fix and the quota move), correct
stale entries, record the dependency-rule breaks found in the import graph
with removal conditions, add the close-out table of remaining legacy paths
with their decided target owners, recount the mulga-side follow-ups, and
fix stale old-path references in docs and comments.
* test: move KV migration engine tests to foundation/state/migrate
The Registry and RunKV tests exercise foundation/state/migrate exclusively
but lived in spinifex/migrate's test file. Relocate them, with their NATS
helpers, to the package that owns the engine. No test name, assertion or
behaviour changes.
* refactor: move config and object-store upgrade workflow to operator/upgrade
Relocate spinifex/migrate, the operator-driven config-file and object-store
upgrade workflow behind spx admin upgrade, to operator/upgrade. Package
renamed to upgrade (imported as opupgrade where a local upgrade identifier
exists). It stays distinct from foundation/state/migrate, the generic KV
migration engine; no registry types, migration names, versions, file
formats or command names change.
* docs: record migrate move to operator/upgrade
* test: enforce ADR-0001 layering with a blocking ratchet
Add spinifex/architecture/layering, a test-only package that parses the
module's import graph, maps every production package to an ADR-0001
layer and checks each dependency rule as its own subtest. Existing
violations are allow-listed against their recorded debt; a new forbidden
edge, an unmapped package, a stale allowlist entry or a new package
under a frozen legacy root fails local preflight and CI.
Record the nine edges the test found missing from the migration record
(quota -> EC2 instance types, STS/Ochre/ELBv2 -> IAM, ELBv2
cross-domain, installer UI -> admin), name the importers of the utils
residuals and gateway/ec2_idempotency.go, and correct the handlers/iam
consumer list.
* refactor: drop the legacy handlers_ecr import alias
Import domains/ecr under its package name, or as ecrdomain where the
file also imports the AWS SDK ecr package, matching acmdomain. Also fix
a stale daemon file reference in the integration ECR daemon comment.
* chore: mark spf13/pflag as a direct dependency
operator/cli/wiring.go imports pflag directly. Same version; no upgrade.
* docs: record test-package-check and nilaway as unverified gates
* refactor: move the layering test to module-root architecture/
The ADR-0001 layering ratchet is repository governance tooling, not
Spinifex product code, so it sits beside tests/ rather than under the
spinifex/ product tree. make test, test-cover and test-race now include
./architecture/..., and the graph walk excludes architecture/.
* fix: update paths left stale by the package moves
The MariaDB image stamp test read the RDS engine from
handlers/rds/engine_mariadb.go, so test-images failed in CI; it now reads
domains/rds/engine/mariadb.go. Comments citing moved files now name
their new locations.
* refactor: move the default-VPC subject into contracts/ec2/v1
utils.SubjectEnsureDefaultVpc becomes ec2v1.EnsureDefaultVpcSubject, the
request/reply route between gateway account creation and the daemon.
Only the subject moves; the request and reply bodies are still anonymous
structs on each side and are recorded as such in the contract README.
The layering freeze now allows two utils files.
* docs: qualify the migrate split and record admin and Viperblock findings
The migrate row is a split, not an exact ADR-0001 map row, and needs a map
clarification. Correct the stale admin consumers, record the operator-only
admin files as movable, the vbwire prerequisite for the Viperblock adapter
and the capability decision encryption.go needs.
* refactor: move operator-only admin code to operator/admin
Config generation, certificate issuance, host discovery, telemetry,
email validation and image import/removal/orphan-volume tooling move
to operator/admin. Legacy admin keeps only the symbols with
non-operator consumers: default account identifiers, AWS credential
generators and system image promotion.
foundation/pki and the tenant ACM CA are unchanged. The readAMI
helper and the removal test fixtures are duplicated privately in
legacy admin rather than shared across the split. The now-stale
operator/installer/ui -> admin layering allowlist entry is removed.
* docs: record operator/admin split in package-boundary migration
* test: pin viperblockd fence, abandon and dirty-marker wire shapes
Characterization tests against the current vbwire package, before it
moves to contracts/viperblockd/legacy/v1: both subject forms for the
fenced and abandon routes, golden JSON for VolumeFencedEvent,
VolumeAbandonRequest, VolumeAbandonResponse and DirtyRecord, the dirty
bucket name, and the volume-key pattern shared by the lease and dirty
stores.
* refactor: move viperblockd fence, abandon and dirty-marker vocabulary to contracts
Move the fence and abandon subjects/DTOs, the dirty bucket name,
DirtyRecord and the shared volume-key pattern out of vbwire and into
contracts/viperblockd/legacy/v1, alongside the package's existing
mount/unmount/delete/sync/config-update routes. vbwire keeps only the
I/O helpers (BindDirtyBucket, ListUnsealedVolumes), now built on the
contract vocabulary instead of defining it.
Record the approved dirty-marker ownership boundary in the contract
README: the Viperblock adapter is the sole producer, viperblockd was
only its transitional custodian. Document the fence and abandon routes
and the dirty bucket in docs/service-interfaces.yaml.
* refactor: split the Viperblock EBS adapter out of services/viperblockd
Move the EBS provider adapter (volume lease, fence, dirty marker,
nbdkit recovery, provider handlers, encryption key loading) to
providers/ebs/viperblock. services/viperblockd keeps only process
lifecycle and the NATS subscription wiring that starts and stops it.
The dirty-bucket read path moves to providers/ebs/viperblock/dirty as
a read-only ListUnsealedVolumes capability; the adapter's write, clear
and purge logic stays unexported in the adapter package. Delete
services/viperblockd/vbwire now that its vocabulary lives in
contracts/viperblockd/legacy/v1 and its I/O helpers live in dirty.
Update operator/cli, the runtime service registry, qemunbdd's config
comment and the integration test harness to the new import paths, and
drop the now-stale architecture/layering allowlist and frozen-package
entries for the deleted package. Behaviour, persisted keys and NATS
subjects are unchanged.
* refactor: move the viperblockd role to runtime/roles/viperblockd
With the EBS adapter in providers/ebs/viperblock, services/viperblockd
holds only role supervision and the opt-in live data-path tests, so it
joins the other runtime roles. It was the last package under services/;
the frozen legacy list drops it.
* docs: record viperblockd contract, adapter and role moves
* test: characterise vpc.* network-realisation routes
Pin every vpc.* subject, JSON payload shape and the request/reply
{success,error} ack envelope exactly as they exist today, ahead of
introducing contracts/network/v1. Covers the 22 subjects registered
by domains/network/subscribers, the vpc.igw-attach/detach subjects
and InternetGatewayEvent already owned by contracts/ec2/v1, the
utils.AddNAT/PublishNATEvent client wire behaviour including the
delete-nat retry-on-ErrNoResponders budget, and the EC2-side NATEvent
and SGEvent/SGRule duplicates in domains/ec2/vpc.
Also adds the six vpc.* route-table and system-egress subjects that
were missing from service-interfaces.yaml despite being live
production routes (vpc.add/delete-igw-route, vpc.gate/ungate-subnet-
egress, vpc.add/delete-system-egress).
* feat: add contracts/network/v1 for vpc.* realisation routes
Declare the versioned wire vocabulary for the vpc.* control-plane-to-
network-realisation boundary: subjects, payload types, the shared
{success,error} ack envelope and the vpcd-workers queue group, one
file per route family. No NATS I/O, no consumer switched over yet;
the package coexists with the fragmented originals pinned in the
prior characterization commit until slice 3 migrates call sites.
* refactor: use contracts/network/v1 for vpc.* producers and consumers
Switch every vpc.* producer (domains/ec2/vpc, eip, instance, natgw,
routetable, igw, handlers/elbv2, handlers/eks, daemon, systemvpc) and the
network/subscribers consumer onto contracts/network/v1 subjects, event
types and the shared AckEnvelope. contracts/network/v1 becomes the sole
owner of vpc.igw-attach/detach, replacing the duplicate definitions in
contracts/ec2/v1.
Deletes the duplicated local subject constants and event structs:
subscribers' Topic* consts and event structs, vpc.NATEvent and SGEvent,
eip's natEvent, natgw's natGatewayEvent, and eks's systemEgressEvent.
domains/ec2/vpc.SGRule survives because EC2 persists it directly in
SecurityGroupRecord; contracts/network/v1.SecurityGroupRule was expanded
in slice 2 to match it field-for-field so wire bytes are unchanged.
No lifecycle or reliability changes: the NAT add barrier, delete
retry-on-ErrNoResponders, and the inconsistent per-call-site NAT
delivery modes (EIP AssociateAddress and DeleteNetworkInterface both stay
fire-and-forget with no ack) are preserved exactly, as are the golden
wire literals pinned in the characterization tests.
* refactor: replace utils/vpcd_event.go with a typed network projection client
Every vpc.* call site now goes through domains/network/projection's
operation-specific methods instead of utils.AddNAT, utils.PublishNATEvent
or utils.RequestEvent, so no caller passes a bare subject string. Each
method states today's delivery mode by name: AddNAT stays strict,
AddNATBestEffort and AnnounceNATAdd/AnnounceNATDelete keep the gaps those
call sites already had, and RemoveNAT keeps the ErrNoResponders retry.
utils/vpcd_event.go and its tests are deleted; the behaviour they pinned
(wire shape, ack envelope, retry-on-no-responders) moved into the
projection package's own golden and nat tests, unchanged. utils/ now
holds only encryption.go, which this does not touch.
* refactor: route the remaining fire-and-forget vpc.* producers through projection
CreateVPC/DeleteVPC, CreateSubnet/DeleteSubnet, AttachInternetGateway/
DetachInternetGateway, AddNATGateway/RemoveNATGateway, AddIGWRoute/
RemoveIGWRoute, GateSubnetEgress/UngateSubnetEgress and RemoveSystemEgress
were still publishing through natsmsg.PublishEvent or a raw nc.Publish
call with a bare subject string, left over from the mechanical slice 3
pass. Each now goes through its named projection.Client method instead,
so every vpc.* producer in the tree uses the typed client.
* refactor: move EBS encryption posture to domains/ec2/ebs/encryption and delete utils
The master-key loader was used only by EC2 image, instance and volume to
derive the AWS-visible Encrypted flag. It becomes ebsencryption.Enabled,
EC2's temporary reading of the provider's key-file setting; the cache,
empty-path meaning, error text and each caller's error handling are
unchanged (image still reports unencrypted when the key will not load).
Its successor is a providers/ebs capability reporting volume encryption
posture.
utils/ is now empty and deleted. The layering test drops utils and
services from the legacy roots, removes their allowlist entries and fails
if either retired root regains files. The S7 subject scan follows the
vpc.* literals from utils to contracts/network so its gap list stays
complete.
* docs: record the network contract, projection client and utils removal
* test: characterise instance-state persistence, recovery and generation handling
Pin the current local-file bytes and schema failure modes, the instance
bucket configuration, record key and wire form, the bucket version gate,
generation preservation across every KV writer and the VM conversion, and
what a returning node does for a missing, foreign, terminal or unreadable
canonical record. These describe current behaviour, not the ADR-0007 target.
* test: freeze direct instance-state consumers
Add an AST ratchet over production code that fails when a new file names
the daemon's instance-state bucket, record keys, local-file API, the raw
vm.InstanceRecord or its conversion, or a JetStreamManager instance-state
method. Today's 73 uses form a shrink-only allowlist keyed to the slice 0
inventory; stale entries fail, and a synthetic-source self-test proves the
matcher detects each kind of use and ignores declarations.
* docs: record ADR-0007 slice 0 instance-state inventory
Record the instance-state persistence formats, the nineteen direct
consumers by process and access kind, the characterised recovery and
schema outcomes, the generation-preservation matrix and the ratchet that
freezes the consumer set. Nothing conforms to ADR-0007 yet and vpcd stays
blocked.
* test: narrow the instance-state ratchet to instance-state access
Stop freezing the JetStreamManager type and constructor, which also serve
cluster state; flag only instance-state methods, bucket and key identifiers,
record codecs and local-state access. The cluster-state-only consumers in
daemon_mgmt_ip.go and operator/cli/cluster.go leave the allowlist (73 to 67).
The self-test now proves a cluster-state use is ignored and an instance-state
method call is caught. Record the owner's disposition of the KV-unavailable
restore result in the migration record.
* docs: record the narrowed instance-state ratchet commit
* docs: design the instance-state transition record and activation gate
* docs: record the instance-state transition-record design commit
* docs: record slice 1a activation-gate decisions
* feat: add instance transition-record codec with format marker
Adds DecodeTransitionRecord and TransitionRecord.Encode in the record's owner
package. A top-level format member selects the shape: absent or 1 decodes the
legacy record and is never given a marker on write; 2 also decodes the new
assignment and observation sections; an integer above 2 decodes its legacy
fields read-only and Encode refuses it; anything else is refused on decode.
Encode applies only the typed paths that differ from the decode-time snapshot
onto the input, so unknown members at every object level and unchanged known
fields keep their read form. A changed array is replaced whole. Decode refuses
malformed JSON, duplicate keys at any depth, keys that alias a known field by
case, trailing data and a non-object record, each with its own sentinel.
No production reader or writer uses the codec yet; the new types are not
part of vm.InstanceRecord, so every existing record encodes unchanged. The
codec file is registered as an owner file of the instance-state ratchet.
* docs: record slice 1b step 1 codec and awsgw rollout qualification
Records the codec added in e62f921d2 as design Section 10 (C1 to C7): its
surface, preservation at every object level, the changed-array limitation,
marker handling, the rejection classes and its capable-writers-only scope. C2
supersedes P1's top-level-only preservation for the codec. G19 records that
old read-only gateway consumers remain safe but DescribeInstanceStatus gains
unconfirmed/degraded visibility only after awsgw upgrades. The migration record
gains the commit row and states that no production path uses the codec yet.
* refactor: move the instance transition-record codec to domains/ec2/instance/state
The codec defines how the canonical persisted instance record evolves, with
its assignment and observation sections, which is EC2 instance ownership rather
than node-local VM supervision. It moves unchanged into its own package and
keeps wrapping the legacy vm.InstanceRecord alias and its spec and status
types, which stay in runtime/compute/vm for now.
The domain-to-runtime import is recorded as new dependency-rule debt with its
removal condition and allow-listed against that entry. The instance-state
ratchet now names the new file as an owner file; its allowlist is unchanged.
* fix: refuse array changes that would drop element extensions
A changed array is replaced whole by the writer's value, which silently lost
any member a newer writer had added inside its elements. Encode now inspects
the array as read: if any element carries a member its typed schema does not
know, at any depth, it returns ErrRecordExtensionConflict naming the array and
the first such member. Arrays without such members are still replaced whole,
unchanged arrays are still kept verbatim, and arrays inside unknown members are
never inspected.
* docs: record the codec move and the array extension-conflict rule
Records 8db9771fb and c617914a1. T1 and Section 10 now name
domains/ec2/instance/state as the eventual owner of the persisted instance
record and its evolution, with vm.InstanceRecord as the temporary legacy type
it wraps. C3 replaces the changed-array limitation with the refusal: a changed
array with element extensions fails with ErrRecordExtensionConflict and is never
silently dropped. P2 stays a conservative rule until finalisation.
* fix: refuse removals that would drop record extension fields
A capable writer may no longer remove or replace a known object, map entry or
array when its stored value carries members the writer's schema does not know.
The check that guarded changed arrays now runs wherever the merge drops or
overwrites a stored value instead of merging into it: a known member deleted or
set to null, a map entry removed or stored under a key the typed decode
rewrote, a value of another JSON kind, and a changed array. Encode returns
ErrRecordExtensionConflict naming the changed container and the first unknown
member. Removal stays allowed when the stored value has no extensions, and a
change merged member by member still keeps unknown members.
* docs: record the uniform extension-conflict rule
Adds design clause C8 for 0d7b4266d: a capable writer may not remove or
replace a known object, map entry or array whose stored value carries unknown
structured extension fields; removal without extensions stays allowed. C3 and
C5 point at it. The migration record gains the commit row, updated subsection
wording and a pointer-bump follow-up deferring make docs-portal to the Mulga
gitlink PR so the portal evaluates the proposed Spinifex revision.
* refactor: inject vpcd's IMDS instance-state readers from the instance state boundary
* docs: cite the vpcd reader injection commit in the migration record
* refactor: move vpcd to runtime/roles/vpcd
* docs: cite the vpcd move commit in the migration record
* refactor: move the ECS agent wire vocabulary to contracts/ecs/v1
Move handlers/ecs/bus to contracts/ecs/v1 (package ecsv1) with no legacy
shim. Subject strings, JSON fields and types are byte-identical; the old-agent
compatibility test moves unchanged and golden pins cover the remaining
message types. The README route table records that only service-reconcile
has a live producer today; the agent reuses these payload types through the
gateway actions. service-interfaces.yaml lists ecs.bus.* under the daemon.
* refactor: move system-image promotion to domains/ec2/image
Move images_promote.go and its tests from admin to domains/ec2/image:
SystemOwnerAlias, PromoteImageOpts/PromoteImageResult, PromoteSystemImage,
GetAMIMetadata and the private snapshot reader. Storage, ordering, errors and
promotion behaviour are unchanged; the daemon, gateway/spx and operator CLI
callers import ec2image. admin now holds only account IDs and key generators.
* docs: record the ECS contract and image-promotion moves and the remaining-roots matrix
* refactor: move the bootstrap account identifiers to foundation/aws/identifiers
DefaultAccountID and DefaultAccountName join GlobalAccountID as the fixed,
platform-reserved identity of the first human-facing bootstrap account.
Names and values are unchanged and stay fixed. Callers in daemon, gateway,
awsgw, accountteardown and the operator CLI use awsidentifiers; admin now
holds only the IAM credential generators.
* docs: cite the bootstrap account identifier move in the migration record
* refactor: inject the STS cluster JWKS lookup from handlers/eks
handlers/sts declares ClusterJWKSResolver and keeps JWKS decoding and
InvalidIdentityToken mapping; handlers/eks implements it as
OIDCJWKSReader beside the bucket and key layout helpers. awsgw and the
integration harness inject the reader. STS no longer imports any EKS
package, including tests, and the handlers/sts -> handlers/eks layering
exception is removed.
* docs: cite the STS JWKS resolver cut in the migration record
* refactor: move the AWS credential-format generators to foundation/aws/credentials and retire admin
GenerateAWSAccessKey and GenerateAWSSecretKey move unchanged to
foundation/aws/credentials as a stateless format primitive. handlers/iam
and operator/cli import it directly. The admin legacy root is deleted
and joins the retired roots so it cannot return.
* docs: cite the admin retirement in the migration record
* revert: remove the uncalled ADR-0007 codec and design stack from the ADR-0001 checkpoint
The instance transition-record codec, its tests and the slice 1a design
had no production caller. They leave this package-boundary checkpoint
and will be reapplied on a dedicated ADR-0007 branch. Slice 0
characterisation and the vpcd IMDS readers in
domains/ec2/instance/state stay.
* docs: cite the ADR-0007 codec revert in the migration record
* fix: declare the ECS action subjects in service-interfaces.yaml
Declaring ecs.bus.* made ecs a managed prefix for manifest-lint, which then
failed on the 34 ECS action subjects the daemon registers. Declare them as
ecs.*, the same wildcard convention as rds.*.
* chore: run manifest-check and manifest-lint in preflight
CI runs both, but preflight did not, so manifest drift could pass local
checks and still fail Build and Test.
* docs: record resumed preflight enforcement and dev merge ports
* fix: restore dev changes dropped by earlier merges
Reapplies the slices.ContainsFunc rewrite in nodeFitsCustomerInstance and the DeleteDBSubnetGroup doc comment, both lost when their files were refactored.
* docs: record accountteardown as a frozen legacy root pending cleanup capabilities
---------
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
The release workflow granted contents: write to every job, including the distro builds and the self-hosted verify job, which only read. Only release, promote, mark-failed and iso create or edit the GitHub release, so they alone get write and the rest default to read.
setup-go caches by default, and a release or dev prerelease run restores caches that any dev or main run saved. A poisoned module cache from one of those runs would then flow into the shipped binaries, so both release builds now start from a clean cache.
Account teardown's force path cleared a volume's attachment and deleted it
without proving the holding instance gone, and the delete guard judged its
instance view complete by counting replies, so one node answering twice
could stand in for the silent node hosting the guest.
The holder must now be terminated or definitively InvalidInstanceID.NotFound
on a describe naming it, confirmed on a listing every configured node
answered by identity. Anything else retains the volume, reports it stuck
naming the volume and instance, and keeps the account TERMINATING.
DeleteVolume's in-use guard takes the configured node set and uses the same
identity-checked listing, so public DeleteVolume also refuses while any
configured node is unanswered. That refusal keeps the listing's retryable
ServiceUnavailable and Retry-After; other guard failures stay InternalError.
Account teardown listed instances account-wide through a path that returns
whatever nodes answered. A silent or erroring node, a node answering twice,
or a failed instance-bucket query gave a short list with no error, which
teardown counted as drained.
Add DescribeAccountInstancesComplete, which requires a clean reply from
every configured node by node ID plus both instance buckets, and otherwise
returns a retryable ServiceUnavailable. Thread the configured node set into
the teardown engine from the admin API and the CLI. The engine already
aborts on a listing error, so the account stays TERMINATING and storage
teardown does not run.
The fake control plane returned the scripted 403 once, then failed every
retry with "returned no config", overwriting the failure the heartbeat
reports. The test only passed if a beat landed within one 1ms retry
interval. A persistent fake error keeps the 403 on every retry.
Set read, write and idle timeouts and a header cap on each per-tap IMDS
http.Server, and cap its listener at 32 connections, so one guest cannot
hold unbounded vpcd descriptors and goroutines with idle keep-alives.
The VPC networking guide had no mention of IPsec, so an operator reading it
had no way to discover that the geneve overlay is encrypted by default or
what turning that off is worth. Add a section covering the default-on
posture, the topologies where disabling is and is not appropriate, the
measured cost on the ConnectX-5 reference cluster, and the full procedure.
Correct the performance-tuning guide's claim that a manual
"ovn-nbctl set NB_Global . ipsec=false" is transient because the daemon
reconciles it. That holds for the enable direction only: ReconcileOVNIPSec
returns early when IPsec is disabled, so it stops requesting encryption but
never clears NB_Global.ipsec or other_config:ipsec_encapsulation. Both are
left reading true, which makes the control plane disagree with the config.
Document the manual steps as a required complement to the config change.
Adding a role to an instance profile, removing it, or deleting the profile
only refreshed the profile list, so the role page kept its cached profiles
for up to five minutes.
Deleting a route, disassociating a subnet or Elastic IP, revoking a
security-group rule, detaching an internet gateway and deleting a launch
template version each ran on one click. They now open a confirmation
dialog, and a failed launch template version delete is reported instead of
silently leaving the row.
A failed RunInstances or ingress authorize left launch-wizard-N behind, so
the retry, which reuses the same default name, failed with
InvalidGroup.Duplicate. The security group list now also refreshes after a
failed launch.
The wizard validated only custom CIDRs, so a default plan that overran a small
VPC, or a custom CIDR overlapping a default, passed the form and left a
half-built VPC. Validation, the mutation, the preview and the CLI panel now
share one resolved CIDR list, and subnet errors are shown.
Subnets are created with allSettled so a failure still reports every subnet
that was made, and the calculator packs private subnets after the public ones
when the VPC midpoint leaves no room.
The custom CIDR used isValidCidr's VPC defaults of /16 to /28, so a single-IP
source such as 203.0.113.5/32, 10.0.0.0/8 and 0.0.0.0/0 were all rejected.
AWS accepts any prefix for a security-group rule source.
Rebuilds frontend/dist, which also picks up the signature-error logout fix.
SignatureDoesNotMatch and IncompleteSignature no longer clear the session:
clock skew and path rewrites also produce them, so the user was logged out
repeatedly over a request problem.
Accept read the protocol-detection byte inline with a 5s deadline, so each
client that sent nothing stalled every new visitor. Accept now returns the
TLS-wrapped conn immediately and the first byte is sniffed on the first
handshake Read, in the connection's own goroutine.
Service limits are per tenancy. vm-single-principal now builds in its own, so
the shape the other topologies share can have zero quota there, and the apply
then fails before anything about the instance principal is exercised.
Measured in ours: standard-e6-core-count is 0 available, against 13 for e5 and
10 for standard3. The first principal run died on 400-LimitExceeded for Core
Instance, which reads as a capacity problem and is a shape with no quota.
OCI_PRINCIPAL_SHAPE overrides it for that topology alone and defaults to the
shape every other topology uses, so nothing changes where quota already exists.
* fix: default the OCI helper to the DEFAULT profile
oci_env.py shipped our own internal Oracle compartment profile as the default for
every reader, with a second internal name as a fallback.
A fresh tenancy's config holds only `[DEFAULT]`, so the shipped default was a name
nobody else can have, and the fallback chain could land a deployment in whichever
tenancy a different profile happened to point at.
The profile is now `DEFAULT`, overridable by `OCI_CLI_PROFILE` or `--profile`, and a
named profile the config does not hold is an error rather than a fallback.
`DEFAULT` needs its own existence test, because configparser treats it as the
fallback for every other section and `has_section` denies it exists even when
`oci setup config` wrote nothing else.
`eip-move-test.sh` and the config table in the README carried the same name.
CI reads its credential from the environment rather than from a profile, so nothing
there referenced it.
* docs: look the OCI compartment OCID up by name
The guide asked the reader to paste `ocid1.compartment.oc1..xxx` three times, and
the `oci` CLI it already requires can resolve the name instead.
The lookup needs its own check. `oci iam compartment list --query ... | [0]` exits 0
and prints nothing when the name matches nothing, so an unset `$CID` would otherwise
reach Terraform and fail with an OCI error naming neither the compartment nor the
cause.
Two compartments in one tenancy can share a name, which the listing command shows
and the lookup cannot resolve, so the paste form stays as the answer for that case.
Records why the tenancy root is the wrong value: the step 2.1 policy is scoped to
whatever `$CID` holds.
* Revert "docs: look the OCI compartment OCID up by name"
This reverts commit 5443149cce.
* Reapply "docs: look the OCI compartment OCID up by name"
This reverts commit 1535d94c8a.
* fix: stop validate-topology.sh failing on macOS bash
`tf_principal_var` is empty on every run that does not pass `--instance-principal`,
and bash before 4.4 treats `"${arr[@]}"` on an empty array as an unbound variable
under `set -u`.
macOS ships bash 3.2, so the documented deployment command aborted at the tf_vars
assembly with `tf_principal_var[@]: unbound variable` before building anything.
Assigning the array was not enough, which is why the earlier `HOSTS=()` fix did not
cover this: the error is in the expansion, not the reference.
Both sites now expand only behind a length test, which is valid on every version.
`capture_journals` had the same shape for `HOSTS`, reachable when a run fails before
the apply.
Reproduced and fixed against bash 3.2.57 in a container, which emits the identical
message, and the script now runs to the terraform call on 3.2 for vm-single,
vm-single-principal and `--instance-principal`, including the `--keep` cleanup path
with no hosts.
* docs: record the bash 3.x requirement in the OCI scripts
These three are the scripts the OCI guide tells an operator to run, and macOS ships
bash 3.2, so a bash 4 construct in any of them breaks a deployment rather than a
test.
The constraint has now been rediscovered twice, through declare -A and mapfile, and
then through an empty array expanded under set -u.
Names the three traps at the top of each file so the next change does not have to
find them by failing on someone's Mac.
* fix: stream the workbook suite instead of hiding it in a file
The nine workbooks ran with stdout redirected to `workbooks.log`, so a run wedged
inside one printed nothing until the whole suite finished.
A CI job then sat silent for hours with no way to tell which workbook it was in, or
whether it was working at all.
The output is now teed, so the file the per-workbook table is parsed from is
unchanged while the run is visible live.
Each of the driver's own RUN/PASS/FAIL lines is stamped with elapsed seconds and the
workbook name, which is what identifies the slow one without reading the log.
`pipefail` carries the ssh exit status through the pipe, so a failed suite still
takes the branch that records the table and captures the journals.
* fix: grant the OCI policy the subnet verb AssignPrivateIP requires
An OCI node with a pool-mode external interface formed, reported healthy, and
then refused every launch wanting a public address with a 404
NotAuthorizedOrNotFound from AssignPrivateIP.
AssignPrivateIP calls CreatePrivateIp, which Oracle documents as needing
PRIVATE_IP_CREATE, PRIVATE_IP_ASSIGN, VNIC_ASSIGN and SUBNET_ATTACH, and
DeletePrivateIp as needing SUBNET_DETACH.
The policy granted use vnics, manage private-ips and manage public-ips, and no
subnet permission at all, so SUBNET_ATTACH was never held.
Only the subnets use verb carries it.
Measured on a live node as the instance principal: GetVnic and ListPrivateIps
succeed, GetSubnet and CreatePrivateIp both 404.
Authentication, the dynamic group's matching rule and the compartment were all
correct, so the credential was authentic and unauthorised.
The documented operator policy for the API-key route carried the same three
statements, so neither credential route could allocate.
A node now proves its own credential at pool start by making one read in each
permission class, and logs the missing verb if one fails.
The topology gate waited for "resolved the external VNIC", which comes from the
metadata service and needs no IAM policy, so an unauthorised node passed the
gate and failed an hour later inside a workbook as capacity.
It now waits for the authorisation line and fails immediately on a refusal.
Reads rather than a real assign-and-release round trip: a round trip also fails
on a full subnet, and reporting that as an authorisation fault sends an operator
to IAM for a capacity problem.
* docs: say what a principal run costs in a tenancy with no dynamic group
No principal topology can pass there, and the only previous hint was one clause
about OCI_INSTANCE_PRINCIPAL needing the group to exist.
* test: cover the subnet read and the authorisation verdict
Diff coverage was 39.1% against a 60% gate. Three of the four gaps were real.
GetSubnet on the SDK-backed client had no test, so the request shape was
unverified: addressing the collection rather than one subnet by OCID would
answer from a different permission and report a credential as working when it is
not. Also pins that a sparse body is not an error and that the unauthorised 404
arrives as ErrNotFound, which is what stops the check retrying a verdict that
will never change.
The fake's GetSubnet was exercised only from the ocinet tests, and coverage is
recorded per package under test, so it read as zero-covered.
reportAuthorisation now has a test asserting the exact log text, because
validate-topology.sh greps for it and a rename would quietly turn the gate into
one that never fires.
The six lines still uncovered are inside FromPoolConfig, which needs a live
JetStream to reach. Diff coverage 87.0%.
* fix: refuse a run whose tfvars file contradicts the compartment it was given
Terraform ranks *.auto.tfvars above TF_VAR_*, so a leftover file wins silently
over an operator targeting a different tenancy by environment, and the guide
tells every operator to write one.
Hit for real: a stale terraform.auto.tfvars naming a different tenancy's
compartment and a bare-metal shape overrode TF_VAR_compartment_ocid, and the
plan was for 11 resources in a compartment nobody had chosen. Only a dry run
caught it.
The run now stops and names both values rather than building.
* fix: collect orphaned OCI public IPs and name the limit that was hit
A reserved public IP attached to nothing is on no VNIC, so the reconcile
that exists to find leaked addresses could not see the ones that matter.
It also ran only at startup, while an address is leaked on release and on
teardown. Twenty accumulated in one compartment and exhausted the
tenancy's limit of 50, after which every launch needing a public address
aborted.
Reconcile now lists reserved public IPs in the compartment and collects
those that carry our prefix, are attached to nothing, are named by no
binding and are older than a grace period. All four are needed: an
allocated but unassociated Elastic IP is detached too and is the
customer's to keep, and a young detached address is what another node's
CreatePublicIP looks like until it attaches. A pool with no binding
record is not swept at all, since that is indistinguishable from a store
that cannot be read.
The pass now repeats on a ticker rather than running once at startup.
Both OCI ceilings also resolved to InsufficientAddressCapacity, which
tells an operator the region is out of addresses when the fix is to
release addresses they already hold. AWS separates them, so a tenancy out
of reserved public IPs is now AddressLimitExceeded and a full VNIC is
PrivateIpAddressLimitExceeded.
* feat: run the OCI principal e2e in a tenancy that can hold its dynamic group
vm-single-principal needs the tenancy's dynamic group, and creating one is a
tenancy-root write that a shared test tenancy does not grant us. The topology
could never pass there however correct the code was, so it now takes its own
credential from OCI_PRINCIPAL_*, all six or none. A partial set is refused
before the build rather than at the apply, because it would build one tenancy's
instances against another's compartment. With none set the topology is recorded
SKIPPED instead of spending a provision to fail.
The sweep takes the same credential, or a leftover principal deployment is
destroyed against a tenancy that never held it and reported clean.
Two defects found while proving it:
--dry-run reported FAIL on every topology with a tree source, because the dry
run skips the build and then refuses the tarball it would have produced. The
flag operators are told to run first could not pass.
The dry run truncated its plan to the last 30 lines, which is the outputs block.
That dropped the resource count and every compartment and shape line, so the
one check a dry run exists for could not be made from its output. It now writes
the whole plan and prints the summary.
Removing a deployment is documented as leaving public IPs behind. Spinifex
allocates them on the running node, so they are in no Terraform state and
destroy cannot see them; twenty accumulated and exhausted a tenancy's limit,
which then refused every launch in every compartment.
* fix: canonicalise the Opc-Next-Page header in the pagination test
CI's linter rejects a non-canonical header name and the local preflight does
not, so this passed here and failed there.
* docs(oci): offer both credential routes, and default the guide to the release
Step 2 was one route only, the scoped API key, with the instance principal
mentioned in a TIP at the end of it. It is now "2. OCI Credentials for
Spinifex" with the two routes as 2.1 and 2.2, a table to choose between them,
and the instance principal first because it leaves no key on any node.
The guide also assumed the dev branch throughout. `git switch dev` and
`--channel dev` are now an optional note for installing a pre-release, and the
documented path is the published release.
* test(oci): add a vm-single-principal topology, and cover the credential branch
Instance-principal authentication was broken for a release while the OCI
nightly stayed green, because the run's two axes never crossed: every topology
that built the address allocator authenticated with an API key, and every run
that authenticated as the instance was dispatched with no external pool, so the
allocator was never constructed and the credential never used.
vm-single-principal closes that. It is vm-single with the credential fixed by
the topology rather than by a separate switch, so the name alone decides it, and
pairing it with --no-external-pool is refused rather than run: without the
allocator there is nothing for it to exercise. It joins the nightly's default
list, and naming it alone is the way to test this one thing in a quarter of the
full run.
A unit test covers the same branch without a tenancy. newClient now reaches its
two constructors through package variables a test can swap, which is the only
way to observe the choice off an OCI instance - both real constructors need
either an instance identity or a key file on disk.
* fix(oci): let the tfvars choose the credential, and stop the guide hardcoding paths
The deploy driver passed instance_principal as a -var, which outranks a
tfvars file, so a deployment that set it there had it silently replaced with
off and the nodes went looking for a key. Same defect class as the shape and
node count, and the same fix: the flag is a choice about one run and still
outranks everything, and without it the deployment's own tfvars decides.
The driver now reads instance_principal from terraform.tfvars and
*.auto.tfvars for its pre-apply gates, taking the last assignment as Terraform
does, and checks that guess against a new Terraform output once the apply has
run. They have to agree, because the credential handoff is skipped on the
strength of it and a wrong answer leaves a node with no credential at all.
The guide now carries one export rather than two more steps. SPX_PRINCIPAL is
set in step 2.1, expanded into the tfvars step 4 already writes, and read back
from there in step 5, so an instance-principal deployment runs the same
command as a key-based one.
Getting into the node needed an address copied out of the log. Every topology
now writes its hosts file, not only the multi-node one that hands it to
install-node.sh, and the run prints the ssh and destroy commands when --keep
leaves the infrastructure up. The guide exports TOPOLOGY once, so the paths
read the same for one VM, three, or bare metal.
* fix(oci): print the ssh key path as ~ rather than expanded
The --keep line read ssh -i /home/you/.ssh/oci-spx, which is the invoking
user's home and differs on every machine. Collapsed to ~ when the key is under
$HOME, left alone when it is not.
Admission clamps on live MemAvailable, so on a loaded box running tests in
parallel any daemon test that allocates could see no capacity. TestMain now
disables the live-memory gate; its own tests inject a reader.
admin init and account create wrote a single profile endpoint_url, so aws s3
reached the gateway on :9999, which does not serve S3. Profiles now carry a
services section pointing s3 at the predastore gate, and the INI rewrite keeps
nested settings instead of flattening them.
Servers reply before ending their span, so the client can return before
the span reaches the recorder. SpansByKind now polls until a server span
has ended and fails the test if none does within 5s.
The dev release tagged itself with git describe plus -dev, and describe
resolved to the previous dev tag, so every tag appended to the last until
the tarball name passed the 255-byte filename limit. Describe against
release tags only, here and in the Makefile and bare-metal VERSION.
* fix(oci): run the deploy driver on the bash macOS ships
`validate-topology.sh` died on line 41 with `bm: unbound variable` on a Mac. macOS
ships bash 3.2, where `declare -A` is an indexed assignment and the subscript is
read as an arithmetic expression, so `[bm]` dereferenced an unset variable and
`set -u` ended the run. `mapfile` further down would have failed next.
The guide names macOS as a supported workstation, so the driver now uses neither:
the per-topology defaults are a `case` in a function, and the host list is a read
loop over a captured string. The output is captured before the loop so a failed
`terraform output` fails there rather than silently producing an empty host list.
Verified: all three topologies resolve their shape and node count, and an unknown
topology still names the valid set.
* docs(oci): end the OCI deploy where the install guides end
The deploy command now carries --skip-workload, so the guide stops at a formed
cluster and a working AWS call rather than at a workbook run, and hands off to
Setting Up Your Cluster the way every other install guide does. A step to SSH to
the node was missing between the deploy and the verification call.
Three things this corrects while the section was open. The expected output block
now matches the command it documents, including the --skip-workload line that
ends the run. The --channel dev callout said OCI support was missing from the
deploy driver, when what is missing is an OCI address allocator in a published
release. The verification output was fenced as json while holding a shell
session, which is neither valid json nor the thing being shown.
The note on probing a guest's public address from outside the VCN moves to the
handoff section, since that is now where a guest first gets launched.
`validate-topology.sh` passed the topology's shape and node count as `-var`,
which is the strongest source Terraform reads, so a `compute_shape` or
`node_count` in `terraform.auto.tfvars` was silently discarded. The OCI guide
told the reader to write both into that file and then to run the driver, so a
deployment asking for `VM.Standard.E5.Flex` got `VM.Standard.E6.Flex` and said
nothing about it. Three of the five shapes the guide recommends were unreachable
by the command it gives, including both bare-metal ones other than the shape CI
happens to use.
They now go in as `TF_VAR_compute_shape` and `TF_VAR_node_count`, the weakest
source, so the tfvars file wins and a deployment needs no flag to size itself.
CI is unaffected: `*.auto.tfvars` is gitignored, so a runner's checkout holds no
such file and a named topology still resolves to the same values every run.
`instance_principal` stays a `-var`, because `--instance-principal` is a choice
about the run rather than about the infrastructure's size and a stale tfvars must
not quietly contradict it.
Because the count can now come from the file, the driver no longer presumes it.
The number of addresses in the `hosts_file` output is the authority for the
single-node init path and the readiness gate, a divergence from the topology's
default is logged rather than fatal, and a new `compute_shape` output lets the run
report the shape it built instead of the one it asked for.
In the guide: the tfvars heredoc is unquoted so `$CID` expands, with a `grep` to
confirm it did, and `node_client_cidr_allow_list` is bracketed, since it is a
`list(string)` and the bare CIDR was an HCL syntax error.
* docs(oci): rewrite both OCI guides in the house voice
Seventy-five em dashes across the two guides, with four of them in headings where GitHub and the docs site slugify them differently and a cross-link resolves on neither. Recast as commas, colons and separate sentences.
The silent-credential failure was stated five times as the same triple, "forms, passes every health check, and then refuses every launch". It now reads in full once, under Credentials, and the other four refer back to it instead of echoing the skeleton.
Also broken up: the parallel stack in "Why run Spinifex on OCI", several five-sentence paragraphs, and a handful of clauses that leaned on a dash where a full stop was clearer.
No technical content changed. Every inline-code token is byte-identical before and after, every fenced block is unchanged apart from one comment line, and the four reworded headings carry no inbound anchor links. The docs build reports zero warnings for either page and both render at the same size as before.
* docs(oci): open the install guide with what Spinifex is, as the other install guides do
* docs(oci): ship the credential installer instead of asking for a hook
The guide told the reader to write their own executable to put the OCI API key
on each node, which is work nobody should have to do to install a product, and
the step was unclear about where the scoping actually happens.
`scripts/terraform/oci-spx/spx-oci-config.sh` now does it. It takes
`validate-topology.sh`'s hook contract, `hook <ssh-key> <host>...`, so it drops
straight into `--credential-hook` and is also runnable on its own against an
existing cluster.
It resolves the credential in four steps and takes the first complete one: the
`OCI_SPX_*` environment variables, the `[spinifex]` profile in `~/.oci/config`,
`~/.oci/oci_api_key_spx.pem` with the Terraform profile's identifiers, then the
Terraform credential itself with a warning that it is wider than the allocator
needs. Steps 2 to 4 go through `scripts/oci_env.py`, the same resolver the apply
uses, so the credential on the nodes and the one that built the infrastructure
cannot come from different profiles. The fingerprint is checked against the key
before anything is written, because a mismatched pair is otherwise an OCI 401 at
the first allocation, long after the deploy has reported success.
The node's filename stays `oci_api_key.pem` while the workstation's is
`oci_api_key_spx.pem`: the suffix exists only to keep the Spinifex key off
whatever `[DEFAULT]` uses, and a node has no other OCI key. Nothing reads the
name anyway, since the daemon learns the path from `key_file` inside
`/etc/spinifex/oci/config`.
Key material reaches the nodes over SSH, in the remote shell's stdin rather than
its arguments where `ps` would expose it, and goes into neither user-data nor
Terraform state. `--credential-hook` still accepts any executable with that
argument shape, so a tenancy that keeps credentials in a vault can substitute
its own.
Also in the guide: the policy statements name the compartment by `compartment
id` rather than by name, which is the form that resolves from a compartment the
policy is not attached at; `$CID` is consistently a compartment OCID; and the
table of contents matches the headings again.
* docs(oci): publish both OCI guides on the docs site
The site extracts only four H2 sections and drops everything else, so both guides
would have published nearly empty: the deployment guide matched none of the four,
and the reference doc only its troubleshooting. Insert the section markers the
other published docs already use, keeping every heading at the level it had so no
anchor moves.
The architecture reference becomes oci-architecture/README.md, because a page is
always <slug>/README.md and a flat URL takes the last path segment -- plain
"architecture" would have published at /docs/architecture.
Two headings lose an em dash. GitHub renders it as a double hyphen in the
fragment and the site's slugify drops it entirely, so the two disagreed and the
cross-links resolved on neither.
* docs(oci): the OCI guides sit in their own Cloud Install section
Their category frontmatter named Install, which is the section they are moving out of.
attachedPolicies resolved each attached ARN and, on any error, logged a warning
and skipped it. A transport failure is indistinguishable from a missing record
there, so one timed-out read reported the policy as not attached.
Two paths then read the same field and disagreed. ListAttachedRolePolicies
returned the filtered list, so a client concluded the attachment was gone and
dropped it from its own state without calling DetachRolePolicy, while DeleteRole
checked the unfiltered field and refused for as long as the ARN was recorded.
Nothing could clear it through the API. This is what wedged the ecs-quickstart
workbook's teardown on DeleteRole(ecsInstanceRole) with a 409.
The state being guarded is also unreachable: DeletePolicy refuses while any
principal attaches the policy and fails closed if the attachment scan errors, so
no API sequence produces a principal holding an ARN with no policy record. AWS
cannot return a short list here either -- an attached policy always exists, and
an internal failure is a 5xx.
AWS-managed ARNs are unaffected. They are deliberately stored without a record
and are still reported from the ARN itself, above the lookup.
Closes mulga-tmk5e
* feat: run every Terraform workbook on OCI, and split the guide in two
The OCI topology driver ran the shared driver's default five workbooks, which left ECS and all three EKS variants untested on every platform even though their assertions already existed. It now passes an explicit nine and imports the three appliance images they need — RDS, ECS and EKS each boot their own, found by a spinifex:managed-by tag and never by name.
demo-app stays out because it is not a workbook: it has no .tf at all, being the container image the EKS workbooks' nested workloads modules deploy, so listing it would fail on a missing root module rather than test anything.
The guide is now two documents. oci-integration/README.md is the end-user path — six steps, deploy first, and it closes the three prerequisites nothing documented before: the clone, the ref, and the SSH keypair the driver refuses to start without. architecture.md holds the sizing, the IAM policy, every variable, the three filtering layers and the troubleshooting.
Its firewall advice was also backwards. node_client_cidr_allow_list is an OCI security-list rule over the whole public subnet, and a guest's public address is a secondary private IP on the node's own VNIC, so narrowing it to the operator's address cuts internet ingress to every guest rather than protecting the node. The hardening section now arms the nft policy and narrows SSH through custom.nft, which is the layer that can tell a node's listener from a guest's.
The deploy step names --channel dev rather than a pinned tag. The dev tag string grows with every cut, so a pinned example goes stale within the week.
* fix(oci): the guide could not work on a fresh tenancy, because the profile name was ours
oci_env.py bridges ~/.oci/config into TF_VAR_* — the provider takes the tenancy, user, fingerprint, key and region as Terraform variables and reads no config file itself — but the profile it looked for was the reference tenancy's, and validate-topology.sh passed no way to change it. A fresh account's config holds [DEFAULT], so every deploy would have stopped at "OCI profile 'apacanzset03child03' was not found" before building anything.
The profile is now resolved as --oci-profile, then $OCI_CLI_PROFILE, then the reference tenancy, then DEFAULT. A name passed explicitly and absent is an error listing the profiles present, never a fallback: the failure worth preventing is building a cluster in whichever tenancy happened to be first. The reference tenancy still outranks DEFAULT, so existing runs are unchanged, and CI is untouched because it takes the environment route rather than a profile.
has_profile exists because configparser treats DEFAULT as the fallback for every other section rather than as a section, so has_section denies it exists even when it is the only thing in the file — which made the first version of this fallback do nothing for exactly the case it was written for.
Three documentation corrections found while checking the guide against the code. The IAM policy is ten operations and needs "use vnics", not "read vnics": registering a secondary private IP is an operation on a VNIC, so OCI checks VNIC_ASSIGN, which lives in the use verb, and with read the node forms and looks healthy and every allocation fails NotAuthorizedOrNotFound. These are now the same three statements instance-principal.tf grants, which is the only version of the policy that has been exercised.
The guide also promised a worked example of the credential hook that does not exist and should not: nothing here distributes a credential. It now states the contract instead — the argument shape, the two files, and that a non-zero exit stops the deploy before anything is built. And the dev channel's rate limit is documented with the tag as its remedy, rather than left in the driver's help where a reader will not see it.
* fix(ci): a failing workbook's log never reached the artifact, so only the runner had it
run-tofu-examples-e2e.sh writes its assertion output to the per-topology workbooks.log in the state directory. "Sweep anything left behind" deletes those directories and runs before "Upload the run", so the upload's three oci-state/.validate-* patterns matched nothing and said so to nobody. The first nine-workbook run lost the one thing it was dispatched to find: ecs-quickstart failed on vm-single and the reason existed only on the runner, which does not resolve from a workstation.
A collect step now copies *.log, *.txt and *.tsv into the artifact directory before the sweep, prefixed by topology so two topologies cannot overwrite each other. The workflow already records that "Publish the result tables" sits ahead of the sweep for this reason; the logs were simply never given the same treatment.
The dead patterns are removed rather than left. A path that cannot match is worse than no path, because it reads as protection that is present — which applies to the credential-hook exclusion too: it now lives in the collect step, which is the only thing that writes into the artifact tree, so the file cannot reach an artifact rather than being filtered out of one.
* fix(ci): a state directory kept the last run's logs, and the artifact published them as this run's
Collecting the per-topology logs surfaced the next defect immediately: a vm-single-only run produced an artifact holding bm-workbooks.log and nine vm-multi node logs, because those state directories still held the previous run's output. Anyone reading that artifact would conclude three topologies ran when one did, which is worse than the missing log it replaced.
Two causes, one each side. validate-topology.sh never cleared its own logs, so a topology's directory accumulated them; it now removes *.log, nodes.txt and cloudinit-*.txt as it starts, before the first line of the new run, and never the tfstate, which is how the sweep finds hosts to destroy. The globs are named rather than a find, so what goes is readable at a glance, and :? stops the shell instead of expanding to /*.log if STATE_DIR is ever unset.
That alone does not cover a topology the run skipped, whose directory is never entered. So the collect step now takes only the topologies this run drove, read back from the per-topology log the harness writes rather than from the dispatch input -- a run that names no topology gets the harness's default, and reading that here would be a second copy of the list to disagree with the first.
Also proved by this: ecs-quickstart passes on a freshly built vm-single, 172s, where it failed at 293s as the seventh of nine. The failure is order or load dependent on one 32 GB node, not a property of the workbook.
* fix(ci): a failed workbook run left no node journal, so the cause was unreadable
The workbook log says which API call failed; only the journal says why, and the
nodes are destroyed minutes later. An IAM teardown failure on the nightly came
down to one swallowed warning that existed nowhere else, and by the time the
artifact was read the host it was logged on had been torn down.
Warnings and above first, since that is where a swallowed error surfaces, then a
bounded tail for context. Unbounded, a multi-hour suite's journal would dwarf the
rest of the artifact. The existing state-directory clear keeps these from going
stale between runs, and the collect step already globs *.log.
* feat(oci): separate using an instance principal from creating one
The allocator needs OCI credentials at runtime, and a node with none forms,
passes every health check, then refuses every launch wanting a public address.
Instance principal is the credential route that puts no key material on a node,
but it needs a dynamic group and a policy, and those are tenancy-root resources
the compartment-scoped user the rest of the configuration runs as cannot create.
enable_instance_principal conflated the two, so the better credential route was
only reachable by a tenancy admin. instance_principal now takes off, adopt or
create: adopt selects the credential and creates nothing, because the matching
rule covers the whole compartment and so names no instance to reference.
setup-identity.sh owns the create case, with its own state. validate-topology.sh
destroys its state at the end of every run, so holding tenancy resources there
would let a teardown delete the tenancy's policy.
Also package install-node.sh and smoke-test.sh, so a cluster can be formed from
a curl | bash install with no checkout. smoke-test.sh is not optional: formation
scp's it from its own directory for the post-formation verification.
* feat(oci): take the credential handoff from an operator hook, and widen the workflow name
An API-key deployment needs a credential on each node, and nothing in this
repository can supply one: a key in user-data is readable from instance metadata
for the life of the instance, and this host runs other people's guests. A manual
install does not survive either, because a topology destroys itself on success.
So the harness calls an operator-owned executable after formation and before the
pool, the only window where a node has /etc/spinifex and has not started the
allocator. Instance principal needs no handoff, so the hook is skipped there.
The workflow is renamed for the vendors it will cover rather than the first one.
Nothing is orphaned in the Actions UI: it has never been on the default branch
and has never run, so there is no history attached to the old filename.
* docs(oci): the API-key path needs the config file, not just the PEM
oci_config_file names an ordinary OCI SDK config and nothing creates it, so a node
holding only the key forms and then fails every allocation with no stated cause.
* fix(ci): the runner context is not available in a job-level env block
A workflow referencing it there is rejected before any job starts, which is why
the rename produced a failed run attributed to the push. The reference had never
been validated: a dispatch-only workflow that is never dispatched is never parsed.
* fix(deps): bump containerd to v2.4.1 for GO-2026-6597
The image-pull DoS via crafted OCI index graph amplification is reachable from the ECS agent's containerd puller, so the vulnerability scan fails preflight on v2.4.0. A patch release with no API change.
* fix(e2e): probe guest public addresses from outside the cloud's own network
A VCN does not reflect its own public addresses back into itself, so a node asking for one gets nothing however well the datapath works. The workbook driver runs on the first node, so every assertion that curls or SSHes a guest's public address failed on OCI against a cluster that was serving those addresses correctly to the internet.
The driver now takes E2E_PUBLIC_PROXY, a SOCKS5 proxy used for public addresses only, and validate-topology.sh supplies one with a remote dynamic forward on the connection that launches the driver. Egress is then the host running the validation, which is outside the VCN and is where a customer sits. Unset leaves every other platform on a direct probe, which is already correct there.
Also adds --version TAG for a repeatable install, separates the workflow's auth choice from its external-pool choice so an API key can run with the allocator live, and captures allocator diagnostics before the teardown destroys them.
* ci: read the Go version from go.mod in the release workflows
Both pinned 1.27.0 by hand while every other workflow reads the file, so the toolchain they build with drifts from the one the module declares the day go.mod moves. release.yml was already inconsistent with itself: its second setup-go step reads spinifex/go.mod.
* feat: publish the OCI topology results as tables on the run page
A green tick on one job with 4,000 lines of log behind it is confidence for
whoever wrote the harness and nothing to anybody else. Someone assessing whether
Spinifex works on their cloud should be able to see that it formed and served
guests on a single VM, on three VMs and on bare metal without opening a log.
record now takes <gate> <status> [detail] and writes results.tsv beside the
human block, so a gate's status is a field the script sets rather than a regex
over prose that reads fine and matches wrong. record_workbooks turns the
driver's own go-test-shaped lines into workbooks.tsv, and runs on the failing
branch too: which four passed is most of what the fifth failing means.
e2e-oci.sh --summary-only renders both tables to stdout and to
GITHUB_STEP_SUMMARY, and the workflow calls it under always() before the sweep,
because the sweep destroys the state directories the tables are read from. A
skipped gate gets its own mark, and a topology that never ran says so: the one
way a results table can do real harm is a reader taking an absent row for a
passing one.
* docs(oci): cut the deploy-by-hand appendix, and validate the config for real
The appendix was 444 of 1014 lines restating an install the Terraform already
does, which made the document read as though deploying by hand were a normal
route. It is not: the configuration builds the VCN, the volumes and both VNICs
per node and renders the cloud-init, in the same order every time. What is
genuinely OCI-specific is four things, each of which already has a section, so
the replacement names those and sends everyone else to the single-node and
multi-node install guides.
Verify end to end told the reader to ping a guest's public address from the
host, which cannot work: a VCN does not route its own public addresses back into
itself. That is the finding this branch is built on and the document was
asserting the opposite. It now says where to probe from and why, and carries the
allocator's `resolved the external VNIC` check, which existed only in the cut
appendix and is the one line proving the credentials and compartment are right.
`spx config validate` is not a command -- spx has admin, get, service, top and
version -- so that line could only ever fail and its output and status were both
discarded. It is replaced by parsing the TOML, which is what was wanted: the
remap and the pool append are textual edits, and a broken result would otherwise
surface as every service failing to start with nothing saying why. The remap now
checks its own substitution count for the same reason, since re.sub with no
match rewrites the file unchanged and exits 0.
Publication of the run-page tables moves to one owner. GITHUB_STEP_SUMMARY is
per step, so appending from the run as well as the publish step would have put
two copies of every table on the run page.
* fix(oci): check the credential hook before building anything
The hook is a path from a CI variable, and whether it exists on the runner is
knowable before Terraform runs. Checked where it was used, a wrong path cost the
apply, the install and the formation first -- forty minutes and a bare-metal bill
to be told a file is missing. Nothing about the check depends on the apply, so it
moves to the top, where --dry-run reaches it too.
* fix(oci): the summary must say what was built and what was not tested
The first CI run headed its tables "Published release `latest channel`" over a
build of a feature branch, which is the one error in a results table that would
mislead a reader who trusted it. The header now states a tree build as a tree
build, and names how the allocator authenticated.
A run with no external pool says so in the header rather than leaving it to a
SKIPPED row further down. Public-address allocation is the claim a green table is
most likely to be read as making, and it is exactly the claim such a run cannot
support. Both flags are passed to the publish step too, so the summary rendered
after a failure describes the same run as the one that failed.
Raise the dev prerelease prune from five to twenty. A pinned dev tag is how an
external evaluator installs a specific build, and five is about a week of them --
short enough that the tag someone was handed stops resolving while they are still
using it.
* docs(oci): the workbook table is a snapshot, and the nightly is the current answer
It is dated against a build five releases of work ago, and the nightly now runs
the same workbooks on all three topologies and publishes its own table. Say which
one to read rather than letting a stale table be taken as current.
* ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file
A workflow's displayed name is captured when GitHub first registers it and is not
refreshed afterwards. Both OCI entries were registered from a run that could not
be parsed -- a job-level env block referencing runner.temp -- so both list as
their path, which reads as a missing name: key and is not one. The 2026-10-02 run
was green on all three topologies and the entry still read as a path, with
updated_at unchanged from the broken first run, so fixing the file is necessary
and not sufficient.
A new path is a new registration, and its first run reads a valid file. Dropping
-nightly is also more honest: there is no schedule: here, because GitHub
registers one only from the default branch, so a cron could never run dev and a
feature branch as separate nightlies.
* Revert "ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file"
The rename makes the workflow undispatchable. `gh workflow run` resolves a
filename against the default branch, and the raw dispatch endpoint returns 404
for a path GitHub has not registered -- checked twice, several minutes apart, so
it is not an indexing delay. The old name works only because it is already
registered, and an unrunnable workflow is a worse problem than a mislabelled one.
The sidebar label needs a fresh registration, which a rename cannot provide from
a feature branch. It corrects itself when the file reaches main, where GitHub
re-reads it; the alternative is deleting the dead registrations' runs, which is an
operator's call.
* fix(ci): never publish the credential hook's output as an artifact
The hook is the operator's own executable, and the one thing known about it is
that it handles a private key. Its stdout and stderr are captured to
credential-hook.log, which the artifact's *.log glob was collecting into a
downloadable archive with a fourteen-day retention.
Excluded rather than trusted to have been quiet. A hook that prints its key once
while someone is debugging it should not thereby publish it.
* fix(oci): the hairpin probe measured nothing on one node, and claimed too much
With a single host, HOSTS[0] and HOSTS[-1] are the same address, so the node
connected to itself -- which always succeeds. It reported "present" on vm-single
and bm for that reason and not as a measurement. It now needs two nodes and says
so otherwise.
The claim it carried was also wrong. "A VCN does not reflect its own public
addresses back into itself" was read off a workstation run whose
node_client_cidr_allow_list was narrowed to one office address, which makes a
node's own public address a non-permitted source: the refusal that follows is
identical to an absent hairpin. With the committed 0.0.0.0/0 default the probe
reports the hairpin present between two nodes, so the original reading was
confounded and is retracted.
None of this changes the remedy. A customer reaches a guest from outside the VCN,
which is where the proxy's egress is, and probing from there is correct whichever
of the two causes is in play -- 15 of 15 workbooks pass through it on all three
topologies. What changes is what the documentation may assert: the guide now names
both causes as indistinguishable rather than stating one as fact.
* docs(oci): a reader could not run the first command, because nothing told them to clone
The guide's first code block was `cd scripts/terraform/oci-spx`, into a directory
no step had obtained. It also never mentioned the SSH keypair the driver refuses to
start without, nor that the configuration is not on main or in v1.21.0 yet, so all
three had to be inferred from the deploy failing.
Restructured so the deploy comes first and reads as four steps and one command:
get the configuration, choose one node or three, write the inputs, run it. The
credential decision stays before step 4, because it changes an argument there and a
node without one forms, looks healthy, and then refuses every launch that wants a
public address. Everything else -- why OCI, sizing, the architecture, the IAM
policy, quotas, the oci CLI, what Terraform builds, every variable -- moves under
Reference, unchanged in substance.
Adds what a successful run prints, naming `allocator ready` as the line to look for:
every line above it appears on a node that cannot allocate, and that is the one
failure the run does not announce.
Also drops an empty verdict.txt committed from a local run. The directory the guide
now tells customers to work in should not ship one run's artifacts.
The installer served by install.mulgadc.com is the latest release's, so it
predates the --channel flag the harness was passing and would have died with an
unknown option. Document the same form in both install guides.
* fix: size the volume lease against the stall the host actually produces
A guest lost its disk whenever JetStream writes failed for fifteen seconds,
which is what one NVMe I/O timeout on the node holding the JetStream leader
produces. The budget composed badly rather than any single number being wrong:
a confirmation could already be volumeLeaseRenewInterval old when an outage
began, and validity expired volumeLeaseValidity after it, so an outage covering
[15s, 30s) surrendered a volume no peer had taken. Measured outages are 10-15s,
so the lease was sized at the edge of a routine event. On one cluster thirty
eight renewal timeouts produced six surrenders and four destroyed guests.
Validity rises to 40s and the renewal interval falls to 10s, taking the
tolerated outage from fifteen seconds to thirty. The validity-to-TTL gap is now
named by volumeLeaseAcquireMargin instead of being implied by two constants:
nothing may acquire until the server-side bucket TTL ages the entry out, so the
margin only has to cover scheduling and request latency on this side.
The renewal timeout rises above the stall bound, so a renewal that begins inside
a stall is answered when it clears rather than being certain to fail. That is
not a constant on its own. renewLoop was one goroutine, so a blocking Update
delayed the validity check by the whole timeout: measured on the old loop with
the new timeout, a holder surrendered 51s after its last confirmation, eleven
seconds past validity and six past the server TTL, which is a window where a
peer may already hold the volume. The renewal now runs on its own goroutine with
one call in flight at a time, the loop keeps testing validity underneath it, and
it waits for the renewal before closing done so release still reads a settled
revision.
deviceStallBound documents nvme_core.io_timeout as a property of the host. It is
not configured or applied here, only asserted against, because it describes the
machines rather than the product.
The sizing is a probability, not a guarantee. An outage longer than validity
still surrenders, and the fence still destroys the guest instead of revoking the
export and leaving it paused, which is the change that would make outage length
stop mattering.
* fix: keep a fenced instance out of every automatic relaunch
A fence does not seal, so the volumes it leaves behind are a state nothing has
shown is mountable. Two paths relaunched one anyway. The restore path consults
the restart window, which forgives on a clock, so a fence days old reports a
full budget and the next daemon start remounts it. The recovery reconciler
excludes a storage fault but not a fence, and a fence kills the export instead
of pausing the guest, so the record publishes no I/O error and reads as an
ordinary desired-running instance on a node that has died.
Both now read vm.VolumeFenced off the StateReason the fence already writes. The
refusal is narrower than StateError on purpose: a recovery that failed for its
own reasons is what the restart budget exists for.
Also separates the two storage quantities the lease was sized against. One
stalled I/O lasts deviceStallBound, which is what a renewal timeout has to
exceed; what the lease has to survive is a run of them, now named
jetstreamOutageBound. They were conflated, so raising the device constant would
have licensed a validity past the server TTL.
A surrender can only ever report validity, so the length of an outage was
unobservable. Renewals now count consecutively and report outage_ms once they
recover, which measures the windows the lease survives.
install.mulgadc.com has served a dev channel since it was written -- setup.sh
understands INSTALL_SPINIFEX_CHANNEL=dev and the Worker resolves it to the newest
GitHub prerelease -- but nothing ever produced one, so /download/dev/amd64
returned 404. This is the producer.
A separate workflow rather than conditionals in release.yml. A dev build is a
different artifact, not a release with flags off: amd64 only, no ISO, no system
images, no bare-metal gate, and nothing on a self-hosted runner. The sibling
repositories are taken at dev rather than their default branch, because dev of
spinifex against main of the others is a pairing nobody runs.
The tag is the describe string plus -dev, which release.yml's own convention
already treats as a prerelease, and which makes the commit recoverable from the
version alone. The flag is set at creation and then the published state is
asserted: a prerelease must never become the latest release, because latest is
what an unqualified curl | bash installs.
setup.sh gains --channel and --version so the instruction is a flag rather than
an environment variable, and the OCI harness gains OCI_CHANNEL so a nightly can
test the path the contacts are actually given.
* fix(oci): make a bare-metal node form and mount its data volume
Bare metal presents VNICs differently from a VM: the first VNIC on a physical
port is the port, and a second one sharing that port is a tagged overlay with no
Linux device at all. The external VNIC was landing in that second slot, so no
interface ever carried it and the WAN bridge could not be built. Attach it at
nic_index 1 on a BM shape, which makes it the first VNIC on the second port and
therefore untagged and present at boot, exactly as it is on a VM.
Two consequences of that followed. The bridge script picked the first non-primary
VNIC from instance metadata, which on bare metal can be one with no interface, so
it now selects the VNIC the kernel actually has a device for and waits for that.
And OCI makes a bare-metal volume attachment multipath, where the Oracle agent
refuses to log in unless multipathd is running -- probing the unit name Debian
retired, so on Ubuntu 26.04 it never sees it. Alias the retired name and assert
the alias on every poll rather than once, because the first run had it created and
then vanish.
Finally, the three cloud-init steps become ordered systemd units. runcmd
continues past a failed step, so a node whose volume never appeared still opened
its service ports with nothing behind them; Requires= makes that impossible and
lets a boot that raced the iSCSI login retry on the next one.
* feat(oci): authenticate the address allocator as the instance principal
The OCI allocator could only authenticate from an API-key config file, so every
node needed a private key copied to it and a config file hand-written beside it.
A node without them forms, passes every health check, and then refuses every
launch that wants a public address with InsufficientAddressCapacity -- the cause
appears only in the journal.
NewInstancePrincipalClient already existed and was called from nowhere. Wire it
in behind a new oci_auth key on the pool, so a node can authenticate with the
certificate its own metadata service serves and hold no key material at all.
Nothing to copy, nothing to rotate, and nothing in Terraform state.
The reason recorded for preferring the config file was that it reads no instance
metadata, because Spinifex's own IMDS endpoints claim 169.254.169.254 and take
the cloud's metadata service with them. That no longer distinguishes the two:
oci_vnic_iface is what the guide recommends and it resolves the VNIC through a
metadata lookup already, so such a node depends on the IMDS remap either way.
The two constructors' comments also contradicted each other on which was
preferred; both now say what they cost.
config_file stays the default, because instance principal needs a dynamic group
and a policy only a tenancy admin can write. The Terraform creates both behind
enable_instance_principal, off by default for the same reason the module creates
no other tenancy-root resource, and the policy grants three verbs on vnics,
private-ips and public-ips in one compartment rather than anything broader.
Naming the method explicitly rather than inferring it is deliberate: a node meant
to use its own identity now fails at config load instead of falling back to a key
file nobody installed. Pairing oci_auth="instance_principal" with a config file
is refused for the same reason, since the certificate is already the credential
and the file would be the half silently ignored.
Terraform also stages the pool fragment on each node, which under instance
principal contains no secret at all.
* feat(oci): validate all three topologies from one script, and document both auth routes
validate-topology.sh builds one topology from nothing, installs the published
release, forms the cluster, configures the pool and the IMDS remap, runs a
workbook against it, proves the workbook actually serves traffic, and destroys
everything. Three topologies because each breaks differently: bare metal presents
VNICs unlike a VM, a single node has no Geneve underlay to get wrong, and only a
cluster exercises RAFT, the gateway chassis and cross-node allocation.
--topology has no default, deliberately. A command aimed at the wrong topology is
the easiest expensive mistake available here, and each keeps its own Terraform
state so two can be built from one checkout without either destroying the other's
instances.
The teardown decides the verdict rather than tidying up after it. A topology or a
workbook that cannot be destroyed is half proved, so destroy runs from an EXIT
trap and a teardown failure fails the run. It also checks the two things a green
apply does not: that vpcd logged resolving the external VNIC, which is the only
evidence the credentials work and br-wan matched a real VNIC, and that the load
balancer returns 200 -- a workbook can create cleanly and carry no traffic.
The guide gains the instance-principal route beside the key file, and loses its
claim that bare metal changes nothing else in the document. That claim was wrong
in exactly two places, and both are now handled rather than left to the reader:
the external VNIC's nic_index and the Oracle agent's multipath unit name. It now
also says plainly that the external-address path on bare metal is unproven.
* test(oci): let a topology be validated without an OCI credential
The external pool is the one phase that needs a key or a dynamic group, and
the VM topologies have to be regression-tested regardless. --no-external-pool
forms and verifies without it, and implies --skip-workload so the run cannot
claim a guest path it never exercised.
* test(oci): one entry point that proves a ref works on OCI, and cleans up after itself
e2e-oci.sh drives every topology from environment variables alone, so a
workstation, a Worker dispatch and e2e-oci-nightly.yml all take the same path
rather than drifting into three answers about whether OCI works.
Three things it deliberately does not do. It does not test the published
release: --distro installs the tarball built from the ref under test, because a
nightly that cannot fail on a bad commit is decoration. It does not carry its
own workbook assertions, running tests/e2e/run-tofu-examples-e2e.sh over the
published workbooks instead, so a workbook that passes on a hypervisor and fails
here is a difference in OCI. And it cannot quietly leave cloud resources
running: --sweep-only destroys whatever an earlier run's state still holds, and
it runs before the topologies, after them, and from an always() step.
The credential now reaches Terraform as a value through OCI_PRIVATE_KEY, so no
runner writes key material to disk, and oci_env.py refuses --shell on that route
because in CI stdout is the job log.
Four harness defects the first real runs exposed, each of which had been giving
a wrong answer:
- oci_env.py announced itself on stdout, corrupting every captured
terraform output. That is also why a single node came back as two hosts.
- terraform output was passed -var, which it rejects, and a silent fallback
then read the default state file instead of the topology's.
- grep -c Ready matches NotReady, so a single-node run whose only node was
NotReady on 0.0.0.0 reported PASS. The status column is parsed now, and
membership is waited for rather than sampled once.
- install-node.sh refuses fewer than two hosts, so a single node forms with
spx admin init the way the guide documents.
Also waits for the dpkg lock: cloud-init being finished does not mean apt is.
* fix(oci): resolve mulga's path explicitly, and refuse an unpushed ref
Two assumptions in the new harness that a run would not have caught quickly.
Deriving mulga's root from spinifex's is right for a submodule checkout and
wrong for CI, where the two are siblings under the workspace, so MULGA_ROOT is
explicit and the workflow sets it.
build-install-artifacts.sh clones each repo from GitHub rather than using the
checkout beside it, and falls back to dev when the ref is not there. A nightly
on an unpushed branch would then pass or fail on dev's code while reporting the
branch. Check the ref is on origin first, and skip the Go suite build: the
workbook driver is a shell script, so those binaries prove nothing here.
* fix(oci): do not read $? after a command when set -e is on
The workbook phase read the driver's exit status on the line after the ssh that
produced it, which under set -e is never reached: a failing run aborted before
printing the driver's own per-workbook diagnostics, losing them exactly when
they are the thing you need. An `if !` keeps them, and names which workbooks
passed and which did not.
* feat: let an OCI e2e dispatch form without the external allocator
A tenancy without the nodes' dynamic group cannot authenticate the
allocator, so every topology failed at it rather than reporting what it
had proved. The no_external_pool input forms with no allocator and
records the allocator and the workbooks as SKIPPED, never PASS.
Also rebuilds the dry-run plan message imperatively: it still referenced
a variable renamed to WORKBOOKS, and under set -u the substitution died
and the text silently vanished from the line.
* fix: name the offending input when an OCI credential is unusable
A credential from the environment reached the SDK unvalidated, so a
secret with a trailing newline failed as an uncaught InvalidConfig
traceback naming an SDK config field that nobody set by that name.
Strip the scalar values, since a secret store written with echo carries
a newline, and translate both InvalidConfig and InvalidPrivateKey into
the variable or config key to go and fix.
* fix: make the OCI nightly able to build, and stop it leaking a dispatch input into the shell
Four defects, found reviewing the workflow rather than running it.
The builder clones all five sub-repos from GitHub over https and they are
all private, so a hosted runner had no credential and the tree build
could never have worked. The app token now covers six repositories and
git carries it for our organisation only.
A dispatch input was interpolated into a run block, where a quote ends
the string and the remainder executes. It goes through the environment
now, which is the only place a caller-supplied value belongs.
The artifact upload took the whole directory, which meant the distro
tarball and the test archive, and would have taken terraform.tfstate had
the state directories sat under it. It is a named list now, and it also
collects the per-topology logs that were never uploaded at all.
* fix: make the OCI nightly's runner a setting, and stop widening the app token
The five sub-repos are public, so build-install-artifacts.sh clones them
without a credential and the token has no reason to reach them. Only
mulga is private, which is what it was scoped to before.
The runner is now inputs.runner, then the OCI_E2E_RUNNER variable, then
hosted. A per-provider self-hosted agent is a reasonable thing to want --
it lifts the six-hour cap a hosted job carries and keeps a long cloud
wait off the hypervisor labels -- and it should not need a workflow edit.
Removing the SSH key is load bearing on that path: a self-hosted runner
keeps $HOME between jobs, so the key would sit there between nightlies.
* fix: install the OCI SDK the harness needs, and keep state out of the checkout
Nothing installed the oci Python module, on any runner: validate-topology.sh
called bare python3 and oci_env.py imports oci, so every topology would have
failed at the first terraform call. The harness now uses a system python3 that
has it and otherwise builds its own virtualenv from requirements.txt, which is
the same behaviour on a workstation, a hosted runner and a self-hosted one.
The state directories move under OCI_STATE_ROOT, defaulting to the script
directory. A persistent runner needs them outside the checkout, because
actions/checkout wipes untracked files there and would remove the state of a
killed run before the sweep could read it, leaving OCI instances running that
nothing is able to find.
* fix: drop the OCI SDK dependency, so the harness runs on any host
python3-venv is not installed on banksia, and the harness needed it: the
environment helper re-executed itself from a virtualenv to import the OCI
SDK. It used the SDK for two things -- parsing an INI file, which
configparser already does, and discovering the tenancy home region, which
no Terraform file consumed. Neither justifies a dependency on every host
that runs this, so the helper is standard library only now.
The credential is checked for shape instead, locally and before any OCI
call, which is a better failure than a 401 forty seconds into a run.
Two things the removal uncovered. The home region is a fixed property of
a tenancy rather than something to rediscover every run, so it is
OCI_HOME_REGION. And OCI serves IAM writes from the home region while the
dynamic group and policy were using the default provider for the build
region, which would have failed when the tenancy applied them: they go
through an aliased provider that defaults to the build region, so a
tenancy whose home region is where it builds sees no change.
DeleteInstanceProfile now deletes via kvutil.DeleteIf at the revision it
checked, so a role attached after the RoleName check sends the delete back
through the DeleteConflict guard instead of removing the profile.
ListInstanceProfiles logs and skips a profile whose attached role no longer
exists rather than failing the whole account listing.
* docs: add work-in-progress Exoscale integration guide with stack diagrams
* feat: source=exoscale external pool allocating Elastic IPs through the exo CLI
* docs: rewrite the Exoscale guide as a short end-user install, and test the exo fake, store and wiring
Fuzz QueryParamsToStruct over query bodies decoded into EC2, ELBv2 and RDS
inputs: no panic, list entries bounded by MaxSliceLen and the input size, and
an entry past the first gap never changes the parse. make fuzz runs every
Fuzz* target for FUZZTIME each.
bodyscope lower-cased names with strings.ToLower and decoded into a map, while
handlers decode with encoding/json. A non-ASCII name that encoding/json folds
onto an ASCII field (repoſitoryName), or a repeated name whose later value is
null or an object, let the gate authorize one resource while the handler acted
on another. Parse now decodes with encoding/json/v2, which refuses repeated
names, and refuses non-ASCII names; both yield ErrAmbiguousBody.
FuzzGateMatchesHandler checks, for every identifier the ECR, ECS, EKS, ACM and
Bedrock resolvers read, that the gate and a handler-shaped decode agree.
Removes directives for linters that are not enabled (noctx, exhaustive,
deadcode) and gosec suppressions already covered by config exclusions.
Tags ochrevector QueryResult and Filter with their existing field names
so musttag passes without changing the NATS wire format, and bounds the
RDS window slot conversion instead of suppressing G115.
Peer /health and predastore readiness probes skipped TLS verification.
Both now trust the cluster CA (nats.cacert) that issues every node's server
cert. A node without the CA stops peer probing, leaving peers reachable,
and reports predastore not ready.
Update request and EC2 error-code counts after the suite artefacts were removed, describe how the sweep now reaches its targets, and point at the plan that traces each finding.
Salts restart per operation, so tag, untag and update cases only reached a resource when the same-numbered create case had made one. Create an IAM, ECS and ECR fixture per run and point every operation but a create or delete at it, and give ECR's policy and manifest documents values AWS accepts.
Clear the warnings the conformance suite caused itself, so what remains is a real divergence from AWS:
- Seed conditionally required members into the base request of RDS CreateDBInstance and RestoreDBInstanceFromDBSnapshot, ECS CreateCapacityProvider and CreateService, and EC2 CreateCapacityReservation.
- Give loosely modelled members values AWS accepts: ELBv2 ProtocolVersion and Matcher.HttpCode, EKS policyArn, addonName and access-entry type, EC2 SourceRegion and STS AccessKeyId.
- Import a CA-signed certificate, key and chain in ACM ImportCertificate.
- Rewrite ACM's lookahead domain pattern as RE2 so RequestCertificate is generated, and give the integration ACM service a tenant CA permitting example.com.
- Run the service daemon-lite behind the cross-tenant boundary test.
- Accept ELBv2's common parameter validation errors in responses.
TestServeQueueGroupBalancesSharedSubjects read both workers' handler counts
immediately after waiting for the mount to land on both of them, on the
argument that ordering on the shared connection meant the earlier capabilities
message must already have been delivered to everyone getting it.
That ordering does not exist. Each async subscription dispatches on its own
goroutine, so a worker's mount handler and its capabilities handler race, and
the mount having landed says nothing about whether the capabilities message has
been counted yet. Race Detection caught it: publishes reached 2 while
capabilities was still 1.
Wait for the expected count and then confirm it does not move. Both halves
matter — how many handlers a message reaches is the server's decision, so the
count converges from below and cannot overshoot, but a broken queue group
delivering to both members would pass through the right value on its way to the
wrong one.
* fix: wait for JetStream quorum before deciding viperblock has no storage
On a cold multi-node start every service comes up at once against a cluster
that has not elected a meta leader yet. IAM retries that window; viperblock did
not. Its lease and dirty bucket opens are both on the path to the daemon
running at all, so the first refusal took the process down with "volume lease
bucket: context deadline exceeded" and systemd crash-looped it into a failed
bootstrap.
The start path now opens both through kvstore.OpenWithRetry, which is what vpcd
and IMDS already use for the same window. The plain constructors are untouched:
they are what the replica-count and cluster-size tests assert fail closed, and
an operator tool reading a refusal as a hang would be worse than the crash.
OpenWithRetry now stops on an undeclared cluster size, matching the IAM and
kvlease loops. Waiting cannot declare one, so retrying it would turn a clear
configuration fault into ninety seconds of silence followed by the same error.
* test: mark the viperblock bucket startup test as in-package
test-package-check requires new test files to be external unless the file says
why it cannot be. The start-path openers and the constructors beneath them are
both unexported, and which of the two waits is the whole subject.
* fix: assert the takeover against the volume that was actually left unsealed
TestCrossNodeStartAfterFailedSealTakesOverLoudly picked its subject in advance
— the attached pattern volume — and treated the stop returning an error as
proof that volume's seal had failed. It is not. The unmount is a NATS request,
and the caller giving up says nothing about the seal behind it; local state
kept with no seal receipt is equally what an in-flight seal looks like.
In the nightly the test then thawed predastore, the pattern volume's seal
finished against a backend that was answering again, and the marker cleared. By
the time the volume moved there were no unsealed writes, so viperblock was
right to say nothing and the test failed it as a silent loss.
The product was loud where it mattered. The instance's root volume has to read
its AMI snapshot to seal, which the freeze does break, so its seal genuinely
failed and the node it landed on logged the takeover naming the node that had
been holding the writes. A blank attached volume has no such read and can seal
straight through the outage.
So the test now reads which volumes the cluster records as unsealed, after the
thaw has settled rather than before, and asserts the takeover against those. If
nothing was left behind it skips: the fault injection did not reproduce, and
asserting a warning there would be asserting a wrong one.
* fix: publish the instance cache resync mark with its readiness, not after it
installSync set ready under c.mu and markResynced ran after the unlock, so
there was a window where the cache was ready and serving records while
resyncAge reported that no sync had ever succeeded. A caller that observes a
ready cache and then asks how stale it is got "never", which reads as a cache
that cannot be trusted, and the resync-age metric was skipped for as long as
the window lasted.
The mark now goes up under the same lock that publishes readiness, so any
reader that takes the lock and sees a ready cache sees the mark too. Both
callers dropped their trailing markResynced, which installSync now does.
Widened under the race detector, which is where TestPeriodicResync_Undecodable
RecordMarksDegraded had been failing intermittently: it reads resyncAge
straight after waiting for readiness.
* fix: replicate every JetStream KV bucket across the cluster
A KV bucket created without a replica count gets one replica, and one
replica lives on one JetStream-chosen server recorded in no config.
Losing, restarting or partitioning that server makes the bucket return
"no responders available" from every node at once, so a single-node event
becomes a cluster-wide outage. The reconciler leader-lease buckets were
the worst case: acquiring a lease is a write, so a lease bucket without a
quorum cannot be acquired by anyone, and every reconciler sharing it stops
cluster-wide at exactly the moment a node has failed.
The cluster's node count is now declared once, in config.LoadConfig, and
read by the one helper every bucket creation goes through. Every process
that creates a bucket loads the cluster config first, so no service can
forget to pass it and no signature has to carry it. An undeclared size is
an error at creation rather than a silent fall back to one replica.
Existing buckets are raised rather than left behind: a service opening one
raises it, the daemon sweeps them all after formation, and
"spx admin kv replicas --repair" does it on demand. Raising reads the live
stream config and changes only the replica count, and never lowers one.
Removes the ~30 replicas/clusterSize parameters that only forwarded a
number, and JetStreamManager.UpdateReplicas, which set len(nodes) with no
clamp and would have failed outright on a cluster of six or more.
Proved by a repo-wide invariant test pinning bucket creation to the one
helper, a whole-cluster sweep in every multi-node nightly cell, and a new
kvquorum suite (cell 33) that takes away the node leading the most buckets
and requires the survivors to keep answering for all of them.
* chore: name cell 33 in the nightly dispatch input
* fix: a KV bucket that cannot be raised must still open
The first version of the replica fix treated a raise it could not place as a
hard error, and on a real three-node cluster that turned the fix into the
outage it prevents. A node whose config named a fourth server wanted four
replicas, NATS answered "no suitable peers for placement" for three, and the
daemon retried for its full five-minute budget without finishing bootstrap.
The bucket was present, quorate and serving on three nodes throughout. A
config naming a node before that node serves is the ordinary middle of
growing a cluster, so that is a state to tolerate rather than refuse.
The line is now drawn between the two cases instead of around both. Creating
a bucket below the cluster's replica count still fails, because that is the
defect and a fallback there reintroduces it where it is hardest to notice.
Failing to raise an existing bucket warns and hands the bucket back, and the
raise is retried on every open besides the daemon sweep and --repair. This
matters most in kvlease, which runs on every reconcile tick: refusing there
would stop every reconciler sharing a lease bucket that was working.
getOrCreateBucket now attaches before it creates. A create asking for more
replicas than the cluster can place is refused before the server looks at the
name, so with create-first "this bucket already exists" was an inference from
which error came back, and the wrong one comes back on a single-node server.
RaiseAllBucketReplicas attempts every bucket and joins the failures, and
--repair keeps going too. Stopping at the first would leave the rest of a
cluster short because one bucket could not be placed.
Also adds nats.ServerName/NodeFromServerName beside the template that renders
server_name, with a test tying them together. The kvquorum suite had mapped
stream peers to nodes by comparing them to node names, which they are not:
they are the node name behind a "spinifex-nats-" prefix, so nothing matched
and the node-loss test could not choose a victim.
* test: wait for the right NATS peer count after a node rejoins
/routez reports a node's routes to its peers, not its own membership, so a
fully formed three-node cluster is two peers per node. The kvquorum suite
waited for three and spent its whole five-minute rejoin budget on a cluster
that had already reformed. Every other caller passes len(Nodes)-1; the
helper's doc comment now says so.
The assertion the test exists for had already passed: node1 was leading 18 of
47 buckets when it was taken away, and node2 and node3 each still answered for
all 47.
* fix: replicate KV buckets across the largest odd count a cluster can place
A bucket was replicated across min(nodes, 5) servers, which stores a fourth
full copy on a four-node cluster and still survives exactly one loss — the
same tolerance R3 gives for three quarters of the write amplification. A Raft
majority needs an odd count, so the rule becomes the largest odd count the
cluster can place, capped at five: 1, 3, 3, 5, 5. It is one expression with two
entry points, so no caller can hold a stale copy of the policy and keep passing
after the policy changes.
Three retry loops could not tell a configuration fault from a cluster that had
yet to form. An undeclared cluster size never becomes declared by waiting, so
each of them spent its whole budget on one and then reported quorum as the
cause. They now stop on a permanent error and say what is actually wrong. This
is what the integration suite was hitting: it builds its gateway by hand and
never loads a cluster config, so the size was never declared and the IAM retry
sat in its sleep until the test timed out. Both ends are fixed.
The audit reported configuration alone. A bucket at three replicas with one
peer offline and one lagging is one failure from unavailable and read as ok.
BucketReport now carries the leader and a count of the peers JetStream itself
calls current and reachable, quorum is measured against that count, and
spx admin kv replicas exits 0, 1 or 2 so a deployment gate can tell a problem
--repair fixes from one needing a human.
Declaration is write-once: daemon.Start refuses a conflicting redeclaration
because it is about to create buckets, and it declares before JetStream init
rather than after it. config.LoadConfig warns instead, because it also reads
config files that are not this process's own cluster.
spx admin init --nodes 2 is refused — no replica count on two nodes both
survives a loss and accepts writes — and single-node replication is asserted in
the single e2e suite, so it runs in every nightly host permutation.
* fix: withhold the vLLM serving image until its artifact is published
The catalog offers ubuntu-26.04-vllm-serving-x86_64, and the qcow2 it names
404s on iso.mulgadc.com, so spx admin images import would fail for every
operator who tried it. Its GPU siblings are published; this one never was.
The entry moves to WithheldImages rather than being deleted, because the entry
is the definition — the URL, the checksum and the two tags resolveServingAMI
filters on. Publishing the artifact is then a move back, not a retype.
Bedrock already handles the image not being registered: resolveServingAMI
returns ErrServingAMINotFound, which is the honest answer while there is
nothing to launch.
The bedrock tag contract is still pinned, now against the map the definition
lives in, and it additionally asserts the entry is not offered for import. The
e2e reachability walk covers AvailableImages alone, so a withheld entry is out
of it by construction.
* fix: drop the cluster size from the call sites dev added
Two constructors on dev still pass a replica count that no longer exists as a
parameter: the cluster size is a declared process fact, so EKSServiceDeps has
no ClusterSize field and NewIAMServiceImpl takes three arguments. The merge is
textually clean and breaks the build, which is the shape of collision a
signature removal always has.
* docs: explain what an operator sees when a cluster server fails
The replication section said how many copies a bucket has and stopped there,
which leaves the question an operator actually has during an incident: one
server is gone, what still works. It now covers the majority side carrying on,
an isolated server refusing changes on purpose and why that is the behaviour
you want, guests continuing to run while anything that has to be recorded does
not, the risk of reading state from a server that is cut off, and a returning
server catching up by itself.
The surrounding prose reads from the operator's side rather than the
implementation's: what you see and what to do, not which consensus protocol is
underneath.
CreateRole, GetRole, ListRoles and instance-profile roles each get their own projection; GetRole carries an empty RoleLastUsed. Policies carry UpdateDate and PermissionsBoundaryUsageCount, CreatePolicy drops Description and ListPolicies drops Tags. CreateUser echoes Tags, list instance-profile actions drop profile Tags, and GetAccountSummary gains AWS's seven missing keys and loses SigningCertificatesPerUser.
AWS answers a bad name charset, a path not bounded by slashes and an
enum violation such as UpdateAccessKey's status with ValidationError.
Length and tag violations keep InvalidInput until AWS's code for them
is captured.
AWS percent-encodes every policy document it returns. Spinifex returned
raw JSON, so a client decoding once read '+' as a space and '%41' as 'A',
giving perpetual Terraform drift, and a lone '%' failed the read.
Encode in the gateway wrappers for CreateRole, GetRole, ListRoles,
GetRolePolicy, GetUserPolicy, GetGroupPolicy, GetPolicyVersion and the
instance-profile reads. The service layer keeps raw JSON because STS and
spx admin parse it in-process. The web UI now decodes role inline
policies like every other document.
Tests added since main used in-package test packages, failing the
test-package check. Convert them to <pkg>_test with export_test.go
hooks; the ecs-agent main package keeps an in-package marker.
A request whose body cannot be read to hash failed through the catch-all
branch, logging a malformed envelope, returning IncompleteSignature and
recording an anonymous lockout strike, so ten dropped connections locked
out an IP. Match sigv4.ErrReadingBody and answer InternalError without
recording a failure. Bumps bluebottle for the new sentinel.
* feat: recover instances automatically from a node that stopped heartbeating
A node that dies hard takes its guests with it, and nothing brings them back
until an operator brings that node back. Every daemon now runs the same small
reconciler: it reads the instance records, selects desired-running instances
whose owner is stale, and relaunches them here.
The claim is a CAS on Status.LastNode and that is the whole of the
coordination. Several survivors race one instance, exactly one revision wins,
and the losers find a record naming a live owner and stop considering it. A
claimant that dies mid-launch leaves the record naming a node that is itself
stale, which is the same candidate the next pass derives again, so there is no
lease to expire and nothing to reconcile by hand.
Ownership is not permission to write. The volume leases still decide whether the
guest can move, and VolumeMounter.Mount runs before startQEMU, so an attempt
that loses the volume race has touched no tap, no OVS port and no OVN binding.
A storage failure is not a host failure and wants the opposite response. A guest
paused by werror=stop publishes Health.IOErrorSince, and a record carrying it is
refused: the volumes are the same objects in the same store from anywhere, so
relaunching moves the pause rather than ending it. A node whose own object store
is not answering stands down entirely, which separates a store broken here from
a store broken everywhere without either having to be diagnosed. A launch that
does not complete backs this node off that instance, doubling to a cap, because
nothing on the mount path can yet say which of the two it hit.
Recovery is off by default and inert below three nodes. It also waits until it
has seen its peers live, or five minutes, so the first node back from a
coordinated shutdown does not recover guests from nodes that are still booting.
* test: prove instance recovery moves a host failure and refuses a storage failure
The e2e half of automatic instance recovery, asserted from both sides because a reconciler that cannot tell the two failures apart passes one and fails the other. InstanceAutoRecovery takes a node down and requires the guest back on exactly one survivor, identified by the qemu process rather than by what the API reports. InstanceRecoveryRefusesStorageFault stops predastore cluster-wide first, so the guest is paused by werror=stop when its node goes away, and requires that nothing moves.
Also fixes the settle window, which re-evaluated every pass and so held a daemon back for its first five minutes over exactly the failure the reconciler exists to handle. It now latches once every peer has been seen live.
harness.InstanceHostingNodes replaces the single-node lookup so a caller can assert there is exactly one host rather than assume it, and skip a node it has deliberately taken down.
* fix: make automatic instance recovery a cluster-wide setting
Recovery was read from the per-node Config, so it had to be written three times on a three-node cluster, once inside each [nodes.<name>] table. A top-level [recovery] table — the obvious thing to write, and what the e2e suite writes — was parsed into nothing and the daemon reported recovery disabled on every node with the setting plainly in the file.
Cluster-wide is also the honest shape. A node with recovery off is not opting out of anything: recovery is what survivors do to a dead node's instances, so the setting belongs where there is one of it.
* test: drive real I/O into the storage-fault case, and freeze the store rather than stop it
Measured on a three-node cluster: predastore stopped cluster-wide for twelve minutes against idle guests paused none of them. Viperblock serves what it has cached and an idle guest issues nothing that reaches the object store, so the test asserted that nothing moved in a run where nothing was ever faulted.
The victim now launches with cloud-init user-data that writes to its root volume continuously. User-data rather than SSH because a routed-NAT cell gives guests no public address, so a test that had to log in could not run on half of them.
The fault is SIGSTOP rather than a unit stop, for the reason the storagefault suite already gives: a frozen process holds its connections open and answers nothing. Under that fault and that load the first BLOCK_IO_ERROR arrived 4m47s later, so the pause budget is now ten minutes rather than three.
* test: give instance recovery its own suite and its own nightly cell
Both tests take a node away and one of them freezes predastore cluster-wide, so anything sharing the environment reports that outage as its own failure — the same argument storagefault already makes for being its own package. They are also slow by nature: a dead owner has to be seen stale twice, a guest has to boot, and the storage case waits for a pause measured at 4m47s and then holds four minutes to prove nothing moved. Roughly twenty minutes on top of a permutation cell budgeted at thirty-five.
Cell 31 is three-node and static, matching storagefault's, with a 75-minute job budget over a 60-minute suite timeout. The suite is eligible on multi-node environments and deliberately not in the nightly permutation set.
* fix: give a guest long enough to open a volume the object store is slow to serve
QEMU opens every drive before its monitor answers a greeting, so the QMP
greeting deadline is really a deadline on opening the guest's volumes. At 30s
it was shorter than a cold viperblock open costs on a cluster that has lost a
node, where every object read is served degraded: measured at 35s with one node
of three away. The launch killed the guest five seconds short of ready and then
did it again on every retry, which is a recovery that can never complete.
A network-backed drive now gets its own deadline for the same reason VFIO
already had one — synchronous work before the monitor answers — and the
instancerecovery suite is what found it.
Also make the suite's predastore thaw best effort per node. It runs after a
node has been taken away, where there is no process to signal, and a node it
cannot reach must not stop it restoring the rest.
* docs: name the two newest cells in the dispatch input
The cells input is the operator's index of what a number means, and it stopped
at 29 while 30 and 31 were added.
* test: prove the guest is writing before freezing the store in the recovery suite
The storage-fault test froze the object store as soon as RunInstances reported
running, which is when QEMU has launched rather than when the guest has booted.
On the nightly cluster the freeze landed while the guest was still in UEFI, so
the only drive it could reach was the EFI varstore — and pflash has no
werror=stop to pause on, so the guest never paused and the test reported a
product failure it had caused itself.
The load driver now prints to the serial console after each pass it has written
and synced, and the test waits for that before freezing. The console is read
through GetConsoleOutput, so this still needs no login and still runs on cells
whose guests have no public address.
* feat: recover instances automatically with nothing to configure
The switch is gone. An instance whose host has failed is down whether or
not recovery is enabled, so the only outcome a disabled reconciler can
produce is leaving it down — which is the state recovery exists to end,
not a safer one to fall back to.
Against that, the switch cost a cluster that can be configured into
never recovering anything, with no symptom until a node dies. This
feature already hit that once: a top-level table parsed into nothing and
every node reported recovery off with the setting plainly in the file.
What makes recovery inert is now only facts about the cluster — no
cluster config, no JetStream, or fewer than three nodes — and each says
in the log which one applied. The e2e no longer writes config or
restarts daemons: it waits for every node to report the reconciler
active on a cluster nothing configured, so reintroducing a switch fails
there as well as in the unit tests.
* feat: stop retrying a recovery that is never going to work
Every failed attempt used to go to the same exponential backoff and stay
there. An instance nothing could launch was retried every fifteen
minutes forever, still observed-running on a node that is gone, with no
account of it anywhere the customer or an operator would look.
A failure is now classified and counted. Two classes are read from what
the launch actually reports: the node it was taken from still holds a
volume lease, which clears on that lease's clock and is retried on a
faster one, and this node has no room, which another survivor may. There
is no terminal class, because nothing on this path can prove one — the
launch collapses everything else into ServerInternal, so a verdict of
"never going to work" would be a guess, and the guess that abandons a
recoverable instance is the expensive one.
The budget bounds the unknown case instead. Every class counts against
it, since a class that did not would be the old unbounded retry under a
narrower condition. Exhausting it leaves the instance where AWS leaves
one whose host failed and could not be brought back: stopped, carrying a
StateReason that names the cause and the last error, one StartInstances
from running again. Writing the desired state down is also what makes
one node's decision cluster-wide, because selection reads desired state
— the same property the claim already has.
A record another node has since claimed is never touched, so a node
exhausting its attempts cannot stop a guest that is now running
elsewhere.
* fix: give a guest the time for every cost it pays before its monitor answers
A GPU instance booting from a network-backed root volume pays both
costs: QEMU opens every drive across the object store and then pins the
whole of guest RAM, one after the other, all before the monitor answers.
The deadline took whichever was larger, so such a guest was given less
time than it needs.
That the two floors are both 180s is why nothing has been seen yet — the
VFIO branch has always returned at least the network-drive deadline. It
is the same defect either way, and the symptom when it lands is a GPU
instance SIGKILLed on every attempt while its plain sibling launches,
which reads as a GPU fault rather than as a deadline.
The costs are now added, the plain QEMU startup is counted once rather
than replaced, and the cap covers both floors together so it cannot clip
a guest before either cost has been waited out.
* test: assert the AWS-visible arc of a host failure end to end
A customer cannot see which host an instance is on, and AWS never tells
them. What they can see is DescribeInstanceStatus, so that is where a
recovery has to be provable, and the suite never looked at it.
Three frames are now required, in order: running with system-status ok
before anything is broken, system-status impaired while the owning node
is gone, and ok again once a survivor is running it. The middle frame is
the one worth having — the window opens when the owner's heartbeat goes
stale, which is strictly before any survivor may claim, and closes when
the relaunch registers, so it is bounded rather than lucky. Reporting ok
throughout would be the more comfortable lie.
The availability zone is compared across the first and last frames as
well. A recovery that crossed one would break every placement promise
made to the customer, and nothing else in the suite would have noticed.
* test: prove a half-healthy node fences itself and is never a second writer
The two existing recovery cases take a node away by stopping its services,
which is the failure this feature handles least interestingly: a node that is
gone has no opinion about who owns its volumes. The one that matters is the node
that is still up, still running QEMU, still answering its own local clients, and
simply cannot reach consensus. That is the state in which two nodes can each
believe they own a volume, and the only thing between that and a corrupted disk
is that the partitioned node gives up first, on a clock it can evaluate without
reaching anything.
So this drops NATS on 4222 and 4248 between one node and its peers and nothing
else, then asserts the ordering the safety argument rests on rather than the
outcome it produces. A watch samples every node every three seconds for the
whole run, including the heal, and fails on any single sample naming two hosts.
Two guests writing one volume through one object store is data loss already
under way, not a race that resolves, and no point-in-time assertion can see one
that appeared between its own samples.
The injector takes its own nft table rather than touching inet spinifex_filter,
because a test that flushed the node's real firewall would leave it either
unfirewalled or misfirewalled with no sign of which, and a separate table at
priority -300 heals in one atomic delete. A transient systemd timer removes it
independently, so a cancelled workflow does not leave a node cut off from its
cluster until somebody notices. It proves SSH still works and proves a peer port
is genuinely unreachable before anything is asserted, since a rule set that
installed cleanly and dropped nothing would read as the system tolerating a
partition.
The suite budget goes to 90m and cell 31 to 105m to fit the third case.
* fix: move an instance's boot state, volumes, logical port and address with it
A record-driven launch assumes the node reading the record is the node that wrote
it. On a daemon restart that holds. On a recovery onto a survivor it does not, and
five things follow from it — one that stops a moved instance booting at all, and
four that stop it running anywhere.
A load balancer never comes back from a host failure. Its boot configuration is a
set of files under the runtime directory and the record holds their paths, so the
survivor restores paths that exist only on the node that died. The hook that
rebuilds them was reachable from Restore alone, and the cross-node path goes
straight to vmMgr.Run — so QEMU is handed a netcfg, an agent env and a CA cert
that are not there. The gateway URL baked into that env names the creating node's
management bridge as well, which the rebuild corrects because it rebuilds from
whichever node is running it. Firing the hook on both paths fixes both, and a
rebuild that fails now refuses the launch: a guest that boots without its network
configuration is worse than one that does not boot, because the records would say
it recovered.
The export outlives the guest and holds the volume lease. A returning node is
healthy, so it renews that lease indefinitely and the new owner never acquires:
the guest is stopped here and refused there, recovery spends its attempt budget on
a volume that was never going to be free, and gives up on an instance that was
never unrecoverable. So the export is given up too, over a new node-addressed
request that tears it down exactly as the fence does. Deliberately not an unmount
— an unmount seals, this node's block map is the stale one, and sealing publishes
it over the winner's. Unlike the fence it deletes the lease key rather than
leaving it to expire, because the lease is still this node's to hand back.
The tap outlives the guest, and the tap is what claims the instance's OVN logical
port. The binding names one chassis, so two nodes offering the same iface-id do
not split the traffic — they contend, and the address answers from whichever won
last. It also blocks the layer above: the external IP's host ingress is pruned on
the evidence that the owning guest sits on another chassis, and that stays false
while this node is still claiming the port.
The management address outlives the guest as a cluster-wide reservation, and the
release was unconditional. br-mgmt is one flat L2 segment, so the node an instance
left was freeing an address still in use and offering it to the next instance
anywhere. Only the node an entry names may release it now, with the takeover
recorded on arrival so the node actually running the instance can still give the
address back. An entry written before the node field existed names nobody and
stays releasable, because there is no other holder to collide with.
The fence path already cleaned up the tap, so the release fix repairs that path
too.
* test: prove a load balancer keeps its name, its address and its traffic when its host dies
Takes away the node running an internet-facing ALB and asserts the four things a
customer of one actually holds: the name keeps resolving, the address behind it
does not move, requests through the name are answered again without anybody being
called, and the backends are healthy again from the recovered load balancer's
point of view.
The backends are placed around the load balancer rather than beside it. A load
balancer lands wherever a node answers its queue group, so its node is discovered
and the backends are relaunched until both are elsewhere; one on the node about to
be taken away would be recovered too, and the run could not then tell an ALB that
came back serving from one that came back with nothing to serve.
DNS is asked of a node's own northstar over the wire, and the HTTP probes resolve
through the same server, so the claim under test is what this cluster answers for
a name rather than what the machine running the tests was configured to forward
to. A watch samples every surviving node every five seconds for the whole run,
because an answer that went missing or pointed elsewhere for one reconcile pass is
the outage, and no single sample reaches it.
The returning node then has to give up what it no longer owns: no second copy of
the guest, and no tap still claiming the load balancer's logical port, since OVN
binds a port to one chassis and a second claimant contends for it rather than
splitting the traffic. The last assertion is that the name still serves with that
node back in the cluster.
* docs: name the new cell in the nightly's cell list
* test: stop the disk-load guest from filling the console it is judged by
The load generator announced every write, once a second, for as long as the
test ran. The console is a ring buffer this suite reads back, so an hour of
that pushes out everything explaining a failure, including the kernel's own
account of the I/O error the test exists to produce. Announce twenty times,
then keep writing quietly.
* docs: say what the OCI guide has been proven on, and what a host failure does
The guide read as a support statement for OCI rather than as an account of one
tested path, promised seven steps and then gave eight, and said nothing about
what happens to a guest or its public address when the node under it fails.
State the evidence, state that the nightly guards the datapath and not the
cloud, and give the failure behaviour as a table with the reason codes a
customer will actually see.
Also say plainly what the volume lease is: one running guest and a bounded,
logged window, not a write barrier the object store enforces. Someone sizing a
workload against this should read that before they read the table.
* docs: say why the mount classifier refuses to retry a lease another node holds
It reads as an omission — a lease held elsewhere is the most transient refusal
there is. It is not. This classifier drives the restore path, where the holder
is a node that is running the guest, so retrying would relaunch against a live
owner. Recovery wants the opposite and reads the refusal itself.
* feat: claim a recovered guest's public address in line, and verify it arrived
A recovered instance was reported running before anything moved its public
address off the node that died, so for up to fifteen seconds the records said
recovered and the address answered nothing. The recovery path now runs the
affinity pass itself, which is idempotent and is the same pass the ticker runs.
Two things the pass could not do before. It abandoned every remaining address
on the first API failure, though a throttle or a denial is per-object and the
guests behind the others are just as dark. And it trusted the move's own
response, which is OCI agreeing to the request rather than OCI having done it,
so an accepted move that never landed read as success. It now re-reads what OCI
delivers here and reports the difference.
An address that will not arrive makes the guest a system-status failure with the
time it went dark, the same shape AWS uses for a host-level fault. The guest
passes every check it can be asked about itself, which is exactly why reporting
it healthy was the worst available answer.
* feat: give a recovery attempt one trace across all three of its phases
The volumes opening, QEMU starting and its monitor answering already trace
themselves, but nothing tied them to the attempt that asked for them: three
orphan traces per recovery, no way to ask which phase a slow one spent its time
in, and nothing naming the node the guest came from. A span over the attempt
makes them its children, and the reconciler's own log lines carry its trace ID
because the handler stamps it from the context.
A failed attempt records the fault and the attempt count on the span, so the
budget running out is readable as a sequence rather than as twelve unrelated
warnings.
* fix: finish a fence once its writer finally exits
A kill that did not take effect was logged, counted and dropped. The export
stayed registered and the lease stayed held, which is correct while the writer
lives and wrong forever after it dies: a lease a healthy node keeps renewing
never expires, so the volume could not start anywhere.
The failure is now re-reported every 30 s until it resolves, and the teardown
completes as soon as the writer is confirmed gone. Only a delivered kill is
waited on — a refused one says nothing about what became of the writer.
* test: stop judging a detached stop or terminate on a transient state
Both tests read the instance state immediately after the ack, but the transition
runs on its own goroutine, so the read races it in both directions: too early
sees running, too late sees the final state. The race gate caught the first of
those on a loaded runner.
* fix: replicate the volume lease and dirty buckets across the cluster
Both were created with the default single replica, so both lived on one node.
That node holds the answer to "who may write this volume" and the marker every
mount reads, and losing it takes volume leasing and mounting down cluster-wide.
A partitioned node was fenced correctly and then no survivor could relaunch its
guest: every attempt failed on "read dirty mark: no responders available", for
ten minutes, until the budget ran out.
The replica count is now the cluster's node count, as the gateway's own buckets
already do. Existing buckets are raised in place rather than requiring a wipe.
The e2e diagnostics that should have said this are fixed too. The fence and the
surrender are logged by the storage service, not the daemon, so the assertions
read the wrong journal; and Eventuallyf evaluates its message at the call site,
so a ten-minute wait was explained by a log that stopped before it began.
spinifex/cloud/oci imports oci-go-sdk directly, but go.mod listed it as indirect
and go.sum was missing three of the modules it needs — sony/gobreaker/v2,
gofrs/flock and youmark/pkcs8. The module therefore only type-checked inside the
go.work workspace, whose go.work.sum happened to carry them, so a standalone
checkout failed to build and the linter reported it as a typecheck error against
an unrelated file.
* test(e2e): make the workbook teardown a gate, not a warning
A customer who cannot remove what they created is as stuck as one who could
not create it, so the teardown is part of what a workbook proves. It logged
WARN and passed the suite regardless, and a bucket that could never be emptied
went unnoticed for weeks behind exactly that line.
The failure it hides is a wedge rather than an error: the AWS provider retries
DeleteBucket against a bucket it cannot empty indefinitely. So the run is
bounded by DESTROY_TIMEOUT, defaulting to 900s, and the last forty lines of its
output are printed when it fails. The cleanup after a failed apply is bounded
the same way for the same reason.
* test(e2e): print a workbook's PASS after its teardown, not before
The verdict now depends on the teardown, so a workbook that built everything
correctly and then could not delete it printed PASS and FAIL for the same run.
A renewal that times out is treated as transient and keeps the lease, correctly,
but the server may have applied it anyway. It is then a revision ahead of what
the holder recorded, so every later conditional update is refused for that
reason alone — and a refusal was read as a peer having taken the volume. The
export was fenced, nbdkit killed and the guest lost its disk, with the log
naming a winner of "unknown" because no peer ever existed. A ten-second
JetStream stall destroyed four running guests this way.
A rejected update now re-reads the entry. One that still carries this lease's
owner and generation is this node's own write coming back, so the revision is
adopted and the lease continues; anything else, or an entry that has gone, is a
real loss. A re-read that cannot be answered keeps the lease without extending
it, because local validity is the bound that depends on reading nothing.
The rule is stated in its own test: a revision mismatch may not fence before a
re-read says the entry is no longer ours, and lapsed local validity always
fences whether or not a winner can be named. The weaker rule — that an
unnameable winner never fences — would leave a partitioned node writing for as
long as it stayed partitioned.
The AWS provider reads a bucket's tags with `ListTagsForResource` on S3 Control when it knows the account ID. S3 Control puts the account in the hostname — `https://{account}.{endpoint}` — so against an IP-addressed endpoint the request fails a DNS lookup before a packet leaves, and no endpoint configuration can help. A tagged bucket therefore fails its first read after being created.
`skip_requesting_account_id` alone does not fix it: `aws-sdk-go-base` takes the account ID from the STS `GetCallerIdentity` credentials check before it consults that flag, so `skip_credentials_validation` is needed alongside it. With no account ID the provider falls back to `GetBucketTagging`, which predastore serves.
`s3-webapp` is the only workbook that creates a bucket, and it uses no `aws_caller_identity` and references no other security group, so a blank account ID costs it nothing.
The bucket also gets `force_destroy`, since the webapp uploads into it and a destroy of a used bucket fails `BucketNotEmpty` without it.
The e2e harness now asserts the bucket carries the tags Terraform set, which is what catches this regressing, and no longer aborts the remaining workbooks on a failure — `s3-webapp` failing says nothing about whether RDS works, and four nights of `rds-quickstart` coverage were lost that way. `STOP_ON_FAIL=1` restores the early abort.
Every workbook's provider versions are pinned, so a run reproduces rather than picking up whatever the registry published that morning.
* feat: ship uninstall-spx.sh and fix the OCI bring-up path
Adds scripts/uninstall-spx.sh, the inverse of setup.sh: it removes the
software where node-reset.sh clears state and leaves the node installed. An
operator who installed with `curl ... | bash` had no teardown path at all.
State teardown is delegated to node-reset.sh rather than reimplemented, and
setup.sh now ships both to /usr/local/share/spinifex so they are reachable
without a checkout.
It refuses on a multi-node cluster and --yes does not override that: a flag
meaning "don't ask me" must not also mean "ignore a safety property".
/var/lib/spinifex and every apt package are kept unless asked for, and both
opt-ins confirm separately even under --yes.
Proved by a full uninstall/reinstall round trip on dev-oci, which found two
bugs neither prod nor dev-prod can reach, because both are ISO-installed:
node-reset.sh exited 2, silently, at its last step and after the wipe.
setup.sh creates the spinifex *group* and the spinifex-* service users but
never a *user* named spinifex, so `getent passwd spinifex` exits 2 and
pipefail carried it into set -e.
The uninstaller broke openvswitch-switch on its first run by removing the
socket-perms helper while leaving the drop-in referencing it, failing the
unit at step EXEC with status 203. Fixed on both sides: the uninstaller
removes drop-ins it wrote into other units' namespaces and restarts anything
it failed, and setup-ovn.sh writes ExecStartPost=- so a missing helper can
never stop another package's daemon.
setup-ovn.sh also learns the shape OCI needs: when --wan-bridge names a Linux
bridge it links it to OVS with a veth pair instead of enslaving the NIC, so
the VNIC keeps its MAC identity and netplan keeps owning the addresses. It
now fails on OVS datapath errors instead of reporting success, and the
--management demotion guard keys on /etc/default/ovn-central rather than on
database presence, which the ovn-central package creates on every node.
Tests: scripts/uninstall-spx_test.sh, wired into make test-build-scripts
along with shellcheck over the four scripts that ship to every node.
* feat: allocate external IPs from the Oracle Cloud API
An OCI VNIC drops any packet whose source is not a registered private IP
object on it, so an external pool cannot be range math there — the address
has to be created through the provider before it exists at all. Add
source = "oci" and the allocator behind it.
One allocation is two OCI objects: a secondary private IP that rides the
wire, and a RESERVED public IP attached to it that the internet reaches.
Allocate returns the public address because that is what AWS semantics are
about, and records the pair so the host plumbing and release path can find
the private half.
The OCI objects are created before the binding is written, and dropped
after it on release — both orders chosen so a crash leaves a discoverable
leak rather than a record that lies. Reconcile squares the two on startup
and is required, not optional: a leaked reserved public IP bills and holds
one of the 50 per-region slots with nothing referencing it. It never
touches an object without our display-name prefix, because the reference
host carries operator-created secondaries on the same VNIC.
Resolve the VNIC from an interface name by MAC through IMDS rather than by
Oracle's VNIC ordering, which is not a documented contract.
Generalise ExternalIPAM.EnableDHCP into InstallAllocator so the OCI wiring
is not a second single-purpose installer.
* fix: refuse source = "oci" outside routed mode, and cover the SDK path
An OCI VNIC accepts exactly one MAC, its own, and delivers every inbound
frame to it regardless of which of its addresses the packet is for.
Measured on the reference host: an internet source reaches 10.200.0.144 as
router-MAC to VNIC-MAC, the public address never appears on the wire at
all, and every off-subnet destination resolves to the same next hop.
Pool mode puts a per-VPC gateway LRP MAC and a per-rule external NAT MAC on
the uplink and ARPs for the address, which policy.go:50 justifies with "the
Linux bridge learns the per-rule external MAC on the veth port like any
other MAC". An OCI VNIC does not learn MACs; it accepts one. So a pool-mode
guest's egress is dropped and its ingress never reaches OVN. Routed mode
keeps the address on the host, which already wears the VNIC's MAC.
Pinning every LRP to that MAC does not rescue pool mode: they share one
external logical switch and OVN resolves inbound frames by an L2 lookup on
it, so duplicate MACs work for one VPC and silently misroute past that.
The refusal names the MAC reason, because a refusal that only says "use nat
mode" reads as arbitrary and gets worked around.
Also test the SDK request shapes against a local endpoint. Two of them are
wrong in ways reading does not catch: pagination stopping early reports a
partial VNIC to reconcile, which then drops live bindings, and a nil
privateIpId on detach is omitted from the body entirely, so OCI reads the
update as a no-op, returns 200, and leaves the address attached and billing.
* fix: install host firewall rules where a distro's catch-all reject cannot precede them
Oracle's Ubuntu cloud image ships a catch-all REJECT in both INPUT and FORWARD.
Two Spinifex datapaths were silently dead behind it on any such host.
EnsureNATEgressRules probed with -C and appended with -A, so its ACCEPT rules
landed after the REJECT and never matched: 24113 packets on the reject, 0 on
both accepts, and guests had no egress. The comment named the case it did not
handle, anticipating a FORWARD policy of DROP — which an append does survive —
rather than an explicit REJECT rule, which it cannot. Insert at the head, and
drain existing copies first, because a rule an earlier build appended still
satisfies -C and would otherwise stay stranded across an upgrade.
Nothing opened INPUT for the IMDS endpoints at all. The responder binds .254
and .253 per instance on the ime- devices, so guest requests are locally
delivered and meet INPUT. Requests arrived and no reply ever left; cloud-init
spent 490s timing out and reported the endpoint as disabled. With one wildcarded
accept the same image boots in 18.6s and finds its datasource.
The natuplink e2e check had the same blind spot the code did, asserting -C and
therefore presence but never position. It now asserts the accepts precede the
first reject.
* feat: OCI address pairs, and an IMDS remap so the host keeps its own metadata
An OCI external address is two objects, not one: a RESERVED public IP that
OCI 1:1-NATs upstream and never puts on the wire, and a secondary private IP
on the VNIC that the wire actually carries. Every AWS-facing record holds the
public half; every datapath object must hold the private one.
The mapping goes in policy.natManager and nowhere else. AddEIP, DeleteEIP,
PruneOrphanEIPs and the HostEIPBinder all converge there, and reconcile
rebuilds the rule set from AWS-facing intent on every pass — so a correctly
mapped rule installed further upstream is swept on the next reconcile and
replaced with one matching an address that never arrives. The seam is a
DatapathResolver defaulting to identity, so every non-OCI deployment is
unchanged. A resolve failure is fatal: falling back to the public address
would install a rule that reads correctly in ovn-nbctl and blackholes.
ocinet.Lookup is the OCI side — read-only, KV-backed, no client and no
credentials, because vpcd is a different process from the daemon that owns
the allocator and the bindings record is all either would consult.
Two live bugs the in-memory test store was hiding, both found by making the
fake report ErrNotFound the way the real store does:
- KVStore.Mutate could not create, so every pool failed its own first
allocation after OCI had already created and billed the pair.
- Reconcile treated a missing record as a failure, aborting the one pass
that exists to find an object created before its binding was written.
Guest IMDS must stay on 169.254.169.254, which is also where a cloud host's
own metadata service lives — the first guest launch made it a local address
and took the host's metadata and DNS with it. The datapath is OpenFlow
steered, so the guest-facing addresses stay put and only the host-side
endpoint address moves, with a PREROUTING DNAT on ime-+ translating and
conntrack un-NATing the reply. Off by default; identity when unset.
The IMDS firewall rules move to vpcd. They are wildcarded over every
endpoint rather than per-launch work, and spinifex-daemon holds no iptables
grant and no CAP_NET_ADMIN by design, so attempting them aborted every
launch.
* fix: name the real constructor in the IMDS bind-address error, and mark the NAT datapath test in-package
The error string pointed at DefaultHostBindAddrs, which does not exist — a
caller following it would have looked for a symbol that was never written.
nat_datapath_test.go reuses nat_test.go's in-package fixtures, so it carries
the //test:in-package marker the package check asks for.
* docs: index what each e2e suite proves
Written because a datapath comment was read as an end-to-end behavioural claim
and the suite that already answered it was found only after the wrong answer
had been given. TestVPCEgressPaths/NATBaseline is the authority on private
subnet egress, and nothing pointed at it.
* fix: source-route an EIP's reply out the VNIC that holds it
An EIP's reply carries the EIP as its source. On a host that pins its
uplink with a policy rule, the reply took the main table instead and left
by the wrong interface — which a cloud VNIC drops, because the address is
registered on a different one. Ingress reached the guest and nothing came
back.
Every bind now mirrors the uplink's rule onto the EIP. The selector is the
rule rather than the interface: both OCI VNICs sit in one subnet, so "the
interface owning the EIP's subnet" names two of them, and two candidate
rules is a refusal rather than a guess. A host with no source routing gets
no rule and no change.
* docs: name the setup-ovn recipe routed mode actually needs on OCI
The OCI guide told operators to run `--wan-bridge=br-wan`, which is
mutually exclusive with `--nat-uplink` and deletes the spx-nat transit
veth routed mode runs on. The node then refuses to bring the uplink up,
with an error naming the flag the guide should have used.
Several messages also claimed routed mode is outbound-only with no
public IPs. That stopped being true when the routed EIP path landed, and
it reads as "what you are about to do cannot work" during the one init
the OCI path documents.
* feat: let routed NAT form a multi-node cluster, and carry every pool through formation
Lifts the --nodes >= 2 refusal on --external-mode=nat. The refusal carried no
technical justification -- it was a v1 scope decision -- but routed NAT is the
only datapath a cloud VNIC can carry, so "routed NAT is single-node" had become
"Spinifex on a cloud host is single-node".
It is not fixed by lifting it. Measured on three OCI nodes: the cluster forms,
all chassis register and guest-to-guest traffic runs at 7.1-7.7 Gbit/s, but
every node claims every EIP, nodes disagree on the VPC's gateway LRP, and only
the gateway chassis reaches its own guests. init now says so on stderr rather
than refusing, because the refusal made the defects unreachable and therefore
unfixable. Distributing the EIP datapath is the actual fix.
Lifting it exposed a second defect that stops formation outright. NetworkConfig
held nine flattened Pool* scalars and the init node set PoolName = "wan"
unconditionally; routed NAT renders two pools and the one that matters --
nat-transit, carrying the gateway-LRP range -- was dropped. Each joiner got a
single pool named "wan" holding the transit gateway and the detected WAN prefix,
and the leader re-rendered its own config from the same struct, so all three
nodes were wrong. Pools is now []admin.PoolData in render order, and PoolData
gains json tags because it is on the wire now rather than a template input.
* feat(install-node): add --external-mode so a cluster can be formed in routed NAT
install-node.sh could not express routed NAT at all. It required
--external-pool/--external-gateway/--external-prefix-len and passed
--external-mode=pool to every spx admin init unconditionally, so the only
supported way to form a routed-NAT cluster was by hand.
--external-mode nat refuses the three pool flags rather than ignoring them, and
threads --nat-uplink into setup-ovn.sh -- without it the WAN interface check
rejects a physical NIC during formation, which is correct in pool mode and
wrong in routed mode, where nothing is bridged.
* fix(ocinet): stop each node deleting its peers' OCI addresses on startup
Reconcile compares the binding record against the private IPs on one VNIC. The
record is one key per pool and therefore cluster-wide, so on more than one node
every peer's binding looks exactly like a binding whose objects OCI has lost.
Each node dropped them, and direction 1 on the owning node then collected the
now-unclaimed objects as leaks -- deleting the reserved public IP and private IP
of a running instance.
Observed on the three-node OCI cluster: node2's startup pass logged "collected a
leaked address" for an address it had allocated ten minutes earlier and "dropped
a stale binding" for one living on node3's VNIC.
The damage did not stop at the OCI objects. With the binding gone,
Lookup.DatapathIP returns the public address unchanged, so the OVN
dnat_and_snat row carries the public address instead of the private half. Guest
egress then leaves the VNIC sourced from a reserved public IP, which OCI drops
because it enforces the registered source per VNIC -- and EnsureEIPSourceRoute
no-ops too, since a public address shares no subnet with any host address. One
destroyed binding takes out both directions.
Binding.VNICID already exists for exactly this ("how a node that comes back from
the dead tells mine from moved on without me") and Reconcile never consulted it.
A binding is now only dropped when it names this node's VNIC; one naming no VNIC
cannot be attributed and is left alone rather than guessed at.
* fix(host): find the EIP's source-routing table when the uplink is held as a /32
uplinkSourceRuleFor identified the interface owning an EIP's subnet by finding
a host address configured with a prefix that contains it. On OCI both VNICs sit
in one subnet, so br-wan has to hold its address as a /32 -- a /23 there puts
the subnet's on-link route in the main table at metric 0 and takes the whole
subnet off the primary VNIC. A /32 contains nothing but itself, so no rule was
ever found, no source rule was installed, and the reply left by the primary
VNIC where OCI dropped it for carrying an unregistered source.
The address is not the only record of who owns the subnet. The rule names a
table, and that table holds the on-link route. Consulted as a fallback, only
when the address prefixes matched nobody, so a host that already resolved this
the direct way keeps its existing answer -- prod and dev-prod are unaffected.
Only scope-link routes count. A default route covers every address and would
make every source-routing table an equally good answer.
* docs(oci): correct the multi-node story and name the oci_vnic_id constraint
The nat-mode warning said EIPs and default egress were both broken on more
than one node. Default egress is fine: delete a guest's dnat_and_snat row and
it reaches 1.1.1.1 in 2.3 ms from a guest on any of three nodes. Only the
public-IP path was broken, and the three defects behind it are now fixed.
What remains is not a bug but the shape of the datapath: routed NAT centralises
the EIP path on the VPC's gateway chassis, ext-shared is a localnet so the other
nodes cannot reach it, and OCI drops any frame whose source is not registered on
the VNIC it leaves by. So every node's pool must pin oci_vnic_id to the gateway
chassis's VNIC rather than resolving its own. Say so in §5.3, where an operator
writing the pool will read it, and in §9, where the cost is that a gateway
change kills every public IP until the pools are repointed.
The single-node cap itself is gone from the doc: a three-node cluster has been
formed and measured, and the warning proceeds rather than refusing.
* feat(nat): distribute the routed-NAT EIP path so it works on more than one node
Routed NAT processed every EIP on the VPC's gateway chassis, and ext-shared is
a localnet, so the transit segment a /32 route pointed into existed only on
that one node. Measured on three nodes: all three claimed all three EIPs, the
nodes disagreed on the gateway LRP, and only guests on the gateway chassis were
reachable.
The mechanism to fix it is not new. Pool mode already distributes EIPs by
putting external_mac and logical_port on the dnat_and_snat row and letting
OVN's ARP responder answer for the external IP on the external segment. Routed
mode has an external segment too, the spx-nat veth pair, and it being per node
is the distribution: only the node running the instance can answer ARP on it.
So a routed rule is now distributed on the same terms as a pool one, and its
host route becomes on-link on the transit veth instead of via a gateway LRP
that may be on another chassis. bindHostEIP stops demanding that address for a
distributed rule, which is what previously failed the bind on every node but
one. vpcd plumbs only the ENIs local OVS reports as its own, asked of OVS
rather than the Southbound DB because the question is about this host.
The gateway LRP keeps its transit-pool address. Default egress routes through
it and works across three nodes today, so making it link-local as pool mode
does would break a working path to fix one distribution has already fixed.
Pool and static are unmoved, and a test asserts that against the same input
rather than leaving it to inspection. A rule with no MAC stays centralised, so
a cluster part-way through an upgrade is never in a state neither shape handles.
Beads: mulga-opyl7, mulga-la0pe
* fix(nat): make routed-EIP host plumbing work on every node
Three guests on three OCI nodes, one guest per node: only one was
reachable. Two defects, both invisible on a single node.
The transit veth took a random MAC per node, but OVN holds one MAC
binding for the transit nexthop across the whole cluster — so every
chassis addressed its egress to whichever node seeded the binding, and
the other kernels dropped it at L2. The pair is node-local, so one
locally-administered address everywhere is unambiguous; setup-ovn.sh
pins it at creation and in the netdev unit, and EnsureUplinkPort repairs
a veth made before that.
Host routes and proxy-ARP also rode the leader-gated reconcile pass, so
a node that never won the lease never plumbed its own guests' EIPs.
BindHostEIPs does that half on its own loop on every node; Bind already
leaves alone an EIP whose ENI is elsewhere, so two nodes never contend.
An unresolvable address skips the sweep rather than shrinking it.
Pool and static are untouched: the new pass is a no-op outside routed
mode, and a test asserts it.
Also adds --teardown and --dry-run to setup-ovn.sh, for re-running it on
a node that is already tangled. It deliberately spares the NB/SB
databases, /etc/spinifex, the system-id and the packages, and refuses to
del-br a Linux bridge.
* fix(nat): assert the transit MAC where vpcd actually runs it
EnsureUplinkPort is never called in production — host.Wiring is
constructed nowhere — so the repair landed in a path the deployed
daemon does not take, and three nodes came back with three different
MACs. vpcd re-asserts it beside the NAT egress rules instead, which is
the routed-NAT branch that does run on every start.
* fix(vpcd): make uplink-mode detection say what it saw, and check it
detectBridgeMode logged only the branch it took, so a node detected as
veth when it was wired for nat looked identical to one that was right.
It now reports both probes every time, errors when a node carries both
uplink veths (set up twice, and --teardown is the way out), and errors
when external_mode and the wiring disagree — the datapath follows the
wiring, so the config silently loses that argument. An explicitly
configured bridge_mode gets the same cross-check, since nothing else
ever compared it against the wiring.
The vpc-networking doc now opens nat with "start with pool unless one
of these applies" and gains a section on nat across more than one node;
the admin warning for --external-mode=nat on two or more nodes says
what was verified by hand and what is still uncovered.
* test(nat): prove routed NAT delivers each node's own guests, on a cluster
natuplink ran single-node only, which is exactly where the two defects
that broke three nodes are invisible. It now admits ModeMultinode, and
phase 9 takes a second shape there: a guest per node in a spread
placement group, and for each one — its public IP plumbed by the node
running it and by no other node, the route on-link with proxy-ARP
answered there, a dnat_and_snat row carrying external_mac and
logical_port, a TCP handshake in from another node, and the guest's own
console verdict for egress. Every node's transit veth is checked
against the one cluster-wide MAC before any guest boots.
Single-node phase 9 gains the same row and on-link assertions, so the
shape is held on both topologies.
Two gates come from what three OCI nodes actually do: every node holds
a gateway_chassis row for every VPC (the active one is the highest
priority) and every node installs the VPC ingress route, so only phase
7 needs gating — it runs where the gateway port and the guest are both
local, and logs which one is missing otherwise.
Cell 20 is real-multi x nat with its own nightly step, leaving the
permutation cells' suite set alone.
* test(nat): run the VPC egress paths in nat mode, not pool mode only
The single suite gated TestVPCEgressPaths and TestWANEgressSMTPBlock on
external_mode = "pool", with a comment saying nat clusters have no
EIP/NAT-GW support. They have had EIPs since the routed path landed, so
the gate now asks whether the cluster can allocate public addresses at
all — pool, or nat with a non-transit pool configured.
The nat-single cell runs TestVPCEgressPaths beside natuplink, which is
the only nightly coverage of a private subnet's isolation and its NAT
gateway in routed mode. natuplink never creates a private subnet.
* docs(config): list nat among external_mode values
* fix(ci): renumber the multi-node nat cell off the bare-metal job
Cell 20 is already the bare-metal single-node job, gated on the same
cells input but declared in the workflow rather than e2e-cells.json.
A matrix entry with the same number runs two unrelated jobs under one
name, and naming 20 in a dispatch selects both.
* feat(images): add Oracle Linux 10.1 and 9.8, and pin their catalog digests
Oracle publishes its qcow2 checksums inline on an HTML page and ships no
sums file, so the four new entries carry the expected digest in the catalog
rather than a URL to fetch one from. Images gains ChecksumDigest for that,
and the catalog import path verifies against it and says which of the two
sources it used, so a recorded digest never reads as a fetched sums file.
Pinning is safe here and only here: each URL names an immutable build
(b291, b293, b178, b182) and never moves. aarch64 has a -kvm-cloud- build
alongside the plain -kvm- one; the cloud build is the cloud-init image and
is the one that matches x86_64's -kvm-.
TestImageCatalogReachable HEADs every URL in the catalog from the general
single-node suite. These artifacts live on hosts nobody here controls, so
an upstream that retires a build breaks `spx admin images import` for a
customer and nothing else in the suite would notice. It also asserts that
every entry sets exactly one checksum source, and that no pinned entry
points at a "latest" URL whose bytes would change under it.
* fix(formation): carry bridge_mode onto the leader and every joiner
applyNetworkConfig copied external_mode, external_interface and the pools
out of the formation NetworkConfig but never bridge_mode, so a multi-node
cluster wrote a spinifex.toml without the key while single-node init wrote
it. Every node then fell back to autodetecting a mode that init had pinned.
Nightly cell 30 caught it: TestNATUplink failed at the config phase on a
spinifex.toml that did not contain bridge_mode = "nat". Deriving it from
the mode rather than adding a NetworkConfig field keeps the leader and the
joiners on the same rule single-node init already uses.
* feat(ui): show how to ssh into an instance, and which user to use
An instance detail page gave the addresses and the key pair name but never
the login user, which is the one part of the command nobody can derive. It
belongs to the AMI, not to Spinifex: nothing in the launch path creates or
renames an account, so stock cloud-init applies the distro's own
default_user. Oracle Linux is cloud-user, Debian is debian, Ubuntu is
ubuntu, and guessing from the distro name gets two of those three wrong.
The card resolves the user from the AMI's name and description, and an
unrecognised image yields a placeholder rather than a guess, because a
wrong username in a copy-paste command is worse than an honest gap. A
private-only instance is called out as reachable from inside the VPC only,
rather than presented as a command that will simply time out.
* fix(nat): plumb host state for a NAT gateway's public address
AddNATGateway installed the OVN snat row and stopped there. On a routed-NAT
node the host is what delivers a public address, so an egressing private guest
left with a source the host held no route back to, and the masquerade rule —
which matches the transit 100.127.0.0/24 alone — never saw the packet. The
gateway's address needs the same /32 route into OVN, proxy-ARP and FORWARD
accepts an EIP gets.
hostBindSpecs unions floatingIPSpecs with one deduped spec per NAT gateway
address and feeds BindHostEIPs. It is kept apart from floatingIPSpecs
deliberately: pruneOrphanEIPs and addLive read that set as the complete list of
addresses allowed to hold a dnat_and_snat, and a gateway holds an snat, so
widening it would stop the sweep reclaiming a stale row on an address later
reused as a gateway's.
AddNATGateway also binds directly, on both the create and idempotent-skip
paths, so a fresh gateway is not dark for up to 30s waiting on the host EIP
pass — and so a reboot that keeps the OVN row still replays the volatile half.
Documents NAT gateways in nat mode in vpc-networking, and says at the head of
that section that nat is a supported mode covered by nightly cells 19 and 30.
* docs(oci): document Oracle Linux as a guest image, and that its user is cloud-user
Four catalog entries, all UEFI. The login user is the one that catches
people: opc is the default on the images Oracle publishes into OCI itself,
while these are the generic KVM cloud images from yum.oracle.com, whose
cloud-init default_user is cloud-user. Verified on 10.1 and 9.8 — opc, oracle
and ec2-user are all refused.
Also records why these four pin a checksum digest inline (Oracle ships no sums
file, and each URL names an immutable build), why arm64 takes the -kvm-cloud-
build, and that SELinux ships enforcing.
* fix(reconcile): stop reading an unreadable intent as an empty one
A wedged JetStream KV returned no keys for the ENI bucket while writes were
still landing. loadPorts mapped that to an empty port map, pruneOrphanPorts
read every guest LSP as an orphan against it, and every guest on the node lost
DHCP and IMDS while the API kept answering and the instances stayed "running".
The existing re-read guard did not help: it re-reads through the same path.
Three defences, because no single one covers every shape of the fault.
A failed kv.Get now fails the pass instead of dropping the record. Dropping one
VPC narrows localVPCs, which filters out every ENI in it, which the sweep reads
as permission to delete every guest port in that VPC — one failed read, same
outcome, and it needs no exotic KV state. Unmarshal failures stay skips: that
is a poison record rather than an unhealthy store, and failing on it would
wedge reconcile forever.
The port sweep now asks local OVS which taps are plugged in here and never
deletes a port that has one. A running guest's port cannot be an orphan
whatever intent says, and OVS is the one liveness signal that does not come
through the KV read path the sweep is otherwise entirely driven by. Wired from
vpcd; unwired callers fall back to the guard below. An unreadable OVS skips the
sweep rather than deciding on intent alone.
Each sweep also refuses outright when the keep set it would delete against
holds nothing at all — no ENIs, no SGs, or no ports and no external IPs. That
is the case local taps cannot cover, because the drift sweep is leader-gated
and lists NB cluster-wide, so the leader holds no tap for another node's guest.
The cost is real: a genuinely orphaned row now survives until one live record
of its class exists. Leaking a stale row is the recoverable half.
Three existing prune tests drove their fixtures to a wholly empty intent, which
is now the refusal case; they keep one live record of the class so they still
prove the prune. The Get-failure path has no unit test — there is no
jetstream.JetStream fake in the tree and an embedded server cannot be made to
list a key it will then refuse.
* docs(oci): separate the two address quotas, and the principal that can read them
The regional reserved-public-IP ceiling is tenancy-wide and does not grow with
node count, so a cluster hits it before the per-VNIC 64, which does. Records
the commands that read both, and that a detached reserved public IP still
bills.
Reading limits or usage needs a tenancy-level grant, which cannot be
compartment-scoped the way the node's policy is. That is a second, read-only
principal kept off every node, not an addition to SpinifexOperators.
* docs(oci): give quotas their own troubleshooting section, and say we enforce none
Spinifex hardcodes no address ceiling: it attempts the allocation and maps
OCI's own refusal to InsufficientAddressCapacity. So a quota is raised with
Oracle and nothing here is reconfigured or restarted — which is what an
operator needs to know before it bites, not after.
Separates the two quotas by how they scale: the regional reserved-public-IP
default is tenancy-wide and raisable by ticket, the per-VNIC 64 is per node and
is not.
* fix(oci): follow a guest to its new node when its EIP moves
OCI delivers a public address to whichever VNIC holds the private IP behind
it, and nothing ever moved that private IP. Binding.VNICID was written once at
allocate time, as the VNIC of whichever queue-group daemon happened to serve
the AWS request, and never updated — so an address landed on the right node
only by chance, and a guest that stopped and started elsewhere went dark while
every OVN row, host route and firewall rule on the new node was correct.
ClaimLocalAddresses closes the gap from the node that should hold the address.
Local OVS decides ownership, the same authority the host EIP binder already
trusts for the same question, and UpdatePrivateIp with a new VnicId does the
move in one server-side call. The private IP OCID survives it, so the reserved
public IP attached to it follows without being touched and the customer's
address never changes.
A loop per node, not leader-gated: the question is about this host's guests,
and a node that never wins the reconcile lease would otherwise never claim the
addresses of the instances it is running.
* fix(oci): record which ENI an EIP is associated with
An EIP is allocated bare and attached by a later AssociateAddress, so the OCI
binding written at allocate time named no ENI and nothing ever filled it in.
The address then stayed on whichever node happened to serve the allocation:
the affinity pass could follow an auto-assigned address, which is allocated
with its ENI, but never an EIP — the case customers actually use.
BindOwner is on an optional OwnerTracker interface rather than Allocator,
because only OCI has a notion of an address being delivered to one node. A
static or DHCP address is on the wire everywhere its node can reach, those
allocators do not implement it, and ExternalIPAM treats a pool that does not
as a success with nothing to do.
The same gap made the release owner guard inert for EIPs: it compares against
a binding ENI that was always empty, so a stale teardown could free an address
already handed to another instance.
* fix(ec2): report the address an instance is actually reachable on
Three faults kept DescribeInstances answering with the address launch assigned,
whatever happened afterwards.
The launch-time stamp lives on the stored ec2.Instance, and the stopped path
only declined to add an address rather than removing one, so a stopped instance
went on advertising an address it had released and that may already belong to
someone else. The projection now clears it, and the NIC association with it,
working on its copy so a restart can still report the address.
The running path filled the public address only when it was nil, which the
stamp never is, so an EIP could not be reported however correctly it was
recorded. The runtime address now wins.
And nothing told the rest of the system about an association: AssociateAddress
wrote neither the ENI record, which is what IMDS and the network reconcile
read, nor the instance record, which is what a describe answers from. The ENI
write is direct; the instance write is a fan-out, because the queue-group
worker that serves AssociateAddress is rarely the node running the instance.
* fix(net): take the host EIP plumbing off a node its guest has left
Bind refuses an EIP whose ENI is on another chassis; the prune beside it kept
anything still wanted cluster-wide. A guest that moved therefore left the old
node holding the /32 route into OVN and the proxy-ARP entry on the uplink, with
nothing that would ever remove them. On a shared segment that node goes on
answering ARP for an address another node is now serving.
The two halves now ask the same question by the same authority, in the loop
that already runs per node. An unreadable local OVS is not a yes: tearing down
a live address on a guess would fire everywhere at once, so it keeps the
binding and the next pass tries again. A binder with no chassis test behaves
exactly as before, and a centralised EIP is exempt because it hairpins through
the gateway chassis and every node plumbs it legitimately.
* fix(net): refuse to free an Elastic IP on behalf of a departing interface
Releasing an instance's public address on teardown releases it by address,
scoped to the ENI holding it. Since an association now writes the EIP onto the
instance record, that address is the EIP — so terminating an instance gave a
customer's Elastic IP back to the pool while describe-addresses still reported
it as theirs, free to be handed to another guest.
The existing ownership guard cannot catch this. It only fires when the stored
allocation already names an ENI, and an EIP is allocated bare and associated
later, so the stored ENI is empty and the guard passes straight through.
The distinction that holds is who is asking. ReleaseAddress is the address's own
owner and names no interface; instance teardown names the interface it is tearing
down. So a release that names an owner may free an auto-assigned address and
never an EIP, which is enforced in each of the three allocators.
DHCP needed the scope carried on the wire, since it releases by address and
resolves the lease afterwards. Absent on an older caller, which reads as false
and keeps the previous behaviour.
Worst on the two providers where it is not recoverable: on DHCP the lease is the
only thing holding the address and the server promises nothing about handing it
back, and on OCI the address is a reserved public IP object that the release
destroys outright.
* test(e2e): prove an Elastic IP survives a stop and start
Instance teardown releases the public address by value, and since the
association writes the EIP onto the instance record, that release aims
straight at the customer's Elastic IP. The allocator guards now refuse it,
but nothing end to end proved the address comes back on a restart.
The step goes inside EIPFlip, which already runs last on the shared public
guest for the same reason a stop is safe there: the address the guest
returns on is the allocation's, not the pool's. It runs on the static
cells and the DHCP ones alike, and on DHCP it doubles as the only outside
view of the lease still renewing while the guest is down.
* feat: release an auto-assigned public address on stop, take a new one on start
AWS lends an instance a public address for as long as it runs and gives it a
different one on the next start. Spinifex kept the address across a stop, which
holds a pool slot for a stopped instance and, on a DHCP pool, keeps renewing a
lease nothing is using.
A stop now offers the address back through the instance cleaner, which refuses
anything an Elastic IP is behind — the EIP association record answers that, the
same question an ENI delete already asks — and records on the VM that the start
owes it a new one. A host drain keeps the address: Restore relaunches the guest
on the next boot, and changing its address under it is an outage nobody asked
for.
The start allocates before the launch, so an exhausted pool refuses the start
rather than booting a guest the customer cannot reach, adopts an Elastic IP
associated while the instance was stopped instead of assigning a second address
beside it, and hands the address back if the launch then fails.
e2e: a StopStartAddress stage on the public guest asserts a stopped instance
reports no address and comes back on a different one from a static pool, and a
StopStartKeepsTheEIP step asserts the opposite for an Elastic IP.
* docs: restructure the OCI guide around two deployment paths, with diagrams
The guide had grown into a numbered wall with no shape: no frontmatter, no
contents, and a reader with an OCI account had no way to tell where to start.
It now follows the multi-node install template — overview, prerequisites, then
the deployment path — and splits that path in two: Terraform for one node or
three, which is the preferred route, and a manual single-node walkthrough for
an instance that already exists. Both meet at one shared configuration section
rather than repeating it.
Two ASCII diagrams carry what the prose could not: one OCI instance with the
whole AWS surface inside it, and three nodes with the same surface distributed,
showing which plane each VNIC carries and how an address follows a guest that
moves.
The limits section was materially out of date and said so in the places that
mattered most. Multi-node is no longer experimental, public addresses are no
longer pinned to one node — each allocates on its own VNIC and the affinity
pass moves the private half to wherever the guest is — and an auto-assigned
address is now returned to OCI on stop, so a stopped fleet stops paying for
addresses. What changed is recorded rather than quietly edited.
The Terraform configuration moves in from the monorepo as
scripts/terraform/oci-spx, so the guide points at something in this repository.
Also marks the two new lifecycle test files in-package, with the reason each
one needs to be: both substitute unexported dependencies.
* docs: rework the OCI guide — SVG diagrams, Terraform variables, iSCSI and br-wan
Replaces the two ASCII diagrams with SVGs in the house style, one for a
single node and one for three. Renames the two deployment sections to
Terraform deployment and Manual deployment, documents what the Terraform
builds per node and every variable worth setting, and explains that the
data volume is an iSCSI attachment rather than a disk.
The manual path now covers the iSCSI login, the fstab entry and the
netplan source routing that Terraform writes, and states what br-wan is
for on a routed-NAT node: it names the VNIC for the OCI allocator and
holds the registered addresses, but is never bridged into OVS.
Opens with what Spinifex is and why OCI is worth running it on.
* fix: make gateway convergence work on a multi-node routed-NAT cluster
Three defects each stopped a multi-node cluster from ever attaching an
internet gateway, and all three produced the same uninformative symptom:
`terraform apply` sitting on `aws_internet_gateway.igw: Still creating...`
forever while every service reported healthy.
ovn-northd was given only its local OVSDB socket as a client connection.
A RAFT follower does not forward writes -- it answers "not cluster leader,
trying another server" -- so with one entry in the list, northd stops
translating Northbound intent into Southbound flows the moment database
leadership moves to another node. Measured on the OCI cluster:
SB_Global.nb_cfg seventeen transactions behind NB_Global.nb_cfg, with no
chassisredirect port for any VPC created after leadership moved.
setup-ovn.sh wrote the member list only when the caller passed
--ovn-remote, which only install-node.sh did; it now derives the list from
the live RAFT membership otherwise, and says so when it finds a single
member, because that node needs a second pass once the others join.
The gateway datapath probe ran wherever the reconcile lease landed rather
than on the chassis owning the port. Both probes leave from the local
uplink, and under routed NAT the transit subnet exists identically on every
node, so a remote probe reported a healthy gateway as dead and aimed a
repair at a datapath the node does not own. GatewayPortLocal gates it.
On OCI the probe then ARPed the tenant-visible public IP, which the cloud
NATs and which appears on no interface anywhere. The reconciler now maps
the address through the same DatapathResolver the NAT rules are built from.
Also adds DescribeVpcEndpoints, which the AWS provider calls on every VPC
delete to sweep GuardDuty-managed endpoints; an InvalidAction there failed
the destroy and stranded the VPC.
The OCI guide gains a Terraform section recording which shipped workbooks
were validated against a three-node OCI cluster and which are still open.
* docs: record the validated Terraform workbook results for the three-node OCI cluster
Six of the seven workbooks run against the OCI cluster pass, including the
three the section exists to prove: RDS, ECS and EKS. Records nginx-alb and the
clean eks-quickstart re-run, and names the two lifecycle gaps that bit during
the run so an operator recognises them rather than reading them as a broken
workbook.
* fix: route the NAT gateway's SNAT source through the datapath resolver
AddNATGateway was the one NAT path that did not map its external address to
the one that rides the wire. Where the public half of an allocation is NAT'd by
the cloud it is on no interface anywhere, so every private-subnet guest egressed
with a source that was dropped, while the gateway reported available and DNS
kept working because the VPC resolver is host-local.
The unmapped address also failed the host EIP sweep, which works in resolved
addresses, so the bind was pruned as foreign on the next pass and reinstalled on
the one after. The public half now survives as an external ID instead.
Also restructures the OCI guide around numbered steps through to running
Terraform, and moves the cloud-init and metadata sections into troubleshooting
where they are read when something is wrong rather than followed when it is not.
* fix: place a NAT gateway's OCI address on the node that forwards for it
OCI drops a packet whose source is not a registered private IP on the VNIC it
leaves from, so a NAT gateway's SNAT source has to be on the gateway chassis.
Two independent placements decided those and nothing reconciled them: the
address was created by whichever worker served AllocateAddress, while the
chassis is a fixed ordering the reconcile pass asserts. On a three-node cluster
that made private-subnet egress a coin flip, and the same Terraform passed or
died depending on which node answered.
The affinity pass already solves this for guests, by asking local OVS where the
tap is. A NAT gateway has no tap, and it skipped anything with no ENI — which
is exactly and only this address. The equivalent local fact is the VPC's
chassisredirect binding, so ownership now has two answers rather than one, and
an address that is neither a guest's nor a gateway's is still left alone.
The VPC is recorded from the reconcile pass rather than from CreateNatGateway,
because the answer is the gateway chassis and reconcile is what decides it.
* fix: plumb a NAT gateway's host ingress only on its gateway chassis
A NAT gateway's address has no ENI, so neither the bind's locality test nor
the prune's could ever fire for it and every node installed its /32 route into
OVN. Latent on bare metal, where pool addresses are disjoint from the node
subnet; on a cloud that hands out the address from the nodes' own subnet it
outranks the connected route and swallows node-to-node traffic.
Give that address the same second authority the ocinet affinity pass has: the
VPC's chassisredirect binding, read from the local ovn-controller. An
unanswerable question still leaves the binding alone, so a single node whose
gateway port is not yet claimed behaves exactly as before.
The orphan EIP prune built its live set from ENI and user EIPs alone while also
sweeping host ingress, so every drift pass tore down the ingress of every live
NAT gateway and left it dark until the next host EIP tick.
* fix: ask the gateway chassis only about a NAT gateway's address
gatewayElsewhere skipped the NAT gateway marker it was named for and answered
for every spec, so on a node that is not the VPC's gateway chassis every EIP —
including a guest's, whose locality is its own port — was left unplumbed. Two
of three nodes bound nothing.
* test: assert the two private-subnet EKS workbooks in the tofu harness
eks-https-ingress and eks-gitops-argocd put their workers in private subnets
behind a NAT gateway, so a worker only registers Ready if its egress to the
cluster endpoint and to ECR survives that path. That makes them the workbooks
that catch a NAT gateway fault, and neither had an assertion, so neither could
be run at all — run_workbook calls assert_<name> by construction.
Both share eks-quickstart's readiness check, extracted unchanged. They also
name the worker size node_instance_type rather than instance_type, and tofu
rejects a -var for an undeclared variable outright, so the name is read from
the config instead of kept in a list here.
* test: count agents, not every node, when asserting EKS workers Ready
The control plane is three K3s servers, so an unfiltered node count reaches a
one-worker cluster's desired size before any worker joins — the assertion
passed on exactly the failure it exists to catch, reporting "4/1 workers
Ready". Filtering on the control-plane role label makes it 1/1, which is the
number that says a node-group worker registered.
Matters most for the two private-subnet workbooks, where the worker joining is
the whole proof that egress through the NAT gateway works.
* docs: record the two private-subnet EKS workbooks passing on OCI
DescribeInstanceAttribute always returned a Value for userData, an empty string when the instance was launched without any. AWS returns the attribute with no Value at all in that case.
The Terraform provider reads user_data only when Value is non-nil, so an empty string made it record the hash of the empty string in state and propose a change on every plan for an instance whose configuration names no user data.
One 30s timeout bounded every ebs.provider.v1.* request. capabilities, volume.describe and volume.list answer from state already held; volume.create clones an AMI snapshot. The cheap calls set the budget and the clone inherited it.
On a freshly restarted cluster the first clone of an AMI does not finish inside that budget, so RunInstances took a context deadline for a preparation failure and terminated the instance, taking its ENI and volumes with it. A retry of the identical request succeeded once the snapshot had been read once.
The subjects that move or reference data now carry 180s, matching unmountSealTimeout, the existing budget here for an EBS operation the caller cannot bound. The classification lives with the subject constants, so a caller constructing a provider cannot reintroduce one flat value, and a new subject stays fast until someone classifies it.
Retrying a timed-out create is deliberately not attempted: the responder may still be running one, and that needs idempotency this change does not establish.
RunInstances defaulted the root device to /dev/vda, the path the guest sees, so an instance launched without a block device mapping reported a DeviceName that contradicted its own RootDeviceName of /dev/sda1.
ECS capacity and EKS nodegroups named the same guest path explicitly, so every internally launched instance carried it too. All three now name ebsmetadata.RootDeviceName, which is what AWS returns and what the guest already resolves through its own device links.
BlockDeviceMappings[].DeviceName was overwritten with the path the guest kernel assigns, on every launch, start and hot-attach. The name is ForceNew on aws_instance.ebs_block_device and root_block_device, so a value that never reads back as it was sent is not drift to reconcile: every plan destroyed and recreated the instance and everything on its volumes, and a stock terraform-aws-modules/ec2-instance configuration never converged.
The convention it served — lsblk inside the guest showing the same names as DescribeInstances — is not one AWS keeps either. On Nitro the API reports /dev/sda1 while the guest shows /dev/nvme0n1p1, and IMDS block-device-mapping reports the API name too. The API name is a label, and the wire contract owes the caller the label it supplied.
UpdateGuestDeviceNames becomes LogGuestDeviceMap: the guest paths are still discovered and logged for an operator correlating a volume against what the guest sees, and no longer recorded as the device the caller asked for. AttachVolume records the API name it already returns and already persists to volume metadata. IMDS projects the same field and so becomes AWS-shaped with it.
The Terraform AWS provider compares the stored url against the configured one through suppressOpenIDURL, which parses the stored value and forces the https scheme onto it. An issuer carrying a port is not parseable without a scheme — Go reads the host as one and rejects the rest — so the parse fails, the suppressor returns false, and the ForceNew url attribute replaces the provider on every plan. Every issuer this platform mints carries a port, so the AWS-shaped stripped form is the one that cannot round-trip here. The ARN keeps the stripped host-and-path form, which is what AWS uses and what issuerFromOIDCProviderARN reverses.
An instance whose relaunch fails during daemon-startup recovery goes to
StateError, and nothing ever retried it: every later restart read the same
StateError and skipped it again, permanently. The usual cause is a dependency
that was not ready at boot, which is over seconds later, so the instance stayed
down for a condition that had already cleared.
The manager already bounds restarts for a crashing instance, through
Health.CrashCount and Health.FirstCrashTime against MaxRestartsInWindow. A
recovery failure wrote none of it, so it was invisible to the only budget there
is. MarkRecoveryFailed now counts one, the window's two rules move into
recordCrash and rollRestartWindow, and MaybeRestart is rewritten over them with
the same constants and comparisons it had.
With a failure counted, restore can retry: an instance in StateError with budget
left returns to Pending and goes through the existing relaunch, including its
mount-not-ready backoff. With the budget spent it stays in StateError for the
operator and is only announced to the recovering hook, as before. The window
expires, so a later restart tries again rather than writing the instance off.
StartStoppedInstance is the operator's remedy and it was refusing to work. The
shared stopped store never holds a StateError record, so the load missed, and
the fallback treated the presence of the dead local record as proof the instance
was already running. It reads the record's state now. StateError is started
locally through StartInstance, which already accepts it; terminated answers
InvalidInstanceID.NotFound; everything else keeps IncorrectInstanceState,
because a claim inserts the instance before it launches, so a local record in a
startable state is a start another caller is part-way through.
The agent resolved the container runtime once, in New. A containerd socket that was not accepting during the first second of boot left the agent runtime-less for the life of the process, and nothing retried. It registered and heartbeated normally, so the scheduler saw a healthy instance and kept placing tasks that died immediately at the nil runtime check.
The runtime now lives behind a gate that a watch loop keeps current: it dials while the gate is empty, and probes with a container list while it is full, clearing the gate when the probe fails so the next tick re-dials. That covers a socket that is not ready yet at boot and a containerd restarted underneath a running agent alike, and everything that used the puller or the runner reads through the gate, so a runtime arriving late is picked up without a restart.
An agent with no runtime advertises no capacity. Registration and the heartbeat report zero CPU and zero memory while the gate is empty, so placement skips the host, and the control plane reads that pair of zeros as an involuntary drain: the instance becomes DRAINING with agentConnected false and zero registered resources, which is visible through DescribeContainerInstances rather than only in the node's log. Capacity returning clears it through the branch a reaped instance already used. A host with no CPU cannot exist, so the zero needs no new wire field. An operator drain is no longer cleared by a bus registration either, which it previously was.
A deployment whose tasks keep failing to start now backs off. Deployment gains NextLaunchAt, and past the failure count the circuit breaker already tracks, the reconciler widens the gap between launch passes up to a cap. It is keyed on that counter but applies whether or not the breaker is enabled, because AWS leaves it off by default and an unbraked service relaunches a task that cannot start on every pass, forever.
An instance left in StateError could not be terminated through any path, and the gateway reported success anyway, so a Terraform destroy waited on a record the platform had already claimed was gone.
classifyRestoredInstances skipped a StateError instance with the comment that the operator must retry or terminate it, but continued before firing OnInstanceRecovering, which is what binds ec2.cmd.<id>. The topic the terminate travels on therefore had no responder and the stated remedy was unreachable.
With no responder there, TerminateInstances fell back to the ec2.terminate topic, whose handler reads the stopped KV bucket only and answered InvalidInstanceID.NotFound for a record held in the active registry. The gateway discarded that payload without reading its code and took the idempotent-absence branch, returning terminated for a record it had not removed.
The StateError branch now announces the instance to the recovering hook before skipping relaunch, and the gateway reads the refusal: only a genuine absence falls through to the idempotent branch, and any other code is logged and returned.
RunInstances created exactly one volume, the root, plus the EFI variable store, and refused a request naming any other EBS device. An instance could not be launched with a data volume attached, so terraform-aws-modules/ec2-instance with an ebs_block_device failed the apply and the workaround was CreateVolume plus AttachVolume afterwards.
The launcher was never the constraint: buildDrives already has a data-volume branch and already allocates a hot-plug port for a volume attached at RunInstances time. The control plane now creates a volume for every EBS mapping through the volume service, which is where CreateVolume's validation and metadata write already live, and records each as a non-boot request at the device name the caller named.
A data volume is created empty, so a mapping naming a snapshot or naming no size is still refused by name rather than served as something it did not ask for. A launch that fails partway deletes the volumes it created.
The filter was never read, so ACTIVE, DRAINING and no filter at all returned every registration in the cluster. A caller could not ask which container instances can take work without describing every one of them, and a cluster carrying a drained node read as if every node were live.
The status a record holds is the one DescribeContainerInstances reports, so a filtered list and a describe of what it returns agree. REGISTERING, DEREGISTERING and REGISTRATION_FAILED are accepted and match nothing, because no record here ever holds them; a status outside the enum is refused.
An unserved object sub-resource fell through to PutObject, GetObject or DeleteObject, so a write to one stored the object itself. The pin also carries the meta leader health check fix and the router's single query parse.
Both install guides had grown sections that were not install steps, which is what made the multi-node one hard to follow. Each now ends on a step that hands off to Setting Up Your Cluster, and the reference material moves under Administration where it can be read on its own.
- Single-node and multi-node both end on a "Set Up Your Cluster" step rather than a trailing sentence.
- The multi-node cluster-state check via `spx admin kv digest` is gone. Verification is now every node Ready, capacity pooled, and the AWS API answering.
- "How Multi-Node Storage Works" becomes docs/admin/predastore-distributed-storage, expanded with the host and node model, the write and read paths, degraded writes, hinted handoff, repair, encryption at rest, and what each class of failure costs.
- "Firewall and Cluster Membership" becomes docs/admin/host-firewall, expanded with the full public, cluster and encap port groups, how to narrow SSH, and which files under /etc/spinifex/firewall survive an upgrade.
- Setting Up Your Cluster gains a closing step that arms the firewall and narrows SSH, so the securing work lands after a cluster that works rather than during the install.
install-usb and vpc-networking followed the two moved sections to their new paths.
Three omissions left the launched root volume reading back as something the caller never asked for.
DescribeInstances returned a null RootDeviceName and RootDeviceType while DescribeImages populated both for the same AMI, so the two answers disagreed about the same image. The launch path now projects both onto the instance.
The root volume's EBS request carried no device name, so the launcher wrote an empty string to the volume's metadata and DescribeVolumes reported an attached volume with an empty Attachments[].Device. The name parseVolumeParams already resolved now travels with the request.
A volume-scoped TagSpecification was discarded: only instance-scoped tags were read, and the comment claimed volume specs were handled elsewhere, which nothing did. They are now written to the root volume's metadata document.
BlockDeviceMappings[].DeviceName still reports the guest path rather than the requested name. That rewrite is deliberate and is left alone here.
* fix: route the EKS update reads so they refuse instead of reading as a typo
UpdateClusterVersion was registered at POST /clusters/{name}/update-version, which is the nodegroup pattern and not a path the SDK ever calls; AWS puts it at POST /clusters/{name}/updates. The refusal written for it was therefore unreachable, and the router answered InvalidAction, which tells a caller they typed something that does not exist.
ListUpdates and DescribeUpdate had no route at all and answered the same way. Both now refuse with NotImplemented at the gateway rather than through a service method: the update surface they would read is absent, so a round-trip to reach a constant refusal is surface for its own sake.
The absence of the cluster-level update surface is unchanged. This only stops the platform blaming the caller's spelling for its own routing.
* test: account for the two EKS update reads in the cross-tenant table
Both name their cluster by a path segment that carries no account segment, so
neither can name another tenant's resource and neither needs a case of its own.
The agent acked a stop in the same loop iteration that launched the reap, so
the directive was deleted before the reap had done anything. When the reap
could not run — no container runtime, or a failing List — nothing re-posted it,
and the task held its ENI, its capacity reservation and its target-group
registrations forever. The in-memory `stopping` map made that permanent: it was
set at dispatch and never cleared, so the agent would not re-dispatch even if
the directive did come back.
A stop ledger now separates a reap in flight from a reap that reported STOPPED.
Only the second is acked, so a failed reap leaves the directive in the inbox to
be redelivered and retried, while the assign loop keeps reading both halves and
so still refuses to start a task that is stopping or stopped.
* fix: persist security group rule tags and report -1 ports for all-protocol rules
A security group rule accepted a security-group-rule TagSpecification and kept
none of it, so DescribeSecurityGroupRules answered with no tags and every
subsequent plan offered to add them again. SGRule now carries a Tags map,
populated on both authorize paths and projected into the central tag store, and
DescribeSecurityGroupRules evaluates tag: and tag-key filters against it instead
of excluding every rule.
CreateTags and DeleteTags on an sgr- id were a silent no-op because
updateRecordTags routed only subnet-, vpc-, sg- and eni-. It now finds the rule
inside its parent group's record and mutates it there, and a revoked rule's
central entry is cleared with it.
An all-protocol rule reported the ports it was stored with, 0 and 0, where AWS
reports -1 and -1, which is the same never-converging plan by another route.
* fix: recognise sgr- as a security-group-rule resource id
DescribeTags reported ResourceType "unknown" for a security group rule, so a
resource-type filter naming security-group-rule matched none of them, and the
authorizer could build no ARN for a rule id and fell back to the any-resource
wildcard. Both read the same prefix table, which had no entry for sgr-.
RunInstances read BlockDeviceMappings[0] and nothing else, so a request carrying a root mapping and a data mapping created one volume sized from whichever happened to be first, and the first mapping became the root whatever device it named. A module emitting the data mapping first produced a root volume of the data volume's size and no data volume at all, with no error.
The root is now the mapping naming the AMI root device, with position deciding only when nothing names it, so the single /dev/vda mapping ECS and EKS launch with still works. Any further EBS mapping is refused in ValidateRunInstancesInput with InvalidParameterValue naming BlockDeviceMappings and the devices it cannot attach, rather than being discarded silently. Ephemeral mappings carry no volume and are ignored.
Creating the additional volumes at launch remains unimplemented; this makes its absence visible.
The deploy builds with GOWORK=off, so spx links the pin rather than the sibling
checkout, and the versioning handlers are absent from every deployed binary
until it moves.
GitHub registers a schedule: only from a repository's default branch, so this cron could drive exactly one branch and always executed main's copy of this file whatever it tested. main was never exercised by the nightly, and a fix to the workflow itself could not be proved until it had already shipped — which is how cell-24's unguarded pkill stayed live on the schedule after its fix merged to dev.
A Cloudflare Worker now dispatches this workflow once on dev and once on main. A dispatch takes both the workflow file and the source under test from its ref, so GIT_BRANCH is simply github.ref_name and every schedule-only expression goes with the cron.
cell-24 clears leftover predastore nodes with `pkill -KILL -f '/bin/s3d
-config'`, which matches every predastore node on the runner and not only the
ones this run left behind. The predastore repository runs the same harnesses on
the same machine, so "still running" does not mean "stale" — on 2026-09-16 this
step killed a predastore benchmark 32 seconds into its workload, and the run
failed with 61 connection-refused errors that look nothing like their cause.
Both kills now run under the host benchmark lock the harnesses themselves take,
so anything alive while it is held really is a leftover. The second one needs it
as much as the first: by the time cleanup runs the harness has released the
lock, so another repository's benchmark can already be up.
Every deployed binary links whatever go.mod pins, so the routing fix for bucket sub-resources is absent from the gateway until this moves. Without it a provider-level default_tags block still leaves no bucket creatable by Terraform.
DescribeTags kept answering for resources that no longer exist: the central tag store's DeleteAllTags had one production caller, the instance terminate path, so every other resource type left its entry behind forever.
CentralTagWriter becomes CentralTagStore and gains DeleteAllTags, which TagsServiceImpl already implemented, so the daemon wiring is a rename rather than new plumbing. clearRecordTags mirrors projectRecordTags: a nil store is a no-op and a failure is logged rather than returned.
The clear runs after the record delete has succeeded, never before, so a delete that fails leaves a live resource with its tags intact. deleteNetworkInterface covers both the public delete and the forced instance teardown, and deleteSecurityGroupInternal covers the default group that only the DeleteVpc cascade ever removes.
An AWS SDK's default endpoint resolution builds
{service}.{region}.{services/domain}. IMDS previously answered that
leaf with the literal amazonaws.com, so a workload that forgot its
endpoint override silently reached real AWS instead of failing.
Now that service-endpoint DNS resolves cluster-side, serve the
cluster's configured AWS.ServicesDomain instead, threaded from
config through vpcd.Config into NewIMDSServiceImpl. An empty value
fails closed with a 404 rather than ever falling back to
amazonaws.com. services/partition is unchanged: it still returns
aws, since AWS SDKs carry hardcoded partition tables.
A tag set at creation time through TagSpecifications was written only onto the resource's own KV record. DescribeTags reads a separate central index at tags/{accountID}/{resourceID}.json, which only CreateTags writes, so a resource born with tags described as having none until something tagged it again.
Terraform finds this immediately: terraform-aws-modules/eks reads a cluster security group's tags back after creating it, and aws_ec2_tag fails against a resource whose tags the tag API cannot see.
The four VPC create paths now project their stored tags into the central store through an optional CentralTagWriter, injected after construction like the external IPAM. A nil writer leaves create behaviour unchanged, and a write failure is logged rather than failing the create — the resource exists either way, and failing a create over a tag index would be worse than the divergence it repairs.
Only vpc, subnet, security group and network interface are covered here. Image, volume, snapshot, route table, internet gateway, egress-only internet gateway, elastic IP, NAT gateway, key pair and launch template have the same split and are follow-up work.
* fix(ecs): make awsvpc ENI slots a placement resource
A container instance carries a finite number of awsvpc task ENIs — the hot-plug PCIe root ports the daemon pre-allocates from the instance type's cap — and nothing outside the daemon knew the limit existed. The scheduler's capacity model was CPU, memory and GPU, so every over-capacity task was placed, created an ENI, failed to hot-plug it and rolled it back. A service asking for more tasks than the cluster has slots churned ENI create/delete indefinitely instead of reporting a placement failure once. On a t3.micro the third task and everything after it failed with "attach task ENI eni-… -> i-…: ServerInternal".
The instance record now carries the instance type, reported the way AWS carries it, as the ecs.instance-type container-instance attribute; the ENI cap is derived from it rather than stored, so a corrected limit table applies on the next read. An instance whose type has not been reported is not gated at all: a recognised type always has at least one slot, so zero can only mean unknown, and refusing every awsvpc placement against an older agent would turn a mixed-version cluster into an outage.
Both placement paths gate on it. A task declined for want of an interface is reported as RESOURCE:ENI, spelled as AWS spells it, and separately from a plain capacity refusal. The caller-visible detail is a fixed sentence; the wrapped error keeps its identifiers and server error codes in the log, where someone can act on them.
DescribeInstanceTypes now reports NetworkInfo.MaximumNetworkInterfaces, from the same table the slot allocator enforces, so a caller sizing an autoscaling group has something to size against. Fields this stack does not implement are left unset rather than guessed.
Finally, the daemon records the slot it actually allocated as the ENI's device index. The requested index was stored verbatim, so every task ENI on an instance reported index 1 while landing on a different PCIe slot; on AWS a device index is unique per instance.
* fix(ecs): send an ENI capacity refusal as the reason alone
AWS returns RESOURCE:* placement failures with no detail. Carrying a sentence
there gives a caller a second string to parse that says nothing the reason does
not already say.
CreateTags is a read-modify-write of one resource's whole tag map. It was
serialised by a sync.Mutex on the service struct, which exists once per
process, while the handler answers on the spinifex-workers queue group and so
runs on whichever node NATS picks. Two nodes read the same map, each added its
own key, and the later write won. Every call returned success.
Six parallel writers against one security group on a three-node cluster left
four tags. On a single node the mutex hides it entirely, which is why no test
caught it.
The tag index moves from one S3 object per resource to a JetStream KV bucket,
where the merge runs inside kvutil.Update and commits at the revision it read.
A writer that loses the race re-reads the winner's keys and adds its own on
top. S3 could not host the fix: predastore's PutObject has no If-Match, so
there is nothing to compare and swap against.
Tags written before the move are not migrated in a batch. A batch that
half-failed would leave a resource looking untagged, and the next write would
commit that as the truth. Instead the CAS seeds a missing entry from the old
object inside the attempt, so a resource migrates on its first write and a
failure to read the old object fails that one call. DescribeTags reads the
union of both key sets, preferring the bucket, and DeleteAllTags clears both so
old tags cannot reappear through the fallback.
The process-local mutex is removed rather than left beside the CAS.
The regression test drives six separate service values over one bucket,
because that is what makes it able to fail: with the old code every caller
inside one process was already serialised, and only a second node could lose an
update.
kvutil.BucketNames ranged the KeyValueStoreNames lister channel using whatever context the caller supplied, and the EKS desired-set builder supplies one that lives as long as the daemon. A lost stream-names reply therefore left the channel open with nothing ever sent on it and parked the caller for the life of the process. kvutil.Keys, immediately below it, already bounds exactly this case and says why; BucketNames never got the same treatment.
The DNS reconcile made that hang cluster-wide. A pass takes the reconcile lease before computing its batch and releases it on a deferred call, and the lease renews on a background timer for as long as the holder lives, so a pass that never returns holds it forever and the other nodes never take over. Observed on a three-node cluster: the zone object stopped being written after a rolling deploy, and every name published afterwards stayed NXDOMAIN until the holder was restarted.
The listing now carries its own deadline and reports a timeout as an error, never as a short list, because callers prune on absence. The pass bounds the batch computation as well, so any other deadline-free call reached from the desired-set builders costs a cycle rather than the cluster. Only the read side is bounded: it has no side effects, so abandoning it is safe, while the apply stays inline under the lease.
ListTasks read only the cluster and returned every task record under it, discarding DesiredStatus, Family, ServiceName, StartedBy and ContainerInstance. On a cluster with one running task and a hundred stopped ones, both status filters returned the same hundred-and-one ARNs.
A taskFilter now carries the input's matchable fields and is applied per record. An omitted desiredStatus means RUNNING, as it does on AWS, so an unfiltered call no longer hands back everything the cluster has ever run.
recordTaskState moves a self-exited task's desired status to STOPPED alongside its last status. Without that the filter would be honoured and still wrong: a task that died on its own would answer the RUNNING filter for as long as its record survived.
Container-instance counts came from len(PlacedTasks), which is the set of tasks still owing capacity rather than a status tally, so a PENDING task counted as running and a task whose release did not complete counted as running forever. PendingTasksCount was never set at all. Both are now tallied from the task records, the way the cluster-level counts already were. DeregisterContainerInstance passes zero deliberately — the instance is gone, and counting there would report the tasks it just force-stopped.
launchType is not filtered because TaskRecord has no launch-type field to match on, and maxResults/nextToken are left alone because no List call in this package paginates; adding a token scheme to one would leave the other four inconsistent.
* feat: publish AWS service endpoint names in DNS and match gateway cert SANs
ec2.<region>.spinifex.internal and its sibling AWS service names (sts,
elasticloadbalancing, ecs, eks, acm) NXDOMAINed because the internal
suffix was only ever a cert SAN and Host-header match, never a
northstar zone. BootstrapBaseZone now seeds it as a third zone, guarded
three-way so a suffix colliding with the default or internal domain
seeds no duplicate.
The reconciler publishes each name as a set-valued A record spanning
every cluster node's own gateway address, since a service endpoint's
natural target is every node rather than one resource address that
answers the same from anywhere. ConfigArr.SetRecordSet backs this in
northstar; UpsertRecord's single-record semantics are untouched for
EC2/ELB/EKS/RDS. A non-IP gateway host is skipped rather than written
as a malformed record, and prune authority for the new class is scoped
to its own zone so it can never act on the base or private zone.
admin.AWSGWServiceDNSNames and handlers/dns.ServiceEndpointNames both
consume config.AWSGWServiceNames, so the cert SANs and the DNS records
they must match can never drift apart.
Depends on github.com/mulgadc/northstar#31 (ConfigArr.SetRecordSet);
go.mod pins that branch's commit until it merges and tags.
* refactor(config): rename AWS.InternalSuffix to AWS.ServicesDomain
ServicesDomain names the value IMDS serves at
/latest/meta-data/services/domain, the slot an AWS SDK substitutes
into {service}.{region}.{domain} for default endpoint resolution.
The old name read as a sibling of Northstar's compute.internal while
meaning something entirely different, which invited confusion.
The default changes from spinifex.internal to services.internal, role-
named like compute.internal (per-instance hostnames) now that
services.internal holds service endpoints instead.
Deployed clusters have internal_suffix baked into spinifex.toml and
their gateway cert SANs built from it, so LoadConfig now reads
services_domain, falls back to the legacy internal_suffix key when
that is unset, and only then applies the new default. Every consumer
of the field and the renamed DefaultAWSServicesDomain constant is
updated accordingly, including the spx admin init template and the
e2e test harness's own config loader.
* chore: pin northstar to the merged set-valued write path
* fix(gateway): resolve DescribeRegions endpoint from the gateway's advertised host
DescribeRegions hardcoded https://localhost:9999 as the Region's endpoint
regardless of where the gateway actually runs, so a workload doing endpoint
discovery through this API was pointed at its own loopback instead of a
reachable address. The endpoint is now resolved by the caller from
GatewayConfig.RegistryHost/RegistryPort, the same advertised host:port ECR
already uses (concrete AWSGW bind host, else AdvertiseIP), and passed into
DescribeRegions rather than invented there. Falls back to localhost:9999
only when no concrete host is configured, which is also the only case where
that value is actually correct.
Closes mulga-0bqih.
* style: trim DescribeRegions endpoint comments
* feat(imds): serve block-device-mapping from instance block device data
The aws-ebs-csi-driver's IMDS client fails outright on this 404,
aborting IMDSInstanceInfo before it reaches the Kubernetes fallback the
driver otherwise uses. Serve ami, root, and one ebsN per attached EBS
volume from the instance's existing BlockDeviceMappings, root device
first, matching how the driver counts "ebs" occurrences in the listing.
* style: trim block-device-mapping comments
AuthorizeSecurityGroupIngress and AuthorizeSecurityGroupEgress rejected any
IpPermission carrying an Ipv6Ranges entry. The security-group module defaults
egress_ipv6_cidr_blocks to ::/0 and the provider revokes the group's own
default egress before authorizing the replacement, so the rejection left the
group with no egress rules at all, which is treated as deny.
SGRule gains CidrIpv6. The rule is stored, deduped, revoked and projected like
any other, and the two policy.Rule adapters drop it before the ACL builder,
which is IPv4-only. No interface here has an IPv6 address, so the rule matches
nothing - which is what an IPv6 rule does on AWS in a VPC with no IPv6 CIDR.
Revoking an IPv6 rule that does not exist now returns InvalidPermission.NotFound
rather than succeeding as a no-op. The no-op existed only because authorize
refused to store the rule, so a caller's state could never match.
Closes mulga-6bu36.
DescribeTasks returned an empty attachments array for every STOPPED awsvpc task. On env19, 134 of 135 tasks in a cluster reported attachments: [], and only the one PENDING task carried an ElasticNetworkInterface block. On AWS a stopped task keeps its attachment for the retention window, with status DELETED and the networkInterfaceId still readable, because it is part of the forensic record of a task that has stopped.
taskToAWS gates the attachment on ENIID being set, and the ENI release retry had made a successful release clear that field so a non-empty ENIID would mean "still owed" and the sweep could retry from it. The signal was right; carrying it in the emptiness of ENIID was not, because the same field is the projection's only source.
The state moves to an ENIReleased flag of its own. ENIID stays put, so the attachment survives as long as the record does, and the sweep's owed test, its refusal to prune, and the placement rollback all read the flag instead. A released record ages out on retention exactly as it did before the retry landed. The attachment reports DELETED once released rather than echoing the task's own status.
This also restores the evidence path the ENI-leak probe depends on: it follows a stopped task to its ENI, and with no attachment to follow it had been reporting skipped on every run.
Closes mulga-5bnx3.
Releasing an awsvpc task ENI was best effort: reclaimTaskENI logged a failure and returned, and nothing ever tried again. Its own comment said the ENI fields stayed on the record for a later retry, but no later retry existed, and the stopped-task sweep pruned the record an hour later without looking at those fields — deleting the only pointer to the leaked interface. An orphaned task ENI still references its security group, so DeleteSecurityGroup fails with DependencyViolation and no awsvpc stack can be destroyed without deleting the ENIs by hand.
A non-empty ENIID on a stopped record now means exactly one thing: this ENI is still owed. reclaimTaskENI clears the identity when Release succeeds and leaves it when it does not, forceStopTask reclaims before its single write so the cleared fields land in it, and the stopped-task sweep gains the job of paying the debt off — retrying with a backoff that doubles from 30s to a 15m ceiling, capped at twenty attempts per pass because Release is a 30s NATS round trip and a bucket full of them would stall the sweep for the whole fleet.
A record that still owes an ENI is never pruned, however far past retention it sits. It is the only pointer to the interface, so dropping it converts a recoverable leak into a permanent one.
The placement rollback had the same hole from the other end: an attach failure cleared the ENI fields unconditionally and RunTask never persisted the record, so a failed release there leaked with no trace at all. It now clears only when the release succeeded, and otherwise persists the record as STOPPED so the sweep owns it like any other.
Closes mulga-1ysjx.
An AMI imported on a node whose timezone is not UTC was stamped with local wall-clock time carrying a Z suffix, so on a +1000 node an 11:42Z import reported 21:42Z — ten hours in the future. The layout "2006-01-02T15:04:05.000Z" treats the trailing Z as a literal character rather than a zone directive, so Format printed whatever zone the value carried and then asserted UTC regardless.
DescribeImages now converts before formatting, which also corrects records already stored, since the stored time.Time still carries its original zone. The four sites that stamp a new image now store time.Now().UTC() as well, so the value is right on both sides of the round trip.
Terraform's most_recent sorts on this field, so before this an AMI imported on a +1000 node outranked a genuinely newer one imported on a UTC node for ten hours.
Closes mulga-il7w1.
DescribeServices returned no availabilityZoneRebalancing field, so the terraform provider read the computed attribute as empty and planned an in-place update to DISABLED on every plan, including one taken immediately after a clean apply.
The pinned aws-sdk-go v1.55.8 is the last v1 release and its ECS shapes predate the field, so it can be neither parsed from an input nor set on an output through the SDK types. Embedding ecs.Service in a wrapper does not work either: jsonutil nests an embedded struct rather than flattening it, so the service arrives under its own key.
The field is read off the raw request body and written onto the encoded response instead. checkAZRebalancing refuses anything but DISABLED with InvalidParameterException, because nothing here rebalances tasks across zones and a silently dropped ENABLED would report a service as rebalancing when nothing does. addSDKGapFields runs from WriteJSONResponse so every service-carrying action is covered from one place, decodes with UseNumber to keep the epoch-second times jsonutil emitted, and leaves a value already present alone.
Closes mulga-t239t.
oci.WithTTY put Terminal in the OCI spec while the task was created with a cio
creator that asked for no terminal, so the shim made no console socket and runc
refused to start the container at all. The terminal flag now reaches the FIFO
set, and a terminal carries no separate stderr stream.
* fix(ecs): enforce or refuse container runtime fields (TFC-0040)
user, mountPoints, volumesFrom, readonlyRootFilesystem, privileged,
pseudoTerminal, interactive, linuxParameters, systemControls,
startTimeout and stopTimeout registered successfully but were never
stored, sent to the agent, enforced, or echoed back on describe.
Store and echo the fields the agent can enforce (user, readonly root
filesystem, privileged, pseudo terminal, interactive, systemControls,
capability add/drop, start/stop timeout), apply them through containerd
via OCI spec options and a graceful Stop, and refuse at
RegisterTaskDefinition the fields the agent cannot honor (mountPoints,
volumesFrom, linuxParameters.devices/initProcessEnabled/
sharedMemorySize/tmpfs). Refusal only triggers on a value that asks
for the restriction; empty lists and false/unset flags are accepted
and echoed as before.
* test: move the bus wire-contract test to an external package
* fix(ecs-agent): release an interactive container's stdin so its copier exits
* fix(ecs): return a caller-supplied empty collection instead of omitting it
CreateNetworkInterface accepted a caller-supplied PrivateIpAddress
verbatim, with no subnet-range check and no IPAM reservation, so a
direct call could collide with a concurrent AllocateIP handing out the
same address. Add IPAM.ClaimIP, a CAS-guarded sibling of AllocateIP
that validates the address against the subnet and rejects it if
already allocated, and wire it into CreateNetworkInterface with the
matching AWS error codes.
RunInstances --private-ip-address silently returned a different,
pool-assigned address instead of honouring the request. Forward
PrivateIpAddress (including the NetworkInterfaces[].PrivateIpAddresses
primary entry) into the auto-created ENI's CreateNetworkInterfaceInput,
and reject a pinned address on a launch with MaxCount > 1 since AWS
only allows it for a single instance.
* fix: decompress single-file zst/bz2/xz/gz disk images into tmpdir
ExtractDiskImageFromFile only handled tar archives and a broken plain
.xz branch that ran xz -dk beside the source and then tried to parse
tar chatter for a path, so it silently returned an empty string. Talos
(.raw.zst, .raw.xz), Fedora CoreOS (.qcow2.xz, .raw.xz) and Flatcar
(.img.bz2) publish only single-file compressed artifacts, so none of
them could be imported.
Add streaming decompression for .zst, .bz2, .xz and .gz that writes
into the caller's tmpdir and returns that path directly, instead of
inferring it from tar output. A decompressed .qcow2 payload now flows
through the existing qcow2-to-raw qemu-img conversion via a shared
finalizeDiskImage helper. zstd uses klauspost/compress, already a
transitive dependency via containerd, for a pure-Go streaming decoder.
bz2 and gzip use the standard library. xz shells out to the xz binary
since no pure-Go xz decoder is already a dependency, and the existing
.tar.xz branch already requires that binary on the host via tar's -J
flag, so this adds no new host requirement.
* fix: write sparse holes when decompressing single-file disk images
Verified live on env13: a Talos raw.zst imported successfully but the
decompressed output was written literally rather than sparsely, an
11 GiB file on disk for content that decompresses to 183 MiB with
xz -dc | dd conv=sparse. This is a regression relative to the
qcow2-to-raw path beside it, which already writes sparse via
qemu-img convert.
decompressToTemp now reads in 1 MiB blocks, seeks past all-zero
blocks instead of writing them, and truncates to the final size so a
trailing zero run is still recorded as a hole. 1 MiB amortises the
read/seek syscall overhead over a multi-GiB image while still being
small enough to catch the large contiguous zero runs that dominate a
sparse raw disk; a zero run shorter than one block is written
literally, which is the safe direction for a false negative to fail
in.
zstd, bzip2 and gzip all route through decompressToTemp, so all three
gain sparse output from this change. The plain-.xz path shells out to
the xz binary with output piped straight to a file and does not go
through decompressToTemp, so it is unaffected by this fix and remains
a known follow-up.
* fix: make plain-.xz decompression sparse too
decompressXz shelled its output straight to a file, bypassing the
sparse-hole handling just added to decompressToTemp for zstd, bzip2
and gzip. .raw.xz is the format Talos and Flatcar actually publish
alongside .zst, so this path is the one most likely hit in practice,
and left dense the exact disk-size regression this fix targets still
fires for it.
exec.Cmd already runs its own copier goroutine when Stdout is not an
*os.File, and Wait joins it. decompressXz now hands the xz process an
io.Pipe write end and reuses decompressToTemp on the read end, so
plain .xz gets identical sparse-hole handling to the other codecs. A
separate goroutine closes the pipe with the process's exit error once
Wait returns, so a read past the end of a failed decompression
surfaces that error, stderr included, instead of a bare EOF; the
error path also closes the read end explicitly so a stuck copier
cannot wedge Wait forever if decompressToTemp gives up early.
AssociateAccessPolicy recorded a policy association but never granted
it in the cluster. Bake static ClusterRoleBindings for the mulga:
groups into the eks-node image, project a principal's cluster-scoped
associated policies onto those groups in the TokenReview webhook, and
reject namespace scope at association time since there is no
namespace-scoped RBAC writer.
* feat(iam): evaluate trust-policy conditions and accept aws:SourceAccount
evalTrustPolicy read Effect, Action and Principal but never Condition, so a
stored aws:SourceAccount clause was a control nothing enforced. Give it a
condition arm evaluated in both the Deny and Allow passes, following the
StringEquals-only, fail-closed-on-unknown-key shape
evalTrustPolicyForWebIdentity already uses. The value is the caller ARN's own
account, derived server-side rather than caller-asserted.
Relax validateWebIdentityCondition from a single-action gate to a per-action
allowlist: sts:AssumeRoleWithWebIdentity keeps its existing IRSA keys, and
sts:AssumeRole now accepts aws:SourceAccount under StringEquals. Everything
else still fails at write time, naming the operator, key and action rather
than a bare MalformedPolicyDocument, and CreateRole/UpdateAssumeRolePolicy now
carry that reason to the caller instead of swallowing it into slog.Debug.
Extend the identity-policy refusal at validateStatementRestrictions with the
reason for aws:SourceArn and aws:RequestTag/*, the two keys a companion change
records as divergences in rfc0-7_aws-divergence.md.
* test(iam): fix trust-policy reason tests after aws:SourceAccount now valid
PR #1046 merged into dev with fixtures asserting aws:SourceAccount plus
StringEquals on sts:AssumeRole is refused. Part B of mulga-yxd9p makes
that combination valid, so the fixtures now exercise an unsupported
action and an unsupported key instead.
* docs: drop bead and plan-decision references from test comments
Bump bluebottle for the new key. requestConditionKeys now maps this
door's internal principal-type spelling to the canonical AWS value
through the bluebottle constants, and omits the key for an
unrecognized type rather than supplying an empty string.
ValidateTrustPolicyDocument builds precise errors naming the statement index
and the offending element, and both callers discarded them into slog.Debug
and returned a bare MalformedPolicyDocument. Debug is below the default
level, so the reason reached nobody: not the caller, and not the operator
reading the gateway log.
The contrast with the identity-policy path was stark. CreatePolicy names the
operator and the key exactly; CreateRole answered only 'The policy document
is malformed.', so an operator could not tell whether Principal, Action, a
size limit or a Condition block was at fault. The trust gate is the one the
terraform ECS module hits, which is what made this worth fixing on its own.
Both sites now use awserrors.Errorf, the same mechanism the identity path
uses. The separate Debug line is dropped rather than duplicated: the gateway
already logs a returned error at Error level, so one message now serves both
the client and the operator.
* feat(ecs): populate service event ring on DescribeServices
Events was always projected as null since ServiceRecord carried no event
history. Add a capped, newest-first event ring populated at the rollout
transitions the reconciler already computes: steady state reached, a
circuit-breaker trip, and a placement shortfall it cannot resolve this pass.
Emission is edge-triggered on the state change already guarding each site, so
the repeated reconcile loop does not refill the ring with one repeated
message every pass.
* feat(ecs): project createdAt on services and deployments
CreatedAt was already computed and persisted on ServiceRecord and Deployment
at creation time, but serviceToAWS never echoed it, so DescribeServices could
never answer which build wrote a given record. Project ServiceRecord.CreatedAt
and Deployment.CreatedAt/UpdatedAt, omitting the key entirely for a legacy
record whose timestamp predates the field rather than emitting the 1970 epoch.
CreateCluster refused accessConfig.authenticationMode = API_AND_CONFIG_MAP,
which is both AWS's default since 1.23 and the terraform-aws-eks module
default, and named no parameter in the refusal. The mode is now stored on
the cluster record and echoed by DescribeCluster, so a caller does not see
a permanent access_config diff on every subsequent plan; bare CONFIG_MAP
still fails, since Spinifex has no aws-auth ConfigMap path. The refusal
now names accessConfig.authenticationMode via awserrors.Errorf.
nodegroupRecordToAWS also only set Health when a node group had a status
reason, so a healthy node group described back with health: null instead
of the empty-issues object AWS always returns. Health is now always
constructed with a non-nil issues slice.
The unserved-service gate ran before signature verification but answered
SignatureDoesNotMatch, falsely blaming credentials that were never checked.
It now returns InvalidAction, the same code an unimplemented action on a
served service already gets, so the behaviour is consistent rather than
novel.
The envelope was also always XML, because jsonErrorService only lists
scopes the gateway serves and an unserved scope is by definition absent
from it. A JSON-protocol caller such as sqs or dynamodb therefore could
not deserialize the body and never saw the true error. writeSigV4Error now
falls back to the request's own X-Amz-Target and Content-Type headers to
pick JSON when the scope is not in that list, leaving every served scope's
envelope selection unchanged.
RegisterTargets read the target group, appended in memory and wrote the whole record back with no revision guard, so two concurrent registrations lost one. Terraform parallelises by default, so any target group with more than one member hit it, and the apply exited 0 with half the fleet not behind the load balancer.
Every writer of the record now goes through the store's existing CAS rather than a wholesale Put, because a CAS loop on one writer is defeated by a non-CAS writer on the same key. Contention stays per target group; no global lock is introduced.
RequestCertificate returned InternalErrorException for a missing tenant CA, which is a client-actionable precondition rather than a server fault. Worse, that code is classified retryable, so the provider backed off and retried instead of failing, and a single certificate sat at Still creating for over nine minutes. It now returns ACM's not-found code, which is terminal, and carries a message naming the missing CA and how to create one.
ListCertificates omitted CertificateSummaryList entirely on an empty account, since the SDK marshaller drops a nil slice unconditionally.
Closes: mulga-c6khe
Closes: mulga-wbjal
Closes: mulga-l6yft
Closes: mulga-61h3h
* fix(iam): persist policy tags and count attachments across all principal kinds
CreatePolicy built its record with an empty tag list and never read input.Tags, and GetPolicy, ListPolicies and CreatePolicy's own response projected no Tags field at all. Both halves are fixed, matching the role path.
buildAttachmentCounts scanned only the users bucket, so a policy attached to a role or group counted as unattached. That is not just a misleading number: DeletePolicy gates its conflict check on the same count, so an attached policy could be deleted, leaving the principal holding a dangling ARN.
The count and the reverse lookup are the same traversal with the names kept or discarded, so both now derive from one buildPolicyAttachments over users, roles and groups. Nothing is stored, so no counter can drift from the attachments it counts.
ListEntitiesForPolicy is implemented on that traversal, honouring EntityFilter and PathPrefix.
Closes: mulga-fvhlc
Closes: mulga-9i6uh
Closes: mulga-ryu30
Closes: mulga-8ranb
* test(iam): account for ListEntitiesForPolicy in the cross-tenant and policy-ARN gates
The action resolves PolicyArn through the same branch as GetPolicy, so it
belongs with that helper's siblings; it was in neither gate, which is what
the accounting test exists to catch.
* fix(ecr): store and project repository encryption, scanning and tags
CreateRepository parsed none of encryptionConfiguration, imageScanningConfiguration or tags, so none were stored and none came back. Both blocks are now parsed, persisted on RepoMeta and projected, defaulting to AES256 and scanOnPush false as AWS does.
encryptionType KMS is rejected rather than echoed. Predastore seals every fragment with AES-256-GCM under a required master key, so reporting AES256 states something true, but no customer-managed key material is plumbed anywhere and accepting a KMS key ARN would assert a protection that does not exist.
Repository tags are stored on the same record, and TagResource and UntagResource now work instead of returning NotImplemented.
PutImageScanningConfiguration gets its own handler: asking for scanning off succeeds as a no-op, which is the already-true state, while asking for it on still returns OperationNotSupportedException because no scanner exists. Reporting scanning as enabled when nothing scans would read as a false all-clear.
The AWS Repository shape was built in three places with no shared helper, so every new field would have been added three times. They now share one buildRepository projection.
Closes: mulga-ywvtu
Closes: mulga-j2y54
Closes: mulga-3whvb
* fix: accept and store repository tags on ECR CreateRepository
The AWS API takes a tags parameter on CreateRepository; the gateway accepted
the request and discarded them, leaving the repository untagged with no error.
Decode them into the RepoMeta tag map at create time, rejecting an empty key
the same way TagResource does.
DescribeTaskDefinition now echoes portMappings[].name and runtimePlatform,
DescribeServices projects the stored networkConfiguration and
enableECSManagedTags, DescribeClusters returns the four instance, task and
service counts, and the five list operations return their ARN key as a
present empty list instead of omitting it.
UpdateService now honours networkConfiguration and enableECSManagedTags so a
changed value converges instead of producing a permanent diff.
The eleven container runtime fields the task definition also drops are
deliberately left absent: nothing between the daemon and the on-host agent
carries them, and reporting a container as privileged or read-only-root when
nothing enforces it is worse than reporting nothing.
Closes: mulga-0t9q2
Closes: mulga-7thn7
Closes: mulga-d8kif
Closes: mulga-5z8z9
Three defects at the gateway's HTTP error boundary, none of which change an authorisation outcome.
xmlErrorBody defaulted to the EC2 query envelope, so any scope it did not enumerate, including every service the gateway does not serve, received a shape a REST-XML client cannot deserialize and the real code never surfaced. The generic ErrorResponse is now the default and the query-protocol services are named explicitly. An unparsed signature keeps the EC2 shape, since no service is known there and that path is not what this changes.
The supportedServices gate logged the truthful reason and then emitted the canned credentials text, because writeSigV4Error had no parameter to carry a message. It takes one now, and the gate names the service it refused. The signature is never verified at that point, so blaming the caller's credentials was actively misleading.
ctxAction was set only for the query protocol, so every JSON-1.1 request shared the throttle bucket "<account>:unknown" and no per-action limit could ever match. It now resolves from X-Amz-Target, and for the path-routed rest-json services from the route's own lookup function.
Also sets X-Amzn-Errortype on the four JSON emitters, which JSON-protocol SDKs read to resolve the modelled exception type.
Recovery tested whether the QEMU process was still running before it looked at the instance's state, so a live process short-circuited the transitional-state handling and reconnectInstance then set the instance back to running. A stop or terminate the caller had already been told was under way was silently discarded.
Check for StateStopping and StateShuttingDown first, kill a QEMU that outlived the transition, and finalize. reconnectInstance now refuses a transitional instance outright rather than promoting it.
Whether br-mgmt is a per-host island or one L2 segment spanning the cluster
decides whether a system instance can reach the gateway address it was baked
with. No artifact recorded which one a run built, and the environment is
destroyed before it can be asked, so the decisive evidence for an LB agent
heartbeat failure was gone by the time anyone looked.
Records ports, address and fail_mode per node, then probes every node's
br-mgmt address from every node. Islands reach only themselves and spanned
segments reach everything; a mixed result is the failure this catches.
* fix: carry a handler's own refusal message to the client without leaking wrapper context
A daemon that explains a refusal had that explanation discarded one step short of the client, so every NATS-backed error surfaced as the generic ErrorLookup blurb for its code. The message crossed NATS intact; resolveErrorDetail then dropped it because decodedNATSError was not the shape it recognises.
Rebuild the decoded response through awserrors.Errorf so the walk finds the message, and send only ResolveErrorDetail's curated message from the two encode sites instead of err.Error(). The latter rendered every %w layer, including context a call site added purely for its own logs, so restoring the message without also fixing the encode side would have turned a fidelity bug into an information disclosure.
Two existing tests pinned that wrapper context onto the wire, directly contradicting the gateway's own no-leak assertion; both are rewritten to pin the sanitized contract instead.
Also name the rejected filter in the twenty-four ParseFilters sites that logged the detail and returned a bare code, and give the ACM force-renew refusals a registered code so their reason still reaches the caller.
* fix: stop the filter rejection repeating its own error code to the client
ParseFilters spelled the code into its message text, so once the code began travelling separately the client saw it twice: "InvalidParameterValue: InvalidParameterValue: The filter 'x' is invalid".
Give ParseFilters the code directly via awserrors.Errorf and have its twenty-five callers return that error unwrapped rather than re-wrapping it. All of them already mapped it to InvalidParameterValue, so nothing changes but the duplication.
* fix: surface why a security group rule was rejected
ipPermissionsToSGRules already names the reason, "IPv6 rules are not supported" or "IpPermission must specify at least one IpRange or UserIdGroupPair", and both Authorize call sites logged it and returned a bare InvalidParameterValue, so the caller saw only the canned blurb.
This is the case mulga-e9qqx names specifically. It does not make IPv6 rules work; it makes the refusal legible.
CreateCluster and CreateNodegroup accepted fields that DescribeCluster and DescribeNodegroup never returned. The unmodified terraform-aws-modules/eks/aws module reads those fields back, finds them empty, and either aborts the apply or proposes a destroy of a cluster that is already correct.
Two of the omissions aborted the apply outright. A module precondition on data.aws_eks_cluster failed with cluster_service_cidr is "" because kubernetesNetworkConfig was absent, and the tag resource address built from clusterSecurityGroupId came out as (,suite) because that field was empty.
Store what the caller asked for on ClusterMeta and NodegroupRecord at create time, and project it back. Both records are encoding/json with omitempty, so records already in KV stay readable and project exactly as they did before.
The read-backs echo the request rather than synthesising an AWS default, because UpdateClusterConfig is not implemented — a value the caller did not ask for would be a diff they could never converge. Where a field does report a default it is the value genuinely in force: 10.43.0.0/16 is k3s's own service CIDR, API is the only authentication mode implemented, and logging reports every type disabled because no control-plane log export exists.
resourcesVpcConfig.securityGroupIds previously overwrote the caller's own groups with the platform's pair, which live in a VPC the caller cannot see. It now echoes the caller. The customer-facing cluster security group is created in the caller's account and VPC at cluster-create time, so the id reported by clusterSecurityGroupId resolves for the tenant that owns it. That creation is idempotent with the one launchNodegroupInfra already makes, and matches AWS, which creates the group with the cluster.
updateNodegroupConfig accepted UpdateConfig and Taints, ignored both, and returned SUCCESSFUL. It now applies them under the existing compare-and-swap, so a plan converges instead of asking forever. Caller-supplied labels and taints still never reach kubelet; that gap predates this change and is tracked separately.
Verified on env19: the module applies clean and a re-plan reports no changes.
Six defects in the EC2 describe path, found by running unmodified Terraform Registry modules against a live cluster and planning immediately after a clean apply. They share one shape: the write succeeds, the read-back is incomplete, and the caller cannot tell.
Populate Association on the primary network interface. The public IP was projected onto the instance but never onto the NIC, so the AWS provider read associate_public_ip_address as false for an instance that had a working public address. That attribute is force-new, so a no-op plan proposed destroying and recreating the instance.
Stamp EnclaveOptions. AWS always returns it, and its absence rendered as a permanent enclave_options block in every plan.
Stop emitting empty Reservations. The reservation map entry was created before the filter predicate ran, so a filter that excluded every instance in a reservation still emitted the reservation with no instances. Projection and filtering now run before the insert, in both the live and the KV path.
Merge reservations by ID in the gateway. Response frames were concatenated rather than merged, so one reservation could appear twice in a single response and which copy landed first varied between calls.
Mint the reservation ID once per RunInstances call. Each node minted its own, so a launch spread across nodes came back as several reservations. The gateway now allocates the ID and passes it to every node in a NATS header; a node that does not read the header mints its own as before, so the change is additive in both directions.
Answer an incomplete DescribeInstances sweep with a retryable ServiceUnavailable. Suppressing a false InvalidInstanceID.NotFound when a node had not answered was correct, but no branch was ever written for that case, so the caller received an empty 200 that read as authoritative. A filters-only query with no matches is still a legitimate empty result.
Accept the AWS spellings of the DescribeVpcs filters. The filters were implemented but named cidr-block and is-default where AWS uses cidr and isDefault, so Terraform's aws_vpc data source was rejected outright. Both spellings now work, since an internal caller depends on the old one. The parse error also names the offending filter to the client instead of only the log.
Resolve image owner aliases before filtering on them. describe-images --owners self compared an alias against a numeric account ID and matched nothing; the correct resolution already existed but ran after the filter.
Also builds the segscan test oracle with -buildvcs=false. That is unrelated to the above and fails on current toolchains regardless of this change, but it blocks preflight.
Host reservations, ECR image scanning and the flow-logs integration template
were declared not applicable, and the reserved instance family with them. None
of those are ruled out: reservations and scheduled instances can be mocked over
capacity reservations the way spot instances already are, scanning needs only a
scanner behind the API, and the template needs CloudFormation.
What stays is what the platform cannot have: retired AWS features, operations
whose content is an AWS price or counterparty, and AWS-hosted services with
nothing behind them here.
The roadmap carries IPv4 pools, local zones as an installation option, and an
S3 gap closure naming inventory and intelligent tiering. All three were
published as operations this platform will never serve.
Twenty-nine EC2 operations — local gateways, CoIP pools, BYOIP and the zone
group opt-in — and nine S3 operations are no longer declared. Carrier gateways
stay, as a route into a telco's 5G network in a Wavelength Zone.
The internal report listed every unimplemented operation together, so the 87
EC2 operations declared not applicable sat in the same list as the work, where
anything reading it for candidates would pick them up.
They are now listed apart, grouped under the reason they were declared with,
and the counts line reports them as not applicable rather than as gaps.
EC2 published one not-applicable operation against 625 in the model, so the
page was silent about a large surface that is absent for a structural reason
rather than an unfinished one.
Eighty-six more are declared across nine families: EC2-Classic, the 2016
resource ID migration, FPGA images, Elastic Graphics, Outposts and the edge
zone gateways, capacity purchasing, AWS backbone telemetry, Marketplace
product codes and BYOIP. S3 loses its replication and analytics declarations,
which named work that is not ruled out.
Multi-AZ clusters, read replicas and point-in-time restore were declared not
applicable, which says the platform will never serve them. All three are
planned, so the pages claimed something the roadmap contradicts.
Only a declaration now publishes that claim. A handler that refuses an
operation today establishes nothing about tomorrow, so it reads as the gap it
is rather than a permanent one.
Splitting them into a second table repeated a reason on every row that shared
it and separated an operation from the rest of the surface it belongs to. One
table carries every operation the platform answers, served or refused, and the
reasons return to numbered footnotes below it.
S3 was the one served surface with no coverage page. The gateway has no
operation-name dispatch table for it, so there was nothing to compare against
the model.
Predastore now declares the operations each of its S3 routes answers, so the
generator compares that inventory the same way it compares the other nine
services. The page credits Predastore rather than Spinifex, and its intro
separates what is routed from what conforms, pointing at the ceph/s3-tests
report for the second.
The index listed a count per service and left the reader to add them up. It
now opens with the total and closes the table with it, counting only the
services it links.
The coverage pages led with a percentage and an "x of y" count, and every
table listed the modelled operations Spinifex does not implement, which reads
as a scorecard against AWS rather than a statement of what the platform serves.
A published page now opens with the operation count alone, tables the
operations the gateway serves, and carries a separate not-applicable table with
each reason inline instead of footnotes. The unimplemented operations and the
counts move to docs/coverage/internal-report.md, which no docs-site slug points
at. A service with no implemented operations has no publishable page.
Every node's startup pass, the account.created event and the gateway's
EnsureDefaultVpc request can each build an account's defaults on a different
node. Each checked for an existing record, then wrote a fresh random ID with
Put, so racing nodes built a VPC, security group, route table and internet
gateway each. A per-account claim, written with Create, now fixes the IDs
before anything is built; every record is created under those IDs with
Create, and only the creator announces it to vpcd. nextVNI retries its CAS
instead of failing the loser.
spx admin init now discards the node's pre-formation JetStream store by default for --nodes >= 2; --discard-jetstream=false restores the refusal. spx admin kv digest gains --json with stream replica, retention and limit settings and per-sequence subject, time and TTL. The new spx admin kv compare reads every node's digest and fails only on real divergence: different content at one sequence, a missing message nothing explains, or wrong replica placement. install-node.sh gates formation on it and gains --ipsec on|off, verified on every node. The multi-node guide documents both.
A weekend gap is two runs of evidence lost on a suite whose remaining
failures are a rate rather than a bug. GitHub queues scheduled runs behind
every other top-of-hour cron and sheds them under load, which is why
2026-09-14 never fired and why every run started 3h15m-3h45m late.
* fix: reboot a guest by asking it, not by pulling its reset line
RebootInstances was QMP system_reset alone. The guest kernel is never told,
so it does not sync, does not unmount and loses every dirty page - which is
how a reboot issued seconds after a write loses that write, and how a guest
rebooted during cloud-init can fail to come back at all.
Guests now launch with -action shutdown=pause, so a clean guest poweroff
leaves QEMU paused rather than gone. Reboot presses the power button, waits
for the guest to reach that state, then resets and resumes the same process:
no volume, tap or state churn, and no window where the guest is missing. A
guest that will not go quietly is hard-reset, which is what AWS documents.
The button is pressed again every 10s while the guest ignores it. One signal
is not a shutdown request on a guest whose acpid has not started, which is
the same boot race that produced the failure.
Two consequences handled here: a guest that powers itself off now leaves
QEMU paused, so the heartbeat quits it; and a guest launched before the flag
exits on powerdown rather than pausing, so it is reset without being asked.
* fix: answer a reboot when it is accepted, not when the guest is back
Asking a guest to shut down takes as long as the guest takes. RebootInstances
gives the daemon five seconds to reply, so a reboot that waited for the guest
reported a reboot that had in fact succeeded as InvalidInstanceID.NotFound.
Manager.Reboot now admits the reboot — the instance exists and is running — and
runs the powerdown and reset in the background, which is the contract EC2
documents. The two admission failures are still reported; nothing after them
can be, so a reset that fails is logged and left to the heartbeat.
The gateway also stops calling a NATS timeout NotFound. Only the absence of a
subscriber says anything about where the instance is, which is how the sibling
start and terminate paths already read it.
* test: prove the churn reboot by its boot ID, not by how fast it happened
The subtest waited ten seconds and then read /proc/uptime. A reboot is
asynchronous, so the guest answering at that point may be the one that has
not shut down yet, and its uptime is whatever it was before — which the test
reads as "reboot may not have occurred".
Waiting for the boot ID to change is positive proof the guest rebooted and
does not depend on how long the guest takes to go down. The uptime assertion
stays, now measured against a boot that is known to be new.
* test: prove the graceful reboot against real guests and a real engine
Two tests the reboot design asked for and the unit tests cannot give.
Cell 18 launches two guests that differ only in how they answer the power
button. One installs a shutdown hook writing an fsynced line a second, so the
file that survives measures how much of its shutdown it was allowed to run —
a guest reset instead of asked leaves none of it. The other ignores the power
key entirely and must be reset once its budget expires, with the daemon saying
so. Each variant asserts the evidence only it can produce, and the slow one
asserts the absence of the line the wedged one requires.
Cell 27 loads a PostgreSQL instance past its shared buffers with pgbench and
reboots it dirty. The probe is an UNLOGGED table: PostgreSQL writes no WAL for
one, so a clean shutdown checkpoints it and crash recovery truncates it
unconditionally. Rows committed normally survive a hard reset too, so the
obvious assertion would pass against the bug.
The rds suite and cell 27 both gain budget for the load phase.
* fix: stop the process-group test misreading a reaped child as alive
The test polled until the child was gone, then checked once more. A task
being reaped shows state X for an instant, which processRunning counted as
running, so that second look failed about 1 run in 50 under -race.
* fix: wait for the recovery retry loop before removing a test's data dir
restoreInstances starts the retry loop untracked, and it rewrites the local
state file when it ends. A write landing during t.TempDir cleanup failed
TestRestoreInstances_* with RemoveAll: directory not empty.
* fix: read NB_Global.ipsec through the NB remote list so followers and compute nodes answer quietly
* fix: stop probing and nudging the EIP datapath of a stopped instance's port
* fix: log the real client IP, request ID and duration_ms in the awsgw access line
* test: mark the UI proxy header test as in-package
* fix: discard pre-formation JetStream data when forming a multi-node cluster
A node that ran as its own single-node cluster carries KV streams into
formation. nats-server adopts a local stream with the same name as the
clustered one, and raft catch-up only fetches sequences beyond a
follower's last, so the conflicting prefix is never repaired and replicas
diverge permanently.
- spx admin join removes the local JetStream store once formation
succeeds. A store holding streams now needs --force, and the
identity-discard warning names it.
- spx admin init --nodes >= 2 refuses a store that holds streams unless
--discard-jetstream is given, and removes it once every node has joined.
Keeping it as the seed is unsafe: stream leadership ignores who holds
data, and unassigned local streams are deleted as orphans.
- Neither removes a store while `spx service nats` runs, --force included.
- install-node.sh passes --discard-jetstream and waits for every
`spx service` process to exit before formation.
* feat: add spx admin kv digest for per-replica JetStream consistency checks
Replicas of a stream can hold different content while raft reports them
current, and nothing in the cluster detects it. A direct get over NATS is
no way to inspect one replica: a client's request goes to a random member
of the direct-get queue group, routed replicas included.
kv digest copies this node's $G streams tree (without consumer state) to a
temporary directory, opens the copy in a private in-process server with no
listener, and prints first/last sequence, message count and a SHA-256 over
every stored message per stream. Running it on each node and diffing the
output shows which streams diverge; --seqs locates the first differing
sequence. The live store is only ever read.
* fix: let spx admin kv digest read replicated streams
A replicated stream's meta.inf carries num_replicas > 1, which a standalone
server refuses to recover, so the digest could not read any production store.
Rewrite the copy's config to one replica with no expiry and a recomputed
checksum, and test it against a three-node cluster and an expired bucket.
* test: use assert.ErrorIs for the removed JetStream store
* test: mark the JetStream store and digest tests as in-package
* fix: stop the process-group test misreading a reaped child as alive
The test polled until the child was gone, then checked once more. A task
being reaped shows state X for an instant, which processRunning counted as
running, so that second look failed about 1 run in 50 under -race.
* fix: wait for the recovery retry loop before removing a test's data dir
restoreInstances starts the retry loop untracked, and it rewrites the local
state file when it ends. A write landing during t.TempDir cleanup failed
TestRestoreInstances_* with RemoveAll: directory not empty.
* fix: stop the process-group test misreading a reaped child as alive
The test polled until the child was gone, then checked once more. A task
being reaped shows state X for an instant, which processRunning counted as
running, so that second look failed about 1 run in 50 under -race.
* fix: wait for the recovery retry loop before removing a test's data dir
restoreInstances starts the retry loop untracked, and it rewrites the local
state file when it ends. A write landing during t.TempDir cleanup failed
TestRestoreInstances_* with RemoveAll: directory not empty.
* fix: retry a recovery relaunch that lost a race with a starting dependency
A guest drain-stopped by a coordinated cluster shutdown could be lost for good.
The relaunch ran before JetStream was serving, the volume lease claim timed out,
and the failure was recorded as permanent.
The backoff this needed already existed. relaunchWithRetry retries five times,
but only on ErrMountRetryable, and mountErrRetryable recognised exactly two
viperblock state-load sentinels. A lease claim returning context deadline
exceeded matched neither, so the mount was reported as permanently failed and
the backoff never engaged once.
- Classify a lease store that cannot answer as ErrLeaseStoreUnavailable, and
include it in mountErrRetryable. A lease held by another node is deliberately
excluded: that is an answer, not an outage.
- Probe readiness rather than connectedness. checkViperblockReady tested
natsConn.IsConnected(), which is true against a server whose JetStream is
still starting; it now calls AccountInfo with a timeout.
- Reconsider a latched failure. RetryRecoveryFailed relaunches instances parked
in error by a dependency-unavailable reason, once the backing store is
confirmed ready, for a bounded number of passes, then leaves them alone.
Only mount-state and launch failures qualify; a failed reconnect or
pre-relaunch hook can mean the guest itself needs looking at.
Closes the e2e gap that made this feel covered. The node-failure tests stop the
service units directly so guests keep running and recovery is a QMP re-attach to
a live process. Nothing stopped a guest, so nothing relaunched one from a sealed
volume against dependencies that were still starting.
TestMultinodeClusterRestartResumesGuests does, including the wait for every spx
process to exit that makes it a real restart rather than a stop that returned
early.
Beads mulga-v7kp4, mulga-kyb1d.
* fix: stop every spx process by unit in the cluster-restart helper
The first CI run of TestMultinodeClusterRestartResumesGuests failed at the
process wait with "node1 still has 2 spx process(es)", and the same thing
happened by hand on a prod node.
Every service is the same binary, and the stop was driven from a hard-coded
list. StopNode names seven units and omits spinifex-northstar and
spinifex-qmp-collector, which is exactly the two survivors. The unit name now
comes from the surviving process's cgroup, so the set that stops is whoever is
still running the binary and the list cannot go stale when a service is added.
StopNode itself is left alone: its list is deliberate for the hard outage it
simulates, where the target's drain must not fire.
Also marks recovery_retry_test.go as //test:in-package — the retry policy is
unexported and the tests drive it through the seams restore_test.go uses.
* fix: stop spinifex.target in the cluster-restart helper, not the units
The second CI run reformed NATS to 2 peers of 3: node3's NATS was stopped and
never started again.
StopNode stops the service units directly and leaves spinifex.target active.
That is right for the hard outage it simulates and wrong for a real shutdown —
the target then reads as satisfied while its units are dead, and the later start
raced a stop job that was still queued.
The helper now runs what the operator procedure and update-nodes.sh run: stop
spinifex.target, wait for every spx process to exit, and additionally wait for
systemd to have no spinifex job left queued before starting. That last wait is
the one that closes this race.
* fix: wait for the right NATS peer count in the cluster-restart helper
WaitNATSPeers counts remote peers, so a healthy three-node cluster reports two.
Asking for three could never be satisfied, and the run that hit it had in fact
restarted all three NATS correctly.
The QEMU guard never fired. pgrep -x and pkill -x match comm, which the kernel
truncates to 15 characters, so qemu-system-x86_64 is stored as qemu-system-x86
and neither the -x name nor the -x alternation could ever match. The script
therefore never signalled a guest and never waited, and wiped viperblock state
out from under live guests. Match the 11-character prefix against comm, the same
rule qemuProcessPrefix already applies in vm/orphan_scan.go, and scope it to the
user the daemon runs guests as so a host carrying libvirt guests alongside
Spinifex ones does not lose the wrong set. Not -f: that matches the whole
command line, where argv[0] may carry a path.
Nothing waited for spx either. Stopping spinifex.target returns once the target
is inactive, but spinifex-shutdown.service is ordered After= the storage
services, so their stop jobs queue behind its ExecStop and NATS can still be
running. JetStream then rewrites its index after the rm and before the
verification, failing the reset on a file the script had already deleted. Add
the wait update-nodes.sh has carried since d9bc6e35.
The firewall restore was gated on a mode file the failed run had already
deleted, so a retry read the absence as "never armed", skipped the restore, and
install-node.sh honoured that and formed the cluster around an open node. Stash
the mode outside every wiped tree, and fail rather than warn when a node that
arrived armed cannot be put back.
install-node.sh had the other half of that cascade: it disarms every host to let
the OVN raft ports through and only re-arms at the end, so any abort in between
left them all open. Re-arm from an EXIT trap, and say so loudly when a host
arrives with no policy.
The wipe loop deletes files but keeps every directory, so one empty
$G/streams/KV_<name> directory per stream survived the reset. NATS enumerates
those on start and reports them as HA assets it must restore, leaving the meta
group at size=4 with no leader over hundreds of assets that have no data behind
them. Every service then fails to reach JetStream and the daemon aborts its
bootstrap.
Remove the tree outright, mirroring the predastore/cluster removal above it.
The existing docker push e2e test pushes a single 4KB scratch layer, too
small to exercise predastore's storage path for a real multi-hundred-MB
blob. Add a 3-layer (100+80+60MiB) push/pull case that verifies re-pulled
layer digests against the pushed image and directly checks Content-Length
on a raw HTTP GET of the largest blob, targeting a reported defect where
large blob GETs returned 200 with content-length: 0. Re-pull uses
`docker builder prune -af`, not just `rmi`, since buildx's build cache can
otherwise satisfy the pull without touching the registry.
* ci: benchmark Predastore across a real 3-node Spinifex env
The nightly throughput job runs three s3d processes on loopback aliases on a
single box against local NVMe, so nothing measures Predastore in the shape it
ships in: three machines, with reads crossing a real NIC between hosts.
This provisions a 3-node env the way the multi-node e2e job does, then hands
its three gates to the existing harness through PERF_EXTERNAL_HOSTS. The
harness skips its own start and stop and measures them directly, so the
workloads and analysis are unchanged and the two numbers stay comparable.
It runs on a systemd LAN runner rather than the ephemeral pool because a pool
VM sits behind the geneve overlay at MTU 1408, which would put the runner's
own network path inside the measurement.
* ci: trigger the 3-node benchmark on its own branch for a first run
A workflow_dispatch only resolves against the default branch, so this cannot
be dispatched until it is merged. The push trigger is scoped to this branch
and comes back out once the workflow is on dev.
* fix: measure as the identity the provisioned cluster actually mints
spx admin init generates a random access key pair at bootstrap and writes it
to node1's ~/.aws/credentials, so the benchmark's built-in key is one this
cluster has never heard of and every run died on the first CreateBucket.
The pair is read and used inside a single step, the way e2e-ddil.yml does it:
a step-level env: block is echoed verbatim into the run log, a shell variable
is not.
* ci: default the 3-node benchmark to the compare preset
smoke was the right default while the pipeline was unproven. It is green now,
and compare is what the nightly loopback run uses, so it is the preset whose
numbers can be set beside it.
* ci: make the cluster's ref a knob, pinned to v1.18.0 for one run
The harness stays on this branch while the cluster is built from whatever ref
is asked for, so an old release is measured with the same ruler as the current
one. run-info.txt records the deployed ref rather than the harness checkout.
The v1.18.0 default is temporary: a push event carries no dispatch inputs, and
this cannot be dispatched until it is on dev. It comes back out after the run.
* test: probe ranged GETs, which no workload here exercises
Warp reads whole objects, so the throughput figures stay healthy while ranged
GETs read the whole object per request. The probe reads the same boundary
crossing range from a 64 MiB and a 256 MiB object: under read amplification
the cost tracks object size, and otherwise the two match.
Cluster ref is pinned to the fix branch for this run and comes back out after.
* test: run the ranged GET probe against dev to prove it fails there
A probe that has only ever been green proves nothing about the fix. Pointing
the cluster at dev, which still has the defect, is the control that shows the
probe detects it. The pin comes back out after.
* ci: drop the branch push trigger and the cluster ref pin
Both were scaffolding to run this before it existed on dev. The workflow is
dispatch only again, and cluster_ref defaults to the branch under test.
<svgxmlns="http://www.w3.org/2000/svg"width="1020"height="836"viewBox="0 0 1020 836"role="img"aria-label="Spinifex on a single Exoscale instance: the instance sits on the zone's public network with one interface, eth0, holding its own address and the default route. Manual Elastic IPs are delivered to eth0 and routed by the host into OVN through the spx-nat transit veth, and an Exoscale block storage volume backs /var/lib/spinifex.">
<textx="28"y="42"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="11.5"fill="#EE4B12"font-weight="600"text-anchor="start"letter-spacing="2.5">SPINIFEX ON EXOSCALE — SINGLE NODE</text>
<textx="50"y="218"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="11.5"fill="#EE4B12"font-weight="600"text-anchor="start"letter-spacing="2">EXOSCALE ZONE DE-FRA-1 · PUBLIC NETWORK 91.92.116.0/23</text>
<textx="970"y="218"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="10.5"fill="#9AA1A8"font-weight="400"text-anchor="end">gateway 91.92.116.1 · security group open to all</text>
<textx="932"y="328"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="10.5"fill="#5B636C"font-weight="400"text-anchor="end">attached to the instance</text>
<textx="538"y="356"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="12"fill="#F3F1EC"font-weight="400"text-anchor="start">89.145.163.1/32 → delivered to eth0</text>
<textx="538"y="378"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"text-anchor="start">the host routes each /32 into OVN, never owns it</text>
<textx="296"y="449"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#9AA1A8"font-weight="400"text-anchor="start">transit 100.127.0.0/24 · EIP/32 routed in · masquerade out eth0 for the rest</text>
<textx="88"y="512"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="12"fill="#9AA1A8"font-weight="700"text-anchor="start"letter-spacing="1.5">SPINIFEX CONTROL PLANE</text>
<textx="932"y="512"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="10.5"fill="#5B636C"font-weight="400"text-anchor="end">every service is one binary: spx service <name></text>
<textx="28"y="796"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"text-anchor="start">A guest's public address is a manual Exoscale Elastic IP, delivered to eth0 addressed to itself — no private/public pair to translate.</text>
<textx="28"y="814"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"text-anchor="start">The host routes it into OVN, so inbound and outbound are the same address and the guest sees its private IP only.</text>
<svgxmlns="http://www.w3.org/2000/svg"width="1020"height="796"viewBox="0 0 1020 796"role="img"aria-label="Spinifex across three Exoscale instances in one zone, spread by an anti-affinity group. Each has a public eth0 carrying its own Elastic IPs, a private-network eth1 for the cluster mesh, its own block storage volume and its share of the guests. Geneve, NATS JetStream, predastore erasure coding and the OVN raft join them east-west over the private network. This topology is not yet tested.">
<textx="28"y="42"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="11.5"fill="#EE4B12"font-weight="600"text-anchor="start"letter-spacing="2.5">SPINIFEX ON EXOSCALE — THREE NODES</text>
<textx="992"y="42"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="11.5"fill="#9AA1A8"font-weight="400"text-anchor="end">anti-affinity group · one zone · not yet tested</text>
<textx="48"y="676"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="12"fill="#9AA1A8"font-weight="700"text-anchor="start"letter-spacing="1.5">EAST–WEST, OVER THE EXOSCALE PRIVATE NETWORK</text>
<textx="972"y="676"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="10.5"fill="#5B636C"font-weight="400"text-anchor="end">no Elastic IP is involved · MTU 1500</text>
<textx="28"y="744"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"text-anchor="start">Guest-to-guest traffic rides Geneve between the nodes' private-network interfaces — it never touches eth0 or an Elastic IP.</text>
<textx="28"y="762"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"text-anchor="start">Each node carries the Elastic IPs of its own guests. Moving one between nodes is an Exoscale API call, not yet automated.</text>
<textx="28"y="780"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"text-anchor="start">Three nodes is the smallest cluster that tolerates losing one; the anti-affinity group keeps them on different hypervisors.</text>
<svgxmlns="http://www.w3.org/2000/svg"width="1020"height="836"viewBox="0 0 1020 836"role="img"aria-label="Spinifex on a single OCI instance: the instance sits inside the VCN public subnet 10.200.0.0/23; its primary VNIC carries the default route and the cluster mesh, its secondary VNIC is the br-wan bridge holding registered secondary private IPs paired with reserved public IPs; guests reach them through the spx-nat transit veth, and an iSCSI block volume backs /var/lib/spinifex.">
<textx="28"y="42"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="11.5"fill="#EE4B12"font-weight="600"text-anchor="start"letter-spacing="2.5">SPINIFEX ON OCI — SINGLE NODE</text>
<textx="992"y="42"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="11.5"fill="#9AA1A8"font-weight="400"text-anchor="end">one instance · routed NAT · real public addresses</text>
<textx="88"y="512"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="12"fill="#9AA1A8"font-weight="700"text-anchor="start"letter-spacing="1.5">SPINIFEX CONTROL PLANE</text>
<textx="932"y="512"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="10.5"fill="#5B636C"font-weight="400"text-anchor="end">every service is one binary: spx service <name></text>
<textx="28"y="796"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"text-anchor="start">A guest's public address is a real OCI reserved public IP, attached to a secondary private IP on br-wan.</text>
<textx="28"y="814"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"text-anchor="start">Inbound and outbound are therefore the same address, and OCI's per-VNIC source check passes.</text>
<svgxmlns="http://www.w3.org/2000/svg"width="1020"height="796"viewBox="0 0 1020 796"role="img"aria-label="Spinifex across three OCI instances: all three sit inside the VCN public subnet 10.200.0.0/23, one per fault domain. Each carries a primary VNIC for the cluster mesh and a br-wan secondary VNIC holding its reserved public addresses, its own iSCSI block volume, and its share of the guests. Geneve, NATS JetStream, predastore erasure coding and the OVN raft join them east-west.">
<textx="28"y="42"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="11.5"fill="#EE4B12"font-weight="600"text-anchor="start"letter-spacing="2.5">SPINIFEX ON OCI — THREE NODES</text>
<textx="992"y="42"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="11.5"fill="#9AA1A8"font-weight="400"text-anchor="end">three fault domains · one region · one API endpoint</text>
<textx="68"y="554"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="12"fill="#9AA1A8"font-weight="700"text-anchor="start"letter-spacing="1.5">EAST–WEST, INSIDE THE VCN</text>
<textx="952"y="554"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="10.5"fill="#5B636C"font-weight="400"text-anchor="end">no public address is involved</text>
<textx="66"y="612"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"text-anchor="start">Guest-to-guest traffic rides Geneve between the nodes' primary VNICs — it never leaves the subnet and never touches br-wan.</text>
<textx="48"y="688"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="13"fill="#F5F3EE"font-weight="700"text-anchor="start">A guest that moves takes its address with it.</text>
<textx="48"y="705"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#9AA1A8"font-weight="400"text-anchor="start">The reserved public IP follows the secondary private IP to whichever node's br-wan the guest lands on — the tenant sees no change.</text>
<textx="28"y="756"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"text-anchor="start">Three nodes is the smallest cluster that tolerates losing one: NATS JetStream keeps quorum, predastore RS(2,1)</text>
<textx="28"y="774"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"text-anchor="start">keeps the data readable, and the OVN Northbound raft keeps a leader. One node per fault domain is what makes that real.</text>
<!-- Complete this section only when the PR changes account-scoped resource lifecycle, account termination, resource cleanup, or a lifecycle boundary. Otherwise write: N/A. -->
- Lifecycle owner and resource family:
- Owner-local ordering, recovery, retained-state, or force impact:
- [ ] All community contributions in this pull request are licensed to Mulga under the terms of the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). By creating this pull request I represent that I have the right to license these contributions to Mulga under the Apache License 2.0.
# Dispatched by the same Cloudflare Worker that drives e2e-nightly.yml, once per
# ref, rather than by a `schedule:` here. GitHub registers a schedule only from
# the default branch, so a cron in this file could never run `dev` and a feature
# branch as separate nightlies, and it would always execute the default branch's
# copy of this file -- leaving a fix to the workflow itself untestable until it
# had already shipped. A dispatch takes both the file and the source from its ref.
on:
workflow_dispatch:
inputs:
topologies:
description:"Space-separated topologies (blank = vm-single vm-multi bm vm-single-principal). One name runs one topology."
required:false
workbooks:
description:"Space-separated workbooks (blank = the driver's default list)"
required:false
source:
description:"tree (build this ref) or release (the published installer)"
required:false
default:tree
channel:
description:"With source=release: latest, or dev for the newest prerelease"
required:false
default:latest
keep_on_fail:
description:"Leave a failed topology up to inspect. It keeps billing."
required:false
type:boolean
default:false
auth:
description:"How nodes reach the OCI API: instance_principal (needs the tenancy dynamic group) or api_key (needs the credential hook). The vm-single-principal topology always uses instance_principal."
required:false
default:instance_principal
type:choice
options:[instance_principal, api_key]
no_external_pool:
description:"Form without the OCI allocator. Skips every workbook needing a public address. Not applied to vm-single-principal."
required:false
type:boolean
default:false
runner:
description:"Runner label (blank = the OCI_E2E_RUNNER variable, else ubuntu-26.04)"
required:false
permissions:
contents:read
# Every topology is built in one OCI compartment under one set of resource names,
# so two runs at once would each see the other's instances and the sweep in
# e2e-oci.sh could destroy a cluster the other run was still testing. Queue the
# second rather than cancelling it: a dispatch is usually a verification run.
concurrency:
group:e2e-oci
cancel-in-progress:false
jobs:
oci:
# Hosted by default: this job only drives Terraform and SSH, and the work
# happens on OCI, so nothing here needs a hypervisor. Overridable because the
# argument for a per-provider self-hosted runner is control, not capacity --
# a dedicated label keeps a two-hour cloud wait off the hypervisor labels and
description:"Comma-separated cell numbers to run, no spaces (blank = all). 1-3, 8-10, 17-19 tofu permutation matrix, 34-36 Oracle Linux single node (basic, reboot, routed NAT), 20 baremetal, 21-22 ISO, 23 predastore throughput, 24 predastore fault injection, 25-27 service suites (eks, ecs, rds), 28 guest disk performance, 29 storage fault injection, 30 routed NAT on three nodes, 31 instance recovery on three nodes, 32 load balancer recovery on three nodes, 33 KV bucket quorum under node loss on three nodes"
required:false
permissions:
contents:read
# actions: read lets the "Resolve CI job id" step below look up this job's
# numeric id via the Actions API — github.job is only the YAML key, no
# context exposes the id itself.
actions:read
# Every cell builds its cluster on the same five hypervisors, so two runs in
@@ -21,37 +22,29 @@ permissions:
# second rather than cancelling it: a dispatch is usually a verification run
# and discarding it loses the result someone is waiting on.
#
# The group is shared with e2e.yml, e2e-suites.yml and e2e-ddil.yml, which draw
# on the same runner pools. A per-workflow group let a push-triggered E2E land
# on top of a nightly cell and take libvirt down under both.
# The group is shared with e2e-suites.yml and e2e-ddil.yml, which draw on the
# same runner pools. e2e.yml is deliberately outside it: a single-cluster push
# run is cheap enough to overlap, and grouping it stalled every push behind a
# multi-hour nightly.
concurrency:
group:e2e-hypervisors
cancel-in-progress:false
# Scheduled runs are loaded from the default branch (main), so the schedule
# trigger must stay in sync on main for cron to fire — and so must the
# concurrency block above. main currently has none, which is why the nightly
# cron runs ungrouped and a dispatch on any branch can land on top of it.
Report security vulnerabilities privately through [SECURITY.md](SECURITY.md), not as a public issue.
**AWS parity is the design rule.** When Spinifex's behaviour is in question, the answer is what AWS does. If you believe Spinifex diverges from AWS, state what AWS does and show it with a test or a reproduction against a running cluster.
## Understanding the Codebase
Every AWS call enters through a SigV4-authenticated gateway, is published to NATS, and is answered by the daemon that claims it. Start with these:
User-facing docs are published at [docs.mulgadc.com](https://docs.mulgadc.com).
## Development Setup
These steps build Spinifex from source and run it as a single-node cluster, following the [Source Install](https://docs.mulgadc.com/docs/install-source) guide.
### Prerequisites
- **Ubuntu 26.04** or **Debian 13**, on x86_64 or aarch64
- A host with hardware virtualisation (KVM) and `sudo` access
- The host's WAN interface enslaved to a Linux bridge named `br-wan`, with the host IP, default route and DHCP on the bridge rather than the bare NIC (see [VPC Networking → Bridge Setup](https://docs.mulgadc.com/docs/vpc-networking#bridge-setup-physical-network-wiring))
The web console additionally needs `pnpm` and the Node.js version pinned in `spinifex/runtime/roles/spinifexui/frontend/.nvmrc`.
## Development Workflow
| Command | Use it to |
|---|---|
| `make deploy` | Rebuild, install `spx`, and restart `spinifex.target` after a code change. |
| `make reinstall` | Re-run `dev-install.sh` after changing systemd units, helper scripts, or logrotate config. |
| `./scripts/reset-dev-env.sh` | Wipe all cluster state and reinstall from scratch, keeping your network topology. |
| `make test` | Run unit tests (`make test-race` for the race detector). |
| `make test-integration` | Run the real gateway router against embedded NATS JetStream. |
| `make preflight` | Run the pre-commit gate: lint, `govulncheck`, coverage, and manifest and script checks. Must pass. |
Follow the services with `journalctl -u 'spinifex-*' -f`.
The web console lives in `spinifex/runtime/roles/spinifexui/frontend/` (`pnpm install`, `pnpm dev`, `pnpm lint`, `pnpm test`). `spx` embeds its checked-in `dist/`, so run `make build-ui` and commit the rebuilt `dist/` with your change.
## Creating a Pull Request
1. Fork [mulgadc/spinifex](https://github.com/mulgadc/spinifex) and clone your fork into the same parent directory as the sibling repositories, branching off `dev`:
```bash
cd ~/Development/mulga
git clone git@github.com:YOUR_USERNAME/spinifex.git && cd spinifex
- `go mod tidy` leaves only intended changes in `go.mod` and `go.sum`, with none of the `clone-deps.sh` `replace` lines.
- Runtime changes are exercised on your dev cluster with `make deploy`.
3. Commit using [Conventional Commits](https://www.conventionalcommits.org/) (`feat:`, `fix:`, `refactor:`, `docs:`, `chore:`), push, and open a pull request against `dev`, filling in the template.
$(_Q)LOG_IGNORE=1AWS_MODEL_CONFORMANCE_MODE=$(AWS_MODEL_CONFORMANCE_MODE)AWS_MODEL_CONFORMANCE_REPORT=$(AWS_MODEL_CONFORMANCE_REPORT) go test -count=1 -tags=integration -timeout 60s ./tests/integration/... $(_RACEQ)
$(_Q)LOG_IGNORE=1AWS_MODEL_CONFORMANCE_MODE=$(AWS_MODEL_CONFORMANCE_MODE)AWS_MODEL_CONFORMANCE_REPORT=$(AWS_MODEL_CONFORMANCE_REPORT) go test -count=1 -tags=integration -timeout 120s ./tests/integration/... $(_RACEQ)
# Spinifex: the open, AWS-compatible cloud you run yourself
Spinifex, built by [Mulga](https://mulgadc.com), recreates the AWS services your software already uses (EC2, EBS, S3, VPC, IAM) on infrastructure you control. Move workloads off the hyperscalers, run AI at a fraction of the cost, or take the cloud to the edge, without rewriting a line. The exact build you ship to AWS, Spinifex serves on your own hardware: same APIs, same Terraform, same SDKs.
Spinifex, built by [Mulga](https://mulgadc.com), recreates the AWS services your software already uses on infrastructure you control. Move workloads off the hyperscalers, run AI at a fraction of the cost, or take the cloud to the edge, without rewriting a line.
## What is Spinifex?
Spinifex is an open-source infrastructure platform that recreates the AWS service surface (EC2, EBS, S3, VPC, IAM) on bare-metal, edge, and on-premises environments. Run cloud-native software without a hyperscaler.
Built for teams that need:
- The AWS tooling you already use (AWS CLI, SDKs, Terraform), with nothing to rewrite
- Full control of the stack: your hardware, your network, your data, your keys
- A cloud that keeps running offline, through disconnection, with no external control plane
- An open core (AGPL-3.0), auditable and yours to keep, with no lock-in
- **Your AWS tooling, unchanged.** The AWS CLI, SDKs, and Terraform work as they are.
- **Full control of the stack.** Your hardware, data, and keys, with no external control plane, running through disconnection.
- **Open core under AGPL-3.0.** Auditable, yours to keep, with no lock-in.
You change *where* your software runs, not *what* it is.
## The Platform
## Try It Live
Get a real Spinifex endpoint in under a minute, with no card and no install. The sandbox is yours for 72 hours, with access keys, the web console, and EC2, EBS, S3, VPC, IAM, EKS, and RDS ready to call. Point your existing AWS CLI or Terraform straight at it.
**[Create your sandbox →](https://mulgadc.com/signup)**
For Terraform, start from the [example workbooks](https://docs.mulgadc.com/docs/nginx-webserver).
## Why Not LocalStack or Floci?
[LocalStack](https://github.com/localstack/localstack) and [Floci](https://github.com/floci-io/floci) are emulators for local development and CI testing. An emulator returns a plausible response; Spinifex boots the machine. A call launches a real KVM guest, attaches a replicated Viperblock volume, or writes an erasure-coded object into Predastore, so Spinifex can run your production workloads on hardware you own. Test against an emulator, then run in production on Spinifex.
## Services
The platform serves **392 AWS API operations** across the services below. See exactly which ones in the [coverage report](https://docs.mulgadc.com/coverage).
| Service | What it is | Built on | Status |
| --- | --- | --- | --- |
| **EC2** | Compute | QEMU/KVM, with spot instances and GPU passthrough | Available |
| **EBS** | Block storage | [Viperblock](https://github.com/mulgadc/viperblock): replicated, with a WAL on local NVMe | Available |
| **VPC** | Networking | OVN, with security groups | Available |
| **IAM** | Identity & auth | SigV4 across every service | Available |
| **ALB / NLB** | Load balancers | HAProxy (L7), nginx `stream` (L4), with ACM certificates | Available |
| **EKS** | Kubernetes | k3s, with HA control planes and managed node groups | Available |
| **ECR** | Container registry | Predastore, deduplicated per account | Available |
| **ECS** | Container service | Spinifex instances | Available |
| **RDS** | Databases | PostgreSQL and MariaDB, with automated backups | Available |
| **Bedrock** | AI deployment | vLLM for text generation and Hugging Face TEI for embeddings and reranking | Q4 2026 |
| **Lambda** | Serverless functions | | Q1 2027 |
Roadmap services ship under the same AWS API surface, so code written for AWS today works the moment they land. [Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).
## Deploy Anywhere
<p align="center">
<img src=".github/assets/platform.svg" alt="The Spinifex stack: your apps and AWS tooling on top, the Spinifex AWS-compatible service layer, running on standard Linux on CPU and GPU hardware, deployable to Neocloud, on-premise, or the edge." width="900">
</p>
From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
- **Neocloud:** lift and shift onto partner GPU capacity (H100 / H200 / B200). Change an endpoint, keep the software.
- **On-premise:** a multi-node HA cluster in your own data centre, so the stack, the data, and the jurisdiction stay yours.
- **Edge:** air-gapped sites, vehicles, vessels, factories, and clinics, with compute next to the data.
## Three Ways To Deploy
## Install
The same AWS-compatible platform runs in three shapes. Pick the one that matches your hardware.
Spinifex runs on Ubuntu 26.04 or Debian 13. The full guides are at [docs.mulgadc.com](https://docs.mulgadc.com).
### Neocloud
>*Prerequisite:* Spinifex requires a Linux bridge configured on the host for VM networking. See the [single-node install guide](https://docs.mulgadc.com/docs/install#prerequisites) for setup details.
Lift and shift onto a partner Neocloud without rewriting workloads. GPU capacity (H100 / H200 / B200), cheaper, available now. Change an endpoint, keep the software.
### On-premise
Host Spinifex on your own hardware in your own data centre. You get a multi-node HA cluster integrated with your storage and networking, so the stack, the data, and the jurisdiction stay yours.
### Edge
Cloud where the cloud can't reach. Air-gapped sites, vehicles, vessels, factories, and clinics. Compute next to the data, running through disconnection, on hardware you own. The same AWS APIs your software already uses.
## AWS Compatibility
Spinifex speaks the AWS API surface. The AWS SDKs, AWS CLI, and Terraform all work unchanged, whether you deploy at the edge, on-premise, or on a partner Neocloud.
| Service | What it is | Status |
| --- | --- | --- |
| **EC2** | Compute | Available |
| **EBS** | Block Storage | Available |
| **S3** | Object Storage | Available |
| **VPC** | Networking | Available |
| **IAM** | Identity & Auth | Available |
| **ALB / NLB** | Load Balancers | Available |
| **EKS** | Kubernetes | Available |
| **ECR** | Container Registry | Available |
| **ECS** | Container Service | Available |
| **RDS** | Databases | Available |
| **Bedrock** | AI Deployment | Q3 2026 |
Roadmap items ship under the same AWS API surface. Code written for AWS today keeps working the moment they land. [Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).
## Live Demo
Create a live sandbox in under a minute. The fastest way to experience the stack. Try Spinifex against one of your own workloads on infrastructure we run for you.
* Live in under a minute. No card, no install.
* Yours for 72 hours to test and experiment.
* Real AWS-compatible endpoint, access keys and console.
* EC2, EBS, S3, VPC, IAM, EKS and RDS, ready to call.
* Point your existing AWS CLI or Terraform straight at it.
Sign up at [https://mulgadc.com/signup](https://mulgadc.com/signup)
Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing an EC2 compatible API. It manages lifecycle operations like start, stop, and terminate through QEMU's QMP interface. Designed to be straightforward and scriptable, Spinifex lets you launch VMs with the AWS CLI, SDKs, or Terraform, without Kubernetes or a heavyweight orchestrator.
The managed services build on that same instance layer. An EKS node group, an ECS container instance, and an RDS database are Spinifex VMs on your own hardware, each driven through its own AWS API.
- **EC2** – VM lifecycle on bare metal, cloud-init and IMDS metadata, key pairs, tags, spot instances, capacity reservations, and GPU passthrough
- **EKS** – Managed Kubernetes on k3s, with HA control planes, managed node groups, cluster add-ons, an OIDC provider, and IAM access entries
- **ECS** – Task and service scheduling across container instances, with awsvpc ENIs, capacity providers, rolling deployments, and GPU-aware placement
- **ECR** – Private container registry backed by Predastore, with content-addressed blobs deduplicated per account and lifecycle policies
- **RDS** – Managed PostgreSQL and MariaDB, with parameter groups, automated backups, and DNS endpoints served by Northstar
- **ALB / NLB** – Layer 7 load balancing on HAProxy and layer 4 on nginx `stream`, with health checks and ACM-issued listener certificates
- **VPC and IAM** – OVN-backed networking and security groups, with roles, policies, and SigV4 auth applied across every service
### Viperblock (Block Storage – EBS Alternative)
[Viperblock](https://github.com/mulgadc/viperblock) is a high-performance, WAL-backed block storage service that replicates volumes across multiple nodes. It's built for reliability and speed, with support for snapshots, recovery, and direct connection to QEMU instances using NBD or virtio-blk.
- Fast, durable virtual disks
- Replication for resilience
- Exposed over NBD or embedded in VMs
- High performance WAL logs on local NVMe drives, reducing IO traffic to S3
- In-memory read/write block cache
### Predastore (Object Storage – S3-Compatible)
[Predastore](https://github.com/mulgadc/predastore) is a fully S3-compatible object storage system. It supports the AWS S3 API, including Signature V4 authentication, multipart uploads, and Terraform provisioning. Predastore chunks data and distributes it across nodes using Reed-Solomon erasure coding, so it survives node loss and suits large-scale or low-bandwidth sites.
- S3-compatible API and auth
- Multipart uploads, streaming reads/writes
- Data redundancy with Reed-Solomon encoding
### Northstar (Authoritative DNS)
[Northstar](https://github.com/mulgadc/northstar) is a lightweight authoritative DNS server that handles both internal service discovery and public-facing DNS for a Spinifex cluster. Zones are human-readable TOML files, served from the local filesystem or synced from Predastore, keeping DNS configuration in the same object storage as the rest of your infrastructure.
- Authoritative DNS over UDP, TCP, and DNS-over-TLS
- Service discovery for cluster components via SRV records
- Zone files stored locally or in S3-compatible storage
## Architecture at a Glance
<p align="center">
<img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus answered by stateless ec2/ebs/s3/vpc/iam daemons over local backends." width="900">
</p>
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless: scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Just systemd units, a NATS cluster, and your hardware. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
## Self-hosted Installation
Installation requires an Ubuntu 26.04 or Debian 13 system. See the detailed documentation at [docs.mulgadc.com](https://docs.mulgadc.com) for installing Spinifex.
### Bare Metal ISO
The recommended installation is a [bootable x86 installer](https://iso.mulgadc.com/spinifex.iso) for bare-metal hardware.
**Bare metal (recommended):** download the [bootable x86 installer](https://iso.mulgadc.com/spinifex.iso) and follow the [USB install guide](https://docs.mulgadc.com/docs/install-usb).
```bash
curl -fLO https://iso.mulgadc.com/spinifex.iso
```
Follow the [USB install guide](https://docs.mulgadc.com/docs/install-usb) to write the ISO to USB and install on your hardware.
### Single Node Install
>*Prerequisite:* Spinifex requires a Linux bridge configured on the host for VM networking. See the [single-node install guide](https://docs.mulgadc.com/docs/install#prerequisites) for setup details.
**From source:** see the [Source Install](https://docs.mulgadc.com/docs/install-source) guide.
For a complete development environment see the [Source Install](https://docs.mulgadc.com/docs/install-source) documentation.
## Spinifex UI
Spinifex ships with a built-in web console, an optional alternative to the AWS CLI, SDKs, and Terraform. If you know the AWS Management Console, the Spinifex UI fills the same role: a browser-based view of your instances, volumes, buckets, VPCs, and IAM resources, without leaving your own network.
## Web Console
<p align="center">
<img src=".github/assets/spinifex-ui.jpg" alt="Spinifex web console dashboard" width="900">
</p>
Each node serves the console on port `3000` over TLS, and it comes up with `spinifex.target`:
Spinifex ships with a built-in web console for your instances, volumes, buckets, networks, and IAM. See [Launching the Web UI](https://docs.mulgadc.com/docs/setting-up-your-cluster#7-launching-the-web-ui).
- **Same API, different surface.** The UI issues the same AWS SigV4 calls the CLI does, so RBAC, audit trails, and IAM policies apply the same way.
- **Single sign-on against your AWS credentials.** Log in with the access keys from `~/.aws/credentials` on the node where Spinifex is installed. There is no separate user database.
- **Self-hosted, works offline.** The Spinifex binary embeds the UI and each node serves it directly, with no external CDN or analytics calls.
## Architecture
For the full walkthrough, covering first-time TLS certificate trust, login, and a feature tour, see [**Launching the Web UI**](https://docs.mulgadc.com/docs/setting-up-your-cluster#7-launching-the-web-ui) in the cluster setup guide.
<p align="center">
<img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus answered by stateless ec2/ebs/s3/vpc/iam daemons over local backends." width="900">
</p>
## Development Philosophy
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless, so you scale horizontally by starting more. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
### Built by Engineers, For Engineers
## How We Build Spinifex
Spinifex is developed by experienced infrastructure engineers with deep AWS expertise, including former AWS team members who understand the intricacies of building production-grade cloud services. Our team brings decades of combined experience from AWS, enterprise infrastructure, and edge computing environments.
Spinifex is built by infrastructure engineers with deep AWS experience, including former AWS team members, drawing on decades of combined work across AWS, enterprise infrastructure, and edge computing. Engineers architect and implement it, with **Claude Code** accelerating parts of the work.
### AI-Assisted Development
Spinifex is architected and implemented by experienced engineers, with **Claude Code** accelerating certain development tasks. The split keeps the quality and reliability standards that production infrastructure demands:
**How We Use Claude Code:**
- **Code Generation**: Boilerplate AWS API structures and handlers
- **Documentation**: Comprehensive development guides and API documentation
- **Testing**: Test case generation and validation scenarios
- **Refactoring**: Large-scale code restructuring and optimization
**What Remains Human-Driven:**
- **Architecture Decisions**: Core system design and scalability choices
- **Security Implementation**: Authentication, encryption, and threat modeling
- **Performance Optimization**: Real-world performance tuning and benchmarking
- **Production Operations**: Deployment strategies and operational procedures
| Claude Code assists with | Engineers own |
| --- | --- |
| Boilerplate AWS API structures and handlers | Architecture and scalability decisions |
| Development guides and API documentation | Authentication, encryption, and threat modelling |
| Test case generation and validation scenarios | Real-world performance tuning and benchmarking |
| Large-scale restructuring and refactoring | Deployment strategy and production operations |
## Trademarks
@@ -227,4 +141,4 @@ Spinifex is architected and implemented by experienced engineers, with **Claude
## License
Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE), with a commercial license available. You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.
Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE), with a commercial license available.
t.Errorf("injected use of %s was not detected; got %v",sym,s.sorted())
}
}
iflen(s)!=len(want){
t.Errorf("want exactly %d uses (the manager type, its constructor, cluster-state methods, field names, method declarations and an unrelated LoadState excluded), got %v",
{From:"agents/ecs/agent",To:"domains/ecs",Debt:"agents/ecs/agent",Reason:"assignment/state wire types from the ECS implementation; needs a guest/controller contract"},
{From:"agents/eks/tokenwebhook",To:"domains/eks/cluster",Debt:"agents/eks/tokenwebhook",Reason:"WebhookTokenReviewResult from the EKS implementation; needs an EKS contract or projection"},
{From:"agents/rds/agent",To:"domains/rds/instance",Debt:"agents/rds/agent (dependency, narrowed)",Reason:"command protocol, wire and record types; needs a guest/controller wire contract"},
{From:"cmd/aws-model-coverage",To:"domains/acm/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared ECR/ACM/Route 53 inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/ecr/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared ECR/ACM/Route 53 inventories; named exception, must not construct a registration"},
{From:"cmd/aws-model-coverage",To:"domains/route53/awsapi",Debt:"cmd/aws-model-coverage",Reason:"reads declared ECR/ACM/Route 53 inventories; named exception, must not construct a registration"},
{From:"runtime/daemon",To:"domains/ec2/awsapi/instance",Debt:"Instance adapter consumers outside EC2",Reason:"runtime/daemon/eks_cp_control.go and rds_deps.go call the instance adapter; needs an EC2 capability"},
{From:"domains/acm",To:"domains/iam",Debt:"ACM: the IAM AES-GCM helper import (ACM relocation residual)",Reason:"IAM AES-GCM helpers; needs a shared crypto boundary"},
{From:"domains/admission/quota",To:"domains/ec2/awsapi/eip",Debt:"domains/admission/quota (dependency, moved from handlers/quota unchanged)",Reason:"describe adapter used as a counter; needs a reader capability"},
{From:"domains/admission/quota",To:"domains/ec2/awsapi/instance",Debt:"domains/admission/quota (dependency, moved from handlers/quota unchanged)",Reason:"vcpu.go counts instances through the instance adapter; needs an EC2 reader capability"},
{From:"domains/admission/quota",To:"domains/ec2/awsapi/volume",Debt:"domains/admission/quota (dependency, moved from handlers/quota unchanged)",Reason:"describe adapter used as a counter; needs a reader capability"},
{From:"domains/admission/quota",To:"domains/ec2/awsapi/vpc",Debt:"domains/admission/quota (dependency, moved from handlers/quota unchanged)",Reason:"describe adapter used as a counter; needs a reader capability"},
{From:"domains/admission/quota",To:"domains/ec2/instancetypes",Debt:"domains/admission/quota → domains/ec2/instancetypes",Reason:"DefaultVCPUs sizing; needs a consumer-owned capability or EC2 projection"},
{From:"domains/admission/quota",To:"domains/elbv2/awsapi",Debt:"domains/admission/quota (dependency, moved from handlers/quota unchanged)",Reason:"live ELBv2 counter; needs a reader capability"},
{From:"domains/admission/quota",To:"domains/rds/instance",Debt:"domains/admission/quota (dependency, moved from handlers/quota unchanged)",Reason:"live RDS counter; needs a reader capability"},
{From:"domains/admission/quota",To:"runtime/compute/vm",Debt:"domains/admission/quota (dependency, moved from handlers/quota unchanged)",Reason:"instance-record vCPU sweep; needs a reader capability"},
{From:"domains/dns",To:"domains/network/reconcile",Debt:"domains/dns",Reason:"takes the reconcile lock from network's implementation"},
{From:"domains/dns",To:"runtime/compute/vm",Debt:"domains/dns",Reason:"domain-to-runtime import of VM state"},
{From:"domains/ec2/awsapi/instance",To:"domains/iam",Debt:"domains/ec2/awsapi/instance (dependency)",Reason:"IAM edge needs an IAM capability"},
{From:"domains/ec2/awsapi/instance",To:"runtime/compute/cache",Debt:"domains/ec2/awsapi/instance (dependency)",Reason:"node liveness and VM state; needs an EC2-owned projection"},
{From:"domains/ec2/awsapi/instance",To:"runtime/compute/vm",Debt:"domains/ec2/awsapi/instance (dependency)",Reason:"node liveness and VM state; needs an EC2-owned projection"},
{From:"domains/ec2/awsapi/spotinstance",To:"domains/admission/quota",Debt:"domains/ec2/awsapi/spotinstance (dependency)",Reason:"RequestSpotInstances.go uses IAMService and quota.Service"},
{From:"domains/ec2/awsapi/spotinstance",To:"domains/iam",Debt:"domains/ec2/awsapi/spotinstance (dependency)",Reason:"RequestSpotInstances.go uses IAMService and quota.Service"},
{From:"domains/ec2/eip",To:"domains/network/topology",Debt:"EC2 → network implementation",Reason:"EC2 reads network implementation; clears with the EC2-owned projection"},
{From:"domains/ec2/guestmetadata",To:"domains/dns",Debt:"Guest metadata: the cross-domain and runtime imports above",Reason:"guest-metadata relocation residual; needs capabilities, projections or contracts"},
{From:"domains/ec2/guestmetadata",To:"domains/iam",Debt:"Guest metadata: the cross-domain and runtime imports above",Reason:"guest-metadata relocation residual; needs capabilities, projections or contracts"},
{From:"domains/ec2/guestmetadata",To:"domains/sts",Debt:"Guest metadata: the cross-domain and runtime imports above",Reason:"guest-metadata relocation residual; needs capabilities, projections or contracts"},
{From:"domains/ec2/guestmetadata",To:"runtime/compute/vm",Debt:"Guest metadata: the cross-domain and runtime imports above",Reason:"guest-metadata relocation residual; needs capabilities, projections or contracts"},
{From:"domains/ec2/instance",To:"domains/dns",Debt:"domains/ec2/instance",Reason:"cross-domain implementation import of DNS"},
{From:"domains/ec2/instance",To:"domains/network/topology",Debt:"EC2 → network implementation",Reason:"EC2 reads network implementation; clears with the EC2-owned projection"},
{From:"domains/ec2/instance",To:"runtime/compute/gpu",Debt:"domains/ec2/instance",Reason:"VM/GPU runtime state read directly; needs a domain-owned projection"},
{From:"domains/ec2/instance",To:"runtime/compute/vm",Debt:"domains/ec2/instance",Reason:"VM/GPU runtime state read directly; needs a domain-owned projection"},
{From:"domains/ec2/instance/state",To:"runtime/daemon",Debt:"domains/ec2/instance/state (dependency, IMDS readers)",Reason:"wraps the daemon's local-file reader and JetStreamManager bucket open unchanged for vpcd's IMDS lookup"},
{From:"domains/ec2/instance/state",To:"runtime/compute/vm",Debt:"domains/ec2/instance/state (dependency)",Reason:"the vpcd IMDS local reader returns vm.VM"},
{From:"domains/ec2/volume",To:"runtime/compute/vm",Debt:"Domain → runtime",Reason:"VM runtime state read directly; needs a domain-owned projection"},
{From:"domains/ec2/vpc",To:"domains/network/external",Debt:"EC2 → network implementation",Reason:"EC2 reads network implementation; clears with the EC2-owned projection"},
{From:"domains/ec2/vpc",To:"domains/network/external/dhcp",Debt:"EC2 → network implementation",Reason:"EC2 reads network implementation; clears with the EC2-owned projection"},
{From:"domains/ec2/vpc",To:"domains/network/identifiers",Debt:"EC2 → network implementation",Reason:"EC2 reads network implementation; clears with the EC2-owned projection"},
{From:"domains/ec2/vpc",To:"domains/network/topology",Debt:"EC2 → network implementation",Reason:"EC2 reads network implementation; clears with the EC2-owned projection"},
{From:"domains/ecr/auth",To:"domains/iam",Debt:"Cross-domain",Reason:"IAM implementation import; needs an IAM capability"},
{From:"domains/network/reconcile",To:"domains/ec2/eip",Debt:"domains/network",Reason:"reconcile reads EC2 private records; clears with the EC2-owned projection"},
{From:"domains/network/reconcile",To:"domains/ec2/igw",Debt:"domains/network",Reason:"reconcile reads EC2 private records; clears with the EC2-owned projection"},
{From:"domains/network/reconcile",To:"domains/ec2/natgw",Debt:"domains/network",Reason:"reconcile reads EC2 private records; clears with the EC2-owned projection"},
{From:"domains/network/reconcile",To:"domains/ec2/routetable",Debt:"domains/network",Reason:"reconcile reads EC2 private records; clears with the EC2-owned projection"},
{From:"domains/network/reconcile",To:"domains/ec2/vpc",Debt:"domains/network",Reason:"reconcile reads EC2 private records; clears with the EC2-owned projection"},
{From:"domains/ochre",To:"domains/ec2/systeminstance",Debt:"Cross-domain",Reason:"consumes another domain's implementation rather than a capability"},
{From:"domains/ochre",To:"domains/iam",Debt:"IAM implementation consumers in STS, Ochre and ELBv2",Reason:"secret encrypt/decrypt helpers; needs a shared crypto boundary"},
{From:"domains/ochre",To:"domains/network/systemvpc",Debt:"Cross-domain",Reason:"consumes another domain's implementation rather than a capability"},
{From:"domains/ochre",To:"ingress/aws/bodyscope",Debt:"domains/ochre (action adapters)",Reason:"Bedrock action adapters still live in the Ochre package; they move to domains/ochre/awsapi"},
{From:"domains/ochre",To:"runtime/compute/gpu",Debt:"Domain → runtime",Reason:"GPU runtime state read directly; needs a domain-owned projection"},
{From:"domains/ochre/vector",To:"domains/iam",Debt:"Cross-domain",Reason:"consumes another domain's implementation rather than a capability"},
{From:"gateway",To:"domains/ec2/awsapi",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/account",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/capacityreservation",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/eigw",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/eip",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/idem",Debt:"gateway/ec2.go (registration; ec2_idempotency.go)",Reason:"dispatch-time idempotency wrapper; moves with the EC2 dispatch map"},
{From:"gateway",To:"domains/ec2/awsapi/igw",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/image",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/instance",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/key",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/launchtemplate",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/natgw",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/placementgroup",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/routetable",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/snapshot",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/spotinstance",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/tags",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/volume",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/vpc",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ec2/awsapi/zone",Debt:"gateway/ec2.go (registration)",Reason:"EC2 dispatch map stays in gateway until the ingress registration seam"},
{From:"domains/rds/awsapi",To:"domains/ec2/awsapi/instance",Debt:"Instance adapter consumers outside EC2",Reason:"enginecatalog.go calls the instance adapter; needs an EC2 capability"},
{From:"domains/sts/awsapi",To:"domains/iam",Debt:"IAM implementation consumers in STS, Ochre and ELBv2",Reason:"GetCallerIdentity takes IAMService; needs an IAM capability"},
{From:"gateway",To:"domains/eks/awsapi",Debt:"gateway (service registration)",Reason:"IAM, STS, EKS and RDS dispatch stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/iam/awsapi",Debt:"gateway (service registration)",Reason:"IAM, STS, EKS and RDS dispatch stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/rds/awsapi",Debt:"gateway (service registration)",Reason:"IAM, STS, EKS and RDS dispatch stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/sts/awsapi",Debt:"gateway (service registration)",Reason:"IAM, STS, EKS and RDS dispatch stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/ecs/awsapi",Debt:"gateway (service registration)",Reason:"ECS and ELBv2 dispatch stays in gateway until the ingress registration seam"},
{From:"gateway",To:"domains/elbv2/awsapi",Debt:"gateway (service registration)",Reason:"ECS and ELBv2 dispatch stays in gateway until the ingress registration seam"},
{From:"domains/ecs",To:"domains/ec2/ebs/metadata",Debt:"close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)",Reason:"legacy ECS calls another domain's implementation as an internal API"},
{From:"domains/ecs",To:"domains/ec2/eip",Debt:"close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)",Reason:"legacy ECS calls another domain's implementation as an internal API"},
{From:"domains/ecs",To:"domains/ec2/instancetypes",Debt:"close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)",Reason:"legacy ECS calls another domain's implementation as an internal API"},
{From:"domains/ecs",To:"domains/elbv2",Debt:"close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)",Reason:"legacy ECS calls another domain's implementation as an internal API"},
{From:"domains/ecs",To:"domains/iam",Debt:"close-out: handlers/ecs (ADR-0004 blocked-dependency list; inventory-ecs boundary imports)",Reason:"legacy ECS calls another domain's implementation as an internal API"},
{From:"domains/eks/cluster",To:"domains/dns",Debt:"close-out: handlers/eks (ADR-0004 blocked-dependency list; inventory-eks boundary imports)",Reason:"legacy EKS calls another domain's or runtime implementation directly"},
{From:"domains/eks/cluster",To:"domains/ec2/ebs/metadata",Debt:"close-out: handlers/eks (ADR-0004 blocked-dependency list; inventory-eks boundary imports)",Reason:"legacy EKS calls another domain's or runtime implementation directly"},
{From:"domains/eks/cluster",To:"domains/ec2/instancetypes",Debt:"close-out: handlers/eks (ADR-0004 blocked-dependency list; inventory-eks boundary imports)",Reason:"legacy EKS calls another domain's or runtime implementation directly"},
{From:"domains/eks/cluster",To:"domains/ec2/placementgroup",Debt:"close-out: handlers/eks (ADR-0004 blocked-dependency list; inventory-eks boundary imports)",Reason:"legacy EKS calls another domain's or runtime implementation directly"},
{From:"domains/eks/cluster",To:"domains/ec2/systeminstance",Debt:"close-out: handlers/eks (ADR-0004 blocked-dependency list; inventory-eks boundary imports)",Reason:"legacy EKS calls another domain's or runtime implementation directly"},
{From:"domains/eks/cluster",To:"domains/network/systemvpc",Debt:"close-out: handlers/eks (ADR-0004 blocked-dependency list; inventory-eks boundary imports)",Reason:"legacy EKS calls another domain's or runtime implementation directly"},
{From:"domains/eks/cluster",To:"domains/iam",Debt:"close-out: handlers/eks (ADR-0004 blocked-dependency list; inventory-eks boundary imports)",Reason:"legacy EKS calls another domain's or runtime implementation directly"},
{From:"domains/eks/cluster",To:"runtime/compute/vm",Debt:"close-out: handlers/eks (ADR-0004 blocked-dependency list; inventory-eks boundary imports)",Reason:"legacy EKS calls another domain's or runtime implementation directly"},
{From:"domains/elbv2",To:"domains/acm",Debt:"domains/elbv2 (cross-domain)",Reason:"ELBv2 cross-domain import; clears when ELBv2 is extracted behind consumer-owned capabilities"},
{From:"domains/elbv2",To:"domains/dns",Debt:"domains/elbv2 (cross-domain)",Reason:"ELBv2 cross-domain import; clears when ELBv2 is extracted behind consumer-owned capabilities"},
{From:"domains/elbv2",To:"domains/ec2/systeminstance",Debt:"domains/elbv2 (cross-domain)",Reason:"ELBv2 cross-domain import; clears when ELBv2 is extracted behind consumer-owned capabilities"},
{From:"domains/elbv2",To:"domains/ec2/vpc",Debt:"domains/elbv2 (cross-domain)",Reason:"ELBv2 cross-domain import; clears when ELBv2 is extracted behind consumer-owned capabilities"},
{From:"domains/elbv2",To:"domains/network/topology",Debt:"domains/elbv2 (cross-domain)",Reason:"ELBv2 cross-domain import; clears when ELBv2 is extracted behind consumer-owned capabilities"},
{From:"domains/elbv2",To:"domains/iam",Debt:"IAM implementation consumers in STS, Ochre and ELBv2",Reason:"system-instance role ensurer; needs an IAM capability"},
{From:"domains/elbv2",To:"agents/elbv2",Debt:"domains/elbv2/{agent_types,haproxy,health_checker,nginx}.go import lbagent",Reason:"health-report and cert/PID-path vocabulary; needs a guest/controller contract"},
{From:"domains/rds/instance",To:"domains/dns",Debt:"close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)",Reason:"legacy RDS calls another domain's or runtime implementation directly"},
{From:"domains/rds/instance",To:"domains/ec2/instance",Debt:"close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)",Reason:"legacy RDS calls another domain's or runtime implementation directly"},
{From:"domains/rds/instance",To:"domains/ec2/instancetypes",Debt:"close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)",Reason:"legacy RDS calls another domain's or runtime implementation directly"},
{From:"domains/rds/instance",To:"domains/ec2/systeminstance",Debt:"close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)",Reason:"legacy RDS calls another domain's or runtime implementation directly"},
{From:"domains/rds/instance",To:"domains/network/systemvpc",Debt:"close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)",Reason:"legacy RDS calls another domain's or runtime implementation directly"},
{From:"domains/rds/instance",To:"domains/iam",Debt:"close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)",Reason:"legacy RDS calls another domain's or runtime implementation directly"},
{From:"domains/rds/instance",To:"runtime/compute/vm",Debt:"close-out: handlers/rds (ADR-0004 blocked-dependency list; inventory-rds boundary imports)",Reason:"legacy RDS calls another domain's or runtime implementation directly"},
{From:"domains/sts",To:"domains/iam",Debt:"IAM implementation consumers in STS, Ochre and ELBv2",Reason:"STS reads IAM service, trust-policy and OIDC-provider state; needs an IAM capability"},
{From:"operator/cli",To:"gateway",Debt:"operator/cli (dependency)",Reason:"command tree imports implementations; needs public capabilities"},
// importsDomainImplementation is any domain-owned package other than a named
// projection, in its target or legacy home.
funcimportsDomainImplementation(toclass)bool{
returnto.domainShaped()&&to.kind!=domainProj
}
varrules=[]rule{
{
id:"R1_FoundationImportsOnlyFoundation",
clause:"rule 1: foundation imports neither domains, ingress, runtime nor operator (enforced conservatively: foundation imports no module package above foundation)",
hint:"move the dependency's needed primitive into foundation or invert it through a parameter",
clause:"rule 3: a domain does not read another domain's private records, migrations or repository; it uses a consumer-owned capability, a named projection, or a versioned contracts/<domain>/v1 contract",
hint:"declare a narrow capability in the consumer, or import the owner's projection or contracts/<domain>/v1",
clause:"rule 4: a provider adapter implements a provider contract and cannot acquire AWS-visible resource authority by reading a domain's private state",
hint:"pass the needed values into the provider contract instead of importing the domain",
clause:"rule 5: operator and runtime call public use cases; a domain's awsapi action adapters are owned by that domain and wired only by the composition root",
hint:"call an owner-provided capability instead of the AWS action adapter",
subject:func(_class,_string)bool{returntrue},
forbids:func(f,tclass,fromNamestring)bool{
ift.kind!=domainAWSAPI||fromName==awsgwRole{
returnfalse
}
return!f.domainShaped()||f.domain!=t.domain
},
},
{
id:"R5_OperatorImportsNoLegacyImplementation",
clause:"rule 5 and the cmd/spinifex map row: operator use cases reach domains through public capabilities and do not read domain-private state",
hint:"reach the domain through its public capability; do not import a legacy implementation root",
clause:"decision item 1 and the area table: contracts own cross-process vocabulary, not application interfaces, domain persistence or transport implementation",
hint:"keep wire types self-contained; depend only on foundation or another contract",
clause:"area table and domain module shape: a domain owns no other domain's implementation; only its awsapi adapters may use the generic ingress/aws machinery",
hint:"depend on a contract or capability; leave ingress use to the domain's awsapi package",
t.Errorf("port/pollWait = %d/%v, want the defaults",cfg.EnginePort,cfg.PollWait)
}
}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Julian Sommer
