gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-19 10:52:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
dev
spinifex/docs/COMMANDS.md
T
Josh Siviour b5879f44ff ADR-0001 package-boundary migration checkpoint (#1190) 
* refactor: move telemetry file naming to contract

* docs: record telemetry contract move

* refactor: move aws xml response helpers

* docs: record aws xml helper move

* refactor: move aws tag projections to foundation

* docs: record aws tag projection move

* refactor: move account key to state foundation

* docs: record account key state move

* refactor: move host privilege escalation to runtime

* docs: record host privilege move

* chore: resolve lint findings from package-boundary moves

* refactor: move process supervision helpers to runtime host

* docs: record process supervision move

* refactor: move nbd endpoint helpers to compute runtime

* docs: record nbd endpoint move

* refactor: move image catalogue to operator

* docs: record image catalogue move

* refactor: move resource id generation to aws foundation

* docs: record resource id move

* refactor: move predastore role to runtime roles

* docs: point service manifest at acm domain

* docs: record predastore role move

* refactor: move northstar role to runtime roles

* docs: record northstar role move

* refactor: move awsgw role to runtime roles

* docs: record awsgw role move

* refactor: move error payload helpers to aws errors

* docs: record error payload move

* refactor: move account identifiers to aws identifiers

* docs: record account identifier move

* refactor: move NATS transport to messaging foundation

* docs: record NATS transport move

* refactor: move system instance contract to EC2 domain

* docs: record system instance move

* refactor: move system VPC builder to network domain

* docs: record system VPC move

* refactor: move instance metadata service to EC2 domain

* docs: record guest metadata move and dependency-rule debt

* refactor: move DNS control plane to DNS domain

* docs: record DNS domain move

* refactor: move base zone bootstrap to DNS domain

* docs: record base zone bootstrap move

* refactor: replace SystemAccountID with GlobalAccountID

* docs: record SystemAccountID removal

* refactor: extract CA key-pair loader to foundation pki

* docs: record foundation pki extraction

* refactor: move tenant CA paths to ACM domain

* docs: record tenant CA path move

* refactor: move IPsec certificate handling to network host

* docs: record IPsec certificate move

* refactor: move Ochre serving lifecycle to the Ochre domain

handlers/bedrock becomes domains/ochre. Ochre is the platform service;
Bedrock names only its AWS API surface. Persisted and observable
identifiers keep their bedrock names.

* docs: record Ochre domain move

* refactor: move Ochre vector store to the Ochre domain

handlers/ochrevector becomes domains/ochre/vector. Callers import it as
ochrevector because vector is a common local name in the gateway.

* docs: record Ochre vector store move

* refactor: move Hugging Face Hub client to the Ochre domain

gateway/bedrock/hfhub becomes domains/ochre/hfhub. Its only caller is
the spx admin ochre weights pull.

* docs: record Hugging Face Hub client move and Ochre gateway debt

* refactor: move EC2 tag mutation helpers to the EC2 domain

utils/tagmirror.go becomes domains/ec2/tagmirror. It is EC2-wide tag
mutation behaviour, and the tags resource package cannot host it
without a resource-to-resource import cycle.

* docs: record EC2 tag mutation move and migrate split finding

* refactor: move network packages under the network domain

Move spinifex/network/* to spinifex/domains/network/* as one enclosing
move so the L0-L5 layer invariants keep classifying every package from
a single root. Package names and dependency direction are unchanged;
invariant scan roots, the RG-10 sudo allow-list path, CODEOWNERS and
the repo-root walk in the IMDS netns unit test follow the new path.

* docs: record network domain move

* refactor: move standalone EC2 resource packages to the EC2 domain

Move handlers/ec2/{account,key,launchtemplate,placementgroup,
spotinstance,volumestate} to domains/ec2/<resource>. Each package is
named after its directory and callers import it as ec2<resource>;
volumestate already carried its directory name.

* docs: record standalone EC2 resource moves

* refactor: move EC2 VPC networking packages to the EC2 domain

Move handlers/ec2/{vpc,eigw,eip,igw,natgw,routetable} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>. The reconcile drift-watch source scan now matches the
ec2<resource> bucket qualifiers; its rule is unchanged.

* docs: record EC2 VPC networking moves

* refactor: move EC2 compute and storage packages to the EC2 domain

Move handlers/ec2/{instance,volume,snapshot,image,tags} to
domains/ec2/<resource>, each named after its directory and imported as
ec2<resource>, which empties handlers/ec2. The daemon's e2e suite
selection now lists the moved EC2 resource paths in place of
handlers/ec2, so EC2-only changes select the daemon suites again.

* docs: record EC2 compute and storage moves

* refactor: move ECR credential provider to the EKS agent tree

Move the kubelet exec credential provider implementation from
cmd/ecr-credential-provider to agents/eks/credentialprovider, exposing
one Main(in, out) entry point. The binary keeps its name and FIPS boot
import as a thin main. Moving out of cmd/ brings the code under gosec;
the one G101 hit on the kubelet API version constant is annotated.

* docs: record ECR credential provider move

* refactor: move EKS gateway publish helper to the EKS agent tree

Move the eks-gateway-publish implementation to agents/eks/gatewaypublish
behind Run(ctx, Config) error. The binary keeps flag definitions,
environment defaults, parsing, its log prefix and exit code; each former
fatal site returns an error with the same text. The retry wait now also
honours context cancellation, which the binary never triggers.

* docs: record EKS gateway publish move

* refactor: move EKS gateway fetch helper to the EKS agent tree

Move the eks-gateway-fetch implementation to agents/eks/gatewayfetch
behind Run(ctx, Config) error, writing its TSV to Config.Out. The binary
keeps flag definitions, environment defaults, parsing, its log prefix and
exit code; each former fatal site returns an error with the same text.

* docs: record EKS gateway fetch move

* refactor: move EKS webhook cert helper to the EKS agent tree

cmd/eks-webhook-cert keeps flag parsing, the log prefix and the exit;
agents/eks/webhookcert owns Config and Run. Output, error text, file
modes and exit codes are unchanged.

* docs: record EKS webhook cert move

* refactor: move EKS konnectivity cert helper to the EKS agent tree

cmd/eks-konnectivity-cert keeps flag parsing, the log prefix and the
exit; agents/eks/konnectivitycert owns Config and Run. Output, error
text, file modes and exit codes are unchanged.

* docs: record EKS konnectivity cert move

* refactor: move EKS token webhook to the EKS agent tree

cmd/eks-token-webhook keeps the -addr flag, environment-derived
config, logging setup and exits; agents/eks/tokenwebhook owns Config
and Run. The handlers/eks TokenReview result type stays as recorded
temporary debt.

* docs: record EKS token webhook move

* refactor: move ECS agent runtime and credentials packages to the ECS agent tree

* docs: record ECS agent subpackage move

* refactor: move ECS agent implementation to the ECS agent tree

cmd/ecs-agent keeps process lifecycle: signal context, the separate New
and Run calls, both failure log lines, the exit and the ldflags-set
build version, now passed in through Config.AgentVersion.
agents/ecs/agent owns Config, LoadConfig and the agent.

* docs: record ECS agent move

* refactor: move RDS guest agent to agents/rds/agent

* docs: record RDS guest agent move

* refactor: move installer packages to operator/installer

* docs: record installer move

* refactor: move spx command tree to operator/cli

Moving the command tree out of cmd/ brings it under errcheck and gosec.
Flag and config wiring now fails loudly instead of silently dropping a
binding, listing commands fail when they cannot write their table, and
the formation server's unauthenticated TLS is annotated as a known gap.

* docs: record spx command tree move

* chore: accept known vulnerabilities with an expiry

The vulnerability scan failed preflight on a containerd advisory that dev
already carries and that is deferred to a separate dependency update. The
gate now passes only reachable findings listed with an expiry date, fails
on any new one or on an expired acceptance, and warns when an acceptance
is no longer needed.

* refactor: move EC2 account action adapters to domains/ec2/awsapi

* refactor: move EC2 capacityreservation action adapters to domains/ec2/awsapi

* refactor: move EC2 key action adapters to domains/ec2/awsapi

* refactor: move EC2 launchtemplate action adapters to domains/ec2/awsapi

* refactor: move EC2 placementgroup action adapters to domains/ec2/awsapi

* docs: record EC2 standalone action adapter moves

* refactor: move EC2 zone action adapters to domains/ec2/awsapi

* refactor: move EC2 vpc action adapters to domains/ec2/awsapi

* refactor: move EC2 eigw action adapters to domains/ec2/awsapi

* refactor: move EC2 eip action adapters to domains/ec2/awsapi

* refactor: move EC2 igw action adapters to domains/ec2/awsapi

* refactor: move EC2 natgw action adapters to domains/ec2/awsapi

* refactor: move EC2 routetable action adapters to domains/ec2/awsapi

* docs: record EC2 VPC networking action adapter moves

* refactor: move EC2 image action adapters to domains/ec2/awsapi

* refactor: move EC2 snapshot action adapters to domains/ec2/awsapi

* refactor: move EC2 volume action adapters to domains/ec2/awsapi

* refactor: move EC2 tags action adapters to domains/ec2/awsapi

* docs: record EC2 storage and tag action adapter moves

* refactor: move EC2 resource authorization scopes to domains/ec2/awsapi

* refactor: move EC2 idempotency adapter to domains/ec2/awsapi

* refactor: move EC2 instance action adapters to domains/ec2/awsapi

* refactor: move EC2 spotinstance action adapters to domains/ec2/awsapi

* docs: record EC2 instance action adapter moves

* refactor: move AWS REST-JSON router to ingress/aws/rest

* refactor: move SigV4 auth-failure rate limiter to ingress/aws/ratelimit

* docs: record AWS ingress helper moves and tooling conformance

* docs: correct historical lifecycle references

* test: characterize ECR control-plane dispatch before registration seam

* feat: add AWS ingress service registration contract

* docs: record ingress registration phases 0 and 1

* refactor: route registered AWS services through the ingress dispatch registry

The gateway now consumes ingress/aws/dispatch. A registered service is
accepted by the auth gate, resolves its own action label, renders errors in
its declared envelope and is dispatched with an authenticated Invocation
(account, region, policy authorisation and a lazy verified actor). Every
other service keeps the existing switch, now named as the legacy path.

The awsgw role wires an empty registry and refuses to start if a registered
name is also served by the legacy table. No service is registered yet, so
behaviour for every existing service is unchanged.

* docs: record ingress dispatch registry consumption

* refactor: register the ECR control plane from its domain

domains/ecr/awsapi now registers the "ecr" AWS JSON 1.1 surface with the
ingress dispatch registry: action resolution, resource scope, authorisation
through the invocation, the composed capabilities and the success writer all
live with the domain. The awsgw role is the only serving composition point;
the gateway no longer imports the ECR API package and ECR leaves the legacy
service table, JSON error table and dispatch switch. The OCI /v2 route is
unchanged.

Operation coverage reads ECR's declared inventory, which the registration
uses verbatim; coverage.json is byte-identical.

* docs: record ECR control-plane registration

* fix: compose the ECR control plane in the integration harness

The harness stopped wiring ECR's composed capabilities when the
authorization-token action was composed, so every integration test that
creates a repository or mints a token got ServerInternal. It now composes
ECR as the awsgw role does, sharing the OCI registry, meta store and token
issuer the /v2 data plane uses.

* docs: record ingress pipeline deferral

* refactor: extract AWS error envelope rendering to ingress

ingress/aws/envelope owns the EC2, IAM-style and S3 XML error envelopes, the
AWS JSON 1.1 error body and X-Amzn-Errortype value, the JSON-protocol header
detector and the two error content types. Each shape keeps its own builder.

The gateway error writers, ErrorHandler and the legacy per-service envelope
selection stay in gateway and call the new package; gateway/eks renders its
JSON errors through it directly. A 140-case status/header/body golden
comparison across services and error writers was identical before and after.

* docs: record error envelope extraction

* refactor: move AWS service selection into ingress dispatch

Selector answers served, action, error protocol, registered entry and
startup overlap. The gateway supplies its legacy tables through the
dispatch.Legacy interface; legacy dispatch and invocation building stay
in gateway.

* docs: record ingress service selection move

* test: characterize ACM control-plane dispatch before registration

Pins full-pipeline error envelopes, a successful action, target parsing
agreement between dispatch and telemetry, authorization before the NATS
call, 413 and 503 paths, throttle/trace/audit action and the inventory.

* refactor: register ACM control plane from its domain

domains/acm/awsapi registers the acm JSON 1.1 surface through the
ingress dispatch registry, wired by the awsgw role. The gateway drops
ACM_Request, the ACM action table and every legacy acm entry; the
import invariant now covers each registered domain.

* docs: record ACM registration

* docs: record ECS, EKS and RDS source inventory and first-resource ranking

* docs: record EKS control-plane secrets persisted in the cluster record

* chore: drop containerd advisory acceptance now that dev ships v2.4.1

* test: characterize RDS DB subnet group lifecycle before extraction

* docs: record RDS DB subnet group lifecycle contract

* refactor: extract the RDS DB subnet group lifecycle owner

domains/rds/subnetgroup owns the db-subnet-groups/{name} record, member
subnet validation and the delete in-use guard. handlers/rds keeps AWS
validation, projection and paging and binds two consumer-owned
capabilities: the owner's Dependants (instances naming the group) and the
instance side's subnetGroupPlacement. Stored shape and behaviour are
unchanged.

* docs: record RDS DB subnet group owner extraction

* docs: correct RDS subnet group tagging debt and conformance scope

* test: characterize ECS task definition lifecycle before extraction

* docs: record ECS task definition lifecycle contract

* refactor: extract ECS task definition owner into domains/ecs/taskdefinition

Move the task definition record, container types, keys, reference parsing,
ARN helpers and the register/deregister/resolve/list logic into a resource
owner package. Stored JSON field names and KV keys are unchanged, and
revision allocation keeps its existing non-atomic read-then-write.

The owner declares its own PortMapping and SystemControl persisted types so
it no longer depends on handlers/ecs/bus; the handler translates them onto
the assign payload. Task and service launches reach task definitions only
through a launch-side taskDefinitionResolver. AWS conversion, the guest
projection and tags stay in handlers/ecs, and gateway authz only switches to
the moved reference/ARN helpers.

List handlers/ecs and the new package under spinifex_daemon so ECS-only
changes select e2e-ecs.

* docs: record ECS task definition owner extraction

* test: characterize EKS access entry lifecycle before extraction

* docs: record EKS access entry lifecycle contract

* refactor: extract EKS access entry owner into domains/eks/access

Move the access entry record, associated policy and scope types, keys,
PrincipalARNHash, the policy catalogue and group mapping into a resource
owner package. Stored JSON field names and KV keys are unchanged.

Owner is the only writer: Create, SeedCreatorAdmin, Update, Associate,
Disassociate and Delete carry the existing logic verbatim, and the put,
compare-and-set and delete primitives are private. The handler keeps AWS
validation, the cluster existence check, error mapping and projection.
Token review still reads records through the exported Get, and the cluster
prefix purge still sweeps entry keys.

List handlers/eks and the new package under spinifex_daemon so EKS-only
changes select the EKS suites.

* docs: record EKS access entry owner extraction

* test: characterize RDS engine catalogue before extraction

* fix: return InvalidParameterValue for a missing RDS master password

ValidateMasterUserPassword built the empty-password error with a bare
errors.New carrying the code only in its text, which awserrors does not
resolve, so CreateDBInstance without MasterUserPassword returned HTTP 500
ServerInternal. Build it with awserrors.Errorf like the other branches so
the gateway returns 400 InvalidParameterValue, and cover the full gateway
to RDS service path with a regression test.

* docs: record RDS engine catalogue boundary and evidence

* refactor: extract RDS engine catalogue into domains/rds/engine

Move engine identity and validation, the per-engine parameter catalogues,
effective-parameter resolution, db.* instance classes and engine/orderable
selection into an engine package. Behaviour, error codes and ordering are
unchanged.

The package imports no EC2 package: instance-class memory comes from an
immutable Sizing value built from an engine-declared InstanceTypes
interface, backed by the fixed instance-type table in handlers/rds.
Effective parameters leave as an engine-owned Setting, converted to
Parameter in handlers/rds. AWS output building, paging, apply-method
mapping, storage limits and the gateway runnable check stay in the
adapter path.

The RDS agent declares its own EngineCatalog interface and types and
receives an engine-backed implementation from cmd/rds-agent, so agent
code does not import the catalogue. Cross-engine tests use the two real
engines instead of a registered fake.

* docs: record RDS engine catalogue owner move

* test: characterize RDS DB parameter group lifecycle before extraction

* docs: record RDS DB parameter group lifecycle contract

* refactor: extract RDS DB parameter group owner

Move the parameter group and override records, keys and listing into
domains/rds/parametergroup behind an Owner. Delete asks an owner-declared
Dependants interface; instance-side resolution reads through its own
parameterGroupValues interface. SetOverrides refuses an unknown group so
no override key is written without its group. Stored shape, keys, public
errors and propagation are unchanged.

* docs: record RDS DB parameter group owner extraction

* test: characterize EKS add-on lifecycle and guest wire

Pin the persisted record and manifest layout, create/update/delete and
status-report transitions, the ignored client token and report version,
the missing serviceAccountRoleArn PassRole check, and the exact internal
add-ons HTTP body and agent TSV output as they stand today.

* docs: record EKS add-on lifecycle contract

* fix: require iam:PassRole for EKS add-on service account role

CreateAddon and UpdateAddon now pass serviceAccountRoleArn through the
existing EKS PassedRoleARNs path, so the caller needs iam:PassRole on the
role for eks.amazonaws.com, matching CreateCluster and CreateNodegroup.

* docs: record EKS add-on PassRole check

* refactor: extract EKS add-on owner

Move the add-on catalogue, record and manifest keys, CAS, status
transitions and staging into domains/eks/addon behind an Owner that is
the sole writer. The handler keeps AWS validation, error mapping and
projection; the cluster reconciler delegates reports through the narrow
addonReports interface; node groups call EnsureGPUDevicePlugin. Stored
keys, JSON shape, wire types and lifecycle behaviour are unchanged.

* docs: record EKS add-on owner extraction

* refactor: extract EKS add-on guest wire contract

Move the add-on status subject, delivery phase, status report, staged
manifest and internal add-ons response into contracts/eks/v1 and use it
from the handler, gateway and gatewayfetch, replacing the agent's
duplicate. Wire bytes are unchanged, including the internal add-ons body
that the SDK renderer emits with Go field names; compatibility tests pin
the JSON, the guest printf form and the TSV.

* docs: record EKS add-on wire contract extraction

* docs: update EKS add-on inventory and remaining edges

* docs: draft EKS add-on lifecycle correction design for review

* chore: run contracts tests in the default test, coverage and race targets

* docs: apply review decisions to EKS add-on lifecycle design

* test: characterize EKS add-on v1 compatibility with added fields

Pin which owner writes drop unknown record and manifest fields, that
reads and deletes ignore or erase them, that one undecodable entry fails
a listing, and that v1 wire consumers ignore extra fields.

* docs: add EKS add-on executor, bundle, v1 compatibility and release inventories

* docs: draft EKS add-on v2 guest contract design for review

* docs: qualify historical architecture references

* docs: apply review decisions and ADR-0006 to EKS add-on v2 design

* fix(eks): bind internal report publishing to the caller's cluster

PublishInternal was authorised by IAM policy alone, with its resource
built from the body accountId and the URL cluster. The control-plane
role grants it on "*", so any control-plane VM session, any tenant
holding eks:*, or static system credentials could publish cluster
state, bootstrap and add-on reports for another cluster or account.

PublishInternal now passes the existing internal-route gate: the caller
must be a control-plane instance-role session that is a member of the
URL cluster in the body account. The gate and the handler decode the
body through one shared decoder, so the account checked is the account
the subject is built from. Every unauthorised target returns the same
AccessDenied, and invalid account or cluster names are treated as not
found. Valid publishes keep their subjects and payload bytes.

* fix(eks): bind webhook token review to the caller's cluster

WebhookTokenReview took its account from the request body and was
authorised by IAM policy alone, with the control-plane role granting it
on "*". A control-plane VM, any tenant holding eks:*, or static system
credentials could resolve a token against another cluster or account
and learn the Kubernetes groups it maps to there. Error responses also
distinguished absent accounts from absent clusters.

WebhookTokenReview now passes the same internal-route gate as
PublishInternal: a control-plane instance-role session that is a member
of the URL cluster in the body account. The gate and the handler share
one body decoder, each body-scoped internal route must declare its
reader, and every unauthorised target returns the same AccessDenied
without reaching STS verification. Same-cluster responses are unchanged.

* fix(eks): validate add-on serviceAccountRoleArn syntax

CreateAddon and UpdateAddon stored serviceAccountRoleArn without
checking it was an ARN, and the guest renders the value into the
add-on's manifests. A non-empty value must now be syntactically an IAM
role ARN (12-digit account, no region, role resource, IAM name and path
rules); anything else returns InvalidParameterException before any
record is written or manifest staged.

The syntax check is foundation/aws/arn.ValidateRoleARN. iam:PassRole at
the gateway remains the permission and existence check. Stored values
are not rewritten, and an UpdateAddon that omits the field still
succeeds on a record holding a legacy value.

* docs: record EKS internal-route and role-ARN security corrections

* docs: add EKS add-on release membership and activation case study

* fix: point UI build, CI and lint rules at moved package paths

* docs: record EKS add-on activation review decisions

* refactor: move quota admission to domains/admission/quota

Relocate the quota service, live counters and vCPU records from
handlers/quota to domains/admission/quota. Package renamed from
handlers_quota to quota; callers with a local quota identifier import it
as admissionquota. KV buckets, subjects and error codes are unchanged, and
the package's existing debt edges move with it unchanged.

* docs: reconcile package-boundary record for ADR-0001 close-out

Add the missing slice rows (the commits that dissolved spinifex/types, the
ECR repository projection, the stale-path fix and the quota move), correct
stale entries, record the dependency-rule breaks found in the import graph
with removal conditions, add the close-out table of remaining legacy paths
with their decided target owners, recount the mulga-side follow-ups, and
fix stale old-path references in docs and comments.

* test: move KV migration engine tests to foundation/state/migrate

The Registry and RunKV tests exercise foundation/state/migrate exclusively
but lived in spinifex/migrate's test file. Relocate them, with their NATS
helpers, to the package that owns the engine. No test name, assertion or
behaviour changes.

* refactor: move config and object-store upgrade workflow to operator/upgrade

Relocate spinifex/migrate, the operator-driven config-file and object-store
upgrade workflow behind spx admin upgrade, to operator/upgrade. Package
renamed to upgrade (imported as opupgrade where a local upgrade identifier
exists). It stays distinct from foundation/state/migrate, the generic KV
migration engine; no registry types, migration names, versions, file
formats or command names change.

* docs: record migrate move to operator/upgrade

* test: enforce ADR-0001 layering with a blocking ratchet

Add spinifex/architecture/layering, a test-only package that parses the
module's import graph, maps every production package to an ADR-0001
layer and checks each dependency rule as its own subtest. Existing
violations are allow-listed against their recorded debt; a new forbidden
edge, an unmapped package, a stale allowlist entry or a new package
under a frozen legacy root fails local preflight and CI.

Record the nine edges the test found missing from the migration record
(quota -> EC2 instance types, STS/Ochre/ELBv2 -> IAM, ELBv2
cross-domain, installer UI -> admin), name the importers of the utils
residuals and gateway/ec2_idempotency.go, and correct the handlers/iam
consumer list.

* refactor: drop the legacy handlers_ecr import alias

Import domains/ecr under its package name, or as ecrdomain where the
file also imports the AWS SDK ecr package, matching acmdomain. Also fix
a stale daemon file reference in the integration ECR daemon comment.

* chore: mark spf13/pflag as a direct dependency

operator/cli/wiring.go imports pflag directly. Same version; no upgrade.

* docs: record test-package-check and nilaway as unverified gates

* refactor: move the layering test to module-root architecture/

The ADR-0001 layering ratchet is repository governance tooling, not
Spinifex product code, so it sits beside tests/ rather than under the
spinifex/ product tree. make test, test-cover and test-race now include
./architecture/..., and the graph walk excludes architecture/.

* fix: update paths left stale by the package moves

The MariaDB image stamp test read the RDS engine from
handlers/rds/engine_mariadb.go, so test-images failed in CI; it now reads
domains/rds/engine/mariadb.go. Comments citing moved files now name
their new locations.

* refactor: move the default-VPC subject into contracts/ec2/v1

utils.SubjectEnsureDefaultVpc becomes ec2v1.EnsureDefaultVpcSubject, the
request/reply route between gateway account creation and the daemon.
Only the subject moves; the request and reply bodies are still anonymous
structs on each side and are recorded as such in the contract README.
The layering freeze now allows two utils files.

* docs: qualify the migrate split and record admin and Viperblock findings

The migrate row is a split, not an exact ADR-0001 map row, and needs a map
clarification. Correct the stale admin consumers, record the operator-only
admin files as movable, the vbwire prerequisite for the Viperblock adapter
and the capability decision encryption.go needs.

* refactor: move operator-only admin code to operator/admin

Config generation, certificate issuance, host discovery, telemetry,
email validation and image import/removal/orphan-volume tooling move
to operator/admin. Legacy admin keeps only the symbols with
non-operator consumers: default account identifiers, AWS credential
generators and system image promotion.

foundation/pki and the tenant ACM CA are unchanged. The readAMI
helper and the removal test fixtures are duplicated privately in
legacy admin rather than shared across the split. The now-stale
operator/installer/ui -> admin layering allowlist entry is removed.

* docs: record operator/admin split in package-boundary migration

* test: pin viperblockd fence, abandon and dirty-marker wire shapes

Characterization tests against the current vbwire package, before it
moves to contracts/viperblockd/legacy/v1: both subject forms for the
fenced and abandon routes, golden JSON for VolumeFencedEvent,
VolumeAbandonRequest, VolumeAbandonResponse and DirtyRecord, the dirty
bucket name, and the volume-key pattern shared by the lease and dirty
stores.

* refactor: move viperblockd fence, abandon and dirty-marker vocabulary to contracts

Move the fence and abandon subjects/DTOs, the dirty bucket name,
DirtyRecord and the shared volume-key pattern out of vbwire and into
contracts/viperblockd/legacy/v1, alongside the package's existing
mount/unmount/delete/sync/config-update routes. vbwire keeps only the
I/O helpers (BindDirtyBucket, ListUnsealedVolumes), now built on the
contract vocabulary instead of defining it.

Record the approved dirty-marker ownership boundary in the contract
README: the Viperblock adapter is the sole producer, viperblockd was
only its transitional custodian. Document the fence and abandon routes
and the dirty bucket in docs/service-interfaces.yaml.

* refactor: split the Viperblock EBS adapter out of services/viperblockd

Move the EBS provider adapter (volume lease, fence, dirty marker,
nbdkit recovery, provider handlers, encryption key loading) to
providers/ebs/viperblock. services/viperblockd keeps only process
lifecycle and the NATS subscription wiring that starts and stops it.

The dirty-bucket read path moves to providers/ebs/viperblock/dirty as
a read-only ListUnsealedVolumes capability; the adapter's write, clear
and purge logic stays unexported in the adapter package. Delete
services/viperblockd/vbwire now that its vocabulary lives in
contracts/viperblockd/legacy/v1 and its I/O helpers live in dirty.

Update operator/cli, the runtime service registry, qemunbdd's config
comment and the integration test harness to the new import paths, and
drop the now-stale architecture/layering allowlist and frozen-package
entries for the deleted package. Behaviour, persisted keys and NATS
subjects are unchanged.

* refactor: move the viperblockd role to runtime/roles/viperblockd

With the EBS adapter in providers/ebs/viperblock, services/viperblockd
holds only role supervision and the opt-in live data-path tests, so it
joins the other runtime roles. It was the last package under services/;
the frozen legacy list drops it.

* docs: record viperblockd contract, adapter and role moves

* test: characterise vpc.* network-realisation routes

Pin every vpc.* subject, JSON payload shape and the request/reply
{success,error} ack envelope exactly as they exist today, ahead of
introducing contracts/network/v1. Covers the 22 subjects registered
by domains/network/subscribers, the vpc.igw-attach/detach subjects
and InternetGatewayEvent already owned by contracts/ec2/v1, the
utils.AddNAT/PublishNATEvent client wire behaviour including the
delete-nat retry-on-ErrNoResponders budget, and the EC2-side NATEvent
and SGEvent/SGRule duplicates in domains/ec2/vpc.

Also adds the six vpc.* route-table and system-egress subjects that
were missing from service-interfaces.yaml despite being live
production routes (vpc.add/delete-igw-route, vpc.gate/ungate-subnet-
egress, vpc.add/delete-system-egress).

* feat: add contracts/network/v1 for vpc.* realisation routes

Declare the versioned wire vocabulary for the vpc.* control-plane-to-
network-realisation boundary: subjects, payload types, the shared
{success,error} ack envelope and the vpcd-workers queue group, one
file per route family. No NATS I/O, no consumer switched over yet;
the package coexists with the fragmented originals pinned in the
prior characterization commit until slice 3 migrates call sites.

* refactor: use contracts/network/v1 for vpc.* producers and consumers

Switch every vpc.* producer (domains/ec2/vpc, eip, instance, natgw,
routetable, igw, handlers/elbv2, handlers/eks, daemon, systemvpc) and the
network/subscribers consumer onto contracts/network/v1 subjects, event
types and the shared AckEnvelope. contracts/network/v1 becomes the sole
owner of vpc.igw-attach/detach, replacing the duplicate definitions in
contracts/ec2/v1.

Deletes the duplicated local subject constants and event structs:
subscribers' Topic* consts and event structs, vpc.NATEvent and SGEvent,
eip's natEvent, natgw's natGatewayEvent, and eks's systemEgressEvent.
domains/ec2/vpc.SGRule survives because EC2 persists it directly in
SecurityGroupRecord; contracts/network/v1.SecurityGroupRule was expanded
in slice 2 to match it field-for-field so wire bytes are unchanged.

No lifecycle or reliability changes: the NAT add barrier, delete
retry-on-ErrNoResponders, and the inconsistent per-call-site NAT
delivery modes (EIP AssociateAddress and DeleteNetworkInterface both stay
fire-and-forget with no ack) are preserved exactly, as are the golden
wire literals pinned in the characterization tests.

* refactor: replace utils/vpcd_event.go with a typed network projection client

Every vpc.* call site now goes through domains/network/projection's
operation-specific methods instead of utils.AddNAT, utils.PublishNATEvent
or utils.RequestEvent, so no caller passes a bare subject string. Each
method states today's delivery mode by name: AddNAT stays strict,
AddNATBestEffort and AnnounceNATAdd/AnnounceNATDelete keep the gaps those
call sites already had, and RemoveNAT keeps the ErrNoResponders retry.

utils/vpcd_event.go and its tests are deleted; the behaviour they pinned
(wire shape, ack envelope, retry-on-no-responders) moved into the
projection package's own golden and nat tests, unchanged. utils/ now
holds only encryption.go, which this does not touch.

* refactor: route the remaining fire-and-forget vpc.* producers through projection

CreateVPC/DeleteVPC, CreateSubnet/DeleteSubnet, AttachInternetGateway/
DetachInternetGateway, AddNATGateway/RemoveNATGateway, AddIGWRoute/
RemoveIGWRoute, GateSubnetEgress/UngateSubnetEgress and RemoveSystemEgress
were still publishing through natsmsg.PublishEvent or a raw nc.Publish
call with a bare subject string, left over from the mechanical slice 3
pass. Each now goes through its named projection.Client method instead,
so every vpc.* producer in the tree uses the typed client.

* refactor: move EBS encryption posture to domains/ec2/ebs/encryption and delete utils

The master-key loader was used only by EC2 image, instance and volume to
derive the AWS-visible Encrypted flag. It becomes ebsencryption.Enabled,
EC2's temporary reading of the provider's key-file setting; the cache,
empty-path meaning, error text and each caller's error handling are
unchanged (image still reports unencrypted when the key will not load).
Its successor is a providers/ebs capability reporting volume encryption
posture.

utils/ is now empty and deleted. The layering test drops utils and
services from the legacy roots, removes their allowlist entries and fails
if either retired root regains files. The S7 subject scan follows the
vpc.* literals from utils to contracts/network so its gap list stays
complete.

* docs: record the network contract, projection client and utils removal

* test: characterise instance-state persistence, recovery and generation handling

Pin the current local-file bytes and schema failure modes, the instance
bucket configuration, record key and wire form, the bucket version gate,
generation preservation across every KV writer and the VM conversion, and
what a returning node does for a missing, foreign, terminal or unreadable
canonical record. These describe current behaviour, not the ADR-0007 target.

* test: freeze direct instance-state consumers

Add an AST ratchet over production code that fails when a new file names
the daemon's instance-state bucket, record keys, local-file API, the raw
vm.InstanceRecord or its conversion, or a JetStreamManager instance-state
method. Today's 73 uses form a shrink-only allowlist keyed to the slice 0
inventory; stale entries fail, and a synthetic-source self-test proves the
matcher detects each kind of use and ignores declarations.

* docs: record ADR-0007 slice 0 instance-state inventory

Record the instance-state persistence formats, the nineteen direct
consumers by process and access kind, the characterised recovery and
schema outcomes, the generation-preservation matrix and the ratchet that
freezes the consumer set. Nothing conforms to ADR-0007 yet and vpcd stays
blocked.

* test: narrow the instance-state ratchet to instance-state access

Stop freezing the JetStreamManager type and constructor, which also serve
cluster state; flag only instance-state methods, bucket and key identifiers,
record codecs and local-state access. The cluster-state-only consumers in
daemon_mgmt_ip.go and operator/cli/cluster.go leave the allowlist (73 to 67).
The self-test now proves a cluster-state use is ignored and an instance-state
method call is caught. Record the owner's disposition of the KV-unavailable
restore result in the migration record.

* docs: record the narrowed instance-state ratchet commit

* docs: design the instance-state transition record and activation gate

* docs: record the instance-state transition-record design commit

* docs: record slice 1a activation-gate decisions

* feat: add instance transition-record codec with format marker

Adds DecodeTransitionRecord and TransitionRecord.Encode in the record's owner
package. A top-level format member selects the shape: absent or 1 decodes the
legacy record and is never given a marker on write; 2 also decodes the new
assignment and observation sections; an integer above 2 decodes its legacy
fields read-only and Encode refuses it; anything else is refused on decode.

Encode applies only the typed paths that differ from the decode-time snapshot
onto the input, so unknown members at every object level and unchanged known
fields keep their read form. A changed array is replaced whole. Decode refuses
malformed JSON, duplicate keys at any depth, keys that alias a known field by
case, trailing data and a non-object record, each with its own sentinel.

No production reader or writer uses the codec yet; the new types are not
part of vm.InstanceRecord, so every existing record encodes unchanged. The
codec file is registered as an owner file of the instance-state ratchet.

* docs: record slice 1b step 1 codec and awsgw rollout qualification

Records the codec added in e62f921d2 as design Section 10 (C1 to C7): its
surface, preservation at every object level, the changed-array limitation,
marker handling, the rejection classes and its capable-writers-only scope. C2
supersedes P1's top-level-only preservation for the codec. G19 records that
old read-only gateway consumers remain safe but DescribeInstanceStatus gains
unconfirmed/degraded visibility only after awsgw upgrades. The migration record
gains the commit row and states that no production path uses the codec yet.

* refactor: move the instance transition-record codec to domains/ec2/instance/state

The codec defines how the canonical persisted instance record evolves, with
its assignment and observation sections, which is EC2 instance ownership rather
than node-local VM supervision. It moves unchanged into its own package and
keeps wrapping the legacy vm.InstanceRecord alias and its spec and status
types, which stay in runtime/compute/vm for now.

The domain-to-runtime import is recorded as new dependency-rule debt with its
removal condition and allow-listed against that entry. The instance-state
ratchet now names the new file as an owner file; its allowlist is unchanged.

* fix: refuse array changes that would drop element extensions

A changed array is replaced whole by the writer's value, which silently lost
any member a newer writer had added inside its elements. Encode now inspects
the array as read: if any element carries a member its typed schema does not
know, at any depth, it returns ErrRecordExtensionConflict naming the array and
the first such member. Arrays without such members are still replaced whole,
unchanged arrays are still kept verbatim, and arrays inside unknown members are
never inspected.

* docs: record the codec move and the array extension-conflict rule

Records 8db9771fb and c617914a1. T1 and Section 10 now name
domains/ec2/instance/state as the eventual owner of the persisted instance
record and its evolution, with vm.InstanceRecord as the temporary legacy type
it wraps. C3 replaces the changed-array limitation with the refusal: a changed
array with element extensions fails with ErrRecordExtensionConflict and is never
silently dropped. P2 stays a conservative rule until finalisation.

* fix: refuse removals that would drop record extension fields

A capable writer may no longer remove or replace a known object, map entry or
array when its stored value carries members the writer's schema does not know.
The check that guarded changed arrays now runs wherever the merge drops or
overwrites a stored value instead of merging into it: a known member deleted or
set to null, a map entry removed or stored under a key the typed decode
rewrote, a value of another JSON kind, and a changed array. Encode returns
ErrRecordExtensionConflict naming the changed container and the first unknown
member. Removal stays allowed when the stored value has no extensions, and a
change merged member by member still keeps unknown members.

* docs: record the uniform extension-conflict rule

Adds design clause C8 for 0d7b4266d: a capable writer may not remove or
replace a known object, map entry or array whose stored value carries unknown
structured extension fields; removal without extensions stays allowed. C3 and
C5 point at it. The migration record gains the commit row, updated subsection
wording and a pointer-bump follow-up deferring make docs-portal to the Mulga
gitlink PR so the portal evaluates the proposed Spinifex revision.

* refactor: inject vpcd's IMDS instance-state readers from the instance state boundary

* docs: cite the vpcd reader injection commit in the migration record

* refactor: move vpcd to runtime/roles/vpcd

* docs: cite the vpcd move commit in the migration record

* refactor: move the ECS agent wire vocabulary to contracts/ecs/v1

Move handlers/ecs/bus to contracts/ecs/v1 (package ecsv1) with no legacy
shim. Subject strings, JSON fields and types are byte-identical; the old-agent
compatibility test moves unchanged and golden pins cover the remaining
message types. The README route table records that only service-reconcile
has a live producer today; the agent reuses these payload types through the
gateway actions. service-interfaces.yaml lists ecs.bus.* under the daemon.

* refactor: move system-image promotion to domains/ec2/image

Move images_promote.go and its tests from admin to domains/ec2/image:
SystemOwnerAlias, PromoteImageOpts/PromoteImageResult, PromoteSystemImage,
GetAMIMetadata and the private snapshot reader. Storage, ordering, errors and
promotion behaviour are unchanged; the daemon, gateway/spx and operator CLI
callers import ec2image. admin now holds only account IDs and key generators.

* docs: record the ECS contract and image-promotion moves and the remaining-roots matrix

* refactor: move the bootstrap account identifiers to foundation/aws/identifiers

DefaultAccountID and DefaultAccountName join GlobalAccountID as the fixed,
platform-reserved identity of the first human-facing bootstrap account.
Names and values are unchanged and stay fixed. Callers in daemon, gateway,
awsgw, accountteardown and the operator CLI use awsidentifiers; admin now
holds only the IAM credential generators.

* docs: cite the bootstrap account identifier move in the migration record

* refactor: inject the STS cluster JWKS lookup from handlers/eks

handlers/sts declares ClusterJWKSResolver and keeps JWKS decoding and
InvalidIdentityToken mapping; handlers/eks implements it as
OIDCJWKSReader beside the bucket and key layout helpers. awsgw and the
integration harness inject the reader. STS no longer imports any EKS
package, including tests, and the handlers/sts -> handlers/eks layering
exception is removed.

* docs: cite the STS JWKS resolver cut in the migration record

* refactor: move the AWS credential-format generators to foundation/aws/credentials and retire admin

GenerateAWSAccessKey and GenerateAWSSecretKey move unchanged to
foundation/aws/credentials as a stateless format primitive. handlers/iam
and operator/cli import it directly. The admin legacy root is deleted
and joins the retired roots so it cannot return.

* docs: cite the admin retirement in the migration record

* revert: remove the uncalled ADR-0007 codec and design stack from the ADR-0001 checkpoint

The instance transition-record codec, its tests and the slice 1a design
had no production caller. They leave this package-boundary checkpoint
and will be reapplied on a dedicated ADR-0007 branch. Slice 0
characterisation and the vpcd IMDS readers in
domains/ec2/instance/state stay.

* docs: cite the ADR-0007 codec revert in the migration record

* fix: declare the ECS action subjects in service-interfaces.yaml

Declaring ecs.bus.* made ecs a managed prefix for manifest-lint, which then
failed on the 34 ECS action subjects the daemon registers. Declare them as
ecs.*, the same wildcard convention as rds.*.

* chore: run manifest-check and manifest-lint in preflight

CI runs both, but preflight did not, so manifest drift could pass local
checks and still fail Build and Test.

* docs: record resumed preflight enforcement and dev merge ports

* fix: restore dev changes dropped by earlier merges

Reapplies the slices.ContainsFunc rewrite in nodeFitsCustomerInstance and the DeleteDBSubnetGroup doc comment, both lost when their files were refactored.

* docs: record accountteardown as a frozen legacy root pending cleanup capabilities

---------

Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
2026-10-15 14:42:53 +10:00

213 lines
46 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Spinifex Admin Commands
## Spinifex Admin CLI
Platform management commands not exposed via the AWS gateway API. These are CLI-only commands.
### Service Management
Services run under systemd. Units live in `spinifex/build/systemd/`, installed by `scripts/setup.sh` on install and reconciled by `spx admin upgrade`.
#### Units
Every service is `PartOf=spinifex.target` and ordered by its own `After=`/`Wants=`.
| Unit | ExecStart | Description |
|------|-----------|-------------|
| `spinifex.target` | — | The whole stack: every service below plus the watchdog timer, openvswitch and ovn-controller |
| `spinifex-nats.service` | `spx service nats start` | NATS message bus with JetStream. Config at `/etc/spinifex/nats/nats.conf`; dependants block on `wait-for-nats.sh` as an `ExecStartPre` |
| `spinifex-predastore.service` | `spx service predastore start` | S3-compatible object storage, serving that host's nodes and the S3 gate among them |
| `spinifex-northstar.service` | `spx service northstar start` | Authoritative DNS for `*.spx3.net`, recursive via upstream forwarders, syncing zones from its S3 bucket |
| `spinifex-viperblock.service` | `spx service viperblock start` | Block storage with the NBD plugin. `KillMode=process`: nbdkit serving a guest survives a restart of this unit |
| `spinifex-daemon.service` | `spx service spinifex start` | VM orchestration, NATS subscriptions, health endpoint. `KillMode=process`: guest QEMU survives a restart and the new daemon reattaches. `SIGHUP` reloads config |
| `spinifex-awsgw.service` | `spx service awsgw start` | AWS-compatible gateway: SigV4 auth, IAM policy enforcement, TLS. Also serves the Private Admin API |
| `spinifex-vpcd.service` | `spx service vpcd start` | VPC daemon: translates `vpc.*` NATS events to OVN logical switches, ports and routers |
| `spinifex-ui.service` | `spx service spinifex-ui start` | Web UI serving the React frontend |
| `spinifex-qmp-collector.service` | `spx service qmp-collector start` | Reads the `contracts/telemetry/v1` per-VM sidecar, polls telemetry QMP sockets and tap counters, publishes to NATS `metrics.ec2.*` |
| `spinifex-shutdown.service` | `/bin/true`; the work is its `ExecStop` | Runs `spx admin node drain --local --unless-restarting`. `After=` every service, so it stops first and drains guests while the stack is still up |
| `spinifex-nats-watchdog.service` / `.timer` | `nats-js-watchdog.sh` | JetStream liveness probe every 30s. `try-restart`s NATS when JetStream stops accepting writes while the process stays alive — what `Restart=on-failure` cannot catch |
| `spinifex-firewall.service` | `spinifex-firewall-apply` | Oneshot nftables input policy, `WantedBy=sysinit.target` rather than the target. No `ExecStop`: remove it with `nft delete table inet spinifex_filter` |
| `regenerate-ssh-host-keys.service` | `ssh-keygen -A` | Oneshot for AMI-built nodes, conditional on the host key being absent |
#### Operations
| Command | Description |
|---------|-------------|
| `sudo systemctl start spinifex.target` | Starts the stack in dependency order |
| `sudo systemctl stop spinifex.target` | Stops the stack, draining guests first via `spinifex-shutdown.service` |
| `sudo systemctl restart spinifex.target` | Restarts the stack. Does **not** drain: QEMU and nbdkit survive on `KillMode=process` and the daemon reattaches |
| `sudo systemctl restart spinifex-daemon` | Restarts one service. Required after a binary swap — services keep executing the replaced inode until they restart (`spx admin preflight` reports this) |
| `systemctl status 'spinifex-*'` | Per-service state, PID, memory and recent log lines |
| `journalctl -u spinifex-daemon -f` | Follows one service; `-u 'spinifex-*'` follows all |
| `sudo systemctl kill -s HUP spinifex-daemon` | Reloads daemon config without a restart |
| `sudo systemctl edit spinifex-daemon` | Drop-in override. Preferred over editing the installed unit, which `spx admin upgrade` then reports as operator-modified and refuses to touch |
A cluster-wide stop is `spx admin cluster shutdown`, not a `systemctl` loop across nodes — it sequences GATE → DRAIN → STORAGE → PERSIST → INFRA with a per-phase ACK from every node.
#### Service configuration
Under systemd the services take no flags: each unit supplies `Environment=` entries, and the flags below are reachable as their `SPINIFEX_*` equivalents. Precedence is flag, then env var, then the node's `spinifex.toml` section, then the default. Environment files are `/etc/spinifex/telemetry.env` (all services), `/etc/spinifex/systemd.env` (viperblock's plugin path) and `/etc/spinifex/host.env` (the daemon's `SPINIFEX_RESERVED_MEM_GB`).
| Service | Flag | Environment variable | Default / notes |
|---------|------|----------------------|-----------------|
| predastore | `--port` | `SPINIFEX_PREDASTORE_PORT` | 8443, the host's S3 gate port |
| predastore | `--host` | `SPINIFEX_PREDASTORE_HOST` | 0.0.0.0, the address the host's sockets bind |
| predastore | `--base-path` | `SPINIFEX_PREDASTORE_BASE_PATH` | Directory holding the service pid file; the unit sets `SPINIFEX_BASE_DIR` |
| predastore | `--config-path` | `SPINIFEX_PREDASTORE_CONFIG_PATH` | Required |
| predastore | `--tls-cert` / `--tls-key` | `SPINIFEX_PREDASTORE_TLS_CERT` / `_TLS_KEY` | Required |
| predastore | `--encryption-key-file` | `SPINIFEX_PREDASTORE_ENCRYPTION_KEY_FILE` | Required |
| predastore | `--host-id` | `SPINIFEX_PREDASTORE_HOST_ID` | Required; names a `[[host]]` in the predastore config. When `SPINIFEX_CONFIG_PATH` points at a cluster spinifex.toml, host/port/host-id default to that node's `[nodes.<node>.predastore]` section rather than the defaults above. `host_id` absent from spinifex.toml resolves to 0, which names no host and is rejected |
| viperblock | `--s3-host` | `SPINIFEX_VIPERBLOCK_S3_HOST` | 0.0.0.0:8443 |
| viperblock | `--s3-bucket` | `SPINIFEX_VIPERBLOCK_S3_BUCKET` | predastore |
| viperblock | `--s3-region` | `SPINIFEX_VIPERBLOCK_S3_REGION` | ap-southeast-2 |
| viperblock | `--plugin-path` | `SPINIFEX_VIPERBLOCK_PLUGIN_PATH` | `/opt/spinifex/lib/nbdkit-viperblock-plugin.so`; set in `/etc/spinifex/systemd.env` |
| viperblock | `--encryption-key-file` | `SPINIFEX_VIPERBLOCK_ENCRYPTION_KEY_FILE` | 32-byte AES-256 master key for at-rest encryption; empty disables it |
| viperblock | `--debug` | `SPINIFEX_VIPERBLOCK_DEBUG` | |
| nats | `--port` / `--host` / `--debug` | `SPINIFEX_NATS_PORT` / `SPINIFEX_NATS_HOST` / `SPINIFEX_NATS_DEBUG` | 4222 / 0.0.0.0 |
| nats | `--data-dir` / `--jetstream` | `SPINIFEX_NATS_DATA_DIR` / `SPINIFEX_NATS_JETSTREAM` | The unit sets neither: it points `SPINIFEX_CONFIG_PATH` at `nats.conf`, which carries both |
| spinifex (daemon) | `--wal-dir` | `SPINIFEX_WAL_DIR` | The unit sets it alongside `SPINIFEX_BASE_DIR` |
| awsgw | `--host` | `SPINIFEX_AWSGW_HOST` | 0.0.0.0:9999 |
| awsgw | `--tls-cert` / `--tls-key` | `SPINIFEX_AWSGW_TLS_CERT` / `_TLS_KEY` | `/etc/spinifex/server.pem` and `server.key` |
| awsgw | `--debug` | `SPINIFEX_AWSGW_DEBUG` | |
| spinifex-ui | `--port` / `--host` | `SPINIFEX_UI_PORT` / `SPINIFEX_UI_HOST` | 3000 / 0.0.0.0. Command aliases: `ui`, `spinifexui` |
| spinifex-ui | `--tls-cert` / `--tls-key` | `SPINIFEX_UI_TLS_CERT` / `SPINIFEX_UI_TLS_KEY` | `/etc/spinifex/server.pem` and `server.key` |
| spinifex-ui | `--base-dir` | `SPINIFEX_UI_BASE_DIR` | `/var/lib/spinifex/spinifex-ui/` |
| northstar | `--northstar-config` | `SPINIFEX_NORTHSTAR_CONFIG` | Overrides `nodes.<node>.northstar.config_path` |
### Cluster Inspection
Operational commands for inspecting cluster state. These fan out NATS requests to all nodes and aggregate responses.
| Command | Flags | Description |
|---------|-------|-------------|
| `spx get nodes` | `--timeout` (default: 3s) | Loads config → publishes to `spinifex.node.status` fan-out topic → collects responses within timeout → merges config-known nodes with NATS responders → nodes that don't respond shown as `NotReady` |
| `spx get vms` | `--timeout` (default: 3s) | Publishes to `spinifex.node.vms` fan-out topic → collects VM info from all nodes → sorts by node then instance ID → prints table. Alias: `spx get instances` |
| `spx get dhcp-leases` | `--timeout` (default: 3s), `--orphans` (show only leases whose owning resource is gone), `--skip-owner-check` (skip the owner lookup; faster, OWNER reads as unchecked) | Lists the upstream DHCP leases vpcd holds for `source="dhcp"` external pools, with the resource each was taken for and whether that resource still exists. An OWNER of `gone` is a leaked address — nothing renews it on purpose and nothing will release it, so it stays out of the upstream scope until its TTL expires. `unknown` means the owning service could not be reached, which is not the same finding. Alias: `spx get leases` |
### Resource Monitoring
| Command | Flags | Description |
|---------|-------|-------------|
| `spx top nodes` | `--timeout` (default: 3s) | Publishes to `spinifex.node.status` fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability |
### Cluster Initialization
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin init` | `--nodes`, `--node`, `--bind`, `--advertise` (external IP off-host clients dial; auto-detected from the default route), `--port`, `--region`, `--az`, `--cluster-name`, `--cluster-bind`, `--cluster-routes`, `--predastore-nodes`, `--predastore-compaction-interval` (seconds; test clusters set a short one), `--services`, `--formation-timeout`, `--token-ttl`, `--force`, `--discard-jetstream` (with `--nodes` >= 2, remove this node's existing JetStream streams once every node has joined; without it a multi-node init over a store that holds streams is refused, and it never proceeds while NATS is running), `--skip-host-dns`, `--email` (operator address for update and security notices), `--no-telemetry`, `--gpu-passthrough`, `--ipsec` (default true; encrypt intra-AZ Geneve via OVN native IPsec cluster-wide — disable only for a trusted single-rack lab), `--host` (leader to join; falls back to multicast discovery), `--external-mode` (`pool` \| `nat` routed NAT for non-bridgeable uplinks, single-node; pair with `setup-ovn.sh --nat-uplink`; note: system instances (ECS/EKS/load-balancer agents) are unsupported in nat v1 — they require `pool` mode), `--external-iface` (WAN NIC for br-external, auto-detected), `--external-source` (`dhcp` \| `static`), `--external-pool` (`start-end`), `--external-prefix-len` (default 24), `--external-gateway` (auto-detected), `--external-bind-bridge` (default `br-wan` when `--external-source=dhcp`), `--gateway-ip` (OVN gateway router's external IP for SNAT; defaults to the pool's range start, required for `nat` mode without DHCP) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to `join-token` file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI `spx` profile → creates directory structure under `~/spinifex/` → points the host resolver at this node's local northstar so the Spinifex zones (ELB/EKS names) resolve from the node itself (opt out with `--skip-host-dns`). `--force` on an already-initialized node is idempotent for crypto: it preserves the existing keys (master.key, predastore/viperblock encryption keys), system + admin credentials, and CA, and refreshes only the config files and the CA-signed server certificate (so a changed bind IP / SANs is picked up without breaking already-joined nodes or CA-baked AMIs). A genuine clean slate requires removing the data dirs first. |
| `spx admin join` | `--host` (required), `--node` (required), `--token` (required), `--bind`, `--advertise` (defaults to `--bind`, or the auto-detected WAN IP when `--bind` is 0.0.0.0), `--port`, `--region`, `--az`, `--cluster-bind`, `--cluster-routes`, `--data-dir`, `--services`, `--skip-host-dns`, `--email`, `--no-telemetry`, `--predastore-compaction-interval`, `--join-timeout` (how long to keep retrying while the formation server is unreachable, default 20m), `--force` (join even though this node is already initialized or holds JetStream streams, **discarding its own CA, master key and JetStream store**; the store is removed only after formation succeeds, and the join is refused while NATS is running even with `--force`) | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations → points the host resolver at this node's local northstar so the Spinifex zones resolve from the node itself (opt out with `--skip-host-dns`) |
### Version
| Command | Flags | Description |
|---------|-------|-------------|
| `spx version` | — | Prints Spinifex version, commit hash, OS, and architecture (populated via build-time ldflags) |
### Cluster Operations
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin kv digest` | `--store-dir` (default `<spinifex-dir>/nats/jetstream`), `--work-dir` (holds the temporary copy, default the system temp dir), `--stream` (repeatable filter), `--seqs` (also print a hash per sequence), `--json` (print one JSON document, the input `spx admin kv compare` reads) | Read-only per-replica consistency check. Copies this node's `$G` streams tree (without consumer state) to a temporary directory, opens the copy in a private in-process NATS server with no listener, and prints one line per stream with first/last sequence, message count and a SHA-256 over every stored message (sequence, subject, timestamp, headers, data). Run it on every node with `sudo` and diff the output with the `#` lines removed: replicas of one stream must match. A direct get over NATS is not used because any replica may answer it. Writes landing during the copy can change a replica's tail, so gate API writes or re-run before calling a tail difference divergence. Prefer `--json --seqs` on every node followed by `spx admin kv compare`, which tells a diverged replica from one that is merely behind. |
| `spx admin kv compare <digest.json>...` | — (`-` reads any number of digests from stdin) | Reads the `spx admin kv digest --json --seqs` output of every node and checks that each stream's replicas hold the same content. A replica that is merely behind is not divergence; these are: the same sequence holding different content on two replicas, a message missing from one replica that nothing on it explains (no later write to the same subject, no expiry under `max_age` or a message TTL, no stream limit dropping its head), a stream held by fewer or more nodes than its replica count, replicas disagreeing on that count, or no sequence range common to every replica. Sequences past the slowest replica's last are not compared. Exits 0 when every stream is consistent, 1 on divergence, 2 on bad input. Digests are taken from live stores seconds apart, so re-run before trusting a failure that a write landing between two copies could explain. `install-node.sh` gates cluster formation on it. |
| `spx admin cluster shutdown` | `--force` (shutdown even if nodes don't respond), `--timeout` (max wait per phase, default 120s), `--dry-run` (print phase plan without executing) | Performs coordinated, phased shutdown of entire cluster. Phases execute in order: GATE (stop API/UI) → DRAIN (stop VMs) → STORAGE (stop viperblock) → PERSIST (stop predastore) → INFRA (stop NATS/daemon). Each phase waits for all nodes to ACK before proceeding. Uses JetStream state tracking. |
| `spx admin cluster drain-dhcp` | `--timeout` (reply-collection window, default 30s) | Asks each vpcd to DHCPRELEASE every external-pool DHCP lease it currently holds, returning them to the upstream DHCP server. Run on teardown before stopping services — an env reset otherwise strands held leases upstream until their TTL expires, eventually exhausting the upstream scope. Best-effort: warns and exits 0 if the cluster is already down. |
| `spx admin node drain --local` | `--local` (drain the local node only, required), `--timeout` (max wait per phase, default 120s), `--unless-restarting` (drain on a real host shutdown or a plain target stop, and skip only when a restart of the stack is already queued; unset, it always drains) | Runs the GATE and DRAIN phases against the local node only: powers down its guests via QMP and unmounts their volumes (flushing the viperblock WAL) while every service is still up. STORAGE/PERSIST/INFRA are left to systemd's ordered teardown. Wired as the `ExecStop` of `spinifex-shutdown.service`, so `systemctl stop spinifex.target` and host reboot/poweroff drain guests before any storage service stops. |
### Certificate Management
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin cert renew` | `--extra-ip` (additional IPs for SANs), `--extra-dns` (additional DNS names for SANs) | Reads existing CA → regenerates server certificate with all current network interface IPs and machine hostname in SANs → writes new cert. Use after adding a new network interface or changing IP addresses. |
| `spx admin cert create-tenant-ca` | `--domain` (required, repeatable — permitted domains baked into the CA's name constraints), `--regenerate` (replace an existing tenant CA, requires confirmation), `--yes` (skip the `--regenerate` confirmation prompt) | Creates the tenant root CA that ACM's `PRIVATE_CA` validation mode signs leaf certificates from — a separate, independent root from the platform CA. Idempotent: running it again against an existing tenant CA reports the current state rather than regenerating; `--regenerate` is required to replace the root, which invalidates every device's existing trust. |
| `spx admin cert force-renew` | `--arn` (required, certificate ARN), `--account` (defaults to the bootstrap account) | Reissues an ACM certificate under its existing ARN immediately, instead of waiting for the renewal worker's proportional window. Only `PRIVATE_CA` (tenant-CA-issued) certificates can be force-renewed; imported and public ones are refused. Talks to the running daemon over NATS and takes the same per-certificate lease the scheduled worker uses, so it is safe against a live deployment. It exists mainly to verify the ELBv2 fan-out — that every load balancer referencing the certificate re-renders with the new leaf — without waiting out the real window. |
### Node Diagnostics
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin preflight` | `--quiet` (print only non-OK rows) | Reports whether this host has the privileged helper scripts and sudoers grants the installed build depends on. `spx version` knows the build but not the host: `setup.sh` and `setup-ovn.sh` install those assets, and a binary-swap deploy (`make deploy`, or a manual copy) skips them. It also reports any running service still executing a replaced binary — overwriting `/usr/local/bin/spx` while `spinifex.target` is up does not move services onto it, so each keeps executing the replaced inode until it restarts and a node can serve a mixture of two builds indefinitely. Checks compare content, not existence, so a present-but-stale asset is caught. Exits non-zero if any asset is Missing, Stale or Ungranted. |
| `spx admin node js-probe` | `--timeout` (default: 10s) | Round-trips a canary key through the cluster-state KV bucket and exits non-zero if the write or the read-back fails. JetStream can latch a write error per message block and keep serving reads — nats-server's filestore records the failure on the block and returns it on every later write, and nothing clears it short of a restart. The HTTP health endpoint reports the server-wide JetStream state, which that latch never touches, so only an actual write observes it. |
| `spx admin banner` | `--boot-check` | Writes the node information banner to `/etc/issue` (shown before login on the physical/serial console) and `/etc/motd` (shown after SSH login). With `--boot-check` it also detects a management IP that has changed since last boot and updates `/etc/spinifex/node.conf` accordingly. |
### Storage Inspection
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin volumes orphans` | — | Lists volumes the storage provider holds that the control plane has no record of. Their blocks consume space but they have no API handle, so they can be neither described nor deleted through the EC2 API. **Reports only, never deletes**: the evidence that an orphan is still wanted is the record that went missing, so removal is an operator decision taken per volume. |
| `spx admin volumes unsealed` | — | Lists volumes a node has open, or had open, without a confirmed seal to the backend — the node named holds writes predastore may not have. Starting such a volume elsewhere is allowed, and is preferred over leaving the instance unable to run, but it opens from the last checkpoint that did reach the backend. This is the list of volumes where that would cost something. |
### Upgrade Management
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin upgrade` | `--yes` (apply without prompting), `--dry-run` (report pending config and unit changes without applying), `--units-only` (reconcile systemd units only, skip config migrations), `--skip-units` (apply config migrations only, skip unit reconciliation), `--config-dir` (persistent, default: `~/spinifex/config`), `--spinifex-dir` (persistent, default: `~/spinifex`) | Reads current config versions from registry → computes pending config migrations (from→to per target) → reads installed systemd unit versions and compares each against the copy embedded in the binary (`systemd.Reconcile` with `DryRun: true`) → prints both summaries → prompts for confirmation unless `--yes` or `--dry-run` → runs `migrate.DefaultRegistry.RunAllConfig()` to apply migrations to config files → runs `systemd.Reconcile` for real to install missing units and replace stale ones (after a timestamped backup), then `systemctl daemon-reload`. Units whose installed marker version matches the embedded one but whose content differs are left untouched and reported as operator-modified — never overwritten. Writing `/etc/systemd/system` requires root; without it, drift is still reported and the command exits non-zero rather than writing partially. Never restarts services — a unit change applies to that service's next stop. Intended for upgrades between Spinifex versions, including nodes that were upgraded by swapping the `spx` binary directly rather than by re-running the installer. Operators can skip migrations during install by setting `INSTALL_SPINIFEX_SKIP_MIGRATE=1`, then run `spx admin upgrade` manually to review and apply. After completion, services must be restarted with `sudo systemctl restart spinifex.target` to pick up config changes. Invoked non-interactively by `scripts/setup.sh` with `--yes`. |
### Account Management
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin account create` | `--name` (required) | Connects to NATS → refuses if an account already holds the name (case- and whitespace-insensitive) → reserves the name in `spinifex-account-names` KV → CAS loop on `spinifex-account-counter:next_id` for sequential 12-digit ID → creates Account record in `spinifex-accounts` KV → creates default VPC → creates `admin` user, AdministratorAccess policy (Action:*, Resource:*) and one access key → prints credentials and writes a `spinifex-<name>` AWS CLI profile |
| `spx admin account create --remote` | `--name` (required), `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA), `--client-token` (default: generated), `--source` (default: `spx-cli`) | Same account, created through `POST /admin/CreateAccount` over SigV4 instead of NATS — the path the public signup form uses, so a shell can exercise it. Credentials come from the standard AWS chain (env vars or `AWS_PROFILE`), never from the cluster master key; the caller must be an IAM user in the super-admin account holding `spinifex:CreateAccount`. Reuse the printed `--client-token` to retry: a fresh token would create a second account. |
| `spx admin account list` | — | Connects to NATS → IAMService.ListAccounts() → prints table with Account ID, Name, Status, Created |
| `spx admin account list --remote` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Same listing read through `POST /admin/ListAccounts` over SigV4 instead of NATS, so an off-cluster operator or the load-test harness sees its tenants without SSH access to a node. Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:ListAccounts`. |
| `spx admin account quota get <account-id>` | `--endpoint` (default: this node's AWS gateway), `--region` (default: this node's region), `--ca-bundle` (default: this node's CA) | Reads the limits in force for one account through `POST /admin/GetAccountQuota` over SigV4, and reports for each dimension whether it came from the `[quota]` block in `awsgw.toml` (`config`) or from an override stored against the account (`override`). Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:GetAccountQuota`. |
| `spx admin account quota set <account-id>` | `--vcpus`, `--vpcs`, `--subnets`, `--eips`, `--volumes`, `--volumes-gib`, `--rds-instances`, `--load-balancers`, `--clear`, plus `--endpoint`, `--region`, `--ca-bundle` | Overrides individual limits for one account through `POST /admin/PutAccountQuota`, leaving every dimension not named inheriting the configured baseline. `-1` uncaps a dimension; `0` is a real limit that denies every request on it. `--clear` removes the whole override record and cannot be combined with a limit flag. The account must already exist. Prints the resulting limits, so a caller never has to guess what it applied. The caller must hold `spinifex:PutAccountQuota`. |
| `spx admin account delete <account-id>` | `--dry-run`, `--force`, `--yes` (requires `--name`), `--name` (account name, must match the stored record) | Prints the inventory it is about to remove → confirms the account name (prompted unless `--name` is given) → marks the account `TERMINATING`, which blocks its credentials cluster-wide → deletes its resources in dependency order (compute, attachments, storage, network, platform, identity, account), waiting for each stage to actually drain before starting the next → releases the name reservation and quota counter → deletes the account record. Anything that will not delete is reported as **stuck** with the reason, and the account stays `TERMINATING` so the residue keeps an owner. `000000000000` and `000000000001` are refused. `--dry-run` lists and changes nothing. The compute stage reaps ECS clusters, EKS nodegroups and clusters, and RDS instances before it reaps plain instances, because their backing VMs belong to the system account and a tenant-scoped instance listing cannot see them. The storage stage reaps the account's S3 buckets: uploads in flight are aborted, then the objects are removed a page at a time, then the bucket. Only buckets the account owns are ever listed, so the shared platform buckets are out of reach rather than filtered out. Platform-managed serving endpoints are reaped ahead of instances, because only their own delete releases the ENI and attached data volume alongside the VM, and spot instance requests are cancelled first so a live request cannot launch a replacement for something already terminated. The network stage reaps load balancers, then target groups, then NAT gateways and egress-only internet gateways before the subnets, security groups and addresses they hold. The platform stage reaps ACM certificates. |
| `spx admin account delete --force` | as above | Escalates only for resources the ordinary API refuses: it clears a volume attachment in the control plane without the guest's cooperation (`ec2.ForceDetachVolume`), hard-destroys after a graceful attempt times out, and treats an already-missing resource as deleted. It never reorders the stages and never widens scope beyond the account. Use it once the guest is being destroyed — this is the escape from a volume stuck attached to an instance that will not terminate, where neither can otherwise be deleted. |
| `spx admin account delete --remote` | as above, plus `--endpoint`, `--region`, `--ca-bundle`, `--client-token` | Same teardown driven through `POST /admin/DeleteAccount` over SigV4 instead of NATS, then follows `DescribeAccountDeletion` to completion, printing each stage as it lands. Credentials come from the standard AWS chain; the caller must be an IAM user in the super-admin account holding `spinifex:DeleteAccount` and `spinifex:DescribeAccountDeletion`. |
| `spx admin principal create <name>` | `--grant` (repeatable or comma-separated; default: every admin method — `CreateAccount`, `DeleteAccount`, `DescribeAccountDeletion`, `GetAccountQuota`, `ListAccounts`, `PutAccountQuota`) | Creates an IAM user in the super-admin account with an inline `spx-admin-methods` policy allowing the requested methods, revokes any access key left from a previous run, and mints one new key. Each method is named rather than granted as `spinifex:*`, so a later addition to the admin surface is not authorised by an existing key. The principal's grants are exactly that one policy: re-running with fewer `--grant` values narrows it, and any other inline policy on the user is removed, so a rename cannot leave a wider grant behind. The secret is printed once; store it in a secret manager or an AWS profile, never in a config file. `admin` is refused as a name — it is the account's own administrator and rotating its key would revoke the bootstrap credential. |
| `spx admin principal list` | — | Lists every principal in the super-admin account with its access-key count and what it may call. An attached policy is reported by name (`AdministratorAccess (attached)`) rather than expanded, which is how an unscoped principal is spotted. |
| `spx admin principal revoke <name>` | — | Deletes every access key the named principal holds, leaving the user and its policy in place. Immediate and cluster-wide; re-issue with `principal create`. |
| `spx admin principal delete <name>` | — | Removes the principal entirely: its access keys, its inline policies and the user. `admin` is refused, as it is for `create`. Use `revoke` instead when the principal should stay and only its credential should stop working. |
| `spx admin principal audit` | — | Reports roles in the super-admin account whose trust policy names the account, its root ARN, or a wildcard. STS does not evaluate the caller's identity policy on `AssumeRole`, so any such role is assumable by a scoped admin principal, which then inherits its permissions. Exits non-zero if any role is flagged. |
### Image Management
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin images import` | `--name`, `--file`, `--distro`, `--version`, `--arch`, `--platform`, `--boot-mode` (bios/uefi/uefi-preferred), `--tag`, `--force`, `--skip-verify`, `--checksum`, `--ami-name` (override the registered `DescribeImages` name, which otherwise defaults to `ami-{distro}-{version}-{arch}`; used for locally-built appliances such as `--ami-name spinifex-eks-node`), `--tmp-dir` (default: `<spinifex-dir>/images`) | Catalog imports (`--name`) download the image, fetch the catalog `Checksum` URL, verify the SHA-256/SHA-512 digest, and inherit `BootMode` from the catalog entry. `--boot-mode` overrides the catalog value when set. Mismatch fails closed; the cached file is left on disk and `--force` re-downloads. `--file` imports require an explicit `--boot-mode` because there is no catalog metadata to inherit from, and are verified only when `--checksum <sums-file>` is passed. `--checksum` takes the sums file the image was published with (`SHA256SUMS`, `SHA512SUMS`, `<image>.sha256`/`.sha512`, coreutils, BSD or bare-digest format), looks up the entry for the image's basename, and infers sha256 or sha512 from the digest length. A mismatch exits 1 before anything is written to storage and leaves the file in place. `--checksum` proves only that the image matches the sums file supplied with it, not that either is authentic; verify the sums file's vendor signature separately. `--checksum` is refused without `--file` or with `--skip-verify`. A `--file` import without `--checksum` prints `Importing <file> without checksum validation` to stderr and proceeds; `--name --file` does not fall back to the catalog digest. Every import records the source artifact's digest on the AMI with who verified it (`operator`, `catalog` or `unverified`, the unverified cases hashed as sha256), prints it on success, and shows it in `spx admin images describe`. The digest is of the file as supplied, not of the stored volume. `--skip-verify` bypasses verification for catalog imports and prints a warning to stderr; use only for debugging or when upstream mirrors are confirmed-broken. Import never modifies the image: the bytes written to storage are the bytes that were verified. Compressed and qcow2 images are extracted into a hidden `.import-*` scratch directory under `--tmp-dir`, which is removed when the import succeeds or fails, so the catalog directory keeps only the downloaded source (for `--name`) and the AMI manifest. Point `--tmp-dir` at a disk-backed filesystem with room for the full raw image, never at a tmpfs such as `/tmp`. A guest that needs to trust the deployment CA for in-guest SDK-over-TLS calls fetches it at runtime from `http://169.254.169.254/spinifex/ca.pem` (see the IMDS section), so a CA rotation reaches guests without re-importing anything. |
| `spx admin images list` | — | Lists available OS images that can be imported or downloaded |
| `spx admin images describe` | `--image-id` (required) | Read-only. Prints the AMI's name, owner, creation date, boot mode and architecture, then its source digest: algorithm and value, verification (`operator` with the sums file name, `catalog` with the sums URL, or `unverified`), and the source file name. AMIs imported before digest recording print `Source digest: not recorded`. The digest is not exposed through `DescribeImages`. |
| `spx admin images promote` | `--image-id` (required), `--yes` | Reads `ami-<id>/config.json`, validates the AMI is account-owned, then rewrites `ImageOwnerAlias` to `"system"` in-place. No block data is copied. The change takes effect immediately — the AMI becomes visible to all accounts via `DescribeImages`. Prompts for confirmation (skipped with `--yes`). Already-system AMIs are refused. |
| `spx admin images remove` | `--image-id` (required), `--force`, `--yes` | Loads `ami-<id>/config.json`, walks transitive dependents — copied snapshots whose `VolumeID == imageID`, volumes whose `SnapshotID` references the internal `snap-ami-<id>` or any derived snap, and account AMIs created via `CopyImage` whose `SnapshotID` is a derived snap — then prompts (skipped with `--yes`) before deleting `ami-<id>/config.json` (the DescribeImages barrier) followed by the rest of `ami-<id>/` and `snap-ami-<id>/`. Account-owned AMIs are refused with a hint pointing at `aws ec2 deregister-image` + `aws ec2 delete-snapshot`. `--force` bypasses the dependency, ownership and config-corrupt checks for salvage of orphaned blocks. |
### GPU Management
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin gpu status` | `--node` (default: local node) | Queries `spinifex.node.status` fan-out → finds the target node response → prints Node, GPU hardware (model list or "none detected"), IOMMU state, vfio-pci state, passthrough enabled/disabled, and GPU pool allocation (`allocated/total`). Also lists GPU-capable instance types when passthrough is active. |
| `spx admin gpu enable` | — | Checks current passthrough state via NATS → errors if already enabled or prerequisites not met (directs to `setup`) → writes `gpu_passthrough = true` to `spinifex.toml` via `admin.SetGPUPassthrough` → sends SIGHUP to `spinifex-daemon` via `systemctl kill -s HUP` → polls node status for up to 30 s until daemon confirms new state → prints final `gpu status` output. Must be run directly on the target host. |
| `spx admin gpu disable` | — | Checks current passthrough state via NATS → errors if already disabled or if `AllocGPUs > 0` (must terminate GPU instances first) → writes `gpu_passthrough = false` → sends SIGHUP to `spinifex-daemon` → polls for up to 30 s → prints final `gpu status` output. Must be run directly on the target host. |
| `spx admin gpu setup` | — | Idempotent host configuration for GPU passthrough. Steps (skipped if already applied): detect GPUs via `gpu.Discover` → collect PCI IDs for all IOMMU-group siblings → check/enable IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`) → write vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`) → blacklist nouveau (`/etc/modprobe.d/blacklist-nouveau.conf`) → blacklist amdgpu if AMD GPU present (`/etc/modprobe.d/blacklist-amdgpu.conf`) → write vfio-pci early binding config (`/etc/modprobe.d/vfio-pci.conf`) → add vfio modules to initramfs. If any change requires a reboot: runs `update-initramfs -u` and exits with reboot instructions. After reboot: verifies `vfio_pci` module is loaded → verifies each GPU is bound to `vfio-pci` (binds explicitly via `driver_override` if unbound) → calls `gpu enable` to activate passthrough. |
| `spx admin gpu mig status` | — (must be run on target host) | Runs `gpu.Discover()` locally → for each GPU prints: PCI address, model, MIG capability, MIG mode (enabled/disabled/N/A); for GPUs with MIG enabled lists active MIG slices with GI ID, profile name, and mdev path. |
| `spx admin gpu mig enable` | `--profile <name>` (required, e.g. `1g.10gb`), `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → enables MIG mode on each target (`gpu.EnableMIGMode`) → lists available profiles (`gpu.ListProfiles`) and validates requested profile name → destroys any existing instances (`gpu.DestroyAllInstances`) → creates new instances filling GPU capacity (`gpu.CreateInstances`) → writes `mig_profile` to `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. |
| `spx admin gpu mig disable` | `--gpu <pci-addr>` (optional, default: all MIG-capable GPUs) | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via `gpu.Discover()` (filtered by `--gpu` if set) → destroys all GPU instances (`gpu.DestroyAllInstances`) → disables MIG mode (`gpu.DisableMIGMode`) → clears `mig_profile` in `spinifex.toml` via `admin.SetMIGProfile` → sends SIGHUP to `spinifex-daemon`. Must be run directly on the target host. |
### EKS Control-Plane Disaster Recovery
| Command | Flags | Description |
|---------|-------|-------------|
| `spx admin eks restore-snapshot` | `--cluster` (required), `--snapshot` (optional, defaults to the latest snapshot in predastore), `--account` (optional, defaults to the bootstrap account) | Single-CP total-loss DR path (fail-safe): validates the snapshot exists in predastore BEFORE any mutation (a typo'd/missing key hard-fails, never resets into an empty datastore) → launches a fresh control-plane VM as a cluster-init seed (replaying the persisted create-time launch template) → sets a required-snapshot `RecoveryDirective` (`cluster-reset`) so the boot-time recovery agent aborts rather than resets-into-empty if it cannot fetch the snapshot → persists the replacement in cluster meta BEFORE re-pointing the NLB (so an NLB failure is convergeable by the reconciler, returned as a provisional status, not a hard error) → re-points the cluster NLB's apiserver and konnectivity target groups from the old CP's ENI to the new one → fences the old CP with retries, failing loudly if it cannot be confirmed terminated (split-brain guard). Any failure before the meta commit unwinds the fresh CP (terminate + clear directive) so a re-run does not stack a second resetting control plane. The returned status is provisional — success means the sequence completed, not that etcd is restored and serving; verify cluster health. HA clusters (a spread with a potentially surviving quorum) are rejected — recover those via quorum reformation instead. |
## Private Admin API
Not an AWS API: a Spinifex-internal surface on the AWS gateway, JSON in and JSON out, one method per path segment at `POST /admin/<Method>`. It exists so a self-service signup form, a provisioning system or a load-test harness can create and remove tenant accounts without an operator shell on a node.
Requests are SigV4-signed with `service=spinifex` and the gateway's configured region (`region` in `awsgw.toml`; production is `us-west-1`). The caller must be an IAM **user** in the super-admin account (`000000000001`) holding `spinifex:<Method>` — assumed-role sessions are refused outright. Every gate denies with the same `AccessDenied`, so a caller cannot learn which one it failed. Every response carries `X-Amzn-RequestId`, repeated as `requestId` in an error body.
| Method | Request | Response | Notes |
|--------|---------|----------|-------|
| `POST /admin/CreateAccount` | `name` (email address, required), `clientToken` (32–128 chars of `[A-Za-z0-9_-]`, required), `source` (free-form provenance tag, max 64 chars) | `accountId`, `accountName`, `adminUser`, `accessKeyId`, `secretAccessKey`, `defaultVpcId` | Creates a tenant account with an `admin` user, an AdministratorAccess policy, one access key and a default VPC. The secret is returned **once**; replaying the same `clientToken` within 24 hours returns the identical response and is the only way to re-obtain it. |
| `POST /admin/DeleteAccount` | `accountId` (12 digits, required), `accountName` (must match the stored record), `clientToken` (32–128 chars), `force`, `dryRun` | `deletionId`, `accountId`, `state`; plus `inventory` when `dryRun` | Marks the account `TERMINATING` and tears down everything it owns in dependency order, then deletes the account record. Returns as soon as the teardown starts — it takes minutes, so progress comes from `DescribeAccountDeletion`. `dryRun` needs neither the name nor a token, answers inline and changes nothing. One job per account: a second call with a different token gets `OperationInProgress` while one is running. `000000000000` and `000000000001` return `AccessDenied`, the same answer an unauthorized caller gets. |
| `POST /admin/DescribeAccountDeletion` | `accountId` (12 digits, required) | `deletionId`, `state` (`RUNNING`, `COMPLETED`, `FAILED`), `startedAt`, `updatedAt`, `finishedAt`, `stages[]` with per-stage `deleted[]`, `stuck[]` and `elapsed`, `error` | The record outlives the account, so this stays answerable after the deletion completes. `NoSuchEntity` means no teardown was ever started for that account. A `RUNNING` job whose `updatedAt` has not advanced for five minutes is treated as abandoned — the gateway running it restarted — and a fresh `DeleteAccount` takes it over. |
| `POST /admin/ListAccounts` | — | `accounts[]` of `accountId`, `accountName`, `status`, `createdAt`; `count` | Enumerates every account. No credential material, and no paging: `[signup] max_accounts` bounds the listing. `status` is how a teardown stuck in `TERMINATING` is noticed. |
| `POST /admin/GetAccountQuota` | `accountId` (12 digits, required) | per-dimension limits, each with the source it came from | Reads the limits in force for one account, reporting for each dimension whether the value came from the `[quota]` block in `awsgw.toml` (`config`) or from an override stored against the account (`override`). Driven by `spx admin account quota get`. |
| `POST /admin/PutAccountQuota` | `accountId` (12 digits, required), the limit dimensions to override, or a clear directive | the resulting limits | Overrides individual limits for one account, leaving every dimension not named inheriting the configured baseline. `-1` uncaps a dimension; `0` is a real limit that denies every request on it. The account must already exist. Driven by `spx admin account quota set`. |
Errors are `{"error":{"code":…,"message":…},"requestId":…}`. `OperationInProgress` (409), `ServiceUnavailable` (503) and `InternalError` (500) are retryable with backoff, **always reusing the same `clientToken`** — a fresh token after a failure creates a duplicate account. `AccountAlreadyExists` (409), `IdempotentParameterMismatch` (400), `LimitExceeded` (400, `[signup] max_accounts` reached), `InvalidParameterValue`, `MissingParameter`, `InvalidRequest`, `InvalidAction`, `MethodNotAllowed` (405) and `AccessDenied` (403) are not.
Each method is a separate grant, and `spx admin principal create` decides which a credential holds. `spx admin principal create operator` grants all six — the operator and load-test credential. `spx admin principal create signup --grant CreateAccount` grants one, which is what an internet-facing signup form should hold: that key cannot reach deletion, so a compromise of the form's host cannot remove a tenant. Separate principals are the point rather than an accident: revoking either costs nothing else.
The surface is unreachable until an operator runs `spx admin principal create`; there is no config toggle and no default credential for it. The super-admin account's own `admin` user does hold these grants already, through the `AdministratorAccess` policy every account's administrator carries (`Action: "*"`), so its bootstrap key can call the surface without any further setup — but it is unscoped and unrotatable without locking out the operator, which is why automation should use a named principal. Revocation is `spx admin principal revoke <name>` (or `aws iam delete-access-key`), which is immediate and cluster-wide. `[signup] max_accounts` in `awsgw.toml` caps how many accounts may exist; it defaults to 128 when the section is absent, and 0 means uncapped.
Reference in New Issue View Git Blame Copy Permalink