mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Josh Siviour
bbff004b45
* feat(eks): cluster_state KV helpers — ClusterMeta + status CAS + prefix sweep
Stage 1 of Sprint 6b CreateCluster lifecycle. Introduces the first
read/write helpers against the eks-account-{accountID} bucket so later
stages (oidc keypair, NLB wiring, cluster reconciler) can persist state
without re-inventing CAS/sweep logic.
- ClusterMeta struct holds the DescribeCluster source of truth (status,
endpoint, OIDC issuer, CA, VPC config, control-plane VM + ENI, NLB ARN).
- ClusterStatus enum mirrors the AWS EKS string set verbatim.
- PutClusterMeta is unconditional (CreateCluster path); GetClusterMeta
returns ErrClusterNotFound on key absence so the service boundary can
map to ResourceNotFoundException.
- SetClusterStatus is revision-checked with bounded CAS retries so the
reconciler and CreateCluster body cannot wedge each other.
- DeleteClusterPrefix sweeps every clusters/{name}/* key; called from
DeleteCluster after the OIDC private key is zeroized.
Tests cover round-trip, missing/corrupt blob, idempotent status set,
DELETING transition from every prior state, concurrent revision bump
recovery, sibling-cluster preservation, empty-bucket no-op.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(eks): per-cluster ECDSA-P256 OIDC keypair generation + zeroize
Stage 2 of Sprint 6b. Lays the at-rest primitives the CreateCluster body
will call before launching the K3s server VM. Pure crypto helpers — no
daemon, no launcher, no NATS bus.
- GenerateClusterOIDCKeypair mints an ECDSA-P256 keypair per cluster
(matches eks-v1.md Q8), PEM-marshals the private as PKCS8,
envelope-encrypts it via handlers_iam.EncryptSecret (AES-256-GCM,
AwsGW masterKey), writes the ciphertext to OIDCSigningKeyKey, and
publishes the public JWKS document to OIDCJWKSKey. Returns the JWKS
bytes so CreateCluster can hand them straight to user-data without
a follow-up KV read.
- JWKS shape mirrors what handlers_sts.JWKS already decodes for IRSA
verification (RFC 7517 EC P-256, kid = base64url SHA-256 of SPKI DER,
alg ES256, use sig). Types are defined locally to avoid a
handlers_eks → handlers_sts → handlers_eks import cycle; JSON wire
compat is the contract.
- LoadClusterOIDCPrivateKey is the recovery-path read for future
stages (reconciler restart, restore-snapshot tooling). Validates
PKCS8 parses to an ECDSA P-256 *PrivateKey and rejects anything else.
- ZeroizeClusterOIDCKey overwrites the encrypted blob with zeros of
the same length, then deletes. DeleteCluster calls this *first* so a
crash mid-teardown leaves no recoverable key material. Idempotent
on missing key — supports retry semantics.
Tests cover JWKS shape (RFC 7517 EC P-256 with 32-byte SEC1
coordinates), per-cluster key uniqueness, round-trip private-key
decrypt matches JWKS public, wrong-master-key fails, missing-key
ErrClusterNotFound mapping, zeroize idempotency, JWKS survives
zeroize (carries no secret material), padCoord left-pad correctness.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(eks): cluster SG + NLB ensure/delete helpers
Stage 3 of Sprint 6b CreateCluster lifecycle. Adds the customer-account
SG provisioning + per-cluster NLB plumbing the CreateCluster body will
call between OIDC keypair (Stage 2) and K3s VM launch (Stage 4). Pure
interface-call helpers — no daemon, no KV, no NATS bus.
- EnsureClusterSGs creates (or returns) eks-cluster-{name}-control-plane-sg
and eks-cluster-{name}-nodegroup-sg in the customer VPC, tagged
spinifex:managed-by=eks + spinifex:eks-cluster=<name> + spinifex:eks-role.
Idempotent on group-name + vpc-id pair. DeleteClusterSGs is best-effort
with continue-on-error sweep.
- EnsureClusterNLB provisions the eks-<name> NLB (internal scheme), the
eks-<name>-cp target group (TCP, port 6443, target-type ip per eks-v1.md
Q13), and the TCP :443 forward listener. Returns ClusterNLB
{LoadBalancerArn, TargetGroupArn, ListenerArn, DNSName} so CreateCluster
can persist every ARN in ClusterMeta. Idempotent on cluster name;
resource-name length validated against the AWS 32-char cap so a too-long
cluster name fails with a clear error instead of a generic
InvalidParameterValue downstream.
- RegisterClusterTarget + DeregisterClusterTarget bind the K3s server VM
ENI IP to the TG on :6443. Stage 4 calls Register once the VM has its
ENI IP; DeleteCluster calls Deregister before VM termination.
- DeleteClusterNLB tears down NLB + TG. DeleteLoadBalancer already
cascades listener cleanup inside the ELBv2 impl. LB sweep error takes
precedence so DeleteCluster surfaces the more visible failure first
while the TG sweep still runs.
Narrow sgProvisioner + nlbProvisioner interfaces in-package give tests
hand-rolled recorder fakes; the production VPCService + ELBv2Service
satisfy them structurally so no adapter is needed at Stage 6 wire-up.
Tests cover fresh-creates with right names/tags/ports, idempotent reuse,
mixed-existence (control-plane exists, nodegroup created), name-too-long
rejection, empty-inputs rejection, missing-resource no-op on delete, and
first-error-surfaced sweep continuation on delete.
Plan: docs/development/feature/eks-6b-create-cluster.md
* feat(ec2): RunInstances honors pre-created NetworkInterfaceId
Closes the AWS-parity gap where RunInstances always auto-created a
primary ENI even when the caller specified
NetworkInterfaces[0].NetworkInterfaceId. The EKS Sprint 6b CreateCluster
launcher (Stage 4) needs to pre-create the K3s server VM's ENI so the
ENI can be tagged/owned distinctly from the launcher's instance; the
existing path silently created a second ENI and left the caller's
unattached.
Changes:
- ENICreator interface gains GetENI(accountID, eniID) → *ENIInfo. ENIInfo
is a translated struct (SubnetID, VpcID, PrivateIp, MAC, Status,
SecurityGroupIDs) so handlers/ec2/instance stays free of an import on
handlers/ec2/vpc.ENIRecord.
- daemon/vpc_adapter.go gains daemonENICreator.GetENI translating
VPCServiceImpl.GetENIRecord into ENIInfo.
- PrepareRunInstances inspects NetworkInterfaces[0].NetworkInterfaceId
after the existing subnet/SG lift block. If non-empty, the loop
dispatches to a new attachPreCreatedENI helper:
- Looks up the ENI under accountID.
- Rejects with InvalidNetworkInterface.InUse if Status == "in-use".
- Calls AttachENI to mark the ENI in-use and bind it to the new VM.
- Populates instance.ENIId / ENIMac and ec2.Instance network fields
(PrivateIp, SubnetId, VpcId, SecurityGroups, NetworkInterfaces[0]).
- Skips the auto-create + public-IP-auto-assign paths; the caller is
expected to manage public IP out-of-band.
- Auto-create path is left untouched for every other code path: callers
that pass SubnetId without a NetworkInterfaceId still get a fresh ENI
with the existing semantics.
Tests cover the four shapes: happy path (auto-create not called, attach
called once, instance populated), in-use ENI rejected with the canonical
AWS error code, lookup error surfaces verbatim, attach error surfaces
verbatim.
* feat(eks): K3s control-plane VM launcher with cloud-init user-data
LaunchK3sServerVM resolves the eks-server AMI by name, pre-creates the
customer-account control-plane ENI in the supplied subnet with the
control-plane SG, renders cloud-init user-data (first-boot env, OIDC
private key, K3s config.yaml with tls-san + service-account args), then
calls RunInstances with NetworkInterfaces[0].NetworkInterfaceId pointing
at the pre-created ENI. RunInstances failures roll back the ENI so the
caller does not leak a customer-account resource.
TerminateK3sServerVM tears the instance and ENI down for DeleteCluster;
missing identifiers are a no-op so retries stay safe.
ENI + VM both live in the customer account (deviation from plan §7 which
specified a SystemAccount-owned VM); cross-account ENI attach is deferred
to a follow-up bead.
* feat(eks): NATS bootstrap subscriber + cluster reconciler + leader lease
NATSBootstrap subscribes to the four single-shot subjects the K3s server VM
publishes (k3s-bootstrap-token, k3s-admin-kubeconfig, k3s-oidc-jwks, k3s-ca)
on eks.bus.{accountID}.{clusterName}.{kind}. Token and admin kubeconfig are
envelope-encrypted via handlers_iam.EncryptSecret and persisted to KV. The
published JWKS is cross-checked against the controller-generated keypair by
kid and rejected on mismatch. The CA lands on meta.CertificateAuthorityB64
via the new SetClusterCertificateAuthority CAS updater.
ClusterReconciler holds a leader lease on spinifex-eks-leader/{accountID}/
{clusterName} (60s TTL, 20s refresh) and drives:
- CREATING -> ACTIVE once the four bootstrap KV artifacts are present and
NLB /healthz responds 200.
- ACTIVE -> probe /healthz on the configured interval, warn on failure.
- DELETING / FAILED -> exit (DeleteCluster body owns teardown).
Loss of the leader lease (CAS conflict or different holderID) returns
ErrReconcilerLeaseLost so the caller can re-acquire from another node.
Tested with embedded JetStream + a stub HTTPDoer; race-clean.
* feat(eks): OIDC issuer URL builder + reconciler registry
ClusterOIDCIssuer renders the {gatewayBaseURL}/oidc/eks/{region}/{accountID}/
{clusterName} URL kube-apiserver advertises as service-account-issuer and
that STS / external IRSA validators dereference. Validates URL safety on the
path segments so a hostile cluster name cannot inject scheme/path fragments.
ReconcilerRegistry tracks per-(accountID, clusterName) reconciler goroutines
with idempotent Spawn (drop duplicate spawn calls), Stop (cancel one + wait
for release), and StopAll (graceful daemon shutdown). RunClusterReconciler
is the production SpawnReconcilerFn: constructs a ClusterReconciler,
acquires the leader lease, runs Run in a goroutine. Both helpers consumed
by the upcoming service_impl CreateCluster body.
* feat(eks): CreateCluster/DescribeCluster/ListClusters/DeleteCluster bodies
Wires the EKSServiceImpl orchestration deps (VPC SG/K3s/Subnet, NLB,
Instance launcher, AMI resolver, master key, gateway base URL, region,
holder ID, reconciler registry) and implements the four lifecycle
methods:
- CreateCluster: validate → CREATING meta → SGs → NLB → OIDC issuer +
keypair → K3s server VM (cloud-init carries OIDC private PEM) →
register ENI IP with NLB target group → spawn NATS bootstrap
subscriber + per-cluster reconciler via the registry.
- DescribeCluster: KV meta read → eks.Cluster shape, with
ErrClusterNotFound translated to ResourceNotFoundException at the
service boundary.
- ListClusters: KV key-prefix scan + sorted maxResults/nextToken
pagination.
- DeleteCluster: SetClusterStatus DELETING → zeroize OIDC key → stop
reconciler via registry → deregister NLB target + delete NLB →
terminate K3s VM → delete SGs → delete cluster KV prefix.
Daemon wiring:
- buildEKSServiceDeps loads MasterKey from BaseDir/config/master.key
best-effort, resolves the gateway base URL via the same precedence the
LB-agent block applies later, and passes VPCService + ELBv2Service +
ImageService directly while adapting InstanceServiceImpl via
daemonEKSInstanceLauncher (RunInstances passes through; Terminate-
Instances loops vmMgr.Get + StopOrTerminateInstance in-process).
- SubnetVPCResolver is satisfied by daemonEKSSubnetResolver wrapping
VPCService.GetSubnet.
- Startup spawns reconcilers for every persisted CREATING/ACTIVE cluster
via SpawnRegisteredReconcilers; shutdown calls eksService.Shutdown so
every per-cluster goroutine releases its lease before exit.
Test surface:
- service_impl_test: TestEKSServiceImpl_ClusterLifecycleShimMode covers
the four lifecycle methods when constructed via the back-compat shim
(CreateCluster/DeleteCluster → ServerInternal, DescribeCluster →
ResourceNotFound on empty bucket, ListClusters → empty list).
- daemon_handlers_eks_test: drop the four lifecycle subjects from the
NotImplemented routing loop; handler-side coverage now lives in
handlers/eks.
- awserrors: add ResourceNotFound (404) for the DescribeCluster path.
* fix(eks): load master.key from config dir not service data dir
eks_deps read master.key from {BaseDir}/config/master.key, but the daemon
systemd unit sets SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex (a per-service
data dir). admin init writes master.key to the shared config dir, so the
daemon looked in a path that only resolved when a {BaseDir}/config symlink
existed — awsgw has one, the daemon's nested data dir does not. Result:
MasterKey=nil, depsReadyForOrchestration() false, CreateCluster/DeleteCluster
collapse to ServerInternal.
master.key is a config artifact written next to spinifex.toml. Read it from
filepath.Dir(d.configPath), matching how the daemon already resolves
predastore.toml and TLS certs. No symlink or provisioning change needed.
* fix(eks): register + resolve eks-server AMI via tag, add import --ami-name
CreateCluster's K3s launch failed 'eks: AMI eks-server not found' on every
clean install. Two defects:
1. spx admin images import had no way to name a locally-built appliance AMI:
--name is a catalog lookup (rejects non-catalog values) and the AMI name
was hardcoded to ami-{distro}-{version}-{arch}. build-system-image.sh ran
'import --name $AMI_NAME' for eks-server/eks-agent/GPU images, which all
failed 'Image name not found in available images'.
2. The launcher resolved DescribeImages by name=='eks-server', a name the
pipeline could never produce.
Add --ami-name to override the registered AMI name on --file imports (fixes
eks-server, eks-agent, and GPU appliance imports). Resolve the EKS AMI by the
spinifex:managed-by=eks tag the pipeline already stamps, newest-wins on
multiple matches. Tag-based resolution survives the planned server+agent →
single EKS AMI unify untouched (role chosen per-instance at launch).
build-system-image.sh now passes --ami-name; Makefile gains
import-eks-server-image (IMPORT=1) and dev-install.sh imports the AMI when
DEV_INSTALL_EKS=1.
* fix(eks): give K3s server VM NATS token + CA so bootstrap can publish
The K3s control-plane VM was launched with NATSURL=ConnectedUrl() (loopback,
unreachable from the VM) and no auth material, while spinifex NATS requires a
shared token over TLS. The VM could not connect to NATS, so it never published
its four one-shot bootstrap messages and the cluster stayed in CREATING.
Hand the VM the daemon's shared NATS token + CA PEM and a VM-reachable NATS URL
(mirrors the ELBv2 LB-agent shared-credential pattern; spinifex has no nkeys
hierarchy to scope per-principal creds against).
- K3sServerInput: drop dead NATSCredsContent; add NATSToken + NATSCACert,
required by validateK3sServerInput. user-data writes SPINIFEX_NATS_TOKEN,
SPINIFEX_NATS_CA + the CA file; first-boot.env tightened to 0600.
- EKSServiceDeps + CreateCluster: pass NATSURL/NATSToken/NATSCACert.
- buildEKSServiceDeps: read CA from config.NATS.CACert; extract
resolveGatewayHost(); add resolveNATSURL() (same reachable host, NATS port).
- k3s-first-boot.sh: build nats CLI args with --token + --tlsca.
* fix(eks): retry transient bootstrap persists, fail cluster on permanent error
The NATS bootstrap subscriber tore down on the first persist error and the
cluster sat in CREATING forever. Core-NATS delivers a publication once, so a
transient KV blip during persist was unrecoverable and a malformed payload
(bad JWKS kid, non-base64 CA, empty envelope) silently stranded the cluster.
- persistWithRetry rides out transient persist failures in-handler (3 attempts,
200ms backoff) instead of unsubscribing and waiting for a re-publish that
never comes.
- errBootstrapPermanent classifies validation failures (malformed envelope,
empty fields, JWKS kid mismatch, non-base64 CA) as non-retryable so they
return immediately.
- spawnBootstrap marks the cluster FAILED (with reason) when Run exits with a
non-context error, so DescribeCluster surfaces the cause and DeleteCluster
can reclaim resources.
- MarkClusterFailed transitions to FAILED+StatusReason only from CREATING, so a
late bootstrap error cannot clobber a concurrent delete or healthy cluster.
- casUpdateMeta extracts the revision-checked RMW loop shared by the meta
setters.
* fix(eks): resume NATS bootstrap for CREATING clusters on daemon restart
SpawnRegisteredReconcilers re-spawned only the reconciler on boot, never the
bootstrap subscriber. A daemon restart while a cluster was CREATING (K3s VM
still booting) lost the subscriptions; the VM published its four one-shot
core-NATS messages to nobody and the cluster hung in CREATING forever.
- BootstrapPendingKinds reports which artifacts (token, kubeconfig, CA) are
still missing from KV. JWKS is excluded: it persists no artifact and
re-waiting on a once-published subject would deadlock — its cross-check is
best-effort across a restart.
- RunForKinds subscribes to a subject subset (nil = all four); Run delegates to
it. subjectHandlers maps each kind to its handler and rejects unknown kinds.
- SpawnRegisteredReconcilers resumes bootstrap for CREATING clusters, waiting
only on the missing artifacts, so a mid-CREATING restart still drives the
cluster to ACTIVE.
* fix(eks): DeleteCluster must not wipe KV or report success on teardown failure
DeleteCluster ran zeroize, NLB teardown, and VM termination as warn-and-continue
then unconditionally swept the per-cluster KV prefix and returned success. A
failed NLB or VM teardown leaked billable resources with no owning record (the
meta holding their ARNs/IDs was erased) and told the operator the cluster was
gone. Zeroize — a stated security guarantee of no recoverable key material —
had its failure ignored too.
- Zeroize, NLB delete, and VM terminate are now blocking: their failures
accumulate (errors.Join) and abort the call before the KV sweep, leaving the
cluster DELETING and returning the error so a retry can still reach the ARNs.
- KV prefix sweep runs only after NLB + VM teardown confirm.
- SG cleanup stays best-effort (plan step 5) — a leaked SG strands no owning
record — and target deregister stays best-effort since DeleteClusterNLB tears
down the whole NLB + target group.
* fix(eks): persist NLB + control-plane ARNs mid-CreateCluster so failures don't leak
CreateCluster set meta.NLBArn / target-group / instance / ENI in memory and
persisted them only in the final PutClusterMeta, after LaunchK3sServerVM. Any
failure between EnsureClusterNLB and that persist left the NLB + target group +
listener + backing LB VM created but unrecorded; markFailed did no rollback and
DeleteCluster keys teardown off the persisted ARNs, so the resources leaked.
- Persist meta immediately after EnsureClusterNLB (records NLB/TG ARNs) and
again after LaunchK3sServerVM (records instance + ENI), before the next
fallible step.
- A failed create now leaves a FAILED meta carrying the resource references, so
the DeleteCluster teardown path (165.9) reclaims every NLB/TG/listener/VM.
- Tests: NLB-arn-persisted-before-later-failure, failed-create-then-delete
reclaims NLB, and a CreateCluster happy path. Shared eksServiceFixture
extracted from the delete-cluster fixture.
* fix(sts): parse path-based EKS OIDC issuer so IRSA accepts 6b cluster tokens
ParseEKSIssuerURL required an `oidc.eks.` host prefix and an exactly-2-segment
path (AWS hosted-EKS shape), but ClusterOIDCIssuer emits — and awsgw serves —
https://{gateway-host}/oidc/eks/{region}/{accountID}/{clusterName}. Every 6b
cluster's iss claim was rejected with "issuer host must start with oidc.eks",
so AssumeRoleWithWebIdentity would fail all IRSA tokens once 6e serves them.
Reconcile on the path-based shape, which is what is actually reachable on
bare-metal/on-prem (no oidc.eks.* vhost or DNS exists). Host trust comes from
the IAM OIDC-provider registration (the full issuer string is the registry key)
and the JWKS signature check, not a host pattern — so the parser drops the host
prefix rule and instead pins accountID/clusterName at fixed positions in a
strict 5-segment /oidc/eks/{region}/{accountID}/{clusterName} path. A crafted
iss with extra or missing segments still fails closed.
Adds a cross-component round-trip test asserting a ClusterOIDCIssuer-built iss
parses back to the same (accountID, clusterName).
* fix(eks): reclaim FAILED clusters on retry + cluster-scoped exists/not-found errors
A failed CreateCluster left a FAILED ClusterMeta that poisoned retries: the
preflight returned ErrorELBv2TargetGroupInUse ("The target group is currently
in use...") for a cluster-exists condition, and because that maps to a
retryable code the SDK's auto-retry surfaced ResourceInUseException instead of
the original failure. FAILED clusters were also a dead state — visible to
neither a clean recreate nor a clear lifecycle.
- Add EKS-specific ResourceInUseException / ResourceNotFoundException codes with
cluster-scoped messages; CreateCluster's exists check and DescribeCluster /
DeleteCluster not-found now use them instead of the ELBv2 / EC2 codes.
- CreateCluster reclaims a FAILED cluster: it purges the prior attempt's
recorded resources and clears KV before recreating, so a retry (including the
SDK auto-retry) is idempotent rather than blocked. Teardown failure surfaces
rather than recreating on top of orphans.
- Extract purgeClusterInfra, shared by DeleteCluster and the reclaim path
(blocking zeroize/NLB/VM, best-effort SG, KV sweep only after teardown
confirms).
- FAILED clusters remain observable via Describe (status=FAILED) and List — no
state where create blocks but describe/list show nothing.
* fix(eks): fail CREATING clusters that exceed create timeout
A per-cluster reconciler that never observes a ready control plane (bootstrap
artifacts missing or healthz unreachable) previously stayed CREATING forever.
Add a configurable create timeout (default 15m): once meta.CreatedAt is older
than the deadline and the cluster is still not ready, the reconciler marks it
FAILED with a reason, exiting the loop. Within the deadline it stays CREATING.
* fix(eks): surface ACTIVE cluster health in describe-cluster
The reconciler only slog.Warn'd a failing /healthz probe for an ACTIVE
cluster, so describe-cluster kept reporting a dead control plane as healthy.
Add HealthIssue + LastHealthProbe to ClusterMeta, write them from the ACTIVE
reconcile branch (recording the failure reason, clearing on recovery), and map
a non-empty HealthIssue to a ClusterHealth ClusterUnreachable issue in
describe-cluster.
* test(eks): add plan-mandated CreateCluster/DeleteCluster service cases
service_impl_test only covered shim mode. Add the four cases the plan's Testing
section requires: CONFIG_MAP auth mode rejected (and API accepted), missing
subnetIds rejected, DescribeCluster 404 with full deps wired (past the shim
short-circuit), and the DeleteCluster zeroize-before-teardown ordering proven
by forcing a teardown failure and asserting the OIDC key is already wiped.
* refactor(eks): return OIDC private-key PEM from generator, drop re-read
CreateCluster discarded GenerateClusterOIDCKeypair's return then did a second
KV get + AES-GCM decrypt via loadOIDCPrivateKeyPEM to re-derive the same PEM the
generator already had in hand. Return the plaintext PEM (alongside the JWKS)
from the generator and use it directly; remove loadOIDCPrivateKeyPEM and the now
-unused crypto/x509 + encoding/pem imports in service_impl.
* fix(eks): make CreateCluster/DeleteCluster failures diagnosable
Unmet orchestration deps and a missing eks-server AMI both collapsed to a bare
ServerInternal with no request-time log, leaving the client a 'contact AWS
re:Post' message and zero cause. Enumerate the specific missing deps and log
them at ERROR, returning ServiceUnavailable (an operator/config fault, not an
unrecoverable internal one). Add an ErrEKSServerAMINotFound sentinel so a
missing control-plane AMI is logged and surfaced as ServiceUnavailable too.
* fix(eks): serialize REST-JSON responses via jsonutil so the AWS SDK can parse them
WriteJSONResponse marshaled aws-sdk-go *Output structs with encoding/json,
which ignores their locationName tags and emits Go PascalCase keys. EKS
clients parse restjson on the lowercase locationName, so every field of
list/describe/create/delete responses was silently dropped (empty endpoint,
null identity.oidc.issuer). Marshal via jsonutil.BuildJSON — the same
marshaler the SDK uses for restjson bodies — which honors locationName.
Adds wire-shape tests asserting lowercase keys present and PascalCase absent.
* feat(eks): launch K3s server VM via boot-agnostic system-instance path
The EKS K3s control-plane VM launched through the plain tenant RunInstances
path, so it had only its VPC ENI and no management-bridge NIC -- no route to
the daemon's NATS endpoint. It never published its bootstrap subjects and the
cluster stayed CREATING.
Generalize LaunchSystemInstance to a boot-agnostic contract supporting both
system-instance boot styles:
- BootDirect: direct-boot microVM (fw_cfg blobs) -- ELBv2 LB agent
- BootAMI: AMI boot + cloud-init seed (q35) -- EKS K3s server VM
The launch contract moves to a neutral handlers/sysinstance package so EKS can
launch system VMs without an eks->elbv2 dependency; handlers/elbv2 re-exports
the types as aliases. The BootAMI branch mirrors the daemon's
Prepare -> Insert -> Launch split and allocates the mgmt NIC on the VM record
between Insert and Launch, so the cloud-init network-config renders a static
mgmt0 interface. The EKS NATSURL is now sourced from the mgmt-bridge IP (the
on-link address the VM's mgmt NIC can reach), falling back to the gateway-host
resolution when no mgmt bridge exists.
Verified: K3s VM now boots from the eks-server AMI under q35 with a mgmt0 NIC
(10.15.8.x), reachable from the host over br-mgmt, with NATS on-link at the
bridge IP. No AMI rebuild required.
* fix(eks): make DeleteCluster K3s teardown idempotent so it can't wedge
TerminateK3sServerVM treated every DeleteNetworkInterface error -- and every
TerminateSystemInstance error -- as blocking. Because instance terminate is
async, the immediate ENI delete races it: the first attempt fails
InvalidNetworkInterface.InUse, and once the instance-terminate cascade releases
the ENI every retry fails InvalidNetworkInterfaceID.NotFound. Both joined into
purgeClusterInfra's teardownErrs gate, which short-circuits before the SG sweep
and KV prefix delete -- so once the ENI was gone the cluster was stuck DELETING
permanently, leaking its control-plane + nodegroup SGs and KV meta.
Classify the teardown errors instead of blanket-propagating:
- ENI InvalidNetworkInterfaceID.NotFound / InvalidNetworkInterface.NotFound
-> idempotent success (the cascade or a prior retry already deleted it);
must not block the downstream SG + KV sweep.
- ENI InvalidNetworkInterface.InUse (VM still terminating) and any other
error -> retryable; surfaced so the cluster stays DELETING and the
reconciler retries once the instance releases the ENI.
- Instance already gone -> idempotent. TerminateSystemInstance now returns a
sysinstance.ErrSystemInstanceNotFound sentinel that the EKS path tolerates
via errors.Is, so a retried delete after the VM drained completes.
Add awserrors.IsErrorCode(err, code) as the shared delete-error classifier
(substring match through %w wrapping) and route the lone ad-hoc strings.Contains
guard in daemon/vm_adapters.go through it.
Verified: delete-cluster on three clusters left wedged DELETING / FAILED (ENI +
VM already gone) all complete -- clusters erased, SGs + ENIs reclaimed, no leak.
* fix(eks): gate ACTIVE on verified JWKS cross-check, not pre-seeded key
The controller pre-seeds OIDCJWKSKey at create time, so its presence
proved nothing about the running cluster's actual signing key. A cluster
could reach ACTIVE — and start signing IRSA tokens — before the K3s VM's
published JWKS was ever cross-checked against the generated keypair.
Add an OIDCJWKSVerifiedKey marker written by persistJWKS only after
assertJWKSMatch passes. The reconciler's ACTIVE gate and
BootstrapPendingKinds now key off the marker, not the pre-seeded JWKS.
assertJWKSMatch is map-based and also rejects kty mismatch.
* fix(eks): collapse duplicated gateway-host switch into resolveGatewayHost
startCluster reimplemented the 5-case gateway-host precedence inline,
duplicating resolveGatewayHost (used by the OIDC issuer and EKS NATS URL).
If one drifted, the OIDC issuer host embedded in cluster tokens would
diverge from the host LB VMs use to reach the daemon -> IRSA iss mismatch
and cluster-wide token rejection.
startCluster now calls resolveGatewayHost and derives the multi-node
mgmt host route (mgmtRouteVia) from the resolved host. resolveGatewayHost
carries the canonical case enumeration; a table test pins all five
branches plus the case-1 skip guards and the no-host fallthrough.
* fix(eks): make eks-server VM boot diagnosable + fix first-boot publisher crash
Three issues found while diagnosing why K3s clusters never leave CREATING:
1. setup.sh: the stock Alpine cloud image lists `console=tty0` LAST in
default_kernel_opts, so Linux binds /dev/console to the framebuffer and the
host-side serial capture sees only kernel dmesg — OpenRC, cloud-init, and
k3s-first-boot output were invisible. Reorder so console=ttyS0 is last.
2. k3s-first-boot.initd: command_background was "no" while output_log/error_log
were set; start-stop-daemon rejects --stdout/--stderr without --background
("Failed to start k3s-first-boot"), so the bootstrap publisher never ran and
the cluster wedged in CREATING. Background it (the publisher waits up to 5min
on K3s /healthz and must not block the runlevel anyway) and route its output
to the serial console, which the orchestrator captures (the VM has no login).
3. k3s-first-boot.sh: on the /healthz timeout, dump the tail of /var/log/k3s.log
to the console so the apiserver's own failure is visible host-side.
* fix(eks): seed K3s apiserver a public OIDC key, not the private key
buildK3sUserData pointed both kube-apiserver service-account args at the same
private-key file. --service-account-key-file requires a PUBLIC key — k8s
ParsePublicKeysPEM rejects a private-key PEM with "data does not contain any
valid RSA or ECDSA public keys" and the apiserver crash-loops, so K3s never
reaches /healthz, first-boot never publishes, and the cluster wedges in
CREATING.
Add PublicKeyPEMFromPrivate (PKIX public PEM from the PKCS#8 private key) and
seed it to a separate /etc/rancher/k3s/oidc-signing-key.pub.pem. Point
service-account-key-file at the public file; keep service-account-signing-key-file
on the private file. Runtime-verified: the apiserver crash is gone and K3s now
starts and schedules system pods.
* feat(eks): egress-only SNAT + DNS for K3s control-plane VM
The K3s server VM (a hidden system instance) could not pull system-pod
images: it had no internet egress and no working resolver, so
coredns/traefik/metrics-server never started and /healthz timed out.
Egress: give the single control-plane VM egress-only internet, confined
to its /32, with no inbound exposure and no customer-visible resource.
- policy/route.go: SystemInstanceEgressPriority (1200, above the drop
gate) + AddSystemInstanceEgress/Delete — a reroute scoped to
ip4.src == <vmIP>/32 so peers in a shared subnet are unaffected.
- policy/nat.go: AddSystemInstanceSNAT/Delete — a plain snat (not
dnat_and_snat, so no inbound path), idempotent on its pool external IP.
- external/igw.go: L5 EnsureSystemInstanceEgress/Remove resolves the
gateway nexthop and attaches the reroute + snat (mirrors the NAT-GW
egress path; ADR-0006 S1/S4/S5 preserved).
- subscribers: vpc.add/delete-system-egress topics + handlers.
- handlers/eks: allocate a hidden (ManagedBy=eks) pool address on
CreateCluster and publish add; release + publish delete on teardown.
EIP dep wired through EKSServiceDeps + the daemon.
DNS: the eks-server AMI is Alpine/dhcpcd and /etc/resolv.conf is a
dangling symlink, so DHCP-supplied resolvers are never persisted. Write
a real /etc/resolv.conf from the K3s VM cloud-config via bootcmd (not
write_files — a write_files entry follows the dead symlink, fails, and
aborts the whole block, dropping first-boot.env + the k3s config).
* feat: EKS 6b single-node SQLite pivot + bootstrap-envelope fixes
Switch the K3s server VM from embedded etcd to embedded SQLite (kine)
for the single-node v1, so CreateCluster reaches a healthy control plane.
etcd's per-commit fsync stalls past the apiserver deadline on viperblock's
deferred-durability block path; SQLite tolerates it.
- k3s server: omit cluster-init (selects SQLite/kine), add anonymous-auth=true
so the reconciler's anonymous /healthz NLB probe returns 200.
- first-boot readiness gate: anonymous /healthz (401 under CIS anonymous-auth
=false) -> authenticated 'kubectl get --raw /readyz'.
- first-boot publish: emit a JSON BootstrapEnvelope for all four subjects
(token, admin-kubeconfig, oidc-jwks, ca); jwks and ca were absent before.
- nats pub: --stdin -> --force-stdin (nats CLI 0.4.0).
- manifest: add jq (builds the bootstrap-message JSON envelopes).
* feat: EKS reconciler probes control-plane mgmt IP until DNS lands
The host-side cluster reconciler cannot resolve or route to the VPC-internal
NLB DNS endpoint (no authoritative DNS / Eclipso yet), so CreateCluster timed
out in CREATING and was marked FAILED even though the control plane was healthy.
Probe the control-plane VM's br-mgmt NIC address directly — the one apiserver
endpoint reachable from the daemon. Capture the mgmt IP from the launch result
(SystemInstanceOutput.MgmtIP -> K3sServerOutput.MgmtIP -> ClusterMeta.
ControlPlaneMgmtIP) and build the reconciler health URL as
https://<mgmtIP>:6443/healthz. The probe client already sets InsecureSkipVerify,
so the mgmt IP need not be in the apiserver cert SAN.
Workaround until authoritative DNS + a host-reachable NLB front-end land, at
which point spawnReconciler reverts to the NLB endpoint.
64 KiB
64 KiB
Command Implementation Matrix
Spinifex Admin CLI
Platform management commands not exposed via the AWS gateway API. These are CLI-only commands.
Service Management
Service lifecycle commands for starting, stopping, and checking status of all Spinifex cluster services. Each service subcommand supports start, stop, and status operations.
| Command | Flags | Prerequisites | Basic Logic | Status |
|---|---|---|---|---|
spx service predastore start |
--port (default: 8443), --host (default: 0.0.0.0), --base-path, --config-path, --debug, --tls-cert, --tls-key, --backend (distributed/filesystem, default: distributed), --node-id (default: 0), --pprof, --pprof-output |
TLS cert/key, base-path, config-path required | Creates predastore service instance with S3-compatible storage backend → starts service | DONE |
spx service predastore stop |
— | Predastore must be running | Stops the predastore service | DONE |
spx service predastore status |
— | None | Reports predastore service status | DONE |
spx service viperblock start |
--s3-host (default: 0.0.0.0:8443), --s3-bucket (default: predastore), --s3-region (default: ap-southeast-2), --plugin-path (auto-detected via nbdkit --dump-config plugindir; typically /usr/lib/x86_64-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so on amd64, overridable via SPINIFEX_VIPERBLOCK_PLUGIN_PATH in /etc/spinifex/systemd.env) |
--config required, plugin-path must exist on disk |
Loads cluster config → connects to NATS and Predastore → starts viperblock block storage service with NBD plugin | DONE |
spx service viperblock stop |
— | Viperblock must be running | Stops the viperblock service | DONE |
spx service viperblock status |
— | None | Reports viperblock service status | DONE |
spx service nats start |
--port (default: 4222), --host (default: 0.0.0.0), --debug, --data-dir, --jetstream |
None | Starts embedded NATS server with optional JetStream | DONE |
spx service nats stop |
— | NATS must be running | Stops the NATS service | DONE |
spx service nats status |
— | None | Reports NATS service status | DONE |
spx service spinifex start |
--wal-dir |
--config required |
Loads cluster config → starts spinifex daemon (VM orchestration, NATS subscriptions, health endpoint) | DONE |
spx service spinifex stop |
— | Spinifex daemon must be running | Stops the spinifex daemon service | DONE |
spx service spinifex status |
— | None | Reports spinifex daemon service status | DONE |
spx service awsgw start |
--host (default: 0.0.0.0:9999), --tls-cert, --tls-key, --debug |
--config required |
Loads cluster config → starts AWS-compatible gateway with SigV4 auth, IAM policy enforcement, TLS | DONE |
spx service awsgw stop |
— | AWS gateway must be running | Stops the AWS gateway service | DONE |
spx service awsgw status |
— | None | Reports AWS gateway service status | DONE |
spx service spinifex-ui start |
--port (default: 3000), --host (default: 0.0.0.0), --tls-cert, --tls-key |
None | Starts embedded web UI server serving the React frontend. Aliases: ui, spinifexui |
DONE |
spx service spinifex-ui stop |
— | spinifex-ui must be running | Stops the spinifex-ui service | DONE |
spx service spinifex-ui status |
— | None | Reports spinifex-ui service status | DONE |
spx service vpcd start |
— | --config required, OVN/OVS installed |
Loads cluster config → starts VPC daemon (subscribes to vpc.* NATS events, translates to OVN logical switches/ports/routers) |
DONE |
spx service vpcd stop |
— | vpcd must be running | Stops the vpcd service | DONE |
spx service vpcd status |
— | None | Reports vpcd service status | DONE |
Cluster Inspection
Operational commands for inspecting cluster state. These fan out NATS requests to all nodes and aggregate responses.
| Command | Flags | Prerequisites | Basic Logic | Output Columns | Status |
|---|---|---|---|---|---|
spx get nodes |
--timeout (default: 3s) |
Cluster must be running (NATS) | Loads config → publishes to spinifex.node.status fan-out topic → collects responses within timeout → merges config-known nodes with NATS responders → nodes that don't respond shown as NotReady |
NAME, STATUS, IP, REGION, AZ, UPTIME, VMs, SERVICES | DONE |
spx get vms |
--timeout (default: 3s) |
Cluster must be running (NATS) | Publishes to spinifex.node.vms fan-out topic → collects VM info from all nodes → sorts by node then instance ID → prints table. Alias: spx get instances |
INSTANCE, STATUS, TYPE, VCPU, MEM, NODE, IP, AGE | DONE |
Resource Monitoring
| Command | Flags | Prerequisites | Basic Logic | Output | Status |
|---|---|---|---|---|---|
spx top nodes |
--timeout (default: 3s) |
Cluster must be running (NATS) | Publishes to spinifex.node.status fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability |
Table 1: NAME, CPU (used/total), MEM (used/total), GPU (used/total), VMs — GPU column shows allocated/total when passthrough is enabled, 0/count* when the node is GPU-capable but passthrough is disabled, or - when no GPU is present. Table 2: INSTANCE TYPE, AVAILABLE, VCPU, MEMORY |
DONE |
Cluster Initialization
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|
spx admin init |
--nodes, --node, --bind, --port, --region, --az, --cluster-name, --cluster-bind, --cluster-routes, --predastore-nodes, --services, --formation-timeout, --token-ttl, --force |
None (first-time setup) | Generates root IAM credentials (AKIA-prefixed access key + secret) → creates master.key (AES-256, 32 bytes, 0600) → writes bootstrap.json (consumed on first start) → generates CA + server TLS certificates → generates join token (written to join-token file, displayed in join command) → creates NATS config with auth token → writes spinifex.toml, awsgw.toml, predastore.toml → configures AWS CLI spx profile → creates directory structure under ~/spinifex/ |
1. Init creates all config files 2. Root credentials printed once 3. master.key is 32 bytes, mode 0600 4. bootstrap.json consumed on first start 5. --force re-initializes existing config6. AWS CLI profile spx auto-configured7. Multi-node init generates join token and writes to <config-dir>/join-token |
DONE |
spx admin join |
--host (required), --node (required), --token (required), --bind, --port, --region, --az, --cluster-bind, --cluster-routes, --data-dir, --services |
Leader node must be running | Connects to leader node with join token (Authorization: Bearer header) → retrieves cluster configuration → configures local node to join cluster and participate in distributed operations | 1. Join existing cluster 2. Missing host (error) 3. Missing node name (error) 4. Missing token (error) 5. Invalid token (401) 6. Expired token (401) |
DONE |
Version
| Command | Flags | Prerequisites | Basic Logic | Status |
|---|---|---|---|---|
spx version |
— | None | Prints Spinifex version, commit hash, OS, and architecture (populated via build-time ldflags) | DONE |
Cluster Operations
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|
spx admin cluster shutdown |
--force (shutdown even if nodes don't respond), --timeout (max wait per phase, default 120s), --dry-run (print phase plan without executing) |
Cluster must be running | Performs coordinated, phased shutdown of entire cluster. Phases execute in order: GATE (stop API/UI) → DRAIN (stop VMs) → STORAGE (stop viperblock) → PERSIST (stop predastore) → INFRA (stop NATS/daemon). Each phase waits for all nodes to ACK before proceeding. Uses JetStream state tracking. | 1. Shutdown running cluster 2. All nodes stop cleanly 3. Force shutdown with unresponsive nodes 4. Dry-run prints plan |
DONE |
Certificate Management
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|
spx admin cert renew |
--extra-ip (additional IPs for SANs), --extra-dns (additional DNS names for SANs) |
Existing CA from spx admin init |
Reads existing CA → regenerates server certificate with all current network interface IPs and machine hostname in SANs → writes new cert. Use after adding a new network interface or changing IP addresses. | 1. Renew with auto-detected IPs 2. Renew with extra IPs 3. Renew with extra DNS names |
DONE |
Upgrade Management
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|
spx admin upgrade |
--yes (apply without prompting), --config-dir (persistent, default: ~/spinifex/config), --spinifex-dir (persistent, default: ~/spinifex) |
Existing installation (spinifex.toml must exist in config dir) |
Reads current config versions from registry → computes pending config migrations (from→to per target) → prints version summary and pending list → prompts for confirmation unless --yes → runs migrate.DefaultRegistry.RunAllConfig() to apply migrations to config files. Intended for upgrades between Spinifex versions. Operators can skip migrations during install by setting INSTALL_SPINIFEX_SKIP_MIGRATE=1, then run spx admin upgrade manually to review and apply. After completion, services must be restarted with sudo systemctl restart spinifex.target. Invoked non-interactively by scripts/setup.sh with --yes. |
1. No installation present (error, suggests spx admin init)2. No pending migrations (prints "No pending config migrations") 3. Pending migrations with interactive confirmation (y/yes applies, anything else aborts) 4. Pending migrations with --yes (applies without prompt)5. Migration failure surfaces error and exits non-zero |
DONE |
Account Management
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|
spx admin account create |
--name |
Cluster must be running (NATS), master key must exist | Connects to NATS → CAS loop on spinifex-account-counter:next_id for sequential 12-digit ID → creates Account record in spinifex-accounts KV → creates admin user in new account → creates access key for admin → creates AdministratorAccess policy (Action:, Resource:) → attaches policy → prints credentials |
1. Create first account (ID 000000000001) 2. Sequential IDs (000000000002, etc.) 3. Admin user has full access 4. Credentials printed once |
DONE |
spx admin account list |
— | Cluster must be running (NATS), master key must exist | Connects to NATS → IAMService.ListAccounts() → prints table with Account ID, Name, Status, Created | 1. List all accounts 2. Empty list (only global account) |
DONE |
Image Management
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|
spx admin images import |
--name, --ami-name, --file, --distro, --version, --arch, --platform, --boot-mode (bios/uefi/uefi-preferred), --tag, --force, --skip-verify |
Cluster must be running; either --name (catalog download) or --file (operator-supplied media). --file mode additionally requires --distro, --version, --arch, --boot-mode. |
Catalog imports (--name) download the image, fetch the catalog Checksum URL, verify the SHA-256/SHA-512 digest, and inherit BootMode from the catalog entry. --boot-mode overrides the catalog value when set. Mismatch fails closed; the cached file is left on disk and --force re-downloads. --file imports skip checksum verification (operator-supplied media is outside Spinifex's trust boundary, the skip is logged at INFO for audit) and require an explicit --boot-mode because there is no catalog metadata to inherit from. --ami-name overrides the registered AMI name (the DescribeImages name), which otherwise defaults to ami-{distro}-{version}-{arch}; use it for locally-built appliance images (e.g. --ami-name spinifex-eks-server) not in the catalog. --skip-verify bypasses verification for catalog imports and emits a WARN slog + stderr notice; use only for debugging or when upstream mirrors are confirmed-broken. |
1. Import valid catalog image (verifies checksum) 2. Tampered cache hit fails with ErrChecksumMismatch3. --force recovers after a mismatch4. --file import skips verification5. --skip-verify bypasses checksum with WARN6. --file without --boot-mode rejected7. --boot-mode=invalid rejected |
DONE |
spx admin images list |
— | None | Lists available OS images that can be imported or downloaded | 1. List available images | DONE |
spx admin images remove |
--image-id (required), --force, --yes |
Cluster must be running; predastore reachable. Targets AMIs imported via spx admin images import (non-account ImageOwnerAlias, e.g. "system"). |
Loads ami-<id>/config.json, walks transitive dependents — copied snapshots whose VolumeID == imageID, volumes whose SnapshotID references the internal snap-ami-<id> or any derived snap, and account AMIs created via CopyImage whose SnapshotID is a derived snap — then prompts (skipped with --yes) before deleting ami-<id>/config.json (the DescribeImages barrier) followed by the rest of ami-<id>/ and snap-ami-<id>/. Account-owned AMIs are refused with a hint pointing at aws ec2 deregister-image + aws ec2 delete-snapshot. --force bypasses the dependency, ownership and config-corrupt checks for salvage of orphaned blocks. |
1. Happy path: system AMI, no deps → deleted 2. Account-owned → refused with AWS-flow hint 3. Missing/corrupt config → InvalidAMIID.NotFound (salvageable with --force)4. Dependent volume (direct or via copied snap) → refused 5. Dependent account AMI → refused 6. --force overrides dependents7. Idempotent re-run after partial delete |
DONE |
Image integrity verification (CMMC SI.L1-3.14.2)
Catalog imports (spx admin images import --name <name>) verify the image
against the catalog-declared SHA-256/SHA-512 digest before extraction. The sums
file is fetched from the catalog Checksum URL over HTTPS only (cross-scheme
redirects refused), and verification runs on both fresh downloads and cache
hits so a poisoned cache is caught on the next import.
On mismatch the import exits non-zero, the cached file is left on disk for
inspection, and the printed guidance is spx admin images import --name <name> --force to re-download.
--file imports skip verification by design: operator-supplied media is
outside Spinifex's trust boundary and the operator is responsible for
integrity (e.g. sha256sum against a trusted upstream digest before import).
The skip is recorded as an INFO slog event with reason=local-file-import
so a CMMC assessor can audit the decision from journald.
--skip-verify bypasses the checksum step for catalog imports. The command
still downloads via the catalog URL but does not compare the image digest
against the sums file. Intended for narrow cases such as debugging upstream
mirror issues or running against a transiently-broken latest/ path; the
skip is logged at WARN with reason=skip-verify-flag and printed to stderr
so operators and assessors see it. Prefer --file with an out-of-band
verified image over --skip-verify whenever possible.
Limitation: verification confirms the image matches the digest the mirror served. A mirror compromise that swaps both image and sums file is not detected; closing that gap requires GPG signature verification of the sums file, deferred to a later phase.
spx admin images remove caveats
Admin-imported AMIs (ImageOwnerAlias = "system") live
under the ami-<id>/ S3 prefix and use a viperblock-internal snap checkpoint
at snap-ami-<id>/ — there is no snap-<id>/metadata.json. The AWS handlers
(DeregisterImage, DeleteSnapshot) reject system owners with
UnauthorizedOperation, which is the right behaviour for tenant API callers
but leaves no AWS-flow path to reclaim space. spx admin images remove is
the admin-trust-boundary counterpart that performs the dependency walk and
hard-deletes the blocks directly against predastore.
CopyImage of a system AMI is metadata-only: it writes a fresh
snap-<acct>/metadata.json whose VolumeID points at ami-<sys> and a new
ami-<acct>/config.json referencing that snap. Volumes launched from the
copied AMI read transitively from ami-<sys>/chunks/.... The remove command
walks this transitive set and refuses if anything references the target.
TOCTOU window: between the safety scan and the config.json delete a
concurrent RunInstances against the AMI could create a new dependent
volume. The window is sub-second on a healthy cluster. The admin running
this command is expected to know the fleet's operational state; if the race
fires the result is a vol-<id> with deleted backing blocks, recovered by
terminating the orphaned instance.
--force bypasses every safety check (dependents, ownership, missing /
corrupt config.json). Use only for salvage of orphaned blocks. Running it
against a live system AMI corrupts every dependent volume on the next disk
read.
GPU Management
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|---|---|---|---|---|---|
spx admin gpu status |
--node (default: local node) |
Cluster must be running (NATS) | Queries spinifex.node.status fan-out → finds the target node response → prints Node, GPU hardware (model list or "none detected"), IOMMU state, vfio-pci state, passthrough enabled/disabled, and GPU pool allocation (allocated/total). Also lists GPU-capable instance types when passthrough is active. |
1. Local node with passthrough enabled (shows pool and instance types) 2. Local node capable but passthrough disabled (shows "run enable") 3. Node without GPU hardware (shows "none detected") 4. --node targets a remote node5. Node not found or not responding (error) |
DONE |
spx admin gpu enable |
— | Cluster must be running (NATS); node must be GPU-capable (IOMMU + vfio-pci configured) | Checks current passthrough state via NATS → errors if already enabled or prerequisites not met (directs to setup) → writes gpu_passthrough = true to spinifex.toml via admin.SetGPUPassthrough → sends SIGHUP to spinifex-daemon via systemctl kill -s HUP → polls node status for up to 30 s until daemon confirms new state → prints final gpu status output. Must be run directly on the target host. |
1. Enable on capable node 2. Already enabled (no-op) 3. Prerequisites not met (error, suggests setup)4. Daemon does not confirm within 30 s (timeout error) |
DONE |
spx admin gpu disable |
— | Cluster must be running (NATS); no GPU instances may be running | Checks current passthrough state via NATS → errors if already disabled or if AllocGPUs > 0 (must terminate GPU instances first) → writes gpu_passthrough = false → sends SIGHUP to spinifex-daemon → polls for up to 30 s → prints final gpu status output. Must be run directly on the target host. |
1. Disable with no GPU instances running 2. Already disabled (no-op) 3. GPU instances running (error with count) 4. Daemon does not confirm within 30 s (timeout error) |
DONE |
spx admin gpu setup |
— | Must be run as root; NVIDIA or AMD GPU must be present | Idempotent host configuration for GPU passthrough. Steps (skipped if already applied): detect GPUs via gpu.Discover → collect PCI IDs for all IOMMU-group siblings → check/enable IOMMU in GRUB (intel_iommu=on iommu=pt or amd_iommu=on iommu=pt) → write vfio udev rule (/etc/udev/rules.d/99-spinifex-vfio.rules) → blacklist nouveau (/etc/modprobe.d/blacklist-nouveau.conf) → blacklist amdgpu if AMD GPU present (/etc/modprobe.d/blacklist-amdgpu.conf) → write vfio-pci early binding config (/etc/modprobe.d/vfio-pci.conf) → add vfio modules to initramfs. If any change requires a reboot: runs update-initramfs -u and exits with reboot instructions. After reboot: verifies vfio_pci module is loaded → verifies each GPU is bound to vfio-pci (binds explicitly via driver_override if unbound) → calls gpu enable to activate passthrough. |
1. First run on unconfigured host (GRUB updated, reboot required) 2. Re-run after reboot (verifies bindings, enables passthrough) 3. Already fully configured (all steps skipped, enables passthrough) 4. AMD GPU present (blacklists amdgpu in addition to nouveau) 5. Not run as root (error) 6. No GPU hardware detected (error) |
DONE |
spx admin gpu mig status |
— (must be run on target host) | NVIDIA GPU with MIG support present | Runs gpu.Discover() locally → for each GPU prints: PCI address, model, MIG capability, MIG mode (enabled/disabled/N/A); for GPUs with MIG enabled lists active MIG slices with GI ID, profile name, and mdev path. |
1. Node with MIG enabled (shows slices) 2. MIG capable but not enabled (shows disabled) 3. No MIG-capable GPU (shows N/A for MIG fields) |
DONE |
spx admin gpu mig enable |
--profile <name> (required, e.g. 1g.10gb), --gpu <pci-addr> (optional, default: all MIG-capable GPUs) |
Must be run as root; MIG-capable GPU present; no GPU instances running | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via gpu.Discover() (filtered by --gpu if set) → enables MIG mode on each target (gpu.EnableMIGMode) → lists available profiles (gpu.ListProfiles) and validates requested profile name → destroys any existing instances (gpu.DestroyAllInstances) → creates new instances filling GPU capacity (gpu.CreateInstances) → writes mig_profile to spinifex.toml via admin.SetMIGProfile → sends SIGHUP to spinifex-daemon. Must be run directly on the target host. |
1. Enable MIG with valid profile 2. Invalid profile name (error with available profile list) 3. No MIG-capable GPU (error) 4. GPU instances running (error) 5. --gpu targets specific GPU6. Not run as root (error) |
DONE |
spx admin gpu mig disable |
--gpu <pci-addr> (optional, default: all MIG-capable GPUs) |
Must be run as root; no GPU instances running | Checks no GPU instances running (NATS) → discovers MIG-capable GPUs via gpu.Discover() (filtered by --gpu if set) → destroys all GPU instances (gpu.DestroyAllInstances) → disables MIG mode (gpu.DisableMIGMode) → clears mig_profile in spinifex.toml via admin.SetMIGProfile → sends SIGHUP to spinifex-daemon. Must be run directly on the target host. |
1. Disable with no running GPU instances 2. GPU instances running (error) 3. MIG not enabled on GPU (skips that GPU) 4. --gpu targets specific GPU5. Not run as root (error) |
DONE |
AWS-Compatible API
EC2 — Instance Management
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
run-instances |
--image-id, --instance-type, --count, --key-name, --user-data, --subnet-id, --block-device-mappings (DeviceName, VolumeSize, VolumeType, Iops, DeleteOnTermination), --placement (GroupName), --iam-instance-profile (Name/Arn) |
--security-group-ids, --tag-specifications, --dry-run, --client-token, --disable-api-termination, --ebs-optimized, --network-interfaces, --private-ip-address, --monitoring, --credit-specification, --cpu-options, --metadata-options, --launch-template, --hibernate-options |
DONE |
describe-instances |
--instance-ids, --filters (instance-state-name, instance-id, instance-type, vpc-id, subnet-id, tag:*, tag-key, tag-value) |
--max-results, --next-token, --dry-run |
DONE |
start-instances |
--instance-ids |
--dry-run, --force |
DONE |
stop-instances |
--instance-ids |
--force, --hibernate, --dry-run |
DONE |
terminate-instances |
--instance-ids, DeleteOnTermination (per-volume) |
--dry-run |
DONE |
reboot-instances |
--instance-ids |
--dry-run |
DONE |
describe-instance-types |
--filters (capacity only) |
--instance-types, --max-results, --next-token, --dry-run, other filters |
DONE |
modify-instance-attribute |
--instance-id, --instance-type, --user-data, --disable-api-termination |
--ebs-optimized, --source-dest-check, --instance-initiated-shutdown-behavior, --block-device-mappings, --groups, --ena-support, --sriov-net-support |
DONE |
get-console-output |
--instance-id |
--latest, --dry-run |
DONE |
describe-instance-attribute |
--instance-id, --attribute (instanceType, userData, disableApiTermination, instanceInitiatedShutdownBehavior, disableApiStop, ebsOptimized, enaSupport, sourceDestCheck, rootDeviceName, kernel, ramdisk) |
--dry-run |
DONE |
describe-instance-credit-specifications |
--instance-ids |
--filters, --max-results, --dry-run |
DONE (stub — always returns standard) |
describe-instance-status |
--instance-ids, --include-all-instances, --filters (availability-zone, instance-state-code, instance-state-name, tag:*) |
--max-results, --next-token, --dry-run, event/instance-status/system-status filters |
DONE (static health) |
monitor-instances |
— | --instance-ids |
NOT STARTED |
unmonitor-instances |
— | --instance-ids |
NOT STARTED |
run-instances --iam-instance-profile enforces iam:PassRole on the contained
role ARN and rejects cross-account profile ARNs. Placement strategies: spread
(1 per node, atomic CAS) and cluster (pin all to one node).
EC2 — IAM Instance Profile Associations
Stored as vm.VM.IamInstanceProfileArn + IamInstanceProfileAssociationId
(one ARN per instance). Association IDs (iip-assoc-) are regenerated on every
Associate/Replace and never reused. Auto-disassociated on terminate; preserved
across stop/start.
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
associate-iam-instance-profile |
--instance-id, --iam-instance-profile (Name/Arn) |
--dry-run |
DONE |
disassociate-iam-instance-profile |
--association-id |
--dry-run |
DONE |
replace-iam-instance-profile-association |
--association-id, --iam-instance-profile (Name/Arn) |
--dry-run |
DONE |
describe-iam-instance-profile-associations |
--association-ids, --filters (instance-id, state) |
--max-results, --next-token, --dry-run |
DONE |
EC2 — Key Pairs
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-key-pair |
--key-name, --key-type (rsa/ed25519) |
--key-format, --tag-specifications, --dry-run |
DONE |
describe-key-pairs |
--key-names, --key-pair-ids, --filters (key-pair-id, key-name, fingerprint, tag:*) |
--max-results, --dry-run |
DONE |
delete-key-pair |
--key-name, --key-pair-id |
--dry-run |
DONE |
import-key-pair |
--key-name, --public-key-material |
--tag-specifications, --dry-run |
DONE |
EC2 — AMI Images
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
describe-images |
--image-ids, --owners (self/account-id/alias), --filters (name, state, architecture, image-id, is-public, owner-id, description, image-type, tag:*) |
--executable-users, --include-deprecated, --include-disabled, --max-results, --next-token, --dry-run |
DONE |
create-image |
--instance-id, --name, --description, --tag-specifications |
--no-reboot, --block-device-mappings, --dry-run |
DONE |
register-image |
--name, --description, --architecture (x86_64/arm64/i386), --root-device-name, --virtualization-type (hvm), --boot-mode (bios/uefi/uefi-preferred), --block-device-mappings (root w/ Ebs.SnapshotId+VolumeSize), --tag-specifications |
--billing-products, --uefi-data |
DONE |
deregister-image |
--image-id |
--dry-run |
DONE |
copy-image |
--source-image-id, --source-region, --name, --description, --client-token, --copy-image-tags, --tag-specifications (image only) |
--encrypted, --kms-key-id, --destination-outpost-arn, --dry-run |
DONE |
describe-image-attribute |
--image-id, --attribute (description, blockDeviceMapping) |
--dry-run, other attributes (launchPermission, bootMode, kernel, ramdisk, sriovNetSupport, productCodes, tpmSupport, uefiData, imdsSupport, lastLaunchedTime, deregistrationProtection) |
DONE |
modify-image-attribute |
--image-id, --description (top-level or structured) |
--launch-permission, --imds-support, --operation-type, --user-ids, --user-groups, --organization-arns, --product-codes, --dry-run, other --attribute values |
DONE |
reset-image-attribute |
--image-id, --attribute description |
--attribute launchPermission, --dry-run |
DONE |
import-image |
— | --disk-containers, --description, --architecture, --platform |
NOT STARTED |
copy-image is metadata-only — no block copy. The new snapshot inherits the
source VolumeID. register-image/copy-image accept uefi-preferred as
input but treat it as uefi at launch.
EC2 — Volumes (EBS)
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
describe-volumes |
--volume-ids, --filters (volume-id, status, size, volume-type, attachment.instance-id, attachment.status, attachment.device, availability-zone, tag:*), persisted DeleteOnTermination |
--max-results, --next-token, --dry-run |
DONE |
create-volume |
--size, --availability-zone, --volume-type (gp3), --snapshot-id |
--iops (hardcoded 3000), --encrypted (hardcoded false), --throughput, --tag-specifications |
DONE |
delete-volume |
--volume-id |
--dry-run |
DONE |
modify-volume |
--volume-id, --size, --volume-type, --iops |
--throughput, --dry-run, --multi-attach-enabled |
DONE |
attach-volume |
--volume-id, --instance-id, --device (auto-assigns /dev/sd[f-p]) |
--dry-run |
DONE |
detach-volume |
--volume-id, --instance-id (optional), --device, --force |
--dry-run |
DONE |
describe-volume-status |
--volume-ids, --filters (volume-id, volume-status.status, availability-zone) |
--max-results, --next-token, --dry-run |
DONE |
describe-volumes-modifications |
— | --volume-ids, --filters, --max-results |
NOT STARTED |
EC2 — Snapshots
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-snapshot |
--volume-id, --description, --tag-specifications |
--dry-run |
DONE |
delete-snapshot |
--snapshot-id |
--dry-run |
DONE |
describe-snapshots |
--snapshot-ids, --filters (snapshot-id, status, volume-id, volume-size, owner-id, tag:*) |
--owner-ids, --max-results, --dry-run |
DONE |
copy-snapshot |
--source-snapshot-id, --source-region, --description |
--encrypted, --dry-run |
DONE |
create-snapshots |
— | --instance-specification, --description, --tag-specifications |
NOT STARTED |
EC2 — Tags
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-tags |
--resources, --tags |
--dry-run |
DONE |
delete-tags |
--resources, --tags |
--dry-run |
DONE |
describe-tags |
--filters (resource-id, resource-type, key, value) |
--max-results, --next-token, --dry-run |
DONE |
EC2 — Regions, AZs, Account Attributes
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
describe-regions |
— (returns configured region) | --region-names, --filters, --all-regions, --dry-run |
DONE |
describe-availability-zones |
— (returns configured AZ) | --zone-names, --filters, --all-availability-zones |
DONE |
describe-account-attributes |
--attribute-names |
--dry-run |
DONE |
EC2 — Account Settings
Persistence works (NATS JetStream KV) but stored values are not yet enforced by downstream services.
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
enable-ebs-encryption-by-default |
— | --dry-run |
STARTED (enforcement pending) |
disable-ebs-encryption-by-default |
— | --dry-run |
STARTED (enforcement pending) |
get-ebs-encryption-by-default |
— | --dry-run |
STARTED (enforcement pending) |
enable-serial-console-access |
— | --dry-run |
STARTED (enforcement pending) |
disable-serial-console-access |
— | --dry-run |
STARTED (enforcement pending) |
get-serial-console-access-status |
— | --dry-run |
DONE |
enable-snapshot-block-public-access |
— | --state |
NOT STARTED |
disable-snapshot-block-public-access |
— | --dry-run |
NOT STARTED |
get-snapshot-block-public-access-state |
— | --dry-run |
NOT STARTED |
enable-image-block-public-access |
— | --image-block-public-access-state |
NOT STARTED |
disable-image-block-public-access |
— | --dry-run |
NOT STARTED |
get-image-block-public-access-state |
— | --dry-run |
NOT STARTED |
modify-instance-metadata-defaults |
— | --http-tokens, --http-put-response-hop-limit, --http-endpoint, --instance-metadata-tags |
NOT STARTED |
get-instance-metadata-defaults |
— | --dry-run |
NOT STARTED |
EC2 — VPC Core
VPC/Subnet/ENI/SG CRUD stores metadata in NATS KV and publishes events to vpcd for OVN translation. Single AZ for Spinifex v1.
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-vpc |
--cidr-block, --tag-specifications |
--instance-tenancy, --dry-run |
DONE |
delete-vpc |
--vpc-id |
--dry-run |
DONE |
describe-vpcs |
--vpc-ids, --filters (vpc-id, state, cidr-block, is-default, owner-id, tag:*) |
--max-results, --dry-run |
DONE |
modify-vpc-attribute |
--vpc-id, --enable-dns-hostnames, --enable-dns-support, --enable-network-address-usage-metrics |
--dry-run |
DONE |
describe-vpc-attribute |
--vpc-id, --attribute (enableDnsHostnames, enableDnsSupport, enableNetworkAddressUsageMetrics) |
--dry-run |
DONE |
associate-vpc-cidr-block |
— | --vpc-id, --cidr-block |
NOT STARTED |
disassociate-vpc-cidr-block |
— | --association-id |
NOT STARTED |
create-subnet |
--vpc-id, --cidr-block, --availability-zone, --tag-specifications |
--dry-run |
DONE |
delete-subnet |
--subnet-id |
--dry-run |
DONE |
describe-subnets |
--subnet-ids, --filters (vpc-id, subnet-id, availability-zone, cidr-block, state, default-for-az, tag:*) |
--max-results, --dry-run |
DONE |
modify-subnet-attribute |
--subnet-id, --map-public-ip-on-launch |
--assign-ipv6-address-on-creation, --dry-run |
DONE |
associate-subnet-cidr-block |
— | --subnet-id, --ipv6-cidr-block |
NOT STARTED |
disassociate-subnet-cidr-block |
— | --association-id |
NOT STARTED |
EC2 — Security Groups
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-security-group |
--group-name, --description, --vpc-id, --tag-specifications |
--dry-run |
DONE |
delete-security-group |
--group-id |
--dry-run |
DONE |
describe-security-groups |
--group-ids, --filters (vpc-id, group-name, group-id, description, ip-permission.cidr, tag:*) |
--group-names, --max-results, --dry-run |
DONE |
authorize-security-group-ingress |
--group-id, --ip-permissions |
--dry-run |
DONE |
authorize-security-group-egress |
--group-id, --ip-permissions |
--dry-run |
DONE |
revoke-security-group-ingress |
--group-id, --ip-permissions |
--dry-run |
DONE |
revoke-security-group-egress |
--group-id, --ip-permissions |
--dry-run |
DONE |
describe-security-group-rules |
--filters (group-id, security-group-rule-id, tag:*, tag-key), --security-group-rule-ids |
--max-results, --next-token, --dry-run |
DONE |
EC2 — Internet Gateway
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-internet-gateway |
--tag-specifications |
--dry-run |
DONE |
attach-internet-gateway |
--internet-gateway-id, --vpc-id |
--dry-run |
DONE |
detach-internet-gateway |
--internet-gateway-id, --vpc-id |
--dry-run |
DONE |
delete-internet-gateway |
--internet-gateway-id |
--dry-run |
DONE |
describe-internet-gateways |
--internet-gateway-ids, --filters (internet-gateway-id, attachment.vpc-id, attachment.state, tag:*) |
--max-results, --dry-run |
DONE |
EC2 — Egress-Only Internet Gateway
KV CRUD only — no OVN/OVS integration. EIGWs are stored but have no effect on network topology.
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-egress-only-internet-gateway |
--vpc-id, --tag-specifications |
--client-token, --dry-run |
STARTED (KV only, no OVN) |
delete-egress-only-internet-gateway |
--egress-only-internet-gateway-id |
--dry-run |
STARTED (KV only, no OVN) |
describe-egress-only-internet-gateways |
--egress-only-internet-gateway-ids, --filters (egress-only-internet-gateway-id, tag:*) |
--max-results, --next-token, --dry-run |
STARTED (KV only, no OVN) |
EC2 — Route Tables
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-route-table |
--vpc-id |
--tag-specifications, --dry-run |
DONE |
delete-route-table |
--route-table-id |
--dry-run |
DONE |
describe-route-tables |
--route-table-ids, --filters (vpc-id, route-table-id, association.main, association.route-table-association-id, association.subnet-id, route.destination-cidr-block, route.gateway-id) |
--max-results, --next-token, --dry-run |
DONE |
create-route |
--route-table-id, --destination-cidr-block, --gateway-id, --nat-gateway-id |
--egress-only-internet-gateway-id, --vpc-peering-connection-id, --dry-run |
DONE |
delete-route |
--route-table-id, --destination-cidr-block |
--dry-run |
DONE |
replace-route |
--route-table-id, --destination-cidr-block, --gateway-id |
--nat-gateway-id, --dry-run |
DONE |
associate-route-table |
--route-table-id, --subnet-id |
--gateway-id, --dry-run |
DONE |
disassociate-route-table |
--association-id |
--dry-run |
DONE |
replace-route-table-association |
--association-id, --route-table-id |
--dry-run |
DONE |
EC2 — Network Interfaces (ENIs)
ENIs are auto-created by run-instances --subnet-id and auto-deleted on
termination. Standalone attach/detach API is internal-only.
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-network-interface |
--subnet-id, --private-ip-address, --description, --tag-specifications |
--groups, --dry-run |
DONE |
delete-network-interface |
--network-interface-id |
--dry-run |
DONE |
describe-network-interfaces |
--network-interface-ids, --filters (subnet-id, vpc-id, attachment.instance-id) |
--max-results, --dry-run |
DONE |
modify-network-interface-attribute |
— | --network-interface-id, --description, --groups |
DONE |
attach-network-interface |
— | --network-interface-id, --instance-id, --device-index |
NOT STARTED (internal only) |
detach-network-interface |
— | --attachment-id, --force |
NOT STARTED (internal only) |
assign-private-ip-addresses |
— | --network-interface-id, --private-ip-addresses, --secondary-private-ip-address-count |
NOT STARTED |
unassign-private-ip-addresses |
— | --network-interface-id, --private-ip-addresses |
NOT STARTED |
EC2 — Elastic IP
EIP commands are only registered when an external IPAM pool is configured.
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
allocate-address |
--public-ipv4-pool, --tag-specifications |
--domain, --dry-run |
DONE |
release-address |
--allocation-id |
--dry-run |
DONE |
associate-address |
--allocation-id, --network-interface-id, --instance-id, --private-ip-address |
--dry-run, --allow-reassociation |
DONE |
disassociate-address |
--association-id |
--dry-run |
DONE |
describe-addresses |
--allocation-ids, --public-ips, --filters (allocation-id, public-ip, instance-id, association-id, domain, tag:*) |
--dry-run |
DONE |
describe-addresses-attribute |
--allocation-ids |
--attribute, --dry-run, --max-results, --next-token |
DONE |
EC2 — NAT Gateway
Deleted gateways move to a separate KV bucket with 1-hour TTL for Terraform polling.
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-nat-gateway |
--subnet-id, --allocation-id |
--connectivity-type, --tag-specifications, --dry-run |
DONE |
delete-nat-gateway |
--nat-gateway-id |
--dry-run |
DONE |
describe-nat-gateways |
--nat-gateway-ids, --filters (vpc-id, state) |
--max-results, --next-token, --dry-run |
DONE |
assign-private-nat-gateway-address |
— | --nat-gateway-id, --private-ip-addresses |
NOT STARTED |
associate-nat-gateway-address |
— | --nat-gateway-id, --allocation-ids |
NOT STARTED |
EC2 — Placement Groups
Strategies: spread (1 instance per node, strict) and cluster (all pinned
to single node). partition rejected.
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-placement-group |
--group-name, --strategy (spread/cluster) |
--partition-count, --spread-level, --tag-specifications, --dry-run |
DONE |
delete-placement-group |
--group-name |
--dry-run |
DONE |
describe-placement-groups |
--group-names, --group-ids, --filters (strategy, state, spread-level, group-name) |
--dry-run |
DONE |
EC2 — VPC Peering
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-vpc-peering-connection |
— | --vpc-id, --peer-vpc-id, --peer-owner-id, --peer-region, --tag-specifications |
NOT STARTED |
accept-vpc-peering-connection |
— | --vpc-peering-connection-id, --dry-run |
NOT STARTED |
reject-vpc-peering-connection |
— | --vpc-peering-connection-id, --dry-run |
NOT STARTED |
delete-vpc-peering-connection |
— | --vpc-peering-connection-id, --dry-run |
NOT STARTED |
describe-vpc-peering-connections |
— | --vpc-peering-connection-ids, --filters, --max-results |
NOT STARTED |
modify-vpc-peering-connection-options |
— | --vpc-peering-connection-id, --requester-peering-connection-options, --accepter-peering-connection-options |
NOT STARTED |
EC2 — VPC Endpoints
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-vpc-endpoint |
— | --vpc-id, --service-name, --vpc-endpoint-type, --route-table-ids, --subnet-ids, --tag-specifications |
NOT STARTED |
delete-vpc-endpoints |
— | --vpc-endpoint-ids, --dry-run |
NOT STARTED |
describe-vpc-endpoints |
— | --vpc-endpoint-ids, --filters, --max-results |
NOT STARTED |
describe-vpc-endpoint-services |
— | --service-names, --filters, --max-results |
NOT STARTED |
modify-vpc-endpoint |
— | --vpc-endpoint-id, --add-route-table-ids, --remove-route-table-ids, --add-subnet-ids, --remove-subnet-ids |
NOT STARTED |
EC2 — VPN & Customer Gateway
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-customer-gateway |
— | --type, --bgp-asn, --ip-address, --tag-specifications |
NOT STARTED |
delete-customer-gateway |
— | --customer-gateway-id, --dry-run |
NOT STARTED |
describe-customer-gateways |
— | --customer-gateway-ids, --filters |
NOT STARTED |
create-vpn-gateway |
— | --type, --amazon-side-asn, --tag-specifications |
NOT STARTED |
delete-vpn-gateway |
— | --vpn-gateway-id, --dry-run |
NOT STARTED |
attach-vpn-gateway |
— | --vpn-gateway-id, --vpc-id |
NOT STARTED |
detach-vpn-gateway |
— | --vpn-gateway-id, --vpc-id |
NOT STARTED |
describe-vpn-gateways |
— | --vpn-gateway-ids, --filters |
NOT STARTED |
create-vpn-connection |
— | --type, --customer-gateway-id, --vpn-gateway-id, --options, --tag-specifications |
NOT STARTED |
delete-vpn-connection |
— | --vpn-connection-id, --dry-run |
NOT STARTED |
describe-vpn-connections |
— | --vpn-connection-ids, --filters |
NOT STARTED |
modify-vpn-connection |
— | --vpn-connection-id, --vpn-gateway-id, --customer-gateway-id |
NOT STARTED |
modify-vpn-connection-options |
— | --vpn-connection-id, --local-ipv4-network-cidr, --remote-ipv4-network-cidr |
NOT STARTED |
EC2 — Network ACLs
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-network-acl |
— | --vpc-id, --tag-specifications |
NOT STARTED |
delete-network-acl |
— | --network-acl-id, --dry-run |
NOT STARTED |
describe-network-acls |
— | --network-acl-ids, --filters, --max-results |
NOT STARTED |
create-network-acl-entry |
— | --network-acl-id, --rule-number, --protocol, --rule-action, --cidr-block, --ingress/--egress, --port-range |
NOT STARTED |
delete-network-acl-entry |
— | --network-acl-id, --rule-number, --ingress/--egress |
NOT STARTED |
replace-network-acl-association |
— | --association-id, --network-acl-id |
NOT STARTED |
replace-network-acl-entry |
— | --network-acl-id, --rule-number, --protocol, --rule-action, --cidr-block, --ingress/--egress, --port-range |
NOT STARTED |
EC2 — Prefix Lists
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-managed-prefix-list |
— | --prefix-list-name, --address-family, --max-entries, --entries, --tag-specifications |
NOT STARTED |
delete-managed-prefix-list |
— | --prefix-list-id, --dry-run |
NOT STARTED |
describe-managed-prefix-lists |
— | --prefix-list-ids, --filters, --max-results |
NOT STARTED |
modify-managed-prefix-list |
— | --prefix-list-id, --current-version, --add-entries, --remove-entries, --prefix-list-name |
NOT STARTED |
get-managed-prefix-list-entries |
— | --prefix-list-id, --target-version, --max-results |
NOT STARTED |
get-managed-prefix-list-associations |
— | --prefix-list-id, --max-results |
NOT STARTED |
EC2 — Launch Templates
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-launch-template |
— | --launch-template-name, --launch-template-data, --tag-specifications |
NOT STARTED |
create-launch-template-version |
— | --launch-template-id/--launch-template-name, --launch-template-data, --source-version |
NOT STARTED |
delete-launch-template |
— | --launch-template-id/--launch-template-name, --dry-run |
NOT STARTED |
describe-launch-templates |
— | --launch-template-ids, --launch-template-names, --filters |
NOT STARTED |
describe-launch-template-versions |
— | --launch-template-id/--launch-template-name, --versions, --min-version, --max-version |
NOT STARTED |
EC2 — Dedicated Hosts, IPv4 Pools, DHCP, Capacity Reservations
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
allocate-hosts |
— | --availability-zone, --instance-type, --quantity, --auto-placement, --tag-specifications |
NOT STARTED |
describe-hosts |
— | --host-ids, --filters, --max-results |
NOT STARTED |
release-hosts |
— | --host-ids |
NOT STARTED |
create-public-ipv4-pool |
— | --tag-specifications, --dry-run |
NOT STARTED |
delete-public-ipv4-pool |
— | --pool-id, --dry-run |
NOT STARTED |
describe-public-ipv4-pools |
— | --pool-ids, --filters, --max-results |
NOT STARTED |
create-dhcp-options |
— | --dhcp-configurations, --tag-specifications |
NOT STARTED |
delete-dhcp-options |
— | --dhcp-options-id, --dry-run |
NOT STARTED |
describe-dhcp-options |
— | --dhcp-options-ids, --filters, --max-results |
NOT STARTED |
associate-dhcp-options |
— | --dhcp-options-id, --vpc-id, --dry-run |
NOT STARTED |
create-capacity-reservation |
— | --instance-type, --instance-platform, --availability-zone, --instance-count, --end-date, --end-date-type, --instance-match-criteria, --tag-specifications |
NOT STARTED |
cancel-capacity-reservation |
— | --capacity-reservation-id, --dry-run |
NOT STARTED |
describe-capacity-reservations |
— | --capacity-reservation-ids, --filters, --max-results |
NOT STARTED |
modify-capacity-reservation |
— | --capacity-reservation-id, --instance-count, --end-date, --end-date-type |
NOT STARTED |
EC2 — Misc
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
delete-network-interface-permission |
— | --network-interface-permission-id, --force |
NOT STARTED |
enable-address-transfer |
— | --allocation-id, --transfer-account-id |
NOT STARTED |
disable-address-transfer |
— | --allocation-id |
NOT STARTED |
EBS Direct API
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
start-snapshot |
— | --volume-size, --parent-snapshot-id, --description, --encrypted |
NOT STARTED |
put-snapshot-block |
— | --snapshot-id, --block-index, --block-data, --checksum |
NOT STARTED |
get-snapshot-block |
— | --snapshot-id, --block-index |
NOT STARTED |
complete-snapshot |
— | --snapshot-id, --changed-blocks-count |
NOT STARTED |
list-snapshot-blocks |
— | --snapshot-id, --max-results, --next-token |
NOT STARTED |
list-changed-blocks |
— | --second-snapshot-id, --first-snapshot-id, --max-results |
NOT STARTED |
IAM
All IAM operations are account-scoped. Root user (account 000000000000)
bypasses policy evaluation entirely.
IAM — Users
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-user |
--user-name, --path |
--tags, --permissions-boundary |
DONE |
get-user |
--user-name |
— | DONE |
list-users |
--path-prefix |
--max-items, --marker |
DONE |
delete-user |
--user-name |
— | DONE |
IAM — Access Keys
Max 2 keys per user. Secrets encrypted with AES-256-GCM using master key, returned plaintext only at creation.
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-access-key |
--user-name |
— | DONE |
list-access-keys |
--user-name |
--max-items, --marker |
DONE |
delete-access-key |
--access-key-id, --user-name |
— | DONE |
update-access-key |
--access-key-id, --user-name, --status (Active/Inactive) |
— | DONE |
IAM — Policies
Policy documents require Version: "2012-10-17" and a valid Statement array.
Wildcard action matching supported (ec2:*, ec2:Describe*, *). Evaluation
order: explicit Deny > explicit Allow > implicit Deny.
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-policy |
--policy-name, --policy-document, --path, --description |
--tags |
DONE |
get-policy |
--policy-arn |
— | DONE |
get-policy-version |
--policy-arn, --version-id |
— | DONE |
list-policies |
— | --scope, --only-attached, --path-prefix, --max-items, --marker |
DONE |
delete-policy |
--policy-arn |
— | DONE |
attach-user-policy |
--user-name, --policy-arn |
— | DONE |
detach-user-policy |
--user-name, --policy-arn |
— | DONE |
list-attached-user-policies |
--user-name |
--path-prefix, --max-items, --marker |
DONE |
IAM — Policy Evaluation (internal — not an AWS API)
| Feature | Status |
|---|---|
| Root user bypass | DONE |
| Default deny | DONE |
| Explicit allow | DONE |
| Explicit deny | DONE |
Wildcard matching (*, ec2:*, ec2:Describe*) |
DONE |
| Account-scoped evaluation | DONE |
| EC2 action mapping | DONE |
| IAM action mapping (16 actions) | DONE |
IAM — Roles
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-role |
--role-name, --assume-role-policy-document, --path, --description, --max-session-duration, --tags |
--permissions-boundary |
DONE |
get-role |
--role-name |
— | DONE |
list-roles |
--path-prefix |
--max-items, --marker |
DONE |
delete-role |
--role-name |
— | DONE |
update-role |
--role-name, --description, --max-session-duration |
— | DONE |
update-assume-role-policy |
--role-name, --policy-document |
— | DONE |
attach-role-policy |
--role-name, --policy-arn |
— | DONE |
detach-role-policy |
--role-name, --policy-arn |
— | DONE |
list-attached-role-policies |
--role-name, --path-prefix |
--max-items, --marker |
DONE |
IAM — Instance Profiles
Containers for IAM roles that allow EC2 instances to assume a role. Required
for run-instances --iam-instance-profile. Max 1 role per profile.
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-instance-profile |
--instance-profile-name, --path, --tags |
— | DONE |
get-instance-profile |
--instance-profile-name |
— | DONE |
list-instance-profiles |
--path-prefix |
--max-items, --marker |
DONE |
list-instance-profiles-for-role |
--role-name |
--max-items, --marker |
DONE |
delete-instance-profile |
--instance-profile-name |
— | DONE |
add-role-to-instance-profile |
--instance-profile-name, --role-name |
— | DONE |
remove-role-from-instance-profile |
--instance-profile-name, --role-name |
— | DONE |
IAM — Groups
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-group |
— | --group-name, --path |
NOT STARTED |
get-group |
— | --group-name |
NOT STARTED |
list-groups |
— | --path-prefix, --max-items, --marker |
NOT STARTED |
delete-group |
— | --group-name |
NOT STARTED |
add-user-to-group |
— | --group-name, --user-name |
NOT STARTED |
remove-user-from-group |
— | --group-name, --user-name |
NOT STARTED |
list-groups-for-user |
— | --user-name, --max-items, --marker |
NOT STARTED |
attach-group-policy |
— | --group-name, --policy-arn |
NOT STARTED |
detach-group-policy |
— | --group-name, --policy-arn |
NOT STARTED |
list-attached-group-policies |
— | --group-name, --path-prefix, --max-items, --marker |
NOT STARTED |
STS
| Command | Implemented Flags | Missing Flags (rejected if supplied) | Status |
|---|---|---|---|
get-caller-identity |
— | — | DONE |
assume-role |
--role-arn, --role-session-name, --duration-seconds (900–min(role MaxSessionDuration, 43200)) |
--policy, --policy-arns (→ PackedPolicyTooLarge); --tags, --transitive-tag-keys (→ InvalidParameterValue); --serial-number, --token-code (→ InvalidParameterValue); --external-id, --source-identity (accepted and logged, not enforced — no Condition evaluator in v1) |
DONE |
get-session-token |
— | --duration-seconds, --serial-number, --token-code |
NOT STARTED |
assume-role-with-web-identity |
— | --role-arn, --role-session-name, --web-identity-token, --provider-id, --policy, --policy-arns, --duration-seconds |
NOT STARTED |
assume-role-with-saml |
— | --role-arn, --principal-arn, --saml-assertion, --policy, --policy-arns, --duration-seconds |
NOT STARTED |
get-access-key-info |
— | --access-key-id |
NOT STARTED |
get-federation-token |
— | --name, --policy, --policy-arns, --duration-seconds, --tags |
NOT STARTED |
decode-authorization-message |
— | --encoded-message |
NOT STARTED |
Trust policies stored on roles (AssumeRolePolicyDocument) reject Condition,
NotPrincipal, NotAction, empty-string Action elements, and empty
Principal blocks at write time (MalformedPolicyDocument) — v1 has no
Condition evaluator so accepting them would silently allow.
ELBv2 (Application & Network Load Balancer)
The data plane uses a system-managed LB VM running HAProxy, launched automatically during create-load-balancer. HAProxy config is pushed via NATS on listener/target changes.
ELBv2 — Load Balancers
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-load-balancer |
--name, --subnets, --security-groups, --scheme (internet-facing/internal), --tags, --ip-address-type (ipv4) |
--type (hardcoded application), --customer-owned-ipv4-pool, --dry-run |
DONE |
delete-load-balancer |
--load-balancer-arn |
--dry-run |
DONE |
describe-load-balancers |
--load-balancer-arns, --names |
--page-size, --marker, --dry-run |
DONE |
ELBv2 — Target Groups
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-target-group |
--name, --protocol (HTTP), --port, --vpc-id, --target-type (instance), --health-check-protocol, --health-check-port, --health-check-path, --health-check-interval-seconds, --health-check-timeout-seconds, --healthy-threshold-count, --unhealthy-threshold-count, --matcher, --tags |
--health-check-enabled, --protocol-version, --ip-address-type, --dry-run |
DONE |
delete-target-group |
--target-group-arn |
--dry-run |
DONE |
describe-target-groups |
--target-group-arns, --names, --load-balancer-arn |
--page-size, --marker, --dry-run |
DONE |
ELBv2 — Targets
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
register-targets |
--target-group-arn, --targets (Id, Port) |
--dry-run |
DONE |
deregister-targets |
--target-group-arn, --targets (Id, Port) |
--dry-run |
DONE |
describe-target-health |
--target-group-arn, --targets |
--include |
DONE |
ELBv2 — Listeners
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-listener |
--load-balancer-arn, --default-actions (Type=forward, TargetGroupArn), --protocol (HTTP), --port |
--ssl-policy, --certificates, --alpn-policy, --mutual-authentication, --dry-run |
DONE |
delete-listener |
--listener-arn |
--dry-run |
DONE |
describe-listeners |
--load-balancer-arn, --listener-arns |
--page-size, --marker, --dry-run |
DONE |
ELBv2 — Tags
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
describe-tags |
--resource-arns (loadbalancer, targetgroup, listener) |
— | DONE |
ELBv2 — Not Yet Implemented
| Feature | Priority | Status |
|---|---|---|
| Listener rules (CreateRule, DeleteRule, DescribeRules, ModifyRule) | High | NOT STARTED |
| HTTPS/TLS termination (SSL certificates, policies, ALPN) | High | NOT STARTED |
Modify operations (ModifyLoadBalancerAttributes, ModifyTargetGroup, ModifyTargetGroupAttributes, ModifyListener) |
Medium | NOT STARTED |
| Connection draining (deregistration delay) | Medium | NOT STARTED |
| Stickiness / session affinity | Medium | NOT STARTED |
| Active health checking (API-driven, vs. HAProxy-only today) | Medium | NOT STARTED |
| IP and Lambda target types | Low | NOT STARTED |
| S3 access log delivery | Low | NOT STARTED |
| WAF integration | Low | NOT STARTED |
CloudWatch (Basic Monitoring)
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
put-metric-data |
— | --namespace, --metric-data |
NOT STARTED |
get-metric-statistics |
— | --namespace, --metric-name, --start-time, --end-time, --period, --statistics, --dimensions |
NOT STARTED |
list-metrics |
— | --namespace, --metric-name, --dimensions, --recently-active |
NOT STARTED |
describe-alarms |
— | --alarm-names, --alarm-name-prefix, --state-value, --action-prefix |
NOT STARTED |
put-metric-alarm |
— | --alarm-name, --namespace, --metric-name, --statistic, --period, --evaluation-periods, --threshold, --comparison-operator, --alarm-actions, --dimensions |
NOT STARTED |
delete-alarms |
— | --alarm-names |
NOT STARTED |
Auto Scaling
| Command | Implemented Flags | Missing Flags | Status |
|---|---|---|---|
create-auto-scaling-group |
— | --auto-scaling-group-name, --launch-template, --min-size, --max-size, --desired-capacity, --vpc-zone-identifier, --target-group-arns, --health-check-type, --health-check-grace-period, --tags |
NOT STARTED |
update-auto-scaling-group |
— | --auto-scaling-group-name, --min-size, --max-size, --desired-capacity, --launch-template, --health-check-type, --health-check-grace-period |
NOT STARTED |
delete-auto-scaling-group |
— | --auto-scaling-group-name, --force-delete |
NOT STARTED |
describe-auto-scaling-groups |
— | --auto-scaling-group-names, --filters, --max-records |
NOT STARTED |
set-desired-capacity |
— | --auto-scaling-group-name, --desired-capacity, --honor-cooldown |
NOT STARTED |
describe-auto-scaling-instances |
— | --instance-ids, --max-records |
NOT STARTED |
put-scaling-policy |
— | --auto-scaling-group-name, --policy-name, --policy-type, --target-tracking-configuration, --scaling-adjustment, --cooldown |
NOT STARTED |
delete-scaling-policy |
— | --auto-scaling-group-name, --policy-name |
NOT STARTED |