Ben Duncan
225bf65a8e
Merges `feat/ebs-export-tunables` (83 commits) into dev. The branch could not be rebase-merged because it carries five merges from dev, so it lands as a squash. ## What is in it - **EBS export tunables** and the nbdkit/viperblock export path they configure. - **Teardown reapers and backoff** for S3 buckets, load balancers, gateways, spot requests, certificates, endpoints, elastic IPs and network interfaces, so a dependent that keeps failing is backed off rather than retried hot. - **Retry and tolerance paths**: ECS IAM service init, ECR signing keys deleted between the list and the read, an unreadable signing key at awsgw startup, and an ECS instance-profile policy attach converging on a stale read. - **Key pair validation reads the material by name** instead of listing the account, closing the `InvalidKeyPair.NotFound` race that a single-replica LIST could produce. - **Concurrency and fan-out**: concurrent metadata document fetches, bounded metadata reads for describes, and the node-discovery timeout removed from every fan-out call. - **OVN correctness against the configured NB cluster**: the flows barrier, nexthop MAC binding, and stopping a compute node from running the package's standalone database. - **nats-js-watchdog can re-arm itself.** The escalation check ran above the probe loop while the only line that cleared the counter sat below it, so a latched watchdog could never observe a recovery. It now probes first and decides second, and says so once per latch rather than once per tick. - **CI**: 19 commits of nightly and build-pipeline work. ## Verification `make preflight` green locally. Build and Test, Integration Tests, Race Detection, and Lint and Security all green on `c2da0385`. ## Known red, not introduced here Two nightly cells fail on this branch and are pre-existing: - **cell-10** — a partial OVN IPsec enable leaves `NB_Global ipsec=true` asserted cluster-wide while two of three chassis failed their local reconcile, black-holing cross-chassis guest traffic for roughly 5.5 minutes while every health signal stays green. - **cell-19** — a total layer-3 egress blackhole below a correctly and promptly programmed NAT/EIP datapath, aggravated by a natuplink probe whose retry budget outlives the test's own wait, so it can never emit its FAIL verdict. Both are filed separately and should be weighed before a release is cut.
4.5 KiB
title, description, category, tags, resources
| title | description | category | tags | resources | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Spinifex Admin CLI | Complete reference for the Spinifex administration CLI. Manage accounts, nodes, VMs, and services. | Admin |
|
|
Spinifex Admin CLI
Complete reference for the Spinifex administration CLI.
Table of Contents
Overview
The spx binary is the central administration tool for managing your Spinifex infrastructure. It provides commands for cluster initialization, account management, node operations, VM lifecycle, and service control. All services in the Spinifex platform are managed through this single binary.
Binary location: /usr/local/bin/spx
Instructions
Account Management
Create a new isolated account. This provisions a sequential 12-digit account ID, an admin user with an AdministratorAccess policy attached, and an access key pair. The credentials are written to ~/.aws/credentials and ~/.aws/config under a spinifex-<name> profile automatically.
spx admin account create --name myteam
Account created successfully!
Account ID: 000000000002
Account Name: myteam
Admin User: admin
Access Key ID: AKIA1A2B3C4D5E6F7890ABCD
Secret Access Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
AWS Profile: spinifex-myteam
Use with:
AWS_PROFILE=spinifex-myteam aws ec2 describe-instances
The secret access key is only shown once. It is saved to
~/.aws/credentialson the node where you ran the command; copy it from there if you need it elsewhere.
Set the profile to start using the account:
export AWS_PROFILE=spinifex-myteam
To create additional users and scoped permissions within the account, see IAM Users and Policies.
List all accounts:
spx admin account list
ACCOUNT ID NAME STATUS CREATED
---------- ---- ------ -------
000000000000 system ACTIVE 2026-07-03 02:17
000000000001 spinifex ACTIVE 2026-07-03 02:17
000000000002 myteam ACTIVE 2026-07-03 03:32
Deleting an account, and creating or removing accounts from your own signup or provisioning system over the private admin API, are covered in Account Management.
Node Management
List nodes in the cluster:
spx get nodes
NAME STATUS IP REGION AZ UPTIME VMs
node1 Ready 127.0.0.1 ap-southeast-2 ap-southeast-2a 2m 0
node2 Ready 127.0.0.2 ap-southeast-2 ap-southeast-2a 2m 0
node3 Ready 127.0.0.3 ap-southeast-2 ap-southeast-2a 2m 0
Monitor Resources
spx top nodes
Prints per-node CPU/memory usage and cluster-wide instance type availability.
Image Management
spx admin images list
spx admin images import --name debian-13-arm64
Catalog imports verify the image against the catalog-declared SHA-256/SHA-512 digest before extraction. Use --file to import operator-supplied media (verification skipped — operator is responsible for integrity), or --force to re-download after a checksum mismatch.
EKS node image
To run EKS, import the prebuilt node image from the catalog:
spx admin images import --name spinifex-eks-node
This pulls the Alpine + K3s node AMI from iso.mulgadc.com, verifies its checksum, and registers it tagged spinifex:managed-by=eks. eks create-cluster and eks create-nodegroup resolve the boot AMI by that tag, so no further configuration is needed.
Cluster Shutdown
Coordinated, phased shutdown of the entire cluster — running VMs are stopped before storage and control-plane services:
spx admin cluster shutdown
Troubleshooting
Permission Denied Running Spinifex
The binary may not be executable. Fix permissions:
chmod +x /usr/local/bin/spx
If you get permission errors during operations, ensure you're running with appropriate privileges. Some OVN and networking commands require sudo.
Services Fail to Start
Check the daemon logs for specific errors via systemctl/journalctl:
systemctl status 'spinifex-*'
journalctl -u spinifex-daemon -f
journalctl -u 'spinifex-*' -f
Common causes include port conflicts, missing OVN configuration, or untrusted CA certificates.