mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 19:01:42 +00:00
Ben Duncan
225bf65a8e
Merges `feat/ebs-export-tunables` (83 commits) into dev. The branch could not be rebase-merged because it carries five merges from dev, so it lands as a squash. ## What is in it - **EBS export tunables** and the nbdkit/viperblock export path they configure. - **Teardown reapers and backoff** for S3 buckets, load balancers, gateways, spot requests, certificates, endpoints, elastic IPs and network interfaces, so a dependent that keeps failing is backed off rather than retried hot. - **Retry and tolerance paths**: ECS IAM service init, ECR signing keys deleted between the list and the read, an unreadable signing key at awsgw startup, and an ECS instance-profile policy attach converging on a stale read. - **Key pair validation reads the material by name** instead of listing the account, closing the `InvalidKeyPair.NotFound` race that a single-replica LIST could produce. - **Concurrency and fan-out**: concurrent metadata document fetches, bounded metadata reads for describes, and the node-discovery timeout removed from every fan-out call. - **OVN correctness against the configured NB cluster**: the flows barrier, nexthop MAC binding, and stopping a compute node from running the package's standalone database. - **nats-js-watchdog can re-arm itself.** The escalation check ran above the probe loop while the only line that cleared the counter sat below it, so a latched watchdog could never observe a recovery. It now probes first and decides second, and says so once per latch rather than once per tick. - **CI**: 19 commits of nightly and build-pipeline work. ## Verification `make preflight` green locally. Build and Test, Integration Tests, Race Detection, and Lint and Security all green on `c2da0385`. ## Known red, not introduced here Two nightly cells fail on this branch and are pre-existing: - **cell-10** — a partial OVN IPsec enable leaves `NB_Global ipsec=true` asserted cluster-wide while two of three chassis failed their local reconcile, black-holing cross-chassis guest traffic for roughly 5.5 minutes while every health signal stays green. - **cell-19** — a total layer-3 egress blackhole below a correctly and promptly programmed NAT/EIP datapath, aggravated by a natuplink probe whose retry budget outlives the test's own wait, so it can never emit its FAIL verdict. Both are filed separately and should be weighed before a release is cut.
154 lines
4.5 KiB
Markdown
154 lines
4.5 KiB
Markdown
---
|
|
title: "Spinifex Admin CLI"
|
|
description: "Complete reference for the Spinifex administration CLI. Manage accounts, nodes, VMs, and services."
|
|
category: "Admin"
|
|
tags:
|
|
- cli
|
|
- admin
|
|
- reference
|
|
resources:
|
|
- title: "Spinifex Repository"
|
|
url: "https://github.com/mulgadc/spinifex"
|
|
---
|
|
|
|
# Spinifex Admin CLI
|
|
|
|
> Complete reference for the Spinifex administration CLI.
|
|
|
|
## Table of Contents
|
|
|
|
- [Overview](#overview)
|
|
- [Instructions](#instructions)
|
|
- [Troubleshooting](#troubleshooting)
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
The `spx` binary is the central administration tool for managing your Spinifex infrastructure. It provides commands for cluster initialization, account management, node operations, VM lifecycle, and service control. All services in the Spinifex platform are managed through this single binary.
|
|
|
|
**Binary location:** `/usr/local/bin/spx`
|
|
|
|
## Instructions
|
|
|
|
## Account Management
|
|
|
|
Create a new isolated account. This provisions a sequential 12-digit account ID, an `admin` user with an `AdministratorAccess` policy attached, and an access key pair. The credentials are written to `~/.aws/credentials` and `~/.aws/config` under a `spinifex-<name>` profile automatically.
|
|
|
|
```bash
|
|
spx admin account create --name myteam
|
|
```
|
|
|
|
```
|
|
Account created successfully!
|
|
Account ID: 000000000002
|
|
Account Name: myteam
|
|
Admin User: admin
|
|
Access Key ID: AKIA1A2B3C4D5E6F7890ABCD
|
|
Secret Access Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
|
|
AWS Profile: spinifex-myteam
|
|
|
|
Use with:
|
|
AWS_PROFILE=spinifex-myteam aws ec2 describe-instances
|
|
```
|
|
|
|
> **The secret access key is only shown once.** It is saved to `~/.aws/credentials` on the node where you ran the command; copy it from there if you need it elsewhere.
|
|
|
|
Set the profile to start using the account:
|
|
|
|
```bash
|
|
export AWS_PROFILE=spinifex-myteam
|
|
```
|
|
|
|
To create additional users and scoped permissions within the account, see [IAM Users and Policies](/docs/iam-users-and-policies).
|
|
|
|
List all accounts:
|
|
|
|
```bash
|
|
spx admin account list
|
|
```
|
|
|
|
```
|
|
ACCOUNT ID NAME STATUS CREATED
|
|
---------- ---- ------ -------
|
|
000000000000 system ACTIVE 2026-07-03 02:17
|
|
000000000001 spinifex ACTIVE 2026-07-03 02:17
|
|
000000000002 myteam ACTIVE 2026-07-03 03:32
|
|
```
|
|
|
|
Deleting an account, and creating or removing accounts from your own signup or provisioning system over the private admin API, are covered in [Account Management](/docs/admin/account-management).
|
|
|
|
## Node Management
|
|
|
|
List nodes in the cluster:
|
|
|
|
```bash
|
|
spx get nodes
|
|
```
|
|
|
|
```
|
|
NAME STATUS IP REGION AZ UPTIME VMs
|
|
node1 Ready 127.0.0.1 ap-southeast-2 ap-southeast-2a 2m 0
|
|
node2 Ready 127.0.0.2 ap-southeast-2 ap-southeast-2a 2m 0
|
|
node3 Ready 127.0.0.3 ap-southeast-2 ap-southeast-2a 2m 0
|
|
```
|
|
|
|
## Monitor Resources
|
|
|
|
```bash
|
|
spx top nodes
|
|
```
|
|
|
|
Prints per-node CPU/memory usage and cluster-wide instance type availability.
|
|
|
|
## Image Management
|
|
|
|
```bash
|
|
spx admin images list
|
|
spx admin images import --name debian-13-arm64
|
|
```
|
|
|
|
Catalog imports verify the image against the catalog-declared SHA-256/SHA-512 digest before extraction. Use `--file` to import operator-supplied media (verification skipped — operator is responsible for integrity), or `--force` to re-download after a checksum mismatch.
|
|
|
|
### EKS node image
|
|
|
|
To run EKS, import the prebuilt node image from the catalog:
|
|
|
|
```bash
|
|
spx admin images import --name spinifex-eks-node
|
|
```
|
|
|
|
This pulls the Alpine + K3s node AMI from `iso.mulgadc.com`, verifies its checksum, and registers it tagged `spinifex:managed-by=eks`. `eks create-cluster` and `eks create-nodegroup` resolve the boot AMI by that tag, so no further configuration is needed.
|
|
|
|
## Cluster Shutdown
|
|
|
|
Coordinated, phased shutdown of the entire cluster — running VMs are stopped before storage and control-plane services:
|
|
|
|
```bash
|
|
spx admin cluster shutdown
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Permission Denied Running Spinifex
|
|
|
|
The binary may not be executable. Fix permissions:
|
|
|
|
```bash
|
|
chmod +x /usr/local/bin/spx
|
|
```
|
|
|
|
If you get permission errors during operations, ensure you're running with appropriate privileges. Some OVN and networking commands require `sudo`.
|
|
|
|
### Services Fail to Start
|
|
|
|
Check the daemon logs for specific errors via `systemctl`/`journalctl`:
|
|
|
|
```bash
|
|
systemctl status 'spinifex-*'
|
|
journalctl -u spinifex-daemon -f
|
|
journalctl -u 'spinifex-*' -f
|
|
```
|
|
|
|
Common causes include port conflicts, missing OVN configuration, or untrusted CA certificates.
|