mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Josh Siviour
c767f2b6ee
* feat: stamp the owning node's AZ on the instance record RunInstance never persists a node's AZ onto the stored instance, so today only the launching node's own ProjectInstance call can produce a correct AZ for a describe. Add VM.AZ, carry it on InstanceStatus since it is placement observed by the running node, and stamp it in WriteRunningSet alongside Status.LastNode from the node's own config.AZ. * docs: trim the AZ field comment to the project's three-line limit * fix: identify Gather responders by node, not by frame count utils.Gather previously counted reply frames toward completeness, so two subscriptions replying from one host (a daemon mid-restart) could satisfy a fan-out while a genuinely configured node stayed silent. Every daemon reply now carries an X-Node-ID header set by the shared response helpers via nats.Msg.RespondMsg, so Gather can attribute each frame to the node that sent it. Gather gains a CollectionMode (CollectServeData preserves today's early-exit behavior; CollectUntilDeadline collects to the full deadline for absence-proof callers) and returns node-identified Frames plus Responders/ SuccessResponders/ErrorResponders sets. Identity mode retains only the first payload seen per node; a disagreeing later frame from the same node flags a conflict but never overwrites the retained data. Node identity uses one namespace across the reply header, config.ClusterConfig.Nodes keys, daemon.Heartbeat.Node, and vm.VM.LastNode, all traced back to Daemon.node - confirmed to already agree, with a new test asserting it at runtime. This is a mechanical caller-side follow-on: every existing utils.Gather caller now reads frame.Data instead of frame directly. * feat: thread configured node set to DescribeInstances GatewayConfig gains NodeIDs, the keys of config.ClusterConfig.Nodes, distinct from the existing ExpectedNodes count. awsgw builds it from cluster config and wires it into GatewayConfig. The DescribeInstances action passes gw.NodeIDs through instead of relying on gw.DiscoverActiveNodes(ctx) at that call site, so completeness can be judged against the nodes that are actually configured rather than the nodes that happened to answer a discovery probe. * fix: judge DescribeInstances completeness by node identity DescribeInstancesChecked now judges completeness on decoded frames against the configured node set (ValidResponders) rather than on Gather's frame/byte counts, which cannot know the describe schema. A node that answers with a nil Reservations list is a valid responder, not an invalid one - the existing merge guard on Reservations != nil governs how frames combine, not whether the node counted as having answered. The explicit-instance-ID 404 path now collects with CollectUntilDeadline so a node that hasn't replied yet cannot be mistaken for a node that has nothing to report; general listing stays on CollectServeData. Any ambiguity - an unidentified frame, a node appearing in both the success and error sets, a node replying twice with disagreeing frames, a Gather cap hit, or an empty/unset node set - suppresses the NotFound rather than risking a false one. * fix: let a describe by instance id stop once every id is found CollectUntilDeadline made every DescribeInstances with explicit ids wait the full 3s fan-out deadline, measured at 4650ms against 1655ms before, on a healthy cluster where all three nodes answer in about a millisecond. Gather gains a Settled predicate, consulted only under CollectUntilDeadline, that ends collection once the replies already answer the question. Absence still pays the whole deadline, and an instance the caller cannot see is filtered out of every frame so it takes the absence path too, which keeps the two NotFound answers indistinguishable in time. * refactor: resolve IMDS instance metadata local-first, drop the fan-out IMDS only ever serves a guest on its own node, so a DescribeInstances fan-out to the whole cluster is unnecessary latency and load. Resolve from the node's own on-disk VM state first, falling back to a direct record-space read for an instance that has moved off this node. Neither path issues a NATS request. * perf: cache the local instance state parse for IMDS lookups IMDS request rate is guest-driven and unbounded, so re-reading and re-parsing the whole on-disk VM state on every request does not scale with VM count on a node. Cache the parsed state and only re-read when the file's mtime or size changes; a stat or parse failure clears the cache and falls back to the record space rather than serving a stale hit. * fix: share ManagedBy visibility rule between DescribeInstances and IMDS Extracts IsInstanceVisibleToCaller so account ownership and the platform-managed hide rule live in one place instead of two independently maintained copies. * refactor: decouple IMDS instance lookup from the daemon package handlers/imds no longer imports daemon: NewIMDSServiceImpl now takes a localStateReader and a recordLoader interface instead of a dataDir string, and vpcd (the composition root for both planes) constructs both adapters, including the stat-guarded local-state cache that used to live inline in handlers/imds. Either adapter may be nil, degrading that source to a miss rather than failing IMDS to start. * test: move the local state reader test to an external package * chore: exempt export_test.go from the external test package check The check's own remedy text tells you to move unexported hooks into an export_test.go, then flags that file for being in-package — which it is by definition, since exporting internals is its whole purpose. * fix: bound how many describes may hold a full fan-out deadline A describe naming an instance the caller cannot see is filtered out of every frame, so it never settles early and holds a gateway goroutine and inbox subscription for the whole 3s deadline. Issuing those is free for the caller, so nothing stopped a burst of them pinning unbounded gateway state. Absence proofs now draw on a fixed budget. Past it, callers get a retryable RequestLimitExceeded rather than being queued, since queueing would let the backlog grow. The decision is taken before any reply is read, so a nonexistent id and another account's id throttle identically and the throttle cannot become the timing oracle the deadline exists to close. * perf: settle describe-by-id on responder coverage, not the full deadline Every describe-by-id was paying the whole 3s fan-out deadline whenever the requested instance was not in a running-instance frame — which includes a stopped instance, a terminated one, an id belonging to another account, and an id that does not exist. Gather gains ResponderGrace: once every ExpectedResponders node has answered, collection closes after a short grace window instead of running to the deadline. Core NATS gives a request only to the subscribers present when it was published, so an overlapping subscription replies alongside the first and one that arrives later never receives the request at all. Waiting past full coverage cannot find a reply that full coverage missed. The window is armed once and never extended, so a node replying repeatedly cannot hold the fan-out open. The full deadline is now paid only when a node stays silent, which is the case it was always meant to cover. The stopped and terminated bucket queries now run alongside the fan-out rather than after it, so their latency overlaps it instead of adding to it. They feed a shared instanceSettler that ends collection as soon as every requested id has been seen, skipping even the grace window. The timing oracle is unchanged. An instance the caller cannot see is filtered out of every source before the reply is sent, so it takes the same path as an id that does not exist: both answer NotFound after full responder coverage, and remain indistinguishable in time. * fix: stop parallel tests racing the terminate backoff seam terminateRetrySleep is a package global, and every test that needed it neutered swapped it and restored the old value from t.Cleanup. Those tests run in parallel, so two cleanups could write the seam at once — a real data race the detector caught once enough parallel tests shared the binary. Installed once in TestMain instead. Nothing in this package wants the real backoff, so there is no value in restoring it per test.
Spinifex Scripts
Node installation, dev-environment lifecycle, guest image builds, and verification tooling for Spinifex.
Install and node setup
| Script | What it does |
|---|---|
setup.sh |
The production binary installer, served from install.mulgadc.com. Installs deps, users, sudoers, files, systemd units, logrotate and udev rules. SETUP_STAGES runs a subset; ISO_BUILD=1 runs it inside the ISO builder's chroot. |
setup-ovn.sh |
Bootstraps a compute node for OVN VPC networking: installs OVN/OVS, creates br-int, configures the WAN bridge (bridged, DHCP, or routed-NAT uplink), sets the chassis identity, and applies overlay sysctls. --management also starts the OVN central services. |
install-node.sh |
Forms a multi-node cluster across hosts that are already running Spinifex: rebuilds the OVN database in clustered form, runs spx admin init on the first host and spx admin join on the rest, then verifies. The scripted form of install-multi-node. Destroys the joining nodes' logical network state and crypto — see its --help. |
dev-install.sh |
Full local dev setup through the production installer: builds from source, packs a tarball, runs setup.sh, initialises the cluster, and starts services under systemd. |
clone-deps.sh |
Clones or updates the viperblock and predastore checkouts alongside spinifex for cross-repo development. |
Dev environment
| Script | What it does |
|---|---|
reset-dev-env.sh |
Single-node reset: stops services, wipes /etc/, /var/lib/, /var/log/ and /run/spinifex, rebuilds, reinstalls via dev-install.sh, and relaunches a smoke-test instance. Preserves the existing node/network topology from spinifex.toml. |
dev-env/*.conf |
Topology profiles for reset-dev-env.sh --profile=NAME, declaring which bridge carries the wan, lan and vpc planes: single-nic (all collapsed onto wan), two-nic (vpc folds onto lan), three-plane (a NIC each). |
Verification and benchmarks
| Script | What it does |
|---|---|
smoke-test.sh |
End-to-end sanity check on a running node: imports an SSH keypair and the Ubuntu AMI, then launches an instance. --create-vpc builds the VPC, gateway, subnet and routes first and tears them down after; --nodes N launches through a spread placement group so each instance lands on a different physical server. TEST_GPU=1 adds GPU passthrough checks. |
verify-network.sh |
Asserts a node's datapath matches its declared planes — EIP reachability, which plane carries Geneve, and which interfaces services bind. Catches what smoke-test.sh cannot, since none of its traffic leaves the box. |
test-ddil-phase1.sh |
systemd-level smoke test for daemon-local-autonomy tier 1. Destructive on the local node: stops and starts spinifex-nats and restarts spinifex-daemon, restoring both via a cleanup trap. |
check-coverage.sh |
Fails the build if total Go coverage falls below the minimum threshold, excluding packages exempt from the requirement. |
diff-coverage.sh |
Coverage check scoped to changed lines only, against an auto-detected base ref (HEAD~1 on main, origin/main on dev, otherwise origin/dev). |
run-bench.sh |
Launches a benchmark against running Ubuntu instances, auto-detecting the AMI for the host architecture. Tracks nbdkit and perf alongside the run. |
disk-performance.sh |
Runs a fio random 70/30 read-write benchmark swept across 4k/16k/128k/1M. Writes JSON results to /tmp/spinifex-disk-bench; override with BENCH_DIR, OUT_DIR, SIZE, JOBS, BLOCK_SIZES. |
network-performance.sh |
iperf throughput from several clients to one server, driven over SSH, writing a summary.txt of Gbit/s per client. --server-ip sets the address clients dial, so pointing it at instance private addresses measures the VPC overlay rather than the external pool. |
workload-performance.sh |
In-guest memory, disk, CPU and network workload test. Installs Go and runs the Badger test suite with jemalloc disabled and reduced parallelism. |
Guest images
| Script | What it does |
|---|---|
build-system-image.sh |
Builds a minimal system AMI (Alpine or Ubuntu) from a manifest. All customisation happens inside the libguestfs appliance, so the build touches no host block device, no host mount, and needs no sudo. --import also registers it as an AMI. |
publish-system-image.sh |
Converts a built raw image to compressed qcow2, writes the .sha256 sidecar the catalog verifier expects, and uploads both to the R2 bucket behind iso.mulgadc.com/system-ami/. |
build-microvm-image.sh |
Builds the minimal Alpine kernel + initramfs used by the QEMU microvm machine type and the lb-agent, carrying haproxy (ALB) and nginx-stream (NLB) data planes. |
mkosi-build.sh |
Runs an mkosi system-image build inside the pinned builder container, so a runner needs only Docker and the toolchain cannot drift. --image names an output and expands to the right profile list. |
mkosi-publish.sh |
Publishes an mkosi-built image to R2 from inside the same pinned container — qcow2 conversion, .sha256 sidecar, and upload. |
mkosi-common.sh |
Shared plumbing sourced by both mkosi scripts. Builds the toolchain image once, with the builder account matched to the invoking user so bind-mounted output is not written back as root. |
mkosi-builder.Dockerfile |
The pinned image-build toolchain. Debian-hosted regardless of target distro, since mkosi bootstraps the target from its own archive. |
images/ — per-image build inputs
Each subdirectory holds a manifest.conf, init scripts, and a setup.sh that build-system-image.sh runs inside the libguestfs appliance after packages and files are placed. *_test.sh files are shell unit tests for the helper of the same name.
| Path | What it builds |
|---|---|
ecs-agent/ |
The Alpine spinifex-ecs-node container-instance AMI (containerd + ecs-agent). setup.sh sets OpenRC exec bits, wires the CNI bin path, and applies the serial-console tweaks. |
ecs-agent-gpu/ |
The Ubuntu ECS GPU container-instance AMI. Same CNI and containerd wiring adjusted for Ubuntu's package layout, plus the headless NVIDIA driver and nvidia-container-toolkit. |
eks-node/ |
The unified Alpine EKS node AMI (K3s server or agent, selected at first boot). setup.sh installs the pinned, SHA-verified K3s binary and the role init scripts. |
eks-node-gpu/ |
The Ubuntu EKS GPU worker AMI. Agent role only — GPU nodes never run the control plane — plus the NVIDIA GPU stack. |
rds-postgres/ |
The spinifex-rds-postgres AMI. setup.sh sets exec bits on the OpenRC services and creates the root-only directory cloud-init drops the agent env file and gateway CA into. |
rds-mariadb/ |
The spinifex-rds-mariadb AMI. Same layout as rds-postgres/, plus the /etc/my.cnf.d include drop-in and the conf.d/mariadb dependency that keeps the packaged service off an unbootstrapped datadir. |
ubuntu-gpu-{amd,nvidia}-setup.sh |
Chroot setup for GPU-capable Ubuntu guest images: ROCm CLI tools and AMD firmware, or the DKMS-built headless NVIDIA driver. Guest-side only — host VFIO setup is spx admin gpu setup. |
images/eks-node/ — in-guest helpers
Installed into the EKS node AMI and run at boot or on a timer. All host communication goes through the AWS gateway over SigV4; the VM never speaks NATS directly.
| Script | What it does |
|---|---|
eks-node-role.sh |
First-boot role selector. Reads SPINIFEX_K3S_ROLE from the cloud-init env file and enables the matching services — nothing is enabled at bake time. |
k3s-prestart.sh |
Server prestart: refuses to start on a restore-block marker, derives config.yaml from its skeleton, stages the konnectivity-agent and CoreDNS manifests, and waits for the token-webhook kubeconfig. |
k3s-agent-prestart.sh |
Agent prestart shared by the OpenRC and systemd units. Requires K3S_URL and K3S_TOKEN, then resolves the kubelet provider-id from IMDSv2 for EKS parity. |
k3s-first-boot.sh |
Runs once after the server is healthy: reads the node-token and admin kubeconfig, rewrites the kubeconfig server address to the cluster NLB endpoint, and publishes both to the host. |
konnectivity-server-prestart.sh |
Waits for the K3s CA and admin kubeconfig, mints the konnectivity serving cert, and writes the apiserver replica count for the caller to source. |
mulga-eks-addon-sync.sh |
Pulls staged managed-addon descriptors through the gateway, renders each bundle into the K3s auto-deploy dir, and reports per-addon delivery status back. |
mulga-eks-etcd-snapshot.sh |
Snapshots K3s embedded etcd to predastore. Installed into two crond periodic dirs and picks its tier — frequent or daily — from the dir it was invoked from. |
mulga-eks-k3s-recovery.sh |
Applies a control-plane etcd-quorum recovery directive before k3s starts. Pull-based and a no-op unless the reconciler escalated a wedged control plane. |
mulga-eks-state-report.sh |
Publishes server health (healthz, node count) to the cluster reconciler. The apiserver binds the VPC node-ip, which the host daemon cannot probe directly. |
mulga-eks-provider-id.sh |
Writes the K3s config drop-in supplying the metadata the EBS CSI driver needs from IMDS: provider-id plus the region, zone and instance-type labels. |
mulga-ebs-byid.sh |
Mints the EBS-style /dev/disk/by-id symlink the upstream EBS CSI node plugin resolves, from the virtio-blk serial. Alpine's busybox mdev has no eudev to do it. |
mulga-mgmt-net.sh |
Boot oneshot bringing a multi-NIC system VM's interfaces up by MAC from the QEMU fw_cfg netcfg blob, before cloud-init's network stage. cloud-init cannot reliably pick the right NIC out of two. |
mulga-vpc-mtu.sh |
Boot oneshot pinning the VPC data NIC MTU so flannel sizes its VXLAN overlay to fit the OVN geneve underlay. Otherwise large frames are silently dropped while handshakes pass. |
Launch helpers
| Script | What it does |
|---|---|
launch-gpu.sh |
Launches a GPU instance from the AMD or NVIDIA GPU AMI, selecting the AMI from the instance type's GPU brand. --amd / --nvidia override the detection. |
ecs-node-bringup.sh |
Launches an ECS container-instance VM. Renders the cloud-init user-data seeding the agent's control-plane config, IAM credentials and gateway CA — there is no ECS-specific launch API, by design. |
demo/ — GPU demo
| Script | What it does |
|---|---|
deploy.sh |
Deploys and starts the four-instance GPU demo (dashboard, YOLO, Ollama, qwen3-vl) from the *_IP and VIDEO_PATH env vars. |
dashboard_server.py |
Serves the demo web UI and proxies the streams from the GPU instances. |
yolo_stream.py |
Runs YOLO detection over a video file, serving an MJPEG stream of annotated frames and an SSE stream of qwen3-vl scene descriptions. |