gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-15 17:11:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
ae6fcfe7a1a14fff8c58192ea69be8fe4b888de4
spinifex/Makefile
T
Josh Siviour ae6fcfe7a1 Ansible dev extensions (#254) 
* chore(ansible): add dev-status playbook (read-only health doctor)

New `dev-status` role + playbook + Makefile target. Read-only snapshot of
spinifex.target, failed units, port listeners, OVN/OVS state, gateway
round-trip, AMI/keypair/volume/instance counts, and build-vs-install drift
for spx + nbdkit plugin + microvm artifacts. Pairs with dev-preflight as
the post-install / running-system diagnostic.

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.1.

* chore(ansible): add dev-logs playbook (bug-report bundle)

New `logs` role + playbook + Makefile target. Captures journalctl for
spinifex-* / ovn-* / openvswitch-switch / qemu / nbdkit, copies
/var/log/spinifex/, snapshots systemctl status + networking (ip/ss/ovs/
ovn show) + spx/plugin/microvm version manifest + dmesg, then bundles
into /tmp/spinifex-logs-<host>-<ts>.tar.gz owned by the invoking user.

Tail mode deliberately omitted — devs should `journalctl -u 'spinifex-*' -f`
directly; ansible captures stdout and is the wrong tool for follow.

Overrides: -e logs_since=10min, -e logs_output_dir=/tmp,
-e logs_include_dmesg=false.

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.2.

* chore(ansible): add dev-snapshot / dev-restore (state checkpoint)

New snapshot + restore roles + playbooks + Makefile targets.

dev-snapshot quiesces the full spinifex+OVN+OVS stack in dependency
order (spinifex.target → ovn-controller → ovn-central →
openvswitch-switch), kills lingering QEMU VMs, then tars /etc/spinifex,
/var/lib/spinifex (viperblock volumes included), /var/lib/ovn (NB+SB
DBs), and /etc/openvswitch/conf.db into /var/lib/spinifex-snapshots/
<name>.tar.gz with a sibling .manifest, restarts services in reverse,
and waits for the AWS gateway to come back.

dev-restore inverses: stop services, wipe the same path set, untar the
named bundle, copy /etc/spinifex/ca.pem back into the system trust
store and rebuild the CA bundle (the snapshot doesn't include
/usr/local/share/ca-certificates/spinifex-ca.crt), daemon-reload,
restart in reverse order.

Both refuse to run without -e snapshot_name=<name>, validate the name
matches [A-Za-z0-9._-]+, and produce honest summaries when the gateway
fails to come back (does not mask other ongoing issues).

Tested on dev box: snapshot/restore of 2.4 GiB state (large viperblock
volumes) completes cleanly with no data drift; services all return to
active.

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.3.

* fix(ansible): wrap aws/spx calls in sg spinifex to fix group inheritance

Adds scripts/ansible/scripts/aws-as-spinifex.sh and spx-as-spinifex.sh
helpers that re-enter the command under `sg spinifex -c '...'` with
AWS_PROFILE=spinifex preset. Swaps every AWS / spx admin task across
the status, init, deploy, smoketest, snapshot, and restore roles to
use these via spinifex_aws_cli / spinifex_spx_cli vars.

Root cause (mulga-siv-92): ansible's `become_user` is a no-op when the
target user equals the calling user, so sudo never runs and
supplementary groups are inherited from the operator's invoking shell
rather than re-initialised via initgroups(3). On shells that predate
setup.sh's spinifex group add, the spinifex group is missing and
/etc/spinifex/ca.pem (drwxr-x--- root:spinifex, the AWS CLI's
ca_bundle) is unreadable — aws cli fails with SSL Errno 13. The
helpers re-enter the command in a shell that explicitly assumes the
spinifex group via sg(1), bypassing the issue regardless of how
ansible-playbook is invoked.

Side benefit: drops 50 lines of repeated `become: true / become_user /
environment: AWS_PROFILE` boilerplate across the roles.

Closes mulga-siv-92.

* chore(ansible): add dev-version playbook (build vs install drift)

New `version` role + playbook + Makefile target. Compares sha256 of
built artifacts (bin/spx, viperblock/lib/nbdkit-viperblock-plugin.so,
build/microvm/{vmlinuz,initramfs.cpio.gz}) against their installed
counterparts (/usr/local/bin/spx, plugindir, /usr/share/spinifex/
microvm/*). Prints per-artifact drift verdict plus sub-repo HEAD
short-revs (spinifex/viperblock/predastore) for context. Closes with
either "DRIFT detected — run: make ansible-dev-deploy" or "No drift".

Pure read-only. Faster than dev-status (no AWS/OVN/port probing).

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.9.

* chore(ansible): add missing roles/deploy/meta/main.yml

All other roles have meta/main.yml; deploy was the lone outlier.
Consistency fix. Closes mulga-siv-86.10.

* chore(ansible): add dev-vpc playbook (idempotent dev VPC fixture)

New `vpc` role + playbook + Makefile target. Idempotent
present/absent dispatch creates a deterministic spinifex-dev-vpc with
public subnet (10.99.1.0/24, map-public-ip-on-launch), internet
gateway, route table, and default route. Hooks into smoketest's launch
path — smoketest's existing
Name=map-public-ip-on-launch,Values=true filter now picks it up
instead of falling through to the "no subnet → skip launch" branch.

Idempotency: each step describe-then-create, looking up by Name tag
where possible. Re-running on a converged VPC is changed=0, skipped=5
(the create tasks).

Workaround for mulga-siv-94 (gateway silently drops
--tag-specifications and no-ops create-tags for route tables): role
falls back to "RT associated with our subnet" lookup so tag absence
doesn't break idempotency. create-tags task is kept as a no-op-now
forward-looking step for when the gateway gap is closed.

Teardown (-e dev_vpc_state=absent) enumerates ALL non-main route
tables in the VPC (not just the tagged one), so orphans from prior
runs that landed before the fallback was added get cleaned up too.

Tested end-to-end against josh-banksia dev gateway: create / re-run
(idempotent) / teardown all converge cleanly.

Plan: docs/development/improvements/ansible-dev-extensions.md (mulga).
Bead: mulga-siv-86.4.
2026-05-26 16:14:11 +10:00

326 lines
12 KiB
Makefile
Raw Blame History

GO_PROJECT_NAME := spx
SHELL := /bin/bash
# Detect architecture for cross-platform support
ARCH := $(shell uname -m)
ifeq ($(ARCH),x86_64)
GO_ARCH := amd64
AWS_ARCH := x86_64
# On x86, we can run ARM VMs via emulation
QEMU_PACKAGES := qemu-system-x86 qemu-system-arm
else ifeq ($(ARCH),aarch64)
GO_ARCH := arm64
AWS_ARCH := aarch64
# On ARM, we can run x86 VMs via emulation
QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
else ifeq ($(ARCH),arm64)
GO_ARCH := arm64
AWS_ARCH := aarch64
QEMU_PACKAGES := qemu-system-aarch64 qemu-system-x86
else
$(error Unsupported architecture: $(ARCH). Only x86_64 and aarch64/arm64 are supported.)
endif
# Ask Go whether workspace mode is active
IN_WORKSPACE := $(shell go env GOWORK)
# Use -mod=mod unless Go reports an active workspace path
ifeq ($(IN_WORKSPACE),)
GO_BUILD_MOD := -mod=mod
else ifeq ($(IN_WORKSPACE),off)
GO_BUILD_MOD := -mod=mod
else
GO_BUILD_MOD :=
endif
# Quiet-mode filters (active when QUIET=1, set by preflight via recursive make)
# Note: grep pipelines use PIPESTATUS[0] so the exit status of `go test`
# propagates through the filter — otherwise a test failure is swallowed by
# grep's own (success) exit code and preflight prints "passed" on red.
ifdef QUIET
_Q = @
_COVQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' | grep -v 'coverage: 0\.0%' || true; }; exit $${PIPESTATUS[0]}
_RACEQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' || true; }; exit $${PIPESTATUS[0]}
_SECQ = >
else
_Q =
_COVQ =
_RACEQ =
_SECQ = 2>&1 | tee
endif
build: go_build build-installer build-lb-agent
# Build spinifex-ui frontend (requires pnpm)
build-ui:
@echo -e "\n....Building spinifex-ui frontend...."
cd spinifex/services/spinifexui/frontend && pnpm build
# GO commands
VERSION ?= $(shell git describe --tags --always --dirty)
COMMIT ?= $(shell git rev-parse --short HEAD)
LDFLAGS := -s -w -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=$(VERSION) -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Commit=$(COMMIT)
go_build:
@echo -e "\n....Building $(GO_PROJECT_NAME)"
GOFIPS140=v1.0.0 go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go
build-installer:
@echo -e "\n....Building spinifex-installer"
GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/spinifex-installer cmd/installer/main.go
build-lb-agent:
@echo -e "\n....Building lb-agent (static)"
CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
build-system-image: ## Build a system image from manifest (use IMAGE=lb)
ifndef IMAGE
$(error IMAGE is required. Usage: make build-system-image IMAGE=lb)
endif
./scripts/build-system-image.sh scripts/images/$(IMAGE).conf
MICROVM_OUT_DIR := build/microvm
MICROVM_ARTIFACTS := $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz
MICROVM_INPUTS := scripts/build-microvm-image.sh $(MICROVM_OUT_DIR)/init.sh $(MICROVM_OUT_DIR)/inittab bin/lb-agent
# Grouped target — script writes both files in one run.
$(MICROVM_ARTIFACTS) &: $(MICROVM_INPUTS)
./scripts/build-microvm-image.sh
# Only triggers when bin/lb-agent is missing; preserves the artifact's mtime so
# build-microvm-image stays correctly stale-aware.
bin/lb-agent:
$(MAKE) build-lb-agent
build-microvm-image: $(MICROVM_ARTIFACTS) ## Build microVM kernel + initramfs (incremental — skips when up to date)
install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share/spinifex/microvm/
sudo install -d /usr/share/spinifex/microvm
sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
go_run:
@echo -e "\n....Running $(GO_PROJECT_NAME)...."
$(GOPATH)/bin/$(GO_PROJECT_NAME)
# Preflight — runs the same checks as GitHub Actions (lint + vuln + tests).
# Use this before committing to catch CI failures locally.
preflight:
@$(MAKE) --no-print-directory QUIET=1 manifest-check lint govulncheck test-cover diff-coverage test-race test-harness
@echo -e "\n ✅ Preflight passed — safe to commit."
# E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
# default `go test ./spinifex/...`. Runs with mocked AWS clients — no
# infrastructure required, safe to run in CI without a cluster.
test-harness:
@echo -e "\n....Running e2e harness unit tests...."
$(_Q)LOG_IGNORE=1 go test -tags=e2e -timeout 60s ./tests/e2e/harness/... $(_RACEQ)
# Validate docs/service-interfaces.yaml. Schema check + cross-reference
# of services/suites/fixtures + on-disk path existence. Subject content
# vs source is enforced separately in Bead 5 drift lint.
manifest-check:
@echo -e "\n....Checking service-interfaces.yaml...."
@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml
# Run unit tests
test:
@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
LOG_IGNORE=1 go test -timeout 120s ./spinifex/...
# Run unit tests with coverage profile
COVERPROFILE ?= coverage.out
test-cover:
@echo -e "\n....Running tests with coverage for $(GO_PROJECT_NAME)...."
$(_Q)LOG_IGNORE=1 go test -timeout 120s -coverprofile=$(COVERPROFILE) -covermode=atomic ./spinifex/... $(_COVQ)
@scripts/check-coverage.sh $(COVERPROFILE) $(QUIET)
# Run unit tests with race detector
test-race:
@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
$(_Q)LOG_IGNORE=1 go test -race -timeout 300s ./spinifex/... $(_RACEQ)
# Unit tests for in-repo GitHub Actions (e.g. .github/actions/e2e-analyze).
# Kept out of `test-cover` so coverage % isn't diluted by CI-only tooling.
test-actions:
@echo -e "\n....Running action tests...."
LOG_IGNORE=1 go test -timeout 60s ./.github/actions/...
# Check that new/changed code meets coverage threshold (runs tests first)
diff-coverage: test-cover
@QUIET=$(QUIET) scripts/diff-coverage.sh $(COVERPROFILE)
bench:
@echo -e "\n....Running benchmarks for $(GO_PROJECT_NAME)...."
$(MAKE) easyjson
LOG_IGNORE=1 go test -benchmem -run=. -bench=. ./...
run:
$(MAKE) go_build
$(MAKE) go_run
# Fast iteration: build + install binary + restart all services.
# Microvm artifacts are reinstalled when they already exist on disk — the rule's
# input timestamps drive a rebuild only if anything actually changed. On a fresh
# checkout (no build/microvm/vmlinuz yet) the install step is skipped; run
# `make install-microvm` explicitly the first time.
deploy: build
sudo install -m 755 bin/spx /usr/local/bin/spx
@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ]; then \
$(MAKE) install-microvm; \
else \
echo "[deploy] microvm artifacts absent — run 'make install-microvm' for first-time setup"; \
fi
sudo systemctl daemon-reload
sudo systemctl restart spinifex.target
# Re-run setup.sh after changing systemd units, helper scripts, or logrotate config.
# Not needed for code-only changes — use deploy for those.
reinstall:
scripts/dev-install.sh
clean:
rm -f ./bin/$(GO_PROJECT_NAME)
rm -rf spinifex/services/spinifexui/frontend/dist
install-system:
@echo -e "\n....Installing system dependencies for $(ARCH)...."
@echo "QEMU packages: $(QEMU_PACKAGES)"
sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
-o Dpkg::Options::="--force-confdef" \
-o Dpkg::Options::="--force-confold" \
nbdkit nbdkit-plugin-dev pkg-config $(QEMU_PACKAGES) qemu-utils qemu-kvm \
ovmf qemu-efi-aarch64 \
libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
ovn-central ovn-host openvswitch-switch dhcpcd-base
install-go:
@echo -e "\n....Installing Go 1.26.3 for $(ARCH) ($(GO_ARCH))...."
@if [ ! -d "/usr/local/go" ]; then \
curl -L https://go.dev/dl/go1.26.3.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
else \
echo "Go already installed in /usr/local/go"; \
fi
@echo "Go version: $$(go version)"
install-aws:
@echo -e "\n....Installing AWS CLI v2 for $(ARCH) ($(AWS_ARCH))...."
@if ! command -v aws >/dev/null 2>&1; then \
curl "https://awscli.amazonaws.com/awscli-exe-linux-$(AWS_ARCH).zip" -o "awscliv2.zip"; \
unzip -o awscliv2.zip; \
./aws/install; \
rm -rf awscliv2.zip aws/; \
else \
echo "AWS CLI already installed"; \
fi
quickinstall: install-system install-go install-aws
@echo -e "\n✅ Quickinstall complete for $(ARCH)."
@echo " Please ensure /usr/local/go/bin is in your PATH."
@echo " Installed: Go ($(GO_ARCH)), AWS CLI ($(AWS_ARCH)), QEMU ($(QEMU_PACKAGES))"
# Lint all Go code via golangci-lint (replaces check-format, vet, gosec, staticcheck)
lint:
@echo "Running golangci-lint..."
$(_Q)golangci-lint run ./...
@echo " golangci-lint ok"
# Auto-fix all linter issues that have fixers
fix:
golangci-lint run --fix ./...
# Govulncheck — dependency vulnerability scanning (not covered by golangci-lint)
govulncheck:
@echo "Running govulncheck..."
$(_Q)go tool govulncheck ./...
@echo " govulncheck ok"
# Build release tarballs — use distro-ARCH for single arch, distro for both
distro: distro-amd64 distro-arm64
@echo ""
@echo "Distribution tarballs:"
@ls -lh dist/*.tar.gz
@echo ""
@cat dist/*.sha256
distro-amd64:
@echo "Building spinifex $(VERSION) linux/amd64..."
@mkdir -p dist/
docker buildx build \
--platform linux/amd64 \
--build-arg VERSION=$(VERSION) \
-f build/Dockerfile.distro \
--output type=local,dest=dist/amd64/ \
../
@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ] && [ -f $(MICROVM_OUT_DIR)/initramfs.cpio.gz ]; then \
echo "[distro-amd64] staging microvm artifacts into tarball"; \
mkdir -p dist/amd64/microvm; \
cp $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz dist/amd64/microvm/; \
else \
echo "[distro-amd64] WARNING: microvm artifacts missing — tarball will not include them"; \
echo "[distro-amd64] run 'make build-microvm-image' before 'make distro-amd64'"; \
fi
tar -czf dist/spinifex-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
sha256sum dist/spinifex-$(VERSION)-linux-amd64.tar.gz > dist/spinifex-$(VERSION)-linux-amd64.tar.gz.sha256
distro-arm64:
@echo "Building spinifex $(VERSION) linux/arm64..."
@mkdir -p dist/
docker buildx build \
--platform linux/arm64 \
--build-arg VERSION=$(VERSION) \
-f build/Dockerfile.distro \
--output type=local,dest=dist/arm64/ \
../
tar -czf dist/spinifex-$(VERSION)-linux-arm64.tar.gz -C dist/arm64 .
sha256sum dist/spinifex-$(VERSION)-linux-arm64.tar.gz > dist/spinifex-$(VERSION)-linux-arm64.tar.gz.sha256
distro-clean:
rm -rf dist/
# Ansible dev lifecycle (experimental, parallel to dev-*.sh / reset-dev-env.sh).
# See scripts/ansible/README.md and docs/development/improvements/ansible-dev-lifecycle.md.
ansible-dev-preflight:
cd scripts/ansible && ansible-playbook playbooks/dev-preflight.yml
ansible-dev-teardown:
cd scripts/ansible && ansible-playbook playbooks/dev-teardown.yml
ansible-dev-install:
cd scripts/ansible && ansible-playbook playbooks/dev-install.yml
ansible-dev-reset:
cd scripts/ansible && ansible-playbook playbooks/dev-reset.yml
ansible-dev-deploy:
cd scripts/ansible && ansible-playbook playbooks/dev-deploy.yml
ansible-dev-status:
cd scripts/ansible && ansible-playbook playbooks/dev-status.yml
ansible-dev-logs:
cd scripts/ansible && ansible-playbook playbooks/dev-logs.yml
# Snapshot / restore require an explicit -e snapshot_name=<name>. The
# wrapper passes ANSIBLE_EXTRA through so devs can run
# `make ansible-dev-snapshot ANSIBLE_EXTRA='-e snapshot_name=before-merge'`.
ansible-dev-snapshot:
cd scripts/ansible && ansible-playbook playbooks/dev-snapshot.yml $(ANSIBLE_EXTRA)
ansible-dev-restore:
cd scripts/ansible && ansible-playbook playbooks/dev-restore.yml $(ANSIBLE_EXTRA)
ansible-dev-version:
cd scripts/ansible && ansible-playbook playbooks/dev-version.yml
ansible-dev-vpc:
cd scripts/ansible && ansible-playbook playbooks/dev-vpc.yml $(ANSIBLE_EXTRA)
.PHONY: build build-ui build-installer build-lb-agent build-system-image build-microvm-image install-microvm go_build go_run preflight test test-cover test-race diff-coverage bench run test-actions test-harness manifest-check diff-coverage bench run \
deploy reinstall clean \
install-system install-go install-aws quickinstall \
lint fix govulncheck \
distro distro-amd64 distro-arm64 distro-clean \
ansible-dev-preflight ansible-dev-teardown ansible-dev-install ansible-dev-reset ansible-dev-deploy ansible-dev-status ansible-dev-logs ansible-dev-snapshot ansible-dev-restore ansible-dev-version ansible-dev-vpc
Reference in New Issue View Git Blame Copy Permalink