mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
Ben Duncan
c6841ce1f0
* feat(oci): separate using an instance principal from creating one The allocator needs OCI credentials at runtime, and a node with none forms, passes every health check, then refuses every launch wanting a public address. Instance principal is the credential route that puts no key material on a node, but it needs a dynamic group and a policy, and those are tenancy-root resources the compartment-scoped user the rest of the configuration runs as cannot create. enable_instance_principal conflated the two, so the better credential route was only reachable by a tenancy admin. instance_principal now takes off, adopt or create: adopt selects the credential and creates nothing, because the matching rule covers the whole compartment and so names no instance to reference. setup-identity.sh owns the create case, with its own state. validate-topology.sh destroys its state at the end of every run, so holding tenancy resources there would let a teardown delete the tenancy's policy. Also package install-node.sh and smoke-test.sh, so a cluster can be formed from a curl | bash install with no checkout. smoke-test.sh is not optional: formation scp's it from its own directory for the post-formation verification. * feat(oci): take the credential handoff from an operator hook, and widen the workflow name An API-key deployment needs a credential on each node, and nothing in this repository can supply one: a key in user-data is readable from instance metadata for the life of the instance, and this host runs other people's guests. A manual install does not survive either, because a topology destroys itself on success. So the harness calls an operator-owned executable after formation and before the pool, the only window where a node has /etc/spinifex and has not started the allocator. Instance principal needs no handoff, so the hook is skipped there. The workflow is renamed for the vendors it will cover rather than the first one. Nothing is orphaned in the Actions UI: it has never been on the default branch and has never run, so there is no history attached to the old filename. * docs(oci): the API-key path needs the config file, not just the PEM oci_config_file names an ordinary OCI SDK config and nothing creates it, so a node holding only the key forms and then fails every allocation with no stated cause. * fix(ci): the runner context is not available in a job-level env block A workflow referencing it there is rejected before any job starts, which is why the rename produced a failed run attributed to the push. The reference had never been validated: a dispatch-only workflow that is never dispatched is never parsed. * fix(deps): bump containerd to v2.4.1 for GO-2026-6597 The image-pull DoS via crafted OCI index graph amplification is reachable from the ECS agent's containerd puller, so the vulnerability scan fails preflight on v2.4.0. A patch release with no API change. * fix(e2e): probe guest public addresses from outside the cloud's own network A VCN does not reflect its own public addresses back into itself, so a node asking for one gets nothing however well the datapath works. The workbook driver runs on the first node, so every assertion that curls or SSHes a guest's public address failed on OCI against a cluster that was serving those addresses correctly to the internet. The driver now takes E2E_PUBLIC_PROXY, a SOCKS5 proxy used for public addresses only, and validate-topology.sh supplies one with a remote dynamic forward on the connection that launches the driver. Egress is then the host running the validation, which is outside the VCN and is where a customer sits. Unset leaves every other platform on a direct probe, which is already correct there. Also adds --version TAG for a repeatable install, separates the workflow's auth choice from its external-pool choice so an API key can run with the allocator live, and captures allocator diagnostics before the teardown destroys them. * ci: read the Go version from go.mod in the release workflows Both pinned 1.27.0 by hand while every other workflow reads the file, so the toolchain they build with drifts from the one the module declares the day go.mod moves. release.yml was already inconsistent with itself: its second setup-go step reads spinifex/go.mod. * feat: publish the OCI topology results as tables on the run page A green tick on one job with 4,000 lines of log behind it is confidence for whoever wrote the harness and nothing to anybody else. Someone assessing whether Spinifex works on their cloud should be able to see that it formed and served guests on a single VM, on three VMs and on bare metal without opening a log. record now takes <gate> <status> [detail] and writes results.tsv beside the human block, so a gate's status is a field the script sets rather than a regex over prose that reads fine and matches wrong. record_workbooks turns the driver's own go-test-shaped lines into workbooks.tsv, and runs on the failing branch too: which four passed is most of what the fifth failing means. e2e-oci.sh --summary-only renders both tables to stdout and to GITHUB_STEP_SUMMARY, and the workflow calls it under always() before the sweep, because the sweep destroys the state directories the tables are read from. A skipped gate gets its own mark, and a topology that never ran says so: the one way a results table can do real harm is a reader taking an absent row for a passing one. * docs(oci): cut the deploy-by-hand appendix, and validate the config for real The appendix was 444 of 1014 lines restating an install the Terraform already does, which made the document read as though deploying by hand were a normal route. It is not: the configuration builds the VCN, the volumes and both VNICs per node and renders the cloud-init, in the same order every time. What is genuinely OCI-specific is four things, each of which already has a section, so the replacement names those and sends everyone else to the single-node and multi-node install guides. Verify end to end told the reader to ping a guest's public address from the host, which cannot work: a VCN does not route its own public addresses back into itself. That is the finding this branch is built on and the document was asserting the opposite. It now says where to probe from and why, and carries the allocator's `resolved the external VNIC` check, which existed only in the cut appendix and is the one line proving the credentials and compartment are right. `spx config validate` is not a command -- spx has admin, get, service, top and version -- so that line could only ever fail and its output and status were both discarded. It is replaced by parsing the TOML, which is what was wanted: the remap and the pool append are textual edits, and a broken result would otherwise surface as every service failing to start with nothing saying why. The remap now checks its own substitution count for the same reason, since re.sub with no match rewrites the file unchanged and exits 0. Publication of the run-page tables moves to one owner. GITHUB_STEP_SUMMARY is per step, so appending from the run as well as the publish step would have put two copies of every table on the run page. * fix(oci): check the credential hook before building anything The hook is a path from a CI variable, and whether it exists on the runner is knowable before Terraform runs. Checked where it was used, a wrong path cost the apply, the install and the formation first -- forty minutes and a bare-metal bill to be told a file is missing. Nothing about the check depends on the apply, so it moves to the top, where --dry-run reaches it too. * fix(oci): the summary must say what was built and what was not tested The first CI run headed its tables "Published release `latest channel`" over a build of a feature branch, which is the one error in a results table that would mislead a reader who trusted it. The header now states a tree build as a tree build, and names how the allocator authenticated. A run with no external pool says so in the header rather than leaving it to a SKIPPED row further down. Public-address allocation is the claim a green table is most likely to be read as making, and it is exactly the claim such a run cannot support. Both flags are passed to the publish step too, so the summary rendered after a failure describes the same run as the one that failed. Raise the dev prerelease prune from five to twenty. A pinned dev tag is how an external evaluator installs a specific build, and five is about a week of them -- short enough that the tag someone was handed stops resolving while they are still using it. * docs(oci): the workbook table is a snapshot, and the nightly is the current answer It is dated against a build five releases of work ago, and the nightly now runs the same workbooks on all three topologies and publishes its own table. Say which one to read rather than letting a stale table be taken as current. * ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file A workflow's displayed name is captured when GitHub first registers it and is not refreshed afterwards. Both OCI entries were registered from a run that could not be parsed -- a job-level env block referencing runner.temp -- so both list as their path, which reads as a missing name: key and is not one. The 2026-10-02 run was green on all three topologies and the entry still read as a path, with updated_at unchanged from the broken first run, so fixing the file is necessary and not sufficient. A new path is a new registration, and its first run reads a valid file. Dropping -nightly is also more honest: there is no schedule: here, because GitHub registers one only from the default branch, so a cron could never run dev and a feature branch as separate nightlies. * Revert "ci: rename to e2e-cloudvendor.yml so the sidebar entry is registered from a valid file" The rename makes the workflow undispatchable. `gh workflow run` resolves a filename against the default branch, and the raw dispatch endpoint returns 404 for a path GitHub has not registered -- checked twice, several minutes apart, so it is not an indexing delay. The old name works only because it is already registered, and an unrunnable workflow is a worse problem than a mislabelled one. The sidebar label needs a fresh registration, which a rename cannot provide from a feature branch. It corrects itself when the file reaches main, where GitHub re-reads it; the alternative is deleting the dead registrations' runs, which is an operator's call. * fix(ci): never publish the credential hook's output as an artifact The hook is the operator's own executable, and the one thing known about it is that it handles a private key. Its stdout and stderr are captured to credential-hook.log, which the artifact's *.log glob was collecting into a downloadable archive with a fourteen-day retention. Excluded rather than trusted to have been quiet. A hook that prints its key once while someone is debugging it should not thereby publish it. * fix(oci): the hairpin probe measured nothing on one node, and claimed too much With a single host, HOSTS[0] and HOSTS[-1] are the same address, so the node connected to itself -- which always succeeds. It reported "present" on vm-single and bm for that reason and not as a measurement. It now needs two nodes and says so otherwise. The claim it carried was also wrong. "A VCN does not reflect its own public addresses back into itself" was read off a workstation run whose node_client_cidr_allow_list was narrowed to one office address, which makes a node's own public address a non-permitted source: the refusal that follows is identical to an absent hairpin. With the committed 0.0.0.0/0 default the probe reports the hairpin present between two nodes, so the original reading was confounded and is retracted. None of this changes the remedy. A customer reaches a guest from outside the VCN, which is where the proxy's egress is, and probing from there is correct whichever of the two causes is in play -- 15 of 15 workbooks pass through it on all three topologies. What changes is what the documentation may assert: the guide now names both causes as indistinguishable rather than stating one as fact. * docs(oci): a reader could not run the first command, because nothing told them to clone The guide's first code block was `cd scripts/terraform/oci-spx`, into a directory no step had obtained. It also never mentioned the SSH keypair the driver refuses to start without, nor that the configuration is not on main or in v1.21.0 yet, so all three had to be inferred from the deploy failing. Restructured so the deploy comes first and reads as four steps and one command: get the configuration, choose one node or three, write the inputs, run it. The credential decision stays before step 4, because it changes an argument there and a node without one forms, looks healthy, and then refuses every launch that wants a public address. Everything else -- why OCI, sizing, the architecture, the IAM policy, quotas, the oci CLI, what Terraform builds, every variable -- moves under Reference, unchanged in substance. Adds what a successful run prints, naming `allocator ready` as the line to look for: every line above it appears on a node that cannot allocate, and that is the one failure the run does not announce. Also drops an empty verdict.txt committed from a local run. The directory the guide now tells customers to work in should not ship one run's artifacts.
472 lines
21 KiB
Makefile
472 lines
21 KiB
Makefile
GO_PROJECT_NAME := spx
|
|
SHELL := /bin/bash
|
|
|
|
# GOFIPS140 is a global build-config change: it alters every package's action ID,
|
|
# stdlib included, so a target that omits it shares no build-cache entry with one
|
|
# that sets it. Exported once here rather than per-recipe so the two can't drift.
|
|
export GOFIPS140 := v1.0.0
|
|
|
|
# Detect architecture for cross-platform support
|
|
ARCH := $(shell uname -m)
|
|
ifeq ($(ARCH),x86_64)
|
|
GO_ARCH := amd64
|
|
AWS_ARCH := x86_64
|
|
else ifeq ($(ARCH),aarch64)
|
|
GO_ARCH := arm64
|
|
AWS_ARCH := aarch64
|
|
else ifeq ($(ARCH),arm64)
|
|
GO_ARCH := arm64
|
|
AWS_ARCH := aarch64
|
|
else
|
|
$(error Unsupported architecture: $(ARCH). Only x86_64 and aarch64/arm64 are supported.)
|
|
endif
|
|
|
|
# Ask Go whether workspace mode is active
|
|
IN_WORKSPACE := $(shell go env GOWORK)
|
|
|
|
# Use -mod=mod unless Go reports an active workspace path
|
|
ifeq ($(IN_WORKSPACE),)
|
|
GO_BUILD_MOD := -mod=mod
|
|
else ifeq ($(IN_WORKSPACE),off)
|
|
GO_BUILD_MOD := -mod=mod
|
|
else
|
|
GO_BUILD_MOD :=
|
|
endif
|
|
|
|
# Quiet-mode filters (active when QUIET=1, set by preflight via recursive make)
|
|
# Note: grep pipelines use PIPESTATUS[0] so the exit status of `go test`
|
|
# propagates through the filter — otherwise a test failure is swallowed by
|
|
# grep's own (success) exit code and preflight prints "passed" on red.
|
|
ifdef QUIET
|
|
_Q = @
|
|
_COVQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' | grep -v 'coverage: 0\.0%' || true; }; exit $${PIPESTATUS[0]}
|
|
_RACEQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' || true; }; exit $${PIPESTATUS[0]}
|
|
_SECQ = >
|
|
else
|
|
_Q =
|
|
_COVQ =
|
|
_RACEQ =
|
|
_SECQ = 2>&1 | tee
|
|
endif
|
|
|
|
build: go_build build-installer build-lb-agent generate-aws-model-coverage
|
|
|
|
# Build spinifex-ui frontend (requires pnpm)
|
|
build-ui:
|
|
@echo -e "\n....Building spinifex-ui frontend...."
|
|
cd spinifex/services/spinifexui/frontend && pnpm build
|
|
|
|
# GO commands
|
|
VERSION ?= $(shell git describe --tags --always --dirty)
|
|
COMMIT ?= $(shell git rev-parse --short HEAD)
|
|
LDFLAGS := -s -w -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Version=$(VERSION) -X github.com/mulgadc/spinifex/cmd/spinifex/cmd.Commit=$(COMMIT)
|
|
|
|
go_build:
|
|
@echo -e "\n....Building $(GO_PROJECT_NAME)"
|
|
go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go
|
|
|
|
build-installer:
|
|
@echo -e "\n....Building spinifex-installer"
|
|
go build -ldflags "-s -w" -o ./bin/spinifex-installer cmd/installer/main.go
|
|
|
|
build-lb-agent:
|
|
@echo -e "\n....Building lb-agent (static)"
|
|
CGO_ENABLED=0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
|
|
|
|
build-ecs-agent: ## Build the ecs-agent (ships in the ECS-AMI guest; not in host `build`)
|
|
@echo -e "\n....Building ecs-agent (static)"
|
|
CGO_ENABLED=0 go build -ldflags "-s -w -X main.version=$(VERSION)" -o ./bin/ecs-agent ./cmd/ecs-agent
|
|
|
|
build-loadgen: ## Build spx-loadgen (drives the stress harness's measured load; not shipped to nodes)
|
|
@echo -e "\n....Building spx-loadgen"
|
|
go build -ldflags "-s -w" -o ./bin/spx-loadgen ./cmd/spx-loadgen
|
|
|
|
build-system-image: ## Build a system image from manifest (use IMAGE=lb or IMAGE=eks-node)
|
|
ifndef IMAGE
|
|
$(error IMAGE is required. Usage: make build-system-image IMAGE=lb)
|
|
endif
|
|
@if [ -f scripts/images/$(IMAGE).conf ]; then \
|
|
./scripts/build-system-image.sh scripts/images/$(IMAGE).conf $(if $(IMPORT),--import); \
|
|
elif [ -f scripts/images/$(IMAGE)/manifest.conf ]; then \
|
|
./scripts/build-system-image.sh scripts/images/$(IMAGE)/manifest.conf $(if $(IMPORT),--import); \
|
|
else \
|
|
echo "ERROR: no manifest at scripts/images/$(IMAGE).conf or scripts/images/$(IMAGE)/manifest.conf"; \
|
|
exit 1; \
|
|
fi
|
|
|
|
build-eks-node-image: ## Build the unified eks-node AMI (K3s server+agent; role at first boot; IMPORT=1 to register)
|
|
$(MAKE) build-system-image IMAGE=eks-node
|
|
|
|
import-eks-node-image: ## Build + register the eks-node AMI (requires a running cluster)
|
|
$(MAKE) build-system-image IMAGE=eks-node IMPORT=1
|
|
|
|
publish-eks-node-image: ## Build + publish the eks-node AMI to Cloudflare R2 (needs R2_ENDPOINT + AWS_* env)
|
|
./scripts/publish-system-image.sh scripts/images/eks-node/manifest.conf --build
|
|
|
|
# The Ubuntu/systemd EKS image, as opposed to build-eks-node-image's Alpine one.
|
|
# Composition lives in mkosi-build.sh's image_profiles table, so the image name
|
|
# is the whole interface here.
|
|
build-eks-node-mkosi-image: ## Build the Ubuntu mkosi EKS node image (server+agent; role at first boot)
|
|
./scripts/mkosi-build.sh --image spinifex-eks-node
|
|
|
|
build-ecs-node-image: ## Build the spinifex-ecs-node AMI (Alpine + containerd + ecs-agent; IMPORT=1 to register)
|
|
$(MAKE) build-system-image IMAGE=ecs-agent
|
|
|
|
import-ecs-node-image: ## Build + register the ecs-node AMI (requires a running cluster)
|
|
$(MAKE) build-system-image IMAGE=ecs-agent IMPORT=1
|
|
|
|
build-rds-postgres-image: ## Build the spinifex-rds-postgres AMI (Alpine + PostgreSQL 18 + rds-init; IMPORT=1 to register)
|
|
$(MAKE) build-system-image IMAGE=rds-postgres
|
|
|
|
import-rds-postgres-image: ## Build + register the rds-postgres AMI (requires a running cluster)
|
|
$(MAKE) build-system-image IMAGE=rds-postgres IMPORT=1
|
|
|
|
build-rds-mariadb-image: ## Build the spinifex-rds-mariadb AMI (Alpine + MariaDB 11.8 + rds-init; IMPORT=1 to register)
|
|
$(MAKE) build-system-image IMAGE=rds-mariadb
|
|
|
|
import-rds-mariadb-image: ## Build + register the rds-mariadb AMI (requires a running cluster)
|
|
$(MAKE) build-system-image IMAGE=rds-mariadb IMPORT=1
|
|
|
|
MICROVM_OUT_DIR := build/microvm
|
|
MICROVM_ARTIFACTS := $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz
|
|
MICROVM_INPUTS := scripts/build-microvm-image.sh $(MICROVM_OUT_DIR)/init.sh $(MICROVM_OUT_DIR)/inittab bin/lb-agent
|
|
|
|
# Grouped target — script writes both files in one run.
|
|
$(MICROVM_ARTIFACTS) &: $(MICROVM_INPUTS)
|
|
./scripts/build-microvm-image.sh
|
|
|
|
# Only triggers when bin/lb-agent is missing; preserves the artifact's mtime so
|
|
# build-microvm-image stays correctly stale-aware.
|
|
bin/lb-agent:
|
|
$(MAKE) build-lb-agent
|
|
|
|
build-microvm-image: $(MICROVM_ARTIFACTS) ## Build microVM kernel + initramfs (incremental — skips when up to date)
|
|
|
|
install-microvm: $(MICROVM_ARTIFACTS) ## Install microVM artifacts to /usr/share/spinifex/microvm/
|
|
sudo install -d /usr/share/spinifex/microvm
|
|
sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
|
|
sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
|
|
|
|
# the pre-commit gate: manifest checks, lint, vuln, and the unit and e2e-harness tiers.
|
|
# integration and race tests skipped to keep quick, they run in CI.
|
|
#
|
|
# -j2 overlaps the only two heavy gates, lint and test-cover: alone each leaves
|
|
# most of the machine idle, so pairing them cuts a cold run by about a quarter.
|
|
# -Otarget keeps each gate's output in one block instead of interleaving them.
|
|
# manifest-check manifest-lint test-package-check removed, can add back later
|
|
preflight:
|
|
@$(MAKE) --no-print-directory -j2 -Otarget QUIET=1 lint govulncheck test-cover diff-coverage test-harness test-build-scripts
|
|
@echo -e "\n ✅ Preflight passed — safe to commit."
|
|
|
|
# Shell suites + shellcheck for build/scripts/, the systemd-unit helpers that
|
|
# ship on every node (unlike scripts/images/, kept in preflight: a wrong
|
|
# restart decision here is an outage, not asset churn).
|
|
#
|
|
# NODE_SCRIPTS are the four that setup.sh installs into
|
|
# /usr/local/share/spinifex on every node. They are covered here for the same
|
|
# reason: between them they install the software, wire OVN, erase the state and
|
|
# remove the install, so a shellcheck warning in one is an outage or a wrong
|
|
# deletion, not a style note.
|
|
NODE_SCRIPTS := scripts/setup.sh scripts/setup-ovn.sh scripts/install-node.sh scripts/smoke-test.sh scripts/node-reset.sh scripts/uninstall-spx.sh
|
|
|
|
test-build-scripts:
|
|
@echo -e "\n....Running build/scripts/**/*_test.sh...."
|
|
@for t in $$(find build/scripts -name '*_test.sh' | sort); do \
|
|
echo "-- $$t"; \
|
|
sh "$$t" || exit 1; \
|
|
done
|
|
@echo -e "\n....Running scripts/*_test.sh...."
|
|
@for t in $$(find scripts -maxdepth 1 -name '*_test.sh' | sort); do \
|
|
echo "-- $$t"; \
|
|
sh "$$t" || exit 1; \
|
|
done
|
|
@echo -e "\n....Running shellcheck over build/scripts/**/*.sh + the node scripts...."
|
|
shellcheck -S warning $$(find build/scripts -name '*.sh' | sort) $(NODE_SCRIPTS)
|
|
@echo " test-build-scripts ok"
|
|
|
|
# E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
|
|
# default `go test ./spinifex/...`. Runs with mocked AWS clients — no
|
|
# infrastructure required, safe to run in CI without a cluster.
|
|
#
|
|
# diskperf/ is here for the same reason: its baseline comparison and probe
|
|
# accounting decide whether the disk-performance gate passes, and that
|
|
# arithmetic has to be covered off-cluster or the gate is trusted untested.
|
|
test-harness:
|
|
@echo -e "\n....Running e2e harness unit tests...."
|
|
$(_Q)LOG_IGNORE=1 go test -tags=e2e -timeout 60s ./tests/e2e/harness/... ./tests/e2e/diskperf/... $(_RACEQ)
|
|
|
|
# In-process integration tier: the real gateway router against embedded NATS
|
|
# JetStream, with only the daemon-side NATS subjects stubbed.
|
|
AWS_MODEL_CONFORMANCE_REPORT ?= $(CURDIR)/.cache/aws-model-conformance-report.txt
|
|
AWS_MODEL_CONFORMANCE_MODE ?= fail
|
|
AWS_MODEL_OPERATION_COVERAGE_DIR ?= $(CURDIR)/docs/coverage
|
|
# -count=1 is load-bearing: the conformance report is written by the test binary,
|
|
# so a cached pass skips the run, leaves no report, and the cat below fails the
|
|
# target. It also keeps the conformance gate honest — a cached result would mean
|
|
# the check never ran against this commit. The generated request sweep adds
|
|
# about 15s, most of it EC2 fan-outs waiting out their window for instances
|
|
# no daemon-lite owns.
|
|
test-integration:
|
|
@echo -e "\n....Running in-process integration tests...."
|
|
$(_Q)LOG_IGNORE=1 AWS_MODEL_CONFORMANCE_MODE=$(AWS_MODEL_CONFORMANCE_MODE) AWS_MODEL_CONFORMANCE_REPORT=$(AWS_MODEL_CONFORMANCE_REPORT) go test -count=1 -tags=integration -timeout 120s ./tests/integration/... $(_RACEQ)
|
|
@cat $(AWS_MODEL_CONFORMANCE_REPORT)
|
|
@$(MAKE) --no-print-directory generate-aws-model-coverage
|
|
@echo "AWS operation coverage: $(AWS_MODEL_OPERATION_COVERAGE_DIR)"
|
|
|
|
generate-aws-model-coverage:
|
|
@go run ./cmd/aws-model-coverage -out $(AWS_MODEL_OPERATION_COVERAGE_DIR) -json $(AWS_MODEL_OPERATION_COVERAGE_DIR)/coverage.json
|
|
|
|
aws-model-coverage: generate-aws-model-coverage
|
|
@go run ./cmd/aws-model-coverage
|
|
|
|
# Segscan storage oracle: needs the mulga umbrella repo's scripts/segscan
|
|
# checked out alongside spinifex (see spinifex/testutil/segscanoracle), which
|
|
# is not the default local or CI layout, so this is a separate target from
|
|
# test-integration rather than folded into it. Skips itself when segscan's
|
|
# source isn't found.
|
|
test-segscan-oracle:
|
|
@echo -e "\n....Running segscan storage oracle test...."
|
|
$(_Q)LOG_IGNORE=1 go test -tags=integration,segscanoracle -timeout 120s ./tests/integration/... -run TestSegscanOracle $(_RACEQ)
|
|
|
|
# Validate docs/service-interfaces.yaml. Schema check + cross-reference
|
|
# of services/suites/fixtures + on-disk path existence.
|
|
manifest-check:
|
|
@echo -e "\n....Checking service-interfaces.yaml...."
|
|
@go run ./tests/e2e/manifest-check/cmd/manifest-check -repo-root . -manifest docs/service-interfaces.yaml
|
|
|
|
# Drift guards: direct-create fixture lint + NATS subject lint,
|
|
# ratcheted against tests/e2e/manifest-lint/baseline.txt. Fails only on NEW
|
|
# drift beyond the baseline.
|
|
manifest-lint:
|
|
@echo -e "\n....Linting manifest drift (fixtures + subjects)...."
|
|
@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root .
|
|
|
|
# Accept current drift into the baseline. Run after an intentional change.
|
|
manifest-lint-update:
|
|
@go run ./tests/e2e/manifest-lint/cmd/manifest-lint -repo-root . -update
|
|
|
|
# Run unit tests
|
|
test:
|
|
@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
|
|
LOG_IGNORE=1 go test -timeout 180s ./spinifex/... ./cmd/... ./internal/...
|
|
|
|
# Empty locally, where reusing a cached result between runs is the point.
|
|
# CI passes -count=1 so a green run means the suite executed against that commit
|
|
# rather than replaying results the persisted build cache carried over.
|
|
GOTESTFLAGS ?=
|
|
|
|
# Run unit tests with coverage profile
|
|
COVERPROFILE ?= coverage.out
|
|
test-cover:
|
|
@echo -e "\n....Running tests with coverage for $(GO_PROJECT_NAME)...."
|
|
$(_Q)LOG_IGNORE=1 go test $(GOTESTFLAGS) -timeout 180s -coverprofile=$(COVERPROFILE) -covermode=atomic ./spinifex/... ./cmd/... ./internal/... $(_COVQ)
|
|
@scripts/check-coverage.sh $(COVERPROFILE) $(QUIET)
|
|
|
|
# Run unit tests with race detector
|
|
test-race:
|
|
@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
|
|
$(_Q)LOG_IGNORE=1 go test $(GOTESTFLAGS) -race -timeout 300s ./spinifex/... ./cmd/... ./internal/... $(_RACEQ)
|
|
|
|
# Unit tests for in-repo GitHub Actions (e.g. .github/actions/e2e-analyze).
|
|
# Kept out of `test-cover` so coverage % isn't diluted by CI-only tooling.
|
|
test-actions:
|
|
@echo -e "\n....Running action tests...."
|
|
LOG_IGNORE=1 go test $(GOTESTFLAGS) -timeout 60s ./.github/actions/...
|
|
|
|
# Shell suites + shellcheck for scripts/images/ and images/mkosi.profiles/
|
|
# helpers baked into system images. Kept out of `preflight` (a dedicated CI
|
|
# job gates it on scripts/images/**+images/mkosi.profiles/** changes instead)
|
|
# so image-asset churn doesn't run on every Go contributor's commit.
|
|
#
|
|
# images/mkosi.profiles/ ships scripts with no .sh suffix (mkosi's own
|
|
# mkosi.*.chroot lifecycle hooks, and wrapper binaries like vllm-serve under
|
|
# mkosi.extra/) so a plain `-name '*.sh'` glob silently skips exactly the
|
|
# files most worth linting. Sources there are found by shebang instead of
|
|
# name; this also means a profile that ships no scripts at all yields an
|
|
# empty match rather than an error.
|
|
test-images:
|
|
@echo -e "\n....Running scripts/images/**/*_test.sh...."
|
|
@for t in $$(find scripts/images -name '*_test.sh' | sort); do \
|
|
echo "-- $$t"; \
|
|
bash "$$t" || exit 1; \
|
|
done
|
|
@echo -e "\n....Running images/mkosi.profiles/**/*_test.sh...."
|
|
@for t in $$(find images/mkosi.profiles -name '*_test.sh' | sort); do \
|
|
echo "-- $$t"; \
|
|
bash "$$t" || exit 1; \
|
|
done
|
|
@echo -e "\n....Running shellcheck over scripts/images/**/*.sh...."
|
|
shellcheck -S warning $$(find scripts/images -name '*.sh' | sort)
|
|
@echo -e "\n....Running shellcheck over images/mkosi.profiles/** shell sources...."
|
|
@srcs="$$(grep -rlIE '^#!/bin/(bash|sh)' images/mkosi.profiles 2>/dev/null | sort)"; \
|
|
if [ -n "$$srcs" ]; then \
|
|
shellcheck -S warning $$srcs; \
|
|
else \
|
|
echo " (no shell sources found under images/mkosi.profiles)"; \
|
|
fi
|
|
@echo " test-images ok"
|
|
|
|
# Check that new/changed code meets coverage threshold (runs tests first)
|
|
diff-coverage: test-cover
|
|
@QUIET=$(QUIET) scripts/diff-coverage.sh $(COVERPROFILE)
|
|
|
|
# Check that newly added test files use an external test package
|
|
test-package-check:
|
|
@QUIET=$(QUIET) scripts/check-test-package.sh
|
|
|
|
bench:
|
|
@echo -e "\n....Running benchmarks for $(GO_PROJECT_NAME)...."
|
|
LOG_IGNORE=1 go test -benchmem -run=. -bench=. ./...
|
|
|
|
# Fuzz every Fuzz* target for FUZZTIME each; -fuzz takes one target per run.
|
|
# Plain `go test` already replays the seeds and testdata/fuzz on preflight.
|
|
FUZZTIME ?= 30s
|
|
fuzz:
|
|
@set -e; grep -rlE '^func Fuzz\w+\(' --include='*_test.go' spinifex cmd internal | while read -r file; do \
|
|
for name in $$(grep -oE '^func Fuzz\w+' "$$file" | cut -d' ' -f2); do \
|
|
echo -e "\n....Fuzzing $$name ($$(dirname $$file)) for $(FUZZTIME)...."; \
|
|
LOG_IGNORE=1 go test -run='^$$' -fuzz="^$$name$$" -fuzztime=$(FUZZTIME) ./$$(dirname $$file); \
|
|
done; \
|
|
done
|
|
|
|
# Fast iteration: build + install binary + restart all services.
|
|
# Microvm artifacts are reinstalled when they already exist on disk — the rule's
|
|
# input timestamps drive a rebuild only if anything actually changed. On a fresh
|
|
# checkout (no build/microvm/vmlinuz yet) the install step is skipped; run
|
|
# `make install-microvm` explicitly the first time.
|
|
deploy: build
|
|
sudo install -m 755 bin/spx /usr/local/bin/spx
|
|
@if [ "$${SKIP_ASSET_PREFLIGHT:-}" != "1" ]; then \
|
|
if ! sudo /usr/local/bin/spx admin preflight; then \
|
|
echo ""; \
|
|
echo "[deploy] host-asset preflight failed: this node's helper scripts or sudoers"; \
|
|
echo " grants don't match the binary just installed. Remediate with:"; \
|
|
echo " scripts/update-nodes.sh (from the mulga root)"; \
|
|
echo " or:"; \
|
|
echo " make reinstall"; \
|
|
echo " Set SKIP_ASSET_PREFLIGHT=1 to bypass for a deliberate binary-only push."; \
|
|
exit 1; \
|
|
fi; \
|
|
else \
|
|
echo "[deploy] SKIP_ASSET_PREFLIGHT=1 — skipping host-asset preflight"; \
|
|
fi
|
|
@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ]; then \
|
|
$(MAKE) install-microvm; \
|
|
else \
|
|
echo "[deploy] microvm artifacts absent — run 'make install-microvm' for first-time setup"; \
|
|
fi
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart spinifex.target
|
|
|
|
# Re-run setup.sh after changing systemd units, helper scripts, or logrotate config.
|
|
# Not needed for code-only changes — use deploy for those.
|
|
reinstall:
|
|
scripts/dev-install.sh
|
|
|
|
clean:
|
|
rm -f ./bin/$(GO_PROJECT_NAME)
|
|
rm -rf spinifex/services/spinifexui/frontend/dist
|
|
|
|
install-system:
|
|
@echo -e "\n....Installing system dependencies for $(ARCH)...."
|
|
sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
|
|
-o Dpkg::Options::="--force-confdef" \
|
|
-o Dpkg::Options::="--force-confold" \
|
|
nbdkit nbdkit-plugin-dev pkg-config qemu-system-x86 qemu-system-arm qemu-utils \
|
|
ovmf qemu-efi-aarch64 \
|
|
libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
|
|
iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
|
|
ovn-central ovn-host openvswitch-switch dhcpcd-base
|
|
|
|
install-go:
|
|
@echo -e "\n....Installing Go 1.27.0 for $(ARCH) ($(GO_ARCH))...."
|
|
@if [ ! -d "/usr/local/go" ]; then \
|
|
curl -L https://go.dev/dl/go1.27.0.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
|
|
else \
|
|
echo "Go already installed in /usr/local/go"; \
|
|
fi
|
|
@echo "Go version: $$(go version)"
|
|
|
|
install-aws:
|
|
@echo -e "\n....Installing AWS CLI v2 for $(ARCH) ($(AWS_ARCH))...."
|
|
@if ! command -v aws >/dev/null 2>&1; then \
|
|
curl "https://awscli.amazonaws.com/awscli-exe-linux-$(AWS_ARCH).zip" -o "awscliv2.zip"; \
|
|
unzip -o awscliv2.zip; \
|
|
./aws/install; \
|
|
rm -rf awscliv2.zip aws/; \
|
|
else \
|
|
echo "AWS CLI already installed"; \
|
|
fi
|
|
|
|
quickinstall: install-system install-go install-aws
|
|
@echo -e "\n✅ Quickinstall complete for $(ARCH)."
|
|
@echo " Please ensure /usr/local/go/bin is in your PATH."
|
|
|
|
lint:
|
|
@echo "Running golangci-lint..."
|
|
$(_Q)scripts/run-gate.sh golangci-lint golangci-lint run ./...
|
|
@echo " golangci-lint ok"
|
|
|
|
fix:
|
|
golangci-lint run --fix ./...
|
|
|
|
govulncheck:
|
|
@echo "Running govulncheck..."
|
|
$(_Q)scripts/run-gate.sh govulncheck go tool govulncheck ./...
|
|
@echo " govulncheck ok"
|
|
|
|
# NilAway — advisory nil-panic analysis. Not in preflight due to false positives
|
|
nilaway:
|
|
@echo "Running nilaway..."
|
|
$(_Q)scripts/run-gate.sh nilaway go tool nilaway -include-pkgs=github.com/mulgadc/spinifex -exclude-test-files ./...
|
|
@echo " nilaway ok"
|
|
|
|
# Build release tarballs — use distro-ARCH for single arch, distro for both
|
|
distro: distro-amd64 distro-arm64
|
|
@echo ""
|
|
@echo "Distribution tarballs:"
|
|
@ls -lh dist/*.tar.gz
|
|
@echo ""
|
|
@cat dist/*.sha256
|
|
|
|
distro-amd64:
|
|
@echo "Building spinifex $(VERSION) linux/amd64..."
|
|
@mkdir -p dist/
|
|
docker buildx build \
|
|
--platform linux/amd64 \
|
|
--build-arg VERSION=$(VERSION) \
|
|
-f build/Dockerfile.distro \
|
|
--output type=local,dest=dist/amd64/ \
|
|
../
|
|
@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ] && [ -f $(MICROVM_OUT_DIR)/initramfs.cpio.gz ]; then \
|
|
echo "[distro-amd64] staging microvm artifacts into tarball"; \
|
|
mkdir -p dist/amd64/microvm; \
|
|
cp $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz dist/amd64/microvm/; \
|
|
else \
|
|
echo "[distro-amd64] WARNING: microvm artifacts missing — tarball will not include them"; \
|
|
echo "[distro-amd64] run 'make build-microvm-image' before 'make distro-amd64'"; \
|
|
fi
|
|
tar -czf dist/spinifex-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
|
|
sha256sum dist/spinifex-$(VERSION)-linux-amd64.tar.gz > dist/spinifex-$(VERSION)-linux-amd64.tar.gz.sha256
|
|
|
|
distro-arm64:
|
|
@echo "Building spinifex $(VERSION) linux/arm64..."
|
|
@mkdir -p dist/
|
|
docker buildx build \
|
|
--platform linux/arm64 \
|
|
--build-arg VERSION=$(VERSION) \
|
|
-f build/Dockerfile.distro \
|
|
--output type=local,dest=dist/arm64/ \
|
|
../
|
|
tar -czf dist/spinifex-$(VERSION)-linux-arm64.tar.gz -C dist/arm64 .
|
|
sha256sum dist/spinifex-$(VERSION)-linux-arm64.tar.gz > dist/spinifex-$(VERSION)-linux-arm64.tar.gz.sha256
|
|
|
|
distro-clean:
|
|
rm -rf dist/
|
|
|
|
.PHONY: test-package-check build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-rds-postgres-image import-rds-postgres-image build-rds-mariadb-image import-rds-mariadb-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench fuzz test-actions test-images test-build-scripts test-harness test-integration generate-aws-model-coverage aws-model-coverage test-segscan-oracle manifest-check manifest-lint manifest-lint-update \
|
|
deploy reinstall clean \
|
|
install-system install-go install-aws quickinstall \
|
|
lint fix govulncheck nilaway \
|
|
distro distro-amd64 distro-arm64 distro-clean
|