mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-11-07 11:19:56 +00:00
Josh Siviour
b9bce93877
* feat(lambda): add the unregistered RequestResponse Invoke path Invoke resolves the function and qualifier with the existing lookup, authorizes lambda:InvokeFunction on the function ARN, and hands one synchronous attempt to an injected Executor. With no executor it fails closed; a Pending function is ResourceConflictException. An accepted attempt is not cancelled by a caller disconnect: only the wait is bounded. Event, DryRun, LogType Tail and other unimplemented options are refused explicitly. Lambda stays unregistered and no user code runs. Also makes two flaky tests deterministic: the workerboot key-mismatch test could pick an identical key 1 time in 256, and the loadgen first-breach test depended on a 2ms op staying under a 30ms wall-clock p99. * feat: track each Lambda code upload before storing it CreateFunction now writes a per-Region upload record before the name claim and before any code Put, so a failed upload, rollback, expired claim or late original writer cannot leave an untracked copy. Refused or ambiguous outcomes keep the record; nothing is deleted or inferred complete from an expired claim. CodeStored validates the upload record before committing and refuses a missing, unreadable, future-format, mismatched or non-owed record without changing either record. Attach CASes only the checked record. * feat(lambda): run test-link Invoke in a daemon-owned microVM Behind the lambdatestlink build tag only. The gateway executor presigns the exact code digest and sends one NATS request to this node's daemon; the daemon fetches and verifies the code, then runs one fresh microVM per invocation through a runner that holds QEMU's creating thread, kills only through a verified pidfd, and counts a stop only on a recorded exit status. The host enforces the function timeout from dispatch, and a late result is a timeout. Each attempt writes a marker before allocating an address or tap and refuses further test invocations while any marker remains, including after a restart. Taps are created exclusively and removed only while their ifindex and owner alias still match. Anything whose stop or release is not proven stays held. The default build is unchanged: Invoke refuses before presigning, sending or allocating, and Lambda stays unrouted. Limits: functions never leave Pending (the env15 run activated its test record by hand); no grant mechanism; a hung daemon leaves the hard timeout unenforced; held attempts are not recovered after a restart; requests are bounded by the 1 MiB NATS message; a timeout is an unknown outcome rather than AWS's Unhandled error. * feat(lambda): enforce attempt grants and a QEMU hard-kill backstop on the test link Test-only and local: everything here is behind the lambdatestlink build tag except the new domains/lambda/execution owner package. The default gateway still refuses Invoke before presigning, sending or allocating, the normal guest agent stays closed, and Lambda stays unregistered. Attempt grants (domains/lambda/execution): - One owner record per node holds the current attempt; every change is a conditional write, and an unrecognised format is left untouched. - The gateway issues a grant binding request ID, account, Region, function, revision and code digest before anything is sent; a second attempt is refused while one is unresolved. - The daemon starts only under the current matching grant, before it downloads code or allocates anything; stale, replayed or mismatched grants are refused with the record unchanged. - A result counts only if the owner accepts it under the dispatch deadline while the attempt is running; a late or uncertain acceptance is reported as an unknown outcome, never success. - An attempt ends only after the VM stop is proven and resources are released, or when nothing was launched; otherwise it is held as ambiguous. Time alone never releases it. An uncertain stop write is an unknown outcome: it may already have landed. QEMU hard-kill backstop (runtime/compute/lambda/testlink): - A preloaded initializer arms a SIGKILL timer inside QEMU before main and acknowledges with QEMU's PID, the deadline and a per-attempt nonce. Any failure after it loads kills the process before main. - The daemon checks the initializer against a pinned SHA-256 before launch, builds QEMU's environment without inherited LD_* or backstop settings, and sends no invocation frame without the exact acknowledgement; a missing or wrong acknowledgement kills QEMU and keeps state until the stop is proven. Limits, not production readiness: - Not the M1 decision. The grant binds request identity and code digest, not payload, memory or function ARN, and the guest does not yet bind generation or sequence. - If the preload never loads (missing file, static or setuid QEMU), nothing arms; a daemon hung before refusing can leave QEMU alive, and the only guarantee is that no invocation frame was sent. - Hash-then-load has a replacement race and is not a trust boundary; the initializer is built and installed by hand, with no packaging. - A killed QEMU stays a zombie until the daemon reaps it; held attempts are not recovered after restart; duplicate attempt IDs are checked only against the current attempt. * feat(lambda): seal the test-link timer library and bind the guest to its grant Test-link (lambdatestlink) build only. Lambda stays publicly unregistered and the normal guest agent stays fail-closed. Timer library: the daemon reads the hard-kill initializer once, checks those bytes against the pinned digest, copies them into a sealed memfd, verifies the seals and re-hashes the sealed copy, then passes it to QEMU as an inherited fd loaded through /proc/self/fd. The initializer closes that fd before it acknowledges. A kernel that rejects MFD_EXEC, or a memfd_noexec policy that refuses it, refuses the attempt; there is no fallback. Guest binding: the daemon writes a test-grant fw_cfg entry with the owner's grant ID, attempt ID, environment ID and generation, and the guest's scope now comes from that grant. Before installing code or starting the bootstrap, the guest refuses unless the grant names the authenticated environment generation, the first frame carries that grant and attempt ID, and the frame deadline is present and still ahead on the guest clock. Limits: - The digest is only as trustworthy as the build input that pins it; the memfd closes the swap window between check and load, it does not authenticate it. - Not yet shown under the real spinifex-daemon systemd sandbox (MemoryDenyWriteExecute, ProtectProc) or with memfd_noexec set to 1 or 2. - fw_cfg and the request frame both come from the executor, so matching them does not independently authenticate owner authority. This is not the M1 decision. - The guest clock check is a local guard only; it cannot detect clock skew or a delayed, superseded grant. The guest has no channel to the owner and checks the generation only on the first frame. - Held attempts are still not recovered after a daemon restart. * feat(lambda): recover held test-link attempts after a daemon restart Test-link (lambdatestlink) build only. Lambda stays publicly unregistered and the normal guest agent stays fail-closed. Launch identity: attempt.Run passes the session callback its own duplicate of the clone-time pidfd and closes SessionDone before publishing the result. Before the guest is sent anything, the daemon takes the VM's pidfs inode from that exact handle (refusing if the pidfd is not on pidfs or the VM has exited), with the host machine ID, boot ID, PID namespace and the launching daemon's own pidfs identity, and records it in the owner record with a conditional Launched transition. A failed write holds the attempt as ambiguous. Recovery: RecoverLoop retries for a bounded number of rounds on the daemon context, each round holding the node slot so it never runs beside a request. It holds anything it cannot scope to the recorded boot and PID namespace with the launcher gone, and holds an attempt with no launch identity. A PID with no process, or one now held by a process with another pidfs inode, proves the stop without signalling; the recorded VM still running is killed through its own pidfd and counted stopped only once the pidfd reports the exit. Cleanup order: files, tap and address (MgmtIPAllocator.ReleaseChecked, which reports an unconfirmed release; Release is unchanged for its callers), then a new owner state released that still blocks Issue, then the marker, then ended. Recovery finishes from either side of each step. Limits: - Reboot and partition recovery stay held until host identity or fencing is established; a machine ID can be copied, so it is not host proof. - A launch that crashes before its identity is recorded stays held. - No recovery if the daemon cannot restart or hangs. - Not yet run under the real spinifex-daemon systemd unit, and no kernel without pidfs was tested. - This is not the M1 decision. * fix(lambda): refuse ownerless checked mgmt IP releases ReleaseChecked now refuses an entry that names no node, or an allocator with no node bound, and leaves both the cluster and local entries in place. The checked path deletes its local entry only after a confirmed release and only if it still holds the same address. Legacy Release is unchanged. Callers must serialize Allocate and ReleaseChecked for one instance ID; the Lambda test-link slot does. Recovery tests that need real pidfs now skip only when the kernel's pidfds are not on pidfs, and a non-pidfs descriptor is shown to be refused. The runtime still fails closed. Adds an owner test for Store.Launched. * fix(lambda): retry test-link recovery until it settles RecoverLoop now retries on the daemon context until the held attempt settles, starting at 5s and doubling to a 5 minute cap, instead of giving up after ten rounds. Each round still takes the node slot, so it never runs beside Handle. It covers only what a previous daemon left: an attempt a later Handle in the same daemon holds stays held until the next restart. The serving log now says the guest does not verify owner grant authority. * feat(lambda): build the test-link daemon and initializer as one pair Add make build-lambda-testlink. It compiles the hard-kill initializer, checks it against the 2.34 glibc ceiling, pins its SHA-256 into a lambdatestlink spx build, and proves with TestLambdaHardKillPin that the daemon's own pin accepts that file and refuses a changed one. Both files are built in a staging directory and moved together only on success; any earlier pair is removed first, so a failed rebuild leaves no pair. The target refuses to build unless Go targets linux on the host architecture. The initializer now uses _POSIX_C_SOURCE with -std=c17, so it no longer needs __isoc23_strtol (glibc 2.38). The daemon refuses an initializer built for another ELF machine or class. Startup now begins recovery with only the owner store, tap and address cleanup. A missing bridge address, pin, trust store or matching initializer stops new invocations but not recovery. Limits: test-only and unregistered. The pin is only as trustworthy as the build host and toolchain; the initializer is still installed by hand and ships in no release artefact. linux/arm64 is untested. Recovery needs pidfs, which older kernels may lack. This is not the M1 decision. * feat(lambda): fix one CLOCK_BOOTTIME backstop deadline per test-link attempt Test-only (lambdatestlink); the default build and public Lambda route are unchanged. - The daemon reads CLOCK_BOOTTIME once before the grant Start and fixes one absolute deadline: bounded code fetch + function timeout + boot allowance + kill margin. Nothing recomputes or extends it; Handle never retries and recovery never relaunches. - The hard-kill initializer arms its SIGKILL timer at that absolute deadline (SPX_HARDKILL_DEADLINE_NS, TIMER_ABSTIME) instead of relative to QEMU start. - Dispatch is refused once the deadline has passed: before QEMU starts, when the initializer ack is read at or after it (VM killed, stop proven), and immediately before the invocation frame is sent. A clock read failure counts as expired. Limits: tap/IP setup, image read, initializer seal and QEMU start have no caps of their own beyond the boot allowance. An overdue timer is not shown to fire before vCPUs resume after host suspend. Owner-side retention, cross-node ordering, clock-rate assumptions and M1 remain unsolved; nothing is released by time and held attempts still need stop proof. * feat(lambda): boot the test-link runtime image from a pinned digest The test-link daemon now prepares its runtime image once at startup: each boot file is opened without following symlinks or blocking, capped by size, copied into a sealed memfd and hashed there, and the image digest over those sealed copies must equal LAMBDA_IMAGE_DIGEST pinned into the build. QEMU boots only the sealed copies (fds 3-5), and firmware comes from the image directory rather than the shared microvm qboot.rom. The digest identifies the bytes; it does not authenticate where they came from. build-lambda-testlink-image builds a separately tagged guest into its own outputs and prints the digest with the daemon's own file selection. Both test-link targets refuse anything but an x86_64 host building linux/amd64, and the digest input is read raw and validated before it reaches ldflags. * test: cover test-link image load failures and daemon startup release Exercise each sealed-copy failure path through the memfd hook and check partial copies are closed. Extract the daemon's readiness and release steps so startup tests can show the image is released when serving fails and that Handle refuses without a prepared image. * fix: align test-link Invoke errors with AWS and verify guest build tag Malformed JSON payloads return 400 InvalidRequestContentException and oversized requests return 413 with X-Amzn-Errortype RequestEntityTooLargeException, both before the executor runs. Timeouts and lost or failed sends are reported as unknown rather than as a function timeout, and any Sandbox.Timedout result is treated as unknown. Oversized function responses report Unhandled with Function.ResponseSizeTooLarge in the body. Add lambda-image-agent-tag, which reads the agent's build information from each built initramfs and refuses an image whose lambdatestlink tag does not match the target. Add tests that the default gateway cannot invoke and that management IP Release drops the local entry when the address is held elsewhere or the KV is down. * test: use the shared RequireAWSCode helper in Lambda tests The per-package requireCode helpers were replaced by testutil.RequireAWSCode, which makes the same assertions; the Lambda Invoke and upload tests still called the removed helper. * fix: validate and authorize qualified Lambda Invoke requests Invoke now runs every deterministic check before authorization and authorizes the qualified function ARN derived from the same validated target. Qualifiers follow the documented FunctionName and Qualifier patterns and length limits, and a malformed query string is refused. Code download errors no longer carry transport detail, so a presigned credential in a bad redirect cannot reach the reply or the gateway log. A seam built only with lambdatestlink and lambdatestlinkseam drives real session outcomes through the executor and composed gateway in one test. * test: restore the loadgen and workerboot tests to dev * test: drive a bootstrap-posted timeout through the composed Invoke path * fix: carry handler and qualifier to the test-link runtime and refuse function environments The test link now passes the stored handler to the guest as _HANDLER and the validated qualifier into the invoked-function ARN. Functions with environment variables are refused before any presign, grant or dispatch, since delivering them needs a separate secret-transport design. * test: stub QEMU for the test-link package so it runs on hosts without it
602 lines
31 KiB
Makefile
602 lines
31 KiB
Makefile
GO_PROJECT_NAME := spx
|
|
SHELL := /bin/bash
|
|
|
|
# GOFIPS140 is a global build-config change: it alters every package's action ID,
|
|
# stdlib included, so a target that omits it shares no build-cache entry with one
|
|
# that sets it. Exported once here rather than per-recipe so the two can't drift.
|
|
export GOFIPS140 := v1.0.0
|
|
|
|
# Detect architecture for cross-platform support
|
|
ARCH := $(shell uname -m)
|
|
ifeq ($(ARCH),x86_64)
|
|
GO_ARCH := amd64
|
|
AWS_ARCH := x86_64
|
|
else ifeq ($(ARCH),aarch64)
|
|
GO_ARCH := arm64
|
|
AWS_ARCH := aarch64
|
|
else ifeq ($(ARCH),arm64)
|
|
GO_ARCH := arm64
|
|
AWS_ARCH := aarch64
|
|
else
|
|
$(error Unsupported architecture: $(ARCH). Only x86_64 and aarch64/arm64 are supported.)
|
|
endif
|
|
|
|
# Ask Go whether workspace mode is active
|
|
IN_WORKSPACE := $(shell go env GOWORK)
|
|
|
|
# Use -mod=mod unless Go reports an active workspace path
|
|
ifeq ($(IN_WORKSPACE),)
|
|
GO_BUILD_MOD := -mod=mod
|
|
else ifeq ($(IN_WORKSPACE),off)
|
|
GO_BUILD_MOD := -mod=mod
|
|
else
|
|
GO_BUILD_MOD :=
|
|
endif
|
|
|
|
# Quiet-mode filters (active when QUIET=1, set by preflight via recursive make)
|
|
# Note: grep pipelines use PIPESTATUS[0] so the exit status of `go test`
|
|
# propagates through the filter — otherwise a test failure is swallowed by
|
|
# grep's own (success) exit code and preflight prints "passed" on red.
|
|
ifdef QUIET
|
|
_Q = @
|
|
_COVQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' | grep -v 'coverage: 0\.0%' || true; }; exit $${PIPESTATUS[0]}
|
|
_RACEQ = 2>&1 | { grep -Ev '^\s*(ok|PASS|\?|=== RUN|--- PASS:)\s' || true; }; exit $${PIPESTATUS[0]}
|
|
_SECQ = >
|
|
_HDR = :
|
|
_RUNSH = sh "$$t" >"$$log" 2>&1 || { echo "-- $$t"; cat "$$log"; exit 1; }
|
|
else
|
|
_Q =
|
|
_COVQ =
|
|
_RACEQ =
|
|
_SECQ = 2>&1 | tee
|
|
_HDR = echo -e
|
|
_RUNSH = echo "-- $$t"; sh "$$t" || exit 1
|
|
endif
|
|
|
|
build: go_build build-installer build-lb-agent generate-aws-model-coverage
|
|
|
|
# Build spinifex-ui frontend (requires pnpm)
|
|
build-ui:
|
|
@echo -e "\n....Building spinifex-ui frontend...."
|
|
cd spinifex/runtime/roles/spinifexui/frontend && pnpm build
|
|
|
|
# Playwright suite for the console, against Vite dev and the built dist under
|
|
# the console's CSP. Not part of preflight; needs `pnpm exec playwright install chromium`.
|
|
test-ui-e2e: build-ui
|
|
cd spinifex/runtime/roles/spinifexui/frontend && pnpm e2e
|
|
|
|
# The same suite inside the pinned Playwright image CI uses (podman or docker),
|
|
# so screenshots compare as they will on the runner. ui-baselines redraws them.
|
|
ui-e2e-container:
|
|
scripts/ui-playwright.sh compare
|
|
|
|
ui-baselines:
|
|
scripts/ui-playwright.sh baselines
|
|
|
|
# GO commands
|
|
VERSION ?= $(shell git describe --tags --always --dirty --exclude '*-dev')
|
|
COMMIT ?= $(shell git rev-parse --short HEAD)
|
|
LDFLAGS := -s -w -X github.com/mulgadc/spinifex/spinifex/operator/cli.Version=$(VERSION) -X github.com/mulgadc/spinifex/spinifex/operator/cli.Commit=$(COMMIT)
|
|
|
|
go_build:
|
|
@echo -e "\n....Building $(GO_PROJECT_NAME)"
|
|
go build $(GO_BUILD_MOD) -ldflags "$(LDFLAGS)" -o ./bin/$(GO_PROJECT_NAME) cmd/spinifex/main.go
|
|
|
|
# Controlled-test only: builds the tagged daemon and its hard-kill initializer
|
|
# together, pinning the initializer's digest into that daemon build. The daemon
|
|
# keeps its symbols so the pin can be read back from the binary.
|
|
# LAMBDA_IMAGE_DIGEST is the test-link runtime image digest that
|
|
# build-lambda-testlink-image prints; it is read raw and validated before use.
|
|
LAMBDA_TESTLINK_DIR := ./bin/lambda-testlink
|
|
LAMBDA_TESTLINK_PKG := github.com/mulgadc/spinifex/spinifex/runtime/daemon
|
|
LAMBDA_TESTLINK_PIN := $(LAMBDA_TESTLINK_PKG).lambdaHardKillSHA256
|
|
LAMBDA_DIGEST_RE := ^[0-9a-f]{64}$$
|
|
build-lambda-testlink: override export LAMBDA_IMAGE_DIGEST := $(value LAMBDA_IMAGE_DIGEST)
|
|
build-lambda-testlink:
|
|
@[[ "$$LAMBDA_IMAGE_DIGEST" =~ $(LAMBDA_DIGEST_RE) ]] || { echo "LAMBDA_IMAGE_DIGEST must be 64 lowercase hex digits"; exit 1; }
|
|
@host=$$(uname -m); goos=$$(go env GOOS); goarch=$$(go env GOARCH); \
|
|
test "$$host" = x86_64 && test "$$goos/$$goarch" = linux/amd64 || { echo "the test link supports only an x86_64 host building linux/amd64, not $$host building $$goos/$$goarch"; exit 1; }
|
|
@rm -f $(LAMBDA_TESTLINK_DIR)/hardkill.so $(LAMBDA_TESTLINK_DIR)/$(GO_PROJECT_NAME)
|
|
@mkdir -p $(LAMBDA_TESTLINK_DIR).tmp
|
|
@rm -f $(LAMBDA_TESTLINK_DIR).tmp/hardkill.so $(LAMBDA_TESTLINK_DIR).tmp/$(GO_PROJECT_NAME)
|
|
cc -std=c17 -O2 -Wall -Wextra -Werror -shared -fPIC -o $(LAMBDA_TESTLINK_DIR).tmp/hardkill.so spinifex/runtime/compute/lambda/testlink/hardkill/hardkill.c -lrt
|
|
scripts/check-glibc-ceiling.sh 2.34 $(LAMBDA_TESTLINK_DIR).tmp/hardkill.so
|
|
@digest=$$(sha256sum $(LAMBDA_TESTLINK_DIR).tmp/hardkill.so | cut -d' ' -f1); \
|
|
test $${#digest} -eq 64 || { echo "initializer digest missing"; exit 1; }; \
|
|
echo "initializer sha256 $$digest"; \
|
|
pin="-X $(LAMBDA_TESTLINK_PIN)=$$digest -X $(LAMBDA_TESTLINK_PKG).lambdaImageDigest=$$LAMBDA_IMAGE_DIGEST"; \
|
|
LAMBDA_TESTLINK_INITIALIZER=$(abspath $(LAMBDA_TESTLINK_DIR).tmp/hardkill.so) go test -count=1 -tags lambdatestlink -ldflags "$$pin" -run '^TestLambda(HardKillPin|ImageDigestPin)$$' ./spinifex/runtime/daemon/ && \
|
|
go build $(GO_BUILD_MOD) -tags lambdatestlink -ldflags "$(subst -s -w ,,$(LDFLAGS)) $$pin" -o $(LAMBDA_TESTLINK_DIR).tmp/$(GO_PROJECT_NAME) cmd/spinifex/main.go
|
|
@mkdir -p $(LAMBDA_TESTLINK_DIR)
|
|
mv $(LAMBDA_TESTLINK_DIR).tmp/hardkill.so $(LAMBDA_TESTLINK_DIR).tmp/$(GO_PROJECT_NAME) $(LAMBDA_TESTLINK_DIR)/
|
|
@rmdir $(LAMBDA_TESTLINK_DIR).tmp
|
|
|
|
build-installer:
|
|
@echo -e "\n....Building spinifex-installer"
|
|
go build -ldflags "-s -w" -o ./bin/spinifex-installer cmd/installer/main.go
|
|
|
|
build-lb-agent:
|
|
@echo -e "\n....Building lb-agent (static)"
|
|
CGO_ENABLED=0 go build -ldflags "-s -w" -o ./bin/lb-agent cmd/lb-agent/main.go
|
|
|
|
build-ecs-agent: ## Build the ecs-agent (ships in the ECS-AMI guest; not in host `build`)
|
|
@echo -e "\n....Building ecs-agent (static)"
|
|
CGO_ENABLED=0 go build -ldflags "-s -w -X main.version=$(VERSION)" -o ./bin/ecs-agent ./cmd/ecs-agent
|
|
|
|
build-loadgen: ## Build spx-loadgen (drives the stress harness's measured load; not shipped to nodes)
|
|
@echo -e "\n....Building spx-loadgen"
|
|
go build -ldflags "-s -w" -o ./bin/spx-loadgen ./cmd/spx-loadgen
|
|
|
|
build-system-image: ## Build a system image from manifest (use IMAGE=lb or IMAGE=eks-node)
|
|
ifndef IMAGE
|
|
$(error IMAGE is required. Usage: make build-system-image IMAGE=lb)
|
|
endif
|
|
@if [ -f scripts/images/$(IMAGE).conf ]; then \
|
|
./scripts/build-system-image.sh scripts/images/$(IMAGE).conf $(if $(IMPORT),--import); \
|
|
elif [ -f scripts/images/$(IMAGE)/manifest.conf ]; then \
|
|
./scripts/build-system-image.sh scripts/images/$(IMAGE)/manifest.conf $(if $(IMPORT),--import); \
|
|
else \
|
|
echo "ERROR: no manifest at scripts/images/$(IMAGE).conf or scripts/images/$(IMAGE)/manifest.conf"; \
|
|
exit 1; \
|
|
fi
|
|
|
|
build-eks-node-image: ## Build the unified eks-node AMI (K3s server+agent; role at first boot; IMPORT=1 to register)
|
|
$(MAKE) build-system-image IMAGE=eks-node
|
|
|
|
import-eks-node-image: ## Build + register the eks-node AMI (requires a running cluster)
|
|
$(MAKE) build-system-image IMAGE=eks-node IMPORT=1
|
|
|
|
publish-eks-node-image: ## Build + publish the eks-node AMI to Cloudflare R2 (needs R2_ENDPOINT + AWS_* env)
|
|
./scripts/publish-system-image.sh scripts/images/eks-node/manifest.conf --build
|
|
|
|
# The Ubuntu/systemd EKS image, as opposed to build-eks-node-image's Alpine one.
|
|
# Composition lives in mkosi-build.sh's image_profiles table, so the image name
|
|
# is the whole interface here.
|
|
build-eks-node-mkosi-image: ## Build the Ubuntu mkosi EKS node image (server+agent; role at first boot)
|
|
./scripts/mkosi-build.sh --image spinifex-eks-node
|
|
|
|
build-ecs-node-image: ## Build the spinifex-ecs-node AMI (Alpine + containerd + ecs-agent; IMPORT=1 to register)
|
|
$(MAKE) build-system-image IMAGE=ecs-agent
|
|
|
|
import-ecs-node-image: ## Build + register the ecs-node AMI (requires a running cluster)
|
|
$(MAKE) build-system-image IMAGE=ecs-agent IMPORT=1
|
|
|
|
build-rds-postgres-image: ## Build the spinifex-rds-postgres AMI (Alpine + PostgreSQL 18 + rds-init; IMPORT=1 to register)
|
|
$(MAKE) build-system-image IMAGE=rds-postgres
|
|
|
|
import-rds-postgres-image: ## Build + register the rds-postgres AMI (requires a running cluster)
|
|
$(MAKE) build-system-image IMAGE=rds-postgres IMPORT=1
|
|
|
|
build-rds-mariadb-image: ## Build the spinifex-rds-mariadb AMI (Alpine + MariaDB 11.8 + rds-init; IMPORT=1 to register)
|
|
$(MAKE) build-system-image IMAGE=rds-mariadb
|
|
|
|
import-rds-mariadb-image: ## Build + register the rds-mariadb AMI (requires a running cluster)
|
|
$(MAKE) build-system-image IMAGE=rds-mariadb IMPORT=1
|
|
|
|
MICROVM_OUT_DIR := build/microvm
|
|
MICROVM_ARTIFACTS := $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz
|
|
MICROVM_INPUTS := scripts/build-microvm-image.sh $(MICROVM_OUT_DIR)/init.sh $(MICROVM_OUT_DIR)/inittab bin/lb-agent
|
|
|
|
# Grouped target — script writes both files in one run.
|
|
$(MICROVM_ARTIFACTS) &: $(MICROVM_INPUTS)
|
|
./scripts/build-microvm-image.sh
|
|
|
|
# Only triggers when bin/lb-agent is missing; preserves the artifact's mtime so
|
|
# build-microvm-image stays correctly stale-aware.
|
|
bin/lb-agent:
|
|
$(MAKE) build-lb-agent
|
|
|
|
# The dedicated SQS server, built from the go.mod-pinned module so the image
|
|
# ships exactly the reviewed version. Static for the musl guest.
|
|
bin/nats-server: go.mod go.sum
|
|
@mkdir -p bin
|
|
GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags '-s -w' -o $@ github.com/nats-io/nats-server/v2
|
|
|
|
# go_local_srcs lists the main module package dirs and files guest package $(1)
|
|
# links, scanned with the guest build settings; a dir's mtime catches a deleted
|
|
# or renamed file. A failed scan names a missing prerequisite, so make stops.
|
|
go_local_srcs = $(shell GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go list -deps -f '{{if and .Module .Module.Main}}{{.Dir}} {{range .GoFiles}}{{$$.Dir}}/{{.}} {{end}}{{range .EmbedFiles}}{{$$.Dir}}/{{.}} {{end}}{{end}}' $(1) || echo go-list-failed-for-$(1))
|
|
.SECONDEXPANSION:
|
|
|
|
# The SQS microVM image: one guest running its own SQS nats-server and the SQS
|
|
# worker as separate processes. A separate artifact from the LB image above.
|
|
SQS_WORKER_MICROVM_OUT_DIR := $(MICROVM_OUT_DIR)/sqs-worker
|
|
SQS_WORKER_MICROVM_ARTIFACTS := $(SQS_WORKER_MICROVM_OUT_DIR)/vmlinuz $(SQS_WORKER_MICROVM_OUT_DIR)/initramfs.cpio.gz
|
|
SQS_WORKER_MICROVM_INPUTS := scripts/build-sqs-store-microvm-image.sh scripts/lib/microvm-image.sh scripts/lib/sqs-store-microvm-publish.sh $(SQS_WORKER_MICROVM_OUT_DIR)/init.sh $(SQS_WORKER_MICROVM_OUT_DIR)/inittab $(addprefix $(SQS_WORKER_MICROVM_OUT_DIR)/,sqs-worker-lib.sh sqs-worker-prepare sqs-worker-store sqs-worker-ready sqs-worker-stop) bin/nats-server bin/sqs-worker
|
|
|
|
$(SQS_WORKER_MICROVM_ARTIFACTS) &: $(SQS_WORKER_MICROVM_INPUTS)
|
|
./scripts/build-sqs-store-microvm-image.sh
|
|
|
|
bin/sqs-worker: go.mod go.sum $$(call go_local_srcs,./cmd/sqs-worker)
|
|
@mkdir -p bin
|
|
GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags '-s -w' -o $@ ./cmd/sqs-worker
|
|
|
|
build-sqs-worker-microvm-image: $(SQS_WORKER_MICROVM_ARTIFACTS) ## Build the SQS microVM kernel and initramfs
|
|
|
|
# The Lambda runtime microVM image. Built on demand; not shipped or installed.
|
|
LAMBDA_RUNTIME_MICROVM_OUT_DIR := $(MICROVM_OUT_DIR)/lambda-runtime
|
|
LAMBDA_RUNTIME_MICROVM_ARTIFACTS := $(LAMBDA_RUNTIME_MICROVM_OUT_DIR)/vmlinuz $(LAMBDA_RUNTIME_MICROVM_OUT_DIR)/initramfs.cpio.gz
|
|
LAMBDA_RUNTIME_MICROVM_INPUTS := scripts/build-lambda-runtime-microvm-image.sh scripts/lib/microvm-image.sh $(addprefix $(LAMBDA_RUNTIME_MICROVM_OUT_DIR)/,init.sh inittab run-agent) bin/lambda-agent
|
|
|
|
$(LAMBDA_RUNTIME_MICROVM_ARTIFACTS) &: $(LAMBDA_RUNTIME_MICROVM_INPUTS)
|
|
./scripts/build-lambda-runtime-microvm-image.sh
|
|
|
|
bin/lambda-agent: go.mod go.sum $$(call go_local_srcs,./cmd/lambda-agent)
|
|
@mkdir -p bin
|
|
GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags '-s -w' -o $@ ./cmd/lambda-agent
|
|
|
|
build-lambda-runtime-microvm-image: $(LAMBDA_RUNTIME_MICROVM_ARTIFACTS) ## Build the Lambda runtime microVM kernel and initramfs (not shipped)
|
|
GOWORK=off go run ./cmd/lambda-image-agent-tag $(LAMBDA_RUNTIME_MICROVM_OUT_DIR)/initramfs.cpio.gz absent
|
|
|
|
# Controlled-test only: a separately built guest whose agent has the
|
|
# lambdatestlink tag, in its own outputs, never touching bin/lambda-agent or the
|
|
# default image. It prints the digest to pass as LAMBDA_IMAGE_DIGEST.
|
|
LAMBDA_TESTLINK_IMAGE_DIR := $(MICROVM_OUT_DIR)/lambda-testlink-runtime
|
|
build-lambda-testlink-image: ## Build the test-link Lambda guest image and print its digest (not shipped)
|
|
@host=$$(uname -m); goos=$$(go env GOOS); goarch=$$(go env GOARCH); \
|
|
test "$$host" = x86_64 && test "$$goos/$$goarch" = linux/amd64 || { echo "the test link supports only an x86_64 host building linux/amd64, not $$host building $$goos/$$goarch"; exit 1; }
|
|
$(MAKE) --no-print-directory $(MICROVM_OUT_DIR)/qboot.rom
|
|
@mkdir -p bin $(LAMBDA_TESTLINK_IMAGE_DIR)
|
|
GOWORK=off GOFLAGS= CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -tags lambdatestlink -ldflags '-s -w' -o bin/lambda-agent-testlink ./cmd/lambda-agent
|
|
./scripts/build-lambda-runtime-microvm-image.sh bin/lambda-agent-testlink $(LAMBDA_TESTLINK_IMAGE_DIR)
|
|
GOWORK=off go run ./cmd/lambda-image-agent-tag $(LAMBDA_TESTLINK_IMAGE_DIR)/initramfs.cpio.gz present
|
|
install -m 0644 $(MICROVM_OUT_DIR)/qboot.rom $(LAMBDA_TESTLINK_IMAGE_DIR)/qboot.rom
|
|
@digest=$$(GOWORK=off go run -tags lambdatestlink ./cmd/lambda-image-digest $(LAMBDA_TESTLINK_IMAGE_DIR)) && \
|
|
[[ "$$digest" =~ $(LAMBDA_DIGEST_RE) ]] || { echo "image digest not computed"; exit 1; }; \
|
|
echo "LAMBDA_IMAGE_DIGEST=$$digest"
|
|
|
|
# qboot is the microvm firmware. The source tarball ships with each release
|
|
# because qboot is GPL-2.0.
|
|
QBOOT_COMMIT := $(shell sed -n 's/^QBOOT_COMMIT="\(.*\)"$$/\1/p' scripts/build-qboot.sh)
|
|
QBOOT_SOURCE := build/qboot/qboot-$(QBOOT_COMMIT).tar.gz
|
|
QBOOT_ARTIFACTS := $(MICROVM_OUT_DIR)/qboot.rom $(MICROVM_OUT_DIR)/qboot.LICENSE $(QBOOT_SOURCE)
|
|
|
|
$(QBOOT_ARTIFACTS) &: scripts/build-qboot.sh
|
|
./scripts/build-qboot.sh
|
|
|
|
build-microvm-image: $(MICROVM_ARTIFACTS) $(QBOOT_ARTIFACTS) ## Build microVM kernel, initramfs and qboot firmware (incremental — skips when up to date)
|
|
|
|
install-sqs-worker-microvm: $(SQS_WORKER_MICROVM_ARTIFACTS) ## Install the SQS microVM image to /usr/share/spinifex/microvm/sqs-worker/
|
|
sudo install -d /usr/share/spinifex/microvm/sqs-worker
|
|
sudo rm -f /usr/share/spinifex/microvm/sqs-worker/initramfs.cpio.gz
|
|
sudo install -m 0644 $(SQS_WORKER_MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/sqs-worker/vmlinuz
|
|
sudo install -m 0644 $(SQS_WORKER_MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/sqs-worker/initramfs.cpio.gz
|
|
|
|
install-microvm: $(MICROVM_ARTIFACTS) $(QBOOT_ARTIFACTS) ## Install microVM artifacts to /usr/share/spinifex/microvm/
|
|
sudo install -d /usr/share/spinifex/microvm
|
|
sudo install -m 0644 $(MICROVM_OUT_DIR)/vmlinuz /usr/share/spinifex/microvm/vmlinuz
|
|
sudo install -m 0644 $(MICROVM_OUT_DIR)/initramfs.cpio.gz /usr/share/spinifex/microvm/initramfs.cpio.gz
|
|
sudo install -m 0644 $(MICROVM_OUT_DIR)/qboot.rom /usr/share/spinifex/microvm/qboot.rom
|
|
sudo install -m 0644 $(MICROVM_OUT_DIR)/qboot.LICENSE /usr/share/spinifex/microvm/qboot.LICENSE
|
|
|
|
# the pre-commit gate: lint, vuln, and the unit and e2e-harness tiers.
|
|
# integration and race tests skipped to keep quick, they run in CI.
|
|
#
|
|
# -j2 overlaps the only two heavy gates, lint and test-cover: alone each leaves
|
|
# most of the machine idle, so pairing them cuts a cold run by about a quarter.
|
|
# -Otarget keeps each gate's output in one block instead of interleaving them.
|
|
preflight:
|
|
@$(MAKE) --no-print-directory -j2 -Otarget QUIET=1 lint govulncheck test-cover diff-coverage test-harness test-build-scripts
|
|
@echo -e "\n ✅ Preflight passed — safe to commit."
|
|
|
|
# Shell suites + shellcheck for build/scripts/, the systemd-unit helpers that
|
|
# ship on every node (unlike scripts/images/, kept in preflight: a wrong
|
|
# restart decision here is an outage, not asset churn).
|
|
#
|
|
# NODE_SCRIPTS are the four that setup.sh installs into
|
|
# /usr/local/share/spinifex on every node. They are covered here for the same
|
|
# reason: between them they install the software, wire OVN, erase the state and
|
|
# remove the install, so a shellcheck warning in one is an outage or a wrong
|
|
# deletion, not a style note.
|
|
NODE_SCRIPTS := scripts/setup.sh scripts/setup-ovn.sh scripts/install-node.sh scripts/smoke-test.sh scripts/node-reset.sh scripts/uninstall-spx.sh scripts/lib/distro.sh scripts/lib/stage-sqs-store-microvm.sh
|
|
|
|
test-build-scripts:
|
|
@$(_HDR) "\n....Running build/scripts/**/*_test.sh...."
|
|
@log=$$(mktemp); trap 'rm -f "$$log"' EXIT; for t in $$(find build/scripts -name '*_test.sh' | sort); do \
|
|
$(_RUNSH); \
|
|
done
|
|
@$(_HDR) "\n....Running scripts/*_test.sh, scripts/lib/*_test.sh and scripts/appliance/*_test.sh...."
|
|
@log=$$(mktemp); trap 'rm -f "$$log"' EXIT; for t in $$( (find scripts -maxdepth 1 -name '*_test.sh'; find scripts/lib scripts/appliance -name '*_test.sh') | sort); do \
|
|
$(_RUNSH); \
|
|
done
|
|
@$(_HDR) "\n....Running scripts/terraform/latitude-spx/*_test.sh...."
|
|
@log=$$(mktemp); trap 'rm -f "$$log"' EXIT; for t in $$(find scripts/terraform/latitude-spx -maxdepth 1 -name '*_test.sh' | sort); do \
|
|
$(_RUNSH); \
|
|
done
|
|
@$(_HDR) "\n....Running .github/scripts/*_test.sh...."
|
|
@log=$$(mktemp); trap 'rm -f "$$log"' EXIT; for t in $$(find .github/scripts -maxdepth 1 -name '*_test.sh' | sort); do \
|
|
$(_RUNSH); \
|
|
done
|
|
$(_Q)shellcheck -S warning .github/scripts/hypervisor-disk-preflight.sh \
|
|
.github/scripts/wait-for-memory.sh \
|
|
.github/scripts/mulga-design-check.sh
|
|
@$(_HDR) "\n....Running shellcheck over build/scripts/**/*.sh + the node scripts...."
|
|
$(_Q)shellcheck -S warning $$(find build/scripts -name '*.sh' | sort) $(NODE_SCRIPTS)
|
|
$(_Q)shellcheck -S warning -x scripts/build-oracle-appliance.sh $$(find scripts/appliance -name '*.sh' | sort)
|
|
@$(_HDR) " test-build-scripts ok"
|
|
|
|
# E2E harness unit tests. Build-tagged `e2e` so they're skipped by the
|
|
# default `go test ./spinifex/...`. Runs with mocked AWS clients — no
|
|
# infrastructure required, safe to run in CI without a cluster.
|
|
#
|
|
# diskperf/ is here for the same reason: its baseline comparison and probe
|
|
# accounting decide whether the disk-performance gate passes, and that
|
|
# arithmetic has to be covered off-cluster or the gate is trusted untested.
|
|
test-harness:
|
|
@echo -e "\n....Running e2e harness unit tests...."
|
|
$(_Q)LOG_IGNORE=1 go test -tags=e2e -timeout 60s ./tests/e2e/harness/... ./tests/e2e/diskperf/... $(_RACEQ)
|
|
|
|
# In-process integration tier: the real gateway router against embedded NATS
|
|
# JetStream, with only the daemon-side NATS subjects stubbed.
|
|
AWS_MODEL_CONFORMANCE_REPORT ?= $(CURDIR)/.cache/aws-model-conformance-report.txt
|
|
AWS_MODEL_CONFORMANCE_MODE ?= fail
|
|
AWS_MODEL_OPERATION_COVERAGE_DIR ?= $(CURDIR)/docs/coverage
|
|
# -count=1 is load-bearing: the conformance report is written by the test binary,
|
|
# so a cached pass skips the run, leaves no report, and the cat below fails the
|
|
# target. It also keeps the conformance gate honest — a cached result would mean
|
|
# the check never ran against this commit. The generated request sweep adds
|
|
# about 15s, most of it EC2 fan-outs waiting out their window for instances
|
|
# no daemon-lite owns.
|
|
test-integration:
|
|
@echo -e "\n....Running in-process integration tests...."
|
|
$(_Q)LOG_IGNORE=1 AWS_MODEL_CONFORMANCE_MODE=$(AWS_MODEL_CONFORMANCE_MODE) AWS_MODEL_CONFORMANCE_REPORT=$(AWS_MODEL_CONFORMANCE_REPORT) go test -count=1 -tags=integration -timeout 120s ./tests/integration/... $(_RACEQ)
|
|
@cat $(AWS_MODEL_CONFORMANCE_REPORT)
|
|
@$(MAKE) --no-print-directory generate-aws-model-coverage
|
|
@echo "AWS operation coverage: $(AWS_MODEL_OPERATION_COVERAGE_DIR)"
|
|
|
|
generate-aws-model-coverage:
|
|
@go run ./cmd/aws-model-coverage -out $(AWS_MODEL_OPERATION_COVERAGE_DIR) -json $(AWS_MODEL_OPERATION_COVERAGE_DIR)/coverage.json
|
|
|
|
aws-model-coverage: generate-aws-model-coverage
|
|
@go run ./cmd/aws-model-coverage
|
|
|
|
# Segscan storage oracle: needs the mulga umbrella repo's scripts/segscan
|
|
# checked out alongside spinifex (see internal/testkit/segscanoracle), which
|
|
# is not the default local or CI layout, so this is a separate target from
|
|
# test-integration rather than folded into it. Skips itself when segscan's
|
|
# source isn't found.
|
|
test-segscan-oracle:
|
|
@echo -e "\n....Running segscan storage oracle test...."
|
|
$(_Q)LOG_IGNORE=1 go test -tags=integration,segscanoracle -timeout 120s ./tests/integration/... -run TestSegscanOracle $(_RACEQ)
|
|
|
|
# Run unit tests
|
|
test:
|
|
@echo -e "\n....Running tests for $(GO_PROJECT_NAME)...."
|
|
LOG_IGNORE=1 go test -timeout 180s ./spinifex/... ./cmd/... ./internal/... ./contracts/... ./architecture/...
|
|
|
|
# Empty locally, where reusing a cached result between runs is the point.
|
|
# CI passes -count=1 so a green run means the suite executed against that commit
|
|
# rather than replaying results the persisted build cache carried over.
|
|
GOTESTFLAGS ?=
|
|
|
|
# Run unit tests with coverage profile
|
|
COVERPROFILE ?= coverage.out
|
|
test-cover:
|
|
@echo -e "\n....Running tests with coverage for $(GO_PROJECT_NAME)...."
|
|
$(_Q)LOG_IGNORE=1 go test $(GOTESTFLAGS) -timeout 180s -coverprofile=$(COVERPROFILE) -covermode=atomic ./spinifex/... ./cmd/... ./internal/... ./contracts/... ./architecture/... $(_COVQ)
|
|
@scripts/check-coverage.sh $(COVERPROFILE) $(QUIET)
|
|
|
|
# Run unit tests with race detector
|
|
test-race:
|
|
@echo -e "\n....Running tests with race detector for $(GO_PROJECT_NAME)...."
|
|
$(_Q)LOG_IGNORE=1 SPINIFEX_SQS_R3=1 go test $(GOTESTFLAGS) -race -timeout 300s ./spinifex/... ./cmd/... ./internal/... ./contracts/... ./architecture/... $(_RACEQ)
|
|
$(_Q)LOG_IGNORE=1 go test $(GOTESTFLAGS) -race -tags=lambdatestlink -timeout 120s ./cmd/lambda-agent/ ./spinifex/agents/lambda/hostlink/ ./spinifex/runtime/compute/lambda/testlink/ ./spinifex/domains/network/host/ $(_RACEQ)
|
|
$(_Q)LOG_IGNORE=1 go test $(GOTESTFLAGS) -race -tags=lambdatestlink -timeout 120s -run '^TestLambdaTestLink(Executor|Reply)_' ./spinifex/runtime/roles/awsgw/ $(_RACEQ)
|
|
$(_Q)LOG_IGNORE=1 go test $(GOTESTFLAGS) -race -tags=lambdatestlink,lambdatestlinkseam -timeout 120s -run '^TestLambdaTestLinkSeam_' ./spinifex/runtime/roles/awsgw/ $(_RACEQ)
|
|
$(_Q)LOG_IGNORE=1 go test $(GOTESTFLAGS) -race -tags=lambdatestlink -timeout 120s -run '^TestLambdaTestLink' ./spinifex/runtime/daemon/ $(_RACEQ)
|
|
|
|
# Unit tests for in-repo GitHub Actions (e.g. .github/actions/e2e-analyze).
|
|
# Kept out of `test-cover` so coverage % isn't diluted by CI-only tooling.
|
|
test-actions:
|
|
@echo -e "\n....Running action tests...."
|
|
LOG_IGNORE=1 go test $(GOTESTFLAGS) -timeout 60s ./.github/actions/...
|
|
|
|
# Shell suites + shellcheck for scripts/images/ and images/mkosi.profiles/
|
|
# helpers baked into system images. Kept out of `preflight` (a dedicated CI
|
|
# job gates it on scripts/images/**+images/mkosi.profiles/** changes instead)
|
|
# so image-asset churn doesn't run on every Go contributor's commit.
|
|
#
|
|
# images/mkosi.profiles/ ships scripts with no .sh suffix (mkosi's own
|
|
# mkosi.*.chroot lifecycle hooks, and wrapper binaries like vllm-serve under
|
|
# mkosi.extra/) so a plain `-name '*.sh'` glob silently skips exactly the
|
|
# files most worth linting. Sources there are found by shebang instead of
|
|
# name; this also means a profile that ships no scripts at all yields an
|
|
# empty match rather than an error.
|
|
test-images:
|
|
@echo -e "\n....Running scripts/images/**/*_test.sh...."
|
|
@for t in $$(find scripts/images -name '*_test.sh' | sort); do \
|
|
echo "-- $$t"; \
|
|
bash "$$t" || exit 1; \
|
|
done
|
|
@echo -e "\n....Running images/mkosi.profiles/**/*_test.sh...."
|
|
@for t in $$(find images/mkosi.profiles -name '*_test.sh' | sort); do \
|
|
echo "-- $$t"; \
|
|
bash "$$t" || exit 1; \
|
|
done
|
|
@echo -e "\n....Running shellcheck over scripts/images/**/*.sh...."
|
|
shellcheck -S warning $$(find scripts/images -name '*.sh' | sort)
|
|
@echo -e "\n....Running shellcheck over images/mkosi.profiles/** shell sources...."
|
|
@srcs="$$(grep -rlIE '^#!/bin/(bash|sh)' images/mkosi.profiles 2>/dev/null | sort)"; \
|
|
if [ -n "$$srcs" ]; then \
|
|
shellcheck -S warning $$srcs; \
|
|
else \
|
|
echo " (no shell sources found under images/mkosi.profiles)"; \
|
|
fi
|
|
@echo " test-images ok"
|
|
|
|
# Check that new/changed code meets coverage threshold (runs tests first)
|
|
diff-coverage: test-cover
|
|
@QUIET=$(QUIET) scripts/diff-coverage.sh $(COVERPROFILE)
|
|
|
|
# Check that newly added test files use an external test package
|
|
test-package-check:
|
|
@QUIET=$(QUIET) scripts/check-test-package.sh
|
|
|
|
bench:
|
|
@echo -e "\n....Running benchmarks for $(GO_PROJECT_NAME)...."
|
|
LOG_IGNORE=1 go test -benchmem -run=. -bench=. ./...
|
|
|
|
# Fuzz every Fuzz* target for FUZZTIME each; -fuzz takes one target per run.
|
|
# Plain `go test` already replays the seeds and testdata/fuzz on preflight.
|
|
FUZZTIME ?= 30s
|
|
fuzz:
|
|
@set -e; grep -rlE '^func Fuzz\w+\(' --include='*_test.go' spinifex cmd internal | while read -r file; do \
|
|
for name in $$(grep -oE '^func Fuzz\w+' "$$file" | cut -d' ' -f2); do \
|
|
echo -e "\n....Fuzzing $$name ($$(dirname $$file)) for $(FUZZTIME)...."; \
|
|
LOG_IGNORE=1 go test -run='^$$' -fuzz="^$$name$$" -fuzztime=$(FUZZTIME) ./$$(dirname $$file); \
|
|
done; \
|
|
done
|
|
|
|
# Fast iteration: build + install binary + restart all services.
|
|
# Microvm artifacts are reinstalled when they already exist on disk — the rule's
|
|
# input timestamps drive a rebuild only if anything actually changed. On a fresh
|
|
# checkout (no build/microvm/vmlinuz yet) the install step is skipped; run
|
|
# `make install-microvm` explicitly the first time. They go in before the
|
|
# preflight, which checks the microvm firmware.
|
|
deploy: build
|
|
sudo install -m 755 bin/spx /usr/local/bin/spx
|
|
@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ]; then \
|
|
$(MAKE) install-microvm; \
|
|
else \
|
|
echo "[deploy] microvm artifacts absent — run 'make install-microvm' for first-time setup"; \
|
|
fi
|
|
@if [ "$${SKIP_ASSET_PREFLIGHT:-}" != "1" ]; then \
|
|
if ! sudo /usr/local/bin/spx admin preflight; then \
|
|
echo ""; \
|
|
echo "[deploy] host-asset preflight failed: this node's helper scripts, sudoers"; \
|
|
echo " grants, QEMU or microvm firmware don't match the binary just installed. Remediate with:"; \
|
|
echo " scripts/update-nodes.sh (from the mulga root)"; \
|
|
echo " or:"; \
|
|
echo " make reinstall"; \
|
|
echo " Set SKIP_ASSET_PREFLIGHT=1 to bypass for a deliberate binary-only push."; \
|
|
exit 1; \
|
|
fi; \
|
|
else \
|
|
echo "[deploy] SKIP_ASSET_PREFLIGHT=1 — skipping host-asset preflight"; \
|
|
fi
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart spinifex.target
|
|
|
|
# Re-run setup.sh after changing systemd units, helper scripts, or logrotate config.
|
|
# Not needed for code-only changes — use deploy for those.
|
|
reinstall:
|
|
scripts/dev-install.sh
|
|
|
|
clean:
|
|
rm -f ./bin/$(GO_PROJECT_NAME)
|
|
rm -rf spinifex/runtime/roles/spinifexui/frontend/dist
|
|
|
|
# On dnf hosts the runtime packages and repositories come from setup.sh, so the
|
|
# two cannot drift; nbdkit-devel, for building the plugin, is only in CodeReady.
|
|
install-system:
|
|
@echo -e "\n....Installing system dependencies for $(ARCH)...."
|
|
@pm=$$(INSTALL_SPINIFEX_LIB_ONLY=1 bash -c '. scripts/setup.sh && distro_pkg_manager') || exit 1; \
|
|
if [ "$$pm" = dnf ]; then \
|
|
INSTALL_SPINIFEX_LIB_ONLY=1 bash -c '. scripts/setup.sh && setup_sudo && detect_os && install_dnf_deps' && \
|
|
sudo dnf config-manager --set-enabled ol10_codeready_builder && \
|
|
sudo dnf install -y nbdkit-devel pkgconf-pkg-config make gcc openssh-clients git sudo; \
|
|
else \
|
|
sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
|
|
-o Dpkg::Options::="--force-confdef" \
|
|
-o Dpkg::Options::="--force-confold" \
|
|
nbdkit nbdkit-plugin-dev pkg-config qemu-system-x86 qemu-system-arm qemu-utils \
|
|
ovmf qemu-efi-aarch64 \
|
|
libvirt-daemon-system libvirt-clients libvirt-dev make gcc jq curl \
|
|
iproute2 netcat-openbsd openssh-client wget git unzip sudo xz-utils file \
|
|
ovn-central ovn-host openvswitch-switch dhcpcd-base; \
|
|
fi
|
|
|
|
install-go:
|
|
@echo -e "\n....Installing Go 1.27.3 for $(ARCH) ($(GO_ARCH))...."
|
|
@if [ ! -d "/usr/local/go" ]; then \
|
|
curl -L https://go.dev/dl/go1.27.3.linux-$(GO_ARCH).tar.gz | tar -C /usr/local -xz; \
|
|
else \
|
|
echo "Go already installed in /usr/local/go"; \
|
|
fi
|
|
@echo "Go version: $$(go version)"
|
|
|
|
install-aws:
|
|
@echo -e "\n....Installing AWS CLI v2 for $(ARCH) ($(AWS_ARCH))...."
|
|
@if ! command -v aws >/dev/null 2>&1; then \
|
|
curl "https://awscli.amazonaws.com/awscli-exe-linux-$(AWS_ARCH).zip" -o "awscliv2.zip"; \
|
|
unzip -o awscliv2.zip; \
|
|
./aws/install; \
|
|
rm -rf awscliv2.zip aws/; \
|
|
else \
|
|
echo "AWS CLI already installed"; \
|
|
fi
|
|
|
|
quickinstall: install-system install-go install-aws
|
|
@echo -e "\n✅ Quickinstall complete for $(ARCH)."
|
|
@echo " Please ensure /usr/local/go/bin is in your PATH."
|
|
|
|
lint:
|
|
@echo "Running golangci-lint..."
|
|
$(_Q)scripts/run-gate.sh golangci-lint golangci-lint run ./...
|
|
@echo " golangci-lint ok"
|
|
|
|
fix:
|
|
golangci-lint run --fix ./...
|
|
|
|
govulncheck:
|
|
@echo "Running govulncheck..."
|
|
$(_Q)scripts/run-gate.sh govulncheck go tool govulncheck ./...
|
|
@echo " govulncheck ok"
|
|
|
|
# NilAway — advisory nil-panic analysis. Not in preflight due to false positives
|
|
nilaway:
|
|
@echo "Running nilaway..."
|
|
$(_Q)scripts/run-gate.sh nilaway go tool nilaway -include-pkgs=github.com/mulgadc/spinifex -exclude-test-files ./...
|
|
@echo " nilaway ok"
|
|
|
|
# Build release tarballs — use distro-ARCH for single arch, distro for both
|
|
distro: distro-amd64 distro-arm64
|
|
@echo ""
|
|
@echo "Distribution tarballs:"
|
|
@ls -lh dist/*.tar.gz
|
|
@echo ""
|
|
@cat dist/*.sha256
|
|
|
|
distro-amd64:
|
|
@echo "Building spinifex $(VERSION) linux/amd64..."
|
|
@mkdir -p dist/
|
|
docker buildx build \
|
|
--platform linux/amd64 \
|
|
--build-arg VERSION=$(VERSION) \
|
|
-f build/Dockerfile.distro \
|
|
--output type=local,dest=dist/amd64/ \
|
|
../
|
|
@if [ -f $(MICROVM_OUT_DIR)/vmlinuz ] && [ -f $(MICROVM_OUT_DIR)/initramfs.cpio.gz ] && [ -f $(MICROVM_OUT_DIR)/qboot.rom ] && [ -f $(QBOOT_SOURCE) ]; then \
|
|
echo "[distro-amd64] staging microvm artifacts into tarball"; \
|
|
mkdir -p dist/amd64/microvm; \
|
|
cp $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz $(MICROVM_OUT_DIR)/qboot.rom $(MICROVM_OUT_DIR)/qboot.LICENSE dist/amd64/microvm/; \
|
|
cp $(QBOOT_SOURCE) dist/; \
|
|
else \
|
|
echo "[distro-amd64] WARNING: microvm artifacts missing — tarball will not include them"; \
|
|
echo "[distro-amd64] run 'make build-microvm-image' before 'make distro-amd64'"; \
|
|
fi
|
|
./scripts/lib/stage-sqs-store-microvm.sh $(SQS_WORKER_MICROVM_OUT_DIR) dist/amd64 sqs-worker
|
|
tar -czf dist/spinifex-$(VERSION)-linux-amd64.tar.gz -C dist/amd64 .
|
|
sha256sum dist/spinifex-$(VERSION)-linux-amd64.tar.gz > dist/spinifex-$(VERSION)-linux-amd64.tar.gz.sha256
|
|
|
|
distro-arm64:
|
|
@echo "Building spinifex $(VERSION) linux/arm64..."
|
|
@mkdir -p dist/
|
|
docker buildx build \
|
|
--platform linux/arm64 \
|
|
--build-arg VERSION=$(VERSION) \
|
|
-f build/Dockerfile.distro \
|
|
--output type=local,dest=dist/arm64/ \
|
|
../
|
|
tar -czf dist/spinifex-$(VERSION)-linux-arm64.tar.gz -C dist/arm64 .
|
|
sha256sum dist/spinifex-$(VERSION)-linux-arm64.tar.gz > dist/spinifex-$(VERSION)-linux-arm64.tar.gz.sha256
|
|
|
|
distro-clean:
|
|
rm -rf dist/
|
|
|
|
.PHONY: test-package-check build build-lambda-testlink build-lambda-testlink-image build-ui test-ui-e2e ui-e2e-container ui-baselines build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-rds-postgres-image import-rds-postgres-image build-rds-mariadb-image import-rds-mariadb-image build-microvm-image build-sqs-worker-microvm-image build-lambda-runtime-microvm-image install-microvm install-sqs-worker-microvm go_build preflight test test-cover test-race diff-coverage bench fuzz test-actions test-images test-build-scripts test-harness test-integration generate-aws-model-coverage aws-model-coverage test-segscan-oracle \
|
|
deploy reinstall clean \
|
|
install-system install-go install-aws quickinstall \
|
|
lint fix govulncheck nilaway \
|
|
distro distro-amd64 distro-arm64 distro-clean
|