mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-11-05 10:37:23 +00:00
Josh Siviour
b9bce93877
* feat(lambda): add the unregistered RequestResponse Invoke path Invoke resolves the function and qualifier with the existing lookup, authorizes lambda:InvokeFunction on the function ARN, and hands one synchronous attempt to an injected Executor. With no executor it fails closed; a Pending function is ResourceConflictException. An accepted attempt is not cancelled by a caller disconnect: only the wait is bounded. Event, DryRun, LogType Tail and other unimplemented options are refused explicitly. Lambda stays unregistered and no user code runs. Also makes two flaky tests deterministic: the workerboot key-mismatch test could pick an identical key 1 time in 256, and the loadgen first-breach test depended on a 2ms op staying under a 30ms wall-clock p99. * feat: track each Lambda code upload before storing it CreateFunction now writes a per-Region upload record before the name claim and before any code Put, so a failed upload, rollback, expired claim or late original writer cannot leave an untracked copy. Refused or ambiguous outcomes keep the record; nothing is deleted or inferred complete from an expired claim. CodeStored validates the upload record before committing and refuses a missing, unreadable, future-format, mismatched or non-owed record without changing either record. Attach CASes only the checked record. * feat(lambda): run test-link Invoke in a daemon-owned microVM Behind the lambdatestlink build tag only. The gateway executor presigns the exact code digest and sends one NATS request to this node's daemon; the daemon fetches and verifies the code, then runs one fresh microVM per invocation through a runner that holds QEMU's creating thread, kills only through a verified pidfd, and counts a stop only on a recorded exit status. The host enforces the function timeout from dispatch, and a late result is a timeout. Each attempt writes a marker before allocating an address or tap and refuses further test invocations while any marker remains, including after a restart. Taps are created exclusively and removed only while their ifindex and owner alias still match. Anything whose stop or release is not proven stays held. The default build is unchanged: Invoke refuses before presigning, sending or allocating, and Lambda stays unrouted. Limits: functions never leave Pending (the env15 run activated its test record by hand); no grant mechanism; a hung daemon leaves the hard timeout unenforced; held attempts are not recovered after a restart; requests are bounded by the 1 MiB NATS message; a timeout is an unknown outcome rather than AWS's Unhandled error. * feat(lambda): enforce attempt grants and a QEMU hard-kill backstop on the test link Test-only and local: everything here is behind the lambdatestlink build tag except the new domains/lambda/execution owner package. The default gateway still refuses Invoke before presigning, sending or allocating, the normal guest agent stays closed, and Lambda stays unregistered. Attempt grants (domains/lambda/execution): - One owner record per node holds the current attempt; every change is a conditional write, and an unrecognised format is left untouched. - The gateway issues a grant binding request ID, account, Region, function, revision and code digest before anything is sent; a second attempt is refused while one is unresolved. - The daemon starts only under the current matching grant, before it downloads code or allocates anything; stale, replayed or mismatched grants are refused with the record unchanged. - A result counts only if the owner accepts it under the dispatch deadline while the attempt is running; a late or uncertain acceptance is reported as an unknown outcome, never success. - An attempt ends only after the VM stop is proven and resources are released, or when nothing was launched; otherwise it is held as ambiguous. Time alone never releases it. An uncertain stop write is an unknown outcome: it may already have landed. QEMU hard-kill backstop (runtime/compute/lambda/testlink): - A preloaded initializer arms a SIGKILL timer inside QEMU before main and acknowledges with QEMU's PID, the deadline and a per-attempt nonce. Any failure after it loads kills the process before main. - The daemon checks the initializer against a pinned SHA-256 before launch, builds QEMU's environment without inherited LD_* or backstop settings, and sends no invocation frame without the exact acknowledgement; a missing or wrong acknowledgement kills QEMU and keeps state until the stop is proven. Limits, not production readiness: - Not the M1 decision. The grant binds request identity and code digest, not payload, memory or function ARN, and the guest does not yet bind generation or sequence. - If the preload never loads (missing file, static or setuid QEMU), nothing arms; a daemon hung before refusing can leave QEMU alive, and the only guarantee is that no invocation frame was sent. - Hash-then-load has a replacement race and is not a trust boundary; the initializer is built and installed by hand, with no packaging. - A killed QEMU stays a zombie until the daemon reaps it; held attempts are not recovered after restart; duplicate attempt IDs are checked only against the current attempt. * feat(lambda): seal the test-link timer library and bind the guest to its grant Test-link (lambdatestlink) build only. Lambda stays publicly unregistered and the normal guest agent stays fail-closed. Timer library: the daemon reads the hard-kill initializer once, checks those bytes against the pinned digest, copies them into a sealed memfd, verifies the seals and re-hashes the sealed copy, then passes it to QEMU as an inherited fd loaded through /proc/self/fd. The initializer closes that fd before it acknowledges. A kernel that rejects MFD_EXEC, or a memfd_noexec policy that refuses it, refuses the attempt; there is no fallback. Guest binding: the daemon writes a test-grant fw_cfg entry with the owner's grant ID, attempt ID, environment ID and generation, and the guest's scope now comes from that grant. Before installing code or starting the bootstrap, the guest refuses unless the grant names the authenticated environment generation, the first frame carries that grant and attempt ID, and the frame deadline is present and still ahead on the guest clock. Limits: - The digest is only as trustworthy as the build input that pins it; the memfd closes the swap window between check and load, it does not authenticate it. - Not yet shown under the real spinifex-daemon systemd sandbox (MemoryDenyWriteExecute, ProtectProc) or with memfd_noexec set to 1 or 2. - fw_cfg and the request frame both come from the executor, so matching them does not independently authenticate owner authority. This is not the M1 decision. - The guest clock check is a local guard only; it cannot detect clock skew or a delayed, superseded grant. The guest has no channel to the owner and checks the generation only on the first frame. - Held attempts are still not recovered after a daemon restart. * feat(lambda): recover held test-link attempts after a daemon restart Test-link (lambdatestlink) build only. Lambda stays publicly unregistered and the normal guest agent stays fail-closed. Launch identity: attempt.Run passes the session callback its own duplicate of the clone-time pidfd and closes SessionDone before publishing the result. Before the guest is sent anything, the daemon takes the VM's pidfs inode from that exact handle (refusing if the pidfd is not on pidfs or the VM has exited), with the host machine ID, boot ID, PID namespace and the launching daemon's own pidfs identity, and records it in the owner record with a conditional Launched transition. A failed write holds the attempt as ambiguous. Recovery: RecoverLoop retries for a bounded number of rounds on the daemon context, each round holding the node slot so it never runs beside a request. It holds anything it cannot scope to the recorded boot and PID namespace with the launcher gone, and holds an attempt with no launch identity. A PID with no process, or one now held by a process with another pidfs inode, proves the stop without signalling; the recorded VM still running is killed through its own pidfd and counted stopped only once the pidfd reports the exit. Cleanup order: files, tap and address (MgmtIPAllocator.ReleaseChecked, which reports an unconfirmed release; Release is unchanged for its callers), then a new owner state released that still blocks Issue, then the marker, then ended. Recovery finishes from either side of each step. Limits: - Reboot and partition recovery stay held until host identity or fencing is established; a machine ID can be copied, so it is not host proof. - A launch that crashes before its identity is recorded stays held. - No recovery if the daemon cannot restart or hangs. - Not yet run under the real spinifex-daemon systemd unit, and no kernel without pidfs was tested. - This is not the M1 decision. * fix(lambda): refuse ownerless checked mgmt IP releases ReleaseChecked now refuses an entry that names no node, or an allocator with no node bound, and leaves both the cluster and local entries in place. The checked path deletes its local entry only after a confirmed release and only if it still holds the same address. Legacy Release is unchanged. Callers must serialize Allocate and ReleaseChecked for one instance ID; the Lambda test-link slot does. Recovery tests that need real pidfs now skip only when the kernel's pidfds are not on pidfs, and a non-pidfs descriptor is shown to be refused. The runtime still fails closed. Adds an owner test for Store.Launched. * fix(lambda): retry test-link recovery until it settles RecoverLoop now retries on the daemon context until the held attempt settles, starting at 5s and doubling to a 5 minute cap, instead of giving up after ten rounds. Each round still takes the node slot, so it never runs beside Handle. It covers only what a previous daemon left: an attempt a later Handle in the same daemon holds stays held until the next restart. The serving log now says the guest does not verify owner grant authority. * feat(lambda): build the test-link daemon and initializer as one pair Add make build-lambda-testlink. It compiles the hard-kill initializer, checks it against the 2.34 glibc ceiling, pins its SHA-256 into a lambdatestlink spx build, and proves with TestLambdaHardKillPin that the daemon's own pin accepts that file and refuses a changed one. Both files are built in a staging directory and moved together only on success; any earlier pair is removed first, so a failed rebuild leaves no pair. The target refuses to build unless Go targets linux on the host architecture. The initializer now uses _POSIX_C_SOURCE with -std=c17, so it no longer needs __isoc23_strtol (glibc 2.38). The daemon refuses an initializer built for another ELF machine or class. Startup now begins recovery with only the owner store, tap and address cleanup. A missing bridge address, pin, trust store or matching initializer stops new invocations but not recovery. Limits: test-only and unregistered. The pin is only as trustworthy as the build host and toolchain; the initializer is still installed by hand and ships in no release artefact. linux/arm64 is untested. Recovery needs pidfs, which older kernels may lack. This is not the M1 decision. * feat(lambda): fix one CLOCK_BOOTTIME backstop deadline per test-link attempt Test-only (lambdatestlink); the default build and public Lambda route are unchanged. - The daemon reads CLOCK_BOOTTIME once before the grant Start and fixes one absolute deadline: bounded code fetch + function timeout + boot allowance + kill margin. Nothing recomputes or extends it; Handle never retries and recovery never relaunches. - The hard-kill initializer arms its SIGKILL timer at that absolute deadline (SPX_HARDKILL_DEADLINE_NS, TIMER_ABSTIME) instead of relative to QEMU start. - Dispatch is refused once the deadline has passed: before QEMU starts, when the initializer ack is read at or after it (VM killed, stop proven), and immediately before the invocation frame is sent. A clock read failure counts as expired. Limits: tap/IP setup, image read, initializer seal and QEMU start have no caps of their own beyond the boot allowance. An overdue timer is not shown to fire before vCPUs resume after host suspend. Owner-side retention, cross-node ordering, clock-rate assumptions and M1 remain unsolved; nothing is released by time and held attempts still need stop proof. * feat(lambda): boot the test-link runtime image from a pinned digest The test-link daemon now prepares its runtime image once at startup: each boot file is opened without following symlinks or blocking, capped by size, copied into a sealed memfd and hashed there, and the image digest over those sealed copies must equal LAMBDA_IMAGE_DIGEST pinned into the build. QEMU boots only the sealed copies (fds 3-5), and firmware comes from the image directory rather than the shared microvm qboot.rom. The digest identifies the bytes; it does not authenticate where they came from. build-lambda-testlink-image builds a separately tagged guest into its own outputs and prints the digest with the daemon's own file selection. Both test-link targets refuse anything but an x86_64 host building linux/amd64, and the digest input is read raw and validated before it reaches ldflags. * test: cover test-link image load failures and daemon startup release Exercise each sealed-copy failure path through the memfd hook and check partial copies are closed. Extract the daemon's readiness and release steps so startup tests can show the image is released when serving fails and that Handle refuses without a prepared image. * fix: align test-link Invoke errors with AWS and verify guest build tag Malformed JSON payloads return 400 InvalidRequestContentException and oversized requests return 413 with X-Amzn-Errortype RequestEntityTooLargeException, both before the executor runs. Timeouts and lost or failed sends are reported as unknown rather than as a function timeout, and any Sandbox.Timedout result is treated as unknown. Oversized function responses report Unhandled with Function.ResponseSizeTooLarge in the body. Add lambda-image-agent-tag, which reads the agent's build information from each built initramfs and refuses an image whose lambdatestlink tag does not match the target. Add tests that the default gateway cannot invoke and that management IP Release drops the local entry when the address is held elsewhere or the KV is down. * test: use the shared RequireAWSCode helper in Lambda tests The per-package requireCode helpers were replaced by testutil.RequireAWSCode, which makes the same assertions; the Lambda Invoke and upload tests still called the removed helper. * fix: validate and authorize qualified Lambda Invoke requests Invoke now runs every deterministic check before authorization and authorizes the qualified function ARN derived from the same validated target. Qualifiers follow the documented FunctionName and Qualifier patterns and length limits, and a malformed query string is refused. Code download errors no longer carry transport detail, so a presigned credential in a bad redirect cannot reach the reply or the gateway log. A seam built only with lambdatestlink and lambdatestlinkseam drives real session outcomes through the executor and composed gateway in one test. * test: restore the loadgen and workerboot tests to dev * test: drive a bootstrap-posted timeout through the composed Invoke path * fix: carry handler and qualifier to the test-link runtime and refuse function environments The test link now passes the stored handler to the guest as _HANDLER and the validated qualifier into the invoked-function ARN. Functions with environment variables are refused before any presign, grant or dispatch, since delivering them needs a separate secret-transport design. * test: stub QEMU for the test-link package so it runs on hosts without it
82 lines
3.2 KiB
Bash
82 lines
3.2 KiB
Bash
#!/bin/sh
|
|
# build-lambda-runtime-microvm-image_test.sh — a build into a chosen output
|
|
# directory stages and publishes there, and leaves the default input directory
|
|
# untouched.
|
|
#
|
|
# The script runs in a fixture repo against the real library, with only the
|
|
# rootfs, kernel and packing steps replaced so no image is downloaded.
|
|
set -eu
|
|
|
|
SCRIPTS_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
|
|
TMP=$(mktemp -d)
|
|
trap 'rm -rf "$TMP"' EXIT
|
|
|
|
fails=0
|
|
pass() { echo " ok - $1"; }
|
|
fail() { echo " FAIL - $1"; fails=$((fails + 1)); }
|
|
|
|
echo "-- build-lambda-runtime-microvm-image.sh output directory"
|
|
|
|
REPO="$TMP/repo"
|
|
IN="$REPO/build/microvm/lambda-runtime"
|
|
mkdir -p "$REPO/scripts/lib" "$IN" "$REPO/bin" "$TMP/bin"
|
|
cp "$SCRIPTS_DIR/build-lambda-runtime-microvm-image.sh" "$REPO/scripts/"
|
|
for f in init.sh inittab run-agent; do printf '%s\n' "$f" > "$IN/$f"; done
|
|
printf 'default image\n' > "$IN/vmlinuz"
|
|
printf '#!/bin/sh\n' > "$REPO/bin/agent-testlink"
|
|
chmod +x "$REPO/bin/agent-testlink"
|
|
printf '#!/bin/sh\necho "ELF 64-bit, statically linked"\n' > "$TMP/bin/file"
|
|
chmod +x "$TMP/bin/file"
|
|
BEFORE=$(cd "$IN" && ls -A | cksum)
|
|
|
|
# The real library, with the steps that need a rootfs or kernel replaced.
|
|
# Packing runs from inside the chroot as the real step does, and records where
|
|
# the stage is while it still exists.
|
|
cat > "$REPO/scripts/lib/microvm-image.sh" <<EOF
|
|
. "$SCRIPTS_DIR/lib/microvm-image.sh"
|
|
microvm_check_tools() { :; }
|
|
microvm_make_rootfs() { mkdir -p "\$CHROOT_DIR/etc"; }
|
|
microvm_empty_dev() { :; }
|
|
microvm_locate_kernel() { :; }
|
|
microvm_strip_modules() { :; }
|
|
microvm_strip_rootfs() { :; }
|
|
microvm_stage_kernel() { printf 'kernel\n' > "\$STAGE_DIR/vmlinuz"; }
|
|
microvm_pack_initramfs() {
|
|
(cd "\$CHROOT_DIR" && printf 'initramfs\n' | gzip > "\$STAGE_DIR/initramfs.cpio.gz")
|
|
dirname "\$STAGE_DIR" > "$TMP/stage-parent"
|
|
ls -A "$IN" > "$TMP/input-during"
|
|
}
|
|
EOF
|
|
|
|
# The output is given relative to the repo root, as make passes it.
|
|
OUT="$REPO/build/microvm/lambda-testlink-runtime"
|
|
if ! (cd "$REPO" && PATH="$TMP/bin:$PATH" bash scripts/build-lambda-runtime-microvm-image.sh bin/agent-testlink build/microvm/lambda-testlink-runtime) > "$TMP/log" 2>&1; then
|
|
fail "build into output dir: $(tr '\n' ' ' < "$TMP/log")"
|
|
else
|
|
if [ "$(cat "$TMP/stage-parent")" = "$OUT" ]; then
|
|
pass "stage created in the output directory"
|
|
else
|
|
fail "stage created in $(cat "$TMP/stage-parent"), want $OUT"
|
|
fi
|
|
if grep -q '^\.stage\.' "$TMP/input-during"; then
|
|
fail "stage present in the default input directory during the build"
|
|
else
|
|
pass "no stage in the default input directory during the build"
|
|
fi
|
|
if [ "$(cd "$IN" && ls -A | cksum)" = "$BEFORE" ] && grep -qx 'default image' "$IN/vmlinuz"; then
|
|
pass "default input directory unchanged"
|
|
else
|
|
fail "default input directory changed: $(cd "$IN" && ls -A | tr '\n' ' ')"
|
|
fi
|
|
if grep -qx kernel "$OUT/vmlinuz" && gzip -t "$OUT/initramfs.cpio.gz" && [ "$(ls -A "$OUT" | wc -l)" -eq 2 ]; then
|
|
pass "output directory holds exactly the published pair"
|
|
else
|
|
fail "output directory: $(ls -A "$OUT" | tr '\n' ' ')"
|
|
fi
|
|
fi
|
|
|
|
if [ "$fails" -ne 0 ]; then
|
|
echo "$fails failure(s)"
|
|
exit 1
|
|
fi
|