A user who signs up for the public sandbox and logs in at console.spx3.com
lands on the normal Dashboard, which reads "No instances", "No volumes" and
"No VPCs" for a brand new account. Nothing says this is a sandbox and nothing
suggests a first action.
Show a one-time dialog on the Dashboard when the console is reached at the
sandbox hostname. It names the sandbox, sets expectations about the cluster
being shared and reclaimable, and links to the areas worth visiting first.
Dismissal is remembered in localStorage, and storage being unavailable only
means it shows again next visit.
Sandbox-ness is decided from the hostname rather than a server flag: it is a
property of how the user reached the console, not of the node serving it, and
the same nodes answer on a LAN address and on the public name. This follows
isDirectHostAccess, which already gates the certificate banner on the login
page the same way.
The console SPA is embedded in the binary, so the rebuilt bundle is included.
The daemon's connect to the platform pgvector appliance gave up after a
fixed five attempts; once exhausted, startOchreVector returned and the
ochre.vector.query subject stayed unregistered for the daemon's lifetime,
so RAG never recovered after the appliance was torn down and re-adopted
across a spinifex.target restart without a manual daemon bounce.
Retry Ensure-then-Connect until the daemon context is cancelled instead,
via a small generic retryUntilContext helper with a capped doubling
backoff and throttled logging. Each attempt already re-resolves the live
dial IP, so a re-adopted appliance on a new address is picked up and the
query subject registers on its own.
Every non-success on the daemon's NATS path recorded as error, so an
unknown instance id, a bad parameter and a genuine server fault were one
series and a daemon error rate tracked caller behaviour as much as
daemon health. The same conflation at the log level made an expected
fan-out miss write an ERROR line on every node that was never going to
answer it: on a four-node cluster each IMDS metadata fetch by a booting
guest produced three that meant nothing.
Failures are now classified by the status their AWS error code maps to,
through the same function the HTTP middleware calls, so the two paths
cannot drift apart. A caller mistake records client_error and logs at
WARN; a server fault records error and stays at ERROR. The rule is now
that ERROR means the daemon failed, which is broader than the fan-out
case and subsumes it.
Separately, the per-instance command path was subscribed raw and passed
through no metrics wrapper at all, so start, stop, terminate, reboot and
the volume and ENI hot-plug commands produced no data points. They are
now counted under an action naming the command rather than the subject,
which carries the instance id. The drain runs off-thread and records its
own point once the flush finishes.
The gateway derived each request's IAM action prefix from the wire-service
name, so bedrock-runtime, bedrock-agent and bedrock-agent-runtime calls were
checked as bedrock-runtime:ConverseStream, bedrock-agent-runtime:Retrieve, etc.
AWS namespaces every Bedrock action under bedrock:, so an AWS-shaped scoped
policy never matched and only a wildcard admitted the call.
Collapse the Bedrock family to the shared bedrock: namespace in IAMAction, the
single choke point for policy decisions; recordResolvedAction keeps the wire
name for telemetry.
Guest series from qmp-collector carry the owning account as the
CloudWatch dimension account_id. Traces and logs carry the same account
as aws.account_id, and on metrics-apm* the bare account_id name is
already taken by the NATS global account the nats service reports. A
dashboard control built on the trace field therefore cannot reach guest
metrics, which is why the guest panels sit apart from the account
drill-down.
The dimension name is an AWS-facing contract, so it is not renamed. The
OTel bridge mirrors it instead: guest series now export aws.account_id
alongside account_id, and a series with no account exports neither a
blank value nor a placeholder.
ebsRequestWithTrace started its span from context.Background(), so every
ebs.mount, ebs.unmount and ebs.delete rooted a trace of its own. Block storage
work was therefore unreachable from the request that caused it: viperblockd
recorded 10,209 transactions in 24h and not one could be tied to a caller or an
account.
VolumeMounter now takes the caller's context. Launch and hot attach/detach have
one to give. Shutdown, crash recovery and teardown genuinely do not — they
answer to no request — so they pass a background context, which is an honest
root rather than a missing one.
The owning account rides both the span and the NATS header, because
viperblockd's consumer span reads that header to attribute itself. It comes off
the instance where there is one, and is passed explicitly for a lone volume
request, which carries no instance to read it from. An absent account is left
off rather than blanked: recovery and teardown act for nobody, and crediting
that work to whoever happens to be admin would read as tenant activity.
http.Server.WriteTimeout is a total deadline rather than an idle one, so any
value below the RDS command channel's 20s long poll kills every poll mid-flight
and strands agents on a channel no command can reach. Nothing recorded that the
omission was load-bearing, leaving it one hardening pass away from removal.
The sandbox is one cluster with tenancy at the account level, so every
question about it starts with which account. Today the account is on the
gateway's own spans and nowhere else: of 187,317 daemon transactions in 24
hours, none name an account, and the same holds for vpcd, viperblockd and
northstar. A tenant's request is attributable at the front door and anonymous
everywhere it does its work.
The account already crosses NATS on every request as X-Account-ID, so the
spans can name it without new plumbing. StartConsumerSpan reads the header and
startProducerSpan takes the account the caller sent for, which covers every
handler built on ServeNATSRequest and every fan-out through Gather.
Work with no caller - a sweep, a reconciler, a node-to-node probe - is left
unattributed rather than credited to whoever happens to be admin.
Merges `feat/ebs-export-tunables` (83 commits) into dev. The branch could not be rebase-merged because it carries five merges from dev, so it lands as a squash.
## What is in it
- **EBS export tunables** and the nbdkit/viperblock export path they configure.
- **Teardown reapers and backoff** for S3 buckets, load balancers, gateways, spot requests, certificates, endpoints, elastic IPs and network interfaces, so a dependent that keeps failing is backed off rather than retried hot.
- **Retry and tolerance paths**: ECS IAM service init, ECR signing keys deleted between the list and the read, an unreadable signing key at awsgw startup, and an ECS instance-profile policy attach converging on a stale read.
- **Key pair validation reads the material by name** instead of listing the account, closing the `InvalidKeyPair.NotFound` race that a single-replica LIST could produce.
- **Concurrency and fan-out**: concurrent metadata document fetches, bounded metadata reads for describes, and the node-discovery timeout removed from every fan-out call.
- **OVN correctness against the configured NB cluster**: the flows barrier, nexthop MAC binding, and stopping a compute node from running the package's standalone database.
- **nats-js-watchdog can re-arm itself.** The escalation check ran above the probe loop while the only line that cleared the counter sat below it, so a latched watchdog could never observe a recovery. It now probes first and decides second, and says so once per latch rather than once per tick.
- **CI**: 19 commits of nightly and build-pipeline work.
## Verification
`make preflight` green locally. Build and Test, Integration Tests, Race Detection, and Lint and Security all green on `c2da0385`.
## Known red, not introduced here
Two nightly cells fail on this branch and are pre-existing:
- **cell-10** — a partial OVN IPsec enable leaves `NB_Global ipsec=true` asserted cluster-wide while two of three chassis failed their local reconcile, black-holing cross-chassis guest traffic for roughly 5.5 minutes while every health signal stays green.
- **cell-19** — a total layer-3 egress blackhole below a correctly and promptly programmed NAT/EIP datapath, aggravated by a natuplink probe whose retry budget outlives the test's own wait, so it can never emit its FAIL verdict.
Both are filed separately and should be weighed before a release is cut.
The lockout fires after 10 auth failures from an address in 60 seconds and
then answers RequestLimitExceeded to everything from that address. A client
repeating one hopeless request reaches the threshold as fast as an attacker
working through a keyspace, and 503 tells it to retry later - advice it can
never act on, because the fault is a credential that will never resolve. On
wattle four guests holding deleted STS session credentials produced 53,392
rejected requests and 69,820 503s in a day, and could not recover by fixing
their credentials because the lockout is on the address, not the principal.
RecordFailure now takes a fingerprint identifying the attempt, and the window
holds one entry per distinct fingerprint: a repeat refreshes its timestamp
instead of adding to the count. Whatever an attacker must vary is what the
fingerprint carries - the key id, the presented signature, a digest of the
session token - so guessing is throttled exactly as before while one dead
credential presented 14,000 times stays one fault.
A request rejected before its key id is parsed has no identity to protect, so
malformed envelopes and skewed timestamps still count every occurrence.
An unresolvable instance role answers an empty 200 on the
security-credentials list and a 404 on the credential itself. An SDK reads
that as "no EC2 IMDS role found" and keeps signing with whatever it already
holds, so a guest whose credentials have expired retries forever and never
recovers. On wattle four guests did this for five days: 42,517 role-list
requests, no credential fetches, and not one line anywhere saying why.
The absence was invisible because three layers returned a silent nil -
DescribeInstances missing the instance, no instance profile ARN on the
record, and a profile IAM no longer has. profileFor now reports which of
those it hit, and the two credential paths warn on it, throttled to one line
per instance and reason every five minutes so a stuck fleet stays legible at
that request volume. A role-name mismatch is the guest's own doing and stays
silent.
96.7% of spinifex-daemon request points carried no outcome. natsMetricsHandler
recorded around a plain nats.MsgHandler, which writes its own reply and returns
nothing, so the wrapper had nothing to observe and passed an empty outcome. Only
the /health HTTP path, which goes through the OTel HTTP middleware, ever
reported one.
The handler type now reports how it answered:
type natsHandler func(*nats.Msg) string
Making it a distinct type means the compiler enumerates every subscription and
every handler, rather than a correlation map guessing at responders that reply
after the wrapper returns. Four outcomes are defined: success, error, skipped
for a broadcast handler that declines because the request names another node,
and a deferred sentinel for handlers that answer from a goroutine and record
their own point once the work finishes — timing the wrapper would have reported
a five-minute system launch as an instant success.
utils.ServeNATSRequest and ServeNATSRequestCtx now return whether they replied
with a result, so handlers built on them report an outcome without restating the
error handling. Adding the return value leaves existing callers untouched.
Also measures the per-reservation launch subject, which was subscribed raw and
emitted no request points at all. It records under the subject root because the
reservation id is unbounded.
awsgw request telemetry said POST / and nothing else: no AWS action, no caller,
no reason. On wattle that left 69,820 responses with status 503 in a day
unattributable from the request docs — the cause was only findable by leaving
them for separate auth-failure log lines, which do carry sourceIP and
accessKeyID.
The request logger is the outermost middleware, so a context derived downstream
never reaches it. A small mutable record now rides the request: the auth path
writes identity as it resolves it, writeSigV4Error writes the verdict, and the
action enricher writes the action. The logger and the server span both read it,
so a trace search for failing requests answers from where, doing what and
rejected why without a join.
The record holds an access key id and an error code and nothing else that could
hold a credential; a test asserts the field set so that stays true.
vpcd and spinifex-ui never called otelsetup.SetRequestAction, so rpc_method on
their request metrics stayed at the bare HTTP verb the middleware sets. Over 24h
vpcd had exactly two action values, GET and PUT, which left IMDS countable in
aggregate but made /latest/api/token indistinguishable from a credential fetch
at /latest/meta-data/iam/security-credentials/<role>. Guests depend on IMDS for
credentials, so a failure there is user-visible and was not attributable.
Both namers map to a closed set with a single catch-all, because a guest
controls the request path and an unbounded name would put one metric series
behind every invented path. The role name, the interface MAC and the public-key
index are stripped for the same reason, and spinifex-ui names the static and SPA
branches rather than the content-hashed asset filename.
The middleware also treated every 4xx as a success, which is why vpcd and
spinifex-ui reported zero errors over 24h despite IMDS answering 401 for a
missing IMDSv2 token and 404 for an unresolvable ENI. Those now record
client_error. error keeps meaning 5xx so existing error-rate panels are
unchanged, and a 4xx still does not mark the span failed.
The ecs-agent classified only *.dkr.ecr.* hosts as ECR, so a registry
advertised as the gateway's bare host/IP pulled anonymously and 401'd.
Recognise the agent's own gateway host as ECR and mint a token for it.
The instance role — the pull's fallback credential when a task carries no
execution role — granted ecs:* only; add ECR read so the pull authorizes.
* docs: repoint network invariants doc comment at docs/adr/
The region-local ADR set moved from docs/development/proposals/ to
docs/adr/ on acceptance, and 0006 kept its filename but the directory
changed.
* docs: repoint network invariants doc comment at the proposal path
The region-local ADR set was never accepted, so it lives under
docs/development/proposals/region-local-architecture/ rather than
docs/adr/.
---------
Co-authored-by: joshsiv-mulga <engineering@mulgadc.com>
A release for an IP with no tracked lease was answered with a plain error. The
comment at the responder assumed callers log-and-continue, so failing loudly
would surface the strand without wedging teardown. That assumption does not
hold: TerminatedTeardownReaper marks TeardownNAT failed, TeardownComplete stays
false, and the record is re-driven every sweep. No retry can make a deleted
lease reappear, so it never converges.
It also disables its own backstop. The terminated bucket has a 1h TTL meant to
bound the record regardless, but each sweep persists the advanced marks and that
write refreshes the TTL, so the record outlives the very thing meant to catch
it. Measured across four hosts: seventeen stranded addresses, each retried on a
fixed 120s cadence, indefinitely.
Keep the loud failure and make it terminal. The release reply gains a
machine-readable code, the client maps it to ErrLeaseNotTracked, and
ReleasePublicIP treats that as done after recording the leak, since the local
pool slot is already free and the upstream lease cannot be released without a
lease record. Teardown then completes and the record purges.
The strand is no longer silent: mulga.resource.leaked counts it by resource
class and the address is logged at error level, so a leaked address is
queryable rather than inferred from a repeating warning.
Go stores test results in the build cache alongside compiled objects, so now that the cache persists across runs an unchanged package replays its previous result instead of executing.
That is what makes a local run fast and exactly what a merge gate should not do: a green check would no longer mean the suite ran against that commit.
Add a GOTESTFLAGS hook, empty by default so local runs keep reusing results, and pass -count=1 from the two CI steps that gate a PR.
Compilation is still cached either way, so this costs no build time.
setup-go only writes its cache on a key miss, and keyed it on spinifex/go.sum alone.
Once a run saved that key every later run logged "Cache hit ... not saving cache", so the cache stayed frozen at whatever the first job to finish happened to have compiled and nothing built afterwards was ever stored.
All four jobs also shared that one key, so the job that won the save race decided what the other three restored.
The race job was handed artifacts built without -race, and the build job artifacts built without GOFIPS140, neither of which yields a single usable entry.
Manage both caches directly instead: modules keyed on all five go.sum files so the sibling modules stop re-downloading, and build artifacts keyed per job with a per-run suffix so each flavour keeps its own entry and refreshes it every run.
GOFIPS140 is a global build-config change: it alters every package's action ID, stdlib included.
A target that omits it therefore shares no build-cache entry with one that sets it, and the two flavours cost roughly a full cold build to switch between.
Seven targets set it inline while eight go invocations did not, so test-integration, test-harness, test-actions, test-segscan-oracle, bench and the go run tools all built the non-FIPS flavour.
In CI this is load-bearing: the integration job wins the shared setup-go cache-save race, stores a non-FIPS cache, and every FIPS build in the other jobs then misses on every lookup.
Export it once so the two cannot drift apart again.
The case seeded no config, so loadLocalConfig fell back to the host's
installed spinifex.toml. It passed on a provisioned dev box and failed on
a bare runner, where the region pre-flight fires before svc.Start().
* feat(ebs): make nbdkit thread count and per-volume read cache configurable
Both numbers that set a volume's export concurrency and memory were
hard-coded: nbdkit ran on its own default of 16 threads per connection
because no -t was ever passed, and the read cache was a 128 MiB literal in
the mount path. Neither could be changed without a recompile, and neither
was visible in the logs or the process argv.
Add default_threads and cache_size_mb to [nodes.<node>.ebs]. Defaults are
unchanged, so a config that does not set them behaves exactly as before.
CacheSizeMB is a pointer so an explicit 0 (cache disabled) stays
distinguishable from an unset key, matching how ViperblockConfig treats its
own toggles. -efi volumes stay uncached regardless.
Validation is a range check only. Both are operator tunables applied per
nbdkit process, and there is one of those per volume, so the host cost is
the value times the volume count -- left for the operator to weigh rather
than second-guessed here.
* feat(security): arm the firewall by install path and sync the clock at firstboot
The host firewall armed itself from a code default, so a curl-to-bash install
onto an existing Debian or Ubuntu machine applied a default-deny INPUT policy
seconds after bootstrap. Anything the operator was already running that is not
on the allow list stopped accepting connections, with nothing logged and no
config key to set beforehand — no template writes firewall_enabled, so every
install fell through to the default.
Arming is now the install path's decision. setup.sh takes --firewall=on|off and
records it in /etc/spinifex/firewall/mode, defaulting on under ISO_BUILD and off
everywhere else: the ISO is an appliance on hardware we define, a package install
is a guest on a machine that was already doing something.
firewall_enabled becomes a three-state *bool so "unset" and "explicitly false"
stay distinguishable. Flipping the default to false instead would have torn down
a working policy on every existing node, because false is an instruction to
disable rather than an absence. Resolution is config, then mode file, then on, so
a node installed before any of this keeps what it has.
The ISO had the opposite problem. setup.sh is the only place that enables
spinifex-firewall.service and it returns before that line under ISO_BUILD, with
no systemd in the chroot to talk to. Nothing else enabled it, so after a reboot
the ruleset — runtime state — was gone, and ReconcileFirewall short-circuited on
an unchanged peer file and never put it back. An ISO node was firewalled until
its first reboot and open permanently after. firstboot now enables the unit, and
the reconcile checks the table is loaded rather than treating an unchanged peer
set as proof the policy is applied.
Two policy changes: 443 is held for the nginx TLS edge so that rollout needs no
per-node firewall change, and the SSH port is read from sshd_config and its
Include drop-ins instead of being hardcoded to 22 — a host on Port 2222 was
allowed exactly one port that was not the one it uses, and ct state established
kept the operator's session alive until they disconnected. 53 stays open on every
node, now with a comment saying so: northstar serves public authoritative DNS.
Clock: firstboot steps and waits for sync before spx admin init mints
credentials, since a certificate issued against a wrong RTC carries a NotBefore
that outlives the correction. Bounded to 60s and non-fatal so an air-gapped
install still completes, and once is enough because rtcsync writes the corrected
time back. A chrony drop-in replaces Debian's makestep 1 3, which stops stepping
after three updates and then slews at roughly 7 seconds a day — a node that
burned those before the network came up never visibly converged.
* fix(security): let a node join a cluster whose firewall is already armed
Expanding a single-node ISO install into a cluster could not succeed. Two
independent blockers, both from the same root: the peer set is derived from
converged cluster state, but the ports needed to reach convergence are gated
on that same set.
The formation port is peer-scoped, and a node dialling in to join is not a
peer yet, so the handshake was dropped rather than refused and the joiner
retried in silence until it timed out. It cannot simply move to the public
rule: outside formation the same port carries the daemon cluster manager,
whose /health and /local/* routes report node topology, service inventory and
running instances unauthenticated. So it is opened for the formation window
only, by spx admin init, and closed again on every exit path.
The reconcile then deadlocked. Holding out for the full chassis list is right
on a node with no policy loaded, but on one armed for a smaller cluster the
ports a joining node needs in order to register its chassis are exactly the
ones the stale set blocks, so the list can never complete. Widen the peer set
on the tunnel set already loaded, and report the partial state so the caller
retries sooner than the steady-state interval. A converged cluster whose
Southbound DB is briefly unreadable is still left untouched.
* docs(install): explain the firewall around cluster formation
An ISO-installed server is a cluster of one, and its firewall admits internal
cluster traffic only from members it recognises. Forming a cluster out of such
servers therefore needs the firewall off first: step 3 of the multi-node guide
has each server dial node 1's OVN database on the raft ports, and those are
peer-scoped like everything else on the cluster plane. Neither guide said so,
and the failure is a silent drop that looks like a network fault.
Both guides now walk the off / form / on sequence, and the multi-node guide
gains a reference section covering which ports are public, how arming differs
between the ISO and the binary installer, and how to verify membership landed.
Re-enable the boot-time unit whenever the daemon arms the policy. Disabling
the firewall to form a cluster also disables that unit, so re-arming used to
look complete while leaving the node bare after its next reboot.
* fix: make the ISO-to-cluster conversion path work end to end
Forming a cluster from four ISO-imaged servers surfaced a run of bugs that
share a starting state CI never exercises: the ISO delivers an initialized,
armed, running node, while the conversion path was written against the binary
installer's unconfigured one.
- install-node.sh never disarmed the host firewall, so nodes 2 and 3 could not
dial the first node on the OVN raft ports and formation could not proceed. It
now owns the whole lifecycle: disarm, form, re-arm, then verify the loaded
policy against the expected peer set.
- spx admin init left the admin access key and secret empty on the preserve
path. The pair cannot be read back, and the IAM store is rebuilt by formation,
so the empty strings reached every joiner's bootstrap.json and crash-looped
awsgw. It is reissued instead, and IAM seeding now skips an empty admin block
rather than taking the gateway down on every restart.
- node-reset.sh left /etc/default/ovn-central behind. Raft membership lives in
that file, not in the databases the reset deletes, so ovn-central recreated
the database as a cluster member dialling a peer that had itself been wiped.
- node-reset.sh also destroyed the installed firewall policy, which nothing
regenerates. set-peers then failed on every attempt and the node stayed open
permanently and silently. The arming mode is read before the wipe and the
stage regenerated after it.
- spinifex-firewall-apply rejected every verb when the policy file was missing,
including disable. Turning the firewall off is the recovery path, so that left
a node that could be neither armed nor disarmed.
- viperblock exited 0 on a fatal startup error, so Restart=on-failure never
fired. During formation its JetStream bucket has no leader yet, so the EBS
provider died for good and every volume request failed with no responders.
predastore and qemunbd had the same bug.
- spx admin images import created image directories at 0700, so the first sudo
import locked group members out of a tree deliberately made group-writable.
Three of these hid behind checks that cannot succeed: ovn-sbctl show against a
clustered database, and the OVN northbound and southbound waits in setup-ovn.sh.
All three failed to stderr and let the run continue, so a check that could never
pass was indistinguishable from one that was merely slow.
The USB install guide claimed the ISO leaves Spinifex unconfigured and told the
operator to run spx admin init. firstboot has always done that itself, so the
guide now describes what the ISO actually produces and how to start using it.
* fix: let spread placement launch across every node
A spread placement group reserved four distinct nodes correctly and then failed
the launch with ServerInternal, having started only two or three of them. The
scheduler was never at fault; two faults downstream stopped the instances
following the reservation.
vpcd subscribes to vpc.add-nat with QueueSubscribe, which delivers one message
at a time, and the handler holds its reply across the OVN flows barrier. The
45s caller deadline was sized against a single 30s barrier, which is right for
one instance and wrong for the N a spread launch issues at once — the third
request waits behind two full barriers and times out. That one subscription now
overlaps its handlers under a bounded semaphore. Each barrier still syncs after
its own write, so the guarantee is unchanged while N barriers collapse into
roughly one.
With the requests overlapping, every barrier still ran to its alarm.
install-node.sh computed OVN_REMOTE after building the database nodes and gave
it only to compute nodes, so the database nodes were configured without
--ovn-remote. setup-ovn.sh writes ovn-northd's NB/SB connections from that list
and skips it on the localhost default, and it already documents what follows: a
northd dialling only its local member stops advancing nb_cfg once raft
leadership moves, wedging ovn-nbctl --wait=hv. The list is knowable before any
node is touched, so it is computed first and passed to the database nodes too.
Nothing reported an OVN fault while this was happening. Every chassis was
registered, ovn-controller and IPsec were up, and the raft cluster was healthy.
The only signal was the barrier logging that it had overrun and was continuing
without confirmation — which reads like slowness and was a permanent wedge.
* feat: sweep block sizes in the disk benchmark and add an iperf3 network benchmark
* fix: use iperf 2 with CSV output and report why a remote start failed
* fix: stop pkill matching its own shell and verify the iperf server is listening
* ci: run the predastore throughput benchmark at the end of the nightly
* ci: boot the ISO in the nightly, single-node and three-node
* ci: use the existing CI node password secret in the ISO cells
* ci: report free space before the benchmark instead of failing inside warp
* ci: keep the benchmarks and node disks off tmpfs
* ci: export the work directories from RUNNER_TEMP rather than job env
* ci: run the ISO and benchmark workloads from the nvme scratch area
* perf: run guest NICs on vhost-net with multiqueue
Guest-to-guest VPC throughput measured 1.68 Gbit/s single-stream against
21.8 Gbit/s host-to-host on the same underlay, and did not improve with
more streams. QEMU's -netdev tap defaults vhost to off, so the whole
datapath ran in the main loop thread in userspace: that thread sat at
59-65% CPU while all four vCPU threads idled at ~4%.
Every tap netdev now carries vhost=on, and VPC ENIs additionally run
multiqueue at min(vCPUs, 8) queue pairs. Multiqueue needs all three of
the tap, the netdev and the device to agree, so TapNetDev and NetDevice
derive the netdev options and the 2N+2 MSI-X vector count from one queue
figure rather than leaving callers to keep them in step.
The QMP hot-plug path takes the same options as cold boot, with a test
comparing it against the rendered cold-boot strings so the two cannot
drift. MMIO machines stay single-queue: they have no MSI-X.
* ci: check out the sibling repos the go workspace needs in the ISO cells
* ci: skip cell-21 workload cleanup when the workload never ran
* ci: size the cell-21 node at 8 GiB and keep its serial log
* ci: keep the cell-21 node diagnostics alongside the serial log
* fix: let the daemon open /dev/vhost-net and delete multi_queue taps
Enabling vhost=on broke instance launch two ways. DevicePolicy=closed gates
the cgroup device list before capabilities are consulted, so QEMU's open of
/dev/vhost-net returned EPERM despite CAP_DAC_OVERRIDE; add the device to the
allow list and bump the unit version. Separately, `ip tuntap del` re-opens the
device with TUNSETIFF and fails with EINVAL on a multi_queue tap, leaking it on
every teardown; route deletion through `ip link del`, which is flag-agnostic.
* perf: default guest NICs to a single queue
Multiqueue measured a net loss. Spreading one flow across queue pairs reorders
packets and TCP reads that as loss: a 4-vCPU guest fell from 1.76 to 1.50
Gbit/s, with cwnd collapsing on spurious retransmits. The host overlay funnels
a node pair onto one softirq core, so the extra queues have no headroom to win
back what the reordering costs. Keep the plumbing and its tests behind
MultiqueueNICs so re-enabling is one line once that funnel is fixed.
vhost-net stays on unconditionally; it moves the datapath off the QEMU main
loop, measured at 60% of a core under load against 3% with it.
* fix: raise the UDP receive buffer for predastore's QUIC transport
Predastore carries blob, meta and raft over QUIC and quic-go asks for a 7MiB
receive buffer. Left at the kernel default of 208KiB it silently dropped
datagrams under load: the four-node cluster had shed 80k-125k each, which
failed shard writes, put raft into repeated elections, and ultimately handed
guests an I/O error that cost two of them their root filesystems.
Adds a sysctl stage writing /etc/sysctl.d/99-spinifex-net.conf, so the setting
lands in the ISO image and on every node push rather than living in a runbook.
* ci: give the cell-21 workload an identity the node accepts
* feat: tie the guest MTU and NIC multiqueue to ipsec_enabled
ESP costs 34 bytes on top of geneve's 58, so a cluster running the overlay in
plaintext can advertise 1442 instead of 1408 — worth roughly 10% of guest
throughput. Derive it from network.ipsec_enabled rather than hardcoding the
conservative figure.
Multiqueue moves with the same flag, because its sign depends on it. Behind
IPsec it costs ~12%: ESP has no L4 ports, so RSS funnels a node pair onto one
core and extra queues only reorder packets, which TCP reads as loss. Without
IPsec the sending host's single vhost thread saturates first and queues are
worth +54%. IPsec on with multiqueue on is the worst of the four combinations,
so it stops being a standalone knob and becomes vm.Deps.MultiqueueNICs.
The reconciler only ever created a subnet's DHCPOptions row and never revisited
it, so the create-time MTU stuck for the life of the subnet and this change
would have applied to new subnets only. Worse in the other direction: turning
IPsec back on would have left subnets advertising 1442 over ESP, which drops
large segments with no ICMP to recover from. applySubnets now diffs and updates
in place.
reconcile.Config takes IPSecDisabled rather than IPSecEnabled so the zero value
is the conservative MTU — a caller that forgets it is merely slower.
* docs: add an install-time performance tuning guide
Measured on the four-node cluster rather than inferred: the UDP receive buffer
sysctl, what IPsec actually costs and why, the multiqueue sign flip, the overlay
MTU budget, and the jumbo-frame check that must precede enabling it.
A starter doc — several sections end in an open question rather than a
recommendation.
* ci: keep the cell-21 node alive for the workload step
* ci: collect the cell-21 qemu log and widen its timeout
* feat: derive the guest MTU from the fabric, and deepen the virtio NIC config
The overlay overhead is passed on to the guest rather than hidden from it.
Hiding it means running the fabric far enough above 1500 that a standard guest
MTU fits in the headroom, which is what the large public clouds do — but they
own the switches. On a fabric we do not control, the honest figure is the one
the path can carry, and a jumbo underlay should reach the guest: 9000 gives
8942, or 8908 with IPsec. network.underlay_mtu is the lever; SubnetMTU spends
it. Configured rather than probed, because a node's own NIC MTU says nothing
about the switch between it and its peers.
host_mtu sets VIRTIO_NET_F_MTU so a guest that never takes a DHCP lease —
statically addressed, or IPv6-only — still learns the ceiling.
rx/tx_queue_size raise the virtio ring from QEMU's default 256. That default is
a packet-rate ceiling rather than a memory saving: a vCPU preempted for a
fraction of a millisecond fills all 256 slots and the overflow is dropped.
ethtool is the only way to read NIC offload, ring and channel state, so a node
without it cannot be diagnosed for throughput at all.
* ci: give the cell-21 workload its own guest key
* fix: set the guest MTU on the tap, not just in the guest
The tap was created and brought up without an MTU, so it stayed at the
kernel default of 1500 while DHCP and host_mtu told the guest something
else. Harmless at 1442; it silently blackholed every large frame on a
jumbo underlay, with no ICMP back because the drop is at the first hop.
TapSpec.MTU comes from the same guest overlay MTU the DHCP option and
host_mtu already use, and is applied in the bring-up call so a size the
fabric cannot carry fails the launch instead of degrading quietly.
* fix: stop setting tx_queue_size, which QEMU caps at 256 for tap
QEMU only honours a tx_queue_size above 256 for vhost-user and
vhost-vdpa backends. On a tap it rejects the value outright rather than
clamping it, so every launch died with "must be a power of 2 between
256 and 256" and the instance went straight to terminated.
rx_queue_size is unaffected and stays at 1024. A test pins the absence
of tx_queue_size across machine types and queue counts.
* docs: lead the tuning guide with the overlay segmentation finding
The guide's headline was a 5.4x that came almost entirely from turning
IPsec off, and it treated MTU and the fabric as the main levers. The
host reaches 92.6 Gbit/s at MTU 1500 on the same NIC, so the fabric is
not the constraint and the encrypted default is not the thing to tune
away from.
Adds the host-versus-guest comparison up front, documents
network.underlay_mtu, and makes the missing segmentation offload the
first open question.
* ci: stop deselected matrix cells failing in teardown
* docs: quantify what disabling overlay IPsec is worth
The guide described the trade without the numbers an operator needs to
make it. Adds the measured 4.9x, notes that the MTU and multiqueue
changes follow ipsec_enabled rather than being separate knobs, and
records two transition hazards: a stale wide MTU on running guests, and
RunInstances failing while the OVN flows barrier catches up with the new
SAs.
* ci: collect and retain the e2e benchmark and cluster logs
Cells 21 and 22 both copied two paths that only ever exist on the nodes, so
the benchmark JSON was never uploaded and cell-22 collected nothing at all.
Both now gather what they actually produce, and keep it for 90 days.
* fix: quieten spx by installing an explicit CLI logger default
spx never set a process logger, so it inherited whatever a linked package
left behind — until now a JSON handler at Info on stdout, interleaving
library log records with command output and breaking piped use.
Execute() claims the default before any command runs: text on stderr,
LevelError, so commands report failures through their own stderr writes and
libraries stay quiet. --verbose/-v raises to info and SPX_LOG_LEVEL sets an
arbitrary level, the flag winning over the environment. Service subcommands
still call initTelemetry, which replaces the handler outright.
No root --debug flag: viperblock, qemunbd, nats and awsgw each define their
own --debug with a service-specific meaning, and a root persistent flag would
shadow inconsistently depending on the command.
The guard test samples the handler at package-init time rather than inside
the test, because tests in this package start services that install their own
logger legitimately.
* feat: private /admin/CreateAccount for self-service tenant signup
Add POST /admin/<Method> to the AWS gateway: SigV4-signed, JSON in and
JSON out, callable only by an IAM user in the super-admin account holding
spinifex:<Method>. CreateAccount is the one method, so a signup form can
provision a tenant without an operator shell. No new service, port, unit
or certificate — it mounts inside the gateway's existing authenticated
group and inherits its auth, throttling and tracing.
Provisioning is resumable rather than transactional, which fits stores
that have no transaction. Two CAS creates carry the correctness: the
client token admits one in-flight attempt and makes a lost response
replayable for 24 hours, and the name reservation makes duplicate account
names impossible. Every step after the account exists is create-if-absent,
so a retry with the same token finishes the job instead of failing.
The CLI and the endpoint share one provisioning helper, which closes a
gap on both at once: CreateAccount never checked names, so running
`spx admin account create` twice produced two accounts that differed only
by ID.
The response waits for the account's default VPC over a new
ec2.EnsureDefaultVpc request/reply, rather than trusting the existing
fire-and-forget iam.account.created publish — credentials for an account
that cannot launch anything are worse than a slower call.
[signup] max_accounts bounds how many accounts may exist; absent means
uncapped, which is what every cluster does today.
* fix: stop the awsgw template hardcoding one deployment's console URL
console_url shipped as https://console.spx3.com, so every fresh install —
including an on-premise cluster that will never run signup — was
configured to point new accounts at our console. It is the only
deployment-specific domain in any template. It now ships empty, and the
gateway warns when it provisions an account without one, since a blank
sign-in link is otherwise visible only to the customer who receives it.
Extract the remote client from the command body while here. The signing,
request shape and error decoding are the part a Cloudflare Worker
mirrors, and as a function they are testable against a TLS test server
rather than only by hand against a live cluster.
* refactor: drop console_url from signup and default max_accounts to 128
The gateway never read console_url — it passed the value straight through to
the response, and the idempotency record froze it for 24 hours so a corrected
typo still replayed stale. The client that made the request already knows
where it wants to send the new tenant.
max_accounts is now a pointer in the parsed config so an absent [signup]
section takes the 128 default while an explicit 0 still means uncapped.
* feat: run the predastore fault-injection harness as cell-24
Cell-23 asks how fast predastore is when every node is healthy; cell-24 asks
whether it survives one that is not. It runs after cell-23 rather than beside
it: both want a quiet single runner, and a benchmark sharing a machine with a
fault injection run measures the wrong thing.
The port offset keeps the two clear of each other even so, and the job clears
stale nodes on entry and exit, since a run killed before its cleanup trap
leaves one holding ports that fail the next run somewhere unrelated.
* ci: keep the throughput benchmark's output with its results
The run log only existed in the job log, so the artifact it uploaded could not
be read without going back to the run that produced it.
* ci: keep the fault-injection run log with its results
events.txt carries the timeline and the FAIL line, but not the build output or
anything Warp wrote before the harness took over, so a failed cell could only
be read from the job log it was printed into.
* fix: serve the cluster region to the console instead of hardcoding it
The console hardcoded ap-southeast-2 as its SigV4 region in seven files, so on
any other cluster the credential scope was wrong and the gateway rejected every
signed request.
The SPA is embedded in the binary, so a build-time value would pin one binary to
one region. The UI server now serves GET /api/config with the region, no-store
so a proxy fronting several clusters cannot serve one the other's region, and
the SPA loads it once before render.
The region comes from the serving node's config because awsgw pins signature
verification to that same field. Reading the cluster-wide [aws] region instead
would let a cluster whose two keys disagree pass review and still fail to
authenticate.
Also relabels the login card as Spinifex credentials, since the fields are not
AWS ones, and hides the CA install banner unless the console is reached
directly: behind a proxy with a publicly trusted certificate there is nothing
for the cluster CA to fix.
* fix: say once when the host firewall will not be maintained
Both quiet outcomes of the reconcile — no cluster config, and the policy
switched off — return nil without logging, so a node that never arms is
indistinguishable in the journal from one that armed on its first pass.
* test: arm the host firewall before asserting on its peer set
Only the ISO install arms the policy; every other path writes off to the
mode file, so the daemon disables the firewall and never writes a peer
set. The test read that file as its baseline, which it could never have.
It now switches the mode on for its own duration and puts it back, waits
for the peer set the way install-node.sh does, and on failure reports the
mode, the node count and what the daemon said rather than just the
missing file.
* test: move the console region test to the external cmd package
It was in package cmd, which test-package-check rejects, so preflight
failed on the branch. The hook follows the pattern already in
export_test.go.
* test: make the reboot EIP diagnostic capable of naming the drop
The two ofproto/trace probes injected packets with no TTL, so OVN's router
killed them at lr_in_ip_input and both traces ended early for a reason that
had nothing to do with the failure under investigation. A keyword filter then
stripped the table-match lines, hiding where each trace stopped.
Traces now carry a TTL and are kept whole, the real datapath megaflows for the
guest reply are dumped alongside them, and any drop matching the guest source
is resolved from its OpenFlow cookie back to the Southbound logical flow that
installed it.
* fix: exempt an EIP instance from the subnet drop gate with allow, not reroute
A subnet with no route of its own carries a priority-1100 drop policy, and an
instance with an external IP is lifted above it by a priority-1200 policy. That
policy rerouted to the pool gateway, which in distributed NAT is outside every
logical router port subnet, because the gateway LRP is the link-local /30. The
static default route survives that by naming an output port; a routing policy
has no such field, so ovn-northd discarded all three of them and the drop gate
below swallowed everything the instances sent.
The effect was that every external IP answered inbound traffic and none of it
came back: the SYN reached the guest, the guest replied, and the reply died in
the router's policy stage. Guest-initiated egress was lost the same way.
The policy only ever needed to lift the gate. The router's default route already
points at the gateway, so allow leaves the routing decision intact and gives
northd nothing to reject.
* test: match the log the firewall quorum gate actually emits
The assertion waited for "chassis encap addresses; not writing a partial peer
set", which no code has ever logged. The reconcile reports a short chassis list
as "OVN reports N of M chassis encap addresses" and the maintain loop carries
that into its retry warning, so the wait could only ever time out.
Arming the node in the previous commit got the test as far as this assertion for
the first time, which is why it surfaced now. The failure also prints the
journal tail and the same inputs the earlier wait reports.
* fix: stop one unreadable volume document failing every volume listing
A single volume document whose shards no longer reconstruct made
DescribeVolumes, DescribeVolumeStatus and DescribeVolumesModifications
fail for every account, because the listing returned the first fetch
error rather than skipping the document.
ListVolumes now skips a document it cannot read or decode and logs the
skip, mirroring ListAMIs. Callers whose answer would be wrong rather
than merely partial take ListVolumesStrict: the orphan scan and the
snapshot in-use check both gate destructive work on a complete list.
The tolerance covers fetch failures as well as decode failures. An
object whose shards no longer join is as unusable as one whose bytes
will not parse, and it was the fetch path that took the cluster down.
* feat: pull Ochre model weights from a hub into predastore
Add `spx admin ochre weights pull`: fetch a self-host model's weights from
Hugging Face directly into predastore, producing a prefix `weights stage`
consumes. Streams safetensors-only artefacts (plus config/tokenizer and the
shard index), refusing pickle/.bin. Pins an immutable commit SHA, writes an
ochre-pull.json manifest that stage records as SourceRevision, and aborts clean
(removing any written object) on a gated or mid-stream failure. Token resolves
from --hf-token, HF_TOKEN, or a stored platform credential; never logged.
Also add `spx admin ochre credentials set` to store an encrypted provider
credential from stdin, and SourceRevision on the weights KV record + list.
* fix: stream hub downloads without a whole-request timeout
http.Client.Timeout bounds the entire request including the body read,
so a multi-GB safetensors shard was guillotined mid-download at 30s. Bound
dial and response-header latency via the transport instead, leaving the
body free to stream for as long as the server keeps sending.
The InvokeModel OUTPUT guardrail hook logged and forwarded the model response
unguarded when the per-family completion extractor could not parse the body, so
output enforcement failed open while the INPUT path already failed closed.
Both the non-stream OUTPUT hook and the buffered InvokeModelWithResponseStream
source now fail closed: on an unparseable completion (non-stream) or an
undecodable assistant chunk (stream) they load the guardrail view and emit a
blocked response carrying BlockedOutputsMessaging. A genuinely empty completion
(valid chunks, no assistant text) still passes through. Shared resolve/load
prologue extracted into loadGuardrailView.
Self-host InvokeModel/InvokeModelWithResponseStream picked the adapter by tier
and always used the Llama adapter, silently mis-serving a non-Meta model through
Llama's native request schema. Dispatch now selects by catalog Family; an
unhandled self-host family is refused with ValidationException, not mis-served.
Endpoint-resolution failures hardcoded ServiceUnavailableException, discarding
the AWS code the daemon put on the wire. All four self-host resolve sites now
resolve the carried code (ModelNotReadyException, ThrottlingException, ...) and
fall back to ServiceUnavailable only for an un-coded transport failure.
* feat: ochre-vector list-jobs subject
Adds ochre.vector.listJobs to the daemon-side VectorService surface (both
the in-process and NATS-forwarding implementations) so a caller can list
every ingestion job for an account rather than looking one up by id.
Widens jobGetter to expose JobStore.List, registers the subject on the
daemon's post-connect subscription table, and updates the CLI test
double to satisfy the widened VectorService interface.
Also carries the jackc/puddle/v2 indirect dependency go.sum was missing.
* feat: bedrock-agent knowledge-base + data-source CRUD
Adds the emulated AWS bedrock-agent control-plane gateway service:
KnowledgeBase/DataSource CRUD, StartIngestionJob/GetIngestionJob/
ListIngestionJobs, backed by new gateway-owned KBStore/DataSourceStore
JetStream KV persistence (mirroring registry.go's CAS idiom) and the
existing ochre-vector daemon surface for the bound vector index.
One knowledge base binds exactly one vector index. embeddingModelArn
maps to EmbeddingModel/Dimension; storageConfiguration and roleArn are
accepted and stored verbatim but never acted on. CreateDataSource maps
S3 + fixed-size chunking config onto a SourceSpec, converting AWS's
maxTokens/overlapPercentage into internal ChunkSize/ChunkOverlap.
DeleteKnowledgeBase cascades to every data source under it before
dropping the bound index. Wires BedrockAgentKB/BedrockAgentDataSources/
BedrockAgentVector into the gateway and registers bedrock-agent in the
supported-services and dispatch tables.
Runtime (Retrieve/RetrieveAndGenerate) is out of scope here.
* test: cover bedrock-agent routing and KB/DataSource store edges
Adds end-to-end HTTP tests for BedrockAgent_Request's route table and
per-route handler closures (all eleven operations, plus unknown-route/
missing-account/nil-dependency error paths), KBStore/DataSourceStore
PurgeAll and no-JetStream-client error paths, and ListJobs coverage
(vectorService, NATS round trip, per-account scoping) in the existing
ochre-vector API test suite.
* feat: bedrock-agent-runtime retrieve + retrieve-and-generate
Adds the emulated bedrock-agent-runtime data plane: Retrieve resolves a
knowledge base to its bound vector index and runs a similarity query;
RetrieveAndGenerate runs the same retrieval, renders a RAG prompt (default or
caller-supplied template), and calls Converse in-process for generation,
returning citations built from the retrieved chunks and a session id.
Recovers the full RetrievalFilter operator set from the raw request body
since the vendored aws-sdk-go RetrievalFilter type only carries andAll/orAll,
and fixes GetService never routing real signed bedrock-agent traffic (only
bedrock-runtime and bedrock had discriminator rules; bedrock-agent's own
knowledgebases paths had none).
* fix: bedrock-agent KB delete idempotency + ingestion-job ownership
CreateKnowledgeBase: a rollback DeleteIndex failure after a kb.Create
claim failure is now logged at Error level with the orphaned index id
instead of a bare Warn, while the caller still sees kb.Create's own
error unchanged.
DeleteKnowledgeBase: a bound index that DeleteIndex reports already
gone (ErrIndexNotFound) no longer aborts before the KB record itself is
removed, so a KB never dangles pointing at an index that no longer
exists. Any other DeleteIndex error still aborts the cascade.
GetIngestionJob: now verifies the addressed dataSourceId owns the job
(Bucket+Prefix match against the data source's own SourceSpec, same
best-effort check ListIngestionJobs already used), not just the
knowledge base's bound index -- closing a path where a job could be
read through the wrong data source id.
* fix: default bedrock-agent-runtime retrieve to 5 results (AWS parity)
* fix: thread DataSourceID through ingestion jobs
JobRecord and IngestRequest now carry an exact DataSourceID instead of
relying on Bucket+Prefix as a best-effort proxy. StartIngestionJob
stamps it from the DataSource record; GetIngestionJob and
ListIngestionJobs filter on it exactly, treating a job with no
DataSourceID as belonging to no bedrock-agent data source.
* fix: purge KB/DataSource records on appliance teardown
Appliance.Teardown purged the index registry and job store but left
the gateway-owned KB/DataSource KV records dangling, pointing at a
deleted index. WithKBStores attaches those stores the same way
WithStores does for registry/jobs, built before Connect so teardown
still works on a broken appliance, and Teardown purges them too.
The watchdog's is-active guard cannot fire against spinifex-nats.service
(Type=simple reports active the instant the process forks, long before
JetStream replays its state), so it restarted a NATS that was still
recovering. Separately, any non-zero exit from spx js-probe was read as a
JetStream write failure, so a broken spx binary read as a cluster-wide
outage instead of an unrunnable tool.
Replaces the is-active guard with an age gate on
ExecMainStartTimestampMonotonic, adds a post-restart cooldown and a
restart-count escalation that gives up rather than looping, and teaches the
watchdog to treat js-probe exit 2 (and any exit code that isn't 0 or 1) as
inconclusive rather than as proof of failure. Bounds the unit's own start
with TimeoutStartSec=120, matching the existing TimeoutStopSec derivation.
The predastore health probe QUIC-dials the raft meta node, but a single-host
install binds no cluster-plane socket — colocated nodes talk over an in-process
pipe. The daemon runs in a separate process from predastore, so it can never
reach that pipe and the dial is always refused, reporting a permanent false
unreachable that gates ansible deploy readiness.
On a single-host install, probe predastore's one cross-process listener — the
S3 gate — instead. A gate that accepts a connection is a predastore process
serving, and on a single voter raft is trivially leader. Multi-host installs
keep the raft QUIC probe unchanged.
* fix: exit non-zero when a spinifex service fails to start
Every service start/stop subcommand in cmd/spinifex/cmd/service.go used
Run funcs that printed an error and returned, which exits 0. Systemd's
Restart=on-failure never fires on a zero exit, so a fatal startup error
(seen live: viperblockd's volume lease bucket lookup timing out during a
cold JetStream start) left the node permanently down until a manual
systemctl restart.
Converts every fatal error path across predastore, viperblock, qemunbd,
nats, spinifex, awsgw, spinifex-ui, vpcd, northstar and qmp-collector to
RunE returning an error, following the RunE + SilenceErrors/SilenceUsage
pattern the northstar start command already used in this file. root.go's
Execute() already prints the returned error to stderr and calls
os.Exit(1), so this centralizes the exit-code decision instead of
repeating fmt.Fprintln+os.Exit at each call site. Pure status subcommands
with no fallible operations are left as Run.
Adds a subprocess re-exec test proving a fatal predastore start now
exits 1 end to end, plus in-process RunE tests covering each service's
start command.
* test: cover the service startup validation branches
The Ochre vector store change imported github.com/jackc/pgx/v5/pgxpool
without running go mod tidy, so github.com/jackc/puddle/v2 was absent
from go.mod and go.sum. Any build under -mod=readonly failed, breaking
the multi-node E2E job at 'go test -c'. Also promotes pgx/v5 from the
indirect block, since it is imported directly.
* feat: add /v1/embeddings gateway adapter
* feat: Ochre vector store pgvector backend, KV index registry, orchestration
Adds handlers_ochrevector: the VectorBackend interface + pgx/v5 impl
with per-account schema/role isolation enforced via SET LOCAL ROLE +
search_path inside every transaction, the KV index registry
(CREATING/READY/DELETING/STALE), and the create/delete/list/reconcile
orchestration tying them together with crash-safe rollback and
re-entrant recovery of stuck records.
* feat: singleton appliance-ensure state machine for ochre vector store
Adds the encrypted control-plane credential store and re-entrant
Ensure state machine for the platform pgvector appliance (D2): a
fixed-key KV claim mirroring Registry.Reserve elects exactly one
winner to launch the appliance via an injected ApplianceLauncher,
losers wait or resume a stale crashed provision, and the master
password is generated once, stored only AES-256-GCM encrypted, and
decrypted solely inside Connect to build a pgxBackend.
* feat: Ochre vector store async ingestion (chunker, job store, worker)
Adds Stage 3: a deterministic recursive chunk splitter, a per-account
ingestion job-record store mirroring the index registry, the
transactional per-document replace on the backend, and the
IngestService worker that lists a predastore source, chunks and
embeds each object, and replaces its rows idempotently.
* feat: Ochre vector store similarity query and metadata filter grammar
Adds VectorBackend.Query (cosine kNN, k default/cap, iterative HNSW
scan under a filter) and a Bedrock-KB-shaped metadata Filter AST that
compiles to parameterized SQL over jsonb, closing off arbitrary SQL by
allowlisting jsonb keys and always binding values as params. Also
fixes RunJob reporting READY on a job that ingested nothing regardless
of failure cause, and stamps SourceSpec.Metadata onto ingested rows so
the new filters have data to match against.
* feat: Ochre vector store request/response API, VectorService, and CLI
Adds the daemon-facing VectorService interface (index CRUD, ingest,
describe job, query), its server implementation over the existing
registry/backend/embedder, a NATS-forwarding client, and the
`spx admin ochre vector` operator CLI. Query resolves and embeds
against each index's own pinned model; ingest stamps the same
model/dimension onto every source spec server-side. accountID is
always taken from the caller context, never the request payload.
Daemon subscription wiring and manifest registration are a later
change; this is pure request/response plumbing with no construction
into daemon.go or awsgw.go.
* feat: wire Ochre vector store VectorService into the daemon
Constructs the platform Postgres appliance and VectorService behind a new
OchreVector config section that defaults off, so existing daemons are
unaffected. Adds a real RDS-backed ApplianceLauncher that provisions the
appliance over rds.CreateDBInstance/DescribeDBInstances, registers the six
ochre.vector.* NATS subjects mirroring the bedrock endpoint service pattern,
and degrades to a disabled (nil) service on any boot-time appliance failure
rather than blocking daemon startup.
* fix: fix Ochre vector store boot ordering and self-register subjects
startOchreVector ran before subscribeAll, so its appliance launch fired
rds.CreateDBInstance/DescribeDBInstances requests with no responder yet
registered, timing out the launch on a first-enable boot. It also ran
synchronously, blocking daemon startup up to the appliance launch timeout.
Move it to a background goroutine started after subscribeAll returns, and
have it register its own ochre.vector.* subjects once the VectorService is
actually constructed, via a new registerNatsSubs helper shared with
subscribeAll's boot-time table. Guard d.natsSubscriptions with d.mu, since
this background registration can now race the shutdown path's
unsubscribe-all sweep.
* feat: bake pgvector into the shared rds-postgres image
* fix: reach the ochre appliance over a routed daemon port on its ENI IP
The daemon dialled the RDS appliance over rec.Endpoint, a vanity spx3.net
hostname the control-plane host can neither resolve nor route to, so a healthy
appliance was never reachable and the vector store disabled itself.
Resolve the appliance's customer-ENI private IP read-only via
ec2.DescribeNetworkInterfaces filtered on the spinifex:rds-db-instance tag, dial
that IP for the DSN instead of the hostname, and give the daemon its own routed
ENI in the appliance's subnet bound to an OVS host port so the IP is reachable.
Retry appliance startup with bounded backoff instead of hard-disabling on the
first transient.
* fix: install pgvector at bootstrap for the Ochre vector appliance
The daemon connects to the appliance as its NOSUPERUSER master, and the
image's pgvector is not a trusted extension, so CREATE EXTENSION vector
failed with permission denied. rds-init now installs it via the bootstrap
superuser, gated on the appliance's fixed master name so tenant instances
are unaffected.
* fix: grant the Ochre appliance master CREATE on its database
The daemon's EnsureAccount runs CREATE SCHEMA per account as the NOSUPERUSER
master, which does not own the postgres database, so bootstrap now grants it
CREATE on the database alongside the pgvector install. Both are gated to the
appliance master, so tenant instances are unaffected.
* fix: make the Ochre appliance's pgvector reachable under per-account isolation
The daemon connects as a NOSUPERUSER master and runs every account operation
under SET ROLE with a search_path pinned to the account schema. Two gaps blocked
that against a real non-superuser Postgres: the master could not SET ROLE into
the roles it creates, and the account role could not see the vector type.
Bootstrap now grants the appliance master createrole_self_grant and installs
pgvector into a read-only extensions schema granted to PUBLIC; the backend
carries that schema on each account search_path so the vector type resolves
without weakening per-account write isolation. All appliance-scoped.
* fix: prune stale VPC host port before installing the daemon appliance port
A replaced Ochre appliance is handed the same subnet IP, so a new daemon ENI
mints a fresh vhp- port at an address a superseded port still holds. The old
port survives at the OS/OVS layer even after its ENI record is wiped, so two
interfaces claim one address and the kernel can route to the dead one
(EHOSTUNREACH). installVPCHostPort now prunes any other vhp- port holding the
target address before installing.
* fix: gate VM restore's stale-socket kill on backing-store readiness
On daemon start, a live VM whose NBD sockets fail to dial was force-killed as an
orphan. While predastore/viperblock is still coming up after a restart those
sockets are transiently unreachable, so a healthy VM (the Ochre appliance was
the observed victim) was destroyed and its RDS record flipped to failed. The
kill now only fires when the backing store is confirmed up; otherwise the live
QEMU is kept and QMP reconnected. Genuine orphan reaping under a healthy store
is unchanged.
* fix: drive ochre ingest jobs from a daemon-side sweep scheduler
* feat: add operator teardown verb for the ochre platform appliance
Adds Appliance.Teardown to remove the singleton platform Postgres
appliance (RDS instance, host port, KV record), an admin-only NATS
subject and daemon handler to drive it, and `spx admin ochre appliance
teardown --confirm` to invoke it without exposing the action through
the tenant awsgw surface.
* fix: register ochre appliance teardown before Connect so it survives a broken appliance
* fix: purge ochre index and job metadata on appliance teardown
* test: mark ochre in-package test files for the external-test gate
/health reported service_health.predastore == "ok" for every node simply
because predastore was named in spinifex.toml, without contacting it — on
env19 all three nodes stayed green for the whole time raft had no leader and
volume creates were failing. A probe registry now backs /health's predastore
entry with a real raft status dial (ok/no_leader/unreachable, cached briefly
and bounded by a short timeout), while every other service keeps its
unprobed "ok" default unchanged.
spx admin storage status had the same root problem from the other side: it
dialed HTTPS /health and /status endpoints predastore's meta nodes never
served (they speak an opcode-multiplexed RPC), so every meta node always
reported unhealthy with blank raft fields. It now dials the same raft status
RPC predastore#82 exposes, verified against the cluster CA, with the JSON
output shape unchanged.
checkSocketScope compared a socket's bind address to node.Addr and called
it the node's WAN address, but node.Addr is the cluster peer address
(harness.Node's own doc). On CI's single-homed tofu nodes this is the
node's only address, so every Cluster-scope service failed no matter what
it bound. Compare against a positively-known SPINIFEX_WAN_IP instead,
distinct from the peer address, and record via harness.Detail when no
such address is available so the skipped check is visible rather than
silently passing.
* fix(e2e): resolve the listener inventory without a Go toolchain on the target node
tests/e2e/multinode/listener_invariant_test.go resolved
docs/security/network-connections/README.md at runtime via `go env
GOMOD`, which works under `go test` but not inside a compiled .test
binary shipped to a node with no Go toolchain. Embed the doc in a new
docs/security/network-connections package instead, so the content is
compiled into the binary and every caller parses bytes rather than a
resolved path. RepoRoot and DocPath are now unused and removed.
* docs: trim the embed package comment to the three-line standard
* fix(e2e): kill the node-failure phase abruptly instead of draining it
Phase 8 used `systemctl stop spinifex.target`, which now drains guests
gracefully since the target-stop drain fix landed, so it tested an
orderly shutdown rather than a failure. Since a target unit has no
process of its own and cannot be signalled, and PartOf= only
propagates stop/restart jobs rather than kill, SIGKILL each core
service's main process directly instead, leaving orphaned guest
qemu/nbdkit processes for recovery to reattach.
Also fix a stale comment on spinifex-predastore.service's
TimeoutStopSec: it claimed to match viperblock's drain budget, which
is now 150s rather than 30s. The value was already correct; only the
rationale was outdated.
* fix(e2e): hold node2's outage open through Phase 8's assertions
Restart=on-failure was respawning each killed unit within ~5s, so
every Phase 8 assertion ran against an already-recovered node2 and
DEGRADED_PEERS never actually dropped. Stop the same units right
after the kill to cancel the queued restart and keep them down for
the rest of the phase, and make the NATS degraded-peer count a real
assertion (sized off NODE_COUNT) now that it can genuinely fail.
* test: add the listener invariant static guard and e2e runtime check
Adds both halves the plan called for: a static test in
spinifex/network/invariants that scans install scripts and config
templates for a bare OVSDB ptcp remote, a wildcard bind on a
Cluster/Encap port, or a bind port absent from the inventory, and an
e2e test in tests/e2e/multinode that asserts the same on live ss
-tulnp output. Both read
docs/security/network-connections/README.md "## 1. Inbound
Listeners" as their fixture through a shared, non-test parser package
(spinifex/network/listenerinventory), so a wildcard exception has to
be declared in the doc's own prose rather than hardcoded in either
test.
* fix: make the listener wildcard exception fail closed on an explicit marker
The prior heuristic granted an exception for any unnegated "wildcard"
or "0.0.0.0" mention within 40 characters of the port's row. That
fails open: a future row phrased as "avoids the wildcard" or with a
negation further upstream than the window would silently exempt
itself. Replace it with a single exact literal phrase, "binds the
wildcard by design", that a row must contain to grant the exception —
no other wording counts, so an unrecognized mention denies rather
than grants. Update the two rows that need it (5300, 500/4500) and
document the marker's role in the inventory's own prose.
* docs: record that the Geneve tunnel socket binds the wildcard by design
* fix: reap orphaned nbdkit processes recovery cannot adopt
recoverMountedVolumes already rebuilds MountedVolumes from surviving
nbdkit processes, but an unadoptable backend (buildVB fails, most
commonly a deleted volume) or a duplicate corroborated process for an
already-adopted volume was logged and left running forever, with no
later pass reconsidering it.
Reap both classes, but only when a scan of every other process's
/proc/<pid>/cmdline on the host finds no reference to the backend's
NBD endpoint -- its socket path, or its 127.0.0.1:<port> pair, checked
both joined and split across server.host=/server.port=. A referencing
process, or a scan that could not complete, both leave the process
running and log loudly instead, since signalling the wrong process
destroys a running guest's data path.
* fix: treat an unreadable /proc candidate as inconclusive, not absent
endpointReferencer skipped any os.ReadFile failure on a candidate's
cmdline, which silently counted a read it could not perform as
evidence the candidate does not reference the endpoint. Under a /proc
mounted with hidepid, a guest QEMU owned by another uid is invisible
this way, so an in-use backend could be reaped.
Only os.IsNotExist (the candidate exited between ReadDir and the read)
continues the scan. Any other error now returns non-nil from
endpointReferencer, which reapOrphanedNbdkit already treats as an
inconclusive scan: leave the process running, log at WARN.
* fix: stop orphan nbdkit reaping from killing live backends under hidepid=invisible
hidepid=invisible hides another uid's /proc/<pid> directory entry outright,
not just its contents, so a live QEMU referencer (owned by spinifex-daemon)
never appeared in viperblockd's scan at all -- a clean "no referencer" result
that was actually just a hidden one. reapOrphanedNbdkit now detects the
hazardous mount via /proc/self/mountinfo and declines to reap rather than
trust a scan that could be blind to a live guest.
spinifex-viperblock.service drops ProtectProc=invisible (now default, same
tradeoff already accepted by spinifex-qmp-collector.service) so the scan can
actually see spinifex-daemon's processes again once it's safe to trust.
Overwriting /usr/local/bin/spx while spinifex.target is running does not move
the services onto it. Each keeps executing the replaced inode until it
restarts, so a node serves a mixture of two builds, and because the handlers
are NATS queue-group workers the skew shows up as roughly one request in three
behaving like the old build — an intermittent fault rather than a broken node.
The ansible deploy path already detects and remediates this, but only there:
an out-of-band binary copy leaves nothing to report it, and on env19 that
state went unnoticed for a full day. spx admin preflight now reports it, so an
operator has a command to run and a script has a non-zero exit to gate on.
Detection follows the deploy path's: a (deleted) marker on /proc/<pid>/exe, a
target path that no longer resolves, or a path that resolves to a different
inode than the one being executed.
* fix(shutdown): scope a local drain to its own node and stop counting failed phases as done
A node's spinifex-shutdown.service publishes its GATE and DRAIN requests to
the shared spinifex.cluster.shutdown.* subjects, which every daemon subscribes
to as a fan-out. The node filter only decided which ACK the stopping node
waited for, so by the time it applied, every other node had already stopped
its gateway, UI and VPC daemon and powered down its guests. Stopping one node
took down every guest in the cluster.
Requests now carry the node they apply to, and a daemon drops one addressed
elsewhere without ACKing. A cluster shutdown leaves the field empty, so it
still reaches every node.
The STORAGE phase also ran 'pkill -f nbdkit', which matches every nbdkit on
the host: another node's backends on a shared host, and a live guest's backend
that survived a daemon restart under KillMode=process. Cleanup now sweeps this
node's own nbdkit-vol-*.pid files and signals a PID only while /proc still
agrees it is nbdkit, so a recycled PID is never hit.
Finally, the coordinator counted errored ACKs toward its per-phase quorum, so
a node that failed to drain reported N/N and the run advanced into STORAGE and
PERSIST with guests still running. Only clean ACKs count now, and any errored
one aborts the phase unless --force.
* test(shutdown): cover the phase gate, every handler's target check and the real proc paths
Diff coverage came in at 52.9% against a 60% threshold. The gaps were the
per-phase target checks beyond GATE, the real procComm and signalProcess
bodies that every cleanup test stubs out, and the coordinator's abort branch,
which sat inside runClusterShutdown where no unit test could reach it.
The abort decision moves into shutdownPhaseOutcome, a pure function returning
the completed count and the message to print, so the rule that an errored node
does not count toward the quorum is testable on its own rather than only
reachable through a full cluster shutdown.
Image import shelled out to virt-customize to install the deployment CA into
an imported image's trust store. The step was best-effort and skipped whenever
virt-customize was absent, which is the default: nothing in provisioning
installs guestfs-tools on a node. When it did run it mutated the image after
its checksum had been verified, it could not handle Windows guests, and it tied
CA rotation to re-importing every affected AMI.
Nothing the stack launches depended on it. ECS container instances, EKS k3s
servers and BootDirect microVMs all receive the gateway CA at launch via
user-data or fw_cfg, and boot itself needs no CA because IMDS is plain HTTP.
The only uncovered case is a customer instance whose own workload calls a
Spinifex HTTPS endpoint, which needs a pull mechanism rather than an image
mutation.
The IMDS responder now serves the CA at /spinifex/ca.pem, outside the /latest
tree so the AWS-compatible surface stays byte-identical to EC2's. The route is
token-free because a CA certificate is public material and the console already
serves it unauthenticated; the X-Forwarded-For SSRF guard still applies. The
PEM is cached and re-read on modtime change, so rotating ca.pem on a node
reaches guests without restarting vpcd.
* refactor: detach abandoned viperblock engines instead of stopping their goroutines by hand
viperblockd opens an engine for a single operation in ten places and ends it
by stopping the background goroutines, never reaching Close. That leaves the
chunk GC sweeper running on an engine the caller has finished with, and leaves
the volume recorded as still held, so nothing can tell an engine that was
handed over from two engines holding one volume at once.
Replace each with vb.Detach(), which stops all three goroutines and reports
the release. One site keeps its bare StopChunkUploader: the daemon-side VB
that tracks state while the nbdkit plugin owns the data path is not finished
with the volume, and must not report otherwise.
An invariants test scans the package for a goroutine stopped outside Detach,
so the pattern cannot come back one call site at a time.
* chore: bump the viperblock pin to the release carrying Detach
Detach does not exist in the previous pin, so the detach work in this branch
cannot build without it. Targeted go get of one module, no tidy — viperblock
is the only requirement that moves.
* refactor(test): share the IAM ensurer and instance store test doubles
The IAM system-instance-role ensurer was hand-faked in seven packages and a
stopped-instance store in eight types across three, so one contract had many
independently drifting doubles. Add mock packages for both, following the
shape of network/ovn/mock, and use them where behaviour matches.
handlers/iam's own tests move to package handlers_iam_test: the mock imports
handlers_iam, so an in-package test importing the mock would cycle.
The vm package keeps its local state store fakes. Moving them would mean
converting shutdown_test.go, manager_deps_test.go and restore_test.go, which
lean on unexported Manager internals, for no gain to other packages. The
gateway IAM tests keep theirs too: they mock the ~50-method IAMService, a
genuinely different contract.
* fix: keep mock writes out of the claimable stopped pool
WriteStoppedInstance was inserting into Stopped, the map
ClaimStoppedInstance draws from. A caller that claimed an instance,
failed downstream and rolled back by writing the record made it
claimable again, so a concurrent caller could claim the same instance
and both would reach vmMgr.Run. This broke
TestStartStoppedInstance_ConcurrentClaimRace, which the local fake it
replaced had passed: that fake recorded writes only.
Writes now land in WroteStopped alone. Also flips both mock test files
to external test packages to satisfy test-package-check.
* fix(test): restore in-package tests that reach unexported symbols
admin_upgrade_test.go and units_invariants_test.go were moved to external
test packages while, in parallel, validateUpgradeFlags and unitVersionRe grew
references in those same files. Both sides merged cleanly and the result did
not compile, which silenced the systemd unit guards entirely.
Also regenerates units_gen.go, which had drifted from build/systemd: it was
missing spinifex-firewall.service and carried a stale spinifex-daemon.service.
The drift landed unnoticed because those guards were the tests that could not
build.
Note that `go generate ./...` does not regenerate this file. The directive
lives in generate.go, which is //go:build ignore, so go generate never scans
it; the generator has to be run directly from spinifex/systemd.
* fix(systemd): bump the daemon unit version so the firewall path reaches hosts
The nftables firewall work added `-/etc/spinifex/firewall` to the daemon
unit's ReadWritePaths but left the version marker at 1. Reconcile replaces a
unit only when the installed version is lower than the embedded one; same
version with different content is treated as operator-modified and left
alone. An upgraded host would therefore keep its old unit and the daemon
would never be able to write peers.nft.
Bump the daemon unit to version 2 and give the firewall unit a marker of its
own — it had none, so it read as version 0. Pin both body hashes.
* fix(systemd): make go generate actually regenerate units_gen.go
The directive sat in generate.go, which is //go:build ignore, so go generate
never scanned the file and never ran it. Every drift failure told the reader
to run `go generate ./...`, which silently did nothing for this file, and the
units_gen.go drift that hid the daemon unit version bug landed that way.
Move the directive to reconcile.go, which is part of the package and is
scanned, and drop the dead one.
* refactor(test): route duplicated NATS bootstraps through testutil
Thirteen test sites hand-rolled a NATS or JetStream server bootstrap that
spinifex/testutil already provides, and one seedKV clone duplicated
testutil.SeedKV. Route them through the shared helpers.
Sites whose options genuinely diverge are left alone: fixed ports needed for
reconnect and restart scenarios, custom TLS and token auth, a custom
MaxPayload, and TestMain entry points that have no testing.TB for t.Cleanup.
The natgw and routetable seedKV clones use CreateOrUpdateKeyValue rather than
CreateKeyValue, so their failure behaviour differs and they stay as they are.
Also moves eight test files to external test packages where they declare no
helper used by a sibling in the same package. Coverage is unaffected: Go
compiles internal and external test packages into one binary per package, so
attribution is identical either way.
* chore(test): require new test files to use an external test package
Adds a preflight check that fails when a newly added test file declares an
in-package test. External test packages bind tests to the exported API and
let packages share test doubles, which an in-package double cannot be. Only
added files are checked, so the existing in-package tests are untouched.
e2e and integration suites are exempt: both share a package-level fixture
built in main_test.go, so their files cannot move individually. A file that
needs unexported access can opt out with a //test:in-package comment.
The fixture pinned its S3 gate to 127.0.0.1:18443. Three test binaries import
it and go test runs packages as concurrent processes, so two clusters raced
for the port. The loser did not fail cleanly: waitForReady probed the winner's
gate on the same port and reported healthy, so the setup client talked to
another process's cluster and failed with BucketAlreadyOwnedByYou.
Reserve a port per process instead, and reach the gate only through
Fixture.Host so it cannot be hardcoded again. A Start that fails partway now
drains the cluster before returning; its nodes hold process-wide pipe names
that a later attempt would otherwise collide with.
* feat(security): add an nftables host firewall scoping cluster ports to peers
Nodes shipped no host firewall, so every cluster listener — OVN NB/SB, the
formation port, NATS, predastore — was reachable from anything that could route
to the node. On a single-NIC node the planes collapse onto the public address,
which made that the default rather than the exception.
setup.sh installs a policy and an apply helper but does not enable them: the
peer set is unknown until the cluster's planes resolve, and a drop policy with
an empty set would cut the first node that applied it out of its own formation.
install-node.sh writes the peers and enables the unit after formation.
The policy owns only table inet spinifex_filter and replaces it in one
transaction. vpcd writes MASQUERADE and per-EIP FORWARD rules into the ip
nat/filter tables and reinstalls them only on start, so a whole-ruleset flush —
what Debian's nftables.service does on both start and stop — would break
elastic IPs silently. INPUT only, for the same reason: a forward-hook drop here
could not be rescued by an accept in the other table.
Guest traffic keeps working: IMDS and VPC DNS terminate on ime-* ports in the
host netns, and every guest agent reaches its control plane over the AWS
gateway rather than NATS, so no cluster port needs to face the guests.
* feat: reconcile the host firewall peer set from the daemon
The peer set was written by install-node.sh and by cluster-bootstrap.yml. That
missed two of the four formation paths (bm-bootstrap.sh and the ISO firstboot),
duplicated the same logic in each, and went stale as soon as a node was added or
replaced. host.ReconcileFirewall now owns it: every formation path starts the
daemon, and it re-runs on each startup.
A node's own config names its peers by their wan address only, so a set built
from config alone drops every peer's NATS, OVSDB and formation traffic. The
lan plane comes from the generated NATS cluster routes and the OVN remote lists,
and the vpc plane from the Southbound Encap table; an empty encap answer is an
error rather than a narrower set.
ProtectSystem=full applies to the sudo child too, so the root helper could not
write /etc/spinifex/firewall either — it is now in the daemon's ReadWritePaths,
optional so a node installed before the firewall stage still starts.
The apply helper gains set-peers and disable verbs, re-validates every address
as a dotted quad with in-range octets, and rolls the peer file back if the
ruleset will not load.
* fix: retry the host firewall reconcile until every chassis registers
The peer set is only safe to write once OVN reports an encap address for
every node. On a fresh bootstrap the daemon routinely wins the race against
ovn-controller, so a single startup attempt saw a partial chassis list and
either wrote a set that dropped Geneve from the nodes missing from it, or
failed outright and left the node with no policy until a restart.
Gate on the full chassis count and schedule the reconcile instead of running
it once: back off from 15s to 5m while it fails, then re-run every 5m so a
node added or replaced after formation is picked up without a daemon restart.
An unchanged peer set short-circuits before the helper runs.
* test: cover the firewall chassis quorum gate in the multinode e2e suite
A partial chassis list is the normal state early in a bootstrap, and writing a
peer set from one produces a firewall that drops Geneve from every node missing
from it. The gate that refuses that write had no live coverage: a 5-node
bootstrap converges too fast for any daemon to observe a partial list, so the
verification run only ever exercised the OVN-unreachable path.
Reach the branch by inflating the expected count instead of breaking OVN: a
phantom node in one node's config makes the cluster one chassis short, with
nothing in the Southbound DB disturbed. Asserts the gate logs, that no peer
file is written while it holds, that the loaded ruleset survives, and that the
node recovers to a byte-identical peer set once the phantom is removed.
* feat: reconcile systemd unit files on upgrade
Unit files were written once at install time by install_systemd() in
scripts/setup.sh and never re-asserted. A node upgraded by swapping the
spx binary kept whatever units it was installed with, so any fix that
lives in a unit rather than in code never reached it.
This is not theoretical. A node whose spinifex-viperblock.service
predates KillMode=process defaults to KillMode=control-group, so on a
target stop TimeoutStopSec expires mid-flush and systemd cgroup-SIGKILLs
every nbdkit while guest QEMU survives in the daemon's own cgroup: the
guest keeps running with no storage path.
Units are now stamped with a spinifex-unit-version marker, embedded in
the binary via a generated file, and reconciled by spx admin upgrade.
Missing or version-stale units are replaced after a timestamped backup;
a unit at the current version whose content differs is reported as
operator-modified and never overwritten. Drop-ins under <unit>.d are
untouched, so systemctl edit overrides survive.
Nothing is restarted. A daemon-reload is enough for KillMode and
TimeoutStopSec to apply to the next stop, so drift is corrected without
disturbing a running guest.
Adds --dry-run, --units-only and --skip-units to spx admin upgrade.
* test: cover systemd reconcile error paths and the admin upgrade CLI flow
Diff coverage on the unit-migration PR sat at 45.8%, below the 60% gate.
Adds tests for reconcile.go's error/edge paths (unreadable unit, backup
and write failures, daemon-reload failure, malformed version markers,
downgrade-is-conflict) and drives admin_upgrade.go's CLI flow directly
via an injectable exit and a decision matrix extracted into
validateUpgradeFlags/upgradeSummary for testability.
The catalog image-import command wrote a volume's blocks through the
EBS provider but never wrote the EC2 control plane's
{snapshotID}/metadata.json, so the imported AMI's snapshot was
invisible to DescribeSnapshots and any resolution path without the
system-AMI fallback. Register it the same way the ochre weights
import already does, and log when the system-AMI fallback masks a
missing document instead of silently succeeding.
AMI metadata moved to ebsmetadata documents and the legacy
ami-<id>/config.json read path was removed rather than migrated, so an
AMI imported before the change is invisible to the control plane after
upgrading. Nothing reports it as pending, because the gap is in stored
data rather than in a config file. Warn about it alongside the existing
object-storage exemption, and say how to check and recover.
Also document the binary-swap trap: replacing spx under a running
target leaves services on the replaced inode, and because handlers are
queue-group workers a skewed node answers a fraction of requests with
the old behaviour, which reads as an intermittent fault.
fakeSystemVPC.seq and fakeSystemVPC.igwCreated were mutated without
synchronisation while the service under test runs launches concurrently,
so the race detector intermittently failed
TestList_ReturnsEndpointsAcrossAllAccountsIncludingPinned with a data
race between CreateSubnet and AssociateRouteTable, both reaching id().
Guard the fake's mutable state with a mutex. Test-double only, no
production code involved.
* feat: introduce the EBSProvider interface and control-plane metadata store
Phase 1 of decoupling the control plane from viperblock. Adds a
provider-neutral block-storage contract that spinifex owns end to end.
ebsprovider defines EBSProvider, whose operation set and idempotency rules
follow the CSI controller service, with NATS request/reply as the transport
rather than CSI's gRPC. Every request and response carries an explicit schema
version so wire skew fails loudly at the boundary instead of silently decoding
a changed payload. The package deliberately exposes no viperblock types:
provider-specific state is an opaque handle. Ships with a NATS client against
the ebs.provider.v1.* subjects and an in-memory fake for tests.
ebsmetadata gives spinifex its own volume and AMI documents under a spinifex
key prefix, so the control plane can answer DescribeVolumes and DescribeImages
without reading the storage engine's persisted state.
The EC2 volume and snapshot handlers gain provider-backed paths for create,
describe, modify, attach-state and delete. Provider injection is optional for
now, so the existing engine path stays the default until the viperblockd side
serves the new subjects and the metadata migration lands.
vm/lifecycle.go no longer imports viperblock; FindFreePort was the only symbol
it used and is now a local helper.
* feat: serve the EBS provider contract from viperblockd
Gives the ebsprovider NATS client a server. Volume create, describe, expand
and delete, plus snapshot create and delete, are now handled by the daemon
that owns BaseDir and the mounted-volume registry, which is where viperblock
engine construction belongs.
Every handler prefers an already-mounted engine over opening a second one on
the same volume, and expand refuses outright while the volume is mounted.
Opening a second engine on a live volume is the double-writer hazard this
decoupling exists to remove, so the boundary enforces it rather than trusting
callers. Requests carrying an unrecognised schema version are rejected at the
boundary instead of being decoded on a best-effort basis.
Snapshot creation replies immediately with an operation ID and completes
asynchronously on a per-snapshot subject, since a large volume will outrun the
request timeout. Draining the volume stays with the caller: it coordinates
with the guest and is not something the storage boundary can reach.
The snapshot create subject gains an explicit create verb. Addressing it as
ebs.provider.v1.snapshot.<volumeID> collided with the snapshot delete subject
and left the wildcard unsubscribable.
Publish and unpublish are not served yet; the mount and unmount logic they
would front is still inline in launchService and needs extracting first.
Also adds a conformance suite that runs one set of contract assertions against
both the in-memory fake and the NATS client, so each side is checked against
the contract rather than against the other's structs.
* feat: let an operator select the EBS provider
Adds an [ebs] provider key and wires the NATS provider into the EC2 volume,
snapshot, image and instance services at the daemon's composition root. Until
now nothing injected a provider, so the provider-backed code paths in those
services were unreachable.
The key is provider-neutral rather than nested under [viperblock], because it
names the boundary rather than one provider's settings; a second provider will
not need it renamed. An empty value resolves to "embedded", so deployed config
files that predate the key keep today's behaviour, and an unrecognised value
is a startup error rather than a silent fallback.
Startup probes the provider's capability verb and warns when nothing answers.
The daemon and viperblockd start independently, so an unanswered probe is not
fatal, but without it a misconfigured node fails much later as an unexplained
request timeout on the first EBS call.
"viperblockd" stays experimental: under it volume metadata lives in the
control plane's own document, and the leak reaper, GC and admin paths still
read viperblock's config.json, so they cannot see tenant or tag data for
volumes created that way.
* fix: make every volume-metadata consumer provider-aware
Several readers and writers of volume metadata were never given a provider
branch, so under the viperblockd provider they silently used the wrong store.
Each of these is a correctness bug rather than a missing feature, because the
control plane had already stopped writing the document they read.
Modifications had nowhere to live: ModifyVolume built its result in memory and
dropped it, and DescribeVolumesModifications read it back out of viperblock's
config.json, so a describe straight after a successful modify returned nothing.
Volume metadata now carries its own modification record.
The leak reaper read attachment and tags from config.json and wrote orphan
markers to tags.json, neither of which exists for a provider-managed volume, so
a leaked volume was invisible to it and any mark it did write could not be read
back. Tag mutations had the same split: a tag applied after creation never
reached the document DescribeVolumes reads.
Snapshot deletion could not see a provider-managed clone, because a clone
records its source only in the control-plane document. That let a delete strip
chunks a live clone still reads, so it is fixed here rather than left for the
migration.
Volume enumeration and DescribeVolumeStatus were likewise scanning object
prefixes that a provider-managed volume does not create.
Also adds an encryption flag to the volume document, which DescribeVolumes had
been reporting as false unconditionally and snapshots could not carry at all,
and a list operation for AMIs so the image path can follow later.
The legacy path is unchanged; its invariant tests pass unmodified.
* feat: backfill ebsmetadata documents from legacy volume and AMI state
DescribeVolumes under the viperblockd provider enumerates only ebsmetadata
documents, so a volume created before the switch had no document and was
invisible rather than merely slow to read. A per-object read fallback cannot
fix enumeration, so both a backfill and a fallback are needed.
Add an object-store migration kind to the migrate framework. Predastore has no
conditional write, so an object-based version stamp would be last-writer-wins
across nodes; RunObject stamps progress in a caller-supplied JetStream KV
bucket instead, exactly as RunKV stamps a bucket's own version. The stamp is a
read-then-write rather than a compare-and-swap, so correctness rests on every
ObjectMigration.Run being safe to re-execute.
The backfill itself lives in migrate/ebsmetadatabackfill rather than beside the
numbered config migrations: it needs viperblock's VBState schema to decode the
legacy config.json, and migrate is imported by every handler that stamps a KV
bucket version. It converts what it can reach and skips unreadable objects with
a log, so one corrupt volume cannot hide the rest of the fleet.
ebsmetadata gains the legacy read fallback for the window before the migration
has run everywhere. It stays free of viperblock by taking the decode as an
injected function value, wired at the composition root, and the fallback is
retired by leaving it unset. A fallback read never writes the converted value
back, which would race the migration.
Prefix enumeration pages to exhaustion. Stopping at the first page would let
the backfill stamp itself complete having converted only the volumes it
happened to see, reintroducing the invisible-volume failure it exists to
prevent.
The migration runs from configureEBSProvider, so only when the provider is
selected; backfilling while the embedded engine is still authoritative would
produce documents that immediately go stale. Failure is fatal rather than
advisory, unlike the capability probe beside it. This makes the switch to
viperblockd one-way, which is now recorded on the config field.
* feat: serve volume publish and unpublish from viperblockd
PublishVolume and UnpublishVolume were the last unserved verbs on the provider
contract, and they are the ones that attach a volume to a VM. Until they are
served the provider path cannot boot an instance, so provider = "viperblockd"
cannot be verified end to end.
The mount and unmount logic was two large anonymous handler bodies inline in
launchService. Rather than write a second mount path, extract each into a named
function and have both the legacy ebs.mount/ebs.unmount handlers and the new
provider handlers call it. Two divergent implementations of nbdkit spawning and
the seal decision would be far worse than the inline bodies. The extraction
relocates code without rewriting it; response publishing stays in each handler
so the provider never publishes to ebs.mount.response.
Publish is idempotent: a volume this node already has mounted returns its
existing attachment rather than starting a second nbdkit against it, which is
the double-writer hazard the boundary exists to prevent. Unpublish preserves
the invariant that a failed seal leaves the volume mounted so a retry can
re-attempt it, and reports an error rather than success.
Both verbs are node-addressed, so they subscribe directly rather than through
the spinifex-workers queue group, and are registered only when NodeName is set.
There is no queue-group fallback: a subject that already names one node has
nothing sensible to fall back to.
Read-only attachment is refused rather than silently granted read-write, since
nbdkit is started read-write or not at all.
Also fixes a latent panic carried by the extracted mount path: viperblock.New
returns a nil VB alongside its error, and SetCacheSize was called on it before
the error was checked, so a failing New would have taken down the handler.
* feat: read and write AMI metadata through ebsmetadata
The backfill migration wrote ebsmetadata AMI documents that nothing read: every
AMI path still went through viperblock.AMIMetadata inside a VBState wrapper at
{imageID}/config.json. Give the image service provider branches for the AMI
read, write and enumerate paths, with enumeration going through ListAMIs rather
than an object-prefix scan, as DescribeVolumes already does.
AMIMetaLoader declared GetAMIConfig returning viperblock.AMIMetadata, putting a
viperblock type on an interface boundary in the instance package. It now returns
ebsmetadata.AMI on both paths, the embedded one converting before it returns, so
the boundary names no viperblock type whichever provider is selected.
The legacy-to-document mapping had accumulated three copies. Consolidate them in
ebsmetadata/vblegacy, a sibling package rather than a member so ebsmetadata
itself stays free of viperblock, and cover it with a reflective test that fails
when a field is added to either schema without being wired through. Sixteen
fields copied by hand in three places would have dropped one silently.
The admin image paths read whichever representation exists and write back to
that same one, so a provider-managed AMI never gains a spurious config.json.
FindAMIDependents now unions ebsmetadata volumes with the legacy prefix scan:
missing a dependent there deletes an AMI a live volume still needs, the same
class of bug as the snapshot clone path fixed earlier on this branch.
That union is built by hand rather than by wiring Store.ListVolumes' fallback,
which swallows per-ID read errors. This path deliberately fails closed on a
transport error so a delete cannot proceed on a partial picture, and the
fallback would have quietly turned that into failing open.
The CLI image import now also writes the document, best-effort, so an AMI
imported after the migration is not left reachable only through the fallback.
* fix(ec2): resolve volume config through ebsmetadata when a provider is configured
A provider-created volume has no vol-*/config.json — its metadata lives in
the ebsmetadata document. getBaseVolumeConfigAndEncryption read the legacy
key unconditionally, so AttachVolume reported a volume that DescribeVolumes
had just listed as available as InvalidVolume.NotFound.
Routing through ebsmetadata.Store also puts this path behind the legacy
fallback, so a volume created before the provider switch stays attachable.
The NoSuchKey remap now names the key it missed; its silence made the
original failure invisible in the daemon logs on every node.
* test(bench): add an EC2 control-plane latency benchmark
Measures the EBS control-plane operations either side of the provider
boundary, so the cost of the NATS hop can be stated rather than guessed.
Reports median and p95 with a bootstrap CI on the median difference, and
labels a delta indicative unless that CI excludes zero.
* fix(ec2): drain the volume before a provider snapshot
The legacy CreateSnapshot branch drains first so the snapshot reads a
current checkpoint; the provider branch did not, and viperblockd documents
the drain as the caller's responsibility. Both sides deferred to the other,
so an attached volume snapshotted stale data with no error raised.
DrainVolume and volumeAttachment now take the fallback state and instance
as strings rather than a viperblock.VolumeMetadata, removing a viperblock
type from an exported signature.
* feat(ec2): route the instance root volume through the EBS provider
RunInstances built its own viperblock engine to clone the AMI snapshot into
the root volume, which made every launch a second writer against a volume
viperblockd already serves. The launch now calls the provider's CreateVolume
with the AMI's snapshot and source volume, and records the result in
ebsmetadata so DescribeVolumes and attach can see it.
A snapshot clone needs both the snapshot ID and its source volume ID to
resolve base blocks, so CreateVolume in the volume handler now sends
SourceVolumeID alongside SourceSnapshotID; without it the daemon rejected
every create-volume --snapshot-id outright.
The gp3 IOPS and throughput constants move to spinifex/types so the instance
and volume handlers share one definition rather than duplicating literals.
* feat(ebsprovider): add a bounded volume seed to CreateVolume
The EFI variable store must be initialised from the launching node's firmware
VARS template, byte for byte. Passing a host path would not work: CreateVolume
is queue-subscribed, so any node can service it, and a foreign node's OVMF
package would bind the guest's pflash region to the wrong firmware build.
CreateVolumeRequest carries the bytes instead, as SeedData, and viperblockd
writes them at offset 0 in the engine it already builds. The seed applies only
when the create actually allocates the volume, so a relaunch never overwrites
a BootOrder the guest has since written.
JSON base64-encodes a []byte, inflating the payload by 4/3, so MaxSeedBytes
bounds the seed well inside the NATS max_payload the cluster runs with and
both ends reject an oversized seed with a clear error rather than letting the
publish fail. VolumeSeeding advertises the behaviour so a provider that cannot
seed says so instead of silently dropping the bytes.
* feat(ec2): route the EFI variable store through the EBS provider
prepareEFIVolume built a control-plane viperblock engine to seed the variable
store from the firmware VARS template, making every UEFI launch a second
writer against a volume viperblockd already serves. It now calls the
provider's CreateVolume with the template as seed bytes, so the store is
allocated and written by the daemon that owns the volume.
The template is still read on the launching node and shipped with the request.
CreateVolume is queue-subscribed, so any node can service it, and a VARS
region seeded from a foreign node's OVMF build would bind the guest's pflash
to the wrong firmware.
pflash refuses a VARS region that is not byte-exact, so the returned capacity
is checked against the template length; a provider that rounded up to a GiB
boundary would otherwise yield a guest that cannot boot.
The engine path stays behind a nil-provider guard until the legacy branch goes.
* feat(ec2): route image snapshots through the EBS provider
snapshotRunningVolume and snapshotStoppedVolume each built a control-plane
viperblock engine to take the snapshot backing a new AMI, making CreateImage
a second writer against a volume viperblockd already serves. Both now call
the provider's CreateSnapshot with the volume's provider handle.
The running path keeps its drain, which is what makes the snapshot current
rather than one predating a guest-triggered write. The stopped path needs no
drain, and the daemon already prefers a live mounted engine over opening a
second one, so it handles both cases itself.
getVolumeConfig read the legacy config.json, which a provider-created volume
does not have, so every CreateImage against one failed. It now reads the
ebsmetadata document under a provider, the same way attach does.
* fix(daemon): only the owning node creates an image for a stopped instance
handleEC2CreateImage is a fan-out. Ownership is established by vmMgr, which
only the node holding the VM passes, so a running instance is imaged once.
A stopped instance lives in the cluster-replicated KV instead, which every
node reads successfully, so all of them ran the full snapshot and AMI
creation. The caller saw the first reply and the rest leaked: on a three-node
cluster one create-image produced three AMIs and three snapshots.
A stopped instance records its owner in LastNode, so a node that is not the
owner now declines. Gather tallies error frames and keeps waiting, returning
only on a success, so declining cannot preempt the owner's reply.
An instance stopped by a build that never set LastNode elects an owner
through the KV's compare-and-swap, since having every node decline would make
create-image fail permanently for those records.
* perf(ec2): serve DescribeVolumes by ID from the volume documents
The provider branch called ListVolumes unconditionally and filtered in
memory, so asking for one volume read and decoded every volume document in
the bucket. The non-provider branch already had the equivalent fast path.
Explicit IDs now fetch their documents directly. The tenant check moves with
them: a document belonging to another account reports not-found rather than
being returned, matching what the enumeration filter did.
Measured against a memory object store, the fast path stays flat near 20us
regardless of bucket size while the enumeration grows linearly, reaching
4.6ms at 500 volumes.
* chore(lint): ban the viperblock import from the EC2 control plane
The control plane reaches block storage through ebsprovider now, but nothing
stopped a handler from importing the engine again. That is not hypothetical:
the instance handler first imported viperblock for a config struct alone, and
the engine construction followed thirteen days later.
depguard denies the import across handlers/, which matters because
viperblock.New starts a chunk uploader — every construction is a writer
against a volume viperblockd already serves. A types-only import is denied
too, since that is exactly how the last one started.
The four files still carrying the pre-provider engine path behind a
nil-provider guard are exempted by name. The exemption list is the remaining
work: it shrinks to nothing when that path goes.
* fix(ec2): refuse CopySnapshot instead of creating an unusable snapshot
CopySnapshot wrote a control-plane snapshot document under the new ID and
nothing else. The copy described correctly and CreateVolume against it
succeeded, so it looked like it worked, but the new ID had no viperblock
snapshot metadata behind it and the volume failed when the guest attached it.
Producing the metadata means re-sealing it under the destination ID. On an
encrypted volume it is AEAD-sealed with the snapshot ID in the AAD and the
sealing helpers are unexported, so spinifex cannot write it. Copying only the
unencrypted form would be dead code on any cluster with encryption enabled.
Refusing with UnsupportedOperation fails at the call the caller can act on
rather than at attach time. Source lookup and the ownership check still run
first, so a bad ID or another tenant's snapshot reports that instead.
* test(ec2): give handler test fixtures their own legacy config.json shape
Handler tests built the legacy config.json fixture by importing
viperblock's VBState/VolumeConfig/VolumeMetadata/AMIMetadata directly.
Mirror that shape in ebsmetadata/vblegacy, next to the production code
that already reads it (ebsmetadatabackfill), and switch the
fixture-only usages across the volume/snapshot/image handler tests to
it. A byte-for-byte equivalence test against the real viperblock types
guards the mirror from drifting out of sync.
Call sites that hand a viperblock type into putVolumeConfig,
mergeVolumeConfig, or buildVBConfig are untouched, since those
functions still name the real type in their signature.
* feat(ebsprovider): add CopySnapshot to the EBS provider contract
Adds a CopySnapshot operation to EBSProvider (types, ebs.provider.v1.snapshot.copy
NATS subject, MemoryProvider and NATSProvider implementations, conformance
coverage) and a viperblockd handler that duplicates a snapshot's metadata via
viperblock.CopySnapshotMeta on the source volume's VB.
* feat(ec2): serve CopySnapshot through the EBS provider
CopySnapshot has been refusing with UnsupportedOperation because a copy needs
its own snapshot metadata on the backend under the new ID, and on an encrypted
volume that metadata is AEAD-sealed with the snapshot ID in the AAD, so the
control plane cannot re-seal it. The provider contract now carries a copy
operation, so the storage engine does it.
The provider path generates a new snapshot ID, has the provider duplicate the
backend metadata under it, then writes the control-plane document. Rollback
mirrors CreateSnapshot: a failure after the copy deletes the provider snapshot
rather than leaving one nothing references. Ownership is still checked against
the source before anything is created.
The no-provider path keeps refusing. It has no way to produce the backend
metadata, and the copy it used to write described fine and then failed at
attach — a resource that fails as far as possible from the call that made it.
Adds MemoryProvider.Snapshot so a control-plane test can assert a snapshot
reached the provider rather than existing only as a control-plane document.
That assertion is what catches the old behaviour.
* refactor: remove the embedded EBS engine path from the EC2 control plane
viperblockd is now the only EBS provider. The pre-provider path that
constructed a viperblock engine in-process is deleted, and with it every
github.com/mulgadc/viperblock import under spinifex/handlers/ (tests
included). The depguard ec2-control-plane rule drops all four file
exemptions and the !$test carve-out, so the rule now reads simply as
"nothing under spinifex/handlers/** may import viperblock".
viperblock.VolumeConfig was also a data carrier on the provider path. It
is replaced by the spinifex-native ebsmetadata.Volume rather than the
vblegacy mirror: vblegacy mirrors a legacy on-disk shape whose only job is
decoding old objects, while ebsmetadata.Volume already carries every field
the call sites read, so the conversion step disappears instead of being
reimplemented against a legacy shape.
The embedded-only helpers (volumeConfigWrapper, putVolumeConfig,
mergeVolumeConfig, configIsEncrypted, putVolumeConfigViaDetached,
deleteS3Prefix, buildVBConfig, snapshotVolume, the ami-*/config.json
prefix scan) had exactly one production caller chain each, all through a
nil-provider branch, and are removed. The snapshot-dependency guard on
DeleteVolume is provider-agnostic and is kept, now running on the provider
path where it previously did not.
config: viperblockd is the default and an empty Provider resolves to it.
An explicit provider = "embedded" is rejected at config load rather than
silently upgraded, because starting anyway would migrate a node's volumes
to ebsmetadata one-way without the operator asking. The generated
spinifex.toml template emits viperblockd.
* fix(images): write the imported AMI document from the config the import wrote
ImportDiskImage creates the AMI's snapshot and records it on its own
*viperblock.VB, which viperblock.New allocates fresh, so the value lands only
in {imageID}/config.json. The import command was persisting the ebsmetadata
document from its pre-import manifest copy, whose SnapshotID is still empty.
Store.GetAMI only consults the legacy config.json fallback when the document
is missing, so that empty-SnapshotID document permanently shadowed the real
value and every launch from an imported AMI failed the zero-copy clone check
with ServerInternal.
Rebuild the document by reading back the config.json the import just wrote. A
missing or unreadable config is skipped rather than written from the stale
manifest: no document still resolves correctly through the legacy fallback.
* test(e2e): read NODE from spx get vms instead of VCPU in the spread check
spx get vms gained HEALTH and CRASHES columns, shifting NODE from field 6 to
field 8. The spread placement check still read field 6, so every instance
reported its vCPU count instead of its node, the unique-node count always
collapsed to 1, and the check failed on clusters that had spread correctly.
Read field 8 and strip the ANSI colour codes wrapping each cell.
* feat(viperblockd): add a --debug flag so viperblock debug logging is reachable
viperblockd hardcoded Info, and viperblock's VB.logger() falls back to
slog.Default() because vb.log is only ever set by SetDebug. That left the
engine's debug records, including the chunk uploader's start and disabled
lines emitted during New(), unreachable in a deployed environment.
Binds the namespaced viper key viperblock-debug and SPINIFEX_VIPERBLOCK_DEBUG
rather than the bare debug key, which several commands share. Also sets the
Config.Debug field, which existed but was set only by tests.
* fix(viperblockd): release the chunk uploader when openVolumeVB fails
viperblock.New starts a chunk uploader and a WAL syncer before the backend is
touched, so a failure in Backend.Init or loadStateWithRetry returned nil to the
caller while leaving both running with no handle to stop them. Repeated backend
failures on CopySnapshot, ExpandVolume or DescribeVolume accumulated uploaders
on a 30s ticker for the life of the process.
Adds invariant tests for both halves of the ownership rule: a mounted volume's
snapshot create and copy reuse the live engine and log zero volume opens, an
unmounted volume opens exactly one, and a failed open leaves no uploader behind.
The open counts are read from the same log record a production alarm watches.
* fix(ebsprovider): route requests to the node that owns the mount
Every viperblockd queue-subscribed the ebs.provider.v1.* verbs in one
spinifex-workers group, so NATS delivered each request to a random node. Only
the node holding the mount can reuse the live engine through findMountedVolume;
any other node constructed a second viperblock engine, with a live 30s chunk
uploader, against a volume nbdkit was serving on a different host. Measured on
a three-node cluster, this was roughly two snapshots in three.
Rather than resolve the owner, let the owner answer. A node subscribes
ebs.provider.v1.owner.<volumeID>.<verb> as a plain subscription when it mounts
a volume and drops it on unmount, mirroring the per-volume ebs.config
subscription. Clients try that subject first and fall back to the queue group
only on ErrNoResponders, which means nothing is mounted and opening an engine
is correct. A timeout deliberately does not fall back, since the owner may
still be running the operation.
Applies to snapshot create and copy, expand and describe, the four verbs that
reuse a mounted engine. Delete keeps queue dispatch.
* fix: rebuild viperblockd mounted-volume registry after a restart
Config.MountedVolumes is in-memory and rebuilt from nothing on start, so a
viperblockd restart while a VM keeps running leaves nbdkit serving a volume
the daemon has forgotten. Owner routing then infers from ErrNoResponders that
nothing is mounted and falls back to the queue group, which opens a second
viperblock engine, with a live chunk uploader, on another node. The same empty
registry also defeats handlePublishVolume's double-mount guard, so a republish
could start a second nbdkit.
Discover surviving nbdkit processes from /proc at startup and rebuild the
registry from them, following orphan_scan.go's precedent: match comm exactly,
parse the argv for volume=<id> and --unix/-p, and adopt a PID only when
nbdkit-vol-<volume>.pid names that same PID. Nothing is started, signalled or
killed; anything that fails corroboration is logged and skipped. Recovery runs
before the first subscription so no handler can race it.
Extract mountVolume's engine construction into constructMountedVB so mount and
recovery cannot drift, and release the WAL syncer viperblock.New starts when
construction fails after it.
* fix: corroborate recovered nbdkit processes from argv, not a pidfile
nbdkit is launched with -f (required for the Go plugin) and never writes the
--pidfile it is given, so no such file exists on any node; the daemon's own
/tmp is a PrivateTmp namespace besides. Corroborating against it rejected
every genuine mount, making recovery a no-op in production.
Corroborate against what the kernel actually has: the process must serve our
plugin, name this daemon's base dir, and still hold its socket. Adopt a volume
once, so two live nbdkits for it leave one registry entry and a warning rather
than two conflicting ones.
* feat: add a second EBS provider backed by qcow2 and qemu-nbd
The decoupling was justified by swappability that nothing demonstrated: the
conformance suite ran against MemoryProvider and against a NATSProvider wired
to a MemoryProvider, so both arms were the same implementation behind
different transports. A fake is not a second implementation.
Add three pieces. natsserve serves the ebs.provider.v1.* contract from any
EBSProvider, depending on nothing but the interface, and replaces the
test-only server twin so conformance now exercises production code. The
conformance suite moves to its own package and is exported so implementations
outside ebsprovider can run it, with no assertion changed. qemunbdd is the
second implementation: one qcow2 file per volume, qemu-nbd for publish, real
snapshot copies instead of chunk references, and every exec behind a runner
seam so its tests need no qemu.
Owner-subject routing is deliberately not served neutrally: it needs a
per-node mounted-volume registry the interface does not model, so it stays in
viperblockd and a neutral server's clients correctly fall back to the queue
group.
* test: run the conformance suite against the qcow2 provider
The suite passes unmodified against an implementation sharing no code with
viperblock, both in-process and through natsserve over the real wire
contract, which is the swappability claim the provider boundary was built to
support.
Two real defects it caught. ExpandVolume and GetVolume inspected the image
before consulting the publication map, so qemu-img could not open a volume
qemu-nbd held the write lock on and the refusal surfaced as an internal
error instead of ErrVolumeInUse; qemu-img info now runs with --force-share,
so describing a volume no longer depends on whether it is attached. Publish
also failed with qemu-nbd's raw exit status when the socket path exceeded the
108-byte AF_UNIX limit, and now reports that reason itself.
* feat: make the EBS provider selectable by config
Add the qemunbd selector alongside viperblockd, a spx service command that
serves the qcow2 provider over natsserve, and service registration. Config
validation accepts the new value everywhere viperblockd is accepted, while
embedded stays rejected by name.
The daemon logged a hard-coded viperblockd constant as the active provider
rather than the resolved one, in configureEBSProvider and in both of
probeEBSProvider's lines; all three now report what is actually configured.
Both daemons answer the same subjects, so exactly one may run on a cluster.
* test: add a live lifecycle check for the qcow2 provider
Drives create, publish, write, snapshot, restore and read-back entirely
through ebsprovider.NewNATSProvider against a real qemunbdd daemon on a
cluster node, so the client under test is the same shape the control plane
uses and never touches the provider type. Skipped unless SPINIFEX_QEMUNBD_LIVE
is set, so it stays out of preflight.
* fix: publish NBD URIs in the form standard clients accept
PublishedVolume.NBDURI carried nbd:unix:/path, which is QEMU's legacy
pre-URI filename syntax rather than an NBD URI. libnbd rejects it outright
with "URI must begin with 'nbd://'", so no NBD client but QEMU could ever
consume what the EBS provider boundary hands back.
Emit nbd+unix:///?socket=/path instead, the form libnbd, qemu and nbdkit
all accept. ParseNBDURI now takes both that and the TCP form, and still
accepts the legacy nbd:unix: syntax for state written by older daemons, so
a conformant third-party provider's URI is no longer rejected on the way
in either.
Verified against qemu 10.0.11: qemu-img and a real -drive option string
both open the URI form, and qemu reports both forms as nbd+unix://?socket=,
so it already treats the URI as canonical.
* test: validate published exports with libnbd's client tools
Every other assertion about the EBS provider goes through our own Go
client, so nothing has ever checked whether the export we publish is usable
by an NBD client that knows nothing about this codebase. That is the layer
neither the conformance suite nor a CSI-shaped contract can reach.
RunNBDClientSuite drives nbdinfo and nbdcopy against a published volume:
the URI connects, the export advertises its size and block-size
constraints, data written externally reads back byte for byte, a ReadOnly
publish sets the read-only transmission flag, and UnpublishVolume tears the
export down. It skips when libnbd-bin is absent rather than passing
vacuously.
Wired to qemunbdd, where all five failed before the URI fix. The publish
assertion in the main suite now requires a dialable URI instead of merely
a non-empty string, which is what let the wrong form through.
* test: run the external NBD client suite against a live viperblockd
The suite so far only ever saw qemu-nbd. Pointing it at viperblockd's nbdkit
export is the comparison worth having: one external client, two providers it
cannot tell apart, and every difference is a candidate leak in the boundary.
RunNBDClientSuiteWithConfig takes the node name and volume prefix a live
cluster needs, and publishForNBD now unpublishes and deletes what it created
so a run against a real cluster leaves nothing behind. Subtests dial a
normalised URI so a provider emitting the legacy syntax still gets its export
tested rather than every subtest failing on the same string; whether the URI
itself is standard is now its own subtest.
Adds a reconnect check, since publication holds the volume rather than any one
connection. It retries the real operation because a cheaper liveness probe is
itself a connection and re-triggers whatever refused it.
* feat: make the conformance suite capability-driven
The suite asserted a fixed Capabilities literal, so it could only ever pass
against a provider advertising exactly that set. Every implementation had to
match the suite rather than the suite testing the contract. RunSuite now reads
GetCapabilities from the provider under test and gates optional behaviour on
what it advertises, and the fixed literal is demoted to ReferenceCapabilities,
a convenience for callers constructing a MemoryProvider.
Add ReadOnlyPublish to Capabilities. A provider that leaves it false must
refuse a ReadOnly publish rather than hand back a writable export, which is
what viperblockd already does; it now has a way to say so up front.
Add ErrUnsupportedCapability and the unsupported_capability error code. A
request for optional behaviour a provider does not advertise is well formed,
so classifying it as invalid_argument told the caller to fix a request that
was never wrong. viperblockd's ReadOnly refusal is reclassified accordingly.
Declared capabilities now match observed behaviour: qemunbdd advertises
SparseExtentReporting (qemu-nbd exports base:allocation, confirmed with
nbdinfo) and ReadOnlyPublish (PublishVolume passes -r). viperblockd advertises
neither: the nbdkit Go binding exposes no extents callback, and the mount path
never passes -r.
The external NBD client suite gains a sparse-extent subtest that checks the
export's advertised contexts against the capability, so SparseExtentReporting
is falsifiable from outside rather than taken on trust, and its read-only
subtest now branches on the capability instead of assuming refusal.
Cover the new branches with a MemoryProvider advertising no optional
capabilities and one advertising OnlineExpansion, the branch no provider we
ship exercises.
* refactor: rename SourceVolumeID to SourceSnapshotVolumeID
CreateVolumeRequest.SourceVolumeID meant three different things. viperblockd
required it whenever SourceSnapshotID was set and used it to resolve the
snapshot's chunk references, so there it named the snapshot's origin volume.
MemoryProvider ignored it entirely. qemunbdd implemented it as an independent
volume-to-volume clone, which is what the name and doc comment implied. The
conformance suite could not catch the divergence because the reference
provider had no branch for the field at all.
The contract already has a field meaning "the volume this snapshot came from":
Snapshot.SourceVolumeID, which every provider populates. So the request field
is the same fact travelling back the other way, and its name was the problem.
Rename it to SourceSnapshotVolumeID, document it as the source snapshot's
origin volume, and state that it is required with SourceSnapshotID and
meaningless without it.
All three providers now enforce one rule: a source snapshot must be
accompanied by its origin volume, or the request is invalid_argument.
MemoryProvider additionally rejects an origin that does not match the
snapshot's own SourceVolumeID, so a wrong name fails instead of resolving
blocks against a volume the snapshot never came from. qemunbdd loses its
volume-to-volume clone branch, which implemented behaviour the contract no
longer offers and no caller ever requested.
The suite pins both the requirement and a full create-snapshot-restore round
trip, so the field is exercised end to end rather than only on the error path.
* feat: make the conformance suite runnable against a live provider
The suite had only ever run against MemoryProvider and qemunbdd, both of which
are discarded with the test binary. viperblockd, the provider production
actually uses, was the one implementation the suite never judged, because the
suite assumed each subtest got fresh state: hard-coded volume IDs collide with
an earlier run's leftovers on a store that keeps them, and a hard-coded node ID
addresses a node that does not exist.
SuiteConfig adds NamePrefix, NodeID and OtherNodeID. A tracking wrapper records
every volume, snapshot and publication the suite creates and tears them down in
dependency order, logging rather than failing so a teardown problem cannot mask
the assertion a subtest was making. Two new tests cover this by running the
whole suite twice against one shared provider instance, which passes only
because cleanup works, and once per prefix to cover concurrent runs sharing a
daemon.
TestLive_ViperblockdConformance wires this to a real cluster behind
SPINIFEX_VIPERBLOCK_LIVE, alongside the existing external NBD client suite.
* fix(viperblockd): close the four contract gaps the live conformance run found
Unpublishing a volume this node has no mount for now succeeds instead of
returning not_found, matching the legacy unmount path: nothing mounted here is
already the state the request asks for.
CreateSnapshot refuses a snapshot ID that already exists against a different
source volume with already_exists, rather than overwriting it.
DeleteVolume refuses a published volume with volume_in_use instead of tearing
the mount down, on this node or on whichever node answers the volume's owner
subject. The export and the guest writing through it survive the metadata
deletion, so the old behaviour surfaced later as corruption.
CreateVolume reads the source snapshot before writing the volume, returning
not_found when it is absent and invalid_argument when it was taken from a
different volume than the request names. Previously both produced a blank
volume and a successful response.
CopySnapshot classifies viperblock's foreign-volume refusal as invalid_argument
now that it carries a sentinel.
* feat(ebsprovider): pin the source snapshot's origin volume in the contract
The suite only required source_snapshot_volume_id to be present, so a caller
naming a volume the snapshot never came from was accepted everywhere. The
snapshot records its own origin, so this is checkable: creating from a snapshot
with a mismatched source volume is now invalid_argument for every provider,
enforced by qemunbdd against its snapshot metadata and by viperblockd against
the snapshot's persisted SourceVolumeName.
* test(ebsprovider): run the contract suite against viperblockd in CI
CI installed neither qemu nor nbdkit, so every conformance test that needs a
real implementation skipped silently and MemoryProvider was the only provider
the suite judged. CI now installs both, builds viperblock's nbdkit plugin, and
sets SPINIFEX_REQUIRE_CONFORMANCE_TOOLS so a runner image that loses them fails
instead of skipping.
TestViperblockdConformance stands the provider handlers up over an embedded
NATS and a real predastore and runs the same suite against them, which is what
caught CopySnapshot accepting a foreign volume's snapshot on unencrypted
volumes.
* feat(viperblockd): honour read-only publish and advertise block size constraints
ReadOnlyPublish was advertised false and read_only=true was rejected,
so the capability was true only by omission. nbdkit -r now starts the
export when the request asks for it, which sets the NBD read-only
transmission flag and makes the plugin refuse writes. Access mode is
fixed when nbdkit starts, so a republish asking for the other mode is
refused with VolumeInUse rather than handed the running export.
Block size constraints were absent entirely: the nbdkit Go binding has
no block-size callback, so the plugin cannot advertise them. The
blocksize-policy filter can, without a binding callback. Minimum 1
because viperblock's WriteAt handles unaligned requests with a
read-modify-write, preferred 4096 to match its block size, maximum 32M
to match what qemu-nbd advertises.
* test(conformance): judge sparse reporting by the allocation map, not the context
The subtest asserted that base:allocation was offered. nbdkit offers
that context whether or not the plugin can compute extents, answering
'all allocated' when it cannot, so the assertion passed for a server
with no extent support at all — it could not fail. What separates the
two is the answer: a volume with nothing written to it is entirely
holes to a server that really tracks allocation.
Read the map with nbdinfo --map --json and compare holes against the
advertised capability. qemunbdd reports holes and advertises true;
viperblockd reports none and advertises false.
* test(fixtures): keep SSL_CERT_FILE set for the fixture's whole lifetime
The predastore fixture is one daemon per test binary, but its cert was
exported with t.Setenv, which reverts when the first test that called
Start finishes. In-process clients did not notice because crypto/x509
caches the pool behind a sync.Once, loaded while the variable was still
set. Child processes did: nbdkit spawned by a later test inherited an
environment with no SSL_CERT_FILE and could not verify the daemon, so
the whole NBD client suite failed when the package ran as a whole while
passing when run alone.
Also install libnbd-bin in CI, so the NBD client suite runs there
instead of failing on missing tools.
* ci: install nbdkit-plugin-dev so the viperblock plugin can build
* test(volume): give the snapshot fixture an owner
CreateVolume now fails closed on a snapshot whose owner it cannot match,
including an unset owner_id. The fixture predates that and wrote none,
so the provider test failed with InvalidSnapshot.NotFound before ever
reaching the provider.
* test(integration): give the volume harness a real provider
The control plane no longer builds volumes itself, so CreateVolume in
the integration harness failed with "no EBS provider configured": the
harness wired a VolumeServiceImpl with nothing behind it. It now serves
viperblockd's provider subjects against the same predastore fixture, so
the blocks are still written by real viperblock to a real daemon.
Three persisted-state assertions moved with it. TenantID, VolumeType
and AvailabilityZone are control-plane facts, held in the ebsmetadata
document rather than viperblock's own state, so the storage oracle now
asserts their absence and DescribeVolumes -- whose answer is
account-scoped -- is what checks them.
* refactor: route image import through the EBS provider and drop the legacy metadata path
The control plane could still only run against viperblock: the AMI import and
the ochre weights import both linked the storage library directly, and the
ebsmetadata store kept a read fallback onto viperblock's config.json.
Import now goes through the provider contract with no new verb: CreateVolume,
PublishVolume, a bulk copy over the published NBD export with qemu-img, then
UnpublishVolume and CreateSnapshot. Bulk data rides NBD rather than NATS,
whose request size the contract caps far below a disk image.
The migration scaffolding goes with it. ebsmetadata documents are now the only
thing the control plane reads: no legacy prefix scan, no config.json fallback,
no backfill on daemon start. config.json belongs to the provider, which is the
single writer of it.
ListAMIs keeps its tolerant/strict split. Skipping an undecodable document is
right for DescribeImages, where one corrupt document must not hide the fleet,
and wrong for a name-uniqueness check, which cannot rule out a name it failed
to read.
spinifex and cmd no longer import viperblock outside services/viperblockd and
testutil/vbscan.
* feat: add a volume enumeration verb so a lost document cannot strand blocks
Making ebsmetadata documents the sole index of volumes left one failure with
no recovery: a volume whose document is gone disappears from DescribeVolumes,
from the leak reaper (which enumerates documents), and from the provider,
whose contract had no verb that asks what it holds.
ListVolumes closes that. It is capability-gated on VolumeEnumeration and
paginated, following CSI's optional LIST_VOLUMES rather than inventing a
shape. Refs are deliberately not Volumes: enumeration answers what exists,
and a provider must not open every volume's engine to say so.
Each implementation reads what actually exists rather than what it remembers.
qemunbdd walks its volumes directory, not its in-memory map, which does not
survive a restart; viperblockd walks the bucket's top-level prefixes. A
volume the provider has forgotten is precisely what a caller is looking for.
The consumer is a read-only orphan report: provider volumes minus documented
ones, exposed as spx admin volumes orphans. It never deletes. An EFI store is
excused only when its base volume is documented, so a stranded pair is still
reported. Listing documents stays strict, because reporting a volume as
orphaned on the strength of a document that failed to decode would point an
operator at live data.
* fix: stop the orphan scan accusing storage of holding things it does not
A live env19 run reported three orphans against a healthy cluster: an AMI's
backing volume, a snapshot, and the cluster key store. All three were wrong,
and a report that cries wolf on a clean cluster is worse than no report.
Enumeration treated every top-level bucket prefix as a volume. Snapshots sit
beside volumes at that level and have a config of their own, so presence of a
config cannot separate them; the snap- name is what does, which is already how
the AMI removal path reads them back. keys/ and spinifex/ are not volumes at
all.
An AMI's blocks are a genuine provider volume, so enumeration keeps reporting
them. What was missing is on the reading side: an AMI is recorded as an image
document, not a volume one, so the scan now treats a documented image as
known. Listing images is strict for the same reason listing volumes is —
unreadable and absent are different answers, and only one means orphaned.
* fix: let an orphaned image's blocks be reclaimed
The orphan report tells an operator that reclaiming a stranded volume is their
decision, and images remove --force is the tool it points them at. On env19
that tool failed on exactly those volumes: removal drops the ebsmetadata
document first, and predastore answers NoSuchKey for a document that is
already gone, which is the definition of an orphan.
Deleting the document stays first and stays fatal. That ordering is the
barrier that stops a launch landing on blocks about to be deleted, and an AMI
still advertised after its blocks are gone is the worse failure. Only an
already-absent document is now tolerated: the barrier is satisfied, and the
blocks are the sole thing left to reclaim.
S3ObjectStore.DeleteObject normalises a missing key the way GetObject already
does, so this is recognised rather than matched on error text.
The obvious unit test passed without the fix, because the in-memory store is
silently tolerant on delete and real S3 answers 204 — neither behaves like the
backend this runs against. The test now drives a store that reports the miss,
and fails without the fix.
* feat: add per-verb OTel spans to the EBS provider contract
The ten provider verbs cross a single NATS seam and nothing measured it.
The only spans on the storage path were the legacy ebs.mount/ebs.unmount
pair, so control-plane storage latency could not be attributed to a verb
or compared across provider implementations.
NATSProvider.request is the one client-side funnel every verb passes
through, including both legs of an owner-subject fallback, so a single
span there covers the whole contract and measures what the control plane
experiences regardless of which provider answers.
Span names come from a closed verb set rather than the subject. Owner
and snapshot-create subjects embed a volume ID and the mount/unmount
subjects embed a node ID; naming spans after them would give every
volume a span name of its own. The IDs move to attributes and an
unrecognised subject collapses to a single unknown verb.
Trace context rides the message headers, so the serving provider joins
the caller's trace instead of rooting one of its own. The carrier is a
copy of the one in spinifex/utils rather than an import: that package
pulls in the aws-sdk and the cluster key store, which the wire contract
must not depend on.
Responses carry no shared accessor for their error field, so both sides
re-read just that field from the encoded reply, and only when the span
is recording. Without it a failed verb looks like a successful round
trip.
viperblockd serves the contract with its own handlers rather than
natsserve, so its handlers now take the per-message context and its
publish/unpublish pair drops the subject-named consumer span it opened
for itself.
* feat: add null and hostile providers, and a per-verb bench suite
The provider boundary made it possible to put something other than
viperblock behind the control plane, and nothing yet did.
nullprovider is the floor. It is not a new type: MemoryProvider already
answers every verb from a map and natsserve already serves any
EBSProvider, so the package names that composition and states what it
measures. On loopback with no storage behind it a round trip costs
around 195us against 74ns in process, which is the seam's own price and
is not attributable to any storage engine.
conformance.RunBenchSuite prices one round trip per verb and drives
every implementation through the same calls in the same order, so two
providers' numbers are comparable. It measures control operations only:
guest I/O never crosses these verbs.
hostile wraps any provider in one that misbehaves. Faults derive from
the seed and a per-target sequence rather than a global counter, so
concurrent callers on different volumes cannot reorder each other's
faults and a failure found deep into a soak run replays exactly. Error
and error-after-work are separate faults on purpose: the second reports
failure after the work has already happened, which is how an orphan is
made, and is what makes an enumeration scan the right oracle for a soak.
Enumeration itself is never faulted, since an oracle that can be told
what to say cannot falsify anything.
The soak asserts that every volume the provider still holds is
attributable to an injected fault. A leftover with nothing behind it is
the control plane losing track of storage on its own.
Two bugs were found by falsifying the new tests rather than by running
them. The FNV-derived draw clustered badly enough that two fault bands
were never reachable in 1200 draws, so the draw now goes through a
SplitMix64 finalizer. And the first version of the lie test was
worthless: MemoryProvider returns copies, so a lie could not corrupt it
whether or not the injector copied, and the test now drives a provider
that hands out pointers into its own state.
* test: bench the provider contract against qemunbdd and a live viperblockd
RunBenchSuite was only ever pointed at the null provider, so the numbers
priced the seam and nothing else. Point it at the two real implementations.
BenchConfig gains a VolumeBytes knob: 1 GiB per iteration is fine against a
provider that stores nothing and wrong against one that does, so qemunbdd and
viperblockd are both measured at 64 MiB.
liveProvider splits into a testing.TB flavour. Benchmarks need the same
connection and the same node and cannot be handed a *testing.T.
* fix(viperblockd): carry the trace context into every viperblock engine call
Both S3 clients only emit a span when the request already carries one, so
calling the no-context Init, LoadState, SaveState, LoadBlockState and Close
left the whole engine open untraceable. Every provider verb traced as one flat
server span, and the 342ms describe measured on env19 could not be attributed
to anything.
viperblock already exposes the Ctx variants and Backend already declares
InitCtx, so this is threading, not new instrumentation. openVolumeVB,
openLoadedVolumeVB, sealVolumeVB, describeVolumeEngine, snapshotVolumeEngine,
copySnapshotOnVolume, seedVolume and applyConfigUpdate now take a context, and
mount/unmount keep the context their own span derives instead of dropping it.
completeCreateSnapshot gets WithoutCancel: the background half of
accept-then-publish has to outlive the request it was accepted by and still be
attributable to it.
The per-volume ebs.config handler opens a consumer span like the other legacy
subjects, so a config update is no longer the one path with no trace at all.
* perf: describe a volume without opening an engine on it
describeVolumeEngine fell through to openVolumeVB for any volume not mounted
on the node serving the request, which constructed a full viperblock engine
to read one integer. On env19 that decomposed as a 22 ms reachability list, a
3 ms state read and a 16 ms PutObject of the volume's own config.json — the
write coming from the SeqNum bootstrap a read-write open performs.
Use viperblock's read-only state load instead. A describe is now the 3 ms
read: no background goroutines, no reachability probe, no SeqNum window
claimed, and no write issued by a worker that does not own the volume.
Falsified: restoring the openVolumeVB call fails
TestDescribeVolume_ReadsWithoutOpeningTheVolume on both the write and the
list, with the fake backend answering each so neither assertion hides behind
the other.
* perf: share one HTTP client across every viperblock backend
Four sites built the same S3Config by hand, each leaving HTTPClient unset, so
every engine open and state read got its own connection pool and paid its own
TLS handshake. That was invisible while Backend.Init issued a whole-bucket
list — the list paid for the connection and the state read rode it — and it
became the dominant cost once describe was reduced to a single request.
Route all four through cfg.volumeS3Config, which builds the client once.
Falsified: dropping the HTTPClient field takes
TestDescribeVolume_ReusesOneConnection from one connection to three.
* perf: wait for nbdkit to be ready instead of waiting a fixed second
mountVolume spawned nbdkit and then ran a confirm loop until a one-second
deadline passed. The loop only broke early when the process exited, so the
fast-failure path worked and the success path always cost the full second.
That second was ~1015 ms of a ~1040 ms publish, and it emitted no spans, so
the timings never contradicted the assumption that it was process spawn.
Wait for the endpoint to accept a connection instead. That is also the
question the caller needs answered, since it is about to hand the URI to
something that will connect to it — where 'still alive after a second' only
correlates with it. A nbdkit that exits during startup still fails the mount
immediately, and one that never comes up now fails on a deadline rather than
being assumed healthy.
Falsified: reintroducing a fixed sleep in pollUntil fails both the readiness
timing assertion and the fast-failure one.
* test: measure the data path, not just the control verbs
The conformance benchmarks price create, publish and describe. Guest
I/O never crosses those verbs, so none of those numbers say anything
about what a guest experiences. This drives the published export with
qemu-img bench, which sits on the same block layer virtio-blk does.
It asserts only that the I/O completes and logs a table: what a given
store is capable of is not a contract, and pinning a number here would
only produce a flaky test.
Read profiles stay inside the region the write profiles covered, or
they would be timing how fast the export answers with zeroes, and one
write profile flushes, because an unflushed write measures how fast the
WAL accepts data rather than how fast the store takes it.
Profiles are spaced apart because the plugin tears an engine down per
disconnect and removes the volume's local directory as it goes, so a
connection opened while the previous one is closing fails its
handshake.
* test: run data-path profiles back to back
The spacing was a workaround for a volume lock that did not hold across
close. With that fixed, consecutive profiles are worth more than the
workaround was: each is its own NBD connection, so a run that completes
has also shown that an arriving connection waits out the previous one's
close rather than racing it.
* fix: refuse a delete that cannot establish whether a volume is published
volumeMountedElsewhere returned false on every failure path, including a
request timeout, so a busy or partitioned owner was indistinguishable
from no owner at all and the delete proceeded. Deleting a published
volume leaves the nbdkit export and the guest writing through it alive
over storage that is going away.
Report the three outcomes separately. NATS distinguishes no-responders
from no-answer, so only the former means unmounted; an unanswered probe
is refused and the caller may retry once the owner is reachable.
* feat: exclude a second viperblock engine with a JetStream volume lease
Nothing in the stack stopped two viperblock engines opening one volume from
different nodes. For an encrypted volume that is not just a consistency risk:
the AES-GCM nonce is fully consumed by SeqNum, VolumeUUID and a domain byte,
and the SeqNum window claim is a read-then-write with no CAS, so two engines
opening concurrently issue overlapping nonces under the same key.
Every engine open now takes a lease first, keyed by volume in a JetStream KV
bucket. A lease is a KV create, which JetStream serialises, so a second
claimant loses instead of both winning. The revision that won is the
generation: monotonic across the bucket, so a later opener always carries a
higher one than the writer it displaced. Entries carry a TTL and a renewal
loop, so a node that dies mid-mount does not wedge its volumes forever.
Losing a lease is logged, not enforced. Until the object store can refuse a
stale writer's PUT, failing live guest I/O on lease loss would add a failure
mode without stopping the zombie it is aimed at.
A daemon with no lease store refuses to open rather than opening blind, so
nbdkit recovery binds the store before it adopts surviving exports. Repeat
acquisitions on one node share a lease: the unmount seal opens a detached
engine while the mount being torn down still holds it, and cross-node
exclusion is what the lease is for. A restart reclaims this node's own stale
entries, since the exports that outlived the daemon are still running.
* refactor: consume masterkey from bluebottle in the test this branch added
* feat(ebsprovider): classify refusals a caller may retry
Every ErrorCode said what went wrong and none said whether the same
request could succeed later, so a caller could not tell a permanent
refusal from a transient one. Transient conditions shipped as internal,
which tells a caller to give up when it should retry.
Adds ErrUnavailable / ErrorCodeUnavailable and ErrorCode.Retryable(),
which callers branch on instead of on the code itself. DeleteVolume's
"could not establish whether the volume is published" refusal moves off
internal onto it: the owner that did not answer may answer later.
TestErrorCodeRetryable names every code, so adding one without deciding
its retry meaning fails rather than silently defaulting to permanent.
TestClassifyError pins the sentinel-to-wire-code mapping, which had no
coverage at all; a sentinel missing from classifyError falls through to
internal and loses both the reason and the retry signal.
* fix(ebsprovider): drop CopyOnWriteClone, which no request can ask for
Four providers advertised the capability and nothing consumed it: there
is no clone verb on the interface. CreateVolume's SourceSnapshotID
restores from a snapshot and CopySnapshot duplicates snapshot metadata;
neither is a volume clone, and the field documenting SourceSnapshotID
says so explicitly. CSI expresses clone as a volume_content_source with
a volume source, which this contract does not implement.
Advertising it told a caller a feature was available and then gave them
no way to use it. Removed rather than backed by a method: adding a clone
verb is a feature, not a contract fix, and the capability can come back
alongside it. The Capabilities doc comment now states the rule, and
qemunbdd's stale comment about sparse extents is corrected while there.
* feat(ebsprovider): add ListSnapshots so snapshots are reconcilable
The contract could create, copy and delete snapshots but never say which
ones it held. That left the control-plane metadata document as the only
index of them, so losing one stranded the data it pointed at with no way
to find it again. This is the same exposure ListVolumes closed for
volumes, and it stayed invisible because ListSnapshots is optional in
CSI too.
ListSnapshots pages exactly as ListVolumes does, gated on a new
SnapshotEnumeration capability. SnapshotRef is deliberately not a
Snapshot: enumeration answers what exists, and reading a snapshot's size
or state can cost far more than listing it.
viperblockd walks the bucket's snap- prefixes, sharing the top-level
walk with ListVolumes. It leaves SourceVolumeID empty because the link
lives inside the snapshot's own encrypted state and reading it would
mean a fetch per snapshot listed. qemunbdd walks its snapshots
directory rather than its in-memory map, so a snapshot that outlived the
process still appears; its SourceVolumeID is empty in that case for the
same reason CreateSnapshot refuses to reuse the ID.
Five conformance subtests cover it, and they run green against
MemoryProvider, qemunbdd and viperblockd over a real predastore.
* feat(ebsprovider): state the exclusion guarantee in the contract
A caller could not tell which providers prevent two writers from holding
one volume, at what scope, or for how long. The guarantee existed in
viperblockd and was invisible from the outside, which is why the
conformance suite had no concurrency dimension at all.
Capabilities.Exclusion states it, and deliberately separates three
questions the contract used to run together: how far exclusion reaches
(none, node, cluster), whether a dead owner's claim ever expires, and
whether a writer that lost its claim is actually stopped. That last one
matters most and is the least flattering: exclusion refuses the second
opener, fencing stops the first. viperblockd advertises cluster scope
with a 45s TTL and FencesLostClaim false, because a partitioned node
whose lease lapsed keeps writing until the object store can refuse it.
Four new conformance subtests. Two required a second viperblockd node in
the harness: cluster exclusion cannot be exercised by one node, because
the second publish goes to a subject nobody serves and times out rather
than being refused. Both nodes share the NATS the leases live in and the
bucket the volumes live in, so a refusal comes from the lease.
Standing that second node up immediately found a real gap, in a test
that had been silently skipping for want of one. A publish refused by
another node's lease came back as internal rather than volume_in_use, so
a caller was told something had broken when the answer was that someone
else held the volume. Mapped in handlePublishVolume.
* build: pin viperblock to the release carrying the provider's dependencies
The branch has been building against a local replace pointing at a
viperblock worktree, so its committed pin never actually built: it named
a version without CopySnapshotMeta, ErrSnapshotVolumeMismatch,
NewStateReader or s3.NewHTTPClient, all of which this branch calls.
Points at the merged viperblock dev instead and drops the replace.
GOWORK=off now builds clean, which is the deploy path ansible uses.
* fix: let viperblockd's delete sweep tolerate an object another sweep removed
deleteObjectPrefix claimed to mirror handlers/ec2/volume's deleteS3Prefix and
give DeleteVolume and DeleteSnapshot idempotent-when-absent behaviour, but it
returned on any DeleteObject error. Two overlapping deletes for one volume left
the loser reporting an internal failure once the winner had removed a chunk
between the list and the delete.
The control-plane copy that carried this fix is gone with the embedded engine
path, so the provider path reintroduced the bug it had just fixed.
* fix: stop CreateVolume leaving local state on whichever node served it
A state read prefers this node's local copy over the backend, and provider RPCs
are not owner-routed, so the node that happened to serve a create kept a copy
that answered describe as if the volume still existed after another node
deleted it. The two-node conformance harness caught it as DeleteVolume/success
failing about one run in three.
The seeded path already removed its local files after closing. The unseeded
path saved state and left them, which was the only difference between the two.
* fix: guard the mount path itself against a second export of one volume
handlePublishVolume refused a volume this node already exported, but
mountVolume did not, and the legacy ebs.<node>.mount subject is the route
production takes for instance launch and hot-attach. A retried attach racing a
fresh one started two real nbdkit against one volume, with no refusal and no
warning: the volume lease only refuses a different node, and a same-node
re-acquire finds the volume already held and increments its refcount.
That also made the advertised exclusion inaccurate. viperblockd claims
cluster-scoped exclusion on the grounds that a second opener is refused, while
the conformance subtests reach it only through PublishVolume, so they passed
with the legacy route unguarded. Moving the check into mountVolume gives both
entry points the same behaviour and makes the claim true for the path
production uses.
* fix: tolerate a missing legacy AMI config on deregister
DeregisterImage deletes the AMI document, then the pre-provider
{imageID}/config.json. Predastore answers a delete of an absent key with
NoSuchKey, and that was treated as fatal, so every AMI created since the
metadata store landed failed to deregister after its document was already
gone. A retrying caller then saw InvalidAMIID.NotFound for an image it had
just been told it could not delete.
The regression test uses a store that answers a missing-key delete the way
Predastore does; MemoryObjectStore silently succeeds, which is why the unit
suite stayed green.
* fix: reap ENIs left behind by an abandoned instance launch
A launch creates the instance's ENI and then attaches it. When the daemon
dies in between, the record survives with no instance and nothing ever looks
at it again. The security group dependency check counts an ENI that lists the
group, so the group cannot be deleted for as long as the record exists.
The in-process launch path already rolls back on error; a process that dies
cannot. This adds a cluster-wide sweep over records that are launch-created,
never attached, and older than the launch RPC timeout by a wide margin. The
description prefix is now a shared constant because it is the only thing
separating an abandoned launch from a standalone ENI a caller asked for.
deleteS3Prefix returned on any DeleteObject error, so two overlapping deletes
for one volume left the loser reporting ServerInternal once the winner had
removed a chunk between the list and the delete. DeleteDBInstance surfaced that
as HTTP 500.
S3ObjectStore.DeleteObject now reports a missing key as NoSuchKeyError, matching
GetObject and HeadObject, and the sweep treats an already-deleted object as
done. S3 answers 204 here but Predastore answers 404, and both mean the same
thing to a delete sweep.
The daemon only called EnableOVNIPSec when network.ipsec_enabled was true, so
the disabled case was never reached and nothing ever turned the host's IPsec
services off. Since setup.sh installs openvswitch-ipsec and strongswan-charon
unconditionally and the packages enable their units, charon listened on
0.0.0.0:500 and 0.0.0.0:4500 on every node -- including single-node clusters,
where EnableOVNIPSec short-circuits and IPsec is never switched on at all.
ReconcileOVNIPSec runs on every startup and resolves the state both ways. It
self-heals: a single-node box that later joins an IPsec-on cluster gets the
services back through the same call, which a static install-time mask could not
do because setup.sh runs before the topology is known.
The unit changes go through a new fixed-verb root helper rather than systemctl.
spinifex-daemon holds three narrow sudo grants and none of them is systemctl,
which is deliberate: a NOPASSWD systemctl rule takes an arbitrary unit name and
is root-equivalent. The helper takes only on|off and names the units itself,
matching the existing spinifex-set-endpoint-sysctl pattern. Only the read-only
is-enabled and is-active probes are run directly, and a test pins that.
strongswan-starter.service is disabled in both states. ovs-monitor-ipsec execs
the strongSwan starter itself rather than going through that unit, so it only
ever competes for UDP 500/4500 -- confirmed on a live IPsec-enabled cluster,
where charon's parent is /usr/lib/ipsec/starter and the unit is inactive.
A nil cluster config leaves the services alone rather than guessing.
This is the bind-address half only. On a multi-node cluster charon must run,
and it still binds the wildcard; restricting that reach to the encap plane is
the host firewall's job.
The inventory was missing northstar (53 on the advertise address, 5300 on the
wildcard and dialled cross-node by every node's per-instance DNS shim), charon
(500/4500 plus ESP), Geneve 6081, the per-instance IMDS and VPC-DNS listeners,
and chronyd. Section 6 asks the operator to compare the table against ss -tulnp,
which was not possible while six listeners were absent.
Two classifications were wrong. 8443 was Cluster, but predastore binds the S3
gate to 0.0.0.0 deliberately because S3 is a public plane. 4432 was External,
but it binds the node's --bind address and is transient.
The scope vocabulary predated the three-plane model, so it could not express the
encap plane or record that a node with fewer NICs collapses the planes onto its
public address. Cluster and Encap now state intent rather than a guarantee.
Replaces the reference nftables ruleset. The previous one had no conntrack,
loopback, ICMP or ime-* rules and ended in a blanket drop, so an operator who
applied it lost guest metadata, guest DNS, all egress return traffic and their
own SSH session. The replacement is input-only in a dedicated table, because
nftables verdicts are per-table and a default-deny forward chain would override
the accepts vpcd writes to ip filter and break every routed-NAT instance.
* fix: drain guests on a plain target stop, not just restart
`systemctl is-system-running` cannot tell a plain `systemctl stop
spinifex.target` apart from the stop half of a restart -- both report
`running`, and the old `--only-if-host-stopping` gate skipped the drain
for both. That left guests running with no QMP shutdown and no volume
unmount while predastore/viperblock stopped underneath them, so a real
stop discarded any writes still in flight.
Add a second signal: `systemctl list-jobs` shows a pending START job for
spinifex.target during a restart (the target's stop leg resolves
instantly, since it owns no processes) or a pending RESTART job for
spinifex-shutdown.service itself when that unit is restarted directly
(PartOf propagates target-to-member only). Renamed the flag to
`--unless-restarting` to match: drain on a real shutdown or a plain
stop, skip only when one of those jobs shows a restart is already
queued. Any step systemctl can't answer fails toward draining, and a
skipped drain now warns loudly (naming each instance still running) so
an operator sees the risk in the journal instead of discovering it via
`ps`. `--only-if-host-stopping` stays as a deprecated, still-functional
alias so a partial upgrade between unit file and binary doesn't break.
* fix: retry the local drain GATE/DRAIN ACK instead of failing on first miss
A drain that fails at GATE is indistinguishable from a skipped drain from
the guest's perspective: it keeps running while storage goes away
underneath it. Stopping spinifex.target shortly after a restart hit exactly
this: the daemon had not finished re-subscribing to NATS, so the ACK
request came back with zero subscribers and NATS bounced it back near-
instantly ("no responders"), which is why the failure logged in about a
second despite --timeout=120s. collectLocalShutdownACK now retries with
backoff bounded by the phase timeout instead of failing on the first bounce,
and an exhausted retry budget logs at ERROR and names the guests still
running via a severity-aware refactor of warnIfGuestsLeftRunning, so a
failed drain is unmistakable in the journal rather than silent.
* fix: reap idle nbdkit concurrently and re-derive viperblock's stop budget
KillMode=process made viperblockd the sole reaper of nbdkit on SIGTERM, but
shutdownVolumes reaped them one at a time and TimeoutStopSec was never
re-derived for that change. With several mounted volumes the serial loop
blew past the 30s budget, systemd SIGKILLed viperblockd mid-loop, and
whichever nbdkit hadn't been reaped yet was orphaned.
Fan the per-volume utils.KillProcess calls out concurrently so the reap
costs one worst-case nbdkit, not N of them summed, and raise
TimeoutStopSec to 150s to strictly exceed that single-volume worst case
(120s KillProcess grace + nbdkit's own 20s drain + headroom). Add an
invariant asserting the two stay in that relationship, and a test proving
the reap is actually concurrent (fails against the old serial loop).
* docs: document the host and guest lifecycle contract
States what happens to running guests for every host lifecycle event: reboot, target restart, target stop, daemon restart, daemon crash, power loss and a coordinated cluster shutdown. Records the older-release behaviour where a plain target stop left guests with no data path, and how to tell which behaviour a node has.
make test-integration cats the AWS model conformance report immediately after
the test run, but the report is written by the test binary itself. Go's test
cache means an unchanged package reports "ok (cached)" without executing, so
no report is written and the cat fails the target with exit 1.
Any branch whose changes do not invalidate that package's cache hits it, which
is why it fires on unrelated branches -- a docs-only branch cannot break a build
yet failed the same way.
-count=1 forces the run. That also keeps the conformance gate honest: a cached
result would mean the check never ran against this commit. Making the cat
tolerant instead would let the gate silently no-op.
The bare-checkout build on the CI runner resolves these from go.mod alone,
where the tagged v1.16.0 releases still carry the pre-bluebottle masterkey
and otelsetup layouts.
otelsetup/otelsetup.go and otelsetup/slog.go were a near-copy of the shared
bootstrap. Entrypoints now call bluebottle directly, passing
WithEnvironmentFallback for SPINIFEX_CI_ENV; the package keeps only the HTTP
middleware and request metrics. Init reattaches the OTLP bridge to a default
logger installed before it ran, so the local addFanoutHandler is no longer
needed.
The bridge is now level-gated, so services no longer ship Debug records to
the sink while running at Info.
Takes iampolicy, masterkey and sigv4 from github.com/mulgadc/bluebottle rather
than predastore, so the control plane no longer depends on the S3 server for
shared auth and policy code.
setup-ovn.sh set the OVN Northbound and Southbound client remotes with a bare
set-connection ptcp:6641 / ptcp:6642. The address-free ptcp: form binds every
interface, so on a multi-plane node the unauthenticated, unencrypted OVSDB
control plane was reachable from the public interface. Write access to the
Southbound database is control of the whole tenant network fabric.
ovsdb-server takes remotes from two additive sources: the Connection table,
which RAFT replicates cluster-wide, and local ovn-ctl command-line flags. That
split is what the fix uses. Loopback goes in the replicated row, because
127.0.0.1 is correct on every node, so replication is harmless and local
clients keep working. The lan address goes on the per-node command line via
--db-nb-addr and --db-nb-create-insecure-remote, which never replicates, so
each node binds its own address without pushing it to its peers.
A new --lan-addr flag carries that address, falling back to
--db-cluster-local-addr so clustered nodes stay correct without caller changes.
The standalone branch now writes /etc/default/ovn-central too, since a two-node
install still has a remote ovn-controller that needs the lan listener. The
guard on --db-nb-addr is load-bearing: it defaults to 0.0.0.0, so emitting
create-insecure-remote without a resolved address would recreate the bug.
install-node.sh passes the lan address on both branches and warns when the lan
plane has collapsed onto the public address, since the listener then lands on
the public interface and a firewall is the only remaining control.
* feat: render predastore templates with the cluster and S3 planes split
The templates set one bind_addr per host, which put raft and blob wherever
S3 was published. A host now binds the cluster plane on the address peers
dial and the gate carries its own bind_addr for the S3 API, so the installer
can hand S3 the WAN without exposing replication with it.
The single-node template also drops to one node of each role. Extra blob
nodes only hash-partition keys across more badger indexes on the same drive,
and extra meta replicas add fsyncs for a quorum that dies with the machine.
* docs: regenerate the AWS model operation coverage table
* fix: repin predastore to the merged dev commit
Rework the frontmatter titles and descriptions for the five Identity
docs so each leads with the AWS concept term users actually search, and
add contextual in-prose links between them. Only the roles and instance
profiles page currently earns search impressions, so it now links to all
four siblings and each sibling links back into the cluster.
The branch imports predastore/pkg/auth, predastore/pkg/ratelimit and the
root predastore package, none of which exist in the pinned v1.15.0. Every
build that goes through the go.work workspace resolved those from the
sibling checkout and passed; the e2e binary build on the CI runner uses a
bare checkout governed by go.mod alone, and failed to resolve them.
The port tables were wrong in both directions: inter-node traffic is QUIC over
UDP rather than TCP, and the shard range 9991-9993 no longer exists. Ports are
unique within a host rather than across the cluster, so every machine uses the
same three, and a single-node install opens only the S3 port because colocated
nodes talk over an in-process pipe.
The reference architectures scripted a move of node data out of a directory
layout that no longer exists, which would have broken any install following
them. The update guide now says object storage has no migration path.
The fixture wrote a config the strict decoder rejects and built its cluster
through clusterrun, so it now renders a v1 TOML, loads it through LoadConfig and
serves under Run. Loading the real file means the fixture trips over the same
validation an install does. Its data directory holds node-<id> directly, and it
drains on cancellation rather than being abandoned at exit.
The runtime tests drove buildRuntime, serve and the old on-disk layout, all of
which predastore owns now, so they go. What replaces them covers what spinifex
still decides: the flag merge and its precedence, a host id or gate node the
config does not have, and the pid file behind Stop and Status.
Two fixes to earlier commits in this series: the host id conversion moves behind
its own guard, and the node status probes use WaitGroup.Go.
Hosts carry their nodes under [[host.node]] now, and a node's address is its
host's addr plus its own port rather than a single host address it shared. The
gate belongs to neither storage table, so it is matched explicitly and an
unknown role still warns.
The reported vocabulary follows predastore's: db_nodes and shard_nodes become
meta_nodes and blob_nodes through the daemon, the gateway and the UI. Bucket
type is gone from predastore entirely, so the field goes with it.
The facade owns every node of a host, the S3 gate among them, so the service no
longer builds a cluster runtime or a gateway of its own. Waiting for a leader,
draining the gateway and the shutdown grace all moved behind Run; pprof and the
base path have no equivalent and are gone.
spinifex.toml stays authoritative for the S3 endpoint: the host entry is
settled from the service flags before Run, and the gate is found by role rather
than by position. A host id is now required, so init writes one for single-node
installs too, where the topology was previously left implicit.
BasePath survives as the pid directory alone. Stop and Status now read the same
value start writes, which they did not before.
The decoder is strict now, so every key the reorganised predastore no longer
reads is a startup failure rather than an ignored line. Both templates move to
nested [[host.node]] tables with the gate, blob and meta roles, portless
bind_addr and addr, singular [[bucket]], and the host TLS and encryption-key
paths the schema requires. version resets to 1 with the format.
A gate node is what serves the S3 API, so a host without one answers nothing.
Ports are unique within a host rather than across the cluster: gate 8443, blob
from 6660, meta from 7660. Colocated nodes bind no socket and still need one.
The daemon dialled a /status endpoint on each meta node to report which held
raft leadership. That endpoint lived in predastore's s3db package; after the
transport collapse the meta nodes speak rpc over QUIC and the in-process pipe
and expose no HTTP surface, so the probe dialled a dead port.
NodeStatusResponse.PredastoreRole is retained and left empty. Every consumer
already guards on the empty string, so the CLI and the UI simply omit the
badge, and restoring the probe later needs no contract change.
The migration rewrote a v4 topology into v5 and recovered raft state through
predastore's clusterrun package, which no longer exists. The on-disk layout it
relocated data into has moved too: node directories are now node-<id> under the
host root, with no host-N level. Nothing migrates a pre-cutover install; it is
re-initialised from scratch.
node-reset.sh keeps directories by design, so it now removes the predastore
cluster tree outright rather than leaving the old skeleton standing.
* feat: add guardrail CRUD store and ARN support to bedrock gateway
Stage 1 of Ochre Guardrails: control-plane store for CreateGuardrail,
CreateGuardrailVersion, GetGuardrail, ListGuardrails, UpdateGuardrail,
and DeleteGuardrail, backed by a JetStream KV bucket keyed per account.
Records keep the full policy config so later stages can build the
filter engine and inference enforcement on top without re-deriving it.
* feat: add deterministic guardrail filter engine and ApplyGuardrail op
Adds the pure wordPolicy/sensitiveInformationPolicy filter (regex-detectable
PII plus a starter profanity list), scaffolds content/topic/contextualGrounding
as NONE, and exposes it all through the bedrock-runtime ApplyGuardrail route.
* feat: enforce guardrails on Converse and ConverseStream
Wire GuardrailConfig enforcement into Router.Converse and
ConverseStream via a shared enforceGuardrail helper (resolve, load,
apply the deterministic filter engine). INPUT blocks skip the
backend entirely; OUTPUT blocks or ANONYMIZE redactions rewrite the
completion before it reaches the caller. ConverseStream buffers
delta text and assesses it at block/message-stop boundaries rather
than streaming unassessed model output. Both self-host and
Anthropic-direct backends are covered automatically since the
enforcement sits above Router's per-backend dispatch. Requests
without a GuardrailConfig are unaffected. An unresolvable or foreign
guardrail fails closed with ResourceNotFoundException.
* feat: enforce guardrails on InvokeModel and InvokeModelWithResponseStream
Adds the Invoke half of Ochre guardrails (plan §5), reusing the Stage 3
enforceGuardrail helper and Stage 2 filter engine unchanged. Guardrail
selection comes from the X-Amzn-Bedrock-Guardrail{Identifier,Version}
headers. Since InvokeModel bodies are family-native/opaque at the
Router, text is extracted/reinjected per family (llama self-host,
anthropic) in the new guardrail_invoke.go. INPUT is checked before the
backend is ever called (fail closed on an unresolved guardrail, no
backend call); OUTPUT is checked after, handling block/anonymize.
InvokeModelWithResponseStream buffers the inner stream and assesses
once at end-of-stream via a guardrailInvokeStreamSource decorator,
mirroring Stage 3's guardrailStreamSource.
* fix: route ApplyGuardrail through bedrock-runtime and use guardrail-specific RNF text
The service discriminator only promoted /model/... paths from bedrock to
bedrock-runtime, so ApplyGuardrail's singular /guardrail/.../apply route fell
through to the control-plane handler and was rejected. Extend the
discriminator to also promote /guardrail/..., which can't collide with the
plural /guardrails control-plane CRUD routes.
Also give guardrail-not-found paths their own ResourceNotFoundException
message instead of the generic "could not resolve the foundation model"
wording shared with model-lookup failures.
CI builds spinifex in a workspace over viperblock and predastore only, so
northstar resolves from the proxy. Its v1.15.0 log sanitiser predates the
attribute.Value migration and does not compile against otel/log v0.21.0.
The log API moved bodies and attributes onto attribute.Value in v0.21.0, so the
otelslog bridge and the sibling log sanitisers have to move in the same window.
No spinifex source change is needed; the workspace resolves the migrated
predastore and viperblock.
* feat: parameterise bedrock endpoint create/delete by account and pinned
Widens EnsureEndpointInput/DescribeEndpointInput/DeleteEndpointInput
with AccountID and (Ensure-only) Pinned, and keys the daemon's
Ensure/Describe/Delete on the resolved account instead of the
hardcoded GlobalAccountID. An empty AccountID still resolves to
GlobalAccountID and Pinned defaults to false, so every existing
caller (resolver, reaper, evictForCapacity, the NATS gateway client)
is unchanged. This is stage 1 of provisioned throughput: it reuses
the existing launch/capacity/readiness machinery for a dedicated,
account-scoped, pinned endpoint without adding a parallel surface.
* feat(bedrock): add provisioned-throughput CRUD and PT ARN
Implements CreateProvisionedModelThroughput, Get, List, Update, and
Delete over a new bedrock-provisioned KV store, plus the
provisioned-model ARN format/parse. Create rejects provider-hosted
models and requests a pinned, account-scoped endpoint from the daemon;
Get derives status live from the pinned endpoint's state; Update
honours name changes only and rejects a served-model swap; Delete
tears down the pinned endpoint before removing the record.
EndpointProvisioner is declared in gateway_bedrock with primitive-typed
methods (handlers_bedrock already imports gateway_bedrock, so it can't
be imported back for its EndpointService interface directly); a small
adapter in handlers_bedrock bridges the two, always setting Pinned:true
and the caller's real account on every Ensure.
Does not yet wire a PT ARN into the Converse/InvokeModel inference
path (next stage).
* feat(bedrock): accept a provisioned-throughput ARN on Converse/InvokeModel
Translates a PT ARN before resolve: the gateway inference entry parses it,
loads the bedrock-provisioned record, and resolves against the pinned
endpoint for the commitment's own account rather than the GlobalAccountID
shorthand every bare modelId uses. Adds EndpointResolver.EndpointForAccount
(gateway-side interface, implemented by DynamicEndpointResolver) so the seam
stays import-cycle-safe, and threads it through vllm.go/invoke_llama.go's
resolve call sites. A foreign-account or unknown PT ARN reads as
ResourceNotFoundException, same as any other unresolvable model identifier.
* fix(bedrock): accept a provisioned-throughput ARN on streaming inference
ConverseStream and InvokeModelWithResponseStream built their router/adapter
with a nil ProvisionedStore and the GlobalAccountID-shorthand provider, so a
PT ARN was silently never translated on the streaming path even though the
non-stream Converse/InvokeModel siblings already handled it. Thread the
gateway's provisioned store through both entry points and give
Router.ConverseStream / InvokeStreamRouter.InvokeModelWithResponseStream the
same translate-before-resolve treatment Converse/InvokeModel already have.
* fix(bedrock): surface pinned endpoints in the operator CLI
Service.List only ever enumerated the GlobalAccountID key prefix, and
'endpoint describe' always looked up GlobalAccountID too, so a pinned,
account-scoped endpoint held a GPU while reading ABSENT and missing from
list. List now scans every account's records, and describe gains an
optional --account flag (default empty, preserving GlobalAccountID
resolution) to look up a specific account's endpoint. list/describe both
render the endpoint's account and pinned status.
* fix(ochre): self-heal weights stage when the recorded snapshot is gone
runStageWeights's idempotency check trusted a matching source URI without
verifying the snapshot it names still exists, so a host rebuild or volume
GC that lost the snapshot left stage permanently short-circuited on a
useless KV record until an operator ran 'weights remove' first. Inject a
weightsSnapshotChecker, verify liveness before the no-op return, and fall
through to re-materialise when the snapshot is gone. 'weights list' uses
the same checker to mark dangling entries so an operator can see which
staged models can actually serve without waiting for endpoint launch to
fail.
* docs: note weights stage self-heal and list STATUS column
Adds resolveServingAMI test cases: single match, multiple matches
selected deterministically by newest CreationDate independent of
describe-order, zero matches, and nil/empty ImageId entries skipped.
A binary-swap deploy (make deploy or a manual spx copy) ships a new binary
whose host-config prerequisites — the privileged helper scripts and their
sudoers grant, placed only by setup.sh / setup-ovn.sh — it does not ship.
The node looks healthy and the gap surfaces later as a cryptic launch
failure or a silent security regression.
Add a preflight package that checks the host against what this build
expects, comparing asset content, not existence, so a stale-but-present
asset is caught too. The managed helpers install from quoted heredocs, so
their on-disk bytes equal the heredoc body verbatim; the binary carries
the canonical sha256s and compares.
- spinifex/preflight: CheckHost / CheckHostAt, content hash of the two
managed helpers and a NOPASSWD grant-line check for the sysctl helper.
- Generated canonicalHashes plus a drift guard that re-extracts the
heredocs and fails if the committed manifest no longer matches.
- Three surfaces: spx admin preflight, a make deploy gate
(SKIP_ASSET_PREFLIGHT=1 to bypass), and a non-fatal daemon boot warning.
Release reported "nothing was claimed" as an error, and the terminated-instance
teardown reaper persisted that as the gpu teardown mark. The mark could then
never reach done, so the record was never purged and the GC re-drove it every
two minutes for as long as it existed, failing identically each time because
the condition being retried was that the work had already been done.
Stop-then-terminate is ordinary: the stop releases the GPU and the later
terminate finds nothing left to release. Two instances on wattle had been in
this state for days, at 2251 and 640 retries, logging an ERROR naming the
node's only GPU and falsely claiming it might need a manual rebind.
Release now returns a matchable ErrNoGPUClaimed and the teardown cleaner reads
it as success, which is what the reaper already documents its cleaners to do.
The sentinel stays an error rather than becoming nil so the launch-rollback
path can still tell that its claim/release pair is out of step, and so a real
vfio rebind failure - which clears Available, the health signal the Bedrock
capacity gate reads - stays distinct from the benign case.
checkCapacity read only PoolEntry.Available, but that field tracks device
health: Release clears it when a vfio rebind fails, so a device that cannot
be handed back is not offered again. Occupancy is the separate InstanceID
field, which PoolEntry documents as empty when free, and gpu.Manager.Claim
tests both before taking a device.
Reading Available alone let a GPU already held by a running serving VM admit.
Ensure never took its capacity refusal, so evictForCapacity was never called
and LRU eviction never fired: on a single-GPU node the launch instead
proceeded to STARTING and failed later with InsufficientInstanceCapacity,
with no eviction attempted and no reason logged.
The existing table had a case named 'already claimed' that set Available to
false, which is not the shape Claim leaves behind. That encoded the wrong
model and is why the gap went unnoticed, so it is renamed to what it really
covers and the real claimed-device shapes are added alongside.
Also stop daemonGPUSnapshotter capturing d.gpuManager by value.
applyGPUConfig swaps that pointer on a passthrough toggle and re-points
daemonGPUClaimer but not the snapshotter, which would leave bedrock reading
an orphaned pool that sees no claims and admits everything.
The catalog carried exactly one self-host entry, so two models could never
contend for a GPU: an Ensure for the model already running returns the
existing record before reaching the capacity check. LRU eviction was
therefore built and unit-tested but unable to fire in production.
Llama 3.2 3B Instruct is a real AWS-published Bedrock model ID, so it keeps
the drop-in-SDK promise, and it is a genuine quality tier above the 1B model
rather than a fixture added only to unblock a test.
Staying inside the Meta family is deliberate. InvokeRouter dispatches
self-host requests to llamaInvokeAdapter on tier alone, so a model of another
family would be served with Llama's native request and response schema
instead of being refused. A new test pins that assumption to the catalog so
it fails loudly rather than silently mis-serving.
Sizing for the RTX A1000: 3.21B params at bf16 is roughly 6124 MiB of weights
against 8188 MiB of VRAM, so --enforce-eager skips CUDA graph capture to free
several hundred MiB and --max-model-len is half the 1B entry's. fp8 is not an
option on this GPU: GA107 is compute capability 8.6, and fp8 needs sm_89.
These figures are estimates to be corrected against a live reading.
Weights gating is now measurably per-entry rather than all-or-nothing, which
only became observable once two self-host entries could disagree.
Three gaps found while verifying idle reclaim on live hardware.
The reclaim log line passed a time.Duration, which slog renders as an int64
nanosecond count in JSON: the line read idleFor:135000000000 rather than 2m15s,
in journalctl and in the metrics sink alike. It now logs an integer second
count under a unit-suffixed field, which is both readable and still
aggregatable — a string would have cost the latter. The same shape appears at
around ten other sites across the fleet and is tracked separately.
The endpoint command's help still told operators the gateway could not request
endpoints itself, which stopped being true when dynamic resolution landed. It
now describes the lifecycle that actually runs, and says what the manual
commands are still for.
describe showed nothing the idle sweep observed, so an operator asking why an
endpoint was or was not reclaimed had to read the KV. It now reports in-flight
count, last activity, idle duration, pinned status and consecutive scrape
failures for a READY endpoint, and nothing for any other state, since no other
state is swept.
LastActive moves onto EndpointRecord so the idle clock, the LRU eviction order
and this output all read one definition rather than three.
A refused launch and an idle GPU could coexist indefinitely: capacity said no
while a device sat under a model nobody was calling. Ensure now makes room once
before giving up, choosing the least recently used endpoint that is safe to
stop — READY, nothing in flight, not pinned, past minResidency, and on this
node, since GPU capacity is node-local and stopping an endpoint elsewhere frees
nothing here.
The eviction goes through Delete, so the device is released by the same
DRAINING -> terminate -> release path an operator delete takes rather than
being taken from underneath a running process. Nothing evictable returns the
original ModelNotReadyException unchanged: capacity contention breaches no
quota and must not claim to.
minResidency defaults to 5m so two models cannot thrash each other with a full
cold boot per call. Beyond that the deployment is under-provisioned, and the
eviction rate is what says so.
Also adds coldStartWait (OCHRE_COLD_START_WAIT), default 0. At zero the
fail-fast contract is unchanged and costs no extra describe. Above zero the
resolver polls inside its existing singleflight, so a burst of cold callers
shares one wait and one launch. The default stays 0 because the measurement
behind it has not changed: cold start is minutes, and no client retry policy
spans that.
A launched model held its GPU until an operator deleted the endpoint. On a
single-GPU deployment that made every other model in the catalog permanently
unreachable, with no path to recovery that did not involve a human.
The reaper scrapes each READY endpoint's vLLM Prometheus /metrics and reads
idleness from the serving process itself rather than from gateway-reported
activity, which keeps the daemon the single writer of endpoint state and works
unchanged with any number of gateways. Idle means both scheduler queues are
empty and the completed-request counter has not advanced since the previous
sweep: the counter clause is what catches a request that started and finished
entirely between two sweeps.
Two rules bound the reclaim. An endpoint is never reaped within idleTTL of
readyAt, so it cannot be taken back between reaching READY and the request that
launched it arriving. A failed scrape is unknown, never idle; five consecutive
failures terminate and relaunch, so a wedged VM is not mistaken for a quiet one
and left holding a device.
Observations are persisted on the record rather than kept in the reaper's
memory: a leader change must not reset the idle clock, and the eviction
decision that follows in the next change runs on whichever replica the queue
group picked. A sweep writes only when what it saw actually changed.
The inference path resolved self-host models from a static pinned map and
never asked for a launch, so on any deployment that had not pinned an endpoint
by hand a staged model was advertised, accepted by the access gate, and then
rejected by every invoke forever with no VM ever started.
Add a resolver over the endpoint registry, satisfying the existing
EndpointResolver interface so the provider and router layers are unchanged.
It lives in handlers/bedrock rather than gateway/bedrock because that package
cannot import this one; restating the subjects and payloads gateway-side would
be two definitions of one wire contract, free to drift.
Resolution order keeps the escape hatch first: OCHRE_VLLM_ENDPOINTS wins, then
a short-TTL cache of READY base URLs, then describe. STARTING and DRAINING do
not resolve and do not re-request, since a launch or a teardown is already in
flight. ABSENT requests one and reports not-resolved, which the invoke paths
turn into a retryable ModelNotReadyException — the cold-start contract.
Concurrent cold calls for one model collapse through singleflight, so a burst
costs one describe and one ensure rather than a pair per request. Describe and
ensure errors propagate rather than being flattened into a cold start, so a
dead control plane stays distinguishable from a model that is merely starting,
and the daemon's own refusals keep their AWS codes.
finalizeTerminated treated a refused terminated -> terminated transition as a
failure, but reaching the state we were driving to is convergence: the VM GC
had reconciled the same instance while terminateCleanup was running.
The caller pays for that error. Observed on wattle: an Ochre endpoint delete
wrote DRAINING, lost this race, and returned before the DRAINING -> ABSENT
write, so the VM was gone and the GPU released while the record stayed
DRAINING. Delete only accepted READY, so no retry could clear it and the model
was unreachable until the KV was edited by hand. ELB, EKS and RDS system
instances race the same GC.
So converge in the vm layer when the instance did reach terminated, keeping
the refusal for one that did not, and let Delete resume from DRAINING: a
teardown that failed partway has to be retryable.
* feat(bedrock): gate Ochre models behind per-account access grants
Self-host models were invokable by any authenticated account: tieredCatalog
returned them unconditionally and no runtime router checked the caller.
GetFoundationModel also described models the caller could not use.
Add a bedrock-model-access JetStream KV bucket holding presence-only grants,
resolved through a new AccessResolver. Access is deny-by-default: a gateway
with no store configured falls back to DenyAllAccessResolver, so an
unconfigured deployment serves nothing rather than everything. The system
account bypasses grants, matching how quotas exempt it.
All four runtime routers share one grantedCatalogEntry gate so they cannot
drift on who may invoke what. List and describe agree: an ungranted model is
absent from ListFoundationModels and reported as ResourceNotFoundException by
GetFoundationModel, so the error cannot confirm a model exists.
Administration lands on two surfaces: Grant/Revoke/ListModelAccess actions on
the spinifex gateway service for the UI, and `spx admin ochre access` for
operators and provisioning, which opens the bucket over NATS directly rather
than requiring a SigV4 client the CLI does not have.
* feat(bedrock): seed the admin account's model grants at gateway start
Deny-by-default left a fresh install with a catalog nobody could see, and the
first cut filled that gap from ansible — which is a dev tool, so any install
that never ran it came up unusable.
Seed from the product instead: on start awsgw grants the platform admin
account the full catalog, once, behind a marker key in the same bucket. The
marker makes it a bootstrap step rather than a policy: tenant accounts still
begin with no access, and an operator who revokes a grant does not have it
restored by the next restart. Grants are written before the marker so a crash
midway repeats the idempotent grants rather than leaving a half-seeded
account, and the marker is a conditional create so only the first node to
reach it need win.
Seeding is best-effort: a KV failure logs and leaves the gateway serving,
and the next start retries because the marker is written only on success.
* test(bedrock): cover the model-access CLI and gateway admin actions
Route the access CLI runners through the same loadConfigAndConnectFn /
ochreExit seam the weights runners use, so grant, revoke and list can be
driven against an embedded JetStream server, and add gateway tests for the
grant admin actions including the non-admin denial.
* feat(images): add a vllm-serving AMI as a stacked mkosi profile
LaunchServingVM resolves its AMI by the tags spinifex:managed-by=bedrock and
spinifex:bedrock-role=vllm-serving. No image carried either, so every Ensure
failed at resolveServingAMI. The only GPU image ships the NVIDIA driver and
Docker but no Python, no vLLM and no serving unit, so tagging it would have
moved the failure from admission to first boot.
Add a vllm profile that stacks onto gpu-nvidia and owns the whole guest side
of the launcher contract: the runtime, vllm-serve.service, the weights mount
and the tags. vLLM is baked into a uv-managed venv rather than pulled as a
container at boot — a runtime docker pull of vllm/vllm-openai is known to
stall this platform's write plane, and mkosi's unprivileged build has no
daemon to bake an image from anyway.
Also bring images/mkosi.profiles/ under make test-images. Its scripts have no
.sh suffix, so the existing name glob skipped exactly the files worth linting;
sources are found by shebang instead.
* fix(images): make the vllm postinst survive the build sandbox
Two failures found by actually running the build.
uv installs to /usr/local/bin, which mkosi's postinst PATH need not carry, so
`command -v uv` reported a good install as missing. Call it by absolute path,
and check it the same way.
`vllm --version` initialises device config and dies with "Failed to infer
device type" on a builder with no GPU, so it could never have verified
anything. Assert the pinned version through package metadata instead, which
also catches a version mismatch rather than merely a broken entrypoint.
* fix(images): pin the serving AMI datasource and mirror cloud-init to the console
A serving VM boots through LaunchSystemInstance, so it comes up dual-NIC with a
fw_cfg netcfg blob and takes its user-data solely from IMDS — the same shape as
the EKS control plane. That path carries a cloud-init datasource pin; the vllm
profile never got one, and the base config deliberately leaves datasource_list
unpinned, so the serving image was the only system VM booting without it.
Reuse the EKS pin rather than copying it. The 99- prefix is load-bearing:
read_conf_d() applies cloud.cfg.d in REVERSE and mergemanydict() is
first-source-wins, so a 90- name loses to cloud-init's own 90_dpkg.cfg ('_'
sorts above '-') and the pin is discarded in silence. The postinst asserts the
MERGED datasource_list via cloud-init's own read_conf_with_confd rather than
checking the file exists, because it existed at 90- too and did nothing there.
Also mirror every cloud-init stage to the serial console. A serving VM is
headless, and a guest that wedges before handover never surrenders
/var/log/cloud-init-output.log, so the console get-console-output reads is the
only channel an operator has.
The images_test.go case covers the two tags resolveServingAMI filters on; it
matches on those alone and never on name, so a typo hides the image entirely.
* fix(images): make the dhcpcd data-NIC lease genuinely one-shot
dhcp_oneshot() promised to configure the link and exit "leaving no daemon
to fight cloud-init", but the dhcpcd branch passed only -q, which is quiet
rather than one-shot. Every Ubuntu image takes that branch, so a dhcpcd
daemon survived on the data NIC holding the control socket.
cloud-init's Ec2 datasource then runs its own ephemeral dhcpcd on the same
NIC. Instead of spawning, dhcpcd forwards the invocation to the resident
daemon as a control command and exits 0, and --dumplease returns the
resident daemon's lease, so discovery looks successful. Cleanup then waits
for a pid file at /run/dhcpcd/<iface>-4.pid that no process ever wrote,
spinning its full 30000 x 10ms budget: init-local stalls for exactly 300s
on every system-instance boot.
Adding -1 exits once the interface is configured and -p keeps the address
and routes across that exit. Both are needed; -1 alone tears the link back
down. The test now models the real clients — the stub stays resident unless
given the flag that makes its client exit, and the dhcpcd branch is forced
by dropping udhcpc from PATH rather than left to whether the host happens
to have /usr/share/udhcpc/default.script.
* fix(images): serve without a CUDA toolkit and stop vllm-serve crash-looping
vLLM picks FlashInfer for top-k/top-p sampling, and FlashInfer JIT-compiles
that kernel on first use. The serving image ships the driver and the
container toolkit but no nvcc, so the sampler warmup died with "Could not
find nvcc and default cuda_home='/usr/local/cuda' doesn't exist" minutes
into a GPU cold start, after the model had already loaded.
VLLM_USE_FLASHINFER_SAMPLER=0 selects vLLM's native top-k/top-p sampler,
which is the same operation with no CUDA toolchain at runtime. Everything
else on the serving path is already precompiled in the wheels: the attention
backend resolves to FLASH_ATTN and CUDA graph capture completes, both
without nvcc. Baking FlashInfer's AOT modules is the alternative if the
fused kernel is ever wanted, and it keeps the toolchain in the builder
rather than the appliance.
The unit had Restart=on-failure with no bound, so an unfixable start failure
restarted forever and buried its own first traceback under identical copies
on the serial console — the only view an operator has of a headless guest.
StartLimitIntervalSec/StartLimitBurst make it report instead.
The postinst asserts the installed unit carries the sampler setting, for the
same reason the datasource pin is asserted there: the alternative is finding
out at runtime, on a GPU, five minutes in.
Also corrects the shared datasource pin's header, which still claimed it
installs as 90-spinifex-ec2-imds.cfg. It is installed at 99- by the vllm
profile precisely because a 90- name loses the reverse-sorted, first-source-
wins cloud.cfg.d merge to cloud-init's own 90_dpkg.cfg.
* fix(bedrock): raise the serving-endpoint readiness bound to 15 minutes
The 5 minute bound was chosen before any cold start had been measured, and
its own comment asked to be corrected once a model actually ran on wattle.
It now has: Llama 3.2 1B on an RTX A1000 reached "Application startup
complete" 257.6s after launch, leaving ~42s of margin on the smallest model
this platform can serve, with a cold torch.compile accounting for 26.3s of
it. A bigger card and a bigger model are unmeasured.
The two failure modes are not symmetric. Too low tears down an endpoint that
was about to come up and reports it as a failed launch; too high only makes
a genuinely stuck launch take longer to give up. So the bound is set well
clear of the single data point rather than close to it.
The CLI's own wait moves to 20 minutes to stay above it. The daemon starts
its clock when the VM is launched, up to a minute after the CLI starts its
own, so leaving both at 15 would expire the client first and report a bare
"timed out" in place of the daemon's abort, which carries the reason.
* fix: give the Ochre daemon a routed port in the Bedrock system VPC
The readiness probe and every InvokeModel call dial a serving VM's
system-VPC address from inside the daemon, but that subnet lives in OVN
on br-int and the host has no route to it, so a healthy VM never reached
READY.
Each daemon now owns an ENI in the system VPC's private subnet, bound to
an OVS internal port on br-int by iface-id, exactly as ovn-controller
binds a guest tap. The ENI address is installed at the subnet's prefix
length, which is what supplies the connected route. BaseURL is unchanged:
the address it already carried is now reachable.
Describe-or-create keyed on a per-node description, since ENIs carry no
tag filter, so a restart or a second endpoint reuses one address instead
of leaking one per launch. The port is ensured before anything else the
launch creates: a VM the daemon cannot dial holds a GPU for the whole
readiness window before unwinding.
Both actions carried no scope, so their policy check evaluated against "*".
Matching is pattern-to-value, so a statement whose Resource is a specific ARN
never matches "*" — a Deny fencing an instance blocked the scoped
ModifyDBInstance and DeleteDBInstance but not a snapshot of the same instance.
The principal could snapshot the fenced instance, restore it under a name the
Deny does not cover, set a master password of their choosing and read the data.
Each is scoped to its source: the instance for a create, the snapshot for a
restore, which breaks the chain at its first step. Both identifiers are required
on their action and ResourceARN already falls back to "*" when absent, so this
narrows where the identifier is present and is a no-op where it is not.
Evaluating both ARNs per action, for AWS parity, is deliberately left out: it
turns the scope into a slice across all 13 scoped entries and touches the shared
dispatcher call site.
* feat: add spx admin ochre endpoint ensure/describe/list/delete
The daemon subscribes all four bedrock.endpoint.* subjects, but nothing in
production calls their NATS client, so the serving-endpoint lifecycle could
not be exercised at all in a running system. This gives an operator a direct
way to drive it.
ensure --wait polls describe until READY and reports the elapsed time, since
the daemon replies STARTING as soon as it claims the model and the launch
outlives the request. That is the measured cold start.
Also corrects the weights stage tokenizer rule in COMMANDS.md, which still
described both tokenizer files as required rather than either.
* test: cover the ochre endpoint Run wrappers' connect and exit paths
* fix: treat an empty endpoint bucket as no endpoints, not an error
ListEndpoints was the only kvutil.Keys caller not handling ErrNoKeysFound,
so 'ochre endpoint list' failed on a cluster that had never launched one.
weights stage created the snapshot at the viperblock layer only, writing
the block checkpoint and config.json but not the metadata.json that
CreateVolume resolves a SnapshotId through. The endpoint launcher clones
the weights volume via ec2.CreateVolume, so every launch failed with
InvalidSnapshot.NotFound against a snapshot that was present on disk.
The GPU model table was only ever used to report VRAM, so wrong or missing
entries were cosmetic. Bedrock's capacity check is the first code that rejects
on VRAM, which turns every latent data error into an admission decision.
Correct seven entries verified against pci.ids — most seriously 1002:7448,
labelled Instinct MI300X at 192GB when it is a Radeon Pro W7900 at 48GB, a
4x over-report that admits a model which then OOMs at weight load. Record the
smallest capacity where one PCI ID spans several cards, since under-reporting
refuses an endpoint an operator can see and override while over-reporting
fails later and looks like a vLLM bug.
Widen the table so common datacenter, workstation and consumer cards resolve
without an operator-supplied override, and apply GPUModelOverrides to the pool
devices as well as the advertised model, so the override reaches the code that
actually gates. An override with no memory_mib is left alone rather than
wiping discovered VRAM, since the config cannot distinguish it from a
deliberate zero.
MemoryMiB 0 stays the "undiscoverable" sentinel, but checkCapacity no longer
counts it as a device with no VRAM: an otherwise-free device now reports which
card is unknown and how to fix it.
* fix: make the predastore readiness gate a real readiness probe
checkPredastoreReady only did a TCP dial + close, so a listening socket
satisfied it well before predastore could serve TLS. In CI this passed
in 170 microseconds, 0.72s after predastore opened its listener, and
recovery started against a backend that wasn't serving yet — predastore
logged the probe itself as a TLS handshake EOF.
Replace the dial with an HTTPS GET against the configured predastore
endpoint on a package-level client (hoisted out of the 2s poll loop),
with a per-request 3s context timeout. Any HTTP response counts as
ready, including 403 from this deliberately unsigned request, which is
exactly what an unauthenticated S3 GET returns. Only a transport or TLS
failure means not-ready.
This narrows the race window and removes the spurious TLS-handshake-EOF
log lines predastore emits on every daemon start, but it does not close
the window on its own: predastore can serve TLS before its storage
backend answers reads correctly. The load-bearing fix is the viperblock
VBState read retry landing separately.
* style: trim the predastore readiness comments to three lines
GetPasswordData returned an empty string both when the guest had not emitted a
password yet and when it never could, because the launch key pair was ED25519
and cannot perform PKCS#1 v1.5 encryption. The two are indistinguishable, so a
caller in the second case polls forever with no signal.
An empty result on a Windows instance whose launch key is not RSA now returns
InvalidKeyPair.Type naming the cause and the fix. Every uncertainty keeps the
old empty result: a populated blob always wins, Linux is never judged, and an
unresolvable key is not treated as evidence of the wrong type.
get-password-data --priv-launch-key requires PEM, but ssh-keygen writes the
OpenSSH container by default, so even an explicitly-RSA key pair failed with
"Unable to decrypt password data using provided private key file" — which
reads like a padding problem rather than a format one. AWS returns PKCS#1 PEM
for RSA key pairs. OpenSSH reads that format for SSH either way.
The create-key-pair UI gains a key-type selector, defaulting to the rsa the
page already sent, with a hint that RSA is required for Windows password
retrieval.
The API default stays ed25519. Changing it is a separate decision.
ResourceNotFoundException is a shared wire code, so Bedrock aliases the EKS
error constant and inherited its message: a caller asking for a model was
told to list its EKS clusters. Register Bedrock overrides in the existing
errorLookupByService table, under both service keys since the gateway splits
bedrock from bedrock-runtime on the /model/ path prefix.
PutModelInvocationLoggingConfiguration also accepted embeddingDataDeliveryEnabled
and dropped it: the internal LoggingConfig had no such field, so the flag never
survived to Get. Carry it through both translation directions and fold it into
the bucket-required check, since embedding delivery with no bucket is as
undeliverable as text or image.
cloudbase-init has no IMDSv2 token support in any release, so a Windows
instance launched without an explicit MetadataOptions block 401s on every
metadata GET and never reaches its main plugin stage. Derive the launch
default from the resolved image's platform: Windows gets optional, every
other platform keeps required, and an explicit request value always wins.
* feat: add spx admin ochre weights stage/list/remove
Give operators a supported path to stage a self-hosted model's
weights: point at a predastore S3 prefix holding ordinary Hugging
Face files, and stage materialises them into a viperblock volume,
snapshots it, and records the source URI and snapshot ID in the
bedrock-weights KV bucket.
stage refuses before downloading anything if the model is not a
self-host catalog entry, or if the prefix is missing any required
file. Re-staging the same source is a no-op; a different source
replaces the entry and reports the previous snapshot for separate
reclamation.
WeightsStore gains GetWeights and a source-URI field on its record
to support this. LookupServingSpec (catalog.go) is reused unchanged
to validate --model-id.
* fix: unify ServingSpec after rebase onto dev's daemon-launcher spec
Rebasing onto dev pulled in mulga-vmfng.7.5's daemon-side ServingSpec
and LookupServingSpec, textually duplicating the ochre weights stage
CLI's own versions in catalog.go. Collapse both into one type carrying
ModelID and VLLMArgs, and one three-value LookupServingSpec so callers
distinguish an unknown model (found=false) from a provider-tier one
(selfHost=false). Update the daemon launcher's call site accordingly.
* test: cover ochre weights stage/list/remove command behaviour
Splits runOchreWeightsStage/List/Remove into testable decision functions
(runStageWeights, listWeightsOutput, removeWeights) separate from cobra and
NATS wiring, and makes mkfs.ext4 execution and the connect/exit steps
swappable so the full command flow can be driven against an embedded
JetStream server instead of live infrastructure. Also relaxes the tokenizer
file requirement to accept either tokenizer.json or tokenizer.model, since
canonical Llama 3.x repos ship only the former at the prefix root.
* fix: resolve mkfs.ext4 from sbin when it is absent from PATH
e2fsprogs installs mkfs.ext4 into sbin, which a non-login shell omits, so
weights staging failed on an unprivileged operator's otherwise fine host.
* fix: size the weights volume to a whole GiB so the tail block fits
viperblock rejects a write whose block overruns VolumeSize, so importing an
image of unaligned length failed on its last block. Rounding up also makes
the volume's byte size and its manifest SizeGiB describe the same volume.
Name the volume in the import error so a part-written import is traceable.
DescribeImages skipped any ami-* prefix whose config.json could not be
fetched, read, or parsed, logging each at DEBUG. At the daemon's default
INFO level the result was an empty Images list and a completion line
reading count=0, indistinguishable from a healthy empty registry.
Promote the three fault branches plus the body-close failure to WARN,
leave NoSuchKey at DEBUG since a prefix with no config is a legitimate
non-AMI directory, and report prefixes examined alongside images
returned so a mismatch is visible without raising the log level.
SetServiceOwnership chowned per-service directories to service users
without propagating a missing user, so spx admin init reported success
while leaving /etc/spinifex/northstar root-owned 0600. ChownRecursive
and SetServiceOwnership now return errors, failures are aggregated
across every service rather than stopping at the first, and
finalizeNodeSetup prints the unapplied paths instead of a success mark.
The per-tap datapath only matched IP, on the assumption that a guest reaches
169.254.169.254 through its default gateway. A guest that treats 169.254.0.0/16
as on-link resolves the address itself instead, and nothing on br-imds, br-int
or in OVN answered that ARP, so it could not reach IMDS at all.
Windows does exactly this, which left cloudbase-init timing out on every
metadata request and skipping the rest of its plugin stage.
Install a per-tap ARP responder for each captured address under the existing
cookie, replying with the gateway MAC the egress flow already presents.
The viperblock start command applies --s3-host, --s3-bucket and --s3-region over
spinifex.toml whenever viper reports a non-empty value. s3-bucket and s3-region
carried non-empty defaults, so the override fired on every start even though the
service unit passes neither flag nor environment variable, and the region read
from the config was replaced with ap-southeast-2.
Any cluster deployed to another region could not launch an instance: viperblockd
signed its S3 requests with ap-southeast-2, predastore verified them against its
own region and rejected them as AuthorizationHeaderMalformed, the volume mount
failed and RunInstances ended in launch_failed. The bare-metal e2e cell, which
installs with --region bare-metal, failed every instance-backed test this way.
Default all three to empty, matching s3-host, so an unset flag leaves the
configured value alone while an explicit flag or SPINIFEX_VIPERBLOCK_* still
wins. The predastore.toml template also pinned the main bucket to a literal
ap-southeast-2 while the top-level key and the northstar bucket both rendered
the configured region; it now renders it too.
Introduces handlers/bedrock, the daemon-side owner of vLLM serving VM
placement and lifecycle. State lives in a new JetStream KV bucket
(bedrock-endpoints) keyed {accountID}/{modelID}, cycling through
ABSENT -> STARTING -> READY -> DRAINING -> ABSENT. The daemon is the
sole KV writer; the gateway (or any other caller) requests transitions
over NATS via bedrock.endpoint.{ensure,describe,list,delete}, wired
into subscribeAll exactly as handlers/eks registers its own subjects.
Ensure is idempotent and single-flighted at two layers: a local
per-key mutex collapses concurrent same-process callers, and a
JetStream KV Create/CAS on the record's generation collapses races
across daemon replicas, so a burst of concurrent Ensure calls for one
model launches exactly one VM either way. Capacity is admission-checked
against gpu.Manager's free devices before a launch is claimed, so a
refusal never leaves a stuck STARTING record. Readiness is an explicit
poll of the serving VM's /v1/models endpoint until 200 or
startupTimeout, never a sleep.
Serving VMs boot via the system-instance path against a dedicated
system VPC, and follow the Phase-0 weights resolution: a snapshot
staged out-of-band is COW-cloned per endpoint into a data volume the
VM attaches post-boot, decoupled from the boot AMI.
Adds the .7.6 metering slice on top of .7.4's invocation stream: a
second durable consumer (ochre-usage-metering) aggregates per-account
token counts and external cost into a new bedrock-usage KV bucket,
keyed {accountID}/{modelID}/{period}. RequestID dedupe (an atomic
create-if-absent marker in bedrock-usage-dedupe) makes redelivery
count-neutral rather than double-billing a customer.
Prices are integer micro-USD per million tokens (D12): in-tree
catalog defaults with KV overrides in bedrock-prices, mirroring the
serving-spec shape from .7.3. Self-hosted models always resolve a
known zero; a model with neither an override nor a default resolves
Known=false, which the usage counter surfaces as a sticky CostUnknown
flag distinct from a genuine zero-cost period.
handlers_quota gains two Bedrock dimensions, both defaulting to
disabled independently of the block's existing Enabled toggle:
tokens-per-month (a check-against-history hard cap reading the
stream-fed counter, tolerant of a few seconds of lag) and
requests-per-minute (a local per-account token bucket enforced
immediately in-process, with a periodic KV sync for cross-node
observability only — enforcement never round-trips). Both are wired
into BedrockRuntime_Request ahead of every Converse/InvokeModel
variant.
* feat(ec2): serve the Windows administrator password via GetPasswordData
Adds the GetPasswordData EC2 API, mirroring the existing GetConsoleOutput
path: a per-instance NATS subscription, a daemon handler, and a gateway
shim registered in ec2Actions. The guest emits its encrypted administrator
password to the serial console at first boot; the daemon scans the whole
console log (not just the AWS 64KB tail, since the password would scroll
off a long-running instance) for the last <Password>...</Password> block
and returns its base64 ciphertext unchanged. Spinifex never handles key
material and never sees the plaintext password.
onInstanceUpHook's per-instance topic binding is refactored from two
hand-rolled subscribe-then-rollback blocks into a loop over the topic set,
so adding this third topic doesn't nest the rollback logic further and a
failure partway through still unwinds every topic already bound in that
call.
* test(ec2): cover the GetPasswordData gateway shim and remaining daemon branches
The gateway shim had no test file, leaving diff coverage below the
preflight threshold. Adds GetPasswordData_test.go covering input
validation, the happy path against a stubbed NATS responder, the
ErrNoResponders and ErrTimeout mappings to InvalidInstanceID.NotFound, a
daemon error payload, and a malformed JSON response. The request timeout
is now a package variable so the timeout test doesn't have to wait out
the real 5 seconds.
Also adds daemon-side coverage for the missing-InstanceId, malformed
input payload, and cross-account ownership-denied branches of
handleEC2GetPasswordData.
DescribeImages and DescribeInstances only ever populated PlatformDetails,
which is a billing string covering every OS. The AWS SDKs and the Terraform
provider key off the separate Platform field, which is either "windows" or
absent, so a Windows AMI read as Linux.
Derive Platform from PlatformDetails rather than persisting a new AMI field,
so already-imported Windows AMIs are fixed without a re-import.
The v4 to v5 topology migration runs as root under `spx admin upgrade`,
but predastore runs as spinifex-storage with an empty capability
bounding set. Two paths left it locked out of its own state.
The migration created the cluster data dir as root:root 0750, so
spinifex-storage could not traverse into the node dirs the rename had
placed under it, and clusterrun could not create new ones. Separately,
the raft recovery that follows each state replica move reopens badger
and writes a fresh snapshot as root, leaving root-owned files inside a
node dir that the rename had otherwise left correctly owned.
Both now take their ownership from the data being moved rather than
from a service user name, so a dev install running as the invoking user
needs no special case. The data dir chown is unconditional so a re-run
repairs a dir left behind by an earlier failed run, and the tree chown
runs even when there was nothing to recover, since that path still
opens the stores.
* feat: record every Bedrock invocation to JetStream, deliver to S3 and logs
Publishes an InvocationRecord for every Converse/InvokeModel call (streaming
or not) to a durable JetStream stream, then fans it out via a delivery
consumer to the account's own predastore bucket (body text, gated per
account by PutModelInvocationLoggingConfiguration) and a metadata-only log
line. Publish failures increment a drop counter instead of failing the
invocation. Streaming pumps record on every exit path via defer, including
client disconnect, and Close() now cancels the upstream request outright so
a disconnected client actually frees backend generation. vLLM streams
estimate output tokens from content-delta counts when real usage isn't
available yet, flagged usage_estimated; Anthropic always reports real usage.
* fix: deliver invocation records to S3 regardless of body, replicate stream
deliver() was gating the S3 write itself on InputText/OutputText being
non-empty, so an account with a destination bucket configured but body
delivery left off got nothing — the delivery flags already gate the body
fields upstream in Record(), and shouldn't gate delivery a second time.
EnsureInvocationStream defaulted to a single replica on a multi-node
cluster, unlike its sibling KV stores, so losing one node lost billing and
audit records without the control plane itself going down. Also renames
the awsgw predastore client from ecrStore to objStore now that it backs
both the ECR registry and Bedrock invocation delivery.
* feat: in-tree serving specs + KV weights override for ochre catalog
Extend catalogEntry with MinVRAMMiB, InstanceType and VLLMArgs so a
self-host model's serving requirements ship with the catalog rather
than living only in an operator's head. Add WeightsStore, a
deployment-scoped (not account-scoped) JetStream KV override in a new
bedrock-weights bucket that resolves a model's staged weights snapshot
ID, following the same shape as the existing CredentialStore. It is
plumbed into tieredCatalog and GetFoundationModel through a
package-level resolver (SetWeightsResolver/currentWeightsResolver),
since both are called from gateway/bedrock.go's fixed-arity route
table, which is out of scope here.
A self-host model with no resolvable weights snapshot is not
servable and must not be advertised: tieredCatalog filters it out and
GetFoundationModel returns ResourceNotFoundException for it.
Replace the meta.llama3-70b-instruct-v1:0 self-host entry, which
needs roughly 140 GB of VRAM in fp16 and has never actually run, with
meta.llama3-2-1b-instruct-v1:0, a real AWS-published Bedrock model
ID. At bf16 its 1.24B params are roughly 2.5 GiB of weights, well
inside wattle's single RTX A1000 (8188 MiB); MinVRAMMiB is set to
4096 to budget headroom for CUDA context, activations and KV cache on
top of that. This is an estimate, not a measurement off wattle -
.7.5's launcher should correct it against a real nvidia-smi reading
once the model is actually running there.
lookupCatalogEntry, used by the untiered invoke/converse routing
path, is left unchanged and still ignores weights resolvability;
gating that path is out of scope for this bead.
* fix: separate weights resolve errors from not-found in ochre catalog
An internal fault (e.g. a JetStream outage) reaching the KV weights
resolver was being collapsed into the same "not servable" verdict as
a clean not-found: tieredCatalog would silently unadvertise every
self-host model, and GetFoundationModel would return
ResourceNotFoundException for a model that does exist but whose
resolver is temporarily broken. Both call sites now distinguish the
two: a resolve error is logged and surfaced as its own error (mapped
to ServerInternal at the AWS error boundary), and only a clean
not-resolvable result yields ResourceNotFoundException.
tieredCatalog's signature changes to return an error alongside the
entries so ListFoundationModels can propagate a resolve fault instead
of silently returning a thinned-out list.
Also raises the self-host entry's gpu-memory-utilization from 0.5 to
0.6 (roughly 4913 MiB of vLLM's own pool) and MinVRAMMiB from 4096 to
5120 to match: at 0.5, weights (~2.5 GiB) left under 1.5 GiB for KV
cache and activations at max-model-len=8192, likely too tight for any
real concurrency. The two numbers are meant to agree: MinVRAMMiB is
the admission-gate floor, gpu-memory-utilization is what tells vLLM
to actually claim that much on an 8188 MiB device.
Comments trimmed to the repo's 3-line limit and stripped of plan/bead
references; the VRAM reasoning and model-choice rationale live in the
commit body and the plan doc instead.
Threading the weights resolver through as an explicit parameter
(mirroring CredentialResolver) rather than a package-level global was
considered per review. Doing so needs edits to gateway/bedrock.go
(bedrockRouteHandler's signature, both route closures, the
Bedrock_Request call site, a new bedrockWeightsResolver() method) and
gateway/gateway.go (a new BedrockWeights field on GatewayConfig).
Both are outside gateway/bedrock/ and are explicitly out of scope for
this change, owned by a parallel workstream already modifying the
same route table. The global resolver stays, now with the
error-handling fix above.
* fix: stop e2e GetFoundationModel regressing on the new weights gate
TestGetFoundationModel's "known model" subtest asserted on
meta.llama3-2-1b-instruct-v1:0 unconditionally, but that entry is now
gated on a weights snapshot staged in the bedrock-weights KV bucket.
On any real deployment that bucket starts empty, so the subtest would
fail with ResourceNotFoundException outside of OCHRE_E2E_SELFHOST.
The "known model" assertion now targets the Anthropic entry, which
resolves unconditionally and keeps testing what the subtest was
actually there for: a known model round-trips, an unknown one is
ResourceNotFoundException. The self-host assertion moves to a new
TestGetFoundationModelSelfHost, gated behind OCHRE_E2E_SELFHOST=1
alongside TestConverseSelfHost, where weights are expected staged.
tests/integration/bedrock_test.go's TestListFoundationModels already
wires a weights-resolver stub before asserting the self-host entry is
present, so it does not regress the same way.
* test: correct a stale assertion message on the ochre catalog weights gate
Viperblock's ordered extent index pulls in tidwall/btree, so the bump needs
the matching go.sum entry or the build fails resolving a transitive import.
Spinifex imports github.com/mulgadc/predastore/clusterrun, but go.mod still
pinned the pre-merge pseudo-version 608939ef2bb4, which predates that package.
Off-workspace builds (CI, which has no go.work) fell back to resolving the
import at @latest — v1.14.0 — and failed.
* match new predastore config surface area
* fix(migrate): carry existing installs onto the predastore v5 topology
The cluster topology refactor breaks an install three ways at once. The old
config carries a top-level host key, which the new schema binds to the [[host]]
table array, so TOML rejects the document and the daemon dies before it reads
any topology. Node state moved from distributed/{db,nodes}/node-N to a per-host
data directory keyed by a node id that is now unique across roles. And because
object metadata records which node holds each shard, the shard nodes had to
keep their ids, so the state replicas took new ones and every replica's
persisted raft configuration stopped listing it.
predastore.toml gains a 4 to 5 migration covering all three: it rewrites the
tables, relocates whichever node directories exist on this machine, and
recovers each relocated state replica onto the new member set. The rewrite is
line-oriented rather than a re-render from the template, so buckets, auth,
rate limits and iam settings an operator edited survive.
spinifex.toml gains a 3 to 4 migration renaming the predastore node_id to
host_id, which predastore-start.sh greps to select the host this machine runs.
The predastore migration lives in its own package rather than in migrate,
which every handler that stamps a KV bucket version imports; reaching the
cluster runtime from there would pull raft, badger and the QUIC client into all
of them. admin_upgrade.go blank-imports it, so only the binary that runs the
upgrade links it, and a test pins both registrations.
* test(predastore): cover the service runtime, gateway and shutdown paths
The service wrapper's new seams had no unit tests: buildRuntime, serve and
the errgroup Start runs the cluster and the S3 gateway under. CI's diff
coverage gate measured the PR at 34.6% against a 60% threshold as a result.
Cover every buildRuntime error wrap and its host selection, both ways the
gateway can fail serve, and the full lifecycle: Start brings the nodes and
the gateway up together, Shutdown drains both by cancelling the context
they share. Topologies are written per test as real config, with node ids
from a process-wide counter — the pipe transport's listener registry is
global, and this package's integration tests hold a fixture daemon on the
low ids for the life of the test binary.
setup.sh aborts twice on resolute. qemu-kvm is a virtual package with two
providers there, so apt refuses to resolve it; the token was redundant anyway
since $QEMU_PACKAGES already installs qemu-system-x86/arm and spinifex execs
the qemu-system-* binaries directly.
sudo-rs, Ubuntu's default sudo since 25.10, then rejects the sudoers file: it
implements no pam_session Defaults setting, and forbids a wildcard inside a
command argument. pam_session is now emitted only for classic sudo, and the
sysctl grant names a fixed-verb helper instead of a wildcard.
The helper is narrower than the rule it replaces: it accepts one interface, a
key restricted to rp_filter or accept_local, and a value restricted to 0 or 1,
where the old grant permitted any net.ipv4.conf key at any value. Granting
sysctl with an unconstrained argument was rejected as root-equivalent.
Adds scripts/lint-apt-packages.sh and an Apt Packages workflow so the runtime
package list is resolved against each supported suite's real indices in CI.
Spinifex rejected HttpTokens=optional and served 401 to every untokened
GET, which is stricter than AWS, whose own default is optional. That made
Windows guests impossible: cloudbase-init is the only realistic Windows
bootstrap agent and its EC2Service has no IMDSv2 token support in any
released version, so a Windows instance never received its hostname,
injected password or user-data.
HttpTokens is now settable at launch and via ModifyInstanceMetadataOptions,
accepting optional or required. The launch default stays required, so an
instance that does not ask for IMDSv1 never gets it. Every other
metadata-option field keeps its existing rejection.
The untokened path is reachable without credentials, so the per-instance
lookup is cached with a short TTL; without that a guest could drive
unbounded DescribeInstances fan-outs. A valid token short-circuits before
the lookup, so compliant IMDSv2 clients are unaffected.
* feat: spread HA control-plane placement across distinct AZs first
SchedulableHosts now orders eligible nodes by AZ round-robin (one host
per AZ before repeating any AZ) instead of raw fan-out arrival order.
ReserveSpreadNodes takes a strict order-preserving prefix of that list,
so an N-way HA reservation lands on N distinct AZs whenever that many
exist, degrading to today's node-only spread when AZ data is uniform
or absent. No change to the placement-group wire format.
* feat: honour ClientRequestToken idempotency for EKS CreateCluster
Mirrors the EC2 RunInstances ClientToken pattern with a TTL JetStream
KV bucket scoped per EKSServiceImpl instance. Same token and identical
params replays the in-progress/created cluster; same token with
different params returns IdempotentParameterMismatch; a distinct token
reusing the same cluster name still hits the existing atomic
name-claim and returns ResourceInUse.
* style: trim EKS spread comments to the 3-line max and drop a bead reference
* fix: namespace remaining viper keys shared across service subcommands
predastore, nats and awsgw registered "port", "host", "debug", "tls-cert",
"tls-key", "encryption-key-file" and "node-id" as bare, unqualified viper
keys. viper.BindPFlag overwrites the same key's registered pflag object on
each call, so only the last subcommand registered in init() kept its CLI
flag live; the others read a foreign flag's (usually unset) value.
Bare keys are also vulnerable to AutomaticEnv: viper checks the
SPINIFEX_<KEY> derived name before an explicit BindEnv target, so a
same-named env var set for one subcommand also feeds viper.Get calls in
every other subcommand sharing that key, or a generic SPINIFEX_DEBUG-style
var could shadow all of them at once.
Namespace each affected key per subcommand ("predastore-port",
"nats-debug", "awsgw-host", etc.) following the pattern already used for
predastore-host/base-path/config-path and spinifex-ui-*, so each key's
AutomaticEnv-derived name matches its own BindEnv target. CLI flag names
and environment variable names are unchanged. nats's own bind-host flag
uses "nats-service-host" rather than "nats-host", since that key is already
bound by rootCmd's own cluster-wide --nats-host override.
* fix: bind viperblock flags through the command that declares them
s3-host, s3-bucket, s3-region and plugin-path are declared on viperblockCmd
but were looked up on predastoreCmd, which declares none of them. The lookup
returned nil, BindPFlag rejected it silently, and the discarded error hid it.
Both the flags and their declared defaults were invisible to viper, so
s3-bucket resolved "" rather than "predastore".
* fix: bump goleak past the elided-frames stack-parse panic
goleak v1.3.0's stack parser treats a "...N frames elided..." line as
a function name and panics when it can't parse one. Go elides frames
on deep/recursive stacks, which is rare but load-dependent, so this
surfaced as an intermittent panic inside goleak itself (not a real
leak) during TestDrainVolume_DispatchedGoroutineDoesNotLeak under a
full parallel preflight run. Upstream fixed this in goleak@b62053b,
unreleased as a tagged version, so this pins to the latest commit.
* test: guard goleak against panicking on elided-frame stacks
* style: trim goleak regression test comments to the 3-line max
* feat: add segscan storage oracle for the real-predastore integration tier
Adds spinifex/testutil/segscanoracle, which copies a fixture node data dir
and execs the mulga umbrella repo's scripts/segscan (a separate Go module
that cannot be imported) to assert live/dead-tombstoned/dead-orphan byte
accounting against real predastore writes. Kept off the default PR path: a
new segscanoracle build tag excludes the test from make test-integration,
and a separate segscan-oracle.yml workflow (push to main/dev + manual
dispatch only, mirroring e2e.yml's non-gating trigger) checks out
scripts/segscan rather than adding that checkout cost to spinifex.yml.
* test: cover segscanoracle's pure logic to satisfy diff coverage
segscanoracle has no build tag and always compiles into the default
build, but nothing in that build exercised it, so every new line
counted as uncovered against the 60% diff-coverage gate. Adds unit
tests for JSON decoding (including the malformed-payload path), the
Locate resolution order, and the directory-copy routine, plus a
throwaway stand-in module (not the real scripts/segscan) to exercise
build+exec+decode end to end.
Pulls decodeReport out of Run and resolveSegscanDir out of Locate so
the pure logic is testable without a *testing.T or a real binary.
* feat: accept and surface gp3 Throughput on CreateVolume/DescribeVolumes
Honors input.Throughput (125-1000 MiB/s, default 125 when omitted) on
CreateVolume, persists it to VolumeMetadata, and surfaces it via
DescribeVolumes (getVolumeByID), mirroring the Iops handling shipped
in mulga-siv-380. Pins viperblock to the commit that adds the
VolumeMetadata.Throughput field.
Zero Throughput on a read volume means it predates this field; it is
omitted from the response rather than surfaced as a misleading 0
MiB/s, matching the existing IOPS convention.
* chore: pin viperblock to the merged gp3 Throughput commit
* fix: stop viper AutomaticEnv from shadowing predastore's own env vars
Viper resolves AutomaticEnv before explicit BindEnv, so predastore's
bare "config-path", "host" and "base-path" viper keys auto-derived to
SPINIFEX_CONFIG_PATH, SPINIFEX_HOST and SPINIFEX_BASE_PATH -
coincidentally the same names used elsewhere (the global cluster
config path, and root's global --host override). Setting
SPINIFEX_CONFIG_PATH for the cluster config (needed by
derivePredastoreBind) silently made predastore parse the cluster
config as its own S3 config, breaking auth for every signed request.
Rename predastore's host/base-path/config-path viper keys to
"predastore-"-prefixed keys whose AutomaticEnv-derived name is
identical to their explicit BindEnv target, so no shadow is possible
regardless of what else is set in the environment.
* style: trim predastore env-binding comments to the 3-line limit
* ci: skip building unused E2E suite binaries in e2e.yml's single/multi jobs
These two jobs immediately rebuild suite-aware binaries after Build
Install Artifacts and never consume the tar's _bin contents, so the
cross-compiled suite build was pure waste. build-install-artifacts.sh
gains an optional --skip-e2e-build flag (positional contract unchanged);
e2e-nightly.yml and the other callers keep building the full set.
* refactor: replace hand-rolled IMDS client with AWS SDK v2
internal/imdscreds, cmd/ecs-agent's metadata fetch and its credentials
provider used a hand-written IMDSv2 token/GET flow with manual RFC3339
expiry parsing. A malformed Expiration silently fell back to "never
expires" instead of failing.
Replace it with feature/ec2/imds, credentials/ec2rolecreds and
aws.CredentialsCache, following the internal/gwsign/signer.go
precedent. An unparseable Expiration is now a hard error from Fetch,
covered by TestFetch_BadExpirationErrors. Output shapes to callers are
unchanged, and IMDS base URLs ending in /latest keep working.
* refactor: derive predastore bind config from spinifex.toml in spx itself
predastore-start.sh awk-parsed spinifex.toml for the bind host/port and
node_id, exported them as env vars, then exec'd spx. Move that
derivation into predastoreStartCmd via derivePredastoreBind, which
reads the raw (non-DIAL-normalized) nodes.<node>.predastore section,
defaulting node_id to -1 (co-located) when the key is absent, same as
the script. An explicit --host/--port/--node-id flag or
SPINIFEX_PREDASTORE_* env var still overrides the derived value.
Delete the now-unused script and point the systemd unit's ExecStart
directly at spx service predastore start, matching the
spinifex-viperblock.service precedent.
* feat: close IMDS and vm_adapters trace-linking gaps
Wrap the IMDS mux with otelsetup.HTTPMiddleware (outermost, ahead of
rejectForwarded) so guest metadata requests parent their DescribeInstances/
Attribute NATS fan-out instead of rooting a trace per call, including
requests the SSRF guard rejects.
Route daemon/vm_adapters.go's ebs.mount/ebs.unmount NATS requests through a
new ebsRequestWithTrace helper that opens a client span and injects it into
the message headers, so viperblockd's consumer spans join the producer trace
instead of rooting their own.
* feat: trace the ebs.delete request from the instance cleaner
* fix: preserve running guests across target restart, harden state write
spinifex-shutdown.service's ExecStop fired on any spinifex.target stop,
including the stop half of `systemctl restart spinifex.target`, draining
every guest via QMP even though the host never went down. Gate the drain
behind `systemctl is-system-running` == "stopping", which only holds once
systemd is genuinely unwinding into shutdown.target (reboot/poweroff), so
a target restart or plain `systemctl stop` now leaves QEMU/nbdkit running
(KillMode=process) for the daemon to reconnect to on the way back up.
Daemon.WriteState returned as soon as the local state file write failed,
skipping the independent JetStream KV write entirely. A crash in that
window left a live VM with no durable record anywhere. Attempt the KV
write regardless of local disk health, then still surface the local
write error to callers, mirroring finalizeTerminated's handling of the
same fault class on the terminate path.
* fix: gate the shutdown drain via an spx flag, not a shell script
Replace the shutdown-drain-gate.sh gate with a --only-if-host-stopping
flag on `spx admin node drain`. The unit now points at /usr/local/bin/spx
directly (already required for the node to function) instead of a script
installed under the data directory, which a partial rebuild of that
directory could silently drop without ever failing ExecStop. The gate
logic also becomes a plain Go function with injectable state lookup, so
it gets real table-driven coverage instead of only a live reboot.
hostIsStopping() drains only on a literal "stopping" from `systemctl
is-system-running`. Every other recognized state (running/degraded/...)
and any unrecognized output skip the drain; a fully unreachable systemctl
fails toward draining, since a skipped drain on a real shutdown hard-kills
guests and loses unflushed writes, while a spurious drain on a restart
only costs an extra graceful stop/relaunch cycle. Run by hand without the
flag, the command still drains unconditionally as it always has.
* refactor: split the shutdown-gate comments to sit with the code they explain
* fix(vm): durably record terminate even when local state persistence fails
finalizeTerminated bailed out of WriteTerminatedInstance whenever the
transition to terminated hit a persistence-only failure (e.g. ENOSPC writing
local state), even though the in-memory status had already reached
terminated. That left the instance owned by nothing: StuckTerminateReaper
skips it (no longer shutting-down) and TerminatedTeardownReaper never sees
it (never reached the terminated KV bucket). Only an operator daemon restart
recovered it.
WriteTerminatedInstance is a JetStream KV write, independent of local disk
space, so keep going into it whenever the in-memory status did reach
terminated, distinguishing that case from a genuine invalid/raced
transition. Once the disk has space again, teardown now completes on its
own via the existing idempotent TerminatedTeardownReaper.
* fix(systemd): cap per-service memory under spinifex-system.slice
Every service under spinifex.slice ran with MemoryMax=infinity /
MemoryHigh=infinity, so one runaway service could exhaust the whole slice
budget and the slice-level OOM would then reap an arbitrary victim instead
of the offender.
Add per-unit MemoryHigh/MemoryMax as percentages of physical RAM (so the
same units scale from a dev box to bare metal). spinifex-daemon.service
still hosts every guest QEMU process in its own cgroup (no per-instance
slice yet), so it gets by far the largest allowance, just under the slice's
own 80%/90%, to avoid starving VM launches. predastore is the legitimately
memory-hungry service and gets generous headroom above its idle footprint.
The remaining infra services are small, single-purpose Go processes and get
a shared conservative bound.
* fix(viperblockd): remove local volume directory on ebs.delete
DeleteVolume deleted the S3 data and notified viperblockd, but the ebs.delete
handler only stopped the volume's goroutines, killed nbdkit, and removed the
NBD socket — it never removed the local WAL/checkpoint cache directory under
BaseDir. The only path that removed it (VB.Close's RemoveLocalFiles) only
ran from the unmount handler's seal step, which -efi volumes never reach
(they carry no durable data to seal) and which a main volume also skips
whenever a prior seal failed (e.g. disk full, deliberately keeping local
files for recovery). Since terminate always unmounts before deleting, and
delete is permanent, retention had no remaining purpose by the time
ebs.delete ran.
Unconditionally remove BaseDir/<volume> in the ebs.delete handler regardless
of mount-tracking state, closing the leak for both the root volume and its
-efi companion.
* fix(viperblockd): validate volume names before removing local state
ebs.delete unmarshalled the volume name straight off the wire and
handed it to os.RemoveAll: an empty name collapsed to BaseDir itself,
and ".." components could escape it entirely. localVolumeDir now
rejects an empty/invalid name or one that resolves outside BaseDir
before any removal, and ebs.delete returns Success: false with an
error instead of silently skipping. Applied the same gate to
ebs.mount and ebs.config, the other two handlers that turn a
wire-supplied volume name into a BaseDir-relative path.
* chore: pin viperblock to the merged concurrent-open corruption fix
The ansible build role builds with GOWORK=off, so the deployed binary links
whatever go.mod pins rather than the go.work overlay. Move the pin to the
revision carrying the per-volume flock and the WAL/state write fixes.
* fix(vpc): close the route-table and ENI teardown leaks
DeleteVpc validated subnets and non-default security groups but never
non-main route tables, so it deleted the VPC out from under them and left
records naming a VPC that no longer resolves. It now returns
DependencyViolation, alongside the existing checks and before any cascade,
and still cascades the main route table on its own.
DeleteSubnet left route table associations naming the deleted subnet.
DeleteRouteTable blocks on any non-main association carrying a subnet ID,
and that subnet could never be deleted again to clear it, so the table was
pinned forever. Associations are now cleared before the subnet record goes,
leaving a mid-way failure retryable. routetable imports vpc, so the record
cannot be typed here; only the associations key is rewritten, and every
other field is passed through untouched.
DeleteLoadBalancer detached an ENI and then re-read it to delete, where a
lagging R3 replica could report it still in use. The failure was logged at
WARN and the LB record deleted anyway, orphaning the ENI and its security
group. DetachAndDeleteENI now reads once and carries the detach revision
into the delete via LastRevision, retrying the whole flow from a fresh read
on a conflict, bounded at 3 attempts. The IPAM release happens after the
delete, not before, so a retry cannot hand the same IP back twice. When
ENIs survive, the LB record is kept naming them and the call returns
DependencyViolation rather than dropping the only reference to the leak.
Instance termination routes through the same call, and both ENI paths now
distinguish a real delete from an already-absent one instead of logging a
deletion that did not happen.
TestRLC5_NoOrphanedRouteTableAfterSubnetThenVpcTeardown covers the ordering
that defeated the per-operation tests: subnet first, then VPC.
* fix(elbv2): route SetSubnets ENI teardown through the atomic detach+delete
SetSubnets detached every ENI in one loop, then deleted the removed subnets'
ENIs with force=false in another. That is the same shape as the
DeleteLoadBalancer leak: the delete re-reads the ENI, a lagging replica can
still show it attached, and the delete fails as in-use while the LB moves on
without it.
Removed subnets now go through DetachAndDeleteENI, which carries the detach
revision into the delete under one read. ENIs that survive the reshape are
still detached on their own, since they are re-attached to the relaunched VM.
isProductionLayout() keyed off /etc/spinifex alone. A node reset legitimately
removes that directory, so spx silently fell back to the dev layout and rebuilt
the whole node — configs, keys, nats.conf — under the invoking user's home while
the systemd units kept reading /etc/spinifex. Cluster formation reported success
and every service then failed to start.
Detection now also accepts the installed systemd target, which setup.sh writes
and a reset never touches, so the layout survives the reset that caused this.
init and join no longer mkdir the production config dir either. Only setup.sh
can rebuild the per-service subtree and its ownership, so recreating the top
level would still leave a node that inits cleanly and dies at service start.
They now refuse with the remedy instead.
/var/log/spinifex is 0775 root:spinifex so each service user can write its own
log. logrotate refuses to rotate inside a group-writable directory unless the
config says which user to rotate as, so every entry was skipped and the logs
grew without bound.
Verified on env13: before, logrotate -d reported 'skipping
/var/log/spinifex/nats.log because parent directory has insecure permissions'
and nothing had ever rotated. After, the debug run is clean and forced
rotations produce nats.log.1 and a compressed nats.log.2.gz.
* fix(gateway): log the two SigV4 paths that reject silently
Three code paths answer SignatureDoesNotMatch, but only the Verify mismatch
logged anything. A skewed request timestamp and an unsupported service in the
credential scope both returned the same wire error with no log line at all, so
they were indistinguishable from a canonicalisation defect after the fact.
Both now log at Warn. The skew path records the client and server timestamps
plus the window, which is what separates clock drift from a signing bug.
Adds signingTime, since Parse rejects a skewed request before returning a
SignedRequest, leaving the raw header or presigned query arg as the only copy
of the timestamp available to log.
* fix(gateway): log the malformed-envelope rejection too
The parse-error default branch answers IncompleteSignature and was the last
auth rejection path with no log line, so a malformed Authorization header,
unsupported algorithm or missing content hash left no record of which
validation stage rejected the request. The parse error names it.
shutdownVolumes deliberately spares nbdkit that a guest is still writing
through, because tearing one out mid-write corrupts that guest's filesystem.
The unit left KillMode at the default, so the cgroup SIGTERMed those children
anyway and SIGKILLed them at TimeoutStopSec -- causing both a 30s stop stall
and the corruption the skip exists to prevent.
KillMode=process matches spinifex-daemon, where guest QEMU already survives a
restart for DDIL reattach. Idle nbdkit is still reaped explicitly.
Measured on env12 with a guest attached: control-group stopped in 30.08s with
'Failed with result timeout' and SIGKILLed nbdkit, leaving QEMU running with
zero ESTAB nbd sockets. KillMode=process stops sub-second, both nbdkit
survive, and the guest keeps its device.
* fix(ec2): drain the attached volume before CreateImage reads a live checkpoint
CreateImage against a running instance (--no-reboot) read the volume's
checkpoint directly from predastore without draining in-flight writes
first, unlike CreateSnapshot. A checkpoint captured before a guest-
triggered write (e.g. growpart rewriting the GPT after a root volume
is launched larger than the base image) could be stale, producing an
AMI that boots to a UEFI shell instead of the OS.
Extracts the existing snapshot-package drain logic into a shared,
reusable DrainVolume so both live-snapshot call sites route through
the same drain-before-read guarantee instead of being able to diverge.
* test(e2e): boot-verify CreateImage AMIs instead of trusting State=available
describe-images reporting State=available is not evidence an AMI
boots — a well-formed but unbootable AMI (stuck at the UEFI shell)
still registers cleanly. Extend the CreateImage e2e test to launch a
probe instance from each produced AMI (running-path and stopped-path)
and require a real SSH login, with console-dump-on-failure wired up
for diagnosis.
* fix(images): sparsify the Alpine guestfish .raw before conversion
The Alpine build path grows the qcow2 in place via qemu-img resize +
guestfish resize2fs, which can leave metadata scattered across the
newly-grown region. virt-sparsify zeroes the guest's unused blocks
before the raw conversion so they collapse into real holes, matching
the Ubuntu virt-resize path's sparseness.
DescribeInstances silently returned an empty Reservations list for an
instance ID that never existed, so callers could not distinguish "gone"
from "never existed". Mirror the two-pass found-set pattern from
DescribeSecurityGroups, but only assert absence when the NATS fan-out and
both stopped/terminated KV bucket queries are provably complete — a
node timing out during the fan-out must never produce a false NotFound
for an instance that actually exists on that node.
The assertion is split into a new DescribeInstancesChecked, used only by
the gateway's customer-facing action. The plain DescribeInstances stays
silent for internal callers (IMDS instance lookup, the RDS instance-state
resolver, the EKS control-plane reconciler) that treat "not found" as
"not currently present" rather than a fault.
The watchdog probed only healthz?js-enabled-only=true, which reports the
server-wide JetStream state. A per-message-block write error is recorded on
the block and returned on every later write, and nothing clears it short of
a restart -- the server-wide state is never touched, so healthz stays 200
while writes fail indefinitely after the disk recovers.
spx admin node js-probe round-trips a canary key through the cluster-state
bucket, which is the only operation that observes the latch. It probes that
bucket rather than one of its own because a dedicated bucket is a separate
stream that can be placed elsewhere and stay writable.
* fix: stop passing nbdkit S3 credentials as literal argv
access_key/secret_key were interpolated into the nbdkit plugin argv,
which is world-readable via /proc/<pid>/cmdline for the life of the
process. Credentials now travel via cmd.Env (VB_ACCESS_KEY/
VB_SECRET_KEY) instead. Requires a viperblock nbdkit plugin build that
reads these environment variables; inert against an older plugin
until that pin moves.
* chore: pin viperblock to the VB_ACCESS_KEY/VB_SECRET_KEY fallback
The spinifex half sets these env vars on the nbdkit child, but the
deploy builds with GOWORK=off, so viperblock's matching read of them
only ships once go.mod points at 8a4787a.
* fix(lbagent): stop heartbeating a load balancer once it is confirmed deleted
A LoadBalancerNotFound response is the gateway's positive confirmation that
the store has no record for the LB, not a connectivity failure — unlike a
dial error, timeout, or 5xx, which leave its existence unknown and must keep
retrying. Past the startup registration grace window (or after any prior
successful heartbeat), it can no longer be the record's own replication
race, so the agent now stops rather than retrying forever and filling the
log with heartbeat noise after teardown.
* fix(nats): enable the JetStream ENOSPC-latch watchdog timer
nats-server disables JetStream process-wide on ENOSPC (server/jetstream.go
handleOutOfSpace -> ShutdownJetStream) and never re-enables it itself —
that decision is nats-server's own internal state, not spinifex's, so a
restart is the only way back. The spinifex-nats-watchdog.timer + service
already probe /healthz?js-enabled-only=true and try-restart on a sustained
failure, but the timer's WantedBy=timers.target never took effect: neither
setup.sh nor firstboot enabled it, so the watchdog shipped inert and a full
disk needed a manual restart forever. Enable --now it alongside the rest of
the unit set.
* fix: bound EKS DELETING re-drive with backoff and a terminal give-up
reapCluster re-drove purgeClusterInfra on every GC tick with no attempt
tracking, so a permanently-failing teardown (e.g. an unretriable
DependencyViolation) was retried forever. Track attempt count and last
attempt time on ClusterMeta, back off exponentially between re-drives,
and give up after maxDeleteReapAttempts with a single error-level log.
Status stays DELETING on give-up so the cluster's infra remains tracked
and billable pending operator intervention, preserving the ADR-0006 §6
billable-before-sweep invariant.
* fix: create the EKS e2e node role independently in each subtest
runEBSCSIVolume string-built the worker node role ARN without ever
creating it, only passing because runIRSAPod's t.Cleanup discarded
DeleteRole's error while the role was still attached to its instance
profile, leaving it behind for EBSCSIVolume to reuse by naming
convention. Extract ensureNodeRole and call it independently from both
subtests, detaching the role from its instance profile before deleting
it and asserting that error instead of discarding it.
* fix: reclaim CSI-provisioned EBS volumes on EKS cluster delete
purgeClusterInfra tore down the control plane, nodegroups, and
networking but never looked at EBS volumes, so any volume the
in-cluster CSI driver had provisioned for a PVC was left behind
untracked and unbilled for. Add a report-only reclaim step that scans
for volumes tagged with the cluster's CSI owner tag and logs them for
operator follow-up.
Volumes are never auto-deleted: a PV's reclaim policy (Delete vs
Retain) is a Kubernetes-side property the CSI driver does not tag onto
the EC2 volume, so there is no safe way to tell from tags alone
whether deleting a given volume is intended. This mirrors the
volume-leak reaper's existing mark-and-alarm policy for the same
reason.
The cluster-owner tag requires the EBS CSI controller's
--k8s-tag-cluster-id flag, added here to the controller manifest;
without it a CSI volume carries no cluster-scoped tag and reclaim
would have to guess across every cluster sharing the account.
* fix: persist the last purge error on a wedged DELETING cluster
DeleteReapExhausted told an operator a cluster's teardown had given
up, but not why: nothing recorded the failing purge's error.
Persist it on every failed re-drive, not only once the reaper
exhausts its attempts, so it is visible on a still-retrying cluster
too. Status stays DELETING throughout, preserving the billing
invariant that untorn-down infra remains tracked.
* fix: make the EKS e2e suite confirm its CSI volume is actually deleted
The EBSCSIVolume subtest deleted its PVC on cleanup but never
confirmed the underlying volume was reclaimed, so a CSI driver that
silently failed to delete it would leak a volume with nobody
noticing. The suite provisions and owns this volume, so unlike
production teardown it can safely wait on its StorageClass's Delete
reclaim policy: capture the CSI volumeHandle while the PVC is bound,
then have cleanup poll EC2 for that exact volume to disappear before
the addon/nodegroup that run the CSI controller are torn down.
* test: gate scripts/images/ shell suites + shellcheck in CI
The eks-node/rds-postgres *_test.sh suites and shellcheck only ever ran
by hand: spinifex.yml filters pushes on Go-only paths and no job
invoked them, so a broken helper could land unnoticed. Add a
`make test-images` target and a path-filtered image-scripts.yml
workflow so both run automatically on scripts/images/** and images/**
changes, kept separate from preflight so image-asset churn doesn't run
on every Go contributor's commit. Also clears the shellcheck findings
that would have started this gate red (SC1090 on a runtime-resolved
source path, SC2154 x2 on eval-assigned variables shellcheck can't trace).
* test: pin vCPU quota gate ordering in RequestSpotInstances
Nothing asserted that RequestSpotInstances runs the vCPU quota gate
before dispatching capacity or launching a VM, so a regression that
dropped or reordered EnforceLaunch would pass silently. Add an
ordering test using a real quota.Service (embedded JetStream KV) and
real NATS responders standing in for the capacity and launch daemons,
recording a global call sequence and asserting the gate's index
precedes both. Verified by removing the gate: the first quota-KV read
then comes from the post-launch charge instead, landing after capacity
dispatch and launch, and the ordering assertion fails as expected.
* fix(systemd): give every unit an explicit TimeoutStopSec
9 of 12 spinifex systemd units had no TimeoutStopSec and inherited
systemd's 90s default, which is either far more or far less time than
their own SIGTERM handling needs. Size each one to what its shutdown
path actually does: predastore matches viperblock's 30s WAL-drain
budget since viperblock flushes through it (After= already sequences
viperblock's stop first; this is the backstop if that's ever bypassed),
spinifex-ui covers its 30s internal shutdown context plus margin for
long-lived streaming proxy connections, nats covers its own
JetStream-shutdown/raft-stepdown/client-drain signal handler, and
services with no SIGTERM handler at all (awsgw) get a short bound
since they exit immediately regardless.
Extends the existing unit-file invariant test with a check that every
.service file declares TimeoutStopSec, so a newly added unit that
forgets one fails CI instead of silently falling back to 90s.
* fix(vm): reconcile QEMU processes with no instance record on restore
classifyRestoredInstances only walks Manager.Snapshot(), so a
qemu-system process with no instance record at all -- a crash
mid-launch, a lost KV write, a stale process surviving a wiped state
file -- was never handed to killOrphanedQEMU and stayed permanently
invisible to recovery.
Add a host-process scan that diffs live qemu-system PIDs (matched via
/proc/<pid>/comm) against the PIDs claimed by known instances' own
-pidfile files. It only reports what it finds, never kills: this
daemon cannot distinguish "record not loaded yet" or "owned by another
daemon/tenant on this host" from a genuine orphan, and killing a
running customer VM to fix an invisibility gap is the wrong trade.
Runs at the end of Restore, once every known instance has been
classified and relaunched, so a live process still unaccounted for at
that point genuinely has no record anywhere.
* fix(vm): distinguish recordless QEMU orphans by pidfile ownership
A recordless qemu-system process with a stale spinifex pidfile is
positively this daemon's own orphan; one with no pidfile at all could
belong to another tenant on a shared host. Log the two cases distinctly
instead of one blanket warning, so an operator can tell them apart.
Also sandbox XDG_RUNTIME_DIR in the shared daemon test harness so
Restore()-driven tests never read this box's real runtime directory.
* fix(spinifexui): treat ErrServerClosed as a clean shutdown, not a failure
Serve returns http.ErrServerClosed after Shutdown is called deliberately
(the SIGTERM handler above does exactly that), but Start propagated it
as an error, making systemd log a graceful stop as exit code 1 and
Result: exit-code -- indistinguishable from a real crash.
Query-protocol services (EC2, IAM, STS...) get their action from the
SigV4 Action= form param before dispatch, so traceActionEnricher can
rename the span and set aws.action/aws.service from that context.
REST-JSON services (EKS, ECS, ACM, tagging, ECR, bedrock...) only
resolve their action inside their own dispatcher, well after that
middleware runs, so the enrichment never fired for them.
checkPolicyResource is the one call every dispatcher already routes
its resolved (service, action) pair through, so move the enrichment
there via a shared recordResolvedAction helper instead of duplicating
it per REST-JSON handler.
BytesFreed counted logical object sizes deleted from the S3 namespace,
but predastore only reclaims that disk space later via background
compaction. The CLI printed "freed X GiB" immediately on delete, which
made a compaction outage that never reclaimed the space invisible.
Rename to BytesDeleted and reword the CLI output to say the bytes are
marked for deletion and reclaimed by background compaction, not freed
immediately.
* fix(ec2): rename ambiguous 'code' slog key to firmwarePath
The EFI variable store prep log used 'code' for the OVMF firmware
path, which reads as an error/status code in log search. Rename to
firmwarePath to avoid confusion.
* fix(spinifexui): write pid file to a service-owned BaseDir
spinifexui.Start() called the legacy utils.WritePidFile, which has no
MkdirAll and falls back to $HOME/spinifex. Under the spinifex-ui unit's
ProtectHome=yes this directory is neither guaranteed to exist nor
writable. Add Config.BaseDir, switch to utils.WritePidFileTo (which
does MkdirAll) for Start/Stop/Status, wire it through the CLI as a
spinifex-ui-base-dir flag/SPINIFEX_UI_BASE_DIR env var, provision
/var/lib/spinifex/spinifex-ui in setup.sh, and add it to the unit's
ReadWritePaths.
* test(viperblockd): replace fixed sleep with a readiness poll
TestEBSConfigQueueGroup_DetachedOpenFails slept 500ms before assuming
launchService's ebs.config subscriber was bound, then raced a hardcoded
5s request deadline. Poll with a cheap malformed request (rejected at
json.Unmarshal, before the slow S3 path) until the subscriber answers,
then issue the real request with room for openVolumeVB's S3 backoff.
* fix(e2e): capture GPU worker console log on test failure
Wire harness.OnFailure into TestEKSGPUPodExposure to dump the GPU
worker's serial console via the established EC2 GetConsoleOutput path,
matching lb_test.go's captureLBConsoleOnFailure. A driver or
device-plugin hang after boot only ever surfaces on the guest console.
* fix(ec2): render empty Describe result containers and add requestId
xmlutil.BuildXML omits a nil slice's XML container entirely but renders
an empty one for a non-nil empty slice, so every EC2 Describe with zero
results (DescribeKeyPairs, DescribeVolumes, DescribeInstances, etc.)
came back as a bare "<FooResponse></FooResponse>" instead of AWS's
"<FooResponse><requestId/><fooSet/></FooResponse>". botocore parses that
body as having no result key at all, so the CLI silently prints nothing
and exits 0 — this broke the ansible key-import guard that gates
cluster bootstrap on describe-key-pairs' output.
Fix normalizes nil result slices to empty ones and synthesizes a
requestId in the single marshal chokepoint shared by every EC2 action
(marshalEC2Response, used by both ec2Handler and ec2HandlerWithReq),
rather than patching each handler individually.
* fix(ec2): return InvalidKeyPair.NotFound / InvalidSnapshot.NotFound for missing explicit IDs
DescribeKeyPairs and DescribeSnapshots silently returned an empty list
when the caller named a specific, nonexistent KeyName/KeyPairId or
SnapshotId, instead of erroring like real AWS (and like the existing
DescribeSecurityGroups/DescribeImages/DescribeVolumes/DescribeSubnets
handlers already do). botocore treats the empty-list response as
success, so ansible's cluster-post key-import gate — which checks
`describe-key-pairs --key-names spinifex-key` for a nonzero exit code —
never fires, and spinifex-key is never imported during bootstrap.
Mirrors the two-pass found-set pattern already used by
DescribeSecurityGroups: build the set of names/IDs actually present in
the filtered results, then error if any explicitly requested name/ID
is absent. An unfiltered list or a --filters query that matches
nothing is unaffected and still returns an empty list with no error.
DescribeInstances has the same gap but needs a design decision on how
a NotFound check interacts with the fan-out's existing partial-result
leniency, so it is left for separate follow-up.
TanStack router-plugin <1.168.21 sorted routes by coercing route nodes
to strings, so output order followed filesystem-walk order and flipped
depending on which generator version a contributor's node_modules
resolved. Pin the version so it cannot drift, commit the sorted output
from the fixed generator, and fail CI if a build produces a different
route tree than the one committed.
On a zero-worker or GPU-only cluster no node tolerates the DaemonSet,
so it has zero desired pods. The apiserver then has no tunnel to
kube-system pods (including metrics-server), v1beta1.metrics.k8s.io
never goes Available, and namespace GC hangs on the kubernetes
finalizer for every namespace, not just ones using metrics.
Add the same CriticalAddonsOnly + nvidia.com/gpu tolerations already
used by every other control-plane-critical addon in this repo
(aws-load-balancer-controller, ebs-csi-driver, argocd) so the agent
also schedules onto the control plane.
A tool whose child dies by signal can still exit 0 through go tool, so
govulncheck printed 'ok' after being OOM-killed and preflight passed
without a vulnerability scan having run. Route lint, govulncheck and
nilaway through scripts/run-gate.sh, which fails on a non-zero exit or
on signal-death output.
RegisterTaskDefinition let any caller with ecs:RegisterTaskDefinition name
an arbitrary taskRoleArn/executionRoleArn, and RunTask/StartTask/CreateService/
UpdateService could launch tasks under a previously registered definition's
roles, all without holding iam:PassRole on those roles. Mirror the EC2
gateway's PassRoleChecker closure pattern to close both paths.
ClaimLBName's crash-orphan reclaim treated any owner with no persisted
LB record as abandoned, but a legitimate in-flight create has no record
yet either during the gap between claiming the name and PutLoadBalancer.
Two concurrent creates for the same name could both win.
Track a claim timestamp and only allow reclaim once it exceeds a TTL,
so a fresh in-flight claim blocks a second claimer like a live LB would,
while a claim abandoned by a genuine crash is still reclaimed.
gateway.ErrorHandler unconditionally overwrote every error with
ErrorLookup's static per-code text, discarding both a caller's own
formatted message and any per-service distinction for wire codes
shared across services (e.g. ACM and EKS both use
ResourceInUseException). ResolveErrorDetail now also returns a
message when the producing call site used awserrors.Errorf, and
LookupErrorMessage resolves service-scoped wording before falling
back to the global default. DeleteVpc's DependencyViolation call
sites now name the blocking subnet or security group.
* fix: scope golangci-lint cache per worktree
golangci-lint's default cache under ~/.cache/golangci-lint is keyed by
module path rather than worktree path, so parallel worktrees of this
module collide and surface each other's lint issues, including from
worktrees that have since been deleted.
* fix: correct eks-node-gpu catalog description to match agent-only image
The GPU node image never installs eks-token-webhook or a server/agent
role selector, per scripts/images/eks-node-gpu/manifest.conf; it ships
ecr-credential-provider only and is agent-only because GPU nodes never
run the EKS control plane.
instanceCleanerAdapter.DeleteVolumes only called DetachVolumeOnTerminate for
a still-attached Boot volume, assuming shutdownAndUnmount's Unmount already
cleared a non-Boot volume's attachment. That assumption breaks whenever the
unmount seal fails (Unmount swallows the error) or never runs at all, as on
the stuck-terminate force-complete path. A leaked volume then stays in-use
with no recovery path once the instance record is deleted: detach-volume
--force fails InvalidInstanceID.NotFound and delete-volume fails
VolumeInUse. Call the same explicit, idempotent detach for every
non-DeleteOnTermination volume regardless of Boot.
DescribeImages hardcoded every AMI as "available" regardless of import
progress, so a failed backend import that never reached the flush was
indistinguishable from a complete one. Project the AMI's own State,
treating empty as available so pre-existing AMIs registered before the
field existed stay visible. CreateImageFromInstance, RegisterImage, and
CopyImage now set State explicitly since each completes synchronously
before the config write.
* fix(vm): report ENOSPC to the guest on data volumes instead of pausing the VM
QEMU's default werror=enospc pauses the whole VM when a data volume's
backend runs out of space. The boot drive already avoids this via
werror=report,rerror=report; data volumes had no on-error policy at
all in either the cold-boot -device or the hotplug device_add.
Set werror=report,rerror=report on the virtio-blk-pci device itself,
since a data volume's blockdev-add NBD node has no BlockBackend to
carry it the way -drive does for the boot drive.
* fix(vm): stamp ShuttingDownAt even when the state transition write fails
transitionWithPrecheck returned before stampShuttingDownAt whenever
TransitionState failed, even though the in-memory status had already
moved to the target. On a full disk this leaves ShuttingDownAt zero,
and StuckTerminateReaper.Sweep skips any VM whose ShuttingDownAt is
zero — so the backstop meant to bound a wedged terminate is disabled
by exactly the condition it exists to catch.
Stamp on the persistence-failure branch too, using the same helper
and locking as the success path, while still returning the original
error to the caller. The raced branch (TransitionState failed and the
in-memory status never reached target) is left unstamped, since the
target state was never actually entered there.
Start registers startCluster as a member of shutdownWg and then waits on
that group. startCluster's tail waited on the same group, so it could not
return until the counter it was waiting on reached zero, and the counter
could not reach zero until it returned.
The SIGTERM handler completed and logged "Shutdown complete" in ~20ms, but
the process never exited: it sat until systemd's TimeoutStopSec=15 and was
SIGKILLed, on every stop. Because the member stop chain is serialized,
that 15s also gated viperblock, predastore and nats, which each stop in
milliseconds once reached.
The duplicate setupShutdown in the same tail registered a second
signal.Notify and a second handler goroutine, which is where the
duplicated shutdown log lines and NATS unsubscribe pass came from.
Drop both. Start already installs the handler and owns the single wait.
The invariant test pins each helper to one call site, since a second call
site is the defect rather than a symptom of it.
Five manager tests waited on a fake DHCP counter and then read the KV
store. Fake.Acquire increments that counter before Manager.doAcquire
persists the entry, so the assertion ran in the gap between the two and
read the pre-acquire address whenever the write lost the race.
Idle machines always won the race, which is why this only ever failed
under load: the last three Build and Test failures on dev were this test
twice, on a diff that never touched the package.
Make the persisted KV state the barrier, matching the shape already used
by TestManagerStartKeepsNewLeaseWhenStaleReleaseFails, and keep the
counter as a plain assertion once the write is observed.
* fix(ec2): release post-launch ENIs on instance terminate
Terminate released only the ENI recorded at RunInstances time. Attachments
made after boot — ECS awsvpc task ENIs, or any direct AttachNetworkInterface
— are written to the spinifex-vpc-enis KV and never reach instance.ENIId, so
both terminate paths walked straight past them. The survivors held their
security group, blocking the SG, subnet and VPC deletes behind
DependencyViolation.
Both paths now sweep the KV for records naming the instance, independent of
the scalar. The sweep is additive, so an instance with no secondary ENIs
behaves exactly as before, and best-effort per ENI so one failure neither
aborts the rest nor blocks convergence on re-run.
DeleteOnTermination was advertised by DescribeNetworkInterfaces but backed by
nothing; ENIRecord now carries it, defaulting to true, and false detaches
without deleting.
* test(ec2): cover ENI release branches
Add table-driven and NATS-backed tests for the terminate-time ENI
sweep: DetachENI/ForceDeleteInstanceENI failure tolerance,
DeleteOnTermination skip, multi-ENI happy path, and
ListInstanceENIs error mapping on both the daemon and instance
service adapters.
viperblock#64 stops WriteAtCtx returning ErrNoSpace on predastore's nearfull
pressure signal, which rides on a successful write and means slow down rather
than stop. Without this, spx refused every write once the store fell below 15%
free, 10 points above the watermark at which predastore actually rejects.
Needed by spx itself: image import runs in-process against the module-cache
build, so the plugin-side fix alone left the control plane on the old
behaviour.
Six merged viperblock changes behind one repin: local files kept when the
WAL cannot be uploaded, stranded WAL generations retried after a failed
chunk upload, the flush-covers-acked-writes invariant pinned, the nbdkit
plugin's shutdown drain bounded at 20s, seal receipts written after a clean
seal, and FLUSH fsyncing the WAL before it reports success.
The seal receipts pair with the consumer already on dev, and the 20s drain
bound pairs with TimeoutStopSec=30 on spinifex-viperblock.service.
The unmount path decided whether to seal by stat-ing BaseDir/<volume>, but
the nbdkit plugin deletes exactly that directory as the last step of a
successful seal. So the fallback seal was unreachable whenever the plugin
exited cleanly, and a WARN claiming there was no local state fired on every
healthy unmount.
Branch on the receipt the plugin now leaves instead. Local state present
still means the plugin's seal failed or was cut short, so the fallback runs
and is finally reachable; a receipt means the plugin sealed and cleaned up;
neither keeps the original WARN, now meaning what it says.
The aux-volume check moves out of volumeNeedsSeal and to the front of the
chain: split out but left second, an -efi volume with local state would
newly reach a seal it has always been excluded from.
Stale receipts are cleared at mount rather than matched by PID at unmount,
so nothing depends on nbdkit's pid surviving to plugin load.
In-use nbdkit children are SIGTERMed with this cgroup and run the plugin's
shutdown drain. That drain is now bounded at 20s in the plugin itself, so
30s here lets the plugin's own bound fire first and keeps SIGKILL
exceptional. At the 90s default a slow or unreachable backend stalled the
whole target-stop transaction.
Picks up viperblock #57, which routes the JSON slog handler to stderr so
the nbdkit plugin's logs reach journald — nbdkit repoints a plugin's fd 1
at /dev/null, so the whole volume seal path was discarded. Also picks up
#56, removing a per-call allocation from the per-block cache instrument.
No spinifex code depends on either change: spx does not import
viperblock/telemetry. The bump is needed because the build role compares
the viperblock source behind spx (module cache, GOWORK=off) against the
one behind the nbdkit plugin (the submodule tree) and fails on a mismatch.
lb-agent configured no logger at all, so it fell back to slog's default
text handler. That folds attributes into the message, leaving the cause of
a failure trapped in a string: 'Heartbeat failed err="gateway returned
400"' arrived greppable but never queryable.
No OTLP export: the agent's only egress is the gateway it is watching, so
the host-side serial console remains the path that survives a blackout.
Init bolts the OTLP bridge onto whatever slog default is installed at the
time. SetupRoutes then called SetDefaultJSONLogger purely to set a level,
and that bare slog.SetDefault replaced the fanout with a stdout-only
handler, so the gateway went dark to the sink for the rest of the process.
One nightly cell shipped 9 app-log docs, all within 30ms of startup, while
its journal held 574 request lines.
Backs the stdout handler with a LevelVar and adds SetLevel, so verbosity
changes no longer touch the handler chain.
* feat(gateway): log the canonical request and action on signature mismatch
A SignatureDoesNotMatch told us nothing beyond err="signature mismatch":
the canonical request the server reconstructed was never logged, and the
APM trace is a bare "POST /" because auth fails before the dispatcher
parses the action. An intermittent mismatch was therefore undiagnosable.
Logs the server-side canonical request, with the session token masked, plus
the recovered action. Warn rather than Debug because the failures worth
catching never reproduce on demand; the path is already rate-limited.
* chore(deps): bump predastore for SignedRequest.CanonicalRequest
Picks up the exported canonical request the mismatch logging needs.
Render summed FailCount across suites and printed "No failures across any
suite" whenever the total was zero, without asking whether any test had run.
A cell that dies during provisioning emits junit with no testcases at all, so
it took the clean path. Nightly run 30467638631 cell 20 failed with "install
callback never fired after 3 attempts; PXE never triggered install" and its
analysis.md read "No failures across any suite" — absence of evidence rendered
as positive evidence.
Zero testcases now renders a warning naming the layer above the suites. A run
with tests and no failures is unchanged.
Cells 2, 3, 9 and 10 ran on spinifex-image-builder-ubuntu-24.04.qcow2, built
2026-04-20 — five weeks before openvswitch-ipsec and strongswan-charon were
added to the image builder. Those cells therefore had no OVN IPsec support and
ran intra-AZ Geneve in plaintext. Every other workflow is already on 26.04.
The host_image_override guards now match ubuntu-* rather than a pinned version.
An exact match is what let this drift go unnoticed: bumping the matrix release
silently stops overriding the image and drops the cell onto the debian gold.
Cells 2, 3, 9 and 10 ran on spinifex-image-builder-ubuntu-24.04.qcow2, built
2026-04-20 — five weeks before openvswitch-ipsec and strongswan-charon were
added to the image builder. Those cells therefore had no OVN IPsec support and
ran intra-AZ Geneve in plaintext. Every other workflow is already on 26.04.
The host_image_override guards now match ubuntu-* rather than a pinned version.
An exact match is what let this drift go unnoticed: bumping the matrix release
silently stops overriding the image and drops the cell onto the debian gold.
* fix(e2e): fail TestMultinodeIPSec when a cluster that requested IPsec has none
An empty ipsec_encapsulation was treated as "bootstrapped without IPsec" and
skipped. On ubuntu-24.04 openvswitch-ipsec.service is inactive at daemon start,
so the enable guard correctly refuses to flip the flag and the cluster runs
plaintext intra-AZ Geneve — but the test then skipped and the cell reported
green, masking the gap on 2 of 3 multinode cells.
Cross-check network.ipsec_enabled from the node config: an explicit false still
skips, anything else fails with the plaintext condition named.
* fix(e2e): read only the ipsec_enabled value, not the whole config line
Detail() rendered "ipsec_enabled=ipsec_enabled = true" because the raw grep
line was logged as the value. Take the first token after "=" so a trailing
comment cannot be read as the value either, and compare exactly rather than
by substring.
Every daemon read-modify-wrote the same zone object in s3://northstar with no
mutual exclusion. A NATS queue group balances load, it does not serialise
distinct messages, so concurrent daemons lost each other's records and could
leave the object unparseable as TOML. Once unparseable it stayed that way: both
the writer and the drift reconciler must parse the zone before they can repair
it, so DNS changes stopped applying until the object was edited by hand.
The exposure did not need user activity. The drift backstop ran on every node,
unelected, on a 60s ticker, so an idle three-node cluster raced its own zone
once per tick. Five publisher families feed the same object, so ordinary
concurrent API calls collided too.
- Add a per-zone JetStream KV lock, following the CAS patterns already used by
the vpcd reconcile leader and the vCPU quota store. One key per zone, so
unrelated zones never block. The wait budget stays under requestTimeout so a
producer's ack is not lost to contention, and a write aborts rather than land
outside an expired lease.
- Rebuild a zone whose stored bytes do not parse instead of failing every write
forever. The reconciler reports a corrupt zone as absent, which suppresses
pruning while still publishing the full desired set. Detection matches the
decoder's error type so it works against the pinned northstar release.
- Elect one node per drift cycle, removing the idle-cluster self-race.
- Correct three comments claiming the queue group serialised zone writes; they
are why the hazard went unnoticed.
- Run the LB e2e internet-facing subtests in parallel. Their serialisation
workaround exempted internal LBs on the false premise that those use a
different DNS path; ELBChanges sets the same base-domain zone for both.
* fix: group-scope the OVS/OVN sockets and stop sudoing read-only probes
The OVS DB socket and both sets of ctl sockets were chmod 0666 by setup-ovn.sh,
with a systemd override re-applying it on every openvswitch-switch restart. Write
access to db.sock is full control of the local datapath, so this handed the
datapath to any account on the box. Both service users run with Group=spinifex,
so the access is now 0660 root:spinifex, /var/run/ovn is 0750, and a shared
helper re-applies it from ExecStartPost on openvswitch-switch and ovn-controller
alike — the latter is new, and matters because the ovn-controller ctl socket name
embeds the pid and is recreated on every start.
That also fixes the health probe reporting ovn-controller as not_running on a
healthy node. It shelled out to `sudo ovn-appctl`, which only spinifex-vpcd is
granted; as spinifex-daemon sudo demanded a password and the swallowed error read
as OVN being down. With the ctl socket group-owned the probe needs no privilege,
so health.go moves to a non-escalating Runner rather than the alternative of
granting the daemon a root-equivalent ovn-appctl rule to read a status.
ListIMDSTaps ran `list-ports` plus one `get` per patch port on vpcd's 15s tick,
so 1+N sudo invocations every tick, each one logging a pam_limits warning about
the core rlimit the unit deliberately pins to 0. It now issues a single JSON
query per pass, and the sudoers drop-in disables the PAM session for both service
users, which also leaves sudo children inheriting the unit's rlimits.
An absent iface-id on a patch port is now skipped with a warning instead of
aborting the pass: it is the only record of the full ENI, so such a port cannot
join the live set either way, and failing the pass stalled reconcile for every
healthy port behind it. A transient query failure still aborts.
* fix: group-own the OVS/OVN pidfiles too, and cover the ipsec ctl socket
Group-owning the ctl sockets was not enough for `ovn-appctl -t ovn-controller`:
it resolves the pid-named socket through the pidfile, and OVS opens that
O_RDWR to test the fcntl liveness lock, so a group-readable pidfile still fails
the probe with EACCES. The pidfiles now get the same 0660 root:spinifex, which
also tightens them from the 0644 they ship with.
openvswitch-ipsec joins the override list because setup-ovn.sh restarts it
after the sweep runs, leaving its ctl socket as the one file still at the
shipped mode.
Verified on the dev node: every socket and pidfile is 0660 root:spinifex and
survives a restart of all three units, an unprivileged user is refused, and
/health reports ovn-controller ok where the sudo probe read not_running.
* refactor: stop escalating the OVS/OVN socket clients
Now that setup-ovn.sh group-owns the control sockets, ovs-vsctl, ovs-appctl,
ovn-nbctl, ovn-sbctl, ovn-appctl and systemctl reach their daemons as the
service user. utils.NeedsPrivilege carries that policy and both command
builders consult it, so ip, iptables, dhcpcd, sysctl and arping still escalate
while the socket clients no longer do. Their sudoers grants are removed: each
took unrestricted args and all of them accept --log-file=PATH, so every one was
a root-equivalent grant handed out to run a query.
ovs-ofctl keeps its grant. It talks to a per-bridge <bridge>.mgmt socket that
ovs-vswitchd creates when the bridge appears, including bridges spinifex
creates at runtime, so the provisioning sweep cannot group-own it up front.
The NB and SB database sockets join the sweep for ovn-nbctl and ovn-sbctl.
vpcd carried its own sudoCommand copy that escalated unconditionally, bypassing
the policy entirely. It survived only because the grants existed and broke the
OVN flows-ready barrier the moment they were removed, so it now delegates to
utils.SudoCommand. TestRG10_SudoOnlyThroughThePolicy fails the build if another
package reintroduces a local copy.
Verified on the dev node: an instance launches, IMDS is served, the flows-ready
barrier completes, and no sudo denial appears anywhere in the journal.
* refactor: drop every spinifex-vpcd sudoers grant for ambient capabilities
vpcd held NOPASSWD grants for ip, iptables, sysctl and arping. The ip grant
alone was root-equivalent: sudoers cannot constrain its arguments, so
`sudo ip netns exec <ns> /bin/sh` was a root shell for the service account.
The unit already grants CAP_NET_ADMIN and CAP_NET_RAW ambiently, and the kernel
passes an ambient set to each child on exec, so those tools work as the service
user with no escalation at all. Verified on a live node: all sixteen ip
subcommands vpcd issues, plus iptables reads and writes, sysctl -w on the
net.ipv4.neigh tree, and arping, all succeed under exactly the unit's ambient
set. spinifex-vpcd now has no sudoers rules.
NeedsPrivilege had to become capability-aware to do this. spx is one binary
shared by both service units, and the daemon holds CAP_SYS_ADMIN and
CAP_DAC_OVERRIDE but no CAP_NET_ADMIN, so a static tool list cannot express
"vpcd runs ip directly, the daemon escalates it". The decision now reads CapAmb
from /proc/self/status once and matches it against the capability each
invocation needs. sysctl is resolved per key, since CAP_NET_ADMIN covers the
net.* trees and nothing else.
The dhcpcd grants go too: neither daemon invokes it. Only setup-ovn.sh, which
runs as the provisioning user, and the in-guest ecs-agent.
TestRG10_VPCDHoldsNoSudoersGrant pins both halves together, so a future grant
cannot be re-added without also explaining why the capability stopped working.
* chore: restore routeTree.gen.ts swept in by mistake
* fix: probe ovn-controller with ovn-appctl, not ovs-appctl
checkOVNController's first probe passed a bare -t ovn-controller to ovs-appctl,
which resolves it against OVS_RUNDIR — /var/run/openvswitch. ovn-controller's
socket and pidfile are in /var/run/ovn, so the probe could never succeed and
logged "ovn-controller.pid: open: No such file or directory" on every check.
Not a functional break: the next probe globs the full /var/run/ovn path and
succeeds, so the function still returned nil. It cost an exec per check and
made a healthy node look like it was failing an OVN liveness probe.
ovn-appctl defaults OVS_RUNDIR to /var/run/ovn, so the common case now matches
first. The glob and systemctl fallbacks stay for older layouts.
* fix(dhcp): release stranded leases and label them with the owning node
Two paths dropped a lease from KV without returning it to the upstream
server, so a source="dhcp" pool leaked addresses permanently.
Manager.Start deleted an expired entry outright. Dropping the entry alone
abandons the binding rather than returning it, and a server that ages
leases by liveness holds the address for as long as anything answers ARP
for it. Adoption now sends a bounded, best-effort RELEASE first, skipping
leases with no stored offer/ack because nclient4 cannot rebuild those.
Release-by-IP answered SUCCESS when the IP was absent from KV, telling
the caller the address was freed when nothing went on the wire. It now
returns an error. Every ReleaseIP call site logs and continues, so this
surfaces the strand without wedging teardown.
Leases were also unattributable upstream: chaddr is a derived HashMAC and
option 61 carries that same MAC, so no field identified the host that took
the lease. Option 12 is now "<node>-<client-id>" and option 60 is
"mulga-spinifex/<node>", giving the lease table a per-lease identity and a
per-host column to group on. HashMAC is untouched — encoding the node
there would cut per-node entropy and reverse a documented decision.
Observed live: 20 office-LAN addresses held by one node's orphaned OVN
gateway LRPs, with no lease records and no renewals.
* fix: rebind the datapath when a DHCP lease moves to a new address
A lease re-issued on a different IP was persisted to KV with no notification
of any kind. doAcquire replaced the lease after an expiry and doRenew accepted
whatever a rebind answered with, while the gateway LRP's address is written
once at attach and never revisited. Each such event burned one address
permanently: the old one squatted by OVN's ARP responder with no renewer and
no DHCPRELEASE, the new one renewed forever attached to nothing.
Route both paths through applyLease, which releases the superseded address and
invokes an IPChangeHook so the datapath can follow. vpcd registers a hook that
moves the gateway LRP's networks and gateway_ip stamp, re-runs the routed leg
so the VPC SNAT and host ingress route follow, and waits on the flows barrier.
EIP, ENI-public and NATGW addresses are recorded outside vpcd in
DescribeAddresses and DescribeInstances, so the hook reports those rather than
repointing a datapath the control plane no longer agrees with. The release half
still applies to every purpose, so the leak itself is closed either way.
A failed rebind does not fail the lease loop: the new address is held
regardless, and dropping it would strand both.
* fix: reconcile API-visible records when their DHCP lease moves
An EIP or auto-assigned public IP whose lease was re-issued elsewhere left
DescribeAddresses and DescribeNetworkInterfaces advertising an address vpcd had
already released. The datapath kept the old dnat_and_snat too, so the address
was dead on both sides while the upstream server was free to hand it to
someone else.
vpcd cannot fix this alone: it holds the lease, but the records naming those
addresses live daemon-side. Add vpc.dhcp.lease-changed, which vpcd requests and
the daemon serves, dispatching to the service owning the moved address.
RebindPublicIP and RebindENIPublicIP commit the new external IP through
vpc.add-nat first — AddEIP scrubs the predecessor row keyed on the logical IP,
so one commit is enough — and only then move the record, so a NAT failure never
leaves a record advertising an address that does not route.
Both refuse to move a record naming neither address, so a stale request cannot
overwrite an allocation the lease does not own, and both are idempotent under
duplicate delivery.
A failed or unanswered reconcile propagates back to the lease loop rather than
reading as success. vpc.dhcp.lease-expired, listed in the service manifest but
never implemented, is replaced by the subject that now exists.
* fix: dedupe retried AllocateAddress so one call takes one address
EC2 AllocateAddress carries no ClientToken, and vpcd's DORA ladder can
outlast the AWS client read timeout, so every SDK-level retry minted a
fresh allocation ID and drew another upstream DHCP lease. The caller kept
one EIP and the rest stayed leased with no record naming them, so nothing
ever released them.
Retries are now identified by the SDK's amz-sdk-invocation-id, which is
stable across attempts of one call. The gateway lifts it off the request,
NATSRequest forwards it as a header, and the daemon puts it back on the
context, so the EIP service sees the same token every attempt. singleflight
joins a retry to the call still in flight — the common case — and a TTL'd
KV record answers retries that arrive after the first call completed.
* fix: reclaim DHCP leases whose owner is gone and stop dropping expired ones
Three leaks that all end the same way: an address stays leased upstream with
nothing renewing it on purpose and nothing that will ever release it.
Manager.Start deleted any lease that expired while vpcd was down, after
releasing it. The resource it was taken for still existed, so the datapath kept
forwarding on an address the upstream server now considers free — a
duplicate-IP conflict waiting for the server to reissue it — and nothing ever
re-acquired. Expired leases now get a renewal loop like any other, which
re-DORAs on its first pass; the existing IP-change hook rebinds the datapath
when the new address differs. The reaffirm RENEW is skipped for these, since a
lease the server has already aged out will only fail it.
Nothing reconciled leases against the resources that justify them, so a
resource deleted out from under its lease left the address renewed forever.
ReapOrphans sweeps them: vpcd asks the daemon over vpc.dhcp.owner-check
whether the EIP, ENI or VPC still exists, and releases the lease when it does
not. A gateway lease also has a router port to remove, so DetachIGW runs
before the address goes back — releasing it while OVN still answers ARP for it
invites the same conflict. Anything the daemon cannot resolve is reported
unknown and kept: an unreachable daemon must never read as a fleet-wide
deletion.
The acquire ladder ran to ~62s, just past the 60s read timeout botocore
defaults to, which is what turned one AllocateAddress into two leases. It now
totals 40s with a 45s budget, and the DORA RPC 50s, so the caller hears the
outcome instead of giving up and retrying. This is a mitigation, not a fix, for
callers that send no retry token: botocore emits no amz-sdk-invocation-id, so
the dedupe added earlier cannot see their retries at all.
Fixes mulga-j19gj, mulga-9exe6, mulga-7qyzj.
* feat: add spx get dhcp-leases so held external addresses are visible
The upstream leases behind a source="dhcp" external pool lived in a per-AZ KV
bucket with no read path, so working out which addresses were held, by what,
and whether the owner still existed meant writing a NATS client or reading
journals. That was the main obstacle to diagnosing every leak in this series.
The command lists each lease with its purpose, VPC, address, MAC, holding node
and remaining time, and resolves the owner over vpc.dhcp.owner-check so an
orphan is visible as such. --orphans narrows the output to leaks;
--skip-owner-check drops the lookup when only the addresses matter.
Fixes mulga-655iu.
* fix(vpcd): reconcile gateway ports onto their lease address at startup
A gateway LRP configured from a static pool keeps its address when the pool is
switched to source="dhcp", so the VPC forwards on an address nothing leases
while the lease it holds goes unused. No lease event ever fires to correct it.
vpcd now sweeps every gw-lrp lease on startup and rebinds the port to the
address the lease actually holds. RebindGatewayIP is idempotent, so a port
already in agreement costs one OVSDB read.
* test: cover dhcp lease owner-check handlers and the lease CLI
The owner-check path decides whether the reaper releases an address, so every
inconclusive branch is asserted to report unknown rather than gone.
* fix(dhcp): make the outcome of a stale lease release observable
A stale RELEASE was logged only when the transport failed. A lease with no raw
offer/ack was skipped in silence, and a RELEASE the server ignored looked
identical to one it honoured, so an address left bound upstream could not be
traced to either cause.
The skip now warns, and a successful release logs the chaddr it used, which is
what a server matches a RELEASE against.
* fix(dhcp): take the RELEASE chaddr from the lease being released
A server matches a RELEASE to a binding on chaddr, and the release path
re-resolved it live. For a UseIfaceMAC lease whose interface MAC had changed
since, that sends an address the binding never had: the server drops the RELEASE
and says nothing, so the address stays held with no renewer.
The chaddr now comes from the lease record, which carries what took the lease
out, falling back to live resolution only when the record has none.
* fix(eks): GPU node AMI declares uefi boot mode
The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but
the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under
legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at
firmware with zero serial console output.
* fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs
Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
initialize under the closed/proprietary kernel module: dmesg shows
RmInitAdapter failing with "requires use of the NVIDIA open kernel
modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to
nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia
recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no
separate open variant (userspace tools only), so it is unchanged.
* fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons
A cluster with a GPU-only nodegroup has no untainted node: the CP
carries CriticalAddonsOnly (EKS parity keeps user workloads off it)
and every worker carries nvidia.com/gpu. Without tolerations for both,
ebs-csi-controller and aws-load-balancer-controller have nowhere to
schedule and stay Pending forever. Real EKS never needs this since its
control plane isn't a schedulable node, so the stock upstream
manifests carry no such toleration.
* fix(images): give the libguestfs appliance a working resolver
The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
(127.0.0.53), which isn't running inside the appliance chroot, so DNS
is dead until package installs need it. Point it at a real resolver
before package installs, then restore the stub symlink afterward so
the shipped AMI still gets its DNS from systemd-resolved at boot like
a stock cloud image.
* fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind
A GPU's IOMMU group can include an upstream PCIe bridge when the
platform lacks ACS isolation between them. vfio-pci refuses to bind
non-endpoint bridge devices, so Claim/ReclaimByAddress must filter
bridge-class (0x06) members out of the group before touching driver
bindings, rather than failing the whole claim or leaving the bridge
rebound incorrectly.
* fix(ecs): GPU node AMI declares uefi boot mode
The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the
Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS
no OVMF/pflash firmware is attached, so the guest hangs at firmware with
zero serial console output.
Same defect and same fix as the eks-node-gpu entry: both images are built
by the same script from the same stock Ubuntu cloudimg, so both were
EFI-only and only one was corrected.
* fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch
ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary
kernel module. NVIDIA ships no proprietary module for Blackwell (GB202)
and later, so ECS GPU container-instances could not bind those cards on
any driver branch. It is the third verbatim copy of the routine and was
missed when the other two were fixed.
Also pins the driver branch across all three copies. The branch was
resolved at build time via apt-cache search | sort -rn | head -1, which
sorts on the number in the package name — but that number is a branch
alias, not a driver version (nvidia-dkms-535-server installs 580.126.20),
so the ordering key need not match what installs. Resolving also let
identical source produce different drivers on different days.
The pin lives in setup.sh rather than manifest.conf because manifest
variables are not passed into the guest: the setup script is uploaded and
executed by virt-customize --run, with no environment carried across.
* fix(eks): tolerate CP and GPU taints in argocd addon
All six argocd Deployments and its StatefulSet carried no tolerations. A
cluster with a GPU-only nodegroup has no untainted node to place them on:
the control plane carries CriticalAddonsOnly and every worker carries
nvidia.com/gpu, so the pods stay Pending forever.
Same defect the ebs-csi-controller and load-balancer-controller addons
already had. Real EKS never hits it because its control plane is not a
schedulable node, so the stock upstream manifests carry no such
toleration and the divergence is ours to maintain.
* refactor(images): extract eks-node initd logic into init-agnostic helpers
The OpenRC runscripts carried their start_pre logic inline, so porting a
service to systemd meant re-deriving that logic under a distro change
rather than just rewriting the init wiring. Move it into /bin/sh helpers
that either init system can drive, keeping eks-node on Alpine/OpenRC.
k3s-prestart takes the restore-block guard, config.yaml derivation,
konnectivity-agent and CoreDNS staging, and the token-webhook wait.
konnectivity-server-prestart takes the CA wait and cert mint, plus the
server-count derivation that used to run at parse time. Both keep every
guard, bound and message as-is.
Values flow back the way the GPU image already does it: the helper writes
shell assignments to a /run file and the init system sources them, so
command_args is built in start_pre where it used to be a top-level
assignment.
k3s-agent-prestart moves to eks-node and becomes the single copy. The GPU
manifest claimed its helpers were reused verbatim from eks-node while
hand-copying this one; it now points at the shared file, so the two
cannot drift.
k3s-agent.initd still sources agent.env and exports itself rather than
delegating: OpenRC has no EnvironmentFile= equivalent, and the daemon
inherits only what the runscript exports, so a helper subprocess sourcing
it would not reach k3s.
The helpers run under set -eu, which the runscripts did not. The awk over
first-boot.env is guarded on the konnectivity manifest, not on the env
file, so a missing env file exits awk nonzero; without || true that turned
a warn-and-continue into a refusal to start k3s. Covered by a test that
fails on rc=2 if the guard is dropped.
* feat(images): build system images in a pinned mkosi container
Move the image-build toolchain into a container so a runner needs only
Docker. The toolchain cannot drift between runners, and the same build
runs identically on a developer's machine.
mkosi builds in a user namespace, which two of Docker's default
confinements block: the seccomp profile masks CLONE_NEWUSER, and the
docker-default AppArmor profile separately blocks unsharing the mount
namespace. Both are relaxed; neither is a privilege grant, as
capabilities stay dropped and no devices are exposed. mkosi also runs as
an ordinary user rather than root, since as root it takes a path that
needs real CAP_SYS_ADMIN.
Verified end to end: an Ubuntu 26.04 Format=disk image builds from the
Debian container and the artefact is a valid GPT disk, with the package
cache on a named volume cutting a forced rebuild from ~2min to ~34s.
* feat(images): publish system images from the build container
Convert mkosi's raw output to a compressed qcow2, write the .sha256
sidecar the spx catalog verifier matches, and upload both to R2 using the
same pinned toolchain as the build, so publishing needs nothing on the
host either. The toolchain image build moves to a shared helper rather
than being duplicated across the two entry points.
Publishing needs no relaxed confinement, unlike the build: qemu-img and
aws create no namespaces, so it keeps Docker's default seccomp and
AppArmor profiles.
The object path carries no version and the catalog resolves newest-wins,
so an upload overwrites the live image with no way back. Copy the current
object to a timestamped archive key first, making a bad publish a restore
rather than an unrecoverable loss.
* fix(images): build the image on the output's mount, not the cache volume
mkosi assembles the image in its workspace and moves the result to the
output directory. The workspace lived under the cache volume, which is a
different mount from the output bind mount even when both sit on one
filesystem, so the move failed EXDEV and mkosi fell back to copying —
requiring the finished image to exist twice at once.
That is invisible on a small image and fatal on a real one: a ~4G GPU
image needed ~8G to land and died with ENOSPC after a successful build.
Point the workspace at the output's own mount so the move is a rename
again and the image only exists once. The package cache stays on the
volume, which is the part worth persisting across runs.
* feat(images): add mkosi base config and gpu-nvidia profile
Establish images/ with a base mkosi.conf plus a gpu-nvidia profile,
replacing the throwaway smoke-test config. The base builds a bootable
UEFI disk; the profile adds the NVIDIA stack.
Most of ubuntu-gpu-nvidia-setup.sh does not carry over. Its kernel-version
hunt, the do_bootloader=no kernel-img.conf hack and the manual
update-initramfs all exist only because that pipeline chroots into a
prebuilt cloud image that already has a kernel. mkosi installs the kernel
and owns the bootloader, so the version is known rather than detected, and
the postinst asserts exactly one kernel instead of searching for it. The
static configuration (nouveau blacklist, unattended-upgrades blacklist,
cloud-init docker group) becomes files under mkosi.extra/ rather than
heredocs.
linux-image-generic drags in every linux-firmware split package, none of
it relevant to a headless VM guest; left at the default the modules initrd
embeds all of it into the UKI, which grew to ~880MB and overflowed the
512M ESP. The initrd module set is trimmed to mkosi-initrd's curated
default, which is a boot fix and not only a size one.
Verified in the built artefact, not just the build log: nvidia.ko.zst and
the four sibling modules under /usr/lib/modules/<kver>/updates/dkms, the
nouveau blacklist, and the nvidia runtime in docker's daemon.json.
* fix(images): unblock guest boot on wait-online and firstboot
A guest built from these images stalled twice on the way to a login prompt,
both times on a unit waiting for input that never arrives in a VM.
systemd-networkd-wait-online holds network-online.target until every managed
link is up, and docker.service pulls that target in with Wants=, so any image
carrying docker inherits the wait whether or not it wants it. A guest whose
links are slow or absent ran the unit to its full 120s default with
multi-user.target queued behind it. Override it to --any with a short timeout:
one link up is what a single-NIC guest actually needs.
systemd-firstboot separately prompts on the console for anything the image
does not already carry, before multi-user.target. With no timezone baked in,
boot stopped dead at "Please enter the new timezone name" and waited for a
keystroke. Setting locale, keymap and timezone at build time leaves firstboot
with nothing to ask.
Both are baked into the base rather than delivered as first-boot drop-ins:
an image built from scratch has no cloud-init to write them.
Measured on wattle with an A1000 passed through: login at 42.8s, down from
~210s, and the firstboot prompt is gone. nvidia-smi in the guest reports
"NVIDIA RTX A1000, 595.71.05, 8188 MiB".
Also widen the images gitignore to output-*/ so a scratch build cannot leave
multi-GB artefacts stageable.
* feat(images): make the base image reachable — cloud-init, sshd, console
An image built from scratch launched unreachable. The images this replaces
start from a stock Ubuntu cloud image and inherit cloud-init, sshd, sudo, a
network renderer, dbus and a serial console from it; mkosi installs only what
is listed, so none of it was there and nothing said so.
Guests get their SSH key and user-data from IMDS via cloud-init, so without
the package a launched instance has no authorized_keys at all — no error, just
no way in. It also made any cloud-init-delivered config inert, including the
gpu profile's docker-group drop-in.
Each addition is load-bearing on its own:
cloud-init consumes IMDS: keys, user-data, hostname, network
openssh-server the key needs a sshd to be installed into
netplan.io cloud-init's renderer on Ubuntu; with no renderer the NIC is
never configured, IMDS is never reached, and cloud-init
falls back to DataSourceNone
dbus systemd only Recommends it and mkosi installs no Recommends,
so systemctl could not reach the system bus — and the EKS and
ECS bootstraps drive guests with systemctl from user-data
lsb-release cloud-init shells out to it; without it every boot reports
"degraded" forever and the status stops being a usable signal
The kernel also needs console=ttyS0 baked in. The launcher gives every guest a
serial port and get-console-output reads it, but a kernel with no console=
writes to none of it: the console stopped after the UEFI bootloader and every
boot failure was invisible. ttyS0 goes last because Linux makes the final
console= the controlling /dev/console.
manage_etc_hosts is set because cloud-init renames the guest per launch but at
its default never writes a matching hosts entry, so sudo warned on every call.
datasource_list is deliberately NOT pinned: the guest identifies the platform
from the SMBIOS system UUID the launcher spoofs, ds-identify narrows to Ec2 on
its own, and pinning per image is the customisation this migration deletes.
Verified on wattle with an A1000 passed through: ssh in as ubuntu with
spinifex-key, cloud-id "aws", cloud-init status done (not degraded), no failed
units, nvidia-smi reports the A1000 on 595.71.05, docker active, boot 26.7s.
* feat(images): let the guest grow its root into the volume it was given
mkosi sizes the disk to its content, so the image ships with a root
partition that fills the disk to its last sector and ~1.4G free. Nothing
in it can expand that: cloud-init's growpart module is enabled, but the
growpart binary lives in cloud-guest-utils, which is not installed and is
not pulled in as a Recommends by anything in the package set. systemd-repart
is not an alternative — Ubuntu's systemd package carries no repart binary,
unit or definitions at all.
The result was a node handed a larger volume that it could only ever use
the first few GiB of, with the rest allocated and unreachable. That is
fatal for a worker whose job is to pull container images.
gdisk is deliberately left out despite being a cloud-init Recommends:
growpart resizes GPT through sfdisk, which arrives via the hard fdisk
dependency, and e2fsprogs brings the resize2fs that the resizefs module
runs afterwards.
Verified on wattle: the gpu-nvidia image launched with a 20G root reports
20G on df -h / (was 4.4G), via
"'/' resized: changed (/dev/vda2) from 4770582528 to 20936900096", with
nvidia-smi still reporting the A1000 and no failed units.
* feat(images): move eks-node-gpu and ecs-agent-gpu onto mkosi profiles
Adds the eks-common, eks-agent and ecs profiles, so both GPU node images
build from the composed base rather than a hand-authored tree, and the three
verbatim copies of install_nvidia_gpu_stack() collapse into gpu-nvidia alone.
Splits Docker out of gpu-nvidia into its own profile. Docker belongs to the
standalone GPU AMI, whose purpose is `docker run --gpus all`; the EKS and ECS
node images run k3s or containerd and reach the GPU through CDI. Fusing the
two was fatal for ECS rather than merely wasteful: gpu-nvidia installed
containerd.io from Docker's repository, which Conflicts/Replaces the Ubuntu
containerd the ecs profile needs, so gpu-nvidia + ecs could not resolve at
all. nvidia-container-toolkit stays in gpu-nvidia — every GPU image needs
nvidia-ctk to generate its CDI spec.
Closes three gaps the pilot left, each of which only shows up on a booted
guest: /etc/modules-load.d/nvidia.conf, without which nothing loads the
modules on a headless node and CDI generation finds no GPU;
mulga-cdi-generate.service, which no profile shipped; and an apt-mark hold on
the toolkit packages, which the unattended-upgrades blacklist does not cover
for a plain `apt upgrade`.
mkosi-build.sh gains --image, naming an output and expanding it to an ordered
profile list. The order is load-bearing: mkosi runs each profile's postinst in
the order the profiles are passed, so docker's nvidia-ctk wiring must follow
gpu-nvidia. Leaving that to callers meant an argument-order mistake produced a
healthy-looking image with no GPU wiring, so docker's postinst also fails
loudly when gpu-nvidia is selected but nvidia-ctk is absent.
Profiles reference build inputs where they already live via ExtraTrees=
instead of copying them, keeping one copy of the helpers shared with the
Alpine image. Go binaries are built on the host into a staging tree, since the
builder container carries no Go on purpose: go.mod is the single pin for the
toolchain and a second pin in the Dockerfile could disagree with it silently.
* fix(images): install systemd-resolved so the guest has a DNS resolver
mkosi installs only the packages named in Packages= plus their hard dependency
closure, and systemd-resolved is merely a Recommends of systemd. So it was never
installed: the image shipped a 0-byte /etc/resolv.conf that nothing populated,
because netplan renders to systemd-networkd and networkd alone does not write
resolv.conf. glibc then fell back to localhost and every lookup failed with
"[::1]:53 connection refused", which reads as a network fault rather than a
missing package.
A node that cannot resolve cannot pull a container image. The ECS GPU node
booted, registered as a container instance, then failed every task at image
pull. The EKS image carries the same defect but is masked by the cloud-init
bootcmd in nodegroup_userdata.go that writes resolv.conf at launch; the ECS
launch path has no equivalent, so only ECS exposed it.
* fix(images): name build output by image and never report a skipped build as success
Two defects in the build wrapper that let the wrong bytes reach a boot test.
Every composition wrote the same spinifex.raw, taken from the base config's
ImageId, so an artefact carried no record of which profiles produced it and an
EKS image was distinguishable from an ECS one only by which build ran last.
Passing --image-id names the output after the image, and as a side effect the
name lands in the guest's /usr/lib/os-release. Publishing correspondingly loses
its default input: --name is the published object and --image the local
artefact, and with nothing tying the two together a defaulted input meant
whatever raw happened to be in the output directory could be uploaded under the
name the caller asked for.
mkosi also skips the build whenever the output path merely exists. It does not
compare inputs, so it never notices an edited mkosi.conf or a restaged binary,
prints "Use --force to rebuild" and exits 0 — a stale image indistinguishable
from a fresh one. Builds are now always forced, and the run additionally asserts
the expected raw was written during this invocation rather than trusting the
exit status. The cache worth keeping is the package download volume, which
--force does not touch.
* feat: add gpu-amd mkosi profile for the ROCm AMI
Ports scripts/images/ubuntu-gpu-amd-setup.sh so ubuntu-gpu-amd builds under
mkosi as 'gpu-amd docker'. Without it the legacy scripts/images/ tree has to
stay alive for one remaining image.
Pins the ROCm repo by apt priority rather than only adding it: Ubuntu 26.04
universe ships its own vendored rocminfo, which unpinned apt preferred over
AMD's, mixing release trains within one image.
Uses a systemd unit for render/video group membership instead of a cloud-init
runcmd, because cloud-init's config merge takes a duplicated runcmd key
wholesale from one fragment and silently discards the other, which would have
dropped the docker profile's own usermod.
Runtime GPU verification is deferred — no AMD hardware is available.
* feat(images): add eks-server, ec2-node and CPU eks-node/ecs-node mkosi images
Ports the K3s server side of EKS off the legacy Alpine/OpenRC builder: nine
systemd units, a templated etcd-snapshot service with daily and 15-minute
timers, the addon/coredns/konnectivity assets, and the six server binaries.
konnectivity-server is not an in-repo package, so it gets a pinned host-side
clone-and-build rather than an entry in profile_binaries().
Splits ec2-node out of eks-common. mulga-ebs-byid mints the by-id links for any
attached volume and mulga-mgmt-net configures any multi-NIC system VM, so
neither belongs behind an EKS profile that a database image would have to
compose to reach them.
Start-at-boot moves into preset files. mkosi runs preset-all after every
postinst, so a postinst enable is discarded before the image is finished, and
with no policy file the fallthrough is enable — which built an image that ran a
control plane and a worker at once.
Installs the cloud-init datasource pin as 99- rather than 90-. cloud-init-base
ships its own 90_dpkg.cfg declaring datasource_list, read_conf_d sorts the
directory in reverse and the merge keeps the first source, so a 90- prefixed
file lost the key silently. The postinst now asserts the merged result.
* refactor(eks-node): make the role selector init-agnostic
eks-node-role and k3s-first-boot drove OpenRC directly, so on the systemd image
the selector enabled nothing and the node never started k3s. Both now dispatch
through svc_enable/svc_start/svc_disable on a detected init, with EKS_NODE_INIT
overriding detection so either branch is testable on a host that has only one.
Detection prefers rc-update over systemctl: the test suite stubs rc-update on
PATH, and preferring systemctl would have silently rerouted those cases to a
path they were not written for.
Enable and start stay two separate passes. The server path starts
mulga-eks-k3s-recovery before k3s so a recovery directive applies before the
apiserver comes up, and collapsing to systemctl enable --now would interleave
them.
The dispatch is duplicated in both scripts rather than shared: they are
single-file deliverables with no library-path convention between them.
Tests cover both branches across all three roles, including that server-join
still gets no k3s-first-boot, and that rc-update wins when both inits are
present.
* fix(eks-node): enqueue role-selected units instead of blocking on them
eks-node-role.service declares Before=k3s.service, and
mulga-eks-k3s-recovery.service declares After=eks-node-role.service, so a
blocking systemctl start deadlocks: systemd will not run the queued job while
the selector is still activating, and the selector cannot finish until that job
runs. Type=oneshot defaults TimeoutStartSec=infinity, so nothing breaks the tie
and the node wedges mid-boot with the role written but no control plane. On a
booted guest the selector sat in activating for 261s until the VM was torn down.
Start with --no-block on systemd. The ordering the blocking call was implicitly
providing is already declared between the units themselves, so systemd
sequences them correctly on its own. OpenRC runs the script directly rather
than scheduling it, so that branch keeps blocking and keeps surfacing failures
through set -e.
* fix: escape $ in konnectivity-server ExecStart so server-count survives
systemd expands $VAR/${VAR} in Exec*= from the unit's own environment before
the command runs, substituting an empty string for a name it does not know.
KONN_SERVER_COUNT is not a unit variable — konnectivity-server-prestart writes
it to /run/konnectivity-server-args.env, which the wrapper shell sources at
runtime — so systemd rewrote the flag to `--server-count=` before /bin/sh saw
it. konnectivity-server then died on flag parsing, which is why the unit logged
Started and produced nothing despite --logtostderr=true and journal+console,
and why the apiserver's cluster egress selector found no UDS for the life of
the cluster.
This is a port artefact. konnectivity-server.initd is flag-for-flag identical
and shares the same prestart script, but under OpenRC the shell was the only
thing reading ${KONN_SERVER_COUNT}, so the same text was correct.
Escape it as $$ to defer expansion to the wrapper shell, and add
lint_unit_expansions() to reject any unescaped $VAR/${VAR} in an Exec*= line of
a profile's units. The linter joins line continuations first, since the
offending reference sat mid-way through a wrapped multi-line ExecStart, and
runs on the host before the container starts so a bad unit fails the build in
seconds rather than at first boot. $(cmd) is deliberately not flagged: systemd
leaves it literal.
Verified on env12 with a rebuilt image: TestEKS/IRSAPod and
TestEKS/EBSCSIVolume, which both previously failed on `kubectl exec ... timed
out` through the egress tunnel, now pass, with zero UDS connect errors across
the run.
* fix: refresh the stale qcow2 mkosi-build leaves beside a fresh raw
mkosi only emits the raw. A qcow2 next to it came from a previous run's hand
conversion, and nothing about its name says which build it belongs to, so it
survives a rebuild and silently ships the old image to a node while the raw
beside it carries the new one. Observed on a rebuild that left the raw at 15:34
carrying a fix and the qcow2 at 12:45 carrying the pre-fix unit.
This is the same failure the existing freshness assertion guards against, one
file over, so extend that block rather than add a second check. Refresh rather
than delete: the qcow2 is what gets copied to a node and imported, so removing
it would push the manual conversion, and this trap, back onto the caller.
Convert via a temp name and rename so an interrupted run leaves the old file
rather than a half-written one that looks current.
* fix: resolve the awsvpc task DHCP client instead of hardcoding udhcpc
ecs-agent leases a task ENI's OVN-assigned address inside a per-task netns,
which systemd-networkd does not manage, so it shells out to a DHCP client. That
call was hardcoded to busybox udhcpc, which Alpine ships and Ubuntu does not.
`ip netns exec` exits 1 rather than 127 when it cannot exec the binary, so the
failure surfaced as an opaque "network setup failed" with the task stuck in
STOPPED, and it took every awsvpc-mode workload with it: TaskAwsvpc and
ServiceWithELB both failed on the mkosi image, the latter as runningCount=0
because its target group targets task ENI IPs.
Resolve the client at setup time, preferring udhcpc so Alpine behaviour is
unchanged and falling back to dhcpcd. Presence of the udhcpc binary is not
sufficient to select it: Debian and Ubuntu's busybox ship the applet without
/usr/share/udhcpc/default.script, and udhcpc then takes the lease and exits
nonzero having configured nothing. Same order and same reasoning as
dhcp_oneshot() in mulga-mgmt-net.sh. Both invocations stay one-shot, leaving no
daemon behind in the netns.
Install dhcpcd-base in base, because neither client was present in any profile.
That also repairs mulga-mgmt-net.sh, whose own udhcpc/dhcpcd fallback resolved
to nothing on Ubuntu and would have failed the first time a cross-host
control-plane data NIC needed it — the S5 path the step-6 plan notes has never
been exercised.
The DHCP-client choice is injected so the tests assert a named platform's
branch rather than whatever happens to be installed on the machine running
them; without that the same test exercises different code on an Alpine CI box
and an Ubuntu dev box.
* refactor: retire the ec2-node profile, fold its helpers into the base image
ec2-node carried mulga-ebs-byid and mulga-mgmt-net, and every EKS composition
had to remember to list it — eks-common even guarded on its presence and
aborted the build without it. ECS did not list it, so the ECS image shipped
with no by-id symlinks for hot-attached volumes at all.
Neither helper is answerable at build time: any image may later be attached a
volume, and any image may later be launched as a control plane. Both are inert
when unused (mulga-ebs-byid is udev-driven; mulga-mgmt-net exits 0 when its
fw_cfg blob is absent), so the base image is where they belong. That deletes
the composition rule, the guard, and the ECS gap in one move.
Also drops ec2-node from the two eks image compositions in mkosi-build.sh.
Verified on freshly built spinifex-ecs-node and spinifex-eks-node raws: both
carry the helpers and the udev rule, mulga-ebs-byid is enabled via
multi-user.target.wants and mulga-mgmt-net via cloud-init.target.wants.
* fix(eks): force UseMTU=false on netplan units so DHCP renewal can't reinflate the pinned link
mulga-vpc-mtu only won the boot race by ordering: netplan renders
UseMTU=true (it does not inherit networkd's own false default), so a
DHCP lease renewal at T1 could silently re-apply the advertised MTU to
the link. usemtu_off() forces UseMTU=false via a runtime drop-in in
the systemd-documented [DHCPv4] section for every rendered unit and
reloads networkd, removing the dependency on boot ordering entirely.
assert_pinned() then verifies the link and its routes actually match
before exiting, so any regression is a loud boot-time failure instead
of a silent one. Both are no-ops on the Alpine/dhcpcd image, which has
no systemd-networkd.
* fix(images): state UseMTU=false under both the current section and the legacy alias
netplan renders the DHCP settings under [DHCP] while systemd documents
[DHCPv4]. A drop-in naming only one parses cleanly and does nothing if the
running systemd disagrees, which is the silent no-op the drop-in exists to
prevent.
* fix(ecs): install iptables on both ecs-node images, cover task-role credential fetch
credendpoint.go DNATs the ECS task credential endpoint via iptables; neither
the mkosi ecs profile nor the legacy Alpine manifest installed it, so the
endpoint silently failed to start and any task with a task role could not
fetch credentials while the instance still looked healthy.
Also fixes a latent bug in the e2e task-role trust policy (Service-principal-only,
which this platform's HTTPS AssumeRole path can never match) and adds a
TaskRoleCredentials subtest that fetches real credentials from inside a task
container, so a dead endpoint fails the suite instead of passing unnoticed.
* fix(images): ship preset files for base, eks-common and ecs start-at-boot
mkosi runs systemctl preset-all after every profile's postinst, discarding
whatever systemctl enable a postinst performed. eks-server and eks-agent
already carry preset files for this reason; base, eks-common and ecs still
relied on the default-enable fallthrough, which happens to match today but
is one upstream package change from reversing.
Adds 10-mulga-base.preset (mulga-ebs-byid, mulga-mgmt-net),
30-mulga-eks-common.preset (mulga-eks-provider-id, mulga-vpc-mtu) and
40-mulga-ecs.preset (containerd, ecs-ca-install, ecs-agent; also disables
cni-dhcp.service/socket, which containernetworking-plugins enables by
default with nothing to serve under the baked host-local IPAM conflist).
Removes the now-redundant systemctl enable calls from each postinst,
matching the eks-server/eks-agent precedent.
* fix(sts): attribute ecs-tasks.amazonaws.com to container-instance callers
The HTTPS AssumeRole path passed an empty principalSource, so matchServicePrincipal
denied every Service clause other than the IMDS-only EC2 one. A role carrying the
trust policy AWS documents for an ECS task role was unassumable by construction.
Service attribution is now derived from the authenticated SigV4 caller: an
assumed-role session of the account's ecsInstanceRole — the only credential the
in-guest agent holds — is attributed ecs-tasks.amazonaws.com, and only for roles
in its own account. Nothing in the request body contributes, so a task holding
its own task-role session cannot claim the service principal.
* fix(images): remove baked SSH host keys, pin Seed= for reproducibility
openssh-server's postinst bakes an identical host key pair into every
build, making every guest from one build mutually MITM-able; sshd-keygen
already regenerates keys on first boot (ConditionFirstBoot=yes), so the
baked pair is pure non-determinism with no purpose. Also pin Seed= so
systemd-repart stops assigning fresh partition UUIDs on every build.
Verified on a built spinifex-ecs-node image: no ssh_host_* files ship,
cloud-init still lists ssh in cloud_init_modules, and sshd-keygen.service
is wired via ssh.service.wants/ (redundant regeneration path). A two-build
comparison dropped content-differing files from 9 to 1 (only the UKI,
whose zstd internals remain sensitive to unrelated input changes).
---------
Co-authored-by: tomnewton-mulga <tom.newton@mulgadc.com>
* fix: fan out certificate re-import to referencing load balancers
Re-importing new material under an existing ACM ARN previously updated
only the stored CertRecord; no load balancer referencing that ARN ever
re-rendered its config, so HAProxy kept serving the old leaf until it
expired while the API reported success.
Add an InUseBy reverse index on CertRecord, maintained via snapshot-diff
around every listener create/modify/delete and cert attach/detach in
handlers/elbv2. On re-import, ACMServiceImpl invokes a fan-out hook
(wired in daemon.go to avoid an import cycle) that re-runs
updateStoredConfig for every load balancer in the ARN's InUseBy set,
mirroring the existing target-group fan-out. lbagent is unchanged: it
already polls ConfigHash and reloads on change.
* fix: make InUseBy index updates CAS-safe under concurrent writers
AddInUseBy/RemoveInUseBy previously did a plain get/mutate/put, which
loses concurrent writers silently: two listeners on different load
balancers attaching the same certificate at once could drop one
entry with no error. Because InUseBy is the sole mechanism that fans
a renewed certificate out to the data plane, a lost entry means that
load balancer never re-renders and its certificate silently expires
in HAProxy while ACM reports it as healthy.
Switch both to a JetStream KV revision-based compare-and-swap loop
with jittered backoff, retrying on a concurrent writer instead of
overwriting it. Replace the old concurrency test, which only
asserted no errors/panics and so could not detect a lost index
entry, with one that asserts exact final state across N concurrent
writers.
* fix: encrypt ACM certificate private keys at rest
CertRecord.PrivateKey was persisted to the spinifex-acm JetStream KV
bucket as plaintext PEM. Encrypt it with AES-256-GCM via the existing
handlers_iam.EncryptSecret/DecryptSecret helpers, keyed by master.key.
Store.PutCert encrypts when a master key is set; GetCert/ListCerts
decrypt, falling back to a narrowly-bounded legacy-plaintext read (only
when the stored value is unambiguously a PEM private key block) so
pre-existing records keep working and are re-encrypted on next write.
A Store never given a key (ELBv2's independent read-only acmStore)
behaves exactly as before — encryption is opt-in per Store instance via
the new SetMasterKey method, so no other package needs to change.
NewACMServiceImplWithNATS now requires a master key and fails
construction without one; daemon.go loads master.key inside the
initServiceWithRetry closure so a missing key fails daemon startup
after the retry window instead of leaving the service permanently
degraded.
* fix: wire ELBv2's ACM store with the deployment master key
ELBv2 constructed its own handlers_acm.Store without a master key, so
once ACM started encrypting CertRecord.PrivateKey at rest, ELBv2 could
no longer resolve listener certificates: it read back ciphertext where
it expected PEM. Require NewStore(ctx, nc, masterKey) to always take a
key so an unkeyed Store can no longer be constructed, thread the
deployment master key through NewELBv2ServiceImplWithNATS and the
daemon's ELBv2 init closure (mirroring the ACM closure), and make ACM
store construction failure fatal instead of silently degrading every
HTTPS listener.
* feat(acm): tenant Private CA with x509 name constraints
Adds an independent tenant root CA (separate from the platform CA) that
signs leaf certificates only within its baked-in permitted domains.
Name constraints (PermittedDNSDomains, excluded IP ranges, excluded
email/URI) are frozen at creation and drift is rejected rather than
silently regenerated, so a device's trust store is never invalidated
without an explicit operator action.
* docs(acm): record OpenSSL verification of the empty email/URI exclusions
Confirms the zero-length ExcludedEmailAddresses/ExcludedURIDomains
entries parse cleanly and enforce as expected under OpenSSL 3.5.6, not
just Go's own verifier, since this CA is meant for device trust stores.
* feat: implement RequestCertificate with PRIVATE_CA managed issuance
Adds the ACM RequestCertificate API surface: mode derivation
(PROVIDER_API/MANUAL_TXT/PRIVATE_CA), synchronous PRIVATE_CA issuance via an
injected CertAuthority, and correctly-shaped PENDING_VALIDATION records for
the deferred modes. Extends CertRecord with issuance/validation/renewal
state, wires the gateway and daemon NATS dispatch, and updates COMMANDS.md.
* feat: wire tenant CA into the running spinifex daemon
Add LoadTenantCA as a load-only entry point (never creates), refactor
LoadOrCreateTenantCA to share the load path. Add 'spx admin cert
create-tenant-ca' to create the tenant CA with explicit permitted
domains, idempotent unless --regenerate is passed. Wire tenant CA
loading into daemon startup: missing tenant CA logs a warning and
degrades PRIVATE_CA issuance only, it does not block startup like the
master key does. Fold tenant CA paths into admin's CA path helpers,
keeping filenames distinct from the platform CA (tenant-ca.pem/.key
vs ca.pem/.key) so the two roots can never be confused.
Also give ACM's DeleteCertificate in-use refusal its own awserrors
alias instead of reusing the EKS-labelled one directly, and correct
COMMANDS.md to reflect that the tagging endpoints are implemented.
* fix: derive ACM validation mode from the requested domain, not a global flag
Mode derivation tested cfg.Northstar.DefaultDomain != "", which says only that
northstar is configured, never whether it is authoritative for the domain being
requested. Any deployment with a default domain set therefore returned
MANUAL_TXT for every request, including domains hosted nowhere, with a TXT
record the caller could not publish.
Resolve the requested domain against northstar's actual hosted zones, and put
MANUAL_TXT last. A name-constrained tenant CA is a definite local capability the
operator declared explicitly, whereas MANUAL_TXT assumes the domain is publicly
delegated so a public CA can resolve the challenge - which is never true of an
internal northstar zone. A tenant CA covering every requested domain now wins,
and a configured DNS provider still takes precedence over the manual path so
operators with a real domain are not downgraded to a private certificate.
* feat: renew PRIVATE_CA certificates automatically before they expire
Managed issuance signed a leaf once and never renewed it. With 90-day leaves
every certificate silently expired a quarter after the first apply, which is
the treadmill managed issuance exists to remove.
Renewal is scoped to PRIVATE_CA: re-signing from the tenant root is fully
offline, so this needs no ACME client and no network access. The window is
proportional - two thirds of the certificate's own lifetime, floored at a
minimum - so it self-tunes as issuance lifetimes change rather than degenerating
into 'renew immediately, forever' the way a fixed window would. A deterministic
per-ARN jitter keeps certificates issued in one apply from all renewing in the
same minute.
Reissue keeps the certificate ARN and generates a fresh leaf key, then pushes
through the ELBv2 fan-out. The push is mandatory rather than incidental:
GetLBConfig returns stored configuration and never re-resolves certificate
material, so a renewal that did not push would leave HAProxy serving the old
leaf until something unrelated changed the config hash.
A per-certificate lease keyed on the ARN means one node drives a given renewal
without serialising all renewals behind a single leader. Certificates whose
domains have fallen outside the tenant CA's name constraints record a failure
reason instead of retrying in a tight loop.
The renewal scan reads metadata only and never decrypts stored private keys;
the lease is a raw compare-and-swap that leaves the encrypted key untouched.
* fix: carry handler error messages over the daemon NATS transport
handleNATSRequest responded with the sanitized error code alone, so any
handler error without a registered code collapsed to a bare ServerInternal
and its reason was visible only in the daemon log. utils.ServeNATSRequestCtx
has always preserved the message; the two implementations had drifted.
Adds respondWithServiceError and routes the 16 service-error sites through
it, plus transport-level tests pinning that the message survives for an
uncoded error, that a wrapped coded error keeps its real code, and that a
bare code carries no message.
* feat: implement ACM GetCertificate and stop decrypting keys on read paths
GetCertificate was unimplemented end to end: it now returns the PEM
certificate body and chain for an owned ARN, ResourceNotFound for an unknown
or cross-account ARN, and RequestInProgressException for a certificate that
has a record but no body yet, wired through the service interface, the NATS
client, the daemon subscription and the gateway action map.
Store gains GetCertMetadata alongside GetCert, and ListCerts becomes
ListCertMetadata. Every read that answers a question about a certificate
rather than serving it - ownership checks, Describe, Delete, tag reads,
GetCertificate and listing - now leaves PrivateKey empty instead of running an
AES-GCM open per record to produce plaintext key material no caller reads.
* fix: reconcile the ACM master-key wiring with dev's new tests
The InUseBy and cert fan-out tests that landed on dev construct the ACM and
ELBv2 services directly, and both constructors now take the deployment master
key. Pass one shared key in the fan-out test - two services over the same
bucket disagreeing on the key is the exact failure the wiring exists to stop.
A Store is never unkeyed now, so the two test helpers collapse into one.
* fix: rebuild the ACM InUseBy index at daemon startup
The index that drives certificate fan-out is written only by listener
mutations, so a listener whose certificates have not been touched since it
was created has no entry. A certificate with an empty index then neither
fans out on renewal nor trips DeleteCertificate's in-use guard, and both
failures are silent: the renewal reports success while the load balancer
keeps serving the old leaf, and a certificate a live listener depends on
can be deleted.
Reconcile the index from the persisted listener records once at startup.
Walking every certificate rather than only the referenced ones makes it a
rebuild rather than a backfill, so it also drops entries naming a load
balancer that no longer exists. Writing only differences keeps it
idempotent, and the index writes are already CAS-guarded, so the nodes of
a cluster reconciling the same bucket converge.
* fix: deliver listener SNI certificates to HAProxy
Add/RemoveListenerCertificates persisted the listener record and then
returned, so the config was never re-rendered and lbagent had nothing to
poll for. Re-render through reloadListenerLB, the same helper the rule
mutations use.
That alone changed nothing, because the renderer dropped SNI certificates:
resolveFrontendCert returned only the default certificate and the bind line
emitted a single crt, with the PEM path keyed on the listener so a second
certificate had nowhere to live. Render every attached certificate instead,
default first so it stays the no-SNI fallback, each with its own path.
HAProxy accepts crt more than once on a bind line and selects by SNI.
The default certificate keeps its existing filename so an upgrade does not
move a deployed PEM, and the nginx renderer keeps the single-certificate
resolver it still relies on.
* fix: prune stale TLS certs from the lb-agent cert dir
A delivered PEM is cert + chain + private key. writeCertFiles only ever
wrote, so every certificate detached from a listener left its key on the
load balancer VM's disk indefinitely. Prune after a successful reload,
scoped to .pem files in the cert dir's immediate entries.
* fix: serve every attached certificate on the nginx NLB engine
nginx allows one ssl_certificate per key type in a server block, and the
NLB config is a stream block where server_name does not exist. Select by
SNI instead: give ssl_certificate a variable and map $ssl_server_name to
each certificate's path, keyed on the leaf's DNS SANs. A single-certificate
listener keeps a literal path, since a variable costs a per-handshake read.
The cert-path helpers were HAProxy-only by hard-coding lbagent.CertDir, so
they now take the dir and both engines share one multi-cert resolver.
* fix: emit ValidationMethod in DomainValidationOptions
DescribeCertificate returned each domain validation entry without a
ValidationMethod, so the AWS provider had nothing to compare against
aws_acm_certificate.validation_method. Emit DNS for the DNS-based modes.
PRIVATE_CA stays empty on purpose: it validates nothing, and real ACM omits
the method on a private certificate, where the provider reads it as NONE
whatever the request asked for.
Persist and echo requiresCompatibilities on task definitions, so a Terraform round trip no longer reads null as drift and force-replaces the definition on every apply.
Report a task whose containers all exited while the ecs-agent was down. reconcile only re-adopted running containers, so the gateway held such a task RUNNING forever, its service never replaced it, and the load balancer target stayed unhealthy.
Single-node OVN RAFT bring-up gated only on the Northbound DB socket, then
restarted ovn-controller without waiting for the Southbound RAFT to elect a
leader, serve clients, and for northd to write the gateway logical flows. With
no join step to absorb the boot window the controller attached mid-election,
programmed a partial datapath (SNAT ct-commit without output/delivery) and
never reconverged, so NAT-GW egress and EIP ingress intermittently black-holed
on the single-node baremetal CI host.
Add an SB readiness gate before ovn-controller starts (SB reachable and, when
clustered, an elected leader), and after ovn-controller restart wait for its SB
connection then force inc-engine/recompute to rebuild all OpenFlow from the
converged Southbound. Applies to 1/2/3-node RAFT alike.
The failed state existed and was unreachable: nothing drove available →
failed, so DescribeDBInstances reported available for a database that had
been dead for a week. Add the failure classifier on top of the rds-4
reconciler.
An instance is failed when its system VM is not running AND its agent
heartbeat is stale — never either alone, since a stale beat under a live VM
is an agent fault rather than a dead database. The transition records a
timestamped reason that DescribeDBInstances and DescribeEvents can explain,
and a healthy heartbeat clears it again, which is the recovery the AMI's
in-guest Restart=on-failure and EC2's VM auto-restart provide underneath.
The recovery ladder and its fencing gate stay deferred to v1.1.
The failure clock is persisted rather than held in leader memory, so a
leader change does not restart the window, and it resets only on a healthy
heartbeat so a VM that boots and immediately wedges cannot mask a
persistent fault. Every lifecycle-owned state is skipped outright, which is
what keeps the classifier from racing rds-5a/rds-5b over the same VM.
A persisted heartbeat is judged against the stale window plus the persist
floor: beats are queue-group delivered, so the leader mostly sees them only
through the record, which a healthy agent refreshes no more often than
every five beats. The steady-state path is answered from the record alone,
so a healthy fleet costs no VM describe fan-outs.
* fix(test): reclaim fixture scratch dirs stranded in /tmp
The shared predastore fixture and the integration suite's JetStream store both
outlive any individual test, so neither can use t.TempDir() - the first test to
finish would delete files the daemon still has open. Both used os.MkdirTemp
with nothing removing the directory afterwards, so every run stranded one, and
a run killed by a timeout panic or the OOM killer stranded it regardless. /tmp
is a tmpfs on the dev boxes, so the leak is charged to RAM and swap, and
swapped tmpfs pages have no backing store to be dropped to.
Add a shared sweep that reclaims a directory only when it matches both the
prefix and an age no live run can reach. The NATS store also has a clean
shutdown point, so TestMain now removes it outright and the sweep is left to
cover the killed-run case. The predastore daemon runs until the process exits
and has no such point, so the sweep is its only mechanism.
* chore: repin viperblock to the dev tip carrying the scratch-dir fix
Test-only upstream change, so the build output is unchanged - this keeps
go.mod aligned with the submodule pointer rather than delivering a fix.
* fix(dhcp): retransmit within a DORA transaction instead of restarting it
nclient4's retryFn re-sends the same packet under the same xid with a
doubling read deadline, which is RFC 2131 4.1 retransmission. WithRetry(1)
runs that loop exactly once, so the client transmitted a single DISCOVER
per attempt and never retransmitted it. socketTimeout then handed that
lone transmission the caller's whole budget, so it sat silent for 4s,
then 8s, then 16s, then 32s.
Manager does retry, on that schedule, but each attempt builds a fresh
client and therefore a fresh xid. Every attempt is a new transaction
rather than a retransmission of the previous one, so an OFFER answering
an earlier exchange is discarded as belonging to an unknown transaction.
Send six transmissions per exchange starting from a 1s read, spanning a
63s ladder — longer than the widest window Manager schedules, so an
attempt ends on the caller's context rather than on an exhausted ladder.
socketTimeout now clamps to the caller's remaining budget instead of
consuming it in one silent read. Observed live recovering an exchange
whose OFFER arrived on the third transmission, which a single-shot
client would have missed.
Both allocation sites also stop replacing the allocator's error with a
bare InsufficientAddressCapacity. ResolveErrorCode unwraps, so a
genuinely exhausted static pool still surfaces that code while a DHCP
fault reports as itself. The flat mapping is why every failure on this
path read as a capacity problem regardless of cause.
* chore(dhcptest): add --broadcast so the probe can mirror vpcd's Acquire path
* fix(dhcp): resolve the chaddr on renew and release, not just on acquire
Acquire derived a chaddr when the request carried none. Renew and Release
read lease.HWAddr straight out of the store and handed it to nclient4, so
a lease persisted before the chaddr was carried renewed with all zeros in
chaddr and option 61 fell back to its untyped form. The lease then lands
upstream with no hardware address against it — the outcome option 61 was
fixed to stop, reached by a different route — and because the server sees
an unknown client each time, every renewal takes a new lease.
resolveHWAddr now serves all three paths and treats all-zeros as unusable
rather than merely empty: six zero bytes passes a width check yet is the
exact value that produces the unattributable entry.
* test(dhcp): cover chaddr resolution and the unmasked allocation errors
* fix(eks): GPU node AMI declares uefi boot mode
The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but
the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under
legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at
firmware with zero serial console output.
* fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs
Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
initialize under the closed/proprietary kernel module: dmesg shows
RmInitAdapter failing with "requires use of the NVIDIA open kernel
modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to
nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia
recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no
separate open variant (userspace tools only), so it is unchanged.
* fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons
A cluster with a GPU-only nodegroup has no untainted node: the CP
carries CriticalAddonsOnly (EKS parity keeps user workloads off it)
and every worker carries nvidia.com/gpu. Without tolerations for both,
ebs-csi-controller and aws-load-balancer-controller have nowhere to
schedule and stay Pending forever. Real EKS never needs this since its
control plane isn't a schedulable node, so the stock upstream
manifests carry no such toleration.
* fix(images): give the libguestfs appliance a working resolver
The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
(127.0.0.53), which isn't running inside the appliance chroot, so DNS
is dead until package installs need it. Point it at a real resolver
before package installs, then restore the stub symlink afterward so
the shipped AMI still gets its DNS from systemd-resolved at boot like
a stock cloud image.
* fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind
A GPU's IOMMU group can include an upstream PCIe bridge when the
platform lacks ACS isolation between them. vfio-pci refuses to bind
non-endpoint bridge devices, so Claim/ReclaimByAddress must filter
bridge-class (0x06) members out of the group before touching driver
bindings, rather than failing the whole claim or leaving the bridge
rebound incorrectly.
* fix(ecs): GPU node AMI declares uefi boot mode
The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the
Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS
no OVMF/pflash firmware is attached, so the guest hangs at firmware with
zero serial console output.
Same defect and same fix as the eks-node-gpu entry: both images are built
by the same script from the same stock Ubuntu cloudimg, so both were
EFI-only and only one was corrected.
* fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch
ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary
kernel module. NVIDIA ships no proprietary module for Blackwell (GB202)
and later, so ECS GPU container-instances could not bind those cards on
any driver branch. It is the third verbatim copy of the routine and was
missed when the other two were fixed.
Also pins the driver branch across all three copies. The branch was
resolved at build time via apt-cache search | sort -rn | head -1, which
sorts on the number in the package name — but that number is a branch
alias, not a driver version (nvidia-dkms-535-server installs 580.126.20),
so the ordering key need not match what installs. Resolving also let
identical source produce different drivers on different days.
The pin lives in setup.sh rather than manifest.conf because manifest
variables are not passed into the guest: the setup script is uploaded and
executed by virt-customize --run, with no environment carried across.
* fix(eks): tolerate CP and GPU taints in argocd addon
All six argocd Deployments and its StatefulSet carried no tolerations. A
cluster with a GPU-only nodegroup has no untainted node to place them on:
the control plane carries CriticalAddonsOnly and every worker carries
nvidia.com/gpu, so the pods stay Pending forever.
Same defect the ebs-csi-controller and load-balancer-controller addons
already had. Real EKS never hits it because its control plane is not a
schedulable node, so the stock upstream manifests carry no such
toleration and the divergence is ours to maintain.
---------
Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>
* fix: detach before delete on instance terminate to stop root-volume leak
TerminateStoppedInstance and the running-instance terminate path both
deleted a DeleteOnTermination root volume without first clearing its
attachment, so DeleteVolume's in-use guard rejected it and the error was
swallowed (stopped path) or only alarmed (running path), leaking the
volume permanently.
Add VolumeServiceImpl.DeleteVolumeOnTerminate, which clears the stale
attachment via UpdateVolumeState (terminate implies detach) before
deleting, and route both terminate paths through it. Stamp
Teardown[TeardownVolumes] on the stopped path so a delete failure is
visible to the volume leak reaper instead of being lost.
* fix: detach non-DeleteOnTermination root on terminate; cover teardown self-heal
A DeleteOnTermination=false root volume stayed attached to the terminated
instance on both terminate paths: Stop's Unmount deliberately skips Boot
volumes, so the root kept its attachment across stop and neither terminate
path detached it, stranding it attached to an instance that no longer
exists. Add VolumeServiceImpl.DetachVolumeOnTerminate (metadata-only
attachment clear, no delete) and route the DeleteOnTermination=false Boot
volume through it on both paths, matching AWS terminate-implies-detach.
Also add a regression test proving the TerminatedTeardownReaper re-drives a
failed volumes-teardown through the now-detach-first cleaner and self-heals
the leak, so a transient delete failure at terminate does not strand a
volume permanently.
* fix: bound detach chain and reconcile vanished-QEMU wedge
DetachVolume gains a bounded aggregate context deadline over the QMP
hot-unplug chain and a confirmed-dead-QEMU short-circuit that skips the
QMP steps (which would only hang on the wedged process) and seals
directly, so a detach can no longer pin attachMu forever when nbdkit or
QEMU die mid-flight.
OrphanQEMUReaper now also finalizes shutting-down instances whose QEMU
has vanished: a terminate that wedged downstream and never reached
finalizeTerminated is driven to terminated, with outstanding teardown
stamped failed so TerminatedTeardownReaper completes it idempotently.
Gated on a dead process, so a live in-progress terminate is never
stolen.
* feat: add delete-authorized stuck-terminate backstop
A terminate that wedges with QEMU still alive is invisible to the
OrphanQEMUReaper dead-QEMU reconcile and would sit shutting-down forever.
StuckTerminateReaper force-completes it: past stuckTerminateTimeout it
force-kills the wedged process, reclaims DeleteOnTermination volume space
through the cleaner, stamps remaining teardown for TerminatedTeardownReaper,
and drives the record to terminated.
It is the one reaper explicitly authorized to delete volume data, but only
for a DeleteOnTermination volume on an already-terminating instance — a far
narrower licence than ADR-0005 §3 denies VolumeLeakReaper, which stays
mark-and-alarm-only. A new ShuttingDownAt timestamp, stamped once on the
transition, bounds how long a terminate may wedge; a missing timestamp reads
as not-yet-stuck so the backstop never acts on an unbounded record.
TestNPassOverwrite detaches after every pass, so viperblock's drain and GC
only ever run at reopen — it cannot show whether GC reclaims superseded
chunks while a volume stays open, and its serial write rate never nears the
predastore rate limiter's burst. TestContinuousChurn attaches once and drives
concurrent 4 MiB-aligned raw overwrites of per-writer regions for several
minutes without detaching, so a sustained superseded-chunk backlog builds
under one long open, spanning multiple GC ticks.
The workload is bounded and self-contained: each dd writes one chunk and the
clock is re-checked after every write, the loop runs as root under a single
sudo bash with a process-group kill trap, and it is wrapped in a guest-side
timeout. Root ownership is load-bearing — the writers open a raw device, so
only a root-owned process group can reap them; a best-effort guest pkill runs
before detach as a backstop so a stray writer cannot wedge the detach.
Picks up viperblock 3dcfbb7 (#49): the chunk GC sweep now runs on a
dedicated goroutine instead of sharing the chunk-uploader select, so
sustained overwrite churn no longer starves GC and wedges the volume at
the internal space cap.
This reverts commit f9c9f0b38d.
#580 switched setup.sh to read the northstar :53 listener and zones from
northstar.toml, so the resolved stage now defers at install time (before
formation) and must be re-run afterward. Only firstboot re-runs it; the
manual setup.sh + spx admin init install and CI never do, so those hosts
get no host DNS and cluster-zone names (ELB/EKS) fail to resolve from the node.
Reverting restores the pre-#580 behaviour: systemd-resolved is configured
at install time from the WAN IP, with no post-formation re-run needed. This
reintroduces the original gap where resolvconf hosts get no host DNS. The
proper fix configures host DNS from spx admin init/join for both resolver
managers (mulga-x30yv).
A freshly launched LB agent begins heartbeating the instant the VM boots,
but the gateway node fielding that first heartbeat may not yet have the LB
record — it is still replicating from the node that created it — so the
gateway answers 400 LoadBalancerNotFound. The IMDS credential chain is
likewise not always warm on the first tick. Both self-heal within a few
ticks, yet every early failure was logged at ERROR, flooding the console
telemetry with startup noise (46 such lines across 8 agents in one run).
Bound these to a registration grace window: before any heartbeat has ever
succeeded, and only within the window, a failure logs at INFO. Past the
window with no success — the fingerprint of a genuinely blackholed agent —
or after any prior success, a failure logs at ERROR as before, so the
serial-console telemetry still surfaces a real blackhole. The only change
for a persistent failure is a bounded delay before it escalates.
* fix: make the seal-failure unmount test deterministic
The test forced the seal to fail by pointing S3Host at an unreachable
endpoint, so the handler only returned once the S3 client had exhausted
its jittered backoff. Measured handler time spanned 1.8s to 5.5s against
a 5s request deadline, which failed 2-3 runs in 10 on an idle machine and
worse on a loaded CI runner.
Config gains a sealVolume seam, nil meaning the real seal, and the test
injects an immediate failure instead. It now runs in 0.53s, 50 for 50,
and still asserts what it was written to assert -- plus that the handler
actually attempted the seal, which the old form could not distinguish.
* chore: bump viperblock to dev tip (GC observability + scratch sweep)
Picks up viperblock #47 (every chunk GC sweep logs its outcome at Info)
and #48 (TestMain reclaims scratch dirs abandoned by a panicking run).
The tier ran inside the Build and Test job, serialised ahead of test-cover, and
again in preflight. Neither placement bought signal that a parallel job does not:
it provisions nothing and finishes in seconds, so the only effect was wall time
on the two paths developers wait for most.
It stays PR-blocking as its own job, which is what distinguishes it from
e2e.yml's front gate — that one is push-triggered on a self-hosted runner and so
never gates a PR before merge.
MgmtIPAllocator allocated from a process-local map, but br-mgmt is
trunked onto a shared physical NIC making the /24 one flat L2 segment
across the whole cluster. Every node independently allocated from
.10, causing collisions and ARP-race losers being blackholed.
Rewrite the allocator on the spinifex-cluster-state NATS KV bucket
using compare-and-swap, keyed by mgmt-ipam.<subnet>, modeled on IPAM.
Bind the KV handle in startCluster (not startLocal) to respect the
DDIL boundary. Serve already-allocated instance IDs from the local
cache when KV is unhealthy, but refuse new allocations until it
recovers. Rebuild now reconciles this node's running VMs into the
cluster-wide record instead of just refreshing the local cache.
* feat(config): enable viperblock chunk GC in rendered node configs
The gc_enabled toggle was plumbed through every VB construction site but
nothing ever set it. GCEnabled is a *bool that every consumer treats as
off when nil, and the config template rendered shardwal while omitting
gc_enabled, so no deployed node ever carried the key and chunk GC was off
everywhere by construction. Superseded chunks were never reclaimed under
overwrite churn.
Render gc_enabled = true alongside shardwal, and cover it with a test that
loads the rendered file through config.LoadConfig rather than matching the
template text. A string match would still pass if the value failed to parse
into a non-nil pointer, which is the exact way the setting reaches its
consumers.
Existing deployments keep the old rendering until their config is
regenerated; the template governs install and bootstrap only.
* chore: pin viperblock to the build with the cross-process GC snapshot marker
Enabling chunk GC in rendered node configs is only safe once the sweeping
engine can see a snapshot taken by another process. viperblock 3b56d61 adds the
per-volume snapshot marker that closes that window, and 2a34add fixes the
concurrent-stop panic in the background goroutine lifecycle that GC's own ticker
would otherwise widen. Pin both before turning GC on.
LaunchRunInstances is a best-effort batch launcher that reports failure only
through VM state: volume preparation, GPU claim and vmMgr.Run errors each call
MarkFailed and continue. launchAMISystemInstance ignored that and always
returned a SystemInstanceOutput, so an EKS control-plane server, ELB or Bedrock
serving VM whose launch failed handed its caller an instance ID for a VM that
was already being torn down.
Check the instance reached running state before returning. Cleanup stays with
the MarkFailed chain LaunchRunInstances has already run.
buildDrives emitted every non-boot data volume as a bare -drive with no
id= and no node-name=, so a stop/start (or reboot/resize) relaunch left
the volume anonymous: device_del/blockdev-del had nothing to address,
QEMU kept the NBD socket open, and DetachVolume stalled through
nbdkit's full kill grace before giving up.
Add -blockdev support to the QEMU config builder and use it for
non-boot, non-EFI volumes, sharing the node-name/device-id/iothread-id
naming and virtio-blk-pci device arguments with AttachVolume's hot-plug
path so cold-boot and hot-attach can no longer drift apart. A data
volume with no persisted HotplugPort now gets one allocated and written
back during buildDrives instead of landing on an unplugged bus.
Also raise unmountSealTimeout past viperblockd's kill grace (it runs
the seal after the SIGKILL, so matching the grace meant an unmount
that used the whole grace could never be observed as successful), and
bound the e2e harness's forced teardown detach by the teardown
deadline so a slow detach is reported as slow rather than as an
unexplained leak.
* test: assert the gateway->vpcd->OVN crossing in the integration tier
The tier already ran the real VPCServiceImpl for the AWS-facing half, but
nothing consumed the vpc.* events it published: the SG topics were answered
by a canned {"success":true} and the fire-and-forget topology topics were
dropped. Both sides of that wire were tested only against stubs, so the join
was asserted nowhere below the live e2e tier.
Wire a real subscribers.Subscriber over a real in-process OVN Northbound DB
(ovntest.StartNB), opt-in per test so only tests that assert OVN state pay
the startup cost. StartDaemonLite gains WithRealVPCD to suppress its stubs,
since a stub and the subscriber both answering one request-reply would race.
Scope is the crossing, not NB row shapes — those stay with the reconcile
scenario tests. Everything host-dependent (ovn-controller, OVS, taps, netns,
nftables, packet paths) remains the live tier's claim.
* fix: make the integration tier re-runnable and quiet the ensure probe
Two defects the vpcd/OVN work surfaced.
The tier was green at -count=1 and red package-wide at -count=2. Test
accounts were keyed by t.Name(), which is unique among tests but repeats
across passes in one TestMain, so a second pass inherited the first pass's
JetStream namespace — including an ECR signing key sealed with a master key
StartGateway regenerates per call. Key each account off a counter instead.
TestRunInstances_ClientTokenIdempotency then collided with itself for the
same reason: its dedup store outlives a pass, so its fixed ClientToken made
the second pass's opening launch look like a replay. Suffix the tokens per
invocation.
ensureNamedRowOps' wait-op is built to fail fast when the row it guards
already exists, which is the ordinary idempotent path, but transactOps
reported it as "OVSDB operation failed" at ERROR. Every re-ensure emitted
one, drowning real faults in the log sink. Classify that one result as the
probe outcome it is and log it at Debug.
* test: cover transactOps' failed-result paths
The classifier split one log line into two branches and neither was
exercised: transactOps sat at 31% because EnsureLogicalRouter's cache
lookup absorbs a re-ensure before it ever transacts, so no unit test
reached a failed operation result.
Drive both kinds directly against a real NB — the guard probe finding an
existing row, and two same-named inserts violating the schema's name
index — since only the second is a fault and they now log at different
levels. transactOps goes to 94%; the remainder is the transport-error
return, which needs a broken connection to reach.
* feat(ci): extend per-cell telemetry scoping to every e2e workflow
ci.cell only reached e2e-nightly and e2e-suites, so a run of e2e, e2e-ddil
or gpu-e2e still landed in the sink as one undifferentiated bucket. Stamp it
on the remaining workflows: single/multi on e2e's two provisioning jobs, ddil
and gpu on their single-job workflows.
The best-effort numeric job id lookup moves out of e2e-nightly into a shared
composite action, so the four other provisioning jobs get ci.job_id without
copying the gh/jq incantation. The lookup now matches on env.JOB_NAME rather
than interpolating the name into the jq filter, which broke on names carrying
quotes or non-ASCII separators, and no-ops when telemetry is disabled.
gpu-e2e gets ci.cell but deliberately no job id lookup: nothing in that job
bakes the attrs into a daemon, so the API call would be paid for an env var
no exporter reads.
* fix(ci): define e2e suite topology eligibility once
Three workflows carried three different answers to which suites a topology
can run. e2e-suites and e2e-nightly ran lb on their single-node leg, where
every internet-facing subtest skips for want of a peer, so a fraction of the
suite reported as a green lb run. e2e-suites' multi leg ran multinode alone,
dropping the lb and cert coverage that only a multi-node env can complete —
so full lb coverage never ran there at all.
.github/scripts/suite-sets.sh is now the single definition, sourced by all
three. e2e.yml's two allow-list regexes derive from it, e2e-suites derives
both matrices from it (multi_provision gains the suite-matrix step
single_provision already had), and e2e-nightly's deliberately narrower
per-cell lists are declared there too, guarded by an assert that fails the
cell if a subset ever names a suite its topology cannot fully exercise.
The nightly stays narrow on purpose: its cells have a 35-minute budget and
exist to prove each install/network/host-OS permutation boots and serves.
Suite breadth is the PR gate's job and already runs on every push. That is
now stated in one place rather than implied by a third divergent list.
* test(volume): assert chunk GC stays wired on in CreateVolume
GCEnabled was zero-valued at every viperblock construction site for a long
time and no test noticed. It is not part of viperblock.VBState, so it never
round-trips through a saved config.json and cannot be observed by reading
volume state back.
Extract the viperblock.VB assembly out of CreateVolume into buildVBConfig so
the GC wiring is assertable directly against the returned config, without a
live viperblock or S3 backend. The config is returned by pointer because
viperblock.VB carries atomic.Uint64 fields that vet's copylocks check
rightly refuses to see copied.
* test(e2e): snapshot backend byte accounting before volume purge
DeleteVolume purges a volume's entire backend prefix, including any chunks a
leak left behind, before anything measures it. That is the same masking that
hid the storage-growth leak in production: a passing E2E cannot fail on
backend growth when the evidence is deleted first.
Snapshot the volume's backend chunk footprint in EnsureVolume's cleanup
closure, immediately before DeleteVolume, so every volume the shared teardown
path purges is measured rather than silently discarded. The snapshot is
always taken and logged; the hard ratio check additionally runs for callers
that registered a guest-written-bytes baseline, since most callers never
attach or write to the volume they create and so have no baseline to compare
against.
The tolerance is the RS(2,1) erasure floor widened for per-chunk header and
tail-chunk overhead. It is sized to distinguish unbounded growth from a
footprint settling at the floor, not to catch bounded residual garbage.
The check is non-fatal inside the cleanup callback: t.Fatalf's runtime.Goexit
would abort the DeleteVolume call queued behind it and turn a reported leak
into a real one.
* test(e2e): add memory-constrained concurrent-write durability regression
The existing durability tests write one 4MiB sentinel, sequentially, on a
comfortably sized guest, which is the exact inverse of the known-open
data-loss bug: acked writes read back zero-filled after a cold reread when a
guest is memory-starved during concurrent writeback. No happy-path test can
reach it, because the fault lives in the intersection of memory pressure,
write concurrency and cache-cold reread.
Add a suite that encodes those three conditions together: a guest discovered
at or below the working set, N parallel large writers launched as background
jobs in a single command so they genuinely overlap, swap disabled, then sync
and drop_caches before rereading. Assertions are per stream rather than
aggregate so a failure names the losing writer instead of leaving whoever
reads it to re-derive which of N writers lost data from one combined digest.
GuestExec gains a timeout-parameterised variant. Its fixed two-minute default
is sized for sub-second payloads, and a multi-gigabyte write to a
network-backed volume would exceed it, so the suite would have failed by
timeout while looking like it had caught the corruption. The budget scales
with the working set, and a deadline is reported distinctly from a data
mismatch so the two red states are never confused.
The suite is dispatchable but deliberately absent from the default SUITES:
the bug it reproduces is still open, so it is expected to fail and must not
sit in a merge-gating path. It is registered in the workflow allow-list so it
stays discoverable rather than being buried behind a skip.
Also record on the live IAM instance-profile test why its assertions were not
moved down to the integration tier, so the trace is not repeated.
* test(e2e): drop backend byte accounting, assert a node-wide delta instead
The accounting helper listed the internal predastore bucket to attribute
bytes per volume. That listing is refused unconditionally: predastore
authorises bucket access by same-account ownership, and the bucket belongs
to the service account, so a test caller gets 403 no matter what it holds.
Granting the test a service credential would invert the trust boundary to
buy a measurement, which is the wrong trade.
Replace it with what the S3 surface actually permits: sample node-wide
allocated bytes either side of the write and bound the delta against the
bytes the guest wrote. Coarser, but it measures something real rather than
depending on a call that cannot succeed.
The tolerance moves into the suite that uses it. It encodes RS(2,1)'s 1.5x
expansion floor plus headroom; a cluster running the recommended RS(3,2)
has a 1.667x floor and only ~12% of margin left, so the constant needs
revisiting if the encoding changes.
* revert: drop memdurability e2e suite
The suite asserted the guest read back what it wrote after a
memory-constrained concurrent write, but the assertion could not fail for
the reason it claimed: it compared the guest's own read against the
guest's own write with the page cache still warm, so it passed whether or
not the data reached the backend. A test that is green on a build with a
known-open durability bug is worse than no test — it advertises coverage
that does not exist.
Removes the suite and its Makefile/workflow wiring, plus
DiscoverInstanceTypeAtLeastMemory, which had no other caller.
GuestExecTimeout stays: GuestExec now delegates to it, and the old
two-minute hard cap it replaced was a latent trap for any long guest
command.
* test(e2e): add partial-block concurrent-write durability suite
Replaces the removed memdurability suite, which passed on a build that
still carried the bug it claimed to test.
The engine reaches its read-modify-write path only for a partial block;
a fully covered block takes a full-overwrite branch that reads nothing
and can lose nothing. The old workload wrote through ext4 with 1 MiB
buffers, which the filesystem coalesced into aligned full-block
requests: block-layer tracing recorded zero unaligned and zero sub-4096
requests, so the path under test never ran, and writers 4 MiB apart
could not collide on a block even in principle.
This suite writes the shape the defect needs. Two writers issue
concurrent O_DIRECT writes into disjoint halves of the same 4096-byte
blocks on a raw, unformatted device, stamping every half with the round
number, then drop the page cache and reread with O_DIRECT asserting
every half carries the final round. One half reading back at an earlier
round is the signature of the lost update.
The old suite also could not fail even had it reached the path: its
write phase ended in 'wait; sync; sha256sum' and reported sha256sum's
exit status, while bare wait returns 0 unconditionally, so writer errors
were invisible and a backend ENOSPC presented as a pass. Here each
writer's exit status and exact byte count are asserted before any
durability claim is made.
Working set is 8 MiB rather than 2048 MiB, and the volume is deleted on
teardown; the predecessor left roughly 3 GiB of backend per run and
twice drove a node under the nearfull watermark.
Dispatchable but not in the default suite list until a green run and a
demonstrated fail-first against a pre-fix engine exist.
* test(e2e): let the fixture own the working-set volume delete
The suite deleted the volume in its own cleanup and the fixture deleted
it again on close, so every run ended with an InvalidVolume.NotFound
from the second attempt. Detach stays — DeleteVolume against a still
attached volume is what leaks volumes on teardown — and the delete is
left to the fixture so the volume is released exactly once.
* test(e2e): gate merges on the partial-block durability suite
Verified on one node with the same test binary and workload, varying
only the engine: 1758 of 8192 halves reverted a generation against the
pre-fix build, 0 of 8192 against the fixed one. That is the fail-first
evidence the suite needed before it could be trusted to gate anything.
Adding it to the default suite list rather than leaving it dispatchable
is the point. Its predecessor sat outside every path that runs, which
is how it survived long enough to be discovered green on a build that
still lost data.
* test(integration): cover the ECR data plane in the front gate
The live ecr/ suite drove docker, crane and skopeo against a real
registry, yet none of it boots a guest: it needs a registry, an object
store and an HTTP client. The in-process gateway already serves the OCI
Distribution v2 surface on an ephemeral port with a memory-backed object
store, so these run here in 0.18s instead of behind full provisioning.
Covers the three claims the live tests made beyond authorization, which
is already down here: a push/pull round trip asserting blobs and
manifests read back byte-identical, refusal to auto-create a repository
on push, and retagging an existing manifest without disturbing the
original tag.
Content is compared rather than status codes alone. A registry that
stores a blob under the wrong digest, truncates it, or serves a
different one answers 200 to every request in the sequence.
The capacity/load leg deliberately stays live: it exists to put real
parallel pressure on a real predastore, and a memory-backed object store
cannot make that claim.
* fix(e2e): make volume teardown release the volume before deleting it
Fixture cleanups drain LIFO, so a suite that creates its instance before its
volume tears the volume down first, while it is still attached. DeleteVolume
answered VolumeInUse, nothing retried, and the volume was stranded on the node.
Reordering the chain cannot fix this — registration order has no way to express
the instance/volume dependency. Instead teardown is now self-sufficient for any
creation order: it force-detaches, waits for the volume to reach available, and
only then deletes. Instance teardown waits for terminated for the same reason,
since TerminateInstances returns as soon as the request is accepted.
Two things kept the leak invisible. A failing cleanup was logged and forgotten,
so a run stayed green with resources left behind; failures are now reported as
leaks, failing the bound test or surfacing through Close() for TestMain to fail
the run. And a panic in one cleanup aborted the whole chain, stranding every
resource behind it; each callback is now isolated.
RegisterVolumeTeardown and the fixture had grown separate teardown paths, only
one of which detached. Both now share it.
Re-creating a nodegroup of the same name could never reach ACTIVE. The
Ready-gate took its baseline from the live Ready count for that name, so
a terraform replace after CREATE_FAILED — which deletes then re-creates
— captured the previous incarnation's workers, demanded old+new, then
watched the count fall as the old workers were terminated. The target
was unreachable by construction.
Observed on env19 with a perfectly healthy cluster: three new workers
all joined and the create still failed with
reports 3 Ready nodes, want >= 5 (baseline 2 + 3 workers)
and each retry raised the bar again (baseline 3 => want 6), so the only
way out was untainting the resource by hand.
A create means every worker under that name is new, so the gate is just
"desired Ready nodes carrying this nodegroup label" — baseline 0. The
trade-off is stated in the comment: a stale incarnation's Ready nodes
can briefly inflate the count and satisfy the gate early, but they go
NotReady within about a minute of termination, whereas baselining on
them fails every replacement permanently.
The test models the real sequence — delete, stale Ready count still
reported, re-create — and fails without the fix with the same shape as
the field error.
* fix(dhcp): derive nclient4 read deadline from the caller's context
nclient4 races its own read deadline against the caller's context and the
shorter one ends the attempt. Acquire and Renew passed a fixed 5s, so every
window in Manager's 4s/8s/16s/32s retransmission schedule past the first was
truncated to 5s: the client listened for ~19s of its 90s budget and a server
that went quiet for longer was never waited out.
Measured in the sink, the split was total — attempt 1 (jittered under 5s)
always failed with the caller's context deadline, attempts 2 to 4 always
failed with nclient4's ErrNoResponse — and only 27% of clients that lost
attempt 1 were rescued by attempt 2.
Track the context's remaining time instead, keeping the configured value as
the fallback its doc comment already describes.
* fix(dhcp): keep the socket deadline strictly behind the caller's
Setting the nclient4 read deadline to exactly the context's remaining time
makes the two expire together, so a timed-out attempt reports either
context.DeadlineExceeded or nclient4's ErrNoResponse depending on which
goroutine wins. Observed live on a settled bridge: three attempts reported the
context deadline and the fourth reported ErrNoResponse, which reads as a
packet-matching fault rather than the plain timeout it is.
Give the socket a second of grace so ctx.Done() is always what fires.
* fix(dhcp): encode option 61 with its hardware-type byte
The client identifier was sent as a bare string, so the server read the
identifier's first character as the hardware type — "dhcp-gw-lrp-vpc-x"
arrived as type 100 ('d') with the identifier truncated. Since that is not
Ethernet, the server recorded no hardware address, and the upstream lease
table listed our leases with a MAC of all zeroes.
Send the chaddr typed as Ethernet, falling back to a type-0 identifier when
no hardware address is available. RELEASE now encodes identically, since the
server matches a release to a lease by client identifier.
Option 61 no longer carries the readable id, so restore it via hostname and
vendor class, which were specified but never populated. dhcptest builds its
options through the same helper rather than its own copy, which had drifted
into reproducing the broken encoding.
* fix(dhcptest): derive the probe MAC the way vpcd does
The probe carried its own copy of the MAC derivation, still on the 24-bit
02:00:00:xx:xx:xx hash that vpcd moved off when widening it to a SHA-256
prefix. So --spinifex-id leased under a MAC the daemon would never use:
"gateway-wan1" resolved to 02:00:00:7c:b7:b6 here and 02:54:df:71:22:5c in
vpcd, which makes the probe unable to reproduce a daemon lease at all.
Call DeriveMAC instead and drop the local copy.
The nightly matrix runs every cell under a single ci.run_id, and each cell
names its hosts node1/node2/node3, so telemetry from all cells collapses into
one undifferentiated bucket in the ES sink. Root-causing a single failed cell
means aggregating across every other cell in the run.
Stamp ci.cell and ci.matrix alongside the existing ci.run_id. Workflow-level
env cannot reference the matrix context, so the attrs are overridden in each
matrixed job's own env block, leaving the workflow-level value as the
non-matrix fallback.
The numeric GitHub job id is not exposed in any context (github.job is the
YAML key, not the id), so resolve it best-effort via the jobs API and append
it if found. The lookup never fails the job.
e2e-suites can only distinguish single from multi: daemon telemetry is baked
into the VM at provision time, before the suite matrix exists.
Baremetal shipped no telemetry at all — neither the endpoint nor the attrs
crossed the workflow_call boundary. Pass them through as an explicit input so
the cell reaches the sink for the first time.
* fix(eks-node): stop the DHCP MTU overriding the pinned link MTU
Pinning eth0 to 1320 was never enough. This image runs dhcpcd, which
honours DHCP option 26 and stamps the advertised MTU onto every route it
installs:
default via 10.32.101.1 dev eth0 proto dhcp src 10.32.101.7 \
metric 1002 mtu 1442
Linux prefers a route's MTU metric over the device MTU, so the node kept
advertising an MSS derived from 1442 while the real path MTU through the
NAT gateway is 1408. Small packets pass and full-MSS inbound segments —
a TLS ServerHello with its certificate chain, image layers — are
silently dropped. It presents as `net/http: TLS handshake timeout` on
image pulls, not as anything MTU-shaped, and it only affects the host
netns, so containerd pulls fail while pod traffic sized off cni0 is
fine. Per-destination PMTU exceptions expiring in and out of the route
cache made it look intermittent and host-specific.
Three changes, innermost first:
- dhcpcd.conf gets `nooption interface_mtu`, so no lease can stamp a
route MTU. mulga-vpc-mtu is the single source of truth for the MTU
of these nodes.
- mulga-vpc-mtu re-stamps any route that already carries a different
MTU, covering an image built before this fix or a dhcpcd already
running. Routes with no MTU metric inherit the device MTU and are
left alone.
- the subnet DHCP option drops 1442 to 1408. 1442 is geneve-only and
ignores the IPsec ESP overhead on the NAT egress path, so it
overstates the usable MTU for every guest, not just these nodes.
Also corrects the header comment, which claimed Alpine's udhcpc ignores
option 26 — this image runs dhcpcd, which honours it. That wrong premise
is why the route metric was never considered.
Adds a stubbed-`ip` test covering the rewrite, the dev token that
`ip route show dev X` omits, idempotency, and routes without an MTU
metric.
* fix(eks): sweep the CP VPC by identity so teardown can finish
DeleteClusterCPVPC finds the managed control-plane VPC's subnets and SGs
by role tag. A create that fails partway — the first control-plane
server failing to launch, say — leaves resources behind that never got
tagged, so the sweep cannot see them.
DeleteVpc rejects while any subnet or non-default SG remains, and
teardown is re-driven on a timer, so the cluster sits in DELETING
forever retrying the same doomed delete every couple of minutes:
eks-deleting: re-driving wedged DELETING teardown cluster=toc
eks-deleting: re-drive teardown failed
err="delete managed CP VPC: eks: delete cp vpc vpc-...: DependencyViolation"
Nothing converges: the cluster name cannot be reused and terraform's
delete wait can never pass. Observed on env19, wedged for ~11 hours
until this change went in, at which point the next re-drive cleared it.
After the tagged sweep, clear anything still in the VPC scoped by
vpc-id: subnets, and non-default SGs (the default SG goes with the VPC
cascade). Teardown had no SG surface, so CPVPCDeps gains an optional SG
provisioner — nil is tolerated, since only teardown uses it.
* chore: pin viperblock + predastore to the merged storage fixes
viperblock f8cfd77 (#42) makes read-modify-write atomic per block,
removing the lost update that silently reverted a 4096-byte block to a
previous generation of itself. Confirmed as the env19 corruption cause
by an A/B: the pre-fix build corrupted image layers within ~15 minutes
(failed to extract layer / invalid tar header / unpigz: corrupted) while
the fixed build ran 5+ hours clean, absorbing 6407 same-block RMW
conflicts on the worker root disks.
predastore c0a90d2 (#63) batches FSM.Restore. Without it, any predastore
restart on a store whose metadata set outgrew one badger transaction
fails with "Txn is too big to fit into one request" on every node at
once, taking the metadata plane — and with it the AMI catalogue and
cluster creation — down permanently.
Both were verified in the built artifact rather than from go.mod alone:
go version -m reports the new pseudo-versions, and the rmw_conflicts
counter plus the batched-restore error strings are present in the
binary. Pinning by go.mod while the submodule pointer says otherwise is
how an earlier analysis ended up reading dead code.
* test: record the corrected DHCP MTU in the kitchen-sink golden
The NB golden pins the DHCP options published per subnet, including the
advertised MTU. Dropping the subnet option from 1442 to 1408 changes
what the reconciler writes, so the golden diverged and the live scenario
test failed — the test doing exactly its job.
Regenerated with UPDATE_GOLDEN=1. The diff is only mtu:1442 -> mtu:1408
on the two subnet DHCP records; nothing else in the snapshot moves.
* test: add in-process integration test tier
Roughly half of the behavioural tests under tests/e2e/ assert control-plane
API shapes, authz and error codes that need no live environment, yet they can
only run against a tofu-provisioned environment on a scarce self-hosted
runner. This adds the tier they can move down to.
The real gateway router runs in-process: SetupRoutes() is a pure function of
GatewayConfig, so it serves from an httptest server over embedded NATS
JetStream with real IAM/STS services and genuine SigV4 verification. Only the
daemon side of the wire is faked -- the ec2.*/ebs.* NATS subjects a live node
would answer are stubbed per test.
Nothing is provisioned and no fixed ports are bound, so the tier runs
concurrently on a shared build host and is safe as a CI front gate ahead of
the live suites. It is build-tagged and kept out of preflight, which stays
unit-only.
* test(integration): add IAM/STS client accessors and port the zero-stub authz tests
Adds IAMClient/STSClient plus ClientsWithCreds/ClientsWithSessionCreds so a
test can act as a principal other than the seeded root user, and stubs the
node-discovery subject in StartGateway so call paths that trigger discovery
resolve immediately instead of riding out the fan-out timeout.
Ports the four enforcement tests that guard on ec2:DescribeRegions, a
gateway-local action, so they need no daemon-side stub at all.
* test(integration): port remaining IAM/STS authz tests from e2e
Ports TestIAMUserCRUD, TestIAMAccessKeyLifecycle, TestIAMUserAuthentication,
TestIAMPolicyCRUD, TestIAMPolicyAttachmentEnforcement, TestIAMPolicyLifecycle,
TestIAMGroupsLifecycle, TestIAMRolesAndProfiles and
TestSTSAssumeRoleAndGetCallerIdentity from tests/e2e/iam/ into the in-process
integration tier. Each test now self-bootstraps its own users/policies/roles
instead of relying on shared suite-level state, since every integration test
gets its own gateway and embedded NATS. TestIAMRolesAndProfiles stubs
ec2.IamProfileAssociation.describe because DeleteInstanceProfile always
counts live associations over NATS regardless of which guard actually fires.
TestIAMCleanup is not ported: it only swept a shared long-lived E2E
environment, and every ordering/dependency assertion it made is already
covered by TestIAMPolicyLifecycle, TestIAMGroupsLifecycle and
TestIAMRolesAndProfiles's own delete-conflict guards.
* test(e2e): remove IAM/STS authz tests migrated to the integration tier
All 13 signed-request authz tests plus TestIAMCleanup now have faithful,
mutation-proven equivalents in tests/integration/ — the gateway dispatches
IAM/STS actions directly with no daemon hop, so the in-process tier is a
faithful and far cheaper substitute. The iam/ e2e suite now runs only
TestIAMInstanceProfileAssociation and TestIMDS.
Drop the now fully-dead iam_helpers_test.go alongside the six deleted test
files, and remove harness.IAMGroupARN/IAMDeleteGroupBestEffort, which had no
remaining caller anywhere in the repo once iam_groups_test.go was gone.
* feat(integration): daemon-lite harness for key/tags/routetable/vpc
Adds DaemonLite, an in-process stand-in for a live spinifex daemon that
subscribes the real key, tags, route table, and VPC/subnet/SG/IGW service
implementations directly to their NATS subjects, backed by memory and
embedded-JetStream stores. Ports TestKeyPairs, TestTagManagement,
TestRouteTableValidation, and TestReplaceRouteConvergence from
tests/e2e/single so the integration tier drives genuine daemon-side
business logic instead of StubSubject canned replies.
vpcd (OVN) is out of scope, so its "vpc.create-sg"/"vpc.delete-sg"
acknowledgements are stubbed with a fixed success reply — the SG record
itself is still written by the real service impl before that event fires,
so the stub only unblocks the synchronous round-trip rather than
replacing tested behavior.
* ci(e2e): gate live E2E jobs behind the in-process integration tier
Runs the in-process integration suite first, on the ci-single runner host
rather than inside a provisioned VM, so a broken control plane is caught
before either live job pays the Spinifex install. Both e2e_single and
e2e_multi now require it, and e2e_status treats a skipped integration job as
a failure so the gating status cannot go green on a gate that never ran.
Drops the shadow-mode suite selector: targeted per-diff suite selection is
not the direction being pursued, and leaving a non-functional shadow job in
the gating workflow only invited it to be mistaken for a live gate.
* test(e2e): remove single/ tests superseded by daemon-lite ports
TestKeyPairs, TestTagManagement, TestRouteTableValidation, and
TestReplaceRouteConvergence now have proven-equivalent ports in
tests/integration/ (mutation testing confirmed parity). Delete the
e2e originals and their runX helpers, along with keypair_test.go's
and tags_test.go's file-local helpers left with no caller.
generateImportPubKey is shared by other single/ tests so it moves to
helpers_test.go instead of being deleted. Slims the manifest-lint
baseline for the removed create-call entries.
* ci: run the lb e2e suite on multinode only
Both e2e jobs filter SUITES by exclusion, and neither excluded lb, so the
suite ran twice per e2e run: 354s on single and 654s on multi. Balancing
across instances on one node is not a deployment mode we support, and
ELBv2 proper requires two AZs, so the single-node run was redundant.
Drops ~354s from the single-node job, which is the gate critical path.
* test(integration): port 3 of 7 ECR tests to the in-process tier
Wires the gateway's ECR control plane (repository/policy/lifecycle
metadata over an in-process daemon-lite NATS subscriber, blob and
manifest bytes via a memory object store, JWT issuer/verifier keyed off
the existing IAM master key) into the integration harness, and ports
TestECRControlPlane, TestECRGetLoginPassword, and
TestECRIAMAuthorization_AssumedRolePullOnly from tests/e2e/ecr — each
proven equivalent by mutation (evaluatePrincipalPolicy forced to allow,
and the repo-describe path forced to not-found) before its E2E original
was removed.
The other 4 authorization tests (PullOnly, ReadOnly,
DetachPolicyDeniesImmediately, DeactivatedKeyReturns401) stay on the E2E
tier: AttachUserPolicy lacks the AWS-managed-ARN existence-check bypass
that AttachRolePolicy/AttachGroupPolicy already have, so attaching
AmazonEC2ContainerRegistryPullOnly/ReadOnly straight to a user fails
NoSuchEntity in both tiers alike. Left untouched pending a fix to that
shared IAM code.
* fix(iam): accept AWS-managed policy ARNs on AttachUserPolicy
AttachUserPolicy called getPolicyByARN unconditionally, so attaching an
AWS-managed ARN to a user returned NoSuchEntity. AttachRolePolicy and
AttachGroupPolicy already skip that lookup for AWS-managed ARNs and store
them opaquely; the user path was missed when that guard was introduced.
ListAttachedUserPolicies had the mirror of the same gap: it resolved every
ARN through getPolicyByARN and silently dropped the ones it could not find,
so a managed ARN would disappear from the listing rather than round-trip.
Evaluation already handled these correctly via resolveAttachedPolicy, so
only the write and list paths needed the guard.
* test(integration): port remaining 4 ECR authorization tests
Ports TestECRIAMAuthorization_PullOnly, _ReadOnly,
_DetachPolicyDeniesImmediately, and _DeactivatedKeyReturns401 from
tests/e2e/ecr now that AttachUserPolicy accepts AWS-managed policy ARNs.
Each was proven equivalent by mutation before its E2E original was
removed: PullOnly/ReadOnly/DetachPolicyDeniesImmediately fail when
evaluatePrincipalPolicy is forced to always allow, and
DeactivatedKeyReturns401 fails when the access-key-active check in
resolveECRLongLivedPrincipal is bypassed. All mutants were reverted and
a clean run confirmed before committing.
tests/e2e/ecr/iam_authorization_test.go is now fully removed — every
test it held has an in-process equivalent, and its helpers had no other
callers in the package.
* ci: select e2e suites by allow-list instead of deny-list
Both jobs filtered SUITES by exclusion, so any suite named in neither
deny-list ran in both jobs. That is how lb came to run twice per e2e run,
and it makes adding a suite silently fan it out everywhere.
Each job now names the suites it is eligible to run. Selection is still
driven by SUITES, so this is behaviour-preserving for the current default:
single/iam/cert on the single-node job, multinode/lb/cert on the multi-node
job, exactly as before. eks, ecs and storagegrowth are listed as eligible on
the single-node job but are not yet in the default SUITES.
* test(e2e): delete seven tests with no coverage
TestFinalClusterStats and TestInstanceLaunch assert nothing beyond
what other callers of their shared helpers already imply. The
multinode baseline, volume, guest-SSH, and preflight tests duplicate
single-node coverage or exercise per-guest behaviour that isn't
cluster-specific. TestMultinodeInstanceDistribution can't fail: three
independent RunInstances calls with no placement group make a strict
spread assertion flaky and a loose one meaningless.
Preflight checks (/dev/kvm writability, peer SSH reachability) move
into requireMultiNodeFixture so the package fixture singleton still
fails fast with a clear message instead of disappearing along with
the dedicated test. The remaining callers of needInstanceTrio
(CrossNodeOps, CrossNodeGateway, NodeFailure, VolumeDurability) still
trigger the shared trio launch on their own, so removing the
distribution and volume-lifecycle tests doesn't leave it unbuilt.
Also slims the manifest-lint baseline entry for the deleted
multinode volume test.
* fix(e2e): require full 3-node spread in TestMultinodeSpread
SpreadPlacement passed at unique>=3, warned via t.Logf at unique==2,
and only failed below that. A 33% spread failure (two instances
colocated) passed with a log line. The 243.6s NAT-egress scenario that
follows only proves what it claims to if the private trio actually
landed on three distinct chassis, so that premise needs to be a hard
assertion, not a warning.
* refactor(e2e): drop the unused spread-assertion harness helpers
AssertSpreadAcrossNodes and CountInstancesPerNode lost their only caller when
the instance-distribution test was removed. Beyond being dead, the assertion
helper documented minNodes=1 as an acceptable logging-only mode inherited from
the bash driver, which is the pattern that let a placement test pass while
every instance landed on one node.
InstanceHostingNode and its ps helpers stay; five multinode tests use them.
* fix(ec2): stop leaked goroutines on prepareEFIVolume error paths
newViperblock starts the chunk uploader and WAL syncer unconditionally,
but prepareEFIVolume only released them on the happy path via Close().
Six error returns between acquisition and Close left both goroutines
running. Add a defer immediately after acquisition, matching the
sibling prepareRootVolume; Close() remains the durability boundary.
* test(predastore): add real predastore fixture for integration tests
Extract predastore_integration_test.go's inline daemon startup into a
reusable testutil/predastore package so tests/integration can point
DaemonLite-style wiring at a real S3-compatible backend instead of an
unreachable host. Kept as its own leaf package rather than folding into
the shared testutil package: it transitively imports predastore's
quicclient, whose package-level connection pool starts a permanent
goroutine at init that would trip goleak checks in any of the ~80
other packages that import testutil for unrelated NATS helpers.
* test(integration): migrate CreateVolume to assert real predastore persistence
Add StartVolumeDaemonLite, an opt-in DaemonLite variant wired to the
real predastore fixture, and migrate TestCreateVolume_PersistsToRealPredastore
off memory-backed DaemonLite so CreateVolume/DescribeVolumes exercise
viperblock's real S3 backend end to end instead of stopping at input
validation.
* test(integration): add real storage-state oracle for CreateVolume
DescribeVolumes only proves the handler's own read path agrees with its
own write path, both via the same deserialization; it cannot catch a bug
in a field it never re-derives (e.g. a unit-conversion error in the bytes
handed to viperblock, since VolumeMetadata.SizeGiB is copied from the
request directly). vbscan reads a volume's persisted config.json and
chunk objects straight over S3, independent of any spinifex handler code,
so TestCreateVolume_PersistsToRealPredastore can assert on what viperblock
actually wrote rather than only on what the handler claims.
vbscan is a local reimplementation of scripts/vbrefscan's read-only scan
rather than an import: spinifex cannot depend on mulga, the module that
contains it. It only reimplements the config/chunk-listing half of
vbrefscan — the checkpoint-based reference-integrity half needs
viperblock.ParseBlockCheckpointBytes, which does not exist in the
viperblock version spinifex currently depends on (v1.12.1) and is also
moot for this volume: CreateVolume never calls SaveLiveCheckpoint, so a
freshly created volume has no checkpoint to scan regardless of version.
* refactor(test): move the predastore daemon fixture out of the coverage scope
test-cover runs go test ./spinifex/..., so anything under spinifex/ is
compiled and measured. The daemon fixture cannot be exercised by unit
tests, so it reported a coverage gap no unit test could close. The e2e
harness already avoids this by living under tests/; the fixture now
does the same. vbscan stays put, being ordinary library code with its
own unit tests.
* test(integration): migrate RunInstances multi-count expansion off live guests
RunInstances with MinCount/MaxCount=2 expands into a single per-node
targeted launch (distributeInstances in gateway/ec2/instance/placement.go)
before the daemon-facing NATS hop, so the real gateway router plus a
stubbed daemon response is sufficient to prove it — no throwaway VM is
needed to exercise the expansion logic itself.
The live single-node E2E test's WaitForInstanceState/TerminateInstances
sequence was teardown hygiene for the real guests it booted, not an
assertion of the expansion behavior, so it is dropped rather than
emulated. The migrated test additionally asserts the per-node launch
request itself carries MinCount=MaxCount=2, proving the count actually
propagates rather than the launch succeeding by coincidence or via the
zero-daemon degenerate path in placement.go.
* docs: drop a doc reference to the deleted live test
* test(integration): migrate launch-template RunInstances expansion off live guests
RunInstances --launch-template resolves ImageId/InstanceType from the
stored template entirely gateway-side (expandLaunchTemplate, before
validation/placement/the daemon-facing NATS hop), so proving it forwards
the resolved fields never needed a real guest boot. Ports the assertion
to tests/integration via a new opt-in StartLaunchTemplateDaemonLite (real
LaunchTemplateServiceImpl over embedded JetStream) plus a direct
subscription on the per-node launch subject to capture what the gateway
actually dispatched, rather than only the client-visible response.
The live test keeps its CRUD/versioning lifecycle (create/modify/delete
version, tag-filtered describe) — none of it touches a guest — and drops
only the RunInstances-from-template + wait-for-running + terminate tail,
whose only job was waiting out a real boot to read back fields already
proven resolved.
* test(integration): migrate KV-only account-scoping checks off live tenants
Ports the KeyPair/VPC-Subnet/IGW-EIGW/AccountSettings cross-account
isolation assertions from tests/e2e/single/account_test.go to the
in-process integration tier. Each of these resources resolves ownership
from a plain account-scoped KV key (utils.AccountKey(accountID, id), or
an account-keyed settings record) with no vm.Manager or real guest
involved, so DaemonLite now also wires the real EgressOnlyIGWServiceImpl
and AccountSettingsServiceImpl alongside the existing VPC/IGW/Key
services. Two independently minted tenant accounts (via
IAMService.CreateAccount/CreateUser/CreateAccessKey/CreatePolicy/
AttachUserPolicy, in-process — the same calls runAccountCreate makes
without the CLI/stdout round-trip) replace the live tier's carousel so
isolation is checked between real, separately owned accounts.
Left live: instance scoping (Step2) and volume scoping (Step3) —
checkInstanceOwnership, the actual cross-account gate for stop/start/
reboot/terminate/attach-volume, runs daemon-side only after a real
vm.Manager lookup, so it cannot be exercised without a live guest.
Snapshot scoping (Step5) needs a real backing volume for CreateSnapshot
(viperblock, out of DaemonLite's scope), so it stays live entirely rather
than split off only its two KV-only assertions. Global-resources (Step9)
carries no account branching in the handlers at all, so asserting its
equality across two accounts wouldn't exercise anything a fake node
responder didn't already trivially satisfy — not migrated.
* test(integration): migrate launch-template CRUD lifecycle off live guests
TestLaunchTemplates booted zero guests: its RunInstances coverage had
already moved to TestRunInstances_LaunchTemplateExpansion, leaving only
template CRUD/versioning assertions occupying a full live-cluster slot.
Port the remaining create/describe/version/modify/delete lifecycle,
including the SourceVersion merge and a nested timestamp round-trip, into
TestLaunchTemplates_CRUDLifecycle against the in-process integration
gateway.
* refactor(tests): move spot instance lifecycle coverage to integration tier
RequestSpotInstances is a mock over on-demand RunInstances (no bidding,
interruption, or reclamation), so booting real VMs in the e2e tier proved
nothing spot-specific. Migrate the SIR CRUD lifecycle (request, describe,
cancel, close-on-terminate, insufficient-capacity error) to a NATS-level
integration test that exercises the real gateway orchestration and daemon
service impl without a live guest. Real-VM-boot assertions are already
covered by TestRunInstances_MultiCountExpansion and the e2e baseline tests.
* fix(tests): share one embedded NATS server across the integration package
ClientTokenStore in spinifex/gateway/ec2/instance is a process-wide
sync.Once singleton bound to whichever *nats.Conn first initialises it.
Every integration test booted its own embedded NATS server, so the first
RunInstances-calling test captured the singleton, and any later test
inherited a KV handle bound to that server's now-closed connection.
TestMain now boots one embedded NATS+JetStream server for the whole
package; every StartGateway call connects into its own dynamically
provisioned NATS account (accountAuthenticator in shared_nats.go), giving
each test a private JetStream namespace despite the shared server, and
none of the per-test connections are closed until every test has finished
- whichever one the singleton latches onto stays live for the run.
Also wire the suite into the required PR gate: it was previously only
invoked from e2e.yml's push-triggered (main/dev), self-hosted "Integration
(front gate)" job, so a regression here never blocked a PR. The whole
package runs in single-digit seconds, so it now runs inline in
`make preflight` and as its own step in spinifex.yml's build_and_test job.
* test(integration): migrate serial-console access checks off live guests
Enable/Disable/GetSerialConsoleAccessEnabled is pure gateway+account-settings
logic with no guest involved, so it moves to the in-process integration tier
against the real AccountSettingsServiceImpl wired into DaemonLite.
* test(integration): migrate DescribeImages round-trip off live guests
runImage's AMI describe round-trip needs no real cluster: it registers an
AMI against a hand-seeded snapshot via the real ImageServiceImpl, backed by
a new StartImageDaemonLite helper wiring RegisterImage/DescribeImages onto an
in-memory object store.
* test(integration): migrate predastore object lifecycle off live guests
runPredastoreObjectLifecycle only needs a real predastore daemon, not a
tofu-provisioned cluster: it now runs against tests/fixtures/predastore's
in-process fixture, covering the same create/put/get/delete/list contract.
* test(integration): migrate ListFoundationModels catalog check off live cluster
ListFoundationModels is pure gateway-side static catalog logic with no
NATS/daemon hop, so it moves to the integration tier via a new BedrockClient
harness accessor; this tier configures no credential resolver, so the
Anthropic entry's absence is asserted explicitly rather than left optional.
* test(integration): cover RunInstances client-token idempotency
Proves replaying the same ClientToken through the real gateway returns the original reservation without a second daemon dispatch, and a different token launches independently.
* test(integration): cover RunInstances launch-template expansion
Proves version selection (omitted Version resolves to the template default, explicit Version selects that version) and field precedence (explicit RunInstances parameters override the template, unset fields inherit it) through the real gateway.
* test(integration): cover RunInstances malformed-input rejection
Proves a malformed ImageId and a malformed targeted-launch CapacityReservationId are rejected before any NATS dispatch, with no daemon stub registered so a regression that let either through would fail fast instead of hanging.
* test(integration): cover RunInstances iam:PassRole enforcement
Proves a caller granted ec2:RunInstances but not iam:PassRole on an instance profile's role is denied, and the identical session succeeds once PassRole is granted for that exact role ARN.
* test(integration): cover RunInstances placement-group strategy routing
Adds a placement-group daemon-lite wiring a real PlacementGroupServiceImpl over NATS, then proves spread strategy assigns exactly one instance per node while cluster strategy pins the whole launch to the single highest-capacity node.
* test(integration): cover RunInstances capacity distribution across nodes
Proves the default (non-placement-group) path packs a multi-instance launch across nodes proportionally to spare capacity rather than exhausting one node first.
* test(e2e): merge SameSGComms and SGToSGDatapath around one client/target pair
Both booted their own instance pair to check SG-to-SG datapath enforcement
and same-default-SG east-west connectivity. Merging them around one shared
client/target pair cuts four boots to two, and converting the inline
harness.Step sequence into named t.Run stages (PortGroupMembership,
AllowedTraffic, DeniedTraffic, RevokeAndVerifyBlocked,
ReauthorizeAndVerifyRestored, SameSGComms) gives each its own pass/fail
attribution instead of one opaque test result.
SameSGComms now runs last and joins the default SG onto each ENI via
ModifyNetworkInterfaceAttribute only after every SG-to-SG allow/deny
assertion has already recorded its own result, since the default SG's
self-reference rule admits all traffic between co-members and would
otherwise defeat the denied-traffic check.
* test(e2e): merge default-SG reachability, guest DNS, and egress enforcement
DefaultSGReachabilityBaseline, GuestDNSResolution, and SecurityGroupEgress
each booted their own instance to check one facet of "a guest is reachable
only when policy says so": ingress deny/authorize, resolver + WAN egress,
and egress ACL enforcement. Merging them around one shared guest cuts
three boots to one, with named t.Run stages (Deny, Authorize, Resolver,
WANEgress, RevokeEgress/RestoreEgress) giving each its own attribution.
The egress revoke/restore round-trip now mutates this scenario's own
dedicated SG instead of the shared default SG the rest of the suite
depends on, isolating its egress-rule mutation from every other test.
* fix(ci): single-quote e2e test_run regex across the ssh boundary
* fix(e2e): anchor SG egress-enforcement probe to a WAN IP, not the gateway
* test(e2e): merge single-node guest churn tests into one durability scenario
Combines TestSSHProbe, TestENIHotplug, TestENIHotplugReconcile,
TestVolumeDurability, TestModifyInstanceAttribute, and TestRebootInstance
into TestGuestChurnDurability, a single shared-guest scenario that
re-verifies a data-integrity sentinel sha256 after every perturbation
stage (ENI hotplug, daemon restart, ENI detach, instance-type resize,
stop/start, reboot). This proves the sentinel survives the full churn
sequence rather than just one isolated stop/start.
* fix(vpc): clear subnet drop gate when installing egress reroute
A subnet with an attached IGW but no 0.0.0.0/0 route carries a
priority-1100 drop gate on the VPC router. When a route later lands
(NATGW reroute at 900, IGW reroute at 1000) the reroute is installed but
the drop gate was never removed, so the higher-priority gate wins and
black-holes the subnet's egress in lr_in_policy before it reaches routing
or SNAT. This left NAT-gateway private-subnet egress with 100% packet
loss: the SNAT stage saw zero packets and no conntrack was committed.
Remove the subnet's drop gate for the same prefix once a reroute is
installed. Idempotent and prefix-scoped, so a specific-prefix reroute is
a no-op and leaves the default gate intact.
* test(e2e): merge VPC subnet, NAT gateway, and EIP single-node tests
Merges the former TestVPCSubnetE2E, TestNewVPCEgressBaseline,
TestNATGateway, and TestInstanceEIP into one staged scenario,
TestVPCEgressPaths, around one scenario-owned VPC and its two guests,
cutting five guest boots to three.
The public guest serves PublicSubnetEgress, then the NAT Gateway
bastion hop, then (last, since association rewrites its public IP)
the EIP flip. The private guest serves the NAT Gateway path, launched
directly into the same VPC's already-existing private subnet instead
of a dedicated one. RouteBeforeSubnet keeps its own second VPC and
instance since it exercises the implicit main-route-table code path,
a different one than the explicit route table this scenario's own VPC
uses for its public subnet — collapsing the two would drop coverage of
whichever path got merged away.
The NAT Gateway up/down flip now repeats for an env-tunable round
count (SPINIFEX_VPCEGRESS_NAT_ROUNDS, default 1), matching every other
merged single-node scenario.
* test: TEMPORARY mutation-check — revert a795c2b2 drop-gate clear
Reverts the RemoveSubnetEgressDrop call in ensureSubnetEgressAtPriority
to confirm TestVPCEgressPaths/Round1/NATGatewayUp actually detects the
regression it guards against. This commit will be reverted immediately
after the live run confirms a failure.
* test: TEMPORARY mutation-check v2 — neutralize RemoveSubnetEgressDrop
Restores the real a795c2b2 fix in ensureSubnetEgressAtPriority (the
first mutation attempt only reverted this call site and was masked by
the independent gate-recompute path in routetable service_impl.go's
CreateRoute, which also clears the drop gate on every 0.0.0.0/0 route
add and predates a795c2b2). Neutralizing RemoveSubnetEgressDrop itself
disables gate-clearing regardless of caller, reproducing the literal
regression. Will be reverted immediately after live verification.
* Revert "test: TEMPORARY mutation-check v2 — neutralize RemoveSubnetEgressDrop"
Mutation-check confirmed: neutralizing RemoveSubnetEgressDrop reproduced
a795c2b2's regression exactly (stale priority-1100 drop gate outranking
the priority-1000 IGW reroute, confirmed via OVN flow dump; PublicSubnetEgress
failed with an SSH handshake timeout to the public instance). Restoring
the real fix; igw.go is now byte-identical to a795c2b2.
* fix(e2e): stop pruning shared multinode SSH key pem mid-run
The multinode package fixture cached Fixture.Artifacts once, computed
from harness.ArtifactDir(t, env) against whichever top-level Test*
happened to win the pkgFixOnce race. That helper both names the
directory after that one test and registers a t.Cleanup that removes
it once that test passes. needKeyPair wrote the shared EC2 key pair's
pem into that same directory, so as soon as the winning test finished,
its own artifact-dir cleanup deleted the pem every later test in the
package still needed to SSH into fresh guests, even though the
underlying EC2 key pair itself was untouched (its DeleteKeyPair
cleanup already correctly drains at process Close() in TestMain).
Fix: give Fixture an ArtifactDir(t) method — the same per-call pattern
already used by the gpu/single/storagegrowth/iam fixtures — so each
diagnostic dump site recomputes its own dir against the CURRENT test.
The shared key pair pem now lives under Fixture.TmpDir, which is
genuinely package-scoped and only removed once in TestMain after
m.Run() completes.
* refactor(e2e): parallelize LB e2e scenarios on multinode clusters
Each of the 8 TestLoadBalancer scenarios already stands up its own
dedicated LB + target group (ModifyListener/ListenerRules mutate their
LB's listener in place, so it can never be shared). The bottleneck was
running them serially: each teardown made createActiveLB back off
waiting for the previous LB's sys.micro slot to be reclaimed before the
next one could stand up.
On multinode clusters (peer != "", i.e. the 3-node cluster this suite
targets) there's spare capacity to stand up all 8 dedicated LBs at once,
so scenarios now run as t.Parallel() subtests instead of serial
create/teardown/backoff. Since each scenario owns an independent LB,
one never reaching active only fails that scenario — it no longer
cascades into five sequential 5-minute timeouts, so the serial fail-fast
gate is skipped in this mode; concurrency itself bounds the worst case
to a single timeout window.
Single-node dev boxes (no peer) keep the original serial path,
including the fail-fast gate, since they lack the spare capacity for
concurrent LB standup. No subtest body, assertion, or call signature
changed — only the dispatch/gating logic in TestLoadBalancer.
* fix(ci): gate e2e jobs on the suites they actually run
* fix(e2e): serialize internet-facing LB subtests among themselves
Live multinode verification showed InternetFacing_ALB and
InternetFacing_NLB running concurrently each clobber the other's
public-LB DNS record mid-registration, so neither ever resolves within
the 90s wait. The old serial suite never had two internet-facing LBs
alive at once, so this never surfaced.
Drop t.Parallel() from the three internet-facing subtests so their
register+resolve windows never overlap each other, while leaving the
internal subtests parallel as before. They may still run alongside the
internal group, which resolves internal (not public) DNS and isn't
affected.
* fix(e2e): give each ALB LB scenario its own security group
Concurrent internal subtests all created ALBs with no explicit
SecurityGroups, so each ALB's ENI fell back to the one shared VPC
default SG. One scenario authorizing/reconfiguring its own listener
racing another mutating the same SG made Internal_ALB_ModifyListener
intermittently fail to serve on the modified port under parallel load.
Each ALB scenario now creates a dedicated SG (createScenarioSG),
authorized only for the ports that scenario's listener(s) use, and
passes it to CreateLoadBalancer. The shared app-instance/probe-client
SG is untouched, since SG ingress is enforced at the destination ENI
regardless of the source ENI's SG. NLBs are unaffected: they already
get an auto-managed per-LB SG when none is supplied.
* test(e2e): annotate intentional create-path calls for manifest-lint
The vpcegress and per-scenario LB security-group tests create raw AWS
resources on purpose — the egress datapath, the route-before-subnet
ordering, and per-scenario SG isolation are the subjects under test, so
they cannot route through a shared Ensure* fixture. Mark each with
// e2e:allow-create and slim the baseline of entries for the removed
single-node network tests.
* test(e2e): restore live spot terminate-close and cancel-running coverage
The integration-tier spot test calls CloseForInstance directly and infers
cancel-keeps-running from the SIR's InstanceId, because DaemonLite has no
vm.Manager to drive the real TerminateInstances -> daemon teardown cleaner
-> RemoveFromSpotRequest -> CloseForInstance chain or to observe a real
VM's running state. Add a thin live e2e test that drives both terminal
transitions through a real backing VM: cancel one spot-backed instance and
observe it stays running, terminate the other and observe its SIR reach
closed through the genuine async delegation.
* fix(network): refresh orphan-EIP prune liveness to spare mid-pass launches
pruneOrphanEIPs listed OVN NAT rows live but matched them against the intent
snapshot captured at the start of the reconcile pass. Because the apply phase
can block for tens of seconds on guest-port datapath waits, a guest launched
during the pass had a live dnat_and_snat row absent from the stale snapshot, so
the prune swept it and blackholed the guest's auto-assigned public IP until the
next pass re-added it — an intermittent multinode SSH-dark flake.
Add an optional FreshIntent re-read callback (wired in vpcd to LoadIntentFromKV)
and union its ports into the live set before pruning, so a mid-pass launch
counts as live. Skip the prune entirely when the re-read fails rather than risk
a false sweep against a snapshot known to be stale. Self-healing and idempotent:
a genuine orphan absent from both snapshot and fresh read is still pruned next
drift. Single-node behaviour is unchanged (callback nil in tests).
* feat: plumb GCEnabled config through to every VB construction site
viperblock chunk GC is gated behind VB.GCEnabled, but spinifex never
set it, so every deployed node ran with GC off by construction
regardless of which viperblock binary shipped. Add a
[viperblock] gc_enabled config key (default false, mirrors shardwal)
and thread it through to the three places spinifex builds a VB: the
nbdkit plugin it spawns per mounted volume (via a new gc_enabled
plugin param), viperblockd's detached-volume opens, and the volume
service's CreateVolume.
* chore: repin viperblock to GC-enable lineage
The gc-enable plumbing references viperblock.VB.GCEnabled, which is
absent from the previously pinned v1.12.1. Repin to the feat/nbd-gc-enable
commit so a standalone (GOWORK=off) build — the path the ansible deploy
uses — resolves the field, and spx plus the nbdkit plugin build from the
same GC-bearing viperblock lineage.
* fix(storagegrowth): stop launching a nano for image-pull repro
ensureImagePullInstance shared harness.DiscoverNanoInstanceType with the
synthetic npass suite. A 512MB guest OOM-thrashes containerd before any
disk-driven ENOSPC and conflates the storage leak with guest-memory
pressure. Add a package-local imagePullInstanceType helper: an exact
override via SPINIFEX_STORAGEGROWTH_INSTANCE_TYPE, otherwise the smallest
advertised type at or above SPINIFEX_STORAGEGROWTH_MIN_MEM_MIB (default
16384 MiB). npass keeps using the nano selector unchanged.
* feat: report guest block write errors so backend ENOSPC reaches the guest
Set werror=report and rerror=report on the guest boot drive so a viperblock
ENOSPC (or any other backend write failure) is surfaced to the guest as a
real I/O error instead of QEMU silently pausing the VM. Without this the
guest kernel never learns its disk is out of space, so it can't fail
writes cleanly or let an operator react.
* chore: repin predastore and viperblock to the storage-capacity-safety lineage
Drop the local-path replace for predastore (a test shortcut) and pin both
predastore and viperblock to their feat/storage-capacity-safety commits, so
a standalone (GOWORK=off) build resolves the free-space watermark, ENOSPC
classification, and backpressure plumbing that spinifex's werror/rerror
change depends on.
* chore: bump viperblock to 054a44a3f0f5
Realigns spx's linked viperblock with the submodule working tree, which the
build role's source-drift gate compares against.
* chore: repin predastore and viperblock to their dev lineage
Both were pinned to pseudo-versions off feat/storage-capacity-safety. That work
has now landed on dev in each repo, and both PRs were squash-merged, so the
branch commits those pins named are unreachable from any branch.
Repin to the dev commits carrying the same code: predastore 5699329 and
viperblock 35dd9f2.
Northstar carried working northstar.dns.* instruments but shipped no
telemetry, so its dashboard was empty and the only northstar data in
Elasticsearch was journald scraped under the systemd unit name.
Two gaps, both outside the instrumentation. northstarStartCmd never called
initTelemetry, unlike every other service command, so the instruments
recorded into a no-op provider. The unit also did not source
/etc/spinifex/telemetry.env, which every other application unit reads for
OTEL_EXPORTER_OTLP_ENDPOINT, MULGA_ENV and MULGA_SOURCE — so even an
initialised SDK would have had no endpoint and no env label.
Add an invariant covering the second gap: any unit launching an spx service
must source the telemetry drop-in. A service missing it starts healthy and
silently reports nothing, which only surfaces when someone notices an empty
dashboard. Verified the test fails against the unpatched unit.
Add e2e_es_summary, a self-hosted job that aggregates suite-status.txt
from both e2e legs and POSTs one doc per run to the Elastic sink, keyed
by labels.ci_run_id, so Kibana can answer "did run X pass, and where
are its artifacts" without hand-aggregating traces-apm*/logs-apm.error*.
Runs on ci-single (LAN route to the sink) since e2e_status stays on
GitHub-hosted ubuntu-26.04 for the PR status-report check. State is
computed with the same single_ok/multi_ok logic as e2e_status so the
two can never disagree.
* feat(bedrock): Ochre streaming — ConverseStream + InvokeModelWithResponseStream
Adds the two streaming bedrock-runtime operations behind a shared
eventstream.Encoder frame writer: ConverseStream reframes vLLM/Anthropic SSE
into the normalized Bedrock event taxonomy (messageStart -> contentBlock*
-> messageStop -> metadata), InvokeModelWithResponseStream wraps each
provider-native chunk in a PayloadPart frame. Fixes otelsetup's
statusRecorder to implement Flusher/Unwrap so flushes actually reach the
socket through the middleware chain, which streaming depends on.
* fix(bedrock): keep vLLM ConverseStream taxonomy ordered on edge cases
Route contentBlockStop/messageStop through an idempotent ensureStarted so a
usage-only chunk arriving before any finish_reason no longer emits metadata
ahead of messageStop, and a zero-content stream no longer emits stop events
without a preceding messageStart. Regression tests for both.
* fix(bedrock): route /model/ data-plane calls to bedrock-runtime
AWS SDKs sign bedrock-runtime with the SigV4 signing name "bedrock", the
same as the control plane; the two services are separated only by endpoint
hostname. The gateway serves one endpoint and routed purely on the signing
name, so every /model/{id}/converse*|invoke* call landed in Bedrock_Request
(control plane) and returned InvalidAction — no standard SDK/CLI could reach
any bedrock-runtime endpoint.
Disambiguate by path in GetService: a "bedrock"-scoped request to a /model/
path (data-plane-exclusive) resolves to bedrock-runtime. Control-plane paths
(/foundation-models, /custom-models, /model-invocation-job) are unaffected.
* test(bedrock): satisfy testifylint go-require/empty in streaming tests
Use assert (guarded where a value is dereferenced afterward) instead of
require inside httptest handler closures, and assert.Empty for empty-string
checks, per dev's stricter testifylint config.
The Build/Lint/Race jobs checked out viperblock and predastore at a hardcoded
ref: dev and overlaid them with a go workspace, so the go.mod pins were ignored
and a spinifex change depending on an unmerged sibling change could never
compile in CI. Resolve each sibling to a branch of the same name when it exists
on that repo, else dev, mirroring the e2e path in
scripts/tofu-cluster/build-install-artifacts.sh so unit CI and e2e CI agree.
Treat the ECR registry JWT as a signed identity pointer rather than a
permission grant. Every /v2/* request now rehydrates the token's access
key against current IAM/STS state, maps the OCI route to an ecr: action
and repository ARN, and runs the identity-policy evaluator before
dispatch. PullOnly/ReadOnly principals can no longer push, overwrite
tags, or delete manifests, and key/session/user/role/account/policy
revocation takes effect on the next request instead of at token expiry.
Cross-repository blob mounts now require source-read permission and
proof the digest is referenced by a manifest in the named source repo,
closing an account-wide blob-fishing gap.
A CreateLoadBalancerSync whose data-plane launch failed returned without
rolling back the NLB managed security group. The orphaned SG sat in the EKS
managed control-plane VPC and blocked DeleteVpc with DependencyViolation
indefinitely, wedging the cluster in DELETING. The failure was also flattened
to ServerInternal, discarding the only useful diagnostic.
The sync path now unwinds fully on launch failure (ENIs, managed SG, name
claim, record) and maps the launch error onto its real cause: capacity
shortfalls pass through as their own code, anything else stays ServerInternal.
The async path deliberately keeps its failed record — the caller holds the ARN
and can reclaim it via DeleteLoadBalancer.
The image-pull storage-growth workload HEADed <volID>/checkpoints/blocks.live.bin
with the operator box's gateway identity, which has no S3 read grant on the
bucket and was refused 403 on every sample, so checkpoint_bytes read 0 across a
whole run. It now HEADs the object on the node over PeerSSH using predastore's
own [[auth]] credentials from predastore.toml (root-owned, so read via
passwordless sudo; the CA is readable through the spinifex group). The
credentials are read into shell locals, used inline for one aws s3api
head-object, then unset — never echoed, never leaving the node. A probe failure
that is not a legitimate 404 voids the sample (checkpoint_bytes -1 plus a probe
error) rather than reporting a false zero, so a dead instrument can never look
like no growth. The dead operator-side S3 client path is removed.
Align the top of the README with mulgadc.com: new headline and intro,
rewritten "What is Spinifex?", a new "Three ways to deploy" section
(Neocloud / on-premise / edge) and a new "AWS compatibility" service
matrix (EC2, EBS, S3, VPC, IAM, ALB/NLB, EKS, ECR, ECS available; RDS
Q3 2026) linking the GitHub releases. Refresh the architecture prose and
Key Features, add Deploy and AWS-services nav links, and swap banner.svg
for the ember-on-ink mini-stack design matching platform.svg.
Rebuild the message-driven architecture diagram in the ember-on-ink
theme to match platform.svg: ember gradient NATS bus, ember-tinted
gateway and daemon cards, neutral clients and backend cards. Content
unchanged (clients → gateway → NATS → ec2/ebs/s3/vpc/iam daemons →
local backends). De-slop the copy and refresh the img alt text.
Replace the platform diagram with a static SVG matching the mulgadc.com
front-page stack diagram: ember accent, ink panel, the portable-stack
box (Neocloud / on-prem / edge) wrapping the Spinifex AWS-compatible
service layer, over standard Linux on CPU/GPU hardware. Self-contained
and GitHub-safe (presentation attributes only). Refresh the img alt text.
* fix(admin): round the AMI root volume up so the image fits
`spx admin images import` sized the root volume with integer division, which
floors: a 4.94 GiB image registered a 4 GiB volume. The image is then copied
into a volume too small to hold it, is truncated, and the guest boots with no
root partition — it stalls on the root device until systemd gives up, and
nothing anywhere reports an undersized volume.
floorVolumeSizeToAMI already exists to prevent exactly this ("Under-sizing
hangs the guest in dracut"), but it raises the caller's requested size to the
AMI's recorded VolumeSizeGiB — the same floored value — so it inherited the
mistake instead of catching it.
This stayed hidden because every current system image is an exact multiple of
1 GiB, where flooring and rounding up agree. It surfaced on a 5306470400-byte
image, which would not boot.
Round up instead, via a helper so the boundaries are testable: an exact
multiple must not gain a spare GiB, and one byte over must.
* fix(eks): default a nodegroup's root volume to 20 GiB, as AWS does
CreateNodegroup stored diskSize verbatim, so omitting it persisted 0 and
the launch skipped its block device mapping entirely, dropping the worker
onto whatever volume the AMI itself declares.
That was harmless while the node AMIs baked generous virtual disks — the
GPU image declares 16 GiB for ~2 GiB of content, so the fall-through
happened to leave plenty of room. It stops being harmless as soon as an
image is sized to its contents: the worker then boots onto a volume with
barely a spare GiB and cannot pull the container images it exists to run.
Sizing the node is the nodegroup's decision either way, not a property
the image should be able to dictate by accident.
20 GiB matches the AWS EKS managed-nodegroup default. It is resolved at
create rather than at launch so DescribeNodegroup reports the size the
node actually gets.
TestCreateNodegroup_NoDiskSizeOmitsBlockDeviceMapping asserted the old
fall-through and is replaced by its inverse; the explicit-size case is
already covered by TestCreateNodegroup_DiskSizePropagatesToBlockDeviceMapping.
* fix(ecs): size a container instance's root volume to 30 GiB by default
ProvisionCapacity built its RunInstances input without a block device
mapping, so a container instance inherited whatever volume its AMI
declares. EC2 has no default of its own to fall back on — the AMI is the
only thing that decides — and a node AMI is sized to hold the image, which
leaves the agent no room for the container images it exists to pull.
30 GiB mirrors what the AWS ECS-optimized AMI declares. ProvisionCapacity
is spinifex's own API rather than an AWS one, so DiskSize is added to its
input for callers that want to size a cluster explicitly, defaulting when
omitted the same way InstanceType and Count already do.
* fix(images): pull the Ubuntu catalog images from releases, not a pinned daily
Both Ubuntu catalog entries pointed at a dated build under the codename
dailies tree, which Canonical prunes as builds age out. That path now 404s,
so "spx admin images import --name ubuntu-26.04-x86_64" fails outright for
x86_64 and arm64 alike — the entries had rotted with nothing to catch it.
Release builds are retained rather than pruned, so the pin stays resolvable
while keeping a specific, reproducible build. The filenames under releases/
carry the version instead of the codename, which matters because the sums
entry is matched on the URL's basename.
CI is unaffected and could not have caught this: it fetches resolute/current/,
a symlink that never rots, so nothing exercised the pinned path.
Debian pins a dated directory too but is fine — those builds are retained,
and its entries still resolve.
Pulls a container image inside a guest onto a viperblock volume and samples
guest used bytes, host backing-store bytes, nbdkit RSS and the live
checkpoint size as it goes, so backing-store cost can be fitted against the
bytes the guest actually wrote.
Complements the N-pass overwrite workload, which rewrites one extent and so
pins the block map size — that shape cannot show whether cost grows in
proportion to bytes written or faster than them.
Redirects both dockerd's data-root and containerd's root onto the target
volume. Redirecting data-root alone leaves containerd extracting layers into
/var/lib/containerd on the AMI's small root disk, which fails with ENOSPC
while the target volume sits empty.
The storagegrowth gate called ResolveCACert unconditionally before
building the AWS client, even though newAWSClient itself skips CA
resolution entirely when SPINIFEX_AWS_INSECURE=1. That blocked the
documented runner-resident remote path (env12 and similar clusters
with no local CA cert on disk) and the skip message falsely claimed
no node was running when the real cause was a missing CA. Skip the
CA check under SPINIFEX_AWS_INSECURE=1 and state the actual cause
plus the escape hatch in the skip message.
Rewrites one logical extent N times through full attach/format-write/detach
cycles, so the guest-visible footprint stays constant while total bytes written
scales with N. That shape isolates the leak where viperblock mints chunk objects
under unique monotonic keys and never deletes the superseded ones: a sequential
write-once supersedes nothing and so cannot exercise it.
The suite drives the workload only and deliberately does not measure the
backend. Measuring on-disk bytes from here would mean either a host du (which
cannot separate live from dead bytes, and so cannot distinguish this leak from
the unrelated one where an overwritten key's old extent is orphaned without a
tombstone) or a dependency on tooling outside this repository, which would stop
the suite building standalone. It instead reports what it did to a caller-named
result file and asserts only what a guest can observe: a sha256 round-trip
through a fresh mount per pass, and a footprint that stays constant across them.
The volume is retained on request, because deleting it removes the evidence:
DeleteVolume enumerates the bucket by prefix and drops every key under it,
including the leaked chunks.
Phase 8 of TestNATUplink checked the second VPC router's lr-nat-list with a
single non-polling assert; the igw-default-snat row commits a beat after the
gateway LRP appears, so the read raced the OVN commit and intermittently saw
an empty list (flaky CI red). Wrap it in EventuallyErr, matching the sibling
LRP and host-route polls in the same phase.
AddSNAT logged only on the idempotent-skip path, so the first install was
silent and RCA could not tell a late SNAT from a missing one. Emit an install
log keyed by router + vpc_cidr + external_ip.
Server-cert IP SANs were built solely from live interface enumeration
(DiscoverLocalIPs) plus bindIP, so the canonical mgmt-bridge IP entered
the SAN set only when br-mgmt happened to be up at cert-generation time.
On a host where br-mgmt is down when the cert is minted, 10.15.8.1 was
silently dropped and the EKS control-plane publish to https://10.15.8.1
failed cert verify, leaving the cluster stuck CREATING.
Pin config.DefaultMgmtBridgeIP into the extra SANs for both the init and
join cert paths so publish succeeds regardless of interface state at
generation time. Add a regression test asserting the SAN is present when
br-mgmt is not a live interface.
Add an IRSAPod subtest that applies a real pod wired for IRSA — a projected
ServiceAccount token (aud sts.amazonaws.com) plus the AWS_ROLE_ARN /
AWS_WEB_IDENTITY_TOKEN_FILE / AWS_ENDPOINT_URL_STS / AWS_CA_BUNDLE env the
addon-sync agent injects — and asserts the pod's in-cluster
aws sts get-caller-identity resolves to the assumed IRSA role. This closes the
gap left by IRSAWebIdentity, which only exchanges the token host-side.
Extract registerOIDCRole (shared with IRSAWebIdentity); the pod runs on a
1-node worker nodegroup (the control-plane node is CriticalAddonsOnly-tainted)
and reaches awsgw STS over the worker subnet NAT.
The build script derives BUILD_DIR from SYSTEM_IMAGE_BUILD_DIR (the AMI
publish workflow points it off tmpfs so the 16G Ubuntu GPU raw fits), but
the publish step hardcoded /tmp and could not find the freshly built raw.
GPU legs failed 'Raw image not found'; Alpine legs silently reused stale
pre-existing /tmp raws on the persistent runner. Mirror the build script's
derivation so publish looks where the build actually wrote.
The publish-eks-ami matrix fans out four legs on the single persistent
ci-single runner, all reusing one workspace root. go work init is
create-only and checkout never scrubs the root go.work, so legs after the
first fail with 'go.work already exists'. Remove any stale go.work before
init so each leg re-inits cleanly.
Expand the publish workflow to build the two Ubuntu NVIDIA GPU variants
alongside the Alpine eks-node/ecs-agent AMIs. The image build+publish
scripts are already manifest-driven and Ubuntu-aware; only the workflow
lacked the GPU legs.
Refactor the single-job body into a matrix over the four image manifests,
resolved by a new 'variant' input (all|cpu|gpu) so the slow 16G GPU legs
can be skipped when only the CPU AMIs need republishing. GPU images do not
fit a RAM-backed /tmp, so set SYSTEM_IMAGE_BUILD_DIR onto the runner's
on-disk _temp; legs run serially (max-parallel 1) on the single ci runner.
The nodegroup label was only written to K3S_NODE_LABEL in agent.env, but
k3s has no such env var, so non-GPU workers never received the label.
waitWorkersReady buckets Ready nodes by that label (added in #511), so
the affected nodegroups never counted toward their ACTIVE transition and
stayed CREATING until timeout.
Ride the label on a real k3s node-label config drop-in
(15-nodegroup-label.yaml), delivered via cloud-init user-data. Drop the
inert K3S_NODE_LABEL from agent.env and the initd export.
WaitForInstanceState now aborts immediately with the recorded StateReason
message when it observes a reaped state (shutting-down/terminated) while
waiting for a live target, instead of burning the full 5-minute timeout on
an instance that can never reach it.
MarkFailed takes a context and records the failure on the active span via
span.RecordError so a reaped launch surfaces in the APM error stream,
correlated by trace id. Only the LaunchRunInstances path carries a live
span; other callers pass context.Background() and the record is a no-op.
Surfaced by RCA of E2E run 29258398516 (TestAccountScoping/Step2).
TestNATGateway flaked with 100% egress loss: the tenant VPC gateway router's
OVN MAC_Binding had no entry for its upstream default-route nexthop. OVN
resolves that nexthop via lazy dynamic ARP on first egress; if that ARP is
lost during gateway bring-up, SNAT'd packets die in lr_in_arp_resolve and no
reply returns. MAC_Binding is per (datapath, logical_port, ip), so a sibling
VPC router resolving the same nexthop does not help.
Deterministically seed a static MAC binding for the nexthop at IGW attach. The
nexthop MAC is the host's own default-gateway MAC (the host shares the physical
uplink), read from ip neigh and written via ovn-nbctl static-mac-binding-add
(northd programs it into SB MAC_Binding, GC-safe). Host layer L0 owns the
shell-out per ADR-0006 S2; igw calls it via an injected hook. Best-effort and
idempotent: a missing MAC leaves dynamic ARP as the fallback, and it re-runs on
every AttachIGW so a binding lost to a later flap self-heals.
utils.GetOrCreateKVBucket hardcoded Replicas=1, so the ~38 handlers that
create their KV buckets lazily (VPC/subnet/ENI/EIP/ELBv2/ECS/ECR/ACM/
natgw/snapshot/placementgroup/IPAM/account-settings/dhcp/static-pool …)
created them at R1 on a multi-node cluster. upgradeJetStreamReplicas runs
once at startup, so any bucket created lazily after boot stayed R1
permanently — and an R1 stream's single Raft replica is transiently
leaderless during formation/join/rebalance, surfacing as
ErrNoResponders/ErrTimeout on KV ops.
Add a package-level default replica count that the daemon sets once at
boot (startCluster, after initJetStream) to len(clusterConfig.Nodes),
before any handler creates a bucket. GetOrCreateKVBucket now creates at
that default, so lazily-created buckets are born quorate. The one-shot
startup UpdateReplicas remains as a safety net for pre-existing R1
buckets from older deploys.
Zero call-site churn: the default flows through the existing
GetOrCreateKVBucketWithReplicas path already used by eks/ecs/iam.
TestSpawnRegisteredReconcilers_ResumesNonTerminal flaked two ways. First,
the shared fixture meta stamps a weeks-old CreatedAt, so a resumed CREATING
reconciler instantly trips the CREATE timeout, marks the cluster FAILED and
deregisters before the registry.Has assertion runs — the test was racing a
self-removal. Stamp a recent CreatedAt so the reconciler stays CREATING.
Second, SpawnRegisteredReconcilers ran a one-shot JetStream enumeration
(KeyValueStoreNames + kv.Keys), which is eventually consistent: under load a
just-committed cluster can be missing and never resumed, silently on a real
daemon. Re-scan until the observed CREATING/ACTIVE count stops growing, capped
and short-backed-off; Spawn is idempotent so re-runs only pick up stragglers.
discoverSingleNodeIPs seeds 127.0.0.1 then appends net.InterfaceAddrs()
in undefined order, so WANHost=nodeIPs[0] was loopback and the guest
gateway URL helpers fell through to the first non-loopback interface
addr — an arbitrary co-tenant bridge on multi-bridge shared hosts
(wattle: br-lan-env1..23, br-wan), leaving guest VMs unable to reach the
advertised gateway to register.
Resolve WANHost from spinifex.toml nodes.<node>.advertise (falling back
to that node's host), mirroring awsgwBindIP's toml scan; fall back to the
first non-loopback discovered addr only when config is unreadable. This
removes the need for the SPINIFEX_WAN_IP override on shared hosts.
The crash-supervisor auto-restart path (qmpHeartbeat -> HandleCrash ->
MaybeRestart -> startQEMU -> newQMPClientWithHandshake -> dialQMPWithRetry)
already unlinks the stale QMP socket and retries a refused/absent connect. But
a freshly relaunched QEMU can accept the connect then reset the connection
mid-greeting while it initialises, surfacing 'connection reset by peer'
immediately after 'QEMU started successfully' as a ServerInternal.
Add ECONNRESET and EPIPE to isTransientDialError so the dial is retried until
QEMU services the monitor, closing the auto-restart QMP-connect race.
Add non-streaming InvokeModel to bedrock-runtime. Unlike Converse's unified
schema, InvokeModel passes model-provider-native bodies through, so route by
model family: Anthropic Claude bodies forward to the Messages API (drop the
Bedrock-only anthropic_version, inject model, return verbatim); Meta Llama
bodies translate to/from the self-host vLLM OpenAI completions endpoint.
Raw-body dispatch bypasses the JSON response path (WriteRawResponse). Reuses
the Phase 1 credential + endpoint resolvers and upstream-status mapping.
Unit-tested (routing, body rewrite, translation, error classes) plus gated
InvokeModel E2E.
Both services already run otelsetup.Init and ship logs/metrics but never
opened a span, so they were invisible in the Kibana APM service map.
qmp-collector now opens a root span per per-instance poll tick (marks
error on QMP collect/publish failure); spinifex-ui now wraps its HTTP
handler chain with otelsetup.HTTPMiddleware and its reverse-proxy
transport with otelhttp, so UI requests get a server span and proxied
calls to awsgw/predastore carry traceparent instead of stopping the
trace at the UI hop.
nats and eks are intentionally left uninstrumented in this pass: nats is
the vendored message-broker process with no message-level hook to attach
spans to (the traffic it carries is already traced end-to-end at its
producer and consumer endpoints); eks request/response calls already
produce spans today under service.name=spinifex-daemon since the
handlers run in that process, and its async reconciler/state-machine
goroutines need a separate span design across a much larger surface.
Skip ERROR log when a subscription is already invalid
(nats.ErrBadSubscription) during graceful shutdown; log at debug
instead. Cosmetic cleanup race, no functional impact.
golangci-lint canonicalheader flagged 6 lowercase header literals in the
Ochre/Bedrock gateway (anthropic.go, anthropic_test.go, vllm.go). Canonical
form is wire-equivalent; canonicalise to X-Api-Key, Anthropic-Version and
Content-Type to restore CI lint.
* feat(bedrock): Ochre runtime MVP — Converse over self-host + Anthropic-direct
Add the AWS-Bedrock-compatible gateway services (bedrock, bedrock-runtime)
following the EKS REST-JSON pattern: ListFoundationModels / GetFoundationModel
over a static connectivity-tiered catalog, and non-streaming Converse routed
per modelId to one of two provider backends — self-hosted vLLM (OpenAI wire)
and Anthropic-direct (Anthropic Messages wire).
Per-account provider API keys are a master-key-encrypted JetStream KV secret
with an optional platform default; the advertised catalog gates external
models on a resolvable credential. Self-host endpoints are pinned via config
(OCHRE_VLLM_ENDPOINTS) behind an EndpointResolver seam.
Adds gateway/bedrock/ (catalog, provider router, Anthropic + vLLM adapters,
credential store), awserrors REST-JSON codes, and the bedrock E2E suite
(list/get light; Converse gated on live backends). Daemon/NATS control plane
and dynamic endpoint lifecycle are deferred to Phase 3.
* test(bedrock): cover dispatch, router, credentials KV + endpoint config
Add unit coverage for the Ochre Phase 1 diff: Bedrock_Request /
BedrockRuntime_Request end-to-end dispatch, provider router, JSON
response/error writers, upstream-status mapping, the credential-store KV
round-trip, the Anthropic bound-provider path, and parseBedrockEndpoints.
Lifts PR diff coverage above the 60% gate.
* ci: GPU e2e workflow on ci-gpu self-hosted runner
Runs tests/e2e/gpu (EC2/ECS/EKS GPU passthrough) against the persistent,
hand-bootstrapped wattle shared host, driven over SSH by a banksia runner
labelled ci-gpu (matches the ci-single/ci-multi convention) — no runner
installed on the shared box.
- runs-on [self-hosted, ci-gpu]; concurrency ci-gpu, cancel-in-progress
false (single A1000 = single-flight, queue not stomp)
- workflow_dispatch + GPU-path filter (not every push)
- manual-work mutex gate (~/.ci-gpu-paused / CI_GPU_PAUSED) so hands-on GPU
testing blocks CI
- if: always() cleanup terminates launched GPU instances; never resets the
box (no setup-ovn.sh/dev-reset/cluster-reset)
- drives the control plane with spx + the already-configured aws spinifex
profile directly (no wrapper scripts, nothing to install on wattle)
* ci(gpu-e2e): advisory AMI preflight, fail only if zero GPU AMIs
The suite is self-guarding — each test skips when its AMI is absent. The
hard-fail on the base ubuntu GPU AMI blocked the whole run even when the
ECS GPU AMI (the only one imported) could execute. Inventory the AMIs and
fail only when none are present.
* ci(gpu-e2e): self-provision base GPU AMI, hard-require it in preflight
The GPU E2E workflow was green-because-skipped: with no GPU AMIs imported
the self-guarding suite skipped every test, passing as a false green. Make
the raw-EC2 GPU passthrough test the always-on baseline.
- Add an import-if-absent setup step that pulls the prebuilt base GPU image
from the catalog by name (ubuntu-26.04-nvidia-gpu-x86_64), naming the AMI
exactly what tests/e2e/gpu filters for. Idempotent; caches on wattle's
persistent loop-fs. Runs spx under sg spinifex (local storage write).
- Preflight now hard-requires the base AMI (fail, not skip) so a silent
import failure can't pass as green; ECS/EKS node AMIs stay opt-in warn.
- Reconcile the local build path: ubuntu-gpu-nvidia.conf AMI_NAME and
launch-gpu.sh's nvidia default now use the same canonical name, so a
locally built+imported image also satisfies the suite.
* ci(gpu-e2e): pass --ami-name on base GPU import so the AMI name matches
spx admin images import --name selects the catalog entry but names the
resulting AMI generically (ami-<distro>-<version>-<arch>), so the import
landed as ami-ubuntu-26.04-x86_64 and the tests/e2e/gpu name filter never
matched. Pass --ami-name ubuntu-26.04-nvidia-gpu-x86_64 explicitly (same
flag build-system-image.sh uses) and poll briefly for availability.
* fix(network,vm): reconcile auto-assigned public-IP NAT + scale VFIO QMP timeout
Auto-assigned public IPs were never carried through the network reconciler.
applyEIPs ran AddEIP + ensureGuestPortDatapath for user EIPs only; auto-assigned
IPs (intent.Ports[].PublicIP) got egress exemption but no NAT re-assert and no
gatewayLRP->guest convergence. Their dnat_and_snat was created once at
RunInstances and never touched, so a vpcd/OVN rebuild dropped it and the
guest-port hop never converged: inbound DNATs at the gateway but never reaches
the guest (TCP/SSH dark) while ICMP is answered by the gateway LR (ping OK).
floatingIPSpecs now folds every auto-assigned/ELB public IP into the same
AddEIP + guest-port convergence path as user EIPs (deduped by private IP; a user
EIP wins), making the datapath self-healing across reconcile ticks and deploys.
qmpGreetingTimeout for VFIO guests now scales with guest RAM (base 30s + 8s/GiB,
floored at 180s, capped at 600s) instead of a flat 180s, so a large guest is not
SIGKILLed mid-pin while the whole guest RAM is IOMMU-locked before QMP answers.
* fix(vm): scale QMP handshake command deadline for VFIO launches
The greeting timeout scaling alone did not fix the VFIO launch SIGKILL: a GPU
guest sends its QMP greeting before the RAM pin completes, then cannot service
qmp_capabilities until the pin finishes. sendQMPCommand hardcoded a 30s response
deadline, so the handshake decode timed out (~43s for a 16 GiB g5.xlarge) and
the launch cleanup SIGKILLed a QEMU that would have come up.
Parameterize the command deadline (sendQMPCommandWithTimeout); the handshake now
uses the RAM-scaled qmpGreetingTimeout (base 30s + 8s/GiB, floor 180s, cap 600s)
for qmp_capabilities. Other callers keep the 30s default.
* test(gpu-e2e): share one GPU instance across passthrough subtests to avoid mid-suite IP recycle
Recycling both the floating and private IP onto a new ENI left the ingress
datapath targeting the terminated instance, so the successor was unreachable
on its EIP while IMDS stayed up.
- AddEIP re-points the dnat_and_snat row when the owning ENI changed even
though the external+logical IP are unchanged, keyed on the portable
spinifex:logical_port external-id (centralised mode leaves the native
LogicalPort column empty). The genuine same-ENI case still skips row churn.
- Flush the SB MAC_Binding for a private IP on ENI teardown and successor
bind so a reused address re-resolves to the live owner's MAC, via a new
MACBindingFlusher backed by ovn-sbctl.
- Reconciler prunes orphan dnat_and_snat rows whose stamped owning ENI is
absent from intent (leaked by lost vpc.delete-nat across dead VPCs).
- IMDS tap responder suspends retries for an ENI whose record is gone,
stopping the 15s reconcile churn.
* test(daemon): cover GPU inventory in GetNodes and GPU attachment in GetVMs
Adds unit tests for buildGPUInventory/handleNodeStatus (whole-GPU claimed/
free, MIG slices claimed/free, no-GPU node) and resolveVMGPU/handleNodeVMs
(whole-GPU attachment, MIG attachment, no attachment). Production code for
these paths (types.GPUInfo/GPUSliceInfo/VMGPUInfo, buildGPUInventory,
resolveVMGPU) and EC2 DescribeInstanceTypes GpuInfo.Gpus population
(instancetypes.GenerateGPUTypes/GenerateMIGTypes/IsGPUType) already landed
via the daemon GPU-admission work and were already test-covered for part C.
* feat(ui): shared GpuInstanceTypeSelect grouped instance-type picker
Reusable Select wrapper for EC2/EKS/ECS instance-type pickers: groups
Standard vs GPU-Accelerated types via structural GpuInfo.Gpus detection
(AWS SDK shape, no name heuristics), annotates GPU items with VRAM from
MemoryInfo.SizeInMiB and an optional per-type availability count, and
falls back to a single ungrouped list when no GPU types are present.
* feat(ui): EC2 GPU inventory, instance GPU card, shared picker in run-instances
* feat(ui): EKS nodegroup GPU picker + AMI/taint note
Add-nodegroup form now uses the shared GpuInstanceTypeSelect, grouping
and VRAM-annotating GPU instance types instead of a free-text field.
Selecting a GPU type surfaces a note that the GPU node AMI is
auto-selected and the nodegroup is tainted
nvidia.com/gpu=present:NoSchedule by default. Nodegroup rows badge GPU
nodegroups, derived from the primary instance type since
NodegroupRecord does not expose a GPU flag over the API.
* feat(ui): ECS GPU capacity picker, task-def GPU count, task gpuIds
* feat(eks): expose nodegroup GPU signal in Describe/ListNodegroups + UI badge
* chore(ui): rebuild embedded console dist for GPU UI
Bump github.com/mulgadc/viperblock to v1.11.1-0.20260713024608-eb4a376becc8
(dev tip through viperblock #34): bounded parallel chunk upload + coalesced
checkpoint (edou7 sec 7 items 2 and 9), sweeping the require up from #32 and
picking up the #33 write backpressure on the way. The nbdkit plugin the VM
write plane uses is built from the submodule source; this bump aligns the
viperblockd library that spx links.
* fix(daemon): gate GPU instance admission on cpu/mem and GPU-slot availability
canAllocateLocked early-returned count for whole-GPU (non-MIG) types,
bypassing the vCPU/memory admission calc and the live MemAvailable gate.
canAllocateCount already had a requiresGPU/availGPU branch, but every
production caller passed requiresGPU=false, making it dead code.
GPU instance types (whole-GPU and MIG) now flow through the same
cpu/mem + live-mem gate as every other type, with GPU-slot count wired
in as an additional hard constraint from gpuManager.Available(). This
also gives MIG a real up-front GPU-slice gate instead of only failing
late at Claim.
* fix(vm): release leaked GPU pool slot on crash-to-error and add reconciler backstop
HandleCrash freed vCPU/memory via deallocateResources but never called
InstanceCleaner.ReleaseGPU, since it bypasses the stop/terminate cleanup
chain that normally carries that call. A successful restart reuses the
still-set GPUAttachments, so that path is untouched, but when restarts
are exhausted, the instance type is unknown, there isn't enough capacity
to restart, or the relaunch itself fails, the instance settles
permanently in StateError still holding its GPU pool entry with nothing
to reclaim it.
Release the GPU and clear GPUAttachments on each of those terminal
paths. Add a GPU-pool reconciler mirroring the existing ENI reconciler,
wired into the daemon's reconcile loop, that periodically frees any pool
entry whose instance is absent or has settled into a non-running state,
as a backstop for a node-down mid-cascade or any other gap in the
synchronous release.
* fix(eks): keep derived ELBv2 names under 32 chars and surface CREATE_FAILED reason
Long cluster names made the derived "eks-<name>[-cp|-konn]" NLB/target-group
names exceed the ELBv2 32-char limit, failing cluster creation asynchronously
with no visible cause. safeELBv2Name now deterministically truncates the name
and appends a stable hash of the full cluster name when needed, so the same
name is derived on every call (create, lookup, delete). DescribeCluster also
now surfaces ClusterMeta.StatusReason as a Health.Issues entry when a cluster
is FAILED, instead of leaving callers to guess why.
* fix(eks): plumb per-nodegroup Ready-node counts through state reports
The on-VM state-report script now emits nodegroup_ready, a per-nodegroup Ready
count keyed by the eks.amazonaws.com/nodegroup label value, alongside the
existing cluster-wide node_count. ServerStateReport, the ClusterReconciler's
observe step, and SetClusterHealthState now carry this map through to
ClusterMeta.NodegroupNodeCounts, so per-nodegroup consumers (wired in the next
commit) can gate on a nodegroup's own workers instead of the cluster total.
* fix(eks): scope nodegroup Ready-gate per nodegroup and retry failed worker launches
waitWorkersReady compared the cluster-wide Ready-node count against a
nodegroup's baseline+desired, so in a multi-nodegroup cluster one nodegroup's
Ready workers could mask another whose own workers never registered. It now
reads meta.NodegroupNodeCounts[ng], scoped to nodes carrying the
eks.amazonaws.com/nodegroup=<name> label.
Separately, launchNodegroupInfra never retried a failed worker launch (a
transient QMP timeout or host capacity pressure), leaving the nodegroup stuck
CREATING with zero workers. The new launchWorkersUntilDesired wrapper relaunches
only the shortfall on failure, logging each retry, until the desired count is
reached or workerLaunchRetryTimeout elapses.
Both fixes are covered by regression tests: a two-nodegroup scenario where only
one has Ready workers, a transient-failure-then-success relaunch, and an
exhausted-retry-budget CREATE_FAILED path. Not covered here: a worker that dies
after the nodegroup is already ACTIVE — that needs an ongoing reconciler, which
this change does not add.
* fix(eks): tolerate GPU NoSchedule taint in konnectivity-agent DaemonSet
A GPU-only worker is tainted nvidia.com/gpu=present:NoSchedule, so the
konnectivity-agent DaemonSet (no tolerations) showed DESIRED=0 there and
kubectl logs/exec against pods on that node failed "No agent available".
Adds an Exists toleration for nvidia.com/gpu:NoSchedule.
Live-cluster validation (DESIRED count, kubectl exec) is deferred — no cluster
available to test against right now.
* fix: tolerate nvidia.com/gpu taint on ebs-csi-node and coredns DaemonSets so GPU workers get storage and DNS
A re-driven DeleteCluster whose managed control-plane VPC was already gone
(EC2 InvalidVpcID.NotFound) kept looping DependencyViolation: the internal
cp-vpc state record (ClusterMeta.ManagedCPVPC) was never cleared unless the
*entire* purge succeeded, so any unrelated flaky step retried CP-VPC/SG API
calls against a stale VpcId forever instead of converging.
purgeClusterInfra now clears ManagedCPVPC (via a new CAS-based
ClearClusterManagedCPVPC) as soon as DeleteClusterCPVPC + its SG cleanup both
succeed, independent of whatever else in that run still fails.
DeleteClusterCPVPC also now republishes vpc.delete for the CP VPC on every
call — including the already-gone fast path, falling back to the persisted
VpcId when the live tag-indexed describe can no longer name it — so vpcd's
topology manager gets another chance to GC an orphaned logical
router/subnets/SubnetEgressPriorityDrop policies if the original delete's
fire-and-forget event was ever lost.
handleRelease cancelled the per-lease renewal goroutine but deleted the
KV entry without waiting for it to exit. An in-flight reaffirm/renew Put
could land after the Delete, re-creating the lease and leaking a KV
record. Add a per-loop done channel and block on it after cancel so any
pending Put completes before the delete, making DrainAll's release path
deterministically synchronous.
* feat(eks): expose NVIDIA GPUs to pods via CDI device plugin
GPU nodegroups now label workers nvidia.com/gpu.present=true and taint them
nvidia.com/gpu=present:NoSchedule (via a k3s config.yaml.d drop-in, since k3s
has no K3S_NODE_TAINT env), and stage a hidden nvidia-device-plugin addon that
addon-sync renders into the K3s auto-deploy dir. The device plugin runs with
--device-list-strategy=cdi-cri and is gated to GPU nodes by nodeSelector +
toleration, advertising nvidia.com/gpu allocatable so pods land on GPU nodes
and get the device injected via the CDI spec baked by mulga-cdi-generate.
Relies on containerd 2.0.5 default enable_cdi=true (no config fork).
* fix(vm): scale QMP greeting deadline for VFIO passthrough guests
A GPU passthrough guest (-device vfio-pci + large -m) must lock and DMA-map
its entire guest RAM through the IOMMU before QEMU's QMP monitor answers,
which exceeds the fixed 30s greeting deadline and gets the VM SIGKILLed at
launch. Parameterize the greeting timeout (NewQMPClientWithGreetingTimeout)
and scale it to 180s for VMs with GPU attachments; plain VMs keep the 30s
default and reconnects against an already-running QEMU keep it too.
* fix(eks): register GPU workers reliably and expose GPUs to pods
Three fixes let a GPU nodegroup worker boot, register, and advertise its GPU:
- k3s-agent.service gains Delegate=yes (plus KillMode/TasksMax/limits and
overlay/br_netfilter modprobe): k3s embeds containerd+kubelet in-process, so
on cgroup-v2 the kubelet cannot create /kubepods without a delegated subtree
and ContainerManager.Start fails before Node registration.
- The GPU node label rides a k3s config drop-in (20-gpu.yaml node-label +
node-taint) instead of K3S_NODE_LABEL, which k3s ignores; without the label
the nvidia-device-plugin nodeSelector never matches.
- Raise the nodegroup ready timeout to 20m for the ~7min GPU first boot.
Adds the GPU e2e (VPC + NAT-GW egress harness) and a slog-restore unit test.
* fix(eks): install ecr-credential-provider on the GPU node AMI
The GPU worker's kubelet is started (from nodegroup userdata) with
--image-credential-provider-config + --image-credential-provider-bin-dir=
/usr/local/bin, but the eks-node-gpu AMI never built or installed the
ecr-credential-provider binary. kubelet treats a referenced-but-missing
provider binary as a fatal RegisterCredentialProviderPlugins error, so the
node crash-loops and never registers (while the Alpine eks-node worker, which
ships the binary, joins fine). Build + install it in the GPU manifest; the
server-only binaries stay omitted since GPU nodes are agent-only.
Also tee k3s-agent stderr (and its embedded kubelet klog) to the serial
console: Ubuntu logs only to journald, which is off-box-invisible on a worker
in an isolated customer VPC, so a kubelet that fails to register left no trace.
* fix(eks): give the nvidia-device-plugin its own GPU access
The CDI-mode device plugin (--device-list-strategy=cdi-cri) must run
nvml.Init and generate the CDI spec in its own container, but the DS had
no GPU wiring, so nvml.Init crash-looped and nvidia.com/gpu was never
advertised. Run the DS under the k3s-auto-configured nvidia runtimeClass
with NVIDIA_VISIBLE_DEVICES/CAPABILITIES=all, the host root mounted
read-only at /driver-root so it can locate the driver libs + ld.so.cache,
and /var/run/cdi mounted writable so the generated spec is visible to
containerd. No enable_cdi drop-in is needed on k3s v1.32.5 / containerd
2.0.5. GPU workload pods still need only resources.limits nvidia.com/gpu.
Live-validated end-to-end on the single-GPU host (RTX A1000): baked
control-plane AMI, GPU nodegroup, nvidia.com/gpu allocatable, nvidia-smi
pod, zero manual intervention.
* test(ec2): drop obsolete restoreSlogDefault test
Rebased onto dev, which dropped the restoreSlogDefault workaround in #506
(viperblock no longer mutates the global slog). The test referenced the
now-deleted helper and no longer compiles. Real GPU/registration diff is
independently covered (diff coverage 95.3%).
When no br-mgmt bridge is present (supported single-node config where system
instances reach AWSGW via the advertise IP), LaunchSystemInstance correctly
skips mgmt-tap creation but the NIC list still carried the mgmt slot, so
buildNICNetdevs emitted a QEMU netdev/device for a tap that was never created
and QEMU aborted (could not configure /dev/net/tun: Operation not permitted),
failing every CreateLoadBalancer/CreateCluster. Skip the mgmt NIC (index 1)
when its MAC is unset across netdevs, kernel-cmdline mmio slots, and the guest
fw_cfg netcfg blob; multi-node (br-mgmt present) is unchanged.
- 4ncdg: NATSRequest now round-trips the AWS error Message, not just the
sanitized Code. Server sends GenerateErrorPayloadWithMessage(code,
err.Error()); the client returns the Message when present so the CLI prints
the real reason (e.g. "snapshot not found; refusing to restore") instead of
a bare "ServerInternal". One code path fixes every NATS-driven command.
- fbmhl: the UI node view rendered raw host totals (vCPU:80, 0/125.5GB),
ignoring the daemon reserve. It now shows schedulable = total - reserved,
the capacity a tenant can actually consume. reserved_vcpu/reserved_mem_gb
added to the NodeInfo type (already present in the API payload).
Three related nat-mode fixes:
- ura01: nat-transit pool now emits gw_lrp_range_start/end. admin init
sets 100.127.0.16-.254 (239 VPC gateways) instead of the auto-derived
top-16 (~15). Adds GwLrpRange* to PoolData + the spinifex.toml template
so operators can widen any pool.
- 724mm: detectPoolMode returns true only for external_mode=pool. nat and
disabled clusters now skip Phase 8b/8d, which otherwise failed on
AllocateAddress UnsupportedOperation.
- pnz2i: document that system instances (ECS/EKS/LB agents) are
unsupported in nat v1 (require pool mode) in the vpc-networking README
and COMMANDS.md; document the gw_lrp_range_* fields.
* fix(eks): write etcd-snapshot.env at CP launch so snapshots actually run
* test(eks): cover resolveSystemPredastoreURL and buildEKSServiceDeps
Diff-coverage gate for 852f61d9 was unmet (48.8% < 60%) because the
mgmt-bridge branches and EKSServiceDeps assembly had no direct test.
* feat(eks): restore-snapshot DR path (fresh CP + cluster-reset restore + NLB re-point)
* fix(eks): make restore-snapshot fail-safe (validate snapshot, unwind on partial failure, honest status)
* fix(eks): sign etcd-snapshot uploads with deployment region (was 403ing)
* fix(eks): apply recovery directive on DR-seed first boot + single-node replacement host
* fix(daemon): log NATS handler failures at Error (were only on the trace span)
* feat(gpu): Ubuntu NVIDIA GPU node image + GPU AMI selection
Epic A of GPU passthrough to EKS pods / ECS task containers.
- Add gpu-vendor AMI tag constants + GPUVendorForType/IsGPUTypeName helpers
(instance-family prefix -> vendor from knownGPUModels).
- EKS + ECS worker AMI resolution branches to a GPU-tagged node image for GPU
instance types; no fallback to the driverless image (hard error instead).
- Exclude gpu-vendor-tagged AMIs from the plain (non-GPU) lookup so a newer GPU
AMI cannot hijack ordinary nodes (managed-by tag is shared; filters have no
negation, so exclude client-side).
- NodegroupRecord caches GPUEnabled/GPUVendor from InstanceTypes.
- Catalog entries spinifex-eks-node-gpu / spinifex-ecs-node-gpu (dual-tagged).
- New Ubuntu 26.04 GPU node images (eks-node-gpu, ecs-agent-gpu): reuse the
proven DKMS -server driver recipe (drop Docker CE + dev payload), add
nvidia-container-toolkit, OpenRC->systemd port of the worker/agent service
subset, udev EBS by-id bridge, nvidia modules-load, first-boot CDI generation.
- build-system-image.sh: SYSTEM_TAG supports multiple space-separated tags.
* build: allow SYSTEM_IMAGE_BUILD_DIR override for image build dir
Ubuntu GPU images are 16G; a RAM-backed /tmp tmpfs is too small to hold the
qcow2 + expanded raw. Default stays /tmp; SYSTEM_IMAGE_BUILD_DIR redirects the
build onto a real disk.
An ELBv2 LB whose system VM launches but whose in-guest lb-agent never
delivers a first heartbeat previously stayed in provisioning forever with
an empty StateReason: promotion to active happens only on the first
heartbeat and the only provisioning->failed transitions were at
create/launch time. A single wedged LB in the multi-node e2e lb suite
leaked its listener/target-group and cascaded into ResourceInUse and
DependencyViolation on VPC teardown.
- Add a lifecycle reaper that marks failed any LB still provisioning past
lbProvisioningTimeout (4m) with no heartbeat, reusing markLBFailed, and
populate a provisioning StateReason so a stuck LB is self-describing.
Started once at daemon startup; stopped on service Close.
- Harness: add ErrLBProvisioningTimeout and treat a provisioning-timeout
as retryable in createActiveLB; register idempotent LB/listener teardown
up-front so no exit path leaks into the shared VPC.
- IMDS: bounded retry+backoff on the instance-fact and user-data lookups so
a transient gather timeout under multi-node load does not starve the
lb-agent bootstrap.
The coordinated host-shutdown DRAIN phase transitioned every running VM to
StateStopped and migrated it to the operator-stopped shared KV bucket, evicting
it from node-local running state. On the next boot Restore only loads local
running state and treated every StateStopped instance as user-stopped, so
previously-running VMs never relaunched after a host reboot.
Gate on the existing operator-intent signal Attributes.StopInstance (set only by
StopInstances, cleared by StartInstance): stopOne keeps drain-stopped VMs in
local running state instead of migrating them, and Restore relaunches
drain-stopped VMs while operator-stopped ones stay stopped. markUnschedulable
now stamps StopInstance so capacity-demoted instances are not relaunched.
Close four pre-existing lifecycle races surfaced during env19 hardening.
Volume seal/reaper (42hbm, c7ce2): the ebs.unmount handler now seals the
volume before removing it from MountedVolumes, so a retry after a
timed-out-but-completed seal reliably reports NotFound. EBSUnMountResponse
carries a NotFound flag and unmountResponseError treats it as seal success,
so state reconciles to available/detached instead of leaking. The volume
leak reaper reconciles the attachment record (never the data) for volumes
whose owning instance is terminated, so GC can delete them.
Stopped-instance CAS (ozlbq, wlmea): add UpdateStoppedInstance, a
mutate-callback backed by casUpdate(createIfAbsent=false), and migrate
TagStoppedInstance and ModifyInstanceAttribute off the resurrect-prone
Load->mutate->WriteStoppedInstance path. A tag/attribute write racing a
winning start-claim now fails cleanly instead of re-creating the deleted
entry. StartStoppedInstance returns IncorrectInstanceState (not
InvalidInstanceID.NotFound) when the stopped entry is absent but the
instance is present locally.
Every tofu-based e2e workflow set TOFU_LOG to a /tmp path outside
ARTIFACT_DIR. ci-fmt.sh's quiet_run helper appends all tofu init/apply/
destroy output to that file, but the Teardown step (which runs the
destroy) always ran AFTER the existing upload-artifact step, so the
full log — including destroy-time failures — was silently discarded
on every run.
Add an if: always() upload-artifact step after Teardown in each
tofu-based job (e2e.yml, e2e-suites.yml, e2e-nightly.yml, e2e-ddil.yml)
that uploads $TOFU_LOG directly. Since quiet_run appends both apply and
destroy output to the same file, one upload after Teardown now
captures everything, recoverable even when destroy itself fails.
Shipping this log to Elasticsearch/OTLP is deferred to mulga-5n7fe,
which will consume this artifact.
Init now installs an sdk/log LoggerProvider inside the existing
exportEnabled() gate, sharing the same resource as the tracer/meter (incl.
ci.run_id via OTEL_RESOURCE_ATTRIBUTES) and joining lp.Shutdown into the
returned shutdown func. The process default slog handler is fanned out to
both the existing stdout/journald handler and an otelslog bridge, so app
logs export over OTLP with trace correlation while journald output is
unchanged. With no OTLP endpoint configured Init remains a no-op: no
LoggerProvider is installed and the default logger is left untouched.
PR #487 turned attach/detach flaky: reconnectQMP dialed a fresh QMP
client before closing the old connection. QEMU's monitor is a
single-client `server,nowait` listener, so while the old conn holds the
slot the redial gets no greeting, blocks ~30s, and fails, leaving
q.Dead permanently true and every later attach/detach ServerInternal.
Fix the ordering (close, then dial via dialQMPWithRetry to absorb the
brief post-close ECONNREFUSED window).
waitForDeviceDeletedEvent marked q.Dead on any decode error, including
a benign read-deadline timeout — routine, since DEVICE_DELETED is
frequently pre-swallowed by device_del's own reply loop. The
underlying connection survives a timeout, but encoding/json.Decoder
permanently caches read errors on itself, so on a timeout the Decoder
is swapped for a fresh one over the same conn instead of forcing a
full QMP reconnect; q.Dead is now only set on a genuine decode error.
Also: an AttachVolume re-attach to the same instance with a different
device now returns VolumeInUse (matching AWS) instead of
InvalidParameterValue.
* feat(ec2,ebs): harden volume attach/detach + instance lifecycle state machine
Batch of related EBS/instance-lifecycle correctness fixes verified together.
- u94vs: add compare-and-swap (casUpdate/casPut via nats KV Update revision)
to the multi-writer lifecycle KV sites (WriteTerminatedInstance,
WriteStoppedInstance, WriteClusterShutdown); make the teardown-reaper
read-modify-write atomic via a merge callback so retries never re-run
cleaner side effects.
- yi2ed: add ClaimStoppedInstance, a revision-checked delete-as-claim on the
stopped bucket performed before Allocate/Insert/Run, so a forwarded start
and a local fallback cannot double-start one instance onto the same volume.
- siv-473: make AttachVolume idempotent — return attached success when the
volume is already attached to the requesting instance; a conflicting
requested device returns InvalidParameterValue.
- siv-462: reconnect now re-asserts in-use for all attached volumes (boot and
non-boot data volumes), not just boot, so a daemon restart no longer leaves
stateful-pod data volumes split-brain; EFI pflash is skipped.
- 9w2mv: wait for the QMP DEVICE_DELETED event (bounded) before blockdev-del
instead of a blind sleep, so a slow guest unplug no longer exhausts retries.
- soyah: poll (50ms) instead of fixed 1s sleeps in KillProcess and the
viperblockd nbdkit confirm-running path, cutting volume-lifecycle latency.
- siv-481: a cross-node start forward that times out now falls back to a local
start; this is safe because the yi2ed claim is the single serialization point.
* fix(eks): create per-account and leader KV buckets with cluster replica count
The EKS per-account KV bucket and the leader lease bucket were created at
Replicas=1 and never upgraded, since they're created lazily after
upgradeJetStreamReplicas runs at startup. On multi-node clusters this leaves
their single-replica Raft group transiently leaderless during formation,
causing KV timeouts that surface as a deterministic 503 on ListClusters.
Thread clusterSize through GetOrCreateAccountBucket/InitLeaderBucket (backed
by a new utils.GetOrCreateKVBucketWithReplicas) so both buckets are created
at max(clusterSize,1) replicas, matching every other node-count-scaled
JetStream stream.
A fresh cluster or SB re-bootstrap could leave a node's ovn-controller with
its OVSDB client latched on stale Southbound RAFT cluster state, looping on
"clustered database server has stale data" and silently no longer realising
new Port_Bindings. Cross-host CP data-NICs then never bind: no DHCP, no IMDS,
degraded boot. A manual `systemctl restart ovn-controller` was the only
recovery. Root cause is deterministic: setup-ovn.sh restarts ovn-controller
per node as each RAFT member forms, so a controller started for an early
member races the SB membership changes that later joins trigger.
Add three layers of recovery (the ansible bring-up settle pass that fixes this
at the source lives in the mulga repo):
- Per-host wedge watchdog (vpcd/ovn_controller_watchdog.go): not leader-gated
since the wedge is per chassis. Polls SB connection-status; after a stale
threshold issues sb-cluster-state-reset, throttled by a cooldown. Decision
logic takes an injected clock for deterministic tests.
- Reconcile-loop escalation: ensureGatewayClaimed and ensureGuestPortDatapath
escalate to one sb-cluster-state-reset after repeated recompute misses only
when the SB is not connected — a recompute cannot clear a stale-SB wedge.
- Prober primitives SBConnectionState/ResetSBClusterState on GatewayClaimProber
plus the GatewayClaimVerifier interface extension.
Also fix the awsgw OVN health probe, which passed a slashed `-t` socket path
that never resolves and so always reported ovn-controller not_running. Probe
connection-status instead: OVNControllerUp is now true only when connected, so
healthy nodes read ok and a real wedge correctly reads not_running.
The EIP "verify unreachable after disassociate" step polls runSSHQuiet
inside EventuallyErr(90s), but a wedged ssh made os/exec Wait block for
14 minutes — EventuallyErr cannot interrupt an in-flight CombinedOutput,
so one probe consumed the whole 30-min test-binary budget and panicked.
The panic skipped t.Cleanup, leaking the EIP and NAT-gateway VMs, which
held single-node capacity and starved the lb suite's sys.micro launch
(the daemon unsubscribes the full topic, so NATS returns no-responders).
Bound the ssh child at the process layer with exec.CommandContext (30s
deadline) plus WaitDelay, returning a descriptive timeout error. This is
version- and phase-independent, unlike ssh's own ConnectTimeout. 30s is
safe for all callers (worst legit call is ~15s) and well under the 90s
poll budget, so a wedged probe now yields a clean bounded failure with
cleanup intact.
* feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper
* feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode
* feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules)
* feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode
* feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks
* feat(scripts): add --nat-uplink routed mode to setup-ovn.sh
* test(e2e): add routed-NAT single-node natuplink suite
Covers the external_mode=nat lane end to end on a nat-initialized node:
host transit wiring (veth, br-ext port, ip_forward, iptables egress
rules), spinifex.toml contents, disabled EIP surface (empty
describe-addresses, UnsupportedOperation on allocate), default subnet
MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit
/24, guest outbound egress proven via a serial-console probe (nat mode
has no inbound path), and unique transit gateway IPs across VPCs.
Suite self-gates on external_mode=nat; registered as e2e-natuplink and
in SUITES_BUILD so it compiles in CI. Manifest entry stays commented
until CI can provision a nat-mode environment.
* feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column
Adds the exempted_ext_ips column (optional strong Address_Set ref) to the
nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an
in-place SetNATExemptedExtIPs update on the client interface, live client,
and mock. Test NB schema gains the column and covers insert-resolves-UUID,
converge, clear, and missing-NAT paths.
* feat(policy): exempt address set on routed SNAT/EIP rows
Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the
spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW
snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows
un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR)
with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip
path patch the ref in place on pre-existing rows. The set is a singleton
strong ref and is never deleted. Non-routed modes are unchanged.
* feat(host): VPC ingress route primitives for routed-NAT mode
EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that
sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the
spx-nat veth. Replace/del semantics keep both idempotent; a missing route
on removal is not an error.
* feat(external): routed ingress leg on IGW attach/detach, reconcile-safe
AttachIGW's routed leg now also installs the host VPC ingress route via
RoutedIngressHooks, and re-runs on the already-attached path so the
reconcile loop restores host routes (and patches legacy SNAT exempt refs)
after a reboot — OVN state survives restarts but host routes do not.
DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and
the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs,
validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct
pass nil hooks and are unchanged.
* test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern
New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR
via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row
pointing at spinifex_nat_exempt (containing the transit /24), and — after
opening the SG from the transit net (default-closed, AWS parity) — ping and
an sshd TCP handshake against the instance private IP from the host, which
proves the un-SNATted return path. Phase 8 additionally waits for the
second VPC's ingress route to appear on IGW attach. vpc-networking README
documents the jumpbox pattern, SG note, and nat_exempt_cidrs.
* feat(config,admin): public wan pool alongside nat-transit
Restructure admin config generation from single-pool scalars to a Pools
slice so nat mode can emit both the transit pool and an optional public
wan pool. nat mode now accepts --external-pool/--external-source (plus
--external-bind-bridge/--gateway-ip when a public pool is requested),
validated by the shared resolvePublicPoolFlags helper also used by pool
mode. Transit pool always renders first so the IGW allocator picks it.
Formation wire format unchanged (nat mode is single-node only).
* feat(daemon): enable EIP service and auto-assign in nat mode with public pool
External IPAM (and with it the real EIP service, RunInstances public IP
auto-assign, and MapPublicIpOnLaunch on default subnets) now activates
whenever public pools exist: pool mode as before, plus nat mode when a
wan pool rides alongside the transit segment. The transit pool is
filtered out of IPAM — its addresses are gateway-LRP plumbing, never
allocatable EIPs. Transit-only nat configs keep the disabled EIP stub.
* feat(host): per-EIP ingress primitives for routed-NAT mode
DetectUplinkFor parses ip route get to find the interface and source IP
facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move).
EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC
gateway LRP transit IP (src-hinted so host-local traffic still DNATs),
answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC,
WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for
DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated.
* feat(policy,vpcd): host EIP binder option on NATManager
WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds
after priming on both the fresh-add and idempotent-skip paths (host
routes and proxy-ARP are volatile even when the OVN row survives, so
reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort.
The next hop comes from the new gatewayPortIP helper reading the gateway
LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID
key now an exported policy const shared with the external package.
vpcd selects the IGW pool by name instead of index — with two pools in
nat mode the gateway allocator must stay on the transit pool — and wires
the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public
pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced
pools (gateway unknown until lease) proxy-ARP on their bind bridge.
* feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection
dhcp_mac = "derived" (default) | "interface" on external pools. The
interface variant leases with the bind interface's own MAC — WiFi/WWAN
uplinks drop frames with foreign source MACs, so derived chaddr MACs
never complete DORA there. Option 61 stays the lease discriminator.
UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and
persists through KV so restart re-acquires repeat it.
Routers that key leases on MAC and ignore client-id would hand every
lease the same IP; the manager detects an ACK matching the interface's
own address or another client-id's live lease, releases the colliding
lease, and hard-errors advising a static range instead.
admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp
pools bound to non-bridgeable uplinks (wlan/wwan/ppp).
* test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs
The natuplink suite now detects a public (non-transit) pool in spinifex.toml
and switches lanes: Tier-1-only configs keep the EIP-disabled and
no-public-IP assertions; public-pool configs assert the EIP surface is
enabled, the default subnet has MapPublicIpOnLaunch=true, instances
auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress
lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink,
per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the
EIP, reconcile replay across a vpcd restart, and teardown on disassociate.
Docs: vpc-networking README gains a public-pool section for routed-NAT nodes
(init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated
mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool.
* fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd
Routed-NAT mode shells out to iptables for the transit masquerade and
per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for
host-delivered EIPs. Without these sudoers entries vpcd crash-loops with
"command not allowed" on nat-mode nodes.
* fix(ovn): scope DeleteStaticRoute prefix match to the target router
Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache
match could grab another router's row and fail with a referential
integrity violation, wedging IGW attach on multi-VPC nodes. Match only
rows referenced by the target router, mirroring FindStaticRoute. Mock
updated to the same semantics, with a regression test covering drift
replacement across two routers.
* fix(reconcile): always replay AttachIGW so host ingress state is re-ensured
applyIGWs skipped AttachIGW whenever the gateway switch port existed in
actual state, so the already-attached re-ensure path never ran and
routed-NAT host ingress routes lost on reboot were never reinstalled.
AttachIGW is idempotent; run it unconditionally.
* fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins
Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes
collide on the single host route table and the last IGW attach silently
stole the route (route replace). Pass vpcID and the gateway LRP IP
through RoutedIngressHooks; the bootstrap VPC always installs its route,
other VPCs only install when the CIDR is free and only remove a route
their own gateway IP holds. Adds host.VPCIngressRouteVia to read the
current route holder.
* ci(e2e): add nightly cell 19 for routed-NAT single-node suite
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
* fix(admin): nat-mode init summary reflects configured public pool
The summary printed 'outbound-only — no public IPs/EIPs' even when
--external-pool/--external-source configured a public wan pool. Print
the pool range (or dhcp bind) and note EIPs are enabled.
* fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites
Reconcile the routed-NAT branch with dev's ctx-first service APIs after
rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService
interface, the vm_adapters public-IP resolver asserts the ctx-signed
AssociatedPublicIPForInstance, and the default-subnet mapping test threads
ctx into DescribeSubnets.
* feat(otelsetup): OTel init, resource attrs, trace-stamping slog handler
Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT: with no endpoint only the W3C
propagator is installed and globals stay no-op, so instrumented binaries
ship everywhere at zero cost. Resource carries service/host/mulga.env/
mulga.source plus OTEL_RESOURCE_ATTRIBUTES (ci.run_id in CI).
* feat(otelsetup): wire telemetry init into all spx services
Every spx service start command installs the trace-stamping JSON slog
default and the OTel providers via initTelemetry, flushed on shutdown.
Gateway's own slog setup now routes through otelsetup so its level
handling keeps trace correlation.
* feat(gateway): HTTP tracing middleware on awsgw and daemon cluster API
Server span per request with W3C traceparent extraction; awsgw spans are
renamed to the resolved SigV4 service.Action post-auth with account and
region attributes. Request log lines carry trace_id via context.
* feat(nats): propagate W3C trace context over command subjects
NATSRequestCtx injects traceparent beside X-Account-ID and opens a client
span per hop; ServeNATSRequest opens a consumer span joining the producer
trace, with a Ctx variant for handlers that thread context. RunInstances
is plumbed end-to-end (gateway dispatch through placement fan-out to the
daemon handler) and viperblockd ebs.mount/unmount join inbound traces.
* feat(systemd): source optional /etc/spinifex/telemetry.env in service units
Lets operators enable OTLP export (OTEL_EXPORTER_OTLP_ENDPOINT,
MULGA_ENV, MULGA_SOURCE) by dropping a single env file, with no unit
edits. File is optional; absent means telemetry stays disabled.
* feat(ci): export OTel telemetry from E2E envs to the central Elastic sink
All E2E workflows set SPINIFEX_OTLP_ENDPOINT (repo variable override,
default wattle) plus ci.run_id/ci.workflow resource attributes, so
bootstrap-install.sh writes telemetry.env on every node. Teardown runs
flush-telemetry.sh before the final tofu destroy so buffered spans and
metrics reach the sink.
* feat(otelsetup): request metrics and Go runtime metrics via OTLP
mulga.requests counter + mulga.request.duration histogram recorded by the
same middleware that opens spans: awsgw carries the resolved SigV4 action,
the daemon cluster API uses the chi route pattern, and NATS handlers record
per-topic at the subscribe chokepoints (node names stripped to keep
cardinality flat). Go runtime metrics start inside the same env gate.
* feat(daemon): deep spans for QMP commands, VM launch phases, EBS ops
Single chokepoint span per QMP command in sendQMPCommand covering all
callsites; vm.launch phase children (mount_volumes, start_qemu, qmp_connect,
persist_state); ec2.cmd consumer span with traceparent extraction plus
start/stop/reboot/terminate child ops; ebs.mount/unmount spans in
viperblockd with volume attributes.
* feat(telemetry): thread request context through NATS, object store and VM paths
Migrate utils.NATSRequest call sites to NATSRequestCtx (241 sites; W3C
traceparent now flows gateway -> daemon on every request), add GatherCtx and
migrate all Gather call sites, inject traceparent on manual EKS publishes.
ObjectStore interface is ctx-first with aws-sdk WithContext calls and an
otelhttp transport, so predastore spans join the caller trace. NATS consumer
spans record error status when the response envelope carries an error.
vm.Manager and the ec2 handler service interfaces take ctx-first so vm.launch
and QMP spans join the request trace; detached background work uses explicit
context.Background(). Hot-path slog calls use *Context variants for trace_id
binding.
* feat(telemetry): suppress health-check, heartbeat and unrooted trace noise
Health probes record metrics only (no span per poll); the QMP heartbeat
poller marks its context so query-status ticks stop rooting a trace each.
rootSampler honors MULGA_ROOT_TRACE_RATIO (ParentBased ratio, default 1.0)
and the predastore unit sets 0.05: guest block I/O roots are sampled while
request-linked spans are always kept.
* feat(otel): export deployment.environment resource attribute
Elastic APM maps it to service.environment, enabling the native environment
selector across the APM UI (Services, Traces, Dependencies).
* fix(telemetry): stop objectstore rooting method-only HTTP transactions
Background S3 calls (state sync, cleanup loops) carried no caller span, so
otelhttp rooted a trace per request and APM surfaced thousands of parentless
'HTTP GET/PUT/DELETE' transactions with expected-404 probes counted as
failures. Emit the client span only when the request context already carries
one, matching the viperblock backend. Bump viperblock/predastore to branch
heads so CI builds also pick up the equivalent span filter and root sampler.
* feat(telemetry): per-VM guest metrics via dedicated QMP socket + qmp-collector
Goanna Phase 1 producer, operator-consumed first. Each VM gets a second
-qmp chardev (qmp-telemetry-<id>.sock) plus a sidecar discovery file
carrying instance/account/vcpus/taps/monitoring-tier; the new
qmp-collector service polls query-cpus-fast (vCPU thread /proc jiffies),
query-blockstats, query-balloon and host tap sysfs counters, and
publishes CloudWatch-mappable goanna_ec2_* series to NATS
metrics.ec2.<instance-id>. A queue-group bridge in the same process
forwards the series to the node-local OTLP receiver so they reach the
operator sink until Goanna consumes the subjects. The collector never
touches the manager's control QMP socket; telemetry failures never
block a launch.
* fix(qmp-collector): dial NATS via DialTarget, drop ProtectProc, fix meta race
The collector connected to the raw NATS host (0.0.0.0), failing TLS
verification against a cert valid for 127.0.0.1; route through
admin.DialTarget like the other daemons.
ProtectProc=invisible hid the QEMU vCPU threads (dumpable=0) that the
collector must read from /proc/<tid>/stat, so every tick failed and
re-primed without ever publishing. Drop it to default.
Surface collect failures at Warn so silent metric loss is visible, drop
per-tick publish to Debug, and read InstanceID under the lock in run()
to fix a race with updateMeta on ENI hot-plug.
* chore(deps): re-pin predastore + viperblock to dev-landed commits
* feat(eks): reconciler in-place control-plane restart on sustained degradation
Phase 1 of EKS CP recovery (siv-243). The ACTIVE-branch reconciler only
recorded health; a CP VM that died (OOM/host crash) left the cluster wedged
ACTIVE-but-unreachable with no recovery path. Add a CPInstanceControl surface
and bounded in-place restart: once an ACTIVE cluster's CP stays unhealthy past
a grace window, StartInstance is issued (backoff-spaced, capped) when the
instance is stopped/error, re-mounting the same root volume so embedded etcd
survives. Degradation stays AWS-faithful ACTIVE + a ClusterHealth issue rather
than a non-AWS DEGRADED status enum. cpControl nil keeps the reflect-only path.
Wiring of a real CPInstanceControl in the daemon and the error-state
manager-eviction fix are the remaining Phase-1 integration steps; Phase 2
(etcd snapshot cadence + restore-snapshot) is tracked separately.
* feat(eks): wire control-plane restart into the reconciler via instance service
Threads a CPInstanceControl into the per-cluster reconciler (siv-243 Phase 1).
cpControlAdapter binds the daemon instance service + account: InstanceState
reads the CP's EC2 state via DescribeInstances, StartInstance restarts a stopped
CP via StartStoppedInstance, which loads from the shared stopped store and
re-mounts the same root volume so embedded etcd survives. Wired as
EKSServiceDeps.CPControl = d.instanceService and passed through spawnReconciler.
Recovers a cleanly-stopped control plane co-located with the reconciler leader.
Remaining: DescribeInstances is node-local so a cross-node stopped CP needs an
aggregating describe; and an error/crashed CP (not in the stopped store) needs a
dedicated error-state restart path — both tracked as Phase-1 follow-ups.
* feat(eks): cross-node CP describe + error-state control-plane restart
The reconciler's control-plane recovery was wired to the node-local instance
service: DescribeInstances only saw the leader's node and StartStoppedInstance
only handled a cleanly-stopped CP from the shared KV. A crashed/OOM CP ends in
error state, still resident in its owning node's manager map (not the stopped
store), so recovery never reached it — the actual incident in the recovery plan.
Wire EKSServiceDeps.CPControl to a NATS-backed eksCPControl. DescribeInstances
fans out across every host (gateway scatter-gather) so the reconciler observes a
CP on any node. RecoverInstance drives the gateway StartInstances two-step:
ec2.cmd.<id> recovers a live error/running owner in place (the owner stays
subscribed through the crash; vmMgr.Start relaunches error->pending, re-mounting
the etcd root volume), falling back to ec2.start shared-KV rehydration for a
stopped CP.
The CP VM is ManagedBy=EKS, hidden from customer DescribeInstances; relax that
hide so the owning account resolves its own managed CP when targeting it by
explicit instance ID (bulk listings still hide it). EC2APIState already maps the
internal error state to the AWS closed-enum stopped, which cpRestartable accepts.
* fix(eks): reconcile CP as system account, not customer
The control-plane VMs are launched under the system account
(admin.SystemAccountID()), not the customer account that owns the cluster
record. spawnReconciler was binding cpControlAdapter to the customer account,
so InstanceState/RecoverInstance ran as the customer — which cannot see or own
its own cluster's system-owned CP VMs. DescribeInstances returned zero and the
reconciler logged 'control-plane instance not found', never restarting a wedged
CP. Bind the adapter to the system account instead.
Revert the earlier DescribeInstances managed-VM relaxation: it was based on the
mistaken assumption that CP VMs are customer-owned. Visibility is resolved by
the system-account adapter, so the explicit-ID escape hatch is dead code.
* fix(vm): pre-create mgmt tap on disk-boot restart path
The direct-boot branch pre-creates the management tap via NetworkPlumber.SetupTap
(owned by the daemon euid), but the disk-boot/else branch only appended the QEMU
tap,id=mgmt0 device without creating the tap. The daemon runs as a non-root user
without CAP_NET_ADMIN, so QEMU can only attach a tap it already owns; on the
disk-boot path QEMU tried to create /dev/net/tun itself and failed with
'Operation not permitted', failing every stopped-instance restart (StartInstances
returned ServerInternal). This blocked EKS control-plane recovery: the reconciler
found the stopped CP and issued the restart, but the VM never booted.
Call SetupTap for the mgmt tap in the disk-boot branch, mirroring direct-boot.
* feat(eks): recover all HA control-plane members, not just primary
The reconciler restarted only ControlPlaneInstanceID (the primary CP
member), so on an HA cluster the other server VMs stayed stopped and
etcd never regained quorum (1/3) — apiserver healthz stayed fail even
after recovery ran. maybeRecoverControlPlane now iterates every member
in ControlPlaneNodes (falling back to the scalar ID for single-node
clusters), restarting each stopped/error member per pass while
preserving the grace window, attempt cap, and backoff gates.
Refs mulga-siv-509
* fix(eks): keep cloud-init polling IMDS so primary CP gets its user-data
The eks-node system VM's user-data is delivered only via IMDS (the
NoCloud seed was retired in #411). IMDS (169.254.169.254) is reachable
only once mulga-mgmt-net brings up the data NIC via DHCP and the per-tap
br-imds datapath reconciles, which on the first-launched (primary) CP
can lag cloud-init's stock 120s probe window. With the default max_wait
cloud-init falls back to DataSourceNone, permanently losing the launch
user-data (no SPINIFEX_K3S_ROLE -> no k3s server) and wedging the cluster
in CREATING.
Bake a cloud.cfg.d drop-in that pins DataSourceEc2 and raises max_wait to
600s so cloud-init keeps polling until the datapath is ready instead of
falling through to None. Worst case a slow boot, never a lost role.
* fix(eks): warm IMDS instance-role creds before first SigV4 sign
On IMDS-credentials mode the on-VM SigV4 helpers (eks-token-webhook,
eks-gateway-publish, mulga-eks-state-report) sign via the SDK default
chain, which resolves in-VM IMDS instance-role creds. The per-tap
br-imds datapath lags VM boot by minutes (observed ~264s), and the SDK's
IMDS client fast-fails each probe (250ms dial), so early Retrieve calls
return 'context deadline exceeded' / 'no EC2 IMDS role found'.
eks-gateway-publish only retries for 30x5s=150s, shorter than the
datapath warm time, so it exhausts its budget and drops the
bootstrap/state publish -> cluster hangs in CREATING.
The credential chain is sound (the CP role trusts ec2.amazonaws.com and
AssumeRoleForInstance evaluates that principal); the failure is timing.
Add gwsign.Signer.EnsureCredentials and block in eksgw.New until the
instance-role creds first retrieve (2s interval, 10m ceiling aligned
with cloud-init max_wait), so every helper's first signed request
already holds creds instead of racing a cold datapath. Static-key mode
is unaffected.
* fix(eks): retry CP data-NIC DHCP until cross-host datapath is up
A single ~16s DHCP one-shot on the primary data NIC lost the race against
the cross-host OVN datapath (logical-port binding + per-tap reconcile on the
remote placement host), stranding the NIC IP-less so the Ec2 datasource never
reached IMDS and the CP member wedged. Add dhcp_acquire: retry the one-shot on
a budget aligned with cloud-init's Ec2 max_wait (600s), confirming an inet
address before success. Each attempt still exits leaving no daemon, so
cloud-init's later ENI management is unaffected.
* fix(eks): drop reconciler registry entry when Run self-exits
ReconcilerRegistry.Spawn only cleaned up its holder on ctx cancel
(Stop/StopAll). When a ClusterReconciler's Run returned on its own — a
terminal cluster state or a lost lease — nothing cancelled the ctx, so
the holder leaked and a later same-name create silently no-opped at the
head of Spawn, never reconciling to ACTIVE.
Thread a finished channel from RunClusterReconciler's Run goroutine back
to Spawn's watcher, which now cleans up on whichever fires first: an
external cancel or Run exiting. On self-exit it deletes its own holder
and releases the lease, so a fresh same-name create re-spawns and
reconciles.
* fix(eks): run mulga-mgmt-net in boot runlevel before cloud-init-local
CP user-data is delivered only via IMDS since the NoCloud seed was retired
(#411), so cloud-init DataSourceEc2 must reach 169.254.169.254. mulga-mgmt-net
DHCPs the data NIC that makes IMDS reachable, but it was added to the default
runlevel and only declared `before cloud-init` (the network-stage service).
cloud-init-local therefore ran its init-local Ec2 crawl first with no data-NIC
route and burned its whole probe window (~4 min) before mulga-mgmt-net brought
eth0 up, regressing CP bringup from ~1m50s to ~5m54s.
Add mulga-mgmt-net to the boot runlevel in setup.sh instead of default, drop it
from ENABLE_SERVICES, and order it `before cloud-init-local` so the data NIC and
IMDS route are up before the init-local crawl. Keeps the IMDS-only user-data
path; the raised Ec2 max_wait stays as a safety net for host br-imds datapath lag.
* fix(vm): refresh LaunchTime on crash-restart so watchdog spares relaunch
RestartCrashedInstance re-launched a crashed VM without refreshing
Instance.LaunchTime. The pending watchdog measures now-LaunchTime against
PendingWatchdogTimeout, so the stale original launch time tripped
launch_timeout on the next tick and terminated the crash-restart
mid-flight. Refresh LaunchTime under the state lock, mirroring the
daemon-restart restore path (restore.go), so the relaunch gets a fresh
timeout window. Adds a regression test asserting LaunchTime advances.
* feat(eks): reconcile control-plane member count after a lost member
The cluster reconciler self-heals a stopped/error control-plane VM in
place but never replaced a terminated/gone member, so an HA cluster ran
below haControlPlaneCount permanently. Add a member-count reconcile pass
that provisions one replacement CP joining the surviving etcd quorum.
- CPProvisioner + EKSServiceImpl.ProvisionReplacementCP: replay the
persisted create template on a free host, the single-node analogue of
the spread join branch, with rotating creds re-derived.
- ControlPlaneTemplate persisted on ClusterMeta at create (per-node and
rotating-cred fields cleared). SwapControlPlaneMember CAS swaps the dead
node for the replacement and refreshes the [0] scalar mirrors.
- maybeReplaceControlPlaneMember fires only when no member is restartable
and one is truly lost, a healthy majority survives, bounded by grace,
backoff and an attempt cap; placement excludes live-member hosts.
- Replacement prunes the superseded member from etcd once joined:
k3s-first-boot deletes the stale Node (EKS_ETCD_PRUNE_PEER_IP) so quorum
width returns to N. json tags added to K3sServerInput (now persisted).
* feat(eks): in-guest control-plane self-diagnosis on healthz fail
When a control-plane apiserver reports unhealthy, the reconciler only
saw healthz=fail with no cause, and the guest has no on-demand command
channel (serial root login is password-gated, no guest agent). Make the
CP self-diagnose and relay the cause through the one-way state report.
- mulga-eks-state-report.sh: diagnose() collects failing /readyz?verbose
subchecks, isolated /healthz/etcd reachability, and etcd-db-dir free
space; on fail write a verbose block to the host-captured serial
console and add a JSON-safe reason to the state payload.
- ServerStateReport: add Reason (omitempty; empty from a healthy CP or
an older AMI predating the field).
- cluster_reconciler observe(): surface the reason in the health issue.
- Tests: POSIX unit test for the agent (stubbed kubectl/df); Go tests
for the reason-bearing and terse health-issue paths.
* feat(eks): 15-min etcd snapshot cadence alongside nightly
Nightly alone is too coarse a recovery point for a young cluster. Install the
same mulga-eks-etcd-snapshot script into a second Alpine crond periodic dir
(/etc/periodic/15min) so it also runs on a ~15-minute RPO window; the script
picks its cadence tier from the dir it is invoked from ($0 sniff) and encodes
the tier in the snapshot key (etcd-{tier}-{ts}.snap), so a predastore S3
lifecycle rule can retain the frequent rolling window shallow and the nightly
copies deep. Server-role only (agents exit early); no restore-path change.
Add a self-contained POSIX unit test (stub k3s/curl/logger, symlinked crond
dirs to exercise the real $0 tier sniff) covering agent no-op, frequent + daily
key prefixes, local-snapshot prune, and quiet no-op without creds.
* feat(eks): auto-reform wedged etcd quorum via boot-time cluster-reset
An HA k3s control plane does not self-heal after a simultaneous full-CP
restart: embedded etcd never reforms quorum (etcd:unreachable sustained),
and in-place VM restarts cannot fix it. Add a last-resort escalation that
drives the documented k3s cluster-reset recovery, pull-based so the host
never needs an inbound channel into the guest.
Reconciler: maybeReformEtcdQuorum escalates only when every CP member is
VM-running but the health issue carries etcd:unreachable, the cluster is
HA, and a grace window has elapsed. It sets per-member recovery directives
(seed = cluster-reset, others = wipe-rejoin), epoch-bumped and keyed by
instance ID, then restarts the members. Bounded by grace/backoff/attempts;
a recovered cluster clears the directives once. Enabled via
WithEtcdResetRecovery alongside the existing CP control wiring.
Directive plane: per-member KV directive {epoch, action, snapshot} with
store helpers, an EKS service Get/Set pair over NATS-RPC, and an
internal-recovery gateway route (system-cred carve-out, same as
internal-addons). The CP server instance role gains eks:GetRecoveryDirective.
Guest: mulga-eks-k3s-recovery is a pre-k3s oneshot (server role only) that
fetches its directive through the AWS gateway (eks-gateway-fetch -resource
recovery), epoch-guards one-shot application, then runs
`k3s server --cluster-reset` (optionally restoring a predastore snapshot)
or wipes the local etcd datastore to rejoin the reset seed. Wired into the
image manifest, setup.sh perms, and the eks-node-role selector.
Tests: reconciler escalation gates + recovery clear, directive store epoch
increment + service validation, and a POSIX unit test for the guest agent.
* fix(eks): reboot running CP members to apply etcd cluster-reset directive
The etcd quorum-reformation escalation drove StartInstance on the seed and
followers, but the reset gate only fires when every member is VM-running. The
StartInstances path rejects a running instance with IncorrectInstanceState, so
the guest never re-entered its boot sequence and the cluster-reset/wipe-rejoin
directive was never applied — the wedged control plane stayed wedged.
Reboot the running members (QMP system_reset) instead, re-entering the boot
sequence so the on-VM recovery agent applies the directive. Adds RebootInstance
to the CP control surface (interface, adapter, NATS-backed daemon impl) and
switches maybeReformEtcdQuorum to it; the stopped/error in-place restart path
keeps StartInstance.
* fix(eks): graceful stop for etcd reset recovery
The etcd quorum-reformation reset path forced running CP members through a
hard QMP system_reset to re-enter the boot sequence and apply their recovery
directives. A hard reset skips a clean unmount: on the next boot e2fsck
cleared the recovery and k3s runlevel entries (dangling inodes), so the
cluster-reset never ran and k3s was unstartable — the reset defeated recovery.
Replace the reboot with a graceful StopInstance (QMP system_powerdown, clean
unmount, SIGKILL fallback). The existing in-place restart path
(maybeRecoverControlPlane) then boots each stopped member clean on a later
tick, and the on-VM recovery agent applies its directive against an intact
filesystem. Reuses existing lifecycle machinery; the stop also tolerates a
wedged QMP client via the SIGKILL fallback.
Swaps RebootInstance -> StopInstance across CPInstanceControl,
cpInstanceController, the cpControlAdapter, and the NATS-backed eksCPControl
(gateway RebootInstances -> StopInstances), plus the reset/recovery test fakes.
* fix(vm): retry QMP dial and unlink stale socket on relaunch
A QEMU killed with SIGKILL (crash or etcd-reset restart) leaves its
qmp-<id>.sock inode behind. On relaunch the startup os.Stat probe and the
QMP dial both raced that dead listener, so a single dial returned
'connection refused': StartStoppedInstance then reported ServerInternal and
left the instance KV-stopped even though the guest booted fully. The wedged
member stayed invisible to the reconciler's cpLive gate, stalling recovery.
Unlink the stale socket before launch so the inode that appears is the fresh
QEMU's, and retry the QMP connect with a short backoff to cover the window
where the socket is bound but not yet listening. Handshake failures are not
retried — a listener answered, so the error surfaces immediately.
* feat(eks-ui): surface cluster Health.Issues on the EKS console
DescribeCluster is AWS-faithful (Status stays ACTIVE, degradation lives in
Health.Issues), but the cluster detail page only rendered the status badge, so
a degraded control plane still showed as ACTIVE with no visible signal. Add a
destructive health badge next to the status and a Health row that lists the
issue messages, mirroring the addons tab. Healthy clusters read OK.
* chore: rebuild ui
* fix(viperblockd): stop mount-path daemon VB uploader to prevent checkpoint clobber
The daemon-side VB created in the ebs.mount handler is stored in
MountedVolume.VB for the whole mount, and its background chunk uploader was
only stopped at daemon shutdown. That state-only VB ran DrainToBackend every
30s, overwriting the live checkpoint the nbdkit plugin VB (the data-path
writer) keeps updating, leaving SeqNums inconsistent with sealed chunk
ciphertext and causing transient AEAD auth failures on cloned encrypted
volumes. Stop its uploader right after New, matching the other short-lived VB
call sites.
Also bump viperblock to the atomic chunk-ObjectID allocation fix.
* build(deps): pin viperblock to landed AEAD concurrency fix
Bump github.com/mulgadc/viperblock to the dev commit b08bea6 that
eliminates "cipher: message authentication failed" on encrypted volumes
under concurrent read-modify-write + background chunk drain. Removes the
local-path replace used while the fix was in flight.
* fix(viperblock): stop chunk-uploader goroutine leak at all VB call sites
viperblock.New() unconditionally starts a background chunk-uploader
goroutine (30 s interval) that only VB.Close() stops. Seven call sites
created short-lived or state-only VB instances without calling Close(),
leaking one uploader goroutine per instance. On a single node the leaked
goroutines exhaust memory (predastore RSS grew ~1.3 GB/5 min at idle
post-single-suite), causing liveMemGate to reject launches and the lb
suite to fail.
The leaked uploaders are also the primary cause of the AEAD auth
failures (mulga-be2ev): a leaked VB from prepareRootVolume holds the
clone-point BlockLookup and overwrites the correct live checkpoint in S3
every 30 s, leaving SeqNums inconsistent with sealed chunk ciphertext.
Fix all seven sites:
- instance/service_impl.go prepareRootVolume: defer StopChunkUploader +
StopWALSyncer after successful viperblock.New (mirrors the EFI path's
intent; full Close is unsafe here because cloneAMIToVolume already
called SaveBlockState + RemoveLocalFiles)
- viperblockd.go ebs.delete handler: add StopChunkUploader alongside
existing StopWALSyncer
- viperblockd.go ebs.unmount handler: same
- viperblockd.go shutdownVolumes: same
- snapshot/service_impl.go snapshotVolume: defer both stops after New
- volume/service_impl.go CreateVolume: defer both stops after New
- image/service_impl.go snapshotRunningVolume: defer both stops after New
- image/service_impl.go snapshotStoppedVolume: defer both stops after New
Closes mulga-0zzci
Closes mulga-ji5fi
* fix(deps): bump viperblock to 01a7e686 (AEAD regression fix)
The prior pin (6af89855) carried the snapshot-AMI AEAD regression that
corrupts block storage on restore. Point at the merged viperblock dev fix.
---------
Co-authored-by: joshsiv-mulga <mrk@mulgadc.com>
Failed LBs previously reported an empty State.Reason, so e2e failures
read 'entered terminal state failed:' with no cause. Record the cause
on every failed transition — launch error (e.g. insufficient sys.micro
capacity), missing system credentials, missing internal mgmt return
route, and async-launch panic — persist it as StateReason, and map it
to LoadBalancerState.Reason in describe responses. Cleared when the
first heartbeat transitions the LB to active.
The internal lb subtests raced the previous subtest's sys.micro VM
deallocation: fixed 15s sleeps were not enough on capacity-tight
single nodes, so the LB entered terminal state failed (insufficient
capacity, then nats no-responders after the daemon unsubscribed the
full topic).
Replace the sleeps with createActiveLB: create LB + listener, wait for
active via the new non-fatal harness WaitForLBActiveErr, and on a
terminal failed state (wraps ErrLBTerminalFailed) tear down fully and
retry with backoff, up to 3 attempts.
go.mod referenced v1.10.0 which pre-dates LoadLiveCheckpoint. CI builds
without go.work so e2e-build failed to compile. Pin to viperblock dev tip
6af8985 (feat/snapshot-ami merged) until v1.11.0 is tagged.
* feat(gpu): add H200/RTX Pro 6000 SE models and fix MIG host resource allocation
- Fix H100 SXM/PCIe device ID labels swapped in models.go (10de:2330=SXM, 10de:2331=PCIe)
- Add H200 SXM (10de:2335) to knownModels, computeModels, and migCapableModels
- Add NVIDIArtxPro6000BlackwellSE to definitions.go mapped to g7e family (matching AWS g7e instance type)
- Add MIGHostResources() policy table: 4 vCPUs / 32 GiB host RAM per compute slice, covering all known NVIDIA MIG profiles
- GenerateMIGTypes now emits non-zero vCPU/memory for EC2 instance type descriptors
* fix(daemon): enforce CPU/memory limits for MIG instance types
Whole-GPU passthrough remains bypass-only (gpuManager.Claim is the sole gate).
MIG types now fall through to canAllocateCount so host CPU and RAM are checked
before launch — prevents over-committing a host where one physical GPU backs
multiple concurrent slices, each with non-trivial vCPU/memory allocation.
* feat: Add snapshot image promotion flow
* fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs
* fix: test coverage and more consistent prefix check
* fixed testing
* more tests
* fix: added new graceful process killer in utils, bumped NATS timeout for testing
* fix: complete at-rest encryption launch + volume lifecycle (#361)
* chore: bump viperblock to encrypted-snapshot VolumeUUID nonce fix
Pull viperblock 1044c2a (feat/snapshot-uuid-nonce-fix) so E2E builds against
the fix for the encrypted AMI snapshot metadata tag-verify failure
(cipher: message authentication failed) on instance launch. Pairs with
viperblock PR #22.
* fix: forward viperblock master key to nbdkit mount path
viperblockd loaded the master key but never passed it to the nbdkit
plugin it spawns for volume mounts, so encrypted boot volumes opened
in cleartext and failed LoadState with an encryption-state mismatch.
Forward cfg.EncryptionKeyFile into NBDKitConfig and emit the plugin's
encryption_key_file= argument when set.
* chore: bump viperblock to fresh-volume VolumeUUID mint fix
* chore: bump viperblock to snapshot-link LoadState ordering fix
* fix: route encrypted-volume config writes through viperblock keyholder
config.json is a sealed VBState whose AES-GCM tag and StateSeqNum-derived
nonce can only be advanced by the master-key holder that owns the volume.
The EC2 volume handler wrote it directly to S3, so mergeVolumeConfig had to
refuse encrypted volumes (rewriting would strip the tag) — bricking
UpdateVolumeState (attach in-use / detach available) and ModifyVolume.
Route encrypted-volume config updates through viperblockd, mirroring the
ebs.snapshot idiom: per-volume ebs.config.{volumeID} hits the owning node's
live VB (the single StateSeqNum writer); on ErrNoResponders the ebs.config
queue group opens the detached volume exclusively and reseals. Cleartext
volumes keep the direct-S3 fast path. No crypto leaves viperblockd.
mulga-siv-323
* fix: unwrap encryption envelope when snapshot handler reads config.json
CreateSnapshot and snapshotInUseByVolumes decoded config.json straight into
a VBState. For encrypted volumes config.json is an at-rest envelope
{payload, authtag}; a raw decode yields a zero VBState (SizeGiB==0), so
CreateSnapshot 500s on the size guard and the in-use scan silently drops
encrypted volumes' snapshots. Unwrap via viperblock.StateBody first,
mirroring the volume handler's mergeVolumeConfig path.
mulga-siv-324
* test: cover viperblockd ebs.config handlers
Unit/integration coverage for applyConfigUpdate, makeConfigUpdateHandler
(ebs.config.{volumeID}), and the ebs.config queue-group fallback: grow-only
size reconcile, live-VB apply, invalid JSON, apply errors, and the
detached-volume open path. Clears the diff-coverage gate (68.8%).
* chore: pin viperblock to dev tip (#22 merged)
* fix(viperblockd): seal volume block-map to predastore on detach + durability e2es (#371)
* feat: cloud-init EC2 datasource (#411)
* feat: Add snapshot image promotion flow
* fix: Small changes to viperblock/spinifex so snapshot captures delta properly and instances can boot from snapshot AMIs
* fix: test coverage and more consistent prefix check
* merge changes
* chore: remove dead SnapshotSub field and SIGTERM snapshot remnants from viperblockd
SnapshotSub was added during the NATS/SIGTERM snapshot attempts but is never
set post-rebase. Removes the field from MountedVolume and the two nil-checked
unsubscribe blocks. Re-adds the nbd import which is still needed for NBDKitConfig.
* fix: post-rebase gofmt fixes and remove dead snapshot handler code
Split gofmt-squashed lines in go.mod, vm_adapters.go, nbd_test.go,
viperblockd.go, volumes.go, and volume_durability_test.go. Removes the
dead unix-socket snapshot handler tests from viperblockd_handlers_test.go.
* test: add viperblock snapshot S3 coverage tests
Exercises snapshotVolume, snapshotRunningVolume, and snapshotStoppedVolume
past the nil-config guard using an httptest server that returns 403.
The AWS SDK credential check fails synchronously without network I/O,
keeping the tests fast while covering the S3Config construction, master key
load, and viperblock.New code paths. Brings diff coverage from 45% to 69.9%.
* fix: cloneAMIToVolume reads config via format-agnostic path
LoadStateRequest expects a viperblock binary envelope (encrypted),
but CreateImageFromInstance writes the AMI config as plain JSON via
putAMIConfig. Replace LoadStateRequest with s.amiLoader.GetAMIConfig
which uses viperblock.StateBody to handle both formats, matching the
path used by ListImages and all other AMI readers.
* fix: pin viperblock to feat/snapshot-ami pseudo-version for LoadLiveCheckpoint
go.mod referenced v1.10.0 (released from dev) which pre-dates LoadLiveCheckpoint.
CI builds without go.work so it resolved to the published version and failed.
Pin to the snapshot-ami HEAD pseudo-version until the branch is merged and tagged.
* fix: replace indexOf with slices.Index, revert e2e runner and viperblock version
- nbd_test.go: remove local indexOf helper, use slices.Index from stdlib
- e2e-suites.yml: revert ubuntu-latest back to ubuntu-26.04 with correct formatting
- go.mod: revert viperblock back to v1.10.0 (will resolve once viperblock PR merges)
* fix: remove pseudo-version pin from go.sum
* feat: drain viperblock via snapshot socket before CreateSnapshot
Before LoadLiveCheckpoint(), dial {data_dir}/viperblock/{vol}/snapshot.sock
and wait for DrainToBackend() to complete (30s timeout). This ensures WAL
chunks and the live checkpoint are current in S3 before the snapshot VB
reads them. If the socket is absent the instance is stopped and Close()
already drained.
fix: socket path used s.config.WalDir which is empty in the daemon config;
changed to s.config.DataDir + "/viperblock" to match the path the NBD
plugin uses when creating the socket.
* feat: wire snapshot/AMI UI actions and add PromoteImage NATS endpoint
- Register as AMI from snapshot detail (RegisterImage with BDM pointing at snapshot)
- Deregister from image detail (DeregisterImage with navigate-away on success)
- Promote to System from image detail (admin-only, irreversible via UI)
- NATS subject spinifex.image.promote wired through gateway → daemon → existing admin.PromoteSystemImage
- signedFetch extended with optional params to support action-keyed extra fields
* refactor: address PR review on PromoteImage types and subscription ordering
- Merge PromoteImageInput/PromoteImageOutput into PromoteImageOpts/PromoteImageResult
(add JSON tags; drop the duplicate pair)
- Inline the input variable in gateway/spx/promote_image.go
- Move spinifex.image.promote subscription to the spinifex.* group in daemon.go
* test: cover snapshotVolume/Running/Stopped error paths for diff coverage
Add encryption-key-load-error and LoadState-failure tests for the three
new viperblock snapshot paths (snapshotVolume in snapshot service,
snapshotRunningVolume and snapshotStoppedVolume in image service).
Diff coverage: 58.8% → 65.5%.
---------
Co-authored-by: Josh Siviour <mrk@mulgadc.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
* fix: ECS StopTask reaps container before releasing capacity
StopTask marked a task lastStatus=STOPPED and released its capacity/ENI in
the control plane, but nothing ever told the ecs-agent to kill the container.
The container kept holding its host ports, so DescribeTasks reported a STOPPED
task whose container was still RUNNING, and a replacement task placed on the
freed capacity that rebound the same host port hit EADDRINUSE.
Wire the agent-driven cooperative stop the design intended (ecs-v1.md:165):
- Add a per-instance stop inbox (StopsPrefix/StopKey) and bus.StopDirective.
- requestStopTask marks DesiredStatus=STOPPED and posts a stop directive,
leaving lastStatus/capacity/ENI/targets untouched. StopTask, DeleteService,
and service scale-in use it; the heartbeat reaper and cluster/instance
teardown keep the immediate forceStopTask since no live agent can reap.
- PollAssignments returns pending stops; the agent lists containers by
mulga.ecs.taskID label, Removes them (kill+delete, freeing ports/netns),
then reports STOPPED. recordTaskState performs the single capacity release
on the agent's STOPPED report and reclaims the stop inbox.
- listServiceTasks excludes DesiredStatus=STOPPED tasks so a cooperatively
stopped task drops out of the service's running total immediately.
DescribeTasks now keeps the task RUNNING with desiredStatus=STOPPED until the
container is actually reaped, matching AWS ordering and eliminating the port
race.
* test: cover StopTask reap paths — poll stop-inbox drain, unplaced force-stop, agent stopTask edges
* ci: retrigger PR checks
- RegisterTaskDefinition rejects container secrets[] (InvalidParameterException)
and warns on unsupported log drivers instead of silently dropping them.
- Capture logConfiguration; only host-side json-file is collected, other drivers
warn at register and reach the agent for parity. Document json-file retrieval.
- Plumb executionRoleArn taskdef -> assign -> agent; the agent assumes it to
authorize ECR pulls, falling back to the instance role when unset.
- Add deployment objects with a health-gated rolling update honoring
deploymentConfiguration minimumHealthyPercent/maximumPercent, plus a deployment
circuit breaker with optional rollback to the last-good task definition.
EnsureLogicalSwitch/Router/PortGroup returned a client-side named-uuid on the
fresh-insert path but the real server uuid when the row already existed, so a
caller storing the returned uuid then doing Where(uuid) would miss. Resolve the
persisted uuid from the monitored cache after insert and add an explicit created
flag; topology callers now branch on created instead of uuid equality.
ReplaceACLs deleted and recreated the whole ACL set every reconcile, minting new
row uuids even for an unchanged SG and forcing ovn-northd flow recompute on every
drift tick. Diff current vs desired ACLs and no-op when unchanged (live + mock).
* test(network): in-process OVN NB reconcile harness
Add ovntest.StartNB: an in-process OVN Northbound DB backed by libovsdb's
pure-Go OVSDB server (no ovsdb-server/ovn-northd C daemons) plus a vendored
NB schema subset matching nbdb. Real LiveClient connects over a unix socket,
so reconcile scenarios exercise intent generation, reconcile wiring, LiveClient
and OVSDB transaction semantics in milliseconds without VMs.
Add live reconcile scenarios asserting NB rows and idempotency:
- VPC + subnet + SG + ENI from a hand-built IntentState
- the same seeded through JetStream KV via LoadIntentFromKV (front-to-back)
- IGW attach + drop gate + public-instance exemption (Logical_Router_Policy)
The idempotency snapshot normalizes UUID and pointer churn so it asserts
logical stability; datapath and NB->SB translation stay in the e2e suites.
* test: add orphan-prune and kitchen-sink golden live NB scenarios
Extend the in-process OVN reconcile harness with the two P0 completeness
scenarios. TestScenario_OrphanPrune_Live locks the teardown path and the
ReconcileApplyOnly startup-race guard: apply-only must not prune, full
Reconcile must, and re-pruning is idempotent. TestScenario_KitchenSinkGolden_Live
reconciles a maximal intent exercising every applyX path (VPC, public+private
subnets, two ENIs, SG with tenant rules, IGW, associated EIP, NAT gateway,
IGW/NATGW reroutes, drop gates) and asserts the normalized NB snapshot against
a checked-in golden, mechanically catching an unwired or mis-wired resource
type. Regenerate the golden with UPDATE_GOLDEN=1 go test.
* fix: repoint DB-node ovn-northd at RAFT member list; extend add-nat deadline
Two related fixes for multi-node OVN clusters:
- setup-ovn.sh (mulga-siv-504): in the clustered --management branch, point
ovn-northd's client NB/SB connections at the full RAFT member list derived
from OVN_REMOTE, so the active northd follows the leader instead of dialing
its local member only. Without this, northd stops advancing SB_Global.nb_cfg
once leadership moves off the node, wedging the ovn-nbctl --wait=hv barrier.
Guarded on a real member list so the standalone single-DB-node path is
unchanged.
- utils/nats.go (mulga-js-206): raise the add-nat request deadline from 10s to
45s so the client no longer rolls back before the server-side --wait=hv flows
barrier (30s) completes.
* fix: stop ovn-northd restart from tearing down split NB/SB ovsdb servers
The packaged ovn-northd.service ExecStop runs 'ovn-ctl stop_northd' without
--ovn-manage-ovsdb=no, so restarting northd to apply OVN_CTL_OPTS also stops the
separately-managed NB/SB ovsdb-server units. On a clean clustered bring-up this
races the freshly-started DBs and kills them, so setup-ovn's wait for the NB
socket times out and the RAFT create/join aborts. Install an ovn-northd.service
drop-in overriding ExecStop with --ovn-manage-ovsdb=no so northd never touches
the DB lifecycle (mulga-siv-504).
* test(e2e): capture SNAT datapath on NAT GW egress failure
TestNATGateway captured nothing when the private-VM egress probe failed, so
the baremetal nightly's intermittent 100% packet loss left no OVN/OVS state in
the failure bundle to root-cause it. Register harness.OnFailure before the
egress-verify poll to dump the SNAT datapath (OVN show, NAT rules, port
bindings, br-int flows, conntrack, ARP) keyed on the NAT GW external IP,
matching the eip/vpc datapath tests.
Also add a br-ext localnet flow capture to dumpDatapathState: br-int flows miss
the egress hop where SNAT'd traffic leaves for the physical uplink, which is
exactly where an egress black-hole would show.
* test(e2e): add OVN pipeline diagnostics to NAT GW egress capture
Capture the gateway router's per-stage logical-flow packet counts
(ovn-sbctl --stats lflow-list), MAC_Binding, static routes and the VM's
logical switch port alongside the existing OVS/conntrack/ARP dumps. When
the SNAT flow reads n_packets=0 while the guest transmits, these pinpoint
which pipeline stage drops the packet upstream of lr_out_snat.
* test(e2e): dump full lflow-list + datapath ct/flows for egress RCA
The IP-filtered lflow dump hides the default-drop ACL flow (its match
carries no IP), so a pre-SNAT egress drop at ls_out_acl only showed the
allow-flows at n_packets=0, never the deny that ate the packets. Replace
it with a full ovn-sbctl --stats lflow-list, echoed so it survives in the
CI stdout log. Swap the conntrack capture to ovs-appctl dpctl/dump-conntrack
(the standalone conntrack binary is absent on the baremetal runner) and add
dpctl/dump-flows -m for kernel megaflow hit counts.
* fix: claim gateway cr-port on every reconcile miss and stop masking launch failures
ensureGatewayClaimed nudged ovn-controller recompute only once then polled
until timeout, so a fresh VPC bring-up or chassis flap that fired the single
nudge too early left the gateway chassisredirect port permanently unclaimed.
Recompute on every miss instead, mirroring ensureGuestPortDatapath.
aggregateResults and distributeInstancesSpread masked any node launch failure
(including the resulting 5-minute launch timeout) as InsufficientInstance
Capacity. Surface the real node error when one exists; keep the capacity code
only for genuine capacity shortfalls.
* feat(eks): multinode control-plane networking, IAM, and CoreDNS parity
Integrates the EKS multinode fix set for the managed-control-plane topology
so a 3-worker cluster brings up ingress/ALB and cross-node DNS the way AWS
EKS does. Squashed from feat/eks-workload-lifecycle, rebased onto dev; the
node-IAM piece (4f6ea8cf) is dropped as dev's lazy IAMProvider supersedes it.
EKS:
- Route apiserver egress (exec/logs/webhooks) through konnectivity
(apiserver-network-proxy): CP servers run konnectivity-server behind the
cluster NLB (:8132), workers run konnectivity-agent, so apiserver->pod
traffic rides the agent-initiated reverse tunnel instead of an unroutable
CP->VPC path. Server-count fan-out gives every apiserver a tunnel to every
replica (HA-correct).
- Pin k3s node-ip to the CP-VPC primary on overlay re-home; CP-overlay
flannel mesh SG rule authorized at cluster launch with ordered teardown.
- Pin VPC data NIC MTU to 1320 so flannel fits the OVN underlay.
- Pin the CoreDNS DaemonSet to workers (drop CriticalAddonsOnly + CP/master
tolerations, add anti-control-plane nodeAffinity) for AWS-EKS parity.
- Spread nodegroup workers across hosts; addon readiness falls back on
workload rollout when gvks are absent.
- elbv2: allow the proxy_protocol_v2.enabled target-group attribute.
EBS:
- Move volume attachment state to state.json so a live nbdkit VB cannot
clobber it; route mounted-volume state updates through the live VB with a
retry for in-use marks racing the mount responder; keep the boot volume
in-use across stop/crash unmount.
* fix(eks): gate orchestration on the resolved IAM ensurer, not raw deps.IAM
missingOrchestrationDeps gated on s.deps.IAM, but production wires only the
lazy deps.IAMProvider and leaves deps.IAM nil (set directly by tests only).
The gate therefore reported missing=[IAM] on every node forever, so
CreateCluster/CreateNodegroup were rejected with ServiceUnavailable even
though the IAM service was fully up via the provider.
Gate on s.iamEnsurer() instead — the same resolver (deps.IAM, else
deps.IAMProvider()) the nodegroup/role code already uses — so the protective
check still rejects a node with no IAM service but passes when the provider
supplies one. Add a regression test that sets only IAMProvider and asserts the
gate is satisfied; the existing nil-both test still rejects.
* fix(eks): let eks-token-webhook use IMDS creds when static keys absent
The CP VM launches on an instance profile, so buildK3sUserData omits
EKS_ACCESS_KEY/EKS_SECRET_KEY. loadConfig() rejected absent static keys
before eksgw.New could fall back to the IMDS SDK chain, so the webhook
aborted on boot, never wrote token-webhook.kubeconfig, and the apiserver
crashed on the missing --authentication-token-webhook-config-file file,
stalling the cluster at CREATING until it timed out to FAILED.
Drop the static-key precondition (matching the sibling eks-gateway-publish
and eks-gateway-fetch helpers); keep the genuinely-required gateway/account/
cluster gates.
* fix(eks): stop dhcpcd IPv4LL hijacking IMDS on multi-NIC control plane
The K3s control-plane system VM is dual-NIC (data ENI + mgmt NIC on br-mgmt,
which has no DHCP server). dhcpcd's no-lease IPv4LL fallback assigned a
169.254.x.x address and a 169.254.0.0/16 route on the mgmt NIC, capturing
169.254.169.254 and steering IMDS off the data NIC away from the per-tap
br-imds datapath, where the host RST it. eks-gateway-publish then failed to
sign requests and no bootstrap artifacts were ever published, so clusters sat
CREATING until they timed out to FAILED.
Bake noipv4ll into /etc/dhcpcd.conf so dhcpcd never assigns an IPv4LL address
or /16 route on a NIC that fails DHCP; real DHCP leases (workers, data ENI)
are unaffected and IMDS falls through the data NIC's default route to the
br-imds demux, matching the single-NIC path. Defense in depth: mulga-mgmt-net
deletes any default route on a NIC declared NIC<n>_DEFAULT=0.
* fix(eks): grant eks:WebhookTokenReview to the control-plane instance role
The eks-token-webhook relays `aws eks get-token` bearer tokens to the gateway
broker at POST /clusters/{name}/token-review, signing with the control-plane
VM's IMDS instance-role creds (spinifex-eks-server). That route authorizes
against eks:WebhookTokenReview, but the role's inline policy granted only
eks:PublishInternal and eks:ListInternalAddons, so the gateway 403'd every
relay and the apiserver returned 401 to all external kubectl.
Add eks:WebhookTokenReview to eksServerInlinePolicy. EnsureSystemInstanceProfile
re-asserts the inline policy on every control-plane instance-profile ensure, and
the gateway evaluates authorization per request, so existing clusters converge
without a rebuild once the shared role is updated.
* feat: add OVSDB RAFT cluster primitives to setup-ovn.sh
Add --db-cluster-local-addr / --db-cluster-remote-addr flags so a
management node can run the OVN NB/SB databases as a native OVSDB RAFT
cluster instead of a single non-clustered ovn-central. When a local addr
is given, write a shared OVN_CTL_OPTS to /etc/default/ovn-central (NB
RAFT 6643 / SB RAFT 6644) and restart the per-DB units directly, since
the ovn-central aggregator is ExecStart=/bin/true and won't restart its
children. Empty remote addr creates the cluster (init node); a set remote
addr joins it. NB/SB client set-connection runs only on the create node
(RAFT replicates it cluster-wide).
The single-node --management path is unchanged when no cluster addrs are
supplied, so dev boxes and existing single-node clusters keep working.
--ovn-remote already passes through verbatim, so a comma-separated SB
endpoint list works for compute nodes; documented in usage.
* feat: OVN client multi-endpoint failover for RAFT clusters
Accept a comma-separated OVN NB/SB endpoint list so vpcd can target a
clustered DB. config.ParseEndpoints splits the list (trimming whitespace,
dropping empties); live.go Connect passes one libovsdb WithEndpoint per
endpoint, so the client tries them in order, fails over on disconnect, and
follows RAFT leader redirects. Single-endpoint strings still yield one
endpoint, so single-node dev is unchanged.
ovn-sbctl --db= already accepts the raw comma list, so the SB CLI
consumers (discoverChassis, gateway_claim) need no change.
* feat: write clustered OVN NB/SB endpoints across the RAFT quorum
spx admin init/join now emit comma-separated OVN NB/SB endpoint lists for
the first three quorum nodes (formation.BuildOVNDBAddrs) instead of loopback
or a single leader address, so vpcd fails over across the cluster.
firstboot defers OVN bring-up to the provisioning controller when it owns
formation (SkipFormation): a standalone single-node .db would otherwise block
the controller's clustered create/join.
* test: add multinode OVN RAFT cluster + failover e2e; document clustered topology
ovn_raft_test.go asserts NB/SB cluster/status reports a 3-server quorum with
one leader on every DB node, then stops the NB leader's ovn-central and proves
an NB write still lands via a surviving endpoint.
Docs: vpc-networking gains the multi-endpoint vpcd config and a RAFT
cluster-status debugging section; network-connections lists the 6643/6644 RAFT
ports.
vpcd's libovsdb NB client was a single long-lived connection with no
reconnect logic. After the NB server restarted (host reboot, service
restart) every vpcd holding a dead/half-open socket never re-established
it, so OVN-write handlers blocked until the caller timeout and CreateVpc
returned ServerInternal intermittently (queue-distributed across vpcd).
Add WithInactivityCheck to the OVSDB client: a periodic Echo probe
detects the dead peer and triggers reconnect against the configured
endpoint, with infinite backoff so a daemon keeps retrying across a long
outage. Bound the SG subscriber handlers with a timeout context so a
stuck OVN op frees the worker instead of hanging on context.Background.
NLBs hard-rejected customer security groups and always minted a managed SG that
auto-opened listener ports, so aws-load-balancer-controller (which assigns SGs to
type=external NLBs) got a 400 and looped. Accept up to 5 customer SGs on an NLB:
when supplied they replace the managed SG on the ENIs and the caller owns the
listener-port rules (CreateListener already no-ops its auto-open when there is no
managed SG); without them the managed-SG default is unchanged. SetSecurityGroups
follows AWS create-time semantics — replace SGs on an NLB created with SGs, reject
adding them to one created without.
CreateVolume hardcoded iops to 3000, silently dropping a caller-supplied Iops,
and rejected non-gp3 VolumeType with the generic InvalidParameterValue. Honor
input.Iops within the gp3 envelope (3000 baseline, 500 IOPS/GiB, 16000 ceiling)
and return the volume-type-specific UnknownVolumeType for unsupported types.
Closes EBS CSI API gaps #3-4 (siv-380); gp3 Throughput (gap #2) deferred.
* feat: IMDS-capable gateway signer for lb-agent + EKS clients
Add internal/gwsign, a SigV4 signer that draws credentials from either
fixed static keys or the AWS SDK default credential chain. In a guest VM
the chain falls back to the in-VM IMDS endpoint, yielding scoped,
auto-rotating instance-role credentials; a session token is signed as
X-Amz-Security-Token so the gateway can verify temporary (ASIA) creds.
Wire it into the lb-agent (lbagent.Agent) and the EKS on-VM client
(internal/eksgw). When static keys are absent the clients now sign with
IMDS credentials instead of failing; when present, behaviour is
unchanged (static path). This is the client-side foundation for dropping
long-lived secrets from VM user-data.
Credentials are retrieved per request so rotated IMDS creds take effect
without an agent restart.
* feat: IMDS instance-role creds for EKS server + LB VMs (Phase B/C)
Attach a Spinifex-managed system instance profile to the k3s control-plane VM
and the elbv2 LB VM so both authenticate to the gateway with scoped, rotating
IMDS instance-role credentials instead of a long-lived static secret baked into
user-data.
- Add a shared handlers/iam.EnsureSystemInstanceProfile that find-or-creates a
role (EC2 trust) plus its instance profile and asserts an inline permission
policy. The gateway enforces per-action authz on assumed-role principals, so
the roles carry exactly the actions each VM calls:
- EKS server: eks:PublishInternal, eks:ListInternalAddons.
- LB agent: elasticloadbalancing:LBAgentHeartbeat, GetLBConfig.
- Plumb IamInstanceProfileArn through sysinstance.SystemInstanceInput and both
daemon launch paths (BootAMI via PrepareRunInstances->RunInstance; direct-boot
microVM via RunInstance), which persist it so the in-VM IMDS endpoint serves
the role.
- Drop the baked secrets when a profile is attached: omit EKS_ACCESS_KEY/
EKS_SECRET_KEY from the k3s first-boot env and LB_ACCESS_KEY/LB_SECRET_KEY from
the lb-agent fw_cfg env; relax k3s server validation to accept profile-only;
drop the EKS key requires from the first-boot and state-report scripts.
- Both verticals fall back to baked static creds when IAM is unwired, so an
absent master key never blocks a launch.
* fix: resolve IMDS instance-role creds under the instance account for system VMs
Three boot-path defects kept the LB VM's lb-agent from authenticating to the
gateway with IMDS-served instance-role creds; each produced a silent empty IMDS
role list and a heartbeat failing with "unexpected empty EC2 IMDS role list".
Found and fixed against env19.
- Build the system IAM ensurer lazily on the first system-VM launch instead of
eagerly at daemon startup, where it raced the NATS KV backend and fell back to
baked static creds permanently. ELBv2 and the EKS deps now hold an IAMProvider
resolved at launch time.
- Create the LB instance profile in the system account the LB VM runs under
(utils.GlobalAccountID), not the LB-owner account. IMDS resolves the profile
under the instance's account, so a customer-account role was invisible to it.
- Record the attached instance's owning account on the ENI (InstanceOwnerId,
mirroring AWS Attachment.InstanceOwnerId) and resolve the instance and every
IAM/STS lookup under it. The LB ENI lives in the customer account while the
system LB VM and its role live in the system account; keying the whole IMDS
chain off the ENI account left the instance invisible and the role lookup
missing. VPC leaves (subnet/VPC/SG CIDR) keep using the ENI account.
A fresh LB on env19 then authenticated end-to-end: IMDS serves a non-empty role
list plus ASIA creds, and the gateway accepts the heartbeat as an assumed-role
principal with no static keys in user-data.
* test(e2e): assert LB VM authenticates via IMDS instance-role creds
The LB suite's targets-healthy assertion already depends on the lb-agent
authenticating to the gateway, and with static keys dropped from user-data that
auth is IMDS-only. Add an explicit check that a recent gateway authentication
for the LB used an assumed-role (ASIA) principal, so a regression to the
instance-account IMDS resolution fails with a clear message instead of a
mysterious datapath timeout. The LB VM is a system-account instance invisible to
the customer AWS API, so the assertion observes the gateway journal on the
cluster nodes; it is time-bounded to avoid the stale-console-log false-fail and
skips when no node is SSH-reachable.
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
Add a second build+publish step to the EKS AMI workflow so the same
manual run also builds the ecs-agent system AMI and uploads its qcow2 +
.sha256 sidecar to R2. Reuses the shared checkout, Go, and tooling setup;
renames the workflow to cover both node images.
2026-06-30 10:52:26 +10:00
2643 changed files with 318390 additions and 44996 deletions
Daemons are <tspanfill="#F87171"font-weight="600">stateless</tspan> — scale horizontally by starting more.
No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.
</text>
</g>
</svg>
<svgxmlns="http://www.w3.org/2000/svg"width="1020"height="770"viewBox="0 0 1020 770"role="img"aria-label="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus; stateless ec2/ebs/s3/vpc/iam daemons answer and drive local backends (QEMU/KVM, Viperblock, Predastore, OVN/OVS, identity store).">
<textx="48.0"y="697.0"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"><tspanfill="#EE4B12"font-weight="700">Queue groups</tspan><tspanfill="#5B636C"> load-balance ops across daemon replicas. </tspan><tspanfill="#EE4B12"font-weight="700">Fan-out</tspan><tspanfill="#5B636C"> (no queue group) dispatches describe calls to every node.</tspan></text>
<textx="48.0"y="720.0"font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif"font-size="12"fill="#5B636C"font-weight="400"><tspanfill="#5B636C">Daemons are </tspan><tspanfill="#EE4B12"font-weight="700">stateless</tspan><tspanfill="#5B636C">: scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Systemd units, a NATS cluster, and your hardware.</tspan></text>
<textx="450"y="58"font-family="'Segoe UI', system-ui, sans-serif"font-size="20"font-weight="600"fill="#f5f5f5"text-anchor="middle">AWS-style workflows, on your own stack</text>
<svgxmlns="http://www.w3.org/2000/svg"width="1000"height="664"viewBox="0 0 1000 664"role="img"aria-label="The Spinifex stack: applications and AWS tooling on top, the Spinifex AWS-compatible service layer, running on standard Linux on CPU/GPU hardware, deployable to Neocloud, on-prem or edge.">
<textx="136.0"y="249.8"font-family="ui-monospace, 'SFMono-Regular', Menlo, Consolas, monospace"font-size="11"fill="#9AA1A8"font-weight="400"text-anchor="start">Runs the same on a single node, cluster, edge, or air-gapped rack.</text>
echo"::error::${label} names suite '${suite}', which is not eligible for that topology (eligible: ${eligible}). Fix the list in .github/scripts/suite-sets.sh." >&2
description:"OTEL_RESOURCE_ATTRIBUTES value for this run, e.g. ci.run_id=123,ci.cell=20 (caller-specific, so it's an input rather than a workflow-level default)"
required:false
type:string
default:""
permissions:
contents:read
@@ -16,6 +21,10 @@ permissions:
env:
GIT_BRANCH:${{ github.ref_name }}
# Same central ES/APM sink as the other four e2e workflows. Bare-metal had
# no default here before — this cell's telemetry never reached the sink at
# all (not just missing ci.cell), see bm-bootstrap.sh's telemetry.env step.
# Manual-work mutex gate — see setup note 4 above. Checked BEFORE any
# SSH to wattle so a paused run touches the box zero times.
- name:Manual-work mutex gate
id:gate
env:
REPO_VAR_PAUSED:${{ vars.CI_GPU_PAUSED }}
run:|
set -euo pipefail
PAUSED=false
if [ -f "$HOME/.ci-gpu-paused" ]; then
PAUSED=true
echo "::warning::ci-gpu runner paused via ~/.ci-gpu-paused sentinel on the runner — skipping this run (manual GPU work in progress on wattle). Remove the file to resume CI."
fi
if [ "$(echo "${REPO_VAR_PAUSED:-}" | tr '[:upper:]' '[:lower:]')" = "true" ]; then
PAUSED=true
echo "::warning::ci-gpu paused via repo/environment variable CI_GPU_PAUSED=true — skipping this run."
fi
echo "paused=${PAUSED}" >> "$GITHUB_OUTPUT"
- name:Paused — nothing to do
if:steps.gate.outputs.paused == 'true'
run:echo "ci-gpu is paused; job exits cleanly without touching wattle. See gate step above for the reason."
echo "::error::GPU e2e suite failed (rc=${{ steps.e2e.outputs.rc }}) — see analysis above; raw ES/APM traces at ${SPINIFEX_OTLP_ENDPOINT} (ci.run_id=${{ github.run_id }})"
$(_Q)LOG_IGNORE=1AWS_MODEL_CONFORMANCE_MODE=$(AWS_MODEL_CONFORMANCE_MODE)AWS_MODEL_CONFORMANCE_REPORT=$(AWS_MODEL_CONFORMANCE_REPORT) go test -count=1 -tags=integration -timeout 60s ./tests/integration/... $(_RACEQ)
<img src=".github/assets/banner.svg" alt="Spinifex: the open, AWS-compatible cloud you run yourself. EC2, EBS, S3, VPC, and IAM on your infrastructure, at the edge, or on a Neocloud." width="900">
# Spinifex: An Open Source AWS-Compatible Stack for Bare-Metal, Edge, and On-Prem Deployments
# Spinifex: the open, AWS-compatible cloud you run yourself
Spinifex developed by [Mulga Defense Corporation](https://mulgadc.com) is an open source infrastructure platform that brings the core services of AWS—like EC2, VPC, EBS, and S3—to environments where running in the cloud isn't an option. Whether you're deploying to edge sites, private data-centers, or need to operate in low-connectivity or highly contested environments, Spinifex gives you AWS-style workflows on your own hardware.
Spinifex, built by [Mulga](https://mulgadc.com), recreates the AWS services your software already uses (EC2, EBS, S3, VPC, IAM) on infrastructure you control. Move workloads off the hyperscalers, run AI at a fraction of the cost, or take the cloud to the edge, without rewriting a line. The exact build you ship to AWS, Spinifex serves on your own hardware: same APIs, same Terraform, same SDKs.
## What is Spinifex?
Spinifex replicates essential AWS primitives—virtual machines, block volumes, and object storage—using lightweight, self-contained components that are easy to deploy and integrate.
Spinifex is an open-source infrastructureplatform that recreates the AWS service surface (EC2, EBS, S3, VPC, IAM) on bare-metal, edge, and on-premises environments. Run cloud-native software without a hyperscaler.
It's designed for developers and operators who need:
Most AWS alternatives wrap an API and call it a cloud. We rebuilt the engineering underneath: distributed object storage with erasure coding, block storage built to survive failure, and compute on bare metal. That depth is the reason your software runs unchanged, instead of rewritten.
- Cloud-like infrastructure without the cloud
- Minimal dependencies, full control
- Drop-in compatibility with tools like the AWS CLI, SDKs, and Terraform
- A secure cloud environment you control and own the entire hardware, network and software stack
It's built for teams that need:
You can run Spinifex on a few servers in a rack, a field site, or anywhere centralized cloud services aren't feasible.
- The AWS tooling they already use (AWS CLI, SDKs, Terraform), with nothing to rewrite
- Full control of the stack: your hardware, your network, your data, your keys
- A cloud that keeps running offline, through disconnection, with no external control plane
- An open core (AGPL-3.0), auditable and yours to keep, with no lock-in
You change *where* your software runs, not *what* it is.
## The Platform
<p align="center">
<img src=".github/assets/platform.svg" alt="The Mulga Platform — Hardware to Applications" width="900">
<img src=".github/assets/platform.svg" alt="The Spinifex stack: your apps and AWS tooling on top, the Spinifex AWS-compatible service layer, running on standard Linux on CPU and GPU hardware, deployable to Neocloud, on-premise, or the edge." width="900">
</p>
From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
## Three Ways To Deploy
One AWS-compatible surface, three deployment shapes. Pick the one that matches your reality; the platform on top is the same.
### Neocloud
Lift and shift onto a partner Neocloud. Move workloads off the hyperscalers and onto our Neocloud partner ecosystem without rewriting them. GPU capacity (H100 / H200 / B200), cheaper, available now. Change an endpoint, keep the software.
### On-premise
Bring your own hardware and host Spinifex in your own data centre. A real multi-node HA cluster integrated with your storage and networking, with full control of stack, data, and jurisdiction. A predictable bill, no egress surprises, an auditable open-source core.
### Edge
Cloud where the cloud can't reach. Air-gapped sites, vehicles, vessels, factories, and clinics. Compute next to the data, running through disconnection, on hardware you own. The same AWS APIs your software already uses.
## AWS Compatibility
Speak the AWS API surface, natively. The AWS SDKs, AWS CLI, and Terraform: everything you deploy on AWS deploys on Spinifex unchanged. At the edge, on-premise, or on a partner Neocloud.
| Service | What it is | Status |
| --- | --- | --- |
| **EC2** | Compute | Available |
| **EBS** | Block Storage | Available |
| **S3** | Object Storage | Available |
| **VPC** | Networking | Available |
| **IAM** | Identity & Auth | Available |
| **ALB / NLB** | Load Balancers | Available |
| **EKS** | Kubernetes | Available |
| **ECR** | Container Registry | Available |
| **ECS** | Container Service | Available |
| **RDS** | Databases | Available |
| **Bedrock** | AI Deployment | Q3 2026 |
Roadmap items ship under the same AWS API surface. Code written for AWS today keeps working the moment they land. [Track what's shipped in the release notes](https://github.com/mulgadc/spinifex/releases).
## Core Components
### Spinifex (Compute Service – EC2 Alternative)
@@ -72,26 +112,34 @@ Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs
- Multipart uploads, streaming reads/writes
- Data redundancy with Reed-Solomon encoding
### Northstar (Authoritative DNS)
[Northstar](https://github.com/mulgadc/northstar) is a lightweight authoritative DNS server that handles both internal service discovery and public-facing DNS for a Spinifex cluster. Zones are human-readable TOML files, served from the local filesystem or synced from Predastore, keeping DNS configuration in the same object storage as the rest of your infrastructure.
- Authoritative DNS over UDP, TCP, and DNS-over-TLS
- Service discovery for cluster components via SRV records
- Zone files stored locally or in S3-compatible storage
<img src=".github/assets/architecture.svg" alt="Spinifex message-driven architecture: standard AWS clients call the AWS Gateway over HTTPS with SigV4, which publishes to a NATS bus answered by stateless ec2/ebs/s3/vpc/iam daemons over local backends." width="900">
</p>
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless — scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
Every AWS API call is authenticated at the gateway, published to a NATS subject, and answered by whichever daemon claims it. Daemons are stateless: scale horizontally by starting more. No etcd, no Kubernetes, no external control plane. Just systemd units, a NATS cluster, and your hardware. Deep dive: **[`docs/DESIGN.md`](docs/DESIGN.md)**.
## Key Features
AWS-Compatible Interfaces – Provision infrastructure with awscli, Terraform, or SDKs you already use.
-Designed for Control – Run on your own terms, whether that's in a datacenter, remote site, or sensitive environment.
-Minimal Dependencies – Each service is standalone and avoids complex orchestration layers.
-Works Offline – No reliance on centralized cloud services or external networks.
- Open Source – Licensed under AGPL3.0. Fork it, extend it, or deploy it as-is.
- **AWS-compatible APIs.** Use the AWS CLI, SDKs, and Terraform you already know. Repoint your endpoint and ship; nothing to rewrite.
- **Zero cloud dependency.** Runs entirely on your hardware. No phone-home, no control plane, no external authority. Works fully offline.
-**Bare-metal compute.** QEMU-based with direct hardware access. No hypervisor tax, no abstraction overhead.
-**Built-in storage.** Block and object storage included: NVMe caching, Reed-Solomon erasure coding, and replication out of the box.
-**Edge-first architecture.** Designed for disconnected, contested, and resource-constrained environments from day one.
-**Open core, no lock-in.** AGPL-3.0 with a commercial option. Inspect it, modify it, deploy it. The platform is yours to keep.
## Installation
Installation requires an Ubuntu / Debian system. See the detailed documentation at [docs.mulgadc.com](https://docs.mulgadc.com) for maintaining and installing Spinifex.
Installation requires an Ubuntu 26.04 or Debian 13 system. See the detailed documentation at [docs.mulgadc.com](https://docs.mulgadc.com) for installing Spinifex.
### Bare Metal ISO
@@ -101,16 +149,11 @@ The recommended installation is a [bootable x86 installer](https://iso.mulgadc.c
curl -fLO https://iso.mulgadc.com/spinifex.iso
```
Follow the [USB install guide](https://docs.mulgadc.com/docs/install-usb) to write the ISO to USB and install on your hardware. The install guide walks through the full process.
Follow the [USB install guide](https://docs.mulgadc.com/docs/install-usb) to write the ISO to USB and install on your hardware.
### Single Node Install
The installation is straightforward to set up and running on a single node for testing purposes. Debian 13 is currently supported, additional Linux distributions are on the immediate roadmap.
>*Prerequisite:* Linux bridge for networking.
Spinifex requires a Linux bridge configured on the host for VM networking. See the [single-node install guide](https://docs.mulgadc.com/docs/install#prerequisites) prerequisites for setup details.
>*Prerequisite:* Spinifex requires a Linux bridge configured on the host for VM networking. See the [single-node install guide](https://docs.mulgadc.com/docs/install#prerequisites) for setup details.
- **[Northstar](https://github.com/mulgadc/northstar)** - Authoritative DNS server
Each component can be developed independently. See component-specific documentation for focused development guides.
@@ -149,10 +193,6 @@ Spinifex ships with a built-in web console — an optional alternative to the AW
The console is served by each node on port `3000` over TLS, and becomes available as soon as `spinifex.target` is up:
```bash
open https://YOUR_NODE_IP:3000
```
- **Same API, different surface.** Every action in the UI is the same AWS SigV4 call the CLI makes — so RBAC, audit trails, and IAM policies apply uniformly.
- **Single sign-on against your AWS credentials.** Log in with the access keys from `~/.aws/credentials` on the node where Spinifex is installed — no separate user database.
- **Self-hosted, works offline.** The UI is embedded in the Spinifex binary and served from the node itself. No external CDN, no analytics calls, no cloud dependency.
@@ -192,6 +232,10 @@ While Spinifex is architected and implemented by experienced engineers, we lever
This hybrid approach ensures Spinifex benefits from both proven engineering expertise and modern development acceleration, while maintaining the quality and reliability standards required for production infrastructure.
## Trademarks
"AWS", "Amazon Web Services", and all related service names (EC2, EBS, S3, VPC, IAM, and others) are trademarks of Amazon.com, Inc. or its affiliates. Mulga Defense Corporation and Spinifex are independent and not affiliated with, endorsed by, or sponsored by Amazon.com, Inc. or Amazon Web Services, Inc. References to AWS services describe interoperability and compatibility only.
## License
Spinifex is open source under the [GNU Affero General Public License v3.0](LICENSE). You're free to use, modify, and deploy it anywhere you need reliable infrastructure without depending on centralized cloud platforms.
# Recovery: drop the history so a later failure gets the full
# restart budget again, and re-arm the escalation message.
rm -f "$RESTART_COUNT_FILE""$ESCALATED_STAMP"
exit0
fi
if["$wrc" -eq 2];then
echo"nats-js-watchdog: js-probe inconclusive (config, connect, bucket-open, or spx itself failed); not evidence of a JetStream failure, taking no action" >&2
exit0
fi
failure="JetStream accepts no writes (canary round-trip failed) while $HEALTH_URL reports healthy"
else
failure="JetStream unhealthy at $HEALTH_URL"
fi
if["$i" -lt "$SAMPLES"];then
sleep "$SAMPLE_GAP"
@@ -46,5 +167,24 @@ while [ "$i" -le "$SAMPLES" ]; do
echo"nats-js-watchdog: already restarted $SERVICE$restart_count time(s) without recovery; giving up rather than looping (clear $RESTART_COUNT_FILE to re-arm)" >&2
fi
exit0
fi
echo"nats-js-watchdog: $failure on $SAMPLES consecutive probes; restarting $SERVICE" >&2
echo"nats-js-watchdog: restart $restart_count of $RESTART_ESCALATE_AT before escalation; if $SERVICE is still failing after this, the watchdog will stop acting until $RESTART_COUNT_FILE is cleared" >&2
retries:=flag.Int("retries",3,"Number of DORA attempts before giving up")
verbose:=flag.Bool("verbose",false,"Print full DHCP packet contents")
broadcast:=flag.Bool("broadcast",false,"Set the BOOTP broadcast flag, as vpcd's Acquire path does, so replies come back to ff:ff:ff:ff:ff:ff rather than unicast to the bridge MAC")
flag.Parse()
// Resolve the hardware address to use.
@@ -44,20 +46,25 @@ func main() {
switch{
case*spinifexID!="":
hwAddr,err=net.ParseMAC(generateMAC(*spinifexID))
// vpcd's own derivation, so the probe leases under the MAC the daemon
// would have used for this id.
hwAddr,err=dhcp.DeriveMAC(*spinifexID)
iferr!=nil{
log.Fatalf("generateMAC(%q): %v",*spinifexID,err)
slog.Error("Could not derive MAC","spinifex-id",*spinifexID,"error",err)
os.Exit(1)
}
fmt.Printf("spinifex virtual MAC for %q: %s\n",*spinifexID,hwAddr)
case*macStr!="":
hwAddr,err=net.ParseMAC(*macStr)
iferr!=nil{
log.Fatalf("parse --mac %q: %v",*macStr,err)
slog.Error("Could not parse --mac","mac",*macStr,"error",err)
os.Exit(1)
}
default:
hwAddr,err=ifaceMAC(*iface)
iferr!=nil{
log.Fatalf("get MAC for %s: %v",*iface,err)
slog.Error("Could not get MAC for interface","iface",*iface,"error",err)
os.Exit(1)
}
fmt.Printf("using real interface MAC: %s\n",hwAddr)
}
@@ -83,7 +90,7 @@ func main() {
if*promisc{
iferr:=setPromisc(*iface,true);err!=nil{
log.Printf("WARNING: failed to set promisc on %s: %v",*iface,err)
slog.Warn("Failed to set promisc on interface","iface",*iface,"error",err)
// (awsvpc task ENI). Empty keeps the container in the host (VM) netns —
// bridge/host mode behaviour.
NetnsPathstring
// GPU is the whole-GPU count requested for this container.
GPUint
// GPUIDs are the ledger-pinned device UUIDs assigned to this container, one
// per GPU requested. Empty for non-GPU containers or if pinning fell short.
// The runner injects these as CDI devices into the OCI spec.
GPUIDs[]string
}
// RunStatus is a finished container's outcome.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Julian Sommer
