Every non-success on the daemon's NATS path recorded as error, so an
unknown instance id, a bad parameter and a genuine server fault were one
series and a daemon error rate tracked caller behaviour as much as
daemon health. The same conflation at the log level made an expected
fan-out miss write an ERROR line on every node that was never going to
answer it: on a four-node cluster each IMDS metadata fetch by a booting
guest produced three that meant nothing.
Failures are now classified by the status their AWS error code maps to,
through the same function the HTTP middleware calls, so the two paths
cannot drift apart. A caller mistake records client_error and logs at
WARN; a server fault records error and stays at ERROR. The rule is now
that ERROR means the daemon failed, which is broader than the fan-out
case and subsumes it.
Separately, the per-instance command path was subscribed raw and passed
through no metrics wrapper at all, so start, stop, terminate, reboot and
the volume and ENI hot-plug commands produced no data points. They are
now counted under an action naming the command rather than the subject,
which carries the instance id. The drain runs off-thread and records its
own point once the flush finishes.
The gateway derived each request's IAM action prefix from the wire-service
name, so bedrock-runtime, bedrock-agent and bedrock-agent-runtime calls were
checked as bedrock-runtime:ConverseStream, bedrock-agent-runtime:Retrieve, etc.
AWS namespaces every Bedrock action under bedrock:, so an AWS-shaped scoped
policy never matched and only a wildcard admitted the call.
Collapse the Bedrock family to the shared bedrock: namespace in IAMAction, the
single choke point for policy decisions; recordResolvedAction keeps the wire
name for telemetry.
Guest series from qmp-collector carry the owning account as the
CloudWatch dimension account_id. Traces and logs carry the same account
as aws.account_id, and on metrics-apm* the bare account_id name is
already taken by the NATS global account the nats service reports. A
dashboard control built on the trace field therefore cannot reach guest
metrics, which is why the guest panels sit apart from the account
drill-down.
The dimension name is an AWS-facing contract, so it is not renamed. The
OTel bridge mirrors it instead: guest series now export aws.account_id
alongside account_id, and a series with no account exports neither a
blank value nor a placeholder.
ebsRequestWithTrace started its span from context.Background(), so every
ebs.mount, ebs.unmount and ebs.delete rooted a trace of its own. Block storage
work was therefore unreachable from the request that caused it: viperblockd
recorded 10,209 transactions in 24h and not one could be tied to a caller or an
account.
VolumeMounter now takes the caller's context. Launch and hot attach/detach have
one to give. Shutdown, crash recovery and teardown genuinely do not — they
answer to no request — so they pass a background context, which is an honest
root rather than a missing one.
The owning account rides both the span and the NATS header, because
viperblockd's consumer span reads that header to attribute itself. It comes off
the instance where there is one, and is passed explicitly for a lone volume
request, which carries no instance to read it from. An absent account is left
off rather than blanked: recovery and teardown act for nobody, and crediting
that work to whoever happens to be admin would read as tenant activity.
http.Server.WriteTimeout is a total deadline rather than an idle one, so any
value below the RDS command channel's 20s long poll kills every poll mid-flight
and strands agents on a channel no command can reach. Nothing recorded that the
omission was load-bearing, leaving it one hardening pass away from removal.
The sandbox is one cluster with tenancy at the account level, so every
question about it starts with which account. Today the account is on the
gateway's own spans and nowhere else: of 187,317 daemon transactions in 24
hours, none name an account, and the same holds for vpcd, viperblockd and
northstar. A tenant's request is attributable at the front door and anonymous
everywhere it does its work.
The account already crosses NATS on every request as X-Account-ID, so the
spans can name it without new plumbing. StartConsumerSpan reads the header and
startProducerSpan takes the account the caller sent for, which covers every
handler built on ServeNATSRequest and every fan-out through Gather.
Work with no caller - a sweep, a reconciler, a node-to-node probe - is left
unattributed rather than credited to whoever happens to be admin.
Merges `feat/ebs-export-tunables` (83 commits) into dev. The branch could not be rebase-merged because it carries five merges from dev, so it lands as a squash.
## What is in it
- **EBS export tunables** and the nbdkit/viperblock export path they configure.
- **Teardown reapers and backoff** for S3 buckets, load balancers, gateways, spot requests, certificates, endpoints, elastic IPs and network interfaces, so a dependent that keeps failing is backed off rather than retried hot.
- **Retry and tolerance paths**: ECS IAM service init, ECR signing keys deleted between the list and the read, an unreadable signing key at awsgw startup, and an ECS instance-profile policy attach converging on a stale read.
- **Key pair validation reads the material by name** instead of listing the account, closing the `InvalidKeyPair.NotFound` race that a single-replica LIST could produce.
- **Concurrency and fan-out**: concurrent metadata document fetches, bounded metadata reads for describes, and the node-discovery timeout removed from every fan-out call.
- **OVN correctness against the configured NB cluster**: the flows barrier, nexthop MAC binding, and stopping a compute node from running the package's standalone database.
- **nats-js-watchdog can re-arm itself.** The escalation check ran above the probe loop while the only line that cleared the counter sat below it, so a latched watchdog could never observe a recovery. It now probes first and decides second, and says so once per latch rather than once per tick.
- **CI**: 19 commits of nightly and build-pipeline work.
## Verification
`make preflight` green locally. Build and Test, Integration Tests, Race Detection, and Lint and Security all green on `c2da0385`.
## Known red, not introduced here
Two nightly cells fail on this branch and are pre-existing:
- **cell-10** — a partial OVN IPsec enable leaves `NB_Global ipsec=true` asserted cluster-wide while two of three chassis failed their local reconcile, black-holing cross-chassis guest traffic for roughly 5.5 minutes while every health signal stays green.
- **cell-19** — a total layer-3 egress blackhole below a correctly and promptly programmed NAT/EIP datapath, aggravated by a natuplink probe whose retry budget outlives the test's own wait, so it can never emit its FAIL verdict.
Both are filed separately and should be weighed before a release is cut.
The lockout fires after 10 auth failures from an address in 60 seconds and
then answers RequestLimitExceeded to everything from that address. A client
repeating one hopeless request reaches the threshold as fast as an attacker
working through a keyspace, and 503 tells it to retry later - advice it can
never act on, because the fault is a credential that will never resolve. On
wattle four guests holding deleted STS session credentials produced 53,392
rejected requests and 69,820 503s in a day, and could not recover by fixing
their credentials because the lockout is on the address, not the principal.
RecordFailure now takes a fingerprint identifying the attempt, and the window
holds one entry per distinct fingerprint: a repeat refreshes its timestamp
instead of adding to the count. Whatever an attacker must vary is what the
fingerprint carries - the key id, the presented signature, a digest of the
session token - so guessing is throttled exactly as before while one dead
credential presented 14,000 times stays one fault.
A request rejected before its key id is parsed has no identity to protect, so
malformed envelopes and skewed timestamps still count every occurrence.
An unresolvable instance role answers an empty 200 on the
security-credentials list and a 404 on the credential itself. An SDK reads
that as "no EC2 IMDS role found" and keeps signing with whatever it already
holds, so a guest whose credentials have expired retries forever and never
recovers. On wattle four guests did this for five days: 42,517 role-list
requests, no credential fetches, and not one line anywhere saying why.
The absence was invisible because three layers returned a silent nil -
DescribeInstances missing the instance, no instance profile ARN on the
record, and a profile IAM no longer has. profileFor now reports which of
those it hit, and the two credential paths warn on it, throttled to one line
per instance and reason every five minutes so a stuck fleet stays legible at
that request volume. A role-name mismatch is the guest's own doing and stays
silent.
96.7% of spinifex-daemon request points carried no outcome. natsMetricsHandler
recorded around a plain nats.MsgHandler, which writes its own reply and returns
nothing, so the wrapper had nothing to observe and passed an empty outcome. Only
the /health HTTP path, which goes through the OTel HTTP middleware, ever
reported one.
The handler type now reports how it answered:
type natsHandler func(*nats.Msg) string
Making it a distinct type means the compiler enumerates every subscription and
every handler, rather than a correlation map guessing at responders that reply
after the wrapper returns. Four outcomes are defined: success, error, skipped
for a broadcast handler that declines because the request names another node,
and a deferred sentinel for handlers that answer from a goroutine and record
their own point once the work finishes — timing the wrapper would have reported
a five-minute system launch as an instant success.
utils.ServeNATSRequest and ServeNATSRequestCtx now return whether they replied
with a result, so handlers built on them report an outcome without restating the
error handling. Adding the return value leaves existing callers untouched.
Also measures the per-reservation launch subject, which was subscribed raw and
emitted no request points at all. It records under the subject root because the
reservation id is unbounded.
awsgw request telemetry said POST / and nothing else: no AWS action, no caller,
no reason. On wattle that left 69,820 responses with status 503 in a day
unattributable from the request docs — the cause was only findable by leaving
them for separate auth-failure log lines, which do carry sourceIP and
accessKeyID.
The request logger is the outermost middleware, so a context derived downstream
never reaches it. A small mutable record now rides the request: the auth path
writes identity as it resolves it, writeSigV4Error writes the verdict, and the
action enricher writes the action. The logger and the server span both read it,
so a trace search for failing requests answers from where, doing what and
rejected why without a join.
The record holds an access key id and an error code and nothing else that could
hold a credential; a test asserts the field set so that stays true.
vpcd and spinifex-ui never called otelsetup.SetRequestAction, so rpc_method on
their request metrics stayed at the bare HTTP verb the middleware sets. Over 24h
vpcd had exactly two action values, GET and PUT, which left IMDS countable in
aggregate but made /latest/api/token indistinguishable from a credential fetch
at /latest/meta-data/iam/security-credentials/<role>. Guests depend on IMDS for
credentials, so a failure there is user-visible and was not attributable.
Both namers map to a closed set with a single catch-all, because a guest
controls the request path and an unbounded name would put one metric series
behind every invented path. The role name, the interface MAC and the public-key
index are stripped for the same reason, and spinifex-ui names the static and SPA
branches rather than the content-hashed asset filename.
The middleware also treated every 4xx as a success, which is why vpcd and
spinifex-ui reported zero errors over 24h despite IMDS answering 401 for a
missing IMDSv2 token and 404 for an unresolvable ENI. Those now record
client_error. error keeps meaning 5xx so existing error-rate panels are
unchanged, and a 4xx still does not mark the span failed.
The ecs-agent classified only *.dkr.ecr.* hosts as ECR, so a registry
advertised as the gateway's bare host/IP pulled anonymously and 401'd.
Recognise the agent's own gateway host as ECR and mint a token for it.
The instance role — the pull's fallback credential when a task carries no
execution role — granted ecs:* only; add ECR read so the pull authorizes.
* docs: repoint network invariants doc comment at docs/adr/
The region-local ADR set moved from docs/development/proposals/ to
docs/adr/ on acceptance, and 0006 kept its filename but the directory
changed.
* docs: repoint network invariants doc comment at the proposal path
The region-local ADR set was never accepted, so it lives under
docs/development/proposals/region-local-architecture/ rather than
docs/adr/.
---------
Co-authored-by: joshsiv-mulga <engineering@mulgadc.com>
A release for an IP with no tracked lease was answered with a plain error. The
comment at the responder assumed callers log-and-continue, so failing loudly
would surface the strand without wedging teardown. That assumption does not
hold: TerminatedTeardownReaper marks TeardownNAT failed, TeardownComplete stays
false, and the record is re-driven every sweep. No retry can make a deleted
lease reappear, so it never converges.
It also disables its own backstop. The terminated bucket has a 1h TTL meant to
bound the record regardless, but each sweep persists the advanced marks and that
write refreshes the TTL, so the record outlives the very thing meant to catch
it. Measured across four hosts: seventeen stranded addresses, each retried on a
fixed 120s cadence, indefinitely.
Keep the loud failure and make it terminal. The release reply gains a
machine-readable code, the client maps it to ErrLeaseNotTracked, and
ReleasePublicIP treats that as done after recording the leak, since the local
pool slot is already free and the upstream lease cannot be released without a
lease record. Teardown then completes and the record purges.
The strand is no longer silent: mulga.resource.leaked counts it by resource
class and the address is logged at error level, so a leaked address is
queryable rather than inferred from a repeating warning.
Go stores test results in the build cache alongside compiled objects, so now that the cache persists across runs an unchanged package replays its previous result instead of executing.
That is what makes a local run fast and exactly what a merge gate should not do: a green check would no longer mean the suite ran against that commit.
Add a GOTESTFLAGS hook, empty by default so local runs keep reusing results, and pass -count=1 from the two CI steps that gate a PR.
Compilation is still cached either way, so this costs no build time.
setup-go only writes its cache on a key miss, and keyed it on spinifex/go.sum alone.
Once a run saved that key every later run logged "Cache hit ... not saving cache", so the cache stayed frozen at whatever the first job to finish happened to have compiled and nothing built afterwards was ever stored.
All four jobs also shared that one key, so the job that won the save race decided what the other three restored.
The race job was handed artifacts built without -race, and the build job artifacts built without GOFIPS140, neither of which yields a single usable entry.
Manage both caches directly instead: modules keyed on all five go.sum files so the sibling modules stop re-downloading, and build artifacts keyed per job with a per-run suffix so each flavour keeps its own entry and refreshes it every run.
GOFIPS140 is a global build-config change: it alters every package's action ID, stdlib included.
A target that omits it therefore shares no build-cache entry with one that sets it, and the two flavours cost roughly a full cold build to switch between.
Seven targets set it inline while eight go invocations did not, so test-integration, test-harness, test-actions, test-segscan-oracle, bench and the go run tools all built the non-FIPS flavour.
In CI this is load-bearing: the integration job wins the shared setup-go cache-save race, stores a non-FIPS cache, and every FIPS build in the other jobs then misses on every lookup.
Export it once so the two cannot drift apart again.
The case seeded no config, so loadLocalConfig fell back to the host's
installed spinifex.toml. It passed on a provisioned dev box and failed on
a bare runner, where the region pre-flight fires before svc.Start().
* feat(ebs): make nbdkit thread count and per-volume read cache configurable
Both numbers that set a volume's export concurrency and memory were
hard-coded: nbdkit ran on its own default of 16 threads per connection
because no -t was ever passed, and the read cache was a 128 MiB literal in
the mount path. Neither could be changed without a recompile, and neither
was visible in the logs or the process argv.
Add default_threads and cache_size_mb to [nodes.<node>.ebs]. Defaults are
unchanged, so a config that does not set them behaves exactly as before.
CacheSizeMB is a pointer so an explicit 0 (cache disabled) stays
distinguishable from an unset key, matching how ViperblockConfig treats its
own toggles. -efi volumes stay uncached regardless.
Validation is a range check only. Both are operator tunables applied per
nbdkit process, and there is one of those per volume, so the host cost is
the value times the volume count -- left for the operator to weigh rather
than second-guessed here.
* feat(security): arm the firewall by install path and sync the clock at firstboot
The host firewall armed itself from a code default, so a curl-to-bash install
onto an existing Debian or Ubuntu machine applied a default-deny INPUT policy
seconds after bootstrap. Anything the operator was already running that is not
on the allow list stopped accepting connections, with nothing logged and no
config key to set beforehand — no template writes firewall_enabled, so every
install fell through to the default.
Arming is now the install path's decision. setup.sh takes --firewall=on|off and
records it in /etc/spinifex/firewall/mode, defaulting on under ISO_BUILD and off
everywhere else: the ISO is an appliance on hardware we define, a package install
is a guest on a machine that was already doing something.
firewall_enabled becomes a three-state *bool so "unset" and "explicitly false"
stay distinguishable. Flipping the default to false instead would have torn down
a working policy on every existing node, because false is an instruction to
disable rather than an absence. Resolution is config, then mode file, then on, so
a node installed before any of this keeps what it has.
The ISO had the opposite problem. setup.sh is the only place that enables
spinifex-firewall.service and it returns before that line under ISO_BUILD, with
no systemd in the chroot to talk to. Nothing else enabled it, so after a reboot
the ruleset — runtime state — was gone, and ReconcileFirewall short-circuited on
an unchanged peer file and never put it back. An ISO node was firewalled until
its first reboot and open permanently after. firstboot now enables the unit, and
the reconcile checks the table is loaded rather than treating an unchanged peer
set as proof the policy is applied.
Two policy changes: 443 is held for the nginx TLS edge so that rollout needs no
per-node firewall change, and the SSH port is read from sshd_config and its
Include drop-ins instead of being hardcoded to 22 — a host on Port 2222 was
allowed exactly one port that was not the one it uses, and ct state established
kept the operator's session alive until they disconnected. 53 stays open on every
node, now with a comment saying so: northstar serves public authoritative DNS.
Clock: firstboot steps and waits for sync before spx admin init mints
credentials, since a certificate issued against a wrong RTC carries a NotBefore
that outlives the correction. Bounded to 60s and non-fatal so an air-gapped
install still completes, and once is enough because rtcsync writes the corrected
time back. A chrony drop-in replaces Debian's makestep 1 3, which stops stepping
after three updates and then slews at roughly 7 seconds a day — a node that
burned those before the network came up never visibly converged.
* fix(security): let a node join a cluster whose firewall is already armed
Expanding a single-node ISO install into a cluster could not succeed. Two
independent blockers, both from the same root: the peer set is derived from
converged cluster state, but the ports needed to reach convergence are gated
on that same set.
The formation port is peer-scoped, and a node dialling in to join is not a
peer yet, so the handshake was dropped rather than refused and the joiner
retried in silence until it timed out. It cannot simply move to the public
rule: outside formation the same port carries the daemon cluster manager,
whose /health and /local/* routes report node topology, service inventory and
running instances unauthenticated. So it is opened for the formation window
only, by spx admin init, and closed again on every exit path.
The reconcile then deadlocked. Holding out for the full chassis list is right
on a node with no policy loaded, but on one armed for a smaller cluster the
ports a joining node needs in order to register its chassis are exactly the
ones the stale set blocks, so the list can never complete. Widen the peer set
on the tunnel set already loaded, and report the partial state so the caller
retries sooner than the steady-state interval. A converged cluster whose
Southbound DB is briefly unreadable is still left untouched.
* docs(install): explain the firewall around cluster formation
An ISO-installed server is a cluster of one, and its firewall admits internal
cluster traffic only from members it recognises. Forming a cluster out of such
servers therefore needs the firewall off first: step 3 of the multi-node guide
has each server dial node 1's OVN database on the raft ports, and those are
peer-scoped like everything else on the cluster plane. Neither guide said so,
and the failure is a silent drop that looks like a network fault.
Both guides now walk the off / form / on sequence, and the multi-node guide
gains a reference section covering which ports are public, how arming differs
between the ISO and the binary installer, and how to verify membership landed.
Re-enable the boot-time unit whenever the daemon arms the policy. Disabling
the firewall to form a cluster also disables that unit, so re-arming used to
look complete while leaving the node bare after its next reboot.
* fix: make the ISO-to-cluster conversion path work end to end
Forming a cluster from four ISO-imaged servers surfaced a run of bugs that
share a starting state CI never exercises: the ISO delivers an initialized,
armed, running node, while the conversion path was written against the binary
installer's unconfigured one.
- install-node.sh never disarmed the host firewall, so nodes 2 and 3 could not
dial the first node on the OVN raft ports and formation could not proceed. It
now owns the whole lifecycle: disarm, form, re-arm, then verify the loaded
policy against the expected peer set.
- spx admin init left the admin access key and secret empty on the preserve
path. The pair cannot be read back, and the IAM store is rebuilt by formation,
so the empty strings reached every joiner's bootstrap.json and crash-looped
awsgw. It is reissued instead, and IAM seeding now skips an empty admin block
rather than taking the gateway down on every restart.
- node-reset.sh left /etc/default/ovn-central behind. Raft membership lives in
that file, not in the databases the reset deletes, so ovn-central recreated
the database as a cluster member dialling a peer that had itself been wiped.
- node-reset.sh also destroyed the installed firewall policy, which nothing
regenerates. set-peers then failed on every attempt and the node stayed open
permanently and silently. The arming mode is read before the wipe and the
stage regenerated after it.
- spinifex-firewall-apply rejected every verb when the policy file was missing,
including disable. Turning the firewall off is the recovery path, so that left
a node that could be neither armed nor disarmed.
- viperblock exited 0 on a fatal startup error, so Restart=on-failure never
fired. During formation its JetStream bucket has no leader yet, so the EBS
provider died for good and every volume request failed with no responders.
predastore and qemunbd had the same bug.
- spx admin images import created image directories at 0700, so the first sudo
import locked group members out of a tree deliberately made group-writable.
Three of these hid behind checks that cannot succeed: ovn-sbctl show against a
clustered database, and the OVN northbound and southbound waits in setup-ovn.sh.
All three failed to stderr and let the run continue, so a check that could never
pass was indistinguishable from one that was merely slow.
The USB install guide claimed the ISO leaves Spinifex unconfigured and told the
operator to run spx admin init. firstboot has always done that itself, so the
guide now describes what the ISO actually produces and how to start using it.
* fix: let spread placement launch across every node
A spread placement group reserved four distinct nodes correctly and then failed
the launch with ServerInternal, having started only two or three of them. The
scheduler was never at fault; two faults downstream stopped the instances
following the reservation.
vpcd subscribes to vpc.add-nat with QueueSubscribe, which delivers one message
at a time, and the handler holds its reply across the OVN flows barrier. The
45s caller deadline was sized against a single 30s barrier, which is right for
one instance and wrong for the N a spread launch issues at once — the third
request waits behind two full barriers and times out. That one subscription now
overlaps its handlers under a bounded semaphore. Each barrier still syncs after
its own write, so the guarantee is unchanged while N barriers collapse into
roughly one.
With the requests overlapping, every barrier still ran to its alarm.
install-node.sh computed OVN_REMOTE after building the database nodes and gave
it only to compute nodes, so the database nodes were configured without
--ovn-remote. setup-ovn.sh writes ovn-northd's NB/SB connections from that list
and skips it on the localhost default, and it already documents what follows: a
northd dialling only its local member stops advancing nb_cfg once raft
leadership moves, wedging ovn-nbctl --wait=hv. The list is knowable before any
node is touched, so it is computed first and passed to the database nodes too.
Nothing reported an OVN fault while this was happening. Every chassis was
registered, ovn-controller and IPsec were up, and the raft cluster was healthy.
The only signal was the barrier logging that it had overrun and was continuing
without confirmation — which reads like slowness and was a permanent wedge.
* feat: sweep block sizes in the disk benchmark and add an iperf3 network benchmark
* fix: use iperf 2 with CSV output and report why a remote start failed
* fix: stop pkill matching its own shell and verify the iperf server is listening
* ci: run the predastore throughput benchmark at the end of the nightly
* ci: boot the ISO in the nightly, single-node and three-node
* ci: use the existing CI node password secret in the ISO cells
* ci: report free space before the benchmark instead of failing inside warp
* ci: keep the benchmarks and node disks off tmpfs
* ci: export the work directories from RUNNER_TEMP rather than job env
* ci: run the ISO and benchmark workloads from the nvme scratch area
* perf: run guest NICs on vhost-net with multiqueue
Guest-to-guest VPC throughput measured 1.68 Gbit/s single-stream against
21.8 Gbit/s host-to-host on the same underlay, and did not improve with
more streams. QEMU's -netdev tap defaults vhost to off, so the whole
datapath ran in the main loop thread in userspace: that thread sat at
59-65% CPU while all four vCPU threads idled at ~4%.
Every tap netdev now carries vhost=on, and VPC ENIs additionally run
multiqueue at min(vCPUs, 8) queue pairs. Multiqueue needs all three of
the tap, the netdev and the device to agree, so TapNetDev and NetDevice
derive the netdev options and the 2N+2 MSI-X vector count from one queue
figure rather than leaving callers to keep them in step.
The QMP hot-plug path takes the same options as cold boot, with a test
comparing it against the rendered cold-boot strings so the two cannot
drift. MMIO machines stay single-queue: they have no MSI-X.
* ci: check out the sibling repos the go workspace needs in the ISO cells
* ci: skip cell-21 workload cleanup when the workload never ran
* ci: size the cell-21 node at 8 GiB and keep its serial log
* ci: keep the cell-21 node diagnostics alongside the serial log
* fix: let the daemon open /dev/vhost-net and delete multi_queue taps
Enabling vhost=on broke instance launch two ways. DevicePolicy=closed gates
the cgroup device list before capabilities are consulted, so QEMU's open of
/dev/vhost-net returned EPERM despite CAP_DAC_OVERRIDE; add the device to the
allow list and bump the unit version. Separately, `ip tuntap del` re-opens the
device with TUNSETIFF and fails with EINVAL on a multi_queue tap, leaking it on
every teardown; route deletion through `ip link del`, which is flag-agnostic.
* perf: default guest NICs to a single queue
Multiqueue measured a net loss. Spreading one flow across queue pairs reorders
packets and TCP reads that as loss: a 4-vCPU guest fell from 1.76 to 1.50
Gbit/s, with cwnd collapsing on spurious retransmits. The host overlay funnels
a node pair onto one softirq core, so the extra queues have no headroom to win
back what the reordering costs. Keep the plumbing and its tests behind
MultiqueueNICs so re-enabling is one line once that funnel is fixed.
vhost-net stays on unconditionally; it moves the datapath off the QEMU main
loop, measured at 60% of a core under load against 3% with it.
* fix: raise the UDP receive buffer for predastore's QUIC transport
Predastore carries blob, meta and raft over QUIC and quic-go asks for a 7MiB
receive buffer. Left at the kernel default of 208KiB it silently dropped
datagrams under load: the four-node cluster had shed 80k-125k each, which
failed shard writes, put raft into repeated elections, and ultimately handed
guests an I/O error that cost two of them their root filesystems.
Adds a sysctl stage writing /etc/sysctl.d/99-spinifex-net.conf, so the setting
lands in the ISO image and on every node push rather than living in a runbook.
* ci: give the cell-21 workload an identity the node accepts
* feat: tie the guest MTU and NIC multiqueue to ipsec_enabled
ESP costs 34 bytes on top of geneve's 58, so a cluster running the overlay in
plaintext can advertise 1442 instead of 1408 — worth roughly 10% of guest
throughput. Derive it from network.ipsec_enabled rather than hardcoding the
conservative figure.
Multiqueue moves with the same flag, because its sign depends on it. Behind
IPsec it costs ~12%: ESP has no L4 ports, so RSS funnels a node pair onto one
core and extra queues only reorder packets, which TCP reads as loss. Without
IPsec the sending host's single vhost thread saturates first and queues are
worth +54%. IPsec on with multiqueue on is the worst of the four combinations,
so it stops being a standalone knob and becomes vm.Deps.MultiqueueNICs.
The reconciler only ever created a subnet's DHCPOptions row and never revisited
it, so the create-time MTU stuck for the life of the subnet and this change
would have applied to new subnets only. Worse in the other direction: turning
IPsec back on would have left subnets advertising 1442 over ESP, which drops
large segments with no ICMP to recover from. applySubnets now diffs and updates
in place.
reconcile.Config takes IPSecDisabled rather than IPSecEnabled so the zero value
is the conservative MTU — a caller that forgets it is merely slower.
* docs: add an install-time performance tuning guide
Measured on the four-node cluster rather than inferred: the UDP receive buffer
sysctl, what IPsec actually costs and why, the multiqueue sign flip, the overlay
MTU budget, and the jumbo-frame check that must precede enabling it.
A starter doc — several sections end in an open question rather than a
recommendation.
* ci: keep the cell-21 node alive for the workload step
* ci: collect the cell-21 qemu log and widen its timeout
* feat: derive the guest MTU from the fabric, and deepen the virtio NIC config
The overlay overhead is passed on to the guest rather than hidden from it.
Hiding it means running the fabric far enough above 1500 that a standard guest
MTU fits in the headroom, which is what the large public clouds do — but they
own the switches. On a fabric we do not control, the honest figure is the one
the path can carry, and a jumbo underlay should reach the guest: 9000 gives
8942, or 8908 with IPsec. network.underlay_mtu is the lever; SubnetMTU spends
it. Configured rather than probed, because a node's own NIC MTU says nothing
about the switch between it and its peers.
host_mtu sets VIRTIO_NET_F_MTU so a guest that never takes a DHCP lease —
statically addressed, or IPv6-only — still learns the ceiling.
rx/tx_queue_size raise the virtio ring from QEMU's default 256. That default is
a packet-rate ceiling rather than a memory saving: a vCPU preempted for a
fraction of a millisecond fills all 256 slots and the overflow is dropped.
ethtool is the only way to read NIC offload, ring and channel state, so a node
without it cannot be diagnosed for throughput at all.
* ci: give the cell-21 workload its own guest key
* fix: set the guest MTU on the tap, not just in the guest
The tap was created and brought up without an MTU, so it stayed at the
kernel default of 1500 while DHCP and host_mtu told the guest something
else. Harmless at 1442; it silently blackholed every large frame on a
jumbo underlay, with no ICMP back because the drop is at the first hop.
TapSpec.MTU comes from the same guest overlay MTU the DHCP option and
host_mtu already use, and is applied in the bring-up call so a size the
fabric cannot carry fails the launch instead of degrading quietly.
* fix: stop setting tx_queue_size, which QEMU caps at 256 for tap
QEMU only honours a tx_queue_size above 256 for vhost-user and
vhost-vdpa backends. On a tap it rejects the value outright rather than
clamping it, so every launch died with "must be a power of 2 between
256 and 256" and the instance went straight to terminated.
rx_queue_size is unaffected and stays at 1024. A test pins the absence
of tx_queue_size across machine types and queue counts.
* docs: lead the tuning guide with the overlay segmentation finding
The guide's headline was a 5.4x that came almost entirely from turning
IPsec off, and it treated MTU and the fabric as the main levers. The
host reaches 92.6 Gbit/s at MTU 1500 on the same NIC, so the fabric is
not the constraint and the encrypted default is not the thing to tune
away from.
Adds the host-versus-guest comparison up front, documents
network.underlay_mtu, and makes the missing segmentation offload the
first open question.
* ci: stop deselected matrix cells failing in teardown
* docs: quantify what disabling overlay IPsec is worth
The guide described the trade without the numbers an operator needs to
make it. Adds the measured 4.9x, notes that the MTU and multiqueue
changes follow ipsec_enabled rather than being separate knobs, and
records two transition hazards: a stale wide MTU on running guests, and
RunInstances failing while the OVN flows barrier catches up with the new
SAs.
* ci: collect and retain the e2e benchmark and cluster logs
Cells 21 and 22 both copied two paths that only ever exist on the nodes, so
the benchmark JSON was never uploaded and cell-22 collected nothing at all.
Both now gather what they actually produce, and keep it for 90 days.
* fix: quieten spx by installing an explicit CLI logger default
spx never set a process logger, so it inherited whatever a linked package
left behind — until now a JSON handler at Info on stdout, interleaving
library log records with command output and breaking piped use.
Execute() claims the default before any command runs: text on stderr,
LevelError, so commands report failures through their own stderr writes and
libraries stay quiet. --verbose/-v raises to info and SPX_LOG_LEVEL sets an
arbitrary level, the flag winning over the environment. Service subcommands
still call initTelemetry, which replaces the handler outright.
No root --debug flag: viperblock, qemunbd, nats and awsgw each define their
own --debug with a service-specific meaning, and a root persistent flag would
shadow inconsistently depending on the command.
The guard test samples the handler at package-init time rather than inside
the test, because tests in this package start services that install their own
logger legitimately.
* feat: private /admin/CreateAccount for self-service tenant signup
Add POST /admin/<Method> to the AWS gateway: SigV4-signed, JSON in and
JSON out, callable only by an IAM user in the super-admin account holding
spinifex:<Method>. CreateAccount is the one method, so a signup form can
provision a tenant without an operator shell. No new service, port, unit
or certificate — it mounts inside the gateway's existing authenticated
group and inherits its auth, throttling and tracing.
Provisioning is resumable rather than transactional, which fits stores
that have no transaction. Two CAS creates carry the correctness: the
client token admits one in-flight attempt and makes a lost response
replayable for 24 hours, and the name reservation makes duplicate account
names impossible. Every step after the account exists is create-if-absent,
so a retry with the same token finishes the job instead of failing.
The CLI and the endpoint share one provisioning helper, which closes a
gap on both at once: CreateAccount never checked names, so running
`spx admin account create` twice produced two accounts that differed only
by ID.
The response waits for the account's default VPC over a new
ec2.EnsureDefaultVpc request/reply, rather than trusting the existing
fire-and-forget iam.account.created publish — credentials for an account
that cannot launch anything are worse than a slower call.
[signup] max_accounts bounds how many accounts may exist; absent means
uncapped, which is what every cluster does today.
* fix: stop the awsgw template hardcoding one deployment's console URL
console_url shipped as https://console.spx3.com, so every fresh install —
including an on-premise cluster that will never run signup — was
configured to point new accounts at our console. It is the only
deployment-specific domain in any template. It now ships empty, and the
gateway warns when it provisions an account without one, since a blank
sign-in link is otherwise visible only to the customer who receives it.
Extract the remote client from the command body while here. The signing,
request shape and error decoding are the part a Cloudflare Worker
mirrors, and as a function they are testable against a TLS test server
rather than only by hand against a live cluster.
* refactor: drop console_url from signup and default max_accounts to 128
The gateway never read console_url — it passed the value straight through to
the response, and the idempotency record froze it for 24 hours so a corrected
typo still replayed stale. The client that made the request already knows
where it wants to send the new tenant.
max_accounts is now a pointer in the parsed config so an absent [signup]
section takes the 128 default while an explicit 0 still means uncapped.
* feat: run the predastore fault-injection harness as cell-24
Cell-23 asks how fast predastore is when every node is healthy; cell-24 asks
whether it survives one that is not. It runs after cell-23 rather than beside
it: both want a quiet single runner, and a benchmark sharing a machine with a
fault injection run measures the wrong thing.
The port offset keeps the two clear of each other even so, and the job clears
stale nodes on entry and exit, since a run killed before its cleanup trap
leaves one holding ports that fail the next run somewhere unrelated.
* ci: keep the throughput benchmark's output with its results
The run log only existed in the job log, so the artifact it uploaded could not
be read without going back to the run that produced it.
* ci: keep the fault-injection run log with its results
events.txt carries the timeline and the FAIL line, but not the build output or
anything Warp wrote before the harness took over, so a failed cell could only
be read from the job log it was printed into.
* fix: serve the cluster region to the console instead of hardcoding it
The console hardcoded ap-southeast-2 as its SigV4 region in seven files, so on
any other cluster the credential scope was wrong and the gateway rejected every
signed request.
The SPA is embedded in the binary, so a build-time value would pin one binary to
one region. The UI server now serves GET /api/config with the region, no-store
so a proxy fronting several clusters cannot serve one the other's region, and
the SPA loads it once before render.
The region comes from the serving node's config because awsgw pins signature
verification to that same field. Reading the cluster-wide [aws] region instead
would let a cluster whose two keys disagree pass review and still fail to
authenticate.
Also relabels the login card as Spinifex credentials, since the fields are not
AWS ones, and hides the CA install banner unless the console is reached
directly: behind a proxy with a publicly trusted certificate there is nothing
for the cluster CA to fix.
* fix: say once when the host firewall will not be maintained
Both quiet outcomes of the reconcile — no cluster config, and the policy
switched off — return nil without logging, so a node that never arms is
indistinguishable in the journal from one that armed on its first pass.
* test: arm the host firewall before asserting on its peer set
Only the ISO install arms the policy; every other path writes off to the
mode file, so the daemon disables the firewall and never writes a peer
set. The test read that file as its baseline, which it could never have.
It now switches the mode on for its own duration and puts it back, waits
for the peer set the way install-node.sh does, and on failure reports the
mode, the node count and what the daemon said rather than just the
missing file.
* test: move the console region test to the external cmd package
It was in package cmd, which test-package-check rejects, so preflight
failed on the branch. The hook follows the pattern already in
export_test.go.
* test: make the reboot EIP diagnostic capable of naming the drop
The two ofproto/trace probes injected packets with no TTL, so OVN's router
killed them at lr_in_ip_input and both traces ended early for a reason that
had nothing to do with the failure under investigation. A keyword filter then
stripped the table-match lines, hiding where each trace stopped.
Traces now carry a TTL and are kept whole, the real datapath megaflows for the
guest reply are dumped alongside them, and any drop matching the guest source
is resolved from its OpenFlow cookie back to the Southbound logical flow that
installed it.
* fix: exempt an EIP instance from the subnet drop gate with allow, not reroute
A subnet with no route of its own carries a priority-1100 drop policy, and an
instance with an external IP is lifted above it by a priority-1200 policy. That
policy rerouted to the pool gateway, which in distributed NAT is outside every
logical router port subnet, because the gateway LRP is the link-local /30. The
static default route survives that by naming an output port; a routing policy
has no such field, so ovn-northd discarded all three of them and the drop gate
below swallowed everything the instances sent.
The effect was that every external IP answered inbound traffic and none of it
came back: the SYN reached the guest, the guest replied, and the reply died in
the router's policy stage. Guest-initiated egress was lost the same way.
The policy only ever needed to lift the gate. The router's default route already
points at the gateway, so allow leaves the routing decision intact and gives
northd nothing to reject.
* test: match the log the firewall quorum gate actually emits
The assertion waited for "chassis encap addresses; not writing a partial peer
set", which no code has ever logged. The reconcile reports a short chassis list
as "OVN reports N of M chassis encap addresses" and the maintain loop carries
that into its retry warning, so the wait could only ever time out.
Arming the node in the previous commit got the test as far as this assertion for
the first time, which is why it surfaced now. The failure also prints the
journal tail and the same inputs the earlier wait reports.
* fix: stop one unreadable volume document failing every volume listing
A single volume document whose shards no longer reconstruct made
DescribeVolumes, DescribeVolumeStatus and DescribeVolumesModifications
fail for every account, because the listing returned the first fetch
error rather than skipping the document.
ListVolumes now skips a document it cannot read or decode and logs the
skip, mirroring ListAMIs. Callers whose answer would be wrong rather
than merely partial take ListVolumesStrict: the orphan scan and the
snapshot in-use check both gate destructive work on a complete list.
The tolerance covers fetch failures as well as decode failures. An
object whose shards no longer join is as unusable as one whose bytes
will not parse, and it was the fetch path that took the cluster down.
* feat: pull Ochre model weights from a hub into predastore
Add `spx admin ochre weights pull`: fetch a self-host model's weights from
Hugging Face directly into predastore, producing a prefix `weights stage`
consumes. Streams safetensors-only artefacts (plus config/tokenizer and the
shard index), refusing pickle/.bin. Pins an immutable commit SHA, writes an
ochre-pull.json manifest that stage records as SourceRevision, and aborts clean
(removing any written object) on a gated or mid-stream failure. Token resolves
from --hf-token, HF_TOKEN, or a stored platform credential; never logged.
Also add `spx admin ochre credentials set` to store an encrypted provider
credential from stdin, and SourceRevision on the weights KV record + list.
* fix: stream hub downloads without a whole-request timeout
http.Client.Timeout bounds the entire request including the body read,
so a multi-GB safetensors shard was guillotined mid-download at 30s. Bound
dial and response-header latency via the transport instead, leaving the
body free to stream for as long as the server keeps sending.
The InvokeModel OUTPUT guardrail hook logged and forwarded the model response
unguarded when the per-family completion extractor could not parse the body, so
output enforcement failed open while the INPUT path already failed closed.
Both the non-stream OUTPUT hook and the buffered InvokeModelWithResponseStream
source now fail closed: on an unparseable completion (non-stream) or an
undecodable assistant chunk (stream) they load the guardrail view and emit a
blocked response carrying BlockedOutputsMessaging. A genuinely empty completion
(valid chunks, no assistant text) still passes through. Shared resolve/load
prologue extracted into loadGuardrailView.
Self-host InvokeModel/InvokeModelWithResponseStream picked the adapter by tier
and always used the Llama adapter, silently mis-serving a non-Meta model through
Llama's native request schema. Dispatch now selects by catalog Family; an
unhandled self-host family is refused with ValidationException, not mis-served.
Endpoint-resolution failures hardcoded ServiceUnavailableException, discarding
the AWS code the daemon put on the wire. All four self-host resolve sites now
resolve the carried code (ModelNotReadyException, ThrottlingException, ...) and
fall back to ServiceUnavailable only for an un-coded transport failure.
Josh Siviour