slog renders a bare time.Duration as an int64 nanosecond count and names
the unit nowhere, so every duration field in the fleet read like
"elapsed":135000000000. The repo idiom of .Round(time.Second) changed
the value without changing the shape and truncated every sub-second
measurement to zero.
otelsetup.Millis converts a duration for logging under a snake_case key
ending in _ms, and TestSlogDurationConvention type-checks ./spinifex/...
and ./cmd/... to fail the build when a raw Duration reaches an slog call
or a Millis value is logged under a key without the suffix. It found 55
sites across 34 files, well past the sites a grep for time.Since shows,
because most durations come from a config field or a package constant.
A handler-level ReplaceAttr would fix every site at once but can only
emit d.String(), which Elasticsearch maps as a keyword and nothing can
aggregate over.
A user who signs up for the public sandbox and logs in at console.spx3.com
lands on the normal Dashboard, which reads "No instances", "No volumes" and
"No VPCs" for a brand new account. Nothing says this is a sandbox and nothing
suggests a first action.
Show a one-time dialog on the Dashboard when the console is reached at the
sandbox hostname. It names the sandbox, sets expectations about the cluster
being shared and reclaimable, and links to the areas worth visiting first.
Dismissal is remembered in localStorage, and storage being unavailable only
means it shows again next visit.
Sandbox-ness is decided from the hostname rather than a server flag: it is a
property of how the user reached the console, not of the node serving it, and
the same nodes answer on a LAN address and on the public name. This follows
isDirectHostAccess, which already gates the certificate banner on the login
page the same way.
The console SPA is embedded in the binary, so the rebuilt bundle is included.
The daemon's connect to the platform pgvector appliance gave up after a
fixed five attempts; once exhausted, startOchreVector returned and the
ochre.vector.query subject stayed unregistered for the daemon's lifetime,
so RAG never recovered after the appliance was torn down and re-adopted
across a spinifex.target restart without a manual daemon bounce.
Retry Ensure-then-Connect until the daemon context is cancelled instead,
via a small generic retryUntilContext helper with a capped doubling
backoff and throttled logging. Each attempt already re-resolves the live
dial IP, so a re-adopted appliance on a new address is picked up and the
query subject registers on its own.
Every non-success on the daemon's NATS path recorded as error, so an
unknown instance id, a bad parameter and a genuine server fault were one
series and a daemon error rate tracked caller behaviour as much as
daemon health. The same conflation at the log level made an expected
fan-out miss write an ERROR line on every node that was never going to
answer it: on a four-node cluster each IMDS metadata fetch by a booting
guest produced three that meant nothing.
Failures are now classified by the status their AWS error code maps to,
through the same function the HTTP middleware calls, so the two paths
cannot drift apart. A caller mistake records client_error and logs at
WARN; a server fault records error and stays at ERROR. The rule is now
that ERROR means the daemon failed, which is broader than the fan-out
case and subsumes it.
Separately, the per-instance command path was subscribed raw and passed
through no metrics wrapper at all, so start, stop, terminate, reboot and
the volume and ENI hot-plug commands produced no data points. They are
now counted under an action naming the command rather than the subject,
which carries the instance id. The drain runs off-thread and records its
own point once the flush finishes.
The gateway derived each request's IAM action prefix from the wire-service
name, so bedrock-runtime, bedrock-agent and bedrock-agent-runtime calls were
checked as bedrock-runtime:ConverseStream, bedrock-agent-runtime:Retrieve, etc.
AWS namespaces every Bedrock action under bedrock:, so an AWS-shaped scoped
policy never matched and only a wildcard admitted the call.
Collapse the Bedrock family to the shared bedrock: namespace in IAMAction, the
single choke point for policy decisions; recordResolvedAction keeps the wire
name for telemetry.
Guest series from qmp-collector carry the owning account as the
CloudWatch dimension account_id. Traces and logs carry the same account
as aws.account_id, and on metrics-apm* the bare account_id name is
already taken by the NATS global account the nats service reports. A
dashboard control built on the trace field therefore cannot reach guest
metrics, which is why the guest panels sit apart from the account
drill-down.
The dimension name is an AWS-facing contract, so it is not renamed. The
OTel bridge mirrors it instead: guest series now export aws.account_id
alongside account_id, and a series with no account exports neither a
blank value nor a placeholder.
ebsRequestWithTrace started its span from context.Background(), so every
ebs.mount, ebs.unmount and ebs.delete rooted a trace of its own. Block storage
work was therefore unreachable from the request that caused it: viperblockd
recorded 10,209 transactions in 24h and not one could be tied to a caller or an
account.
VolumeMounter now takes the caller's context. Launch and hot attach/detach have
one to give. Shutdown, crash recovery and teardown genuinely do not — they
answer to no request — so they pass a background context, which is an honest
root rather than a missing one.
The owning account rides both the span and the NATS header, because
viperblockd's consumer span reads that header to attribute itself. It comes off
the instance where there is one, and is passed explicitly for a lone volume
request, which carries no instance to read it from. An absent account is left
off rather than blanked: recovery and teardown act for nobody, and crediting
that work to whoever happens to be admin would read as tenant activity.
http.Server.WriteTimeout is a total deadline rather than an idle one, so any
value below the RDS command channel's 20s long poll kills every poll mid-flight
and strands agents on a channel no command can reach. Nothing recorded that the
omission was load-bearing, leaving it one hardening pass away from removal.
The sandbox is one cluster with tenancy at the account level, so every
question about it starts with which account. Today the account is on the
gateway's own spans and nowhere else: of 187,317 daemon transactions in 24
hours, none name an account, and the same holds for vpcd, viperblockd and
northstar. A tenant's request is attributable at the front door and anonymous
everywhere it does its work.
The account already crosses NATS on every request as X-Account-ID, so the
spans can name it without new plumbing. StartConsumerSpan reads the header and
startProducerSpan takes the account the caller sent for, which covers every
handler built on ServeNATSRequest and every fan-out through Gather.
Work with no caller - a sweep, a reconciler, a node-to-node probe - is left
unattributed rather than credited to whoever happens to be admin.
Merges `feat/ebs-export-tunables` (83 commits) into dev. The branch could not be rebase-merged because it carries five merges from dev, so it lands as a squash.
## What is in it
- **EBS export tunables** and the nbdkit/viperblock export path they configure.
- **Teardown reapers and backoff** for S3 buckets, load balancers, gateways, spot requests, certificates, endpoints, elastic IPs and network interfaces, so a dependent that keeps failing is backed off rather than retried hot.
- **Retry and tolerance paths**: ECS IAM service init, ECR signing keys deleted between the list and the read, an unreadable signing key at awsgw startup, and an ECS instance-profile policy attach converging on a stale read.
- **Key pair validation reads the material by name** instead of listing the account, closing the `InvalidKeyPair.NotFound` race that a single-replica LIST could produce.
- **Concurrency and fan-out**: concurrent metadata document fetches, bounded metadata reads for describes, and the node-discovery timeout removed from every fan-out call.
- **OVN correctness against the configured NB cluster**: the flows barrier, nexthop MAC binding, and stopping a compute node from running the package's standalone database.
- **nats-js-watchdog can re-arm itself.** The escalation check ran above the probe loop while the only line that cleared the counter sat below it, so a latched watchdog could never observe a recovery. It now probes first and decides second, and says so once per latch rather than once per tick.
- **CI**: 19 commits of nightly and build-pipeline work.
## Verification
`make preflight` green locally. Build and Test, Integration Tests, Race Detection, and Lint and Security all green on `c2da0385`.
## Known red, not introduced here
Two nightly cells fail on this branch and are pre-existing:
- **cell-10** — a partial OVN IPsec enable leaves `NB_Global ipsec=true` asserted cluster-wide while two of three chassis failed their local reconcile, black-holing cross-chassis guest traffic for roughly 5.5 minutes while every health signal stays green.
- **cell-19** — a total layer-3 egress blackhole below a correctly and promptly programmed NAT/EIP datapath, aggravated by a natuplink probe whose retry budget outlives the test's own wait, so it can never emit its FAIL verdict.
Both are filed separately and should be weighed before a release is cut.
The lockout fires after 10 auth failures from an address in 60 seconds and
then answers RequestLimitExceeded to everything from that address. A client
repeating one hopeless request reaches the threshold as fast as an attacker
working through a keyspace, and 503 tells it to retry later - advice it can
never act on, because the fault is a credential that will never resolve. On
wattle four guests holding deleted STS session credentials produced 53,392
rejected requests and 69,820 503s in a day, and could not recover by fixing
their credentials because the lockout is on the address, not the principal.
RecordFailure now takes a fingerprint identifying the attempt, and the window
holds one entry per distinct fingerprint: a repeat refreshes its timestamp
instead of adding to the count. Whatever an attacker must vary is what the
fingerprint carries - the key id, the presented signature, a digest of the
session token - so guessing is throttled exactly as before while one dead
credential presented 14,000 times stays one fault.
A request rejected before its key id is parsed has no identity to protect, so
malformed envelopes and skewed timestamps still count every occurrence.
An unresolvable instance role answers an empty 200 on the
security-credentials list and a 404 on the credential itself. An SDK reads
that as "no EC2 IMDS role found" and keeps signing with whatever it already
holds, so a guest whose credentials have expired retries forever and never
recovers. On wattle four guests did this for five days: 42,517 role-list
requests, no credential fetches, and not one line anywhere saying why.
The absence was invisible because three layers returned a silent nil -
DescribeInstances missing the instance, no instance profile ARN on the
record, and a profile IAM no longer has. profileFor now reports which of
those it hit, and the two credential paths warn on it, throttled to one line
per instance and reason every five minutes so a stuck fleet stays legible at
that request volume. A role-name mismatch is the guest's own doing and stays
silent.
96.7% of spinifex-daemon request points carried no outcome. natsMetricsHandler
recorded around a plain nats.MsgHandler, which writes its own reply and returns
nothing, so the wrapper had nothing to observe and passed an empty outcome. Only
the /health HTTP path, which goes through the OTel HTTP middleware, ever
reported one.
The handler type now reports how it answered:
type natsHandler func(*nats.Msg) string
Making it a distinct type means the compiler enumerates every subscription and
every handler, rather than a correlation map guessing at responders that reply
after the wrapper returns. Four outcomes are defined: success, error, skipped
for a broadcast handler that declines because the request names another node,
and a deferred sentinel for handlers that answer from a goroutine and record
their own point once the work finishes — timing the wrapper would have reported
a five-minute system launch as an instant success.
utils.ServeNATSRequest and ServeNATSRequestCtx now return whether they replied
with a result, so handlers built on them report an outcome without restating the
error handling. Adding the return value leaves existing callers untouched.
Also measures the per-reservation launch subject, which was subscribed raw and
emitted no request points at all. It records under the subject root because the
reservation id is unbounded.
awsgw request telemetry said POST / and nothing else: no AWS action, no caller,
no reason. On wattle that left 69,820 responses with status 503 in a day
unattributable from the request docs — the cause was only findable by leaving
them for separate auth-failure log lines, which do carry sourceIP and
accessKeyID.
The request logger is the outermost middleware, so a context derived downstream
never reaches it. A small mutable record now rides the request: the auth path
writes identity as it resolves it, writeSigV4Error writes the verdict, and the
action enricher writes the action. The logger and the server span both read it,
so a trace search for failing requests answers from where, doing what and
rejected why without a join.
The record holds an access key id and an error code and nothing else that could
hold a credential; a test asserts the field set so that stays true.
vpcd and spinifex-ui never called otelsetup.SetRequestAction, so rpc_method on
their request metrics stayed at the bare HTTP verb the middleware sets. Over 24h
vpcd had exactly two action values, GET and PUT, which left IMDS countable in
aggregate but made /latest/api/token indistinguishable from a credential fetch
at /latest/meta-data/iam/security-credentials/<role>. Guests depend on IMDS for
credentials, so a failure there is user-visible and was not attributable.
Both namers map to a closed set with a single catch-all, because a guest
controls the request path and an unbounded name would put one metric series
behind every invented path. The role name, the interface MAC and the public-key
index are stripped for the same reason, and spinifex-ui names the static and SPA
branches rather than the content-hashed asset filename.
The middleware also treated every 4xx as a success, which is why vpcd and
spinifex-ui reported zero errors over 24h despite IMDS answering 401 for a
missing IMDSv2 token and 404 for an unresolvable ENI. Those now record
client_error. error keeps meaning 5xx so existing error-rate panels are
unchanged, and a 4xx still does not mark the span failed.
The ecs-agent classified only *.dkr.ecr.* hosts as ECR, so a registry
advertised as the gateway's bare host/IP pulled anonymously and 401'd.
Recognise the agent's own gateway host as ECR and mint a token for it.
The instance role — the pull's fallback credential when a task carries no
execution role — granted ecs:* only; add ECR read so the pull authorizes.
* docs: repoint network invariants doc comment at docs/adr/
The region-local ADR set moved from docs/development/proposals/ to
docs/adr/ on acceptance, and 0006 kept its filename but the directory
changed.
* docs: repoint network invariants doc comment at the proposal path
The region-local ADR set was never accepted, so it lives under
docs/development/proposals/region-local-architecture/ rather than
docs/adr/.
---------
Co-authored-by: joshsiv-mulga <engineering@mulgadc.com>
A release for an IP with no tracked lease was answered with a plain error. The
comment at the responder assumed callers log-and-continue, so failing loudly
would surface the strand without wedging teardown. That assumption does not
hold: TerminatedTeardownReaper marks TeardownNAT failed, TeardownComplete stays
false, and the record is re-driven every sweep. No retry can make a deleted
lease reappear, so it never converges.
It also disables its own backstop. The terminated bucket has a 1h TTL meant to
bound the record regardless, but each sweep persists the advanced marks and that
write refreshes the TTL, so the record outlives the very thing meant to catch
it. Measured across four hosts: seventeen stranded addresses, each retried on a
fixed 120s cadence, indefinitely.
Keep the loud failure and make it terminal. The release reply gains a
machine-readable code, the client maps it to ErrLeaseNotTracked, and
ReleasePublicIP treats that as done after recording the leak, since the local
pool slot is already free and the upstream lease cannot be released without a
lease record. Teardown then completes and the record purges.
The strand is no longer silent: mulga.resource.leaked counts it by resource
class and the address is logged at error level, so a leaked address is
queryable rather than inferred from a repeating warning.
Go stores test results in the build cache alongside compiled objects, so now that the cache persists across runs an unchanged package replays its previous result instead of executing.
That is what makes a local run fast and exactly what a merge gate should not do: a green check would no longer mean the suite ran against that commit.
Add a GOTESTFLAGS hook, empty by default so local runs keep reusing results, and pass -count=1 from the two CI steps that gate a PR.
Compilation is still cached either way, so this costs no build time.
setup-go only writes its cache on a key miss, and keyed it on spinifex/go.sum alone.
Once a run saved that key every later run logged "Cache hit ... not saving cache", so the cache stayed frozen at whatever the first job to finish happened to have compiled and nothing built afterwards was ever stored.
All four jobs also shared that one key, so the job that won the save race decided what the other three restored.
The race job was handed artifacts built without -race, and the build job artifacts built without GOFIPS140, neither of which yields a single usable entry.
Manage both caches directly instead: modules keyed on all five go.sum files so the sibling modules stop re-downloading, and build artifacts keyed per job with a per-run suffix so each flavour keeps its own entry and refreshes it every run.
GOFIPS140 is a global build-config change: it alters every package's action ID, stdlib included.
A target that omits it therefore shares no build-cache entry with one that sets it, and the two flavours cost roughly a full cold build to switch between.
Seven targets set it inline while eight go invocations did not, so test-integration, test-harness, test-actions, test-segscan-oracle, bench and the go run tools all built the non-FIPS flavour.
In CI this is load-bearing: the integration job wins the shared setup-go cache-save race, stores a non-FIPS cache, and every FIPS build in the other jobs then misses on every lookup.
Export it once so the two cannot drift apart again.
The case seeded no config, so loadLocalConfig fell back to the host's
installed spinifex.toml. It passed on a provisioned dev box and failed on
a bare runner, where the region pre-flight fires before svc.Start().
Josh Siviour