predastore pinned MULGA_ROOT_TRACE_RATIO=0.05 so guest block I/O, which is
locally rooted and has no caller trace, would not drown request traces. That
worked, but TraceIDRatioBased decides when the root span is created, before
any error exists, so it discarded 95% of predastore failures along with the
successes. An RCA on a rare locally-rooted failure had a 1-in-20 chance of
having the span at all.
The sink now runs a tail-sampling tier in front of APM Server that keeps every
errored trace, keeps every trace slower than 2s, and samples the rest at 10%.
It judges a trace once it is whole, so it can shed the same volume while
keeping the traces worth having. Dropping the head ratio is what lets it see
them: at 0.05 the tail sampler only ever saw 5% of predastore roots, so its
error rule could keep at most 5% of the failures.
Sized before changing: predastore's true local root rate is ~434 spans/s
across six active CI environments, against 21.7/s reaching the sink today.
Driving the tier at 508 spans/s held it at 88 MiB of a 1.5 GiB cap and 1.8%
CPU with zero drops, and trace storage goes from 14.9 to 22.5 GB per 7 days
on a volume with 735 GB free.
* docs: bring doc titles and descriptions into SEO length limits
Rewrite the frontmatter title and description of every published doc so the
rendered <title> lands in 50-60 characters (including the site suffix) and the
meta description lands in 150-160 characters.
* docs: move SEO titles to a seoTitle field and restore display titles
The frontmatter title renders as the page H1 and the sidebar label, so tuning it
for the title tag shortened nothing and lengthened the visible navigation. Put
the search-facing string in seoTitle and give title back its original value.
* docs: document the frontmatter length rules for the docs site
The artifact bundle carried the guest console and the spinifex journals but
nothing from OVN, so a phase 6 egress failure could not be attributed to an
unresolved gateway nexthop MAC versus a return-path or upstream problem.
Register an OnFailure at the top of phase 6 that writes
static-mac-binding-list, the SB MAC_Binding table and the VPC router's
lr-route-list into the bundle.
The nightly collected only spinifex-* journal units, so a cross-chassis
black-hole could not be attributed to IPsec SA establishment versus SB
port binding. ovn-controller, ovn-northd and ovs-vswitchd log to files
rather than journald, so they were absent from the bundle entirely.
Adds the openvswitch-ipsec.service journal per node, mirroring e2e.yml,
and pulls /var/log/ovn and /var/log/openvswitch off each node.
* docs: bring doc titles and descriptions into SEO length limits
Rewrite the frontmatter title and description of every published doc so the
rendered <title> lands in 50-60 characters (including the site suffix) and the
meta description lands in 150-160 characters.
* docs: move SEO titles to a seoTitle field and restore display titles
The frontmatter title renders as the page H1 and the sidebar label, so tuning it
for the title tag shortened nothing and lengthened the visible navigation. Put
the search-facing string in seoTitle and give title back its original value.
The guest fail-loop ran 36 iterations at ~28 s each (two 10 s curl timeouts,
a 3 s ping, a 5 s sleep), so NAT-E2E-EGRESS-FAIL first printed at ~16m50s
while the console wait gave up at 8m. The marker was structurally
unreachable and an egress failure reported only a byte count.
Derive the wait budget from the loop instead of hard-coding both ends: 10
iterations at a 30 s per-iteration allowance plus 3m for boot and cloud-init,
keeping the same 8m ceiling. The user-data script is now generated from those
constants and the existing marker consts. Dump the last console read via
OnFailure so a budget overrun also leaves the console in artifacts.
TestSGPolicyDatapath/AllowedTraffic waited 60s for curl to reach the
target's HTTP server but nothing ever waited for that server to start.
Its cloud-init runs later than the client's sshd, which is what the
preceding stage actually gates on, so the budget was measured from the
wrong baseline and the gap between the two is unbounded on a loaded
node. The failures reported curl exit 7, a refused connection: a
missing SG allow is a drop ACL here, which would time out as exit 28.
The target guest now reports its listener on the serial console, the
only channel back to a guest whose SG has no SSH ingress, and a new
TargetHTTPReady stage waits for that marker before AllowedTraffic runs
at all. AllowedTraffic keeps its 60s budget, which now only has to
cover chassis flow install, and dumps VPC flow diagnostics when it does
fail so a genuine propagation failure is attributable from CI artifacts.
test-package-check fails on dev without it: the test mutates the
unexported guestPortBackoffBase / guestPortBackoffMax and drives
ensureGuestPortDatapath and recordPortFailure directly.
A node whose NewExternalIPAM call lost the race with the JetStream meta
leader logs a WARN and carries on with d.externalIPAM nil. That nil
concrete pointer was handed straight to the PublicIPAllocator interface,
where it reads as non-nil, so the guard in PrepareRunInstances passed and
findPool derefed a nil receiver. RunInstances into a subnet with
MapPublicIpOnLaunch took the daemon down with SIGSEGV, and because the ENI
was already persisted and the gateway saw only a NATS timeout, nothing
rolled it back: the orphan ENI later rejected DeleteSecurityGroup with
DependencyViolation.
The concrete value now crosses into PublicIPAllocator and PublicIPReleaser
only once init succeeded, and AllocateIP/ReleaseIP tolerate a nil receiver
so another caller repeating the mistake degrades instead of panicking.
Making the interface genuinely nil is not enough on its own: the existing
nil check would then have skipped assignment altogether and booted an
instance with no public address on a subnet that promised one. The
wantPublic decision now runs unconditionally and a missing allocator takes
the allocation-failure path, which detaches and deletes the ENI, returns
capacity and reports InsufficientAddressCapacity.
Launches that do not ask for a public IP are untouched: subnets default to
MapPublicIpOnLaunch only when the cluster has public pools, which is the
same condition that initialises the IPAM.
* feat: ship per-test JUnit results to the Elasticsearch sink
Every e2e workflow already converts its test log to JUnit XML and renders
it into the job summary, then discards it with the artifact. mulga-ci-runs
records which suite failed; nothing records which test failed, so
root-causing a failure means reading raw Actions logs across historical
runs.
The junit-to-es action bulk-indexes one doc per leaf test case into
mulga-ci-test-results: name, subtest, status, duration, failure text, run
id, cell, suite, branch and sha. Parent t.Run wrappers are dropped so a
failing subtest counts once, matching the job summary renderer. The doc id
is deterministic, so a replayed post overwrites rather than
double-counting.
Called from every workflow that produces JUnit XML, and tolerated on
failure: an unreachable sink must not turn a green test run red.
* feat: classify unparsed JUnit cases as unknown, not error
go-junit-report emits <error message="No test result found"> when a
=== RUN line has no matching result line, which means the log it was fed
was truncated or filtered. Recording that as a failure would put 78
phantom failures per green bare-metal run into the pass rate.
Two ports on env19 have no tap behind them anywhere in the cluster:
their NB Logical_Switch_Port reads up: false and the SB Port_Binding
has chassis: [], so nothing can ever claim them. ensureGuestPortDatapath
is already bounded, but the outer reconcile loop calls it again every
cycle, so each dead port pays the full 45s nudge sequence and files
another ERROR roughly twelve times an hour, which is 1500 of vpcd's 1691
daily WARN lines and all of its ERRORs.
A port that burns its convergence deadline now backs off, doubling from
a minute to a thirty-minute cap, and clears the moment it binds. The
failure and recovery lines also carry the ENI, the EIP and the guest IP:
they previously named only the VPC and the LSP, so the guest behind a
stuck port could not be identified from telemetry.
slog renders a bare time.Duration as an int64 nanosecond count and names
the unit nowhere, so every duration field in the fleet read like
"elapsed":135000000000. The repo idiom of .Round(time.Second) changed
the value without changing the shape and truncated every sub-second
measurement to zero.
otelsetup.Millis converts a duration for logging under a snake_case key
ending in _ms, and TestSlogDurationConvention type-checks ./spinifex/...
and ./cmd/... to fail the build when a raw Duration reaches an slog call
or a Millis value is logged under a key without the suffix. It found 55
sites across 34 files, well past the sites a grep for time.Since shows,
because most durations come from a config field or a package constant.
A handler-level ReplaceAttr would fix every site at once but can only
emit d.String(), which Elasticsearch maps as a keyword and nothing can
aggregate over.
A user who signs up for the public sandbox and logs in at console.spx3.com
lands on the normal Dashboard, which reads "No instances", "No volumes" and
"No VPCs" for a brand new account. Nothing says this is a sandbox and nothing
suggests a first action.
Show a one-time dialog on the Dashboard when the console is reached at the
sandbox hostname. It names the sandbox, sets expectations about the cluster
being shared and reclaimable, and links to the areas worth visiting first.
Dismissal is remembered in localStorage, and storage being unavailable only
means it shows again next visit.
Sandbox-ness is decided from the hostname rather than a server flag: it is a
property of how the user reached the console, not of the node serving it, and
the same nodes answer on a LAN address and on the public name. This follows
isDirectHostAccess, which already gates the certificate banner on the login
page the same way.
The console SPA is embedded in the binary, so the rebuilt bundle is included.
The daemon's connect to the platform pgvector appliance gave up after a
fixed five attempts; once exhausted, startOchreVector returned and the
ochre.vector.query subject stayed unregistered for the daemon's lifetime,
so RAG never recovered after the appliance was torn down and re-adopted
across a spinifex.target restart without a manual daemon bounce.
Retry Ensure-then-Connect until the daemon context is cancelled instead,
via a small generic retryUntilContext helper with a capped doubling
backoff and throttled logging. Each attempt already re-resolves the live
dial IP, so a re-adopted appliance on a new address is picked up and the
query subject registers on its own.
Every non-success on the daemon's NATS path recorded as error, so an
unknown instance id, a bad parameter and a genuine server fault were one
series and a daemon error rate tracked caller behaviour as much as
daemon health. The same conflation at the log level made an expected
fan-out miss write an ERROR line on every node that was never going to
answer it: on a four-node cluster each IMDS metadata fetch by a booting
guest produced three that meant nothing.
Failures are now classified by the status their AWS error code maps to,
through the same function the HTTP middleware calls, so the two paths
cannot drift apart. A caller mistake records client_error and logs at
WARN; a server fault records error and stays at ERROR. The rule is now
that ERROR means the daemon failed, which is broader than the fan-out
case and subsumes it.
Separately, the per-instance command path was subscribed raw and passed
through no metrics wrapper at all, so start, stop, terminate, reboot and
the volume and ENI hot-plug commands produced no data points. They are
now counted under an action naming the command rather than the subject,
which carries the instance id. The drain runs off-thread and records its
own point once the flush finishes.
The gateway derived each request's IAM action prefix from the wire-service
name, so bedrock-runtime, bedrock-agent and bedrock-agent-runtime calls were
checked as bedrock-runtime:ConverseStream, bedrock-agent-runtime:Retrieve, etc.
AWS namespaces every Bedrock action under bedrock:, so an AWS-shaped scoped
policy never matched and only a wildcard admitted the call.
Collapse the Bedrock family to the shared bedrock: namespace in IAMAction, the
single choke point for policy decisions; recordResolvedAction keeps the wire
name for telemetry.
Guest series from qmp-collector carry the owning account as the
CloudWatch dimension account_id. Traces and logs carry the same account
as aws.account_id, and on metrics-apm* the bare account_id name is
already taken by the NATS global account the nats service reports. A
dashboard control built on the trace field therefore cannot reach guest
metrics, which is why the guest panels sit apart from the account
drill-down.
The dimension name is an AWS-facing contract, so it is not renamed. The
OTel bridge mirrors it instead: guest series now export aws.account_id
alongside account_id, and a series with no account exports neither a
blank value nor a placeholder.
ebsRequestWithTrace started its span from context.Background(), so every
ebs.mount, ebs.unmount and ebs.delete rooted a trace of its own. Block storage
work was therefore unreachable from the request that caused it: viperblockd
recorded 10,209 transactions in 24h and not one could be tied to a caller or an
account.
VolumeMounter now takes the caller's context. Launch and hot attach/detach have
one to give. Shutdown, crash recovery and teardown genuinely do not — they
answer to no request — so they pass a background context, which is an honest
root rather than a missing one.
The owning account rides both the span and the NATS header, because
viperblockd's consumer span reads that header to attribute itself. It comes off
the instance where there is one, and is passed explicitly for a lone volume
request, which carries no instance to read it from. An absent account is left
off rather than blanked: recovery and teardown act for nobody, and crediting
that work to whoever happens to be admin would read as tenant activity.
http.Server.WriteTimeout is a total deadline rather than an idle one, so any
value below the RDS command channel's 20s long poll kills every poll mid-flight
and strands agents on a channel no command can reach. Nothing recorded that the
omission was load-bearing, leaving it one hardening pass away from removal.
The sandbox is one cluster with tenancy at the account level, so every
question about it starts with which account. Today the account is on the
gateway's own spans and nowhere else: of 187,317 daemon transactions in 24
hours, none name an account, and the same holds for vpcd, viperblockd and
northstar. A tenant's request is attributable at the front door and anonymous
everywhere it does its work.
The account already crosses NATS on every request as X-Account-ID, so the
spans can name it without new plumbing. StartConsumerSpan reads the header and
startProducerSpan takes the account the caller sent for, which covers every
handler built on ServeNATSRequest and every fan-out through Gather.
Work with no caller - a sweep, a reconciler, a node-to-node probe - is left
unattributed rather than credited to whoever happens to be admin.
Merges `feat/ebs-export-tunables` (83 commits) into dev. The branch could not be rebase-merged because it carries five merges from dev, so it lands as a squash.
## What is in it
- **EBS export tunables** and the nbdkit/viperblock export path they configure.
- **Teardown reapers and backoff** for S3 buckets, load balancers, gateways, spot requests, certificates, endpoints, elastic IPs and network interfaces, so a dependent that keeps failing is backed off rather than retried hot.
- **Retry and tolerance paths**: ECS IAM service init, ECR signing keys deleted between the list and the read, an unreadable signing key at awsgw startup, and an ECS instance-profile policy attach converging on a stale read.
- **Key pair validation reads the material by name** instead of listing the account, closing the `InvalidKeyPair.NotFound` race that a single-replica LIST could produce.
- **Concurrency and fan-out**: concurrent metadata document fetches, bounded metadata reads for describes, and the node-discovery timeout removed from every fan-out call.
- **OVN correctness against the configured NB cluster**: the flows barrier, nexthop MAC binding, and stopping a compute node from running the package's standalone database.
- **nats-js-watchdog can re-arm itself.** The escalation check ran above the probe loop while the only line that cleared the counter sat below it, so a latched watchdog could never observe a recovery. It now probes first and decides second, and says so once per latch rather than once per tick.
- **CI**: 19 commits of nightly and build-pipeline work.
## Verification
`make preflight` green locally. Build and Test, Integration Tests, Race Detection, and Lint and Security all green on `c2da0385`.
## Known red, not introduced here
Two nightly cells fail on this branch and are pre-existing:
- **cell-10** — a partial OVN IPsec enable leaves `NB_Global ipsec=true` asserted cluster-wide while two of three chassis failed their local reconcile, black-holing cross-chassis guest traffic for roughly 5.5 minutes while every health signal stays green.
- **cell-19** — a total layer-3 egress blackhole below a correctly and promptly programmed NAT/EIP datapath, aggravated by a natuplink probe whose retry budget outlives the test's own wait, so it can never emit its FAIL verdict.
Both are filed separately and should be weighed before a release is cut.
Julian Sommer