mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Josh Siviour
52f12586f5
predastore pinned MULGA_ROOT_TRACE_RATIO=0.05 so guest block I/O, which is locally rooted and has no caller trace, would not drown request traces. That worked, but TraceIDRatioBased decides when the root span is created, before any error exists, so it discarded 95% of predastore failures along with the successes. An RCA on a rare locally-rooted failure had a 1-in-20 chance of having the span at all. The sink now runs a tail-sampling tier in front of APM Server that keeps every errored trace, keeps every trace slower than 2s, and samples the rest at 10%. It judges a trace once it is whole, so it can shed the same volume while keeping the traces worth having. Dropping the head ratio is what lets it see them: at 0.05 the tail sampler only ever saw 5% of predastore roots, so its error rule could keep at most 5% of the failures. Sized before changing: predastore's true local root rate is ~434 spans/s across six active CI environments, against 21.7/s reaching the sink today. Driving the tier at 508 spans/s held it at 88 MiB of a 1.5 GiB cap and 1.8% CPU with zero drops, and trace storage goes from 14.9 to 22.5 GB per 7 days on a volume with 735 GB free.