mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 08:18:18 +00:00
refactor(obs): make dial9 telemetry opt-in and actually record events (#4663)
* refactor(obs): make dial9 telemetry opt-in and actually record events
The dial9 Tokio-runtime profiler was disabled by default, yet every build
paid for it, and enabling it produced trace files with no events in them.
Recorded empty traces
---------------------
`build_traced_runtime` called `TracedRuntime::builder()...build(..)`, but dial9
only starts recording in `build_and_start*`. `build` still returns a live guard
whose `is_enabled()` reports true, and still creates and seals segment files —
they just contain a header and no events. It also skipped `with_trace_path`, so
the background worker driving the segment pipeline was never spawned.
Measured on the new smoke example: 310 bytes of bare segment header, against
5640 bytes for the same workload once recording actually starts.
Switch to `with_trace_path(..).build_and_start(..)`.
Cost was unconditional
----------------------
`--cfg tokio_unstable` was a global `[build] rustflags` entry and `rustfs-obs`
depended on `dial9-tokio-telemetry` unconditionally, so all builds depended on
Tokio's non-semver API. Worse, an environment `RUSTFLAGS` replaces (never
appends to) the config-file value, so any caller exporting their own RUSTFLAGS
silently dropped the flag — the long comment in build.yml was a scar from that.
dial9 is now an opt-in feature (`dial9`, plus `dial9-s3` and `dial9-taskdump`),
the global rustflag is gone, and `crates/obs/build.rs` fails the compile if the
feature is on without the flag. Telemetry builds go through `make build-profiling`.
Metrics that could not lie
--------------------------
`rustfs_dial9_{events_total,bytes_written_total,rotations_total,cpu_overhead_percent}`
were hard-coded to zero — a Counter pinned at 0 reads as "nothing happened".
Removed. `rustfs_dial9_enabled` was sourced from the environment, so it read 1
even when the traced runtime failed and the process fell back to a standard
runtime; it is replaced by `rustfs_dial9_supported` (compile-time),
`rustfs_dial9_configured` (intent) and `rustfs_dial9_active_sessions` (reality).
No `writer_healthy` gauge is exported: dial9's `RotatingWriter` can enter its
`Finished` state and stop writing, but exposes no way to observe that, so the
gauge could only ever be hard-coded to 1. Documented as a known gap instead.
Final events were lost
----------------------
The `TelemetryGuard` lived in a `static OnceLock`, which is never dropped, so
buffered events were never flushed at exit. `build_tokio_runtime` now returns
the guard and `run_process` drops it before any exit path.
Also
----
- `disk_usage_bytes` was a `read_dir` + per-file `stat` on the metrics
collection path. It is now sampled by a background task into an atomic.
- `SAMPLING_RATE`/`S3_BUCKET`/`S3_PREFIX` were parsed, warned about, and
discarded. S3 upload is now wired to dial9's `with_s3_uploader` behind
`dial9-s3`; `SAMPLING_RATE` has no upstream equivalent and is removed.
- Wire `with_task_dumps` (async backtraces of stalled tasks), configurable via
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_{ENABLED,IDLE_THRESHOLD_MS}`.
- Split `telemetry/dial9.rs` into `config`/`state`/`enabled`/`disabled`; the
stub keeps the public API identical so callers need no `#[cfg]`.
- Drop four print-only examples and the manual test bin that exercised the
removed `init_session` scaffolding.
Verified: cargo check/clippy/test across default, `dial9`, and `dial9-s3`;
build.rs correctly rejects `dial9` without `--cfg tokio_unstable`;
`make pre-commit` passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): document dial9 as an on-demand profiler
scripts/run.sh advertised a `SAMPLING_RATE` knob that was never passed to dial9,
and claimed "CPU overhead < 5% (with sampling rate 1.0)" and "lower values reduce
CPU overhead" on the strength of it. The knob is gone; the guidance built on it
had to go too.
Replace it with what is actually true: dial9 needs a `make build-profiling`
binary, its disk budget evicts oldest-first (so a high poll rate can overwrite
the incident you are chasing), and it cannot be toggled without a restart.
Add docs/operations/dial9-runtime-profiling.md covering the build variants, an
investigation walkthrough, the configuration table, how to read the three
supported/configured/active_sessions gauges against each other, and the upstream
gap that makes writer death only indirectly observable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(obs): add a dial9 smoke example that proves events are recorded
The bug this guards against is invisible to every existing signal: with
`build` instead of `build_and_start`, dial9 creates the trace file, seals
segments, and reports `TelemetryGuard::is_enabled() == true` — it simply
records no events. Only the segment's byte count tells the two apart.
Measured on this workload: 5640 bytes when recording, 310 bytes (a bare
segment header) when not. The example asserts >= 2048 bytes, and was verified
to fail with the `build` call restored.
Also correct the comment on the `is_enabled` check in `finish_traced_runtime`.
It claimed to catch "recording silently off"; it does not. It only rejects the
inert guard a lenient config yields after a build failure. Recording is
guaranteed by `build_and_start`, not by that check.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): accept Unsupported runtime telemetry capability
A binary built without the `dial9` feature now reports the runtime-telemetry
capability as `Unsupported` rather than `Disabled`. The distinction matters to
operators: `Disabled` implies the capability can be switched on by setting an
environment variable, which is not true here — telemetry needs a rebuild.
Widen the assertion and pin the new semantics: when `dial9::is_supported()` is
false, the state must be exactly `Unsupported`.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): drop the dial9-s3 feature, its TLS stack is vulnerable
CI's Dependency Review and `cargo deny` both reject the branch: dial9's
`worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3, which pins
aws-smithy-http-client onto hyper-rustls 0.24 and rustls-webpki 0.101.7. That
webpki carries RUSTSEC-2026-0098, -0099 and -0104.
0.1.3 is the latest release of the transfer manager, and 1.2.0 the latest of the
smithy client, so there is nothing to upgrade to. Cargo's feature unification can
add features but cannot drop a transitive dependency, so it cannot be worked
around from here either — the rest of the workspace already resolves to the safe
rustls-webpki 0.103 / hyper-rustls 0.27.
Remove the `dial9-s3` feature and the `with_s3_uploader` wiring. The two S3
environment variables stay parsed and warned about, now naming the real reason
rather than a missing build feature. Trace segments are collected from the output
directory instead. Tracked as D9-14 in rustfs/backlog#1157.
With this, Cargo.lock is byte-identical to main: the PR no longer touches the
dependency graph at all.
Also correct the `dial9-taskdump` documentation. It claimed the feature "compiles
to a no-op elsewhere"; in fact `tokio/taskdump` raises a `compile_error!` on any
target other than linux/{aarch64,x86,x86_64}. Verified by trying to build it on
macOS, which is how the claim was found to be wrong.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
@@ -0,0 +1,138 @@
|
||||
# dial9 Tokio Runtime Profiling
|
||||
|
||||
`dial9-tokio-telemetry` records Tokio runtime-level events — poll start/end,
|
||||
worker park/unpark, task spawn/terminate, and optionally async backtraces of
|
||||
stalled tasks — into binary trace segments.
|
||||
|
||||
It answers questions that Prometheus metrics and `tracing` spans cannot:
|
||||
|
||||
- A request took 200 ms. Was a worker thread blocked, or was it waiting on I/O?
|
||||
- Which task held a worker for 40 ms without yielding?
|
||||
- Are workers parking because there is no work, or because the queue is starved?
|
||||
|
||||
These are the failure modes behind drive stalls, `io_uring`/`O_DIRECT` regressions,
|
||||
and object-data-cache fill contention. They are invisible to the rest of the obs stack.
|
||||
|
||||
## This is a profiler, not telemetry
|
||||
|
||||
Three properties follow from that, and all three matter operationally.
|
||||
|
||||
**The stock binary cannot run it.** dial9 hooks Tokio's unstable runtime API, which
|
||||
requires `--cfg tokio_unstable`. Release binaries are built without it, so they neither
|
||||
depend on Tokio's non-semver surface nor pay its cost. Setting
|
||||
`RUSTFS_RUNTIME_DIAL9_ENABLED=true` on a stock binary logs a warning and records nothing.
|
||||
|
||||
**Traces are written continuously and evicted.** The retained budget is
|
||||
`MAX_FILE_SIZE × ROTATION_COUNT` (1 GB by default). Once exceeded, the *oldest*
|
||||
segments are deleted. Under a high poll rate that budget can wrap in minutes,
|
||||
which means the incident you are chasing may already have been overwritten.
|
||||
Size the budget against the window you need, and prefer short, targeted runs.
|
||||
|
||||
**There is no runtime toggle.** Telemetry is installed when the Tokio runtime is
|
||||
constructed, so enabling or disabling it requires a process restart.
|
||||
|
||||
## Building
|
||||
|
||||
```bash
|
||||
make build-profiling
|
||||
```
|
||||
|
||||
That is `cargo build --release --bin rustfs --features dial9` with
|
||||
`RUSTFLAGS="--cfg tokio_unstable"`. One optional feature layers on top:
|
||||
|
||||
```bash
|
||||
# Async backtraces of stalled tasks. Linux aarch64/x86/x86_64 only —
|
||||
# `tokio/taskdump` hard-errors at compile time on other targets, so this
|
||||
# will not build on macOS.
|
||||
DIAL9_RUSTFLAGS="--cfg tokio_unstable --cfg tokio_taskdump" \
|
||||
DIAL9_FEATURES="dial9-taskdump" make build-profiling
|
||||
```
|
||||
|
||||
`crates/obs/build.rs` fails the build if the `dial9` feature is enabled without
|
||||
`--cfg tokio_unstable`. This is deliberate: an environment `RUSTFLAGS` *replaces*
|
||||
the value from `.cargo/config.toml` rather than appending to it, so the flag used
|
||||
to disappear silently whenever anything else set `RUSTFLAGS`.
|
||||
|
||||
For CPU profiling with usable stacks, add `-C force-frame-pointers=yes`.
|
||||
|
||||
## Running an investigation
|
||||
|
||||
```bash
|
||||
export RUSTFS_RUNTIME_DIAL9_ENABLED=true
|
||||
export RUSTFS_RUNTIME_DIAL9_OUTPUT_DIR=/tmp/rustfs-telemetry-investigation
|
||||
export RUSTFS_RUNTIME_DIAL9_ROTATION_COUNT=3 # 300 MB total, ~short window
|
||||
export RUSTFS_RUNTIME_DIAL9_TASK_DUMP_ENABLED=true # if built with dial9-taskdump
|
||||
```
|
||||
|
||||
Reproduce the fault, then stop the process **gracefully**. The final buffered
|
||||
events are flushed when the telemetry guard drops during shutdown; a `SIGKILL`
|
||||
loses them, and they are usually the interesting ones.
|
||||
|
||||
Segments land in `$OUTPUT_DIR/$FILE_PREFIX.N.bin`. Analyse them with upstream's
|
||||
`TRACE_ANALYSIS_GUIDE.md`.
|
||||
|
||||
Turn `RUSTFS_RUNTIME_DIAL9_ENABLED` back off when you are done.
|
||||
|
||||
## Configuration
|
||||
|
||||
| Variable | Default | Notes |
|
||||
|---|---|---|
|
||||
| `RUSTFS_RUNTIME_DIAL9_ENABLED` | `false` | Needs a `dial9` build to take effect |
|
||||
| `RUSTFS_RUNTIME_DIAL9_OUTPUT_DIR` | `/var/log/rustfs/telemetry` | Must be writable |
|
||||
| `RUSTFS_RUNTIME_DIAL9_FILE_PREFIX` | `rustfs-tokio` | |
|
||||
| `RUSTFS_RUNTIME_DIAL9_MAX_FILE_SIZE` | `104857600` | Bytes per segment |
|
||||
| `RUSTFS_RUNTIME_DIAL9_ROTATION_COUNT` | `10` | Total budget = size × count |
|
||||
| `RUSTFS_RUNTIME_DIAL9_TASK_DUMP_ENABLED` | `false` | Needs `dial9-taskdump` (Linux only) |
|
||||
| `RUSTFS_RUNTIME_DIAL9_TASK_DUMP_IDLE_THRESHOLD_MS` | `10` | Mean idle for Poisson sampling |
|
||||
| `RUSTFS_RUNTIME_DIAL9_S3_BUCKET` | unset | **Not honoured**, see below |
|
||||
| `RUSTFS_RUNTIME_DIAL9_S3_PREFIX` | unset | **Not honoured**, see below |
|
||||
|
||||
Variables whose build feature is absent are ignored, with a warning naming the
|
||||
feature to rebuild with.
|
||||
|
||||
### S3 upload is unavailable
|
||||
|
||||
dial9's `worker-s3` feature depends on `aws-sdk-s3-transfer-manager` 0.1.3 (its
|
||||
latest release), which pins `aws-smithy-http-client` onto `hyper-rustls` 0.24 and
|
||||
`rustls-webpki` 0.101.7. That webpki carries RUSTSEC-2026-0098, -0099 and -0104,
|
||||
and the repository's `cargo deny` gate rejects it.
|
||||
|
||||
Cargo's feature unification can add features but cannot drop a transitive
|
||||
dependency, so this cannot be worked around downstream — it needs an upstream
|
||||
release. RustFS therefore builds no S3 uploader at all. Setting the two variables
|
||||
above logs a warning and changes nothing; retrieve trace segments from
|
||||
`OUTPUT_DIR` directly. Tracked as D9-14 in rustfs/backlog#1157.
|
||||
|
||||
## Metrics
|
||||
|
||||
| Metric | Meaning |
|
||||
|---|---|
|
||||
| `rustfs_dial9_supported` | Binary was compiled with telemetry support |
|
||||
| `rustfs_dial9_configured` | Operator asked for telemetry via the environment |
|
||||
| `rustfs_dial9_active_sessions` | A session is actually recording (0 or 1) |
|
||||
| `rustfs_dial9_disk_usage_bytes` | Trace segment bytes on disk, refreshed every 60 s |
|
||||
| `rustfs_dial9_errors_total` | Telemetry setup failures |
|
||||
|
||||
The first three are separate on purpose, because they disagree in exactly the
|
||||
cases you need to diagnose:
|
||||
|
||||
- `configured=1, supported=0` — wrong binary. Rebuild with `make build-profiling`.
|
||||
- `configured=1, supported=1, active_sessions=0` — telemetry failed to start
|
||||
(usually an unwritable `OUTPUT_DIR`) and the process fell back to a standard
|
||||
runtime. Check `rustfs_dial9_errors_total` and the startup logs.
|
||||
|
||||
The session metrics are not exported at all unless telemetry is running. A counter
|
||||
pinned at zero reads as "nothing happened", which is worse than a missing series.
|
||||
|
||||
### Known gap: writer death is not directly observable
|
||||
|
||||
dial9's `RotatingWriter` stops accepting writes (its internal `Finished` state) if
|
||||
the output directory is removed underneath it or a segment cannot be sealed —
|
||||
a real risk in containers where `/var/log` gets cleaned. Upstream exposes no way
|
||||
to observe this: `TelemetryGuard::is_enabled()` reports how the session was *built*,
|
||||
not whether it is still writing.
|
||||
|
||||
There is therefore no `writer_healthy` gauge, because it could only ever be
|
||||
hard-coded to `1`. Instead, watch `rustfs_dial9_disk_usage_bytes`: a session with
|
||||
`active_sessions=1` whose disk usage has stopped growing has most likely hit this
|
||||
state, and needs a restart.
|
||||
Reference in New Issue
Block a user