# dial9 Tokio Runtime Profiling

`dial9-tokio-telemetry` records Tokio runtime-level events — poll start/end,
worker park/unpark, task spawn/terminate, and optionally async backtraces of
stalled tasks — into binary trace segments.

It answers questions that Prometheus metrics and `tracing` spans cannot:

- A request took 200 ms. Was a worker thread blocked, or was it waiting on I/O?
- Which task held a worker for 40 ms without yielding?
- Are workers parking because there is no work, or because the queue is starved?

These are the failure modes behind drive stalls, `io_uring`/`O_DIRECT` regressions,
and object-data-cache fill contention. They are invisible to the rest of the obs stack.

## This is a profiler, not telemetry

Three properties follow from that, and all three matter operationally.

**The stock binary cannot run it.** dial9 hooks Tokio's unstable runtime API, which
requires `--cfg tokio_unstable`. Release binaries are built without it, so they neither
depend on Tokio's non-semver surface nor pay its cost. Setting
`RUSTFS_RUNTIME_DIAL9_ENABLED=true` on a stock binary logs a warning and records nothing.

**Traces are written continuously and evicted.** The retained budget is
`MAX_FILE_SIZE × ROTATION_COUNT` (1 GB by default). Once exceeded, the *oldest*
segments are deleted. Under a high poll rate that budget can wrap in minutes,
which means the incident you are chasing may already have been overwritten.
Size the budget against the window you need, and prefer short, targeted runs.

**There is no runtime toggle.** Telemetry is installed when the Tokio runtime is
constructed, so enabling or disabling it requires a process restart.

## Building

```bash
make build-profiling
```

That is `cargo build --release --bin rustfs --features dial9` with
`RUSTFLAGS="--cfg tokio_unstable"`. One optional feature layers on top:

```bash
# Async backtraces of stalled tasks. Linux aarch64/x86/x86_64 only —
# `tokio/taskdump` hard-errors at compile time on other targets, so this
# will not build on macOS.
DIAL9_RUSTFLAGS="--cfg tokio_unstable --cfg tokio_taskdump" \
  DIAL9_FEATURES="dial9-taskdump" make build-profiling
```

`crates/obs/build.rs` fails the build if the `dial9` feature is enabled without
`--cfg tokio_unstable`. This is deliberate: an environment `RUSTFLAGS` *replaces*
the value from `.cargo/config.toml` rather than appending to it, so the flag used
to disappear silently whenever anything else set `RUSTFLAGS`.

For CPU profiling with usable stacks, add `-C force-frame-pointers=yes`.

## Running an investigation

```bash
export RUSTFS_RUNTIME_DIAL9_ENABLED=true
export RUSTFS_RUNTIME_DIAL9_OUTPUT_DIR=/tmp/rustfs-telemetry-investigation
export RUSTFS_RUNTIME_DIAL9_ROTATION_COUNT=3      # 300 MB total, ~short window
export RUSTFS_RUNTIME_DIAL9_TASK_DUMP_ENABLED=true # if built with dial9-taskdump
```

Reproduce the fault, then stop the process **gracefully**. The final buffered
events are flushed when the telemetry guard drops during shutdown; a `SIGKILL`
loses them, and they are usually the interesting ones.

Segments land in `$OUTPUT_DIR/$FILE_PREFIX.N.bin`. Analyse them with upstream's
`TRACE_ANALYSIS_GUIDE.md`.

Turn `RUSTFS_RUNTIME_DIAL9_ENABLED` back off when you are done.

## Configuration

| Variable | Default | Notes |
|---|---|---|
| `RUSTFS_RUNTIME_DIAL9_ENABLED` | `false` | Needs a `dial9` build to take effect |
| `RUSTFS_RUNTIME_DIAL9_OUTPUT_DIR` | `/var/log/rustfs/telemetry` | Must be writable |
| `RUSTFS_RUNTIME_DIAL9_FILE_PREFIX` | `rustfs-tokio` | |
| `RUSTFS_RUNTIME_DIAL9_MAX_FILE_SIZE` | `104857600` | Bytes per segment |
| `RUSTFS_RUNTIME_DIAL9_ROTATION_COUNT` | `10` | Total budget = size × count |
| `RUSTFS_RUNTIME_DIAL9_TASK_DUMP_ENABLED` | `false` | Needs `dial9-taskdump` (Linux only) |
| `RUSTFS_RUNTIME_DIAL9_TASK_DUMP_IDLE_THRESHOLD_MS` | `10` | Mean idle for Poisson sampling |
| `RUSTFS_RUNTIME_DIAL9_S3_BUCKET` | unset | **Not honoured**, see below |
| `RUSTFS_RUNTIME_DIAL9_S3_PREFIX` | unset | **Not honoured**, see below |

Variables whose build feature is absent are ignored, with a warning naming the
feature to rebuild with.

### S3 upload is unavailable

dial9's `worker-s3` feature depends on `aws-sdk-s3-transfer-manager` 0.1.3 (its
latest release), which pins `aws-smithy-http-client` onto `hyper-rustls` 0.24 and
`rustls-webpki` 0.101.7. That webpki carries RUSTSEC-2026-0098, -0099 and -0104,
and the repository's `cargo deny` gate rejects it.

Cargo's feature unification can add features but cannot drop a transitive
dependency, so this cannot be worked around downstream — it needs an upstream
release. RustFS therefore builds no S3 uploader at all. Setting the two variables
above logs a warning and changes nothing; retrieve trace segments from
`OUTPUT_DIR` directly. Tracked as D9-14 in rustfs/backlog#1157.

## Metrics

| Metric | Meaning |
|---|---|
| `rustfs_dial9_supported` | Binary was compiled with telemetry support |
| `rustfs_dial9_configured` | Operator asked for telemetry via the environment |
| `rustfs_dial9_active_sessions` | A session is actually recording (0 or 1) |
| `rustfs_dial9_disk_usage_bytes` | Trace segment bytes on disk, refreshed every 60 s |
| `rustfs_dial9_errors_total` | Telemetry setup failures |

The first three are separate on purpose, because they disagree in exactly the
cases you need to diagnose:

- `configured=1, supported=0` — wrong binary. Rebuild with `make build-profiling`.
- `configured=1, supported=1, active_sessions=0` — telemetry failed to start
  (usually an unwritable `OUTPUT_DIR`) and the process fell back to a standard
  runtime. Check `rustfs_dial9_errors_total` and the startup logs.

The session metrics are not exported at all unless telemetry is running. A counter
pinned at zero reads as "nothing happened", which is worse than a missing series.

### Known gap: writer death is not directly observable

dial9's `RotatingWriter` stops accepting writes (its internal `Finished` state) if
the output directory is removed underneath it or a segment cannot be sealed —
a real risk in containers where `/var/log` gets cleaned. Upstream exposes no way
to observe this: `TelemetryGuard::is_enabled()` reports how the session was *built*,
not whether it is still writing.

There is therefore no `writer_healthy` gauge, because it could only ever be
hard-coded to `1`. Instead, watch `rustfs_dial9_disk_usage_bytes`: a session with
`active_sessions=1` whose disk usage has stopped growing has most likely hit this
state, and needs a restart.