gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-26 08:18:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(obs): remove the dial9 task-dump switch that could never work; correct measured claims (#4688)

Browse Source 
* fix(obs): remove the dial9 task-dump switch that could never do anything

Measured on a bench host (Linux x86_64) against the code merged in #4663: with
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_ENABLED=true`, a `dial9-taskdump` build, and
`--cfg tokio_taskdump`, dial9 recorded **zero** TaskDump events.

dial9 captures a task dump only for futures it wrapped itself — those spawned
through `dial9_tokio_telemetry::spawn`, which is where `TaskDumped<F>` gets
applied. `tokio::spawn` gets no wrapper, and RustFS spawns with `tokio::spawn`
throughout. Same workload, same binary, only the spawner changed:

    tokio::spawn   ->      0 dumps
    dial9::spawn   ->  14709 dumps, all with callchains

Upstream documents this (README line 151) and tracks the doc gap at
dial9-rs/dial9#477. I did not read it before wiring `with_task_dumps` in #4663,
and so shipped exactly the kind of lying configuration knob that PR set out to
delete. Remove it: the two environment variables, the config fields, the
`with_task_dumps` call, and the `dial9-taskdump` feature — whose only effect was
to constrain the build to Linux while recording nothing.

Re-adding it only makes sense together with migrating the paths under
investigation to dial9's spawner. Tracked as D9-16 in rustfs/backlog#1157.

Also drop the `--cfg tokio_taskdump` requirement from the Makefile. Measured:
dumps are captured with and without it (14709 vs 14674, within noise), and
upstream never asked for it. That requirement was mine, invented and untested.

Cargo.lock loses tokio's `backtrace` dependency, which `tokio/taskdump` pulled in.

Co-Authored-By: heihutu <heihutu@gmail.com>

* docs(obs): replace guessed dial9 retention numbers with measured ones

Three corrections, all to claims I wrote in #4663 without measuring them.

"Under a high poll rate that budget can wrap in minutes" was a guess. Measured on
a single-node 4-drive cluster under warp mixed (66 MiB/s, 110 obj/s, 32 concurrent):
13023 events/s, 0.16 MiB/s, so the default 1 GiB budget wraps after roughly 108
minutes. Even at ten times the throughput that is ~11 minutes. State the measured
rate and how to scale it instead.

dial9 was described as the tool for drive stalls. It is not. RustFS does disk I/O
on the blocking pool and through io_uring, never on an async worker, so a slow
drive never lengthens a poll. Injecting 200 ms of latency on one of four drives
cut throughput by 64% and left the poll distribution unchanged (polls >= 5 ms:
49 -> 56; p999: 2.67 ms -> 2.75 ms). Enabling dial9's CPU and sched profilers
does not help: sched events are per-worker only, and the CPU profiler samples
on-CPU while a stalled drive is an off-CPU wait. Say so plainly, and point at
the `rustfs_io_*` metrics instead.

What dial9 *is* good for, on the same traces: single polls of 418-625 ms with no
fault injected at all — real worker stalls nothing else in the obs stack surfaces.
Lead with that.

Also link the two upstream issues filed for the gaps we documented:
dial9-rs/dial9#658 (writer death unobservable) and #659 (worker-s3 CVEs).

Measurements: rustfs/backlog#1157 (D9-11, D9-13, D9-18).

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
2026-07-11 01:34:59 +08:00
committed by GitHub
parent 6f05a740b3
commit 60ad15a7f9
10 changed files with 87 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/obs/Cargo.toml
+9 -5
View File
@@ -31,11 +31,15 @@ default = []
# build script fails the compile when that flag is missing. Off by default so
# ordinary builds neither pay for nor depend on Tokio's unstable API.
dial9 = ["dep:dial9-tokio-telemetry"]
# Capture async backtraces of stalled tasks. Enables `tokio/taskdump`, which
# hard-errors at compile time on anything other than linux/{aarch64,x86,x86_64},
# and additionally needs `--cfg tokio_taskdump` to record anything. Linux-only,
# and not a silent no-op elsewhere: enabling it on macOS fails the build.
dial9-taskdump = ["dial9", "dial9-tokio-telemetry/taskdump"]
#
# NOTE: there is deliberately no `dial9-taskdump` feature. dial9 only captures a
# task dump for futures it wrapped itself, i.e. those spawned via
# `dial9_tokio_telemetry::spawn`. RustFS spawns with `tokio::spawn` throughout,
# so enabling `tokio/taskdump` would cost a Linux-only build constraint and
# record nothing. Measured on an identical workload: 0 dumps via `tokio::spawn`,
# 14709 via `dial9::spawn`. Re-adding this feature only makes sense together
# with migrating the paths under investigation to dial9's spawner.
# See rustfs/backlog#1157 (D9-16) and dial9-rs/dial9#477.
#
# NOTE: there is deliberately no `dial9-s3` feature. dial9's `worker-s3` feature
# pulls aws-sdk-s3-transfer-manager 0.1.3 (its latest), which pins
crates/obs/src/telemetry/dial9/config.rs
+3 -19
View File
@@ -22,14 +22,12 @@
use super::state::dial9_runtime_state;
use rustfs_config::{
DEFAULT_RUNTIME_DIAL9_ENABLED, DEFAULT_RUNTIME_DIAL9_FILE_PREFIX, DEFAULT_RUNTIME_DIAL9_MAX_FILE_SIZE,
DEFAULT_RUNTIME_DIAL9_OUTPUT_DIR, DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT, DEFAULT_RUNTIME_DIAL9_TASK_DUMP_ENABLED,
DEFAULT_RUNTIME_DIAL9_TASK_DUMP_IDLE_THRESHOLD_MS, ENV_RUNTIME_DIAL9_ENABLED, ENV_RUNTIME_DIAL9_FILE_PREFIX,
ENV_RUNTIME_DIAL9_MAX_FILE_SIZE, ENV_RUNTIME_DIAL9_OUTPUT_DIR, ENV_RUNTIME_DIAL9_ROTATION_COUNT, ENV_RUNTIME_DIAL9_S3_BUCKET,
ENV_RUNTIME_DIAL9_S3_PREFIX, ENV_RUNTIME_DIAL9_TASK_DUMP_ENABLED, ENV_RUNTIME_DIAL9_TASK_DUMP_IDLE_THRESHOLD_MS,
DEFAULT_RUNTIME_DIAL9_OUTPUT_DIR, DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT, ENV_RUNTIME_DIAL9_ENABLED,
ENV_RUNTIME_DIAL9_FILE_PREFIX, ENV_RUNTIME_DIAL9_MAX_FILE_SIZE, ENV_RUNTIME_DIAL9_OUTPUT_DIR,
ENV_RUNTIME_DIAL9_ROTATION_COUNT, ENV_RUNTIME_DIAL9_S3_BUCKET, ENV_RUNTIME_DIAL9_S3_PREFIX,
};
use rustfs_utils::{get_env_bool, get_env_opt_str, get_env_opt_u64, get_env_opt_usize, get_env_str};
use std::path::PathBuf;
use std::time::Duration;
use tracing::warn;
use super::{EVENT_DIAL9_STATE, LOG_COMPONENT_OBS, LOG_SUBSYSTEM_DIAL9};
@@ -92,12 +90,6 @@ pub struct Dial9Config {
/// Optional key prefix for uploaded segments
pub s3_prefix: Option<String>,
/// Whether to capture async backtraces for tasks that stall
pub task_dump_enabled: bool,
/// Mean idle duration for task-dump Poisson sampling
pub task_dump_idle_threshold: Duration,
}
impl Default for Dial9Config {
@@ -110,8 +102,6 @@ impl Default for Dial9Config {
rotation_count: DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT,
s3_bucket: None,
s3_prefix: None,
task_dump_enabled: DEFAULT_RUNTIME_DIAL9_TASK_DUMP_ENABLED,
task_dump_idle_threshold: Duration::from_millis(DEFAULT_RUNTIME_DIAL9_TASK_DUMP_IDLE_THRESHOLD_MS),
}
}
}
@@ -165,12 +155,6 @@ impl Dial9Config {
rotation_count,
s3_bucket,
s3_prefix,
task_dump_enabled: get_env_bool(ENV_RUNTIME_DIAL9_TASK_DUMP_ENABLED, DEFAULT_RUNTIME_DIAL9_TASK_DUMP_ENABLED),
task_dump_idle_threshold: Duration::from_millis(
get_env_opt_u64(ENV_RUNTIME_DIAL9_TASK_DUMP_IDLE_THRESHOLD_MS)
.filter(|ms| *ms > 0)
.unwrap_or(DEFAULT_RUNTIME_DIAL9_TASK_DUMP_IDLE_THRESHOLD_MS),
),
};
dial9_runtime_state().record_config(&config);
crates/obs/src/telemetry/dial9/enabled.rs
+8 -12
View File
@@ -23,7 +23,7 @@ use super::config::Dial9Config;
use super::state::{dial9_runtime_state, measure_disk_usage_bytes};
use super::{EVENT_DIAL9_STATE, LOG_COMPONENT_OBS, LOG_SUBSYSTEM_DIAL9};
use crate::TelemetryError;
use dial9_tokio_telemetry::telemetry::{ProcessResourceUsageConfig, RotatingWriter, TaskDumpConfig, TracedRuntime};
use dial9_tokio_telemetry::telemetry::{ProcessResourceUsageConfig, RotatingWriter, TracedRuntime};
use std::time::Duration;
use tracing::{info, warn};
@@ -115,19 +115,16 @@ pub fn build_traced_runtime(
.with_runtime_name(RUNTIME_NAME)
.with_process_resource_usage(ProcessResourceUsageConfig::default());
let traced = if config.task_dump_enabled {
traced.with_task_dumps(
TaskDumpConfig::builder()
.idle_threshold(config.task_dump_idle_threshold)
.build(),
)
} else {
traced
};
// `build_and_start` rather than `build`: `build` returns a live guard that
// never records, writing segments that contain only a header.
//
// No `with_task_dumps` here. dial9 captures a task dump only for futures it
// wrapped itself, i.e. those spawned via `dial9_tokio_telemetry::spawn`;
// `tokio::spawn` gets no wrapper. RustFS spawns with `tokio::spawn`
// throughout, so calling `with_task_dumps` records nothing. Measured on an
// identical workload: 0 dumps via `tokio::spawn`, 14709 via `dial9::spawn`.
// See rustfs/backlog#1157 (D9-16) and dial9-rs/dial9#477.
//
// No `with_s3_uploader` here: dial9's `worker-s3` feature carries a
// vulnerable TLS stack. See the note in `crates/obs/Cargo.toml`.
finish_traced_runtime(traced.build_and_start(builder, writer), config)
@@ -165,7 +162,6 @@ fn finish_traced_runtime(
output_dir = %config.output_dir,
file_prefix = %config.file_prefix,
disk_budget_bytes = config.total_disk_budget(),
task_dumps = config.task_dump_enabled,
"dial9 state changed"
);
crates/obs/src/telemetry/dial9/mod.rs
+16 -2
View File
@@ -31,8 +31,21 @@
//!
//! Trace segments are written to disk continuously and evicted oldest-first
//! once `RUSTFS_RUNTIME_DIAL9_MAX_FILE_SIZE * RUSTFS_RUNTIME_DIAL9_ROTATION_COUNT`
//! bytes are retained. Under a high poll rate the budget can wrap in minutes,
//! so size it against the window you need to capture.
//! bytes are retained. Measured at ~0.16 MiB/s (13k events/s) under a 66 MiB/s
//! warp workload, so the default 1 GiB budget wraps after roughly 108 minutes.
//! Scale that by your own event rate before relying on a long capture.
//!
//! # What it does not see
//!
//! A drive stall does not show up here. RustFS performs disk I/O on the blocking
//! pool and through io_uring, never on an async worker, so a slow drive does not
//! lengthen any poll. Injecting 200 ms of drive latency cut throughput by 64%
//! and left the poll-duration distribution unchanged. Use the `rustfs_io_*`
//! metrics and the drive-stall budget for that; dial9 answers a different
//! question — which task held a worker, and for how long.
//!
//! Task dumps would answer *where* it was stuck, but dial9 only captures them
//! for futures spawned through its own `spawn`. See rustfs/backlog#1157 (D9-16).
//!
//! # Known observability gap
//!
@@ -43,6 +56,7 @@
//! is therefore no `writer_healthy` metric: it could only ever be hard-coded to
//! `1`. Watch `rustfs_dial9_disk_usage_bytes` — a session that is recording but
//! whose disk usage stops growing has most likely hit this state.
//! Reported upstream as dial9-rs/dial9#658.
mod config;
mod state;