gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-13 08:36:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(obs): make dial9 telemetry opt-in and actually record events (#4663)

Browse Source 
* refactor(obs): make dial9 telemetry opt-in and actually record events

The dial9 Tokio-runtime profiler was disabled by default, yet every build
paid for it, and enabling it produced trace files with no events in them.

Recorded empty traces
---------------------
`build_traced_runtime` called `TracedRuntime::builder()...build(..)`, but dial9
only starts recording in `build_and_start*`. `build` still returns a live guard
whose `is_enabled()` reports true, and still creates and seals segment files —
they just contain a header and no events. It also skipped `with_trace_path`, so
the background worker driving the segment pipeline was never spawned.

Measured on the new smoke example: 310 bytes of bare segment header, against
5640 bytes for the same workload once recording actually starts.

Switch to `with_trace_path(..).build_and_start(..)`.

Cost was unconditional
----------------------
`--cfg tokio_unstable` was a global `[build] rustflags` entry and `rustfs-obs`
depended on `dial9-tokio-telemetry` unconditionally, so all builds depended on
Tokio's non-semver API. Worse, an environment `RUSTFLAGS` replaces (never
appends to) the config-file value, so any caller exporting their own RUSTFLAGS
silently dropped the flag — the long comment in build.yml was a scar from that.

dial9 is now an opt-in feature (`dial9`, plus `dial9-s3` and `dial9-taskdump`),
the global rustflag is gone, and `crates/obs/build.rs` fails the compile if the
feature is on without the flag. Telemetry builds go through `make build-profiling`.

Metrics that could not lie
--------------------------
`rustfs_dial9_{events_total,bytes_written_total,rotations_total,cpu_overhead_percent}`
were hard-coded to zero — a Counter pinned at 0 reads as "nothing happened".
Removed. `rustfs_dial9_enabled` was sourced from the environment, so it read 1
even when the traced runtime failed and the process fell back to a standard
runtime; it is replaced by `rustfs_dial9_supported` (compile-time),
`rustfs_dial9_configured` (intent) and `rustfs_dial9_active_sessions` (reality).

No `writer_healthy` gauge is exported: dial9's `RotatingWriter` can enter its
`Finished` state and stop writing, but exposes no way to observe that, so the
gauge could only ever be hard-coded to 1. Documented as a known gap instead.

Final events were lost
----------------------
The `TelemetryGuard` lived in a `static OnceLock`, which is never dropped, so
buffered events were never flushed at exit. `build_tokio_runtime` now returns
the guard and `run_process` drops it before any exit path.

Also
----
- `disk_usage_bytes` was a `read_dir` + per-file `stat` on the metrics
  collection path. It is now sampled by a background task into an atomic.
- `SAMPLING_RATE`/`S3_BUCKET`/`S3_PREFIX` were parsed, warned about, and
  discarded. S3 upload is now wired to dial9's `with_s3_uploader` behind
  `dial9-s3`; `SAMPLING_RATE` has no upstream equivalent and is removed.
- Wire `with_task_dumps` (async backtraces of stalled tasks), configurable via
  `RUSTFS_RUNTIME_DIAL9_TASK_DUMP_{ENABLED,IDLE_THRESHOLD_MS}`.
- Split `telemetry/dial9.rs` into `config`/`state`/`enabled`/`disabled`; the
  stub keeps the public API identical so callers need no `#[cfg]`.
- Drop four print-only examples and the manual test bin that exercised the
  removed `init_session` scaffolding.

Verified: cargo check/clippy/test across default, `dial9`, and `dial9-s3`;
build.rs correctly rejects `dial9` without `--cfg tokio_unstable`;
`make pre-commit` passes.

Co-Authored-By: heihutu <heihutu@gmail.com>

* docs(obs): document dial9 as an on-demand profiler

scripts/run.sh advertised a `SAMPLING_RATE` knob that was never passed to dial9,
and claimed "CPU overhead < 5% (with sampling rate 1.0)" and "lower values reduce
CPU overhead" on the strength of it. The knob is gone; the guidance built on it
had to go too.

Replace it with what is actually true: dial9 needs a `make build-profiling`
binary, its disk budget evicts oldest-first (so a high poll rate can overwrite
the incident you are chasing), and it cannot be toggled without a restart.

Add docs/operations/dial9-runtime-profiling.md covering the build variants, an
investigation walkthrough, the configuration table, how to read the three
supported/configured/active_sessions gauges against each other, and the upstream
gap that makes writer death only indirectly observable.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(obs): add a dial9 smoke example that proves events are recorded

The bug this guards against is invisible to every existing signal: with
`build` instead of `build_and_start`, dial9 creates the trace file, seals
segments, and reports `TelemetryGuard::is_enabled() == true` — it simply
records no events. Only the segment's byte count tells the two apart.

Measured on this workload: 5640 bytes when recording, 310 bytes (a bare
segment header) when not. The example asserts >= 2048 bytes, and was verified
to fail with the `build` call restored.

Also correct the comment on the `is_enabled` check in `finish_traced_runtime`.
It claimed to catch "recording silently off"; it does not. It only rejects the
inert guard a lenient config yields after a build failure. Recording is
guaranteed by `build_and_start`, not by that check.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(rustfs): accept Unsupported runtime telemetry capability

A binary built without the `dial9` feature now reports the runtime-telemetry
capability as `Unsupported` rather than `Disabled`. The distinction matters to
operators: `Disabled` implies the capability can be switched on by setting an
environment variable, which is not true here — telemetry needs a rebuild.

Widen the assertion and pin the new semantics: when `dial9::is_supported()` is
false, the state must be exactly `Unsupported`.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): drop the dial9-s3 feature, its TLS stack is vulnerable

CI's Dependency Review and `cargo deny` both reject the branch: dial9's
`worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3, which pins
aws-smithy-http-client onto hyper-rustls 0.24 and rustls-webpki 0.101.7. That
webpki carries RUSTSEC-2026-0098, -0099 and -0104.

0.1.3 is the latest release of the transfer manager, and 1.2.0 the latest of the
smithy client, so there is nothing to upgrade to. Cargo's feature unification can
add features but cannot drop a transitive dependency, so it cannot be worked
around from here either — the rest of the workspace already resolves to the safe
rustls-webpki 0.103 / hyper-rustls 0.27.

Remove the `dial9-s3` feature and the `with_s3_uploader` wiring. The two S3
environment variables stay parsed and warned about, now naming the real reason
rather than a missing build feature. Trace segments are collected from the output
directory instead. Tracked as D9-14 in rustfs/backlog#1157.

With this, Cargo.lock is byte-identical to main: the PR no longer touches the
dependency graph at all.

Also correct the `dial9-taskdump` documentation. It claimed the feature "compiles
to a no-op elsewhere"; in fact `tokio/taskdump` raises a `compile_error!` on any
target other than linux/{aarch64,x86,x86_64}. Verified by trying to build it on
macOS, which is how the claim was found to be wrong.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
2026-07-10 18:52:48 +08:00
committed by GitHub
parent f83f9ada13
commit 00536da80c
30 changed files with 1406 additions and 1197 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/obs/src/metrics/collectors/dial9.rs
+82 -115
View File
@@ -14,100 +14,65 @@
//! dial9 Tokio runtime telemetry metrics collector.
//!
//! This module provides metrics for monitoring the health and performance
//! of the dial9 telemetry system itself.
#![allow(dead_code)]
//! Reports the health of the telemetry system itself, not the runtime events it
//! records — those live in the trace segments on disk.
//!
//! Every metric here is backed by a value the process actually observes. A
//! counter that cannot be sourced is not exported at all: a series pinned at
//! zero reads as "nothing happened", which is worse than a missing series.
use crate::MetricType;
use crate::metrics::report::PrometheusMetric;
use crate::telemetry::dial9::runtime_stats_snapshot;
use rustfs_config::{DEFAULT_RUNTIME_DIAL9_ENABLED, ENV_RUNTIME_DIAL9_ENABLED};
use rustfs_utils::get_env_bool;
use crate::telemetry::dial9::{is_configured, is_enabled, is_supported, runtime_stats_snapshot};
/// Dial9 telemetry system statistics.
/// Health of the dial9 telemetry system.
#[derive(Debug, Clone, Default)]
pub struct Dial9Stats {
/// Total number of telemetry events recorded
pub events_total: u64,
/// Total bytes written to trace files
pub bytes_written: u64,
/// Number of file rotations that have occurred
pub rotation_count: u64,
/// Total number of dial9 errors
/// Cumulative count of telemetry setup failures.
pub errors_total: u64,
/// Estimated CPU overhead percentage (if available)
pub cpu_overhead_percent: f64,
/// Current disk usage by trace files in bytes
/// Bytes on disk held by trace segments, as of the last background refresh.
pub disk_usage_bytes: u64,
/// Number of active sessions
/// Number of active telemetry sessions (0 or 1).
pub active_sessions: u64,
}
/// Collect dial9 telemetry metrics.
/// Convert dial9 health statistics into Prometheus metrics.
///
/// This function converts dial9 statistics into Prometheus metrics format.
///
/// # Arguments
///
/// * `stats` - Dial9 statistics to report
///
/// # Returns
///
/// A vector of Prometheus metrics for dial9 telemetry statistics.
/// `rustfs_dial9_supported` reflects compile-time support, `rustfs_dial9_configured`
/// reflects operator intent, and `rustfs_dial9_active_sessions` reflects reality.
/// They disagree when a build lacks the feature, or when the traced runtime
/// failed to build and the process fell back to a standard runtime — read all
/// three before concluding that telemetry is running.
pub fn collect_dial9_metrics(stats: &Dial9Stats) -> Vec<PrometheusMetric> {
let enabled = is_dial9_enabled();
let enabled_value = if enabled { 1.0 } else { 0.0 };
let mut metrics = vec![
PrometheusMetric::new(
"rustfs_dial9_supported",
MetricType::Gauge,
"Whether this binary was compiled with dial9 telemetry support (1) or not (0)",
bool_gauge(is_supported()),
),
PrometheusMetric::new(
"rustfs_dial9_configured",
MetricType::Gauge,
"Whether dial9 telemetry is requested via environment configuration (1) or not (0)",
bool_gauge(is_configured()),
),
];
let mut metrics = vec![PrometheusMetric::new(
"rustfs_dial9_enabled",
MetricType::Gauge,
"Whether dial9 telemetry is enabled (1) or disabled (0)",
enabled_value,
)];
// If dial9 is disabled, return just the enabled flag
if !enabled {
// Without a running session the remaining series carry no information;
// exporting them would only publish zeros.
if !is_enabled() {
return metrics;
}
// Add detailed metrics when enabled
metrics.extend(vec![
metrics.extend([
PrometheusMetric::new(
"rustfs_dial9_events_total",
MetricType::Counter,
"Total number of Tokio runtime events recorded by dial9",
stats.events_total as f64,
),
PrometheusMetric::new(
"rustfs_dial9_bytes_written_total",
MetricType::Counter,
"Total bytes written to dial9 trace files",
stats.bytes_written as f64,
),
PrometheusMetric::new(
"rustfs_dial9_rotations_total",
MetricType::Counter,
"Total number of trace file rotations",
stats.rotation_count as f64,
),
PrometheusMetric::new(
"rustfs_dial9_errors_total",
MetricType::Counter,
"Total number of dial9 telemetry errors",
stats.errors_total as f64,
),
PrometheusMetric::new(
"rustfs_dial9_cpu_overhead_percent",
"rustfs_dial9_active_sessions",
MetricType::Gauge,
"Estimated CPU overhead percentage from dial9 telemetry",
stats.cpu_overhead_percent,
"Number of active dial9 telemetry sessions",
stats.active_sessions as f64,
),
PrometheusMetric::new(
"rustfs_dial9_disk_usage_bytes",
@@ -116,80 +81,82 @@ pub fn collect_dial9_metrics(stats: &Dial9Stats) -> Vec<PrometheusMetric> {
stats.disk_usage_bytes as f64,
),
PrometheusMetric::new(
"rustfs_dial9_active_sessions",
MetricType::Gauge,
"Number of active dial9 telemetry sessions",
stats.active_sessions as f64,
"rustfs_dial9_errors_total",
MetricType::Counter,
"Total number of dial9 telemetry setup errors",
stats.errors_total as f64,
),
]);
metrics
}
fn bool_gauge(value: bool) -> f64 {
if value { 1.0 } else { 0.0 }
}
/// Collect dial9 metrics from the current runtime snapshot.
///
/// Reads cached atomics only; performs no I/O.
pub fn collect_current_dial9_metrics() -> Vec<PrometheusMetric> {
let snapshot = runtime_stats_snapshot();
let stats = Dial9Stats {
errors_total: snapshot.errors_total,
disk_usage_bytes: snapshot.disk_usage_bytes,
active_sessions: snapshot.active_sessions,
..Dial9Stats::default()
};
collect_dial9_metrics(&stats)
}
/// Check if dial9 telemetry is enabled via environment variable.
/// Whether dial9 telemetry is actually running: compiled in *and* configured.
pub fn is_dial9_enabled() -> bool {
get_env_bool(ENV_RUNTIME_DIAL9_ENABLED, DEFAULT_RUNTIME_DIAL9_ENABLED)
is_enabled()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_dial9_stats_default() {
let stats = Dial9Stats::default();
assert_eq!(stats.events_total, 0);
assert_eq!(stats.bytes_written, 0);
assert_eq!(stats.rotation_count, 0);
assert_eq!(stats.errors_total, 0);
assert_eq!(stats.cpu_overhead_percent, 0.0);
assert_eq!(stats.disk_usage_bytes, 0);
assert_eq!(stats.active_sessions, 0);
fn metric_names(metrics: &[PrometheusMetric]) -> Vec<String> {
metrics.iter().map(|m| m.name.to_string()).collect()
}
#[test]
fn test_collect_dial9_metrics() {
let stats = Dial9Stats {
events_total: 100,
bytes_written: 1024,
..Default::default()
};
let metrics = collect_dial9_metrics(&stats);
// Should always have at least the enabled flag
assert!(!metrics.is_empty());
fn always_reports_support_and_intent() {
let names = metric_names(&collect_dial9_metrics(&Dial9Stats::default()));
assert!(names.iter().any(|n| n == "rustfs_dial9_supported"));
assert!(names.iter().any(|n| n == "rustfs_dial9_configured"));
}
#[test]
fn test_collect_dial9_metrics_with_values() {
let stats = Dial9Stats {
events_total: 10000,
bytes_written: 1024000,
rotation_count: 5,
errors_total: 0,
cpu_overhead_percent: 2.5,
disk_usage_bytes: 2048000,
active_sessions: 1,
};
fn omits_session_metrics_when_telemetry_is_not_running() {
// Under `cargo test` dial9 is neither compiled in nor configured, so the
// collector must stop after the two compile-time/intent gauges rather
// than publish zeroed session series.
if is_enabled() {
return;
}
let metrics = collect_dial9_metrics(&Dial9Stats::default());
assert_eq!(metrics.len(), 2);
let names = metric_names(&metrics);
assert!(!names.iter().any(|n| n == "rustfs_dial9_disk_usage_bytes"));
assert!(!names.iter().any(|n| n == "rustfs_dial9_active_sessions"));
}
let metrics = collect_dial9_metrics(&stats);
#[test]
fn supported_gauge_tracks_compile_time_feature() {
let metrics = collect_dial9_metrics(&Dial9Stats::default());
let supported = metrics
.iter()
.find(|m| m.name == "rustfs_dial9_supported")
.expect("supported gauge is always exported");
assert_eq!(supported.value, bool_gauge(cfg!(feature = "dial9")));
}
// When dial9 is enabled, should have all metrics
// Note: This test assumes dial9 is enabled in the test environment
// If disabled, only the enabled flag metric will be present
assert!(!metrics.is_empty());
#[test]
fn bool_gauge_maps_to_one_and_zero() {
assert_eq!(bool_gauge(true), 1.0);
assert_eq!(bool_gauge(false), 0.0);
}
}