refactor(obs): make dial9 telemetry opt-in and actually record events (#4663)

* refactor(obs): make dial9 telemetry opt-in and actually record events

The dial9 Tokio-runtime profiler was disabled by default, yet every build
paid for it, and enabling it produced trace files with no events in them.

Recorded empty traces
---------------------
`build_traced_runtime` called `TracedRuntime::builder()...build(..)`, but dial9
only starts recording in `build_and_start*`. `build` still returns a live guard
whose `is_enabled()` reports true, and still creates and seals segment files —
they just contain a header and no events. It also skipped `with_trace_path`, so
the background worker driving the segment pipeline was never spawned.

Measured on the new smoke example: 310 bytes of bare segment header, against
5640 bytes for the same workload once recording actually starts.

Switch to `with_trace_path(..).build_and_start(..)`.

Cost was unconditional
----------------------
`--cfg tokio_unstable` was a global `[build] rustflags` entry and `rustfs-obs`
depended on `dial9-tokio-telemetry` unconditionally, so all builds depended on
Tokio's non-semver API. Worse, an environment `RUSTFLAGS` replaces (never
appends to) the config-file value, so any caller exporting their own RUSTFLAGS
silently dropped the flag — the long comment in build.yml was a scar from that.

dial9 is now an opt-in feature (`dial9`, plus `dial9-s3` and `dial9-taskdump`),
the global rustflag is gone, and `crates/obs/build.rs` fails the compile if the
feature is on without the flag. Telemetry builds go through `make build-profiling`.

Metrics that could not lie
--------------------------
`rustfs_dial9_{events_total,bytes_written_total,rotations_total,cpu_overhead_percent}`
were hard-coded to zero — a Counter pinned at 0 reads as "nothing happened".
Removed. `rustfs_dial9_enabled` was sourced from the environment, so it read 1
even when the traced runtime failed and the process fell back to a standard
runtime; it is replaced by `rustfs_dial9_supported` (compile-time),
`rustfs_dial9_configured` (intent) and `rustfs_dial9_active_sessions` (reality).

No `writer_healthy` gauge is exported: dial9's `RotatingWriter` can enter its
`Finished` state and stop writing, but exposes no way to observe that, so the
gauge could only ever be hard-coded to 1. Documented as a known gap instead.

Final events were lost
----------------------
The `TelemetryGuard` lived in a `static OnceLock`, which is never dropped, so
buffered events were never flushed at exit. `build_tokio_runtime` now returns
the guard and `run_process` drops it before any exit path.

Also
----
- `disk_usage_bytes` was a `read_dir` + per-file `stat` on the metrics
  collection path. It is now sampled by a background task into an atomic.
- `SAMPLING_RATE`/`S3_BUCKET`/`S3_PREFIX` were parsed, warned about, and
  discarded. S3 upload is now wired to dial9's `with_s3_uploader` behind
  `dial9-s3`; `SAMPLING_RATE` has no upstream equivalent and is removed.
- Wire `with_task_dumps` (async backtraces of stalled tasks), configurable via
  `RUSTFS_RUNTIME_DIAL9_TASK_DUMP_{ENABLED,IDLE_THRESHOLD_MS}`.
- Split `telemetry/dial9.rs` into `config`/`state`/`enabled`/`disabled`; the
  stub keeps the public API identical so callers need no `#[cfg]`.
- Drop four print-only examples and the manual test bin that exercised the
  removed `init_session` scaffolding.

Verified: cargo check/clippy/test across default, `dial9`, and `dial9-s3`;
build.rs correctly rejects `dial9` without `--cfg tokio_unstable`;
`make pre-commit` passes.

Co-Authored-By: heihutu <heihutu@gmail.com>

* docs(obs): document dial9 as an on-demand profiler

scripts/run.sh advertised a `SAMPLING_RATE` knob that was never passed to dial9,
and claimed "CPU overhead < 5% (with sampling rate 1.0)" and "lower values reduce
CPU overhead" on the strength of it. The knob is gone; the guidance built on it
had to go too.

Replace it with what is actually true: dial9 needs a `make build-profiling`
binary, its disk budget evicts oldest-first (so a high poll rate can overwrite
the incident you are chasing), and it cannot be toggled without a restart.

Add docs/operations/dial9-runtime-profiling.md covering the build variants, an
investigation walkthrough, the configuration table, how to read the three
supported/configured/active_sessions gauges against each other, and the upstream
gap that makes writer death only indirectly observable.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(obs): add a dial9 smoke example that proves events are recorded

The bug this guards against is invisible to every existing signal: with
`build` instead of `build_and_start`, dial9 creates the trace file, seals
segments, and reports `TelemetryGuard::is_enabled() == true` — it simply
records no events. Only the segment's byte count tells the two apart.

Measured on this workload: 5640 bytes when recording, 310 bytes (a bare
segment header) when not. The example asserts >= 2048 bytes, and was verified
to fail with the `build` call restored.

Also correct the comment on the `is_enabled` check in `finish_traced_runtime`.
It claimed to catch "recording silently off"; it does not. It only rejects the
inert guard a lenient config yields after a build failure. Recording is
guaranteed by `build_and_start`, not by that check.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(rustfs): accept Unsupported runtime telemetry capability

A binary built without the `dial9` feature now reports the runtime-telemetry
capability as `Unsupported` rather than `Disabled`. The distinction matters to
operators: `Disabled` implies the capability can be switched on by setting an
environment variable, which is not true here — telemetry needs a rebuild.

Widen the assertion and pin the new semantics: when `dial9::is_supported()` is
false, the state must be exactly `Unsupported`.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): drop the dial9-s3 feature, its TLS stack is vulnerable

CI's Dependency Review and `cargo deny` both reject the branch: dial9's
`worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3, which pins
aws-smithy-http-client onto hyper-rustls 0.24 and rustls-webpki 0.101.7. That
webpki carries RUSTSEC-2026-0098, -0099 and -0104.

0.1.3 is the latest release of the transfer manager, and 1.2.0 the latest of the
smithy client, so there is nothing to upgrade to. Cargo's feature unification can
add features but cannot drop a transitive dependency, so it cannot be worked
around from here either — the rest of the workspace already resolves to the safe
rustls-webpki 0.103 / hyper-rustls 0.27.

Remove the `dial9-s3` feature and the `with_s3_uploader` wiring. The two S3
environment variables stay parsed and warned about, now naming the real reason
rather than a missing build feature. Trace segments are collected from the output
directory instead. Tracked as D9-14 in rustfs/backlog#1157.

With this, Cargo.lock is byte-identical to main: the PR no longer touches the
dependency graph at all.

Also correct the `dial9-taskdump` documentation. It claimed the feature "compiles
to a no-op elsewhere"; in fact `tokio/taskdump` raises a `compile_error!` on any
target other than linux/{aarch64,x86,x86_64}. Verified by trying to build it on
macOS, which is how the claim was found to be wrong.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>

rustfs/Cargo.toml

@@ -35,12 +35,6 @@ path = "src/lib.rs"
 name = "rustfs"
 path = "src/main.rs"
 
-[[bin]]
-name = "manual-test-dial9"
-path = "tests/manual/test_dial9.rs"
-test = false
-bench = false
-
 [[bench]]
 name = "s3_operations"
 harness = false
@@ -60,6 +54,9 @@ full = ["metrics-gpu", "ftps", "swift", "webdav", "sftp", "pyroscope"]
 manual-test-runners = []
 rio-v2 = ["rustfs-ecstore/rio-v2"]
 pyroscope = ["rustfs-obs/pyroscope"]
+# Tokio runtime telemetry. Requires `--cfg tokio_unstable`; use `make build-profiling`.
+dial9 = ["rustfs-obs/dial9"]
+dial9-taskdump = ["dial9", "rustfs-obs/dial9-taskdump"]
 hotpath = [
     "dep:hotpath",
     "hotpath/hotpath",

rustfs/src/runtime_capabilities.rs

@@ -24,6 +24,7 @@ const EBPF_LINUX_ONLY: &str = "eBPF support is only available on linux targets";
 const STORAGE_MEDIA_NOT_REPORTED: &str = "storage media not reported by endpoints";
 const FAILURE_DOMAIN_NOT_REPORTED: &str = "failure domain labels not reported by endpoints";
 const NUMA_NOT_WIRED: &str = "NUMA topology not wired into runtime";
+const RUNTIME_TELEMETRY_NOT_COMPILED_IN: &str = "dial9 telemetry not compiled into this binary";
 const NUMA_LINUX_ONLY: &str = "NUMA topology reporting currently targets linux runtimes";
 
 #[derive(Debug, Default, Clone, Copy)]
@@ -99,7 +100,10 @@ pub fn topology_snapshot_from_endpoint_pools(endpoint_pools: &EndpointServerPool
 }
 
 fn runtime_telemetry_status() -> CapabilityStatus {
-    if rustfs_obs::dial9::is_enabled() {
+    if !rustfs_obs::dial9::is_supported() {
+        return CapabilityStatus::unsupported().with_reason(RUNTIME_TELEMETRY_NOT_COMPILED_IN);
+    }
+    if rustfs_obs::dial9::is_configured() {
         CapabilityStatus::supported()
     } else {
         CapabilityStatus::disabled()
@@ -190,10 +194,16 @@ mod tests {
         assert_eq!(snapshot.platform.os.as_deref(), Some(std::env::consts::OS));
         assert_eq!(snapshot.platform.arch.as_deref(), Some(std::env::consts::ARCH));
         assert_eq!(snapshot.platform.target_triple.as_deref(), Some(compiled_target_triple().as_str()));
+        // A binary built without the `dial9` feature reports `Unsupported`
+        // rather than `Disabled`: telemetry cannot be turned on by configuration
+        // alone, and reporting it as merely disabled would imply that it can.
         assert!(matches!(
             snapshot.runtime_telemetry.state,
-            CapabilityState::Supported | CapabilityState::Disabled
+            CapabilityState::Supported | CapabilityState::Disabled | CapabilityState::Unsupported
         ));
+        if !rustfs_obs::dial9::is_supported() {
+            assert!(matches!(snapshot.runtime_telemetry.state, CapabilityState::Unsupported));
+        }
         assert!(
             snapshot
                 .platform

rustfs/src/server/runtime.rs

@@ -12,15 +12,10 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use std::sync::OnceLock;
 use std::time::Duration;
 use sysinfo::{RefreshKind, System};
 
-// Import TelemetryGuard from rustfs_obs re-export
-use rustfs_obs::dial9::TelemetryGuard;
-
-// Global storage for TelemetryGuard to keep it alive for the program duration
-static DIAL9_TELEMETRY_GUARD: OnceLock<TelemetryGuard> = OnceLock::new();
+use rustfs_obs::dial9::Dial9SessionGuard;
 
 #[inline]
 fn compute_default_thread_stack_size() -> usize {
@@ -166,56 +161,43 @@ fn print_tokio_thread_enable() -> bool {
     rustfs_utils::get_env_bool(rustfs_config::ENV_THREAD_PRINT_ENABLED, rustfs_config::DEFAULT_THREAD_PRINT_ENABLED)
 }
 
-/// Build Tokio runtime with optional dial9 telemetry support.
+/// Build the Tokio runtime, attaching dial9 telemetry when it is enabled.
 ///
-/// If dial9 is enabled via environment variables, creates a TracedRuntime
-/// and stores the TelemetryGuard globally to keep it alive for the
-/// duration of the program.
+/// The returned [`Dial9SessionGuard`] owns the telemetry session. It **must**
+/// be dropped before the process exits: dropping it flushes buffered events and
+/// seals the final trace segment. Parking it in a `static` would mean it is
+/// never dropped and the last events — usually the ones that explain a stall —
+/// are lost.
+///
+/// When telemetry is requested but cannot be built, this falls back to a
+/// standard runtime and returns `None` for the guard rather than failing
+/// startup. The `rustfs_dial9_active_sessions` metric reports that outcome.
 ///
 /// # Returns
 ///
-/// * `Ok(runtime)` - Successfully created runtime
-/// * `Err(e)` - Failed to create runtime
+/// The runtime, plus `Some(guard)` when a telemetry session is recording.
 ///
 /// # Errors
 ///
-/// Returns an error if:
-/// - The Tokio runtime builder fails
-/// - Dial9 is enabled but fails to initialize (falls back to standard runtime)
-///
-/// # Examples
-///
-/// ```no_run
-/// // build_tokio_runtime is pub(crate) - call it from within the rustfs binary:
-/// // let runtime = build_tokio_runtime().expect("Failed to build runtime");
-/// // runtime.block_on(async { /* ... */ })
-/// ```
-pub fn build_tokio_runtime() -> Result<tokio::runtime::Runtime, BuildError> {
-    let mut builder = tokio_runtime_builder();
-
-    // Check if dial9 is enabled
-    if rustfs_obs::dial9::is_enabled() {
-        tracing::info!("Dial9 telemetry enabled, building TracedRuntime");
-
-        return match rustfs_obs::dial9::build_traced_runtime(builder) {
-            Ok((runtime, guard)) => {
-                // Store guard in global static to keep it alive for the program duration
-                let _ = DIAL9_TELEMETRY_GUARD.set(guard);
-                tracing::info!("TracedRuntime created successfully, guard stored globally");
-                Ok(runtime)
-            }
-            Err(e) => {
-                tracing::warn!("Failed to build TracedRuntime: {}", e);
-                tracing::warn!("Falling back to standard Tokio runtime");
-                // Rebuild the builder for standard runtime
-                let mut builder = tokio_runtime_builder();
-                builder.build().map_err(BuildError::Runtime)
-            }
-        };
+/// Returns [`BuildError::Runtime`] if the Tokio runtime builder itself fails.
+pub fn build_tokio_runtime() -> Result<(tokio::runtime::Runtime, Option<Dial9SessionGuard>), BuildError> {
+    if !rustfs_obs::dial9::is_enabled() {
+        let runtime = tokio_runtime_builder().build().map_err(BuildError::Runtime)?;
+        return Ok((runtime, None));
     }
 
-    // Standard runtime
-    builder.build().map_err(BuildError::Runtime)
+    tracing::info!("Dial9 telemetry enabled, building TracedRuntime");
+    match rustfs_obs::dial9::build_traced_runtime(tokio_runtime_builder()) {
+        Ok((runtime, guard)) => {
+            tracing::info!("TracedRuntime created and recording");
+            Ok((runtime, Some(guard)))
+        }
+        Err(e) => {
+            tracing::warn!("Failed to build TracedRuntime: {e}; falling back to standard Tokio runtime");
+            let runtime = tokio_runtime_builder().build().map_err(BuildError::Runtime)?;
+            Ok((runtime, None))
+        }
+    }
 }
 
 /// Error type for runtime building failures.

rustfs/src/startup_entrypoint.rs

@@ -32,8 +32,14 @@ const OBSERVABILITY_INIT_FATAL_ALREADY_REPORTED: &str = "observability initializ
 
 pub fn run_process() {
     // Building the process runtime is a startup fatal boundary.
-    let runtime = crate::server::build_tokio_runtime().expect("Failed to build Tokio runtime");
+    let (runtime, dial9_guard) = crate::server::build_tokio_runtime().expect("Failed to build Tokio runtime");
     let result = runtime.block_on(async_main());
+
+    // Flush and seal the trace segment before any exit path. `process::exit`
+    // below does not run destructors, and neither does returning from `main`
+    // for a guard held in a `static`.
+    drop(dial9_guard);
+
     if let Err(ref e) = result {
         if e.to_string() != OBSERVABILITY_INIT_FATAL_ALREADY_REPORTED {
             // Tracing may not be initialized when startup fails this early.

rustfs/src/startup_runtime_hooks.rs

@@ -17,7 +17,7 @@ use crate::startup_runtime_sources;
 use rustls::crypto::aws_lc_rs::default_provider;
 use std::future::Future;
 use std::io::Result;
-use tracing::{debug, info};
+use tracing::{debug, info, warn};
 
 const LOG_COMPONENT_EMBEDDED: &str = "embedded";
 const LOG_COMPONENT_MAIN: &str = "main";
@@ -35,24 +35,39 @@ pub(crate) fn log_startup_runtime_diagnostics() {
 }
 
 fn log_dial9_runtime_status() {
-    if rustfs_obs::dial9::is_enabled() {
-        info!(
+    let supported = rustfs_obs::dial9::is_supported();
+    let configured = rustfs_obs::dial9::is_configured();
+
+    match (supported, configured) {
+        (true, true) => info!(
             target: "rustfs::main",
             event = EVENT_DIAL9_RUNTIME_STATUS,
             component = LOG_COMPONENT_MAIN,
             subsystem = LOG_SUBSYSTEM_STARTUP,
             enabled = true,
             "Dial9 Tokio runtime telemetry is enabled"
-        );
-    } else {
-        debug!(
+        ),
+        // Requested but unavailable. This is the case an operator most needs to
+        // see: the process otherwise runs to completion recording nothing.
+        (false, true) => warn!(
             target: "rustfs::main",
             event = EVENT_DIAL9_RUNTIME_STATUS,
             component = LOG_COMPONENT_MAIN,
             subsystem = LOG_SUBSYSTEM_STARTUP,
             enabled = false,
+            supported = false,
+            remedy = "rebuild with --features dial9 (see `make build-profiling`)",
+            "Dial9 Tokio runtime telemetry is configured but not compiled into this binary"
+        ),
+        (_, false) => debug!(
+            target: "rustfs::main",
+            event = EVENT_DIAL9_RUNTIME_STATUS,
+            component = LOG_COMPONENT_MAIN,
+            subsystem = LOG_SUBSYSTEM_STARTUP,
+            enabled = false,
+            supported,
             "Dial9 Tokio runtime telemetry is disabled"
-        );
+        ),
     }
 }
 

rustfs/tests/manual/test_dial9.rs

@@ -1,95 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-// Manual Dial9 integration runner.
-//
-// Run with:
-// `cargo run -p rustfs --features manual-test-runners --bin manual-test-dial9`
-//
-// This file lives under `rustfs/tests/manual` and is registered explicitly in
-// `rustfs/Cargo.toml` so it stays out of `cargo test` auto-discovery.
-use rustfs_obs::dial9::{Dial9Config, Dial9SessionGuard};
-use tokio::time::{Duration, sleep};
-
-#[tokio::main]
-async fn main() -> Result<(), Box<dyn std::error::Error>> {
-    println!("=== Dial9 Integration Test ===\n");
-
-    // Test 1: Check default dial9 configuration
-    println!("Test 1: Default configuration");
-    let default_config = Dial9Config::default();
-    println!("  default enabled: {}", default_config.enabled);
-    println!("  default output_dir: {}", default_config.output_dir);
-    println!("  default file_prefix: {}", default_config.file_prefix);
-    println!("  ✓ PASS: Default configuration loaded\n");
-
-    // Test 2: Create explicit dial9 configuration
-    println!("Test 2: Explicit dial9 configuration");
-    let config = Dial9Config {
-        enabled: true,
-        output_dir: "/tmp/rustfs-test-telemetry".to_string(),
-        sampling_rate: 0.5,
-        ..Dial9Config::default()
-    };
-    println!("  config.enabled: {}", config.enabled);
-    println!("  config.output_dir: {}", config.output_dir);
-    println!("  config.file_prefix: {}", config.file_prefix);
-    println!("  config.sampling_rate: {}", config.sampling_rate);
-
-    assert!(config.enabled);
-    assert_eq!(config.output_dir, "/tmp/rustfs-test-telemetry");
-    assert_eq!(config.sampling_rate, 0.5);
-    println!("  ✓ PASS: Configuration loaded correctly\n");
-
-    // Test 3: Initialize dial9 session
-    println!("Test 3: Initialize dial9 session");
-    match Dial9SessionGuard::new(config.clone()).await {
-        Ok(Some(guard)) => {
-            println!("  Dial9 session initialized successfully");
-            println!("  guard.is_active(): {}", guard.is_active());
-            println!("  ✓ PASS: Session initialized\n");
-
-            // Test 4: Generate some async activity
-            println!("Test 4: Generate async activity for tracing");
-            let handle = tokio::spawn(async {
-                for i in 1..=5 {
-                    println!("  Task iteration {}", i);
-                    sleep(Duration::from_millis(50)).await;
-                }
-            });
-            handle.await?;
-            println!("  ✓ PASS: Async activity completed\n");
-
-            // Test 5: Session shutdown
-            println!("Test 5: Session cleanup");
-            drop(guard);
-            println!("  ✓ PASS: Session cleaned up\n");
-        }
-        Ok(None) => {
-            println!("  ⚠ SKIP: Dial9 session not created (configuration validation may have failed)\n");
-        }
-        Err(e) => {
-            println!("  ✗ FAIL: {:?}", e);
-            return Err(e.into());
-        }
-    }
-
-    // Cleanup
-    if let Err(err) = tokio::fs::remove_dir_all(&config.output_dir).await {
-        println!("  ⚠ SKIP: Failed to remove output directory: {}", err);
-    }
-
-    println!("=== All Tests Passed! ===");
-    Ok(())
-}