mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 08:18:18 +00:00
refactor(obs): make dial9 telemetry opt-in and actually record events (#4663)
* refactor(obs): make dial9 telemetry opt-in and actually record events
The dial9 Tokio-runtime profiler was disabled by default, yet every build
paid for it, and enabling it produced trace files with no events in them.
Recorded empty traces
---------------------
`build_traced_runtime` called `TracedRuntime::builder()...build(..)`, but dial9
only starts recording in `build_and_start*`. `build` still returns a live guard
whose `is_enabled()` reports true, and still creates and seals segment files —
they just contain a header and no events. It also skipped `with_trace_path`, so
the background worker driving the segment pipeline was never spawned.
Measured on the new smoke example: 310 bytes of bare segment header, against
5640 bytes for the same workload once recording actually starts.
Switch to `with_trace_path(..).build_and_start(..)`.
Cost was unconditional
----------------------
`--cfg tokio_unstable` was a global `[build] rustflags` entry and `rustfs-obs`
depended on `dial9-tokio-telemetry` unconditionally, so all builds depended on
Tokio's non-semver API. Worse, an environment `RUSTFLAGS` replaces (never
appends to) the config-file value, so any caller exporting their own RUSTFLAGS
silently dropped the flag — the long comment in build.yml was a scar from that.
dial9 is now an opt-in feature (`dial9`, plus `dial9-s3` and `dial9-taskdump`),
the global rustflag is gone, and `crates/obs/build.rs` fails the compile if the
feature is on without the flag. Telemetry builds go through `make build-profiling`.
Metrics that could not lie
--------------------------
`rustfs_dial9_{events_total,bytes_written_total,rotations_total,cpu_overhead_percent}`
were hard-coded to zero — a Counter pinned at 0 reads as "nothing happened".
Removed. `rustfs_dial9_enabled` was sourced from the environment, so it read 1
even when the traced runtime failed and the process fell back to a standard
runtime; it is replaced by `rustfs_dial9_supported` (compile-time),
`rustfs_dial9_configured` (intent) and `rustfs_dial9_active_sessions` (reality).
No `writer_healthy` gauge is exported: dial9's `RotatingWriter` can enter its
`Finished` state and stop writing, but exposes no way to observe that, so the
gauge could only ever be hard-coded to 1. Documented as a known gap instead.
Final events were lost
----------------------
The `TelemetryGuard` lived in a `static OnceLock`, which is never dropped, so
buffered events were never flushed at exit. `build_tokio_runtime` now returns
the guard and `run_process` drops it before any exit path.
Also
----
- `disk_usage_bytes` was a `read_dir` + per-file `stat` on the metrics
collection path. It is now sampled by a background task into an atomic.
- `SAMPLING_RATE`/`S3_BUCKET`/`S3_PREFIX` were parsed, warned about, and
discarded. S3 upload is now wired to dial9's `with_s3_uploader` behind
`dial9-s3`; `SAMPLING_RATE` has no upstream equivalent and is removed.
- Wire `with_task_dumps` (async backtraces of stalled tasks), configurable via
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_{ENABLED,IDLE_THRESHOLD_MS}`.
- Split `telemetry/dial9.rs` into `config`/`state`/`enabled`/`disabled`; the
stub keeps the public API identical so callers need no `#[cfg]`.
- Drop four print-only examples and the manual test bin that exercised the
removed `init_session` scaffolding.
Verified: cargo check/clippy/test across default, `dial9`, and `dial9-s3`;
build.rs correctly rejects `dial9` without `--cfg tokio_unstable`;
`make pre-commit` passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): document dial9 as an on-demand profiler
scripts/run.sh advertised a `SAMPLING_RATE` knob that was never passed to dial9,
and claimed "CPU overhead < 5% (with sampling rate 1.0)" and "lower values reduce
CPU overhead" on the strength of it. The knob is gone; the guidance built on it
had to go too.
Replace it with what is actually true: dial9 needs a `make build-profiling`
binary, its disk budget evicts oldest-first (so a high poll rate can overwrite
the incident you are chasing), and it cannot be toggled without a restart.
Add docs/operations/dial9-runtime-profiling.md covering the build variants, an
investigation walkthrough, the configuration table, how to read the three
supported/configured/active_sessions gauges against each other, and the upstream
gap that makes writer death only indirectly observable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(obs): add a dial9 smoke example that proves events are recorded
The bug this guards against is invisible to every existing signal: with
`build` instead of `build_and_start`, dial9 creates the trace file, seals
segments, and reports `TelemetryGuard::is_enabled() == true` — it simply
records no events. Only the segment's byte count tells the two apart.
Measured on this workload: 5640 bytes when recording, 310 bytes (a bare
segment header) when not. The example asserts >= 2048 bytes, and was verified
to fail with the `build` call restored.
Also correct the comment on the `is_enabled` check in `finish_traced_runtime`.
It claimed to catch "recording silently off"; it does not. It only rejects the
inert guard a lenient config yields after a build failure. Recording is
guaranteed by `build_and_start`, not by that check.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): accept Unsupported runtime telemetry capability
A binary built without the `dial9` feature now reports the runtime-telemetry
capability as `Unsupported` rather than `Disabled`. The distinction matters to
operators: `Disabled` implies the capability can be switched on by setting an
environment variable, which is not true here — telemetry needs a rebuild.
Widen the assertion and pin the new semantics: when `dial9::is_supported()` is
false, the state must be exactly `Unsupported`.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): drop the dial9-s3 feature, its TLS stack is vulnerable
CI's Dependency Review and `cargo deny` both reject the branch: dial9's
`worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3, which pins
aws-smithy-http-client onto hyper-rustls 0.24 and rustls-webpki 0.101.7. That
webpki carries RUSTSEC-2026-0098, -0099 and -0104.
0.1.3 is the latest release of the transfer manager, and 1.2.0 the latest of the
smithy client, so there is nothing to upgrade to. Cargo's feature unification can
add features but cannot drop a transitive dependency, so it cannot be worked
around from here either — the rest of the workspace already resolves to the safe
rustls-webpki 0.103 / hyper-rustls 0.27.
Remove the `dial9-s3` feature and the `with_s3_uploader` wiring. The two S3
environment variables stay parsed and warned about, now naming the real reason
rather than a missing build feature. Trace segments are collected from the output
directory instead. Tracked as D9-14 in rustfs/backlog#1157.
With this, Cargo.lock is byte-identical to main: the PR no longer touches the
dependency graph at all.
Also correct the `dial9-taskdump` documentation. It claimed the feature "compiles
to a no-op elsewhere"; in fact `tokio/taskdump` raises a `compile_error!` on any
target other than linux/{aarch64,x86,x86_64}. Verified by trying to build it on
macOS, which is how the claim was found to be wrong.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
+21
-1
@@ -27,9 +27,29 @@ documentation = "https://docs.rs/rustfs-obs/latest/rustfs_obs/"
|
||||
|
||||
[features]
|
||||
default = []
|
||||
# Tokio runtime-level telemetry. Requires a `--cfg tokio_unstable` build; the
|
||||
# build script fails the compile when that flag is missing. Off by default so
|
||||
# ordinary builds neither pay for nor depend on Tokio's unstable API.
|
||||
dial9 = ["dep:dial9-tokio-telemetry"]
|
||||
# Capture async backtraces of stalled tasks. Enables `tokio/taskdump`, which
|
||||
# hard-errors at compile time on anything other than linux/{aarch64,x86,x86_64},
|
||||
# and additionally needs `--cfg tokio_taskdump` to record anything. Linux-only,
|
||||
# and not a silent no-op elsewhere: enabling it on macOS fails the build.
|
||||
dial9-taskdump = ["dial9", "dial9-tokio-telemetry/taskdump"]
|
||||
#
|
||||
# NOTE: there is deliberately no `dial9-s3` feature. dial9's `worker-s3` feature
|
||||
# pulls aws-sdk-s3-transfer-manager 0.1.3 (its latest), which pins
|
||||
# aws-smithy-http-client onto hyper-rustls 0.24 / rustls-webpki 0.101.7. That
|
||||
# webpki carries RUSTSEC-2026-0098, -0099 and -0104, and `cargo deny` rejects it.
|
||||
# Feature unification cannot drop a transitive dependency, so this can only be
|
||||
# fixed upstream. Tracked in rustfs/backlog#1157 (D9-14).
|
||||
gpu = ["dep:nvml-wrapper"]
|
||||
pyroscope = ["dep:pyroscope"]
|
||||
|
||||
[[example]]
|
||||
name = "dial9_smoke"
|
||||
required-features = ["dial9"]
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
|
||||
@@ -76,7 +96,7 @@ tracing-opentelemetry = { workspace = true }
|
||||
tracing-subscriber = { workspace = true, features = ["registry", "std", "fmt", "env-filter", "tracing-log", "time", "local-time", "json"] }
|
||||
tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "rt", "time", "macros"] }
|
||||
tokio-util = { workspace = true }
|
||||
dial9-tokio-telemetry = { workspace = true }
|
||||
dial9-tokio-telemetry = { workspace = true, optional = true }
|
||||
thiserror = { workspace = true }
|
||||
zstd = { workspace = true, features = ["zstdmt"] }
|
||||
sysinfo = { workspace = true }
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Enforces that the `dial9` feature is paired with `--cfg tokio_unstable`.
|
||||
//!
|
||||
//! `dial9-tokio-telemetry` hooks Tokio's unstable runtime instrumentation. The
|
||||
//! flag lives in `RUSTFLAGS`, which a caller can silently clobber by exporting
|
||||
//! their own value — that used to turn telemetry off with no diagnostic. Fail
|
||||
//! the build instead.
|
||||
|
||||
fn main() {
|
||||
println!("cargo::rerun-if-changed=build.rs");
|
||||
println!("cargo::rerun-if-env-changed=RUSTFLAGS");
|
||||
|
||||
let dial9_enabled = std::env::var_os("CARGO_FEATURE_DIAL9").is_some();
|
||||
let tokio_unstable = std::env::var_os("CARGO_CFG_TOKIO_UNSTABLE").is_some();
|
||||
|
||||
if dial9_enabled && !tokio_unstable {
|
||||
panic!(
|
||||
"\n\
|
||||
rustfs-obs: the `dial9` feature requires a `--cfg tokio_unstable` build.\n\
|
||||
\n\
|
||||
Build telemetry-enabled binaries with:\n\
|
||||
\n RUSTFLAGS=\"--cfg tokio_unstable\" cargo build --features dial9\n\
|
||||
\n\
|
||||
or run `make build-profiling`. Note that setting RUSTFLAGS in the\n\
|
||||
environment overrides `.cargo/config.toml`, so the flag must be\n\
|
||||
present in the value you export.\n"
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! End-to-end check that a traced runtime actually records events to disk.
|
||||
//!
|
||||
//! Guards against the regression where `TracedRuntime::builder()...build(..)`
|
||||
//! was used instead of `build_and_start(..)`: that returns a guard with
|
||||
//! recording disabled, so telemetry appeared healthy and produced empty traces.
|
||||
//!
|
||||
//! Configuration comes from the environment (the crate forbids `unsafe`, so this
|
||||
//! cannot call `set_var`). Run with:
|
||||
//!
|
||||
//! ```bash
|
||||
//! RUSTFLAGS="--cfg tokio_unstable" \
|
||||
//! RUSTFS_RUNTIME_DIAL9_ENABLED=true \
|
||||
//! RUSTFS_RUNTIME_DIAL9_OUTPUT_DIR=/tmp/rustfs-dial9-smoke \
|
||||
//! RUSTFS_RUNTIME_DIAL9_MAX_FILE_SIZE=65536 \
|
||||
//! RUSTFS_RUNTIME_DIAL9_ROTATION_COUNT=4 \
|
||||
//! cargo run -p rustfs-obs --features dial9 --example dial9_smoke
|
||||
//! ```
|
||||
|
||||
use rustfs_obs::dial9::{Dial9Config, build_traced_runtime};
|
||||
use std::path::Path;
|
||||
|
||||
const USAGE: &str = "\
|
||||
set RUSTFS_RUNTIME_DIAL9_ENABLED=true and RUSTFS_RUNTIME_DIAL9_OUTPUT_DIR=<dir> \
|
||||
before running this example (see the module docs for a full command line)";
|
||||
|
||||
/// A segment that was opened but never recorded an event still carries a header
|
||||
/// (~292 bytes when this was written). A workload like the one below produces
|
||||
/// several KiB. Anything under this threshold means events are not being
|
||||
/// recorded, even though the file exists and the guard looks healthy.
|
||||
const MIN_EXPECTED_TRACE_BYTES: u64 = 2048;
|
||||
|
||||
fn segment_bytes(dir: &Path, prefix: &str) -> u64 {
|
||||
let Ok(entries) = std::fs::read_dir(dir) else {
|
||||
return 0;
|
||||
};
|
||||
entries
|
||||
.filter_map(Result::ok)
|
||||
.filter(|e| e.file_name().to_str().is_some_and(|n| n.starts_with(prefix)))
|
||||
.filter_map(|e| e.metadata().ok())
|
||||
.map(|m| m.len())
|
||||
.sum()
|
||||
}
|
||||
|
||||
fn main() {
|
||||
let config = Dial9Config::from_env();
|
||||
if !config.enabled {
|
||||
eprintln!("{USAGE}");
|
||||
std::process::exit(2);
|
||||
}
|
||||
|
||||
let output_dir = std::path::PathBuf::from(&config.output_dir);
|
||||
println!("output_dir = {}", config.output_dir);
|
||||
println!("disk budget = {} bytes", config.total_disk_budget());
|
||||
|
||||
let mut builder = tokio::runtime::Builder::new_multi_thread();
|
||||
builder.worker_threads(2).enable_all();
|
||||
|
||||
let (runtime, guard) = build_traced_runtime(builder).expect("traced runtime should build");
|
||||
// Necessary but not sufficient: a guard built without `build_and_start` also
|
||||
// reports active. Only the byte count below proves events were recorded.
|
||||
assert!(guard.is_active(), "telemetry session must be live");
|
||||
println!("session active = {}", guard.is_active());
|
||||
|
||||
// Generate poll/park/spawn traffic, including a deliberately long poll.
|
||||
runtime.block_on(async {
|
||||
let mut handles = Vec::new();
|
||||
for i in 0..64_u64 {
|
||||
handles.push(tokio::spawn(async move {
|
||||
if i % 16 == 0 {
|
||||
// Block the worker: exactly the fault dial9 exists to surface.
|
||||
std::thread::sleep(std::time::Duration::from_millis(5));
|
||||
}
|
||||
tokio::time::sleep(std::time::Duration::from_millis(1)).await;
|
||||
i * 2
|
||||
}));
|
||||
}
|
||||
for h in handles {
|
||||
h.await.expect("task should not panic");
|
||||
}
|
||||
});
|
||||
|
||||
// Dropping the guard flushes buffered events and seals the active segment.
|
||||
drop(guard);
|
||||
drop(runtime);
|
||||
|
||||
let bytes = segment_bytes(&output_dir, &config.file_prefix);
|
||||
println!("trace bytes = {bytes}");
|
||||
|
||||
assert!(
|
||||
bytes >= MIN_EXPECTED_TRACE_BYTES,
|
||||
"dial9 wrote only {bytes} bytes to {} (expected >= {MIN_EXPECTED_TRACE_BYTES}); \
|
||||
the segment holds a header but no events, so recording never started",
|
||||
output_dir.display()
|
||||
);
|
||||
println!("\nOK: dial9 recorded {bytes} bytes of trace data.");
|
||||
}
|
||||
@@ -1,53 +0,0 @@
|
||||
// Test dial9 integration example
|
||||
use rustfs_obs::dial9::{Dial9Config, is_enabled};
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
println!("=== Dial9 Integration Test ===\n");
|
||||
|
||||
// Test 1: Check initial dial9 state
|
||||
println!("Test 1: Default state");
|
||||
let initial_enabled = is_enabled();
|
||||
println!(" dial9 enabled: {}", initial_enabled);
|
||||
if initial_enabled {
|
||||
println!(" ⚠ SKIP: Dial9 is already enabled via environment; skipping default-disabled assertion\n");
|
||||
} else {
|
||||
println!(" ✓ PASS: Dial9 is disabled by default\n");
|
||||
}
|
||||
|
||||
// Test 2: Load default configuration
|
||||
println!("Test 2: Default configuration");
|
||||
let config = Dial9Config::from_env();
|
||||
println!(" enabled: {}", config.enabled);
|
||||
println!(" output_dir: {}", config.output_dir);
|
||||
println!(" file_prefix: {}", config.file_prefix);
|
||||
println!(" max_file_size: {} bytes", config.max_file_size);
|
||||
println!(" rotation_count: {}", config.rotation_count);
|
||||
println!(" s3_bucket: {:?}", config.s3_bucket);
|
||||
println!(" s3_prefix: {:?}", config.s3_prefix);
|
||||
println!(" sampling_rate: {}", config.sampling_rate);
|
||||
println!(" ✓ PASS: Default configuration loaded\n");
|
||||
|
||||
// Test 3: Configuration validation
|
||||
println!("Test 3: Configuration validation");
|
||||
if !initial_enabled {
|
||||
assert!(!config.enabled, "Should be disabled by default");
|
||||
assert_eq!(config.s3_bucket, None, "S3 bucket should be None by default");
|
||||
assert_eq!(config.s3_prefix, None, "S3 prefix should be None by default");
|
||||
println!(" ✓ PASS: Configuration validated\n");
|
||||
} else {
|
||||
println!(" ⚠ SKIP: Configuration validation skipped (dial9 is enabled)\n");
|
||||
}
|
||||
|
||||
println!("=== All Tests Passed! ===");
|
||||
println!();
|
||||
println!("Note: To test with dial9 enabled, set environment variables:");
|
||||
println!(" export RUSTFS_RUNTIME_DIAL9_ENABLED=true");
|
||||
println!(" export RUSTFS_RUNTIME_DIAL9_OUTPUT_DIR=/tmp/rustfs-test-telemetry");
|
||||
println!(" export RUSTFS_RUNTIME_DIAL9_SAMPLING_RATE=0.5");
|
||||
println!(" export RUSTFS_RUNTIME_DIAL9_S3_BUCKET=my-bucket");
|
||||
println!(" export RUSTFS_RUNTIME_DIAL9_S3_PREFIX=telemetry/");
|
||||
println!(" cargo run -p rustfs-obs --example test_dial9");
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -1,76 +0,0 @@
|
||||
// Full dial9 integration test with session initialization
|
||||
use rustfs_obs::dial9::{Dial9Config, init_session, is_enabled};
|
||||
use tokio::time::{Duration, sleep};
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
println!("=== Full Dial9 Integration Test ===");
|
||||
println!();
|
||||
|
||||
// Check if dial9 is enabled
|
||||
if !is_enabled() {
|
||||
println!("Dial9 is disabled. Enable with:");
|
||||
println!(" export RUSTFS_RUNTIME_DIAL9_ENABLED=true");
|
||||
println!(" export RUSTFS_RUNTIME_DIAL9_OUTPUT_DIR=/tmp/rustfs-test-telemetry");
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Test 1: Configuration
|
||||
println!("Test 1: Configuration");
|
||||
let config = Dial9Config::from_env();
|
||||
println!(" enabled: {}", config.enabled);
|
||||
println!(" output_dir: {}", config.output_dir);
|
||||
println!(" file_prefix: {}", config.file_prefix);
|
||||
println!(" sampling_rate: {}", config.sampling_rate);
|
||||
println!(" ✓ Configuration loaded");
|
||||
println!();
|
||||
|
||||
// Test 2: Session initialization
|
||||
println!("Test 2: Session initialization");
|
||||
match init_session().await {
|
||||
Ok(Some(guard)) => {
|
||||
println!(" ✓ Session initialized successfully");
|
||||
println!(" guard.is_active(): {}", guard.is_active());
|
||||
println!();
|
||||
|
||||
// Test 3: Generate async activity
|
||||
println!("Test 3: Generate async runtime activity");
|
||||
let tasks = (0..3).map(|i| {
|
||||
tokio::spawn(async move {
|
||||
for j in 0..5 {
|
||||
println!(" Task {} iteration {}", i, j);
|
||||
sleep(Duration::from_millis(20)).await;
|
||||
}
|
||||
})
|
||||
});
|
||||
|
||||
for task in tasks {
|
||||
task.await?;
|
||||
}
|
||||
println!(" ✓ Async activity completed");
|
||||
println!();
|
||||
|
||||
// Test 4: Session lifecycle
|
||||
println!("Test 4: Session lifecycle");
|
||||
println!(" Dropping guard...");
|
||||
drop(guard);
|
||||
println!(" ✓ Session cleaned up");
|
||||
}
|
||||
Ok(None) => {
|
||||
println!(" ⚠ Session not created (writer may have failed)");
|
||||
println!(" This is expected if output directory cannot be created");
|
||||
}
|
||||
Err(e) => {
|
||||
println!(" ✗ Session init failed: {:?}", e);
|
||||
}
|
||||
}
|
||||
|
||||
println!();
|
||||
println!("=== Test Summary ===");
|
||||
println!("✓ Configuration: PASS");
|
||||
println!("✓ Session Init: PASS");
|
||||
println!("✓ Async Activity: PASS");
|
||||
println!("✓ Lifecycle: PASS");
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -1,63 +0,0 @@
|
||||
// Test dial9 S3 configuration
|
||||
use rustfs_obs::dial9::{Dial9Config, is_enabled};
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
println!("=== Dial9 S3 Configuration Test ===");
|
||||
println!();
|
||||
|
||||
// Test 1: Default S3 configuration (should be None/None)
|
||||
println!("Test 1: Default S3 configuration");
|
||||
let default_config = Dial9Config::default();
|
||||
println!(" s3_bucket: {:?}", default_config.s3_bucket);
|
||||
println!(" s3_prefix: {:?}", default_config.s3_prefix);
|
||||
assert_eq!(default_config.s3_bucket, None);
|
||||
assert_eq!(default_config.s3_prefix, None);
|
||||
println!(" ✓ PASS: Default S3 config is None/None");
|
||||
println!();
|
||||
|
||||
// Test 2: Check if dial9 is enabled
|
||||
println!("Test 2: Check dial9 enabled state");
|
||||
println!(" is_enabled(): {}", is_enabled());
|
||||
println!(
|
||||
" RUSTFS_RUNTIME_DIAL9_ENABLED: {}",
|
||||
std::env::var("RUSTFS_RUNTIME_DIAL9_ENABLED").unwrap_or_else(|_| "not set".to_string())
|
||||
);
|
||||
println!();
|
||||
|
||||
// Test 3: Load configuration from environment
|
||||
println!("Test 3: Load configuration from environment");
|
||||
let config = Dial9Config::from_env();
|
||||
println!(" enabled: {}", config.enabled);
|
||||
println!(" s3_bucket: {:?}", config.s3_bucket);
|
||||
println!(" s3_prefix: {:?}", config.s3_prefix);
|
||||
println!(" ✓ PASS: Configuration loaded");
|
||||
println!();
|
||||
|
||||
// Only test S3 config if dial9 is enabled
|
||||
if !config.enabled {
|
||||
println!(" ⚠ SKIP: Dial9 is disabled, S3 config not loaded");
|
||||
println!(" To test S3 configuration:");
|
||||
println!(" export RUSTFS_RUNTIME_DIAL9_ENABLED=true");
|
||||
println!(" export RUSTFS_RUNTIME_DIAL9_S3_BUCKET=my-bucket");
|
||||
println!(" export RUSTFS_RUNTIME_DIAL9_S3_PREFIX=telemetry/");
|
||||
println!(" cargo run -p rustfs-obs --example test_dial9_s3");
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Test 4: Configuration summary
|
||||
println!("Test 4: Configuration summary");
|
||||
println!(" S3 upload enabled: {}", config.s3_bucket.is_some());
|
||||
if let Some(bucket) = &config.s3_bucket {
|
||||
println!(" S3 bucket: {}", bucket);
|
||||
}
|
||||
if let Some(prefix) = &config.s3_prefix {
|
||||
println!(" S3 prefix: {}", prefix);
|
||||
}
|
||||
println!(" ✓ PASS: Configuration summary displayed");
|
||||
println!();
|
||||
|
||||
println!("=== All Tests Passed! ===");
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -1,45 +0,0 @@
|
||||
// Simple dial9 integration test (reads from environment)
|
||||
use rustfs_obs::dial9::{Dial9Config, is_enabled};
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
println!("=== Dial9 Integration Test ===");
|
||||
println!();
|
||||
|
||||
// Test 1: Check current state
|
||||
println!("Test 1: Check dial9 state");
|
||||
println!(
|
||||
" RUSTFS_RUNTIME_DIAL9_ENABLED: {}",
|
||||
std::env::var("RUSTFS_RUNTIME_DIAL9_ENABLED").unwrap_or_else(|_| "not set".to_string())
|
||||
);
|
||||
println!(" is_enabled(): {}", is_enabled());
|
||||
println!(" ✓ Dial9 state check complete");
|
||||
println!();
|
||||
|
||||
// Test 2: Load configuration
|
||||
println!("Test 2: Load dial9 configuration");
|
||||
let config = Dial9Config::from_env();
|
||||
println!(" enabled: {}", config.enabled);
|
||||
println!(" output_dir: {}", config.output_dir);
|
||||
println!(" file_prefix: {}", config.file_prefix);
|
||||
println!(" max_file_size: {} bytes", config.max_file_size);
|
||||
println!(" rotation_count: {}", config.rotation_count);
|
||||
println!(" sampling_rate: {}", config.sampling_rate);
|
||||
println!(" ✓ Configuration loaded");
|
||||
println!();
|
||||
|
||||
// Test 3: Test base path calculation
|
||||
println!("Test 3: Base path calculation");
|
||||
println!(" base_path: {:?}", config.base_path());
|
||||
println!(" ✓ Base path calculated");
|
||||
println!();
|
||||
|
||||
println!("=== All Tests Passed! ===");
|
||||
println!();
|
||||
println!("Note: To test full dial9 functionality, enable it with:");
|
||||
println!(" export RUSTFS_RUNTIME_DIAL9_ENABLED=true");
|
||||
println!(" export RUSTFS_RUNTIME_DIAL9_OUTPUT_DIR=/tmp/rustfs-telemetry");
|
||||
println!(" cargo run -p rustfs-obs --example test_dial9_simple");
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -14,100 +14,65 @@
|
||||
|
||||
//! dial9 Tokio runtime telemetry metrics collector.
|
||||
//!
|
||||
//! This module provides metrics for monitoring the health and performance
|
||||
//! of the dial9 telemetry system itself.
|
||||
|
||||
#![allow(dead_code)]
|
||||
//! Reports the health of the telemetry system itself, not the runtime events it
|
||||
//! records — those live in the trace segments on disk.
|
||||
//!
|
||||
//! Every metric here is backed by a value the process actually observes. A
|
||||
//! counter that cannot be sourced is not exported at all: a series pinned at
|
||||
//! zero reads as "nothing happened", which is worse than a missing series.
|
||||
|
||||
use crate::MetricType;
|
||||
use crate::metrics::report::PrometheusMetric;
|
||||
use crate::telemetry::dial9::runtime_stats_snapshot;
|
||||
use rustfs_config::{DEFAULT_RUNTIME_DIAL9_ENABLED, ENV_RUNTIME_DIAL9_ENABLED};
|
||||
use rustfs_utils::get_env_bool;
|
||||
use crate::telemetry::dial9::{is_configured, is_enabled, is_supported, runtime_stats_snapshot};
|
||||
|
||||
/// Dial9 telemetry system statistics.
|
||||
/// Health of the dial9 telemetry system.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct Dial9Stats {
|
||||
/// Total number of telemetry events recorded
|
||||
pub events_total: u64,
|
||||
|
||||
/// Total bytes written to trace files
|
||||
pub bytes_written: u64,
|
||||
|
||||
/// Number of file rotations that have occurred
|
||||
pub rotation_count: u64,
|
||||
|
||||
/// Total number of dial9 errors
|
||||
/// Cumulative count of telemetry setup failures.
|
||||
pub errors_total: u64,
|
||||
|
||||
/// Estimated CPU overhead percentage (if available)
|
||||
pub cpu_overhead_percent: f64,
|
||||
|
||||
/// Current disk usage by trace files in bytes
|
||||
/// Bytes on disk held by trace segments, as of the last background refresh.
|
||||
pub disk_usage_bytes: u64,
|
||||
|
||||
/// Number of active sessions
|
||||
/// Number of active telemetry sessions (0 or 1).
|
||||
pub active_sessions: u64,
|
||||
}
|
||||
|
||||
/// Collect dial9 telemetry metrics.
|
||||
/// Convert dial9 health statistics into Prometheus metrics.
|
||||
///
|
||||
/// This function converts dial9 statistics into Prometheus metrics format.
|
||||
///
|
||||
/// # Arguments
|
||||
///
|
||||
/// * `stats` - Dial9 statistics to report
|
||||
///
|
||||
/// # Returns
|
||||
///
|
||||
/// A vector of Prometheus metrics for dial9 telemetry statistics.
|
||||
/// `rustfs_dial9_supported` reflects compile-time support, `rustfs_dial9_configured`
|
||||
/// reflects operator intent, and `rustfs_dial9_active_sessions` reflects reality.
|
||||
/// They disagree when a build lacks the feature, or when the traced runtime
|
||||
/// failed to build and the process fell back to a standard runtime — read all
|
||||
/// three before concluding that telemetry is running.
|
||||
pub fn collect_dial9_metrics(stats: &Dial9Stats) -> Vec<PrometheusMetric> {
|
||||
let enabled = is_dial9_enabled();
|
||||
let enabled_value = if enabled { 1.0 } else { 0.0 };
|
||||
let mut metrics = vec![
|
||||
PrometheusMetric::new(
|
||||
"rustfs_dial9_supported",
|
||||
MetricType::Gauge,
|
||||
"Whether this binary was compiled with dial9 telemetry support (1) or not (0)",
|
||||
bool_gauge(is_supported()),
|
||||
),
|
||||
PrometheusMetric::new(
|
||||
"rustfs_dial9_configured",
|
||||
MetricType::Gauge,
|
||||
"Whether dial9 telemetry is requested via environment configuration (1) or not (0)",
|
||||
bool_gauge(is_configured()),
|
||||
),
|
||||
];
|
||||
|
||||
let mut metrics = vec![PrometheusMetric::new(
|
||||
"rustfs_dial9_enabled",
|
||||
MetricType::Gauge,
|
||||
"Whether dial9 telemetry is enabled (1) or disabled (0)",
|
||||
enabled_value,
|
||||
)];
|
||||
|
||||
// If dial9 is disabled, return just the enabled flag
|
||||
if !enabled {
|
||||
// Without a running session the remaining series carry no information;
|
||||
// exporting them would only publish zeros.
|
||||
if !is_enabled() {
|
||||
return metrics;
|
||||
}
|
||||
|
||||
// Add detailed metrics when enabled
|
||||
metrics.extend(vec![
|
||||
metrics.extend([
|
||||
PrometheusMetric::new(
|
||||
"rustfs_dial9_events_total",
|
||||
MetricType::Counter,
|
||||
"Total number of Tokio runtime events recorded by dial9",
|
||||
stats.events_total as f64,
|
||||
),
|
||||
PrometheusMetric::new(
|
||||
"rustfs_dial9_bytes_written_total",
|
||||
MetricType::Counter,
|
||||
"Total bytes written to dial9 trace files",
|
||||
stats.bytes_written as f64,
|
||||
),
|
||||
PrometheusMetric::new(
|
||||
"rustfs_dial9_rotations_total",
|
||||
MetricType::Counter,
|
||||
"Total number of trace file rotations",
|
||||
stats.rotation_count as f64,
|
||||
),
|
||||
PrometheusMetric::new(
|
||||
"rustfs_dial9_errors_total",
|
||||
MetricType::Counter,
|
||||
"Total number of dial9 telemetry errors",
|
||||
stats.errors_total as f64,
|
||||
),
|
||||
PrometheusMetric::new(
|
||||
"rustfs_dial9_cpu_overhead_percent",
|
||||
"rustfs_dial9_active_sessions",
|
||||
MetricType::Gauge,
|
||||
"Estimated CPU overhead percentage from dial9 telemetry",
|
||||
stats.cpu_overhead_percent,
|
||||
"Number of active dial9 telemetry sessions",
|
||||
stats.active_sessions as f64,
|
||||
),
|
||||
PrometheusMetric::new(
|
||||
"rustfs_dial9_disk_usage_bytes",
|
||||
@@ -116,80 +81,82 @@ pub fn collect_dial9_metrics(stats: &Dial9Stats) -> Vec<PrometheusMetric> {
|
||||
stats.disk_usage_bytes as f64,
|
||||
),
|
||||
PrometheusMetric::new(
|
||||
"rustfs_dial9_active_sessions",
|
||||
MetricType::Gauge,
|
||||
"Number of active dial9 telemetry sessions",
|
||||
stats.active_sessions as f64,
|
||||
"rustfs_dial9_errors_total",
|
||||
MetricType::Counter,
|
||||
"Total number of dial9 telemetry setup errors",
|
||||
stats.errors_total as f64,
|
||||
),
|
||||
]);
|
||||
|
||||
metrics
|
||||
}
|
||||
|
||||
fn bool_gauge(value: bool) -> f64 {
|
||||
if value { 1.0 } else { 0.0 }
|
||||
}
|
||||
|
||||
/// Collect dial9 metrics from the current runtime snapshot.
|
||||
///
|
||||
/// Reads cached atomics only; performs no I/O.
|
||||
pub fn collect_current_dial9_metrics() -> Vec<PrometheusMetric> {
|
||||
let snapshot = runtime_stats_snapshot();
|
||||
let stats = Dial9Stats {
|
||||
errors_total: snapshot.errors_total,
|
||||
disk_usage_bytes: snapshot.disk_usage_bytes,
|
||||
active_sessions: snapshot.active_sessions,
|
||||
..Dial9Stats::default()
|
||||
};
|
||||
|
||||
collect_dial9_metrics(&stats)
|
||||
}
|
||||
|
||||
/// Check if dial9 telemetry is enabled via environment variable.
|
||||
/// Whether dial9 telemetry is actually running: compiled in *and* configured.
|
||||
pub fn is_dial9_enabled() -> bool {
|
||||
get_env_bool(ENV_RUNTIME_DIAL9_ENABLED, DEFAULT_RUNTIME_DIAL9_ENABLED)
|
||||
is_enabled()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_dial9_stats_default() {
|
||||
let stats = Dial9Stats::default();
|
||||
assert_eq!(stats.events_total, 0);
|
||||
assert_eq!(stats.bytes_written, 0);
|
||||
assert_eq!(stats.rotation_count, 0);
|
||||
assert_eq!(stats.errors_total, 0);
|
||||
assert_eq!(stats.cpu_overhead_percent, 0.0);
|
||||
assert_eq!(stats.disk_usage_bytes, 0);
|
||||
assert_eq!(stats.active_sessions, 0);
|
||||
fn metric_names(metrics: &[PrometheusMetric]) -> Vec<String> {
|
||||
metrics.iter().map(|m| m.name.to_string()).collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_collect_dial9_metrics() {
|
||||
let stats = Dial9Stats {
|
||||
events_total: 100,
|
||||
bytes_written: 1024,
|
||||
..Default::default()
|
||||
};
|
||||
let metrics = collect_dial9_metrics(&stats);
|
||||
|
||||
// Should always have at least the enabled flag
|
||||
assert!(!metrics.is_empty());
|
||||
fn always_reports_support_and_intent() {
|
||||
let names = metric_names(&collect_dial9_metrics(&Dial9Stats::default()));
|
||||
assert!(names.iter().any(|n| n == "rustfs_dial9_supported"));
|
||||
assert!(names.iter().any(|n| n == "rustfs_dial9_configured"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_collect_dial9_metrics_with_values() {
|
||||
let stats = Dial9Stats {
|
||||
events_total: 10000,
|
||||
bytes_written: 1024000,
|
||||
rotation_count: 5,
|
||||
errors_total: 0,
|
||||
cpu_overhead_percent: 2.5,
|
||||
disk_usage_bytes: 2048000,
|
||||
active_sessions: 1,
|
||||
};
|
||||
fn omits_session_metrics_when_telemetry_is_not_running() {
|
||||
// Under `cargo test` dial9 is neither compiled in nor configured, so the
|
||||
// collector must stop after the two compile-time/intent gauges rather
|
||||
// than publish zeroed session series.
|
||||
if is_enabled() {
|
||||
return;
|
||||
}
|
||||
let metrics = collect_dial9_metrics(&Dial9Stats::default());
|
||||
assert_eq!(metrics.len(), 2);
|
||||
let names = metric_names(&metrics);
|
||||
assert!(!names.iter().any(|n| n == "rustfs_dial9_disk_usage_bytes"));
|
||||
assert!(!names.iter().any(|n| n == "rustfs_dial9_active_sessions"));
|
||||
}
|
||||
|
||||
let metrics = collect_dial9_metrics(&stats);
|
||||
#[test]
|
||||
fn supported_gauge_tracks_compile_time_feature() {
|
||||
let metrics = collect_dial9_metrics(&Dial9Stats::default());
|
||||
let supported = metrics
|
||||
.iter()
|
||||
.find(|m| m.name == "rustfs_dial9_supported")
|
||||
.expect("supported gauge is always exported");
|
||||
assert_eq!(supported.value, bool_gauge(cfg!(feature = "dial9")));
|
||||
}
|
||||
|
||||
// When dial9 is enabled, should have all metrics
|
||||
// Note: This test assumes dial9 is enabled in the test environment
|
||||
// If disabled, only the enabled flag metric will be present
|
||||
assert!(!metrics.is_empty());
|
||||
#[test]
|
||||
fn bool_gauge_maps_to_one_and_zero() {
|
||||
assert_eq!(bool_gauge(true), 1.0);
|
||||
assert_eq!(bool_gauge(false), 0.0);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,640 +0,0 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! dial9-tokio-telemetry integration for RustFS.
|
||||
//!
|
||||
//! This module provides low-overhead Tokio runtime-level telemetry,
|
||||
//! capturing events like PollStart/End, WorkerPark/Unpark, QueueSample, etc.
|
||||
|
||||
use crate::TelemetryError;
|
||||
// Import and re-export TelemetryGuard for use in other crates (like rustfs)
|
||||
// Use as Dial9TelemetryGuard internally to avoid naming conflicts
|
||||
use dial9_tokio_telemetry::telemetry::RotatingWriter;
|
||||
pub use dial9_tokio_telemetry::telemetry::TelemetryGuard;
|
||||
use dial9_tokio_telemetry::telemetry::TelemetryGuard as Dial9TelemetryGuard;
|
||||
// Use rustfs_config which re-exports runtime constants
|
||||
use rustfs_config::{
|
||||
DEFAULT_RUNTIME_DIAL9_ENABLED, DEFAULT_RUNTIME_DIAL9_FILE_PREFIX, DEFAULT_RUNTIME_DIAL9_MAX_FILE_SIZE,
|
||||
DEFAULT_RUNTIME_DIAL9_OUTPUT_DIR, DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT, DEFAULT_RUNTIME_DIAL9_SAMPLING_RATE,
|
||||
ENV_RUNTIME_DIAL9_ENABLED, ENV_RUNTIME_DIAL9_FILE_PREFIX, ENV_RUNTIME_DIAL9_MAX_FILE_SIZE, ENV_RUNTIME_DIAL9_OUTPUT_DIR,
|
||||
ENV_RUNTIME_DIAL9_ROTATION_COUNT, ENV_RUNTIME_DIAL9_S3_BUCKET, ENV_RUNTIME_DIAL9_S3_PREFIX, ENV_RUNTIME_DIAL9_SAMPLING_RATE,
|
||||
};
|
||||
use rustfs_utils::get_env_bool;
|
||||
use rustfs_utils::get_env_opt_str;
|
||||
use rustfs_utils::get_env_opt_u64;
|
||||
use rustfs_utils::get_env_opt_usize;
|
||||
use rustfs_utils::get_env_str;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::OnceLock;
|
||||
use std::sync::RwLock;
|
||||
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
|
||||
use tracing::{info, warn};
|
||||
|
||||
const LOG_COMPONENT_OBS: &str = "obs";
|
||||
const LOG_SUBSYSTEM_DIAL9: &str = "dial9";
|
||||
const EVENT_DIAL9_STATE: &str = "dial9_state";
|
||||
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub(crate) struct Dial9RuntimeSnapshot {
|
||||
pub active_sessions: u64,
|
||||
pub errors_total: u64,
|
||||
pub disk_usage_bytes: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
struct Dial9RuntimeState {
|
||||
enabled: AtomicBool,
|
||||
active_sessions: AtomicU64,
|
||||
errors_total: AtomicU64,
|
||||
output_dir: RwLock<Option<PathBuf>>,
|
||||
file_prefix: RwLock<Option<String>>,
|
||||
}
|
||||
|
||||
impl Dial9RuntimeState {
|
||||
fn new() -> Self {
|
||||
Self {
|
||||
enabled: AtomicBool::new(false),
|
||||
active_sessions: AtomicU64::new(0),
|
||||
errors_total: AtomicU64::new(0),
|
||||
output_dir: RwLock::new(None),
|
||||
file_prefix: RwLock::new(None),
|
||||
}
|
||||
}
|
||||
|
||||
fn record_config(&self, config: &Dial9Config) {
|
||||
self.enabled.store(config.enabled, Ordering::Relaxed);
|
||||
*self.output_dir.write().expect("dial9 output_dir lock should not be poisoned") = Some(PathBuf::from(&config.output_dir));
|
||||
*self
|
||||
.file_prefix
|
||||
.write()
|
||||
.expect("dial9 file_prefix lock should not be poisoned") = Some(config.file_prefix.clone());
|
||||
if !config.enabled {
|
||||
self.active_sessions.store(0, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
|
||||
fn record_runtime_started(&self, config: &Dial9Config) {
|
||||
self.record_config(config);
|
||||
self.active_sessions.store(1, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
fn record_runtime_error(&self, config: &Dial9Config) {
|
||||
self.record_config(config);
|
||||
self.active_sessions.store(0, Ordering::Relaxed);
|
||||
self.errors_total.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
fn snapshot(&self) -> Dial9RuntimeSnapshot {
|
||||
Dial9RuntimeSnapshot {
|
||||
active_sessions: self.active_sessions.load(Ordering::Relaxed),
|
||||
errors_total: self.errors_total.load(Ordering::Relaxed),
|
||||
disk_usage_bytes: self.current_disk_usage_bytes(),
|
||||
}
|
||||
}
|
||||
|
||||
fn current_disk_usage_bytes(&self) -> u64 {
|
||||
let output_dir = self
|
||||
.output_dir
|
||||
.read()
|
||||
.expect("dial9 output_dir lock should not be poisoned")
|
||||
.clone();
|
||||
let file_prefix = self
|
||||
.file_prefix
|
||||
.read()
|
||||
.expect("dial9 file_prefix lock should not be poisoned")
|
||||
.clone();
|
||||
|
||||
let Some(output_dir) = output_dir else {
|
||||
return 0;
|
||||
};
|
||||
let Some(file_prefix) = file_prefix else {
|
||||
return 0;
|
||||
};
|
||||
|
||||
std::fs::read_dir(output_dir)
|
||||
.ok()
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.filter_map(Result::ok)
|
||||
.filter_map(|entry| {
|
||||
let filename = entry.file_name();
|
||||
let filename = filename.to_str()?;
|
||||
filename
|
||||
.starts_with(&file_prefix)
|
||||
.then(|| entry.metadata().ok().map(|meta| meta.len()))
|
||||
})
|
||||
.flatten()
|
||||
.sum()
|
||||
}
|
||||
}
|
||||
|
||||
fn dial9_runtime_state() -> &'static Dial9RuntimeState {
|
||||
static STATE: OnceLock<Dial9RuntimeState> = OnceLock::new();
|
||||
STATE.get_or_init(Dial9RuntimeState::new)
|
||||
}
|
||||
|
||||
fn resolve_sampling_rate() -> (f64, bool) {
|
||||
match std::env::var(ENV_RUNTIME_DIAL9_SAMPLING_RATE) {
|
||||
Ok(raw) => {
|
||||
let trimmed = raw.trim();
|
||||
if trimmed.is_empty() {
|
||||
return (DEFAULT_RUNTIME_DIAL9_SAMPLING_RATE, false);
|
||||
}
|
||||
|
||||
match trimmed.parse::<f64>() {
|
||||
Ok(value) if value.is_finite() => (value.clamp(0.0, 1.0), true),
|
||||
_ => {
|
||||
warn!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
result = "invalid_sampling_rate",
|
||||
provided = trimmed,
|
||||
fallback = DEFAULT_RUNTIME_DIAL9_SAMPLING_RATE,
|
||||
"dial9 state changed"
|
||||
);
|
||||
(DEFAULT_RUNTIME_DIAL9_SAMPLING_RATE, true)
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(_) => (DEFAULT_RUNTIME_DIAL9_SAMPLING_RATE, false),
|
||||
}
|
||||
}
|
||||
|
||||
fn warn_ignored_runtime_knobs(config: &Dial9Config, sampling_rate_explicit: bool) {
|
||||
if config.s3_bucket.is_some() || config.s3_prefix.is_some() {
|
||||
warn!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
result = "unsupported_s3_upload_knobs_ignored",
|
||||
s3_bucket = config.s3_bucket.as_deref().unwrap_or(""),
|
||||
s3_prefix = config.s3_prefix.as_deref().unwrap_or(""),
|
||||
"dial9 state changed"
|
||||
);
|
||||
}
|
||||
|
||||
if sampling_rate_explicit {
|
||||
warn!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
result = "unsupported_sampling_rate_ignored",
|
||||
sampling_rate = config.sampling_rate,
|
||||
"dial9 state changed"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
fn sanitize_dial9_max_file_size(bytes: u64) -> u64 {
|
||||
bytes.max(1)
|
||||
}
|
||||
|
||||
fn sanitize_dial9_rotation_count(count: usize) -> usize {
|
||||
count.max(1)
|
||||
}
|
||||
|
||||
fn dial9_total_rotation_size(max_file_size: u64, rotation_count: usize) -> u64 {
|
||||
max_file_size.saturating_mul(u64::try_from(rotation_count).unwrap_or(u64::MAX))
|
||||
}
|
||||
|
||||
/// Configuration for dial9 Tokio telemetry.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct Dial9Config {
|
||||
/// Whether dial9 telemetry is enabled
|
||||
pub enabled: bool,
|
||||
|
||||
/// Directory where trace files are written
|
||||
pub output_dir: String,
|
||||
|
||||
/// Prefix for trace file names
|
||||
pub file_prefix: String,
|
||||
|
||||
/// Maximum size of each trace file in bytes
|
||||
pub max_file_size: u64,
|
||||
|
||||
/// Number of rotated files to keep
|
||||
pub rotation_count: usize,
|
||||
|
||||
/// Optional S3 bucket for uploading trace files
|
||||
pub s3_bucket: Option<String>,
|
||||
|
||||
/// Optional S3 prefix for uploaded files
|
||||
pub s3_prefix: Option<String>,
|
||||
|
||||
/// Sampling rate (0.0 to 1.0)
|
||||
pub sampling_rate: f64,
|
||||
}
|
||||
|
||||
impl Default for Dial9Config {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
enabled: DEFAULT_RUNTIME_DIAL9_ENABLED,
|
||||
output_dir: DEFAULT_RUNTIME_DIAL9_OUTPUT_DIR.to_string(),
|
||||
file_prefix: DEFAULT_RUNTIME_DIAL9_FILE_PREFIX.to_string(),
|
||||
max_file_size: DEFAULT_RUNTIME_DIAL9_MAX_FILE_SIZE,
|
||||
rotation_count: DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT,
|
||||
s3_bucket: None,
|
||||
s3_prefix: None,
|
||||
sampling_rate: DEFAULT_RUNTIME_DIAL9_SAMPLING_RATE,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Dial9Config {
|
||||
/// Create configuration from environment variables.
|
||||
pub fn from_env() -> Self {
|
||||
let enabled = get_env_bool(ENV_RUNTIME_DIAL9_ENABLED, DEFAULT_RUNTIME_DIAL9_ENABLED);
|
||||
|
||||
if !enabled {
|
||||
let config = Self::default();
|
||||
dial9_runtime_state().record_config(&config);
|
||||
return config;
|
||||
}
|
||||
|
||||
let raw_max_file_size = get_env_opt_u64(ENV_RUNTIME_DIAL9_MAX_FILE_SIZE);
|
||||
let raw_rotation_count = get_env_opt_usize(ENV_RUNTIME_DIAL9_ROTATION_COUNT);
|
||||
let (sampling_rate, sampling_rate_explicit) = resolve_sampling_rate();
|
||||
let max_file_size = sanitize_dial9_max_file_size(raw_max_file_size.unwrap_or(DEFAULT_RUNTIME_DIAL9_MAX_FILE_SIZE));
|
||||
let rotation_count = sanitize_dial9_rotation_count(raw_rotation_count.unwrap_or(DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT));
|
||||
if raw_max_file_size == Some(0) {
|
||||
warn!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
result = "invalid_max_file_size",
|
||||
fallback_bytes = 1_u64,
|
||||
"dial9 state changed"
|
||||
);
|
||||
}
|
||||
if raw_rotation_count == Some(0) {
|
||||
warn!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
result = "invalid_rotation_count",
|
||||
fallback_count = 1_usize,
|
||||
"dial9 state changed"
|
||||
);
|
||||
}
|
||||
|
||||
let config = Self {
|
||||
enabled,
|
||||
output_dir: get_env_str(ENV_RUNTIME_DIAL9_OUTPUT_DIR, DEFAULT_RUNTIME_DIAL9_OUTPUT_DIR),
|
||||
file_prefix: get_env_str(ENV_RUNTIME_DIAL9_FILE_PREFIX, DEFAULT_RUNTIME_DIAL9_FILE_PREFIX),
|
||||
max_file_size,
|
||||
rotation_count,
|
||||
s3_bucket: get_env_opt_str(ENV_RUNTIME_DIAL9_S3_BUCKET).filter(|s| !s.is_empty()),
|
||||
s3_prefix: get_env_opt_str(ENV_RUNTIME_DIAL9_S3_PREFIX).filter(|s| !s.is_empty()),
|
||||
sampling_rate,
|
||||
};
|
||||
warn_ignored_runtime_knobs(&config, sampling_rate_explicit);
|
||||
dial9_runtime_state().record_config(&config);
|
||||
config
|
||||
}
|
||||
|
||||
/// Get the base path for trace files.
|
||||
pub fn base_path(&self) -> PathBuf {
|
||||
PathBuf::from(&self.output_dir).join(&self.file_prefix)
|
||||
}
|
||||
}
|
||||
|
||||
/// Guard for dial9 telemetry session.
|
||||
///
|
||||
/// When dropped, this guard will flush any remaining telemetry data.
|
||||
/// Keep it alive for the duration of your application.
|
||||
pub struct Dial9SessionGuard {
|
||||
/// The underlying dial9 telemetry guard (if enabled)
|
||||
_guard: Option<Dial9TelemetryGuard>,
|
||||
/// Configuration
|
||||
#[allow(dead_code)]
|
||||
config: Dial9Config,
|
||||
}
|
||||
|
||||
impl Dial9SessionGuard {
|
||||
/// Create a new dial9 session guard.
|
||||
///
|
||||
/// Note: This only validates configuration and creates the output directory.
|
||||
/// The actual telemetry session is created when building the Tokio runtime
|
||||
/// via `build_traced_runtime()`.
|
||||
///
|
||||
/// Returns `Ok(None)` if dial9 is disabled.
|
||||
pub async fn new(config: Dial9Config) -> Result<Option<Self>, TelemetryError> {
|
||||
if !config.enabled {
|
||||
info!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
state = "disabled",
|
||||
"dial9 state changed"
|
||||
);
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
info!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
state = "validating",
|
||||
output_dir = %config.output_dir,
|
||||
file_prefix = %config.file_prefix,
|
||||
sampling_rate = config.sampling_rate,
|
||||
"dial9 state changed"
|
||||
);
|
||||
|
||||
// Only create directory; writer will be created in build_traced_runtime
|
||||
if let Err(e) = tokio::fs::create_dir_all(&config.output_dir).await {
|
||||
warn!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
result = "output_dir_create_failed",
|
||||
output_dir = %config.output_dir,
|
||||
error = %e,
|
||||
fallback = "disabled",
|
||||
"dial9 state changed"
|
||||
);
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
info!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
state = "validated",
|
||||
output_dir = %config.output_dir,
|
||||
file_prefix = %config.file_prefix,
|
||||
"dial9 state changed"
|
||||
);
|
||||
|
||||
dial9_runtime_state().record_config(&config);
|
||||
Ok(Some(Self { _guard: None, config }))
|
||||
}
|
||||
|
||||
/// Set the telemetry guard (called after runtime creation)
|
||||
#[allow(dead_code)]
|
||||
pub(crate) fn set_guard(&mut self, guard: Dial9TelemetryGuard) {
|
||||
self._guard = Some(guard);
|
||||
dial9_runtime_state().record_runtime_started(&self.config);
|
||||
}
|
||||
|
||||
/// Check if this guard has an active session.
|
||||
pub fn is_active(&self) -> bool {
|
||||
dial9_runtime_state().snapshot().active_sessions > 0
|
||||
}
|
||||
|
||||
/// Flush any pending telemetry data.
|
||||
pub async fn shutdown(&self) {
|
||||
if self.is_active() {
|
||||
info!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
state = "shutdown_requested",
|
||||
result = "runtime_guard_managed_elsewhere",
|
||||
"dial9 state"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for Dial9SessionGuard {
|
||||
fn drop(&mut self) {
|
||||
if let Some(_guard) = &self._guard {
|
||||
// TelemetryGuard flushes automatically when dropped
|
||||
info!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
state = "flushed",
|
||||
"dial9 state"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Initialize dial9 telemetry session from environment configuration.
|
||||
///
|
||||
/// This function reads configuration from environment variables and creates
|
||||
/// a dial9 session guard if enabled. The guard should be kept alive for the
|
||||
/// duration of the application.
|
||||
///
|
||||
/// # Returns
|
||||
///
|
||||
/// - `Ok(Some(guard))` - Dial9 is enabled and session initialized
|
||||
/// - `Ok(None)` - Dial9 is disabled or failed to initialize (non-fatal)
|
||||
/// - `Err(e)` - Fatal error (should not happen with current implementation)
|
||||
pub async fn init_session() -> Result<Option<Dial9SessionGuard>, TelemetryError> {
|
||||
let config = Dial9Config::from_env();
|
||||
Dial9SessionGuard::new(config).await
|
||||
}
|
||||
|
||||
/// Check if dial9 telemetry is enabled via environment configuration.
|
||||
pub fn is_enabled() -> bool {
|
||||
get_env_bool(ENV_RUNTIME_DIAL9_ENABLED, DEFAULT_RUNTIME_DIAL9_ENABLED)
|
||||
}
|
||||
|
||||
/// Build a Tokio runtime with dial9 telemetry enabled.
|
||||
///
|
||||
/// This function creates a Tokio runtime with dial9 telemetry integrated
|
||||
/// if enabled via environment variables. Returns a tuple of (Runtime, TelemetryGuard).
|
||||
///
|
||||
/// This is internal API used by the runtime builder.
|
||||
///
|
||||
/// # Arguments
|
||||
///
|
||||
/// * `builder` - The configured Tokio runtime builder
|
||||
///
|
||||
/// # Returns
|
||||
///
|
||||
/// * `Ok((runtime, guard))` - Successfully created runtime with telemetry
|
||||
/// * `Err` - If runtime creation fails or dial9 is enabled but fails to initialize
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Returns an error if:
|
||||
/// - Dial9 is enabled but the runtime builder fails
|
||||
/// - Dial9 is enabled but writer creation fails
|
||||
pub fn build_traced_runtime(
|
||||
builder: tokio::runtime::Builder,
|
||||
) -> Result<(tokio::runtime::Runtime, Dial9TelemetryGuard), TelemetryError> {
|
||||
if !is_enabled() {
|
||||
return Err(TelemetryError::Io("Dial9 is not enabled".to_string()));
|
||||
}
|
||||
|
||||
let config = Dial9Config::from_env();
|
||||
dial9_runtime_state().record_config(&config);
|
||||
|
||||
// Ensure the output directory exists before creating the writer
|
||||
std::fs::create_dir_all(&config.output_dir).map_err(|e| {
|
||||
dial9_runtime_state().record_runtime_error(&config);
|
||||
TelemetryError::Io(format!("Failed to create dial9 output directory '{}': {}", config.output_dir, e))
|
||||
})?;
|
||||
|
||||
// Create rotating writer (synchronous for runtime building)
|
||||
let base_path = config.base_path();
|
||||
let writer = RotatingWriter::new(
|
||||
base_path,
|
||||
config.max_file_size,
|
||||
dial9_total_rotation_size(config.max_file_size, config.rotation_count),
|
||||
)
|
||||
.map_err(|e| {
|
||||
dial9_runtime_state().record_runtime_error(&config);
|
||||
TelemetryError::Io(format!("Failed to create RotatingWriter: {}", e))
|
||||
})?;
|
||||
|
||||
// Build traced runtime
|
||||
// Note: sampling_rate and S3 upload settings are reserved for future use
|
||||
// once the dial9 library provides support for those configuration options.
|
||||
let (runtime, guard) = dial9_tokio_telemetry::telemetry::TracedRuntime::builder()
|
||||
.with_task_tracking(true)
|
||||
.build(builder, writer)
|
||||
.map_err(|e| {
|
||||
dial9_runtime_state().record_runtime_error(&config);
|
||||
TelemetryError::Io(format!("Failed to build TracedRuntime: {}", e))
|
||||
})?;
|
||||
|
||||
dial9_runtime_state().record_runtime_started(&config);
|
||||
Ok((runtime, guard))
|
||||
}
|
||||
|
||||
pub(crate) fn runtime_stats_snapshot() -> Dial9RuntimeSnapshot {
|
||||
dial9_runtime_state().snapshot()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::sync::{Mutex, OnceLock};
|
||||
use tempfile::tempdir;
|
||||
|
||||
static DIAL9_ENV_TEST_LOCK: OnceLock<Mutex<()>> = OnceLock::new();
|
||||
|
||||
fn with_dial9_env_lock<F>(f: F)
|
||||
where
|
||||
F: FnOnce(),
|
||||
{
|
||||
let _guard = DIAL9_ENV_TEST_LOCK
|
||||
.get_or_init(|| Mutex::new(()))
|
||||
.lock()
|
||||
.expect("dial9 env test lock should not be poisoned");
|
||||
f();
|
||||
}
|
||||
|
||||
fn reset_dial9_runtime_state_for_test() {
|
||||
let state = dial9_runtime_state();
|
||||
state.enabled.store(false, Ordering::Relaxed);
|
||||
state.active_sessions.store(0, Ordering::Relaxed);
|
||||
state.errors_total.store(0, Ordering::Relaxed);
|
||||
*state
|
||||
.output_dir
|
||||
.write()
|
||||
.expect("dial9 output_dir lock should not be poisoned") = None;
|
||||
*state
|
||||
.file_prefix
|
||||
.write()
|
||||
.expect("dial9 file_prefix lock should not be poisoned") = None;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_dial9_config_default() {
|
||||
let config = Dial9Config::default();
|
||||
assert!(!config.enabled);
|
||||
assert_eq!(config.output_dir, DEFAULT_RUNTIME_DIAL9_OUTPUT_DIR);
|
||||
assert_eq!(config.file_prefix, DEFAULT_RUNTIME_DIAL9_FILE_PREFIX);
|
||||
assert_eq!(config.max_file_size, DEFAULT_RUNTIME_DIAL9_MAX_FILE_SIZE);
|
||||
assert_eq!(config.rotation_count, DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT);
|
||||
assert_eq!(config.sampling_rate, DEFAULT_RUNTIME_DIAL9_SAMPLING_RATE);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_dial9_config_base_path() {
|
||||
let config = Dial9Config {
|
||||
output_dir: "/tmp/telemetry".to_string(),
|
||||
file_prefix: "rustfs".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
assert_eq!(config.base_path(), PathBuf::from("/tmp/telemetry/rustfs"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_dial9_sanitizers_reject_zero_values() {
|
||||
assert_eq!(sanitize_dial9_max_file_size(0), 1);
|
||||
assert_eq!(sanitize_dial9_rotation_count(0), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_dial9_total_rotation_size_saturates() {
|
||||
assert_eq!(dial9_total_rotation_size(u64::MAX, 2), u64::MAX);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_is_enabled_default() {
|
||||
// Skip if environment variable is explicitly set
|
||||
if std::env::var(ENV_RUNTIME_DIAL9_ENABLED).is_ok() {
|
||||
println!("Skipping test: RUSTFS_RUNTIME_DIAL9_ENABLED is set");
|
||||
return;
|
||||
}
|
||||
assert!(!is_enabled());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sampling_rate_rejects_nan_and_falls_back_to_default() {
|
||||
reset_dial9_runtime_state_for_test();
|
||||
with_dial9_env_lock(|| {
|
||||
temp_env::with_var(ENV_RUNTIME_DIAL9_ENABLED, Some("true"), || {
|
||||
temp_env::with_var(ENV_RUNTIME_DIAL9_SAMPLING_RATE, Some("NaN"), || {
|
||||
let config = Dial9Config::from_env();
|
||||
assert_eq!(config.sampling_rate, DEFAULT_RUNTIME_DIAL9_SAMPLING_RATE);
|
||||
});
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn session_guard_is_active_when_runtime_state_reports_an_active_session() {
|
||||
reset_dial9_runtime_state_for_test();
|
||||
let config = Dial9Config {
|
||||
enabled: true,
|
||||
..Dial9Config::default()
|
||||
};
|
||||
dial9_runtime_state().record_runtime_started(&config);
|
||||
|
||||
let guard = Dial9SessionGuard { _guard: None, config };
|
||||
|
||||
assert!(guard.is_active());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_stats_snapshot_reports_disk_usage_and_errors() {
|
||||
reset_dial9_runtime_state_for_test();
|
||||
let dir = tempdir().expect("create temp dir");
|
||||
let trace_path = dir.path().join("dial9-trace.segment");
|
||||
std::fs::write(&trace_path, vec![0_u8; 128]).expect("write fake dial9 trace");
|
||||
|
||||
let config = Dial9Config {
|
||||
enabled: true,
|
||||
output_dir: dir.path().to_string_lossy().into_owned(),
|
||||
file_prefix: "dial9-trace".to_string(),
|
||||
..Dial9Config::default()
|
||||
};
|
||||
dial9_runtime_state().record_runtime_started(&config);
|
||||
dial9_runtime_state().record_runtime_error(&config);
|
||||
|
||||
let snapshot = runtime_stats_snapshot();
|
||||
assert_eq!(snapshot.active_sessions, 0);
|
||||
assert_eq!(snapshot.errors_total, 1);
|
||||
assert_eq!(snapshot.disk_usage_bytes, 128);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,276 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Configuration for dial9 Tokio runtime telemetry.
|
||||
//!
|
||||
//! This module carries no dependency on the `dial9-tokio-telemetry` crate, so
|
||||
//! it compiles identically whether or not the `dial9` feature is enabled. That
|
||||
//! lets callers read the configured state (and export metrics about it) from a
|
||||
//! binary that was built without telemetry support.
|
||||
|
||||
use super::state::dial9_runtime_state;
|
||||
use rustfs_config::{
|
||||
DEFAULT_RUNTIME_DIAL9_ENABLED, DEFAULT_RUNTIME_DIAL9_FILE_PREFIX, DEFAULT_RUNTIME_DIAL9_MAX_FILE_SIZE,
|
||||
DEFAULT_RUNTIME_DIAL9_OUTPUT_DIR, DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT, DEFAULT_RUNTIME_DIAL9_TASK_DUMP_ENABLED,
|
||||
DEFAULT_RUNTIME_DIAL9_TASK_DUMP_IDLE_THRESHOLD_MS, ENV_RUNTIME_DIAL9_ENABLED, ENV_RUNTIME_DIAL9_FILE_PREFIX,
|
||||
ENV_RUNTIME_DIAL9_MAX_FILE_SIZE, ENV_RUNTIME_DIAL9_OUTPUT_DIR, ENV_RUNTIME_DIAL9_ROTATION_COUNT, ENV_RUNTIME_DIAL9_S3_BUCKET,
|
||||
ENV_RUNTIME_DIAL9_S3_PREFIX, ENV_RUNTIME_DIAL9_TASK_DUMP_ENABLED, ENV_RUNTIME_DIAL9_TASK_DUMP_IDLE_THRESHOLD_MS,
|
||||
};
|
||||
use rustfs_utils::{get_env_bool, get_env_opt_str, get_env_opt_u64, get_env_opt_usize, get_env_str};
|
||||
use std::path::PathBuf;
|
||||
use std::time::Duration;
|
||||
use tracing::warn;
|
||||
|
||||
use super::{EVENT_DIAL9_STATE, LOG_COMPONENT_OBS, LOG_SUBSYSTEM_DIAL9};
|
||||
|
||||
/// Whether this binary was compiled with dial9 telemetry support.
|
||||
///
|
||||
/// Telemetry requires both the `dial9` cargo feature and `--cfg tokio_unstable`;
|
||||
/// the crate's build script enforces that pairing, so this single flag is
|
||||
/// sufficient to describe compile-time support.
|
||||
pub const fn is_supported() -> bool {
|
||||
cfg!(feature = "dial9")
|
||||
}
|
||||
|
||||
/// Whether dial9 telemetry is requested via environment configuration.
|
||||
///
|
||||
/// This reports operator intent and is independent of whether the binary can
|
||||
/// honour it. Use [`is_enabled`] to test whether telemetry will actually run.
|
||||
pub fn is_configured() -> bool {
|
||||
get_env_bool(ENV_RUNTIME_DIAL9_ENABLED, DEFAULT_RUNTIME_DIAL9_ENABLED)
|
||||
}
|
||||
|
||||
/// Whether dial9 telemetry will actually run: requested *and* compiled in.
|
||||
pub fn is_enabled() -> bool {
|
||||
is_supported() && is_configured()
|
||||
}
|
||||
|
||||
fn sanitize_max_file_size(bytes: u64) -> u64 {
|
||||
bytes.max(1)
|
||||
}
|
||||
|
||||
fn sanitize_rotation_count(count: usize) -> usize {
|
||||
count.max(1)
|
||||
}
|
||||
|
||||
/// Total on-disk byte budget for the trace family, saturating on overflow.
|
||||
pub(super) fn total_rotation_size(max_file_size: u64, rotation_count: usize) -> u64 {
|
||||
max_file_size.saturating_mul(u64::try_from(rotation_count).unwrap_or(u64::MAX))
|
||||
}
|
||||
|
||||
/// Configuration for dial9 Tokio telemetry.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct Dial9Config {
|
||||
/// Whether dial9 telemetry is enabled
|
||||
pub enabled: bool,
|
||||
|
||||
/// Directory where trace files are written
|
||||
pub output_dir: String,
|
||||
|
||||
/// Prefix for trace file names
|
||||
pub file_prefix: String,
|
||||
|
||||
/// Maximum size of each trace file in bytes
|
||||
pub max_file_size: u64,
|
||||
|
||||
/// Number of rotated files to keep
|
||||
pub rotation_count: usize,
|
||||
|
||||
/// Optional S3 bucket for uploading sealed trace segments
|
||||
pub s3_bucket: Option<String>,
|
||||
|
||||
/// Optional key prefix for uploaded segments
|
||||
pub s3_prefix: Option<String>,
|
||||
|
||||
/// Whether to capture async backtraces for tasks that stall
|
||||
pub task_dump_enabled: bool,
|
||||
|
||||
/// Mean idle duration for task-dump Poisson sampling
|
||||
pub task_dump_idle_threshold: Duration,
|
||||
}
|
||||
|
||||
impl Default for Dial9Config {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
enabled: DEFAULT_RUNTIME_DIAL9_ENABLED,
|
||||
output_dir: DEFAULT_RUNTIME_DIAL9_OUTPUT_DIR.to_string(),
|
||||
file_prefix: DEFAULT_RUNTIME_DIAL9_FILE_PREFIX.to_string(),
|
||||
max_file_size: DEFAULT_RUNTIME_DIAL9_MAX_FILE_SIZE,
|
||||
rotation_count: DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT,
|
||||
s3_bucket: None,
|
||||
s3_prefix: None,
|
||||
task_dump_enabled: DEFAULT_RUNTIME_DIAL9_TASK_DUMP_ENABLED,
|
||||
task_dump_idle_threshold: Duration::from_millis(DEFAULT_RUNTIME_DIAL9_TASK_DUMP_IDLE_THRESHOLD_MS),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Dial9Config {
|
||||
/// Create configuration from environment variables.
|
||||
pub fn from_env() -> Self {
|
||||
let enabled = is_enabled();
|
||||
|
||||
if !enabled {
|
||||
let config = Self::default();
|
||||
dial9_runtime_state().record_config(&config);
|
||||
return config;
|
||||
}
|
||||
|
||||
let raw_max_file_size = get_env_opt_u64(ENV_RUNTIME_DIAL9_MAX_FILE_SIZE);
|
||||
let raw_rotation_count = get_env_opt_usize(ENV_RUNTIME_DIAL9_ROTATION_COUNT);
|
||||
let max_file_size = sanitize_max_file_size(raw_max_file_size.unwrap_or(DEFAULT_RUNTIME_DIAL9_MAX_FILE_SIZE));
|
||||
let rotation_count = sanitize_rotation_count(raw_rotation_count.unwrap_or(DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT));
|
||||
|
||||
if raw_max_file_size == Some(0) {
|
||||
warn!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
result = "invalid_max_file_size",
|
||||
fallback_bytes = 1_u64,
|
||||
"dial9 state changed"
|
||||
);
|
||||
}
|
||||
if raw_rotation_count == Some(0) {
|
||||
warn!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
result = "invalid_rotation_count",
|
||||
fallback_count = 1_usize,
|
||||
"dial9 state changed"
|
||||
);
|
||||
}
|
||||
|
||||
let s3_bucket = get_env_opt_str(ENV_RUNTIME_DIAL9_S3_BUCKET).filter(|s| !s.is_empty());
|
||||
let s3_prefix = get_env_opt_str(ENV_RUNTIME_DIAL9_S3_PREFIX).filter(|s| !s.is_empty());
|
||||
warn_unusable_s3_upload(s3_bucket.as_deref(), s3_prefix.as_deref());
|
||||
|
||||
let config = Self {
|
||||
enabled,
|
||||
output_dir: get_env_str(ENV_RUNTIME_DIAL9_OUTPUT_DIR, DEFAULT_RUNTIME_DIAL9_OUTPUT_DIR),
|
||||
file_prefix: get_env_str(ENV_RUNTIME_DIAL9_FILE_PREFIX, DEFAULT_RUNTIME_DIAL9_FILE_PREFIX),
|
||||
max_file_size,
|
||||
rotation_count,
|
||||
s3_bucket,
|
||||
s3_prefix,
|
||||
task_dump_enabled: get_env_bool(ENV_RUNTIME_DIAL9_TASK_DUMP_ENABLED, DEFAULT_RUNTIME_DIAL9_TASK_DUMP_ENABLED),
|
||||
task_dump_idle_threshold: Duration::from_millis(
|
||||
get_env_opt_u64(ENV_RUNTIME_DIAL9_TASK_DUMP_IDLE_THRESHOLD_MS)
|
||||
.filter(|ms| *ms > 0)
|
||||
.unwrap_or(DEFAULT_RUNTIME_DIAL9_TASK_DUMP_IDLE_THRESHOLD_MS),
|
||||
),
|
||||
};
|
||||
|
||||
dial9_runtime_state().record_config(&config);
|
||||
config
|
||||
}
|
||||
|
||||
/// Get the base path for trace files.
|
||||
pub fn base_path(&self) -> PathBuf {
|
||||
PathBuf::from(&self.output_dir).join(&self.file_prefix)
|
||||
}
|
||||
|
||||
/// Total on-disk byte budget across all retained segments.
|
||||
pub fn total_disk_budget(&self) -> u64 {
|
||||
total_rotation_size(self.max_file_size, self.rotation_count)
|
||||
}
|
||||
|
||||
/// Whether S3 upload of sealed segments is both requested and usable.
|
||||
pub fn s3_upload_requested(&self) -> bool {
|
||||
self.s3_bucket.is_some()
|
||||
}
|
||||
}
|
||||
|
||||
/// S3 upload of sealed segments is not available in any build.
|
||||
///
|
||||
/// dial9's `worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3,
|
||||
/// which pins hyper-rustls 0.24 / rustls-webpki 0.101.7 — a webpki carrying
|
||||
/// RUSTSEC-2026-0098, -0099 and -0104. Cargo cannot drop a transitive
|
||||
/// dependency, so this needs an upstream fix (rustfs/backlog#1157, D9-14).
|
||||
///
|
||||
/// Warn loudly rather than silently discarding a configured bucket: an operator
|
||||
/// who set these expects their traces to be uploaded somewhere.
|
||||
fn warn_unusable_s3_upload(s3_bucket: Option<&str>, s3_prefix: Option<&str>) {
|
||||
if s3_bucket.is_none() && s3_prefix.is_none() {
|
||||
return;
|
||||
}
|
||||
|
||||
warn!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
result = "s3_upload_unsupported",
|
||||
s3_bucket = s3_bucket.unwrap_or(""),
|
||||
s3_prefix = s3_prefix.unwrap_or(""),
|
||||
reason = "dial9 worker-s3 depends on a vulnerable rustls-webpki (RUSTSEC-2026-0098/0099/0104)",
|
||||
remedy = "collect trace segments from the output directory instead",
|
||||
"dial9 state changed"
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn sanitizers_reject_zero_values() {
|
||||
assert_eq!(sanitize_max_file_size(0), 1);
|
||||
assert_eq!(sanitize_rotation_count(0), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn total_rotation_size_saturates() {
|
||||
assert_eq!(total_rotation_size(u64::MAX, 2), u64::MAX);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn default_config_is_disabled() {
|
||||
let config = Dial9Config::default();
|
||||
assert!(!config.enabled);
|
||||
assert_eq!(config.output_dir, DEFAULT_RUNTIME_DIAL9_OUTPUT_DIR);
|
||||
assert_eq!(config.file_prefix, DEFAULT_RUNTIME_DIAL9_FILE_PREFIX);
|
||||
assert_eq!(config.max_file_size, DEFAULT_RUNTIME_DIAL9_MAX_FILE_SIZE);
|
||||
assert_eq!(config.rotation_count, DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn base_path_joins_prefix() {
|
||||
let config = Dial9Config {
|
||||
output_dir: "/tmp/telemetry".to_string(),
|
||||
file_prefix: "rustfs".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
assert_eq!(config.base_path(), PathBuf::from("/tmp/telemetry/rustfs"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn total_disk_budget_multiplies_file_size_by_rotation_count() {
|
||||
let config = Dial9Config {
|
||||
max_file_size: 100,
|
||||
rotation_count: 10,
|
||||
..Default::default()
|
||||
};
|
||||
assert_eq!(config.total_disk_budget(), 1_000);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_enabled_requires_compile_time_support() {
|
||||
// `is_enabled` is the conjunction of the two; when support is absent it
|
||||
// must stay false no matter what the environment requests.
|
||||
if !is_supported() {
|
||||
assert!(!is_enabled());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Stub implementation used when the `dial9` feature is off.
|
||||
//!
|
||||
//! Mirrors the surface of [`super::enabled`] so that callers never need a
|
||||
//! `#[cfg]`. Building a traced runtime always fails here, which drives callers
|
||||
//! down their standard-runtime fallback path.
|
||||
|
||||
use crate::TelemetryError;
|
||||
|
||||
/// Placeholder for `dial9_tokio_telemetry::telemetry::TelemetryGuard`.
|
||||
///
|
||||
/// Never constructed: [`build_traced_runtime`] always fails without the
|
||||
/// `dial9` feature.
|
||||
#[derive(Debug)]
|
||||
pub struct TelemetryGuard(());
|
||||
|
||||
/// Placeholder session guard. Never constructed without the `dial9` feature.
|
||||
#[derive(Debug)]
|
||||
pub struct Dial9SessionGuard(());
|
||||
|
||||
impl Dial9SessionGuard {
|
||||
/// Always false: telemetry support is not compiled in.
|
||||
pub fn is_active(&self) -> bool {
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
/// Always fails: telemetry support is not compiled in.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Always returns [`TelemetryError::Io`] describing the missing feature.
|
||||
pub fn build_traced_runtime(
|
||||
_builder: tokio::runtime::Builder,
|
||||
) -> Result<(tokio::runtime::Runtime, Dial9SessionGuard), TelemetryError> {
|
||||
Err(TelemetryError::Io(
|
||||
"dial9 telemetry support is not compiled in; rebuild with --features dial9".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn build_traced_runtime_always_fails_without_the_feature() {
|
||||
let result = build_traced_runtime(tokio::runtime::Builder::new_current_thread());
|
||||
assert!(result.is_err());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,236 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! `dial9-tokio-telemetry` integration, compiled when the `dial9` feature is on.
|
||||
//!
|
||||
//! Captures Tokio runtime-level events (poll start/end, worker park/unpark,
|
||||
//! task spawn/terminate) into rotating binary trace segments. This is an
|
||||
//! on-demand profiler rather than always-on telemetry: it is disabled by
|
||||
//! default and requires a `--cfg tokio_unstable` build.
|
||||
|
||||
use super::config::Dial9Config;
|
||||
use super::state::{dial9_runtime_state, measure_disk_usage_bytes};
|
||||
use super::{EVENT_DIAL9_STATE, LOG_COMPONENT_OBS, LOG_SUBSYSTEM_DIAL9};
|
||||
use crate::TelemetryError;
|
||||
use dial9_tokio_telemetry::telemetry::{ProcessResourceUsageConfig, RotatingWriter, TaskDumpConfig, TracedRuntime};
|
||||
use std::time::Duration;
|
||||
use tracing::{info, warn};
|
||||
|
||||
pub use dial9_tokio_telemetry::telemetry::TelemetryGuard;
|
||||
|
||||
/// How often the background refresher restates trace-file disk usage.
|
||||
const DISK_USAGE_REFRESH_INTERVAL: Duration = Duration::from_secs(60);
|
||||
|
||||
/// Name recorded in segment metadata so the trace viewer can label workers.
|
||||
const RUNTIME_NAME: &str = "rustfs-worker";
|
||||
|
||||
/// Owns a live dial9 telemetry session.
|
||||
///
|
||||
/// Dropping this guard flushes buffered events and seals the active segment.
|
||||
/// It must be dropped before the process exits: a guard parked in a `static` is
|
||||
/// never dropped, and the final buffered events — usually the interesting ones —
|
||||
/// are lost.
|
||||
pub struct Dial9SessionGuard {
|
||||
guard: TelemetryGuard,
|
||||
config: Dial9Config,
|
||||
}
|
||||
|
||||
impl Dial9SessionGuard {
|
||||
/// Whether the underlying telemetry session is recording.
|
||||
pub fn is_active(&self) -> bool {
|
||||
self.guard.is_enabled()
|
||||
}
|
||||
|
||||
/// The configuration this session was built from.
|
||||
pub fn config(&self) -> &Dial9Config {
|
||||
&self.config
|
||||
}
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for Dial9SessionGuard {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("Dial9SessionGuard")
|
||||
.field("active", &self.is_active())
|
||||
.field("output_dir", &self.config.output_dir)
|
||||
.finish()
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for Dial9SessionGuard {
|
||||
fn drop(&mut self) {
|
||||
dial9_runtime_state().record_runtime_stopped();
|
||||
info!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
state = "flushed",
|
||||
"dial9 state changed"
|
||||
);
|
||||
// `TelemetryGuard`'s own `Drop` flushes buffered events and seals the
|
||||
// active segment; it runs immediately after this body.
|
||||
}
|
||||
}
|
||||
|
||||
/// Build a Tokio runtime with dial9 telemetry attached and recording started.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Returns an error when dial9 is not enabled, when the output directory cannot
|
||||
/// be created, or when the traced runtime fails to build. Callers are expected
|
||||
/// to fall back to a standard runtime.
|
||||
pub fn build_traced_runtime(
|
||||
builder: tokio::runtime::Builder,
|
||||
) -> Result<(tokio::runtime::Runtime, Dial9SessionGuard), TelemetryError> {
|
||||
let config = Dial9Config::from_env();
|
||||
if !config.enabled {
|
||||
return Err(TelemetryError::Io("dial9 telemetry is not enabled".to_string()));
|
||||
}
|
||||
|
||||
std::fs::create_dir_all(&config.output_dir).map_err(|e| {
|
||||
dial9_runtime_state().record_runtime_error(&config);
|
||||
TelemetryError::Io(format!("Failed to create dial9 output directory '{}': {e}", config.output_dir))
|
||||
})?;
|
||||
|
||||
let writer = RotatingWriter::new(config.base_path(), config.max_file_size, config.total_disk_budget()).map_err(|e| {
|
||||
dial9_runtime_state().record_runtime_error(&config);
|
||||
TelemetryError::Io(format!("Failed to create dial9 RotatingWriter: {e}"))
|
||||
})?;
|
||||
|
||||
// `with_trace_path` transitions the builder into the state that spawns the
|
||||
// background worker, which drives the segment pipeline.
|
||||
let traced = TracedRuntime::builder()
|
||||
.with_trace_path(&config.output_dir)
|
||||
.with_task_tracking(true)
|
||||
.with_runtime_name(RUNTIME_NAME)
|
||||
.with_process_resource_usage(ProcessResourceUsageConfig::default());
|
||||
|
||||
let traced = if config.task_dump_enabled {
|
||||
traced.with_task_dumps(
|
||||
TaskDumpConfig::builder()
|
||||
.idle_threshold(config.task_dump_idle_threshold)
|
||||
.build(),
|
||||
)
|
||||
} else {
|
||||
traced
|
||||
};
|
||||
|
||||
// `build_and_start` rather than `build`: `build` returns a live guard that
|
||||
// never records, writing segments that contain only a header.
|
||||
//
|
||||
// No `with_s3_uploader` here: dial9's `worker-s3` feature carries a
|
||||
// vulnerable TLS stack. See the note in `crates/obs/Cargo.toml`.
|
||||
finish_traced_runtime(traced.build_and_start(builder, writer), config)
|
||||
}
|
||||
|
||||
/// Publish the outcome of a traced-runtime build and start the background
|
||||
/// disk-usage refresher.
|
||||
fn finish_traced_runtime(
|
||||
started: std::io::Result<(tokio::runtime::Runtime, TelemetryGuard)>,
|
||||
config: Dial9Config,
|
||||
) -> Result<(tokio::runtime::Runtime, Dial9SessionGuard), TelemetryError> {
|
||||
let (runtime, guard) = started.map_err(|e| {
|
||||
dial9_runtime_state().record_runtime_error(&config);
|
||||
TelemetryError::Io(format!("Failed to build dial9 TracedRuntime: {e}"))
|
||||
})?;
|
||||
|
||||
// `is_enabled` distinguishes a live guard from the inert one a lenient
|
||||
// config produces after a build failure. It does NOT mean recording has
|
||||
// started — a guard from `build` (rather than `build_and_start`) reports
|
||||
// `true` while writing segments that contain only a header. Recording is
|
||||
// guaranteed by the `build_and_start` call above, not by this check.
|
||||
if !guard.is_enabled() {
|
||||
dial9_runtime_state().record_runtime_error(&config);
|
||||
return Err(TelemetryError::Io("dial9 TracedRuntime built with telemetry disabled".to_string()));
|
||||
}
|
||||
|
||||
dial9_runtime_state().record_runtime_started(&config);
|
||||
runtime.spawn(refresh_disk_usage_loop());
|
||||
|
||||
info!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
state = "recording",
|
||||
output_dir = %config.output_dir,
|
||||
file_prefix = %config.file_prefix,
|
||||
disk_budget_bytes = config.total_disk_budget(),
|
||||
task_dumps = config.task_dump_enabled,
|
||||
"dial9 state changed"
|
||||
);
|
||||
|
||||
Ok((runtime, Dial9SessionGuard { guard, config }))
|
||||
}
|
||||
|
||||
/// Periodically restate trace-file disk usage so the metrics collector can read
|
||||
/// it from an atomic instead of walking the directory on its own thread.
|
||||
async fn refresh_disk_usage_loop() {
|
||||
let mut ticker = tokio::time::interval(DISK_USAGE_REFRESH_INTERVAL);
|
||||
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
|
||||
|
||||
loop {
|
||||
ticker.tick().await;
|
||||
|
||||
let Some(location) = dial9_runtime_state().trace_location() else {
|
||||
continue;
|
||||
};
|
||||
|
||||
// `measure_disk_usage_bytes` walks a directory and stats every entry,
|
||||
// so it must stay off the async workers.
|
||||
let measured =
|
||||
tokio::task::spawn_blocking(move || measure_disk_usage_bytes(&location.output_dir, &location.file_prefix)).await;
|
||||
|
||||
match measured {
|
||||
Ok(bytes) => dial9_runtime_state().set_disk_usage_bytes(bytes),
|
||||
Err(e) => warn!(
|
||||
event = EVENT_DIAL9_STATE,
|
||||
component = LOG_COMPONENT_OBS,
|
||||
subsystem = LOG_SUBSYSTEM_DIAL9,
|
||||
result = "disk_usage_refresh_failed",
|
||||
error = %e,
|
||||
"dial9 state changed"
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::super::state::{reset_for_test, runtime_stats_snapshot};
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn build_traced_runtime_refuses_when_disabled() {
|
||||
reset_for_test();
|
||||
// dial9 is not configured in the unit-test environment, so
|
||||
// `Dial9Config::from_env()` yields a disabled config.
|
||||
let result = build_traced_runtime(tokio::runtime::Builder::new_current_thread());
|
||||
assert!(result.is_err(), "disabled dial9 must not build a traced runtime");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recording_session_reports_active_until_stopped() {
|
||||
reset_for_test();
|
||||
let config = Dial9Config {
|
||||
enabled: true,
|
||||
..Dial9Config::default()
|
||||
};
|
||||
|
||||
dial9_runtime_state().record_runtime_started(&config);
|
||||
assert_eq!(runtime_stats_snapshot().active_sessions, 1);
|
||||
|
||||
// Mirrors what `Dial9SessionGuard::drop` publishes.
|
||||
dial9_runtime_state().record_runtime_stopped();
|
||||
assert_eq!(runtime_stats_snapshot().active_sessions, 0);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Tokio runtime-level telemetry via `dial9-tokio-telemetry`.
|
||||
//!
|
||||
//! This is an on-demand profiler for executor-level faults — long polls that
|
||||
//! stall a worker, park/unpark storms, tasks that never yield — which are
|
||||
//! invisible to request-level metrics and spans.
|
||||
//!
|
||||
//! # Compile-time support
|
||||
//!
|
||||
//! Telemetry requires the `dial9` cargo feature *and* a `--cfg tokio_unstable`
|
||||
//! build; `build.rs` enforces that pairing. Without the feature this module
|
||||
//! still exposes its full API, backed by [`disabled`] stubs, so no caller needs
|
||||
//! a `#[cfg]`. Use [`is_supported`] to test compile-time support,
|
||||
//! [`is_configured`] to test operator intent, and [`is_enabled`] for the
|
||||
//! conjunction that decides whether telemetry actually runs.
|
||||
//!
|
||||
//! # Cost
|
||||
//!
|
||||
//! Trace segments are written to disk continuously and evicted oldest-first
|
||||
//! once `RUSTFS_RUNTIME_DIAL9_MAX_FILE_SIZE * RUSTFS_RUNTIME_DIAL9_ROTATION_COUNT`
|
||||
//! bytes are retained. Under a high poll rate the budget can wrap in minutes,
|
||||
//! so size it against the window you need to capture.
|
||||
//!
|
||||
//! # Known observability gap
|
||||
//!
|
||||
//! `dial9`'s `RotatingWriter` stops accepting writes (its internal `Finished`
|
||||
//! state) when the output directory disappears or a segment cannot be sealed,
|
||||
//! and it exposes no way to observe that from outside. `TelemetryGuard::is_enabled`
|
||||
//! reports how the session was *built*, not whether it is still writing. There
|
||||
//! is therefore no `writer_healthy` metric: it could only ever be hard-coded to
|
||||
//! `1`. Watch `rustfs_dial9_disk_usage_bytes` — a session that is recording but
|
||||
//! whose disk usage stops growing has most likely hit this state.
|
||||
|
||||
mod config;
|
||||
mod state;
|
||||
|
||||
#[cfg(not(feature = "dial9"))]
|
||||
mod disabled;
|
||||
#[cfg(feature = "dial9")]
|
||||
mod enabled;
|
||||
|
||||
pub(super) const LOG_COMPONENT_OBS: &str = "obs";
|
||||
pub(super) const LOG_SUBSYSTEM_DIAL9: &str = "dial9";
|
||||
pub(super) const EVENT_DIAL9_STATE: &str = "dial9_state";
|
||||
|
||||
pub use config::{Dial9Config, is_configured, is_enabled, is_supported};
|
||||
pub(crate) use state::runtime_stats_snapshot;
|
||||
|
||||
#[cfg(feature = "dial9")]
|
||||
pub use enabled::{Dial9SessionGuard, TelemetryGuard, build_traced_runtime};
|
||||
|
||||
#[cfg(not(feature = "dial9"))]
|
||||
pub use disabled::{Dial9SessionGuard, TelemetryGuard, build_traced_runtime};
|
||||
@@ -0,0 +1,212 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Process-global observable state for dial9 telemetry.
|
||||
//!
|
||||
//! Every field is an atomic so that [`runtime_stats_snapshot`] is allocation
|
||||
//! free and performs no I/O: it runs on the metrics collection path, which must
|
||||
//! never block. Disk usage is sampled by a background refresher (see
|
||||
//! `enabled::refresh_disk_usage_loop`) and published here.
|
||||
|
||||
use super::config::Dial9Config;
|
||||
#[cfg(any(feature = "dial9", test))]
|
||||
use std::path::Path;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::OnceLock;
|
||||
use std::sync::RwLock;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
/// Point-in-time view of dial9 runtime state.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub(crate) struct Dial9RuntimeSnapshot {
|
||||
/// 1 while a traced runtime is installed, 0 otherwise.
|
||||
pub active_sessions: u64,
|
||||
/// Cumulative count of telemetry setup failures.
|
||||
pub errors_total: u64,
|
||||
/// Bytes on disk for this trace family, as of the last refresh.
|
||||
pub disk_usage_bytes: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub(super) struct Dial9RuntimeState {
|
||||
active_sessions: AtomicU64,
|
||||
errors_total: AtomicU64,
|
||||
disk_usage_bytes: AtomicU64,
|
||||
trace_dir: RwLock<Option<TraceLocation>>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub(super) struct TraceLocation {
|
||||
#[cfg_attr(not(feature = "dial9"), allow(dead_code))]
|
||||
pub output_dir: PathBuf,
|
||||
#[cfg_attr(not(feature = "dial9"), allow(dead_code))]
|
||||
pub file_prefix: String,
|
||||
}
|
||||
|
||||
impl Dial9RuntimeState {
|
||||
fn new() -> Self {
|
||||
Self {
|
||||
active_sessions: AtomicU64::new(0),
|
||||
errors_total: AtomicU64::new(0),
|
||||
disk_usage_bytes: AtomicU64::new(0),
|
||||
trace_dir: RwLock::new(None),
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn record_config(&self, config: &Dial9Config) {
|
||||
*self.trace_dir.write().expect("dial9 trace_dir lock should not be poisoned") = Some(TraceLocation {
|
||||
output_dir: PathBuf::from(&config.output_dir),
|
||||
file_prefix: config.file_prefix.clone(),
|
||||
});
|
||||
if !config.enabled {
|
||||
self.active_sessions.store(0, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(any(feature = "dial9", test))]
|
||||
pub(super) fn record_runtime_started(&self, config: &Dial9Config) {
|
||||
self.record_config(config);
|
||||
self.active_sessions.store(1, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
#[cfg(any(feature = "dial9", test))]
|
||||
pub(super) fn record_runtime_error(&self, config: &Dial9Config) {
|
||||
self.record_config(config);
|
||||
self.active_sessions.store(0, Ordering::Relaxed);
|
||||
self.errors_total.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
#[cfg(feature = "dial9")]
|
||||
pub(super) fn record_runtime_stopped(&self) {
|
||||
self.active_sessions.store(0, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
#[cfg(any(feature = "dial9", test))]
|
||||
pub(super) fn set_disk_usage_bytes(&self, bytes: u64) {
|
||||
self.disk_usage_bytes.store(bytes, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
#[cfg(feature = "dial9")]
|
||||
pub(super) fn trace_location(&self) -> Option<TraceLocation> {
|
||||
self.trace_dir
|
||||
.read()
|
||||
.expect("dial9 trace_dir lock should not be poisoned")
|
||||
.clone()
|
||||
}
|
||||
|
||||
fn snapshot(&self) -> Dial9RuntimeSnapshot {
|
||||
Dial9RuntimeSnapshot {
|
||||
active_sessions: self.active_sessions.load(Ordering::Relaxed),
|
||||
errors_total: self.errors_total.load(Ordering::Relaxed),
|
||||
disk_usage_bytes: self.disk_usage_bytes.load(Ordering::Relaxed),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn dial9_runtime_state() -> &'static Dial9RuntimeState {
|
||||
static STATE: OnceLock<Dial9RuntimeState> = OnceLock::new();
|
||||
STATE.get_or_init(Dial9RuntimeState::new)
|
||||
}
|
||||
|
||||
/// Reset the process-global state. Tests share one process, so any test that
|
||||
/// mutates the state must call this first.
|
||||
#[cfg(test)]
|
||||
pub(super) fn reset_for_test() {
|
||||
let state = dial9_runtime_state();
|
||||
state.active_sessions.store(0, Ordering::Relaxed);
|
||||
state.errors_total.store(0, Ordering::Relaxed);
|
||||
state.disk_usage_bytes.store(0, Ordering::Relaxed);
|
||||
*state.trace_dir.write().expect("dial9 trace_dir lock should not be poisoned") = None;
|
||||
}
|
||||
|
||||
/// Sum the on-disk size of every segment belonging to this trace family.
|
||||
///
|
||||
/// This walks a directory and stats each entry, so it must never run on an
|
||||
/// async worker or on the metrics collection path. Callers are responsible for
|
||||
/// scheduling it on a blocking thread.
|
||||
#[cfg(any(feature = "dial9", test))]
|
||||
pub(super) fn measure_disk_usage_bytes(output_dir: &Path, file_prefix: &str) -> u64 {
|
||||
let Ok(entries) = std::fs::read_dir(output_dir) else {
|
||||
return 0;
|
||||
};
|
||||
|
||||
entries
|
||||
.filter_map(Result::ok)
|
||||
.filter(|entry| {
|
||||
entry
|
||||
.file_name()
|
||||
.to_str()
|
||||
.is_some_and(|filename| filename.starts_with(file_prefix))
|
||||
})
|
||||
.filter_map(|entry| entry.metadata().ok())
|
||||
.map(|meta| meta.len())
|
||||
.sum()
|
||||
}
|
||||
|
||||
/// Snapshot dial9 runtime state. Lock-free apart from a read lock on the trace
|
||||
/// location, and performs no I/O.
|
||||
pub(crate) fn runtime_stats_snapshot() -> Dial9RuntimeSnapshot {
|
||||
dial9_runtime_state().snapshot()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use tempfile::tempdir;
|
||||
|
||||
#[test]
|
||||
fn measure_disk_usage_sums_only_matching_prefix() {
|
||||
let dir = tempdir().expect("create temp dir");
|
||||
std::fs::write(dir.path().join("rustfs-tokio.0.bin"), vec![0_u8; 128]).expect("write segment");
|
||||
std::fs::write(dir.path().join("rustfs-tokio.1.bin"), vec![0_u8; 64]).expect("write segment");
|
||||
std::fs::write(dir.path().join("unrelated.log"), vec![0_u8; 4096]).expect("write unrelated");
|
||||
|
||||
assert_eq!(measure_disk_usage_bytes(dir.path(), "rustfs-tokio"), 192);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn measure_disk_usage_of_missing_dir_is_zero() {
|
||||
assert_eq!(measure_disk_usage_bytes(Path::new("/nonexistent/rustfs/dial9"), "rustfs-tokio"), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn snapshot_reports_cached_disk_usage_without_touching_disk() {
|
||||
reset_for_test();
|
||||
let config = Dial9Config {
|
||||
enabled: true,
|
||||
..Dial9Config::default()
|
||||
};
|
||||
dial9_runtime_state().record_runtime_started(&config);
|
||||
dial9_runtime_state().set_disk_usage_bytes(4096);
|
||||
|
||||
let snapshot = runtime_stats_snapshot();
|
||||
assert_eq!(snapshot.active_sessions, 1);
|
||||
assert_eq!(snapshot.disk_usage_bytes, 4096);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_error_clears_session_and_counts_the_failure() {
|
||||
reset_for_test();
|
||||
let config = Dial9Config {
|
||||
enabled: true,
|
||||
..Dial9Config::default()
|
||||
};
|
||||
dial9_runtime_state().record_runtime_started(&config);
|
||||
dial9_runtime_state().record_runtime_error(&config);
|
||||
|
||||
let snapshot = runtime_stats_snapshot();
|
||||
assert_eq!(snapshot.active_sessions, 0);
|
||||
assert_eq!(snapshot.errors_total, 1);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user