#4663 added a full S3 upload path (config fields + env parsing + wiring), then
reverted the wiring when its dependency turned out to carry RUSTSEC advisories
(D9-14). The revert stopped at enabled.rs and left the config layer half torn
down: `Dial9Config` still carried `s3_bucket`/`s3_prefix` with no consumer, plus
a dead `s3_upload_requested()`, and `Dial9SessionGuard::config()` was likewise
never called. All three are zero-consumer dead code.
Since S3 upload cannot be built at all, the config type should not model it.
Drop the two fields and both accessors. `from_env` still reads the S3 env vars
and warns about them (an operator who set them deserves to know why they do
nothing — that warning is deliberate and stays), but drops the values instead
of storing configuration nothing reads.
While here, collapse `from_env`'s two `record_config` calls into one by
extracting `from_env_enabled`, so the enabled/disabled paths share a single
publish point.
No behavior change: the S3 warning fires identically, and every other field is
untouched. Net -13 lines, and `Dial9Config` drops from 8 fields to 6.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(obs): remove the dial9 task-dump switch that could never do anything
Measured on a bench host (Linux x86_64) against the code merged in #4663: with
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_ENABLED=true`, a `dial9-taskdump` build, and
`--cfg tokio_taskdump`, dial9 recorded **zero** TaskDump events.
dial9 captures a task dump only for futures it wrapped itself — those spawned
through `dial9_tokio_telemetry::spawn`, which is where `TaskDumped<F>` gets
applied. `tokio::spawn` gets no wrapper, and RustFS spawns with `tokio::spawn`
throughout. Same workload, same binary, only the spawner changed:
tokio::spawn -> 0 dumps
dial9::spawn -> 14709 dumps, all with callchains
Upstream documents this (README line 151) and tracks the doc gap at
dial9-rs/dial9#477. I did not read it before wiring `with_task_dumps` in #4663,
and so shipped exactly the kind of lying configuration knob that PR set out to
delete. Remove it: the two environment variables, the config fields, the
`with_task_dumps` call, and the `dial9-taskdump` feature — whose only effect was
to constrain the build to Linux while recording nothing.
Re-adding it only makes sense together with migrating the paths under
investigation to dial9's spawner. Tracked as D9-16 in rustfs/backlog#1157.
Also drop the `--cfg tokio_taskdump` requirement from the Makefile. Measured:
dumps are captured with and without it (14709 vs 14674, within noise), and
upstream never asked for it. That requirement was mine, invented and untested.
Cargo.lock loses tokio's `backtrace` dependency, which `tokio/taskdump` pulled in.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): replace guessed dial9 retention numbers with measured ones
Three corrections, all to claims I wrote in #4663 without measuring them.
"Under a high poll rate that budget can wrap in minutes" was a guess. Measured on
a single-node 4-drive cluster under warp mixed (66 MiB/s, 110 obj/s, 32 concurrent):
13023 events/s, 0.16 MiB/s, so the default 1 GiB budget wraps after roughly 108
minutes. Even at ten times the throughput that is ~11 minutes. State the measured
rate and how to scale it instead.
dial9 was described as the tool for drive stalls. It is not. RustFS does disk I/O
on the blocking pool and through io_uring, never on an async worker, so a slow
drive never lengthens a poll. Injecting 200 ms of latency on one of four drives
cut throughput by 64% and left the poll distribution unchanged (polls >= 5 ms:
49 -> 56; p999: 2.67 ms -> 2.75 ms). Enabling dial9's CPU and sched profilers
does not help: sched events are per-worker only, and the CPU profiler samples
on-CPU while a stalled drive is an off-CPU wait. Say so plainly, and point at
the `rustfs_io_*` metrics instead.
What dial9 *is* good for, on the same traces: single polls of 418-625 ms with no
fault injected at all — real worker stalls nothing else in the obs stack surfaces.
Lead with that.
Also link the two upstream issues filed for the gaps we documented:
dial9-rs/dial9#658 (writer death unobservable) and #659 (worker-s3 CVEs).
Measurements: rustfs/backlog#1157 (D9-11, D9-13, D9-18).
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): make body-cache hook re-registrable + add e2e regressions
ODC-21 (backlog#1126): the GET body-cache hook lived in a first-wins
OnceLock. When AppContext is rebuilt (config reload, test re-init) a fresh
ObjectDataCacheAdapter is constructed and re-registered, but the OnceLock
kept ecstore's GET probe pointed at adapter #1 while every usecase-layer
fill and invalidation targeted adapter #2 — silently degrading the feature
to a 0% hit rate with no error, log, or metric, and stranding entries in the
unreachable cache until their TTL.
Replace the slot with RwLock<Option<Arc<dyn GetObjectBodyCacheHook>>> so
re-registration atomically swaps to the newest adapter, and log at WARN when
a swap replaces a *different* instance (Arc::ptr_eq). RwLock over
ArcSwapOption because arc-swap's RefCnt is impl<T> (Sized, thin *mut T) and
cannot hold an Arc<dyn Trait> without a sized newtype wrapper; the probe
reads the slot once per full-object GET but only clones an Arc, negligible
next to the metadata quorum fan-out already done before the probe. Add a
test-only clear_get_object_body_cache_hook so tests register/unregister
deterministically.
With the hook now re-registrable, add true end-to-end regressions that drive
get_object_reader (not the full_object_plaintext_len predicate) against a
real erasure-coded, genuinely-compressed object via the blackbox
make_local_set_disks harness, with a stand-in hook playing the app-layer
cache (the injection point production uses; the adapter itself lives above
ecstore). These close the gap the predicate-only tests left — a caller that
opens a new shortcut serving the cached body directly, the original form of
both P0s:
- backlog#1108: a raw_data_movement_read must yield the STORED (compressed)
bytes, never the cached plaintext.
- backlog#1109: a compressed cache hit must publish the DECOMPRESSED length
as object_info.size (the UploadPartCopy invariant), with the streamed
length matching.
- backlog#1146: a restore read (restore_request.days) must serve STORED
bytes, not the cache.
Mutation-verified each e2e test bites: dropping the raw_data_movement_read
gate serves plaintext (fails #1108); removing the hit-site size republication
publishes 2972 vs 660000 (fails #1109); dropping the restore gate serves
plaintext (fails #1146).
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(obs): make dial9 telemetry opt-in and actually record events
The dial9 Tokio-runtime profiler was disabled by default, yet every build
paid for it, and enabling it produced trace files with no events in them.
Recorded empty traces
---------------------
`build_traced_runtime` called `TracedRuntime::builder()...build(..)`, but dial9
only starts recording in `build_and_start*`. `build` still returns a live guard
whose `is_enabled()` reports true, and still creates and seals segment files —
they just contain a header and no events. It also skipped `with_trace_path`, so
the background worker driving the segment pipeline was never spawned.
Measured on the new smoke example: 310 bytes of bare segment header, against
5640 bytes for the same workload once recording actually starts.
Switch to `with_trace_path(..).build_and_start(..)`.
Cost was unconditional
----------------------
`--cfg tokio_unstable` was a global `[build] rustflags` entry and `rustfs-obs`
depended on `dial9-tokio-telemetry` unconditionally, so all builds depended on
Tokio's non-semver API. Worse, an environment `RUSTFLAGS` replaces (never
appends to) the config-file value, so any caller exporting their own RUSTFLAGS
silently dropped the flag — the long comment in build.yml was a scar from that.
dial9 is now an opt-in feature (`dial9`, plus `dial9-s3` and `dial9-taskdump`),
the global rustflag is gone, and `crates/obs/build.rs` fails the compile if the
feature is on without the flag. Telemetry builds go through `make build-profiling`.
Metrics that could not lie
--------------------------
`rustfs_dial9_{events_total,bytes_written_total,rotations_total,cpu_overhead_percent}`
were hard-coded to zero — a Counter pinned at 0 reads as "nothing happened".
Removed. `rustfs_dial9_enabled` was sourced from the environment, so it read 1
even when the traced runtime failed and the process fell back to a standard
runtime; it is replaced by `rustfs_dial9_supported` (compile-time),
`rustfs_dial9_configured` (intent) and `rustfs_dial9_active_sessions` (reality).
No `writer_healthy` gauge is exported: dial9's `RotatingWriter` can enter its
`Finished` state and stop writing, but exposes no way to observe that, so the
gauge could only ever be hard-coded to 1. Documented as a known gap instead.
Final events were lost
----------------------
The `TelemetryGuard` lived in a `static OnceLock`, which is never dropped, so
buffered events were never flushed at exit. `build_tokio_runtime` now returns
the guard and `run_process` drops it before any exit path.
Also
----
- `disk_usage_bytes` was a `read_dir` + per-file `stat` on the metrics
collection path. It is now sampled by a background task into an atomic.
- `SAMPLING_RATE`/`S3_BUCKET`/`S3_PREFIX` were parsed, warned about, and
discarded. S3 upload is now wired to dial9's `with_s3_uploader` behind
`dial9-s3`; `SAMPLING_RATE` has no upstream equivalent and is removed.
- Wire `with_task_dumps` (async backtraces of stalled tasks), configurable via
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_{ENABLED,IDLE_THRESHOLD_MS}`.
- Split `telemetry/dial9.rs` into `config`/`state`/`enabled`/`disabled`; the
stub keeps the public API identical so callers need no `#[cfg]`.
- Drop four print-only examples and the manual test bin that exercised the
removed `init_session` scaffolding.
Verified: cargo check/clippy/test across default, `dial9`, and `dial9-s3`;
build.rs correctly rejects `dial9` without `--cfg tokio_unstable`;
`make pre-commit` passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): document dial9 as an on-demand profiler
scripts/run.sh advertised a `SAMPLING_RATE` knob that was never passed to dial9,
and claimed "CPU overhead < 5% (with sampling rate 1.0)" and "lower values reduce
CPU overhead" on the strength of it. The knob is gone; the guidance built on it
had to go too.
Replace it with what is actually true: dial9 needs a `make build-profiling`
binary, its disk budget evicts oldest-first (so a high poll rate can overwrite
the incident you are chasing), and it cannot be toggled without a restart.
Add docs/operations/dial9-runtime-profiling.md covering the build variants, an
investigation walkthrough, the configuration table, how to read the three
supported/configured/active_sessions gauges against each other, and the upstream
gap that makes writer death only indirectly observable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(obs): add a dial9 smoke example that proves events are recorded
The bug this guards against is invisible to every existing signal: with
`build` instead of `build_and_start`, dial9 creates the trace file, seals
segments, and reports `TelemetryGuard::is_enabled() == true` — it simply
records no events. Only the segment's byte count tells the two apart.
Measured on this workload: 5640 bytes when recording, 310 bytes (a bare
segment header) when not. The example asserts >= 2048 bytes, and was verified
to fail with the `build` call restored.
Also correct the comment on the `is_enabled` check in `finish_traced_runtime`.
It claimed to catch "recording silently off"; it does not. It only rejects the
inert guard a lenient config yields after a build failure. Recording is
guaranteed by `build_and_start`, not by that check.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): accept Unsupported runtime telemetry capability
A binary built without the `dial9` feature now reports the runtime-telemetry
capability as `Unsupported` rather than `Disabled`. The distinction matters to
operators: `Disabled` implies the capability can be switched on by setting an
environment variable, which is not true here — telemetry needs a rebuild.
Widen the assertion and pin the new semantics: when `dial9::is_supported()` is
false, the state must be exactly `Unsupported`.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): drop the dial9-s3 feature, its TLS stack is vulnerable
CI's Dependency Review and `cargo deny` both reject the branch: dial9's
`worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3, which pins
aws-smithy-http-client onto hyper-rustls 0.24 and rustls-webpki 0.101.7. That
webpki carries RUSTSEC-2026-0098, -0099 and -0104.
0.1.3 is the latest release of the transfer manager, and 1.2.0 the latest of the
smithy client, so there is nothing to upgrade to. Cargo's feature unification can
add features but cannot drop a transitive dependency, so it cannot be worked
around from here either — the rest of the workspace already resolves to the safe
rustls-webpki 0.103 / hyper-rustls 0.27.
Remove the `dial9-s3` feature and the `with_s3_uploader` wiring. The two S3
environment variables stay parsed and warned about, now naming the real reason
rather than a missing build feature. Trace segments are collected from the output
directory instead. Tracked as D9-14 in rustfs/backlog#1157.
With this, Cargo.lock is byte-identical to main: the PR no longer touches the
dependency graph at all.
Also correct the `dial9-taskdump` documentation. It claimed the feature "compiles
to a no-op elsewhere"; in fact `tokio/taskdump` raises a `compile_error!` on any
target other than linux/{aarch64,x86,x86_64}. Verified by trying to build it on
macOS, which is how the claim was found to be wrong.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(allocator): restore validated jemalloc target gating
Restrict the global allocator and jemalloc profiling paths to linux-gnu-x86_64 so Linux ARM64 builds fall back to mimalloc again.
Update the runtime profiling, allocator reclaim, and admin profiling handlers to use the same target gating and avoid exposing jemalloc-only code on unsupported targets.
Verification:
- cargo fmt --all
- cargo fmt --all --check
- cargo check -p rustfs
- make pre-commit
* fix(profiling): restore non-jemalloc platform behavior
Restore CPU profiling support on macOS while keeping jemalloc-backed memory profiling restricted to validated linux-gnu-x86_64 targets.
Also restore Unix log directory permission hardening so the allocator regression fix does not roll back unrelated observability behavior on other supported platforms.
Verification in progress:
- cargo fmt --all
- cargo check -p rustfs
- make pre-commit (running)
* fix(profiling): qualify periodic memory log macros
Use explicit tracing macro paths in periodic jemalloc memory profiling logging so builds do not fail when the local target configuration excludes those branches from the current import set.
Verification:
- cargo fmt --all
- cargo check -p rustfs
* fix(profiling): gate unsupported memory helper
Restrict the unsupported memory profiling helper to non-linux-gnu-x86_64 targets so Linux builds do not emit dead_code warnings for an unreachable fallback.
Verification:
- cargo fmt --all
- cargo check -p rustfs
houseme