fix(obs): remove the dial9 task-dump switch that could never work; correct measured claims (#4688)

* fix(obs): remove the dial9 task-dump switch that could never do anything

Measured on a bench host (Linux x86_64) against the code merged in #4663: with
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_ENABLED=true`, a `dial9-taskdump` build, and
`--cfg tokio_taskdump`, dial9 recorded **zero** TaskDump events.

dial9 captures a task dump only for futures it wrapped itself — those spawned
through `dial9_tokio_telemetry::spawn`, which is where `TaskDumped<F>` gets
applied. `tokio::spawn` gets no wrapper, and RustFS spawns with `tokio::spawn`
throughout. Same workload, same binary, only the spawner changed:

    tokio::spawn   ->      0 dumps
    dial9::spawn   ->  14709 dumps, all with callchains

Upstream documents this (README line 151) and tracks the doc gap at
dial9-rs/dial9#477. I did not read it before wiring `with_task_dumps` in #4663,
and so shipped exactly the kind of lying configuration knob that PR set out to
delete. Remove it: the two environment variables, the config fields, the
`with_task_dumps` call, and the `dial9-taskdump` feature — whose only effect was
to constrain the build to Linux while recording nothing.

Re-adding it only makes sense together with migrating the paths under
investigation to dial9's spawner. Tracked as D9-16 in rustfs/backlog#1157.

Also drop the `--cfg tokio_taskdump` requirement from the Makefile. Measured:
dumps are captured with and without it (14709 vs 14674, within noise), and
upstream never asked for it. That requirement was mine, invented and untested.

Cargo.lock loses tokio's `backtrace` dependency, which `tokio/taskdump` pulled in.

Co-Authored-By: heihutu <heihutu@gmail.com>

* docs(obs): replace guessed dial9 retention numbers with measured ones

Three corrections, all to claims I wrote in #4663 without measuring them.

"Under a high poll rate that budget can wrap in minutes" was a guess. Measured on
a single-node 4-drive cluster under warp mixed (66 MiB/s, 110 obj/s, 32 concurrent):
13023 events/s, 0.16 MiB/s, so the default 1 GiB budget wraps after roughly 108
minutes. Even at ten times the throughput that is ~11 minutes. State the measured
rate and how to scale it instead.

dial9 was described as the tool for drive stalls. It is not. RustFS does disk I/O
on the blocking pool and through io_uring, never on an async worker, so a slow
drive never lengthens a poll. Injecting 200 ms of latency on one of four drives
cut throughput by 64% and left the poll distribution unchanged (polls >= 5 ms:
49 -> 56; p999: 2.67 ms -> 2.75 ms). Enabling dial9's CPU and sched profilers
does not help: sched events are per-worker only, and the CPU profiler samples
on-CPU while a stalled drive is an off-CPU wait. Say so plainly, and point at
the `rustfs_io_*` metrics instead.

What dial9 *is* good for, on the same traces: single polls of 418-625 ms with no
fault injected at all — real worker stalls nothing else in the obs stack surfaces.
Lead with that.

Also link the two upstream issues filed for the gaps we documented:
dial9-rs/dial9#658 (writer death unobservable) and #659 (worker-s3 CVEs).

Measurements: rustfs/backlog#1157 (D9-11, D9-13, D9-18).

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>

scripts/run.sh

@@ -116,9 +116,13 @@ export RUSTFS_RUNTIME_GLOBAL_QUEUE_INTERVAL=31
 # dial9 Tokio Runtime Telemetry Configuration
 # ============================================================================
 # dial9 captures Tokio runtime-level events (poll start/end, worker park/unpark,
-# task spawn/terminate) into binary trace segments. It sees executor-level faults
-# — a long poll stalling a worker, a task that never yields — that request-level
-# metrics and spans cannot.
+# task spawn/terminate) into binary trace segments. It sees long polls that stall
+# a worker — something request-level metrics and spans cannot show.
+#
+# It does NOT see a drive stall: RustFS does its disk I/O on the blocking pool
+# and via io_uring, never on an async worker, so a slow drive does not lengthen
+# any poll. Measured: injecting 200ms of drive latency cut throughput by 64% and
+# left the poll-duration distribution unchanged.
 #
 # This is an on-demand profiler, not always-on telemetry:
 #
@@ -126,8 +130,9 @@ export RUSTFS_RUNTIME_GLOBAL_QUEUE_INTERVAL=31
 #   which enables the `dial9` feature and `--cfg tokio_unstable`. Setting the
 #   variables below on a stock binary logs a warning and changes nothing.
 # - Trace segments are written continuously, and the oldest are deleted once
-#   MAX_FILE_SIZE * ROTATION_COUNT bytes are retained. Under a high poll rate
-#   that budget can wrap in minutes — size it against the window you need.
+#   MAX_FILE_SIZE * ROTATION_COUNT bytes are retained. Measured at ~0.16 MiB/s
+#   under a 66 MiB/s warp workload, so the default 1 GiB budget wraps after
+#   roughly 108 minutes. Scale that by your event rate.
 # - Turn it off again when the investigation is over.
 #
 # See docs/operations/dial9-runtime-profiling.md.
@@ -148,12 +153,6 @@ export RUSTFS_RUNTIME_GLOBAL_QUEUE_INTERVAL=31
 # MAX_FILE_SIZE * ROTATION_COUNT; older segments are evicted, not kept.
 #export RUSTFS_RUNTIME_DIAL9_ROTATION_COUNT=10
 
-# Capture async backtraces of tasks that stall (default: false).
-# Needs a binary built with the `dial9-taskdump` feature (Linux only).
-#export RUSTFS_RUNTIME_DIAL9_TASK_DUMP_ENABLED=true
-# Mean idle duration for task-dump Poisson sampling, in ms (default: 10)
-#export RUSTFS_RUNTIME_DIAL9_TASK_DUMP_IDLE_THRESHOLD_MS=10
-
 # S3 upload is NOT available: dial9's uploader depends on a rustls-webpki with
 # known CVEs, so the feature is not built. These are parsed and warned about,
 # never honoured. Collect segments from OUTPUT_DIR instead.
@@ -165,10 +164,10 @@ export RUSTFS_RUNTIME_GLOBAL_QUEUE_INTERVAL=31
 #export RUSTFS_RUNTIME_DIAL9_OUTPUT_DIR="$current_dir/deploy/telemetry"
 
 # --- Scenario 2: investigating a worker stall ---
-# Task dumps show where stalled tasks are parked. Keep the run short.
+# Trace shows which task held a worker and for how long, but not where it was
+# stuck: task dumps need dial9's own spawner (see backlog#1157 D9-16).
 #export RUSTFS_RUNTIME_DIAL9_ENABLED=true
 #export RUSTFS_RUNTIME_DIAL9_OUTPUT_DIR=/tmp/rustfs-telemetry-investigation
-#export RUSTFS_RUNTIME_DIAL9_TASK_DUMP_ENABLED=true
 #export RUSTFS_RUNTIME_DIAL9_ROTATION_COUNT=3
 
 export OTEL_INSTRUMENTATION_NAME="rustfs"