perf(io-metrics): add a global switch to gate general metric emission (#4733)

The io-metrics crate already had per-stage GET/PUT switches, but the request
headers/summaries and ~40 general recorders (I/O scheduler, bytes-pool,
zero-copy, bandwidth, system-resource, error/timeout/retry) emitted
unconditionally, paying their label allocations and arithmetic even when no
metric recorder is installed.

Add `METRICS_ENABLED: AtomicBool` (default false) with `set_metrics_enabled()`
/ `metrics_enabled()`, isomorphic to the existing stage switches, and gate:

- GET headers/summaries via `get_stage_metrics_enabled()`:
  record_get_object_request_start / _request_started / _request_result /
  _timeout / _completion / _total_duration_with_path, plus legacy
  record_get_object.
- PUT headers via `put_stage_metrics_enabled()`:
  record_put_object_request_start / _request_result, plus legacy
  record_put_object.
- 41 general recorders via `metrics_enabled()`.

Startup enables all three switches together under
`observability_metric_enabled()` (startup_observability.rs), with a passthrough
in startup_runtime_sources.rs. The global recorder is itself only installed
when observability metric export is on, so gating off is pure cost savings when
export is disabled and a no-op when it is enabled.

Deliberately NOT gated (would break function, not just observation): the EC
encode in-flight and GET buffered-bytes accounting guards
(add/remove_ec_encode_inflight_bytes, track_get_object_buffered_bytes) whose
values are read back by current_*; and the stateful sibling modules
(lock/deadlock/backpressure/autotuner/adaptive_ttl/collector/internode). The
console realtime metrics endpoint (/admin/v3/metrics) samples system state and
internode metrics directly, so it does not depend on the gated recorders.

Adds a toggle test exercising both the enabled and disabled paths, serialized
on the existing METRICS_FLAG_LOCK to stay robust under nextest's shared-process
model.

Co-authored-by: heihutu <heihutu@gmail.com>

rustfs/src/startup_observability.rs

@@ -29,6 +29,7 @@ pub(crate) async fn init_observability_runtime(store: Arc<ECStore>, ctx: Cancell
         init_compression_total_memory_from_backend(store).await;
         startup_runtime_sources::set_put_stage_metrics_enabled(true);
         startup_runtime_sources::set_get_stage_metrics_enabled(true);
+        startup_runtime_sources::set_metrics_enabled(true);
         startup_runtime_sources::init_metrics_runtime(ctx.clone());
         crate::memory_observability::init_memory_observability(ctx.clone());
         init_auto_tuner(ctx).await;

rustfs/src/startup_runtime_sources.rs

@@ -104,6 +104,10 @@ pub(crate) fn set_get_stage_metrics_enabled(enabled: bool) {
     rustfs_io_metrics::set_get_stage_metrics_enabled(enabled);
 }
 
+pub(crate) fn set_metrics_enabled(enabled: bool) {
+    rustfs_io_metrics::set_metrics_enabled(enabled);
+}
+
 pub(crate) fn init_tls_metrics() {
     rustfs_tls_runtime::init_tls_metrics();
 }