mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-31 10:32:24 +00:00
Zhengchao An
e2257325a2
peer_rest_recovery_probe_logs_keep_request_id_span_context failed ~10% of `cargo test -p rustfs-ecstore --lib -- cluster::rpc::` runs with left: "request-span", right: "recovery-monitor". The recovery-monitor info_span! was evaluating to Span::none(), so the probe's log line landed under the caller's span. tracing caches each callsite's Interest in process-global state, and the first thread to reach a callsite fixes that value. While at most one dispatcher is registered, tracing-core takes a fast path that derives the interest from the registering thread's own subscriber, and registration is once-only (CAS). Under libtest a sibling test reaches recovery_monitor_span via mark_offline_and_spawn_recovery from a thread with no subscriber, so the interest is derived from NoSubscriber and cached as Interest::never() for the whole process. Add pin_callsite_interest_for_test(): registering a second, inert dispatcher rebuilds every registered callsite's interest against the live dispatcher set (repairing a poisoned value) and keeps tracing-core off the single-dispatcher fast path (preventing new ones). This also covers the production marked_suspect / recovery_monitor_started event callsites that remote_disk_network_error_starts_recovery_monitor_with_request_context asserts on. rename_data_response_accepts_legacy_json_without_decode_error is a separate root cause: it snapshots the process-global internode metrics and asserts the decode-error counter did not move, which siblings that record decode errors (or reset the counters) invalidate. Put the 11 tests that observe those counters in one #[serial(internode_metrics)] group. Both races are impossible under nextest, which runs each test in its own process, so neither test belongs in the ecstore-serial-flaky test-group (that serializes across process boundaries) nor in the ci-profile quarantine (they never redden CI). Verified: cluster::rpc:: subset 0/30 failures under libtest (was 3/30); target test paired with its poisoner 0/30 (was 4/20); 5/5 clean under nextest at 179/179.