Add a replacement recovery peer RPC so Admin v4 can distinguish definitive cluster proofs from unsupported, unavailable, or conflicting peer state without extending the existing background heal v3/v1 status protocol.
Co-authored-by: heihutu <heihutu@gmail.com>
rename_all_missing_source_still_warns and rename_all_real_failure_still_warns
assert that `warn_reliable_rename_failure` emitted its WARN, but that is a
single production callsite shared with tests that call rename_all *without*
installing a subscriber — rename_all_missing_source_returns_file_not_found,
two tests above, is one of them.
tracing caches each callsite's Interest process-globally and the first thread
to reach a callsite fixes that value; while at most one dispatcher is
registered, tracing-core derives it from the registering thread's own
subscriber, and registration is once-only. When the subscriber-less sibling
wins, the callsite is cached as Interest::never() and the WARN never fires,
so the assertion sees empty output:
ordinary missing-source failures must keep the WARN, got:
Reproduced at 3/25 with `disk::os::tests::rename_all_missing_source` (both
tests), against 0/20 for the victim alone. Fixed by pinning callsite interest
inside warn_capture(), so every current and future user of that helper is
covered rather than just the two tests that happen to fail today.
pin_callsite_interest_for_test() moves from cluster::rpc::background_monitor
to a new crate-level test_tracing module: it is domain-neutral and now has
consumers in two unrelated subsystems, and disk::os should not have to reach
into a cluster::rpc test helper.
Verified: repro filter 0/30 (was 3/25); disk::os:: 0/12; cluster::rpc:: 0/12
and its poisoner pair 0/15, confirming the moved helper still holds.
Follow-up to #5438. Closes the last item in #5439.
peer_rest_recovery_probe_logs_keep_request_id_span_context failed ~10% of
`cargo test -p rustfs-ecstore --lib -- cluster::rpc::` runs with
left: "request-span", right: "recovery-monitor". The recovery-monitor
info_span! was evaluating to Span::none(), so the probe's log line landed
under the caller's span.
tracing caches each callsite's Interest in process-global state, and the
first thread to reach a callsite fixes that value. While at most one
dispatcher is registered, tracing-core takes a fast path that derives the
interest from the registering thread's own subscriber, and registration is
once-only (CAS). Under libtest a sibling test reaches recovery_monitor_span
via mark_offline_and_spawn_recovery from a thread with no subscriber, so
the interest is derived from NoSubscriber and cached as Interest::never()
for the whole process.
Add pin_callsite_interest_for_test(): registering a second, inert
dispatcher rebuilds every registered callsite's interest against the live
dispatcher set (repairing a poisoned value) and keeps tracing-core off the
single-dispatcher fast path (preventing new ones). This also covers the
production marked_suspect / recovery_monitor_started event callsites that
remote_disk_network_error_starts_recovery_monitor_with_request_context
asserts on.
rename_data_response_accepts_legacy_json_without_decode_error is a
separate root cause: it snapshots the process-global internode metrics and
asserts the decode-error counter did not move, which siblings that record
decode errors (or reset the counters) invalidate. Put the 11 tests that
observe those counters in one #[serial(internode_metrics)] group.
Both races are impossible under nextest, which runs each test in its own
process, so neither test belongs in the ecstore-serial-flaky test-group
(that serializes across process boundaries) nor in the ci-profile
quarantine (they never redden CI).
Verified: cluster::rpc:: subset 0/30 failures under libtest (was 3/30);
target test paired with its poisoner 0/30 (was 4/20); 5/5 clean under
nextest at 179/179.
Retry peer tier config reloads after committed mutations so recovered nodes converge without requiring a second admin change.
Co-authored-by: heihutu <heihutu@gmail.com>
Keep remote peer topology slots observable when peer client construction cannot build a dialing client, and publish admin server_info cache/failure state only after a complete probe round.
Covers rustfs/backlog#1426 and rustfs/backlog#1430.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(admin): use raw topology host for peer mapping
Backlog: rustfs/backlog#1427
Keep endpoint topology membership keyed by raw host:port when mapping peers to grid hosts, while preserving DNS resolution only for the PeerRestClient dialing host.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(admin): reconcile v3 info with topology
Backlog: rustfs/backlog#1425
Backlog: rustfs/backlog#1428
Synthesize additive unknown server rows from configured endpoint topology before v3 backend counters are computed, and add exact drive-identity coverage tests so balanced totals cannot hide a missing member.
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(admin): collapse topology reconciliation
Build a single endpoint topology index for v3 admin server reconciliation and completeness reporting, and reuse the raw peer/grid mapping when constructing peer clients.
Backlog: rustfs/backlog#1424
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): lock tier config mutations
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(tier): add mutation RPC auth contract (#5082)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add peer mutation handler core (#5084)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add mutation control rpc service (#5087)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): recover prepared mutation drains (#5093)
Recover prepared tier mutation intent records into the local tier runtime so a restarted peer fails closed before issuing new remote-tier operation leases or conflicting admin publishes.
Reconcile the recovered block map on each scan so committed, aborted, or removed intents clear stale local blocks instead of wedging the peer until process restart.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): prove zero references before tier removal (#5092)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): clear peer mutation runtime blocks (#5094)
Install prepared mutation runtime blocks when peer prepare requests are applied or replayed so followers fail closed immediately before restart recovery.
Clear the in-memory block once peer commit or abort reaches a durable terminal state, including delayed duplicate prepare requests that observe a committed or aborted record.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add peer mutation control client
Add signed tier mutation prepare, commit, and abort client calls for peer fanout while preserving the existing protobuf and RPC contract.
Verify response proofs before interpreting peer outcomes, fail closed on invalid states, and reject oversized payloads before dialing peers.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(scanner): back off clean single-disk cycles
* fix(scanner): extend idle backoff across erasure clusters
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Two follow-up fixes to #4607 (both verified real bugs, each with a regression
test):
1. The in-call server_info retry did not re-dial on network errors. A
network-like first failure runs through `finalize_result`, which sets the
client's `offline` gate; the retry then called `evict_connection` and
re-invoked `server_info`, but `get_client` short-circuits on that gate and
returns "temporarily offline" without dialing. Only the async recovery
monitor would clear it. So the retry re-dialed only in the timeout branch and
was a no-op in the transport/half-open branch it was meant to cover. Add
`PeerRestClient::prepare_retry` (evict + clear the offline gate) and use it
before the retry.
2. Synthesized/degraded drive lists went empty on hostname deployments.
`PeerRestClient::host` is an `XHost` that `hosts_sorted` builds via
`XHost::try_from` -> `to_socket_addrs`, so it is the resolved `IP:port`; but
`synthesized_disks`/`peer_disk_health` compared it against
`Endpoint::host_port()`, which is the raw `hostname:port`. On hostname
clusters the compare missed, the drive list came back empty, and
`unknownDisks` stayed 0 — reproducing the "drives vanish from the summary"
regression #4607 fixed (also affected the pre-existing offline path). Compare
through a shared `endpoint_host_matches` helper that canonicalizes the
endpoint side through the same `XHost` resolution.
Tests: `peer_rest_client_prepare_retry_clears_offline_gate`,
`endpoint_host_matches_direct_and_canonicalized` (uses localhost, no external
DNS).
Follow-up to rustfs/rustfs#4607; tracked in rustfs/backlog#1049.
Co-authored-by: heihutu <heihutu@gmail.com>
A node could show OFFLINE in the console with zero logs explaining why
(rustfs/backlog#888, reported in rustfs#4304): the consecutive-failure
threshold crossing in handle_server_info_failure marked the peer offline
silently, the per-call warnings only existed on the observing node and
never named the transition, and the PeerRestClient offline flag plus its
background recovery monitor ran without any start/success log.
Logging-only change, no behavior change:
- handle_server_info_failure / handle_storage_info_failure: WARN
event="peer_marked_offline" exactly once at the threshold crossing
(later failures while already offline stay DEBUG), and DEBUG
event="peer_probe_failure" while returning cached state below the
threshold.
- update_server_info_cache / update_storage_info_cache: INFO
event="peer_recovered_online" when a probe succeeds after the peer had
been reported offline.
- PeerRestClient::mark_offline_and_spawn_recovery: WARN
event="peer_connection_marked_offline" when the offline flag is first
set (guarded by the recovery_running CAS so repeated failures do not
spam), and INFO event="peer_connection_recovered" with the attempt
count when connectivity is restored.
An "offline then back" episode now leaves a complete, correlatable
trace: N probe failures -> peer_marked_offline -> recovery monitor ->
peer_recovered_online.
Verification:
- cargo test -p rustfs-ecstore --lib (notification + peer_rest suites)
- make pre-commit
Ref: rustfs/backlog#888, rustfs#4304
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(server): make startup readiness wait configurable and raise default (#4264)
The startup runtime-readiness wait was a hardcoded 30s constant with no env
override. On slow multi-node cold starts (Docker/K8s/Synology NAS) this window
is shorter than the internal startup budgets it depends on — the endpoint
DNS-retry window (~90s) and the format-load retry loop (~100s worst case) — so
readiness times out and the node exits with
`startup readiness timed out after 30s: storage_ready=false, lock_quorum_ready=false`
before storage/lock quorum can converge, feeding the restart storm reported in
the issue.
- Add `RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS` (default 120s), documented in
rustfs-config health constants.
- Resolve the wait at runtime via `startup_runtime_readiness_max_wait()`; a
value of `0` falls back to the default instead of timing out instantly.
- Repoint `STARTUP_RUNTIME_READINESS_MAX_WAIT` at the shared config default so
there is a single source of truth, and cover the getter with unit tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): stop peer/disk background monitors on graceful shutdown (#4264)
Long-lived peer health/recovery and remote-disk monitors are detached
`tokio::spawn` tasks that each hold a `tracing::Span` via `.instrument(..)` for
their whole lifetime. Nothing cancelled them at shutdown, so on the normal
return path the Tokio runtime was dropped while they were still alive and their
`Span`s were dropped during worker-thread thread-local-storage (TLS)
destruction. At that point `tracing-subscriber`'s fmt `on_close` can touch an
already-destroyed TLS slot and panic with
`cannot access a Thread Local Storage value during or after destruction`, which
escalates to a panic-during-panic abort (SIGILL / exit 132) — the crash
reported on Synology in issue #4264, amplified by the restart storm.
- Add `cluster::rpc::background_monitor` with a process-global shutdown token,
`spawn_background_monitor()` (races the monitor future against that token so
its span drops while the runtime is alive), and public
`shutdown_background_monitors()`.
- Route every span-holding peer_s3 / peer_rest / remote_disk monitor spawn
through `spawn_background_monitor` instead of `tokio::spawn(..).instrument()`.
- Expose `rustfs_ecstore::shutdown_background_monitors()` and call it from the
graceful shutdown sequence (right after `ctx.cancel()`, before runtime
teardown) via the `storage_api` compatibility boundary.
Existing recovery-probe span-context tests still pass, confirming log
correlation is preserved.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Zhengchao An