mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-07 13:53:12 +00:00
houseme
f73880f354
* fix(server): make startup readiness wait configurable and raise default (#4264) The startup runtime-readiness wait was a hardcoded 30s constant with no env override. On slow multi-node cold starts (Docker/K8s/Synology NAS) this window is shorter than the internal startup budgets it depends on — the endpoint DNS-retry window (~90s) and the format-load retry loop (~100s worst case) — so readiness times out and the node exits with `startup readiness timed out after 30s: storage_ready=false, lock_quorum_ready=false` before storage/lock quorum can converge, feeding the restart storm reported in the issue. - Add `RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS` (default 120s), documented in rustfs-config health constants. - Resolve the wait at runtime via `startup_runtime_readiness_max_wait()`; a value of `0` falls back to the default instead of timing out instantly. - Repoint `STARTUP_RUNTIME_READINESS_MAX_WAIT` at the shared config default so there is a single source of truth, and cover the getter with unit tests. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): stop peer/disk background monitors on graceful shutdown (#4264) Long-lived peer health/recovery and remote-disk monitors are detached `tokio::spawn` tasks that each hold a `tracing::Span` via `.instrument(..)` for their whole lifetime. Nothing cancelled them at shutdown, so on the normal return path the Tokio runtime was dropped while they were still alive and their `Span`s were dropped during worker-thread thread-local-storage (TLS) destruction. At that point `tracing-subscriber`'s fmt `on_close` can touch an already-destroyed TLS slot and panic with `cannot access a Thread Local Storage value during or after destruction`, which escalates to a panic-during-panic abort (SIGILL / exit 132) — the crash reported on Synology in issue #4264, amplified by the restart storm. - Add `cluster::rpc::background_monitor` with a process-global shutdown token, `spawn_background_monitor()` (races the monitor future against that token so its span drops while the runtime is alive), and public `shutdown_background_monitors()`. - Route every span-holding peer_s3 / peer_rest / remote_disk monitor spawn through `spawn_background_monitor` instead of `tokio::spawn(..).instrument()`. - Expose `rustfs_ecstore::shutdown_background_monitors()` and call it from the graceful shutdown sequence (right after `ctx.cancel()`, before runtime teardown) via the `storage_api` compatibility boundary. Existing recovery-probe span-context tests still pass, confirming log correlation is preserved. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>
70 lines
2.1 KiB
Rust
70 lines
2.1 KiB
Rust
// Copyright 2024 RustFS Team
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
pub mod api;
|
|
mod bucket;
|
|
mod cache_value;
|
|
mod cluster;
|
|
mod config;
|
|
mod core;
|
|
mod data_movement;
|
|
mod data_usage;
|
|
mod diagnostics;
|
|
mod disk;
|
|
mod erasure;
|
|
mod error;
|
|
mod io_support;
|
|
pub(crate) mod layout;
|
|
mod object_api;
|
|
mod runtime;
|
|
mod services;
|
|
mod set_disk;
|
|
mod storage_api_contracts;
|
|
mod store;
|
|
|
|
// pub mod checksum;
|
|
mod client;
|
|
mod event;
|
|
|
|
use rustfs_concurrency::WorkloadAdmissionSnapshotProvider;
|
|
use std::sync::Arc;
|
|
|
|
pub type WorkloadAdmissionSnapshotProviderRef = Arc<dyn WorkloadAdmissionSnapshotProvider + Send + Sync>;
|
|
|
|
pub fn set_workload_admission_snapshot_provider(
|
|
provider: WorkloadAdmissionSnapshotProviderRef,
|
|
) -> std::result::Result<(), WorkloadAdmissionSnapshotProviderRef> {
|
|
runtime::sources::set_workload_admission_snapshot_provider(provider)
|
|
}
|
|
|
|
/// Request shutdown of all long-lived peer/disk background monitor tasks.
|
|
///
|
|
/// Call this during graceful shutdown, *before* the Tokio runtime is dropped, so
|
|
/// each monitor future (and the `tracing::Span` it holds) is dropped while the
|
|
/// runtime and tracing subscriber are still alive. This avoids the
|
|
/// thread-local-storage `on_close` panic that can otherwise abort the process
|
|
/// during worker-thread teardown (issue #4264). Idempotent and cheap.
|
|
pub fn shutdown_background_monitors() {
|
|
cluster::rpc::shutdown_background_monitors();
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod rio_tests {
|
|
#[test]
|
|
fn uses_expected_rio_backend() {
|
|
let expected = if cfg!(feature = "rio-v2") { "rio-v2" } else { "legacy-rio" };
|
|
assert_eq!(crate::io_support::rio::backend_name(), expected);
|
|
}
|
|
}
|