mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-13 16:46:55 +00:00
houseme
f73880f354
* fix(server): make startup readiness wait configurable and raise default (#4264) The startup runtime-readiness wait was a hardcoded 30s constant with no env override. On slow multi-node cold starts (Docker/K8s/Synology NAS) this window is shorter than the internal startup budgets it depends on — the endpoint DNS-retry window (~90s) and the format-load retry loop (~100s worst case) — so readiness times out and the node exits with `startup readiness timed out after 30s: storage_ready=false, lock_quorum_ready=false` before storage/lock quorum can converge, feeding the restart storm reported in the issue. - Add `RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS` (default 120s), documented in rustfs-config health constants. - Resolve the wait at runtime via `startup_runtime_readiness_max_wait()`; a value of `0` falls back to the default instead of timing out instantly. - Repoint `STARTUP_RUNTIME_READINESS_MAX_WAIT` at the shared config default so there is a single source of truth, and cover the getter with unit tests. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): stop peer/disk background monitors on graceful shutdown (#4264) Long-lived peer health/recovery and remote-disk monitors are detached `tokio::spawn` tasks that each hold a `tracing::Span` via `.instrument(..)` for their whole lifetime. Nothing cancelled them at shutdown, so on the normal return path the Tokio runtime was dropped while they were still alive and their `Span`s were dropped during worker-thread thread-local-storage (TLS) destruction. At that point `tracing-subscriber`'s fmt `on_close` can touch an already-destroyed TLS slot and panic with `cannot access a Thread Local Storage value during or after destruction`, which escalates to a panic-during-panic abort (SIGILL / exit 132) — the crash reported on Synology in issue #4264, amplified by the restart storm. - Add `cluster::rpc::background_monitor` with a process-global shutdown token, `spawn_background_monitor()` (races the monitor future against that token so its span drops while the runtime is alive), and public `shutdown_background_monitors()`. - Route every span-holding peer_s3 / peer_rest / remote_disk monitor spawn through `spawn_background_monitor` instead of `tokio::spawn(..).instrument()`. - Expose `rustfs_ecstore::shutdown_background_monitors()` and call it from the graceful shutdown sequence (right after `ctx.cancel()`, before runtime teardown) via the `storage_api` compatibility boundary. Existing recovery-probe span-context tests still pass, confirming log correlation is preserved. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>
42 lines
1.7 KiB
Rust
42 lines
1.7 KiB
Rust
// Copyright 2024 RustFS Team
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
pub(crate) mod background_monitor;
|
|
pub(crate) mod client;
|
|
pub(crate) mod context_propagation;
|
|
pub(crate) mod http_auth;
|
|
pub(crate) mod internode_data_transport;
|
|
pub(crate) mod peer_rest_client;
|
|
pub(crate) mod peer_s3_client;
|
|
pub(crate) mod remote_disk;
|
|
pub(crate) mod remote_locker;
|
|
pub(crate) mod runtime_sources;
|
|
|
|
pub use background_monitor::shutdown_background_monitors;
|
|
pub(crate) use background_monitor::spawn_background_monitor;
|
|
pub use client::{
|
|
TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client, node_service_time_out_client_no_auth,
|
|
};
|
|
pub use http_auth::{TONIC_RPC_PREFIX, build_auth_headers, gen_signature_headers, verify_rpc_signature};
|
|
#[cfg(test)]
|
|
pub(crate) use internode_data_transport::TcpHttpInternodeDataTransport;
|
|
pub use internode_data_transport::build_internode_data_transport_from_env;
|
|
pub use peer_rest_client::{
|
|
PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC,
|
|
};
|
|
pub(crate) use peer_s3_client::heal_bucket_local_on_disks;
|
|
pub use peer_s3_client::{LocalPeerS3Client, PeerS3Client, S3PeerSys};
|
|
pub use remote_disk::RemoteDisk;
|
|
pub use remote_locker::RemoteClient;
|