mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-21 11:56:38 +00:00
houseme
f73880f354
* fix(server): make startup readiness wait configurable and raise default (#4264) The startup runtime-readiness wait was a hardcoded 30s constant with no env override. On slow multi-node cold starts (Docker/K8s/Synology NAS) this window is shorter than the internal startup budgets it depends on — the endpoint DNS-retry window (~90s) and the format-load retry loop (~100s worst case) — so readiness times out and the node exits with `startup readiness timed out after 30s: storage_ready=false, lock_quorum_ready=false` before storage/lock quorum can converge, feeding the restart storm reported in the issue. - Add `RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS` (default 120s), documented in rustfs-config health constants. - Resolve the wait at runtime via `startup_runtime_readiness_max_wait()`; a value of `0` falls back to the default instead of timing out instantly. - Repoint `STARTUP_RUNTIME_READINESS_MAX_WAIT` at the shared config default so there is a single source of truth, and cover the getter with unit tests. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): stop peer/disk background monitors on graceful shutdown (#4264) Long-lived peer health/recovery and remote-disk monitors are detached `tokio::spawn` tasks that each hold a `tracing::Span` via `.instrument(..)` for their whole lifetime. Nothing cancelled them at shutdown, so on the normal return path the Tokio runtime was dropped while they were still alive and their `Span`s were dropped during worker-thread thread-local-storage (TLS) destruction. At that point `tracing-subscriber`'s fmt `on_close` can touch an already-destroyed TLS slot and panic with `cannot access a Thread Local Storage value during or after destruction`, which escalates to a panic-during-panic abort (SIGILL / exit 132) — the crash reported on Synology in issue #4264, amplified by the restart storm. - Add `cluster::rpc::background_monitor` with a process-global shutdown token, `spawn_background_monitor()` (races the monitor future against that token so its span drops while the runtime is alive), and public `shutdown_background_monitors()`. - Route every span-holding peer_s3 / peer_rest / remote_disk monitor spawn through `spawn_background_monitor` instead of `tokio::spawn(..).instrument()`. - Expose `rustfs_ecstore::shutdown_background_monitors()` and call it from the graceful shutdown sequence (right after `ctx.cancel()`, before runtime teardown) via the `storage_api` compatibility boundary. Existing recovery-probe span-context tests still pass, confirming log correlation is preserved. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>
68 lines
3.8 KiB
Rust
68 lines
3.8 KiB
Rust
// Copyright 2024 RustFS Team
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
/// Enable or disable public `/health` and `/health/ready` endpoints.
|
|
/// When disabled, the routes are not registered and return 404.
|
|
pub const ENV_HEALTH_ENDPOINT_ENABLE: &str = "RUSTFS_HEALTH_ENDPOINT_ENABLE";
|
|
pub const DEFAULT_HEALTH_ENDPOINT_ENABLE: bool = true;
|
|
|
|
/// Cache TTL for storage readiness runtime-state evaluation (milliseconds).
|
|
/// This reduces storage-layer pressure when probes are called at high frequency.
|
|
pub const ENV_HEALTH_READINESS_CACHE_TTL_MS: &str = "RUSTFS_HEALTH_READINESS_CACHE_TTL_MS";
|
|
pub const DEFAULT_HEALTH_READINESS_CACHE_TTL_MS: u64 = 1000;
|
|
|
|
/// Timeout for cluster health readiness collectors (milliseconds).
|
|
/// This bounds expensive storage and lock quorum checks used by cluster probes.
|
|
pub const ENV_HEALTH_CLUSTER_TIMEOUT_MS: &str = "RUSTFS_HEALTH_CLUSTER_TIMEOUT_MS";
|
|
pub const DEFAULT_HEALTH_CLUSTER_TIMEOUT_MS: u64 = 2000;
|
|
|
|
/// Maximum time to wait for local node runtime readiness (storage / IAM / lock
|
|
/// quorum) during startup before failing fast (seconds).
|
|
///
|
|
/// On slow or multi-node cold starts — e.g. Docker/Kubernetes/NAS deployments
|
|
/// where peer DNS records and erasure-format quorum take time to converge — this
|
|
/// budget must be large enough to outlast the internal startup DNS-retry window
|
|
/// and the format-load retry loop. Increase it for slow NAS/edge clusters; lower
|
|
/// it for fast single-node setups that should fail fast. A value of `0` is
|
|
/// treated as the default rather than an instant timeout.
|
|
pub const ENV_STARTUP_READINESS_MAX_WAIT_SECS: &str = "RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS";
|
|
pub const DEFAULT_STARTUP_READINESS_MAX_WAIT_SECS: u64 = 120;
|
|
|
|
/// Enable minimal health payload mode for GET `/health*` responses.
|
|
/// When enabled, only `status` and `ready` fields are returned.
|
|
pub const ENV_HEALTH_MINIMAL_RESPONSE_ENABLE: &str = "RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE";
|
|
pub const DEFAULT_HEALTH_MINIMAL_RESPONSE_ENABLE: bool = false;
|
|
|
|
/// Enable busy protection for health probes.
|
|
/// When enabled with a positive request threshold, alias health probes may
|
|
/// return 429 when active HTTP requests exceed the threshold.
|
|
pub const ENV_HEALTH_COMPAT_BUSY_CHECK_ENABLE: &str = "RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE";
|
|
pub const DEFAULT_HEALTH_COMPAT_BUSY_CHECK_ENABLE: bool = false;
|
|
|
|
/// Max active HTTP requests; alias health probes report busy (429) when active requests reach or exceed this value.
|
|
/// Set to 0 to disable thresholding even when busy protection is enabled.
|
|
pub const ENV_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS: &str = "RUSTFS_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS";
|
|
pub const DEFAULT_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS: usize = 0;
|
|
|
|
/// Enable KMS readiness check for alias readiness probes.
|
|
/// When enabled, `/health/ready` additionally requires KMS service to be
|
|
/// in running state if a global KMS manager exists.
|
|
pub const ENV_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE: &str = "RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE";
|
|
pub const DEFAULT_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE: bool = false;
|
|
|
|
/// Enable peer-health readiness impact.
|
|
/// When disabled, peer-health state is reported but does not affect readiness.
|
|
pub const ENV_HEALTH_PEER_READY_CHECK_ENABLE: &str = "RUSTFS_HEALTH_PEER_READY_CHECK_ENABLE";
|
|
pub const DEFAULT_HEALTH_PEER_READY_CHECK_ENABLE: bool = false;
|