mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-10 15:16:56 +00:00
feat(ecstore): converge startup topology under a wait-mode policy (#4631)
Multi-node startups (Kubernetes StatefulSets, bare-metal, VMs) can hit transient DNS/local-host resolution failures while peers and headless service records are still coming up. Previously endpoint resolution retried DNS for a fixed 90s window and then returned an error, which propagated out of run() and exited the process; in Kubernetes this drove repeated kubelet restarts even though the cluster eventually converged. Introduce a startup topology convergence policy with three modes: - orchestrated: wait effectively indefinitely for recoverable DNS/topology errors so the process stays Running (readiness stays false) instead of exiting; defer the DNS-IP cross-port validation because headless-service records can still be flapping. - bounded: wait a finite window (default 3m) then fail with an action-oriented error listing host/stage/elapsed and remediation hints. - fail-fast: fail on the first non-transient resolution error. Mode is auto-detected (Kubernetes -> orchestrated, distributed URL endpoints -> bounded, local path endpoints -> fail-fast) and can be overridden via RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE, RUSTFS_STARTUP_TOPOLOGY_WAIT_TIMEOUT and RUSTFS_STARTUP_TOPOLOGY_RETRY_MAX_DELAY. Also normalize divergent local/remote verdicts for endpoints that share a host:port, throttle retry warnings, and keep the DNS-independent local same-path checks in every mode. Configuration error handling is unchanged: malformed volumes, mixed styles/schemes, duplicate or root paths, and non-transient errors still fail fast. Read the topology env vars through rustfs_utils::get_env_opt_str for consistent alias handling, and simplify the log_once poisoned-lock branch to unwrap_or_else(PoisonError::into_inner). Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
@@ -79,10 +79,7 @@ static WARNED_ENV_MESSAGES: OnceLock<Mutex<HashSet<String>>> = OnceLock::new();
|
||||
|
||||
fn log_once(key: &str, message: impl FnOnce() -> String) {
|
||||
let seen = WARNED_ENV_MESSAGES.get_or_init(|| Mutex::new(HashSet::new()));
|
||||
let mut seen = match seen.lock() {
|
||||
Ok(seen) => seen,
|
||||
Err(poisoned) => poisoned.into_inner(),
|
||||
};
|
||||
let mut seen = seen.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
if seen.insert(key.to_string()) {
|
||||
warn!("{}", message());
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user