feat(ecstore): converge startup topology under a wait-mode policy (#4631)

Multi-node startups (Kubernetes StatefulSets, bare-metal, VMs) can hit
transient DNS/local-host resolution failures while peers and headless
service records are still coming up. Previously endpoint resolution
retried DNS for a fixed 90s window and then returned an error, which
propagated out of run() and exited the process; in Kubernetes this drove
repeated kubelet restarts even though the cluster eventually converged.

Introduce a startup topology convergence policy with three modes:

- orchestrated: wait effectively indefinitely for recoverable
  DNS/topology errors so the process stays Running (readiness stays
  false) instead of exiting; defer the DNS-IP cross-port validation
  because headless-service records can still be flapping.
- bounded: wait a finite window (default 3m) then fail with an
  action-oriented error listing host/stage/elapsed and remediation hints.
- fail-fast: fail on the first non-transient resolution error.

Mode is auto-detected (Kubernetes -> orchestrated, distributed URL
endpoints -> bounded, local path endpoints -> fail-fast) and can be
overridden via RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE,
RUSTFS_STARTUP_TOPOLOGY_WAIT_TIMEOUT and
RUSTFS_STARTUP_TOPOLOGY_RETRY_MAX_DELAY.

Also normalize divergent local/remote verdicts for endpoints that share
a host:port, throttle retry warnings, and keep the DNS-independent local
same-path checks in every mode. Configuration error handling is
unchanged: malformed volumes, mixed styles/schemes, duplicate or root
paths, and non-transient errors still fail fast.

Read the topology env vars through rustfs_utils::get_env_opt_str for
consistent alias handling, and simplify the log_once poisoned-lock branch
to unwrap_or_else(PoisonError::into_inner).

Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
2026-07-10 01:57:13 +08:00
committed by GitHub
parent da755eb66b
commit 6a81353785
3 changed files with 494 additions and 83 deletions
+1 -4
View File
@@ -79,10 +79,7 @@ static WARNED_ENV_MESSAGES: OnceLock<Mutex<HashSet<String>>> = OnceLock::new();
fn log_once(key: &str, message: impl FnOnce() -> String) {
let seen = WARNED_ENV_MESSAGES.get_or_init(|| Mutex::new(HashSet::new()));
let mut seen = match seen.lock() {
Ok(seen) => seen,
Err(poisoned) => poisoned.into_inner(),
};
let mut seen = seen.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
if seen.insert(key.to_string()) {
warn!("{}", message());
}