gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-20 03:22:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(ecstore): converge startup topology under a wait-mode policy (#4631)

Browse Source 
Multi-node startups (Kubernetes StatefulSets, bare-metal, VMs) can hit
transient DNS/local-host resolution failures while peers and headless
service records are still coming up. Previously endpoint resolution
retried DNS for a fixed 90s window and then returned an error, which
propagated out of run() and exited the process; in Kubernetes this drove
repeated kubelet restarts even though the cluster eventually converged.

Introduce a startup topology convergence policy with three modes:

- orchestrated: wait effectively indefinitely for recoverable
  DNS/topology errors so the process stays Running (readiness stays
  false) instead of exiting; defer the DNS-IP cross-port validation
  because headless-service records can still be flapping.
- bounded: wait a finite window (default 3m) then fail with an
  action-oriented error listing host/stage/elapsed and remediation hints.
- fail-fast: fail on the first non-transient resolution error.

Mode is auto-detected (Kubernetes -> orchestrated, distributed URL
endpoints -> bounded, local path endpoints -> fail-fast) and can be
overridden via RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE,
RUSTFS_STARTUP_TOPOLOGY_WAIT_TIMEOUT and
RUSTFS_STARTUP_TOPOLOGY_RETRY_MAX_DELAY.

Also normalize divergent local/remote verdicts for endpoints that share
a host:port, throttle retry warnings, and keep the DNS-independent local
same-path checks in every mode. Configuration error handling is
unchanged: malformed volumes, mixed styles/schemes, duplicate or root
paths, and non-transient errors still fail fast.

Read the topology env vars through rustfs_utils::get_env_opt_str for
consistent alias handling, and simplify the log_once poisoned-lock branch
to unwrap_or_else(PoisonError::into_inner).

Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
2026-07-10 01:57:13 +08:00
committed by GitHub
parent da755eb66b
commit 6a81353785
3 changed files with 494 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/config/src/constants/app.rs
+35
View File
@@ -143,6 +143,41 @@ pub const ENV_MINIO_CI: &str = "MINIO_CI";
/// Default flag value for bypassing local physical disk independence checks.
pub const DEFAULT_UNSAFE_BYPASS_DISK_CHECK: bool = false;
/// Environment variable selecting how startup waits for DNS/topology
/// convergence in multi-node setups. One of `auto` (default), `orchestrated`,
/// `bounded`, or `fail-fast`.
///
/// - `orchestrated`: wait effectively indefinitely for recoverable DNS/topology
/// errors so the process stays Running (readiness stays false) instead of
/// exiting and triggering a Kubernetes pod restart loop.
/// - `bounded`: wait for a finite window (see
/// `ENV_STARTUP_TOPOLOGY_WAIT_TIMEOUT`) then fail with an actionable error.
/// - `fail-fast`: fail on the first non-transient resolution error (CI/local).
/// - `auto`: Kubernetes -> orchestrated, distributed URL endpoints -> bounded,
/// local path endpoints -> fail-fast.
pub const ENV_STARTUP_TOPOLOGY_WAIT_MODE: &str = "RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE";
/// Environment variable bounding how long startup waits for topology/DNS
/// convergence before failing in `bounded` mode. Accepts durations such as
/// `3m`, `90s`, `500ms`, or a bare number of seconds.
pub const ENV_STARTUP_TOPOLOGY_WAIT_TIMEOUT: &str = "RUSTFS_STARTUP_TOPOLOGY_WAIT_TIMEOUT";
/// Environment variable capping the per-attempt backoff delay while retrying
/// startup topology/DNS convergence. Accepts durations such as `8s`.
pub const ENV_STARTUP_TOPOLOGY_RETRY_MAX_DELAY: &str = "RUSTFS_STARTUP_TOPOLOGY_RETRY_MAX_DELAY";
/// Environment variable injected into every Kubernetes pod; its presence marks
/// an orchestrated deployment for startup topology auto-detection.
pub const ENV_KUBERNETES_SERVICE_HOST: &str = "KUBERNETES_SERVICE_HOST";
/// Default bounded-mode wait window (seconds) for non-Kubernetes distributed
/// deployments before startup topology convergence is declared failed.
pub const DEFAULT_STARTUP_TOPOLOGY_WAIT_TIMEOUT_SECS: u64 = 180;
/// Default per-attempt backoff cap (seconds) while retrying startup
/// topology/DNS convergence.
pub const DEFAULT_STARTUP_TOPOLOGY_RETRY_MAX_DELAY_SECS: u64 = 8;
/// Environment variable for server access key.
pub const ENV_RUSTFS_ACCESS_KEY: &str = "RUSTFS_ACCESS_KEY";