The startup topology convergence auto-detection (added in #4631) checked
Kubernetes before endpoint style, so a local path deployment (single or
multi-drive, non-distributed) running inside Kubernetes was classified as
orchestrated with an effectively unbounded wait window. Path-style
endpoints have no hostnames to resolve, so the wait never actually
triggers and runtime behavior is unchanged, but the startup log then
advertised mode=orchestrated / wait_timeout=MAX for a purely local
deployment, which is misleading and contradicts the documented policy
(local path endpoints -> fail-fast).
Order the auto-detection by endpoint style first: only distributed URL
endpoints, which resolve hostnames, pick orchestrated (Kubernetes) or
bounded (otherwise); local path endpoints stay fail-fast regardless of
the platform. Update the doc comment to match and add a policy case
locking Kubernetes + local path to fail-fast.
Co-authored-by: heihutu <heihutu@gmail.com>
Multi-node startups (Kubernetes StatefulSets, bare-metal, VMs) can hit
transient DNS/local-host resolution failures while peers and headless
service records are still coming up. Previously endpoint resolution
retried DNS for a fixed 90s window and then returned an error, which
propagated out of run() and exited the process; in Kubernetes this drove
repeated kubelet restarts even though the cluster eventually converged.
Introduce a startup topology convergence policy with three modes:
- orchestrated: wait effectively indefinitely for recoverable
DNS/topology errors so the process stays Running (readiness stays
false) instead of exiting; defer the DNS-IP cross-port validation
because headless-service records can still be flapping.
- bounded: wait a finite window (default 3m) then fail with an
action-oriented error listing host/stage/elapsed and remediation hints.
- fail-fast: fail on the first non-transient resolution error.
Mode is auto-detected (Kubernetes -> orchestrated, distributed URL
endpoints -> bounded, local path endpoints -> fail-fast) and can be
overridden via RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE,
RUSTFS_STARTUP_TOPOLOGY_WAIT_TIMEOUT and
RUSTFS_STARTUP_TOPOLOGY_RETRY_MAX_DELAY.
Also normalize divergent local/remote verdicts for endpoints that share
a host:port, throttle retry warnings, and keep the DNS-independent local
same-path checks in every mode. Configuration error handling is
unchanged: malformed volumes, mixed styles/schemes, duplicate or root
paths, and non-transient errors still fail fast.
Read the topology env vars through rustfs_utils::get_env_opt_str for
consistent alias handling, and simplify the log_once poisoned-lock branch
to unwrap_or_else(PoisonError::into_inner).
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): retry transient endpoint DNS failures
* fix(ecstore): satisfy DNS retry clippy check
* fix(ecstore): harden endpoint DNS retry
* test(ecstore): stabilize DNS retry error test
houseme