fix(ecstore): fail fast on missing RPC secret in distributed startup (#5248)

When endpoints include remote nodes and no internode RPC secret is
resolvable, every remote format read fails client-side with "No valid
auth token" and startup dies ~2 minutes later with the misleading
"store init failed to load formats after 10 retries: erasure read
quorum" error (issues #4939, #5153).

Add a preflight to ECStore init that resolves the RPC secret once
before any disk opens: if any endpoint is non-local and resolution
fails, abort immediately with the operator-facing remediation message.
The preflight also emits the "RPC auth secret resolution failed" log
line so the log-analyzer rpc-secret-resolution rule keeps firing for
this scenario. Single-node (all-local) topologies never invoke the
resolver.
This commit is contained in:
Zhengchao An
2026-07-26 08:28:38 +08:00
committed by GitHub
parent 44d2c3bd34
commit 5c99ca1328
2 changed files with 171 additions and 5 deletions
@@ -123,8 +123,10 @@ target/bench/internode-transport/
Captured while running the first real A/B on a 4-node ansible cluster; needed before any live run:
- **RPC secret is mandatory on current `main`.** Internode RPC fails closed: default creds
(`RUSTFS_SECRET_KEY=rustfsadmin`) with no `RUSTFS_RPC_SECRET``No valid auth token` → the cluster
never reaches `storage_quorum`. Set a **non-default** `RUSTFS_RPC_SECRET`, identical on every node.
(`RUSTFS_SECRET_KEY=rustfsadmin`) with no `RUSTFS_RPC_SECRET`the node aborts startup immediately
with `store init aborted: endpoints include remote nodes but RPC authentication secret is not
configured` (builds before the preflight instead showed `No valid auth token` and never reached
`storage_quorum`). Set a **non-default** `RUSTFS_RPC_SECRET`, identical on every node.
- **systemd start timeout.** The install unit is `Type=notify`; READY only fires after quorum, which on
freshly-purged disks exceeds a 30 s `TimeoutStartSec` → crash loop. Use a drop-in `TimeoutStartSec=infinity`.
- **Server-side metrics need OTLP.** RustFS has no Prometheus pull endpoint (`/admin/v3/metrics` is NDJSON,