mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-09 06:39:25 +00:00
fix(ecstore): fail fast on missing RPC secret in distributed startup (#5248)
When endpoints include remote nodes and no internode RPC secret is resolvable, every remote format read fails client-side with "No valid auth token" and startup dies ~2 minutes later with the misleading "store init failed to load formats after 10 retries: erasure read quorum" error (issues #4939, #5153). Add a preflight to ECStore init that resolves the RPC secret once before any disk opens: if any endpoint is non-local and resolution fails, abort immediately with the operator-facing remediation message. The preflight also emits the "RPC auth secret resolution failed" log line so the log-analyzer rpc-secret-resolution rule keeps firing for this scenario. Single-node (all-local) topologies never invoke the resolver.
This commit is contained in:
@@ -123,8 +123,10 @@ target/bench/internode-transport/
|
||||
Captured while running the first real A/B on a 4-node ansible cluster; needed before any live run:
|
||||
|
||||
- **RPC secret is mandatory on current `main`.** Internode RPC fails closed: default creds
|
||||
(`RUSTFS_SECRET_KEY=rustfsadmin`) with no `RUSTFS_RPC_SECRET` → `No valid auth token` → the cluster
|
||||
never reaches `storage_quorum`. Set a **non-default** `RUSTFS_RPC_SECRET`, identical on every node.
|
||||
(`RUSTFS_SECRET_KEY=rustfsadmin`) with no `RUSTFS_RPC_SECRET` → the node aborts startup immediately
|
||||
with `store init aborted: endpoints include remote nodes but RPC authentication secret is not
|
||||
configured` (builds before the preflight instead showed `No valid auth token` and never reached
|
||||
`storage_quorum`). Set a **non-default** `RUSTFS_RPC_SECRET`, identical on every node.
|
||||
- **systemd start timeout.** The install unit is `Type=notify`; READY only fires after quorum, which on
|
||||
freshly-purged disks exceeds a 30 s `TimeoutStartSec` → crash loop. Use a drop-in `TimeoutStartSec=infinity`.
|
||||
- **Server-side metrics need OTLP.** RustFS has no Prometheus pull endpoint (`/admin/v3/metrics` is NDJSON,
|
||||
|
||||
Reference in New Issue
Block a user