feat(api): wire opt-in per-client S3 API rate limiting (backlog#1191)
RustFS shipped three rate-limiter implementations and none was wired to
any request path: the tower layer never returned 429 (its over-limit
branch passed requests through) and was never instantiated, the console
env switches only logged, and the Swift token bucket was never called.
Replace them with one working, default-off implementation:
- Rewrite rustfs/src/server/rate_limit.rs as a sharded per-client-IP
token-bucket limiter (32 mutex shards instead of one global RwLock
write per request), bounded at 100k tracked IPs with lossless
refilled-idle sweeps, returning 429 + Retry-After + x-ratelimit-*
headers and an S3-style XML body.
- Key on trusted-proxy-validated ClientInfo.real_ip, else the socket
peer address; never read spoofable X-Forwarded-For/X-Real-IP headers.
Requests without a resolvable identity fail open. The echoed request
id is charset-gated to prevent reflected XML injection.
- Wire the layer once at startup via option_layer between
CatchPanicLayer and ReadinessGateLayer (external stack only), gated by
new RUSTFS_API_RATE_LIMIT_ENABLE/_RPM/_BURST constants; health and
profiling probes, internode RPC/gRPC, and the console are exempt.
- Make RUSTFS_CONSOLE_RATE_LIMIT_ENABLE/_RPM actually enforce by
reusing the same limiter core through an axum middleware.
- Delete the dead Swift ratelimit module, its isolated tests, and the
stale logging-guardrail entry; keep the live SwiftError 429 mapping.
- Add unit tests (exhaustion/recovery with injected time, concurrency,
cap eviction, spoofed-header and fail-open behavior, env matrix) and
e2e tests proving 429 + Retry-After on the real server and zero
behavior change with default configuration.
Zhengchao An