feat(api): wire opt-in per-client S3 API rate limiting (429 + Retry-After) (#4895)

feat(api): wire opt-in per-client S3 API rate limiting (backlog#1191)

RustFS shipped three rate-limiter implementations and none was wired to
any request path: the tower layer never returned 429 (its over-limit
branch passed requests through) and was never instantiated, the console
env switches only logged, and the Swift token bucket was never called.

Replace them with one working, default-off implementation:

- Rewrite rustfs/src/server/rate_limit.rs as a sharded per-client-IP
  token-bucket limiter (32 mutex shards instead of one global RwLock
  write per request), bounded at 100k tracked IPs with lossless
  refilled-idle sweeps, returning 429 + Retry-After + x-ratelimit-*
  headers and an S3-style XML body.
- Key on trusted-proxy-validated ClientInfo.real_ip, else the socket
  peer address; never read spoofable X-Forwarded-For/X-Real-IP headers.
  Requests without a resolvable identity fail open. The echoed request
  id is charset-gated to prevent reflected XML injection.
- Wire the layer once at startup via option_layer between
  CatchPanicLayer and ReadinessGateLayer (external stack only), gated by
  new RUSTFS_API_RATE_LIMIT_ENABLE/_RPM/_BURST constants; health and
  profiling probes, internode RPC/gRPC, and the console are exempt.
- Make RUSTFS_CONSOLE_RATE_LIMIT_ENABLE/_RPM actually enforce by
  reusing the same limiter core through an axum middleware.
- Delete the dead Swift ratelimit module, its isolated tests, and the
  stale logging-guardrail entry; keep the live SwiftError 429 mapping.
- Add unit tests (exhaustion/recovery with injected time, concurrency,
  cap eviction, spoofed-header and fail-open behavior, env matrix) and
  e2e tests proving 429 + Retry-After on the real server and zero
  behavior change with default configuration.

crates/config/src/constants/api.rs

@@ -0,0 +1,52 @@
+//  Copyright 2024 RustFS Team
+//
+//  Licensed under the Apache License, Version 2.0 (the "License");
+//  you may not use this file except in compliance with the License.
+//  You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+//  Unless required by applicable law or agreed to in writing, software
+//  distributed under the License is distributed on an "AS IS" BASIS,
+//  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+//  See the License for the specific language governing permissions and
+//  limitations under the License.
+
+/// Enable or disable per-client rate limiting for the S3 API.
+///
+/// When enabled (and `RUSTFS_API_RATE_LIMIT_RPM` > 0), requests are throttled
+/// per client IP using a token bucket; over-limit requests receive
+/// `429 Too Many Requests` with a `Retry-After` header. Internode RPC/gRPC,
+/// health probes, and the console (which has its own limiter) are exempt.
+/// Environment variable: RUSTFS_API_RATE_LIMIT_ENABLE
+/// Example: RUSTFS_API_RATE_LIMIT_ENABLE=true
+pub const ENV_API_RATE_LIMIT_ENABLE: &str = "RUSTFS_API_RATE_LIMIT_ENABLE";
+
+/// Default for `RUSTFS_API_RATE_LIMIT_ENABLE`.
+///
+/// Disabled by default: RustFS ships permissive and operators opt in to
+/// abuse-protection hardening. When disabled the request path is unchanged.
+pub const DEFAULT_API_RATE_LIMIT_ENABLE: bool = false;
+
+/// Sustained S3 API request budget per client IP, in requests per minute.
+///
+/// `0` means unlimited (rate limiting stays inert even when enabled).
+/// Environment variable: RUSTFS_API_RATE_LIMIT_RPM
+/// Example: RUSTFS_API_RATE_LIMIT_RPM=6000
+pub const ENV_API_RATE_LIMIT_RPM: &str = "RUSTFS_API_RATE_LIMIT_RPM";
+
+/// Default for `RUSTFS_API_RATE_LIMIT_RPM`.
+///
+/// `0` (unlimited) so that setting only the enable switch cannot throttle
+/// traffic by surprise; operators must choose an explicit budget.
+pub const DEFAULT_API_RATE_LIMIT_RPM: u32 = 0;
+
+/// Burst capacity per client IP (maximum tokens in the bucket).
+///
+/// Allows short spikes above the sustained rate. `0` means "same as RPM".
+/// Environment variable: RUSTFS_API_RATE_LIMIT_BURST
+/// Example: RUSTFS_API_RATE_LIMIT_BURST=200
+pub const ENV_API_RATE_LIMIT_BURST: &str = "RUSTFS_API_RATE_LIMIT_BURST";
+
+/// Default for `RUSTFS_API_RATE_LIMIT_BURST` (`0` = same as RPM).
+pub const DEFAULT_API_RATE_LIMIT_BURST: u32 = 0;

crates/config/src/constants/mod.rs

@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+pub(crate) mod api;
 pub(crate) mod app;
 pub(crate) mod body_limits;
 pub(crate) mod capacity;

crates/config/src/lib.rs

@@ -15,6 +15,8 @@
 #[cfg(feature = "constants")]
 pub mod constants;
 #[cfg(feature = "constants")]
+pub use constants::api::*;
+#[cfg(feature = "constants")]
 pub use constants::app::*;
 #[cfg(feature = "constants")]
 pub use constants::body_limits::*;