gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-19 11:06:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(api): add opt-in per-bucket dimension to the S3 API rate limiter (#4949)

Browse Source 
Follow-up to #4895 (backlog#1191 deferred sub-item). The client-IP
dimension gives per-client fairness; this adds a collective per-bucket
budget so one hot bucket cannot monopolize the server regardless of how
many client IPs the traffic is spread across.

- Generalize RateLimiter over its key type with a Borrow-based check()
  so &str lookups against String bucket keys stay allocation-free on
  the hit path; client-IP call sites are unchanged in behavior.
- RateLimitLayer now carries optional client and bucket limiters; a
  request must pass every configured dimension, and rejections report
  which one tripped via a new 'dimension' metric label.
- Bucket extraction mirrors s3s host routing: virtual-hosted-style
  resolves the Host/authority prefix against the same expanded domain
  set (with port variants) the s3s router uses; otherwise the first
  path segment. Admin and table-catalog namespaces are never buckets.
- New env vars RUSTFS_API_RATE_LIMIT_BUCKET_RPM/_BURST (default 0 =
  dimension off) under the existing enable switch; bucket-only
  configurations (client RPM 0) are supported.
- Bucket names are attacker-chosen, so the bounded-shards design
  (100k keys, lossless idle sweeps, most-idle eviction) is the memory
  defense; a test floods 10k random names and asserts the cap holds.
- Unit tests for extraction, shared bucket budgets across IPs, both-
  dimensions interaction, and the extended env matrix; e2e test proves
  bucket-only throttling on the real server with an unrelated bucket
  unaffected.
This commit is contained in:
Zhengchao An
2026-07-17 19:01:18 +08:00
committed by GitHub
parent e1fc4b12ea
commit 97edb2e5cf
5 changed files with 536 additions and 144 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/config/src/constants/api.rs
+23
View File
@@ -50,3 +50,26 @@ pub const ENV_API_RATE_LIMIT_BURST: &str = "RUSTFS_API_RATE_LIMIT_BURST";
/// Default for `RUSTFS_API_RATE_LIMIT_BURST` (`0` = same as RPM).
pub const DEFAULT_API_RATE_LIMIT_BURST: u32 = 0;
/// Sustained S3 API request budget per addressed bucket, in requests per
/// minute — a collective ceiling shared by all clients of that bucket.
///
/// Complements the per-client-IP dimension: it protects the server from one
/// hot bucket regardless of how many client IPs the traffic comes from. `0`
/// disables the bucket dimension. Requires `RUSTFS_API_RATE_LIMIT_ENABLE`.
/// Environment variable: RUSTFS_API_RATE_LIMIT_BUCKET_RPM
/// Example: RUSTFS_API_RATE_LIMIT_BUCKET_RPM=60000
pub const ENV_API_RATE_LIMIT_BUCKET_RPM: &str = "RUSTFS_API_RATE_LIMIT_BUCKET_RPM";
/// Default for `RUSTFS_API_RATE_LIMIT_BUCKET_RPM` (`0` = dimension disabled).
pub const DEFAULT_API_RATE_LIMIT_BUCKET_RPM: u32 = 0;
/// Burst capacity per bucket (maximum tokens in the bucket-dimension bucket).
///
/// `0` means "same as `RUSTFS_API_RATE_LIMIT_BUCKET_RPM`".
/// Environment variable: RUSTFS_API_RATE_LIMIT_BUCKET_BURST
/// Example: RUSTFS_API_RATE_LIMIT_BUCKET_BURST=2000
pub const ENV_API_RATE_LIMIT_BUCKET_BURST: &str = "RUSTFS_API_RATE_LIMIT_BUCKET_BURST";
/// Default for `RUSTFS_API_RATE_LIMIT_BUCKET_BURST` (`0` = same as bucket RPM).
pub const DEFAULT_API_RATE_LIMIT_BUCKET_BURST: u32 = 0;