mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-28 16:07:05 +00:00
fix(memory): cgroup-aware resource detection for container environments (#6536)
* fix(iam): raise recursion limit for migration test Co-Authored-By: heihutu <heihutu@gmail.com> * fix(memory): cgroup-aware resource detection for container environments Issue #5803 reported memory RSS regression since beta.9: - RSS memory steps ~+300 MiB on tiny S3 bursts and never returns - Daily OOMKills in 1 GiB containers - Root cause: RustFS uses host memory/CPU instead of container cgroup limits Changes: - Add cgroup_resources.rs: cgroup v1/v2 CPU and memory detection - Add container_config.rs: container configuration with env overrides - Fix memory_observability.rs: use effective memory (cgroup-aware) - Fix server/runtime.rs: use cgroup-aware CPU detection for Tokio - Cap max_blocking_threads to 256 for small containers (<=4 cores) - Add new metrics: rustfs_memory_effective_total_bytes, rustfs_cgroup_* - Add startup logging of detected container resources New environment variables: - RUSTFS_DISABLE_CGROUP_DETECTION: disable cgroup detection - RUSTFS_OVERRIDE_CPU_CORES: override detected CPU cores - RUSTFS_OVERRIDE_MEMORY_BYTES: override detected memory limit Fixes: rustfs/rustfs#5803 Tracking: rustfs/backlog#2012 Co-Authored-By: heihutu <heihutu@gmail.com> * style: apply cargo fmt formatting Co-Authored-By: heihutu <heihutu@gmail.com> * fix: cross-platform compatibility for cgroup detection - Move CHANGES_SUMMARY.md and FINAL_SUMMARY.md to docs/operations/ - Add platform-specific cgroup detection (Linux only) - Non-Linux platforms (macOS, Windows) fall back to host values - Add platform-specific tests for cgroup detection - Remove unused imports for non-Linux builds Co-Authored-By: heihutu <heihutu@gmail.com> * fix: clippy warnings for cgroup_resources - Remove unused import super::CgroupResources - Use derive(Default) instead of manual impl - Remove redundant trim() before split_whitespace() - Fix absurd_extreme_comparisons (quota <= 0 for u64) - Use div_ceil() instead of manual implementation Co-Authored-By: heihutu <heihutu@gmail.com> * refactor: consolidate cgroup detection into single module - Merge cgroup_resources.rs and container_config.rs into unified module - Remove duplicate test file cgroup_resources_test.rs - Remove redundant CHANGES_SUMMARY.md and FINAL_SUMMARY.md - Simplify memory_observability.rs to use unified API - Simplify server/runtime.rs to use unified API - All cgroup detection logic now in single source of truth - Environment variable overrides integrated into main module - Clippy and fmt clean Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
@@ -0,0 +1,203 @@
|
||||
# Container Resource Detection
|
||||
|
||||
RustFS automatically detects container resource limits (CPU and memory) from cgroup v1/v2. This ensures correct resource allocation and accurate metrics in containerized environments (Kubernetes, Docker, etc.).
|
||||
|
||||
## Problem
|
||||
|
||||
When RustFS runs in a container, the underlying system libraries report the **host's** total CPU cores and memory, not the container's limits. This leads to:
|
||||
|
||||
1. **Over-provisioned Tokio threads**: Too many worker and blocking threads
|
||||
2. **Incorrect memory metrics**: `rustfs_memory_usage_percent` shows host-based percentage
|
||||
3. **Memory budget errors**: Object data cache sized to host RAM instead of container limit
|
||||
4. **OOMKills**: Container exceeds its memory limit and gets killed
|
||||
|
||||
## Solution
|
||||
|
||||
RustFS now detects cgroup limits directly from the filesystem:
|
||||
|
||||
- **CPU**: `/sys/fs/cgroup/cpu.max` (v2) or `/sys/fs/cgroup/cpu/cpu.cfs_quota_us` (v1)
|
||||
- **Memory**: `/sys/fs/cgroup/memory.max` (v2) or `/sys/fs/cgroup/memory/memory.limit_in_bytes` (v1)
|
||||
|
||||
The effective resource limits are the **minimum** of host and cgroup values.
|
||||
|
||||
## Detection Logic
|
||||
|
||||
### CPU Detection
|
||||
|
||||
1. Read cgroup v2 `/sys/fs/cgroup/cpu.max`
|
||||
- Format: `"$QUOTA $PERIOD"` or `"max"` (unlimited)
|
||||
- Calculate: `cores = ceil(quota / period)`
|
||||
2. Fallback to cgroup v1 `/sys/fs/cgroup/cpu/cpu.cfs_quota_us`
|
||||
- Calculate: `cores = ceil(quota / period)`
|
||||
3. Fallback to host CPU count from `sysinfo`
|
||||
|
||||
### Memory Detection
|
||||
|
||||
1. Read cgroup v2 `/sys/fs/cgroup/memory.max`
|
||||
- Value in bytes or `"max"` (unlimited)
|
||||
2. Fallback to cgroup v1 `/sys/fs/cgroup/memory/memory.limit_in_bytes`
|
||||
- Very large values (≥2^62) indicate unlimited
|
||||
3. Fallback to host memory from `sysinfo`
|
||||
|
||||
## Environment Variables
|
||||
|
||||
### Disable Cgroup Detection
|
||||
|
||||
```bash
|
||||
RUSTFS_DISABLE_CGROUP_DETECTION=1
|
||||
```
|
||||
|
||||
Disables cgroup detection entirely. Useful for testing or when cgroup filesystem is not accessible.
|
||||
|
||||
### Override CPU Cores
|
||||
|
||||
```bash
|
||||
RUSTFS_OVERRIDE_CPU_CORES=4
|
||||
```
|
||||
|
||||
Overrides detected CPU cores. Takes precedence over cgroup detection.
|
||||
|
||||
### Override Memory Limit
|
||||
|
||||
```bash
|
||||
RUSTFS_OVERRIDE_MEMORY_BYTES=2147483648
|
||||
```
|
||||
|
||||
Overrides detected memory limit in bytes. Takes precedence over cgroup detection.
|
||||
|
||||
## Metrics
|
||||
|
||||
### New Metrics
|
||||
|
||||
| Metric | Description |
|
||||
|--------|-------------|
|
||||
| `rustfs_memory_effective_total_bytes` | Effective memory total (host or cgroup) |
|
||||
| `rustfs_cgroup_detected` | Whether cgroup limits were detected (1=yes, 0=no) |
|
||||
| `rustfs_cgroup_cpu_cores_limit` | Detected CPU cores limit |
|
||||
| `rustfs_cgroup_memory_limit_bytes` | Detected memory limit |
|
||||
|
||||
### Updated Metrics
|
||||
|
||||
| Metric | Change |
|
||||
|--------|--------|
|
||||
| `rustfs_memory_total_bytes` | Now uses effective memory (cgroup-aware) |
|
||||
| `rustfs_memory_usage_percent` | Now calculated against effective memory |
|
||||
|
||||
## Startup Logging
|
||||
|
||||
RustFS logs detected container resources at startup:
|
||||
|
||||
```
|
||||
INFO container resources (detected from cgroup) cpu_cores=2 memory_bytes=1073741824 memory_mib=1024
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```
|
||||
INFO container resources (overridden by environment variables) cpu_cores=4 memory_bytes=2147483648 memory_mib=2048
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Kubernetes with Resource Limits
|
||||
|
||||
```yaml
|
||||
resources:
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: "1Gi"
|
||||
requests:
|
||||
cpu: "500m"
|
||||
memory: "512Mi"
|
||||
```
|
||||
|
||||
RustFS will detect:
|
||||
- CPU cores: 2
|
||||
- Memory: 1 GiB (1073741824 bytes)
|
||||
|
||||
### Docker with CPU and Memory Limits
|
||||
|
||||
```bash
|
||||
docker run --cpus=2 --memory=1g rustfs/rustfs:latest
|
||||
```
|
||||
|
||||
RustFS will detect:
|
||||
- CPU cores: 2
|
||||
- Memory: 1 GiB
|
||||
|
||||
### Manual Override
|
||||
|
||||
```bash
|
||||
export RUSTFS_OVERRIDE_CPU_CORES=4
|
||||
export RUSTFS_OVERRIDE_MEMORY_BYTES=2147483648
|
||||
```
|
||||
|
||||
RustFS will use:
|
||||
- CPU cores: 4
|
||||
- Memory: 2 GiB
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Cgroup Detection Not Working
|
||||
|
||||
1. Check if cgroup filesystem is mounted:
|
||||
```bash
|
||||
ls -la /sys/fs/cgroup/
|
||||
```
|
||||
|
||||
2. Check cgroup version:
|
||||
```bash
|
||||
stat -fc %T /sys/fs/cgroup/
|
||||
```
|
||||
- `cgroup2fs` = cgroup v2
|
||||
- `tmpfs` = cgroup v1
|
||||
|
||||
3. Check if limits are set:
|
||||
```bash
|
||||
# cgroup v2
|
||||
cat /sys/fs/cgroup/cpu.max
|
||||
cat /sys/fs/cgroup/memory.max
|
||||
|
||||
# cgroup v1
|
||||
cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
|
||||
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
|
||||
```
|
||||
|
||||
### Metrics Show Host Values
|
||||
|
||||
If `rustfs_memory_effective_total_bytes` shows host memory instead of cgroup limit:
|
||||
|
||||
1. Verify cgroup detection is not disabled:
|
||||
```bash
|
||||
echo $RUSTFS_DISABLE_CGROUP_DETECTION
|
||||
```
|
||||
|
||||
2. Check startup logs for cgroup detection:
|
||||
```bash
|
||||
grep "container resources" /logs/rustfs.log
|
||||
```
|
||||
|
||||
3. Use environment variable override as workaround:
|
||||
```bash
|
||||
export RUSTFS_OVERRIDE_MEMORY_BYTES=1073741824
|
||||
```
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Files Modified
|
||||
|
||||
- `rustfs/src/cgroup_resources.rs` - Core cgroup detection logic
|
||||
- `rustfs/src/container_config.rs` - Container configuration with overrides
|
||||
- `rustfs/src/memory_observability.rs` - Updated memory metrics
|
||||
- `rustfs/src/server/runtime.rs` - Updated Tokio runtime configuration
|
||||
- `rustfs/src/startup_entrypoint.rs` - Startup logging
|
||||
|
||||
### Performance Impact
|
||||
|
||||
- **Startup**: One-time detection adds ~1ms overhead
|
||||
- **Runtime**: Cached values, no repeated filesystem reads
|
||||
- **Memory**: Negligible (<1KB for cached values)
|
||||
|
||||
### Thread Safety
|
||||
|
||||
All detection functions are thread-safe and use `OnceLock` for caching.
|
||||
Reference in New Issue
Block a user