gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-26 08:18:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore: remove skills and docs dir. (#1631)

Browse Source
This commit is contained in:
安正超
2026-01-27 22:43:22 +08:00
committed by GitHub
parent 26c0230e8f
commit fff175dcdd
167 changed files with 2 additions and 31691 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/COMPLETE_SUMMARY.md
-306
View File
@@ -1,306 +0,0 @@
# Adaptive Buffer Sizing - Complete Implementation Summary
## English Version
### Overview
This implementation provides a comprehensive adaptive buffer sizing optimization system for RustFS, enabling intelligent
buffer size selection based on file size and workload characteristics. The complete migration path (Phases 1-4) has been
successfully implemented with full backward compatibility.
### Key Features
#### 1. Workload Profile System
- **6 Predefined Profiles**: GeneralPurpose, AiTraining, DataAnalytics, WebWorkload, IndustrialIoT, SecureStorage
- **Custom Configuration Support**: Flexible buffer size configuration with validation
- **OS Environment Detection**: Automatic detection of secure Chinese OS environments (Kylin, NeoKylin, UOS, OpenKylin)
- **Thread-Safe Global Configuration**: Atomic flags and immutable configuration structures
#### 2. Intelligent Buffer Sizing
- **File Size Aware**: Automatically adjusts buffer sizes from 32KB to 4MB based on file size
- **Profile-Based Optimization**: Different buffer strategies for different workload types
- **Unknown Size Handling**: Special handling for streaming and chunked uploads
- **Performance Metrics**: Optional metrics collection via feature flag
#### 3. Integration Points
- **put_object**: Optimized buffer sizing for object uploads
- **put_object_extract**: Special handling for archive extraction
- **upload_part**: Multipart upload optimization
### Implementation Phases
#### Phase 1: Infrastructure (Completed)
- Created workload profile module (`rustfs/src/config/workload_profiles.rs`)
- Implemented core data structures (WorkloadProfile, BufferConfig, RustFSBufferConfig)
- Added configuration validation and testing framework
#### Phase 2: Opt-In Usage (Completed)
- Added global configuration management
- Implemented `RUSTFS_BUFFER_PROFILE_ENABLE` and `RUSTFS_BUFFER_PROFILE` configuration
- Integrated buffer sizing into core upload functions
- Maintained backward compatibility with legacy behavior
#### Phase 3: Default Enablement (Completed)
- Changed default to enabled with GeneralPurpose profile
- Replaced opt-in with opt-out mechanism (`--buffer-profile-disable`)
- Created comprehensive migration guide (MIGRATION_PHASE3.md)
- Ensured zero-impact migration for existing deployments
#### Phase 4: Full Integration (Completed)
- Unified profile-only implementation
- Removed hardcoded buffer values
- Added optional performance metrics collection
- Cleaned up deprecated code and improved documentation
### Technical Details
#### Buffer Size Ranges by Profile
| Profile | Min Buffer | Max Buffer | Optimal For |
|----------------|------------|------------|-------------------------------|
| GeneralPurpose | 64KB | 1MB | Mixed workloads |
| AiTraining | 512KB | 4MB | Large files, sequential I/O |
| DataAnalytics | 128KB | 2MB | Mixed read-write patterns |
| WebWorkload | 32KB | 256KB | Small files, high concurrency |
| IndustrialIoT | 64KB | 512KB | Real-time streaming |
| SecureStorage | 32KB | 256KB | Compliance environments |
#### Configuration Options
**Environment Variables:**
- `RUSTFS_BUFFER_PROFILE`: Select workload profile (default: GeneralPurpose)
- `RUSTFS_BUFFER_PROFILE_DISABLE`: Disable profiling (opt-out)
**Command-Line Flags:**
- `--buffer-profile <PROFILE>`: Set workload profile
- `--buffer-profile-disable`: Disable workload profiling
### Performance Impact
- **Default (GeneralPurpose)**: Same performance as original implementation
- **AiTraining**: Up to 4x throughput improvement for large files (>500MB)
- **WebWorkload**: Lower memory usage, better concurrency for small files
- **Metrics Collection**: < 1% CPU overhead when enabled
### Code Quality
- **30+ Unit Tests**: Comprehensive test coverage for all profiles and scenarios
- **1200+ Lines of Documentation**: Complete usage guides, migration guides, and API documentation
- **Thread-Safe Design**: Atomic flags, immutable configurations, zero data races
- **Memory Safe**: All configurations validated, bounded buffer sizes
### Files Changed
```
rustfs/src/config/mod.rs | 10 +
rustfs/src/config/workload_profiles.rs | 650 +++++++++++++++++
rustfs/src/storage/ecfs.rs | 200 ++++++
rustfs/src/main.rs | 40 ++
docs/adaptive-buffer-sizing.md | 550 ++++++++++++++
docs/IMPLEMENTATION_SUMMARY.md | 380 ++++++++++
docs/MIGRATION_PHASE3.md | 380 ++++++++++
docs/PHASE4_GUIDE.md | 425 +++++++++++
docs/README.md | 3 +
```
### Backward Compatibility
- ✅ Zero breaking changes
- ✅ Default behavior matches original implementation
- ✅ Opt-out mechanism available
- ✅ All existing tests pass
- ✅ No configuration required for migration
### Usage Examples
**Default (Recommended):**
```bash
./rustfs /data
```
**Custom Profile:**
```bash
export RUSTFS_BUFFER_PROFILE=AiTraining
./rustfs /data
```
**Opt-Out:**
```bash
export RUSTFS_BUFFER_PROFILE_DISABLE=true
./rustfs /data
```
**With Metrics:**
```bash
cargo build --features metrics --release
./target/release/rustfs /data
```
---
## 中文版本
### 概述
本实现为 RustFS 提供了全面的自适应缓冲区大小优化系统，能够根据文件大小和工作负载特性智能选择缓冲区大小。完整的迁移路径（阶段
1-4）已成功实现，完全向后兼容。
### 核心功能
#### 1. 工作负载配置文件系统
- **6 种预定义配置文件**：通用、AI 训练、数据分析、Web 工作负载、工业物联网、安全存储
- **自定义配置支持**：灵活的缓冲区大小配置和验证
- **操作系统环境检测**：自动检测中国安全操作系统环境（麒麟、中标麒麟、统信、开放麒麟）
- **线程安全的全局配置**：原子标志和不可变配置结构
#### 2. 智能缓冲区大小调整
- **文件大小感知**：根据文件大小自动调整 32KB 到 4MB 的缓冲区
- **基于配置文件的优化**：不同工作负载类型的不同缓冲区策略
- **未知大小处理**：流式传输和分块上传的特殊处理
- **性能指标**：通过功能标志可选的指标收集
#### 3. 集成点
- **put_object**：对象上传的优化缓冲区大小
- **put_object_extract**：存档提取的特殊处理
- **upload_part**：多部分上传优化
### 实现阶段
#### 阶段 1：基础设施（已完成）
- 创建工作负载配置文件模块（`rustfs/src/config/workload_profiles.rs`）
- 实现核心数据结构（WorkloadProfile、BufferConfig、RustFSBufferConfig）
- 添加配置验证和测试框架
#### 阶段 2：选择性启用（已完成）
- 添加全局配置管理
- 实现 `RUSTFS_BUFFER_PROFILE_ENABLE` 和 `RUSTFS_BUFFER_PROFILE` 配置
- 将缓冲区大小调整集成到核心上传函数中
- 保持与旧版行为的向后兼容性
#### 阶段 3：默认启用（已完成）
- 将默认值更改为使用通用配置文件启用
- 将选择性启用替换为选择性退出机制（`--buffer-profile-disable`）
- 创建全面的迁移指南（MIGRATION_PHASE3.md）
- 确保现有部署的零影响迁移
#### 阶段 4：完全集成（已完成）
- 统一的纯配置文件实现
- 移除硬编码的缓冲区值
- 添加可选的性能指标收集
- 清理弃用代码并改进文档
### 技术细节
#### 按配置文件划分的缓冲区大小范围
| 配置文件 | 最小缓冲 | 最大缓冲 | 最适合 |
|----------|-------|-------|------------|
| 通用 | 64KB | 1MB | 混合工作负载 |
| AI 训练 | 512KB | 4MB | 大文件、顺序 I/O |
| 数据分析 | 128KB | 2MB | 混合读写模式 |
| Web 工作负载 | 32KB | 256KB | 小文件、高并发 |
| 工业物联网 | 64KB | 512KB | 实时流式传输 |
| 安全存储 | 32KB | 256KB | 合规环境 |
#### 配置选项
**环境变量：**
- `RUSTFS_BUFFER_PROFILE`：选择工作负载配置文件（默认：通用）
- `RUSTFS_BUFFER_PROFILE_DISABLE`：禁用配置文件（选择性退出）
**命令行标志：**
- `--buffer-profile <配置文件>`：设置工作负载配置文件
- `--buffer-profile-disable`：禁用工作负载配置文件
### 性能影响
- **默认（通用）**：与原始实现性能相同
- **AI 训练**：大文件（>500MB）吞吐量提升最多 4 倍
- **Web 工作负载**：小文件的内存使用更低、并发性更好
- **指标收集**：启用时 CPU 开销 < 1%
### 代码质量
- **30+ 单元测试**：全面覆盖所有配置文件和场景
- **1200+ 行文档**：完整的使用指南、迁移指南和 API 文档
- **线程安全设计**：原子标志、不可变配置、零数据竞争
- **内存安全**：所有配置经过验证、缓冲区大小有界
### 文件变更
```
rustfs/src/config/mod.rs | 10 +
rustfs/src/config/workload_profiles.rs | 650 +++++++++++++++++
rustfs/src/storage/ecfs.rs | 200 ++++++
rustfs/src/main.rs | 40 ++
docs/adaptive-buffer-sizing.md | 550 ++++++++++++++
docs/IMPLEMENTATION_SUMMARY.md | 380 ++++++++++
docs/MIGRATION_PHASE3.md | 380 ++++++++++
docs/PHASE4_GUIDE.md | 425 +++++++++++
docs/README.md | 3 +
```
### 向后兼容性
- ✅ 零破坏性更改
- ✅ 默认行为与原始实现匹配
- ✅ 提供选择性退出机制
- ✅ 所有现有测试通过
- ✅ 迁移无需配置
### 使用示例
**默认（推荐）：**
```bash
./rustfs /data
```
**自定义配置文件：**
```bash
export RUSTFS_BUFFER_PROFILE=AiTraining
./rustfs /data
```
**选择性退出：**
```bash
export RUSTFS_BUFFER_PROFILE_DISABLE=true
./rustfs /data
```
**启用指标：**
```bash
cargo build --features metrics --release
./target/release/rustfs /data
```
### 总结
本实现为 RustFS 提供了企业级的自适应缓冲区优化能力，通过完整的四阶段迁移路径实现了从基础设施到完全集成的平滑过渡。系统默认启用，完全向后兼容，并提供了强大的工作负载优化功能，使不同场景下的性能得到显著提升。
完整的文档、全面的测试覆盖和生产就绪的实现确保了系统的可靠性和可维护性。通过可选的性能指标收集，运维团队可以持续监控和优化缓冲区配置，实现数据驱动的性能调优。
docs/CONCURRENCY_ARCHITECTURE.md
-601
View File
@@ -1,601 +0,0 @@
# Concurrent GetObject Performance Optimization - Complete Architecture Design
## Executive Summary
This document provides a comprehensive architectural analysis of the concurrent GetObject performance optimization implemented in RustFS. The solution addresses Issue #911 where concurrent GetObject latency degraded exponentially (59ms → 110ms → 200ms for 1→2→4 requests).
## Table of Contents
1. [Problem Statement](#problem-statement)
2. [Architecture Overview](#architecture-overview)
3. [Module Analysis: concurrency.rs](#module-analysis-concurrencyrs)
4. [Module Analysis: ecfs.rs](#module-analysis-ecfsrs)
5. [Critical Analysis: helper.complete() for Cache Hits](#critical-analysis-helpercomplete-for-cache-hits)
6. [Adaptive I/O Strategy Design](#adaptive-io-strategy-design)
7. [Cache Architecture](#cache-architecture)
8. [Metrics and Monitoring](#metrics-and-monitoring)
9. [Performance Characteristics](#performance-characteristics)
10. [Future Enhancements](#future-enhancements)
---
## Problem Statement
### Original Issue (#911)
Users observed exponential latency degradation under concurrent load:
| Concurrent Requests | Observed Latency | Expected Latency |
|---------------------|------------------|------------------|
| 1 | 59ms | ~60ms |
| 2 | 110ms | ~60ms |
| 4 | 200ms | ~60ms |
| 8 | 400ms+ | ~60ms |
### Root Causes Identified
1. **Fixed Buffer Sizes**: 1MB buffers for all requests caused memory contention
2. **No I/O Rate Limiting**: Unlimited concurrent disk reads saturated I/O queues
3. **No Object Caching**: Repeated reads of same objects hit disk every time
4. **Lock Contention**: RwLock-based caching (if any) created bottlenecks
---
## Architecture Overview
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ GetObject Request Flow │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ 1. Request Tracking (GetObjectGuard - RAII) │
│ - Atomic increment of ACTIVE_GET_REQUESTS │
│ - Start time capture for latency metrics │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ 2. OperationHelper Initialization │
│ - Event: ObjectAccessedGet / s3:GetObject │
│ - Used for S3 bucket notifications │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ 3. Cache Lookup (if enabled) │
│ - Key: "{bucket}/{key}" or "{bucket}/{key}?versionId={vid}" │
│ - Conditions: cache_enabled && !part_number && !range │
│ - On HIT: Return immediately with CachedGetObject │
│ - On MISS: Continue to storage backend │
└─────────────────────────────────────────────────────────────────────────────┘
│
┌───────────────┴───────────────┐
│ │
Cache HIT Cache MISS
│ │
▼ ▼
┌──────────────────────────────┐ ┌───────────────────────────────────────────┐
│ Return CachedGetObject │ │ 4. Adaptive I/O Strategy │
│ - Parse last_modified │ │ - Acquire disk_permit (semaphore) │
│ - Construct GetObjectOutput │ │ - Calculate IoStrategy from wait time │
│ - ** CALL helper.complete **│ │ - Select buffer_size, readahead, etc. │
│ - Return S3Response │ │ │
└──────────────────────────────┘ └───────────────────────────────────────────┘
│
▼
┌───────────────────────────────────────────┐
│ 5. Storage Backend Read │
│ - Get object info (metadata) │
│ - Validate conditions (ETag, etc.) │
│ - Stream object data │
└───────────────────────────────────────────┘
│
▼
┌───────────────────────────────────────────┐
│ 6. Cache Writeback (if eligible) │
│ - Conditions: size <= 10MB, no enc. │
│ - Background: tokio::spawn() │
│ - Store: CachedGetObject with metadata│
└───────────────────────────────────────────┘
│
▼
┌───────────────────────────────────────────┐
│ 7. Response Construction │
│ - Build GetObjectOutput │
│ - Call helper.complete(&result) │
│ - Return S3Response │
└───────────────────────────────────────────┘
```
---
## Module Analysis: concurrency.rs
### Purpose
The `concurrency.rs` module provides intelligent concurrency management to prevent performance degradation under high concurrent load. It implements:
1. **Request Tracking**: Atomic counters for active requests
2. **Adaptive Buffer Sizing**: Dynamic buffer allocation based on load
3. **Moka Cache Integration**: Lock-free object caching
4. **Adaptive I/O Strategy**: Load-aware I/O parameter selection
5. **Disk I/O Rate Limiting**: Semaphore-based throttling
### Key Components
#### 1. IoLoadLevel Enum
```rust
pub enum IoLoadLevel {
Low, // < 10ms wait - ample I/O capacity
Medium, // 10-50ms wait - moderate load
High, // 50-200ms wait - significant load
Critical, // > 200ms wait - severe congestion
}
```
**Design Rationale**: These thresholds are calibrated for NVMe SSD characteristics. Adjustments may be needed for HDD or cloud storage.
#### 2. IoStrategy Struct
```rust
pub struct IoStrategy {
pub buffer_size: usize, // Calculated buffer size (32KB-1MB)
pub buffer_multiplier: f64, // 0.4 - 1.0 of base buffer
pub enable_readahead: bool, // Disabled under high load
pub cache_writeback_enabled: bool, // Disabled under critical load
pub use_buffered_io: bool, // Always enabled
pub load_level: IoLoadLevel,
pub permit_wait_duration: Duration,
}
```
**Strategy Selection Matrix**:
| Load Level | Buffer Mult | Readahead | Cache WB | Rationale |
|------------|-------------|-----------|----------|-----------|
| Low | 1.0 (100%) | ✓ Yes | ✓ Yes | Maximize throughput |
| Medium | 0.75 (75%) | ✓ Yes | ✓ Yes | Balance throughput/fairness |
| High | 0.5 (50%) | ✗ No | ✓ Yes | Reduce I/O amplification |
| Critical | 0.4 (40%) | ✗ No | ✗ No | Prevent memory exhaustion |
#### 3. IoLoadMetrics
Rolling window statistics for load tracking:
- `average_wait()`: Smoothed average for stable decisions
- `p95_wait()`: Tail latency indicator
- `max_wait()`: Peak contention detection
#### 4. GetObjectGuard (RAII)
Automatic request lifecycle management:
```rust
impl Drop for GetObjectGuard {
fn drop(&mut self) {
ACTIVE_GET_REQUESTS.fetch_sub(1, Ordering::Relaxed);
// Record metrics...
}
}
```
**Guarantees**:
- Counter always decremented, even on panic
- Request duration always recorded
- No resource leaks
#### 5. ConcurrencyManager
Central coordination point:
```rust
pub struct ConcurrencyManager {
pub cache: HotObjectCache, // Moka-based object cache
disk_permit: Semaphore, // I/O rate limiter
cache_enabled: bool, // Feature flag
io_load_metrics: Mutex<IoLoadMetrics>, // Load tracking
}
```
**Key Methods**:
| Method | Purpose |
|--------|---------|
| `track_request()` | Create RAII guard for request tracking |
| `acquire_disk_read_permit()` | Rate-limited disk access |
| `calculate_io_strategy()` | Compute adaptive I/O parameters |
| `get_cached_object()` | Lock-free cache lookup |
| `put_cached_object()` | Background cache writeback |
| `invalidate_cache()` | Cache invalidation on writes |
---
## Module Analysis: ecfs.rs
### get_object Implementation
The `get_object` function is the primary focus of optimization. Key integration points:
#### Line ~1678: OperationHelper Initialization
```rust
let mut helper = OperationHelper::new(&req, EventName::ObjectAccessedGet, "s3:GetObject");
```
**Purpose**: Prepares S3 bucket notification event. The `complete()` method MUST be called before returning to trigger notifications.
#### Lines ~1694-1756: Cache Lookup
```rust
if manager.is_cache_enabled() && part_number.is_none() && range.is_none() {
if let Some(cached) = manager.get_cached_object(&cache_key).await {
// Build response from cache
return Ok(S3Response::new(output)); // <-- ISSUE: helper.complete() NOT called!
}
}
```
**CRITICAL ISSUE IDENTIFIED**: The current cache hit path does NOT call `helper.complete(&result)`, which means S3 bucket notifications are NOT triggered for cache hits.
#### Lines ~1800-1830: Adaptive I/O Strategy
```rust
let permit_wait_start = std::time::Instant::now();
let _disk_permit = manager.acquire_disk_read_permit().await;
let permit_wait_duration = permit_wait_start.elapsed();
// Calculate adaptive I/O strategy from permit wait time
let io_strategy = manager.calculate_io_strategy(permit_wait_duration, base_buffer_size);
// Record metrics
#[cfg(feature = "metrics")]
{
histogram!("rustfs.disk.permit.wait.duration.seconds").record(...);
gauge!("rustfs.io.load.level").set(io_strategy.load_level as f64);
gauge!("rustfs.io.buffer.multiplier").set(io_strategy.buffer_multiplier);
}
```
#### Lines ~2100-2150: Cache Writeback
```rust
if should_cache && io_strategy.cache_writeback_enabled {
// Read stream into memory
// Background cache via tokio::spawn()
// Serve from InMemoryAsyncReader
}
```
#### Line ~2273: Final Response
```rust
let result = Ok(S3Response::new(output));
let _ = helper.complete(&result); // <-- Correctly called for cache miss path
result
```
---
## Critical Analysis: helper.complete() for Cache Hits
### Problem
When serving from cache, the current implementation returns early WITHOUT calling `helper.complete(&result)`. This has the following consequences:
1. **Missing S3 Bucket Notifications**: `s3:GetObject` events are NOT sent
2. **Incomplete Audit Trail**: Object access events are not logged
3. **Event-Driven Workflows Break**: Lambda triggers, SNS notifications fail
### Solution
The cache hit path MUST properly configure the helper with object info and version_id, then call `helper.complete(&result)` before returning:
```rust
if manager.is_cache_enabled() && part_number.is_none() && range.is_none() {
if let Some(cached) = manager.get_cached_object(&cache_key).await {
// ... build response output ...
// CRITICAL: Build ObjectInfo for event notification
let event_info = ObjectInfo {
bucket: bucket.clone(),
name: key.clone(),
storage_class: cached.storage_class.clone(),
mod_time: cached.last_modified.as_ref().and_then(|s| {
time::OffsetDateTime::parse(s, &Rfc3339).ok()
}),
size: cached.content_length,
actual_size: cached.content_length,
is_dir: false,
user_defined: cached.user_metadata.clone(),
version_id: cached.version_id.as_ref().and_then(|v| Uuid::parse_str(v).ok()),
delete_marker: cached.delete_marker,
content_type: cached.content_type.clone(),
content_encoding: cached.content_encoding.clone(),
etag: cached.e_tag.clone(),
..Default::default()
};
// Set object info and version_id on helper for proper event notification
let version_id_str = req.input.version_id.clone().unwrap_or_default();
helper = helper.object(event_info).version_id(version_id_str);
let result = Ok(S3Response::new(output));
// Trigger S3 bucket notification event
let _ = helper.complete(&result);
return result;
}
}
```
### Key Points for Proper Event Notification
1. **ObjectInfo Construction**: The `event_info` must be built from cached metadata to provide:
- `bucket` and `name` (key) for object identification
- `size` and `actual_size` for event payload
- `etag` for integrity verification
- `version_id` for versioned object access
- `storage_class`, `content_type`, and other metadata
2. **helper.object(event_info)**: Sets the object information for the notification event. This ensures:
- Lambda triggers receive proper object metadata
- SNS/SQS notifications include complete information
- Audit logs contain accurate object details
3. **helper.version_id(version_id_str)**: Sets the version ID for versioned bucket access:
- Enables version-specific event routing
- Supports versioned object lifecycle policies
- Provides complete audit trail for versioned access
4. **Performance**: The `helper.complete()` call may involve async I/O (SQS, SNS). Consider:
- Fire-and-forget with `tokio::spawn()` for minimal latency impact
- Accept slight latency increase for correctness
5. **Metrics Alignment**: Ensure cache hit metrics don't double-count
```
---
## Adaptive I/O Strategy Design
### Goal
Automatically tune I/O parameters based on observed system load to prevent:
- Memory exhaustion under high concurrency
- I/O queue saturation
- Latency spikes
- Unfair resource distribution
### Algorithm
```
1. ACQUIRE disk_permit from semaphore
2. MEASURE wait_duration = time spent waiting for permit
3. CLASSIFY load_level from wait_duration:
- Low: wait < 10ms
- Medium: 10ms <= wait < 50ms
- High: 50ms <= wait < 200ms
- Critical: wait >= 200ms
4. CALCULATE strategy based on load_level:
- buffer_multiplier: 1.0 / 0.75 / 0.5 / 0.4
- enable_readahead: true / true / false / false
- cache_writeback: true / true / true / false
5. APPLY strategy to I/O operations
6. RECORD metrics for monitoring
```
### Feedback Loop
```
┌──────────────────────────┐
│ IoLoadMetrics │
│ (rolling window) │
└──────────────────────────┘
▲
│ record_permit_wait()
│
┌───────────────────┐ ┌─────────────┐ ┌─────────────────────┐
│ Disk Permit Wait │──▶│ IoStrategy │──▶│ Buffer Size, etc. │
│ (observed latency)│ │ Calculation │ │ (applied to I/O) │
└───────────────────┘ └─────────────┘ └─────────────────────┘
│
▼
┌──────────────────────────┐
│ Prometheus Metrics │
│ - io.load.level │
│ - io.buffer.multiplier │
└──────────────────────────┘
```
---
## Cache Architecture
### HotObjectCache (Moka-based)
```rust
pub struct HotObjectCache {
bytes_cache: Cache<String, Arc<CachedObjectData>>, // Legacy byte cache
response_cache: Cache<String, Arc<CachedGetObject>>, // Full response cache
}
```
### CachedGetObject Structure
```rust
pub struct CachedGetObject {
pub body: bytes::Bytes, // Object data
pub content_length: i64, // Size in bytes
pub content_type: Option<String>, // MIME type
pub e_tag: Option<String>, // Entity tag
pub last_modified: Option<String>, // RFC3339 timestamp
pub expires: Option<String>, // Expiration
pub cache_control: Option<String>, // Cache-Control header
pub content_disposition: Option<String>,
pub content_encoding: Option<String>,
pub content_language: Option<String>,
pub storage_class: Option<String>,
pub version_id: Option<String>, // Version support
pub delete_marker: bool,
pub tag_count: Option<i32>,
pub replication_status: Option<String>,
pub user_metadata: HashMap<String, String>,
}
```
### Cache Key Strategy
| Scenario | Key Format |
|----------|------------|
| Latest version | `"{bucket}/{key}"` |
| Specific version | `"{bucket}/{key}?versionId={vid}"` |
### Cache Invalidation
Invalidation is triggered on all write operations:
| Operation | Invalidation Target |
|-----------|---------------------|
| `put_object` | Latest + specific version |
| `copy_object` | Destination object |
| `delete_object` | Deleted object |
| `delete_objects` | Each deleted object |
| `complete_multipart_upload` | Completed object |
---
## Metrics and Monitoring
### Request Metrics
| Metric | Type | Description |
|--------|------|-------------|
| `rustfs.get.object.requests.total` | Counter | Total GetObject requests |
| `rustfs.get.object.requests.completed` | Counter | Completed requests |
| `rustfs.get.object.duration.seconds` | Histogram | Request latency |
| `rustfs.concurrent.get.requests` | Gauge | Current concurrent requests |
### Cache Metrics
| Metric | Type | Description |
|--------|------|-------------|
| `rustfs.object.cache.hits` | Counter | Cache hits |
| `rustfs.object.cache.misses` | Counter | Cache misses |
| `rustfs.get.object.cache.served.total` | Counter | Requests served from cache |
| `rustfs.get.object.cache.serve.duration.seconds` | Histogram | Cache serve latency |
| `rustfs.object.cache.writeback.total` | Counter | Cache writeback operations |
### I/O Metrics
| Metric | Type | Description |
|--------|------|-------------|
| `rustfs.disk.permit.wait.duration.seconds` | Histogram | Disk permit wait time |
| `rustfs.io.load.level` | Gauge | Current I/O load level (0-3) |
| `rustfs.io.buffer.multiplier` | Gauge | Current buffer multiplier |
| `rustfs.io.strategy.selected` | Counter | Strategy selections by level |
### Prometheus Queries
```promql
# Cache hit rate
sum(rate(rustfs_object_cache_hits[5m])) /
(sum(rate(rustfs_object_cache_hits[5m])) + sum(rate(rustfs_object_cache_misses[5m])))
# P95 GetObject latency
histogram_quantile(0.95, rate(rustfs_get_object_duration_seconds_bucket[5m]))
# Average disk permit wait
rate(rustfs_disk_permit_wait_duration_seconds_sum[5m]) /
rate(rustfs_disk_permit_wait_duration_seconds_count[5m])
# I/O load level distribution
sum(rate(rustfs_io_strategy_selected_total[5m])) by (level)
```
---
## Performance Characteristics
### Expected Improvements
| Concurrent Requests | Before | After (Cache Miss) | After (Cache Hit) |
|---------------------|--------|--------------------|--------------------|
| 1 | 59ms | ~55ms | < 5ms |
| 2 | 110ms | 60-70ms | < 5ms |
| 4 | 200ms | 75-90ms | < 5ms |
| 8 | 400ms | 90-120ms | < 5ms |
| 16 | 800ms | 110-145ms | < 5ms |
### Resource Usage
| Resource | Impact |
|----------|--------|
| Memory | Reduced under high load via adaptive buffers |
| CPU | Slight increase for strategy calculation |
| Disk I/O | Smoothed via semaphore limiting |
| Cache | 100MB default, automatic eviction |
---
## Future Enhancements
### 1. Dynamic Semaphore Sizing
Automatically adjust disk permit count based on observed throughput:
```rust
if avg_wait > 100ms && current_permits > MIN_PERMITS {
reduce_permits();
} else if avg_wait < 10ms && throughput < MAX_THROUGHPUT {
increase_permits();
}
```
### 2. Predictive Caching
Analyze access patterns to pre-warm cache:
- Track frequently accessed objects
- Prefetch predicted objects during idle periods
### 3. Tiered Caching
Implement multi-tier cache hierarchy:
- L1: Process memory (current Moka cache)
- L2: Redis cluster (shared across instances)
- L3: Local SSD cache (persistent across restarts)
### 4. Request Priority
Implement priority queuing for latency-sensitive requests:
```rust
pub enum RequestPriority {
RealTime, // < 10ms SLA
Standard, // < 100ms SLA
Batch, // Best effort
}
```
---
## Conclusion
The concurrent GetObject optimization architecture provides a comprehensive solution to the exponential latency degradation issue. Key components work together:
1. **Request Tracking** (GetObjectGuard) ensures accurate concurrency measurement
2. **Adaptive I/O Strategy** prevents system overload under high concurrency
3. **Moka Cache** provides sub-5ms response times for hot objects
4. **Disk Permit Semaphore** prevents I/O queue saturation
5. **Comprehensive Metrics** enable observability and tuning
**Critical Fix Required**: The cache hit path must call `helper.complete(&result)` to ensure S3 bucket notifications are triggered for all object access events.
---
## Document Information
- **Version**: 1.0
- **Created**: 2025-11-29
- **Author**: RustFS Team
- **Related Issues**: #911
- **Status**: Implemented and Verified
docs/CONCURRENT_GETOBJECT_IMPLEMENTATION_SUMMARY.md
-465
View File
@@ -1,465 +0,0 @@
# Concurrent GetObject Performance Optimization - Implementation Summary
## Executive Summary
Successfully implemented a comprehensive solution to address exponential performance degradation in concurrent GetObject requests. The implementation includes three key optimizations that work together to significantly improve performance under concurrent load while maintaining backward compatibility.
## Problem Statement
### Observed Behavior
| Concurrent Requests | Latency per Request | Performance Degradation |
|---------------------|---------------------|------------------------|
| 1 | 59ms | Baseline |
| 2 | 110ms | 1.9x slower |
| 4 | 200ms | 3.4x slower |
### Root Causes Identified
1. **Fixed buffer sizing** regardless of concurrent load led to memory contention
2. **No I/O concurrency control** caused disk saturation
3. **No caching** resulted in redundant disk reads for hot objects
4. **Lack of fairness** allowed large requests to starve smaller ones
## Solution Architecture
### 1. Concurrency-Aware Adaptive Buffer Sizing
#### Implementation
```rust
pub fn get_concurrency_aware_buffer_size(file_size: i64, base_buffer_size: usize) -> usize {
let concurrent_requests = ACTIVE_GET_REQUESTS.load(Ordering::Relaxed);
let adaptive_multiplier = match concurrent_requests {
0..=2 => 1.0, // Low: 100% buffer
3..=4 => 0.75, // Medium: 75% buffer
5..=8 => 0.5, // High: 50% buffer
_ => 0.4, // Very high: 40% buffer
};
(base_buffer_size as f64 * adaptive_multiplier) as usize
.clamp(min_buffer, max_buffer)
}
```
#### Benefits
- **Reduced memory pressure**: Smaller buffers under high concurrency
- **Better cache utilization**: More data fits in CPU cache
- **Improved fairness**: Prevents large requests from monopolizing resources
- **Automatic adaptation**: No manual tuning required
#### Metrics
- `rustfs_concurrent_get_requests`: Tracks active request count
- `rustfs_buffer_size_bytes`: Histogram of buffer sizes used
### 2. Hot Object Caching (LRU)
#### Implementation
```rust
struct HotObjectCache {
max_object_size: 10 * MI_B, // 10MB limit per object
max_cache_size: 100 * MI_B, // 100MB total capacity
cache: RwLock<lru::LruCache<String, Arc<CachedObject>>>,
}
```
#### Features
- **LRU eviction policy**: Automatic management of cache memory
- **Eligibility filtering**: Only small (<= 10MB), complete objects cached
- **Atomic size tracking**: Thread-safe cache size management
- **Read-optimized**: RwLock allows concurrent reads
#### Current Limitations
- **Cache insertion not yet implemented**: Framework exists but streaming cache insertion requires TeeReader implementation
- **Cache can be populated manually**: Via admin API or background processes
- **Cache lookup functional**: Objects in cache will be served from memory
#### Benefits (once fully implemented)
- **Eliminates disk I/O**: Memory access is 100-1000x faster
- **Reduces contention**: Cached objects don't compete for disk I/O permits
- **Improves scalability**: Cache hit ratio increases with concurrent load
#### Metrics
- `rustfs_object_cache_hits`: Count of successful cache lookups
- `rustfs_object_cache_misses`: Count of cache misses
- `rustfs_object_cache_size_bytes`: Current cache memory usage
- `rustfs_object_cache_insertions`: Count of cache additions
### 3. I/O Concurrency Control
#### Implementation
```rust
struct ConcurrencyManager {
disk_read_semaphore: Arc<Semaphore>, // 64 permits
}
// In get_object:
let _permit = manager.acquire_disk_read_permit().await;
// Permit automatically released when dropped
```
#### Benefits
- **Prevents I/O saturation**: Limits queue depth to optimal level (64)
- **Predictable latency**: Avoids exponential increase under extreme load
- **Fair queuing**: FIFO order for disk access
- **Graceful degradation**: Queues requests instead of thrashing
#### Tuning
The default of 64 concurrent disk reads is suitable for most scenarios:
- **SSD/NVMe**: Can handle higher queue depths efficiently
- **HDD**: May benefit from lower values (32-48) to reduce seeks
- **Network storage**: Depends on network bandwidth and latency
### 4. Request Tracking (RAII)
#### Implementation
```rust
pub struct GetObjectGuard {
start_time: Instant,
}
impl Drop for GetObjectGuard {
fn drop(&mut self) {
ACTIVE_GET_REQUESTS.fetch_sub(1, Ordering::Relaxed);
// Record metrics
}
}
// Usage:
let _guard = ConcurrencyManager::track_request();
// Automatically decrements counter on drop
```
#### Benefits
- **Zero overhead**: Tracking happens automatically
- **Leak-proof**: Counter always decremented, even on panics
- **Accurate metrics**: Reflects actual concurrent load
- **Duration tracking**: Captures request completion time
## Integration Points
### GetObject Handler
```rust
async fn get_object(&self, req: S3Request<GetObjectInput>) -> S3Result<S3Response<GetObjectOutput>> {
// 1. Track request (RAII guard)
let _request_guard = ConcurrencyManager::track_request();
// 2. Try cache lookup (fast path)
if let Some(cached_data) = manager.get_cached(&cache_key).await {
return serve_from_cache(cached_data);
}
// 3. Acquire I/O permit (rate limiting)
let _disk_permit = manager.acquire_disk_read_permit().await;
// 4. Read from storage with optimal buffer
let optimal_buffer_size = get_concurrency_aware_buffer_size(
response_content_length,
base_buffer_size
);
// 5. Stream response
let body = StreamingBlob::wrap(
ReaderStream::with_capacity(final_stream, optimal_buffer_size)
);
Ok(S3Response::new(output))
}
```
### Workload Profile Integration
The solution integrates with the existing workload profile system:
```rust
let base_buffer_size = get_buffer_size_opt_in(file_size);
let optimal_buffer_size = get_concurrency_aware_buffer_size(file_size, base_buffer_size);
```
This two-stage approach provides:
1. **Workload-specific sizing**: Based on file size and workload type
2. **Concurrency adaptation**: Further adjusted for current load
## Testing
### Test Coverage
#### Unit Tests (in concurrency.rs)
- `test_concurrent_request_tracking`: RAII guard functionality
- `test_adaptive_buffer_sizing`: Buffer size calculation
- `test_hot_object_cache`: Cache operations
- `test_cache_eviction`: LRU eviction behavior
- `test_concurrency_manager_creation`: Initialization
- `test_disk_read_permits`: Semaphore behavior
#### Integration Tests (in concurrent_get_object_test.rs)
- `test_concurrent_request_tracking`: End-to-end tracking
- `test_adaptive_buffer_sizing`: Multi-level concurrency
- `test_buffer_size_bounds`: Boundary conditions
- `bench_concurrent_requests`: Performance benchmarking
- `test_disk_io_permits`: Permit acquisition
- `test_cache_operations`: Cache lifecycle
- `test_large_object_not_cached`: Size filtering
- `test_cache_eviction`: Memory pressure handling
### Running Tests
```bash
# Run all tests
cargo test --test concurrent_get_object_test
# Run specific test
cargo test --test concurrent_get_object_test test_adaptive_buffer_sizing
# Run with output
cargo test --test concurrent_get_object_test -- --nocapture
```
### Performance Validation
To validate the improvements in a real environment:
```bash
# 1. Create test object (32MB)
dd if=/dev/random of=test.bin bs=1M count=32
mc cp test.bin rustfs/test/bxx
# 2. Run concurrent load test (Go client from issue)
for concurrency in 1 2 4 8 16; do
echo "Testing concurrency: $concurrency"
# Run your Go test client with this concurrency level
# Record average latency
done
# 3. Monitor metrics
curl http://localhost:9000/metrics | grep rustfs_get_object
```
## Expected Performance Improvements
### Latency Improvements
| Concurrent Requests | Before | After (Expected) | Improvement |
|---------------------|--------|------------------|-------------|
| 1 | 59ms | 55-60ms | Baseline |
| 2 | 110ms | 65-75ms | ~40% faster |
| 4 | 200ms | 80-100ms | ~50% faster |
| 8 | 400ms | 100-130ms | ~65% faster |
| 16 | 800ms | 120-160ms | ~75% faster |
### Scaling Characteristics
- **Sub-linear latency growth**: Latency increases at < O(n)
- **Bounded maximum latency**: Upper bound even under extreme load
- **Fair resource allocation**: All requests make progress
- **Predictable behavior**: Consistent performance across load levels
## Monitoring and Observability
### Key Metrics
#### Request Metrics
```promql
# P95 latency
histogram_quantile(0.95,
rate(rustfs_get_object_duration_seconds_bucket[5m])
)
# Concurrent request count
rustfs_concurrent_get_requests
# Request rate
rate(rustfs_get_object_requests_completed[5m])
```
#### Cache Metrics
```promql
# Cache hit ratio
sum(rate(rustfs_object_cache_hits[5m]))
/
(sum(rate(rustfs_object_cache_hits[5m])) + sum(rate(rustfs_object_cache_misses[5m])))
# Cache memory usage
rustfs_object_cache_size_bytes
# Cache entries
rustfs_object_cache_entries
```
#### Buffer Metrics
```promql
# Average buffer size
avg(rustfs_buffer_size_bytes)
# Buffer size distribution
histogram_quantile(0.95, rustfs_buffer_size_bytes_bucket)
```
### Dashboards
Recommended Grafana panels:
1. **Request Latency**: P50, P95, P99 over time
2. **Concurrency Level**: Active requests gauge
3. **Cache Performance**: Hit ratio and memory usage
4. **Buffer Sizing**: Distribution and adaptation
5. **I/O Permits**: Available vs. in-use permits
## Code Quality
### Review Findings and Fixes
All code review issues have been addressed:
1. **✅ Race condition in cache size tracking**
- Fixed by using consistent atomic operations within write lock
2. **✅ Incorrect buffer sizing thresholds**
- Corrected: 1-2 (100%), 3-4 (75%), 5-8 (50%), >8 (40%)
3. **✅ Unhelpful error message**
- Improved semaphore acquire failure message
4. **✅ Incomplete cache implementation**
- Documented limitation and added detailed TODO
### Security Considerations
- **No new attack surface**: Only internal optimizations
- **Resource limits enforced**: Cache size and I/O permits bounded
- **No data exposure**: Cache respects existing access controls
- **Thread-safe**: All shared state properly synchronized
### Memory Safety
- **No unsafe code**: Pure safe Rust
- **RAII for cleanup**: Guards ensure resource cleanup
- **Bounded memory**: Cache size limited to 100MB
- **No memory leaks**: All resources automatically dropped
## Deployment Considerations
### Configuration
Default values are production-ready but can be tuned:
```rust
// In concurrency.rs
const HIGH_CONCURRENCY_THRESHOLD: usize = 8;
const MEDIUM_CONCURRENCY_THRESHOLD: usize = 4;
// Cache settings
max_object_size: 10 * MI_B, // 10MB per object
max_cache_size: 100 * MI_B, // 100MB total
disk_read_semaphore: Semaphore::new(64), // 64 concurrent reads
```
### Rollout Strategy
1. **Phase 1**: Deploy with monitoring (current state)
- All optimizations active
- Collect baseline metrics
2. **Phase 2**: Validate performance improvements
- Compare metrics before/after
- Adjust thresholds if needed
3. **Phase 3**: Implement streaming cache (future)
- Add TeeReader for cache insertion
- Enable automatic cache population
### Rollback Plan
If issues arise:
1. No code changes needed - optimizations degrade gracefully
2. Monitor for any unexpected behavior
3. File size limits prevent memory exhaustion
4. I/O semaphore prevents disk saturation
## Future Enhancements
### Short Term (Next Sprint)
1. **Implement Streaming Cache**
```rust
// Potential approach with TeeReader
let (cache_sink, response_stream) = tee_reader(original_stream);
tokio::spawn(async move {
let data = read_all(cache_sink).await?;
manager.cache_object(key, data).await;
});
return response_stream;
```
2. **Add Admin API for Cache Management**
- Cache statistics endpoint
- Manual cache invalidation
- Pre-warming capability
### Medium Term
1. **Request Prioritization**
- Small files get priority
- Age-based queuing to prevent starvation
- QoS classes per tenant
2. **Advanced Caching**
- Partial object caching (hot blocks)
- Predictive prefetching
- Distributed cache across nodes
3. **I/O Scheduling**
- Batch similar requests for sequential I/O
- Deadline-based scheduling
- NUMA-aware buffer allocation
### Long Term
1. **ML-Based Optimization**
- Learn access patterns
- Predict hot objects
- Adaptive threshold tuning
2. **Compression**
- Transparent cache compression
- CPU-aware compression level
- Deduplication for similar objects
## Success Criteria
### Quantitative Metrics
- ✅ **Latency reduction**: 40-75% improvement under concurrent load
- ✅ **Memory efficiency**: Sub-linear growth with concurrency
- ✅ **I/O optimization**: Bounded queue depth
- 🔄 **Cache hit ratio**: >70% for hot objects (once implemented)
### Qualitative Goals
- ✅ **Maintainability**: Clear, well-documented code
- ✅ **Reliability**: No crashes or resource leaks
- ✅ **Observability**: Comprehensive metrics
- ✅ **Compatibility**: No breaking changes
## Conclusion
This implementation successfully addresses the concurrent GetObject performance issue through three complementary optimizations:
1. **Adaptive buffer sizing** eliminates memory contention
2. **I/O concurrency control** prevents disk saturation
3. **Hot object caching** framework reduces redundant disk I/O (full implementation pending)
The solution is production-ready, well-tested, and provides a solid foundation for future enhancements. Performance improvements of 40-75% are expected under concurrent load, with predictable behavior even under extreme conditions.
## References
- **Implementation PR**: [Link to PR]
- **Original Issue**: User reported 2x-3.4x slowdown with concurrency
- **Technical Documentation**: `docs/CONCURRENT_PERFORMANCE_OPTIMIZATION.md`
- **Test Suite**: `rustfs/tests/concurrent_get_object_test.rs`
- **Core Module**: `rustfs/src/storage/concurrency.rs`
## Contact
For questions or issues:
- File issue on GitHub
- Tag @houseme or @copilot
- Reference this document and the implementation PR
docs/CONCURRENT_PERFORMANCE_OPTIMIZATION.md
-319
View File
@@ -1,319 +0,0 @@
# Concurrent GetObject Performance Optimization
## Problem Statement
When multiple concurrent GetObject requests are made to RustFS, performance degrades exponentially:
| Concurrency Level | Single Request Latency | Performance Impact |
|------------------|----------------------|-------------------|
| 1 request | 59ms | Baseline |
| 2 requests | 110ms | 1.9x slower |
| 4 requests | 200ms | 3.4x slower |
## Root Cause Analysis
The performance degradation was caused by several factors:
1. **Fixed Buffer Sizing**: Using `DEFAULT_READ_BUFFER_SIZE` (1MB) for all requests, regardless of concurrent load
- High memory contention under concurrent load
- Inefficient cache utilization
- CPU context switching overhead
2. **No Concurrency Control**: Unlimited concurrent disk reads causing I/O saturation
- Disk I/O queue depth exceeded optimal levels
- Increased seek times on traditional disks
- Resource contention between requests
3. **Lack of Caching**: Repeated reads of the same objects
- No reuse of frequently accessed data
- Unnecessary disk I/O for hot objects
## Solution Architecture
### 1. Concurrency-Aware Adaptive Buffer Sizing
The system now dynamically adjusts buffer sizes based on the current number of concurrent GetObject requests:
```rust
let optimal_buffer_size = get_concurrency_aware_buffer_size(file_size, base_buffer_size);
```
#### Buffer Sizing Strategy
| Concurrent Requests | Buffer Size Multiplier | Typical Buffer | Rationale |
|--------------------|----------------------|----------------|-----------|
| 1-2 (Low) | 1.0x (100%) | 512KB-1MB | Maximize throughput with large buffers |
| 3-4 (Medium) | 0.75x (75%) | 256KB-512KB | Balance throughput and fairness |
| 5-8 (High) | 0.5x (50%) | 128KB-256KB | Improve fairness, reduce memory pressure |
| 9+ (Very High) | 0.4x (40%) | 64KB-128KB | Ensure fair scheduling, minimize memory |
#### Benefits
- **Reduced memory pressure**: Smaller buffers under high concurrency prevent memory exhaustion
- **Better cache utilization**: More requests fit in CPU cache with smaller buffers
- **Improved fairness**: Prevents large requests from starving smaller ones
- **Adaptive performance**: Automatically tunes for different workload patterns
### 2. Hot Object Caching (LRU)
Implemented an intelligent LRU cache for frequently accessed small objects:
```rust
pub struct HotObjectCache {
max_object_size: usize, // Default: 10MB
max_cache_size: usize, // Default: 100MB
cache: RwLock<lru::LruCache<String, Arc<CachedObject>>>,
}
```
#### Caching Policy
- **Eligible objects**: Size ≤ 10MB, complete object reads (no ranges)
- **Eviction**: LRU (Least Recently Used)
- **Capacity**: Up to 1000 objects, 100MB total
- **Exclusions**: Encrypted objects, partial reads, multipart
#### Benefits
- **Reduced disk I/O**: Cache hits eliminate disk reads entirely
- **Lower latency**: Memory access is 100-1000x faster than disk
- **Higher throughput**: Free up disk bandwidth for cache misses
- **Better scalability**: Cache hit ratio improves with concurrent load
### 3. Disk I/O Concurrency Control
Added a semaphore to limit maximum concurrent disk reads:
```rust
disk_read_semaphore: Arc<Semaphore> // Default: 64 permits
```
#### Benefits
- **Prevents I/O saturation**: Limits queue depth to optimal levels
- **Predictable latency**: Avoids exponential latency increase
- **Protects disk health**: Reduces excessive seek operations
- **Graceful degradation**: Queues requests rather than thrashing
### 4. Request Tracking and Monitoring
Implemented RAII-based request tracking with automatic cleanup:
```rust
pub struct GetObjectGuard {
start_time: Instant,
}
impl Drop for GetObjectGuard {
fn drop(&mut self) {
ACTIVE_GET_REQUESTS.fetch_sub(1, Ordering::Relaxed);
// Record metrics
}
}
```
#### Metrics Collected
- `rustfs_concurrent_get_requests`: Current concurrent request count
- `rustfs_get_object_requests_completed`: Total completed requests
- `rustfs_get_object_duration_seconds`: Request duration histogram
- `rustfs_object_cache_hits`: Cache hit count
- `rustfs_object_cache_misses`: Cache miss count
- `rustfs_buffer_size_bytes`: Buffer size distribution
## Performance Expectations
### Expected Improvements
Based on the optimizations, we expect:
| Concurrency Level | Before | After (Expected) | Improvement |
|------------------|--------|------------------|-------------|
| 1 request | 59ms | 55-60ms | Similar (baseline) |
| 2 requests | 110ms | 65-75ms | ~40% faster |
| 4 requests | 200ms | 80-100ms | ~50% faster |
| 8 requests | 400ms | 100-130ms | ~65% faster |
| 16 requests | 800ms | 120-160ms | ~75% faster |
### Key Performance Characteristics
1. **Sub-linear scaling**: Latency increases sub-linearly with concurrency
2. **Cache benefits**: Hot objects see near-zero latency from cache hits
3. **Predictable behavior**: Bounded latency even under extreme load
4. **Memory efficiency**: Lower memory usage under high concurrency
## Implementation Details
### Integration Points
The optimization is integrated at the GetObject handler level:
```rust
async fn get_object(&self, req: S3Request<GetObjectInput>) -> S3Result<S3Response<GetObjectOutput>> {
// 1. Track request
let _request_guard = ConcurrencyManager::track_request();
// 2. Try cache
if let Some(cached_data) = manager.get_cached(&cache_key).await {
return Ok(S3Response::new(output)); // Fast path
}
// 3. Acquire I/O permit
let _disk_permit = manager.acquire_disk_read_permit().await;
// 4. Calculate optimal buffer size
let optimal_buffer_size = get_concurrency_aware_buffer_size(
response_content_length,
base_buffer_size
);
// 5. Stream with optimal buffer
let body = StreamingBlob::wrap(
ReaderStream::with_capacity(final_stream, optimal_buffer_size)
);
}
```
### Configuration
All defaults can be tuned via code changes:
```rust
// In concurrency.rs
const HIGH_CONCURRENCY_THRESHOLD: usize = 8;
const MEDIUM_CONCURRENCY_THRESHOLD: usize = 4;
// Cache settings
max_object_size: 10 * MI_B, // 10MB
max_cache_size: 100 * MI_B, // 100MB
disk_read_semaphore: Semaphore::new(64), // 64 concurrent reads
```
## Testing Recommendations
### 1. Concurrent Load Testing
Use the provided Go client to test different concurrency levels:
```go
concurrency := []int{1, 2, 4, 8, 16, 32}
for _, c := range concurrency {
// Run test with c concurrent goroutines
// Measure average latency and P50/P95/P99
}
```
### 2. Hot Object Testing
Test cache effectiveness with repeated reads:
```bash
# Read same object 100 times with 10 concurrent clients
for i in {1..10}; do
for j in {1..100}; do
mc cat rustfs/test/bxx > /dev/null
done &
done
wait
```
### 3. Mixed Workload Testing
Simulate real-world scenarios:
- 70% small objects (<1MB) - should see high cache hit rate
- 20% medium objects (1-10MB) - partial cache benefit
- 10% large objects (>10MB) - adaptive buffer sizing benefit
### 4. Stress Testing
Test system behavior under extreme load:
```bash
# 100 concurrent clients, continuous reads
ab -n 10000 -c 100 http://rustfs:9000/test/bxx
```
## Monitoring and Observability
### Key Metrics to Watch
1. **Latency Percentiles**
- P50, P95, P99 request duration
- Should show sub-linear growth with concurrency
2. **Cache Performance**
- Cache hit ratio (target: >70% for hot objects)
- Cache memory usage
- Eviction rate
3. **Resource Utilization**
- Memory usage per concurrent request
- Disk I/O queue depth
- CPU utilization
4. **Throughput**
- Requests per second
- Bytes per second
- Concurrent request count
### Prometheus Queries
```promql
# Average request duration by concurrency level
histogram_quantile(0.95,
rate(rustfs_get_object_duration_seconds_bucket[5m])
)
# Cache hit ratio
sum(rate(rustfs_object_cache_hits[5m]))
/
(sum(rate(rustfs_object_cache_hits[5m])) + sum(rate(rustfs_object_cache_misses[5m])))
# Concurrent requests over time
rustfs_concurrent_get_requests
# Memory efficiency (bytes per request)
rustfs_object_cache_size_bytes / rustfs_concurrent_get_requests
```
## Future Enhancements
### Potential Improvements
1. **Request Prioritization**
- Prioritize small requests over large ones
- Age-based priority to prevent starvation
- QoS classes for different clients
2. **Advanced Caching**
- Partial object caching (hot blocks)
- Predictive prefetching based on access patterns
- Distributed cache across multiple nodes
3. **I/O Scheduling**
- Batch similar requests for sequential I/O
- Deadline-based I/O scheduling
- NUMA-aware buffer allocation
4. **Adaptive Tuning**
- Machine learning based buffer sizing
- Dynamic cache size adjustment
- Workload-aware optimization
5. **Compression**
- Transparent compression for cached objects
- Adaptive compression based on CPU availability
- Deduplication for similar objects
## References
- [Issue #XXX](https://github.com/rustfs/rustfs/issues/XXX): Original performance issue
- [PR #XXX](https://github.com/rustfs/rustfs/pull/XXX): Implementation PR
- [MinIO Best Practices](https://min.io/docs/minio/linux/operations/install-deploy-manage/performance-and-optimization.html)
- [LRU Cache Design](https://leetcode.com/problems/lru-cache/)
- [Tokio Concurrency Patterns](https://tokio.rs/tokio/tutorial/shared-state)
## Conclusion
The concurrency-aware optimization addresses the root causes of performance degradation:
1. ✅ **Adaptive buffer sizing** reduces memory contention and improves cache utilization
2. ✅ **Hot object caching** eliminates redundant disk I/O for frequently accessed files
3. ✅ **I/O concurrency control** prevents disk saturation and ensures predictable latency
4. ✅ **Comprehensive monitoring** enables performance tracking and tuning
These changes should significantly improve performance under concurrent load while maintaining compatibility with existing clients and workloads.
docs/DEVELOPMENT.md
-71
View File
@@ -1,71 +0,0 @@
# RustFS Local Development Guide
This guide explains how to set up and run a local development environment for RustFS using Docker. This approach allows you to build and run the code from source in a consistent environment without needing to install the Rust toolchain on your host machine.
## Prerequisites
- [Docker](https://docs.docker.com/get-docker/)
- [Docker Compose](https://docs.docker.com/compose/install/)
## Quick Start
The development environment is configured as a Docker Compose profile named `dev`.
### 1. Setup Console UI (Optional)
If you want to use the Console UI, you must download the static assets first. The default source checkout does not include them.
```bash
bash scripts/static.sh
```
### 2. Start the Environment
To start the development container:
```bash
docker compose --profile dev up -d rustfs-dev
```
**Note**: The first run will take some time (5-10 minutes) because it builds the docker image and compiles all Rust dependencies from source. Subsequent runs will be much faster.
### 3. View Logs
To follow the application logs:
```bash
docker compose --profile dev logs -f rustfs-dev
```
### 4. Access the Services
- **S3 API**: `http://localhost:9010`
- **Console UI**: `http://localhost:9011/rustfs/console/index.html`
## Workflow
### Making Changes
The source code from your local `rustfs` directory is mounted into the container at `/app`. You can edit files in your preferred IDE on your host machine.
### Applying Changes
Since the application runs via `cargo run`, you need to restart the container to pick up changes. Thanks to incremental compilation, this is fast.
```bash
docker compose --profile dev restart rustfs-dev
```
### Rebuilding Dependencies
If you modify `Cargo.toml` or `Cargo.lock`, you generally need to rebuild the Docker image to update the cached dependencies layer:
```bash
docker compose --profile dev build rustfs-dev
```
## Troubleshooting
### `VolumeNotFound` Error
If you see an error like `Error: Custom { kind: Other, error: VolumeNotFound }`, it means the `rustfs` binary was started without valid volume arguments.
The development image uses `entrypoint.sh` to parse the `RUSTFS_VOLUMES` environment variable (supporting `{N..M}` syntax), create the directories, and pass them to `cargo run`. Ensure your `RUSTFS_VOLUMES` variable is correctly formatted.
### Slow Initial Build
This is expected. The `dev` stage in `Dockerfile.source` compiles all dependencies from scratch. Because the `/usr/local/cargo/registry` is mounted as a volume, these compiled artifacts are preserved between restarts, making future builds fast.
docs/ENVIRONMENT_VARIABLES.md
-202
View File
@@ -1,202 +0,0 @@
# RustFS Environment Variables
This document describes the environment variables that can be used to configure RustFS behavior.
## Background Services Control
### RUSTFS_ENABLE_SCANNER
Controls whether the data scanner service should be started.
- **Default**: `true`
- **Valid values**: `true`, `false`
- **Description**: When enabled, the data scanner will run background scans to detect inconsistencies and corruption in stored data.
**Examples**:
```bash
# Disable scanner
export RUSTFS_ENABLE_SCANNER=false
# Enable scanner (default behavior)
export RUSTFS_ENABLE_SCANNER=true
```
### RUSTFS_ENABLE_HEAL
Controls whether the auto-heal service should be started.
- **Default**: `true`
- **Valid values**: `true`, `false`
- **Description**: When enabled, the heal manager will automatically repair detected data inconsistencies and corruption.
**Examples**:
```bash
# Disable auto-heal
export RUSTFS_ENABLE_HEAL=false
# Enable auto-heal (default behavior)
export RUSTFS_ENABLE_HEAL=true
```
### RUSTFS_ENABLE_LOCKS
Controls whether the distributed lock system should be enabled.
- **Default**: `true`
- **Valid values**: `true`, `false`, `1`, `0`, `yes`, `no`, `on`, `off`, `enabled`, `disabled` (case insensitive)
- **Description**: When enabled, provides distributed locking for concurrent object operations. When disabled, all lock operations immediately return success without actual locking.
**Examples**:
```bash
# Disable lock system
export RUSTFS_ENABLE_LOCKS=false
# Enable lock system (default behavior)
export RUSTFS_ENABLE_LOCKS=true
```
## Service Combinations
The scanner and heal services can be independently controlled:
| RUSTFS_ENABLE_SCANNER | RUSTFS_ENABLE_HEAL | Result |
|----------------------|-------------------|--------|
| `true` (default) | `true` (default) | Both scanner and heal are active |
| `true` | `false` | Scanner runs without heal capabilities |
| `false` | `true` | Heal manager is available but no scanning |
| `false` | `false` | No background maintenance services |
## Use Cases
### Development Environment
For development or testing environments where you don't need background maintenance:
```bash
export RUSTFS_ENABLE_SCANNER=false
export RUSTFS_ENABLE_HEAL=false
./rustfs --address 127.0.0.1:9000 ...
```
### Scan-Only Mode
For environments where you want to detect issues but not automatically fix them:
```bash
export RUSTFS_ENABLE_SCANNER=true
export RUSTFS_ENABLE_HEAL=false
./rustfs --address 127.0.0.1:9000 ...
```
### Heal-Only Mode
For environments where external tools trigger healing but no automatic scanning:
```bash
export RUSTFS_ENABLE_SCANNER=false
export RUSTFS_ENABLE_HEAL=true
./rustfs --address 127.0.0.1:9000 ...
```
### Production Environment (Default)
For production environments where both services should be active:
```bash
# These are the defaults, so no need to set explicitly
# export RUSTFS_ENABLE_SCANNER=true
# export RUSTFS_ENABLE_HEAL=true
./rustfs --address 127.0.0.1:9000 ...
```
### No-Lock Development
For single-node development where locking is not needed:
```bash
export RUSTFS_ENABLE_LOCKS=false
./rustfs --address 127.0.0.1:9000 ...
```
## Protocol Servers
### RUSTFS_FTPS_ENABLE
Controls whether the FTPS (FTP over TLS) server should be started.
- **Default**: `false`
- **Valid values**: `true`, `false`
- **Description**: When enabled, starts an FTPS server for secure file transfers over TLS.
### RUSTFS_FTPS_ADDRESS
FTPS server bind address.
- **Default**: `0.0.0.0:8021`
- **Valid values**: Valid IP:PORT combination
- **Description**: The address and port where the FTPS server will listen for connections.
### RUSTFS_FTPS_CERTS_FILE
Path to FTPS server TLS certificate file.
- **Default**: None (required when FTPS is enabled)
- **Valid values**: Path to a PEM-encoded certificate file
- **Description**: TLS certificate used for securing FTPS connections.
### RUSTFS_FTPS_KEY_FILE
Path to FTPS server TLS private key file.
- **Default**: None (required when FTPS is enabled)
- **Valid values**: Path to a PEM-encoded private key file
- **Description**: TLS private key corresponding to the certificate.
### RUSTFS_FTPS_PASSIVE_PORTS
Passive port range for FTPS data connections.
- **Default**: None (system-assigned ports)
- **Valid values**: Port range in format "START-END" (e.g., "40000-50000")
- **Description**: Range of ports for FTPS passive mode data connections.
### RUSTFS_FTPS_EXTERNAL_IP
External IP address for FTPS passive mode.
- **Default**: None (auto-detected)
- **Valid values**: Valid IP address
- **Description**: External IP address advertised to FTPS clients for passive mode, useful for NAT setups.
### RUSTFS_SFTP_ENABLE
Controls whether the SFTP (SSH File Transfer Protocol) server should be started.
- **Default**: `false`
- **Valid values**: `true`, `false`
- **Description**: When enabled, starts an SFTP server for secure file transfers over SSH.
### RUSTFS_SFTP_ADDRESS
SFTP server bind address.
- **Default**: `0.0.0.0:8022`
- **Valid values**: Valid IP:PORT combination
- **Description**: The address and port where the SFTP server will listen for connections.
### RUSTFS_SFTP_HOST_KEY
Path to SFTP server SSH host key file.
- **Default**: None (required when SFTP is enabled)
- **Valid values**: Path to an SSH host key file
- **Description**: SSH host key used for server identification.
### RUSTFS_SFTP_AUTHORIZED_KEYS
Path to SFTP authorized keys file.
- **Default**: None (required when SFTP is enabled)
- **Valid values**: Path to a file containing OpenSSH public keys
- **Description**: File containing authorized SSH public keys for client authentication.
## Performance Impact
- **Scanner**: Light to moderate CPU/IO impact during scans
- **Heal**: Moderate to high CPU/IO impact during healing operations
- **Locks**: Minimal CPU/memory overhead for coordination; disabling can improve throughput in single-client scenarios
- **Memory**: Each service uses additional memory for processing queues and metadata
- **FTPS**: Moderate CPU/memory overhead for TLS operations and connection management
- **SFTP**: Moderate CPU/memory overhead for SSH operations and key management
Disabling these services in resource-constrained environments can improve performance for primary storage operations.
docs/FINAL_OPTIMIZATION_SUMMARY.md
-398
View File
@@ -1,398 +0,0 @@
# Final Optimization Summary - Concurrent GetObject Performance
## Overview
This document provides a comprehensive summary of all optimizations made to address the concurrent GetObject performance degradation issue, incorporating all feedback and implementing best practices as a senior Rust developer.
## Problem Statement
**Original Issue**: GetObject performance degraded exponentially under concurrent load:
- 1 concurrent request: 59ms
- 2 concurrent requests: 110ms (1.9x slower)
- 4 concurrent requests: 200ms (3.4x slower)
**Root Causes Identified**:
1. Fixed 1MB buffer size caused memory contention
2. No I/O concurrency control led to disk saturation
3. Absence of caching for frequently accessed objects
4. Inefficient lock management in concurrent scenarios
## Solution Architecture
### 1. Optimized LRU Cache Implementation (lru 0.16.2)
#### Read-First Access Pattern
Implemented an optimistic locking strategy using the `peek()` method from lru 0.16.2:
```rust
async fn get(&self, key: &str) -> Option<Arc<Vec<u8>>> {
// Phase 1: Read lock with peek (no LRU modification)
let cache = self.cache.read().await;
if let Some(cached) = cache.peek(key) {
let data = Arc::clone(&cached.data);
drop(cache);
// Phase 2: Write lock only for LRU promotion
let mut cache_write = self.cache.write().await;
if let Some(cached) = cache_write.get(key) {
cached.hit_count.fetch_add(1, Ordering::Relaxed);
return Some(data);
}
}
None
}
```
**Benefits**:
- **50% reduction** in write lock acquisitions
- Multiple readers can peek simultaneously
- Write lock only when promoting in LRU order
- Maintains proper LRU semantics
#### Advanced Cache Operations
**Batch Operations**:
```rust
// Single lock for multiple objects
pub async fn get_cached_batch(&self, keys: &[String]) -> Vec<Option<Arc<Vec<u8>>>>
```
**Cache Warming**:
```rust
// Pre-populate cache on startup
pub async fn warm_cache(&self, objects: Vec<(String, Vec<u8>)>)
```
**Hot Key Tracking**:
```rust
// Identify most accessed objects
pub async fn get_hot_keys(&self, limit: usize) -> Vec<(String, usize)>
```
**Cache Management**:
```rust
// Lightweight checks and explicit invalidation
pub async fn is_cached(&self, key: &str) -> bool
pub async fn remove_cached(&self, key: &str) -> bool
```
### 2. Advanced Buffer Sizing
#### Standard Concurrency-Aware Sizing
| Concurrent Requests | Buffer Multiplier | Rationale |
|--------------------|-------------------|-----------|
| 1-2 | 1.0x (100%) | Maximum throughput |
| 3-4 | 0.75x (75%) | Balanced performance |
| 5-8 | 0.5x (50%) | Fair resource sharing |
| >8 | 0.4x (40%) | Memory efficiency |
#### Advanced File-Pattern-Aware Sizing
```rust
pub fn get_advanced_buffer_size(
file_size: i64,
base_buffer_size: usize,
is_sequential: bool
) -> usize
```
**Optimizations**:
1. **Small files (<256KB)**: Use 25% of file size (16-64KB range)
2. **Sequential reads**: 1.5x multiplier at low concurrency
3. **Large files + high concurrency**: 0.8x for better parallelism
**Example**:
```rust
// 32MB file, sequential read, low concurrency
let buffer = get_advanced_buffer_size(
32 * 1024 * 1024, // file_size
256 * 1024, // base_buffer (256KB)
true // is_sequential
);
// Result: ~384KB buffer (256KB * 1.5)
```
### 3. I/O Concurrency Control
**Semaphore-Based Rate Limiting**:
- Default: 64 concurrent disk reads
- Prevents disk I/O saturation
- FIFO queuing ensures fairness
- Tunable based on storage type:
- NVMe SSD: 128-256
- HDD: 32-48
- Network storage: Based on bandwidth
### 4. RAII Request Tracking
```rust
pub struct GetObjectGuard {
start_time: Instant,
}
impl Drop for GetObjectGuard {
fn drop(&mut self) {
ACTIVE_GET_REQUESTS.fetch_sub(1, Ordering::Relaxed);
// Record metrics
}
}
```
**Benefits**:
- Zero overhead tracking
- Automatic cleanup on drop
- Panic-safe counter management
- Accurate concurrent load measurement
## Performance Analysis
### Cache Performance
| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| Cache hit (read-heavy) | 2-3ms | <1ms | 2-3x faster |
| Cache hit (with promotion) | 2-3ms | 2-3ms | Same (required) |
| Batch get (10 keys) | 20-30ms | 5-10ms | 2-3x faster |
| Cache miss | 50-800ms | 50-800ms | Same (disk bound) |
### Overall Latency Impact
| Concurrent Requests | Original | Optimized | Improvement |
|---------------------|----------|-----------|-------------|
| 1 | 59ms | 50-55ms | ~10% |
| 2 | 110ms | 60-70ms | ~40% |
| 4 | 200ms | 75-90ms | ~55% |
| 8 | 400ms | 90-120ms | ~70% |
| 16 | 800ms | 110-145ms | ~75% |
**With cache hits**: <5ms regardless of concurrency level
### Memory Efficiency
| Scenario | Buffer Size | Memory Impact | Efficiency Gain |
|----------|-------------|---------------|-----------------|
| Small files (128KB) | 32KB (was 256KB) | 8x more objects | 8x improvement |
| Sequential reads | 1.5x base | Better throughput | 50% faster |
| High concurrency | 0.32x base | 3x more requests | Better fairness |
## Test Coverage
### Comprehensive Test Suite (15 Tests)
**Request Tracking**:
1. `test_concurrent_request_tracking` - RAII guard functionality
**Buffer Sizing**:
2. `test_adaptive_buffer_sizing` - Multi-level concurrency adaptation
3. `test_buffer_size_bounds` - Boundary conditions
4. `test_advanced_buffer_sizing` - File pattern optimization
**Cache Operations**:
5. `test_cache_operations` - Basic cache lifecycle
6. `test_large_object_not_cached` - Size filtering
7. `test_cache_eviction` - LRU eviction behavior
8. `test_cache_batch_operations` - Batch retrieval efficiency
9. `test_cache_warming` - Pre-population mechanism
10. `test_hot_keys_tracking` - Access frequency tracking
11. `test_cache_removal` - Explicit invalidation
12. `test_is_cached_no_promotion` - Peek behavior verification
**Performance**:
13. `bench_concurrent_requests` - Concurrent request handling
14. `test_concurrent_cache_access` - Performance under load
15. `test_disk_io_permits` - Semaphore behavior
## Code Quality Standards
### Documentation
✅ **All documentation in English** following Rust documentation conventions
✅ **Comprehensive inline comments** explaining design decisions
✅ **Usage examples** in doc comments
✅ **Module-level documentation** with key features and characteristics
### Safety and Correctness
✅ **Thread-safe** - Proper use of Arc, RwLock, AtomicUsize
✅ **Panic-safe** - RAII guards ensure cleanup
✅ **Memory-safe** - No unsafe code
✅ **Deadlock-free** - Careful lock ordering and scope management
### API Design
✅ **Clear separation of concerns** - Public vs private APIs
✅ **Consistent naming** - Follows Rust naming conventions
✅ **Type safety** - Strong typing prevents misuse
✅ **Ergonomic** - Easy to use correctly, hard to use incorrectly
## Production Deployment Guide
### Configuration
```rust
// Adjust based on your environment
const CACHE_SIZE_MB: usize = 200; // For more hot objects
const MAX_OBJECT_SIZE_MB: usize = 20; // For larger hot objects
const DISK_CONCURRENCY: usize = 64; // Based on storage type
```
### Cache Warming Example
```rust
async fn init_cache_on_startup(manager: &ConcurrencyManager) {
// Load known hot objects
let hot_objects = vec![
("config/settings.json".to_string(), load_config()),
("common/logo.png".to_string(), load_logo()),
// ... more hot objects
];
manager.warm_cache(hot_objects).await;
info!("Cache warmed with {} objects", hot_objects.len());
}
```
### Monitoring
```rust
// Periodic cache metrics
tokio::spawn(async move {
loop {
tokio::time::sleep(Duration::from_secs(60)).await;
let stats = manager.cache_stats().await;
gauge!("cache_size_bytes").set(stats.size as f64);
gauge!("cache_entries").set(stats.entries as f64);
let hot_keys = manager.get_hot_keys(10).await;
for (key, hits) in hot_keys {
info!("Hot: {} ({} hits)", key, hits);
}
}
});
```
### Prometheus Metrics
```promql
# Cache hit ratio
sum(rate(rustfs_object_cache_hits[5m]))
/
(sum(rate(rustfs_object_cache_hits[5m])) + sum(rate(rustfs_object_cache_misses[5m])))
# P95 latency
histogram_quantile(0.95, rate(rustfs_get_object_duration_seconds_bucket[5m]))
# Concurrent requests
rustfs_concurrent_get_requests
# Cache efficiency
rustfs_object_cache_size_bytes / rustfs_object_cache_entries
```
## File Structure
```
rustfs/
├── src/
│ └── storage/
│ ├── concurrency.rs # Core concurrency management
│ ├── concurrent_get_object_test.rs # Comprehensive tests
│ ├── ecfs.rs # GetObject integration
│ └── mod.rs # Module declarations
├── Cargo.toml # lru = "0.16.2"
└── docs/
├── CONCURRENT_PERFORMANCE_OPTIMIZATION.md
├── ENHANCED_CACHING_OPTIMIZATION.md
├── PR_ENHANCEMENTS_SUMMARY.md
└── FINAL_OPTIMIZATION_SUMMARY.md # This document
```
## Migration Guide
### Backward Compatibility
✅ **100% backward compatible** - No breaking changes
✅ **Automatic optimization** - Existing code benefits immediately
✅ **Opt-in advanced features** - Use when needed
### Using New Features
```rust
// Basic usage (automatic)
let _guard = ConcurrencyManager::track_request();
if let Some(data) = manager.get_cached(&key).await {
return serve_from_cache(data);
}
// Advanced usage (explicit)
let results = manager.get_cached_batch(&keys).await;
manager.warm_cache(hot_objects).await;
let hot = manager.get_hot_keys(10).await;
// Advanced buffer sizing
let buffer = get_advanced_buffer_size(file_size, base, is_sequential);
```
## Future Enhancements
### Short Term
1. Implement TeeReader for automatic cache insertion from streams
2. Add Admin API for cache management
3. Distributed cache invalidation across cluster nodes
### Medium Term
1. Predictive prefetching based on access patterns
2. Tiered caching (Memory + SSD + Remote)
3. Smart eviction considering factors beyond LRU
### Long Term
1. ML-based optimization and prediction
2. Content-addressable storage with deduplication
3. Adaptive tuning based on observed patterns
## Success Metrics
### Quantitative Goals
✅ **Latency reduction**: 40-75% improvement under concurrent load
✅ **Memory efficiency**: Sub-linear growth with concurrency
✅ **Cache effectiveness**: <5ms for cache hits
✅ **I/O optimization**: Bounded queue depth
### Qualitative Goals
✅ **Maintainability**: Clear, well-documented code
✅ **Reliability**: No crashes or resource leaks
✅ **Observability**: Comprehensive metrics
✅ **Compatibility**: No breaking changes
## Conclusion
This optimization successfully addresses the concurrent GetObject performance issue through a comprehensive solution:
1. **Optimized Cache** (lru 0.16.2) with read-first pattern
2. **Advanced buffer sizing** adapting to concurrency and file patterns
3. **I/O concurrency control** preventing disk saturation
4. **Batch operations** for efficiency
5. **Comprehensive testing** ensuring correctness
6. **Production-ready** features and monitoring
The solution is backward compatible, well-tested, thoroughly documented in English, and ready for production deployment.
## References
- **Issue**: #911 - Concurrent GetObject performance degradation
- **Final Commit**: 010e515 - Complete optimization with lru 0.16.2
- **Implementation**: `rustfs/src/storage/concurrency.rs`
- **Tests**: `rustfs/src/storage/concurrent_get_object_test.rs`
- **LRU Crate**: https://crates.io/crates/lru (version 0.16.2)
## Contact
For questions or issues related to this optimization:
- File issue on GitHub referencing #911
- Tag @houseme or @copilot
- Reference this document and commit 010e515
docs/IMPLEMENTATION_SUMMARY.md
-412
View File
@@ -1,412 +0,0 @@
# Adaptive Buffer Sizing Implementation Summary
## Overview
This implementation extends PR #869 with a comprehensive adaptive buffer sizing optimization system that provides intelligent buffer size selection based on file size and workload type.
## What Was Implemented
### 1. Workload Profile System
**File:** `rustfs/src/config/workload_profiles.rs` (501 lines)
A complete workload profiling system with:
- **6 Predefined Profiles:**
- `GeneralPurpose`: Balanced performance (default)
- `AiTraining`: Optimized for large sequential reads
- `DataAnalytics`: Mixed read-write patterns
- `WebWorkload`: Small file intensive
- `IndustrialIoT`: Real-time streaming
- `SecureStorage`: Security-first, memory-constrained
- **Custom Configuration Support:**
```rust
WorkloadProfile::Custom(BufferConfig {
min_size: 16 * 1024,
max_size: 512 * 1024,
default_unknown: 128 * 1024,
thresholds: vec![...],
})
```
- **Configuration Validation:**
- Ensures min_size > 0
- Validates max_size >= min_size
- Checks threshold ordering
- Validates buffer sizes within bounds
### 2. Enhanced Buffer Sizing Algorithm
**File:** `rustfs/src/storage/ecfs.rs` (+156 lines)
- **Backward Compatible:**
- Preserved original `get_adaptive_buffer_size()` function
- Existing code continues to work without changes
- **New Enhanced Function:**
```rust
fn get_adaptive_buffer_size_with_profile(
file_size: i64,
profile: Option<WorkloadProfile>
) -> usize
```
- **Auto-Detection:**
- Automatically detects Chinese secure OS (Kylin, NeoKylin, UOS, OpenKylin)
- Falls back to GeneralPurpose if no special environment detected
### 3. Comprehensive Testing
**Location:** `rustfs/src/storage/ecfs.rs` and `rustfs/src/config/workload_profiles.rs`
- Unit tests for all 6 workload profiles
- Boundary condition testing
- Configuration validation tests
- Custom configuration tests
- Unknown file size handling tests
- Total: 15+ comprehensive test cases
### 4. Complete Documentation
**Files:**
- `docs/adaptive-buffer-sizing.md` (460 lines)
- `docs/README.md` (updated with navigation)
Documentation includes:
- Overview and architecture
- Detailed profile descriptions
- Usage examples
- Performance considerations
- Best practices
- Troubleshooting guide
- Migration guide from PR #869
## Design Decisions
### 1. Backward Compatibility
**Decision:** Keep original `get_adaptive_buffer_size()` function unchanged.
**Rationale:**
- Ensures no breaking changes
- Existing code continues to work
- Gradual migration path available
### 2. Profile-Based Configuration
**Decision:** Use enum-based profiles instead of global configuration.
**Rationale:**
- Type-safe profile selection
- Compile-time validation
- Easy to extend with new profiles
- Clear documentation of available options
### 3. Separate Module for Profiles
**Decision:** Create dedicated `workload_profiles` module.
**Rationale:**
- Clear separation of concerns
- Easy to locate and maintain
- Can be used across the codebase
- Facilitates testing
### 4. Conservative Default Values
**Decision:** Use moderate buffer sizes by default.
**Rationale:**
- Prevents excessive memory usage
- Suitable for most workloads
- Users can opt-in to larger buffers
## Performance Characteristics
### Memory Usage by Profile
| Profile | Min Buffer | Max Buffer | Memory Footprint |
|---------|-----------|-----------|------------------|
| GeneralPurpose | 64KB | 1MB | Low-Medium |
| AiTraining | 512KB | 4MB | High |
| DataAnalytics | 128KB | 2MB | Medium |
| WebWorkload | 32KB | 256KB | Low |
| IndustrialIoT | 64KB | 512KB | Low |
| SecureStorage | 32KB | 256KB | Low |
### Throughput Impact
- **Small buffers (32-64KB):** Better for high concurrency, many small files
- **Medium buffers (128-512KB):** Balanced for mixed workloads
- **Large buffers (1-4MB):** Maximum throughput for large sequential I/O
## Usage Patterns
### Simple Usage (Backward Compatible)
```rust
// Existing code works unchanged
let buffer_size = get_adaptive_buffer_size(file_size);
```
### Profile-Aware Usage
```rust
// For AI/ML workloads
let buffer_size = get_adaptive_buffer_size_with_profile(
file_size,
Some(WorkloadProfile::AiTraining)
);
// Auto-detect environment
let buffer_size = get_adaptive_buffer_size_with_profile(file_size, None);
```
### Custom Configuration
```rust
let custom = BufferConfig {
min_size: 16 * 1024,
max_size: 512 * 1024,
default_unknown: 128 * 1024,
thresholds: vec![
(1024 * 1024, 64 * 1024),
(i64::MAX, 256 * 1024),
],
};
let profile = WorkloadProfile::Custom(custom);
let buffer_size = get_adaptive_buffer_size_with_profile(file_size, Some(profile));
```
## Integration Points
The new functionality can be integrated into:
1. **`put_object`**: Choose profile based on object metadata or headers
2. **`put_object_extract`**: Use appropriate profile for archive extraction
3. **`upload_part`**: Apply profile for multipart uploads
Example integration (future enhancement):
```rust
async fn put_object(&self, req: S3Request<PutObjectInput>) -> S3Result<S3Response<PutObjectOutput>> {
// Detect workload from headers or configuration
let profile = detect_workload_from_request(&req);
let buffer_size = get_adaptive_buffer_size_with_profile(
size,
Some(profile)
);
let body = tokio::io::BufReader::with_capacity(buffer_size, reader);
// ... rest of implementation
}
```
## Security Considerations
### Memory Safety
1. **Bounded Buffer Sizes:**
- All configurations enforce min and max limits
- Prevents out-of-memory conditions
- Validation at configuration creation time
2. **Immutable Configurations:**
- All config structures are immutable after creation
- Thread-safe by design
- No risk of race conditions
3. **Secure OS Detection:**
- Read-only access to `/etc/os-release`
- No privilege escalation required
- Graceful fallback on error
### No New Vulnerabilities
- Only adds new functionality
- Does not modify existing security-critical paths
- Preserves all existing security measures
- All new code is defensive and validated
## Testing Strategy
### Unit Tests
- Located in both modules with `#[cfg(test)]`
- Test all workload profiles
- Validate configuration logic
- Test boundary conditions
### Integration Testing
Future integration tests should cover:
- Actual file upload/download with different profiles
- Performance benchmarks for each profile
- Memory usage monitoring
- Concurrent operations
## Future Enhancements
### 1. Runtime Configuration
Add environment variables or config file support:
```bash
RUSTFS_BUFFER_PROFILE=AiTraining
RUSTFS_BUFFER_MIN_SIZE=32768
RUSTFS_BUFFER_MAX_SIZE=1048576
```
### 2. Dynamic Profiling
Collect metrics and automatically adjust profile:
```rust
// Monitor actual I/O patterns and adjust buffer sizes
let optimal_profile = analyze_io_patterns();
```
### 3. Per-Bucket Configuration
Allow different profiles per bucket:
```rust
// Configure profiles via bucket metadata
bucket.set_buffer_profile(WorkloadProfile::WebWorkload);
```
### 4. Performance Metrics
Add metrics to track buffer effectiveness:
```rust
metrics::histogram!("buffer_utilization", utilization);
metrics::counter!("buffer_resizes", 1);
```
## Migration Path
### Phase 1: Current State ✅
- Infrastructure in place
- Backward compatible
- Fully documented
- Tested
### Phase 2: Opt-In Usage ✅ **IMPLEMENTED**
- ✅ Configuration option to enable profiles (`RUSTFS_BUFFER_PROFILE_ENABLE`)
- ✅ Workload profile selection (`RUSTFS_BUFFER_PROFILE`)
- ✅ Default to existing behavior when disabled
- ✅ Global configuration management
- ✅ Integration in `put_object`, `put_object_extract`, and `upload_part`
- ✅ Command-line and environment variable support
- ✅ Performance monitoring ready
**How to Use:**
```bash
# Enable with environment variables
export RUSTFS_BUFFER_PROFILE_ENABLE=true
export RUSTFS_BUFFER_PROFILE=AiTraining
./rustfs /data
# Or use command-line flags
./rustfs --buffer-profile-enable --buffer-profile WebWorkload /data
```
### Phase 3: Default Enablement ✅ **IMPLEMENTED**
- ✅ Profile-aware buffer sizing enabled by default
- ✅ Default profile: `GeneralPurpose` (same behavior as PR #869 for most files)
- ✅ Backward compatibility via `--buffer-profile-disable` flag
- ✅ Easy profile switching via `--buffer-profile` or `RUSTFS_BUFFER_PROFILE`
- ✅ Updated documentation with Phase 3 examples
**Default Behavior:**
```bash
# Phase 3: Enabled by default with GeneralPurpose profile
./rustfs /data
# Change to a different profile
./rustfs --buffer-profile AiTraining /data
# Opt-out to legacy behavior if needed
./rustfs --buffer-profile-disable /data
```
**Key Changes from Phase 2:**
- Phase 2: Required `--buffer-profile-enable` to opt-in
- Phase 3: Enabled by default, use `--buffer-profile-disable` to opt-out
- Maintains full backward compatibility
- No breaking changes for existing deployments
### Phase 4: Full Integration ✅ **IMPLEMENTED**
- ✅ Deprecated legacy `get_adaptive_buffer_size()` function
- ✅ Profile-only implementation via `get_buffer_size_opt_in()`
- ✅ Performance metrics collection capability (with `metrics` feature)
- ✅ Consolidated buffer sizing logic
- ✅ All buffer sizes come from workload profiles
**Implementation Details:**
```rust
// Phase 4: Single entry point for buffer sizing
fn get_buffer_size_opt_in(file_size: i64) -> usize {
// Uses workload profiles exclusively
// Legacy function deprecated but maintained for compatibility
// Metrics collection integrated for performance monitoring
}
```
**Key Changes from Phase 3:**
- Legacy function marked as `#[deprecated]` but still functional
- Single, unified buffer sizing implementation
- Performance metrics tracking (optional, via feature flag)
- Even disabled mode uses GeneralPurpose profile (profile-only)
## Maintenance Guidelines
### Adding New Profiles
1. Add enum variant to `WorkloadProfile`
2. Implement config method
3. Add tests
4. Update documentation
5. Add usage examples
### Modifying Existing Profiles
1. Update threshold values in config method
2. Update tests to match new values
3. Update documentation
4. Consider migration impact
### Performance Tuning
1. Collect metrics from production
2. Analyze buffer hit rates
3. Adjust thresholds based on data
4. A/B test changes
5. Update documentation with findings
## Conclusion
This implementation provides a solid foundation for adaptive buffer sizing in RustFS:
- ✅ Comprehensive workload profiling system
- ✅ Backward compatible design
- ✅ Extensive testing
- ✅ Complete documentation
- ✅ Secure and memory-safe
- ✅ Ready for production use
The modular design allows for gradual adoption and future enhancements without breaking existing functionality.
## References
- [PR #869: Fix large file upload freeze with adaptive buffer sizing](https://github.com/rustfs/rustfs/pull/869)
- [Adaptive Buffer Sizing Documentation](./adaptive-buffer-sizing.md)
- [Performance Testing Guide](./PERFORMANCE_TESTING.md)
docs/MIGRATION_PHASE3.md
-284
View File
@@ -1,284 +0,0 @@
# Migration Guide: Phase 2 to Phase 3
## Overview
Phase 3 of the adaptive buffer sizing feature makes workload profiles **enabled by default**. This document helps you understand the changes and how to migrate smoothly.
## What Changed
### Phase 2 (Opt-In)
- Buffer profiling was **disabled by default**
- Required explicit enabling via `--buffer-profile-enable` or `RUSTFS_BUFFER_PROFILE_ENABLE=true`
- Used legacy PR #869 behavior unless explicitly enabled
### Phase 3 (Default Enablement)
- Buffer profiling is **enabled by default** with `GeneralPurpose` profile
- No configuration needed for default behavior
- Can opt-out via `--buffer-profile-disable` or `RUSTFS_BUFFER_PROFILE_DISABLE=true`
- Maintains full backward compatibility
## Impact Analysis
### For Most Users (No Action Required)
The `GeneralPurpose` profile (default in Phase 3) provides the **same buffer sizes** as PR #869 for most file sizes:
- Small files (< 1MB): 64KB buffer
- Medium files (1MB-100MB): 256KB buffer
- Large files (≥ 100MB): 1MB buffer
**Result:** Your existing deployments will work exactly as before, with no performance changes.
### For Users Who Explicitly Enabled Profiles in Phase 2
If you were using:
```bash
# Phase 2
export RUSTFS_BUFFER_PROFILE_ENABLE=true
export RUSTFS_BUFFER_PROFILE=AiTraining
./rustfs /data
```
You can simplify to:
```bash
# Phase 3
export RUSTFS_BUFFER_PROFILE=AiTraining
./rustfs /data
```
The `RUSTFS_BUFFER_PROFILE_ENABLE` variable is no longer needed (but still respected for compatibility).
### For Users Who Want Exact Legacy Behavior
If you need the guaranteed exact behavior from PR #869 (before any profiling):
```bash
# Phase 3 - Opt out to legacy behavior
export RUSTFS_BUFFER_PROFILE_DISABLE=true
./rustfs /data
# Or via command-line
./rustfs --buffer-profile-disable /data
```
## Migration Scenarios
### Scenario 1: Default Deployment (No Changes Needed)
**Phase 2:**
```bash
./rustfs /data
# Used PR #869 fixed algorithm
```
**Phase 3:**
```bash
./rustfs /data
# Uses GeneralPurpose profile (same buffer sizes as PR #869 for most cases)
```
**Action:** None required. Behavior is essentially identical.
### Scenario 2: Using Custom Profile in Phase 2
**Phase 2:**
```bash
export RUSTFS_BUFFER_PROFILE_ENABLE=true
export RUSTFS_BUFFER_PROFILE=WebWorkload
./rustfs /data
```
**Phase 3 (Simplified):**
```bash
export RUSTFS_BUFFER_PROFILE=WebWorkload
./rustfs /data
# RUSTFS_BUFFER_PROFILE_ENABLE no longer needed
```
**Action:** Remove `RUSTFS_BUFFER_PROFILE_ENABLE=true` from your configuration.
### Scenario 3: Explicitly Disabled in Phase 2
**Phase 2:**
```bash
# Or just not setting RUSTFS_BUFFER_PROFILE_ENABLE
./rustfs /data
```
**Phase 3 (If you want to keep legacy behavior):**
```bash
export RUSTFS_BUFFER_PROFILE_DISABLE=true
./rustfs /data
```
**Action:** Set `RUSTFS_BUFFER_PROFILE_DISABLE=true` if you want to guarantee exact PR #869 behavior.
### Scenario 4: AI/ML Workloads
**Phase 2:**
```bash
export RUSTFS_BUFFER_PROFILE_ENABLE=true
export RUSTFS_BUFFER_PROFILE=AiTraining
./rustfs /data
```
**Phase 3 (Simplified):**
```bash
export RUSTFS_BUFFER_PROFILE=AiTraining
./rustfs /data
```
**Action:** Remove `RUSTFS_BUFFER_PROFILE_ENABLE=true`.
## Configuration Reference
### Phase 3 Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `RUSTFS_BUFFER_PROFILE` | `GeneralPurpose` | The workload profile to use |
| `RUSTFS_BUFFER_PROFILE_DISABLE` | `false` | Disable profiling and use legacy behavior |
### Phase 3 Command-Line Flags
| Flag | Default | Description |
|------|---------|-------------|
| `--buffer-profile <PROFILE>` | `GeneralPurpose` | Set the workload profile |
| `--buffer-profile-disable` | disabled | Disable profiling (opt-out) |
### Deprecated (Still Supported for Compatibility)
| Variable | Status | Replacement |
|----------|--------|-------------|
| `RUSTFS_BUFFER_PROFILE_ENABLE` | Deprecated | Profiling is enabled by default; use `RUSTFS_BUFFER_PROFILE_DISABLE` to opt-out |
## Performance Expectations
### GeneralPurpose Profile (Default)
Same performance as PR #869 for most workloads:
- Small files: Same 64KB buffer
- Medium files: Same 256KB buffer
- Large files: Same 1MB buffer
### Specialized Profiles
When you switch to a specialized profile, you get optimized buffer sizes:
| Profile | Performance Benefit | Use Case |
|---------|-------------------|----------|
| `AiTraining` | Up to 4x throughput on large files | ML model files, training datasets |
| `WebWorkload` | Lower memory, higher concurrency | Static assets, CDN |
| `DataAnalytics` | Balanced for mixed patterns | Data warehouses, BI |
| `IndustrialIoT` | Low latency, memory-efficient | Sensor data, telemetry |
| `SecureStorage` | Compliance-focused, minimal memory | Government, healthcare |
## Testing Your Migration
### Step 1: Test Default Behavior
```bash
# Start with default configuration
./rustfs /data
# Verify it works as expected
# Check logs for: "Using buffer profile: GeneralPurpose"
```
### Step 2: Test Your Workload Profile (If Using)
```bash
# Set your specific profile
export RUSTFS_BUFFER_PROFILE=AiTraining
./rustfs /data
# Verify in logs: "Using buffer profile: AiTraining"
```
### Step 3: Test Opt-Out (If Needed)
```bash
# Disable profiling
export RUSTFS_BUFFER_PROFILE_DISABLE=true
./rustfs /data
# Verify in logs: "using legacy adaptive buffer sizing"
```
## Rollback Plan
If you encounter any issues with Phase 3, you can easily roll back:
### Option 1: Disable Profiling
```bash
export RUSTFS_BUFFER_PROFILE_DISABLE=true
./rustfs /data
```
This gives you the exact PR #869 behavior.
### Option 2: Use GeneralPurpose Profile Explicitly
```bash
export RUSTFS_BUFFER_PROFILE=GeneralPurpose
./rustfs /data
```
This uses profiling but with conservative buffer sizes.
## FAQ
### Q: Will Phase 3 break my existing deployment?
**A:** No. The default `GeneralPurpose` profile uses the same buffer sizes as PR #869 for most scenarios. Your deployment will work exactly as before.
### Q: Do I need to change my configuration?
**A:** Only if you were explicitly using profiles in Phase 2. You can simplify by removing `RUSTFS_BUFFER_PROFILE_ENABLE=true`.
### Q: What if I want the exact legacy behavior?
**A:** Set `RUSTFS_BUFFER_PROFILE_DISABLE=true` to use the exact PR #869 algorithm.
### Q: Can I still use RUSTFS_BUFFER_PROFILE_ENABLE?
**A:** Yes, it's still supported for backward compatibility, but it's no longer necessary.
### Q: How do I know which profile is active?
**A:** Check the startup logs for messages like:
- "Using buffer profile: GeneralPurpose"
- "Buffer profiling is disabled, using legacy adaptive buffer sizing"
### Q: Should I switch to a specialized profile?
**A:** Only if you have specific workload characteristics:
- AI/ML with large files → `AiTraining`
- Web applications → `WebWorkload`
- Secure/compliance environments → `SecureStorage`
- Default is fine for most general-purpose workloads
## Support
If you encounter issues during migration:
1. Check logs for buffer profile information
2. Try disabling profiling with `--buffer-profile-disable`
3. Report issues with:
- Your workload type
- File sizes you're working with
- Performance observations
- Log excerpts showing buffer profile initialization
## Timeline
- **Phase 1:** Infrastructure (✅ Complete)
- **Phase 2:** Opt-In Usage (✅ Complete)
- **Phase 3:** Default Enablement (✅ Current - You are here)
- **Phase 4:** Full Integration (Future)
## Conclusion
Phase 3 represents a smooth evolution of the adaptive buffer sizing feature. The default behavior remains compatible with PR #869, while providing an easy path to optimize for specific workloads when needed.
Most users can migrate without any changes, and those who need the exact legacy behavior can easily opt-out.
docs/MOKA_CACHE_MIGRATION.md
-569
View File
@@ -1,569 +0,0 @@
# Moka Cache Migration and Metrics Integration
## Overview
This document describes the complete migration from `lru` to `moka` cache library and the comprehensive metrics collection system integrated into the GetObject operation.
## Why Moka?
### Performance Advantages
| Feature | LRU 0.16.2 | Moka 0.12.11 | Benefit |
|---------|------------|--------------|---------|
| **Concurrent reads** | RwLock (shared lock) | Lock-free | 10x+ faster reads |
| **Concurrent writes** | RwLock (exclusive lock) | Lock-free | No write blocking |
| **Expiration** | Manual implementation | Built-in TTL/TTI | Automatic cleanup |
| **Size tracking** | Manual atomic counters | Weigher function | Accurate & automatic |
| **Async support** | Manual wrapping | Native async/await | Better integration |
| **Memory management** | Manual eviction | Automatic LRU | Less complexity |
| **Performance scaling** | O(log n) with lock | O(1) lock-free | Better at scale |
### Key Improvements
1. **True Lock-Free Access**: No locks for reads or writes, enabling true parallel access
2. **Automatic Expiration**: TTL and TTI handled by the cache itself
3. **Size-Based Eviction**: Weigher function ensures accurate memory tracking
4. **Native Async**: Built for tokio from the ground up
5. **Better Concurrency**: Scales linearly with concurrent load
## Implementation Details
### Cache Configuration
```rust
let cache = Cache::builder()
.max_capacity(100 * MI_B as u64) // 100MB total
.weigher(|_key: &String, value: &Arc<CachedObject>| -> u32 {
value.size.min(u32::MAX as usize) as u32
})
.time_to_live(Duration::from_secs(300)) // 5 minutes TTL
.time_to_idle(Duration::from_secs(120)) // 2 minutes TTI
.build();
```
**Configuration Rationale**:
- **Max Capacity (100MB)**: Balances memory usage with cache hit rate
- **Weigher**: Tracks actual object size for accurate eviction
- **TTL (5 min)**: Ensures objects don't stay stale too long
- **TTI (2 min)**: Evicts rarely accessed objects automatically
### Data Structures
#### HotObjectCache
```rust
#[derive(Clone)]
struct HotObjectCache {
cache: Cache<String, Arc<CachedObject>>,
max_object_size: usize,
hit_count: Arc<AtomicU64>,
miss_count: Arc<AtomicU64>,
}
```
**Changes from LRU**:
- Removed `RwLock` wrapper (Moka is lock-free)
- Removed manual `current_size` tracking (Moka handles this)
- Added global hit/miss counters for statistics
- Made struct `Clone` for easier sharing
#### CachedObject
```rust
#[derive(Clone)]
struct CachedObject {
data: Arc<Vec<u8>>,
cached_at: Instant,
size: usize,
access_count: Arc<AtomicU64>, // Changed from AtomicUsize
}
```
**Changes**:
- `access_count` now `AtomicU64` for larger counts
- Struct is `Clone` for compatibility with Moka
### Core Methods
#### get() - Lock-Free Retrieval
```rust
async fn get(&self, key: &str) -> Option<Arc<Vec<u8>>> {
match self.cache.get(key).await {
Some(cached) => {
cached.access_count.fetch_add(1, Ordering::Relaxed);
self.hit_count.fetch_add(1, Ordering::Relaxed);
#[cfg(feature = "metrics")]
{
counter!("rustfs_object_cache_hits").increment(1);
counter!("rustfs_object_cache_access_count", "key" => key)
.increment(1);
}
Some(Arc::clone(&cached.data))
}
None => {
self.miss_count.fetch_add(1, Ordering::Relaxed);
#[cfg(feature = "metrics")]
{
counter!("rustfs_object_cache_misses").increment(1);
}
None
}
}
}
```
**Benefits**:
- No locks acquired
- Automatic LRU promotion by Moka
- Per-key and global metrics tracking
- O(1) average case performance
#### put() - Automatic Eviction
```rust
async fn put(&self, key: String, data: Vec<u8>) {
let size = data.len();
if size == 0 || size > self.max_object_size {
return;
}
let cached_obj = Arc::new(CachedObject {
data: Arc::new(data),
cached_at: Instant::now(),
size,
access_count: Arc::new(AtomicU64::new(0)),
});
self.cache.insert(key.clone(), cached_obj).await;
#[cfg(feature = "metrics")]
{
counter!("rustfs_object_cache_insertions").increment(1);
gauge!("rustfs_object_cache_size_bytes")
.set(self.cache.weighted_size() as f64);
gauge!("rustfs_object_cache_entry_count")
.set(self.cache.entry_count() as f64);
}
}
```
**Simplifications**:
- No manual eviction loop (Moka handles automatically)
- No size tracking (weigher function handles this)
- Direct cache access without locks
#### stats() - Accurate Reporting
```rust
async fn stats(&self) -> CacheStats {
self.cache.run_pending_tasks().await; // Ensure accuracy
CacheStats {
size: self.cache.weighted_size() as usize,
entries: self.cache.entry_count() as usize,
max_size: 100 * MI_B,
max_object_size: self.max_object_size,
hit_count: self.hit_count.load(Ordering::Relaxed),
miss_count: self.miss_count.load(Ordering::Relaxed),
}
}
```
**Improvements**:
- `run_pending_tasks()` ensures accurate stats
- Direct access to `weighted_size()` and `entry_count()`
- Includes hit/miss counters
## Comprehensive Metrics Integration
### Metrics Architecture
```
┌─────────────────────────────────────────────────────────┐
│ GetObject Flow │
├─────────────────────────────────────────────────────────┤
│ │
│ 1. Request Start │
│ ↓ rustfs_get_object_requests_total (counter) │
│ ↓ rustfs_concurrent_get_object_requests (gauge) │
│ │
│ 2. Cache Lookup │
│ ├─ Hit → rustfs_object_cache_hits (counter) │
│ │ rustfs_get_object_cache_served_total │
│ │ rustfs_get_object_cache_serve_duration │
│ │ │
│ └─ Miss → rustfs_object_cache_misses (counter) │
│ │
│ 3. Disk Permit Acquisition │
│ ↓ rustfs_disk_permit_wait_duration_seconds │
│ │
│ 4. Disk Read │
│ ↓ (existing storage metrics) │
│ │
│ 5. Response Build │
│ ↓ rustfs_get_object_response_size_bytes │
│ ↓ rustfs_get_object_buffer_size_bytes │
│ │
│ 6. Request Complete │
│ ↓ rustfs_get_object_requests_completed │
│ ↓ rustfs_get_object_total_duration_seconds │
│ │
└─────────────────────────────────────────────────────────┘
```
### Metric Catalog
#### Request Metrics
| Metric | Type | Description | Labels |
|--------|------|-------------|--------|
| `rustfs_get_object_requests_total` | Counter | Total GetObject requests received | - |
| `rustfs_get_object_requests_completed` | Counter | Completed GetObject requests | - |
| `rustfs_concurrent_get_object_requests` | Gauge | Current concurrent requests | - |
| `rustfs_get_object_total_duration_seconds` | Histogram | End-to-end request duration | - |
#### Cache Metrics
| Metric | Type | Description | Labels |
|--------|------|-------------|--------|
| `rustfs_object_cache_hits` | Counter | Cache hits | - |
| `rustfs_object_cache_misses` | Counter | Cache misses | - |
| `rustfs_object_cache_access_count` | Counter | Per-object access count | key |
| `rustfs_get_object_cache_served_total` | Counter | Objects served from cache | - |
| `rustfs_get_object_cache_serve_duration_seconds` | Histogram | Cache serve latency | - |
| `rustfs_get_object_cache_size_bytes` | Histogram | Cached object sizes | - |
| `rustfs_object_cache_insertions` | Counter | Cache insertions | - |
| `rustfs_object_cache_size_bytes` | Gauge | Total cache memory usage | - |
| `rustfs_object_cache_entry_count` | Gauge | Number of cached entries | - |
#### I/O Metrics
| Metric | Type | Description | Labels |
|--------|------|-------------|--------|
| `rustfs_disk_permit_wait_duration_seconds` | Histogram | Time waiting for disk permit | - |
#### Response Metrics
| Metric | Type | Description | Labels |
|--------|------|-------------|--------|
| `rustfs_get_object_response_size_bytes` | Histogram | Response payload sizes | - |
| `rustfs_get_object_buffer_size_bytes` | Histogram | Buffer sizes used | - |
### Prometheus Query Examples
#### Cache Performance
```promql
# Cache hit rate
sum(rate(rustfs_object_cache_hits[5m]))
/
(sum(rate(rustfs_object_cache_hits[5m])) + sum(rate(rustfs_object_cache_misses[5m])))
# Cache memory utilization
rustfs_object_cache_size_bytes / (100 * 1024 * 1024)
# Cache effectiveness (objects served directly)
rate(rustfs_get_object_cache_served_total[5m])
/
rate(rustfs_get_object_requests_completed[5m])
# Average cache serve latency
rate(rustfs_get_object_cache_serve_duration_seconds_sum[5m])
/
rate(rustfs_get_object_cache_serve_duration_seconds_count[5m])
# Top 10 most accessed cached objects
topk(10, rate(rustfs_object_cache_access_count[5m]))
```
#### Request Performance
```promql
# P50, P95, P99 latency
histogram_quantile(0.50, rate(rustfs_get_object_total_duration_seconds_bucket[5m]))
histogram_quantile(0.95, rate(rustfs_get_object_total_duration_seconds_bucket[5m]))
histogram_quantile(0.99, rate(rustfs_get_object_total_duration_seconds_bucket[5m]))
# Request rate
rate(rustfs_get_object_requests_completed[5m])
# Average concurrent requests
avg_over_time(rustfs_concurrent_get_object_requests[5m])
# Request success rate
rate(rustfs_get_object_requests_completed[5m])
/
rate(rustfs_get_object_requests_total[5m])
```
#### Disk Contention
```promql
# Average disk permit wait time
rate(rustfs_disk_permit_wait_duration_seconds_sum[5m])
/
rate(rustfs_disk_permit_wait_duration_seconds_count[5m])
# P95 disk wait time
histogram_quantile(0.95,
rate(rustfs_disk_permit_wait_duration_seconds_bucket[5m])
)
# Percentage of time waiting for disk permits
(
rate(rustfs_disk_permit_wait_duration_seconds_sum[5m])
/
rate(rustfs_get_object_total_duration_seconds_sum[5m])
) * 100
```
#### Resource Usage
```promql
# Average response size
rate(rustfs_get_object_response_size_bytes_sum[5m])
/
rate(rustfs_get_object_response_size_bytes_count[5m])
# Average buffer size
rate(rustfs_get_object_buffer_size_bytes_sum[5m])
/
rate(rustfs_get_object_buffer_size_bytes_count[5m])
# Cache vs disk reads ratio
rate(rustfs_get_object_cache_served_total[5m])
/
(rate(rustfs_get_object_requests_completed[5m]) - rate(rustfs_get_object_cache_served_total[5m]))
```
## Performance Comparison
### Benchmark Results
| Scenario | LRU (ms) | Moka (ms) | Improvement |
|----------|----------|-----------|-------------|
| Single cache hit | 0.8 | 0.3 | 2.7x faster |
| 10 concurrent hits | 2.5 | 0.8 | 3.1x faster |
| 100 concurrent hits | 15.0 | 2.5 | 6.0x faster |
| Cache miss + insert | 1.2 | 0.5 | 2.4x faster |
| Hot key (1000 accesses) | 850 | 280 | 3.0x faster |
### Memory Usage
| Metric | LRU | Moka | Difference |
|--------|-----|------|------------|
| Overhead per entry | ~120 bytes | ~80 bytes | 33% less |
| Metadata structures | ~8KB | ~4KB | 50% less |
| Lock contention memory | High | None | 100% reduction |
## Migration Guide
### Code Changes
**Before (LRU)**:
```rust
// Manual RwLock management
let mut cache = self.cache.write().await;
if let Some(cached) = cache.get(key) {
// Manual hit count
cached.hit_count.fetch_add(1, Ordering::Relaxed);
return Some(Arc::clone(&cached.data));
}
// Manual eviction
while current + size > max {
if let Some((_, evicted)) = cache.pop_lru() {
current -= evicted.size;
}
}
```
**After (Moka)**:
```rust
// Direct access, no locks
match self.cache.get(key).await {
Some(cached) => {
// Automatic LRU promotion
cached.access_count.fetch_add(1, Ordering::Relaxed);
Some(Arc::clone(&cached.data))
}
None => None
}
// Automatic eviction by Moka
self.cache.insert(key, value).await;
```
### Configuration Changes
**Before**:
```rust
cache: RwLock::new(lru::LruCache::new(
std::num::NonZeroUsize::new(1000).unwrap()
)),
current_size: AtomicUsize::new(0),
```
**After**:
```rust
cache: Cache::builder()
.max_capacity(100 * MI_B)
.weigher(|_, v| v.size as u32)
.time_to_live(Duration::from_secs(300))
.time_to_idle(Duration::from_secs(120))
.build(),
```
### Testing Migration
All existing tests work without modification. The cache behavior is identical from an API perspective, but internal implementation is more efficient.
## Monitoring Recommendations
### Dashboard Layout
**Panel 1: Request Overview**
- Request rate (line graph)
- Concurrent requests (gauge)
- P95/P99 latency (line graph)
**Panel 2: Cache Performance**
- Hit rate percentage (gauge)
- Cache memory usage (line graph)
- Cache entry count (line graph)
**Panel 3: Cache Effectiveness**
- Objects served from cache (rate)
- Cache serve latency (histogram)
- Top cached objects (table)
**Panel 4: Disk I/O**
- Disk permit wait time (histogram)
- Disk wait percentage (gauge)
**Panel 5: Resource Usage**
- Response sizes (histogram)
- Buffer sizes (histogram)
### Alerts
**Critical**:
```promql
# Cache disabled or failing
rate(rustfs_object_cache_hits[5m]) + rate(rustfs_object_cache_misses[5m]) == 0
# Very high disk wait times
histogram_quantile(0.95,
rate(rustfs_disk_permit_wait_duration_seconds_bucket[5m])
) > 1.0
```
**Warning**:
```promql
# Low cache hit rate
(
rate(rustfs_object_cache_hits[5m])
/
(rate(rustfs_object_cache_hits[5m]) + rate(rustfs_object_cache_misses[5m]))
) < 0.5
# High concurrent requests
rustfs_concurrent_get_object_requests > 100
```
## Future Enhancements
### Short Term
1. **Dynamic TTL**: Adjust TTL based on access patterns
2. **Regional Caches**: Separate caches for different regions
3. **Compression**: Compress cached objects to save memory
### Medium Term
1. **Tiered Caching**: Memory + SSD + Remote
2. **Predictive Prefetching**: ML-based cache warming
3. **Distributed Cache**: Sync across cluster nodes
### Long Term
1. **Content-Aware Caching**: Different policies for different content types
2. **Cost-Based Eviction**: Consider fetch cost in eviction decisions
3. **Cache Analytics**: Deep analysis of access patterns
## Troubleshooting
### High Miss Rate
**Symptoms**: Cache hit rate < 50%
**Possible Causes**:
- Objects too large (> 10MB)
- High churn rate (TTL too short)
- Working set larger than cache size
**Solutions**:
```rust
// Increase cache size
.max_capacity(200 * MI_B)
// Increase TTL
.time_to_live(Duration::from_secs(600))
// Increase max object size
max_object_size: 20 * MI_B
```
### Memory Growth
**Symptoms**: Cache memory exceeds expected size
**Possible Causes**:
- Weigher function incorrect
- Too many small objects
- Memory fragmentation
**Solutions**:
```rust
// Fix weigher to include overhead
.weigher(|_k, v| (v.size + 100) as u32)
// Add min object size
if size < 1024 { return; } // Don't cache < 1KB
```
### High Disk Wait Times
**Symptoms**: P95 disk wait > 100ms
**Possible Causes**:
- Not enough disk permits
- Slow disk I/O
- Cache not effective
**Solutions**:
```rust
// Increase permits for NVMe
disk_read_semaphore: Arc::new(Semaphore::new(128))
// Improve cache hit rate
.max_capacity(500 * MI_B)
```
## References
- **Moka GitHub**: https://github.com/moka-rs/moka
- **Moka Documentation**: https://docs.rs/moka/0.12.11
- **Original Issue**: #911
- **Implementation Commit**: 3b6e281
- **Previous LRU Implementation**: Commit 010e515
## Conclusion
The migration to Moka provides:
- **10x better concurrent performance** through lock-free design
- **Automatic memory management** with TTL/TTI
- **Comprehensive metrics** for monitoring and optimization
- **Production-ready** solution with proven scalability
This implementation sets the foundation for future enhancements while immediately improving performance for concurrent workloads.
docs/MOKA_TEST_SUITE.md
-472
View File
@@ -1,472 +0,0 @@
# Moka Cache Test Suite Documentation
## Overview
This document describes the comprehensive test suite for the Moka-based concurrent GetObject optimization. The test suite validates all aspects of the concurrency management system including cache operations, buffer sizing, request tracking, and performance characteristics.
## Test Organization
### Test File Location
```
rustfs/src/storage/concurrent_get_object_test.rs
```
### Total Tests: 18
## Test Categories
### 1. Request Management Tests (3 tests)
#### test_concurrent_request_tracking
**Purpose**: Validates RAII-based request tracking
**What it tests**:
- Request count increments when guards are created
- Request count decrements when guards are dropped
- Automatic cleanup (RAII pattern)
**Expected behavior**:
```rust
let guard = ConcurrencyManager::track_request();
// count += 1
drop(guard);
// count -= 1 (automatic)
```
#### test_adaptive_buffer_sizing
**Purpose**: Validates concurrency-aware buffer size adaptation
**What it tests**:
- Buffer size reduces with increasing concurrency
- Multipliers: 1→2 req (1.0x), 3-4 (0.75x), 5-8 (0.5x), >8 (0.4x)
- Proper scaling for memory efficiency
**Test cases**:
| Concurrent Requests | Expected Multiplier | Description |
|---------------------|---------------------|-------------|
| 1-2 | 1.0 | Full buffer for throughput |
| 3-4 | 0.75 | Medium reduction |
| 5-8 | 0.5 | High concurrency |
| >8 | 0.4 | Maximum reduction |
#### test_buffer_size_bounds
**Purpose**: Validates buffer size constraints
**What it tests**:
- Minimum buffer size (64KB)
- Maximum buffer size (10MB)
- File size smaller than buffer uses file size
### 2. Cache Operations Tests (8 tests)
#### test_moka_cache_operations
**Purpose**: Basic Moka cache functionality
**What it tests**:
- Cache insertion
- Cache retrieval
- Stats accuracy (entries, size)
- Missing key handling
- Cache clearing
**Key difference from LRU**:
- Requires `sleep()` delays for Moka's async processing
- Eventual consistency model
```rust
manager.cache_object(key.clone(), data).await;
sleep(Duration::from_millis(50)).await; // Give Moka time
let cached = manager.get_cached(&key).await;
```
#### test_large_object_not_cached
**Purpose**: Validates size limit enforcement
**What it tests**:
- Objects > 10MB are rejected
- Cache remains empty after rejection
- Size limit protection
#### test_moka_cache_eviction
**Purpose**: Validates Moka's automatic eviction
**What it tests**:
- Cache stays within 100MB limit
- LRU eviction when capacity exceeded
- Automatic memory management
**Behavior**:
- Cache 20 × 6MB objects (120MB total)
- Moka automatically evicts to stay under 100MB
- Older objects evicted first (LRU)
#### test_cache_batch_operations
**Purpose**: Batch retrieval efficiency
**What it tests**:
- Multiple keys retrieved in single operation
- Mixed existing/non-existing keys handled
- Efficiency vs individual gets
**Benefits**:
- Single function call for multiple objects
- Lock-free parallel access with Moka
- Better performance than sequential gets
#### test_cache_warming
**Purpose**: Pre-population functionality
**What it tests**:
- Batch insertion via warm_cache()
- All objects successfully cached
- Startup optimization support
**Use case**: Server startup can pre-load known hot objects
#### test_hot_keys_tracking
**Purpose**: Access pattern analysis
**What it tests**:
- Per-object access counting
- Sorted results by access count
- Top-N key retrieval
**Validation**:
- Hot keys sorted descending by access count
- Most accessed objects identified correctly
- Useful for cache optimization
#### test_cache_removal
**Purpose**: Explicit cache invalidation
**What it tests**:
- Remove cached object
- Verify removal
- Handle non-existent key
**Use case**: Manual cache invalidation when data changes
#### test_is_cached_no_side_effects
**Purpose**: Side-effect-free existence check
**What it tests**:
- contains() doesn't increment access count
- Doesn't affect LRU ordering
- Lightweight check operation
**Important**: This validates that checking existence doesn't pollute metrics
### 3. Performance Tests (4 tests)
#### test_concurrent_cache_access
**Purpose**: Lock-free concurrent access validation
**What it tests**:
- 100 concurrent cache reads
- Completion time < 500ms
- No lock contention
**Moka advantage**: Lock-free design enables true parallel access
```rust
let tasks: Vec<_> = (0..100)
.map(|i| {
tokio::spawn(async move {
let _ = manager.get_cached(&key).await;
})
})
.collect();
// Should complete quickly due to lock-free design
```
#### test_cache_hit_rate
**Purpose**: Hit rate calculation validation
**What it tests**:
- Hit/miss tracking accuracy
- Percentage calculation
- 50/50 mix produces ~50% hit rate
**Metrics**:
```rust
let hit_rate = manager.cache_hit_rate();
// Returns percentage: 0.0 - 100.0
```
#### test_advanced_buffer_sizing
**Purpose**: File pattern-aware buffer optimization
**What it tests**:
- Small file optimization (< 256KB)
- Sequential read enhancement (1.5x)
- Large file + high concurrency reduction (0.8x)
**Patterns**:
| Pattern | Buffer Adjustment | Reason |
|---------|-------------------|---------|
| Small file | Reduce to 0.25x file size | Don't over-allocate |
| Sequential | Increase to 1.5x | Prefetch optimization |
| Large + concurrent | Reduce to 0.8x | Memory efficiency |
#### bench_concurrent_cache_performance
**Purpose**: Performance benchmark
**What it tests**:
- Sequential vs concurrent access
- Speedup measurement
- Lock-free advantage quantification
**Expected results**:
- Concurrent should be faster or similar
- Demonstrates Moka's scalability
- No significant slowdown under concurrency
### 4. Advanced Features Tests (3 tests)
#### test_disk_io_permits
**Purpose**: I/O rate limiting
**What it tests**:
- Semaphore permit acquisition
- 64 concurrent permits (default)
- FIFO queuing behavior
**Purpose**: Prevents disk I/O saturation
#### test_ttl_expiration
**Purpose**: TTL configuration validation
**What it tests**:
- Cache configured with TTL (5 min)
- Cache configured with TTI (2 min)
- Automatic expiration mechanism exists
**Note**: Full TTL test would require 5 minute wait; this just validates configuration
## Test Patterns and Best Practices
### Moka-Specific Patterns
#### 1. Async Processing Delays
Moka processes operations asynchronously. Always add delays after operations:
```rust
// Insert
manager.cache_object(key, data).await;
sleep(Duration::from_millis(50)).await; // Allow processing
// Bulk operations need more time
manager.warm_cache(objects).await;
sleep(Duration::from_millis(100)).await; // Allow batch processing
// Eviction tests
// ... cache many objects ...
sleep(Duration::from_millis(200)).await; // Allow eviction
```
#### 2. Eventual Consistency
Moka's lock-free design means eventual consistency:
```rust
// May not be immediately available
let cached = manager.get_cached(&key).await;
// Better: wait and retry if critical
sleep(Duration::from_millis(50)).await;
let cached = manager.get_cached(&key).await;
```
#### 3. Concurrent Testing
Use Arc for sharing across tasks:
```rust
let manager = Arc::new(ConcurrencyManager::new());
let tasks: Vec<_> = (0..100)
.map(|i| {
let mgr = Arc::clone(&manager);
tokio::spawn(async move {
// Use mgr here
})
})
.collect();
```
### Assertion Patterns
#### Descriptive Messages
Always include context in assertions:
```rust
// Bad
assert!(cached.is_some());
// Good
assert!(
cached.is_some(),
"Object {} should be cached after insertion",
key
);
```
#### Tolerance for Timing
Account for async processing and system variance:
```rust
// Allow some tolerance
assert!(
stats.entries >= 8,
"Most objects should be cached (got {}/10)",
stats.entries
);
// Rather than exact
assert_eq!(stats.entries, 10); // May fail due to timing
```
#### Range Assertions
For performance tests, use ranges:
```rust
assert!(
elapsed < Duration::from_millis(500),
"Should complete quickly, took {:?}",
elapsed
);
```
## Running Tests
### All Tests
```bash
cargo test --package rustfs concurrent_get_object
```
### Specific Test
```bash
cargo test --package rustfs test_moka_cache_operations
```
### With Output
```bash
cargo test --package rustfs concurrent_get_object -- --nocapture
```
### Specific Test with Output
```bash
cargo test --package rustfs test_concurrent_cache_access -- --nocapture
```
## Performance Expectations
| Test | Expected Duration | Notes |
|------|-------------------|-------|
| test_concurrent_request_tracking | <50ms | Simple counter ops |
| test_moka_cache_operations | <100ms | Single object ops |
| test_cache_eviction | <500ms | Many insertions + eviction |
| test_concurrent_cache_access | <500ms | 100 concurrent tasks |
| test_cache_warming | <200ms | 5 object batch |
| bench_concurrent_cache_performance | <1s | Comparative benchmark |
## Debugging Failed Tests
### Common Issues
#### 1. Timing Failures
**Symptom**: Test fails intermittently
**Cause**: Moka async processing not complete
**Fix**: Increase sleep duration
```rust
// Before
sleep(Duration::from_millis(50)).await;
// After
sleep(Duration::from_millis(100)).await;
```
#### 2. Assertion Exact Match
**Symptom**: Expected exact count, got close
**Cause**: Async processing, eviction timing
**Fix**: Use range assertions
```rust
// Before
assert_eq!(stats.entries, 10);
// After
assert!(stats.entries >= 8 && stats.entries <= 10);
```
#### 3. Concurrent Test Failures
**Symptom**: Concurrent tests timeout or fail
**Cause**: Resource contention, slow system
**Fix**: Increase timeout, reduce concurrency
```rust
// Before
let tasks: Vec<_> = (0..1000).map(...).collect();
// After
let tasks: Vec<_> = (0..100).map(...).collect();
```
## Test Coverage Report
### By Feature
| Feature | Tests | Coverage |
|---------|-------|----------|
| Request tracking | 1 | ✅ Complete |
| Buffer sizing | 3 | ✅ Complete |
| Cache operations | 5 | ✅ Complete |
| Batch operations | 2 | ✅ Complete |
| Hot keys | 1 | ✅ Complete |
| Hit rate | 1 | ✅ Complete |
| Eviction | 1 | ✅ Complete |
| TTL/TTI | 1 | ✅ Complete |
| Concurrent access | 2 | ✅ Complete |
| Disk I/O control | 1 | ✅ Complete |
### By API Method
| Method | Tested | Test Name |
|--------|--------|-----------|
| `track_request()` | ✅ | test_concurrent_request_tracking |
| `get_cached()` | ✅ | test_moka_cache_operations |
| `cache_object()` | ✅ | test_moka_cache_operations |
| `cache_stats()` | ✅ | test_moka_cache_operations |
| `clear_cache()` | ✅ | test_moka_cache_operations |
| `is_cached()` | ✅ | test_is_cached_no_side_effects |
| `get_cached_batch()` | ✅ | test_cache_batch_operations |
| `remove_cached()` | ✅ | test_cache_removal |
| `get_hot_keys()` | ✅ | test_hot_keys_tracking |
| `cache_hit_rate()` | ✅ | test_cache_hit_rate |
| `warm_cache()` | ✅ | test_cache_warming |
| `acquire_disk_read_permit()` | ✅ | test_disk_io_permits |
| `buffer_size()` | ✅ | test_advanced_buffer_sizing |
## Continuous Integration
### Pre-commit Hook
```bash
# Run all concurrency tests before commit
cargo test --package rustfs concurrent_get_object
```
### CI Pipeline
```yaml
- name: Test Concurrency Features
run: |
cargo test --package rustfs concurrent_get_object -- --nocapture
cargo test --package rustfs bench_concurrent_cache_performance -- --nocapture
```
## Future Test Enhancements
### Planned Tests
1. **Distributed cache coherency** - Test cache sync across nodes
2. **Memory pressure** - Test behavior under low memory
3. **Long-running TTL** - Full TTL expiration cycle
4. **Cache poisoning resistance** - Test malicious inputs
5. **Metrics accuracy** - Validate all Prometheus metrics
### Performance Benchmarks
1. **Latency percentiles** - P50, P95, P99 under load
2. **Throughput scaling** - Requests/sec vs concurrency
3. **Memory efficiency** - Memory usage vs cache size
4. **Eviction overhead** - Cost of eviction operations
## Conclusion
The Moka test suite provides comprehensive coverage of all concurrency features with proper handling of Moka's async, lock-free design. The tests validate both functional correctness and performance characteristics, ensuring the optimization delivers the expected improvements.
**Key Achievements**:
- ✅ 18 comprehensive tests
- ✅ 100% API coverage
- ✅ Performance validation
- ✅ Moka-specific patterns documented
- ✅ Production-ready test suite
docs/PERFORMANCE_TESTING.md
-329
View File
@@ -1,329 +0,0 @@
# RustFS Performance Testing Guide
This document describes the recommended tools and workflows for benchmarking RustFS and analyzing performance bottlenecks.
## Overview
RustFS exposes several complementary tooling options:
1. **Profiling** – collect CPU samples through the built-in `pprof` endpoints.
2. **Load testing** – drive concurrent requests with dedicated client utilities.
3. **Monitoring and analysis** – inspect collected metrics to locate hotspots.
## Prerequisites
### 1. Enable profiling support
Set the profiling environment variable before launching RustFS:
```bash
export RUSTFS_ENABLE_PROFILING=true
./rustfs
```
### 2. Install required tooling
Make sure the following dependencies are available:
```bash
# Base tools
curl # HTTP requests
jq # JSON processing (optional)
# Analysis tools
go # Go pprof CLI (optional, required for protobuf output)
python3 # Python load-testing scripts
# macOS users
brew install curl jq go python3
# Ubuntu/Debian users
sudo apt-get install curl jq golang-go python3
```
## Performance Testing Methods
### Method 1: Use the dedicated profiling script (recommended)
The repository ships with a helper script for common profiling flows:
```bash
# Show command help
./scripts/profile_rustfs.sh help
# Check profiler status
./scripts/profile_rustfs.sh status
# Capture a 30 second flame graph
./scripts/profile_rustfs.sh flamegraph
# Download protobuf-formatted samples
./scripts/profile_rustfs.sh protobuf
# Collect both formats
./scripts/profile_rustfs.sh both
# Provide custom arguments
./scripts/profile_rustfs.sh -d 60 -u http://192.168.1.100:9000 both
```
### Method 2: Run the Python end-to-end tester
A Python utility combines background load generation with profiling:
```bash
# Launch the integrated test harness
python3 test_load.py
```
The script will:
1. Launch multi-threaded S3 operations as load.
2. Pull profiling samples in parallel.
3. Produce a flame graph for investigation.
### Method 3: Simple shell-based load test
For quick smoke checks, a lightweight bash script is also provided:
```bash
# Execute a lightweight benchmark
./simple_load_test.sh
```
## Profiling Output Formats
### 1. Flame graph (SVG)
- **Purpose**: Visualize CPU time distribution.
- **File name**: `rustfs_profile_TIMESTAMP.svg`
- **How to view**: Open the SVG in a browser.
- **Interpretation tips**:
- Width reflects CPU time per function.
- Height illustrates call-stack depth.
- Click to zoom into specific frames.
```bash
# Example: open the file in a browser
open profiles/rustfs_profile_20240911_143000.svg
```
### 2. Protobuf samples
- **Purpose**: Feed data to the `go tool pprof` command.
- **File name**: `rustfs_profile_TIMESTAMP.pb`
- **Tooling**: `go tool pprof`
```bash
# Analyze the protobuf output
go tool pprof profiles/rustfs_profile_20240911_143000.pb
# Common pprof commands
(pprof) top # Show hottest call sites
(pprof) list func # Display annotated source for a function
(pprof) web # Launch the web UI (requires graphviz)
(pprof) png # Render a PNG flame chart
(pprof) help # List available commands
```
## API Usage
### Check profiling status
```bash
curl "http://127.0.0.1:9000/rustfs/admin/debug/pprof/status"
```
Sample response:
```json
{
"enabled": "true",
"sampling_rate": "100"
}
```
### Capture profiling data
```bash
# Fetch a 30-second flame graph
curl "http://127.0.0.1:9000/rustfs/admin/debug/pprof/profile?seconds=30&format=flamegraph" \
-o profile.svg
# Fetch protobuf output
curl "http://127.0.0.1:9000/rustfs/admin/debug/pprof/profile?seconds=30&format=protobuf" \
-o profile.pb
```
**Parameters**
- `seconds`: Duration between 1 and 300 seconds.
- `format`: Output format (`flamegraph`/`svg` or `protobuf`/`pb`).
## Load Testing Scenarios
### 1. S3 API workload
Use the Python harness to exercise a complete S3 workflow:
```python
# Basic configuration
tester = S3LoadTester(
endpoint="http://127.0.0.1:9000",
access_key="rustfsadmin",
secret_key="rustfsadmin"
)
# Execute the load test
# Four threads, ten operations each
tester.run_load_test(num_threads=4, operations_per_thread=10)
```
Each iteration performs:
1. Upload a 1 MB object.
2. Download the object.
3. Delete the object.
### 2. Custom load scenarios
```bash
# Create a test bucket
curl -X PUT "http://127.0.0.1:9000/test-bucket"
# Concurrent uploads
for i in {1..10}; do
echo "test data $i" | curl -X PUT "http://127.0.0.1:9000/test-bucket/object-$i" -d @- &
done
wait
# Concurrent downloads
for i in {1..10}; do
curl "http://127.0.0.1:9000/test-bucket/object-$i" > /dev/null &
done
wait
```
## Profiling Best Practices
### 1. Environment preparation
- Confirm that `RUSTFS_ENABLE_PROFILING=true` is set.
- Use an isolated benchmark environment to avoid interference.
- Reserve disk space for generated profile artifacts.
### 2. Data collection tips
- **Warm-up**: Run a light workload for 5–10 minutes before sampling.
- **Sampling window**: Capture 30–60 seconds under steady load.
- **Multiple samples**: Take several runs to compare results.
### 3. Analysis focus areas
When inspecting flame graphs, pay attention to:
1. **The widest frames** – most CPU time consumed.
2. **Flat plateaus** – likely bottlenecks.
3. **Deep call stacks** – recursion or complex logic.
4. **Unexpected syscalls** – I/O stalls or allocation churn.
### 4. Common issues
- **Lock contention**: Investigate frames under `std::sync`.
- **Memory allocation**: Search for `alloc`-related frames.
- **I/O wait**: Review filesystem or network call stacks.
- **Serialization overhead**: Look for JSON/XML parsing hotspots.
## Troubleshooting
### 1. Profiling disabled
Error: `{"enabled":"false"}`
**Fix**:
```bash
export RUSTFS_ENABLE_PROFILING=true
# Restart RustFS
```
### 2. Connection refused
Error: `Connection refused`
**Checklist**:
- Confirm RustFS is running.
- Ensure the port number is correct (default 9000).
- Verify firewall rules.
### 3. Oversized profile output
If artifacts become too large:
- Shorten the capture window (e.g., 15–30 seconds).
- Reduce load-test concurrency.
- Prefer protobuf output instead of SVG.
## Configuration Parameters
### Environment variables
| Variable | Default | Description |
|------|--------|------|
| `RUSTFS_ENABLE_PROFILING` | `false` | Enable profiling support |
| `RUSTFS_URL` | `http://127.0.0.1:9000` | RustFS endpoint |
| `PROFILE_DURATION` | `30` | Profiling duration in seconds |
| `OUTPUT_DIR` | `./profiles` | Output directory |
### Script arguments
```bash
./scripts/profile_rustfs.sh [OPTIONS] [COMMAND]
OPTIONS:
-u, --url URL RustFS URL
-d, --duration SECONDS Profile duration
-o, --output DIR Output directory
COMMANDS:
status Check profiler status
flamegraph Collect a flame graph
protobuf Collect protobuf samples
both Collect both formats (default)
```
## Output Locations
- **Script output**: `./profiles/`
- **Python script**: `/tmp/rustfs_profiles/`
- **File naming**: `rustfs_profile_TIMESTAMP.{svg|pb}`
## Example Workflow
1. **Launch RustFS**
```bash
RUSTFS_ENABLE_PROFILING=true ./rustfs
```
2. **Verify profiling availability**
```bash
./scripts/profile_rustfs.sh status
```
3. **Start a load test**
```bash
python3 test_load.py &
```
4. **Collect samples**
```bash
./scripts/profile_rustfs.sh -d 60 both
```
5. **Inspect the results**
```bash
# Review the flame graph
open profiles/rustfs_profile_*.svg
# Or analyze the protobuf output
go tool pprof profiles/rustfs_profile_*.pb
```
Following this workflow helps you understand RustFS performance characteristics, locate bottlenecks, and implement targeted optimizations.
docs/PHASE4_GUIDE.md
-383
View File
@@ -1,383 +0,0 @@
# Phase 4: Full Integration Guide
## Overview
Phase 4 represents the final stage of the adaptive buffer sizing migration path. It provides a unified, profile-based implementation with deprecated legacy functions and optional performance metrics.
## What's New in Phase 4
### 1. Deprecated Legacy Function
The `get_adaptive_buffer_size()` function is now deprecated:
```rust
#[deprecated(
since = "Phase 4",
note = "Use workload profile configuration instead."
)]
fn get_adaptive_buffer_size(file_size: i64) -> usize
```
**Why Deprecated?**
- Profile-based approach is more flexible and powerful
- Encourages use of the unified configuration system
- Simplifies maintenance and future enhancements
**Still Works:**
- Function is maintained for backward compatibility
- Internally delegates to GeneralPurpose profile
- No breaking changes for existing code
### 2. Profile-Only Implementation
All buffer sizing now goes through workload profiles:
**Before (Phase 3):**
```rust
fn get_buffer_size_opt_in(file_size: i64) -> usize {
if is_buffer_profile_enabled() {
// Use profiles
} else {
// Fall back to hardcoded get_adaptive_buffer_size()
}
}
```
**After (Phase 4):**
```rust
fn get_buffer_size_opt_in(file_size: i64) -> usize {
if is_buffer_profile_enabled() {
// Use configured profile
} else {
// Use GeneralPurpose profile (no hardcoded values)
}
}
```
**Benefits:**
- Consistent behavior across all modes
- Single source of truth for buffer sizes
- Easier to test and maintain
### 3. Performance Metrics
Optional metrics collection for monitoring and optimization:
```rust
#[cfg(feature = "metrics")]
{
metrics::histogram!("buffer_size_bytes", buffer_size as f64);
metrics::counter!("buffer_size_selections", 1);
if file_size >= 0 {
let ratio = buffer_size as f64 / file_size as f64;
metrics::histogram!("buffer_to_file_ratio", ratio);
}
}
```
## Migration Guide
### From Phase 3 to Phase 4
**Good News:** No action required for most users!
Phase 4 is fully backward compatible with Phase 3. Your existing configurations and deployments continue to work without changes.
### If You Have Custom Code
If your code directly calls `get_adaptive_buffer_size()`:
**Option 1: Update to use the profile system (Recommended)**
```rust
// Old code
let buffer_size = get_adaptive_buffer_size(file_size);
// New code - let the system handle it
// (buffer sizing happens automatically in put_object, upload_part, etc.)
```
**Option 2: Suppress deprecation warnings**
```rust
// If you must keep calling it directly
#[allow(deprecated)]
let buffer_size = get_adaptive_buffer_size(file_size);
```
**Option 3: Use the new API explicitly**
```rust
// Use the profile system directly
use rustfs::config::workload_profiles::{WorkloadProfile, RustFSBufferConfig};
let config = RustFSBufferConfig::new(WorkloadProfile::GeneralPurpose);
let buffer_size = config.get_buffer_size(file_size);
```
## Performance Metrics
### Enabling Metrics
**At Build Time:**
```bash
cargo build --features metrics --release
```
**In Cargo.toml:**
```toml
[dependencies]
rustfs = { version = "*", features = ["metrics"] }
```
### Available Metrics
| Metric Name | Type | Description |
|------------|------|-------------|
| `buffer_size_bytes` | Histogram | Distribution of selected buffer sizes |
| `buffer_size_selections` | Counter | Total number of buffer size calculations |
| `buffer_to_file_ratio` | Histogram | Ratio of buffer size to file size |
### Using Metrics
**With Prometheus:**
```rust
// Metrics are automatically exported to Prometheus format
// Access at http://localhost:9090/metrics
```
**With Custom Backend:**
```rust
// Use the metrics crate's recorder interface
use metrics_exporter_prometheus::PrometheusBuilder;
PrometheusBuilder::new()
.install()
.expect("failed to install Prometheus recorder");
```
### Analyzing Metrics
**Buffer Size Distribution:**
```promql
# Most common buffer sizes
histogram_quantile(0.5, buffer_size_bytes) # Median
histogram_quantile(0.95, buffer_size_bytes) # 95th percentile
histogram_quantile(0.99, buffer_size_bytes) # 99th percentile
```
**Buffer Efficiency:**
```promql
# Average ratio of buffer to file size
avg(buffer_to_file_ratio)
# Files where buffer is > 10% of file size
buffer_to_file_ratio > 0.1
```
**Usage Patterns:**
```promql
# Rate of buffer size selections
rate(buffer_size_selections[5m])
# Total selections over time
increase(buffer_size_selections[1h])
```
## Optimizing Based on Metrics
### Scenario 1: High Memory Usage
**Symptom:** Most buffers are at maximum size
```promql
histogram_quantile(0.9, buffer_size_bytes) > 1048576 # 1MB
```
**Solution:**
- Switch to a more conservative profile
- Use SecureStorage or WebWorkload profile
- Or create custom profile with lower max_size
### Scenario 2: Poor Throughput
**Symptom:** Buffer-to-file ratio is very small
```promql
avg(buffer_to_file_ratio) < 0.01 # Less than 1%
```
**Solution:**
- Switch to a more aggressive profile
- Use AiTraining or DataAnalytics profile
- Increase buffer sizes for your workload
### Scenario 3: Mismatched Profile
**Symptom:** Wide distribution of file sizes with single profile
```promql
# High variance in buffer sizes
stddev(buffer_size_bytes) > 500000
```
**Solution:**
- Consider per-bucket profiles (future feature)
- Use GeneralPurpose for mixed workloads
- Or implement custom thresholds
## Testing Phase 4
### Unit Tests
Run the Phase 4 specific tests:
```bash
cd /home/runner/work/rustfs/rustfs
cargo test test_phase4_full_integration
```
### Integration Tests
Test with different configurations:
```bash
# Test default behavior
./rustfs /data
# Test with different profiles
export RUSTFS_BUFFER_PROFILE=AiTraining
./rustfs /data
# Test opt-out mode
export RUSTFS_BUFFER_PROFILE_DISABLE=true
./rustfs /data
```
### Metrics Verification
With metrics enabled:
```bash
# Build with metrics
cargo build --features metrics --release
# Run and check metrics endpoint
./target/release/rustfs /data &
curl http://localhost:9090/metrics | grep buffer_size
```
## Troubleshooting
### Q: I'm getting deprecation warnings
**A:** You're calling `get_adaptive_buffer_size()` directly. Options:
1. Remove the direct call (let the system handle it)
2. Use `#[allow(deprecated)]` to suppress warnings
3. Migrate to the profile system API
### Q: How do I know which profile is being used?
**A:** Check the startup logs:
```
Buffer profiling is enabled by default (Phase 3), profile: GeneralPurpose
Using buffer profile: GeneralPurpose
```
### Q: Can I still opt-out in Phase 4?
**A:** Yes! Use `--buffer-profile-disable`:
```bash
export RUSTFS_BUFFER_PROFILE_DISABLE=true
./rustfs /data
```
This uses GeneralPurpose profile (same buffer sizes as PR #869).
### Q: What's the difference between opt-out in Phase 3 vs Phase 4?
**A:**
- **Phase 3**: Opt-out uses hardcoded legacy function
- **Phase 4**: Opt-out uses GeneralPurpose profile
- **Result**: Identical buffer sizes, but Phase 4 is profile-based
### Q: Do I need to enable metrics?
**A:** No, metrics are completely optional. They're useful for:
- Production monitoring
- Performance analysis
- Profile optimization
- Capacity planning
If you don't need these, skip the metrics feature.
## Best Practices
### 1. Let the System Handle Buffer Sizing
**Don't:**
```rust
// Avoid direct calls
let buffer_size = get_adaptive_buffer_size(file_size);
let reader = BufReader::with_capacity(buffer_size, file);
```
**Do:**
```rust
// Let put_object/upload_part handle it automatically
// Buffer sizing happens transparently
```
### 2. Use Appropriate Profiles
Match your profile to your workload:
- AI/ML models: `AiTraining`
- Static assets: `WebWorkload`
- Mixed files: `GeneralPurpose`
- Compliance: `SecureStorage`
### 3. Monitor in Production
Enable metrics in production:
```bash
cargo build --features metrics --release
```
Use the data to:
- Validate profile choice
- Identify optimization opportunities
- Plan capacity
### 4. Test Profile Changes
Before changing profiles in production:
```bash
# Test in staging
export RUSTFS_BUFFER_PROFILE=AiTraining
./rustfs /staging-data
# Monitor metrics for a period
# Compare with baseline
# Roll out to production when validated
```
## Future Enhancements
Based on collected metrics, future versions may include:
1. **Auto-tuning**: Automatically adjust profiles based on observed patterns
2. **Per-bucket profiles**: Different profiles for different buckets
3. **Dynamic thresholds**: Adjust thresholds based on system load
4. **ML-based optimization**: Use machine learning to optimize buffer sizes
5. **Adaptive limits**: Automatically adjust max_size based on available memory
## Conclusion
Phase 4 represents the mature state of the adaptive buffer sizing system:
- ✅ Unified, profile-based implementation
- ✅ Deprecated legacy code (but backward compatible)
- ✅ Optional performance metrics
- ✅ Production-ready and battle-tested
- ✅ Future-proof and extensible
Most users can continue using the system without any changes, while advanced users gain powerful new capabilities for monitoring and optimization.
## References
- [Adaptive Buffer Sizing Guide](./adaptive-buffer-sizing.md)
- [Implementation Summary](./IMPLEMENTATION_SUMMARY.md)
- [Phase 3 Migration Guide](./MIGRATION_PHASE3.md)
- [Performance Testing Guide](./PERFORMANCE_TESTING.md)
docs/README.md
-250
View File
@@ -1,250 +0,0 @@
# RustFS Documentation Center
Welcome to the RustFS distributed file system documentation center!
## 📚 Documentation Navigation
### ⚡ Performance Optimization
RustFS provides intelligent performance optimization features for different workloads.
| Document | Description | Audience |
|------|------|----------|
| [Adaptive Buffer Sizing](./adaptive-buffer-sizing.md) | Intelligent buffer sizing optimization for optimal performance across workload types | Developers and system administrators |
| [Phase 3 Migration Guide](./MIGRATION_PHASE3.md) | Migration guide from Phase 2 to Phase 3 (Default Enablement) | Operations and DevOps teams |
| [Phase 4 Full Integration Guide](./PHASE4_GUIDE.md) | Complete guide to Phase 4 features: deprecated legacy functions, performance metrics | Advanced users and performance engineers |
| [Performance Testing Guide](./PERFORMANCE_TESTING.md) | Performance benchmarking and optimization guide | Performance engineers |
### 🔐 KMS (Key Management Service)
RustFS KMS delivers enterprise-grade key management and data encryption.
| Document | Description | Audience |
|------|------|----------|
| [KMS User Guide](./kms/README.md) | Comprehensive KMS guide with quick start, configuration, and deployment steps | Required reading for all users |
| [HTTP API Reference](./kms/http-api.md) | HTTP REST API reference with usage examples | Administrators and operators |
| [Programming API Reference](./kms/api.md) | Rust library APIs and code samples | Developers |
| [Configuration Reference](./kms/configuration.md) | Complete configuration options and environment variables | System administrators |
| [Troubleshooting](./kms/troubleshooting.md) | Diagnosis tips and solutions for common issues | Operations engineers |
| [Security Guide](./kms/security.md) | Security best practices and compliance guidance | Security architects |
## 🚀 Quick Start
### 1. Deploy KMS in 5 Minutes
**Production (Vault backend)**
```bash
# 1. Enable the Vault feature flag
cargo build --features vault --release
# 2. Configure environment variables
export RUSTFS_VAULT_ADDRESS=https://vault.company.com:8200
export RUSTFS_VAULT_TOKEN=hvs.CAESIJ...
# 3. Launch the service
./target/release/rustfs server
```
**Development & Testing (Local backend)**
```bash
# 1. Build a release binary
cargo build --release
# 2. Configure local storage
export RUSTFS_KMS_BACKEND=Local
export RUSTFS_KMS_LOCAL_KEY_DIR=/tmp/rustfs-keys
# 3. Launch the service
./target/release/rustfs server
```
### 2. S3-Compatible Encryption
```bash
# Upload an encrypted object
curl -X PUT https://rustfs.company.com/bucket/sensitive.txt \
-H "x-amz-server-side-encryption: AES256" \
--data-binary @sensitive.txt
# Download with automatic decryption
curl https://rustfs.company.com/bucket/sensitive.txt
```
## 🏗️ Architecture Overview
### Three-Layer KMS Security Architecture
```
┌─────────────────────────────────────────────────┐
│ Application Layer │
│ ┌─────────────┐ ┌─────────────┐ │
│ │ S3 API │ │ REST API │ │
│ └─────────────┘ └─────────────┘ │
├─────────────────────────────────────────────────┤
│ Encryption Layer │
│ ┌─────────────┐ Encrypt ┌─────────────────┐ │
│ │ Object Data │ ◄──────► │ Data Key (DEK) │ │
│ └─────────────┘ └─────────────────┘ │
├─────────────────────────────────────────────────┤
│ Key Management Layer │
│ ┌─────────────────┐ Encrypt ┌──────────────┐ │
│ │ Data Key (DEK) │ ◄───────│ Master Key │ │
│ └─────────────────┘ │ (Vault/HSM) │ │
│ └──────────────┘ │
└─────────────────────────────────────────────────┘
```
### Key Features
- ✅ **Multi-layer encryption**: Master Key → DEK → Object Data
- ✅ **High performance**: 1 MB streaming encryption with large file support
- ✅ **Multiple backends**: Vault (production) + Local (testing)
- ✅ **S3 compatibility**: Supports standard SSE-S3/SSE-KMS headers
- ✅ **Enterprise-ready**: Auditing, monitoring, and compliance features
## 📖 Learning Paths
### 👨‍💻 Developers
1. Read the [Programming API Reference](./kms/api.md) to learn the Rust library
2. Review the sample code to understand integration patterns
3. Consult [Troubleshooting](./kms/troubleshooting.md) when issues occur
### 👨‍💼 System Administrators
1. Start with the [KMS User Guide](./kms/README.md)
2. Learn the [HTTP API Reference](./kms/http-api.md) for management tasks
3. Study the [Configuration Reference](./kms/configuration.md) in depth
4. Configure monitoring and logging
### 👨‍🔧 Operations Engineers
1. Become familiar with the [HTTP API Reference](./kms/http-api.md) for day-to-day work
2. Master the [Troubleshooting](./kms/troubleshooting.md) procedures
3. Understand the requirements in the [Security Guide](./kms/security.md)
4. Establish operational runbooks
### 🔒 Security Architects
1. Dive into the [Security Guide](./kms/security.md)
2. Evaluate threat models and risk posture
3. Define security policies
## 🤝 Contribution Guide
We welcome community contributions!
### Documentation Contributions
```bash
# 1. Fork the repository
git clone https://github.com/your-username/rustfs.git
# 2. Create a documentation branch
git checkout -b docs/improve-kms-guide
# 3. Edit the documentation
# Update Markdown files under docs/kms/
# 4. Commit the changes
git add docs/
git commit -m "docs: improve KMS configuration examples"
# 5. Open a Pull Request
gh pr create --title "Improve KMS documentation"
```
### Documentation Guidelines
- Use clear headings and structure
- Provide runnable code examples
- Include warnings and tips where appropriate
- Support multiple usage scenarios
- Keep the content up to date
## 📞 Support & Feedback
### Getting Help
- **GitHub Issues**: https://github.com/rustfs/rustfs/issues
- **Discussion Forum**: https://github.com/rustfs/rustfs/discussions
- **Documentation Questions**: Open an issue on the relevant document
- **Security Concerns**: security@rustfs.com
### Issue Reporting Template
When reporting a problem, please provide:
```markdown
**Environment**
- RustFS version: v1.0.0
- Operating system: Ubuntu 20.04
- Rust version: 1.75.0
**Issue Description**
Summarize the problem you encountered...
**Reproduction Steps**
1. Step one
2. Step two
3. Step three
**Expected Behavior**
Describe what you expected to happen...
**Actual Behavior**
Describe what actually happened...
**Relevant Logs**
```bash
# Paste relevant log excerpts
```
**Additional Information**
Any other details that may help...
```
## 📈 Release History
| Version | Release Date | Highlights |
|------|----------|----------|
| v1.0.0 | 2024-01-15 | 🎉 First official release with full KMS functionality |
| v0.9.0 | 2024-01-01 | 🔐 KMS system refactor with performance optimizations |
| v0.8.0 | 2023-12-15 | ⚡ Streaming encryption with 1 MB block size tuning |
## 🗺️ Roadmap
### Coming Soon (v1.1.0)
- [ ] Automatic key rotation
- [ ] HSM integration support
- [ ] Web UI management console
- [ ] Additional compliance support (SOC2, HIPAA)
### Long-Term Plans
- [ ] Multi-tenant key isolation
- [ ] Key import/export tooling
- [ ] Performance benchmarking suite
- [ ] Kubernetes Operator
## 📋 Documentation Feedback
Help us improve the documentation!
**Was this documentation helpful?**
- 👍 Very helpful
- 👌 Mostly satisfied
- 👎 Needs improvement
**Suggestions for improvement:**
Share specific ideas via GitHub Issues.
---
**Last Updated**: 2024-01-15
**Documentation Version**: v1.0.0
*Thank you for using RustFS! We are committed to delivering the best distributed file system solution.*
docs/SECURITY_SUMMARY_special_chars.md
-241
View File
@@ -1,241 +0,0 @@
# Security Summary: Special Characters in Object Paths
## Overview
This document summarizes the security implications of the changes made to handle special characters in S3 object paths.
## Changes Made
### 1. Control Character Validation
**Files Modified**: `rustfs/src/storage/ecfs.rs`
**Change**: Added validation to reject object keys containing control characters:
```rust
// Validate object key doesn't contain control characters
if key.contains(['\0', '\n', '\r']) {
return Err(S3Error::with_message(
S3ErrorCode::InvalidArgument,
format!("Object key contains invalid control characters: {:?}", key)
));
}
```
**Security Impact**: ✅ **Positive**
- **Prevents injection attacks**: Null bytes, newlines, and carriage returns could be used for various injection attacks
- **Improves error messages**: Clear rejection of invalid input
- **No breaking changes**: Valid UTF-8 object names still work
- **Defense in depth**: Adds additional validation layer
### 2. Debug Logging
**Files Modified**: `rustfs/src/storage/ecfs.rs`
**Change**: Added debug logging for keys with special characters:
```rust
// Log debug info for keys with special characters
if key.contains([' ', '+', '%']) {
debug!("PUT object with special characters in key: {:?}", key);
}
```
**Security Impact**: ✅ **Neutral**
- **Information disclosure**: Debug level logs are only enabled when explicitly configured
- **Helps debugging**: Assists in diagnosing client-side encoding issues
- **No sensitive data**: Only logs the object key (which is not secret)
- **Production safe**: Debug logs disabled by default in production
## Security Considerations
### Path Traversal
**Risk**: Could special characters enable path traversal attacks?
**Analysis**: ✅ **No Risk**
- Object keys are not directly used as filesystem paths
- RustFS uses a storage abstraction layer (ecstore)
- Path sanitization occurs at multiple levels
- Our validation rejects control characters that could be used in attacks
**Evidence**:
```rust
// From path utilities - already handles path traversal
pub fn clean(path: &str) -> String {
// Normalizes paths, removes .. and . components
}
```
### URL Encoding/Decoding Vulnerabilities
**Risk**: Could double-encoding or encoding issues lead to security issues?
**Analysis**: ✅ **No Risk**
- s3s library (well-tested) handles URL decoding
- We receive already-decoded keys from s3s
- No manual URL decoding in our code (avoids double-decode bugs)
- Control character validation prevents encoded null bytes
**Evidence**:
```rust
// From s3s-0.12.0-rc.4/src/ops/mod.rs:
let decoded_uri_path = urlencoding::decode(req.uri.path())
.map_err(|_| S3ErrorCode::InvalidURI)?
.into_owned();
```
### Injection Attacks
**Risk**: Could special characters enable SQL injection, command injection, or other attacks?
**Analysis**: ✅ **No Risk**
- Object keys are not used in SQL queries (no SQL database)
- Object keys are not passed to shell commands
- Object keys are not evaluated as code
- Our control character validation prevents most injection vectors
**Mitigations**:
1. Control character rejection (null bytes, newlines)
2. UTF-8 validation (already present in Rust strings)
3. Storage layer abstraction (no direct filesystem operations)
### Information Disclosure
**Risk**: Could debug logging expose sensitive information?
**Analysis**: ✅ **Low Risk**
- Debug logs are opt-in (RUST_LOG=rustfs=debug)
- Only object keys are logged (not content)
- Object keys are part of the S3 API (not secret)
- Production deployments should not enable debug logging
**Best Practices**:
```bash
# Development
RUST_LOG=rustfs=debug ./rustfs server /data
# Production (no debug logs)
RUST_LOG=info ./rustfs server /data
```
### Denial of Service
**Risk**: Could malicious object keys cause DoS?
**Analysis**: ✅ **Low Risk**
- Control character validation has O(n) complexity (acceptable)
- No unbounded loops or recursion added
- Validation is early in the request pipeline
- AWS S3 API already has key length limits (1024 bytes)
## Vulnerability Assessment
### Known Vulnerabilities: **None**
The changes introduce:
- ✅ **Defensive validation** (improves security)
- ✅ **Better error messages** (improves UX)
- ✅ **Debug logging** (improves diagnostics)
- ❌ **No new attack vectors**
- ❌ **No security regressions**
### Security Testing
**Manual Review**: ✅ Completed
- Code reviewed for injection vulnerabilities
- URL encoding handling verified via s3s source inspection
- Path traversal risks analyzed
**Automated Testing**: ⚠️ CodeQL timed out
- CodeQL analysis timed out due to large codebase
- Changes are minimal (3 validation blocks + logging)
- No complex logic or unsafe operations added
- Recommend manual security review (completed above)
**E2E Testing**: ✅ Test suite created
- Tests cover edge cases with special characters
- Tests verify correct handling of spaces, plus signs, etc.
- Tests would catch security regressions
## Security Recommendations
### For Deployment
1. **Logging Configuration**:
- Production: `RUST_LOG=info` or `RUST_LOG=warn`
- Development: `RUST_LOG=debug` is safe
- Never log to publicly accessible locations
2. **Input Validation**:
- Our validation is defensive (not primary security)
- Trust s3s library for primary validation
- Monitor logs for validation errors
3. **Client Security**:
- Educate users to use proper S3 SDKs
- Warn against custom HTTP clients (easy to make mistakes)
- Provide client security guidelines
### For Future Development
1. **Additional Validation** (optional):
- Consider max key length validation
- Consider Unicode normalization
- Consider additional control character checks
2. **Security Monitoring**:
- Monitor for repeated validation errors (could indicate attack)
- Track unusual object key patterns
- Alert on control character rejection attempts
3. **Documentation**:
- Keep security docs updated
- Document security considerations for contributors
- Maintain threat model
## Compliance
### Standards Compliance
✅ **RFC 3986** (URI Generic Syntax):
- URL encoding handled by s3s library
- Follows standard URI rules
✅ **AWS S3 API Specification**:
- Compatible with AWS S3 behavior
- Follows object key naming rules
- Matches AWS error codes
✅ **OWASP Top 10**:
- A03:2021 – Injection: Control character validation
- A05:2021 – Security Misconfiguration: Clear error messages
- A09:2021 – Security Logging: Appropriate debug logging
## Conclusion
### Security Assessment: ✅ **APPROVED**
The changes to handle special characters in object paths:
- **Improve security** through control character validation
- **Introduce no new vulnerabilities**
- **Follow security best practices**
- **Maintain backward compatibility**
- **Are production-ready**
### Risk Level: **LOW**
- Changes are minimal and defensive
- No unsafe operations introduced
- Existing security mechanisms unchanged
- Well-tested s3s library handles encoding
### Recommendation: **MERGE**
These changes can be safely merged and deployed to production.
---
**Security Review Date**: 2025-12-09
**Reviewer**: Automated Analysis + Manual Review
**Risk Level**: Low
**Status**: Approved
**Next Review**: After deployment (monitor for any issues)
docs/adaptive-buffer-sizing.md
-765
View File
@@ -1,765 +0,0 @@
# Adaptive Buffer Sizing Optimization
RustFS implements intelligent adaptive buffer sizing optimization that automatically adjusts buffer sizes based on file size and workload type to achieve optimal balance between performance, memory usage, and security.
## Overview
The adaptive buffer sizing system provides:
- **Automatic buffer size selection** based on file size
- **Workload-specific optimizations** for different use cases
- **Special environment support** (Kylin, NeoKylin, Unity OS, etc.)
- **Memory pressure awareness** with configurable limits
- **Unknown file size handling** for streaming scenarios
## Workload Profiles
### GeneralPurpose (Default)
Balanced performance and memory usage for general-purpose workloads.
**Buffer Sizing:**
- Small files (< 1MB): 64KB buffer
- Medium files (1MB-100MB): 256KB buffer
- Large files (≥ 100MB): 1MB buffer
**Best for:**
- General file storage
- Mixed workloads
- Default configuration when workload type is unknown
### AiTraining
Optimized for AI/ML training workloads with large sequential reads.
**Buffer Sizing:**
- Small files (< 10MB): 512KB buffer
- Medium files (10MB-500MB): 2MB buffer
- Large files (≥ 500MB): 4MB buffer
**Best for:**
- Machine learning model files
- Training datasets
- Large sequential data processing
- Maximum throughput requirements
### DataAnalytics
Optimized for data analytics with mixed read-write patterns.
**Buffer Sizing:**
- Small files (< 5MB): 128KB buffer
- Medium files (5MB-200MB): 512KB buffer
- Large files (≥ 200MB): 2MB buffer
**Best for:**
- Data warehouse operations
- Analytics workloads
- Business intelligence
- Mixed access patterns
### WebWorkload
Optimized for web applications with small file intensive operations.
**Buffer Sizing:**
- Small files (< 512KB): 32KB buffer
- Medium files (512KB-10MB): 128KB buffer
- Large files (≥ 10MB): 256KB buffer
**Best for:**
- Web assets (images, CSS, JavaScript)
- Static content delivery
- CDN origin storage
- High concurrency scenarios
### IndustrialIoT
Optimized for industrial IoT with real-time streaming requirements.
**Buffer Sizing:**
- Small files (< 1MB): 64KB buffer
- Medium files (1MB-50MB): 256KB buffer
- Large files (≥ 50MB): 512KB buffer (capped for memory constraints)
**Best for:**
- Sensor data streams
- Real-time telemetry
- Edge computing scenarios
- Low latency requirements
- Memory-constrained devices
### SecureStorage
Security-first configuration with strict memory limits for compliance.
**Buffer Sizing:**
- Small files (< 1MB): 32KB buffer
- Medium files (1MB-50MB): 128KB buffer
- Large files (≥ 50MB): 256KB buffer (strict limit)
**Best for:**
- Compliance-heavy environments
- Secure government systems (Kylin, NeoKylin, UOS)
- Financial services
- Healthcare data storage
- Memory-constrained secure environments
**Auto-Detection:**
This profile is automatically selected when running on Chinese secure operating systems:
- Kylin
- NeoKylin
- UOS (Unity OS)
- OpenKylin
## Usage
### Using Default Configuration
The system automatically uses the `GeneralPurpose` profile by default:
```rust
// The buffer size is automatically calculated based on file size
// Uses GeneralPurpose profile by default
let buffer_size = get_adaptive_buffer_size(file_size);
```
### Using Specific Workload Profile
```rust
use rustfs::config::workload_profiles::WorkloadProfile;
// For AI/ML workloads
let buffer_size = get_adaptive_buffer_size_with_profile(
file_size,
Some(WorkloadProfile::AiTraining)
);
// For web workloads
let buffer_size = get_adaptive_buffer_size_with_profile(
file_size,
Some(WorkloadProfile::WebWorkload)
);
// For secure storage
let buffer_size = get_adaptive_buffer_size_with_profile(
file_size,
Some(WorkloadProfile::SecureStorage)
);
```
### Auto-Detection Mode
The system can automatically detect the runtime environment:
```rust
// Auto-detects OS environment or falls back to GeneralPurpose
let buffer_size = get_adaptive_buffer_size_with_profile(file_size, None);
```
### Custom Configuration
For specialized requirements, create a custom configuration:
```rust
use rustfs::config::workload_profiles::{BufferConfig, WorkloadProfile};
let custom_config = BufferConfig {
min_size: 16 * 1024, // 16KB minimum
max_size: 512 * 1024, // 512KB maximum
default_unknown: 128 * 1024, // 128KB for unknown sizes
thresholds: vec![
(1024 * 1024, 64 * 1024), // < 1MB: 64KB
(50 * 1024 * 1024, 256 * 1024), // 1MB-50MB: 256KB
(i64::MAX, 512 * 1024), // >= 50MB: 512KB
],
};
let profile = WorkloadProfile::Custom(custom_config);
let buffer_size = get_adaptive_buffer_size_with_profile(file_size, Some(profile));
```
## Phase 3: Default Enablement (Current Implementation)
**⚡ NEW: Workload profiles are now enabled by default!**
Starting from Phase 3, adaptive buffer sizing with workload profiles is **enabled by default** using the `GeneralPurpose` profile. This provides improved performance out-of-the-box while maintaining full backward compatibility.
### Default Behavior
```bash
# Phase 3: Profile-aware buffer sizing enabled by default with GeneralPurpose profile
./rustfs /data
```
This now automatically uses intelligent buffer sizing based on file size and workload characteristics.
### Changing the Workload Profile
```bash
# Use a different profile (AI/ML workloads)
export RUSTFS_BUFFER_PROFILE=AiTraining
./rustfs /data
# Or via command-line
./rustfs --buffer-profile AiTraining /data
# Use web workload profile
./rustfs --buffer-profile WebWorkload /data
```
### Opt-Out (Legacy Behavior)
If you need the exact behavior from PR #869 (fixed algorithm), you can disable profiling:
```bash
# Disable buffer profiling (revert to PR #869 behavior)
export RUSTFS_BUFFER_PROFILE_DISABLE=true
./rustfs /data
# Or via command-line
./rustfs --buffer-profile-disable /data
```
### Available Profile Names
The following profile names are supported (case-insensitive):
| Profile Name | Aliases | Description |
|-------------|---------|-------------|
| `GeneralPurpose` | `general` | Default balanced configuration (same as PR #869 for most files) |
| `AiTraining` | `ai` | Optimized for AI/ML workloads |
| `DataAnalytics` | `analytics` | Mixed read-write patterns |
| `WebWorkload` | `web` | Small file intensive operations |
| `IndustrialIoT` | `iot` | Real-time streaming |
| `SecureStorage` | `secure` | Security-first, memory constrained |
### Behavior Summary
**Phase 3 Default (Enabled):**
- Uses workload-aware buffer sizing with `GeneralPurpose` profile
- Provides same buffer sizes as PR #869 for most scenarios
- Allows easy switching to specialized profiles
- Buffer sizes: 64KB, 256KB, 1MB based on file size (GeneralPurpose)
**With `RUSTFS_BUFFER_PROFILE_DISABLE=true`:**
- Uses the exact original adaptive buffer sizing from PR #869
- For users who want guaranteed legacy behavior
- Buffer sizes: 64KB, 256KB, 1MB based on file size
**With Different Profiles:**
- `AiTraining`: 512KB, 2MB, 4MB - maximize throughput
- `WebWorkload`: 32KB, 128KB, 256KB - optimize concurrency
- `SecureStorage`: 32KB, 128KB, 256KB - compliance-focused
- And more...
### Migration Examples
**Phase 2 → Phase 3 Migration:**
```bash
# Phase 2 (Opt-In): Had to explicitly enable
export RUSTFS_BUFFER_PROFILE_ENABLE=true
export RUSTFS_BUFFER_PROFILE=GeneralPurpose
./rustfs /data
# Phase 3 (Default): Enabled automatically
./rustfs /data # ← Same behavior, no configuration needed!
```
**Using Different Profiles:**
```bash
# AI/ML workloads - larger buffers for maximum throughput
export RUSTFS_BUFFER_PROFILE=AiTraining
./rustfs /data
# Web workloads - smaller buffers for high concurrency
export RUSTFS_BUFFER_PROFILE=WebWorkload
./rustfs /data
# Secure environments - compliance-focused
export RUSTFS_BUFFER_PROFILE=SecureStorage
./rustfs /data
```
**Reverting to Legacy Behavior:**
```bash
# If you encounter issues or need exact PR #869 behavior
export RUSTFS_BUFFER_PROFILE_DISABLE=true
./rustfs /data
```
## Phase 4: Full Integration (Current Implementation)
**🚀 NEW: Profile-only implementation with performance metrics!**
Phase 4 represents the final stage of the adaptive buffer sizing system, providing a unified, profile-based approach with optional performance monitoring.
### Key Features
1. **Deprecated Legacy Function**
- `get_adaptive_buffer_size()` is now deprecated
- Maintained for backward compatibility only
- All new code uses the workload profile system
2. **Profile-Only Implementation**
- Single entry point: `get_buffer_size_opt_in()`
- All buffer sizes come from workload profiles
- Even "disabled" mode uses GeneralPurpose profile (no hardcoded values)
3. **Performance Metrics** (Optional)
- Built-in metrics collection with `metrics` feature flag
- Tracks buffer size selections
- Monitors buffer-to-file size ratios
- Helps optimize profile configurations
### Unified Buffer Sizing
```rust
// Phase 4: Single, unified implementation
fn get_buffer_size_opt_in(file_size: i64) -> usize {
// Enabled by default (Phase 3)
// Uses workload profiles exclusively
// Optional metrics collection
}
```
### Performance Monitoring
When compiled with the `metrics` feature flag:
```bash
# Build with metrics support
cargo build --features metrics
# Run and collect metrics
./rustfs /data
# Metrics collected:
# - buffer_size_bytes: Histogram of selected buffer sizes
# - buffer_size_selections: Counter of buffer size calculations
# - buffer_to_file_ratio: Ratio of buffer size to file size
```
### Migration from Phase 3
No action required! Phase 4 is fully backward compatible with Phase 3:
```bash
# Phase 3 usage continues to work
./rustfs /data
export RUSTFS_BUFFER_PROFILE=AiTraining
./rustfs /data
# Phase 4 adds deprecation warnings for direct legacy function calls
# (if you have custom code calling get_adaptive_buffer_size)
```
### What Changed
| Aspect | Phase 3 | Phase 4 |
|--------|---------|---------|
| Legacy Function | Active | Deprecated (still works) |
| Implementation | Hybrid (legacy fallback) | Profile-only |
| Metrics | None | Optional via feature flag |
| Buffer Source | Profiles or hardcoded | Profiles only |
### Benefits
1. **Simplified Codebase**
- Single implementation path
- Easier to maintain and optimize
- Consistent behavior across all scenarios
2. **Better Observability**
- Optional metrics for performance monitoring
- Data-driven profile optimization
- Production usage insights
3. **Future-Proof**
- No legacy code dependencies
- Easy to add new profiles
- Extensible for future enhancements
### Code Example
**Phase 3 (Still Works):**
```rust
// Enabled by default
let buffer_size = get_buffer_size_opt_in(file_size);
```
**Phase 4 (Recommended):**
```rust
// Same call, but now with optional metrics and profile-only implementation
let buffer_size = get_buffer_size_opt_in(file_size);
// Metrics automatically collected if feature enabled
```
**Deprecated (Backward Compatible):**
```rust
// This still works but generates deprecation warnings
#[allow(deprecated)]
let buffer_size = get_adaptive_buffer_size(file_size);
```
### Enabling Metrics
Add to `Cargo.toml`:
```toml
[dependencies]
rustfs = { version = "*", features = ["metrics"] }
```
Or build with feature flag:
```bash
cargo build --features metrics --release
```
### Metrics Dashboard
When metrics are enabled, you can visualize:
- **Buffer Size Distribution**: Most common buffer sizes used
- **Profile Effectiveness**: How well profiles match actual workloads
- **Memory Efficiency**: Buffer-to-file size ratios
- **Usage Patterns**: File size distribution and buffer selection trends
Use your preferred metrics backend (Prometheus, InfluxDB, etc.) to collect and visualize these metrics.
## Phase 2: Opt-In Usage (Previous Implementation)
**Note:** Phase 2 documentation is kept for historical reference. The current version uses Phase 4 (Full Integration).
<details>
<summary>Click to expand Phase 2 documentation</summary>
Starting from Phase 2 of the migration path, workload profiles can be enabled via environment variables or command-line arguments.
### Environment Variables
Enable workload profiling using these environment variables:
```bash
# Enable buffer profiling (opt-in)
export RUSTFS_BUFFER_PROFILE_ENABLE=true
# Set the workload profile
export RUSTFS_BUFFER_PROFILE=AiTraining
# Start RustFS
./rustfs /data
```
### Command-Line Arguments
Alternatively, use command-line flags:
```bash
# Enable buffer profiling with AI training profile
./rustfs --buffer-profile-enable --buffer-profile AiTraining /data
# Enable buffer profiling with web workload profile
./rustfs --buffer-profile-enable --buffer-profile WebWorkload /data
# Disable buffer profiling (use legacy behavior)
./rustfs /data
```
### Behavior
When `RUSTFS_BUFFER_PROFILE_ENABLE=false` (default in Phase 2):
- Uses the original adaptive buffer sizing from PR #869
- No breaking changes to existing deployments
- Buffer sizes: 64KB, 256KB, 1MB based on file size
When `RUSTFS_BUFFER_PROFILE_ENABLE=true`:
- Uses the configured workload profile
- Allows for workload-specific optimizations
- Buffer sizes vary based on the selected profile
</details>
## Configuration Validation
All buffer configurations are validated to ensure correctness:
```rust
let config = BufferConfig { /* ... */ };
config.validate()?; // Returns Err if invalid
```
**Validation Rules:**
- `min_size` must be > 0
- `max_size` must be >= `min_size`
- `default_unknown` must be between `min_size` and `max_size`
- Thresholds must be in ascending order
- Buffer sizes in thresholds must be within `[min_size, max_size]`
## Environment Detection
The system automatically detects special operating system environments by reading `/etc/os-release` on Linux systems:
```rust
if let Some(profile) = WorkloadProfile::detect_os_environment() {
// Returns SecureStorage profile for Kylin, NeoKylin, UOS, etc.
let buffer_size = profile.config().calculate_buffer_size(file_size);
}
```
**Detected Environments:**
- Kylin (麒麟)
- NeoKylin (中标麒麟)
- UOS / Unity OS (统信)
- OpenKylin (开放麒麟)
## Performance Considerations
### Memory Usage
Different profiles have different memory footprints:
| Profile | Min Buffer | Max Buffer | Typical Memory |
|---------|-----------|-----------|----------------|
| GeneralPurpose | 64KB | 1MB | Low-Medium |
| AiTraining | 512KB | 4MB | High |
| DataAnalytics | 128KB | 2MB | Medium |
| WebWorkload | 32KB | 256KB | Low |
| IndustrialIoT | 64KB | 512KB | Low |
| SecureStorage | 32KB | 256KB | Low |
### Throughput Impact
Larger buffers generally provide better throughput for large files by reducing system call overhead:
- **Small buffers (32-64KB)**: Lower memory, more syscalls, suitable for many small files
- **Medium buffers (128-512KB)**: Balanced approach for mixed workloads
- **Large buffers (1-4MB)**: Maximum throughput, best for large sequential reads
### Concurrency Considerations
For high-concurrency scenarios (e.g., WebWorkload):
- Smaller buffers reduce per-connection memory
- Allows more concurrent connections
- Better overall system resource utilization
## Best Practices
### 1. Choose the Right Profile
Select the profile that matches your primary workload:
```rust
// AI/ML training
WorkloadProfile::AiTraining
// Web application
WorkloadProfile::WebWorkload
// General purpose storage
WorkloadProfile::GeneralPurpose
```
### 2. Monitor Memory Usage
In production, monitor memory consumption:
```rust
// For memory-constrained environments, use smaller buffers
WorkloadProfile::SecureStorage // or IndustrialIoT
```
### 3. Test Performance
Benchmark your specific workload to verify the profile choice:
```bash
# Run performance tests with different profiles
cargo test --release -- --ignored performance_tests
```
### 4. Consider File Size Distribution
If you know your typical file sizes:
- Mostly small files (< 1MB): Use `WebWorkload` or `SecureStorage`
- Mostly large files (> 100MB): Use `AiTraining` or `DataAnalytics`
- Mixed sizes: Use `GeneralPurpose`
### 5. Compliance Requirements
For regulated environments:
```rust
// Automatically uses SecureStorage on detected secure OS
let config = RustFSBufferConfig::with_auto_detect();
// Or explicitly set SecureStorage
let config = RustFSBufferConfig::new(WorkloadProfile::SecureStorage);
```
## Integration Examples
### S3 Put Object
```rust
async fn put_object(&self, req: S3Request<PutObjectInput>) -> S3Result<S3Response<PutObjectOutput>> {
let size = req.input.content_length.unwrap_or(-1);
// Use workload-aware buffer sizing
let buffer_size = get_adaptive_buffer_size_with_profile(
size,
Some(WorkloadProfile::GeneralPurpose)
);
let body = tokio::io::BufReader::with_capacity(
buffer_size,
StreamReader::new(body)
);
// Process upload...
}
```
### Multipart Upload
```rust
async fn upload_part(&self, req: S3Request<UploadPartInput>) -> S3Result<S3Response<UploadPartOutput>> {
let size = req.input.content_length.unwrap_or(-1);
// For large multipart uploads, consider using AiTraining profile
let buffer_size = get_adaptive_buffer_size_with_profile(
size,
Some(WorkloadProfile::AiTraining)
);
let body = tokio::io::BufReader::with_capacity(
buffer_size,
StreamReader::new(body_stream)
);
// Process part upload...
}
```
## Troubleshooting
### High Memory Usage
If experiencing high memory usage:
1. Switch to a more conservative profile:
```rust
WorkloadProfile::WebWorkload // or SecureStorage
```
2. Set explicit memory limits in custom configuration:
```rust
let config = BufferConfig {
min_size: 16 * 1024,
max_size: 128 * 1024, // Cap at 128KB
// ...
};
```
### Low Throughput
If experiencing low throughput for large files:
1. Use a more aggressive profile:
```rust
WorkloadProfile::AiTraining // or DataAnalytics
```
2. Increase buffer sizes in custom configuration:
```rust
let config = BufferConfig {
max_size: 4 * 1024 * 1024, // 4MB max buffer
// ...
};
```
### Streaming/Unknown Size Handling
For chunked transfers or streaming:
```rust
// Pass -1 for unknown size
let buffer_size = get_adaptive_buffer_size_with_profile(-1, None);
// Returns the profile's default_unknown size
```
## Technical Implementation
### Algorithm
The buffer size is selected based on file size thresholds:
```rust
pub fn calculate_buffer_size(&self, file_size: i64) -> usize {
if file_size < 0 {
return self.default_unknown;
}
for (threshold, buffer_size) in &self.thresholds {
if file_size < *threshold {
return (*buffer_size).clamp(self.min_size, self.max_size);
}
}
self.max_size
}
```
### Thread Safety
All configuration structures are:
- Immutable after creation
- Safe to share across threads
- Cloneable for per-thread customization
### Performance Overhead
- Configuration lookup: O(n) where n = number of thresholds (typically 2-4)
- Negligible overhead compared to I/O operations
- Configuration can be cached per-connection
## Migration Guide
### From PR #869
The original `get_adaptive_buffer_size` function is preserved for backward compatibility:
```rust
// Old code (still works)
let buffer_size = get_adaptive_buffer_size(file_size);
// New code (recommended)
let buffer_size = get_adaptive_buffer_size_with_profile(
file_size,
Some(WorkloadProfile::GeneralPurpose)
);
```
### Upgrading Existing Code
1. **Identify workload type** for each use case
2. **Replace** `get_adaptive_buffer_size` with `get_adaptive_buffer_size_with_profile`
3. **Choose** appropriate profile
4. **Test** performance impact
## References
- [PR #869: Fix large file upload freeze with adaptive buffer sizing](https://github.com/rustfs/rustfs/pull/869)
- [Performance Testing Guide](./PERFORMANCE_TESTING.md)
- [Configuration Documentation](./ENVIRONMENT_VARIABLES.md)
## License
Copyright 2024 RustFS Team
Licensed under the Apache License, Version 2.0.
docs/ansible/REAEME.md
-106
View File
@@ -1,106 +0,0 @@
# Install rustfs with mnmd mode using ansible
This chapter show how to install rustfs with mnmd(multiple nodes multiple disks) using ansible playbook.Two installation method are available, namely binary and docker compose.
## Requirements
- Multiple nodes(At least 4 nodes,each has private IP and public IP)
- Multiple disks(At least 1 disk per nodes, 4 disks is a better choice)
- Ansible should be available
- Docker should be available(only for docker compose installation)
## Binary installation and uninstallation
### Installation
For binary installation([script installation](https://rustfs.com/en/download/),you should modify the below part of the playbook,
```
- name: Modify Target Server's hosts file
blockinfile:
path: /etc/hosts
block: |
172.92.20.199 rustfs-node1
172.92.20.200 rustfs-node2
172.92.20.201 rustfs-node3
172.92.20.202 rustfs-node4
```
Replacing the IP with your nodes' **private IP**.If you have more than 4 nodes, adding the ip in order.
Running the command to install rustfs
```
ansible-playbook --skip-tags rustfs_uninstall binary-mnmd.yml
```
After installation success, you can access the rustfs cluster via any node's public ip and 9000 port. Both default username and password are `rustfsadmin`.
### Uninstallation
Running the command to uninstall rustfs
```
ansible-playbook --tags rustfs_uninstall binary-mnmd.yml
```
## Docker compose installation and uninstallation
**NOTE**: For docker compose installation,playbook contains docker installation task,
```
tasks:
- name: Install docker
shell: |
apt-get remove -y docker docker.io containerd runc || true
apt-get update -y
apt-get install -y ca-certificates curl gnupg lsb-release
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | gpg --dearmor --yes -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://mirrors.aliyun.com/docker-ce/linux/ubuntu \
$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update -y
apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
become: yes
register: docker_installation_result
changed_when: false
- name: Installation check
debug:
var: docker_installation_result.stdout
```
If your node already has docker environment,you can add `tags` in the playbook and skip this task in the follow installation.By the way, the docker installation only for `Ubuntu` OS,if you have the different OS,you should modify this task as well.
For docker compose installation,you should also modify the below part of the playbook,
```
extra_hosts:
- "rustfs-node1:172.20.92.202"
- "rustfs-node2:172.20.92.201"
- "rustfs-node3:172.20.92.200"
- "rustfs-node4:172.20.92.199"
```
Replacing the IP with your nodes' **private IP**.If you have more than 4 nodes, adding the ip in order.
Running the command to install rustfs,
```
ansible-playbook --skip-tags docker_uninstall docker-compose-mnmd.yml
```
After installation success, you can access the rustfs cluster via any node's public ip and 9000 port. Both default username and password are `rustfsadmin`.
### Uninstallation
Running the command to uninstall rustfs
```
ansible-playbook --tags docker_uninstall docker-compose-mnmd.yml
```
docs/ansible/binary-mnmd.yml
-166
View File
@@ -1,166 +0,0 @@
---
- name: Prepare for RustFS installation
hosts: rustfs
become: yes
vars:
ansible_python_interpreter: /usr/bin/python3
remote_user: root
tasks:
- name: Create Workspace
file:
path: /opt/rustfs
state: directory
mode: '0755'
register: create_dir_result
- name: Dir Creation Result Check
debug:
msg: "RustFS dir created successfully"
when: create_dir_result.changed
- name: Modify Target Server's hosts file
blockinfile:
path: /etc/hosts
block: |
127.0.0.1 localhost
172.20.92.199 rustfs-node1
172.20.92.200 rustfs-node2
172.20.92.201 rustfs-node3
172.20.92.202 rustfs-node4
- name: Create rustfs group
group:
name: rustfs
system: yes
state: present
- name: Create rustfs user
user:
name: rustfs
shell: /bin/bash
system: yes
create_home: no
group: rustfs
state: present
- name: Get rustfs user id
command: id -u rustfs
register: rustfs_uid
changed_when: false
ignore_errors: yes
- name: Check rustfs user id
debug:
msg: "rustfs uid is {{ rustfs_uid.stdout }}"
- name: Create volume dir
file:
path: "{{ item }}"
state: directory
owner: rustfs
group: rustfs
mode: '0755'
loop:
- /data/rustfs0
- /data/rustfs1
- /data/rustfs2
- /data/rustfs3
- /var/logs/rustfs
- name: Install RustFS
hosts: rustfs
become: yes
vars:
ansible_python_interpreter: /usr/bin/python3
install_script_url: "https://rustfs.com/install_rustfs.sh"
install_script_tmp: "/opt/rustfs/install_rustfs.sh"
tags: rustfs_install
tasks:
- name: Prepare configuration file
copy:
dest: /etc/default/rustfs
content: |
RUSTFS_ACCESS_KEY=rustfsadmin
RUSTFS_SECRET_KEY=rustfsadmin
RUSTFS_VOLUMES="http://rustfs-node{1...4}:9000/data/rustfs{0...3}"
RUSTFS_ADDRESS=":9000"
RUSTFS_CONSOLE_ENABLE=true
RUSTFS_OBS_LOG_DIRECTORY="/var/logs/rustfs/"
owner: root
group: root
mode: '0644'
- name: Install unzip
apt:
name: unzip
state: present
update_cache: yes
- name: Download the rustfs install script
get_url:
url: "{{ install_script_url }}"
dest: "{{ install_script_tmp }}"
mode: '0755'
- name: Run rustfs installation script
expect:
command: bash "{{install_script_tmp}}"
responses:
'.*Enter your choice.*': "1\n"
'.*Please enter RustFS service port.*': "9000\n"
'.*Please enter RustFS console port.*': "9001\n"
'.*Please enter data storage directory.*': "http://rustfs-node{1...4}:9000/data/rustfs{0...3}\n"
timeout: 300
register: rustfs_install_result
tags:
- rustfs_install
- name: Debug installation output
debug:
var: rustfs_install_result.stdout_lines
- name: Installation confirmation
command: rustfs --version
register: rustfs_version
changed_when: false
failed_when: rustfs_version.rc != 0
- name: Show rustfs version
debug:
msg: "RustFS version is {{ rustfs_version.stdout }}"
- name: Uninstall RustFS
hosts: rustfs
become: yes
vars:
install_script_tmp: /opt/rustfs/install_rustfs.sh
ansible_python_interpreter: /usr/bin/python3
tags: rustfs_uninstall
tasks:
- name: Run rustfs uninstall script
expect:
command: bash "{{ install_script_tmp }}"
responses:
'Enter your choice.*': "2\n"
'Are you sure you want to uninstall RustFS.*': "y\n"
timeout: 300
register: rustfs_uninstall_result
tags: rustfs_uninstall
- name: Debug uninstall output
debug:
var: rustfs_uninstall_result.stdout_lines
- name: Delete data dir
file:
path: "{{ item }}"
state: absent
loop:
- /data/rustfs0
- /data/rustfs1
- /data/rustfs2
- /data/rustfs3
- /var/logs/rustfs
docs/ansible/docker-compose-mnmd.yml
-114
View File
@@ -1,114 +0,0 @@
---
- name: Prepare for RustFS installation
hosts: rustfs
become: yes
vars:
ansible_python_interpreter: /usr/bin/python3
install_script_url: "https://rustfs.com/install_rustfs.sh"
docker_compose: "/opt/rustfs/docker-compose"
remote_user: root
tasks:
- name: Install docker
tags: docker_install
shell: |
apt-get remove -y docker docker.io containerd runc || true
apt-get update -y
apt-get install -y ca-certificates curl gnupg lsb-release
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | gpg --dearmor --yes -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://mirrors.aliyun.com/docker-ce/linux/ubuntu \
$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update -y
apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
become: yes
register: docker_installation_result
changed_when: false
when: ansible_facts['distribution'] == "Ubuntu"
- name: Installation check
debug:
var: docker_installation_result.stdout
when: ansible_facts['distribution'] == "Ubuntu"
- name: Create docker compose dir
file:
path: "{{ docker_compose }}"
state: directory
mode: '0755'
- name: Prepare docker compose file
copy:
content: |
services:
rustfs:
image: rustfs/rustfs:latest
container_name: rustfs
hostname: rustfs
network_mode: host
environment:
# Use service names and correct disk indexing (1..4 to match mounted paths)
- RUSTFS_VOLUMES=http://rustfs-node{1...4}:9000/data/rustfs{1...4}
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
- RUSTFS_ACCESS_KEY=rustfsadmin
- RUSTFS_SECRET_KEY=rustfsadmin
- RUSTFS_CMD=rustfs
command: ["sh", "-c", "sleep 3 && rustfs"]
healthcheck:
test:
[
"CMD-SHELL",
"curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/health || exit 1"
]
interval: 10s
timeout: 5s
retries: 3
start_period: 30s
ports:
- "9000:9000" # API endpoint
- "9001:9001" # Console
volumes:
- rustfs-data1:/data/rustfs1
- rustfs-data2:/data/rustfs2
- rustfs-data3:/data/rustfs3
- rustfs-data4:/data/rustfs4
extra_hosts:
- "rustfs-node1:172.20.92.202"
- "rustfs-node2:172.20.92.201"
- "rustfs-node3:172.20.92.200"
- "rustfs-node4:172.20.92.199"
volumes:
rustfs-data1:
rustfs-data2:
rustfs-data3:
rustfs-data4:
dest: "{{ docker_compose }}/docker-compose.yml"
mode: '0644'
- name: Install rustfs using docker compose
tags: rustfs_install
command: docker compose -f "{{ docker_compose}}/docker-compose.yml" up -d
args:
chdir: "{{ docker_compose }}"
- name: Get docker compose output
command: docker compose ps
args:
chdir: "{{ docker_compose }}"
register: docker_compose_output
- name: Check the docker compose installation output
debug:
msg: "{{ docker_compose_output.stdout }}"
- name: Uninstall rustfs using docker compose
tags: rustfs_uninstall
command: docker compose -f "{{ docker_compose}}/docker-compose.yml" down
args:
chdir: "{{ docker_compose }}"
docs/bug_resolution_report_issue_1013.md
-174
View File
@@ -1,174 +0,0 @@
# Bug Resolution Report: Jemalloc Page Size Crash on Raspberry Pi (AArch64)
**Status:** Resolved and Verified
**Issue Reference:** GitHub Issue #1013
**Target Architecture:** Linux AArch64 (Raspberry Pi 5, Apple Silicon VMs)
**Date:** December 7, 2025
---
## 1. Executive Summary
This document details the analysis, resolution, and verification of a critical startup crash affecting `rustfs` on
Raspberry Pi 5 and other AArch64 Linux environments. The issue was identified as a memory page size mismatch between the
compiled `jemalloc` allocator (4KB) and the runtime kernel configuration (16KB).
The fix involves a dynamic, architecture-aware allocator configuration that automatically switches to `mimalloc` on
AArch64 systems while retaining the high-performance `jemalloc` for standard x86_64 server environments. This solution
ensures 100% stability on ARM hardware without introducing performance regressions on existing platforms.
---
## 2. Issue Analysis
### 2.1 Symptom
The application crashes immediately upon startup, including during simple version checks (`rustfs -version`).
**Error Message:**
```text
<jemalloc>: Unsupported system page size
```
### 2.2 Environment
* **Hardware:** Raspberry Pi 5 (and compatible AArch64 systems).
* **OS:** Debian Trixie (Linux AArch64).
* **Kernel Configuration:** 16KB system page size (common default for modern ARM performance).
### 2.3 Root Cause
The crash stems from a fundamental incompatibility in the `tikv-jemallocator` build configuration:
1. **Static Configuration:** Experimental builds of `jemalloc` are often compiled expecting a standard **4KB memory page**.
2. **Runtime Mismatch:** Modern AArch64 kernels (like on RPi 5) often use **16KB or 64KB pages** for improved TLB
efficiency.
3. **Fatal Error:** When `jemalloc` initializes, it detects that the actual system page size exceeds its compiled
support window. This is treated as an unrecoverable error, triggering an immediate panic before `main()` is even
entered.
---
## 3. Impact Assessment
### 3.1 Critical Bottleneck
**Zero-Day Blocker:** The mismatch acts as a hard blocker. The binaries produced were completely non-functional on the
impacted hardware.
### 3.2 Scope
* **Affected:** Linux AArch64 systems with non-standard (non-4KB) page sizes.
* **Unaffected:** Standard x86_64 servers, MacOS, and Windows environments.
---
## 4. Solution Strategy
### 4.1 Selected Fix: Architecture-Aware Allocator Switching
We opted to replace the allocator specifically for the problematic architecture.
* **For AArch64 (Target):** Switch to **`mimalloc`**.
* *Rationale:* `mimalloc` is a robust, high-performance allocator that is inherently agnostic to specific system
page sizes (supports 4KB/16KB/64KB natively). It is already used in `musl` builds, proving its reliability.
* **For x86_64 (Standard):** Retain **`jemalloc`**.
* *Rationale:* `jemalloc` is deeply optimized for server workloads. Keeping it ensures no changes to the performance
profile of the primary production environment.
### 4.2 Alternatives Rejected
* **Recompiling Jemalloc:** Attempting to force `jemalloc` to support 64KB pages (`--with-lg-page=16`) via
`tikv-jemallocator` features was deemed too complex and fragile. It would require forking the wrapper crate or complex
build script overrides, increasing maintenance burden.
---
## 5. Implementation Details
The fix was implemented across three key areas of the codebase to ensure "Secure by Design" principles.
### 5.1 Dependency Management (`rustfs/Cargo.toml`)
We used Cargo's platform-specific configuration to isolate dependencies. `jemalloc` is now mathematically impossible to
link on AArch64.
* **Old Config:** `jemalloc` included for all Linux GNU targets.
* **New Config:**
* `mimalloc` enabled for `not(all(target_os = "linux", target_env = "gnu", target_arch = "x86_64"))` (i.e.,
everything except Linux GNU x86_64).
* `tikv-jemallocator` restricted to `all(target_os = "linux", target_env = "gnu", target_arch = "x86_64")`.
### 5.2 Global Allocator Logic (`rustfs/src/main.rs`)
The global allocator is now conditionally selected at compile time:
```rust
#[cfg(all(target_os = "linux", target_env = "gnu", target_arch = "x86_64"))]
#[global_allocator]
static GLOBAL: tikv_jemallocator::Jemalloc = tikv_jemallocator::Jemalloc;
#[cfg(not(all(target_os = "linux", target_env = "gnu", target_arch = "x86_64")))]
#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
```
### 5.3 Safe Fallbacks (`rustfs/src/profiling.rs`)
Since `jemalloc` provides specific profiling features (memory dumping) that `mimalloc` does not mirror 1:1, we added
feature guards.
* **Guard:** `#[cfg(all(target_os = "linux", target_env = "gnu", target_arch = "x86_64"))]` (profiling enabled only on
Linux GNU x86_64)
* **Behavior:** On all other platforms (including AArch64), calls to dump memory profiles now return a "Not Supported"
error log instead of crashing or failing to compile.
---
## 6. Verification and Testing
To ensure the fix is 100% effective, we employed **Cross-Architecture Dependency Tree Analysis**. This method
mathematically proves which libraries are linked for a specific target.
### 6.1 Test 1: Replicating the Bugged Environment (AArch64)
We checked if the crashing library (`jemalloc`) was still present for the ARM64 target.
* **Command:** `cargo tree --target aarch64-unknown-linux-gnu -i tikv-jemallocator`
* **Result:** `warning: nothing to print.`
* **Conclusion:** **Passed.** `jemalloc` is completely absent from the build graph. The crash is impossible.
### 6.2 Test 2: Verifying the Fix (AArch64)
We confirmed that the safe allocator (`mimalloc`) was correctly substituted.
* **Command:** `cargo tree --target aarch64-unknown-linux-gnu -i mimalloc`
* **Result:**
```text
mimalloc v0.1.48
└── rustfs v0.0.5 ...
```
* **Conclusion:** **Passed.** The system is correctly configured to use the page-agnostic allocator.
### 6.3 Test 3: Regression Safety (x86_64)
We ensured that standard servers were not accidentally downgraded to `mimalloc` (unless desired).
* **Command:** `cargo tree --target x86_64-unknown-linux-gnu -i tikv-jemallocator`
* **Result:**
```text
tikv-jemallocator v0.6.1
└── rustfs v0.0.5 ...
```
* **Conclusion:** **Passed.** No regression. High-performance allocator retained for standard hardware.
---
## 7. Conclusion
The codebase is now **110% secure** against the "Unsupported system page size" crash.
* **Robustness:** Achieved via reliable, architecture-native allocators (`mimalloc` on ARM).
* **Stability:** Build process is deterministic; no "lucky" builds.
* **Maintainability:** Uses standard Cargo features (`cfg`) without custom build scripts or hacks.
docs/client-special-characters-guide.md
-442
View File
@@ -1,442 +0,0 @@
# Working with Special Characters in Object Names
## Overview
This guide explains how to properly handle special characters (spaces, plus signs, etc.) in S3 object names when using RustFS.
## Quick Reference
| Character | What You Type | How It's Stored | How to Access It |
|-----------|---------------|-----------------|------------------|
| Space | `my file.txt` | `my file.txt` | Use proper S3 client/SDK |
| Plus | `test+file.txt` | `test+file.txt` | Use proper S3 client/SDK |
| Percent | `test%file.txt` | `test%file.txt` | Use proper S3 client/SDK |
**Key Point**: Use a proper S3 SDK or client. They handle URL encoding automatically!
## Recommended Approach: Use S3 SDKs
The easiest and most reliable way to work with object names containing special characters is to use an official S3 SDK. These handle all encoding automatically.
### AWS CLI
```bash
# Works correctly - AWS CLI handles encoding
aws --endpoint-url=http://localhost:9000 s3 cp file.txt "s3://mybucket/path with spaces/file.txt"
aws --endpoint-url=http://localhost:9000 s3 ls "s3://mybucket/path with spaces/"
# Works with plus signs
aws --endpoint-url=http://localhost:9000 s3 cp data.json "s3://mybucket/ES+net/data.json"
```
### MinIO Client (mc)
```bash
# Configure RustFS endpoint
mc alias set myrustfs http://localhost:9000 ACCESS_KEY SECRET_KEY
# Upload with spaces in path
mc cp README.md "myrustfs/mybucket/a f+/b/c/3/README.md"
# List contents
mc ls "myrustfs/mybucket/a f+/"
mc ls "myrustfs/mybucket/a f+/b/c/3/"
# Works with plus signs
mc cp file.txt "myrustfs/mybucket/ES+net/file.txt"
```
### Python (boto3)
```python
import boto3
# Configure client
s3 = boto3.client(
's3',
endpoint_url='http://localhost:9000',
aws_access_key_id='ACCESS_KEY',
aws_secret_access_key='SECRET_KEY'
)
# Upload with spaces - boto3 handles encoding automatically
s3.put_object(
Bucket='mybucket',
Key='path with spaces/file.txt',
Body=b'file content'
)
# List objects - boto3 encodes prefix automatically
response = s3.list_objects_v2(
Bucket='mybucket',
Prefix='path with spaces/'
)
for obj in response.get('Contents', []):
print(obj['Key']) # Will print: "path with spaces/file.txt"
# Works with plus signs
s3.put_object(
Bucket='mybucket',
Key='ES+net/LHC+Data+Challenge/file.json',
Body=b'data'
)
```
### Go (AWS SDK)
```go
package main
import (
"bytes"
"fmt"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
func main() {
// Configure session
sess := session.Must(session.NewSession(&aws.Config{
Endpoint: aws.String("http://localhost:9000"),
Region: aws.String("us-east-1"),
Credentials: credentials.NewStaticCredentials("ACCESS_KEY", "SECRET_KEY", ""),
S3ForcePathStyle: aws.Bool(true),
}))
svc := s3.New(sess)
// Upload with spaces - SDK handles encoding
_, err := svc.PutObject(&s3.PutObjectInput{
Bucket: aws.String("mybucket"),
Key: aws.String("path with spaces/file.txt"),
Body: bytes.NewReader([]byte("content")),
})
if err != nil {
panic(err)
}
// List objects - SDK handles encoding
result, err := svc.ListObjectsV2(&s3.ListObjectsV2Input{
Bucket: aws.String("mybucket"),
Prefix: aws.String("path with spaces/"),
})
if err != nil {
panic(err)
}
for _, obj := range result.Contents {
fmt.Println(*obj.Key)
}
}
```
### Node.js (AWS SDK v3)
```javascript
const { S3Client, PutObjectCommand, ListObjectsV2Command } = require("@aws-sdk/client-s3");
// Configure client
const client = new S3Client({
endpoint: "http://localhost:9000",
region: "us-east-1",
credentials: {
accessKeyId: "ACCESS_KEY",
secretAccessKey: "SECRET_KEY",
},
forcePathStyle: true,
});
// Upload with spaces - SDK handles encoding
async function upload() {
const command = new PutObjectCommand({
Bucket: "mybucket",
Key: "path with spaces/file.txt",
Body: "file content",
});
await client.send(command);
}
// List objects - SDK handles encoding
async function list() {
const command = new ListObjectsV2Command({
Bucket: "mybucket",
Prefix: "path with spaces/",
});
const response = await client.send(command);
for (const obj of response.Contents || []) {
console.log(obj.Key);
}
}
```
## Advanced: Manual HTTP Requests
**⚠️ Not Recommended**: Only use if you can't use an S3 SDK.
If you must make raw HTTP requests, you need to manually URL-encode the object key in the path:
### URL Encoding Rules
| Character | Encoding | Example |
|-----------|----------|---------|
| Space | `%20` | `my file.txt` → `my%20file.txt` |
| Plus | `%2B` | `test+file.txt` → `test%2Bfile.txt` |
| Percent | `%25` | `test%file.txt` → `test%25file.txt` |
| Slash (in name) | `%2F` | `test/file.txt` → `test%2Ffile.txt` |
**Important**: In URL **paths** (not query parameters):
- `%20` = space
- `+` = literal plus sign (NOT space!)
- To represent a plus sign, use `%2B`
### Example: Manual curl Request
```bash
# Upload object with spaces
curl -X PUT "http://localhost:9000/mybucket/path%20with%20spaces/file.txt" \
-H "Authorization: AWS4-HMAC-SHA256 ..." \
-d "file content"
# Upload object with plus signs
curl -X PUT "http://localhost:9000/mybucket/ES%2Bnet/file.txt" \
-H "Authorization: AWS4-HMAC-SHA256 ..." \
-d "data"
# List objects (prefix in query parameter)
curl "http://localhost:9000/mybucket?prefix=path%20with%20spaces/"
# Note: You'll also need to compute AWS Signature V4
# This is complex - use an SDK instead!
```
## Troubleshooting
### Issue: "UI can navigate to folder but can't list contents"
**Symptom**:
- You uploaded: `mc cp file.txt "myrustfs/bucket/a f+/b/c/file.txt"`
- You can see folder `"a f+"` in the UI
- But clicking on it shows "No Objects"
**Root Cause**: The UI may not be properly URL-encoding the prefix when making the LIST request.
**Solution**:
1. **Use CLI instead**: `mc ls "myrustfs/bucket/a f+/b/c/"` works correctly
2. **Check UI console**: Open browser DevTools, look at Network tab, check if the request is properly encoded
3. **Report UI bug**: If using RustFS web console, this is a UI bug to report
**Workaround**:
Use the CLI for operations with special characters until UI is fixed.
### Issue: "400 Bad Request: Invalid Argument"
**Symptom**:
```
Error: api error InvalidArgument: Invalid argument
```
**Possible Causes**:
1. **Client not encoding plus signs**
- Problem: Client sends `/bucket/ES+net/file.txt`
- Solution: Client should send `/bucket/ES%2Bnet/file.txt`
- Fix: Use a proper S3 SDK
2. **Control characters in key**
- Problem: Key contains null bytes, newlines, etc.
- Solution: Remove invalid characters from key name
3. **Double-encoding**
- Problem: Client encodes twice: `%20` → `%2520`
- Solution: Only encode once, or use SDK
**Debugging**:
Enable debug logging on RustFS:
```bash
RUST_LOG=rustfs=debug ./rustfs server /data
```
Look for log lines like:
```
DEBUG rustfs::storage::ecfs: PUT object with special characters in key: "a f+/file.txt"
DEBUG rustfs::storage::ecfs: LIST objects with special characters in prefix: "ES+net/"
```
### Issue: "NoSuchKey error but file exists"
**Symptom**:
- Upload: `PUT /bucket/test+file.txt` works
- List: `GET /bucket?prefix=test` shows: `test+file.txt`
- Get: `GET /bucket/test+file.txt` fails with NoSuchKey
**Root Cause**: Key was stored with one encoding, requested with another.
**Diagnosis**:
```bash
# Check what name is actually stored
mc ls --recursive myrustfs/bucket/
# Try different encodings
curl "http://localhost:9000/bucket/test+file.txt" # Literal +
curl "http://localhost:9000/bucket/test%2Bfile.txt" # Encoded +
curl "http://localhost:9000/bucket/test%20file.txt" # Space (if + was meant as space)
```
**Solution**: Use a consistent S3 client/SDK for all operations.
### Issue: "Special characters work in CLI but not in UI"
**Root Cause**: This is a UI bug. The backend (RustFS) handles special characters correctly when accessed via proper S3 clients.
**Verification**:
```bash
# These should all work:
mc cp file.txt "myrustfs/bucket/test with spaces/file.txt"
mc ls "myrustfs/bucket/test with spaces/"
aws --endpoint-url=http://localhost:9000 s3 cp file.txt "s3://bucket/test with spaces/file.txt"
aws --endpoint-url=http://localhost:9000 s3 ls "s3://bucket/test with spaces/"
```
**Solution**: Report as UI bug. Use CLI for now.
## Best Practices
### 1. Use Simple Names When Possible
Avoid special characters if you don't need them:
- ✅ Good: `my-file.txt`, `data_2024.json`, `report-final.pdf`
- ⚠️ Acceptable but complex: `my file.txt`, `data+backup.json`, `report (final).pdf`
### 2. Always Use S3 SDKs/Clients
Don't try to build raw HTTP requests yourself. Use:
- AWS CLI
- MinIO client (mc)
- AWS SDKs (Python/boto3, Go, Node.js, Java, etc.)
- Other S3-compatible SDKs
### 3. Understand URL Encoding
If you must work with URLs directly:
- **In URL paths**: Space=`%20`, Plus=`%2B`, `+` means literal plus
- **In query params**: Space=`%20` or `+`, Plus=`%2B`
- Use a URL encoding library in your language
### 4. Test Your Client
Before deploying:
```bash
# Test with spaces
mc cp test.txt "myrustfs/bucket/test with spaces/file.txt"
mc ls "myrustfs/bucket/test with spaces/"
# Test with plus
mc cp test.txt "myrustfs/bucket/test+plus/file.txt"
mc ls "myrustfs/bucket/test+plus/"
# Test with mixed
mc cp test.txt "myrustfs/bucket/test with+mixed/file.txt"
mc ls "myrustfs/bucket/test with+mixed/"
```
## Technical Details
### How RustFS Handles Special Characters
1. **Request Reception**: Client sends HTTP request with URL-encoded path
```
PUT /bucket/test%20file.txt
```
2. **URL Decoding**: s3s library decodes the path
```rust
let decoded = urlencoding::decode("/bucket/test%20file.txt")
// Result: "/bucket/test file.txt"
```
3. **Storage**: Object stored with decoded name
```
Stored as: "test file.txt"
```
4. **Retrieval**: Object retrieved by decoded name
```rust
let key = "test file.txt"; // Already decoded by s3s
store.get_object(bucket, key)
```
5. **Response**: Key returned in response (decoded)
```xml
<Key>test file.txt</Key>
```
6. **Client Display**: S3 clients display the decoded name
```
Shows: test file.txt
```
### URL Encoding Standards
RustFS follows:
- **RFC 3986**: URI Generic Syntax
- **AWS S3 API**: Object key encoding rules
- **HTTP/1.1**: URL encoding in request URIs
Key points:
- Keys are UTF-8 strings
- URL encoding is only for HTTP transport
- Keys are stored and compared in decoded form
## FAQs
**Q: Can I use spaces in object names?**
A: Yes, but use an S3 SDK which handles encoding automatically.
**Q: Why does `+` not work as a space?**
A: In URL paths, `+` represents a literal plus sign. Only in query parameters does `+` mean space. Use `%20` for spaces in paths.
**Q: Does RustFS support Unicode in object names?**
A: Yes, object names are UTF-8 strings. They support any valid UTF-8 character.
**Q: What characters are forbidden?**
A: Control characters (null byte, newline, carriage return) are rejected. All printable characters are allowed.
**Q: How do I fix "UI can't list folder" issue?**
A: Use the CLI (mc or aws-cli) instead. This is a UI bug, not a backend issue.
**Q: Why do some clients work but others don't?**
A: Proper S3 SDKs handle encoding correctly. Custom clients may have bugs. Always use official SDKs.
## Getting Help
If you encounter issues:
1. **Check this guide first**
2. **Verify you're using an S3 SDK** (not raw HTTP)
3. **Test with mc client** to isolate if issue is backend or client
4. **Enable debug logging** on RustFS: `RUST_LOG=rustfs=debug`
5. **Report issues** at: https://github.com/rustfs/rustfs/issues
Include in bug reports:
- Client/SDK used (and version)
- Exact object name causing issue
- Whether mc client works
- Debug logs from RustFS
---
**Last Updated**: 2025-12-09
**RustFS Version**: 0.0.5+
**Related Documents**:
- [Special Characters Analysis](./special-characters-in-path-analysis.md)
- [Special Characters Solution](./special-characters-solution.md)
docs/cluster_recovery.md
-156
View File
@@ -1,156 +0,0 @@
# Resolution Report: Issue #1001 - Cluster Recovery from Abrupt Power-Off
## 1. Issue Description
**Problem**: The cluster failed to recover gracefully when a node experienced an abrupt power-off (hard failure).
**Symptoms**:
- The application became unable to upload files.
- The Console Web UI became unresponsive across the cluster.
- The `rustfsadmin` user was unable to log in after a server power-off.
- The performance page displayed 0 storage, 0 objects, and 0 servers online/offline.
- The system "hung" indefinitely, unlike the immediate recovery observed during a graceful process termination (`kill`).
**Root Cause (Multi-Layered)**:
1. **TCP Connection Issue**: The standard TCP protocol does not immediately detect a silent peer disappearance (power loss) because no `FIN` or `RST` packets are sent.
2. **Stale Connection Cache**: Cached gRPC connections in `GLOBAL_Conn_Map` were reused even when the peer was dead, causing blocking on every RPC call.
3. **Blocking IAM Notifications**: Login operations blocked waiting for ALL peers to acknowledge user/policy changes.
4. **No Per-Peer Timeouts**: Console aggregation calls like `server_info()` and `storage_info()` could hang waiting for dead peers.
---
## 2. Technical Approach
To resolve this, we implemented a comprehensive multi-layered resilience strategy.
### Key Objectives:
1. **Fail Fast**: Detect dead peers in seconds, not minutes.
2. **Evict Stale Connections**: Automatically remove dead connections from cache to force reconnection.
3. **Non-Blocking Operations**: Auth and IAM operations should not wait for dead peers.
4. **Graceful Degradation**: Console should show partial data from healthy nodes, not hang.
---
## 3. Implemented Solution
### Solution Overview
The fix implements a multi-layered detection strategy covering both Control Plane (RPC) and Data Plane (Streaming):
1. **Control Plane (gRPC)**:
* Enabled `http2_keep_alive_interval` (5s) and `keep_alive_timeout` (3s) in `tonic` clients.
* Enforced `tcp_keepalive` (10s) on underlying transport.
* Context: Ensures cluster metadata operations (raft, status checks) fail fast if a node dies.
2. **Data Plane (File Uploads/Downloads)**:
* **Client (Rio)**: Updated `reqwest` client builder in `crates/rio` to enable TCP Keepalive (10s) and HTTP/2 Keepalive (5s). This prevents hangs during large file streaming (e.g., 1GB uploads).
* **Server**: Enabled `SO_KEEPALIVE` on all incoming TCP connections in `rustfs/src/server/http.rs` to forcefully close sockets from dead clients.
3. **Cross-Platform Build Stability**:
* Guarded Linux-specific profiling code (`jemalloc_pprof`) with `#[cfg(target_os = "linux")]` to fix build failures on macOS/AArch64.
### Configuration Changes
```rust
pub async fn storage_info<S: StorageAPI>(&self, api: &S) -> rustfs_madmin::StorageInfo {
let peer_timeout = Duration::from_secs(2);
for client in self.peer_clients.iter() {
futures.push(async move {
if let Some(client) = client {
match timeout(peer_timeout, client.local_storage_info()).await {
Ok(Ok(info)) => Some(info),
Ok(Err(_)) | Err(_) => {
// Return offline status for dead peer
Some(rustfs_madmin::StorageInfo {
disks: get_offline_disks(&host, &endpoints),
..Default::default()
})
}
}
}
});
}
// Rest continues even if some peers are down
}
```
### Fix 4: Enhanced gRPC Client Configuration
**File Modified**: `crates/protos/src/lib.rs`
**Configuration**:
```rust
const CONNECT_TIMEOUT_SECS: u64 = 3; // Reduced from 5s
const TCP_KEEPALIVE_SECS: u64 = 10; // OS-level keepalive
const HTTP2_KEEPALIVE_INTERVAL_SECS: u64 = 5; // HTTP/2 PING interval
const HTTP2_KEEPALIVE_TIMEOUT_SECS: u64 = 3; // PING ACK timeout
const RPC_TIMEOUT_SECS: u64 = 30; // Reduced from 60s
let connector = Endpoint::from_shared(addr.to_string())?
.connect_timeout(Duration::from_secs(CONNECT_TIMEOUT_SECS))
.tcp_keepalive(Some(Duration::from_secs(TCP_KEEPALIVE_SECS)))
.http2_keep_alive_interval(Duration::from_secs(HTTP2_KEEPALIVE_INTERVAL_SECS))
.keep_alive_timeout(Duration::from_secs(HTTP2_KEEPALIVE_TIMEOUT_SECS))
.keep_alive_while_idle(true)
.timeout(Duration::from_secs(RPC_TIMEOUT_SECS));
```
---
## 4. Files Changed Summary
| File | Change |
|------|--------|
| `crates/common/src/globals.rs` | Added `evict_connection()`, `has_cached_connection()`, `clear_all_connections()` |
| `crates/common/Cargo.toml` | Added `tracing` dependency |
| `crates/protos/src/lib.rs` | Refactored to use constants, added `evict_failed_connection()`, improved documentation |
| `crates/protos/Cargo.toml` | Added `tracing` dependency |
| `crates/ecstore/src/rpc/peer_rest_client.rs` | Added auto-eviction on RPC failure for `server_info()` and `local_storage_info()` |
| `crates/ecstore/src/notification_sys.rs` | Added per-peer timeout to `storage_info()` |
| `crates/iam/src/sys.rs` | Made `notify_for_user()`, `notify_for_service_account()`, `notify_for_group()` non-blocking |
---
## 5. Test Results
All 299 tests pass:
```
test result: ok. 299 passed; 0 failed; 0 ignored
```
---
## 6. Expected Behavior After Fix
| Scenario | Before | After |
|----------|--------|-------|
| Node power-off | Cluster hangs indefinitely | Cluster recovers in ~8 seconds |
| Login during node failure | Login hangs | Login succeeds immediately |
| Console during node failure | Shows 0/0/0 | Shows partial data from healthy nodes |
| Upload during node failure | Upload stops | Upload fails fast, can be retried |
| Stale cached connection | Blocks forever | Auto-evicted, fresh connection attempted |
---
## 7. Verification Steps
1. **Start a 3+ node RustFS cluster**
2. **Test Console Recovery**:
- Access console dashboard
- Forcefully kill one node (e.g., `kill -9`)
- Verify dashboard updates within 10 seconds showing offline status
3. **Test Login Recovery**:
- Kill a node while logged out
- Attempt login with `rustfsadmin`
- Verify login succeeds within 5 seconds
4. **Test Upload Recovery**:
- Start a large file upload
- Kill the target node mid-upload
- Verify upload fails fast (not hangs) and can be retried
---
## 8. Related Issues
- Issue #1001: Cluster Recovery from Abrupt Power-Off
- PR #1035: fix(net): resolve 1GB upload hang and macos build
## 9. Contributors
- Initial keepalive fix: Original PR #1035
- Deep-rooted reliability fix: This update
docs/compression-best-practices.md
-444
View File
@@ -1,444 +0,0 @@
# HTTP Response Compression Best Practices in RustFS
## Overview
This document outlines best practices for HTTP response compression in RustFS, based on lessons learned from fixing the
NoSuchKey error response regression (Issue #901) and the whitelist-based compression redesign (Issue #902).
## Whitelist-Based Compression (Issue #902)
### Design Philosophy
After Issue #901, we identified that the blacklist approach (compress everything except known problematic types) was
still causing issues with browser downloads showing "unknown file size". In Issue #902, we redesigned the compression
system using a **whitelist approach** aligned with MinIO's behavior:
1. **Compression is disabled by default** - Opt-in rather than opt-out
2. **Only explicitly configured content types are compressed** - Preserves Content-Length for all other responses
3. **Fine-grained configuration** - Control via file extensions, MIME types, and size thresholds
4. **Skip already-encoded content** - Avoid double compression
### Configuration Options
RustFS provides flexible compression configuration via environment variables and command-line arguments:
| Environment Variable | CLI Argument | Default | Description |
|---------------------|--------------|---------|-------------|
| `RUSTFS_COMPRESS_ENABLE` | | `false` | Enable/disable compression |
| `RUSTFS_COMPRESS_EXTENSIONS` | | `""` | File extensions to compress (e.g., `.txt,.log,.csv`) |
| `RUSTFS_COMPRESS_MIME_TYPES` | | `text/*,application/json,...` | MIME types to compress (supports wildcards) |
| `RUSTFS_COMPRESS_MIN_SIZE` | | `1000` | Minimum file size (bytes) for compression |
### Usage Examples
```bash
# Enable compression for text files and JSON
RUSTFS_COMPRESS_ENABLE=on \
RUSTFS_COMPRESS_EXTENSIONS=.txt,.log,.csv,.json,.xml \
RUSTFS_COMPRESS_MIME_TYPES=text/*,application/json,application/xml \
RUSTFS_COMPRESS_MIN_SIZE=1000 \
rustfs /data
# Or using command-line arguments
rustfs /data \
--compress-enable \
--compress-extensions ".txt,.log,.csv" \
--compress-mime-types "text/*,application/json" \
--compress-min-size 1000
```
### Implementation Details
The `CompressionPredicate` implements intelligent compression decisions:
```rust
impl Predicate for CompressionPredicate {
fn should_compress<B>(&self, response: &Response<B>) -> bool {
// 1. Check if compression is enabled
if !self.config.enabled { return false; }
// 2. Never compress error responses
if status.is_client_error() || status.is_server_error() { return false; }
// 3. Skip already-encoded content (gzip, br, deflate, etc.)
if has_content_encoding(response) { return false; }
// 4. Check minimum size threshold
if content_length < self.config.min_size { return false; }
// 5. Check whitelist: extension OR MIME type must match
if matches_extension(response) || matches_mime_type(response) {
return true;
}
// 6. Default: don't compress (whitelist approach)
false
}
}
```
### Benefits of Whitelist Approach
| Aspect | Blacklist (Old) | Whitelist (New) |
|--------|-----------------|-----------------|
| Default behavior | Compress most content | No compression |
| Content-Length | Often removed | Preserved for unmatched types |
| Browser downloads | "Unknown file size" | Accurate file size shown |
| Configuration | Complex exclusion rules | Simple inclusion rules |
| MinIO compatibility | Different behavior | Aligned behavior |
## Key Principles
### 1. Never Compress Error Responses
**Rationale**: Error responses are typically small (100-500 bytes) and need to be transmitted accurately. Compression
can:
- Introduce Content-Length header mismatches
- Add unnecessary overhead for small payloads
- Potentially corrupt error details during buffering
**Implementation**:
```rust
// Always check status code first
if status.is_client_error() || status.is_server_error() {
return false; // Don't compress
}
```
**Affected Status Codes**:
- 4xx Client Errors (400, 403, 404, etc.)
- 5xx Server Errors (500, 502, 503, etc.)
### 2. Size-Based Compression Threshold
**Rationale**: Compression has overhead in terms of CPU and potentially network roundtrips. For very small responses:
- Compression overhead > space savings
- May actually increase payload size
- Adds latency without benefit
**Recommended Threshold**: 1000 bytes minimum (configurable via `RUSTFS_COMPRESS_MIN_SIZE`)
**Implementation**:
```rust
if let Some(content_length) = response.headers().get(CONTENT_LENGTH) {
if let Ok(length) = content_length.to_str()?.parse::<u64>()? {
if length < self.config.min_size {
return false; // Don't compress small responses
}
}
}
```
### 3. Skip Already-Encoded Content
**Rationale**: If the response already has a `Content-Encoding` header (e.g., gzip, br, deflate, zstd), the content
is already compressed. Re-compressing provides no benefit and may cause issues:
- Double compression wastes CPU cycles
- May corrupt data or increase size
- Breaks decompression on client side
**Implementation**:
```rust
// Skip if content is already encoded (e.g., gzip, br, deflate, zstd)
if let Some(content_encoding) = response.headers().get(CONTENT_ENCODING) {
if let Ok(encoding) = content_encoding.to_str() {
let encoding_lower = encoding.to_lowercase();
// "identity" means no encoding, so we can still compress
if encoding_lower != "identity" && !encoding_lower.is_empty() {
debug!("Skipping compression for already encoded response: {}", encoding);
return false;
}
}
}
```
**Common Content-Encoding Values**:
- `gzip` - GNU zip compression
- `br` - Brotli compression
- `deflate` - Deflate compression
- `zstd` - Zstandard compression
- `identity` - No encoding (compression allowed)
### 4. Maintain Observability
**Rationale**: Compression decisions can affect debugging and troubleshooting. Always log when compression is skipped.
**Implementation**:
```rust
debug!(
"Skipping compression for error response: status={}",
status.as_u16()
);
```
**Log Analysis**:
```bash
# Monitor compression decisions
RUST_LOG=rustfs::server::http=debug ./target/release/rustfs
# Look for patterns
grep "Skipping compression" logs/rustfs.log | wc -l
```
## Common Pitfalls
### ❌ Compressing All Responses Blindly
```rust
// BAD - No filtering
.layer(CompressionLayer::new())
```
**Problem**: Can cause Content-Length mismatches with error responses and browser download issues
### ❌ Using Blacklist Approach
```rust
// BAD - Blacklist approach (compress everything except...)
fn should_compress(&self, response: &Response<B>) -> bool {
// Skip images, videos, archives...
if is_already_compressed_type(content_type) { return false; }
true // Compress everything else
}
```
**Problem**: Removes Content-Length for many file types, causing "unknown file size" in browsers
### ✅ Using Whitelist-Based Predicate
```rust
// GOOD - Whitelist approach with configurable predicate
.layer(CompressionLayer::new().compress_when(CompressionPredicate::new(config)))
```
### ❌ Ignoring Content-Encoding Header
```rust
// BAD - May double-compress already compressed content
fn should_compress(&self, response: &Response<B>) -> bool {
matches_mime_type(response) // Missing Content-Encoding check
}
```
**Problem**: Double compression wastes CPU and may corrupt data
### ✅ Comprehensive Checks
```rust
// GOOD - Multi-criteria whitelist decision
fn should_compress(&self, response: &Response<B>) -> bool {
// 1. Must be enabled
if !self.config.enabled { return false; }
// 2. Skip error responses
if response.status().is_error() { return false; }
// 3. Skip already-encoded content
if has_content_encoding(response) { return false; }
// 4. Check minimum size
if get_content_length(response) < self.config.min_size { return false; }
// 5. Must match whitelist (extension OR MIME type)
matches_extension(response) || matches_mime_type(response)
}
```
## Performance Considerations
### CPU Usage
- **Compression CPU Cost**: ~1-5ms for typical responses
- **Benefit**: 70-90% size reduction for text/json
- **Break-even**: Responses > 512 bytes on fast networks
### Network Latency
- **Savings**: Proportional to size reduction
- **Break-even**: ~256 bytes on typical connections
- **Diminishing Returns**: Below 128 bytes
### Memory Usage
- **Buffer Size**: Usually 4-16KB per connection
- **Trade-off**: Memory vs. bandwidth
- **Recommendation**: Profile in production
## Testing Guidelines
### Unit Tests
Test compression predicate logic:
```rust
#[test]
fn test_should_not_compress_errors() {
let predicate = ShouldCompress;
let response = Response::builder()
.status(404)
.body(())
.unwrap();
assert!(!predicate.should_compress(&response));
}
#[test]
fn test_should_not_compress_small_responses() {
let predicate = ShouldCompress;
let response = Response::builder()
.status(200)
.header(CONTENT_LENGTH, "100")
.body(())
.unwrap();
assert!(!predicate.should_compress(&response));
}
```
### Integration Tests
Test actual S3 API responses:
```rust
#[tokio::test]
async fn test_error_response_not_truncated() {
let response = client
.get_object()
.bucket("test")
.key("nonexistent")
.send()
.await;
// Should get proper error, not truncation error
match response.unwrap_err() {
SdkError::ServiceError(err) => {
assert!(err.is_no_such_key());
}
other => panic!("Expected ServiceError, got {:?}", other),
}
}
```
## Monitoring and Alerts
### Metrics to Track
1. **Compression Ratio**: `compressed_size / original_size`
2. **Compression Skip Rate**: `skipped_count / total_count`
3. **Error Response Size Distribution**
4. **CPU Usage During Compression**
### Alert Conditions
```yaml
# Prometheus alert rules
- alert: HighCompressionSkipRate
expr: |
rate(http_compression_skipped_total[5m])
/ rate(http_responses_total[5m]) > 0.5
annotations:
summary: "More than 50% of responses skipping compression"
- alert: LargeErrorResponses
expr: |
histogram_quantile(0.95,
rate(http_error_response_size_bytes_bucket[5m])) > 1024
annotations:
summary: "Error responses larger than 1KB"
```
## Migration Guide
### Migrating from Blacklist to Whitelist Approach
If you're upgrading from an older RustFS version with blacklist-based compression:
1. **Compression is now disabled by default**
- Set `RUSTFS_COMPRESS_ENABLE=on` to enable
- This ensures backward compatibility for existing deployments
2. **Configure your whitelist**
```bash
# Example: Enable compression for common text formats
RUSTFS_COMPRESS_ENABLE=on
RUSTFS_COMPRESS_EXTENSIONS=.txt,.log,.csv,.json,.xml,.html,.css,.js
RUSTFS_COMPRESS_MIME_TYPES=text/*,application/json,application/xml,application/javascript
RUSTFS_COMPRESS_MIN_SIZE=1000
```
3. **Verify browser downloads**
- Check that file downloads show accurate file sizes
- Verify Content-Length headers are preserved for non-compressed content
### Updating Existing Code
If you're adding compression to an existing service:
1. **Start with compression disabled** (default)
2. **Define your whitelist**: Identify content types that benefit from compression
3. **Set appropriate thresholds**: Start with 1KB minimum size
4. **Enable and monitor**: Watch CPU, latency, and download behavior
### Rollout Strategy
1. **Stage 1**: Deploy to canary (5% traffic)
- Monitor for 24 hours
- Check error rates and latency
- Verify browser download behavior
2. **Stage 2**: Expand to 25% traffic
- Monitor for 48 hours
- Validate compression ratios
- Check Content-Length preservation
3. **Stage 3**: Full rollout (100% traffic)
- Continue monitoring for 1 week
- Document any issues
- Fine-tune whitelist based on actual usage
## Related Documentation
- [Fix NoSuchKey Regression](./fix-nosuchkey-regression.md)
- [tower-http Compression](https://docs.rs/tower-http/latest/tower_http/compression/)
- [HTTP Content-Encoding](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding)
## Architecture
### Module Structure
The compression functionality is organized in a dedicated module for maintainability:
```
rustfs/src/server/
├── compress.rs # Compression configuration and predicate
├── http.rs # HTTP server (uses compress module)
└── mod.rs # Module declarations
```
### Key Components
1. **`CompressionConfig`** - Stores compression settings parsed from environment/CLI
2. **`CompressionPredicate`** - Implements `tower_http::compression::predicate::Predicate`
3. **Configuration Constants** - Defined in `crates/config/src/constants/compress.rs`
## References
1. Issue #901: NoSuchKey error response regression
2. Issue #902: Whitelist-based compression redesign
3. [Google Web Fundamentals - Text Compression](https://web.dev/reduce-network-payloads-using-text-compression/)
4. [AWS Best Practices - Response Compression](https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/)
---
**Last Updated**: 2025-12-13
**Maintainer**: RustFS Team
docs/console-separation.md
-1334
View File
File diff suppressed because it is too large Load Diff
docs/examples/README.md
-85
View File
@@ -1,85 +0,0 @@
# RustFS Deployment Examples
This directory contains practical deployment examples and configurations for RustFS.
## Available Examples
### [MNMD (Multi-Node Multi-Drive)](./mnmd/)
Complete Docker Compose example for deploying RustFS in a 4-node, 4-drive-per-node configuration.
**Features:**
- Proper disk indexing (1..4) to avoid VolumeNotFound errors
- Startup coordination via `wait-and-start.sh` script
- Service discovery using Docker service names
- Health checks with alternatives for different base images
- Comprehensive documentation and verification checklist
**Use Case:** Production-ready multi-node deployment for high availability and performance.
**Quick Start:**
```bash
cd docs/examples/mnmd
docker-compose up -d
```
**See also:**
- [MNMD README](./mnmd/README.md) - Detailed usage guide
- [MNMD CHECKLIST](./mnmd/CHECKLIST.md) - Step-by-step verification
## Other Deployment Examples
For additional deployment examples, see:
- [`docker/`](./docker/) - Root-level examples directory with:
- `docker-quickstart.sh` - Quick start script for basic deployments, Quickstart script (basic
/dev/prod/status/test/cleanup)
- `enhanced-docker-deployment.sh` - Advanced deployment scenarios, Advanced deployment script with multiple
scenarios and detailed logs (basic /dev/prod/all/status/test/logs/cleanup)
- `enhanced-security-deployment.sh` - Production-ready scripts with TLS, throttling, and secure credential
generation
- `docker-comprehensive.yml` - Docker Compose with multiple profiles, Docker Compose files containing multiple
profiles (basic / dev / production / enterprise / api-only / nginx, etc.)
- Usage example:
```bash
# Rapid development environment
./docs/examples/docker/docker-quickstart.sh dev
# Start dev profile using Docker Compose
docker-compose -f docs/examples/docker/docker-comprehensive.yml --profile dev up -d
# Secure deployment
./docs/examples/docker/enhanced-security-deployment.sh
```
- Note: If the original CI or other documents refer to the old path `examples/`, please update it to
`docs/examples/docker/`. Relative links within the document are already in this README.
- [`.docker/compose/`](/.docker/compose/) - Docker Compose configurations:
- `docker-compose.cluster.yaml` - Basic cluster setup
- `docker-compose.observability.yaml` - Observability stack integration
## Related Documentation
- [Console & Endpoint Service Separation](../console-separation.md)
- [Environment Variables](../ENVIRONMENT_VARIABLES.md)
- [Performance Testing](../PERFORMANCE_TESTING.md)
## Contributing
When adding new examples:
1. Create a dedicated subdirectory under `docs/examples/`
2. Include a comprehensive README.md
3. Provide working configuration files
4. Add verification steps or checklists
5. Document common issues and troubleshooting
## Support
For issues or questions:
- GitHub Issues: https://github.com/rustfs/rustfs/issues
- Documentation: https://rustfs.com/docs
docs/examples/docker/README.md
-282
View File
@@ -1,282 +0,0 @@
# RustFS Docker Deployment Examples
This directory contains various deployment scripts and configuration files for RustFS with console and endpoint service
separation.
## Quick Start Scripts
### `docker-quickstart.sh`
The fastest way to get RustFS running with different configurations.
```bash
# Basic deployment (ports 9000-9001)
./docker-quickstart.sh basic
# Development environment (ports 9010-9011)
./docker-quickstart.sh dev
# Production-like deployment (ports 9020-9021)
./docker-quickstart.sh prod
# Check status of all deployments
./docker-quickstart.sh status
# Test health of all running services
./docker-quickstart.sh test
# Clean up all containers
./docker-quickstart.sh cleanup
```
### `enhanced-docker-deployment.sh`
Comprehensive deployment script with multiple scenarios and detailed logging.
```bash
# Deploy individual scenarios
./enhanced-docker-deployment.sh basic # Basic setup with port mapping
./enhanced-docker-deployment.sh dev # Development environment
./enhanced-docker-deployment.sh prod # Production-like with security
# Deploy all scenarios at once
./enhanced-docker-deployment.sh all
# Check status and test services
./enhanced-docker-deployment.sh status
./enhanced-docker-deployment.sh test
# View logs for specific container
./enhanced-docker-deployment.sh logs rustfs-dev
# Complete cleanup
./enhanced-docker-deployment.sh cleanup
```
### `enhanced-security-deployment.sh`
Production-ready deployment with enhanced security features including TLS, rate limiting, and secure credential
generation.
```bash
# Deploy with security hardening
./enhanced-security-deployment.sh
# Features:
# - Automatic TLS certificate generation
# - Secure credential generation
# - Rate limiting configuration
# - Console access restrictions
# - Health check validation
```
## Docker Compose Examples
### `docker-comprehensive.yml`
Complete Docker Compose configuration with multiple deployment profiles.
```bash
# Deploy specific profiles
docker-compose -f docker-comprehensive.yml --profile basic up -d
docker-compose -f docker-comprehensive.yml --profile dev up -d
docker-compose -f docker-comprehensive.yml --profile production up -d
docker-compose -f docker-comprehensive.yml --profile enterprise up -d
docker-compose -f docker-comprehensive.yml --profile api-only up -d
# Deploy with reverse proxy
docker-compose -f docker-comprehensive.yml --profile production --profile nginx up -d
```
#### Available Profiles:
- **basic**: Simple deployment for testing (ports 9000-9001)
- **dev**: Development environment with debug logging (ports 9010-9011)
- **production**: Production deployment with security (ports 9020-9021)
- **enterprise**: Full enterprise setup with TLS (ports 9030-9443)
- **api-only**: API endpoint without console (port 9040)
## Usage Examples by Scenario
### Development Setup
```bash
# Quick development start
./docker-quickstart.sh dev
# Or use enhanced deployment for more features
./enhanced-docker-deployment.sh dev
# Or use Docker Compose
docker-compose -f docker-comprehensive.yml --profile dev up -d
```
**Access Points:**
- API: http://localhost:9010 (or 9030 for enhanced)
- Console: http://localhost:9011/rustfs/console/ (or 9031 for enhanced)
- Credentials: dev-admin / dev-secret
### Production Deployment
```bash
# Security-hardened deployment
./enhanced-security-deployment.sh
# Or production profile
./enhanced-docker-deployment.sh prod
```
**Features:**
- TLS encryption for console
- Rate limiting enabled
- Restricted CORS policies
- Secure credential generation
- Console bound to localhost only
### Testing and CI/CD
```bash
# API-only deployment for testing
docker-compose -f docker-comprehensive.yml --profile api-only up -d
# Quick basic setup for integration tests
./docker-quickstart.sh basic
```
## Configuration Examples
### Environment Variables
All deployment scripts support customization via environment variables:
```bash
# Custom image and ports
export RUSTFS_IMAGE="rustfs/rustfs:custom-tag"
export CONSOLE_PORT="8001"
export API_PORT="8000"
# Custom data directories
export DATA_DIR="/custom/data/path"
export CERTS_DIR="/custom/certs/path"
# Run with custom configuration
./enhanced-security-deployment.sh
```
### Common Configurations
```bash
# Development - permissive CORS
RUSTFS_CORS_ALLOWED_ORIGINS="*"
RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*"
# Production - restrictive CORS
RUSTFS_CORS_ALLOWED_ORIGINS="https://myapp.com,https://api.myapp.com"
RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="https://admin.myapp.com"
# Security hardening
RUSTFS_CONSOLE_RATE_LIMIT_ENABLE="true"
RUSTFS_CONSOLE_RATE_LIMIT_RPM="60"
RUSTFS_CONSOLE_AUTH_TIMEOUT="1800"
```
## Monitoring and Health Checks
All deployments include health check endpoints:
```bash
# Test API health
curl http://localhost:9000/health
# Test console health
curl http://localhost:9001/health
# Test all deployments
./docker-quickstart.sh test
./enhanced-docker-deployment.sh test
```
## Network Architecture
### Port Mappings
| Deployment | API Port | Console Port | Description |
|------------|----------|--------------|-------------------------|
| Basic | 9000 | 9001 | Simple deployment |
| Dev | 9010 | 9011 | Development environment |
| Prod | 9020 | 9021 | Production-like setup |
| Enterprise | 9030 | 9443 | Enterprise with TLS |
| API-Only | 9040 | - | API endpoint only |
### Network Isolation
Production deployments use network isolation:
- **Public API Network**: Exposes API endpoints to external clients
- **Internal Console Network**: Restricts console access to internal networks
- **Secure Network**: Isolated network for enterprise deployments
## Security Considerations
### Development
- Permissive CORS policies for easy testing
- Debug logging enabled
- Default credentials for simplicity
### Production
- Restrictive CORS policies
- TLS encryption for console
- Rate limiting enabled
- Secure credential generation
- Console bound to localhost
- Network isolation
### Enterprise
- Complete TLS encryption
- Advanced rate limiting
- Authentication timeouts
- Secret management
- Network segregation
## Troubleshooting
### Common Issues
1. **Port Conflicts**: Use different ports via environment variables
2. **CORS Errors**: Check origin configuration and browser network tab
3. **Health Check Failures**: Verify services are running and ports are accessible
4. **Permission Issues**: Check volume mount permissions and certificate file permissions
### Debug Commands
```bash
# Check container logs
docker logs rustfs-container
# Check container environment
docker exec rustfs-container env | grep RUSTFS
# Test connectivity
docker exec rustfs-container curl http://localhost:9000/health
docker exec rustfs-container curl http://localhost:9001/health
# Check listening ports
docker exec rustfs-container netstat -tulpn | grep -E ':(9000|9001)'
```
## Migration from Previous Versions
See [docs/console-separation.md](../../console-separation.md) for detailed migration instructions from single-port
deployments to the separated architecture.
## Additional Resources
- [Console Separation Documentation](../../console-separation.md)
- [Docker Compose Configuration](../../../docker-compose.yml)
- [Main Dockerfile](../../../Dockerfile)
- [Security Best Practices](../../console-separation.md#security-hardening)
docs/examples/docker/docker-comprehensive.yml
-219
View File
@@ -1,219 +0,0 @@
# RustFS Comprehensive Docker Deployment Examples
# This file demonstrates various deployment scenarios for RustFS with console separation
version: "3.8"
services:
# Basic deployment with default settings
rustfs-basic:
image: rustfs/rustfs:latest
container_name: rustfs-basic
ports:
- "9000:9000" # API endpoint
- "9001:9001" # Console interface
environment:
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
- RUSTFS_CORS_ALLOWED_ORIGINS=http://127.0.0.1:9001
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
- RUSTFS_ACCESS_KEY=admin
- RUSTFS_SECRET_KEY=password
volumes:
- rustfs-basic-data:/data
networks:
- rustfs-network
restart: unless-stopped
healthcheck:
test: [ "CMD", "sh", "-c", "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health" ]
interval: 30s
timeout: 10s
retries: 3
profiles:
- basic
# Development environment with debug logging
rustfs-dev:
image: rustfs/rustfs:latest
container_name: rustfs-dev
ports:
- "9010:9000" # API endpoint
- "9011:9001" # Console interface
environment:
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
- RUSTFS_CORS_ALLOWED_ORIGINS=*
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
- RUSTFS_ACCESS_KEY=dev-admin
- RUSTFS_SECRET_KEY=dev-password
- RUSTFS_OBS_LOGGER_LEVEL=debug
volumes:
- rustfs-dev-data:/data
- rustfs-dev-logs:/logs
networks:
- rustfs-network
restart: unless-stopped
healthcheck:
test: [ "CMD", "sh", "-c", "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health" ]
interval: 30s
timeout: 10s
retries: 3
profiles:
- dev
# Production environment with security hardening
rustfs-production:
image: rustfs/rustfs:latest
container_name: rustfs-production
ports:
- "9020:9000" # API endpoint (public)
- "127.0.0.1:9021:9001" # Console (localhost only)
environment:
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
- RUSTFS_CORS_ALLOWED_ORIGINS=https://myapp.com,https://api.myapp.com
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=https://admin.myapp.com
- RUSTFS_CONSOLE_RATE_LIMIT_ENABLE=true
- RUSTFS_CONSOLE_RATE_LIMIT_RPM=60
- RUSTFS_CONSOLE_AUTH_TIMEOUT=1800
- RUSTFS_ACCESS_KEY_FILE=/run/secrets/rustfs_access_key
- RUSTFS_SECRET_KEY_FILE=/run/secrets/rustfs_secret_key
volumes:
- rustfs-production-data:/data
- rustfs-production-logs:/logs
- rustfs-certs:/certs:ro
networks:
- rustfs-network
secrets:
- rustfs_access_key
- rustfs_secret_key
restart: unless-stopped
healthcheck:
test: [ "CMD", "sh", "-c", "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health" ]
interval: 30s
timeout: 10s
retries: 3
profiles:
- production
# Enterprise deployment with TLS and full security
rustfs-enterprise:
image: rustfs/rustfs:latest
container_name: rustfs-enterprise
ports:
- "9030:9000" # API endpoint
- "127.0.0.1:9443:9001" # Console with TLS (localhost only)
environment:
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
- RUSTFS_TLS_PATH=/certs
- RUSTFS_CORS_ALLOWED_ORIGINS=https://enterprise.com
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=https://admin.enterprise.com
- RUSTFS_CONSOLE_RATE_LIMIT_ENABLE=true
- RUSTFS_CONSOLE_RATE_LIMIT_RPM=30
- RUSTFS_CONSOLE_AUTH_TIMEOUT=900
volumes:
- rustfs-enterprise-data:/data
- rustfs-enterprise-logs:/logs
- rustfs-enterprise-certs:/certs:ro
networks:
- rustfs-secure-network
secrets:
- rustfs_enterprise_access_key
- rustfs_enterprise_secret_key
restart: unless-stopped
healthcheck:
test: [ "CMD", "sh", "-c", "curl -f http://127.0.0.1:9000/health && curl -k -f https://127.0.0.1:9001/rustfs/console/health" ]
interval: 30s
timeout: 10s
retries: 3
profiles:
- enterprise
# API-only deployment (console disabled)
rustfs-api-only:
image: rustfs/rustfs:latest
container_name: rustfs-api-only
ports:
- "9040:9000" # API endpoint only
environment:
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ENABLE=false
- RUSTFS_CORS_ALLOWED_ORIGINS=https://client-app.com
- RUSTFS_ACCESS_KEY=api-only-key
- RUSTFS_SECRET_KEY=api-only-secret
volumes:
- rustfs-api-data:/data
networks:
- rustfs-network
restart: unless-stopped
healthcheck:
test: [ "CMD", "curl", "-f", "http://127.0.0.1:9000/health" ]
interval: 30s
timeout: 10s
retries: 3
profiles:
- api-only
# Nginx reverse proxy for production
nginx-proxy:
image: nginx:alpine
container_name: rustfs-nginx
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./nginx/ssl:/etc/nginx/ssl:ro
networks:
- rustfs-network
restart: unless-stopped
depends_on:
- rustfs-production
profiles:
- production
- enterprise
networks:
rustfs-network:
driver: bridge
ipam:
config:
- subnet: 172.20.0.0/16
rustfs-secure-network:
driver: bridge
internal: true
ipam:
config:
- subnet: 172.21.0.0/16
volumes:
rustfs-basic-data:
driver: local
rustfs-dev-data:
driver: local
rustfs-dev-logs:
driver: local
rustfs-production-data:
driver: local
rustfs-production-logs:
driver: local
rustfs-enterprise-data:
driver: local
rustfs-enterprise-logs:
driver: local
rustfs-enterprise-certs:
driver: local
rustfs-api-data:
driver: local
rustfs-certs:
driver: local
secrets:
rustfs_access_key:
external: true
rustfs_secret_key:
external: true
rustfs_enterprise_access_key:
external: true
rustfs_enterprise_secret_key:
external: true
docs/examples/docker/docker-quickstart.sh
-292
View File
@@ -1,292 +0,0 @@
#!/usr/bin/env bash
# RustFS Docker Quick Start Script
# This script provides easy deployment commands for different scenarios
set -e
# Colors for output
GREEN='\033[0;32m'
BLUE='\033[0;34m'
YELLOW='\033[1;33m'
RED='\033[0;31m'
NC='\033[0m' # No Color
log() {
echo -e "${GREEN}[RustFS]${NC} $1"
}
info() {
echo -e "${BLUE}[INFO]${NC} $1"
}
warn() {
echo -e "${YELLOW}[WARN]${NC} $1"
}
error() {
echo -e "${RED}[ERROR]${NC} $1"
}
# Print banner
print_banner() {
echo -e "${BLUE}"
echo "=================================================="
echo " RustFS Docker Quick Start"
echo " Console & Endpoint Separation"
echo "=================================================="
echo -e "${NC}"
}
# Check Docker availability
check_docker() {
if ! command -v docker &> /dev/null; then
error "Docker is not installed or not available in PATH"
exit 1
fi
info "Docker is available: $(docker --version)"
}
# Quick start - basic deployment
quick_basic() {
log "Starting RustFS basic deployment..."
docker run -d \
--name rustfs-quick \
-p 9000:9000 \
-p 9001:9001 \
-e RUSTFS_CORS_ALLOWED_ORIGINS="http://localhost:9001" \
-v rustfs-quick-data:/data \
rustfs/rustfs:latest
echo
info "✅ RustFS deployed successfully!"
info "🌐 API Endpoint: http://localhost:9000"
info "🖥️ Console UI: http://localhost:9001/rustfs/console/"
info "🔐 Credentials: rustfsadmin / rustfsadmin"
info "🏥 Health Check: curl http://localhost:9000/health"
echo
info "To stop: docker stop rustfs-quick"
info "To remove: docker rm rustfs-quick && docker volume rm rustfs-quick-data"
}
# Development deployment with debug logging
quick_dev() {
log "Starting RustFS development environment..."
docker run -d \
--name rustfs-dev \
-p 9010:9000 \
-p 9011:9001 \
-e RUSTFS_CORS_ALLOWED_ORIGINS="*" \
-e RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
-e RUSTFS_ACCESS_KEY="dev-admin" \
-e RUSTFS_SECRET_KEY="dev-secret" \
-e RUSTFS_OBS_LOGGER_LEVEL="debug" \
-v rustfs-dev-data:/data \
rustfs/rustfs:latest
echo
info "✅ RustFS development environment ready!"
info "🌐 API Endpoint: http://localhost:9010"
info "🖥️ Console UI: http://localhost:9011/rustfs/console/"
info "🔐 Credentials: dev-admin / dev-secret"
info "📊 Debug logging enabled"
echo
info "To stop: docker stop rustfs-dev"
}
# Production-like deployment
quick_prod() {
log "Starting RustFS production-like deployment..."
# Generate secure credentials
ACCESS_KEY="prod-$(openssl rand -hex 8)"
SECRET_KEY="$(openssl rand -hex 24)"
docker run -d \
--name rustfs-prod \
-p 9020:9000 \
-p 127.0.0.1:9021:9001 \
-e RUSTFS_CORS_ALLOWED_ORIGINS="https://myapp.com" \
-e RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="https://admin.myapp.com" \
-e RUSTFS_CONSOLE_RATE_LIMIT_ENABLE="true" \
-e RUSTFS_CONSOLE_RATE_LIMIT_RPM="60" \
-e RUSTFS_ACCESS_KEY="$ACCESS_KEY" \
-e RUSTFS_SECRET_KEY="$SECRET_KEY" \
-v rustfs-prod-data:/data \
rustfs/rustfs:latest
# Save credentials
echo "RUSTFS_ACCESS_KEY=$ACCESS_KEY" > rustfs-prod-credentials.txt
echo "RUSTFS_SECRET_KEY=$SECRET_KEY" >> rustfs-prod-credentials.txt
chmod 600 rustfs-prod-credentials.txt
echo
info "✅ RustFS production deployment ready!"
info "🌐 API Endpoint: http://localhost:9020 (public)"
info "🖥️ Console UI: http://127.0.0.1:9021/rustfs/console/ (localhost only)"
info "🔐 Credentials saved to rustfs-prod-credentials.txt"
info "🔒 Console restricted to localhost for security"
echo
warn "⚠️ Change default CORS origins for production use"
}
# Stop and cleanup
cleanup() {
log "Cleaning up RustFS deployments..."
docker stop rustfs-quick rustfs-dev rustfs-prod 2>/dev/null || true
docker rm rustfs-quick rustfs-dev rustfs-prod 2>/dev/null || true
info "Containers stopped and removed"
echo
info "To also remove data volumes, run:"
info "docker volume rm rustfs-quick-data rustfs-dev-data rustfs-prod-data"
}
# Show status of all deployments
status() {
log "RustFS deployment status:"
echo
if docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" | grep -q rustfs; then
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" | head -n1
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" | grep rustfs
else
info "No RustFS containers are currently running"
fi
echo
info "Available endpoints:"
if docker ps --filter "name=rustfs-quick" --format "{{.Names}}" | grep -q rustfs-quick; then
echo " Basic: http://localhost:9000 (API) | http://localhost:9001/rustfs/console/ (Console)"
fi
if docker ps --filter "name=rustfs-dev" --format "{{.Names}}" | grep -q rustfs-dev; then
echo " Dev: http://localhost:9010 (API) | http://localhost:9011/rustfs/console/ (Console)"
fi
if docker ps --filter "name=rustfs-prod" --format "{{.Names}}" | grep -q rustfs-prod; then
echo " Prod: http://localhost:9020 (API) | http://127.0.0.1:9021/rustfs/console/ (Console)"
fi
}
# Test deployments
test_deployments() {
log "Testing RustFS deployments..."
echo
# Test basic deployment
if docker ps --filter "name=rustfs-quick" --format "{{.Names}}" | grep -q rustfs-quick; then
info "Testing basic deployment..."
if curl -s -f http://localhost:9000/health | grep -q "ok"; then
echo " ✅ API health check: PASS"
else
echo " ❌ API health check: FAIL"
fi
if curl -s -f http://localhost:9001/health | grep -q "console"; then
echo " ✅ Console health check: PASS"
else
echo " ❌ Console health check: FAIL"
fi
fi
# Test dev deployment
if docker ps --filter "name=rustfs-dev" --format "{{.Names}}" | grep -q rustfs-dev; then
info "Testing development deployment..."
if curl -s -f http://localhost:9010/health | grep -q "ok"; then
echo " ✅ Dev API health check: PASS"
else
echo " ❌ Dev API health check: FAIL"
fi
if curl -s -f http://localhost:9011/health | grep -q "console"; then
echo " ✅ Dev Console health check: PASS"
else
echo " ❌ Dev Console health check: FAIL"
fi
fi
# Test prod deployment
if docker ps --filter "name=rustfs-prod" --format "{{.Names}}" | grep -q rustfs-prod; then
info "Testing production deployment..."
if curl -s -f http://localhost:9020/health | grep -q "ok"; then
echo " ✅ Prod API health check: PASS"
else
echo " ❌ Prod API health check: FAIL"
fi
if curl -s -f http://127.0.0.1:9021/health | grep -q "console"; then
echo " ✅ Prod Console health check: PASS"
else
echo " ❌ Prod Console health check: FAIL"
fi
fi
}
# Show help
show_help() {
print_banner
echo "Usage: $0 [command]"
echo
echo "Commands:"
echo " basic Start basic RustFS deployment (ports 9000-9001)"
echo " dev Start development deployment with debug logging (ports 9010-9011)"
echo " prod Start production-like deployment with security (ports 9020-9021)"
echo " status Show status of running deployments"
echo " test Test health of all running deployments"
echo " cleanup Stop and remove all RustFS containers"
echo " help Show this help message"
echo
echo "Examples:"
echo " $0 basic # Quick start with default settings"
echo " $0 dev # Development environment with debug logs"
echo " $0 prod # Production-like setup with security"
echo " $0 status # Check what's running"
echo " $0 test # Test all deployments"
echo " $0 cleanup # Clean everything up"
echo
echo "For more advanced deployments, see:"
echo " - examples/enhanced-docker-deployment.sh"
echo " - examples/enhanced-security-deployment.sh"
echo " - examples/docker-comprehensive.yml"
echo " - docs/console-separation.md"
echo
}
# Main execution
case "${1:-help}" in
"basic")
print_banner
check_docker
quick_basic
;;
"dev")
print_banner
check_docker
quick_dev
;;
"prod")
print_banner
check_docker
quick_prod
;;
"status")
print_banner
status
;;
"test")
print_banner
test_deployments
;;
"cleanup")
print_banner
cleanup
;;
"help"|*)
show_help
;;
esac
docs/examples/docker/enhanced-docker-deployment.sh
-318
View File
@@ -1,318 +0,0 @@
#!/usr/bin/env bash
# RustFS Enhanced Docker Deployment Examples
# This script demonstrates various deployment scenarios for RustFS with console separation
set -e
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
log_info() {
echo -e "${GREEN}[INFO]${NC} $1"
}
log_warn() {
echo -e "${YELLOW}[WARN]${NC} $1"
}
log_error() {
echo -e "${RED}[ERROR]${NC} $1"
}
log_section() {
echo -e "\n${BLUE}========================================${NC}"
echo -e "${BLUE}$1${NC}"
echo -e "${BLUE}========================================${NC}\n"
}
# Function to clean up existing containers
cleanup() {
log_info "Cleaning up existing RustFS containers..."
docker stop rustfs-basic rustfs-dev rustfs-prod 2>/dev/null || true
docker rm rustfs-basic rustfs-dev rustfs-prod 2>/dev/null || true
}
# Function to wait for service to be ready
wait_for_service() {
local url=$1
local service_name=$2
local max_attempts=30
local attempt=0
log_info "Waiting for $service_name to be ready at $url..."
while [ $attempt -lt $max_attempts ]; do
if curl -s -f "$url" > /dev/null 2>&1; then
log_info "$service_name is ready!"
return 0
fi
attempt=$((attempt + 1))
sleep 1
done
log_error "$service_name failed to start within ${max_attempts}s"
return 1
}
# Scenario 1: Basic deployment with port mapping
deploy_basic() {
log_section "Scenario 1: Basic Docker Deployment with Port Mapping"
log_info "Starting RustFS with port mapping 9020:9000 and 9021:9001"
docker run -d \
--name rustfs-basic \
-p 9020:9000 \
-p 9021:9001 \
-e RUSTFS_CORS_ALLOWED_ORIGINS="http://localhost:9021,http://127.0.0.1:9021" \
-e RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
-e RUSTFS_ACCESS_KEY="basic-access" \
-e RUSTFS_SECRET_KEY="basic-secret" \
-v rustfs-basic-data:/data \
rustfs/rustfs:latest
# Wait for services to be ready
wait_for_service "http://localhost:9020/health" "API Service"
wait_for_service "http://localhost:9021/health" "Console Service"
log_info "Basic deployment ready!"
log_info "🌐 API endpoint: http://localhost:9020"
log_info "🖥️ Console UI: http://localhost:9021/rustfs/console/"
log_info "🔐 Credentials: basic-access / basic-secret"
log_info "🏥 Health checks:"
log_info " API: curl http://localhost:9020/health"
log_info " Console: curl http://localhost:9021/health"
}
# Scenario 2: Development environment
deploy_development() {
log_section "Scenario 2: Development Environment"
log_info "Starting RustFS development environment"
docker run -d \
--name rustfs-dev \
-p 9030:9000 \
-p 9031:9001 \
-e RUSTFS_CORS_ALLOWED_ORIGINS="*" \
-e RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
-e RUSTFS_ACCESS_KEY="dev-access" \
-e RUSTFS_SECRET_KEY="dev-secret" \
-e RUSTFS_OBS_LOGGER_LEVEL="debug" \
-v rustfs-dev-data:/data \
rustfs/rustfs:latest
# Wait for services to be ready
wait_for_service "http://localhost:9030/health" "Dev API Service"
wait_for_service "http://localhost:9031/health" "Dev Console Service"
log_info "Development deployment ready!"
log_info "🌐 API endpoint: http://localhost:9030"
log_info "🖥️ Console UI: http://localhost:9031/rustfs/console/"
log_info "🔐 Credentials: dev-access / dev-secret"
log_info "📊 Debug logging enabled"
log_info "🏥 Health checks:"
log_info " API: curl http://localhost:9030/health"
log_info " Console: curl http://localhost:9031/health"
}
# Scenario 3: Production-like environment with security
deploy_production() {
log_section "Scenario 3: Production-like Deployment"
log_info "Starting RustFS production-like environment with security"
# Generate secure credentials
ACCESS_KEY=$(openssl rand -hex 16)
SECRET_KEY=$(openssl rand -hex 32)
# Save credentials for reference
cat > rustfs-prod-credentials.env << EOF
# RustFS Production Deployment Credentials
# Generated: $(date)
RUSTFS_ACCESS_KEY=$ACCESS_KEY
RUSTFS_SECRET_KEY=$SECRET_KEY
EOF
chmod 600 rustfs-prod-credentials.env
docker run -d \
--name rustfs-prod \
-p 9040:9000 \
-p 127.0.0.1:9041:9001 \
-e RUSTFS_ADDRESS="0.0.0.0:9000" \
-e RUSTFS_CONSOLE_ADDRESS="0.0.0.0:9001" \
-e RUSTFS_CORS_ALLOWED_ORIGINS="https://myapp.example.com" \
-e RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="https://admin.example.com" \
-e RUSTFS_ACCESS_KEY="$ACCESS_KEY" \
-e RUSTFS_SECRET_KEY="$SECRET_KEY" \
-v rustfs-prod-data:/data \
rustfs/rustfs:latest
# Wait for services to be ready
wait_for_service "http://localhost:9040/health" "Prod API Service"
wait_for_service "http://127.0.0.1:9041/health" "Prod Console Service"
log_info "Production deployment ready!"
log_info "🌐 API endpoint: http://localhost:9040 (public)"
log_info "🖥️ Console UI: http://127.0.0.1:9041/rustfs/console/ (localhost only)"
log_info "🔐 Credentials: $ACCESS_KEY / $SECRET_KEY"
log_info "🔒 Security: Console restricted to localhost"
log_info "🏥 Health checks:"
log_info " API: curl http://localhost:9040/health"
log_info " Console: curl http://127.0.0.1:9041/health"
log_warn "⚠️ Console is restricted to localhost for security"
log_warn "⚠️ Credentials saved to rustfs-prod-credentials.env file"
}
# Function to show service status
show_status() {
log_section "Service Status"
echo "Running containers:"
docker ps --filter "name=rustfs-" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
echo -e "\nService endpoints:"
if docker ps --filter "name=rustfs-basic" --format "{{.Names}}" | grep -q rustfs-basic; then
echo " Basic API: http://localhost:9020"
echo " Basic Console: http://localhost:9021/rustfs/console/"
fi
if docker ps --filter "name=rustfs-dev" --format "{{.Names}}" | grep -q rustfs-dev; then
echo " Dev API: http://localhost:9030"
echo " Dev Console: http://localhost:9031/rustfs/console/"
fi
if docker ps --filter "name=rustfs-prod" --format "{{.Names}}" | grep -q rustfs-prod; then
echo " Prod API: http://localhost:9040"
echo " Prod Console: http://127.0.0.1:9041/rustfs/console/"
fi
}
# Function to test services
test_services() {
log_section "Testing Services"
# Test basic deployment
if docker ps --filter "name=rustfs-basic" --format "{{.Names}}" | grep -q rustfs-basic; then
log_info "Testing basic deployment..."
if curl -s http://localhost:9020/health | grep -q "ok"; then
log_info "✓ Basic API health check passed"
else
log_error "✗ Basic API health check failed"
fi
if curl -s http://localhost:9021/health | grep -q "console"; then
log_info "✓ Basic Console health check passed"
else
log_error "✗ Basic Console health check failed"
fi
fi
# Test development deployment
if docker ps --filter "name=rustfs-dev" --format "{{.Names}}" | grep -q rustfs-dev; then
log_info "Testing development deployment..."
if curl -s http://localhost:9030/health | grep -q "ok"; then
log_info "✓ Dev API health check passed"
else
log_error "✗ Dev API health check failed"
fi
if curl -s http://localhost:9031/health | grep -q "console"; then
log_info "✓ Dev Console health check passed"
else
log_error "✗ Dev Console health check failed"
fi
fi
# Test production deployment
if docker ps --filter "name=rustfs-prod" --format "{{.Names}}" | grep -q rustfs-prod; then
log_info "Testing production deployment..."
if curl -s http://localhost:9040/health | grep -q "ok"; then
log_info "✓ Prod API health check passed"
else
log_error "✗ Prod API health check failed"
fi
if curl -s http://127.0.0.1:9041/health | grep -q "console"; then
log_info "✓ Prod Console health check passed"
else
log_error "✗ Prod Console health check failed"
fi
fi
}
# Function to show logs
show_logs() {
log_section "Service Logs"
if [ -n "$1" ]; then
docker logs "$1"
else
echo "Available containers:"
docker ps --filter "name=rustfs-" --format "{{.Names}}"
echo -e "\nUsage: $0 logs <container-name>"
fi
}
# Main menu
case "${1:-menu}" in
"basic")
cleanup
deploy_basic
;;
"dev")
cleanup
deploy_development
;;
"prod")
cleanup
deploy_production
;;
"all")
cleanup
deploy_basic
deploy_development
deploy_production
show_status
;;
"status")
show_status
;;
"test")
test_services
;;
"logs")
show_logs "$2"
;;
"cleanup")
cleanup
docker volume rm rustfs-basic-data rustfs-dev-data rustfs-prod-data 2>/dev/null || true
log_info "Cleanup completed"
;;
"menu"|*)
echo "RustFS Enhanced Docker Deployment Examples"
echo ""
echo "Usage: $0 [command]"
echo ""
echo "Commands:"
echo " basic - Deploy basic RustFS with port mapping"
echo " dev - Deploy development environment"
echo " prod - Deploy production-like environment"
echo " all - Deploy all scenarios"
echo " status - Show status of running containers"
echo " test - Test all running services"
echo " logs - Show logs for specific container"
echo " cleanup - Clean up all containers and volumes"
echo ""
echo "Examples:"
echo " $0 basic # Deploy basic setup"
echo " $0 status # Check running services"
echo " $0 logs rustfs-dev # Show dev container logs"
echo " $0 cleanup # Clean everything up"
;;
esac
docs/examples/docker/enhanced-security-deployment.sh
-206
View File
@@ -1,206 +0,0 @@
#!/usr/bin/env bash
# RustFS Enhanced Security Deployment Script
# This script demonstrates production-ready deployment with enhanced security features
set -e
# Configuration
RUSTFS_IMAGE="${RUSTFS_IMAGE:-rustfs/rustfs:latest}"
CONTAINER_NAME="${CONTAINER_NAME:-rustfs-secure}"
DATA_DIR="${DATA_DIR:-./data}"
CERTS_DIR="${CERTS_DIR:-./certs}"
CONSOLE_PORT="${CONSOLE_PORT:-9443}"
API_PORT="${API_PORT:-9000}"
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
log() {
echo -e "${BLUE}[INFO]${NC} $1"
}
warn() {
echo -e "${YELLOW}[WARN]${NC} $1"
}
error() {
echo -e "${RED}[ERROR]${NC} $1"
exit 1
}
success() {
echo -e "${GREEN}[SUCCESS]${NC} $1"
}
# Check if Docker is available
check_docker() {
if ! command -v docker &> /dev/null; then
error "Docker is not installed or not in PATH"
fi
log "Docker is available"
}
# Generate TLS certificates for console
generate_certs() {
if [[ ! -d "$CERTS_DIR" ]]; then
mkdir -p "$CERTS_DIR"
log "Created certificates directory: $CERTS_DIR"
fi
if [[ ! -f "$CERTS_DIR/console.crt" ]] || [[ ! -f "$CERTS_DIR/console.key" ]]; then
log "Generating TLS certificates for console..."
openssl req -x509 -newkey rsa:4096 \
-keyout "$CERTS_DIR/console.key" \
-out "$CERTS_DIR/console.crt" \
-days 365 -nodes \
-subj "/C=US/ST=CA/L=SF/O=RustFS/CN=localhost"
chmod 600 "$CERTS_DIR/console.key"
chmod 644 "$CERTS_DIR/console.crt"
success "TLS certificates generated"
else
log "TLS certificates already exist"
fi
}
# Create data directory
create_data_dir() {
if [[ ! -d "$DATA_DIR" ]]; then
mkdir -p "$DATA_DIR"
log "Created data directory: $DATA_DIR"
fi
}
# Generate secure credentials
generate_credentials() {
if [[ -z "$RUSTFS_ACCESS_KEY" ]]; then
export RUSTFS_ACCESS_KEY="admin-$(openssl rand -hex 8)"
log "Generated access key: $RUSTFS_ACCESS_KEY"
fi
if [[ -z "$RUSTFS_SECRET_KEY" ]]; then
export RUSTFS_SECRET_KEY="$(openssl rand -hex 32)"
log "Generated secret key: [HIDDEN]"
fi
# Save credentials to .env file
cat > .env << EOF
RUSTFS_ACCESS_KEY=$RUSTFS_ACCESS_KEY
RUSTFS_SECRET_KEY=$RUSTFS_SECRET_KEY
EOF
chmod 600 .env
success "Credentials saved to .env file"
}
# Stop existing container
stop_existing() {
if docker ps -a --format "table {{.Names}}" | grep -q "^$CONTAINER_NAME\$"; then
log "Stopping existing container: $CONTAINER_NAME"
docker stop "$CONTAINER_NAME" 2>/dev/null || true
docker rm "$CONTAINER_NAME" 2>/dev/null || true
fi
}
# Deploy RustFS with enhanced security
deploy_rustfs() {
log "Deploying RustFS with enhanced security..."
docker run -d \
--name "$CONTAINER_NAME" \
--restart unless-stopped \
-p "$CONSOLE_PORT:9001" \
-p "$API_PORT:9000" \
-v "$(pwd)/$DATA_DIR:/data" \
-v "$(pwd)/$CERTS_DIR:/certs:ro" \
-e RUSTFS_CONSOLE_TLS_ENABLE=true \
-e RUSTFS_CONSOLE_TLS_CERT=/certs/console.crt \
-e RUSTFS_CONSOLE_TLS_KEY=/certs/console.key \
-e RUSTFS_CONSOLE_RATE_LIMIT_ENABLE=true \
-e RUSTFS_CONSOLE_RATE_LIMIT_RPM=60 \
-e RUSTFS_CONSOLE_AUTH_TIMEOUT=1800 \
-e RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="https://localhost:$CONSOLE_PORT" \
-e RUSTFS_CORS_ALLOWED_ORIGINS="http://localhost:$API_PORT" \
-e RUSTFS_ACCESS_KEY="$RUSTFS_ACCESS_KEY" \
-e RUSTFS_SECRET_KEY="$RUSTFS_SECRET_KEY" \
"$RUSTFS_IMAGE" /data
# Wait for container to start
sleep 5
if docker ps --format "table {{.Names}}" | grep -q "^$CONTAINER_NAME\$"; then
success "RustFS deployed successfully"
else
error "Failed to deploy RustFS"
fi
}
# Check service health
check_health() {
log "Checking service health..."
# Check console health
if curl -k -s "https://localhost:$CONSOLE_PORT/health" | jq -e '.status == "ok"' > /dev/null 2>&1; then
success "Console service is healthy"
else
warn "Console service health check failed"
fi
# Check API health
if curl -s "http://localhost:$API_PORT/health" | jq -e '.status == "ok"' > /dev/null 2>&1; then
success "API service is healthy"
else
warn "API service health check failed"
fi
}
# Display access information
show_access_info() {
echo
echo "=========================================="
echo " RustFS Access Information"
echo "=========================================="
echo
echo "🌐 Console (HTTPS): https://localhost:$CONSOLE_PORT/rustfs/console/"
echo "🔧 API Endpoint: http://localhost:$API_PORT"
echo "🏥 Console Health: https://localhost:$CONSOLE_PORT/health"
echo "🏥 API Health: http://localhost:$API_PORT/health"
echo
echo "🔐 Credentials:"
echo " Access Key: $RUSTFS_ACCESS_KEY"
echo " Secret Key: [Check .env file]"
echo
echo "📝 Logs: docker logs $CONTAINER_NAME"
echo "🛑 Stop: docker stop $CONTAINER_NAME"
echo
echo "⚠️ Note: Console uses self-signed certificate"
echo " Accept the certificate warning in your browser"
echo
}
# Main deployment flow
main() {
log "Starting RustFS Enhanced Security Deployment"
check_docker
create_data_dir
generate_certs
generate_credentials
stop_existing
deploy_rustfs
# Wait a bit for services to start
sleep 10
check_health
show_access_info
success "Deployment completed successfully!"
}
# Run main function
main "$@"
docs/examples/mnmd/CHECKLIST.md
-329
View File
@@ -1,329 +0,0 @@
# MNMD Deployment Checklist
This checklist provides step-by-step verification for deploying RustFS in MNMD (Multi-Node Multi-Drive) mode using
Docker.
## Pre-Deployment Checks
### 1. System Requirements
- [ ] Docker Engine 20.10+ installed
- [ ] Docker Compose 2.0+ installed
- [ ] At least 8GB RAM available
- [ ] At least 40GB disk space available (for 4 nodes × 4 volumes)
Verify with:
```bash
docker --version
docker-compose --version
free -h
df -h
```
### 2. File System Checks
- [ ] Using XFS, ext4, or another suitable filesystem (not NFS for production)
- [ ] File system supports extended attributes
Verify with:
```bash
df -T | grep -E '(xfs|ext4)'
```
### 3. Permissions and SELinux
- [ ] Current user is in `docker` group or can run `sudo docker`
- [ ] SELinux is properly configured (if enabled)
Verify with:
```bash
groups | grep docker
getenforce # If enabled, should show "Permissive" or "Enforcing" with proper policies
```
### 4. Network Configuration
- [ ] Ports 9000-9031 are available
- [ ] No firewall blocking Docker bridge network
Verify with:
```bash
# Check if ports are free
netstat -tuln | grep -E ':(9000|9001|9010|9011|9020|9021|9030|9031)'
# Should return nothing if ports are free
```
### 5. Files Present
- [ ] `docker-compose.yml` exists in current directory
Verify with:
```bash
cd docs/examples/mnmd
ls -la
chmod +x wait-and-start.sh # If needed
```
## Deployment Steps
### 1. Start the Cluster
- [ ] Navigate to the example directory
- [ ] Pull the latest RustFS image
- [ ] Start the cluster
```bash
cd docs/examples/mnmd
docker-compose pull
docker-compose up -d
```
### 2. Monitor Startup
- [ ] Watch container logs during startup
- [ ] Verify no VolumeNotFound errors
- [ ] Check that peer discovery completes
```bash
# Watch all logs
docker-compose logs -f
# Watch specific node
docker-compose logs -f rustfs-node1
# Look for successful startup messages
docker-compose logs | grep -i "ready\|listening\|started"
```
### 3. Verify Container Status
- [ ] All 4 containers are running
- [ ] All 4 containers show as healthy
```bash
docker-compose ps
# Expected output: 4 containers in "Up" state with "healthy" status
```
### 4. Check Health Endpoints
- [ ] API health endpoints respond on all nodes
- [ ] Console health endpoints respond on all nodes
```bash
# Test API endpoints
curl http://localhost:9000/health
curl http://localhost:9010/health
curl http://localhost:9020/health
curl http://localhost:9030/health
# Test Console endpoints
curl http://localhost:9001/health
curl http://localhost:9011/health
curl http://localhost:9021/health
curl http://localhost:9031/health
# All should return successful health status
```
## Post-Deployment Verification
### 1. In-Container Checks
- [ ] Data directories exist
- [ ] Directories have correct permissions
- [ ] RustFS process is running
```bash
# Check node1
docker exec rustfs-node1 ls -la /data/
docker exec rustfs-node1 ps aux | grep rustfs
# Verify all 4 data directories exist
docker exec rustfs-node1 ls -d /data/rustfs{1..4}
```
### 2. DNS and Network Validation
- [ ] Service names resolve correctly
- [ ] Inter-node connectivity works
```bash
# DNS resolution test
docker exec rustfs-node1 nslookup rustfs-node2
docker exec rustfs-node1 nslookup rustfs-node3
docker exec rustfs-node1 nslookup rustfs-node4
# Connectivity test (using nc if available)
docker exec rustfs-node1 nc -zv rustfs-node2 9000
docker exec rustfs-node1 nc -zv rustfs-node3 9000
docker exec rustfs-node1 nc -zv rustfs-node4 9000
# Or using telnet/curl
docker exec rustfs-node1 curl -v http://rustfs-node2:9000/health
```
### 3. Volume Configuration Validation
- [ ] RUSTFS_VOLUMES environment variable is correct
- [ ] All 16 endpoints are configured (4 nodes × 4 drives)
```bash
# Check environment variable
docker exec rustfs-node1 env | grep RUSTFS_VOLUMES
# Expected output:
# RUSTFS_VOLUMES=http://rustfs-node{1...4}:9000/data/rustfs{1...4}
```
### 4. Cluster Functionality
- [ ] Can list buckets via API
- [ ] Can create a bucket
- [ ] Can upload an object
- [ ] Can download an object
```bash
# Configure AWS CLI or s3cmd
export AWS_ACCESS_KEY_ID=rustfsadmin
export AWS_SECRET_ACCESS_KEY=rustfsadmin
# Using AWS CLI (if installed)
aws --endpoint-url http://localhost:9000 s3 mb s3://test-bucket
aws --endpoint-url http://localhost:9000 s3 ls
echo "test content" > test.txt
aws --endpoint-url http://localhost:9000 s3 cp test.txt s3://test-bucket/
aws --endpoint-url http://localhost:9000 s3 ls s3://test-bucket/
aws --endpoint-url http://localhost:9000 s3 cp s3://test-bucket/test.txt downloaded.txt
cat downloaded.txt
# Or using curl
curl -X PUT http://localhost:9000/test-bucket \
-H "Host: localhost:9000" \
--user rustfsadmin:rustfsadmin
```
### 5. Healthcheck Verification
- [ ] Docker reports all services as healthy
- [ ] Healthcheck scripts work in containers
```bash
# Check Docker health status
docker inspect rustfs-node1 --format='{{.State.Health.Status}}'
docker inspect rustfs-node2 --format='{{.State.Health.Status}}'
docker inspect rustfs-node3 --format='{{.State.Health.Status}}'
docker inspect rustfs-node4 --format='{{.State.Health.Status}}'
# All should return "healthy"
# Test healthcheck command manually
docker exec rustfs-node1 nc -z localhost 9000
echo $? # Should be 0
```
## Troubleshooting Checks
### If VolumeNotFound Error Occurs
- [ ] Verify volume indexing starts at 1, not 0
- [ ] Check that RUSTFS_VOLUMES matches mounted paths
- [ ] Ensure all /data/rustfs{1..4} directories exist
```bash
# Check mounted volumes
docker inspect rustfs-node1 | jq '.[].Mounts'
# Verify directories in container
docker exec rustfs-node1 ls -la /data/
```
### If Healthcheck Fails
- [ ] Check if `nc` is available in the image
- [ ] Try alternative healthcheck (curl/wget)
- [ ] Increase `start_period` in docker-compose.yml
```bash
# Check if nc is available
docker exec rustfs-node1 which nc
# Test healthcheck manually
docker exec rustfs-node1 nc -z localhost 9000
# Check logs for errors
docker-compose logs rustfs-node1 | grep -i error
```
### If Startup Takes Too Long
- [ ] Check peer discovery timeout in logs
- [ ] Verify network connectivity between nodes
- [ ] Consider increasing timeout in wait-and-start.sh
```bash
# Check startup logs
docker-compose logs rustfs-node1 | grep -i "waiting\|peer\|timeout"
# Check network
docker network inspect mnmd_rustfs-mnmd
```
### If Containers Crash or Restart
- [ ] Review container logs
- [ ] Check resource usage (CPU/Memory)
- [ ] Verify no port conflicts
```bash
# View last crash logs
docker-compose logs --tail=100 rustfs-node1
# Check resource usage
docker stats --no-stream
# Check restart count
docker-compose ps
```
## Cleanup Checklist
When done testing:
- [ ] Stop the cluster: `docker-compose down`
- [ ] Remove volumes (optional, destroys data): `docker-compose down -v`
- [ ] Clean up dangling images: `docker image prune`
- [ ] Verify ports are released: `netstat -tuln | grep -E ':(9000|9001|9010|9011|9020|9021|9030|9031)'`
## Production Deployment Additional Checks
Before deploying to production:
- [ ] Change default credentials (RUSTFS_ACCESS_KEY, RUSTFS_SECRET_KEY)
- [ ] Configure TLS certificates
- [ ] Set up proper logging and monitoring
- [ ] Configure backups for volumes
- [ ] Review and adjust resource limits
- [ ] Set up external load balancer (if needed)
- [ ] Document disaster recovery procedures
- [ ] Test failover scenarios
- [ ] Verify data persistence after container restart
## Summary
This checklist ensures:
- ✓ Correct disk indexing (1..4 instead of 0..3)
- ✓ Proper startup coordination via wait-and-start.sh
- ✓ Service discovery via Docker service names
- ✓ Health checks function correctly
- ✓ All 16 endpoints (4 nodes × 4 drives) are operational
- ✓ No VolumeNotFound errors occur
For more details, see [README.md](./README.md) in this directory.
docs/examples/mnmd/README.md
-268
View File
@@ -1,268 +0,0 @@
# RustFS MNMD (Multi-Node Multi-Drive) Docker Example
This directory contains a complete, ready-to-use MNMD deployment example for RustFS with 4 nodes and 4 drives per node (
4x4 configuration).
## Overview
This example addresses common deployment issues including:
- **VolumeNotFound errors** - Fixed by using correct disk indexing (`/data/rustfs{1...4}` instead of
`/data/rustfs{0...3}`)
- **Startup race conditions** - Solved with a simple `sleep` command in each service.
- **Service discovery** - Uses Docker service names (`rustfs-node{1..4}`) instead of hard-coded IPs
- **Health checks** - Implements proper health monitoring with `nc` (with alternatives documented)
## Quick Start
From this directory (`docs/examples/mnmd`), run:
```bash
# Start the cluster
docker-compose up -d
# Check the status
docker-compose ps
# View logs
docker-compose logs -f
# Test the deployment
curl http://localhost:9000/health
curl http://localhost:9001/rustfs/console/health
# Run comprehensive tests
./test-deployment.sh
# Stop the cluster
docker-compose down
# Clean up volumes (WARNING: deletes all data)
docker-compose down -v
```
## Configuration Details
### Volume Configuration
The example uses the following volume configuration:
```bash
RUSTFS_VOLUMES=http://rustfs-node{1...4}:9000/data/rustfs{1...4}
```
This expands to 16 endpoints (4 nodes × 4 drives):
- Node 1: `/data/rustfs1`, `/data/rustfs2`, `/data/rustfs3`, `/data/rustfs4`
- Node 2: `/data/rustfs1`, `/data/rustfs2`, `/data/rustfs3`, `/data/rustfs4`
- Node 3: `/data/rustfs1`, `/data/rustfs2`, `/data/rustfs3`, `/data/rustfs4`
- Node 4: `/data/rustfs1`, `/data/rustfs2`, `/data/rustfs3`, `/data/rustfs4`
**Important:** Disk indexing starts at 1 to match the mounted paths (`/data/rustfs1..4`).
### Port Mappings
| Node | API Port | Console Port |
|-------|----------|--------------|
| node1 | 9000 | 9001 |
| node2 | 9010 | 9011 |
| node3 | 9020 | 9021 |
| node4 | 9030 | 9031 |
### Startup Coordination
To prevent race conditions during startup where nodes might not find each other, a simple `sleep 3` command is added to
each service's command. This provides a brief delay, allowing the network and other services to initialize before RustFS
starts. For more complex scenarios, a more robust health-check dependency or an external entrypoint script might be
required.
### Health Checks
Default health check using `nc` (netcat):
```yaml
healthcheck:
test: [ "CMD-SHELL", "nc -z localhost 9000 || exit 1" ]
interval: 10s
timeout: 5s
retries: 3
start_period: 30s
```
#### Alternative Health Checks
If your base image lacks `nc`, use one of these alternatives:
**Using curl:**
```yaml
healthcheck:
test: [ "CMD-SHELL", "curl -f http://localhost:9000/health || exit 1" ]
interval: 10s
timeout: 5s
retries: 3
start_period: 30s
```
**Using wget:**
```yaml
healthcheck:
test: [ "CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1" ]
interval: 10s
timeout: 5s
retries: 3
start_period: 30s
```
### Brace Expansion Alternatives
If your Docker Compose runtime doesn't support brace expansion (`{1...4}`), replace with explicit endpoints:
```yaml
environment:
- RUSTFS_VOLUMES=http://rustfs-node1:9000/data/rustfs1,http://rustfs-node1:9000/data/rustfs2,http://rustfs-node1:9000/data/rustfs3,http://rustfs-node1:9000/data/rustfs4,http://rustfs-node2:9000/data/rustfs1,http://rustfs-node2:9000/data/rustfs2,http://rustfs-node2:9000/data/rustfs3,http://rustfs-node2:9000/data/rustfs4,http://rustfs-node3:9000/data/rustfs1,http://rustfs-node3:9000/data/rustfs2,http://rustfs-node3:9000/data/rustfs3,http://rustfs-node3:9000/data/rustfs4,http://rustfs-node4:9000/data/rustfs1,http://rustfs-node4:9000/data/rustfs2,http://rustfs-node4:9000/data/rustfs3,http://rustfs-node4:9000/data/rustfs4
```
## Using RUSTFS_CMD
The `RUSTFS_CMD` environment variable provides a fallback when no command is specified:
```yaml
environment:
- RUSTFS_CMD=rustfs # Default fallback command
```
This allows the entrypoint to execute the correct command when Docker doesn't provide one.
## Testing the Deployment
After starting the cluster, verify it's working:
### Automated Testing
Use the provided test script for comprehensive validation:
```bash
./test-deployment.sh
```
This script tests:
- Container status (4/4 running)
- Health checks (4/4 healthy)
- API endpoints (4 ports)
- Console endpoints (4 ports)
- Inter-node connectivity
- Data directory existence
### Manual Testing
For manual verification:
```bash
# 1. Check all containers are healthy
docker-compose ps
# 2. Test API endpoints
for port in 9000 9010 9020 9030; do
echo "Testing port $port..."
curl -s http://localhost:${port}/health | jq '.'
done
# 3. Test console endpoints
for port in 9001 9011 9021 9031; do
echo "Testing console port $port..."
curl -s http://localhost:${port}/rustfs/console/health | jq '.'
done
# 4. Check inter-node connectivity
docker exec rustfs-node1 nc -zv rustfs-node2 9000
docker exec rustfs-node1 nc -zv rustfs-node3 9000
docker exec rustfs-node1 nc -zv rustfs-node4 9000
```
## Troubleshooting
### VolumeNotFound Error
**Symptom:** Error message about `/data/rustfs0` not found.
**Solution:** This example uses `/data/rustfs{1...4}` indexing to match the mounted Docker volumes. Ensure your
`RUSTFS_VOLUMES` configuration starts at index 1, not 0.
### Health Check Failures
**Symptom:** Containers show as unhealthy.
**Solutions:**
1. Check if `nc` is available: `docker exec rustfs-node1 which nc`
2. Use alternative health checks (curl/wget) as documented above
3. Increase `start_period` if nodes need more time to initialize
### Startup Timeouts
**Symptom:** Services timeout waiting for peers.
**Solutions:**
1. Check logs: `docker-compose logs rustfs-node1`
2. Verify network connectivity: `docker-compose exec rustfs-node1 ping rustfs-node2`
3. Consider increasing the `sleep` duration in the `docker-compose.yml` `command` directive if a longer delay is needed.
### Permission Issues
**Symptom:** Cannot create directories or write data.
**Solution:** Ensure volumes have correct permissions or set `RUSTFS_UID` and `RUSTFS_GID` environment variables.
## Advanced Configuration
### Custom Credentials
Replace default credentials in production:
```yaml
environment:
- RUSTFS_ACCESS_KEY=your_access_key
- RUSTFS_SECRET_KEY=your_secret_key
```
### TLS Configuration
Add TLS certificates:
```yaml
volumes:
- ./certs:/opt/tls:ro
environment:
- RUSTFS_TLS_PATH=/opt/tls
```
### Resource Limits
Add resource constraints:
```yaml
deploy:
resources:
limits:
cpus: '2'
memory: 4G
reservations:
cpus: '1'
memory: 2G
```
## See Also
- [CHECKLIST.md](./CHECKLIST.md) - Step-by-step verification guide
- [../../console-separation.md](../../console-separation.md) - Console & endpoint service separation guide
- [../../../examples/docker-comprehensive.yml](../../../examples/docker-comprehensive.yml) - More deployment examples
- [Issue #618](https://github.com/rustfs/rustfs/issues/618) - Original VolumeNotFound issue
## References
- RustFS Documentation: https://rustfs.com
- Docker Compose Documentation: https://docs.docker.com/compose/
docs/examples/mnmd/docker-compose.mtls.yml
-32
View File
@@ -1,32 +0,0 @@
services:
mnmd:
image: ghcr.io/your-org/mnmd:latest
container_name: mnmd
ports:
- "8443:8443"
volumes:
- ./tls:/tls:ro
environment:
# Example mnmd settings (adapt to your image)
- MNMD_LISTEN_ADDR=0.0.0.0:8443
- MNMD_TLS_CERT=/tls/server_cert.pem
- MNMD_TLS_KEY=/tls/server_key.pem
- MNMD_TLS_CLIENT_CA=/tls/ca.crt
rustfs:
image: ghcr.io/rustfs/rustfs:latest
container_name: rustfs
depends_on:
- mnmd
environment:
- RUSTFS_TLS_PATH=/tls
- RUSTFS_TRUST_SYSTEM_CA=false
- RUSTFS_TRUST_LEAF_CERT_AS_CA=false
# Enable outbound mTLS (client identity) for MNMD
- RUSTFS_MTLS_CLIENT_CERT=/tls/client_cert.pem
- RUSTFS_MTLS_CLIENT_KEY=/tls/client_key.pem
# MNMD address configured to https
- RUSTFS_MNMD_ADDR=https://mnmd:8443
- RUSTFS_MNMD_DOMAIN=mnmd
volumes:
- ./tls:/tls:ro
docs/examples/mnmd/docker-compose.yml
-121
View File
@@ -1,121 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# MNMD (Multi-Node Multi-Drive) Docker Compose Example
# 4 nodes x 4 drives configuration
# This example demonstrates a complete, ready-to-use MNMD deployment
# addressing startup coordination and VolumeNotFound issues.
x-node-template: &node-template
image: rustfs/rustfs:latest
environment:
# Use service names and correct disk indexing (1..4 to match mounted paths)
- RUSTFS_VOLUMES=http://rustfs-node{1...4}:9000/data/rustfs{1...4}
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
- RUSTFS_ACCESS_KEY=rustfsadmin
- RUSTFS_SECRET_KEY=rustfsadmin
- RUSTFS_CMD=rustfs
- RUSTFS_OBS_LOG_DIRECTORY=/logs
command: [ "sh", "-c", "sleep 3 && rustfs" ]
healthcheck:
test:
[
"CMD",
"sh", "-c",
"curl -f http://localhost:9000/health && curl -f http://localhost:9001/rustfs/console/health"
]
interval: 10s
timeout: 5s
retries: 3
start_period: 30s
networks:
- rustfs-mnmd
services:
rustfs-node1:
<<: *node-template
container_name: rustfs-node1
hostname: rustfs-node1
ports:
- "9000:9000" # API endpoint
- "9001:9001" # Console
volumes:
- node1-data1:/data/rustfs1
- node1-data2:/data/rustfs2
- node1-data3:/data/rustfs3
- node1-data4:/data/rustfs4
rustfs-node2:
<<: *node-template
container_name: rustfs-node2
hostname: rustfs-node2
ports:
- "9010:9000" # API endpoint
- "9011:9001" # Console
volumes:
- node2-data1:/data/rustfs1
- node2-data2:/data/rustfs2
- node2-data3:/data/rustfs3
- node2-data4:/data/rustfs4
rustfs-node3:
<<: *node-template
container_name: rustfs-node3
hostname: rustfs-node3
ports:
- "9020:9000" # API endpoint
- "9021:9001" # Console
volumes:
- node3-data1:/data/rustfs1
- node3-data2:/data/rustfs2
- node3-data3:/data/rustfs3
- node3-data4:/data/rustfs4
rustfs-node4:
<<: *node-template
container_name: rustfs-node4
hostname: rustfs-node4
ports:
- "9030:9000" # API endpoint
- "9031:9001" # Console
volumes:
- node4-data1:/data/rustfs1
- node4-data2:/data/rustfs2
- node4-data3:/data/rustfs3
- node4-data4:/data/rustfs4
networks:
rustfs-mnmd:
driver: bridge
volumes:
node1-data1:
node1-data2:
node1-data3:
node1-data4:
node2-data1:
node2-data2:
node2-data3:
node2-data4:
node3-data1:
node3-data2:
node3-data3:
node3-data4:
node4-data1:
node4-data2:
node4-data3:
node4-data4:
docs/examples/mnmd/test-deployment.sh
-172
View File
@@ -1,172 +0,0 @@
#!/usr/bin/env bash
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# test-deployment.sh - Quick test script for MNMD deployment
# Usage: ./test-deployment.sh
set -e
# Colors for output
GREEN='\033[0;32m'
RED='\033[0;31m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
echo "========================================="
echo "RustFS MNMD Deployment Test"
echo "========================================="
echo ""
# Test 1: Check if all containers are running
echo "Test 1: Checking container status..."
RUNNING=$(docker-compose ps | grep -c "Up" || echo "0")
if [ "$RUNNING" -eq 4 ]; then
echo -e "${GREEN}✓ All 4 containers are running${NC}"
else
echo -e "${RED}✗ Only $RUNNING/4 containers are running${NC}"
docker-compose ps
exit 1
fi
echo ""
# Test 2: Check health status
echo "Test 2: Checking health status..."
HEALTHY=0
for node in rustfs-node1 rustfs-node2 rustfs-node3 rustfs-node4; do
STATUS=$(docker inspect "$node" --format='{{.State.Health.Status}}' 2>/dev/null || echo "unknown")
if [ "$STATUS" = "healthy" ]; then
echo -e " ${GREEN}✓ $node is healthy${NC}"
HEALTHY=$((HEALTHY + 1))
elif [ "$STATUS" = "starting" ]; then
echo -e " ${YELLOW}⚠ $node is starting (wait a moment)${NC}"
else
echo -e " ${RED}✗ $node status: $STATUS${NC}"
fi
done
if [ "$HEALTHY" -eq 4 ]; then
echo -e "${GREEN}✓ All containers are healthy${NC}"
elif [ "$HEALTHY" -gt 0 ]; then
echo -e "${YELLOW}⚠ $HEALTHY/4 containers are healthy (some may still be starting)${NC}"
else
echo -e "${RED}✗ No containers are healthy${NC}"
exit 1
fi
echo ""
# Test 3: Check API endpoints
echo "Test 3: Testing API endpoints..."
PORTS=(9000 9010 9020 9030)
API_SUCCESS=0
for port in "${PORTS[@]}"; do
if curl -sf http://localhost:${port}/health >/dev/null 2>&1; then
echo -e " ${GREEN}✓ API on port $port is responding${NC}"
API_SUCCESS=$((API_SUCCESS + 1))
else
echo -e " ${RED}✗ API on port $port is not responding${NC}"
fi
done
if [ "$API_SUCCESS" -eq 4 ]; then
echo -e "${GREEN}✓ All API endpoints are working${NC}"
else
echo -e "${YELLOW}⚠ $API_SUCCESS/4 API endpoints are working${NC}"
fi
echo ""
# Test 4: Check Console endpoints
echo "Test 4: Testing Console endpoints..."
CONSOLE_PORTS=(9001 9011 9021 9031)
CONSOLE_SUCCESS=0
for port in "${CONSOLE_PORTS[@]}"; do
if curl -sf http://localhost:${port}/rustfs/console/health >/dev/null 2>&1; then
echo -e " ${GREEN}✓ Console on port $port is responding${NC}"
CONSOLE_SUCCESS=$((CONSOLE_SUCCESS + 1))
else
echo -e " ${RED}✗ Console on port $port is not responding${NC}"
fi
done
if [ "$CONSOLE_SUCCESS" -eq 4 ]; then
echo -e "${GREEN}✓ All Console endpoints are working${NC}"
else
echo -e "${YELLOW}⚠ $CONSOLE_SUCCESS/4 Console endpoints are working${NC}"
fi
echo ""
# Test 5: Check inter-node connectivity
echo "Test 5: Testing inter-node connectivity..."
CONN_SUCCESS=0
for node in rustfs-node2 rustfs-node3 rustfs-node4; do
if docker exec rustfs-node1 nc -z "$node" 9000 2>/dev/null; then
echo -e " ${GREEN}✓ node1 → $node connection OK${NC}"
CONN_SUCCESS=$((CONN_SUCCESS + 1))
else
echo -e " ${RED}✗ node1 → $node connection failed${NC}"
fi
done
if [ "$CONN_SUCCESS" -eq 3 ]; then
echo -e "${GREEN}✓ All inter-node connections are working${NC}"
else
echo -e "${YELLOW}⚠ $CONN_SUCCESS/3 inter-node connections are working${NC}"
fi
echo ""
# Test 6: Verify data directories
echo "Test 6: Verifying data directories..."
DIR_SUCCESS=0
for i in {1..4}; do
if docker exec rustfs-node1 test -d "/data/rustfs${i}"; then
DIR_SUCCESS=$((DIR_SUCCESS + 1))
else
echo -e " ${RED}✗ /data/rustfs${i} not found in node1${NC}"
fi
done
if [ "$DIR_SUCCESS" -eq 4 ]; then
echo -e "${GREEN}✓ All data directories exist${NC}"
else
echo -e "${RED}✗ Only $DIR_SUCCESS/4 data directories exist${NC}"
fi
echo ""
# Summary
echo "========================================="
echo "Test Summary"
echo "========================================="
echo "Containers running: $RUNNING/4"
echo "Healthy containers: $HEALTHY/4"
echo "API endpoints: $API_SUCCESS/4"
echo "Console endpoints: $CONSOLE_SUCCESS/4"
echo "Inter-node connections: $CONN_SUCCESS/3"
echo "Data directories: $DIR_SUCCESS/4"
echo ""
TOTAL=$((RUNNING + HEALTHY + API_SUCCESS + CONSOLE_SUCCESS + CONN_SUCCESS + DIR_SUCCESS))
MAX_SCORE=23
if [ "$TOTAL" -eq "$MAX_SCORE" ]; then
echo -e "${GREEN}✓ All tests passed! Deployment is working correctly.${NC}"
exit 0
elif [ "$TOTAL" -ge 20 ]; then
echo -e "${YELLOW}⚠ Most tests passed. Some components may still be starting up.${NC}"
echo " Try running this script again in a few moments."
exit 0
else
echo -e "${RED}✗ Some tests failed. Check the output above and logs for details.${NC}"
echo " Run 'docker-compose logs' for more information."
exit 1
fi
docs/fix-large-file-upload-freeze.md
-192
View File
@@ -1,192 +0,0 @@
# Fix for Large File Upload Freeze Issue
## Problem Description
When uploading large files (10GB-20GB) consecutively, uploads may freeze with the following error:
```
[2025-11-10 14:29:22.110443 +00:00] ERROR [s3s::service]
AwsChunkedStreamError: Underlying: error reading a body from connection
```
## Root Cause Analysis
### 1. Small Default Buffer Size
The issue was caused by using `tokio_util::io::StreamReader::new()` which has a default buffer size of only **8KB**. This is far too small for large file uploads and causes:
- **Excessive system calls**: For a 10GB file with 8KB buffer, approximately **1.3 million read operations** are required
- **High CPU overhead**: Each read involves AWS chunked encoding/decoding overhead
- **Memory allocation pressure**: Frequent small allocations and deallocations
- **Increased timeout risk**: Slow read pace can trigger connection timeouts
### 2. AWS Chunked Encoding Overhead
AWS S3 uses chunked transfer encoding which adds metadata to each chunk. With a small buffer:
- More chunks need to be processed
- More metadata parsing operations
- Higher probability of parsing errors or timeouts
### 3. Connection Timeout Under Load
When multiple large files are uploaded consecutively:
- Small buffers lead to slow data transfer rates
- Network connections may timeout waiting for data
- The s3s library reports "error reading a body from connection"
## Solution
Wrap `StreamReader::new()` with `tokio::io::BufReader::with_capacity()` using a 1MB buffer size (`DEFAULT_READ_BUFFER_SIZE = 1024 * 1024`).
### Changes Made
Modified three critical locations in `rustfs/src/storage/ecfs.rs`:
1. **put_object** (line ~2338): Standard object upload
2. **put_object_extract** (line ~376): Archive file extraction and upload
3. **upload_part** (line ~2864): Multipart upload
### Before
```rust
let body = StreamReader::new(
body.map(|f| f.map_err(|e| std::io::Error::other(e.to_string())))
);
```
### After
```rust
// Use a larger buffer size (1MB) for StreamReader to prevent chunked stream read timeouts
// when uploading large files (10GB+). The default 8KB buffer is too small and causes
// excessive syscalls and potential connection timeouts.
let body = tokio::io::BufReader::with_capacity(
DEFAULT_READ_BUFFER_SIZE,
StreamReader::new(body.map(|f| f.map_err(|e| std::io::Error::other(e.to_string())))),
);
```
## Performance Impact
### For a 10GB File Upload:
| Metric | Before (8KB buffer) | After (1MB buffer) | Improvement |
|--------|--------------------|--------------------|-------------|
| Read operations | ~1,310,720 | ~10,240 | **99.2% reduction** |
| System call overhead | High | Low | Significantly reduced |
| Memory allocations | Frequent small | Less frequent large | More efficient |
| Timeout risk | High | Low | Much more stable |
### Benefits
1. **Reduced System Calls**: ~99% reduction in read operations for large files
2. **Lower CPU Usage**: Less AWS chunked encoding/decoding overhead
3. **Better Memory Efficiency**: Fewer allocations and better cache locality
4. **Improved Reliability**: Significantly reduced timeout probability
5. **Higher Throughput**: Better network utilization
## Testing Recommendations
To verify the fix works correctly, test the following scenarios:
1. **Single Large File Upload**
- Upload a 10GB file
- Upload a 20GB file
- Monitor for timeout errors
2. **Consecutive Large File Uploads**
- Upload 5 files of 10GB each consecutively
- Upload 3 files of 20GB each consecutively
- Ensure no freezing or timeout errors
3. **Multipart Upload**
- Upload large files using multipart upload
- Test with various part sizes
- Verify all parts complete successfully
4. **Archive Extraction**
- Upload large tar/gzip files with X-Amz-Meta-Snowball-Auto-Extract
- Verify extraction completes without errors
## Monitoring
After deployment, monitor these metrics:
- Upload completion rate for files > 1GB
- Average upload time for large files
- Frequency of chunked stream errors
- CPU usage during uploads
- Memory usage during uploads
## Related Configuration
The buffer size is defined in `crates/ecstore/src/set_disk.rs`:
```rust
pub const DEFAULT_READ_BUFFER_SIZE: usize = 1024 * 1024; // 1 MB
```
This value is used consistently across the codebase for stream reading operations.
## Additional Considerations
### Implementation Details
The solution uses `tokio::io::BufReader` to wrap the `StreamReader`, as `tokio-util 0.7.17` does not provide a `StreamReader::with_capacity()` method. The `BufReader` provides the same buffering benefits while being compatible with the current tokio-util version.
### Adaptive Buffer Sizing (Implemented)
The fix now includes **dynamic adaptive buffer sizing** based on file size for optimal performance and memory usage:
```rust
/// Calculate adaptive buffer size based on file size for optimal streaming performance.
fn get_adaptive_buffer_size(file_size: i64) -> usize {
match file_size {
// Unknown size or negative (chunked/streaming): use 1MB buffer for safety
size if size < 0 => 1024 * 1024,
// Small files (< 1MB): use 64KB to minimize memory overhead
size if size < 1_048_576 => 65_536,
// Medium files (1MB - 100MB): use 256KB for balanced performance
size if size < 104_857_600 => 262_144,
// Large files (>= 100MB): use 1MB buffer for maximum throughput
_ => 1024 * 1024,
}
}
```
**Benefits**:
- **Memory Efficiency**: Small files use smaller buffers (64KB), reducing memory overhead
- **Balanced Performance**: Medium files use 256KB buffers for optimal balance
- **Maximum Throughput**: Large files (100MB+) use 1MB buffers to minimize syscalls
- **Automatic Selection**: Buffer size is chosen automatically based on content-length
**Performance Impact by File Size**:
| File Size | Buffer Size | Memory Saved vs Fixed 1MB | Syscalls (approx) |
|-----------|-------------|--------------------------|-------------------|
| 100 KB | 64 KB | 960 KB (94% reduction) | ~2 |
| 10 MB | 256 KB | 768 KB (75% reduction) | ~40 |
| 100 MB | 1 MB | 0 KB (same) | ~100 |
| 10 GB | 1 MB | 0 KB (same) | ~10,240 |
### Future Improvements
1. **Connection Keep-Alive**: Ensure HTTP keep-alive is properly configured for consecutive uploads
2. **Rate Limiting**: Consider implementing upload rate limiting to prevent resource exhaustion
3. **Configurable Thresholds**: Make buffer size thresholds configurable via environment variables or config file
### Alternative Approaches Considered
1. **Increase s3s timeout**: Would only mask the problem, not fix the root cause
2. **Retry logic**: Would increase complexity and potentially make things worse
3. **Connection pooling**: Already handled by underlying HTTP stack
4. **Upgrade tokio-util**: Would provide `StreamReader::with_capacity()` but requires testing entire dependency tree
## References
- Issue: "Uploading files of 10GB or 20GB consecutively may cause the upload to freeze"
- Error: `AwsChunkedStreamError: Underlying: error reading a body from connection`
- Library: `tokio_util::io::StreamReader`
- Default buffer: 8KB (tokio_util default)
- New buffer: 1MB (`DEFAULT_READ_BUFFER_SIZE`)
## Conclusion
This fix addresses the root cause of large file upload freezes by using an appropriately sized buffer for stream reading. The 1MB buffer significantly reduces system call overhead, improves throughput, and eliminates timeout issues during consecutive large file uploads.
docs/fix-nosuchkey-regression.md
-141
View File
@@ -1,141 +0,0 @@
# Fix for NoSuchKey Error Response Regression (Issue #901)
## Problem Statement
In RustFS version 1.0.69, a regression was introduced where attempting to download a non-existent or deleted object would return a networking error instead of the expected `NoSuchKey` S3 error:
```
Expected: Aws::S3::Errors::NoSuchKey
Actual: Seahorse::Client::NetworkingError: "http response body truncated, expected 119 bytes, received 0 bytes"
```
## Root Cause Analysis
The issue was caused by the `CompressionLayer` middleware being applied to **all** HTTP responses, including S3 error responses. The sequence of events that led to the bug:
1. Client requests a non-existent object via `GetObject`
2. RustFS determines the object doesn't exist
3. The s3s library generates a `NoSuchKey` error response (XML format, ~119 bytes)
4. HTTP headers are written, including `Content-Length: 119`
5. The `CompressionLayer` attempts to compress the error response body
6. Due to compression buffering or encoding issues with small payloads, the body becomes empty (0 bytes)
7. The client receives `Content-Length: 119` but the actual body is 0 bytes
8. AWS SDK throws a "truncated body" networking error instead of parsing the S3 error
## Solution
The fix implements an intelligent compression predicate (`ShouldCompress`) that excludes certain responses from compression:
### Exclusion Criteria
1. **Error Responses (4xx and 5xx)**: Never compress error responses to ensure error details are preserved and transmitted accurately
2. **Small Responses (< 256 bytes)**: Skip compression for very small responses where compression overhead outweighs benefits
### Implementation Details
```rust
impl Predicate for ShouldCompress {
fn should_compress<B>(&self, response: &Response<B>) -> bool
where
B: http_body::Body,
{
let status = response.status();
// Never compress error responses (4xx and 5xx status codes)
if status.is_client_error() || status.is_server_error() {
debug!("Skipping compression for error response: status={}", status.as_u16());
return false;
}
// Check Content-Length header to avoid compressing very small responses
if let Some(content_length) = response.headers().get(http::header::CONTENT_LENGTH) {
if let Ok(length_str) = content_length.to_str() {
if let Ok(length) = length_str.parse::<u64>() {
if length < 256 {
debug!("Skipping compression for small response: size={} bytes", length);
return false;
}
}
}
}
// Compress successful responses with sufficient size
true
}
}
```
## Benefits
1. **Correctness**: Error responses are now transmitted with accurate Content-Length headers
2. **Compatibility**: AWS SDKs and other S3 clients correctly receive and parse error responses
3. **Performance**: Small responses avoid unnecessary compression overhead
4. **Observability**: Debug logging provides visibility into compression decisions
## Testing
Comprehensive test coverage was added to prevent future regressions:
### Test Cases
1. **`test_get_deleted_object_returns_nosuchkey`**: Verifies that getting a deleted object returns NoSuchKey
2. **`test_head_deleted_object_returns_nosuchkey`**: Verifies HeadObject also returns NoSuchKey for deleted objects
3. **`test_get_nonexistent_object_returns_nosuchkey`**: Tests objects that never existed
4. **`test_multiple_gets_deleted_object`**: Ensures stability across multiple consecutive requests
### Running Tests
```bash
# Run the specific test
cargo test --test get_deleted_object_test -- --ignored
# Or start RustFS server and run tests
./scripts/dev_rustfs.sh
cargo test --test get_deleted_object_test
```
## Impact Assessment
### Affected APIs
- `GetObject`
- `HeadObject`
- Any S3 API that returns 4xx/5xx error responses
### Backward Compatibility
- **No breaking changes**: The fix only affects error response handling
- **Improved compatibility**: Better alignment with S3 specification and AWS SDK expectations
- **No performance degradation**: Small responses were already not compressed by default in most cases
## Deployment Considerations
### Verification Steps
1. Deploy the fix to a staging environment
2. Run the provided Ruby reproduction script to verify the fix
3. Monitor error logs for any compression-related warnings
4. Verify that large successful responses are still being compressed
### Monitoring
Enable debug logging to observe compression decisions:
```bash
RUST_LOG=rustfs::server::http=debug
```
Look for log messages like:
- `Skipping compression for error response: status=404`
- `Skipping compression for small response: size=119 bytes`
## Related Issues
- Issue #901: Regression in exception when downloading non-existent key in alpha 69
- Commit: 86185703836c9584ba14b1b869e1e2c4598126e0 (getobjectlength fix)
## References
- [AWS S3 Error Responses](https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html)
- [tower-http CompressionLayer](https://docs.rs/tower-http/latest/tower_http/compression/index.html)
- [s3s Library](https://github.com/Nugine/s3s)
docs/kms/README.md
-151
View File
@@ -1,151 +0,0 @@
# RustFS Key Management Service
The RustFS Key Management Service (KMS) provides end-to-end key orchestration, envelope encryption, and S3-compatible semantics for encrypted object storage. It sits between the RustFS API surface and the underlying encryption primitives, ensuring that data at rest and in flight remains protected while keeping operational workflows simple.
## Highlights
- **Multiple backends** – plug in Vault for production or use the Local filesystem backend for development and CI.
- **Envelope encryption** – master keys protect data-encryption keys (DEKs); DEKs protect object payloads with AES-256-GCM streaming.
- **S3 compatibility** – works transparently with `SSE-S3`, `SSE-KMS`, and `SSE-C` headers so existing tools continue to function.
- **Dynamic lifecycle** – configure, rotate, or swap backends at runtime by calling the admin REST API; no server restart is required.
- **Caching & resilience** – built-in caching minimises latency, while health probes, retries, and metrics help operators keep track of the service.
## Architecture
```
┌──────────────────────────────────────────────────────────┐
│ RustFS Frontend │
│ (S3 compatible API, IAM, policy engine, bucket logic) │
└──────────────┬───────────────────────────────────────────┘
│
▼
┌──────────────────────────────┐
│ Encryption Service Manager │
│ • Applies admin config │
│ • Controls backend runtime │
│ • Exposes metrics / health │
└──────────────┬──────────────┘
│
┌─────────┴─────────┐
│ │
▼ ▼
┌────────────────┐ ┌────────────────────┐
│ Local Backend │ │ Vault Backend │
│ • File-based │ │ • Transit engine │
│ • Dev / CI │ │ • Production ready │
└────────────────┘ └────────────────────┘
```
### Components at a Glance
| Component | Responsibility |
|------------------------------|-------------------------------------------------------------------------|
| `rustfs::kms::manager` | Owns backend lifecycle, caching, and key orchestration. |
| `rustfs::kms::encryption` | Encrypts/decrypts payloads, issues data keys, validates headers. |
| Admin REST handlers | Accept configuration requests (`configure`, `start`, `status`, etc.). |
| Backends | `local` (filesystem) and `vault` (Transit) implementations. |
## Supported Backends
| Backend | When to use | Key storage | Authentication | Notes |
|---------|-------------|-------------|----------------|-------|
| Local | Development, CI, integration tests | JSON-encoded key blobs on disk | none | Simple, fast to bootstrap, not secure for production. |
| Vault | Production or pre-production | Vault Transit & KV engines | token or AppRole | Supports rotation, audit logging, sealed-state recovery, TLS. |
Refer to [configuration.md](configuration.md) for static configuration details and [dynamic-configuration-guide.md](dynamic-configuration-guide.md) for the runtime workflow.
## Encryption Workflows
RustFS KMS supports the same S3 semantics users expect:
- **SSE-S3** – RustFS manages the data key lifecycle and returns the `x-amz-server-side-encryption` header.
- **SSE-KMS** – RustFS issues per-object data keys bound to the configured KMS backend, exposing the `x-amz-server-side-encryption` header with value `aws:kms`.
- **SSE-C** – Clients provide a 256-bit key and MD5 checksum per request; RustFS uses KMS to encrypt metadata, while encrypted payloads are streamed with the customer key.
Internally, every object follows the envelope-encryption flow below:
1. Determine the logical key-id (default, explicit header, or SSE-C customer key).
2. Ask the configured backend for a DEK or encryption context.
3. Stream-encrypt the payload with AES-256-GCM (1 MiB chunking, authenticated headers).
4. Persist metadata (IV, checksum, key-id) alongside object state.
5. During GET/HEAD, the same process runs in reverse with integrity checks.
## Quick Start
1. **Build RustFS** – `cargo build --release` or run the project-specific build helper.
2. **Prepare credentials** – ensure you have admin access keys; for Vault, export `VAULT_ADDR` and a root or scoped token.
3. **Launch RustFS** – `./target/release/rustfs server` (KMS starts in `NotConfigured`).
4. **Configure the backend**:
```bash
# Local backend (ephemeral testing)
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST -d '{
"backend_type": "local",
"key_dir": "/var/lib/rustfs/kms-keys",
"default_key_id": "rustfs-master"
}' \
http://localhost:9000/rustfs/admin/v3/kms/configure
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST http://localhost:9000/rustfs/admin/v3/kms/start
```
```bash
# Vault backend (production)
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST -d '{
"backend_type": "vault",
"address": "https://vault.example.com:8200",
"auth_method": {
"token": "s.XYZ..."
},
"mount_path": "transit",
"kv_mount": "secret",
"key_path_prefix": "rustfs/kms/keys",
"default_key_id": "rustfs-master"
}' \
https://rustfs.example.com/rustfs/admin/v3/kms/configure
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST https://rustfs.example.com/rustfs/admin/v3/kms/start
```
5. **Verify**:
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
http://localhost:9000/rustfs/admin/v3/kms/status
```
The response should include `"status": "Running"` and the configured backend summary.
## Documentation Map
| Topic | Description |
|-------|-------------|
| [http-api.md](http-api.md) | Formal REST endpoint reference with request/response samples. |
| [dynamic-configuration-guide.md](dynamic-configuration-guide.md) | Gradual rollout, rotation, and failover playbooks. |
| [configuration.md](configuration.md) | Static configuration files, environment variables, Helm/Ansible hints. |
| [api.md](api.md) | Rust crate interfaces (`ConfigureKmsRequest`, `KmsManager`, encryption helpers). |
| [sse-integration.md](sse-integration.md) | Mapping between S3 headers and RustFS behaviour, client examples. |
| [security.md](security.md) | Threat model, access control, TLS, auditing, secrets hygiene. |
| [test_suite_integration.md](test_suite_integration.md) | Running e2e, Vault, and regression test suites. |
| [troubleshooting.md](troubleshooting.md) | Common errors and recovery steps. |
## Terminology
| Term | Definition |
|------|------------|
| **KMS backend** | Implementation that holds master keys (Local filesystem or Vault Transit/ KV). |
| **Master Key** | Root key stored in the backend; encrypts data keys. |
| **Data Encryption Key (DEK)** | Per-object key that encrypts payload chunks. |
| **Envelope Encryption** | Wrapping DEKs with a higher-level key before persisting. |
| **SSE-S3 / SSE-KMS / SSE-C** | Amazon S3-compatible encryption modes supported by RustFS. |
For deeper dives continue with the documents referenced above. EOF
docs/kms/api.md
-169
View File
@@ -1,169 +0,0 @@
# RustFS KMS Developer API
This document targets developers extending RustFS or embedding the KMS primitives directly. The `rustfs-kms` crate exposes building blocks for configuration, backend orchestration, and data-key lifecycle management.
## Crate Overview
Add the crate to your workspace (already included in RustFS):
```toml
[dependencies]
rustfs-kms = { path = "crates/kms" }
```
Key namespaces:
| Module | Purpose |
|--------|---------|
| `rustfs_kms::config` | Typed configuration objects for local/vault backends. |
| `rustfs_kms::manager::KmsManager` | High-level coordinator that proxies operations to a backend. |
| `rustfs_kms::encryption::service::EncryptionService` | Frontend consumed by RustFS S3 handlers. |
| `rustfs_kms::backends` | Backend trait definitions and concrete implementations. |
| `rustfs_kms::types` | Request/response DTOs used by the REST handlers and manager. |
| `rustfs_kms::service_manager` | Async runtime that powers `/kms/configure`, `/kms/start`, etc. |
## Constructing a Configuration
```rust
use rustfs_kms::config::{BackendConfig, KmsBackend, KmsConfig, LocalConfig, VaultConfig, VaultAuthMethod};
let config = KmsConfig {
backend: KmsBackend::Vault,
backend_config: BackendConfig::Vault(VaultConfig {
address: "https://vault.example.com:8200".parse().unwrap(),
auth_method: VaultAuthMethod::Token { token: "s.XYZ".into() },
namespace: None,
mount_path: "transit".into(),
kv_mount: "secret".into(),
key_path_prefix: "rustfs/kms/keys".into(),
tls: None,
}),
default_key_id: Some("rustfs-master".into()),
timeout: std::time::Duration::from_secs(30),
retry_attempts: 3,
enable_cache: true,
cache_config: Default::default(),
};
```
To build configurations from the admin REST payloads, use `api_types::ConfigureKmsRequest`:
```rust
use rustfs_kms::api_types::{ConfigureKmsRequest, ConfigureVaultKmsRequest};
let request = ConfigureKmsRequest::Vault(ConfigureVaultKmsRequest {
address: "https://vault.example.com:8200".into(),
auth_method: VaultAuthMethod::Token { token: "s.XYZ".into() },
namespace: None,
mount_path: Some("transit".into()),
kv_mount: Some("secret".into()),
key_path_prefix: Some("rustfs/kms/keys".into()),
default_key_id: Some("rustfs-master".into()),
skip_tls_verify: Some(false),
timeout_seconds: Some(30),
retry_attempts: Some(5),
enable_cache: Some(true),
max_cached_keys: Some(2048),
cache_ttl_seconds: Some(600),
});
let kms_config: KmsConfig = (&request).into();
```
## Service Manager Lifecycle
The admin layer interacts with a `ServiceManager` singleton that wraps `KmsManager`:
```rust
use rustfs_kms::{init_global_kms_service_manager, get_global_kms_service_manager};
let manager = init_global_kms_service_manager();
manager.configure(config).await?;
manager.start().await?;
let status = manager.get_status().await; // -> KmsServiceStatus::Running
```
`get_global_encryption_service()` returns the `EncryptionService` façade that the S3 request handlers call. The service exposes async methods mirroring AWS KMS semantics:
```rust
use rustfs_kms::types::{CreateKeyRequest, KeyUsage, GenerateDataKeyRequest, KeySpec};
use rustfs_kms::get_global_encryption_service;
let service = get_global_encryption_service().await.expect("service not initialised");
let create = CreateKeyRequest {
key_name: None,
key_usage: KeyUsage::EncryptDecrypt,
description: Some("project-alpha".into()),
tags: Default::default(),
origin: None,
policy: None,
};
let created = service.create_key(create).await?;
let data_key = service
.generate_data_key(GenerateDataKeyRequest {
key_id: created.key_id.clone(),
key_spec: KeySpec::Aes256,
encryption_context: Default::default(),
})
.await?;
```
## Backend Integration Points
To add a custom backend:
1. Implement the `KmsBackend` trait (see `crates/kms/src/backends/mod.rs`).
2. Provide conversions from `ConfigureKmsRequest` into your backend’s config struct.
3. Register the backend in `BackendFactory` and extend the admin handlers to accept the new `backend_type` string.
The trait contract requires implementing methods such as `create_key`, `encrypt`, `decrypt`, `generate_data_key`, `list_keys`, and `health_check`.
```rust
#[async_trait::async_trait]
pub trait KmsBackend: Send + Sync {
async fn create_key(&self, request: CreateKeyRequest) -> Result<CreateKeyResponse>;
async fn encrypt(&self, request: EncryptRequest) -> Result<EncryptResponse>;
async fn decrypt(&self, request: DecryptRequest) -> Result<DecryptResponse>;
async fn generate_data_key(&self, request: GenerateDataKeyRequest) -> Result<GenerateDataKeyResponse>;
async fn describe_key(&self, request: DescribeKeyRequest) -> Result<DescribeKeyResponse>;
async fn list_keys(&self, request: ListKeysRequest) -> Result<ListKeysResponse>;
async fn delete_key(&self, request: DeleteKeyRequest) -> Result<DeleteKeyResponse>;
async fn cancel_key_deletion(&self, request: CancelKeyDeletionRequest) -> Result<CancelKeyDeletionResponse>;
async fn health_check(&self) -> Result<bool>;
}
```
## Encryption Pipeline Helpers
`EncryptionService` contains two methods used by the S3 PUT/GET pipeline:
- `encrypt_stream` (invoked by `PutObject` and multipart uploads) obtains DEKs, encrypts payload chunks with AES-256-GCM, and returns headers.
- `decrypt_stream` resolves metadata, fetches the required DEK or customer key, and streams plaintext back to the client.
Both rely on `ObjectCipher` implementations defined in `crates/kms/src/encryption/ciphers.rs`. When adjusting chunk sizes or cipher suites, update these implementations and the SSE documentation.
## Testing Utilities
- `rustfs_kms::mock` contains in-memory backends used by unit tests.
- The e2e crate (`crates/e2e_test`) exposes helpers such as `LocalKMSTestEnvironment` and `VaultTestEnvironment` for integration testing.
- Run the full suite: `cargo test --workspace --exclude e2e_test` for unit coverage, `cargo test -p e2e_test kms:: -- --nocapture` for end-to-end validation.
## Error Handling Conventions
All public async methods return `rustfs_kms::error::Result<T>`. Errors are categorised as:
| Variant | Meaning |
|---------|---------|
| `KmsError::Configuration` | Invalid or missing backend configuration. |
| `KmsError::Backend` | Underlying backend failure (Vault error, disk I/O, etc.). |
| `KmsError::Crypto` | Integrity or cryptographic failure. |
| `KmsError::Cache` | Cache lookup or eviction failure. |
Map these errors to HTTP responses using the helper macros in `rustfs/src/admin/handlers`.
---
For operational workflows continue with [http-api.md](http-api.md) and [dynamic-configuration-guide.md](dynamic-configuration-guide.md). For encryption semantics, see [sse-integration.md](sse-integration.md).
docs/kms/configuration.md
-125
View File
@@ -1,125 +0,0 @@
# KMS Configuration Guide
This guide describes the configuration surfaces for the RustFS Key Management Service. RustFS can be configured statically at process start or dynamically via the admin REST API. Most operators start with a static bootstrap (CLI flags, configuration file, or environment variables) and then rely on dynamic configuration to rotate keys or swap backends.
## Configuration Sources
| Mechanism | When to use | Notes |
|---------------------|----------------------------------------------------------|-------|
| CLI flags | Local development, ad-hoc testing | `rustfs server --kms-enable --kms-backend vault ...` |
| Environment vars | Container/Helm/Ansible deployments | Prefix variables with `RUSTFS_` (see table below). |
| Static config file | Use your orchestration tooling to render TOML/YAML, then pass the corresponding flags during startup. |
| Dynamic REST API | Post-start updates without restarting (see [dynamic-configuration-guide.md](dynamic-configuration-guide.md)). |
## CLI Flags & Environment Variables
| CLI flag | Env variable | Description |
|-----------------------------|--------------------------------|-------------|
| `--kms-enable` | `RUSTFS_KMS_ENABLE` | Enables KMS at startup. Defaults to `false`. |
| `--kms-backend <local\|vault>` | `RUSTFS_KMS_BACKEND` | Selects the backend implementation. Defaults to `local`. |
| `--kms-key-dir <path>` | `RUSTFS_KMS_KEY_DIR` | Required when `kms-backend=local`; directory that stores wrapped master keys. |
| `--kms-vault-address <url>` | `RUSTFS_KMS_VAULT_ADDRESS` | Vault base URL (e.g. `https://vault.example.com:8200`). |
| `--kms-vault-token <token>` | `RUSTFS_KMS_VAULT_TOKEN` | Token used for Vault authentication. Prefer AppRole or short-lived tokens. |
| `--kms-default-key-id <id>` | `RUSTFS_KMS_DEFAULT_KEY_ID` | Default key used when clients omit `x-amz-server-side-encryption-aws-kms-key-id`. |
> **Tip:** Even when you plan to reconfigure the backend dynamically, setting `--kms-enable` is useful because it instantiates the global manager eagerly and surfaces better error messages when configuration fails.
## Static TOML Example (Local Backend)
```toml
# rustfs.toml
[kms]
enabled = true
backend = "local"
key_dir = "/var/lib/rustfs/kms-keys"
default_key_id = "rustfs-master"
```
Render this file using your favourite template tool and translate it to CLI flags when launching RustFS:
```bash
rustfs server \
--kms-enable \
--kms-backend local \
--kms-key-dir /var/lib/rustfs/kms-keys \
--kms-default-key-id rustfs-master
```
## Static TOML Example (Vault Backend)
```toml
[kms]
enabled = true
backend = "vault"
vault_address = "https://vault.example.com:8200"
# Supply either a token or render AppRole credentials dynamically
vault_token = "s.XYZ..."
default_key_id = "rustfs-master"
```
Ensure that the Vault binary is reachable and the Transit engine is initialised before starting RustFS:
```bash
vault secrets enable transit
vault secrets enable -path=secret kv-v2
vault write transit/keys/rustfs-master type=aes256-gcm96
```
If you prefer AppRole authentication, omit `vault_token` and set the token dynamically via the REST API once RustFS is online (see [dynamic-configuration-guide.md](dynamic-configuration-guide.md)).
## Backend-Specific Options
### Local Backend
| Field | Description |
|------------------|-------------|
| `key_dir` | Directory where wrapped master keys are stored (`*.key` JSON files). Ensure it is backed up securely in persistent deployments. |
| `default_key_id` | Optional; if not provided, SSE-S3 uploads require an explicit header. |
| `file_permissions` (REST only) | Octal permissions applied to generated key files (`0o600` by default). |
| `master_key` (REST only) | Base64-encoded wrapping key used to protect DEKs on disk. Leave unset to generate one automatically. |
During development you can generate a default key manually:
```bash
mkdir -p /tmp/rustfs-keys
openssl rand -hex 32 > /tmp/rustfs-keys/rustfs-master.material
```
The KMS e2e tests also demonstrate programmatic key creation using the `/kms/keys` API.
### Vault Backend
| Field | Description |
|---------------------|-------------|
| `address` | Base URL including scheme. TLS is strongly recommended. |
| `auth_method` | `Token { token: "..." }` or `AppRole { role_id, secret_id }`. Tokens should be renewable or short-lived. |
| `mount_path` | Transit engine mount (default `transit`). |
| `kv_mount` | KV v2 engine used to stash wrapped keys or metadata. |
| `key_path_prefix` | Prefix under the KV mount (e.g. `rustfs/kms/keys`). |
| `namespace` | Vault enterprise namespace (optional). |
| `skip_tls_verify` | Development convenience; avoid using this in production. |
| `default_key_id` | Transit key to use when clients omit `x-amz-server-side-encryption-aws-kms-key-id`. |
## Advanced Runtime Knobs (REST API)
The dynamic API exposes additional fields not available on the CLI:
| Field | Purpose |
|-------|---------|
| `timeout_seconds` | Backend operation timeout (defaults to 30s). |
| `retry_attempts` | Number of retries for transient backend failures (defaults to 3). |
| `enable_cache` | Enables in-memory cache of DEKs and metadata. |
| `max_cached_keys` / `cache_ttl_seconds` | Cache size and TTL limits. |
These options are mostly relevant for large deployments; configure them via the `/kms/configure` REST call once the service is online.
## Bootstrapping Workflow
1. Pick a backend (`local` or `vault`).
2. Ensure the required infrastructure is ready (filesystem permissions or Vault engines).
3. Start RustFS with `--kms-enable` and the minimal bootstrap flags.
4. Call the REST API to refine configuration (timeouts, cache, AppRole, etc.).
5. Verify with `/kms/status` and issue a test `PutObject` using SSE headers.
6. Record the configuration in your infra-as-code tooling for repeatability.
For runtime reconfiguration (rotating keys, swapping from local to Vault) follow the step-by-step guide in [dynamic-configuration-guide.md](dynamic-configuration-guide.md).
docs/kms/dynamic-configuration-guide.md
-155
View File
@@ -1,155 +0,0 @@
# Dynamic KMS Configuration Playbook
RustFS exposes a first-class admin REST API that allows you to configure, start, stop, and reconfigure the KMS subsystem without restarting the server. This document walks through common operational scenarios.
## Prerequisites
- RustFS is running and reachable on the admin endpoint (typically `http(s)://<host>/rustfs/admin/v3`).
- You have admin access and credentials (access key/secret or session token) with the `ServerInfoAdminAction` permission.
- Optional: `awscurl` or another SigV4-aware HTTP client to sign admin requests.
Before starting, confirm the KMS service manager is initialised:
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
http://localhost:9000/rustfs/admin/v3/kms/status
```
The initial response shows `"status": "NotConfigured"`.
## Initial Configuration Flow
1. **Submit the configuration**
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST -d '{
"backend_type": "local",
"key_dir": "/var/lib/rustfs/kms-keys",
"default_key_id": "rustfs-master",
"enable_cache": true,
"cache_ttl_seconds": 900
}' \
http://localhost:9000/rustfs/admin/v3/kms/configure
```
2. **Start the service**
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST http://localhost:9000/rustfs/admin/v3/kms/start
```
3. **Verify**
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
http://localhost:9000/rustfs/admin/v3/kms/status
```
Look for `"status": "Running"` and a backend summary.
## Switching to Vault
To migrate from the local backend to Vault:
1. Prepare Vault:
```bash
vault secrets enable transit
vault secrets enable -path=secret kv-v2
vault write transit/keys/rustfs-master type=aes256-gcm96
```
2. Configure the new backend without stopping service:
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST -d '{
"backend_type": "vault",
"address": "https://vault.example.com:8200",
"auth_method": { "approle": { "role_id": "...", "secret_id": "..." } },
"mount_path": "transit",
"kv_mount": "secret",
"key_path_prefix": "rustfs/kms/keys",
"default_key_id": "rustfs-master",
"retry_attempts": 5,
"timeout_seconds": 60
}' \
http://localhost:9000/rustfs/admin/v3/kms/reconfigure
```
3. Confirm the new backend is active via `/kms/status`.
4. Run test uploads with `SSE-KMS` headers to ensure the new backend is serving requests.
## Rotating the Default Key
1. **Create a new key** using the key management API:
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST -d '{ "KeyUsage": "ENCRYPT_DECRYPT", "Description": "rotation-2024-09" }' \
http://localhost:9000/rustfs/admin/v3/kms/keys
```
2. **Set it as default** via `reconfigure`:
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST -d '{
"backend_type": "vault",
"default_key_id": "rotation-2024-09"
}' \
http://localhost:9000/rustfs/admin/v3/kms/reconfigure
```
Only the fields supplied in the payload are updated; omitted fields keep their previous values.
3. **Validate** by uploading a new object and checking that `x-amz-server-side-encryption-aws-kms-key-id` reports the new key.
## Rolling Cache or Timeout Changes
Caching knobs help tune latency. To adjust them at runtime:
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST -d '{
"enable_cache": true,
"max_cached_keys": 2048,
"cache_ttl_seconds": 600,
"timeout_seconds": 20
}' \
http://localhost:9000/rustfs/admin/v3/kms/reconfigure
```
## Pausing the KMS Service
Stopping the service keeps configuration in place but disables new KMS operations. Existing SSE objects remain accessible only if their metadata allows offline decryption.
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST http://localhost:9000/rustfs/admin/v3/kms/stop
```
Restart later with `/kms/start`.
## Automation Tips
- Wrap REST calls in an idempotent script (see `scripts/` for examples) so you can re-run configuration safely.
- Use `--test-threads=1` when running KMS e2e suites in CI; they spin up real servers and Vault instances.
- In Kubernetes, run the configuration script as an init job that waits for both RustFS and Vault readiness before calling `/kms/configure`.
- Emit events to your observability platform: successful reconfigurations generate structured logs with the backend summary.
## Rollback Strategy
If a new configuration introduces errors:
1. Call `/kms/reconfigure` with the previous payload (keep a snapshot in version control).
2. If the backend is unreachable, call `/kms/stop` to protect data from partial writes.
3. Investigate logs under `rustfs::kms::*` and Vault audit logs.
4. Once the issue is resolved, reapply the desired configuration and restart.
Dynamic configuration makes backend maintenance safe and repeatable—ensure every change is scripted and traceable.
docs/kms/frontend-api-guide-zh.md
-372
View File
@@ -1,372 +0,0 @@
# RustFS KMS Frontend Integration Guide
This document targets frontend engineers who need to integrate with the RustFS Key Management Service (KMS). It provides a complete API reference, usage notes, and example implementations.
## 📋 Contents
1. [Quick Start](#quick-start)
2. [Authentication & Permissions](#authentication--permissions)
3. [API Catalog](#api-catalog)
4. [Service Management APIs](#service-management-apis)
5. [Key Management APIs](#key-management-apis)
6. [Data Encryption APIs](#data-encryption-apis)
7. [Bucket Encryption Configuration APIs](#bucket-encryption-configuration-apis)
8. [Monitoring & Cache APIs](#monitoring--cache-apis)
9. [Common Error Codes](#common-error-codes)
10. [Data Types](#data-types)
11. [Implementation Examples](#implementation-examples)
## Quick Start
### Base configuration
| Setting | Value |
|---------|-------|
| **Base URL** | `http://localhost:9000/rustfs/admin/v3` (local development) |
| **Production URL** | `https://your-rustfs-domain.com/rustfs/admin/v3` |
| **Request format** | `application/json` |
| **Response format** | `application/json` |
| **Authentication** | AWS Signature Version 4 |
| **Encoding** | UTF-8 |
### Common request headers
| Header | Required | Value |
|--------|----------|-------|
| `Content-Type` | ✅ | `application/json` |
| `Authorization` | ✅ | `AWS4-HMAC-SHA256 Credential=...` |
| `X-Amz-Date` | ✅ | ISO 8601 timestamp |
## Authentication & Permissions
### Required IAM permissions
Clients must have `ServerInfoAdminAction` to invoke KMS APIs.
### AWS SigV4 signing
All requests must be signed with SigV4.
- **Access Key ID** – account access key
- **Secret Access Key** – corresponding secret key
- **Region** – `us-east-1`
- **Service** – `execute-api`
## API Catalog
### Service management
| Method | Path | Description | Status |
|--------|------|-------------|--------|
| `POST` | `/kms/configure` | Configure the KMS service | ✅ Available |
| `POST` | `/kms/start` | Start the service | ✅ Available |
| `POST` | `/kms/stop` | Stop the service | ✅ Available |
| `GET` | `/kms/service-status` | Retrieve service status | ✅ Available |
| `POST` | `/kms/reconfigure` | Reconfigure and restart | ✅ Available |
### Key management
| Method | Path | Description | Status |
|--------|------|-------------|--------|
| `POST` | `/kms/keys` | Create a master key | ✅ Available |
| `GET` | `/kms/keys` | List keys | ✅ Available |
| `GET` | `/kms/keys/{key_id}` | Get key metadata | ✅ Available |
| `DELETE` | `/kms/keys/delete` | Schedule key deletion | ✅ Available |
| `POST` | `/kms/keys/cancel-deletion` | Cancel key deletion | ✅ Available |
### Data encryption
| Method | Path | Description | Status |
|--------|------|-------------|--------|
| `POST` | `/kms/generate-data-key` | Generate a data key | ✅ Available |
| `POST` | `/kms/decrypt` | Decrypt a data key | ⚠️ Not implemented |
### Bucket encryption configuration
| Method | Path | Description | Status |
|--------|------|-------------|--------|
| `GET` | `/api/v1/buckets` | List buckets | ✅ Available |
| `GET` | `/api/v1/bucket-encryption/{bucket}` | Get default encryption | ✅ Available |
| `PUT` | `/api/v1/bucket-encryption/{bucket}` | Set default encryption | ✅ Available |
| `DELETE` | `/api/v1/bucket-encryption/{bucket}` | Remove default encryption | ✅ Available |
### Monitoring & cache
| Method | Path | Description | Status |
|--------|------|-------------|--------|
| `GET` | `/kms/config` | Retrieve KMS configuration | ✅ Available |
| `POST` | `/kms/clear-cache` | Clear the KMS cache | ✅ Available |
### Legacy compatibility endpoints
| Method | Path | Description | Status |
|--------|------|-------------|--------|
| `POST` | `/kms/create-key` | Create key (legacy) | ✅ Available |
| `GET` | `/kms/describe-key` | Describe key (legacy) | ✅ Available |
| `GET` | `/kms/list-keys` | List keys (legacy) | ✅ Available |
| `GET` | `/kms/status` | KMS status (legacy) | ✅ Available |
> ✅ **Available** – implemented and usable.
> ⚠️ **Not implemented** – API shape defined but backend missing.
> Prefer the new endpoints; legacy routes exist for backwards compatibility.
## Service Management APIs
### 1. Configure KMS
`POST /kms/configure`
Parameters:
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `backend_type` | string | ✅ | `"local"` or `"vault"` |
| `key_directory` | string | Cond. | Local backend key directory |
| `default_key_id` | string | ✅ | Default master key ID |
| `enable_cache` | boolean | ❌ | Toggle cache (default `true`) |
| `cache_ttl_seconds` | integer | ❌ | Cache TTL (default `600`) |
| `timeout_seconds` | integer | ❌ | Operation timeout (default `30`) |
| `retry_attempts` | integer | ❌ | Retry attempts (default `3`) |
| `address` | string | Cond. | Vault server address |
| `auth_method` | object | Cond. | Vault auth config |
| `mount_path` | string | Cond. | Vault transit mount path |
| `kv_mount` | string | Cond. | Vault KV mount |
| `key_path_prefix` | string | Cond. | Vault key prefix |
Vault `auth_method` fields:
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `token` | string | ✅ | Vault token |
Response
```json
{
"success": boolean,
"message": string,
"config_id": string?
}
```
### 2. Start KMS
`POST /kms/start`
Response fields: `success`, `message`, `status` (`Running`, `Stopped`, `Error`).
### 3. Stop KMS
`POST /kms/stop`
Same response structure as `/kms/start`.
### 4. Service status
`GET /kms/service-status`
Response
```json
{
"status": "Running" | "Stopped" | "NotConfigured" | "Error",
"backend_type": "local" | "vault",
"healthy": boolean,
"config_summary": {
"backend_type": string,
"default_key_id": string,
"timeout_seconds": integer,
"retry_attempts": integer,
"enable_cache": boolean
}
}
```
### 5. Reconfigure
`POST /kms/reconfigure`
Accepts the same payload as `/kms/configure` and restarts the service.
## Key Management APIs
### 1. Create key
`POST /kms/keys`
Parameters:
| Name | Type | Required | Description |
|------|------|----------|-------------|
| `KeyUsage` | string | ✅ | `"ENCRYPT_DECRYPT"` |
| `Description` | string | ❌ | Description (≤256 chars) |
| `Tags` | object | ❌ | Key/value tag map |
Response includes `key_id` and `key_metadata` (enabled, usage, creation date, etc.).
### 2. Key metadata
`GET /kms/keys/{key_id}` returns the `key_metadata` object.
### 3. List keys
`GET /kms/keys?limit=&marker=` with pagination support.
### 4. Schedule deletion
`DELETE /kms/keys/delete`
Parameters: `key_id`, optional `pending_window_in_days` (7–30, default 7).
### 5. Cancel deletion
`POST /kms/keys/cancel-deletion`
Provide `key_id`; response returns updated metadata with `deletion_date = null`.
## Data Encryption APIs
### 1. Generate data key
`POST /kms/generate-data-key`
Parameters: `key_id`, optional `key_spec` (`AES_256` or `AES_128`), optional `encryption_context` map.
Response contains `plaintext_key` (Base64) and `ciphertext_blob` (Base64).
### 2. Decrypt data key
`POST /kms/decrypt`
> ⚠️ Not yet implemented. Expect parameters `ciphertext_blob` and optional `encryption_context`. A future response will expose `key_id` and `plaintext`.
## Bucket Encryption Configuration APIs
RustFS exposes S3-compatible endpoints via the AWS SDK.
### 1. List buckets
Use `ListBuckets` from the AWS SDK.
### 2. Get default encryption
`GetBucketEncryption` returns SSE rules (`SSEAlgorithm`, optional `KMSMasterKeyID`). A 404 indicates no configuration.
### 3. Set default encryption
`PutBucketEncryption` supports SSE-S3 (`AES256`) or SSE-KMS (`aws:kms` + key ID).
### 4. Delete default encryption
`DeleteBucketEncryption` removes the configuration.
Example composable and helper utilities are provided in the original Chinese document; port them as needed.
## Monitoring & Cache APIs
### 1. Get KMS config
`GET /kms/config` returns backend, cache settings, and default key ID.
### 2. Clear cache
`POST /kms/clear-cache` invalidates cached key metadata.
### 3. Legacy status
`GET /kms/status` (legacy) provides cache hit/miss stats.
## Common Error Codes
### HTTP status codes
| Code | Error | Description |
|------|-------|-------------|
| 200 | – | Success |
| 400 | `InvalidRequest` | Bad request or parameters |
| 401 | `AccessDenied` | Authentication failure |
| 403 | `AccessDenied` | Authorization failure |
| 404 | `NotFound` | Resource not found |
| 409 | `Conflict` | Resource conflict |
| 500 | `InternalError` | Server error |
### Error payload
```json
{
"error": {
"code": string,
"message": string,
"request_id": string?
}
}
```
### Specific codes
- `InvalidRequest` – check payload
- `AccessDenied` – verify credentials/permissions
- `KeyNotFound` – key ID incorrect
- `InvalidKeyState` – key disabled or invalid
- `ServiceNotConfigured` – configure KMS first
- `ServiceNotRunning` – start the service
- `BackendError` – backend failure
- `EncryptionFailed` / `DecryptionFailed` – inspect ciphertext/context
## Data Types
### `KeyMetadata`
| Field | Type | Description |
|-------|------|-------------|
| `key_id` | string | UUID |
| `description` | string | Key description |
| `enabled` | boolean | Whether the key is enabled |
| `key_usage` | string | Always `ENCRYPT_DECRYPT` |
| `creation_date` | string | ISO 8601 timestamp |
| `rotation_enabled` | boolean | Rotation status |
| `deletion_date` | string? | Scheduled deletion timestamp |
### `ConfigSummary`
| Field | Type | Description |
|-------|------|-------------|
| `backend_type` | string | `local` or `vault` |
| `default_key_id` | string | Default master key |
| `timeout_seconds` | integer | Operation timeout |
| `retry_attempts` | integer | Retry attempts |
| `enable_cache` | boolean | Cache toggle |
### Enumerations
- `ServiceStatus` – `Running`, `Stopped`, `NotConfigured`, `Error`
- `BackendType` – `local`, `vault`
- `KeyUsage` – `ENCRYPT_DECRYPT`
- `KeySpec` – `AES_256`, `AES_128`
## Implementation Examples
The original guide included extensive code samples covering bucket encryption flows, Vue/React composables, and full application scaffolding. The key patterns are:
1. **Signed requests** – Use AWS SigV4 (via AWS SDK or manual signing) to call `/rustfs/admin/v3` endpoints.
2. **Multipart encryption flow** – Request a data key, encrypt data locally, upload ciphertext, and store the encrypted key blob.
3. **Bucket encryption lifecycle** – Use the S3 SDK to configure default SSE policies, optionally provisioning dedicated KMS keys per bucket.
4. **Health monitoring** – Periodically poll `/kms/status` or `/kms/config` to ensure the service is healthy and cache hit ratios remain acceptable.
## Troubleshooting & Support
If issues arise:
1. Verify the KMS service is healthy via `/kms/service-status`.
2. Confirm Vault or local backend configuration.
3. Inspect server logs for detailed error messages.
4. Run `cargo test -p e2e_test kms:: -- --nocapture` to validate the setup.
5. Ensure your AWS SDK version supports the required S3/KMS calls.
Common questions:
- **Bucket encryption fails with insufficient permissions** – Ensure the IAM policy grants `s3:GetBucketEncryption`, `s3:PutBucketEncryption`, `s3:DeleteBucketEncryption`, and (for SSE-KMS) `kms:DescribeKey`.
- **Unable to select a KMS key** – Confirm the KMS service is running, the key is enabled, and `KeyUsage` is `ENCRYPT_DECRYPT`.
- **Frontend shows incorrect encryption state** – A 404 during `GetBucketEncryption` is normal (no configuration). Allow for network latency before refreshing the status.
---
_Last updated: 2024-09-22_
docs/kms/http-api.md
-248
View File
@@ -1,248 +0,0 @@
# KMS Admin HTTP API Reference
The RustFS KMS admin API is exposed under the admin prefix (`/rustfs/admin/v3`). Requests must be signed with SigV4 credentials that have the `ServerInfoAdminAction` permission. All request and response bodies use JSON, and all endpoints return standard HTTP status codes.
- Base URL examples: `http://localhost:9000/rustfs/admin/v3`, `https://rustfs.example.com/rustfs/admin/v3`.
- Headers: set `Content-Type: application/json` for requests with bodies.
- Authentication: sign with SigV4 (`awscurl`, `aws-signature-v4`, or the official SDKs).
## Service Lifecycle
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/kms/configure` | POST | Apply the initial backend configuration. Does not start the service. |
| `/kms/reconfigure` | POST | Merge a new configuration on top of the existing one. |
| `/kms/start` | POST | Start the configured backend. |
| `/kms/stop` | POST | Stop the backend; configuration is kept. |
| `/kms/status` | GET | Lightweight status summary (`Running`, `Configured`, etc.). |
| `/kms/service-status` | GET | Backward-compatible alias for `/kms/status`. |
| `/kms/config` | GET | Returns the cached configuration summary. |
| `/kms/clear-cache` | POST | Clears in-memory DEK and metadata caches. |
### Configure / Reconfigure
**Request**
```json
{
"backend_type": "vault",
"address": "https://vault.example.com:8200",
"auth_method": { "token": "s.XYZ" },
"mount_path": "transit",
"kv_mount": "secret",
"key_path_prefix": "rustfs/kms/keys",
"default_key_id": "rustfs-master",
"enable_cache": true,
"cache_ttl_seconds": 600,
"timeout_seconds": 30,
"retry_attempts": 3
}
```
**Response**
```json
{
"success": true,
"message": "KMS configured successfully",
"status": "Configured"
}
```
> **Partial updates:** `/kms/reconfigure` updates only the fields present in the payload. Use this to rotate tokens or adjust cache parameters without resubmitting the full configuration.
### Start / Stop
**Start response**
```json
{
"success": true,
"message": "KMS service started successfully",
"status": "Running"
}
```
**Stop response**
```json
{
"success": true,
"message": "KMS service stopped successfully",
"status": "Configured"
}
```
### Status & Config
`GET /kms/status`
```json
{
"status": "Running",
"backend_type": "vault",
"healthy": true,
"config_summary": {
"backend_type": "vault",
"default_key_id": "rustfs-master",
"timeout_seconds": 30,
"retry_attempts": 3,
"enable_cache": true,
"cache_summary": {
"max_keys": 1024,
"ttl_seconds": 600,
"enable_metrics": true
},
"backend_summary": {
"backend_type": "vault",
"address": "https://vault.example.com:8200",
"auth_method_type": "token",
"namespace": null,
"mount_path": "transit",
"kv_mount": "secret",
"key_path_prefix": "rustfs/kms/keys"
}
}
}
```
`GET /kms/config`
```json
{
"backend": "vault",
"cache_enabled": true,
"cache_max_keys": 1024,
"cache_ttl_seconds": 600,
"default_key_id": "rustfs-master"
}
```
`POST /kms/clear-cache` returns HTTP `204` with an empty body when successful.
## Key Management
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/kms/keys` | POST | Create a new master key in the backend. |
| `/kms/keys` | GET | List master keys (paginated). |
| `/kms/keys/{key_id}` | GET | Retrieve metadata for a specific key. |
| `/kms/keys/delete` | DELETE | Schedule key deletion. |
| `/kms/keys/cancel-deletion` | POST | Cancel a pending deletion request. |
### Create Key
**Request**
```json
{
"KeyUsage": "ENCRYPT_DECRYPT",
"Description": "project-alpha",
"Tags": {
"owner": "security",
"env": "prod"
}
}
```
**Response**
```json
{
"key_id": "fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85",
"key_metadata": {
"key_id": "fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85",
"description": "project-alpha",
"enabled": true,
"key_usage": "ENCRYPT_DECRYPT",
"creation_date": "2024-09-18T07:10:42.012345Z",
"rotation_enabled": false
}
}
```
### List Keys
`GET /kms/keys?limit=50&marker=<token>`
```json
{
"keys": [
{ "key_id": "fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85", "description": "project-alpha" }
],
"truncated": false,
"next_marker": null
}
```
### Describe Key
`GET /kms/keys/fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85`
```json
{
"key_metadata": {
"key_id": "fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85",
"description": "project-alpha",
"enabled": true,
"key_usage": "ENCRYPT_DECRYPT",
"creation_date": "2024-09-18T07:10:42.012345Z",
"deletion_date": null
}
}
```
### Delete & Cancel Deletion
**Delete request**
```json
{
"key_id": "fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85",
"pending_window_in_days": 7
}
```
**Cancel deletion**
```json
{
"key_id": "fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85"
}
```
Both endpoints respond with the updated `key_metadata`.
## Data Key Operations
`POST /kms/generate-data-key`
**Request**
```json
{
"key_id": "fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85",
"key_spec": "AES_256",
"encryption_context": {
"bucket": "analytics-data",
"object": "2024/09/18/report.parquet"
}
}
```
**Response**
```json
{
"key_id": "fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85",
"plaintext_key": "sQW6qt0yS7CqD6c8hY7GZg==",
"ciphertext_blob": "gAAAAABlLK..."
}
```
- `plaintext_key` is Base64-encoded and must be zeroised after use.
- `ciphertext_blob` can be stored alongside object metadata for future re-wraps.
## Error Handling
| Code | Meaning | Example payload |
|------|---------|-----------------|
| `400 Bad Request` | Malformed JSON or missing required fields. | `{ "code": "InvalidRequest", "message": "invalid JSON" }` |
| `401 Unauthorized` | Request was not signed or credentials are invalid. | `{ "code": "AccessDenied", "message": "authentication required" }` |
| `403 Forbidden` | Caller lacks admin permissions. | `{ "code": "AccessDenied", "message": "unauthorised" }` |
| `409 Conflict` | Backend already configured in an incompatible way. | `{ "code": "Conflict", "message": "KMS already running" }` |
| `500 Internal Server Error` | Backend failure or transient issue. Logs include details. | `{ "code": "InternalError", "message": "failed to create key: ..." }` |
## Useful Utilities
- [`awscurl`](https://github.com/okigan/awscurl) for quick SigV4 requests.
- The `scripts/` directory contains example shell scripts to configure local and Vault backends automatically.
- The e2e test harness (`cargo test -p e2e_test kms:: -- --nocapture`) demonstrates end-to-end API usage against both backends.
For dynamic workflows and automation strategies, continue with [dynamic-configuration-guide.md](dynamic-configuration-guide.md).
docs/kms/security.md
-65
View File
@@ -1,65 +0,0 @@
# KMS Security Guidelines
This document summarises the security posture of the RustFS KMS subsystem and offers guidance for safe production deployment.
## Threat Model
- Attackers might obtain network access to RustFS or Vault.
- Leaked admin credentials could manipulate KMS configuration.
- Misconfigured SSE-C clients could expose plaintext keys.
- Insider threats may attempt to extract master keys from disk-based storage.
RustFS mitigates these risks via access control, auditability, and best practices outlined below.
## Authentication & Authorisation
- The admin API requires SigV4 credentials with `ServerInfoAdminAction`. Restrict these credentials to trusted automation.
- Do **not** share admin credentials with regular S3 clients. Provision separate IAM users for data-plane traffic.
- When running behind a reverse proxy, ensure the proxy passes through headers required for SigV4 signature validation.
## Network Security
- Enforce TLS for both RustFS and Vault deployments. Set `skip_tls_verify=false` in production.
- Use mTLS or private network peering between RustFS and Vault where possible.
- Restrict Vault transit endpoints using network ACLs or service meshes so only RustFS can reach them.
## Secret Management
- Never store Vault tokens directly in configuration files. Prefer AppRole or short-lived tokens injected at runtime.
- If you must render a token (e.g. in CI), use environment variables with limited scope and rotate them frequently.
- For the local backend, keep the key directory on encrypted disks with tight POSIX permissions (default `0o600`).
## Vault Hardening Checklist
- Enable audit logging (`vault audit enable file file_path=/var/log/vault_audit.log`).
- Create a dedicated policy granting access only to the `transit` and `secret` paths used by RustFS.
- Configure automatic token renewal or rely on `vault agent` to manage token lifetimes.
- Monitor the health endpoint (`/v1/sys/health`) and integrate it into your on-call alerts.
## Caching & Memory Hygiene
- When `enable_cache=true`, DEKs are stored in memory for the configured TTL. Tune `max_cached_keys` and TTL to balance latency versus exposure.
- The encryption service zeroises plaintext keys after use. Avoid logging plaintext keys or contexts in custom code.
- For workloads that require strict FIPS compliance, disable caching and rely on Vault for each request.
## SSE-C Considerations
- Clients are responsible for providing 256-bit keys and MD5 hashes. Reject uploads where the digest does not match.
- Educate clients that SSE-C keys are never stored server side; losing the key means losing access to the object.
- Use HTTPS for all client connections to prevent key disclosure.
## Audit & Monitoring
- Capture structured logs emitted under the `rustfs::kms` target. Each admin call logs request principals and outcomes.
- Export metrics such as cache hit ratio, backend latency, and failure counts to your observability stack.
- Periodically run the e2e Vault suite in a staging environment to verify backup/restore procedures.
## Incident Response
1. Stop the KMS service (`POST /kms/stop`) to freeze new operations.
2. Rotate admin credentials and Vault tokens.
3. Examine audit logs to determine the blast radius.
4. Restore keys from backups or Vault versions if tampering occurred.
5. Reconfigure the backend using trusted credentials and restart the service.
By adhering to these practices, you can deploy RustFS KMS with confidence across regulated or high-security environments.
docs/kms/sse-integration.md
-91
View File
@@ -1,91 +0,0 @@
# Server-Side Encryption Integration
RustFS implements Amazon S3-compatible server-side encryption semantics. This document outlines how each mode maps to KMS operations and how clients should format requests.
## Supported Modes
| Mode | Request Headers | Managed by | Notes |
|------|-----------------|------------|-------|
| `SSE-S3` | `x-amz-server-side-encryption: AES256` | RustFS KMS using the configured default key. | Simplest option; clients do not manage keys. |
| `SSE-KMS` | `x-amz-server-side-encryption: aws:kms`<br>`x-amz-server-side-encryption-aws-kms-key-id: <key-id>` (optional) | RustFS KMS + backend (Vault/Local). | Specify a key-id to override the default. |
| `SSE-C` | `x-amz-server-side-encryption-customer-algorithm: AES256`<br>`x-amz-server-side-encryption-customer-key: <Base64 key>`<br>`x-amz-server-side-encryption-customer-key-MD5: <Base64 MD5>` | Customer provided | RustFS never stores the plaintext key; clients must supply it on every request. |
## Request Examples
### SSE-S3 Upload & Download
```bash
# Upload
aws s3api put-object \
--endpoint-url http://localhost:9000 \
--bucket demo --key obj.txt --body file.txt \
--server-side-encryption AES256
# Download
aws s3api get-object \
--endpoint-url http://localhost:9000 \
--bucket demo --key obj.txt out.txt
```
### SSE-KMS with Explicit Key ID
```bash
aws s3api put-object \
--endpoint-url http://localhost:9000 \
--bucket demo --key report.csv --body report.csv \
--server-side-encryption aws:kms \
--ssekms-key-id rotation-2024-09
```
If `--ssekms-key-id` is omitted, RustFS uses the configured `default_key_id`.
### SSE-C Multipart Upload
SSE-C requires additional care:
1. Generate a 256-bit key and compute its MD5 digest.
2. For multipart uploads, every request (initiate, upload-part, complete, GET) must include the SSE-C headers.
3. Keep part sizes ≥ 5 MiB to avoid falling back to inline storage which complicates key handling.
```bash
KEY="01234567890123456789012345678901"
KEY_B64=$(echo -n "$KEY" | base64)
KEY_MD5=$(echo -n "$KEY" | md5 | awk '{print $1}')
aws s3api create-multipart-upload \
--endpoint-url http://localhost:9000 \
--bucket demo --key video.mp4 \
--server-side-encryption-customer-algorithm AES256 \
--server-side-encryption-customer-key "$KEY_B64" \
--server-side-encryption-customer-key-MD5 "$KEY_MD5"
# Upload all parts with the same trio of headers
```
On download, supply the same headers; otherwise the request fails with `AccessDenied`.
## Response Headers
| Header | SSE-S3 | SSE-KMS | SSE-C |
|--------|--------|---------|-------|
| `x-amz-server-side-encryption` | `AES256` | `aws:kms` | _absent_ |
| `x-amz-server-side-encryption-aws-kms-key-id` | _default key id_ | Provided key id | _absent_ |
| `x-amz-server-side-encryption-customer-algorithm` | _absent_ | _absent_ | `AES256` |
| `x-amz-server-side-encryption-customer-key-MD5` | _absent_ | _absent_ | MD5 of supplied key |
## Error Scenarios
| Scenario | Error | Resolution |
|----------|-------|------------|
| SSE-C key/MD5 mismatch | `AccessDenied` | Regenerate the MD5 digest, ensure Base64 encoding is correct. |
| Missing SSE-C headers on GET | `InvalidRequest` | Provide the same `sse-c` headers used during upload. |
| Invalid key id for SSE-KMS | `NotFound` | Call `GET /kms/keys` to retrieve the valid IDs or create one via the admin API. |
| KMS backend offline | `InternalError` | Check `/kms/status`, restart or reconfigure the backend. |
## Best Practices
- Always use HTTPS endpoints when supplying SSE-C headers.
- Log the key-id used for SSE-KMS uploads to simplify forensic analysis.
- For compliance workloads, disable cache or lower cache TTL via `/kms/reconfigure` so data keys are short-lived.
- Test multipart SSE-C flows regularly; the e2e suite (`test_comprehensive_kms_full_workflow`) covers this scenario.
For the administrative API and configuration specifics, refer to [http-api.md](http-api.md) and [configuration.md](configuration.md).
docs/kms/test_suite_integration.md
-77
View File
@@ -1,77 +0,0 @@
# KMS Test Suite Integration
RustFS ships with an extensive set of automated tests that exercise the KMS stack. This guide explains how to run them locally and in CI.
## Crate Overview
- `crates/kms` – unit tests for configuration, caching, and backend adapters.
- `crates/e2e_test/src/kms` – end-to-end suites for Local and Vault backends, multipart uploads, edge cases, and fault recovery.
- `crates/e2e_test/src/kms/common.rs` – reusable test environments (spins up RustFS, configures Vault, manages buckets).
## Prerequisites
| Requirement | Purpose |
|-------------|---------|
| `vault` binary (>=1.15) | Required for Vault end-to-end tests. Install from Vault releases. |
| `awscurl` (optional) | Debugging helper to hit admin endpoints. |
| `openssl`, `md5` | Used by SSE-C helpers during tests. |
| Local ports | Tests bind ephemeral ports (ensure `127.0.0.1:<random>` is free). |
## Running Unit Tests
```bash
cargo test --workspace --exclude e2e_test
```
This covers the core KMS crate plus supporting libraries.
## Running End-to-End Suites
### All KMS Tests
```bash
NO_PROXY=127.0.0.1,localhost \
HTTP_PROXY= HTTPS_PROXY= \
cargo test -p e2e_test kms:: -- --nocapture --test-threads=1
```
- `--nocapture` streams logs to stdout for troubleshooting.
- `--test-threads=1` ensures serial execution; most tests spawn standalone RustFS and Vault processes.
### Local Backend Only
```bash
cargo test -p e2e_test kms::kms_local_test:: -- --nocapture --test-threads=1
```
### Vault Backend Only
```bash
vault server -dev -dev-root-token-id=dev-root-token &
VAULT_PID=$!
cargo test -p e2e_test kms::kms_vault_test:: -- --nocapture --test-threads=1
kill $VAULT_PID
```
The tests can also start Vault automatically if the binary is found on `PATH`. When running in CI, whitelist the `vault` executable in the sandbox or mark the job as privileged.
## Updating Fixtures
- Adjustment to SSE behaviour or multipart limits often requires touching `crates/e2e_test/src/kms/common.rs`. Keep helpers generic so multiple tests can reuse them.
- When fixing bugs, add targeted coverage in the relevant suite (e.g. `kms_fault_recovery_test.rs`).
- Vault-specific fixtures live in `crates/e2e_test/src/kms/common.rs::VaultTestEnvironment`.
## Debugging Tips
- Use the `CLAUDE DEBUG` log lines (left intentionally verbose) to inspect the RustFS server flow during tests.
- If a test fails with `Operation not permitted`, rerun with sandbox overrides (`cargo test ...` with elevated permissions) as shown above.
- Attach `RUST_LOG=rustfs::kms=debug` to surface detailed backend interactions.
## CI Recommendations
- Split KMS tests into a dedicated job so slower suites (Vault) do not gate unrelated changes.
- Cache the Vault binary and reuse it across runs to minimise setup time.
- Surface logs and `target/debug/e2e_test-*` binaries as artifacts when failures occur.
For API usage examples and configuration reference, consult [http-api.md](http-api.md) and [dynamic-configuration-guide.md](dynamic-configuration-guide.md).
docs/kms/troubleshooting.md
-55
View File
@@ -1,55 +0,0 @@
# KMS Troubleshooting
Use this checklist to diagnose and resolve common KMS-related issues.
## Quick Diagnostics
1. **Check status**
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
http://localhost:9000/rustfs/admin/v3/kms/status
```
2. **Inspect logs** – enable `RUST_LOG=rustfs::kms=debug`.
3. **Verify backend reachability** – for Vault, run `vault status` and test network connectivity.
4. **Run a smoke test** – upload a small object with `--server-side-encryption AES256`.
## Common Issues
| Symptom | Likely Cause | Resolution |
|---------|--------------|------------|
| `status: NotConfigured` | KMS was never configured or configuration failed. | POST `/kms/configure`, then `/kms/start`. Check logs for JSON parsing errors. |
| `healthy: false` | Backend health probe failed; Vault sealed or filesystem inaccessible. | Unseal Vault, confirm permissions on the key directory, re-run `/kms/start`. |
| `InternalError: failed to create key` | Backend rejected the request (e.g. Vault policy). | Review Vault audit logs and ensure the RustFS policy has `transit/keys/*` access. |
| `AccessDenied` when downloading SSE-C objects | Missing/incorrect SSE-C headers. | Provide the same `x-amz-server-side-encryption-customer-*` headers used during upload. |
| Multipart SSE-C download truncated | Parts smaller than 5 MiB stored inline; older builds mishandled them. | Re-upload with ≥5 MiB parts or upgrade to the latest RustFS build. |
| `Operation not permitted` during tests | OS sandbox blocked launching `vault` or `rustfs`. | Re-run tests with elevated permissions (`cargo test ...` with sandbox overrides). |
| `KMS key directory is required for local backend` | Started RustFS with `--kms-backend local` but no `--kms-key-dir`. | Supply the flag or use the dynamic API to set the directory before calling `/kms/start`. |
## Clearing the Cache
If data keys become stale (e.g. after manual rotation in Vault), clear the cache:
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST http://localhost:9000/rustfs/admin/v3/kms/clear-cache
```
## Resetting the Service
1. `POST /kms/stop`
2. `POST /kms/configure` with the known-good payload
3. `POST /kms/start`
4. Verify with `/kms/status`
## Support Data Collection
When opening an issue, capture:
- Output of `/kms/status` and `/kms/config`
- Relevant RustFS logs (`rustfs::kms=*`)
- Vault audit log snippets (if using Vault)
- The SSE headers used by the failing client request
Providing these artifacts drastically speeds up triage.
docs/nosuchkey-fix-comprehensive-analysis.md
-396
View File
@@ -1,396 +0,0 @@
# Comprehensive Analysis: NoSuchKey Error Fix and Related Improvements
## Overview
This document provides a comprehensive analysis of the complete solution for Issue #901 (NoSuchKey regression),
including related improvements from PR #917 that were merged into this branch.
## Problem Statement
**Issue #901**: In RustFS 1.0.69, attempting to download a non-existent or deleted object returns a networking error
instead of the expected `NoSuchKey` S3 error.
**Error Observed**:
```
Class: Seahorse::Client::NetworkingError
Message: "http response body truncated, expected 119 bytes, received 0 bytes"
```
**Expected Behavior**:
```ruby
assert_raises(Aws::S3::Errors::NoSuchKey) do
s3.get_object(bucket: 'some-bucket', key: 'some-key-that-was-deleted')
end
```
## Complete Solution Analysis
### 1. HTTP Compression Layer Fix (Primary Issue)
**File**: `rustfs/src/server/http.rs`
**Root Cause**: The `CompressionLayer` was being applied to all responses, including error responses. When s3s generates
a NoSuchKey error response (~119 bytes XML), the compression layer interferes, causing Content-Length mismatch.
**Solution**: Implemented `ShouldCompress` predicate that intelligently excludes:
- Error responses (4xx/5xx status codes)
- Small responses (< 256 bytes)
**Code Changes**:
```rust
impl Predicate for ShouldCompress {
fn should_compress<B>(&self, response: &Response<B>) -> bool
where
B: http_body::Body,
{
let status = response.status();
// Never compress error responses
if status.is_client_error() || status.is_server_error() {
debug!("Skipping compression for error response: status={}", status.as_u16());
return false;
}
// Skip compression for small responses
if let Some(content_length) = response.headers().get(http::header::CONTENT_LENGTH) {
if let Ok(length_str) = content_length.to_str() {
if let Ok(length) = length_str.parse::<u64>() {
if length < 256 {
debug!("Skipping compression for small response: size={} bytes", length);
return false;
}
}
}
}
true
}
}
```
**Impact**: Ensures error responses are transmitted with accurate Content-Length headers, preventing AWS SDK truncation
errors.
### 2. Content-Length Calculation Fix (Related Issue from PR #917)
**File**: `rustfs/src/storage/ecfs.rs`
**Problem**: The content-length was being calculated incorrectly for certain object types (compressed, encrypted).
**Changes**:
```rust
// Before:
let mut content_length = info.size;
let content_range = if let Some(rs) = & rs {
let total_size = info.get_actual_size().map_err(ApiError::from) ?;
// ...
}
// After:
let mut content_length = info.get_actual_size().map_err(ApiError::from) ?;
let content_range = if let Some(rs) = & rs {
let total_size = content_length;
// ...
}
```
**Rationale**:
- `get_actual_size()` properly handles compressed and encrypted objects
- Returns the actual decompressed size when needed
- Avoids duplicate calls and potential inconsistencies
**Impact**: Ensures Content-Length header accurately reflects the actual response body size.
### 3. Delete Object Metadata Fix (Related Issue from PR #917)
**File**: `crates/filemeta/src/filemeta.rs`
#### Change 1: Version Update Logic (Line 618)
**Problem**: Incorrect version update logic during delete operations.
```rust
// Before:
let mut update_version = fi.mark_deleted;
// After:
let mut update_version = false;
```
**Rationale**:
- The previous logic would always update version when `mark_deleted` was true
- This could cause incorrect version state transitions
- The new logic only updates version in specific replication scenarios
- Prevents spurious version updates during delete marker operations
**Impact**: Ensures correct version management when objects are deleted, which is critical for subsequent GetObject
operations to correctly determine that an object doesn't exist.
#### Change 2: Version ID Filtering (Lines 1711, 1815)
**Problem**: Nil UUIDs were not being filtered when converting to FileInfo.
```rust
// Before:
pub fn into_fileinfo(&self, volume: &str, path: &str, all_parts: bool) -> FileInfo {
// let version_id = self.version_id.filter(|&vid| !vid.is_nil());
// ...
FileInfo {
version_id: self.version_id,
// ...
}
}
// After:
pub fn into_fileinfo(&self, volume: &str, path: &str, all_parts: bool) -> FileInfo {
let version_id = self.version_id.filter(|&vid| !vid.is_nil());
// ...
FileInfo {
version_id,
// ...
}
}
```
**Rationale**:
- Nil UUIDs (all zeros) are not valid version IDs
- Filtering them ensures cleaner semantics
- Aligns with S3 API expectations where no version ID means None, not a nil UUID
**Impact**:
- Improves correctness of version tracking
- Prevents confusion with nil UUIDs in debugging and logging
- Ensures proper behavior in versioned bucket scenarios
## How the Pieces Work Together
### Scenario: GetObject on Deleted Object
1. **Client Request**: `GET /bucket/deleted-object`
2. **Object Lookup**:
- RustFS queries metadata using `FileMeta`
- Version ID filtering ensures nil UUIDs don't interfere (filemeta.rs change)
- Delete state is correctly maintained (filemeta.rs change)
3. **Error Generation**:
- Object not found or marked as deleted
- Returns `ObjectNotFound` error
- Converted to S3 `NoSuchKey` error by s3s library
4. **Response Serialization**:
- s3s serializes error to XML (~119 bytes)
- Sets `Content-Length: 119`
5. **Compression Decision** (NEW):
- `ShouldCompress` predicate evaluates response
- Detects 4xx status code → Skip compression
- Detects small size (119 < 256) → Skip compression
6. **Response Transmission**:
- Full 119-byte XML error body is sent
- Content-Length matches actual body size
- AWS SDK successfully parses NoSuchKey error
### Without the Fix
The problematic flow:
1. Steps 1-4 same as above
2. **Compression Decision** (OLD):
- No filtering, all responses compressed
- Attempts to compress 119-byte error response
3. **Response Transmission**:
- Compression layer buffers/processes response
- Body becomes corrupted or empty (0 bytes)
- Headers already sent with Content-Length: 119
- AWS SDK receives 0 bytes, expects 119 bytes
- Throws "truncated body" networking error
## Testing Strategy
### Comprehensive Test Suite
**File**: `crates/e2e_test/src/reliant/get_deleted_object_test.rs`
Four test cases covering different scenarios:
1. **`test_get_deleted_object_returns_nosuchkey`**
- Upload object → Delete → GetObject
- Verifies NoSuchKey error, not networking error
2. **`test_head_deleted_object_returns_nosuchkey`**
- Tests HeadObject on deleted objects
- Ensures consistency across API methods
3. **`test_get_nonexistent_object_returns_nosuchkey`**
- Tests objects that never existed
- Validates error handling for truly non-existent keys
4. **`test_multiple_gets_deleted_object`**
- 5 consecutive GetObject calls on deleted object
- Ensures stability and no race conditions
### Running Tests
```bash
# Start RustFS server
./scripts/dev_rustfs.sh
# Run specific test
cargo test --test get_deleted_object_test -- test_get_deleted_object_returns_nosuchkey --ignored
# Run all deletion tests
cargo test --test get_deleted_object_test -- --ignored
```
## Performance Impact Analysis
### Compression Skip Rate
**Before Fix**: 0% (all responses compressed)
**After Fix**: ~5-10% (error responses + small responses)
**Calculation**:
- Error responses: ~3-5% of total traffic (typical)
- Small responses: ~2-5% of successful responses
- Total skip rate: ~5-10%
**CPU Impact**:
- Reduced CPU usage from skipped compression
- Estimated savings: 1-2% overall CPU reduction
- No negative impact on latency
### Memory Impact
**Before**: Compression buffers allocated for all responses
**After**: Fewer compression buffers needed
**Savings**: ~5-10% reduction in compression buffer memory
### Network Impact
**Before Fix (Errors)**:
- Attempted compression of 119-byte error responses
- Often resulted in 0-byte transmissions (bug)
**After Fix (Errors)**:
- Direct transmission of 119-byte responses
- No bandwidth savings, but correct behavior
**After Fix (Small Responses)**:
- Skip compression for responses < 256 bytes
- Minimal bandwidth impact (~1-2% increase)
- Better latency for small responses
## Monitoring and Observability
### Key Metrics
1. **Compression Skip Rate**
```
rate(http_compression_skipped_total[5m]) / rate(http_responses_total[5m])
```
2. **Error Response Size**
```
histogram_quantile(0.95, rate(http_error_response_size_bytes[5m]))
```
3. **NoSuchKey Error Rate**
```
rate(s3_errors_total{code="NoSuchKey"}[5m])
```
### Debug Logging
Enable detailed logging:
```bash
RUST_LOG=rustfs::server::http=debug ./target/release/rustfs
```
Look for:
- `Skipping compression for error response: status=404`
- `Skipping compression for small response: size=119 bytes`
## Deployment Checklist
### Pre-Deployment
- [x] Code review completed
- [x] All tests passing
- [x] Clippy checks passed
- [x] Documentation updated
- [ ] Performance testing in staging
- [ ] Security scan (CodeQL)
### Deployment Strategy
1. **Canary (5% traffic)**: Monitor for 24 hours
2. **Partial (25% traffic)**: Monitor for 48 hours
3. **Full rollout (100% traffic)**: Continue monitoring for 1 week
### Rollback Plan
If issues detected:
1. Revert compression predicate changes
2. Keep metadata fixes (they're beneficial regardless)
3. Investigate and reapply compression fix
## Related Issues and PRs
- Issue #901: NoSuchKey error regression
- PR #917: Fix/objectdelete (content-length and delete fixes)
- Commit: 86185703836c9584ba14b1b869e1e2c4598126e0 (getobjectlength)
## Future Improvements
### Short-term
1. Add metrics for nil UUID filtering
2. Add delete marker specific metrics
3. Implement versioned bucket deletion tests
### Long-term
1. Consider gRPC compression strategy
2. Implement adaptive compression thresholds
3. Add response size histograms per S3 operation
## Conclusion
This comprehensive fix addresses the NoSuchKey regression through a multi-layered approach:
1. **HTTP Layer**: Intelligent compression predicate prevents error response corruption
2. **Storage Layer**: Correct content-length calculation for all object types
3. **Metadata Layer**: Proper version management and UUID filtering for deleted objects
The solution is:
- ✅ **Correct**: Fixes the regression completely
- ✅ **Performant**: No negative performance impact, potential improvements
- ✅ **Robust**: Comprehensive test coverage
- ✅ **Maintainable**: Well-documented with clear rationale
- ✅ **Observable**: Debug logging and metrics support
---
**Author**: RustFS Team
**Date**: 2025-11-24
**Version**: 1.0
docs/rustfs-trending.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 41 KiB

docs/security/dos-prevention-body-limits.md
-42
View File
@@ -1,42 +0,0 @@
# DoS Prevention: Request/Response Body Size Limits
## Executive Summary
This document describes the implementation of request and response body size limits in RustFS to prevent Denial of Service (DoS) attacks through unbounded memory allocation. The previous use of `usize::MAX` with `store_all_limited()` posed a critical security risk allowing attackers to exhaust server memory.
## Security Risk Assessment
### Vulnerability: Unbounded Memory Allocation
**Severity**: High
**Impact**: Server memory exhaustion, service unavailability
**Likelihood**: High (easily exploitable)
**Previous Code** (vulnerable):
```rust
let body = input.store_all_limited(usize::MAX).await?;
```
On a 64-bit system, `usize::MAX` is approximately 18 exabytes, effectively unlimited.
## Implemented Limits
| Limit | Size | Use Cases |
|-------|------|-----------|
| `MAX_ADMIN_REQUEST_BODY_SIZE` | 1 MB | User management, policies, tier/KMS/event configs |
| `MAX_IAM_IMPORT_SIZE` | 10 MB | IAM import/export (ZIP archives) |
| `MAX_BUCKET_METADATA_IMPORT_SIZE` | 100 MB | Bucket metadata import |
| `MAX_HEAL_REQUEST_SIZE` | 1 MB | Healing operations |
| `MAX_S3_RESPONSE_SIZE` | 10 MB | S3 client responses from remote services |
## Rationale
- AWS IAM policy limit: 6KB-10KB
- Typical payloads: < 100KB
- 1MB-100MB limits provide generous headroom while preventing DoS
- Based on real-world usage analysis and industry standards
## Files Modified
- 22 files updated across admin handlers and S3 client modules
- 2 new files: `rustfs/src/admin/constants.rs`, `crates/ecstore/src/client/body_limits.rs`
docs/special-characters-README.md
-220
View File
@@ -1,220 +0,0 @@
# Special Characters in Object Paths - Complete Documentation
This directory contains comprehensive documentation for handling special characters (spaces, plus signs, percent signs, etc.) in S3 object paths with RustFS.
## Quick Links
- **For Users**: Start with [Client Guide](./client-special-characters-guide.md)
- **For Developers**: Read [Solution Document](./special-characters-solution.md)
- **For Deep Dive**: See [Technical Analysis](./special-characters-in-path-analysis.md)
## Document Overview
### 1. [Client Guide](./client-special-characters-guide.md)
**Target Audience**: Application developers, DevOps engineers, end users
**Contents**:
- How to upload files with spaces, plus signs, etc.
- Examples for all major S3 SDKs (Python, Go, Node.js, AWS CLI, mc)
- Troubleshooting common issues
- Best practices
- FAQ
**When to Read**: You're experiencing issues with special characters in object names.
### 2. [Solution Document](./special-characters-solution.md)
**Target Audience**: RustFS developers, contributors, maintainers
**Contents**:
- Root cause analysis
- Technical explanation of URL encoding
- Why the backend is correct
- Why issues occur in UI/clients
- Implementation recommendations
- Testing strategy
**When to Read**: You need to understand the technical solution or contribute to the codebase.
### 3. [Technical Analysis](./special-characters-in-path-analysis.md)
**Target Audience**: Senior architects, security reviewers, technical deep-dive readers
**Contents**:
- Comprehensive technical analysis
- URL encoding standards (RFC 3986, AWS S3 API)
- Deep dive into s3s library behavior
- Edge cases and security considerations
- Multiple solution approaches evaluated
- Complete implementation plan
**When to Read**: You need detailed technical understanding or are making architectural decisions.
## TL;DR - The Core Issue
### What Happened
Users reported:
1. **Part A**: UI can navigate to folders with special chars but can't list contents
2. **Part B**: 400 errors when uploading files with `+` in the path
### Root Cause
**Backend (RustFS) is correct** ✅
- The s3s library properly URL-decodes object keys from HTTP requests
- RustFS stores and retrieves objects with special characters correctly
- CLI tools (mc, aws-cli) work perfectly → proves backend is working
**Client/UI is the issue** ❌
- Some clients don't properly URL-encode requests
- UI may not encode prefixes when making LIST requests
- Custom HTTP clients may have encoding bugs
### Solution
1. **For Users**: Use proper S3 SDKs/clients (they handle encoding automatically)
2. **For Developers**: Backend needs no fixes, but added defensive validation and logging
3. **For UI**: UI needs to properly URL-encode all requests (if applicable)
## Quick Examples
### ✅ Works Correctly (Using mc)
```bash
# Upload
mc cp file.txt "myrustfs/bucket/path with spaces/file.txt"
# List
mc ls "myrustfs/bucket/path with spaces/"
# Result: ✅ Success - mc properly encodes the request
```
### ❌ May Not Work (Raw HTTP without encoding)
```bash
# Wrong: Not encoded
curl "http://localhost:9000/bucket/path with spaces/file.txt"
# Result: ❌ May fail - spaces not encoded
```
### ✅ Correct Raw HTTP
```bash
# Correct: Properly encoded
curl "http://localhost:9000/bucket/path%20with%20spaces/file.txt"
# Result: ✅ Success - spaces encoded as %20
```
## URL Encoding Quick Reference
| Character | Display | In URL Path | In Query Param |
|-----------|---------|-------------|----------------|
| Space | ` ` | `%20` | `%20` or `+` |
| Plus | `+` | `%2B` | `%2B` |
| Percent | `%` | `%25` | `%25` |
**Critical**: In URL **paths**, `+` = literal plus (NOT space). Only `%20` = space in paths!
## Implementation Status
### ✅ Completed
1. **Backend Validation**: Added control character validation (rejects null bytes, newlines)
2. **Debug Logging**: Added logging for keys with special characters
3. **Tests**: Created comprehensive e2e test suite
4. **Documentation**:
- Client guide with SDK examples
- Solution document for developers
- Technical analysis for architects
### 📋 Recommended Next Steps
1. **Run Tests**: Execute e2e tests to verify backend behavior
```bash
cargo test --package e2e_test special_chars
```
2. **UI Review** (if applicable): Check if RustFS UI properly encodes requests
3. **User Communication**:
- Update user documentation
- Add troubleshooting to FAQ
- Communicate known UI limitations (if any)
## Related GitHub Issues
- Original Issue: Special Chars in path (#???)
- Referenced PR: #1072 (mentioned in issue comments)
## Support
If you encounter issues with special characters:
1. **First**: Check the [Client Guide](./client-special-characters-guide.md)
2. **Try**: Use mc or AWS CLI to isolate the issue
3. **Enable**: Debug logging: `RUST_LOG=rustfs=debug`
4. **Report**: Create an issue with:
- Client/SDK used
- Exact object name causing issues
- Whether mc works (to isolate backend vs client)
- Debug logs
## Contributing
When contributing related fixes:
1. Read the [Solution Document](./special-characters-solution.md)
2. Understand that backend is working correctly via s3s
3. Focus on UI/client improvements or documentation
4. Add tests to verify behavior
5. Update relevant documentation
## Testing
### Run Special Character Tests
```bash
# All special character tests
cargo test --package e2e_test special_chars -- --nocapture
# Specific test
cargo test --package e2e_test test_object_with_space_in_path -- --nocapture
cargo test --package e2e_test test_object_with_plus_in_path -- --nocapture
cargo test --package e2e_test test_issue_scenario_exact -- --nocapture
```
### Test with Real Clients
```bash
# MinIO client
mc alias set test http://localhost:9000 minioadmin minioadmin
mc cp README.md "test/bucket/test with spaces/README.md"
mc ls "test/bucket/test with spaces/"
# AWS CLI
aws --endpoint-url=http://localhost:9000 s3 cp README.md "s3://bucket/test with spaces/README.md"
aws --endpoint-url=http://localhost:9000 s3 ls "s3://bucket/test with spaces/"
```
## Version History
- **v1.0** (2025-12-09): Initial documentation
- Comprehensive analysis completed
- Root cause identified (UI/client issue)
- Backend validation and logging added
- Client guide created
- E2E tests added
## See Also
- [AWS S3 API Documentation](https://docs.aws.amazon.com/AmazonS3/latest/API/)
- [RFC 3986: URI Generic Syntax](https://tools.ietf.org/html/rfc3986)
- [s3s Library Documentation](https://docs.rs/s3s/)
- [URL Encoding Best Practices](https://developer.mozilla.org/en-US/docs/Glossary/Percent-encoding)
---
**Maintained by**: RustFS Team
**Last Updated**: 2025-12-09
**Status**: Complete - Ready for Use
docs/special-characters-README_ZH.md
-185
View File
@@ -1,185 +0,0 @@
# 对象路径中的特殊字符 - 完整文档
本目录包含关于在 RustFS 中处理 S3 对象路径中特殊字符(空格、加号、百分号等)的完整文档。
## 快速链接
- **用户指南**: [客户端指南](./client-special-characters-guide.md)
- **开发者文档**: [解决方案文档](./special-characters-solution.md)
- **深入分析**: [技术分析](./special-characters-in-path-analysis.md)
## 核心问题说明
### 问题现象
用户报告了两个问题:
1. **问题 A**: UI 可以导航到包含特殊字符的文件夹,但无法列出其中的内容
2. **问题 B**: 上传路径中包含 `+` 号的文件时出现 400 错误
### 根本原因
经过深入调查,包括检查 s3s 库的源代码,我们发现:
**后端 (RustFS) 工作正常** ✅
- s3s 库正确地对 HTTP 请求中的对象键进行 URL 解码
- RustFS 正确存储和检索包含特殊字符的对象
- 命令行工具(mc, aws-cli)完美工作 → 证明后端正确处理特殊字符
**问题出在 UI/客户端层** ❌
- 某些客户端未正确进行 URL 编码
- UI 可能在发出 LIST 请求时未对前缀进行编码
- 自定义 HTTP 客户端可能存在编码错误
### 解决方案
1. **用户**: 使用正规的 S3 SDK/客户端(它们会自动处理编码)
2. **开发者**: 后端无需修复,但添加了防御性验证和日志
3. **UI**: UI 需要正确对所有请求进行 URL 编码(如适用)
## URL 编码快速参考
| 字符 | 显示 | URL 路径中 | 查询参数中 |
|------|------|-----------|-----------|
| 空格 | ` ` | `%20` | `%20` 或 `+` |
| 加号 | `+` | `%2B` | `%2B` |
| 百分号 | `%` | `%25` | `%25` |
**重要**: 在 URL **路径**中,`+` = 字面加号(不是空格)。只有 `%20` = 空格!
## 快速示例
### ✅ 正确使用(使用 mc)
```bash
# 上传
mc cp file.txt "myrustfs/bucket/路径 包含 空格/file.txt"
# 列出
mc ls "myrustfs/bucket/路径 包含 空格/"
# 结果: ✅ 成功 - mc 正确编码了请求
```
### ❌ 可能失败(原始 HTTP 未编码)
```bash
# 错误: 未编码
curl "http://localhost:9000/bucket/路径 包含 空格/file.txt"
# 结果: ❌ 可能失败 - 空格未编码
```
### ✅ 正确的原始 HTTP
```bash
# 正确: 已正确编码
curl "http://localhost:9000/bucket/%E8%B7%AF%E5%BE%84%20%E5%8C%85%E5%90%AB%20%E7%A9%BA%E6%A0%BC/file.txt"
# 结果: ✅ 成功 - 空格编码为 %20
```
## 实施状态
### ✅ 已完成
1. **后端验证**: 添加了控制字符验证(拒绝空字节、换行符)
2. **调试日志**: 为包含特殊字符的键添加了日志记录
3. **测试**: 创建了综合 e2e 测试套件
4. **文档**:
- 包含 SDK 示例的客户端指南
- 开发者解决方案文档
- 架构师技术分析
- 安全摘要
### 📋 建议的后续步骤
1. **运行测试**: 执行 e2e 测试以验证后端行为
```bash
cargo test --package e2e_test special_chars
```
2. **UI 审查**(如适用): 检查 RustFS UI 是否正确编码请求
3. **用户沟通**:
- 更新用户文档
- 在 FAQ 中添加故障排除
- 传达已知的 UI 限制(如有)
## 测试
### 运行特殊字符测试
```bash
# 所有特殊字符测试
cargo test --package e2e_test special_chars -- --nocapture
# 特定测试
cargo test --package e2e_test test_object_with_space_in_path -- --nocapture
cargo test --package e2e_test test_object_with_plus_in_path -- --nocapture
cargo test --package e2e_test test_issue_scenario_exact -- --nocapture
```
### 使用真实客户端测试
```bash
# MinIO 客户端
mc alias set test http://localhost:9000 minioadmin minioadmin
mc cp README.md "test/bucket/测试 包含 空格/README.md"
mc ls "test/bucket/测试 包含 空格/"
# AWS CLI
aws --endpoint-url=http://localhost:9000 s3 cp README.md "s3://bucket/测试 包含 空格/README.md"
aws --endpoint-url=http://localhost:9000 s3 ls "s3://bucket/测试 包含 空格/"
```
## 支持
如果遇到特殊字符问题:
1. **首先**: 查看[客户端指南](./client-special-characters-guide.md)
2. **尝试**: 使用 mc 或 AWS CLI 隔离问题
3. **启用**: 调试日志: `RUST_LOG=rustfs=debug`
4. **报告**: 创建问题,包含:
- 使用的客户端/SDK
- 导致问题的确切对象名称
- mc 是否工作(以隔离后端与客户端)
- 调试日志
## 相关文档
- [客户端指南](./client-special-characters-guide.md) - 用户必读
- [解决方案文档](./special-characters-solution.md) - 开发者指南
- [技术分析](./special-characters-in-path-analysis.md) - 深入分析
- [安全摘要](./SECURITY_SUMMARY_special_chars.md) - 安全审查
## 常见问题
**问: 可以在对象名称中使用空格吗?**
答: 可以,但请使用能自动处理编码的 S3 SDK。
**问: 为什么 `+` 不能用作空格?**
答: 在 URL 路径中,`+` 表示字面加号。只有在查询参数中 `+` 才表示空格。在路径中使用 `%20` 表示空格。
**问: RustFS 支持对象名称中的 Unicode 吗?**
答: 支持,对象名称是 UTF-8 字符串。它们支持任何有效的 UTF-8 字符。
**问: 哪些字符是禁止的?**
答: 控制字符(空字节、换行符、回车符)被拒绝。所有可打印字符都是允许的。
**问: 如何修复"UI 无法列出文件夹"的问题?**
答: 使用 CLI(mc 或 aws-cli)代替。这是 UI 错误,不是后端问题。
## 版本历史
- **v1.0** (2025-12-09): 初始文档
- 完成综合分析
- 确定根本原因(UI/客户端问题)
- 添加后端验证和日志
- 创建客户端指南
- 添加 E2E 测试
---
**维护者**: RustFS 团队
**最后更新**: 2025-12-09
**状态**: 完成 - 可供使用
docs/special-characters-in-path-analysis.md
-536
View File
@@ -1,536 +0,0 @@
# Special Characters in Object Path - Comprehensive Analysis and Solution
## Executive Summary
This document provides an in-depth analysis of the issues with special characters (spaces, plus signs, etc.) in object paths within RustFS, along with a comprehensive solution strategy.
## Problem Statement
### Issue Description
Users encounter problems when working with object paths containing special characters:
**Part A: Spaces in Paths**
```bash
mc cp README.md "local/dummy/a%20f+/b/c/3/README.md"
```
- The UI allows navigation to the folder `%20f+/`
- However, it cannot display the contents within that folder
- CLI tools like `mc ls` correctly show the file exists
**Part B: Plus Signs in Paths**
```
Error: blob (key "/test/data/org_main-org/dashboards/ES+net/LHC+Data+Challenge/firefly-details.json")
api error InvalidArgument: Invalid argument
```
- Files with `+` signs in paths cause 400 (Bad Request) errors
- This affects clients using the Go Cloud Development Kit or similar libraries
## Root Cause Analysis
### URL Encoding in S3 API
According to the AWS S3 API specification:
1. **Object keys in HTTP URLs MUST be URL-encoded**
- Space character → `%20`
- Plus sign → `%2B`
- Literal `+` in URL path → stays as `+` (represents itself, not space)
2. **URL encoding rules for S3 paths:**
- In HTTP URLs: `/bucket/path%20with%20spaces/file%2Bname.txt`
- Decoded key: `path with spaces/file+name.txt`
- Note: `+` in URL path represents a literal `+`, NOT a space
3. **Important distinction:**
- In **query parameters**, `+` represents space (form URL encoding)
- In **URL paths**, `+` represents a literal plus sign
- Space in paths must be encoded as `%20`
### The s3s Library Behavior
The s3s library (version 0.12.0-rc.4) handles HTTP request parsing and URL decoding:
1. **Expected behavior**: s3s should URL-decode the path from HTTP requests before passing keys to our handlers
2. **Current observation**: There appears to be inconsistency or a bug in how keys are decoded
3. **Hypothesis**: The library may not be properly handling certain special characters or edge cases
### Where the Problem Manifests
The issue affects multiple operations:
1. **PUT Object**: Uploading files with special characters in path
2. **GET Object**: Retrieving files with special characters
3. **LIST Objects**: Listing directory contents with special characters in path
4. **DELETE Object**: Deleting files with special characters
### Consistency Issues
The core problem is **inconsistency** in how paths are handled:
- **Storage layer**: May store objects with URL-encoded names
- **Retrieval layer**: May expect decoded names
- **Comparison layer**: Path matching fails when encoding differs
- **List operation**: Returns encoded or decoded names inconsistently
## Technical Analysis
### Current Implementation
#### 1. Storage Layer (ecfs.rs)
```rust
// In put_object
let PutObjectInput {
bucket,
key, // ← This comes from s3s, should be URL-decoded
...
} = input;
store.put_object(&bucket, &key, &mut reader, &opts).await
```
#### 2. List Objects Implementation
```rust
// In list_objects_v2
let object_infos = store
.list_objects_v2(
&bucket,
&prefix, // ← Should this be decoded?
continuation_token,
delimiter.clone(),
max_keys,
fetch_owner.unwrap_or_default(),
start_after,
incl_deleted,
)
.await
```
#### 3. Object Retrieval
The key (object name) needs to match exactly between:
- How it's stored (during PUT)
- How it's queried (during GET/LIST)
- How it's compared (path matching)
### The URL Encoding Problem
Consider this scenario:
1. Client uploads: `PUT /bucket/a%20f+/file.txt`
2. s3s decodes to: `a f+/file.txt` (correct: %20→space, +→plus)
3. We store as: `a f+/file.txt`
4. Client lists: `GET /bucket?prefix=a%20f+/`
5. s3s decodes to: `a f+/`
6. We search for: `a f+/`
7. Should work! ✓
But what if s3s is NOT decoding properly? Or decoding inconsistently?
1. Client uploads: `PUT /bucket/a%20f+/file.txt`
2. s3s passes: `a%20f+/file.txt` (BUG: not decoded!)
3. We store as: `a%20f+/file.txt`
4. Client lists: `GET /bucket?prefix=a%20f+/`
5. s3s passes: `a%20f+/`
6. We search for: `a%20f+/`
7. Works by accident! ✓
But then:
8. Client lists: `GET /bucket?prefix=a+f%2B/` (encoding + as %2B)
9. s3s passes: `a+f%2B/` or `a+f+/` ??
10. We search for that, but stored name was `a%20f+/`
11. Mismatch! ✗
## Solution Strategy
### Approach 1: Trust s3s Library (Recommended)
**Assumption**: s3s library correctly URL-decodes all keys from HTTP requests
**Strategy**:
1. Assume all keys received from s3s are already decoded
2. Store objects with decoded names (UTF-8 strings with literal special chars)
3. Use decoded names for all operations (GET, LIST, DELETE)
4. Never manually URL-encode/decode keys in our handlers
5. Trust s3s to handle HTTP-level encoding/decoding
**Advantages**:
- Follows separation of concerns
- Simpler code
- Relies on well-tested library behavior
**Risks**:
- If s3s has a bug, we're affected
- Need to verify s3s actually does this correctly
### Approach 2: Explicit URL Decoding (Defensive)
**Assumption**: s3s may not decode keys properly, or there are edge cases
**Strategy**:
1. Explicitly URL-decode all keys when received from s3s
2. Use `urlencoding::decode()` on all keys in handlers
3. Store and operate on decoded names
4. Add safety checks and error handling
**Implementation**:
```rust
use urlencoding::decode;
// In put_object
let key = decode(&input.key)
.map_err(|e| s3_error!(InvalidArgument, format!("Invalid URL encoding in key: {}", e)))?
.into_owned();
```
**Advantages**:
- More defensive
- Explicit control
- Handles s3s bugs or limitations
**Risks**:
- Double-decoding if s3s already decodes
- May introduce new bugs
- More complex code
### Approach 3: Hybrid Strategy (Most Robust)
**Strategy**:
1. Add logging to understand what s3s actually passes us
2. Create tests with various special characters
3. Determine if s3s decodes correctly
4. If yes → use Approach 1
5. If no → use Approach 2 with explicit decoding
## Recommended Implementation Plan
### Phase 1: Investigation & Testing
1. **Create comprehensive tests** for special characters:
- Spaces (` ` / `%20`)
- Plus signs (`+` / `%2B`)
- Percent signs (`%` / `%25`)
- Slashes in names (usually not allowed, but test edge cases)
- Unicode characters
- Mixed special characters
2. **Add detailed logging**:
```rust
debug!("Received key from s3s: {:?}", key);
debug!("Key bytes: {:?}", key.as_bytes());
```
3. **Test with real S3 clients**:
- AWS SDK
- MinIO client (mc)
- Go Cloud Development Kit
- boto3 (Python)
### Phase 2: Fix Implementation
Based on Phase 1 findings, implement one of:
#### Option A: s3s handles decoding correctly
- Add tests to verify behavior
- Document the assumption
- Add assertions or validation
#### Option B: s3s has bugs or doesn't decode
- Add explicit URL decoding to all handlers
- Use `urlencoding::decode()` consistently
- Add error handling for invalid encoding
- Document the workaround
### Phase 3: Ensure Consistency
1. **Audit all key usage**:
- PutObject
- GetObject
- DeleteObject
- ListObjects/ListObjectsV2
- CopyObject (source and destination)
- HeadObject
- Multi-part upload operations
2. **Standardize key handling**:
- Create a helper function `normalize_object_key()`
- Use it consistently everywhere
- Add validation
3. **Update path utilities** (`crates/utils/src/path.rs`):
- Ensure path manipulation functions handle special chars
- Add tests for path operations with special characters
### Phase 4: Testing & Validation
1. **Unit tests**:
```rust
#[test]
fn test_object_key_with_space() {
let key = "path with spaces/file.txt";
// test PUT, GET, LIST operations
}
#[test]
fn test_object_key_with_plus() {
let key = "path+with+plus/file+name.txt";
// test all operations
}
#[test]
fn test_object_key_with_mixed_special_chars() {
let key = "complex/path with spaces+plus%percent.txt";
// test all operations
}
```
2. **Integration tests**:
- Test with real S3 clients
- Test mc (MinIO client) scenarios from the issue
- Test Go Cloud Development Kit scenario
- Test AWS SDK compatibility
3. **Regression testing**:
- Ensure existing tests still pass
- Test with normal filenames (no special chars)
- Test with existing data
## Implementation Details
### Key Functions to Modify
1. **rustfs/src/storage/ecfs.rs**:
- `put_object()` - line ~2763
- `get_object()` - find implementation
- `list_objects_v2()` - line ~2564
- `delete_object()` - find implementation
- `copy_object()` - handle source and dest keys
- `head_object()` - find implementation
2. **Helper function to add**:
```rust
/// Normalizes an object key by ensuring it's properly URL-decoded
/// and contains only valid UTF-8 characters.
///
/// This function should be called on all object keys received from
/// the S3 API to ensure consistent handling of special characters.
fn normalize_object_key(key: &str) -> S3Result<String> {
// If s3s already decodes, this is a no-op validation
// If not, this explicitly decodes
match urlencoding::decode(key) {
Ok(decoded) => Ok(decoded.into_owned()),
Err(e) => Err(s3_error!(
InvalidArgument,
format!("Invalid URL encoding in object key: {}", e)
)),
}
}
```
### Testing Strategy
Create a new test module:
```rust
// crates/e2e_test/src/special_chars_test.rs
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn test_put_get_object_with_space() {
// Upload file with space in path
let bucket = "test-bucket";
let key = "folder/file with spaces.txt";
let content = b"test content";
// PUT
put_object(bucket, key, content).await.unwrap();
// GET
let retrieved = get_object(bucket, key).await.unwrap();
assert_eq!(retrieved, content);
// LIST
let objects = list_objects(bucket, "folder/").await.unwrap();
assert!(objects.iter().any(|obj| obj.key == key));
}
#[tokio::test]
async fn test_put_get_object_with_plus() {
let bucket = "test-bucket";
let key = "folder/ES+net/file+name.txt";
// ... similar test
}
#[tokio::test]
async fn test_mc_client_scenario() {
// Reproduce the exact scenario from the issue
let bucket = "dummy";
let key = "a f+/b/c/3/README.md"; // Decoded form
// ... test with mc client or simulate its behavior
}
}
```
## Edge Cases and Considerations
### 1. Directory Markers
RustFS uses `__XLDIR__` suffix for directories:
- Ensure special characters in directory names are handled
- Test: `"folder with spaces/__XLDIR__"`
### 2. Multipart Upload
- Upload ID and part operations must handle special chars
- Test: Multipart upload of object with special char path
### 3. Copy Operations
CopyObject has both source and destination keys:
```rust
// Both need consistent handling
let src_key = input.copy_source.key();
let dest_key = input.key;
```
### 4. Presigned URLs
If RustFS supports presigned URLs, they need special attention:
- URL encoding in presigned URLs
- Signature calculation with encoded vs decoded keys
### 5. Event Notifications
Events include object keys:
- Ensure event payloads have properly encoded/decoded keys
- Test: Webhook target receives correct key format
### 6. Versioning
Version IDs with special character keys:
- Test: List object versions with special char keys
## Security Considerations
### Path Traversal
Ensure URL decoding doesn't enable path traversal:
```rust
// BAD: Don't allow
key = "../../../etc/passwd"
// After decoding:
key = "..%2F..%2F..%2Fetc%2Fpasswd" → "../../../etc/passwd"
// Solution: Validate decoded keys
fn validate_object_key(key: &str) -> S3Result<()> {
if key.contains("..") {
return Err(s3_error!(InvalidArgument, "Invalid object key"));
}
if key.starts_with('/') {
return Err(s3_error!(InvalidArgument, "Object key cannot start with /"));
}
Ok(())
}
```
### Null Bytes
Ensure no null bytes in decoded keys:
```rust
if key.contains('\0') {
return Err(s3_error!(InvalidArgument, "Object key contains null byte"));
}
```
## Testing with Real Clients
### MinIO Client (mc)
```bash
# Test space in path (from issue)
mc cp README.md "local/dummy/a%20f+/b/c/3/README.md"
mc ls "local/dummy/a%20f+/"
mc ls "local/dummy/a%20f+/b/c/3/"
# Test plus in path
mc cp test.txt "local/bucket/ES+net/file+name.txt"
mc ls "local/bucket/ES+net/"
# Test mixed
mc cp data.json "local/bucket/folder%20with%20spaces+plus/file.json"
```
### AWS CLI
```bash
# Upload with space
aws --endpoint-url=http://localhost:9000 s3 cp test.txt "s3://bucket/path with spaces/file.txt"
# List
aws --endpoint-url=http://localhost:9000 s3 ls "s3://bucket/path with spaces/"
```
### Go Cloud Development Kit
```go
import "gocloud.dev/blob"
// Test the exact scenario from the issue
key := "/test/data/org_main-org/dashboards/ES+net/LHC+Data+Challenge/firefly-details.json"
err := bucket.WriteAll(ctx, key, data, nil)
```
## Success Criteria
The fix is successful when:
1. ✅ mc client can upload files with spaces in path
2. ✅ UI correctly displays folders with special characters
3. ✅ UI can list contents of folders with special characters
4. ✅ Files with `+` in path can be uploaded without errors
5. ✅ All S3 operations (PUT, GET, LIST, DELETE) work with special chars
6. ✅ Go Cloud Development Kit can upload files with `+` in path
7. ✅ All existing tests still pass (no regressions)
8. ✅ New tests cover various special character scenarios
## Documentation Updates
After implementation, update:
1. **API Documentation**: Document how special characters are handled
2. **Developer Guide**: Best practices for object naming
3. **Migration Guide**: If storage format changes
4. **FAQ**: Common issues with special characters
5. **This Document**: Final solution and lessons learned
## References
- AWS S3 API Specification: https://docs.aws.amazon.com/AmazonS3/latest/API/
- URL Encoding RFC 3986: https://tools.ietf.org/html/rfc3986
- s3s Library: https://docs.rs/s3s/0.12.0-rc.4/
- urlencoding crate: https://docs.rs/urlencoding/
- Issue #1072 (referenced in comments)
## Conclusion
The issue with special characters in object paths is a critical correctness bug that affects S3 API compatibility. The solution requires:
1. **Understanding** how s3s library handles URL encoding
2. **Implementing** consistent key handling across all operations
3. **Testing** thoroughly with real S3 clients
4. **Validating** that all edge cases are covered
The recommended approach is to start with investigation and testing (Phase 1) to understand the current behavior, then implement the appropriate fix with comprehensive test coverage.
---
**Document Version**: 1.0
**Date**: 2025-12-09
**Author**: RustFS Team
**Status**: Draft - Awaiting Investigation Results
docs/special-characters-solution.md
-311
View File
@@ -1,311 +0,0 @@
# Special Characters in Object Path - Solution Implementation
## Executive Summary
After comprehensive investigation, the root cause analysis reveals:
1. **Backend (rustfs) is handling URL encoding correctly** via the s3s library
2. **The primary issue is likely in the UI/client layer** where URL encoding is not properly handled
3. **Backend enhancements needed** to ensure robustness and better error messages
## Root Cause Analysis
### What s3s Library Does
The s3s library (version 0.12.0-rc.4) **correctly** URL-decodes object keys from HTTP requests:
```rust
// From s3s-0.12.0-rc.4/src/ops/mod.rs, line 261:
let decoded_uri_path = urlencoding::decode(req.uri.path())
.map_err(|_| S3ErrorCode::InvalidURI)?
.into_owned();
```
This means:
- Client sends: `PUT /bucket/a%20f+/file.txt`
- s3s decodes to: `a f+/file.txt`
- Our handler receives: `key = "a f+/file.txt"` (already decoded)
### What Our Backend Does
1. **Storage**: Stores objects with decoded names (e.g., `"a f+/file.txt"`)
2. **Retrieval**: Returns objects with decoded names in LIST responses
3. **Path operations**: Rust's `Path` APIs preserve special characters correctly
### The Real Problems
#### Problem 1: UI Client Issue (Part A)
**Symptom**: UI can navigate TO folder but can't LIST contents
**Diagnosis**:
- User uploads: `PUT /bucket/a%20f+/b/c/3/README.md` ✅ Works
- CLI lists: `GET /bucket?prefix=a%20f+/` ✅ Works (mc properly encodes)
- UI navigates: Shows folder "a f+" ✅ Works
- UI lists folder: `GET /bucket?prefix=a f+/` ❌ Fails (UI doesn't encode!)
**Root Cause**: The UI is not URL-encoding the prefix when making the LIST request. It should send `prefix=a%20f%2B/` but likely sends `prefix=a f+/` which causes issues.
**Evidence**:
- mc (MinIO client) works → proves backend is correct
- UI doesn't work → proves UI encoding is wrong
#### Problem 2: Client Encoding Issue (Part B)
**Symptom**: 400 error with plus signs
**Error Message**: `api error InvalidArgument: Invalid argument`
**Diagnosis**:
The plus sign (`+`) has special meaning in URL query parameters (represents space in form encoding) but not in URL paths. Clients must encode `+` as `%2B` in paths.
**Example**:
- Correct: `/bucket/ES%2Bnet/file.txt` → decoded to `ES+net/file.txt`
- Wrong: `/bucket/ES+net/file.txt` → might be misinterpreted
### URL Encoding Rules
According to RFC 3986 and AWS S3 API:
| Character | In URL Path | In Query Param | Decoded Result |
|-----------|-------------|----------------|----------------|
| Space | `%20` | `%20` or `+` | ` ` (space) |
| Plus | `%2B` | `%2B` | `+` (plus) |
| Percent | `%25` | `%25` | `%` (percent) |
**Critical Note**: In URL **paths** (not query params), `+` represents a literal plus sign, NOT a space. Only `%20` represents space in paths.
## Solution Implementation
### Phase 1: Backend Validation & Logging (Low Risk)
Add defensive validation and better logging to help diagnose issues:
```rust
// In rustfs/src/storage/ecfs.rs
/// Validate that an object key doesn't contain problematic characters
/// that might indicate client-side encoding issues
fn log_potential_encoding_issues(key: &str) {
// Check for unencoded special chars that might indicate problems
if key.contains('\n') || key.contains('\r') || key.contains('\0') {
warn!("Object key contains control characters: {:?}", key);
}
// Log debug info for troubleshooting
debug!("Processing object key: {:?} (bytes: {:?})", key, key.as_bytes());
}
```
**Benefit**: Helps diagnose client-side issues without changing behavior.
### Phase 2: Enhanced Error Messages (Low Risk)
When validation fails, provide helpful error messages:
```rust
// Check for invalid UTF-8 or suspicious patterns
if !key.is_ascii() && !key.is_char_boundary(key.len()) {
return Err(S3Error::with_message(
S3ErrorCode::InvalidArgument,
"Object key contains invalid UTF-8. Ensure keys are properly URL-encoded."
));
}
```
### Phase 3: Documentation (No Risk)
1. **API Documentation**: Document URL encoding requirements
2. **Client Guide**: Explain how to properly encode object keys
3. **Troubleshooting Guide**: Common issues and solutions
### Phase 4: UI Fix (If Applicable)
If RustFS includes a web UI/console:
1. **Ensure UI properly URL-encodes all requests**:
```javascript
// When making requests, encode the key:
const encodedKey = encodeURIComponent(key);
fetch(`/bucket/${encodedKey}`);
// When making LIST requests, encode the prefix:
const encodedPrefix = encodeURIComponent(prefix);
fetch(`/bucket?prefix=${encodedPrefix}`);
```
2. **Decode when displaying**:
```javascript
// When showing keys in UI, decode for display:
const displayKey = decodeURIComponent(key);
```
## Testing Strategy
### Test Cases
Our e2e tests in `crates/e2e_test/src/special_chars_test.rs` cover:
1. ✅ Spaces in paths: `"a f+/b/c/3/README.md"`
2. ✅ Plus signs in paths: `"ES+net/LHC+Data+Challenge/file.json"`
3. ✅ Mixed special characters
4. ✅ PUT, GET, LIST, DELETE operations
5. ✅ Exact scenario from issue
### Running Tests
```bash
# Run special character tests
cargo test --package e2e_test special_chars -- --nocapture
# Run specific test
cargo test --package e2e_test test_issue_scenario_exact -- --nocapture
```
### Expected Results
All tests should **pass** because:
- s3s correctly decodes URL-encoded keys
- Rust Path APIs preserve special characters
- ecstore stores/retrieves keys correctly
- AWS SDK (used in tests) properly encodes keys
If tests fail, it would indicate a bug in our backend implementation.
## Client Guidelines
### For Application Developers
When using RustFS with any S3 client:
1. **Use a proper S3 SDK**: AWS SDK, MinIO SDK, etc. handle encoding automatically
2. **If using raw HTTP**: Manually URL-encode object keys in paths
3. **Remember**:
- Space → `%20` (not `+` in paths!)
- Plus → `%2B`
- Percent → `%25`
### Example: Correct Client Usage
```python
# Python boto3 - handles encoding automatically
import boto3
s3 = boto3.client('s3', endpoint_url='http://localhost:9000')
# These work correctly - boto3 encodes automatically:
s3.put_object(Bucket='test', Key='path with spaces/file.txt', Body=b'data')
s3.put_object(Bucket='test', Key='path+with+plus/file.txt', Body=b'data')
s3.list_objects_v2(Bucket='test', Prefix='path with spaces/')
```
```go
// Go AWS SDK - handles encoding automatically
package main
import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/s3"
)
func main() {
svc := s3.New(session.New())
// These work correctly - SDK encodes automatically:
svc.PutObject(&s3.PutObjectInput{
Bucket: aws.String("test"),
Key: aws.String("path with spaces/file.txt"),
Body: bytes.NewReader([]byte("data")),
})
svc.ListObjectsV2(&s3.ListObjectsV2Input{
Bucket: aws.String("test"),
Prefix: aws.String("path with spaces/"),
})
}
```
```bash
# MinIO mc client - handles encoding automatically
mc cp file.txt "local/bucket/path with spaces/file.txt"
mc ls "local/bucket/path with spaces/"
```
### Example: Manual HTTP Requests
If making raw HTTP requests (not recommended):
```bash
# Correct: URL-encode the path
curl -X PUT "http://localhost:9000/bucket/path%20with%20spaces/file.txt" \
-H "Content-Type: text/plain" \
-d "data"
# Correct: Encode plus as %2B
curl -X PUT "http://localhost:9000/bucket/ES%2Bnet/file.txt" \
-H "Content-Type: text/plain" \
-d "data"
# List with encoded prefix
curl "http://localhost:9000/bucket?prefix=path%20with%20spaces/"
```
## Monitoring and Debugging
### Backend Logs
Enable debug logging to see key processing:
```bash
RUST_LOG=rustfs=debug cargo run
```
Look for log messages showing:
- Received keys
- Validation errors
- Storage operations
### Common Issues
| Symptom | Likely Cause | Solution |
|---------|--------------|----------|
| 400 "InvalidArgument" | Client not encoding properly | Use S3 SDK or manually encode |
| 404 "NoSuchKey" but file exists | Encoding mismatch | Check client encoding |
| UI shows folder but can't list | UI bug - not encoding prefix | Fix UI to encode requests |
| Works with CLI, fails with UI | UI implementation issue | Compare UI requests vs CLI |
## Conclusion
### Backend Status: ✅ Working Correctly
The RustFS backend correctly handles URL-encoded object keys through the s3s library. No backend code changes are required for basic functionality.
### Client/UI Status: ❌ Needs Attention
The issues described appear to be client-side or UI-side problems:
1. **Part A**: UI not properly encoding LIST prefix requests
2. **Part B**: Client not encoding `+` as `%2B` in paths
### Recommendations
1. **Short-term**:
- Add logging and better error messages (Phase 1-2)
- Document client requirements (Phase 3)
- Fix UI if applicable (Phase 4)
2. **Long-term**:
- Add comprehensive e2e tests (already done!)
- Monitor for encoding-related errors
- Educate users on proper S3 client usage
3. **For Users Experiencing Issues**:
- Use proper S3 SDKs (AWS, MinIO, etc.)
- If using custom clients, ensure proper URL encoding
- If using RustFS UI, report UI bugs separately
---
**Document Version**: 1.0
**Date**: 2025-12-09
**Status**: Final - Ready for Implementation
**Next Steps**: Implement Phase 1-3, run tests, update user documentation
docs/tls.md
-63
View File
@@ -1,63 +0,0 @@
# TLS / mTLS configuration
RustFS supports TLS for serving HTTPS and for outbound gRPC connections (MNMD).
It also supports optional client certificate authentication (mTLS) for outbound gRPC:
if a client identity is configured, RustFS will present it; otherwise it will use
server-authenticated TLS only.
## Recommended `tls/` directory layout
Place these files in a directory (default: `./tls`, configurable via `RUSTFS_TLS_PATH`).
```
TLS_DIR/
ca.crt # PEM bundle of CA/root certificates to trust (recommended)
public.crt # optional extra root bundle (PEM)
rustfs_cert.pem # server leaf certificate (PEM) used by the RustFS server
rustfs_key.pem # server private key (PEM) used by the RustFS server
# Optional: outbound mTLS client identity for MNMD
client_cert.pem # client certificate chain (PEM)
client_key.pem # client private key (PEM)
# Optional: server-side mTLS (inbound client certificate verification)
client_ca.crt # PEM bundle of CA certificates to verify client certificates
```
## Environment variables
### Root trust
- `RUSTFS_TLS_PATH` (default: `tls`): TLS directory.
- `RUSTFS_TRUST_SYSTEM_CA` (default: `false`): When `true`, include the platform/system
trust store as additional roots. When `false`, system roots are not used.
- `RUSTFS_TRUST_LEAF_CERT_AS_CA` (default: `false`): Compatibility switch. If `true`,
RustFS will also load `rustfs_cert.pem` into the root store (treating leaf certificates
as trusted roots). Prefer providing `ca.crt` instead.
### Outbound mTLS identity
- `RUSTFS_MTLS_CLIENT_CERT` (default: `${RUSTFS_TLS_PATH}/client_cert.pem`): path to PEM client cert/chain.
- `RUSTFS_MTLS_CLIENT_KEY` (default: `${RUSTFS_TLS_PATH}/client_key.pem`): path to PEM private key.
If both files exist, RustFS enables outbound mTLS. If either is missing, RustFS proceeds
with server-only TLS.
### Server-side mTLS (inbound client certificate verification)
- `RUSTFS_SERVER_MTLS_ENABLE` (default: `false`): When `true`, the RustFS server requires
clients to present valid certificates signed by a trusted CA for authentication.
When enabled, RustFS loads client CA certificates from:
1. `${RUSTFS_TLS_PATH}/client_ca.crt` (preferred)
2. `${RUSTFS_TLS_PATH}/ca.crt` (fallback if `client_ca.crt` does not exist)
**Important**: Server mTLS is disabled by default. When enabled but no valid CA bundle is
found, RustFS will fail to start with a clear error message. This ensures that server mTLS
cannot be accidentally enabled without proper client CA configuration.
## Failure mode for HTTPS without roots
When connecting to an `https://` MNMD address, RustFS requires at least one configured
trusted root. If none are loaded (no `ca.crt`/`public.crt` and system roots disabled),
RustFS fails fast with a clear error message.