mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-29 16:37:07 +00:00
Fix/getobjectlength (#920)
* fix getobject content length resp * Fix regression in exception handling for non-existent key with enhanced compression predicate and metadata improvements (#915) * Initial plan * Fix GetObject regression by excluding error responses from compression The issue was that CompressionLayer was attempting to compress error responses, which could cause Content-Length header mismatches. By excluding 4xx and 5xx responses from compression, we ensure error responses (like NoSuchKey) are sent correctly without body truncation. Co-authored-by: houseme <4829346+houseme@users.noreply.github.com> * Enhance NoSuchKey fix with improved compression predicate and comprehensive tests - Enhanced ShouldCompress predicate with size-based exclusion (< 256 bytes) - Added detailed documentation explaining the compression logic - Added debug logging for better observability - Created comprehensive test suite with 4 test cases: - test_get_deleted_object_returns_nosuchkey - test_head_deleted_object_returns_nosuchkey - test_get_nonexistent_object_returns_nosuchkey - test_multiple_gets_deleted_object - Added extensive inline documentation and comments - Created docs/fix-nosuchkey-regression.md with full analysis Co-authored-by: houseme <4829346+houseme@users.noreply.github.com> * Add compression best practices documentation Added comprehensive guide covering: - Best practices for HTTP response compression - Common pitfalls and solutions - Performance considerations and trade-offs - Testing guidelines and examples - Monitoring and alerting recommendations - Migration guide for existing services Co-authored-by: houseme <4829346+houseme@users.noreply.github.com> * fix * fmt * fmt * Fix/objectdelete (#917) * fix getobject content length resp * fix delete object --------- Co-authored-by: houseme <housemecn@gmail.com> * Add comprehensive analysis of NoSuchKey fix and related improvements Created detailed documentation analyzing: - HTTP compression layer fix (primary issue) - Content-length calculation fix from PR #917 - Delete object metadata fixes from PR #917 - How all components work together - Complete scenario walkthrough - Performance impact analysis - Testing strategy and deployment checklist This ties together all the changes in the PR branch including the merged improvements from PR #917. Co-authored-by: houseme <4829346+houseme@users.noreply.github.com> * replace `once_cell` to `std` * fmt --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: houseme <4829346+houseme@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: weisd <im@weisd.in> * fmt --------- Co-authored-by: weisd <weishidavip@163.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: houseme <4829346+houseme@users.noreply.github.com> Co-authored-by: weisd <im@weisd.in>
This commit is contained in:
houseme
+52
-21
@@ -3,23 +3,29 @@
|
||||
## English Version
|
||||
|
||||
### Overview
|
||||
This implementation provides a comprehensive adaptive buffer sizing optimization system for RustFS, enabling intelligent buffer size selection based on file size and workload characteristics. The complete migration path (Phases 1-4) has been successfully implemented with full backward compatibility.
|
||||
|
||||
This implementation provides a comprehensive adaptive buffer sizing optimization system for RustFS, enabling intelligent
|
||||
buffer size selection based on file size and workload characteristics. The complete migration path (Phases 1-4) has been
|
||||
successfully implemented with full backward compatibility.
|
||||
|
||||
### Key Features
|
||||
|
||||
#### 1. Workload Profile System
|
||||
|
||||
- **6 Predefined Profiles**: GeneralPurpose, AiTraining, DataAnalytics, WebWorkload, IndustrialIoT, SecureStorage
|
||||
- **Custom Configuration Support**: Flexible buffer size configuration with validation
|
||||
- **OS Environment Detection**: Automatic detection of secure Chinese OS environments (Kylin, NeoKylin, UOS, OpenKylin)
|
||||
- **Thread-Safe Global Configuration**: Atomic flags and immutable configuration structures
|
||||
|
||||
#### 2. Intelligent Buffer Sizing
|
||||
|
||||
- **File Size Aware**: Automatically adjusts buffer sizes from 32KB to 4MB based on file size
|
||||
- **Profile-Based Optimization**: Different buffer strategies for different workload types
|
||||
- **Unknown Size Handling**: Special handling for streaming and chunked uploads
|
||||
- **Performance Metrics**: Optional metrics collection via feature flag
|
||||
|
||||
#### 3. Integration Points
|
||||
|
||||
- **put_object**: Optimized buffer sizing for object uploads
|
||||
- **put_object_extract**: Special handling for archive extraction
|
||||
- **upload_part**: Multipart upload optimization
|
||||
@@ -27,23 +33,27 @@ This implementation provides a comprehensive adaptive buffer sizing optimization
|
||||
### Implementation Phases
|
||||
|
||||
#### Phase 1: Infrastructure (Completed)
|
||||
|
||||
- Created workload profile module (`rustfs/src/config/workload_profiles.rs`)
|
||||
- Implemented core data structures (WorkloadProfile, BufferConfig, RustFSBufferConfig)
|
||||
- Added configuration validation and testing framework
|
||||
|
||||
#### Phase 2: Opt-In Usage (Completed)
|
||||
|
||||
- Added global configuration management
|
||||
- Implemented `RUSTFS_BUFFER_PROFILE_ENABLE` and `RUSTFS_BUFFER_PROFILE` configuration
|
||||
- Integrated buffer sizing into core upload functions
|
||||
- Maintained backward compatibility with legacy behavior
|
||||
|
||||
#### Phase 3: Default Enablement (Completed)
|
||||
|
||||
- Changed default to enabled with GeneralPurpose profile
|
||||
- Replaced opt-in with opt-out mechanism (`--buffer-profile-disable`)
|
||||
- Created comprehensive migration guide (MIGRATION_PHASE3.md)
|
||||
- Ensured zero-impact migration for existing deployments
|
||||
|
||||
#### Phase 4: Full Integration (Completed)
|
||||
|
||||
- Unified profile-only implementation
|
||||
- Removed hardcoded buffer values
|
||||
- Added optional performance metrics collection
|
||||
@@ -53,22 +63,24 @@ This implementation provides a comprehensive adaptive buffer sizing optimization
|
||||
|
||||
#### Buffer Size Ranges by Profile
|
||||
|
||||
| Profile | Min Buffer | Max Buffer | Optimal For |
|
||||
|---------|-----------|-----------|-------------|
|
||||
| GeneralPurpose | 64KB | 1MB | Mixed workloads |
|
||||
| AiTraining | 512KB | 4MB | Large files, sequential I/O |
|
||||
| DataAnalytics | 128KB | 2MB | Mixed read-write patterns |
|
||||
| WebWorkload | 32KB | 256KB | Small files, high concurrency |
|
||||
| IndustrialIoT | 64KB | 512KB | Real-time streaming |
|
||||
| SecureStorage | 32KB | 256KB | Compliance environments |
|
||||
| Profile | Min Buffer | Max Buffer | Optimal For |
|
||||
|----------------|------------|------------|-------------------------------|
|
||||
| GeneralPurpose | 64KB | 1MB | Mixed workloads |
|
||||
| AiTraining | 512KB | 4MB | Large files, sequential I/O |
|
||||
| DataAnalytics | 128KB | 2MB | Mixed read-write patterns |
|
||||
| WebWorkload | 32KB | 256KB | Small files, high concurrency |
|
||||
| IndustrialIoT | 64KB | 512KB | Real-time streaming |
|
||||
| SecureStorage | 32KB | 256KB | Compliance environments |
|
||||
|
||||
#### Configuration Options
|
||||
|
||||
**Environment Variables:**
|
||||
|
||||
- `RUSTFS_BUFFER_PROFILE`: Select workload profile (default: GeneralPurpose)
|
||||
- `RUSTFS_BUFFER_PROFILE_DISABLE`: Disable profiling (opt-out)
|
||||
|
||||
**Command-Line Flags:**
|
||||
|
||||
- `--buffer-profile <PROFILE>`: Set workload profile
|
||||
- `--buffer-profile-disable`: Disable workload profiling
|
||||
|
||||
@@ -111,23 +123,27 @@ docs/README.md | 3 +
|
||||
### Usage Examples
|
||||
|
||||
**Default (Recommended):**
|
||||
|
||||
```bash
|
||||
./rustfs /data
|
||||
```
|
||||
|
||||
**Custom Profile:**
|
||||
|
||||
```bash
|
||||
export RUSTFS_BUFFER_PROFILE=AiTraining
|
||||
./rustfs /data
|
||||
```
|
||||
|
||||
**Opt-Out:**
|
||||
|
||||
```bash
|
||||
export RUSTFS_BUFFER_PROFILE_DISABLE=true
|
||||
./rustfs /data
|
||||
```
|
||||
|
||||
**With Metrics:**
|
||||
|
||||
```bash
|
||||
cargo build --features metrics --release
|
||||
./target/release/rustfs /data
|
||||
@@ -138,23 +154,28 @@ cargo build --features metrics --release
|
||||
## 中文版本
|
||||
|
||||
### 概述
|
||||
本实现为 RustFS 提供了全面的自适应缓冲区大小优化系统，能够根据文件大小和工作负载特性智能选择缓冲区大小。完整的迁移路径（阶段 1-4）已成功实现，完全向后兼容。
|
||||
|
||||
本实现为 RustFS 提供了全面的自适应缓冲区大小优化系统，能够根据文件大小和工作负载特性智能选择缓冲区大小。完整的迁移路径（阶段
|
||||
1-4）已成功实现，完全向后兼容。
|
||||
|
||||
### 核心功能
|
||||
|
||||
#### 1. 工作负载配置文件系统
|
||||
- **6 种预定义配置文件**：通用、AI训练、数据分析、Web工作负载、工业物联网、安全存储
|
||||
|
||||
- **6 种预定义配置文件**：通用、AI 训练、数据分析、Web 工作负载、工业物联网、安全存储
|
||||
- **自定义配置支持**：灵活的缓冲区大小配置和验证
|
||||
- **操作系统环境检测**：自动检测中国安全操作系统环境（麒麟、中标麒麟、统信、开放麒麟）
|
||||
- **线程安全的全局配置**：原子标志和不可变配置结构
|
||||
|
||||
#### 2. 智能缓冲区大小调整
|
||||
|
||||
- **文件大小感知**：根据文件大小自动调整 32KB 到 4MB 的缓冲区
|
||||
- **基于配置文件的优化**：不同工作负载类型的不同缓冲区策略
|
||||
- **未知大小处理**：流式传输和分块上传的特殊处理
|
||||
- **性能指标**：通过功能标志可选的指标收集
|
||||
|
||||
#### 3. 集成点
|
||||
|
||||
- **put_object**：对象上传的优化缓冲区大小
|
||||
- **put_object_extract**：存档提取的特殊处理
|
||||
- **upload_part**：多部分上传优化
|
||||
@@ -162,23 +183,27 @@ cargo build --features metrics --release
|
||||
### 实现阶段
|
||||
|
||||
#### 阶段 1：基础设施（已完成）
|
||||
|
||||
- 创建工作负载配置文件模块（`rustfs/src/config/workload_profiles.rs`）
|
||||
- 实现核心数据结构（WorkloadProfile、BufferConfig、RustFSBufferConfig）
|
||||
- 添加配置验证和测试框架
|
||||
|
||||
#### 阶段 2：选择性启用（已完成）
|
||||
|
||||
- 添加全局配置管理
|
||||
- 实现 `RUSTFS_BUFFER_PROFILE_ENABLE` 和 `RUSTFS_BUFFER_PROFILE` 配置
|
||||
- 将缓冲区大小调整集成到核心上传函数中
|
||||
- 保持与旧版行为的向后兼容性
|
||||
|
||||
#### 阶段 3：默认启用（已完成）
|
||||
|
||||
- 将默认值更改为使用通用配置文件启用
|
||||
- 将选择性启用替换为选择性退出机制（`--buffer-profile-disable`）
|
||||
- 创建全面的迁移指南（MIGRATION_PHASE3.md）
|
||||
- 确保现有部署的零影响迁移
|
||||
|
||||
#### 阶段 4：完全集成（已完成）
|
||||
|
||||
- 统一的纯配置文件实现
|
||||
- 移除硬编码的缓冲区值
|
||||
- 添加可选的性能指标收集
|
||||
@@ -188,30 +213,32 @@ cargo build --features metrics --release
|
||||
|
||||
#### 按配置文件划分的缓冲区大小范围
|
||||
|
||||
| 配置文件 | 最小缓冲 | 最大缓冲 | 最适合 |
|
||||
|---------|---------|---------|--------|
|
||||
| 通用 | 64KB | 1MB | 混合工作负载 |
|
||||
| AI训练 | 512KB | 4MB | 大文件、顺序I/O |
|
||||
| 数据分析 | 128KB | 2MB | 混合读写模式 |
|
||||
| Web工作负载 | 32KB | 256KB | 小文件、高并发 |
|
||||
| 工业物联网 | 64KB | 512KB | 实时流式传输 |
|
||||
| 安全存储 | 32KB | 256KB | 合规环境 |
|
||||
| 配置文件 | 最小缓冲 | 最大缓冲 | 最适合 |
|
||||
|----------|-------|-------|------------|
|
||||
| 通用 | 64KB | 1MB | 混合工作负载 |
|
||||
| AI 训练 | 512KB | 4MB | 大文件、顺序 I/O |
|
||||
| 数据分析 | 128KB | 2MB | 混合读写模式 |
|
||||
| Web 工作负载 | 32KB | 256KB | 小文件、高并发 |
|
||||
| 工业物联网 | 64KB | 512KB | 实时流式传输 |
|
||||
| 安全存储 | 32KB | 256KB | 合规环境 |
|
||||
|
||||
#### 配置选项
|
||||
|
||||
**环境变量：**
|
||||
|
||||
- `RUSTFS_BUFFER_PROFILE`：选择工作负载配置文件（默认：通用）
|
||||
- `RUSTFS_BUFFER_PROFILE_DISABLE`：禁用配置文件（选择性退出）
|
||||
|
||||
**命令行标志：**
|
||||
|
||||
- `--buffer-profile <配置文件>`：设置工作负载配置文件
|
||||
- `--buffer-profile-disable`：禁用工作负载配置文件
|
||||
|
||||
### 性能影响
|
||||
|
||||
- **默认（通用）**：与原始实现性能相同
|
||||
- **AI训练**：大文件（>500MB）吞吐量提升最多 4倍
|
||||
- **Web工作负载**：小文件的内存使用更低、并发性更好
|
||||
- **AI 训练**：大文件（>500MB）吞吐量提升最多 4 倍
|
||||
- **Web 工作负载**：小文件的内存使用更低、并发性更好
|
||||
- **指标收集**：启用时 CPU 开销 < 1%
|
||||
|
||||
### 代码质量
|
||||
@@ -246,23 +273,27 @@ docs/README.md | 3 +
|
||||
### 使用示例
|
||||
|
||||
**默认（推荐）：**
|
||||
|
||||
```bash
|
||||
./rustfs /data
|
||||
```
|
||||
|
||||
**自定义配置文件：**
|
||||
|
||||
```bash
|
||||
export RUSTFS_BUFFER_PROFILE=AiTraining
|
||||
./rustfs /data
|
||||
```
|
||||
|
||||
**选择性退出：**
|
||||
|
||||
```bash
|
||||
export RUSTFS_BUFFER_PROFILE_DISABLE=true
|
||||
./rustfs /data
|
||||
```
|
||||
|
||||
**启用指标：**
|
||||
|
||||
```bash
|
||||
cargo build --features metrics --release
|
||||
./target/release/rustfs /data
|
||||
|
||||
Reference in New Issue
Block a user