gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-22 04:16:38 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' of https://github.com/rustfs/s3-rustfs into feature/ilm

Browse Source 
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/utils/Cargo.toml
#	crates/utils/src/net.rs
#	ecstore/Cargo.toml
#	ecstore/src/set_disk.rs
#	rustfs/src/storage/ecfs.rs
This commit is contained in:
likewu
2025-06-23 16:42:18 +08:00
parent 2c176dd864 dc93398768
commit 81f0c9763f
225 changed files with 14913 additions and 6941 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ecstore/BENCHMARK.md
-270
View File
@@ -1,270 +0,0 @@
# Reed-Solomon Erasure Coding Performance Benchmark
This directory contains a comprehensive benchmark suite for comparing the performance of different Reed-Solomon implementations.
## 📊 Test Overview
### Supported Implementation Modes
#### 🏛️ Pure Erasure Mode (Default, Recommended)
- **Stable and Reliable**: Uses mature reed-solomon-erasure implementation
- **Wide Compatibility**: Supports arbitrary shard sizes
- **Memory Efficient**: Optimized memory usage patterns
- **Predictable**: Performance insensitive to shard size
- **Use Case**: Default choice for production environments, suitable for most application scenarios
#### 🎯 SIMD Mode (`reed-solomon-simd` feature)
- **High Performance Optimization**: Uses SIMD instruction sets for high-performance encoding/decoding
- **Performance Oriented**: Focuses on maximizing processing performance
- **Target Scenarios**: High-performance scenarios for large data processing
- **Use Case**: Scenarios requiring maximum performance, suitable for handling large amounts of data
### Test Dimensions
- **Encoding Performance** - Speed of encoding data into erasure code shards
- **Decoding Performance** - Speed of recovering original data from erasure code shards
- **Shard Size Sensitivity** - Impact of different shard sizes on performance
- **Erasure Code Configuration** - Performance impact of different data/parity shard ratios
- **SIMD Mode Performance** - Performance characteristics of SIMD optimization
- **Concurrency Performance** - Performance in multi-threaded environments
- **Memory Efficiency** - Memory usage patterns and efficiency
- **Error Recovery Capability** - Recovery performance under different numbers of lost shards
## 🚀 Quick Start
### Run Quick Tests
```bash
# Run quick performance comparison tests (default pure Erasure mode)
./run_benchmarks.sh quick
```
### Run Complete Comparison Tests
```bash
# Run detailed implementation comparison tests
./run_benchmarks.sh comparison
```
### Run Specific Mode Tests
```bash
# Test default pure erasure mode (recommended)
./run_benchmarks.sh erasure
# Test SIMD mode
./run_benchmarks.sh simd
```
## 📈 Manual Benchmark Execution
### Basic Usage
```bash
# Run all benchmarks (default pure erasure mode)
cargo bench
# Run specific benchmark files
cargo bench --bench erasure_benchmark
cargo bench --bench comparison_benchmark
```
### Compare Different Implementation Modes
```bash
# Test default pure erasure mode
cargo bench --bench comparison_benchmark
# Test SIMD mode
cargo bench --bench comparison_benchmark \
--features reed-solomon-simd
# Save baseline for comparison
cargo bench --bench comparison_benchmark \
-- --save-baseline erasure_baseline
# Compare SIMD mode performance with baseline
cargo bench --bench comparison_benchmark \
--features reed-solomon-simd \
-- --baseline erasure_baseline
```
### Filter Specific Tests
```bash
# Run only encoding tests
cargo bench encode
# Run only decoding tests
cargo bench decode
# Run tests for specific data sizes
cargo bench 1MB
# Run tests for specific configurations
cargo bench "4+2"
```
## 📊 View Results
### HTML Reports
Benchmark results automatically generate HTML reports:
```bash
# Start local server to view reports
cd target/criterion
python3 -m http.server 8080
# Access in browser
open http://localhost:8080/report/index.html
```
### Command Line Output
Benchmarks display in terminal:
- Operations per second (ops/sec)
- Throughput (MB/s)
- Latency statistics (mean, standard deviation, percentiles)
- Performance trend changes
## 🔧 Test Configuration
### Data Sizes
- **Small Data**: 1KB, 8KB - Test small file scenarios
- **Medium Data**: 64KB, 256KB - Test common file sizes
- **Large Data**: 1MB, 4MB - Test large file processing and SIMD optimization
- **Very Large Data**: 16MB+ - Test high throughput scenarios
### Erasure Code Configurations
- **(4,2)** - Common configuration, 33% redundancy
- **(6,3)** - 50% redundancy, balanced performance and reliability
- **(8,4)** - 50% redundancy, more parallelism
- **(10,5)**, **(12,6)** - High parallelism configurations
### Shard Sizes
Test different shard sizes from 32 bytes to 8KB, with special focus on:
- **Memory Alignment**: 64, 128, 256 bytes - Impact of memory alignment on performance
- **Cache Friendly**: 1KB, 2KB, 4KB - CPU cache-friendly sizes
## 📝 Interpreting Test Results
### Performance Metrics
1. **Throughput**
- Unit: MB/s or GB/s
- Measures data processing speed
- Higher is better
2. **Latency**
- Unit: microseconds (μs) or milliseconds (ms)
- Measures single operation time
- Lower is better
3. **CPU Efficiency**
- Bytes processed per CPU cycle
- Reflects algorithm efficiency
### Expected Results
**Pure Erasure Mode (Default)**:
- Stable performance, insensitive to shard size
- Best compatibility, supports all configurations
- Stable and predictable memory usage
**SIMD Mode (`reed-solomon-simd` feature)**:
- High-performance SIMD optimized implementation
- Suitable for large data processing scenarios
- Focuses on maximizing performance
**Shard Size Sensitivity**:
- SIMD mode may be more sensitive to shard sizes
- Pure Erasure mode relatively insensitive to shard size
**Memory Usage**:
- SIMD mode may have specific memory alignment requirements
- Pure Erasure mode has more stable memory usage
## 🛠️ Custom Testing
### Adding New Test Scenarios
Edit `benches/erasure_benchmark.rs` or `benches/comparison_benchmark.rs`:
```rust
// Add new test configuration
let configs = vec![
// Your custom configuration
BenchConfig::new(10, 4, 2048 * 1024, 2048 * 1024), // 10+4, 2MB
];
```
### Adjust Test Parameters
```rust
// Modify sampling and test time
group.sample_size(20); // Sample count
group.measurement_time(Duration::from_secs(10)); // Test duration
```
## 🐛 Troubleshooting
### Common Issues
1. **Compilation Errors**: Ensure correct dependencies are installed
```bash
cargo update
cargo build --all-features
```
2. **Performance Anomalies**: Check if running in correct mode
```bash
# Check current configuration
cargo bench --bench comparison_benchmark -- --help
```
3. **Tests Taking Too Long**: Adjust test parameters
```bash
# Use shorter test duration
cargo bench -- --quick
```
### Performance Analysis
Use tools like `perf` for detailed performance analysis:
```bash
# Analyze CPU usage
cargo bench --bench comparison_benchmark &
perf record -p $(pgrep -f comparison_benchmark)
perf report
```
## 🤝 Contributing
Welcome to submit new benchmark scenarios or optimization suggestions:
1. Fork the project
2. Create feature branch: `git checkout -b feature/new-benchmark`
3. Add test cases
4. Commit changes: `git commit -m 'Add new benchmark for XYZ'`
5. Push to branch: `git push origin feature/new-benchmark`
6. Create Pull Request
## 📚 References
- [reed-solomon-erasure crate](https://crates.io/crates/reed-solomon-erasure)
- [reed-solomon-simd crate](https://crates.io/crates/reed-solomon-simd)
- [Criterion.rs benchmark framework](https://bheisler.github.io/criterion.rs/book/)
- [Reed-Solomon error correction principles](https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction)
---
💡 **Tips**:
- Recommend using the default pure Erasure mode, which provides stable performance across various scenarios
- Consider SIMD mode for high-performance requirements
- Benchmark results may vary based on hardware, operating system, and compiler versions
- Suggest running tests in target deployment environment for most accurate performance data
ecstore/BENCHMARK_ZH.md
-270
View File
@@ -1,270 +0,0 @@
# Reed-Solomon 纠删码性能基准测试
本目录包含了比较不同 Reed-Solomon 实现性能的综合基准测试套件。
## 📊 测试概述
### 支持的实现模式
#### 🏛️ 纯 Erasure 模式（默认，推荐）
- **稳定可靠**: 使用成熟的 reed-solomon-erasure 实现
- **广泛兼容**: 支持任意分片大小
- **内存高效**: 优化的内存使用模式
- **可预测性**: 性能对分片大小不敏感
- **使用场景**: 生产环境默认选择，适合大多数应用场景
#### 🎯 SIMD模式（`reed-solomon-simd` feature）
- **高性能优化**: 使用SIMD指令集进行高性能编码解码
- **性能导向**: 专注于最大化处理性能
- **适用场景**: 大数据量处理的高性能场景
- **使用场景**: 需要最大化性能的场景，适合处理大量数据
### 测试维度
- **编码性能** - 数据编码成纠删码分片的速度
- **解码性能** - 从纠删码分片恢复原始数据的速度
- **分片大小敏感性** - 不同分片大小对性能的影响
- **纠删码配置** - 不同数据/奇偶分片比例的性能影响
- **SIMD模式性能** - SIMD优化的性能表现
- **并发性能** - 多线程环境下的性能表现
- **内存效率** - 内存使用模式和效率
- **错误恢复能力** - 不同丢失分片数量下的恢复性能
## 🚀 快速开始
### 运行快速测试
```bash
# 运行快速性能对比测试（默认纯Erasure模式）
./run_benchmarks.sh quick
```
### 运行完整对比测试
```bash
# 运行详细的实现对比测试
./run_benchmarks.sh comparison
```
### 运行特定模式的测试
```bash
# 测试默认纯 erasure 模式（推荐）
./run_benchmarks.sh erasure
# 测试SIMD模式
./run_benchmarks.sh simd
```
## 📈 手动运行基准测试
### 基本使用
```bash
# 运行所有基准测试（默认纯 erasure 模式）
cargo bench
# 运行特定的基准测试文件
cargo bench --bench erasure_benchmark
cargo bench --bench comparison_benchmark
```
### 对比不同实现模式
```bash
# 测试默认纯 erasure 模式
cargo bench --bench comparison_benchmark
# 测试SIMD模式
cargo bench --bench comparison_benchmark \
--features reed-solomon-simd
# 保存基线进行对比
cargo bench --bench comparison_benchmark \
-- --save-baseline erasure_baseline
# 与基线比较SIMD模式性能
cargo bench --bench comparison_benchmark \
--features reed-solomon-simd \
-- --baseline erasure_baseline
```
### 过滤特定测试
```bash
# 只运行编码测试
cargo bench encode
# 只运行解码测试
cargo bench decode
# 只运行特定数据大小的测试
cargo bench 1MB
# 只运行特定配置的测试
cargo bench "4+2"
```
## 📊 查看结果
### HTML 报告
基准测试结果会自动生成 HTML 报告：
```bash
# 启动本地服务器查看报告
cd target/criterion
python3 -m http.server 8080
# 在浏览器中访问
open http://localhost:8080/report/index.html
```
### 命令行输出
基准测试会在终端显示：
- 每秒操作数 (ops/sec)
- 吞吐量 (MB/s)
- 延迟统计 (平均值、标准差、百分位数)
- 性能变化趋势
## 🔧 测试配置
### 数据大小
- **小数据**: 1KB, 8KB - 测试小文件场景
- **中等数据**: 64KB, 256KB - 测试常见文件大小
- **大数据**: 1MB, 4MB - 测试大文件处理和 SIMD 优化
- **超大数据**: 16MB+ - 测试高吞吐量场景
### 纠删码配置
- **(4,2)** - 常用配置，33% 冗余
- **(6,3)** - 50% 冗余，平衡性能和可靠性
- **(8,4)** - 50% 冗余，更多并行度
- **(10,5)**, **(12,6)** - 高并行度配置
### 分片大小
测试从 32 字节到 8KB 的不同分片大小，特别关注：
- **内存对齐**: 64, 128, 256 字节 - 内存对齐对性能的影响
- **Cache 友好**: 1KB, 2KB, 4KB - CPU 缓存友好的大小
## 📝 解读测试结果
### 性能指标
1. **吞吐量 (Throughput)**
- 单位: MB/s 或 GB/s
- 衡量数据处理速度
- 越高越好
2. **延迟 (Latency)**
- 单位: 微秒 (μs) 或毫秒 (ms)
- 衡量单次操作时间
- 越低越好
3. **CPU 效率**
- 每 CPU 周期处理的字节数
- 反映算法效率
### 预期结果
**纯 Erasure 模式（默认）**:
- 性能稳定，对分片大小不敏感
- 兼容性最佳，支持所有配置
- 内存使用稳定可预测
**SIMD模式（`reed-solomon-simd` feature）**:
- 高性能SIMD优化实现
- 适合大数据量处理场景
- 专注于最大化性能
**分片大小敏感性**:
- SIMD模式对分片大小可能更敏感
- 纯 Erasure 模式对分片大小相对不敏感
**内存使用**:
- SIMD模式可能有特定的内存对齐要求
- 纯 Erasure 模式内存使用更稳定
## 🛠️ 自定义测试
### 添加新的测试场景
编辑 `benches/erasure_benchmark.rs` 或 `benches/comparison_benchmark.rs`：
```rust
// 添加新的测试配置
let configs = vec![
// 你的自定义配置
BenchConfig::new(10, 4, 2048 * 1024, 2048 * 1024), // 10+4, 2MB
];
```
### 调整测试参数
```rust
// 修改采样和测试时间
group.sample_size(20); // 样本数量
group.measurement_time(Duration::from_secs(10)); // 测试时间
```
## 🐛 故障排除
### 常见问题
1. **编译错误**: 确保安装了正确的依赖
```bash
cargo update
cargo build --all-features
```
2. **性能异常**: 检查是否在正确的模式下运行
```bash
# 检查当前配置
cargo bench --bench comparison_benchmark -- --help
```
3. **测试时间过长**: 调整测试参数
```bash
# 使用更短的测试时间
cargo bench -- --quick
```
### 性能分析
使用 `perf` 等工具进行更详细的性能分析：
```bash
# 分析 CPU 使用情况
cargo bench --bench comparison_benchmark &
perf record -p $(pgrep -f comparison_benchmark)
perf report
```
## 🤝 贡献
欢迎提交新的基准测试场景或优化建议：
1. Fork 项目
2. 创建特性分支: `git checkout -b feature/new-benchmark`
3. 添加测试用例
4. 提交更改: `git commit -m 'Add new benchmark for XYZ'`
5. 推送到分支: `git push origin feature/new-benchmark`
6. 创建 Pull Request
## 📚 参考资料
- [reed-solomon-erasure crate](https://crates.io/crates/reed-solomon-erasure)
- [reed-solomon-simd crate](https://crates.io/crates/reed-solomon-simd)
- [Criterion.rs 基准测试框架](https://bheisler.github.io/criterion.rs/book/)
- [Reed-Solomon 纠删码原理](https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction)
---
💡 **提示**:
- 推荐使用默认的纯Erasure模式，它在各种场景下都有稳定的表现
- 对于高性能需求可以考虑SIMD模式
- 基准测试结果可能因硬件、操作系统和编译器版本而异
- 建议在目标部署环境中运行测试以获得最准确的性能数据
ecstore/Cargo.toml
+11 -7
View File
@@ -11,12 +11,10 @@ rust-version.workspace = true
workspace = true
[features]
default = ["reed-solomon-simd"]
reed-solomon-simd = []
reed-solomon-erasure = []
default = []
[dependencies]
rustfs-config = { workspace = true }
rustfs-config = { workspace = true, features = ["constants"] }
async-trait.workspace = true
backon.workspace = true
blake2 = { workspace = true }
@@ -43,7 +41,6 @@ http-body-util = "0.1.1"
highway = { workspace = true }
url.workspace = true
uuid = { workspace = true, features = ["v4", "fast-rng", "serde"] }
reed-solomon-erasure = { version = "6.0.0", features = ["simd-accel"] }
reed-solomon-simd = { version = "3.0.0" }
transform-stream = "0.3.1"
lazy_static.workspace = true
@@ -59,8 +56,12 @@ tokio-util = { workspace = true, features = ["io", "compat"] }
crc32fast = { workspace = true }
siphasher = { workspace = true }
base64-simd = { workspace = true }
sha1 = { workspace = true }
base64 = { workspace = true }
hmac = { workspace = true }
sha2 = { workspace = true }
sha1 = { workspace = true }
hex-simd = { workspace = true }
path-clean = { workspace = true }
tempfile.workspace = true
@@ -94,6 +95,8 @@ shadow-rs.workspace = true
rustfs-filemeta.workspace = true
rustfs-utils ={workspace = true, features=["full"]}
rustfs-rio.workspace = true
futures-util.workspace = true
serde_urlencoded.workspace = true
reader = { workspace = true }
[target.'cfg(not(windows))'.dependencies]
@@ -106,6 +109,7 @@ winapi = { workspace = true }
[dev-dependencies]
tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }
criterion = { version = "0.5", features = ["html_reports"] }
temp-env = "0.2.0"
[build-dependencies]
shadow-rs = { workspace = true, features = ["build", "metadata"] }
@@ -116,4 +120,4 @@ harness = false
[[bench]]
name = "comparison_benchmark"
harness = false
harness = false
ecstore/IMPLEMENTATION_COMPARISON.md
-333
View File
@@ -1,333 +0,0 @@
# Reed-Solomon Implementation Comparison Analysis
## 🔍 Issue Analysis
With the optimized SIMD mode design, we provide high-performance Reed-Solomon implementation. The system can now deliver optimal performance across different scenarios.
## 📊 Implementation Mode Comparison
### 🏛️ Pure Erasure Mode (Default, Recommended)
**Default Configuration**: No features specified, uses stable reed-solomon-erasure implementation
**Characteristics**:
- ✅ **Wide Compatibility**: Supports any shard size from byte-level to GB-level
- 📈 **Stable Performance**: Performance insensitive to shard size, predictable
- 🔧 **Production Ready**: Mature and stable implementation, widely used in production
- 💾 **Memory Efficient**: Optimized memory usage patterns
- 🎯 **Consistency**: Completely consistent behavior across all scenarios
**Use Cases**:
- Default choice for most production environments
- Systems requiring completely consistent and predictable performance behavior
- Performance-change-sensitive systems
- Scenarios mainly processing small files or small shards
- Systems requiring strict memory usage control
### 🎯 SIMD Mode (`reed-solomon-simd` feature)
**Configuration**: `--features reed-solomon-simd`
**Characteristics**:
- 🚀 **High-Performance SIMD**: Uses SIMD instruction sets for high-performance encoding/decoding
- 🎯 **Performance Oriented**: Focuses on maximizing processing performance
- ⚡ **Large Data Optimization**: Suitable for high-throughput scenarios with large data processing
- 🏎️ **Speed Priority**: Designed for performance-critical applications
**Use Cases**:
- Application scenarios requiring maximum performance
- High-throughput systems processing large amounts of data
- Scenarios with extremely high performance requirements
- CPU-intensive workloads
## 📏 Shard Size vs Performance Comparison
Performance across different configurations:
| Data Size | Config | Shard Size | Pure Erasure Mode (Default) | SIMD Mode Strategy | Performance Comparison |
|-----------|--------|------------|----------------------------|-------------------|----------------------|
| 1KB | 4+2 | 256 bytes | Erasure implementation | SIMD implementation | SIMD may be faster |
| 1KB | 6+3 | 171 bytes | Erasure implementation | SIMD implementation | SIMD may be faster |
| 1KB | 8+4 | 128 bytes | Erasure implementation | SIMD implementation | SIMD may be faster |
| 64KB | 4+2 | 16KB | Erasure implementation | SIMD optimization | SIMD mode faster |
| 64KB | 6+3 | 10.7KB | Erasure implementation | SIMD optimization | SIMD mode faster |
| 1MB | 4+2 | 256KB | Erasure implementation | SIMD optimization | SIMD mode significantly faster |
| 16MB | 8+4 | 2MB | Erasure implementation | SIMD optimization | SIMD mode substantially faster |
## 🎯 Benchmark Results Interpretation
### Pure Erasure Mode Example (Default) ✅
```
encode_comparison/implementation/1KB_6+3_erasure
time: [245.67 ns 256.78 ns 267.89 ns]
thrpt: [3.73 GiB/s 3.89 GiB/s 4.07 GiB/s]
💡 Consistent Erasure performance - All configurations use the same implementation
```
```
encode_comparison/implementation/64KB_4+2_erasure
time: [2.3456 μs 2.4567 μs 2.5678 μs]
thrpt: [23.89 GiB/s 24.65 GiB/s 25.43 GiB/s]
💡 Stable and reliable performance - Suitable for most production scenarios
```
### SIMD Mode Success Examples ✅
**Large Shard SIMD Optimization**:
```
encode_comparison/implementation/64KB_4+2_simd
time: [1.2345 μs 1.2567 μs 1.2789 μs]
thrpt: [47.89 GiB/s 48.65 GiB/s 49.43 GiB/s]
💡 Using SIMD optimization - Shard size: 16KB, high-performance processing
```
**Small Shard SIMD Processing**:
```
encode_comparison/implementation/1KB_6+3_simd
time: [234.56 ns 245.67 ns 256.78 ns]
thrpt: [3.89 GiB/s 4.07 GiB/s 4.26 GiB/s]
💡 SIMD processing small shards - Shard size: 171 bytes
```
## 🛠️ Usage Guide
### Selection Strategy
#### 1️⃣ Recommended: Pure Erasure Mode (Default)
```bash
# No features needed, use default configuration
cargo run
cargo test
cargo bench
```
**Applicable Scenarios**:
- 📊 **Consistency Requirements**: Need completely predictable performance behavior
- 🔬 **Production Environment**: Best choice for most production scenarios
- 💾 **Memory Sensitive**: Strict requirements for memory usage patterns
- 🏗️ **Stable and Reliable**: Mature and stable implementation
#### 2️⃣ High Performance Requirements: SIMD Mode
```bash
# Enable SIMD mode for maximum performance
cargo run --features reed-solomon-simd
cargo test --features reed-solomon-simd
cargo bench --features reed-solomon-simd
```
**Applicable Scenarios**:
- 🎯 **High Performance Scenarios**: Processing large amounts of data requiring maximum throughput
- 🚀 **Performance Optimization**: Want optimal performance for large data
- ⚡ **Speed Priority**: Scenarios with extremely high speed requirements
- 🏎️ **Compute Intensive**: CPU-intensive workloads
### Configuration Optimization Recommendations
#### Based on Data Size
**Small Files Primarily** (< 64KB):
```toml
# Recommended to use default pure Erasure mode
# No special configuration needed, stable and reliable performance
```
**Large Files Primarily** (> 1MB):
```toml
# Recommend enabling SIMD mode for higher performance
# features = ["reed-solomon-simd"]
```
**Mixed Scenarios**:
```toml
# Default pure Erasure mode suits most scenarios
# For maximum performance, enable: features = ["reed-solomon-simd"]
```
#### Recommendations Based on Erasure Coding Configuration
| Config | Small Data (< 64KB) | Large Data (> 1MB) | Recommended Mode |
|--------|-------------------|-------------------|------------------|
| 4+2 | Pure Erasure | Pure Erasure / SIMD Mode | Pure Erasure (Default) |
| 6+3 | Pure Erasure | Pure Erasure / SIMD Mode | Pure Erasure (Default) |
| 8+4 | Pure Erasure | Pure Erasure / SIMD Mode | Pure Erasure (Default) |
| 10+5 | Pure Erasure | Pure Erasure / SIMD Mode | Pure Erasure (Default) |
### Production Environment Deployment Recommendations
#### 1️⃣ Default Deployment Strategy
```bash
# Production environment recommended configuration: Use pure Erasure mode (default)
cargo build --release
```
**Advantages**:
- ✅ Maximum compatibility: Handle data of any size
- ✅ Stable and reliable: Mature implementation, predictable behavior
- ✅ Zero configuration: No complex performance tuning needed
- ✅ Memory efficient: Optimized memory usage patterns
#### 2️⃣ High Performance Deployment Strategy
```bash
# High performance scenarios: Enable SIMD mode
cargo build --release --features reed-solomon-simd
```
**Advantages**:
- ✅ Optimal performance: SIMD instruction set optimization
- ✅ High throughput: Suitable for large data processing
- ✅ Performance oriented: Focuses on maximizing processing speed
- ✅ Modern hardware: Fully utilizes modern CPU features
#### 2️⃣ Monitoring and Tuning
```rust
// Choose appropriate implementation based on specific scenarios
match data_size {
size if size > 1024 * 1024 => {
// Large data: Consider using SIMD mode
println!("Large data detected, SIMD mode recommended");
}
_ => {
// General case: Use default Erasure mode
println!("Using default Erasure mode");
}
}
```
#### 3️⃣ Performance Monitoring Metrics
- **Throughput Monitoring**: Monitor encoding/decoding data processing rates
- **Latency Analysis**: Analyze processing latency for different data sizes
- **CPU Utilization**: Observe CPU utilization efficiency of SIMD instructions
- **Memory Usage**: Monitor memory allocation patterns of different implementations
## 🔧 Troubleshooting
### Performance Issue Diagnosis
#### Issue 1: Performance Not Meeting Expectations
**Symptom**: SIMD mode performance improvement not significant
**Cause**: Data size may not be suitable for SIMD optimization
**Solution**:
```rust
// Check shard size and data characteristics
let shard_size = data.len().div_ceil(data_shards);
println!("Shard size: {} bytes", shard_size);
if shard_size >= 1024 {
println!("Good candidate for SIMD optimization");
} else {
println!("Consider using default Erasure mode");
}
```
#### Issue 2: Compilation Errors
**Symptom**: SIMD-related compilation errors
**Cause**: Platform not supported or missing dependencies
**Solution**:
```bash
# Check platform support
cargo check --features reed-solomon-simd
# If failed, use default mode
cargo check
```
#### Issue 3: Abnormal Memory Usage
**Symptom**: Memory usage exceeds expectations
**Cause**: Memory alignment requirements of SIMD implementation
**Solution**:
```bash
# Use pure Erasure mode for comparison
cargo run --features reed-solomon-erasure
```
### Debugging Tips
#### 1️⃣ Performance Comparison Testing
```bash
# Test pure Erasure mode performance
cargo bench --features reed-solomon-erasure
# Test SIMD mode performance
cargo bench --features reed-solomon-simd
```
#### 2️⃣ Analyze Data Characteristics
```rust
// Statistics of data characteristics in your application
let data_sizes: Vec<usize> = data_samples.iter()
.map(|data| data.len())
.collect();
let large_data_count = data_sizes.iter()
.filter(|&&size| size >= 1024 * 1024)
.count();
println!("Large data (>1MB): {}/{} ({}%)",
large_data_count,
data_sizes.len(),
large_data_count * 100 / data_sizes.len()
);
```
#### 3️⃣ Benchmark Comparison
```bash
# Generate detailed performance comparison report
./run_benchmarks.sh comparison
# View HTML report to analyze performance differences
cd target/criterion && python3 -m http.server 8080
```
## 📈 Performance Optimization Recommendations
### Application Layer Optimization
#### 1️⃣ Data Chunking Strategy
```rust
// Optimize data chunking for SIMD mode
const OPTIMAL_BLOCK_SIZE: usize = 1024 * 1024; // 1MB
const MIN_EFFICIENT_SIZE: usize = 64 * 1024; // 64KB
let block_size = if data.len() < MIN_EFFICIENT_SIZE {
data.len() // Small data can consider default mode
} else {
OPTIMAL_BLOCK_SIZE.min(data.len()) // Use optimal block size
};
```
#### 2️⃣ Configuration Tuning
```rust
// Choose erasure coding configuration based on typical data size
let (data_shards, parity_shards) = if typical_file_size > 1024 * 1024 {
(8, 4) // Large files: more parallelism, utilize SIMD
} else {
(4, 2) // Small files: simple configuration, reduce overhead
};
```
### System Layer Optimization
#### 1️⃣ CPU Feature Detection
```bash
# Check CPU supported SIMD instruction sets
lscpu | grep -i flags
cat /proc/cpuinfo | grep -i flags | head -1
```
#### 2️⃣ Memory Alignment Optimization
```rust
// Ensure data memory alignment to improve SIMD performance
use aligned_vec::AlignedVec;
let aligned_data = AlignedVec::<u8, aligned_vec::A64>::from_slice(&data);
```
---
💡 **Key Conclusions**:
- 🎯 **Pure Erasure mode (default) is the best general choice**: Stable and reliable, suitable for most scenarios
- 🚀 **SIMD mode suitable for high-performance scenarios**: Best choice for large data processing
- 📊 **Choose based on data characteristics**: Small data use Erasure, large data consider SIMD
- 🛡️ **Stability priority**: Production environments recommend using default Erasure mode
ecstore/IMPLEMENTATION_COMPARISON_ZH.md
-333
View File
@@ -1,333 +0,0 @@
# Reed-Solomon 实现对比分析
## 🔍 问题分析
随着SIMD模式的优化设计，我们提供了高性能的Reed-Solomon实现。现在系统能够在不同场景下提供最优的性能表现。
## 📊 实现模式对比
### 🏛️ 纯 Erasure 模式（默认，推荐）
**默认配置**: 不指定任何 feature，使用稳定的 reed-solomon-erasure 实现
**特点**:
- ✅ **广泛兼容**: 支持任意分片大小，从字节级到 GB 级
- 📈 **稳定性能**: 性能对分片大小不敏感，可预测
- 🔧 **生产就绪**: 成熟稳定的实现，已在生产环境广泛使用
- 💾 **内存高效**: 优化的内存使用模式
- 🎯 **一致性**: 在所有场景下行为完全一致
**使用场景**:
- 大多数生产环境的默认选择
- 需要完全一致和可预测的性能行为
- 对性能变化敏感的系统
- 主要处理小文件或小分片的场景
- 需要严格的内存使用控制
### 🎯 SIMD模式（`reed-solomon-simd` feature）
**配置**: `--features reed-solomon-simd`
**特点**:
- 🚀 **高性能SIMD**: 使用SIMD指令集进行高性能编码解码
- 🎯 **性能导向**: 专注于最大化处理性能
- ⚡ **大数据优化**: 适合大数据量处理的高吞吐量场景
- 🏎️ **速度优先**: 为性能关键型应用设计
**使用场景**:
- 需要最大化性能的应用场景
- 处理大量数据的高吞吐量系统
- 对性能要求极高的场景
- CPU密集型工作负载
## 📏 分片大小与性能对比
不同配置下的性能表现：
| 数据大小 | 配置 | 分片大小 | 纯 Erasure 模式（默认） | SIMD模式策略 | 性能对比 |
|---------|------|----------|------------------------|-------------|----------|
| 1KB | 4+2 | 256字节 | Erasure 实现 | SIMD 实现 | SIMD可能更快 |
| 1KB | 6+3 | 171字节 | Erasure 实现 | SIMD 实现 | SIMD可能更快 |
| 1KB | 8+4 | 128字节 | Erasure 实现 | SIMD 实现 | SIMD可能更快 |
| 64KB | 4+2 | 16KB | Erasure 实现 | SIMD 优化 | SIMD模式更快 |
| 64KB | 6+3 | 10.7KB | Erasure 实现 | SIMD 优化 | SIMD模式更快 |
| 1MB | 4+2 | 256KB | Erasure 实现 | SIMD 优化 | SIMD模式显著更快 |
| 16MB | 8+4 | 2MB | Erasure 实现 | SIMD 优化 | SIMD模式大幅领先 |
## 🎯 基准测试结果解读
### 纯 Erasure 模式示例（默认） ✅
```
encode_comparison/implementation/1KB_6+3_erasure
time: [245.67 ns 256.78 ns 267.89 ns]
thrpt: [3.73 GiB/s 3.89 GiB/s 4.07 GiB/s]
💡 一致的 Erasure 性能 - 所有配置都使用相同实现
```
```
encode_comparison/implementation/64KB_4+2_erasure
time: [2.3456 μs 2.4567 μs 2.5678 μs]
thrpt: [23.89 GiB/s 24.65 GiB/s 25.43 GiB/s]
💡 稳定可靠的性能 - 适合大多数生产场景
```
### SIMD模式成功示例 ✅
**大分片 SIMD 优化**:
```
encode_comparison/implementation/64KB_4+2_simd
time: [1.2345 μs 1.2567 μs 1.2789 μs]
thrpt: [47.89 GiB/s 48.65 GiB/s 49.43 GiB/s]
💡 使用 SIMD 优化 - 分片大小: 16KB，高性能处理
```
**小分片 SIMD 处理**:
```
encode_comparison/implementation/1KB_6+3_simd
time: [234.56 ns 245.67 ns 256.78 ns]
thrpt: [3.89 GiB/s 4.07 GiB/s 4.26 GiB/s]
💡 SIMD 处理小分片 - 分片大小: 171字节
```
## 🛠️ 使用指南
### 选择策略
#### 1️⃣ 推荐：纯 Erasure 模式（默认）
```bash
# 无需指定 feature，使用默认配置
cargo run
cargo test
cargo bench
```
**适用场景**:
- 📊 **一致性要求**: 需要完全可预测的性能行为
- 🔬 **生产环境**: 大多数生产场景的最佳选择
- 💾 **内存敏感**: 对内存使用模式有严格要求
- 🏗️ **稳定可靠**: 成熟稳定的实现
#### 2️⃣ 高性能需求：SIMD模式
```bash
# 启用SIMD模式获得最大性能
cargo run --features reed-solomon-simd
cargo test --features reed-solomon-simd
cargo bench --features reed-solomon-simd
```
**适用场景**:
- 🎯 **高性能场景**: 处理大量数据需要最大吞吐量
- 🚀 **性能优化**: 希望在大数据时获得最佳性能
- ⚡ **速度优先**: 对处理速度有极高要求的场景
- 🏎️ **计算密集**: CPU密集型工作负载
### 配置优化建议
#### 针对数据大小的配置
**小文件为主** (< 64KB):
```toml
# 推荐使用默认纯 Erasure 模式
# 无需特殊配置，性能稳定可靠
```
**大文件为主** (> 1MB):
```toml
# 建议启用SIMD模式获得更高性能
# features = ["reed-solomon-simd"]
```
**混合场景**:
```toml
# 默认纯 Erasure 模式适合大多数场景
# 如需最大性能可启用: features = ["reed-solomon-simd"]
```
#### 针对纠删码配置的建议
| 配置 | 小数据 (< 64KB) | 大数据 (> 1MB) | 推荐模式 |
|------|----------------|----------------|----------|
| 4+2 | 纯 Erasure | 纯 Erasure / SIMD模式 | 纯 Erasure（默认） |
| 6+3 | 纯 Erasure | 纯 Erasure / SIMD模式 | 纯 Erasure（默认） |
| 8+4 | 纯 Erasure | 纯 Erasure / SIMD模式 | 纯 Erasure（默认） |
| 10+5 | 纯 Erasure | 纯 Erasure / SIMD模式 | 纯 Erasure（默认） |
### 生产环境部署建议
#### 1️⃣ 默认部署策略
```bash
# 生产环境推荐配置：使用纯 Erasure 模式（默认）
cargo build --release
```
**优势**:
- ✅ 最大兼容性：处理任意大小数据
- ✅ 稳定可靠：成熟的实现，行为可预测
- ✅ 零配置：无需复杂的性能调优
- ✅ 内存高效：优化的内存使用模式
#### 2️⃣ 高性能部署策略
```bash
# 高性能场景：启用SIMD模式
cargo build --release --features reed-solomon-simd
```
**优势**:
- ✅ 最优性能：SIMD指令集优化
- ✅ 高吞吐量：适合大数据处理
- ✅ 性能导向：专注于最大化处理速度
- ✅ 现代硬件：充分利用现代CPU特性
#### 2️⃣ 监控和调优
```rust
// 根据具体场景选择合适的实现
match data_size {
size if size > 1024 * 1024 => {
// 大数据：考虑使用SIMD模式
println!("Large data detected, SIMD mode recommended");
}
_ => {
// 一般情况：使用默认Erasure模式
println!("Using default Erasure mode");
}
}
```
#### 3️⃣ 性能监控指标
- **吞吐量监控**: 监控编码/解码的数据处理速率
- **延迟分析**: 分析不同数据大小的处理延迟
- **CPU使用率**: 观察SIMD指令的CPU利用效率
- **内存使用**: 监控不同实现的内存分配模式
## 🔧 故障排除
### 性能问题诊断
#### 问题1: 性能不符合预期
**现象**: SIMD模式性能提升不明显
**原因**: 可能数据大小不适合SIMD优化
**解决**:
```rust
// 检查分片大小和数据特征
let shard_size = data.len().div_ceil(data_shards);
println!("Shard size: {} bytes", shard_size);
if shard_size >= 1024 {
println!("Good candidate for SIMD optimization");
} else {
println!("Consider using default Erasure mode");
}
```
#### 问题2: 编译错误
**现象**: SIMD相关的编译错误
**原因**: 平台不支持或依赖缺失
**解决**:
```bash
# 检查平台支持
cargo check --features reed-solomon-simd
# 如果失败，使用默认模式
cargo check
```
#### 问题3: 内存使用异常
**现象**: 内存使用超出预期
**原因**: SIMD实现的内存对齐要求
**解决**:
```bash
# 使用纯 Erasure 模式进行对比
cargo run --features reed-solomon-erasure
```
### 调试技巧
#### 1️⃣ 性能对比测试
```bash
# 测试纯 Erasure 模式性能
cargo bench --features reed-solomon-erasure
# 测试SIMD模式性能
cargo bench --features reed-solomon-simd
```
#### 2️⃣ 分析数据特征
```rust
// 统计你的应用中的数据特征
let data_sizes: Vec<usize> = data_samples.iter()
.map(|data| data.len())
.collect();
let large_data_count = data_sizes.iter()
.filter(|&&size| size >= 1024 * 1024)
.count();
println!("Large data (>1MB): {}/{} ({}%)",
large_data_count,
data_sizes.len(),
large_data_count * 100 / data_sizes.len()
);
```
#### 3️⃣ 基准测试对比
```bash
# 生成详细的性能对比报告
./run_benchmarks.sh comparison
# 查看 HTML 报告分析性能差异
cd target/criterion && python3 -m http.server 8080
```
## 📈 性能优化建议
### 应用层优化
#### 1️⃣ 数据分块策略
```rust
// 针对SIMD模式优化数据分块
const OPTIMAL_BLOCK_SIZE: usize = 1024 * 1024; // 1MB
const MIN_EFFICIENT_SIZE: usize = 64 * 1024; // 64KB
let block_size = if data.len() < MIN_EFFICIENT_SIZE {
data.len() // 小数据可以考虑默认模式
} else {
OPTIMAL_BLOCK_SIZE.min(data.len()) // 使用最优块大小
};
```
#### 2️⃣ 配置调优
```rust
// 根据典型数据大小选择纠删码配置
let (data_shards, parity_shards) = if typical_file_size > 1024 * 1024 {
(8, 4) // 大文件：更多并行度，利用 SIMD
} else {
(4, 2) // 小文件：简单配置，减少开销
};
```
### 系统层优化
#### 1️⃣ CPU 特性检测
```bash
# 检查 CPU 支持的 SIMD 指令集
lscpu | grep -i flags
cat /proc/cpuinfo | grep -i flags | head -1
```
#### 2️⃣ 内存对齐优化
```rust
// 确保数据内存对齐以提升 SIMD 性能
use aligned_vec::AlignedVec;
let aligned_data = AlignedVec::<u8, aligned_vec::A64>::from_slice(&data);
```
---
💡 **关键结论**:
- 🎯 **纯Erasure模式（默认）是最佳通用选择**：稳定可靠，适合大多数场景
- 🚀 **SIMD模式适合高性能场景**：大数据处理的最佳选择
- 📊 **根据数据特征选择**：小数据用Erasure，大数据考虑SIMD
- 🛡️ **稳定性优先**：生产环境建议使用默认Erasure模式
ecstore/README_cn.md
+42 -55
View File
@@ -1,38 +1,15 @@
# ECStore - Erasure Coding Storage
ECStore provides erasure coding functionality for the RustFS project, supporting multiple Reed-Solomon implementations for optimal performance and compatibility.
ECStore provides erasure coding functionality for the RustFS project, using high-performance Reed-Solomon SIMD implementation for optimal performance.
## Reed-Solomon Implementations
## Reed-Solomon Implementation
### Available Backends
### SIMD Backend (Only)
#### `reed-solomon-erasure` (Default)
- **Stability**: Mature and well-tested implementation
- **Performance**: Good performance with SIMD acceleration when available
- **Compatibility**: Works with any shard size
- **Memory**: Efficient memory usage
- **Use case**: Recommended for production use
#### `reed-solomon-simd` (Optional)
- **Performance**: Optimized SIMD implementation for maximum speed
- **Limitations**: Has restrictions on shard sizes (must be >= 64 bytes typically)
- **Memory**: May use more memory for small shards
- **Use case**: Best for large data blocks where performance is critical
### Feature Flags
Configure the Reed-Solomon implementation using Cargo features:
```toml
# Use default implementation (reed-solomon-erasure)
ecstore = "0.0.1"
# Use SIMD implementation for maximum performance
ecstore = { version = "0.0.1", features = ["reed-solomon-simd"], default-features = false }
# Use traditional implementation explicitly
ecstore = { version = "0.0.1", features = ["reed-solomon-erasure"], default-features = false }
```
- **Performance**: Uses SIMD optimization for high-performance encoding/decoding
- **Compatibility**: Works with any shard size through SIMD implementation
- **Reliability**: High-performance SIMD implementation for large data processing
- **Use case**: Optimized for maximum performance in large data processing scenarios
### Usage Example
@@ -68,42 +45,52 @@ assert_eq!(&recovered, data);
## Performance Considerations
### When to use `reed-solomon-simd`
- Large block sizes (>= 1KB recommended)
- High-throughput scenarios
- CPU-intensive workloads where encoding/decoding is the bottleneck
### When to use `reed-solomon-erasure`
- Small block sizes
- Memory-constrained environments
- General-purpose usage
- Production deployments requiring maximum stability
### SIMD Implementation Benefits
- **High Throughput**: Optimized for large block sizes (>= 1KB recommended)
- **CPU Optimization**: Leverages modern CPU SIMD instructions
- **Scalability**: Excellent performance for high-throughput scenarios
### Implementation Details
#### `reed-solomon-erasure`
- **Instance Reuse**: The encoder instance is cached and reused across multiple operations
- **Thread Safety**: Thread-safe with interior mutability
- **Memory Efficiency**: Lower memory footprint for small data
#### `reed-solomon-simd`
- **Instance Creation**: New encoder/decoder instances are created for each operation
- **API Design**: The SIMD implementation's API is designed for single-use instances
- **Performance Trade-off**: While instances are created per operation, the SIMD optimizations provide significant performance benefits for large data blocks
- **Optimization**: Future versions may implement instance pooling if the underlying API supports reuse
- **Instance Caching**: Encoder/decoder instances are cached and reused for optimal performance
- **Thread Safety**: Thread-safe with RwLock-based caching
- **SIMD Optimization**: Leverages CPU SIMD instructions for maximum performance
- **Reset Capability**: Cached instances are reset for different parameters, avoiding unnecessary allocations
### Performance Tips
1. **Batch Operations**: When possible, batch multiple small operations into larger blocks
2. **Block Size Optimization**: Use block sizes that are multiples of 64 bytes for SIMD implementations
2. **Block Size Optimization**: Use block sizes that are multiples of 64 bytes for optimal SIMD performance
3. **Memory Allocation**: Pre-allocate buffers when processing multiple blocks
4. **Feature Selection**: Choose the appropriate feature based on your data size and performance requirements
4. **Cache Warming**: Initial operations may be slower due to cache setup, subsequent operations benefit from caching
## Cross-Platform Compatibility
Both implementations support:
- x86_64 with SIMD acceleration
- aarch64 (ARM64) with optimizations
The SIMD implementation supports:
- x86_64 with advanced SIMD instructions (AVX2, SSE)
- aarch64 (ARM64) with NEON SIMD optimizations
- Other architectures with fallback implementations
The `reed-solomon-erasure` implementation provides better cross-platform compatibility and is recommended for most use cases.
The implementation automatically selects the best available SIMD instructions for the target platform, providing optimal performance across different architectures.
## Testing and Benchmarking
Run performance benchmarks:
```bash
# Run erasure coding benchmarks
cargo bench --bench erasure_benchmark
# Run comparison benchmarks
cargo bench --bench comparison_benchmark
# Generate benchmark reports
./run_benchmarks.sh
```
## Error Handling
All operations return `Result` types with comprehensive error information:
- Encoding errors: Invalid parameters, insufficient memory
- Decoding errors: Too many missing shards, corrupted data
- Configuration errors: Invalid shard counts, unsupported parameters
ecstore/benches/comparison_benchmark.rs
+102 -115
View File
@@ -1,29 +1,28 @@
//! 专门比较 Pure Erasure 和 Hybrid (SIMD) 模式性能的基准测试
//! Reed-Solomon SIMD performance analysis benchmarks
//!
//! 这个基准测试使用不同的feature编译配置来直接对比两种实现的性能。
//! This benchmark analyzes the performance characteristics of the SIMD Reed-Solomon implementation
//! across different data sizes, shard configurations, and usage patterns.
//!
//! ## 运行比较测试
//! ## Running Performance Analysis
//!
//! ```bash
//! # 测试 Pure Erasure 实现 (默认)
//! # Run all SIMD performance tests
//! cargo bench --bench comparison_benchmark
//!
//! # 测试 Hybrid (SIMD) 实现
//! cargo bench --bench comparison_benchmark --features reed-solomon-simd
//! # Generate detailed performance report
//! cargo bench --bench comparison_benchmark -- --save-baseline simd_analysis
//!
//! # 测试强制 erasure-only 模式
//! cargo bench --bench comparison_benchmark --features reed-solomon-erasure
//!
//! # 生成对比报告
//! cargo bench --bench comparison_benchmark -- --save-baseline erasure
//! cargo bench --bench comparison_benchmark --features reed-solomon-simd -- --save-baseline hybrid
//! # Run specific test categories
//! cargo bench --bench comparison_benchmark encode_analysis
//! cargo bench --bench comparison_benchmark decode_analysis
//! cargo bench --bench comparison_benchmark shard_analysis
//! ```
use criterion::{BenchmarkId, Criterion, Throughput, black_box, criterion_group, criterion_main};
use ecstore::erasure_coding::Erasure;
use std::time::Duration;
/// 基准测试数据配置
/// Performance test data configuration
struct TestData {
data: Vec<u8>,
size_name: &'static str,
@@ -36,41 +35,41 @@ impl TestData {
}
}
/// 生成不同大小的测试数据集
/// Generate different sized test datasets for performance analysis
fn generate_test_datasets() -> Vec<TestData> {
vec![
TestData::new(1024, "1KB"), // 小数据
TestData::new(8 * 1024, "8KB"), // 中小数据
TestData::new(64 * 1024, "64KB"), // 中等数据
TestData::new(256 * 1024, "256KB"), // 中大数据
TestData::new(1024 * 1024, "1MB"), // 大数据
TestData::new(4 * 1024 * 1024, "4MB"), // 超大数据
TestData::new(1024, "1KB"), // Small data
TestData::new(8 * 1024, "8KB"), // Medium-small data
TestData::new(64 * 1024, "64KB"), // Medium data
TestData::new(256 * 1024, "256KB"), // Medium-large data
TestData::new(1024 * 1024, "1MB"), // Large data
TestData::new(4 * 1024 * 1024, "4MB"), // Extra large data
]
}
/// 编码性能比较基准测试
fn bench_encode_comparison(c: &mut Criterion) {
/// SIMD encoding performance analysis
fn bench_encode_analysis(c: &mut Criterion) {
let datasets = generate_test_datasets();
let configs = vec![
(4, 2, "4+2"), // 常用配置
(6, 3, "6+3"), // 50%冗余
(8, 4, "8+4"), // 50%冗余，更多分片
(4, 2, "4+2"), // Common configuration
(6, 3, "6+3"), // 50% redundancy
(8, 4, "8+4"), // 50% redundancy, more shards
];
for dataset in &datasets {
for (data_shards, parity_shards, config_name) in &configs {
let test_name = format!("{}_{}_{}", dataset.size_name, config_name, get_implementation_name());
let test_name = format!("{}_{}_{}", dataset.size_name, config_name, "simd");
let mut group = c.benchmark_group("encode_comparison");
let mut group = c.benchmark_group("encode_analysis");
group.throughput(Throughput::Bytes(dataset.data.len() as u64));
group.sample_size(20);
group.measurement_time(Duration::from_secs(10));
// 检查是否能够创建erasure实例（某些配置在纯SIMD模式下可能失败）
// Test SIMD encoding performance
match Erasure::new(*data_shards, *parity_shards, dataset.data.len()).encode_data(&dataset.data) {
Ok(_) => {
group.bench_with_input(
BenchmarkId::new("implementation", &test_name),
BenchmarkId::new("simd_encode", &test_name),
&(&dataset.data, *data_shards, *parity_shards),
|b, (data, data_shards, parity_shards)| {
let erasure = Erasure::new(*data_shards, *parity_shards, data.len());
@@ -82,7 +81,7 @@ fn bench_encode_comparison(c: &mut Criterion) {
);
}
Err(e) => {
println!("⚠️ 跳过测试 {} - 配置不支持: {}", test_name, e);
println!("⚠️ Skipping test {} - configuration not supported: {}", test_name, e);
}
}
group.finish();
@@ -90,35 +89,35 @@ fn bench_encode_comparison(c: &mut Criterion) {
}
}
/// 解码性能比较基准测试
fn bench_decode_comparison(c: &mut Criterion) {
/// SIMD decoding performance analysis
fn bench_decode_analysis(c: &mut Criterion) {
let datasets = generate_test_datasets();
let configs = vec![(4, 2, "4+2"), (6, 3, "6+3"), (8, 4, "8+4")];
for dataset in &datasets {
for (data_shards, parity_shards, config_name) in &configs {
let test_name = format!("{}_{}_{}", dataset.size_name, config_name, get_implementation_name());
let test_name = format!("{}_{}_{}", dataset.size_name, config_name, "simd");
let erasure = Erasure::new(*data_shards, *parity_shards, dataset.data.len());
// 预先编码数据 - 检查是否支持此配置
// Pre-encode data - check if this configuration is supported
match erasure.encode_data(&dataset.data) {
Ok(encoded_shards) => {
let mut group = c.benchmark_group("decode_comparison");
let mut group = c.benchmark_group("decode_analysis");
group.throughput(Throughput::Bytes(dataset.data.len() as u64));
group.sample_size(20);
group.measurement_time(Duration::from_secs(10));
group.bench_with_input(
BenchmarkId::new("implementation", &test_name),
BenchmarkId::new("simd_decode", &test_name),
&(&encoded_shards, *data_shards, *parity_shards),
|b, (shards, data_shards, parity_shards)| {
let erasure = Erasure::new(*data_shards, *parity_shards, dataset.data.len());
b.iter(|| {
// 模拟最大可恢复的数据丢失
// Simulate maximum recoverable data loss
let mut shards_opt: Vec<Option<Vec<u8>>> =
shards.iter().map(|shard| Some(shard.to_vec())).collect();
// 丢失等于奇偶校验分片数量的分片
// Lose up to parity_shards number of shards
for item in shards_opt.iter_mut().take(*parity_shards) {
*item = None;
}
@@ -131,33 +130,33 @@ fn bench_decode_comparison(c: &mut Criterion) {
group.finish();
}
Err(e) => {
println!("⚠️ 跳过解码测试 {} - 配置不支持: {}", test_name, e);
println!("⚠️ Skipping decode test {} - configuration not supported: {}", test_name, e);
}
}
}
}
}
/// 分片大小敏感性测试
fn bench_shard_size_sensitivity(c: &mut Criterion) {
/// Shard size sensitivity analysis for SIMD optimization
fn bench_shard_size_analysis(c: &mut Criterion) {
let data_shards = 4;
let parity_shards = 2;
// 测试不同的分片大小，特别关注SIMD的临界点
// Test different shard sizes, focusing on SIMD optimization thresholds
let shard_sizes = vec![32, 64, 128, 256, 512, 1024, 2048, 4096, 8192];
let mut group = c.benchmark_group("shard_size_sensitivity");
let mut group = c.benchmark_group("shard_size_analysis");
group.sample_size(15);
group.measurement_time(Duration::from_secs(8));
for shard_size in shard_sizes {
let total_size = shard_size * data_shards;
let data = (0..total_size).map(|i| (i % 256) as u8).collect::<Vec<u8>>();
let test_name = format!("{}B_shard_{}", shard_size, get_implementation_name());
let test_name = format!("{}B_shard_simd", shard_size);
group.throughput(Throughput::Bytes(total_size as u64));
// 检查此分片大小是否支持
// Check if this shard size is supported
let erasure = Erasure::new(data_shards, parity_shards, data.len());
match erasure.encode_data(&data) {
Ok(_) => {
@@ -170,15 +169,15 @@ fn bench_shard_size_sensitivity(c: &mut Criterion) {
});
}
Err(e) => {
println!("⚠️ 跳过分片大小测试 {} - 不支持: {}", test_name, e);
println!("⚠️ Skipping shard size test {} - not supported: {}", test_name, e);
}
}
}
group.finish();
}
/// 高负载并发测试
fn bench_concurrent_load(c: &mut Criterion) {
/// High-load concurrent performance analysis
fn bench_concurrent_analysis(c: &mut Criterion) {
use std::sync::Arc;
use std::thread;
@@ -186,14 +185,14 @@ fn bench_concurrent_load(c: &mut Criterion) {
let data = Arc::new((0..data_size).map(|i| (i % 256) as u8).collect::<Vec<u8>>());
let erasure = Arc::new(Erasure::new(4, 2, data_size));
let mut group = c.benchmark_group("concurrent_load");
let mut group = c.benchmark_group("concurrent_analysis");
group.throughput(Throughput::Bytes(data_size as u64));
group.sample_size(10);
group.measurement_time(Duration::from_secs(15));
let test_name = format!("1MB_concurrent_{}", get_implementation_name());
let test_name = "1MB_concurrent_simd";
group.bench_function(&test_name, |b| {
group.bench_function(test_name, |b| {
b.iter(|| {
let handles: Vec<_> = (0..4)
.map(|_| {
@@ -214,42 +213,44 @@ fn bench_concurrent_load(c: &mut Criterion) {
group.finish();
}
/// 错误恢复能力测试
fn bench_error_recovery_performance(c: &mut Criterion) {
let data_size = 256 * 1024; // 256KB
/// Error recovery performance analysis
fn bench_error_recovery_analysis(c: &mut Criterion) {
let data_size = 512 * 1024; // 512KB
let data = (0..data_size).map(|i| (i % 256) as u8).collect::<Vec<u8>>();
let configs = vec![
(4, 2, 1), // 丢失1个分片
(4, 2, 2), // 丢失2个分片（最大可恢复）
(6, 3, 2), // 丢失2个分片
(6, 3, 3), // 丢失3个分片（最大可恢复）
(8, 4, 3), // 丢失3个分片
(8, 4, 4), // 丢失4个分片（最大可恢复）
// Test different error recovery scenarios
let scenarios = vec![
(4, 2, 1, "single_loss"), // Lose 1 shard
(4, 2, 2, "double_loss"), // Lose 2 shards (maximum)
(6, 3, 1, "single_loss_6_3"), // Lose 1 shard with 6+3
(6, 3, 3, "triple_loss_6_3"), // Lose 3 shards (maximum)
(8, 4, 2, "double_loss_8_4"), // Lose 2 shards with 8+4
(8, 4, 4, "quad_loss_8_4"), // Lose 4 shards (maximum)
];
let mut group = c.benchmark_group("error_recovery");
let mut group = c.benchmark_group("error_recovery_analysis");
group.throughput(Throughput::Bytes(data_size as u64));
group.sample_size(15);
group.measurement_time(Duration::from_secs(8));
group.measurement_time(Duration::from_secs(10));
for (data_shards, parity_shards, lost_shards) in configs {
for (data_shards, parity_shards, loss_count, scenario_name) in scenarios {
let erasure = Erasure::new(data_shards, parity_shards, data_size);
let test_name = format!("{}+{}_lost{}_{}", data_shards, parity_shards, lost_shards, get_implementation_name());
// 检查此配置是否支持
match erasure.encode_data(&data) {
Ok(encoded_shards) => {
let test_name = format!("{}+{}_{}", data_shards, parity_shards, scenario_name);
group.bench_with_input(
BenchmarkId::new("recovery", &test_name),
&(&encoded_shards, data_shards, parity_shards, lost_shards),
|b, (shards, data_shards, parity_shards, lost_shards)| {
&(&encoded_shards, data_shards, parity_shards, loss_count),
|b, (shards, data_shards, parity_shards, loss_count)| {
let erasure = Erasure::new(*data_shards, *parity_shards, data_size);
b.iter(|| {
// Simulate specific number of shard losses
let mut shards_opt: Vec<Option<Vec<u8>>> = shards.iter().map(|shard| Some(shard.to_vec())).collect();
// 丢失指定数量的分片
for item in shards_opt.iter_mut().take(*lost_shards) {
// Lose the specified number of shards
for item in shards_opt.iter_mut().take(*loss_count) {
*item = None;
}
@@ -260,71 +261,57 @@ fn bench_error_recovery_performance(c: &mut Criterion) {
);
}
Err(e) => {
println!("⚠️ 跳过错误恢复测试 {} - 配置不支持: {}", test_name, e);
println!("⚠️ Skipping recovery test {}: {}", scenario_name, e);
}
}
}
group.finish();
}
/// 内存效率测试
fn bench_memory_efficiency(c: &mut Criterion) {
let data_shards = 4;
let parity_shards = 2;
let data_size = 1024 * 1024; // 1MB
/// Memory efficiency analysis
fn bench_memory_analysis(c: &mut Criterion) {
let data_sizes = vec![64 * 1024, 256 * 1024, 1024 * 1024]; // 64KB, 256KB, 1MB
let config = (4, 2); // 4+2 configuration
let mut group = c.benchmark_group("memory_efficiency");
group.throughput(Throughput::Bytes(data_size as u64));
group.sample_size(10);
let mut group = c.benchmark_group("memory_analysis");
group.sample_size(15);
group.measurement_time(Duration::from_secs(8));
let test_name = format!("memory_pattern_{}", get_implementation_name());
for data_size in data_sizes {
let data = (0..data_size).map(|i| (i % 256) as u8).collect::<Vec<u8>>();
let size_name = format!("{}KB", data_size / 1024);
// 测试连续多次编码对内存的影响
group.bench_function(format!("{}_continuous", test_name), |b| {
let erasure = Erasure::new(data_shards, parity_shards, data_size);
b.iter(|| {
for i in 0..10 {
let data = vec![(i % 256) as u8; data_size];
let shards = erasure.encode_data(black_box(&data)).unwrap();
group.throughput(Throughput::Bytes(data_size as u64));
// Test instance reuse vs new instance creation
group.bench_with_input(BenchmarkId::new("reuse_instance", &size_name), &data, |b, data| {
let erasure = Erasure::new(config.0, config.1, data.len());
b.iter(|| {
let shards = erasure.encode_data(black_box(data)).unwrap();
black_box(shards);
}
});
});
});
// 测试大量小编码任务
group.bench_function(format!("{}_small_chunks", test_name), |b| {
let chunk_size = 1024; // 1KB chunks
let erasure = Erasure::new(data_shards, parity_shards, chunk_size);
b.iter(|| {
for i in 0..1024 {
let data = vec![(i % 256) as u8; chunk_size];
let shards = erasure.encode_data(black_box(&data)).unwrap();
group.bench_with_input(BenchmarkId::new("new_instance", &size_name), &data, |b, data| {
b.iter(|| {
let erasure = Erasure::new(config.0, config.1, data.len());
let shards = erasure.encode_data(black_box(data)).unwrap();
black_box(shards);
}
});
});
});
}
group.finish();
}
/// 获取当前实现的名称
fn get_implementation_name() -> &'static str {
#[cfg(feature = "reed-solomon-simd")]
return "hybrid";
#[cfg(not(feature = "reed-solomon-simd"))]
return "erasure";
}
// Benchmark group configuration
criterion_group!(
benches,
bench_encode_comparison,
bench_decode_comparison,
bench_shard_size_sensitivity,
bench_concurrent_load,
bench_error_recovery_performance,
bench_memory_efficiency
bench_encode_analysis,
bench_decode_analysis,
bench_shard_size_analysis,
bench_concurrent_analysis,
bench_error_recovery_analysis,
bench_memory_analysis
);
criterion_main!(benches);
ecstore/benches/erasure_benchmark.rs
+118 -170
View File
@@ -1,25 +1,23 @@
//! Reed-Solomon erasure coding performance benchmarks.
//! Reed-Solomon SIMD erasure coding performance benchmarks.
//!
//! This benchmark compares the performance of different Reed-Solomon implementations:
//! - Default (Pure erasure): Stable reed-solomon-erasure implementation
//! - `reed-solomon-simd` feature: SIMD mode with optimized performance
//! This benchmark tests the performance of the high-performance SIMD Reed-Solomon implementation.
//!
//! ## Running Benchmarks
//!
//! ```bash
//! # 运行所有基准测试
//! # Run all benchmarks
//! cargo bench
//!
//! # 运行特定的基准测试
//! # Run specific benchmark
//! cargo bench --bench erasure_benchmark
//!
//! # 生成HTML报告
//! # Generate HTML report
//! cargo bench --bench erasure_benchmark -- --output-format html
//!
//! # 只测试编码性能
//! # Test encoding performance only
//! cargo bench encode
//!
//! # 只测试解码性能
//! # Test decoding performance only
//! cargo bench decode
//! ```
//!
@@ -29,24 +27,24 @@
//! - Different data sizes: 1KB, 64KB, 1MB, 16MB
//! - Different erasure coding configurations: (4,2), (6,3), (8,4)
//! - Both encoding and decoding operations
//! - Small vs large shard scenarios for SIMD optimization
//! - SIMD optimization for different shard sizes
use criterion::{BenchmarkId, Criterion, Throughput, black_box, criterion_group, criterion_main};
use ecstore::erasure_coding::{Erasure, calc_shard_size};
use std::time::Duration;
/// 基准测试配置结构体
/// Benchmark configuration structure
#[derive(Clone, Debug)]
struct BenchConfig {
/// 数据分片数量
/// Number of data shards
data_shards: usize,
/// 奇偶校验分片数量
/// Number of parity shards
parity_shards: usize,
/// 测试数据大小（字节）
/// Test data size (bytes)
data_size: usize,
/// 块大小（字节）
/// Block size (bytes)
block_size: usize,
/// 配置名称
/// Configuration name
name: String,
}
@@ -62,27 +60,27 @@ impl BenchConfig {
}
}
/// 生成测试数据
/// Generate test data
fn generate_test_data(size: usize) -> Vec<u8> {
(0..size).map(|i| (i % 256) as u8).collect()
}
/// 基准测试: 编码性能对比
/// Benchmark: Encoding performance
fn bench_encode_performance(c: &mut Criterion) {
let configs = vec![
// 小数据量测试 - 1KB
// Small data tests - 1KB
BenchConfig::new(4, 2, 1024, 1024),
BenchConfig::new(6, 3, 1024, 1024),
BenchConfig::new(8, 4, 1024, 1024),
// 中等数据量测试 - 64KB
// Medium data tests - 64KB
BenchConfig::new(4, 2, 64 * 1024, 64 * 1024),
BenchConfig::new(6, 3, 64 * 1024, 64 * 1024),
BenchConfig::new(8, 4, 64 * 1024, 64 * 1024),
// 大数据量测试 - 1MB
// Large data tests - 1MB
BenchConfig::new(4, 2, 1024 * 1024, 1024 * 1024),
BenchConfig::new(6, 3, 1024 * 1024, 1024 * 1024),
BenchConfig::new(8, 4, 1024 * 1024, 1024 * 1024),
// 超大数据量测试 - 16MB
// Extra large data tests - 16MB
BenchConfig::new(4, 2, 16 * 1024 * 1024, 16 * 1024 * 1024),
BenchConfig::new(6, 3, 16 * 1024 * 1024, 16 * 1024 * 1024),
];
@@ -90,13 +88,13 @@ fn bench_encode_performance(c: &mut Criterion) {
for config in configs {
let data = generate_test_data(config.data_size);
// 测试当前默认实现（通常是SIMD）
let mut group = c.benchmark_group("encode_current");
// Test SIMD encoding performance
let mut group = c.benchmark_group("encode_simd");
group.throughput(Throughput::Bytes(config.data_size as u64));
group.sample_size(10);
group.measurement_time(Duration::from_secs(5));
group.bench_with_input(BenchmarkId::new("current_impl", &config.name), &(&data, &config), |b, (data, config)| {
group.bench_with_input(BenchmarkId::new("simd_impl", &config.name), &(&data, &config), |b, (data, config)| {
let erasure = Erasure::new(config.data_shards, config.parity_shards, config.block_size);
b.iter(|| {
let shards = erasure.encode_data(black_box(data)).unwrap();
@@ -105,99 +103,55 @@ fn bench_encode_performance(c: &mut Criterion) {
});
group.finish();
// 如果SIMD feature启用，测试专用的erasure实现对比
#[cfg(feature = "reed-solomon-simd")]
{
use ecstore::erasure_coding::ReedSolomonEncoder;
// Test direct SIMD implementation for large shards (>= 512 bytes)
let shard_size = calc_shard_size(config.data_size, config.data_shards);
if shard_size >= 512 {
let mut simd_group = c.benchmark_group("encode_simd_direct");
simd_group.throughput(Throughput::Bytes(config.data_size as u64));
simd_group.sample_size(10);
simd_group.measurement_time(Duration::from_secs(5));
let mut erasure_group = c.benchmark_group("encode_erasure_only");
erasure_group.throughput(Throughput::Bytes(config.data_size as u64));
erasure_group.sample_size(10);
erasure_group.measurement_time(Duration::from_secs(5));
simd_group.bench_with_input(BenchmarkId::new("simd_direct", &config.name), &(&data, &config), |b, (data, config)| {
b.iter(|| {
// Direct SIMD implementation
let per_shard_size = calc_shard_size(data.len(), config.data_shards);
match reed_solomon_simd::ReedSolomonEncoder::new(config.data_shards, config.parity_shards, per_shard_size) {
Ok(mut encoder) => {
// Create properly sized buffer and fill with data
let mut buffer = vec![0u8; per_shard_size * config.data_shards];
let copy_len = data.len().min(buffer.len());
buffer[..copy_len].copy_from_slice(&data[..copy_len]);
erasure_group.bench_with_input(
BenchmarkId::new("erasure_impl", &config.name),
&(&data, &config),
|b, (data, config)| {
let encoder = ReedSolomonEncoder::new(config.data_shards, config.parity_shards).unwrap();
b.iter(|| {
// 创建编码所需的数据结构
let per_shard_size = calc_shard_size(data.len(), config.data_shards);
let total_size = per_shard_size * (config.data_shards + config.parity_shards);
let mut buffer = vec![0u8; total_size];
buffer[..data.len()].copy_from_slice(data);
let slices: smallvec::SmallVec<[&mut [u8]; 16]> = buffer.chunks_exact_mut(per_shard_size).collect();
encoder.encode(black_box(slices)).unwrap();
black_box(&buffer);
});
},
);
erasure_group.finish();
}
// 如果使用SIMD feature，测试直接SIMD实现对比
#[cfg(feature = "reed-solomon-simd")]
{
// 只对大shard测试SIMD（小于512字节的shard SIMD性能不佳）
let shard_size = calc_shard_size(config.data_size, config.data_shards);
if shard_size >= 512 {
let mut simd_group = c.benchmark_group("encode_simd_direct");
simd_group.throughput(Throughput::Bytes(config.data_size as u64));
simd_group.sample_size(10);
simd_group.measurement_time(Duration::from_secs(5));
simd_group.bench_with_input(
BenchmarkId::new("simd_impl", &config.name),
&(&data, &config),
|b, (data, config)| {
b.iter(|| {
// 直接使用SIMD实现
let per_shard_size = calc_shard_size(data.len(), config.data_shards);
match reed_solomon_simd::ReedSolomonEncoder::new(
config.data_shards,
config.parity_shards,
per_shard_size,
) {
Ok(mut encoder) => {
// 创建正确大小的缓冲区，并填充数据
let mut buffer = vec![0u8; per_shard_size * config.data_shards];
let copy_len = data.len().min(buffer.len());
buffer[..copy_len].copy_from_slice(&data[..copy_len]);
// 按正确的分片大小添加数据分片
for chunk in buffer.chunks_exact(per_shard_size) {
encoder.add_original_shard(black_box(chunk)).unwrap();
}
let result = encoder.encode().unwrap();
black_box(result);
}
Err(_) => {
// SIMD不支持此配置，跳过
black_box(());
}
// Add data shards with correct shard size
for chunk in buffer.chunks_exact(per_shard_size) {
encoder.add_original_shard(black_box(chunk)).unwrap();
}
});
},
);
simd_group.finish();
}
let result = encoder.encode().unwrap();
black_box(result);
}
Err(_) => {
// SIMD doesn't support this configuration, skip
black_box(());
}
}
});
});
simd_group.finish();
}
}
}
/// 基准测试: 解码性能对比
/// Benchmark: Decoding performance
fn bench_decode_performance(c: &mut Criterion) {
let configs = vec![
// 中等数据量测试 - 64KB
// Medium data tests - 64KB
BenchConfig::new(4, 2, 64 * 1024, 64 * 1024),
BenchConfig::new(6, 3, 64 * 1024, 64 * 1024),
// 大数据量测试 - 1MB
// Large data tests - 1MB
BenchConfig::new(4, 2, 1024 * 1024, 1024 * 1024),
BenchConfig::new(6, 3, 1024 * 1024, 1024 * 1024),
// 超大数据量测试 - 16MB
// Extra large data tests - 16MB
BenchConfig::new(4, 2, 16 * 1024 * 1024, 16 * 1024 * 1024),
];
@@ -205,25 +159,25 @@ fn bench_decode_performance(c: &mut Criterion) {
let data = generate_test_data(config.data_size);
let erasure = Erasure::new(config.data_shards, config.parity_shards, config.block_size);
// 预先编码数据
// Pre-encode data
let encoded_shards = erasure.encode_data(&data).unwrap();
// 测试当前默认实现的解码性能
let mut group = c.benchmark_group("decode_current");
// Test SIMD decoding performance
let mut group = c.benchmark_group("decode_simd");
group.throughput(Throughput::Bytes(config.data_size as u64));
group.sample_size(10);
group.measurement_time(Duration::from_secs(5));
group.bench_with_input(
BenchmarkId::new("current_impl", &config.name),
BenchmarkId::new("simd_impl", &config.name),
&(&encoded_shards, &config),
|b, (shards, config)| {
let erasure = Erasure::new(config.data_shards, config.parity_shards, config.block_size);
b.iter(|| {
// 模拟数据丢失 - 丢失一个数据分片和一个奇偶分片
// Simulate data loss - lose one data shard and one parity shard
let mut shards_opt: Vec<Option<Vec<u8>>> = shards.iter().map(|shard| Some(shard.to_vec())).collect();
// 丢失最后一个数据分片和第一个奇偶分片
// Lose last data shard and first parity shard
shards_opt[config.data_shards - 1] = None;
shards_opt[config.data_shards] = None;
@@ -234,58 +188,52 @@ fn bench_decode_performance(c: &mut Criterion) {
);
group.finish();
// 如果使用混合模式（默认），测试SIMD解码性能
#[cfg(not(feature = "reed-solomon-erasure"))]
{
let shard_size = calc_shard_size(config.data_size, config.data_shards);
if shard_size >= 512 {
let mut simd_group = c.benchmark_group("decode_simd_direct");
simd_group.throughput(Throughput::Bytes(config.data_size as u64));
simd_group.sample_size(10);
simd_group.measurement_time(Duration::from_secs(5));
// Test direct SIMD decoding for large shards
let shard_size = calc_shard_size(config.data_size, config.data_shards);
if shard_size >= 512 {
let mut simd_group = c.benchmark_group("decode_simd_direct");
simd_group.throughput(Throughput::Bytes(config.data_size as u64));
simd_group.sample_size(10);
simd_group.measurement_time(Duration::from_secs(5));
simd_group.bench_with_input(
BenchmarkId::new("simd_impl", &config.name),
&(&encoded_shards, &config),
|b, (shards, config)| {
b.iter(|| {
let per_shard_size = calc_shard_size(config.data_size, config.data_shards);
match reed_solomon_simd::ReedSolomonDecoder::new(
config.data_shards,
config.parity_shards,
per_shard_size,
) {
Ok(mut decoder) => {
// 添加可用的分片（除了丢失的）
for (i, shard) in shards.iter().enumerate() {
if i != config.data_shards - 1 && i != config.data_shards {
if i < config.data_shards {
decoder.add_original_shard(i, black_box(shard)).unwrap();
} else {
let recovery_idx = i - config.data_shards;
decoder.add_recovery_shard(recovery_idx, black_box(shard)).unwrap();
}
simd_group.bench_with_input(
BenchmarkId::new("simd_direct", &config.name),
&(&encoded_shards, &config),
|b, (shards, config)| {
b.iter(|| {
let per_shard_size = calc_shard_size(config.data_size, config.data_shards);
match reed_solomon_simd::ReedSolomonDecoder::new(config.data_shards, config.parity_shards, per_shard_size)
{
Ok(mut decoder) => {
// Add available shards (except lost ones)
for (i, shard) in shards.iter().enumerate() {
if i != config.data_shards - 1 && i != config.data_shards {
if i < config.data_shards {
decoder.add_original_shard(i, black_box(shard)).unwrap();
} else {
let recovery_idx = i - config.data_shards;
decoder.add_recovery_shard(recovery_idx, black_box(shard)).unwrap();
}
}
}
let result = decoder.decode().unwrap();
black_box(result);
}
Err(_) => {
// SIMD不支持此配置，跳过
black_box(());
}
let result = decoder.decode().unwrap();
black_box(result);
}
});
},
);
simd_group.finish();
}
Err(_) => {
// SIMD doesn't support this configuration, skip
black_box(());
}
}
});
},
);
simd_group.finish();
}
}
}
/// 基准测试: 不同分片大小对性能的影响
/// Benchmark: Impact of different shard sizes on performance
fn bench_shard_size_impact(c: &mut Criterion) {
let shard_sizes = vec![64, 128, 256, 512, 1024, 2048, 4096, 8192];
let data_shards = 4;
@@ -301,8 +249,8 @@ fn bench_shard_size_impact(c: &mut Criterion) {
group.throughput(Throughput::Bytes(total_data_size as u64));
// 测试当前实现
group.bench_with_input(BenchmarkId::new("current", format!("shard_{}B", shard_size)), &data, |b, data| {
// Test SIMD implementation
group.bench_with_input(BenchmarkId::new("simd", format!("shard_{}B", shard_size)), &data, |b, data| {
let erasure = Erasure::new(data_shards, parity_shards, total_data_size);
b.iter(|| {
let shards = erasure.encode_data(black_box(data)).unwrap();
@@ -313,19 +261,19 @@ fn bench_shard_size_impact(c: &mut Criterion) {
group.finish();
}
/// 基准测试: 编码配置对性能的影响
/// Benchmark: Impact of coding configurations on performance
fn bench_coding_configurations(c: &mut Criterion) {
let configs = vec![
(2, 1), // 最小冗余
(3, 2), // 中等冗余
(4, 2), // 常用配置
(6, 3), // 50%冗余
(8, 4), // 50%冗余，更多分片
(10, 5), // 50%冗余，大量分片
(12, 6), // 50%冗余，更大量分片
(2, 1), // Minimal redundancy
(3, 2), // Medium redundancy
(4, 2), // Common configuration
(6, 3), // 50% redundancy
(8, 4), // 50% redundancy, more shards
(10, 5), // 50% redundancy, many shards
(12, 6), // 50% redundancy, very many shards
];
let data_size = 1024 * 1024; // 1MB测试数据
let data_size = 1024 * 1024; // 1MB test data
let data = generate_test_data(data_size);
let mut group = c.benchmark_group("coding_configurations");
@@ -347,17 +295,17 @@ fn bench_coding_configurations(c: &mut Criterion) {
group.finish();
}
/// 基准测试: 内存使用模式
/// Benchmark: Memory usage patterns
fn bench_memory_patterns(c: &mut Criterion) {
let data_shards = 4;
let parity_shards = 2;
let block_size = 1024 * 1024; // 1MB块
let block_size = 1024 * 1024; // 1MB block
let mut group = c.benchmark_group("memory_patterns");
group.sample_size(10);
group.measurement_time(Duration::from_secs(5));
// 测试重复使用同一个Erasure实例
// Test reusing the same Erasure instance
group.bench_function("reuse_erasure_instance", |b| {
let erasure = Erasure::new(data_shards, parity_shards, block_size);
let data = generate_test_data(block_size);
@@ -368,7 +316,7 @@ fn bench_memory_patterns(c: &mut Criterion) {
});
});
// 测试每次创建新的Erasure实例
// Test creating new Erasure instance each time
group.bench_function("new_erasure_instance", |b| {
let data = generate_test_data(block_size);
@@ -382,7 +330,7 @@ fn bench_memory_patterns(c: &mut Criterion) {
group.finish();
}
// 基准测试组配置
// Benchmark group configuration
criterion_group!(
benches,
bench_encode_performance,
ecstore/run_benchmarks.sh
+74 -88
View File
@@ -1,54 +1,48 @@
#!/bin/bash
# Reed-Solomon 实现性能比较脚本
#
# 这个脚本将运行不同的基准测试来比较SIMD模式和纯Erasure模式的性能
#
# 使用方法:
# ./run_benchmarks.sh [quick|full|comparison]
#
# quick - 快速测试主要场景
# full - 完整基准测试套件
# comparison - 专门对比两种实现模式
# Reed-Solomon SIMD 性能基准测试脚本
# 使用高性能 SIMD 实现进行纠删码性能测试
set -e
# 颜色输出
# ANSI 颜色码
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
PURPLE='\033[0;35m'
NC='\033[0m' # No Color
# 输出带颜色的信息
# 打印带颜色的消息
print_info() {
echo -e "${BLUE}[INFO]${NC} $1"
echo -e "${BLUE}ℹ️ $1${NC}"
}
print_success() {
echo -e "${GREEN}[SUCCESS]${NC} $1"
echo -e "${GREEN}✅ $1${NC}"
}
print_warning() {
echo -e "${YELLOW}[WARNING]${NC} $1"
echo -e "${YELLOW}⚠️ $1${NC}"
}
print_error() {
echo -e "${RED}[ERROR]${NC} $1"
echo -e "${RED}❌ $1${NC}"
}
# 检查是否安装了必要工具
# 检查系统要求
check_requirements() {
print_info "检查系统要求..."
# 检查 Rust
if ! command -v cargo &> /dev/null; then
print_error "cargo 未安装，请先安装 Rust 工具链"
print_error "Cargo 未找到，请确保已安装 Rust"
exit 1
fi
# 检查是否安装了 criterion
if ! grep -q "criterion" Cargo.toml; then
print_error "Cargo.toml 中未找到 criterion 依赖"
# 检查 criterion
if ! cargo --list | grep -q "bench"; then
print_error "未找到基准测试支持，请确保使用的是支持基准测试的 Rust 版本"
exit 1
fi
@@ -62,28 +56,15 @@ cleanup() {
print_success "清理完成"
}
# 运行纯 Erasure 模式基准测试
run_erasure_benchmark() {
print_info "🏛️ 开始运行纯 Erasure 模式基准测试..."
echo "================================================"
cargo bench --bench comparison_benchmark \
--features reed-solomon-erasure \
-- --save-baseline erasure_baseline
print_success "纯 Erasure 模式基准测试完成"
}
# 运行SIMD模式基准测试
# 运行 SIMD 模式基准测试
run_simd_benchmark() {
print_info "🎯 开始运行SIMD模式基准测试..."
print_info "🎯 开始运行 SIMD 模式基准测试..."
echo "================================================"
cargo bench --bench comparison_benchmark \
--features reed-solomon-simd \
-- --save-baseline simd_baseline
print_success "SIMD模式基准测试完成"
print_success "SIMD 模式基准测试完成"
}
# 运行完整的基准测试套件
@@ -91,33 +72,42 @@ run_full_benchmark() {
print_info "🚀 开始运行完整基准测试套件..."
echo "================================================"
# 运行详细的基准测试（使用默认纯Erasure模式）
# 运行详细的基准测试
cargo bench --bench erasure_benchmark
print_success "完整基准测试套件完成"
}
# 运行性能对比测试
run_comparison_benchmark() {
print_info "📊 开始运行性能对比测试..."
# 运行性能测试
run_performance_test() {
print_info "📊 开始运行性能测试..."
echo "================================================"
print_info "步骤 1: 测试纯 Erasure 模式..."
print_info "步骤 1: 运行编码基准测试..."
cargo bench --bench comparison_benchmark \
--features reed-solomon-erasure \
-- --save-baseline erasure_baseline
-- encode --save-baseline encode_baseline
print_info "步骤 2: 测试SIMD模式并与 Erasure 模式对比..."
print_info "步骤 2: 运行解码基准测试..."
cargo bench --bench comparison_benchmark \
--features reed-solomon-simd \
-- --baseline erasure_baseline
-- decode --save-baseline decode_baseline
print_success "性能对比测试完成"
print_success "性能测试完成"
}
# 运行大数据集测试
run_large_data_test() {
print_info "🗂️ 开始运行大数据集测试..."
echo "================================================"
cargo bench --bench erasure_benchmark \
-- large_data --save-baseline large_data_baseline
print_success "大数据集测试完成"
}
# 生成比较报告
generate_comparison_report() {
print_info "📊 生成性能比较报告..."
print_info "📊 生成性能报告..."
if [ -d "target/criterion" ]; then
print_info "基准测试结果已保存到 target/criterion/ 目录"
@@ -138,49 +128,48 @@ generate_comparison_report() {
run_quick_test() {
print_info "🏃 运行快速性能测试..."
print_info "测试纯 Erasure 模式..."
print_info "测试 SIMD 编码性能..."
cargo bench --bench comparison_benchmark \
--features reed-solomon-erasure \
-- encode_comparison --quick
-- encode --quick
print_info "测试SIMD模式..."
print_info "测试 SIMD 解码性能..."
cargo bench --bench comparison_benchmark \
--features reed-solomon-simd \
-- encode_comparison --quick
-- decode --quick
print_success "快速测试完成"
}
# 显示帮助信息
show_help() {
echo "Reed-Solomon 性能基准测试脚本"
echo "Reed-Solomon SIMD 性能基准测试脚本"
echo ""
echo "实现模式："
echo " 🏛️ 纯 Erasure 模式（默认）- 稳定兼容的 reed-solomon-erasure 实现"
echo " 🎯 SIMD模式 - 高性能SIMD优化实现"
echo " 🎯 SIMD 模式 - 高性能 SIMD 优化的 reed-solomon-simd 实现"
echo ""
echo "使用方法:"
echo " $0 [command]"
echo ""
echo "命令:"
echo " quick 运行快速性能测试"
echo " full 运行完整基准测试套件（默认Erasure模式）"
echo " comparison 运行详细的实现模式对比测试"
echo " erasure 只测试纯 Erasure 模式"
echo " simd 只测试SIMD模式"
echo " full 运行完整基准测试套件"
echo " performance 运行详细的性能测试"
echo " simd 运行 SIMD 模式测试"
echo " large 运行大数据集测试"
echo " clean 清理测试结果"
echo " help 显示此帮助信息"
echo ""
echo "示例:"
echo " $0 quick # 快速测试两种模式"
echo " $0 comparison # 详细对比测试"
echo " $0 full # 完整测试套件（默认Erasure模式）"
echo " $0 simd # 只测试SIMD模式"
echo " $0 erasure # 只测试纯 Erasure 模式"
echo " $0 quick # 快速性能测试"
echo " $0 performance # 详细性能测试"
echo " $0 full # 完整测试套件"
echo " $0 simd # SIMD 模式测试"
echo " $0 large # 大数据集测试"
echo ""
echo "模式说明:"
echo " Erasure模式: 使用reed-solomon-erasure实现，稳定可靠"
echo " SIMD模式: 使用reed-solomon-simd实现，高性能优化"
echo "实现特性:"
echo " - 使用 reed-solomon-simd 高性能 SIMD 实现"
echo " - 支持编码器/解码器实例缓存"
echo " - 优化的内存管理和线程安全"
echo " - 跨平台 SIMD 指令支持"
}
# 显示测试配置信息
@@ -196,22 +185,22 @@ show_test_info() {
if [ -f "/proc/cpuinfo" ]; then
echo " - CPU 型号: $(grep 'model name' /proc/cpuinfo | head -1 | cut -d: -f2 | xargs)"
if grep -q "avx2" /proc/cpuinfo; then
echo " - SIMD 支持: AVX2 ✅ (SIMD模式将利用SIMD优化)"
echo " - SIMD 支持: AVX2 ✅ (将使用高级 SIMD 优化)"
elif grep -q "sse4" /proc/cpuinfo; then
echo " - SIMD 支持: SSE4 ✅ (SIMD模式将利用SIMD优化)"
echo " - SIMD 支持: SSE4 ✅ (将使用 SIMD 优化)"
else
echo " - SIMD 支持: 未检测到高级 SIMD 特性"
echo " - SIMD 支持: 基础 SIMD 特性"
fi
fi
echo " - 默认模式: 纯Erasure模式 (稳定可靠)"
echo " - 高性能模式: SIMD模式 (性能优化)"
echo " - 实现: reed-solomon-simd (高性能 SIMD 优化)"
echo " - 特性: 实例缓存、线程安全、跨平台 SIMD"
echo ""
}
# 主函数
main() {
print_info "🧪 Reed-Solomon 实现性能基准测试"
print_info "🧪 Reed-Solomon SIMD 实现性能基准测试"
echo "================================================"
check_requirements
@@ -227,14 +216,9 @@ main() {
run_full_benchmark
generate_comparison_report
;;
"comparison")
"performance")
cleanup
run_comparison_benchmark
generate_comparison_report
;;
"erasure")
cleanup
run_erasure_benchmark
run_performance_test
generate_comparison_report
;;
"simd")
@@ -242,6 +226,11 @@ main() {
run_simd_benchmark
generate_comparison_report
;;
"large")
cleanup
run_large_data_test
generate_comparison_report
;;
"clean")
cleanup
;;
@@ -257,10 +246,7 @@ main() {
esac
print_success "✨ 基准测试执行完成!"
print_info "💡 提示: 推荐使用默认的纯Erasure模式，对于高性能需求可考虑SIMD模式"
}
# 如果直接运行此脚本，调用主函数
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi
# 启动脚本
main "$@"
ecstore/src/admin_server_info.rs
+1 -1
View File
@@ -93,7 +93,7 @@ async fn is_server_resolvable(endpoint: &Endpoint) -> Result<()> {
// 构造 PingRequest
let request = Request::new(PingRequest {
version: 1,
body: finished_data.to_vec(),
body: bytes::Bytes::copy_from_slice(finished_data),
});
// 发送请求并获取响应
ecstore/src/bitrot.rs
+9 -3
View File
@@ -28,7 +28,7 @@ pub async fn create_bitrot_reader(
checksum_algo: HashAlgorithm,
) -> disk::error::Result<Option<BitrotReader<Box<dyn AsyncRead + Send + Sync + Unpin>>>> {
// Calculate the total length to read, including the checksum overhead
let length = offset.div_ceil(shard_size) * checksum_algo.size() + length;
let length = length.div_ceil(shard_size) * checksum_algo.size() + length;
if let Some(data) = inline_data {
// Use inline data
@@ -68,14 +68,20 @@ pub async fn create_bitrot_writer(
disk: Option<&DiskStore>,
volume: &str,
path: &str,
length: usize,
length: i64,
shard_size: usize,
checksum_algo: HashAlgorithm,
) -> disk::error::Result<BitrotWriterWrapper> {
let writer = if is_inline_buffer {
CustomWriter::new_inline_buffer()
} else if let Some(disk) = disk {
let length = length.div_ceil(shard_size) * checksum_algo.size() + length;
let length = if length > 0 {
let length = length as usize;
(length.div_ceil(shard_size) * checksum_algo.size() + length) as i64
} else {
0
};
let file = disk.create_file("", volume, path, length).await?;
CustomWriter::new_tokio_writer(file)
} else {
ecstore/src/bucket/metadata.rs
+1 -1
View File
@@ -45,7 +45,7 @@ pub const BUCKET_TARGETS_FILE: &str = "bucket-targets.json";
pub struct BucketMetadata {
pub name: String,
pub created: OffsetDateTime,
pub lock_enabled: bool, // 虽然标记为不使用，但可能需要保留
pub lock_enabled: bool, // While marked as unused, it may need to be retained
pub policy_config_json: Vec<u8>,
pub notification_config_xml: Vec<u8>,
pub lifecycle_config_xml: Vec<u8>,
ecstore/src/bucket/metadata_sys.rs
-1
View File
@@ -443,7 +443,6 @@ impl BucketMetadataSys {
let bm = match self.get_config(bucket).await {
Ok((res, _)) => res,
Err(err) => {
warn!("get_object_lock_config err {:?}", &err);
return if err == Error::ConfigNotFound {
Err(BucketMetadataError::BucketObjectLockConfigNotFound.into())
} else {
ecstore/src/cache_value/metacache_set.rs
+53 -7
View File
@@ -1,7 +1,7 @@
use crate::disk::error::DiskError;
use crate::disk::{self, DiskAPI, DiskStore, WalkDirOptions};
use futures::future::join_all;
use rustfs_filemeta::{MetaCacheEntries, MetaCacheEntry, MetacacheReader};
use rustfs_filemeta::{MetaCacheEntries, MetaCacheEntry, MetacacheReader, is_io_eof};
use std::{future::Future, pin::Pin, sync::Arc};
use tokio::{spawn, sync::broadcast::Receiver as B_Receiver};
use tracing::error;
@@ -50,7 +50,6 @@ impl Clone for ListPathRawOptions {
}
pub async fn list_path_raw(mut rx: B_Receiver<bool>, opts: ListPathRawOptions) -> disk::error::Result<()> {
// println!("list_path_raw {},{}", &opts.bucket, &opts.path);
if opts.disks.is_empty() {
return Err(DiskError::other("list_path_raw: 0 drives provided"));
}
@@ -59,12 +58,13 @@ pub async fn list_path_raw(mut rx: B_Receiver<bool>, opts: ListPathRawOptions) -
let mut readers = Vec::with_capacity(opts.disks.len());
let fds = Arc::new(opts.fallback_disks.clone());
let (cancel_tx, cancel_rx) = tokio::sync::broadcast::channel::<bool>(1);
for disk in opts.disks.iter() {
let opdisk = disk.clone();
let opts_clone = opts.clone();
let fds_clone = fds.clone();
// let (m_tx, m_rx) = mpsc::channel::<MetaCacheEntry>(100);
// readers.push(m_rx);
let mut cancel_rx_clone = cancel_rx.resubscribe();
let (rd, mut wr) = tokio::io::duplex(64);
readers.push(MetacacheReader::new(rd));
jobs.push(spawn(async move {
@@ -92,7 +92,13 @@ pub async fn list_path_raw(mut rx: B_Receiver<bool>, opts: ListPathRawOptions) -
need_fallback = true;
}
if cancel_rx_clone.try_recv().is_ok() {
// warn!("list_path_raw: cancel_rx_clone.try_recv().await.is_ok()");
return Ok(());
}
while need_fallback {
// warn!("list_path_raw: while need_fallback start");
let disk = match fds_clone.iter().find(|d| d.is_some()) {
Some(d) => {
if let Some(disk) = d.clone() {
@@ -130,6 +136,7 @@ pub async fn list_path_raw(mut rx: B_Receiver<bool>, opts: ListPathRawOptions) -
}
}
// warn!("list_path_raw: while need_fallback done");
Ok(())
}));
}
@@ -143,9 +150,15 @@ pub async fn list_path_raw(mut rx: B_Receiver<bool>, opts: ListPathRawOptions) -
loop {
let mut current = MetaCacheEntry::default();
// warn!(
// "list_path_raw: loop start, bucket: {}, path: {}, current: {:?}",
// opts.bucket, opts.path, &current.name
// );
if rx.try_recv().is_ok() {
return Err(DiskError::other("canceled"));
}
let mut top_entries: Vec<Option<MetaCacheEntry>> = vec![None; readers.len()];
let mut at_eof = 0;
@@ -168,31 +181,47 @@ pub async fn list_path_raw(mut rx: B_Receiver<bool>, opts: ListPathRawOptions) -
} else {
// eof
at_eof += 1;
// warn!("list_path_raw: peek eof, disk: {}", i);
continue;
}
}
Err(err) => {
if err == rustfs_filemeta::Error::Unexpected {
at_eof += 1;
// warn!("list_path_raw: peek err eof, disk: {}", i);
continue;
} else if err == rustfs_filemeta::Error::FileNotFound {
}
// warn!("list_path_raw: peek err00, err: {:?}", err);
if is_io_eof(&err) {
at_eof += 1;
// warn!("list_path_raw: peek eof, disk: {}", i);
continue;
}
if err == rustfs_filemeta::Error::FileNotFound {
at_eof += 1;
fnf += 1;
// warn!("list_path_raw: peek fnf, disk: {}", i);
continue;
} else if err == rustfs_filemeta::Error::VolumeNotFound {
at_eof += 1;
fnf += 1;
vnf += 1;
// warn!("list_path_raw: peek vnf, disk: {}", i);
continue;
} else {
has_err += 1;
errs[i] = Some(err.into());
// warn!("list_path_raw: peek err, disk: {}", i);
continue;
}
}
};
// warn!("list_path_raw: loop entry: {:?}, disk: {}", &entry.name, i);
// If no current, add it.
if current.name.is_empty() {
top_entries[i] = Some(entry.clone());
@@ -228,10 +257,12 @@ pub async fn list_path_raw(mut rx: B_Receiver<bool>, opts: ListPathRawOptions) -
}
if vnf > 0 && vnf >= (readers.len() - opts.min_disks) {
// warn!("list_path_raw: vnf > 0 && vnf >= (readers.len() - opts.min_disks) break");
return Err(DiskError::VolumeNotFound);
}
if fnf > 0 && fnf >= (readers.len() - opts.min_disks) {
// warn!("list_path_raw: fnf > 0 && fnf >= (readers.len() - opts.min_disks) break");
return Err(DiskError::FileNotFound);
}
@@ -250,6 +281,10 @@ pub async fn list_path_raw(mut rx: B_Receiver<bool>, opts: ListPathRawOptions) -
_ => {}
});
error!(
"list_path_raw: has_err > 0 && has_err > opts.disks.len() - opts.min_disks break, err: {:?}",
&combined_err.join(", ")
);
return Err(DiskError::other(combined_err.join(", ")));
}
@@ -263,6 +298,7 @@ pub async fn list_path_raw(mut rx: B_Receiver<bool>, opts: ListPathRawOptions) -
}
}
// error!("list_path_raw: at_eof + has_err == readers.len() break {:?}", &errs);
break;
}
@@ -272,12 +308,16 @@ pub async fn list_path_raw(mut rx: B_Receiver<bool>, opts: ListPathRawOptions) -
}
if let Some(agreed_fn) = opts.agreed.as_ref() {
// warn!("list_path_raw: agreed_fn start, current: {:?}", &current.name);
agreed_fn(current).await;
// warn!("list_path_raw: agreed_fn done");
}
continue;
}
// warn!("list_path_raw: skip start, current: {:?}", &current.name);
for (i, r) in readers.iter_mut().enumerate() {
if top_entries[i].is_some() {
let _ = r.skip(1).await;
@@ -291,7 +331,12 @@ pub async fn list_path_raw(mut rx: B_Receiver<bool>, opts: ListPathRawOptions) -
Ok(())
});
jobs.push(revjob);
if let Err(err) = revjob.await.map_err(std::io::Error::other)? {
error!("list_path_raw: revjob err {:?}", err);
let _ = cancel_tx.send(true);
return Err(err);
}
let results = join_all(jobs).await;
for result in results {
@@ -300,5 +345,6 @@ pub async fn list_path_raw(mut rx: B_Receiver<bool>, opts: ListPathRawOptions) -
}
}
// warn!("list_path_raw: done");
Ok(())
}
ecstore/src/cmd/bucket_replication.rs
+12 -12
View File
@@ -6,7 +6,7 @@ use crate::bucket::metadata_sys::get_replication_config;
use crate::bucket::versioning_sys::BucketVersioningSys;
use crate::error::Error;
use crate::new_object_layer_fn;
use crate::peer::RemotePeerS3Client;
use crate::rpc::RemotePeerS3Client;
use crate::store;
use crate::store_api::ObjectIO;
use crate::store_api::ObjectInfo;
@@ -26,8 +26,6 @@ use futures::stream::FuturesUnordered;
use http::HeaderMap;
use http::Method;
use lazy_static::lazy_static;
use std::str::FromStr;
use std::sync::Arc;
// use std::time::SystemTime;
use once_cell::sync::Lazy;
use regex::Regex;
@@ -44,6 +42,8 @@ use std::collections::HashMap;
use std::collections::HashSet;
use std::fmt;
use std::iter::Iterator;
use std::str::FromStr;
use std::sync::Arc;
use std::sync::atomic::AtomicI32;
use std::sync::atomic::Ordering;
use std::vec;
@@ -512,8 +512,8 @@ pub async fn get_heal_replicate_object_info(
let mut result = ReplicateObjectInfo {
name: oi.name.clone(),
size: oi.size as i64,
actual_size: asz as i64,
size: oi.size,
actual_size: asz,
bucket: oi.bucket.clone(),
//version_id: oi.version_id.clone(),
version_id: oi
@@ -815,8 +815,8 @@ impl ReplicationPool {
vsender.pop(); // Dropping the sender will close the channel
}
self.workers_sender = vsender;
warn!("self sender size is {:?}", self.workers_sender.len());
warn!("self sender size is {:?}", self.workers_sender.len());
// warn!("self sender size is {:?}", self.workers_sender.len());
// warn!("self sender size is {:?}", self.workers_sender.len());
}
async fn resize_failed_workers(&self, _count: usize) {
@@ -1759,13 +1759,13 @@ pub async fn schedule_replication(oi: ObjectInfo, o: Arc<store::ECStore>, dsc: R
let replication_timestamp = Utc::now(); // Placeholder for timestamp parsing
let replication_state = oi.replication_state();
let actual_size = oi.actual_size.unwrap_or(0);
let actual_size = oi.actual_size;
//let ssec = oi.user_defined.contains_key("ssec");
let ssec = false;
let ri = ReplicateObjectInfo {
name: oi.name,
size: oi.size as i64,
size: oi.size,
bucket: oi.bucket,
version_id: oi
.version_id
@@ -2019,8 +2019,8 @@ impl ReplicateObjectInfo {
mod_time: Some(
OffsetDateTime::from_unix_timestamp(self.mod_time.timestamp()).unwrap_or_else(|_| OffsetDateTime::now_utc()),
),
size: self.size as usize,
actual_size: Some(self.actual_size as usize),
size: self.size,
actual_size: self.actual_size,
is_dir: false,
user_defined: None, // 可以按需从别处导入
parity_blocks: 0,
@@ -2319,7 +2319,7 @@ impl ReplicateObjectInfo {
// 设置对象大小
//rinfo.size = object_info.actual_size.unwrap_or(0);
rinfo.size = object_info.actual_size.map_or(0, |v| v as i64);
rinfo.size = object_info.actual_size;
//rinfo.replication_action = object_info.
rinfo.replication_status = ReplicationStatusType::Completed;
ecstore/src/cmd/bucket_targets.rs
+4 -4
View File
@@ -4,11 +4,11 @@ use crate::{
StorageAPI,
bucket::{metadata_sys, target::BucketTarget},
endpoints::Node,
peer::{PeerS3Client, RemotePeerS3Client},
rpc::{PeerS3Client, RemotePeerS3Client},
};
use crate::{
bucket::{self, target::BucketTargets},
new_object_layer_fn, peer, store_api,
new_object_layer_fn, store_api,
};
//use tokio::sync::RwLock;
use aws_sdk_s3::Client as S3Client;
@@ -24,7 +24,7 @@ use tokio::sync::RwLock;
pub struct TClient {
pub s3cli: S3Client,
pub remote_peer_client: peer::RemotePeerS3Client,
pub remote_peer_client: RemotePeerS3Client,
pub arn: String,
}
impl TClient {
@@ -444,7 +444,7 @@ impl BucketTargetSys {
grid_host: "".to_string(),
};
let cli = peer::RemotePeerS3Client::new(Some(node), None);
let cli = RemotePeerS3Client::new(Some(node), None);
match cli
.get_bucket_info(&tgt.target_bucket, &store_api::BucketOptions::default())
ecstore/src/compress.rs
+115
View File
@@ -0,0 +1,115 @@
use rustfs_utils::string::has_pattern;
use rustfs_utils::string::has_string_suffix_in_slice;
use std::env;
use tracing::error;
pub const MIN_COMPRESSIBLE_SIZE: usize = 4096;
// 环境变量名称，用于控制是否启用压缩
pub const ENV_COMPRESSION_ENABLED: &str = "RUSTFS_COMPRESSION_ENABLED";
// Some standard object extensions which we strictly dis-allow for compression.
pub const STANDARD_EXCLUDE_COMPRESS_EXTENSIONS: &[&str] = &[
".gz", ".bz2", ".rar", ".zip", ".7z", ".xz", ".mp4", ".mkv", ".mov", ".jpg", ".png", ".gif",
];
// Some standard content-types which we strictly dis-allow for compression.
pub const STANDARD_EXCLUDE_COMPRESS_CONTENT_TYPES: &[&str] = &[
"video/*",
"audio/*",
"application/zip",
"application/x-gzip",
"application/x-zip-compressed",
"application/x-compress",
"application/x-spoon",
];
pub fn is_compressible(headers: &http::HeaderMap, object_name: &str) -> bool {
// 检查环境变量是否启用压缩，默认关闭
if let Ok(compression_enabled) = env::var(ENV_COMPRESSION_ENABLED) {
if compression_enabled.to_lowercase() != "true" {
error!("Compression is disabled by environment variable");
return false;
}
} else {
// 环境变量未设置时默认关闭
return false;
}
let content_type = headers.get("content-type").and_then(|s| s.to_str().ok()).unwrap_or("");
// TODO: crypto request return false
if has_string_suffix_in_slice(object_name, STANDARD_EXCLUDE_COMPRESS_EXTENSIONS) {
error!("object_name: {} is not compressible", object_name);
return false;
}
if !content_type.is_empty() && has_pattern(STANDARD_EXCLUDE_COMPRESS_CONTENT_TYPES, content_type) {
error!("content_type: {} is not compressible", content_type);
return false;
}
true
// TODO: check from config
}
#[cfg(test)]
mod tests {
use super::*;
use temp_env;
#[test]
fn test_is_compressible() {
use http::HeaderMap;
let headers = HeaderMap::new();
// 测试环境变量控制
temp_env::with_var(ENV_COMPRESSION_ENABLED, Some("false"), || {
assert!(!is_compressible(&headers, "file.txt"));
});
temp_env::with_var(ENV_COMPRESSION_ENABLED, Some("true"), || {
assert!(is_compressible(&headers, "file.txt"));
});
temp_env::with_var_unset(ENV_COMPRESSION_ENABLED, || {
assert!(!is_compressible(&headers, "file.txt"));
});
temp_env::with_var(ENV_COMPRESSION_ENABLED, Some("true"), || {
let mut headers = HeaderMap::new();
// 测试不可压缩的扩展名
headers.insert("content-type", "text/plain".parse().unwrap());
assert!(!is_compressible(&headers, "file.gz"));
assert!(!is_compressible(&headers, "file.zip"));
assert!(!is_compressible(&headers, "file.mp4"));
assert!(!is_compressible(&headers, "file.jpg"));
// 测试不可压缩的内容类型
headers.insert("content-type", "video/mp4".parse().unwrap());
assert!(!is_compressible(&headers, "file.txt"));
headers.insert("content-type", "audio/mpeg".parse().unwrap());
assert!(!is_compressible(&headers, "file.txt"));
headers.insert("content-type", "application/zip".parse().unwrap());
assert!(!is_compressible(&headers, "file.txt"));
headers.insert("content-type", "application/x-gzip".parse().unwrap());
assert!(!is_compressible(&headers, "file.txt"));
// 测试可压缩的情况
headers.insert("content-type", "text/plain".parse().unwrap());
assert!(is_compressible(&headers, "file.txt"));
assert!(is_compressible(&headers, "file.log"));
headers.insert("content-type", "text/html".parse().unwrap());
assert!(is_compressible(&headers, "file.html"));
headers.insert("content-type", "application/json".parse().unwrap());
assert!(is_compressible(&headers, "file.json"));
});
}
}
ecstore/src/config/com.rs
+51 -43
View File
@@ -41,6 +41,7 @@ pub async fn read_config_with_metadata<S: StorageAPI>(
if err == Error::FileNotFound || matches!(err, Error::ObjectNotFound(_, _)) {
Error::ConfigNotFound
} else {
warn!("read_config_with_metadata: err: {:?}, file: {}", err, file);
err
}
})?;
@@ -92,9 +93,13 @@ pub async fn delete_config<S: StorageAPI>(api: Arc<S>, file: &str) -> Result<()>
}
pub async fn save_config_with_opts<S: StorageAPI>(api: Arc<S>, file: &str, data: Vec<u8>, opts: &ObjectOptions) -> Result<()> {
let _ = api
if let Err(err) = api
.put_object(RUSTFS_META_BUCKET, file, &mut PutObjReader::from_vec(data), opts)
.await?;
.await
{
error!("save_config_with_opts: err: {:?}, file: {}", err, file);
return Err(err);
}
Ok(())
}
@@ -110,59 +115,62 @@ async fn new_and_save_server_config<S: StorageAPI>(api: Arc<S>) -> Result<Config
Ok(cfg)
}
pub async fn read_config_without_migrate<S: StorageAPI>(api: Arc<S>) -> Result<Config> {
let config_file = format!("{}{}{}", CONFIG_PREFIX, SLASH_SEPARATOR, CONFIG_FILE);
let data = match read_config(api.clone(), config_file.as_str()).await {
Ok(res) => res,
Err(err) => {
return if err == Error::ConfigNotFound {
warn!("config not found, start to init");
let cfg = new_and_save_server_config(api).await?;
warn!("config init done");
Ok(cfg)
} else {
error!("read config err {:?}", &err);
Err(err)
};
}
};
fn get_config_file() -> String {
format!("{}{}{}", CONFIG_PREFIX, SLASH_SEPARATOR, CONFIG_FILE)
}
read_server_config(api, data.as_slice()).await
/// Handle the situation where the configuration file does not exist, create and save a new configuration
async fn handle_missing_config<S: StorageAPI>(api: Arc<S>, context: &str) -> Result<Config> {
warn!("Configuration not found ({}): Start initializing new configuration", context);
let cfg = new_and_save_server_config(api).await?;
warn!("Configuration initialization complete ({})", context);
Ok(cfg)
}
/// Handle configuration file read errors
fn handle_config_read_error(err: Error, file_path: &str) -> Result<Config> {
error!("Read configuration failed (path: '{}'): {:?}", file_path, err);
Err(err)
}
pub async fn read_config_without_migrate<S: StorageAPI>(api: Arc<S>) -> Result<Config> {
let config_file = get_config_file();
// Try to read the configuration file
match read_config(api.clone(), &config_file).await {
Ok(data) => read_server_config(api, &data).await,
Err(Error::ConfigNotFound) => handle_missing_config(api, "Read the main configuration").await,
Err(err) => handle_config_read_error(err, &config_file),
}
}
async fn read_server_config<S: StorageAPI>(api: Arc<S>, data: &[u8]) -> Result<Config> {
let cfg = {
if data.is_empty() {
let config_file = format!("{}{}{}", CONFIG_PREFIX, SLASH_SEPARATOR, CONFIG_FILE);
let cfg_data = match read_config(api.clone(), config_file.as_str()).await {
Ok(res) => res,
Err(err) => {
return if err == Error::ConfigNotFound {
warn!("config not found init start");
let cfg = new_and_save_server_config(api).await?;
warn!("config not found init done");
Ok(cfg)
} else {
error!("read config err {:?}", &err);
Err(err)
};
}
};
// TODO: decrypt
// If the provided data is empty, try to read from the file again
if data.is_empty() {
let config_file = get_config_file();
warn!("Received empty configuration data, try to reread from '{}'", config_file);
Config::unmarshal(cfg_data.as_slice())?
} else {
Config::unmarshal(data)?
// Try to read the configuration again
match read_config(api.clone(), &config_file).await {
Ok(cfg_data) => {
// TODO: decrypt
let cfg = Config::unmarshal(&cfg_data)?;
return Ok(cfg.merge());
}
Err(Error::ConfigNotFound) => return handle_missing_config(api, "Read alternate configuration").await,
Err(err) => return handle_config_read_error(err, &config_file),
}
};
}
// Process non-empty configuration data
let cfg = Config::unmarshal(data)?;
Ok(cfg.merge())
}
async fn save_server_config<S: StorageAPI>(api: Arc<S>, cfg: &Config) -> Result<()> {
pub async fn save_server_config<S: StorageAPI>(api: Arc<S>, cfg: &Config) -> Result<()> {
let data = cfg.marshal()?;
let config_file = format!("{}{}{}", CONFIG_PREFIX, SLASH_SEPARATOR, CONFIG_FILE);
let config_file = get_config_file();
save_config(api, &config_file, data).await
}
ecstore/src/config/mod.rs
+12 -4
View File
@@ -18,6 +18,14 @@ lazy_static! {
pub static ref GLOBAL_ConfigSys: ConfigSys = ConfigSys::new();
}
/// Standard config keys and values.
pub const ENABLE_KEY: &str = "enable";
pub const COMMENT_KEY: &str = "comment";
/// Enable values
pub const ENABLE_ON: &str = "on";
pub const ENABLE_OFF: &str = "off";
pub const ENV_ACCESS_KEY: &str = "RUSTFS_ACCESS_KEY";
pub const ENV_SECRET_KEY: &str = "RUSTFS_SECRET_KEY";
pub const ENV_ROOT_USER: &str = "RUSTFS_ROOT_USER";
@@ -56,7 +64,7 @@ pub struct KV {
}
#[derive(Debug, Deserialize, Serialize, Clone)]
pub struct KVS(Vec<KV>);
pub struct KVS(pub Vec<KV>);
impl Default for KVS {
fn default() -> Self {
@@ -83,7 +91,7 @@ impl KVS {
}
#[derive(Debug, Clone)]
pub struct Config(HashMap<String, HashMap<String, KVS>>);
pub struct Config(pub HashMap<String, HashMap<String, KVS>>);
impl Default for Config {
fn default() -> Self {
@@ -99,8 +107,8 @@ impl Config {
cfg
}
pub fn get_value(&self, subsys: &str, key: &str) -> Option<KVS> {
if let Some(m) = self.0.get(subsys) {
pub fn get_value(&self, sub_sys: &str, key: &str) -> Option<KVS> {
if let Some(m) = self.0.get(sub_sys) {
m.get(key).cloned()
} else {
None
ecstore/src/config/storageclass.rs
+9 -2
View File
@@ -6,7 +6,8 @@ use serde::{Deserialize, Serialize};
use std::env;
use tracing::warn;
// default_parity_count 默认配置，根据磁盘总数分配校验磁盘数量
/// Default parity count for a given drive count
/// The default configuration allocates the number of check disks based on the total number of disks
pub fn default_parity_count(drive: usize) -> usize {
match drive {
1 => 0,
@@ -112,7 +113,13 @@ impl Config {
}
}
pub fn should_inline(&self, shard_size: usize, versioned: bool) -> bool {
pub fn should_inline(&self, shard_size: i64, versioned: bool) -> bool {
if shard_size < 0 {
return false;
}
let shard_size = shard_size as usize;
let mut inline_block = DEFAULT_INLINE_BLOCK;
if self.initialized {
inline_block = self.inline_block;
ecstore/src/disk/error.rs
+1 -1
View File
@@ -124,7 +124,7 @@ pub enum DiskError {
#[error("erasure read quorum")]
ErasureReadQuorum,
#[error("io error")]
#[error("io error {0}")]
Io(io::Error),
}
ecstore/src/disk/fs.rs
+21 -15
View File
@@ -109,7 +109,7 @@ pub async fn access(path: impl AsRef<Path>) -> io::Result<()> {
}
pub fn access_std(path: impl AsRef<Path>) -> io::Result<()> {
std::fs::metadata(path)?;
tokio::task::block_in_place(|| std::fs::metadata(path))?;
Ok(())
}
@@ -118,7 +118,7 @@ pub async fn lstat(path: impl AsRef<Path>) -> io::Result<Metadata> {
}
pub fn lstat_std(path: impl AsRef<Path>) -> io::Result<Metadata> {
std::fs::metadata(path)
tokio::task::block_in_place(|| std::fs::metadata(path))
}
pub async fn make_dir_all(path: impl AsRef<Path>) -> io::Result<()> {
@@ -146,21 +146,27 @@ pub async fn remove_all(path: impl AsRef<Path>) -> io::Result<()> {
#[tracing::instrument(level = "debug", skip_all)]
pub fn remove_std(path: impl AsRef<Path>) -> io::Result<()> {
let meta = std::fs::metadata(path.as_ref())?;
if meta.is_dir() {
std::fs::remove_dir(path.as_ref())
} else {
std::fs::remove_file(path.as_ref())
}
let path = path.as_ref();
tokio::task::block_in_place(|| {
let meta = std::fs::metadata(path)?;
if meta.is_dir() {
std::fs::remove_dir(path)
} else {
std::fs::remove_file(path)
}
})
}
pub fn remove_all_std(path: impl AsRef<Path>) -> io::Result<()> {
let meta = std::fs::metadata(path.as_ref())?;
if meta.is_dir() {
std::fs::remove_dir_all(path.as_ref())
} else {
std::fs::remove_file(path.as_ref())
}
let path = path.as_ref();
tokio::task::block_in_place(|| {
let meta = std::fs::metadata(path)?;
if meta.is_dir() {
std::fs::remove_dir_all(path)
} else {
std::fs::remove_file(path)
}
})
}
pub async fn mkdir(path: impl AsRef<Path>) -> io::Result<()> {
@@ -172,7 +178,7 @@ pub async fn rename(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<
}
pub fn rename_std(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<()> {
std::fs::rename(from, to)
tokio::task::block_in_place(|| std::fs::rename(from, to))
}
#[tracing::instrument(level = "debug", skip_all)]
ecstore/src/disk/local.rs
+81 -57
View File
@@ -38,6 +38,7 @@ use rustfs_utils::path::{
};
use crate::erasure_coding::bitrot_verify;
use bytes::Bytes;
use common::defer;
use path_absolutize::Absolutize;
use rustfs_filemeta::{
@@ -67,7 +68,7 @@ use uuid::Uuid;
#[derive(Debug)]
pub struct FormatInfo {
pub id: Option<Uuid>,
pub data: Vec<u8>,
pub data: Bytes,
pub file_info: Option<Metadata>,
pub last_check: Option<OffsetDateTime>,
}
@@ -82,6 +83,12 @@ impl FormatInfo {
}
}
/// A helper enum to handle internal buffer types for writing data.
pub enum InternalBuf<'a> {
Ref(&'a [u8]),
Owned(Bytes),
}
pub struct LocalDisk {
pub root: PathBuf,
pub format_path: PathBuf,
@@ -131,7 +138,7 @@ impl LocalDisk {
let mut format_last_check = None;
if !format_data.is_empty() {
let s = format_data.as_slice();
let s = format_data.as_ref();
let fm = FormatV3::try_from(s).map_err(Error::other)?;
let (set_idx, disk_idx) = fm.find_disk_index_by_disk_id(fm.erasure.this)?;
@@ -595,8 +602,14 @@ impl LocalDisk {
let volume_dir = self.get_bucket_path(volume)?;
self.write_all_private(volume, format!("{}/{}", path, STORAGE_FORMAT_FILE).as_str(), &buf, true, volume_dir)
.await?;
self.write_all_private(
volume,
format!("{}/{}", path, STORAGE_FORMAT_FILE).as_str(),
buf.into(),
true,
&volume_dir,
)
.await?;
Ok(())
}
@@ -609,13 +622,14 @@ impl LocalDisk {
let tmp_volume_dir = self.get_bucket_path(super::RUSTFS_META_TMP_BUCKET)?;
let tmp_file_path = tmp_volume_dir.join(Path::new(Uuid::new_v4().to_string().as_str()));
self.write_all_internal(&tmp_file_path, buf, sync, tmp_volume_dir).await?;
self.write_all_internal(&tmp_file_path, InternalBuf::Ref(buf), sync, &tmp_volume_dir)
.await?;
rename_all(tmp_file_path, file_path, volume_dir).await
}
// write_all_public for trail
async fn write_all_public(&self, volume: &str, path: &str, data: Vec<u8>) -> Result<()> {
async fn write_all_public(&self, volume: &str, path: &str, data: Bytes) -> Result<()> {
if volume == RUSTFS_META_BUCKET && path == super::FORMAT_CONFIG_FILE {
let mut format_info = self.format_info.write().await;
format_info.data.clone_from(&data);
@@ -623,47 +637,55 @@ impl LocalDisk {
let volume_dir = self.get_bucket_path(volume)?;
self.write_all_private(volume, path, &data, true, volume_dir).await?;
self.write_all_private(volume, path, data, true, &volume_dir).await?;
Ok(())
}
// write_all_private with check_path_length
#[tracing::instrument(level = "debug", skip_all)]
pub async fn write_all_private(
&self,
volume: &str,
path: &str,
buf: &[u8],
sync: bool,
skip_parent: impl AsRef<Path>,
) -> Result<()> {
pub async fn write_all_private(&self, volume: &str, path: &str, buf: Bytes, sync: bool, skip_parent: &Path) -> Result<()> {
let volume_dir = self.get_bucket_path(volume)?;
let file_path = volume_dir.join(Path::new(&path));
check_path_length(file_path.to_string_lossy().as_ref())?;
self.write_all_internal(file_path, buf, sync, skip_parent).await
self.write_all_internal(&file_path, InternalBuf::Owned(buf), sync, skip_parent)
.await
}
// write_all_internal do write file
pub async fn write_all_internal(
&self,
file_path: impl AsRef<Path>,
data: impl AsRef<[u8]>,
file_path: &Path,
data: InternalBuf<'_>,
sync: bool,
skip_parent: impl AsRef<Path>,
skip_parent: &Path,
) -> Result<()> {
let flags = O_CREATE | O_WRONLY | O_TRUNC;
let mut f = {
if sync {
// TODO: suport sync
self.open_file(file_path.as_ref(), flags, skip_parent.as_ref()).await?
self.open_file(file_path, flags, skip_parent).await?
} else {
self.open_file(file_path.as_ref(), flags, skip_parent.as_ref()).await?
self.open_file(file_path, flags, skip_parent).await?
}
};
f.write_all(data.as_ref()).await.map_err(to_file_error)?;
match data {
InternalBuf::Ref(buf) => {
f.write_all(buf).await.map_err(to_file_error)?;
}
InternalBuf::Owned(buf) => {
// Reduce one copy by using the owned buffer directly.
// It may be more efficient for larger writes.
let mut f = f.into_std().await;
let task = tokio::task::spawn_blocking(move || {
use std::io::Write as _;
f.write_all(buf.as_ref()).map_err(to_file_error)
});
task.await??;
}
}
Ok(())
}
@@ -703,7 +725,7 @@ impl LocalDisk {
let meta = file.metadata().await.map_err(to_file_error)?;
let file_size = meta.len() as usize;
bitrot_verify(Box::new(file), file_size, part_size, algo, sum.to_vec(), shard_size)
bitrot_verify(Box::new(file), file_size, part_size, algo, bytes::Bytes::copy_from_slice(sum), shard_size)
.await
.map_err(to_file_error)?;
@@ -751,7 +773,7 @@ impl LocalDisk {
Ok(res) => res,
Err(e) => {
if e != DiskError::VolumeNotFound && e != Error::FileNotFound {
info!("scan list_dir {}, err {:?}", &current, &e);
debug!("scan list_dir {}, err {:?}", &current, &e);
}
if opts.report_notfound && e == Error::FileNotFound && current == &opts.base_dir {
@@ -821,13 +843,14 @@ impl LocalDisk {
let name = decode_dir_object(format!("{}/{}", &current, &name).as_str());
out.write_obj(&MetaCacheEntry {
name,
name: name.clone(),
metadata,
..Default::default()
})
.await?;
*objs_returned += 1;
// warn!("scan list_dir {}, write_obj done, name: {:?}", &current, &name);
return Ok(());
}
}
@@ -848,6 +871,7 @@ impl LocalDisk {
for entry in entries.iter() {
if opts.limit > 0 && *objs_returned >= opts.limit {
// warn!("scan list_dir {}, limit reached 2", &current);
return Ok(());
}
@@ -923,6 +947,7 @@ impl LocalDisk {
while let Some(dir) = dir_stack.pop() {
if opts.limit > 0 && *objs_returned >= opts.limit {
// warn!("scan list_dir {}, limit reached 3", &current);
return Ok(());
}
@@ -943,6 +968,7 @@ impl LocalDisk {
}
}
// warn!("scan list_dir {}, done", &current);
Ok(())
}
}
@@ -952,13 +978,13 @@ fn is_root_path(path: impl AsRef<Path>) -> bool {
}
// 过滤 std::io::ErrorKind::NotFound
pub async fn read_file_exists(path: impl AsRef<Path>) -> Result<(Vec<u8>, Option<Metadata>)> {
pub async fn read_file_exists(path: impl AsRef<Path>) -> Result<(Bytes, Option<Metadata>)> {
let p = path.as_ref();
let (data, meta) = match read_file_all(&p).await {
Ok((data, meta)) => (data, Some(meta)),
Err(e) => {
if e == Error::FileNotFound {
(Vec::new(), None)
(Bytes::new(), None)
} else {
return Err(e);
}
@@ -973,13 +999,13 @@ pub async fn read_file_exists(path: impl AsRef<Path>) -> Result<(Vec<u8>, Option
Ok((data, meta))
}
pub async fn read_file_all(path: impl AsRef<Path>) -> Result<(Vec<u8>, Metadata)> {
pub async fn read_file_all(path: impl AsRef<Path>) -> Result<(Bytes, Metadata)> {
let p = path.as_ref();
let meta = read_file_metadata(&path).await?;
let data = fs::read(&p).await.map_err(to_file_error)?;
Ok((data, meta))
Ok((data.into(), meta))
}
pub async fn read_file_metadata(p: impl AsRef<Path>) -> Result<Metadata> {
@@ -1103,7 +1129,7 @@ impl DiskAPI for LocalDisk {
format_info.id = Some(disk_id);
format_info.file_info = Some(file_meta);
format_info.data = b;
format_info.data = b.into();
format_info.last_check = Some(OffsetDateTime::now_utc());
Ok(Some(disk_id))
@@ -1111,7 +1137,7 @@ impl DiskAPI for LocalDisk {
#[tracing::instrument(skip(self))]
async fn set_disk_id(&self, id: Option<Uuid>) -> Result<()> {
// 本地不需要设置
// No setup is required locally
// TODO: add check_id_store
let mut format_info = self.format_info.write().await;
format_info.id = id;
@@ -1119,7 +1145,7 @@ impl DiskAPI for LocalDisk {
}
#[tracing::instrument(skip(self))]
async fn read_all(&self, volume: &str, path: &str) -> Result<Vec<u8>> {
async fn read_all(&self, volume: &str, path: &str) -> Result<Bytes> {
if volume == RUSTFS_META_BUCKET && path == super::FORMAT_CONFIG_FILE {
let format_info = self.format_info.read().await;
if !format_info.data.is_empty() {
@@ -1134,7 +1160,7 @@ impl DiskAPI for LocalDisk {
}
#[tracing::instrument(level = "debug", skip_all)]
async fn write_all(&self, volume: &str, path: &str, data: Vec<u8>) -> Result<()> {
async fn write_all(&self, volume: &str, path: &str, data: Bytes) -> Result<()> {
self.write_all_public(volume, path, data).await
}
@@ -1179,7 +1205,7 @@ impl DiskAPI for LocalDisk {
let err = self
.bitrot_verify(
&part_path,
erasure.shard_file_size(part.size),
erasure.shard_file_size(part.size as i64) as usize,
checksum_info.algorithm,
&checksum_info.hash,
erasure.shard_size(),
@@ -1220,7 +1246,7 @@ impl DiskAPI for LocalDisk {
resp.results[i] = CHECK_PART_FILE_NOT_FOUND;
continue;
}
if (st.len() as usize) < fi.erasure.shard_file_size(part.size) {
if (st.len() as i64) < fi.erasure.shard_file_size(part.size as i64) {
resp.results[i] = CHECK_PART_FILE_CORRUPT;
continue;
}
@@ -1250,7 +1276,7 @@ impl DiskAPI for LocalDisk {
}
#[tracing::instrument(level = "debug", skip(self))]
async fn rename_part(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str, meta: Vec<u8>) -> Result<()> {
async fn rename_part(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str, meta: Bytes) -> Result<()> {
let src_volume_dir = self.get_bucket_path(src_volume)?;
let dst_volume_dir = self.get_bucket_path(dst_volume)?;
if !skip_access_checks(src_volume) {
@@ -1372,9 +1398,7 @@ impl DiskAPI for LocalDisk {
}
#[tracing::instrument(level = "debug", skip(self))]
async fn create_file(&self, origvolume: &str, volume: &str, path: &str, _file_size: usize) -> Result<FileWriter> {
// warn!("disk create_file: origvolume: {}, volume: {}, path: {}", origvolume, volume, path);
async fn create_file(&self, origvolume: &str, volume: &str, path: &str, _file_size: i64) -> Result<FileWriter> {
if !origvolume.is_empty() {
let origvolume_dir = self.get_bucket_path(origvolume)?;
if !skip_access_checks(origvolume) {
@@ -1405,8 +1429,6 @@ impl DiskAPI for LocalDisk {
#[tracing::instrument(level = "debug", skip(self))]
// async fn append_file(&self, volume: &str, path: &str, mut r: DuplexStream) -> Result<File> {
async fn append_file(&self, volume: &str, path: &str) -> Result<FileWriter> {
warn!("disk append_file: volume: {}, path: {}", volume, path);
let volume_dir = self.get_bucket_path(volume)?;
if !skip_access_checks(volume) {
access(&volume_dir)
@@ -1471,7 +1493,9 @@ impl DiskAPI for LocalDisk {
return Err(DiskError::FileCorrupt);
}
f.seek(SeekFrom::Start(offset as u64)).await?;
if offset > 0 {
f.seek(SeekFrom::Start(offset as u64)).await?;
}
Ok(Box::new(f))
}
@@ -1667,7 +1691,7 @@ impl DiskAPI for LocalDisk {
let new_dst_buf = xlmeta.marshal_msg()?;
self.write_all(src_volume, format!("{}/{}", &src_path, STORAGE_FORMAT_FILE).as_str(), new_dst_buf)
self.write_all(src_volume, format!("{}/{}", &src_path, STORAGE_FORMAT_FILE).as_str(), new_dst_buf.into())
.await?;
if let Some((src_data_path, dst_data_path)) = has_data_dir_path.as_ref() {
let no_inline = fi.data.is_none() && fi.size > 0;
@@ -1690,7 +1714,7 @@ impl DiskAPI for LocalDisk {
.write_all_private(
dst_volume,
format!("{}/{}/{}", &dst_path, &old_data_dir.to_string(), STORAGE_FORMAT_FILE).as_str(),
&dst_buf,
dst_buf.into(),
true,
&skip_parent,
)
@@ -1833,11 +1857,11 @@ impl DiskAPI for LocalDisk {
}
})?;
if !FileMeta::is_xl2_v1_format(buf.as_slice()) {
if !FileMeta::is_xl2_v1_format(buf.as_ref()) {
return Err(DiskError::FileVersionNotFound);
}
let mut xl_meta = FileMeta::load(buf.as_slice())?;
let mut xl_meta = FileMeta::load(buf.as_ref())?;
xl_meta.update_object_version(fi)?;
@@ -1869,7 +1893,7 @@ impl DiskAPI for LocalDisk {
let fm_data = meta.marshal_msg()?;
self.write_all(volume, format!("{}/{}", path, STORAGE_FORMAT_FILE).as_str(), fm_data)
self.write_all(volume, format!("{}/{}", path, STORAGE_FORMAT_FILE).as_str(), fm_data.into())
.await?;
Ok(())
@@ -2043,7 +2067,7 @@ impl DiskAPI for LocalDisk {
}
res.exists = true;
res.data = data;
res.data = data.into();
res.mod_time = match meta.modified() {
Ok(md) => Some(OffsetDateTime::from(md)),
Err(_) => {
@@ -2206,7 +2230,7 @@ impl DiskAPI for LocalDisk {
let mut obj_deleted = false;
for info in obj_infos.iter() {
let done = ScannerMetrics::time(ScannerMetric::ApplyVersion);
let sz: usize;
let sz: i64;
(obj_deleted, sz) = item.apply_actions(info, &mut size_s).await;
done();
@@ -2227,7 +2251,7 @@ impl DiskAPI for LocalDisk {
size_s.versions += 1;
}
size_s.total_size += sz;
size_s.total_size += sz as usize;
if info.delete_marker {
continue;
@@ -2428,8 +2452,8 @@ mod test {
disk.make_volume("test-volume").await.unwrap();
// Test write and read operations
let test_data = vec![1, 2, 3, 4, 5];
disk.write_all("test-volume", "test-file.txt", test_data.clone())
let test_data: Vec<u8> = vec![1, 2, 3, 4, 5];
disk.write_all("test-volume", "test-file.txt", test_data.clone().into())
.await
.unwrap();
@@ -2554,7 +2578,7 @@ mod test {
// Valid format info
let valid_format_info = FormatInfo {
id: Some(Uuid::new_v4()),
data: vec![1, 2, 3],
data: vec![1, 2, 3].into(),
file_info: Some(fs::metadata(".").await.unwrap()),
last_check: Some(now),
};
@@ -2563,7 +2587,7 @@ mod test {
// Invalid format info (missing id)
let invalid_format_info = FormatInfo {
id: None,
data: vec![1, 2, 3],
data: vec![1, 2, 3].into(),
file_info: Some(fs::metadata(".").await.unwrap()),
last_check: Some(now),
};
@@ -2573,7 +2597,7 @@ mod test {
let old_time = OffsetDateTime::now_utc() - time::Duration::seconds(10);
let old_format_info = FormatInfo {
id: Some(Uuid::new_v4()),
data: vec![1, 2, 3],
data: vec![1, 2, 3].into(),
file_info: Some(fs::metadata(".").await.unwrap()),
last_check: Some(old_time),
};
@@ -2594,7 +2618,7 @@ mod test {
// Test existing file
let (data, metadata) = read_file_exists(test_file).await.unwrap();
assert_eq!(data, b"test content");
assert_eq!(data.as_ref(), b"test content");
assert!(metadata.is_some());
// Clean up
@@ -2611,7 +2635,7 @@ mod test {
// Test reading file
let (data, metadata) = read_file_all(test_file).await.unwrap();
assert_eq!(data, test_content);
assert_eq!(data.as_ref(), test_content);
assert!(metadata.is_file());
assert_eq!(metadata.len(), test_content.len() as u64);
ecstore/src/disk/mod.rs
+10 -11
View File
@@ -6,7 +6,6 @@ pub mod format;
pub mod fs;
pub mod local;
pub mod os;
pub mod remote;
pub const RUSTFS_META_BUCKET: &str = ".rustfs.sys";
pub const RUSTFS_META_MULTIPART_BUCKET: &str = ".rustfs.sys/multipart";
@@ -22,12 +21,13 @@ use crate::heal::{
data_usage_cache::{DataUsageCache, DataUsageEntry},
heal_commands::{HealScanMode, HealingTracker},
};
use crate::rpc::RemoteDisk;
use bytes::Bytes;
use endpoint::Endpoint;
use error::DiskError;
use error::{Error, Result};
use local::LocalDisk;
use madmin::info_commands::DiskMetrics;
use remote::RemoteDisk;
use rustfs_filemeta::{FileInfo, RawFileInfo};
use serde::{Deserialize, Serialize};
use std::{fmt::Debug, path::PathBuf, sync::Arc};
@@ -36,7 +36,6 @@ use tokio::{
io::{AsyncRead, AsyncWrite},
sync::mpsc::Sender,
};
use tracing::warn;
use uuid::Uuid;
pub type DiskStore = Arc<Disk>;
@@ -303,7 +302,7 @@ impl DiskAPI for Disk {
}
#[tracing::instrument(skip(self))]
async fn create_file(&self, _origvolume: &str, volume: &str, path: &str, _file_size: usize) -> Result<FileWriter> {
async fn create_file(&self, _origvolume: &str, volume: &str, path: &str, _file_size: i64) -> Result<FileWriter> {
match self {
Disk::Local(local_disk) => local_disk.create_file(_origvolume, volume, path, _file_size).await,
Disk::Remote(remote_disk) => remote_disk.create_file(_origvolume, volume, path, _file_size).await,
@@ -319,7 +318,7 @@ impl DiskAPI for Disk {
}
#[tracing::instrument(skip(self))]
async fn rename_part(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str, meta: Vec<u8>) -> Result<()> {
async fn rename_part(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str, meta: Bytes) -> Result<()> {
match self {
Disk::Local(local_disk) => local_disk.rename_part(src_volume, src_path, dst_volume, dst_path, meta).await,
Disk::Remote(remote_disk) => {
@@ -363,7 +362,7 @@ impl DiskAPI for Disk {
}
#[tracing::instrument(skip(self))]
async fn write_all(&self, volume: &str, path: &str, data: Vec<u8>) -> Result<()> {
async fn write_all(&self, volume: &str, path: &str, data: Bytes) -> Result<()> {
match self {
Disk::Local(local_disk) => local_disk.write_all(volume, path, data).await,
Disk::Remote(remote_disk) => remote_disk.write_all(volume, path, data).await,
@@ -371,7 +370,7 @@ impl DiskAPI for Disk {
}
#[tracing::instrument(skip(self))]
async fn read_all(&self, volume: &str, path: &str) -> Result<Vec<u8>> {
async fn read_all(&self, volume: &str, path: &str) -> Result<Bytes> {
match self {
Disk::Local(local_disk) => local_disk.read_all(volume, path).await,
Disk::Remote(remote_disk) => remote_disk.read_all(volume, path).await,
@@ -490,10 +489,10 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
async fn read_file(&self, volume: &str, path: &str) -> Result<FileReader>;
async fn read_file_stream(&self, volume: &str, path: &str, offset: usize, length: usize) -> Result<FileReader>;
async fn append_file(&self, volume: &str, path: &str) -> Result<FileWriter>;
async fn create_file(&self, origvolume: &str, volume: &str, path: &str, file_size: usize) -> Result<FileWriter>;
async fn create_file(&self, origvolume: &str, volume: &str, path: &str, file_size: i64) -> Result<FileWriter>;
// ReadFileStream
async fn rename_file(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str) -> Result<()>;
async fn rename_part(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str, meta: Vec<u8>) -> Result<()>;
async fn rename_part(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str, meta: Bytes) -> Result<()>;
async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()>;
// VerifyFile
async fn verify_file(&self, volume: &str, path: &str, fi: &FileInfo) -> Result<CheckPartsResp>;
@@ -503,8 +502,8 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
// ReadParts
async fn read_multiple(&self, req: ReadMultipleReq) -> Result<Vec<ReadMultipleResp>>;
// CleanAbandonedData
async fn write_all(&self, volume: &str, path: &str, data: Vec<u8>) -> Result<()>;
async fn read_all(&self, volume: &str, path: &str) -> Result<Vec<u8>>;
async fn write_all(&self, volume: &str, path: &str, data: Bytes) -> Result<()>;
async fn read_all(&self, volume: &str, path: &str) -> Result<Bytes>;
async fn disk_info(&self, opts: &DiskInfoOptions) -> Result<DiskInfo>;
async fn ns_scanner(
&self,
ecstore/src/endpoints.rs
+8 -2
View File
@@ -680,7 +680,7 @@ mod test {
),
(
vec!["ftp://server/d1", "http://server/d2", "http://server/d3", "http://server/d4"],
Some(Error::other("'ftp://server/d1': io error")),
Some(Error::other("'ftp://server/d1': io error invalid URL endpoint format")),
10,
),
(
@@ -719,7 +719,13 @@ mod test {
(None, Ok(_)) => {}
(Some(e), Ok(_)) => panic!("{}: error: expected = {}, got = <nil>", test_case.2, e),
(Some(e), Err(e2)) => {
assert_eq!(e.to_string(), e2.to_string(), "{}: error: expected = {}, got = {}", test_case.2, e, e2)
assert!(
e2.to_string().starts_with(&e.to_string()),
"{}: error: expected = {}, got = {}",
test_case.2,
e,
e2
)
}
}
}
ecstore/src/erasure_coding/bitrot.rs
+48 -40
View File
@@ -1,6 +1,9 @@
use bytes::Bytes;
use pin_project_lite::pin_project;
use rustfs_utils::{HashAlgorithm, read_full, write_all};
use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite};
use rustfs_utils::HashAlgorithm;
use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt};
use tracing::error;
use uuid::Uuid;
pin_project! {
/// BitrotReader reads (hash+data) blocks from an async reader and verifies hash integrity.
@@ -11,10 +14,11 @@ pin_project! {
shard_size: usize,
buf: Vec<u8>,
hash_buf: Vec<u8>,
hash_read: usize,
data_buf: Vec<u8>,
data_read: usize,
hash_checked: bool,
// hash_read: usize,
// data_buf: Vec<u8>,
// data_read: usize,
// hash_checked: bool,
id: Uuid,
}
}
@@ -31,10 +35,11 @@ where
shard_size,
buf: Vec::new(),
hash_buf: vec![0u8; hash_size],
hash_read: 0,
data_buf: Vec::new(),
data_read: 0,
hash_checked: false,
// hash_read: 0,
// data_buf: Vec::new(),
// data_read: 0,
// hash_checked: false,
id: Uuid::new_v4(),
}
}
@@ -50,30 +55,31 @@ where
let hash_size = self.hash_algo.size();
// Read hash
let mut hash_buf = vec![0u8; hash_size];
if hash_size > 0 {
self.inner.read_exact(&mut hash_buf).await?;
self.inner.read_exact(&mut self.hash_buf).await.map_err(|e| {
error!("bitrot reader read hash error: {}", e);
e
})?;
}
let data_len = read_full(&mut self.inner, out).await?;
// // Read data
// let mut data_len = 0;
// while data_len < out.len() {
// let n = self.inner.read(&mut out[data_len..]).await?;
// if n == 0 {
// break;
// }
// data_len += n;
// // Only read up to one shard_size block
// if data_len >= self.shard_size {
// break;
// }
// }
// Read data
let mut data_len = 0;
while data_len < out.len() {
let n = self.inner.read(&mut out[data_len..]).await.map_err(|e| {
error!("bitrot reader read data error: {}", e);
e
})?;
if n == 0 {
break;
}
data_len += n;
}
if hash_size > 0 {
let actual_hash = self.hash_algo.hash_encode(&out[..data_len]);
if actual_hash != hash_buf {
if actual_hash.as_ref() != self.hash_buf.as_slice() {
error!("bitrot reader hash mismatch, id={} data_len={}, out_len={}", self.id, data_len, out.len());
return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, "bitrot hash mismatch"));
}
}
@@ -139,27 +145,25 @@ where
if hash_algo.size() > 0 {
let hash = hash_algo.hash_encode(buf);
self.buf.extend_from_slice(&hash);
self.buf.extend_from_slice(hash.as_ref());
}
self.buf.extend_from_slice(buf);
// Write hash+data in one call
let mut n = write_all(&mut self.inner, &self.buf).await?;
self.inner.write_all(&self.buf).await?;
if n < hash_algo.size() {
return Err(std::io::Error::new(
std::io::ErrorKind::WriteZero,
"short write: not enough bytes written",
));
}
// self.inner.flush().await?;
n -= hash_algo.size();
let n = buf.len();
self.buf.clear();
Ok(n)
}
pub async fn shutdown(&mut self) -> std::io::Result<()> {
self.inner.shutdown().await
}
}
pub fn bitrot_shard_file_size(size: usize, shard_size: usize, algo: HashAlgorithm) -> usize {
@@ -174,7 +178,7 @@ pub async fn bitrot_verify<R: AsyncRead + Unpin + Send>(
want_size: usize,
part_size: usize,
algo: HashAlgorithm,
_want: Vec<u8>,
_want: Bytes, // FIXME: useless parameter?
mut shard_size: usize,
) -> std::io::Result<()> {
let mut hash_buf = vec![0; algo.size()];
@@ -196,7 +200,7 @@ pub async fn bitrot_verify<R: AsyncRead + Unpin + Send>(
let read = r.read_exact(&mut buf).await?;
let actual_hash = algo.hash_encode(&buf);
if actual_hash != hash_buf[0..n] {
if actual_hash.as_ref() != &hash_buf[0..n] {
return Err(std::io::Error::other("bitrot hash mismatch"));
}
@@ -329,6 +333,10 @@ impl BitrotWriterWrapper {
self.bitrot_writer.write(buf).await
}
pub async fn shutdown(&mut self) -> std::io::Result<()> {
self.bitrot_writer.shutdown().await
}
/// Extract the inline buffer data, consuming the wrapper
pub fn into_inline_data(self) -> Option<Vec<u8>> {
match self.writer_type {
ecstore/src/erasure_coding/decode.rs
+37 -27
View File
@@ -30,7 +30,7 @@ where
// readers传入前应处理disk错误，确保每个reader达到可用数量的BitrotReader
pub fn new(readers: Vec<Option<BitrotReader<R>>>, e: Erasure, offset: usize, total_length: usize) -> Self {
let shard_size = e.shard_size();
let shard_file_size = e.shard_file_size(total_length);
let shard_file_size = e.shard_file_size(total_length as i64) as usize;
let offset = (offset / e.block_size) * shard_size;
@@ -67,36 +67,34 @@ where
}
// 使用并发读取所有分片
let mut read_futs = Vec::with_capacity(self.readers.len());
let read_futs: Vec<_> = self
.readers
.iter_mut()
.enumerate()
.map(|(i, opt_reader)| {
if let Some(reader) = opt_reader.as_mut() {
for (i, opt_reader) in self.readers.iter_mut().enumerate() {
let future = if let Some(reader) = opt_reader.as_mut() {
Box::pin(async move {
let mut buf = vec![0u8; shard_size];
// 需要move i, buf
Some(async move {
match reader.read(&mut buf).await {
Ok(n) => {
buf.truncate(n);
(i, Ok(buf))
}
Err(e) => (i, Err(Error::from(e))),
match reader.read(&mut buf).await {
Ok(n) => {
buf.truncate(n);
(i, Ok(buf))
}
})
} else {
None
}
})
.collect();
Err(e) => (i, Err(Error::from(e))),
}
}) as std::pin::Pin<Box<dyn std::future::Future<Output = (usize, Result<Vec<u8>, Error>)> + Send>>
} else {
// reader是None时返回FileNotFound错误
Box::pin(async move { (i, Err(Error::FileNotFound)) })
as std::pin::Pin<Box<dyn std::future::Future<Output = (usize, Result<Vec<u8>, Error>)> + Send>>
};
read_futs.push(future);
}
// 过滤掉None，join_all
let mut results = join_all(read_futs.into_iter().flatten()).await;
let results = join_all(read_futs).await;
let mut shards: Vec<Option<Vec<u8>>> = vec![None; self.readers.len()];
let mut errs = vec![None; self.readers.len()];
for (i, shard) in results.drain(..) {
for (i, shard) in results.into_iter() {
match shard {
Ok(data) => {
if !data.is_empty() {
@@ -104,7 +102,7 @@ where
}
}
Err(e) => {
error!("Error reading shard {}: {}", i, e);
// error!("Error reading shard {}: {}", i, e);
errs[i] = Some(e);
}
}
@@ -142,6 +140,7 @@ where
W: tokio::io::AsyncWrite + Send + Sync + Unpin,
{
if get_data_block_len(en_blocks, data_blocks) < length {
error!("write_data_blocks get_data_block_len < length");
return Err(io::Error::new(ErrorKind::UnexpectedEof, "Not enough data blocks to write"));
}
@@ -150,6 +149,7 @@ where
for block_op in &en_blocks[..data_blocks] {
if block_op.is_none() {
error!("write_data_blocks block_op.is_none()");
return Err(io::Error::new(ErrorKind::UnexpectedEof, "Missing data block"));
}
@@ -164,7 +164,10 @@ where
offset = 0;
if write_left < block.len() {
writer.write_all(&block_slice[..write_left]).await?;
writer.write_all(&block_slice[..write_left]).await.map_err(|e| {
error!("write_data_blocks write_all err: {}", e);
e
})?;
total_written += write_left;
break;
@@ -172,7 +175,10 @@ where
let n = block_slice.len();
writer.write_all(block_slice).await?;
writer.write_all(block_slice).await.map_err(|e| {
error!("write_data_blocks write_all2 err: {}", e);
e
})?;
write_left -= n;
@@ -228,6 +234,7 @@ impl Erasure {
};
if block_length == 0 {
// error!("erasure decode decode block_length == 0");
break;
}
@@ -242,12 +249,14 @@ impl Erasure {
}
if !reader.can_decode(&shards) {
error!("erasure decode can_decode errs: {:?}", &errs);
ret_err = Some(Error::ErasureReadQuorum.into());
break;
}
// Decode the shards
if let Err(e) = self.decode_data(&mut shards) {
error!("erasure decode decode_data err: {:?}", e);
ret_err = Some(e);
break;
}
@@ -255,6 +264,7 @@ impl Erasure {
let n = match write_data_blocks(writer, &shards, self.data_shards, block_offset, block_length).await {
Ok(n) => n,
Err(e) => {
error!("erasure decode write_data_blocks err: {:?}", e);
ret_err = Some(e);
break;
}
ecstore/src/erasure_coding/encode.rs
+44 -20
View File
@@ -4,10 +4,13 @@ use crate::disk::error::Error;
use crate::disk::error_reduce::count_errs;
use crate::disk::error_reduce::{OBJECT_OP_IGNORED_ERRS, reduce_write_quorum_errs};
use bytes::Bytes;
use futures::StreamExt;
use futures::stream::FuturesUnordered;
use std::sync::Arc;
use std::vec;
use tokio::io::AsyncRead;
use tokio::sync::mpsc;
use tracing::error;
pub(crate) struct MultiWriter<'a> {
writers: &'a mut [Option<BitrotWriterWrapper>],
@@ -25,33 +28,41 @@ impl<'a> MultiWriter<'a> {
}
}
#[allow(clippy::needless_range_loop)]
pub async fn write(&mut self, data: Vec<Bytes>) -> std::io::Result<()> {
for i in 0..self.writers.len() {
if self.errs[i].is_some() {
continue; // Skip if we already have an error for this writer
}
let writer_opt = &mut self.writers[i];
let shard = &data[i];
if let Some(writer) = writer_opt {
async fn write_shard(writer_opt: &mut Option<BitrotWriterWrapper>, err: &mut Option<Error>, shard: &Bytes) {
match writer_opt {
Some(writer) => {
match writer.write(shard).await {
Ok(n) => {
if n < shard.len() {
self.errs[i] = Some(Error::ShortWrite);
self.writers[i] = None; // Mark as failed
*err = Some(Error::ShortWrite);
*writer_opt = None; // Mark as failed
} else {
self.errs[i] = None;
*err = None;
}
}
Err(e) => {
self.errs[i] = Some(Error::from(e));
*err = Some(Error::from(e));
}
}
} else {
self.errs[i] = Some(Error::DiskNotFound);
}
None => {
*err = Some(Error::DiskNotFound);
}
}
}
pub async fn write(&mut self, data: Vec<Bytes>) -> std::io::Result<()> {
assert_eq!(data.len(), self.writers.len());
{
let mut futures = FuturesUnordered::new();
for ((writer_opt, err), shard) in self.writers.iter_mut().zip(self.errs.iter_mut()).zip(data.iter()) {
if err.is_some() {
continue; // Skip if we already have an error for this writer
}
futures.push(Self::write_shard(writer_opt, err, shard));
}
while let Some(()) = futures.next().await {}
}
let nil_count = self.errs.iter().filter(|&e| e.is_none()).count();
@@ -60,6 +71,13 @@ impl<'a> MultiWriter<'a> {
}
if let Some(write_err) = reduce_write_quorum_errs(&self.errs, OBJECT_OP_IGNORED_ERRS, self.write_quorum) {
error!(
"reduce_write_quorum_errs: {:?}, offline-disks={}/{}, errs={:?}",
write_err,
count_errs(&self.errs, &Error::DiskNotFound),
self.writers.len(),
self.errs
);
return Err(std::io::Error::other(format!(
"Failed to write data: {} (offline-disks={}/{})",
write_err,
@@ -79,6 +97,13 @@ impl<'a> MultiWriter<'a> {
.join(", ")
)))
}
pub async fn _shutdown(&mut self) -> std::io::Result<()> {
for writer in self.writers.iter_mut().flatten() {
writer.shutdown().await?;
}
Ok(())
}
}
impl Erasure {
@@ -96,8 +121,8 @@ impl Erasure {
let task = tokio::spawn(async move {
let block_size = self.block_size;
let mut total = 0;
let mut buf = vec![0u8; block_size];
loop {
let mut buf = vec![0u8; block_size];
match rustfs_utils::read_full(&mut reader, &mut buf).await {
Ok(n) if n > 0 => {
total += n;
@@ -114,7 +139,6 @@ impl Erasure {
return Err(e);
}
}
buf.clear();
}
Ok((reader, total))
@@ -130,7 +154,7 @@ impl Erasure {
}
let (reader, total) = task.await??;
// writers.shutdown().await?;
Ok((reader, total))
}
}
ecstore/src/erasure_coding/erasure.rs
+93 -224
View File
@@ -1,27 +1,15 @@
//! Erasure coding implementation supporting multiple Reed-Solomon backends.
//! Erasure coding implementation using Reed-Solomon SIMD backend.
//!
//! This module provides erasure coding functionality with support for two different
//! Reed-Solomon implementations:
//! This module provides erasure coding functionality with high-performance SIMD
//! Reed-Solomon implementation:
//!
//! ## Reed-Solomon Implementations
//! ## Reed-Solomon Implementation
//!
//! ### Pure Erasure Mode (Default)
//! - **Stability**: Pure erasure implementation, mature and well-tested
//! - **Performance**: Good performance with consistent behavior
//! - **Compatibility**: Works with any shard size
//! - **Use case**: Default behavior, recommended for most production use cases
//!
//! ### SIMD Mode (`reed-solomon-simd` feature)
//! ### SIMD Mode (Only)
//! - **Performance**: Uses SIMD optimization for high-performance encoding/decoding
//! - **Compatibility**: Works with any shard size through SIMD implementation
//! - **Reliability**: High-performance SIMD implementation for large data processing
//! - **Use case**: Use when maximum performance is needed for large data processing
//!
//! ## Feature Flags
//!
//! - Default: Use pure reed-solomon-erasure implementation (stable and reliable)
//! - `reed-solomon-simd`: Use SIMD mode for optimal performance
//! - `reed-solomon-erasure`: Explicitly enable pure erasure mode (same as default)
//! - **Use case**: Optimized for maximum performance in large data processing scenarios
//!
//! ## Example
//!
@@ -35,8 +23,6 @@
//! ```
use bytes::{Bytes, BytesMut};
use reed_solomon_erasure::galois_8::ReedSolomon as ReedSolomonErasure;
#[cfg(feature = "reed-solomon-simd")]
use reed_solomon_simd;
use smallvec::SmallVec;
use std::io;
@@ -44,38 +30,23 @@ use tokio::io::AsyncRead;
use tracing::warn;
use uuid::Uuid;
/// Reed-Solomon encoder variants supporting different implementations.
#[allow(clippy::large_enum_variant)]
pub enum ReedSolomonEncoder {
/// SIMD mode: High-performance SIMD implementation (when reed-solomon-simd feature is enabled)
#[cfg(feature = "reed-solomon-simd")]
SIMD {
data_shards: usize,
parity_shards: usize,
// 使用RwLock确保线程安全，实现Send + Sync
encoder_cache: std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonEncoder>>,
decoder_cache: std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonDecoder>>,
},
/// Pure erasure mode: default and when reed-solomon-erasure feature is specified
Erasure(Box<ReedSolomonErasure>),
/// Reed-Solomon encoder using SIMD implementation.
pub struct ReedSolomonEncoder {
data_shards: usize,
parity_shards: usize,
// 使用RwLock确保线程安全，实现Send + Sync
encoder_cache: std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonEncoder>>,
decoder_cache: std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonDecoder>>,
}
impl Clone for ReedSolomonEncoder {
fn clone(&self) -> Self {
match self {
#[cfg(feature = "reed-solomon-simd")]
ReedSolomonEncoder::SIMD {
data_shards,
parity_shards,
..
} => ReedSolomonEncoder::SIMD {
data_shards: *data_shards,
parity_shards: *parity_shards,
// 为新实例创建空的缓存，不共享缓存
encoder_cache: std::sync::RwLock::new(None),
decoder_cache: std::sync::RwLock::new(None),
},
ReedSolomonEncoder::Erasure(encoder) => ReedSolomonEncoder::Erasure(encoder.clone()),
Self {
data_shards: self.data_shards,
parity_shards: self.parity_shards,
// 为新实例创建空的缓存，不共享缓存
encoder_cache: std::sync::RwLock::new(None),
decoder_cache: std::sync::RwLock::new(None),
}
}
}
@@ -83,81 +54,50 @@ impl Clone for ReedSolomonEncoder {
impl ReedSolomonEncoder {
/// Create a new Reed-Solomon encoder with specified data and parity shards.
pub fn new(data_shards: usize, parity_shards: usize) -> io::Result<Self> {
#[cfg(feature = "reed-solomon-simd")]
{
// SIMD mode when reed-solomon-simd feature is enabled
Ok(ReedSolomonEncoder::SIMD {
data_shards,
parity_shards,
encoder_cache: std::sync::RwLock::new(None),
decoder_cache: std::sync::RwLock::new(None),
})
}
#[cfg(not(feature = "reed-solomon-simd"))]
{
// Pure erasure mode when reed-solomon-simd feature is not enabled (default or reed-solomon-erasure)
let encoder = ReedSolomonErasure::new(data_shards, parity_shards)
.map_err(|e| io::Error::other(format!("Failed to create erasure encoder: {:?}", e)))?;
Ok(ReedSolomonEncoder::Erasure(Box::new(encoder)))
}
Ok(ReedSolomonEncoder {
data_shards,
parity_shards,
encoder_cache: std::sync::RwLock::new(None),
decoder_cache: std::sync::RwLock::new(None),
})
}
/// Encode data shards with parity.
pub fn encode(&self, shards: SmallVec<[&mut [u8]; 16]>) -> io::Result<()> {
match self {
#[cfg(feature = "reed-solomon-simd")]
ReedSolomonEncoder::SIMD {
data_shards,
parity_shards,
encoder_cache,
..
} => {
let mut shards_vec: Vec<&mut [u8]> = shards.into_vec();
if shards_vec.is_empty() {
return Ok(());
}
let mut shards_vec: Vec<&mut [u8]> = shards.into_vec();
if shards_vec.is_empty() {
return Ok(());
}
// 使用 SIMD 进行编码
let simd_result = self.encode_with_simd(*data_shards, *parity_shards, encoder_cache, &mut shards_vec);
// 使用 SIMD 进行编码
let simd_result = self.encode_with_simd(&mut shards_vec);
match simd_result {
Ok(()) => Ok(()),
Err(simd_error) => {
warn!("SIMD encoding failed: {}", simd_error);
Err(simd_error)
}
}
match simd_result {
Ok(()) => Ok(()),
Err(simd_error) => {
warn!("SIMD encoding failed: {}", simd_error);
Err(simd_error)
}
ReedSolomonEncoder::Erasure(encoder) => encoder
.encode(shards)
.map_err(|e| io::Error::other(format!("Erasure encode error: {:?}", e))),
}
}
#[cfg(feature = "reed-solomon-simd")]
fn encode_with_simd(
&self,
data_shards: usize,
parity_shards: usize,
encoder_cache: &std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonEncoder>>,
shards_vec: &mut [&mut [u8]],
) -> io::Result<()> {
fn encode_with_simd(&self, shards_vec: &mut [&mut [u8]]) -> io::Result<()> {
let shard_len = shards_vec[0].len();
// 获取或创建encoder
let mut encoder = {
let mut cache_guard = encoder_cache
let mut cache_guard = self
.encoder_cache
.write()
.map_err(|_| io::Error::other("Failed to acquire encoder cache lock"))?;
match cache_guard.take() {
Some(mut cached_encoder) => {
// 使用reset方法重置现有encoder以适应新的参数
if let Err(e) = cached_encoder.reset(data_shards, parity_shards, shard_len) {
if let Err(e) = cached_encoder.reset(self.data_shards, self.parity_shards, shard_len) {
warn!("Failed to reset SIMD encoder: {:?}, creating new one", e);
// 如果reset失败，创建新的encoder
reed_solomon_simd::ReedSolomonEncoder::new(data_shards, parity_shards, shard_len)
reed_solomon_simd::ReedSolomonEncoder::new(self.data_shards, self.parity_shards, shard_len)
.map_err(|e| io::Error::other(format!("Failed to create SIMD encoder: {:?}", e)))?
} else {
cached_encoder
@@ -165,14 +105,14 @@ impl ReedSolomonEncoder {
}
None => {
// 第一次使用，创建新encoder
reed_solomon_simd::ReedSolomonEncoder::new(data_shards, parity_shards, shard_len)
reed_solomon_simd::ReedSolomonEncoder::new(self.data_shards, self.parity_shards, shard_len)
.map_err(|e| io::Error::other(format!("Failed to create SIMD encoder: {:?}", e)))?
}
}
};
// 添加原始shards
for (i, shard) in shards_vec.iter().enumerate().take(data_shards) {
for (i, shard) in shards_vec.iter().enumerate().take(self.data_shards) {
encoder
.add_original_shard(shard)
.map_err(|e| io::Error::other(format!("Failed to add shard {}: {:?}", i, e)))?;
@@ -185,15 +125,16 @@ impl ReedSolomonEncoder {
// 将恢复shards复制到输出缓冲区
for (i, recovery_shard) in result.recovery_iter().enumerate() {
if i + data_shards < shards_vec.len() {
shards_vec[i + data_shards].copy_from_slice(recovery_shard);
if i + self.data_shards < shards_vec.len() {
shards_vec[i + self.data_shards].copy_from_slice(recovery_shard);
}
}
// 将encoder放回缓存（在result被drop后encoder自动重置，可以重用）
drop(result); // 显式drop result，确保encoder被重置
*encoder_cache
*self
.encoder_cache
.write()
.map_err(|_| io::Error::other("Failed to return encoder to cache"))? = Some(encoder);
@@ -202,39 +143,19 @@ impl ReedSolomonEncoder {
/// Reconstruct missing shards.
pub fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
match self {
#[cfg(feature = "reed-solomon-simd")]
ReedSolomonEncoder::SIMD {
data_shards,
parity_shards,
decoder_cache,
..
} => {
// 使用 SIMD 进行重构
let simd_result = self.reconstruct_with_simd(*data_shards, *parity_shards, decoder_cache, shards);
// 使用 SIMD 进行重构
let simd_result = self.reconstruct_with_simd(shards);
match simd_result {
Ok(()) => Ok(()),
Err(simd_error) => {
warn!("SIMD reconstruction failed: {}", simd_error);
Err(simd_error)
}
}
match simd_result {
Ok(()) => Ok(()),
Err(simd_error) => {
warn!("SIMD reconstruction failed: {}", simd_error);
Err(simd_error)
}
ReedSolomonEncoder::Erasure(encoder) => encoder
.reconstruct(shards)
.map_err(|e| io::Error::other(format!("Erasure reconstruct error: {:?}", e))),
}
}
#[cfg(feature = "reed-solomon-simd")]
fn reconstruct_with_simd(
&self,
data_shards: usize,
parity_shards: usize,
decoder_cache: &std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonDecoder>>,
shards: &mut [Option<Vec<u8>>],
) -> io::Result<()> {
fn reconstruct_with_simd(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
// Find a valid shard to determine length
let shard_len = shards
.iter()
@@ -243,17 +164,18 @@ impl ReedSolomonEncoder {
// 获取或创建decoder
let mut decoder = {
let mut cache_guard = decoder_cache
let mut cache_guard = self
.decoder_cache
.write()
.map_err(|_| io::Error::other("Failed to acquire decoder cache lock"))?;
match cache_guard.take() {
Some(mut cached_decoder) => {
// 使用reset方法重置现有decoder
if let Err(e) = cached_decoder.reset(data_shards, parity_shards, shard_len) {
if let Err(e) = cached_decoder.reset(self.data_shards, self.parity_shards, shard_len) {
warn!("Failed to reset SIMD decoder: {:?}, creating new one", e);
// 如果reset失败，创建新的decoder
reed_solomon_simd::ReedSolomonDecoder::new(data_shards, parity_shards, shard_len)
reed_solomon_simd::ReedSolomonDecoder::new(self.data_shards, self.parity_shards, shard_len)
.map_err(|e| io::Error::other(format!("Failed to create SIMD decoder: {:?}", e)))?
} else {
cached_decoder
@@ -261,7 +183,7 @@ impl ReedSolomonEncoder {
}
None => {
// 第一次使用，创建新decoder
reed_solomon_simd::ReedSolomonDecoder::new(data_shards, parity_shards, shard_len)
reed_solomon_simd::ReedSolomonDecoder::new(self.data_shards, self.parity_shards, shard_len)
.map_err(|e| io::Error::other(format!("Failed to create SIMD decoder: {:?}", e)))?
}
}
@@ -270,12 +192,12 @@ impl ReedSolomonEncoder {
// Add available shards (both data and parity)
for (i, shard_opt) in shards.iter().enumerate() {
if let Some(shard) = shard_opt {
if i < data_shards {
if i < self.data_shards {
decoder
.add_original_shard(i, shard)
.map_err(|e| io::Error::other(format!("Failed to add original shard for reconstruction: {:?}", e)))?;
} else {
let recovery_idx = i - data_shards;
let recovery_idx = i - self.data_shards;
decoder
.add_recovery_shard(recovery_idx, shard)
.map_err(|e| io::Error::other(format!("Failed to add recovery shard for reconstruction: {:?}", e)))?;
@@ -289,7 +211,7 @@ impl ReedSolomonEncoder {
// Fill in missing data shards from reconstruction result
for (i, shard_opt) in shards.iter_mut().enumerate() {
if shard_opt.is_none() && i < data_shards {
if shard_opt.is_none() && i < self.data_shards {
for (restored_index, restored_data) in result.restored_original_iter() {
if restored_index == i {
*shard_opt = Some(restored_data.to_vec());
@@ -302,7 +224,8 @@ impl ReedSolomonEncoder {
// 将decoder放回缓存（在result被drop后decoder自动重置，可以重用）
drop(result); // 显式drop result，确保decoder被重置
*decoder_cache
*self
.decoder_cache
.write()
.map_err(|_| io::Error::other("Failed to return decoder to cache"))? = Some(decoder);
@@ -469,22 +392,27 @@ impl Erasure {
}
/// Calculate the total erasure file size for a given original size.
// Returns the final erasure size from the original size
pub fn shard_file_size(&self, total_length: usize) -> usize {
pub fn shard_file_size(&self, total_length: i64) -> i64 {
if total_length == 0 {
return 0;
}
if total_length < 0 {
return total_length;
}
let total_length = total_length as usize;
let num_shards = total_length / self.block_size;
let last_block_size = total_length % self.block_size;
let last_shard_size = calc_shard_size(last_block_size, self.data_shards);
num_shards * self.shard_size() + last_shard_size
(num_shards * self.shard_size() + last_shard_size) as i64
}
/// Calculate the offset in the erasure file where reading begins.
// Returns the offset in the erasure file where reading begins
pub fn shard_file_offset(&self, start_offset: usize, length: usize, total_length: usize) -> usize {
let shard_size = self.shard_size();
let shard_file_size = self.shard_file_size(total_length);
let shard_file_size = self.shard_file_size(total_length as i64) as usize;
let end_shard = (start_offset + length) / self.block_size;
let mut till_offset = end_shard * shard_size + shard_size;
if till_offset > shard_file_size {
@@ -550,6 +478,13 @@ mod tests {
use super::*;
#[test]
fn test_shard_file_size_cases2() {
let erasure = Erasure::new(12, 4, 1024 * 1024);
assert_eq!(erasure.shard_file_size(1572864), 131074);
}
#[test]
fn test_shard_file_size_cases() {
let erasure = Erasure::new(4, 2, 8);
@@ -572,25 +507,18 @@ mod tests {
assert_eq!(erasure.shard_file_size(1248739), 312186); // 1248739/8=156092, last=3, 3 div_ceil 4=1, 156092*2+1=312185
assert_eq!(erasure.shard_file_size(43), 12); // 43/8=5, last=3, 3 div_ceil 4=1, 5*2+1=11
assert_eq!(erasure.shard_file_size(1572864), 393216); // 43/8=5, last=3, 3 div_ceil 4=1, 5*2+1=11
}
#[test]
fn test_encode_decode_roundtrip() {
let data_shards = 4;
let parity_shards = 2;
// Use different block sizes based on feature
#[cfg(not(feature = "reed-solomon-simd"))]
let block_size = 8; // Pure erasure mode (default)
#[cfg(feature = "reed-solomon-simd")]
let block_size = 1024; // SIMD mode - SIMD with fallback
let block_size = 1024; // SIMD mode
let erasure = Erasure::new(data_shards, parity_shards, block_size);
// Use different test data based on feature
#[cfg(not(feature = "reed-solomon-simd"))]
let test_data = b"hello world".to_vec(); // Small data for erasure (default)
#[cfg(feature = "reed-solomon-simd")]
// Use sufficient test data for SIMD optimization
let test_data = b"SIMD mode test data for encoding and decoding roundtrip verification with sufficient length to ensure shard size requirements are met for proper SIMD optimization.".repeat(20); // ~3KB for SIMD
let data = &test_data;
@@ -618,13 +546,7 @@ mod tests {
fn test_encode_decode_large_1m() {
let data_shards = 4;
let parity_shards = 2;
// Use different block sizes based on feature
#[cfg(feature = "reed-solomon-simd")]
let block_size = 512 * 3; // SIMD mode
#[cfg(not(feature = "reed-solomon-simd"))]
let block_size = 8192; // Pure erasure mode (default)
let erasure = Erasure::new(data_shards, parity_shards, block_size);
// Generate 1MB test data
@@ -672,9 +594,14 @@ mod tests {
#[test]
fn test_shard_file_offset() {
let erasure = Erasure::new(4, 2, 8);
let offset = erasure.shard_file_offset(0, 16, 32);
let erasure = Erasure::new(8, 8, 1024 * 1024);
let offset = erasure.shard_file_offset(0, 86, 86);
println!("offset={}", offset);
assert!(offset > 0);
let total_length = erasure.shard_file_size(86);
println!("total_length={}", total_length);
assert!(total_length > 0);
}
#[tokio::test]
@@ -685,16 +612,10 @@ mod tests {
let data_shards = 4;
let parity_shards = 2;
// Use different block sizes based on feature
#[cfg(feature = "reed-solomon-simd")]
let block_size = 1024; // SIMD mode
#[cfg(not(feature = "reed-solomon-simd"))]
let block_size = 8; // Pure erasure mode (default)
let erasure = Arc::new(Erasure::new(data_shards, parity_shards, block_size));
// Use test data suitable for both modes
// Use test data suitable for SIMD mode
let data =
b"Async error test data with sufficient length to meet requirements for proper testing and validation.".repeat(20); // ~2KB
@@ -728,13 +649,7 @@ mod tests {
let data_shards = 4;
let parity_shards = 2;
// Use different block sizes based on feature
#[cfg(feature = "reed-solomon-simd")]
let block_size = 1024; // SIMD mode
#[cfg(not(feature = "reed-solomon-simd"))]
let block_size = 8; // Pure erasure mode (default)
let erasure = Arc::new(Erasure::new(data_shards, parity_shards, block_size));
// Use test data that fits in exactly one block to avoid multi-block complexity
@@ -742,8 +657,6 @@ mod tests {
b"Channel async callback test data with sufficient length to ensure proper operation and validation requirements."
.repeat(8); // ~1KB
// let data = b"callback".to_vec(); // 8 bytes to fit exactly in one 8-byte block
let data_clone = data.clone(); // Clone for later comparison
let mut reader = Cursor::new(data);
let (tx, mut rx) = mpsc::channel::<Vec<Bytes>>(8);
@@ -782,8 +695,7 @@ mod tests {
assert_eq!(&recovered, &data_clone);
}
// Tests specifically for SIMD mode
#[cfg(feature = "reed-solomon-simd")]
// SIMD mode specific tests
mod simd_tests {
use super::*;
@@ -1152,47 +1064,4 @@ mod tests {
assert_eq!(&recovered, &data_clone);
}
}
// Comparative tests between different implementations
#[cfg(not(feature = "reed-solomon-simd"))]
mod comparative_tests {
use super::*;
#[test]
fn test_implementation_consistency() {
let data_shards = 4;
let parity_shards = 2;
let block_size = 2048; // Large enough for SIMD requirements
// Create test data that ensures each shard is >= 512 bytes (SIMD minimum)
let test_data = b"This is test data for comparing reed-solomon-simd and reed-solomon-erasure implementations to ensure they produce consistent results when given the same input parameters and data. This data needs to be sufficiently large to meet SIMD requirements.";
let data = test_data.repeat(50); // Create much larger data: ~13KB total, ~3.25KB per shard
// Test with erasure implementation (default)
let erasure_erasure = Erasure::new(data_shards, parity_shards, block_size);
let erasure_shards = erasure_erasure.encode_data(&data).unwrap();
// Test data integrity with erasure
let mut erasure_shards_opt: Vec<Option<Vec<u8>>> = erasure_shards.iter().map(|shard| Some(shard.to_vec())).collect();
// Lose some shards
erasure_shards_opt[1] = None; // Data shard
erasure_shards_opt[4] = None; // Parity shard
erasure_erasure.decode_data(&mut erasure_shards_opt).unwrap();
let mut erasure_recovered = Vec::new();
for shard in erasure_shards_opt.iter().take(data_shards) {
erasure_recovered.extend_from_slice(shard.as_ref().unwrap());
}
erasure_recovered.truncate(data.len());
// Verify erasure implementation works correctly
assert_eq!(&erasure_recovered, &data, "Erasure implementation failed to recover data correctly");
println!("✅ Both implementations are available and working correctly");
println!("✅ Default (reed-solomon-erasure): Data recovery successful");
println!("✅ SIMD tests are available as separate test suite");
}
}
}
ecstore/src/global.rs
+41 -9
View File
@@ -1,12 +1,3 @@
use lazy_static::lazy_static;
use std::{
collections::HashMap,
sync::{Arc, OnceLock},
time::SystemTime,
};
use tokio::sync::{OnceCell, RwLock};
use uuid::Uuid;
use crate::heal::mrf::MRFState;
use crate::{
bucket::lifecycle::bucket_lifecycle_ops::LifecycleSys,
@@ -17,6 +8,15 @@ use crate::{
store::ECStore,
tier::tier::TierConfigMgr,
};
use lazy_static::lazy_static;
use policy::auth::Credentials;
use std::{
collections::HashMap,
sync::{Arc, OnceLock},
time::SystemTime,
};
use tokio::sync::{OnceCell, RwLock};
use uuid::Uuid;
pub const DISK_ASSUME_UNKNOWN_SIZE: u64 = 1 << 30;
pub const DISK_MIN_INODES: u64 = 1000;
@@ -50,6 +50,38 @@ pub static ref GLOBAL_LocalNodeName: String = "127.0.0.1:9000".to_string();
pub static ref GLOBAL_LocalNodeNameHex: String = rustfs_utils::crypto::hex(GLOBAL_LocalNodeName.as_bytes());
pub static ref GLOBAL_NodeNamesHex: HashMap<String, ()> = HashMap::new();}
static GLOBAL_ACTIVE_CRED: OnceLock<Credentials> = OnceLock::new();
pub fn init_global_action_cred(ak: Option<String>, sk: Option<String>) {
let ak = {
if let Some(k) = ak {
k
} else {
rustfs_utils::string::gen_access_key(20).unwrap_or_default()
}
};
let sk = {
if let Some(k) = sk {
k
} else {
rustfs_utils::string::gen_secret_key(32).unwrap_or_default()
}
};
GLOBAL_ACTIVE_CRED
.set(Credentials {
access_key: ak,
secret_key: sk,
..Default::default()
})
.unwrap();
}
pub fn get_global_action_cred() -> Option<Credentials> {
GLOBAL_ACTIVE_CRED.get().cloned()
}
/// Get the global rustfs port
pub fn global_rustfs_port() -> u16 {
if let Some(p) = GLOBAL_RUSTFS_PORT.get() {
ecstore/src/heal/data_scanner.rs
+10 -10
View File
@@ -63,8 +63,8 @@ use crate::{
heal_ops::{BG_HEALING_UUID, HealSource},
},
new_object_layer_fn,
peer::is_reserved_or_invalid_bucket,
store::ECStore,
store_utils::is_reserved_or_invalid_bucket,
};
use crate::{disk::DiskAPI, store_api::ObjectInfo};
use crate::{
@@ -612,7 +612,7 @@ impl ScannerItem {
cumulative_size += obj_info.size;
}
if cumulative_size >= SCANNER_EXCESS_OBJECT_VERSIONS_TOTAL_SIZE.load(Ordering::SeqCst) as usize {
if cumulative_size >= SCANNER_EXCESS_OBJECT_VERSIONS_TOTAL_SIZE.load(Ordering::SeqCst) as i64 {
//todo
}
@@ -718,7 +718,7 @@ impl ScannerItem {
Ok(object_infos)
}
pub async fn apply_actions(&mut self, oi: &ObjectInfo, _size_s: &mut SizeSummary) -> (bool, usize) {
pub async fn apply_actions(&mut self, oi: &ObjectInfo, _size_s: &mut SizeSummary) -> (bool, i64) {
let done = ScannerMetrics::time(ScannerMetric::Ilm);
let (action, size) = self.apply_lifecycle(oi).await;
@@ -807,21 +807,21 @@ impl ScannerItem {
match tgt_status {
ReplicationStatusType::Pending => {
tgt_size_s.pending_count += 1;
tgt_size_s.pending_size += oi.size;
tgt_size_s.pending_size += oi.size as usize;
size_s.pending_count += 1;
size_s.pending_size += oi.size;
size_s.pending_size += oi.size as usize;
}
ReplicationStatusType::Failed => {
tgt_size_s.failed_count += 1;
tgt_size_s.failed_size += oi.size;
tgt_size_s.failed_size += oi.size as usize;
size_s.failed_count += 1;
size_s.failed_size += oi.size;
size_s.failed_size += oi.size as usize;
}
ReplicationStatusType::Completed | ReplicationStatusType::CompletedLegacy => {
tgt_size_s.replicated_count += 1;
tgt_size_s.replicated_size += oi.size;
tgt_size_s.replicated_size += oi.size as usize;
size_s.replicated_count += 1;
size_s.replicated_size += oi.size;
size_s.replicated_size += oi.size as usize;
}
_ => {}
}
@@ -829,7 +829,7 @@ impl ScannerItem {
if matches!(oi.replication_status, ReplicationStatusType::Replica) {
size_s.replica_count += 1;
size_s.replica_size += oi.size;
size_s.replica_size += oi.size as usize;
}
}
}
ecstore/src/heal/heal_commands.rs
+1 -1
View File
@@ -232,7 +232,7 @@ impl HealingTracker {
if let Some(disk) = &self.disk {
let file_path = Path::new(BUCKET_META_PREFIX).join(HEALING_TRACKER_FILENAME);
disk.write_all(RUSTFS_META_BUCKET, file_path.to_str().unwrap(), htracker_bytes)
disk.write_all(RUSTFS_META_BUCKET, file_path.to_str().unwrap(), htracker_bytes.into())
.await?;
}
Ok(())
ecstore/src/lib.rs
+5 -3
View File
@@ -1,9 +1,12 @@
extern crate core;
pub mod admin_server_info;
pub mod bitrot;
pub mod bucket;
pub mod cache_value;
mod chunk_stream;
pub mod cmd;
pub mod compress;
pub mod config;
pub mod disk;
pub mod disks_layout;
@@ -14,17 +17,16 @@ pub mod global;
pub mod heal;
pub mod metrics_realtime;
pub mod notification_sys;
pub mod peer;
pub mod peer_rest_client;
pub mod pools;
pub mod rebalance;
pub mod rpc;
pub mod set_disk;
mod sets;
pub mod store;
pub mod store_api;
mod store_init;
pub mod store_list_objects;
mod store_utils;
pub mod store_utils;
pub mod checksum;
pub mod client;
ecstore/src/notification_sys.rs
+13 -2
View File
@@ -2,7 +2,7 @@ use crate::StorageAPI;
use crate::admin_server_info::get_commit_id;
use crate::error::{Error, Result};
use crate::global::{GLOBAL_BOOT_TIME, get_global_endpoints};
use crate::peer_rest_client::PeerRestClient;
use crate::rpc::PeerRestClient;
use crate::{endpoints::EndpointServerPools, new_object_layer_fn};
use futures::future::join_all;
use lazy_static::lazy_static;
@@ -143,7 +143,11 @@ impl NotificationSys {
#[tracing::instrument(skip(self))]
pub async fn load_rebalance_meta(&self, start: bool) {
let mut futures = Vec::with_capacity(self.peer_clients.len());
for client in self.peer_clients.iter().flatten() {
for (i, client) in self.peer_clients.iter().flatten().enumerate() {
warn!(
"notification load_rebalance_meta start: {}, index: {}, client: {:?}",
start, i, client.host
);
futures.push(client.load_rebalance_meta(start));
}
@@ -158,11 +162,16 @@ impl NotificationSys {
}
pub async fn stop_rebalance(&self) {
warn!("notification stop_rebalance start");
let Some(store) = new_object_layer_fn() else {
error!("stop_rebalance: not init");
return;
};
// warn!("notification stop_rebalance load_rebalance_meta");
// self.load_rebalance_meta(false).await;
// warn!("notification stop_rebalance load_rebalance_meta done");
let mut futures = Vec::with_capacity(self.peer_clients.len());
for client in self.peer_clients.iter().flatten() {
futures.push(client.stop_rebalance());
@@ -175,7 +184,9 @@ impl NotificationSys {
}
}
warn!("notification stop_rebalance stop_rebalance start");
let _ = store.stop_rebalance().await;
warn!("notification stop_rebalance stop_rebalance done");
}
}
ecstore/src/pools.rs
+6 -6
View File
@@ -24,7 +24,7 @@ use futures::future::BoxFuture;
use http::HeaderMap;
use rmp_serde::{Deserializer, Serializer};
use rustfs_filemeta::{MetaCacheEntries, MetaCacheEntry, MetadataResolutionParams};
use rustfs_rio::HashReader;
use rustfs_rio::{HashReader, WarpReader};
use rustfs_utils::path::{SLASH_SEPARATOR, encode_dir_object, path_join};
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
@@ -33,7 +33,7 @@ use std::io::{Cursor, Write};
use std::path::PathBuf;
use std::sync::Arc;
use time::{Duration, OffsetDateTime};
use tokio::io::AsyncReadExt;
use tokio::io::{AsyncReadExt, BufReader};
use tokio::sync::broadcast::Receiver as B_Receiver;
use tracing::{error, info, warn};
@@ -1254,6 +1254,7 @@ impl ECStore {
}
if let Err(err) = self
.clone()
.complete_multipart_upload(
&bucket,
&object_info.name,
@@ -1275,10 +1276,9 @@ impl ECStore {
return Ok(());
}
let mut data = PutObjReader::new(
HashReader::new(rd.stream, object_info.size as i64, object_info.size as i64, None, false)?,
object_info.size,
);
let reader = BufReader::new(rd.stream);
let hrd = HashReader::new(Box::new(WarpReader::new(reader)), object_info.size, object_info.size, None, false)?;
let mut data = PutObjReader::new(hrd);
if let Err(err) = self
.put_object(
ecstore/src/rebalance.rs
+226 -136
View File
@@ -1,7 +1,3 @@
use std::io::Cursor;
use std::sync::Arc;
use std::time::SystemTime;
use crate::StorageAPI;
use crate::cache_value::metacache_set::{ListPathRawOptions, list_path_raw};
use crate::config::com::{read_config_with_metadata, save_config_with_opts};
@@ -16,19 +12,21 @@ use crate::store_api::{CompletePart, GetObjectReader, ObjectIO, ObjectOptions, P
use common::defer;
use http::HeaderMap;
use rustfs_filemeta::{FileInfo, MetaCacheEntries, MetaCacheEntry, MetadataResolutionParams};
use rustfs_rio::HashReader;
use rustfs_rio::{HashReader, WarpReader};
use rustfs_utils::path::encode_dir_object;
use serde::{Deserialize, Serialize};
use tokio::io::AsyncReadExt;
use std::io::Cursor;
use std::sync::Arc;
use time::OffsetDateTime;
use tokio::io::{AsyncReadExt, BufReader};
use tokio::sync::broadcast::{self, Receiver as B_Receiver};
use tokio::time::{Duration, Instant};
use tracing::{error, info, warn};
use uuid::Uuid;
use workers::workers::Workers;
const REBAL_META_FMT: u16 = 1; // Replace with actual format value
const REBAL_META_VER: u16 = 1; // Replace with actual version value
const REBAL_META_NAME: &str = "rebalance_meta";
const REBAL_META_NAME: &str = "rebalance.bin";
#[derive(Debug, Default, Clone, Serialize, Deserialize)]
pub struct RebalanceStats {
@@ -64,7 +62,7 @@ impl RebalanceStats {
self.num_versions += 1;
let on_disk_size = if !fi.deleted {
fi.size as i64 * (fi.erasure.data_blocks + fi.erasure.parity_blocks) as i64 / fi.erasure.data_blocks as i64
fi.size * (fi.erasure.data_blocks + fi.erasure.parity_blocks) as i64 / fi.erasure.data_blocks as i64
} else {
0
};
@@ -123,9 +121,9 @@ pub enum RebalSaveOpt {
#[derive(Debug, Default, Clone, Serialize, Deserialize)]
pub struct RebalanceInfo {
#[serde(rename = "startTs")]
pub start_time: Option<SystemTime>, // Time at which rebalance-start was issued
pub start_time: Option<OffsetDateTime>, // Time at which rebalance-start was issued
#[serde(rename = "stopTs")]
pub end_time: Option<SystemTime>, // Time at which rebalance operation completed or rebalance-stop was called
pub end_time: Option<OffsetDateTime>, // Time at which rebalance operation completed or rebalance-stop was called
#[serde(rename = "status")]
pub status: RebalStatus, // Current state of rebalance operation
}
@@ -137,14 +135,14 @@ pub struct DiskStat {
pub available_space: u64,
}
#[derive(Debug, Default, Serialize, Deserialize)]
#[derive(Debug, Default, Serialize, Deserialize, Clone)]
pub struct RebalanceMeta {
#[serde(skip)]
pub cancel: Option<broadcast::Sender<bool>>, // To be invoked on rebalance-stop
#[serde(skip)]
pub last_refreshed_at: Option<SystemTime>,
pub last_refreshed_at: Option<OffsetDateTime>,
#[serde(rename = "stopTs")]
pub stopped_at: Option<SystemTime>, // Time when rebalance-stop was issued
pub stopped_at: Option<OffsetDateTime>, // Time when rebalance-stop was issued
#[serde(rename = "id")]
pub id: String, // ID of the ongoing rebalance operation
#[serde(rename = "pf")]
@@ -164,29 +162,29 @@ impl RebalanceMeta {
pub async fn load_with_opts<S: StorageAPI>(&mut self, store: Arc<S>, opts: ObjectOptions) -> Result<()> {
let (data, _) = read_config_with_metadata(store, REBAL_META_NAME, &opts).await?;
if data.is_empty() {
warn!("rebalanceMeta: no data");
warn!("rebalanceMeta load_with_opts: no data");
return Ok(());
}
if data.len() <= 4 {
return Err(Error::other("rebalanceMeta: no data"));
return Err(Error::other("rebalanceMeta load_with_opts: no data"));
}
// Read header
match u16::from_le_bytes([data[0], data[1]]) {
REBAL_META_FMT => {}
fmt => return Err(Error::other(format!("rebalanceMeta: unknown format: {}", fmt))),
fmt => return Err(Error::other(format!("rebalanceMeta load_with_opts: unknown format: {}", fmt))),
}
match u16::from_le_bytes([data[2], data[3]]) {
REBAL_META_VER => {}
ver => return Err(Error::other(format!("rebalanceMeta: unknown version: {}", ver))),
ver => return Err(Error::other(format!("rebalanceMeta load_with_opts: unknown version: {}", ver))),
}
let meta: Self = rmp_serde::from_read(Cursor::new(&data[4..]))?;
*self = meta;
self.last_refreshed_at = Some(SystemTime::now());
self.last_refreshed_at = Some(OffsetDateTime::now_utc());
warn!("rebalanceMeta: loaded meta done");
warn!("rebalanceMeta load_with_opts: loaded meta done");
Ok(())
}
@@ -196,6 +194,7 @@ impl RebalanceMeta {
pub async fn save_with_opts<S: StorageAPI>(&self, store: Arc<S>, opts: ObjectOptions) -> Result<()> {
if self.pool_stats.is_empty() {
warn!("rebalanceMeta save_with_opts: no pool stats");
return Ok(());
}
@@ -218,7 +217,7 @@ impl ECStore {
#[tracing::instrument(skip_all)]
pub async fn load_rebalance_meta(&self) -> Result<()> {
let mut meta = RebalanceMeta::new();
warn!("rebalanceMeta: load rebalance meta");
warn!("rebalanceMeta: store load rebalance meta");
match meta.load(self.pools[0].clone()).await {
Ok(_) => {
warn!("rebalanceMeta: rebalance meta loaded0");
@@ -244,7 +243,7 @@ impl ECStore {
return Err(err);
}
error!("rebalanceMeta: not found, rebalance not started");
warn!("rebalanceMeta: not found, rebalance not started");
}
}
@@ -255,9 +254,18 @@ impl ECStore {
pub async fn update_rebalance_stats(&self) -> Result<()> {
let mut ok = false;
let pool_stats = {
let rebalance_meta = self.rebalance_meta.read().await;
rebalance_meta.as_ref().map(|v| v.pool_stats.clone()).unwrap_or_default()
};
warn!("update_rebalance_stats: pool_stats: {:?}", &pool_stats);
for i in 0..self.pools.len() {
if self.find_index(i).await.is_none() {
if pool_stats.get(i).is_none() {
warn!("update_rebalance_stats: pool {} not found", i);
let mut rebalance_meta = self.rebalance_meta.write().await;
warn!("update_rebalance_stats: pool {} not found, add", i);
if let Some(meta) = rebalance_meta.as_mut() {
meta.pool_stats.push(RebalanceStats::default());
}
@@ -267,23 +275,24 @@ impl ECStore {
}
if ok {
let mut rebalance_meta = self.rebalance_meta.write().await;
if let Some(meta) = rebalance_meta.as_mut() {
warn!("update_rebalance_stats: save rebalance meta");
let rebalance_meta = self.rebalance_meta.read().await;
if let Some(meta) = rebalance_meta.as_ref() {
meta.save(self.pools[0].clone()).await?;
}
drop(rebalance_meta);
}
Ok(())
}
async fn find_index(&self, index: usize) -> Option<usize> {
if let Some(meta) = self.rebalance_meta.read().await.as_ref() {
return meta.pool_stats.get(index).map(|_v| index);
}
// async fn find_index(&self, index: usize) -> Option<usize> {
// if let Some(meta) = self.rebalance_meta.read().await.as_ref() {
// return meta.pool_stats.get(index).map(|_v| index);
// }
None
}
// None
// }
#[tracing::instrument(skip(self))]
pub async fn init_rebalance_meta(&self, bucktes: Vec<String>) -> Result<String> {
@@ -310,7 +319,7 @@ impl ECStore {
let mut pool_stats = Vec::with_capacity(self.pools.len());
let now = SystemTime::now();
let now = OffsetDateTime::now_utc();
for disk_stat in disk_stats.iter() {
let mut pool_stat = RebalanceStats {
@@ -369,20 +378,26 @@ impl ECStore {
#[tracing::instrument(skip(self))]
pub async fn next_rebal_bucket(&self, pool_index: usize) -> Result<Option<String>> {
warn!("next_rebal_bucket: pool_index: {}", pool_index);
let rebalance_meta = self.rebalance_meta.read().await;
warn!("next_rebal_bucket: rebalance_meta: {:?}", rebalance_meta);
if let Some(meta) = rebalance_meta.as_ref() {
if let Some(pool_stat) = meta.pool_stats.get(pool_index) {
if pool_stat.info.status == RebalStatus::Completed || !pool_stat.participating {
warn!("next_rebal_bucket: pool_index: {} completed or not participating", pool_index);
return Ok(None);
}
if pool_stat.buckets.is_empty() {
warn!("next_rebal_bucket: pool_index: {} buckets is empty", pool_index);
return Ok(None);
}
warn!("next_rebal_bucket: pool_index: {} bucket: {}", pool_index, pool_stat.buckets[0]);
return Ok(Some(pool_stat.buckets[0].clone()));
}
}
warn!("next_rebal_bucket: pool_index: {} None", pool_index);
Ok(None)
}
@@ -392,18 +407,28 @@ impl ECStore {
if let Some(meta) = rebalance_meta.as_mut() {
if let Some(pool_stat) = meta.pool_stats.get_mut(pool_index) {
warn!("bucket_rebalance_done: buckets {:?}", &pool_stat.buckets);
if let Some(idx) = pool_stat.buckets.iter().position(|b| b.as_str() == bucket.as_str()) {
warn!("bucket_rebalance_done: bucket {} rebalanced", &bucket);
pool_stat.buckets.remove(idx);
pool_stat.rebalanced_buckets.push(bucket);
// 使用 retain 来过滤掉要删除的 bucket
let mut found = false;
pool_stat.buckets.retain(|b| {
if b.as_str() == bucket.as_str() {
found = true;
pool_stat.rebalanced_buckets.push(b.clone());
false // 删除这个元素
} else {
true // 保留这个元素
}
});
if found {
warn!("bucket_rebalance_done: bucket {} rebalanced", &bucket);
return Ok(());
} else {
warn!("bucket_rebalance_done: bucket {} not found", bucket);
}
}
}
warn!("bucket_rebalance_done: bucket {} not found", bucket);
Ok(())
}
@@ -411,18 +436,28 @@ impl ECStore {
let rebalance_meta = self.rebalance_meta.read().await;
if let Some(ref meta) = *rebalance_meta {
if meta.stopped_at.is_some() {
warn!("is_rebalance_started: rebalance stopped");
return false;
}
meta.pool_stats.iter().enumerate().for_each(|(i, v)| {
warn!(
"is_rebalance_started: pool_index: {}, participating: {:?}, status: {:?}",
i, v.participating, v.info.status
);
});
if meta
.pool_stats
.iter()
.any(|v| v.participating && v.info.status != RebalStatus::Completed)
{
warn!("is_rebalance_started: rebalance started");
return true;
}
}
warn!("is_rebalance_started: rebalance not started");
false
}
@@ -462,10 +497,11 @@ impl ECStore {
{
let mut rebalance_meta = self.rebalance_meta.write().await;
if let Some(meta) = rebalance_meta.as_mut() {
meta.cancel = Some(tx)
} else {
error!("start_rebalance: rebalance_meta is None exit");
warn!("start_rebalance: rebalance_meta is None exit");
return;
}
@@ -474,19 +510,25 @@ impl ECStore {
let participants = {
if let Some(ref meta) = *self.rebalance_meta.read().await {
if meta.stopped_at.is_some() {
warn!("start_rebalance: rebalance already stopped exit");
return;
}
// if meta.stopped_at.is_some() {
// warn!("start_rebalance: rebalance already stopped exit");
// return;
// }
let mut participants = vec![false; meta.pool_stats.len()];
for (i, pool_stat) in meta.pool_stats.iter().enumerate() {
if pool_stat.info.status == RebalStatus::Started {
participants[i] = pool_stat.participating;
warn!("start_rebalance: pool {} status: {:?}", i, pool_stat.info.status);
if pool_stat.info.status != RebalStatus::Started {
warn!("start_rebalance: pool {} not started, skipping", i);
continue;
}
warn!("start_rebalance: pool {} participating: {:?}", i, pool_stat.participating);
participants[i] = pool_stat.participating;
}
participants
} else {
warn!("start_rebalance:2 rebalance_meta is None exit");
Vec::new()
}
};
@@ -497,11 +539,13 @@ impl ECStore {
continue;
}
if get_global_endpoints()
.as_ref()
.get(idx)
.is_none_or(|v| v.endpoints.as_ref().first().is_none_or(|e| e.is_local))
{
if !get_global_endpoints().as_ref().get(idx).is_some_and(|v| {
warn!("start_rebalance: pool {} endpoints: {:?}", idx, v.endpoints);
v.endpoints.as_ref().first().is_some_and(|e| {
warn!("start_rebalance: pool {} endpoint: {:?}, is_local: {}", idx, e, e.is_local);
e.is_local
})
}) {
warn!("start_rebalance: pool {} is not local, skipping", idx);
continue;
}
@@ -522,13 +566,13 @@ impl ECStore {
}
#[tracing::instrument(skip(self, rx))]
async fn rebalance_buckets(self: &Arc<Self>, rx: B_Receiver<bool>, pool_index: usize) -> Result<()> {
async fn rebalance_buckets(self: &Arc<Self>, mut rx: B_Receiver<bool>, pool_index: usize) -> Result<()> {
let (done_tx, mut done_rx) = tokio::sync::mpsc::channel::<Result<()>>(1);
// Save rebalance metadata periodically
let store = self.clone();
let save_task = tokio::spawn(async move {
let mut timer = tokio::time::interval_at(Instant::now() + Duration::from_secs(10), Duration::from_secs(10));
let mut timer = tokio::time::interval_at(Instant::now() + Duration::from_secs(30), Duration::from_secs(10));
let mut msg: String;
let mut quit = false;
@@ -537,14 +581,15 @@ impl ECStore {
// TODO: cancel rebalance
Some(result) = done_rx.recv() => {
quit = true;
let now = SystemTime::now();
let now = OffsetDateTime::now_utc();
let state = match result {
Ok(_) => {
warn!("rebalance_buckets: completed");
msg = format!("Rebalance completed at {:?}", now);
RebalStatus::Completed},
Err(err) => {
warn!("rebalance_buckets: error: {:?}", err);
// TODO: check stop
if err.to_string().contains("canceled") {
msg = format!("Rebalance stopped at {:?}", now);
@@ -557,9 +602,11 @@ impl ECStore {
};
{
warn!("rebalance_buckets: save rebalance meta, pool_index: {}, state: {:?}", pool_index, state);
let mut rebalance_meta = store.rebalance_meta.write().await;
if let Some(rbm) = rebalance_meta.as_mut() {
warn!("rebalance_buckets: save rebalance meta2, pool_index: {}, state: {:?}", pool_index, state);
rbm.pool_stats[pool_index].info.status = state;
rbm.pool_stats[pool_index].info.end_time = Some(now);
}
@@ -568,7 +615,7 @@ impl ECStore {
}
_ = timer.tick() => {
let now = SystemTime::now();
let now = OffsetDateTime::now_utc();
msg = format!("Saving rebalance metadata at {:?}", now);
}
}
@@ -576,7 +623,7 @@ impl ECStore {
if let Err(err) = store.save_rebalance_stats(pool_index, RebalSaveOpt::Stats).await {
error!("{} err: {:?}", msg, err);
} else {
info!(msg);
warn!(msg);
}
if quit {
@@ -588,30 +635,41 @@ impl ECStore {
}
});
warn!("Pool {} rebalancing is started", pool_index + 1);
warn!("Pool {} rebalancing is started", pool_index);
while let Some(bucket) = self.next_rebal_bucket(pool_index).await? {
warn!("Rebalancing bucket: start {}", bucket);
if let Err(err) = self.rebalance_bucket(rx.resubscribe(), bucket.clone(), pool_index).await {
if err.to_string().contains("not initialized") {
warn!("rebalance_bucket: rebalance not initialized, continue");
continue;
}
error!("Error rebalancing bucket {}: {:?}", bucket, err);
done_tx.send(Err(err)).await.ok();
loop {
if let Ok(true) = rx.try_recv() {
warn!("Pool {} rebalancing is stopped", pool_index);
done_tx.send(Err(Error::other("rebalance stopped canceled"))).await.ok();
break;
}
warn!("Rebalance bucket: done {} ", bucket);
self.bucket_rebalance_done(pool_index, bucket).await?;
if let Some(bucket) = self.next_rebal_bucket(pool_index).await? {
warn!("Rebalancing bucket: start {}", bucket);
if let Err(err) = self.rebalance_bucket(rx.resubscribe(), bucket.clone(), pool_index).await {
if err.to_string().contains("not initialized") {
warn!("rebalance_bucket: rebalance not initialized, continue");
continue;
}
error!("Error rebalancing bucket {}: {:?}", bucket, err);
done_tx.send(Err(err)).await.ok();
break;
}
warn!("Rebalance bucket: done {} ", bucket);
self.bucket_rebalance_done(pool_index, bucket).await?;
} else {
warn!("Rebalance bucket: no bucket to rebalance");
break;
}
}
warn!("Pool {} rebalancing is done", pool_index + 1);
warn!("Pool {} rebalancing is done", pool_index);
done_tx.send(Ok(())).await.ok();
save_task.await.ok();
warn!("Pool {} rebalancing is done2", pool_index);
Ok(())
}
@@ -622,6 +680,7 @@ impl ECStore {
if let Some(pool_stat) = meta.pool_stats.get_mut(pool_index) {
// Check if the pool's rebalance status is already completed
if pool_stat.info.status == RebalStatus::Completed {
warn!("check_if_rebalance_done: pool {} is already completed", pool_index);
return true;
}
@@ -631,7 +690,8 @@ impl ECStore {
// Mark pool rebalance as done if within 5% of the PercentFreeGoal
if (pfi - meta.percent_free_goal).abs() <= 0.05 {
pool_stat.info.status = RebalStatus::Completed;
pool_stat.info.end_time = Some(SystemTime::now());
pool_stat.info.end_time = Some(OffsetDateTime::now_utc());
warn!("check_if_rebalance_done: pool {} is completed, pfi: {}", pool_index, pfi);
return true;
}
}
@@ -641,24 +701,30 @@ impl ECStore {
}
#[allow(unused_assignments)]
#[tracing::instrument(skip(self, wk, set))]
#[tracing::instrument(skip(self, set))]
async fn rebalance_entry(
&self,
self: Arc<Self>,
bucket: String,
pool_index: usize,
entry: MetaCacheEntry,
set: Arc<SetDisks>,
wk: Arc<Workers>,
// wk: Arc<Workers>,
) {
defer!(|| async {
wk.give().await;
});
warn!("rebalance_entry: start rebalance_entry");
// defer!(|| async {
// warn!("rebalance_entry: defer give worker start");
// wk.give().await;
// warn!("rebalance_entry: defer give worker done");
// });
if entry.is_dir() {
warn!("rebalance_entry: entry is dir, skipping");
return;
}
if self.check_if_rebalance_done(pool_index).await {
warn!("rebalance_entry: rebalance done, skipping pool {}", pool_index);
return;
}
@@ -666,6 +732,7 @@ impl ECStore {
Ok(fivs) => fivs,
Err(err) => {
error!("rebalance_entry Error getting file info versions: {}", err);
warn!("rebalance_entry: Error getting file info versions, skipping");
return;
}
};
@@ -676,7 +743,7 @@ impl ECStore {
let expired: usize = 0;
for version in fivs.versions.iter() {
if version.is_remote() {
info!("rebalance_entry Entry {} is remote, skipping", version.name);
warn!("rebalance_entry Entry {} is remote, skipping", version.name);
continue;
}
// TODO: filterLifecycle
@@ -684,7 +751,7 @@ impl ECStore {
let remaining_versions = fivs.versions.len() - expired;
if version.deleted && remaining_versions == 1 {
rebalanced += 1;
info!("rebalance_entry Entry {} is deleted and last version, skipping", version.name);
warn!("rebalance_entry Entry {} is deleted and last version, skipping", version.name);
continue;
}
let version_id = version.version_id.map(|v| v.to_string());
@@ -735,6 +802,7 @@ impl ECStore {
}
for _i in 0..3 {
warn!("rebalance_entry: get_object_reader, bucket: {}, version: {}", &bucket, &version.name);
let rd = match set
.get_object_reader(
bucket.as_str(),
@@ -753,6 +821,10 @@ impl ECStore {
Err(err) => {
if is_err_object_not_found(&err) || is_err_version_not_found(&err) {
ignore = true;
warn!(
"rebalance_entry: get_object_reader, bucket: {}, version: {}, ignore",
&bucket, &version.name
);
break;
}
@@ -762,10 +834,10 @@ impl ECStore {
}
};
if let Err(err) = self.rebalance_object(pool_index, bucket.clone(), rd).await {
if let Err(err) = self.clone().rebalance_object(pool_index, bucket.clone(), rd).await {
if is_err_object_not_found(&err) || is_err_version_not_found(&err) || is_err_data_movement_overwrite(&err) {
ignore = true;
info!("rebalance_entry {} Entry {} is already deleted, skipping", &bucket, version.name);
warn!("rebalance_entry {} Entry {} is already deleted, skipping", &bucket, version.name);
break;
}
@@ -780,7 +852,7 @@ impl ECStore {
}
if ignore {
info!("rebalance_entry {} Entry {} is already deleted, skipping", &bucket, version.name);
warn!("rebalance_entry {} Entry {} is already deleted, skipping", &bucket, version.name);
continue;
}
@@ -812,13 +884,13 @@ impl ECStore {
{
error!("rebalance_entry: delete_object err {:?}", &err);
} else {
info!("rebalance_entry {} Entry {} deleted successfully", &bucket, &entry.name);
warn!("rebalance_entry {} Entry {} deleted successfully", &bucket, &entry.name);
}
}
}
#[tracing::instrument(skip(self, rd))]
async fn rebalance_object(&self, pool_idx: usize, bucket: String, rd: GetObjectReader) -> Result<()> {
async fn rebalance_object(self: Arc<Self>, pool_idx: usize, bucket: String, rd: GetObjectReader) -> Result<()> {
let object_info = rd.object_info.clone();
// TODO: check : use size or actual_size ?
@@ -897,6 +969,7 @@ impl ECStore {
}
if let Err(err) = self
.clone()
.complete_multipart_upload(
&bucket,
&object_info.name,
@@ -917,8 +990,9 @@ impl ECStore {
return Ok(());
}
let hrd = HashReader::new(rd.stream, object_info.size as i64, object_info.size as i64, None, false)?;
let mut data = PutObjReader::new(hrd, object_info.size);
let reader = BufReader::new(rd.stream);
let hrd = HashReader::new(Box::new(WarpReader::new(reader)), object_info.size, object_info.size, None, false)?;
let mut data = PutObjReader::new(hrd);
if let Err(err) = self
.put_object(
@@ -957,26 +1031,29 @@ impl ECStore {
let pool = self.pools[pool_index].clone();
let wk = Workers::new(pool.disk_set.len() * 2).map_err(Error::other)?;
let mut jobs = Vec::new();
// let wk = Workers::new(pool.disk_set.len() * 2).map_err(Error::other)?;
// wk.clone().take().await;
for (set_idx, set) in pool.disk_set.iter().enumerate() {
wk.clone().take().await;
let rebalance_entry: ListCallback = Arc::new({
let this = Arc::clone(self);
let bucket = bucket.clone();
let wk = wk.clone();
// let wk = wk.clone();
let set = set.clone();
move |entry: MetaCacheEntry| {
let this = this.clone();
let bucket = bucket.clone();
let wk = wk.clone();
// let wk = wk.clone();
let set = set.clone();
Box::pin(async move {
wk.take().await;
tokio::spawn(async move {
this.rebalance_entry(bucket, pool_index, entry, set, wk).await;
});
warn!("rebalance_entry: rebalance_entry spawn start");
// wk.take().await;
// tokio::spawn(async move {
warn!("rebalance_entry: rebalance_entry spawn start2");
this.rebalance_entry(bucket, pool_index, entry, set).await;
warn!("rebalance_entry: rebalance_entry spawn done");
// });
})
}
});
@@ -984,62 +1061,68 @@ impl ECStore {
let set = set.clone();
let rx = rx.resubscribe();
let bucket = bucket.clone();
let wk = wk.clone();
tokio::spawn(async move {
// let wk = wk.clone();
let job = tokio::spawn(async move {
if let Err(err) = set.list_objects_to_rebalance(rx, bucket, rebalance_entry).await {
error!("Rebalance worker {} error: {}", set_idx, err);
} else {
info!("Rebalance worker {} done", set_idx);
}
wk.clone().give().await;
// wk.clone().give().await;
});
jobs.push(job);
}
wk.wait().await;
// wk.wait().await;
for job in jobs {
job.await.unwrap();
}
warn!("rebalance_bucket: rebalance_bucket done");
Ok(())
}
#[tracing::instrument(skip(self))]
pub async fn save_rebalance_stats(&self, pool_idx: usize, opt: RebalSaveOpt) -> Result<()> {
// TODO: NSLOOK
// TODO: lock
let mut meta = RebalanceMeta::new();
meta.load_with_opts(
self.pools[0].clone(),
ObjectOptions {
no_lock: true,
..Default::default()
},
)
.await?;
if opt == RebalSaveOpt::StoppedAt {
meta.stopped_at = Some(SystemTime::now());
}
let mut rebalance_meta = self.rebalance_meta.write().await;
if let Some(rb) = rebalance_meta.as_mut() {
if opt == RebalSaveOpt::Stats {
meta.pool_stats[pool_idx] = rb.pool_stats[pool_idx].clone();
if let Err(err) = meta.load(self.pools[0].clone()).await {
if err != Error::ConfigNotFound {
warn!("save_rebalance_stats: load err: {:?}", err);
return Err(err);
}
*rb = meta;
} else {
*rebalance_meta = Some(meta);
}
if let Some(meta) = rebalance_meta.as_mut() {
meta.save_with_opts(
self.pools[0].clone(),
ObjectOptions {
no_lock: true,
..Default::default()
},
)
.await?;
match opt {
RebalSaveOpt::Stats => {
{
let mut rebalance_meta = self.rebalance_meta.write().await;
if let Some(rbm) = rebalance_meta.as_mut() {
meta.pool_stats[pool_idx] = rbm.pool_stats[pool_idx].clone();
}
}
if let Some(pool_stat) = meta.pool_stats.get_mut(pool_idx) {
pool_stat.info.end_time = Some(OffsetDateTime::now_utc());
}
}
RebalSaveOpt::StoppedAt => {
meta.stopped_at = Some(OffsetDateTime::now_utc());
}
}
{
let mut rebalance_meta = self.rebalance_meta.write().await;
*rebalance_meta = Some(meta.clone());
}
warn!(
"save_rebalance_stats: save rebalance meta, pool_idx: {}, opt: {:?}, meta: {:?}",
pool_idx, opt, meta
);
meta.save(self.pools[0].clone()).await?;
Ok(())
}
}
@@ -1052,12 +1135,15 @@ impl SetDisks {
bucket: String,
cb: ListCallback,
) -> Result<()> {
warn!("list_objects_to_rebalance: start list_objects_to_rebalance");
// Placeholder for actual object listing logic
let (disks, _) = self.get_online_disks_with_healing(false).await;
if disks.is_empty() {
warn!("list_objects_to_rebalance: no disk available");
return Err(Error::other("errNoDiskAvailable"));
}
warn!("list_objects_to_rebalance: get online disks with healing");
let listing_quorum = self.set_drive_count.div_ceil(2);
let resolver = MetadataResolutionParams {
@@ -1075,7 +1161,10 @@ impl SetDisks {
bucket: bucket.clone(),
recursice: true,
min_disks: listing_quorum,
agreed: Some(Box::new(move |entry: MetaCacheEntry| Box::pin(cb1(entry)))),
agreed: Some(Box::new(move |entry: MetaCacheEntry| {
warn!("list_objects_to_rebalance: agreed: {:?}", &entry.name);
Box::pin(cb1(entry))
})),
partial: Some(Box::new(move |entries: MetaCacheEntries, _: &[Option<DiskError>]| {
// let cb = cb.clone();
let resolver = resolver.clone();
@@ -1083,11 +1172,11 @@ impl SetDisks {
match entries.resolve(resolver) {
Some(entry) => {
warn!("rebalance: list_objects_to_decommission get {}", &entry.name);
warn!("list_objects_to_rebalance: list_objects_to_decommission get {}", &entry.name);
Box::pin(async move { cb(entry).await })
}
None => {
warn!("rebalance: list_objects_to_decommission get none");
warn!("list_objects_to_rebalance: list_objects_to_decommission get none");
Box::pin(async {})
}
}
@@ -1097,6 +1186,7 @@ impl SetDisks {
)
.await?;
warn!("list_objects_to_rebalance: list_objects_to_rebalance done");
Ok(())
}
}
ecstore/src/rpc/http_auth.rs
+375
View File
@@ -0,0 +1,375 @@
use crate::global::get_global_action_cred;
use base64::Engine as _;
use base64::engine::general_purpose;
use hmac::{Hmac, Mac};
use http::HeaderMap;
use http::HeaderValue;
use http::Method;
use http::Uri;
use sha2::Sha256;
use time::OffsetDateTime;
use tracing::error;
type HmacSha256 = Hmac<Sha256>;
const SIGNATURE_HEADER: &str = "x-rustfs-signature";
const TIMESTAMP_HEADER: &str = "x-rustfs-timestamp";
const SIGNATURE_VALID_DURATION: i64 = 300; // 5 minutes
/// Get the shared secret for HMAC signing
fn get_shared_secret() -> String {
if let Some(cred) = get_global_action_cred() {
cred.secret_key
} else {
// Fallback to environment variable if global credentials are not available
std::env::var("RUSTFS_RPC_SECRET").unwrap_or_else(|_| "rustfs-default-secret".to_string())
}
}
/// Generate HMAC-SHA256 signature for the given data
fn generate_signature(secret: &str, url: &str, method: &Method, timestamp: i64) -> String {
let uri: Uri = url.parse().expect("Invalid URL");
let path_and_query = uri.path_and_query().unwrap();
let url = path_and_query.to_string();
let data = format!("{}|{}|{}", url, method, timestamp);
let mut mac = HmacSha256::new_from_slice(secret.as_bytes()).expect("HMAC can take key of any size");
mac.update(data.as_bytes());
let result = mac.finalize();
general_purpose::STANDARD.encode(result.into_bytes())
}
/// Build headers with authentication signature
pub fn build_auth_headers(url: &str, method: &Method, headers: &mut HeaderMap) {
let secret = get_shared_secret();
let timestamp = OffsetDateTime::now_utc().unix_timestamp();
let signature = generate_signature(&secret, url, method, timestamp);
headers.insert(SIGNATURE_HEADER, HeaderValue::from_str(&signature).unwrap());
headers.insert(TIMESTAMP_HEADER, HeaderValue::from_str(&timestamp.to_string()).unwrap());
}
/// Verify the request signature for RPC requests
pub fn verify_rpc_signature(url: &str, method: &Method, headers: &HeaderMap) -> std::io::Result<()> {
let secret = get_shared_secret();
// Get signature from header
let signature = headers
.get(SIGNATURE_HEADER)
.and_then(|v| v.to_str().ok())
.ok_or_else(|| std::io::Error::other("Missing signature header"))?;
// Get timestamp from header
let timestamp_str = headers
.get(TIMESTAMP_HEADER)
.and_then(|v| v.to_str().ok())
.ok_or_else(|| std::io::Error::other("Missing timestamp header"))?;
let timestamp: i64 = timestamp_str
.parse()
.map_err(|_| std::io::Error::other("Invalid timestamp format"))?;
// Check timestamp validity (prevent replay attacks)
let current_time = OffsetDateTime::now_utc().unix_timestamp();
if current_time.saturating_sub(timestamp) > SIGNATURE_VALID_DURATION {
return Err(std::io::Error::other("Request timestamp expired"));
}
// Generate expected signature
let expected_signature = generate_signature(&secret, url, method, timestamp);
// Compare signatures
if signature != expected_signature {
error!(
"verify_rpc_signature: Invalid signature: secret {}, url {}, method {}, timestamp {}, signature {}, expected_signature {}",
secret, url, method, timestamp, signature, expected_signature
);
return Err(std::io::Error::other("Invalid signature"));
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use http::{HeaderMap, Method};
use time::OffsetDateTime;
#[test]
fn test_get_shared_secret() {
let secret = get_shared_secret();
assert!(!secret.is_empty(), "Secret should not be empty");
let url = "http://node1:7000/rustfs/rpc/read_file_stream?disk=http%3A%2F%2Fnode1%3A7000%2Fdata%2Frustfs3&volume=.rustfs.sys&path=pool.bin%2Fdd0fd773-a962-4265-b543-783ce83953e9%2Fpart.1&offset=0&length=44";
let method = Method::GET;
let mut headers = HeaderMap::new();
build_auth_headers(url, &method, &mut headers);
let url = "/rustfs/rpc/read_file_stream?disk=http%3A%2F%2Fnode1%3A7000%2Fdata%2Frustfs3&volume=.rustfs.sys&path=pool.bin%2Fdd0fd773-a962-4265-b543-783ce83953e9%2Fpart.1&offset=0&length=44";
let result = verify_rpc_signature(url, &method, &headers);
assert!(result.is_ok(), "Valid signature should pass verification");
}
#[test]
fn test_generate_signature_deterministic() {
let secret = "test-secret";
let url = "http://example.com/api/test";
let method = Method::GET;
let timestamp = 1640995200; // Fixed timestamp
let signature1 = generate_signature(secret, url, &method, timestamp);
let signature2 = generate_signature(secret, url, &method, timestamp);
assert_eq!(signature1, signature2, "Same inputs should produce same signature");
assert!(!signature1.is_empty(), "Signature should not be empty");
}
#[test]
fn test_generate_signature_different_inputs() {
let secret = "test-secret";
let url = "http://example.com/api/test";
let method = Method::GET;
let timestamp = 1640995200;
let signature1 = generate_signature(secret, url, &method, timestamp);
let signature2 = generate_signature(secret, "http://different.com/api/test2", &method, timestamp);
let signature3 = generate_signature(secret, url, &Method::POST, timestamp);
let signature4 = generate_signature(secret, url, &method, timestamp + 1);
assert_ne!(signature1, signature2, "Different URLs should produce different signatures");
assert_ne!(signature1, signature3, "Different methods should produce different signatures");
assert_ne!(signature1, signature4, "Different timestamps should produce different signatures");
}
#[test]
fn test_build_auth_headers() {
let url = "http://example.com/api/test";
let method = Method::POST;
let mut headers = HeaderMap::new();
build_auth_headers(url, &method, &mut headers);
// Verify headers are present
assert!(headers.contains_key(SIGNATURE_HEADER), "Should contain signature header");
assert!(headers.contains_key(TIMESTAMP_HEADER), "Should contain timestamp header");
// Verify header values are not empty
let signature = headers.get(SIGNATURE_HEADER).unwrap().to_str().unwrap();
let timestamp_str = headers.get(TIMESTAMP_HEADER).unwrap().to_str().unwrap();
assert!(!signature.is_empty(), "Signature should not be empty");
assert!(!timestamp_str.is_empty(), "Timestamp should not be empty");
// Verify timestamp is a valid integer
let timestamp: i64 = timestamp_str.parse().expect("Timestamp should be valid integer");
let current_time = OffsetDateTime::now_utc().unix_timestamp();
// Should be within a reasonable range (within 1 second of current time)
assert!((current_time - timestamp).abs() <= 1, "Timestamp should be close to current time");
}
#[test]
fn test_verify_rpc_signature_success() {
let url = "http://example.com/api/test";
let method = Method::GET;
let mut headers = HeaderMap::new();
// Build headers with valid signature
build_auth_headers(url, &method, &mut headers);
// Verify should succeed
let result = verify_rpc_signature(url, &method, &headers);
assert!(result.is_ok(), "Valid signature should pass verification");
}
#[test]
fn test_verify_rpc_signature_invalid_signature() {
let url = "http://example.com/api/test";
let method = Method::GET;
let mut headers = HeaderMap::new();
// Build headers with valid signature first
build_auth_headers(url, &method, &mut headers);
// Tamper with the signature
headers.insert(SIGNATURE_HEADER, HeaderValue::from_str("invalid-signature").unwrap());
// Verify should fail
let result = verify_rpc_signature(url, &method, &headers);
assert!(result.is_err(), "Invalid signature should fail verification");
let error = result.unwrap_err();
assert_eq!(error.to_string(), "Invalid signature");
}
#[test]
fn test_verify_rpc_signature_expired_timestamp() {
let url = "http://example.com/api/test";
let method = Method::GET;
let mut headers = HeaderMap::new();
// Set expired timestamp (older than SIGNATURE_VALID_DURATION)
let expired_timestamp = OffsetDateTime::now_utc().unix_timestamp() - SIGNATURE_VALID_DURATION - 10;
let secret = get_shared_secret();
let signature = generate_signature(&secret, url, &method, expired_timestamp);
headers.insert(SIGNATURE_HEADER, HeaderValue::from_str(&signature).unwrap());
headers.insert(TIMESTAMP_HEADER, HeaderValue::from_str(&expired_timestamp.to_string()).unwrap());
// Verify should fail due to expired timestamp
let result = verify_rpc_signature(url, &method, &headers);
assert!(result.is_err(), "Expired timestamp should fail verification");
let error = result.unwrap_err();
assert_eq!(error.to_string(), "Request timestamp expired");
}
#[test]
fn test_verify_rpc_signature_missing_signature_header() {
let url = "http://example.com/api/test";
let method = Method::GET;
let mut headers = HeaderMap::new();
// Add only timestamp header, missing signature
let timestamp = OffsetDateTime::now_utc().unix_timestamp();
headers.insert(TIMESTAMP_HEADER, HeaderValue::from_str(&timestamp.to_string()).unwrap());
// Verify should fail
let result = verify_rpc_signature(url, &method, &headers);
assert!(result.is_err(), "Missing signature header should fail verification");
let error = result.unwrap_err();
assert_eq!(error.to_string(), "Missing signature header");
}
#[test]
fn test_verify_rpc_signature_missing_timestamp_header() {
let url = "http://example.com/api/test";
let method = Method::GET;
let mut headers = HeaderMap::new();
// Add only signature header, missing timestamp
headers.insert(SIGNATURE_HEADER, HeaderValue::from_str("some-signature").unwrap());
// Verify should fail
let result = verify_rpc_signature(url, &method, &headers);
assert!(result.is_err(), "Missing timestamp header should fail verification");
let error = result.unwrap_err();
assert_eq!(error.to_string(), "Missing timestamp header");
}
#[test]
fn test_verify_rpc_signature_invalid_timestamp_format() {
let url = "http://example.com/api/test";
let method = Method::GET;
let mut headers = HeaderMap::new();
headers.insert(SIGNATURE_HEADER, HeaderValue::from_str("some-signature").unwrap());
headers.insert(TIMESTAMP_HEADER, HeaderValue::from_str("invalid-timestamp").unwrap());
// Verify should fail
let result = verify_rpc_signature(url, &method, &headers);
assert!(result.is_err(), "Invalid timestamp format should fail verification");
let error = result.unwrap_err();
assert_eq!(error.to_string(), "Invalid timestamp format");
}
#[test]
fn test_verify_rpc_signature_url_mismatch() {
let original_url = "http://example.com/api/test";
let different_url = "http://example.com/api/different";
let method = Method::GET;
let mut headers = HeaderMap::new();
// Build headers for one URL
build_auth_headers(original_url, &method, &mut headers);
// Try to verify with a different URL
let result = verify_rpc_signature(different_url, &method, &headers);
assert!(result.is_err(), "URL mismatch should fail verification");
let error = result.unwrap_err();
assert_eq!(error.to_string(), "Invalid signature");
}
#[test]
fn test_verify_rpc_signature_method_mismatch() {
let url = "http://example.com/api/test";
let original_method = Method::GET;
let different_method = Method::POST;
let mut headers = HeaderMap::new();
// Build headers for one method
build_auth_headers(url, &original_method, &mut headers);
// Try to verify with a different method
let result = verify_rpc_signature(url, &different_method, &headers);
assert!(result.is_err(), "Method mismatch should fail verification");
let error = result.unwrap_err();
assert_eq!(error.to_string(), "Invalid signature");
}
#[test]
fn test_signature_valid_duration_boundary() {
let url = "http://example.com/api/test";
let method = Method::GET;
let secret = get_shared_secret();
let mut headers = HeaderMap::new();
let current_time = OffsetDateTime::now_utc().unix_timestamp();
// Test timestamp just within valid duration
let valid_timestamp = current_time - SIGNATURE_VALID_DURATION + 1;
let signature = generate_signature(&secret, url, &method, valid_timestamp);
headers.insert(SIGNATURE_HEADER, HeaderValue::from_str(&signature).unwrap());
headers.insert(TIMESTAMP_HEADER, HeaderValue::from_str(&valid_timestamp.to_string()).unwrap());
let result = verify_rpc_signature(url, &method, &headers);
assert!(result.is_ok(), "Timestamp within valid duration should pass");
// Test timestamp just outside valid duration
let mut headers = HeaderMap::new();
let invalid_timestamp = current_time - SIGNATURE_VALID_DURATION - 15;
let signature = generate_signature(&secret, url, &method, invalid_timestamp);
headers.insert(SIGNATURE_HEADER, HeaderValue::from_str(&signature).unwrap());
headers.insert(TIMESTAMP_HEADER, HeaderValue::from_str(&invalid_timestamp.to_string()).unwrap());
let result = verify_rpc_signature(url, &method, &headers);
assert!(result.is_err(), "Timestamp outside valid duration should fail");
}
#[test]
fn test_round_trip_authentication() {
let test_cases = vec![
("http://example.com/api/test", Method::GET),
("https://api.rustfs.com/v1/bucket", Method::POST),
("http://localhost:9000/admin/info", Method::PUT),
("https://storage.example.com/path/to/object?query=param", Method::DELETE),
];
for (url, method) in test_cases {
let mut headers = HeaderMap::new();
// Build authentication headers
build_auth_headers(url, &method, &mut headers);
// Verify the signature should succeed
let result = verify_rpc_signature(url, &method, &headers);
assert!(result.is_ok(), "Round-trip test failed for {} {}", method, url);
}
}
}
ecstore/src/rpc/mod.rs
+11
View File
@@ -0,0 +1,11 @@
mod http_auth;
mod peer_rest_client;
mod peer_s3_client;
mod remote_disk;
mod tonic_service;
pub use http_auth::{build_auth_headers, verify_rpc_signature};
pub use peer_rest_client::PeerRestClient;
pub use peer_s3_client::{LocalPeerS3Client, PeerS3Client, RemotePeerS3Client, S3PeerSys};
pub use remote_disk::RemoteDisk;
pub use tonic_service::make_server;
ecstore/src/peer_rest_client.rs → ecstore/src/rpc/peer_rest_client.rs
+3 -3
View File
@@ -292,8 +292,8 @@ impl PeerRestClient {
let mut buf_o = Vec::new();
opts.serialize(&mut Serializer::new(&mut buf_o))?;
let request = Request::new(GetMetricsRequest {
metric_type: buf_t,
opts: buf_o,
metric_type: buf_t.into(),
opts: buf_o.into(),
});
let response = client.get_metrics(request).await?.into_inner();
@@ -664,7 +664,7 @@ impl PeerRestClient {
let response = client.load_rebalance_meta(request).await?.into_inner();
warn!("load_rebalance_meta response {:?}", response);
warn!("load_rebalance_meta response {:?}, grid_host: {:?}", response, &self.grid_host);
if !response.success {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
ecstore/src/peer.rs → ecstore/src/rpc/peer_s3_client.rs
+1 -58
View File
@@ -8,6 +8,7 @@ use crate::heal::heal_commands::{
};
use crate::heal::heal_ops::RUSTFS_RESERVED_BUCKET;
use crate::store::all_local_disk;
use crate::store_utils::is_reserved_or_invalid_bucket;
use crate::{
disk::{self, VolumeInfo},
endpoints::{EndpointServerPools, Node},
@@ -20,7 +21,6 @@ use protos::node_service_time_out_client;
use protos::proto_gen::node_service::{
DeleteBucketRequest, GetBucketInfoRequest, HealBucketRequest, ListBucketRequest, MakeBucketRequest,
};
use regex::Regex;
use std::{collections::HashMap, fmt::Debug, sync::Arc};
use tokio::sync::RwLock;
use tonic::Request;
@@ -622,63 +622,6 @@ impl PeerS3Client for RemotePeerS3Client {
}
}
// 检查桶名是否有效
fn check_bucket_name(bucket_name: &str, strict: bool) -> Result<()> {
if bucket_name.trim().is_empty() {
return Err(Error::other("Bucket name cannot be empty"));
}
if bucket_name.len() < 3 {
return Err(Error::other("Bucket name cannot be shorter than 3 characters"));
}
if bucket_name.len() > 63 {
return Err(Error::other("Bucket name cannot be longer than 63 characters"));
}
let ip_address_regex = Regex::new(r"^(\d+\.){3}\d+$").unwrap();
if ip_address_regex.is_match(bucket_name) {
return Err(Error::other("Bucket name cannot be an IP address"));
}
let valid_bucket_name_regex = if strict {
Regex::new(r"^[a-z0-9][a-z0-9\.\-]{1,61}[a-z0-9]$").unwrap()
} else {
Regex::new(r"^[A-Za-z0-9][A-Za-z0-9\.\-_:]{1,61}[A-Za-z0-9]$").unwrap()
};
if !valid_bucket_name_regex.is_match(bucket_name) {
return Err(Error::other("Bucket name contains invalid characters"));
}
// 检查包含 "..", ".-", "-."
if bucket_name.contains("..") || bucket_name.contains(".-") || bucket_name.contains("-.") {
return Err(Error::other("Bucket name contains invalid characters"));
}
Ok(())
}
// 检查是否为 元数据桶
fn is_meta_bucket(bucket_name: &str) -> bool {
bucket_name == disk::RUSTFS_META_BUCKET
}
// 检查是否为 保留桶
fn is_reserved_bucket(bucket_name: &str) -> bool {
bucket_name == "rustfs"
}
// 检查桶名是否为保留名或无效名
pub fn is_reserved_or_invalid_bucket(bucket_entry: &str, strict: bool) -> bool {
if bucket_entry.is_empty() {
return true;
}
let bucket_entry = bucket_entry.trim_end_matches('/');
let result = check_bucket_name(bucket_entry, strict).is_err();
result || is_meta_bucket(bucket_entry) || is_reserved_bucket(bucket_entry)
}
pub async fn heal_bucket_local(bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
let disks = clone_drives().await;
let before_state = Arc::new(RwLock::new(vec![String::new(); disks.len()]));
ecstore/src/disk/remote.rs → ecstore/src/rpc/remote_disk.rs
+123 -66
View File
@@ -1,36 +1,27 @@
use std::path::PathBuf;
use bytes::Bytes;
use futures::lock::Mutex;
use http::{HeaderMap, Method};
use http::{HeaderMap, HeaderValue, Method, header::CONTENT_TYPE};
use protos::{
node_service_time_out_client,
proto_gen::node_service::{
CheckPartsRequest, DeletePathsRequest, DeleteRequest, DeleteVersionRequest, DeleteVersionsRequest, DeleteVolumeRequest,
DiskInfoRequest, ListDirRequest, ListVolumesRequest, MakeVolumeRequest, MakeVolumesRequest, NsScannerRequest,
ReadAllRequest, ReadMultipleRequest, ReadVersionRequest, ReadXlRequest, RenameDataRequest, RenameFileRequest,
StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WalkDirRequest, WriteAllRequest, WriteMetadataRequest,
StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest,
},
};
use rmp_serde::Serializer;
use rustfs_filemeta::{FileInfo, MetaCacheEntry, MetacacheWriter, RawFileInfo};
use rustfs_rio::{HttpReader, HttpWriter};
use serde::Serialize;
use tokio::{
io::AsyncWrite,
sync::mpsc::{self, Sender},
};
use tokio_stream::{StreamExt, wrappers::ReceiverStream};
use tonic::Request;
use tracing::info;
use uuid::Uuid;
use super::error::{Error, Result};
use super::{
use crate::disk::{
CheckPartsResp, DeleteOptions, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, FileInfoVersions,
ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
endpoint::Endpoint,
};
use crate::{
disk::error::{Error, Result},
rpc::build_auth_headers,
};
use crate::{
disk::{FileReader, FileWriter},
heal::{
@@ -39,6 +30,16 @@ use crate::{
heal_commands::{HealScanMode, HealingTracker},
},
};
use rustfs_filemeta::{FileInfo, RawFileInfo};
use rustfs_rio::{HttpReader, HttpWriter};
use tokio::{
io::AsyncWrite,
sync::mpsc::{self, Sender},
};
use tokio_stream::{StreamExt, wrappers::ReceiverStream};
use tonic::Request;
use tracing::info;
use uuid::Uuid;
use protos::proto_gen::node_service::RenamePartRequest;
@@ -255,47 +256,55 @@ impl DiskAPI for RemoteDisk {
Ok(())
}
// FIXME: TODO: use writer
#[tracing::instrument(skip(self, wr))]
async fn walk_dir<W: AsyncWrite + Unpin + Send>(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> {
let now = std::time::SystemTime::now();
info!("walk_dir {}/{}/{:?}", self.endpoint.to_string(), opts.bucket, opts.filter_prefix);
let mut wr = wr;
let mut out = MetacacheWriter::new(&mut wr);
let mut buf = Vec::new();
opts.serialize(&mut Serializer::new(&mut buf))?;
let mut client = node_service_time_out_client(&self.addr)
.await
.map_err(|err| Error::other(format!("can not get client, err: {}", err)))?;
let request = Request::new(WalkDirRequest {
disk: self.endpoint.to_string(),
walk_dir_options: buf,
});
let mut response = client.walk_dir(request).await?.into_inner();
// // FIXME: TODO: use writer
// #[tracing::instrument(skip(self, wr))]
// async fn walk_dir<W: AsyncWrite + Unpin + Send>(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> {
// let now = std::time::SystemTime::now();
// info!("walk_dir {}/{}/{:?}", self.endpoint.to_string(), opts.bucket, opts.filter_prefix);
// let mut wr = wr;
// let mut out = MetacacheWriter::new(&mut wr);
// let mut buf = Vec::new();
// opts.serialize(&mut Serializer::new(&mut buf))?;
// let mut client = node_service_time_out_client(&self.addr)
// .await
// .map_err(|err| Error::other(format!("can not get client, err: {}", err)))?;
// let request = Request::new(WalkDirRequest {
// disk: self.endpoint.to_string(),
// walk_dir_options: buf.into(),
// });
// let mut response = client.walk_dir(request).await?.into_inner();
loop {
match response.next().await {
Some(Ok(resp)) => {
if !resp.success {
return Err(Error::other(resp.error_info.unwrap_or_default()));
}
let entry = serde_json::from_str::<MetaCacheEntry>(&resp.meta_cache_entry)
.map_err(|_| Error::other(format!("Unexpected response: {:?}", response)))?;
out.write_obj(&entry).await?;
}
None => break,
_ => return Err(Error::other(format!("Unexpected response: {:?}", response))),
}
}
// loop {
// match response.next().await {
// Some(Ok(resp)) => {
// if !resp.success {
// if let Some(err) = resp.error_info {
// if err == "Unexpected EOF" {
// return Err(Error::Io(std::io::Error::new(std::io::ErrorKind::UnexpectedEof, err)));
// } else {
// return Err(Error::other(err));
// }
// }
info!(
"walk_dir {}/{:?} done {:?}",
opts.bucket,
opts.filter_prefix,
now.elapsed().unwrap_or_default()
);
Ok(())
}
// return Err(Error::other("unknown error"));
// }
// let entry = serde_json::from_str::<MetaCacheEntry>(&resp.meta_cache_entry)
// .map_err(|_| Error::other(format!("Unexpected response: {:?}", response)))?;
// out.write_obj(&entry).await?;
// }
// None => break,
// _ => return Err(Error::other(format!("Unexpected response: {:?}", response))),
// }
// }
// info!(
// "walk_dir {}/{:?} done {:?}",
// opts.bucket,
// opts.filter_prefix,
// now.elapsed().unwrap_or_default()
// );
// Ok(())
// }
#[tracing::instrument(skip(self))]
async fn delete_version(
@@ -558,6 +567,29 @@ impl DiskAPI for RemoteDisk {
Ok(response.volumes)
}
#[tracing::instrument(skip(self, wr))]
async fn walk_dir<W: AsyncWrite + Unpin + Send>(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> {
info!("walk_dir {}", self.endpoint.to_string());
let url = format!(
"{}/rustfs/rpc/walk_dir?disk={}",
self.endpoint.grid_host(),
urlencoding::encode(self.endpoint.to_string().as_str()),
);
let opts = serde_json::to_vec(&opts)?;
let mut headers = HeaderMap::new();
headers.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
build_auth_headers(&url, &Method::GET, &mut headers);
let mut reader = HttpReader::new(url, Method::GET, headers, Some(opts)).await?;
tokio::io::copy(&mut reader, wr).await?;
Ok(())
}
#[tracing::instrument(level = "debug", skip(self))]
async fn read_file(&self, volume: &str, path: &str) -> Result<FileReader> {
info!("read_file {}/{}", volume, path);
@@ -572,12 +604,22 @@ impl DiskAPI for RemoteDisk {
0
);
Ok(Box::new(HttpReader::new(url, Method::GET, HeaderMap::new()).await?))
let mut headers = HeaderMap::new();
headers.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
build_auth_headers(&url, &Method::GET, &mut headers);
Ok(Box::new(HttpReader::new(url, Method::GET, headers, None).await?))
}
#[tracing::instrument(level = "debug", skip(self))]
async fn read_file_stream(&self, volume: &str, path: &str, offset: usize, length: usize) -> Result<FileReader> {
info!("read_file_stream {}/{}/{}", self.endpoint.to_string(), volume, path);
// warn!(
// "disk remote read_file_stream {}/{}/{} offset={} length={}",
// self.endpoint.to_string(),
// volume,
// path,
// offset,
// length
// );
let url = format!(
"{}/rustfs/rpc/read_file_stream?disk={}&volume={}&path={}&offset={}&length={}",
self.endpoint.grid_host(),
@@ -588,7 +630,10 @@ impl DiskAPI for RemoteDisk {
length
);
Ok(Box::new(HttpReader::new(url, Method::GET, HeaderMap::new()).await?))
let mut headers = HeaderMap::new();
headers.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
build_auth_headers(&url, &Method::GET, &mut headers);
Ok(Box::new(HttpReader::new(url, Method::GET, headers, None).await?))
}
#[tracing::instrument(level = "debug", skip(self))]
@@ -605,12 +650,21 @@ impl DiskAPI for RemoteDisk {
0
);
Ok(Box::new(HttpWriter::new(url, Method::PUT, HeaderMap::new()).await?))
let mut headers = HeaderMap::new();
headers.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
build_auth_headers(&url, &Method::PUT, &mut headers);
Ok(Box::new(HttpWriter::new(url, Method::PUT, headers).await?))
}
#[tracing::instrument(level = "debug", skip(self))]
async fn create_file(&self, _origvolume: &str, volume: &str, path: &str, file_size: usize) -> Result<FileWriter> {
info!("create_file {}/{}/{}", self.endpoint.to_string(), volume, path);
async fn create_file(&self, _origvolume: &str, volume: &str, path: &str, file_size: i64) -> Result<FileWriter> {
// warn!(
// "disk remote create_file {}/{}/{} file_size={}",
// self.endpoint.to_string(),
// volume,
// path,
// file_size
// );
let url = format!(
"{}/rustfs/rpc/put_file_stream?disk={}&volume={}&path={}&append={}&size={}",
@@ -622,7 +676,10 @@ impl DiskAPI for RemoteDisk {
file_size
);
Ok(Box::new(HttpWriter::new(url, Method::PUT, HeaderMap::new()).await?))
let mut headers = HeaderMap::new();
headers.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
build_auth_headers(&url, &Method::PUT, &mut headers);
Ok(Box::new(HttpWriter::new(url, Method::PUT, headers).await?))
}
#[tracing::instrument(level = "debug", skip(self))]
@@ -649,7 +706,7 @@ impl DiskAPI for RemoteDisk {
}
#[tracing::instrument(skip(self))]
async fn rename_part(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str, meta: Vec<u8>) -> Result<()> {
async fn rename_part(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str, meta: Bytes) -> Result<()> {
info!("rename_part {}/{}", src_volume, src_path);
let mut client = node_service_time_out_client(&self.addr)
.await
@@ -773,7 +830,7 @@ impl DiskAPI for RemoteDisk {
}
#[tracing::instrument(skip(self))]
async fn write_all(&self, volume: &str, path: &str, data: Vec<u8>) -> Result<()> {
async fn write_all(&self, volume: &str, path: &str, data: Bytes) -> Result<()> {
info!("write_all");
let mut client = node_service_time_out_client(&self.addr)
.await
@@ -795,7 +852,7 @@ impl DiskAPI for RemoteDisk {
}
#[tracing::instrument(skip(self))]
async fn read_all(&self, volume: &str, path: &str) -> Result<Vec<u8>> {
async fn read_all(&self, volume: &str, path: &str) -> Result<Bytes> {
info!("read_all {}/{}", volume, path);
let mut client = node_service_time_out_client(&self.addr)
.await
ecstore/src/rpc/tonic_service.rs
+3627
View File
File diff suppressed because it is too large Load Diff
ecstore/src/set_disk.rs
+190 -68
View File
@@ -52,6 +52,7 @@ use crate::{
heal::data_scanner::{HEAL_DELETE_DANGLING, globalHealConfig},
store_api::ListObjectVersionsInfo,
};
use bytes::Bytes;
use bytesize::ByteSize;
use chrono::Utc;
use futures::future::join_all;
@@ -61,13 +62,14 @@ use lock::{LockApi, namespace_lock::NsLockMap};
use madmin::heal_commands::{HealDriveInfo, HealResultItem};
use md5::{Digest as Md5Digest, Md5};
use rand::{Rng, seq::SliceRandom};
use rustfs_filemeta::headers::RESERVED_METADATA_PREFIX_LOWER;
use rustfs_filemeta::{
FileInfo, FileMeta, FileMetaShallowVersion, MetaCacheEntries, MetaCacheEntry, MetadataResolutionParams, ObjectPartInfo,
RawFileInfo, file_info_from_raw,
headers::{AMZ_OBJECT_TAGGING, AMZ_STORAGE_CLASS},
merge_file_meta_versions,
};
use rustfs_rio::{EtagResolvable, HashReader};
use rustfs_rio::{EtagResolvable, HashReader, TryGetIndex as _, WarpReader};
use rustfs_utils::{
HashAlgorithm,
crypto::{base64_decode, base64_encode, hex},
@@ -497,7 +499,7 @@ impl SetDisks {
src_object: &str,
dst_bucket: &str,
dst_object: &str,
meta: Vec<u8>,
meta: Bytes,
write_quorum: usize,
) -> disk::error::Result<Vec<Option<DiskStore>>> {
let src_bucket = Arc::new(src_bucket.to_string());
@@ -867,6 +869,8 @@ impl SetDisks {
};
if let Some(err) = reduce_read_quorum_errs(errs, OBJECT_OP_IGNORED_ERRS, expected_rquorum) {
// let object = parts_metadata.first().map(|v| v.name.clone()).unwrap_or_default();
// error!("object_quorum_from_meta: {:?}, errs={:?}, object={:?}", err, errs, object);
return Err(err);
}
@@ -879,6 +883,7 @@ impl SetDisks {
let parity_blocks = Self::common_parity(&parities, default_parity_count as i32);
if parity_blocks < 0 {
error!("object_quorum_from_meta: parity_blocks < 0, errs={:?}", errs);
return Err(DiskError::ErasureReadQuorum);
}
@@ -943,6 +948,7 @@ impl SetDisks {
Self::object_quorum_from_meta(&parts_metadata, &errs, self.default_parity_count).map_err(map_err_notfound)?;
if read_quorum < 0 {
error!("check_upload_id_exists: read_quorum < 0, errs={:?}", errs);
return Err(Error::ErasureReadQuorum);
}
@@ -984,6 +990,7 @@ impl SetDisks {
quorum: usize,
) -> disk::error::Result<FileInfo> {
if quorum < 1 {
error!("find_file_info_in_quorum: quorum < 1");
return Err(DiskError::ErasureReadQuorum);
}
@@ -1042,6 +1049,7 @@ impl SetDisks {
}
if max_count < quorum {
error!("find_file_info_in_quorum: max_count < quorum, max_val={:?}", max_val);
return Err(DiskError::ErasureReadQuorum);
}
@@ -1086,7 +1094,7 @@ impl SetDisks {
return Ok(fi);
}
warn!("QuorumError::Read, find_file_info_in_quorum fileinfo not found");
error!("find_file_info_in_quorum: fileinfo not found");
Err(DiskError::ErasureReadQuorum)
}
@@ -1770,10 +1778,18 @@ impl SetDisks {
let _min_disks = self.set_drive_count - self.default_parity_count;
let (read_quorum, _) = Self::object_quorum_from_meta(&parts_metadata, &errs, self.default_parity_count)
.map_err(|err| to_object_err(err.into(), vec![bucket, object]))?;
let (read_quorum, _) = match Self::object_quorum_from_meta(&parts_metadata, &errs, self.default_parity_count)
.map_err(|err| to_object_err(err.into(), vec![bucket, object]))
{
Ok(v) => v,
Err(e) => {
// error!("Self::object_quorum_from_meta: {:?}, bucket: {}, object: {}", &e, bucket, object);
return Err(e);
}
};
if let Some(err) = reduce_read_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, read_quorum as usize) {
error!("reduce_read_quorum_errs: {:?}, bucket: {}, object: {}", &err, bucket, object);
return Err(to_object_err(err.into(), vec![bucket, object]));
}
@@ -1811,7 +1827,7 @@ impl SetDisks {
bucket: &str,
object: &str,
offset: usize,
length: usize,
length: i64,
writer: &mut W,
fi: FileInfo,
files: Vec<FileInfo>,
@@ -1824,11 +1840,16 @@ impl SetDisks {
{
let (disks, files) = Self::shuffle_disks_and_parts_metadata_by_index(disks, &files, &fi);
let total_size = fi.size;
let total_size = fi.size as usize;
let length = { if length == 0 { total_size - offset } else { length } };
let length = if length < 0 {
fi.size as usize - offset
} else {
length as usize
};
if offset > total_size || offset + length > total_size {
error!("get_object_with_fileinfo offset out of range: {}, total_size: {}", offset, total_size);
return Err(Error::other("offset out of range"));
}
@@ -1846,13 +1867,6 @@ impl SetDisks {
let (last_part_index, _) = fi.to_part_offset(end_offset)?;
// debug!(
// "get_object_with_fileinfo end offset:{}, last_part_index:{},part_offset:{}",
// end_offset, last_part_index, 0
// );
// let erasure = Erasure::new(fi.erasure.data_blocks, fi.erasure.parity_blocks, fi.erasure.block_size);
let erasure = erasure_coding::Erasure::new(fi.erasure.data_blocks, fi.erasure.parity_blocks, fi.erasure.block_size);
let mut total_readed = 0;
@@ -1864,7 +1878,7 @@ impl SetDisks {
let part_number = fi.parts[i].number;
let part_size = fi.parts[i].size;
let mut part_length = part_size - part_offset;
if part_length > length - total_readed {
if part_length > (length - total_readed) {
part_length = length - total_readed
}
@@ -1903,9 +1917,10 @@ impl SetDisks {
let nil_count = errors.iter().filter(|&e| e.is_none()).count();
if nil_count < erasure.data_shards {
if let Some(read_err) = reduce_read_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, erasure.data_shards) {
error!("create_bitrot_reader reduce_read_quorum_errs {:?}", &errors);
return Err(to_object_err(read_err.into(), vec![bucket, object]));
}
error!("create_bitrot_reader not enough disks to read: {:?}", &errors);
return Err(Error::other(format!("not enough disks to read: {:?}", errors)));
}
@@ -2252,7 +2267,8 @@ impl SetDisks {
erasure_coding::Erasure::default()
};
result.object_size = ObjectInfo::from_file_info(&lastest_meta, bucket, object, true).get_actual_size()?;
result.object_size =
ObjectInfo::from_file_info(&lastest_meta, bucket, object, true).get_actual_size()? as usize;
// Loop to find number of disks with valid data, per-drive
// data state and a list of outdated disks on which data needs
// to be healed.
@@ -2514,7 +2530,7 @@ impl SetDisks {
disk.as_ref(),
RUSTFS_META_TMP_BUCKET,
&format!("{}/{}/part.{}", tmp_id, dst_data_dir, part.number),
erasure.shard_file_size(part.size),
erasure.shard_file_size(part.size as i64),
erasure.shard_size(),
HashAlgorithm::HighwayHash256,
)
@@ -2596,13 +2612,15 @@ impl SetDisks {
part.size,
part.mod_time,
part.actual_size,
part.index.clone(),
);
if is_inline_buffer {
if let Some(writer) = writers[index].take() {
// if let Some(w) = writer.as_any().downcast_ref::<BitrotFileWriter>() {
// parts_metadata[index].data = Some(w.inline_data().to_vec());
// }
parts_metadata[index].data = Some(writer.into_inline_data().unwrap_or_default());
parts_metadata[index].data =
Some(writer.into_inline_data().map(bytes::Bytes::from).unwrap_or_default());
}
parts_metadata[index].set_inline_data();
} else {
@@ -2826,7 +2844,7 @@ impl SetDisks {
heal_item_type: HEAL_ITEM_OBJECT.to_string(),
bucket: bucket.to_string(),
object: object.to_string(),
object_size: lfi.size,
object_size: lfi.size as usize,
version_id: version_id.to_string(),
disk_count: disk_len,
..Default::default()
@@ -2948,6 +2966,7 @@ impl SetDisks {
}
Ok(m)
} else {
error!("delete_if_dang_ling: is_object_dang_ling errs={:?}", errs);
Err(DiskError::ErasureReadQuorum)
}
}
@@ -3010,13 +3029,25 @@ impl SetDisks {
}
let (buckets_results_tx, mut buckets_results_rx) = mpsc::channel::<DataUsageEntryInfo>(disks.len());
// 新增：从环境变量读取基础间隔，默认 30 秒
let set_disk_update_interval_secs = std::env::var("RUSTFS_NS_SCANNER_INTERVAL")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.unwrap_or(30);
let update_time = {
let mut rng = rand::rng();
Duration::from_secs(30) + Duration::from_secs_f64(10.0 * rng.random_range(0.0..1.0))
Duration::from_secs(set_disk_update_interval_secs) + Duration::from_secs_f64(10.0 * rng.random_range(0.0..1.0))
};
let mut ticker = interval(update_time);
let task = tokio::spawn(async move {
// 检查是否需要运行后台任务
let skip_background_task = std::env::var("RUSTFS_SKIP_BACKGROUND_TASK")
.ok()
.and_then(|v| v.parse::<bool>().ok())
.unwrap_or(false);
let task = if !skip_background_task {
Some(tokio::spawn(async move {
let last_save = Some(SystemTime::now());
let mut need_loop = true;
while need_loop {
@@ -3044,7 +3075,10 @@ impl SetDisks {
}
}
}
});
}))
} else {
None
};
// Restrict parallelism for disk usage scanner
let max_procs = num_cpus::get();
@@ -3148,7 +3182,9 @@ impl SetDisks {
info!("ns_scanner start");
let _ = join_all(futures).await;
if let Some(task) = task {
let _ = task.await;
}
info!("ns_scanner completed");
Ok(())
}
@@ -3474,7 +3510,7 @@ impl SetDisks {
if let (Some(started), Some(mod_time)) = (started, version.mod_time) {
if mod_time > started {
version_not_found += 1;
if send(heal_entry_skipped(version.size)).await {
if send(heal_entry_skipped(version.size as usize)).await {
defer.await;
return;
}
@@ -3518,10 +3554,10 @@ impl SetDisks {
if version_healed {
bg_seq.count_healed(HEAL_ITEM_OBJECT.to_string()).await;
result = heal_entry_success(version.size);
result = heal_entry_success(version.size as usize);
} else {
bg_seq.count_failed(HEAL_ITEM_OBJECT.to_string()).await;
result = heal_entry_failure(version.size);
result = heal_entry_failure(version.size as usize);
match version.version_id {
Some(version_id) => {
info!("unable to heal object {}/{}-v({})", bucket, version.name, version_id);
@@ -3876,7 +3912,7 @@ impl ObjectIO for SetDisks {
let is_inline_buffer = {
if let Some(sc) = GLOBAL_StorageClass.get() {
sc.should_inline(erasure.shard_file_size(data.content_length), opts.versioned)
sc.should_inline(erasure.shard_file_size(data.size()), opts.versioned)
} else {
false
}
@@ -3891,7 +3927,7 @@ impl ObjectIO for SetDisks {
Some(disk),
RUSTFS_META_TMP_BUCKET,
&tmp_object,
erasure.shard_file_size(data.content_length),
erasure.shard_file_size(data.size()),
erasure.shard_size(),
HashAlgorithm::HighwayHash256,
)
@@ -3937,15 +3973,34 @@ impl ObjectIO for SetDisks {
return Err(Error::other(format!("not enough disks to write: {:?}", errors)));
}
let stream = mem::replace(&mut data.stream, HashReader::new(Box::new(Cursor::new(Vec::new())), 0, 0, None, false)?);
let stream = mem::replace(
&mut data.stream,
HashReader::new(Box::new(WarpReader::new(Cursor::new(Vec::new()))), 0, 0, None, false)?,
);
let (reader, w_size) = Arc::new(erasure).encode(stream, &mut writers, write_quorum).await?; // TODO: 出错，删除临时目录
let (reader, w_size) = match Arc::new(erasure).encode(stream, &mut writers, write_quorum).await {
Ok((r, w)) => (r, w),
Err(e) => {
error!("encode err {:?}", e);
return Err(e.into());
}
}; // TODO: 出错，删除临时目录
let _ = mem::replace(&mut data.stream, reader);
// if let Err(err) = close_bitrot_writers(&mut writers).await {
// error!("close_bitrot_writers err {:?}", err);
// }
if (w_size as i64) < data.size() {
return Err(Error::other("put_object write size < data.size()"));
}
if user_defined.contains_key(&format!("{}compression", RESERVED_METADATA_PREFIX_LOWER)) {
user_defined.insert(format!("{}compression-size", RESERVED_METADATA_PREFIX_LOWER), w_size.to_string());
}
let index_op = data.stream.try_get_index().map(|v| v.clone().into_vec());
//TODO: userDefined
let etag = data.stream.try_resolve_etag().unwrap_or_default();
@@ -3956,6 +4011,14 @@ impl ObjectIO for SetDisks {
// get content-type
}
let mut actual_size = data.actual_size();
if actual_size < 0 {
let is_compressed = fi.is_compressed();
if !is_compressed {
actual_size = w_size as i64;
}
}
if let Some(sc) = user_defined.get(AMZ_STORAGE_CLASS) {
if sc == storageclass::STANDARD {
let _ = user_defined.remove(AMZ_STORAGE_CLASS);
@@ -3967,19 +4030,21 @@ impl ObjectIO for SetDisks {
for (i, fi) in parts_metadatas.iter_mut().enumerate() {
if is_inline_buffer {
if let Some(writer) = writers[i].take() {
fi.data = Some(writer.into_inline_data().unwrap_or_default());
fi.data = Some(writer.into_inline_data().map(bytes::Bytes::from).unwrap_or_default());
}
fi.set_inline_data();
}
fi.metadata = user_defined.clone();
fi.mod_time = Some(now);
fi.size = w_size;
fi.size = w_size as i64;
fi.versioned = opts.versioned || opts.version_suspended;
fi.add_object_part(1, etag.clone(), w_size, fi.mod_time, w_size);
fi.add_object_part(1, etag.clone(), w_size, fi.mod_time, actual_size, index_op.clone());
fi.set_inline_data();
// debug!("put_object fi {:?}", &fi)
if opts.data_movement {
fi.set_data_moved();
}
}
let (online_disks, _, op_old_dir) = Self::rename_data(
@@ -4036,7 +4101,7 @@ impl StorageAPI for SetDisks {
async fn local_storage_info(&self) -> madmin::StorageInfo {
let disks = self.get_disks_internal().await;
let mut local_disks: Vec<Option<Arc<crate::disk::Disk>>> = Vec::new();
let mut local_disks: Vec<Option<Arc<disk::Disk>>> = Vec::new();
let mut local_endpoints = Vec::new();
for (i, ep) in self.set_endpoints.iter().enumerate() {
@@ -4843,7 +4908,7 @@ impl StorageAPI for SetDisks {
Some(disk),
RUSTFS_META_TMP_BUCKET,
&tmp_part_path,
erasure.shard_file_size(data.content_length),
erasure.shard_file_size(data.size()),
erasure.shard_size(),
HashAlgorithm::HighwayHash256,
)
@@ -4882,16 +4947,33 @@ impl StorageAPI for SetDisks {
return Err(Error::other(format!("not enough disks to write: {:?}", errors)));
}
let stream = mem::replace(&mut data.stream, HashReader::new(Box::new(Cursor::new(Vec::new())), 0, 0, None, false)?);
let stream = mem::replace(
&mut data.stream,
HashReader::new(Box::new(WarpReader::new(Cursor::new(Vec::new()))), 0, 0, None, false)?,
);
let (reader, w_size) = Arc::new(erasure).encode(stream, &mut writers, write_quorum).await?; // TODO: 出错，删除临时目录
let _ = mem::replace(&mut data.stream, reader);
if (w_size as i64) < data.size() {
return Err(Error::other("put_object_part write size < data.size()"));
}
let index_op = data.stream.try_get_index().map(|v| v.clone().into_vec());
let mut etag = data.stream.try_resolve_etag().unwrap_or_default();
if let Some(ref tag) = opts.preserve_etag {
etag = tag.clone(); // TODO: 需要验证 etag 是否一致
etag = tag.clone();
}
let mut actual_size = data.actual_size();
if actual_size < 0 {
let is_compressed = fi.is_compressed();
if !is_compressed {
actual_size = w_size as i64;
}
}
let part_info = ObjectPartInfo {
@@ -4899,7 +4981,8 @@ impl StorageAPI for SetDisks {
number: part_id,
size: w_size,
mod_time: Some(OffsetDateTime::now_utc()),
actual_size: data.content_length,
actual_size,
index: index_op,
..Default::default()
};
@@ -4916,7 +4999,7 @@ impl StorageAPI for SetDisks {
&tmp_part_path,
RUSTFS_META_MULTIPART_BUCKET,
&part_path,
fi_buff,
fi_buff.into(),
write_quorum,
)
.await?;
@@ -4926,6 +5009,7 @@ impl StorageAPI for SetDisks {
part_num: part_id,
last_mod: Some(OffsetDateTime::now_utc()),
size: w_size,
actual_size,
};
// error!("put_object_part ret {:?}", &ret);
@@ -5209,7 +5293,7 @@ impl StorageAPI for SetDisks {
// complete_multipart_upload 完成
#[tracing::instrument(skip(self))]
async fn complete_multipart_upload(
&self,
self: Arc<Self>,
bucket: &str,
object: &str,
upload_id: &str,
@@ -5251,12 +5335,15 @@ impl StorageAPI for SetDisks {
for (i, res) in part_files_resp.iter().enumerate() {
let part_id = uploaded_parts[i].part_num;
if !res.error.is_empty() || !res.exists {
// error!("complete_multipart_upload part_id err {:?}", res);
error!("complete_multipart_upload part_id err {:?}, exists={}", res, res.exists);
return Err(Error::InvalidPart(part_id, bucket.to_owned(), object.to_owned()));
}
let part_fi = FileInfo::unmarshal(&res.data).map_err(|_e| {
// error!("complete_multipart_upload FileInfo::unmarshal err {:?}", e);
let part_fi = FileInfo::unmarshal(&res.data).map_err(|e| {
error!(
"complete_multipart_upload FileInfo::unmarshal err {:?}, part_id={}, bucket={}, object={}",
e, part_id, bucket, object
);
Error::InvalidPart(part_id, bucket.to_owned(), object.to_owned())
})?;
let part = &part_fi.parts[0];
@@ -5266,11 +5353,18 @@ impl StorageAPI for SetDisks {
// debug!("complete part {} object info {:?}", part_num, &part);
if part_id != part_num {
// error!("complete_multipart_upload part_id err part_id != part_num {} != {}", part_id, part_num);
error!("complete_multipart_upload part_id err part_id != part_num {} != {}", part_id, part_num);
return Err(Error::InvalidPart(part_id, bucket.to_owned(), object.to_owned()));
}
fi.add_object_part(part.number, part.etag.clone(), part.size, part.mod_time, part.actual_size);
fi.add_object_part(
part.number,
part.etag.clone(),
part.size,
part.mod_time,
part.actual_size,
part.index.clone(),
);
}
let (shuffle_disks, mut parts_metadatas) = Self::shuffle_disks_and_parts_metadata_by_index(&disks, &files_metas, &fi);
@@ -5280,24 +5374,35 @@ impl StorageAPI for SetDisks {
fi.parts = Vec::with_capacity(uploaded_parts.len());
let mut object_size: usize = 0;
let mut object_actual_size: usize = 0;
let mut object_actual_size: i64 = 0;
for (i, p) in uploaded_parts.iter().enumerate() {
let has_part = curr_fi.parts.iter().find(|v| v.number == p.part_num);
if has_part.is_none() {
// error!("complete_multipart_upload has_part.is_none() {:?}", has_part);
error!(
"complete_multipart_upload has_part.is_none() {:?}, part_id={}, bucket={}, object={}",
has_part, p.part_num, bucket, object
);
return Err(Error::InvalidPart(p.part_num, "".to_owned(), p.etag.clone().unwrap_or_default()));
}
let ext_part = &curr_fi.parts[i];
if p.etag != Some(ext_part.etag.clone()) {
error!(
"complete_multipart_upload etag err {:?}, part_id={}, bucket={}, object={}",
p.etag, p.part_num, bucket, object
);
return Err(Error::InvalidPart(p.part_num, ext_part.etag.clone(), p.etag.clone().unwrap_or_default()));
}
// TODO: crypto
if (i < uploaded_parts.len() - 1) && !is_min_allowed_part_size(ext_part.size) {
if (i < uploaded_parts.len() - 1) && !is_min_allowed_part_size(ext_part.actual_size) {
error!(
"complete_multipart_upload is_min_allowed_part_size err {:?}, part_id={}, bucket={}, object={}",
ext_part.actual_size, p.part_num, bucket, object
);
return Err(Error::InvalidPart(p.part_num, ext_part.etag.clone(), p.etag.clone().unwrap_or_default()));
}
@@ -5310,11 +5415,12 @@ impl StorageAPI for SetDisks {
size: ext_part.size,
mod_time: ext_part.mod_time,
actual_size: ext_part.actual_size,
index: ext_part.index.clone(),
..Default::default()
});
}
fi.size = object_size;
fi.size = object_size as i64;
fi.mod_time = opts.mod_time;
if fi.mod_time.is_none() {
fi.mod_time = Some(OffsetDateTime::now_utc());
@@ -5331,6 +5437,18 @@ impl StorageAPI for SetDisks {
fi.metadata.insert("etag".to_owned(), etag);
fi.metadata
.insert(format!("{}actual-size", RESERVED_METADATA_PREFIX_LOWER), object_actual_size.to_string());
if fi.is_compressed() {
fi.metadata
.insert(format!("{}compression-size", RESERVED_METADATA_PREFIX_LOWER), object_size.to_string());
}
if opts.data_movement {
fi.set_data_moved();
}
// TODO: object_actual_size
let _ = object_actual_size;
@@ -5402,17 +5520,6 @@ impl StorageAPI for SetDisks {
)
.await?;
for (i, op_disk) in online_disks.iter().enumerate() {
if let Some(disk) = op_disk {
if disk.is_online().await {
fi = parts_metadatas[i].clone();
break;
}
}
}
fi.is_latest = true;
// debug!("complete fileinfo {:?}", &fi);
// TODO: reduce_common_data_dir
@@ -5434,7 +5541,22 @@ impl StorageAPI for SetDisks {
.await;
}
let _ = self.delete_all(RUSTFS_META_MULTIPART_BUCKET, &upload_id_path).await;
let upload_id_path = upload_id_path.clone();
let store = self.clone();
let _cleanup_handle = tokio::spawn(async move {
let _ = store.delete_all(RUSTFS_META_MULTIPART_BUCKET, &upload_id_path).await;
});
for (i, op_disk) in online_disks.iter().enumerate() {
if let Some(disk) = op_disk {
if disk.is_online().await {
fi = parts_metadatas[i].clone();
break;
}
}
}
fi.is_latest = true;
Ok(ObjectInfo::from_file_info(&fi, bucket, object, opts.versioned || opts.version_suspended))
}
@@ -5794,7 +5916,7 @@ async fn disks_with_all_parts(
let verify_err = bitrot_verify(
Box::new(Cursor::new(data.clone())),
data_len,
meta.erasure.shard_file_size(meta.size),
meta.erasure.shard_file_size(meta.size) as usize,
checksum_info.algorithm,
checksum_info.hash,
meta.erasure.shard_size(),
@@ -6006,8 +6128,8 @@ pub async fn stat_all_dirs(disks: &[Option<DiskStore>], bucket: &str, prefix: &s
}
const GLOBAL_MIN_PART_SIZE: ByteSize = ByteSize::mib(5);
fn is_min_allowed_part_size(size: usize) -> bool {
size as u64 >= GLOBAL_MIN_PART_SIZE.as_u64()
fn is_min_allowed_part_size(size: i64) -> bool {
size >= GLOBAL_MIN_PART_SIZE.as_u64() as i64
}
fn get_complete_multipart_md5(parts: &[CompletePart]) -> String {
ecstore/src/sets.rs
+1 -1
View File
@@ -651,7 +651,7 @@ impl StorageAPI for Sets {
#[tracing::instrument(skip(self))]
async fn complete_multipart_upload(
&self,
self: Arc<Self>,
bucket: &str,
object: &str,
upload_id: &str,
ecstore/src/store.rs
+37 -14
View File
@@ -31,7 +31,7 @@ use crate::{
bucket::{lifecycle::bucket_lifecycle_ops::TransitionState, metadata::BucketMetadata},
disk::{BUCKET_META_PREFIX, DiskOption, DiskStore, RUSTFS_META_BUCKET, new_disk},
endpoints::EndpointServerPools,
peer::S3PeerSys,
rpc::S3PeerSys,
sets::Sets,
store_api::{
BucketInfo, BucketOptions, CompletePart, DeleteBucketOptions, DeletedObject, GetObjectReader, HTTPRangeSpec,
@@ -53,6 +53,7 @@ use rustfs_utils::crypto::base64_decode;
use rustfs_utils::path::{SLASH_SEPARATOR, decode_dir_object, encode_dir_object, path_join_buf};
use s3s::dto::{BucketVersioningStatus, ObjectLockConfiguration, ObjectLockEnabled, VersioningConfiguration};
use std::cmp::Ordering;
use std::net::SocketAddr;
use std::process::exit;
use std::slice::Iter;
use std::time::SystemTime;
@@ -101,7 +102,7 @@ pub struct ECStore {
impl ECStore {
#[allow(clippy::new_ret_no_self)]
#[tracing::instrument(level = "debug", skip(endpoint_pools))]
pub async fn new(_address: String, endpoint_pools: EndpointServerPools) -> Result<Arc<Self>> {
pub async fn new(address: SocketAddr, endpoint_pools: EndpointServerPools) -> Result<Arc<Self>> {
// let layouts = DisksLayout::from_volumes(endpoints.as_slice())?;
let mut deployment_id = None;
@@ -115,12 +116,17 @@ impl ECStore {
let mut local_disks = Vec::new();
init_local_peer(
&endpoint_pools,
&GLOBAL_Rustfs_Host.read().await.to_string(),
&GLOBAL_Rustfs_Port.read().await.to_string(),
)
.await;
info!("ECStore new address: {}", address.to_string());
let mut host = address.ip().to_string();
if host.is_empty() {
host = GLOBAL_Rustfs_Host.read().await.to_string()
}
let mut port = address.port().to_string();
if port.is_empty() {
port = GLOBAL_Rustfs_Port.read().await.to_string()
}
info!("ECStore new host: {}, port: {}", host, port);
init_local_peer(&endpoint_pools, &host, &port).await;
// debug!("endpoint_pools: {:?}", endpoint_pools);
@@ -856,9 +862,26 @@ impl ECStore {
let (update_closer_tx, mut update_close_rx) = mpsc::channel(10);
let mut ctx_clone = cancel.subscribe();
let all_buckets_clone = all_buckets.clone();
// 新增：从环境变量读取interval，默认30秒
let ns_scanner_interval_secs = std::env::var("RUSTFS_NS_SCANNER_INTERVAL")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.unwrap_or(30);
// 检查是否跳过后台任务
let skip_background_task = std::env::var("RUSTFS_SKIP_BACKGROUND_TASK")
.ok()
.and_then(|v| v.parse::<bool>().ok())
.unwrap_or(false);
if skip_background_task {
info!("跳过后台任务执行: RUSTFS_SKIP_BACKGROUND_TASK=true");
return Ok(());
}
let task = tokio::spawn(async move {
let mut last_update: Option<SystemTime> = None;
let mut interval = interval(Duration::from_secs(30));
let mut interval = interval(Duration::from_secs(ns_scanner_interval_secs));
let all_merged = Arc::new(RwLock::new(DataUsageCache::default()));
loop {
select! {
@@ -1225,7 +1248,7 @@ impl ObjectIO for ECStore {
return self.pools[0].put_object(bucket, object.as_str(), data, opts).await;
}
let idx = self.get_pool_idx(bucket, &object, data.content_length as i64).await?;
let idx = self.get_pool_idx(bucket, &object, data.size()).await?;
if opts.data_movement && idx == opts.src_pool_idx {
return Err(StorageError::DataMovementOverwriteErr(
@@ -1500,9 +1523,7 @@ impl StorageAPI for ECStore {
// TODO: nslock
let pool_idx = self
.get_pool_idx_no_lock(src_bucket, &src_object, src_info.size as i64)
.await?;
let pool_idx = self.get_pool_idx_no_lock(src_bucket, &src_object, src_info.size).await?;
if cp_src_dst_same {
if let (Some(src_vid), Some(dst_vid)) = (&src_opts.version_id, &dst_opts.version_id) {
@@ -2029,7 +2050,7 @@ impl StorageAPI for ECStore {
#[tracing::instrument(skip(self))]
async fn complete_multipart_upload(
&self,
self: Arc<Self>,
bucket: &str,
object: &str,
upload_id: &str,
@@ -2040,6 +2061,7 @@ impl StorageAPI for ECStore {
if self.single_pool() {
return self.pools[0]
.clone()
.complete_multipart_upload(bucket, object, upload_id, uploaded_parts, opts)
.await;
}
@@ -2049,6 +2071,7 @@ impl StorageAPI for ECStore {
continue;
}
let pool = pool.clone();
let err = match pool
.complete_multipart_upload(bucket, object, upload_id, uploaded_parts.clone(), opts)
.await
ecstore/src/store_api.rs
+112 -42
View File
@@ -12,24 +12,24 @@ use crate::{
use crate::{disk::DiskStore, heal::heal_commands::HealOpts};
use http::{HeaderMap, HeaderValue};
use madmin::heal_commands::HealResultItem;
use rustfs_filemeta::headers::RESERVED_METADATA_PREFIX_LOWER;
use rustfs_filemeta::{FileInfo, MetaCacheEntriesSorted, ObjectPartInfo, headers::AMZ_OBJECT_TAGGING};
use rustfs_rio::{HashReader, Reader};
use rustfs_rio::{DecompressReader, HashReader, LimitReader, WarpReader};
use rustfs_utils::CompressionAlgorithm;
use rustfs_utils::path::decode_dir_object;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::fmt::Debug;
use std::io::Cursor;
use std::str::FromStr as _;
use std::sync::Arc;
use time::OffsetDateTime;
use tokio::io::AsyncReadExt;
use tokio::io::{AsyncRead, AsyncReadExt};
use tracing::warn;
use uuid::Uuid;
pub const ERASURE_ALGORITHM: &str = "rs-vandermonde";
pub const BLOCK_SIZE_V2: usize = 1024 * 1024; // 1M
pub const RESERVED_METADATA_PREFIX: &str = "X-Rustfs-Internal-";
pub const RESERVED_METADATA_PREFIX_LOWER: &str = "x-rustfs-internal-";
pub const RUSTFS_HEALING: &str = "X-Rustfs-Internal-healing";
pub const RUSTFS_DATA_MOVE: &str = "X-Rustfs-Internal-data-mov";
#[derive(Debug, Default, Serialize, Deserialize)]
pub struct MakeBucketOptions {
@@ -58,46 +58,50 @@ pub struct DeleteBucketOptions {
pub struct PutObjReader {
pub stream: HashReader,
pub content_length: usize,
}
impl Debug for PutObjReader {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("PutObjReader")
.field("content_length", &self.content_length)
.finish()
f.debug_struct("PutObjReader").finish()
}
}
impl PutObjReader {
pub fn new(stream: HashReader, content_length: usize) -> Self {
PutObjReader { stream, content_length }
pub fn new(stream: HashReader) -> Self {
PutObjReader { stream }
}
pub fn from_vec(data: Vec<u8>) -> Self {
let content_length = data.len();
let content_length = data.len() as i64;
PutObjReader {
stream: HashReader::new(Box::new(Cursor::new(data)), content_length as i64, content_length as i64, None, false)
stream: HashReader::new(Box::new(WarpReader::new(Cursor::new(data))), content_length, content_length, None, false)
.unwrap(),
content_length,
}
}
pub fn size(&self) -> i64 {
self.stream.size()
}
pub fn actual_size(&self) -> i64 {
self.stream.actual_size()
}
}
pub struct GetObjectReader {
pub stream: Box<dyn Reader>,
pub stream: Box<dyn AsyncRead + Unpin + Send + Sync>,
pub object_info: ObjectInfo,
}
impl GetObjectReader {
#[tracing::instrument(level = "debug", skip(reader))]
pub fn new(
reader: Box<dyn Reader>,
reader: Box<dyn AsyncRead + Unpin + Send + Sync>,
rs: Option<HTTPRangeSpec>,
oi: &ObjectInfo,
opts: &ObjectOptions,
_h: &HeaderMap<HeaderValue>,
) -> Result<(Self, usize, usize)> {
) -> Result<(Self, usize, i64)> {
let mut rs = rs;
if let Some(part_number) = opts.part_number {
@@ -106,6 +110,47 @@ impl GetObjectReader {
}
}
// TODO:Encrypted
let (algo, is_compressed) = oi.is_compressed_ok()?;
// TODO: check TRANSITION
if is_compressed {
let actual_size = oi.get_actual_size()?;
let (off, length) = (0, oi.size);
let (_dec_off, dec_length) = (0, actual_size);
if let Some(_rs) = rs {
// TODO: range spec is not supported for compressed object
return Err(Error::other("The requested range is not satisfiable"));
// let (off, length) = rs.get_offset_length(actual_size)?;
}
let dec_reader = DecompressReader::new(reader, algo);
let actual_size = if actual_size > 0 {
actual_size as usize
} else {
return Err(Error::other(format!("invalid decompressed size {}", actual_size)));
};
warn!("actual_size: {}", actual_size);
let dec_reader = LimitReader::new(dec_reader, actual_size);
let mut oi = oi.clone();
oi.size = dec_length;
warn!("oi.size: {}, off: {}, length: {}", oi.size, off, length);
return Ok((
GetObjectReader {
stream: Box::new(dec_reader),
object_info: oi,
},
off,
length,
));
}
if let Some(rs) = rs {
let (off, length) = rs.get_offset_length(oi.size)?;
@@ -147,8 +192,8 @@ impl GetObjectReader {
#[derive(Debug)]
pub struct HTTPRangeSpec {
pub is_suffix_length: bool,
pub start: usize,
pub end: Option<usize>,
pub start: i64,
pub end: i64,
}
impl HTTPRangeSpec {
@@ -157,29 +202,38 @@ impl HTTPRangeSpec {
return None;
}
let mut start = 0;
let mut end = -1;
let mut start = 0i64;
let mut end = -1i64;
for i in 0..oi.parts.len().min(part_number) {
start = end + 1;
end = start + oi.parts[i].size as i64 - 1
end = start + (oi.parts[i].size as i64) - 1
}
Some(HTTPRangeSpec {
is_suffix_length: false,
start: start as usize,
end: { if end < 0 { None } else { Some(end as usize) } },
start,
end,
})
}
pub fn get_offset_length(&self, res_size: usize) -> Result<(usize, usize)> {
pub fn get_offset_length(&self, res_size: i64) -> Result<(usize, i64)> {
let len = self.get_length(res_size)?;
let mut start = self.start;
if self.is_suffix_length {
start = res_size - self.start
start = res_size + self.start;
if start < 0 {
start = 0;
}
}
Ok((start, len))
Ok((start as usize, len))
}
pub fn get_length(&self, res_size: usize) -> Result<usize> {
pub fn get_length(&self, res_size: i64) -> Result<i64> {
if res_size < 0 {
return Err(Error::other("The requested range is not satisfiable"));
}
if self.is_suffix_length {
let specified_len = self.start; // 假设 h.start 是一个 i64 类型
let mut range_length = specified_len;
@@ -195,8 +249,8 @@ impl HTTPRangeSpec {
return Err(Error::other("The requested range is not satisfiable"));
}
if let Some(end) = self.end {
let mut end = end;
if self.end > -1 {
let mut end = self.end;
if res_size <= end {
end = res_size - 1;
}
@@ -205,7 +259,7 @@ impl HTTPRangeSpec {
return Ok(range_length);
}
if self.end.is_none() {
if self.end == -1 {
let range_length = res_size - self.start;
return Ok(range_length);
}
@@ -285,6 +339,7 @@ pub struct PartInfo {
pub last_mod: Option<OffsetDateTime>,
pub size: usize,
pub etag: Option<String>,
pub actual_size: i64,
}
#[derive(Debug, Clone, Default)]
@@ -307,9 +362,9 @@ pub struct ObjectInfo {
pub bucket: String,
pub name: String,
pub mod_time: Option<OffsetDateTime>,
pub size: usize,
pub size: i64,
// Actual size is the real size of the object uploaded by client.
pub actual_size: Option<usize>,
pub actual_size: i64,
pub is_dir: bool,
pub user_defined: Option<HashMap<String, String>>,
pub parity_blocks: usize,
@@ -375,27 +430,41 @@ impl Clone for ObjectInfo {
impl ObjectInfo {
pub fn is_compressed(&self) -> bool {
if let Some(meta) = &self.user_defined {
meta.contains_key(&format!("{}compression", RESERVED_METADATA_PREFIX))
meta.contains_key(&format!("{}compression", RESERVED_METADATA_PREFIX_LOWER))
} else {
false
}
}
pub fn is_compressed_ok(&self) -> Result<(CompressionAlgorithm, bool)> {
let scheme = self
.user_defined
.as_ref()
.and_then(|meta| meta.get(&format!("{}compression", RESERVED_METADATA_PREFIX_LOWER)).cloned());
if let Some(scheme) = scheme {
let algorithm = CompressionAlgorithm::from_str(&scheme)?;
Ok((algorithm, true))
} else {
Ok((CompressionAlgorithm::None, false))
}
}
pub fn is_multipart(&self) -> bool {
self.etag.as_ref().is_some_and(|v| v.len() != 32)
}
pub fn get_actual_size(&self) -> std::io::Result<usize> {
if let Some(actual_size) = self.actual_size {
return Ok(actual_size);
pub fn get_actual_size(&self) -> std::io::Result<i64> {
if self.actual_size > 0 {
return Ok(self.actual_size);
}
if self.is_compressed() {
if let Some(meta) = &self.user_defined {
if let Some(size_str) = meta.get(&format!("{}actual-size", RESERVED_METADATA_PREFIX)) {
if let Some(size_str) = meta.get(&format!("{}actual-size", RESERVED_METADATA_PREFIX_LOWER)) {
if !size_str.is_empty() {
// Todo: deal with error
let size = size_str.parse::<usize>().map_err(|e| std::io::Error::other(e.to_string()))?;
let size = size_str.parse::<i64>().map_err(|e| std::io::Error::other(e.to_string()))?;
return Ok(size);
}
}
@@ -406,8 +475,9 @@ impl ObjectInfo {
actual_size += part.actual_size;
});
if actual_size == 0 && actual_size != self.size {
return Err(std::io::Error::other("invalid decompressed size"));
return Err(std::io::Error::other(format!("invalid decompressed size {} {}", actual_size, self.size)));
}
return Ok(actual_size);
}
@@ -827,7 +897,7 @@ pub trait StorageAPI: ObjectIO {
// ListObjectParts
async fn abort_multipart_upload(&self, bucket: &str, object: &str, upload_id: &str, opts: &ObjectOptions) -> Result<()>;
async fn complete_multipart_upload(
&self,
self: Arc<Self>,
bucket: &str,
object: &str,
upload_id: &str,
ecstore/src/store_init.rs
+2 -2
View File
@@ -256,7 +256,7 @@ pub async fn load_format_erasure(disk: &DiskStore, heal: bool) -> disk::error::R
_ => e,
})?;
let mut fm = FormatV3::try_from(data.as_slice())?;
let mut fm = FormatV3::try_from(data.as_ref())?;
if heal {
let info = disk
@@ -311,7 +311,7 @@ pub async fn save_format_file(disk: &Option<DiskStore>, format: &Option<FormatV3
let tmpfile = Uuid::new_v4().to_string();
let disk = disk.as_ref().unwrap();
disk.write_all(RUSTFS_META_BUCKET, tmpfile.as_str(), json_data.into_bytes())
disk.write_all(RUSTFS_META_BUCKET, tmpfile.as_str(), json_data.into_bytes().into())
.await?;
disk.rename_file(RUSTFS_META_BUCKET, tmpfile.as_str(), RUSTFS_META_BUCKET, FORMAT_CONFIG_FILE)
ecstore/src/store_list_objects.rs
+2 -1
View File
@@ -7,10 +7,10 @@ use crate::disk::{DiskInfo, DiskStore};
use crate::error::{
Error, Result, StorageError, is_all_not_found, is_all_volume_not_found, is_err_bucket_not_found, to_object_err,
};
use crate::peer::is_reserved_or_invalid_bucket;
use crate::set_disk::SetDisks;
use crate::store::check_list_objs_args;
use crate::store_api::{ListObjectVersionsInfo, ListObjectsInfo, ObjectInfo, ObjectOptions};
use crate::store_utils::is_reserved_or_invalid_bucket;
use crate::{store::ECStore, store_api::ListObjectsV2Info};
use futures::future::join_all;
use rand::seq::SliceRandom;
@@ -364,6 +364,7 @@ impl ECStore {
max_keys: i32,
) -> Result<ListObjectVersionsInfo> {
if marker.is_none() && version_marker.is_some() {
warn!("inner_list_object_versions: marker is none and version_marker is some");
return Err(StorageError::NotImplemented);
}
ecstore/src/store_utils.rs
+60
View File
@@ -1,7 +1,10 @@
use crate::config::storageclass::STANDARD;
use crate::disk::RUSTFS_META_BUCKET;
use regex::Regex;
use rustfs_filemeta::headers::AMZ_OBJECT_TAGGING;
use rustfs_filemeta::headers::AMZ_STORAGE_CLASS;
use std::collections::HashMap;
use std::io::{Error, Result};
pub fn clean_metadata(metadata: &mut HashMap<String, String>) {
remove_standard_storage_class(metadata);
@@ -19,3 +22,60 @@ pub fn clean_metadata_keys(metadata: &mut HashMap<String, String>, key_names: &[
metadata.remove(key.to_owned());
}
}
// 检查是否为 元数据桶
fn is_meta_bucket(bucket_name: &str) -> bool {
bucket_name == RUSTFS_META_BUCKET
}
// 检查是否为 保留桶
fn is_reserved_bucket(bucket_name: &str) -> bool {
bucket_name == "rustfs"
}
// 检查桶名是否为保留名或无效名
pub fn is_reserved_or_invalid_bucket(bucket_entry: &str, strict: bool) -> bool {
if bucket_entry.is_empty() {
return true;
}
let bucket_entry = bucket_entry.trim_end_matches('/');
let result = check_bucket_name(bucket_entry, strict).is_err();
result || is_meta_bucket(bucket_entry) || is_reserved_bucket(bucket_entry)
}
// 检查桶名是否有效
fn check_bucket_name(bucket_name: &str, strict: bool) -> Result<()> {
if bucket_name.trim().is_empty() {
return Err(Error::other("Bucket name cannot be empty"));
}
if bucket_name.len() < 3 {
return Err(Error::other("Bucket name cannot be shorter than 3 characters"));
}
if bucket_name.len() > 63 {
return Err(Error::other("Bucket name cannot be longer than 63 characters"));
}
let ip_address_regex = Regex::new(r"^(\d+\.){3}\d+$").unwrap();
if ip_address_regex.is_match(bucket_name) {
return Err(Error::other("Bucket name cannot be an IP address"));
}
let valid_bucket_name_regex = if strict {
Regex::new(r"^[a-z0-9][a-z0-9\.\-]{1,61}[a-z0-9]$").unwrap()
} else {
Regex::new(r"^[A-Za-z0-9][A-Za-z0-9\.\-_:]{1,61}[A-Za-z0-9]$").unwrap()
};
if !valid_bucket_name_regex.is_match(bucket_name) {
return Err(Error::other("Bucket name contains invalid characters"));
}
// 检查包含 "..", ".-", "-."
if bucket_name.contains("..") || bucket_name.contains(".-") || bucket_name.contains("-.") {
return Err(Error::other("Bucket name contains invalid characters"));
}
Ok(())
}