mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-24 21:26:28 +00:00
feat: migrate to reed-solomon-simd only implementation
- Remove reed-solomon-erasure dependency and all related code - Simplify ReedSolomonEncoder from enum to struct with SIMD-only implementation - Eliminate all conditional compilation (#[cfg(feature = ...)]) - Add instance caching with RwLock-based encoder/decoder reuse - Implement reset mechanism to avoid unnecessary allocations - Ensure thread safety with proper cache management - Update documentation and benchmark scripts for SIMD-only approach - Apply code formatting across all files Breaking Changes: - Removes support for reed-solomon-erasure feature flag - API remains compatible but implementation is now SIMD-only Performance Impact: - Improved encoding/decoding performance through SIMD optimization - Reduced memory allocations via instance caching - Enhanced thread safety and concurrency support
This commit is contained in:
weisd
+1
-4
@@ -11,9 +11,7 @@ rust-version.workspace = true
|
||||
workspace = true
|
||||
|
||||
[features]
|
||||
default = ["reed-solomon-simd"]
|
||||
reed-solomon-simd = []
|
||||
reed-solomon-erasure = []
|
||||
default = []
|
||||
|
||||
[dependencies]
|
||||
rustfs-config = { workspace = true, features = ["constants"] }
|
||||
@@ -40,7 +38,6 @@ http.workspace = true
|
||||
highway = { workspace = true }
|
||||
url.workspace = true
|
||||
uuid = { workspace = true, features = ["v4", "fast-rng", "serde"] }
|
||||
reed-solomon-erasure = { version = "6.0.0", features = ["simd-accel"] }
|
||||
reed-solomon-simd = { version = "3.0.0" }
|
||||
transform-stream = "0.3.1"
|
||||
lazy_static.workspace = true
|
||||
|
||||
+42
-55
@@ -1,38 +1,15 @@
|
||||
# ECStore - Erasure Coding Storage
|
||||
|
||||
ECStore provides erasure coding functionality for the RustFS project, supporting multiple Reed-Solomon implementations for optimal performance and compatibility.
|
||||
ECStore provides erasure coding functionality for the RustFS project, using high-performance Reed-Solomon SIMD implementation for optimal performance.
|
||||
|
||||
## Reed-Solomon Implementations
|
||||
## Reed-Solomon Implementation
|
||||
|
||||
### Available Backends
|
||||
### SIMD Backend (Only)
|
||||
|
||||
#### `reed-solomon-erasure` (Default)
|
||||
- **Stability**: Mature and well-tested implementation
|
||||
- **Performance**: Good performance with SIMD acceleration when available
|
||||
- **Compatibility**: Works with any shard size
|
||||
- **Memory**: Efficient memory usage
|
||||
- **Use case**: Recommended for production use
|
||||
|
||||
#### `reed-solomon-simd` (Optional)
|
||||
- **Performance**: Optimized SIMD implementation for maximum speed
|
||||
- **Limitations**: Has restrictions on shard sizes (must be >= 64 bytes typically)
|
||||
- **Memory**: May use more memory for small shards
|
||||
- **Use case**: Best for large data blocks where performance is critical
|
||||
|
||||
### Feature Flags
|
||||
|
||||
Configure the Reed-Solomon implementation using Cargo features:
|
||||
|
||||
```toml
|
||||
# Use default implementation (reed-solomon-erasure)
|
||||
ecstore = "0.0.1"
|
||||
|
||||
# Use SIMD implementation for maximum performance
|
||||
ecstore = { version = "0.0.1", features = ["reed-solomon-simd"], default-features = false }
|
||||
|
||||
# Use traditional implementation explicitly
|
||||
ecstore = { version = "0.0.1", features = ["reed-solomon-erasure"], default-features = false }
|
||||
```
|
||||
- **Performance**: Uses SIMD optimization for high-performance encoding/decoding
|
||||
- **Compatibility**: Works with any shard size through SIMD implementation
|
||||
- **Reliability**: High-performance SIMD implementation for large data processing
|
||||
- **Use case**: Optimized for maximum performance in large data processing scenarios
|
||||
|
||||
### Usage Example
|
||||
|
||||
@@ -68,42 +45,52 @@ assert_eq!(&recovered, data);
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### When to use `reed-solomon-simd`
|
||||
- Large block sizes (>= 1KB recommended)
|
||||
- High-throughput scenarios
|
||||
- CPU-intensive workloads where encoding/decoding is the bottleneck
|
||||
|
||||
### When to use `reed-solomon-erasure`
|
||||
- Small block sizes
|
||||
- Memory-constrained environments
|
||||
- General-purpose usage
|
||||
- Production deployments requiring maximum stability
|
||||
### SIMD Implementation Benefits
|
||||
- **High Throughput**: Optimized for large block sizes (>= 1KB recommended)
|
||||
- **CPU Optimization**: Leverages modern CPU SIMD instructions
|
||||
- **Scalability**: Excellent performance for high-throughput scenarios
|
||||
|
||||
### Implementation Details
|
||||
|
||||
#### `reed-solomon-erasure`
|
||||
- **Instance Reuse**: The encoder instance is cached and reused across multiple operations
|
||||
- **Thread Safety**: Thread-safe with interior mutability
|
||||
- **Memory Efficiency**: Lower memory footprint for small data
|
||||
|
||||
#### `reed-solomon-simd`
|
||||
- **Instance Creation**: New encoder/decoder instances are created for each operation
|
||||
- **API Design**: The SIMD implementation's API is designed for single-use instances
|
||||
- **Performance Trade-off**: While instances are created per operation, the SIMD optimizations provide significant performance benefits for large data blocks
|
||||
- **Optimization**: Future versions may implement instance pooling if the underlying API supports reuse
|
||||
- **Instance Caching**: Encoder/decoder instances are cached and reused for optimal performance
|
||||
- **Thread Safety**: Thread-safe with RwLock-based caching
|
||||
- **SIMD Optimization**: Leverages CPU SIMD instructions for maximum performance
|
||||
- **Reset Capability**: Cached instances are reset for different parameters, avoiding unnecessary allocations
|
||||
|
||||
### Performance Tips
|
||||
|
||||
1. **Batch Operations**: When possible, batch multiple small operations into larger blocks
|
||||
2. **Block Size Optimization**: Use block sizes that are multiples of 64 bytes for SIMD implementations
|
||||
2. **Block Size Optimization**: Use block sizes that are multiples of 64 bytes for optimal SIMD performance
|
||||
3. **Memory Allocation**: Pre-allocate buffers when processing multiple blocks
|
||||
4. **Feature Selection**: Choose the appropriate feature based on your data size and performance requirements
|
||||
4. **Cache Warming**: Initial operations may be slower due to cache setup, subsequent operations benefit from caching
|
||||
|
||||
## Cross-Platform Compatibility
|
||||
|
||||
Both implementations support:
|
||||
- x86_64 with SIMD acceleration
|
||||
- aarch64 (ARM64) with optimizations
|
||||
The SIMD implementation supports:
|
||||
- x86_64 with advanced SIMD instructions (AVX2, SSE)
|
||||
- aarch64 (ARM64) with NEON SIMD optimizations
|
||||
- Other architectures with fallback implementations
|
||||
|
||||
The `reed-solomon-erasure` implementation provides better cross-platform compatibility and is recommended for most use cases.
|
||||
The implementation automatically selects the best available SIMD instructions for the target platform, providing optimal performance across different architectures.
|
||||
|
||||
## Testing and Benchmarking
|
||||
|
||||
Run performance benchmarks:
|
||||
```bash
|
||||
# Run erasure coding benchmarks
|
||||
cargo bench --bench erasure_benchmark
|
||||
|
||||
# Run comparison benchmarks
|
||||
cargo bench --bench comparison_benchmark
|
||||
|
||||
# Generate benchmark reports
|
||||
./run_benchmarks.sh
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
All operations return `Result` types with comprehensive error information:
|
||||
- Encoding errors: Invalid parameters, insufficient memory
|
||||
- Decoding errors: Too many missing shards, corrupted data
|
||||
- Configuration errors: Invalid shard counts, unsupported parameters
|
||||
@@ -12,7 +12,7 @@
|
||||
//! cargo bench --bench comparison_benchmark --features reed-solomon-simd
|
||||
//!
|
||||
//! # 测试强制 erasure-only 模式
|
||||
//! cargo bench --bench comparison_benchmark --features reed-solomon-erasure
|
||||
//! cargo bench --bench comparison_benchmark
|
||||
//!
|
||||
//! # 生成对比报告
|
||||
//! cargo bench --bench comparison_benchmark -- --save-baseline erasure
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
//! Reed-Solomon erasure coding performance benchmarks.
|
||||
//!
|
||||
//! This benchmark compares the performance of different Reed-Solomon implementations:
|
||||
//! - Default (Pure erasure): Stable reed-solomon-erasure implementation
|
||||
//! - SIMD mode: High-performance reed-solomon-simd implementation
|
||||
//! - `reed-solomon-simd` feature: SIMD mode with optimized performance
|
||||
//!
|
||||
//! ## Running Benchmarks
|
||||
@@ -235,7 +235,7 @@ fn bench_decode_performance(c: &mut Criterion) {
|
||||
group.finish();
|
||||
|
||||
// 如果使用混合模式（默认），测试SIMD解码性能
|
||||
#[cfg(not(feature = "reed-solomon-erasure"))]
|
||||
|
||||
{
|
||||
let shard_size = calc_shard_size(config.data_size, config.data_shards);
|
||||
if shard_size >= 512 {
|
||||
|
||||
+74
-88
@@ -1,54 +1,48 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Reed-Solomon 实现性能比较脚本
|
||||
#
|
||||
# 这个脚本将运行不同的基准测试来比较SIMD模式和纯Erasure模式的性能
|
||||
#
|
||||
# 使用方法:
|
||||
# ./run_benchmarks.sh [quick|full|comparison]
|
||||
#
|
||||
# quick - 快速测试主要场景
|
||||
# full - 完整基准测试套件
|
||||
# comparison - 专门对比两种实现模式
|
||||
# Reed-Solomon SIMD 性能基准测试脚本
|
||||
# 使用高性能 SIMD 实现进行纠删码性能测试
|
||||
|
||||
set -e
|
||||
|
||||
# 颜色输出
|
||||
# ANSI 颜色码
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
PURPLE='\033[0;35m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# 输出带颜色的信息
|
||||
# 打印带颜色的消息
|
||||
print_info() {
|
||||
echo -e "${BLUE}[INFO]${NC} $1"
|
||||
echo -e "${BLUE}ℹ️ $1${NC}"
|
||||
}
|
||||
|
||||
print_success() {
|
||||
echo -e "${GREEN}[SUCCESS]${NC} $1"
|
||||
echo -e "${GREEN}✅ $1${NC}"
|
||||
}
|
||||
|
||||
print_warning() {
|
||||
echo -e "${YELLOW}[WARNING]${NC} $1"
|
||||
echo -e "${YELLOW}⚠️ $1${NC}"
|
||||
}
|
||||
|
||||
print_error() {
|
||||
echo -e "${RED}[ERROR]${NC} $1"
|
||||
echo -e "${RED}❌ $1${NC}"
|
||||
}
|
||||
|
||||
# 检查是否安装了必要工具
|
||||
# 检查系统要求
|
||||
check_requirements() {
|
||||
print_info "检查系统要求..."
|
||||
|
||||
# 检查 Rust
|
||||
if ! command -v cargo &> /dev/null; then
|
||||
print_error "cargo 未安装，请先安装 Rust 工具链"
|
||||
print_error "Cargo 未找到，请确保已安装 Rust"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 检查是否安装了 criterion
|
||||
if ! grep -q "criterion" Cargo.toml; then
|
||||
print_error "Cargo.toml 中未找到 criterion 依赖"
|
||||
# 检查 criterion
|
||||
if ! cargo --list | grep -q "bench"; then
|
||||
print_error "未找到基准测试支持，请确保使用的是支持基准测试的 Rust 版本"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -62,28 +56,15 @@ cleanup() {
|
||||
print_success "清理完成"
|
||||
}
|
||||
|
||||
# 运行纯 Erasure 模式基准测试
|
||||
run_erasure_benchmark() {
|
||||
print_info "🏛️ 开始运行纯 Erasure 模式基准测试..."
|
||||
echo "================================================"
|
||||
|
||||
cargo bench --bench comparison_benchmark \
|
||||
--features reed-solomon-erasure \
|
||||
-- --save-baseline erasure_baseline
|
||||
|
||||
print_success "纯 Erasure 模式基准测试完成"
|
||||
}
|
||||
|
||||
# 运行SIMD模式基准测试
|
||||
# 运行 SIMD 模式基准测试
|
||||
run_simd_benchmark() {
|
||||
print_info "🎯 开始运行SIMD模式基准测试..."
|
||||
print_info "🎯 开始运行 SIMD 模式基准测试..."
|
||||
echo "================================================"
|
||||
|
||||
cargo bench --bench comparison_benchmark \
|
||||
--features reed-solomon-simd \
|
||||
-- --save-baseline simd_baseline
|
||||
|
||||
print_success "SIMD模式基准测试完成"
|
||||
print_success "SIMD 模式基准测试完成"
|
||||
}
|
||||
|
||||
# 运行完整的基准测试套件
|
||||
@@ -91,33 +72,42 @@ run_full_benchmark() {
|
||||
print_info "🚀 开始运行完整基准测试套件..."
|
||||
echo "================================================"
|
||||
|
||||
# 运行详细的基准测试（使用默认纯Erasure模式）
|
||||
# 运行详细的基准测试
|
||||
cargo bench --bench erasure_benchmark
|
||||
|
||||
print_success "完整基准测试套件完成"
|
||||
}
|
||||
|
||||
# 运行性能对比测试
|
||||
run_comparison_benchmark() {
|
||||
print_info "📊 开始运行性能对比测试..."
|
||||
# 运行性能测试
|
||||
run_performance_test() {
|
||||
print_info "📊 开始运行性能测试..."
|
||||
echo "================================================"
|
||||
|
||||
print_info "步骤 1: 测试纯 Erasure 模式..."
|
||||
print_info "步骤 1: 运行编码基准测试..."
|
||||
cargo bench --bench comparison_benchmark \
|
||||
--features reed-solomon-erasure \
|
||||
-- --save-baseline erasure_baseline
|
||||
-- encode --save-baseline encode_baseline
|
||||
|
||||
print_info "步骤 2: 测试SIMD模式并与 Erasure 模式对比..."
|
||||
print_info "步骤 2: 运行解码基准测试..."
|
||||
cargo bench --bench comparison_benchmark \
|
||||
--features reed-solomon-simd \
|
||||
-- --baseline erasure_baseline
|
||||
-- decode --save-baseline decode_baseline
|
||||
|
||||
print_success "性能对比测试完成"
|
||||
print_success "性能测试完成"
|
||||
}
|
||||
|
||||
# 运行大数据集测试
|
||||
run_large_data_test() {
|
||||
print_info "🗂️ 开始运行大数据集测试..."
|
||||
echo "================================================"
|
||||
|
||||
cargo bench --bench erasure_benchmark \
|
||||
-- large_data --save-baseline large_data_baseline
|
||||
|
||||
print_success "大数据集测试完成"
|
||||
}
|
||||
|
||||
# 生成比较报告
|
||||
generate_comparison_report() {
|
||||
print_info "📊 生成性能比较报告..."
|
||||
print_info "📊 生成性能报告..."
|
||||
|
||||
if [ -d "target/criterion" ]; then
|
||||
print_info "基准测试结果已保存到 target/criterion/ 目录"
|
||||
@@ -138,49 +128,48 @@ generate_comparison_report() {
|
||||
run_quick_test() {
|
||||
print_info "🏃 运行快速性能测试..."
|
||||
|
||||
print_info "测试纯 Erasure 模式..."
|
||||
print_info "测试 SIMD 编码性能..."
|
||||
cargo bench --bench comparison_benchmark \
|
||||
--features reed-solomon-erasure \
|
||||
-- encode_comparison --quick
|
||||
-- encode --quick
|
||||
|
||||
print_info "测试SIMD模式..."
|
||||
print_info "测试 SIMD 解码性能..."
|
||||
cargo bench --bench comparison_benchmark \
|
||||
--features reed-solomon-simd \
|
||||
-- encode_comparison --quick
|
||||
-- decode --quick
|
||||
|
||||
print_success "快速测试完成"
|
||||
}
|
||||
|
||||
# 显示帮助信息
|
||||
show_help() {
|
||||
echo "Reed-Solomon 性能基准测试脚本"
|
||||
echo "Reed-Solomon SIMD 性能基准测试脚本"
|
||||
echo ""
|
||||
echo "实现模式："
|
||||
echo " 🏛️ 纯 Erasure 模式（默认）- 稳定兼容的 reed-solomon-erasure 实现"
|
||||
echo " 🎯 SIMD模式 - 高性能SIMD优化实现"
|
||||
echo " 🎯 SIMD 模式 - 高性能 SIMD 优化的 reed-solomon-simd 实现"
|
||||
echo ""
|
||||
echo "使用方法:"
|
||||
echo " $0 [command]"
|
||||
echo ""
|
||||
echo "命令:"
|
||||
echo " quick 运行快速性能测试"
|
||||
echo " full 运行完整基准测试套件（默认Erasure模式）"
|
||||
echo " comparison 运行详细的实现模式对比测试"
|
||||
echo " erasure 只测试纯 Erasure 模式"
|
||||
echo " simd 只测试SIMD模式"
|
||||
echo " full 运行完整基准测试套件"
|
||||
echo " performance 运行详细的性能测试"
|
||||
echo " simd 运行 SIMD 模式测试"
|
||||
echo " large 运行大数据集测试"
|
||||
echo " clean 清理测试结果"
|
||||
echo " help 显示此帮助信息"
|
||||
echo ""
|
||||
echo "示例:"
|
||||
echo " $0 quick # 快速测试两种模式"
|
||||
echo " $0 comparison # 详细对比测试"
|
||||
echo " $0 full # 完整测试套件（默认Erasure模式）"
|
||||
echo " $0 simd # 只测试SIMD模式"
|
||||
echo " $0 erasure # 只测试纯 Erasure 模式"
|
||||
echo " $0 quick # 快速性能测试"
|
||||
echo " $0 performance # 详细性能测试"
|
||||
echo " $0 full # 完整测试套件"
|
||||
echo " $0 simd # SIMD 模式测试"
|
||||
echo " $0 large # 大数据集测试"
|
||||
echo ""
|
||||
echo "模式说明:"
|
||||
echo " Erasure模式: 使用reed-solomon-erasure实现，稳定可靠"
|
||||
echo " SIMD模式: 使用reed-solomon-simd实现，高性能优化"
|
||||
echo "实现特性:"
|
||||
echo " - 使用 reed-solomon-simd 高性能 SIMD 实现"
|
||||
echo " - 支持编码器/解码器实例缓存"
|
||||
echo " - 优化的内存管理和线程安全"
|
||||
echo " - 跨平台 SIMD 指令支持"
|
||||
}
|
||||
|
||||
# 显示测试配置信息
|
||||
@@ -196,22 +185,22 @@ show_test_info() {
|
||||
if [ -f "/proc/cpuinfo" ]; then
|
||||
echo " - CPU 型号: $(grep 'model name' /proc/cpuinfo | head -1 | cut -d: -f2 | xargs)"
|
||||
if grep -q "avx2" /proc/cpuinfo; then
|
||||
echo " - SIMD 支持: AVX2 ✅ (SIMD模式将利用SIMD优化)"
|
||||
echo " - SIMD 支持: AVX2 ✅ (将使用高级 SIMD 优化)"
|
||||
elif grep -q "sse4" /proc/cpuinfo; then
|
||||
echo " - SIMD 支持: SSE4 ✅ (SIMD模式将利用SIMD优化)"
|
||||
echo " - SIMD 支持: SSE4 ✅ (将使用 SIMD 优化)"
|
||||
else
|
||||
echo " - SIMD 支持: 未检测到高级 SIMD 特性"
|
||||
echo " - SIMD 支持: 基础 SIMD 特性"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo " - 默认模式: 纯Erasure模式 (稳定可靠)"
|
||||
echo " - 高性能模式: SIMD模式 (性能优化)"
|
||||
echo " - 实现: reed-solomon-simd (高性能 SIMD 优化)"
|
||||
echo " - 特性: 实例缓存、线程安全、跨平台 SIMD"
|
||||
echo ""
|
||||
}
|
||||
|
||||
# 主函数
|
||||
main() {
|
||||
print_info "🧪 Reed-Solomon 实现性能基准测试"
|
||||
print_info "🧪 Reed-Solomon SIMD 实现性能基准测试"
|
||||
echo "================================================"
|
||||
|
||||
check_requirements
|
||||
@@ -227,14 +216,9 @@ main() {
|
||||
run_full_benchmark
|
||||
generate_comparison_report
|
||||
;;
|
||||
"comparison")
|
||||
"performance")
|
||||
cleanup
|
||||
run_comparison_benchmark
|
||||
generate_comparison_report
|
||||
;;
|
||||
"erasure")
|
||||
cleanup
|
||||
run_erasure_benchmark
|
||||
run_performance_test
|
||||
generate_comparison_report
|
||||
;;
|
||||
"simd")
|
||||
@@ -242,6 +226,11 @@ main() {
|
||||
run_simd_benchmark
|
||||
generate_comparison_report
|
||||
;;
|
||||
"large")
|
||||
cleanup
|
||||
run_large_data_test
|
||||
generate_comparison_report
|
||||
;;
|
||||
"clean")
|
||||
cleanup
|
||||
;;
|
||||
@@ -257,10 +246,7 @@ main() {
|
||||
esac
|
||||
|
||||
print_success "✨ 基准测试执行完成!"
|
||||
print_info "💡 提示: 推荐使用默认的纯Erasure模式，对于高性能需求可考虑SIMD模式"
|
||||
}
|
||||
|
||||
# 如果直接运行此脚本，调用主函数
|
||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||
main "$@"
|
||||
fi
|
||||
# 启动脚本
|
||||
main "$@"
|
||||
@@ -1,6 +1,7 @@
|
||||
#![allow(unused_variables)]
|
||||
#![allow(dead_code)]
|
||||
// use error::Error;
|
||||
use crate::StorageAPI;
|
||||
use crate::bucket::metadata_sys::get_replication_config;
|
||||
use crate::bucket::versioning_sys::BucketVersioningSys;
|
||||
use crate::error::Error;
|
||||
@@ -11,26 +12,25 @@ use crate::store_api::ObjectIO;
|
||||
use crate::store_api::ObjectInfo;
|
||||
use crate::store_api::ObjectOptions;
|
||||
use crate::store_api::ObjectToDelete;
|
||||
use crate::StorageAPI;
|
||||
use aws_sdk_s3::Client as S3Client;
|
||||
use aws_sdk_s3::Config;
|
||||
use aws_sdk_s3::config::BehaviorVersion;
|
||||
use aws_sdk_s3::config::Credentials;
|
||||
use aws_sdk_s3::config::Region;
|
||||
use aws_sdk_s3::Client as S3Client;
|
||||
use aws_sdk_s3::Config;
|
||||
use bytes::Bytes;
|
||||
use chrono::DateTime;
|
||||
use chrono::Duration;
|
||||
use chrono::Utc;
|
||||
use futures::stream::FuturesUnordered;
|
||||
use futures::StreamExt;
|
||||
use futures::stream::FuturesUnordered;
|
||||
use http::HeaderMap;
|
||||
use http::Method;
|
||||
use lazy_static::lazy_static;
|
||||
// use std::time::SystemTime;
|
||||
use once_cell::sync::Lazy;
|
||||
use regex::Regex;
|
||||
use rustfs_rsc::provider::StaticProvider;
|
||||
use rustfs_rsc::Minio;
|
||||
use rustfs_rsc::provider::StaticProvider;
|
||||
use s3s::dto::DeleteMarkerReplicationStatus;
|
||||
use s3s::dto::DeleteReplicationStatus;
|
||||
use s3s::dto::ExistingObjectReplicationStatus;
|
||||
@@ -43,14 +43,14 @@ use std::collections::HashSet;
|
||||
use std::fmt;
|
||||
use std::iter::Iterator;
|
||||
use std::str::FromStr;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::AtomicI32;
|
||||
use std::sync::atomic::Ordering;
|
||||
use std::sync::Arc;
|
||||
use std::vec;
|
||||
use time::OffsetDateTime;
|
||||
use tokio::sync::mpsc::{Receiver, Sender};
|
||||
use tokio::sync::Mutex;
|
||||
use tokio::sync::RwLock;
|
||||
use tokio::sync::mpsc::{Receiver, Sender};
|
||||
use tokio::task;
|
||||
use tracing::{debug, error, info, warn};
|
||||
use uuid::Uuid;
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
use super::{storageclass, Config, GLOBAL_StorageClass};
|
||||
use super::{Config, GLOBAL_StorageClass, storageclass};
|
||||
use crate::disk::RUSTFS_META_BUCKET;
|
||||
use crate::error::{Error, Result};
|
||||
use crate::store_api::{ObjectInfo, ObjectOptions, PutObjReader, StorageAPI};
|
||||
|
||||
@@ -5,7 +5,7 @@ pub mod storageclass;
|
||||
|
||||
use crate::error::Result;
|
||||
use crate::store::ECStore;
|
||||
use com::{lookup_configs, read_config_without_migrate, STORAGE_CLASS_SUB_SYS};
|
||||
use com::{STORAGE_CLASS_SUB_SYS, lookup_configs, read_config_without_migrate};
|
||||
use lazy_static::lazy_static;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::HashMap;
|
||||
|
||||
@@ -1,27 +1,15 @@
|
||||
//! Erasure coding implementation supporting multiple Reed-Solomon backends.
|
||||
//! Erasure coding implementation using Reed-Solomon SIMD backend.
|
||||
//!
|
||||
//! This module provides erasure coding functionality with support for two different
|
||||
//! Reed-Solomon implementations:
|
||||
//! This module provides erasure coding functionality with high-performance SIMD
|
||||
//! Reed-Solomon implementation:
|
||||
//!
|
||||
//! ## Reed-Solomon Implementations
|
||||
//! ## Reed-Solomon Implementation
|
||||
//!
|
||||
//! ### Pure Erasure Mode (Default)
|
||||
//! - **Stability**: Pure erasure implementation, mature and well-tested
|
||||
//! - **Performance**: Good performance with consistent behavior
|
||||
//! - **Compatibility**: Works with any shard size
|
||||
//! - **Use case**: Default behavior, recommended for most production use cases
|
||||
//!
|
||||
//! ### SIMD Mode (`reed-solomon-simd` feature)
|
||||
//! ### SIMD Mode (Only)
|
||||
//! - **Performance**: Uses SIMD optimization for high-performance encoding/decoding
|
||||
//! - **Compatibility**: Works with any shard size through SIMD implementation
|
||||
//! - **Reliability**: High-performance SIMD implementation for large data processing
|
||||
//! - **Use case**: Use when maximum performance is needed for large data processing
|
||||
//!
|
||||
//! ## Feature Flags
|
||||
//!
|
||||
//! - Default: Use pure reed-solomon-erasure implementation (stable and reliable)
|
||||
//! - `reed-solomon-simd`: Use SIMD mode for optimal performance
|
||||
//! - `reed-solomon-erasure`: Explicitly enable pure erasure mode (same as default)
|
||||
//! - **Use case**: Optimized for maximum performance in large data processing scenarios
|
||||
//!
|
||||
//! ## Example
|
||||
//!
|
||||
@@ -35,8 +23,6 @@
|
||||
//! ```
|
||||
|
||||
use bytes::{Bytes, BytesMut};
|
||||
use reed_solomon_erasure::galois_8::ReedSolomon as ReedSolomonErasure;
|
||||
#[cfg(feature = "reed-solomon-simd")]
|
||||
use reed_solomon_simd;
|
||||
use smallvec::SmallVec;
|
||||
use std::io;
|
||||
@@ -44,38 +30,23 @@ use tokio::io::AsyncRead;
|
||||
use tracing::warn;
|
||||
use uuid::Uuid;
|
||||
|
||||
/// Reed-Solomon encoder variants supporting different implementations.
|
||||
#[allow(clippy::large_enum_variant)]
|
||||
pub enum ReedSolomonEncoder {
|
||||
/// SIMD mode: High-performance SIMD implementation (when reed-solomon-simd feature is enabled)
|
||||
#[cfg(feature = "reed-solomon-simd")]
|
||||
SIMD {
|
||||
data_shards: usize,
|
||||
parity_shards: usize,
|
||||
// 使用RwLock确保线程安全，实现Send + Sync
|
||||
encoder_cache: std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonEncoder>>,
|
||||
decoder_cache: std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonDecoder>>,
|
||||
},
|
||||
/// Pure erasure mode: default and when reed-solomon-erasure feature is specified
|
||||
Erasure(Box<ReedSolomonErasure>),
|
||||
/// Reed-Solomon encoder using SIMD implementation.
|
||||
pub struct ReedSolomonEncoder {
|
||||
data_shards: usize,
|
||||
parity_shards: usize,
|
||||
// 使用RwLock确保线程安全，实现Send + Sync
|
||||
encoder_cache: std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonEncoder>>,
|
||||
decoder_cache: std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonDecoder>>,
|
||||
}
|
||||
|
||||
impl Clone for ReedSolomonEncoder {
|
||||
fn clone(&self) -> Self {
|
||||
match self {
|
||||
#[cfg(feature = "reed-solomon-simd")]
|
||||
ReedSolomonEncoder::SIMD {
|
||||
data_shards,
|
||||
parity_shards,
|
||||
..
|
||||
} => ReedSolomonEncoder::SIMD {
|
||||
data_shards: *data_shards,
|
||||
parity_shards: *parity_shards,
|
||||
// 为新实例创建空的缓存，不共享缓存
|
||||
encoder_cache: std::sync::RwLock::new(None),
|
||||
decoder_cache: std::sync::RwLock::new(None),
|
||||
},
|
||||
ReedSolomonEncoder::Erasure(encoder) => ReedSolomonEncoder::Erasure(encoder.clone()),
|
||||
Self {
|
||||
data_shards: self.data_shards,
|
||||
parity_shards: self.parity_shards,
|
||||
// 为新实例创建空的缓存，不共享缓存
|
||||
encoder_cache: std::sync::RwLock::new(None),
|
||||
decoder_cache: std::sync::RwLock::new(None),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -83,81 +54,50 @@ impl Clone for ReedSolomonEncoder {
|
||||
impl ReedSolomonEncoder {
|
||||
/// Create a new Reed-Solomon encoder with specified data and parity shards.
|
||||
pub fn new(data_shards: usize, parity_shards: usize) -> io::Result<Self> {
|
||||
#[cfg(feature = "reed-solomon-simd")]
|
||||
{
|
||||
// SIMD mode when reed-solomon-simd feature is enabled
|
||||
Ok(ReedSolomonEncoder::SIMD {
|
||||
data_shards,
|
||||
parity_shards,
|
||||
encoder_cache: std::sync::RwLock::new(None),
|
||||
decoder_cache: std::sync::RwLock::new(None),
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "reed-solomon-simd"))]
|
||||
{
|
||||
// Pure erasure mode when reed-solomon-simd feature is not enabled (default or reed-solomon-erasure)
|
||||
let encoder = ReedSolomonErasure::new(data_shards, parity_shards)
|
||||
.map_err(|e| io::Error::other(format!("Failed to create erasure encoder: {:?}", e)))?;
|
||||
Ok(ReedSolomonEncoder::Erasure(Box::new(encoder)))
|
||||
}
|
||||
Ok(ReedSolomonEncoder {
|
||||
data_shards,
|
||||
parity_shards,
|
||||
encoder_cache: std::sync::RwLock::new(None),
|
||||
decoder_cache: std::sync::RwLock::new(None),
|
||||
})
|
||||
}
|
||||
|
||||
/// Encode data shards with parity.
|
||||
pub fn encode(&self, shards: SmallVec<[&mut [u8]; 16]>) -> io::Result<()> {
|
||||
match self {
|
||||
#[cfg(feature = "reed-solomon-simd")]
|
||||
ReedSolomonEncoder::SIMD {
|
||||
data_shards,
|
||||
parity_shards,
|
||||
encoder_cache,
|
||||
..
|
||||
} => {
|
||||
let mut shards_vec: Vec<&mut [u8]> = shards.into_vec();
|
||||
if shards_vec.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
let mut shards_vec: Vec<&mut [u8]> = shards.into_vec();
|
||||
if shards_vec.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// 使用 SIMD 进行编码
|
||||
let simd_result = self.encode_with_simd(*data_shards, *parity_shards, encoder_cache, &mut shards_vec);
|
||||
// 使用 SIMD 进行编码
|
||||
let simd_result = self.encode_with_simd(&mut shards_vec);
|
||||
|
||||
match simd_result {
|
||||
Ok(()) => Ok(()),
|
||||
Err(simd_error) => {
|
||||
warn!("SIMD encoding failed: {}", simd_error);
|
||||
Err(simd_error)
|
||||
}
|
||||
}
|
||||
match simd_result {
|
||||
Ok(()) => Ok(()),
|
||||
Err(simd_error) => {
|
||||
warn!("SIMD encoding failed: {}", simd_error);
|
||||
Err(simd_error)
|
||||
}
|
||||
ReedSolomonEncoder::Erasure(encoder) => encoder
|
||||
.encode(shards)
|
||||
.map_err(|e| io::Error::other(format!("Erasure encode error: {:?}", e))),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "reed-solomon-simd")]
|
||||
fn encode_with_simd(
|
||||
&self,
|
||||
data_shards: usize,
|
||||
parity_shards: usize,
|
||||
encoder_cache: &std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonEncoder>>,
|
||||
shards_vec: &mut [&mut [u8]],
|
||||
) -> io::Result<()> {
|
||||
fn encode_with_simd(&self, shards_vec: &mut [&mut [u8]]) -> io::Result<()> {
|
||||
let shard_len = shards_vec[0].len();
|
||||
|
||||
// 获取或创建encoder
|
||||
let mut encoder = {
|
||||
let mut cache_guard = encoder_cache
|
||||
let mut cache_guard = self
|
||||
.encoder_cache
|
||||
.write()
|
||||
.map_err(|_| io::Error::other("Failed to acquire encoder cache lock"))?;
|
||||
|
||||
match cache_guard.take() {
|
||||
Some(mut cached_encoder) => {
|
||||
// 使用reset方法重置现有encoder以适应新的参数
|
||||
if let Err(e) = cached_encoder.reset(data_shards, parity_shards, shard_len) {
|
||||
if let Err(e) = cached_encoder.reset(self.data_shards, self.parity_shards, shard_len) {
|
||||
warn!("Failed to reset SIMD encoder: {:?}, creating new one", e);
|
||||
// 如果reset失败，创建新的encoder
|
||||
reed_solomon_simd::ReedSolomonEncoder::new(data_shards, parity_shards, shard_len)
|
||||
reed_solomon_simd::ReedSolomonEncoder::new(self.data_shards, self.parity_shards, shard_len)
|
||||
.map_err(|e| io::Error::other(format!("Failed to create SIMD encoder: {:?}", e)))?
|
||||
} else {
|
||||
cached_encoder
|
||||
@@ -165,14 +105,14 @@ impl ReedSolomonEncoder {
|
||||
}
|
||||
None => {
|
||||
// 第一次使用，创建新encoder
|
||||
reed_solomon_simd::ReedSolomonEncoder::new(data_shards, parity_shards, shard_len)
|
||||
reed_solomon_simd::ReedSolomonEncoder::new(self.data_shards, self.parity_shards, shard_len)
|
||||
.map_err(|e| io::Error::other(format!("Failed to create SIMD encoder: {:?}", e)))?
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
// 添加原始shards
|
||||
for (i, shard) in shards_vec.iter().enumerate().take(data_shards) {
|
||||
for (i, shard) in shards_vec.iter().enumerate().take(self.data_shards) {
|
||||
encoder
|
||||
.add_original_shard(shard)
|
||||
.map_err(|e| io::Error::other(format!("Failed to add shard {}: {:?}", i, e)))?;
|
||||
@@ -185,15 +125,16 @@ impl ReedSolomonEncoder {
|
||||
|
||||
// 将恢复shards复制到输出缓冲区
|
||||
for (i, recovery_shard) in result.recovery_iter().enumerate() {
|
||||
if i + data_shards < shards_vec.len() {
|
||||
shards_vec[i + data_shards].copy_from_slice(recovery_shard);
|
||||
if i + self.data_shards < shards_vec.len() {
|
||||
shards_vec[i + self.data_shards].copy_from_slice(recovery_shard);
|
||||
}
|
||||
}
|
||||
|
||||
// 将encoder放回缓存（在result被drop后encoder自动重置，可以重用）
|
||||
drop(result); // 显式drop result，确保encoder被重置
|
||||
|
||||
*encoder_cache
|
||||
*self
|
||||
.encoder_cache
|
||||
.write()
|
||||
.map_err(|_| io::Error::other("Failed to return encoder to cache"))? = Some(encoder);
|
||||
|
||||
@@ -202,39 +143,19 @@ impl ReedSolomonEncoder {
|
||||
|
||||
/// Reconstruct missing shards.
|
||||
pub fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
|
||||
match self {
|
||||
#[cfg(feature = "reed-solomon-simd")]
|
||||
ReedSolomonEncoder::SIMD {
|
||||
data_shards,
|
||||
parity_shards,
|
||||
decoder_cache,
|
||||
..
|
||||
} => {
|
||||
// 使用 SIMD 进行重构
|
||||
let simd_result = self.reconstruct_with_simd(*data_shards, *parity_shards, decoder_cache, shards);
|
||||
// 使用 SIMD 进行重构
|
||||
let simd_result = self.reconstruct_with_simd(shards);
|
||||
|
||||
match simd_result {
|
||||
Ok(()) => Ok(()),
|
||||
Err(simd_error) => {
|
||||
warn!("SIMD reconstruction failed: {}", simd_error);
|
||||
Err(simd_error)
|
||||
}
|
||||
}
|
||||
match simd_result {
|
||||
Ok(()) => Ok(()),
|
||||
Err(simd_error) => {
|
||||
warn!("SIMD reconstruction failed: {}", simd_error);
|
||||
Err(simd_error)
|
||||
}
|
||||
ReedSolomonEncoder::Erasure(encoder) => encoder
|
||||
.reconstruct(shards)
|
||||
.map_err(|e| io::Error::other(format!("Erasure reconstruct error: {:?}", e))),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "reed-solomon-simd")]
|
||||
fn reconstruct_with_simd(
|
||||
&self,
|
||||
data_shards: usize,
|
||||
parity_shards: usize,
|
||||
decoder_cache: &std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonDecoder>>,
|
||||
shards: &mut [Option<Vec<u8>>],
|
||||
) -> io::Result<()> {
|
||||
fn reconstruct_with_simd(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
|
||||
// Find a valid shard to determine length
|
||||
let shard_len = shards
|
||||
.iter()
|
||||
@@ -243,17 +164,18 @@ impl ReedSolomonEncoder {
|
||||
|
||||
// 获取或创建decoder
|
||||
let mut decoder = {
|
||||
let mut cache_guard = decoder_cache
|
||||
let mut cache_guard = self
|
||||
.decoder_cache
|
||||
.write()
|
||||
.map_err(|_| io::Error::other("Failed to acquire decoder cache lock"))?;
|
||||
|
||||
match cache_guard.take() {
|
||||
Some(mut cached_decoder) => {
|
||||
// 使用reset方法重置现有decoder
|
||||
if let Err(e) = cached_decoder.reset(data_shards, parity_shards, shard_len) {
|
||||
if let Err(e) = cached_decoder.reset(self.data_shards, self.parity_shards, shard_len) {
|
||||
warn!("Failed to reset SIMD decoder: {:?}, creating new one", e);
|
||||
// 如果reset失败，创建新的decoder
|
||||
reed_solomon_simd::ReedSolomonDecoder::new(data_shards, parity_shards, shard_len)
|
||||
reed_solomon_simd::ReedSolomonDecoder::new(self.data_shards, self.parity_shards, shard_len)
|
||||
.map_err(|e| io::Error::other(format!("Failed to create SIMD decoder: {:?}", e)))?
|
||||
} else {
|
||||
cached_decoder
|
||||
@@ -261,7 +183,7 @@ impl ReedSolomonEncoder {
|
||||
}
|
||||
None => {
|
||||
// 第一次使用，创建新decoder
|
||||
reed_solomon_simd::ReedSolomonDecoder::new(data_shards, parity_shards, shard_len)
|
||||
reed_solomon_simd::ReedSolomonDecoder::new(self.data_shards, self.parity_shards, shard_len)
|
||||
.map_err(|e| io::Error::other(format!("Failed to create SIMD decoder: {:?}", e)))?
|
||||
}
|
||||
}
|
||||
@@ -270,12 +192,12 @@ impl ReedSolomonEncoder {
|
||||
// Add available shards (both data and parity)
|
||||
for (i, shard_opt) in shards.iter().enumerate() {
|
||||
if let Some(shard) = shard_opt {
|
||||
if i < data_shards {
|
||||
if i < self.data_shards {
|
||||
decoder
|
||||
.add_original_shard(i, shard)
|
||||
.map_err(|e| io::Error::other(format!("Failed to add original shard for reconstruction: {:?}", e)))?;
|
||||
} else {
|
||||
let recovery_idx = i - data_shards;
|
||||
let recovery_idx = i - self.data_shards;
|
||||
decoder
|
||||
.add_recovery_shard(recovery_idx, shard)
|
||||
.map_err(|e| io::Error::other(format!("Failed to add recovery shard for reconstruction: {:?}", e)))?;
|
||||
@@ -289,7 +211,7 @@ impl ReedSolomonEncoder {
|
||||
|
||||
// Fill in missing data shards from reconstruction result
|
||||
for (i, shard_opt) in shards.iter_mut().enumerate() {
|
||||
if shard_opt.is_none() && i < data_shards {
|
||||
if shard_opt.is_none() && i < self.data_shards {
|
||||
for (restored_index, restored_data) in result.restored_original_iter() {
|
||||
if restored_index == i {
|
||||
*shard_opt = Some(restored_data.to_vec());
|
||||
@@ -302,7 +224,8 @@ impl ReedSolomonEncoder {
|
||||
// 将decoder放回缓存（在result被drop后decoder自动重置，可以重用）
|
||||
drop(result); // 显式drop result，确保decoder被重置
|
||||
|
||||
*decoder_cache
|
||||
*self
|
||||
.decoder_cache
|
||||
.write()
|
||||
.map_err(|_| io::Error::other("Failed to return decoder to cache"))? = Some(decoder);
|
||||
|
||||
@@ -592,19 +515,10 @@ mod tests {
|
||||
fn test_encode_decode_roundtrip() {
|
||||
let data_shards = 4;
|
||||
let parity_shards = 2;
|
||||
|
||||
// Use different block sizes based on feature
|
||||
#[cfg(not(feature = "reed-solomon-simd"))]
|
||||
let block_size = 8; // Pure erasure mode (default)
|
||||
#[cfg(feature = "reed-solomon-simd")]
|
||||
let block_size = 1024; // SIMD mode - SIMD with fallback
|
||||
|
||||
let block_size = 1024; // SIMD mode
|
||||
let erasure = Erasure::new(data_shards, parity_shards, block_size);
|
||||
|
||||
// Use different test data based on feature
|
||||
#[cfg(not(feature = "reed-solomon-simd"))]
|
||||
let test_data = b"hello world".to_vec(); // Small data for erasure (default)
|
||||
#[cfg(feature = "reed-solomon-simd")]
|
||||
// Use sufficient test data for SIMD optimization
|
||||
let test_data = b"SIMD mode test data for encoding and decoding roundtrip verification with sufficient length to ensure shard size requirements are met for proper SIMD optimization.".repeat(20); // ~3KB for SIMD
|
||||
|
||||
let data = &test_data;
|
||||
@@ -632,13 +546,7 @@ mod tests {
|
||||
fn test_encode_decode_large_1m() {
|
||||
let data_shards = 4;
|
||||
let parity_shards = 2;
|
||||
|
||||
// Use different block sizes based on feature
|
||||
#[cfg(feature = "reed-solomon-simd")]
|
||||
let block_size = 512 * 3; // SIMD mode
|
||||
#[cfg(not(feature = "reed-solomon-simd"))]
|
||||
let block_size = 8192; // Pure erasure mode (default)
|
||||
|
||||
let erasure = Erasure::new(data_shards, parity_shards, block_size);
|
||||
|
||||
// Generate 1MB test data
|
||||
@@ -704,16 +612,10 @@ mod tests {
|
||||
|
||||
let data_shards = 4;
|
||||
let parity_shards = 2;
|
||||
|
||||
// Use different block sizes based on feature
|
||||
#[cfg(feature = "reed-solomon-simd")]
|
||||
let block_size = 1024; // SIMD mode
|
||||
#[cfg(not(feature = "reed-solomon-simd"))]
|
||||
let block_size = 8; // Pure erasure mode (default)
|
||||
|
||||
let erasure = Arc::new(Erasure::new(data_shards, parity_shards, block_size));
|
||||
|
||||
// Use test data suitable for both modes
|
||||
// Use test data suitable for SIMD mode
|
||||
let data =
|
||||
b"Async error test data with sufficient length to meet requirements for proper testing and validation.".repeat(20); // ~2KB
|
||||
|
||||
@@ -747,13 +649,7 @@ mod tests {
|
||||
|
||||
let data_shards = 4;
|
||||
let parity_shards = 2;
|
||||
|
||||
// Use different block sizes based on feature
|
||||
#[cfg(feature = "reed-solomon-simd")]
|
||||
let block_size = 1024; // SIMD mode
|
||||
#[cfg(not(feature = "reed-solomon-simd"))]
|
||||
let block_size = 8; // Pure erasure mode (default)
|
||||
|
||||
let erasure = Arc::new(Erasure::new(data_shards, parity_shards, block_size));
|
||||
|
||||
// Use test data that fits in exactly one block to avoid multi-block complexity
|
||||
@@ -761,8 +657,6 @@ mod tests {
|
||||
b"Channel async callback test data with sufficient length to ensure proper operation and validation requirements."
|
||||
.repeat(8); // ~1KB
|
||||
|
||||
// let data = b"callback".to_vec(); // 8 bytes to fit exactly in one 8-byte block
|
||||
|
||||
let data_clone = data.clone(); // Clone for later comparison
|
||||
let mut reader = Cursor::new(data);
|
||||
let (tx, mut rx) = mpsc::channel::<Vec<Bytes>>(8);
|
||||
@@ -801,8 +695,7 @@ mod tests {
|
||||
assert_eq!(&recovered, &data_clone);
|
||||
}
|
||||
|
||||
// Tests specifically for SIMD mode
|
||||
#[cfg(feature = "reed-solomon-simd")]
|
||||
// SIMD mode specific tests
|
||||
mod simd_tests {
|
||||
use super::*;
|
||||
|
||||
@@ -1171,47 +1064,4 @@ mod tests {
|
||||
assert_eq!(&recovered, &data_clone);
|
||||
}
|
||||
}
|
||||
|
||||
// Comparative tests between different implementations
|
||||
#[cfg(not(feature = "reed-solomon-simd"))]
|
||||
mod comparative_tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_implementation_consistency() {
|
||||
let data_shards = 4;
|
||||
let parity_shards = 2;
|
||||
let block_size = 2048; // Large enough for SIMD requirements
|
||||
|
||||
// Create test data that ensures each shard is >= 512 bytes (SIMD minimum)
|
||||
let test_data = b"This is test data for comparing reed-solomon-simd and reed-solomon-erasure implementations to ensure they produce consistent results when given the same input parameters and data. This data needs to be sufficiently large to meet SIMD requirements.";
|
||||
let data = test_data.repeat(50); // Create much larger data: ~13KB total, ~3.25KB per shard
|
||||
|
||||
// Test with erasure implementation (default)
|
||||
let erasure_erasure = Erasure::new(data_shards, parity_shards, block_size);
|
||||
let erasure_shards = erasure_erasure.encode_data(&data).unwrap();
|
||||
|
||||
// Test data integrity with erasure
|
||||
let mut erasure_shards_opt: Vec<Option<Vec<u8>>> = erasure_shards.iter().map(|shard| Some(shard.to_vec())).collect();
|
||||
|
||||
// Lose some shards
|
||||
erasure_shards_opt[1] = None; // Data shard
|
||||
erasure_shards_opt[4] = None; // Parity shard
|
||||
|
||||
erasure_erasure.decode_data(&mut erasure_shards_opt).unwrap();
|
||||
|
||||
let mut erasure_recovered = Vec::new();
|
||||
for shard in erasure_shards_opt.iter().take(data_shards) {
|
||||
erasure_recovered.extend_from_slice(shard.as_ref().unwrap());
|
||||
}
|
||||
erasure_recovered.truncate(data.len());
|
||||
|
||||
// Verify erasure implementation works correctly
|
||||
assert_eq!(&erasure_recovered, &data, "Erasure implementation failed to recover data correctly");
|
||||
|
||||
println!("✅ Both implementations are available and working correctly");
|
||||
println!("✅ Default (reed-solomon-erasure): Data recovery successful");
|
||||
println!("✅ SIMD tests are available as separate test suite");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+24
-24
@@ -1,8 +1,8 @@
|
||||
use crate::bitrot::{create_bitrot_reader, create_bitrot_writer};
|
||||
use crate::disk::error_reduce::{reduce_read_quorum_errs, reduce_write_quorum_errs, OBJECT_OP_IGNORED_ERRS};
|
||||
use crate::disk::error_reduce::{OBJECT_OP_IGNORED_ERRS, reduce_read_quorum_errs, reduce_write_quorum_errs};
|
||||
use crate::disk::{
|
||||
self, conv_part_err_to_int, has_part_err, CHECK_PART_DISK_NOT_FOUND, CHECK_PART_FILE_CORRUPT,
|
||||
CHECK_PART_FILE_NOT_FOUND, CHECK_PART_SUCCESS,
|
||||
self, CHECK_PART_DISK_NOT_FOUND, CHECK_PART_FILE_CORRUPT, CHECK_PART_FILE_NOT_FOUND, CHECK_PART_SUCCESS,
|
||||
conv_part_err_to_int, has_part_err,
|
||||
};
|
||||
use crate::erasure_coding;
|
||||
use crate::erasure_coding::bitrot_verify;
|
||||
@@ -12,24 +12,24 @@ use crate::heal::data_usage_cache::DataUsageCache;
|
||||
use crate::heal::heal_ops::{HealEntryFn, HealSequence};
|
||||
use crate::store_api::ObjectToDelete;
|
||||
use crate::{
|
||||
cache_value::metacache_set::{list_path_raw, ListPathRawOptions},
|
||||
config::{storageclass, GLOBAL_StorageClass},
|
||||
cache_value::metacache_set::{ListPathRawOptions, list_path_raw},
|
||||
config::{GLOBAL_StorageClass, storageclass},
|
||||
disk::{
|
||||
endpoint::Endpoint, error::DiskError, format::FormatV3, new_disk, CheckPartsResp, DeleteOptions, DiskAPI, DiskInfo,
|
||||
DiskInfoOptions, DiskOption, DiskStore, FileInfoVersions, ReadMultipleReq, ReadMultipleResp,
|
||||
ReadOptions, UpdateMetadataOpts, RUSTFS_META_BUCKET, RUSTFS_META_MULTIPART_BUCKET, RUSTFS_META_TMP_BUCKET,
|
||||
CheckPartsResp, DeleteOptions, DiskAPI, DiskInfo, DiskInfoOptions, DiskOption, DiskStore, FileInfoVersions,
|
||||
RUSTFS_META_BUCKET, RUSTFS_META_MULTIPART_BUCKET, RUSTFS_META_TMP_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions,
|
||||
UpdateMetadataOpts, endpoint::Endpoint, error::DiskError, format::FormatV3, new_disk,
|
||||
},
|
||||
error::{to_object_err, StorageError},
|
||||
error::{StorageError, to_object_err},
|
||||
global::{
|
||||
get_global_deployment_id, is_dist_erasure, GLOBAL_BackgroundHealState, GLOBAL_LOCAL_DISK_MAP,
|
||||
GLOBAL_LOCAL_DISK_SET_DRIVES,
|
||||
GLOBAL_BackgroundHealState, GLOBAL_LOCAL_DISK_MAP, GLOBAL_LOCAL_DISK_SET_DRIVES, get_global_deployment_id,
|
||||
is_dist_erasure,
|
||||
},
|
||||
heal::{
|
||||
data_usage::{DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT},
|
||||
data_usage_cache::{DataUsageCacheInfo, DataUsageEntry, DataUsageEntryInfo},
|
||||
heal_commands::{
|
||||
HealOpts, HealScanMode, HealingTracker, DRIVE_STATE_CORRUPT, DRIVE_STATE_MISSING, DRIVE_STATE_OFFLINE,
|
||||
DRIVE_STATE_OK, HEAL_DEEP_SCAN, HEAL_ITEM_OBJECT, HEAL_NORMAL_SCAN,
|
||||
DRIVE_STATE_CORRUPT, DRIVE_STATE_MISSING, DRIVE_STATE_OFFLINE, DRIVE_STATE_OK, HEAL_DEEP_SCAN, HEAL_ITEM_OBJECT,
|
||||
HEAL_NORMAL_SCAN, HealOpts, HealScanMode, HealingTracker,
|
||||
},
|
||||
heal_ops::BG_HEALING_UUID,
|
||||
},
|
||||
@@ -42,7 +42,7 @@ use crate::{
|
||||
};
|
||||
use crate::{disk::STORAGE_FORMAT_FILE, heal::mrf::PartialOperation};
|
||||
use crate::{
|
||||
heal::data_scanner::{globalHealConfig, HEAL_DELETE_DANGLING},
|
||||
heal::data_scanner::{HEAL_DELETE_DANGLING, globalHealConfig},
|
||||
store_api::ListObjectVersionsInfo,
|
||||
};
|
||||
use bytes::Bytes;
|
||||
@@ -51,22 +51,22 @@ use chrono::Utc;
|
||||
use futures::future::join_all;
|
||||
use glob::Pattern;
|
||||
use http::HeaderMap;
|
||||
use lock::{namespace_lock::NsLockMap, LockApi};
|
||||
use lock::{LockApi, namespace_lock::NsLockMap};
|
||||
use madmin::heal_commands::{HealDriveInfo, HealResultItem};
|
||||
use md5::{Digest as Md5Digest, Md5};
|
||||
use rand::{seq::SliceRandom, Rng};
|
||||
use rand::{Rng, seq::SliceRandom};
|
||||
use rustfs_filemeta::headers::RESERVED_METADATA_PREFIX_LOWER;
|
||||
use rustfs_filemeta::{
|
||||
file_info_from_raw, headers::{AMZ_OBJECT_TAGGING, AMZ_STORAGE_CLASS}, merge_file_meta_versions, FileInfo, FileMeta, FileMetaShallowVersion, MetaCacheEntries,
|
||||
MetaCacheEntry, MetadataResolutionParams,
|
||||
ObjectPartInfo,
|
||||
RawFileInfo,
|
||||
FileInfo, FileMeta, FileMetaShallowVersion, MetaCacheEntries, MetaCacheEntry, MetadataResolutionParams, ObjectPartInfo,
|
||||
RawFileInfo, file_info_from_raw,
|
||||
headers::{AMZ_OBJECT_TAGGING, AMZ_STORAGE_CLASS},
|
||||
merge_file_meta_versions,
|
||||
};
|
||||
use rustfs_rio::{EtagResolvable, HashReader, TryGetIndex as _, WarpReader};
|
||||
use rustfs_utils::{
|
||||
crypto::{base64_decode, base64_encode, hex},
|
||||
path::{encode_dir_object, has_suffix, path_join_buf, SLASH_SEPARATOR},
|
||||
HashAlgorithm,
|
||||
crypto::{base64_decode, base64_encode, hex},
|
||||
path::{SLASH_SEPARATOR, encode_dir_object, has_suffix, path_join_buf},
|
||||
};
|
||||
use sha2::Sha256;
|
||||
use std::hash::Hash;
|
||||
@@ -82,7 +82,7 @@ use std::{
|
||||
use time::OffsetDateTime;
|
||||
use tokio::{
|
||||
io::AsyncWrite,
|
||||
sync::{broadcast, RwLock},
|
||||
sync::{RwLock, broadcast},
|
||||
};
|
||||
use tokio::{
|
||||
select,
|
||||
@@ -5837,9 +5837,9 @@ fn get_complete_multipart_md5(parts: &[CompletePart]) -> String {
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::disk::error::DiskError;
|
||||
use crate::disk::CHECK_PART_UNKNOWN;
|
||||
use crate::disk::CHECK_PART_VOLUME_NOT_FOUND;
|
||||
use crate::disk::error::DiskError;
|
||||
use crate::store_api::CompletePart;
|
||||
use rustfs_filemeta::ErasureInfo;
|
||||
use std::collections::HashMap;
|
||||
|
||||
@@ -8,10 +8,10 @@ use crate::{disk::DiskStore, heal::heal_commands::HealOpts};
|
||||
use http::{HeaderMap, HeaderValue};
|
||||
use madmin::heal_commands::HealResultItem;
|
||||
use rustfs_filemeta::headers::RESERVED_METADATA_PREFIX_LOWER;
|
||||
use rustfs_filemeta::{headers::AMZ_OBJECT_TAGGING, FileInfo, MetaCacheEntriesSorted, ObjectPartInfo};
|
||||
use rustfs_filemeta::{FileInfo, MetaCacheEntriesSorted, ObjectPartInfo, headers::AMZ_OBJECT_TAGGING};
|
||||
use rustfs_rio::{DecompressReader, HashReader, LimitReader, WarpReader};
|
||||
use rustfs_utils::path::decode_dir_object;
|
||||
use rustfs_utils::CompressionAlgorithm;
|
||||
use rustfs_utils::path::decode_dir_object;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::HashMap;
|
||||
use std::fmt::Debug;
|
||||
|
||||
@@ -1,24 +1,24 @@
|
||||
use crate::StorageAPI;
|
||||
use crate::bucket::metadata_sys::get_versioning_config;
|
||||
use crate::bucket::versioning::VersioningApi;
|
||||
use crate::cache_value::metacache_set::{list_path_raw, ListPathRawOptions};
|
||||
use crate::cache_value::metacache_set::{ListPathRawOptions, list_path_raw};
|
||||
use crate::disk::error::DiskError;
|
||||
use crate::disk::{DiskInfo, DiskStore};
|
||||
use crate::error::{
|
||||
is_all_not_found, is_all_volume_not_found, is_err_bucket_not_found, to_object_err, Error, Result, StorageError,
|
||||
Error, Result, StorageError, is_all_not_found, is_all_volume_not_found, is_err_bucket_not_found, to_object_err,
|
||||
};
|
||||
use crate::set_disk::SetDisks;
|
||||
use crate::store::check_list_objs_args;
|
||||
use crate::store_api::{ListObjectVersionsInfo, ListObjectsInfo, ObjectInfo, ObjectOptions};
|
||||
use crate::store_utils::is_reserved_or_invalid_bucket;
|
||||
use crate::StorageAPI;
|
||||
use crate::{store::ECStore, store_api::ListObjectsV2Info};
|
||||
use futures::future::join_all;
|
||||
use rand::seq::SliceRandom;
|
||||
use rustfs_filemeta::{
|
||||
merge_file_meta_versions, FileInfo, MetaCacheEntries, MetaCacheEntriesSorted, MetaCacheEntriesSortedResult, MetaCacheEntry,
|
||||
MetadataResolutionParams,
|
||||
FileInfo, MetaCacheEntries, MetaCacheEntriesSorted, MetaCacheEntriesSortedResult, MetaCacheEntry, MetadataResolutionParams,
|
||||
merge_file_meta_versions,
|
||||
};
|
||||
use rustfs_utils::path::{self, base_dir_from_prefix, SLASH_SEPARATOR};
|
||||
use rustfs_utils::path::{self, SLASH_SEPARATOR, base_dir_from_prefix};
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use tokio::sync::broadcast::{self, Receiver as B_Receiver};
|
||||
|
||||
Reference in New Issue
Block a user