rustfs/ecstore/BENCHMARK.md

Reed-Solomon Erasure Coding Performance Benchmark

This directory contains a comprehensive benchmark suite for comparing the performance of different Reed-Solomon implementations.

📊 Test Overview

Supported Implementation Modes

🏛️ Pure Erasure Mode (Default, Recommended)

• Stable and Reliable: Uses mature reed-solomon-erasure implementation
• Wide Compatibility: Supports arbitrary shard sizes
• Memory Efficient: Optimized memory usage patterns
• Predictable: Performance insensitive to shard size
• Use Case: Default choice for production environments, suitable for most application scenarios

🎯 SIMD Mode (reed-solomon-simd feature)

• High Performance Optimization: Uses SIMD instruction sets for high-performance encoding/decoding
• Performance Oriented: Focuses on maximizing processing performance
• Target Scenarios: High-performance scenarios for large data processing
• Use Case: Scenarios requiring maximum performance, suitable for handling large amounts of data

Test Dimensions

• Encoding Performance - Speed of encoding data into erasure code shards
• Decoding Performance - Speed of recovering original data from erasure code shards
• Shard Size Sensitivity - Impact of different shard sizes on performance
• Erasure Code Configuration - Performance impact of different data/parity shard ratios
• SIMD Mode Performance - Performance characteristics of SIMD optimization
• Concurrency Performance - Performance in multi-threaded environments
• Memory Efficiency - Memory usage patterns and efficiency
• Error Recovery Capability - Recovery performance under different numbers of lost shards

🚀 Quick Start

Run Quick Tests

# Run quick performance comparison tests (default pure Erasure mode)
./run_benchmarks.sh quick

Run Complete Comparison Tests

# Run detailed implementation comparison tests
./run_benchmarks.sh comparison

Run Specific Mode Tests

# Test default pure erasure mode (recommended)
./run_benchmarks.sh erasure

# Test SIMD mode
./run_benchmarks.sh simd

📈 Manual Benchmark Execution

Basic Usage

# Run all benchmarks (default pure erasure mode)
cargo bench

# Run specific benchmark files
cargo bench --bench erasure_benchmark
cargo bench --bench comparison_benchmark

Compare Different Implementation Modes

# Test default pure erasure mode
cargo bench --bench comparison_benchmark

# Test SIMD mode
cargo bench --bench comparison_benchmark \
    --features reed-solomon-simd

# Save baseline for comparison
cargo bench --bench comparison_benchmark \
    -- --save-baseline erasure_baseline

# Compare SIMD mode performance with baseline
cargo bench --bench comparison_benchmark \
    --features reed-solomon-simd \
    -- --baseline erasure_baseline

Filter Specific Tests

# Run only encoding tests
cargo bench encode

# Run only decoding tests  
cargo bench decode

# Run tests for specific data sizes
cargo bench 1MB

# Run tests for specific configurations
cargo bench "4+2"

📊 View Results

HTML Reports

Benchmark results automatically generate HTML reports:

# Start local server to view reports
cd target/criterion
python3 -m http.server 8080

# Access in browser
open http://localhost:8080/report/index.html

Command Line Output

Benchmarks display in terminal:

• Operations per second (ops/sec)
• Throughput (MB/s)
• Latency statistics (mean, standard deviation, percentiles)
• Performance trend changes

🔧 Test Configuration

Data Sizes

• Small Data: 1KB, 8KB - Test small file scenarios
• Medium Data: 64KB, 256KB - Test common file sizes
• Large Data: 1MB, 4MB - Test large file processing and SIMD optimization
• Very Large Data: 16MB+ - Test high throughput scenarios

Erasure Code Configurations

• (4,2) - Common configuration, 33% redundancy
• (6,3) - 50% redundancy, balanced performance and reliability
• (8,4) - 50% redundancy, more parallelism
• (10,5), (12,6) - High parallelism configurations

Shard Sizes

Test different shard sizes from 32 bytes to 8KB, with special focus on:

• Memory Alignment: 64, 128, 256 bytes - Impact of memory alignment on performance
• Cache Friendly: 1KB, 2KB, 4KB - CPU cache-friendly sizes

📝 Interpreting Test Results

Performance Metrics

1. Throughput

   • Unit: MB/s or GB/s
   • Measures data processing speed
   • Higher is better

2. Latency

   • Unit: microseconds (μs) or milliseconds (ms)
   • Measures single operation time
   • Lower is better

3. CPU Efficiency

   • Bytes processed per CPU cycle
   • Reflects algorithm efficiency

Expected Results

Pure Erasure Mode (Default):

• Stable performance, insensitive to shard size
• Best compatibility, supports all configurations
• Stable and predictable memory usage

SIMD Mode (reed-solomon-simd feature):

• High-performance SIMD optimized implementation
• Suitable for large data processing scenarios
• Focuses on maximizing performance

Shard Size Sensitivity:

• SIMD mode may be more sensitive to shard sizes
• Pure Erasure mode relatively insensitive to shard size

Memory Usage:

• SIMD mode may have specific memory alignment requirements
• Pure Erasure mode has more stable memory usage

🛠️ Custom Testing

Adding New Test Scenarios

Edit benches/erasure_benchmark.rs or benches/comparison_benchmark.rs:

// Add new test configuration
let configs = vec![
    // Your custom configuration
    BenchConfig::new(10, 4, 2048 * 1024, 2048 * 1024), // 10+4, 2MB
];

Adjust Test Parameters

// Modify sampling and test time
group.sample_size(20);  // Sample count
group.measurement_time(Duration::from_secs(10));  // Test duration

🐛 Troubleshooting

Common Issues

1. Compilation Errors: Ensure correct dependencies are installed

cargo update
cargo build --all-features

2. Performance Anomalies: Check if running in correct mode

# Check current configuration
cargo bench --bench comparison_benchmark -- --help

3. Tests Taking Too Long: Adjust test parameters

# Use shorter test duration
cargo bench -- --quick

Performance Analysis

Use tools like perf for detailed performance analysis:

# Analyze CPU usage
cargo bench --bench comparison_benchmark & 
perf record -p $(pgrep -f comparison_benchmark)
perf report

🤝 Contributing

Welcome to submit new benchmark scenarios or optimization suggestions:

1. Fork the project
2. Create feature branch: git checkout -b feature/new-benchmark
3. Add test cases
4. Commit changes: git commit -m 'Add new benchmark for XYZ'
5. Push to branch: git push origin feature/new-benchmark
6. Create Pull Request

📚 References

• reed-solomon-erasure crate
• reed-solomon-simd crate
• Criterion.rs benchmark framework
• Reed-Solomon error correction principles

---

💡 Tips:

• Recommend using the default pure Erasure mode, which provides stable performance across various scenarios
• Consider SIMD mode for high-performance requirements
• Benchmark results may vary based on hardware, operating system, and compiler versions
• Suggest running tests in target deployment environment for most accurate performance data