RustFS ECStore is the core storage engine of the RustFS distributed object storage system. It provides enterprise-grade erasure coding capabilities, data integrity protection, and high-performance object storage operations. This module serves as the foundation for RustFS's distributed storage architecture.

Note: This is a core submodule of RustFS and provides the primary storage capabilities for the distributed object storage system. For the complete RustFS experience, please visit the main RustFS repository.

✨ Features

🔧 Erasure Coding Storage

Reed-Solomon Erasure Coding: Advanced error correction with configurable redundancy
Data Durability: Protection against disk failures and bit rot
Automatic Repair: Self-healing capabilities for corrupted or missing data
Configurable Parity: Flexible parity configurations (4+2, 8+4, 16+4, etc.)

💾 Storage Management

Multi-Disk Support: Intelligent disk management and load balancing
Storage Classes: Support for different storage tiers and policies
Bucket Management: Advanced bucket operations and lifecycle management
Object Versioning: Complete versioning support with metadata tracking

🚀 Performance & Scalability

High Throughput: Optimized for large-scale data operations
Parallel Processing: Concurrent read/write operations across multiple disks
Memory Efficient: Smart caching and memory management
SIMD Optimization: Hardware-accelerated erasure coding operations

🛡️ Data Integrity

Bitrot Detection: Real-time data corruption detection
Checksum Verification: Multiple checksum algorithms (MD5, SHA256, XXHash)
Healing System: Automatic background healing and repair
Data Scrubbing: Proactive data integrity scanning

🔄 Advanced Features

Compression: Built-in compression support for space optimization
Replication: Cross-region replication capabilities
Notification System: Real-time event notifications
Metrics & Monitoring: Comprehensive performance metrics

🏗️ Architecture

Storage Layout

ECStore Architecture:
┌─────────────────────────────────────────────────────────────┐
│                    Storage API Layer                        │
├─────────────────────────────────────────────────────────────┤
│  Bucket Management  │  Object Operations  │  Metadata Mgmt  │
├─────────────────────────────────────────────────────────────┤
│              Erasure Coding Engine                          │
├─────────────────────────────────────────────────────────────┤
│    Disk Management    │    Healing System    │    Cache     │
├─────────────────────────────────────────────────────────────┤
│              Physical Storage Devices                       │
└─────────────────────────────────────────────────────────────┘

Erasure Coding Schemes

Configuration	Data Drives	Parity Drives	Fault Tolerance	Storage Efficiency
4+2	4	2	2 disk failures	66.7%
8+4	8	4	4 disk failures	66.7%
16+4	16	4	4 disk failures	80%
Custom	N	K	K disk failures	N/(N+K)

📦 Installation

Add this to your Cargo.toml:

[dependencies]
rustfs-ecstore = "0.1.0"

🔧 Usage

Basic Storage Operations

use rustfs_ecstore::{StorageAPI, new_object_layer_fn};
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize storage layer
    let storage = new_object_layer_fn("/path/to/storage").await?;

    // Create a bucket
    storage.make_bucket("my-bucket", None).await?;

    // Put an object
    let data = b"Hello, RustFS!";
    storage.put_object("my-bucket", "hello.txt", data.to_vec()).await?;

    // Get an object
    let retrieved = storage.get_object("my-bucket", "hello.txt", None).await?;
    println!("Retrieved: {}", String::from_utf8_lossy(&retrieved.data));

    Ok(())
}

Advanced Configuration

use rustfs_ecstore::{StorageAPI, config::Config};

async fn setup_storage_with_config() -> Result<(), Box<dyn std::error::Error>> {
    let config = Config {
        erasure_sets: vec![
            // 8+4 configuration for high durability
            ErasureSet::new(8, 4, vec![
                "/disk1", "/disk2", "/disk3", "/disk4",
                "/disk5", "/disk6", "/disk7", "/disk8",
                "/disk9", "/disk10", "/disk11", "/disk12"
            ])
        ],
        healing_enabled: true,
        compression_enabled: true,
        ..Default::default()
    };

    let storage = new_object_layer_fn("/path/to/storage")
        .with_config(config)
        .await?;

    Ok(())
}

Bucket Management

use rustfs_ecstore::{StorageAPI, bucket::BucketInfo};

async fn bucket_operations(storage: Arc<dyn StorageAPI>) -> Result<(), Box<dyn std::error::Error>> {
    // Create bucket with specific configuration
    let bucket_info = BucketInfo {
        name: "enterprise-bucket".to_string(),
        versioning_enabled: true,
        lifecycle_config: Some(lifecycle_config()),
        ..Default::default()
    };

    storage.make_bucket_with_config(bucket_info).await?;

    // List buckets
    let buckets = storage.list_buckets().await?;
    for bucket in buckets {
        println!("Bucket: {}, Created: {}", bucket.name, bucket.created);
    }

    // Set bucket policy
    storage.set_bucket_policy("enterprise-bucket", policy_json).await?;

    Ok(())
}

Healing and Maintenance

use rustfs_ecstore::{heal::HealingManager, StorageAPI};

async fn healing_operations(storage: Arc<dyn StorageAPI>) -> Result<(), Box<dyn std::error::Error>> {
    // Check storage health
    let health = storage.storage_info().await?;
    println!("Storage Health: {:?}", health);

    // Trigger healing for specific bucket
    let healing_result = storage.heal_bucket("my-bucket").await?;
    println!("Healing completed: {:?}", healing_result);

    // Background healing status
    let healing_status = storage.healing_status().await?;
    println!("Background healing: {:?}", healing_status);

    Ok(())
}

🧪 Testing

Run the test suite:

# Run all tests
cargo test

# Run tests with specific features
cargo test --features "compression,healing"

# Run benchmarks
cargo bench

# Run erasure coding benchmarks
cargo bench --bench erasure_benchmark

# Run comparison benchmarks
cargo bench --bench comparison_benchmark

📊 Performance Benchmarks

ECStore is designed for high-performance storage operations:

Throughput Performance

Sequential Write: Up to 10GB/s on NVMe storage
Sequential Read: Up to 12GB/s with parallel reads
Random I/O: 100K+ IOPS for small objects
Erasure Coding: 5GB/s encoding/decoding throughput

Scalability Metrics

Storage Capacity: Exabyte-scale deployments
Concurrent Operations: 10,000+ concurrent requests
Disk Scaling: Support for 1000+ disks per node
Fault Tolerance: Up to 50% disk failure resilience

🔧 Configuration

Storage Configuration

[storage]
# Erasure coding configuration
erasure_set_size = 12  # Total disks per set
data_drives = 8        # Data drives per set
parity_drives = 4      # Parity drives per set

# Performance tuning
read_quorum = 6        # Minimum disks for read
write_quorum = 7       # Minimum disks for write
parallel_reads = true  # Enable parallel reads
compression = true     # Enable compression

# Healing configuration
healing_enabled = true
healing_interval = "24h"
bitrot_check_interval = "168h"  # Weekly bitrot check

Advanced Features

use rustfs_ecstore::config::StorageConfig;

let config = StorageConfig {
    // Enable advanced features
    bitrot_protection: true,
    automatic_healing: true,
    compression_level: 6,
    checksum_algorithm: ChecksumAlgorithm::XXHash64,

    // Performance tuning
    read_buffer_size: 1024 * 1024,  // 1MB read buffer
    write_buffer_size: 4 * 1024 * 1024,  // 4MB write buffer
    concurrent_operations: 1000,

    // Storage optimization
    small_object_threshold: 128 * 1024,  // 128KB
    large_object_threshold: 64 * 1024 * 1024,  // 64MB

    ..Default::default()
};

🤝 Integration with RustFS

ECStore integrates seamlessly with other RustFS components:

API Server: Provides S3-compatible storage operations
IAM Module: Handles authentication and authorization
Policy Engine: Implements bucket policies and access controls
Notification System: Publishes storage events
Monitoring: Provides detailed metrics and health status

📋 Requirements

Rust: 1.70.0 or later
Platforms: Linux, macOS, Windows
Storage: Local disks, network storage, cloud storage
Memory: Minimum 4GB RAM (8GB+ recommended)
Network: High-speed network for distributed deployments

🚀 Performance Tuning

Optimization Tips

Disk Configuration:
- Use dedicated disks for each erasure set
- Prefer NVMe over SATA for better performance
- Ensure consistent disk sizes within erasure sets
Memory Settings:
- Allocate sufficient memory for caching
- Tune read/write buffer sizes based on workload
- Enable memory-mapped files for large objects
Network Optimization:
- Use high-speed network connections
- Configure proper MTU sizes
- Enable network compression for WAN scenarios
CPU Optimization:
- Utilize SIMD instructions for erasure coding
- Balance CPU cores across erasure sets
- Enable hardware-accelerated checksums

🐛 Troubleshooting

Common Issues

Disk Failures:
- Check disk health using storage_info()
- Trigger healing with heal_bucket()
- Replace failed disks and re-add to cluster
Performance Issues:
- Monitor disk I/O utilization
- Check network bandwidth usage
- Verify erasure coding configuration
Data Integrity:
- Run bitrot detection scans
- Verify checksums for critical data
- Check healing system status

This module is part of the RustFS ecosystem:

RustFS Main - Core distributed storage system
RustFS Crypto - Cryptographic operations
RustFS IAM - Identity and access management
RustFS Policy - Policy engine
RustFS FileMeta - File metadata management

📚 Documentation

For comprehensive documentation, visit:

🔗 Links

Documentation - Complete RustFS manual
Changelog - Release notes and updates
GitHub Discussions - Community support

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details on:

Storage engine architecture and design patterns
Erasure coding implementation guidelines
Performance optimization techniques
Testing procedures for storage operations
Documentation standards for storage APIs

Development Setup

# Clone the repository
git clone https://github.com/rustfs/rustfs.git
cd rustfs

# Navigate to ECStore module
cd crates/ecstore

# Install dependencies
cargo build

# Run tests
cargo test

# Run benchmarks
cargo bench

# Format code
cargo fmt

# Run linter
cargo clippy

💬 Getting Help

Documentation: docs.rustfs.com
Issues: GitHub Issues
Discussions: GitHub Discussions
Storage Support: storage-support@rustfs.com

📞 Contact

Bugs: GitHub Issues
Business: hello@rustfs.com
Jobs: jobs@rustfs.com
General Discussion: GitHub Discussions

👥 Contributors

This module is maintained by the RustFS storage team and community contributors. Special thanks to all who have contributed to making RustFS storage reliable and efficient.

📄 License

Licensed under the Apache License, Version 2.0. See LICENSE for details.

Copyright 2024 RustFS Team

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

RustFS is a trademark of RustFS, Inc.
All other trademarks are the property of their respective owners.

Made with ❤️ by the RustFS Storage Team

README.md

RustFS ECStore - Erasure Coding Storage Engine

📖 Overview