feat: add comprehensive README documentation for all RustFS submodules (#48)

2026-08-26 05:56:50 +00:00 · 2025-07-04 23:02:13 +08:00
parent c41299a29f
commit fa17f7b1e3
20 changed files with 8752 additions and 172 deletions
@@ -0,0 +1,414 @@
+[![RustFS](https://rustfs.com/images/rustfs-github.png)](https://rustfs.com)
+
+# RustFS Rio - High-Performance I/O
+
+<p align="center">
+  <strong>High-performance asynchronous I/O operations for RustFS distributed object storage</strong>
+</p>
+
+<p align="center">
+  <a href="https://github.com/rustfs/rustfs/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rustfs/rustfs/actions/workflows/ci.yml/badge.svg" /></a>
+  <a href="https://docs.rustfs.com/en/">📖 Documentation</a>
+  · <a href="https://github.com/rustfs/rustfs/issues">🐛 Bug Reports</a>
+  · <a href="https://github.com/rustfs/rustfs/discussions">💬 Discussions</a>
+</p>
+
+---
+
+## 📖 Overview
+
+**RustFS Rio** provides high-performance asynchronous I/O operations for the [RustFS](https://rustfs.com) distributed object storage system. It implements efficient data streaming, encryption, compression, and integrity checking with zero-copy operations and optimized buffering strategies.
+
+> **Note:** This is a performance-critical submodule of RustFS that provides essential I/O capabilities for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+
+## ✨ Features
+
+### 🚀 High-Performance I/O
+
+- **Zero-Copy Operations**: Efficient data movement without unnecessary copying
+- **Async Streaming**: Non-blocking streaming I/O with backpressure handling
+- **Vectored I/O**: Scatter-gather operations for improved throughput
+- **Buffer Management**: Intelligent buffer pooling and reuse
+
+### 🔐 Cryptographic Operations
+
+- **AES-GCM Encryption**: Hardware-accelerated encryption/decryption
+- **Streaming Encryption**: Encrypt data on-the-fly without buffering
+- **Key Management**: Secure key derivation and rotation
+- **Digital Signatures**: Data integrity verification
+
+### 📦 Compression Support
+
+- **Multi-Algorithm**: Support for various compression algorithms
+- **Streaming Compression**: Real-time compression during transfer
+- **Adaptive Compression**: Dynamic algorithm selection based on data
+- **Compression Levels**: Configurable compression vs. speed tradeoffs
+
+### 🔧 Data Integrity
+
+- **CRC32 Checksums**: Fast integrity checking
+- **MD5 Hashing**: Legacy compatibility and verification
+- **Merkle Trees**: Hierarchical integrity verification
+- **Error Correction**: Automatic error detection and correction
+
+## 📦 Installation
+
+Add this to your `Cargo.toml`:
+
+```toml
+[dependencies]
+rustfs-rio = "0.1.0"
+```
+
+## 🔧 Usage
+
+### Basic Streaming I/O
+
+```rust
+use rustfs_rio::{StreamReader, StreamWriter, BufferPool};
+use tokio::io::{AsyncReadExt, AsyncWriteExt};
+
+#[tokio::main]
+async fn main() -> Result<(), Box<dyn std::error::Error>> {
+    // Create buffer pool for efficient memory management
+    let buffer_pool = BufferPool::new(64 * 1024, 100); // 64KB buffers, 100 in pool
+
+    // Create streaming reader
+    let mut reader = StreamReader::new(input_source, buffer_pool.clone());
+
+    // Create streaming writer
+    let mut writer = StreamWriter::new(output_destination, buffer_pool.clone());
+
+    // High-performance streaming copy
+    let mut buffer = vec![0u8; 8192];
+    loop {
+        let n = reader.read(&mut buffer).await?;
+        if n == 0 {
+            break;
+        }
+        writer.write_all(&buffer[..n]).await?;
+    }
+
+    writer.flush().await?;
+    Ok(())
+}
+```
+
+### Encrypted Streaming
+
+```rust
+use rustfs_rio::{EncryptedWriter, EncryptedReader, EncryptionKey};
+use aes_gcm::{Aes256Gcm, Key, Nonce};
+
+async fn encrypted_streaming_example() -> Result<(), Box<dyn std::error::Error>> {
+    // Generate encryption key
+    let key = EncryptionKey::generate()?;
+
+    // Create encrypted writer
+    let mut encrypted_writer = EncryptedWriter::new(
+        output_stream,
+        key.clone(),
+        Aes256Gcm::new(&key.into())
+    )?;
+
+    // Write encrypted data
+    encrypted_writer.write_all(b"Hello, encrypted world!").await?;
+    encrypted_writer.finalize().await?;
+
+    // Create encrypted reader
+    let mut encrypted_reader = EncryptedReader::new(
+        input_stream,
+        key.clone(),
+        Aes256Gcm::new(&key.into())
+    )?;
+
+    // Read decrypted data
+    let mut decrypted_data = Vec::new();
+    encrypted_reader.read_to_end(&mut decrypted_data).await?;
+
+    println!("Decrypted: {}", String::from_utf8(decrypted_data)?);
+    Ok(())
+}
+```
+
+### Compressed Streaming
+
+```rust
+use rustfs_rio::{CompressedWriter, CompressedReader, CompressionAlgorithm};
+
+async fn compressed_streaming_example() -> Result<(), Box<dyn std::error::Error>> {
+    // Create compressed writer
+    let mut compressed_writer = CompressedWriter::new(
+        output_stream,
+        CompressionAlgorithm::Zstd,
+        6 // compression level
+    )?;
+
+    // Write compressed data
+    compressed_writer.write_all(b"This data will be compressed").await?;
+    compressed_writer.write_all(b"and streamed efficiently").await?;
+    compressed_writer.finish().await?;
+
+    // Create compressed reader
+    let mut compressed_reader = CompressedReader::new(
+        input_stream,
+        CompressionAlgorithm::Zstd
+    )?;
+
+    // Read decompressed data
+    let mut decompressed_data = Vec::new();
+    compressed_reader.read_to_end(&mut decompressed_data).await?;
+
+    println!("Decompressed: {}", String::from_utf8(decompressed_data)?);
+    Ok(())
+}
+```
+
+### Integrity Checking
+
+```rust
+use rustfs_rio::{ChecksumWriter, ChecksumReader, ChecksumAlgorithm};
+
+async fn integrity_checking_example() -> Result<(), Box<dyn std::error::Error>> {
+    // Create checksum writer
+    let mut checksum_writer = ChecksumWriter::new(
+        output_stream,
+        ChecksumAlgorithm::Crc32
+    );
+
+    // Write data with checksum calculation
+    checksum_writer.write_all(b"Data with integrity checking").await?;
+    let write_checksum = checksum_writer.finalize().await?;
+
+    println!("Write checksum: {:08x}", write_checksum);
+
+    // Create checksum reader
+    let mut checksum_reader = ChecksumReader::new(
+        input_stream,
+        ChecksumAlgorithm::Crc32
+    );
+
+    // Read data with checksum verification
+    let mut data = Vec::new();
+    checksum_reader.read_to_end(&mut data).await?;
+    let read_checksum = checksum_reader.checksum();
+
+    println!("Read checksum: {:08x}", read_checksum);
+
+    // Verify integrity
+    if write_checksum == read_checksum {
+        println!("Data integrity verified!");
+    } else {
+        println!("Data corruption detected!");
+    }
+
+    Ok(())
+}
+```
+
+### Multi-Layer Streaming
+
+```rust
+use rustfs_rio::{MultiLayerWriter, MultiLayerReader, Layer};
+
+async fn multi_layer_streaming_example() -> Result<(), Box<dyn std::error::Error>> {
+    // Create multi-layer writer (compression + encryption + checksum)
+    let mut writer = MultiLayerWriter::new(output_stream)
+        .add_layer(Layer::Compression(CompressionAlgorithm::Zstd, 6))
+        .add_layer(Layer::Encryption(encryption_key.clone()))
+        .add_layer(Layer::Checksum(ChecksumAlgorithm::Crc32))
+        .build()?;
+
+    // Write data through all layers
+    writer.write_all(b"This data will be compressed, encrypted, and checksummed").await?;
+    let final_checksum = writer.finalize().await?;
+
+    // Create multi-layer reader (reverse order)
+    let mut reader = MultiLayerReader::new(input_stream)
+        .add_layer(Layer::Checksum(ChecksumAlgorithm::Crc32))
+        .add_layer(Layer::Decryption(encryption_key.clone()))
+        .add_layer(Layer::Decompression(CompressionAlgorithm::Zstd))
+        .build()?;
+
+    // Read data through all layers
+    let mut data = Vec::new();
+    reader.read_to_end(&mut data).await?;
+
+    // Verify final checksum
+    if reader.verify_checksum(final_checksum)? {
+        println!("All layers verified successfully!");
+    }
+
+    Ok(())
+}
+```
+
+### Vectored I/O Operations
+
+```rust
+use rustfs_rio::{VectoredWriter, VectoredReader, IoVec};
+
+async fn vectored_io_example() -> Result<(), Box<dyn std::error::Error>> {
+    // Create vectored writer
+    let mut vectored_writer = VectoredWriter::new(output_stream);
+
+    // Prepare multiple buffers
+    let header = b"HEADER";
+    let data = b"Important data content";
+    let footer = b"FOOTER";
+
+    // Write multiple buffers in one operation
+    let io_vecs = vec![
+        IoVec::new(header),
+        IoVec::new(data),
+        IoVec::new(footer),
+    ];
+
+    let bytes_written = vectored_writer.write_vectored(&io_vecs).await?;
+    println!("Wrote {} bytes in vectored operation", bytes_written);
+
+    // Create vectored reader
+    let mut vectored_reader = VectoredReader::new(input_stream);
+
+    // Read into multiple buffers
+    let mut header_buf = vec![0u8; 6];
+    let mut data_buf = vec![0u8; 22];
+    let mut footer_buf = vec![0u8; 6];
+
+    let mut read_vecs = vec![
+        IoVec::new_mut(&mut header_buf),
+        IoVec::new_mut(&mut data_buf),
+        IoVec::new_mut(&mut footer_buf),
+    ];
+
+    let bytes_read = vectored_reader.read_vectored(&mut read_vecs).await?;
+    println!("Read {} bytes in vectored operation", bytes_read);
+
+    Ok(())
+}
+```
+
+### Async Stream Processing
+
+```rust
+use rustfs_rio::{AsyncStreamProcessor, ProcessorChain};
+use futures::StreamExt;
+
+async fn stream_processing_example() -> Result<(), Box<dyn std::error::Error>> {
+    // Create processor chain
+    let processor = ProcessorChain::new()
+        .add_processor(Box::new(CompressionProcessor::new(CompressionAlgorithm::Zstd)))
+        .add_processor(Box::new(EncryptionProcessor::new(encryption_key)))
+        .add_processor(Box::new(ChecksumProcessor::new(ChecksumAlgorithm::Crc32)));
+
+    // Create async stream processor
+    let mut stream_processor = AsyncStreamProcessor::new(input_stream, processor);
+
+    // Process stream chunks
+    while let Some(chunk) = stream_processor.next().await {
+        let processed_chunk = chunk?;
+
+        // Handle processed chunk
+        output_stream.write_all(&processed_chunk).await?;
+    }
+
+    Ok(())
+}
+```
+
+## 🏗️ Architecture
+
+### Rio Architecture
+
+```
+Rio I/O Architecture:
+┌─────────────────────────────────────────────────────────────┐
+│                    Stream API Layer                         │
+├─────────────────────────────────────────────────────────────┤
+│ Encryption  │ Compression │ Checksum    │ Vectored I/O     │
+├─────────────────────────────────────────────────────────────┤
+│              Buffer Management                               │
+├─────────────────────────────────────────────────────────────┤
+│   Zero-Copy   │   Async I/O   │   Backpressure Control     │
+├─────────────────────────────────────────────────────────────┤
+│              Tokio Runtime Integration                       │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### Performance Features
+
+| Feature | Benefit | Implementation |
+|---------|---------|----------------|
+| Zero-Copy | Reduced memory usage | Direct buffer operations |
+| Async I/O | High concurrency | Tokio-based operations |
+| Vectored I/O | Reduced syscalls | Scatter-gather operations |
+| Buffer Pooling | Memory efficiency | Reusable buffer management |
+
+## 🧪 Testing
+
+Run the test suite:
+
+```bash
+# Run all tests
+cargo test
+
+# Test streaming operations
+cargo test streaming
+
+# Test encryption
+cargo test encryption
+
+# Test compression
+cargo test compression
+
+# Run benchmarks
+cargo bench
+```
+
+## 📋 Requirements
+
+- **Rust**: 1.70.0 or later
+- **Platforms**: Linux, macOS, Windows
+- **Dependencies**: Tokio async runtime
+- **Hardware**: AES-NI support recommended for encryption
+
+## 🌍 Related Projects
+
+This module is part of the RustFS ecosystem:
+
+- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
+- [RustFS Utils](../utils) - Utility functions
+- [RustFS Crypto](../crypto) - Cryptographic operations
+
+## 📚 Documentation
+
+For comprehensive documentation, visit:
+
+- [RustFS Documentation](https://docs.rustfs.com)
+- [Rio API Reference](https://docs.rustfs.com/rio/)
+- [Performance Guide](https://docs.rustfs.com/performance/)
+
+## 🔗 Links
+
+- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
+- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
+- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
+
+## 🤝 Contributing
+
+We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+
+## 📄 License
+
+Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
+
+---
+
+<p align="center">
+  <strong>RustFS</strong> is a trademark of RustFS, Inc.<br>
+  All other trademarks are the property of their respective owners.
+</p>
+
+<p align="center">
+  Made with 🚀 by the RustFS Team
+</p>