mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-08 22:33:22 +00:00
安正超
408 lines
14 KiB
Markdown
408 lines
14 KiB
Markdown
[](https://rustfs.com)
|
|
|
|
# RustFS Zip - Compression & Archiving
|
|
|
|
<p align="center">
|
|
<strong>High-performance compression and archiving for RustFS object storage</strong>
|
|
</p>
|
|
|
|
<p align="center">
|
|
<a href="https://github.com/rustfs/rustfs/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rustfs/rustfs/actions/workflows/ci.yml/badge.svg" /></a>
|
|
<a href="https://docs.rustfs.com/en/">📖 Documentation</a>
|
|
· <a href="https://github.com/rustfs/rustfs/issues">🐛 Bug Reports</a>
|
|
· <a href="https://github.com/rustfs/rustfs/discussions">💬 Discussions</a>
|
|
</p>
|
|
|
|
---
|
|
|
|
## 📖 Overview
|
|
|
|
**RustFS Zip** provides high-performance compression and archiving capabilities for the [RustFS](https://rustfs.com) distributed object storage system. It supports multiple compression algorithms, streaming compression, and efficient archiving operations optimized for storage systems.
|
|
|
|
> **Note:** This is a performance-critical submodule of RustFS that provides essential compression capabilities for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
|
|
|
|
## ✨ Features
|
|
|
|
### 📦 Compression Algorithms
|
|
|
|
- **Zstandard (Zstd)**: Fast compression with excellent ratios
|
|
- **LZ4**: Ultra-fast compression for real-time applications
|
|
- **Gzip**: Industry-standard compression for compatibility
|
|
- **Brotli**: Web-optimized compression for text content
|
|
|
|
### 🚀 Performance Features
|
|
|
|
- **Streaming Compression**: Compress data on-the-fly without buffering
|
|
- **Parallel Processing**: Multi-threaded compression for large files
|
|
- **Adaptive Compression**: Automatic algorithm selection based on data
|
|
- **Hardware Acceleration**: Leverage CPU-specific optimizations
|
|
|
|
### 🔧 Archive Management
|
|
|
|
- **ZIP Format**: Standard ZIP archive creation and extraction
|
|
- **TAR Format**: UNIX-style tar archive support
|
|
- **Custom Formats**: RustFS-optimized archive formats
|
|
- **Metadata Preservation**: Maintain file attributes and timestamps
|
|
|
|
### 📊 Compression Analytics
|
|
|
|
- **Ratio Analysis**: Detailed compression statistics
|
|
- **Performance Metrics**: Compression speed and efficiency
|
|
- **Content-Type Detection**: Automatic compression algorithm selection
|
|
- **Deduplication**: Identify and handle duplicate content
|
|
|
|
## 📦 Installation
|
|
|
|
Add this to your `Cargo.toml`:
|
|
|
|
```toml
|
|
[dependencies]
|
|
rustfs-zip = "0.1.0"
|
|
```
|
|
|
|
## 🔧 Usage
|
|
|
|
### Basic Compression
|
|
|
|
```rust
|
|
use rustfs_zip::{Compressor, CompressionLevel, CompressionAlgorithm};
|
|
|
|
#[tokio::main]
|
|
async fn main() -> Result<(), Box<dyn std::error::Error>> {
|
|
// Create compressor
|
|
let compressor = Compressor::new(CompressionAlgorithm::Zstd, CompressionLevel::Default);
|
|
|
|
// Compress data
|
|
let input_data = b"Hello, World! This is some test data to compress.";
|
|
let compressed = compressor.compress(input_data).await?;
|
|
|
|
println!("Original size: {} bytes", input_data.len());
|
|
println!("Compressed size: {} bytes", compressed.len());
|
|
println!("Compression ratio: {:.2}%",
|
|
(1.0 - compressed.len() as f64 / input_data.len() as f64) * 100.0);
|
|
|
|
// Decompress data
|
|
let decompressed = compressor.decompress(&compressed).await?;
|
|
assert_eq!(input_data, decompressed.as_slice());
|
|
|
|
Ok(())
|
|
}
|
|
```
|
|
|
|
### Streaming Compression
|
|
|
|
```rust
|
|
use rustfs_zip::{StreamingCompressor, StreamingDecompressor};
|
|
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
|
|
|
async fn streaming_compression_example() -> Result<(), Box<dyn std::error::Error>> {
|
|
// Create streaming compressor
|
|
let mut compressor = StreamingCompressor::new(
|
|
CompressionAlgorithm::Zstd,
|
|
CompressionLevel::Fast,
|
|
)?;
|
|
|
|
// Compress streaming data
|
|
let input = tokio::fs::File::open("large_file.txt").await?;
|
|
let output = tokio::fs::File::create("compressed_file.zst").await?;
|
|
|
|
let mut reader = tokio::io::BufReader::new(input);
|
|
let mut writer = tokio::io::BufWriter::new(output);
|
|
|
|
// Stream compression
|
|
let mut buffer = vec![0u8; 8192];
|
|
loop {
|
|
let bytes_read = reader.read(&mut buffer).await?;
|
|
if bytes_read == 0 {
|
|
break;
|
|
}
|
|
|
|
let compressed_chunk = compressor.compress_chunk(&buffer[..bytes_read]).await?;
|
|
writer.write_all(&compressed_chunk).await?;
|
|
}
|
|
|
|
// Finalize compression
|
|
let final_chunk = compressor.finalize().await?;
|
|
writer.write_all(&final_chunk).await?;
|
|
writer.flush().await?;
|
|
|
|
Ok(())
|
|
}
|
|
```
|
|
|
|
### Archive Creation
|
|
|
|
```rust
|
|
use rustfs_zip::{ZipArchive, ArchiveBuilder, CompressionMethod};
|
|
|
|
async fn create_archive_example() -> Result<(), Box<dyn std::error::Error>> {
|
|
// Create archive builder
|
|
let mut builder = ArchiveBuilder::new("backup.zip".to_string());
|
|
|
|
// Add files to archive
|
|
builder.add_file("config.json", "config/app.json", CompressionMethod::Deflate).await?;
|
|
builder.add_file("data.txt", "data/sample.txt", CompressionMethod::Store).await?;
|
|
|
|
// Add directory
|
|
builder.add_directory("logs/", "application_logs/").await?;
|
|
|
|
// Create archive
|
|
let archive = builder.build().await?;
|
|
|
|
println!("Archive created: {}", archive.path());
|
|
println!("Total files: {}", archive.file_count());
|
|
println!("Compressed size: {} bytes", archive.compressed_size());
|
|
println!("Uncompressed size: {} bytes", archive.uncompressed_size());
|
|
|
|
Ok(())
|
|
}
|
|
```
|
|
|
|
### Archive Extraction
|
|
|
|
```rust
|
|
use rustfs_zip::{ZipExtractor, ExtractionOptions};
|
|
|
|
async fn extract_archive_example() -> Result<(), Box<dyn std::error::Error>> {
|
|
// Create extractor
|
|
let extractor = ZipExtractor::new("backup.zip".to_string());
|
|
|
|
// List archive contents
|
|
let entries = extractor.list_entries().await?;
|
|
for entry in &entries {
|
|
println!("File: {} ({} bytes)", entry.name, entry.size);
|
|
}
|
|
|
|
// Extract specific file
|
|
let file_data = extractor.extract_file("config.json").await?;
|
|
println!("Extracted config.json: {} bytes", file_data.len());
|
|
|
|
// Extract all files
|
|
let extraction_options = ExtractionOptions {
|
|
output_directory: "extracted/".to_string(),
|
|
preserve_paths: true,
|
|
overwrite_existing: false,
|
|
};
|
|
|
|
extractor.extract_all(extraction_options).await?;
|
|
println!("Archive extracted successfully");
|
|
|
|
Ok(())
|
|
}
|
|
```
|
|
|
|
### Adaptive Compression
|
|
|
|
```rust
|
|
use rustfs_zip::{AdaptiveCompressor, ContentAnalyzer, CompressionProfile};
|
|
|
|
async fn adaptive_compression_example() -> Result<(), Box<dyn std::error::Error>> {
|
|
// Create adaptive compressor
|
|
let mut compressor = AdaptiveCompressor::new();
|
|
|
|
// Configure compression profiles
|
|
compressor.add_profile(CompressionProfile {
|
|
content_type: "text/*".to_string(),
|
|
algorithm: CompressionAlgorithm::Brotli,
|
|
level: CompressionLevel::High,
|
|
min_size: 1024,
|
|
});
|
|
|
|
compressor.add_profile(CompressionProfile {
|
|
content_type: "image/*".to_string(),
|
|
algorithm: CompressionAlgorithm::Lz4,
|
|
level: CompressionLevel::Fast,
|
|
min_size: 10240,
|
|
});
|
|
|
|
// Compress different types of content
|
|
let text_content = std::fs::read("document.txt")?;
|
|
let image_content = std::fs::read("photo.jpg")?;
|
|
|
|
// Analyze and compress
|
|
let text_result = compressor.compress_adaptive(&text_content, Some("text/plain")).await?;
|
|
let image_result = compressor.compress_adaptive(&image_content, Some("image/jpeg")).await?;
|
|
|
|
println!("Text compression: {} -> {} bytes ({})",
|
|
text_content.len(), text_result.compressed_size, text_result.algorithm);
|
|
println!("Image compression: {} -> {} bytes ({})",
|
|
image_content.len(), image_result.compressed_size, image_result.algorithm);
|
|
|
|
Ok(())
|
|
}
|
|
```
|
|
|
|
### Parallel Compression
|
|
|
|
```rust
|
|
use rustfs_zip::{ParallelCompressor, CompressionJob};
|
|
|
|
async fn parallel_compression_example() -> Result<(), Box<dyn std::error::Error>> {
|
|
// Create parallel compressor
|
|
let compressor = ParallelCompressor::new(4); // 4 worker threads
|
|
|
|
// Prepare compression jobs
|
|
let jobs = vec![
|
|
CompressionJob {
|
|
id: "file1".to_string(),
|
|
data: std::fs::read("file1.txt")?,
|
|
algorithm: CompressionAlgorithm::Zstd,
|
|
level: CompressionLevel::Default,
|
|
},
|
|
CompressionJob {
|
|
id: "file2".to_string(),
|
|
data: std::fs::read("file2.txt")?,
|
|
algorithm: CompressionAlgorithm::Lz4,
|
|
level: CompressionLevel::Fast,
|
|
},
|
|
CompressionJob {
|
|
id: "file3".to_string(),
|
|
data: std::fs::read("file3.txt")?,
|
|
algorithm: CompressionAlgorithm::Gzip,
|
|
level: CompressionLevel::High,
|
|
},
|
|
];
|
|
|
|
// Execute parallel compression
|
|
let results = compressor.compress_batch(jobs).await?;
|
|
|
|
for result in results {
|
|
println!("Job {}: {} -> {} bytes",
|
|
result.job_id, result.original_size, result.compressed_size);
|
|
}
|
|
|
|
Ok(())
|
|
}
|
|
```
|
|
|
|
### Content Deduplication
|
|
|
|
```rust
|
|
use rustfs_zip::{DeduplicationCompressor, ContentHash};
|
|
|
|
async fn deduplication_example() -> Result<(), Box<dyn std::error::Error>> {
|
|
// Create deduplication compressor
|
|
let mut compressor = DeduplicationCompressor::new();
|
|
|
|
// Add files for compression
|
|
let file1 = std::fs::read("document1.txt")?;
|
|
let file2 = std::fs::read("document2.txt")?;
|
|
let file3 = std::fs::read("document1.txt")?; // Duplicate of file1
|
|
|
|
// Compress with deduplication
|
|
let result1 = compressor.compress_with_dedup("doc1", &file1).await?;
|
|
let result2 = compressor.compress_with_dedup("doc2", &file2).await?;
|
|
let result3 = compressor.compress_with_dedup("doc3", &file3).await?;
|
|
|
|
println!("File 1: {} bytes -> {} bytes", file1.len(), result1.compressed_size);
|
|
println!("File 2: {} bytes -> {} bytes", file2.len(), result2.compressed_size);
|
|
println!("File 3: {} bytes -> {} bytes (deduplicated: {})",
|
|
file3.len(), result3.compressed_size, result3.is_deduplicated);
|
|
|
|
// Get deduplication statistics
|
|
let stats = compressor.get_dedup_stats();
|
|
println!("Deduplication saved: {} bytes", stats.bytes_saved);
|
|
println!("Duplicate files found: {}", stats.duplicate_count);
|
|
|
|
Ok(())
|
|
}
|
|
```
|
|
|
|
## 🏗️ Architecture
|
|
|
|
### Zip Module Architecture
|
|
|
|
```
|
|
Zip Architecture:
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Compression API │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ Algorithm │ Streaming │ Archive │ Adaptive │
|
|
│ Selection │ Compression │ Management │ Compression│
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ Compression Engines │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ Zstd │ LZ4 │ Gzip │ Brotli │ Custom │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ Low-Level Compression │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Compression Algorithms
|
|
|
|
| Algorithm | Speed | Ratio | Use Case |
|
|
|-----------|-------|-------|----------|
|
|
| LZ4 | Very Fast | Good | Real-time compression |
|
|
| Zstd | Fast | Excellent | General purpose |
|
|
| Gzip | Medium | Good | Web compatibility |
|
|
| Brotli | Slow | Excellent | Text/web content |
|
|
|
|
## 🧪 Testing
|
|
|
|
Run the test suite:
|
|
|
|
```bash
|
|
# Run all tests
|
|
cargo test
|
|
|
|
# Test compression algorithms
|
|
cargo test algorithms
|
|
|
|
# Test streaming compression
|
|
cargo test streaming
|
|
|
|
# Test archive operations
|
|
cargo test archive
|
|
|
|
# Benchmark compression performance
|
|
cargo bench
|
|
```
|
|
|
|
## 📋 Requirements
|
|
|
|
- **Rust**: 1.70.0 or later
|
|
- **Platforms**: Linux, macOS, Windows
|
|
- **Dependencies**: Native compression libraries
|
|
- **Memory**: Sufficient RAM for compression buffers
|
|
|
|
## 🌍 Related Projects
|
|
|
|
This module is part of the RustFS ecosystem:
|
|
|
|
- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
|
|
- [RustFS Rio](../rio) - High-performance I/O
|
|
- [RustFS Utils](../utils) - Utility functions
|
|
|
|
## 📚 Documentation
|
|
|
|
For comprehensive documentation, visit:
|
|
|
|
- [RustFS Documentation](https://docs.rustfs.com)
|
|
- [Zip API Reference](https://docs.rustfs.com/zip/)
|
|
- [Compression Guide](https://docs.rustfs.com/compression/)
|
|
|
|
## 🔗 Links
|
|
|
|
- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
|
|
- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
|
|
- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
|
|
|
|
## 🤝 Contributing
|
|
|
|
We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
|
|
|
|
## 📄 License
|
|
|
|
Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
|
|
|
|
---
|
|
|
|
<p align="center">
|
|
<strong>RustFS</strong> is a trademark of RustFS, Inc.<br>
|
|
All other trademarks are the property of their respective owners.
|
|
</p>
|
|
|
|
<p align="center">
|
|
Made with 📦 by the RustFS Team
|
|
</p>
|