gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-30 16:59:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #477 from rustfs/docker-images

Browse Source 
feat: Add comprehensive Docker build pipeline for multi-architecture images.
This commit is contained in:
安正超
2025-06-17 23:55:43 +08:00
committed by GitHub
parent 65ff5faada 0f7a98a91f
commit 81a790f13f
14 changed files with 1352 additions and 1279 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.cursorrules
+90
View File
@@ -3,6 +3,7 @@
## ⚠️ CRITICAL DEVELOPMENT RULES ⚠️ ## ⚠️ CRITICAL DEVELOPMENT RULES ⚠️
### 🚨 NEVER COMMIT DIRECTLY TO MASTER/MAIN BRANCH 🚨 ### 🚨 NEVER COMMIT DIRECTLY TO MASTER/MAIN BRANCH 🚨
- **This is the most important rule - NEVER modify code directly on main or master branch** - **This is the most important rule - NEVER modify code directly on main or master branch**
- **Always work on feature branches and use pull requests for all changes** - **Always work on feature branches and use pull requests for all changes**
- **Any direct commits to master/main branch are strictly forbidden** - **Any direct commits to master/main branch are strictly forbidden**
@@ -15,23 +16,27 @@
6. Create a pull request for review 6. Create a pull request for review
## Project Overview ## Project Overview
RustFS is a high-performance distributed object storage system written in Rust, compatible with S3 API. The project adopts a modular architecture, supporting erasure coding storage, multi-tenant management, observability, and other enterprise-level features. RustFS is a high-performance distributed object storage system written in Rust, compatible with S3 API. The project adopts a modular architecture, supporting erasure coding storage, multi-tenant management, observability, and other enterprise-level features.
## Core Architecture Principles ## Core Architecture Principles
### 1. Modular Design ### 1. Modular Design
- Project uses Cargo workspace structure, containing multiple independent crates - Project uses Cargo workspace structure, containing multiple independent crates
- Core modules: `rustfs` (main service), `ecstore` (erasure coding storage), `common` (shared components) - Core modules: `rustfs` (main service), `ecstore` (erasure coding storage), `common` (shared components)
- Functional modules: `iam` (identity management), `madmin` (management interface), `crypto` (encryption), etc. - Functional modules: `iam` (identity management), `madmin` (management interface), `crypto` (encryption), etc.
- Tool modules: `cli` (command line tool), `crates/*` (utility libraries) - Tool modules: `cli` (command line tool), `crates/*` (utility libraries)
### 2. Asynchronous Programming Pattern ### 2. Asynchronous Programming Pattern
- Comprehensive use of `tokio` async runtime - Comprehensive use of `tokio` async runtime
- Prioritize `async/await` syntax - Prioritize `async/await` syntax
- Use `async-trait` for async methods in traits - Use `async-trait` for async methods in traits
- Avoid blocking operations, use `spawn_blocking` when necessary - Avoid blocking operations, use `spawn_blocking` when necessary
### 3. Error Handling Strategy ### 3. Error Handling Strategy
- **Use modular, type-safe error handling with `thiserror`** - **Use modular, type-safe error handling with `thiserror`**
- Each module should define its own error type using `thiserror::Error` derive macro - Each module should define its own error type using `thiserror::Error` derive macro
- Support error chains and context information through `#[from]` and `#[source]` attributes - Support error chains and context information through `#[from]` and `#[source]` attributes
@@ -54,6 +59,7 @@ RustFS is a high-performance distributed object storage system written in Rust,
## Code Style Guidelines ## Code Style Guidelines
### 1. Formatting Configuration ### 1. Formatting Configuration
```toml ```toml
max_width = 130 max_width = 130
fn_call_width = 90 fn_call_width = 90
@@ -69,21 +75,25 @@ single_line_let_else_max_width = 100
Before every commit, you **MUST**: Before every commit, you **MUST**:
1. **Format your code**: 1. **Format your code**:
```bash ```bash
cargo fmt --all cargo fmt --all
``` ```
2. **Verify formatting**: 2. **Verify formatting**:
```bash ```bash
cargo fmt --all --check cargo fmt --all --check
``` ```
3. **Pass clippy checks**: 3. **Pass clippy checks**:
```bash ```bash
cargo clippy --all-targets --all-features -- -D warnings cargo clippy --all-targets --all-features -- -D warnings
``` ```
4. **Ensure compilation**: 4. **Ensure compilation**:
```bash ```bash
cargo check --all-targets cargo check --all-targets
``` ```
@@ -158,6 +168,7 @@ Example output when formatting fails:
``` ```
### 3. Naming Conventions ### 3. Naming Conventions
- Use `snake_case` for functions, variables, modules - Use `snake_case` for functions, variables, modules
- Use `PascalCase` for types, traits, enums - Use `PascalCase` for types, traits, enums
- Constants use `SCREAMING_SNAKE_CASE` - Constants use `SCREAMING_SNAKE_CASE`
@@ -167,6 +178,7 @@ Example output when formatting fails:
- Choose names that clearly express the purpose and intent - Choose names that clearly express the purpose and intent
### 4. Type Declaration Guidelines ### 4. Type Declaration Guidelines
- **Prefer type inference over explicit type declarations** when the type is obvious from context - **Prefer type inference over explicit type declarations** when the type is obvious from context
- Let the Rust compiler infer types whenever possible to reduce verbosity and improve maintainability - Let the Rust compiler infer types whenever possible to reduce verbosity and improve maintainability
- Only specify types explicitly when: - Only specify types explicitly when:
@@ -176,6 +188,7 @@ Example output when formatting fails:
- Needed to resolve ambiguity between multiple possible types - Needed to resolve ambiguity between multiple possible types
**Good examples (prefer these):** **Good examples (prefer these):**
```rust ```rust
// Compiler can infer the type // Compiler can infer the type
let items = vec![1, 2, 3, 4]; let items = vec![1, 2, 3, 4];
@@ -187,6 +200,7 @@ let filtered: Vec<_> = items.iter().filter(|&&x| x > 2).collect();
``` ```
**Avoid unnecessary explicit types:** **Avoid unnecessary explicit types:**
```rust ```rust
// Unnecessary - type is obvious // Unnecessary - type is obvious
let items: Vec<i32> = vec![1, 2, 3, 4]; let items: Vec<i32> = vec![1, 2, 3, 4];
@@ -195,6 +209,7 @@ let result: ProcessResult = process_data(&input);
``` ```
**When explicit types are beneficial:** **When explicit types are beneficial:**
```rust ```rust
// API boundaries - always specify types // API boundaries - always specify types
pub fn process_data(input: &[u8]) -> Result<ProcessResult, Error> { ... } pub fn process_data(input: &[u8]) -> Result<ProcessResult, Error> { ... }
@@ -207,6 +222,7 @@ let cache: HashMap<String, Arc<Mutex<CacheEntry>>> = HashMap::new();
``` ```
### 5. Documentation Comments ### 5. Documentation Comments
- Public APIs must have documentation comments - Public APIs must have documentation comments
- Use `///` for documentation comments - Use `///` for documentation comments
- Complex functions add `# Examples` and `# Parameters` descriptions - Complex functions add `# Examples` and `# Parameters` descriptions
@@ -215,6 +231,7 @@ let cache: HashMap<String, Arc<Mutex<CacheEntry>>> = HashMap::new();
- Avoid meaningless comments like "debug 111" or placeholder text - Avoid meaningless comments like "debug 111" or placeholder text
### 6. Import Guidelines ### 6. Import Guidelines
- Standard library imports first - Standard library imports first
- Third-party crate imports in the middle - Third-party crate imports in the middle
- Project internal imports last - Project internal imports last
@@ -223,6 +240,7 @@ let cache: HashMap<String, Arc<Mutex<CacheEntry>>> = HashMap::new();
## Asynchronous Programming Guidelines ## Asynchronous Programming Guidelines
### 1. Trait Definition ### 1. Trait Definition
```rust ```rust
#[async_trait::async_trait] #[async_trait::async_trait]
pub trait StorageAPI: Send + Sync { pub trait StorageAPI: Send + Sync {
@@ -231,6 +249,7 @@ pub trait StorageAPI: Send + Sync {
``` ```
### 2. Error Handling ### 2. Error Handling
```rust ```rust
// Use ? operator to propagate errors // Use ? operator to propagate errors
async fn example_function() -> Result<()> { async fn example_function() -> Result<()> {
@@ -241,6 +260,7 @@ async fn example_function() -> Result<()> {
``` ```
### 3. Concurrency Control ### 3. Concurrency Control
- Use `Arc` and `Mutex`/`RwLock` for shared state management - Use `Arc` and `Mutex`/`RwLock` for shared state management
- Prioritize async locks from `tokio::sync` - Prioritize async locks from `tokio::sync`
- Avoid holding locks for long periods - Avoid holding locks for long periods
@@ -248,6 +268,7 @@ async fn example_function() -> Result<()> {
## Logging and Tracing Guidelines ## Logging and Tracing Guidelines
### 1. Tracing Usage ### 1. Tracing Usage
```rust ```rust
#[tracing::instrument(skip(self, data))] #[tracing::instrument(skip(self, data))]
async fn process_data(&self, data: &[u8]) -> Result<()> { async fn process_data(&self, data: &[u8]) -> Result<()> {
@@ -257,6 +278,7 @@ async fn process_data(&self, data: &[u8]) -> Result<()> {
``` ```
### 2. Log Levels ### 2. Log Levels
- `error!`: System errors requiring immediate attention - `error!`: System errors requiring immediate attention
- `warn!`: Warning information that may affect functionality - `warn!`: Warning information that may affect functionality
- `info!`: Important business information - `info!`: Important business information
@@ -264,6 +286,7 @@ async fn process_data(&self, data: &[u8]) -> Result<()> {
- `trace!`: Detailed execution paths - `trace!`: Detailed execution paths
### 3. Structured Logging ### 3. Structured Logging
```rust ```rust
info!( info!(
counter.rustfs_api_requests_total = 1_u64, counter.rustfs_api_requests_total = 1_u64,
@@ -276,6 +299,7 @@ info!(
## Error Handling Guidelines ## Error Handling Guidelines
### 1. Error Type Definition ### 1. Error Type Definition
```rust ```rust
// Use thiserror for module-specific error types // Use thiserror for module-specific error types
#[derive(thiserror::Error, Debug)] #[derive(thiserror::Error, Debug)]
@@ -301,6 +325,7 @@ pub type Result<T> = core::result::Result<T, MyError>;
``` ```
### 2. Error Helper Methods ### 2. Error Helper Methods
```rust ```rust
impl MyError { impl MyError {
/// Create error from any compatible error type /// Create error from any compatible error type
@@ -314,6 +339,7 @@ impl MyError {
``` ```
### 3. Error Conversion Between Modules ### 3. Error Conversion Between Modules
```rust ```rust
// Convert between different module error types // Convert between different module error types
impl From<ecstore::error::StorageError> for MyError { impl From<ecstore::error::StorageError> for MyError {
@@ -340,6 +366,7 @@ impl From<MyError> for ecstore::error::StorageError {
``` ```
### 4. Error Context and Propagation ### 4. Error Context and Propagation
```rust ```rust
// Use ? operator for clean error propagation // Use ? operator for clean error propagation
async fn example_function() -> Result<()> { async fn example_function() -> Result<()> {
@@ -359,6 +386,7 @@ fn process_with_context(path: &str) -> Result<()> {
``` ```
### 5. API Error Conversion (S3 Example) ### 5. API Error Conversion (S3 Example)
```rust ```rust
// Convert storage errors to API-specific errors // Convert storage errors to API-specific errors
use s3s::{S3Error, S3ErrorCode}; use s3s::{S3Error, S3ErrorCode};
@@ -404,6 +432,7 @@ impl From<ApiError> for S3Error {
### 6. Error Handling Best Practices ### 6. Error Handling Best Practices
#### Pattern Matching and Error Classification #### Pattern Matching and Error Classification
```rust ```rust
// Use pattern matching for specific error handling // Use pattern matching for specific error handling
async fn handle_storage_operation() -> Result<()> { async fn handle_storage_operation() -> Result<()> {
@@ -428,6 +457,7 @@ async fn handle_storage_operation() -> Result<()> {
``` ```
#### Error Aggregation and Reporting #### Error Aggregation and Reporting
```rust ```rust
// Collect and report multiple errors // Collect and report multiple errors
pub fn validate_configuration(config: &Config) -> Result<()> { pub fn validate_configuration(config: &Config) -> Result<()> {
@@ -452,6 +482,7 @@ pub fn validate_configuration(config: &Config) -> Result<()> {
``` ```
#### Contextual Error Information #### Contextual Error Information
```rust ```rust
// Add operation context to errors // Add operation context to errors
#[tracing::instrument(skip(self))] #[tracing::instrument(skip(self))]
@@ -468,11 +499,13 @@ async fn upload_file(&self, bucket: &str, key: &str, data: Vec<u8>) -> Result<()
## Performance Optimization Guidelines ## Performance Optimization Guidelines
### 1. Memory Management ### 1. Memory Management
- Use `Bytes` instead of `Vec<u8>` for zero-copy operations - Use `Bytes` instead of `Vec<u8>` for zero-copy operations
- Avoid unnecessary cloning, use reference passing - Avoid unnecessary cloning, use reference passing
- Use `Arc` for sharing large objects - Use `Arc` for sharing large objects
### 2. Concurrency Optimization ### 2. Concurrency Optimization
```rust ```rust
// Use join_all for concurrent operations // Use join_all for concurrent operations
let futures = disks.iter().map(|disk| disk.operation()); let futures = disks.iter().map(|disk| disk.operation());
@@ -480,12 +513,14 @@ let results = join_all(futures).await;
``` ```
### 3. Caching Strategy ### 3. Caching Strategy
- Use `lazy_static` or `OnceCell` for global caching - Use `lazy_static` or `OnceCell` for global caching
- Implement LRU cache to avoid memory leaks - Implement LRU cache to avoid memory leaks
## Testing Guidelines ## Testing Guidelines
### 1. Unit Tests ### 1. Unit Tests
```rust ```rust
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
@@ -546,10 +581,12 @@ mod tests {
``` ```
### 2. Integration Tests ### 2. Integration Tests
- Use `e2e_test` module for end-to-end testing - Use `e2e_test` module for end-to-end testing
- Simulate real storage environments - Simulate real storage environments
### 3. Test Quality Standards ### 3. Test Quality Standards
- Write meaningful test cases that verify actual functionality - Write meaningful test cases that verify actual functionality
- Avoid placeholder or debug content like "debug 111", "test test", etc. - Avoid placeholder or debug content like "debug 111", "test test", etc.
- Use descriptive test names that clearly indicate what is being tested - Use descriptive test names that clearly indicate what is being tested
@@ -559,9 +596,11 @@ mod tests {
## Cross-Platform Compatibility Guidelines ## Cross-Platform Compatibility Guidelines
### 1. CPU Architecture Compatibility ### 1. CPU Architecture Compatibility
- **Always consider multi-platform and different CPU architecture compatibility** when writing code - **Always consider multi-platform and different CPU architecture compatibility** when writing code
- Support major architectures: x86_64, aarch64 (ARM64), and other target platforms - Support major architectures: x86_64, aarch64 (ARM64), and other target platforms
- Use conditional compilation for architecture-specific code: - Use conditional compilation for architecture-specific code:
```rust ```rust
#[cfg(target_arch = "x86_64")] #[cfg(target_arch = "x86_64")]
fn optimized_x86_64_function() { /* x86_64 specific implementation */ } fn optimized_x86_64_function() { /* x86_64 specific implementation */ }
@@ -574,16 +613,19 @@ fn generic_function() { /* Generic fallback implementation */ }
``` ```
### 2. Platform-Specific Dependencies ### 2. Platform-Specific Dependencies
- Use feature flags for platform-specific dependencies - Use feature flags for platform-specific dependencies
- Provide fallback implementations for unsupported platforms - Provide fallback implementations for unsupported platforms
- Test on multiple architectures in CI/CD pipeline - Test on multiple architectures in CI/CD pipeline
### 3. Endianness Considerations ### 3. Endianness Considerations
- Use explicit byte order conversion when dealing with binary data - Use explicit byte order conversion when dealing with binary data
- Prefer `to_le_bytes()`, `from_le_bytes()` for consistent little-endian format - Prefer `to_le_bytes()`, `from_le_bytes()` for consistent little-endian format
- Use `byteorder` crate for complex binary format handling - Use `byteorder` crate for complex binary format handling
### 4. SIMD and Performance Optimizations ### 4. SIMD and Performance Optimizations
- Use portable SIMD libraries like `wide` or `packed_simd` - Use portable SIMD libraries like `wide` or `packed_simd`
- Provide fallback implementations for non-SIMD architectures - Provide fallback implementations for non-SIMD architectures
- Use runtime feature detection when appropriate - Use runtime feature detection when appropriate
@@ -591,10 +633,12 @@ fn generic_function() { /* Generic fallback implementation */ }
## Security Guidelines ## Security Guidelines
### 1. Memory Safety ### 1. Memory Safety
- Disable `unsafe` code (workspace.lints.rust.unsafe_code = "deny") - Disable `unsafe` code (workspace.lints.rust.unsafe_code = "deny")
- Use `rustls` instead of `openssl` - Use `rustls` instead of `openssl`
### 2. Authentication and Authorization ### 2. Authentication and Authorization
```rust ```rust
// Use IAM system for permission checks // Use IAM system for permission checks
let identity = iam.authenticate(&access_key, &secret_key).await?; let identity = iam.authenticate(&access_key, &secret_key).await?;
@@ -604,11 +648,13 @@ iam.authorize(&identity, &action, &resource).await?;
## Configuration Management Guidelines ## Configuration Management Guidelines
### 1. Environment Variables ### 1. Environment Variables
- Use `RUSTFS_` prefix - Use `RUSTFS_` prefix
- Support both configuration files and environment variables - Support both configuration files and environment variables
- Provide reasonable default values - Provide reasonable default values
### 2. Configuration Structure ### 2. Configuration Structure
```rust ```rust
#[derive(Debug, Deserialize, Clone)] #[derive(Debug, Deserialize, Clone)]
pub struct Config { pub struct Config {
@@ -622,10 +668,12 @@ pub struct Config {
## Dependency Management Guidelines ## Dependency Management Guidelines
### 1. Workspace Dependencies ### 1. Workspace Dependencies
- Manage versions uniformly at workspace level - Manage versions uniformly at workspace level
- Use `workspace = true` to inherit configuration - Use `workspace = true` to inherit configuration
### 2. Feature Flags ### 2. Feature Flags
```rust ```rust
[features] [features]
default = ["file"] default = ["file"]
@@ -636,15 +684,18 @@ kafka = ["dep:rdkafka"]
## Deployment and Operations Guidelines ## Deployment and Operations Guidelines
### 1. Containerization ### 1. Containerization
- Provide Dockerfile and docker-compose configuration - Provide Dockerfile and docker-compose configuration
- Support multi-stage builds to optimize image size - Support multi-stage builds to optimize image size
### 2. Observability ### 2. Observability
- Integrate OpenTelemetry for distributed tracing - Integrate OpenTelemetry for distributed tracing
- Support Prometheus metrics collection - Support Prometheus metrics collection
- Provide Grafana dashboards - Provide Grafana dashboards
### 3. Health Checks ### 3. Health Checks
```rust ```rust
// Implement health check endpoint // Implement health check endpoint
async fn health_check() -> Result<HealthStatus> { async fn health_check() -> Result<HealthStatus> {
@@ -655,6 +706,7 @@ async fn health_check() -> Result<HealthStatus> {
## Code Review Checklist ## Code Review Checklist
### 1. **Code Formatting and Quality (MANDATORY)** ### 1. **Code Formatting and Quality (MANDATORY)**
- [ ] **Code is properly formatted** (`cargo fmt --all --check` passes) - [ ] **Code is properly formatted** (`cargo fmt --all --check` passes)
- [ ] **All clippy warnings are resolved** (`cargo clippy --all-targets --all-features -- -D warnings` passes) - [ ] **All clippy warnings are resolved** (`cargo clippy --all-targets --all-features -- -D warnings` passes)
- [ ] **Code compiles successfully** (`cargo check --all-targets` passes) - [ ] **Code compiles successfully** (`cargo check --all-targets` passes)
@@ -662,27 +714,32 @@ async fn health_check() -> Result<HealthStatus> {
- [ ] **No formatting-related changes** mixed with functional changes (separate commits) - [ ] **No formatting-related changes** mixed with functional changes (separate commits)
### 2. Functionality ### 2. Functionality
- [ ] Are all error cases properly handled? - [ ] Are all error cases properly handled?
- [ ] Is there appropriate logging? - [ ] Is there appropriate logging?
- [ ] Is there necessary test coverage? - [ ] Is there necessary test coverage?
### 3. Performance ### 3. Performance
- [ ] Are unnecessary memory allocations avoided? - [ ] Are unnecessary memory allocations avoided?
- [ ] Are async operations used correctly? - [ ] Are async operations used correctly?
- [ ] Are there potential deadlock risks? - [ ] Are there potential deadlock risks?
### 4. Security ### 4. Security
- [ ] Are input parameters properly validated? - [ ] Are input parameters properly validated?
- [ ] Are there appropriate permission checks? - [ ] Are there appropriate permission checks?
- [ ] Is information leakage avoided? - [ ] Is information leakage avoided?
### 5. Cross-Platform Compatibility ### 5. Cross-Platform Compatibility
- [ ] Does the code work on different CPU architectures (x86_64, aarch64)? - [ ] Does the code work on different CPU architectures (x86_64, aarch64)?
- [ ] Are platform-specific features properly gated with conditional compilation? - [ ] Are platform-specific features properly gated with conditional compilation?
- [ ] Is byte order handling correct for binary data? - [ ] Is byte order handling correct for binary data?
- [ ] Are there appropriate fallback implementations for unsupported platforms? - [ ] Are there appropriate fallback implementations for unsupported platforms?
### 6. Code Commits and Documentation ### 6. Code Commits and Documentation
- [ ] Does it comply with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)? - [ ] Does it comply with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)?
- [ ] Are commit messages concise and under 72 characters for the title line? - [ ] Are commit messages concise and under 72 characters for the title line?
- [ ] Commit titles should be concise and in English, avoid Chinese - [ ] Commit titles should be concise and in English, avoid Chinese
@@ -691,6 +748,7 @@ async fn health_check() -> Result<HealthStatus> {
## Common Patterns and Best Practices ## Common Patterns and Best Practices
### 1. Resource Management ### 1. Resource Management
```rust ```rust
// Use RAII pattern for resource management // Use RAII pattern for resource management
pub struct ResourceGuard { pub struct ResourceGuard {
@@ -705,6 +763,7 @@ impl Drop for ResourceGuard {
``` ```
### 2. Dependency Injection ### 2. Dependency Injection
```rust ```rust
// Use dependency injection pattern // Use dependency injection pattern
pub struct Service { pub struct Service {
@@ -714,6 +773,7 @@ pub struct Service {
``` ```
### 3. Graceful Shutdown ### 3. Graceful Shutdown
```rust ```rust
// Implement graceful shutdown // Implement graceful shutdown
async fn shutdown_gracefully(shutdown_rx: &mut Receiver<()>) { async fn shutdown_gracefully(shutdown_rx: &mut Receiver<()>) {
@@ -732,16 +792,19 @@ async fn shutdown_gracefully(shutdown_rx: &mut Receiver<()>) {
## Domain-Specific Guidelines ## Domain-Specific Guidelines
### 1. Storage Operations ### 1. Storage Operations
- All storage operations must support erasure coding - All storage operations must support erasure coding
- Implement read/write quorum mechanisms - Implement read/write quorum mechanisms
- Support data integrity verification - Support data integrity verification
### 2. Network Communication ### 2. Network Communication
- Use gRPC for internal service communication - Use gRPC for internal service communication
- HTTP/HTTPS support for S3-compatible API - HTTP/HTTPS support for S3-compatible API
- Implement connection pooling and retry mechanisms - Implement connection pooling and retry mechanisms
### 3. Metadata Management ### 3. Metadata Management
- Use FlatBuffers for serialization - Use FlatBuffers for serialization
- Support version control and migration - Support version control and migration
- Implement metadata caching - Implement metadata caching
@@ -751,6 +814,7 @@ These rules should serve as guiding principles when developing the RustFS projec
### 4. Code Operations ### 4. Code Operations
#### Branch Management #### Branch Management
- **🚨 CRITICAL: NEVER modify code directly on main or master branch - THIS IS ABSOLUTELY FORBIDDEN 🚨** - **🚨 CRITICAL: NEVER modify code directly on main or master branch - THIS IS ABSOLUTELY FORBIDDEN 🚨**
- **⚠️ ANY DIRECT COMMITS TO MASTER/MAIN WILL BE REJECTED AND MUST BE REVERTED IMMEDIATELY ⚠️** - **⚠️ ANY DIRECT COMMITS TO MASTER/MAIN WILL BE REJECTED AND MUST BE REVERTED IMMEDIATELY ⚠️**
- **Always work on feature branches - NO EXCEPTIONS** - **Always work on feature branches - NO EXCEPTIONS**
@@ -768,6 +832,7 @@ These rules should serve as guiding principles when developing the RustFS projec
- Ensure all changes are made on feature branches and merged through pull requests - Ensure all changes are made on feature branches and merged through pull requests
#### Development Workflow #### Development Workflow
- Use English for all code comments, documentation, and variable names - Use English for all code comments, documentation, and variable names
- Write meaningful and descriptive names for variables, functions, and methods - Write meaningful and descriptive names for variables, functions, and methods
- Avoid meaningless test content like "debug 111" or placeholder values - Avoid meaningless test content like "debug 111" or placeholder values
@@ -788,3 +853,28 @@ These rules should serve as guiding principles when developing the RustFS projec
- Any breaking changes or migration notes if applicable - Any breaking changes or migration notes if applicable
- Testing information and verification steps - Testing information and verification steps
- **Provide PR descriptions in copyable markdown format** enclosed in code blocks for easy one-click copying - **Provide PR descriptions in copyable markdown format** enclosed in code blocks for easy one-click copying
## 🚫 AI 文档生成限制
### 禁止生成总结文档
- **严格禁止创建任何形式的AI生成总结文档**
- **不得创建包含大量表情符号、详细格式化表格和典型AI风格的文档**
- **不得在项目中生成以下类型的文档：**
- 基准测试总结文档（BENCHMARK*.md）
- 实现对比分析文档（IMPLEMENTATION_COMPARISON*.md）
- 性能分析报告文档
- 架构总结文档
- 功能对比文档
- 任何带有大量表情符号和格式化内容的文档
- **如果需要文档，请只在用户明确要求时创建，并保持简洁实用的风格**
- **文档应当专注于实际需要的信息，避免过度格式化和装饰性内容**
- **任何发现的AI生成总结文档都应该立即删除**
### 允许的文档类型
- README.md（项目介绍，保持简洁）
- 技术文档（仅在明确需要时创建）
- 用户手册（仅在明确需要时创建）
- API文档（从代码生成）
- 变更日志（CHANGELOG.md）
.docker/Dockerfile.devenv
+2 -5
View File
@@ -1,4 +1,4 @@
FROM m.daocloud.io/docker.io/library/ubuntu:22.04 FROM ubuntu:22.04
ENV LANG C.UTF-8 ENV LANG C.UTF-8
@@ -18,10 +18,7 @@ RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.
&& mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc && rm -rf Linux.flatc.binary.g++-13.zip && mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc && rm -rf Linux.flatc.binary.g++-13.zip
# install rust # install rust
ENV RUSTUP_DIST_SERVER="https://rsproxy.cn" RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV RUSTUP_UPDATE_ROOT="https://rsproxy.cn/rustup"
RUN curl -o rustup-init.sh --proto '=https' --tlsv1.2 -sSf https://rsproxy.cn/rustup-init.sh \
&& sh rustup-init.sh -y && rm -rf rustup-init.sh
COPY .docker/cargo.config.toml /root/.cargo/config.toml COPY .docker/cargo.config.toml /root/.cargo/config.toml
.docker/Dockerfile.rockylinux9.3
+2 -5
View File
@@ -1,4 +1,4 @@
FROM m.daocloud.io/docker.io/library/rockylinux:9.3 AS builder FROM rockylinux:9.3 AS builder
ENV LANG C.UTF-8 ENV LANG C.UTF-8
@@ -25,10 +25,7 @@ RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.
&& rm -rf Linux.flatc.binary.g++-13.zip && rm -rf Linux.flatc.binary.g++-13.zip
# install rust # install rust
ENV RUSTUP_DIST_SERVER="https://rsproxy.cn" RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV RUSTUP_UPDATE_ROOT="https://rsproxy.cn/rustup"
RUN curl -o rustup-init.sh --proto '=https' --tlsv1.2 -sSf https://rsproxy.cn/rustup-init.sh \
&& sh rustup-init.sh -y && rm -rf rustup-init.sh
COPY .docker/cargo.config.toml /root/.cargo/config.toml COPY .docker/cargo.config.toml /root/.cargo/config.toml
.docker/Dockerfile.ubuntu22.04
+2 -5
View File
@@ -1,4 +1,4 @@
FROM m.daocloud.io/docker.io/library/ubuntu:22.04 FROM ubuntu:22.04
ENV LANG C.UTF-8 ENV LANG C.UTF-8
@@ -18,10 +18,7 @@ RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.
&& mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc && rm -rf Linux.flatc.binary.g++-13.zip && mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc && rm -rf Linux.flatc.binary.g++-13.zip
# install rust # install rust
ENV RUSTUP_DIST_SERVER="https://rsproxy.cn" RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV RUSTUP_UPDATE_ROOT="https://rsproxy.cn/rustup"
RUN curl -o rustup-init.sh --proto '=https' --tlsv1.2 -sSf https://rsproxy.cn/rustup-init.sh \
&& sh rustup-init.sh -y && rm -rf rustup-init.sh
COPY .docker/cargo.config.toml /root/.cargo/config.toml COPY .docker/cargo.config.toml /root/.cargo/config.toml
.docker/cargo.config.toml
-8
View File
@@ -1,13 +1,5 @@
[source.crates-io] [source.crates-io]
registry = "https://github.com/rust-lang/crates.io-index" registry = "https://github.com/rust-lang/crates.io-index"
replace-with = 'rsproxy-sparse'
[source.rsproxy]
registry = "https://rsproxy.cn/crates.io-index"
[registries.rsproxy]
index = "https://rsproxy.cn/crates.io-index"
[source.rsproxy-sparse]
registry = "sparse+https://rsproxy.cn/index/"
[net] [net]
git-fetch-with-cli = true git-fetch-with-cli = true
.github/workflows/docker.yml
+314
View File
@@ -0,0 +1,314 @@
name: Build and Push Docker Images
on:
push:
branches:
- main
tags:
- 'v*'
pull_request:
branches:
- main
workflow_dispatch:
inputs:
push_to_registry:
description: 'Push images to registry'
required: false
default: 'true'
type: boolean
env:
REGISTRY_IMAGE_DOCKERHUB: rustfs/rustfs
REGISTRY_IMAGE_GHCR: ghcr.io/${{ github.repository }}
jobs:
# Skip duplicate job runs
skip-check:
permissions:
actions: write
contents: read
runs-on: ubuntu-latest
outputs:
should_skip: ${{ steps.skip_check.outputs.should_skip }}
steps:
- id: skip_check
uses: fkirc/skip-duplicate-actions@v5
with:
concurrent_skipping: 'same_content_newer'
cancel_others: true
paths_ignore: '["*.md", "docs/**"]'
# Build RustFS binary for different platforms
build-binary:
needs: skip-check
if: needs.skip-check.outputs.should_skip != 'true'
strategy:
matrix:
include:
- target: x86_64-unknown-linux-musl
os: ubuntu-latest
arch: amd64
use_cross: false
- target: aarch64-unknown-linux-gnu
os: ubuntu-latest
arch: arm64
use_cross: true
runs-on: ${{ matrix.os }}
timeout-minutes: 120
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Setup Rust toolchain
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
target: ${{ matrix.target }}
components: rustfmt, clippy
- name: Install cross-compilation dependencies (native build)
if: matrix.use_cross == false
run: |
sudo apt-get update
sudo apt-get install -y musl-tools
- name: Install cross tool (cross compilation)
if: matrix.use_cross == true
uses: taiki-e/install-action@v2
with:
tool: cross
- name: Install protoc
uses: arduino/setup-protoc@v3
with:
version: "31.1"
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Install flatc
uses: Nugine/setup-flatc@v1
with:
version: "25.2.10"
- name: Cache cargo dependencies
uses: actions/cache@v3
with:
path: |
~/.cargo/registry
~/.cargo/git
target
key: ${{ runner.os }}-cargo-${{ matrix.target }}-${{ hashFiles('**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-${{ matrix.target }}-
${{ runner.os }}-cargo-
- name: Generate protobuf code
run: cargo run --bin gproto
- name: Build RustFS binary (native)
if: matrix.use_cross == false
run: |
cargo build --release --target ${{ matrix.target }} --bin rustfs
- name: Build RustFS binary (cross)
if: matrix.use_cross == true
run: |
cross build --release --target ${{ matrix.target }} --bin rustfs
- name: Upload binary artifact
uses: actions/upload-artifact@v4
with:
name: rustfs-${{ matrix.arch }}
path: target/${{ matrix.target }}/release/rustfs
retention-days: 1
# Build and push Docker images
build-images:
needs: [skip-check, build-binary]
if: needs.skip-check.outputs.should_skip != 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
strategy:
matrix:
image-type: [production, ubuntu, rockylinux, devenv]
platform: [linux/amd64, linux/arm64]
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Download binary artifacts
uses: actions/download-artifact@v4
with:
path: ./artifacts
- name: Setup binary files
run: |
mkdir -p target/x86_64-unknown-linux-musl/release
mkdir -p target/aarch64-unknown-linux-gnu/release
cp artifacts/rustfs-amd64/rustfs target/x86_64-unknown-linux-musl/release/
cp artifacts/rustfs-arm64/rustfs target/aarch64-unknown-linux-gnu/release/
chmod +x target/*/release/rustfs
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Login to Docker Hub
if: github.event_name != 'pull_request' && (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/'))
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
if: github.event_name != 'pull_request' && (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/'))
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set Dockerfile and context
id: dockerfile
run: |
case "${{ matrix.image-type }}" in
production)
echo "dockerfile=Dockerfile" >> $GITHUB_OUTPUT
echo "context=." >> $GITHUB_OUTPUT
echo "suffix=" >> $GITHUB_OUTPUT
;;
ubuntu)
echo "dockerfile=.docker/Dockerfile.ubuntu22.04" >> $GITHUB_OUTPUT
echo "context=." >> $GITHUB_OUTPUT
echo "suffix=-ubuntu22.04" >> $GITHUB_OUTPUT
;;
rockylinux)
echo "dockerfile=.docker/Dockerfile.rockylinux9.3" >> $GITHUB_OUTPUT
echo "context=." >> $GITHUB_OUTPUT
echo "suffix=-rockylinux9.3" >> $GITHUB_OUTPUT
;;
devenv)
echo "dockerfile=.docker/Dockerfile.devenv" >> $GITHUB_OUTPUT
echo "context=." >> $GITHUB_OUTPUT
echo "suffix=-devenv" >> $GITHUB_OUTPUT
;;
esac
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: |
${{ env.REGISTRY_IMAGE_DOCKERHUB }}
${{ env.REGISTRY_IMAGE_GHCR }}
tags: |
type=ref,event=branch,suffix=${{ steps.dockerfile.outputs.suffix }}
type=ref,event=pr,suffix=${{ steps.dockerfile.outputs.suffix }}
type=semver,pattern={{version}},suffix=${{ steps.dockerfile.outputs.suffix }}
type=semver,pattern={{major}}.{{minor}},suffix=${{ steps.dockerfile.outputs.suffix }}
type=semver,pattern={{major}},suffix=${{ steps.dockerfile.outputs.suffix }}
type=raw,value=latest,suffix=${{ steps.dockerfile.outputs.suffix }},enable={{is_default_branch}}
flavor: |
latest=false
- name: Build and push Docker image
uses: docker/build-push-action@v5
with:
context: ${{ steps.dockerfile.outputs.context }}
file: ${{ steps.dockerfile.outputs.dockerfile }}
platforms: ${{ matrix.platform }}
push: ${{ github.event_name != 'pull_request' && (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/')) }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha,scope=${{ matrix.image-type }}-${{ matrix.platform }}
cache-to: type=gha,mode=max,scope=${{ matrix.image-type }}-${{ matrix.platform }}
build-args: |
BUILDTIME=${{ fromJSON(steps.meta.outputs.json).labels['org.opencontainers.image.created'] }}
VERSION=${{ fromJSON(steps.meta.outputs.json).labels['org.opencontainers.image.version'] }}
REVISION=${{ fromJSON(steps.meta.outputs.json).labels['org.opencontainers.image.revision'] }}
# Create multi-arch manifests
create-manifest:
needs: [skip-check, build-images]
if: needs.skip-check.outputs.should_skip != 'true' && github.event_name != 'pull_request' && (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/'))
runs-on: ubuntu-latest
strategy:
matrix:
image-type: [production, ubuntu, rockylinux, devenv]
steps:
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set image suffix
id: suffix
run: |
case "${{ matrix.image-type }}" in
production) echo "suffix=" >> $GITHUB_OUTPUT ;;
ubuntu) echo "suffix=-ubuntu22.04" >> $GITHUB_OUTPUT ;;
rockylinux) echo "suffix=-rockylinux9.3" >> $GITHUB_OUTPUT ;;
devenv) echo "suffix=-devenv" >> $GITHUB_OUTPUT ;;
esac
- name: Create and push manifest
run: |
# Set tag based on ref
if [[ $GITHUB_REF == refs/tags/* ]]; then
TAG=${GITHUB_REF#refs/tags/}
else
TAG="main"
fi
SUFFIX="${{ steps.suffix.outputs.suffix }}"
# Docker Hub manifest
docker buildx imagetools create -t ${REGISTRY_IMAGE_DOCKERHUB}:${TAG}${SUFFIX} \
${REGISTRY_IMAGE_DOCKERHUB}:${TAG}${SUFFIX}-linux-amd64 \
${REGISTRY_IMAGE_DOCKERHUB}:${TAG}${SUFFIX}-linux-arm64
# GitHub Container Registry manifest
docker buildx imagetools create -t ${REGISTRY_IMAGE_GHCR}:${TAG}${SUFFIX} \
${REGISTRY_IMAGE_GHCR}:${TAG}${SUFFIX}-linux-amd64 \
${REGISTRY_IMAGE_GHCR}:${TAG}${SUFFIX}-linux-arm64
# Create latest tag for main branch
if [[ $GITHUB_REF == refs/heads/main ]]; then
docker buildx imagetools create -t ${REGISTRY_IMAGE_DOCKERHUB}:latest${SUFFIX} \
${REGISTRY_IMAGE_DOCKERHUB}:${TAG}${SUFFIX}-linux-amd64 \
${REGISTRY_IMAGE_DOCKERHUB}:${TAG}${SUFFIX}-linux-arm64
docker buildx imagetools create -t ${REGISTRY_IMAGE_GHCR}:latest${SUFFIX} \
${REGISTRY_IMAGE_GHCR}:${TAG}${SUFFIX}-linux-amd64 \
${REGISTRY_IMAGE_GHCR}:${TAG}${SUFFIX}-linux-arm64
fi
# Security scanning
security-scan:
needs: [skip-check, build-images]
if: needs.skip-check.outputs.should_skip != 'true'
runs-on: ubuntu-latest
strategy:
matrix:
image-type: [production]
steps:
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@master
with:
image-ref: ${{ env.REGISTRY_IMAGE_GHCR }}:main
format: 'sarif'
output: 'trivy-results.sarif'
- name: Upload Trivy scan results to GitHub Security tab
uses: github/codeql-action/upload-sarif@v2
if: always()
with:
sarif_file: 'trivy-results.sarif'
Dockerfile
+25 -5
View File
@@ -1,17 +1,37 @@
FROM alpine:latest FROM alpine:latest
# RUN apk add --no-cache <package-name> # Install runtime dependencies
RUN apk add --no-cache \
ca-certificates \
tzdata \
&& rm -rf /var/cache/apk/*
# Create rustfs user and group
RUN addgroup -g 1000 rustfs && \
adduser -D -s /bin/sh -u 1000 -G rustfs rustfs
WORKDIR /app WORKDIR /app
RUN mkdir -p /data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3 # Create data directories
RUN mkdir -p /data/rustfs{0,1,2,3} && \
chown -R rustfs:rustfs /data /app
COPY ./target/x86_64-unknown-linux-musl/release/rustfs /app/rustfs # Copy binary based on target architecture
COPY --chown=rustfs:rustfs \
target/*/release/rustfs \
/app/rustfs
RUN chmod +x /app/rustfs RUN chmod +x /app/rustfs
EXPOSE 9000 # Switch to non-root user
EXPOSE 9001 USER rustfs
# Expose ports
EXPOSE 9000 9001
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1
# Set default command
CMD ["/app/rustfs"] CMD ["/app/rustfs"]
Dockerfile.multi-stage
+121
View File
@@ -0,0 +1,121 @@
# Multi-stage Dockerfile for RustFS
# Supports cross-compilation for amd64 and arm64 architectures
ARG TARGETPLATFORM
ARG BUILDPLATFORM
# Build stage
FROM --platform=$BUILDPLATFORM rust:1.85-bookworm AS builder
# Install required build dependencies
RUN apt-get update && apt-get install -y \
wget \
git \
curl \
unzip \
gcc \
pkg-config \
libssl-dev \
lld \
&& rm -rf /var/lib/apt/lists/*
# Install cross-compilation tools for ARM64
RUN if [ "$TARGETPLATFORM" = "linux/arm64" ]; then \
apt-get update && \
apt-get install -y gcc-aarch64-linux-gnu && \
rm -rf /var/lib/apt/lists/*; \
fi
# Install protoc
RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protoc-31.1-linux-x86_64.zip \
&& unzip protoc-31.1-linux-x86_64.zip -d protoc3 \
&& mv protoc3/bin/* /usr/local/bin/ && chmod +x /usr/local/bin/protoc \
&& mv protoc3/include/* /usr/local/include/ && rm -rf protoc-31.1-linux-x86_64.zip protoc3
# Install flatc
RUN wget https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.flatc.binary.g++-13.zip \
&& unzip Linux.flatc.binary.g++-13.zip \
&& mv flatc /usr/local/bin/ && chmod +x /usr/local/bin/flatc && rm -rf Linux.flatc.binary.g++-13.zip
# Set up Rust targets based on platform
RUN case "$TARGETPLATFORM" in \
"linux/amd64") rustup target add x86_64-unknown-linux-gnu ;; \
"linux/arm64") rustup target add aarch64-unknown-linux-gnu ;; \
*) echo "Unsupported platform: $TARGETPLATFORM" && exit 1 ;; \
esac
# Set up environment for cross-compilation
ENV CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc
ENV CC_aarch64_unknown_linux_gnu=aarch64-linux-gnu-gcc
ENV CXX_aarch64_unknown_linux_gnu=aarch64-linux-gnu-g++
WORKDIR /usr/src/rustfs
# Copy Cargo files for dependency caching
COPY Cargo.toml Cargo.lock ./
COPY */Cargo.toml ./*/
# Create dummy main.rs files for dependency compilation
RUN find . -name "Cargo.toml" -not -path "./Cargo.toml" | \
xargs -I {} dirname {} | \
xargs -I {} sh -c 'mkdir -p {}/src && echo "fn main() {}" > {}/src/main.rs'
# Build dependencies only (cache layer)
RUN case "$TARGETPLATFORM" in \
"linux/amd64") cargo build --release --target x86_64-unknown-linux-gnu ;; \
"linux/arm64") cargo build --release --target aarch64-unknown-linux-gnu ;; \
esac
# Copy source code
COPY . .
# Generate protobuf code
RUN cargo run --bin gproto
# Build the actual application
RUN case "$TARGETPLATFORM" in \
"linux/amd64") \
cargo build --release --target x86_64-unknown-linux-gnu --bin rustfs && \
cp target/x86_64-unknown-linux-gnu/release/rustfs /usr/local/bin/rustfs \
;; \
"linux/arm64") \
cargo build --release --target aarch64-unknown-linux-gnu --bin rustfs && \
cp target/aarch64-unknown-linux-gnu/release/rustfs /usr/local/bin/rustfs \
;; \
esac
# Runtime stage - Ubuntu minimal for better compatibility
FROM ubuntu:22.04
# Install runtime dependencies
RUN apt-get update && apt-get install -y \
ca-certificates \
tzdata \
wget \
&& rm -rf /var/lib/apt/lists/*
# Create rustfs user and group
RUN groupadd -g 1000 rustfs && \
useradd -d /app -g rustfs -u 1000 -s /bin/bash rustfs
WORKDIR /app
# Create data directories
RUN mkdir -p /data/rustfs{0,1,2,3} && \
chown -R rustfs:rustfs /data /app
# Copy binary from builder stage
COPY --from=builder /usr/local/bin/rustfs /app/rustfs
RUN chmod +x /app/rustfs && chown rustfs:rustfs /app/rustfs
# Switch to non-root user
USER rustfs
# Expose ports
EXPOSE 9000 9001
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:9000/health || exit 1
# Set default command
CMD ["/app/rustfs"]
docker-compose.yml
+221
View File
@@ -0,0 +1,221 @@
version: '3.8'
services:
# RustFS main service
rustfs:
image: rustfs/rustfs:latest
container_name: rustfs-server
build:
context: .
dockerfile: Dockerfile.multi-stage
args:
TARGETPLATFORM: linux/amd64
ports:
- "9000:9000" # S3 API port
- "9001:9001" # Console port
environment:
- RUSTFS_VOLUMES=/data/rustfs0,/data/rustfs1,/data/rustfs2,/data/rustfs3
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
- RUSTFS_ACCESS_KEY=rustfsadmin
- RUSTFS_SECRET_KEY=rustfsadmin
- RUSTFS_LOG_LEVEL=info
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
volumes:
- rustfs_data_0:/data/rustfs0
- rustfs_data_1:/data/rustfs1
- rustfs_data_2:/data/rustfs2
- rustfs_data_3:/data/rustfs3
- ./logs:/app/logs
networks:
- rustfs-network
restart: unless-stopped
healthcheck:
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:9000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
depends_on:
- otel-collector
# Development environment
rustfs-dev:
image: rustfs/rustfs:devenv
container_name: rustfs-dev
build:
context: .
dockerfile: .docker/Dockerfile.devenv
ports:
- "9010:9000"
- "9011:9001"
environment:
- RUSTFS_VOLUMES=/data/rustfs0,/data/rustfs1
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
- RUSTFS_ACCESS_KEY=devadmin
- RUSTFS_SECRET_KEY=devadmin
- RUSTFS_LOG_LEVEL=debug
volumes:
- .:/root/s3-rustfs
- rustfs_dev_data:/data
networks:
- rustfs-network
restart: unless-stopped
profiles:
- dev
# OpenTelemetry Collector
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
container_name: otel-collector
command:
- --config=/etc/otelcol-contrib/otel-collector.yml
volumes:
- ./.docker/observability/otel-collector.yml:/etc/otelcol-contrib/otel-collector.yml:ro
ports:
- "4317:4317" # OTLP gRPC receiver
- "4318:4318" # OTLP HTTP receiver
- "8888:8888" # Prometheus metrics
- "8889:8889" # Prometheus exporter metrics
networks:
- rustfs-network
restart: unless-stopped
profiles:
- observability
# Jaeger for tracing
jaeger:
image: jaegertracing/all-in-one:latest
container_name: jaeger
ports:
- "16686:16686" # Jaeger UI
- "14250:14250" # Jaeger gRPC
environment:
- COLLECTOR_OTLP_ENABLED=true
networks:
- rustfs-network
restart: unless-stopped
profiles:
- observability
# Prometheus for metrics
prometheus:
image: prom/prometheus:latest
container_name: prometheus
ports:
- "9090:9090"
volumes:
- ./.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--storage.tsdb.retention.time=200h'
- '--web.enable-lifecycle'
networks:
- rustfs-network
restart: unless-stopped
profiles:
- observability
# Grafana for visualization
grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=admin
volumes:
- grafana_data:/var/lib/grafana
- ./.docker/observability/grafana/provisioning:/etc/grafana/provisioning:ro
- ./.docker/observability/grafana/dashboards:/var/lib/grafana/dashboards:ro
networks:
- rustfs-network
restart: unless-stopped
profiles:
- observability
# MinIO for S3 API testing
minio:
image: minio/minio:latest
container_name: minio-test
ports:
- "9020:9000"
- "9021:9001"
environment:
- MINIO_ROOT_USER=minioadmin
- MINIO_ROOT_PASSWORD=minioadmin
volumes:
- minio_data:/data
command: server /data --console-address ":9001"
networks:
- rustfs-network
restart: unless-stopped
profiles:
- testing
# Redis for caching (optional)
redis:
image: redis:7-alpine
container_name: redis
ports:
- "6379:6379"
volumes:
- redis_data:/data
networks:
- rustfs-network
restart: unless-stopped
profiles:
- cache
# NGINX reverse proxy (optional)
nginx:
image: nginx:alpine
container_name: nginx-proxy
ports:
- "80:80"
- "443:443"
volumes:
- ./.docker/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./.docker/nginx/ssl:/etc/nginx/ssl:ro
networks:
- rustfs-network
restart: unless-stopped
profiles:
- proxy
depends_on:
- rustfs
networks:
rustfs-network:
driver: bridge
ipam:
config:
- subnet: 172.20.0.0/16
volumes:
rustfs_data_0:
driver: local
rustfs_data_1:
driver: local
rustfs_data_2:
driver: local
rustfs_data_3:
driver: local
rustfs_dev_data:
driver: local
prometheus_data:
driver: local
grafana_data:
driver: local
minio_data:
driver: local
redis_data:
driver: local
docs/docker-build.md
+530
View File
@@ -0,0 +1,530 @@
# RustFS Docker Build and Deployment Guide
This document describes how to build and deploy RustFS using Docker, including the automated GitHub Actions workflow for building and pushing images to Docker Hub and GitHub Container Registry.
## 🚀 Quick Start
### Using Pre-built Images
```bash
# Pull and run the latest RustFS image
docker run -d \
--name rustfs \
-p 9000:9000 \
-p 9001:9001 \
-v rustfs_data:/data \
-e RUSTFS_VOLUMES=/data/rustfs0,/data/rustfs1,/data/rustfs2,/data/rustfs3 \
-e RUSTFS_ACCESS_KEY=rustfsadmin \
-e RUSTFS_SECRET_KEY=rustfsadmin \
-e RUSTFS_CONSOLE_ENABLE=true \
rustfs/rustfs:latest
```
### Using Docker Compose
```bash
# Basic deployment
docker-compose up -d
# Development environment
docker-compose --profile dev up -d
# With observability stack
docker-compose --profile observability up -d
# Full stack with all services
docker-compose --profile dev --profile observability --profile testing up -d
```
## 📦 Available Images
Our GitHub Actions workflow builds multiple image variants:
### Image Registries
- **Docker Hub**: `rustfs/rustfs`
- **GitHub Container Registry**: `ghcr.io/rustfs/s3-rustfs`
### Image Variants
| Variant | Tag Suffix | Description | Use Case |
|---------|------------|-------------|----------|
| Production | *(none)* | Minimal Ubuntu-based runtime | Production deployment |
| Ubuntu | `-ubuntu22.04` | Ubuntu 22.04 based build environment | Development/Testing |
| Rocky Linux | `-rockylinux9.3` | Rocky Linux 9.3 based build environment | Enterprise environments |
| Development | `-devenv` | Full development environment | Development/Debugging |
### Supported Architectures
All images support multi-architecture:
- `linux/amd64` (x86_64-unknown-linux-musl)
- `linux/arm64` (aarch64-unknown-linux-gnu)
### Tag Examples
```bash
# Latest production image
rustfs/rustfs:latest
rustfs/rustfs:main
# Specific version
rustfs/rustfs:v1.0.0
rustfs/rustfs:v1.0.0-ubuntu22.04
# Development environment
rustfs/rustfs:latest-devenv
rustfs/rustfs:main-devenv
```
## 🔧 GitHub Actions Workflow
The Docker build workflow (`.github/workflows/docker.yml`) automatically:
1. **Builds cross-platform binaries** for `amd64` and `arm64`
2. **Creates Docker images** for all variants
3. **Pushes to registries** (Docker Hub and GitHub Container Registry)
4. **Creates multi-arch manifests** for seamless platform selection
5. **Performs security scanning** using Trivy
### Cross-Compilation Strategy
To handle complex native dependencies, we use different compilation strategies:
- **x86_64**: Native compilation with `x86_64-unknown-linux-musl` for static linking
- **aarch64**: Cross-compilation with `aarch64-unknown-linux-gnu` using the `cross` tool
This approach ensures compatibility with various C libraries while maintaining performance.
### Workflow Triggers
- **Push to main branch**: Builds and pushes `main` and `latest` tags
- **Tag push** (`v*`): Builds and pushes version tags
- **Pull requests**: Builds images without pushing
- **Manual trigger**: Workflow dispatch with options
### Required Secrets
Configure these secrets in your GitHub repository:
```bash
# Docker Hub credentials
DOCKERHUB_USERNAME=your-dockerhub-username
DOCKERHUB_TOKEN=your-dockerhub-access-token
# GitHub token is automatically available
GITHUB_TOKEN=automatically-provided
```
## 🏗️ Building Locally
### Prerequisites
- Docker with BuildKit enabled
- Rust toolchain (1.85+)
- Protocol Buffers compiler (protoc 31.1+)
- FlatBuffers compiler (flatc 25.2.10+)
- `cross` tool for ARM64 compilation
### Installation Commands
```bash
# Install Rust targets
rustup target add x86_64-unknown-linux-musl
rustup target add aarch64-unknown-linux-gnu
# Install cross for ARM64 compilation
cargo install cross --git https://github.com/cross-rs/cross
# Install protoc (macOS)
brew install protobuf
# Install protoc (Ubuntu)
sudo apt-get install protobuf-compiler
# Install flatc
# Download from: https://github.com/google/flatbuffers/releases
```
### Build Commands
```bash
# Test cross-compilation setup
./scripts/test-cross-build.sh
# Build production image for local platform
docker build -t rustfs:local .
# Build multi-stage production image
docker build -f Dockerfile.multi-stage -t rustfs:multi-stage .
# Build specific variant
docker build -f .docker/Dockerfile.ubuntu22.04 -t rustfs:ubuntu .
# Build for specific platform
docker build --platform linux/amd64 -t rustfs:amd64 .
docker build --platform linux/arm64 -t rustfs:arm64 .
# Build multi-platform image
docker buildx build --platform linux/amd64,linux/arm64 -t rustfs:multi .
```
### Cross-Compilation
```bash
# Generate protobuf code first
cargo run --bin gproto
# Native x86_64 build
cargo build --release --target x86_64-unknown-linux-musl --bin rustfs
# Cross-compile for ARM64
cross build --release --target aarch64-unknown-linux-gnu --bin rustfs
```
### Build with Docker Compose
```bash
# Build all services
docker-compose build
# Build specific service
docker-compose build rustfs
# Build development environment
docker-compose build rustfs-dev
```
## 🚀 Deployment Options
### 1. Single Container
```bash
docker run -d \
--name rustfs \
--restart unless-stopped \
-p 9000:9000 \
-p 9001:9001 \
-v /data/rustfs:/data \
-e RUSTFS_VOLUMES=/data/rustfs0,/data/rustfs1,/data/rustfs2,/data/rustfs3 \
-e RUSTFS_ADDRESS=0.0.0.0:9000 \
-e RUSTFS_CONSOLE_ENABLE=true \
-e RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001 \
-e RUSTFS_ACCESS_KEY=rustfsadmin \
-e RUSTFS_SECRET_KEY=rustfsadmin \
rustfs/rustfs:latest
```
### 2. Docker Compose Profiles
```bash
# Production deployment
docker-compose up -d
# Development with debugging
docker-compose --profile dev up -d
# With monitoring stack
docker-compose --profile observability up -d
# Complete testing environment
docker-compose --profile dev --profile observability --profile testing up -d
```
### 3. Kubernetes Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: rustfs
spec:
replicas: 3
selector:
matchLabels:
app: rustfs
template:
metadata:
labels:
app: rustfs
spec:
containers:
- name: rustfs
image: rustfs/rustfs:latest
ports:
- containerPort: 9000
- containerPort: 9001
env:
- name: RUSTFS_VOLUMES
value: "/data/rustfs0,/data/rustfs1,/data/rustfs2,/data/rustfs3"
- name: RUSTFS_ADDRESS
value: "0.0.0.0:9000"
- name: RUSTFS_CONSOLE_ENABLE
value: "true"
- name: RUSTFS_CONSOLE_ADDRESS
value: "0.0.0.0:9001"
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: rustfs-data
```
## ⚙️ Configuration
### Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `RUSTFS_VOLUMES` | Comma-separated list of data volumes | Required |
| `RUSTFS_ADDRESS` | Server bind address | `0.0.0.0:9000` |
| `RUSTFS_CONSOLE_ENABLE` | Enable web console | `false` |
| `RUSTFS_CONSOLE_ADDRESS` | Console bind address | `0.0.0.0:9001` |
| `RUSTFS_ACCESS_KEY` | S3 access key | `rustfsadmin` |
| `RUSTFS_SECRET_KEY` | S3 secret key | `rustfsadmin` |
| `RUSTFS_LOG_LEVEL` | Log level | `info` |
| `RUSTFS_OBS_ENDPOINT` | Observability endpoint | `""` |
| `RUSTFS_TLS_PATH` | TLS certificates path | `""` |
### Volume Mounts
- **Data volumes**: `/data/rustfs{0,1,2,3}` - RustFS data storage
- **Logs**: `/app/logs` - Application logs
- **Config**: `/etc/rustfs/` - Configuration files
- **TLS**: `/etc/ssl/rustfs/` - TLS certificates
### Ports
- **9000**: S3 API endpoint
- **9001**: Web console (if enabled)
- **9002**: Admin API (if enabled)
- **50051**: gRPC API (if enabled)
## 🔍 Monitoring and Observability
### Health Checks
The Docker images include built-in health checks:
```bash
# Check container health
docker ps --filter "name=rustfs" --format "table {{.Names}}\t{{.Status}}"
# View health check logs
docker inspect rustfs --format='{{json .State.Health}}'
```
### Metrics and Tracing
When using the observability profile:
- **Prometheus**: http://localhost:9090
- **Grafana**: http://localhost:3000 (admin/admin)
- **Jaeger**: http://localhost:16686
- **OpenTelemetry Collector**: http://localhost:8888/metrics
### Log Collection
```bash
# View container logs
docker logs rustfs -f
# Export logs
docker logs rustfs > rustfs.log 2>&1
```
## 🛠️ Development
### Development Environment
```bash
# Start development container
docker-compose --profile dev up -d rustfs-dev
# Access development container
docker exec -it rustfs-dev bash
# Mount source code for live development
docker run -it --rm \
-v $(pwd):/root/s3-rustfs \
-p 9000:9000 \
rustfs/rustfs:devenv \
bash
```
### Building from Source in Container
```bash
# Use development image for building
docker run --rm \
-v $(pwd):/root/s3-rustfs \
-w /root/s3-rustfs \
rustfs/rustfs:ubuntu22.04 \
cargo build --release --bin rustfs
```
### Testing Cross-Compilation
```bash
# Run the test script to verify cross-compilation setup
./scripts/test-cross-build.sh
# This will test:
# - x86_64-unknown-linux-musl compilation
# - aarch64-unknown-linux-gnu cross-compilation
# - Docker builds for both architectures
```
## 🔐 Security
### Security Scanning
The workflow includes Trivy security scanning:
```bash
# Run security scan locally
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
-v $HOME/Library/Caches:/root/.cache/ \
aquasec/trivy:latest image rustfs/rustfs:latest
```
### Security Best Practices
1. **Use non-root user**: Images run as `rustfs` user (UID 1000)
2. **Minimal base images**: Ubuntu minimal for production
3. **Security updates**: Regular base image updates
4. **Secret management**: Use Docker secrets or environment files
5. **Network security**: Use Docker networks and proper firewall rules
## 📝 Troubleshooting
### Common Issues
#### 1. Cross-Compilation Failures
**Problem**: ARM64 build fails with linking errors
```bash
error: linking with `aarch64-linux-gnu-gcc` failed
```
**Solution**: Use the `cross` tool instead of native cross-compilation:
```bash
# Install cross tool
cargo install cross --git https://github.com/cross-rs/cross
# Use cross for ARM64 builds
cross build --release --target aarch64-unknown-linux-gnu --bin rustfs
```
#### 2. Protobuf Generation Issues
**Problem**: Missing protobuf definitions
```bash
error: failed to run custom build command for `protos`
```
**Solution**: Generate protobuf code first:
```bash
cargo run --bin gproto
```
#### 3. Docker Build Failures
**Problem**: Binary not found in Docker build
```bash
COPY failed: file not found in build context
```
**Solution**: Ensure binaries are built before Docker build:
```bash
# Build binaries first
cargo build --release --target x86_64-unknown-linux-musl --bin rustfs
cross build --release --target aarch64-unknown-linux-gnu --bin rustfs
# Then build Docker image
docker build .
```
### Debug Commands
```bash
# Check container status
docker ps -a
# View container logs
docker logs rustfs --tail 100
# Access container shell
docker exec -it rustfs bash
# Check resource usage
docker stats rustfs
# Inspect container configuration
docker inspect rustfs
# Test cross-compilation setup
./scripts/test-cross-build.sh
```
## 🔄 CI/CD Integration
### GitHub Actions
The provided workflow can be customized:
```yaml
# Override image names
env:
REGISTRY_IMAGE_DOCKERHUB: myorg/rustfs
REGISTRY_IMAGE_GHCR: ghcr.io/myorg/rustfs
```
### GitLab CI
```yaml
build:
stage: build
image: docker:latest
services:
- docker:dind
script:
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
```
### Jenkins Pipeline
```groovy
pipeline {
agent any
stages {
stage('Build') {
steps {
script {
docker.build("rustfs:${env.BUILD_ID}")
}
}
}
stage('Push') {
steps {
script {
docker.withRegistry('https://registry.hub.docker.com', 'dockerhub-credentials') {
docker.image("rustfs:${env.BUILD_ID}").push()
}
}
}
}
}
}
```
## 📚 Additional Resources
- [Docker Official Documentation](https://docs.docker.com/)
- [Docker Compose Reference](https://docs.docker.com/compose/)
- [GitHub Actions Documentation](https://docs.github.com/en/actions)
- [Cross-compilation with Rust](https://rust-lang.github.io/rustup/cross-compilation.html)
- [Cross tool documentation](https://github.com/cross-rs/cross)
- [RustFS Configuration Guide](../README.md)
ecstore/BENCHMARK.md
-270
View File
@@ -1,270 +0,0 @@
# Reed-Solomon Erasure Coding Performance Benchmark
This directory contains a comprehensive benchmark suite for comparing the performance of different Reed-Solomon implementations.
## 📊 Test Overview
### Supported Implementation Modes
#### 🏛️ Pure Erasure Mode (Default, Recommended)
- **Stable and Reliable**: Uses mature reed-solomon-erasure implementation
- **Wide Compatibility**: Supports arbitrary shard sizes
- **Memory Efficient**: Optimized memory usage patterns
- **Predictable**: Performance insensitive to shard size
- **Use Case**: Default choice for production environments, suitable for most application scenarios
#### 🎯 SIMD Mode (`reed-solomon-simd` feature)
- **High Performance Optimization**: Uses SIMD instruction sets for high-performance encoding/decoding
- **Performance Oriented**: Focuses on maximizing processing performance
- **Target Scenarios**: High-performance scenarios for large data processing
- **Use Case**: Scenarios requiring maximum performance, suitable for handling large amounts of data
### Test Dimensions
- **Encoding Performance** - Speed of encoding data into erasure code shards
- **Decoding Performance** - Speed of recovering original data from erasure code shards
- **Shard Size Sensitivity** - Impact of different shard sizes on performance
- **Erasure Code Configuration** - Performance impact of different data/parity shard ratios
- **SIMD Mode Performance** - Performance characteristics of SIMD optimization
- **Concurrency Performance** - Performance in multi-threaded environments
- **Memory Efficiency** - Memory usage patterns and efficiency
- **Error Recovery Capability** - Recovery performance under different numbers of lost shards
## 🚀 Quick Start
### Run Quick Tests
```bash
# Run quick performance comparison tests (default pure Erasure mode)
./run_benchmarks.sh quick
```
### Run Complete Comparison Tests
```bash
# Run detailed implementation comparison tests
./run_benchmarks.sh comparison
```
### Run Specific Mode Tests
```bash
# Test default pure erasure mode (recommended)
./run_benchmarks.sh erasure
# Test SIMD mode
./run_benchmarks.sh simd
```
## 📈 Manual Benchmark Execution
### Basic Usage
```bash
# Run all benchmarks (default pure erasure mode)
cargo bench
# Run specific benchmark files
cargo bench --bench erasure_benchmark
cargo bench --bench comparison_benchmark
```
### Compare Different Implementation Modes
```bash
# Test default pure erasure mode
cargo bench --bench comparison_benchmark
# Test SIMD mode
cargo bench --bench comparison_benchmark \
--features reed-solomon-simd
# Save baseline for comparison
cargo bench --bench comparison_benchmark \
-- --save-baseline erasure_baseline
# Compare SIMD mode performance with baseline
cargo bench --bench comparison_benchmark \
--features reed-solomon-simd \
-- --baseline erasure_baseline
```
### Filter Specific Tests
```bash
# Run only encoding tests
cargo bench encode
# Run only decoding tests
cargo bench decode
# Run tests for specific data sizes
cargo bench 1MB
# Run tests for specific configurations
cargo bench "4+2"
```
## 📊 View Results
### HTML Reports
Benchmark results automatically generate HTML reports:
```bash
# Start local server to view reports
cd target/criterion
python3 -m http.server 8080
# Access in browser
open http://localhost:8080/report/index.html
```
### Command Line Output
Benchmarks display in terminal:
- Operations per second (ops/sec)
- Throughput (MB/s)
- Latency statistics (mean, standard deviation, percentiles)
- Performance trend changes
## 🔧 Test Configuration
### Data Sizes
- **Small Data**: 1KB, 8KB - Test small file scenarios
- **Medium Data**: 64KB, 256KB - Test common file sizes
- **Large Data**: 1MB, 4MB - Test large file processing and SIMD optimization
- **Very Large Data**: 16MB+ - Test high throughput scenarios
### Erasure Code Configurations
- **(4,2)** - Common configuration, 33% redundancy
- **(6,3)** - 50% redundancy, balanced performance and reliability
- **(8,4)** - 50% redundancy, more parallelism
- **(10,5)**, **(12,6)** - High parallelism configurations
### Shard Sizes
Test different shard sizes from 32 bytes to 8KB, with special focus on:
- **Memory Alignment**: 64, 128, 256 bytes - Impact of memory alignment on performance
- **Cache Friendly**: 1KB, 2KB, 4KB - CPU cache-friendly sizes
## 📝 Interpreting Test Results
### Performance Metrics
1. **Throughput**
- Unit: MB/s or GB/s
- Measures data processing speed
- Higher is better
2. **Latency**
- Unit: microseconds (μs) or milliseconds (ms)
- Measures single operation time
- Lower is better
3. **CPU Efficiency**
- Bytes processed per CPU cycle
- Reflects algorithm efficiency
### Expected Results
**Pure Erasure Mode (Default)**:
- Stable performance, insensitive to shard size
- Best compatibility, supports all configurations
- Stable and predictable memory usage
**SIMD Mode (`reed-solomon-simd` feature)**:
- High-performance SIMD optimized implementation
- Suitable for large data processing scenarios
- Focuses on maximizing performance
**Shard Size Sensitivity**:
- SIMD mode may be more sensitive to shard sizes
- Pure Erasure mode relatively insensitive to shard size
**Memory Usage**:
- SIMD mode may have specific memory alignment requirements
- Pure Erasure mode has more stable memory usage
## 🛠️ Custom Testing
### Adding New Test Scenarios
Edit `benches/erasure_benchmark.rs` or `benches/comparison_benchmark.rs`:
```rust
// Add new test configuration
let configs = vec![
// Your custom configuration
BenchConfig::new(10, 4, 2048 * 1024, 2048 * 1024), // 10+4, 2MB
];
```
### Adjust Test Parameters
```rust
// Modify sampling and test time
group.sample_size(20); // Sample count
group.measurement_time(Duration::from_secs(10)); // Test duration
```
## 🐛 Troubleshooting
### Common Issues
1. **Compilation Errors**: Ensure correct dependencies are installed
```bash
cargo update
cargo build --all-features
```
2. **Performance Anomalies**: Check if running in correct mode
```bash
# Check current configuration
cargo bench --bench comparison_benchmark -- --help
```
3. **Tests Taking Too Long**: Adjust test parameters
```bash
# Use shorter test duration
cargo bench -- --quick
```
### Performance Analysis
Use tools like `perf` for detailed performance analysis:
```bash
# Analyze CPU usage
cargo bench --bench comparison_benchmark &
perf record -p $(pgrep -f comparison_benchmark)
perf report
```
## 🤝 Contributing
Welcome to submit new benchmark scenarios or optimization suggestions:
1. Fork the project
2. Create feature branch: `git checkout -b feature/new-benchmark`
3. Add test cases
4. Commit changes: `git commit -m 'Add new benchmark for XYZ'`
5. Push to branch: `git push origin feature/new-benchmark`
6. Create Pull Request
## 📚 References
- [reed-solomon-erasure crate](https://crates.io/crates/reed-solomon-erasure)
- [reed-solomon-simd crate](https://crates.io/crates/reed-solomon-simd)
- [Criterion.rs benchmark framework](https://bheisler.github.io/criterion.rs/book/)
- [Reed-Solomon error correction principles](https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction)
---
💡 **Tips**:
- Recommend using the default pure Erasure mode, which provides stable performance across various scenarios
- Consider SIMD mode for high-performance requirements
- Benchmark results may vary based on hardware, operating system, and compiler versions
- Suggest running tests in target deployment environment for most accurate performance data
ecstore/BENCHMARK_ZH.md
-270
View File
@@ -1,270 +0,0 @@
# Reed-Solomon 纠删码性能基准测试
本目录包含了比较不同 Reed-Solomon 实现性能的综合基准测试套件。
## 📊 测试概述
### 支持的实现模式
#### 🏛️ 纯 Erasure 模式（默认，推荐）
- **稳定可靠**: 使用成熟的 reed-solomon-erasure 实现
- **广泛兼容**: 支持任意分片大小
- **内存高效**: 优化的内存使用模式
- **可预测性**: 性能对分片大小不敏感
- **使用场景**: 生产环境默认选择，适合大多数应用场景
#### 🎯 SIMD模式（`reed-solomon-simd` feature）
- **高性能优化**: 使用SIMD指令集进行高性能编码解码
- **性能导向**: 专注于最大化处理性能
- **适用场景**: 大数据量处理的高性能场景
- **使用场景**: 需要最大化性能的场景，适合处理大量数据
### 测试维度
- **编码性能** - 数据编码成纠删码分片的速度
- **解码性能** - 从纠删码分片恢复原始数据的速度
- **分片大小敏感性** - 不同分片大小对性能的影响
- **纠删码配置** - 不同数据/奇偶分片比例的性能影响
- **SIMD模式性能** - SIMD优化的性能表现
- **并发性能** - 多线程环境下的性能表现
- **内存效率** - 内存使用模式和效率
- **错误恢复能力** - 不同丢失分片数量下的恢复性能
## 🚀 快速开始
### 运行快速测试
```bash
# 运行快速性能对比测试（默认纯Erasure模式）
./run_benchmarks.sh quick
```
### 运行完整对比测试
```bash
# 运行详细的实现对比测试
./run_benchmarks.sh comparison
```
### 运行特定模式的测试
```bash
# 测试默认纯 erasure 模式（推荐）
./run_benchmarks.sh erasure
# 测试SIMD模式
./run_benchmarks.sh simd
```
## 📈 手动运行基准测试
### 基本使用
```bash
# 运行所有基准测试（默认纯 erasure 模式）
cargo bench
# 运行特定的基准测试文件
cargo bench --bench erasure_benchmark
cargo bench --bench comparison_benchmark
```
### 对比不同实现模式
```bash
# 测试默认纯 erasure 模式
cargo bench --bench comparison_benchmark
# 测试SIMD模式
cargo bench --bench comparison_benchmark \
--features reed-solomon-simd
# 保存基线进行对比
cargo bench --bench comparison_benchmark \
-- --save-baseline erasure_baseline
# 与基线比较SIMD模式性能
cargo bench --bench comparison_benchmark \
--features reed-solomon-simd \
-- --baseline erasure_baseline
```
### 过滤特定测试
```bash
# 只运行编码测试
cargo bench encode
# 只运行解码测试
cargo bench decode
# 只运行特定数据大小的测试
cargo bench 1MB
# 只运行特定配置的测试
cargo bench "4+2"
```
## 📊 查看结果
### HTML 报告
基准测试结果会自动生成 HTML 报告：
```bash
# 启动本地服务器查看报告
cd target/criterion
python3 -m http.server 8080
# 在浏览器中访问
open http://localhost:8080/report/index.html
```
### 命令行输出
基准测试会在终端显示：
- 每秒操作数 (ops/sec)
- 吞吐量 (MB/s)
- 延迟统计 (平均值、标准差、百分位数)
- 性能变化趋势
## 🔧 测试配置
### 数据大小
- **小数据**: 1KB, 8KB - 测试小文件场景
- **中等数据**: 64KB, 256KB - 测试常见文件大小
- **大数据**: 1MB, 4MB - 测试大文件处理和 SIMD 优化
- **超大数据**: 16MB+ - 测试高吞吐量场景
### 纠删码配置
- **(4,2)** - 常用配置，33% 冗余
- **(6,3)** - 50% 冗余，平衡性能和可靠性
- **(8,4)** - 50% 冗余，更多并行度
- **(10,5)**, **(12,6)** - 高并行度配置
### 分片大小
测试从 32 字节到 8KB 的不同分片大小，特别关注：
- **内存对齐**: 64, 128, 256 字节 - 内存对齐对性能的影响
- **Cache 友好**: 1KB, 2KB, 4KB - CPU 缓存友好的大小
## 📝 解读测试结果
### 性能指标
1. **吞吐量 (Throughput)**
- 单位: MB/s 或 GB/s
- 衡量数据处理速度
- 越高越好
2. **延迟 (Latency)**
- 单位: 微秒 (μs) 或毫秒 (ms)
- 衡量单次操作时间
- 越低越好
3. **CPU 效率**
- 每 CPU 周期处理的字节数
- 反映算法效率
### 预期结果
**纯 Erasure 模式（默认）**:
- 性能稳定，对分片大小不敏感
- 兼容性最佳，支持所有配置
- 内存使用稳定可预测
**SIMD模式（`reed-solomon-simd` feature）**:
- 高性能SIMD优化实现
- 适合大数据量处理场景
- 专注于最大化性能
**分片大小敏感性**:
- SIMD模式对分片大小可能更敏感
- 纯 Erasure 模式对分片大小相对不敏感
**内存使用**:
- SIMD模式可能有特定的内存对齐要求
- 纯 Erasure 模式内存使用更稳定
## 🛠️ 自定义测试
### 添加新的测试场景
编辑 `benches/erasure_benchmark.rs` 或 `benches/comparison_benchmark.rs`：
```rust
// 添加新的测试配置
let configs = vec![
// 你的自定义配置
BenchConfig::new(10, 4, 2048 * 1024, 2048 * 1024), // 10+4, 2MB
];
```
### 调整测试参数
```rust
// 修改采样和测试时间
group.sample_size(20); // 样本数量
group.measurement_time(Duration::from_secs(10)); // 测试时间
```
## 🐛 故障排除
### 常见问题
1. **编译错误**: 确保安装了正确的依赖
```bash
cargo update
cargo build --all-features
```
2. **性能异常**: 检查是否在正确的模式下运行
```bash
# 检查当前配置
cargo bench --bench comparison_benchmark -- --help
```
3. **测试时间过长**: 调整测试参数
```bash
# 使用更短的测试时间
cargo bench -- --quick
```
### 性能分析
使用 `perf` 等工具进行更详细的性能分析：
```bash
# 分析 CPU 使用情况
cargo bench --bench comparison_benchmark &
perf record -p $(pgrep -f comparison_benchmark)
perf report
```
## 🤝 贡献
欢迎提交新的基准测试场景或优化建议：
1. Fork 项目
2. 创建特性分支: `git checkout -b feature/new-benchmark`
3. 添加测试用例
4. 提交更改: `git commit -m 'Add new benchmark for XYZ'`
5. 推送到分支: `git push origin feature/new-benchmark`
6. 创建 Pull Request
## 📚 参考资料
- [reed-solomon-erasure crate](https://crates.io/crates/reed-solomon-erasure)
- [reed-solomon-simd crate](https://crates.io/crates/reed-solomon-simd)
- [Criterion.rs 基准测试框架](https://bheisler.github.io/criterion.rs/book/)
- [Reed-Solomon 纠删码原理](https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction)
---
💡 **提示**:
- 推荐使用默认的纯Erasure模式，它在各种场景下都有稳定的表现
- 对于高性能需求可以考虑SIMD模式
- 基准测试结果可能因硬件、操作系统和编译器版本而异
- 建议在目标部署环境中运行测试以获得最准确的性能数据
ecstore/IMPLEMENTATION_COMPARISON.md
-333
View File
@@ -1,333 +0,0 @@
# Reed-Solomon Implementation Comparison Analysis
## 🔍 Issue Analysis
With the optimized SIMD mode design, we provide high-performance Reed-Solomon implementation. The system can now deliver optimal performance across different scenarios.
## 📊 Implementation Mode Comparison
### 🏛️ Pure Erasure Mode (Default, Recommended)
**Default Configuration**: No features specified, uses stable reed-solomon-erasure implementation
**Characteristics**:
- ✅ **Wide Compatibility**: Supports any shard size from byte-level to GB-level
- 📈 **Stable Performance**: Performance insensitive to shard size, predictable
- 🔧 **Production Ready**: Mature and stable implementation, widely used in production
- 💾 **Memory Efficient**: Optimized memory usage patterns
- 🎯 **Consistency**: Completely consistent behavior across all scenarios
**Use Cases**:
- Default choice for most production environments
- Systems requiring completely consistent and predictable performance behavior
- Performance-change-sensitive systems
- Scenarios mainly processing small files or small shards
- Systems requiring strict memory usage control
### 🎯 SIMD Mode (`reed-solomon-simd` feature)
**Configuration**: `--features reed-solomon-simd`
**Characteristics**:
- 🚀 **High-Performance SIMD**: Uses SIMD instruction sets for high-performance encoding/decoding
- 🎯 **Performance Oriented**: Focuses on maximizing processing performance
- ⚡ **Large Data Optimization**: Suitable for high-throughput scenarios with large data processing
- 🏎️ **Speed Priority**: Designed for performance-critical applications
**Use Cases**:
- Application scenarios requiring maximum performance
- High-throughput systems processing large amounts of data
- Scenarios with extremely high performance requirements
- CPU-intensive workloads
## 📏 Shard Size vs Performance Comparison
Performance across different configurations:
| Data Size | Config | Shard Size | Pure Erasure Mode (Default) | SIMD Mode Strategy | Performance Comparison |
|-----------|--------|------------|----------------------------|-------------------|----------------------|
| 1KB | 4+2 | 256 bytes | Erasure implementation | SIMD implementation | SIMD may be faster |
| 1KB | 6+3 | 171 bytes | Erasure implementation | SIMD implementation | SIMD may be faster |
| 1KB | 8+4 | 128 bytes | Erasure implementation | SIMD implementation | SIMD may be faster |
| 64KB | 4+2 | 16KB | Erasure implementation | SIMD optimization | SIMD mode faster |
| 64KB | 6+3 | 10.7KB | Erasure implementation | SIMD optimization | SIMD mode faster |
| 1MB | 4+2 | 256KB | Erasure implementation | SIMD optimization | SIMD mode significantly faster |
| 16MB | 8+4 | 2MB | Erasure implementation | SIMD optimization | SIMD mode substantially faster |
## 🎯 Benchmark Results Interpretation
### Pure Erasure Mode Example (Default) ✅
```
encode_comparison/implementation/1KB_6+3_erasure
time: [245.67 ns 256.78 ns 267.89 ns]
thrpt: [3.73 GiB/s 3.89 GiB/s 4.07 GiB/s]
💡 Consistent Erasure performance - All configurations use the same implementation
```
```
encode_comparison/implementation/64KB_4+2_erasure
time: [2.3456 μs 2.4567 μs 2.5678 μs]
thrpt: [23.89 GiB/s 24.65 GiB/s 25.43 GiB/s]
💡 Stable and reliable performance - Suitable for most production scenarios
```
### SIMD Mode Success Examples ✅
**Large Shard SIMD Optimization**:
```
encode_comparison/implementation/64KB_4+2_simd
time: [1.2345 μs 1.2567 μs 1.2789 μs]
thrpt: [47.89 GiB/s 48.65 GiB/s 49.43 GiB/s]
💡 Using SIMD optimization - Shard size: 16KB, high-performance processing
```
**Small Shard SIMD Processing**:
```
encode_comparison/implementation/1KB_6+3_simd
time: [234.56 ns 245.67 ns 256.78 ns]
thrpt: [3.89 GiB/s 4.07 GiB/s 4.26 GiB/s]
💡 SIMD processing small shards - Shard size: 171 bytes
```
## 🛠️ Usage Guide
### Selection Strategy
#### 1️⃣ Recommended: Pure Erasure Mode (Default)
```bash
# No features needed, use default configuration
cargo run
cargo test
cargo bench
```
**Applicable Scenarios**:
- 📊 **Consistency Requirements**: Need completely predictable performance behavior
- 🔬 **Production Environment**: Best choice for most production scenarios
- 💾 **Memory Sensitive**: Strict requirements for memory usage patterns
- 🏗️ **Stable and Reliable**: Mature and stable implementation
#### 2️⃣ High Performance Requirements: SIMD Mode
```bash
# Enable SIMD mode for maximum performance
cargo run --features reed-solomon-simd
cargo test --features reed-solomon-simd
cargo bench --features reed-solomon-simd
```
**Applicable Scenarios**:
- 🎯 **High Performance Scenarios**: Processing large amounts of data requiring maximum throughput
- 🚀 **Performance Optimization**: Want optimal performance for large data
- ⚡ **Speed Priority**: Scenarios with extremely high speed requirements
- 🏎️ **Compute Intensive**: CPU-intensive workloads
### Configuration Optimization Recommendations
#### Based on Data Size
**Small Files Primarily** (< 64KB):
```toml
# Recommended to use default pure Erasure mode
# No special configuration needed, stable and reliable performance
```
**Large Files Primarily** (> 1MB):
```toml
# Recommend enabling SIMD mode for higher performance
# features = ["reed-solomon-simd"]
```
**Mixed Scenarios**:
```toml
# Default pure Erasure mode suits most scenarios
# For maximum performance, enable: features = ["reed-solomon-simd"]
```
#### Recommendations Based on Erasure Coding Configuration
| Config | Small Data (< 64KB) | Large Data (> 1MB) | Recommended Mode |
|--------|-------------------|-------------------|------------------|
| 4+2 | Pure Erasure | Pure Erasure / SIMD Mode | Pure Erasure (Default) |
| 6+3 | Pure Erasure | Pure Erasure / SIMD Mode | Pure Erasure (Default) |
| 8+4 | Pure Erasure | Pure Erasure / SIMD Mode | Pure Erasure (Default) |
| 10+5 | Pure Erasure | Pure Erasure / SIMD Mode | Pure Erasure (Default) |
### Production Environment Deployment Recommendations
#### 1️⃣ Default Deployment Strategy
```bash
# Production environment recommended configuration: Use pure Erasure mode (default)
cargo build --release
```
**Advantages**:
- ✅ Maximum compatibility: Handle data of any size
- ✅ Stable and reliable: Mature implementation, predictable behavior
- ✅ Zero configuration: No complex performance tuning needed
- ✅ Memory efficient: Optimized memory usage patterns
#### 2️⃣ High Performance Deployment Strategy
```bash
# High performance scenarios: Enable SIMD mode
cargo build --release --features reed-solomon-simd
```
**Advantages**:
- ✅ Optimal performance: SIMD instruction set optimization
- ✅ High throughput: Suitable for large data processing
- ✅ Performance oriented: Focuses on maximizing processing speed
- ✅ Modern hardware: Fully utilizes modern CPU features
#### 2️⃣ Monitoring and Tuning
```rust
// Choose appropriate implementation based on specific scenarios
match data_size {
size if size > 1024 * 1024 => {
// Large data: Consider using SIMD mode
println!("Large data detected, SIMD mode recommended");
}
_ => {
// General case: Use default Erasure mode
println!("Using default Erasure mode");
}
}
```
#### 3️⃣ Performance Monitoring Metrics
- **Throughput Monitoring**: Monitor encoding/decoding data processing rates
- **Latency Analysis**: Analyze processing latency for different data sizes
- **CPU Utilization**: Observe CPU utilization efficiency of SIMD instructions
- **Memory Usage**: Monitor memory allocation patterns of different implementations
## 🔧 Troubleshooting
### Performance Issue Diagnosis
#### Issue 1: Performance Not Meeting Expectations
**Symptom**: SIMD mode performance improvement not significant
**Cause**: Data size may not be suitable for SIMD optimization
**Solution**:
```rust
// Check shard size and data characteristics
let shard_size = data.len().div_ceil(data_shards);
println!("Shard size: {} bytes", shard_size);
if shard_size >= 1024 {
println!("Good candidate for SIMD optimization");
} else {
println!("Consider using default Erasure mode");
}
```
#### Issue 2: Compilation Errors
**Symptom**: SIMD-related compilation errors
**Cause**: Platform not supported or missing dependencies
**Solution**:
```bash
# Check platform support
cargo check --features reed-solomon-simd
# If failed, use default mode
cargo check
```
#### Issue 3: Abnormal Memory Usage
**Symptom**: Memory usage exceeds expectations
**Cause**: Memory alignment requirements of SIMD implementation
**Solution**:
```bash
# Use pure Erasure mode for comparison
cargo run --features reed-solomon-erasure
```
### Debugging Tips
#### 1️⃣ Performance Comparison Testing
```bash
# Test pure Erasure mode performance
cargo bench --features reed-solomon-erasure
# Test SIMD mode performance
cargo bench --features reed-solomon-simd
```
#### 2️⃣ Analyze Data Characteristics
```rust
// Statistics of data characteristics in your application
let data_sizes: Vec<usize> = data_samples.iter()
.map(|data| data.len())
.collect();
let large_data_count = data_sizes.iter()
.filter(|&&size| size >= 1024 * 1024)
.count();
println!("Large data (>1MB): {}/{} ({}%)",
large_data_count,
data_sizes.len(),
large_data_count * 100 / data_sizes.len()
);
```
#### 3️⃣ Benchmark Comparison
```bash
# Generate detailed performance comparison report
./run_benchmarks.sh comparison
# View HTML report to analyze performance differences
cd target/criterion && python3 -m http.server 8080
```
## 📈 Performance Optimization Recommendations
### Application Layer Optimization
#### 1️⃣ Data Chunking Strategy
```rust
// Optimize data chunking for SIMD mode
const OPTIMAL_BLOCK_SIZE: usize = 1024 * 1024; // 1MB
const MIN_EFFICIENT_SIZE: usize = 64 * 1024; // 64KB
let block_size = if data.len() < MIN_EFFICIENT_SIZE {
data.len() // Small data can consider default mode
} else {
OPTIMAL_BLOCK_SIZE.min(data.len()) // Use optimal block size
};
```
#### 2️⃣ Configuration Tuning
```rust
// Choose erasure coding configuration based on typical data size
let (data_shards, parity_shards) = if typical_file_size > 1024 * 1024 {
(8, 4) // Large files: more parallelism, utilize SIMD
} else {
(4, 2) // Small files: simple configuration, reduce overhead
};
```
### System Layer Optimization
#### 1️⃣ CPU Feature Detection
```bash
# Check CPU supported SIMD instruction sets
lscpu | grep -i flags
cat /proc/cpuinfo | grep -i flags | head -1
```
#### 2️⃣ Memory Alignment Optimization
```rust
// Ensure data memory alignment to improve SIMD performance
use aligned_vec::AlignedVec;
let aligned_data = AlignedVec::<u8, aligned_vec::A64>::from_slice(&data);
```
---
💡 **Key Conclusions**:
- 🎯 **Pure Erasure mode (default) is the best general choice**: Stable and reliable, suitable for most scenarios
- 🚀 **SIMD mode suitable for high-performance scenarios**: Best choice for large data processing
- 📊 **Choose based on data characteristics**: Small data use Erasure, large data consider SIMD
- 🛡️ **Stability priority**: Production environments recommend using default Erasure mode
ecstore/IMPLEMENTATION_COMPARISON_ZH.md
-333
View File
@@ -1,333 +0,0 @@
# Reed-Solomon 实现对比分析
## 🔍 问题分析
随着SIMD模式的优化设计，我们提供了高性能的Reed-Solomon实现。现在系统能够在不同场景下提供最优的性能表现。
## 📊 实现模式对比
### 🏛️ 纯 Erasure 模式（默认，推荐）
**默认配置**: 不指定任何 feature，使用稳定的 reed-solomon-erasure 实现
**特点**:
- ✅ **广泛兼容**: 支持任意分片大小，从字节级到 GB 级
- 📈 **稳定性能**: 性能对分片大小不敏感，可预测
- 🔧 **生产就绪**: 成熟稳定的实现，已在生产环境广泛使用
- 💾 **内存高效**: 优化的内存使用模式
- 🎯 **一致性**: 在所有场景下行为完全一致
**使用场景**:
- 大多数生产环境的默认选择
- 需要完全一致和可预测的性能行为
- 对性能变化敏感的系统
- 主要处理小文件或小分片的场景
- 需要严格的内存使用控制
### 🎯 SIMD模式（`reed-solomon-simd` feature）
**配置**: `--features reed-solomon-simd`
**特点**:
- 🚀 **高性能SIMD**: 使用SIMD指令集进行高性能编码解码
- 🎯 **性能导向**: 专注于最大化处理性能
- ⚡ **大数据优化**: 适合大数据量处理的高吞吐量场景
- 🏎️ **速度优先**: 为性能关键型应用设计
**使用场景**:
- 需要最大化性能的应用场景
- 处理大量数据的高吞吐量系统
- 对性能要求极高的场景
- CPU密集型工作负载
## 📏 分片大小与性能对比
不同配置下的性能表现：
| 数据大小 | 配置 | 分片大小 | 纯 Erasure 模式（默认） | SIMD模式策略 | 性能对比 |
|---------|------|----------|------------------------|-------------|----------|
| 1KB | 4+2 | 256字节 | Erasure 实现 | SIMD 实现 | SIMD可能更快 |
| 1KB | 6+3 | 171字节 | Erasure 实现 | SIMD 实现 | SIMD可能更快 |
| 1KB | 8+4 | 128字节 | Erasure 实现 | SIMD 实现 | SIMD可能更快 |
| 64KB | 4+2 | 16KB | Erasure 实现 | SIMD 优化 | SIMD模式更快 |
| 64KB | 6+3 | 10.7KB | Erasure 实现 | SIMD 优化 | SIMD模式更快 |
| 1MB | 4+2 | 256KB | Erasure 实现 | SIMD 优化 | SIMD模式显著更快 |
| 16MB | 8+4 | 2MB | Erasure 实现 | SIMD 优化 | SIMD模式大幅领先 |
## 🎯 基准测试结果解读
### 纯 Erasure 模式示例（默认） ✅
```
encode_comparison/implementation/1KB_6+3_erasure
time: [245.67 ns 256.78 ns 267.89 ns]
thrpt: [3.73 GiB/s 3.89 GiB/s 4.07 GiB/s]
💡 一致的 Erasure 性能 - 所有配置都使用相同实现
```
```
encode_comparison/implementation/64KB_4+2_erasure
time: [2.3456 μs 2.4567 μs 2.5678 μs]
thrpt: [23.89 GiB/s 24.65 GiB/s 25.43 GiB/s]
💡 稳定可靠的性能 - 适合大多数生产场景
```
### SIMD模式成功示例 ✅
**大分片 SIMD 优化**:
```
encode_comparison/implementation/64KB_4+2_simd
time: [1.2345 μs 1.2567 μs 1.2789 μs]
thrpt: [47.89 GiB/s 48.65 GiB/s 49.43 GiB/s]
💡 使用 SIMD 优化 - 分片大小: 16KB，高性能处理
```
**小分片 SIMD 处理**:
```
encode_comparison/implementation/1KB_6+3_simd
time: [234.56 ns 245.67 ns 256.78 ns]
thrpt: [3.89 GiB/s 4.07 GiB/s 4.26 GiB/s]
💡 SIMD 处理小分片 - 分片大小: 171字节
```
## 🛠️ 使用指南
### 选择策略
#### 1️⃣ 推荐：纯 Erasure 模式（默认）
```bash
# 无需指定 feature，使用默认配置
cargo run
cargo test
cargo bench
```
**适用场景**:
- 📊 **一致性要求**: 需要完全可预测的性能行为
- 🔬 **生产环境**: 大多数生产场景的最佳选择
- 💾 **内存敏感**: 对内存使用模式有严格要求
- 🏗️ **稳定可靠**: 成熟稳定的实现
#### 2️⃣ 高性能需求：SIMD模式
```bash
# 启用SIMD模式获得最大性能
cargo run --features reed-solomon-simd
cargo test --features reed-solomon-simd
cargo bench --features reed-solomon-simd
```
**适用场景**:
- 🎯 **高性能场景**: 处理大量数据需要最大吞吐量
- 🚀 **性能优化**: 希望在大数据时获得最佳性能
- ⚡ **速度优先**: 对处理速度有极高要求的场景
- 🏎️ **计算密集**: CPU密集型工作负载
### 配置优化建议
#### 针对数据大小的配置
**小文件为主** (< 64KB):
```toml
# 推荐使用默认纯 Erasure 模式
# 无需特殊配置，性能稳定可靠
```
**大文件为主** (> 1MB):
```toml
# 建议启用SIMD模式获得更高性能
# features = ["reed-solomon-simd"]
```
**混合场景**:
```toml
# 默认纯 Erasure 模式适合大多数场景
# 如需最大性能可启用: features = ["reed-solomon-simd"]
```
#### 针对纠删码配置的建议
| 配置 | 小数据 (< 64KB) | 大数据 (> 1MB) | 推荐模式 |
|------|----------------|----------------|----------|
| 4+2 | 纯 Erasure | 纯 Erasure / SIMD模式 | 纯 Erasure（默认） |
| 6+3 | 纯 Erasure | 纯 Erasure / SIMD模式 | 纯 Erasure（默认） |
| 8+4 | 纯 Erasure | 纯 Erasure / SIMD模式 | 纯 Erasure（默认） |
| 10+5 | 纯 Erasure | 纯 Erasure / SIMD模式 | 纯 Erasure（默认） |
### 生产环境部署建议
#### 1️⃣ 默认部署策略
```bash
# 生产环境推荐配置：使用纯 Erasure 模式（默认）
cargo build --release
```
**优势**:
- ✅ 最大兼容性：处理任意大小数据
- ✅ 稳定可靠：成熟的实现，行为可预测
- ✅ 零配置：无需复杂的性能调优
- ✅ 内存高效：优化的内存使用模式
#### 2️⃣ 高性能部署策略
```bash
# 高性能场景：启用SIMD模式
cargo build --release --features reed-solomon-simd
```
**优势**:
- ✅ 最优性能：SIMD指令集优化
- ✅ 高吞吐量：适合大数据处理
- ✅ 性能导向：专注于最大化处理速度
- ✅ 现代硬件：充分利用现代CPU特性
#### 2️⃣ 监控和调优
```rust
// 根据具体场景选择合适的实现
match data_size {
size if size > 1024 * 1024 => {
// 大数据：考虑使用SIMD模式
println!("Large data detected, SIMD mode recommended");
}
_ => {
// 一般情况：使用默认Erasure模式
println!("Using default Erasure mode");
}
}
```
#### 3️⃣ 性能监控指标
- **吞吐量监控**: 监控编码/解码的数据处理速率
- **延迟分析**: 分析不同数据大小的处理延迟
- **CPU使用率**: 观察SIMD指令的CPU利用效率
- **内存使用**: 监控不同实现的内存分配模式
## 🔧 故障排除
### 性能问题诊断
#### 问题1: 性能不符合预期
**现象**: SIMD模式性能提升不明显
**原因**: 可能数据大小不适合SIMD优化
**解决**:
```rust
// 检查分片大小和数据特征
let shard_size = data.len().div_ceil(data_shards);
println!("Shard size: {} bytes", shard_size);
if shard_size >= 1024 {
println!("Good candidate for SIMD optimization");
} else {
println!("Consider using default Erasure mode");
}
```
#### 问题2: 编译错误
**现象**: SIMD相关的编译错误
**原因**: 平台不支持或依赖缺失
**解决**:
```bash
# 检查平台支持
cargo check --features reed-solomon-simd
# 如果失败，使用默认模式
cargo check
```
#### 问题3: 内存使用异常
**现象**: 内存使用超出预期
**原因**: SIMD实现的内存对齐要求
**解决**:
```bash
# 使用纯 Erasure 模式进行对比
cargo run --features reed-solomon-erasure
```
### 调试技巧
#### 1️⃣ 性能对比测试
```bash
# 测试纯 Erasure 模式性能
cargo bench --features reed-solomon-erasure
# 测试SIMD模式性能
cargo bench --features reed-solomon-simd
```
#### 2️⃣ 分析数据特征
```rust
// 统计你的应用中的数据特征
let data_sizes: Vec<usize> = data_samples.iter()
.map(|data| data.len())
.collect();
let large_data_count = data_sizes.iter()
.filter(|&&size| size >= 1024 * 1024)
.count();
println!("Large data (>1MB): {}/{} ({}%)",
large_data_count,
data_sizes.len(),
large_data_count * 100 / data_sizes.len()
);
```
#### 3️⃣ 基准测试对比
```bash
# 生成详细的性能对比报告
./run_benchmarks.sh comparison
# 查看 HTML 报告分析性能差异
cd target/criterion && python3 -m http.server 8080
```
## 📈 性能优化建议
### 应用层优化
#### 1️⃣ 数据分块策略
```rust
// 针对SIMD模式优化数据分块
const OPTIMAL_BLOCK_SIZE: usize = 1024 * 1024; // 1MB
const MIN_EFFICIENT_SIZE: usize = 64 * 1024; // 64KB
let block_size = if data.len() < MIN_EFFICIENT_SIZE {
data.len() // 小数据可以考虑默认模式
} else {
OPTIMAL_BLOCK_SIZE.min(data.len()) // 使用最优块大小
};
```
#### 2️⃣ 配置调优
```rust
// 根据典型数据大小选择纠删码配置
let (data_shards, parity_shards) = if typical_file_size > 1024 * 1024 {
(8, 4) // 大文件：更多并行度，利用 SIMD
} else {
(4, 2) // 小文件：简单配置，减少开销
};
```
### 系统层优化
#### 1️⃣ CPU 特性检测
```bash
# 检查 CPU 支持的 SIMD 指令集
lscpu | grep -i flags
cat /proc/cpuinfo | grep -i flags | head -1
```
#### 2️⃣ 内存对齐优化
```rust
// 确保数据内存对齐以提升 SIMD 性能
use aligned_vec::AlignedVec;
let aligned_data = AlignedVec::<u8, aligned_vec::A64>::from_slice(&data);
```
---
💡 **关键结论**:
- 🎯 **纯Erasure模式（默认）是最佳通用选择**：稳定可靠，适合大多数场景
- 🚀 **SIMD模式适合高性能场景**：大数据处理的最佳选择
- 📊 **根据数据特征选择**：小数据用Erasure，大数据考虑SIMD
- 🛡️ **稳定性优先**：生产环境建议使用默认Erasure模式