mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-19 19:16:17 +00:00
junxiang Mu
557 lines
16 KiB
Markdown
557 lines
16 KiB
Markdown
# RustFS Advanced Health & Metrics (AHM) System Architecture
|
|
|
|
## Overview
|
|
|
|
The RustFS AHM system is a newly designed distributed storage health monitoring and repair system that provides intelligent scanning, automatic repair, rich metrics, and policy-driven management capabilities.
|
|
|
|
## System Architecture
|
|
|
|
### Overall Architecture Diagram
|
|
|
|
```
|
|
┌─────────────────────────────────────┐
|
|
│ API Layer (REST/gRPC) │
|
|
├─────────────────────────────────────┤
|
|
│ Policy & Configuration │
|
|
├─────────────────────────────────────┤
|
|
│ Core Coordination Engine │
|
|
├─────────────────────────────────────┤
|
|
│ Scanner Engine │ Heal Engine │
|
|
├─────────────────────────────────────┤
|
|
│ Metrics & Observability │
|
|
├─────────────────────────────────────┤
|
|
│ Storage Abstraction │
|
|
└─────────────────────────────────────┘
|
|
```
|
|
|
|
### Module Structure
|
|
|
|
```
|
|
rustfs/crates/ecstore/src/ahm/
|
|
├── mod.rs # Module entry point and public interfaces
|
|
├── core/ # Core engines
|
|
│ ├── coordinator.rs # Distributed coordinator - event routing and state management
|
|
│ ├── scheduler.rs # Task scheduler - priority queue and work assignment
|
|
│ └── lifecycle.rs # Lifecycle manager - system startup/shutdown control
|
|
├── scanner/ # Scanning system
|
|
│ ├── engine.rs # Scan engine - scan process control
|
|
│ ├── object_scanner.rs # Object scanner - object-level integrity checks
|
|
│ ├── disk_scanner.rs # Disk scanner - disk-level health checks
|
|
│ ├── metrics_collector.rs # Metrics collector - scan process data collection
|
|
│ └── bandwidth_limiter.rs # Bandwidth limiter - I/O resource control
|
|
├── heal/ # Repair system
|
|
│ ├── engine.rs # Heal engine - repair process control
|
|
│ ├── priority_queue.rs # Priority queue - repair task ordering
|
|
│ ├── repair_worker.rs # Repair worker - actual repair execution
|
|
│ └── validation.rs # Repair validator - repair result verification
|
|
├── metrics/ # Metrics system
|
|
│ ├── collector.rs # Metrics collector - real-time data collection
|
|
│ ├── aggregator.rs # Metrics aggregator - data aggregation and computation
|
|
│ ├── storage.rs # Metrics storage - time-series data storage
|
|
│ └── reporter.rs # Metrics reporter - external system export
|
|
├── policy/ # Policy system
|
|
│ ├── scan_policy.rs # Scan policy - scan behavior configuration
|
|
│ ├── heal_policy.rs # Heal policy - repair priority and strategy
|
|
│ └── retention_policy.rs # Retention policy - data lifecycle management
|
|
└── api/ # API interfaces
|
|
├── admin_api.rs # Admin API - system management operations
|
|
├── metrics_api.rs # Metrics API - metrics query and export
|
|
└── status_api.rs # Status API - system status monitoring
|
|
```
|
|
|
|
## Core Design Principles
|
|
|
|
### 1. Event-Driven Architecture
|
|
|
|
```rust
|
|
pub enum SystemEvent {
|
|
ObjectDiscovered { bucket: String, object: String, metadata: ObjectMetadata },
|
|
HealthIssueDetected { issue_type: HealthIssueType, severity: Severity },
|
|
HealCompleted { result: HealResult },
|
|
ScanCycleCompleted { statistics: ScanStatistics },
|
|
ResourceUsageUpdated { usage: ResourceUsage },
|
|
}
|
|
```
|
|
|
|
- **Scanner** generates discovery events
|
|
- **Heal** responds to repair events
|
|
- **Metrics** collects all event statistics
|
|
- **Policy** controls event processing strategies
|
|
|
|
### 2. Layered Modular Design
|
|
|
|
#### **API Layer**: REST/gRPC interfaces
|
|
- Unified response format
|
|
- Comprehensive error handling
|
|
- Authentication and authorization support
|
|
|
|
#### **Policy Layer**: Configurable business rules
|
|
- Scan frequency and depth control
|
|
- Repair priority policies
|
|
- Data retention rules
|
|
|
|
#### **Coordination Layer**: System coordination and scheduling
|
|
- Event routing and distribution
|
|
- Resource management and allocation
|
|
- Task scheduling and execution
|
|
|
|
#### **Engine Layer**: Core business logic
|
|
- Intelligent scanning algorithms
|
|
- Adaptive repair strategies
|
|
- Performance optimization control
|
|
|
|
#### **Metrics Layer**: Observability support
|
|
- Real-time metrics collection
|
|
- Historical trend analysis
|
|
- Multi-format export
|
|
|
|
### 3. Multi-Mode Scanning Strategies
|
|
|
|
```rust
|
|
pub enum ScanStrategy {
|
|
Full { mode: ScanMode, scope: ScanScope }, // Full scan
|
|
Incremental { since: Instant, mode: ScanMode }, // Incremental scan
|
|
Smart { sample_rate: f64, favor_unscanned: bool }, // Smart sampling
|
|
Targeted { targets: Vec<ObjectTarget>, mode: ScanMode }, // Targeted scan
|
|
}
|
|
|
|
pub enum ScanMode {
|
|
Quick, // Quick scan - metadata only
|
|
Normal, // Normal scan - basic integrity verification
|
|
Deep, // Deep scan - includes bit-rot detection
|
|
}
|
|
```
|
|
|
|
### 4. Priority-Based Repair System
|
|
|
|
```rust
|
|
pub enum HealPriority {
|
|
Low = 0,
|
|
Normal = 1,
|
|
High = 2,
|
|
Critical = 3,
|
|
Emergency = 4,
|
|
}
|
|
|
|
pub enum HealMode {
|
|
RealTime, // Real-time repair - triggered on GET/PUT
|
|
Background, // Background repair - scheduled tasks
|
|
OnDemand, // On-demand repair - admin triggered
|
|
Emergency, // Emergency repair - critical issues
|
|
}
|
|
```
|
|
|
|
## API Usage Guide
|
|
|
|
### 1. System Management API
|
|
|
|
#### Start AHM System
|
|
|
|
```http
|
|
POST /admin/system/start
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"coordinator": {
|
|
"event_buffer_size": 10000,
|
|
"max_concurrent_operations": 1000
|
|
},
|
|
"scanner": {
|
|
"default_scan_mode": "Normal",
|
|
"scan_interval": "24h"
|
|
},
|
|
"heal": {
|
|
"max_workers": 16,
|
|
"queue_capacity": 50000
|
|
}
|
|
}
|
|
```
|
|
|
|
**Response Example:**
|
|
```json
|
|
{
|
|
"success": true,
|
|
"data": {
|
|
"system_id": "ahm-001",
|
|
"status": "Running",
|
|
"started_at": "2024-01-15T10:30:00Z"
|
|
},
|
|
"timestamp": "2024-01-15T10:30:00Z"
|
|
}
|
|
```
|
|
|
|
#### Get System Status
|
|
|
|
```http
|
|
GET /status/health
|
|
```
|
|
|
|
**Response Example:**
|
|
```json
|
|
{
|
|
"success": true,
|
|
"data": {
|
|
"status": "Running",
|
|
"version": "1.0.0",
|
|
"uptime_seconds": 3600,
|
|
"subsystems": {
|
|
"scanner": {
|
|
"status": "Scanning",
|
|
"last_check": "2024-01-15T10:29:00Z",
|
|
"error_message": null
|
|
},
|
|
"heal": {
|
|
"status": "Idle",
|
|
"last_check": "2024-01-15T10:29:00Z",
|
|
"error_message": null
|
|
},
|
|
"metrics": {
|
|
"status": "Running",
|
|
"last_check": "2024-01-15T10:29:00Z",
|
|
"error_message": null
|
|
}
|
|
}
|
|
},
|
|
"timestamp": "2024-01-15T10:30:00Z"
|
|
}
|
|
```
|
|
|
|
### 2. Scan Management API
|
|
|
|
#### Start Scan Task
|
|
|
|
```http
|
|
POST /admin/scan/start
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"strategy": {
|
|
"type": "Full",
|
|
"mode": "Normal",
|
|
"scope": {
|
|
"buckets": ["important-data", "user-uploads"],
|
|
"include_system_objects": false,
|
|
"max_objects": 1000000
|
|
}
|
|
},
|
|
"priority": "High"
|
|
}
|
|
```
|
|
|
|
**Response Example:**
|
|
```json
|
|
{
|
|
"success": true,
|
|
"data": {
|
|
"scan_id": "scan-12345",
|
|
"status": "Started",
|
|
"estimated_duration": "2h30m",
|
|
"estimated_objects": 850000
|
|
},
|
|
"timestamp": "2024-01-15T10:30:00Z"
|
|
}
|
|
```
|
|
|
|
#### Query Scan Status
|
|
|
|
```http
|
|
GET /admin/scan/{scan_id}/status
|
|
```
|
|
|
|
**Response Example:**
|
|
```json
|
|
{
|
|
"success": true,
|
|
"data": {
|
|
"scan_id": "scan-12345",
|
|
"status": "Scanning",
|
|
"progress": {
|
|
"objects_scanned": 425000,
|
|
"bytes_scanned": 1073741824000,
|
|
"issues_detected": 23,
|
|
"completion_percentage": 50.0,
|
|
"scan_rate_ops": 117.5,
|
|
"scan_rate_bps": 268435456,
|
|
"elapsed_time": "1h15m",
|
|
"estimated_remaining": "1h15m"
|
|
},
|
|
"issues": [
|
|
{
|
|
"issue_type": "MissingShards",
|
|
"severity": "High",
|
|
"bucket": "user-uploads",
|
|
"object": "photos/IMG_001.jpg",
|
|
"description": "Missing 1 data shard",
|
|
"detected_at": "2024-01-15T11:15:00Z"
|
|
}
|
|
]
|
|
},
|
|
"timestamp": "2024-01-15T11:45:00Z"
|
|
}
|
|
```
|
|
|
|
### 3. Heal Management API
|
|
|
|
#### Submit Heal Request
|
|
|
|
```http
|
|
POST /admin/heal/request
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"bucket": "user-uploads",
|
|
"object": "photos/IMG_001.jpg",
|
|
"version_id": null,
|
|
"priority": "High",
|
|
"mode": "OnDemand",
|
|
"max_retries": 3
|
|
}
|
|
```
|
|
|
|
**Response Example:**
|
|
```json
|
|
{
|
|
"success": true,
|
|
"data": {
|
|
"heal_request_id": "heal-67890",
|
|
"status": "Queued",
|
|
"priority": "High",
|
|
"estimated_start": "2024-01-15T11:50:00Z",
|
|
"queue_position": 5
|
|
},
|
|
"timestamp": "2024-01-15T11:45:00Z"
|
|
}
|
|
```
|
|
|
|
#### Query Heal Status
|
|
|
|
```http
|
|
GET /admin/heal/{heal_request_id}/status
|
|
```
|
|
|
|
**Response Example:**
|
|
```json
|
|
{
|
|
"success": true,
|
|
"data": {
|
|
"heal_request_id": "heal-67890",
|
|
"status": "Completed",
|
|
"result": {
|
|
"success": true,
|
|
"shards_repaired": 1,
|
|
"total_shards": 8,
|
|
"duration": "45s",
|
|
"strategy_used": "ParityShardRepair",
|
|
"validation_results": [
|
|
{
|
|
"validation_type": "Checksum",
|
|
"passed": true,
|
|
"details": "Object checksum verified",
|
|
"duration": "2s"
|
|
},
|
|
{
|
|
"validation_type": "ShardCount",
|
|
"passed": true,
|
|
"details": "All 8 shards present",
|
|
"duration": "1s"
|
|
}
|
|
]
|
|
}
|
|
},
|
|
"timestamp": "2024-01-15T11:46:00Z"
|
|
}
|
|
```
|
|
|
|
### 4. Metrics Query API
|
|
|
|
#### Get System Metrics
|
|
|
|
```http
|
|
GET /metrics/system?period=1h&metrics=objects_total,scan_rate,heal_success_rate
|
|
```
|
|
|
|
**Response Example:**
|
|
```json
|
|
{
|
|
"success": true,
|
|
"data": {
|
|
"period": "1h",
|
|
"timestamp_range": {
|
|
"start": "2024-01-15T10:45:00Z",
|
|
"end": "2024-01-15T11:45:00Z"
|
|
},
|
|
"metrics": {
|
|
"objects_total": {
|
|
"value": 2500000,
|
|
"unit": "count",
|
|
"labels": {}
|
|
},
|
|
"scan_rate_objects_per_second": {
|
|
"value": 117.5,
|
|
"unit": "ops",
|
|
"labels": {}
|
|
},
|
|
"heal_success_rate": {
|
|
"value": 0.98,
|
|
"unit": "ratio",
|
|
"labels": {}
|
|
}
|
|
}
|
|
},
|
|
"timestamp": "2024-01-15T11:45:00Z"
|
|
}
|
|
```
|
|
|
|
#### Export Prometheus Format Metrics
|
|
|
|
```http
|
|
GET /metrics/prometheus
|
|
```
|
|
|
|
**Response Example:**
|
|
```
|
|
# HELP rustfs_objects_total Total number of objects in the system
|
|
# TYPE rustfs_objects_total gauge
|
|
rustfs_objects_total 2500000
|
|
|
|
# HELP rustfs_scan_rate_objects_per_second Object scanning rate
|
|
# TYPE rustfs_scan_rate_objects_per_second gauge
|
|
rustfs_scan_rate_objects_per_second 117.5
|
|
|
|
# HELP rustfs_heal_success_rate Healing operation success rate
|
|
# TYPE rustfs_heal_success_rate gauge
|
|
rustfs_heal_success_rate 0.98
|
|
|
|
# HELP rustfs_health_issues_total Total health issues detected
|
|
# TYPE rustfs_health_issues_total counter
|
|
rustfs_health_issues_total{severity="critical"} 0
|
|
rustfs_health_issues_total{severity="high"} 3
|
|
rustfs_health_issues_total{severity="medium"} 15
|
|
rustfs_health_issues_total{severity="low"} 45
|
|
```
|
|
|
|
### 5. Policy Configuration API
|
|
|
|
#### Update Scan Policy
|
|
|
|
```http
|
|
PUT /admin/policy/scan
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"default_scan_interval": "12h",
|
|
"deep_scan_probability": 0.1,
|
|
"bandwidth_limit_mbps": 100,
|
|
"concurrent_scanners": 4,
|
|
"skip_system_objects": true,
|
|
"priority_buckets": ["critical-data", "user-data"]
|
|
}
|
|
```
|
|
|
|
#### Update Heal Policy
|
|
|
|
```http
|
|
PUT /admin/policy/heal
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"max_concurrent_heals": 8,
|
|
"emergency_heal_timeout": "5m",
|
|
"auto_heal_enabled": true,
|
|
"heal_verification_required": true,
|
|
"priority_mapping": {
|
|
"critical_buckets": "Emergency",
|
|
"important_buckets": "High",
|
|
"standard_buckets": "Normal"
|
|
}
|
|
}
|
|
```
|
|
|
|
## Usage Examples
|
|
|
|
### Complete Monitoring and Repair Workflow
|
|
|
|
```bash
|
|
# 1. Start AHM system
|
|
curl -X POST http://localhost:9000/admin/system/start \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"scanner": {"default_scan_mode": "Normal"}}'
|
|
|
|
# 2. Start full scan
|
|
SCAN_ID=$(curl -X POST http://localhost:9000/admin/scan/start \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"strategy": {"type": "Full", "mode": "Normal"}}' | \
|
|
jq -r '.data.scan_id')
|
|
|
|
# 3. Monitor scan progress
|
|
watch "curl -s http://localhost:9000/admin/scan/$SCAN_ID/status | jq '.data.progress'"
|
|
|
|
# 4. View discovered issues
|
|
curl -s http://localhost:9000/admin/scan/$SCAN_ID/status | \
|
|
jq '.data.issues[]'
|
|
|
|
# 5. Start repair for discovered issues
|
|
HEAL_ID=$(curl -X POST http://localhost:9000/admin/heal/request \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"bucket": "user-uploads",
|
|
"object": "photos/IMG_001.jpg",
|
|
"priority": "High"
|
|
}' | jq -r '.data.heal_request_id')
|
|
|
|
# 6. Monitor repair progress
|
|
watch "curl -s http://localhost:9000/admin/heal/$HEAL_ID/status | jq '.data'"
|
|
|
|
# 7. View system metrics
|
|
curl -s http://localhost:9000/metrics/system?period=1h | jq '.data.metrics'
|
|
|
|
# 8. Export Prometheus metrics
|
|
curl -s http://localhost:9000/metrics/prometheus
|
|
```
|
|
|
|
## Key Features
|
|
|
|
### 1. Intelligent Scanning
|
|
- **Multi-level scan modes**: Quick/Normal/Deep three depths
|
|
- **Adaptive sampling**: Intelligent object selection based on historical data
|
|
- **Bandwidth control**: Configurable I/O resource limits
|
|
- **Incremental scanning**: Timestamp-based change detection
|
|
|
|
### 2. Intelligent Repair
|
|
- **Priority queue**: Repair ordering based on business importance
|
|
- **Multiple repair strategies**: Data shard, parity shard, hybrid repair
|
|
- **Real-time validation**: Post-repair integrity verification
|
|
- **Retry mechanism**: Configurable failure retry policies
|
|
|
|
### 3. Rich Metrics
|
|
- **Real-time statistics**: Object counts, storage usage, performance metrics
|
|
- **Historical trends**: Time-series data storage and analysis
|
|
- **Multi-format export**: Prometheus, JSON, CSV formats
|
|
- **Custom metrics**: Extensible metrics definition framework
|
|
|
|
### 4. Policy-Driven
|
|
- **Configurable policies**: Independent configuration for scan, heal, retention policies
|
|
- **Dynamic adjustment**: Runtime policy updates without restart
|
|
- **Business alignment**: Differentiated handling based on business importance
|
|
|
|
## Deployment Recommendations
|
|
|
|
### 1. Resource Configuration
|
|
- **CPU**: Recommended 16+ cores for parallel scanning and repair
|
|
- **Memory**: Recommended 32GB+ for metrics cache and task queues
|
|
- **Network**: Recommended gigabit+ bandwidth for cross-node data sync
|
|
- **Storage**: Recommended SSD for metrics data storage
|
|
|
|
### 2. Monitoring Integration
|
|
- **Prometheus**: Metrics collection and alerting
|
|
- **Grafana**: Visualization dashboards
|
|
- **ELK Stack**: Log aggregation and analysis
|
|
- **Jaeger**: Distributed tracing
|
|
|
|
### 3. High Availability Deployment
|
|
- **Multi-instance deployment**: Avoid single points of failure
|
|
- **Load balancing**: API request distribution
|
|
- **Data backup**: Metrics and configuration data backup
|
|
- **Failover**: Automatic failure detection and switching
|
|
|
|
This architecture design provides RustFS with modern, scalable, and highly observable health monitoring and repair capabilities that meet the operational requirements of enterprise-grade distributed storage systems. |