gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-20 03:22:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf(ecstore): improve erasure write diagnostics and single-block performance (#3280)

Browse Source 
* docs(object-capacity): add localized crate docs

* fix(ecstore): improve quorum and transport diagnostics

* perf(ecstore): add safe single-block write fast path

* refactor(ecstore): collapse layered small write paths

* chore(docs): keep issue 662 design note tracked

* fix(docs): restore issue 662 design note

* chore(docs): keep issue 662 design local only

* feat(obs): add internode reliability metrics and dashboard

* feat(obs): extend internode diagnostics and service logging

* fix(docs): use AGENTS guide filename

* perf(ecstore): reuse owned buffer in small encode

* fix(ecstore): tighten small write diagnostics

---------

Co-authored-by: cxymds <Cxymds@qq.com>
This commit is contained in:
houseme
2026-06-08 17:45:56 +08:00
committed by GitHub
parent ddd35badad
commit ed73952cb6
16 changed files with 2455 additions and 123 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/object-capacity/README.md
+353
View File
@@ -0,0 +1,353 @@
# rustfs-object-capacity
`rustfs-object-capacity` is the core object-capacity statistics component in RustFS. It scans local data directories, maintains a capacity cache, triggers incremental refreshes after writes, and provides the admin layer with a used-capacity result that is as inexpensive and resilient as possible.
This crate is not meant to measure total filesystem capacity. Its job is to answer: "How many bytes are currently occupied by RustFS object data?" It makes practical tradeoffs between accuracy, freshness, and scan cost.
## Core Responsibilities
- Scan one or more local data-disk roots and aggregate used bytes and file counts.
- Reduce scan cost on large directories with an "exact prefix + sampled overflow" strategy.
- Return usable degraded results when scans time out, traversal stalls, or some directories fail, instead of failing the entire request immediately.
- Maintain a global `HybridCapacityManager` cache with scheduled refresh, write-triggered refresh, foreground blocking refresh, and background refresh.
- Track which disks were affected by writes so the system can refresh only the dirty subset after a complete per-disk cache is available.
- Emit capacity-related metrics for observability and benchmarks.
## Module Layout
- `src/lib.rs`
Re-exports `scan_used_capacity_disks`, `CapacityDiskRef`, and `CapacityScanSummary`.
- `src/types.rs`
Defines scan input/output types, including `CapacityDiskRef`, the internal `CapacityScanResult`, and the public `CapacityScanSummary`.
- `src/scan.rs`
Implements directory traversal, sampled estimation, timeout/stall detection, multi-disk concurrent scans, and conversion into `CapacityUpdate`.
- `src/capacity_manager.rs`
Owns caching, write-frequency tracking, singleflight refresh coordination, background tasks, dirty-subset merge logic, and the global singleton manager.
- `src/capacity_scope.rs`
Tracks "which disks were touched by a write", including token-bound local scopes and the global dirty-scope registry.
- `benches/capacity_scan.rs`
Exercises the public scan API with benchmark scenarios for exact, sampled, and multi-disk scans.
## Data Model
### `CapacityDiskRef`
```rust
pub struct CapacityDiskRef {
pub endpoint: String,
pub drive_path: String,
}
```
This is the minimal unit required for a scan:
- `endpoint` is used to distinguish metrics and logs.
- `drive_path` is the local disk root path.
### `CapacityScanSummary`
```rust
pub struct CapacityScanSummary {
pub used_bytes: u64,
pub file_count: usize,
pub sampled_count: usize,
pub is_estimated: bool,
pub had_partial_errors: bool,
pub scan_duration: Duration,
}
```
Field meanings:
- `used_bytes`: the computed or estimated used capacity.
- `file_count`: the number of regular files traversed.
- `sampled_count`: the number of overflow files sampled after crossing the threshold.
- `is_estimated`: whether the result is estimated instead of exact.
- `had_partial_errors`: whether traversal encountered local errors while still producing a result.
- `scan_duration`: total scan duration.
## Scan Algorithm
The directory scan lives in `scan.rs::get_dir_size_async` and works as follows:
1. Wrap blocking directory traversal in `tokio::task::spawn_blocking` so the async runtime is not blocked.
2. Walk the directory tree with `WalkDir` and count only regular files.
3. If the file count stays below `DEFAULT_MAX_FILES_THRESHOLD` (default `200_000`), add every file size exactly.
4. After crossing the threshold:
- Keep the first `max_files_threshold` files as an exact prefix.
- Sample every `sample_rate` file after that and estimate the overflow portion from sampled bytes.
5. Periodically perform progress checks:
- If total elapsed time exceeds the timeout, attempt to fall back to a sampled estimate.
- If no file progress is observed within `stall_timeout`, treat the traversal as stalled.
6. If some directory entries or metadata reads fail:
- As long as at least one disk scan succeeds, return a partial-success result.
- Mark the result with `had_partial_errors = true`.
### Scan Concurrency
- Multi-disk scans run concurrently through `buffer_unordered`.
- The current hard-coded maximum concurrency is `4` disks.
- A failure on one disk does not immediately stop scans for the others.
### Timeout and Estimation Fallback
This crate is intentionally not "timeout means hard failure":
- If enough sampled data has already been collected, a timeout or stall produces an estimated result.
- Only when no usable estimate is available does the scan return an error.
- This keeps capacity queries useful for large directories, slow disks, and temporary I/O stalls.
### Symlink Handling
- Symlinks are not followed by default: `RUSTFS_CAPACITY_FOLLOW_SYMLINKS=false`.
- If enabled, the scan applies circular-reference detection and a maximum follow depth.
- The default maximum depth is `3`.
## Capacity Cache and Refresh Strategy
`HybridCapacityManager` is the state center of this crate.
### Cached State
- Latest total capacity value `total_used`
- Last refresh time `last_update`
- File count `file_count`
- Estimated/exact flag `is_estimated`
- Data source `DataSource`
- Per-disk cache `disk_cache`
- Dirty-disk set
- Recent 60-second write buckets
### `DataSource`
- `RealTime`
Foreground real-time refresh when no cache exists yet.
- `Scheduled`
Background refresh triggered by the scheduled task.
- `WriteTriggered`
Refresh triggered when write frequency is high and the cache is old enough.
- `Fallback`
Fallback to externally supplied disk-used capacity when all scans fail.
### Refresh Entry Points
- `refresh_or_join`
A singleflight foreground refresh. If another refresh is already running, callers join and wait for the shared result.
- `spawn_refresh_if_needed`
A background refresh. If another refresh is already running, it is skipped.
- `start_background_task`
Starts two background tasks:
- the scheduled capacity refresh task
- the runtime summary logging task
### Singleflight Semantics
`refresh_or_join` and `spawn_refresh_if_needed` use a `watch` channel to coordinate refresh cycles:
- Only one leader performs the actual refresh at a time.
- Joiners share the same published result after the leader completes.
- Panics inside the refresh function are caught and converted into errors so callers do not crash with the leader.
## Dirty Scope and Subset Refresh
One of the main optimizations in this crate is "refresh only the disks dirtied by writes".
### Scope Propagation
`capacity_scope.rs` provides two ways to propagate dirty disks:
- token scope
- The caller first binds a write operation to a disk set with `record_capacity_scope(token, scope)`.
- Later, `record_write_operation_with_scope_token(Some(token))` consumes that scope and marks the disks dirty.
- global dirty scope
- `record_global_dirty_scope(scope)` records dirty disks directly in the global registry.
- The manager drains and merges them during `get_dirty_disks()`.
### When Dirty-Subset Refresh Is Allowed
Refreshing only dirty disks is safe only when:
- `disk_cache_complete == true`
- which means the system has already completed at least one full refresh without partial errors
- and the per-disk cache is fully populated
If the per-disk cache is incomplete, or there are no dirty disks, the system falls back to a full refresh.
### Merge Rules After a Subset Refresh
- On a successful full refresh, `per_disk` replaces the entire `disk_cache`.
- On a successful dirty-subset refresh, only the affected per-disk entries are updated.
- The total capacity is recomputed from the updated `disk_cache` instead of trusting the subset sum directly.
- If a dirty-subset refresh reports partial errors, that cycle fails and the caller falls back to a full refresh to recover consistency.
## Relationship to the RustFS Main Flow
This crate provides capacity primitives only. The actual RustFS integration lives in `rustfs/src/capacity/service.rs`.
The high-level flow is:
1. Startup calls `init_capacity_management_for_local_disks()`.
2. It collects all local disks and calls `capacity_manager::start_background_task(...)`.
3. Admin used-capacity queries first try the `HybridCapacityManager` cache.
4. If the cache is fresh enough, the cached value is returned directly.
5. If the cache is stale but still acceptable, the stale value is served and a background refresh is triggered.
6. If the cache is very stale and the write rate is high, the request blocks on a foreground refresh.
7. If the initial real-time scan fails, the service falls back to externally supplied disk-used capacity and stores it as `Fallback`.
`crates/ecstore/src/set_disk.rs` is responsible for recording capacity scopes during object writes, heal operations, data movement, and related flows, so this crate can learn which disks were affected.
## Public API
### 1. Direct Scan
This is useful for benchmarks, operational tooling, or isolated validation.
```rust
use rustfs_object_capacity::{CapacityDiskRef, scan_used_capacity_disks};
let disks = vec![
CapacityDiskRef {
endpoint: "node-a".to_string(),
drive_path: "/data/disk1".to_string(),
},
];
let summary = scan_used_capacity_disks(&disks).await?;
println!(
"used={} files={} estimated={}",
summary.used_bytes, summary.file_count, summary.is_estimated
);
# Ok::<(), Box<dyn std::error::Error>>(())
```
### 2. Use the Global Manager
This is useful for in-service caching and refresh orchestration.
```rust
use rustfs_object_capacity::capacity_manager::{DataSource, get_capacity_manager};
let manager = get_capacity_manager();
if let Some(cached) = manager.get_capacity().await {
println!("cached bytes={}", cached.total_used);
}
manager.record_write_operation().await;
let _ = manager
.refresh_or_join(DataSource::Scheduled, || async {
rustfs_object_capacity::scan::refresh_capacity_with_scope(
vec![rustfs_object_capacity::CapacityDiskRef {
endpoint: "node-a".to_string(),
drive_path: "/data/disk1".to_string(),
}],
false,
)
.await
})
.await;
```
### 3. Propagate a Dirty Scope
```rust
use rustfs_object_capacity::capacity_scope::{
CapacityScope, CapacityScopeDisk, record_capacity_scope,
};
use rustfs_object_capacity::capacity_manager::get_capacity_manager;
use uuid::Uuid;
let token = Uuid::new_v4();
record_capacity_scope(
token,
CapacityScope {
disks: vec![CapacityScopeDisk {
endpoint: "node-a".to_string(),
drive_path: "/data/disk1".to_string(),
}],
},
);
get_capacity_manager()
.record_write_operation_with_scope_token(Some(token))
.await;
```
## Environment Variables and Defaults
The configuration constants are defined in `crates/config/src/constants/capacity.rs`.
| Environment Variable | Default | Description |
| --- | --- | --- |
| `RUSTFS_CAPACITY_SCHEDULED_INTERVAL` | `120s` | Scheduled refresh interval |
| `RUSTFS_CAPACITY_WRITE_TRIGGER_DELAY` | `5s` | Debounce delay after writes |
| `RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD` | `5` | Recent 60-second write-frequency threshold |
| `RUSTFS_CAPACITY_FAST_UPDATE_THRESHOLD` | `30s` | Cache age required before fast refresh is considered |
| `RUSTFS_CAPACITY_MAX_FILES_THRESHOLD` | `200000` | Exact-count file threshold |
| `RUSTFS_CAPACITY_STAT_TIMEOUT` | `3s` | Base scan timeout |
| `RUSTFS_CAPACITY_SAMPLE_RATE` | `200` | Overflow-file sampling interval |
| `RUSTFS_CAPACITY_METRICS_INTERVAL` | `600s` | Runtime summary emission interval |
| `RUSTFS_CAPACITY_FOLLOW_SYMLINKS` | `false` | Whether to follow symlinks |
| `RUSTFS_CAPACITY_MAX_SYMLINK_DEPTH` | `3` | Maximum symlink follow depth |
| `RUSTFS_CAPACITY_ENABLE_DYNAMIC_TIMEOUT` | `true` | Whether to enable dynamic timeout scaling |
| `RUSTFS_CAPACITY_MIN_TIMEOUT` | `2s` | Dynamic-timeout lower bound |
| `RUSTFS_CAPACITY_MAX_TIMEOUT` | `15s` | Dynamic-timeout upper bound |
| `RUSTFS_CAPACITY_STALL_TIMEOUT` | `20s` | Stall-detection threshold |
### Configuration-Caching Note
In non-test builds, configuration is cached behind `OnceLock`:
- Environment variables are effectively read once on first access.
- Updating `RUSTFS_CAPACITY_*` during runtime usually does not take effect immediately.
- A process restart is normally required to apply configuration changes reliably.
## Metrics
This crate reports multiple metric families to `rustfs-io-metrics::capacity_metrics`, including:
- cache hit / miss / served state
- refresh inflight, joiners, and success / error outcomes
- current capacity bytes
- write frequency
- dirty-disk count
- per-disk scan duration, sampling mode, timeout fallback, stall detection, and symlink statistics
So this crate is both a capacity-calculation component and an important producer of runtime observability data.
## Benchmarks
Run the benchmark suite with:
```bash
cargo bench -p rustfs-object-capacity --bench capacity_scan
```
Current benchmark scenarios:
- `capacity_scan_exact`
Single-disk exact scan over 10k files.
- `capacity_scan_sampled`
Single-disk scan over 202,048 files that triggers sampled estimation.
- `capacity_scan_multi_disk`
Four-disk exact scan with mixed directory sizes.
## Known Boundaries and Tradeoffs
- It sums file sizes under RustFS object-data directories; it is not a full replacement for filesystem-level `du`.
- Estimated mode prioritizes bounded cost and usable results over perfect per-run precision.
- Dirty-subset refresh is safe only after a complete per-disk cache has been established.
- Partial errors intentionally try to return a degraded result, which improves availability but means callers should pay attention to `had_partial_errors`.
- Symlink following is disabled by default for safety and determinism.
## Relevant Source Entry Points
- [src/lib.rs](./src/lib.rs)
- [src/scan.rs](./src/scan.rs)
- [src/capacity_manager.rs](./src/capacity_manager.rs)
- [src/capacity_scope.rs](./src/capacity_scope.rs)
- [src/types.rs](./src/types.rs)
- [benches/capacity_scan.rs](./benches/capacity_scan.rs)
- [../../rustfs/src/capacity/service.rs](../../rustfs/src/capacity/service.rs)
- [../config/src/constants/capacity.rs](../config/src/constants/capacity.rs)