Compare commits

...

9 Commits

Author SHA1 Message Date
overtrue bb68ff4fbd fix(ecstore): classify decommission stage failures by type, not by message
T2 of backlog#1827. `data_movement_stage_error` flattened every stage failure into `Error::other(format!(...))`, discarding the typed error. The cost was visible in tree: `is_decommission_target_capacity_error` had to match rendered text —

    let message = err.to_string();
    message.contains(&disk_full) || message.contains(&storage_full)

— to notice that the destination pool had filled up, and `is_decommission_copy_cleanup_safe_error` could not see a not-found that surfaced from inside a stage at all.

The wrapper now carries what it wrapped. `DataMovementStageError` renders the same string and returns the original through `source()`; `Error::other` boxes it through `std::io::Error`, so `data_movement_stage_source` recovers it by downcast. Both classifiers unwrap before matching, keeping their substring paths for errors that arrive through some other wrapper.

The rendered message is unchanged, which a test now pins against the exact string the old `format!` produced rather than against a `contains`. Three more cover the round trip for `DiskFull`, `StorageFull`, `FileNotFound` and `SlowDown`, that unrelated errors are not mistaken for stage wrappers, and — the case the issue names — that a not-found surfacing from inside a stage is judged cleanup-safe by the decommission loop exactly as a direct one is.

Refs backlog#1827
2026-08-19 18:38:37 +08:00
houseme 1f8359537b docs(operations): restore the truncated tail of the English audit baseline (#6268)
The merge of rustfs#6261 lost the last 64 lines of the English
translation: merging main (to pick up rustfs#6258) resolved the
conflict on the renamed file by cutting it mid-table in section 6,
which dropped section 7 (backlog/history index), section 8 (audit
method and limitations) and section 9 (landing results) that the
Chinese counterpart still carries. Restore them verbatim from the
translation commit (0e051602f) so both language versions are complete
568-line mirrors of the full 0-9 baseline, as the PR body promised.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-19 18:15:03 +08:00
cxymds d7609b68a6 fix(lock): reject stale lease snapshots (#6249) 2026-08-19 14:47:33 +08:00
houseme 3958781320 feat(io-metrics): attribute ReadVersion RPC stages (#6262) 2026-08-19 14:47:17 +08:00
Zhengchao An 5cb12300bc refactor(sse): keep one bucket-default algorithm mapping for every writer (#6251) 2026-08-19 14:30:04 +08:00
Zhengchao An bce0c05f3c chore(rustfs): adjudicate the remaining 36 bare dead_code allows (#6259) 2026-08-19 14:28:57 +08:00
houseme 07cef6789b feat(ecstore): expose rename sync tail metrics (#6257)
Add default-off PUT stage helpers for fdatasync batch shape and rename quorum fanout shape so #925 follow-up probes can distinguish shard sync batching opportunities from fanout convergence.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-19 14:26:43 +08:00
houseme 05e6dc5f4a docs(operations): add an English counterpart of the heal/scanner audit baseline (#6261)
* docs(operations): land the heal/scanner MinIO audit baseline with closure results

Move the comprehensive heal/scanner vs MinIO analysis (2026-08-16) into
docs/operations/ so it finally enters the tree — the docs/ root is
ignored by the gitignore whitelist, which is why the baseline the audit
issue referenced as "to be merged with a PR" never landed. Append §9
closure results: all 14 backlog sub-issues (#1865-#1878) closed with the
per-item PR map, two further misjudgment corrections (HS-17 was already
implemented; HS-14's MinIO idle semantics drifted upstream), HS-12/HS-18
audit conclusions, and the registered follow-ups.

Backlog issue: rustfs/backlog#1862

Co-Authored-By: heihutu <heihutu@gmail.com>

* docs(operations): add an English counterpart of the audit baseline

Rename the Chinese analysis to *_zh.md (matching the repo's bilingual
convention of scanner-excess-alerts.md / _zh.md) and add a full English
translation at the original path, cross-linked at the top of both files.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-19 06:10:34 +00:00
cxymds 6f3f2f5f62 test(lifecycle): cover noncurrent marker cleanup cascade (#6252) 2026-08-19 14:06:38 +08:00
44 changed files with 2151 additions and 665 deletions
+170
View File
@@ -168,6 +168,24 @@ async fn wait_for_version_expired(
} }
} }
async fn wait_for_key_versions_empty(client: &Client, bucket: &str, key: &str, deadline: StdDuration) -> TestResult {
let start = std::time::Instant::now();
loop {
let listing = client.list_object_versions().bucket(bucket).prefix(key).send().await?;
if listing.versions().is_empty() && listing.delete_markers().is_empty() {
return Ok(());
}
if start.elapsed() >= deadline {
return Err(format!(
"object {bucket}/{key} still had versions or delete markers after {}s: {listing:?}",
deadline.as_secs()
)
.into());
}
tokio::time::sleep(StdDuration::from_millis(500)).await;
}
}
/// Build a prefix-scoped `Days`-based expiration rule. /// Build a prefix-scoped `Days`-based expiration rule.
fn expiration_rule(id: &str, prefix: &str, days: i32) -> Result<LifecycleRule, Box<dyn std::error::Error + Send + Sync>> { fn expiration_rule(id: &str, prefix: &str, days: i32) -> Result<LifecycleRule, Box<dyn std::error::Error + Send + Sync>> {
let rule = LifecycleRule::builder() let rule = LifecycleRule::builder()
@@ -193,6 +211,21 @@ fn noncurrent_expiration_rule(
Ok(rule) Ok(rule)
} }
fn noncurrent_expiration_with_delete_marker_cleanup_rule(
id: &str,
prefix: &str,
days: i32,
) -> Result<LifecycleRule, Box<dyn std::error::Error + Send + Sync>> {
let rule = LifecycleRule::builder()
.id(id)
.filter(LifecycleRuleFilter::builder().prefix(prefix).build())
.expiration(LifecycleExpiration::builder().expired_object_delete_marker(true).build())
.noncurrent_version_expiration(NoncurrentVersionExpiration::builder().noncurrent_days(days).build())
.status(ExpirationStatus::Enabled)
.build()?;
Ok(rule)
}
async fn put_expiration_config(client: &Client, bucket: &str, rule: LifecycleRule) -> TestResult { async fn put_expiration_config(client: &Client, bucket: &str, rule: LifecycleRule) -> TestResult {
let lifecycle = BucketLifecycleConfiguration::builder().rules(rule).build()?; let lifecycle = BucketLifecycleConfiguration::builder().rules(rule).build()?;
client client
@@ -412,6 +445,143 @@ async fn test_lifecycle_noncurrent_version_expiry_removes_only_old_version() ->
Ok(()) Ok(())
} }
/// A combined `NoncurrentDays=1` and `ExpiredObjectDeleteMarker=true` rule
/// must remove a noncurrent data version and then its sole latest delete
/// marker, without expiring current-only objects. A second prefix with only
/// noncurrent expiry proves that marker cleanup comes from EODM.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_lifecycle_noncurrent_expiry_then_cleans_expired_delete_marker() -> TestResult {
let mut env = RustFSTestEnvironment::new().await?;
let mut extra_env = fast_lifecycle_env();
extra_env.push(("RUSTFS_ILM_DEBUG_DAY_SECS", "2"));
env.start_rustfs_server_with_env(vec![], &extra_env).await?;
let client = env.create_s3_client();
let bucket = "ilm-expired-delete-marker";
client.create_bucket().bucket(bucket).send().await?;
client
.put_bucket_versioning()
.bucket(bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await?;
let cascade_key = "cascade/deleted.txt";
let cascade_put = client
.put_object()
.bucket(bucket)
.key(cascade_key)
.body(ByteStream::from_static(b"cascade payload"))
.send()
.await?;
let cascade_data_version = cascade_put
.version_id()
.map(str::to_string)
.expect("cascade PUT returns a version id");
let cascade_delete = client.delete_object().bucket(bucket).key(cascade_key).send().await?;
let cascade_marker_version = cascade_delete
.version_id()
.map(str::to_string)
.expect("cascade DELETE returns a marker version id");
assert_eq!(cascade_delete.delete_marker(), Some(true));
let survivor_key = "cascade/current-only.txt";
client
.put_object()
.bucket(bucket)
.key(survivor_key)
.body(ByteStream::from_static(b"current payload"))
.send()
.await?;
let survivor_before = client.get_object().bucket(bucket).key(survivor_key).send().await?;
assert_eq!(survivor_before.body.collect().await?.into_bytes().as_ref(), b"current payload");
let control_key = "nve-only/deleted.txt";
let control_put = client
.put_object()
.bucket(bucket)
.key(control_key)
.body(ByteStream::from_static(b"control payload"))
.send()
.await?;
let control_data_version = control_put
.version_id()
.map(str::to_string)
.expect("control PUT returns a version id");
let control_delete = client.delete_object().bucket(bucket).key(control_key).send().await?;
let control_marker_version = control_delete
.version_id()
.map(str::to_string)
.expect("control DELETE returns a marker version id");
assert_eq!(control_delete.delete_marker(), Some(true));
let cascade_before = client
.list_object_versions()
.bucket(bucket)
.prefix(cascade_key)
.send()
.await?;
assert!(
cascade_before
.versions()
.iter()
.any(|version| version.version_id() == Some(cascade_data_version.as_str())),
"cascade data version must exist before lifecycle is installed: {cascade_before:?}"
);
assert!(
cascade_before
.delete_markers()
.iter()
.any(|marker| { marker.version_id() == Some(cascade_marker_version.as_str()) && marker.is_latest() == Some(true) }),
"cascade latest delete marker must exist before lifecycle is installed: {cascade_before:?}"
);
let lifecycle = BucketLifecycleConfiguration::builder()
.rules(noncurrent_expiration_with_delete_marker_cleanup_rule(
"expire-and-clean-marker",
"cascade/",
1,
)?)
.rules(noncurrent_expiration_rule("expire-only", "nve-only/", 1)?)
.build()?;
client
.put_bucket_lifecycle_configuration()
.bucket(bucket)
.lifecycle_configuration(lifecycle)
.send()
.await?;
wait_for_key_versions_empty(&client, bucket, cascade_key, StdDuration::from_secs(90)).await?;
wait_for_version_expired(&client, bucket, control_key, &control_data_version, StdDuration::from_secs(90)).await?;
let survivor = client.get_object().bucket(bucket).key(survivor_key).send().await?;
assert_eq!(survivor.body.collect().await?.into_bytes().as_ref(), b"current payload");
let control_after = client
.list_object_versions()
.bucket(bucket)
.prefix(control_key)
.send()
.await?;
assert!(
control_after.versions().is_empty(),
"NVE-only control must remove its data version: {control_after:?}"
);
assert!(
control_after
.delete_markers()
.iter()
.any(|marker| { marker.version_id() == Some(control_marker_version.as_str()) && marker.is_latest() == Some(true) }),
"NVE-only control must preserve its latest delete marker: {control_after:?}"
);
Ok(())
}
/// `Days=0` expiration is invalid per S3 semantics (`Days` must be a positive /// `Days=0` expiration is invalid per S3 semantics (`Days` must be a positive
/// integer >= 1). A `PutBucketLifecycleConfiguration` carrying a zero-day rule /// integer >= 1). A `PutBucketLifecycleConfiguration` carrying a zero-day rule
/// must be rejected with `InvalidArgument` (HTTP 400) - see crates/lifecycle /// must be rejected with `InvalidArgument` (HTTP 400) - see crates/lifecycle
+81 -6
View File
@@ -41,6 +41,10 @@ use bytes::Bytes;
use futures::lock::Mutex; use futures::lock::Mutex;
use metrics::counter; use metrics::counter;
use rustfs_filemeta::{FileInfo, ObjectPartInfo, RawFileInfo}; use rustfs_filemeta::{FileInfo, ObjectPartInfo, RawFileInfo};
use rustfs_io_metrics::internode_metrics::{
INTERNODE_STAGE_READ_VERSION_REQUEST_ENCODE, INTERNODE_STAGE_READ_VERSION_RESPONSE_DECODE,
INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP,
};
use rustfs_protos::ChannelClass; use rustfs_protos::ChannelClass;
use rustfs_protos::evict_failed_connection; use rustfs_protos::evict_failed_connection;
use rustfs_protos::proto_gen::node_service::RenamePartRequest; use rustfs_protos::proto_gen::node_service::RenamePartRequest;
@@ -64,7 +68,7 @@ use std::{
atomic::{AtomicBool, AtomicU32, Ordering}, atomic::{AtomicBool, AtomicU32, Ordering},
}, },
task::{Context, Poll}, task::{Context, Poll},
time::Duration, time::{Duration, Instant},
}; };
use tokio::time; use tokio::time;
use tokio::{ use tokio::{
@@ -1790,6 +1794,16 @@ fn decode_msgpack_or_json<T: DeserializeOwned>(binary: &[u8], json: &str, value_
} }
} }
fn read_version_stage_timer(attribution_enabled: bool) -> Option<Instant> {
attribution_enabled.then(Instant::now)
}
fn record_read_version_stage(stage: &'static str, started_at: Option<Instant>) {
if let Some(started_at) = started_at {
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_stage(stage, started_at.elapsed());
}
}
/// Aggregate encoded size (bytes) of a `ReadMultiple` response, preferring the msgpack payloads /// Aggregate encoded size (bytes) of a `ReadMultiple` response, preferring the msgpack payloads
/// and falling back to the JSON compatibility strings. Used to size the RPC for the payload /// and falling back to the JSON compatibility strings. Used to size the RPC for the payload
/// histogram / large-payload alerting (grpc-optimization P0 instrumentation). /// histogram / large-payload alerting (grpc-optimization P0 instrumentation).
@@ -2705,8 +2719,11 @@ impl DiskAPI for RemoteDisk {
state = "started", state = "started",
"Remote disk RPC started" "Remote disk RPC started"
); );
let opts_str = compat_json(opts)?; let read_version_attribution_enabled = rustfs_io_metrics::get_stage_metrics_enabled();
let opts_bin = encode_msgpack(opts)?; let encode_started = read_version_stage_timer(read_version_attribution_enabled);
let encoded_opts = compat_json(opts).and_then(|opts_str| encode_msgpack(opts).map(|opts_bin| (opts_str, opts_bin)));
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_REQUEST_ENCODE, encode_started);
let (opts_str, opts_bin) = encoded_opts?;
// Idempotent version read: eligible for the bounded transient-network retry so a single // Idempotent version read: eligible for the bounded transient-network retry so a single
// reset-by-peer during the read-after-write window does not erode the metadata read // reset-by-peer during the read-after-write window does not erode the metadata read
@@ -2722,6 +2739,14 @@ impl DiskAPI for RemoteDisk {
.get_client() .get_client()
.await .await
.map_err(|err| Error::other(format!("can not get client, err: {err}")))?; .map_err(|err| Error::other(format!("can not get client, err: {err}")))?;
let request_payload_bytes = read_version_attribution_enabled.then(|| {
disk.len()
.saturating_add(volume.len())
.saturating_add(path.len())
.saturating_add(version_id.len())
.saturating_add(opts_str.len())
.saturating_add(opts_bin.len())
});
let request = Request::new(ReadVersionRequest { let request = Request::new(ReadVersionRequest {
disk, disk,
volume: volume.to_string(), volume: volume.to_string(),
@@ -2731,14 +2756,47 @@ impl DiskAPI for RemoteDisk {
opts_bin: opts_bin.into(), opts_bin: opts_bin.into(),
}); });
let response = client.read_version(request).await?.into_inner(); crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request();
if let Some(request_payload_bytes) = request_payload_bytes {
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_sent_bytes(request_payload_bytes);
}
let rpc_started = read_version_stage_timer(read_version_attribution_enabled);
let response = match client.read_version(request).await {
Ok(response) => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
response.into_inner()
}
Err(err) => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_error();
return Err(err.into());
}
};
if !response.success { if !response.success {
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_error();
return Err(response.error.unwrap_or_default().into()); return Err(response.error.unwrap_or_default().into());
} }
let file_info = decode_msgpack_or_json::<FileInfo>(&response.file_info_bin, &response.file_info, "FileInfo")?; crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_recv_bytes(
validate_decoded_file_info(&file_info)?; response.file_info.len().saturating_add(response.file_info_bin.len()),
);
let decode_started = read_version_stage_timer(read_version_attribution_enabled);
let file_info = match decode_msgpack_or_json::<FileInfo>(&response.file_info_bin, &response.file_info, "FileInfo")
.and_then(|file_info| {
validate_decoded_file_info(&file_info)?;
Ok(file_info)
}) {
Ok(file_info) => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_RESPONSE_DECODE, decode_started);
file_info
}
Err(err) => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_RESPONSE_DECODE, decode_started);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_error();
return Err(err);
}
};
Ok(file_info) Ok(file_info)
}, },
@@ -7931,12 +7989,17 @@ mod tests {
} }
#[tokio::test] #[tokio::test]
#[serial]
async fn read_version_uses_the_metadata_timeout_on_a_stalled_peer() { async fn read_version_uses_the_metadata_timeout_on_a_stalled_peer() {
runtime_sources::ensure_test_rpc_secret(); runtime_sources::ensure_test_rpc_secret();
let Some((base_addr, accept_task)) = spawn_stalled_grpc_peer().await else { let Some((base_addr, accept_task)) = spawn_stalled_grpc_peer().await else {
return; return;
}; };
let remote_disk = remote_disk_for_addr(&base_addr).await; let remote_disk = remote_disk_for_addr(&base_addr).await;
let metrics = rustfs_io_metrics::internode_metrics::global_internode_metrics();
let previous_stage_metrics = rustfs_io_metrics::get_stage_metrics_enabled();
metrics.reset_for_test();
rustfs_io_metrics::set_get_stage_metrics_enabled(true);
temp_env::async_with_vars( temp_env::async_with_vars(
[ [
@@ -7960,6 +8023,18 @@ mod tests {
) )
.await; .await;
rustfs_io_metrics::set_get_stage_metrics_enabled(previous_stage_metrics);
let snapshot = metrics.snapshot();
assert!(
snapshot.outgoing_requests_total >= 1,
"ReadVersion call site should record outgoing attempts when attribution is enabled"
);
assert!(
snapshot.sent_bytes_total > 0,
"ReadVersion call site should record request payload bytes when attribution is enabled"
);
metrics.reset_for_test();
remote_disk.cancel_token.cancel(); remote_disk.cancel_token.cancel();
accept_task.abort(); accept_task.abort();
} }
@@ -14,10 +14,11 @@
use rustfs_io_metrics::internode_metrics::{ use rustfs_io_metrics::internode_metrics::{
INTERNODE_MSGPACK_CODEC_JSON, INTERNODE_MSGPACK_CODEC_MSGPACK, INTERNODE_MSGPACK_DIRECTION_RESPONSE, INTERNODE_MSGPACK_CODEC_JSON, INTERNODE_MSGPACK_CODEC_MSGPACK, INTERNODE_MSGPACK_DIRECTION_RESPONSE,
INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_READ_MULTIPLE, INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_READ_MULTIPLE, INTERNODE_OPERATION_GRPC_READ_VERSION,
INTERNODE_OPERATION_PUT_FILE_STREAM, INTERNODE_OPERATION_READ_FILE_STREAM, INTERNODE_TRANSPORT_BACKEND_GRPC, INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_OPERATION_PUT_FILE_STREAM, INTERNODE_OPERATION_READ_FILE_STREAM,
INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, global_internode_metrics, INTERNODE_TRANSPORT_BACKEND_GRPC, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, global_internode_metrics,
}; };
use std::time::Duration;
#[cfg(test)] #[cfg(test)]
use rustfs_io_metrics::internode_metrics::InternodeMetricsSnapshot; use rustfs_io_metrics::internode_metrics::InternodeMetricsSnapshot;
@@ -82,6 +83,59 @@ pub(crate) fn record_remote_disk_grpc_read_all_request() {
.record_outgoing_request_for_operation_and_backend(INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC); .record_outgoing_request_for_operation_and_backend(INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC);
} }
pub(crate) fn record_remote_disk_grpc_read_version_request() {
if !rustfs_io_metrics::get_stage_metrics_enabled() {
return;
}
global_internode_metrics().record_outgoing_request_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
);
}
pub(crate) fn record_remote_disk_grpc_read_version_error() {
if !rustfs_io_metrics::get_stage_metrics_enabled() {
return;
}
global_internode_metrics()
.record_error_for_operation_and_backend(INTERNODE_OPERATION_GRPC_READ_VERSION, INTERNODE_TRANSPORT_BACKEND_GRPC);
}
pub(crate) fn record_remote_disk_grpc_read_version_sent_bytes(bytes: usize) {
if !rustfs_io_metrics::get_stage_metrics_enabled() {
return;
}
global_internode_metrics().record_sent_bytes_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
bytes,
);
}
pub(crate) fn record_remote_disk_grpc_read_version_recv_bytes(bytes: usize) {
if !rustfs_io_metrics::get_stage_metrics_enabled() {
return;
}
global_internode_metrics().record_recv_bytes_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
bytes,
);
record_grpc_payload_size(INTERNODE_OPERATION_GRPC_READ_VERSION, bytes);
}
pub(crate) fn record_remote_disk_grpc_read_version_stage(stage: &'static str, duration: Duration) {
if !rustfs_io_metrics::get_stage_metrics_enabled() {
return;
}
global_internode_metrics().record_stage_duration_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
stage,
duration,
);
}
pub(crate) fn record_remote_disk_grpc_read_all_recv_bytes(bytes: usize) { pub(crate) fn record_remote_disk_grpc_read_all_recv_bytes(bytes: usize) {
global_internode_metrics().record_recv_bytes_for_operation_and_backend( global_internode_metrics().record_recv_bytes_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_READ_ALL,
+44 -1
View File
@@ -1041,7 +1041,13 @@ fn should_count_decommission_version_complete(ignore: bool, cleanup_ignored: boo
fn is_decommission_copy_cleanup_safe_error(err: &Error) -> bool { fn is_decommission_copy_cleanup_safe_error(err: &Error) -> bool {
// DataMovementOverwriteErr only means source and destination pool resolved to // DataMovementOverwriteErr only means source and destination pool resolved to
// the same pool. Without a target equivalence check it is not cleanup-safe. // the same pool. Without a target equivalence check it is not cleanup-safe.
is_err_object_not_found(err) || is_err_version_not_found(err) if is_err_object_not_found(err) || is_err_version_not_found(err) {
return true;
}
// A not-found surfacing from inside a data-movement stage is the same
// condition once the wrapper is unwrapped (backlog#1827 T2).
crate::data_movement::data_movement_stage_source(err).is_some_and(is_decommission_copy_cleanup_safe_error)
} }
fn is_decommission_target_capacity_error(err: &Error) -> bool { fn is_decommission_target_capacity_error(err: &Error) -> bool {
@@ -1049,6 +1055,13 @@ fn is_decommission_target_capacity_error(err: &Error) -> bool {
return true; return true;
} }
// A stage failure keeps the error it wrapped, so classify by type rather
// than by the rendered message (backlog#1827 T2). The substring fallback
// stays for errors that reached here through some other wrapper.
if let Some(source) = crate::data_movement::data_movement_stage_source(err) {
return is_decommission_target_capacity_error(source);
}
let message = err.to_string(); let message = err.to_string();
let disk_full = Error::DiskFull.to_string(); let disk_full = Error::DiskFull.to_string();
let storage_full = Error::StorageFull.to_string(); let storage_full = Error::StorageFull.to_string();
@@ -4427,6 +4440,36 @@ mod tests {
assert!(is_decommission_target_capacity_error(&Error::StorageFull)); assert!(is_decommission_target_capacity_error(&Error::StorageFull));
} }
/// The decommission loop classifies errors that came back through a
/// data-movement stage wrapper. Before backlog#1827 T2 the wrapper flattened
/// everything into `Error::other(String)`, so these two classifiers had to
/// match on rendered text; now the wrapped error is recoverable by type.
#[test]
fn decommission_classifiers_see_through_a_stage_wrapper() {
let wrap = |inner: Error| {
crate::data_movement::data_movement_stage_error_for_test(
"decommission_object",
"put_object",
"bucket-a",
"object-a",
inner,
)
};
// Capacity: the target pool filling up must still stop the loop.
assert!(is_decommission_target_capacity_error(&wrap(Error::DiskFull)));
assert!(is_decommission_target_capacity_error(&wrap(Error::StorageFull)));
assert!(!is_decommission_target_capacity_error(&wrap(Error::SlowDown)));
// Cleanup safety: a not-found surfacing from inside a stage is the same
// condition as one surfacing directly, so the source entry stays
// eligible for cleanup.
let not_found = Error::ObjectNotFound("bucket-a".to_string(), "object-a".to_string());
assert!(is_decommission_copy_cleanup_safe_error(&not_found));
assert!(is_decommission_copy_cleanup_safe_error(&wrap(not_found)));
assert!(!is_decommission_copy_cleanup_safe_error(&wrap(Error::SlowDown)));
}
#[test] #[test]
fn decommission_target_capacity_error_accepts_wrapped_capacity_errors() { fn decommission_target_capacity_error_accepts_wrapped_capacity_errors() {
let disk_full = Error::other(format!("decommission_object: put_object failed for bucket/object: {}", Error::DiskFull)); let disk_full = Error::other(format!("decommission_object: put_object failed for bucket/object: {}", Error::DiskFull));
+88 -2
View File
@@ -471,8 +471,60 @@ fn resolve_data_movement_abort_result(
)) ))
} }
fn data_movement_stage_error(op_label: &str, stage: &str, bucket: &str, object: &str, err: impl std::fmt::Display) -> Error { /// A data-movement stage failure that keeps the error it wrapped.
Error::other(format!("{op_label}: {stage} failed for {bucket}/{object}: {err}")) ///
/// The rendered message is byte-identical to the `format!` this replaced, so
/// logs and any message-matching callers are unaffected. What changes is that
/// the original error stays reachable through `source()`, which is what lets
/// the decommission loop classify by type instead of by substring
/// (backlog#1827 T2).
#[derive(Debug)]
struct DataMovementStageError {
rendered: String,
source: Box<dyn std::error::Error + Send + Sync>,
}
impl std::fmt::Display for DataMovementStageError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(&self.rendered)
}
}
impl std::error::Error for DataMovementStageError {
fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
Some(self.source.as_ref())
}
}
fn data_movement_stage_error<E>(op_label: &str, stage: &str, bucket: &str, object: &str, err: E) -> Error
where
E: std::error::Error + Send + Sync + 'static,
{
let rendered = format!("{op_label}: {stage} failed for {bucket}/{object}: {err}");
Error::other(DataMovementStageError {
rendered,
source: Box::new(err),
})
}
#[cfg(test)]
pub(crate) fn data_movement_stage_error_for_test(op_label: &str, stage: &str, bucket: &str, object: &str, err: Error) -> Error {
data_movement_stage_error(op_label, stage, bucket, object, err)
}
/// Recover the error a [`data_movement_stage_error`] wrapped, if this is one.
///
/// `Error::other` boxes through `std::io::Error`, so the chain is
/// `StorageError::Io` -> `DataMovementStageError` -> the original error.
pub(crate) fn data_movement_stage_source(err: &Error) -> Option<&Error> {
let Error::Io(io_err) = err else {
return None;
};
io_err
.get_ref()?
.downcast_ref::<DataMovementStageError>()?
.source
.downcast_ref::<Error>()
} }
fn schedule_data_movement_multipart_abort_cleanup( fn schedule_data_movement_multipart_abort_cleanup(
@@ -1865,6 +1917,40 @@ mod tests {
assert!(message.contains(Error::SlowDown.to_string().as_str())); assert!(message.contains(Error::SlowDown.to_string().as_str()));
} }
#[test]
fn stage_error_renders_exactly_as_the_format_it_replaced() {
// The wrapper gained a source; its message must not have moved, or log
// scrapers and any message-matching caller would break (backlog#1827 T2).
// `Error::other` renders through `StorageError::Io`, which prefixes
// "Io error: " — that was true of the `format!` this replaced too, so
// the full string is what must stay stable.
let err = data_movement_stage_error("rebalance_object", "put_object", "bucket-a", "object-a", Error::SlowDown);
assert_eq!(
err.to_string(),
format!("Io error: rebalance_object: put_object failed for bucket-a/object-a: {}", Error::SlowDown)
);
assert_eq!(
err.to_string(),
Error::other(format!("rebalance_object: put_object failed for bucket-a/object-a: {}", Error::SlowDown)).to_string()
);
}
#[test]
fn stage_error_keeps_the_wrapped_error_recoverable() {
for original in [Error::DiskFull, Error::StorageFull, Error::FileNotFound, Error::SlowDown] {
let wrapped =
data_movement_stage_error("decommission_object", "put_object", "bucket-a", "object-a", original.clone());
let recovered = data_movement_stage_source(&wrapped).expect("the wrapped error must be recoverable");
assert_eq!(recovered.to_string(), original.to_string());
}
}
#[test]
fn stage_source_ignores_errors_it_did_not_wrap() {
assert!(data_movement_stage_source(&Error::DiskFull).is_none());
assert!(data_movement_stage_source(&Error::other("plain io error")).is_none());
}
#[test] #[test]
fn test_data_movement_part_stage_error_includes_stage_object_and_part() { fn test_data_movement_part_stage_error_includes_stage_object_and_part() {
let err = let err =
+8
View File
@@ -1098,6 +1098,10 @@ pub(crate) async fn sync_dir_files_with_limiter(dir: impl AsRef<Path>, disk_perm
let files = run_file_sync_blocking(disk_permits.clone(), move || { let files = run_file_sync_blocking(disk_permits.clone(), move || {
let files = regular_files(&scan_dir)?; let files = regular_files(&scan_dir)?;
if files.len() < PARALLEL_FILE_SYNC_THRESHOLD { if files.len() < PARALLEL_FILE_SYNC_THRESHOLD {
rustfs_io_metrics::record_put_rename_fdatasync_batch(
rustfs_io_metrics::PUT_RENAME_FDATASYNC_BATCH_MODE_SERIAL,
files.len(),
);
sync_files(&files)?; sync_files(&files)?;
let fsync_started = rustfs_io_metrics::put_stage_timer(); let fsync_started = rustfs_io_metrics::put_stage_timer();
let result = fsync_dir_std(scan_dir); let result = fsync_dir_std(scan_dir);
@@ -1115,6 +1119,10 @@ pub(crate) async fn sync_dir_files_with_limiter(dir: impl AsRef<Path>, disk_perm
let Some(files) = files else { let Some(files) = files else {
return Ok(()); return Ok(());
}; };
rustfs_io_metrics::record_put_rename_fdatasync_batch(
rustfs_io_metrics::PUT_RENAME_FDATASYNC_BATCH_MODE_PARALLEL,
files.len(),
);
futures::stream::iter(files.into_iter().map(Ok::<_, io::Error>)) futures::stream::iter(files.into_iter().map(Ok::<_, io::Error>))
.try_for_each_concurrent(MAX_PARALLEL_FILE_SYNCS, |path| { .try_for_each_concurrent(MAX_PARALLEL_FILE_SYNCS, |path| {
let disk_permits = disk_permits.clone(); let disk_permits = disk_permits.clone();
@@ -3417,6 +3417,25 @@ impl SetDisks {
quorum_wait_started, quorum_wait_started,
); );
let (results, mut file_infos) = fanout_result.map_err(|_| DiskError::Unexpected)?; let (results, mut file_infos) = fanout_result.map_err(|_| DiskError::Unexpected)?;
if rustfs_io_metrics::put_stage_metrics_enabled() {
let mut fanout_success = 0;
let mut fanout_error = 0;
let mut fanout_panic = 0;
for result in &results {
match result {
Ok(Ok(_)) => fanout_success += 1,
Ok(Err(_)) => fanout_error += 1,
Err(_) => fanout_panic += 1,
}
}
rustfs_io_metrics::record_put_rename_quorum_wait_fanout(
results.len(),
write_quorum,
fanout_success,
fanout_error,
fanout_panic,
);
}
for (idx, result) in results.iter().enumerate() { for (idx, result) in results.iter().enumerate() {
match result { match result {
+120 -27
View File
@@ -47,6 +47,13 @@ pub const INTERNODE_MSGPACK_DIRECTION_REQUEST: &str = "request";
pub const INTERNODE_MSGPACK_DIRECTION_RESPONSE: &str = "response"; pub const INTERNODE_MSGPACK_DIRECTION_RESPONSE: &str = "response";
pub const INTERNODE_MSGPACK_CODEC_MSGPACK: &str = "msgpack"; pub const INTERNODE_MSGPACK_CODEC_MSGPACK: &str = "msgpack";
pub const INTERNODE_MSGPACK_CODEC_JSON: &str = "json"; pub const INTERNODE_MSGPACK_CODEC_JSON: &str = "json";
pub const INTERNODE_STAGE_READ_VERSION_REQUEST_ENCODE: &str = "read_version_request_encode";
pub const INTERNODE_STAGE_READ_VERSION_REQUEST_DECODE: &str = "read_version_request_decode";
pub const INTERNODE_STAGE_READ_VERSION_DISK_READ: &str = "read_version_disk_read";
pub const INTERNODE_STAGE_READ_VERSION_RESPONSE_JSON_ENCODE: &str = "read_version_response_json_encode";
pub const INTERNODE_STAGE_READ_VERSION_RESPONSE_MSGPACK_ENCODE: &str = "read_version_response_msgpack_encode";
pub const INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP: &str = "read_version_rpc_roundtrip";
pub const INTERNODE_STAGE_READ_VERSION_RESPONSE_DECODE: &str = "read_version_response_decode";
const OPERATION_LABEL: &str = "operation"; const OPERATION_LABEL: &str = "operation";
const BACKEND_LABEL: &str = "backend"; const BACKEND_LABEL: &str = "backend";
@@ -67,6 +74,7 @@ const INTERNODE_OPERATION_REQUESTS_OUTGOING_TOTAL: &str = "rustfs_system_network
const INTERNODE_OPERATION_REQUESTS_INCOMING_TOTAL: &str = "rustfs_system_network_internode_operation_requests_incoming_total"; const INTERNODE_OPERATION_REQUESTS_INCOMING_TOTAL: &str = "rustfs_system_network_internode_operation_requests_incoming_total";
const INTERNODE_OPERATION_ERRORS_TOTAL: &str = "rustfs_system_network_internode_operation_errors_total"; const INTERNODE_OPERATION_ERRORS_TOTAL: &str = "rustfs_system_network_internode_operation_errors_total";
const INTERNODE_OPERATION_DURATION_MS: &str = "rustfs_system_network_internode_operation_duration_ms"; const INTERNODE_OPERATION_DURATION_MS: &str = "rustfs_system_network_internode_operation_duration_ms";
const INTERNODE_OPERATION_STAGE_DURATION_MS: &str = "rustfs_system_network_internode_operation_stage_duration_ms";
const INTERNODE_OPERATION_CLASSIFIED_ERRORS_TOTAL: &str = "rustfs_system_network_internode_operation_classified_errors_total"; const INTERNODE_OPERATION_CLASSIFIED_ERRORS_TOTAL: &str = "rustfs_system_network_internode_operation_classified_errors_total";
const INTERNODE_OPERATION_RETRIES_TOTAL: &str = "rustfs_system_network_internode_operation_retries_total"; const INTERNODE_OPERATION_RETRIES_TOTAL: &str = "rustfs_system_network_internode_operation_retries_total";
const INTERNODE_OPERATION_RETRY_SUCCESSES_TOTAL: &str = "rustfs_system_network_internode_operation_retry_successes_total"; const INTERNODE_OPERATION_RETRY_SUCCESSES_TOTAL: &str = "rustfs_system_network_internode_operation_retry_successes_total";
@@ -105,6 +113,7 @@ const SERVER_OPERATION_BACKEND_HTTP_VERSION_LABELS: &[&str] = &[SERVER_LABEL, OP
const SERVER_OPERATION_BACKEND_FAILURE_REASON_LABELS: &[&str] = const SERVER_OPERATION_BACKEND_FAILURE_REASON_LABELS: &[&str] =
&[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, FAILURE_REASON_LABEL]; &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, FAILURE_REASON_LABEL];
const SERVER_OPERATION_BACKEND_RPC_PATH_LABELS: &[&str] = &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, RPC_PATH_LABEL]; const SERVER_OPERATION_BACKEND_RPC_PATH_LABELS: &[&str] = &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, RPC_PATH_LABEL];
const SERVER_OPERATION_BACKEND_STAGE_LABELS: &[&str] = &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, STAGE_LABEL];
const SERVER_LABELS: &[&str] = &[SERVER_LABEL]; const SERVER_LABELS: &[&str] = &[SERVER_LABEL];
const SERVER_REASON_LABELS: &[&str] = &[SERVER_LABEL, REASON_LABEL]; const SERVER_REASON_LABELS: &[&str] = &[SERVER_LABEL, REASON_LABEL];
const SERVER_QUORUM_FAILURE_LABELS: &[&str] = &[SERVER_LABEL, STAGE_LABEL, DOMINANT_ERROR_LABEL]; const SERVER_QUORUM_FAILURE_LABELS: &[&str] = &[SERVER_LABEL, STAGE_LABEL, DOMINANT_ERROR_LABEL];
@@ -134,6 +143,10 @@ pub const INTERNODE_OPERATION_METRICS: &[InternodeOperationMetricDescriptor] = &
name: INTERNODE_OPERATION_DURATION_MS, name: INTERNODE_OPERATION_DURATION_MS,
labels: SERVER_OPERATION_BACKEND_LABELS, labels: SERVER_OPERATION_BACKEND_LABELS,
}, },
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_STAGE_DURATION_MS,
labels: SERVER_OPERATION_BACKEND_STAGE_LABELS,
},
InternodeOperationMetricDescriptor { InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_CLASSIFIED_ERRORS_TOTAL, name: INTERNODE_OPERATION_CLASSIFIED_ERRORS_TOTAL,
labels: SERVER_OPERATION_BACKEND_CLASSIFICATION_LABELS, labels: SERVER_OPERATION_BACKEND_CLASSIFICATION_LABELS,
@@ -394,6 +407,24 @@ impl InternodeMetrics {
.record(duration_ms); .record(duration_ms);
} }
pub fn record_stage_duration_for_operation_and_backend(
&self,
operation: &'static str,
backend: &'static str,
stage: &'static str,
duration: Duration,
) {
let duration_ms = duration.as_secs_f64() * 1000.0;
metrics::histogram!(
INTERNODE_OPERATION_STAGE_DURATION_MS,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend,
STAGE_LABEL => stage
)
.record(duration_ms);
}
pub fn record_classified_error_for_operation_and_backend( pub fn record_classified_error_for_operation_and_backend(
&self, &self,
operation: &'static str, operation: &'static str,
@@ -988,42 +1019,90 @@ mod tests {
assert_eq!(snapshot.replay_cache_evictions_total, 3); assert_eq!(snapshot.replay_cache_evictions_total, 3);
} }
#[test]
fn operation_stage_duration_records_low_cardinality_stage_labels() {
let recorder = DebuggingRecorder::new();
let snapshotter = recorder.snapshotter();
let metrics = InternodeMetrics::default();
with_local_recorder(&recorder, || {
metrics.record_stage_duration_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP,
Duration::from_micros(125),
);
});
let entries: Vec<_> = snapshotter
.snapshot()
.into_vec()
.into_iter()
.filter(|(composite, _, _, _)| composite.key().name() == INTERNODE_OPERATION_STAGE_DURATION_MS)
.collect();
assert_eq!(entries.len(), 1);
let labels: HashMap<_, _> = entries[0]
.0
.key()
.labels()
.map(|label| (label.key().to_string(), label.value().to_string()))
.collect();
assert_eq!(
labels.get(OPERATION_LABEL).map(String::as_str),
Some(INTERNODE_OPERATION_GRPC_READ_VERSION)
);
assert_eq!(labels.get(BACKEND_LABEL).map(String::as_str), Some(INTERNODE_TRANSPORT_BACKEND_GRPC));
assert_eq!(
labels.get(STAGE_LABEL).map(String::as_str),
Some(INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP)
);
assert!(labels.get(SERVER_LABEL).is_some_and(|value| !value.is_empty()));
match &entries[0].3 {
DebugValue::Histogram(samples) => assert_eq!(samples.iter().map(|sample| sample.0).collect::<Vec<_>>(), vec![0.125]),
other => panic!("{INTERNODE_OPERATION_STAGE_DURATION_MS} must be a histogram, got {other:?}"),
}
}
#[test] #[test]
fn operation_metric_descriptors_include_backend_and_operation_labels() { fn operation_metric_descriptors_include_backend_and_operation_labels() {
assert_eq!(INTERNODE_OPERATION_METRICS.len(), 21); assert_eq!(INTERNODE_OPERATION_METRICS.len(), 22);
for metric in &INTERNODE_OPERATION_METRICS[..6] { for metric in &INTERNODE_OPERATION_METRICS[..6] {
assert_eq!(metric.labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]); assert_eq!(metric.labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
} }
for metric in &INTERNODE_OPERATION_METRICS[6..9] { assert_eq!(
INTERNODE_OPERATION_METRICS[6].labels,
&[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, STAGE_LABEL]
);
for metric in &INTERNODE_OPERATION_METRICS[7..10] {
assert_eq!(metric.labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, CLASSIFICATION_LABEL]); assert_eq!(metric.labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, CLASSIFICATION_LABEL]);
} }
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[9].labels, INTERNODE_OPERATION_METRICS[10].labels,
&[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, HTTP_VERSION_LABEL] &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, HTTP_VERSION_LABEL]
); );
for metric in &INTERNODE_OPERATION_METRICS[10..12] { for metric in &INTERNODE_OPERATION_METRICS[11..13] {
assert_eq!(metric.labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]); assert_eq!(metric.labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
} }
assert_eq!(
INTERNODE_OPERATION_METRICS[12].labels,
&[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, FAILURE_REASON_LABEL]
);
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[13].labels, INTERNODE_OPERATION_METRICS[13].labels,
&[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, RPC_PATH_LABEL] &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, FAILURE_REASON_LABEL]
); );
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[14].labels, INTERNODE_OPERATION_METRICS[14].labels,
&[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, RPC_PATH_LABEL] &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, RPC_PATH_LABEL]
); );
for metric in &INTERNODE_OPERATION_METRICS[15..17] { assert_eq!(
INTERNODE_OPERATION_METRICS[15].labels,
&[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, RPC_PATH_LABEL]
);
for metric in &INTERNODE_OPERATION_METRICS[16..18] {
assert_eq!(metric.labels, &[SERVER_LABEL]); assert_eq!(metric.labels, &[SERVER_LABEL]);
} }
assert_eq!(INTERNODE_OPERATION_METRICS[17].labels, &[SERVER_LABEL, REASON_LABEL]); assert_eq!(INTERNODE_OPERATION_METRICS[18].labels, &[SERVER_LABEL, REASON_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[18].labels, &[SERVER_LABEL, STAGE_LABEL, DOMINANT_ERROR_LABEL]); assert_eq!(INTERNODE_OPERATION_METRICS[19].labels, &[SERVER_LABEL, STAGE_LABEL, DOMINANT_ERROR_LABEL]);
// Payload histogram + large-payload counter carry operation+backend labels. // Payload histogram + large-payload counter carry operation+backend labels.
assert_eq!(INTERNODE_OPERATION_METRICS[19].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[20].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]); assert_eq!(INTERNODE_OPERATION_METRICS[20].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[21].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
} }
#[test] #[test]
@@ -1054,62 +1133,66 @@ mod tests {
); );
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[6].name, INTERNODE_OPERATION_METRICS[6].name,
"rustfs_system_network_internode_operation_classified_errors_total" "rustfs_system_network_internode_operation_stage_duration_ms"
); );
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[7].name, INTERNODE_OPERATION_METRICS[7].name,
"rustfs_system_network_internode_operation_retries_total" "rustfs_system_network_internode_operation_classified_errors_total"
); );
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[8].name, INTERNODE_OPERATION_METRICS[8].name,
"rustfs_system_network_internode_operation_retry_successes_total" "rustfs_system_network_internode_operation_retries_total"
); );
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[9].name, INTERNODE_OPERATION_METRICS[9].name,
"rustfs_system_network_internode_operation_http_versions_total" "rustfs_system_network_internode_operation_retry_successes_total"
); );
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[10].name, INTERNODE_OPERATION_METRICS[10].name,
"rustfs_system_network_internode_operation_stall_timeouts_total" "rustfs_system_network_internode_operation_http_versions_total"
); );
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[11].name, INTERNODE_OPERATION_METRICS[11].name,
"rustfs_system_network_internode_operation_write_shutdown_errors_total" "rustfs_system_network_internode_operation_stall_timeouts_total"
); );
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[12].name, INTERNODE_OPERATION_METRICS[12].name,
"rustfs_system_network_internode_rpc_auth_failures_total" "rustfs_system_network_internode_operation_write_shutdown_errors_total"
); );
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[13].name, INTERNODE_OPERATION_METRICS[13].name,
"rustfs_system_network_internode_replay_cache_overflow_by_operation_total" "rustfs_system_network_internode_rpc_auth_failures_total"
); );
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[14].name, INTERNODE_OPERATION_METRICS[14].name,
"rustfs_system_network_internode_replay_cache_records_total" "rustfs_system_network_internode_replay_cache_overflow_by_operation_total"
); );
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[15].name, INTERNODE_OPERATION_METRICS[15].name,
"rustfs_system_network_internode_replay_cache_entries" "rustfs_system_network_internode_replay_cache_records_total"
); );
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[16].name, INTERNODE_OPERATION_METRICS[16].name,
"rustfs_system_network_internode_replay_cache_capacity" "rustfs_system_network_internode_replay_cache_entries"
); );
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[17].name, INTERNODE_OPERATION_METRICS[17].name,
"rustfs_system_network_internode_replay_cache_evictions_total" "rustfs_system_network_internode_replay_cache_capacity"
); );
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[18].name, INTERNODE_OPERATION_METRICS[18].name,
"rustfs_system_storage_erasure_write_quorum_failures_total" "rustfs_system_network_internode_replay_cache_evictions_total"
); );
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[19].name, INTERNODE_OPERATION_METRICS[19].name,
"rustfs_system_network_internode_operation_payload_bytes" "rustfs_system_storage_erasure_write_quorum_failures_total"
); );
assert_eq!( assert_eq!(
INTERNODE_OPERATION_METRICS[20].name, INTERNODE_OPERATION_METRICS[20].name,
"rustfs_system_network_internode_operation_payload_bytes"
);
assert_eq!(
INTERNODE_OPERATION_METRICS[21].name,
"rustfs_system_network_internode_operation_large_payloads_total" "rustfs_system_network_internode_operation_large_payloads_total"
); );
assert_eq!(INTERNODE_OPERATION_GRPC_READ_MULTIPLE, "grpc_read_multiple"); assert_eq!(INTERNODE_OPERATION_GRPC_READ_MULTIPLE, "grpc_read_multiple");
@@ -1129,6 +1212,16 @@ mod tests {
assert_eq!(INTERNODE_MSGPACK_DIRECTION_RESPONSE, "response"); assert_eq!(INTERNODE_MSGPACK_DIRECTION_RESPONSE, "response");
assert_eq!(INTERNODE_MSGPACK_CODEC_MSGPACK, "msgpack"); assert_eq!(INTERNODE_MSGPACK_CODEC_MSGPACK, "msgpack");
assert_eq!(INTERNODE_MSGPACK_CODEC_JSON, "json"); assert_eq!(INTERNODE_MSGPACK_CODEC_JSON, "json");
assert_eq!(INTERNODE_STAGE_READ_VERSION_REQUEST_ENCODE, "read_version_request_encode");
assert_eq!(INTERNODE_STAGE_READ_VERSION_REQUEST_DECODE, "read_version_request_decode");
assert_eq!(INTERNODE_STAGE_READ_VERSION_DISK_READ, "read_version_disk_read");
assert_eq!(INTERNODE_STAGE_READ_VERSION_RESPONSE_JSON_ENCODE, "read_version_response_json_encode");
assert_eq!(
INTERNODE_STAGE_READ_VERSION_RESPONSE_MSGPACK_ENCODE,
"read_version_response_msgpack_encode"
);
assert_eq!(INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP, "read_version_rpc_roundtrip");
assert_eq!(INTERNODE_STAGE_READ_VERSION_RESPONSE_DECODE, "read_version_response_decode");
assert_eq!( assert_eq!(
INTERNODE_SIGNATURE_V1_FALLBACK_TOTAL, INTERNODE_SIGNATURE_V1_FALLBACK_TOTAL,
"rustfs_system_network_internode_signature_v1_fallback_total" "rustfs_system_network_internode_signature_v1_fallback_total"
+110
View File
@@ -120,6 +120,14 @@ pub const PUT_STAGE_SET_DISK_RENAME_BACKUP_DIR_FSYNC: &str = "set_disk_rename_ba
pub const PUT_STAGE_SET_DISK_RENAME_ANCESTOR_DIR_FSYNC: &str = "set_disk_rename_ancestor_dir_fsync"; pub const PUT_STAGE_SET_DISK_RENAME_ANCESTOR_DIR_FSYNC: &str = "set_disk_rename_ancestor_dir_fsync";
pub const PUT_STAGE_SET_DISK_RENAME_RENAME_SYSCALL: &str = "set_disk_rename_rename_syscall"; pub const PUT_STAGE_SET_DISK_RENAME_RENAME_SYSCALL: &str = "set_disk_rename_rename_syscall";
pub const PUT_RENAME_FDATASYNC_BATCH_MODE_SERIAL: &str = "serial";
pub const PUT_RENAME_FDATASYNC_BATCH_MODE_PARALLEL: &str = "parallel";
pub const PUT_RENAME_QUORUM_FANOUT_STATE_SCHEDULED: &str = "scheduled";
pub const PUT_RENAME_QUORUM_FANOUT_STATE_WRITE_QUORUM: &str = "write_quorum";
pub const PUT_RENAME_QUORUM_FANOUT_STATE_SUCCESS: &str = "success";
pub const PUT_RENAME_QUORUM_FANOUT_STATE_ERROR: &str = "error";
pub const PUT_RENAME_QUORUM_FANOUT_STATE_PANIC: &str = "panic";
#[inline(always)] #[inline(always)]
pub fn get_stage_metrics_enabled() -> bool { pub fn get_stage_metrics_enabled() -> bool {
GET_STAGE_METRICS_ENABLED.load(Ordering::Relaxed) GET_STAGE_METRICS_ENABLED.load(Ordering::Relaxed)
@@ -2042,6 +2050,44 @@ pub fn record_put_object_stage_duration_from(stage: &'static str, started_at: Op
} }
} }
#[inline(always)]
fn put_stage_count_value(value: usize) -> f64 {
match u32::try_from(value) {
Ok(value) => f64::from(value),
Err(_) => f64::from(u32::MAX),
}
}
#[inline(always)]
pub fn record_put_rename_fdatasync_batch(mode: &'static str, files: usize) {
if !put_stage_metrics_enabled() {
return;
}
histogram!("rustfs_s3_put_object_rename_fdatasync_batch_files", "mode" => mode).record(put_stage_count_value(files));
}
#[inline(always)]
pub fn record_put_rename_quorum_wait_fanout(
scheduled: usize,
write_quorum: usize,
success: usize,
error: usize,
panicked: usize,
) {
if !put_stage_metrics_enabled() {
return;
}
for (state, count) in [
(PUT_RENAME_QUORUM_FANOUT_STATE_SCHEDULED, scheduled),
(PUT_RENAME_QUORUM_FANOUT_STATE_WRITE_QUORUM, write_quorum),
(PUT_RENAME_QUORUM_FANOUT_STATE_SUCCESS, success),
(PUT_RENAME_QUORUM_FANOUT_STATE_ERROR, error),
(PUT_RENAME_QUORUM_FANOUT_STATE_PANIC, panicked),
] {
histogram!("rustfs_s3_put_object_rename_quorum_wait_fanout_disks", "state" => state).record(put_stage_count_value(count));
}
}
/// Record generic internal operation stage duration (non-PUT paths). /// Record generic internal operation stage duration (non-PUT paths).
/// Use this for metacache walks, listing, lifecycle, and other background /// Use this for metacache walks, listing, lifecycle, and other background
/// operations that are NOT part of the PUT object hot path. /// operations that are NOT part of the PUT object hot path.
@@ -3122,6 +3168,70 @@ mod tests {
assert!(stages.iter().all(|stage| recorded.contains(*stage))); assert!(stages.iter().all(|stage| recorded.contains(*stage)));
} }
#[test]
fn put_rename_code_level_metrics_are_static_and_gated() {
let _guard = METRICS_FLAG_LOCK.lock().unwrap_or_else(|e| e.into_inner());
let recorder = DebuggingRecorder::new();
let snapshotter = recorder.snapshotter();
metrics::with_local_recorder(&recorder, || {
set_put_stage_metrics_enabled(false);
record_put_rename_fdatasync_batch(PUT_RENAME_FDATASYNC_BATCH_MODE_SERIAL, 2);
record_put_rename_quorum_wait_fanout(4, 3, 3, 1, 0);
set_put_stage_metrics_enabled(true);
record_put_rename_fdatasync_batch(PUT_RENAME_FDATASYNC_BATCH_MODE_PARALLEL, 9);
record_put_rename_quorum_wait_fanout(4, 3, 3, 1, 0);
set_put_stage_metrics_enabled(false);
});
let rows = snapshotter.snapshot().into_vec();
assert_eq!(histogram_samples(&rows, "rustfs_s3_put_object_rename_fdatasync_batch_files"), vec![9.0]);
let batch_modes = rows
.iter()
.filter(|(composite, _, _, _)| {
composite.kind() == MetricKind::Histogram
&& composite.key().name() == "rustfs_s3_put_object_rename_fdatasync_batch_files"
})
.flat_map(|(composite, _, _, _)| {
composite
.key()
.labels()
.filter(|label| label.key() == "mode")
.map(|label| label.value().to_string())
.collect::<Vec<_>>()
})
.collect::<HashSet<_>>();
assert_eq!(batch_modes, HashSet::from([PUT_RENAME_FDATASYNC_BATCH_MODE_PARALLEL.to_string()]));
let quorum_samples = histogram_samples(&rows, "rustfs_s3_put_object_rename_quorum_wait_fanout_disks");
assert_eq!(quorum_samples, vec![0.0, 1.0, 3.0, 3.0, 4.0]);
let quorum_states = rows
.iter()
.filter(|(composite, _, _, _)| {
composite.kind() == MetricKind::Histogram
&& composite.key().name() == "rustfs_s3_put_object_rename_quorum_wait_fanout_disks"
})
.flat_map(|(composite, _, _, _)| {
composite
.key()
.labels()
.filter(|label| label.key() == "state")
.map(|label| label.value().to_string())
.collect::<Vec<_>>()
})
.collect::<HashSet<_>>();
assert_eq!(
quorum_states,
HashSet::from([
PUT_RENAME_QUORUM_FANOUT_STATE_SCHEDULED.to_string(),
PUT_RENAME_QUORUM_FANOUT_STATE_WRITE_QUORUM.to_string(),
PUT_RENAME_QUORUM_FANOUT_STATE_SUCCESS.to_string(),
PUT_RENAME_QUORUM_FANOUT_STATE_ERROR.to_string(),
PUT_RENAME_QUORUM_FANOUT_STATE_PANIC.to_string(),
])
);
}
#[test] #[test]
fn test_put_object_diagnostic_buckets() { fn test_put_object_diagnostic_buckets() {
assert_eq!(put_object_size_bucket(0), "unknown"); assert_eq!(put_object_size_bucket(0), "unknown");
+20 -1
View File
@@ -48,6 +48,7 @@ pub struct LocalClient {
struct LocalGuardEntry { struct LocalGuardEntry {
guard: FastLockGuard, guard: FastLockGuard,
acquired_at: SystemTime, acquired_at: SystemTime,
last_refreshed: SystemTime,
expires_at: SystemTime, expires_at: SystemTime,
deadline: Instant, deadline: Instant,
ttl: Duration, ttl: Duration,
@@ -60,6 +61,7 @@ impl LocalGuardEntry {
Self { Self {
guard, guard,
acquired_at, acquired_at,
last_refreshed: acquired_at,
expires_at: acquired_at.checked_add(ttl).unwrap_or(acquired_at), expires_at: acquired_at.checked_add(ttl).unwrap_or(acquired_at),
deadline: monotonic_now.checked_add(ttl).unwrap_or(monotonic_now), deadline: monotonic_now.checked_add(ttl).unwrap_or(monotonic_now),
ttl, ttl,
@@ -74,6 +76,7 @@ impl LocalGuardEntry {
let now = SystemTime::now(); let now = SystemTime::now();
let monotonic_now = Instant::now(); let monotonic_now = Instant::now();
self.expires_at = now.checked_add(self.ttl).unwrap_or(now); self.expires_at = now.checked_add(self.ttl).unwrap_or(now);
self.last_refreshed = now;
self.deadline = monotonic_now.checked_add(self.ttl).unwrap_or(monotonic_now); self.deadline = monotonic_now.checked_add(self.ttl).unwrap_or(monotonic_now);
} }
} }
@@ -347,7 +350,7 @@ impl LockClient for LocalClient {
owner: entry.guard.owner().to_string(), owner: entry.guard.owner().to_string(),
acquired_at: entry.acquired_at, acquired_at: entry.acquired_at,
expires_at: entry.expires_at, expires_at: entry.expires_at,
last_refreshed: SystemTime::now(), last_refreshed: entry.last_refreshed,
metadata: LockMetadata::default(), metadata: LockMetadata::default(),
priority: LockPriority::Normal, priority: LockPriority::Normal,
wait_start_time: None, wait_start_time: None,
@@ -371,6 +374,7 @@ impl LockClient for LocalClient {
owner: entry.guard.owner().to_string(), owner: entry.guard.owner().to_string(),
acquired_at: entry.acquired_at, acquired_at: entry.acquired_at,
remaining_ttl: entry.deadline.saturating_duration_since(Instant::now()), remaining_ttl: entry.deadline.saturating_duration_since(Instant::now()),
guard_id: (!entry.guard.is_disabled()).then(|| entry.guard.guard_id()),
})); }));
} }
leases leases
@@ -484,6 +488,12 @@ mod tests {
.success .success
); );
let initial = client.list_lock_leases().await.pop().expect("acquired lock should be listed"); let initial = client.list_lock_leases().await.pop().expect("acquired lock should be listed");
let initial_status = client
.check_status(&lock_id)
.await
.expect("initial lock status should be readable")
.expect("newly acquired lock should remain held");
assert_eq!(initial_status.last_refreshed, initial_status.acquired_at);
tokio::time::advance(Duration::from_secs(20)).await; tokio::time::advance(Duration::from_secs(20)).await;
let aging = client let aging = client
@@ -492,6 +502,13 @@ mod tests {
.pop() .pop()
.expect("held lock should remain listed before refresh"); .expect("held lock should remain listed before refresh");
assert_eq!(aging.remaining_ttl, Duration::from_secs(10)); assert_eq!(aging.remaining_ttl, Duration::from_secs(10));
let aging_status = client
.check_status(&lock_id)
.await
.expect("aging lock status should be readable")
.expect("aging lock should remain held");
assert_eq!(aging_status.last_refreshed, initial_status.last_refreshed);
assert!(client.refresh(&lock_id).await.expect("refresh should return a result")); assert!(client.refresh(&lock_id).await.expect("refresh should return a result"));
let refreshed = client let refreshed = client
@@ -506,7 +523,9 @@ mod tests {
.expect("refreshed lock should remain held"); .expect("refreshed lock should remain held");
assert_eq!(refreshed.acquired_at, initial.acquired_at); assert_eq!(refreshed.acquired_at, initial.acquired_at);
assert_eq!(refreshed.guard_id, initial.guard_id);
assert_eq!(status.acquired_at, initial.acquired_at); assert_eq!(status.acquired_at, initial.acquired_at);
assert!(status.last_refreshed > initial_status.last_refreshed);
assert_eq!(refreshed.remaining_ttl, Duration::from_secs(30)); assert_eq!(refreshed.remaining_ttl, Duration::from_secs(30));
tokio::time::advance(Duration::from_secs(30)).await; tokio::time::advance(Duration::from_secs(30)).await;
+39 -6
View File
@@ -100,7 +100,7 @@ impl FastObjectLockManager {
Ok(()) => { Ok(()) => {
let guard = FastLockGuard::new(request.key, request.mode, request.owner, shard.clone()); let guard = FastLockGuard::new(request.key, request.mode, request.owner, shard.clone());
// Register guard to prevent premature cleanup // Register guard to prevent premature cleanup
shard.register_guard(guard.guard_id()); shard.register_guard_with_info(guard.guard_id(), guard.key(), guard.mode(), guard.owner());
Ok(guard) Ok(guard)
} }
Err(err) => Err(err), Err(err) => Err(err),
@@ -223,7 +223,7 @@ impl FastObjectLockManager {
if acquired { if acquired {
let guard = FastLockGuard::new(key.clone(), mode, owner.clone(), shard.clone()); let guard = FastLockGuard::new(key.clone(), mode, owner.clone(), shard.clone());
shard.register_guard(guard.guard_id()); shard.register_guard_with_info(guard.guard_id(), guard.key(), guard.mode(), guard.owner());
all_successful.push(key); all_successful.push(key);
guards.push(guard); guards.push(guard);
} }
@@ -252,7 +252,7 @@ impl FastObjectLockManager {
match shard.acquire_lock(request).await { match shard.acquire_lock(request).await {
Ok(()) => { Ok(()) => {
let guard = FastLockGuard::new(request.key.clone(), request.mode, request.owner.clone(), shard.clone()); let guard = FastLockGuard::new(request.key.clone(), request.mode, request.owner.clone(), shard.clone());
shard.register_guard(guard.guard_id()); shard.register_guard_with_info(guard.guard_id(), guard.key(), guard.mode(), guard.owner());
acquired_guards.push(guard); acquired_guards.push(guard);
} }
Err(err) => { Err(err) => {
@@ -310,6 +310,15 @@ impl FastObjectLockManager {
infos infos
} }
/// Enumerate held locks with holder counts and stable holder identities.
pub fn list_locks_with_holder_generations(&self) -> Vec<(crate::fast_lock::types::ObjectLockInfo, u32, Option<Vec<u64>>)> {
let mut infos = Vec::new();
for shard in &self.shards {
infos.extend(shard.list_locks_with_holder_generations());
}
infos
}
/// Force-release every holder of the lock on `key`. /// Force-release every holder of the lock on `key`.
/// ///
/// Returns the number of owners released (0 if the resource was not locked). /// Returns the number of owners released (0 if the resource was not locked).
@@ -556,15 +565,15 @@ mod tests {
let write_key = ObjectKey::new("bucket", "write-object"); let write_key = ObjectKey::new("bucket", "write-object");
let read_key = ObjectKey::new("bucket", "read-object"); let read_key = ObjectKey::new("bucket", "read-object");
let _write_guard = manager let write_guard = manager
.acquire_write_lock(write_key.clone(), "writer") .acquire_write_lock(write_key.clone(), "writer")
.await .await
.expect("write lock should acquire"); .expect("write lock should acquire");
let _read_guard = manager let read_guard = manager
.acquire_read_lock(read_key.clone(), "reader") .acquire_read_lock(read_key.clone(), "reader")
.await .await
.expect("read lock should acquire"); .expect("read lock should acquire");
let _second_read_guard = manager let second_read_guard = manager
.acquire_read_lock(read_key.clone(), "reader") .acquire_read_lock(read_key.clone(), "reader")
.await .await
.expect("second read lock should acquire"); .expect("second read lock should acquire");
@@ -593,6 +602,30 @@ mod tests {
.expect("write holder count listed"); .expect("write holder count listed");
assert_eq!(*write_holder_count, 1); assert_eq!(*write_holder_count, 1);
let generations = manager.list_locks_with_holder_generations();
let (_, _, read_generations) = generations
.iter()
.find(|(info, _, _)| info.key == read_key)
.expect("read holder generations listed");
let mut expected_read_generations = vec![read_guard.guard_id(), second_read_guard.guard_id()];
expected_read_generations.sort_unstable();
assert_eq!(read_generations.as_ref(), Some(&expected_read_generations));
let (_, _, write_generations) = generations
.iter()
.find(|(info, _, _)| info.key == write_key)
.expect("write holder generation listed");
assert_eq!(write_generations.as_ref(), Some(&vec![write_guard.guard_id()]));
drop(read_guard);
let remaining = manager.list_locks_with_holder_generations();
let (_, remaining_count, remaining_generations) = remaining
.iter()
.find(|(info, _, _)| info.key == read_key)
.expect("remaining read holder generation listed");
assert_eq!(*remaining_count, 1);
assert_eq!(remaining_generations.as_ref(), Some(&vec![second_read_guard.guard_id()]));
manager.shutdown().await; manager.shutdown().await;
} }
+74 -6
View File
@@ -24,7 +24,20 @@ use crate::fast_lock::{
state::ObjectLockState, state::ObjectLockState,
types::{LockMode, LockResult, ObjectKey, ObjectLockRequest}, types::{LockMode, LockResult, ObjectKey, ObjectLockRequest},
}; };
use std::collections::HashSet;
#[derive(Debug)]
struct ActiveGuardInfo {
key: ObjectKey,
mode: LockMode,
owner: Arc<str>,
}
#[derive(Debug, PartialEq, Eq, Hash)]
struct GuardHolderKey {
key: ObjectKey,
mode: LockMode,
owner: Arc<str>,
}
/// Lock shard to reduce global contention /// Lock shard to reduce global contention
#[derive(Debug)] #[derive(Debug)]
@@ -38,7 +51,7 @@ pub struct LockShard {
/// Shard ID for debugging /// Shard ID for debugging
_shard_id: usize, _shard_id: usize,
/// Active guard IDs to prevent cleanup of locks with live guards /// Active guard IDs to prevent cleanup of locks with live guards
active_guards: parking_lot::Mutex<HashSet<u64>>, active_guards: parking_lot::Mutex<HashMap<u64, Option<ActiveGuardInfo>>>,
} }
/// Cancellation-safe waiter counter ticket. /// Cancellation-safe waiter counter ticket.
@@ -84,7 +97,7 @@ impl LockShard {
object_pool: ObjectStatePool::new(), object_pool: ObjectStatePool::new(),
metrics: ShardMetrics::new(), metrics: ShardMetrics::new(),
_shard_id: shard_id, _shard_id: shard_id,
active_guards: parking_lot::Mutex::new(HashSet::new()), active_guards: parking_lot::Mutex::new(HashMap::new()),
} }
} }
@@ -327,7 +340,7 @@ impl LockShard {
// First, try to remove the guard from active set // First, try to remove the guard from active set
let guard_was_active = { let guard_was_active = {
let mut guards = self.active_guards.lock(); let mut guards = self.active_guards.lock();
guards.remove(&guard_id) guards.remove(&guard_id).is_some()
}; };
// If guard was not active, this is a double-release attempt // If guard was not active, this is a double-release attempt
@@ -375,8 +388,19 @@ impl LockShard {
/// Register a guard to prevent premature cleanup /// Register a guard to prevent premature cleanup
pub fn register_guard(&self, guard_id: u64) { pub fn register_guard(&self, guard_id: u64) {
self.active_guards.lock().insert(guard_id, None);
}
pub(crate) fn register_guard_with_info(&self, guard_id: u64, key: &ObjectKey, mode: LockMode, owner: &Arc<str>) {
let mut guards = self.active_guards.lock(); let mut guards = self.active_guards.lock();
guards.insert(guard_id); guards.insert(
guard_id,
Some(ActiveGuardInfo {
key: key.clone(),
mode,
owner: owner.clone(),
}),
);
} }
/// Unregister a guard (called when guard is dropped) /// Unregister a guard (called when guard is dropped)
@@ -396,7 +420,7 @@ impl LockShard {
#[cfg(test)] #[cfg(test)]
pub fn is_guard_active(&self, guard_id: u64) -> bool { pub fn is_guard_active(&self, guard_id: u64) -> bool {
let guards = self.active_guards.lock(); let guards = self.active_guards.lock();
guards.contains(&guard_id) guards.contains_key(&guard_id)
} }
/// Calculate adaptive timeout based on current system load and request priority /// Calculate adaptive timeout based on current system load and request priority
@@ -602,6 +626,50 @@ impl LockShard {
infos infos
} }
pub(crate) fn list_locks_with_holder_generations(
&self,
) -> Vec<(crate::fast_lock::types::ObjectLockInfo, u32, Option<Vec<u64>>)> {
// Snapshot lock state before guard registrations. Acquires register after
// mutating state, while releases unregister before mutating state, so a
// concurrent transition can only make the cohort mismatch and fall back.
let infos = self.list_locks_with_holder_counts();
let guards = self.active_guards.lock();
let mut guard_ids_by_holder: HashMap<GuardHolderKey, Vec<u64>> = HashMap::with_capacity(guards.len());
for (&guard_id, guard) in guards
.iter()
.filter_map(|(guard_id, guard)| guard.as_ref().map(|guard| (guard_id, guard)))
{
let key = GuardHolderKey {
key: guard.key.clone(),
mode: guard.mode,
owner: guard.owner.clone(),
};
guard_ids_by_holder
.entry(key)
.and_modify(|guard_ids| guard_ids.push(guard_id))
.or_insert_with(|| vec![guard_id]);
}
drop(guards);
for guard_ids in guard_ids_by_holder.values_mut() {
guard_ids.sort_unstable();
}
infos
.into_iter()
.map(|(info, holder_count)| {
let key = GuardHolderKey {
key: info.key.clone(),
mode: info.mode,
owner: info.owner.clone(),
};
let generation = guard_ids_by_holder
.remove(&key)
.filter(|guard_ids| u32::try_from(guard_ids.len()).ok() == Some(holder_count));
(info, holder_count, generation)
})
.collect()
}
/// Force-release every holder of a lock on `key`, regardless of owner. /// Force-release every holder of a lock on `key`, regardless of owner.
/// ///
/// Returns the number of owners that were released. Used by the admin /// Returns the number of owners that were released. Used by the admin
+1 -1
View File
@@ -257,7 +257,7 @@ impl std::fmt::Display for ObjectKey {
} }
/// Lock type for object operations /// Lock type for object operations
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]
pub enum LockMode { pub enum LockMode {
/// Shared lock for read operations /// Shared lock for read operations
Shared, Shared,
+2
View File
@@ -90,6 +90,8 @@ pub struct LockLeaseInfo {
pub owner: String, pub owner: String,
/// Original acquisition time. Refreshes do not change this value. /// Original acquisition time. Refreshes do not change this value.
pub acquired_at: SystemTime, pub acquired_at: SystemTime,
/// Opaque guard identity used to reject stale diagnostic snapshots.
pub guard_id: Option<u64>,
/// Remaining lease duration derived from the monotonic lease deadline. /// Remaining lease duration derived from the monotonic lease deadline.
pub remaining_ttl: Duration, pub remaining_ttl: Duration,
} }
@@ -1,566 +1,568 @@
# RustFS heal / scanner 全量功能分析与 MinIO 对标(v2) # RustFS heal & scanner vs MinIO — comprehensive parity analysis (v2, 2026-08-16)
- 日期:2026-08-16(基于 main 分支当日代码,审计时 HEAD ≈ `a118d7e4f` > English | [中文版](rustfs-heal-scanner-vs-minio-comprehensive-analysis-2026-08-16_zh.md)
- 范围:`crates/heal`src 19,560 行 + tests 2,274 行)、`crates/scanner`src 约 26,000 行 + tests)、`crates/data-usage``crates/ecstore` 中 heal/heal_walk/bitrot_self_verify 与 config、`crates/common/src/heal_channel.rs``crates/madmin`heal/scanner wire 类型)、`rustfs/src`startup wiring、admin handlers、集群 RPC
- 对标基线:minio/minio masterHEAD `7aac2a2c5b`,仓库已进入维护模式,master 冻结,即最终态) - Date: 2026-08-16 (based on that day's `main` code; audit HEAD ≈ `a118d7e4f`)
- 方法:四路并行审计(heal crate / scanner crate / ecstore 集成层 / MinIO 源码研究),关键结论逐条人工抽验(文内标注"已亲验"处为一手验证) - Scope: `crates/heal` (src 19,560 lines + tests 2,274 lines), `crates/scanner` (src ~26,000 lines + tests), `crates/data-usage`, the heal/heal_walk/bitrot_self_verify and config parts of `crates/ecstore`, `crates/common/src/heal_channel.rs`, `crates/madmin` (heal/scanner wire types), `rustfs/src` (startup wiring, admin handlers, cluster RPC)
- 本文档取代 `docs/rustfs-heal-scanner-vs-minio-parity-assessment.md`2026-06-15 v1)。v1 之后 heal/scanner 相关提交超过 80 个(换盘自动修复全链路、resume 状态机、usage 收敛权威化、集群级 heal 协调、ILM restore 语义等),v1 的功能清单与差距判断已全面过时;v1 中"bloom filter 缺失"等结论经本次核实为**误判**(详见 §5.4)。 - Parity baseline: minio/minio master (HEAD `7aac2a2c5b`; the repo has entered maintenance mode with master frozen, i.e. its final state)
- Method: four parallel audit tracks (heal crate / scanner crate / ecstore integration layer / MinIO source study), with key conclusions verified by hand one by one (points marked "verified first-hand" below were checked against the source directly)
- This document supersedes `docs/rustfs-heal-scanner-vs-minio-parity-assessment.md` (2026-06-15, v1). Since v1 there have been more than 80 heal/scanner commits (the full automatic drive-replacement healing chain, the resume state machine, making usage convergence authoritative, cluster-level heal coordination, ILM restore semantics, etc.), so v1's feature inventory and gap judgments are comprehensively outdated; v1 conclusions such as "bloom filter missing" were verified this round to be **misjudgments** (see §5.4).
--- ---
## 0. 结论摘要 ## 0. Conclusion summary
1. **总体判断:heal 与 scanner 的核心功能链路已经完整**。对象级 healquorum 仲裁 + ETag 兜底 + bitrot Deep 校验 + dangling 处理)、erasure set 深扫(per-set disk-walk 并集枚举)、按版本断点续扫(schema 化持久层 + CAS 原子发布 + 崩溃窗口补齐)、换盘自动修复(readiness 校验 + 身份围栏 + durable intent + completion proof)、scanner 周期循环(leader lock + 持久化 leader-epoch 围栏)、data usage 统计(桶级/集群级、主+备+观测快照、epoch/cycle 防回退)、ILM 全动作(expiry/transition/noncurrent/free-version/delete-marker 清理)、admin Start/Query/Cancel 协议(clientToken 语义对齐 madmin)——以上均有实现且带回归测试。两个 crate 内**没有空实现/早退桩**,异常路径全部有日志 + 指标 + 错误语义。 1. **Overall verdict: the core functional chains of heal and scanner are complete.** Object-level heal (quorum arbitration + ETag fallback + bitrot Deep verification + dangling handling), erasure set deep scans (per-set disk-walk union enumeration), per-version resumable scans (schema'd persistence layer + CAS atomic publish + crash-window backfill), automatic drive-replacement healing (readiness validation + identity fencing + durable intent + completion proof), the scanner cycle loop (leader lock + persisted leader-epoch fence), data usage statistics (bucket-level/cluster-level, primary + backup + observed snapshots, epoch/cycle anti-rollback), the full ILM action set (expiry/transition/noncurrent/free-version/delete-marker cleanup), and the admin Start/Query/Cancel protocol (clientToken semantics aligned with madmin) — all of these are implemented and carry regression tests. There are **no empty implementations / early-return stubs** inside the two crates; every exceptional path has logs + metrics + error semantics.
2. **主要缺口集中在"入口与观测面",而不是修复算法本身**MRF/ECDecode/Metadata 三类任务执行体已实现但无生产触发入口(`HealEvent` 完全未接线);`CheckAbandonedParts` 在 ecstore 三层全部 `NotImplemented`heal/scanner trace 通道缺失;scanner 超限 S3 事件缺失;madmin 客户端方法缺失(只有 wire 类型);heal 字节级进度/ETA 未实现。 2. **The main gaps concentrate on "entry points and the observability surface", not on the repair algorithms themselves**: the MRF/ECDecode/Metadata task executors are implemented but have no production trigger entry (`HealEvent` is entirely unwired); `CheckAbandonedParts` is `NotImplemented` at all three ecstore layers; the heal/scanner trace channels are missing; scanner excess S3 events are missing; madmin client methods are missing (only wire types exist); heal byte-level progress/ETA is not implemented.
3. **与 v1 认知的重要修正**bloom filter 在 MinIO 当前 master **已删除**`.bloomcycle.bin` 只存 cycle 计数),RustFS 现状与 MinIO 一致;MinIO scanner 同样是**集群级 leader 单例**RustFS leader.lock 模型与 MinIO 同型;RustFS ETag 多数派兜底仲裁已实现(`crates/ecstore/src/set_disk/ops/heal.rs:525-567,679`,已亲验),v1 担心的仲裁缺口不存在。 3. **Important corrections to the v1 understanding**: the bloom filter has been **removed** from current MinIO master (`.bloomcycle.bin` stores only a cycle count), so RustFS's current state matches MinIO; the MinIO scanner is likewise a **cluster-level leader singleton**, and RustFS's leader.lock model is the same shape as MinIO's; RustFS's ETag majority-fallback arbitration is already implemented (`crates/ecstore/src/set_disk/ops/heal.rs:525-567,679`, verified first-hand) — the arbitration gap v1 worried about does not exist.
4. **RustFS 在多处超出 MinIO**remote_scanner RPC 协议(远端 peer 本地扫描而非 leader 跨网读远盘)、持久化 leader-epoch CAS 围栏、周期预算与 per-set/per-disk 并发闸、pending-heal 账本、durable replacement intent + completion proof 状态机、前台压力门控(mainline throttle)、集群 heal control coordinator + envelope 重放防护。 4. **RustFS exceeds MinIO in several places**: the remote_scanner RPC protocol (remote peers scan locally instead of the leader reading remote drives across the network), the persisted leader-epoch CAS fence, cycle budgets and per-set/per-disk concurrency gates, the pending-heal ledger, the durable replacement intent + completion proof state machine, foreground pressure gating (mainline throttle), and the cluster heal control coordinator + envelope replay protection.
5. 差距分级统计:P1(行为/运维对齐缺口)8 项,P2(完善性)9 项,P3(清理/低风险)3 项,"按设计不追平"7 项。完整清单见 §6 5. Gap severity tally: 8 P1 items (behavioral/operational alignment gaps), 9 P2 items (completeness), 3 P3 items (cleanup/low risk), and 7 items of "not pursuing parity by design". Full list in §6.
--- ---
## 1. 架构总览 ## 1. Architecture overview
### 1.1 RustFS 三层架构 ### 1.1 RustFS's three-layer architecture
RustFS 把 MinIO 在 `cmd/` 内单体的 heal/scanner 拆成三层 + 两个独立 crate RustFS splits the heal/scanner functionality that MinIO keeps inside the `cmd/` monolith into three layers plus two standalone crates:
| 层 | 位置 | 职责 | | Layer | Location | Responsibilities |
|---|---|---| |---|---|---|
| 原语层 | `crates/ecstore/src/set_disk/ops/heal.rs`~3,240 行)、`ops/heal_walk.rs``ops/bitrot_self_verify.rs`;上层封装 `store/heal.rs``store/heal_walk.rs``core/sets.rs` | 对象/桶/format/替换盘格式修复、disk-walk 并集枚举、写入路径 bitrot 自校验;由 `SetDisks`/`Sets`/`ECStore` 实现 `rustfs_storage_api::HealOperations` 契约(`crates/storage-api/src/object.rs:503-519` | | Primitives layer | `crates/ecstore/src/set_disk/ops/heal.rs` (~3,240 lines), `ops/heal_walk.rs`, `ops/bitrot_self_verify.rs`; upper wrappers `store/heal.rs`, `store/heal_walk.rs`, `core/sets.rs` | Object/bucket/format/replacement-drive format repair, disk-walk union enumeration, write-path bitrot self-verification; the `rustfs_storage_api::HealOperations` contract is implemented by `SetDisks`/`Sets`/`ECStore` (`crates/storage-api/src/object.rs:503-519`) |
| heal 运行时 | `crates/heal` | 进程级 HealManager(优先级队列/调度器/auto disk scanner/断点续传 resume)、HealChannelProcessor(消费全局 heal channel)、换盘替换恢复状态机 | | heal runtime | `crates/heal` | Process-level HealManager (priority queue/scheduler/auto disk scanner/resumable resume), HealChannelProcessor (consumes the global heal channel), drive-replacement recovery state machine |
| scanner 运行时 | `crates/scanner` | 数据使用扫描、ILM 评估与入队、heal 候选生产、复制用量统计、remote scanner RPC | | scanner runtime | `crates/scanner` | Data usage scanning, ILM evaluation and enqueueing, heal candidate production, replication usage statistics, remote scanner RPC |
| 共享协议 | `crates/common/src/heal_channel.rs`~776 行) | Start/Query/Cancel 命令通道、`HealOpts`/`HealScanMode`/`HealRequestSource`/`HealAdmission*` 共享类型、`HealResultItem`madmin | | Shared protocol | `crates/common/src/heal_channel.rs` (~776 lines) | Start/Query/Cancel command channel, `HealOpts`/`HealScanMode`/`HealRequestSource`/`HealAdmission*` shared types, `HealResultItem` (madmin) |
| 共享数据 | `crates/data-usage` | `DataUsageEntry/Info`、直方图、`hash_path`scanner 产生、ecstore/admin 消费 | | Shared data | `crates/data-usage` | `DataUsageEntry/Info`, histograms, `hash_path`; produced by the scanner, consumed by ecstore/admin |
启动链路(已亲验 wiring): Startup chain (wiring verified first-hand):
1. `rustfs/src/startup_services.rs:93``init_background_service_runtime(store)` 1. `rustfs/src/startup_services.rs:93``init_background_service_runtime(store)`.
2. `rustfs/src/startup_background.rs:41-81`:创建全局 heal 服务取消令牌;读 `RUSTFS_SCANNER_ENABLED`(别名 `RUSTFS_ENABLE_SCANNER`,默认 true)与 `RUSTFS_HEAL_ENABLED`(别名 `RUSTFS_ENABLE_HEAL`,默认 true);**只要 heal scanner 任一开启就初始化 heal manager**scanner 产生的 heal 候选需要消费端;两者都关时 heal channel 不初始化,`send_heal_request` "Heal channel not initialized")。 2. `rustfs/src/startup_background.rs:41-81`: create the global heal service cancel token; read `RUSTFS_SCANNER_ENABLED` (alias `RUSTFS_ENABLE_SCANNER`, default true) and `RUSTFS_HEAL_ENABLED` (alias `RUSTFS_ENABLE_HEAL`, default true); **the heal manager is initialized whenever either heal or scanner is enabled** (heal candidates produced by the scanner need a consumer; with both off, the heal channel is not initialized and `send_heal_request` reports "Heal channel not initialized").
3. `crates/heal/src/lib.rs:142-216`owned task 内原子初始化(caller 取消不会遗留半初始化 manager,`lib.rs:123-131``GLOBAL_HEAL_RUNTIME_INIT` 互斥单飞)`HealManager::start()``rustfs_common::heal_channel::init_heal_channels()` → spawn `HealChannelProcessor::start_with_receipts` 3. `crates/heal/src/lib.rs:142-216`: atomic initialization inside an owned task (a caller cancel cannot leave a half-initialized manager behind, `lib.rs:123-131`; `GLOBAL_HEAL_RUNTIME_INIT` mutex single-flight) `HealManager::start()``rustfs_common::heal_channel::init_heal_channels()` → spawn `HealChannelProcessor::start_with_receipts`.
4. `crates/heal/src/heal/manager.rs:1301-1356` `HealManager::start``start_scheduler()``manager.rs:2394-2461`interval 默认 10s + `Notify` 事件驱动唤醒)`process_unclean_shutdown()``manager.rs:1362-1695`)→ `enable_auto_heal`(默认 true)时 `start_auto_disk_scanner()``manager.rs:2464-2999`)。 4. `crates/heal/src/heal/manager.rs:1301-1356` `HealManager::start`: `start_scheduler()` (`manager.rs:2394-2461`, interval default 10s + `Notify` event-driven wakeup) `process_unclean_shutdown()` (`manager.rs:1362-1695`) → when `enable_auto_heal` (default true), `start_auto_disk_scanner()` (`manager.rs:2464-2999`).
5. server ready `rustfs/src/startup_lifecycle.rs:150-152``enable_scanner` `init_data_scanner(token, store)``crates/scanner/src/scanner.rs:1293-1372`)。 5. After the server is ready, `rustfs/src/startup_lifecycle.rs:150-152`: when `enable_scanner`, `init_data_scanner(token, store)` (`crates/scanner/src/scanner.rs:1293-1372`).
6. 优雅停机:`rustfs/src/startup_shutdown.rs:308` `shutdown_ahm_services()`(取消令牌);`:414` `clear_unclean_shutdown_markers()` 6. Graceful shutdown: `rustfs/src/startup_shutdown.rs:308` `shutdown_ahm_services()` (cancel token); `:414` `clear_unclean_shutdown_markers()`.
### 1.2 MinIO 对应结构(master 最终态) ### 1.2 MinIO's corresponding structure (final master state)
| MinIO 文件 | 职责 | | MinIO file | Responsibilities |
|---|---| |---|---|
| `cmd/admin-heal-ops.go` | 手动 admin heal 序列(healSequenceclientToken/forceStart/forceStop | | `cmd/admin-heal-ops.go` | Manual admin heal sequence (healSequence, clientToken/forceStart/forceStop) |
| `cmd/global-heal.go` | 常驻后台 heal 队列(newBgHealSequencetoken 固定 `0000-…`,永不结束)+ `healErasureSet`(逐 set 全量对象 heal | | `cmd/global-heal.go` | Resident background heal queue (newBgHealSequence, token fixed `0000-…`, never ends) + `healErasureSet` (full-object heal per set) |
| `cmd/background-heal-ops.go` | healRoutine worker 池(`_MINIO_HEAL_WORKERS`,默认 GOMAXPROCS/2)消费 healTask | | `cmd/background-heal-ops.go` | healRoutine worker pool (`_MINIO_HEAL_WORKERS`, default GOMAXPROCS/2) consuming healTask |
| `cmd/mrf.go` | MRFMost Recent Fail)队列(容量 100,000),进程退出时持久化 `.minio.sys/buckets/.heal/mrf/list.bin` 并启动回放 | | `cmd/mrf.go` | MRF (Most Recent Fail) queue (capacity 100,000), persisted at process exit to `.minio.sys/buckets/.heal/mrf/list.bin` with startup replay |
| `cmd/background-newdisks-heal-ops.go` | 新盘/换盘自动 resyncmonitorLocalDisksAndHeal 10s 轮询 + healFreshDisk + healingTracker | | `cmd/background-newdisks-heal-ops.go` | Automatic resync for new/replaced drives (monitorLocalDisksAndHeal 10s polling + healFreshDisk + healingTracker) |
| `cmd/erasure-healing.go` / `erasure-healing-common.go` | 对象级 heal 核心(~800 行)、listAndHeal | | `cmd/erasure-healing.go` / `erasure-healing-common.go` | Object-level heal core (~800 lines), listAndHeal |
| `cmd/data-scanner.go` | scanner 循环(globalLeaderLock 集群单例)+ folderScanner + applyActions | | `cmd/data-scanner.go` | Scanner loop (globalLeaderLock cluster singleton) + folderScanner + applyActions |
| `cmd/erasure.go`nsScanner/ `erasure-server-pool.go` | NSScanner 三层结构 | | `cmd/erasure.go` (nsScanner) / `erasure-server-pool.go` | NSScanner three-layer structure |
| `cmd/bucket-lifecycle.go` | ILM 执行器(expiry/transition worker 池) | | `cmd/bucket-lifecycle.go` | ILM executor (expiry/transition worker pools) |
| `cmd/xl-storage.go` | DiskInfo.HealingCheckParts/VerifyFileCleanAbandonedDataRenameData healing 分支 | | `cmd/xl-storage.go` | DiskInfo.Healing, CheckParts/VerifyFile, CleanAbandonedData, RenameData healing branch |
| `cmd/prepare-storage.go` | waitForFormatErasure 新盘启动握手 | | `cmd/prepare-storage.go` | waitForFormatErasure new-drive startup handshake |
### 1.3 架构级差异(设计取舍,非缺陷) ### 1.3 Architecture-level differences (design trade-offs, not defects)
1. **heal 队列模型**MinIO 所有 healscanner 抽样/MRF/admin/新盘 resync)汇入单 channel + 固定 worker 池(新盘 resync 另有 per-drive worker 池);RustFS 是优先级堆 + 去重合并 + 容量分级丢弃 + per-set bulkhead + 前台压力门控的多策略调度器(`manager.rs:3003-3420`)。RustFS 表达力更强,代价是"重复请求被合并"的可观测性问题(v1 已指出,现有 `HealAdmissionReceipt` canonical task_id + alias 机制回应了它,`manager.rs:1759-1846`)。 1. **heal queue model**: MinIO funnels every heal (scanner sampling/MRF/admin/new-disk resync) into a single channel + a fixed worker pool (new-disk resync additionally has a per-drive worker pool); RustFS is a multi-policy scheduler built from a priority heap + dedup-merge + capacity-tiered dropping + per-set bulkhead + foreground pressure gating (`manager.rs:3003-3420`). RustFS is more expressive, at the cost of an observability question around "duplicate requests being merged" (already pointed out in v1; the current `HealAdmissionReceipt` canonical task_id + alias mechanism answers it, `manager.rs:1759-1846`).
2. **scanner 远端盘访问**MinIO leader 通过磁盘抽象层透明读写远端节点磁盘;RustFS leader 通过 remote_scanner RPC 把扫描执行下放到远端 peer 本地进行(`crates/scanner/src/remote_scanner.rs`),只回传结果与进度心跳。两者都是集群单 leaderRustFS 方案省 leader↔远端的元数据读放大,代价是需要维护独立 RPC 协议(HMAC 逐帧认证、会话重放缓存、fence 复验,`remote_scanner.rs:52-61,405-496,1024-1065`)。 2. **scanner remote-drive access**: the MinIO leader transparently reads and writes remote-node drives through the disk abstraction layer; the RustFS leader pushes scan execution down to the remote peer to run locally via the remote_scanner RPC (`crates/scanner/src/remote_scanner.rs`), with only results and progress heartbeats sent back. Both are cluster single-leader. RustFS's approach saves the leader↔remote metadata read amplification, at the cost of maintaining a separate RPC protocol (HMAC per-frame authentication, session replay cache, fence re-validation, `remote_scanner.rs:52-61,405-496,1024-1065`).
3. **heal 状态持久化**MinIO 用单文件 `.healing.bin`msgp healingTrackerdiskID 不匹配即重置);RustFS schema 化多文件(resume/checkpoint/intent/seal/proof 各自 CAS 发布,`resume.rs:38-61`),崩溃窗口显式补齐(`erasure_healer.rs:389-402``resume.rs:1027-1057`)。 3. **heal state persistence**: MinIO uses a single file `.healing.bin` (msgp healingTracker, reset whenever the diskID mismatches); RustFS uses a schema'd multi-file layout (resume/checkpoint/intent/seal/proof, each CAS-published, `resume.rs:38-61`), with the crash window explicitly backfilled (`erasure_healer.rs:389-402`, `resume.rs:1027-1057`).
4. **写路径自保护**MinIO 写入后靠后台 heal 收敛;RustFS 在 PutObject/CompleteMultipartUpload 提交 rename 后主动检查 `convergence.needs_heal()` 并立即入队对象 heal`set_disk/ops/object.rs:2291-2306``ops/multipart.rs:2574-2589`),另有读修复 read repair`io_primitives.rs:1040-1160`)。 4. **write-path self-protection**: MinIO relies on background heal to converge after writes; RustFS, after the commit rename in PutObject/CompleteMultipartUpload, actively checks `convergence.needs_heal()` and immediately enqueues an object heal (`set_disk/ops/object.rs:2291-2306`, `ops/multipart.rs:2574-2589`), and additionally has read repair (`io_primitives.rs:1040-1160`).
--- ---
## 2. Heal 已实现功能全景 ## 2. Heal implemented-feature panorama
### 2.1 任务类型(`HealType``crates/heal/src/heal/task.rs:85-111` ### 2.1 Task types (`HealType`, `crates/heal/src/heal/task.rs:85-111`)
| 类型 | 语义 | 执行体 | 生产触发方 | | Type | Semantics | Executor | Production trigger |
|---|---|---|---| |---|---|---|---|
| `Cluster` | 所有 bucket 依次 heal(结构 + 可选递归对象),批内重试 ≤3 | `heal_cluster` task.rs:1420-1490 | channelbucket 为空即 Clusterchannel.rs:576-577 | | `Cluster` | all buckets healed in turn (structure + optional recursive objects), in-batch retry ≤3 | `heal_cluster` task.rs:1420-1490 | channel: empty bucket means Cluster (channel.rs:576-577) |
| `Object{bucket,object,version_id}` | 单对象/版本;不存在时按 `recreate_missing` 重建或报错 | `heal_object` task.rs:855-1146 | adminscannerread-repair、写路径收敛、add_partial | | `Object{bucket,object,version_id}` | single object/version; when absent, rebuild per `recreate_missing` or error out | `heal_object` task.rs:855-1146 | admin, scanner, read-repair, write-path convergence, add_partial |
| `Bucket{bucket}` | 桶元数据/结构;`recursive` 再遍历全部对象版本 | `heal_bucket` task.rs:1284-1418 + `heal_bucket_objects` task.rs:1508-1698 | adminPOST /v3/heal/{bucket})、scanner `build_bucket_heal_request` | | `Bucket{bucket}` | bucket metadata/structure; `recursive` additionally walks all object versions | `heal_bucket` task.rs:1284-1418 + `heal_bucket_objects` task.rs:1508-1698 | admin (POST /v3/heal/{bucket}), scanner `build_bucket_heal_request` |
| `Prefix{bucket,prefix}` | 按前缀递归 | `heal_prefix` task.rs:1492-1506 | channel`recursive && prefix` 非空(channel.rs:578-585 | | `Prefix{bucket,prefix}` | recursive by prefix | `heal_prefix` task.rs:1492-1506 | channel: `recursive && prefix` non-empty (channel.rs:578-585) |
| `ErasureSet{buckets,set_disk_id}` | format 修复 + healing 标记 + 逐桶预处理 + 可恢复逐版本深扫 | `heal_erasure_set` task.rs:2158-2642 | adminpool/set 参数)、auto disk scannerunclean shutdownrenew_diskdurable replacement 恢复 | | `ErasureSet{buckets,set_disk_id}` | format repair + healing marker + per-bucket preprocessing + resumable per-version deep scan | `heal_erasure_set` task.rs:2158-2642 | admin (pool/set params), auto disk scanner, unclean shutdown, renew_disk, durable replacement recovery |
| `Metadata{bucket,object}` | 仅元数据(Deep、不重建数据) | `heal_metadata` task.rs:1700-1859 | **无生产触发方**§6 HS-01 | | `Metadata{bucket,object}` | metadata only (Deep, does not rebuild data) | `heal_metadata` task.rs:1700-1859 | **no production trigger** (§6 HS-01) |
| `MRF{meta_path}` | 失败路径驱动的 Deep 修复(recursive+update_parity | `heal_mrf` task.rs:1861-1992 | **无生产触发方**(仅 `HealEvent` 可生成,未接线) | | `MRF{meta_path}` | failure-path-driven Deep repair (recursive+update_parity) | `heal_mrf` task.rs:1861-1992 | **no production trigger** (only `HealEvent` can generate it, unwired) |
| `ECDecode{bucket,object,version_id}` | EC 解码重建(Deep+recreate+update_parity),Urgent 优先级 | `heal_ec_decode` task.rs:1994-2156 | **无生产触发方**(仅 `HealEvent` 可生成,未接线) | | `ECDecode{bucket,object,version_id}` | EC decode rebuild (Deep+recreate+update_parity), Urgent priority | `heal_ec_decode` task.rs:1994-2156 | **no production trigger** (only `HealEvent` can generate it, unwired) |
优先级 `Low/Normal/High/Urgent`task.rs:168-179);状态机 `Pending/Running/Retrying/Completed/Failed/Cancelled/Timeout`task.rs:225-241)。 Priorities `Low/Normal/High/Urgent` (task.rs:168-179); state machine `Pending/Running/Retrying/Completed/Failed/Cancelled/Timeout` (task.rs:225-241).
### 2.2 触发路径全景(admin 之外) ### 2.2 Trigger-path panorama (beyond admin)
| 通道 | source | 优先级 | 证据 | | Channel | source | Priority | Evidence |
|---|---|---|---| |---|---|---|---|
| Scanner 周期抽样(1/1024`RUSTFS_HEAL_OBJECT_SELECT_PROB` | Scanner | Low | `scanner_folder.rs:2117-2136``:1150``remove_corrupted=HEAL_DELETE_DANGLING(true)``recreate_missing=false``common/heal_channel.rs:24``scanner_folder.rs:510-511` | | Scanner periodic sampling (1/1024, `RUSTFS_HEAL_OBJECT_SELECT_PROB`) | Scanner | Low | `scanner_folder.rs:2117-2136`, `:1150`; `remove_corrupted=HEAL_DELETE_DANGLING(true)`, `recreate_missing=false` (`common/heal_channel.rs:24`, `scanner_folder.rs:510-511`) |
| Scanner 元数据损坏(get_size 失败分类 HealMetadata | Scanner | High | `scanner_folder.rs:2147-2208``:1244-1260` | | Scanner metadata corruption (get_size failure classified HealMetadata) | Scanner | High | `scanner_folder.rs:2147-2208`, `:1244-1260` |
| Scanner abandoned children(缓存有、盘上无,list_path_raw quorum 核查) | Scanner | High(桶级+对象级) | `scanner_folder.rs:2528-2792` | | Scanner abandoned children (present in cache, absent on disk, list_path_raw quorum verification) | Scanner | High (bucket-level + object-level) | `scanner_folder.rs:2528-2792` |
| Scanner pending-heal 账本重试(heal 通道满被拒后持久化,每桶每轮 ≤128 条、上限 10k) | Scanner | 原优先级 | `scanner_folder.rs:1721-1763``:99-100` | | Scanner pending-heal ledger retry (persisted after rejection by a full heal channel, ≤128 per bucket per round, 10k cap) | Scanner | original priority | `scanner_folder.rs:1721-1763`, `:99-100` |
| auto disk scannerunformatted 盘经 replacement_readiness 确认 / `runtime_state=="returning"` / durable intent 重入) | AutoHeal | Low | `manager.rs:2464-2999` | | auto disk scanner (unformatted drive confirmed via replacement_readiness / `runtime_state=="returning"` drive / durable-intent re-entry) | AutoHeal | Low | `manager.rs:2464-2999` |
| unclean shutdown 恢复(启动读 `unclean-shutdown` 标记 → 全部本地 set ErasureSet heal | AutoHeal | Low | `manager.rs:1362-1695` | | unclean shutdown recovery (startup reads the `unclean-shutdown` marker → ErasureSet heal for all local sets) | AutoHeal | Low | `manager.rs:1362-1695` |
| 写路径收敛(PutObject/CompleteMultipartUpload `convergence.needs_heal()` | Internal | Normal | `set_disk/ops/object.rs:2291-2306``ops/multipart.rs:2574-2589` | | write-path convergence (after PutObject/CompleteMultipartUpload, `convergence.needs_heal()`) | Internal | Normal | `set_disk/ops/object.rs:2291-2306`, `ops/multipart.rs:2574-2589` |
| 部分对象 healadd_partial | Internal | Normal | `set_disk/ops/object.rs:5808-5825` | | partial-object heal (add_partial) | Internal | Normal | `set_disk/ops/object.rs:5808-5825` |
| 旧数据目录清理残留 enqueue | Internal | Normal | `set_disk/core/io_primitives.rs:3880-3907` | | stale data-directory cleanup leftover enqueue | Internal | Normal | `set_disk/core/io_primitives.rs:3880-3907` |
| 读修复(metadata_read_error / missing_shards / decode_errorTTL 去重缓存) | ReadRepair | Low | `set_disk/read.rs:407,995,1079``submit_read_repair_heal``io_primitives.rs:1105-1160`),`recreate_missing=true` | | read repair (metadata_read_error / missing_shards / decode_error, TTL dedup cache) | ReadRepair | Low | `set_disk/read.rs:407,995,1079``submit_read_repair_heal` (`io_primitives.rs:1105-1160`), `recreate_missing=true` |
| 盘重连遇 UnformattedDisk → send_heal_disk | AutoHeal | Normal | `set_disk/ops/locking.rs:339-347` | | drive reconnect hits UnformattedDisk → send_heal_disk | AutoHeal | Normal | `set_disk/ops/locking.rs:339-347` |
| Admin API(含集群 coordinator 路由) | Admin | High | `rustfs/src/admin/handlers/heal.rs:174-212``:771-930` | | Admin API (incl. cluster coordinator routing) | Admin | High | `rustfs/src/admin/handlers/heal.rs:174-212`, `:771-930` |
| 集群 RPC healpeer 调用) | — | — | `rustfs/src/storage/rpc/node_service/heal.rs``ecstore/src/cluster/rpc/peer_s3_client.rs:296,1209` | | cluster RPC heal (peer invocation) | — | — | `rustfs/src/storage/rpc/node_service/heal.rs`, `ecstore/src/cluster/rpc/peer_s3_client.rs:296,1209` |
注意:MinIO MRF 通道(读路径检出 part 缺失/损坏即时投递 + 队列持久化 + shutdown 回放,`cmd/mrf.go``erasure-object.go:395-410,800-812`)在 RustFS read-repair + 写路径收敛**部分替代**`HealType::MRF`/`ECDecode`/`Metadata` 三个执行体没有生产入口(详见 §6 HS-01)。 Note: MinIO's MRF channel (read-path immediate delivery on missing/corrupt parts + queue persistence + shutdown replay, `cmd/mrf.go`, `erasure-object.go:395-410,800-812`) is **partially replaced** in RustFS by read-repair + write-path convergence; the three executors `HealType::MRF`/`ECDecode`/`Metadata` have no production entry (see §6 HS-01 for details).
### 2.3 对象级 heal 语义(ecstore `set_disk/ops/heal.rs` ### 2.3 Object-level heal semantics (ecstore `set_disk/ops/heal.rs`)
流程(`heal_object_with_explicit_version_regen` :426 起): Flow (`heal_object_with_explicit_version_regen` from :426):
1. 取对象写锁(除非 `no_lock`);`object` `/` 结尾走对象目录 heal`heal_object_dir_locked` :1587-1717dangling 判定 + `remove` 删除 + 缺 volume 重建)。 1. Take the object write lock (unless `no_lock`); an `object` ending with `/` goes through object-directory heal (`heal_object_dir_locked` :1587-1717: dangling determination + `remove` deletion + missing-volume rebuild).
2. `read_all_fileinfo` 全盘读 xl.meta,全部 not-found 视为已删除返回。 2. `read_all_fileinfo` reads xl.meta from all disks; all-not-found is treated as already deleted and returns.
3. **quorum 仲裁 + ETag 兜底**(已亲验):`list_online_disks` mod-time quorum 为准;quorum 失效时回退 ETag 多数派仲裁(`:525-567` `filter_by_etag`/`quorum_etag`);`pick_valid_fileinfo` canonical 元数据;"meta 坏盘数 > parity" 的 cannotHeal 判定在 ETag 全盘一致时豁免(`:679`)。与 MinIO `filterDisksByETag` 双仲裁一致。 3. **quorum arbitration + ETag fallback** (verified first-hand): `list_online_disks` treats the mod-time quorum as authoritative; when quorum fails it falls back to ETag majority arbitration (`:525-567` `filter_by_etag`/`quorum_etag`); `pick_valid_fileinfo` picks the canonical metadata; the cannotHeal determination for "number of bad-meta disks > parity" is waived when the ETag agrees across all disks (`:679`). Matches MinIO's dual arbitration in `filterDisksByETag`.
4. `disks_with_all_parts`:562-572)按 `scan_mode` 校验 part**Normal statCheckParts 语义),Deep 做全量 bitrot 校验(VerifyFile 语义)**Normal 扫描检出 `FileCorrupt` 自动升级 Deep 重试一次(`:2022-2031`,与 MinIO erasure-healing.go:1101-1106 同型);无 parity 对象(EC:0)bitrot 失败判不可恢复(`:700-726`)。 4. `disks_with_all_parts` (:562-572) validates parts per `scan_mode`: **Normal only stats (CheckParts semantics), Deep does full bitrot verification (VerifyFile semantics)**; when a Normal scan detects `FileCorrupt` it automatically escalates to Deep and retries once (`:2022-2031`, same shape as MinIO erasure-healing.go:1101-1106); a no-parity object (EC:0) with a bitrot failure is judged unrecoverable (`:700-726`).
5. `should_heal_object_on_disk`:606-650)逐盘分类 missing/corrupt/offline/outdated → 重建:per-part bitrot reader/writer(用 per-part checksum + 算法)、写临时卷后 rename 提交(`HEAL_RENAME_INCOMPLETE` 重试语义 :24);dangling 删除安全检查 `dangling_delete_safety`:1488);**孤儿数据目录回收 `reclaim_orphan_data_dirs_best_effort`:1428**——这部分覆盖了 MinIO `CleanAbandonedData` 的主场景(但无独立 `CheckAbandonedParts` API,见 §6 HS-02)。 5. `should_heal_object_on_disk` (:606-650) classifies each disk as missing/corrupt/offline/outdated → rebuild: per-part bitrot reader/writer (using per-part checksum + algorithm), write into a temporary volume then rename to commit (`HEAL_RENAME_INCOMPLETE` retry semantics :24); dangling-deletion safety check `dangling_delete_safety` (:1488); **orphan data-directory reclamation `reclaim_orphan_data_dirs_best_effort` (:1428)** — this part covers the main scenarios of MinIO's `CleanAbandonedData` (but there is no standalone `CheckAbandonedParts` API, see §6 HS-02).
6. 版本化对象:枚举"每个版本"(`storage.rs:1494-1530`);delete-marker 路径由 `latest_meta.deleted` 决定(`storage.rs:262-277` 注释);回归测试 `tests/heal_b5_versioned_regression_test.rs:282,334` 6. Versioned objects: enumerate "every version" (`storage.rs:1494-1530`); the delete-marker path is decided by `latest_meta.deleted` (`storage.rs:262-277` comment); regression tests `tests/heal_b5_versioned_regression_test.rs:282,334`.
7. 显式版本重建 `try_regenerate_explicit_version_meta`:1318);transitioned 对象本地残留清理。 7. Explicit-version rebuild `try_regenerate_explicit_version_meta` (:1318); cleanup of local leftovers of transitioned objects.
8. 写入路径另有 shard 级 bitrot 自校验 `verify_written_bitrot_shards``ops/bitrot_self_verify.rs:45-129`HighwayHash256S,最终 rename 前校验刚写出的 shard,服务 EC:0 parity 场景)——**注意这不是后台 bitrot 巡检**;后台巡检由 scanner bitrot_cycle 驱动 Deep heal 承担。 8. The write path additionally has shard-level bitrot self-verification `verify_written_bitrot_shards` (`ops/bitrot_self_verify.rs:45-129`, HighwayHash256S, verifying freshly written shards right before the final rename, serving the EC:0 no-parity case) — **note this is not background bitrot patrol**; background patrol is carried by scanner bitrot_cycle-driven Deep heal.
heal crate 侧包装(`task.rs:855-1146`):存在性检查(瞬时错误转 `TransientSkip` 不误判失败 :551-569);scanner 合成目录规范化(:1148-1180);`recreate_missing` 重建(:1183-1282);data-usage-cache 对象锁超时豁免(:571-653);not-found → treated_as_deleted 成功(:1012-1029);结果 `HealResultItem` 保留至多 1024 条 + truncated 标志(:50,845-852)。 heal-crate-side wrapper (`task.rs:855-1146`): existence check (transient errors become `TransientSkip` to avoid false failures :551-569); scanner synthetic-directory normalization (:1148-1180); `recreate_missing` rebuild (:1183-1282); data-usage-cache object-lock timeout exemption (:571-653); not-found → treated_as_deleted success (:1012-1029); results `HealResultItem` keep at most 1024 entries + truncated flag (:50,845-852).
递归遍历(`heal_bucket_objects` task.rs:1508-1698):分页枚举全部版本含 delete marker、瞬时错误指数退避重试 ≤32^n + 抖动 :620-627)、失败样本日志截断 ≤5 条、聚合 `BatchHealFailure` Recursive walk (`heal_bucket_objects` task.rs:1508-1698): paginated enumeration of all versions including delete markers, transient-error exponential-backoff retry ≤3 (2^n + jitter :620-627), failure-sample log truncation ≤5 entries, aggregated `BatchHealFailure`.
### 2.4 erasure set heal 与断点续扫 ### 2.4 erasure set heal and resumable scans
`heal_erasure_set`task.rs:2158-2642)四阶段(4 步进度跟踪): `heal_erasure_set` (task.rs:2158-2642) runs in four phases (4-step progress tracking):
1. **替换意图与恢复盘选择**(仅 AutoHeal + heal_endpoints 非空):复用 durable intent 所在盘 / 排除目标端点选幸存盘;已完成代(CleanupPending)幂等收尾。 1. **Replacement intent and recovery-drive selection** (AutoHeal only + non-empty heal_endpoints): reuse the drive holding the durable intent / exclude the target endpoints and pick surviving drives; already-completed generations get an idempotent CleanupPending wrap-up.
2. **格式修复**`heal_replacement_format(dry_run, pool, set, targets)``storage.rs:1372-1384`trait 默认实现 fail-closed);逐目标盘结果必须全 ok`erasure_healer.rs:97-102`+ 身份围栏复核(task.rs:2410-2420)。 2. **Format repair**: `heal_replacement_format(dry_run, pool, set, targets)` (`storage.rs:1372-1384`, trait default fail-closed); per-target-drive results must all be ok (`erasure_healer.rs:97-102`) + identity-fence re-check (task.rs:2410-2420).
3. **healing 标记**:对目标盘写 owner CAS 标记 `{set_disk_id}:{task_id}``mod.rs:80-229`CAS + 回滚 + 并发唯一 owner),使 `DiskInfo.healing` 为真(已亲验赋值链 `set_disk/mod.rs:4988`)。 3. **healing marker**: write an owner CAS marker `{set_disk_id}:{task_id}` to the target drive (`mod.rs:80-229`, CAS + rollback + unique concurrent owner), which makes `DiskInfo.healing` true (assignment chain verified first-hand `set_disk/mod.rs:4988`).
4. **逐桶预处理 + 可恢复深扫**`ErasureSetHealer::heal_erasure_set``erasure_healer.rs:242-278`)。 4. **Per-bucket preprocessing + resumable deep scan**: `ErasureSetHealer::heal_erasure_set` (`erasure_healer.rs:242-278`).
`ErasureSetHealer` 扫描细节(对标 MinIO `healErasureSet``heal_walk.rs:15-23` 模块注释明确引用 MinIO `global-heal.go` listPathRaw + objQuorum=1 + mergeXLV2Versions): `ErasureSetHealer` scan details (benchmarked against MinIO `healErasureSet`; the `heal_walk.rs:15-23` module comment explicitly cites MinIO `global-heal.go`'s listPathRaw + objQuorum=1 + mergeXLV2Versions):
- **枚举器选择(backlog#920**Deep AutoHeal → per-set **disk-walk 并集枚举** `list_versions_for_heal_page_disk_walk`"任意盘上存在"即 sub-quorum 可重建;`storage.rs:1559-1644`,页界 1000 对象/10,000 版本,`dw1:` cursor);普通请求走 read-quorum `list_object_versions` - **Enumerator choice (backlog#920)**: Deep or AutoHeal → per-set **disk-walk union enumeration** `list_versions_for_heal_page_disk_walk` ("exists on any drive" means sub-quorum reconstructible; `storage.rs:1559-1644`, page bounds 1,000 objects/10,000 versions, `dw1:` cursor); ordinary requests go through read-quorum `list_object_versions`.
- **续扫游标**:权威 cursor opaque continuation token`v1:`=marker JSON`dw1:`=disk-walk key,两命名空间互斥防误读,`storage.rs:81-260`);每完成一页先持久化 cursor 再清 dedup 集合(`erasure_healer.rs:922-927`)。 - **Resume cursor**: the authoritative cursor is an opaque continuation token (`v1:` = marker JSON, `dw1:` = disk-walk key; the two namespaces are mutually exclusive against misreads, `storage.rs:81-260`); after each completed page, persist the cursor first, then clear the dedup set (`erasure_healer.rs:922-927`).
- **页内并发**FuturesUnordered + Semaphore,默认 `RUSTFS_HEAL_PAGE_OBJECT_CONCURRENCY=8`Deep/AutoHeal 强制 1`erasure_healer.rs:105-142`)。 - **In-page concurrency**: FuturesUnordered + Semaphore, default `RUSTFS_HEAL_PAGE_OBJECT_CONCURRENCY=8`, Deep/AutoHeal forces 1 (`erasure_healer.rs:105-142`).
- **per-version dedup**`compose_key` 长度前缀注入编码(`resume.rs:281-288`)。 - **per-version dedup**: `compose_key` length-prefix injection encoding (`resume.rs:281-288`).
- **错误分类**:真缺席(FileNotFound 等)→ Absent(计成功);基础设施瞬时(quorum/DiskNotFound/SlowDown 等)→ Transient(计 skipped);其余 Failed`erasure_healer.rs:148-182`,注释引 backlog#856/#799 B7:离线盘不得记 healed/absent)。 - **Error classification**: truly absent (FileNotFound etc.) → Absent (counted as success); infrastructure-transient (quorum/DiskNotFound/SlowDown etc.) → Transient (counted as skipped); everything else Failed (`erasure_healer.rs:148-182`; the comment cites backlog#856/#799 B7: offline drives must not be recorded healed/absent).
- **防死循环**:空页 truncated 或页尾版本身份不前进即中止(:933-949)。 - **Loop protection**: abort when an empty page is truncated or the page-tail version identity does not advance (:933-949).
- **完成判定**failed/skipped/failed_buckets 任一 >0 不标记完成,`schedule_retry()` 复位 resume+checkpoint 两层(:561-626backlog#855/B6/#1033skip 轮不得标记完成)。 - **Completion determination**: if any of failed/skipped/failed_buckets is >0, do not mark complete; `schedule_retry()` resets both the resume and checkpoint layers (:561-626; backlog#855/B6/#1033: a skip round must not be marked complete).
- **替换盘提交证据**:目标端点物理回读 `replacement_targets_have_version``ops/heal.rs:340-412`),未确认 → transient skip - **Replacement-drive commit proof**: physical read-back on the target endpoints `replacement_targets_have_version` (`ops/heal.rs:340-412`); unconfirmed → transient skip.
### 2.5 换盘自动修复(replacement recovery ### 2.5 Automatic drive-replacement healing (replacement recovery)
- **识别**`replacement_readiness.rs:25-73`):`replacement_mount_lease_root()` 存在、canonicalize 成功、是挂载点、物理设备 id 非空、与根设备不相交、不与兄弟盘共享物理设备(Linux /proc/self/mountinfo mount-id+dev+ino)。非 root 挂载检查有回归测试(`manager.rs:3549`)。 - **Identification** (`replacement_readiness.rs:25-73`): `replacement_mount_lease_root()` exists, canonicalize succeeds, is a mount point, the physical device id is non-empty, disjoint from the root device, and shares no physical device with sibling drives (Linux uses /proc/self/mountinfo mount-id+dev+ino). The non-root mount check has a regression test (`manager.rs:3549`).
- **状态机**`resume.rs:63-73`):`Intent → Rebuilding →(写 proofVerified → CleanupPending → 清理``Abandoned` 终态;跨状态迁移先写持久层再变更(`save_state_strict`)。 - **State machine** (`resume.rs:63-73`): `Intent → Rebuilding → (write proof) Verified → CleanupPending → cleanup`; `Abandoned` is a terminal state; state transitions write the persistence layer first, then mutate (`save_state_strict`).
- **持久化**`resume.rs:38-61`schema ResumeState=5/Checkpoint=5/proof=1):`{task_id}_ahm_resume_state.json``_ahm_checkpoint.json``buckets/ahm-replacement/` 命名空间下 intent/seal/completion_prooftorn write + seal 可识别并原子重建(:1316-1338);CAS 发布、拒绝覆盖并发有效 proof:1512-1585)。 - **Persistence** (`resume.rs:38-61`, schema ResumeState=5/Checkpoint=5/proof=1): `{task_id}_ahm_resume_state.json`, `_ahm_checkpoint.json`, and intent/seal/completion_proof under the `buckets/ahm-replacement/` namespace; torn write + no seal is recognizable and rebuilt atomically (:1316-1338); CAS publish, refuses to overwrite a concurrently valid proof (:1512-1585).
- **恢复**unclean shutdown 与周期扫描都从幸存盘恢复未完成/待清理替换代(`manager.rs:1435-1640,2663-2815`);多代冲突/校验失败 → 冻结该 set`replacement_recovery_blocked_sets``manager.rs:69-87,2782-2815`)。 - **Recovery**: both unclean shutdown and the periodic scan recover unfinished/pending-cleanup replacement generations from surviving drives (`manager.rs:1435-1640,2663-2815`); multi-generation conflict / validation failure → freeze that set (`replacement_recovery_blocked_sets`, `manager.rs:69-87,2782-2815`).
- **对外快照**`current_replacement_recovery_snapshot``lib.rs:262-333`)合并本地幸存盘记录,冲突 → Unknown/非 definitiveadmin `GET /v4/heal/replacement-recovery` - **External snapshot**: `current_replacement_recovery_snapshot` (`lib.rs:262-333`) merges local surviving-drive records; conflict → Unknown / non-definitive; admin `GET /v4/heal/replacement-recovery`.
### 2.6 调度器(manager.rs ### 2.6 Scheduler (manager.rs)
- 优先级堆 + 同优先级 FIFO:148-191,330-347);dedup key 按类型(:469-506);入队三态查重 active→queued→retrying:1759-1785);重复默认 Merged 并返回 canonical task_id`HealAdmissionReceipt`:1821-1846+ client token alias:1219-1246)。 - Priority heap + FIFO within the same priority (:148-191,330-347); dedup key per type (:469-506); enqueue three-state dedup active→queued→retrying (:1759-1785); duplicates default to Merged and return the canonical task_id (`HealAdmissionReceipt`, :1821-1846) + client token alias (:1219-1246).
- 容量:队列满时 best-effort 来源(Scanner/AutoHeal/ReadRepair)或低优先级被 Dropped(QueueFull)Admin/Internal 可驱逐低优先级排队项(`push_displacing_lower_priority` :353-396);80%/95% 压力分级(:885-909)。 - Capacity: when the queue is full, best-effort sources (Scanner/AutoHeal/ReadRepair) or low-priority items get Dropped(QueueFull); Admin/Internal may evict queued lower-priority items (`push_displacing_lower_priority` :353-396); 80%/95% tiered pressure handling (:885-909).
- 并发:全局 `max_concurrent_heals`(默认 4+ per-set bulkhead `max_concurrent_per_set`(默认 1)(:3040-3073,3434-3447)。 - Concurrency: global `max_concurrent_heals` (default 4) + per-set bulkhead `max_concurrent_per_set` (default 1) (:3040-3073,3434-3447).
- 前台压力门控 mainline throttle:前台读/写 permit 利用率 ≥80% 时延迟 best-effort 任务(:919-1009,2999-3020)。 - Foreground pressure gating, mainline throttle: delay best-effort tasks when foreground read/write permit utilization is ≥80% (:919-1009,2999-3020).
- 超时:任务级聚合超时(默认 300s),跨重试保留剩余预算(task.rs:444-451PR #6101)。 - Timeout: task-level aggregate timeout (default 300s), remaining budget preserved across retries (task.rs:444-451, PR #6101).
- 可恢复重试:`is_recoverable_heal()`error.rs:83-136≤3 次、2^n 退避封顶 30sretry 在独立 backoff task 中持有所有权(:3235-3382)。 - Recoverable retry: `is_recoverable_heal()` (error.rs:83-136) ≤3 attempts, 2^n backoff capped at 30s; retries hold ownership inside a standalone backoff task (:3235-3382).
- 完成态保留 10 分钟供查询(:42)。 - Completion states are retained for 10 minutes for querying (:42).
### 2.7 Admin API 与集群协调 ### 2.7 Admin API and cluster coordination
- 路由(`rustfs/src/admin/handlers/heal.rs:174-212`):`POST /rustfs/admin/v3/heal/``/heal/{bucket}``/heal/{bucket}/{prefix}`(同一 POST 按 query `clientToken/forceStart/forceStop` 区分 start/query/cancel,与 mc admin heal 语义对齐);`POST /v3/background-heal/status``GET /v4/heal/replacement-recovery`。权限 `HealAdminAction`route_policy.rs:334-341)。 - Routes (`rustfs/src/admin/handlers/heal.rs:174-212`): `POST /rustfs/admin/v3/heal/`, `/heal/{bucket}`, `/heal/{bucket}/{prefix}` (the same POST distinguishes start/query/cancel by the query `clientToken/forceStart/forceStop`, aligned with mc admin heal semantics); `POST /v3/background-heal/status`; `GET /v4/heal/replacement-recovery`. Permission `HealAdminAction` (route_policy.rs:334-341).
- 集群协调(heal.rs:771-930 + `node_service.rs:514-606`):`heal_topology_fingerprint` + 按拓扑确定性选 coordinator 节点 + coordinator epochenvelope 校验 + SHA256 digest 重放缓防重放;coordinator 非本机走 peer gRPC `heal_control``probe_heal_control` 能力探测(滚动升级场景)。 - Cluster coordination (heal.rs:771-930 + `node_service.rs:514-606`): `heal_topology_fingerprint` + deterministic-by-topology coordinator-node selection + coordinator epoch; envelope validation + SHA256 digest replay protection; when the coordinator is not local, go through peer gRPC `heal_control`; `probe_heal_control` capability probe (rolling-upgrade scenario).
- 请求:body `HealOpts``recursive/dryRun/remove/recreate/scanMode(0/1/2)/updateParity/nolock/pool/set`serde camelCase,与 madmin.HealOpts 字段对齐);根 heal start `recursive=true` `pool+set` 成对;body 上限 1MB - Request: the body is `HealOpts` (`recursive/dryRun/remove/recreate/scanMode(0/1/2)/updateParity/nolock/pool/set`, serde camelCase, fields aligned with madmin.HealOpts); a root heal start requires `recursive=true` or a `pool+set` pair; body cap 1MB.
- 响应:`HealStartSuccess{clientToken, clientAddress, startTime}``HealTaskStatus{summary, detail, startTime, settings, items, truncated, progress}`summary ∈ running/finished/stopped/notFound);`BackgroundHealStatus`bitrot 起始时间/周期/当前模式 + `disabled/uninitialized/idle/active/degraded` 状态——peer 不可达显式 degraded 不冒充 idleissue #5850 + `healOperations` 按优先级×来源矩阵 + 集群进度)。 - Response: `HealStartSuccess{clientToken, clientAddress, startTime}`; `HealTaskStatus{summary, detail, startTime, settings, items, truncated, progress}` (summary ∈ running/finished/stopped/notFound); `BackgroundHealStatus` (bitrot start time/cycle/current mode + `disabled/uninitialized/idle/active/degraded` states — an unreachable peer is explicitly degraded rather than impersonating idle, issue #5850) + `healOperations` as a priority×source matrix + cluster progress.
- `HealResultItem`/`HealDriveInfo`/`HealItemType`/DriveState 枚举与 madmin JSON 兼容(`crates/madmin/src/heal_commands.rs:19-65`)。 - `HealResultItem`/`HealDriveInfo`/`HealItemType`/DriveState enums are JSON-compatible with madmin (`crates/madmin/src/heal_commands.rs:19-65`).
- 状态 payload 8MiB 对折截断(channel.rs:37,73-104);path-token 校验(错误 token 拒绝,空 path 仅匹配 Cluster)。 - A status payload over 8MiB is truncated by halving (channel.rs:37,73-104); path-token validation (wrong token rejected; an empty path matches Cluster only).
### 2.8 heal 指标与日志 ### 2.8 heal metrics and logs
指标:`rustfs_heal_admission_total{source,result,reason,context}``rustfs_heal_task_start_total``rustfs_heal_task_running{type,set}``rustfs_heal_queue_delay_seconds``rustfs_heal_scheduler_skip_total``rustfs_heal_mainline_throttle_total``rustfs_heal_page_concurrency_current{set}``rustfs_heal_candidate_enqueue/merge/drop/priority_reject_total``rustfs_heal_read_repair_dedup_total{reason}` 等。日志全部结构化 event stylePR #5720);per-object 日志降级防风暴(`demote_to_debug_when!`#5716/#5719/#5727)。 Metrics: `rustfs_heal_admission_total{source,result,reason,context}`, `rustfs_heal_task_start_total`, `rustfs_heal_task_running{type,set}`, `rustfs_heal_queue_delay_seconds`, `rustfs_heal_scheduler_skip_total`, `rustfs_heal_mainline_throttle_total`, `rustfs_heal_page_concurrency_current{set}`, `rustfs_heal_candidate_enqueue/merge/drop/priority_reject_total`, `rustfs_heal_read_repair_dedup_total{reason}`, etc. All logs are structured event style (PR #5720); per-object logs are demoted to prevent storms (`demote_to_debug_when!`, #5716/#5719/#5727).
--- ---
## 3. Scanner 已实现功能全景 ## 3. Scanner implemented-feature panorama
### 3.1 循环、leader、立即触发 ### 3.1 Loop, leader, immediate triggering
- **集群单 leader**:分布式 ns 写锁 `leader.lock``scanner.rs:3156-3207`,超时默认 5s+ **持久化 leader-epoch CAS 围栏**leader 用 ETag 前置条件向 `.bloomcycle.bin``RSCYC001` 编码的 (cycle, leader_epoch)`scanner.rs:118,1850-1861,2177-2334`);usage 快照再打 epoch fence:2087-2153)。锁丢失 → 取消当前周期,30s 收敛(:108-111,2623-2642)。 - **Cluster single leader**: distributed ns write lock `leader.lock` (`scanner.rs:3156-3207`, timeout default 5s) + **persisted leader-epoch CAS fence**: the leader writes (cycle, leader_epoch) encoded as `RSCYC001` into `.bloomcycle.bin` using an ETag precondition (`scanner.rs:118,1850-1861,2177-2334`); usage snapshots additionally carry an epoch fence (:2087-2153). Lock lost → cancel the current cycle, converging within 30s (:108-111,2623-2642).
- 抢锁后立即执行一轮;周期 = `RUSTFS_SCANNER_CYCLE` > config cycle > start_delay > 部署默认 > 速度档位(±10% 抖动、下限 1s)。 - One round executes immediately after the lock is acquired; cycle = `RUSTFS_SCANNER_CYCLE` > config cycle > start_delay > deployment default > speed tier (±10% jitter, floor 1s).
- **clean-idle 指数退避**:连续完整无脏周期间隔 ×2(封顶 24hbitrot 周期压缩上限;桶有 lifecycle/replication 活动规则禁用,:383-456,1382-1512)。 - **clean-idle exponential backoff**: consecutive fully-clean idle intervals double (capped at 24h; bitrot-cycle compression cap; disabled when a bucket has active lifecycle/replication rules, :383-456,1382-1512).
- **superseded/deferred 退避**5s 起指数退避封顶 30min:105-106,3432-3438);维护探测失败独立退避(:459-505)。 - **superseded/deferred backoff**: exponential backoff from 5s capped at 30min (:105-106,3432-3438); maintenance probing failures get an independent backoff (:459-505).
- **立即唤醒**① dirty-usage 快路径——写路径 put/delete/multipart/bucket 操作调用 `record_dirty_usage_bucket``scanner_io.rs:222-235`;调用点 `rustfs/src/app/object_usecase.rs:6221` 等),自增 generation Notify 唤醒 leader,脏桶优先排队(`scanner_io.rs:462-488`);② 维护配置变更(lifecycle/replication 设置时 `record_scanner_maintenance_change`);③ 运行时配置热更 generation+Notify;④ 集群活动快照变化。 - **Immediate wakeup**: ① dirty-usage fast path — write-path put/delete/multipart/bucket operations call `record_dirty_usage_bucket` (`scanner_io.rs:222-235`; call sites include `rustfs/src/app/object_usecase.rs:6221`), bump the generation and Notify-wake the leader; dirty buckets are queued first (`scanner_io.rs:462-488`); ② maintenance-config changes (lifecycle/replication settings call `record_scanner_maintenance_change`); ③ runtime-config hot updates generation+Notify; ④ cluster activity snapshot changes.
- **集群协调**`probe_scanner_activity` 汇集本机+peer `ScannerNodeActivity`instance_id/namespace_generation/maintenance_generation/protocol_version/topology_digest/data_movement_active/dirty usage),拓扑摘要覆盖 pools/sets/drives URL,协议版本不齐拒绝共享缓存锁(`scanner.rs:970-1068`);**数据迁移(rebalance/decommission)期间推迟周期**`scanner_io.rs:2226-2374`);周期结束逐 peer RPC 确认 dirty-usage ack`scanner.rs:2925-2952`)。 - **Cluster coordination**: `probe_scanner_activity` gathers this node's and peers' `ScannerNodeActivity` (instance_id/namespace_generation/maintenance_generation/protocol_version/topology_digest/data_movement_active/dirty usage); the topology digest covers pools/sets/drives URLs; a mismatched protocol version refuses to share the cache lock (`scanner.rs:970-1068`); **cycles are deferred during data movement (rebalance/decommission)** (`scanner_io.rs:2226-2374`); at cycle end, per-peer RPC confirms the dirty-usage ack (`scanner.rs:2925-2952`).
### 3.2 遍历模型 ### 3.2 Traversal model
- 主遍历是**全量目录 walk**tokio::fs::read_dir 递归,`scanner_folder.rs:1915-2234`),不走 metacachemetacache/`list_path_raw` 仅用于 abandoned children 跨盘核查(:2528-2792)。 - The main traversal is a **full directory walk** (tokio::fs::read_dir recursion, `scanner_folder.rs:1915-2234`), not via metacache; metacache/`list_path_raw` is used only for the abandoned-children cross-drive verification (:2528-2792).
- 三级并发:leader → per-set(信号量默认 4)→ per-disk 桶扫描(默认 4)→ 单盘递归;每桶每 set 缓存锁 `.scanner-cycle.lock.pool-N.set-M`(锁丢失取消该桶扫描,锁竞争重排队);每盘单扫描准入(本地盘也走信号量,`scanner_io.rs:3246-3274`)。 - Three-level concurrency: leader → per-set (semaphore default 4) → per-disk bucket scans (default 4) → single-drive recursion; a cache lock per bucket per set `.scanner-cycle.lock.pool-N.set-M` (losing the lock cancels that bucket's scan; lock contention re-queues); single-scan admission per drive (local drives also go through the semaphore, `scanner_io.rs:3246-3274`).
- 桶顺序:shuffle 后按 dirty → 未缓存 → 已缓存重排(`scanner_io.rs:2947-2949,462-488`);目录内按名字排序 + resume 提示旋转(`scanner_folder.rs:333-359`)。 - Bucket ordering: after shuffle, re-ordered as dirty → uncached → cached (`scanner_io.rs:2947-2949,462-488`); entries within a directory sorted by name + resume-hint rotation (`scanner_folder.rs:333-359`).
- **断点续扫**`DataUsageScanCheckpoint{version,resume_after,reason}` 持久于缓存 info`data_usage_define.rs:68,293-307`);预算耗尽/取消写入,恢复有 Used/Stale/NoHint 指标;续扫单位是目录(无跨周期对象级分页)。 - **Resumable scanning**: `DataUsageScanCheckpoint{version,resume_after,reason}` persisted in the cache info (`data_usage_define.rs:68,293-307`); written on budget exhaustion/cancel; resumption has Used/Stale/NoHint metrics; the resume unit is a directory (no cross-cycle object-level pagination).
- erasure 语义:发现 `xl.meta` 即对象边界不下钻;UUID data-dir 候选最多探测 64 entry;有数据无元数据 → 记 failed + 高优 healsymlink 目录忽略/环跳过。 - Erasure semantics: finding `xl.meta` marks an object boundary with no descent; at most 64 UUID data-dir candidate entries probed; data without metadata → record failed + high-priority heal; symlink directories ignored / cycles skipped.
- 协作让出:每 N 对象(默认 128`yield_now` - Cooperative yielding: `yield_now` every N objects (default 128).
### 3.3 大桶跳过策略(对标 MinIO compaction ### 3.3 Large-bucket skip strategy (benchmarked against MinIO compaction)
1. 缓存当前性复用:桶与扫描计划未变(name/source/snapshot_complete/plan digest/next_cycle/leader_epoch/cache_key_format 全匹配)整桶跳过(`scanner_io.rs:1062-1109`)。 1. Cache-currency reuse: if the bucket and scan plan are unchanged (name/source/snapshot_complete/plan digest/next_cycle/leader_epoch/cache_key_format all match), the whole bucket is skipped (`scanner_io.rs:1062-1109`).
2. compacted 目录 16 周期轮换窗口:`hash mod (next_cycle, 16)` 命中才重扫,否则从旧缓存拷贝(`scanner_folder.rs:74,2429-2442`)。 2. compacted-directory 16-cycle rotation window: rescan only when `hash mod (next_cycle, 16)` hits, otherwise copy from the old cache (`scanner_folder.rs:74,2429-2442`).
3. compaction 阈值:子项 <500 或纯对象叶子压缩为单 entry;子文件夹 ≥2500(根 10000)预压缩;children ≥10000 归约(:75-78,2314-2340,2846-2887)。 3. compaction thresholds: children <500 or pure-object leaves compress into a single entry; subfolders ≥2500 (root 10000) pre-compressed; children ≥10000 reduced (:75-78,2314-2340,2846-2887).
4. 失败对象 TTL 跳过:86400s/最多 10000 条(:88-91,1354-1381)。 4. failed-object TTL skip: 86400s / at most 10,000 entries (:88-91,1354-1381).
MinIO master 对比:MinIO 的跳过策略同样是 hash-mod-16 周期 + compaction 阈值树(500/10000/2500),**bloom filter 已从 master 删除**。RustFS 的常量与结构与 MinIO 现状同源(MinIO 未采用跨盘 dirty-generation 优先,RustFS 额外多两层跳过——plan digest 与缓存当前性校验)。 Compared with MinIO master: MinIO's skip strategy is likewise hash-mod-16 cycles + a compaction threshold tree (500/10000/2500), and the **bloom filter has been removed from master**. RustFS's constants and structure share the same origin as MinIO's current state (MinIO does not adopt cross-drive dirty-generation prioritization; RustFS additionally has two more skip layers — plan digest and cache-currency validation).
### 3.4 data usage 统计 ### 3.4 data usage statistics
- 维度:每目录 entrysize/objects/versions/delete_markers/大小直方图/版本直方图/复制统计/failed_objects/per-tier stats/children/compacted`data-usage/src/data_usage.rs:661-679`);每对象 SizeSummary(含 per-ARN 复制目标统计、tier 统计,tier 分类:transitioned 完成记入其 tier 否则按 storage classfree version 不计);桶级 `BucketUsageInfo`;集群级 `DataUsageInfo`(含 scanner_cycle/scanner_epoch 围栏 + usage_snapshot_complete)。 - Dimensions: per-directory entry (size/objects/versions/delete_markers/size histogram/version histogram/replication stats/failed_objects/per-tier stats/children/compacted, `data-usage/src/data_usage.rs:661-679`); per-object SizeSummary (incl. per-ARN replication-target stats and tier stats; tier classification: fully transitioned counts toward its tier, otherwise by storage class; free versions not counted); bucket-level `BucketUsageInfo`; cluster-level `DataUsageInfo` (incl. scanner_cycle/scanner_epoch fence + usage_snapshot_complete).
- 存储:每桶每 set `{bucket}/.usage-cache.bin`(主 + `.bkp` 备份 + CAS 重试);权威集群快照 `buckets/data-usage/data-usage.json`(每 10 周期同步 `.bkp`,legacy 路径兼容);陈旧快照拒绝写入(epoch/cycle/last_update 三重判定);被竞争 superseded 的观测快照另存 `data-usage-observed.json` - Storage: per bucket per set `{bucket}/.usage-cache.bin` (primary + `.bkp` backup + CAS retry); the authoritative cluster snapshot `buckets/data-usage/data-usage.json` (`.bkp` synced every 10 cycles, legacy path compatible); stale snapshots rejected on write (triple epoch/cycle/last_update determination); observation snapshots superseded by a race are stored separately as `data-usage-observed.json`.
- 消费:`replace_bucket_usage_memory_from_info` 刷新桶用量内存 + 两层缓存失效(`scanner.rs:4142-4152`→ bucket stats/quota/admin account_info/system;写路径内存实时叠加 overlay;启动读快照判断冷缓存跳过启动延迟。 - Consumption: `replace_bucket_usage_memory_from_info` refreshes bucket-usage memory + two-level cache invalidation (`scanner.rs:4142-4152`) → bucket stats/quota/admin account_info/system; the write path overlays memory in real time; at startup, reading the snapshot detects a cold cache and skips startup delay.
- 未完成 multipart 不参与统计(与 MinIO 一致,MinIO 也不扫 multipart 桶)。 - Incomplete multipart uploads are not counted (consistent with MinIO, which also does not scan the multipart bucket).
### 3.5 ILM 集成 ### 3.5 ILM integration
- 每对象 `ScannerItem::apply_actions``scanner_folder.rs:747-1032`):`Evaluator::new(lifecycle).with_lock_retention(...).with_replication_config(...).eval()` 批量评估。 - Per object `ScannerItem::apply_actions` (`scanner_folder.rs:747-1032`): `Evaluator::new(lifecycle).with_lock_retention(...).with_replication_config(...).eval()` batch evaluation.
- 已实现动作(IlmAction 全集,`common/src/metrics.rs:34-45`):expiry 删除(Delete/DeleteRestored/DeleteRestoredVersion)、全版本删除(DeleteAllVersions/DelMarkerDeleteAllVersions,处理后停止后续版本)、transitionTransition/TransitionVersiontier 列表运行时读取)、noncurrent 批量(DeleteVersionAction → `enqueue_by_newer_noncurrent`)、free-version 清理(`enqueue_free_version`)、object-lock retention 约束。**与 MinIO 9 ILM 动作一一对应** - Implemented actions (the full IlmAction set, `common/src/metrics.rs:34-45`): expiry deletes (Delete/DeleteRestored/DeleteRestoredVersion), all-versions deletes (DeleteAllVersions/DelMarkerDeleteAllVersions, stop further versions after handling), transition (Transition/TransitionVersion, tier list read at runtime), noncurrent batches (DeleteVersionAction → `enqueue_by_newer_noncurrent`), free-version cleanup (`enqueue_free_version`), object-lock retention constraints. **A one-to-one mapping onto MinIO's 9 ILM actions.**
- 执行模型:scanner 是"发现与入队"角色(expiry 队列/transition 队列在 ecstore `bucket_lifecycle_ops.rs`),动作由 worker 池消费——与 MinIO globalExpiryState/globalTransitionState 同型。 - Execution model: the scanner is the "discover and enqueue" role (the expiry/transition queues live in ecstore `bucket_lifecycle_ops.rs`); actions are consumed by worker pools — the same shape as MinIO's globalExpiryState/globalTransitionState.
- AbortIncompleteMultipartUpload 不在 scanner/ILM 内执行(MinIO 同样不在:`internal/bucket/lifecycle/rule.go` FIXME,实际由 `erasureSets.cleanupStaleUploads` 全局例程承担);RustFS 由 ecstore 独立后台任务 `init_background_stale_multipart_upload_cleanup``bucket_lifecycle_ops.rs:3289-3320`+ 桶删除时 on-demand - AbortIncompleteMultipartUpload is not executed inside scanner/ILM (MinIO likewise: `internal/bucket/lifecycle/rule.go` has a FIXME, and it is actually carried by the `erasureSets.cleanupStaleUploads` global routine); in RustFS it is an independent ecstore background task `init_background_stale_multipart_upload_cleanup` (`bucket_lifecycle_ops.rs:3289-3320`) + on-demand at bucket deletion.
- 集成测试覆盖:transition+restorefree-versionnoncurrentdelete-marker0-day、后台扫描过期(`scanner/tests/lifecycle_integration_test.rs:1071-2095`)。 - Integration-test coverage: transition+restore, free-version, noncurrent, delete-marker, 0-day, background-scan expiry (`scanner/tests/lifecycle_integration_test.rs:1071-2095`).
### 3.6 heal 候选生产(scanner 侧) ### 3.6 heal candidate production (scanner side)
- 抽样:`hash mod_alt(next_cycle/prob_div, 1024/prob_div)`,进入 compacted 分支重扫时 prob_div=16 等效概率 ×16(与 MinIO 同款补偿,`scanner_folder.rs:125-127,2117-2122`)。 - Sampling: `hash mod_alt(next_cycle/prob_div, 1024/prob_div)`; when rescanning via the compacted branch, prob_div=16 gives an equivalent ×16 probability (the same compensation as MinIO, `scanner_folder.rs:125-127,2117-2122`).
- deep/normal:周期级 `get_cycle_scan_mode`bitrot_cycle 默认 30d`scanner.rs:1626-1657`)→ 对象级带 `HealScanMode::Deep`;新鲜对象(60s 内修改)降级 Normal:146-155);状态持久 `.background-heal.json``BackgroundHealInfo{bitrot_start_time,bitrot_start_cycle,current_scan_mode}`,与 MinIO 同路径同结构)。 - deep/normal: cycle-level `get_cycle_scan_mode` (bitrot_cycle default 30d, `scanner.rs:1626-1657`) → object-level with `HealScanMode::Deep`; fresh objects (modified within 60s) are demoted to Normal (:146-155); state persisted in `.background-heal.json` (`BackgroundHealInfo{bitrot_start_time,bitrot_start_cycle,current_scan_mode}`, same path and structure as MinIO).
- scanner 只入队不内联执行(内联 heal 已移除,兼容旗标仅告警,`scanner_folder.rs:411-427`);`HealScanMode::Deep` 只是标记,bitrot 校验读发生在 heal 消费端(ecstore Deep 路径)。 - The scanner only enqueues, never executes inline (inline heal was removed; the compat flag only warns, `scanner_folder.rs:411-427`); `HealScanMode::Deep` is just a marker — the bitrot-verification read happens at the heal consumer (the ecstore Deep path).
- 元数据损坏 → 高优 heal`classify_get_size_failure` → HealMetadata);abandoned children → list_path_raw quorum 核查 + 桶级/对象级高优 healhealing 盘粘性跳过(`should_heal` :1628-1648)。 - Metadata corruption → high-priority heal (`classify_get_size_failure` → HealMetadata); abandoned children → list_path_raw quorum verification + bucket-level/object-level high-priority heal; healing drives get sticky skipping (`should_heal` :1628-1648).
- pending-heal 账本:heal 通道满被拒持久化到缓存 info,下轮重试。 - pending-heal ledger: candidates rejected by a full heal channel are persisted into the cache info and retried next round.
- 复制 heal`queue_replication_heal` → replication 队列(走 replication 通道而非 heal channel);per-ARN 复制用量统计。 - Replication heal: `queue_replication_heal` the replication queue (going through the replication channel, not the heal channel); per-ARN replication usage statistics.
### 3.7 remote_scanner RPC 协议(RustFS 特有) ### 3.7 remote_scanner RPC protocol (RustFS-specific)
请求 ≤16KB msgpackversion/request_id/server_epoch/session_id/session_sequence/bucket/next_cycle/leader_epoch/scan_plan_digest/skip_healing/scan_mode/budget);帧 ≤2MBHMAC-SHA256 逐帧认证(域 `rustfs-ns-scanner-frame-v3`);进度心跳 1s(预算模式 250ms);阶段播报 Scanning→PersistingRPC 生命周期上限 24h、断连宽限 2min;防重放 session+sequence 缓存(容量 65536);服务端校验 leader fence 与持久化 cycle 一致 + 每 5s fence 复验;结果 Complete/Partial/NamespaceNotFound/CycleAhead;不支持 v4 协议的远端盘回退 leader 本地扫描(`remote_scanner.rs` 全文件;`scanner_io.rs:2750-2812`)。 Requests ≤16KB msgpack (version/request_id/server_epoch/session_id/session_sequence/bucket/next_cycle/leader_epoch/scan_plan_digest/skip_healing/scan_mode/budget); frames ≤2MB, HMAC-SHA256 per-frame authentication (domain `rustfs-ns-scanner-frame-v3`); progress heartbeats 1s (250ms in budget mode); phase announcements Scanning→Persisting; RPC lifetime cap 24h, disconnect grace 2min; anti-replay session+sequence cache (capacity 65536); the server validates leader-fence and persisted-cycle consistency + fence re-validation every 5s; results Complete/Partial/NamespaceNotFound/CycleAhead; remote drives without v4-protocol support fall back to the leader scanning locally (`remote_scanner.rs` whole file; `scanner_io.rs:2750-2812`).
### 3.8 限速/预算/热更/观测 ### 3.8 Rate limiting / budgets / hot updates / observability
- DynamicSleeper 比例退避(速度档 fastest/fast/default/slow/slowest,同 MinIO 五档参数);idle_mode 总闸;前台 S3 读流量每请求 10ms 封顶 250ms 额外退避。 - DynamicSleeper proportional backoff (speed tiers fastest/fast/default/slow/slowest, same five-tier parameters as MinIO); idle_mode master switch; an extra backoff capped at 250ms per request (10ms base) driven by foreground S3 read traffic.
- 周期预算 ScannerCycleBudgetmax_duration/max_objects/max_directories(默认 0=不限),partial 周期仍推进 cycle 计数。 - Cycle budget ScannerCycleBudget: max_duration/max_objects/max_directories (default 0 = unlimited); partial cycles still advance the cycle count.
- runtime_config 三层来源(env > config > default)逐字段来源标记(Env/Config/ScannerCompatConfig/Default),admin `PUT /v3/config` 热更 → generation+Notify 即时生效;`GET /v3/scanner/status` 返回 enabled/freshness(fresh/stale/unknown)/metrics/cycle_schedule/runtime_config`GET /v3/ilm/expiry/status` 返回 expiry 队列/worker/missed/blocked - runtime_config with three-layer sources (env > config > default) and per-field source markers (Env/Config/ScannerCompatConfig/Default); admin `PUT /v3/config` hot update → generation+Notify takes effect immediately; `GET /v3/scanner/status` returns enabled/freshness(fresh/stale/unknown)/metrics/cycle_schedule/runtime_config; `GET /v3/ilm/expiry/status` returns expiry queue/workers/missed/blocked.
- 指标:leader lock、周期 complete/partial/deferred/supersededversions scannedper-sourceUsage/Lifecycle/BucketReplication/SiteReplication/Heal/Bitrot/Alertschecked/executed/queued/missedcheckpoint set/used/stale、当前路径(per-disk+bucket 实时)、缓存 save 系列、并发系列、告警(excess versions/version size/folders)。 - Metrics: leader lock; cycle complete/partial/deferred/superseded; versions scanned; per-source (Usage/Lifecycle/BucketReplication/SiteReplication/Heal/Bitrot/Alerts) checked/executed/queued/missed; checkpoint set/used/stale; current path (per-disk+bucket in real time); cache save series; concurrency series; alerts (excess versions/version size/folders).
--- ---
## 4. MinIO 逐项对标 ## 4. Item-by-item parity versus MinIO
### 4.1 heal 触发通道对照 ### 4.1 heal trigger-channel comparison
| MinIO 通道 | RustFS 对应 | 状态 | | MinIO channel | RustFS counterpart | Status |
|---|---|---| |---|---|---|
| A. 手动 admin healhealSequenceclientToken/forceStart/forceStop | heal channel Start/Query/Cancel + 集群 coordinator + envelope 重放防护 | ✅ 等价且增强(集群路由);序列语义差异见 §6 HS-06 | | A. Manual admin heal (healSequence, clientToken/forceStart/forceStop) | heal channel Start/Query/Cancel + cluster coordinator + envelope replay protection | ✅ equivalent and enhanced (cluster routing); sequence-semantics differences in §6 HS-06 |
| B. 常驻后台 heal 队列(newBgHealSequence + healRoutine worker 池) | HealManager 常驻调度器 + 优先级队列 + bulkhead | ✅ 等价且增强 | | B. Resident background heal queue (newBgHealSequence + healRoutine worker pool) | HealManager resident scheduler + priority queue + bulkhead | ✅ equivalent and enhanced |
| C. 新盘/换盘自动 resyncmonitorLocalDisksAndHeal 10s + healFreshDisk + healingTracker + waitForFormatErasure 握手) | auto disk scanner10s+ replacement_readiness + durable intent/proof 状态机 + heal_replacement_format | ✅ 等价且增强(identity fence + completion proofMinIO tracker 面向对外可见性更强,见 §6 HS-07 | | C. Automatic new/replaced-drive resync (monitorLocalDisksAndHeal 10s + healFreshDisk + healingTracker + waitForFormatErasure handshake) | auto disk scanner (10s) + replacement_readiness + durable intent/proof state machine + heal_replacement_format | ✅ equivalent and enhanced (identity fence + completion proof; MinIO's tracker is stronger on external visibility, see §6 HS-07) |
| D. MRF(队列 100k + 持久化 list.bin + shutdown 回放 + 读路径 corrupt 投递) | read-repairLow+TTL 去重)+ 写路径 convergence heal 部分承担;`HealType::MRF` 执行体无生产入口 | ⚠️ 部分等价(§6 HS-01 | | D. MRF (100k queue + persisted list.bin + shutdown replay + read-path corrupt delivery) | read-repair (Low + TTL dedup) + write-path convergence heal carry it partially; the `HealType::MRF` executor has no production entry | ⚠️ partially equivalent (§6 HS-01) |
| E. Scanner 抽样 heal1/1024 + compacted ×16 补偿)+ abandoned children | 同款抽样 + ×16 补偿 + abandoned children + pending-heal 账本 | ✅ 等价且增强(账本) | | E. Scanner sampled heal (1/1024 + compacted ×16 compensation) + abandoned children | the same sampling + ×16 compensation + abandoned children + pending-heal ledger | ✅ equivalent and enhanced (the ledger) |
| F. 读路径内联触发 → MRFGetObject part 缺失/损坏、元数据重建 missingBlocks>0 | read repairmissing_shards/decode_error/metadata_read_error 三入口) | ✅ 等价(入 heal 队列而非 MRF 队列) | | F. Read-path inline trigger → MRF (GetObject part missing/corrupt, metadata rebuild missingBlocks>0) | read repair (three entries: missing_shards/decode_error/metadata_read_error) | ✅ equivalent (enqueued into the heal queue rather than the MRF queue) |
### 4.2 对象级 heal 语义对照 ### 4.2 Object-level heal semantics comparison
| 特性 | MinIO | RustFS | 状态 | | Feature | MinIO | RustFS | Status |
|---|---|---|---| |---|---|---|---|
| mod-time quorum 仲裁 | listOnlineDisks | | ✅ | | mod-time quorum arbitration | listOnlineDisks | same | ✅ |
| ETag 多数派兜底(时钟漂移) | filterDisksByETag | `filter_by_etag`/`quorum_etag`heal.rs:525-567 | ✅ 已亲验 | | ETag majority fallback (clock drift) | filterDisksByETag | `filter_by_etag`/`quorum_etag` (heal.rs:525-567) | ✅ verified first-hand |
| cannotHeal ETag 豁免 | ETag 全一致豁免重试 | heal.rs:679 | ✅ | | cannotHeal ETag waiver | waived on all-consistent ETag retry | heal.rs:679 | ✅ |
| Normal=CheckPartsstat/ Deep=VerifyFilebitrot | | `disks_with_all_parts` scan_modeops/heal.rs:562-572,978-1024 | ✅ | | Normal=CheckParts (stat) / Deep=VerifyFile (bitrot) | yes | `disks_with_all_parts` by scan_mode (ops/heal.rs:562-572,978-1024) | ✅ |
| Normal 检出 corrupt 自动升 Deep 重试一次 | erasure-healing.go:1101-1106 | ops/heal.rs:2022-2031 | ✅ | | Normal detecting corrupt auto-escalates to one Deep retry | erasure-healing.go:1101-1106 | ops/heal.rs:2022-2031 | ✅ |
| dangling 判定(not-found > parity+ 删除审计 | isObjectDangling/deleteIfDangling | `dangling_delete_safety`:1488+ scanner HEAL_DELETE_DANGLING | ✅(审计 tags 细节有差异) | | dangling determination (not-found > parity) + deletion auditing | isObjectDangling/deleteIfDangling | `dangling_delete_safety` (:1488) + scanner HEAL_DELETE_DANGLING | ✅ (audit-tags details differ) |
| 孤儿 data-dir/inline 清理(CleanAbandonedData | CheckAbandonedPartsscanner 抽中 + admin Remove 时显式调用) | heal 路径内 `reclaim_orphan_data_dirs_best_effort`:1428);独立 API 三层 NotImplemented | ⚠️ 部分等价(§6 HS-02 | | Orphan data-dir/inline cleanup (CleanAbandonedData) | CheckAbandonedParts (invoked explicitly on scanner sampling + admin Remove) | in-heal-path `reclaim_orphan_data_dirs_best_effort` (:1428); standalone API NotImplemented at all three layers | ⚠️ partially equivalent (§6 HS-02) |
| 版本化/delete-marker heal | HealObject versionIDnullVersionID 特判 | 逐版本枚举 + delete-marker latest healB5 回归) | ✅ | | Versioned/delete-marker heal | HealObject versionID; nullVersionID special case | per-version enumeration + delete-marker latest heal (B5 regression) | ✅ |
| 对象级 healing 元数据标记(x-minio-healingRenameData 跳过版本清理) | 有 | 无对象级标记;依赖盘级 healing.bin + NSLock + rename 语义 | ⚠️ 评估项(§6 HS-12 | | Object-level healing metadata marker (x-minio-healing, RenameData skips version cleanup) | yes | no object-level marker; relies on drive-level healing.bin + NSLock + rename semantics | ⚠️ evaluation item (§6 HS-12) |
| Distribution/Index 一致性三处防线 | 有(manual modification 拒绝) | 目标盘格式结果全 ok 校验 + 身份围栏 | ✅(粒度不同) | | Distribution/Index consistency, three lines of defense | yes (manual modification rejected) | target-drive format results all-ok check + identity fence | ✅ (different granularity) |
| parityEC:0)对象 | bitrot 不可恢复处理 | 判不可恢复(:700-726)+ 写入自校验 | ✅ 增强(写路径自校验) | | no-parity (EC:0) objects | bitrot treated as unrecoverable | judged unrecoverable (:700-726) + write self-verification | ✅ enhanced (write-path self-verification) |
| 三层分布不一致拒绝 heal | | heal_walk 归一化 + 页界防御 | ✅(实现方式不同) | | three-layer distribution inconsistency refuses heal | yes | heal_walk normalization + page-bound defense | ✅ (different implementation approach) |
| multipart 孤儿对账 | CheckAbandonedParts 承担 | 显式 NotImplemented(由 lifecycle 清理承担) | ⚠️ §6 HS-02 | | multipart orphan reconciliation | carried by CheckAbandonedParts | explicitly NotImplemented (carried by lifecycle cleanup) | ⚠️ §6 HS-02 |
| suspended/decommissioned pool 处理 | IsSuspended 跳过 | deferral 语义(store/heal.rs:192-207PR #5876 | ✅ | | suspended/decommissioned pool handling | skipped via IsSuspended | deferral semantics (store/heal.rs:192-207, PR #5876) | ✅ |
| heal 与并发删除互斥 | NSLock + healing 标记 | NSLock + 写锁 | ✅ | | heal mutually exclusive with concurrent deletes | NSLock + healing marker | NSLock + write lock | ✅ |
### 4.3 新盘 resync 对照 ### 4.3 new-drive resync comparison
| MinIO | RustFS | 状态 | | MinIO | RustFS | Status |
|---|---|---| |---|---|---|
| waitForFormatErasure 四类可恢复错误无限等待握手 | startup 盘解析 + renew_disk 重连路径 | ✅(模型不同:RustFS 不在启动时阻塞等待 format | | waitForFormatErasure handshake waiting indefinitely on four classes of recoverable errors | startup drive resolution + renew_disk reconnect path | ✅ (different model: RustFS does not block at startup waiting for format) |
| HealFormat NSLock + errNoHealRequired + refFormat 不一致拒绝 | `heal_format`/`heal_replacement_format` fail-closed + 目标槽位限定(PR #1787 语义) | ✅ 增强 | | HealFormat NSLock + errNoHealRequired + refFormat-mismatch rejection | `heal_format`/`heal_replacement_format` fail-closed + target-slot restriction (PR #1787 semantics) | ✅ enhanced |
| per (pool,set) 分布式锁防并发 resync | set 级队列去重 + bulkheadmanager.rs:2854-2889 | ✅ | | per (pool,set) distributed lock preventing concurrent resync | set-level queue dedup + bulkhead (manager.rs:2854-2889) | ✅ |
| 全新集群检测(待 heal 盘数==总盘数不触发) | replacement_readiness(独立挂载点/物理设备校验,非 root | ✅ 增强 | | brand-new-cluster detection (drives-to-heal == total drives does not trigger) | replacement_readiness (independent mount point / physical-device validation, non-root) | ✅ enhanced |
| healingTracker.healing.binBytes/Items 计数、QueuedBuckets/HealedBucketsResume 快照、RetryAttempts ≤4HealID 联动、diskID 变更重置) | resume/checkpoint schema 化持久层 + durable intent/proofper-task 文件,CAS | ✅ 等价且增强(崩溃窗口补齐);但**对外快照可见性**弱于 MinIO§6 HS-07 | | healingTracker (.healing.bin: Bytes/Items counters, QueuedBuckets/HealedBuckets, Resume snapshot, RetryAttempts ≤4, HealID linkage, diskID-change reset) | resume/checkpoint schema'd persistence + durable intent/proof (per-task files, CAS) | ✅ equivalent and enhanced (crash-window backfill); but **external snapshot visibility** is weaker than MinIO's (§6 HS-07) |
| 跳过 heal 开始后新写入版本(ModTime > Started | 无同款过滤 | ⚠️ §6 HS-13 | | skip versions written after heal start (ModTime > Started) | no such filter | ⚠️ §6 HS-13 |
| 跳过 ILM 已过期版本(filterLifecycle | 无同款过滤 | ⚠️ §6 HS-13 | | skip ILM-expired versions (filterLifecycle) | no such filter | ⚠️ §6 HS-13 |
| worker max(GOMAXPROCS,NR)/4 下限 4heal:drive_workers 覆盖 | 页内并发 8Deep/AutoHeal 强制 1+ per-set bulkhead | ✅(参数模型不同) | | worker count max(GOMAXPROCS,NR)/4 floor 4, heal:drive_workers override | in-page concurrency 8 (Deep/AutoHeal forced to 1) + per-set bulkhead | ✅ (different parameter model) |
| 每 entry waitForLowHTTPReq 让路 | mainline throttle(前台利用率门控) | ✅ 增强 | | waitForLowHTTPReq yield per entry | mainline throttle (foreground-utilization gating) | ✅ enhanced |
| heal 范围含 `.minio.sys/config``.minio.sys/buckets` 两个伪桶;最新桶优先 | ErasureSet 任务逐 bucket 预处理(含 meta bucket 语义由 heal_bucket 承担) | ✅(顺序无"最新优先") | | heal scope includes the two pseudo-buckets `.minio.sys/config` and `.minio.sys/buckets`; newest bucket first | ErasureSet task pre-processes per bucket (meta-bucket semantics carried by heal_bucket) | ✅ (no "newest first" ordering) |
| 失败整体重试 ≤4 次(resetHealing + errRetryHealing | schedule_retry 复位双层 + 可恢复重试 ≤3 | ✅ | | whole-failure retry ≤4 (resetHealing + errRetryHealing) | schedule_retry resets both layers + recoverable retry ≤3 | ✅ |
### 4.4 scanner 对照 ### 4.4 scanner comparison
| MinIO | RustFS | 状态 | | MinIO | RustFS | Status |
|---|---|---| |---|---|---|
| 集群单 leaderglobalLeaderLock | leader.lock + 持久化 leader-epoch CAS 围栏 | ✅ 增强(epoch 围栏防脑裂,MinIO 无持久化 epoch | | cluster single leader (globalLeaderLock) | leader.lock + persisted leader-epoch CAS fence | ✅ enhanced (epoch fence against split-brain; MinIO has no persisted epoch) |
| `.bloomcycle.bin` 只存 cyclebloom 已删除) | 同路径存 cycle+leader_epochRSCYC001 | ✅ 对齐(v1 误判已修正) | | `.bloomcycle.bin` stores only the cycle (bloom removed) | same path stores cycle+leader_epoch (RSCYC001) | ✅ aligned (v1 misjudgment corrected) |
| folderScanner hash-mod-16 + compaction500/10000/2500 | 同款常量 + plan digest + 缓存当前性校验 + dirty 优先 | ✅ 增强 | | folderScanner hash-mod-16 + compaction (500/10000/2500) | same constants + plan digest + cache-currency validation + dirty-first | ✅ enhanced |
| 每盘扫描并行 ≤GOMAXPROCShealing 盘排除 | per-set/per-disk 信号量 + healing 盘粘性跳过 | ✅ | | ≤GOMAXPROCS parallel scans per drive; healing drives excluded | per-set/per-disk semaphores + sticky skip of healing drives | ✅ |
| scannerSleeperfactor 2/max 1sspeed 档热更) | DynamicSleeper 同款 + idle_mode + 前台读退避 | ✅ 增强 | | scannerSleeper (factor 2/max 1s, speed tiers hot-swapped) | DynamicSleeper same + idle_mode + foreground-read backoff | ✅ enhanced |
| idle 语义:`scanner:idle_speed=on`(空闲时段才节流,忙时全速) | `RUSTFS_SCANNER_IDLE_MODE=true`(启用限速总闸) | ⚠️ 语义方向相反,§6 HS-14 | | idle semantics: `scanner:idle_speed=on` (throttle only in idle windows, full speed when busy) | `RUSTFS_SCANNER_IDLE_MODE=true` (master switch for rate limiting) | ⚠️ opposite semantic direction, §6 HS-14 |
| applyActions 顺序(heal→ILM→复制→告警) | apply_actions 同序(heal 候选→ILM→复制 heal→告警) | ✅ | | applyActions order (heal→ILM→replication→alerts) | apply_actions same order (heal candidates→ILM→replication heal→alerts) | ✅ |
| ILM 9 动作 + 批量评估 + DeletePrefixObject 优化 | 同 9 动作 + 批量评估 + expiry 队列 | ✅(DeleteAllVersions 是否单调用优化未逐行核) | | ILM 9 actions + batch evaluation + DeletePrefixObject optimization | same 9 actions + batch evaluation + expiry queue | ✅ (whether DeleteAllVersions has the single-call optimization was not checked line by line) |
| abandoned childrenlistPathRaw minDisks=N/2 发现漏写盘) | list_path_raw + quorum 核查 + 高优 heal | ✅ | | abandoned children (listPathRaw minDisks=N/2 detects under-written drives) | list_path_raw + quorum verification + high-priority heal | ✅ |
| incomplete multipart 独立例程(6h 间隔/24h 过期,rename .trash | ecstore 独立后台任务(可配间隔/过期) | ✅trash 二段清理细节差异,§6 HS-18 | | incomplete multipart independent routine (6h interval/24h expiry, rename into .trash) | ecstore independent background task (configurable interval/expiry) | ✅ (trash two-stage cleanup detail differences, §6 HS-18) |
| usage 维度(size/objects/versions/DM/直方图/复制/tier/bucket 级) | 全覆盖 + 集群快照三重防回退 | ✅ 增强 | | usage dimensions (size/objects/versions/DM/histograms/replication/tier/bucket level) | full coverage + cluster snapshot with triple anti-rollback | ✅ enhanced |
| prefix usageloadPrefixUsageFromBackend,console 消费) | 缓存内有目录树但仅 flatten 桶级 | ❌ §6 HS-08 | | prefix-level usage (loadPrefixUsageFromBackend, consumed by console) | the cache holds the directory tree but flattens only to bucket level | ❌ §6 HS-08 |
| 超限事件 s3:ObjectManyVersions/LargeVersions/PrefixManyFolders + 审计 | 仅指标 alert_excess_*(默认 100/1TiB/65538 vs MinIO 100/1TB/50000 | ⚠️ §6 HS-04/HS-17 | | excess events s3:ObjectManyVersions/LargeVersions/PrefixManyFolders + auditing | metrics alert_excess_* only (defaults 100/1TiB/65538 vs MinIO 100/1TB/50000) | ⚠️ §6 HS-04/HS-17 |
| scanner 指标 v3bucket_scans/directories/objects/versions/last_activity | rustfs_scanner_* 全套 + freshness | ✅(命名体系不同) | | scanner metrics v3 (bucket_scans/directories/objects/versions/last_activity) | full rustfs_scanner_* suite + freshness | ✅ (different naming scheme) |
| TraceScanner / realtime metricsmc admin scanner status/trace | trace 通道;/v3/scanner/status 自有结构 | ⚠️ §6 HS-03 | | TraceScanner / realtime metrics (mc admin scanner status/trace) | no trace channel; /v3/scanner/status has its own structure | ⚠️ §6 HS-03 |
### 4.5 admin/CLI/API 面对照 ### 4.5 admin/CLI/API surface comparison
| MinIO | RustFS | 状态 | | MinIO | RustFS | Status |
|---|---|---| |---|---|---|
| `POST /minio/admin/v3/heal/...` start/status/cancel | `POST /rustfs/admin/v3/heal/...` 同三态 | ✅(路径前缀不同属预期) | | `POST /minio/admin/v3/heal/...` start/status/cancel | `POST /rustfs/admin/v3/heal/...` same three states | ✅ (different path prefix is expected) |
| `HealStartSuccess`/`HealTaskStatus`/`HealResultItem`/DriveState | 同名字段 JSON 兼容 | ✅ | | `HealStartSuccess`/`HealTaskStatus`/`HealResultItem`/DriveState | same-named fields JSON-compatible | ✅ |
| `POST /v3/background-heal/status`BgHealState 聚合) | 同路径 + degraded 语义 + operations 矩阵 | ✅ 增强(MRF per-endpoint 子状态无,因无 MRF | | `POST /v3/background-heal/status` (BgHealState aggregate) | same path + degraded semantics + operations matrix | ✅ enhanced (no MRF per-endpoint sub-state, because there is no MRF) |
| `GET /v3/healthinfo` drive `HealInfo *HealingDisk` | 无同款 healthinfo heal 字段(replacement-recovery v4 承担部分) | ⚠️ §6 HS-07 | | `GET /v3/healthinfo` per-drive `HealInfo *HealingDisk` | no equivalent healthinfo heal field (replacement-recovery v4 covers part of it) | ⚠️ §6 HS-07 |
| madmin 客户端 HealStart/HealStatus/BackgroundHealStatus/ScannerStatus 方法 | 仅 wire 类型,无客户端方法 | ❌ §6 HS-05 | | madmin client HealStart/HealStatus/BackgroundHealStatus/ScannerStatus methods | wire types only, no client methods | ❌ §6 HS-05 |
| mc admin heal --pool/--set--scan-mode--force-start/stop | HealOpts 全字段支持(pool/set/scanMode/forceStart/forceStop | ✅(服务端就绪;缺 mc 侧入口,HS-05 | | mc admin heal --pool/--set, --scan-mode, --force-start/stop | HealOpts full field support (pool/set/scanMode/forceStart/forceStop) | ✅ (server-side ready; missing the mc-side entry, HS-05) |
| ErrHealAlreadyRunning / ErrHealOverlappingPaths 类型化错误 | 去重合并 + 驱逐语义;无类型化重叠拒绝 | ⚠️ §6 HS-06 | | ErrHealAlreadyRunning / ErrHealOverlappingPaths typed errors | dedup-merge + eviction semantics; no typed overlap rejection | ⚠️ §6 HS-06 |
| 结果 backpressuremaxUnconsumedItems=100010s 保活流式、24h 未消费 abort | 快照式查询(1024 条 + 8MiB 截断 + 10min 保留) | ⚠️ §6 HS-06 | | result backpressure (maxUnconsumedItems=1000, 10s keep-alive streaming, 24h unconsumed abort) | snapshot-style query (1024 entries + 8MiB truncation + 10min retention) | ⚠️ §6 HS-06 |
| `mc support inspect`/healing-bin 离线 dump | 无(inspect.rs 存在但 healing dump 未确认) | ⚠️ P3 | | `mc support inspect`/healing-bin offline dump | none (inspect.rs exists but the healing dump is unconfirmed) | ⚠️ P3 |
### 4.6 观测面对照 ### 4.6 observability surface comparison
| 维度 | MinIO | RustFS | 状态 | | Dimension | MinIO | RustFS | Status |
|---|---|---|---| |---|---|---|---|
| heal 指标 | minio_heal_objects_total/heal_total/errors_total/time_last_activity + v3 drive_health 2=healing | rustfs_heal_* 全套(admission/queue delay/running/throttle/page concurrency | ✅RustFS drive_health=healing 单一 gauge 等价物;DiskInfo.healing 已赋值) | | heal metrics | minio_heal_objects_total/heal_total/errors_total/time_last_activity + v3 drive_health 2=healing | full rustfs_heal_* suite (admission/queue delay/running/throttle/page concurrency) | ✅ (RustFS lacks an equivalent of the single drive_health=healing gauge; DiskInfo.healing is already assigned) |
| scanner 指标 | v3 6 + realtime 18 项 | rustfs_scanner_* 全套 + per-source 维度 | ✅ | | scanner metrics | v3 6 + realtime 18 items | full rustfs_scanner_* suite + per-source dimensions | ✅ |
| ILM 指标 | v3 5 个(expiry/transition pending/active/missed + versions_scanned | ilm expiry status API + scanner per-source | ✅(指标与 API 形态不同) | | ILM metrics | v3 5 (expiry/transition pending/active/missed + versions_scanned) | ilm expiry status API + scanner per-source | ✅ (different metrics and API shape) |
| trace | TraceHealing/TraceScanner 两通道 | 无 | ❌ §6 HS-03 | | trace | TraceHealing/TraceScanner channels | none | ❌ §6 HS-03 |
| 审计 | HealObject 事件、dangling 删除审计、scanner:manyversions 等 | 结构化日志(event style+ 指标;无 audit log 事件 | ⚠️ §6 HS-04 | | auditing | HealObject events, dangling-deletion audit, scanner:manyversions etc. | structured logs (event style) + metrics; no audit-log events | ⚠️ §6 HS-04 |
| 进度 | healingTracker Bytes/Items/QueuedBuckets/当前对象 + usage-cache 总量基线 | HealProgress{scanned/healed/failed/bytes/current_object/percentage}bytes_processed 注释为 0、estimated_completion_time None | ⚠️ §6 HS-07 | | progress | healingTracker Bytes/Items/QueuedBuckets/current object + usage-cache total baseline | HealProgress{scanned/healed/failed/bytes/current_object/percentage}; bytes_processed annotated as 0, estimated_completion_time always None | ⚠️ §6 HS-07 |
### 4.7 配置面对照(默认值) ### 4.7 configuration surface comparison (defaults)
| MinIO | RustFS | 备注 | | MinIO | RustFS | Notes |
|---|---|---| |---|---|---|
| `heal:bitrotscan`(默认 offon=每轮;Nm=N×30×24h | `heal.bitrot_cycle` / `RUSTFS_SCANNER_BITROT_CYCLE_SECS`(默认 30d=2592000s0/on=每轮 Deep,off=禁用) | ✅ 同语义(RustFS 默认 30dMinIO 默认 off——**默认值不同**,RustFS 更激进) | | `heal:bitrotscan` (default off; on=every cycle; Nm=N×30×24h) | `heal.bitrot_cycle` / `RUSTFS_SCANNER_BITROT_CYCLE_SECS` (default 30d=2592000s; 0/on=Deep every cycle, off=disabled) | ✅ same semantics (RustFS default 30d, MinIO default off — **different defaults**, RustFS more aggressive) |
| `heal:max_io=100`/`max_sleep=250ms`waitForLowIO | mainline throttle 阈值 80%/80%max_sleep 250ms | ✅ 同型(阈值模型不同) | | `heal:max_io=100`/`max_sleep=250ms` (waitForLowIO) | mainline throttle thresholds 80%/80%, max_sleep 250ms | ✅ same shape (different threshold model) |
| `heal:drive_workers`(默认 -1 自动) | 页内并发 8 + per-set 1 | ✅ 同型 | | `heal:drive_workers` (default -1 auto) | in-page concurrency 8 + per-set 1 | ✅ same shape |
| `_MINIO_HEAL_WORKERS`GOMAXPROCS/2 | `RUSTFS_HEAL_MAX_CONCURRENT_HEALS=4` + `_MAX_CONCURRENT_PER_SET=1` | ✅ | | `_MINIO_HEAL_WORKERS` (GOMAXPROCS/2) | `RUSTFS_HEAL_MAX_CONCURRENT_HEALS=4` + `_MAX_CONCURRENT_PER_SET=1` | ✅ |
| `_MINIO_AUTO_DRIVE_HEALING`on | `RUSTFS_HEAL_AUTO_HEAL_ENABLE=true` | ✅ | | `_MINIO_AUTO_DRIVE_HEALING` (on) | `RUSTFS_HEAL_AUTO_HEAL_ENABLE=true` | ✅ |
| `_MINIO_SCANNER`on | `RUSTFS_SCANNER_ENABLED=true` | ✅ | | `_MINIO_SCANNER` (on) | `RUSTFS_SCANNER_ENABLED=true` | ✅ |
| `scanner:speed` 五档(default=2x/1s/1m | 同五档同名同参数 | ✅ | | `scanner:speed` five tiers (default=2x/1s/1m) | same five tiers, same names, same parameters | ✅ |
| `scanner:idle_speed`on | `RUSTFS_SCANNER_IDLE_MODE`true | ⚠️ 语义方向(HS-14 | | `scanner:idle_speed` (on) | `RUSTFS_SCANNER_IDLE_MODE` (true) | ⚠️ semantic direction (HS-14) |
| `scanner:alert_excess_versions=100` | 100 | ✅ | | `scanner:alert_excess_versions=100` | 100 | ✅ |
| `scanner:alert_excess_folders=50000` | 65538(兼容 PBS 布局) | ⚠️ HS-17 | | `scanner:alert_excess_folders=50000` | 65538 (compatible with the PBS layout) | ⚠️ HS-17 |
| `ilm:expiration_workers=100`/`transition_workers=100` | ecstore expiry/transition worker 池(键见 ilm 子系统) | ✅(默认值未逐项核对) | | `ilm:expiration_workers=100`/`transition_workers=100` | ecstore expiry/transition worker pools (keys under the ilm subsystem) | ✅ (defaults not checked item by item) |
| `api:stale_upload_cleanup_interval=6h`/`expiry=24h` | ecstore 后台任务 env 可配 | ✅(默认值未逐项核对) | | `api:stale_upload_cleanup_interval=6h`/`expiry=24h` | ecstore background task, configurable via env | ✅ (defaults not checked item by item) |
| —(无) | `RUSTFS_HEAL_QUEUE_SIZE=10000``_TASK_TIMEOUT_SECS=300``_INTERVAL_SECS=10``_LOW_PRIORITY_MERGE/DROP``_PAGE_*``_SET_BULKHEAD``_MAINLINE_*``RUSTFS_SCANNER_CYCLE_MAX_*` 预算、`_MAX_CONCURRENT_SET/DISK_SCANS=4``_YIELD_EVERY_N_OBJECTS=128` | RustFS 特有(更细粒度) | | — (none) | `RUSTFS_HEAL_QUEUE_SIZE=10000`, `_TASK_TIMEOUT_SECS=300`, `_INTERVAL_SECS=10`, `_LOW_PRIORITY_MERGE/DROP`, `_PAGE_*`, `_SET_BULKHEAD`, `_MAINLINE_*`, `RUSTFS_SCANNER_CYCLE_MAX_*` budgets, `_MAX_CONCURRENT_SET/DISK_SCANS=4`, `_YIELD_EVERY_N_OBJECTS=128`, etc. | RustFS-specific (finer-grained) |
### 4.8 RustFS 超出 MinIO 的部分 ### 4.8 Where RustFS exceeds MinIO
1. remote_scanner RPC(扫描执行下放远端 peer 本地,含 HMAC 认证/重放缓存/fence 复验/断连宽限)。 1. remote_scanner RPC (scan execution pushed down to the remote peer locally, with HMAC authentication/replay cache/fence re-validation/disconnect grace).
2. 持久化 leader-epoch CAS 围栏 + usage 快照 epoch/cycle 防回退(MinIO 仅锁,无持久 epoch)。 2. Persisted leader-epoch CAS fence + usage-snapshot epoch/cycle anti-rollback (MinIO has only the lock, no persisted epoch).
3. 周期预算(max_duration/objects/directories+ partial 周期推进语义。 3. Cycle budgets (max_duration/objects/directories) + partial-cycle advancement semantics.
4. per-set/per-disk 扫描并发闸 + 每桶每 set 缓存锁。 4. per-set/per-disk scan concurrency gates + a cache lock per bucket per set.
5. pending-heal 账本(heal 通道满不丢候选)。 5. pending-heal ledger (heal candidates are not lost when the heal channel is full).
6. 换盘 durable intent + completion proof 状态机 + 身份围栏(MinIO healingTracker proof)。 6. Drive-replacement durable intent + completion proof state machine + identity fence (MinIO's healingTracker has no proof).
7. mainline throttle 前台压力门控(permit 利用率驱动)。 7. mainline throttle foreground pressure gating (driven by permit utilization).
8. 集群 heal control coordinator + envelope 重放防护 + degraded 显式降级。 8. Cluster heal control coordinator + envelope replay protection + explicit degraded fallback.
9. 写路径 shard bitrot 自校验(EC:0 场景)。 9. Write-path shard bitrot self-verification (the EC:0 case).
10. dirty-usage 快路径唤醒(写路径即时通知 + 脏桶优先)。 10. dirty-usage fast-path wakeup (immediate write-path notification + dirty buckets first).
11. heal 运行时可观测矩阵(优先级×来源 operations snapshot)。 11. heal runtime observability matrix (priority×source operations snapshot).
12. workload admission 联动(heal 调度器读前台压力快照)。 12. workload admission integration (the heal scheduler reads the foreground pressure snapshot).
--- ---
## 5. 差距与改进清单 ## 5. Gap and improvement list
分级定义:P1=行为/运维对齐缺口(影响生产运维或工具链兼容);P2=完善性(功能在但缺一角);P3=清理/低风险。每项含现状证据、MinIO 行为、影响、建议、验收方式。 Severity definitions: P1 = behavioral/operational alignment gap (affects production operations or toolchain compatibility); P2 = completeness (the feature exists but is missing a corner); P3 = cleanup/low risk. Each item includes current-state evidence, MinIO behavior, impact, recommendation, and acceptance.
### P18 项) ### P1 (8 items)
**HS-01 MRF/ECDecode/Metadata 三类 heal 任务无生产触发入口,HealEvent 未接线** **HS-01 The MRF/ECDecode/Metadata heal task types have no production trigger; HealEvent unwired**
- 现状:`HealType::MRF/ECDecode/Metadata` 执行体完整(task.rs:1700-2156)但全仓库无生产触发方;`HealEvent`/`HealEventHandler`event.rs:50-367crate 外零引用(已亲验 grep);channel 转换只产生 Cluster/Object/Bucket/Prefix/ErasureSetchannel.rs:566-601)。 - Current state: the `HealType::MRF/ECDecode/Metadata` executors are complete (task.rs:1700-2156) but have no production trigger anywhere in the repo; `HealEvent`/`HealEventHandler` (event.rs:50-367) has zero references outside the crate (verified first-hand by grep); channel conversion produces only Cluster/Object/Bucket/Prefix/ErasureSet (channel.rs:566-601).
- MinIOmrf.go 独立 MRF 队列(容量 100k,满丢弃计数)、进程退出 msgp 持久化 `.heal/mrf/list.bin` + 启动回放、入队 <1s 延迟 1s(等网络恢复)、healSleeper 限速;读路径 GetObject part 缺失/损坏、元数据重建 missingBlocks>0、Put 部分成功、DeleteObjectmultipartpeer client 共 7+ 投递点。 - MinIO: mrf.go has a standalone MRF queue (capacity 100k, drop-and-count when full), msgp persistence to `.heal/mrf/list.bin` at process exit + startup replay, 1s delay for enqueues <1s (waiting for network recovery), healSleeper rate limiting; on the read path, GetObject part missing/corrupt, metadata rebuild missingBlocks>0, partial Put success, DeleteObject, multipart, and the peer client add up to 7+ delivery points.
- 影响:RustFS read-repair + 写路径收敛覆盖了主场景,但缺少:① 事件驱动的 Urgent ECDecode 重建入口(ecstore 解码失败时目前仅 Low read-repair);② Metadata-only heal 入口(scanner HealMetadata 分类存在但走普通对象 heal);③ MRF 队列持久化(重启丢未消费修复意图——scanner pending-heal 账本部分缓解)。 - Impact: RustFS's read-repair + write-path convergence covers the main scenarios, but lacks: ① an event-driven Urgent ECDecode rebuild entry (on ecstore decode failure there is currently only Low read-repair); ② a metadata-only heal entry (the scanner's HealMetadata classification exists but goes through ordinary object heal); ③ MRF queue persistence (unconsumed repair intents are lost on restart — partially mitigated by the scanner's pending-heal ledger).
- 建议:三选一决策——(a) 接线 HealEvent(在 ecstore 解码失败/metadata 损坏点发事件)+ 实现持久化重试账本;(b) 删除 MRF/ECDecode/Metadata 死代码只保留文档说明;(c) 保留执行体、把 HealEvent 降级为内部 API。推荐 (a) 但需先量化 read-repair 是否已覆盖解码失败场景的响应时间要求。 - Recommendation: a pick-one-of-three decision — (a) wire HealEvent (emit events at ecstore decode-failure/metadata-corruption points) + implement a persistent retry ledger; (b) delete the MRF/ECDecode/Metadata dead code and keep only a documentation note; (c) keep the executors and demote HealEvent to an internal API. (a) is recommended, but first quantify whether read-repair already meets the response-time requirements for decode-failure scenarios.
- 验收:解码失败 → Urgent heal 请求链路 e2e;重启后 pending 修复意图回放;HealEvent 环形缓冲指标。 - Acceptance: an e2e decode-failure → Urgent heal-request chain; replay of pending repair intents after restart; HealEvent ring-buffer metrics.
**HS-02 CheckAbandonedParts 三层 NotImplementedabandoned data 独立对账入口缺失)** **HS-02 CheckAbandonedParts NotImplemented at all three layers (missing standalone abandoned-data reconciliation entry)**
- 现状:`set_disk/ops/heal.rs:2052-2056``core/sets.rs:1144-1148``store/heal.rs:258-266` 三层显式 `Err(NotImplemented)`(已亲验),注释"intentionally retained above the set layer until there is a concrete caller" - Current state: `set_disk/ops/heal.rs:2052-2056`, `core/sets.rs:1144-1148`, `store/heal.rs:258-266` explicitly return `Err(NotImplemented)` at all three layers (verified first-hand); the comment reads "intentionally retained above the set layer until there is a concrete caller".
- MinIO`CheckAbandonedParts`每盘 `CleanAbandonedData`:读 xl.meta → UUID data-dir + inline entries → getDataDirs 差集 → 删多余 data-dir/inline 并重写 xl.meta;由 scanner 抽中 heal 与 admin heal Remove 时显式调用。 - MinIO: `CheckAbandonedParts`per-drive `CleanAbandonedData`: read xl.meta → list UUID data-dirs + inline entries → diff against getDataDirs → delete surplus data-dirs/inline entries and rewrite xl.meta; invoked explicitly on scanner-sampled heals and admin heal Remove.
- 影响:RustFS heal 路径内 `reclaim_orphan_data_dirs_best_effort`:1428)覆盖"heal 时回收孤儿目录",但 ① 无独立触发点(MinIO 在对象未到 heal 阈值时也能清 abandoned data);② inline data 孤儿条目清理未确认;③ multipart 孤儿对账明确不做(设计决定,由 lifecycle 承担)。 - Impact: RustFS's in-heal-path `reclaim_orphan_data_dirs_best_effort` (:1428) covers "reclaim orphan directories while healing", but ① there is no standalone trigger point (MinIO can also clean abandoned data before an object reaches the heal threshold); ② orphan inline-data entry cleanup is unconfirmed; ③ multipart orphan reconciliation is explicitly out of scope (a design decision, carried by lifecycle).
- 建议:评估把 `reclaim_orphan_data_dirs_best_effort` 提升为 heal_object 固定步骤(若尚非)+ 实现 HealOperations::check_abandoned_parts 真实现(调用同一回收逻辑),或明确文档化"由 lifecycle 承担"并关闭 API 面。 - Recommendation: evaluate promoting `reclaim_orphan_data_dirs_best_effort` to a fixed step of heal_object (if it is not already) + implement a real HealOperations::check_abandoned_parts (calling the same reclamation logic), or explicitly document "carried by lifecycle" and close the API surface.
- 验收:构造 data-dir/inline 孤儿 → scanner 抽样/admin heal 后被清理;三层 API 返回成功或显式 NotSupported 文档化。 - Acceptance: construct data-dir/inline orphans → cleaned after scanner sampling/admin heal; the three-layer API returns success or an explicitly documented NotSupported.
**HS-03 heal/scanner trace 通道缺失** **HS-03 heal/scanner trace channels missing**
- 现状:TraceHealing/TraceScanner 零命中(已亲验 grep 全仓库)。 - Current state: zero hits for TraceHealing/TraceScanner (verified first-hand by grepping the whole repo).
- MinIO`madmin.TraceHealing`mc admin trace --healingFuncName=heal.Bucket/heal.Object/heal.CheckAbandonedParts,带 dry/remove/mode/version-id/disks/bytes)、`TraceScanner`mc admin scanner trace,支持 --filter-size/--response-duration)。 - MinIO: `madmin.TraceHealing` (mc admin trace --healing, FuncName=heal.Bucket/heal.Object/heal.CheckAbandonedParts, with dry/remove/mode/version-id/disks/bytes), `TraceScanner` (mc admin scanner trace, supports --filter-size/--response-duration).
- 影响:无法实时观测单个 heal/scanner 动作的耗时与参数;排障只能靠指标聚合与日志。 - Impact: no way to observe in real time the latency and parameters of individual heal/scanner actions; troubleshooting can rely only on aggregated metrics and logs.
- 建议:在 heal channel 执行与 scanner folder/item 处理埋点,接入现有 admin trace 订阅面(若 rustfs 已有 trace 基建则复用,无则按 madmin TraceType 扩展)。 - Recommendation: instrument heal-channel execution and scanner folder/item handling, and hook them into the existing admin trace subscription surface (reuse the rustfs trace infrastructure if it exists; otherwise extend it per madmin TraceType).
- 验收:mc 等价工具能订阅 heal/scanner trace 流。 - Acceptance: an mc-equivalent tool can subscribe to the heal/scanner trace stream.
**HS-04 scanner 超限 S3 事件与审计缺失** **HS-04 Scanner excess S3 events and auditing missing**
- 现状:仅 `rustfs_scanner_excess_*_total` 指标(versions 100/version size 1TiB/folders 65538)。 - Current state: only `rustfs_scanner_excess_*_total` metrics (versions 100 / version size 1TiB / folders 65538).
- MinIO:发 `s3:ObjectManyVersions`>100 版本)、`s3:ObjectLargeVersions`(累计 >1TB)、`s3:PrefixManyFolders`>50000 子目录)事件(UserAgent: Scanner+ scanner:manyversions/largeversions/manyprefixes 审计。 - MinIO: emits `s3:ObjectManyVersions` (>100 versions), `s3:ObjectLargeVersions` (cumulative >1TB), `s3:PrefixManyFolders` (>50000 subdirectories) events (UserAgent: Scanner) + scanner:manyversions/largeversions/manyprefixes auditing.
- 影响:依赖事件订阅做容量治理的用户(console/外部审计)收不到告警。 - Impact: users relying on event subscriptions for capacity governance (console/external auditing) receive no alerts.
- 建议:scanner_folder 告警点接入 notify 事件发布(复用 lifecycle 事件通道语义)。 - Recommendation: hook the scanner_folder alert points into notify event publishing (reusing the lifecycle event-channel semantics).
- 验收:配置桶通知后超限对象触发事件。 - Acceptance: after configuring bucket notifications, an over-threshold object triggers an event.
**HS-05 madmin 客户端方法缺失** **HS-05 madmin client methods missing**
- 现状:`crates/madmin/src/heal_commands.rs` 只有 wire 类型(HealDriveInfo/Infos/HealResultItem);无 HealStart/HealStatus/BackgroundHealStatus/ScannerStatus 客户端方法。 - Current state: `crates/madmin/src/heal_commands.rs` has only wire types (HealDriveInfo/Infos/HealResultItem); no HealStart/HealStatus/BackgroundHealStatus/ScannerStatus client methods.
- MinIOmadmin-go 提供完整客户端;mc admin heal/scanner/status/trace 都建立在上面。 - MinIO: madmin-go provides the full client; mc admin heal/scanner/status/trace are all built on it.
- 影响:mc 等管理工具无法直接对接 RustFS heal/scanner 管理面;自动化运维只能手写 HTTP - Impact: admin tools like mc cannot directly drive the RustFS heal/scanner admin surface; automated operations must hand-write HTTP.
- 建议:按 madmin-go 接口形状补客户端(服务端已就绪,纯客户端工作)。 - Recommendation: add the client following the madmin-go interface shape (the server side is ready; this is pure client work).
- 验收:用 madmin 客户端完成 start→query→cancel 全流程。 - Acceptance: complete the start→query→cancel flow with the madmin client.
**HS-06 admin heal 序列语义与 MinIO 差异** **HS-06 admin heal sequence semantics differ from MinIO**
- 现状:重复/重叠请求被去重合并(返回 canonical task_id)或驱逐;无 ErrHealAlreadyRunning/ErrHealOverlappingPaths 类型化错误(已亲验:manager.rs:1309 already_running 是幂等启动保护,非 admin 语义);结果为快照式查询(1024 条/8MiB 截断/10min 保留),非 MinIO 的流式增量(clientToken 拉增量 + maxUnconsumedItems=1000 backpressure + 10s 保活 + 24h 未消费 abort)。 - Current state: duplicate/overlapping requests are dedup-merged (returning the canonical task_id) or evicted; no ErrHealAlreadyRunning/ErrHealOverlappingPaths typed errors (verified first-hand: manager.rs:1309's already_running is an idempotent-startup guard, not an admin semantic); results are snapshot-style queries (1024 entries/8MiB truncation/10min retention), not MinIO's streaming increments (clientToken pulls increments + maxUnconsumedItems=1000 backpressure + 10s keep-alive + 24h unconsumed abort).
- 影响:mc admin heal 的交互模型(长连接拉增量)对 RustFS 表现为多次快照轮询;自动化脚本难以区分"已合并"与"新启动"。 - Impact: mc admin heal's interaction model (long connection pulling increments) behaves against RustFS as multiple snapshot polls; automation scripts cannot easily distinguish "merged" from "newly started".
- 建议:① 增量语义:channel query 支持自上次 clientToken 起的 items 增量(或 cursor);② 重叠请求返回类型化错误码(或 receipt 中显式 merged_into 字段——现有 alias 机制已有基础);③ forceStart 先停旧再启新语义核对。 - Recommendation: ① incremental semantics: channel query supports item increments since the last clientToken (or a cursor); ② overlapping requests return a typed error code (or an explicit merged_into field in the receipt — the existing alias mechanism already provides the base); ③ verify forceStart's stop-old-then-start-new semantics.
- 验收:madmin 兼容客户端按 MinIO 模式轮询能取得全量 items。 - Acceptance: an madmin-compatible client polling in the MinIO style can retrieve the full item set.
**HS-07 healing 进度与盘级 healing 状态对外可见性不足** **HS-07 healing progress and drive-level healing state insufficiently visible externally**
- 现状:bytes 恢复进度 `progress.bytes_processed = 0 // set to 0 for now`erasure_healer.rs:967);`HealProgress::estimated_completion_time` 恒 None、`HealStatistics::add_healed_objects` 未写入(progress.rs:38,135-139 零调用);healthinfo 无每盘 HealInfo 等价(MinIO HealingDiskBytesDone/Failed/SkippedObjectsTotal 基线、QueuedBuckets/HealedBucketsResume 快照、当前 object);v3 指标无 drive_health=2(healing) 单一 gauge 等价。 - Current state: byte-recovery progress `progress.bytes_processed = 0 // set to 0 for now` (erasure_healer.rs:967); `HealProgress::estimated_completion_time` is always None and `HealStatistics::add_healed_objects` is never written (progress.rs:38,135-139 zero calls); healthinfo has no per-drive HealInfo equivalent (MinIO HealingDisk: BytesDone/Failed/Skipped, ObjectsTotal baseline, QueuedBuckets/HealedBuckets, Resume snapshot, current object); v3 metrics lack an equivalent of the single drive_health=2 (healing) gauge.
- 影响:换盘重建(可能数小时~天)期间运维无法回答"进行到哪/还剩多少/预计何时完成"。 - Impact: during a drive rebuild (potentially hours to days) operations cannot answer "where are we / how much is left / when will it finish".
- 建议:① erasure set heal 统计 bytesheal_object 返回对象大小已可得);② 从 usage-cache 读对象总量基线(MinIO 同款做法);③ admin healthinfo/背景状态暴露每盘 healing 快照(DiskInfo.healing 已有,补聚合暴露);④ ETA 由基线+速率推导。 - Recommendation: ① accumulate bytes in erasure set heal (heal_object already yields the object size); ② read the object-total baseline from usage-cache (the same approach as MinIO); ③ expose a per-drive healing snapshot in admin healthinfo/background status (DiskInfo.healing already exists; add the aggregated exposure); ④ derive the ETA from baseline + rate.
- 验收:换盘重建中 admin 可见 bytes 进度与 ETAmc info 等价输出 Healing 标志。 - Acceptance: during a drive rebuild, admin shows byte progress and ETA; an mc info-equivalent output shows the Healing flag.
**HS-08 prefix usage 未暴露** **HS-08 prefix-level usage not exposed**
- 现状:DataUsageCache 内目录树 entry 存在(hash_path 组织),但 `dui()` flatten 到桶名(data_usage_define.rs:858-915)。 - Current state: the DataUsageCache holds the directory-tree entries (organized by hash_path), but `dui()` flattens only to the bucket name (data_usage_define.rs:858-915).
- MinIO`loadPrefixUsageFromBackend`30s cache)从每 set `.usage-cache.bin` 聚合 prefix usageconsole 桶前缀统计消费。 - MinIO: `loadPrefixUsageFromBackend` (30s cache) aggregates prefix usage from each set's `.usage-cache.bin`, consumed by console bucket-prefix statistics.
- 影响:console/前端无法展示前缀级用量;大桶定位"哪个前缀占空间"无 API。 - Impact: console/front ends cannot show prefix-level usage; there is no API to locate "which prefix is using the space" in a large bucket.
- 建议:实现 flatten 前缀查询 API(数据已在缓存内,纯聚合与暴露工作)。 - Recommendation: implement a prefix-flattening query API (the data is already in the cache; this is pure aggregation and exposure work).
- 验收:ListBuckets/PrefixUsage API 返回与前缀过滤匹配的统计。 - Acceptance: a ListBuckets/PrefixUsage API returns statistics matching the prefix filter.
### P29 项) ### P2 (9 items)
**HS-09 get_disk_status 恒返回 Ok(唯一 TODO**`crates/heal/src/heal/storage.rs:930-943`(已亲验)。当前无生产调用方(低风险)。建议:删除该方法或接 ecstore disk 状态真实现(DiskStatus 枚举已定义)。 **HS-09 get_disk_status always returns Ok (the only TODO)**: `crates/heal/src/heal/storage.rs:930-943` (verified first-hand). Currently no production caller (low risk). Recommendation: delete the method or wire it to the real ecstore disk status (the DiskStatus enum is already defined).
**HS-10 HealStorageAPI 约 1/3 方法为死代码**get_object_meta/get_object_data/put_object_data/delete_object/verify_object_integrity/ec_decode_rebuild/get_disk_status/format_disk/heal_bucket_metadata/get_object_size/get_object_checksum/list_objects_for_heal(非分页版,自带 memory_heavy 警告)均 0 调用方。建议:随 HS-01 决策一并清理或接线(死接口误导后续维护者以为存在调用路径)。 **HS-10 About 1/3 of HealStorageAPI methods are dead code**: get_object_meta/get_object_data/put_object_data/delete_object/verify_object_integrity/ec_decode_rebuild/get_disk_status/format_disk/heal_bucket_metadata/get_object_size/get_object_checksum/list_objects_for_heal (the non-paginated version, with its own memory_heavy warning) all have 0 callers. Recommendation: clean up or wire them together with the HS-01 decision (dead interfaces mislead future maintainers into thinking a call path exists).
**HS-11 bitrot 自检缺失**MinIO 启动时 bitrotSelfTest 对四算法已知向量自检失败即 Fatal(防静默数据损坏)。RustFS 无等价(已亲验 grep)。建议:启动时对 HighwayHash256S 等在用算法做已知向量自检(低成本高价值)。 **HS-11 bitrot self-test missing**: MinIO at startup runs bitrotSelfTest over known vectors for the four algorithms and exits Fatal on failure (guarding against silent data corruption). RustFS has no equivalent (verified first-hand by grep). Recommendation: at startup, run known-vector self-tests for HighwayHash256S and the other algorithms in use (low cost, high value).
**HS-12 对象级 healing 元数据标记评估**MinIO heal 期间对象打 `x-minio-healing:true`RenameData 据此跳过版本清理/legacy purge(漏掉会导致 heal 与并发删除互毁)。RustFS 无对象级标记(已亲验 grep object.rs healing 分支),依赖 NSLock + rename 语义。建议:审计 RustFS rename 提交路径是否存在"heal 提交与并发 delete/version 清理竞争"窗口;若无则文档化差异,若有则补标记等价机制。 **HS-12 object-level healing metadata marker evaluation**: during heal, MinIO tags objects with `x-minio-healing:true`, and RenameData uses it to skip version cleanup/legacy purge (missing it lets heal and concurrent deletes destroy each other). RustFS has no object-level marker (verified first-hand by grep; object.rs has no healing branch) and relies on NSLock + rename semantics. Recommendation: audit whether the RustFS rename-commit path has a "heal commit racing concurrent delete/version cleanup" window; if not, document the difference, and if so, add a marker-equivalent mechanism.
**HS-13 erasure set heal 无"跳过新写入/ILM 已过期版本"过滤**MinIO resync 跳过 ModTime>tracker.Started 的版本(避免 heal 追新写入尾巴)与 ILM 已过期版本(避免白做)。RustFS erasure_healer 未实现同款过滤(按版本 dedup 有,时间/ILM 过滤无)。影响:重建尾部长尾(持续写入的桶 heal 完成判定被新版本推迟)与无效 heal 工作量。建议:disk-walk 枚举处加 started_at 时间过滤 + evaluator 预检。 **HS-13 erasure set heal lacks "skip newly written / ILM-expired versions" filters**: MinIO resync skips versions with ModTime>tracker.Started (so heal does not chase the tail of new writes) and ILM-expired versions (so work is not wasted). RustFS's erasure_healer does not implement such filters (per-version dedup exists; time/ILM filters do not). Impact: a long tail on rebuild completion (the completion decision for a continuously written bucket is pushed out by new versions) and wasted heal work. Recommendation: add a started_at time filter at the disk-walk enumeration point + an evaluator pre-check.
**HS-14 scanner idle 语义方向与 MinIO 相反**MinIO `scanner:idle_speed=on`(默认)= 集群空闲时才节流、忙时全速;RustFS `RUSTFS_SCANNER_IDLE_MODE=true`(默认)= 限速总闸(false=完全不休眠)。两者默认行为可能相近(都限速)但参数语义不可互换,迁移文档需显式说明;若追求 mc config 兼容需重命名/重语义。建议:先文档化差异,评估是否对齐语义。 **HS-14 scanner idle semantics point the opposite way from MinIO**: MinIO `scanner:idle_speed=on` (default) means "throttle only when the cluster is idle, full speed when busy"; RustFS `RUSTFS_SCANNER_IDLE_MODE=true` (default) is a master switch for rate limiting (false = never sleep at all). The default behaviors may end up similar (both throttle), but the parameter semantics are not interchangeable; migration docs must state this explicitly; if mc config compatibility is the goal, a rename/re-semantization is needed. Recommendation: document the difference first, then evaluate aligning the semantics.
**HS-15 alert_excess_folders 默认值差异**RustFS 65538(兼容 PBS/Proxmox 布局,scanner_folder.rs:79vs MinIO 50000。行为差异默认即触发阈值不同。建议:文档化(保留 65538 有本地理由)。 **HS-15 alert_excess_folders default differs**: RustFS 65538 (compatible with the PBS/Proxmox layout, scanner_folder.rs:79) vs MinIO 50000. The behavioral difference is that the trigger threshold differs out of the box. Recommendation: document it (keeping 65538 has local rationale).
**HS-16 单机默认周期钩子未启用**`single_disk_default_cycle_secs(_features) -> None` 恒空(scanner.rs:1428-1430),单机部署无专属默认周期覆盖。建议:决定单机默认周期策略后启用或删除钩子。 **HS-16 single-node default-cycle hook not enabled**: `single_disk_default_cycle_secs(_features) -> None` is always empty (scanner.rs:1428-1430); single-node deployments get no dedicated default-cycle override. Recommendation: after deciding the single-node default-cycle policy, enable or delete the hook.
**HS-17 DeleteAllVersions 批量优化核对**MinIO 用 DeletePrefix+DeletePrefixObject 单调用代替逐版本 fan-out。RustFS expiry 队列路径是否同款优化未逐行核实(集成测试覆盖行为正确性)。建议:核对 `apply_expiry_rule` 全版本删除路径,若无前缀单调用优化则评估补齐。 **HS-17 DeleteAllVersions batch-optimization check**: MinIO uses the single DeletePrefix+DeletePrefixObject call instead of per-version fan-out. Whether RustFS's expiry-queue path has the same optimization was not verified line by line (integration tests cover behavioral correctness). Recommendation: check the `apply_expiry_rule` all-versions delete path; if there is no prefix single-call optimization, evaluate adding it.
### P33 项) ### P3 (3 items)
**HS-18 trash/临时目录二段清理细节核对**:MinIO `.minio.sys/tmp/.trash` 清理(delete_cleanup_interval 默认 5m + deleteCleanupSleeper)与 stale uploads rename-into-trash 二段式。RustFS delete_tail_activity.rs stale multipart 任务,二段语义是否完整对齐未逐行核实。建议:对照补齐或文档化。 **HS-18 trash/temp-directory two-stage cleanup detail check**: MinIO cleans `.minio.sys/tmp/.trash` (delete_cleanup_interval default 5m + deleteCleanupSleeper) and stale uploads are renamed into trash in two stages. RustFS has delete_tail_activity.rs and the stale multipart task; whether the two-stage semantics are fully aligned was not verified line by line. Recommendation: align or document.
**HS-19 root heal 直连死路径清理**`should_handle_root_heal_directly` falseadmin/handlers/heal.rs:1200-1202,测试锁定),store.heal_format 直连分支不可达。建议:删除死分支或恢复直连路径作为集群协调失败的降级。 **HS-19 root-heal direct path is dead code**: `should_handle_root_heal_directly` is always false (admin/handlers/heal.rs:1200-1202, locked by a test); the store.heal_format direct branch is unreachable. Recommendation: delete the dead branch or restore the direct path as a fallback for cluster-coordination failure.
**HS-20 兼容旗标与死指标清理**`RUSTFS_SCANNER_INLINE_HEAL_ENABLE`(开启仅告警)+ `rustfs_scanner_inline_heal_total` 死指标 + `rustfs_common::metrics` 中 scanner 域代码分层迁移(backlog #1843 已登记)。建议:随分层迁移一并清理。 **HS-20 compat flags and dead metrics cleanup**: `RUSTFS_SCANNER_INLINE_HEAL_ENABLE` (enabling only warns) + the dead `rustfs_scanner_inline_heal_total` metric + the scanner-domain code in `rustfs_common::metrics` awaiting layering migration (backlog #1843 already filed). Recommendation: clean up along with the layering migration.
### 按设计不追平(7 项,记录以防后续误判为缺口) ### Not pursuing parity by design (7 items, recorded to prevent later misreading as gaps)
1. **bloom filter**MinIO master 已删除;RustFS `.bloomcycle.bin` 复用为 cycle/epoch 围栏与 MinIO 现状一致。 1. **bloom filter**: removed from MinIO master; RustFS reuses `.bloomcycle.bin` as the cycle/epoch fence, consistent with MinIO's current state.
2. **scanner 集群单 leader**:双方一致;RustFS 额外有 epoch 围栏。 2. **scanner cluster single leader**: both sides agree; RustFS additionally has the epoch fence.
3. **heal 不发 S3 bucket notification**:双方一致(heal 结果走 admin status)。 3. **heal emits no S3 bucket notification**: both sides agree (heal results go through admin status).
4. **incomplete multipart 不在 scanner/ILM 内执行**:双方一致(独立后台例程)。 4. **incomplete multipart not executed inside scanner/ILM**: both sides agree (independent background routine).
5. **内联 heal 移除**RustFS 有意为之(scanner 只入队),MinIO applyHealing 内联路径不做对标。 5. **inline heal removal**: a deliberate RustFS choice (the scanner only enqueues); MinIO's applyHealing inline path is not a parity target.
6. **heal 序列常驻保活(10s 空白回写)**RustFS 快照式查询模型不同,按 HS-06 处理增量语义即可,不复制流式保活。 6. **heal-sequence resident keep-alive (10s blank write-back)**: RustFS's snapshot-query model differs; handling incremental semantics per HS-06 is enough — do not copy the streaming keep-alive.
7. **`.trash`/`tmp-old` 路径名兼容**:RustFS 布局常量独立,不逐字对齐 MinIO 路径。 7. **`.trash`/`tmp-old` path-name compatibility**: RustFS's layout constants are independent; no literal alignment with MinIO paths.
--- ---
## 6. 配置默认值总表(RustFS ## 6. Configuration defaults master table (RustFS)
healenv 前缀 `RUSTFS_HEAL_``crates/config/src/constants/heal.rs`,消费于 `manager.rs:724-800`): heal (env prefix `RUSTFS_HEAL_`, `crates/config/src/constants/heal.rs`, consumed at `manager.rs:724-800`):
| 配置 | 默认 | 热更新 | | Setting | Default | Hot update |
|---|---|---| |---|---|---|
| AUTO_HEAL_ENABLE | true | | | AUTO_HEAL_ENABLE | true | no |
| QUEUE_SIZE | 10000 | | | QUEUE_SIZE | 10000 | no |
| INTERVAL_SECS | 10 | 否(启动时固定) | | INTERVAL_SECS | 10 | no (fixed at startup) |
| TASK_TIMEOUT_SECS | 300 | | | TASK_TIMEOUT_SECS | 300 | no |
| MAX_CONCURRENT_HEALS | 4 | | | MAX_CONCURRENT_HEALS | 4 | no |
| MAX_CONCURRENT_PER_SET | 1≤min(全局,值) | | | MAX_CONCURRENT_PER_SET | 1 (≤min(global, value)) | no |
| LOW_PRIORITY_MERGE_ENABLE | true | | | LOW_PRIORITY_MERGE_ENABLE | true | no |
| LOW_PRIORITY_DROP_WHEN_FULL | true | | | LOW_PRIORITY_DROP_WHEN_FULL | true | no |
| PAGE_OBJECT_CONCURRENCY | 8Deep/AutoHeal 强制 1 | | | PAGE_OBJECT_CONCURRENCY | 8 (Deep/AutoHeal forced to 1) | no |
| EVENT_DRIVEN_SCHEDULER_ENABLE | true | | | EVENT_DRIVEN_SCHEDULER_ENABLE | true | no |
| SET_BULKHEAD_ENABLE | true | | | SET_BULKHEAD_ENABLE | true | no |
| PAGE_PARALLEL_ENABLE | true | | | PAGE_PARALLEL_ENABLE | true | no |
| MAINLINE_THROTTLE_ENABLE | true | | | MAINLINE_THROTTLE_ENABLE | true | no |
| MAINLINE_READ/WRITE_UTILIZATION_HIGH_PERCENT | 80/80 | | | MAINLINE_READ/WRITE_UTILIZATION_HIGH_PERCENT | 80/80 | no |
| MAINLINE_MAX_SLEEP_MS | 250 | | | MAINLINE_MAX_SLEEP_MS | 250 | no |
| (总开关)RUSTFS_HEAL_ENABLED | true | | | (master switch) RUSTFS_HEAL_ENABLED | true | no |
| admin 子系统 heal.bitrot_cycle | 30d | 是(经 scanner runtime config | | admin subsystem heal.bitrot_cycle | 30d | yes (via scanner runtime config) |
scanneradmin 子系统 `scanner``crates/config/src/constants/scanner.rs` + `ecstore/src/config/scanner.rs` + `runtime_config.rs:527-673`): scanner (admin subsystem `scanner`, `crates/config/src/constants/scanner.rs` + `ecstore/src/config/scanner.rs` + `runtime_config.rs:527-673`):
| | env | 默认 | | Key | env | Default |
|---|---|---| |---|---|---|
| speed | RUSTFS_SCANNER_SPEED | default2x/1s/60s | | speed | RUSTFS_SCANNER_SPEED | default (2x/1s/60s) |
| delay / max_wait / cycle / start_delay | RUSTFS_SCANNER_* | 派生/空 | | delay / max_wait / cycle / start_delay | RUSTFS_SCANNER_* | derived/empty |
| cycle_max_duration/objects/directories | …_MAX_* | 0(不限) | | cycle_max_duration/objects/directories | …_MAX_* | 0 (unlimited) |
| bitrot_cycle | …_BITROT_CYCLE_SECS | 259200030d0/on=每轮,off=禁用) | | bitrot_cycle | …_BITROT_CYCLE_SECS | 2592000 (30d; 0/on=every cycle, off=disabled) |
| idle_mode | …_IDLE_MODE | true | | idle_mode | …_IDLE_MODE | true |
| cache_save_timeout | …_CACHE_SAVE_TIMEOUT_SECS | 30s | | cache_save_timeout | …_CACHE_SAVE_TIMEOUT_SECS | 30s |
| max_concurrent_set_scans / disk_scans | …_MAX_CONCURRENT_* | 4/4 | | max_concurrent_set_scans / disk_scans | …_MAX_CONCURRENT_* | 4/4 |
| yield_every_n_objects | …_YIELD_EVERY_N_OBJECTS | 128 | | yield_every_n_objects | …_YIELD_EVERY_N_OBJECTS | 128 |
| alert_excess_versions / version_size / folders | …_ALERT_* | 100 / 1TiB / 65538 | | alert_excess_versions / version_size / folders | …_ALERT_* | 100 / 1TiB / 65538 |
scanner 内部 env`RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=16``RUSTFS_HEAL_OBJECT_SELECT_PROB=1024``RUSTFS_SCANNER_DEEP_VERIFY_COOLDOWN_SECS=60``RUSTFS_DATA_USAGE_FAILED_OBJECT_TTL_SECS=86400`/`_MAX=10000``RUSTFS_LOCK_ACQUIRE_TIMEOUT=5s``RUSTFS_SCANNER_ENABLED=true``RUSTFS_SCANNER_INLINE_HEAL_ENABLE=false`(兼容告警)。 scanner-internal env: `RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=16`, `RUSTFS_HEAL_OBJECT_SELECT_PROB=1024`, `RUSTFS_SCANNER_DEEP_VERIFY_COOLDOWN_SECS=60`, `RUSTFS_DATA_USAGE_FAILED_OBJECT_TTL_SECS=86400`/`_MAX=10000`, `RUSTFS_LOCK_ACQUIRE_TIMEOUT=5s`, `RUSTFS_SCANNER_ENABLED=true`, `RUSTFS_SCANNER_INLINE_HEAL_ENABLE=false` (compat warning).
全部 17 scanner 键支持 env > config 双通道 + admin PUT 热更(generation+Notify 即时生效);heal 运行时参数目前仅 env(无 admin 热更入口,`Arc<RwLock<HealConfig>>` 结构已预留)。 All 17 scanner keys support the env > config dual channel + admin PUT hot update (generation+Notify takes effect immediately); heal runtime parameters are currently env-only (no admin hot-update entry; the `Arc<RwLock<HealConfig>>` structure is already reserved).
--- ---
## 7. 相关 backlog / 历史索引 ## 7. Related backlog / history index
- 换盘自动修复系列(已闭环):backlog #1786(冗余假绿算法)、#1787(目标槽位限定)、#1789resume 与 healing marker 绑定 replacement 实例)、#1791(黑白盒验收矩阵)。 - Automatic drive-replacement healing series (closed loop): backlog #1786 (redundant false-green algorithm), #1787 (target-slot restriction), #1789 (binding resume and the healing marker to the replacement instance), #1791 (black-box/white-box acceptance matrix).
- #801 DiskInfo.healing 从未赋值(已修复闭环,现 `set_disk/mod.rs:4988` 有赋值链)。 - #801 DiskInfo.healing never assigned (fixed and closed; the assignment chain now lives at `set_disk/mod.rs:4988`).
- #1651 Scanner 指标节点/source/bucket-drive 维度(OPEN,本分析 §3.8/§4.6 相关)。 - #1651 Scanner metrics node/source/bucket-drive dimensions (OPEN; related to §3.8/§4.6 of this analysis).
- #1843 crates/common 83% scanner/heal 域代码分层迁移(OPEN,含 HS-20)。 - #1843 crates/common 83% scanner/heal domain code layering migration (OPEN; includes HS-20).
- 代码注释引用的历史缺陷(现已有防护与回归测试):#856/#799 B7(离线盘误记 healed)、#855/B6/#1033skip 不得标记完成)、#920sub-quorum 并集枚举)、#856 B5(按版本续扫)、#5173bitrot trailing bytes)、#5029(回归节点 stale 版本合并)。 - Historical defects cited in code comments (now guarded with regression tests): #856/#799 B7 (offline drive falsely recorded healed), #855/B6/#1033 (a skip round must not be marked complete), #920 (sub-quorum union enumeration), #856 B5 (per-version resume), #5173 (bitrot trailing bytes), #5029 (stale-version merge at regression nodes).
- v1 对标文档:`docs/rustfs-heal-scanner-vs-minio-parity-assessment.md`(本文取代)、落地手册 `docs/rustfs-heal-scanner-vs-minio-improvement-playbook.md`(部分条目已被后续实现超越)。 - v1 parity document: `docs/rustfs-heal-scanner-vs-minio-parity-assessment.md` (superseded by this document); the landing playbook `docs/rustfs-heal-scanner-vs-minio-improvement-playbook.md` (some entries have since been overtaken by implementation).
- 换盘深度分析:`docs/new-disk-replacement-and-healing-deep-analysis-zh.md``docs/node-disk-identity-and-healing-analysis-zh.md` - Drive-replacement deep analyses: `docs/new-disk-replacement-and-healing-deep-analysis-zh.md`, `docs/node-disk-identity-and-healing-analysis-zh.md`.
## 8. 审计方法与局限 ## 8. Audit method and limitations
- 四路并行审计(heal crate 逐文件、scanner crate 逐文件、ecstore 集成层 wiringMinIO master 源码研究)+ 主会话对关键"缺失"结论逐条亲验(get_disk_status TODOHealEvent 零外部引用、.bloomcycle.bin 无 bloom 实现、check_abandoned_parts 三层 NotImplemented、ETag 兜底已实现、trace 通道零命中、already_running 语义)。 - Four parallel audit tracks (heal crate file by file, scanner crate file by file, ecstore integration-layer wiring, MinIO master source study) + the main session verifying each key "missing" conclusion first-hand (the get_disk_status TODO, HealEvent's zero external references, .bloomcycle.bin having no bloom implementation, check_abandoned_parts NotImplemented at all three layers, the ETag fallback being implemented, zero trace-channel hits, the already_running semantics).
- 未逐行核实的点(已在文中标注"未确认/未逐行核"):DeleteAllVersions 前缀单调用优化(HS-17)、trash 二段清理细节(HS-18)、ilm worker 默认值对照、stale multipart 默认值对照、mc CLI flag 逐字拼写(MinIO 侧)。其中 HS-17 HS-18 已于 2026-08-19 完成逐行核实,结论见 §9.2/§9.3 - Points not verified line by line (marked "unconfirmed / not checked line by line" in the text): the DeleteAllVersions prefix single-call optimization (HS-17), trash two-stage cleanup details (HS-18), ilm worker default comparisons, stale multipart default comparisons, mc CLI flag spellings (MinIO side). Of these, HS-17 and HS-18 completed line-by-line verification on 2026-08-19; conclusions in §9.2/§9.3.
- MinIO 侧引用以其 master `7aac2a2c5b` 为准;RustFS 侧行号以 2026-08-16 工作区为准,后续演进请以符号名检索为准。 - MinIO-side references follow its master `7aac2a2c5b`; RustFS-side line numbers follow the 2026-08-16 workspace — for later evolution, search by symbol name instead.
## 9. 落地结果(2026-08-19 更新) ## 9. Landing results (updated 2026-08-19)
本审计衍生的 14 个子 issuebacklog #1865~#1878)已全部闭环。本节为差距清单 HS-01~HS-20 的最终处置记录,也是下一轮对标重审的增量基线。 All 14 sub-issues derived from this audit (backlog #1865~#1878) are closed. This section is the final disposition record for the gap list HS-01~HS-20, and also the incremental baseline for the next parity re-audit.
### 9.1 已落地(PR 均已合并 main ### 9.1 Landed (all PRs merged to main)
- HS-01 MRF 接线 + 持久化修复账本(#1865PR #6189):决策选 (a)。common MRF channelbounded 8192try_send 永不阻塞)+ heal mrf_queue100k / 8MiB 双限环形)+ `buckets/.heal/mrf/journal.bin` CRC 持久化回放(torn tail 截断、回放后删除)+ 三投递点(read decode_error→Urgent ECDecodescanner 元数据损坏→High Metadataadd_partial→Normal+ `RUSTFS_HEAL_MRF_ENABLE` 一键回退。 - HS-01 MRF wiring + persistent repair ledger (#1865, PR #6189): decision (a) chosen. common MRF channel (bounded 8192, try_send never blocks) + heal mrf_queue (100k entries / 8MiB dual-capacity ring) + `buckets/.heal/mrf/journal.bin` CRC-persisted replay (torn tail truncated, deleted after replay) + three delivery points (read decode_error→Urgent ECDecode, scanner metadata corruption→High Metadata, add_partial→Normal) + `RUSTFS_HEAL_MRF_ENABLE` one-switch rollback.
- HS-02 abandoned parts/data-dir 对账(#1866PR #6179):接通 abandoned 检查入口,保留 dry-run / reclaim 计数。 - HS-02 abandoned parts/data-dir reconciliation (#1866, PR #6179): wired up the abandoned-check entry, retaining dry-run / reclaim counters.
- HS-03 heal/scanner trace 通道(#1867PR #6179):进程内 trace bus + `/v3/trace` admin 流式订阅 + heal task / abandoned-parts / scanner folder / ILM / heal-candidate trace producer - HS-03 heal/scanner trace channels (#1867, PR #6179): in-process trace bus + `/v3/trace` admin streaming subscription + heal task / abandoned-parts / scanner folder / ILM / heal-candidate trace producers.
- HS-04 scanner 超限 S3 事件(#1868PR #6176):`s3:Scanner:ManyVersions/LargeVersions/BigPrefix` 三事件 + 24h 边沿冷却;HS-15 阈值差异文档化(`docs/operations/scanner-excess-alerts.md`)。 - HS-04 scanner excess S3 events (#1868, PR #6176): the three events `s3:Scanner:ManyVersions/LargeVersions/BigPrefix` + 24h edge cooldown; the HS-15 threshold delta documented (`docs/operations/scanner-excess-alerts.md`).
- HS-05 madmin 客户端一期(#1869PR #6166):SigV4 admin 客户端 heal/scanner 方法;增量消费方法待 follow-up(协议已由 HS-06 并入)。 - HS-05 madmin client phase 1 (#1869, PR #6166): SigV4 admin client heal/scanner methods; incremental-consumption methods await a follow-up (the protocol was already folded in by HS-06).
- HS-06 admin heal 增量语义与类型化重叠(#1870PR #6206):`sinceSeq/nextSeq/minSeq` 增量游标(wire additive、缺省=全量快照)+ `RUSTFS_HEAL_OVERLAP_POLICY`(默认 merge 不变;minio_error AlreadyRunning/OverlappingPaths 类型化拒绝)+ forceStart 先停旧再启新。 - HS-06 admin heal incremental semantics and typed overlap (#1870, PR #6206): `sinceSeq/nextSeq/minSeq` incremental cursor (wire additive; absent = full snapshot) + `RUSTFS_HEAL_OVERLAP_POLICY` (default merge unchanged; under minio_error, typed AlreadyRunning/OverlappingPaths rejections) + forceStart stops the old sequence before starting the new one.
- HS-07 healing 进度可见性(#1871PR #6179):data-usage 总量基线 + baseline/current/healed 计数。 - HS-07 healing progress visibility (#1871, PR #6179): data-usage total baseline + baseline/current/healed counters.
- HS-08 prefix usage#1872PR #6171):`GET /v3/usage/{bucket}` - HS-08 prefix usage (#1872, PR #6171): `GET /v3/usage/{bucket}`.
- HS-11 bitrot 启动自检(#1873PR #6165)。 - HS-11 bitrot startup self-test (#1873, PR #6165).
- HS-13 heal 跳过过滤(#1875PR #6179):过滤命中版本不再计为失败。 - HS-13 heal skip filters (#1875, PR #6179): filter-hit versions are no longer counted as failures.
- HS-16 单机周期钩子(#1878PR #6250):删恒 None 钩子,决策记录见 `docs/operations/heal-scanner-parity-notes-zh.md` - HS-16 single-node cycle hook (#1878, PR #6250): removed the always-None hook; the decision record is in `docs/operations/heal-scanner-parity-notes-zh.md`.
- HS-09/10/19/20 死代码清理批(#1877PR #6256):净 911 行零行为变更;`get_disk_status` TODO(全仓库唯一产品 TODO)清零;HS-01 联动的 `ec_decode_rebuild`/`get_object_meta` 保留并加 Reserved 注释(MRF 当前经 `heal_object` 执行)。 - HS-09/10/19/20 dead-code cleanup batch (#1877, PR #6256): net 911 lines, zero behavior change; the `get_disk_status` TODO (the repo's only product TODO) cleared to zero; `ec_decode_rebuild`/`get_object_meta`, kept due to the HS-01 linkage, are retained with Reserved annotations (MRF currently executes via `heal_object`).
### 9.2 核对后确认"已实现 / 非缺口"(审计期误判修正,累计四例) ### 9.2 Confirmed "already implemented / not a gap" after verification (audit-period misjudgment corrections, four in total)
- bloom filter(§0 已修正):MinIO master 已删除,双方现状一致。 - bloom filter (corrected in §0): removed from MinIO master; both sides now agree.
- ETag 兜底仲裁(§0 已修正):RustFS 已有实现(`set_disk/ops/heal.rs`)。 - ETag fallback arbitration (corrected in §0): RustFS already has the implementation (`set_disk/ops/heal.rs`).
- HS-17#18762026-08-19 逐行核实后关闭):DeleteAllVersions 前缀单调用优化 RustFS 已完整实现—`apply_expiry_on_non_transitioned_objects` `delete_all()` 两 action 设 `delete_prefix + delete_prefix_object` 后单次 `delete_object``bucket_lifecycle_ops.rs:5047-5056`),SetDisks 分支一次写锁 + 一次全版本 quorum 读 + 内联逐版本 object-lock 检查(`set_disk/ops/object.rs:5566-5612`),与 MinIO `expire.go` `applyExpiryOnNonTransitionedObjects` 逐行对齐。§8 原列"未逐行核实"的本项已有结论:现状即优化路径,无需实现。 - HS-17 (#1876, closed after line-by-line verification on 2026-08-19): the DeleteAllVersions prefix single-call optimization is fully implemented in RustFS — `apply_expiry_on_non_transitioned_objects` sets `delete_prefix + delete_prefix_object` for the two `delete_all()` actions and then performs a single `delete_object` call (`bucket_lifecycle_ops.rs:5047-5056`); the SetDisks branch takes one write lock + one all-version quorum read + inline per-version object-lock checks (`set_disk/ops/object.rs:5566-5612`), aligned line by line with MinIO `expire.go`'s `applyExpiryOnNonTransitionedObjects`. The item §8 listed as "not verified line by line" now has a conclusion: the current state is already the optimized path; nothing to implement.
- HS-14#1878PR #6250 附带核对):MinIO"idle=空闲才节流"是 2024-01 minio/minio#18734 之前的行为(`scannerIdleMode` 现为静态配置,`idle_speed=on` 默认即始终按速度档节流,"idle"命名是历史残留);RustFS `RUSTFS_SCANNER_IDLE_MODE` 与 MinIO 当前语义方向一致,且另有 MinIO 没有的前台读退避下限。真实迁移陷阱(变量须 `RUSTFS_` 前缀、`on/off` vs `true/false` 词表、`false` 连前台保护一起关)已文档化于 `docs/operations/heal-scanner-parity-notes-zh.md` - HS-14 (#1878, checked alongside PR #6250): MinIO's "idle = throttle only when idle" was the behavior before 2024-01 minio/minio#18734 (`scannerIdleMode` is now a static config; `idle_speed=on` by default means always throttling per the speed tier — the "idle" naming is a historical leftover); RustFS's `RUSTFS_SCANNER_IDLE_MODE` points the same way as MinIO's current semantics, and additionally has a foreground-read backoff floor that MinIO lacks. The real migration traps (the variable must carry the `RUSTFS_` prefix, the `on/off` vs `true/false` vocabulary, `false` also turning off foreground protection) are documented in `docs/operations/heal-scanner-parity-notes-zh.md`.
### 9.3 审计型结论(无需改代码) ### 9.3 Audit-style conclusions (no code change needed)
- HS-12#1874PR #6183):不存在 MinIO 用 `x-minio-healing` 防御的那类竞争——所有同 (bucket, object) 提交面在同一把对象级 ns 写锁互斥,heal guard 覆盖 rename 提交全程;交付 2 个并发不变量回归测试 + `docs/operations/heal-concurrency-safety-notes-zh.md` 交点矩阵。 - HS-12 (#1874, PR #6183): the class of race MinIO defends against with `x-minio-healing` does not exist — every commit surface for the same (bucket, object) is mutually exclusive under the same object-level ns write lock, and the heal lock guard covers the whole rename commit; delivered 2 concurrency-invariant regression tests + the intersection matrix in `docs/operations/heal-concurrency-safety-notes-zh.md`.
- HS-18#18782026-08-19 逐行核实):trash/tmp 三段清理全对齐——stale multipart 隔离-清理等价且更安全(`delete_all_with_quorum` 逐盘递归删即 `move_to_trash` rename `.rustfs.sys/tmp/.trash`,另有锁 + fence)、trash 排空基本等价(无逐条 sleeper 节流,5m 周期天然限频)、tmp 非 trash 24h 回收等价(RustFS 5m MinIO 6h 更及时);周期默认 24h/6h/5m 三项全对齐。§8 原列"未逐行核实"的本项已有结论。 - HS-18 (#1878, line-by-line verification on 2026-08-19): trash/tmp three-stage cleanup fully aligned — stale multipart isolation-cleanup is equivalent and safer (`delete_all_with_quorum` recursively deletes per drive, i.e. the `move_to_trash` rename into `.rustfs.sys/tmp/.trash`, plus lock + fence); trash draining is essentially equivalent (no per-entry sleeper throttling; the 5m cycle naturally rate-limits); tmp non-trash 24h reclamation is equivalent (RustFS's 5m is more timely than MinIO's 6h); the three cycle defaults 24h/6h/5m all align. The item §8 listed as "not verified line by line" now has a conclusion.
### 9.4 移交 follow-up(汇总于 backlog#1862 评论区) ### 9.4 Handed over to follow-ups (summarized in the backlog#1862 comment thread)
HS-01 bitrot GET→MRF 全链路 e2ekill -9 journal 回放 e2e、队列满压测 RSS(≤ 预算+10%);HS-05/06 madmin 增量消费方法 + wire 单一来源化 + embedded e2e + 多轮轮询 soakHS-08 多盘 scanner 周期 e2eHS-04 超限审计条目;HS-18 低于 quorum 的 stale-multipart 崩溃残留窗口(扇出中途崩溃且已清盘数 > parity FileNotFound 不在忽略集导致不自然收敛,修复需专用 quorum 变体)。 HS-01 bitrot GET→MRF full-chain e2e, kill -9 journal replay e2e, queue-full RSS stress test (≤ budget+10%); HS-05/06 madmin incremental-consumption methods + single-source wire + embedded e2e + multi-round polling soak; HS-08 multi-drive scanner cycle e2e; HS-04 excess audit entries; HS-18 the stale-multipart crash-residue window below quorum (crashing mid-fan-out with already-cleaned drives > parity means FileNotFound is not in the ignore set, so convergence is unnatural; the fix needs a dedicated quorum variant).
下一轮重审建议:跟随 heal/scanner 下一个大特性落地后触发,以本节为增量基线。 Recommendation for the next re-audit: trigger it after the next big heal/scanner feature lands, using this section as the incremental baseline.
@@ -0,0 +1,568 @@
# RustFS heal / scanner 全量功能分析与 MinIO 对标(v2)
> English version: [rustfs-heal-scanner-vs-minio-comprehensive-analysis-2026-08-16.md](rustfs-heal-scanner-vs-minio-comprehensive-analysis-2026-08-16.md)
- 日期:2026-08-16(基于 main 分支当日代码,审计时 HEAD ≈ `a118d7e4f`
- 范围:`crates/heal`src 19,560 行 + tests 2,274 行)、`crates/scanner`src 约 26,000 行 + tests)、`crates/data-usage``crates/ecstore` 中 heal/heal_walk/bitrot_self_verify 与 config、`crates/common/src/heal_channel.rs``crates/madmin`heal/scanner wire 类型)、`rustfs/src`startup wiring、admin handlers、集群 RPC
- 对标基线:minio/minio masterHEAD `7aac2a2c5b`,仓库已进入维护模式,master 冻结,即最终态)
- 方法:四路并行审计(heal crate / scanner crate / ecstore 集成层 / MinIO 源码研究),关键结论逐条人工抽验(文内标注"已亲验"处为一手验证)
- 本文档取代 `docs/rustfs-heal-scanner-vs-minio-parity-assessment.md`2026-06-15 v1)。v1 之后 heal/scanner 相关提交超过 80 个(换盘自动修复全链路、resume 状态机、usage 收敛权威化、集群级 heal 协调、ILM restore 语义等),v1 的功能清单与差距判断已全面过时;v1 中"bloom filter 缺失"等结论经本次核实为**误判**(详见 §5.4)。
---
## 0. 结论摘要
1. **总体判断:heal 与 scanner 的核心功能链路已经完整**。对象级 healquorum 仲裁 + ETag 兜底 + bitrot Deep 校验 + dangling 处理)、erasure set 深扫(per-set disk-walk 并集枚举)、按版本断点续扫(schema 化持久层 + CAS 原子发布 + 崩溃窗口补齐)、换盘自动修复(readiness 校验 + 身份围栏 + durable intent + completion proof)、scanner 周期循环(leader lock + 持久化 leader-epoch 围栏)、data usage 统计(桶级/集群级、主+备+观测快照、epoch/cycle 防回退)、ILM 全动作(expiry/transition/noncurrent/free-version/delete-marker 清理)、admin Start/Query/Cancel 协议(clientToken 语义对齐 madmin)——以上均有实现且带回归测试。两个 crate 内**没有空实现/早退桩**,异常路径全部有日志 + 指标 + 错误语义。
2. **主要缺口集中在"入口与观测面",而不是修复算法本身**MRF/ECDecode/Metadata 三类任务执行体已实现但无生产触发入口(`HealEvent` 完全未接线);`CheckAbandonedParts` 在 ecstore 三层全部 `NotImplemented`heal/scanner trace 通道缺失;scanner 超限 S3 事件缺失;madmin 客户端方法缺失(只有 wire 类型);heal 字节级进度/ETA 未实现。
3. **与 v1 认知的重要修正**bloom filter 在 MinIO 当前 master **已删除**`.bloomcycle.bin` 只存 cycle 计数),RustFS 现状与 MinIO 一致;MinIO scanner 同样是**集群级 leader 单例**RustFS 的 leader.lock 模型与 MinIO 同型;RustFS 的 ETag 多数派兜底仲裁已实现(`crates/ecstore/src/set_disk/ops/heal.rs:525-567,679`,已亲验),v1 担心的仲裁缺口不存在。
4. **RustFS 在多处超出 MinIO**remote_scanner RPC 协议(远端 peer 本地扫描而非 leader 跨网读远盘)、持久化 leader-epoch CAS 围栏、周期预算与 per-set/per-disk 并发闸、pending-heal 账本、durable replacement intent + completion proof 状态机、前台压力门控(mainline throttle)、集群 heal control coordinator + envelope 重放防护。
5. 差距分级统计:P1(行为/运维对齐缺口)8 项,P2(完善性)9 项,P3(清理/低风险)3 项,"按设计不追平"7 项。完整清单见 §6。
---
## 1. 架构总览
### 1.1 RustFS 三层架构
RustFS 把 MinIO 在 `cmd/` 内单体的 heal/scanner 拆成三层 + 两个独立 crate:
| 层 | 位置 | 职责 |
|---|---|---|
| 原语层 | `crates/ecstore/src/set_disk/ops/heal.rs`~3,240 行)、`ops/heal_walk.rs``ops/bitrot_self_verify.rs`;上层封装 `store/heal.rs``store/heal_walk.rs``core/sets.rs` | 对象/桶/format/替换盘格式修复、disk-walk 并集枚举、写入路径 bitrot 自校验;由 `SetDisks`/`Sets`/`ECStore` 实现 `rustfs_storage_api::HealOperations` 契约(`crates/storage-api/src/object.rs:503-519` |
| heal 运行时 | `crates/heal` | 进程级 HealManager(优先级队列/调度器/auto disk scanner/断点续传 resume)、HealChannelProcessor(消费全局 heal channel)、换盘替换恢复状态机 |
| scanner 运行时 | `crates/scanner` | 数据使用扫描、ILM 评估与入队、heal 候选生产、复制用量统计、remote scanner RPC |
| 共享协议 | `crates/common/src/heal_channel.rs`~776 行) | Start/Query/Cancel 命令通道、`HealOpts`/`HealScanMode`/`HealRequestSource`/`HealAdmission*` 共享类型、`HealResultItem`madmin |
| 共享数据 | `crates/data-usage` | `DataUsageEntry/Info`、直方图、`hash_path`scanner 产生、ecstore/admin 消费 |
启动链路(已亲验 wiring):
1. `rustfs/src/startup_services.rs:93``init_background_service_runtime(store)`
2. `rustfs/src/startup_background.rs:41-81`:创建全局 heal 服务取消令牌;读 `RUSTFS_SCANNER_ENABLED`(别名 `RUSTFS_ENABLE_SCANNER`,默认 true)与 `RUSTFS_HEAL_ENABLED`(别名 `RUSTFS_ENABLE_HEAL`,默认 true);**只要 heal 或 scanner 任一开启就初始化 heal manager**scanner 产生的 heal 候选需要消费端;两者都关时 heal channel 不初始化,`send_heal_request` 报 "Heal channel not initialized")。
3. `crates/heal/src/lib.rs:142-216`owned task 内原子初始化(caller 取消不会遗留半初始化 manager,`lib.rs:123-131``GLOBAL_HEAL_RUNTIME_INIT` 互斥单飞)→ `HealManager::start()``rustfs_common::heal_channel::init_heal_channels()` → spawn `HealChannelProcessor::start_with_receipts`
4. `crates/heal/src/heal/manager.rs:1301-1356` `HealManager::start``start_scheduler()``manager.rs:2394-2461`interval 默认 10s + `Notify` 事件驱动唤醒)→ `process_unclean_shutdown()``manager.rs:1362-1695`)→ `enable_auto_heal`(默认 true)时 `start_auto_disk_scanner()``manager.rs:2464-2999`)。
5. server ready 后 `rustfs/src/startup_lifecycle.rs:150-152``enable_scanner``init_data_scanner(token, store)``crates/scanner/src/scanner.rs:1293-1372`)。
6. 优雅停机:`rustfs/src/startup_shutdown.rs:308` `shutdown_ahm_services()`(取消令牌);`:414` `clear_unclean_shutdown_markers()`
### 1.2 MinIO 对应结构(master 最终态)
| MinIO 文件 | 职责 |
|---|---|
| `cmd/admin-heal-ops.go` | 手动 admin heal 序列(healSequence、clientToken/forceStart/forceStop |
| `cmd/global-heal.go` | 常驻后台 heal 队列(newBgHealSequencetoken 固定 `0000-…`,永不结束)+ `healErasureSet`(逐 set 全量对象 heal |
| `cmd/background-heal-ops.go` | healRoutine worker 池(`_MINIO_HEAL_WORKERS`,默认 GOMAXPROCS/2)消费 healTask |
| `cmd/mrf.go` | MRFMost Recent Fail)队列(容量 100,000),进程退出时持久化 `.minio.sys/buckets/.heal/mrf/list.bin` 并启动回放 |
| `cmd/background-newdisks-heal-ops.go` | 新盘/换盘自动 resyncmonitorLocalDisksAndHeal 10s 轮询 + healFreshDisk + healingTracker |
| `cmd/erasure-healing.go` / `erasure-healing-common.go` | 对象级 heal 核心(~800 行)、listAndHeal |
| `cmd/data-scanner.go` | scanner 循环(globalLeaderLock 集群单例)+ folderScanner + applyActions |
| `cmd/erasure.go`nsScanner/ `erasure-server-pool.go` | NSScanner 三层结构 |
| `cmd/bucket-lifecycle.go` | ILM 执行器(expiry/transition worker 池) |
| `cmd/xl-storage.go` | DiskInfo.Healing、CheckParts/VerifyFile、CleanAbandonedData、RenameData healing 分支 |
| `cmd/prepare-storage.go` | waitForFormatErasure 新盘启动握手 |
### 1.3 架构级差异(设计取舍,非缺陷)
1. **heal 队列模型**MinIO 所有 healscanner 抽样/MRF/admin/新盘 resync)汇入单 channel + 固定 worker 池(新盘 resync 另有 per-drive worker 池);RustFS 是优先级堆 + 去重合并 + 容量分级丢弃 + per-set bulkhead + 前台压力门控的多策略调度器(`manager.rs:3003-3420`)。RustFS 表达力更强,代价是"重复请求被合并"的可观测性问题(v1 已指出,现有 `HealAdmissionReceipt` canonical task_id + alias 机制回应了它,`manager.rs:1759-1846`)。
2. **scanner 远端盘访问**MinIO leader 通过磁盘抽象层透明读写远端节点磁盘;RustFS leader 通过 remote_scanner RPC 把扫描执行下放到远端 peer 本地进行(`crates/scanner/src/remote_scanner.rs`),只回传结果与进度心跳。两者都是集群单 leader。RustFS 方案省 leader↔远端的元数据读放大,代价是需要维护独立 RPC 协议(HMAC 逐帧认证、会话重放缓存、fence 复验,`remote_scanner.rs:52-61,405-496,1024-1065`)。
3. **heal 状态持久化**MinIO 用单文件 `.healing.bin`msgp healingTrackerdiskID 不匹配即重置);RustFS 用 schema 化多文件(resume/checkpoint/intent/seal/proof 各自 CAS 发布,`resume.rs:38-61`),崩溃窗口显式补齐(`erasure_healer.rs:389-402``resume.rs:1027-1057`)。
4. **写路径自保护**MinIO 写入后靠后台 heal 收敛;RustFS 在 PutObject/CompleteMultipartUpload 提交 rename 后主动检查 `convergence.needs_heal()` 并立即入队对象 heal`set_disk/ops/object.rs:2291-2306``ops/multipart.rs:2574-2589`),另有读修复 read repair`io_primitives.rs:1040-1160`)。
---
## 2. Heal 已实现功能全景
### 2.1 任务类型(`HealType``crates/heal/src/heal/task.rs:85-111`
| 类型 | 语义 | 执行体 | 生产触发方 |
|---|---|---|---|
| `Cluster` | 所有 bucket 依次 heal(结构 + 可选递归对象),批内重试 ≤3 | `heal_cluster` task.rs:1420-1490 | channelbucket 为空即 Clusterchannel.rs:576-577 |
| `Object{bucket,object,version_id}` | 单对象/版本;不存在时按 `recreate_missing` 重建或报错 | `heal_object` task.rs:855-1146 | admin、scanner、read-repair、写路径收敛、add_partial |
| `Bucket{bucket}` | 桶元数据/结构;`recursive` 再遍历全部对象版本 | `heal_bucket` task.rs:1284-1418 + `heal_bucket_objects` task.rs:1508-1698 | adminPOST /v3/heal/{bucket})、scanner `build_bucket_heal_request` |
| `Prefix{bucket,prefix}` | 按前缀递归 | `heal_prefix` task.rs:1492-1506 | channel`recursive && prefix` 非空(channel.rs:578-585 |
| `ErasureSet{buckets,set_disk_id}` | format 修复 + healing 标记 + 逐桶预处理 + 可恢复逐版本深扫 | `heal_erasure_set` task.rs:2158-2642 | adminpool/set 参数)、auto disk scanner、unclean shutdown、renew_disk、durable replacement 恢复 |
| `Metadata{bucket,object}` | 仅元数据(Deep、不重建数据) | `heal_metadata` task.rs:1700-1859 | **无生产触发方**(§6 HS-01 |
| `MRF{meta_path}` | 失败路径驱动的 Deep 修复(recursive+update_parity | `heal_mrf` task.rs:1861-1992 | **无生产触发方**(仅 `HealEvent` 可生成,未接线) |
| `ECDecode{bucket,object,version_id}` | EC 解码重建(Deep+recreate+update_parity),Urgent 优先级 | `heal_ec_decode` task.rs:1994-2156 | **无生产触发方**(仅 `HealEvent` 可生成,未接线) |
优先级 `Low/Normal/High/Urgent`task.rs:168-179);状态机 `Pending/Running/Retrying/Completed/Failed/Cancelled/Timeout`task.rs:225-241)。
### 2.2 触发路径全景(admin 之外)
| 通道 | source | 优先级 | 证据 |
|---|---|---|---|
| Scanner 周期抽样(1/1024`RUSTFS_HEAL_OBJECT_SELECT_PROB` | Scanner | Low | `scanner_folder.rs:2117-2136``:1150``remove_corrupted=HEAL_DELETE_DANGLING(true)``recreate_missing=false``common/heal_channel.rs:24``scanner_folder.rs:510-511` |
| Scanner 元数据损坏(get_size 失败分类 HealMetadata | Scanner | High | `scanner_folder.rs:2147-2208``:1244-1260` |
| Scanner abandoned children(缓存有、盘上无,list_path_raw quorum 核查) | Scanner | High(桶级+对象级) | `scanner_folder.rs:2528-2792` |
| Scanner pending-heal 账本重试(heal 通道满被拒后持久化,每桶每轮 ≤128 条、上限 10k) | Scanner | 原优先级 | `scanner_folder.rs:1721-1763``:99-100` |
| auto disk scannerunformatted 盘经 replacement_readiness 确认 / `runtime_state=="returning"` 盘 / durable intent 重入) | AutoHeal | Low | `manager.rs:2464-2999` |
| unclean shutdown 恢复(启动读 `unclean-shutdown` 标记 → 全部本地 set ErasureSet heal | AutoHeal | Low | `manager.rs:1362-1695` |
| 写路径收敛(PutObject/CompleteMultipartUpload 后 `convergence.needs_heal()` | Internal | Normal | `set_disk/ops/object.rs:2291-2306``ops/multipart.rs:2574-2589` |
| 部分对象 healadd_partial | Internal | Normal | `set_disk/ops/object.rs:5808-5825` |
| 旧数据目录清理残留 enqueue | Internal | Normal | `set_disk/core/io_primitives.rs:3880-3907` |
| 读修复(metadata_read_error / missing_shards / decode_errorTTL 去重缓存) | ReadRepair | Low | `set_disk/read.rs:407,995,1079``submit_read_repair_heal``io_primitives.rs:1105-1160`),`recreate_missing=true` |
| 盘重连遇 UnformattedDisk → send_heal_disk | AutoHeal | Normal | `set_disk/ops/locking.rs:339-347` |
| Admin API(含集群 coordinator 路由) | Admin | High | `rustfs/src/admin/handlers/heal.rs:174-212``:771-930` |
| 集群 RPC healpeer 调用) | — | — | `rustfs/src/storage/rpc/node_service/heal.rs``ecstore/src/cluster/rpc/peer_s3_client.rs:296,1209` |
注意:MinIO 的 MRF 通道(读路径检出 part 缺失/损坏即时投递 + 队列持久化 + shutdown 回放,`cmd/mrf.go``erasure-object.go:395-410,800-812`)在 RustFS 由 read-repair + 写路径收敛**部分替代**;`HealType::MRF`/`ECDecode`/`Metadata` 三个执行体没有生产入口(详见 §6 HS-01)。
### 2.3 对象级 heal 语义(ecstore `set_disk/ops/heal.rs`
流程(`heal_object_with_explicit_version_regen` :426 起):
1. 取对象写锁(除非 `no_lock`);`object``/` 结尾走对象目录 heal`heal_object_dir_locked` :1587-1717dangling 判定 + `remove` 删除 + 缺 volume 重建)。
2. `read_all_fileinfo` 全盘读 xl.meta,全部 not-found 视为已删除返回。
3. **quorum 仲裁 + ETag 兜底**(已亲验):`list_online_disks` 以 mod-time quorum 为准;quorum 失效时回退 ETag 多数派仲裁(`:525-567` `filter_by_etag`/`quorum_etag`);`pick_valid_fileinfo` 选 canonical 元数据;"meta 坏盘数 > parity" 的 cannotHeal 判定在 ETag 全盘一致时豁免(`:679`)。与 MinIO `filterDisksByETag` 双仲裁一致。
4. `disks_with_all_parts`:562-572)按 `scan_mode` 校验 part**Normal 仅 statCheckParts 语义),Deep 做全量 bitrot 校验(VerifyFile 语义)**Normal 扫描检出 `FileCorrupt` 自动升级 Deep 重试一次(`:2022-2031`,与 MinIO erasure-healing.go:1101-1106 同型);无 parity 对象(EC:0)bitrot 失败判不可恢复(`:700-726`)。
5. `should_heal_object_on_disk`:606-650)逐盘分类 missing/corrupt/offline/outdated → 重建:per-part bitrot reader/writer(用 per-part checksum + 算法)、写临时卷后 rename 提交(`HEAL_RENAME_INCOMPLETE` 重试语义 :24);dangling 删除安全检查 `dangling_delete_safety`:1488);**孤儿数据目录回收 `reclaim_orphan_data_dirs_best_effort`(:1428**——这部分覆盖了 MinIO `CleanAbandonedData` 的主场景(但无独立 `CheckAbandonedParts` API,见 §6 HS-02)。
6. 版本化对象:枚举"每个版本"(`storage.rs:1494-1530`);delete-marker 路径由 `latest_meta.deleted` 决定(`storage.rs:262-277` 注释);回归测试 `tests/heal_b5_versioned_regression_test.rs:282,334`
7. 显式版本重建 `try_regenerate_explicit_version_meta`:1318);transitioned 对象本地残留清理。
8. 写入路径另有 shard 级 bitrot 自校验 `verify_written_bitrot_shards``ops/bitrot_self_verify.rs:45-129`HighwayHash256S,最终 rename 前校验刚写出的 shard,服务 EC:0 无 parity 场景)——**注意这不是后台 bitrot 巡检**;后台巡检由 scanner bitrot_cycle 驱动 Deep heal 承担。
heal crate 侧包装(`task.rs:855-1146`):存在性检查(瞬时错误转 `TransientSkip` 不误判失败 :551-569);scanner 合成目录规范化(:1148-1180);`recreate_missing` 重建(:1183-1282);data-usage-cache 对象锁超时豁免(:571-653);not-found → treated_as_deleted 成功(:1012-1029);结果 `HealResultItem` 保留至多 1024 条 + truncated 标志(:50,845-852)。
递归遍历(`heal_bucket_objects` task.rs:1508-1698):分页枚举全部版本含 delete marker、瞬时错误指数退避重试 ≤32^n + 抖动 :620-627)、失败样本日志截断 ≤5 条、聚合 `BatchHealFailure`
### 2.4 erasure set heal 与断点续扫
`heal_erasure_set`task.rs:2158-2642)四阶段(4 步进度跟踪):
1. **替换意图与恢复盘选择**(仅 AutoHeal + heal_endpoints 非空):复用 durable intent 所在盘 / 排除目标端点选幸存盘;已完成代(CleanupPending)幂等收尾。
2. **格式修复**`heal_replacement_format(dry_run, pool, set, targets)``storage.rs:1372-1384`trait 默认实现 fail-closed);逐目标盘结果必须全 ok(`erasure_healer.rs:97-102`+ 身份围栏复核(task.rs:2410-2420)。
3. **healing 标记**:对目标盘写 owner CAS 标记 `{set_disk_id}:{task_id}``mod.rs:80-229`CAS + 回滚 + 并发唯一 owner),使 `DiskInfo.healing` 为真(已亲验赋值链 `set_disk/mod.rs:4988`)。
4. **逐桶预处理 + 可恢复深扫**`ErasureSetHealer::heal_erasure_set``erasure_healer.rs:242-278`)。
`ErasureSetHealer` 扫描细节(对标 MinIO `healErasureSet``heal_walk.rs:15-23` 模块注释明确引用 MinIO `global-heal.go` 的 listPathRaw + objQuorum=1 + mergeXLV2Versions):
- **枚举器选择(backlog#920**Deep 或 AutoHeal → per-set **disk-walk 并集枚举** `list_versions_for_heal_page_disk_walk`"任意盘上存在"即 sub-quorum 可重建;`storage.rs:1559-1644`,页界 1000 对象/10,000 版本,`dw1:` cursor);普通请求走 read-quorum `list_object_versions`
- **续扫游标**:权威 cursor 为 opaque continuation token`v1:`=marker JSON、`dw1:`=disk-walk key,两命名空间互斥防误读,`storage.rs:81-260`);每完成一页先持久化 cursor 再清 dedup 集合(`erasure_healer.rs:922-927`)。
- **页内并发**FuturesUnordered + Semaphore,默认 `RUSTFS_HEAL_PAGE_OBJECT_CONCURRENCY=8`Deep/AutoHeal 强制 1`erasure_healer.rs:105-142`)。
- **per-version dedup**`compose_key` 长度前缀注入编码(`resume.rs:281-288`)。
- **错误分类**:真缺席(FileNotFound 等)→ Absent(计成功);基础设施瞬时(quorum/DiskNotFound/SlowDown 等)→ Transient(计 skipped);其余 Failed`erasure_healer.rs:148-182`,注释引 backlog#856/#799 B7:离线盘不得记 healed/absent)。
- **防死循环**:空页 truncated 或页尾版本身份不前进即中止(:933-949)。
- **完成判定**failed/skipped/failed_buckets 任一 >0 不标记完成,`schedule_retry()` 复位 resume+checkpoint 两层(:561-626backlog#855/B6/#1033skip 轮不得标记完成)。
- **替换盘提交证据**:目标端点物理回读 `replacement_targets_have_version``ops/heal.rs:340-412`),未确认 → transient skip。
### 2.5 换盘自动修复(replacement recovery
- **识别**`replacement_readiness.rs:25-73`):`replacement_mount_lease_root()` 存在、canonicalize 成功、是挂载点、物理设备 id 非空、与根设备不相交、不与兄弟盘共享物理设备(Linux 用 /proc/self/mountinfo mount-id+dev+ino)。非 root 挂载检查有回归测试(`manager.rs:3549`)。
- **状态机**`resume.rs:63-73`):`Intent → Rebuilding →(写 proofVerified → CleanupPending → 清理``Abandoned` 终态;跨状态迁移先写持久层再变更(`save_state_strict`)。
- **持久化**`resume.rs:38-61`schema ResumeState=5/Checkpoint=5/proof=1):`{task_id}_ahm_resume_state.json``_ahm_checkpoint.json``buckets/ahm-replacement/` 命名空间下 intent/seal/completion_prooftorn write + 无 seal 可识别并原子重建(:1316-1338);CAS 发布、拒绝覆盖并发有效 proof:1512-1585)。
- **恢复**unclean shutdown 与周期扫描都从幸存盘恢复未完成/待清理替换代(`manager.rs:1435-1640,2663-2815`);多代冲突/校验失败 → 冻结该 set(`replacement_recovery_blocked_sets``manager.rs:69-87,2782-2815`)。
- **对外快照**`current_replacement_recovery_snapshot``lib.rs:262-333`)合并本地幸存盘记录,冲突 → Unknown/非 definitiveadmin `GET /v4/heal/replacement-recovery`
### 2.6 调度器(manager.rs
- 优先级堆 + 同优先级 FIFO:148-191,330-347);dedup key 按类型(:469-506);入队三态查重 active→queued→retrying:1759-1785);重复默认 Merged 并返回 canonical task_id`HealAdmissionReceipt`:1821-1846+ client token alias:1219-1246)。
- 容量:队列满时 best-effort 来源(Scanner/AutoHeal/ReadRepair)或低优先级被 Dropped(QueueFull)Admin/Internal 可驱逐低优先级排队项(`push_displacing_lower_priority` :353-396);80%/95% 压力分级(:885-909)。
- 并发:全局 `max_concurrent_heals`(默认 4+ per-set bulkhead `max_concurrent_per_set`(默认 1)(:3040-3073,3434-3447)。
- 前台压力门控 mainline throttle:前台读/写 permit 利用率 ≥80% 时延迟 best-effort 任务(:919-1009,2999-3020)。
- 超时:任务级聚合超时(默认 300s),跨重试保留剩余预算(task.rs:444-451PR #6101)。
- 可恢复重试:`is_recoverable_heal()`error.rs:83-136)≤3 次、2^n 退避封顶 30sretry 在独立 backoff task 中持有所有权(:3235-3382)。
- 完成态保留 10 分钟供查询(:42)。
### 2.7 Admin API 与集群协调
- 路由(`rustfs/src/admin/handlers/heal.rs:174-212`):`POST /rustfs/admin/v3/heal/``/heal/{bucket}``/heal/{bucket}/{prefix}`(同一 POST 按 query `clientToken/forceStart/forceStop` 区分 start/query/cancel,与 mc admin heal 语义对齐);`POST /v3/background-heal/status``GET /v4/heal/replacement-recovery`。权限 `HealAdminAction`route_policy.rs:334-341)。
- 集群协调(heal.rs:771-930 + `node_service.rs:514-606`):`heal_topology_fingerprint` + 按拓扑确定性选 coordinator 节点 + coordinator epochenvelope 校验 + SHA256 digest 重放缓防重放;coordinator 非本机走 peer gRPC `heal_control``probe_heal_control` 能力探测(滚动升级场景)。
- 请求:body 为 `HealOpts``recursive/dryRun/remove/recreate/scanMode(0/1/2)/updateParity/nolock/pool/set`serde camelCase,与 madmin.HealOpts 字段对齐);根 heal start 需 `recursive=true``pool+set` 成对;body 上限 1MB。
- 响应:`HealStartSuccess{clientToken, clientAddress, startTime}``HealTaskStatus{summary, detail, startTime, settings, items, truncated, progress}`summary ∈ running/finished/stopped/notFound);`BackgroundHealStatus`(bitrot 起始时间/周期/当前模式 + `disabled/uninitialized/idle/active/degraded` 状态——peer 不可达显式 degraded 不冒充 idleissue #5850 + `healOperations` 按优先级×来源矩阵 + 集群进度)。
- `HealResultItem`/`HealDriveInfo`/`HealItemType`/DriveState 枚举与 madmin JSON 兼容(`crates/madmin/src/heal_commands.rs:19-65`)。
- 状态 payload 超 8MiB 对折截断(channel.rs:37,73-104);path-token 校验(错误 token 拒绝,空 path 仅匹配 Cluster)。
### 2.8 heal 指标与日志
指标:`rustfs_heal_admission_total{source,result,reason,context}``rustfs_heal_task_start_total``rustfs_heal_task_running{type,set}``rustfs_heal_queue_delay_seconds``rustfs_heal_scheduler_skip_total``rustfs_heal_mainline_throttle_total``rustfs_heal_page_concurrency_current{set}``rustfs_heal_candidate_enqueue/merge/drop/priority_reject_total``rustfs_heal_read_repair_dedup_total{reason}` 等。日志全部结构化 event stylePR #5720);per-object 日志降级防风暴(`demote_to_debug_when!`#5716/#5719/#5727)。
---
## 3. Scanner 已实现功能全景
### 3.1 循环、leader、立即触发
- **集群单 leader**:分布式 ns 写锁 `leader.lock``scanner.rs:3156-3207`,超时默认 5s+ **持久化 leader-epoch CAS 围栏**leader 用 ETag 前置条件向 `.bloomcycle.bin``RSCYC001` 编码的 (cycle, leader_epoch)`scanner.rs:118,1850-1861,2177-2334`);usage 快照再打 epoch fence:2087-2153)。锁丢失 → 取消当前周期,30s 收敛(:108-111,2623-2642)。
- 抢锁后立即执行一轮;周期 = `RUSTFS_SCANNER_CYCLE` > config cycle > start_delay > 部署默认 > 速度档位(±10% 抖动、下限 1s)。
- **clean-idle 指数退避**:连续完整无脏周期间隔 ×2(封顶 24h;bitrot 周期压缩上限;桶有 lifecycle/replication 活动规则禁用,:383-456,1382-1512)。
- **superseded/deferred 退避**:5s 起指数退避封顶 30min:105-106,3432-3438);维护探测失败独立退避(:459-505)。
- **立即唤醒**:① dirty-usage 快路径——写路径 put/delete/multipart/bucket 操作调用 `record_dirty_usage_bucket``scanner_io.rs:222-235`;调用点 `rustfs/src/app/object_usecase.rs:6221` 等),自增 generation 并 Notify 唤醒 leader,脏桶优先排队(`scanner_io.rs:462-488`);② 维护配置变更(lifecycle/replication 设置时 `record_scanner_maintenance_change`);③ 运行时配置热更 generation+Notify;④ 集群活动快照变化。
- **集群协调**`probe_scanner_activity` 汇集本机+peer 的 `ScannerNodeActivity`instance_id/namespace_generation/maintenance_generation/protocol_version/topology_digest/data_movement_active/dirty usage),拓扑摘要覆盖 pools/sets/drives URL,协议版本不齐拒绝共享缓存锁(`scanner.rs:970-1068`);**数据迁移(rebalance/decommission)期间推迟周期**`scanner_io.rs:2226-2374`);周期结束逐 peer RPC 确认 dirty-usage ack`scanner.rs:2925-2952`)。
### 3.2 遍历模型
- 主遍历是**全量目录 walk**tokio::fs::read_dir 递归,`scanner_folder.rs:1915-2234`),不走 metacachemetacache/`list_path_raw` 仅用于 abandoned children 跨盘核查(:2528-2792)。
- 三级并发:leader → per-set(信号量默认 4)→ per-disk 桶扫描(默认 4)→ 单盘递归;每桶每 set 缓存锁 `.scanner-cycle.lock.pool-N.set-M`(锁丢失取消该桶扫描,锁竞争重排队);每盘单扫描准入(本地盘也走信号量,`scanner_io.rs:3246-3274`)。
- 桶顺序:shuffle 后按 dirty → 未缓存 → 已缓存重排(`scanner_io.rs:2947-2949,462-488`);目录内按名字排序 + resume 提示旋转(`scanner_folder.rs:333-359`)。
- **断点续扫**`DataUsageScanCheckpoint{version,resume_after,reason}` 持久于缓存 info`data_usage_define.rs:68,293-307`);预算耗尽/取消写入,恢复有 Used/Stale/NoHint 指标;续扫单位是目录(无跨周期对象级分页)。
- erasure 语义:发现 `xl.meta` 即对象边界不下钻;UUID data-dir 候选最多探测 64 entry;有数据无元数据 → 记 failed + 高优 healsymlink 目录忽略/环跳过。
- 协作让出:每 N 对象(默认 128)`yield_now`
### 3.3 大桶跳过策略(对标 MinIO compaction
1. 缓存当前性复用:桶与扫描计划未变(name/source/snapshot_complete/plan digest/next_cycle/leader_epoch/cache_key_format 全匹配)整桶跳过(`scanner_io.rs:1062-1109`)。
2. compacted 目录 16 周期轮换窗口:`hash mod (next_cycle, 16)` 命中才重扫,否则从旧缓存拷贝(`scanner_folder.rs:74,2429-2442`)。
3. compaction 阈值:子项 <500 或纯对象叶子压缩为单 entry;子文件夹 ≥2500(根 10000)预压缩;children ≥10000 归约(:75-78,2314-2340,2846-2887)。
4. 失败对象 TTL 跳过:86400s/最多 10000 条(:88-91,1354-1381)。
与 MinIO master 对比:MinIO 的跳过策略同样是 hash-mod-16 周期 + compaction 阈值树(500/10000/2500),**bloom filter 已从 master 删除**。RustFS 的常量与结构与 MinIO 现状同源(MinIO 未采用跨盘 dirty-generation 优先,RustFS 额外多两层跳过——plan digest 与缓存当前性校验)。
### 3.4 data usage 统计
- 维度:每目录 entrysize/objects/versions/delete_markers/大小直方图/版本直方图/复制统计/failed_objects/per-tier stats/children/compacted`data-usage/src/data_usage.rs:661-679`);每对象 SizeSummary(含 per-ARN 复制目标统计、tier 统计,tier 分类:transitioned 完成记入其 tier 否则按 storage classfree version 不计);桶级 `BucketUsageInfo`;集群级 `DataUsageInfo`(含 scanner_cycle/scanner_epoch 围栏 + usage_snapshot_complete)。
- 存储:每桶每 set `{bucket}/.usage-cache.bin`(主 + `.bkp` 备份 + CAS 重试);权威集群快照 `buckets/data-usage/data-usage.json`(每 10 周期同步 `.bkp`,legacy 路径兼容);陈旧快照拒绝写入(epoch/cycle/last_update 三重判定);被竞争 superseded 的观测快照另存 `data-usage-observed.json`
- 消费:`replace_bucket_usage_memory_from_info` 刷新桶用量内存 + 两层缓存失效(`scanner.rs:4142-4152`)→ bucket stats/quota/admin account_info/system;写路径内存实时叠加 overlay;启动读快照判断冷缓存跳过启动延迟。
- 未完成 multipart 不参与统计(与 MinIO 一致,MinIO 也不扫 multipart 桶)。
### 3.5 ILM 集成
- 每对象 `ScannerItem::apply_actions``scanner_folder.rs:747-1032`):`Evaluator::new(lifecycle).with_lock_retention(...).with_replication_config(...).eval()` 批量评估。
- 已实现动作(IlmAction 全集,`common/src/metrics.rs:34-45`):expiry 删除(Delete/DeleteRestored/DeleteRestoredVersion)、全版本删除(DeleteAllVersions/DelMarkerDeleteAllVersions,处理后停止后续版本)、transitionTransition/TransitionVersiontier 列表运行时读取)、noncurrent 批量(DeleteVersionAction → `enqueue_by_newer_noncurrent`)、free-version 清理(`enqueue_free_version`)、object-lock retention 约束。**与 MinIO 的 9 个 ILM 动作一一对应**。
- 执行模型:scanner 是"发现与入队"角色(expiry 队列/transition 队列在 ecstore `bucket_lifecycle_ops.rs`),动作由 worker 池消费——与 MinIO globalExpiryState/globalTransitionState 同型。
- AbortIncompleteMultipartUpload 不在 scanner/ILM 内执行(MinIO 同样不在:`internal/bucket/lifecycle/rule.go` 有 FIXME,实际由 `erasureSets.cleanupStaleUploads` 全局例程承担);RustFS 由 ecstore 独立后台任务 `init_background_stale_multipart_upload_cleanup``bucket_lifecycle_ops.rs:3289-3320`+ 桶删除时 on-demand。
- 集成测试覆盖:transition+restore、free-version、noncurrent、delete-marker、0-day、后台扫描过期(`scanner/tests/lifecycle_integration_test.rs:1071-2095`)。
### 3.6 heal 候选生产(scanner 侧)
- 抽样:`hash mod_alt(next_cycle/prob_div, 1024/prob_div)`,进入 compacted 分支重扫时 prob_div=16 等效概率 ×16(与 MinIO 同款补偿,`scanner_folder.rs:125-127,2117-2122`)。
- deep/normal:周期级 `get_cycle_scan_mode`bitrot_cycle 默认 30d`scanner.rs:1626-1657`)→ 对象级带 `HealScanMode::Deep`;新鲜对象(60s 内修改)降级 Normal:146-155);状态持久 `.background-heal.json``BackgroundHealInfo{bitrot_start_time,bitrot_start_cycle,current_scan_mode}`,与 MinIO 同路径同结构)。
- scanner 只入队不内联执行(内联 heal 已移除,兼容旗标仅告警,`scanner_folder.rs:411-427`);`HealScanMode::Deep` 只是标记,bitrot 校验读发生在 heal 消费端(ecstore Deep 路径)。
- 元数据损坏 → 高优 heal`classify_get_size_failure` → HealMetadata);abandoned children → list_path_raw quorum 核查 + 桶级/对象级高优 heal;healing 盘粘性跳过(`should_heal` :1628-1648)。
- pending-heal 账本:heal 通道满被拒持久化到缓存 info,下轮重试。
- 复制 heal`queue_replication_heal` → replication 队列(走 replication 通道而非 heal channel);per-ARN 复制用量统计。
### 3.7 remote_scanner RPC 协议(RustFS 特有)
请求 ≤16KB msgpackversion/request_id/server_epoch/session_id/session_sequence/bucket/next_cycle/leader_epoch/scan_plan_digest/skip_healing/scan_mode/budget);帧 ≤2MB、HMAC-SHA256 逐帧认证(域 `rustfs-ns-scanner-frame-v3`);进度心跳 1s(预算模式 250ms);阶段播报 Scanning→PersistingRPC 生命周期上限 24h、断连宽限 2min;防重放 session+sequence 缓存(容量 65536);服务端校验 leader fence 与持久化 cycle 一致 + 每 5s fence 复验;结果 Complete/Partial/NamespaceNotFound/CycleAhead;不支持 v4 协议的远端盘回退 leader 本地扫描(`remote_scanner.rs` 全文件;`scanner_io.rs:2750-2812`)。
### 3.8 限速/预算/热更/观测
- DynamicSleeper 比例退避(速度档 fastest/fast/default/slow/slowest,同 MinIO 五档参数);idle_mode 总闸;前台 S3 读流量每请求 10ms 封顶 250ms 额外退避。
- 周期预算 ScannerCycleBudgetmax_duration/max_objects/max_directories(默认 0=不限),partial 周期仍推进 cycle 计数。
- runtime_config 三层来源(env > config > default)逐字段来源标记(Env/Config/ScannerCompatConfig/Default),admin `PUT /v3/config` 热更 → generation+Notify 即时生效;`GET /v3/scanner/status` 返回 enabled/freshness(fresh/stale/unknown)/metrics/cycle_schedule/runtime_config`GET /v3/ilm/expiry/status` 返回 expiry 队列/worker/missed/blocked。
- 指标:leader lock、周期 complete/partial/deferred/superseded、versions scanned、per-sourceUsage/Lifecycle/BucketReplication/SiteReplication/Heal/Bitrot/Alertschecked/executed/queued/missed、checkpoint set/used/stale、当前路径(per-disk+bucket 实时)、缓存 save 系列、并发系列、告警(excess versions/version size/folders)。
---
## 4. 与 MinIO 逐项对标
### 4.1 heal 触发通道对照
| MinIO 通道 | RustFS 对应 | 状态 |
|---|---|---|
| A. 手动 admin healhealSequenceclientToken/forceStart/forceStop | heal channel Start/Query/Cancel + 集群 coordinator + envelope 重放防护 | ✅ 等价且增强(集群路由);序列语义差异见 §6 HS-06 |
| B. 常驻后台 heal 队列(newBgHealSequence + healRoutine worker 池) | HealManager 常驻调度器 + 优先级队列 + bulkhead | ✅ 等价且增强 |
| C. 新盘/换盘自动 resyncmonitorLocalDisksAndHeal 10s + healFreshDisk + healingTracker + waitForFormatErasure 握手) | auto disk scanner10s+ replacement_readiness + durable intent/proof 状态机 + heal_replacement_format | ✅ 等价且增强(identity fence + completion proofMinIO 的 tracker 面向对外可见性更强,见 §6 HS-07) |
| D. MRF(队列 100k + 持久化 list.bin + shutdown 回放 + 读路径 corrupt 投递) | read-repairLow+TTL 去重)+ 写路径 convergence heal 部分承担;`HealType::MRF` 执行体无生产入口 | ⚠️ 部分等价(§6 HS-01) |
| E. Scanner 抽样 heal1/1024 + compacted ×16 补偿)+ abandoned children | 同款抽样 + ×16 补偿 + abandoned children + pending-heal 账本 | ✅ 等价且增强(账本) |
| F. 读路径内联触发 → MRFGetObject part 缺失/损坏、元数据重建 missingBlocks>0 | read repairmissing_shards/decode_error/metadata_read_error 三入口) | ✅ 等价(入 heal 队列而非 MRF 队列) |
### 4.2 对象级 heal 语义对照
| 特性 | MinIO | RustFS | 状态 |
|---|---|---|---|
| mod-time quorum 仲裁 | listOnlineDisks | 同 | ✅ |
| ETag 多数派兜底(时钟漂移) | filterDisksByETag | `filter_by_etag`/`quorum_etag`heal.rs:525-567 | ✅ 已亲验 |
| cannotHeal 的 ETag 豁免 | ETag 全一致豁免重试 | heal.rs:679 | ✅ |
| Normal=CheckPartsstat/ Deep=VerifyFilebitrot | 是 | `disks_with_all_parts` 按 scan_modeops/heal.rs:562-572,978-1024 | ✅ |
| Normal 检出 corrupt 自动升 Deep 重试一次 | erasure-healing.go:1101-1106 | ops/heal.rs:2022-2031 | ✅ |
| dangling 判定(not-found > parity+ 删除审计 | isObjectDangling/deleteIfDangling | `dangling_delete_safety`:1488+ scanner HEAL_DELETE_DANGLING | ✅(审计 tags 细节有差异) |
| 孤儿 data-dir/inline 清理(CleanAbandonedData | CheckAbandonedPartsscanner 抽中 + admin Remove 时显式调用) | heal 路径内 `reclaim_orphan_data_dirs_best_effort`:1428);独立 API 三层 NotImplemented | ⚠️ 部分等价(§6 HS-02 |
| 版本化/delete-marker heal | HealObject versionIDnullVersionID 特判 | 逐版本枚举 + delete-marker latest healB5 回归) | ✅ |
| 对象级 healing 元数据标记(x-minio-healingRenameData 跳过版本清理) | 有 | 无对象级标记;依赖盘级 healing.bin + NSLock + rename 语义 | ⚠️ 评估项(§6 HS-12) |
| Distribution/Index 一致性三处防线 | 有(manual modification 拒绝) | 目标盘格式结果全 ok 校验 + 身份围栏 | ✅(粒度不同) |
| 无 parityEC:0)对象 | bitrot 不可恢复处理 | 判不可恢复(:700-726)+ 写入自校验 | ✅ 增强(写路径自校验) |
| 三层分布不一致拒绝 heal | 有 | heal_walk 归一化 + 页界防御 | ✅(实现方式不同) |
| multipart 孤儿对账 | CheckAbandonedParts 承担 | 显式 NotImplemented(由 lifecycle 清理承担) | ⚠️ §6 HS-02 |
| suspended/decommissioned pool 处理 | IsSuspended 跳过 | deferral 语义(store/heal.rs:192-207PR #5876 | ✅ |
| heal 与并发删除互斥 | NSLock + healing 标记 | NSLock + 写锁 | ✅ |
### 4.3 新盘 resync 对照
| MinIO | RustFS | 状态 |
|---|---|---|
| waitForFormatErasure 四类可恢复错误无限等待握手 | startup 盘解析 + renew_disk 重连路径 | ✅(模型不同:RustFS 不在启动时阻塞等待 format) |
| HealFormat NSLock + errNoHealRequired + refFormat 不一致拒绝 | `heal_format`/`heal_replacement_format` fail-closed + 目标槽位限定(PR #1787 语义) | ✅ 增强 |
| per (pool,set) 分布式锁防并发 resync | set 级队列去重 + bulkheadmanager.rs:2854-2889 | ✅ |
| 全新集群检测(待 heal 盘数==总盘数不触发) | replacement_readiness(独立挂载点/物理设备校验,非 root) | ✅ 增强 |
| healingTracker.healing.binBytes/Items 计数、QueuedBuckets/HealedBuckets、Resume 快照、RetryAttempts ≤4、HealID 联动、diskID 变更重置) | resume/checkpoint schema 化持久层 + durable intent/proofper-task 文件,CAS) | ✅ 等价且增强(崩溃窗口补齐);但**对外快照可见性**弱于 MinIO(§6 HS-07 |
| 跳过 heal 开始后新写入版本(ModTime > Started | 无同款过滤 | ⚠️ §6 HS-13 |
| 跳过 ILM 已过期版本(filterLifecycle | 无同款过滤 | ⚠️ §6 HS-13 |
| worker 数 max(GOMAXPROCS,NR)/4 下限 4heal:drive_workers 覆盖 | 页内并发 8Deep/AutoHeal 强制 1+ per-set bulkhead | ✅(参数模型不同) |
| 每 entry waitForLowHTTPReq 让路 | mainline throttle(前台利用率门控) | ✅ 增强 |
| heal 范围含 `.minio.sys/config``.minio.sys/buckets` 两个伪桶;最新桶优先 | ErasureSet 任务逐 bucket 预处理(含 meta bucket 语义由 heal_bucket 承担) | ✅(顺序无"最新优先" |
| 失败整体重试 ≤4 次(resetHealing + errRetryHealing | schedule_retry 复位双层 + 可恢复重试 ≤3 | ✅ |
### 4.4 scanner 对照
| MinIO | RustFS | 状态 |
|---|---|---|
| 集群单 leaderglobalLeaderLock | leader.lock + 持久化 leader-epoch CAS 围栏 | ✅ 增强(epoch 围栏防脑裂,MinIO 无持久化 epoch |
| `.bloomcycle.bin` 只存 cyclebloom 已删除) | 同路径存 cycle+leader_epochRSCYC001 | ✅ 对齐(v1 误判已修正) |
| folderScanner hash-mod-16 + compaction500/10000/2500 | 同款常量 + plan digest + 缓存当前性校验 + dirty 优先 | ✅ 增强 |
| 每盘扫描并行 ≤GOMAXPROCShealing 盘排除 | per-set/per-disk 信号量 + healing 盘粘性跳过 | ✅ |
| scannerSleeperfactor 2/max 1sspeed 档热更) | DynamicSleeper 同款 + idle_mode + 前台读退避 | ✅ 增强 |
| idle 语义:`scanner:idle_speed=on`(空闲时段才节流,忙时全速) | `RUSTFS_SCANNER_IDLE_MODE=true`(启用限速总闸) | ⚠️ 语义方向相反,§6 HS-14 |
| applyActions 顺序(heal→ILM→复制→告警) | apply_actions 同序(heal 候选→ILM→复制 heal→告警) | ✅ |
| ILM 9 动作 + 批量评估 + DeletePrefixObject 优化 | 同 9 动作 + 批量评估 + expiry 队列 | ✅(DeleteAllVersions 是否单调用优化未逐行核) |
| abandoned childrenlistPathRaw minDisks=N/2 发现漏写盘) | list_path_raw + quorum 核查 + 高优 heal | ✅ |
| incomplete multipart 独立例程(6h 间隔/24h 过期,rename 进 .trash | ecstore 独立后台任务(可配间隔/过期) | ✅(trash 二段清理细节差异,§6 HS-18) |
| usage 维度(size/objects/versions/DM/直方图/复制/tier/bucket 级) | 全覆盖 + 集群快照三重防回退 | ✅ 增强 |
| prefix 级 usageloadPrefixUsageFromBackendconsole 消费) | 缓存内有目录树但仅 flatten 桶级 | ❌ §6 HS-08 |
| 超限事件 s3:ObjectManyVersions/LargeVersions/PrefixManyFolders + 审计 | 仅指标 alert_excess_*(默认 100/1TiB/65538 vs MinIO 100/1TB/50000 | ⚠️ §6 HS-04/HS-17 |
| scanner 指标 v3bucket_scans/directories/objects/versions/last_activity | rustfs_scanner_* 全套 + freshness | ✅(命名体系不同) |
| TraceScanner / realtime metricsmc admin scanner status/trace | 无 trace 通道;/v3/scanner/status 自有结构 | ⚠️ §6 HS-03 |
### 4.5 admin/CLI/API 面对照
| MinIO | RustFS | 状态 |
|---|---|---|
| `POST /minio/admin/v3/heal/...` start/status/cancel | `POST /rustfs/admin/v3/heal/...` 同三态 | ✅(路径前缀不同属预期) |
| `HealStartSuccess`/`HealTaskStatus`/`HealResultItem`/DriveState | 同名字段 JSON 兼容 | ✅ |
| `POST /v3/background-heal/status`BgHealState 聚合) | 同路径 + degraded 语义 + operations 矩阵 | ✅ 增强(MRF per-endpoint 子状态无,因无 MRF |
| `GET /v3/healthinfo` 每 drive `HealInfo *HealingDisk` | 无同款 healthinfo heal 字段(replacement-recovery v4 承担部分) | ⚠️ §6 HS-07 |
| madmin 客户端 HealStart/HealStatus/BackgroundHealStatus/ScannerStatus 方法 | 仅 wire 类型,无客户端方法 | ❌ §6 HS-05 |
| mc admin heal --pool/--set、--scan-mode、--force-start/stop | HealOpts 全字段支持(pool/set/scanMode/forceStart/forceStop | ✅(服务端就绪;缺 mc 侧入口,HS-05) |
| ErrHealAlreadyRunning / ErrHealOverlappingPaths 类型化错误 | 去重合并 + 驱逐语义;无类型化重叠拒绝 | ⚠️ §6 HS-06 |
| 结果 backpressuremaxUnconsumedItems=1000、10s 保活流式、24h 未消费 abort) | 快照式查询(1024 条 + 8MiB 截断 + 10min 保留) | ⚠️ §6 HS-06 |
| `mc support inspect`/healing-bin 离线 dump | 无(inspect.rs 存在但 healing dump 未确认) | ⚠️ P3 |
### 4.6 观测面对照
| 维度 | MinIO | RustFS | 状态 |
|---|---|---|---|
| heal 指标 | minio_heal_objects_total/heal_total/errors_total/time_last_activity + v3 drive_health 2=healing | rustfs_heal_* 全套(admission/queue delay/running/throttle/page concurrency | ✅(RustFS 缺 drive_health=healing 单一 gauge 等价物;DiskInfo.healing 已赋值) |
| scanner 指标 | v3 6 个 + realtime 18 项 | rustfs_scanner_* 全套 + per-source 维度 | ✅ |
| ILM 指标 | v3 5 个(expiry/transition pending/active/missed + versions_scanned | ilm expiry status API + scanner per-source | ✅(指标与 API 形态不同) |
| trace | TraceHealing/TraceScanner 两通道 | 无 | ❌ §6 HS-03 |
| 审计 | HealObject 事件、dangling 删除审计、scanner:manyversions 等 | 结构化日志(event style+ 指标;无 audit log 事件 | ⚠️ §6 HS-04 |
| 进度 | healingTracker Bytes/Items/QueuedBuckets/当前对象 + usage-cache 总量基线 | HealProgress{scanned/healed/failed/bytes/current_object/percentage}bytes_processed 注释为 0、estimated_completion_time 恒 None | ⚠️ §6 HS-07 |
### 4.7 配置面对照(默认值)
| MinIO | RustFS | 备注 |
|---|---|---|
| `heal:bitrotscan`(默认 offon=每轮;Nm=N×30×24h | `heal.bitrot_cycle` / `RUSTFS_SCANNER_BITROT_CYCLE_SECS`(默认 30d=2592000s0/on=每轮 Deepoff=禁用) | ✅ 同语义(RustFS 默认 30dMinIO 默认 off——**默认值不同**RustFS 更激进) |
| `heal:max_io=100`/`max_sleep=250ms`waitForLowIO | mainline throttle 阈值 80%/80%、max_sleep 250ms | ✅ 同型(阈值模型不同) |
| `heal:drive_workers`(默认 -1 自动) | 页内并发 8 + per-set 1 | ✅ 同型 |
| `_MINIO_HEAL_WORKERS`GOMAXPROCS/2 | `RUSTFS_HEAL_MAX_CONCURRENT_HEALS=4` + `_MAX_CONCURRENT_PER_SET=1` | ✅ |
| `_MINIO_AUTO_DRIVE_HEALING`on | `RUSTFS_HEAL_AUTO_HEAL_ENABLE=true` | ✅ |
| `_MINIO_SCANNER`on | `RUSTFS_SCANNER_ENABLED=true` | ✅ |
| `scanner:speed` 五档(default=2x/1s/1m | 同五档同名同参数 | ✅ |
| `scanner:idle_speed`on | `RUSTFS_SCANNER_IDLE_MODE`(true | ⚠️ 语义方向(HS-14 |
| `scanner:alert_excess_versions=100` | 100 | ✅ |
| `scanner:alert_excess_folders=50000` | 65538(兼容 PBS 布局) | ⚠️ HS-17 |
| `ilm:expiration_workers=100`/`transition_workers=100` | ecstore expiry/transition worker 池(键见 ilm 子系统) | ✅(默认值未逐项核对) |
| `api:stale_upload_cleanup_interval=6h`/`expiry=24h` | ecstore 后台任务 env 可配 | ✅(默认值未逐项核对) |
| —(无) | `RUSTFS_HEAL_QUEUE_SIZE=10000``_TASK_TIMEOUT_SECS=300``_INTERVAL_SECS=10``_LOW_PRIORITY_MERGE/DROP``_PAGE_*``_SET_BULKHEAD``_MAINLINE_*``RUSTFS_SCANNER_CYCLE_MAX_*` 预算、`_MAX_CONCURRENT_SET/DISK_SCANS=4``_YIELD_EVERY_N_OBJECTS=128` 等 | RustFS 特有(更细粒度) |
### 4.8 RustFS 超出 MinIO 的部分
1. remote_scanner RPC(扫描执行下放远端 peer 本地,含 HMAC 认证/重放缓存/fence 复验/断连宽限)。
2. 持久化 leader-epoch CAS 围栏 + usage 快照 epoch/cycle 防回退(MinIO 仅锁,无持久 epoch)。
3. 周期预算(max_duration/objects/directories+ partial 周期推进语义。
4. per-set/per-disk 扫描并发闸 + 每桶每 set 缓存锁。
5. pending-heal 账本(heal 通道满不丢候选)。
6. 换盘 durable intent + completion proof 状态机 + 身份围栏(MinIO healingTracker 无 proof)。
7. mainline throttle 前台压力门控(permit 利用率驱动)。
8. 集群 heal control coordinator + envelope 重放防护 + degraded 显式降级。
9. 写路径 shard bitrot 自校验(EC:0 场景)。
10. dirty-usage 快路径唤醒(写路径即时通知 + 脏桶优先)。
11. heal 运行时可观测矩阵(优先级×来源 operations snapshot)。
12. workload admission 联动(heal 调度器读前台压力快照)。
---
## 5. 差距与改进清单
分级定义:P1=行为/运维对齐缺口(影响生产运维或工具链兼容);P2=完善性(功能在但缺一角);P3=清理/低风险。每项含现状证据、MinIO 行为、影响、建议、验收方式。
### P18 项)
**HS-01 MRF/ECDecode/Metadata 三类 heal 任务无生产触发入口,HealEvent 未接线**
- 现状:`HealType::MRF/ECDecode/Metadata` 执行体完整(task.rs:1700-2156)但全仓库无生产触发方;`HealEvent`/`HealEventHandler`event.rs:50-367crate 外零引用(已亲验 grep);channel 转换只产生 Cluster/Object/Bucket/Prefix/ErasureSetchannel.rs:566-601)。
- MinIOmrf.go 独立 MRF 队列(容量 100k,满丢弃计数)、进程退出 msgp 持久化 `.heal/mrf/list.bin` + 启动回放、入队 <1s 延迟 1s(等网络恢复)、healSleeper 限速;读路径 GetObject part 缺失/损坏、元数据重建 missingBlocks>0、Put 部分成功、DeleteObject、multipart、peer client 共 7+ 投递点。
- 影响:RustFS 的 read-repair + 写路径收敛覆盖了主场景,但缺少:① 事件驱动的 Urgent ECDecode 重建入口(ecstore 解码失败时目前仅 Low read-repair);② Metadata-only heal 入口(scanner HealMetadata 分类存在但走普通对象 heal);③ MRF 队列持久化(重启丢未消费修复意图——scanner pending-heal 账本部分缓解)。
- 建议:三选一决策——(a) 接线 HealEvent(在 ecstore 解码失败/metadata 损坏点发事件)+ 实现持久化重试账本;(b) 删除 MRF/ECDecode/Metadata 死代码只保留文档说明;(c) 保留执行体、把 HealEvent 降级为内部 API。推荐 (a) 但需先量化 read-repair 是否已覆盖解码失败场景的响应时间要求。
- 验收:解码失败 → Urgent heal 请求链路 e2e;重启后 pending 修复意图回放;HealEvent 环形缓冲指标。
**HS-02 CheckAbandonedParts 三层 NotImplementedabandoned data 独立对账入口缺失)**
- 现状:`set_disk/ops/heal.rs:2052-2056``core/sets.rs:1144-1148``store/heal.rs:258-266` 三层显式 `Err(NotImplemented)`(已亲验),注释"intentionally retained above the set layer until there is a concrete caller"。
- MinIO`CheckAbandonedParts` → 每盘 `CleanAbandonedData`:读 xl.meta → 列 UUID data-dir + inline entries → 与 getDataDirs 差集 → 删多余 data-dir/inline 并重写 xl.meta;由 scanner 抽中 heal 与 admin heal Remove 时显式调用。
- 影响:RustFS heal 路径内 `reclaim_orphan_data_dirs_best_effort`:1428)覆盖"heal 时回收孤儿目录",但 ① 无独立触发点(MinIO 在对象未到 heal 阈值时也能清 abandoned data);② inline data 孤儿条目清理未确认;③ multipart 孤儿对账明确不做(设计决定,由 lifecycle 承担)。
- 建议:评估把 `reclaim_orphan_data_dirs_best_effort` 提升为 heal_object 固定步骤(若尚非)+ 实现 HealOperations::check_abandoned_parts 真实现(调用同一回收逻辑),或明确文档化"由 lifecycle 承担"并关闭 API 面。
- 验收:构造 data-dir/inline 孤儿 → scanner 抽样/admin heal 后被清理;三层 API 返回成功或显式 NotSupported 文档化。
**HS-03 heal/scanner trace 通道缺失**
- 现状:TraceHealing/TraceScanner 零命中(已亲验 grep 全仓库)。
- MinIO`madmin.TraceHealing`mc admin trace --healingFuncName=heal.Bucket/heal.Object/heal.CheckAbandonedParts,带 dry/remove/mode/version-id/disks/bytes)、`TraceScanner`mc admin scanner trace,支持 --filter-size/--response-duration)。
- 影响:无法实时观测单个 heal/scanner 动作的耗时与参数;排障只能靠指标聚合与日志。
- 建议:在 heal channel 执行与 scanner folder/item 处理埋点,接入现有 admin trace 订阅面(若 rustfs 已有 trace 基建则复用,无则按 madmin TraceType 扩展)。
- 验收:mc 等价工具能订阅 heal/scanner trace 流。
**HS-04 scanner 超限 S3 事件与审计缺失**
- 现状:仅 `rustfs_scanner_excess_*_total` 指标(versions 100/version size 1TiB/folders 65538)。
- MinIO:发 `s3:ObjectManyVersions`>100 版本)、`s3:ObjectLargeVersions`(累计 >1TB)、`s3:PrefixManyFolders`>50000 子目录)事件(UserAgent: Scanner+ scanner:manyversions/largeversions/manyprefixes 审计。
- 影响:依赖事件订阅做容量治理的用户(console/外部审计)收不到告警。
- 建议:scanner_folder 告警点接入 notify 事件发布(复用 lifecycle 事件通道语义)。
- 验收:配置桶通知后超限对象触发事件。
**HS-05 madmin 客户端方法缺失**
- 现状:`crates/madmin/src/heal_commands.rs` 只有 wire 类型(HealDriveInfo/Infos/HealResultItem);无 HealStart/HealStatus/BackgroundHealStatus/ScannerStatus 客户端方法。
- MinIOmadmin-go 提供完整客户端;mc admin heal/scanner/status/trace 都建立在上面。
- 影响:mc 等管理工具无法直接对接 RustFS heal/scanner 管理面;自动化运维只能手写 HTTP。
- 建议:按 madmin-go 接口形状补客户端(服务端已就绪,纯客户端工作)。
- 验收:用 madmin 客户端完成 start→query→cancel 全流程。
**HS-06 admin heal 序列语义与 MinIO 差异**
- 现状:重复/重叠请求被去重合并(返回 canonical task_id)或驱逐;无 ErrHealAlreadyRunning/ErrHealOverlappingPaths 类型化错误(已亲验:manager.rs:1309 的 already_running 是幂等启动保护,非 admin 语义);结果为快照式查询(1024 条/8MiB 截断/10min 保留),非 MinIO 的流式增量(clientToken 拉增量 + maxUnconsumedItems=1000 backpressure + 10s 保活 + 24h 未消费 abort)。
- 影响:mc admin heal 的交互模型(长连接拉增量)对 RustFS 表现为多次快照轮询;自动化脚本难以区分"已合并"与"新启动"。
- 建议:① 增量语义:channel query 支持自上次 clientToken 起的 items 增量(或 cursor);② 重叠请求返回类型化错误码(或 receipt 中显式 merged_into 字段——现有 alias 机制已有基础);③ forceStart 先停旧再启新语义核对。
- 验收:madmin 兼容客户端按 MinIO 模式轮询能取得全量 items。
**HS-07 healing 进度与盘级 healing 状态对外可见性不足**
- 现状:bytes 恢复进度 `progress.bytes_processed = 0 // set to 0 for now`erasure_healer.rs:967);`HealProgress::estimated_completion_time` 恒 None、`HealStatistics::add_healed_objects` 未写入(progress.rs:38,135-139 零调用);healthinfo 无每盘 HealInfo 等价(MinIO HealingDiskBytesDone/Failed/Skipped、ObjectsTotal 基线、QueuedBuckets/HealedBuckets、Resume 快照、当前 object);v3 指标无 drive_health=2(healing) 单一 gauge 等价。
- 影响:换盘重建(可能数小时~天)期间运维无法回答"进行到哪/还剩多少/预计何时完成"。
- 建议:① erasure set heal 统计 bytesheal_object 返回对象大小已可得);② 从 usage-cache 读对象总量基线(MinIO 同款做法);③ admin healthinfo/背景状态暴露每盘 healing 快照(DiskInfo.healing 已有,补聚合暴露);④ ETA 由基线+速率推导。
- 验收:换盘重建中 admin 可见 bytes 进度与 ETAmc info 等价输出 Healing 标志。
**HS-08 prefix 级 usage 未暴露**
- 现状:DataUsageCache 内目录树 entry 存在(hash_path 组织),但 `dui()` 只 flatten 到桶名(data_usage_define.rs:858-915)。
- MinIO`loadPrefixUsageFromBackend`30s cache)从每 set `.usage-cache.bin` 聚合 prefix usageconsole 桶前缀统计消费。
- 影响:console/前端无法展示前缀级用量;大桶定位"哪个前缀占空间"无 API。
- 建议:实现 flatten 前缀查询 API(数据已在缓存内,纯聚合与暴露工作)。
- 验收:ListBuckets/PrefixUsage API 返回与前缀过滤匹配的统计。
### P29 项)
**HS-09 get_disk_status 恒返回 Ok(唯一 TODO**`crates/heal/src/heal/storage.rs:930-943`(已亲验)。当前无生产调用方(低风险)。建议:删除该方法或接 ecstore disk 状态真实现(DiskStatus 枚举已定义)。
**HS-10 HealStorageAPI 约 1/3 方法为死代码**get_object_meta/get_object_data/put_object_data/delete_object/verify_object_integrity/ec_decode_rebuild/get_disk_status/format_disk/heal_bucket_metadata/get_object_size/get_object_checksum/list_objects_for_heal(非分页版,自带 memory_heavy 警告)均 0 调用方。建议:随 HS-01 决策一并清理或接线(死接口误导后续维护者以为存在调用路径)。
**HS-11 bitrot 自检缺失**MinIO 启动时 bitrotSelfTest 对四算法已知向量自检失败即 Fatal(防静默数据损坏)。RustFS 无等价(已亲验 grep)。建议:启动时对 HighwayHash256S 等在用算法做已知向量自检(低成本高价值)。
**HS-12 对象级 healing 元数据标记评估**MinIO heal 期间对象打 `x-minio-healing:true`RenameData 据此跳过版本清理/legacy purge(漏掉会导致 heal 与并发删除互毁)。RustFS 无对象级标记(已亲验 grep object.rs 无 healing 分支),依赖 NSLock + rename 语义。建议:审计 RustFS rename 提交路径是否存在"heal 提交与并发 delete/version 清理竞争"窗口;若无则文档化差异,若有则补标记等价机制。
**HS-13 erasure set heal 无"跳过新写入/ILM 已过期版本"过滤**MinIO resync 跳过 ModTime>tracker.Started 的版本(避免 heal 追新写入尾巴)与 ILM 已过期版本(避免白做)。RustFS erasure_healer 未实现同款过滤(按版本 dedup 有,时间/ILM 过滤无)。影响:重建尾部长尾(持续写入的桶 heal 完成判定被新版本推迟)与无效 heal 工作量。建议:disk-walk 枚举处加 started_at 时间过滤 + evaluator 预检。
**HS-14 scanner idle 语义方向与 MinIO 相反**MinIO `scanner:idle_speed=on`(默认)= 集群空闲时才节流、忙时全速;RustFS `RUSTFS_SCANNER_IDLE_MODE=true`(默认)= 限速总闸(false=完全不休眠)。两者默认行为可能相近(都限速)但参数语义不可互换,迁移文档需显式说明;若追求 mc config 兼容需重命名/重语义。建议:先文档化差异,评估是否对齐语义。
**HS-15 alert_excess_folders 默认值差异**RustFS 65538(兼容 PBS/Proxmox 布局,scanner_folder.rs:79vs MinIO 50000。行为差异默认即触发阈值不同。建议:文档化(保留 65538 有本地理由)。
**HS-16 单机默认周期钩子未启用**:`single_disk_default_cycle_secs(_features) -> None` 恒空(scanner.rs:1428-1430),单机部署无专属默认周期覆盖。建议:决定单机默认周期策略后启用或删除钩子。
**HS-17 DeleteAllVersions 批量优化核对**MinIO 用 DeletePrefix+DeletePrefixObject 单调用代替逐版本 fan-out。RustFS expiry 队列路径是否同款优化未逐行核实(集成测试覆盖行为正确性)。建议:核对 `apply_expiry_rule` 全版本删除路径,若无前缀单调用优化则评估补齐。
### P33 项)
**HS-18 trash/临时目录二段清理细节核对**:MinIO `.minio.sys/tmp/.trash` 清理(delete_cleanup_interval 默认 5m + deleteCleanupSleeper)与 stale uploads rename-into-trash 二段式。RustFS 有 delete_tail_activity.rs 与 stale multipart 任务,二段语义是否完整对齐未逐行核实。建议:对照补齐或文档化。
**HS-19 root heal 直连死路径清理**`should_handle_root_heal_directly` 恒 falseadmin/handlers/heal.rs:1200-1202,测试锁定),store.heal_format 直连分支不可达。建议:删除死分支或恢复直连路径作为集群协调失败的降级。
**HS-20 兼容旗标与死指标清理**:`RUSTFS_SCANNER_INLINE_HEAL_ENABLE`(开启仅告警)+ `rustfs_scanner_inline_heal_total` 死指标 + `rustfs_common::metrics` 中 scanner 域代码分层迁移(backlog #1843 已登记)。建议:随分层迁移一并清理。
### 按设计不追平(7 项,记录以防后续误判为缺口)
1. **bloom filter**MinIO master 已删除;RustFS `.bloomcycle.bin` 复用为 cycle/epoch 围栏与 MinIO 现状一致。
2. **scanner 集群单 leader**:双方一致;RustFS 额外有 epoch 围栏。
3. **heal 不发 S3 bucket notification**:双方一致(heal 结果走 admin status)。
4. **incomplete multipart 不在 scanner/ILM 内执行**:双方一致(独立后台例程)。
5. **内联 heal 移除**RustFS 有意为之(scanner 只入队),MinIO 的 applyHealing 内联路径不做对标。
6. **heal 序列常驻保活(10s 空白回写)**:RustFS 快照式查询模型不同,按 HS-06 处理增量语义即可,不复制流式保活。
7. **`.trash`/`tmp-old` 路径名兼容**:RustFS 布局常量独立,不逐字对齐 MinIO 路径。
---
## 6. 配置默认值总表(RustFS
healenv 前缀 `RUSTFS_HEAL_``crates/config/src/constants/heal.rs`,消费于 `manager.rs:724-800`):
| 配置 | 默认 | 热更新 |
|---|---|---|
| AUTO_HEAL_ENABLE | true | 否 |
| QUEUE_SIZE | 10000 | 否 |
| INTERVAL_SECS | 10 | 否(启动时固定) |
| TASK_TIMEOUT_SECS | 300 | 否 |
| MAX_CONCURRENT_HEALS | 4 | 否 |
| MAX_CONCURRENT_PER_SET | 1(≤min(全局,值) | 否 |
| LOW_PRIORITY_MERGE_ENABLE | true | 否 |
| LOW_PRIORITY_DROP_WHEN_FULL | true | 否 |
| PAGE_OBJECT_CONCURRENCY | 8Deep/AutoHeal 强制 1 | 否 |
| EVENT_DRIVEN_SCHEDULER_ENABLE | true | 否 |
| SET_BULKHEAD_ENABLE | true | 否 |
| PAGE_PARALLEL_ENABLE | true | 否 |
| MAINLINE_THROTTLE_ENABLE | true | 否 |
| MAINLINE_READ/WRITE_UTILIZATION_HIGH_PERCENT | 80/80 | 否 |
| MAINLINE_MAX_SLEEP_MS | 250 | 否 |
| (总开关)RUSTFS_HEAL_ENABLED | true | 否 |
| admin 子系统 heal.bitrot_cycle | 30d | 是(经 scanner runtime config |
scanneradmin 子系统 `scanner``crates/config/src/constants/scanner.rs` + `ecstore/src/config/scanner.rs` + `runtime_config.rs:527-673`):
| 键 | env | 默认 |
|---|---|---|
| speed | RUSTFS_SCANNER_SPEED | default2x/1s/60s |
| delay / max_wait / cycle / start_delay | RUSTFS_SCANNER_* | 派生/空 |
| cycle_max_duration/objects/directories | …_MAX_* | 0(不限) |
| bitrot_cycle | …_BITROT_CYCLE_SECS | 259200030d0/on=每轮,off=禁用) |
| idle_mode | …_IDLE_MODE | true |
| cache_save_timeout | …_CACHE_SAVE_TIMEOUT_SECS | 30s |
| max_concurrent_set_scans / disk_scans | …_MAX_CONCURRENT_* | 4/4 |
| yield_every_n_objects | …_YIELD_EVERY_N_OBJECTS | 128 |
| alert_excess_versions / version_size / folders | …_ALERT_* | 100 / 1TiB / 65538 |
scanner 内部 env`RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=16``RUSTFS_HEAL_OBJECT_SELECT_PROB=1024``RUSTFS_SCANNER_DEEP_VERIFY_COOLDOWN_SECS=60``RUSTFS_DATA_USAGE_FAILED_OBJECT_TTL_SECS=86400`/`_MAX=10000``RUSTFS_LOCK_ACQUIRE_TIMEOUT=5s``RUSTFS_SCANNER_ENABLED=true``RUSTFS_SCANNER_INLINE_HEAL_ENABLE=false`(兼容告警)。
全部 17 个 scanner 键支持 env > config 双通道 + admin PUT 热更(generation+Notify 即时生效);heal 运行时参数目前仅 env(无 admin 热更入口,`Arc<RwLock<HealConfig>>` 结构已预留)。
---
## 7. 相关 backlog / 历史索引
- 换盘自动修复系列(已闭环):backlog #1786(冗余假绿算法)、#1787(目标槽位限定)、#1789resume 与 healing marker 绑定 replacement 实例)、#1791(黑白盒验收矩阵)。
- #801 DiskInfo.healing 从未赋值(已修复闭环,现 `set_disk/mod.rs:4988` 有赋值链)。
- #1651 Scanner 指标节点/source/bucket-drive 维度(OPEN,本分析 §3.8/§4.6 相关)。
- #1843 crates/common 83% scanner/heal 域代码分层迁移(OPEN,含 HS-20)。
- 代码注释引用的历史缺陷(现已有防护与回归测试):#856/#799 B7(离线盘误记 healed)、#855/B6/#1033skip 不得标记完成)、#920sub-quorum 并集枚举)、#856 B5(按版本续扫)、#5173bitrot trailing bytes)、#5029(回归节点 stale 版本合并)。
- v1 对标文档:`docs/rustfs-heal-scanner-vs-minio-parity-assessment.md`(本文取代)、落地手册 `docs/rustfs-heal-scanner-vs-minio-improvement-playbook.md`(部分条目已被后续实现超越)。
- 换盘深度分析:`docs/new-disk-replacement-and-healing-deep-analysis-zh.md``docs/node-disk-identity-and-healing-analysis-zh.md`
## 8. 审计方法与局限
- 四路并行审计(heal crate 逐文件、scanner crate 逐文件、ecstore 集成层 wiring、MinIO master 源码研究)+ 主会话对关键"缺失"结论逐条亲验(get_disk_status TODO、HealEvent 零外部引用、.bloomcycle.bin 无 bloom 实现、check_abandoned_parts 三层 NotImplemented、ETag 兜底已实现、trace 通道零命中、already_running 语义)。
- 未逐行核实的点(已在文中标注"未确认/未逐行核"):DeleteAllVersions 前缀单调用优化(HS-17)、trash 二段清理细节(HS-18)、ilm worker 默认值对照、stale multipart 默认值对照、mc CLI flag 逐字拼写(MinIO 侧)。其中 HS-17 与 HS-18 已于 2026-08-19 完成逐行核实,结论见 §9.2/§9.3。
- MinIO 侧引用以其 master `7aac2a2c5b` 为准;RustFS 侧行号以 2026-08-16 工作区为准,后续演进请以符号名检索为准。
## 9. 落地结果(2026-08-19 更新)
本审计衍生的 14 个子 issuebacklog #1865~#1878)已全部闭环。本节为差距清单 HS-01~HS-20 的最终处置记录,也是下一轮对标重审的增量基线。
### 9.1 已落地(PR 均已合并 main)
- HS-01 MRF 接线 + 持久化修复账本(#1865PR #6189):决策选 (a)。common MRF channelbounded 8192、try_send 永不阻塞)+ heal mrf_queue100k 条 / 8MiB 双限环形)+ `buckets/.heal/mrf/journal.bin` CRC 持久化回放(torn tail 截断、回放后删除)+ 三投递点(read decode_error→Urgent ECDecode、scanner 元数据损坏→High Metadata、add_partial→Normal+ `RUSTFS_HEAL_MRF_ENABLE` 一键回退。
- HS-02 abandoned parts/data-dir 对账(#1866PR #6179):接通 abandoned 检查入口,保留 dry-run / reclaim 计数。
- HS-03 heal/scanner trace 通道(#1867PR #6179):进程内 trace bus + `/v3/trace` admin 流式订阅 + heal task / abandoned-parts / scanner folder / ILM / heal-candidate trace producer。
- HS-04 scanner 超限 S3 事件(#1868PR #6176):`s3:Scanner:ManyVersions/LargeVersions/BigPrefix` 三事件 + 24h 边沿冷却;HS-15 阈值差异文档化(`docs/operations/scanner-excess-alerts.md`)。
- HS-05 madmin 客户端一期(#1869PR #6166):SigV4 admin 客户端 heal/scanner 方法;增量消费方法待 follow-up(协议已由 HS-06 并入)。
- HS-06 admin heal 增量语义与类型化重叠(#1870PR #6206):`sinceSeq/nextSeq/minSeq` 增量游标(wire additive、缺省=全量快照)+ `RUSTFS_HEAL_OVERLAP_POLICY`(默认 merge 不变;minio_error 下 AlreadyRunning/OverlappingPaths 类型化拒绝)+ forceStart 先停旧再启新。
- HS-07 healing 进度可见性(#1871PR #6179):data-usage 总量基线 + baseline/current/healed 计数。
- HS-08 prefix usage#1872PR #6171):`GET /v3/usage/{bucket}`
- HS-11 bitrot 启动自检(#1873PR #6165)。
- HS-13 heal 跳过过滤(#1875PR #6179):过滤命中版本不再计为失败。
- HS-16 单机周期钩子(#1878PR #6250):删恒 None 钩子,决策记录见 `docs/operations/heal-scanner-parity-notes-zh.md`
- HS-09/10/19/20 死代码清理批(#1877PR #6256):净 911 行零行为变更;`get_disk_status` TODO(全仓库唯一产品 TODO)清零;HS-01 联动的 `ec_decode_rebuild`/`get_object_meta` 保留并加 Reserved 注释(MRF 当前经 `heal_object` 执行)。
### 9.2 核对后确认"已实现 / 非缺口"(审计期误判修正,累计四例)
- bloom filter(§0 已修正):MinIO master 已删除,双方现状一致。
- ETag 兜底仲裁(§0 已修正):RustFS 已有实现(`set_disk/ops/heal.rs`)。
- HS-17#18762026-08-19 逐行核实后关闭):DeleteAllVersions 前缀单调用优化 RustFS 已完整实现——`apply_expiry_on_non_transitioned_objects``delete_all()` 两 action 设 `delete_prefix + delete_prefix_object` 后单次 `delete_object``bucket_lifecycle_ops.rs:5047-5056`),SetDisks 分支一次写锁 + 一次全版本 quorum 读 + 内联逐版本 object-lock 检查(`set_disk/ops/object.rs:5566-5612`),与 MinIO `expire.go``applyExpiryOnNonTransitionedObjects` 逐行对齐。§8 原列"未逐行核实"的本项已有结论:现状即优化路径,无需实现。
- HS-14#1878PR #6250 附带核对):MinIO"idle=空闲才节流"是 2024-01 minio/minio#18734 之前的行为(`scannerIdleMode` 现为静态配置,`idle_speed=on` 默认即始终按速度档节流,"idle"命名是历史残留);RustFS `RUSTFS_SCANNER_IDLE_MODE` 与 MinIO 当前语义方向一致,且另有 MinIO 没有的前台读退避下限。真实迁移陷阱(变量须 `RUSTFS_` 前缀、`on/off` vs `true/false` 词表、`false` 连前台保护一起关)已文档化于 `docs/operations/heal-scanner-parity-notes-zh.md`
### 9.3 审计型结论(无需改代码)
- HS-12#1874PR #6183):不存在 MinIO 用 `x-minio-healing` 防御的那类竞争——所有同 (bucket, object) 提交面在同一把对象级 ns 写锁互斥,heal 锁 guard 覆盖 rename 提交全程;交付 2 个并发不变量回归测试 + `docs/operations/heal-concurrency-safety-notes-zh.md` 交点矩阵。
- HS-18#18782026-08-19 逐行核实):trash/tmp 三段清理全对齐——stale multipart 隔离-清理等价且更安全(`delete_all_with_quorum` 逐盘递归删即 `move_to_trash` rename 进 `.rustfs.sys/tmp/.trash`,另有锁 + fence)、trash 排空基本等价(无逐条 sleeper 节流,5m 周期天然限频)、tmp 非 trash 24h 回收等价(RustFS 5m 比 MinIO 6h 更及时);周期默认 24h/6h/5m 三项全对齐。§8 原列"未逐行核实"的本项已有结论。
### 9.4 移交 follow-up(汇总于 backlog#1862 评论区)
HS-01 bitrot GET→MRF 全链路 e2e、kill -9 journal 回放 e2e、队列满压测 RSS(≤ 预算+10%);HS-05/06 madmin 增量消费方法 + wire 单一来源化 + embedded e2e + 多轮轮询 soakHS-08 多盘 scanner 周期 e2eHS-04 超限审计条目;HS-18 低于 quorum 的 stale-multipart 崩溃残留窗口(扇出中途崩溃且已清盘数 > parity 时 FileNotFound 不在忽略集导致不自然收敛,修复需专用 quorum 变体)。
下一轮重审建议:跟随 heal/scanner 下一个大特性落地后触发,以本节为增量基线。
@@ -42,7 +42,6 @@ fn map_data_usage_result<E>(result: Result<DataUsageInfo, E>) -> S3Result<DataUs
result.map_err(|_| S3Error::with_message(S3ErrorCode::InternalError, DATA_USAGE_LOAD_ERROR_MESSAGE)) result.map_err(|_| S3Error::with_message(S3ErrorCode::InternalError, DATA_USAGE_LOAD_ERROR_MESSAGE))
} }
#[allow(dead_code)]
#[derive(Debug, Serialize, Default)] #[derive(Debug, Serialize, Default)]
#[serde(rename_all = "PascalCase", default)] #[serde(rename_all = "PascalCase", default)]
pub struct AccountInfo { pub struct AccountInfo {
-1
View File
@@ -394,7 +394,6 @@ impl Operation for ExportBucketMetadata {
#[derive(Debug, Default, Deserialize)] #[derive(Debug, Default, Deserialize)]
pub struct ImportBucketMetadataQuery { pub struct ImportBucketMetadataQuery {
#[allow(dead_code)]
pub bucket: String, pub bucket: String,
} }
+141 -13
View File
@@ -42,6 +42,7 @@ use s3s::{Body, S3Error, S3ErrorCode, S3Request, S3Response, S3Result, StdError,
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use std::collections::HashMap; use std::collections::HashMap;
use std::pin::Pin; use std::pin::Pin;
use std::sync::Arc;
use std::task::{Context, Poll}; use std::task::{Context, Poll};
use std::time::{Duration, SystemTime}; use std::time::{Duration, SystemTime};
use tokio::sync::{Semaphore, SemaphorePermit, mpsc}; use tokio::sync::{Semaphore, SemaphorePermit, mpsc};
@@ -248,13 +249,14 @@ struct LeaseHolderState {
acquired_at: SystemTime, acquired_at: SystemTime,
ttl_secs: u64, ttl_secs: u64,
holder_count: u32, holder_count: u32,
guard_ids: Option<Vec<u64>>,
} }
fn build_top_locks_response( fn build_top_locks_response(
limit: usize, limit: usize,
now: SystemTime, now: SystemTime,
lease_infos: Vec<LockLeaseInfo>, lease_infos: Vec<LockLeaseInfo>,
fast_infos: Vec<(rustfs_lock::ObjectLockInfo, u32)>, fast_infos: Vec<(rustfs_lock::ObjectLockInfo, u32, Option<Vec<u64>>)>,
) -> TopLocksResponse { ) -> TopLocksResponse {
let mut lease_holders = HashMap::with_capacity(lease_infos.len()); let mut lease_holders = HashMap::with_capacity(lease_infos.len());
@@ -277,17 +279,27 @@ fn build_top_locks_response(
} }
state.ttl_secs = state.ttl_secs.max(ttl_secs); state.ttl_secs = state.ttl_secs.max(ttl_secs);
state.holder_count = state.holder_count.saturating_add(1); state.holder_count = state.holder_count.saturating_add(1);
match (state.guard_ids.as_mut(), info.guard_id) {
(Some(guard_ids), Some(guard_id)) => guard_ids.push(guard_id),
_ => state.guard_ids = None,
}
}) })
.or_insert(LeaseHolderState { .or_insert(LeaseHolderState {
acquired_at: info.acquired_at, acquired_at: info.acquired_at,
ttl_secs, ttl_secs,
holder_count: 1, holder_count: 1,
guard_ids: info.guard_id.map(|guard_id| vec![guard_id]),
}); });
} }
for state in lease_holders.values_mut() {
if let Some(guard_ids) = &mut state.guard_ids {
guard_ids.sort_unstable();
}
}
let mut infos: Vec<_> = fast_infos let mut infos: Vec<_> = fast_infos
.into_iter() .into_iter()
.map(|(info, holder_count)| { .map(|(info, holder_count, guard_ids)| {
let mode = match info.mode { let mode = match info.mode {
LockMode::Shared => TopLockMode::Read, LockMode::Shared => TopLockMode::Read,
LockMode::Exclusive => TopLockMode::Write, LockMode::Exclusive => TopLockMode::Write,
@@ -298,11 +310,10 @@ fn build_top_locks_response(
owner: info.owner.to_string(), owner: info.owner.to_string(),
}; };
let priority = lock_priority_label(info.priority); let priority = lock_priority_label(info.priority);
// Shared-owner timestamps do not roll back when a newer sibling releases, so only their count is stable. // Match the complete holder cohort so replacements cannot reuse stale lease data.
let state = match lease_holders.remove(&key) { let state = match lease_holders.remove(&key) {
Some(lease) Some(lease)
if lease.holder_count == holder_count if lease.holder_count == holder_count && lease.guard_ids.is_some() && lease.guard_ids == guard_ids =>
&& (mode == TopLockMode::Read || info.acquired_at <= lease.acquired_at) =>
{ {
TopLockState { TopLockState {
acquired_at: lease.acquired_at, acquired_at: lease.acquired_at,
@@ -349,8 +360,11 @@ fn build_top_locks_response(
} }
} }
async fn collect_top_locks(limit: usize) -> TopLocksResponse { async fn collect_top_locks_with_clients(
let manager = get_global_lock_manager(); limit: usize,
manager: Arc<rustfs_lock::GlobalLockManager>,
clients: Vec<Arc<dyn rustfs_lock::client::LockClient>>,
) -> TopLocksResponse {
let Some(fast) = manager.as_fast_lock_manager() else { let Some(fast) = manager.as_fast_lock_manager() else {
return TopLocksResponse { return TopLocksResponse {
total: 0, total: 0,
@@ -362,21 +376,29 @@ async fn collect_top_locks(limit: usize) -> TopLocksResponse {
}; };
}; };
let lease_infos = if let Some(clients) = get_global_lock_clients() { let lease_infos = if clients.is_empty() {
join_all(clients.values().map(|client| client.list_lock_leases())) Vec::new()
} else {
join_all(clients.iter().map(|client| client.list_lock_leases()))
.await .await
.into_iter() .into_iter()
.flatten() .flatten()
.collect() .collect()
} else {
Vec::new()
}; };
// Capture holders last so released or replaced lease guards fail the merge checks. // Capture holders last so released or replaced lease guards fail the merge checks.
let fast_infos = fast.list_locks_with_holder_counts(); let fast_infos = fast.list_locks_with_holder_generations();
build_top_locks_response(limit, SystemTime::now(), lease_infos, fast_infos) build_top_locks_response(limit, SystemTime::now(), lease_infos, fast_infos)
} }
async fn collect_top_locks(limit: usize) -> TopLocksResponse {
let manager = get_global_lock_manager();
let clients = get_global_lock_clients()
.map(|clients| clients.values().cloned().collect())
.unwrap_or_default();
collect_top_locks_with_clients(limit, manager, clients).await
}
fn parse_top_locks_limit(uri: &Uri) -> usize { fn parse_top_locks_limit(uri: &Uri) -> usize {
query_value(uri, "count") query_value(uri, "count")
.and_then(|v| v.parse::<usize>().ok()) .and_then(|v| v.parse::<usize>().ok())
@@ -1229,6 +1251,7 @@ mod tests {
let mixed_resource = ObjectKey::new("bucket", "mixed-object"); let mixed_resource = ObjectKey::new("bucket", "mixed-object");
let replaced_resource = ObjectKey::new("bucket", "replaced-object"); let replaced_resource = ObjectKey::new("bucket", "replaced-object");
let remaining_shared_resource = ObjectKey::new("bucket", "remaining-shared-object"); let remaining_shared_resource = ObjectKey::new("bucket", "remaining-shared-object");
let opaque_resource = ObjectKey::new("bucket", "opaque-object");
let response = build_top_locks_response( let response = build_top_locks_response(
TOP_LOCKS_DEFAULT_LIMIT, TOP_LOCKS_DEFAULT_LIMIT,
@@ -1239,6 +1262,7 @@ mod tests {
lock_type: LockType::Shared, lock_type: LockType::Shared,
owner: "owner-a".to_string(), owner: "owner-a".to_string(),
acquired_at: now - Duration::from_secs(50), acquired_at: now - Duration::from_secs(50),
guard_id: Some(11),
remaining_ttl: Duration::from_secs(5), remaining_ttl: Duration::from_secs(5),
}, },
LockLeaseInfo { LockLeaseInfo {
@@ -1246,6 +1270,7 @@ mod tests {
lock_type: LockType::Shared, lock_type: LockType::Shared,
owner: "owner-a".to_string(), owner: "owner-a".to_string(),
acquired_at: now - Duration::from_secs(40), acquired_at: now - Duration::from_secs(40),
guard_id: Some(18),
remaining_ttl: Duration::from_secs(20), remaining_ttl: Duration::from_secs(20),
}, },
LockLeaseInfo { LockLeaseInfo {
@@ -1253,6 +1278,7 @@ mod tests {
lock_type: LockType::Shared, lock_type: LockType::Shared,
owner: "owner-c".to_string(), owner: "owner-c".to_string(),
acquired_at: now - Duration::from_secs(30), acquired_at: now - Duration::from_secs(30),
guard_id: Some(13),
remaining_ttl: Duration::from_secs(25), remaining_ttl: Duration::from_secs(25),
}, },
LockLeaseInfo { LockLeaseInfo {
@@ -1260,6 +1286,7 @@ mod tests {
lock_type: LockType::Exclusive, lock_type: LockType::Exclusive,
owner: "owner-d".to_string(), owner: "owner-d".to_string(),
acquired_at: now - Duration::from_secs(15), acquired_at: now - Duration::from_secs(15),
guard_id: Some(12),
remaining_ttl: Duration::from_secs(18), remaining_ttl: Duration::from_secs(18),
}, },
LockLeaseInfo { LockLeaseInfo {
@@ -1267,6 +1294,7 @@ mod tests {
lock_type: LockType::Exclusive, lock_type: LockType::Exclusive,
owner: "owner-e".to_string(), owner: "owner-e".to_string(),
acquired_at: now - Duration::from_secs(30), acquired_at: now - Duration::from_secs(30),
guard_id: Some(14),
remaining_ttl: Duration::from_secs(25), remaining_ttl: Duration::from_secs(25),
}, },
LockLeaseInfo { LockLeaseInfo {
@@ -1274,8 +1302,17 @@ mod tests {
lock_type: LockType::Shared, lock_type: LockType::Shared,
owner: "owner-f".to_string(), owner: "owner-f".to_string(),
acquired_at: now - Duration::from_secs(30), acquired_at: now - Duration::from_secs(30),
guard_id: Some(16),
remaining_ttl: Duration::from_secs(22), remaining_ttl: Duration::from_secs(22),
}, },
LockLeaseInfo {
resource: opaque_resource.clone(),
lock_type: LockType::Exclusive,
owner: "owner-g".to_string(),
acquired_at: now - Duration::from_secs(30),
guard_id: None,
remaining_ttl: Duration::from_secs(30),
},
], ],
vec![ vec![
( (
@@ -1288,6 +1325,7 @@ mod tests {
priority: rustfs_lock::fast_lock::LockPriority::Normal, priority: rustfs_lock::fast_lock::LockPriority::Normal,
}, },
1, 1,
Some(vec![15]),
), ),
( (
rustfs_lock::ObjectLockInfo { rustfs_lock::ObjectLockInfo {
@@ -1299,6 +1337,7 @@ mod tests {
priority: rustfs_lock::fast_lock::LockPriority::Normal, priority: rustfs_lock::fast_lock::LockPriority::Normal,
}, },
1, 1,
Some(vec![16]),
), ),
( (
rustfs_lock::ObjectLockInfo { rustfs_lock::ObjectLockInfo {
@@ -1310,6 +1349,7 @@ mod tests {
priority: rustfs_lock::fast_lock::LockPriority::Normal, priority: rustfs_lock::fast_lock::LockPriority::Normal,
}, },
1, 1,
Some(vec![12]),
), ),
( (
rustfs_lock::ObjectLockInfo { rustfs_lock::ObjectLockInfo {
@@ -1321,6 +1361,7 @@ mod tests {
priority: rustfs_lock::fast_lock::LockPriority::Normal, priority: rustfs_lock::fast_lock::LockPriority::Normal,
}, },
2, 2,
Some(vec![11, 18]),
), ),
( (
rustfs_lock::ObjectLockInfo { rustfs_lock::ObjectLockInfo {
@@ -1332,6 +1373,7 @@ mod tests {
priority: rustfs_lock::fast_lock::LockPriority::Normal, priority: rustfs_lock::fast_lock::LockPriority::Normal,
}, },
1, 1,
Some(vec![17]),
), ),
( (
rustfs_lock::ObjectLockInfo { rustfs_lock::ObjectLockInfo {
@@ -1343,11 +1385,24 @@ mod tests {
priority: rustfs_lock::fast_lock::LockPriority::Normal, priority: rustfs_lock::fast_lock::LockPriority::Normal,
}, },
2, 2,
None,
),
(
rustfs_lock::ObjectLockInfo {
key: opaque_resource,
mode: LockMode::Exclusive,
owner: "owner-g".into(),
acquired_at: now - Duration::from_secs(4),
expires_at: now + Duration::from_secs(6),
priority: rustfs_lock::fast_lock::LockPriority::Normal,
},
1,
None,
), ),
], ],
); );
assert_eq!(response.total, 6); assert_eq!(response.total, 7);
let leased = response let leased = response
.locks .locks
.iter() .iter()
@@ -1392,6 +1447,47 @@ mod tests {
.find(|entry| entry.object == "remaining-shared-object") .find(|entry| entry.object == "remaining-shared-object")
.expect("an older surviving shared lease should remain lease-backed"); .expect("an older surviving shared lease should remain lease-backed");
assert_eq!(remaining_shared.ttl_secs, 22); assert_eq!(remaining_shared.ttl_secs, 22);
let opaque = response
.locks
.iter()
.find(|entry| entry.object == "opaque-object")
.expect("generation-less holder should remain visible");
assert_eq!(opaque.ttl_secs, 6);
}
#[test]
fn top_locks_rejects_replaced_shared_generation() {
let now = SystemTime::UNIX_EPOCH + Duration::from_secs(100);
let resource = ObjectKey::new("bucket", "replaced-shared-object");
let response = build_top_locks_response(
TOP_LOCKS_DEFAULT_LIMIT,
now,
vec![LockLeaseInfo {
resource: resource.clone(),
lock_type: LockType::Shared,
owner: "owner-a".to_string(),
acquired_at: now - Duration::from_secs(30),
guard_id: Some(1),
remaining_ttl: Duration::from_secs(20),
}],
vec![(
rustfs_lock::ObjectLockInfo {
key: resource,
mode: LockMode::Shared,
owner: "owner-a".into(),
acquired_at: now - Duration::from_secs(2),
expires_at: now + Duration::from_secs(4),
priority: rustfs_lock::fast_lock::LockPriority::Normal,
},
1,
Some(vec![2]),
)],
);
let entry = response.locks.first().expect("replacement remains visible");
assert_eq!(entry.ttl_secs, 4);
assert_eq!(entry.elapsed_secs, 2);
} }
#[tokio::test] #[tokio::test]
@@ -1431,4 +1527,36 @@ mod tests {
drop(guard); drop(guard);
} }
#[tokio::test(start_paused = true)]
async fn collect_top_locks_uses_refreshed_local_lease() {
use rustfs_lock::{FastObjectLockManager, GlobalLockManager, LocalClient, LockClient, LockRequest};
let manager = Arc::new(GlobalLockManager::Enabled(Arc::new(FastObjectLockManager::new())));
let client = Arc::new(LocalClient::with_manager(manager.clone()));
let request = LockRequest::new(ObjectKey::new("diag-bucket", "renewed-object"), LockType::Exclusive, "diag-owner")
.with_ttl(Duration::from_secs(30));
let lock_id = request.lock_id.clone();
assert!(
client
.acquire_lock(&request)
.await
.expect("local lock acquisition should succeed")
.success
);
tokio::time::advance(Duration::from_secs(20)).await;
assert!(client.refresh(&lock_id).await.expect("local lease refresh should succeed"));
let clients: Vec<Arc<dyn rustfs_lock::client::LockClient>> = vec![client.clone()];
let response = collect_top_locks_with_clients(TOP_LOCKS_DEFAULT_LIMIT, manager, clients).await;
let entry = response
.locks
.iter()
.find(|entry| entry.bucket == "diag-bucket" && entry.object == "renewed-object")
.expect("refreshed local lock should be listed");
assert!(entry.ttl_secs >= 29, "collector must use the refreshed lease deadline");
assert!(client.release(&lock_id).await.expect("local lock release should succeed"));
}
} }
-6
View File
@@ -455,12 +455,6 @@ pub fn register_replication_route(r: &mut S3Router<AdminOperation>) -> std::io::
async fn validate_replication_admin_request(req: &S3Request<Body>, action: AdminAction) -> S3Result<Credentials> { async fn validate_replication_admin_request(req: &S3Request<Body>, action: AdminAction) -> S3Result<Credentials> {
authorize_admin_request(req, vec![Action::AdminAction(action)]).await authorize_admin_request(req, vec![Action::AdminAction(action)]).await
} }
#[allow(dead_code)]
fn is_local_host(_host: String) -> bool {
false
}
pub(crate) async fn cluster_replication_stats(bucket: &str, context: Option<Arc<AppContext>>) -> BucketStats { pub(crate) async fn cluster_replication_stats(bucket: &str, context: Option<Arc<AppContext>>) -> BucketStats {
let Some(stats) = current_replication_stats_handle_for_context(context.clone()) else { let Some(stats) = current_replication_stats_handle_for_context(context.clone()) else {
return BucketStats::default(); return BucketStats::default();
-2
View File
@@ -21,7 +21,6 @@ use matchit::Params;
use rustfs_madmin::service_commands::ServiceTraceOpts; use rustfs_madmin::service_commands::ServiceTraceOpts;
use s3s::{Body, S3Request, S3Response, S3Result, s3_error}; use s3s::{Body, S3Request, S3Response, S3Result, s3_error};
#[allow(dead_code)]
fn extract_trace_options(uri: &Uri) -> S3Result<ServiceTraceOpts> { fn extract_trace_options(uri: &Uri) -> S3Result<ServiceTraceOpts> {
let mut st_opts = ServiceTraceOpts::default(); let mut st_opts = ServiceTraceOpts::default();
st_opts st_opts
@@ -31,7 +30,6 @@ fn extract_trace_options(uri: &Uri) -> S3Result<ServiceTraceOpts> {
Ok(st_opts) Ok(st_opts)
} }
#[allow(dead_code)]
pub struct Trace {} pub struct Trace {}
#[async_trait::async_trait] #[async_trait::async_trait]
-1
View File
@@ -19,7 +19,6 @@ pub mod handlers;
mod plugin_contract; mod plugin_contract;
pub(crate) mod replication_metrics_wire; pub(crate) mod replication_metrics_wire;
// Contract inventory is validated by tests before later runtime integration. // Contract inventory is validated by tests before later runtime integration.
#[allow(dead_code)]
pub(crate) mod route_policy; pub(crate) mod route_policy;
pub mod router; pub mod router;
pub(crate) mod runtime_sources; pub(crate) mod runtime_sources;
+4
View File
@@ -1598,6 +1598,10 @@ pub const DEFERRED_ADMIN_ROUTE_POLICIES: &[DeferredAdminRoutePolicy] = &[
), ),
]; ];
#[allow(
dead_code,
reason = "asserted by this file's tests; the lib target cannot see test-only consumers (backlog#1823)"
)]
pub fn validate_admin_route_policy_specs() -> Result<(), AdminRouteMatrixError> { pub fn validate_admin_route_policy_specs() -> Result<(), AdminRouteMatrixError> {
validate_admin_route_specs(ADMIN_ROUTE_POLICY_SPECS) validate_admin_route_specs(ADMIN_ROUTE_POLICY_SPECS)
} }
-1
View File
@@ -5800,7 +5800,6 @@ mod tests {
} }
} }
#[allow(dead_code)]
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct Extra { pub struct Extra {
pub credentials: Option<s3s::auth::Credentials>, pub credentials: Option<s3s::auth::Credentials>,
-2
View File
@@ -45,7 +45,6 @@ pub struct AppContext {
object_store: Arc<ECStore>, object_store: Arc<ECStore>,
iam: Arc<dyn IamInterface>, iam: Arc<dyn IamInterface>,
federated_identity: Arc<dyn FederatedIdentityInterface>, federated_identity: Arc<dyn FederatedIdentityInterface>,
#[allow(dead_code)]
kms: Arc<dyn KmsInterface>, kms: Arc<dyn KmsInterface>,
kms_runtime: Arc<dyn KmsRuntimeInterface>, kms_runtime: Arc<dyn KmsRuntimeInterface>,
outbound_tls_runtime: Arc<dyn OutboundTlsRuntimeInterface>, outbound_tls_runtime: Arc<dyn OutboundTlsRuntimeInterface>,
@@ -162,7 +161,6 @@ impl AppContext {
self.federated_identity.publish_handle(service) self.federated_identity.publish_handle(service)
} }
#[allow(dead_code)]
pub fn kms(&self) -> Arc<dyn KmsInterface> { pub fn kms(&self) -> Arc<dyn KmsInterface> {
self.kms.clone() self.kms.clone()
} }
-2
View File
@@ -49,7 +49,6 @@ use tokio::sync::RwLock;
/// Default IAM interface adapter. /// Default IAM interface adapter.
pub struct IamHandle { pub struct IamHandle {
#[allow(dead_code)]
iam: Arc<IamSys<ObjectStore>>, iam: Arc<IamSys<ObjectStore>>,
} }
@@ -110,7 +109,6 @@ impl FederatedIdentityInterface for FederatedIdentityHandle {
} }
/// Default KMS interface adapter. /// Default KMS interface adapter.
#[allow(dead_code)]
pub struct KmsHandle { pub struct KmsHandle {
kms: Arc<KmsServiceManager>, kms: Arc<KmsServiceManager>,
} }
-2
View File
@@ -36,7 +36,6 @@ use tokio::sync::RwLock;
/// IAM interface for application-layer use-cases. /// IAM interface for application-layer use-cases.
pub trait IamInterface: Send + Sync { pub trait IamInterface: Send + Sync {
#[allow(dead_code)]
fn handle(&self) -> Arc<IamSys<ObjectStore>>; fn handle(&self) -> Arc<IamSys<ObjectStore>>;
fn is_ready(&self) -> bool; fn is_ready(&self) -> bool;
fn token_signing_key(&self) -> Option<String> { fn token_signing_key(&self) -> Option<String> {
@@ -53,7 +52,6 @@ pub trait FederatedIdentityInterface: Send + Sync {
} }
/// KMS interface for application-layer use-cases. /// KMS interface for application-layer use-cases.
#[allow(dead_code)]
pub trait KmsInterface: Send + Sync { pub trait KmsInterface: Send + Sync {
fn handle(&self) -> Arc<KmsServiceManager>; fn handle(&self) -> Arc<KmsServiceManager>;
} }
+6 -25
View File
@@ -97,7 +97,7 @@ use super::storage_api::object_usecase::set_disk::{
}; };
use super::storage_api::object_usecase::sse::{ use super::storage_api::object_usecase::sse::{
DecryptionRequest, EncryptionRequest, SSEType, SseKmsPrincipal, apply_bucket_default_lock_retention, DecryptionRequest, EncryptionRequest, SSEType, SseKmsPrincipal, apply_bucket_default_lock_retention,
authorize_sse_kms_object_read, build_ssec_read_headers, encryption_material_to_metadata, authorize_sse_kms_object_read, bucket_default_write_sse, build_ssec_read_headers, encryption_material_to_metadata,
extract_server_side_encryption_from_headers, extract_ssec_params_from_headers, extract_ssekms_context_from_headers, extract_server_side_encryption_from_headers, extract_ssec_params_from_headers, extract_ssekms_context_from_headers,
get_buffer_size_opt_in, load_bucket_object_lock_config_state, map_get_object_reader_error, sse_decryption, sse_encryption, get_buffer_size_opt_in, load_bucket_object_lock_config_state, map_get_object_reader_error, sse_decryption, sse_encryption,
validate_bucket_object_lock_enabled_state, validate_bucket_object_lock_enabled_state,
@@ -169,8 +169,8 @@ use s3s::dto::{
ObjectLockLegalHoldStatus, ObjectLockMode, ObjectLockRetention, ObjectLockRetentionMode, ObjectPart, PutObjectInput, ObjectLockLegalHoldStatus, ObjectLockMode, ObjectLockRetention, ObjectLockRetentionMode, ObjectPart, PutObjectInput,
PutObjectOutput, Range, RequestCharged, RestoreObjectInput, RestoreObjectOutput, RestoreStatus, SSECustomerAlgorithm, PutObjectOutput, Range, RequestCharged, RestoreObjectInput, RestoreObjectOutput, RestoreStatus, SSECustomerAlgorithm,
SSECustomerKeyMD5, SSEKMSKeyId, SelectObjectContentInput, SelectObjectContentOutput, ServerSideEncryption, SSECustomerKeyMD5, SSEKMSKeyId, SelectObjectContentInput, SelectObjectContentOutput, ServerSideEncryption,
ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, StorageClass, StreamingBlob, TaggingDirective, ServerSideEncryptionConfiguration, StorageClass, StreamingBlob, TaggingDirective, TaggingHeader, Timestamp, TimestampFormat,
TaggingHeader, Timestamp, TimestampFormat, WebsiteRedirectLocation, WebsiteRedirectLocation,
}; };
use s3s::header::{X_AMZ_RESTORE, X_AMZ_RESTORE_OUTPUT_PATH}; use s3s::header::{X_AMZ_RESTORE, X_AMZ_RESTORE_OUTPUT_PATH};
use s3s::stream::{ByteStream, DynByteStream, RemainingLength}; use s3s::stream::{ByteStream, DynByteStream, RemainingLength};
@@ -738,7 +738,7 @@ struct GetObjectPreparedRead {
} }
struct GetObjectStrategyContext { struct GetObjectStrategyContext {
#[allow(dead_code)] #[allow(dead_code, reason = "written but never read back (backlog#1823)")]
io_strategy: concurrency::IoStrategy, io_strategy: concurrency::IoStrategy,
optimal_buffer_size: usize, optimal_buffer_size: usize,
enable_readahead: bool, enable_readahead: bool,
@@ -2676,25 +2676,6 @@ fn has_put_sse_request_headers(headers: &HeaderMap) -> bool {
|| headers.get(AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID).is_some() || headers.get(AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID).is_some()
} }
/// Managed SSE resolved from a bucket default encryption rule on the copy path.
///
/// Unknown algorithms fall back to AES256, the same total mapping as the PUT and
/// extract paths and the storage-layer resolver (`prepare_sse_configuration`), which
/// `sse_encryption` re-runs when it mints the destination DEK. Resolving `None` here
/// instead lets a same-name copy under a malformed bucket default pass the
/// `copy_changes_encryption` guard and take the metadata-only shortcut while the
/// storage layer still encrypts: fresh DEK metadata is committed beside the untouched
/// plaintext blocks and the object becomes unreadable. Reachable only via corrupt or
/// hand-edited bucket metadata — PutBucketEncryption rejects unknown algorithms
/// (backlog#1826).
fn bucket_default_write_sse(sse: &ServerSideEncryptionByDefault) -> ServerSideEncryption {
match sse.sse_algorithm.as_str() {
"AES256" => ServerSideEncryption::from_static(ServerSideEncryption::AES256),
"aws:kms" => ServerSideEncryption::from_static(ServerSideEncryption::AWS_KMS),
_ => ServerSideEncryption::from_static(ServerSideEncryption::AES256),
}
}
/// Resolve the effective server-side encryption for a write against the bucket's /// Resolve the effective server-side encryption for a write against the bucket's
/// default encryption configuration. /// default encryption configuration.
/// ///
@@ -10115,8 +10096,8 @@ mod tests {
DefaultRetention, Delete, DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication, DefaultRetention, Delete, DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication,
DeleteReplicationStatus, Destination, ExistingObjectReplication, ExistingObjectReplicationStatus, ObjectIdentifier, DeleteReplicationStatus, Destination, ExistingObjectReplication, ExistingObjectReplicationStatus, ObjectIdentifier,
ObjectLockConfiguration, ObjectLockEnabled, ObjectLockRule, ReplicaModifications, ReplicaModificationsStatus, ObjectLockConfiguration, ObjectLockEnabled, ObjectLockRule, ReplicaModifications, ReplicaModificationsStatus,
ReplicationConfiguration, ReplicationRule, ReplicationRuleStatus, RestoreRequest, ServerSideEncryptionConfiguration, ReplicationConfiguration, ReplicationRule, ReplicationRuleStatus, RestoreRequest, ServerSideEncryptionByDefault,
ServerSideEncryptionRule, SourceSelectionCriteria, ServerSideEncryptionConfiguration, ServerSideEncryptionRule, SourceSelectionCriteria,
}; };
use std::pin::Pin; use std::pin::Pin;
use std::sync::Arc; use std::sync::Arc;
+3 -2
View File
@@ -1032,8 +1032,9 @@ pub(crate) mod sse {
validate_bucket_object_lock_enabled_state, validate_bucket_object_lock_enabled_state,
}; };
pub(crate) use crate::storage::storage_api::sse_consumer::{ pub(crate) use crate::storage::storage_api::sse_consumer::{
EncryptionKeyKind, SSEType, build_ssec_read_headers, encryption_material_to_metadata, extract_ssec_params_from_headers, EncryptionKeyKind, SSEType, bucket_default_write_sse, build_ssec_read_headers, encryption_material_to_metadata,
extract_ssekms_context_from_headers, map_get_object_reader_error, mark_encrypted_multipart_metadata, extract_ssec_params_from_headers, extract_ssekms_context_from_headers, map_get_object_reader_error,
mark_encrypted_multipart_metadata,
}; };
} }
@@ -31,7 +31,6 @@ pub async fn init_capacity_management_managed() -> Option<CapacityBackgroundTask
} }
/// Get capacity statistics with metrics /// Get capacity statistics with metrics
#[allow(dead_code)]
pub async fn get_capacity_with_metrics() -> Option<(u64, String)> { pub async fn get_capacity_with_metrics() -> Option<(u64, String)> {
get_cached_capacity_with_metrics() get_cached_capacity_with_metrics()
.await .await
-40
View File
@@ -765,7 +765,6 @@ fn resolve_buffer_profile_config(
/// Parse and normalize server address for FTP/FTPS /// Parse and normalize server address for FTP/FTPS
/// Forces IPv4 binding to avoid libunftp IPv6 compatibility issues /// Forces IPv4 binding to avoid libunftp IPv6 compatibility issues
#[allow(dead_code)]
async fn parse_and_normalize_server_address( async fn parse_and_normalize_server_address(
address_str: &str, address_str: &str,
) -> Result<std::net::SocketAddr, Box<dyn std::error::Error + Send + Sync>> { ) -> Result<std::net::SocketAddr, Box<dyn std::error::Error + Send + Sync>> {
@@ -781,45 +780,6 @@ async fn parse_and_normalize_server_address(
Ok(normalized_addr) Ok(normalized_addr)
} }
/// Start FTP/FTPS server in background with shutdown support
/// # Arguments
/// * `server` - The FTP/FTPS server instance
/// * `protocol_name` - Name of the protocol (e.g., "FTP", "FTPS")
#[allow(dead_code)]
fn spawn_server<S>(server: S, protocol_name: &'static str) -> tokio::sync::broadcast::Sender<()>
where
S: std::future::Future<Output = Result<(), Box<dyn std::error::Error>>> + Send + 'static,
{
let (shutdown_tx, _) = tokio::sync::broadcast::channel(1);
tokio::spawn(async move {
if let Err(e) = server.await {
error!(
target: "rustfs::init",
event = "protocol_server_state",
component = LOG_COMPONENT_INIT,
subsystem = LOG_SUBSYSTEM_PROTOCOL,
protocol = protocol_name,
state = "runtime_failed",
error = %e,
"Protocol server failed"
);
}
info!(
target: "rustfs::init",
event = "protocol_server_state",
component = LOG_COMPONENT_INIT,
subsystem = LOG_SUBSYSTEM_PROTOCOL,
protocol = protocol_name,
state = "stopped",
"Protocol server stopped"
);
});
shutdown_tx
}
/// Starts the auto-tuner for performance optimization if enabled via environment variable. /// Starts the auto-tuner for performance optimization if enabled via environment variable.
/// ///
/// The auto-tuner reads `RUSTFS_AUTOTUNER_ENABLED` to decide whether to run. /// The auto-tuner reads `RUSTFS_AUTOTUNER_ENABLED` to decide whether to run.
-1
View File
@@ -211,7 +211,6 @@ fn apply_valid_status(state: &mut LicenseState, token: Token) {
/// ///
/// This is the extension point for OEM/build-time overlays. /// This is the extension point for OEM/build-time overlays.
/// Returns `false` if the verifier was already initialized. /// Returns `false` if the verifier was already initialized.
#[allow(dead_code)]
pub fn set_license_verifier(verifier: SharedLicenseVerifier) -> bool { pub fn set_license_verifier(verifier: SharedLicenseVerifier) -> bool {
LICENSE_VERIFIER.set(verifier).is_ok() LICENSE_VERIFIER.set(verifier).is_ok()
} }
-11
View File
@@ -352,17 +352,6 @@ fn should_fail_test_init_attempt() -> bool {
false false
} }
} }
/// Reset the test failure counter so the next `should_fail_test_init_attempt`
/// call re-reads the environment variable by restoring the sentinel value.
/// Intended for use in integration tests that share a process.
#[doc(hidden)]
#[allow(dead_code)]
pub(crate) fn reset_test_failure_counter() {
use std::sync::atomic::Ordering;
TEST_REMAINING_FAILURES.store(u64::MAX, Ordering::SeqCst);
}
async fn attempt_init_iam_sys( async fn attempt_init_iam_sys(
store: Arc<ECStore>, store: Arc<ECStore>,
) -> std::result::Result<Arc<rustfs_iam::sys::IamSys<rustfs_iam::store::object::ObjectStore>>, std::io::Error> { ) -> std::result::Result<Arc<rustfs_iam::sys::IamSys<rustfs_iam::store::object::ObjectStore>>, std::io::Error> {
+1 -1
View File
@@ -61,7 +61,7 @@ pub(crate) struct ReqInfo {
pub object: Option<String>, pub object: Option<String>,
pub version_id: Option<String>, pub version_id: Option<String>,
pub replication_request_authorized: bool, pub replication_request_authorized: bool,
#[allow(dead_code)] #[allow(dead_code, reason = "written but never read back (backlog#1823)")]
pub region: Option<s3s::region::Region>, pub region: Option<s3s::region::Region>,
pub request_context: Option<RequestContext>, pub request_context: Option<RequestContext>,
/// Set by probe-style callers that treat AccessDenied as an expected filter /// Set by probe-style callers that treat AccessDenied as an expected filter
+1 -2
View File
@@ -50,7 +50,7 @@ pub struct ConcurrencyManager {
/// I/O load metrics for adaptive strategy calculation /// I/O load metrics for adaptive strategy calculation
io_metrics: Arc<Mutex<IoLoadMetrics>>, io_metrics: Arc<Mutex<IoLoadMetrics>>,
/// I/O priority queue for request scheduling /// I/O priority queue for request scheduling
#[allow(dead_code)] #[allow(dead_code, reason = "written but never read back (backlog#1823)")]
priority_queue: Arc<IoPriorityQueue<()>>, priority_queue: Arc<IoPriorityQueue<()>>,
/// Bytes pool for buffer allocation and reuse /// Bytes pool for buffer allocation and reuse
bytes_pool: Arc<BytesPool>, bytes_pool: Arc<BytesPool>,
@@ -131,7 +131,6 @@ pub enum PutObjectAdmission {
Rejected, Rejected,
} }
#[allow(dead_code)]
impl ConcurrencyManager { impl ConcurrencyManager {
/// Create a new concurrency manager with default settings /// Create a new concurrency manager with default settings
/// ///
@@ -64,7 +64,6 @@ impl GetObjectGuard {
} }
/// Get the elapsed time since this guard was created. /// Get the elapsed time since this guard was created.
#[allow(dead_code)]
// This helper is primarily used by unit tests to assert timing. // This helper is primarily used by unit tests to assert timing.
// It's intentionally kept public for callers that may want to inspect // It's intentionally kept public for callers that may want to inspect
// a guard's duration without dropping it. // a guard's duration without dropping it.
+4 -17
View File
@@ -254,7 +254,10 @@ pub(crate) fn apply_bucket_default_lock_retention(
/// ); /// );
/// ``` /// ```
/// ///
#[allow(dead_code)] #[allow(
dead_code,
reason = "exercised by ecfs_test; the lib target cannot see test-only consumers (backlog#1823)"
)]
pub(crate) fn get_adaptive_buffer_size_with_profile(file_size: i64, profile: Option<WorkloadProfile>) -> usize { pub(crate) fn get_adaptive_buffer_size_with_profile(file_size: i64, profile: Option<WorkloadProfile>) -> usize {
let config = match profile { let config = match profile {
Some(p) => RustFSBufferConfig::new(p), Some(p) => RustFSBufferConfig::new(p),
@@ -798,26 +801,10 @@ fn cache_remove(bucket: &str) {
map.remove(bucket); map.remove(bucket);
} }
} }
/// Clear all entries in the cache.
#[allow(dead_code)]
fn cache_clear() {
if let Ok(mut map) = small_cache().write() {
map.clear();
}
}
/// Invalidate the validation cache for a specific bucket. /// Invalidate the validation cache for a specific bucket.
pub fn invalidate_bucket_validation_cache(bucket: &str) { pub fn invalidate_bucket_validation_cache(bucket: &str) {
cache_remove(bucket); cache_remove(bucket);
} }
/// Invalidate all bucket validation cache entries.
#[allow(dead_code)]
pub fn invalidate_all_bucket_validation_cache() {
cache_clear();
}
/// Helper function to get store and validate bucket exists. /// Helper function to get store and validate bucket exists.
/// ///
/// Uses adaptive cache with 5s TTL to avoid repeated stat_volume() calls. /// Uses adaptive cache with 5s TTL to avoid repeated stat_volume() calls.
-9
View File
@@ -15,15 +15,6 @@
use super::ECStore; use super::ECStore;
use crate::storage::storage_api::head_prefix_consumer::contract::list::ListOperations as _; use crate::storage::storage_api::head_prefix_consumer::contract::list::ListOperations as _;
use std::sync::Arc; use std::sync::Arc;
/// Determines if the key "looks like a prefix" (ends with `/`).
/// Note: No special handling for empty strings here; the caller must ensure the key has passed `validate_object_key`.
#[allow(dead_code)]
#[inline]
pub(crate) fn is_prefix_key(key: &str) -> bool {
key.ends_with('/')
}
/// Constructs a more explicit error message when `HEAD` is performed on a `prefix`-style key but the directory marker object is missing. /// Constructs a more explicit error message when `HEAD` is performed on a `prefix`-style key but the directory marker object is missing.
/// ///
/// `has_children`: /// `has_children`:
-20
View File
@@ -1018,12 +1018,6 @@ pub fn parse_copy_source_range(range_str: &str) -> S3Result<HTTPRangeSpec> {
Err(s3_error!(InvalidArgument, "Invalid range format")) Err(s3_error!(InvalidArgument, "Invalid range format"))
} }
} }
#[allow(dead_code)]
pub(crate) fn get_content_sha256(headers: &HeaderMap<HeaderValue>) -> Option<String> {
get_content_sha256_with_query(headers, None)
}
pub(crate) fn get_content_sha256_with_query(headers: &HeaderMap<HeaderValue>, query: Option<&str>) -> Option<String> { pub(crate) fn get_content_sha256_with_query(headers: &HeaderMap<HeaderValue>, query: Option<&str>) -> Option<String> {
match get_request_auth_type_with_query(headers, query) { match get_request_auth_type_with_query(headers, query) {
AuthType::Presigned | AuthType::Signed => { AuthType::Presigned | AuthType::Signed => {
@@ -1036,14 +1030,6 @@ pub(crate) fn get_content_sha256_with_query(headers: &HeaderMap<HeaderValue>, qu
_ => None, _ => None,
} }
} }
/// skip_content_sha256_cksum returns true if caller needs to skip
/// payload checksum, false if not.
#[allow(dead_code)]
fn skip_content_sha256_cksum(headers: &HeaderMap<HeaderValue>) -> bool {
skip_content_sha256_cksum_with_query(headers, None)
}
fn skip_content_sha256_cksum_with_query(headers: &HeaderMap<HeaderValue>, query: Option<&str>) -> bool { fn skip_content_sha256_cksum_with_query(headers: &HeaderMap<HeaderValue>, query: Option<&str>) -> bool {
let include_query_values = matches!(get_request_auth_type_with_query(headers, query), AuthType::Presigned); let include_query_values = matches!(get_request_auth_type_with_query(headers, query), AuthType::Presigned);
let content_sha256 = get_content_sha256_value(headers, query, include_query_values); let content_sha256 = get_content_sha256_value(headers, query, include_query_values);
@@ -1138,12 +1124,6 @@ fn get_content_sha256_value(
.and_then(|v| v.to_str().ok()) .and_then(|v| v.to_str().ok())
.map(str::to_owned) .map(str::to_owned)
} }
#[allow(dead_code)]
fn get_content_sha256_cksum(headers: &HeaderMap<HeaderValue>, service_type: ServiceType) -> String {
get_content_sha256_cksum_with_query(headers, None, service_type)
}
#[cfg(test)] #[cfg(test)]
#[allow(unused_imports)] #[allow(unused_imports)]
mod tests { mod tests {
+156 -27
View File
@@ -23,12 +23,15 @@ use bytes::Bytes;
use rustfs_filemeta::FileInfo; use rustfs_filemeta::FileInfo;
use rustfs_io_metrics::internode_metrics::{ use rustfs_io_metrics::internode_metrics::{
INTERNODE_MSGPACK_CODEC_JSON, INTERNODE_MSGPACK_CODEC_MSGPACK, INTERNODE_MSGPACK_DIRECTION_REQUEST, INTERNODE_MSGPACK_CODEC_JSON, INTERNODE_MSGPACK_CODEC_MSGPACK, INTERNODE_MSGPACK_DIRECTION_REQUEST,
INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC, INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_READ_VERSION, INTERNODE_OPERATION_GRPC_WRITE_ALL,
global_internode_metrics, INTERNODE_STAGE_READ_VERSION_DISK_READ, INTERNODE_STAGE_READ_VERSION_REQUEST_DECODE,
INTERNODE_STAGE_READ_VERSION_RESPONSE_JSON_ENCODE, INTERNODE_STAGE_READ_VERSION_RESPONSE_MSGPACK_ENCODE,
INTERNODE_TRANSPORT_BACKEND_GRPC, global_internode_metrics,
}; };
use rustfs_protos::proto_gen::node_service::*; use rustfs_protos::proto_gen::node_service::*;
use serde::de::DeserializeOwned; use serde::de::DeserializeOwned;
use std::io::Cursor; use std::io::Cursor;
use std::time::Instant;
use tonic::{Request, Response, Status}; use tonic::{Request, Response, Status};
use tracing::debug; use tracing::debug;
@@ -201,6 +204,21 @@ fn encode_read_multiple_response_payloads(
Ok((read_multiple_resps_json, read_multiple_resps_bin)) Ok((read_multiple_resps_json, read_multiple_resps_bin))
} }
fn internode_stage_timer(attribution_enabled: bool) -> Option<Instant> {
attribution_enabled.then(Instant::now)
}
fn record_read_version_stage(stage: &'static str, started_at: Option<Instant>) {
if let Some(started_at) = started_at {
global_internode_metrics().record_stage_duration_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
stage,
started_at.elapsed(),
);
}
}
fn encode_batch_read_version_response_payloads( fn encode_batch_read_version_response_payloads(
batch_read_version_resps: &[BatchReadVersionResp], batch_read_version_resps: &[BatchReadVersionResp],
request_decoded_from_msgpack: bool, request_decoded_from_msgpack: bool,
@@ -685,11 +703,42 @@ impl NodeService {
request: Request<ReadVersionRequest>, request: Request<ReadVersionRequest>,
) -> Result<Response<ReadVersionResponse>, Status> { ) -> Result<Response<ReadVersionResponse>, Status> {
let request = request.into_inner(); let request = request.into_inner();
let metrics = global_internode_metrics();
let read_version_attribution_enabled = rustfs_io_metrics::get_stage_metrics_enabled();
if read_version_attribution_enabled {
metrics.record_incoming_request_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
);
metrics.record_recv_bytes_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
request
.disk
.len()
.saturating_add(request.volume.len())
.saturating_add(request.path.len())
.saturating_add(request.version_id.len())
.saturating_add(request.opts.len())
.saturating_add(request.opts_bin.len()),
);
}
if let Some(disk) = self.find_disk(&request.disk).await { if let Some(disk) = self.find_disk(&request.disk).await {
let request_had_msgpack_payload = !request.opts_bin.is_empty(); let request_had_msgpack_payload = !request.opts_bin.is_empty();
let decode_started = internode_stage_timer(read_version_attribution_enabled);
let opts = match decode_msgpack_or_json::<ReadOptions>(&request.opts_bin, &request.opts, "ReadOptions") { let opts = match decode_msgpack_or_json::<ReadOptions>(&request.opts_bin, &request.opts, "ReadOptions") {
Ok(options) => options, Ok(options) => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_REQUEST_DECODE, decode_started);
options
}
Err(err) => { Err(err) => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_REQUEST_DECODE, decode_started);
if read_version_attribution_enabled {
metrics.record_error_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
);
}
return Ok(Response::new(ReadVersionResponse { return Ok(Response::new(ReadVersionResponse {
success: false, success: false,
file_info: String::new(), file_info: String::new(),
@@ -698,42 +747,88 @@ impl NodeService {
})); }));
} }
}; };
let disk_read_started = internode_stage_timer(read_version_attribution_enabled);
match disk match disk
.read_version("", &request.volume, &request.path, &request.version_id, &opts) .read_version("", &request.volume, &request.path, &request.version_id, &opts)
.await .await
{ {
Ok(file_info) => { Ok(file_info) => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_DISK_READ, disk_read_started);
let json_encode_started = internode_stage_timer(read_version_attribution_enabled);
let file_info_json = compat_response_json(&file_info, request_had_msgpack_payload); let file_info_json = compat_response_json(&file_info, request_had_msgpack_payload);
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_RESPONSE_JSON_ENCODE, json_encode_started);
let msgpack_encode_started = internode_stage_timer(read_version_attribution_enabled);
let file_info_bin = encode_file_info_msgpack(&file_info); let file_info_bin = encode_file_info_msgpack(&file_info);
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_RESPONSE_MSGPACK_ENCODE, msgpack_encode_started);
match (file_info_json, file_info_bin) { match (file_info_json, file_info_bin) {
(Ok(file_info), Ok(file_info_bin)) => Ok(Response::new(ReadVersionResponse { (Ok(file_info), Ok(file_info_bin)) => {
success: true, if read_version_attribution_enabled {
file_info, metrics.record_sent_bytes_for_operation_and_backend(
file_info_bin: file_info_bin.into(), INTERNODE_OPERATION_GRPC_READ_VERSION,
error: None, INTERNODE_TRANSPORT_BACKEND_GRPC,
})), file_info.len().saturating_add(file_info_bin.len()),
(Err(err), _) => Ok(Response::new(ReadVersionResponse { );
success: false, }
file_info: String::new(), Ok(Response::new(ReadVersionResponse {
file_info_bin: Vec::new().into(), success: true,
error: Some(DiskError::other(format!("encode data failed: {err}")).into()), file_info,
})), file_info_bin: file_info_bin.into(),
(_, Err(err)) => Ok(Response::new(ReadVersionResponse { error: None,
success: false, }))
file_info: String::new(), }
file_info_bin: Vec::new().into(), (Err(err), _) => {
error: Some(DiskError::other(format!("encode data failed: {err}")).into()), if read_version_attribution_enabled {
})), metrics.record_error_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
);
}
Ok(Response::new(ReadVersionResponse {
success: false,
file_info: String::new(),
file_info_bin: Vec::new().into(),
error: Some(DiskError::other(format!("encode data failed: {err}")).into()),
}))
}
(_, Err(err)) => {
if read_version_attribution_enabled {
metrics.record_error_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
);
}
Ok(Response::new(ReadVersionResponse {
success: false,
file_info: String::new(),
file_info_bin: Vec::new().into(),
error: Some(DiskError::other(format!("encode data failed: {err}")).into()),
}))
}
} }
} }
Err(err) => Ok(Response::new(ReadVersionResponse { Err(err) => {
success: false, record_read_version_stage(INTERNODE_STAGE_READ_VERSION_DISK_READ, disk_read_started);
file_info: String::new(), if read_version_attribution_enabled {
file_info_bin: Vec::new().into(), metrics.record_error_for_operation_and_backend(
error: Some(err.into()), INTERNODE_OPERATION_GRPC_READ_VERSION,
})), INTERNODE_TRANSPORT_BACKEND_GRPC,
);
}
Ok(Response::new(ReadVersionResponse {
success: false,
file_info: String::new(),
file_info_bin: Vec::new().into(),
error: Some(err.into()),
}))
}
} }
} else { } else {
if read_version_attribution_enabled {
metrics.record_error_for_operation_and_backend(
INTERNODE_OPERATION_GRPC_READ_VERSION,
INTERNODE_TRANSPORT_BACKEND_GRPC,
);
}
Ok(Response::new(ReadVersionResponse { Ok(Response::new(ReadVersionResponse {
success: false, success: false,
file_info: String::new(), file_info: String::new(),
@@ -1520,12 +1615,16 @@ mod tests {
encode_batch_read_version_response_payloads, encode_file_info_msgpack, encode_msgpack, encode_msgpack_named, encode_batch_read_version_response_payloads, encode_file_info_msgpack, encode_msgpack, encode_msgpack_named,
encode_read_multiple_response_payloads, encode_rename_data_response_payloads, encode_read_multiple_response_payloads, encode_rename_data_response_payloads,
}; };
use crate::storage::rpc::node_service::make_server;
use crate::storage::storage_api::ReadMultipleResp; use crate::storage::storage_api::ReadMultipleResp;
use crate::storage::storage_api::RenameDataResp; use crate::storage::storage_api::RenameDataResp;
use crate::storage::storage_api::rpc_consumer::node_service::BatchReadVersionResp; use crate::storage::storage_api::rpc_consumer::node_service::BatchReadVersionResp;
use rustfs_filemeta::FileInfo; use rustfs_filemeta::FileInfo;
use rustfs_io_metrics::internode_metrics::global_internode_metrics; use rustfs_io_metrics::internode_metrics::global_internode_metrics;
use rustfs_protos::proto_gen::node_service::ReadVersionRequest;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use serial_test::serial;
use tonic::Request;
#[derive(Debug, PartialEq, Eq, Serialize, Deserialize)] #[derive(Debug, PartialEq, Eq, Serialize, Deserialize)]
struct SamplePayload { struct SamplePayload {
@@ -1533,6 +1632,36 @@ mod tests {
count: u32, count: u32,
} }
#[tokio::test]
#[serial]
async fn handle_read_version_records_attribution_for_missing_disk() {
let metrics = global_internode_metrics();
let previous_stage_metrics = rustfs_io_metrics::get_stage_metrics_enabled();
metrics.reset_for_test();
rustfs_io_metrics::set_get_stage_metrics_enabled(true);
let response = make_server()
.handle_read_version(Request::new(ReadVersionRequest {
disk: "missing-disk".to_string(),
volume: "bucket".to_string(),
path: "object".to_string(),
version_id: String::new(),
opts: String::new(),
opts_bin: Vec::new().into(),
}))
.await
.expect("ReadVersion handler should return a response")
.into_inner();
rustfs_io_metrics::set_get_stage_metrics_enabled(previous_stage_metrics);
let snapshot = metrics.snapshot();
assert!(!response.success);
assert_eq!(snapshot.incoming_requests_total, 1);
assert_eq!(snapshot.errors_total, 1);
assert!(snapshot.recv_bytes_total > 0);
metrics.reset_for_test();
}
#[test] #[test]
fn decode_msgpack_or_json_prefers_binary_payload() { fn decode_msgpack_or_json_prefers_binary_payload() {
let payload = SamplePayload { let payload = SamplePayload {
+20 -8
View File
@@ -164,7 +164,7 @@ use rustfs_utils::http::headers::{
AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5, AMZ_SERVER_SIDE_ENCRYPTION_KMS_CONTEXT, AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5, AMZ_SERVER_SIDE_ENCRYPTION_KMS_CONTEXT,
}; };
use rustfs_utils::path::path_join_buf; use rustfs_utils::path::path_join_buf;
use s3s::dto::{SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5, SSEKMSKeyId}; use s3s::dto::{SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5, SSEKMSKeyId, ServerSideEncryptionByDefault};
use std::borrow::Cow; use std::borrow::Cow;
// ============================================================================ // ============================================================================
@@ -203,6 +203,24 @@ pub struct SseConfiguration {
/// Effective KMS key ID (after considering bucket defaults) /// Effective KMS key ID (after considering bucket defaults)
pub effective_kms_key_id: Option<SSEKMSKeyId>, pub effective_kms_key_id: Option<SSEKMSKeyId>,
} }
/// Managed SSE resolved from a bucket default encryption rule on a write path.
///
/// The single mapping shared by every writer: this resolver, and the PUT, COPY
/// and extract paths in `app::object_usecase`, which reach it through
/// `resolve_bucket_default_sse`. Unknown algorithms fall back to AES256 rather
/// than to `None`. Resolving `None` instead lets a same-name copy under a
/// malformed bucket default pass the `copy_changes_encryption` guard and take
/// the metadata-only shortcut while this layer still encrypts: fresh DEK
/// metadata is committed beside the untouched plaintext blocks and the object
/// becomes unreadable. Reachable only via corrupt or hand-edited bucket
/// metadata — PutBucketEncryption rejects unknown algorithms (backlog#1826).
pub(crate) fn bucket_default_write_sse(sse: &ServerSideEncryptionByDefault) -> ServerSideEncryption {
match sse.sse_algorithm.as_str() {
"AES256" => ServerSideEncryption::from_static(ServerSideEncryption::AES256),
"aws:kms" => ServerSideEncryption::from_static(ServerSideEncryption::AWS_KMS),
_ => ServerSideEncryption::from_static(ServerSideEncryption::AES256),
}
}
/// Prepare SSE configuration by resolving request parameters with bucket defaults /// Prepare SSE configuration by resolving request parameters with bucket defaults
/// ///
@@ -266,11 +284,7 @@ async fn prepare_sse_configuration(
has_kms_key_id = sse.kms_master_key_id.is_some(), has_kms_key_id = sse.kms_master_key_id.is_some(),
"Bucket SSE default resolved" "Bucket SSE default resolved"
); );
match sse.sse_algorithm.as_str() { bucket_default_write_sse(sse)
"AES256" => ServerSideEncryption::from_static(ServerSideEncryption::AES256),
"aws:kms" => ServerSideEncryption::from_static(ServerSideEncryption::AWS_KMS),
_ => ServerSideEncryption::from_static(ServerSideEncryption::AES256), // fallback
}
}) })
}) })
}); });
@@ -3354,7 +3368,6 @@ async fn get_local_sse_dek_provider() -> Result<Arc<dyn SseDekProvider>, ApiErro
/// Clears GLOBAL_SSE_DEK_PROVIDER (local/test providers) and /// Clears GLOBAL_SSE_DEK_PROVIDER (local/test providers) and
/// GLOBAL_KMS_DEK_PROVIDER (test-injected KMS providers). /// GLOBAL_KMS_DEK_PROVIDER (test-injected KMS providers).
#[cfg(test)] #[cfg(test)]
#[allow(dead_code)]
pub fn reset_sse_dek_provider() { pub fn reset_sse_dek_provider() {
if let Ok(mut slot) = GLOBAL_SSE_DEK_PROVIDER.write() { if let Ok(mut slot) = GLOBAL_SSE_DEK_PROVIDER.write() {
*slot = None; *slot = None;
@@ -3365,7 +3378,6 @@ pub fn reset_sse_dek_provider() {
} }
#[cfg(test)] #[cfg(test)]
#[allow(dead_code)]
pub fn set_sse_dek_provider_for_test(provider: Arc<dyn SseDekProvider>) { pub fn set_sse_dek_provider_for_test(provider: Arc<dyn SseDekProvider>) {
if let Ok(mut slot) = GLOBAL_KMS_DEK_PROVIDER.write() { if let Ok(mut slot) = GLOBAL_KMS_DEK_PROVIDER.write() {
*slot = Some(provider.clone()); *slot = Some(provider.clone());
+3 -3
View File
@@ -344,9 +344,9 @@ pub(crate) mod s3_api_consumer {
pub(crate) mod sse_consumer { pub(crate) mod sse_consumer {
pub(crate) use super::super::sse::{ pub(crate) use super::super::sse::{
EncryptionKeyKind, SSEType, build_ssec_read_headers, encryption_material_to_metadata, extract_ssec_params_from_headers, EncryptionKeyKind, SSEType, bucket_default_write_sse, build_ssec_read_headers, encryption_material_to_metadata,
extract_ssekms_context_from_headers, log_sse_kms_key_policy_mode, map_get_object_reader_error, extract_ssec_params_from_headers, extract_ssekms_context_from_headers, log_sse_kms_key_policy_mode,
mark_encrypted_multipart_metadata, map_get_object_reader_error, mark_encrypted_multipart_metadata,
}; };
pub(crate) use super::{ pub(crate) use super::{
DecryptionRequest, EncryptionRequest, PrepareEncryptionRequest, SseKmsPrincipal, apply_bucket_default_lock_retention, DecryptionRequest, EncryptionRequest, PrepareEncryptionRequest, SseKmsPrincipal, apply_bucket_default_lock_retention,
-1
View File
@@ -16,5 +16,4 @@ pub(crate) use crate::storage::rpc::node_service::make_heal_control_server_with_
#[cfg(test)] #[cfg(test)]
pub(crate) use crate::storage::rpc::node_service::{heal::heal_topology_fingerprint, make_heal_control_server_for_source}; pub(crate) use crate::storage::rpc::node_service::{heal::heal_topology_fingerprint, make_heal_control_server_for_source};
pub use crate::storage::rpc::{make_heal_control_server, make_server, make_tier_mutation_control_server}; pub use crate::storage::rpc::{make_heal_control_server, make_server, make_tier_mutation_control_server};
#[allow(dead_code)]
pub type NodeService = crate::storage::rpc::NodeService; pub type NodeService = crate::storage::rpc::NodeService;
-3
View File
@@ -45,7 +45,6 @@ pub struct VersionInfo {
} }
/// Update check result /// Update check result
#[allow(dead_code)]
#[derive(Debug, Clone, Serialize, Deserialize)] #[derive(Debug, Clone, Serialize, Deserialize)]
pub struct UpdateCheckResult { pub struct UpdateCheckResult {
/// Whether update is available /// Whether update is available
@@ -91,7 +90,6 @@ impl VersionChecker {
} }
/// Create version checker with custom configuration /// Create version checker with custom configuration
#[allow(dead_code)]
pub fn with_config(url: String, timeout: Duration) -> Self { pub fn with_config(url: String, timeout: Duration) -> Self {
let client = reqwest::Client::builder() let client = reqwest::Client::builder()
.timeout(timeout) .timeout(timeout)
@@ -175,7 +173,6 @@ pub async fn check_updates() -> Result<UpdateCheckResult, UpdateCheckError> {
} }
/// Update check with custom URL /// Update check with custom URL
#[allow(dead_code)]
pub async fn check_updates_with_url(url: String) -> Result<UpdateCheckResult, UpdateCheckError> { pub async fn check_updates_with_url(url: String) -> Result<UpdateCheckResult, UpdateCheckError> {
let checker = VersionChecker::with_config(url, Duration::from_secs(10)); let checker = VersionChecker::with_config(url, Duration::from_secs(10));
checker.check_for_updates().await checker.check_for_updates().await