feat(ecstore): type internode client-acquisition failures for quorum buckets (#6619)

* feat(ecstore): type internode client-acquisition failures for stable quorum buckets

Backlog#1845 step 3, first typed family. The largest other(format!) message family in ecstore was 'can not get client, err: {detail}' (~50 production sites): every internode RPC that fails to acquire a client wrapped the dial/auth error with per-peer detail into DiskError::other / StorageError::other, whose Io equality compares the rendered message. N disks failing for this same cause therefore counted as N distinct errors in reduce_errs, starving quorum aggregation, and remote_disk call sites double-wrapped the message on top of get_client's own wrap.

Introduce DiskError::RemoteClientUnavailable(String) (wire code 0x2B) and its StorageError twin (StorageErrorCode 0x54): equality and hashing use the wire code alone, so same-cause failures land in one quorum bucket regardless of per-peer detail, while Display keeps the detail so substring classifiers (network needles, heal recoverability) keep reading it unchanged. Wire encoding carries the rendered detail in error_info and decode restores the typed variant; old peers fall back to the legacy string form gracefully.

Call sites: remote_disk get_client/get_bulk_client/offline-bypass/recovery-probe now construct the typed variant and the ~60 redundant double-wrap map_errs are gone; peer_rest_client's three client getters and offline gates, peer_s3_client, and admin_server_info follow. The tier-config-reload connection classifier's anchored 'can not get client' substring check becomes a typed match on the variant (the string form is retired and now classifies as Terminal, pinned by test).

Ref rustfs/backlog#1845

* chore(ci): refresh error other ratchet baseline

* fix(ecstore): classify typed client network failures
This commit is contained in:
Zhengchao An
2026-08-26 11:24:56 +08:00
committed by GitHub
parent 7cac528de3
commit c0c208d89a
9 changed files with 176 additions and 147 deletions
+12
View File
@@ -221,6 +221,13 @@ pub enum StorageError {
Io(#[source] std::io::Error),
#[error("Lock error: {0}")]
Lock(#[from] rustfs_lock::LockError),
/// Internode RPC client acquisition failed. Mirrors
/// `DiskError::RemoteClientUnavailable`: the detail is diagnostic only and
/// excluded from equality/hashing so same-cause failures bucket together
/// during quorum aggregation (backlog#1845).
#[error("remote rpc client unavailable: {0}")]
RemoteClientUnavailable(String),
}
impl From<crate::erasure::coding::ErasureConstructionError> for StorageError {
@@ -315,6 +322,7 @@ impl From<DiskError> for StorageError {
DiskError::SourceStalled => StorageError::SourceStalled,
DiskError::Timeout => StorageError::Timeout,
DiskError::InvalidPath => StorageError::InvalidPath,
DiskError::RemoteClientUnavailable(detail) => StorageError::RemoteClientUnavailable(detail),
}
}
}
@@ -366,6 +374,7 @@ impl From<StorageError> for DiskError {
StorageError::VolumeNotEmpty => DiskError::VolumeNotEmpty,
StorageError::VolumeAccessDenied => DiskError::VolumeAccessDenied,
StorageError::FileAccessDenied => DiskError::FileAccessDenied,
StorageError::RemoteClientUnavailable(detail) => DiskError::RemoteClientUnavailable(detail),
_ => DiskError::other(val),
}
}
@@ -558,6 +567,7 @@ impl Clone for StorageError {
current: *current,
limit: *limit,
},
StorageError::RemoteClientUnavailable(detail) => StorageError::RemoteClientUnavailable(detail.clone()),
}
}
}
@@ -646,6 +656,7 @@ impl StorageError {
StorageError::InvalidPartNumber(_) => StorageErrorCode::InvalidPartNumber,
StorageError::NamespaceLockQuorumUnavailable { .. } => StorageErrorCode::NamespaceLockQuorumUnavailable,
StorageError::QuotaExceeded { .. } => StorageErrorCode::QuotaExceeded,
StorageError::RemoteClientUnavailable(_) => StorageErrorCode::RemoteClientUnavailable,
}
}
@@ -775,6 +786,7 @@ impl StorageError {
current: Default::default(),
limit: Default::default(),
}),
StorageErrorCode::RemoteClientUnavailable => Some(StorageError::RemoteClientUnavailable(Default::default())),
}
}
}