feat(ecstore): type internode client-acquisition failures for quorum buckets (#6619)

* feat(ecstore): type internode client-acquisition failures for stable quorum buckets

Backlog#1845 step 3, first typed family. The largest other(format!) message family in ecstore was 'can not get client, err: {detail}' (~50 production sites): every internode RPC that fails to acquire a client wrapped the dial/auth error with per-peer detail into DiskError::other / StorageError::other, whose Io equality compares the rendered message. N disks failing for this same cause therefore counted as N distinct errors in reduce_errs, starving quorum aggregation, and remote_disk call sites double-wrapped the message on top of get_client's own wrap.

Introduce DiskError::RemoteClientUnavailable(String) (wire code 0x2B) and its StorageError twin (StorageErrorCode 0x54): equality and hashing use the wire code alone, so same-cause failures land in one quorum bucket regardless of per-peer detail, while Display keeps the detail so substring classifiers (network needles, heal recoverability) keep reading it unchanged. Wire encoding carries the rendered detail in error_info and decode restores the typed variant; old peers fall back to the legacy string form gracefully.

Call sites: remote_disk get_client/get_bulk_client/offline-bypass/recovery-probe now construct the typed variant and the ~60 redundant double-wrap map_errs are gone; peer_rest_client's three client getters and offline gates, peer_s3_client, and admin_server_info follow. The tier-config-reload connection classifier's anchored 'can not get client' substring check becomes a typed match on the variant (the string form is retired and now classifies as Terminal, pinned by test).

Ref rustfs/backlog#1845

* chore(ci): refresh error other ratchet baseline

* fix(ecstore): classify typed client network failures
This commit is contained in:
Zhengchao An
2026-08-26 11:24:56 +08:00
committed by GitHub
parent 7cac528de3
commit c0c208d89a
9 changed files with 176 additions and 147 deletions
+4
View File
@@ -104,6 +104,7 @@ pub enum StorageErrorCode {
Timeout,
InvalidPath,
QuotaExceeded,
RemoteClientUnavailable,
}
impl StorageErrorCode {
@@ -190,6 +191,7 @@ impl StorageErrorCode {
Self::Timeout => 0x51,
Self::InvalidPath => 0x52,
Self::QuotaExceeded => 0x53,
Self::RemoteClientUnavailable => 0x54,
}
}
@@ -276,6 +278,7 @@ impl StorageErrorCode {
0x51 => Some(Self::Timeout),
0x52 => Some(Self::InvalidPath),
0x53 => Some(Self::QuotaExceeded),
0x54 => Some(Self::RemoteClientUnavailable),
_ => None,
}
}
@@ -351,6 +354,7 @@ mod tests {
(StorageErrorCode::OperationCanceled, 0x41),
(StorageErrorCode::NamespaceLockQuorumUnavailable, 0x42),
(StorageErrorCode::QuotaExceeded, 0x53),
(StorageErrorCode::RemoteClientUnavailable, 0x54),
];
const DISK_PRESERVATION_ERROR_CODES: &[(StorageErrorCode, u32)] = &[