refactor(ecstore): make store-to-disk error narrowing a named fallible operation
Backlog#1845 step 4. The blanket impl From<StorageError> for DiskError let ? silently push store-only errors (locks, buckets, quotas) across the disk boundary into DiskError::other, where the rendered message fragments reduce_errs quorum buckets. Same story for the blanket From<StorageError> for rustfs_filemeta::Error and its other() catch-all.
Both impls are replaced by named, fallible methods: StorageError::narrow_to_disk() and StorageError::narrow_to_filemeta(). Variants with an identity on the far side map across unchanged - including the two documented lossy collapses (SlowDown -> TooManyOpenFiles, StorageFull -> DiskFull) that the round-trip tests pin - and everything else returns Err(self) so the call site decides what crossing the boundary means. Removing the impls let the compiler enumerate every conversion site; the census that scoped this issue had found 5, the compiler found 33.
Call sites keep their existing behavior: the io identity bridge and the generic sites fold Err into the io-backed other() exactly as the old catch-all did (identity still recoverable by downcast), listing paths use one shared to_filemeta_err helper, and the two sites that relied on the SlowDown collapse now construct DiskError::TooManyOpenFiles directly so the loss is visible where it happens. No behavior change intended anywhere; the io::Error bridge itself is untouched by design.
Ref rustfs/backlog#1845
* feat(ecstore): type internode client-acquisition failures for stable quorum buckets
Backlog#1845 step 3, first typed family. The largest other(format!) message family in ecstore was 'can not get client, err: {detail}' (~50 production sites): every internode RPC that fails to acquire a client wrapped the dial/auth error with per-peer detail into DiskError::other / StorageError::other, whose Io equality compares the rendered message. N disks failing for this same cause therefore counted as N distinct errors in reduce_errs, starving quorum aggregation, and remote_disk call sites double-wrapped the message on top of get_client's own wrap.
Introduce DiskError::RemoteClientUnavailable(String) (wire code 0x2B) and its StorageError twin (StorageErrorCode 0x54): equality and hashing use the wire code alone, so same-cause failures land in one quorum bucket regardless of per-peer detail, while Display keeps the detail so substring classifiers (network needles, heal recoverability) keep reading it unchanged. Wire encoding carries the rendered detail in error_info and decode restores the typed variant; old peers fall back to the legacy string form gracefully.
Call sites: remote_disk get_client/get_bulk_client/offline-bypass/recovery-probe now construct the typed variant and the ~60 redundant double-wrap map_errs are gone; peer_rest_client's three client getters and offline gates, peer_s3_client, and admin_server_info follow. The tier-config-reload connection classifier's anchored 'can not get client' substring check becomes a typed match on the variant (the string form is retired and now classifies as Terminal, pinned by test).
Ref rustfs/backlog#1845
* chore(ci): refresh error other ratchet baseline
* fix(ecstore): classify typed client network failures
Backlog#1845 step 1 (pure tests, no behavior change): pin the current behavior of every conversion seam an error crosses before heal, replication, or quorum aggregation classifies it, so the later typed-variant and narrow_to_disk() refactors change these expectations deliberately rather than silently.
Covered seams: DiskError <-> StorageError, DiskError <-> node_service wire Error, DiskError/StorageError <-> io::Error (the by-design identity bridge), and StorageError <-> rustfs_filemeta::Error.
Documented lossy edges pinned as-is: SlowDown collapses to TooManyOpenFiles across the disk boundary (StorageFull to DiskFull likewise), the wire Io catch-all re-wraps the rendered message on every hop and drops the io::ErrorKind, and other(format!) messages with per-disk detail fragment reduce_errs quorum buckets while identical messages still bucket together.
Ref rustfs/backlog#1845
* fix(ecstore): handle metadata-less bucket residue
Diagnose metadata-less on-disk residue before non-force DeleteBucket reaches physical deletion, and keep scanner-discovered metadata-missing objects on a non-destructive heal path.
Add explicit heal --remove cleanup for unversioned metadata-less data directories, using the existing data-dir delete primitive and fail-closed shape checks so pre-commit or unknown residue is preserved.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(connect): adapt offline array validator
Wrap the filesystem summary validator in a closure so Option::is_some_and can pass the concrete array reference accepted by serde_json::Value::as_array.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(connect): remove redundant offline test clones
Move the temporary path into the swap closure after deriving the output path, keeping clippy's redundant-clone lint clean for offline bundle tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
The StorageError -> DiskError conversion dropped seven variants with
exact DiskError counterparts (FaultyRemoteDisk, DiskAccessDenied,
DriveIsRoot, IsNotRegular, VolumeNotEmpty, VolumeAccessDenied,
FileAccessDenied) into the DiskError::other fallback, degrading them to
an opaque Io error. Two of them sit on the quorum ignore-lists in
disk/error_reduce.rs, so a degraded instance would stop matching the
ignore list and count toward the dominant error in reduce_errs.
Also mirror the StorageError-side io::Error downdrill in
From<io::Error> for DiskError: recover a StorageError boxed through
From<StorageError> for io::Error instead of wrapping it as Io.
Add a round-trip identity test over every DiskError variant
(DiskError -> StorageError -> DiskError) and a boxed-StorageError
recovery test.
* fix(ecstore): add fallible erasure construction
(cherry picked from commit bd148b20f7)
* fix(ecstore): resolve storage parity per pool
(cherry picked from commit c05c2cb24b)
* fix(ecstore): keep carved per-pool parity core self-contained on main
Fixups so the cherry-picked fallible-erasure + per-pool-parity core builds standalone on current main without the excluded scope-creep commits:
- runtime/sources.rs: re-add backend_storage_class_parities (removed by the per-pool commit; its rebalance caller was updated in an unrelated reporting commit that was left out). Reimplemented over the snapshot API, behavior-identical.
- config/mod.rs: rename the storage-class publish test module (main independently added a mod tests, so the cherry-pick collided).
- rustfs storage_api.rs + startup_storage.rs: route the storage-class ENV consts through the startup storage facade and use a local const for the erasure-set-drive-count env name, satisfying the layer/facade guardrail (main's guardrail is stricter than when the core was authored).
* fix(ecstore): use struct-init in erasure test helper to satisfy clippy field_reassign_with_default
The cherry-picked fallible-erasure commit's `erasure_with_invalid_dimensions` test helper built `Erasure` via `default()` then reassigned fields, which trips `clippy::field_reassign_with_default` under `-D warnings` (only surfaced by `--all-targets`, which lints test code). #4977 fixed this in a later commit that was not part of the carved core. Use struct-init with `..Default::default()`, matching #4977's final form.
* fix(rustfs): gate the test-only storage-class ENV facade re-export behind cfg(test)
The ENV constants (INLINE_BLOCK_ENV/OPTIMIZE_ENV/RRS_ENV/STANDARD_ENV) re-exported through the startup storage facade are only consumed by a #[cfg(test)] test in startup_storage.rs, so in a non-test lib build the re-export is unused and trips -D unused-imports under clippy --all-targets. Gate it with #[cfg(test)], matching #4977's final form.
---------
Co-authored-by: cxymds <cxymds@gmail.com>
is_all_not_found and is_all_volume_not_found delegated to
is_err_bucket_not_found, which matches StorageError::DiskNotFound. When
every erasure set in every pool was unreachable, the per-set aggregate
DiskNotFound slice was classified as "all not found", so list_merged
returned Ok(empty) and walk_result_from_set_errors was fed an
availability failure disguised as an empty listing. Clients saw a
successful empty ListObjects and mistook a full outage for an empty
bucket.
Introduce is_err_strict_not_found (FileNotFound / VolumeNotFound /
FileVersionNotFound / ObjectNotFound / VersionNotFound, excluding
DiskNotFound) for is_all_not_found, mirroring MinIO's isAllNotFound and
DiskError::is_all_not_found. Give is_all_volume_not_found its own
is_err_strict_volume_not_found (VolumeNotFound / BucketNotFound, minus
DiskNotFound) so genuine volume/bucket absence still surfaces while a
missing object under an existing volume is not escalated. Add
is_all_disk_not_found and a pre-check in walk_result_from_set_errors so
an all-offline slice surfaces DiskNotFound instead of being swallowed as
an empty listing, while partial outages (a healthy set present) stay
tolerated as before.
Regression tests cover all-DiskNotFound rejection, genuine not-found
acceptance, volume-not-found semantics, partial/empty short-circuit, and
the walk full-offline vs partial-offline paths.
Co-authored-by: heihutu <heihutu@gmail.com>
test: add insta snapshot test for storage error display format
Add snapshot test to detect unexpected changes in StorageError
display format. This catches output format regressions that
traditional assert tests might miss.
Refs #740
cxymds