A full refresh with partial disk failures used to commit the surviving
subset's sum as a fresh exact cluster total (no field carried the
partial-failure fact), while the complete disk cache kept the failed
disk's old value — so the reported capacity oscillated between the
partial sum and the cache-merged total on alternating refreshes.
- Add a degraded flag to CapacityUpdate/CachedCapacity, set when the
scan behind the update had partial errors; expose it in refresh logs,
admin capacity logs and a new rustfs_capacity_degraded_readings_total
counter.
- On a degraded full refresh, surface only the disks whose own scan
fully succeeded and merge them over a complete disk cache, so failed
disks keep their last-known values and the published total no longer
dips and bounces back. The disk cache is never replaced from a
degraded refresh.
- Without a complete cache, keep the partial sum (unchanged #805
non-pollution behavior) but mark the reading degraded.
Ref: rustfs/backlog#1014 (S06 from audit rustfs/backlog#1010)
* fix(ecstore): make post-commit old data dir cleanup best-effort (backlog#898)
A write is authoritatively committed once rename_data returns Ok (the new
version is durable on >= write_quorum disks and immediately readable). The
subsequent reclamation of the now-dereferenced old object/<data_dir> is pure
space reclamation, yet commit_rename_data_dir propagated a below-quorum GC
failure via `?` into ErasureWriteQuorum -> 503, producing a false-negative
ACK for an already-persisted write. This is a deliberate divergence from
MinIO (erasure-object.go:1577), which couples the two; the divergence is
justified by durability semantics, not parity.
Changes:
- commit_rename_data_dir now returns a structured OldDataDirCleanup receipt
and never returns Err. Adds an old==committed-dir anti-misdelete guard and
a committed_data_dir parameter. Classification is extracted into pure
functions (classify_old_data_dir_cleanup / map_cleanup_join_result /
is_cleanup_not_found) so it is unit-testable. Task panic/cancel is mapped to
a non-ignored DiskError::other (never DiskNotFound), and not-found is
normalized to reclaimed.
- object.rs / multipart.rs consume the receipt instead of `?`. The result
reverts to Ok, so the invalidate_get_object_metadata_cache self-heal and the
capacity/compression accounting that a `?` early-return previously skipped
now run on the cleanup-failure path too.
- On residue, report_old_data_dir_cleanup emits leak metrics and enqueues an
object heal over the existing heal channel (disk-health signal replacing the
503). heal_object -> reclaim_orphan_data_dirs already reclaims unreferenced
local data dirs, closing the loop end to end.
- Adds rustfs_old_data_dir_* counters (attempted/reclaimed/leaked/below_quorum)
as the operator-visible backstop for leaked residue.
- Adds a test-only (#[cfg(test)]) delete fault-injection seam; in production it
inlines to a no-op None and has no behavioral effect.
Tests: pure-function A/C group + join-error mapping + actions decision; A5/A5b
real-disk guard/reclaim integration; end-to-end overwrite returning 200 while
old-data-dir cleanup fails. #864 rollback guard test remains green.
* fix(ecstore): resolve merge conflicts with origin/main in io_primitives.rs
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Zhengchao An