fix(core-storage): fix critical correctness defects from core-storage reliability audit (#4222)

* fix(core-storage): fix critical correctness defects from core-storage audit

Fixes verified defects found in a deep audit of the core storage path
(erasure coding, disk persistence, quorum, heal, replication resync):

- ecstore/disk: rewrite live xl.meta atomically (temp+rename) in
  delete_versions_internal and write_metadata instead of in-place
  truncate, which exposed torn metadata to concurrent readers and
  crashes on the DeleteObjects hot path
- ecstore/erasure: allow heal to reconstruct from exactly data_shards
  bitrot-verified sources; requiring data_shards+1 made objects
  permanently unhealable after losing parity_shards disks
- ecstore/set_disk: direct-memory inline GET applied the erasure
  distribution permutation twice (shuffled inputs re-indexed through
  distribution), concatenating wrong shards into the response body in
  degraded reads; collect from canonical disk-ordered inputs
- ecstore/set_disk: heal now preserves the committed inline layout
  instead of recomputing it with a hardcoded unversioned threshold,
  which split quorum identity of healed replicas and caused endless
  re-heal churn
- ecstore/replication: resync results channel switched from
  broadcast(1) to mpsc; a lagged broadcast receiver ended the stats
  collector and every subsequent failure went uncounted, letting
  failed resyncs be marked completed
- ecstore/replication: ignore an empty persisted resync checkpoint;
  resuming with one skipped every object and marked the resync
  completed without replicating anything
- ecstore/replication: fix inverted not-found error classification in
  replicate_object/replicate_delete logging paths
- ecstore/erasure: guard decode paths against zero block_size or
  data_shards from corrupt on-disk metadata (divide-by-zero panic)
- ecstore/disk: os::read_dir no longer consumes the entry limit on
  entries it does not return (is_empty_dir misjudgment); create_file
  opens with O_TRUNC to avoid stale trailing bytes
- filemeta: treat Some(nil) version id as a null version in
  matches_not_strict; disk-loaded headers never store None, so the
  mod_time quorum guard for unversioned overwrites never fired and an
  interrupted overwrite could displace the committed version in merge
- filemeta: fix msgpack skip lengths for fixext (missed the ext type
  byte) and ext16/32 (over-skipped) unknown fields
- filemeta: return FileCorrupt instead of usize underflow when
  xl.meta is truncated inside the CRC trailer
- filemeta: surface delete-marker insertion failure in delete_version
  instead of reporting success when the data dir is shared

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(replication): drop duplicate cfg(test) etag import from boundary module

The test module already imports content_matches_by_etag locally, so the
top-level cfg(test) import is unused under -D warnings and fails clippy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

crates/ecstore/src/erasure/coding/decode.rs

@@ -1130,6 +1130,13 @@ impl Erasure {
             return (0, Some(io::Error::new(ErrorKind::InvalidInput, "Invalid number of readers")));
         }
 
+        // block_size/data_shards come from on-disk metadata; a corrupt FileInfo with a
+        // zero here must surface as an error, not a divide-by-zero panic on every GET.
+        if self.block_size == 0 || self.data_shards == 0 {
+            record_get_object_pipeline_failure(GET_STAGE_RANGE, GetObjectFailureReason::RangeOrLengthInvalid);
+            return (0, Some(io::Error::new(ErrorKind::InvalidInput, "Invalid erasure coding parameters")));
+        }
+
         let Some(end_offset) = offset.checked_add(length) else {
             record_get_object_pipeline_failure(GET_STAGE_RANGE, GetObjectFailureReason::RangeOrLengthInvalid);
             return (0, Some(io::Error::new(ErrorKind::InvalidInput, "offset + length exceeds total length")));

crates/ecstore/src/erasure/coding/heal.rs

@@ -143,8 +143,11 @@ impl super::Erasure {
             let shard_size = self.shard_size().min(shard_file_size.saturating_sub(shard_offset));
             let (mut shards, errs) = read_heal_shards(&mut readers, shard_size, read_timeout).await;
 
-            // Data reads may use the first read quorum, but heal writes must only
-            // proceed when the source set is strong enough to validate itself.
+            // Every source shard is already bitrot-verified by its reader, so any
+            // data_shards survivors are sufficient to reconstruct — requiring more
+            // would make objects unhealable after losing exactly parity_shards disks,
+            // the failure EC is sized to tolerate. The parity cross-checks below stay
+            // opportunistic: they run whenever surplus source shards exist.
             let available_shards = errs.iter().filter(|e| e.is_none()).count();
             if available_shards < self.data_shards {
                 warn!(
@@ -157,19 +160,6 @@ impl super::Erasure {
                 return Err(Error::ErasureReadQuorum);
             }
 
-            let missing_data_source = shards.iter().take(self.data_shards).any(|shard| shard.is_none());
-            let required_shards = if missing_data_source && self.parity_shards > 0 {
-                self.data_shards + 1
-            } else {
-                self.data_shards
-            };
-            if available_shards < required_shards {
-                return Err(Error::other(format!(
-                    "can not reconstruct data: not enough verified heal source shards (need {}, have {}) {errs:?}",
-                    required_shards, available_shards
-                )));
-            }
-
             let source_parity = shards
                 .iter()
                 .enumerate()