fix(storage): resolve erasure parity per pool (#4977)

* fix(filemeta): add state-aware file info validation * fix(filemeta): validate shard arithmetic and delete paths * fix(ecstore): add fallible erasure construction * fix(ecstore): resolve storage parity per pool * fix(storage): report heterogeneous erasure layouts * fix(admin): publish prepared storage config atomically * fix(storage): harden per-pool parity boundaries * fix(storage): address pre-PR validation findings * test(ci): fix strict-topology validation fixtures * fix(heal): preserve delete markers during repair * refactor(filemeta): drop unused ValidatedFileInfo witness ValidatedFileInfo wrapped an unread `_file_info` reference alongside an `Option<ValidatedErasureLayout>`, but only the layout was ever consumed. Return the layout directly from `FileInfo::validate` so the sole production consumer (`LocalDisk::check_parts`) and the two unit tests read it without the extra witness type and lifetime. No behavior change. * fix(filemeta): keep compressed and MinIO-migrated tiered objects readable The new decode-path validation rejected several legitimate on-disk shapes that older RustFS and MinIO-migrated data carry, turning readable objects into FileCorrupt: - Compressed objects written with an unknown upload size persist a negative per-part actual_size (the documented "unknown size" sentinel that ObjectInfo::get_actual_size already tolerates). validate_collection_contents rejected it via usize::try_from; now a negative actual_size skips shard validation and only real, non-negative sizes are checked. - MinIO-migrated objects transitioned to a versioned remote tier store the tier version id as a UUID string, not 16 raw bytes. MetaObject::into_fileinfo returned FileCorrupt (main tolerated it as None), making all versions of the object unreadable; MetaDeleteMarker free-version records took a Some(nil) sentinel path with the same effect, which also breaks free-version expiry (remote-tier leak). Both now decode through a shared transitioned_version_id_from_meta_sys helper: 16 raw bytes or a UUID string are accepted, anything else is tolerated as None instead of failing the read. Regression tests updated to assert the readable/compat behavior, with new tests covering MinIO string-form recovery. * fix(scanner): build the delete-marker test fixture without erasure geometry get_size_counts_delete_markers_separately_from_versions built its delete marker with `FileInfo::new(object, 1, 1)`, which attaches erasure geometry (data=1/parity=1/distribution). This PR classifies versions by shape via `is_storage_delete_marker()` (no geometry) rather than the raw `deleted` flag, so a geometry-bearing "delete marker" is correctly serialized as a purge-pending payload Object and counted as a version — CI saw summary.versions=3, expected 2. Real delete markers carry no erasure geometry (delete paths build them as `FileInfo { deleted: true, ..Default::default() }`), so construct the fixture the same way. It then classifies as a storage delete marker and the counts (versions=2, delete_markers=1) hold. This keeps the PR's more-correct classification, which prevents a purge-pending object's geometry from being dropped when serialized as a bare delete marker. * docs(changelog): note per-pool parity fix and storage-class startup upgrade caveat Records the #4801 per-pool erasure parity fix under Fixed, and documents the upgrade behavior where a persisted storage class that a small or heterogeneous pool cannot satisfy now fails startup — with the RUSTFS_STORAGE_CLASS_STANDARD recovery steps. Docs-only; covers R4 from the on-disk compatibility audit. * fix(heal): report parity from erasure geometry, not is_valid() heal_object set HealResultItem.parity_blocks via `if lfi.is_valid()`, which was missed by the migration of the other quorum/metadata predicates. With the new `is_valid()` semantics (full payload validation; delete markers now return false), a delete marker or a geometry-bearing version with a benign collection quirk would misreport parity as the pool default instead of its own. Use `has_valid_erasure_geometry()` — the narrow "does this carry erasure geometry" predicate the rest of the migration uses — so reporting matches the object's actual layout. Reporting-only; no data-path change. * fix(filemeta): do not silently serialize a non-canonical deleted FileInfo as an Object `From<FileInfo> for FileMetaVersion` classifies by `is_storage_delete_marker()` (shape), which correctly routes canonical delete markers to Delete and purge-pending payloads (deleted=true with real erasure geometry) to Object. But a `deleted` FileInfo that is neither a canonical marker nor a valid erasure payload would silently serialize as a zero-geometry MetaObject that later fails `validate_for_metadata_read`. Write paths validate first (`validate_for_erasure_write` / `validate_for_metadata_read`), so this is a caller bug; `From` is infallible, so surface it with a structured `warn!` on the malformed branch instead of writing corrupt metadata silently. Legitimate purge-pending objects (valid geometry) are unaffected — the guard only fires for `deleted && !has_valid_erasure_geometry()`. * test(filemeta): assert real historical xl.meta versions pass metadata-read validation Empirical companion to the code-reasoned decode-tolerance invariants (docs/architecture/erasure-coding.md §11) and the rolling-upgrade / MinIO-migration compatibility concern: the tightened `validate_for_metadata_read` runs on every local disk read and peer-RPC-decoded FileInfo, so it must accept every version of real historically-written xl.meta, never reject it as FileCorrupt. Loads five real fixtures — MinIO small-inline, MinIO versioned (two object versions + a delete marker), MinIO large multipart, a legacy V1 (xl.json-derived) object, and a legacy meta_ver 2 object — decodes every version with parts materialized, and asserts validate_for_metadata_read() is Ok for each. Reverting the tolerant handling (delete-marker shape, legacy per-part checksums, string/short transitioned-versionID, negative actual_size) turns this red. * fix(ci): remove duplicate storage test re-exports --------- Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-07-26 08:18:18 +00:00 · 2026-07-19 21:52:31 +08:00
parent 21049401fa
commit b0c6c4cbce
30 changed files with 4613 additions and 427 deletions
@@ -50,6 +50,7 @@ const LOG_SUBSYSTEM_METRICS_COLLECTOR: &str = "metrics_collector";
 const EVENT_METRICS_COLLECTOR_STATE: &str = "metrics_collector_state";

 type ObsStorageInfo = <ObsStore as StorageAdminApi>::StorageInfo;
+type ObsBackendInfo = <ObsStore as StorageAdminApi>::BackendInfo;

 struct ObsDataUsageInfo {
    last_update: Option<SystemTime>,
@@ -789,43 +790,86 @@ pub fn collect_internode_network_stats() -> Option<NetworkStats> {
 }

 /// Collect cluster config metrics from backend parity configuration.
+fn cluster_config_stats_from_backend_parities(
+    rr_sc_parity: Option<usize>,
+    standard_sc_parity: Option<usize>,
+) -> Option<ClusterConfigStats> {
+    Some(ClusterConfigStats {
+        rrs_parity: u32::try_from(rr_sc_parity?).ok()?,
+        standard_parity: u32::try_from(standard_sc_parity?).ok()?,
+    })
+}
+
 pub async fn collect_cluster_config_stats() -> Option<ClusterConfigStats> {
    let store = resolve_obs_object_store_handle()?;
    let backend = StorageAdminApi::backend_info(store.as_ref()).await;

-    Some(ClusterConfigStats {
-        rrs_parity: backend.rr_sc_parity.unwrap_or_default() as u32,
-        standard_parity: backend.standard_sc_parity.unwrap_or_default() as u32,
-    })
+    cluster_config_stats_from_backend_parities(backend.rr_sc_parity, backend.standard_sc_parity)
 }

-/// Collect cluster erasure set metrics from storage and backend topology info.
-pub async fn collect_erasure_set_stats() -> Vec<ErasureSetStats> {
-    let Some(store) = resolve_obs_object_store_handle() else {
-        return Vec::new();
+fn standard_erasure_layout_from_backend(backend: &ObsBackendInfo, pool_idx: usize) -> Option<(usize, usize)> {
+    let drives_per_set = backend.drives_per_set.get(pool_idx).copied()?;
+    if drives_per_set == 0
+        || (!backend.standard_sc_data.is_empty() && backend.standard_sc_data.len() != backend.drives_per_set.len())
+        || (!backend.standard_sc_parities.is_empty() && backend.standard_sc_parities.len() != backend.drives_per_set.len())
+    {
+        return None;
+    }
+
+    let has_data = !backend.standard_sc_data.is_empty();
+    let has_parities = !backend.standard_sc_parities.is_empty();
+    let (data, parity) = match (has_data, has_parities) {
+        (true, true) => (
+            backend.standard_sc_data.get(pool_idx).copied()?,
+            backend.standard_sc_parities.get(pool_idx).copied()?,
+        ),
+        (true, false) => {
+            let data = backend.standard_sc_data.get(pool_idx).copied()?;
+            (data, drives_per_set.checked_sub(data)?)
+        }
+        (false, true) => {
+            let parity = backend.standard_sc_parities.get(pool_idx).copied()?;
+            (drives_per_set.checked_sub(parity)?, parity)
+        }
+        (false, false) => {
+            let parity = backend.standard_sc_parity?;
+            (drives_per_set.checked_sub(parity)?, parity)
+        }
    };

-    let storage_info = StorageAdminApi::storage_info(store.as_ref()).await;
-    let backend = StorageAdminApi::backend_info(store.as_ref()).await;
+    if data == 0 || parity > data || data.checked_add(parity) != Some(drives_per_set) {
+        return None;
+    }
+
+    Some((data, parity))
+}
+
+fn erasure_set_stats_from_backend(storage_info: &ObsStorageInfo, backend: &ObsBackendInfo) -> Vec<ErasureSetStats> {
    let mut grouped: HashMap<(usize, usize), ErasureSetStats> = HashMap::new();

    for disk in &storage_info.disks {
-        let pool_idx = disk.pool_index.max(0) as usize;
-        let set_idx = disk.set_index.max(0) as usize;
-        let set_drive_count = backend.drives_per_set.get(pool_idx).copied().unwrap_or_default();
-        let parity = backend
-            .standard_sc_parities
-            .get(pool_idx)
-            .copied()
-            .or(backend.standard_sc_parity)
-            .unwrap_or(set_drive_count / 2);
+        let (Ok(pool_idx), Ok(set_idx)) = (usize::try_from(disk.pool_index), usize::try_from(disk.set_index)) else {
+            continue;
+        };
+        let Some((_, parity)) = standard_erasure_layout_from_backend(backend, pool_idx) else {
+            continue;
+        };
+        let set_drive_count = backend.drives_per_set[pool_idx];
+        let (Ok(pool_id), Ok(set_id), Ok(size), Ok(parity_metric)) = (
+            u32::try_from(pool_idx),
+            u32::try_from(set_idx),
+            u32::try_from(set_drive_count),
+            u32::try_from(parity),
+        ) else {
+            continue;
+        };
        let quorum_shape = derive_erasure_set_quorum_shape(set_drive_count, parity);

        let entry = grouped.entry((pool_idx, set_idx)).or_insert_with(|| ErasureSetStats {
-            pool_id: pool_idx as u32,
-            set_id: set_idx as u32,
-            size: set_drive_count as u32,
-            parity: parity as u32,
+            pool_id,
+            set_id,
+            size,
+            parity: parity_metric,
            data_shards: quorum_shape.data_shards,
            read_quorum: quorum_shape.read_quorum,
            write_quorum: quorum_shape.write_quorum,
@@ -855,6 +899,17 @@ pub async fn collect_erasure_set_stats() -> Vec<ErasureSetStats> {
    stats
 }

+/// Collect cluster erasure set metrics from storage and backend topology info.
+pub async fn collect_erasure_set_stats() -> Vec<ErasureSetStats> {
+    let Some(store) = resolve_obs_object_store_handle() else {
+        return Vec::new();
+    };
+
+    let storage_info = StorageAdminApi::storage_info(store.as_ref()).await;
+    let backend = StorageAdminApi::backend_info(store.as_ref()).await;
+    erasure_set_stats_from_backend(&storage_info, &backend)
+}
+
 pub async fn collect_iam_stats() -> Option<IamStats> {
    let snapshot = iam_metrics_snapshot()?;

@@ -1119,6 +1174,100 @@ mod tests {
    use std::thread;
    use std::time::Duration;

+    fn storage_info_with_one_online_disk() -> ObsStorageInfo {
+        let mut info = ObsStorageInfo::default();
+        info.disks.push(Default::default());
+        let disk = info.disks.last_mut().expect("inserted disk should exist");
+        disk.pool_index = 0;
+        disk.set_index = 0;
+        disk.disk_index = 0;
+        disk.state = DRIVE_STATE_OK.to_string();
+        disk.runtime_state = Some(DRIVE_STATE_ONLINE.to_string());
+        info
+    }
+
+    #[test]
+    fn cluster_config_stats_accept_homogeneous_backend_parities() {
+        let stats = cluster_config_stats_from_backend_parities(Some(1), Some(2))
+            .expect("homogeneous scalar parities should produce cluster config metrics");
+
+        assert_eq!(stats.rrs_parity, 1);
+        assert_eq!(stats.standard_parity, 2);
+    }
+
+    #[test]
+    fn cluster_config_stats_skip_heterogeneous_backend_parities() {
+        assert!(cluster_config_stats_from_backend_parities(Some(1), None).is_none());
+        assert!(cluster_config_stats_from_backend_parities(None, Some(2)).is_none());
+    }
+
+    #[test]
+    fn cluster_config_stats_reject_parity_larger_than_u32() {
+        let Ok(overflow) = usize::try_from(u64::from(u32::MAX) + 1) else {
+            return;
+        };
+
+        assert!(cluster_config_stats_from_backend_parities(Some(overflow), Some(2)).is_none());
+        assert!(cluster_config_stats_from_backend_parities(Some(1), Some(overflow)).is_none());
+    }
+
+    #[test]
+    fn erasure_set_stats_skip_unknown_backend_layout() {
+        let storage_info = storage_info_with_one_online_disk();
+        let backend = ObsBackendInfo {
+            drives_per_set: vec![8],
+            ..Default::default()
+        };
+
+        assert!(erasure_set_stats_from_backend(&storage_info, &backend).is_empty());
+    }
+
+    #[test]
+    fn erasure_set_stats_skip_inconsistent_exact_layout_without_scalar_fallback() {
+        let storage_info = storage_info_with_one_online_disk();
+        let backend = ObsBackendInfo {
+            standard_sc_data: vec![6],
+            standard_sc_parities: vec![4],
+            standard_sc_parity: Some(2),
+            drives_per_set: vec![8],
+            ..Default::default()
+        };
+
+        assert!(erasure_set_stats_from_backend(&storage_info, &backend).is_empty());
+    }
+
+    #[test]
+    fn erasure_set_stats_accept_truthful_legacy_scalar_layout() {
+        let storage_info = storage_info_with_one_online_disk();
+        let backend = ObsBackendInfo {
+            standard_sc_parity: Some(2),
+            drives_per_set: vec![8],
+            ..Default::default()
+        };
+
+        let stats = erasure_set_stats_from_backend(&storage_info, &backend);
+        assert_eq!(stats.len(), 1);
+        assert_eq!(stats[0].size, 8);
+        assert_eq!(stats[0].parity, 2);
+        assert_eq!(stats[0].data_shards, 6);
+    }
+
+    #[test]
+    fn erasure_set_stats_exact_data_ignores_stale_scalar() {
+        let storage_info = storage_info_with_one_online_disk();
+        let backend = ObsBackendInfo {
+            standard_sc_data: vec![6],
+            standard_sc_parity: Some(4),
+            drives_per_set: vec![8],
+            ..Default::default()
+        };
+
+        let stats = erasure_set_stats_from_backend(&storage_info, &backend);
+        assert_eq!(stats.len(), 1);
+        assert_eq!(stats[0].parity, 2);
+        assert_eq!(stats[0].data_shards, 6);
+    }
+
    fn generate_loopback_traffic() -> std::io::Result<()> {
        let listener = TcpListener::bind(("127.0.0.1", 0))?;
        let addr = listener.local_addr()?;