* fix(filemeta): add state-aware file info validation
* fix(filemeta): validate shard arithmetic and delete paths
* fix(ecstore): add fallible erasure construction
* fix(ecstore): resolve storage parity per pool
* fix(storage): report heterogeneous erasure layouts
* fix(admin): publish prepared storage config atomically
* fix(storage): harden per-pool parity boundaries
* fix(storage): address pre-PR validation findings
* test(ci): fix strict-topology validation fixtures
* fix(heal): preserve delete markers during repair
* refactor(filemeta): drop unused ValidatedFileInfo witness
ValidatedFileInfo wrapped an unread `_file_info` reference alongside an `Option<ValidatedErasureLayout>`, but only the layout was ever consumed. Return the layout directly from `FileInfo::validate` so the sole production consumer (`LocalDisk::check_parts`) and the two unit tests read it without the extra witness type and lifetime.
No behavior change.
* fix(filemeta): keep compressed and MinIO-migrated tiered objects readable
The new decode-path validation rejected several legitimate on-disk shapes that older RustFS and MinIO-migrated data carry, turning readable objects into FileCorrupt:
- Compressed objects written with an unknown upload size persist a negative per-part actual_size (the documented "unknown size" sentinel that ObjectInfo::get_actual_size already tolerates). validate_collection_contents rejected it via usize::try_from; now a negative actual_size skips shard validation and only real, non-negative sizes are checked.
- MinIO-migrated objects transitioned to a versioned remote tier store the tier version id as a UUID string, not 16 raw bytes. MetaObject::into_fileinfo returned FileCorrupt (main tolerated it as None), making all versions of the object unreadable; MetaDeleteMarker free-version records took a Some(nil) sentinel path with the same effect, which also breaks free-version expiry (remote-tier leak). Both now decode through a shared transitioned_version_id_from_meta_sys helper: 16 raw bytes or a UUID string are accepted, anything else is tolerated as None instead of failing the read.
Regression tests updated to assert the readable/compat behavior, with new tests covering MinIO string-form recovery.
* fix(scanner): build the delete-marker test fixture without erasure geometry
get_size_counts_delete_markers_separately_from_versions built its delete marker with `FileInfo::new(object, 1, 1)`, which attaches erasure geometry (data=1/parity=1/distribution). This PR classifies versions by shape via `is_storage_delete_marker()` (no geometry) rather than the raw `deleted` flag, so a geometry-bearing "delete marker" is correctly serialized as a purge-pending payload Object and counted as a version — CI saw summary.versions=3, expected 2.
Real delete markers carry no erasure geometry (delete paths build them as `FileInfo { deleted: true, ..Default::default() }`), so construct the fixture the same way. It then classifies as a storage delete marker and the counts (versions=2, delete_markers=1) hold. This keeps the PR's more-correct classification, which prevents a purge-pending object's geometry from being dropped when serialized as a bare delete marker.
* docs(changelog): note per-pool parity fix and storage-class startup upgrade caveat
Records the #4801 per-pool erasure parity fix under Fixed, and documents the upgrade behavior where a persisted storage class that a small or heterogeneous pool cannot satisfy now fails startup — with the RUSTFS_STORAGE_CLASS_STANDARD recovery steps. Docs-only; covers R4 from the on-disk compatibility audit.
* fix(heal): report parity from erasure geometry, not is_valid()
heal_object set HealResultItem.parity_blocks via `if lfi.is_valid()`, which was missed by the migration of the other quorum/metadata predicates. With the new `is_valid()` semantics (full payload validation; delete markers now return false), a delete marker or a geometry-bearing version with a benign collection quirk would misreport parity as the pool default instead of its own. Use `has_valid_erasure_geometry()` — the narrow "does this carry erasure geometry" predicate the rest of the migration uses — so reporting matches the object's actual layout. Reporting-only; no data-path change.
* fix(filemeta): do not silently serialize a non-canonical deleted FileInfo as an Object
`From<FileInfo> for FileMetaVersion` classifies by `is_storage_delete_marker()` (shape), which correctly routes canonical delete markers to Delete and purge-pending payloads (deleted=true with real erasure geometry) to Object. But a `deleted` FileInfo that is neither a canonical marker nor a valid erasure payload would silently serialize as a zero-geometry MetaObject that later fails `validate_for_metadata_read`. Write paths validate first (`validate_for_erasure_write` / `validate_for_metadata_read`), so this is a caller bug; `From` is infallible, so surface it with a structured `warn!` on the malformed branch instead of writing corrupt metadata silently. Legitimate purge-pending objects (valid geometry) are unaffected — the guard only fires for `deleted && !has_valid_erasure_geometry()`.
* test(filemeta): assert real historical xl.meta versions pass metadata-read validation
Empirical companion to the code-reasoned decode-tolerance invariants (docs/architecture/erasure-coding.md §11) and the rolling-upgrade / MinIO-migration compatibility concern: the tightened `validate_for_metadata_read` runs on every local disk read and peer-RPC-decoded FileInfo, so it must accept every version of real historically-written xl.meta, never reject it as FileCorrupt.
Loads five real fixtures — MinIO small-inline, MinIO versioned (two object versions + a delete marker), MinIO large multipart, a legacy V1 (xl.json-derived) object, and a legacy meta_ver 2 object — decodes every version with parts materialized, and asserts validate_for_metadata_read() is Ok for each. Reverting the tolerant handling (delete-marker shape, legacy per-part checksums, string/short transitioned-versionID, negative actual_size) turns this red.
* fix(ci): remove duplicate storage test re-exports
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
B5 switched heal enumeration to list_object_versions, which only reflects the
read-quorum metadata view: a version present on fewer than read-quorum disks was
never enumerated, so it was never healed. Add a per-erasure-set disk-walk UNION
enumerator (mirrors MinIO global-heal.go objQuorum=1 listPathRaw +
mergeXLV2Versions) that surfaces every (object, version) present on ANY disk and
feeds each to the existing per-version heal_object.
- filemeta: MetaCacheEntries::resolve_union (dir_quorum=1/obj_quorum=1) yields the
cross-disk version union at one tested seam.
- ecstore: SetDisks::heal_walk_versions_page (list_path_raw fan-out, min_disks=1,
dual object/version page bound, inclusive-forward de-overlap) + ECStore delegator
+ HealWalkVersion.
- ecstore data-safety guard: before dangling-delete, try_regenerate_recoverable_meta
physically probes part files via check_parts; when >= data_blocks data shards
survive (meta lost but data recoverable) it regenerates xl.meta from a surviving
FileInfo with the correct per-disk shard index instead of dangling-deleting.
Genuine torn writes (< data_blocks) keep the current behavior — no resurrection.
- heal: dw1: forward-marker cursor codec (reuses ResumeState.resume_cursor,
idempotent restart on foreign tokens); list_versions_for_heal_page_disk_walk
trait method (default falls back to the B5 read-quorum path); heal_bucket_with_resume
selects the disk-walk enumerator when scan_mode==Deep || source==AutoHeal, else
the unchanged B5 path; anti-loop guard aborts on (empty && truncated).
Closesrustfs/backlog#920
MetaObject::into_fileinfo indexed part_sizes[i]/part_actual_sizes[i] by
part_numbers.len() without checking the arrays are the same length,
unlike the adjacent part_etags/part_indices which are length-guarded.
decode_from pushes the three arrays independently and the xl.meta CRC
only covers bytes, so a CRC-valid but internally inconsistent xl.meta
(foreign writer / MinIO interop) triggers an out-of-bounds panic on the
GET/HEAD/LIST decode path.
Guard the three arrays for equal length and return Err(FileCorrupt) so a
divergent shard is skipped and quorum uses the other disks, instead of
panicking the request task. Cascade into_fileinfo to Result across its
callers, and fix io_primitives early-return to derive the version id from
the merged header and fall into the per-disk loop (single-disk survival +
heal). The 2118 merge-first path is left as a documented follow-up.
Refs backlog#900 (filemeta-01).
* fix: add tokio-test
* fix: "called `unwrap` on `v` after checking its variant with `is_some`"
= help: try using `if let` or `match`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_unwrap
= note: `-D clippy::unnecessary-unwrap` implied by `-D warnings`
= help: to override `-D warnings` add `#[allow(clippy::unnecessary_unwrap)]`
* fmt
* set toolchain 1.88.0
* fmt
* fix: cliip
* chore: Add copyright and license headers
This commit adds the Apache 2.0 license and a copyright notice to the header of all source files. This ensures that the licensing and copyright information is clearly stated within the codebase.
* cargo fmt
* fix
* fmt
* fix clippy
cxymds