mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-07 22:03:14 +00:00
7d96ffd7fbba82bb2bac2ca046bb4ea7c2a4b7f5
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 cxymds
|
b0c6c4cbce |
fix(storage): resolve erasure parity per pool (#4977)
* fix(filemeta): add state-aware file info validation
* fix(filemeta): validate shard arithmetic and delete paths
* fix(ecstore): add fallible erasure construction
* fix(ecstore): resolve storage parity per pool
* fix(storage): report heterogeneous erasure layouts
* fix(admin): publish prepared storage config atomically
* fix(storage): harden per-pool parity boundaries
* fix(storage): address pre-PR validation findings
* test(ci): fix strict-topology validation fixtures
* fix(heal): preserve delete markers during repair
* refactor(filemeta): drop unused ValidatedFileInfo witness
ValidatedFileInfo wrapped an unread `_file_info` reference alongside an `Option<ValidatedErasureLayout>`, but only the layout was ever consumed. Return the layout directly from `FileInfo::validate` so the sole production consumer (`LocalDisk::check_parts`) and the two unit tests read it without the extra witness type and lifetime.
No behavior change.
* fix(filemeta): keep compressed and MinIO-migrated tiered objects readable
The new decode-path validation rejected several legitimate on-disk shapes that older RustFS and MinIO-migrated data carry, turning readable objects into FileCorrupt:
- Compressed objects written with an unknown upload size persist a negative per-part actual_size (the documented "unknown size" sentinel that ObjectInfo::get_actual_size already tolerates). validate_collection_contents rejected it via usize::try_from; now a negative actual_size skips shard validation and only real, non-negative sizes are checked.
- MinIO-migrated objects transitioned to a versioned remote tier store the tier version id as a UUID string, not 16 raw bytes. MetaObject::into_fileinfo returned FileCorrupt (main tolerated it as None), making all versions of the object unreadable; MetaDeleteMarker free-version records took a Some(nil) sentinel path with the same effect, which also breaks free-version expiry (remote-tier leak). Both now decode through a shared transitioned_version_id_from_meta_sys helper: 16 raw bytes or a UUID string are accepted, anything else is tolerated as None instead of failing the read.
Regression tests updated to assert the readable/compat behavior, with new tests covering MinIO string-form recovery.
* fix(scanner): build the delete-marker test fixture without erasure geometry
get_size_counts_delete_markers_separately_from_versions built its delete marker with `FileInfo::new(object, 1, 1)`, which attaches erasure geometry (data=1/parity=1/distribution). This PR classifies versions by shape via `is_storage_delete_marker()` (no geometry) rather than the raw `deleted` flag, so a geometry-bearing "delete marker" is correctly serialized as a purge-pending payload Object and counted as a version — CI saw summary.versions=3, expected 2.
Real delete markers carry no erasure geometry (delete paths build them as `FileInfo { deleted: true, ..Default::default() }`), so construct the fixture the same way. It then classifies as a storage delete marker and the counts (versions=2, delete_markers=1) hold. This keeps the PR's more-correct classification, which prevents a purge-pending object's geometry from being dropped when serialized as a bare delete marker.
* docs(changelog): note per-pool parity fix and storage-class startup upgrade caveat
Records the #4801 per-pool erasure parity fix under Fixed, and documents the upgrade behavior where a persisted storage class that a small or heterogeneous pool cannot satisfy now fails startup — with the RUSTFS_STORAGE_CLASS_STANDARD recovery steps. Docs-only; covers R4 from the on-disk compatibility audit.
* fix(heal): report parity from erasure geometry, not is_valid()
heal_object set HealResultItem.parity_blocks via `if lfi.is_valid()`, which was missed by the migration of the other quorum/metadata predicates. With the new `is_valid()` semantics (full payload validation; delete markers now return false), a delete marker or a geometry-bearing version with a benign collection quirk would misreport parity as the pool default instead of its own. Use `has_valid_erasure_geometry()` — the narrow "does this carry erasure geometry" predicate the rest of the migration uses — so reporting matches the object's actual layout. Reporting-only; no data-path change.
* fix(filemeta): do not silently serialize a non-canonical deleted FileInfo as an Object
`From<FileInfo> for FileMetaVersion` classifies by `is_storage_delete_marker()` (shape), which correctly routes canonical delete markers to Delete and purge-pending payloads (deleted=true with real erasure geometry) to Object. But a `deleted` FileInfo that is neither a canonical marker nor a valid erasure payload would silently serialize as a zero-geometry MetaObject that later fails `validate_for_metadata_read`. Write paths validate first (`validate_for_erasure_write` / `validate_for_metadata_read`), so this is a caller bug; `From` is infallible, so surface it with a structured `warn!` on the malformed branch instead of writing corrupt metadata silently. Legitimate purge-pending objects (valid geometry) are unaffected — the guard only fires for `deleted && !has_valid_erasure_geometry()`.
* test(filemeta): assert real historical xl.meta versions pass metadata-read validation
Empirical companion to the code-reasoned decode-tolerance invariants (docs/architecture/erasure-coding.md §11) and the rolling-upgrade / MinIO-migration compatibility concern: the tightened `validate_for_metadata_read` runs on every local disk read and peer-RPC-decoded FileInfo, so it must accept every version of real historically-written xl.meta, never reject it as FileCorrupt.
Loads five real fixtures — MinIO small-inline, MinIO versioned (two object versions + a delete marker), MinIO large multipart, a legacy V1 (xl.json-derived) object, and a legacy meta_ver 2 object — decodes every version with parts materialized, and asserts validate_for_metadata_read() is Ok for each. Reverting the tolerant handling (delete-marker shape, legacy per-part checksums, string/short transitioned-versionID, negative actual_size) turns this red.
* fix(ci): remove duplicate storage test re-exports
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
|
||
|
Zhengchao An
|
7f5873dac8 |
fix(ecstore): resolve erasure parity per pool (#4801) (#5015)
* fix(ecstore): add fallible erasure construction (cherry picked from commit |
||
|
安正超
|
ca58d7f0ec | refactor(config): migrate server config consumers (#3353) | ||
|
houseme
|
8577bd825e |
feat(admin): restore config admin compatibility (#3133)
* feat(admin): restore config admin compatibility Co-authored-by: weisd <im@weisd.in> * fix(admin): align config admin clean rebuild Co-authored-by: weisd <im@weisd.in> * fix(admin): align config history and peer signals * fix(admin): harden config admin mutations * fix(admin): tighten config review follow-ups * perf(admin): reuse env snapshot in config render * fix(ecstore): clean up config admin and listing error handling Remove redundant is_all_volume_not_found check in list_merged, add storage class encode/decode roundtrip tests, fresh boot integration test, and config admin clean rebuild improvements. Co-authored-by: hehutu <heihutu@gmail.com> * fix(admin): sync global server config on mutation and reload Change GLOBAL_SERVER_CONFIG from OnceLock to RwLock so config mutations (set/del/restore/reload) are visible to readers without restart. Call set_global_server_config after every store save and on snapshot reload. Register storage_class as a dynamic config subsystem. Co-authored-by: hehutu <heihutu@gmail.com> * style: apply rustfmt to config and admin tests Co-authored-by: hehutu <heihutu@gmail.com> * fix(test): update signal_service test for storage_class dynamic subsystem storage_class is now a valid dynamic config subsystem, so the "requires object layer" test should expect "storage layer not initialized" instead of "unsupported dynamic config subsystem". Co-authored-by: hehutu <heihutu@gmail.com> * fix(config): publish storage_class runtime config on dynamic reload Change GLOBAL_STORAGE_CLASS from OnceLock to RwLock so runtime updates are possible. apply_storage_class_runtime_config now actually publishes the parsed config via set_global_storage_class instead of dropping it. Addresses review feedback: storage_class was marked as dynamically applied but the parsed result was discarded, so mc admin config set returned config_applied=true while the runtime kept using stale parity settings until restart. Co-authored-by: hehutu <heihutu@gmail.com> * fix(admin): harden config init, history ordering, and env redaction - Change GLOBAL_SERVER_CONFIG from RwLock<Config> to RwLock<Option<Config>> initialized with None, preserving "not initialized" detection via None - Move save_server_config_history before save_server_config_to_store in SetConfigKVHandler, DelConfigKVHandler, and SetConfigHandler so a restore point exists before mutations are persisted - Redact sensitive env override values with *redacted* instead of silently omitting the line, improving admin visibility - Add code comment explaining VolumeNotFound removal rationale in list_merged for listing paths Co-authored-by: hehutu <heihutu@gmail.com> * fix(config): keep in-memory config in sync after set/restore/reload GLOBAL_SERVER_CONFIG was a OnceLock set once at startup and never updated. After mc admin config set writes to the store, any fallback to get_global_server_config() returned stale init-time data. Similarly, reload_runtime_config_snapshot read from the store but discarded the result. - Replace OnceLock with RwLock for GLOBAL_SERVER_CONFIG and GLOBAL_STORAGE_CLASS so they can be updated at runtime - Add set_global_server_config / set_global_storage_class setters - Call set_global_server_config after every config save (set-kv, del-kv, set-config, restore-history) - Re-apply dynamic subsystems (storage_class, audit_webhook, audit_mqtt) and signal peers in reload_runtime_config_snapshot and full-config operations - Fix render_selected_config scope boundary check: track per-scope line count instead of checking global lines.is_empty() - Include STORAGE_CLASS_SUB_SYS in is_dynamic_config_subsystem so apply_storage_class_runtime_config is reachable Co-authored-by: hehutu <heihutu@gmail.com> * fix(storageclass): use CLASS_RRS key in lookup_config for RRS parity lookup_config used kvs.get(RRS) where RRS="REDUCED_REDUNDANCY", but the admin config path writes the key as CLASS_RRS="rrs". This caused RRS values to never be read back, always falling back to default parity. - Changed kvs.get(RRS) to kvs.get(CLASS_RRS) in lookup_config - Added regression tests verifying RRS read/write consistency Co-authored-by: hehutu <heihutu@gmail.com> * fix(config): add peer-side logging and don't swallow apply errors - Add tracing::warn! in reload_dynamic_config_runtime_state and reload_runtime_config_snapshot when config read or subsystem apply fails, so on-host diagnostics show which signal failed and why - Change `let _ = apply_dynamic_config_for_subsystem(...)` to `if let Err(err) = ... { warn!(...) }` in reload_runtime_config_snapshot so per-subsystem failures are logged instead of silently swallowed - Remove weak test global_server_config_returns_none_before_init that had no meaningful assertion due to shared global state Co-authored-by: hehutu <heihutu@gmail.com> * style: apply rustfmt to config and storageclass tests Co-authored-by: hehutu <heihutu@gmail.com> --------- Co-authored-by: weisd <im@weisd.in> Co-authored-by: hehutu <heihutu@gmail.com> |
||
|
安正超
|
642d83f0e4 |
revert: remove #2351 chunk I/O and object-io crate (phase 7) (#2543)
Co-authored-by: houseme <housemecn@gmail.com> |
||
|
houseme
|
32bf8f5bf3 |
feat(storage): add direct chunk GET fast path (#2351)
Signed-off-by: houseme <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: cxymds <Cxymds@qq.com> |
||
|
0xdx2
|
44f3eb7244 |
Fix: add support for additional AWS S3 storage classes and validation logic (#487)
* Fix: add pagination fields to S3 response * Fix: add support for additional AWS S3 storage classes and validation logic * Fix: improve handling of optional fields in S3 response --------- Co-authored-by: DamonXue <damonxue2@gmail.com> |
||
|
houseme
|
c5df1f92c2 |
refactor: replace lazy_static with LazyLock and notify crate registry create_targets_from_config (#311)
* improve code for notify * improve code for logger and fix typo (#272) * Add GNU to build.yml (#275) * fix unzip error * fix url change error fix url change error * Simplify user experience and integrate console and endpoint Simplify user experience and integrate console and endpoint * Add gnu to build.yml * upgrade version * feat: add `cargo clippy --fix --allow-dirty` to pre-commit command (#282) Resolves #277 - Add --fix flag to automatically fix clippy warnings - Add --allow-dirty flag to run on dirty Git trees - Improves code quality in pre-commit workflow * fix: the issue where preview fails when the path length exceeds 255 characters (#280) * fix * fix: improve Windows build support and CI/CD workflow (#283) - Fix Windows zip command issue by using PowerShell Compress-Archive - Add Windows support for OSS upload with ossutil - Replace Chinese comments with English in build.yml - Fix bash syntax error in package_zip function - Improve code formatting and consistency - Update various configuration files for better cross-platform support Resolves Windows build failures in GitHub Actions. * fix: update link in README.md leading to a 404 error (#285) * add rustfs.spec for rustfs (#103) add support on loongarch64 * improve cargo.lock * build(deps): bump the dependencies group with 5 updates (#289) Bumps the dependencies group with 5 updates: | Package | From | To | | --- | --- | --- | | [hyper-util](https://github.com/hyperium/hyper-util) | `0.1.15` | `0.1.16` | | [rand](https://github.com/rust-random/rand) | `0.9.1` | `0.9.2` | | [serde_json](https://github.com/serde-rs/json) | `1.0.140` | `1.0.141` | | [strum](https://github.com/Peternator7/strum) | `0.27.1` | `0.27.2` | | [sysinfo](https://github.com/GuillaumeGomez/sysinfo) | `0.36.0` | `0.36.1` | Updates `hyper-util` from 0.1.15 to 0.1.16 - [Release notes](https://github.com/hyperium/hyper-util/releases) - [Changelog](https://github.com/hyperium/hyper-util/blob/master/CHANGELOG.md) - [Commits](https://github.com/hyperium/hyper-util/compare/v0.1.15...v0.1.16) Updates `rand` from 0.9.1 to 0.9.2 - [Release notes](https://github.com/rust-random/rand/releases) - [Changelog](https://github.com/rust-random/rand/blob/master/CHANGELOG.md) - [Commits](https://github.com/rust-random/rand/compare/rand_core-0.9.1...rand_core-0.9.2) Updates `serde_json` from 1.0.140 to 1.0.141 - [Release notes](https://github.com/serde-rs/json/releases) - [Commits](https://github.com/serde-rs/json/compare/v1.0.140...v1.0.141) Updates `strum` from 0.27.1 to 0.27.2 - [Release notes](https://github.com/Peternator7/strum/releases) - [Changelog](https://github.com/Peternator7/strum/blob/master/CHANGELOG.md) - [Commits](https://github.com/Peternator7/strum/compare/v0.27.1...v0.27.2) Updates `sysinfo` from 0.36.0 to 0.36.1 - [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md) - [Commits](https://github.com/GuillaumeGomez/sysinfo/compare/v0.36.0...v0.36.1) --- updated-dependencies: - dependency-name: hyper-util dependency-version: 0.1.16 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies - dependency-name: rand dependency-version: 0.9.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies - dependency-name: serde_json dependency-version: 1.0.141 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies - dependency-name: strum dependency-version: 0.27.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies - dependency-name: sysinfo dependency-version: 0.36.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * improve code for logger * improve * upgrade * refactor: 优化构建工作流，统一 latest 文件处理和简化制品上传 (#293) * Refactor: DatabaseManagerSystem as global Signed-off-by: junxiang Mu <1948535941@qq.com> * fix: fmt Signed-off-by: junxiang Mu <1948535941@qq.com> * Test: add e2e_test for s3select Signed-off-by: junxiang Mu <1948535941@qq.com> * Test: add test script for e2e Signed-off-by: junxiang Mu <1948535941@qq.com> * improve code for registry and intergation * improve code for registry `create_targets_from_config` * fix * Feature up/ilm (#305) * fix * fix * fix * fix delete-marker expiration. add api_restore. * fix * time retry object upload * lock file * make fmt * fix * restore object * fix * fix * serde-rs-xml -> quick-xml * fix * checksum * fix * fix * fix * fix * fix * fix * fix * transfer lang to english * upgrade clap version from 4.5.41 to 4.5.42 * refactor: replace `lazy_static` with `LazyLock` * add router * fix: modify comment * improve code * fix typos * fix * fix: modify name and fmt * improve code for registry * fix test --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: junxiang Mu <1948535941@qq.com> Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com> Co-authored-by: 安正超 <anzhengchao@gmail.com> Co-authored-by: shiro.lee <69624924+shiroleeee@users.noreply.github.com> Co-authored-by: Marco Orlandin <mipnamic@mipnamic.net> Co-authored-by: zhangwenlong <zhangwenlong@loongson.cn> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: junxiang Mu <1948535941@qq.com> Co-authored-by: likewu <likewu@126.com> |
||
|
houseme
|
5826396cd0 |
refactor: Restructure project layout and clean up dependencies (#30)
This commit introduces a significant reorganization of the project structure to improve maintainability and clarity. Key changes include: - Adjusted the directory layout for a more logical module organization. - Removed unused crate dependencies, reducing the overall project size and potentially speeding up build times. - Updated import paths and configuration files to reflect the structural changes. |