mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-20 03:22:18 +00:00
1d305aeb5b15f5b5ef81e459f4cedafbd703ed39
30 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 houseme
|
9a2d06b370 |
test(heal): lock heal vs delete/overwrite race invariants (HS-12) (#6183)
* test(heal): add concurrency invariants for heal vs delete/overwrite races (HS-12) Audit conclusion for backlog#1874: RustFS does not need a persistent object-level healing marker (MinIO x-minio-healing) because every path that can touch the same (bucket, object) commit surface serializes on the same namespace write lock, and the heal lock guard spans the whole rename commit including the HEAL_RENAME_INCOMPLETE partial path. Lock the conclusion in with two race regression tests: - heal_racing_version_delete_never_resurrects_the_deleted_version: shard damage is injected on the doomed version so a Deep heal has real reconstruction work while a versioned DELETE runs concurrently; the deleted version must stay deleted and the survivor intact. - heal_racing_unversioned_overwrites_preserves_the_last_commit: unversioned overwrites (activating the post-commit tail that deletes the replaced data dir without the ns lock) race a Deep heal in a loop; the final current version must be exactly the last commit. Also adds docs/operations/heal-concurrency-safety-notes-zh.md with the full intersection matrix (17 intersections), lock-coverage argument, and the residual-window classification (commit tail races are fail-into-retry safe; bare prefix delete has zero production callers; admin no_lock is an explicit operator opt-in). Co-Authored-By: heihutu <heihutu@gmail.com> * test: remove redundant heal etag clone Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
360bceafce |
feat(heal): add progress and trace observability (#6179)
* feat(heal): track erasure set progress baseline Record erasure-set heal byte progress from per-object results and seed progress totals from complete usage-cache snapshots when available. Keep usage-cache failures observational so heal execution continues without a baseline. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(heal): skip filtered erasure set versions Skip erasure-set versions written after the durable heal start time, and queue lifecycle-expired versions for expiry before skipping them. Track new-version and ILM-expired skips separately so progress can explain completed baseline work without treating these skips as retry-blocking failures. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(heal): wire abandoned data-dir cleanup check Connect check_abandoned_parts through ECStore, pool, and set layers so heal can invoke the existing orphan data-dir reclaim path instead of returning NotImplemented. Add dry-run support to the reclaim scan and cover dry-run plus scoped set behavior with regression tests. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add heal scanner trace bus Introduce an in-process broadcast trace bus with typed heal and scanner events, lazy event construction, and bounded lagged-subscriber behavior. Cover zero-subscriber publishing, subscription delivery, drop accounting, and lagged receivers with focused common-crate tests. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): stream heal trace events from admin API Wire the admin trace endpoint to the common trace bus for heal/scanner events, including kind, regex, and threshold filtering. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): emit heal trace events Publish heal task lifecycle and abandoned-parts cleanup events through the common trace bus so the admin trace stream has live heal diagnostics. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): emit scanner trace events Publish scanner folder, lifecycle action, and heal-candidate events through the common trace bus for live admin scanner diagnostics. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): route data usage loader through storage api Keep ECStore data-usage facade access behind the heal storage_api boundary so architecture migration guards can validate the heal progress path. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(heal): avoid lifecycle snapshots on ordinary heal pages Only request lifecycle object snapshots when the heal pass has lifecycle expiry context. This keeps ordinary listing and disk-walk pages from cloning FileInfo/ObjectInfo payloads while preserving the skip path that queues expired versions. Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): update bug-fix mocks for lifecycle snapshots Carry the lifecycle snapshot opt-in argument through the remaining heal bug-fix test mocks so all-targets clippy covers the updated storage trait. Co-Authored-By: heihutu <heihutu@gmail.com> * test(rustfs): sync heal storage mock signature Update the rustfs storage RPC test mock for the lifecycle snapshot opt-in argument and cover it with rustfs all-targets clippy. Co-Authored-By: heihutu <heihutu@gmail.com> * test(e2e): allocate smoke ports across nextest processes Serialize E2E port selection with a small /tmp allocator so nextest workers do not reuse the same just-released ephemeral port before RustFS binds it. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
9f02ca6c36 |
fix(ecstore): resolve nine unused bindings in set_disk write and heal paths (#6158)
backlog#1823 step 1, the diagnosis half. Temporarily removing set_disk/mod.rs's #![allow(unused_variables)] surfaced nine bindings. The issue asks that values computed and then dropped on write/quorum paths be diagnosed before being underscored, and that turned out to matter: only four were plain leftovers. Two errors were bound and then left out of the log they were bound for. complete_multipart_upload's checksum failures read `if let Err(err) = ...` and then log part_id, bucket and object with no `err` anywhere in the message, so a checksum failure in production told you which part failed but not why. Both messages now carry the error. One is a lock guard. heal's write_lock_guard holds a namespace write lock for the rest of the scope; renaming it to a bare `_` would drop it immediately and release the lock. It is now `_write_lock_guard`, with a comment saying why it must not be `_`. One was kept alive by a corpse. `errors` in read_multiple_files is read by nothing except two commented-out debug! lines directly below it; the binding and the commented lines go together. One is a cfg split. heal's disk_index is read only inside the #[cfg(test)] fault-injection branch, so underscoring it would break the test build; a `#[cfg(not(test))] let _ = disk_index;` covers the non-test lane instead. The remaining four are genuine leftovers: an unused enumerate index in list_object_parts, a discarded error in a heal reader loop, an inner binding shadowing its own iterator variable, and delete_object's write_quorum. That last one is worth a separate look: delete_object asks get_object_info_and_quorum for a write quorum and never uses it, because delete_object_version below recomputes its own as disks.len() / 2 + 1. The two are not the same number — one comes from the object's erasure configuration, the other is a plain majority of the disk array. Pre-existing behaviour, untouched here. The blankets stay for now. Removing #![allow(unused_imports)] exposes 76 unused imports in set_disk/mod.rs, and they cannot be removed per-lane: cargo fix, working from the lib lane, produced 54 compile errors in the test lane. That needs its own pass with both lanes checked per import. Verification: cargo check -p rustfs-ecstore --tests and --features test-util --tests both warning-free; clippy --lib --tests -D warnings clean; cargo nextest run -p rustfs-ecstore 4101 passed; make pre-commit exit 0. Ref rustfs/backlog#1823 (step 1). |
||
|
Zhengchao An
|
f1f86ee9d0 |
chore(ecstore): drop the set_disk dead_code blanket (#6141)
* chore(ecstore): drop the set_disk dead_code blanket Removing the blanket exposes 39 items; exactly one is deleted. The low share is a finding, not caution: unlike the disk root, where platform gating made local adjudication impossible, here the items were checked and nearly all of them are live. Deleted: HealEntryResult, the only item with no reference anywhere. What the checks turned up, in the order the warnings suggest deleting them: SetDisks::rename_data looked like the head of a dead chain feeding into_legacy_tuple and RenameDataLegacyTuple. It is not: production goes through rename_data_owned, and rename_data itself has test callers at mod.rs:5809 and 5880. The chain below it is therefore live through the tests, and inferring "this is dead, so its callee is dead" would have removed three working items. create_bitrot_readers_until_quorum, read_multiple_files and map_cleanup_join_result all have callers inside their files' test modules, so they only look dead in the lib target. TransitionCommitBarrier and TransitionUploadedSaveProbe, with their install/wait_until_paused/release surfaces, are installed by tests behind #[cfg(all(test, feature = "test-util"))]. ctx.rs's SetDisksCtx accessors are the split seam left by the SetDisks god-object break-up (backlog#815). heal_object_dir's two apparent references are comments, and they document an index-alignment contract that live code maintains for it, so they stay as they are. Worth a maintainer decision: the metadata early-stop switch has a complete percentage-rollout facet — ENV_RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT, get_metadata_early_stop_rollout_pct and should_use_metadata_early_stop — with no caller, no test and no documentation, while its sibling enable flag is live. It is kept with an allow that says so rather than removed, since a rollout knob is a product call. One placement note for anyone adding allows near heal code: check_logging_guardrails.sh requires #[instrument(level = "trace")] to sit immediately before async fn heal_object_dir, so the allow goes above the instrument attribute. Putting it between the two drops the guard's match count and fails the check. Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. cargo nextest run -p rustfs-ecstore 4096 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0. Ref rustfs/backlog#1823 (step 2). * chore(ecstore): fix duplicated and inaccurate dead_code reasons in set_disk format_lock_error carried the same #[allow] twice. Five items in the locking/heal roots were labelled 'asserted by this file's tests' while having no reference at all - heal_object_dir's only two references are comments, as this branch's own notes point out. Say what each item actually is instead, so the next reader does not assume test coverage that is not there. Ref rustfs/backlog#1823. * chore(ecstore): correct the bounded_spare_disk_index dead_code reason The mod.rs copy is an unused test fixture, not something this module's tests assert; the namesake that is exercised lives in the io_primitives test module. Ref rustfs/backlog#1823. |
||
|
houseme
|
0ff3d4cbf4 |
perf(ecstore): borrow rename metadata during commit fanout (#6104)
* perf(ecstore): borrow rename metadata during commit fanout Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): preserve rename_data API compatibility Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: Zhengchao An <anzhengchao@gmail.com> |
||
|
houseme
|
5b54c4303d |
fix(ecstore): reconcile object cleanup receipts (#6077)
* fix(s3): keep multipart completion publication owned Co-Authored-By: heihutu <heihutu@gmail.com> * fix(s3): keep put publication owned Co-Authored-By: heihutu <heihutu@gmail.com> * chore(app): route multipart context through facade Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): gate object transaction fencing Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): fence object transaction epochs Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): reconcile old data cleanup receipts Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
d2b1003612 |
perf(storage): converge Wave 2 hot-path optimizations (#6065)
* perf(get): share inline shards and lock clients Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): converge PUT encoding on contiguous blocks Co-Authored-By: heihutu <heihutu@gmail.com> * perf(get): cache codec streaming gate config Co-Authored-By: heihutu <heihutu@gmail.com> * fix(sse): redact projected customer headers Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): collapse GET metadata snapshots Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): reuse decode stripe scratch Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(ecstore): trim decode scratch adapters Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): adapt transition checks to metadata snapshots Co-Authored-By: heihutu <heihutu@gmail.com> * perf(get): release metadata snapshots at ownership boundary Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(ecstore): close cumulative fast-path findings Co-Authored-By: heihutu <heihutu@gmail.com> * fix(storage): preserve lock and header invariants Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): adapt cumulative paths after rebase Co-Authored-By: heihutu <heihutu@gmail.com> * fix(rio-v2): adapt generated metadata fixture Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
cxymds
|
e11fcfbd08 |
fix(rebalance): converge multipart data movement retries (#6057)
* fix(rebalance): converge multipart data movement retries
* fix(rebalance): harden multipart retry replacement
* fix(rebalance): isolate internal multipart uploads
* test(ecstore): adapt metadata mutation fixtures
* fix(rebalance): preserve transition metadata semantics
* refactor(ecstore): reuse internal metadata matcher
* Revert "refactor(ecstore): reuse internal metadata matcher"
This reverts commit
|
||
|
houseme
|
924958bab5 |
perf(get): slim metadata fanout allocations (#1803) (#5968)
Every GET fans out a `read_version` across all disks to resolve xl.meta. Each fanout allocated an `Arc<ReadOptions>` (3 bools) plus four `Arc<String>` (`Arc::new(x.to_string())` = two allocations each) and cloned them into every spawned task. This trims the per-fanout allocation footprint. - `ReadOptions` is three bools, so it is now `Copy`. The fanout drops the `Arc<ReadOptions>` and hands each spawned task a copy; the two pre-existing `ReadOptions::clone()` sites (set_disk/read.rs, set_disk/ops/heal.rs) stop cloning a `Copy` type. - The four request strings use `Arc::<str>::from(&str)` (one allocation each) instead of `Arc::new(..to_string())` (string buffer + Arc = two each) — four fewer allocations per fanout, transparent to the `read_version(&str)` call. Behavior is unchanged: the fanout still spawns one task per disk (the spawn is deliberate — `read_version_call_counter_observes_spawned_fanout` verifies the process-global counter observes every per-disk increment across workers), quorum / early-stop / full-wait semantics are untouched, and no result ordering or error handling changed. Two larger items from the audit are intentionally NOT in this PR: - `tokio::spawn` -> `FuturesUnordered`: the spawn is a tested, deliberate design (cross-worker counter observation for #1309/#1314), and converting would also change panic isolation. Left as-is. - `vec![FileInfo::default(); N]`: `FileInfo`'s empty containers (String / HashMap / Vec) do not allocate, so this is one `Vec` allocation, not the per-element allocation the audit implied — not a real hot spot. `cargo fmt`, `cargo clippy -p rustfs-ecstore --lib` (0 warnings), `cargo check --lib --tests`, and the 26 fanout / call-counter unit tests pass on macOS (the change is fully cross-platform). Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
houseme
|
276eea1fba |
test(heal): cover replacement target evidence failures (#5919)
Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
f17ea7f146 |
fix(heal): harden replacement rebuild tracking (#5892)
* fix(heal): gate auto replacement formatting Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): require replacement target outcomes Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): bind resumes to replacement targets Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): fence healing marker ownership Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): cover replacement target completion Co-Authored-By: heihutu <heihutu@gmail.com> * docs(heal): clarify replacement recovery status Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): canonicalize replacement target checks Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): satisfy marker test module lint Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): scope automatic replacement format Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): require a mounted replacement target Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): avoid cloned ref slice in test Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): revalidate replacement before scanning Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): reset stale resume checkpoints Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): release scanner disk map before probing Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): persist replacement intent before format Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): fail closed on mountinfo read errors Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): fence replacement target identity Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): order replacement completion cleanup Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): atomically seal replacement completion Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): census replacement target shards Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): fence replacement recovery ownership Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): preserve replacement recovery anchors Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): satisfy replacement recovery lint gates Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): bind replacement identity to mount lease Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): cover durable replacement recovery states Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): validate persisted resume task identifiers Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): avoid blocking replacement marker CAS Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): report failed marker rollback Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): pin replacement resume schema compatibility Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): preserve durable recovery anchors Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): preserve public disk path semantics Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): use canonical replacement task ids Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): cover automatic replacement in 3x4 cluster Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): verify replacement target commits Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): persist replacement completion proof Co-Authored-By: heihutu <heihutu@gmail.com> * feat(heal): expose durable replacement status Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): bound durable replacement discovery Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): remove replacement readiness bypass Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): retry terminal replacement cleanup Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): isolate replacement intents from legacy resume Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): migrate legacy replacement intents at startup Co-Authored-By: heihutu <heihutu@gmail.com> * style(heal): apply strict clippy fix Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): prioritize active replacement recovery state Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): bind readiness to the admitted mount lease Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): atomically publish replacement intents Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): isolate replacement recovery directory Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): tolerate an empty recovery directory Co-Authored-By: heihutu <heihutu@gmail.com> * style(heal): remove redundant disk bytes conversion Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): reconcile proof-first replacement recovery Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): fence torn intent recovery Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): cover replacement migration conflicts Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): fence replacement lease mount identity Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): cover missing replacement path admission Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): reject conflicting legacy completion proof Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): fall back to proc mount identity Co-Authored-By: heihutu <heihutu@gmail.com> * feat(admin): expose replacement recovery status Surface the local durable replacement recovery snapshot in the background heal status response so operators can tell whether replacement cleanup is definitive or still pending. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): keep replacement status compatible Keep the existing background heal status response wire-compatible while retaining the Linux mount lease cleanup needed for the replacement recovery branch. Co-Authored-By: heihutu <heihutu@gmail.com> * style(ecstore): match linux mount lease formatting Keep Linux rustfmt output stable for the replacement mount lease comparison. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): qualify mount lease test constant Use the disk module path for the format config constant in the Linux mount lease regression test. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): keep procfd mount roots directory-safe Use a procfd path with an explicit directory component so Unix directory guards can open the replacement mount lease root with O_NOFOLLOW while preserving handle-relative I/O semantics. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): delete empty leased buckets via dirfd Use the held mount lease fd as the parent for non-force empty bucket deletion on Linux so procfd-rooted paths do not get rejected as BucketNotEmpty. Also make the download-part OpenOptions truncate behavior explicit and keep fsync test recording stable across procfd canonicalization. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): scan leased bucket paths for emptiness Use the local disk I/O root for bucket emptiness probes before non-force bucket deletion and table-bucket metadata checks. This keeps validation on the same mount instance as the subsequent local disk delete path. Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): align lease path test probes Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): block unsafe replacement recovery restarts Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): defer blocked replacement candidates Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): retry transient replacement discovery Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): keep transient recovery errors retryable Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): block corrupt legacy replacement state Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): classify flat replacement intent corruption Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): keep transient resume loads retryable Classify malformed legacy replacement state as blocking corruption while preserving disk and transient load failures for retry. This avoids permanently blocking replacement recovery on temporary storage errors. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): avoid latching transient legacy publishes Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): retry blocked legacy migrations Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): defer blocked startup recoveries Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): preserve disk sync limiter across lease roots Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
3bad829b9a | test(heal): cover partial rename retry admission (#5772) | ||
|
Zhengchao An
|
8550a8f9c3 |
refactor(ecstore): unify remaining heal logs to structured event style (#5720)
PR #5719 fixed the issue #5716 per-object heal log amplification (per-object statements demoted, heal spans forced to TRACE, raw metadata dumps banned by guardrail) and superseded the demotion originally proposed here. This PR now carries only the residual cleanup on top of it: - Convert the remaining bare-field and format-arg heal logs in crates/ecstore/src/set_disk/ops/heal.rs to the file's structured convention (event/component/subsystem + context fields): missing-object skip, disk-marked-for-healing, cannot-reconstruct errors, dangling-cleanup error, missing data_dir error, xl.meta regeneration warn, and orphan-reclaim failure warn. - Demote the last remaining info! in the file — the per-set heal_format "set disk formats success, NoHealRequired" no-op message — to a structured debug! (error_count instead of a raw errs dump), and drop its whitelist exclusion in scripts/check_logging_guardrails.sh so the no-INFO check for set-disk heal files is strict. No control flow or behavior changes. |
||
|
anthonymartin
|
ec135f8c4c |
fix(heal): bound per-object logging (#5719)
Co-authored-by: Anthony Martin <949506+anthonymartin@users.noreply.github.com> |
||
|
cxymds
|
31959b90db |
fix(heal): harden resumable set repair failures (#5693)
* fix(heal): enforce resumable task control * fix(ecstore): surface bucket and metadata heal errors * chore: refresh guardrail path references --------- Signed-off-by: cxymds <cxymds@gmail.com> |
||
|
cxymds
|
67904a6c18 |
fix(ecstore): start with unresolved Kubernetes peers (#5460)
* fix(ecstore): start with unresolved Kubernetes peers * fix(ecstore): infer Kubernetes endpoint identity safely * fix(ecstore): fail closed on unsafe format migration * fix(ecstore): reject poisoned format heal candidates * fix(ecstore): reject unsafe legacy migration outliers * fix(ecstore): resume interrupted format migrations * fix(ecstore): preserve Kubernetes startup compatibility |
||
|
cxymds
|
02f4dbeb68 |
fix(heal): fail closed on ambiguous metadata rescue (#5361)
* fix(heal): fail closed on ambiguous metadata rescue * fix(heal): preserve uncertain dangling state * test(heal): satisfy strict clippy checks * fix(heal): retain explicit version metadata recovery * fix(heal): preserve metadata rescue diagnostics --------- Co-authored-by: houseme <housemecn@gmail.com> |
||
|
cxymds
|
d7afa4e38e |
fix(heal): report partial rename failures (#5355)
* fix(heal): report partial rename failures * fix(log-analyzer): track partial heal rename failures |
||
|
houseme
|
dd46de0945 |
fix(heal): report no-parity bitrot as unrecoverable (#5192)
Keep corrupted no-parity heal results on the integrity-failure path, preserve the object geometry in operator output, and document the recovery boundary for historical bad shards. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
cxymds
|
b0c6c4cbce |
fix(storage): resolve erasure parity per pool (#4977)
* fix(filemeta): add state-aware file info validation
* fix(filemeta): validate shard arithmetic and delete paths
* fix(ecstore): add fallible erasure construction
* fix(ecstore): resolve storage parity per pool
* fix(storage): report heterogeneous erasure layouts
* fix(admin): publish prepared storage config atomically
* fix(storage): harden per-pool parity boundaries
* fix(storage): address pre-PR validation findings
* test(ci): fix strict-topology validation fixtures
* fix(heal): preserve delete markers during repair
* refactor(filemeta): drop unused ValidatedFileInfo witness
ValidatedFileInfo wrapped an unread `_file_info` reference alongside an `Option<ValidatedErasureLayout>`, but only the layout was ever consumed. Return the layout directly from `FileInfo::validate` so the sole production consumer (`LocalDisk::check_parts`) and the two unit tests read it without the extra witness type and lifetime.
No behavior change.
* fix(filemeta): keep compressed and MinIO-migrated tiered objects readable
The new decode-path validation rejected several legitimate on-disk shapes that older RustFS and MinIO-migrated data carry, turning readable objects into FileCorrupt:
- Compressed objects written with an unknown upload size persist a negative per-part actual_size (the documented "unknown size" sentinel that ObjectInfo::get_actual_size already tolerates). validate_collection_contents rejected it via usize::try_from; now a negative actual_size skips shard validation and only real, non-negative sizes are checked.
- MinIO-migrated objects transitioned to a versioned remote tier store the tier version id as a UUID string, not 16 raw bytes. MetaObject::into_fileinfo returned FileCorrupt (main tolerated it as None), making all versions of the object unreadable; MetaDeleteMarker free-version records took a Some(nil) sentinel path with the same effect, which also breaks free-version expiry (remote-tier leak). Both now decode through a shared transitioned_version_id_from_meta_sys helper: 16 raw bytes or a UUID string are accepted, anything else is tolerated as None instead of failing the read.
Regression tests updated to assert the readable/compat behavior, with new tests covering MinIO string-form recovery.
* fix(scanner): build the delete-marker test fixture without erasure geometry
get_size_counts_delete_markers_separately_from_versions built its delete marker with `FileInfo::new(object, 1, 1)`, which attaches erasure geometry (data=1/parity=1/distribution). This PR classifies versions by shape via `is_storage_delete_marker()` (no geometry) rather than the raw `deleted` flag, so a geometry-bearing "delete marker" is correctly serialized as a purge-pending payload Object and counted as a version — CI saw summary.versions=3, expected 2.
Real delete markers carry no erasure geometry (delete paths build them as `FileInfo { deleted: true, ..Default::default() }`), so construct the fixture the same way. It then classifies as a storage delete marker and the counts (versions=2, delete_markers=1) hold. This keeps the PR's more-correct classification, which prevents a purge-pending object's geometry from being dropped when serialized as a bare delete marker.
* docs(changelog): note per-pool parity fix and storage-class startup upgrade caveat
Records the #4801 per-pool erasure parity fix under Fixed, and documents the upgrade behavior where a persisted storage class that a small or heterogeneous pool cannot satisfy now fails startup — with the RUSTFS_STORAGE_CLASS_STANDARD recovery steps. Docs-only; covers R4 from the on-disk compatibility audit.
* fix(heal): report parity from erasure geometry, not is_valid()
heal_object set HealResultItem.parity_blocks via `if lfi.is_valid()`, which was missed by the migration of the other quorum/metadata predicates. With the new `is_valid()` semantics (full payload validation; delete markers now return false), a delete marker or a geometry-bearing version with a benign collection quirk would misreport parity as the pool default instead of its own. Use `has_valid_erasure_geometry()` — the narrow "does this carry erasure geometry" predicate the rest of the migration uses — so reporting matches the object's actual layout. Reporting-only; no data-path change.
* fix(filemeta): do not silently serialize a non-canonical deleted FileInfo as an Object
`From<FileInfo> for FileMetaVersion` classifies by `is_storage_delete_marker()` (shape), which correctly routes canonical delete markers to Delete and purge-pending payloads (deleted=true with real erasure geometry) to Object. But a `deleted` FileInfo that is neither a canonical marker nor a valid erasure payload would silently serialize as a zero-geometry MetaObject that later fails `validate_for_metadata_read`. Write paths validate first (`validate_for_erasure_write` / `validate_for_metadata_read`), so this is a caller bug; `From` is infallible, so surface it with a structured `warn!` on the malformed branch instead of writing corrupt metadata silently. Legitimate purge-pending objects (valid geometry) are unaffected — the guard only fires for `deleted && !has_valid_erasure_geometry()`.
* test(filemeta): assert real historical xl.meta versions pass metadata-read validation
Empirical companion to the code-reasoned decode-tolerance invariants (docs/architecture/erasure-coding.md §11) and the rolling-upgrade / MinIO-migration compatibility concern: the tightened `validate_for_metadata_read` runs on every local disk read and peer-RPC-decoded FileInfo, so it must accept every version of real historically-written xl.meta, never reject it as FileCorrupt.
Loads five real fixtures — MinIO small-inline, MinIO versioned (two object versions + a delete marker), MinIO large multipart, a legacy V1 (xl.json-derived) object, and a legacy meta_ver 2 object — decodes every version with parts materialized, and asserts validate_for_metadata_read() is Ok for each. Reverting the tolerant handling (delete-marker shape, legacy per-part checksums, string/short transitioned-versionID, negative actual_size) turns this red.
* fix(ci): remove duplicate storage test re-exports
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
|
||
|
Zhengchao An
|
7f5873dac8 |
fix(ecstore): resolve erasure parity per pool (#4801) (#5015)
* fix(ecstore): add fallible erasure construction (cherry picked from commit |
||
|
Zhengchao An
|
4d22ed4465 |
perf(capacity): drop per-PUT global lock and per-disk allocation from write dirty-scope (#4933)
perf(capacity): remove per-PUT global lock and per-disk allocation from write dirty-scope Every successful write recorded its capacity dirty scope by allocating an endpoint/path String per online disk, deduplicating through a HashSet, entering the global dirty-scope Mutex, and — in the app response path — taking a global async RwLock to record the write frequency. Under small-object high concurrency this created a global serialization point and O(disks) allocation on the hot path (https://github.com/rustfs/backlog/issues/1315). This change makes the steady-state write path allocation-free and lock-free without altering capacity accounting semantics: - Memoize the per-set dirty scope. Each set resolves its disks' immutable endpoint/path identity lazily into a slot-indexed cache and reuses a shared `Arc<CapacityScope>`; steady-state writes clone the Arc under a read lock instead of rebuilding String/HashSet. The heal path keeps an ad-hoc scope builder because it passes disks in erasure-distribution order rather than physical-slot order. - Add a monotonic generation to the global dirty-scope registry, advanced only when a non-empty drain removes disks. A set upgrades the global registry mutex only on the first write of each generation and then skips it while the generation is unchanged; the observed generation is read under the registry lock so a concurrent drain forces a re-mark, preventing lost updates. The write commits its bytes before recording the scope, so any drain that could remove the mark is ordered after the commit and the following refresh reads the committed bytes. - Replace the write-frequency `RwLock<WriteRecord>` with lock-free atomics: per-second CAS buckets, an atomic last-write timestamp, and an atomic total counter. The frequency window and debounce semantics the refresh scheduler relies on are unchanged. Capacity marking remains a conservative superset of the disks actually written, so admin/scan totals are byte-for-byte identical: extra dirty marks only trigger a re-read of a disk whose usage is unchanged. White-box tests assert the memoized scope equals the previous ad-hoc construction, that the global registry is upgraded exactly once per generation and re-marked after a drain, and that the lock-free write record is exact under concurrent contention. Ref: https://github.com/rustfs/backlog/issues/1315 |
||
|
houseme
|
63b4568f85 |
fix(ecstore): reclaim orphan data dirs on the healthy heal path (#4781)
PR #4356 wired `reclaim_orphan_data_dirs` only into `heal_object`'s post-heal tail, which runs after the `disks_to_heal_count == 0` early return. That early return is exactly the state of the objects the sweep targets: a valid `xl.meta` with all shards present plus a leaked pre-#3510 data dir needs no shard healing, so a healthy heal returned before reclaim and swept nothing. On a healthy deployment (single node, no degraded disks) the reclaim was therefore dead code — an admin heal walked the objects, "healed" them, and reclaimed no leaked space. Run the best-effort reclaim on the `disks_to_heal_count == 0` path as well, gated on `!opts.dry_run`. The shared match+log block is factored into `reclaim_orphan_data_dirs_best_effort` so both exits behave identically. A reclaim failure still never fails the heal. Adds an end-to-end regression: put a healthy non-inline object, plant an unreferenced UUID data dir under it on every disk that holds the object, then drive `heal_object`. A dry-run heal must leave the stray in place; a real heal must reclaim it while preserving the live data dirs, `xl.meta`, and object contents. The test fails against the pre-fix control flow. Refs #3231, #3191, #4356. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
13bdca6762 |
build(toolchain): switch Rust channel to stable (#4775)
* Change Rust toolchain channel to stable Signed-off-by: houseme <housemecn@gmail.com> * style: apply clippy --fix and cargo fix lint suggestions Run `cargo clippy --fix --all-targets --all-features` and `cargo fix --lib --all-targets` across the workspace, then resolve the remaining warnings by hand: - collapse needless borrows in `format!` args, prefer `?` over explicit early returns, and use `.values()` / `.flatten()` iterator adapters - rewrite the `Md5` scan loop via `manual_flatten` and re-indent the `select!` macro body (rustfmt skips macro interiors) - annotate the intentional dead-code `Md5` inherent methods (constructed only by the test factory) with `#[allow(dead_code)]` Behavior is unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Signed-off-by: houseme <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
e3e5693e60 |
fix(ecstore): correct heal result drive-record reporting (#4555)
Two report-only defects on the heal result-reporting surface (they do not affect the data path or heal decisions): - default_heal_result (set_disk/ops/heal.rs): the offline-disk branch pushed an Offline record but fell through into the unconditional push, emitting a second (Corrupt) record for the same disk. This grew before/after.drives to disk_count + offline_count and misaligned every entry after the first offline slot. Add `continue` after the offline push, drive `disk_len` and the loop from a single `self.disks` snapshot, and assert `errs.len() == disk_len`. - Sets::heal_format (core/sets.rs): the before/after drive lists were pre-filled with N default placeholders and then N real entries were pushed, yielding a 2N list whose healed status updates (indexed 0..N) landed on the blank placeholder half. Assign the lists directly from formats_to_drives_info (mirroring the set-level heal_format) so the healed updates hit the real entries. Add regression tests covering offline/online record alignment and the NoHealRequired and heal paths of the pool-level heal_format. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
3531abb34a |
feat(heal): disk-walk UNION enumeration to heal sub-quorum versions (backlog#920) (#4527)
B5 switched heal enumeration to list_object_versions, which only reflects the read-quorum metadata view: a version present on fewer than read-quorum disks was never enumerated, so it was never healed. Add a per-erasure-set disk-walk UNION enumerator (mirrors MinIO global-heal.go objQuorum=1 listPathRaw + mergeXLV2Versions) that surfaces every (object, version) present on ANY disk and feeds each to the existing per-version heal_object. - filemeta: MetaCacheEntries::resolve_union (dir_quorum=1/obj_quorum=1) yields the cross-disk version union at one tested seam. - ecstore: SetDisks::heal_walk_versions_page (list_path_raw fan-out, min_disks=1, dual object/version page bound, inclusive-forward de-overlap) + ECStore delegator + HealWalkVersion. - ecstore data-safety guard: before dangling-delete, try_regenerate_recoverable_meta physically probes part files via check_parts; when >= data_blocks data shards survive (meta lost but data recoverable) it regenerates xl.meta from a surviving FileInfo with the correct per-disk shard index instead of dangling-deleting. Genuine torn writes (< data_blocks) keep the current behavior — no resurrection. - heal: dw1: forward-marker cursor codec (reuses ResumeState.resume_cursor, idempotent restart on foreign tokens); list_versions_for_heal_page_disk_walk trait method (default falls back to the B5 read-quorum path); heal_bucket_with_resume selects the disk-walk enumerator when scan_mode==Deep || source==AutoHeal, else the unchanged B5 path; anti-loop guard aborts on (empty && truncated). Closes rustfs/backlog#920 |
||
|
houseme
|
c9b976ad46 |
feat(ecstore): reclaim orphaned data dirs during heal (#4356)
* feat(ecstore): reclaim orphaned data dirs during heal (backlog #3231/#3191) Pre-#3510, an unversioned overwrite leaked one UUID-named data dir per PUT. The write path now cleans up going forward, but pre-existing strays stay on disk forever: heal's dangling logic only removes whole objects whose data is missing, never surplus data dirs of an otherwise-healthy object. That leaked space is what eventually made ListObjects scan tens of GB and time out. Add SetDisks::reclaim_orphan_data_dirs, a fail-closed, quorum-safe sweep that mirrors purge_orphan_dir_object: - referenced set is the UNION of get_data_dirs() across every online replica's xl.meta, so a dir named by any replica is kept; - if a disk holds the object dir but its xl.meta is missing or unparseable the object is treated as degraded and nothing is removed; - only UUID-named subdirectories are ever considered. Wire it into heal_object's healthy tail (under the object write lock) as a best-effort step so the background heal scanner and admin heal recover the leaked space automatically; a reclaim failure never fails the heal. Covered by four unit tests: removal of an unreferenced dir, no-op when all dirs are referenced, fail-closed abort on missing metadata, and cross-replica union preservation. Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): fix typo unparseable -> unparsable Satisfies the repository typos CI check. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
6297d112c3 |
fix: auto-repair breaking changes from 1b3727e2 (#4324)
fix(heal): remove useless .into() conversion in heal error path Clippy flagged a useless_conversion lint at heal.rs:491 where is redundant because is already of type . |
||
|
Zhengchao An
|
1b3727e2c4 |
fix(heal): clean up .rustfs/tmp healed shards on all failure paths (backlog#799 B20) (#4319)
heal_object wrote healed shards to .rustfs/tmp/<uuid>/ and only removed that tmp dir on the success path (the final delete_all). Three early exits after the tmp shards were written leaked the dir: - `erasure.heal(...)?` failing midway, - the `disks_to_heal_count == 0` early return, and - the `?` on the post-rename remote data-dir delete. Add the tmp cleanup before the first two, and downgrade the remote data-dir cleanup failure to a warning (the healed shard is already renamed into place, so that cleanup failing must not abort the heal or leak the tmp shards). Refs backlog#799 (B20), tracked in rustfs/backlog#863. |
||
|
Zhengchao An
|
55ad8df1c2 |
refactor(ecstore): move HealOperations into set_disk::ops::heal (backlog#817) (#4267)
P1 of the SetDisks split (tracking backlog#815, depends on P0 backlog#816). Give the Heal operation family its own module home: relocate set_disk/heal.rs to set_disk/ops/heal.rs and move the HealOperations storage-api contract impl beside its inherent helpers. The contract stays implemented for SetDisks so its associated-type bounds are unchanged (ecstore_contract_compat_test still covers it). Method bodies are moved unchanged. The four inherent helpers widen from pub(super) to pub(in crate::set_disk) to preserve their exact prior visibility from the deeper module. get_pool_and_set now reads topology through SetDisksCtx to keep the Heal family aligned with the P0 borrow pattern; the read is provably identical (ctx.format()/pool_index() alias the core fields). Runtime behavior is unchanged. |