test(storage): require named core regressions and pinned fixtures (#8330)

* fix(oidc): ignore groups without a matching policy at login

OIDC login failed with "OIDC policy mapping did not resolve to current
policies" whenever any mapped group lacked a policy of the same name.
Directory-backed providers such as Active Directory always emit groups
like `DOMAIN\Domain Users` that can never be mapped, so group-based
authorization was impossible there.

Keep only policy names that resolve to an existing policy and are valid
in session claims, and reject the login only when none remain. The
signed `policy` claim still lists only resolved names, so request-time
evaluation and site replication receivers keep their invariant.

Refs #8163

* fix(oidc): only ignore unmapped groups-claim values

Limit the relaxed resolution to policy names derived solely from the
groups claim. Names from ROLE_POLICY, a dedicated CLAIM_NAME claim, or
an explicit IAM policy mapping must still all resolve.

Reject the login when a mapped name refers to an existing policy whose
name is not allowed in session claims: dropping it could remove an
explicit Deny and broaden access.

Claim-unsafe names are only checked against the in-memory policy cache
and never passed to storage-backed policy loading.

Refs #8163

* docs(oidc): document ignorable roles claim values and cover wiring

Roles claim values are merged into the canonical groups, so values
without a matching policy are ignored like groups-claim values. This
covers built-in provider roles such as Keycloak's `offline_access`.
Document that in the field and method docs and in the provider
requirements, and pin it with a test.

Assert that the OIDC authorization carries the group claim policies so
a regression in the wiring cannot silently disable the relaxation.

Refs #8163

* fix(oidc): ignore group policies only after confirmed absence

merge_policies drops names whose load fails, so a storage or decode
error for an uncached group policy was indistinguishable from a missing
one. The group name was then ignored and the session signed without it,
which could remove an existing Deny that request-time checks cannot
restore.

Add IamSys::policy_exists, which consults the cache and then storage and
reports false only for NoSuchPolicy while returning every other error.
The OIDC binding now ignores a group-derived name only after a confirmed
absence and rejects the login when a lookup fails.

Refs #8163

---------

Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: Chris <anzhengchao@gmail.com>

.agents/references/implementation.md

@@ -58,6 +58,13 @@ are repository-relative. Read only the relevant sections during read-only review
 - Attach error context once where it is actionable. Do not erase typed errors
   below aggregation or quorum layers.
 
+## Rust Boundaries
+
+- Production `unwrap`/`expect` must follow a type guarantee or checked invariant; explain only non-obvious guarantees.
+- Public APIs do not return `Result<_, String>`. Library APIs use domain errors unless intentional erasure is part of the boundary contract; expose a stored inner error through `Error::source()`.
+- Numeric conversions must not silently truncate. Use fallible conversions for untrusted values; validate finiteness, sign, and range before float-to-integer conversion. Clamp/saturate only when required by the domain.
+- Do not add crate-root `#![allow(dead_code)]`.
+
 ## Naming
 
 Use Rust API naming: `SCREAMING_SNAKE_CASE` constants/statics, `snake_case`

.agents/skills/adversarial-validation/references/concurrency-durability.md

@@ -2,8 +2,12 @@
 
 - For every changed lock, enumerate overlapping lock sets and construct the
   ABBA interleaving. Multiple-lock order must be documented and consistent.
-- Mark guard lifetimes and every `.await`, disk, and RPC call inside them.
-  Estimate contention and timeout behavior under concurrent requests.
+- Mark guard lifetimes and every `.await`, disk,
+  and RPC call; estimate contention/timeouts. Tokio read/write guards across
+  `.await` need bounded hold time; long-lived reads can wedge writers (#4195).
+  Keep `std::sync::Mutex` holds brief and never across `.await`.
+- Use direct `fetch_*` for unconditional atomic read-modify-write and
+  `compare_exchange` for conditional updates.
 - Object commits remain fenced if the distributed lock is lost after shard
   writes and before metadata rename.
 - For write/rename changes, trace `write tmp -> sync tmp -> rename -> sync parent

.agents/skills/adversarial-validation/references/correctness.md

@@ -15,6 +15,8 @@ Attack the changed behavior, not every subsystem in the repository.
   bytes and length.
 - For multipart/object commits, fail before/after rename and cleanup; committed
   data must remain readable and pre-commit cleanup must not destroy parts.
+- Bound recursion over untrusted or persisted input; handle corrupted/cyclic
+  tree and cache traversals safely.
 - For version/index ordering, test `len - 1`, `len`, equal timestamps, missing
   versions, and deterministic tie-breaking.
 - For directory-object behavior, trace `__XLDIR__` at the store layer; branches

.agents/skills/adversarial-validation/references/test-coverage.md

@@ -19,7 +19,8 @@
   serialization is required.
 - Internal metadata tests assert both RustFS and MinIO keys, not only read-back
   through a helper that prefers one key.
-- Boundary companions are distinct coverage: `n == max` vs `max + 1`, and
+- Avoid duplicate tests of the same production path and poison-value class.
+  Boundary companions are distinct coverage: `n == max` vs `max + 1`, and
   absent vs empty vs nil UUID.
 - A focused test proves only the targets/features it builds. Add compilation or
   Clippy only for uncovered changed targets.

.agents/skills/code-change-verification/SKILL.md

@@ -1,51 +0,0 @@
----
-name: code-change-verification
-description: Review a commit, PR, or merged patch when the user requests ordinary code-change verification. Do not combine with adversarial-validation; use that skill instead for explicitly adversarial, substantial, or high-risk RustFS reviews.
----
-
-# Code Change Verification
-
-Use this skill for an ordinary requested review. If the root policy or user calls
-for adversarial validation, use `adversarial-validation` instead of running both.
-
-## Core Workflow
-
-### 1) Scope and assumptions
-- Derive the change source, target branch, and relevant runtime/version from the
-  supplied diff and metadata. Ask only when missing context could change the verdict.
-- Focus only on requested scope; avoid reviewing unrelated files.
-
-### 2) Risk map
-- Prioritize in this order:
-  - Data correctness and user-visible behavior
-  - API/contract compatibility
-  - Security and authz/authn boundaries
-  - Concurrency and lifecycle correctness
-  - Performance and resource usage
-- Give higher priority to stateful paths, migration logic, defaults, and error handling.
-
-### 3) Evidence-based inspection
-- Read each modified hunk with neighboring context.
-- Trace call paths and call-site expectations.
-- Check for:
-  - invariant breaks and missing guards
-  - unchecked assumptions and null/empty/error-path handling
-  - stale tests, fixtures, and configs
-  - hidden coupling to shared helpers/constants/features
-- Apply root `AGENTS.md`'s finding standard: try to disprove a candidate before
-  reporting it. Mention an unresolved question only when it could materially
-  change the verdict; do not fill the report with speculative possibilities.
-
-#### Rust-specific checks
-
-For changed Rust behavior, use the matching sections of [rust-code-quality](../rust-code-quality/SKILL.md). Reuse checks already performed by the selected review workflow. Comment-only or formatting-only Rust diffs do not require the full Rust checklist. Carry its P0–P3 ratings over unchanged and use this skill's output format.
-
-### 4) Findings-first output
-- Order supported findings by P0–P3 severity; preserve the Rust ratings above.
-  Include `path:line`, the failure and impact, a focused fix, and its validation.
-- If no supported issues remain, state `No findings` with the reviewed scope and
-  any material verification limitation. Do not append optional improvements to
-  make a clean review look productive.
-
-Close after the required review. Recommend additional verification only for an
-identified unresolved risk or required gate; reuse evidence for unchanged code.

.agents/skills/code-change-verification/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "Code Change Verification"
-  short_description: "Prioritize risks and verify code changes before merge."
-  default_prompt: "Use $code-change-verification for an ordinary requested diff review with prioritized findings."

.agents/skills/issue-triage/SKILL.md

@@ -1,112 +1,13 @@
 ---
 name: issue-triage
-description: Assess whether a GitHub issue is fixed, needs implementation, or can be closed by checking related work and current code. Use for issue completion/triage requests. Status questions are read-only; comment, close, or change labels only when the conversation authorizes that action.
+description: Verify issue completion against current implementation and merged work. Use for issue triage; comment, close, or change labels only when authorized.
 ---
-
 # Issue Triage
 
-Use this skill when the user provides a GitHub issue URL and asks "can this be closed?", "is this already implemented?", "check completion status", or similar triage questions.
+- Resolve the issue repository separately from the implementation repository: `rustfs/backlog` tracks work in `rustfs/rustfs`. Pass `--repo` explicitly to GitHub queries.
+- Read the issue body, relevant comments, linked PRs, checklist, and sub-issues with `gh issue view`. Search explicit links, the full issue URL, qualified references, and subject keywords; an empty search result does not prove implementation is absent.
+- Fetch the implementation base and inspect its current code. For each merged candidate, verify `git merge-base --is-ancestor <merge-commit> <remote>/<base>`; a title, commit message, or merged state alone is insufficient. A local checkout may be stale or on another branch.
+- Verify every checklist/sub-issue before declaring completion. For batches, paginate the entire requested issue scope, exclude PR entries, and filter by author only when requested.
+- Recommend closing only when all requested behavior is present or evidence shows the issue is superseded. Otherwise name what remains. A status request is read-only; reuse existing authority for comments/closing/labels, and follow [Git and PR rules](../../references/pull-requests.md) before posting. Use Chinese for `rustfs/backlog`, existing repository labels only, and a closing comment naming the verified behavior and PRs.
 
-## Workflow
-
-### 1. Fetch issue context
-
-```bash
-gh issue view <N> --repo <owner/repo> --json title,body,state,comments,labels,updatedAt
-```
-
-Read the issue body to understand what was requested. Extract:
-- The specific feature/fix/behavior described.
-- Any linked PRs or commits mentioned in the body or comments.
-- Any checklist items or sub-issues.
-
-Resolve the issue repository and implementation repository separately (for example, `rustfs/backlog` tracks work in `rustfs/rustfs`). Pass the implementation repository explicitly to PR queries; the current checkout may belong to another repository.
-
-### 2. Search for related work
-
-Search git history for commits referencing the issue:
-```bash
-git log --oneline --all --grep="<N>" | head -30
-```
-
-Search for related PRs:
-```bash
-gh pr list --repo <implementation-repo> --search "<issue-url>" --state all --json number,title,state,mergedAt
-```
-
-Also search qualified issue references and subject keywords; for same-repository
-issues, include `#<N>`. Follow explicit links even without a text match. A search
-page with no match does not prove the work is absent.
-
-If the issue mentions specific PRs, check their status:
-```bash
-gh pr view <PR_N> --repo <implementation-repo> --json state,mergedAt,title,mergeCommit,baseRefName
-```
-
-### 3. Verify implementation
-
-Fetch the implementation repository's current base branch. For each merged candidate, verify its merge commit is present and inspect the current code for the claimed behavior; a commit message match alone is not proof:
-```bash
-git fetch <implementation-remote> <base-branch>
-git merge-base --is-ancestor <merge-commit> <implementation-remote>/<base-branch>
-```
-
-If the issue describes a specific defect, inspect the fetched base's code rather than assuming the current checkout contains it:
-```bash
-git show <implementation-remote>/<base-branch>:crates/<relevant>/src/<file>.rs
-```
-
-For issues with checklists, verify each item individually. If sub-items are tracked as separate issues, check those too:
-```bash
-gh issue view <SUB_N> --repo <owner/repo> --json state
-```
-
-### 4. Determine verdict
-
-- **All items fixed and merged**: Recommend closing; name the verified PRs and behavior.
-- **Some items fixed, some remaining**: Keep open; report each remaining item.
-- **Not yet implemented**: Keep open; report what remains.
-- **Superseded or no longer relevant**: Recommend closing with evidence.
-
-### 5. Take action
-
-For a status-only request, return the assessment without GitHub writes. If commenting, closing, or label edits are authorized, perform only those actions; do not ask again for authority already given. Prepare the final assessment before asking for any missing authority. Write `rustfs/backlog` issue content in Chinese.
-
-Close with comment:
-```bash
-gh issue close <N> --repo <owner/repo> --comment "<body>"
-```
-
-Comment without closing:
-```bash
-gh issue comment <N> --repo <owner/repo> --body-file /tmp/triage.md
-```
-
-Update labels only when label changes are authorized, using existing repository labels; never add tool-specific labels:
-```bash
-gh issue edit <N> --repo <owner/repo> --add-label "<existing-label>"
-```
-
-Always use `--body-file` for multiline content, never inline `--body`.
-
-### 6. Handle multi-issue batches
-
-When the user asks to check multiple issues (e.g., "check all issues by user X" or "scan backlog for closable issues"):
-1. List the full requested scope with pagination (for example `gh api --paginate 'repos/<repo>/issues?state=open&per_page=100'`, excluding entries with `pull_request`). Add an author filter only when the user requested one; the default page/limit is not evidence that all issues were checked.
-2. For each issue, run steps 1-5 above.
-3. Report a summary table of all triaged issues with verdicts.
-
-## Report
-
-Identify the issue and current state, verified implementation/PR evidence,
-remaining items, verdict, and action actually taken. Use a table for batches;
-a single issue does not require a heading for each field. Follow step 4's
-verdicts without repeating the assessment in another template.
-
-## Notes
-
-- The user may ask in Chinese ("是否可以关闭", "检查完成情况"); respond in the same language.
-- When closing, always include a summary of what was fixed and which PRs resolved it — this creates a useful audit trail.
-- For issues in `rustfs/backlog`, use `--repo rustfs/backlog`.
-- For issues in `rustfs/rustfs`, use `--repo rustfs/rustfs`.
-- If the issue has sub-issues (GitHub sub-issues API), check each one's state before declaring the parent complete.
+Report the issue, current state, implementation evidence, remaining work, verdict, and action actually taken. Use a table for batches. Write multiline comments via `--body-file`; when closing, post the prepared comment before closing without an inline multiline body.

.agents/skills/pr-review/SKILL.md

@@ -1,93 +1,20 @@
 ---
 name: pr-review
-description: Review a GitHub PR from a URL or number using its actual base/head and risk-appropriate code review. Use when the user asks for a PR review, not a status lookup or PR wording edit. Publish a review only when authorized; delegation and monitoring follow the requested scope and root AGENTS.md.
+description: Review a GitHub PR using its exact base/head and repository risk policy. Use for requested PR reviews; publish or fix only within conversation authorization.
 ---
-
 # PR Review
 
-Use this skill for PR context and review delivery. An ordinary review request is read-only unless the conversation also authorizes posting or fixes. Reuse that authorization without asking again; prepare the review before requesting any missing publication approval.
+1. Resolve the PR repository and remote; the current checkout may belong elsewhere. Read the PR purpose and linked issues, then fetch the actual base and PR head:
 
-## Prerequisites
+   ```bash
+   gh pr view <N> --repo <owner/repo> --json title,author,state,body,baseRefName,headRefName,baseRefOid,headRefOid
+   git fetch <repo-remote> <baseRefName> refs/pull/<N>/head
+   git diff <baseRefOid>...<headRefOid>
+   ```
 
-- Follow root `AGENTS.md`; classify risk with the [review policy](../../references/adversarial-validation.md) and consult relevant [change-style and boundary rules](../../references/implementation.md).
-- Select `code-change-verification` for ordinary review or `adversarial-validation` for explicitly adversarial, substantial, or high-risk review; do not run both on the same diff.
-
-## Workflow
-
-### 1. Gather PR context
-
-```bash
-gh pr view <N> --repo <owner/repo> --json title,author,state,body,additions,deletions,changedFiles,commits,baseRefName,headRefName,baseRefOid,headRefOid
-gh pr diff <N> --repo <owner/repo> --name-only
-```
-
-Read the PR body and linked issues to understand the change's purpose. If the PR references an issue, fetch that too:
-```bash
-gh issue view <ISSUE> --repo <issue-owner/repo> --json title,body,state
-```
-
-### 2. Fetch the diff and classify the change
-
-```bash
-git fetch <repo-remote> <baseRefName> refs/pull/<N>/head
-git diff <baseRefOid>...<headRefOid> --stat
-```
-
-Resolve `<repo-remote>` to the PR repository; do not assume the current checkout's `origin` or `main` matches. Record the exact base/head used. If either moved during fetching, refresh the snapshot before reviewing. Classify using the repository review policy; instruction changes that affect agent execution are mechanical, not exempt.
-
-### 3. Review the changed behavior
-
-Group files by functional area to trace callers and invariants. Use the root risk tier's review shape and only matching lenses. File count does not authorize delegation. When delegation is explicitly authorized, high-risk/substantial reviews use exactly two independent reviewers with the applicable lenses split between them; otherwise use two fresh sequential passes. Reviewers do not spawn further agents.
-
-Findings need a concrete failure scenario with `file:line`; a null verdict briefly names the relevant probes. Reuse existing evidence and choose local checks from the final diff under the root verification policy.
-
-### 4. Check CI status
-
-```bash
-gh pr checks <N> --repo <owner/repo>
-```
-
-Investigate a failed check when it bears on a finding or the user requested CI diagnosis/merge readiness:
-```bash
-gh run view --repo <owner/repo> --log-failed --job=<JOB_ID>
-```
-
-Use current evidence to distinguish pre-existing, flaky, and PR-caused failures. Do not classify them by guesswork or turn a code-only review into unrelated CI repair.
-
-### 5. Synthesize findings
-
-Report the PR, reviewed base/head, and risk tier, then summarize the assessment.
-Use the selected review's P0–P3 ratings and root finding standard: supported
-findings with `file:line`, failure scenario, and fix, or `No findings`.
-State the observed check status, including pending or unavailable checks, and
-the verdict (`APPROVE`, `REQUEST_CHANGES`, or `COMMENT`). Do not infer a pass
-from missing checks or add style nits to populate a clean review.
-
-### 6. Post the review
-
-Only when posting is authorized, write the review body to a temp file and post via CLI. Refresh the PR head first; if it changed, review the delta and update the verdict before posting:
-```bash
-# Request changes
-gh pr review <N> --repo <owner/repo> --request-changes --body-file /tmp/pr_review.md
-
-# Approve
-gh pr review <N> --repo <owner/repo> --approve --body-file /tmp/pr_review.md
-
-# Comment only (no verdict)
-gh pr review <N> --repo <owner/repo> --comment --body-file /tmp/pr_review.md
-```
-
-For authorized inline comments, use [the submission example](references/posting.md).
-
-Always use `--body-file` or `--input`, never inline multiline `--body`.
-
-### 7. Handle follow-up
-
-Follow the [PR lifecycle](../../references/pull-requests.md) and any explicit monitoring request. For follow-up, fetch the new head and compare the recorded reviewed SHA with the new SHA; revisit affected callers and findings. Never use an unfetched `origin/pull/<N>/head` ref as evidence. Update the posted review or resolve addressed threads only within existing authorization.
-
-## Notes
-
-- The user may ask for review in Chinese; respond in the same language but keep the review body in English per AGENTS.md rules.
-- When the user asks for "多角色对抗 review", run the full adversarial validation protocol — this skill's step 3 covers that.
-- If the PR is from a fork, check `maintainerCanModify` before attempting to push fixes.
-- For very large PRs, batch the review by functional area while keeping the same bounded review shape.
+   Record both SHAs; refresh if either moved during fetching. Follow relevant [implementation boundaries](../../references/implementation.md).
+2. Apply the [risk policy](../../references/adversarial-validation.md). Ordinary reviews use the root finding standard directly. For explicit adversarial, substantial, or high-risk reviews, use [the domain probes](../adversarial-validation/SKILL.md). File count does not authorize delegation; use exactly two independent reviewers only when authorized, otherwise two fresh sequential passes for that risk tier. Reviewers do not spawn further agents.
+3. Check `gh pr checks <N> --repo <owner/repo>`. Investigate failures when relevant to a finding or a requested CI/readiness diagnosis. Distinguish pending, unavailable, pre-existing, flaky, and PR-caused states using evidence; a code review does not authorize unrelated repairs.
+4. Report the reviewed SHAs, risk tier, supported findings or `No findings`, material verification limits, check state, and verdict. P0/P1 findings block approval. Severity: P0 = demonstrated data loss, security breach, remote crash, or deadlock; P1 = correctness, compatibility, or material hot-path regression; P2 = a concrete maintainability defect; P3 = optional style, only when requested.
+5. When posting is authorized, follow [Git and PR rules](../../references/pull-requests.md), recheck the remote head, and review any delta before submitting. Use `gh pr review` with `--approve`, `--request-changes`, or `--comment` and `--body-file`; use [the API example](references/posting.md) for authorized inline comments. Reuse existing authorization.
+6. Follow the repository PR lifecycle and explicit monitoring scope. Fetch new heads before comparing to the reviewed SHA; never rely on an unfetched `origin/pull/<N>/head`. Push fixes or resolve threads only when authorized; check `maintainerCanModify` before a fork push.

.agents/skills/rust-code-quality/SKILL.md

@@ -1,116 +0,0 @@
----
-name: rust-code-quality
-description: Run a focused Rust quality review when the user requests one or a selected review workflow needs Rust-specific checks for changed behavior. Do not auto-load for every implementation edit, comment-only or formatting-only Rust diff, or repeat an already completed review.
----
-
-# Rust Code Quality Gate
-
-Use this skill for a dedicated Rust review to cover rules that `cargo clippy`
-does not catch.
-
-Search matches and checklist items are candidates, not findings. Apply the root
-finding standard; distinguish a demonstrated bug, an explicit rule violation,
-and an optional preference. P2/P3 suggestions do not need to be invented or
-included in an otherwise clean correctness review.
-
-## Quick Start
-
-1. Identify changed `.rs` files.
-2. Run the matching candidate searches on changed files.
-3. Apply the manual checklist sections whose behavior the diff touches.
-4. Report or rebut every finding with evidence; P0/P1 findings block approval.
-   Fix them when implementation is authorized; a read-only review reports them.
-
-## Automated Checks
-
-Use these searches to find candidates in changed `.rs` files. Inspect syntax,
-`#[cfg(test)]` scope, and the changed hunk before reporting a finding; text
-filters do not reliably distinguish production code from tests.
-
-```bash
-# 1. unwrap/expect candidates
-rg -n '\.unwrap\(\)|\.expect\(' <changed-files>
-
-# 2. Silent type truncation via `as` cast
-rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
-
-# 3. String as error type
-rg -n 'Result<.*String>' <changed-files>
-
-# 4. Box<dyn Error> in public APIs
-rg -n 'Box<dyn.*Error' <changed-files>
-
-# 5. println/eprintln in production
-rg -n 'println!|eprintln!' <changed-files>
-
-# 6. Ordering::Relaxed usage (verify each is intentional)
-rg -n 'Ordering::Relaxed' <changed-files>
-
-# 7. Default substituted for a possibly-required value (judge each: is the value optional by domain?)
-rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
-```
-
-## Manual Review Checklist
-
-For the Rust diff under review, verify:
-
-### Error Handling
-- [ ] Every production `unwrap()` or `expect()` is infallible by type or a checked invariant; explain only non-obvious invariants, using an existing type, a useful `expect` message, or a concise comment
-- [ ] No `Result<_, String>` in public API signatures
-- [ ] Public library APIs use domain errors unless deliberate error erasure at a boundary is part of the contract
-- [ ] `Error::source()` is overridden when inner error is stored
-- [ ] Error messages are actionable without exposing secret input
-
-### Type Safety
-- [ ] No silent `as` truncation (negative→unsigned, large→small)
-- [ ] Fallible numeric conversions use `TryFrom`/`try_into()` and return a typed error; clamp or saturate only when the domain explicitly requires it
-- [ ] Floating-point to integer conversion validates finiteness, sign, and range before conversion
-
-### Concurrency
-- [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
-- [ ] No `tokio::sync` lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
-- [ ] Atomic read-modify-write uses the direct `fetch_*` operation when possible; use `compare_exchange` only for conditional updates
-- [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
-
-### Memory and Performance
-- [ ] On an identified hot path, report cloning or allocation only with a concrete per-request/per-object cost or benchmark signal
-- [ ] Prefer borrowing, moving, `Bytes`/`Arc`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs
-
-### Recursion Safety
-- [ ] Recursion over untrusted, persisted, or otherwise unbounded input has a depth limit or uses iterative traversal
-- [ ] Tree/cache traversals handle corrupted/cyclic input safely
-
-### Testing
-- [ ] Tests have an observable failure criterion; delegated assertions, `#[should_panic]`, snapshot/property checks, and meaningful `Result` failures do not need a redundant `assert!`
-- [ ] Use `expect` only when its message improves failure diagnosis; do not add boilerplate to self-evident test setup
-- [ ] Test volume and line count are never treated as production-code growth
-
-### Serde
-- [ ] Structs from untrusted input reject unknown fields where the compatibility contract permits; otherwise validate security-critical fields explicitly and test the supported input shape
-- [ ] `#[serde(default)]` not used on security-critical fields without validation
-
-### Code Hygiene
-- [ ] No `#![allow(dead_code)]` at crate root
-- [ ] No camelCase statics or Hungarian notation
-- [ ] New string literals don't duplicate existing constants
-
-### Reuse and Necessity
-- [ ] No new helper duplicates `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, a relevant direct dependency, or plain std/tokio behavior; reused helpers match the call site's semantics
-- [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
-- [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
-- [ ] Comments avoid narration and change history while completely stating non-obvious lock, `SAFETY`, durability, compatibility, and unwrap invariants
-- [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
-
-## Severity Classification
-
-- **P0 (Block merge)**: demonstrated data loss, security breach, remote crash, or deadlock
-- **P1 (Must fix)**: concrete correctness, compatibility, or material hot-path regression
-- **P2 (Should fix)**: avoidable duplication or maintainability issue with a concrete simpler replacement
-- **P3 (Nice to fix)**: local style or clarity issue with no behavioral risk
-
-## Output Template
-
-Use the calling review's output format. For a standalone review, report supported
-findings with severity, location, impact, fix, and validation, or `No findings`.
-Include only material unverified checks. Candidate counts are not a quality
-metric and do not need a separate scan report.

.agents/skills/rustfs-logging-governance/SKILL.md

@@ -1,34 +0,0 @@
----
-name: rustfs-logging-governance
-description: Add or review RustFS `tracing` events with the repository field shape, level policy, privacy boundaries, and guardrails. Use when a change adds or edits a tracing macro/instrumentation site or the logging guardrail script.
----
-
-# RustFS Logging Governance
-
-Apply this skill only to changed logging sites; do not turn a local log edit into
-a broad logging cleanup.
-
-## Workflow
-
-1. Read the changed function/module context and classify the site as lifecycle,
-   request/hot path, fallback, external fetch, or summary.
-2. Match neighboring structured events and reuse existing `EVENT_*`,
-   `LOG_COMPONENT_*`, and `LOG_SUBSYSTEM_*` constants.
-3. Put stable fields first (`event`, `component`, `subsystem`, `state`/`result`,
-   then context) and a short label last.
-4. Select the level by operational meaning:
-   - `error`: behavior/security-affecting failure;
-   - `warn`: degraded/fallback/operator-actionable state;
-   - `info`: low-frequency lifecycle/mode change;
-   - `debug`: targeted diagnostics;
-   - `trace`: repetitive request/object/shard success paths.
-5. Never log secrets, tokens, auth headers, credential payloads, raw
-   attacker-controlled bodies, or merged config dumps. Error strings and
-   `Debug` output are log surfaces too.
-6. Prefer one aggregate summary over inventories or startup banners.
-7. Run `./scripts/check_logging_guardrails.sh` and the checks selected by root
-   `AGENTS.md`.
-
-Read [logging-governance.md](references/logging-governance.md) only for a broad
-logging audit, event-model migration, or guardrail expansion. Ordinary single-
-site edits do not require the full workspace scope map.

.agents/skills/rustfs-logging-governance/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "RustFS Logging Governance"
-  short_description: "Standardize RustFS logs with structured events and guardrails."
-  default_prompt: "Use $rustfs-logging-governance to standardize or review RustFS logging, reduce noise, and update guardrails."

.agents/skills/rustfs-release-publish/SKILL.md

@@ -1,165 +1,42 @@
 ---
 name: rustfs-release-publish
-description: "Run the RustFS console gate, source version bump, preview validation, human confirmation, final-tag publication, and post-release installation and website updates. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
+description: Publish a RustFS version through Console, release-branch CI, preview acceptance, human confirmation, and final publication. Use only for an explicit release request.
 ---
-# RustFS Release Publish (preview-validated pipeline)
+# RustFS Release
 
-This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (invoked here with the authorized commit/push/PR scope) with a mandatory preview-tag validation loop before the final tag is published.
+Require the exact final SemVer target before version edits or publication. If unspecified, inspect current releases/tags and ask with concrete candidates; do not guess the channel or bump. A named bump type resolves to an exact version that still needs confirmation. Report the confirmed target verbatim; resolve any later mismatch before proceeding. Independent read-only preflight may continue.
 
-The binary reports its build tag (`build::TAG` via shadow_rs; `SHORT_VERSION` in
-`rustfs/src/config/cli.rs`), and `build.yml` derives asset names and preview
-classification from that tag. Cargo.toml supplies only the no-tag fallback.
-Preview and final tags must therefore share the validated source commit;
-their tag-dependent version and asset names differ. The channel and cleanup
-constraints are defined once under Preview tag naming and Hard rules below.
+## Invariants
 
-Pipeline shape:
+- Cargo stores `<target>`, never `-preview.N`. Before publication, installation defaults retain the previous available deliverable. Final and preview binaries embed their own tag names, so version/asset names differ despite identical source.
+- Use annotated tags without `v`. Preview names are exactly `<target>-preview.<digits>`; choose the next unused iteration after fetching tags. Preview Releases are prereleases, never Latest; publish only versioned assets/exact Docker tags, never latest assets, R2, Helm, moving Docker channels, or installation/banner updates.
+- `release/<target>` owns the candidate. Retain its history, backport only reviewed fixes, and never merge main wholesale or force-push it. Pin `PREVIEW_HASH` to its exact commit with complete passing CI. Preview and final tags both point there; never tag a later main/branch HEAD or insert another bump.
+- Phase failures block dependents. Keep preview Releases until final publication and notes generation finish; CI removes Releases while retaining tags. Never move an existing preview or final tag.
 
-```
-check console main against its latest Release
-  -> if ahead: publish console -> wait for Release asset + latest API
-  -> bump Cargo.toml and Cargo.lock to <target> -> merge
-  -> tag <preview-tag> at that commit -> CI green
-  -> verify preview Release assets -> run binary locally + console checks
-  -> validate with latest rc client
-  -> report preview acceptance results -> STOP for explicit human confirmation
-  -> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
-  -> CI deletes the <target>-preview.N Releases (tags kept)
-  -> verify published images -> publish Helm chart from the validated source
-  -> update installation references on main -> update rustfs.com announcement
-```
+## Read by Phase
 
-On preview validation failure: fix lands on main via normal PR (Cargo versions are already at `<target>`, no new source bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2. Installation references remain on the previous published deliverable throughout preview validation.
+Read only the current phase's procedure; retain completed evidence across turns. Follow root verification, Git/PR, and authorization rules.
 
-## Required inputs
+| Phase | Procedure |
+|---|---|
+| 0 | Clean tree, fetch main/tags, verify GitHub access, then complete the mandatory [Console release gate](references/console-gate.md). A green Console build alone is insufficient. |
+| 1–2 | [Prepare/reuse the release candidate and run exact-SHA branch CI](references/release-candidate.md). Source version changes use `rustfs-release-version-bump` with authorized delivery scope. |
+| 3 | [Verify the preview build, assets, channels, and notes](references/preview-release.md). |
+| 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. |
+| Confirmation | Complete the gate below and end the turn. |
+| 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. |
+| 7 | [Maintain milestones, installation references, and the website](references/post-release-updates.md) only after complete publication. |
 
-- Final target version, for example `1.0.0-beta.10`.
-- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` after `git fetch --tags`).
+## Human Confirmation
 
-If the target version is missing or ambiguous, collect the current release/tag baseline and ask before version edits or publication. Continue independent read-only preflight while the answer is pending (see the semver gate below).
+After Phases 3–5 pass, recheck that the release branch still equals `PREVIEW_HASH`. Report the target, branch, preview tag/hash, branch CI run, preview Release URL, Console result, and rc matrix. Explicitly ask whether to publish the final tag and end the turn without creating/pushing it.
 
-## Semver gate — resolve the target before version edits or publication
+Only a new affirmative user reply to that exact report authorizes Phase 6. The original release request, earlier confirmation, silence, or automated follow-up does not count. Confirmation binds the target, preview tag, and hash; a changed value or repeated/failed acceptance cycle invalidates it. Revalidate and obtain fresh confirmation before continuing.
 
-Versions follow [SemVer 2.0.0](https://semver.org/). Precedence reminder:
+## Recovery
 
-```
-1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0 < 1.0.1 < 1.1.0 < 2.0.0
-```
+- Before the final tag exists, land fixes on main, backport to the candidate, verify target versions, and restart branch CI/acceptance. Use a new preview iteration once a preview was tagged; main may advance independently.
+- Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases.
+- If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting candidate branches or preview tags requires an explicit decision.
 
-Numeric prerelease identifiers compare numerically (`beta.9 < beta.10`), not lexically — see [semver.org spec item 11](https://semver.org/#spec-item-11). Preview tags are internal validation tags layered on top of the target's prerelease channel — they are never themselves a deliverable version and never appear in version files.
-
-Rules:
-
-- A request like "发个版" / "release the next version" without an exact version string is ALWAYS ambiguous. Derive the current latest tag (`git tag --sort=-v:refname | head`), then ask the user to choose with concrete candidates, e.g. from `1.0.0-beta.10`: next prerelease `1.0.0-beta.11`, promote to `1.0.0-rc.1`, promote to stable `1.0.0`. Never guess between these — they have very different meanings (channel promotion vs. iteration) and different CI classification consequences.
-- After a stable `X.Y.Z` exists, the next version must state which component bumps: patch `X.Y.(Z+1)` for fixes only, minor `X.(Y+1).0` for backward-compatible features, major `(X+1).0.0` for breaking changes. If the user names a bump type but not a number, compute it from the latest stable tag and echo the exact resulting version back for confirmation.
-- Echo the final confirmed version string verbatim in your first status report; every later phase must use exactly that string. If at any point the user's wording and the confirmed version diverge, stop and re-confirm.
-
-## Preview tag naming
-
-- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
-- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
-- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
-- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
-
-## Hard rules
-
-- Before preview, bump only Cargo.toml (workspace package and internal dependency versions) and the corresponding workspace members in Cargo.lock, directly to `<target>`. Keep README installation examples, flake.nix, Chart.yaml, and rustfs.spec on the previous published deliverable until Phase 7. Never write a `-preview.N` suffix into these files; preview identifiers belong to validation tags and artifacts.
-- Publishing a tag alone does not make an installation target available. Verify the exact image manifest, Release assets, and source archive before advancing references that consume them. Helm packaging derives its versions from the final tag and waits for that image; it does not require an early Chart.yaml bump on main.
-- Preview Release assets are versioned and intentionally visible on the Releases page for the duration of validation. Do not label them Latest or use them to update any latest distribution channel.
-- Never delete a preview Release by hand before Phase 6 finishes — Phase 4 downloads its assets and the final Release notes are generated while it still exists. Cleanup is CI's job; only step in manually (`gh release delete "<preview-tag>" --yes`, never `--cleanup-tag`) if `cleanup-preview-releases` failed.
-- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
-- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final. Phase 7 updates installation references on main after publication; neither tag moves to that follow-up commit.
-- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
-- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
-- Phases run in order; a failure blocks dependent steps. For a preview/source acceptance failure before the final tag exists, land the fix, verify Cargo still matches the confirmed target, and restart from Phase 2 with the next preview at the fixed commit. If main has advanced to another target, resolve that source/version mismatch before another preview.
-- Once the final tag exists, preserve its commit. Retry failed publication jobs for that exact tag; do not recreate the tag or restart preview on newer main. If a source change is required after final-tag publication, report the failure and obtain a new target version. Phase 7 failures resume only the failed follow-up after rechecking artifact availability and whether a newer release has superseded it.
-- Completing preview acceptance does not authorize the final tag. After Phases 3–5 pass, report the acceptance evidence and stop until the user explicitly confirms continuation. The original release request, an earlier confirmation, silence, or an automated follow-up does not satisfy this gate.
-- Confirmation is scoped to the reported `<target>`, `<preview-tag>`, and `PREVIEW_HASH`. A failed or repeated acceptance cycle, including any new preview iteration, invalidates prior confirmation and requires a new one.
-- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
-- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
-
-## Phase 0 — Preflight
-
-- `git status --short` clean; `git fetch origin main --tags`.
-- `gh auth status` works; confirm you can view `gh release list -L 3`.
-- Confirm the exact final target version with the user if not explicit.
-
-### Console release gate
-
-Read and complete [the Console gate](references/console-gate.md) before Phase 1. Verify the latest published Console asset and exact commit; if Console main is ahead, complete its release and asset verification first. A successful build alone does not satisfy this gate.
-
-## Phase 1 — Source version bump to the final target (once)
-
-- If Cargo.toml and all workspace members in Cargo.lock already read `<target>` (e.g. this is a restart after a failed preview), verify both and skip the source bump. Installation versions intentionally differ during preview; do not treat them as leftovers. If a previous preparation advanced installation references to an unavailable target, restore those references to the verified published baseline before selecting the next preview commit.
-- Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
-- Get the PR merged into main. Record the resulting main commit:
-
-```bash
-git fetch origin main
-PREVIEW_HASH=$(git rev-parse origin/main)   # must contain the bump PR
-```
-
-`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
-
-## Phase 2 — Publish the preview tag
-
-```bash
-git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
-git push origin "<preview-tag>"
-```
-
-Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
-
-The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
-
-On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
-
-## Phase 3 — CI and preview Release verification
-
-- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
-- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
-- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
-- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
-- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
-
-## Phases 4–5 — Local artifact, Console, and rc acceptance
-
-Read and complete [preview acceptance](references/preview-acceptance.md): verify the downloaded binary's tag/SHA and readiness, exercise Console CRUD with byte-identical download, and pass the full latest-rc command matrix. Any failure blocks final publication. Retain the results for the confirmation gate below.
-
-### Manual confirmation gate
-
-After every Phase 3–5 check passes, report the target, preview tag, `PREVIEW_HASH`, preview Release URL, console result, and rc matrix, then explicitly ask the user whether to publish the final tag. End the turn without creating or pushing `<target>`.
-
-Continue to Phase 6 only after a new user reply explicitly confirms the reported target, preview tag, and commit. A clear affirmative reply to that exact report, such as `确认继续`, is sufficient; if the reply is ambiguous or any reported value changed, ask again.
-
-## Phase 6 — Publish the final tag on the validated commit
-
-No second source version bump. The final tag goes on the exact commit the preview validated:
-
-```bash
-git fetch origin --tags
-git rev-parse "<preview-tag>^{commit}"          # must equal PREVIEW_HASH — abort if not
-git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
-git push origin "<target>"
-```
-
-- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
-- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
-- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
-- Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
-- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
-
-## Phase 7 — Installation references and website announcement
-
-Only after Phase 6 succeeds, complete [post-release updates](references/post-release-updates.md): align installation references on main, then update the existing top banner in `rustfs/rustfs.com`. A failed or incomplete publication leaves both on the previous available version. These follow-up commits never change `PREVIEW_HASH` or either release tag. Track their PR, merge, and deployment states separately from artifact publication; an open PR is not a live website update.
-
-## Output contract
-
-Always report:
-
-- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
-- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
-- Manual confirmation gate status (`WAITING_FOR_CONFIRMATION` or `CONFIRMED`) and its exact target, preview tag, and `PREVIEW_HASH`.
-- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, the rc command matrix, and the preview-Release cleanup result (deleted Releases plus surviving tags).
-- Post-release installation PR and verification results; website banner target, text, link, PR, and observed deployment state. Report any remaining merge authorization or failed deployment explicitly rather than claiming the banner is live.
-- Any deviation from this pipeline and why the user approved it.
+Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization.

.agents/skills/rustfs-release-publish/references/final-release.md

@@ -0,0 +1,22 @@
+# Final Release Publication
+
+Read only after fresh user confirmation of the preview acceptance report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
+
+## Phase 6 — Publish the final tag on the validated commit
+
+No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH`. If it moved after confirmation, the approval no longer applies; validate and obtain confirmation for a new preview. The final tag goes on the exact commit the preview validated:
+
+```bash
+git fetch origin --tags
+git rev-parse "<preview-tag>^{commit}"          # must equal PREVIEW_HASH — abort if not
+git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
+git push origin "<target>"
+```
+
+- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
+- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
+- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
+- Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
+- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
+
+Never pass `--cleanup-tag` when removing preview Releases. Remove them manually only if `cleanup-preview-releases` failed, after final publication and notes generation. Record deleted Releases and surviving tags.

.agents/skills/rustfs-release-publish/references/post-release-updates.md

@@ -1,7 +1,15 @@
-# Post-release Installation and Website Updates
+# Post-release Milestone, Installation, and Website Updates
 
 Read only after the final non-preview tag has passed Phase 6. Tag creation or a green binary build alone is insufficient: the release assets, source archive, container images, Helm package, and applicable latest channels must be available. Preview tags never enter this phase.
 
+## Version milestones
+
+1. Query all pages of both open and closed milestones in the RustFS release repository, matching titles exactly to version strings without a `v` prefix. A failed or incomplete query is not evidence that a milestone is absent; resolve the lookup before making changes.
+2. Close the milestone matching the published `<target>` if it is open; an already closed milestone needs no change. If it is missing, report the gap and continue checking the next version. Do not create a substitute for the released milestone or close/reassign its issues or PRs.
+3. Use an explicitly planned next version when provided; otherwise, for a stable `X.Y.Z`, use `X.Y.(Z+1)`. For example, after `1.0.1` is published, close milestone `1.0.1` and ensure milestone `1.0.2` exists. For an alpha/beta/rc target with a trailing numeric counter, increment that counter within the same channel unless a different next version was specified. If the next version cannot be determined, ask for that version and report this follow-up as blocked. Creating a planning milestone does not confirm the target of a future release or bypass the semver gate.
+4. Reuse an existing next-version milestone, whether open or closed; do not reopen it or create a duplicate. Only when the complete query confirms it is absent, create an open milestone with the exact next-version title. Do not invent a due date or description. After an uncertain creation result, query again before retrying.
+5. Read back both milestones and report their titles, URLs, and actual states, including whether the next milestone was reused or created. A failed close, create, or verification leaves milestone maintenance incomplete; retry only the unfinished operation after refreshing live state. Continue independent installation and website work without moving or republishing the release tag.
+
 ## Installation references
 
 1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.

.agents/skills/rustfs-release-publish/references/preview-release.md

@@ -0,0 +1,24 @@
+# Preview Release Verification
+
+Read for Phase 3 after the candidate passed exact-SHA CI. The [parent release invariants](../SKILL.md) apply.
+
+## Preview channels
+
+- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
+- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
+- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
+- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
+
+## Phase 3 — CI and preview Release verification
+
+- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
+- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
+- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
+- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
+- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
+- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
+
+## Release notes
+
+- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
+- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.

.agents/skills/rustfs-release-publish/references/release-candidate.md

@@ -0,0 +1,37 @@
+# Release Candidate Preparation
+
+Read for Phases 1–2. The [parent release invariants](../SKILL.md) and Console gate apply.
+
+## Candidate Resume Preflight
+
+- `git status --short` clean; `git fetch origin main --tags`.
+- `gh auth status` works; confirm you can view `gh release list -L 3`.
+- Confirm the exact final target version with the user if not explicit.
+- Check whether `release/<target>` already exists before creating it; a resumed release reuses the existing branch and its history. For an in-flight release that already has a preview tag but no branch, anchor the new branch to that tag's commit only after verifying its target versions and prior acceptance evidence.
+
+## Phase 1 — Source version bump to the final target (once)
+
+- If `release/<target>` exists, verify its Cargo.toml and workspace members in Cargo.lock read `<target>` and skip the source bump, even if main has advanced. If the branch versions differ, stop and resolve the mismatch without downgrading main.
+- If no branch exists but an unfinished preview tag does, verify its commit has `<target>` in both version files and skip the bump; Phase 2 anchors the branch to that commit. Stop if those files disagree. Otherwise, if main already has `<target>` in both files, skip the bump. Installation references intentionally retain the previous published deliverable during preview; if they were advanced to an unavailable target, restore that baseline before choosing the next candidate.
+- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
+- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
+
+Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
+
+## Phase 2 — Pin and validate the release candidate, then publish the preview tag
+
+- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
+- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
+- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
+- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
+
+```bash
+git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
+git push origin "<preview-tag>"
+```
+
+Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
+
+The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
+
+On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -1,118 +1,24 @@
 ---
 name: rustfs-release-version-bump
-description: "Prepare Cargo versions for an exact RustFS target, or align installation references after its artifacts are published, with verification and optional commit/push/PR delivery. Use for an explicit version bump or when invoked by the release-publish workflow."
+description: Prepare an exact RustFS Cargo version or align installation references after publication. Use for requested version bumps or the release workflow.
 ---
-# RustFS Release Version Bump
+# Release Version Files
 
-Use this skill to prepare and verify release version files. Commit, push, and PR steps apply only when included in the user's delivery scope; publishing release tags belongs to `rustfs-release-publish`.
+Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those are tag-only. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
 
-## Required inputs
+| Stage | Files and constraints |
+|---|---|
+| `prepare` | Only `Cargo.toml` and `Cargo.lock`: workspace package, internal dependency, and workspace member versions become `<target>`. External dependencies stay unchanged. |
+| `post-release` | Only `README.md`, `README_ZH.md`, `flake.nix`, `helm/rustfs/Chart.yaml`, and `rustfs.spec`. Cargo may already target the next release; leave it untouched. |
 
-- Exact target version, for example `1.0.0-beta.4`.
-- Stage: `prepare` (default) or `post-release`. Infer `post-release` only when the user or parent release workflow explicitly requests installation updates after publication; a normal version bump means `prepare`.
-- Delivery scope: local (`edit/verify`), git (`commit/push`), or GitHub
-  (`commit/push/PR`). Derive it from the conversation; when unspecified, prepare
-  and verify locally without blocking on a delivery question.
+Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Preview preparation retains the previous published installation references; follow-up commits never replace either validated tag.
 
-If target version is missing or ambiguous, stop and ask before editing.
+Packaging policy:
 
-Reject any target version containing `-preview`: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
+- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>` (`beta.N -> 0.N.0`; other versions unchanged). Helm CI derives versions from the final tag, so no early chart bump is required.
+- RPM `Version` is numeric; `Release` is the prerelease suffix or `1` for stable. Expanded `Source0` and the unpack directory must resolve to the exact target including its suffix. Add the changelog entry using current git identity, date, and target; never copy an example identity/date.
+- Changing these mappings requires explicit confirmation.
 
-## Read before editing
+Verification follows root tiers: `prepare` validates Cargo metadata with the updated lockfile and checks every workspace/internal version with no unrelated lockfile updates. `post-release` renders Helm with the verified image, runs `scripts/test_helm_chart_version.sh` for chart version edits, and checks README, Nix, and RPM references. Run `git diff --check`; report unresolved required checks as `BLOCKED` without unrelated fixes.
 
-- `AGENTS.md` (root and nearest path-specific files).
-- `.github/pull_request_template.md` only when preparing a PR.
-- Current branch status and diff against `origin/main`.
-
-## Stage boundaries
-
-`prepare` updates only:
-
-- `Cargo.toml`
-- `Cargo.lock`
-
-`post-release` updates installation and packaging references only:
-
-- `README.md`
-- `README_ZH.md`
-- `flake.nix`
-- `helm/rustfs/Chart.yaml`
-- `rustfs.spec`
-
-During preview, installation references intentionally retain the previous published deliverable. Do not sweep them into a source bump to make every version string match. `flake.nix` builds local source, but its package label is aligned with the other packaging references after publication.
-
-Before `post-release` edits, verify that the exact non-preview target has a published GitHub Release, downloadable assets and source archive, and a pullable `rustfs/rustfs:<target>` image manifest. If any prerequisite is missing, report `BLOCKED` and leave installation references unchanged. If a newer deliverable has already superseded this target, do not downgrade installation defaults during a retry.
-
-Helm CI derives chart versions from the triggering tag, so the final tag can retain the previous Chart.yaml values. Update the repository copy only after the target image and published chart are available. Neither post-release changes nor their merge commit may replace the preview-validated final tag.
-
-## Hard release policy
-
-- Docker doc tags use `<version>` (for example `rustfs/rustfs:1.0.0-beta.4`), not `v<version>`.
-- Derive Helm chart and app versions with `scripts/helm_chart_version.sh <target>`; the existing mapping is `beta.N -> 0.N.0`, with other target versions retained.
-- `rustfs.spec` `Release` uses the prerelease suffix (for example `beta.4`), or `1` for a stable release.
-- Do not change these rules without explicit confirmation.
-
-## Step-by-step workflow
-
-1. Confirm intent and isolate scope
-- Use the exact target and delivery scope already supplied; ask only for a missing or ambiguous target or a material release-policy choice.
-- Inspect current branch and ensure only release-related files are touched for this task.
-
-2. Update workspace versions (`prepare` only)
-- Bump `[workspace.package].version` in `Cargo.toml`.
-- Bump internal workspace crate dependency versions in `Cargo.toml`.
-- Update `Cargo.lock` so workspace package versions match target version.
-- Re-scan Cargo.toml and workspace members in Cargo.lock for partial leftovers; leave external dependency versions unchanged.
-
-3. Update installation references (`post-release` only)
-- `README.md` and `README_ZH.md`: update versioned Docker examples to target version.
-- `flake.nix`: update package version to target version.
-- `helm/rustfs/Chart.yaml`: use the app and chart versions returned by `scripts/helm_chart_version.sh <target>`.
-- `rustfs.spec`:
-- Set `Version` to the numeric version and `Release` to the prerelease suffix (example `beta.4`), or `1` for a stable release. Verify that `Source0` and the unpacked source directory resolve to the exact published target, including its prerelease suffix when present.
-- Add/update top changelog entry with exact format:
-- `* Thu May 20 2026 houseme <housemecn@gmail.com>`
-- `- Update RPM package to RustFS 1.0.0-beta.4`
-- Changelog identity and time must come from current environment:
-- `git config --get user.name`
-- `git config --get user.email`
-- `date '+%a %b %d %Y'`
-- Changelog version text must match target release version exactly.
-
-4. Verify before shipping
-- Follow the root verification tiers for the final diff instead of running a full-workspace gate for version strings.
-- For `prepare`, validate Cargo metadata with the updated lockfile and confirm workspace package/internal dependency versions agree. Do not accept a lockfile containing unrelated dependency updates.
-- For `post-release`, render the Helm chart and confirm its default image is the verified target; run `scripts/test_helm_chart_version.sh` when chart versions change. Check the README image tags, Nix package version, and expanded RPM source URL against that same target.
-- Run `git diff --check` for either stage. Report unresolved required checks as `BLOCKED`; do not silently widen scope to fix unrelated issues.
-
-5. Commit strategy (only when committing is authorized)
-- Use `chore(release): prepare <version>` for `prepare` and `chore(release): align installation references for <version>` for `post-release`.
-- These stages happen on opposite sides of publication; do not combine them in a preview-preparation commit or PR.
-- Stage only intended release files; do not include unrelated working tree changes.
-
-6. Push and PR (only for the authorized delivery scope)
-- Push branch:
-- Use the user-requested or configured push remote: `git push -u <push-remote> <branch>` (first push), or `git push` when tracking is already configured.
-- Create PR with template headings unchanged:
-- `gh pr create --base main --head <branch> --title ... --body-file ...`
-- PR title/body must be English.
-- Use `N/A` for non-applicable template sections.
-- Include verification commands and any `BLOCKED` reason clearly.
-
-## Recommended check commands
-
-- `git status --short --branch`
-- `git diff --name-only origin/main...HEAD`
-- `git diff --stat origin/main...HEAD`
-- `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
-
-## Output contract
-
-When using this skill, always report:
-
-- Target version and stage.
-- Files changed.
-- Any assumptions or uncertainties requiring confirmation.
-- Verification result (`PASSED` or `BLOCKED`) with key evidence.
-- Commit message(s) used.
-- Push status and PR URL when GitHub flow is requested.
+For authorized commits, keep stages separate: `chore(release): prepare <target>` or `chore(release): align installation references for <target>`. Report target/stage, changed files, verification, material uncertainty, and actual commit/push/PR state.

.agents/skills/test-coverage-improver/SKILL.md

@@ -1,57 +0,0 @@
----
-name: test-coverage-improver
-description: Analyze a supplied coverage report or perform an explicitly requested RustFS coverage assessment, rank uncovered risks, and propose focused tests. Do not trigger for ordinary implementation verification, a single regression test, documentation wording, or release preparation without a coverage request.
----
-
-# Test Coverage Improver
-
-Use this skill when you need a prioritized, risk-aware plan to improve tests from coverage results.
-
-## Usage assumptions
-- Focus scope is either changed lines/files, a module, or the whole repository.
-- Reuse a supplied coverage artifact when its revision, scope, and format match.
-- If required context is missing, call out assumptions explicitly before proposing work.
-
-## Workflow
-
-1. Define scope and baseline
-   - Derive the revision and scope from the request, diff, or supplied report.
-   - Default to the affected files/module; whole-workspace coverage requires that
-     scope in the request. Ask only if a wrong scope would change the result.
-
-2. Obtain coverage evidence
-   - First inspect a matching existing artifact; do not regenerate it merely
-     because this skill was selected.
-   - If measurement is needed, read the Coverage section of
-     [the testing guide](../../../docs/testing/README.md#coverage), check disk
-     space/tool availability, and select package/test-scoped `cargo llvm-cov`
-     using the repository's nextest configuration. `make coverage` measures the
-     whole workspace (excluding E2E) and is only for that requested scope.
-   - Collect only metrics the report supports. Missing branch/changed-line
-     coverage is unknown, not zero.
-   - If measurement cannot run, continue with code-based test proposals and
-     mark measured coverage unverified; do not invent a coverage percentage.
-
-3. Rank highest-risk gaps
-   - Prioritize changed code, branch coverage gaps, and low-confidence boundaries.
-   - Apply the risk rubric in [coverage-prioritization.md](references/coverage-prioritization.md).
-   - Report up to 5–8 evidenced gaps; do not pad a small scope.
-   - For each gap, capture: file, lines, uncovered branches, and estimated risk score.
-
-4. Propose high-impact tests
-   - For each gap, name the behavior and regression, distinguishing assertions,
-     relevant normal/edge/failure cases, necessary setup, and estimated effort.
-   - Include only scenarios and setup that apply; reuse shared fixture details.
-
-5. Close with validation plan
-   - State which gaps remain after proposals.
-   - Give a scoped verification command and behavior-based acceptance criterion;
-     use a coverage threshold only when the task or repository requires one.
-   - List assumptions or blockers (environment, fixtures, flaky dependencies).
-
-## Report
-
-Summarize the supported metrics, then combine each ranked gap with its proposed
-test and validation. Include source lines only when supplied or inspected;
-mark missing metrics or locations as unknown. Do not duplicate gaps and tests
-in separate templates or fill empty categories for an otherwise small report.

.agents/skills/test-coverage-improver/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "Test Coverage Improver"
-  short_description: "Find top uncovered risk areas and propose high-impact tests."
-  default_prompt: "Use $test-coverage-improver to analyze coverage for the requested scope, reuse matching reports, and propose tests for evidenced risks."

.agents/skills/test-coverage-improver/references/coverage-prioritization.md

@@ -1,25 +0,0 @@
-# Coverage Gap Prioritization Guide
-
-Use this rubric for each uncovered area.
-
-Score = (Criticality × 2) + CoverageDebt + (Volatility × 0.5)
-
-- Criticality:
-  - 5: authz/authn, data-loss, payment/consistency path
-  - 4: state mutation, cache invalidation, scheduling
-  - 3: error handling + fallbacks in user-visible flows
-  - 2: parsing/format conversion paths
-  - 1: logging-only or low-impact utilities
-
-- CoverageDebt:
-  - 0: 0–5 uncovered lines
-  - 1: 6–20 uncovered lines
-  - 2: 21–40 uncovered lines
-  - 3: 41+ uncovered lines
-
-- Volatility:
-  - 1: stable legacy code with few recent edits
-  - 2: changed in last 2 releases
-  - 3: touched in last 30 days or currently in active PR
-
-Sort by score descending, then by business impact.

.config/e2e-distributed-selection.txt

@@ -1,2 +1,2 @@
-sha256-linux=6fc377fa1f9f06e065f077c7bd53d04efbccf4453f95dde67afaa47778185377
-sha256-darwin=6fc377fa1f9f06e065f077c7bd53d04efbccf4453f95dde67afaa47778185377
+sha256-linux=233bd7a68777eda1f044b6cb77128056be2b6492583c0dfd114b21e98a6102ef
+sha256-darwin=233bd7a68777eda1f044b6cb77128056be2b6492583c0dfd114b21e98a6102ef

.config/e2e-full-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=3b6ff112ff16acbaa4148c4736095a1b6b71cc11af0dd3513ff28c6bdc334ca8
-sha256-linux=4e8bfc01870565a0a608a1634e6919fc593ce120a970aa04d974f7edf2675c21
+sha256-darwin=98dff0961c6eb21d6128bfd2d4a6100a624ddf1fb7db05dba360a5d3d0a310c2
+sha256-linux=9ff0da7406de9b89c0e848ff29f2238b30ba9da89c7e9df97c246ce86631350f

.config/e2e-smoke-selection.txt

@@ -1 +1 @@
-sha256=476cb92aa9e385c9bb509ee5df3e5bd07e8c929730040b60d16791b6c0616d00
+sha256=ba8f392b617cf9f2338142233a417e43927b28096635e628233ce8a60cf49d10

.config/ecstore-required-tests.json

@@ -55,6 +55,211 @@
       "invariant": "odm-source-contract-gcs",
       "suite": "rustfs",
       "name": "on_demand_migration::gcs::tests::gcs_native_backend_satisfies_the_shared_backend_contract"
+    },
+    {
+      "invariant": "rollback-inspection",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::io_primitives::tests::rename_rollback_incomplete_preserves_overwrite_data_dirs_and_staging"
+    },
+    {
+      "invariant": "rollback-cancellation",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::io_primitives::tests::rename_rollback_incomplete_cancelled_task_is_not_success"
+    },
+    {
+      "invariant": "rollback-completion-equivalence",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::io_primitives::tests::rename_rollback_incomplete_matches_early_ack_and_full_wait_after_reopen"
+    },
+    {
+      "invariant": "rollback-physical-owner",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::io_primitives::tests::rename_rollback_children_keep_namespace_ownership_after_coordinator_panic"
+    },
+    {
+      "invariant": "late-ack-retains-commit",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::io_primitives::tests::rename_data_early_ack_post_mutation_tail_error_never_rolls_back_commit"
+    },
+    {
+      "invariant": "tail-drained-cas",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::tail_drained_put_waits_for_tail_and_allows_immediate_cas"
+    },
+    {
+      "invariant": "borrowed-lock-tail",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::no_lock_put_waits_for_rename_tail_under_outer_guard"
+    },
+    {
+      "invariant": "cancelled-writer-owner",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::tail_drained_put_owned_commit_survives_waiter_cancellation"
+    },
+    {
+      "invariant": "get-body-reconstruction",
+      "suite": "rustfs-ecstore",
+      "name": "ecstore_validation_blackbox::blackbox_get_restores_body_after_one_shard_file_is_removed"
+    },
+    {
+      "invariant": "range-reconstruction",
+      "suite": "rustfs-ecstore",
+      "name": "ecstore_validation_blackbox::blackbox_range_read_restores_exact_slice_with_one_offline_disk"
+    },
+    {
+      "invariant": "multipart-abort-linearization",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::multipart::tests::abort_and_complete_linearize_for_plain_sse_and_legacy_layouts"
+    },
+    {
+      "invariant": "multipart-write-quorum",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::multipart::tests::abort_enforces_delete_write_quorum_boundary"
+    },
+    {
+      "invariant": "multipart-physical-owner",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::multipart::tests::complete_multipart_timeout_keeps_namespace_owner_until_physical_publication"
+    },
+    {
+      "invariant": "restore-abort-cleanup",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::object::transition_commit_failure_tests::multipart_restore_aborts_every_post_create_failure"
+    },
+    {
+      "invariant": "legacy-metadata",
+      "suite": "rustfs-filemeta",
+      "name": "filemeta::test::test_issue_2288_legacy_xlmeta_compatibility"
+    },
+    {
+      "invariant": "legacy-nil-version",
+      "suite": "rustfs-filemeta",
+      "name": "filemeta::test::test_into_fileinfo_reads_legacy_nil_uuid_inline_key"
+    },
+    {
+      "invariant": "commit-phase-schedule",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::io_primitives::tests::commit_fault_schedule_keeps_multiple_disk_phases_independent"
+    },
+    {
+      "invariant": "commit-phase-ownership",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::io_primitives::tests::commit_fault_schedule_rejects_duplicate_checkpoint_without_disarming_owner"
+    },
+    {
+      "invariant": "metadata-observation-all-four-slot-states-and-arrival-orders-preserve-reduction",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::metadata_quorum::tests::metadata_observation_all_four_slot_states_and_arrival_orders_preserve_reduction"
+    },
+    {
+      "invariant": "metadata-observation-pending-newer-version-and-same-time-different-directory-force-full-wait",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::metadata_quorum::tests::metadata_observation_pending_newer_version_and_same_time_different_directory_force_full_wait"
+    },
+    {
+      "invariant": "metadata-observation-duplicate-shard-cannot-supply-a-read-reserve",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::metadata_quorum::tests::metadata_observation_duplicate_shard_cannot_supply_a_read_reserve"
+    },
+    {
+      "invariant": "metadata-observation-null-versions-markers-and-invalid-success-keep-their-meaning",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::metadata_quorum::tests::metadata_observation_null_versions_markers_and_invalid_success_keep_their_meaning"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-static-namespace",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_static_namespace"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-cross-subdirectory-limit",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_cross_subdirectory_limit"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-max-and-max-plus-one",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_max_and_max_plus_one"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-fixed-seed-marker-and-prefix",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_fixed_seed_marker_and_prefix"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-offline-corrupt-and-minority-stale",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_offline_corrupt_and_minority_stale"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-versions-delete-markers-and-free-versions",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_versions_delete_markers_and_free_versions"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-orphan-cleanup-requires-authoritative-evidence",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_orphan_cleanup_requires_authoritative_evidence"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-consumer-drop-joins-real-disk-producer",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_consumer_drop_joins_real_disk_producer"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-cancelled-quorum-walk-releases-senders-and-disk-lock",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_cancelled_quorum_walk_releases_senders_and_disk_lock"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-enabled-index-contracts",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_enabled_index_contracts"
+    },
+    {
+      "invariant": "borrowed-context-boundary",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::borrowed_write_context_rejects_wrong_namespace_and_cache_flag"
+    },
+    {
+      "invariant": "borrowed-put-cancellation",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::borrowed_tail_drained_put_keeps_owner_after_caller_cancel"
+    },
+    {
+      "invariant": "borrowed-mpu-cancellation",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::multipart::tests::borrowed_complete_keeps_namespace_owner_after_caller_cancel"
+    },
+    {
+      "invariant": "pool-cas-owner",
+      "suite": "rustfs-ecstore",
+      "name": "core::pools::pools_tests::pool_meta_cas_context_retains_actual_owner_for_exact_target"
+    },
+    {
+      "invariant": "pool-cas-released-owner",
+      "suite": "rustfs-ecstore",
+      "name": "core::pools::pools_tests::pool_meta_cas_rejects_released_owner"
+    },
+    {
+      "invariant": "copy-tail-drained",
+      "suite": "rustfs-ecstore",
+      "name": "store::object::tests::borrowed_copy_tail_drained_waits_for_rename_tail"
+    },
+    {
+      "invariant": "catalog-borrowed-publication",
+      "suite": "rustfs",
+      "name": "admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition"
+    },
+    {
+      "invariant": "directory-marker-quorum",
+      "suite": "rustfs-filemeta",
+      "name": "metacache::tests::resolve_directory_marker_and_prefix_preserve_quorum_in_both_orders"
+    },
+    {
+      "invariant": "logical-prefix-limit",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::test::list_path_gather_results_counts_common_prefixes_before_page_limit"
     }
   ],
   "fixtures": [
@@ -82,6 +287,21 @@
       "path": "crates/ecstore/tests/fixtures/minio/bucket_metadata_full.xlmeta.hex",
       "sha256": "3b6de589519c08a1614c8bd409bb8199c17d42043861b07bce513075e6fbfc12",
       "source": "MinIO RELEASE.2025-07-23T15-54-02Z; crates/ecstore/tests/fixtures/minio/README.md"
+    },
+    {
+      "path": "crates/filemeta/tests/fixtures/issue_2288_legacy_xlmeta.hex",
+      "sha256": "b3e8d32f9950a38b5dc71f679519cc830a2b2a807e8b0b39f78840b238d8eacf",
+      "source": "Legacy xl.meta captured for rustfs/backlog#2288; filemeta::test::test_issue_2288_legacy_xlmeta_compatibility"
+    },
+    {
+      "path": "crates/filemeta/tests/fixtures/issue_2265_legacy_meta_v2_object.hex",
+      "sha256": "89dd7ec92d16e54b5d31e0c49c9edffff8a7aa58890aaeff59bc2ff84da0a62f",
+      "source": "Legacy metadata v2 object captured for rustfs/backlog#2265; crates/filemeta/src/test_data.rs"
+    },
+    {
+      "path": "crates/ecstore/tests/fixtures/list_namespace_keys.json",
+      "sha256": "5afddb60565deba0938e617470255563f0e15a15fab699645c59bb7ccc84898e",
+      "source": "Shared static LIST namespace corpus; ECStore differential tests and ODM list_through_static_namespace_boundary_matrix"
     }
   ]
 }

.config/nextest.toml

@@ -64,6 +64,26 @@ command = ['sh', '-c', 'echo RUST_MIN_STACK=33554432 >> "$NEXTEST_ENV"']
 command = ['sh', '-c', 'echo RUST_MIN_STACK=33554432 >> "$NEXTEST_ENV"']
 
 # --- default profile (local): serialize the flaky groups, never retry --------
+# These real-disk progress oracles exercise fsync, quorum IO, and crash/replay
+# boundaries. Reserve the run's capacity so unrelated fixtures cannot starve
+# their existing deadlines and typed-failure assertions (rustfs/backlog#2706).
+# Keep each test's internal concurrency and zero-retry policy unchanged.
+[[profile.default.overrides]]
+filter = 'package(rustfs) & test(/^connect::diagnostics::schedule::tests::/)'
+threads-required = "num-test-threads"
+
+[[profile.default.overrides]]
+filter = 'package(rustfs-ecstore) & (test(/^set_disk::tests::streaming_get_blocks_concurrent_/) | test(=store::object::tests::retired_marker_converges_after_bucket_recreation_on_sixteen_disks))'
+threads-required = "num-test-threads"
+
+[[profile.default.overrides]]
+filter = 'package(rustfs-heal) & (binary(mrf_partial_write_test) | (binary(mrf_pipeline_test) & test(/^journal_replay_survives_/)) | test(=admin_selector_survives_sigkill_with_original_scope_and_receipts))'
+threads-required = "num-test-threads"
+
+[[profile.default.overrides]]
+filter = 'package(rustfs-scanner) & (test(/^scanner::backlog::tests::native_(writer|retirement)_/) | test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(/^scanner_io::tests::service_cohort::/))'
+threads-required = "num-test-threads"
+
 [[profile.default.scripts]]
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
@@ -82,6 +102,13 @@ setup = 'lifecycle-large-stack'
 filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
 setup = 'ecstore-large-stack'
 
+# These real-disk scanner fixtures assert quiet-workload publication and
+# durable MRF admission. Reserve the pool while their bounded CAS/fsync
+# handoffs run so unrelated disk fanouts cannot consume that fixture budget.
+[[profile.default.overrides]]
+filter = 'package(rustfs-scanner) & (test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(=scanner_io::tests::multi_pool_scanner_cycle_publishes_combined_usage) | test(=scanner_io::tests::pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage) | test(=scanner_folder::tests::mrf_ownership::checkpoint_fixture_metadata_heal_reaches_budgeted_tail))'
+threads-required = "num-test-threads"
+
 [[profile.default.overrides]]
 filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
 test-group = 'ecstore-serial-flaky'
@@ -283,6 +310,24 @@ fail-fast = false
 # marker is the observable signal the flake policy is built around.
 path = "junit.xml"
 
+# Match the local durable-progress reservations. All members, deadlines,
+# assertions, and internal concurrency remain enabled without retries.
+[[profile.ci.overrides]]
+filter = 'package(rustfs) & test(/^connect::diagnostics::schedule::tests::/)'
+threads-required = "num-test-threads"
+
+[[profile.ci.overrides]]
+filter = 'package(rustfs-ecstore) & (test(/^set_disk::tests::streaming_get_blocks_concurrent_/) | test(=store::object::tests::retired_marker_converges_after_bucket_recreation_on_sixteen_disks))'
+threads-required = "num-test-threads"
+
+[[profile.ci.overrides]]
+filter = 'package(rustfs-heal) & (binary(mrf_partial_write_test) | (binary(mrf_pipeline_test) & test(/^journal_replay_survives_/)) | test(=admin_selector_survives_sigkill_with_original_scope_and_receipts))'
+threads-required = "num-test-threads"
+
+[[profile.ci.overrides]]
+filter = 'package(rustfs-scanner) & (test(/^scanner::backlog::tests::native_(writer|retirement)_/) | test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(/^scanner_io::tests::service_cohort::/))'
+threads-required = "num-test-threads"
+
 [[profile.ci.scripts]]
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
@@ -301,6 +346,13 @@ setup = 'lifecycle-large-stack'
 filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
 setup = 'ecstore-large-stack'
 
+# These real-disk scanner fixtures assert quiet-workload publication and
+# durable MRF admission. Reserve the pool while their bounded CAS/fsync
+# handoffs run so unrelated disk fanouts cannot consume that fixture budget.
+[[profile.ci.overrides]]
+filter = 'package(rustfs-scanner) & (test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(=scanner_io::tests::multi_pool_scanner_cycle_publishes_combined_usage) | test(=scanner_io::tests::pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage) | test(=scanner_folder::tests::mrf_ownership::checkpoint_fixture_metadata_heal_reaches_budgeted_tail))'
+threads-required = "num-test-threads"
+
 # ===========================================================================
 # QUARANTINE — flaky tests granted retries = 2 under the ci profile ONLY.
 #
@@ -542,10 +594,11 @@ fail-fast = false
 [profile.e2e-smoke.junit]
 path = "junit.xml"
 
-# These storage-heavy fixtures reserve capacity. Pagination tests bound listing
-# requests separately, after their seed objects have been written.
+# Bound the whole pagination fixture, including startup, seeding, and cleanup;
+# its LIST deadline begins only after the seed objects have been written.
 [[profile.e2e-smoke.overrides]]
 filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
+slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
 threads-required = "num-test-threads"
 
 [[profile.e2e-smoke.overrides]]

.github/workflows/docker.yml

@@ -551,54 +551,6 @@ jobs:
           if-no-files-found: error
           retention-days: 30
 
-  # Docker Hub's overview is a separate `full_description` field that
-  # `docker push` never touches, so it drifted from README.md and kept
-  # linking to pages that no longer exist (#7995). Republish it from the
-  # commit the images were built from, only when images were pushed.
-  sync-dockerhub-description:
-    name: Sync Docker Hub Description
-    needs: [ build-check, build-docker ]
-    if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
-    runs-on: sm-standard-2
-    timeout-minutes: 10
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
-        with:
-          persist-credentials: false
-          ref: ${{ needs.build-check.outputs.source_ref }}
-          sparse-checkout: README.md
-          sparse-checkout-cone-mode: false
-
-      - name: Sync Docker Hub description
-        uses: peter-evans/dockerhub-description@1b9a80c056b620d92cedb9d9b5a223409c68ddfa # v5.0.0
-        with:
-          username: ${{ env.DOCKERHUB_USERNAME }}
-          password: ${{ secrets.DOCKERHUB_TOKEN }}
-          repository: ${{ env.REGISTRY_DOCKERHUB }}
-          readme-filepath: ./README.md
-          # README.md links to docs/ and CONTRIBUTING.md by relative path;
-          # rewrite those to github.com URLs so they resolve on Docker Hub.
-          enable-url-completion: true
-
-      # Docker Hub caps `full_description` at 25,000 bytes and the action
-      # truncates to fit with only a warning, which would publish a README
-      # cut off mid-sentence. README.md is already near the cap, so read the
-      # published overview back and fail if it hit the limit.
-      - name: Check published description was not truncated
-        env:
-          REPOSITORY: ${{ env.REGISTRY_DOCKERHUB }}
-        run: |
-          limit=25000
-          published=$(curl -sSf "https://hub.docker.com/v2/repositories/${REPOSITORY}/" \
-            | jq -j '.full_description' | wc -c | tr -d ' ')
-          echo "Published overview: ${published} bytes (Docker Hub limit: ${limit})"
-          # Truncation lands within one UTF-8 character of the cap.
-          if [ "$published" -ge $((limit - 4)) ]; then
-            echo "::error file=README.md::Docker Hub overview was truncated at ${limit} bytes; trim README.md before the next release."
-            exit 1
-          fi
-
   # Docker build summary
   docker-summary:
     name: Docker Build Summary

.github/workflows/helm-package.yml

@@ -130,6 +130,21 @@ jobs:
       - name: Set up Helm
         uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
 
+      - name: Install yq
+        run: |
+          set -euo pipefail
+          yq_version="4.54.1"
+          yq_sha256="8e34fc298390875de416e6a4afcb8cabeceb25d9aa8506c1a2f9353cf702ea5f"
+          tools_dir="$RUNNER_TEMP/rustfs-helm-tools"
+          mkdir -p "$tools_dir"
+          curl --fail --silent --show-error --location --retry 3 \
+            "https://github.com/mikefarah/yq/releases/download/v${yq_version}/yq_linux_amd64" \
+            --output "$tools_dir/yq"
+          printf '%s  %s\n' "$yq_sha256" "$tools_dir/yq" | sha256sum --check -
+          chmod +x "$tools_dir/yq"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+          "$tools_dir/yq" --version
+
       - name: Test Helm Chart Templates
         run: ./scripts/test_helm_templates.sh
 

AGENTS.md

@@ -17,6 +17,8 @@ rules. A skill cannot expand the user's requested scope or grant authorization.
 
 - Inquiry, diagnosis, review, and planning tasks are read-only unless the user
   explicitly requests changes.
+- Ordinary diff/commit reviews use the finding standard below directly; no
+  generic review skill is required. PR and high-risk reviews retain their workflows.
 - For implementation, read the relevant code, tests, and local guidance, then
   make the smallest change that satisfies the request.
 - State assumptions only when they affect behavior or verification. Ask only
@@ -88,6 +90,9 @@ runtime/build output:
 
 - Run `cargo fmt --all --check` for Rust changes.
 - Run the narrowest test that exercises the changed behavior.
+- For an explicitly requested coverage assessment, use the
+  [coverage guide](docs/testing/README.md#coverage); a regression test alone does
+  not require coverage measurement.
 - Add package-scoped `cargo check` or Clippy only for targets, features, public
   APIs, error handling, or control flow not compiled by the focused test.
 - Use `make pre-commit` only when its repository-wide fast checks add confidence
@@ -150,13 +155,18 @@ For every added or edited `tracing` call:
 
 - Reuse the module's `EVENT_*`, `LOG_COMPONENT_*`, and `LOG_SUBSYSTEM_*`
   constants and field shape.
-- Put fields first and a short label last.
+- Put stable fields first (`event`, `component`, `subsystem`, `state`/`result`,
+  then context when available) and a short label last.
 - Use `error` for behavior/security failure, `warn` for degradation/fallback,
   `info` for low-frequency lifecycle, `debug` for diagnostics, and `trace` for
   repetitive request/object success paths.
-- Never log secrets, credential payloads, or merged configs.
+- Never log secrets, credential payloads, raw untrusted bodies, or merged configs,
+  including through error strings and `Debug` output.
+- Prefer one aggregate summary over inventories or startup banners.
 
-Use `.agents/skills/rustfs-logging-governance/SKILL.md` for logging changes.
+Run `./scripts/check_logging_guardrails.sh` for changed logging sites. For a broad
+logging audit, event-model migration, or guardrail expansion, read
+[logging governance](docs/operations/logging-governance.md).
 
 ## Cross-Cutting Storage Invariants
 

CHANGELOG.md

@@ -25,6 +25,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Presigned URLs honour only signed headers** (GHSA-g8w9-qw9q-fghr): a SigV4 presigned request that carries an `x-amz-*` request header not listed in `X-Amz-SignedHeaders` is now rejected with `403 AccessDenied` ("There were headers present in the request which were not signed"), matching AWS S3. Previously the holder of a presigned `PutObject` URL could add unsigned `x-amz-tagging`, `x-amz-storage-class`, `x-amz-website-redirect-location`, ACL, metadata, Object Lock or SSE headers and have them applied. Presigners that intend a property must set it before signing so the SDK lists the header in `SignedHeaders`; `x-amz-cf-id` (CloudFront) remains tolerated unsigned. Header-signed SigV4 and SigV2 requests are unchanged.
 
 ### Fixed
+- **SSE-C TLS requirement now covers copy-source keys**: with `RUSTFS_SSE_C_REQUIRE_TLS=true`, a `CopyObject` or `UploadPartCopy` that carried only the `x-amz-copy-source-server-side-encryption-customer-*` headers (an SSE-C source copied to a destination that is not SSE-C) was accepted on a plaintext transport and was not counted in `rustfs_ssec_plaintext_requests_total`, although those headers carry the source object's customer key. They now count as SSE-C headers: such requests are counted, and refused with `400 InvalidRequest` when the switch is on. Requests over TLS are unaffected.
+- **SSE-KMS encryption context could replace the location entry**: data keys are wrapped under an encryption-context entry, keyed by the bucket name, that ties them to the object's location. A client `x-amz-server-side-encryption-context` entry under the same key replaced it, and because the client context is stored with the object, the replacement also applied on every read. SSE-KMS writes now refuse such a context with `400 InvalidArgument`; other keys, including the names of other buckets, are unaffected. Objects stored with such an entry by an earlier release remain readable. See `docs/operations/kms-backend-security.md`.
 - **Two admin handlers reported a missing policy or user as `500 InternalError`**: `GET /rustfs/admin/v3/info-canned-policy` and `DELETE /rustfs/admin/v3/remove-user` wrapped every IAM error as an internal fault instead of classifying it, so a lookup for something that does not exist answered `500 InternalError` where the rest of the admin API answers `404 NoSuchResource` — `GET /rustfs/admin/v3/user-info` already did, and `iam_error_to_s3_error` already mapped `NoSuchPolicy`, `NoSuchUser`, `NoSuchServiceAccount` and `NoSuchTempAccount` to `NoSuchResource`. A client could not tell "does not exist" from a real server failure, and a controller that checks existence before creating (a Kubernetes operator reconciling policies or users) could never create the object: the missing-policy lookup failed with a `500` and it retried forever. Both handlers now map their IAM errors through the shared `iam_error_to_s3_error` helper, so a missing policy or user is `404 NoSuchResource` and the `error` is preserved as the S3 error's source; the three IAM calls in the removal path (temporary-user lookup, service-account lookup, delete) are all reclassified, and every other IAM error stays `500 InternalError`. Fixes #8117.
 - **Vault backends accepted key names that escape the key prefix**: only the Local backend refused a key identifier containing a path separator, so on Vault KV2 `POST /rustfs/admin/v3/kms/keys` with a name such as `bad/name` succeeded and created a nested KV2 path that the key listing then reported as a directory rather than a key, and an identifier containing `..` addressed a record outside the configured key prefix once the HTTP client normalised the URL. Both Vault backends now apply the Local backend's containment rule at the point where the identifier becomes a path or a Transit key name: an empty identifier, one containing `/`, `\\` or NUL, or the dot segments `.` and `..` is refused with `400` (`InvalidKey`) before any request reaches Vault. The AWS backend is unaffected, since it addresses keys by ARN and alias. A Vault deployment that created such a key on an earlier build must re-create it under a plain name; objects encrypted under the old name stay readable only until that key is refused, so migrate them first. Refs rustfs/backlog#2474.
 - **Fresh multi-pool bootstrap with distinct format creators**: a new deployment whose pools have their first endpoint on different nodes (for example two single-node pools) could never publish its initial `pool.bin`: each node held fresh-bootstrap proof only for the pool it formatted, the deployment-wide proof collapsed to none, and every node died with `pool metadata recovery required: no durable bootstrap identity or pool.bin replica is available` after the startup retry budget. The first pool's creator now mints the pending cluster identity on its own pool, every other creator copies that nonce-bound identity onto the pool it formatted first-hand, and the elected writer publishes `pool.bin` once every pool replica carries the same pending identity. Corrupt or disagreeing replicas, pools that merely have a format, expansion pools joining an initialized deployment, and restarts without first-hand proof still fail closed. Non-elected nodes that start before `pool.bin` exists, and the elected writer while it waits for the other creators, no longer latch their pool-metadata write gate for the life of the process. Refs rustfs/backlog#2338, rustfs/backlog#2375.

Cargo.lock

@@ -685,9 +685,9 @@ dependencies = [
 
 [[package]]
 name = "async-compression"
-version = "0.4.48"
+version = "0.4.50"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fb61aea1a7def73ee7c350a184f0e70b32c182344e2e75bf70c9b621b83417fd"
+checksum = "ee19bd99b43e3691acbad4e840420a4881cea6c0b66a208125a824f8fd53f5a1"
 dependencies = [
  "compression-codecs",
  "compression-core",
@@ -771,17 +771,6 @@ dependencies = [
  "url",
 ]
 
-[[package]]
-name = "async-recursion"
-version = "1.1.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3b43422f69d8ff38f95f1b2bb76517c91589a924d1559a0e935d7c8ce0274c11"
-dependencies = [
- "proc-macro2",
- "quote",
- "syn 2.0.119",
-]
-
 [[package]]
 name = "async-rs"
 version = "0.8.12"
@@ -1012,9 +1001,9 @@ dependencies = [
 
 [[package]]
 name = "aws-sdk-s3"
-version = "1.150.0"
+version = "1.151.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2fecaa77c862fa1e9d636d54fa06e0049a19a4f071b0d72ea1b9086887d09359"
+checksum = "c82d4b41fe41bfca3c9e1de55645d9a525868ef5ce5e14cee9499ccaf1fc7225"
 dependencies = [
  "arc-swap",
  "aws-credential-types",
@@ -1215,9 +1204,9 @@ dependencies = [
 
 [[package]]
 name = "aws-smithy-http-client"
-version = "1.4.2"
+version = "1.5.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7bd25384a4e437aa8d8f339afad4b69e786b936a7cb10db668a7aaf66717b1a8"
+checksum = "51c89cc3f1f281d659a67a519a1b5c6d445b5ce09fa7e5aee40c2c2707e9509d"
 dependencies = [
  "aws-smithy-async",
  "aws-smithy-protocol-test",
@@ -1297,9 +1286,9 @@ dependencies = [
 
 [[package]]
 name = "aws-smithy-runtime"
-version = "1.15.0"
+version = "1.16.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "591e0024cdf2a8de8711860f7e1d5cfbca26b800ce13793a9aa2f4f5918e6d95"
+checksum = "d6e302ac1d88b99652489df31abdec6ac42a2ab2ac3982ad0ac49f64dfaf28ba"
 dependencies = [
  "aws-smithy-async",
  "aws-smithy-http",
@@ -1321,9 +1310,9 @@ dependencies = [
 
 [[package]]
 name = "aws-smithy-runtime-api"
-version = "1.18.0"
+version = "1.19.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9bb0deb40a69de59f7809251cfe1ab16de135f42c49d222d1202b4d0ce19720f"
+checksum = "c0730c16f91124c6a2abb4932c77e299288b3dd9f967ea2e9ec48cc6731e87a4"
 dependencies = [
  "aws-smithy-async",
  "aws-smithy-runtime-api-macros",
@@ -1544,9 +1533,9 @@ dependencies = [
 
 [[package]]
 name = "bigdecimal"
-version = "0.4.10"
+version = "0.4.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4d6867f1565b3aad85681f1015055b087fcfd840d6aeee6eee7f2da317603695"
+checksum = "8fa3f3d8cbf4dffcfe4991de61d012bef509a409ecbe9dd41049bfe32b4d4653"
 dependencies = [
  "autocfg",
  "libm",
@@ -2209,11 +2198,11 @@ dependencies = [
 
 [[package]]
 name = "compression-codecs"
-version = "0.4.43"
+version = "0.4.45"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bef16c47ba2797aa6a909cc37d39911f3a6743811fe7408ac0b0cc0276b656e9"
+checksum = "98fc98460ba0ad5317075d3632b8dfc45d0be8c4a49347c2a38272019717614a"
 dependencies = [
- "brotli 8.0.4",
+ "brotli 9.0.0",
  "bzip2",
  "compression-core",
  "flate2",
@@ -4804,9 +4793,9 @@ dependencies = [
 
 [[package]]
 name = "google-cloud-gax-internal"
-version = "0.7.19"
+version = "0.7.20"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c47c73e95dc5e1de74bebe21b406394b53ff5ccfdfe0e81a0443cdef468e6744"
+checksum = "e873f7e6e39030c07b6d9697bf3b999a269175c8b433e058f56ae10d129f12f7"
 dependencies = [
  "bytes",
  "futures",
@@ -4820,9 +4809,9 @@ dependencies = [
  "http-body-util",
  "hyper",
  "lazy_static",
- "opentelemetry 0.32.0",
- "opentelemetry-semantic-conventions 0.32.1",
- "opentelemetry_sdk 0.32.1",
+ "opentelemetry",
+ "opentelemetry-semantic-conventions",
+ "opentelemetry_sdk",
  "percent-encoding",
  "pin-project",
  "prost 0.14.4",
@@ -4838,7 +4827,7 @@ dependencies = [
  "tonic-prost",
  "tower",
  "tracing",
- "tracing-opentelemetry 0.33.0",
+ "tracing-opentelemetry",
 ]
 
 [[package]]
@@ -4908,9 +4897,9 @@ dependencies = [
 
 [[package]]
 name = "google-cloud-storage"
-version = "1.19.0"
+version = "1.20.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5f7fa1e66d6b47220dbfc5f243b634136e5b8d14d64cb13419bd59f976681da3"
+checksum = "570eb8d0784118a5f4ae1219fd76ff3b14e90b7b0f877478c02b1374e9bbcb8c"
 dependencies = [
  "async-trait",
  "base64 0.23.1",
@@ -4961,9 +4950,9 @@ dependencies = [
 
 [[package]]
 name = "google-cloud-wkt"
-version = "1.8.0"
+version = "1.9.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ad4b36fefda4485f0922de7a58a3e562e984d513442a1aae2281dce25124be2e"
+checksum = "7f3037907e542cf8b8580aa4cf8ad68568d7ac8e7274519a7d0a121867d434da"
 dependencies = [
  "base64 0.23.1",
  "bytes",
@@ -5282,9 +5271,9 @@ dependencies = [
 
 [[package]]
 name = "hotpath"
-version = "0.26.1"
+version = "0.28.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "036ff36fa1c4ae8dfb686606537eaa15b5d5da0c51bf94abeaf9dfc6ee54cac7"
+checksum = "d8345e171844f80e5ae4cbed8bc8276e094f6ff2293a6f8e547c200183976674"
 dependencies = [
  "arc-swap",
  "async-channel",
@@ -5310,6 +5299,7 @@ dependencies = [
  "rustc-demangle",
  "serde",
  "serde_json",
+ "time",
  "tiny_http",
  "tokio",
 ]
@@ -5322,9 +5312,9 @@ checksum = "e149b660b611650bdf3307941ef3e9b81be3c30ed1dfdbe7f670319674f023a6"
 
 [[package]]
 name = "hotpath-macros"
-version = "0.26.1"
+version = "0.28.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1ff68b15fdbcddfbad685f445597580d6fcbf28b49f01ff975e934351079e13b"
+checksum = "bf73e28fe4a7595c322e4265c1608cb10c215cbbce11e49cfcc258766ba686d6"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -6044,9 +6034,9 @@ dependencies = [
 
 [[package]]
 name = "lazy_static"
-version = "1.5.0"
+version = "1.5.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
+checksum = "20870f649af7073d53e38067b2a84312175d56ea15217e1b15bc83506ec50afb"
 dependencies = [
  "spin 0.9.9",
 ]
@@ -7318,20 +7308,6 @@ version = "0.2.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe"
 
-[[package]]
-name = "opentelemetry"
-version = "0.32.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b0142c63252a9e054e68a4c61a5778f7b14f576274d593f8ce883d191a099682"
-dependencies = [
- "futures-core",
- "futures-sink",
- "js-sys",
- "pin-project-lite",
- "thiserror 2.0.21",
- "tracing",
-]
-
 [[package]]
 name = "opentelemetry"
 version = "0.33.0"
@@ -7352,7 +7328,7 @@ version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "e2b304e8d8f11326ba366a99ca5c7aaf7cd397b67268a332437175aeb42a40db"
 dependencies = [
- "opentelemetry 0.33.0",
+ "opentelemetry",
  "tracing",
  "tracing-core",
  "tracing-log",
@@ -7368,7 +7344,7 @@ dependencies = [
  "async-trait",
  "bytes",
  "http 1.5.0",
- "opentelemetry 0.33.0",
+ "opentelemetry",
  "reqwest",
 ]
 
@@ -7381,10 +7357,10 @@ dependencies = [
  "flate2",
  "http 1.5.0",
  "httpdate",
- "opentelemetry 0.33.0",
+ "opentelemetry",
  "opentelemetry-http",
  "opentelemetry-proto",
- "opentelemetry_sdk 0.33.0",
+ "opentelemetry_sdk",
  "prost 0.14.4",
  "reqwest",
  "thiserror 2.0.21",
@@ -7396,17 +7372,11 @@ version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "25da1ac11a0aeccf38d7f77ee0348715adaf8340f65ad46c94a02c6b20e2f65d"
 dependencies = [
- "opentelemetry 0.33.0",
- "opentelemetry_sdk 0.33.0",
+ "opentelemetry",
+ "opentelemetry_sdk",
  "prost 0.14.4",
 ]
 
-[[package]]
-name = "opentelemetry-semantic-conventions"
-version = "0.32.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c913ac17a6c451661ee255f4625d143e51647ae78ebd969b75e41c4442f4fe47"
-
 [[package]]
 name = "opentelemetry-semantic-conventions"
 version = "0.33.0"
@@ -7420,24 +7390,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "9f71803f481d3f5ccbe477edb42a78f473613712b2327835911a9ea0ab069824"
 dependencies = [
  "chrono",
- "opentelemetry 0.33.0",
- "opentelemetry_sdk 0.33.0",
-]
-
-[[package]]
-name = "opentelemetry_sdk"
-version = "0.32.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9b59f80e1ac4d5ff7a2db8fb6c80badb7f0f3f858211fba08dd9aaec750894f9"
-dependencies = [
- "futures-channel",
- "futures-executor",
- "futures-util",
- "opentelemetry 0.32.0",
- "percent-encoding",
- "portable-atomic",
- "rand 0.9.5",
- "thiserror 2.0.21",
+ "opentelemetry",
+ "opentelemetry_sdk",
 ]
 
 [[package]]
@@ -7449,7 +7403,7 @@ dependencies = [
  "futures-channel",
  "futures-executor",
  "futures-util",
- "opentelemetry 0.33.0",
+ "opentelemetry",
  "percent-encoding",
  "portable-atomic",
  "rand 0.9.5",
@@ -7939,9 +7893,9 @@ checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd"
 
 [[package]]
 name = "pin-utils"
-version = "0.1.0"
+version = "0.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184"
+checksum = "13bee6c73da26345c729282832b60b0363cf3dd9f4bfd81d8551b7a1c889a113"
 
 [[package]]
 name = "piper"
@@ -8473,9 +8427,9 @@ dependencies = [
 
 [[package]]
 name = "pulldown-cmark-to-cmark"
-version = "22.0.1"
+version = "22.0.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ab1ad36992cead65f02aa399a373a42730922f1525d988172634fdefdecb8a60"
+checksum = "84bbb29c624230c4bd1047bbdb2aa47e41c860e9665ce62ba9504eebe91bf867"
 dependencies = [
  "pulldown-cmark",
 ]
@@ -8710,9 +8664,9 @@ dependencies = [
 
 [[package]]
 name = "quinn-proto"
-version = "0.11.18"
+version = "0.11.19"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a9746dbde176634f4f2f1faf2404e30a31b2bc1e9cafb5329c95d8177a18c9fc"
+checksum = "0e750cca55fe4f0439a15d0bb529da9651e79993e8e72c61a899a36d462befbe"
 dependencies = [
  "aws-lc-rs",
  "bytes",
@@ -8733,9 +8687,9 @@ dependencies = [
 
 [[package]]
 name = "quinn-udp"
-version = "0.5.15"
+version = "0.5.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "35a133f956daabe89a61a685c2649f13d82d5aa4bd5d12d1277e1072a21c0694"
+checksum = "af66907df18639dcf4db56ca65490cabc4b27a97dbadd96f2926cca73298f016"
 dependencies = [
  "cfg_aliases",
  "libc",
@@ -9416,9 +9370,9 @@ dependencies = [
 
 [[package]]
 name = "russh-sftp"
-version = "3.0.0"
+version = "3.0.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "093197e526668d92bba562e2bbbe98d1af9831bf080b619c736316ca1fa35101"
+checksum = "9bb17f72aba4314656b359ffba72c68dcbd70ed9e97497898b75498fe7198042"
 dependencies = [
  "bitflags 2.13.2",
  "bytes",
@@ -9554,9 +9508,9 @@ dependencies = [
  "mime_guess",
  "moka",
  "object 0.40.0",
- "opentelemetry 0.33.0",
+ "opentelemetry",
  "opentelemetry-proto",
- "opentelemetry_sdk 0.33.0",
+ "opentelemetry_sdk",
  "p256 0.14.0",
  "parking_lot",
  "percent-encoding",
@@ -9647,7 +9601,7 @@ dependencies = [
  "tower",
  "tower-http 0.7.1",
  "tracing",
- "tracing-opentelemetry 0.34.0",
+ "tracing-opentelemetry",
  "tracing-subscriber",
  "url",
  "urlencoding",
@@ -9797,7 +9751,6 @@ version = "1.0.1"
 dependencies = [
  "arc-swap",
  "async-channel",
- "async-recursion",
  "async-trait",
  "aws-credential-types",
  "aws-sdk-s3",
@@ -9837,8 +9790,8 @@ dependencies = [
  "metrics-util",
  "moka",
  "num_cpus",
- "opentelemetry 0.33.0",
- "opentelemetry_sdk 0.33.0",
+ "opentelemetry",
+ "opentelemetry_sdk",
  "parking_lot",
  "path-absolutize",
  "pin-project-lite",
@@ -9899,7 +9852,7 @@ dependencies = [
  "tower",
  "tracing",
  "tracing-core",
- "tracing-opentelemetry 0.34.0",
+ "tracing-opentelemetry",
  "tracing-subscriber",
  "url",
  "urlencoding",
@@ -9979,6 +9932,7 @@ dependencies = [
  "rustfs-common",
  "rustfs-concurrency",
  "rustfs-config",
+ "rustfs-data-usage",
  "rustfs-ecstore",
  "rustfs-filemeta",
  "rustfs-heal-contracts",
@@ -10416,12 +10370,12 @@ dependencies = [
  "metrics-util",
  "num_cpus",
  "nvml-wrapper",
- "opentelemetry 0.33.0",
+ "opentelemetry",
  "opentelemetry-appender-tracing",
  "opentelemetry-otlp",
- "opentelemetry-semantic-conventions 0.33.0",
+ "opentelemetry-semantic-conventions",
  "opentelemetry-stdout",
- "opentelemetry_sdk 0.33.0",
+ "opentelemetry_sdk",
  "percent-encoding",
  "pyroscope",
  "rcgen",
@@ -10453,7 +10407,7 @@ dependencies = [
  "tracing",
  "tracing-appender",
  "tracing-error",
- "tracing-opentelemetry 0.34.0",
+ "tracing-opentelemetry",
  "tracing-subscriber",
  "url",
  "zstd 0.14.0",
@@ -10762,7 +10716,6 @@ dependencies = [
 name = "rustfs-s3select-query"
 version = "1.0.1"
 dependencies = [
- "async-recursion",
  "async-trait",
  "bytes",
  "datafusion",
@@ -11170,9 +11123,9 @@ dependencies = [
 
 [[package]]
 name = "rustls-connector"
-version = "0.23.8"
+version = "0.23.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1babecfcc65b139b812e74bcc7f9ec7b4e00db659fd42d99567b7e77f0c714c6"
+checksum = "524994fd6957827c775abc941f41a146ce351bbd0a9b8cf632ee37f43a13c205"
 dependencies = [
  "futures-io",
  "futures-rustls",
@@ -12283,9 +12236,9 @@ checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
 
 [[package]]
 name = "suppaftp"
-version = "12.1.0"
+version = "12.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d79f066c76927a775d61b5642e0b712a17582ebe7f62b7e5abba246e619878f7"
+checksum = "b161d71217da0cc91bd09c8f98ac5ad2a5e5b2735c6dfe1c6cb08b4471612f35"
 dependencies = [
  "async-trait",
  "chrono",
@@ -12776,9 +12729,9 @@ dependencies = [
 
 [[package]]
 name = "tokio-rustls"
-version = "0.26.5"
+version = "0.26.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b0c85f2c3ef0b1cd58b36682f4b17aaa995f0e5db534d85692b4903abce21f67"
+checksum = "c9cc2678c2cdd569ef8215e2afd7954ada2ae20b4fdd2c5fe6139a3b02d105db"
 dependencies = [
  "rustls",
  "tokio",
@@ -13086,20 +13039,6 @@ dependencies = [
  "tracing-core",
 ]
 
-[[package]]
-name = "tracing-opentelemetry"
-version = "0.33.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "adbc64cba7137545b8044cb1fe9814f7aacf3c6b5f9b45be8bb5db538befdb26"
-dependencies = [
- "js-sys",
- "opentelemetry 0.32.0",
- "tracing",
- "tracing-core",
- "tracing-subscriber",
- "web-time",
-]
-
 [[package]]
 name = "tracing-opentelemetry"
 version = "0.34.0"
@@ -13107,7 +13046,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "0a904802a1b902f43638b677ff2a650847e3b4404101b6c586d648e8c1e3e8fe"
 dependencies = [
  "js-sys",
- "opentelemetry 0.33.0",
+ "opentelemetry",
  "smallvec",
  "tracing",
  "tracing-core",
@@ -14042,9 +13981,9 @@ dependencies = [
 
 [[package]]
 name = "xxhash-rust"
-version = "0.8.18"
+version = "0.8.19"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "aee1b19627c7c60102ab80d3a9cbe18de90bfe03bfa6c3715447681f0e8c8af6"
+checksum = "550a2b930b62486a393c52d5c3b84bff264b28aa437ed64694d31e93b1757af7"
 
 [[package]]
 name = "yansi"
@@ -14075,9 +14014,9 @@ dependencies = [
 
 [[package]]
 name = "yoke-derive"
-version = "0.8.3"
+version = "0.8.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "33811428bee40dbceb6d545e95754741d17a6aef9a4849f0fd62e2ba4f412a78"
+checksum = "ec8ebde2db3681e8c9980cc27822030e68752690ddfa9473e739aeb4dbde6d71"
 dependencies = [
  "proc-macro2",
  "quote",

Cargo.toml

@@ -146,8 +146,7 @@ rustfs-zip = { path = "./crates/zip", version = "1.0.1" }
 async-channel = "2.5.0"
 async_zip = { default-features = false, version = "0.0.19" }
 mysql_async = { default-features = false, version = "0.37.1" }
-async-compression = { version = "0.4.48" }
-async-recursion = "1.1.1"
+async-compression = { version = "0.4.50" }
 async-trait = "0.1.92"
 async-nats = { version = "0.50.0", default-features = false }
 axum = "0.8.9"
@@ -170,7 +169,7 @@ reqwest = "0.13.5"
 rustfs-kafka-async = { version = "1.3.1" }
 socket2 = { version = "0.6.5" }
 tokio = { version = "1.53.1" }
-tokio-rustls = { default-features = false, version = "0.26.5" }
+tokio-rustls = { default-features = false, version = "0.26.6" }
 tokio-stream = { version = "0.1.19" }
 tokio-test = "0.4.6"
 tokio-util = { version = "0.7.19" }
@@ -246,11 +245,11 @@ atomic_enum = "0.3.0"
 aws-config = { version = "1.12.0" }
 aws-credential-types = { version = "1.3.0" }
 aws-sdk-kms = { default-features = false, version = "1.122.0" }
-aws-sdk-s3 = { default-features = false, version = "1.150.0" }
+aws-sdk-s3 = { default-features = false, version = "1.151.0" }
 aws-sdk-sts = { default-features = false, version = "1.118.0" }
 aws-smithy-async = { version = "1.3.0" }
-aws-smithy-http-client = { default-features = false, version = "1.4.2" }
-aws-smithy-runtime-api = { version = "1.18.0" }
+aws-smithy-http-client = { default-features = false, version = "1.5.0" }
+aws-smithy-runtime-api = { version = "1.19.0" }
 aws-smithy-types = { version = "1.8.1" }
 base64-simd = "0.8.0"
 brotli = "9.0.0"
@@ -269,7 +268,7 @@ enumset = "1.1.14"
 faster-hex = "1.0.0"
 flate2 = "1.1.10"
 glob = "0.3.4"
-google-cloud-storage = "1.19.0"
+google-cloud-storage = "1.20.0"
 google-cloud-auth = "1.17.0"
 hashbrown = { version = "0.17.1" }
 # Base32 for RFC 6238 TOTP shared secrets (RFC 4648 unpadded, the alphabet
@@ -279,7 +278,7 @@ hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
 hostname = "0.4.2"
 ipnetwork = { version = "0.21.1" }
-lazy_static = "1.5.0"
+lazy_static = "1.5.1"
 libc = "0.2.189"
 libsystemd = "0.7.2"
 local-ip-address = "0.6.13"
@@ -346,7 +345,7 @@ walkdir = "2.5.0"
 winapi-util = "0.1.11"
 windows = { version = "0.62.2" }
 windows-sys = "0.61.2"
-xxhash-rust = { version = "0.8.18" }
+xxhash-rust = { version = "0.8.19" }
 zip = "8.6.0"
 zstd = "0.14.0"
 
@@ -366,10 +365,10 @@ pyroscope = { version = "2.1.1" }
 # FTP and SFTP
 libunftp = { version = "0.23.1" }
 unftp-core = "0.1.0"
-suppaftp = { version = "12.1.0" }
+suppaftp = { version = "12.1.1" }
 rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
 russh = { version = "0.63.3" }
-russh-sftp = "3.0.0"
+russh-sftp = "3.0.1"
 
 # WebDAV
 dav-server = "0.11.0"
@@ -377,7 +376,7 @@ dav-server = "0.11.0"
 # Performance Analysis and Memory Profiling
 rustfs-mimalloc = { version = "0.5.6" }
 # Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
-hotpath = { version = "0.26.1", default-features = false }
+hotpath = { version = "0.28.0", default-features = false }
 # Snapshot testing for output format regression detection
 insta = { version = "1.48" }
 

README.md

@@ -141,7 +141,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.1
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -220,7 +220,7 @@ For developers who want to build RustFS Docker images from source with multi-arc
 ./docker-buildx.sh --push
 
 # Build specific version
-./docker-buildx.sh --release v1.0.0 --push
+./docker-buildx.sh --release 1.0.1 --push
 
 # Build for custom registry
 ./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -238,7 +238,7 @@ You can also use Make targets for convenience:
 ```bash
 make docker-buildx                    # Build locally
 make docker-buildx-push               # Build and push
-make docker-buildx-version VERSION=v1.0.0  # Build specific version
+make docker-buildx-version VERSION=1.0.1  # Build specific version
 make help-docker                      # Show all Docker-related commands
 ```
 

README_ZH.md

@@ -138,7 +138,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # 使用指定版本运行
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.1
 ```
 
 如果您通过绑定挂载启用 TLS 证书目录，也请用同样方式准备该目录：
@@ -176,7 +176,7 @@ docker compose -f docker-compose-simple.yml up -d
 ./docker-buildx.sh --push
 
 # 构建指定版本
-./docker-buildx.sh --release v1.0.0 --push
+./docker-buildx.sh --release 1.0.1 --push
 
 # 构建并推送到自定义仓库
 ./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -193,7 +193,7 @@ docker compose -f docker-compose-simple.yml up -d
 ```bash
 make docker-buildx                    # 本地构建
 make docker-buildx-push               # 构建并推送
-make docker-buildx-version VERSION=v1.0.0  # 构建指定版本
+make docker-buildx-version VERSION=1.0.1  # 构建指定版本
 make help-docker                      # 显示所有 Docker 相关命令
 ```
 

crates/config/README.md

@@ -168,9 +168,10 @@ Scanner cycle budget controls:
 
 ## Foreground write admission environment variables
 
-Large direct `PutObject` requests and multipart `UploadPart` requests share one
-per-process permit pool that bounds concurrent body ingest and storage writes.
-Small direct PUTs stay on the legacy path.
+Large direct `PutObject` requests, multipart `UploadPart` requests, and
+server-side multipart `UploadPartCopy` requests share one per-process permit
+pool that bounds concurrent body ingest and storage writes. Small direct PUTs
+stay on the legacy path.
 
 - `RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE`
   - enables the default-on pool; `false` keeps only the soft request counter.
@@ -190,11 +191,11 @@ Small direct PUTs stay on the legacy path.
   - smallest `UploadPart` that takes a permit; `0` gates every part.
   - default is `0`.
 - `RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS`
-  - how long an `UploadPart` waits in the bounded queue for a permit before returning S3 `SlowDown`; `0` rejects immediately when the pool is full.
-  - default is `10000`. Parts wait before body ingest, so SDK-default clients that send every part of an upload concurrently drain through the pool instead of failing on a full pool.
-  - RustFS does not read the request body while a part is queued, so the client's socket write stalls for the whole wait and whatever timeout the client or an intermediary has configured competes with this value. Keep it with margin below the shortest such timeout in use (botocore applies its 60 s `connect_timeout` to the body write; the AWS SDK for Java v2 has a 30 s socket write timeout; reverse proxies add their own body timeouts); a wait that outlives the client timeout surfaces as a dropped connection instead of `SlowDown`.
+  - how long an `UploadPart` or `UploadPartCopy` waits in the bounded queue for a permit before returning S3 `SlowDown`; `0` rejects immediately when the pool is full.
+  - default is `10000`. Parts wait before body ingest or source-object reads, so SDK-default clients that send every part of an upload concurrently drain through the pool instead of failing on a full pool.
+  - RustFS does not read the request body while a part is queued, so the client's socket write stalls for the whole wait and whatever timeout the client or an intermediary has configured competes with this value. `UploadPartCopy` waits before taking bucket lifecycle locks or opening source readers, and uses the unknown-size weight because the authoritative copy length is available only after source metadata and range validation. Keep the wait with margin below the shortest client/proxy timeout in use (botocore applies its 60 s `connect_timeout` to the body write; the AWS SDK for Java v2 has a 30 s socket write timeout; reverse proxies add their own body timeouts); a wait that outlives the client timeout surfaces as a dropped connection instead of `SlowDown`.
 - `RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MAX_PENDING`
-  - maximum `UploadPart` requests waiting for a permit at once; parts beyond it return `SlowDown` without waiting.
+  - maximum `UploadPart` and `UploadPartCopy` requests waiting for a permit at once; parts beyond it return `SlowDown` without waiting.
   - default is `0`, which derives 16 times the large-write slot count, or the explicit request-count limit (512 at stock settings). Automatic subdivision does not enlarge this queue.
   - each queued HTTP/1 part holds whatever unread body the client already pushed into the connection's kernel receive buffer (an HTTP/2 part holds up to its flow-control window in process memory). Autotuned receive buffers can remain large on reused connections; queue depth does not imply a fixed per-connection memory cost.
 - `RUSTFS_PUT_FOREGROUND_ADMISSION_ENABLE`, `RUSTFS_PUT_FOREGROUND_ADMISSION_LIMIT`, `RUSTFS_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS`

crates/config/src/constants/object.rs

@@ -387,7 +387,9 @@ pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 32 * 10
 /// Multipart pressure is often many moderate-sized parts rather than one very
 /// large request. The default gates every multipart part through the same permit
 /// pool as large/unknown-size PutObject while keeping small direct PUTs on the
-/// legacy path.
+/// legacy path. Server-side UploadPartCopy requests use this same multipart
+/// admission path with unknown-size weighting because source metadata and range
+/// validation happen after admission.
 pub const ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: &str =
     "RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES";
 pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 0;
@@ -415,7 +417,9 @@ pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 250;
 /// wait with a 10 s request deadline. The wait must leave margin under the
 /// shortest of those, not merely fall below an SDK default, so the part
 /// receives S3 `SlowDown`/503 for the client to retry instead of losing its
-/// connection (issue #7385). `0` rejects immediately when the pool is full.
+/// connection (issue #7385). Server-side UploadPartCopy waits in the same queue
+/// before taking bucket lifecycle locks or opening source readers. `0` rejects
+/// immediately when the pool is full.
 pub const ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: &str =
     "RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS";
 pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 10_000;
@@ -427,7 +431,7 @@ pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 10_0
 // the wait past any client timeout.
 const _: () = assert!(DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS * 3 <= 30_000);
 
-/// Maximum multipart UploadPart requests waiting for a foreground write permit per process.
+/// Maximum multipart UploadPart or UploadPartCopy requests waiting for a foreground write permit per process.
 ///
 /// Parts beyond this queue depth are rejected with S3 `SlowDown`/503 without
 /// waiting, so a genuinely saturated node still fails fast instead of holding

crates/e2e_test/src/distributed/s3_basic_test.rs

@@ -20,6 +20,60 @@ use aws_sdk_s3::primitives::ByteStream;
 use aws_sdk_s3::types::{Delete, MetadataDirective, ObjectIdentifier};
 use std::time::Duration;
 
+#[tokio::test]
+async fn four_node_bucket_tags_opa_refreshes_warm_peer_caches() -> TestResult {
+    use super::harness::cluster_admin_ok;
+    use crate::sts_query_compat_test::{OPA_AUTH_TOKEN, OpaMock, set_department};
+
+    init_logging();
+    let mut opa = OpaMock::start().await?;
+    let dist = DistCluster::start_with_env(
+        DistLayout::FourNodeFourDisk,
+        &[
+            ("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str()),
+            ("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", OPA_AUTH_TOKEN),
+        ],
+    )
+    .await?;
+    let secret = uuid::Uuid::new_v4().to_string();
+    cluster_admin_ok(
+        &dist.cluster,
+        http::Method::PUT,
+        "/rustfs/admin/v3/add-user?accessKey=opabuckettags",
+        Some(serde_json::json!({"secretKey": secret, "status": "enabled"}).to_string()),
+    )
+    .await?;
+    let bucket = unique_bucket("opa-tags");
+    dist.create_bucket(&bucket).await?;
+    let admin = dist.client(0)?;
+    put_object(&admin, &bucket, "report", b"report".to_vec()).await?;
+    let clients = (0..4)
+        .map(|node| dist.client_with_credentials(node, "opabuckettags", &secret))
+        .collect::<Result<Vec<_>, _>>()?;
+
+    // Warm every peer, then verify the acknowledged updates/removal without
+    // polling or restarting nodes. This exercises healthy-peer metadata reload.
+    for department in [Some("finance"), Some("engineering"), None, Some("finance")] {
+        match department {
+            Some(value) => set_department(&admin, &bucket, value).await?,
+            None => {
+                admin.delete_bucket_tagging().bucket(&bucket).send().await?;
+            }
+        }
+        for client in &clients {
+            let result = client.get_object().bucket(&bucket).key("report").send().await;
+            if department == Some("finance") {
+                assert_eq!(result?.body.collect().await?.into_bytes().as_ref(), b"report");
+            } else {
+                let error = result.expect_err("updated or removed bucket tags must revoke this policy's read access");
+                assert_eq!(error.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
+            }
+            opa.expect_bucket_tags("s3:GetObject", &bucket, department).await?;
+        }
+    }
+    Ok(())
+}
+
 #[tokio::test]
 async fn four_node_four_drive_s3_put_get_head_list_copy_rename_delete_and_presign() -> TestResult {
     init_logging();

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -1450,8 +1450,11 @@ mod tests {
             cluster.set_env("RUSTFS_HEAL_PAGE_PARALLEL_ENABLE", "false");
         }
         // Keep every node's Heal runtime enabled for normal disk registration.
-        let server_rust_log = std::env::var("RUSTFS_HEAL_CHAOS_SERVER_RUST_LOG")
-            .unwrap_or_else(|_| "rustfs::heal::task=info,rustfs=error".to_string());
+        // Capture durable handoff and marker cleanup failures during shutdown.
+        let server_rust_log = std::env::var("RUSTFS_HEAL_CHAOS_SERVER_RUST_LOG").unwrap_or_else(|_| {
+            "rustfs::heal::task=info,rustfs::heal::manager=info,rustfs::main::handle_shutdown=info,rustfs_heal::heal=warn,rustfs=error"
+                .to_string()
+        });
         cluster.set_env("RUST_LOG", server_rust_log);
         let log_dir = std::env::var("RUSTFS_HEAL_CHAOS_LOG_DIR").unwrap_or_else(|_| {
             // Keep explicit directories compatible and isolate parallel suite captures.

crates/e2e_test/src/inline_fast_path_cluster_test.rs

@@ -886,6 +886,8 @@ fn configure_reader_metric_cluster(cluster: &mut RustFSTestClusterEnvironment, c
     cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "true");
     cluster.set_env("RUSTFS_OBS_TRACES_EXPORT_ENABLED", "false");
     cluster.set_env("RUSTFS_OBS_LOGS_EXPORT_ENABLED", "false");
+    // Keep startup diagnostics available when OTLP log export is disabled.
+    cluster.set_env("RUSTFS_OBS_LOG_STDOUT_ENABLED", "true");
     cluster.set_env("RUSTFS_OBS_METER_INTERVAL", "1");
     cluster.set_env("RUSTFS_OBS_USE_STDOUT", "false");
     cluster.set_env("RUSTFS_GET_CODEC_STREAMING_ENABLE", "false");

crates/e2e_test/src/list_objects_v2_pagination_test.rs

@@ -30,6 +30,7 @@ mod tests {
     use crate::common::{RustFSTestEnvironment, init_logging};
     use aws_sdk_s3::Client;
     use aws_sdk_s3::primitives::ByteStream;
+    use futures::{StreamExt, stream};
     use std::collections::HashSet;
     use std::future::Future;
     use std::time::Duration;
@@ -810,23 +811,25 @@ mod tests {
 
         create_bucket(&client, bucket).await.expect("Failed to create bucket");
 
-        // Create 1000 objects: 100 directories with 10 files each
-        let mut all_keys = Vec::new();
-        let dirs: Vec<String> = (0..100).map(|i| format!("dir-{:03}/", i)).collect();
-        for dir in &dirs {
-            for i in 0..10 {
-                let key = format!("{dir}file{:02}.txt", i);
-                client
-                    .put_object()
-                    .bucket(bucket)
-                    .key(&key)
-                    .body(ByteStream::from_static(b"x"))
-                    .send()
-                    .await
-                    .expect("Failed to put object");
-                all_keys.push(key);
-            }
-        }
+        // Keep every fixture key while overlapping durable PUTs within a bounded fanout.
+        let all_keys: Vec<String> = (0..100)
+            .flat_map(|dir| (0..10).map(move |file| format!("dir-{dir:03}/file{file:02}.txt")))
+            .collect();
+        stream::iter(&all_keys)
+            .for_each_concurrent(16, |key| {
+                let client = &client;
+                async move {
+                    client
+                        .put_object()
+                        .bucket(bucket)
+                        .key(key)
+                        .body(ByteStream::from_static(b"x"))
+                        .send()
+                        .await
+                        .unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err}"));
+                }
+            })
+            .await;
 
         eprintln!("Seeded {} objects in {bucket}; starting ListObjectsV2 pagination", all_keys.len());
         let deadline = Instant::now() + PAGINATION_TIMEOUT;
@@ -1040,22 +1043,35 @@ mod tests {
 
         create_bucket(&client, bucket).await.expect("Failed to create bucket");
 
-        // 12 dirs × 100 files = 1200 raw keys
+        // Keep all 1200 raw keys while bounding upload concurrency so fixture setup
+        // leaves the test's timeout budget available for the listing regression.
         let dir_count = 12;
         let files_per_dir = 100;
-        for d in 0..dir_count {
-            for f in 0..files_per_dir {
-                let key = format!("dir-{:02}/file{:03}.txt", d, f);
-                client
-                    .put_object()
-                    .bucket(bucket)
-                    .key(&key)
-                    .body(ByteStream::from_static(b"x"))
-                    .send()
-                    .await
-                    .expect("Failed to put object");
-            }
-        }
+        let fixture_started = std::time::Instant::now();
+        stream::iter((0..dir_count).flat_map(|d| (0..files_per_dir).map(move |f| (d, f))))
+            .for_each_concurrent(16, |(d, f)| {
+                let client = &client;
+                async move {
+                    let key = format!("dir-{d:02}/file{f:03}.txt");
+                    client
+                        .put_object()
+                        .bucket(bucket)
+                        .key(&key)
+                        .body(ByteStream::from_static(b"x"))
+                        .send()
+                        .await
+                        .unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err}"));
+                }
+            })
+            .await;
+        info!(
+            event = "pagination_fixture_ready",
+            component = "e2e_test",
+            subsystem = "list_objects_v2",
+            object_count = dir_count * files_per_dir,
+            elapsed_ms = fixture_started.elapsed().as_millis(),
+            "Pagination fixture ready"
+        );
 
         eprintln!(
             "Seeded {} objects in {bucket}; starting ListObjectsV2 pagination",
@@ -1077,7 +1093,20 @@ mod tests {
             dir_count,
             output.common_prefixes().len()
         );
+        let prefixes = output
+            .common_prefixes()
+            .iter()
+            .map(|entry| entry.prefix().expect("each CommonPrefix must carry its prefix").to_owned())
+            .collect::<Vec<_>>();
+        let expected_prefixes = (0..dir_count).map(|d| format!("dir-{d:02}/")).collect::<Vec<_>>();
+        assert_eq!(prefixes, expected_prefixes);
+        assert!(output.contents().is_empty(), "all fixture objects must collapse into prefixes");
+        assert_eq!(output.key_count(), Some(i32::try_from(dir_count).expect("fixture count fits i32")));
         assert_eq!(output.max_keys(), Some(1000));
+        assert!(
+            output.next_continuation_token().is_none(),
+            "a complete listing must not return a continuation token"
+        );
         // 12 visible < 1000 capped MaxKeys → not truncated
         assert!(
             !output.is_truncated().unwrap_or(false),

crates/e2e_test/src/multipart_storage_class_test.rs

@@ -65,6 +65,130 @@ mod tests {
         Ok(())
     }
 
+    #[tokio::test]
+    async fn list_parts_sdk_paginator_terminates_and_preserves_completion() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
+    {
+        init_logging();
+        let mut env = RustFSTestEnvironment::new().await?;
+        env.start_rustfs_server(Vec::new()).await?;
+        let client = env.create_s3_client();
+        let bucket = "multipart-pagination";
+        let key = "sparse-parts.bin";
+        env.create_test_bucket(bucket).await?;
+        let upload = client.create_multipart_upload().bucket(bucket).key(key).send().await?;
+        let upload_id = upload.upload_id().ok_or("CreateMultipartUpload returned no upload ID")?;
+
+        let mut empty_pages = client
+            .list_parts()
+            .bucket(bucket)
+            .key(key)
+            .upload_id(upload_id)
+            .into_paginator()
+            .stop_on_duplicate_token(false)
+            .send();
+        let empty = tokio::time::timeout(std::time::Duration::from_secs(30), empty_pages.next())
+            .await?
+            .ok_or("empty upload must return one page")??;
+        assert!(empty.parts().is_empty());
+        assert_eq!(empty.is_truncated(), Some(false));
+        assert_eq!(empty.next_part_number_marker(), None);
+        assert!(
+            tokio::time::timeout(std::time::Duration::from_secs(30), empty_pages.next())
+                .await?
+                .is_none()
+        );
+
+        let mut expected_body = Vec::new();
+        let mut completed_parts = Vec::new();
+        for (part_number, body) in [(1, vec![b'a'; PART_SIZE]), (3, vec![b'b'; PART_SIZE]), (10, b"tail".to_vec())] {
+            expected_body.extend_from_slice(&body);
+            let uploaded = client
+                .upload_part()
+                .bucket(bucket)
+                .key(key)
+                .upload_id(upload_id)
+                .part_number(part_number)
+                .body(ByteStream::from(body))
+                .send()
+                .await?;
+            completed_parts.push(
+                CompletedPart::builder()
+                    .part_number(part_number)
+                    .set_e_tag(uploaded.e_tag().map(str::to_owned))
+                    .build(),
+            );
+        }
+
+        for (page_size, expected_pages) in [(1, 3), (2, 2), (3, 1), (1000, 1)] {
+            let mut pages = client
+                .list_parts()
+                .bucket(bucket)
+                .key(key)
+                .upload_id(upload_id)
+                .into_paginator()
+                .page_size(page_size)
+                // The server must terminate pagination even without the SDK's duplicate-token safeguard.
+                .stop_on_duplicate_token(false)
+                .send();
+            let mut page_count = 0;
+            let mut listed_parts = Vec::new();
+            while let Some(page) = tokio::time::timeout(std::time::Duration::from_secs(30), pages.next()).await? {
+                assert!(page_count < expected_pages, "paginator must terminate after the final page");
+                let page = page?;
+                page_count += 1;
+                let truncated = page_count < expected_pages;
+                assert_eq!(page.is_truncated(), Some(truncated));
+                let last_part = page.parts().last().and_then(|part| part.part_number());
+                let expected_marker = truncated.then(|| last_part.expect("truncated page must contain parts").to_string());
+                assert_eq!(page.next_part_number_marker(), expected_marker.as_deref());
+                listed_parts.extend(
+                    page.parts()
+                        .iter()
+                        .map(|part| part.part_number().expect("part number must be present")),
+                );
+            }
+            assert_eq!(page_count, expected_pages);
+            assert_eq!(listed_parts, [1, 3, 10]);
+        }
+
+        let mut resumed_pages = client
+            .list_parts()
+            .bucket(bucket)
+            .key(key)
+            .upload_id(upload_id)
+            .part_number_marker("2")
+            .into_paginator()
+            .page_size(1)
+            .stop_on_duplicate_token(false)
+            .send();
+        for (part_number, next_marker) in [(3, Some("3")), (10, None)] {
+            let page = tokio::time::timeout(std::time::Duration::from_secs(30), resumed_pages.next())
+                .await?
+                .ok_or("parts above a missing marker must still be listed")??;
+            assert_eq!(page.parts().len(), 1);
+            assert_eq!(page.parts()[0].part_number(), Some(part_number));
+            assert_eq!(page.is_truncated(), Some(next_marker.is_some()));
+            assert_eq!(page.next_part_number_marker(), next_marker);
+        }
+        assert!(
+            tokio::time::timeout(std::time::Duration::from_secs(30), resumed_pages.next())
+                .await?
+                .is_none()
+        );
+
+        client
+            .complete_multipart_upload()
+            .bucket(bucket)
+            .key(key)
+            .upload_id(upload_id)
+            .multipart_upload(CompletedMultipartUpload::builder().set_parts(Some(completed_parts)).build())
+            .send()
+            .await?;
+        assert_completed_object(&client, bucket, key, "STANDARD", &expected_body).await?;
+        env.stop_server();
+        Ok(())
+    }
+
     #[tokio::test]
     async fn multipart_upload_preserves_standard_and_rrs_across_retry_and_resume()
     -> Result<(), Box<dyn std::error::Error + Send + Sync>> {

crates/e2e_test/src/protocols/test_runner.rs

@@ -21,7 +21,7 @@ use crate::protocols::sftp_compliance::{
     test_sftp_compliance_readonly, test_sftp_compliance_standalone, test_sftp_compliance_suite,
 };
 use crate::protocols::sftp_core::{test_sftp_core_operations, test_sftp_idle_timeout_disconnects};
-use crate::protocols::webdav_core::test_webdav_core_operations;
+use crate::protocols::webdav_core::{test_webdav_core_operations, test_webdav_quota_reporting};
 use std::time::Instant;
 use tokio::time::{Duration, sleep};
 use tracing::{error, info};
@@ -99,6 +99,10 @@ impl ProtocolTestSuite {
                     info!("=== Starting WebDAV Core Test ===");
                     "WebDAV core operations (MKCOL, PUT, GET, DELETE, PROPFIND)"
                 }
+                "test_webdav_quota_reporting" => {
+                    info!("=== Starting WebDAV Quota Reporting Test ===");
+                    "WebDAV quota reporting (PROPFIND quota properties: bucket-quota sum and physical-capacity fallback)"
+                }
                 "test_sftp_core_operations" => {
                     info!("=== Starting SFTP Core Test ===");
                     "SFTP core operations (banner, mkdir, put, get with SHA compare, rename, delete, rmdir)"
@@ -158,6 +162,7 @@ impl ProtocolTestSuite {
         match test_def.name {
             "test_ftps_core_operations" => test_ftps_core_operations().await.map_err(|e| e.into()),
             "test_webdav_core_operations" => test_webdav_core_operations().await.map_err(|e| e.into()),
+            "test_webdav_quota_reporting" => test_webdav_quota_reporting().await.map_err(|e| e.into()),
             "test_sftp_core_operations" => test_sftp_core_operations().await.map_err(|e| e.into()),
             "test_sftp_compliance_suite" => test_sftp_compliance_suite().await.map_err(|e| e.into()),
             "test_sftp_compliance_readonly" => test_sftp_compliance_readonly().await.map_err(|e| e.into()),
@@ -203,6 +208,10 @@ fn all_protocol_tests() -> Vec<TestDefinition> {
             name: "test_webdav_core_operations",
             required_feature: "webdav",
         },
+        TestDefinition {
+            name: "test_webdav_quota_reporting",
+            required_feature: "webdav",
+        },
         TestDefinition {
             name: "test_sftp_core_operations",
             required_feature: "sftp",
@@ -253,9 +262,10 @@ mod tests {
     fn schedules_all_protocol_tests_without_feature_filter() {
         let names = scheduled_names(ProtocolTestSuite::with_requested_features(None));
 
-        assert_eq!(names.len(), 7);
+        assert_eq!(names.len(), 8);
         assert!(names.contains(&"test_ftps_core_operations"));
         assert!(names.contains(&"test_webdav_core_operations"));
+        assert!(names.contains(&"test_webdav_quota_reporting"));
         assert!(names.contains(&"test_sftp_core_operations"));
         assert!(names.contains(&"test_sftp_compliance_standalone"));
     }
@@ -264,7 +274,14 @@ mod tests {
     fn schedules_only_requested_non_sftp_protocols() {
         let names = scheduled_names(ProtocolTestSuite::with_requested_features(Some("ftps, webdav")));
 
-        assert_eq!(names, vec!["test_ftps_core_operations", "test_webdav_core_operations"]);
+        assert_eq!(
+            names,
+            vec![
+                "test_ftps_core_operations",
+                "test_webdav_core_operations",
+                "test_webdav_quota_reporting"
+            ]
+        );
     }
 
     #[test]
@@ -295,9 +312,10 @@ mod tests {
     fn full_feature_schedules_all_protocol_tests() {
         let names = scheduled_names(ProtocolTestSuite::with_requested_features(Some("full")));
 
-        assert_eq!(names.len(), 7);
+        assert_eq!(names.len(), 8);
         assert!(names.contains(&"test_ftps_core_operations"));
         assert!(names.contains(&"test_webdav_core_operations"));
+        assert!(names.contains(&"test_webdav_quota_reporting"));
         assert!(names.contains(&"test_sftp_core_operations"));
         assert!(names.contains(&"test_sftp_compliance_standalone"));
     }

crates/e2e_test/src/protocols/webdav_core.rs

@@ -33,10 +33,14 @@
 
 use crate::common::rustfs_binary_path_with_features;
 use crate::common::{AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via};
+use crate::common::{FAST_DATA_USAGE_SCANNER_ENV, admin_request};
 use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, ProtocolTestEnvironment};
 use anyhow::Result;
+use http::Method;
 use reqwest::Client;
+use std::time::{Duration, Instant};
 use tokio::process::Command;
+use tokio::time::sleep;
 use tracing::info;
 
 // Fixed WebDAV port for testing
@@ -769,3 +773,261 @@ pub async fn test_webdav_core_operations() -> Result<()> {
 async fn test_webdav_core_operations_direct() -> Result<()> {
     test_webdav_core_operations().await
 }
+
+/// Extract the numeric value of the first occurrence of a DAV property in a
+/// PROPFIND multistatus body. Namespace-prefix agnostic; a self-closing
+/// (unreported) property yields None.
+fn quota_prop_value(body: &str, prop: &str) -> Option<u64> {
+    let prop_start = body.find(prop)?;
+    let after_name = &body[prop_start + prop.len()..];
+    let value_start = after_name.find('>')? + 1;
+    let value_end = after_name[value_start..].find('<')? + value_start;
+    after_name[value_start..value_end].trim().parse().ok()
+}
+
+/// PROPFIND `/` at Depth 0 requesting the two quota properties, returning
+/// (quota-available-bytes, quota-used-bytes).
+async fn propfind_root_quota(client: &Client, base_url: &str, auth_header: &str) -> Result<(u64, u64)> {
+    let body = r#"<?xml version="1.0" encoding="utf-8"?>
+<D:propfind xmlns:D="DAV:"><D:prop><D:quota-available-bytes/><D:quota-used-bytes/></D:prop></D:propfind>"#;
+    let resp = client
+        .request(
+            reqwest::Method::from_bytes(b"PROPFIND").expect("PROPFIND is a valid HTTP method"),
+            base_url,
+        )
+        .header("Authorization", auth_header)
+        .header("Depth", "0")
+        .header("Content-Type", "application/xml")
+        .body(body)
+        .send()
+        .await?;
+    let status = resp.status();
+    let text = resp.text().await?;
+    anyhow::ensure!(status.as_u16() == 207, "PROPFIND / should return 207, got: {status} body: {text}");
+    let available = quota_prop_value(&text, "quota-available-bytes")
+        .ok_or_else(|| anyhow::anyhow!("quota-available-bytes missing from PROPFIND response: {text}"))?;
+    let used = quota_prop_value(&text, "quota-used-bytes")
+        .ok_or_else(|| anyhow::anyhow!("quota-used-bytes missing from PROPFIND response: {text}"))?;
+    Ok((available, used))
+}
+
+/// Set a HARD bucket quota, retrying while the admin API reports the quota
+/// subsystem as not yet ready (503), mirroring quota_test.rs readiness.
+async fn set_bucket_quota_with_retry(admin_base_url: &str, bucket: &str, quota_bytes: u64) -> Result<()> {
+    let quota_path = format!("/rustfs/admin/v3/quota/{bucket}");
+    let quota_config = serde_json::json!({
+        "quota": quota_bytes,
+        "quota_type": "HARD"
+    })
+    .to_string();
+    let deadline = Instant::now() + Duration::from_secs(30);
+    loop {
+        let (status, response) = admin_request(
+            admin_base_url,
+            Method::PUT,
+            &quota_path,
+            Some(quota_config.clone()),
+            DEFAULT_ACCESS_KEY,
+            DEFAULT_SECRET_KEY,
+        )
+        .await
+        .map_err(|e| anyhow::anyhow!("{e}"))?;
+        if status.is_success() {
+            return Ok(());
+        }
+        if status != http::StatusCode::SERVICE_UNAVAILABLE {
+            anyhow::bail!("failed to set quota for {bucket}: {status} {response}");
+        }
+        if Instant::now() >= deadline {
+            anyhow::bail!("quota readiness did not converge for {bucket} within 30 seconds");
+        }
+        sleep(Duration::from_secs(1)).await;
+    }
+}
+
+/// Test WebDAV: PROPFIND quota properties (quota-available-bytes /
+/// quota-used-bytes).
+///
+/// dav-server queries quota once per PROPFIND without a path argument, so the
+/// reported pair is session-scoped:
+///
+/// - Phase A: a visible bucket without quota (or no buckets at all) falls back
+///   to the cluster's usable physical capacity, the console dashboard numbers.
+/// - Phase B: once every visible bucket carries a HARD quota, the pair is the
+///   summed cached bucket usage against the summed quota limits.
+pub async fn test_webdav_quota_reporting() -> Result<()> {
+    // Dedicated ports: the core WebDAV test owns 9010/9080.
+    const QUOTA_S3_ADDRESS: &str = "127.0.0.1:9011";
+    const QUOTA_WEBDAV_PORT: u16 = 9081;
+    const QUOTA_WEBDAV_ADDRESS: &str = "127.0.0.1:9081";
+    const QUOTA_LIMIT: u64 = 1_073_741_824; // 1 GiB
+    const OBJECT_CONTENT: &[u8] = b"Hello, WebDAV!";
+
+    let env = ProtocolTestEnvironment::new().map_err(|e| anyhow::anyhow!("{}", e))?;
+    let admin_base_url = format!("http://{QUOTA_S3_ADDRESS}");
+
+    info!("Starting WebDAV server on {}", QUOTA_WEBDAV_ADDRESS);
+    let binary_path = rustfs_binary_path_with_features(Some("webdav"));
+    let mut server_command = Command::new(&binary_path);
+    server_command
+        .arg("--address")
+        .arg(QUOTA_S3_ADDRESS)
+        .env("RUSTFS_CONSOLE_ENABLE", "false")
+        .env("RUSTFS_WEBDAV_ENABLE", "true")
+        .env("RUSTFS_WEBDAV_ADDRESS", QUOTA_WEBDAV_ADDRESS)
+        .env("RUSTFS_WEBDAV_TLS_ENABLED", "false") // No TLS for testing
+        .arg(&env.temp_dir);
+    // Quota-mode reporting reads the data-usage scanner cache; run the scanner
+    // on a fast cycle so the uploaded object converges quickly.
+    for (key, value) in FAST_DATA_USAGE_SCANNER_ENV {
+        server_command.env(key, value);
+    }
+    let mut server_process = server_command.spawn()?;
+
+    // Ensure server is cleaned up even on failure
+    let result = async {
+        ProtocolTestEnvironment::wait_for_port_ready(QUOTA_WEBDAV_PORT, 30)
+            .await
+            .map_err(|e| anyhow::anyhow!("{}", e))?;
+
+        let client = create_client();
+        let auth_header = basic_auth_header();
+        let base_url = format!("http://{QUOTA_WEBDAV_ADDRESS}");
+
+        // Phase A: physical-capacity fallback (a visible bucket has no quota)
+        info!("Testing WebDAV quota: fallback to physical capacity without bucket quotas");
+        let phase_a_bucket = "quota-fallback-probe";
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MKCOL").expect("MKCOL is a valid HTTP method"),
+                format!("{base_url}/{phase_a_bucket}"),
+            )
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success(),
+            "MKCOL '{phase_a_bucket}' should succeed, got: {}",
+            resp.status()
+        );
+
+        // Storage info may briefly report zero capacity right after startup
+        // (disks still initializing), which omits the quota properties; poll
+        // until they appear.
+        let phase_a_deadline = Instant::now() + Duration::from_secs(30);
+        let (available, _used) = loop {
+            match propfind_root_quota(&client, &base_url, &auth_header).await {
+                Ok(quota) => break quota,
+                Err(e) if Instant::now() < phase_a_deadline => {
+                    info!("quota properties not reported yet, retrying: {e}");
+                    sleep(Duration::from_secs(1)).await;
+                }
+                Err(e) => return Err(e),
+            }
+        };
+        assert!(
+            available > 0,
+            "physical-capacity fallback should report positive quota-available-bytes, got: {available}"
+        );
+        info!("PASS: quota fallback reports physical capacity (available={available})");
+
+        let resp = client
+            .delete(format!("{base_url}/{phase_a_bucket}"))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 204,
+            "DELETE bucket '{phase_a_bucket}' should succeed, got: {}",
+            resp.status()
+        );
+
+        // Phase B: every visible bucket has a HARD quota -> summed usage/quota
+        info!("Testing WebDAV quota: summed bucket quota reporting");
+        let bucket = "quota-probe";
+        let object = "hello.txt";
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MKCOL").expect("MKCOL is a valid HTTP method"),
+                format!("{base_url}/{bucket}"),
+            )
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(resp.status().is_success(), "MKCOL '{bucket}' should succeed, got: {}", resp.status());
+
+        let resp = client
+            .put(format!("{base_url}/{bucket}/{object}"))
+            .header("Authorization", &auth_header)
+            .body(OBJECT_CONTENT)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "PUT '{bucket}/{object}' should succeed, got: {}",
+            resp.status()
+        );
+
+        set_bucket_quota_with_retry(&admin_base_url, bucket, QUOTA_LIMIT).await?;
+
+        // The usage cache trails the scanner; poll PROPFIND until the uploaded
+        // object is reflected (before convergence the missing usage counts as 0).
+        let expected_used = OBJECT_CONTENT.len() as u64;
+        let deadline = Instant::now() + Duration::from_secs(60);
+        let (available, used) = loop {
+            let (available, used) = propfind_root_quota(&client, &base_url, &auth_header).await?;
+            if used == expected_used {
+                break (available, used);
+            }
+            if Instant::now() >= deadline {
+                anyhow::bail!(
+                    "quota-used-bytes did not converge to {expected_used} within 60 seconds (last: used={used} available={available})"
+                );
+            }
+            sleep(Duration::from_secs(2)).await;
+        };
+        assert_eq!(
+            available,
+            QUOTA_LIMIT - expected_used,
+            "quota-available-bytes should be the remaining quota"
+        );
+        info!("PASS: summed quota reporting converged (used={used}, available={available})");
+
+        // Cleanup
+        let resp = client
+            .delete(format!("{base_url}/{bucket}/{object}"))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 204,
+            "DELETE '{bucket}/{object}' should succeed, got: {}",
+            resp.status()
+        );
+        let resp = client
+            .delete(format!("{base_url}/{bucket}"))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 204,
+            "DELETE bucket '{bucket}' should succeed, got: {}",
+            resp.status()
+        );
+
+        info!("WebDAV quota reporting tests passed");
+        Ok(())
+    }
+    .await;
+
+    // Always cleanup server process
+    let _ = server_process.kill().await;
+    let _ = server_process.wait().await;
+
+    result
+}
+
+#[tokio::test]
+async fn test_webdav_quota_reporting_direct() -> Result<()> {
+    test_webdav_quota_reporting().await
+}

crates/e2e_test/src/sts_query_compat_test.rs

@@ -35,7 +35,7 @@ use tokio::time::{Duration, timeout};
 
 type BoxError = Box<dyn Error + Send + Sync>;
 type TestResult = Result<(), BoxError>;
-const OPA_AUTH_TOKEN: &str = "sts-opa-token";
+pub(crate) const OPA_AUTH_TOKEN: &str = "sts-opa-token";
 
 fn sts_client(url: &str, access_key: &str, secret_key: &str, session_token: Option<&str>) -> Client {
     build_test_sts_client(url, access_key, secret_key, session_token, "e2e-sts-query-compat")
@@ -103,6 +103,25 @@ async fn create_user(env: &RustFSTestEnvironment, user: &str, secret: &str) -> T
     Ok(())
 }
 
+pub(crate) async fn set_department(client: &aws_sdk_s3::Client, bucket: &str, department: &str) -> TestResult {
+    client
+        .put_bucket_tagging()
+        .bucket(bucket)
+        .tagging(
+            aws_sdk_s3::types::Tagging::builder()
+                .tag_set(
+                    aws_sdk_s3::types::Tag::builder()
+                        .key("department")
+                        .value(department)
+                        .build()?,
+                )
+                .build()?,
+        )
+        .send()
+        .await?;
+    Ok(())
+}
+
 async fn assert_access_denied(client: &Client, context: &str) -> TestResult {
     let error = client
         .assume_role()
@@ -240,15 +259,19 @@ async fn handle_opa_request(
                 .as_ref()
                 .and_then(|value| value.pointer("/input/action"))
                 .and_then(Value::as_str);
-            let bucket = payload
+            let department = payload
                 .as_ref()
-                .and_then(|value| value.pointer("/input/resource/bucket"))
+                .and_then(|value| value.pointer("/input/context/conditions/ExistingBucketTag~1department/0"))
                 .and_then(Value::as_str);
             matches!(
-                (action, bucket),
-                (Some("s3:ListBucket"), Some("opa-list-visible")) | (Some("s3:GetBucketLocation"), Some("opa-list-location"))
+                (action, department),
+                (Some("s3:ListBucket"), Some("finance")) | (Some("s3:GetBucketLocation"), Some("legal"))
             )
         }
+        Some(Value::String(account)) if account == "opabuckettags" => payload.as_ref().is_some_and(|value| {
+            value["input"]["action"] == "s3:CreateBucket"
+                || value["input"]["context"]["conditions"]["ExistingBucketTag/department"] == serde_json::json!(["finance"])
+        }),
         None => true,
         _ => false,
     };
@@ -270,15 +293,15 @@ enum OpaValidationMode {
     Unavailable,
 }
 
-struct OpaMock {
-    url: String,
+pub(crate) struct OpaMock {
+    pub(crate) url: String,
     requests: mpsc::UnboundedReceiver<Value>,
     validation_started: mpsc::UnboundedReceiver<()>,
     task: JoinHandle<()>,
 }
 
 impl OpaMock {
-    async fn start() -> Result<Self, BoxError> {
+    pub(crate) async fn start() -> Result<Self, BoxError> {
         Self::start_with_mode(OpaValidationMode::Ready, Some(OPA_AUTH_TOKEN)).await
     }
 
@@ -335,6 +358,16 @@ impl OpaMock {
             .ok_or_else(|| "OPA request channel closed".into())
     }
 
+    pub(crate) async fn expect_bucket_tags(&mut self, action: &str, bucket: &str, department: Option<&str>) -> TestResult {
+        let payload = self.next_request().await?;
+        let input = &payload["input"];
+        assert_eq!(input["action"], action);
+        assert_eq!(input["resource"]["bucket"], bucket);
+        let expected = department.map(|value| serde_json::json!([value]));
+        assert_eq!(input["context"]["conditions"].get("ExistingBucketTag/department"), expected.as_ref());
+        Ok(())
+    }
+
     async fn wait_for_validation(&mut self) -> TestResult {
         timeout(Duration::from_secs(5), self.validation_started.recv())
             .await?
@@ -571,8 +604,13 @@ async fn test_list_buckets_opa_contract() -> TestResult {
     .await?;
 
     let admin_client = env.create_s3_client();
-    for bucket in ["opa-list-hidden", "opa-list-location", "opa-list-visible"] {
+    for (bucket, department) in [
+        ("opa-list-hidden", "engineering"),
+        ("opa-list-location", "legal"),
+        ("opa-list-visible", "finance"),
+    ] {
         admin_client.create_bucket().bucket(bucket).send().await?;
+        set_department(&admin_client, bucket, department).await?;
     }
 
     let user = "opalistbuckets";
@@ -640,6 +678,240 @@ async fn test_list_buckets_opa_contract() -> TestResult {
     Ok(())
 }
 
+#[tokio::test]
+async fn test_bucket_tags_opa_contract() -> TestResult {
+    use aws_sdk_s3::primitives::ByteStream;
+    use aws_sdk_s3::types::{
+        BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, Tag, Tagging, VersioningConfiguration,
+    };
+
+    init_logging();
+    let mut opa = OpaMock::start().await?;
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_without_cleanup_with_env(&[
+        ("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str()),
+        ("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", OPA_AUTH_TOKEN),
+        ("NO_PROXY", "127.0.0.1,localhost"),
+    ])
+    .await?;
+    let secret = uuid::Uuid::new_v4().to_string();
+    create_user(&env, "opabuckettags", &secret).await?;
+    let client = aws_sdk_s3::Client::from_conf(build_test_s3_config(&env.url, "opabuckettags", &secret, None, "e2e-bucket-tags"));
+    let admin = env.create_s3_client();
+    let source = "opa-tags-source";
+    let destination = "opa-tags-destination";
+
+    client.create_bucket().bucket(source).send().await?;
+    opa.expect_bucket_tags("s3:CreateBucket", source, None).await?;
+    set_department(&admin, source, "finance").await?;
+    admin
+        .put_bucket_versioning()
+        .bucket(source)
+        .versioning_configuration(
+            VersioningConfiguration::builder()
+                .status(BucketVersioningStatus::Enabled)
+                .build(),
+        )
+        .send()
+        .await?;
+    let written = client
+        .put_object()
+        .bucket(source)
+        .key("report")
+        .tagging("department=engineering")
+        .body(ByteStream::from_static(b"report"))
+        .send()
+        .await?;
+    opa.expect_bucket_tags("s3:PutObject", source, Some("finance")).await?;
+    let version = written.version_id().ok_or("versioned PUT must return a version ID")?;
+    let read = client
+        .get_object()
+        .bucket(source)
+        .key("report")
+        .version_id(version)
+        .send()
+        .await?;
+    assert_eq!(read.body.collect().await?.into_bytes().as_ref(), b"report");
+    opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
+    client
+        .head_object()
+        .bucket(source)
+        .key("report")
+        .version_id(version)
+        .send()
+        .await?;
+    opa.expect_bucket_tags("s3:GetObject", source, Some("finance")).await?;
+    client.list_objects_v2().bucket(source).send().await?;
+    opa.expect_bucket_tags("s3:ListBucket", source, Some("finance")).await?;
+    client.list_object_versions().bucket(source).send().await?;
+    opa.expect_bucket_tags("s3:ListBucketVersions", source, Some("finance"))
+        .await?;
+
+    admin.create_bucket().bucket(destination).send().await?;
+    set_department(&admin, destination, "engineering").await?;
+    let copy_source = format!("{source}/report?versionId={version}");
+    let error = client
+        .copy_object()
+        .bucket(destination)
+        .key("copied")
+        .copy_source(&copy_source)
+        .send()
+        .await
+        .expect_err("source tags must not authorize a different destination");
+    assert_eq!(error.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
+    opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
+    opa.expect_bucket_tags("s3:PutObject", destination, Some("engineering"))
+        .await?;
+    set_department(&admin, destination, "finance").await?;
+    client
+        .copy_object()
+        .bucket(destination)
+        .key("copied")
+        .copy_source(&copy_source)
+        .send()
+        .await?;
+    opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
+    opa.expect_bucket_tags("s3:PutObject", destination, Some("finance")).await?;
+
+    let upload = client
+        .create_multipart_upload()
+        .bucket(destination)
+        .key("multipart")
+        .send()
+        .await?;
+    opa.expect_bucket_tags("s3:PutObject", destination, Some("finance")).await?;
+    let upload_id = upload.upload_id().ok_or("multipart upload ID")?;
+    client
+        .upload_part()
+        .bucket(destination)
+        .key("multipart")
+        .upload_id(upload_id)
+        .part_number(1)
+        .body(ByteStream::from_static(b"part"))
+        .send()
+        .await?;
+    opa.expect_bucket_tags("s3:PutObject", destination, Some("finance")).await?;
+    let copied_part = client
+        .upload_part_copy()
+        .bucket(destination)
+        .key("multipart")
+        .upload_id(upload_id)
+        .part_number(1)
+        .copy_source(&copy_source)
+        .send()
+        .await?;
+    opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
+    opa.expect_bucket_tags("s3:PutObject", destination, Some("finance")).await?;
+    let part = copied_part.copy_part_result().ok_or("copied part result")?;
+    client
+        .complete_multipart_upload()
+        .bucket(destination)
+        .key("multipart")
+        .upload_id(upload_id)
+        .multipart_upload(
+            CompletedMultipartUpload::builder()
+                .parts(
+                    CompletedPart::builder()
+                        .part_number(1)
+                        .e_tag(part.e_tag().ok_or("part ETag")?)
+                        .build(),
+                )
+                .build(),
+        )
+        .send()
+        .await?;
+    opa.expect_bucket_tags("s3:PutObject", destination, Some("finance")).await?;
+    let read = client.get_object().bucket(destination).key("multipart").send().await?;
+    assert_eq!(read.body.collect().await?.into_bytes().as_ref(), b"report");
+    opa.expect_bucket_tags("s3:GetObject", destination, Some("finance")).await?;
+
+    // Mutation authorization uses the old tags, not the proposed replacement.
+    set_department(&client, source, "engineering").await?;
+    opa.expect_bucket_tags("s3:PutBucketTagging", source, Some("finance")).await?;
+    let error = client
+        .list_objects_v2()
+        .bucket(source)
+        .send()
+        .await
+        .expect_err("updated tags revoke access");
+    assert_eq!(error.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
+    opa.expect_bucket_tags("s3:ListBucket", source, Some("engineering")).await?;
+    let error = client
+        .copy_object()
+        .bucket(destination)
+        .key("denied-source")
+        .copy_source(&copy_source)
+        .send()
+        .await
+        .expect_err("destination tags must not authorize a denied source");
+    assert_eq!(error.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
+    opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("engineering"))
+        .await?;
+    set_department(&client, source, "finance")
+        .await
+        .expect_err("requested tags must not authorize their own mutation");
+    opa.expect_bucket_tags("s3:PutBucketTagging", source, Some("engineering"))
+        .await?;
+    set_department(&admin, source, "finance").await?;
+    client.delete_bucket_tagging().bucket(source).send().await?;
+    opa.expect_bucket_tags("s3:PutBucketTagging", source, Some("finance")).await?;
+    let error = client
+        .list_objects_v2()
+        .bucket(source)
+        .send()
+        .await
+        .expect_err("untagged bucket is denied by this policy");
+    assert_eq!(error.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
+    opa.expect_bucket_tags("s3:ListBucket", source, None).await?;
+
+    // Ambiguous stored tags must not turn into an implicit IAM denial that a
+    // separate bucket-policy Allow can override.
+    admin
+        .put_bucket_policy()
+        .bucket(destination)
+        .policy(
+            serde_json::json!({
+                "Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "*"},
+                    "Action": "s3:GetObject", "Resource": format!("arn:aws:s3:::{destination}/*")}]
+            })
+            .to_string(),
+        )
+        .send()
+        .await?;
+    // A normal OPA denial retains the existing bucket-policy Allow fallback;
+    // a metadata lookup error must abort that same authorization path.
+    set_department(&admin, destination, "engineering").await?;
+    let read = client.get_object().bucket(destination).key("multipart").send().await?;
+    assert_eq!(read.body.collect().await?.into_bytes().as_ref(), b"report");
+    opa.expect_bucket_tags("s3:GetObject", destination, Some("engineering"))
+        .await?;
+    admin
+        .put_bucket_tagging()
+        .bucket(destination)
+        .tagging(
+            Tagging::builder()
+                .tag_set(Tag::builder().key("department").value("finance").build()?)
+                .tag_set(Tag::builder().key("department").value("engineering").build()?)
+                .build()?,
+        )
+        .send()
+        .await?;
+    let error = client
+        .get_object()
+        .bucket(destination)
+        .key("multipart")
+        .send()
+        .await
+        .expect_err("metadata failure must not fall back to a bucket-policy Allow");
+    assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(500));
+    assert!(
+        matches!(opa.requests.try_recv(), Err(mpsc::error::TryRecvError::Empty)),
+        "invalid tags must not be sent to OPA as absent"
+    );
+    env.stop_server();
+    Ok(())
+}
+
 #[tokio::test]
 async fn test_sts_and_list_buckets_fail_closed_while_opa_is_initializing() -> TestResult {
     init_logging();

crates/e2e_test/src/upgrade_compatibility_test.rs

@@ -52,6 +52,9 @@ const VERSIONED_BUCKET: &str = "upgrade-versioned-data";
 const MIXED_BUCKET: &str = "upgrade-mixed-version-data";
 const MIXED_NODE_COUNT: usize = 4;
 const UPGRADE_READINESS_BODY: &[u8] = b"upgrade write readiness";
+// Written by the "previous-seed" readiness probe through node 0 before any
+// current binary starts; every later phase must still read it.
+const PREVIOUS_RELEASE_SEED_KEY: &str = ".upgrade-readiness/previous-seed/node-0";
 const MULTIPART_WORKERS: usize = 16;
 const MULTIPART_UPLOADS_PER_WORKER: usize = 16;
 // Peers keep a restarted node's drive in Suspect/Returning for roughly
@@ -68,6 +71,7 @@ const CONFIG_REPLICA_BUCKET: &str = "upgrade-config-replica";
 const ROLLBACK_BUCKET: &str = "rollback-config-data";
 const ROLLBACK_REPLICA_BUCKET: &str = "rollback-config-replica";
 const BUCKET_QUOTA_BYTES: u64 = 64 * 1024 * 1024;
+const QUOTA_USAGE_READINESS_TIMEOUT: Duration = Duration::from_secs(90);
 const LIFECYCLE_RULE_ID: &str = "upgrade-expire-logs";
 const LIFECYCLE_PREFIX: &str = "logs/";
 const LIFECYCLE_DAYS: i32 = 30;
@@ -425,6 +429,11 @@ async fn exercise_mixed_cluster(
     let multipart_keys = write_multipart_load(&clients, phase).await?;
     let expected_count = multipart_keys.len() + 2;
     for (label, client) in [("current", current_client), ("previous", previous_client)] {
+        assert_eq!(
+            read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1,
+            UPGRADE_READINESS_BODY,
+            "{phase}: the {label} node must retain the previous-release seed"
+        );
         wait_for_phase_listing(
             client,
             phase,
@@ -523,9 +532,7 @@ async fn prepare_previous_release_baseline(cluster: &mut RustFSTestClusterEnviro
     wait_for_upgrade_write_readiness(&clients, "previous-baseline", LISTING_CONVERGENCE_TIMEOUT).await?;
     for (reader, client) in clients.iter().enumerate() {
         assert_eq!(
-            read_object(client, MIXED_BUCKET, ".upgrade-readiness/previous-seed/node-0", None)
-                .await?
-                .1,
+            read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1,
             UPGRADE_READINESS_BODY,
             "previous-release node {reader} must retain the seed across its rolling restart"
         );
@@ -935,6 +942,11 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes
     cluster.start_node_from_binary(3, &current_binary).await?;
 
     for (node_idx, client) in cluster.create_all_clients()?.iter().enumerate() {
+        assert_eq!(
+            read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1,
+            UPGRADE_READINESS_BODY,
+            "current node {node_idx} must retain the previous-release seed"
+        );
         for phase in ["one-current-node", "one-previous-node"] {
             wait_for_phase_listing(
                 client,
@@ -962,7 +974,8 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes
                 "assertions": [
                     "current node reads objects written through previous-release client",
                     "previous-release node reads objects written through current client",
-                    "all nodes list every mixed-version object after homogeneous-current convergence"
+                    "all nodes list every mixed-version object after homogeneous-current convergence",
+                    "the previous-release seed survives the baseline restart and every upgrade phase"
                 ],
             }),
         )?;
@@ -1067,12 +1080,23 @@ async fn put_object_through_quota_warmup(client: &Client, bucket: &str, key: &st
 
 async fn get_bucket_quota(env: &RustFSTestEnvironment, bucket: &str) -> Result<Option<u64>, BoxError> {
     let path = format!("/rustfs/admin/v3/quota/{bucket}");
-    let (status, response) = admin_request(&env.url, Method::GET, &path, None, &env.access_key, &env.secret_key).await?;
-    if status != StatusCode::OK {
-        return Err(format!("reading the quota of {bucket} failed: {status} {response}").into());
+    let deadline = Instant::now() + QUOTA_USAGE_READINESS_TIMEOUT;
+    loop {
+        let (status, response) = tokio::time::timeout_at(
+            deadline,
+            admin_request(&env.url, Method::GET, &path, None, &env.access_key, &env.secret_key),
+        )
+        .await
+        .map_err(|_| format!("reading the quota of {bucket} exceeded the usage-readiness timeout"))??;
+        if status == StatusCode::OK {
+            let quota: serde_json::Value = serde_json::from_str(&response)?;
+            return Ok(quota.get("quota").and_then(serde_json::Value::as_u64));
+        }
+        if status != StatusCode::SERVICE_UNAVAILABLE || Instant::now() >= deadline {
+            return Err(format!("reading the quota of {bucket} failed: {status} {response}").into());
+        }
+        sleep(Duration::from_millis(500)).await;
     }
-    let quota: serde_json::Value = serde_json::from_str(&response)?;
-    Ok(quota.get("quota").and_then(serde_json::Value::as_u64))
 }
 
 /// `GET /rustfs/admin/v3/list-remote-targets?bucket=...`.

crates/ecstore/Cargo.toml

@@ -208,7 +208,6 @@ aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a",
 urlencoding = { workspace = true }
 smallvec = { workspace = true, features = ["serde"] }
 shadow-rs = { workspace = true, default-features = false }
-async-recursion.workspace = true
 aws-credential-types = { workspace = true }
 aws-smithy-types = { workspace = true }
 aws-smithy-runtime-api = { workspace = true, features = ["http-1x"] }

crates/ecstore/benches/erasure_benchmark.rs

@@ -307,6 +307,32 @@ fn bench_memory_patterns(c: &mut Criterion) {
     group.sample_size(10);
     group.measurement_time(Duration::from_secs(5));
 
+    // The reader owns an EC-sized buffer before encoding. Keep ingest outside
+    // the timed region to isolate the borrowed-copy versus ownership transfer.
+    let erasure = Erasure::new(data_shards, parity_shards, block_size);
+    let payload = generate_test_data(block_size);
+    for owned in [false, true] {
+        group.bench_function(if owned { "owned_ingest" } else { "borrowed_ingest" }, |b| {
+            b.iter_batched(
+                || {
+                    let mut buffer = bytes::BytesMut::with_capacity(erasure.shard_size() * (data_shards + parity_shards));
+                    buffer.extend_from_slice(&payload);
+                    buffer
+                },
+                |buffer| {
+                    let shards = if owned {
+                        erasure.encode_data_bytes_mut(buffer, block_size)
+                    } else {
+                        erasure.encode_data(&buffer)
+                    }
+                    .expect("encode benchmark block");
+                    black_box(shards)
+                },
+                criterion::BatchSize::PerIteration,
+            );
+        });
+    }
+
     // Test reusing the same Erasure instance
     group.bench_function("reuse_erasure_instance", |b| {
         let erasure = Erasure::new(data_shards, parity_shards, block_size);

crates/ecstore/src/api/mod.rs

@@ -395,6 +395,8 @@ pub mod data_usage {
         ScannerPauseBacklogRetirementPlanner, ScannerPauseBacklogRetirementReplica,
         register_scanner_pause_backlog_retirement_planner,
     };
+    #[cfg(feature = "test-util")]
+    pub use crate::data_usage::seed_bucket_usage_memory_for_test;
     pub use crate::data_usage::{
         DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
         init_compression_total_memory_from_backend, invalidate_admin_data_usage_snapshot_cache,
@@ -406,8 +408,7 @@ pub mod data_usage {
         remove_bucket_usage_from_backend, replace_bucket_usage_memory_from_info, store_compression_total_in_backend,
         store_data_usage_in_backend,
     };
-    #[cfg(feature = "test-util")]
-    pub use crate::data_usage::{get_bucket_usage_memory, seed_bucket_usage_memory_for_test};
+    pub use crate::data_usage::{get_bucket_usage_memory, lookup_degraded_bucket_usage_baseline};
 }
 
 pub mod disk {
@@ -518,7 +519,7 @@ pub mod object {
         ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, QuotaAdmission,
         RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
         SCANNER_PUBLICATION_LEASE_FENCE_METADATA_KEY, ScannerPublicationCommitScope, ScannerPublicationCommitStartError,
-        ScannerPublicationCommitState, ShardIntegrityWriteMode, StreamConsumer, WriteCompletion,
+        ScannerPublicationCommitState, ShardIntegrityWriteMode, StreamConsumer, WriteCommitGuard, WriteCompletion,
         get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
         register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
     };

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -5606,6 +5606,11 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(
 
     //debug!("lc_event.action: {:?}", lc_event.action);
     debug!("expiry_on_non_transitioned_objects opts: {:?}", opts);
+    let usage_accounting = if lc_event.action == IlmAction::DeleteAction && !versioned && !version_suspended && !oi.is_dir {
+        Some(crate::data_usage::begin_expiry_usage_accounting(&oi.bucket).await)
+    } else {
+        None
+    };
     let mut dobj = match api
         .delete_object_with_tier_delete_journal(&oi.bucket, &encode_dir_object(&oi.name), opts)
         .await
@@ -5625,6 +5630,14 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(
             return false;
         }
     };
+    if let Some(accounting) = usage_accounting
+        && !dobj.name.is_empty()
+        && !dobj.delete_marker
+        && !dobj.is_dir
+        && let Ok(size) = crate::data_usage::quota_object_size(&dobj)
+    {
+        accounting.commit(size).await;
+    }
     schedule_lifecycle_replication_delete_if_needed(oi, &dobj).await;
 
     // The object (or all its versions, for delete_all) was expired; evict any
@@ -12891,6 +12904,130 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn ordinary_expiry_accounts_the_committed_receipt_once() {
+        use crate::bucket::quota::{QuotaOperation, checker::QuotaChecker};
+        use crate::data_usage::{apply_bucket_usage_memory_overlay, replace_bucket_usage_memory_from_info};
+        use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
+
+        let (_disk_paths, ecstore) = setup_test_env().await;
+        for size in [0usize, 42] {
+            let bytes = u64::try_from(size).expect("fixture size fits");
+            let bucket = format!("expiry-accounting-{}", Uuid::new_v4().simple());
+            create_test_bucket(&ecstore, &bucket).await;
+            let mut reader = PutObjReader::from_vec(vec![0; size]);
+            let mut queued = ecstore
+                .put_object(&bucket, "expire", &mut reader, &ObjectOptions::default())
+                .await
+                .expect("create expiring object");
+            let mut reader = PutObjReader::from_vec(vec![0; size]);
+            ecstore
+                .put_object(&bucket, "keep", &mut reader, &ObjectOptions::default())
+                .await
+                .expect("create retained object");
+            let mut baseline = DataUsageInfo {
+                last_update: Some(std::time::SystemTime::now()),
+                usage_snapshot_complete: true,
+                buckets_count: 1,
+                ..Default::default()
+            };
+            baseline.buckets_usage.insert(
+                bucket.clone(),
+                BucketUsageInfo {
+                    objects_count: 2,
+                    versions_count: 2,
+                    size: bytes * 2,
+                    ..Default::default()
+                },
+            );
+            baseline.bucket_sizes.insert(bucket.clone(), bytes * 2);
+            baseline.calculate_totals();
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            // Legacy quota admission consumes the same conservative overlay.
+            // Durable reservations have their own storage-commit accounting.
+            metadata_sys::update(
+                &bucket,
+                rustfs_config::QUOTA_CONFIG_FILE,
+                format!(r#"{{"quota":{},"quota_type":"Hard"}}"#, bytes * 2).into_bytes(),
+            )
+            .await
+            .expect("configure the legacy quota");
+            let checker =
+                QuotaChecker::new(Arc::new(tokio::sync::RwLock::new(metadata_sys::BucketMetadataSys::new(ecstore.clone()))));
+            let attempted_size = bytes.max(1);
+            let quota = checker
+                .check_quota(&bucket, QuotaOperation::PutObject, attempted_size)
+                .await
+                .expect("check quota before expiration");
+            assert!(!quota.allowed);
+            assert!(!quota.uses_durable_reservations);
+            assert_eq!(quota.current_usage, Some(bytes * 2));
+            // Queued metadata is not the accounting receipt from the storage commit.
+            queued.size = 1;
+            let event = lifecycle::Event {
+                action: IlmAction::DeleteAction,
+                ..Default::default()
+            };
+            let incarnation = ecstore
+                .bucket_incarnation_id_from_disk(&bucket)
+                .await
+                .expect("bucket incarnation");
+            for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, false)] {
+                assert_eq!(
+                    super::apply_expiry_on_non_transitioned_objects(
+                        ecstore.clone(),
+                        &queued,
+                        &event,
+                        &LcEventSrc::Scanner,
+                        attempt_incarnation,
+                    )
+                    .await,
+                    expected
+                );
+                let mut response = baseline.clone();
+                apply_bucket_usage_memory_overlay(&mut response).await;
+                let remaining = if attempt_incarnation == incarnation { 1 } else { 2 };
+                assert_eq!(response.buckets_usage[&bucket].objects_count, remaining);
+                assert_eq!(response.buckets_usage[&bucket].size, remaining * bytes);
+                let quota = checker
+                    .check_quota(&bucket, QuotaOperation::PutObject, attempted_size)
+                    .await
+                    .expect("check quota before scanner confirmation");
+                assert!(!quota.allowed, "expiry must not release unconfirmed quota credit");
+                assert_eq!(quota.current_usage, Some(bytes * 2));
+            }
+            let mut confirmed = baseline.clone();
+            confirmed.last_update = Some(std::time::SystemTime::now());
+            confirmed.scanner_epoch = Some(7);
+            confirmed.scanner_cycle = Some(10);
+            let usage = confirmed.buckets_usage.get_mut(&bucket).expect("bucket is in the snapshot");
+            usage.size = bytes;
+            usage.objects_count = 1;
+            usage.versions_count = 0; // The scanner does not count unversioned objects as versions.
+            confirmed.bucket_sizes.insert(bucket.clone(), bytes);
+            confirmed.calculate_totals();
+            replace_bucket_usage_memory_from_info(&confirmed).await;
+            let quota = checker
+                .check_quota(&bucket, QuotaOperation::PutObject, attempted_size)
+                .await
+                .expect("check quota after scanner confirmation");
+            assert_eq!(quota.allowed, bytes > 0);
+            assert_eq!(quota.current_usage, Some(bytes));
+            let too_large = checker
+                .check_quota(&bucket, QuotaOperation::PutObject, bytes + 1)
+                .await
+                .expect("check the first byte above the quota limit");
+            assert!(!too_large.allowed);
+            assert!(
+                ecstore
+                    .get_object_info(&bucket, "keep", &ObjectOptions::default())
+                    .await
+                    .is_ok()
+            );
+        }
+    }
+
     #[tokio::test]
     #[serial]
     async fn queued_delete_all_rechecks_a_same_id_rule_moved_into_the_future() {

crates/ecstore/src/bucket/lifecycle/recovery_disposition.rs

@@ -1122,9 +1122,11 @@ pub(crate) async fn resume_recovery_disposition(
     // Lock order is control record, then legacy source. Disposition revisions
     // use ETag CAS and never acquire either namespace lock internally.
     let control_lock = api.new_ns_lock(RUSTFS_META_BUCKET, &control_object).await?;
-    let control_guard = control_lock
-        .get_read_lock(crate::set_disk::get_lock_acquire_timeout())
-        .await?;
+    let control_guard = Arc::new(
+        control_lock
+            .get_read_lock(crate::set_disk::get_lock_acquire_timeout())
+            .await?,
+    );
     let source_lock = api
         .new_ns_lock(RUSTFS_META_BUCKET, &initial.disposition.identity.canonical_source_path)
         .await?;
@@ -1504,7 +1506,7 @@ fn control_is_abandoned_successor(disposition: &IlmRecoveryDisposition, control:
 async fn abandon_recovery_control_fenced(
     api: Arc<ECStore>,
     disposition: &IlmRecoveryDisposition,
-    control_guard: &rustfs_lock::NamespaceLockGuard,
+    control_guard: &Arc<rustfs_lock::NamespaceLockGuard>,
     source_guard: &rustfs_lock::NamespaceLockGuard,
     proof: &IlmRecoveryExportFleetProofToken,
 ) -> EcstoreResult<()> {
@@ -1557,7 +1559,7 @@ async fn abandon_recovery_control_fenced(
         }),
         ..Default::default()
     };
-    options.add_namespace_lock_guard(control_guard);
+    options.add_owned_write_lock(Arc::clone(control_guard), RUSTFS_META_BUCKET, &object);
     options.add_namespace_lock_guard(source_guard);
     let write_result = config_boundary::save_config_with_opts(api.clone(), &object, encoded, &options).await;
     let observed = match load_recovery_control_no_lock(api.clone(), disposition).await {
@@ -1630,7 +1632,7 @@ async fn delete_exact_local_recovery_copy(
     api: Arc<ECStore>,
     copy: &IlmRecoverySourceCopy,
     source_schema: &str,
-    control_guard: &rustfs_lock::NamespaceLockGuard,
+    control_guard: &Arc<rustfs_lock::NamespaceLockGuard>,
     source_guard: &rustfs_lock::NamespaceLockGuard,
 ) -> EcstoreResult<()> {
     let set = recovery_copy_set(&api, &copy.authority)?;
@@ -1704,7 +1706,7 @@ async fn save_recovery_disposition_step_fenced(
     api: Arc<ECStore>,
     current: &ObservedIlmRecoveryDisposition,
     next: &IlmRecoveryDisposition,
-    control_guard: &rustfs_lock::NamespaceLockGuard,
+    control_guard: &Arc<rustfs_lock::NamespaceLockGuard>,
     source_guard: &rustfs_lock::NamespaceLockGuard,
     proof: &IlmRecoveryExportFleetProofToken,
 ) -> EcstoreResult<ObservedIlmRecoveryDisposition> {

crates/ecstore/src/bucket/migration.rs

@@ -280,11 +280,11 @@ where
 
     debug!("Found {} migrating bucket metadata, migrating...", buckets.len());
 
-    let opts = ObjectOptions {
+    let mut opts = ObjectOptions {
         max_parity: true,
-        no_lock: true,
         ..Default::default()
     };
+    opts.use_offline_recovery_write();
     let h = HeaderMap::new();
 
     for bucket in buckets {
@@ -397,11 +397,11 @@ where
         return Ok(());
     }
 
-    let opts = ObjectOptions {
+    let mut opts = ObjectOptions {
         max_parity: true,
-        no_lock: true,
         ..Default::default()
     };
+    opts.use_offline_recovery_write();
     let h = HeaderMap::new();
     let prefix = format!("{IAM_CONFIG_PREFIX}/");
     let mut continuation: Option<String> = None;

crates/ecstore/src/bucket/quota/reservation.rs

@@ -273,7 +273,7 @@ impl QuotaContext {
             reap_stale_reservations(Arc::clone(&store), &ledger_data.bucket, &ledger_data.ledger_object).await?;
 
             let ledger_lock = store.new_ns_lock(RUSTFS_META_BUCKET, &ledger_data.ledger_object).await?;
-            let ledger_guard = ledger_lock.get_write_lock(get_lock_acquire_timeout()).await?;
+            let ledger_guard = Arc::new(ledger_lock.get_write_lock(get_lock_acquire_timeout()).await?);
             fence_namespace_mutations(&store, RUSTFS_META_BUCKET, &ledger_data.ledger_object, None).await?;
             let mut ledger = load_current_ledger_locked(
                 Arc::clone(&store),
@@ -623,7 +623,7 @@ async fn mark_commit_started(data: &LedgerReservationData) -> Result<()> {
     let data = data.clone();
     tokio::spawn(async move {
         let ledger_lock = data.store.new_ns_lock(RUSTFS_META_BUCKET, &data.ledger_object).await?;
-        let ledger_guard = ledger_lock.get_write_lock(get_lock_acquire_timeout()).await?;
+        let ledger_guard = Arc::new(ledger_lock.get_write_lock(get_lock_acquire_timeout()).await?);
         fence_namespace_mutations(&data.store, RUSTFS_META_BUCKET, &data.ledger_object, None).await?;
         let mut ledger = load_ledger_locked(Arc::clone(&data.store), &data.ledger_object).await?;
         ledger.mark_commit_started(data.operation_id, &data.reservation)?;
@@ -644,7 +644,7 @@ async fn settle(data: &LedgerReservationData, committed: bool) -> Result<()> {
         // object lock is released and would otherwise revoke a later write's
         // newly acquired fence for the same object.
         let ledger_lock = store.new_ns_lock(RUSTFS_META_BUCKET, &ledger_object).await?;
-        let ledger_guard = ledger_lock.get_write_lock(get_lock_acquire_timeout()).await?;
+        let ledger_guard = Arc::new(ledger_lock.get_write_lock(get_lock_acquire_timeout()).await?);
         fence_namespace_mutations(&store, RUSTFS_META_BUCKET, &ledger_object, None).await?;
         let mut ledger = load_ledger_locked(Arc::clone(&store), &ledger_object).await?;
         if committed {
@@ -727,7 +727,7 @@ async fn reap_stale_reservations(store: Arc<ECStore>, bucket: &str, ledger_objec
     }
 
     let ledger_lock = store.new_ns_lock(RUSTFS_META_BUCKET, ledger_object).await?;
-    let ledger_guard = ledger_lock.get_write_lock(get_lock_acquire_timeout()).await?;
+    let ledger_guard = Arc::new(ledger_lock.get_write_lock(get_lock_acquire_timeout()).await?);
     fence_namespace_mutations(&store, RUSTFS_META_BUCKET, ledger_object, None).await?;
     let mut ledger = load_ledger_locked(Arc::clone(&store), ledger_object).await?;
     let cursor_changed = next_cursor.is_some() && ledger.reap_cursor != next_cursor;
@@ -887,7 +887,7 @@ async fn save_ledger_locked(
     store: Arc<ECStore>,
     ledger_object: &str,
     ledger: &QuotaLedger,
-    ledger_guard: &NamespaceLockGuard,
+    ledger_guard: &Arc<NamespaceLockGuard>,
 ) -> Result<()> {
     if ledger_guard.is_lock_lost() {
         return Err(StorageError::NamespaceLockQuorumUnavailable {
@@ -908,7 +908,8 @@ async fn save_ledger_locked(
         ..Default::default()
     };
     let _ = opts.set_quota_admission(0, u64::MAX);
-    opts.add_namespace_lock_guard(ledger_guard);
+    opts.add_owned_write_lock(Arc::clone(ledger_guard), RUSTFS_META_BUCKET, ledger_object);
+    opts.write_completion = crate::object_api::WriteCompletion::TailDrained;
     save_config_with_opts(store, ledger_object, serde_json::to_vec(ledger)?, &opts).await
 }
 

crates/ecstore/src/bucket/replication/replication_config_store.rs

@@ -13,7 +13,9 @@
 // limitations under the License.
 
 use super::replication_error_boundary::Result;
-use super::replication_storage_boundary::{HTTPPreconditions, ObjectInfo, ObjectOptions, ReplicationObjectIO};
+use super::replication_storage_boundary::{
+    HTTPPreconditions, ObjectInfo, ObjectOptions, ReplicationObjectIO, WriteCommitGuard, WriteCompletion,
+};
 use crate::config::{com, storageclass};
 use std::sync::Arc;
 
@@ -77,13 +79,6 @@ impl ReplicationConfigStore {
         com::save_config(api, file, data).await
     }
 
-    pub(crate) async fn save_no_lock<S>(api: Arc<S>, file: &str, data: Vec<u8>) -> Result<()>
-    where
-        S: ReplicationObjectIO,
-    {
-        com::save_config_no_lock(api, file, data).await
-    }
-
     pub(crate) async fn save_conditional<S>(
         api: Arc<S>,
         file: &str,
@@ -111,21 +106,18 @@ impl ReplicationConfigStore {
         file: &str,
         data: Vec<u8>,
         http_preconditions: HTTPPreconditions,
+        guard: &WriteCommitGuard,
     ) -> Result<()>
     where
         S: ReplicationObjectIO,
     {
-        com::save_config_with_opts_quiet(
-            api,
-            file,
-            data,
-            &ObjectOptions {
-                max_parity: true,
-                no_lock: true,
-                http_preconditions: Some(http_preconditions),
-                ..Default::default()
-            },
-        )
-        .await
+        let mut opts = ObjectOptions {
+            max_parity: true,
+            http_preconditions: Some(http_preconditions),
+            write_completion: WriteCompletion::TailDrained,
+            ..Default::default()
+        };
+        opts.add_write_commit_guard(guard);
+        com::save_config_with_opts_quiet(api, file, data, &opts).await
     }
 }

crates/ecstore/src/bucket/replication/replication_pool.rs

@@ -44,7 +44,7 @@ use super::replication_resyncer::{
 use super::replication_state::ReplicationStats;
 use super::replication_storage_boundary::{
     HTTPPreconditions, ObjectInfo, ObjectOptions, ObjectToDelete, ReplicationDeletedObject, ReplicationObjectIO,
-    ReplicationStorage,
+    ReplicationStorage, WriteCommitGuard,
 };
 use super::replication_target_boundary::{BucketTargetError, ReplicationTargetStore, replication_object_is_ssec_encrypted};
 use super::replication_versioning_boundary::ReplicationVersioningStore;
@@ -513,7 +513,7 @@ where
             // Lock order is transaction namespace lock -> force-delete journal object lock.
             // Keep the transaction guard alive through the conditional write so legacy
             // writers cannot interleave a read-modify-write transition within this process.
-            let guard = lock.get_write_lock(ReplicationLockTiming::acquire_timeout()).await?;
+            let guard = WriteCommitGuard::acquire(&lock, ReplicationLockTiming::acquire_timeout()).await?;
             let (mut entries, preconditions, exists) = read_force_delete_intents(storage.clone(), file).await?;
             if !update(&mut entries, exists)? {
                 return Ok(());
@@ -565,7 +565,7 @@ async fn read_force_delete_intents<S: ReplicationObjectIO>(
 async fn save_force_delete_intents<S: ReplicationStorage>(
     storage: Arc<S>,
     file: &str,
-    guard: &rustfs_lock::NamespaceLockGuard,
+    guard: &WriteCommitGuard,
     entries: Vec<MrfReplicateEntry>,
     preconditions: HTTPPreconditions,
 ) -> Result<(), EcstoreError> {
@@ -628,7 +628,7 @@ async fn acknowledge_mrf_recovery<S: ReplicationStorage>(
         let lock = storage
             .new_ns_lock(ReplicationMetadataStore::rustfs_meta_bucket(), file)
             .await?;
-        let guard = lock.get_write_lock(ReplicationLockTiming::acquire_timeout()).await?;
+        let guard = WriteCommitGuard::acquire(&lock, ReplicationLockTiming::acquire_timeout()).await?;
         let current = ReplicationConfigStore::read_no_lock_with_metadata_preserve_empty(storage.clone(), file).await;
         let (current_data, current_etag, current_exists) = match current {
             Ok((data, object_info)) => (data, object_info.etag, true),
@@ -658,7 +658,7 @@ async fn acknowledge_mrf_recovery<S: ReplicationStorage>(
         } else {
             encode_mrf_file(&retained)?
         };
-        match ReplicationConfigStore::save_conditional_no_lock(storage.clone(), file, data, preconditions).await {
+        match ReplicationConfigStore::save_conditional_no_lock(storage.clone(), file, data, preconditions, &guard).await {
             Ok(()) => return Ok(retained),
             Err(EcstoreError::PreconditionFailed) => continue,
             Err(error) => return Err(error),
@@ -2522,7 +2522,7 @@ async fn quarantine_mrf_file<S: ReplicationStorage>(storage: &Arc<S>, data: &[u8
                 continue;
             }
         };
-        let guard = match lock.get_write_lock(ReplicationLockTiming::acquire_timeout()).await {
+        let guard = match WriteCommitGuard::acquire(&lock, ReplicationLockTiming::acquire_timeout()).await {
             Ok(guard) => guard,
             Err(error) => {
                 warn!(
@@ -2573,6 +2573,7 @@ async fn quarantine_mrf_file<S: ReplicationStorage>(storage: &Arc<S>, data: &[u8
                     ReplicationMetadataStore::MRF_REPLICATION_FILE,
                     Vec::new(),
                     preconditions,
+                    &guard,
                 )
                 .await
                 {
@@ -2627,13 +2628,12 @@ async fn recover_corrupt_mrf_generation<S: ReplicationStorage>(
     entries: &[MrfReplicateEntry],
     preconditions: Option<HTTPPreconditions>,
     storage: &Arc<S>,
-    guard: &rustfs_lock::NamespaceLockGuard,
+    guard: &WriteCommitGuard,
     pending_payload: &mut Option<PendingMrfAppend>,
     started: Instant,
 ) -> Option<MrfAppendResult> {
     let quarantine_file = format!("{MRF_CORRUPT_FILE_PREFIX}.{}.bin", OffsetDateTime::now_utc().unix_timestamp_nanos());
-    if let Err(error) = ReplicationConfigStore::save_no_lock(storage.clone(), &quarantine_file, corrupt_generation.to_vec()).await
-    {
+    if let Err(error) = ReplicationConfigStore::save(storage.clone(), &quarantine_file, corrupt_generation.to_vec()).await {
         warn!(
             component = LOG_COMPONENT_ECSTORE,
             subsystem = LOG_SUBSYSTEM_REPLICATION,
@@ -2688,6 +2688,7 @@ async fn recover_corrupt_mrf_generation<S: ReplicationStorage>(
         ReplicationMetadataStore::MRF_REPLICATION_FILE,
         data,
         preconditions,
+        guard,
     )
     .await
     {
@@ -2735,7 +2736,7 @@ async fn append_mrf_entries_to_disk<S: ReplicationStorage>(
             return None;
         }
     };
-    let guard = match lock.get_write_lock(ReplicationLockTiming::acquire_timeout()).await {
+    let guard = match WriteCommitGuard::acquire(&lock, ReplicationLockTiming::acquire_timeout()).await {
         Ok(guard) => guard,
         Err(error) => {
             warn!(
@@ -2862,6 +2863,7 @@ async fn append_mrf_entries_to_disk<S: ReplicationStorage>(
         ReplicationMetadataStore::MRF_REPLICATION_FILE,
         data,
         preconditions,
+        &guard,
     )
     .await
     {

crates/ecstore/src/bucket/replication/replication_storage_boundary.rs

@@ -19,7 +19,8 @@ use super::replication_error_boundary::Error;
 use super::replication_filemeta_boundary::{replication_state_from_filemeta, version_purge_status_from_filemeta};
 pub(crate) type ReplicationObjectStore = crate::store::ECStore;
 pub(crate) use crate::object_api::{
-    GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader, ReplicationStatusWritebackCondition, ReplicationStatusWritebackMode,
+    GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader, ReplicationStatusWritebackCondition,
+    ReplicationStatusWritebackMode, WriteCommitGuard, WriteCompletion,
 };
 #[cfg(test)]
 pub(crate) use crate::object_api::{NamespaceLockFence, NamespaceLockSignalTestFence};

crates/ecstore/src/cluster/rpc/network_probe.rs

@@ -56,7 +56,7 @@ impl DiagnosticPacing {
         // Reservation is never refunded: a failed or dropped RPC may have sent some payload.
         let previous = self
             .charged_bytes
-            .fetch_update(Ordering::AcqRel, Ordering::Acquire, |charged| {
+            .try_update(Ordering::AcqRel, Ordering::Acquire, |charged| {
                 charged.checked_add(bytes).filter(|total| *total <= MAX_NETWORK_PROBE_BYTES)
             })
             .map_err(|_| NetworkPeerProbeError::LimitExceeded)?;

crates/ecstore/src/config/com.rs

@@ -74,7 +74,7 @@ fn config_task_join_error(operation: &'static str, error: tokio::task::JoinError
 /// The operation must use the corresponding no-lock read/save functions.
 pub async fn with_server_config_write_lock<F, Fut, T>(store: Arc<ECStore>, operation: F) -> Result<T>
 where
-    F: FnOnce() -> Fut + Send + 'static,
+    F: FnOnce(crate::object_api::WriteCommitGuard) -> Fut + Send + 'static,
     Fut: std::future::Future<Output = T> + Send + 'static,
     T: Send + 'static,
 {
@@ -85,8 +85,8 @@ where
         let transaction_lock = store.new_ns_lock(RUSTFS_META_BUCKET, &transaction_lock).await?;
         let _transaction_guard = transaction_lock.get_write_lock(get_lock_acquire_timeout()).await?;
         let namespace_lock = store.new_ns_lock(RUSTFS_META_BUCKET, SERVER_CONFIG_OBJECT).await?;
-        let _write_guard = namespace_lock.get_write_lock(get_lock_acquire_timeout()).await?;
-        Ok(operation().await)
+        let write_guard = crate::object_api::WriteCommitGuard::acquire(&namespace_lock, get_lock_acquire_timeout()).await?;
+        Ok(operation(write_guard.clone()).await)
     })
     .await
     .map_err(|error| config_task_join_error("server config write", error))?
@@ -119,14 +119,14 @@ where
 /// one metadata config object. The operation must use no-lock object I/O.
 pub async fn with_config_object_write_lock<F, Fut, T>(store: Arc<ECStore>, object: String, operation: F) -> Result<T>
 where
-    F: FnOnce() -> Fut + Send + 'static,
+    F: FnOnce(crate::object_api::WriteCommitGuard) -> Fut + Send + 'static,
     Fut: std::future::Future<Output = T> + Send + 'static,
     T: Send + 'static,
 {
     tokio::spawn(async move {
         let namespace_lock = store.new_ns_lock(RUSTFS_META_BUCKET, &object).await?;
-        let _write_guard = namespace_lock.get_write_lock(get_lock_acquire_timeout()).await?;
-        Ok(operation().await)
+        let write_guard = crate::object_api::WriteCommitGuard::acquire(&namespace_lock, get_lock_acquire_timeout()).await?;
+        Ok(operation(write_guard.clone()).await)
     })
     .await
     .map_err(|error| config_task_join_error("config object write", error))?
@@ -638,7 +638,12 @@ where
     .await
 }
 
-pub async fn save_config_no_lock<S>(api: Arc<S>, file: &str, data: Vec<u8>) -> Result<()>
+pub async fn save_config_no_lock<S>(
+    api: Arc<S>,
+    file: &str,
+    data: Vec<u8>,
+    guard: &crate::object_api::WriteCommitGuard,
+) -> Result<()>
 where
     S: ObjectIO<
             Error = Error,
@@ -650,17 +655,13 @@ where
             PutObjectReader = PutObjReader,
         >,
 {
-    save_config_with_opts(
-        api,
-        file,
-        data,
-        &ObjectOptions {
-            max_parity: true,
-            no_lock: true,
-            ..Default::default()
-        },
-    )
-    .await
+    let mut opts = ObjectOptions {
+        max_parity: true,
+        write_completion: crate::object_api::WriteCompletion::TailDrained,
+        ..Default::default()
+    };
+    opts.add_write_commit_guard(guard);
+    save_config_with_opts(api, file, data, &opts).await
 }
 
 /// `delete_config` with `no_lock` set — for callers already holding the
@@ -846,12 +847,12 @@ where
     Ok(cfg)
 }
 
-async fn new_and_save_server_config_no_lock<S>(api: Arc<S>) -> Result<Config>
+async fn new_and_save_server_config_no_lock<S>(api: Arc<S>, guard: &crate::object_api::WriteCommitGuard) -> Result<Config>
 where
     S: EcstoreObjectIO + StorageAdminApi,
 {
     let cfg = new_server_config();
-    save_server_config_no_lock(api, &cfg).await?;
+    save_server_config_no_lock(api, &cfg, guard).await?;
     Ok(cfg)
 }
 
@@ -2138,14 +2139,18 @@ where
 }
 
 /// Handle the situation where the configuration file does not exist, create and save a new configuration
-async fn handle_missing_config<S>(api: Arc<S>, context: &str, namespace_lock_held: bool) -> Result<Config>
+async fn handle_missing_config<S>(
+    api: Arc<S>,
+    context: &str,
+    write_guard: Option<&crate::object_api::WriteCommitGuard>,
+) -> Result<Config>
 where
     S: EcstoreObjectIO + StorageAdminApi + NamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
 {
     warn!("Configuration not found ({}): Start initializing new configuration", context);
     let cfg = if runtime_sources::first_cluster_node_is_local().await {
-        if namespace_lock_held {
-            new_and_save_server_config_no_lock(api.clone()).await?
+        if let Some(guard) = write_guard {
+            new_and_save_server_config_no_lock(api.clone(), guard).await?
         } else {
             new_and_save_server_config(api.clone()).await?
         }
@@ -2175,13 +2180,13 @@ where
         > + StorageAdminApi,
     S: NamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
 {
-    read_config_without_migrate_inner(api, false).await
+    read_config_without_migrate_inner(api, None).await
 }
 
 /// Reads the server config while an upper layer holds the namespace write lock
 /// for [`SERVER_CONFIG_OBJECT`]. Missing-config initialization uses the matching
 /// no-lock save path and therefore cannot recursively acquire the same lock.
-pub async fn read_config_without_migrate_no_lock<S>(api: Arc<S>) -> Result<Config>
+pub async fn read_config_without_migrate_no_lock<S>(api: Arc<S>, guard: &crate::object_api::WriteCommitGuard) -> Result<Config>
 where
     S: ObjectIO<
             Error = Error,
@@ -2194,7 +2199,7 @@ where
         > + StorageAdminApi
         + NamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
 {
-    read_config_without_migrate_inner(api, true).await
+    read_config_without_migrate_inner(api, Some(guard)).await
 }
 
 /// Reads an already-initialized server config while a caller owns a namespace
@@ -2204,7 +2209,10 @@ pub async fn read_existing_server_config_no_lock(api: Arc<ECStore>) -> Result<Co
     Ok(decode_persisted_server_config(&data)?.merge())
 }
 
-async fn read_config_without_migrate_inner<S>(api: Arc<S>, namespace_lock_held: bool) -> Result<Config>
+async fn read_config_without_migrate_inner<S>(
+    api: Arc<S>,
+    write_guard: Option<&crate::object_api::WriteCommitGuard>,
+) -> Result<Config>
 where
     S: ObjectIO<
             Error = Error,
@@ -2220,19 +2228,23 @@ where
     let config_file = server_config_path();
 
     // Try to read the configuration file.
-    let data = if namespace_lock_held {
+    let data = if write_guard.is_some() {
         read_config_no_lock(api.clone(), &config_file).await
     } else {
         read_config(api.clone(), &config_file).await
     };
     match data {
-        Ok(data) => read_server_config(api, &data, namespace_lock_held).await,
-        Err(Error::ConfigNotFound) => handle_missing_config(api, "Read the main configuration", namespace_lock_held).await,
+        Ok(data) => read_server_config(api, &data, write_guard).await,
+        Err(Error::ConfigNotFound) => handle_missing_config(api, "Read the main configuration", write_guard).await,
         Err(err) => handle_config_read_error(err, &config_file),
     }
 }
 
-async fn read_server_config<S>(api: Arc<S>, data: &[u8], namespace_lock_held: bool) -> Result<Config>
+async fn read_server_config<S>(
+    api: Arc<S>,
+    data: &[u8],
+    write_guard: Option<&crate::object_api::WriteCommitGuard>,
+) -> Result<Config>
 where
     S: EcstoreObjectIO + StorageAdminApi + NamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
 {
@@ -2242,7 +2254,7 @@ where
         warn!("Received empty configuration data, try to reread from '{}'", config_file);
 
         // Try to read the configuration again
-        let data = if namespace_lock_held {
+        let data = if write_guard.is_some() {
             read_config_no_lock(api.clone(), &config_file).await
         } else {
             read_config(api.clone(), &config_file).await
@@ -2253,7 +2265,7 @@ where
                 return Ok(cfg.merge());
             }
             Err(Error::ConfigNotFound) => {
-                return handle_missing_config(api, "Read alternate configuration", namespace_lock_held).await;
+                return handle_missing_config(api, "Read alternate configuration", write_guard).await;
             }
             Err(err) => return handle_config_read_error(err, &config_file),
         }
@@ -2687,7 +2699,7 @@ where
 
 /// Saves the server config while an upper layer holds the namespace write
 /// lock for [`SERVER_CONFIG_OBJECT`].
-pub async fn save_server_config_no_lock<S>(api: Arc<S>, cfg: &Config) -> Result<()>
+pub async fn save_server_config_no_lock<S>(api: Arc<S>, cfg: &Config, guard: &crate::object_api::WriteCommitGuard) -> Result<()>
 where
     S: ObjectIO<
             Error = Error,
@@ -2699,10 +2711,10 @@ where
             PutObjectReader = PutObjReader,
         >,
 {
-    save_server_config_inner(api, cfg, true).await
+    save_server_config_inner(api, cfg, guard).await
 }
 
-async fn save_server_config_inner<S>(api: Arc<S>, cfg: &Config, no_lock: bool) -> Result<()>
+async fn save_server_config_inner<S>(api: Arc<S>, cfg: &Config, guard: &crate::object_api::WriteCommitGuard) -> Result<()>
 where
     S: ObjectIO<
             Error = Error,
@@ -2715,11 +2727,7 @@ where
         >,
 {
     let config_file = get_config_file();
-    let existing = match if no_lock {
-        read_config_no_lock(api.clone(), &config_file).await
-    } else {
-        read_config(api.clone(), &config_file).await
-    } {
+    let existing = match read_config_no_lock(api.clone(), &config_file).await {
         Ok(v) => Some(v),
         Err(Error::ConfigNotFound) => None,
         Err(err) => {
@@ -2743,21 +2751,7 @@ where
         return Ok(());
     }
 
-    if no_lock {
-        save_config_with_opts(
-            api,
-            &config_file,
-            data,
-            &ObjectOptions {
-                max_parity: true,
-                no_lock: true,
-                ..Default::default()
-            },
-        )
-        .await
-    } else {
-        save_config(api, &config_file, data).await
-    }
+    save_config_no_lock(api, &config_file, data, guard).await
 }
 
 pub async fn lookup_configs<S>(cfg: &mut Config, api: Arc<S>) -> Result<()>

crates/ecstore/src/core/pools.rs

@@ -3692,8 +3692,8 @@ fn activation_pool_meta_lock_error(err: rustfs_lock::LockError) -> Error {
 }
 
 pub(crate) struct PoolRebalanceActivationFence {
-    pool_meta_guard: rustfs_lock::NamespaceLockGuard,
-    rebalance_meta_guard: rustfs_lock::NamespaceLockGuard,
+    pool_meta_guard: Arc<rustfs_lock::NamespaceLockGuard>,
+    rebalance_meta_guard: Arc<rustfs_lock::NamespaceLockGuard>,
     fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
     decommission_target_fence_proof: Option<crate::services::notification_sys::DecommissionTargetFenceFleetProofToken>,
     #[cfg(test)]
@@ -3720,7 +3720,12 @@ impl PoolRebalanceActivationFence {
         let forced_lost = self.forced_lost.load(Ordering::Acquire);
         #[cfg(not(test))]
         let forced_lost = false;
-        if forced_lost || self.pool_meta_guard.is_lock_lost() || self.rebalance_meta_guard.is_lock_lost() {
+        if forced_lost
+            || self.pool_meta_guard.is_lock_lost()
+            || self.pool_meta_guard.is_released()
+            || self.rebalance_meta_guard.is_lock_lost()
+            || self.rebalance_meta_guard.is_released()
+        {
             return Err(Error::other("activation lock lost before metadata commit or worker admission"));
         }
         if self
@@ -3742,8 +3747,8 @@ impl PoolRebalanceActivationFence {
     }
 
     pub(crate) fn add_namespace_lock_fence(&self, opts: &mut ObjectOptions) {
-        opts.add_namespace_lock_guard(&self.pool_meta_guard);
-        opts.add_namespace_lock_guard(&self.rebalance_meta_guard);
+        opts.add_owned_write_lock(Arc::clone(&self.pool_meta_guard), RUSTFS_META_BUCKET, POOL_META_NAME);
+        opts.add_owned_write_lock(Arc::clone(&self.rebalance_meta_guard), RUSTFS_META_BUCKET, REBAL_META_NAME);
     }
 
     #[cfg(test)]
@@ -3784,8 +3789,8 @@ where
         .map_err(activation_pool_meta_lock_error)?;
 
     Ok(PoolRebalanceActivationFence {
-        pool_meta_guard,
-        rebalance_meta_guard,
+        pool_meta_guard: Arc::new(pool_meta_guard),
+        rebalance_meta_guard: Arc::new(rebalance_meta_guard),
         fleet_proof,
         decommission_target_fence_proof: None,
         #[cfg(test)]
@@ -4067,7 +4072,7 @@ fn rollback_start_decommission_pool_meta(pool_meta: &mut PoolMeta, previous_pool
 }
 
 fn ensure_pool_meta_write_fence(guard: &rustfs_lock::NamespaceLockGuard, operation: &str) -> Result<()> {
-    if guard.is_lock_lost() {
+    if guard.is_lock_lost() || guard.is_released() {
         return Err(Error::other(format!("{operation}: pool metadata distributed fence was lost")));
     }
     Ok(())
@@ -6339,6 +6344,9 @@ fn encode_pool_meta_v3_envelope(
 }
 
 enum PoolMetaPersistenceFence<'a> {
+    Owned(Arc<rustfs_lock::NamespaceLockGuard>),
+    OfflineRecovery,
+    #[cfg(test)]
     Distributed(Option<Arc<rustfs_lock::distributed_lock::LockLostSignal>>),
     Activation(&'a PoolRebalanceActivationFence),
 }
@@ -6346,6 +6354,10 @@ enum PoolMetaPersistenceFence<'a> {
 impl PoolMetaPersistenceFence<'_> {
     fn ensure_held(&self) -> Result<()> {
         match self {
+            Self::Owned(guard) if guard.is_lock_lost() || guard.is_released() => {
+                Err(Error::other("pool metadata distributed fence was lost before a replica write"))
+            }
+            #[cfg(test)]
             Self::Distributed(Some(signal)) if signal.is_lost() => {
                 Err(Error::other("pool metadata distributed fence was lost before a replica write"))
             }
@@ -6354,14 +6366,30 @@ impl PoolMetaPersistenceFence<'_> {
         }
     }
 
-    fn add_to_options(&self, opts: &mut ObjectOptions) {
+    fn add_to_options(&self, opts: &mut ObjectOptions, object: &str) {
         match self {
+            Self::Owned(guard) => opts.add_owned_write_lock(Arc::clone(guard), RUSTFS_META_BUCKET, object),
+            Self::OfflineRecovery => opts.use_offline_recovery_write(),
+            #[cfg(test)]
             Self::Distributed(Some(signal)) => opts.add_namespace_lock_lost_signal(Arc::clone(signal)),
-            Self::Activation(fence) => fence.add_namespace_lock_fence(opts),
+            Self::Activation(fence) => {
+                fence.add_namespace_lock_fence(opts);
+                opts.add_owned_write_lock(Arc::clone(&fence.pool_meta_guard), RUSTFS_META_BUCKET, object);
+            }
+            #[cfg(test)]
             Self::Distributed(None) => {}
         }
     }
 
+    fn allows_concurrent_phases(&self) -> bool {
+        match self {
+            Self::Owned(guard) => guard.lock_lost_signal().is_some(),
+            #[cfg(test)]
+            Self::Distributed(signal) => signal.is_some(),
+            _ => false,
+        }
+    }
+
     fn is_activation(&self) -> bool {
         matches!(self, Self::Activation(_))
     }
@@ -6662,7 +6690,7 @@ fn pool_meta_cas_options(object: &str, token: &PoolMetaCasToken, fence: &PoolMet
         http_preconditions: Some(pool_meta_cas_preconditions(token, object)?),
         ..Default::default()
     };
-    fence.add_to_options(&mut opts);
+    fence.add_to_options(&mut opts, object);
     Ok(opts)
 }
 
@@ -7049,7 +7077,7 @@ where
         write_state,
         initialized,
         PoolMetaIdentityWriteScope::All,
-        &PoolMetaPersistenceFence::Distributed(None),
+        &PoolMetaPersistenceFence::OfflineRecovery,
         &mut transaction_arm,
     )
     .await?;
@@ -7076,7 +7104,7 @@ where
         write_state,
         false,
         PoolMetaIdentityWriteScope::Pools(pool_indices),
-        &PoolMetaPersistenceFence::Distributed(None),
+        &PoolMetaPersistenceFence::OfflineRecovery,
         &mut transaction_arm,
     )
     .await?;
@@ -8154,11 +8182,16 @@ impl PoolMeta {
             .cloned()
             .ok_or_else(|| Error::other("pool metadata save failed: no storage pools available"))?;
         let pool_meta_lock = pool.new_ns_lock(RUSTFS_META_BUCKET, POOL_META_NAME).await?;
-        let pool_meta_guard = pool_meta_lock.get_write_lock(get_lock_acquire_timeout()).await?;
+        let pool_meta_guard = Arc::new(pool_meta_lock.get_write_lock(get_lock_acquire_timeout()).await?);
         let mut write_state = PoolMetaWriteState::default();
         let indices = (0..self.pools.len()).collect::<Vec<_>>();
         let outcome = self
-            .save_no_lock_armed_scoped(pools, &mut write_state, pool_meta_guard.lock_lost_signal(), Some(&indices))
+            .save_no_lock_armed_scoped(
+                pools,
+                &mut write_state,
+                PoolMetaPersistenceFence::Owned(Arc::clone(&pool_meta_guard)),
+                Some(&indices),
+            )
             .await?;
         outcome.disarm();
         Ok(())
@@ -8182,14 +8215,16 @@ impl PoolMeta {
     where
         S: EcstoreObjectIO,
     {
-        let outcome = self.save_no_lock_armed_scoped(pools, write_state, None, None).await?;
+        let outcome = self
+            .save_no_lock_armed_scoped(pools, write_state, PoolMetaPersistenceFence::OfflineRecovery, None)
+            .await?;
         Ok(outcome.into_committed())
     }
 
     async fn save_no_lock_with_fence<S>(
         &self,
         pools: Vec<Arc<S>>,
-        lock_lost: Option<Arc<rustfs_lock::distributed_lock::LockLostSignal>>,
+        fence: PoolMetaPersistenceFence<'_>,
         indices: &[usize],
     ) -> Result<()>
     where
@@ -8197,7 +8232,7 @@ impl PoolMeta {
     {
         let mut write_state = PoolMetaWriteState::default();
         let outcome = self
-            .save_no_lock_armed_scoped(pools, &mut write_state, lock_lost, Some(indices))
+            .save_no_lock_armed_scoped(pools, &mut write_state, fence, Some(indices))
             .await?;
         outcome.disarm();
         Ok(())
@@ -8233,21 +8268,20 @@ impl PoolMeta {
         &self,
         pools: Vec<Arc<S>>,
         write_state: &mut PoolMetaWriteState,
-        lock_lost: Option<Arc<rustfs_lock::distributed_lock::LockLostSignal>>,
+        fence: PoolMetaPersistenceFence<'_>,
         indices: &[usize],
     ) -> Result<PoolMetaSaveOutcome>
     where
         S: EcstoreObjectIO,
     {
-        self.save_no_lock_armed_scoped(pools, write_state, lock_lost, Some(indices))
-            .await
+        self.save_no_lock_armed_scoped(pools, write_state, fence, Some(indices)).await
     }
 
     async fn save_no_lock_armed_scoped<S>(
         &self,
         pools: Vec<Arc<S>>,
         write_state: &mut PoolMetaWriteState,
-        lock_lost: Option<Arc<rustfs_lock::distributed_lock::LockLostSignal>>,
+        fence: PoolMetaPersistenceFence<'_>,
         indices: Option<&[usize]>,
     ) -> Result<PoolMetaSaveOutcome>
     where
@@ -8258,7 +8292,6 @@ impl PoolMeta {
         // returned outcome is dropped before publication, Drop latches the
         // sticky recovery gate.
         let mut transaction_arm = write_state.arm_transaction();
-        let fence = PoolMetaPersistenceFence::Distributed(lock_lost);
         let committed = self
             .save_no_lock_transaction(pools, write_state, &fence, indices, &mut transaction_arm)
             .await
@@ -8489,7 +8522,7 @@ impl PoolMeta {
             && selection.revision.is_generation_protocol()
             && !selection.replica_state.needs_repair
             && !bootstrap_generation_required
-            && matches!(fence, PoolMetaPersistenceFence::Distributed(Some(_)));
+            && fence.allows_concurrent_phases();
         #[cfg(feature = "e2e-test-hooks")]
         let phase_barrier = if concurrent_phases && pools.len() == 4 {
             PoolMetaPhaseBarrier::bind(&previous, &committed, revision, &durable).await?
@@ -8635,7 +8668,9 @@ impl PoolMeta {
         S: EcstoreObjectIO,
     {
         let indices = (0..self.pools.len()).collect::<Vec<_>>();
-        let outcome = self.save_no_lock_armed(pools, write_state, None, &indices).await?;
+        let outcome = self
+            .save_no_lock_armed(pools, write_state, PoolMetaPersistenceFence::OfflineRecovery, &indices)
+            .await?;
         outcome.disarm();
         Ok(())
     }
@@ -10831,7 +10866,7 @@ impl ECStore {
         &self,
         write_state: &mut PoolMetaWriteState,
         operation: &str,
-    ) -> Result<(rustfs_lock::NamespaceLockGuard, PoolMeta)> {
+    ) -> Result<(Arc<rustfs_lock::NamespaceLockGuard>, PoolMeta)> {
         self.acquire_pool_meta_write_guard_with_lock_error(write_state, operation, activation_pool_meta_lock_error)
             .await
     }
@@ -10841,7 +10876,7 @@ impl ECStore {
         write_state: &mut PoolMetaWriteState,
         operation: &str,
         map_lock_error: F,
-    ) -> Result<(rustfs_lock::NamespaceLockGuard, PoolMeta)>
+    ) -> Result<(Arc<rustfs_lock::NamespaceLockGuard>, PoolMeta)>
     where
         F: FnOnce(rustfs_lock::LockError) -> Error,
     {
@@ -10862,7 +10897,7 @@ impl ECStore {
         let selection = load_pool_meta_replicas_observing(self.pools.clone(), true, write_state).await?;
         write_state.observe_replicas(selection.replica_state);
         write_state.ensure_write_safe(operation)?;
-        Ok((pool_meta_guard, selection.meta))
+        Ok((Arc::new(pool_meta_guard), selection.meta))
     }
 
     async fn acquire_pool_meta_read_guard(
@@ -11132,7 +11167,7 @@ impl ECStore {
     pub(crate) async fn acquire_pool_meta_object_heal_fence(
         &self,
         target_pool_indices: &[usize],
-    ) -> Result<(rustfs_lock::NamespaceLockGuard, Vec<Result<()>>)> {
+    ) -> Result<(Arc<rustfs_lock::NamespaceLockGuard>, Vec<Result<()>>)> {
         let mut save_guard = self.pool_meta_save_gate.lock().await;
         let (pool_meta_guard, snapshot) = self
             .acquire_pool_meta_write_guard_with_lock_error(&mut save_guard, "pool metadata heal admission failed", Error::from)
@@ -11361,7 +11396,12 @@ impl ECStore {
         ensure_exact_delete_capacity_namespace_fences(opts, bucket, object)?;
 
         let outcome = snapshot
-            .save_no_lock_armed(self.pools.clone(), &mut save_guard, write_guard.lock_lost_signal(), &source_pool_indices)
+            .save_no_lock_armed(
+                self.pools.clone(),
+                &mut save_guard,
+                PoolMetaPersistenceFence::Owned(Arc::clone(&write_guard)),
+                &source_pool_indices,
+            )
             .await?;
         ensure_pool_meta_write_fence(&write_guard, "exact delete capacity reconciliation save failed")?;
         for (target_pool_index, target_guard) in &target_guards {
@@ -11470,7 +11510,7 @@ impl ECStore {
             .save_no_lock_armed(
                 self.pools.clone(),
                 &mut save_guard,
-                pool_meta_guard.lock_lost_signal(),
+                PoolMetaPersistenceFence::Owned(Arc::clone(&pool_meta_guard)),
                 &[source_pool_index],
             )
             .await?;
@@ -11659,7 +11699,7 @@ impl ECStore {
             .save_no_lock_armed(
                 self.pools.clone(),
                 &mut save_guard,
-                pool_meta_guard.lock_lost_signal(),
+                PoolMetaPersistenceFence::Owned(Arc::clone(&pool_meta_guard)),
                 &[source_pool_index],
             )
             .await?;
@@ -11965,7 +12005,12 @@ impl ECStore {
         };
         if reservation_grew || pending_added > 0 {
             let outcome = snapshot
-                .save_no_lock_armed(self.pools.clone(), &mut save_guard, write_guard.lock_lost_signal(), &[source_pool_index])
+                .save_no_lock_armed(
+                    self.pools.clone(),
+                    &mut save_guard,
+                    PoolMetaPersistenceFence::Owned(Arc::clone(&write_guard)),
+                    &[source_pool_index],
+                )
                 .await?;
             ensure_pool_meta_write_fence(&write_guard, "decommission target capacity intent save failed")?;
             snapshot = outcome.committed.clone();
@@ -12137,7 +12182,12 @@ impl ECStore {
             return result;
         }
         let outcome = snapshot
-            .save_no_lock_armed(self.pools.clone(), save_guard, write_guard.lock_lost_signal(), &[source_pool_index])
+            .save_no_lock_armed(
+                self.pools.clone(),
+                save_guard,
+                PoolMetaPersistenceFence::Owned(Arc::clone(write_guard)),
+                &[source_pool_index],
+            )
             .await?;
         ensure_pool_meta_write_fence(write_guard, "decommission target capacity progress save failed")?;
         if let Some(target_guard) = target_guard.as_ref() {
@@ -12272,8 +12322,8 @@ impl ECStore {
             .first()
             .ok_or_else(|| Error::other("pool metadata recovery has no storage pools"))?;
         let lock = pool.new_ns_lock(RUSTFS_META_BUCKET, POOL_META_NAME).await?;
-        let guard = lock.get_write_lock(get_lock_acquire_timeout()).await?;
-        let fence = PoolMetaPersistenceFence::Distributed(guard.lock_lost_signal());
+        let guard = Arc::new(lock.get_write_lock(get_lock_acquire_timeout()).await?);
+        let fence = PoolMetaPersistenceFence::Owned(Arc::clone(&guard));
         let mut candidate = PoolMetaWriteState {
             aborted_transaction: Arc::new(AtomicBool::new(false)),
             transaction_failure: Arc::default(),
@@ -12449,7 +12499,12 @@ impl ECStore {
             merge_pool_meta_updates_for_save(&mut snapshot, &pool_meta, indices, "pool metadata save failed")?;
         }
         let outcome = snapshot
-            .save_no_lock_armed(self.pools.clone(), &mut save_guard, pool_meta_guard.lock_lost_signal(), indices)
+            .save_no_lock_armed(
+                self.pools.clone(),
+                &mut save_guard,
+                PoolMetaPersistenceFence::Owned(Arc::clone(&pool_meta_guard)),
+                indices,
+            )
             .await?;
         let mut pool_meta = self.pool_meta.write().await;
         ensure_pool_meta_write_fence(&pool_meta_guard, "pool metadata save failed")?;
@@ -12544,7 +12599,12 @@ impl ECStore {
         };
 
         let outcome = match snapshot
-            .save_no_lock_armed(self.pools.clone(), &mut save_guard, pool_meta_guard.lock_lost_signal(), &[idx])
+            .save_no_lock_armed(
+                self.pools.clone(),
+                &mut save_guard,
+                PoolMetaPersistenceFence::Owned(Arc::clone(&pool_meta_guard)),
+                &[idx],
+            )
             .await
         {
             Ok(outcome) => outcome,
@@ -12610,7 +12670,12 @@ impl ECStore {
         #[cfg(test)]
         decommission_test_wrap_result("bucket_completion_before_save", &bucket.name, &bucket.prefix, 0, Ok(()))?;
         let outcome = snapshot
-            .save_no_lock_armed(self.pools.clone(), &mut save_guard, pool_meta_guard.lock_lost_signal(), &[idx])
+            .save_no_lock_armed(
+                self.pools.clone(),
+                &mut save_guard,
+                PoolMetaPersistenceFence::Owned(Arc::clone(&pool_meta_guard)),
+                &[idx],
+            )
             .await?;
         #[cfg(test)]
         decommission_test_wrap_result("bucket_completion_after_save", &bucket.name, &bucket.prefix, 0, Ok(()))?;
@@ -12744,7 +12809,12 @@ impl ECStore {
             .await?;
         rollback_start_decommission_pool_meta(&mut snapshot, previous_pool_meta, indices);
         let outcome = snapshot
-            .save_no_lock_armed(self.pools.clone(), &mut save_guard, pool_meta_guard.lock_lost_signal(), indices)
+            .save_no_lock_armed(
+                self.pools.clone(),
+                &mut save_guard,
+                PoolMetaPersistenceFence::Owned(Arc::clone(&pool_meta_guard)),
+                indices,
+            )
             .await?;
         let mut pool_meta = self.pool_meta.write().await;
         ensure_pool_meta_write_fence(&pool_meta_guard, "decommission start rollback failed")?;
@@ -12976,7 +13046,7 @@ impl ECStore {
                 .targets
                 .iter()
                 .find(|target| target.pool_index == permitted_target_pool_index)
-                .and_then(&candidate)
+                .and_then(candidate)
             {
                 return Ok(pool_index);
             }
@@ -13044,7 +13114,7 @@ impl ECStore {
         save_pool_meta: Save,
     ) -> Result<()>
     where
-        Save: FnOnce(PoolMeta, Option<Arc<rustfs_lock::distributed_lock::LockLostSignal>>) -> SaveFuture + Send + 'static,
+        Save: FnOnce(PoolMeta, PoolMetaPersistenceFence<'static>) -> SaveFuture + Send + 'static,
         SaveFuture: Future<Output = Result<()>> + Send + 'static,
     {
         let store = self.clone();
@@ -13064,7 +13134,7 @@ impl ECStore {
         save_pool_meta: Save,
     ) -> Result<()>
     where
-        Save: FnOnce(PoolMeta, Option<Arc<rustfs_lock::distributed_lock::LockLostSignal>>) -> SaveFuture,
+        Save: FnOnce(PoolMeta, PoolMetaPersistenceFence<'static>) -> SaveFuture,
         SaveFuture: Future<Output = Result<()>>,
     {
         let owner = owner.as_ref();
@@ -13113,9 +13183,17 @@ impl ECStore {
                 ));
             }
         }
-        let pool_meta_fence = _pool_meta_guard
-            .as_ref()
-            .and_then(rustfs_lock::NamespaceLockGuard::lock_lost_signal);
+        let pool_meta_fence = match _pool_meta_guard.as_ref() {
+            Some(guard) => PoolMetaPersistenceFence::Owned(Arc::clone(guard)),
+            None => {
+                #[cfg(test)]
+                {
+                    PoolMetaPersistenceFence::OfflineRecovery
+                }
+                #[cfg(not(test))]
+                return Err(Error::other("runtime pool metadata cancellation requires its write-lock owner"));
+            }
+        };
 
         // Lock order: start gate, target gates, save gate, distributed pool
         // metadata fence, rebalance_meta, decommission_cancelers, then
@@ -13554,7 +13632,12 @@ impl ECStore {
             let (save_outcome, save_error) = if changed {
                 ensure_decommission_target_fence_fleet_proof(target_fence_proof.as_ref(), target_fence_proof_required)?;
                 match snapshot
-                    .save_no_lock_armed(self.pools.clone(), &mut save_guard, pool_meta_guard.lock_lost_signal(), &changed_indices)
+                    .save_no_lock_armed(
+                        self.pools.clone(),
+                        &mut save_guard,
+                        PoolMetaPersistenceFence::Owned(Arc::clone(&pool_meta_guard)),
+                        &changed_indices,
+                    )
                     .await
                 {
                     Ok(outcome) => (Some(outcome), None),
@@ -13646,7 +13729,12 @@ impl ECStore {
             .await?;
         snapshot.mark_decommission_capacity_blocked(idx, err.to_string(), OffsetDateTime::now_utc())?;
         let outcome = snapshot
-            .save_no_lock_armed(self.pools.clone(), &mut save_guard, pool_meta_guard.lock_lost_signal(), &[idx])
+            .save_no_lock_armed(
+                self.pools.clone(),
+                &mut save_guard,
+                PoolMetaPersistenceFence::Owned(Arc::clone(&pool_meta_guard)),
+                &[idx],
+            )
             .await?;
         ensure_pool_meta_write_fence(&pool_meta_guard, "decommission capacity pause failed")?;
         {
@@ -19858,9 +19946,9 @@ mod tests {
                 .reload_pool_meta()
                 .await
                 .expect("load the active generation on the remote node");
-            other_store
-                .decommission_cancel(0)
+            tokio::time::timeout(std::time::Duration::from_secs(30), other_store.decommission_cancel(0))
                 .await
+                .expect("remote cancel must persist pool metadata without re-entering its state locks")
                 .expect("the remote node should cancel the old generation");
             other_store
                 .clear_decommission(0)
@@ -20125,9 +20213,9 @@ mod tests {
             .expect("the stale operation should have a valid terminal fence plan")
             .expect("the stale operation should have an active reservation");
 
-        first_node
-            .decommission_cancel(0)
+        tokio::time::timeout(std::time::Duration::from_secs(30), first_node.decommission_cancel(0))
             .await
+            .expect("cancel must persist pool metadata without re-entering its state locks")
             .expect("cancel the first durable operation");
         first_node
             .clear_decommission(0)
@@ -20630,8 +20718,9 @@ mod tests {
             .await
             .expect("second target mutation task should not panic")
             .expect("second target mutation should finalize");
-        cancel
+        tokio::time::timeout(std::time::Duration::from_secs(30), cancel)
             .await
+            .expect("cancel must finish after target mutations release their gates")
             .expect("decommission cancellation task should not panic")
             .expect("decommission cancellation should finish after target mutations");
 
@@ -22022,7 +22111,7 @@ mod tests {
             let mut state = store.pool_meta_save_gate.lock().await;
             let (fence, _) = store.acquire_pool_meta_write_guard(&mut state, "test save").await.unwrap();
             let outcome = requested
-                .save_no_lock_armed(store.pools.clone(), &mut state, fence.lock_lost_signal(), &[0])
+                .save_no_lock_armed(store.pools.clone(), &mut state, PoolMetaPersistenceFence::Owned(Arc::clone(&fence)), &[0])
                 .await
                 .unwrap();
             assert_eq!(outcome.committed.pools[0].last_update, requested.pools[0].last_update);
@@ -24824,11 +24913,12 @@ mod pools_tests {
             config[3][0].gate = Some(gate.clone());
             let fixture = Fixture::new(config);
             let mut state = fixture.state();
-            let mut save = Box::pin(
-                fixture
-                    .desired
-                    .save_no_lock_armed(fixture.pools.clone(), &mut state, signal(), &[0]),
-            );
+            let mut save = Box::pin(fixture.desired.save_no_lock_armed(
+                fixture.pools.clone(),
+                &mut state,
+                PoolMetaPersistenceFence::Distributed(signal()),
+                &[0],
+            ));
             tokio::select! { _ = fixture.trace.wait_for(4, 3) => {}, _ = &mut save => panic!("commit crossed the last prepare") }
             assert_eq!(fixture.trace.commit_count(), 0);
             gate.add_permits(1);
@@ -24848,7 +24938,7 @@ mod pools_tests {
             let mut state = fixture.state();
             let err = fixture
                 .desired
-                .save_no_lock_armed(fixture.pools.clone(), &mut state, signal(), &[0])
+                .save_no_lock_armed(fixture.pools.clone(), &mut state, PoolMetaPersistenceFence::Distributed(signal()), &[0])
                 .await
                 .expect_err("a missing conditional-write revision must fail");
             assert!(err.to_string().contains("without a conditional-write revision"));
@@ -24865,11 +24955,12 @@ mod pools_tests {
             config[3][0].gate = Some(gate.clone());
             let fixture = Fixture::new(config);
             let mut state = fixture.state();
-            let mut save = Box::pin(
-                fixture
-                    .desired
-                    .save_no_lock_armed(fixture.pools.clone(), &mut state, signal(), &[0]),
-            );
+            let mut save = Box::pin(fixture.desired.save_no_lock_armed(
+                fixture.pools.clone(),
+                &mut state,
+                PoolMetaPersistenceFence::Distributed(signal()),
+                &[0],
+            ));
             tokio::select! { _ = fixture.trace.wait_for(4, 3) => {}, _ = &mut save => panic!("phase returned with a live replica") }
             assert_eq!(fixture.trace.commit_count(), 0);
             gate.add_permits(1);
@@ -25044,11 +25135,12 @@ mod pools_tests {
                 }
                 let fixture = Fixture::new(config);
                 let mut state = fixture.state();
-                let mut save = Box::pin(
-                    fixture
-                        .desired
-                        .save_no_lock_armed(fixture.pools.clone(), &mut state, signal(), &[0]),
-                );
+                let mut save = Box::pin(fixture.desired.save_no_lock_armed(
+                    fixture.pools.clone(),
+                    &mut state,
+                    PoolMetaPersistenceFence::Distributed(signal()),
+                    &[0],
+                ));
                 tokio::select! { _ = fixture.trace.wait_for(4 * (phase + 1), 4 * phase) => {}, _ = &mut save => panic!("blocked phase finished") }
                 drop(save);
                 assert!(state.ensure_write_safe("cancelled phase").is_err());
@@ -25059,7 +25151,7 @@ mod pools_tests {
             let mut state = fixture.state();
             let outcome = fixture
                 .desired
-                .save_no_lock_armed(fixture.pools.clone(), &mut state, signal(), &[0])
+                .save_no_lock_armed(fixture.pools.clone(), &mut state, PoolMetaPersistenceFence::Distributed(signal()), &[0])
                 .await
                 .expect("durable save before publication");
             drop(outcome);
@@ -25087,6 +25179,7 @@ mod pools_tests {
                 .await
                 .expect("lease request")
                 .expect("lease acquired");
+            let guard = Arc::new(guard);
             let gate = Arc::new(tokio::sync::Semaphore::new(0));
             let mut config = steps(6);
             for step in &mut config {
@@ -25095,7 +25188,7 @@ mod pools_tests {
             let fixture = Fixture::new(config);
             let state = fixture.state();
             let mut arm = state.arm_transaction();
-            let fence = PoolMetaPersistenceFence::Distributed(guard.lock_lost_signal());
+            let fence = PoolMetaPersistenceFence::Owned(Arc::clone(&guard));
             let mut pending = Box::pin(prepare(&fixture, &fence, &mut arm));
             tokio::select! { _ = fixture.trace.wait_for(4, 0) => {}, _ = &mut pending => panic!("gate ignored") }
             tokio::time::timeout(StdDuration::from_secs(3), guard.lock_lost_notified())
@@ -25121,7 +25214,7 @@ mod pools_tests {
                 let mut state = fixture.state();
                 let result = fixture
                     .desired
-                    .save_no_lock_armed(fixture.pools.clone(), &mut state, signal(), &[0])
+                    .save_no_lock_armed(fixture.pools.clone(), &mut state, PoolMetaPersistenceFence::Distributed(signal()), &[0])
                     .await;
                 assert_eq!(fixture.trace.commit_count(), 4);
                 if matches!(fault, WriteFault::TimeoutAfterWrite) {
@@ -25171,7 +25264,7 @@ mod pools_tests {
                 let mut save = Box::pin(fixture.desired.save_no_lock_armed(
                     fixture.pools.clone(),
                     &mut state,
-                    if legacy || needs_repair { signal() } else { None },
+                    PoolMetaPersistenceFence::Distributed(if legacy || needs_repair { signal() } else { None }),
                     &[0],
                 ));
                 tokio::select! { _ = fixture.trace.wait_for(1, 0) => {}, _ = &mut save => panic!("serial write gate ignored") }
@@ -25731,7 +25824,7 @@ mod pools_tests {
         let save_task = tokio::spawn(async move {
             let mut save_guard = save_store.pool_meta_save_gate.lock().await;
             let outcome = snapshot
-                .save_no_lock_armed(vec![save_committed], &mut save_guard, None, &[0])
+                .save_no_lock_armed(vec![save_committed], &mut save_guard, PoolMetaPersistenceFence::OfflineRecovery, &[0])
                 .await?;
             task_publish_started.notify_one();
             task_publish_release.notified().await;
@@ -25758,6 +25851,73 @@ mod pools_tests {
             .expect_err("cancellation after replica save but before publication must keep writes blocked");
     }
 
+    #[tokio::test]
+    async fn pool_meta_cas_context_retains_actual_owner_for_exact_target() {
+        use crate::object_api::write_commit_context::WriteCommitContext;
+
+        let lock = NamespaceLock::with_local_manager("pool-meta-cas-owner".to_string(), Arc::new(GlobalLockManager::new()));
+        let request = LockRequest::new(
+            ObjectKey::new(super::RUSTFS_META_BUCKET, POOL_META_NAME),
+            LockType::Exclusive,
+            "cas-writer",
+        )
+        .with_acquire_timeout(StdDuration::from_secs(1));
+        let owner = Arc::new(
+            lock.acquire_guard(&request)
+                .await
+                .expect("pool metadata lock acquisition should succeed")
+                .expect("pool metadata lock should be available"),
+        );
+        let weak_owner = Arc::downgrade(&owner);
+        let fence = PoolMetaPersistenceFence::Owned(Arc::clone(&owner));
+        let mut contexts = Vec::new();
+        for object in [POOL_META_NAME, POOL_META_IDENTITY_NAME] {
+            let options = super::pool_meta_cas_options(object, &PoolMetaCasToken::Missing, &fence)
+                .expect("a held pool metadata lock should authorize its CAS target");
+            let cloned_options = options.clone();
+            let context = WriteCommitContext::from_options(&cloned_options, super::RUSTFS_META_BUCKET, object, false)
+                .expect("the actual CAS target should retain its borrowed namespace owner");
+            assert!(context.has_borrowed_owner(), "CAS must borrow the real protocol lock owner");
+            assert!(
+                WriteCommitContext::from_options(&cloned_options, super::RUSTFS_META_BUCKET, "unrelated-object", false).is_err(),
+                "a pool metadata CAS capability must not authorize an unrelated object"
+            );
+            drop(options);
+            drop(cloned_options);
+            contexts.push(context);
+        }
+        drop(fence);
+        drop(owner);
+        assert!(
+            weak_owner.upgrade().is_some(),
+            "queued commit contexts must keep the actual lock guard alive"
+        );
+        drop(contexts);
+        assert!(weak_owner.upgrade().is_none(), "the last commit context must release its lock owner");
+    }
+
+    #[tokio::test]
+    async fn pool_meta_cas_rejects_released_owner() {
+        let lock = NamespaceLock::with_local_manager("pool-meta-released-owner".to_string(), Arc::new(GlobalLockManager::new()));
+        let request = LockRequest::new(
+            ObjectKey::new(super::RUSTFS_META_BUCKET, POOL_META_NAME),
+            LockType::Exclusive,
+            "released-cas-writer",
+        )
+        .with_acquire_timeout(StdDuration::from_secs(1));
+        let mut owner = lock
+            .acquire_guard(&request)
+            .await
+            .expect("pool metadata lock acquisition should succeed")
+            .expect("pool metadata lock should be available");
+        assert!(owner.release(), "the test must release a real namespace owner");
+        let fence = PoolMetaPersistenceFence::Owned(Arc::new(owner));
+        assert!(
+            super::pool_meta_cas_options(POOL_META_NAME, &PoolMetaCasToken::Missing, &fence).is_err(),
+            "a released namespace owner must not dispatch another CAS"
+        );
+    }
+
     #[tokio::test]
     async fn test_lost_pool_meta_fence_rejects_replica_write() {
         let client = Arc::new(LocalClient::with_manager(Arc::new(GlobalLockManager::new())));
@@ -25775,6 +25935,7 @@ mod pools_tests {
             .await
             .expect("pool metadata fence acquisition should not error")
             .expect("the stale writer should acquire the pool metadata fence");
+        let guard = Arc::new(guard);
         tokio::time::timeout(StdDuration::from_secs(2), guard.lock_lost_notified())
             .await
             .expect("the stale writer lease should expire");
@@ -25796,7 +25957,12 @@ mod pools_tests {
         };
         let mut write_state = super::PoolMetaWriteState::default();
         let err = snapshot
-            .save_no_lock_armed(vec![storage.clone()], &mut write_state, guard.lock_lost_signal(), &[0])
+            .save_no_lock_armed(
+                vec![storage.clone()],
+                &mut write_state,
+                PoolMetaPersistenceFence::Owned(Arc::clone(&guard)),
+                &[0],
+            )
             .await
             .expect_err("a writer must not persist after losing the distributed pool metadata fence");
 
@@ -25824,6 +25990,7 @@ mod pools_tests {
             .await
             .expect("pool metadata fence acquisition should not error")
             .expect("the publishing writer should acquire the pool metadata fence");
+        let guard = Arc::new(guard);
         let storage = Arc::new(PartialPoolMetaWriteStorage {
             fail_write: false,
             fail_after_first_write: false,
@@ -25844,7 +26011,7 @@ mod pools_tests {
         current.pools[0].last_update = OffsetDateTime::UNIX_EPOCH;
         let mut write_state = super::PoolMetaWriteState::for_test_bootstrap();
         let outcome = saved
-            .save_no_lock_armed(vec![storage], &mut write_state, guard.lock_lost_signal(), &[0])
+            .save_no_lock_armed(vec![storage], &mut write_state, PoolMetaPersistenceFence::Owned(Arc::clone(&guard)), &[0])
             .await
             .expect("the replica save should finish while the fence is valid");
 

crates/ecstore/src/data_movement/scanner_backlog.rs

@@ -306,7 +306,7 @@ pub mod test_util {
         let Some(fault) = fault else { return Ok(None) };
         if fault
             .remaining
-            .fetch_update(Ordering::AcqRel, Ordering::Acquire, |remaining| remaining.checked_sub(1))
+            .try_update(Ordering::AcqRel, Ordering::Acquire, |remaining| remaining.checked_sub(1))
             != Ok(1)
         {
             return Ok(None);

crates/ecstore/src/data_usage/mod.rs

@@ -43,7 +43,7 @@ use std::{
     future::Future,
     sync::{
         Arc, LazyLock, OnceLock,
-        atomic::{AtomicU64, Ordering},
+        atomic::{AtomicU64, AtomicUsize, Ordering},
     },
     time::{Duration, SystemTime},
 };
@@ -67,12 +67,14 @@ struct CachedBucketUsage {
     usage_updated_at: SystemTime,
     // Set by request-path mutations until a scanner snapshot catches up to the same core counts.
     dirty: bool,
-    // Set when a newer scanner snapshot was observed but did not include the dirty counts yet.
+    // Set when an observed snapshot cannot replace this absolute baseline.
     stale_snapshot_pending: bool,
     // First complete scanner generation observed after an unknown-baseline
     // mutation. A strictly later generation is required before the mutation
     // evidence can be discarded.
     pending_scanner_position: Option<(u64, u64)>,
+    // Generation that supplied this absolute baseline, retained across mutations.
+    scanner_position: Option<(u64, u64)>,
     // Deletes are visible to admin immediately, but quota admission keeps
     // them pending until a complete scanner generation reconciles the set.
     // This marker intentionally remains process-local: the delete request
@@ -83,6 +85,79 @@ struct CachedBucketUsage {
     // necessarily a complete scanner reconciliation and therefore creates a
     // fresh cache entry with no pending hold.
     pending_negative_delta: u64,
+    pending_negative_updated_at: Option<SystemTime>,
+    pending_negative_reconciliation: Option<PendingDeleteReconciliation>,
+    in_flight_expirations: Arc<AtomicUsize>,
+}
+
+#[derive(Debug, Clone)]
+struct PendingDeleteReconciliation {
+    bytes: u64,
+    deleted_at: SystemTime,
+    observed: Option<((u64, u64), SystemTime)>,
+}
+
+impl CachedBucketUsage {
+    fn reconcile_pending_deletes(&mut self, position: Option<(u64, u64)>, updated_at: SystemTime) {
+        let Some(position) = position else { return };
+        let Some(pending) = self.pending_negative_reconciliation.as_mut() else { return };
+        match pending.observed {
+            None if updated_at >= pending.deleted_at => pending.observed = Some((position, updated_at)),
+            Some((previous, observed_at)) if position.0 >= previous.0 && position.1 > previous.1 && updated_at >= observed_at => {
+                self.pending_negative_delta = self.pending_negative_delta.saturating_sub(pending.bytes);
+                self.pending_negative_reconciliation = self
+                    .pending_negative_updated_at
+                    .filter(|_| self.pending_negative_delta > 0)
+                    .map(|deleted_at| PendingDeleteReconciliation {
+                        bytes: self.pending_negative_delta,
+                        deleted_at,
+                        observed: (updated_at >= deleted_at).then_some((position, updated_at)),
+                    });
+            }
+            _ => {}
+        }
+    }
+}
+
+/// Keeps a scanner refresh from including a delete before its memory receipt.
+pub(crate) struct ExpiryUsageAccounting<'a> {
+    bucket: &'a str,
+    in_flight: Arc<AtomicUsize>,
+}
+
+impl ExpiryUsageAccounting<'_> {
+    pub(crate) async fn commit(self, deleted_size: u64) {
+        let mut cache = memory_cache().write().await;
+        if let Some(entry) = cache.get_mut(self.bucket)
+            && Arc::ptr_eq(&entry.in_flight_expirations, &self.in_flight)
+        {
+            apply_bucket_object_delete_memory(entry, deleted_size, true);
+        }
+    }
+}
+
+impl Drop for ExpiryUsageAccounting<'_> {
+    fn drop(&mut self) {
+        self.in_flight.fetch_sub(1, Ordering::AcqRel);
+    }
+}
+
+pub(crate) async fn begin_expiry_usage_accounting(bucket: &str) -> ExpiryUsageAccounting<'_> {
+    ensure_bucket_usage_cached(bucket).await;
+    let mut cache = memory_cache().write().await;
+    // The caller owns the bucket name through commit. Avoid allocating either
+    // a receipt name or a map key for every expiry in an already cached bucket.
+    let in_flight = match cache.get(bucket) {
+        Some(entry) => Arc::clone(&entry.in_flight_expirations),
+        None => {
+            let entry = cached_bucket_usage_now(BucketUsageInfo::default());
+            let in_flight = Arc::clone(&entry.in_flight_expirations);
+            cache.insert(bucket.to_owned(), entry);
+            in_flight
+        }
+    };
+    in_flight.fetch_add(1, Ordering::AcqRel);
+    ExpiryUsageAccounting { bucket, in_flight }
 }
 
 type UsageMemoryCache = Arc<RwLock<HashMap<String, CachedBucketUsage>>>;
@@ -1944,7 +2019,11 @@ fn cached_bucket_usage_from_backend(usage: BucketUsageInfo, updated_at: SystemTi
         dirty: false,
         stale_snapshot_pending: false,
         pending_scanner_position: None,
+        scanner_position: None,
         pending_negative_delta: 0,
+        pending_negative_updated_at: None,
+        pending_negative_reconciliation: None,
+        in_flight_expirations: Arc::new(AtomicUsize::new(0)),
     }
 }
 
@@ -1958,7 +2037,11 @@ fn cached_bucket_usage_now(usage: BucketUsageInfo) -> CachedBucketUsage {
         dirty: false,
         stale_snapshot_pending: false,
         pending_scanner_position: None,
+        scanner_position: None,
         pending_negative_delta: 0,
+        pending_negative_updated_at: None,
+        pending_negative_reconciliation: None,
+        in_flight_expirations: Arc::new(AtomicUsize::new(0)),
     }
 }
 
@@ -1974,27 +2057,26 @@ fn bucket_usage_counts_match(left: &BucketUsageInfo, right: &BucketUsageInfo) ->
 }
 
 fn preserve_unknown_dirty_usage(
-    existing: &CachedBucketUsage,
+    existing: &mut CachedBucketUsage,
     snapshot_position: Option<(u64, u64)>,
     snapshot_update: SystemTime,
-) -> Option<CachedBucketUsage> {
+) -> bool {
     if existing.authoritative || !existing.dirty {
-        return None;
+        return false;
     }
     if existing
         .pending_scanner_position
         .zip(snapshot_position)
         .is_some_and(|(previous, current)| current.0 > previous.0 || (current.0 == previous.0 && current.1 > previous.1))
     {
-        return None;
+        return false;
     }
 
-    let mut preserved = existing.clone();
-    preserved.stale_snapshot_pending = true;
-    if preserved.pending_scanner_position.is_none() && snapshot_update >= existing.usage_updated_at {
-        preserved.pending_scanner_position = snapshot_position;
+    existing.stale_snapshot_pending = true;
+    if existing.pending_scanner_position.is_none() && snapshot_update >= existing.usage_updated_at {
+        existing.pending_scanner_position = snapshot_position;
     }
-    Some(preserved)
+    true
 }
 
 #[cfg(any(test, feature = "test-util"))]
@@ -2110,6 +2192,10 @@ pub async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64,
         .entry(bucket.to_string())
         .or_insert_with(|| cached_bucket_usage_now(BucketUsageInfo::default()));
 
+    apply_bucket_object_delete_memory(entry, deleted_size, removed_current_object);
+}
+
+fn apply_bucket_object_delete_memory(entry: &mut CachedBucketUsage, deleted_size: u64, removed_current_object: bool) {
     entry.usage.size = entry.usage.size.saturating_sub(deleted_size);
     entry.pending_negative_delta = entry.pending_negative_delta.saturating_add(deleted_size);
     if removed_current_object {
@@ -2118,6 +2204,16 @@ pub async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64,
     }
 
     let now = SystemTime::now();
+    entry.pending_negative_updated_at = Some(now);
+    // Freeze this tranche even before its first observation. Later deletes
+    // must not keep moving the timestamp that the scanner needs to catch up to.
+    if entry.pending_negative_reconciliation.is_none() && entry.pending_negative_delta > 0 {
+        entry.pending_negative_reconciliation = Some(PendingDeleteReconciliation {
+            bytes: entry.pending_negative_delta,
+            deleted_at: now,
+            observed: None,
+        });
+    }
     entry.refreshed_at = now;
     entry.usage_updated_at = now;
     entry.dirty = true;
@@ -2245,23 +2341,48 @@ async fn replace_bucket_usage_memory_from_info_if_generation(data_usage_info: &D
 
     let usage_updated_at = data_usage_info_updated_at(data_usage_info);
     let snapshot_position = data_usage_info.scanner_epoch.zip(data_usage_info.scanner_cycle);
+    let snapshot_converged = data_usage_info.usage_snapshot_converged == Some(true) && !data_usage_info.usage_snapshot_partial;
     let mut next_cache = HashMap::with_capacity(data_usage_info.buckets_usage.len());
     for (bucket, bucket_usage) in data_usage_info.buckets_usage.iter() {
-        next_cache.insert(
-            bucket.clone(),
-            cached_bucket_usage_from_backend(bucket_usage.clone(), usage_updated_at, true),
-        );
+        let mut entry = cached_bucket_usage_from_backend(bucket_usage.clone(), usage_updated_at, true);
+        entry.scanner_position = snapshot_position;
+        next_cache.insert(bucket.clone(), entry);
     }
 
     let mut cache = memory_cache().write().await;
     if expected_generation.is_some_and(|expected| usage_memory_generation() != expected) {
         return;
     }
-    for (bucket, existing) in cache.iter() {
-        match next_cache.entry(bucket.clone()) {
+    // Move preserved entries instead of cloning their histograms under the lock.
+    for (bucket, mut existing) in std::mem::take(&mut *cache) {
+        let advances_baseline = match (existing.scanner_position, snapshot_position) {
+            (Some(previous), Some(current)) if current.0 >= previous.0 && current.1 >= previous.1 => current.1 > previous.1,
+            (None, _) => true,
+            _ => {
+                existing.stale_snapshot_pending = true;
+                next_cache.insert(bucket, existing);
+                continue;
+            }
+        };
+        existing.reconcile_pending_deletes(snapshot_position, usage_updated_at);
+        if existing.in_flight_expirations.load(Ordering::Acquire) > 0 {
+            next_cache.insert(bucket, existing);
+            continue;
+        }
+        // Both convergence and the two-observation fallback must advance the
+        // baseline's cycle. Epoch-only changes cannot prove a new scan, and
+        // the timestamp check below still protects post-scan mutations.
+        let reconciled_dirty_usage = advances_baseline
+            && snapshot_position.is_some_and(|current| {
+                snapshot_converged
+                    || existing
+                        .pending_scanner_position
+                        .is_some_and(|previous| current.0 >= previous.0 && current.1 > previous.1)
+            });
+        match next_cache.entry(bucket) {
             Entry::Occupied(mut candidate) => {
-                if let Some(preserved) = preserve_unknown_dirty_usage(existing, snapshot_position, usage_updated_at) {
-                    candidate.insert(preserved);
+                if preserve_unknown_dirty_usage(&mut existing, snapshot_position, usage_updated_at) {
+                    candidate.insert(existing);
                     continue;
                 }
                 if existing.authoritative && existing.dirty && bucket_usage_counts_match(&existing.usage, &candidate.get().usage)
@@ -2272,37 +2393,44 @@ async fn replace_bucket_usage_memory_from_info_if_generation(data_usage_info: &D
                     continue;
                 }
                 if existing.authoritative && existing.usage_updated_at > usage_updated_at {
-                    candidate.insert(existing.clone());
+                    candidate.insert(existing);
                     continue;
                 }
-                if existing.authoritative && existing.dirty && !bucket_usage_counts_match(&existing.usage, &candidate.get().usage)
+                if existing.authoritative
+                    && existing.dirty
+                    && !reconciled_dirty_usage
+                    && !bucket_usage_counts_match(&existing.usage, &candidate.get().usage)
                 {
                     // A scanner snapshot can be saved after newer writes but still miss them if it listed the bucket earlier.
-                    let mut preserved = existing.clone();
+                    let mut preserved = existing;
                     if preserved.pending_negative_delta > 0 && preserved.usage.size == candidate.get().usage.size {
                         // The complete scanner snapshot has reconciled the
                         // post-delete byte total. Keep the request-path object
                         // counts until they also converge, but release the
                         // conservative quota hold.
                         preserved.pending_negative_delta = 0;
+                        preserved.pending_negative_reconciliation = None;
                     }
                     preserved.stale_snapshot_pending = true;
+                    preserved.pending_scanner_position = preserved.pending_scanner_position.or(snapshot_position);
                     candidate.insert(preserved);
                 }
             }
             Entry::Vacant(candidate) => {
-                if let Some(preserved) = preserve_unknown_dirty_usage(existing, snapshot_position, usage_updated_at) {
-                    candidate.insert(preserved);
-                } else if existing.authoritative && existing.usage_updated_at > usage_updated_at {
-                    candidate.insert(existing.clone());
-                } else if existing.authoritative && existing.dirty {
-                    let mut preserved = existing.clone();
+                if preserve_unknown_dirty_usage(&mut existing, snapshot_position, usage_updated_at)
+                    || (existing.authoritative && existing.usage_updated_at > usage_updated_at)
+                {
+                    candidate.insert(existing);
+                } else if existing.authoritative && existing.dirty && !reconciled_dirty_usage {
+                    let mut preserved = existing;
                     if preserved.pending_negative_delta > 0 && preserved.usage.size == 0 {
                         // Omitting the bucket from a complete snapshot confirms
                         // that its post-delete byte total is zero.
                         preserved.pending_negative_delta = 0;
+                        preserved.pending_negative_reconciliation = None;
                     }
                     preserved.stale_snapshot_pending = true;
+                    preserved.pending_scanner_position = preserved.pending_scanner_position.or(snapshot_position);
                     candidate.insert(preserved);
                 }
             }
@@ -4352,6 +4480,7 @@ mod tests {
         let mut first_snapshot = data_usage_info_for_test(bucket, 10, 420, mutation_update + Duration::from_nanos(1));
         first_snapshot.scanner_epoch = Some(7);
         first_snapshot.scanner_cycle = Some(10);
+        first_snapshot.usage_snapshot_converged = Some(true);
         replace_bucket_usage_memory_from_info(&first_snapshot).await;
 
         let mut first_response = first_snapshot.clone();
@@ -5405,11 +5534,12 @@ mod tests {
             .expect("delete usage should remain cached")
             .usage_updated_at;
 
-        let stale = data_usage_info_for_test("bucket-a", 2, 768, mutation_update + Duration::from_nanos(1));
+        // Windows SystemTime has 100 ns resolution; retain strict ordering there.
+        let stale = data_usage_info_for_test("bucket-a", 2, 768, mutation_update + Duration::from_micros(1));
         replace_bucket_usage_memory_from_info(&stale).await;
         assert_eq!(get_bucket_usage_memory("bucket-a").await, Some(768));
 
-        let mut too_old_matching = data_usage_info_for_test("bucket-a", 1, 256, mutation_update - Duration::from_nanos(1));
+        let mut too_old_matching = data_usage_info_for_test("bucket-a", 1, 256, mutation_update - Duration::from_micros(1));
         too_old_matching
             .buckets_usage
             .get_mut("bucket-a")
@@ -5423,7 +5553,7 @@ mod tests {
             "a matching byte total from before the delete cannot release the hold"
         );
 
-        let mut reconciled = data_usage_info_for_test("bucket-a", 1, 256, mutation_update + Duration::from_nanos(2));
+        let mut reconciled = data_usage_info_for_test("bucket-a", 1, 256, mutation_update + Duration::from_micros(2));
         reconciled
             .buckets_usage
             .get_mut("bucket-a")
@@ -5759,6 +5889,503 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn complete_scanner_cycles_reconcile_expired_dirty_usage() {
+        for (remaining_objects, omit_bucket) in [(0, false), (1, false), (0, true)] {
+            clear_usage_memory_cache_for_test().await;
+            let bucket = "expired-dirty-usage";
+            let baseline = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+
+            // Lifecycle deletion bypasses request-layer memory accounting.
+            // Repeated complete scans must eventually replace that dirty overlay.
+            for cycle in 1..=3 {
+                let mut snapshot = data_usage_info_for_test(bucket, remaining_objects, remaining_objects * 42, SystemTime::now());
+                snapshot.scanner_epoch = Some(7);
+                snapshot.scanner_cycle = Some(cycle);
+                if omit_bucket {
+                    snapshot.buckets_usage.clear();
+                    snapshot.bucket_sizes.clear();
+                    snapshot.buckets_count = 0;
+                    snapshot.calculate_totals();
+                }
+                replace_bucket_usage_memory_from_info(&snapshot).await;
+                if cycle == 3 {
+                    apply_bucket_usage_memory_overlay_if_authoritative(&mut snapshot, true).await;
+                    assert_eq!(snapshot.objects_total_count, remaining_objects);
+                    assert_eq!(snapshot.objects_total_size, remaining_objects * 42);
+                    if omit_bucket {
+                        assert!(!snapshot.buckets_usage.contains_key(bucket));
+                    } else {
+                        assert_eq!(snapshot.buckets_usage[bucket].objects_count, remaining_objects);
+                        assert_eq!(snapshot.buckets_usage[bucket].size, remaining_objects * 42);
+                    }
+                }
+            }
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expiry_receipts_exclude_refresh_until_committed_or_cancelled() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-receipt";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 2, 84, SystemTime::now())).await;
+        let first = begin_expiry_usage_accounting(bucket).await;
+        let cancelled = begin_expiry_usage_accounting(bucket).await;
+        let mut after_delete = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+        after_delete.scanner_epoch = Some(7);
+        after_delete.scanner_cycle = Some(10);
+        after_delete.usage_snapshot_converged = Some(true);
+        replace_bucket_usage_memory_from_info(&after_delete).await;
+        assert_eq!(memory_cache().read().await[bucket].usage.size, 84);
+        first.commit(42).await;
+        assert_eq!(memory_cache().read().await[bucket].usage.size, 42);
+
+        let mut omitted = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+        omitted.buckets_usage.clear();
+        omitted.bucket_sizes.clear();
+        omitted.buckets_count = 0;
+        omitted.calculate_totals();
+        replace_bucket_usage_memory_from_info(&omitted).await;
+        assert_eq!(memory_cache().read().await[bucket].usage.size, 42);
+        drop(cancelled);
+        assert_eq!(
+            memory_cache().read().await[bucket]
+                .in_flight_expirations
+                .load(Ordering::Acquire),
+            0
+        );
+
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 1, 42, SystemTime::now())).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn cancelled_expiry_receipt_reconciles_without_another_mutation() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-cancelled-after-delete";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 0, 0, SystemTime::now())).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let receipt = begin_expiry_usage_accounting(bucket).await;
+
+        // Storage has deleted one object, but the task has not committed its
+        // memory receipt. Even two complete scans must not replace its baseline.
+        let mut scanned = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+        scanned.scanner_epoch = Some(7);
+        for cycle in [10, 11] {
+            scanned.scanner_cycle = Some(cycle);
+            replace_bucket_usage_memory_from_info(&scanned).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+        }
+        drop(receipt);
+
+        for (cycle, expected) in [(12, 84), (13, 42)] {
+            scanned.scanner_cycle = Some(cycle);
+            replace_bucket_usage_memory_from_info(&scanned).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(expected));
+            let mut displayed = scanned.clone();
+            apply_bucket_usage_memory_overlay(&mut displayed).await;
+            assert_eq!(displayed.buckets_usage[bucket].size, expected);
+            assert_eq!(displayed.buckets_usage[bucket].objects_count, expected / 42);
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expiry_restart_reloads_durable_usage_without_persisting_the_overlay() {
+        use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
+
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        let bucket = "expiry-restart";
+        clear_usage_memory_cache_for_test().await;
+        let baseline = data_usage_info_for_test(bucket, 2, 84, SystemTime::now());
+        store_data_usage_in_backend(baseline.clone(), store.clone())
+            .await
+            .expect("persist the pre-expiration scanner snapshot");
+        replace_bucket_usage_memory_from_info(&baseline).await;
+        begin_expiry_usage_accounting(bucket).await.commit(42).await;
+        let mut displayed = baseline.clone();
+        apply_bucket_usage_memory_overlay(&mut displayed).await;
+        assert_eq!(displayed.buckets_usage[bucket].size, 42);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+
+        // Recreate only process-local usage state, then use the real backend
+        // loader. This is a restart-state regression, not a process restart E2E.
+        clear_usage_memory_cache_for_test().await;
+        invalidate_data_usage_snapshot_cache().await;
+        let recovered = load_data_usage_from_backend_cached(store.clone())
+            .await
+            .expect("reload the durable pre-expiration snapshot");
+        assert_eq!(recovered.buckets_usage[bucket].size, 84);
+        replace_bucket_usage_memory_from_info(&recovered).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+
+        let confirmed = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+        store_data_usage_in_backend(confirmed, store.clone())
+            .await
+            .expect("persist scanner confirmation without another object mutation");
+        let recovered = load_data_usage_from_backend_cached(store.clone())
+            .await
+            .expect("reload the confirmed snapshot");
+        replace_bucket_usage_memory_from_info(&recovered).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+
+        // Historical overlay drift must not contaminate the persisted scan.
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        clear_usage_memory_cache_for_test().await;
+        invalidate_data_usage_snapshot_cache().await;
+        let mut recovered = load_data_usage_from_backend_cached(store)
+            .await
+            .expect("reload the confirmed snapshot after losing the stale overlay");
+        replace_bucket_usage_memory_from_info(&recovered).await;
+        apply_bucket_usage_memory_overlay(&mut recovered).await;
+        assert_eq!(recovered.buckets_usage[bucket].objects_count, 1);
+        assert_eq!(recovered.buckets_usage[bucket].size, 42);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+        clear_usage_memory_cache_for_test().await;
+        invalidate_data_usage_snapshot_cache().await;
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expiry_receipt_cannot_credit_a_replaced_bucket_cache() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-recreated";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 1, 42, SystemTime::now())).await;
+        let receipt = begin_expiry_usage_accounting(bucket).await;
+        clear_usage_memory_cache_for_test().await;
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 2, 84, SystemTime::now())).await;
+        receipt.commit(42).await;
+        assert_eq!(memory_cache().read().await[bucket].usage.objects_count, 2);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn quota_delete_tranches_advance_despite_newer_mutations() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-busy-quota";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 10, 420, SystemTime::now())).await;
+        record_bucket_object_delete_memory(bucket, 42, true).await;
+        let mut observation = data_usage_info_for_test(bucket, 9, 378, SystemTime::now());
+        observation.scanner_epoch = Some(7);
+        observation.scanner_cycle = Some(10);
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        replace_bucket_usage_memory_from_info(&observation).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(462));
+
+        for (epoch, cycle, partial, old_time) in [
+            (8, 10, false, false),
+            (6, 11, false, false),
+            (7, 11, true, false),
+            (7, 11, false, true),
+        ] {
+            let mut rejected = observation.clone();
+            rejected.scanner_epoch = Some(epoch);
+            rejected.scanner_cycle = Some(cycle);
+            rejected.usage_snapshot_complete = !partial;
+            rejected.usage_snapshot_partial = partial;
+            if old_time {
+                rejected.last_update = Some(SystemTime::UNIX_EPOCH);
+            }
+            replace_bucket_usage_memory_from_info(&rejected).await;
+            assert_eq!(memory_cache().read().await[bucket].pending_negative_delta, 42);
+        }
+
+        observation.last_update = Some(SystemTime::now());
+        observation.scanner_cycle = Some(11);
+        // A newer delete must stay charged, without delaying the older receipt.
+        record_bucket_object_delete_memory(bucket, 42, true).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        replace_bucket_usage_memory_from_info(&observation).await;
+        assert_eq!(memory_cache().read().await[bucket].pending_negative_delta, 42);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(462));
+
+        for cycle in [12, 13] {
+            observation.last_update = Some(SystemTime::now());
+            observation.scanner_cycle = Some(cycle);
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            replace_bucket_usage_memory_from_info(&observation).await;
+        }
+        assert_eq!(memory_cache().read().await[bucket].pending_negative_delta, 0);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(504));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn quota_delete_hold_stays_bounded_with_deletes_after_every_observation() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-continuous-quota";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 10, 420, SystemTime::now())).await;
+        record_bucket_object_delete_memory(bucket, 42, true).await;
+        for cycle in 1..=12 {
+            let mut observed = data_usage_info_for_test(bucket, 9, 378, SystemTime::now());
+            observed.scanner_epoch = Some(7);
+            observed.scanner_cycle = Some(cycle);
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            record_bucket_object_delete_memory(bucket, 42, true).await;
+            replace_bucket_usage_memory_from_info(&observed).await;
+            let cache = memory_cache().read().await;
+            assert_eq!(cache[bucket].usage.size, 378);
+            assert!(
+                cache[bucket].pending_negative_delta <= 126,
+                "confirmed older deletes must leave the quota hold"
+            );
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn dirty_usage_reconciliation_requires_a_later_complete_observation() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "dirty-observation-fence";
+        let baseline = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+        replace_bucket_usage_memory_from_info(&baseline).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let mutation_at = memory_cache().read().await[bucket].usage_updated_at;
+        let mut first = data_usage_info_for_test(bucket, 0, 0, mutation_at);
+        first.scanner_epoch = Some(7);
+        first.scanner_cycle = Some(10);
+        let mut partial = first.clone();
+        partial.usage_snapshot_partial = true;
+        partial.usage_snapshot_complete = false;
+        replace_bucket_usage_memory_from_info(&partial).await;
+        assert_eq!(memory_cache().read().await[bucket].pending_scanner_position, None);
+        replace_bucket_usage_memory_from_info(&first).await;
+
+        for (epoch, cycle, partial, update) in [
+            (Some(7), Some(10), false, mutation_at),
+            (Some(8), Some(10), false, mutation_at),
+            (Some(6), Some(11), false, mutation_at),
+            (Some(7), Some(9), false, mutation_at),
+            (Some(7), Some(11), true, mutation_at),
+            (None, Some(11), false, mutation_at),
+            (Some(7), None, false, mutation_at),
+            // Windows `SystemTime` stores 100 ns intervals, so a 1 ns offset is a no-op there.
+            (Some(7), Some(11), false, mutation_at - Duration::from_micros(1)),
+        ] {
+            let mut snapshot = first.clone();
+            snapshot.scanner_epoch = epoch;
+            snapshot.scanner_cycle = cycle;
+            snapshot.usage_snapshot_partial = partial;
+            snapshot.usage_snapshot_complete = !partial;
+            snapshot.last_update = Some(update);
+            replace_bucket_usage_memory_from_info(&snapshot).await;
+            apply_bucket_usage_memory_overlay_if_authoritative(&mut snapshot, true).await;
+            assert_eq!(snapshot.objects_total_count, 1, "epoch={epoch:?}, cycle={cycle:?}, partial={partial}");
+            assert_eq!(snapshot.objects_total_size, 42);
+        }
+
+        let mut successor = first;
+        successor.scanner_epoch = Some(8);
+        successor.scanner_cycle = Some(11);
+        replace_bucket_usage_memory_from_info(&successor).await;
+        apply_bucket_usage_memory_overlay_if_authoritative(&mut successor, true).await;
+        assert_eq!(successor.objects_total_count, 0);
+        assert_eq!(successor.objects_total_size, 0);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn a_new_authoritative_mutation_restarts_reconciliation() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "dirty-mutation-fence";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 0, 0, SystemTime::now())).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let mut snapshot = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+        snapshot.scanner_epoch = Some(7);
+        snapshot.scanner_cycle = Some(10);
+        replace_bucket_usage_memory_from_info(&snapshot).await;
+
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        snapshot.last_update = Some(SystemTime::now());
+        snapshot.scanner_cycle = Some(11);
+        replace_bucket_usage_memory_from_info(&snapshot).await;
+        let mut response = snapshot.clone();
+        apply_bucket_usage_memory_overlay_if_authoritative(&mut response, true).await;
+        assert_eq!(response.objects_total_count, 2);
+        assert_eq!(response.objects_total_size, 84);
+
+        snapshot.scanner_cycle = Some(12);
+        replace_bucket_usage_memory_from_info(&snapshot).await;
+        apply_bucket_usage_memory_overlay_if_authoritative(&mut snapshot, true).await;
+        assert_eq!(snapshot.objects_total_count, 0);
+        assert_eq!(snapshot.objects_total_size, 0);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn converged_scans_repair_historical_usage_during_continuing_writes() {
+        let bucket = "historical-busy";
+        for position in [None, Some((7, 10))] {
+            clear_usage_memory_cache_for_test().await;
+            let mut baseline = data_usage_info_for_test(bucket, 436, 4_587_688_512, SystemTime::now());
+            baseline.scanner_epoch = position.map(|value| value.0);
+            baseline.scanner_cycle = position.map(|value| value.1);
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            for objects in 1..=6 {
+                record_bucket_object_write_memory(bucket, None, 42).await;
+                let mut scanned = data_usage_info_for_test(bucket, objects, objects * 42, SystemTime::now());
+                scanned.scanner_epoch = Some(7);
+                scanned.scanner_cycle = Some(10 + objects);
+                scanned.usage_snapshot_converged = Some(true);
+                replace_bucket_usage_memory_from_info(&scanned).await;
+                assert_eq!(get_bucket_usage_memory(bucket).await, Some(objects * 42));
+                apply_bucket_usage_memory_overlay_if_authoritative(&mut scanned, true).await;
+                assert_eq!(scanned.objects_total_count, objects);
+                assert_eq!(scanned.objects_total_size, objects * 42);
+            }
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn converged_usage_refresh_requires_a_new_valid_scanner_generation() {
+        let bucket = "converged-generation";
+        for (converged, epoch, cycle, partial) in [
+            (None, Some(7), Some(11), false),
+            (Some(false), Some(7), Some(11), false),
+            (Some(true), None, Some(11), false),
+            (Some(true), Some(7), None, false),
+            (Some(true), Some(7), Some(10), false),
+            (Some(true), Some(8), Some(10), false),
+            (Some(true), Some(6), Some(11), false),
+            (Some(true), Some(7), Some(9), false),
+            (Some(true), Some(7), Some(11), true),
+        ] {
+            clear_usage_memory_cache_for_test().await;
+            let mut baseline = data_usage_info_for_test(bucket, 436, 4_587_688_512, SystemTime::now());
+            baseline.scanner_epoch = Some(7);
+            baseline.scanner_cycle = Some(10);
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            let mut scanned = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+            scanned.usage_snapshot_converged = converged;
+            scanned.scanner_epoch = epoch;
+            scanned.scanner_cycle = cycle;
+            scanned.usage_snapshot_partial = partial;
+            replace_bucket_usage_memory_from_info(&scanned).await;
+            assert_eq!(
+                get_bucket_usage_memory(bucket).await,
+                Some(4_587_688_554),
+                "converged={converged:?}, epoch={epoch:?}, cycle={cycle:?}, partial={partial}"
+            );
+            scanned.usage_snapshot_converged = Some(true);
+            scanned.scanner_epoch = Some(8);
+            scanned.scanner_cycle = Some(11);
+            scanned.usage_snapshot_partial = false;
+            replace_bucket_usage_memory_from_info(&scanned).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn converged_usage_refresh_preserves_mutations_after_the_scan() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "converged-write-race";
+        let mut baseline = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+        baseline.scanner_epoch = Some(7);
+        baseline.scanner_cycle = Some(10);
+        replace_bucket_usage_memory_from_info(&baseline).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let mut scanned = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+        scanned.scanner_epoch = Some(7);
+        scanned.scanner_cycle = Some(11);
+        scanned.usage_snapshot_converged = Some(true);
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let after_scan = scanned.last_update.expect("scanner timestamp") + Duration::from_secs(1);
+        memory_cache()
+            .write()
+            .await
+            .get_mut(bucket)
+            .expect("dirty cache")
+            .usage_updated_at = after_scan;
+        replace_bucket_usage_memory_from_info(&scanned).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+        apply_bucket_usage_memory_overlay_if_authoritative(&mut scanned, true).await;
+        assert_eq!(scanned.objects_total_count, 2);
+        assert_eq!(scanned.objects_total_size, 84);
+        let mut successor = data_usage_info_for_test(bucket, 2, 84, after_scan);
+        successor.scanner_epoch = Some(7);
+        successor.scanner_cycle = Some(12);
+        successor.usage_snapshot_converged = Some(true);
+        replace_bucket_usage_memory_from_info(&successor).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn fallback_observations_cannot_bypass_the_cached_scanner_generation() {
+        let bucket = "historical-generation-floor";
+        for (converged, epoch, first_cycle) in [(None, 7, 10), (Some(true), 7, 10), (None, 6, 21), (Some(true), 6, 21)] {
+            clear_usage_memory_cache_for_test().await;
+            let mut baseline = data_usage_info_for_test(bucket, 436, 4_587_688_512, SystemTime::now());
+            baseline.scanner_epoch = Some(7);
+            baseline.scanner_cycle = Some(20);
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            // Wall clocks may step back; a timestamp alone cannot establish
+            // that an asynchronously loaded generation includes this write.
+            let update = memory_cache().read().await[bucket].usage_updated_at + Duration::from_secs(1);
+            for cycle in first_cycle..first_cycle + 3 {
+                let mut scanned = data_usage_info_for_test(bucket, 0, 0, update);
+                scanned.scanner_epoch = Some(epoch);
+                scanned.scanner_cycle = Some(cycle);
+                scanned.usage_snapshot_converged = converged;
+                replace_bucket_usage_memory_from_info(&scanned).await;
+                assert_eq!(
+                    get_bucket_usage_memory(bucket).await,
+                    Some(4_587_688_554),
+                    "older epoch/cycle cannot release quota: converged={converged:?}, epoch={epoch}, cycle={cycle}"
+                );
+                apply_bucket_usage_memory_overlay_if_authoritative(&mut scanned, true).await;
+                assert_eq!(scanned.objects_total_count, 437);
+                assert_eq!(scanned.objects_total_size, 4_587_688_554);
+            }
+            let mut successor = data_usage_info_for_test(bucket, 1, 42, update);
+            successor.scanner_epoch = Some(8);
+            successor.scanner_cycle = Some(21);
+            successor.usage_snapshot_converged = Some(true);
+            replace_bucket_usage_memory_from_info(&successor).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn matching_counts_cannot_discard_the_cached_scanner_generation() {
+        let bucket = "matching-generation-floor";
+        for position in [None, Some((7, 10))] {
+            clear_usage_memory_cache_for_test().await;
+            let mut baseline = data_usage_info_for_test(bucket, 2, 84, SystemTime::now());
+            baseline.scanner_epoch = Some(7);
+            baseline.scanner_cycle = Some(20);
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            let mut matching = baseline.clone();
+            matching.scanner_epoch = position.map(|value| value.0);
+            matching.scanner_cycle = position.map(|value| value.1);
+            replace_bucket_usage_memory_from_info(&matching).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            let update = memory_cache().read().await[bucket].usage_updated_at + Duration::from_secs(1);
+            let mut older = data_usage_info_for_test(bucket, 0, 0, update);
+            older.scanner_epoch = Some(7);
+            older.scanner_cycle = Some(12);
+            older.usage_snapshot_converged = Some(true);
+            replace_bucket_usage_memory_from_info(&older).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(126));
+            apply_bucket_usage_memory_overlay_if_authoritative(&mut older, true).await;
+            assert_eq!(older.objects_total_count, 3);
+            assert_eq!(older.objects_total_size, 126);
+        }
+    }
+
     #[tokio::test]
     #[serial]
     async fn authoritative_refresh_can_reduce_memory_usage() {

crates/ecstore/src/disk/local/commit.rs

@@ -84,46 +84,107 @@ async fn wait_for_heal_commit_test_barrier(root: &Path, bucket: &str, object: &s
     }
 }
 
-fn rollback_committed_rename_std(
-    dst_file_path: &Path,
-    new_data_path: Option<&Path>,
-    rollback_data_dir: Option<Uuid>,
-) -> std::io::Result<()> {
-    if let Some(old_data_dir) = rollback_data_dir {
-        let Some(dst_parent) = dst_file_path.parent() else {
-            return Err(std::io::Error::new(ErrorKind::InvalidInput, "missing object metadata parent"));
-        };
-        let backup_path = dst_parent.join(old_data_dir.to_string()).join(STORAGE_FORMAT_FILE_BACKUP);
-        std::fs::rename(backup_path, dst_file_path)?;
-    } else {
-        remove_file_if_exists(dst_file_path)?;
-    }
-
-    if let Some(new_data_path) = new_data_path {
-        remove_dir_all_if_exists(new_data_path)?;
-    }
-
-    Ok(())
+struct RenameCommitContext<'a> {
+    // Drop the directory identity guard before releasing mutation ownership.
+    directory_guard: os::RenameCommitGuard,
+    publication_root: &'a os::PublicationRoot,
+    mutation_lease: Arc<os::NamespaceMutationLease>,
+    source_metadata: &'a Path,
+    destination_metadata: &'a Path,
+    source_volume_dir: &'a Path,
+    data_paths: Option<&'a (PathBuf, PathBuf)>,
 }
 
-fn rollback_inline_metadata_commit_std(
-    dst_file_path: &Path,
-    rollback_data_dir: Option<Uuid>,
-    local_rollback_path: Option<&Path>,
-) -> std::io::Result<()> {
-    #[cfg(all(test, not(windows)))]
-    os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Rollback, dst_file_path);
-    if let Some(backup_path) = local_rollback_path {
-        // The commit immediately before this rollback renamed the staged
-        // xl.meta from the same directory as `backup_path` onto
-        // `dst_file_path`, proving both paths are on the same filesystem.
-        // Unix rename atomically replaces the committed destination; never
-        // unlink it first or an interrupted rollback could lose xl.meta.
-        std::fs::rename(backup_path, dst_file_path)?;
-    } else {
-        rollback_committed_rename_std(dst_file_path, None, rollback_data_dir)?;
+struct StagedRollbackBackup(PathBuf);
+
+impl StagedRollbackBackup {
+    fn path(&self) -> &Path {
+        &self.0
+    }
+}
+
+impl RenameCommitContext<'_> {
+    fn rollback_metadata_std(&self, rollback_data_dir: Option<Uuid>) -> std::io::Result<()> {
+        if let Some(old_data_dir) = rollback_data_dir {
+            let Some(dst_parent) = self.destination_metadata.parent() else {
+                return Err(std::io::Error::new(ErrorKind::InvalidInput, "missing object metadata parent"));
+            };
+            let backup_path = dst_parent.join(old_data_dir.to_string()).join(STORAGE_FORMAT_FILE_BACKUP);
+            std::fs::rename(backup_path, self.destination_metadata)?;
+        } else {
+            remove_file_if_exists(self.destination_metadata)?;
+        }
+        Ok(())
+    }
+
+    fn rollback_non_inline_std(&self, rollback_data_dir: Option<Uuid>) -> std::io::Result<()> {
+        self.rollback_metadata_std(rollback_data_dir)?;
+        if let Some((_, new_data_path)) = self.data_paths {
+            remove_dir_all_if_exists(new_data_path)?;
+        }
+        Ok(())
+    }
+
+    fn rollback_inline_std(
+        &self,
+        rollback_data_dir: Option<Uuid>,
+        local_backup: Option<&StagedRollbackBackup>,
+    ) -> std::io::Result<()> {
+        #[cfg(all(test, not(windows)))]
+        os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Rollback, self.destination_metadata);
+        if let Some(backup) = local_backup {
+            // The preceding commit used this staged directory on the same
+            // filesystem. Never unlink xl.meta before its atomic replacement.
+            std::fs::rename(backup.path(), self.destination_metadata)?;
+        } else {
+            self.rollback_metadata_std(rollback_data_dir)?;
+        }
+        Ok(())
+    }
+
+    async fn restore_published_data_source(&self) -> Result<()> {
+        let Some((src_data_path, dst_data_path)) = self.data_paths else {
+            return Ok(());
+        };
+        let src_volume_dir = self.source_volume_dir;
+        let publication_root = self.publication_root;
+        let mutation_lease = self.mutation_lease.clone();
+        if fs::symlink_metadata(src_data_path).await.is_ok() {
+            return Ok(());
+        }
+        let result =
+            match os::rename_all_with_lease(dst_data_path, src_data_path, src_volume_dir, publication_root, mutation_lease).await
+            {
+                Ok(()) => Ok(()),
+                Err(DiskError::FileNotFound) => {
+                    let source_exists = fs::symlink_metadata(src_data_path).await.is_ok();
+                    let destination_missing = matches!(
+                        fs::symlink_metadata(dst_data_path).await,
+                        Err(err) if err.kind() == ErrorKind::NotFound
+                    );
+                    if source_exists && destination_missing {
+                        Ok(())
+                    } else {
+                        Err(DiskError::FileNotFound)
+                    }
+                }
+                Err(err) => Err(err),
+            };
+        if let Err(err) = &result {
+            warn!(
+                target: "rustfs_ecstore::disk::local",
+                event = EVENT_DISK_LOCAL_RENAME_REJECTED,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
+                reason = "restore_staged_data_source_failed",
+                src_path = ?src_data_path,
+                dst_path = ?dst_data_path,
+                error = ?err,
+                "Failed to restore staged data after a metadata commit was rejected"
+            );
+        }
+        result
     }
-    Ok(())
 }
 
 #[cfg(any(not(windows), test))]
@@ -212,61 +273,6 @@ async fn read_rename_destination_metadata(
         .map_err(DiskError::from)
 }
 
-async fn restore_renamed_data_source(
-    src_volume_dir: &Path,
-    src_data_path: &Path,
-    dst_data_path: &Path,
-    publication_root: &os::PublicationRoot,
-    mutation_lease: Arc<os::NamespaceMutationLease>,
-) -> Result<()> {
-    if fs::symlink_metadata(src_data_path).await.is_ok() {
-        return Ok(());
-    }
-    let result =
-        match os::rename_all_with_lease(dst_data_path, src_data_path, src_volume_dir, publication_root, mutation_lease).await {
-            Ok(()) => Ok(()),
-            Err(DiskError::FileNotFound) => {
-                let source_exists = fs::symlink_metadata(src_data_path).await.is_ok();
-                let destination_missing = matches!(
-                    fs::symlink_metadata(dst_data_path).await,
-                    Err(err) if err.kind() == ErrorKind::NotFound
-                );
-                if source_exists && destination_missing {
-                    Ok(())
-                } else {
-                    Err(DiskError::FileNotFound)
-                }
-            }
-            Err(err) => Err(err),
-        };
-    if let Err(err) = &result {
-        warn!(
-            target: "rustfs_ecstore::disk::local",
-            event = EVENT_DISK_LOCAL_RENAME_REJECTED,
-            component = LOG_COMPONENT_ECSTORE,
-            subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
-            reason = "restore_staged_data_source_failed",
-            src_path = ?src_data_path,
-            dst_path = ?dst_data_path,
-            error = ?err,
-            "Failed to restore staged data after a metadata commit was rejected"
-        );
-    }
-    result
-}
-
-async fn restore_published_data_source(
-    data_paths: Option<&(PathBuf, PathBuf)>,
-    src_volume_dir: &Path,
-    publication_root: &os::PublicationRoot,
-    mutation_lease: Arc<os::NamespaceMutationLease>,
-) -> Result<()> {
-    let Some((src_data_path, dst_data_path)) = data_paths else {
-        return Ok(());
-    };
-    restore_renamed_data_source(src_volume_dir, src_data_path, dst_data_path, publication_root, mutation_lease).await
-}
-
 /// Proof produced only when the local rename returns at an existing access
 /// preflight, before metadata, backups, or object data can be published.
 #[derive(Debug)]
@@ -457,7 +463,18 @@ impl LocalDisk {
             mutation_lease.clone(),
         )
         .await?;
-        let has_dst_buf = read_rename_destination_metadata(&dst_file_path, &rename_commit_guard, mutation_lease.clone()).await?;
+        let mut commit = RenameCommitContext {
+            directory_guard: rename_commit_guard,
+            publication_root: &self.publication_root,
+            mutation_lease,
+            source_metadata: &src_file_path,
+            destination_metadata: &dst_file_path,
+            source_volume_dir: &src_volume_dir,
+            data_paths: has_data_dir_path.as_ref(),
+        };
+        let has_dst_buf =
+            read_rename_destination_metadata(commit.destination_metadata, &commit.directory_guard, commit.mutation_lease.clone())
+                .await?;
 
         if no_inline {
             // Non-inline: read xl.meta, parse, write, rename data dir, rename xl.meta
@@ -533,18 +550,18 @@ impl LocalDisk {
             // rename below to report through the existing rollback path. Payload
             // durability is kept by both strict and relaxed.
             let tmp_meta_write = {
-                let src_file_path = src_file_path.clone();
+                let src_file_path = commit.source_metadata.to_path_buf();
                 let dst_file_path = dst_file_path.clone();
-                let rename_commit_guard = rename_commit_guard.clone();
-                let mutation_lease = mutation_lease.clone();
+                let owned_directory_guard = commit.directory_guard.clone();
+                let owned_mutation_lease = commit.mutation_lease.clone();
                 async move {
-                    os::run_blocking_namespace_operation(mutation_lease, move || {
+                    os::run_blocking_namespace_operation(owned_mutation_lease, move || {
                         #[cfg(test)]
                         run_owned_file_write_before_open(&src_file_path);
                         let mut prepared_metadata_source = os::create_prepared_rename_source_with_commit_guard(
                             &src_file_path,
                             &dst_file_path,
-                            &rename_commit_guard,
+                            &owned_directory_guard,
                         )?;
                         prepared_metadata_source.write_all(&new_dst_buf, tmp_meta_sync != SyncMode::None)?;
                         Ok(prepared_metadata_source)
@@ -570,14 +587,14 @@ impl LocalDisk {
             // sequential version did.
             let prepared_metadata_source = tmp_meta_res?;
             shard_sync_res?;
-            let rename_commit_guard = remove_dst_base_before_commit(
+            commit.directory_guard = remove_dst_base_before_commit(
                 dst_path,
-                rename_commit_guard,
+                commit.directory_guard,
                 src_file_parent,
                 dst_file_parent,
                 &dst_volume_dir,
-                &self.publication_root,
-                mutation_lease.clone(),
+                commit.publication_root,
+                commit.mutation_lease.clone(),
             )
             .await?;
             if should_remove_staged_meta_before_commit(dst_path) {
@@ -595,7 +612,7 @@ impl LocalDisk {
             if fi_healing
                 && let Some((_, dst_data_path)) = has_data_dir_path.as_ref()
                 && let Err(err) = self
-                    .move_to_trash_with_namespace_owner(dst_data_path, true, false, Some(mutation_lease.clone()))
+                    .move_to_trash_with_namespace_owner(dst_data_path, true, false, Some(commit.mutation_lease.clone()))
                     .await
             {
                 warn!(
@@ -613,9 +630,9 @@ impl LocalDisk {
                     src_data_path,
                     dst_data_path,
                     &skip_parent,
-                    &self.publication_root,
-                    &rename_commit_guard,
-                    mutation_lease.clone(),
+                    commit.publication_root,
+                    &commit.directory_guard,
+                    commit.mutation_lease.clone(),
                 )
                 .await
             {
@@ -630,13 +647,7 @@ impl LocalDisk {
                     error = ?err,
                     "Disk local rename flow failed"
                 );
-                restore_published_data_source(
-                    has_data_dir_path.as_ref(),
-                    &src_volume_dir,
-                    &self.publication_root,
-                    mutation_lease.clone(),
-                )
-                .await?;
+                commit.restore_published_data_source().await?;
                 return Err(err);
             }
             #[cfg(test)]
@@ -661,13 +672,7 @@ impl LocalDisk {
                     reason = "test_fail_before_old_metadata_backup",
                     "Disk local rename flow failed before metadata commit"
                 );
-                restore_published_data_source(
-                    has_data_dir_path.as_ref(),
-                    &src_volume_dir,
-                    &self.publication_root,
-                    mutation_lease.clone(),
-                )
-                .await?;
+                commit.restore_published_data_source().await?;
                 return Err(DiskError::Unexpected);
             }
 
@@ -685,25 +690,16 @@ impl LocalDisk {
                 let backup_parent = dst_file_parent.join(old_data_dir.to_string());
                 #[cfg(not(windows))]
                 if let Err(err) = os::make_dir_all(&backup_parent, &skip_parent).await {
-                    restore_published_data_source(
-                        has_data_dir_path.as_ref(),
-                        &src_volume_dir,
-                        &self.publication_root,
-                        mutation_lease.clone(),
-                    )
-                    .await?;
+                    commit.restore_published_data_source().await?;
                     return Err(err);
                 }
-                let backup_path_guard = match rename_commit_guard.create_destination_directory_for_path_access(&backup_parent) {
+                let backup_path_guard = match commit
+                    .directory_guard
+                    .create_destination_directory_for_path_access(&backup_parent)
+                {
                     Ok(guard) => guard,
                     Err(err) => {
-                        restore_published_data_source(
-                            has_data_dir_path.as_ref(),
-                            &src_volume_dir,
-                            &self.publication_root,
-                            mutation_lease.clone(),
-                        )
-                        .await?;
+                        commit.restore_published_data_source().await?;
                         return Err(DiskError::from(to_file_error(err)));
                     }
                 };
@@ -711,13 +707,7 @@ impl LocalDisk {
                 if let Err(err) = check_path_length(backup_path.to_string_lossy().as_ref()) {
                     #[cfg(windows)]
                     drop(backup_path_guard);
-                    restore_published_data_source(
-                        has_data_dir_path.as_ref(),
-                        &src_volume_dir,
-                        &self.publication_root,
-                        mutation_lease.clone(),
-                    )
-                    .await?;
+                    commit.restore_published_data_source().await?;
                     return Err(err);
                 }
                 let backup_bytes = dst_buf.clone();
@@ -726,7 +716,7 @@ impl LocalDisk {
                 // sync finish. A detached spawn_blocking writer could survive
                 // cancellation and later truncate a newer transaction's
                 // deterministic rollback backup.
-                let write_result = os::run_blocking_namespace_operation(mutation_lease.clone(), move || {
+                let write_result = os::run_blocking_namespace_operation(commit.mutation_lease.clone(), move || {
                     #[cfg(test)]
                     run_owned_file_write_before_open(&backup_path);
                     backup_path_guard.write_file_for_path_access(
@@ -749,13 +739,7 @@ impl LocalDisk {
                         error = ?err,
                         "Disk local rename flow failed"
                     );
-                    restore_published_data_source(
-                        has_data_dir_path.as_ref(),
-                        &src_volume_dir,
-                        &self.publication_root,
-                        mutation_lease.clone(),
-                    )
-                    .await?;
+                    commit.restore_published_data_source().await?;
                     return Err(err);
                 }
             }
@@ -773,9 +757,9 @@ impl LocalDisk {
                 &src_file_path,
                 &dst_file_path,
                 &skip_parent,
-                &self.publication_root,
-                &rename_commit_guard,
-                mutation_lease.clone(),
+                commit.publication_root,
+                &commit.directory_guard,
+                commit.mutation_lease.clone(),
             )
             .await
             {
@@ -790,20 +774,11 @@ impl LocalDisk {
                     error = ?err,
                     "Disk local rename flow failed"
                 );
-                restore_published_data_source(
-                    has_data_dir_path.as_ref(),
-                    &src_volume_dir,
-                    &self.publication_root,
-                    mutation_lease.clone(),
-                )
-                .await?;
+                commit.restore_published_data_source().await?;
                 return Err(err);
             }
-
-            let committed_new_data_path = has_data_dir_path.as_ref().map(|(_, dst_data_path)| dst_data_path.as_path());
             if should_fail_after_metadata_commit(dst_path) {
-                rollback_committed_rename_std(&dst_file_path, committed_new_data_path, rollback_data_dir)
-                    .map_err(to_file_error)?;
+                commit.rollback_non_inline_std(rollback_data_dir).map_err(to_file_error)?;
                 return Err(DiskError::Unexpected);
             }
 
@@ -823,13 +798,12 @@ impl LocalDisk {
                 && let Some(parent) = dst_file_path.parent()
             {
                 let fsync_started = rustfs_io_metrics::put_stage_timer();
-                if let Err(err) = os::fsync_dst_dir_group_commit(parent, Some(mutation_lease.clone())).await {
+                if let Err(err) = os::fsync_dst_dir_group_commit(parent, Some(commit.mutation_lease.clone())).await {
                     rustfs_io_metrics::record_put_object_stage_duration_from(
                         rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DST_DIR_FSYNC,
                         fsync_started,
                     );
-                    rollback_committed_rename_std(&dst_file_path, committed_new_data_path, rollback_data_dir)
-                        .map_err(to_file_error)?;
+                    commit.rollback_non_inline_std(rollback_data_dir).map_err(to_file_error)?;
                     // The commit rename changed the dst part inodes before this fsync
                     // failed and rolled them back; drop any fd cached during that
                     // window so readers re-open the restored inode (rustfs/backlog#1177).
@@ -861,13 +835,12 @@ impl LocalDisk {
                         break;
                     }
                     let fsync_started = rustfs_io_metrics::put_stage_timer();
-                    if let Err(err) = os::fsync_dir_with_owner(dir, Some(mutation_lease.clone())).await {
+                    if let Err(err) = os::fsync_dir_with_owner(dir, Some(commit.mutation_lease.clone())).await {
                         rustfs_io_metrics::record_put_object_stage_duration_from(
                             rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_ANCESTOR_DIR_FSYNC,
                             fsync_started,
                         );
-                        rollback_committed_rename_std(&dst_file_path, committed_new_data_path, rollback_data_dir)
-                            .map_err(to_file_error)?;
+                        commit.rollback_non_inline_std(rollback_data_dir).map_err(to_file_error)?;
                         // Same post-commit rollback window as above — drop cached
                         // dst part fds so readers re-open the restored inode
                         // (rustfs/backlog#1177).
@@ -891,7 +864,7 @@ impl LocalDisk {
             // complete. Do not retain the Windows object identity guard while
             // cleaning staging paths or invalidating cached descriptors.
             #[cfg(windows)]
-            drop(rename_commit_guard);
+            drop(commit.directory_guard);
 
             if let Some(src_file_path_parent) = src_file_path.parent() {
                 if src_volume != super::super::RUSTFS_META_MULTIPART_BUCKET {
@@ -939,7 +912,7 @@ impl LocalDisk {
             let dst_path_for_failpoint = dst_path.to_string();
             #[cfg(windows)]
             let source_parent = src_file_parent.to_path_buf();
-            let rename_commit_guard_for_preparation = rename_commit_guard.clone();
+            let rename_commit_guard_for_preparation = commit.directory_guard.clone();
             let sync = durability.syncs_commit_metadata();
             #[cfg(test)]
             run_inline_before_file_sync_admission(dst_path);
@@ -1022,7 +995,7 @@ impl LocalDisk {
                     if sync {
                         std::fs::File::open(&backup_path)?.sync_data()?;
                     }
-                    staged_rollback_path = Some(backup_path);
+                    staged_rollback_path = Some(StagedRollbackBackup(backup_path));
                 }
 
                 Ok::<_, std::io::Error>((
@@ -1036,9 +1009,10 @@ impl LocalDisk {
                 ))
             };
             let inline_preparation = if let Some(admission) = file_sync_admission.as_ref() {
-                os::run_blocking_namespace_file_sync_operation(mutation_lease.clone(), admission, prepare_inline_metadata).await
+                os::run_blocking_namespace_file_sync_operation(commit.mutation_lease.clone(), admission, prepare_inline_metadata)
+                    .await
             } else {
-                os::run_blocking_namespace_operation(mutation_lease.clone(), prepare_inline_metadata).await
+                os::run_blocking_namespace_operation(commit.mutation_lease.clone(), prepare_inline_metadata).await
             }
             .map_err(to_file_error)
             .map_err(DiskError::from);
@@ -1061,28 +1035,30 @@ impl LocalDisk {
                 }
             };
 
-            let rename_commit_guard = remove_dst_base_before_commit(
+            commit.directory_guard = remove_dst_base_before_commit(
                 dst_path,
-                rename_commit_guard,
+                commit.directory_guard,
                 src_file_parent,
                 dst_file_parent,
                 &dst_volume_dir,
-                &self.publication_root,
-                mutation_lease.clone(),
+                commit.publication_root,
+                commit.mutation_lease.clone(),
             )
             .await?;
 
             if should_remove_staged_meta_before_commit(dst_path) {
                 drop(prepared_metadata_source);
                 let remove_result = std::fs::remove_file(&src_file_path);
-                if let Some(backup_path) = local_rollback_path.as_deref() {
+                if let Some(backup_path) = local_rollback_path.as_ref().map(StagedRollbackBackup::path) {
                     let _ = remove_file_if_exists(backup_path);
                 }
                 remove_result.map_err(to_file_error)?;
                 return Err(DiskError::FileNotFound);
             }
 
-            if let (Some(rollback_data_dir), Some(staged_backup)) = (rollback_data_dir, local_rollback_path.as_deref()) {
+            if let (Some(rollback_data_dir), Some(staged_backup)) =
+                (rollback_data_dir, local_rollback_path.as_ref().map(StagedRollbackBackup::path))
+            {
                 let Some(dst_parent) = dst_file_path.parent() else {
                     return Err(DiskError::other("missing object metadata parent"));
                 };
@@ -1096,8 +1072,8 @@ impl LocalDisk {
                     staged_backup,
                     &backup_path,
                     &dst_volume_dir,
-                    &self.publication_root,
-                    Some(mutation_lease.clone()),
+                    commit.publication_root,
+                    Some(commit.mutation_lease.clone()),
                 )
                 .await
                 {
@@ -1119,7 +1095,8 @@ impl LocalDisk {
                 {
                     let fsync_started = rustfs_io_metrics::put_stage_timer();
                     if let Err(err) =
-                        os::fsync_dir_with_namespace_file_sync_limit(backup_parent, mutation_lease.clone(), admission).await
+                        os::fsync_dir_with_namespace_file_sync_limit(backup_parent, commit.mutation_lease.clone(), admission)
+                            .await
                     {
                         rustfs_io_metrics::record_put_object_stage_duration_from(
                             rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_BACKUP_DIR_FSYNC,
@@ -1143,14 +1120,14 @@ impl LocalDisk {
                     &src_file_path,
                     &dst_file_path,
                     &dst_volume_dir,
-                    &self.publication_root,
-                    &rename_commit_guard,
-                    mutation_lease.clone(),
+                    commit.publication_root,
+                    &commit.directory_guard,
+                    commit.mutation_lease.clone(),
                 )
                 .await
             };
             if let Err(err) = commit_result {
-                if let Some(backup_path) = local_rollback_path.as_deref() {
+                if let Some(backup_path) = local_rollback_path.as_ref().map(StagedRollbackBackup::path) {
                     let _ = remove_file_if_exists(backup_path);
                 }
                 for part_path in &invalidate_part_paths {
@@ -1161,7 +1138,7 @@ impl LocalDisk {
 
             let post_commit = async {
                 if should_fail_after_metadata_commit(dst_path) {
-                    rollback_inline_metadata_commit_std(&dst_file_path, rollback_data_dir, local_rollback_path.as_deref())?;
+                    commit.rollback_inline_std(rollback_data_dir, local_rollback_path.as_ref())?;
                     return Err(std::io::Error::other("test fail after metadata commit"));
                 }
 
@@ -1170,15 +1147,18 @@ impl LocalDisk {
                     && let Some(dst_parent) = dst_file_path.parent()
                 {
                     let fsync_started = rustfs_io_metrics::put_stage_timer();
-                    if let Err(err) =
-                        os::fsync_dst_dir_group_commit_or_namespace_file_sync_limit(dst_parent, mutation_lease.clone(), admission)
-                            .await
+                    if let Err(err) = os::fsync_dst_dir_group_commit_or_namespace_file_sync_limit(
+                        dst_parent,
+                        commit.mutation_lease.clone(),
+                        admission,
+                    )
+                    .await
                     {
                         rustfs_io_metrics::record_put_object_stage_duration_from(
                             rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DST_DIR_FSYNC,
                             fsync_started,
                         );
-                        rollback_inline_metadata_commit_std(&dst_file_path, rollback_data_dir, local_rollback_path.as_deref())?;
+                        commit.rollback_inline_std(rollback_data_dir, local_rollback_path.as_ref())?;
                         return Err(err);
                     }
                     rustfs_io_metrics::record_put_object_stage_duration_from(
@@ -1204,17 +1184,14 @@ impl LocalDisk {
                         }
                         let fsync_started = rustfs_io_metrics::put_stage_timer();
                         if let Err(err) =
-                            os::fsync_dir_with_namespace_file_sync_limit(ancestor_dir, mutation_lease.clone(), admission).await
+                            os::fsync_dir_with_namespace_file_sync_limit(ancestor_dir, commit.mutation_lease.clone(), admission)
+                                .await
                         {
                             rustfs_io_metrics::record_put_object_stage_duration_from(
                                 rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_ANCESTOR_DIR_FSYNC,
                                 fsync_started,
                             );
-                            rollback_inline_metadata_commit_std(
-                                &dst_file_path,
-                                rollback_data_dir,
-                                local_rollback_path.as_deref(),
-                            )?;
+                            commit.rollback_inline_std(rollback_data_dir, local_rollback_path.as_ref())?;
                             return Err(err);
                         }
                         rustfs_io_metrics::record_put_object_stage_duration_from(
@@ -1251,9 +1228,9 @@ impl LocalDisk {
             // The commit no longer has a rollback path. Release the Windows
             // object identity guard before best-effort staging cleanup.
             #[cfg(windows)]
-            drop(rename_commit_guard);
+            drop(commit.directory_guard);
 
-            if let Some(backup_path) = local_rollback_path.as_deref() {
+            if let Some(backup_path) = local_rollback_path.as_ref().map(StagedRollbackBackup::path) {
                 let _ = remove_file_if_exists(backup_path);
             }
 

crates/ecstore/src/disk/uring_result_budget.rs

@@ -73,7 +73,7 @@ impl ProcessResultBudget {
         }
         inner
             .available
-            .fetch_update(Ordering::AcqRel, Ordering::Acquire, |available| available.checked_sub(bytes))
+            .try_update(Ordering::AcqRel, Ordering::Acquire, |available| available.checked_sub(bytes))
             .map(|_| {
                 Some(ResultBudgetReservation {
                     inner: Arc::clone(inner),

crates/ecstore/src/ecstore_validation_blackbox.rs

@@ -259,7 +259,7 @@ async fn blackbox_put_unknown_actual_size_restores_body_and_records_written_size
         .map(|idx| ((idx * 17) % 251) as u8)
         .collect::<Vec<_>>();
     let opts = ObjectOptions {
-        no_lock: true,
+        write_completion: crate::object_api::WriteCompletion::TailDrained,
         ..Default::default()
     };
 
@@ -300,7 +300,7 @@ async fn blackbox_get_restores_body_after_one_shard_file_is_removed() {
         .map(|idx| ((idx * 19) % 251) as u8)
         .collect::<Vec<_>>();
     let opts = ObjectOptions {
-        no_lock: true,
+        write_completion: crate::object_api::WriteCompletion::TailDrained,
         ..Default::default()
     };
 
@@ -406,8 +406,9 @@ async fn blackbox_heal_requests_preserve_repair_scope() {
                         put_object,
                         &mut put_reader,
                         &ObjectOptions {
-                            no_lock: true,
+                            write_completion: crate::object_api::WriteCompletion::TailDrained,
                             versioned: true,
+                            expected_bucket_incarnation_id: Some(put_incarnation),
                             ..Default::default()
                         },
                     )
@@ -468,7 +469,7 @@ async fn blackbox_heal_requests_preserve_repair_scope() {
             put_object,
             &mut healthy_reader,
             &ObjectOptions {
-                no_lock: true,
+                write_completion: crate::object_api::WriteCompletion::TailDrained,
                 ..Default::default()
             },
         )
@@ -508,7 +509,7 @@ async fn blackbox_heal_requests_preserve_repair_scope() {
             .map(|idx| ((idx * 29) % 251) as u8)
             .collect::<Vec<_>>();
         let opts = ObjectOptions {
-            no_lock: true,
+            write_completion: crate::object_api::WriteCompletion::TailDrained,
             ..Default::default()
         };
 
@@ -566,7 +567,7 @@ async fn blackbox_heal_requests_preserve_repair_scope() {
         let suspended_object = "suspended.bin";
         let payload = vec![0x5a; 1 << 20];
         let mpu_opts = ObjectOptions {
-            no_lock: true,
+            write_completion: crate::object_api::WriteCompletion::TailDrained,
             versioned: true,
             ..Default::default()
         };
@@ -674,7 +675,7 @@ async fn blackbox_heal_requests_preserve_repair_scope() {
                 .expect("fourth disk should be online before suspended completion")
         };
         let suspended_opts = ObjectOptions {
-            no_lock: true,
+            write_completion: crate::object_api::WriteCompletion::TailDrained,
             version_suspended: true,
             ..Default::default()
         };
@@ -711,7 +712,7 @@ async fn blackbox_range_read_restores_exact_slice_with_one_offline_disk() {
     let range_start = 513usize;
     let range_len = 8192usize;
     let opts = ObjectOptions {
-        no_lock: true,
+        write_completion: crate::object_api::WriteCompletion::TailDrained,
         ..Default::default()
     };
 
@@ -755,7 +756,7 @@ async fn blackbox_delete_marker_hides_object_body_without_erasing_prior_version_
     let bucket = "bb-delete-marker-read-negative";
     let object = "object.bin";
     let opts = ObjectOptions {
-        no_lock: true,
+        write_completion: crate::object_api::WriteCompletion::TailDrained,
         version_suspended: true,
         object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new(ObjectLockConfigState::ConfirmedAbsent))),
         ..Default::default()
@@ -852,7 +853,7 @@ async fn blackbox_issue3031_diag_covers_put_success_cleanup_and_error_summary()
         let first_payload = b"first diagnostic body".to_vec();
         let second_payload = b"second diagnostic body that replaces the first".to_vec();
         let opts = ObjectOptions {
-            no_lock: true,
+            write_completion: crate::object_api::WriteCompletion::TailDrained,
             ..Default::default()
         };
 
@@ -958,7 +959,7 @@ mod old_current_size_backfill {
             .await
             .expect("bucket should be created");
         let opts = ObjectOptions {
-            no_lock: true,
+            write_completion: crate::object_api::WriteCompletion::TailDrained,
             ..Default::default()
         };
 
@@ -992,7 +993,7 @@ mod old_current_size_backfill {
         // latest's size, not the incoming version's.
         let versioned = "versioned.bin";
         let first_version_opts = ObjectOptions {
-            no_lock: true,
+            write_completion: crate::object_api::WriteCompletion::TailDrained,
             versioned: true,
             version_id: Some(uuid::Uuid::new_v4().to_string()),
             ..Default::default()
@@ -1004,7 +1005,7 @@ mod old_current_size_backfill {
         let expected = prelookup_expectation(&set_disks, bucket, versioned).await;
         assert_eq!(expected, OldCurrentSize::Present(111));
         let second_version_opts = ObjectOptions {
-            no_lock: true,
+            write_completion: crate::object_api::WriteCompletion::TailDrained,
             versioned: true,
             version_id: Some(uuid::Uuid::new_v4().to_string()),
             ..Default::default()
@@ -1036,7 +1037,7 @@ mod old_current_size_backfill {
         let expected = prelookup_expectation(&set_disks, bucket, versioned).await;
         assert_eq!(expected, OldCurrentSize::Present(0));
         let third_version_opts = ObjectOptions {
-            no_lock: true,
+            write_completion: crate::object_api::WriteCompletion::TailDrained,
             versioned: true,
             version_id: Some(uuid::Uuid::new_v4().to_string()),
             ..Default::default()
@@ -1060,7 +1061,7 @@ mod old_current_size_backfill {
             .await
             .expect("bucket should be created");
         let opts = ObjectOptions {
-            no_lock: true,
+            write_completion: crate::object_api::WriteCompletion::TailDrained,
             ..Default::default()
         };
 

crates/ecstore/src/erasure/coding/encode.rs

@@ -43,7 +43,7 @@ const ENV_RUSTFS_ERASURE_ENCODE_BYTESMUT_INGEST: &str = "RUSTFS_ERASURE_ENCODE_B
 const DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES: usize = 32 * 1024 * 1024;
 const DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BLOCKS: usize = 32;
 const DEFAULT_RUSTFS_ERASURE_ENCODE_BATCH_BLOCKS: usize = 4;
-const DEFAULT_RUSTFS_ERASURE_ENCODE_BYTESMUT_INGEST: bool = false;
+const DEFAULT_RUSTFS_ERASURE_ENCODE_BYTESMUT_INGEST: bool = true;
 
 pub(crate) enum IntegrityEncodeMode {
     Inline(usize),
@@ -890,8 +890,6 @@ impl Erasure {
                 // capacity and never reallocates. Reading into uninitialized spare capacity
                 // (instead of resize + slice read) also skips zero-filling each fresh buffer.
                 let ingest_capacity = expanded_block_bytes.max(block_size);
-                // Pre-allocate buffer pool for this encoding session
-                let mut buf_pool: Vec<BytesMut> = Vec::with_capacity(4);
                 let mut buf = BytesMut::with_capacity(ingest_capacity);
                 loop {
                     match read_full_buf_or_eof(&mut reader, &mut buf, block_size).await {
@@ -901,8 +899,6 @@ impl Erasure {
                             total += n;
                             let encode_buf = buf;
                             let res = self.clone().encode_block_bytes_mut(encode_buf, n).await?;
-                            // Try to reuse buffer from pool, or allocate new one
-                            buf = buf_pool.pop().unwrap_or_else(|| BytesMut::with_capacity(ingest_capacity));
                             let queued_bytes = res.queued_bytes();
                             let _producer_stage = rustfs_io_metrics::track_ec_encode_producer_bytes(queued_bytes);
                             let send_wait_stage_start = stage_timer_if_enabled();
@@ -910,11 +906,8 @@ impl Erasure {
                                 return Err(std::io::Error::other(format!("Failed to send encoded data : {err}")));
                             }
                             record_internal_stage_if_enabled("erasure_encode_send_wait", send_wait_stage_start);
-                            // Return buffer to pool if it has sufficient capacity
-                            if buf.capacity() >= ingest_capacity && buf_pool.len() < 4 {
-                                buf_pool.push(buf);
-                                buf = BytesMut::with_capacity(ingest_capacity);
-                            }
+                            // Encoded shards own the previous allocation until writing completes.
+                            buf = BytesMut::with_capacity(ingest_capacity);
                         }
                         Ok(None) => break,
                         Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => {
@@ -1054,18 +1047,18 @@ impl Erasure {
         let mut task = AbortOnDropTask::new(tokio::spawn(async move {
             let block_size = self.block_size;
             let mut total = 0;
-            let mut buf = vec![0u8; block_size];
+            let ingest_capacity = expanded_block_bytes.max(block_size);
+            let mut buf = BytesMut::with_capacity(ingest_capacity);
             let mut pending_batch = Vec::with_capacity(batch_blocks);
             let mut pending_batch_bytes = 0usize;
             let mut pending_batch_stage = None;
             loop {
-                match rustfs_utils::read_full_or_eof(&mut reader, &mut buf).await {
+                match read_full_buf_or_eof(&mut reader, &mut buf, block_size).await {
                     Ok(Some(n)) => {
                         debug_assert!(n > 0, "non-zero block_size prevents zero-length reads");
                         total += n;
-                        let encode_buf = std::mem::take(&mut buf);
-                        let (res, returned_buf) = self.clone().encode_block(encode_buf, n).await?;
-                        buf = returned_buf;
+                        let res = self.clone().encode_block_bytes_mut(buf, n).await?;
+                        buf = BytesMut::with_capacity(ingest_capacity);
                         let queued_bytes = res.queued_bytes();
                         pending_batch_bytes = pending_batch_bytes.saturating_add(queued_bytes);
                         pending_batch.push(res);
@@ -1242,6 +1235,42 @@ mod tests {
     use tokio::io::{AsyncWrite, AsyncWriteExt, ReadBuf};
     use tokio::sync::oneshot;
 
+    #[test]
+    fn bytesmut_ingest_selector_defaults_to_owned_and_honors_overrides() {
+        const CHILD_CASE: &str = "RUSTFS_TEST_BYTESMUT_INGEST_SELECTOR_CASE";
+        if let Ok(case) = std::env::var(CHILD_CASE) {
+            let expected = match case.as_str() {
+                "default" | "true" => true,
+                "false" => false,
+                _ => panic!("unexpected ingest selector case: {case}"),
+            };
+            assert_eq!(use_bytesmut_ingest(), expected, "ingest selector case: {case}");
+            return;
+        }
+
+        // Each selector probe needs a fresh OnceLock and an isolated environment.
+        for case in ["default", "true", "false"] {
+            let mut child = std::process::Command::new(std::env::current_exe().expect("ingest selector test executable"));
+            child.args([
+                "--exact",
+                "erasure::coding::encode::tests::bytesmut_ingest_selector_defaults_to_owned_and_honors_overrides",
+                "--nocapture",
+            ]);
+            child.env(CHILD_CASE, case);
+            child.env_remove(ENV_RUSTFS_ERASURE_ENCODE_BYTESMUT_INGEST);
+            if case != "default" {
+                child.env(ENV_RUSTFS_ERASURE_ENCODE_BYTESMUT_INGEST, case);
+            }
+            let output = child.output().expect("start isolated ingest selector test");
+            assert!(
+                output.status.success(),
+                "ingest selector case {case} failed:\n{}\n{}",
+                String::from_utf8_lossy(&output.stdout),
+                String::from_utf8_lossy(&output.stderr),
+            );
+        }
+    }
+
     struct PendingReader {
         entered: Option<oneshot::Sender<()>>,
         dropped: Option<oneshot::Sender<()>>,

crates/ecstore/src/layout/endpoints.rs

@@ -908,7 +908,7 @@ fn explicit_url_port(raw: &str) -> Result<Option<u16>> {
         return Err(invalid());
     }
 
-    let (scheme, remainder) = raw.split_once("://").ok_or_else(&invalid)?;
+    let (scheme, remainder) = raw.split_once("://").ok_or_else(invalid)?;
     if !scheme.eq_ignore_ascii_case("http") && !scheme.eq_ignore_ascii_case("https") {
         return Err(invalid());
     }
@@ -916,17 +916,17 @@ fn explicit_url_port(raw: &str) -> Result<Option<u16>> {
         .split('/')
         .next()
         .filter(|authority| !authority.is_empty())
-        .ok_or_else(&invalid)?;
+        .ok_or_else(invalid)?;
     if authority.contains('@') {
         return Err(invalid());
     }
 
     let port = if let Some(bracketed) = authority.strip_prefix('[') {
-        let (_, suffix) = bracketed.split_once(']').ok_or_else(&invalid)?;
+        let (_, suffix) = bracketed.split_once(']').ok_or_else(invalid)?;
         if suffix.is_empty() {
             return Ok(None);
         }
-        suffix.strip_prefix(':').ok_or_else(&invalid)?
+        suffix.strip_prefix(':').ok_or_else(invalid)?
     } else if let Some((_, port)) = authority.rsplit_once(':') {
         port
     } else {
@@ -1001,11 +1001,7 @@ fn parse_explicit_local_endpoint_host(raw: &str) -> Result<Host<String>> {
 
     let host = Host::parse(raw).map_err(|_| invalid())?;
     let host = match host {
-        Host::Domain(domain) => Host::Domain(
-            domain_without_optional_trailing_dot(&domain)
-                .ok_or_else(&invalid)?
-                .to_string(),
-        ),
+        Host::Domain(domain) => Host::Domain(domain_without_optional_trailing_dot(&domain).ok_or_else(invalid)?.to_string()),
         host => host,
     };
     if matches!(&host, Host::Domain(domain) if domain.contains('*'))

crates/ecstore/src/object_api/mod.rs

@@ -104,6 +104,7 @@ mod hook_slot;
 mod object_mutation_hook;
 mod readers;
 mod types;
+pub(crate) mod write_commit_context;
 
 #[cfg(test)]
 mod persisted_metadata_keys_tests;
@@ -125,3 +126,4 @@ pub(crate) use object_mutation_hook::notify_object_mutation;
 pub use object_mutation_hook::{ObjectMutationHook, register_object_mutation_hook, unregister_object_mutation_hook};
 pub use readers::*;
 pub use types::*;
+pub use write_commit_context::{WriteCommitGuard, WriteLockContext};

crates/ecstore/src/object_api/types.rs

@@ -935,6 +935,9 @@ pub struct ObjectOptions {
     /// Persisted bucket incarnation observed before authorization.
     pub expected_bucket_incarnation_id: Option<Uuid>,
     pub no_lock: bool,
+    /// A genuine target write guard; loss fences alone do not authorize a borrowed write.
+    #[doc(hidden)]
+    pub write_lock_context: Option<WriteLockContext>,
     /// Internal read-only inspection must not enqueue metadata or payload repairs.
     pub suppress_read_repair: bool,
     /// Control-plane writers that immediately read or CAS the same namespace

crates/ecstore/src/object_api/write_commit_context.rs

@@ -0,0 +1,186 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use super::{ObjectOptions, WriteCompletion};
+use crate::error::{Error, Result};
+use std::sync::Arc;
+use std::time::Duration;
+
+/// A real write guard shared with an inner commit, never a lock-loss signal alone.
+///
+/// Callers cannot fabricate a guard without acquiring its namespace lock.
+///
+/// ```compile_fail
+/// use rustfs_ecstore::api::object::WriteCommitGuard;
+/// use std::sync::Arc;
+///
+/// let guard = WriteCommitGuard {
+///     bucket: "bucket".into(),
+///     object: "object".into(),
+///     guards: Arc::new(Vec::new()),
+/// };
+/// ```
+#[doc(hidden)]
+#[derive(Clone, Debug)]
+pub struct WriteCommitGuard {
+    bucket: Arc<str>,
+    object: Arc<str>,
+    guards: Arc<Vec<Arc<rustfs_lock::NamespaceLockGuard>>>,
+}
+
+impl WriteCommitGuard {
+    /// Acquire the key recorded by the namespace wrapper and retain its owner.
+    pub async fn acquire(
+        lock: &rustfs_lock::NamespaceLockWrapper,
+        timeout: Duration,
+    ) -> std::result::Result<Self, rustfs_lock::LockError> {
+        let guard = Arc::new(lock.get_write_lock(timeout).await?);
+        let key = lock.resource();
+        Ok(Self::from_guard(guard, &key.bucket, &key.object))
+    }
+
+    pub(crate) fn from_guard(guard: Arc<rustfs_lock::NamespaceLockGuard>, bucket: &str, object: &str) -> Self {
+        Self {
+            bucket: bucket.into(),
+            object: object.into(),
+            guards: Arc::new(vec![guard]),
+        }
+    }
+
+    pub fn is_lock_lost(&self) -> bool {
+        self.guards.iter().any(|guard| guard.is_lock_lost() || guard.is_released())
+    }
+
+    fn matches(&self, bucket: &str, object: &str) -> bool {
+        self.bucket.as_ref() == bucket && self.object.as_ref() == object
+    }
+}
+
+/// Opaque boundary capability; only real acquisitions or offline startup construct it.
+///
+/// ```compile_fail
+/// use rustfs_ecstore::object_api::WriteLockContext;
+///
+/// let context = WriteLockContext {};
+/// ```
+#[doc(hidden)]
+#[derive(Clone, Debug)]
+pub struct WriteLockContext {
+    mode: WriteLockMode,
+}
+
+#[derive(Clone, Debug)]
+enum WriteLockMode {
+    Borrowed(Vec<WriteCommitGuard>),
+    OfflineRecovery,
+}
+
+pub(crate) enum WriteCommitContext {
+    Owned,
+    Borrowed(Vec<WriteCommitGuard>),
+    Publication,
+    OfflineRecovery,
+    #[cfg(test)]
+    UnlockedFixture,
+}
+
+impl WriteCommitContext {
+    pub(crate) fn from_options(opts: &ObjectOptions, bucket: &str, object: &str, publication: bool) -> Result<Self> {
+        if publication {
+            if let Some(WriteLockContext {
+                mode: WriteLockMode::Borrowed(guards),
+            }) = opts.write_lock_context.as_ref()
+                && !guards.iter().any(|guard| guard.matches(bucket, object))
+            {
+                return Err(Error::InvalidArgument(
+                    bucket.to_owned(),
+                    object.to_owned(),
+                    "publication carries a different borrowed namespace".to_owned(),
+                ));
+            }
+            return Ok(Self::Publication);
+        }
+        match (opts.no_lock, opts.write_lock_context.as_ref().map(|context| &context.mode)) {
+            (false, None | Some(WriteLockMode::Borrowed(_))) => Ok(Self::Owned),
+            (true, Some(WriteLockMode::Borrowed(guards))) if guards.iter().any(|guard| guard.matches(bucket, object)) => {
+                Ok(Self::Borrowed(guards.clone()))
+            }
+            (true, Some(WriteLockMode::OfflineRecovery)) => Ok(Self::OfflineRecovery),
+            #[cfg(test)]
+            (true, None) if !opts.metadata_cache_safe => Ok(Self::UnlockedFixture),
+            _ => Err(Error::InvalidArgument(
+                bucket.to_owned(),
+                object.to_owned(),
+                "write lock context does not prove the target namespace".to_owned(),
+            )),
+        }
+    }
+
+    pub(crate) fn acquires_namespace(&self) -> bool {
+        matches!(self, Self::Owned)
+    }
+
+    pub(crate) fn has_borrowed_owner(&self) -> bool {
+        matches!(self, Self::Borrowed(_))
+    }
+
+    pub(crate) fn is_lock_lost(&self) -> bool {
+        matches!(self, Self::Borrowed(guards) if guards.iter().any(WriteCommitGuard::is_lock_lost))
+    }
+
+    pub(crate) fn allows_early_ack(&self, completion: WriteCompletion) -> bool {
+        completion == WriteCompletion::Quorum && matches!(self, Self::Owned | Self::Borrowed(_) | Self::Publication)
+    }
+}
+
+impl ObjectOptions {
+    pub fn add_write_commit_guard(&mut self, guard: &WriteCommitGuard) {
+        for owner in guard.guards.iter() {
+            self.add_namespace_lock_guard(owner);
+        }
+        let mut guards = match self.write_lock_context.as_ref() {
+            Some(WriteLockContext {
+                mode: WriteLockMode::Borrowed(existing),
+            }) => existing.clone(),
+            _ => Vec::new(),
+        };
+        if let Some(existing) = guards.iter_mut().find(|held| held.matches(&guard.bucket, &guard.object)) {
+            let owners = Arc::make_mut(&mut existing.guards);
+            for owner in guard.guards.iter() {
+                if !owners.iter().any(|held| Arc::ptr_eq(held, owner)) {
+                    owners.push(Arc::clone(owner));
+                }
+            }
+        } else {
+            guards.push(guard.clone());
+        }
+        self.no_lock = true;
+        self.write_lock_context = Some(WriteLockContext {
+            mode: WriteLockMode::Borrowed(guards),
+        });
+    }
+
+    pub(crate) fn add_owned_write_lock(&mut self, guard: Arc<rustfs_lock::NamespaceLockGuard>, bucket: &str, object: &str) {
+        self.add_write_commit_guard(&WriteCommitGuard::from_guard(guard, bucket, object));
+    }
+
+    /// Only startup migration may publish without a running namespace owner.
+    pub(crate) fn use_offline_recovery_write(&mut self) {
+        self.no_lock = true;
+        self.write_lock_context = Some(WriteLockContext {
+            mode: WriteLockMode::OfflineRecovery,
+        });
+        self.write_completion = WriteCompletion::TailDrained;
+    }
+}

crates/ecstore/src/runtime/instance.rs

@@ -396,7 +396,7 @@ impl InstanceContext {
     pub(crate) fn begin_namespace_commit(self: &Arc<Self>) -> Arc<NamespaceCommitGuard> {
         let counted = self
             .namespace_commits
-            .fetch_update(Ordering::AcqRel, Ordering::Acquire, |count| count.checked_add(1))
+            .try_update(Ordering::AcqRel, Ordering::Acquire, |count| count.checked_add(1))
             .is_ok();
         if counted {
             self.advance_namespace_commit_generation();
@@ -412,7 +412,7 @@ impl InstanceContext {
     fn advance_namespace_commit_generation(&self) {
         let _ = self
             .namespace_commit_generation
-            .fetch_update(Ordering::AcqRel, Ordering::Acquire, |generation| Some(generation.saturating_add(1)));
+            .try_update(Ordering::AcqRel, Ordering::Acquire, |generation| Some(generation.saturating_add(1)));
     }
 
     pub(crate) fn namespace_commit_generation(&self) -> u64 {

crates/ecstore/src/services/rebalance/control.rs

@@ -521,11 +521,12 @@ impl ECStore {
                 .map_err(|err| rebalance_meta_lock_error(err, "write"))
         })
         .await?;
+        let guard = Arc::new(guard);
         let mut opts = ObjectOptions {
-            no_lock: true,
+            write_completion: crate::object_api::WriteCompletion::TailDrained,
             ..Default::default()
         };
-        opts.add_namespace_lock_guard(&guard);
+        opts.add_owned_write_lock(Arc::clone(&guard), crate::disk::RUSTFS_META_BUCKET, REBAL_META_NAME);
 
         merge_and_save_rebalance_meta_no_lock(pool, local_snapshot, stage, opts, None, expected_id).await?;
         if guard.is_lock_lost() {

crates/ecstore/src/set_disk/core/metadata_quorum.rs

@@ -29,6 +29,27 @@ use crate::disk::error_reduce::OBJECT_OP_IGNORED_ERRS;
 use crate::set_disk::file_info_is_valid_for_metadata;
 use rustfs_filemeta::FileInfo;
 
+/// One disk's observation. Pending is neither a successful vote nor an offline
+/// disk: the scheduler may not have issued this slot or may still be waiting.
+// Option<Result<...>> keeps FileInfo inline without a per-response Box.
+// None is pending; Some(Ok(_)) is success; Some(Err(_)) retains a disk error.
+pub(in crate::set_disk) type MetadataDiskResult = Option<crate::disk::error::Result<FileInfo>>;
+
+#[derive(Debug)]
+pub(in crate::set_disk) struct MetadataDiskObservation {
+    pub(in crate::set_disk) disk_index: usize,
+    pub(in crate::set_disk) result: MetadataDiskResult,
+}
+
+impl MetadataDiskObservation {
+    pub(in crate::set_disk) fn file_info(&self) -> Option<&FileInfo> {
+        match &self.result {
+            Some(Ok(metadata)) => Some(metadata),
+            None | Some(Err(_)) => None,
+        }
+    }
+}
+
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
 pub(in crate::set_disk) struct MetadataEarlyStopDecision {
     pub(in crate::set_disk) reason: &'static str,
@@ -59,6 +80,14 @@ pub(in crate::set_disk) struct MetadataQuorumAccumulator {
 }
 
 impl MetadataQuorumAccumulator {
+    pub(in crate::set_disk) fn observe(&mut self, observation: &MetadataDiskObservation) {
+        match &observation.result {
+            None => {}
+            Some(Ok(metadata)) => self.observe_file_info_at(observation.disk_index, metadata),
+            Some(Err(error)) => self.observe_error(error),
+        }
+    }
+
     pub(in crate::set_disk) fn new(total_disks: usize, default_parity_count: usize, allow_early_stop: bool) -> Self {
         Self {
             total_disks,
@@ -86,6 +115,7 @@ impl MetadataQuorumAccumulator {
         self
     }
 
+    #[cfg(test)]
     pub(in crate::set_disk) fn observe_file_info(&mut self, file_info: &FileInfo) {
         self.observe_file_info_with_index(None, file_info);
     }
@@ -383,3 +413,186 @@ pub(in crate::set_disk) fn metadata_early_stop_candidate_matches(left: &FileInfo
 pub(in crate::set_disk) fn is_metadata_fanout_ignored_error(err: &DiskError) -> bool {
     OBJECT_OP_IGNORED_ERRS.iter().any(|ignored| ignored == err)
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::set_disk::SetDisks;
+    use time::OffsetDateTime;
+    use uuid::Uuid;
+
+    fn payload(disk_index: usize) -> FileInfo {
+        let mut metadata = FileInfo::new("object", 2, 2);
+        metadata.volume = "bucket".to_string();
+        metadata.name = "object".to_string();
+        metadata.size = 1;
+        metadata.mod_time = Some(OffsetDateTime::UNIX_EPOCH);
+        metadata.data_dir = Some(Uuid::from_u128(1));
+        metadata.erasure.index = metadata.erasure.distribution[disk_index];
+        metadata.metadata.insert("etag".to_string(), "etag".to_string());
+        metadata.add_object_part(1, "etag".to_string(), 1, None, 1, None, None);
+        assert!(file_info_is_valid_for_metadata(&metadata), "the observation corpus needs a valid payload");
+        metadata
+    }
+
+    fn observation(disk_index: usize, state: usize) -> MetadataDiskObservation {
+        MetadataDiskObservation {
+            disk_index,
+            result: match state {
+                0 => Some(Ok(payload(disk_index))),
+                1 => Some(Err(DiskError::FileNotFound)),
+                2 => Some(Err(DiskError::FileCorrupt)),
+                3 => Some(Err(DiskError::DiskNotFound)),
+                4 => None,
+                _ => panic!("unexpected test state"),
+            },
+        }
+    }
+
+    fn assert_same_reduction(typed: &MetadataQuorumAccumulator, legacy: &MetadataQuorumAccumulator) {
+        assert_eq!(typed.early_stop_decision(), legacy.early_stop_decision());
+        assert_eq!(typed.version_early_stop_decision(), legacy.version_early_stop_decision());
+        assert_eq!(typed.final_miss_reason(), legacy.final_miss_reason());
+        assert_eq!(typed.valid_responses, legacy.valid_responses);
+        assert_eq!(typed.not_found_responses, legacy.not_found_responses);
+        assert_eq!(typed.version_not_found_responses, legacy.version_not_found_responses);
+        assert_eq!(typed.ignored_errors, legacy.ignored_errors);
+        assert_eq!(typed.hard_errors, legacy.hard_errors);
+        assert_eq!(typed.candidate_votes, legacy.candidate_votes);
+        assert_eq!(typed.candidate_shard_mask, legacy.candidate_shard_mask);
+        assert_eq!(typed.matching_version_votes, legacy.matching_version_votes);
+        assert_eq!(typed.delete_marker_votes, legacy.delete_marker_votes);
+        assert_eq!(typed.conflicting_metadata, legacy.conflicting_metadata);
+        assert_eq!(
+            typed.candidate.as_ref().map(SetDisks::file_info_quorum_hash),
+            legacy.candidate.as_ref().map(SetDisks::file_info_quorum_hash)
+        );
+    }
+
+    fn observe_legacy(accumulator: &mut MetadataQuorumAccumulator, observation: &MetadataDiskObservation) {
+        match &observation.result {
+            None => {}
+            Some(Ok(metadata)) => accumulator.observe_file_info_at(observation.disk_index, metadata),
+            Some(Err(error)) => accumulator.observe_error(error),
+        }
+    }
+
+    #[test]
+    fn metadata_observation_all_four_slot_states_and_arrival_orders_preserve_reduction() {
+        let permutations = (0..4)
+            .flat_map(|a| (0..4).filter(move |b| *b != a).map(move |b| (a, b)))
+            .flat_map(|(a, b)| (0..4).filter(move |c| *c != a && *c != b).map(move |c| (a, b, c)))
+            .map(|(a, b, c)| [a, b, c, 6 - a - b - c])
+            .collect::<Vec<_>>();
+        assert_eq!(permutations.len(), 24);
+        // 5^4 states and all 4! arrivals, both with early-stop enabled and disabled.
+        for encoded in 0..625usize {
+            let states = [encoded % 5, encoded / 5 % 5, encoded / 25 % 5, encoded / 125 % 5];
+            let inputs = std::array::from_fn::<_, 4, _>(|index| observation(index, states[index]));
+            for enabled in [false, true] {
+                for order in &permutations {
+                    let mut typed = MetadataQuorumAccumulator::new(4, 2, enabled);
+                    let mut legacy = MetadataQuorumAccumulator::new(4, 2, enabled);
+                    for &index in order {
+                        typed.observe(&inputs[index]);
+                        observe_legacy(&mut legacy, &inputs[index]);
+                        assert_same_reduction(&typed, &legacy);
+                    }
+                    assert_eq!(typed.valid_responses, states.iter().filter(|&&state| state == 0).count());
+                    assert_eq!(typed.not_found_responses, states.iter().filter(|&&state| state == 1).count());
+                    assert_eq!(typed.hard_errors, states.iter().filter(|&&state| state == 2).count());
+                    assert_eq!(typed.ignored_errors, states.iter().filter(|&&state| state == 3).count());
+                    let expected_early_stop =
+                        enabled && typed.valid_responses >= 3 && !states.iter().any(|&state| state == 1 || state == 2);
+                    assert_eq!(
+                        typed.early_stop_decision().is_some(),
+                        expected_early_stop,
+                        "states={states:?}, order={order:?}"
+                    );
+                }
+            }
+        }
+    }
+
+    #[test]
+    fn metadata_observation_pending_newer_version_and_same_time_different_directory_force_full_wait() {
+        let mut accumulator = MetadataQuorumAccumulator::new(4, 2, true);
+        for index in 0..2 {
+            accumulator.observe(&observation(index, 0));
+        }
+        accumulator.observe(&observation(2, 4));
+        assert_eq!(accumulator.candidate_votes, 2, "an outstanding response cannot supply the deciding vote");
+        assert_eq!(accumulator.early_stop_decision(), None);
+
+        let mut newer = payload(2);
+        newer.mod_time = Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(1));
+        accumulator.observe(&MetadataDiskObservation {
+            disk_index: 2,
+            result: Some(Ok(newer)),
+        });
+        assert!(accumulator.conflicting_metadata);
+        assert_eq!(accumulator.early_stop_decision(), None);
+
+        let mut same_time_different_directory = payload(3);
+        same_time_different_directory.data_dir = Some(Uuid::from_u128(2));
+        accumulator.observe(&MetadataDiskObservation {
+            disk_index: 3,
+            result: Some(Ok(same_time_different_directory)),
+        });
+        assert_eq!(accumulator.early_stop_decision(), None);
+    }
+
+    #[test]
+    fn metadata_observation_duplicate_shard_cannot_supply_a_read_reserve() {
+        let mut accumulator = MetadataQuorumAccumulator::new(4, 2, true);
+        let first = payload(0);
+        for disk_index in 0..3 {
+            accumulator.observe(&MetadataDiskObservation {
+                disk_index,
+                result: Some(Ok(first.clone())),
+            });
+        }
+        assert_eq!(accumulator.candidate_votes, 3);
+        assert!(
+            !accumulator.candidate_has_read_reserve(),
+            "copied erasure indexes cannot supply independent data shards"
+        );
+    }
+
+    #[test]
+    fn metadata_observation_null_versions_markers_and_invalid_success_keep_their_meaning() {
+        let mut null_version = MetadataQuorumAccumulator::new(4, 2, true);
+        for index in 0..3 {
+            null_version.observe(&observation(index, 0));
+        }
+        assert!(null_version.early_stop_decision().is_some());
+        assert_eq!(null_version.matching_version_votes, 0);
+
+        let mut marker = MetadataQuorumAccumulator::new(4, 2, true);
+        for disk_index in 0..3 {
+            let metadata = FileInfo {
+                volume: "bucket".to_string(),
+                name: "object".to_string(),
+                deleted: true,
+                mod_time: Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(1)),
+                ..Default::default()
+            };
+            assert!(metadata.is_canonical_delete_marker(), "the null marker fixture must reach marker voting");
+            marker.observe(&MetadataDiskObservation {
+                disk_index,
+                result: Some(Ok(metadata)),
+            });
+        }
+        assert_eq!(marker.delete_marker_votes, 3);
+        assert!(marker.early_stop_decision().is_some());
+
+        let mut invalid = MetadataQuorumAccumulator::new(4, 2, true);
+        invalid.observe(&MetadataDiskObservation {
+            disk_index: 0,
+            result: Some(Ok(FileInfo::default())),
+        });
+        assert_eq!(invalid.valid_responses, 0);
+        assert_eq!(invalid.hard_errors, 1);
+        assert_eq!(invalid.early_stop_decision(), None);
+    }
+}

crates/ecstore/src/set_disk/core/mod.rs

@@ -18,4 +18,5 @@
 //! duplicating read/write/erasure logic.
 
 pub(crate) mod io_primitives;
-mod metadata_quorum;
+pub(in crate::set_disk) mod metadata_quorum;
+pub(in crate::set_disk) mod metadata_read;

crates/ecstore/src/set_disk/mod.rs

@@ -868,7 +868,7 @@ static OBJECT_LOCK_DIAG_ENABLED: OnceLock<bool> = OnceLock::new();
 mod core;
 #[cfg(test)]
 pub(crate) use core::io_primitives::disk_call_counters;
-#[cfg(all(test, feature = "test-util"))]
+#[cfg(test)]
 pub(crate) use core::io_primitives::{ENV_RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE, rename_fanout_barrier};
 mod ctx;
 mod metadata;
@@ -1350,6 +1350,8 @@ mod prepared_get_object_metadata_tests {
                 ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
             ],
             async {
+                // Isolate late body recovery from the independent metadata slow-tail hedge.
+                let _hedge_timer = rename_fanout_barrier::arm(&object, 0, rename_fanout_barrier::PHASE_NON_INLINE_HEDGE_TIMER);
                 let calls = disk_call_counters::observe(&object);
                 let mut reader = set_disks
                     .get_object_reader(bucket, &object, None, HeaderMap::new(), &opts)
@@ -1414,6 +1416,8 @@ mod prepared_get_object_metadata_tests {
                 ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
             ],
             async {
+                // Isolate late body recovery from the independent metadata slow-tail hedge.
+                let _hedge_timer = rename_fanout_barrier::arm(&object, 0, rename_fanout_barrier::PHASE_NON_INLINE_HEDGE_TIMER);
                 let calls = disk_call_counters::observe(&object);
                 let mut reader = set_disks
                     .get_object_reader(bucket, &object, None, HeaderMap::new(), &opts)
@@ -1427,6 +1431,11 @@ mod prepared_get_object_metadata_tests {
                     .expect("late parity should restore the exact GET body");
                 assert_eq!(restored, payload);
                 assert_eq!(calls.total(disk_call_counters::KIND_READ_VERSION), 7);
+                assert_eq!(
+                    calls.for_disk(disk_call_counters::KIND_READ_VERSION, order[3]),
+                    1,
+                    "the omitted parity disk must only be read by the late metadata refresh"
+                );
             },
         )
         .await;
@@ -1491,6 +1500,8 @@ mod prepared_get_object_metadata_tests {
                 ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
             ],
             async {
+                // Isolate late body recovery from the independent metadata slow-tail hedge.
+                let _hedge_timer = rename_fanout_barrier::arm(&object, 0, rename_fanout_barrier::PHASE_NON_INLINE_HEDGE_TIMER);
                 let calls = disk_call_counters::observe(&object);
                 let mut reader = set_disks
                     .get_object_reader(bucket, &object, None, HeaderMap::new(), &opts)
@@ -5516,12 +5527,29 @@ fn collect_inline_data_shard_fileinfos_by_index_or_reason<'a>(
     parts_metadata: &'a [FileInfo],
     fi: &FileInfo,
     data_shards: usize,
+    disk_is_online: impl FnMut(usize) -> bool,
+) -> std::result::Result<Vec<&'a FileInfo>, &'static str> {
+    collect_inline_data_shard_fileinfos_from_observations(
+        parts_metadata
+            .iter()
+            .enumerate()
+            .map(|(index, metadata)| (index, Some(metadata))),
+        fi,
+        data_shards,
+        disk_is_online,
+    )
+}
+
+fn collect_inline_data_shard_fileinfos_from_observations<'a>(
+    observations: impl IntoIterator<Item = (usize, Option<&'a FileInfo>)>,
+    fi: &FileInfo,
+    data_shards: usize,
     mut disk_is_online: impl FnMut(usize) -> bool,
 ) -> std::result::Result<Vec<&'a FileInfo>, &'static str> {
     let distribution = &fi.erasure.distribution;
     let mut data_files = vec![None; data_shards];
 
-    for (disk_index, file_info) in parts_metadata.iter().enumerate() {
+    for (disk_index, file_info) in observations {
         if !disk_is_online(disk_index) {
             continue;
         }
@@ -5531,9 +5559,9 @@ fn collect_inline_data_shard_fileinfos_by_index_or_reason<'a>(
         if block_index == 0 || block_index > data_shards {
             continue;
         }
-        if file_info.name.is_empty() {
+        let Some(file_info) = file_info.filter(|metadata| !metadata.name.is_empty()) else {
             return Err(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_MISSING_SHARD);
-        }
+        };
         if file_info.erasure.index != block_index {
             return Err(GET_METADATA_EARLY_STOP_REASON_DATA_READ_INLINE_IDENTITY_MISMATCH);
         }
@@ -7171,10 +7199,10 @@ fn parts_after_marker(part_numbers: &[usize], part_number_marker: usize) -> Opti
         return Some(part_numbers);
     }
 
-    part_numbers
-        .iter()
-        .position(|&part_number| part_number != 0 && part_number == part_number_marker)
-        .map(|index| &part_numbers[index + 1..])
+    // reduce_quorum_part_numbers returns sorted numbers; the marker need not exist.
+    part_numbers.last().filter(|&&last| part_number_marker <= last)?;
+    let index = part_numbers.partition_point(|&part_number| part_number <= part_number_marker);
+    Some(&part_numbers[index..])
 }
 
 pub fn canonicalize_etag(etag: &str) -> String {
@@ -13972,6 +14000,24 @@ mod tests {
         assert!(parts_after_marker(&part_numbers, 4).is_none());
     }
 
+    #[test]
+    fn parts_after_marker_uses_exclusive_numeric_boundary_for_sparse_parts() {
+        let part_numbers = [1, 3, 10];
+        for (marker, expected) in [
+            (0, Some(&part_numbers[..])),
+            (1, Some(&part_numbers[1..])),
+            (2, Some(&part_numbers[1..])),
+            (3, Some(&part_numbers[2..])),
+            (9, Some(&part_numbers[2..])),
+            (10, Some(&part_numbers[3..])),
+            (11, None),
+            (usize::MAX, None),
+        ] {
+            assert_eq!(parts_after_marker(&part_numbers, marker), expected, "marker {marker}");
+        }
+        assert_eq!(parts_after_marker(&[], 1), None);
+    }
+
     #[test]
     fn delete_file_info_version_id_maps_explicit_null_version_to_stored_null() {
         assert_eq!(delete_file_info_version_id(Some(Uuid::nil())), None);

crates/ecstore/src/set_disk/ops/heal.rs

@@ -598,17 +598,18 @@ impl SetDisks {
         }
 
         scope.check()?;
-        let (_, errors) =
-            Self::read_all_fileinfo(disks.as_slice(), "", bucket, object, &version.to_string(), false, false, false).await?;
+        let metadata_read =
+            Self::read_metadata(disks.as_slice(), "", bucket, object, &version.to_string(), false, false, false).await?;
+        let all_absent = metadata_read
+            .slots
+            .iter()
+            .all(|slot| matches!(&slot.result, Some(Err(DiskError::FileNotFound | DiskError::FileVersionNotFound))));
+
         let selected = self.get_disks_internal().await;
         let same_targets = selected.len() == disks.len() && selected.iter().zip(&disks).all(|(current, original)| {
             matches!((current, original), (Some(current), Some(original)) if std::sync::Arc::ptr_eq(current, original))
         });
-        if !same_targets
-            || !errors
-                .iter()
-                .all(|error| matches!(error, Some(DiskError::FileNotFound | DiskError::FileVersionNotFound)))
-        {
+        if !same_targets || !all_absent {
             return Err(StorageError::SlowDown);
         }
         scope.check()?;
@@ -734,7 +735,7 @@ impl SetDisks {
         object: &str,
         version_id: &str,
     ) -> disk::error::Result<ReadRepairCommitFingerprint> {
-        let (parts_metadata, errs, _) = Self::read_all_fileinfo_observed(
+        let metadata_read = Self::read_metadata_observed(
             disks,
             "",
             bucket,
@@ -747,6 +748,7 @@ impl SetDisks {
             self.default_parity_count,
         )
         .await?;
+        let (parts_metadata, errs, _) = metadata_read.into_legacy();
         let (read_quorum, _) = Self::object_quorum_from_meta(&parts_metadata, &errs, self.default_parity_count)?;
         let read_quorum = usize::try_from(read_quorum).map_err(|_| DiskError::ErasureReadQuorum)?;
         let (_, quorum_mod_time, quorum_etag) = Self::list_online_disks(disks, &parts_metadata, &errs, read_quorum);
@@ -931,8 +933,9 @@ impl SetDisks {
             }
         };
 
-        let (mut parts_metadata, errs) =
-            Self::read_all_fileinfo(&disks, "", bucket, object, version_id, true, true, false).await?;
+        let (mut parts_metadata, errs, _) = Self::read_metadata(&disks, "", bucket, object, version_id, true, true, false)
+            .await?
+            .into_legacy();
 
         trace!(
             event = EVENT_SET_DISK_HEAL,
@@ -1284,8 +1287,10 @@ impl SetDisks {
                             // in `parts_metadata` to defaults. Re-read only before
                             // destructive cleanup so the guard sees every original
                             // identity.
-                            let (delete_guard_metadata, delete_guard_errs) =
-                                Self::read_all_fileinfo(&disks, "", bucket, object, version_id, true, true, false).await?;
+                            let (delete_guard_metadata, delete_guard_errs, _) =
+                                Self::read_metadata(&disks, "", bucket, object, version_id, true, true, false)
+                                    .await?
+                                    .into_legacy();
                             if self
                                 .dangling_delete_safety(bucket, object, &delete_guard_metadata, &delete_guard_errs, &disks)
                                 .await?
@@ -2316,16 +2321,16 @@ impl SetDisks {
                 return Err(DiskError::retired_marker_deferred(format!("conditional marker deletion failed: {error}")));
             }
         }
-        let (_, errors) = Self::read_all_fileinfo(disks, "", bucket, object, &version.to_string(), false, false, false).await?;
+        let metadata_read = Self::read_metadata(disks, "", bucket, object, &version.to_string(), false, false, false).await?;
+        let all_absent = metadata_read
+            .slots
+            .iter()
+            .all(|slot| matches!(&slot.result, Some(Err(DiskError::FileNotFound | DiskError::FileVersionNotFound))));
         let selected = self.get_disks_internal().await;
         let same_targets = selected.len() == disks.len() && selected.iter().zip(disks).all(|(current, original)| {
             matches!((current, original), (Some(current), Some(original)) if std::sync::Arc::ptr_eq(current, original))
         });
-        if !same_targets
-            || !errors
-                .iter()
-                .all(|error| matches!(error, Some(DiskError::FileNotFound | DiskError::FileVersionNotFound)))
-        {
+        if !same_targets || !all_absent {
             return Err(DiskError::retired_marker_deferred(
                 "complete marker absence could not be verified under the original target set",
             ));
@@ -3383,9 +3388,10 @@ impl SetDisks {
         // The inner heal and missing-object report read the registry again;
         // release this snapshot guard before a topology writer can queue between reads.
         let disks = self.get_disks_internal().await;
-        let (_, errs) = Self::read_all_fileinfo(&disks, "", bucket, object, version_id, false, false, false)
+        let (_, errs, _) = Self::read_metadata(&disks, "", bucket, object, version_id, false, false, false)
             .await
-            .map_err(|e| to_object_err(e.into(), vec![bucket, object]))?;
+            .map_err(|e| to_object_err(e.into(), vec![bucket, object]))?
+            .into_legacy();
         if DiskError::is_all_not_found(&errs) {
             debug!(
                 event = EVENT_SET_DISK_HEAL,

crates/ecstore/src/set_disk/ops/multipart.rs

@@ -21,6 +21,7 @@
 //! unchanged; method bodies are moved verbatim and runtime behavior is the same.
 
 use crate::core::pools::DecommissionCapacityAdmission;
+use crate::object_api::write_commit_context::WriteCommitContext;
 
 #[cfg(test)]
 use super::super::GetObjectMetadataCacheKey;
@@ -2125,7 +2126,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
 
         if object_parts.len() > ret.parts.len() {
             ret.is_truncated = true;
-            ret.next_part_number_marker = ret.parts.last().map(|v| v.part_num).unwrap_or_default();
+            ret.next_part_number_marker = ret.parts.last().map(|v| v.part_num);
         }
 
         ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
@@ -2446,17 +2447,19 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
         opts: &ObjectOptions,
     ) -> Result<ObjectInfo> {
         crate::hp_guard!("SetDisks::complete_multipart_upload");
+        let write_context = WriteCommitContext::from_options(opts, bucket, object, false)?;
         self.invalidate_get_object_metadata_cache(bucket, object).await;
 
         let upload_id_path = Self::get_multipart_upload_dir(bucket, object, upload_id, opts.data_movement);
-        let range_seek_rollout_enabled = crate::object_api::legacy_encrypted_range_seek_enabled() && !opts.no_lock;
+        let range_seek_rollout_enabled =
+            crate::object_api::legacy_encrypted_range_seek_enabled() && write_context.acquires_namespace();
         let mut object_lock_guard = None;
         let mut decommission_object_lock_guard = None;
         let mut decommission_target_lock_covered = false;
         let mut decommission_capacity_guard = None;
 
         if opts.http_preconditions.is_some() {
-            if !opts.no_lock {
+            if write_context.acquires_namespace() {
                 object_lock_guard = Some(
                     self.acquire_write_lock_diag("complete_multipart_upload_precondition", bucket, object)
                         .await?,
@@ -2479,7 +2482,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
                     self.pool_index,
                     bucket,
                     object,
-                    opts.no_lock || object_lock_guard.is_some(),
+                    !write_context.acquires_namespace() || object_lock_guard.is_some(),
                     DecommissionCapacityAdmission::ExistingMultipart,
                 )
                 .await?;
@@ -2487,7 +2490,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
             decommission_target_lock_covered = target_lock_covered;
             decommission_capacity_guard = capacity_guard;
         }
-        if !opts.no_lock && object_lock_guard.is_none() && !decommission_target_lock_covered {
+        if write_context.acquires_namespace() && object_lock_guard.is_none() && !decommission_target_lock_covered {
             object_lock_guard = Some(
                 self.acquire_write_lock_diag("complete_multipart_upload_commit", bucket, object)
                     .await?,
@@ -3354,8 +3357,10 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
         // usable as a CopyObject source. Do not return on rename quorum while
         // a tail owner may still hold the object guard and finish shard moves.
         let commit_allows_early_ack = false;
-        let detach_commit_owner = commit_allows_early_ack || upload_guard.is_some() || quota_mutation_fence;
+        let detach_commit_owner =
+            write_context.has_borrowed_owner() || commit_allows_early_ack || upload_guard.is_some() || quota_mutation_fence;
         let commit = async move {
+            let write_context = write_context;
             let mut _object_lock_guard = commit_object_lock_guard;
             let mut _decommission_object_lock_guard = commit_decommission_object_lock_guard;
             let mut _upload_guard = upload_guard;
@@ -3377,6 +3382,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
                 pause_multipart_commit(&commit_bucket, &commit_object, MultipartCommitPause::BeforeQuotaRename).await;
                 if quota_reservation.is_lock_lost()
                     || !quota_reservation.capability_proof_matches()
+                    || write_context.is_lock_lost()
                     || _object_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
                     || _decommission_object_lock_guard
                         .as_ref()
@@ -3431,6 +3437,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
                 }
                 if quota_reservation.is_lock_lost()
                     || !quota_reservation.capability_proof_matches()
+                    || write_context.is_lock_lost()
                     || _object_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
                     || _decommission_object_lock_guard
                         .as_ref()
@@ -3761,6 +3768,7 @@ mod tests {
     use crate::layout::endpoints::SetupType;
     use crate::services::notification_sys::install_remote_version_state_fleet_proof_for_test;
     use crate::set_disk::core::io_primitives::{ENV_RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE, rename_fanout_barrier};
+    use crate::set_disk::get_lock_acquire_timeout;
     // No-locker helpers resolve to the isolated-context variants (see
     // `hermetic_set_disks_isolated`); the guard-based tests build through
     // `hermetic_set_disks_with_lockers`, which stays on the bootstrap context
@@ -4505,6 +4513,54 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn list_object_parts_pagination_preserves_sparse_markers_and_terminates() {
+        let (_temp_dirs, disks, set_disks) = hermetic_set_disks(4).await;
+        let bucket = "multipart-pagination";
+        let object = "object";
+        make_bucket_on_all(&disks, bucket).await;
+        let opts = ObjectOptions::default();
+        let upload = set_disks
+            .new_multipart_upload(bucket, object, &opts)
+            .await
+            .expect("multipart upload should be created");
+
+        let empty = set_disks
+            .list_object_parts(bucket, object, &upload.upload_id, None, 2, &opts)
+            .await
+            .expect("empty upload should be listable");
+        assert!(empty.parts.is_empty());
+        assert!(!empty.is_truncated);
+        assert_eq!(empty.next_part_number_marker, None);
+
+        for part_number in [10, 1, 3] {
+            put_test_part(&set_disks, bucket, object, &upload.upload_id, part_number, b"part", 4).await;
+        }
+
+        for (marker, max_parts, expected_parts, next_marker) in [
+            (None, 0, vec![], None),
+            (None, 1, vec![1], Some(1)),
+            (None, 2, vec![1, 3], Some(3)),
+            (None, 3, vec![1, 3, 10], None),
+            (None, MAX_PARTS_COUNT + 1, vec![1, 3, 10], None),
+            (Some(1), 2, vec![3, 10], None),
+            (Some(2), 1, vec![3], Some(3)),
+            (Some(3), 2, vec![10], None),
+            (Some(10), 2, vec![], None),
+            (Some(11), 2, vec![], None),
+        ] {
+            let page = set_disks
+                .list_object_parts(bucket, object, &upload.upload_id, marker, max_parts, &opts)
+                .await
+                .expect("uploaded parts should be listable");
+            assert_eq!(page.part_number_marker, marker.unwrap_or_default());
+            assert_eq!(page.max_parts, max_parts.min(MAX_PARTS_COUNT));
+            assert_eq!(page.parts.iter().map(|part| part.part_num).collect::<Vec<_>>(), expected_parts);
+            assert_eq!(page.is_truncated, next_marker.is_some());
+            assert_eq!(page.next_part_number_marker, next_marker);
+        }
+    }
+
     async fn object_transaction_epochs(disks: &[DiskStore], bucket: &str, object: &str) -> Vec<Option<Uuid>> {
         let mut epochs = Vec::with_capacity(disks.len());
         let deadline = tokio::time::Instant::now() + Duration::from_secs(30);
@@ -7074,6 +7130,54 @@ mod tests {
         .await;
     }
 
+    #[tokio::test(flavor = "multi_thread")]
+    #[serial]
+    async fn borrowed_complete_keeps_namespace_owner_after_caller_cancel() {
+        let (_dirs, disks, set) = hermetic_set_disks(4).await;
+        let bucket = "multipart-borrowed-cancel";
+        let object = "object";
+        make_bucket_on_all(&disks, bucket).await;
+        let (upload_id, parts) =
+            stage_upload_with_create_opts(&set, bucket, object, &[0x53; 4096], &ObjectOptions::default()).await;
+        let barrier = MultipartCommitBarrier::install(bucket, object, MultipartCommitPause::AfterRename);
+        let writer = Arc::clone(&set);
+        let complete = tokio::spawn(async move {
+            let outer_guard = crate::object_api::WriteCommitGuard::acquire(
+                &writer.new_ns_lock(bucket, object).await.expect("namespace wrapper"),
+                get_lock_acquire_timeout(),
+            )
+            .await
+            .expect("outer MPU namespace guard");
+            let mut opts = ObjectOptions::default();
+            opts.add_write_commit_guard(&outer_guard);
+            writer
+                .complete_multipart_upload(bucket, object, &upload_id, parts, &opts)
+                .await
+        });
+        tokio::time::timeout(Duration::from_secs(30), barrier.wait_until_paused())
+            .await
+            .expect("borrowed completion reaches committed cleanup");
+        complete.abort();
+        assert!(complete.await.expect_err("completion caller canceled").is_cancelled());
+        let mut probe = Box::pin(set.acquire_write_lock_diag("borrowed_complete_probe", bucket, object));
+        assert!(
+            futures::poll!(probe.as_mut()).is_pending(),
+            "borrowed completion retains the real outer namespace owner"
+        );
+        barrier.release();
+        drop(
+            tokio::time::timeout(Duration::from_secs(30), probe)
+                .await
+                .expect("completion owner finishes cleanup")
+                .expect("next writer obtains namespace"),
+        );
+        let info = set
+            .get_object_info(bucket, object, &ObjectOptions::default())
+            .await
+            .expect("borrowed MPU result remains readable");
+        assert_eq!(info.size, 4096);
+    }
+
     #[tokio::test(flavor = "multi_thread")]
     #[serial]
     async fn cancelled_complete_keeps_upload_lock_through_tail_cleanup() {

crates/ecstore/src/set_disk/ops/object.rs

@@ -281,7 +281,7 @@ use super::super::GetObjectMetadataCacheEntry;
 use super::super::GetObjectMetadataCacheKey;
 #[cfg(test)]
 use super::super::capacity_scope_from_disks;
-#[cfg(all(test, feature = "test-util"))]
+#[cfg(test)]
 use super::super::get_lock_acquire_timeout;
 use crate::bucket::lifecycle::{
     tier_delete_journal::{TierDeleteDispatchAuthorization, record_tier_delete_journal_backend_identity},
@@ -320,10 +320,13 @@ use crate::disk::format::FormatV3;
 use crate::disk::new_disk;
 use crate::disk::{DataDirDeleteStatus, OldCurrentSize};
 use crate::error::is_err_invalid_upload_id;
+#[cfg(test)]
+use crate::object_api::WriteCompletion;
+use crate::object_api::write_commit_context::WriteCommitContext;
 use crate::object_api::{GetObjectBodySource, get_object_body_cache_hook_suppressed};
 use crate::object_api::{
     NamespaceLockFence, ReplicationStatusWritebackCondition, ReplicationStatusWritebackMode,
-    SCANNER_PUBLICATION_LEASE_FENCE_METADATA_KEY, WriteCompletion,
+    SCANNER_PUBLICATION_LEASE_FENCE_METADATA_KEY,
 };
 use crate::services::notification_sys::RemoteVersionStateFleetProofToken;
 use crate::services::tier::tier::{TierConfigMgr, TierDestinationId, TierOperationLease, tier_destination_id_from_metadata};
@@ -3499,6 +3502,7 @@ impl SetDisks {
         opts: &ObjectOptions,
         mut publication_fence: Option<RemoteTuplePublicationFence>,
     ) -> Result<(ObjectInfo, Option<OldCurrentSize>)> {
+        let write_context = WriteCommitContext::from_options(opts, bucket, object, publication_fence.is_some())?;
         let protect_write = opts.shard_integrity_write_enabled();
         let source_bucket_incarnation_id = match opts.expected_bucket_incarnation_id {
             Some(incarnation_id) => Some(incarnation_id),
@@ -4026,7 +4030,7 @@ impl SetDisks {
                         self.pool_index,
                         bucket,
                         object,
-                        opts.no_lock || object_lock_guard.is_some(),
+                        !write_context.acquires_namespace() || object_lock_guard.is_some(),
                         DecommissionCapacityAdmission::Mutation,
                     )
                     .await?;
@@ -4034,7 +4038,8 @@ impl SetDisks {
                 decommission_target_lock_covered = target_lock_covered;
                 decommission_capacity_guard = capacity_guard;
             }
-            if publication_fence.is_some() || (!opts.no_lock && object_lock_guard.is_none() && !decommission_target_lock_covered)
+            if publication_fence.is_some()
+                || (write_context.acquires_namespace() && object_lock_guard.is_none() && !decommission_target_lock_covered)
             {
                 #[cfg(any(test, feature = "test-util"))]
                 pause_put_object_commit(bucket, object, PutObjectCommitPause::BeforeNamespace).await;
@@ -4382,10 +4387,11 @@ impl SetDisks {
             // complete rename fan-out drains. Keep this path synchronous so
             // its terminal state is known before the coordinator releases
             // remote leases.
-            let commit_owns_namespace_guard = commit_object_lock_guard.is_some()
+            let commit_owns_namespace_guard = write_context.has_borrowed_owner()
+                || commit_object_lock_guard.is_some()
                 || commit_decommission_object_lock_guard.is_some()
                 || commit_publication_guard.is_some();
-            let commit_allows_early_ack = opts.write_completion == WriteCompletion::Quorum
+            let commit_allows_early_ack = write_context.allows_early_ack(opts.write_completion)
                 && !(opts.data_movement && opts.has_decommission_capacity_reservation())
                 && commit_owns_namespace_guard
                 && commit_scanner_publication_scope.is_none();
@@ -4412,6 +4418,7 @@ impl SetDisks {
             tmp_cleanup_owned = true;
 
             let commit = move |cancellation: Option<CancellationToken>| async move {
+                let write_context = write_context;
                 let mut _object_lock_guard = commit_object_lock_guard;
                 let mut _decommission_object_lock_guard = commit_decommission_object_lock_guard;
                 let mut _publication_guard = commit_publication_guard;
@@ -4427,6 +4434,7 @@ impl SetDisks {
                     pause_put_object_commit(&commit_bucket, &commit_object, PutObjectCommitPause::BeforeQuotaRename).await;
                     if quota_reservation.is_lock_lost()
                         || !quota_reservation.capability_proof_matches()
+                        || write_context.is_lock_lost()
                         || _object_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
                         || _decommission_object_lock_guard
                             .as_ref()
@@ -4486,6 +4494,7 @@ impl SetDisks {
                     }
                     if quota_reservation.is_lock_lost()
                         || !quota_reservation.capability_proof_matches()
+                        || write_context.is_lock_lost()
                         || _object_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
                         || _decommission_object_lock_guard
                             .as_ref()
@@ -4671,6 +4680,7 @@ impl SetDisks {
                             guard_release_rx,
                             (
                                 object_lock_guard,
+                                write_context,
                                 publication_guard,
                                 bucket_lifecycle_guard,
                                 decommission_object_lock_guard,
@@ -4691,6 +4701,7 @@ impl SetDisks {
                             },
                             move |(
                                 object_lock_guard,
+                                write_context,
                                 publication_guard,
                                 bucket_lifecycle_guard,
                                 decommission_object_lock_guard,
@@ -4698,6 +4709,7 @@ impl SetDisks {
                             ),
                                   targets| async move {
                                 drop(object_lock_guard);
+                                drop(write_context);
                                 drop(publication_guard);
                                 drop(bucket_lifecycle_guard);
                                 cleanup_set
@@ -19817,6 +19829,16 @@ mod put_object_tmp_cleanup_tests {
             .await
             .expect("committed object must remain readable despite the failed tail");
         assert_eq!(info.size, TEST_OBJECT_SIZE as i64);
+        let mut read = set
+            .get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
+            .await
+            .expect("committed body must remain readable despite the failed tail");
+        let mut body = Vec::new();
+        read.stream
+            .read_to_end(&mut body)
+            .await
+            .expect("committed body must stream completely after the failed tail");
+        assert_eq!(body, vec![b'1'; TEST_OBJECT_SIZE]);
     }
 
     #[tokio::test]
@@ -20002,6 +20024,93 @@ mod put_object_tmp_cleanup_tests {
         wait_for_tmp_workspace_to_drain(&dirs, "cancelled full-tail commit should release staging ownership").await;
     }
 
+    #[tokio::test]
+    #[serial_test::serial(capacity_dirty_scope)]
+    async fn borrowed_write_context_rejects_wrong_namespace_and_cache_flag() {
+        let (_dirs, _disks, set) = hermetic_set_disks(4).await;
+        let bucket = "borrowed-write-target";
+        let object = "owned-object";
+        let guard = crate::object_api::WriteCommitGuard::acquire(
+            &set.new_ns_lock(bucket, object).await.expect("namespace wrapper"),
+            get_lock_acquire_timeout(),
+        )
+        .await
+        .expect("outer write lock");
+        let mut opts = ObjectOptions::default();
+        opts.add_write_commit_guard(&guard);
+        assert!(matches!(
+            WriteCommitContext::from_options(&opts, bucket, "other-object", false),
+            Err(Error::InvalidArgument(..))
+        ));
+        assert!(matches!(
+            WriteCommitContext::from_options(&opts, bucket, "other-object", true),
+            Err(Error::InvalidArgument(..))
+        ));
+        let fake = ObjectOptions {
+            no_lock: true,
+            metadata_cache_safe: true,
+            ..Default::default()
+        };
+        assert!(matches!(
+            WriteCommitContext::from_options(&fake, bucket, object, false),
+            Err(Error::InvalidArgument(..))
+        ));
+    }
+
+    #[tokio::test]
+    #[serial_test::serial(capacity_dirty_scope)]
+    async fn borrowed_tail_drained_put_keeps_owner_after_caller_cancel() {
+        let (_dirs, disks, set) = hermetic_set_disks(4).await;
+        let bucket = "borrowed-put-cancel";
+        let object = "borrowed-put-cancel-object";
+        for disk in &disks {
+            disk.make_volume(bucket).await.expect("bucket volume");
+        }
+        let barrier = rename_fanout_barrier::arm(object, 0, rename_fanout_barrier::PHASE_RENAME);
+        let tasks = rename_fanout_barrier::observe_tasks(object);
+        let writer = Arc::clone(&set);
+        let put = tokio::spawn(async move {
+            let guard = crate::object_api::WriteCommitGuard::acquire(
+                &writer.new_ns_lock(bucket, object).await.expect("namespace wrapper"),
+                get_lock_acquire_timeout(),
+            )
+            .await
+            .expect("outer write guard");
+            let mut opts = ObjectOptions {
+                write_completion: WriteCompletion::TailDrained,
+                ..Default::default()
+            };
+            opts.add_write_commit_guard(&guard);
+            let mut reader = PutObjReader::from_vec(vec![b'b'; TEST_OBJECT_SIZE]);
+            writer.put_object(bucket, object, &mut reader, &opts).await
+        });
+        tokio::time::timeout(Duration::from_secs(30), barrier.wait_until_paused())
+            .await
+            .expect("borrowed commit reaches rename phase");
+        wait_for_paused_tail_metadata_quorum(&disks, bucket, object).await;
+        assert!(!put.is_finished(), "full-tail borrowed write remains pending");
+        put.abort();
+        assert!(put.await.expect_err("caller was canceled").is_cancelled());
+        let mut probe = Box::pin(set.acquire_write_lock_diag("borrowed_cancel_probe", bucket, object));
+        assert!(
+            futures::poll!(probe.as_mut()).is_pending(),
+            "commit must retain the real borrowed namespace owner"
+        );
+        barrier.release();
+        drop(
+            tokio::time::timeout(Duration::from_secs(30), probe)
+                .await
+                .expect("borrowed owner drains")
+                .expect("next writer acquires"),
+        );
+        assert_eq!(tasks.running(), 0, "all borrowed rename tasks are reaped before releasing the owner");
+        let info = set
+            .get_object_info(bucket, object, &ObjectOptions::default())
+            .await
+            .expect("committed borrowed write remains readable");
+        assert_eq!(info.size, TEST_OBJECT_SIZE as i64);
+    }
+
     #[tokio::test]
     #[serial_test::serial(capacity_dirty_scope)]
     async fn no_lock_put_waits_for_rename_tail_under_outer_guard() {
@@ -20013,27 +20122,23 @@ mod put_object_tmp_cleanup_tests {
                 disk.make_volume(bucket).await.expect("bucket volume should be created");
             }
 
-            let outer_guard = set_disks
-                .acquire_write_lock_diag("outer_no_lock_put", bucket, object)
-                .await
-                .expect("the outer caller should hold the namespace guard");
+            let outer_guard = crate::object_api::WriteCommitGuard::acquire(
+                &set_disks.new_ns_lock(bucket, object).await.expect("outer namespace wrapper"),
+                get_lock_acquire_timeout(),
+            )
+            .await
+            .expect("the outer caller should hold the namespace guard");
             let rename_tasks = rename_fanout_barrier::observe_tasks(object);
             let rename_barrier = rename_fanout_barrier::arm(object, 0, rename_fanout_barrier::PHASE_RENAME);
+            let mut opts = ObjectOptions {
+                write_completion: WriteCompletion::TailDrained,
+                ..Default::default()
+            };
+            opts.add_write_commit_guard(&outer_guard);
             let put_store = Arc::clone(&set_disks);
             let put = tokio::spawn(async move {
                 let mut reader = PutObjReader::from_vec(vec![b'1'; TEST_OBJECT_SIZE]);
-                put_store
-                    .put_object(
-                        bucket,
-                        object,
-                        &mut reader,
-                        &ObjectOptions {
-                            no_lock: true,
-                            write_completion: WriteCompletion::TailDrained,
-                            ..Default::default()
-                        },
-                    )
-                    .await
+                put_store.put_object(bucket, object, &mut reader, &opts).await
             });
 
             tokio::time::timeout(Duration::from_secs(30), rename_barrier.wait_until_paused())

crates/ecstore/src/set_disk/read.rs

@@ -113,7 +113,28 @@ use super::GetCodecStreamingObjectClass;
 use super::GetCodecStreamingRollout;
 #[cfg(test)]
 use super::classify_get_codec_streaming_object_class;
-use super::core::io_primitives::*;
+use super::core::io_primitives::{
+    BitrotReaderSetup, BitrotReaderSetupAttribution, BitrotReaderSetupMode, DeferredReaderReopener, EVENT_SET_DISK_READ,
+    GetCodecStreamingReaderBuildOutcome, MetadataCacheLookup, ObjectBitrotReader, ReadRepairAdmissionSubmitter,
+    ReadRepairHealSubmission, SLOW_OBJECT_READ_LOG_THRESHOLD, codec_streaming_reader_setup_fallback_reason,
+    create_bitrot_readers_until_quorum_with_preference, create_data_block_bitrot_readers, resolved_read_repair_version_id,
+    send_read_repair_heal_request, shard_read_costs_for_disks, submit_read_repair_heal, submit_read_repair_heal_with_submitter,
+};
+#[cfg(test)]
+use super::core::io_primitives::{
+    ENV_RUSTFS_GET_CODEC_STREAMING_DATA_BLOCKS_FIRST_READER_SETUP, ENV_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP,
+    MultipartCodecStreamingReader, ReadRepairAdmissionFuture, ReadRepairAdmissionOutcome, collect_read_multiple_results,
+    collect_read_parts_results, create_bitrot_readers_until_quorum, create_bitrot_readers_until_quorum_all_shards,
+    release_read_repair_heal_reservation, reserve_read_repair_heal, resolve_read_part_from_responses, shard_read_cost_for_disk,
+    shard_read_cost_for_endpoint,
+};
+#[cfg(test)]
+use super::core::metadata_quorum::{MetadataEarlyStopDecision, MetadataQuorumAccumulator};
+#[cfg(test)]
+use super::core::metadata_read::{
+    MetadataFanoutDiagnostics, MetadataFanoutObservation, metadata_early_stop_permitted, should_allow_metadata_early_stop,
+};
+use super::core::metadata_read::{late_materialization_candidate_is_safe, non_inline_data_read_early_stop_allowed};
 #[cfg(test)]
 use super::get_codec_streaming_config_cached_core;
 #[cfg(test)]
@@ -607,11 +628,11 @@ impl SetDisks {
         };
 
         // Early-stop for safe metadata reads is handled inside
-        // read_all_fileinfo_observed (see read_all_fileinfo_early_stop in
-        // core/io_primitives.rs); unsafe requests and callers that opt out
+        // read_metadata_observed (see read_all_fileinfo_early_stop in
+        // core/metadata_read.rs); unsafe requests and callers that opt out
         // (allow_early_stop=false) fall back to full-wait.
-        let (mut parts_metadata, errs, metadata_fanout_diagnostics) = if allow_read_version_coalescing {
-            Self::read_all_fileinfo_observed_for_get_object(
+        let metadata_read = if allow_read_version_coalescing {
+            Self::read_metadata_for_get_object(
                 &disks,
                 "",
                 bucket,
@@ -624,7 +645,7 @@ impl SetDisks {
             )
             .await?
         } else {
-            Self::read_all_fileinfo_observed(
+            Self::read_metadata_observed(
                 &disks,
                 "",
                 bucket,
@@ -638,13 +659,14 @@ impl SetDisks {
             )
             .await?
         };
+        let metadata_fanout_complete = metadata_read.is_complete();
+        let (mut parts_metadata, errs, metadata_fanout_diagnostics) = metadata_read.into_legacy();
         let metadata_metrics_path = if crate::bucket::utils::is_meta_bucketname(bucket) {
             GET_OBJECT_PATH_INTERNAL_META
         } else {
             GET_OBJECT_PATH_LEGACY_DUPLEX
         };
         metadata_fanout_diagnostics.record(metadata_metrics_path);
-        let metadata_fanout_complete = metadata_fanout_diagnostics.total_responses() >= disks.len();
         // warn!("get_object_fileinfo parts_metadata {:?}", &parts_metadata);
         // warn!("get_object_fileinfo {}/{} errs {:?}", bucket, object, &errs);
 
@@ -2117,7 +2139,7 @@ impl SetDisks {
         expected: &LateMetadataIdentity,
         metrics_path: &'static str,
     ) -> Result<(FileInfo, Vec<FileInfo>, Vec<Option<DiskStore>>)> {
-        let (mut parts_metadata, errs, diagnostics) = SetDisks::read_all_fileinfo_observed(
+        let metadata_read = SetDisks::read_metadata_observed(
             fallback_disks,
             "",
             bucket,
@@ -2130,6 +2152,7 @@ impl SetDisks {
             expected.parity_blocks,
         )
         .await?;
+        let (mut parts_metadata, errs, diagnostics) = metadata_read.into_legacy();
         diagnostics.record(metrics_path);
 
         let (read_quorum, write_quorum) = SetDisks::object_quorum_from_meta(&parts_metadata, &errs, expected.parity_blocks)

crates/ecstore/src/store/bucket.rs

@@ -14,7 +14,7 @@
 
 use super::*;
 use crate::bucket::{
-    metadata::{BUCKET_TABLE_RESERVED_PREFIX, table_bucket_catalog_metadata_prefix},
+    metadata::{BUCKET_TABLE_RESERVED_PREFIX, BUCKET_TAGGING_CONFIG, ConfigState, table_bucket_catalog_metadata_prefix},
     utils::is_meta_bucketname,
 };
 use crate::error::is_err_bucket_not_found;
@@ -272,6 +272,32 @@ impl ECStore {
         sys.read().await.get(bucket).await
     }
 
+    /// Resolve stored tags for authorization. Only confirmed absence returns an
+    /// empty map; unavailable, fabricated or malformed metadata must not grant access.
+    pub async fn get_bucket_tags_for_policy(&self, bucket: &str) -> Result<HashMap<String, String>> {
+        let sys = metadata_sys::require_bucket_metadata_sys_in(&self.ctx)?;
+        let metadata = match sys.read().await.get_authoritative_metadata(bucket).await {
+            Ok(metadata) => metadata,
+            Err(Error::ConfigNotFound) => return Ok(HashMap::new()),
+            Err(err) => return Err(err),
+        };
+        let Some(tagging) =
+            ConfigState::of(&metadata.tagging_config_xml, &metadata.tagging_config).require(bucket, BUCKET_TAGGING_CONFIG)?
+        else {
+            return Ok(HashMap::new());
+        };
+        let mut tags = HashMap::with_capacity(tagging.tag_set.len());
+        for tag in &tagging.tag_set {
+            let (Some(key), Some(value)) = (&tag.key, &tag.value) else {
+                return Err(Error::other("stored bucket tag is missing a key or value"));
+            };
+            if key.is_empty() || tags.insert(key.clone(), value.clone()).is_some() {
+                return Err(Error::other("stored bucket tags contain empty or duplicate keys"));
+            }
+        }
+        Ok(tags)
+    }
+
     pub async fn get_bucket_policy(&self, bucket: &str) -> Result<(BucketPolicy, OffsetDateTime)> {
         let sys = metadata_sys::require_bucket_metadata_sys_in(&self.ctx)?;
         sys.read().await.get_bucket_policy(bucket).await
@@ -1212,6 +1238,7 @@ mod tests {
     use rustfs_filemeta::{FileInfo, FileMeta, TRANSITION_COMPLETE};
     use rustfs_lock::{LocalClient, LockRequest, LockType, NamespaceLock, ObjectKey};
     use serial_test::serial;
+    use std::collections::HashMap;
     use std::path::{Path, PathBuf};
     use std::sync::Arc;
     use std::sync::atomic::{AtomicBool, Ordering};
@@ -1619,6 +1646,55 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn bucket_tags_for_policy_distinguish_absence_from_corrupt_metadata() {
+        let (_temp_dir, store) = setup_bucket_quorum_test_env(&[4], None).await;
+        metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await;
+        assert!(
+            store
+                .get_bucket_tags_for_policy("not-created")
+                .await
+                .expect("confirmed missing bucket")
+                .is_empty()
+        );
+
+        let bucket = "policy-bucket-tags";
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("create tagged bucket");
+        let original = store.get_bucket_metadata(bucket).await.expect("read bucket metadata");
+        let cases = &[
+            (b"".as_slice(), Some(HashMap::new())),
+            (b"<Tagging><TagSet/></Tagging>", Some(HashMap::new())),
+            (
+                b"<Tagging><TagSet><Tag><Key>Department</Key><Value>Finance</Value></Tag><Tag><Key>note</Key><Value></Value></Tag></TagSet></Tagging>",
+                Some(HashMap::from([("Department".to_string(), "Finance".to_string()), ("note".to_string(), String::new())])),
+            ),
+            (b"<Tagging><TagSet>", None),
+            (b"<Tagging><TagSet><Tag><Value>finance</Value></Tag></TagSet></Tagging>", None),
+            (b"<Tagging><TagSet><Tag><Key>department</Key></Tag></TagSet></Tagging>", None),
+            (b"<Tagging><TagSet><Tag><Key></Key><Value>finance</Value></Tag></TagSet></Tagging>", None),
+            (
+                b"<Tagging><TagSet><Tag><Key>department</Key><Value>finance</Value></Tag><Tag><Key>department</Key><Value>engineering</Value></Tag></TagSet></Tagging>",
+                None,
+            ),
+        ];
+        for (xml, expected) in cases {
+            let mut metadata = (*original).clone();
+            metadata.tagging_config_xml = xml.to_vec();
+            metadata.tagging_config = crate::bucket::utils::deserialize(xml).ok();
+            metadata_sys::set_bucket_metadata_in(&store.ctx, metadata)
+                .await
+                .expect("inject persisted tag state");
+            let result = store.get_bucket_tags_for_policy(bucket).await;
+            match expected {
+                Some(tags) => assert_eq!(&result.expect("valid tag configuration"), tags),
+                None => assert!(result.is_err(), "corrupt tags must not become an untagged bucket: {xml:?}"),
+            }
+        }
+    }
+
     #[tokio::test]
     async fn request_metadata_methods_fail_closed_before_instance_initialization() {
         let (_temp_dir, store) = setup_multi_pool_bucket_test_env().await;
@@ -1626,6 +1702,10 @@ mod tests {
         let expected = "bucket metadata sys not initialized for this instance";
         let errors = [
             store.get_bucket_metadata("bucket").await.unwrap_err(),
+            store
+                .get_bucket_tags_for_policy("bucket")
+                .await
+                .expect_err("uninitialized metadata must deny tag lookup"),
             store.get_bucket_policy("bucket").await.unwrap_err(),
             store.get_bucket_policy_raw("bucket").await.unwrap_err(),
             store.restricts_public_bucket_access("bucket").await.unwrap_err(),

crates/ecstore/src/store/init.rs

@@ -4362,7 +4362,7 @@ mod tests {
             // successful attempts. Only injected faults spend this global budget.
             // A real failure may consume an attempt, so preserve the final chance.
             faults
-                .fetch_update(Ordering::SeqCst, Ordering::SeqCst, |faults| {
+                .try_update(Ordering::SeqCst, Ordering::SeqCst, |faults| {
                     (faults < crate::core::pools::DECOMMISSION_VERSION_COPY_ATTEMPTS.saturating_sub(1))
                         .then_some(faults.saturating_add(1))
                 })

crates/ecstore/src/store/list_objects.rs

@@ -354,6 +354,15 @@ fn ensure_producer_limit_state(options: &mut ListPathOptions) -> Arc<AtomicBool>
     state
 }
 
+fn list_path_folds_common_prefixes(options: &ListPathOptions) -> bool {
+    !options.versioned
+        && (options.include_directories
+            || options
+                .separator
+                .as_deref()
+                .is_some_and(|separator| !separator.is_empty() && separator != SLASH_SEPARATOR))
+}
+
 async fn can_skip_hidden_prefix_check(options: &ListPathOptions) -> bool {
     if options.recursive || options.incl_deleted || options.versioned {
         return false;
@@ -2440,6 +2449,12 @@ where
             let suffix = key.trim_start_matches(prefix);
             if let Some((common_prefix, _)) = suffix.split_once(separator) {
                 let common_prefix = format!("{prefix}{common_prefix}{separator}");
+                if marker.is_some_and(|marker| common_prefix.as_str() <= marker) {
+                    if collect_stats {
+                        stats.skipped_keys += 1;
+                    }
+                    continue;
+                }
                 if prefix_set.insert(common_prefix.clone()) {
                     if collect_stats {
                         stats.common_prefixes += 1;
@@ -2542,6 +2557,10 @@ fn list_objects_from_metadata_snapshot_candidates(
             let suffix = object.name.trim_start_matches(prefix);
             if let Some((common_prefix, _)) = suffix.split_once(separator) {
                 let common_prefix = format!("{prefix}{common_prefix}{separator}");
+                if marker.is_some_and(|marker| common_prefix.as_str() <= marker) {
+                    stats.skipped_keys += 1;
+                    continue;
+                }
                 if prefix_set.insert(common_prefix.clone()) {
                     stats.common_prefixes += 1;
                     visible_entries.push(VerifiedIndexVisibleEntry::Prefix(common_prefix));
@@ -4962,6 +4981,8 @@ async fn gather_results(
     let gather_started = list_metrics_enabled.then(std::time::Instant::now);
     let mut scanned_entries = 0usize;
     let mut candidate_entries = 0usize;
+    let mut previous_prefix = String::new();
+    let folds_prefixes = list_path_folds_common_prefixes(&opts);
 
     while let Some(mut entry) = recv.recv().await {
         scanned_entries += 1;
@@ -5001,6 +5022,34 @@ async fn gather_results(
             continue;
         }
 
+        // Version LIST projections omit cleanup-only free versions. They must
+        // not consume the lookahead budget before a later visible version.
+        // Cleanup walks consume the raw set stream without this collector.
+        if opts.versioned
+            && is_object
+            && entry
+                .xl_meta()
+                .is_ok_and(|meta| !meta.versions.is_empty() && meta.versions.iter().all(|version| version.header.free_version()))
+        {
+            continue;
+        }
+
+        if folds_prefixes
+            && let Some(separator) = opts.separator.as_deref().filter(|separator| !separator.is_empty())
+            && let Some(offset) = entry.name[opts.prefix.len()..].find(separator)
+        {
+            let common_prefix = &entry.name[..opts.prefix.len() + offset + separator.len()];
+            if common_prefix == previous_prefix
+                || opts.marker.as_deref().is_some_and(|marker| {
+                    (!opts.include_marker && common_prefix <= marker) || (opts.include_marker && common_prefix < marker)
+                })
+            {
+                continue;
+            }
+            previous_prefix.clear();
+            previous_prefix.push_str(common_prefix);
+        }
+
         // TODO(backlog): integrate lifecycle evaluation during object listing
 
         entries.push(Some(entry));
@@ -7023,7 +7072,9 @@ impl SetDisks {
         );
 
         let limit = {
-            if opts.limit > 0 && opts.stop_disk_at_limit {
+            // A raw scan budget can be consumed entirely by keys folding into
+            // one common prefix. Let the logical collector stop these walks.
+            if opts.limit > 0 && opts.stop_disk_at_limit && !list_path_folds_common_prefixes(&opts) {
                 opts.limit + 4 + (opts.limit / 16)
             } else {
                 0
@@ -7248,6 +7299,9 @@ fn calc_common_counter(infos: &[DiskInfo], read_quorum: usize) -> u64 {
 
 // list_path_raw
 
+#[cfg(test)]
+mod differential_tests;
+
 #[cfg(test)]
 mod test {
     use super::{
@@ -7926,6 +7980,117 @@ mod test {
         assert!(cancel.is_cancelled());
     }
 
+    #[tokio::test]
+    async fn list_path_gather_results_counts_common_prefixes_before_page_limit() {
+        for separator in ["/", "-"] {
+            for extra_object in [false, true] {
+                let prefix_name = format!("a{separator}");
+                let first = if separator == "/" {
+                    prefix_name.clone()
+                } else {
+                    format!("{prefix_name}first")
+                };
+                let mut input = vec![test_object_meta_entry(&first)];
+                if separator == "/" {
+                    input.push(test_dir_meta_entry(&prefix_name));
+                } else {
+                    input.push(test_object_meta_entry(&format!("{prefix_name}second")));
+                }
+                input.push(test_object_meta_entry("b"));
+                if extra_object {
+                    input.push(test_object_meta_entry("c"));
+                }
+                let (entry_tx, entry_rx) = mpsc::channel(input.len());
+                let (result_tx, mut result_rx) = mpsc::channel(1);
+                for entry in input {
+                    entry_tx.send(entry).await.expect("delimiter candidates should queue");
+                }
+                drop(entry_tx);
+                let state = gather_results(
+                    CancellationToken::new(),
+                    ListPathOptions {
+                        bucket: "bucket".to_owned(),
+                        separator: Some(separator.to_owned()),
+                        recursive: separator != "/",
+                        include_directories: separator == "/",
+                        limit: 3,
+                        ..Default::default()
+                    },
+                    entry_rx,
+                    result_tx,
+                )
+                .await
+                .expect("delimiter collection should succeed");
+                let result = result_rx.recv().await.expect("delimiter page should arrive");
+                let mut expected = vec![first, "b".to_owned()];
+                if extra_object {
+                    expected.push("c".to_owned());
+                }
+                assert_eq!(
+                    result
+                        .entries
+                        .expect("delimiter entries should exist")
+                        .entries()
+                        .into_iter()
+                        .map(|entry| entry.name.clone())
+                        .collect::<Vec<_>>(),
+                    expected,
+                    "delimiter={separator}, extra={extra_object}"
+                );
+                assert_eq!(result.err.is_none(), extra_object, "n=max must reach EOF; max+1 must retain lookahead");
+                assert_eq!(
+                    state,
+                    if extra_object {
+                        GatherResultsState::LimitReached
+                    } else {
+                        GatherResultsState::InputClosed
+                    }
+                );
+            }
+        }
+        for include_marker in [false, true] {
+            let (entry_tx, entry_rx) = mpsc::channel(2);
+            let (result_tx, mut result_rx) = mpsc::channel(1);
+            for name in ["a/child", "b"] {
+                entry_tx
+                    .send(test_object_meta_entry(name))
+                    .await
+                    .expect("marker candidates should queue");
+            }
+            drop(entry_tx);
+            gather_results(
+                CancellationToken::new(),
+                ListPathOptions {
+                    bucket: "bucket".to_owned(),
+                    separator: Some("/".to_owned()),
+                    recursive: true,
+                    include_directories: true,
+                    marker: Some("a/".to_owned()),
+                    include_marker,
+                    limit: 3,
+                    ..Default::default()
+                },
+                entry_rx,
+                result_tx,
+            )
+            .await
+            .expect("logical marker collection should succeed");
+            let entries = result_rx
+                .recv()
+                .await
+                .expect("logical marker page should arrive")
+                .entries
+                .expect("logical marker entries should exist");
+            let projected = ObjectInfo::from_meta_cache_entries_sorted_infos(&entries, "bucket", "", Some("/".to_owned())).await;
+            let names: Vec<_> = projected.into_iter().map(|object| object.name).collect();
+            assert_eq!(
+                names,
+                if include_marker { vec!["a/", "b"] } else { vec!["b"] },
+                "a folded prefix equal to the marker must follow include_marker={include_marker}"
+            );
+        }
+    }
+
     #[test]
     fn list_versions_pagination_scan_limit_boundaries() {
         for has_version_marker in [false, true] {
@@ -8977,6 +9142,29 @@ mod test {
         assert_eq!(result.info.objects[0].etag.as_deref(), Some("etag-z"));
         assert_eq!(result.info.prefixes, vec!["photos/2026/nested/".to_string()]);
         assert!(!result.info.is_truncated);
+
+        let resumed = list_objects_from_metadata_snapshot_candidates(
+            "bucket",
+            "photos/2026/",
+            Some("photos/2026/nested/"),
+            &Some("/".to_string()),
+            1,
+            &objects,
+        );
+        assert!(
+            resumed.info.prefixes.is_empty(),
+            "metadata-fast cursors must not replay an emitted prefix"
+        );
+        assert_eq!(
+            resumed
+                .info
+                .objects
+                .iter()
+                .map(|object| object.name.as_str())
+                .collect::<Vec<_>>(),
+            ["photos/2026/z.jpg"]
+        );
+        assert!(!resumed.info.is_truncated);
     }
 
     #[test]
@@ -10089,6 +10277,23 @@ mod test {
         assert_eq!(result.prefixes, vec!["photos/2026/archive/".to_string()]);
         assert!(result.is_truncated);
         assert_eq!(result.next_marker.as_deref(), Some("photos/2026/archive/"));
+
+        let resumed = list_objects_from_verified_index_candidates(
+            "photos/2026/",
+            result.next_marker.as_deref(),
+            &Some("/".to_string()),
+            1,
+            &candidates,
+            |key| async move { Ok(Some(test_live_object_info(&key, "live-etag"))) },
+        )
+        .await
+        .expect("verified prefix continuation should succeed");
+        assert!(resumed.prefixes.is_empty(), "verified cursors must not replay an emitted prefix");
+        assert_eq!(
+            resumed.objects.iter().map(|object| object.name.as_str()).collect::<Vec<_>>(),
+            ["photos/2026/d.jpg"]
+        );
+        assert!(!resumed.is_truncated);
     }
 
     #[tokio::test]

crates/ecstore/src/store/object.rs

@@ -985,6 +985,19 @@ impl PreparedGetObjectReader {
         reader.body_source = crate::object_api::GetObjectBodySource::HookMissed;
         Ok(ECStore::attach_read_lock_guard(reader, self.read_lock_guard))
     }
+
+    /// Open the prepared source with bounded copy prefetch and request-owned
+    /// cancellation, including cancellation during reader construction.
+    pub async fn into_reader_for_copy(self) -> Result<(GetObjectReader, tokio_util::sync::DropGuard)> {
+        let cancellation = tokio_util::sync::CancellationToken::new();
+        let guard = cancellation.clone().drop_guard();
+        let reader = crate::set_disk::with_get_object_read_cancellation(
+            cancellation,
+            crate::set_disk::with_get_object_read_policy(crate::set_disk::GetObjectReadPolicy::CopySource, self.into_reader()),
+        )
+        .await?;
+        Ok((reader, guard))
+    }
 }
 
 struct LockGuardedReader {
@@ -1025,8 +1038,23 @@ impl fmt::Display for ObjectLockDiagMode {
     }
 }
 
+enum ObjectLockOwner {
+    Read(rustfs_lock::NamespaceLockGuard),
+    Write(Arc<rustfs_lock::NamespaceLockGuard>),
+}
+
+impl ObjectLockOwner {
+    fn as_ref(&self) -> &rustfs_lock::NamespaceLockGuard {
+        match self {
+            Self::Read(guard) => guard,
+            Self::Write(guard) => guard,
+        }
+    }
+}
+
 pub(crate) struct ObjectLockDiagGuard {
-    guard: rustfs_lock::NamespaceLockGuard,
+    guard: ObjectLockOwner,
+    write_key: Option<(String, String)>,
     #[cfg(test)]
     test_namespace_lock_fence: Option<NamespaceLockFence>,
     enabled: bool,
@@ -1050,7 +1078,11 @@ impl ObjectLockDiagGuard {
         mode: ObjectLockDiagMode,
     ) -> Self {
         Self {
-            guard,
+            guard: match mode {
+                ObjectLockDiagMode::Read => ObjectLockOwner::Read(guard),
+                ObjectLockDiagMode::Write => ObjectLockOwner::Write(Arc::new(guard)),
+            },
+            write_key: None,
             #[cfg(test)]
             test_namespace_lock_fence: None,
             enabled,
@@ -1069,15 +1101,20 @@ impl ObjectLockDiagGuard {
         self
     }
 
+    fn with_write_key(mut self, bucket: &str, object: &str) -> Self {
+        self.write_key = Some((bucket.to_owned(), object.to_owned()));
+        self
+    }
+
     pub(crate) fn lock_lost_signal(&self) -> Option<Arc<rustfs_lock::distributed_lock::LockLostSignal>> {
-        match &self.guard {
+        match self.guard.as_ref() {
             rustfs_lock::NamespaceLockGuard::Standard(guard) => Some(guard.lock_lost()),
             rustfs_lock::NamespaceLockGuard::Fast(_) => None,
         }
     }
 
     pub(crate) fn is_lock_lost(&self) -> bool {
-        self.guard.is_lock_lost() || {
+        self.guard.as_ref().is_lock_lost() || {
             #[cfg(test)]
             {
                 self.test_namespace_lock_fence
@@ -1092,6 +1129,13 @@ impl ObjectLockDiagGuard {
     }
 
     pub(crate) fn add_namespace_lock_fence(&self, opts: &mut ObjectOptions) {
+        if let Some((bucket, object)) = self.write_key.as_ref()
+            && let ObjectLockOwner::Write(guard) = &self.guard
+        {
+            let no_lock = opts.no_lock;
+            opts.add_owned_write_lock(Arc::clone(guard), bucket, object);
+            opts.no_lock = no_lock;
+        }
         opts.ensure_namespace_lock_fence();
         if let Some(signal) = self.lock_lost_signal() {
             opts.add_namespace_lock_lost_signal(signal);
@@ -1288,7 +1332,7 @@ impl RestoreAcceptGuard {
     /// refresh lost quorum). Callers must check this before committing the
     /// restore-status write the guard exists to serialize.
     pub fn is_lock_lost(&self) -> bool {
-        self.0.guard.is_lock_lost()
+        self.0.guard.as_ref().is_lock_lost()
     }
 
     pub fn add_namespace_lock_fence(&self, opts: &mut ObjectOptions) {
@@ -1309,7 +1353,7 @@ impl RestoreWorkerGuard {
 
 impl Drop for ObjectLockDiagGuard {
     fn drop(&mut self) {
-        if !self.enabled || self.guard.is_released() {
+        if !self.enabled || self.guard.as_ref().is_released() {
             return;
         }
 
@@ -3016,7 +3060,8 @@ impl ECStore {
             owner,
             ObjectLockDiagMode::Write,
         )
-        .with_attempt(attempt))
+        .with_attempt(attempt)
+        .with_write_key(bucket, object))
     }
 
     async fn acquire_object_write_lock_if_needed(
@@ -3031,11 +3076,9 @@ impl ECStore {
         }
 
         let guard = self.acquire_object_write_lock(op, bucket, object).await?;
-        if let Some(signal) = guard.lock_lost_signal() {
-            opts.add_namespace_lock_lost_signal(signal);
-        }
-        opts.ensure_namespace_lock_fence();
+        guard.add_namespace_lock_fence(opts);
         opts.no_lock = true;
+        opts.write_completion = crate::object_api::WriteCompletion::TailDrained;
 
         Ok(Some(guard))
     }
@@ -3174,7 +3217,10 @@ impl ECStore {
         let store = Arc::clone(self);
         let write = async move {
             let object = "buckets/.scanner-pause-backlog.json";
-            let mut opts = ObjectOptions::default();
+            let mut opts = ObjectOptions {
+                write_completion: crate::object_api::WriteCompletion::TailDrained,
+                ..Default::default()
+            };
             // Match migration: fixed object namespace -> durable pool metadata ->
             // actual replica namespace. The replica need not be the hash-routed set.
             let object_guard = if store.single_pool() {
@@ -3714,15 +3760,18 @@ impl ECStore {
                     acquire_start.elapsed(),
                     diag_enabled,
                 );
-                guards.push(ObjectLockDiagGuard::new(
-                    guard,
-                    diag_enabled,
-                    op,
-                    diag_enabled.then(|| bucket.to_string()),
-                    diag_enabled.then(|| object.to_string()),
-                    owner,
-                    ObjectLockDiagMode::Write,
-                ));
+                guards.push(
+                    ObjectLockDiagGuard::new(
+                        guard,
+                        diag_enabled,
+                        op,
+                        diag_enabled.then(|| bucket.to_string()),
+                        diag_enabled.then(|| object.to_string()),
+                        owner,
+                        ObjectLockDiagMode::Write,
+                    )
+                    .with_write_key(bucket, object),
+                );
                 locked_sets.push(Arc::clone(set));
             }
         }
@@ -3813,7 +3862,8 @@ impl ECStore {
                 diag_enabled.then(|| object.to_string()),
                 owner,
                 ObjectLockDiagMode::Write,
-            );
+            )
+            .with_write_key(bucket, object);
             #[cfg(test)]
             let guard = {
                 let mut guard = guard;
@@ -4558,11 +4608,13 @@ impl ECStore {
                     shard_integrity_write_mode: dst_opts.shard_integrity_write_mode,
                     version_id: dst_opts.version_id.clone(),
                     no_lock: dst_opts.no_lock,
+                    write_completion: dst_opts.write_completion,
                     mod_time: dst_opts.mod_time,
                     http_preconditions: dst_opts.http_preconditions.clone(),
                     expected_current_version_id: dst_opts.expected_current_version_id.clone(),
                     expected_bucket_incarnation_id: dst_opts.expected_bucket_incarnation_id,
                     namespace_lock_fence: dst_opts.namespace_lock_fence.clone(),
+                    write_lock_context: dst_opts.write_lock_context.clone(),
                     bucket_lifecycle_lock_fence: dst_opts.bucket_lifecycle_lock_fence.clone(),
                     object_lock_config_snapshot: dst_opts.object_lock_config_snapshot.clone(),
                     quota_admission: dst_opts.quota_admission,
@@ -4601,11 +4653,13 @@ impl ECStore {
                         shard_integrity_write_mode: dst_opts.shard_integrity_write_mode,
                         version_id: dst_opts.version_id.clone(),
                         no_lock: dst_opts.no_lock,
+                        write_completion: dst_opts.write_completion,
                         mod_time: dst_opts.mod_time,
                         http_preconditions: dst_opts.http_preconditions.clone(),
                         expected_current_version_id: dst_opts.expected_current_version_id.clone(),
                         expected_bucket_incarnation_id: dst_opts.expected_bucket_incarnation_id,
                         namespace_lock_fence: dst_opts.namespace_lock_fence.clone(),
+                        write_lock_context: dst_opts.write_lock_context.clone(),
                         bucket_lifecycle_lock_fence: dst_opts.bucket_lifecycle_lock_fence.clone(),
                         object_lock_config_snapshot: dst_opts.object_lock_config_snapshot.clone(),
                         quota_admission: dst_opts.quota_admission,
@@ -4654,11 +4708,13 @@ impl ECStore {
             shard_integrity_write_mode: dst_opts.shard_integrity_write_mode,
             version_id: dst_opts.version_id.clone(),
             no_lock: dst_opts.no_lock,
+            write_completion: dst_opts.write_completion,
             mod_time: dst_opts.mod_time,
             http_preconditions: dst_opts.http_preconditions.clone(),
             expected_current_version_id: dst_opts.expected_current_version_id.clone(),
             expected_bucket_incarnation_id: dst_opts.expected_bucket_incarnation_id,
             namespace_lock_fence: dst_opts.namespace_lock_fence.clone(),
+            write_lock_context: dst_opts.write_lock_context.clone(),
             bucket_lifecycle_lock_fence: dst_opts.bucket_lifecycle_lock_fence.clone(),
             object_lock_config_snapshot: dst_opts.object_lock_config_snapshot.clone(),
             quota_admission: dst_opts.quota_admission,
@@ -8708,6 +8764,108 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn borrowed_copy_tail_drained_waits_for_rename_tail() {
+        use crate::object_api::{WriteCommitGuard, WriteCompletion};
+        use crate::set_disk::{ENV_RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE, rename_fanout_barrier};
+
+        temp_env::async_with_vars([(ENV_RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE, Some("true"))], async {
+            let ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
+            let (_dirs, original) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await;
+            let store = Arc::new(new_prepared_reader_test_store_with_ctx(&[original], ctx).await);
+            let bucket = RUSTFS_META_BUCKET;
+            let source = "copy-tail-drained-source";
+            let target = "copy-tail-drained-target";
+            let payload = vec![0x64; 4097];
+            let set = store.pools[0].get_disks_by_key(target);
+            let mut source_info = set
+                .put_object(
+                    bucket,
+                    source,
+                    &mut PutObjReader::from_vec(payload.clone()),
+                    &ObjectOptions {
+                        write_completion: WriteCompletion::TailDrained,
+                        ..Default::default()
+                    },
+                )
+                .await
+                .expect("seed copy source");
+            source_info.put_object_reader = Some(PutObjReader::from_vec(payload.clone()));
+
+            let barrier = rename_fanout_barrier::arm(target, 0, rename_fanout_barrier::PHASE_RENAME);
+            let rename_tasks = rename_fanout_barrier::observe_tasks(target);
+            let copy_store = Arc::clone(&store);
+            let mut copy = tokio::spawn(async move {
+                let lock = copy_store
+                    .new_ns_lock(bucket, target)
+                    .await
+                    .expect("target namespace wrapper");
+                let guard = WriteCommitGuard::acquire(&lock, get_lock_acquire_timeout())
+                    .await
+                    .expect("real Copy target namespace owner");
+                let mut options = ObjectOptions {
+                    write_completion: WriteCompletion::TailDrained,
+                    ..Default::default()
+                };
+                options.add_write_commit_guard(&guard);
+                copy_store
+                    .handle_copy_object(bucket, source, bucket, target, &mut source_info, &ObjectOptions::default(), &options)
+                    .await
+            });
+            let deadline = tokio::time::Instant::now() + Duration::from_secs(30);
+            tokio::time::timeout_at(deadline, barrier.wait_until_paused())
+                .await
+                .expect("copy reaches the paused rename checkpoint");
+            tokio::time::timeout_at(deadline, async {
+                loop {
+                    if let Ok(info) = set
+                        .get_object_info(
+                            bucket,
+                            target,
+                            &ObjectOptions {
+                                no_lock: true,
+                                ..Default::default()
+                            },
+                        )
+                        .await
+                        && info.size == 4097
+                        && rename_tasks.running() == 1
+                    {
+                        break;
+                    }
+                    tokio::task::yield_now().await;
+                }
+            })
+            .await
+            .expect("copy commits a metadata quorum while one rename remains paused");
+            assert!(
+                tokio::time::timeout(Duration::from_secs(1), &mut copy).await.is_err(),
+                "Copy TailDrained must remain pending after metadata quorum"
+            );
+            barrier.release();
+            let result = tokio::time::timeout_at(deadline, copy)
+                .await
+                .expect("copy drains its rename tail")
+                .expect("copy task completes")
+                .expect("copy succeeds");
+            assert_eq!(result.size, 4097);
+            assert_eq!(rename_tasks.running(), 0);
+            let mut reader = store
+                .get_object_reader(bucket, target, None, HeaderMap::new(), &ObjectOptions::default())
+                .await
+                .expect("read copied target");
+            let mut actual = Vec::new();
+            reader
+                .stream
+                .read_to_end(&mut actual)
+                .await
+                .expect("read complete copied bytes");
+            assert_eq!(actual, payload);
+        })
+        .await;
+    }
+
     #[tokio::test]
     #[serial_test::serial]
     async fn object_lock_diagnostics_report_historical_get_blocked_by_put_commit() {

crates/ecstore/tests/fixtures/list_namespace_keys.json

@@ -0,0 +1,14 @@
+[
+  "a",
+  "a/",
+  "a/b",
+  "a/b/child",
+  "a0",
+  "b",
+  "b/leaf",
+  "quote\"&<",
+  "space key",
+  "z",
+  "é",
+  "中/文"
+]

crates/filemeta/src/metacache.rs

@@ -618,6 +618,7 @@ impl MetaCacheEntries {
         }
 
         let mut dir_exists = 0;
+        let mut directory = None;
         let mut selected = None;
 
         params.candidates.clear();
@@ -633,7 +634,7 @@ impl MetaCacheEntries {
             }
             if entry.is_dir() {
                 dir_exists += 1;
-                selected = Some(entry.clone());
+                directory = Some(entry.clone());
                 debug!(entry = %entry.name, "metacache resolve observed directory candidate");
                 continue;
             }
@@ -664,6 +665,10 @@ impl MetaCacheEntries {
             }
         }
 
+        // A prefix and an explicit directory-marker object can share a name.
+        // Keep directory evidence separate so its arrival cannot displace an
+        // object that meets the metadata quorum.
+        let directory = directory.filter(|_| dir_exists >= params.dir_quorum);
         let Some(selected) = selected else {
             debug!(
                 bucket = %params.bucket,
@@ -671,19 +676,9 @@ impl MetaCacheEntries {
                 obj_quorum = params.obj_quorum,
                 "metacache resolve could not select any candidate entry"
             );
-            return None;
+            return directory;
         };
 
-        if selected.is_dir() && dir_exists >= params.dir_quorum {
-            debug!(
-                entry = %selected.name,
-                dir_exists,
-                dir_quorum = params.dir_quorum,
-                "metacache resolve selected directory candidate after quorum reconciliation"
-            );
-            return Some(selected);
-        }
-
         // If we would never be able to reach the required object quorum.
         if objs_valid < params.obj_quorum {
             debug!(
@@ -691,7 +686,7 @@ impl MetaCacheEntries {
                 obj_quorum = params.obj_quorum,
                 "metacache resolve did not have enough valid object candidates to satisfy quorum"
             );
-            return None;
+            return directory;
         }
 
         if objs_agree == objs_valid {
@@ -724,7 +719,7 @@ impl MetaCacheEntries {
 
         let Some(cached) = selected.cached else {
             debug!(selected = %selected.name, "metacache resolve could not merge because selected candidate had no cached metadata");
-            return None;
+            return directory;
         };
 
         let versions = if enforce_write_quorum {
@@ -744,7 +739,7 @@ impl MetaCacheEntries {
                 requested_versions = params.requested_versions,
                 "metacache resolve produced no merged versions after reconciliation"
             );
-            return None;
+            return directory;
         }
 
         let merged_cached = FileMeta {
@@ -761,7 +756,7 @@ impl MetaCacheEntries {
                     error = ?e,
                     "metacache resolve failed to marshal merged metadata entry"
                 );
-                return None;
+                return directory;
             }
         };
 
@@ -2534,6 +2529,57 @@ mod tests {
         assert!(resolved.is_none());
     }
 
+    #[test]
+    fn resolve_directory_marker_and_prefix_preserve_quorum_in_both_orders() {
+        let modified = OffsetDateTime::from_unix_timestamp(1_705_312_300).expect("valid timestamp");
+        let mut object = metacache_entry_with_erasure(modified, "marker-etag", 2, 2);
+        object.name = "folder/".to_string();
+        let directory = metacache_dir_entry("folder/");
+        for enforce_write_quorum in [false, true] {
+            for (object_copies, directory_copies) in [(3, 1), (3, 3), (2, 3), (0, 3), (2, 2)] {
+                for reverse in [false, true] {
+                    let mut entries = vec![Some(object.clone()); object_copies];
+                    entries.extend(vec![Some(directory.clone()); directory_copies]);
+                    if reverse {
+                        entries.reverse();
+                    }
+                    let entries = MetaCacheEntries(entries);
+                    let params = MetadataResolutionParams {
+                        dir_quorum: 3,
+                        obj_quorum: 3,
+                        requested_versions: 1,
+                        bucket: "bucket".to_string(),
+                        strict: true,
+                        ..Default::default()
+                    };
+                    let resolved = if enforce_write_quorum {
+                        entries.resolve_with_write_quorum(params)
+                    } else {
+                        entries.resolve(params)
+                    };
+                    let context = format!(
+                        "objects={object_copies}, directories={directory_copies}, reverse={reverse}, write={enforce_write_quorum}"
+                    );
+                    if object_copies >= 3 {
+                        let resolved = resolved.expect("metadata at quorum must shadow a same-named prefix");
+                        assert!(resolved.is_object(), "{context}");
+                        assert_eq!(
+                            resolved
+                                .to_fileinfo("bucket")
+                                .expect("marker metadata should decode")
+                                .mod_time,
+                            Some(modified)
+                        );
+                    } else if directory_copies >= 3 {
+                        assert!(resolved.expect("prefix at quorum must remain a fallback").is_dir(), "{context}");
+                    } else {
+                        assert!(resolved.is_none(), "{context}");
+                    }
+                }
+            }
+        }
+    }
+
     fn build_hashmap_cache(update_size: usize) -> Arc<Cache<HashMap<usize, usize>>> {
         let generation = Arc::new(AtomicUsize::new(0));
         Arc::new(Cache::new(

crates/heal/Cargo.toml

@@ -73,6 +73,7 @@ hotpath-cpu = [
 [dependencies]
 hotpath.workspace = true
 rustfs-config = { workspace = true }
+rustfs-data-usage = { workspace = true }
 rustfs-concurrency = { workspace = true }
 rustfs-ecstore = { workspace = true }
 rustfs-lock = { workspace = true }

crates/heal/src/error.rs

@@ -177,6 +177,21 @@ impl Error {
         }
     }
 
+    /// Retry an unproven missing object only within the per-object budget.
+    /// Generic task retries retain the narrower classification above.
+    pub(crate) fn is_recoverable_object_heal(&self) -> bool {
+        self.is_recoverable_heal()
+            || matches!(
+                self,
+                Error::Storage(
+                    EcstoreError::FileNotFound
+                        | EcstoreError::FileVersionNotFound
+                        | EcstoreError::ObjectNotFound(_, _)
+                        | EcstoreError::VersionNotFound(_, _, _)
+                ) | Error::Disk(DiskError::FileNotFound | DiskError::FileVersionNotFound)
+            )
+    }
+
     pub(crate) fn dangling_delete_retry_not_before(&self) -> Option<std::time::SystemTime> {
         let after = match self {
             Self::Storage(error) => error.dangling_delete_retry_after(),
@@ -395,6 +410,26 @@ mod tests {
         assert!(Error::Storage(EcstoreError::FaultyRemoteDisk).is_recoverable_heal());
     }
 
+    #[test]
+    fn typed_missing_objects_retry_only_within_the_object_budget() {
+        for error in [
+            Error::Storage(EcstoreError::FileNotFound),
+            Error::Storage(EcstoreError::FileVersionNotFound),
+            Error::Storage(EcstoreError::ObjectNotFound("bucket".to_owned(), "object".to_owned())),
+            Error::Storage(EcstoreError::VersionNotFound(
+                "bucket".to_owned(),
+                "object".to_owned(),
+                "version".to_owned(),
+            )),
+            Error::Disk(DiskError::FileNotFound),
+            Error::Disk(DiskError::FileVersionNotFound),
+        ] {
+            assert!(error.is_recoverable_object_heal(), "{error:?}");
+            assert!(!error.is_recoverable_heal(), "missing objects must not requeue the whole task: {error:?}");
+        }
+        assert!(!Error::other("file not found").is_recoverable_object_heal());
+    }
+
     #[test]
     fn task_timeout_is_terminal() {
         assert!(!Error::TaskTimeout.is_recoverable_heal());

crates/heal/src/heal/erasure_healer/admin.rs

@@ -196,7 +196,7 @@ pub(super) async fn heal_object(
             return ((size, Err(error.unwrap_or(Error::TaskCancelled))), None);
         }
         if let Some(error) = error.as_ref()
-            && error.is_recoverable_heal()
+            && error.is_recoverable_object_heal()
             && failures < MAX_BUCKET_OBJECT_HEAL_RETRIES
         {
             failures += 1;
@@ -224,7 +224,7 @@ pub(super) async fn heal_object(
             None => Ok(!matches!(disposition, HealObjectDisposition::AuthoritativelyAbsent)),
             Some(error) => {
                 failures += 1;
-                if error.is_recoverable_heal() {
+                if error.is_recoverable_object_heal() {
                     disposition = HealObjectDisposition::Deferred {
                         reason: if error.is_dangling_delete_grace() {
                             HealDeferredReason::DanglingDeleteGrace

crates/heal/src/heal/erasure_healer/resume_loop_tests/admin_outcome.rs

@@ -335,9 +335,23 @@ async fn admin_erasure_dispositions_do_not_invent_success_and_retries_count_once
         .expect_err("unhealed objects must retain the failed task status");
     let outcome = task.get_outcome().await;
     let c = &outcome.counters;
-    assert_eq!((c.processed, c.healed, c.unchanged, c.failed, c.skipped, c.unknown), (5, 0, 2, 1, 2, 1));
-    assert_eq!(c.attempt_failures, 5);
+    assert_eq!((c.processed, c.healed, c.unchanged, c.failed, c.skipped, c.unknown), (5, 0, 2, 0, 3, 1));
+    assert_eq!(c.attempt_failures, 8);
+    assert_eq!(storage.calls().iter().filter(|(name, _)| name == "gone").count(), 4);
     assert_eq!(storage.calls().iter().filter(|(name, _)| name == "offline").count(), 4);
+    let gone = outcome
+        .objects
+        .iter()
+        .find(|item| item.identity.object == "gone")
+        .expect("missing version outcome");
+    assert_eq!(
+        gone.disposition,
+        HealObjectDisposition::Deferred {
+            reason: HealDeferredReason::TransientExistenceCheck,
+            retry_not_before: None,
+        },
+        "exhausting retries cannot certify the missing version as absent"
+    );
     assert_partition(&outcome);
 }
 

crates/heal/src/heal/manager/tests/root_recovery.rs

@@ -728,6 +728,90 @@ async fn root_recovery_orphan_report_never_commits_or_retires_pending_work() {
     assert_eq!(manager.root_recovery.pending().await.expect("pending survives GC").len(), 1);
 }
 
+#[cfg(unix)]
+#[tokio::test]
+async fn root_recovery_lock_contention_preserves_a_single_durable_owner() {
+    let (_first_temp, first_disk) = recovery_disk().await;
+    let (_second_temp, second_disk) = recovery_disk().await;
+    let (first_disk, second_disk) = ordered_recovery_disks(first_disk, second_disk);
+    let lock_owner = |disk: &DiskStore| {
+        let path = std::path::Path::new(&disk.endpoint().to_string())
+            .join(RUSTFS_META_BUCKET)
+            .join(".rustfs-cas.lock");
+        let file = std::fs::OpenOptions::new()
+            .create(true)
+            .truncate(false)
+            .read(true)
+            .write(true)
+            .open(path)
+            .expect("open control-file lock");
+        file.lock().expect("hold control-file lock");
+        file
+    };
+    let first_lock = lock_owner(&first_disk);
+    let manager = recovery_manager(vec![first_disk.clone(), second_disk.clone()]);
+    let mut request = admin_request(HealType::Object {
+        bucket: "bucket".to_string(),
+        object: "object".to_string(),
+        version_id: None,
+    });
+    let receipt = manager
+        .submit_heal_request_with_receipt(request.clone())
+        .await
+        .expect("uncontended disk should own the durable heal intent");
+    assert_eq!(receipt.result, HealAdmissionResult::Accepted);
+    let path = format!("root-heal-{}.json", request.id);
+    assert!(matches!(
+        first_disk.read_all(RUSTFS_META_BUCKET, &path).await,
+        Err(DiskError::FileNotFound)
+    ));
+    let original = second_disk.read_all(RUSTFS_META_BUCKET, &path).await.expect("durable intent");
+
+    drop(first_lock);
+    let _second_lock = lock_owner(&second_disk);
+    request.retry_attempts = 1;
+    manager
+        .root_recovery
+        .persist(&request)
+        .await
+        .expect_err("existing ownership must not migrate when its lock is contended");
+    assert!(matches!(
+        first_disk.read_all(RUSTFS_META_BUCKET, &path).await,
+        Err(DiskError::FileNotFound)
+    ));
+    assert_eq!(
+        second_disk
+            .read_all(RUSTFS_META_BUCKET, &path)
+            .await
+            .expect("unchanged owner"),
+        original
+    );
+    let pending = manager.root_recovery.pending().await.expect("single durable owner");
+    assert_eq!(pending.len(), 1);
+    assert_eq!(pending[0].id, request.id);
+    assert_eq!(pending[0].retry_attempts, 0);
+
+    let _first_lock = lock_owner(&first_disk);
+    let blocked = admin_request(HealType::Object {
+        bucket: "bucket".to_string(),
+        object: "blocked".to_string(),
+        version_id: None,
+    });
+    manager
+        .root_recovery
+        .persist(&blocked)
+        .await
+        .expect_err("all owners remain contended");
+    let blocked_path = format!("root-heal-{}.json", blocked.id);
+    for disk in [&first_disk, &second_disk] {
+        assert!(matches!(
+            disk.read_all(RUSTFS_META_BUCKET, &blocked_path).await,
+            Err(DiskError::FileNotFound)
+        ));
+    }
+    assert_eq!(manager.root_recovery.pending().await.expect("original intent remains").len(), 1);
+}
+
 #[cfg(unix)]
 #[tokio::test]
 async fn root_recovery_new_intent_skips_prepublication_read_only_owner() {

crates/heal/src/heal/task/heal_bucket.rs

@@ -762,7 +762,10 @@ impl HealTask {
                                 error = %err,
                                 "Heal bucket object repair skipped due to transient metadata error"
                             );
-                        } else if !age_exhausted && err.is_recoverable_heal() && retry_attempt < MAX_BUCKET_OBJECT_HEAL_RETRIES {
+                        } else if !age_exhausted
+                            && err.is_recoverable_object_heal()
+                            && retry_attempt < MAX_BUCKET_OBJECT_HEAL_RETRIES
+                        {
                             terminal_outcome = false;
                             debug!(
                                 target: "rustfs::heal::task",
@@ -782,13 +785,13 @@ impl HealTask {
                                 inline_retry = Some(item);
                             }
                         } else {
-                            disposition = HealObjectDisposition::Failed(if age_exhausted || err.is_recoverable_heal() {
+                            disposition = HealObjectDisposition::Failed(if age_exhausted || err.is_recoverable_object_heal() {
                                 HealFailureClass::RetryExhausted
                             } else {
                                 HealFailureClass::Permanent
                             });
                             telemetry_unknown |= !increment_counter(&mut failed);
-                            if age_exhausted || err.is_recoverable_heal() {
+                            if age_exhausted || err.is_recoverable_object_heal() {
                                 retryable_failed = retryable_failed.saturating_add(1);
                             } else {
                                 permanent_failed = permanent_failed.saturating_add(1);

crates/heal/src/heal/task/heal_object.rs

@@ -468,7 +468,9 @@ impl HealTask {
         {
             Ok((result, error)) => {
                 if let Some(e) = error {
-                    if self.skip_scanner_synthetic_object_dir_missing(bucket, object, &e).await {
+                    if self.skip_missing_usage_observation(bucket, object, version_id, &e).await
+                        || self.skip_scanner_synthetic_object_dir_missing(bucket, object, &e).await
+                    {
                         return Ok(());
                     }
 
@@ -513,7 +515,9 @@ impl HealTask {
             Err(Error::TaskCancelled) => Err(Error::TaskCancelled),
             Err(Error::TaskTimeout) => Err(Error::TaskTimeout),
             Err(e) => {
-                if self.skip_scanner_synthetic_object_dir_missing(bucket, object, &e).await {
+                if self.skip_missing_usage_observation(bucket, object, version_id, &e).await
+                    || self.skip_scanner_synthetic_object_dir_missing(bucket, object, &e).await
+                {
                     return Ok(());
                 }
 
@@ -536,6 +540,45 @@ impl HealTask {
         }
     }
 
+    async fn skip_missing_usage_observation(&self, bucket: &str, object: &str, version_id: Option<&str>, err: &Error) -> bool {
+        // Usage publication may delete an obsolete observation after a degraded
+        // GET has queued read repair. Its absence is not lost user data. Keep
+        // durable MRF completion on its separate, proof-bearing storage path.
+        if self.source != HealRequestSource::ReadRepair
+            || version_id.is_some()
+            || bucket != RUSTFS_META_BUCKET
+            || object
+                .strip_prefix(BUCKET_META_PREFIX)
+                .and_then(|suffix| suffix.strip_prefix('/'))
+                != Some(rustfs_data_usage::DATA_USAGE_OBSERVED_OBJECT_NAME)
+            || !matches!(
+                err,
+                Error::Disk(DiskError::FileNotFound)
+                    | Error::Storage(EcstoreError::FileNotFound | EcstoreError::ObjectNotFound(_, _))
+            )
+        {
+            return false;
+        }
+
+        debug!(
+            target: "rustfs::heal::task",
+            event = EVENT_HEAL_OBJECT_RESULT,
+            component = LOG_COMPONENT_HEAL,
+            subsystem = LOG_SUBSYSTEM_OBJECT,
+            task_id = %self.id,
+            bucket,
+            object,
+            source = self.source.as_str(),
+            result = "usage_observation_missing",
+            "Heal skipped an absent usage observation"
+        );
+        let mut progress = self.progress.write().await;
+        progress.set_current_object(Some(format!("skipped: {bucket}/{object}")));
+        progress.update_object_progress(1, 0, 0, 1, 0);
+        progress.update_stage(4, 4);
+        true
+    }
+
     /// Durable MRF partial writes may complete only with an incarnation-fenced storage proof.
     async fn heal_mrf_partial_write_object(&self, bucket: &str, object: &str, version_id: Option<&str>) -> Result<()> {
         let heal_opts = HealOpts {

crates/heal/src/heal/task/tests.rs

@@ -18,6 +18,7 @@ use crate::heal::storage::HealStorageObjectResult;
 
 mod concurrent_delete;
 mod deferred_retry;
+mod usage_observation;
 
 mod canonical_outcome {
     use super::*;
@@ -1901,6 +1902,7 @@ fn replacement_identity(
 
 #[derive(Clone)]
 enum MockHealObjectOutcome {
+    MissingObject { outer: bool },
     MissingVersion,
     PermissionDenied,
     RetryableLock,
@@ -2058,6 +2060,14 @@ impl HealStorageAPI for MockStorage {
             .and_then(VecDeque::pop_front)
         {
             return match outcome {
+                MockHealObjectOutcome::MissingObject { outer } => {
+                    let error = Error::Storage(EcstoreError::FileNotFound);
+                    if outer {
+                        Err(error)
+                    } else {
+                        Ok((HealResultItem::default(), Some(error)))
+                    }
+                }
                 MockHealObjectOutcome::MissingVersion => {
                     Ok((HealResultItem::default(), Some(Error::Storage(EcstoreError::FileVersionNotFound))))
                 }
@@ -2118,6 +2128,14 @@ impl HealStorageAPI for MockStorage {
         }
         if let Some(outcome) = self.heal_object_outcome.lock().unwrap().take() {
             return match outcome {
+                MockHealObjectOutcome::MissingObject { outer } => {
+                    let error = Error::Storage(EcstoreError::FileNotFound);
+                    if outer {
+                        Err(error)
+                    } else {
+                        Ok((HealResultItem::default(), Some(error)))
+                    }
+                }
                 MockHealObjectOutcome::MissingVersion => {
                     Ok((HealResultItem::default(), Some(Error::Storage(EcstoreError::FileVersionNotFound))))
                 }

crates/heal/src/heal/task/tests/concurrent_delete.rs

@@ -123,7 +123,7 @@ async fn admin_dry_run_can_observe_without_bucket_identity() {
 #[tokio::test(start_paused = true)]
 async fn admin_traversal_never_converts_unproven_errors_to_absence() {
     for (error, class) in [
-        (MockHealObjectOutcome::MissingVersion, HealFailureClass::Permanent),
+        (MockHealObjectOutcome::MissingVersion, HealFailureClass::RetryExhausted),
         (MockHealObjectOutcome::PermissionDenied, HealFailureClass::Permanent),
         (MockHealObjectOutcome::ErrOther("file not found"), HealFailureClass::Permanent),
         (MockHealObjectOutcome::RetryableReadQuorum, HealFailureClass::RetryExhausted),
@@ -147,3 +147,73 @@ async fn admin_traversal_never_converts_unproven_errors_to_absence() {
         assert_eq!(outcome.objects[0].disposition, HealObjectDisposition::Failed(class));
     }
 }
+
+#[tokio::test(start_paused = true)]
+async fn admin_traversal_retries_unproven_absence_until_storage_certifies_it() {
+    let incarnation = Uuid::new_v4();
+    let receipt = object_receipt("object-a", None, HealObjectDisposition::AuthoritativelyAbsent, incarnation);
+    let storage = Arc::new(MockStorage {
+        bucket_incarnation_id: Mutex::new(Some(incarnation)),
+        heal_object_outcomes: Mutex::new(HashMap::from([(
+            "object-a".to_string(),
+            VecDeque::from([MockHealObjectOutcome::MissingVersion]),
+        )])),
+        heal_object_receipts: Mutex::new(HashMap::from([("object-a".to_string(), VecDeque::from([receipt.clone(), receipt]))])),
+        ..Default::default()
+    });
+    let task = admin_traversal(HealType::Cluster, storage.clone(), false);
+    task.execute()
+        .await
+        .expect("unproven absence must use the existing retry budget");
+    let outcome = task.get_outcome().await;
+    let object = outcome
+        .objects
+        .iter()
+        .find(|item| item.identity.object == "object-a")
+        .expect("object outcome");
+    assert_eq!(object.disposition, HealObjectDisposition::AuthoritativelyAbsent);
+    assert_eq!(object.identity.bucket_incarnation_id, Some(incarnation));
+    assert_eq!(outcome.counters.attempt_failures, 1, "the errored receipt cannot certify absence");
+    assert_eq!(outcome.counters.failed, 0);
+    assert_eq!(
+        storage.heal_object_calls.lock().expect("object calls").as_slice(),
+        ["object-a", "object-b", "object-a"]
+    );
+}
+
+#[tokio::test(start_paused = true)]
+async fn admin_traversal_unproven_absence_exhausts_retries_without_positive_outcome() {
+    let incarnation = Uuid::new_v4();
+    let attempts = MAX_BUCKET_OBJECT_HEAL_RETRIES + 1;
+    let storage = Arc::new(MockStorage {
+        bucket_incarnation_id: Mutex::new(Some(incarnation)),
+        heal_object_outcomes: Mutex::new(HashMap::from([(
+            "object-a".to_string(),
+            (0..attempts).map(|_| MockHealObjectOutcome::MissingVersion).collect(),
+        )])),
+        heal_object_receipts: Mutex::new(HashMap::from([(
+            "object-a".to_string(),
+            (0..attempts)
+                .map(|_| object_receipt("object-a", None, HealObjectDisposition::AuthoritativelyAbsent, incarnation))
+                .collect(),
+        )])),
+        ..Default::default()
+    });
+    let task = admin_traversal(HealType::Cluster, storage.clone(), false);
+    task.execute().await.expect_err("unproven absence cannot finish successfully");
+    let outcome = task.get_outcome().await;
+    let object = outcome
+        .objects
+        .iter()
+        .find(|item| item.identity.object == "object-a")
+        .expect("object outcome");
+    assert_eq!(object.disposition, HealObjectDisposition::Failed(HealFailureClass::RetryExhausted));
+    assert_eq!(outcome.counters.attempt_failures, u64::from(attempts));
+    assert_eq!((outcome.counters.failed, outcome.counters.healed), (1, 0));
+    let calls = storage.heal_object_calls.lock().expect("object calls");
+    assert_eq!(
+        calls.iter().filter(|object| object.as_str() == "object-a").count(),
+        usize::try_from(attempts).expect("attempts fit")
+    );
+    assert_eq!(calls.iter().filter(|object| object.as_str() == "object-b").count(), 1);
+}

crates/heal/src/heal/task/tests/usage_observation.rs

@@ -0,0 +1,137 @@
+// Copyright 2026 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use super::*;
+
+const OBSERVATION: &str = "buckets/.usage.observed.json";
+
+fn task(bucket: &str, object: &str, source: HealRequestSource, version: Option<&str>, storage: Arc<MockStorage>) -> HealTask {
+    let mut request = HealRequest::new(
+        HealType::Object {
+            bucket: bucket.to_string(),
+            object: object.to_string(),
+            version_id: version.map(str::to_string),
+        },
+        HealOptions {
+            recreate_missing: true,
+            ..Default::default()
+        },
+        HealPriority::Low,
+    );
+    request.source = source;
+    HealTask::from_request(request, storage)
+}
+
+fn missing_storage(error: MockHealObjectOutcome) -> Arc<MockStorage> {
+    Arc::new(MockStorage {
+        object_exists: Mutex::new(Some(false)),
+        heal_object_outcome: Mutex::new(Some(error)),
+        ..Default::default()
+    })
+}
+
+#[tokio::test]
+async fn missing_usage_observation_completes_without_claiming_repair_or_absence_proof() {
+    for outer in [false, true] {
+        let storage = missing_storage(MockHealObjectOutcome::MissingObject { outer });
+        let task = task(RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::ReadRepair, None, storage.clone());
+        task.execute()
+            .await
+            .expect("a removed observation is a normal read-repair skip");
+        assert_eq!(task.get_status().await, HealTaskStatus::Completed);
+        let outcome = task.get_outcome().await;
+        assert_eq!(outcome.execution, HealExecutionOutcome::Completed);
+        assert_eq!(outcome.counters.failed, 0);
+        assert_eq!(outcome.counters.skipped, 1);
+        assert_eq!(outcome.objects[0].disposition, HealObjectDisposition::Unknown);
+        let progress = task.get_progress().await;
+        assert_eq!(progress.objects_healed, 0);
+        assert_eq!(progress.skipped_objects, 1);
+        let calls = storage.object_heal_opts.lock().expect("heal calls");
+        assert_eq!(calls.len(), 1);
+        assert!(calls[0].recreate, "the test must reach the actual recreation branch");
+    }
+}
+
+#[tokio::test]
+async fn missing_usage_observation_exclusion_is_limited_to_the_exact_read_repair_target() {
+    for (bucket, object, source, version) in [
+        ("user-bucket", OBSERVATION, HealRequestSource::ReadRepair, None),
+        (RUSTFS_META_BUCKET, "buckets/.usage.json", HealRequestSource::ReadRepair, None),
+        (
+            RUSTFS_META_BUCKET,
+            "buckets/.usage.observed.json.bkp",
+            HealRequestSource::ReadRepair,
+            None,
+        ),
+        (
+            RUSTFS_META_BUCKET,
+            "buckets/nested/.usage.observed.json",
+            HealRequestSource::ReadRepair,
+            None,
+        ),
+        (RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::Admin, None),
+        (RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::Internal, None),
+        (RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::Scanner, None),
+        (RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::AutoHeal, None),
+        (RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::ReadRepair, Some("version-a")),
+    ] {
+        let storage = missing_storage(MockHealObjectOutcome::MissingObject { outer: false });
+        let task = task(bucket, object, source, version, storage);
+        task.execute()
+            .await
+            .expect_err("other requests must retain their missing-object failure");
+        assert_eq!(task.get_outcome().await.counters.failed, 1, "{bucket}/{object}: {source:?}");
+    }
+}
+
+#[tokio::test]
+async fn usage_observation_does_not_hide_non_absence_errors() {
+    for error in [
+        MockHealObjectOutcome::PermissionDenied,
+        MockHealObjectOutcome::OkWithReadQuorum,
+        MockHealObjectOutcome::FaultyStorageDisk(false),
+        MockHealObjectOutcome::MissingVersion,
+        MockHealObjectOutcome::ErrOther("File not found"),
+    ] {
+        let storage = missing_storage(error);
+        let task = task(RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::ReadRepair, None, storage);
+        task.execute().await.expect_err("only a typed object absence may be skipped");
+        assert_eq!(task.get_outcome().await.counters.failed, 1);
+    }
+}
+
+#[tokio::test]
+async fn existing_usage_observation_still_reaches_storage_repair() {
+    let storage = Arc::new(MockStorage::default());
+    let task = task(RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::ReadRepair, None, storage.clone());
+    task.execute().await.expect("existing observation remains repairable");
+    {
+        let calls = storage.object_heal_opts.lock().expect("heal calls");
+        assert_eq!(calls.len(), 1);
+        assert!(calls[0].read_repair);
+    }
+    assert_eq!(task.get_progress().await.skipped_objects, 0);
+}
+
+#[tokio::test]
+async fn missing_usage_observation_mrf_still_requires_storage_proof() {
+    let storage = missing_storage(MockHealObjectOutcome::MissingObject { outer: false });
+    *storage.bucket_incarnation_id.lock().expect("bucket incarnation") = Some(Uuid::new_v4());
+    let task = task(RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::Mrf, None, storage);
+    task.execute()
+        .await
+        .expect_err("MRF responsibility cannot retire on a path-name exception");
+    assert_eq!(task.get_outcome().await.counters.failed, 1);
+}

crates/heal/tests/mrf_partial_write_test.rs

@@ -918,10 +918,19 @@ async fn partial_write_sigkill_replay_scenario(protected: bool) {
             "replayed responsibility must be released only after verified repair"
         );
     } else {
+        // A repair can finish before the next legacy check parks its responsibility.
         assert!(
             wait_until(|| async {
                 let snapshot = manager.operations_snapshot().await;
-                snapshot.queue_length == 0 && snapshot.active_tasks == 0
+                snapshot.queue_length == 0
+                    && snapshot.active_tasks == 0
+                    && mrf_queue::list_legacy_responsibilities(None, 32).await.is_ok_and(|state| {
+                        state.responsibilities.iter().any(|entry| {
+                            entry.bucket == "partial-crash"
+                                && entry.object == "crash.bin"
+                                && entry.status == "held_unverified_legacy"
+                        })
+                    })
             })
             .await,
             "legacy replay attempts must finish"

crates/heal/tests/usage_observation_read_repair_test.rs

@@ -0,0 +1,150 @@
+// Copyright 2026 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#![recursion_limit = "256"]
+
+use http::HeaderMap;
+use rustfs_heal::heal::{
+    channel::HealChannelProcessor,
+    manager::{HealConfig, HealManager},
+    outcome::HealExecutionOutcome,
+    storage::{ECStoreHealStorage, HealStorageAPI},
+    task::HealTaskStatus,
+};
+use rustfs_heal_contracts::heal_channel::{HealChannelCommand, HealRequestSource, init_heal_channel};
+use rustfs_test_utils::TestECStoreEnv;
+use std::{sync::Arc, time::Duration};
+
+mod storage_api;
+use storage_api::integration::{ObjectIO, ObjectOperations, ObjectOptions, PutObjReader, RUSTFS_META_BUCKET, WriteCompletion};
+
+// Owns the process-global heal receiver in this integration-test binary.
+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+async fn usage_observation_read_repair_survives_snapshot_cleanup() {
+    let temp = tempfile::tempdir().expect("test storage directory");
+    let env = TestECStoreEnv::builder().base_dir(temp.path()).build().await;
+    let bucket = RUSTFS_META_BUCKET;
+    let object = "buckets/.usage.observed.json";
+    let payload = br#"{"usageSnapshotConverged":false}"#.to_vec();
+    env.ecstore
+        .put_object(
+            bucket,
+            object,
+            &mut PutObjReader::from_vec(payload.clone()),
+            &ObjectOptions {
+                write_completion: WriteCompletion::TailDrained,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("write the observation to all disks");
+    for disk in &env.disk_paths {
+        assert!(disk.join(bucket).join(object).join("xl.meta").exists());
+    }
+
+    let mut receiver = init_heal_channel().expect("sole heal receiver in this test binary");
+    tokio::fs::remove_dir_all(env.disk_paths[0].join(bucket).join(object))
+        .await
+        .expect("remove one observation replica while retaining read quorum");
+    let mut reader = env
+        .ecstore
+        .get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
+        .await
+        .expect("degraded observation remains readable");
+    assert_eq!(reader.read_all().await.expect("read complete observation"), payload);
+    drop(reader);
+
+    let (request, response_tx) = tokio::time::timeout(Duration::from_secs(10), async {
+        loop {
+            if let HealChannelCommand::Start { request, response_tx } = receiver.recv().await.expect("heal channel open") {
+                assert_eq!(request.bucket, bucket);
+                assert_eq!(request.object_prefix.as_deref(), Some(object));
+                break (request, response_tx);
+            }
+        }
+    })
+    .await
+    .expect("the real degraded GET must emit a repair request");
+    assert_eq!(request.source, HealRequestSource::ReadRepair);
+    assert_eq!(request.recreate_missing, Some(true));
+
+    // Authoritative usage publication deletes the obsolete observation with
+    // these prefix options. Delay admission until that deletion has finished.
+    env.ecstore
+        .delete_object(
+            bucket,
+            object,
+            ObjectOptions {
+                delete_prefix: true,
+                delete_prefix_object: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("clean up the obsolete observation before repair runs");
+    let storage = Arc::new(ECStoreHealStorage::new(env.ecstore.clone()));
+    assert!(
+        !storage
+            .object_exists(bucket, object)
+            .await
+            .expect("check observation absence")
+    );
+
+    let manager = Arc::new(HealManager::new(
+        storage,
+        Some(HealConfig {
+            enable_auto_heal: false,
+            mainline_throttle_enable: false,
+            heal_interval: Duration::from_millis(10),
+            ..Default::default()
+        }),
+    ));
+    manager.start().await.expect("start the real heal scheduler");
+    let processor = HealChannelProcessor::new(manager.clone());
+    let receipt = processor
+        .execute_start_request(request)
+        .await
+        .expect("admit the captured read repair");
+    assert!(receipt.result.is_admitted());
+    response_tx
+        .send(Ok(receipt.result))
+        .expect("acknowledge the GET's repair admission");
+
+    let terminal = tokio::time::timeout(Duration::from_secs(30), async {
+        loop {
+            let report = manager
+                .get_task_report(&receipt.task_id)
+                .await
+                .expect("query scheduled repair");
+            if matches!(report.status, HealTaskStatus::Completed | HealTaskStatus::Failed { .. }) {
+                break report;
+            }
+            tokio::time::sleep(Duration::from_millis(10)).await;
+        }
+    })
+    .await;
+    manager.stop().await.expect("stop the heal scheduler");
+    let report = terminal.expect("repair must reach a terminal result");
+    assert_eq!(report.status, HealTaskStatus::Completed, "{report:?}");
+    let outcome = report.outcome.expect("scheduler retains the repair outcome");
+    assert_eq!(outcome.execution, HealExecutionOutcome::Completed);
+    assert_eq!(outcome.counters.failed, 0);
+    assert_eq!(outcome.counters.skipped, 1);
+    let progress = report.progress.expect("scheduler retains repair progress");
+    assert_eq!(progress.objects_healed, 0);
+    assert_eq!(progress.skipped_objects, 1);
+    for disk in &env.disk_paths {
+        assert!(!disk.join(bucket).join(object).join("xl.meta").exists());
+    }
+}

crates/iam/src/federation/model.rs

@@ -47,6 +47,10 @@ pub struct FederatedAuthorization {
     pub provider_id: String,
     pub claims: FederatedClaims,
     pub policies: Vec<String>,
+    /// Subset of `policies` derived only from the groups claim (including merged roles claim
+    /// values); these may be ignored when no policy of that name exists. Everything else in
+    /// `policies` must resolve.
+    pub group_claim_policies: Vec<String>,
     pub groups: Vec<String>,
     pub roles_claim_key: Option<String>,
     pub roles: Vec<String>,
@@ -126,6 +130,7 @@ mod tests {
                 ..claims("", "", "subject")
             },
             policies,
+            group_claim_policies: Vec::new(),
             groups,
             roles_claim_key: None,
             roles: Vec::new(),

crates/iam/src/federation/oidc/claims.rs

@@ -37,6 +37,7 @@ fn string_list_claim(claims: &OidcClaims, claim_name: &str) -> Vec<String> {
 
 pub(super) fn authorization(oidc: &OidcSys, provider_id: String, claims: OidcClaims) -> FederatedAuthorization {
     let (policies, groups) = oidc.map_claims_to_policies(&provider_id, &claims);
+    let group_claim_policies = oidc.group_claim_policy_names(&provider_id, &claims);
     let roles_claim_key = roles_claim_key(oidc, &provider_id);
     let roles = roles_claim_key
         .as_deref()
@@ -53,6 +54,7 @@ pub(super) fn authorization(oidc: &OidcSys, provider_id: String, claims: OidcCla
             raw: claims.raw,
         },
         policies,
+        group_claim_policies,
         groups,
         roles_claim_key,
         roles,
@@ -135,6 +137,7 @@ mod tests {
         assert_eq!(authorization.claims.groups, ["source-ops", "source-developers", "source-ops"]);
         assert_eq!(authorization.claims.raw, raw);
         assert_eq!(authorization.policies, ["mapped-source-developers", "mapped-source-ops"]);
+        assert_eq!(authorization.group_claim_policies, ["mapped-source-developers", "mapped-source-ops"]);
         assert_eq!(authorization.groups, ["source-developers", "source-ops"]);
         assert_eq!(authorization.roles_claim_key.as_deref(), Some("roles"));
         assert_eq!(authorization.roles, ["reader", "admin"]);