feat(capture): atomically migrate legacy Created history

Co-authored-by: Hauser <housemecn@gmail.com>

.agents/references/implementation.md

@@ -58,6 +58,13 @@ are repository-relative. Read only the relevant sections during read-only review
 - Attach error context once where it is actionable. Do not erase typed errors
   below aggregation or quorum layers.
 
+## Rust Boundaries
+
+- Production `unwrap`/`expect` must follow a type guarantee or checked invariant; explain only non-obvious guarantees.
+- Public APIs do not return `Result<_, String>`. Library APIs use domain errors unless intentional erasure is part of the boundary contract; expose a stored inner error through `Error::source()`.
+- Numeric conversions must not silently truncate. Use fallible conversions for untrusted values; validate finiteness, sign, and range before float-to-integer conversion. Clamp/saturate only when required by the domain.
+- Do not add crate-root `#![allow(dead_code)]`.
+
 ## Naming
 
 Use Rust API naming: `SCREAMING_SNAKE_CASE` constants/statics, `snake_case`

.agents/skills/adversarial-validation/references/concurrency-durability.md

@@ -2,8 +2,12 @@
 
 - For every changed lock, enumerate overlapping lock sets and construct the
   ABBA interleaving. Multiple-lock order must be documented and consistent.
-- Mark guard lifetimes and every `.await`, disk, and RPC call inside them.
-  Estimate contention and timeout behavior under concurrent requests.
+- Mark guard lifetimes and every `.await`, disk,
+  and RPC call; estimate contention/timeouts. Tokio read/write guards across
+  `.await` need bounded hold time; long-lived reads can wedge writers (#4195).
+  Keep `std::sync::Mutex` holds brief and never across `.await`.
+- Use direct `fetch_*` for unconditional atomic read-modify-write and
+  `compare_exchange` for conditional updates.
 - Object commits remain fenced if the distributed lock is lost after shard
   writes and before metadata rename.
 - For write/rename changes, trace `write tmp -> sync tmp -> rename -> sync parent

.agents/skills/adversarial-validation/references/correctness.md

@@ -15,6 +15,8 @@ Attack the changed behavior, not every subsystem in the repository.
   bytes and length.
 - For multipart/object commits, fail before/after rename and cleanup; committed
   data must remain readable and pre-commit cleanup must not destroy parts.
+- Bound recursion over untrusted or persisted input; handle corrupted/cyclic
+  tree and cache traversals safely.
 - For version/index ordering, test `len - 1`, `len`, equal timestamps, missing
   versions, and deterministic tie-breaking.
 - For directory-object behavior, trace `__XLDIR__` at the store layer; branches

.agents/skills/adversarial-validation/references/test-coverage.md

@@ -19,7 +19,8 @@
   serialization is required.
 - Internal metadata tests assert both RustFS and MinIO keys, not only read-back
   through a helper that prefers one key.
-- Boundary companions are distinct coverage: `n == max` vs `max + 1`, and
+- Avoid duplicate tests of the same production path and poison-value class.
+  Boundary companions are distinct coverage: `n == max` vs `max + 1`, and
   absent vs empty vs nil UUID.
 - A focused test proves only the targets/features it builds. Add compilation or
   Clippy only for uncovered changed targets.

.agents/skills/code-change-verification/SKILL.md

@@ -1,51 +0,0 @@
----
-name: code-change-verification
-description: Review a commit, PR, or merged patch when the user requests ordinary code-change verification. Do not combine with adversarial-validation; use that skill instead for explicitly adversarial, substantial, or high-risk RustFS reviews.
----
-
-# Code Change Verification
-
-Use this skill for an ordinary requested review. If the root policy or user calls
-for adversarial validation, use `adversarial-validation` instead of running both.
-
-## Core Workflow
-
-### 1) Scope and assumptions
-- Derive the change source, target branch, and relevant runtime/version from the
-  supplied diff and metadata. Ask only when missing context could change the verdict.
-- Focus only on requested scope; avoid reviewing unrelated files.
-
-### 2) Risk map
-- Prioritize in this order:
-  - Data correctness and user-visible behavior
-  - API/contract compatibility
-  - Security and authz/authn boundaries
-  - Concurrency and lifecycle correctness
-  - Performance and resource usage
-- Give higher priority to stateful paths, migration logic, defaults, and error handling.
-
-### 3) Evidence-based inspection
-- Read each modified hunk with neighboring context.
-- Trace call paths and call-site expectations.
-- Check for:
-  - invariant breaks and missing guards
-  - unchecked assumptions and null/empty/error-path handling
-  - stale tests, fixtures, and configs
-  - hidden coupling to shared helpers/constants/features
-- Apply root `AGENTS.md`'s finding standard: try to disprove a candidate before
-  reporting it. Mention an unresolved question only when it could materially
-  change the verdict; do not fill the report with speculative possibilities.
-
-#### Rust-specific checks
-
-For changed Rust behavior, use the matching sections of [rust-code-quality](../rust-code-quality/SKILL.md). Reuse checks already performed by the selected review workflow. Comment-only or formatting-only Rust diffs do not require the full Rust checklist. Carry its P0–P3 ratings over unchanged and use this skill's output format.
-
-### 4) Findings-first output
-- Order supported findings by P0–P3 severity; preserve the Rust ratings above.
-  Include `path:line`, the failure and impact, a focused fix, and its validation.
-- If no supported issues remain, state `No findings` with the reviewed scope and
-  any material verification limitation. Do not append optional improvements to
-  make a clean review look productive.
-
-Close after the required review. Recommend additional verification only for an
-identified unresolved risk or required gate; reuse evidence for unchanged code.

.agents/skills/code-change-verification/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "Code Change Verification"
-  short_description: "Prioritize risks and verify code changes before merge."
-  default_prompt: "Use $code-change-verification for an ordinary requested diff review with prioritized findings."

.agents/skills/issue-triage/SKILL.md

@@ -1,112 +1,13 @@
 ---
 name: issue-triage
-description: Assess whether a GitHub issue is fixed, needs implementation, or can be closed by checking related work and current code. Use for issue completion/triage requests. Status questions are read-only; comment, close, or change labels only when the conversation authorizes that action.
+description: Verify issue completion against current implementation and merged work. Use for issue triage; comment, close, or change labels only when authorized.
 ---
-
 # Issue Triage
 
-Use this skill when the user provides a GitHub issue URL and asks "can this be closed?", "is this already implemented?", "check completion status", or similar triage questions.
+- Resolve the issue repository separately from the implementation repository: `rustfs/backlog` tracks work in `rustfs/rustfs`. Pass `--repo` explicitly to GitHub queries.
+- Read the issue body, relevant comments, linked PRs, checklist, and sub-issues with `gh issue view`. Search explicit links, the full issue URL, qualified references, and subject keywords; an empty search result does not prove implementation is absent.
+- Fetch the implementation base and inspect its current code. For each merged candidate, verify `git merge-base --is-ancestor <merge-commit> <remote>/<base>`; a title, commit message, or merged state alone is insufficient. A local checkout may be stale or on another branch.
+- Verify every checklist/sub-issue before declaring completion. For batches, paginate the entire requested issue scope, exclude PR entries, and filter by author only when requested.
+- Recommend closing only when all requested behavior is present or evidence shows the issue is superseded. Otherwise name what remains. A status request is read-only; reuse existing authority for comments/closing/labels, and follow [Git and PR rules](../../references/pull-requests.md) before posting. Use Chinese for `rustfs/backlog`, existing repository labels only, and a closing comment naming the verified behavior and PRs.
 
-## Workflow
-
-### 1. Fetch issue context
-
-```bash
-gh issue view <N> --repo <owner/repo> --json title,body,state,comments,labels,updatedAt
-```
-
-Read the issue body to understand what was requested. Extract:
-- The specific feature/fix/behavior described.
-- Any linked PRs or commits mentioned in the body or comments.
-- Any checklist items or sub-issues.
-
-Resolve the issue repository and implementation repository separately (for example, `rustfs/backlog` tracks work in `rustfs/rustfs`). Pass the implementation repository explicitly to PR queries; the current checkout may belong to another repository.
-
-### 2. Search for related work
-
-Search git history for commits referencing the issue:
-```bash
-git log --oneline --all --grep="<N>" | head -30
-```
-
-Search for related PRs:
-```bash
-gh pr list --repo <implementation-repo> --search "<issue-url>" --state all --json number,title,state,mergedAt
-```
-
-Also search qualified issue references and subject keywords; for same-repository
-issues, include `#<N>`. Follow explicit links even without a text match. A search
-page with no match does not prove the work is absent.
-
-If the issue mentions specific PRs, check their status:
-```bash
-gh pr view <PR_N> --repo <implementation-repo> --json state,mergedAt,title,mergeCommit,baseRefName
-```
-
-### 3. Verify implementation
-
-Fetch the implementation repository's current base branch. For each merged candidate, verify its merge commit is present and inspect the current code for the claimed behavior; a commit message match alone is not proof:
-```bash
-git fetch <implementation-remote> <base-branch>
-git merge-base --is-ancestor <merge-commit> <implementation-remote>/<base-branch>
-```
-
-If the issue describes a specific defect, inspect the fetched base's code rather than assuming the current checkout contains it:
-```bash
-git show <implementation-remote>/<base-branch>:crates/<relevant>/src/<file>.rs
-```
-
-For issues with checklists, verify each item individually. If sub-items are tracked as separate issues, check those too:
-```bash
-gh issue view <SUB_N> --repo <owner/repo> --json state
-```
-
-### 4. Determine verdict
-
-- **All items fixed and merged**: Recommend closing; name the verified PRs and behavior.
-- **Some items fixed, some remaining**: Keep open; report each remaining item.
-- **Not yet implemented**: Keep open; report what remains.
-- **Superseded or no longer relevant**: Recommend closing with evidence.
-
-### 5. Take action
-
-For a status-only request, return the assessment without GitHub writes. If commenting, closing, or label edits are authorized, perform only those actions; do not ask again for authority already given. Prepare the final assessment before asking for any missing authority. Write `rustfs/backlog` issue content in Chinese.
-
-Close with comment:
-```bash
-gh issue close <N> --repo <owner/repo> --comment "<body>"
-```
-
-Comment without closing:
-```bash
-gh issue comment <N> --repo <owner/repo> --body-file /tmp/triage.md
-```
-
-Update labels only when label changes are authorized, using existing repository labels; never add tool-specific labels:
-```bash
-gh issue edit <N> --repo <owner/repo> --add-label "<existing-label>"
-```
-
-Always use `--body-file` for multiline content, never inline `--body`.
-
-### 6. Handle multi-issue batches
-
-When the user asks to check multiple issues (e.g., "check all issues by user X" or "scan backlog for closable issues"):
-1. List the full requested scope with pagination (for example `gh api --paginate 'repos/<repo>/issues?state=open&per_page=100'`, excluding entries with `pull_request`). Add an author filter only when the user requested one; the default page/limit is not evidence that all issues were checked.
-2. For each issue, run steps 1-5 above.
-3. Report a summary table of all triaged issues with verdicts.
-
-## Report
-
-Identify the issue and current state, verified implementation/PR evidence,
-remaining items, verdict, and action actually taken. Use a table for batches;
-a single issue does not require a heading for each field. Follow step 4's
-verdicts without repeating the assessment in another template.
-
-## Notes
-
-- The user may ask in Chinese ("是否可以关闭", "检查完成情况"); respond in the same language.
-- When closing, always include a summary of what was fixed and which PRs resolved it — this creates a useful audit trail.
-- For issues in `rustfs/backlog`, use `--repo rustfs/backlog`.
-- For issues in `rustfs/rustfs`, use `--repo rustfs/rustfs`.
-- If the issue has sub-issues (GitHub sub-issues API), check each one's state before declaring the parent complete.
+Report the issue, current state, implementation evidence, remaining work, verdict, and action actually taken. Use a table for batches. Write multiline comments via `--body-file`; when closing, post the prepared comment before closing without an inline multiline body.

.agents/skills/pr-review/SKILL.md

@@ -1,93 +1,20 @@
 ---
 name: pr-review
-description: Review a GitHub PR from a URL or number using its actual base/head and risk-appropriate code review. Use when the user asks for a PR review, not a status lookup or PR wording edit. Publish a review only when authorized; delegation and monitoring follow the requested scope and root AGENTS.md.
+description: Review a GitHub PR using its exact base/head and repository risk policy. Use for requested PR reviews; publish or fix only within conversation authorization.
 ---
-
 # PR Review
 
-Use this skill for PR context and review delivery. An ordinary review request is read-only unless the conversation also authorizes posting or fixes. Reuse that authorization without asking again; prepare the review before requesting any missing publication approval.
+1. Resolve the PR repository and remote; the current checkout may belong elsewhere. Read the PR purpose and linked issues, then fetch the actual base and PR head:
 
-## Prerequisites
+   ```bash
+   gh pr view <N> --repo <owner/repo> --json title,author,state,body,baseRefName,headRefName,baseRefOid,headRefOid
+   git fetch <repo-remote> <baseRefName> refs/pull/<N>/head
+   git diff <baseRefOid>...<headRefOid>
+   ```
 
-- Follow root `AGENTS.md`; classify risk with the [review policy](../../references/adversarial-validation.md) and consult relevant [change-style and boundary rules](../../references/implementation.md).
-- Select `code-change-verification` for ordinary review or `adversarial-validation` for explicitly adversarial, substantial, or high-risk review; do not run both on the same diff.
-
-## Workflow
-
-### 1. Gather PR context
-
-```bash
-gh pr view <N> --repo <owner/repo> --json title,author,state,body,additions,deletions,changedFiles,commits,baseRefName,headRefName,baseRefOid,headRefOid
-gh pr diff <N> --repo <owner/repo> --name-only
-```
-
-Read the PR body and linked issues to understand the change's purpose. If the PR references an issue, fetch that too:
-```bash
-gh issue view <ISSUE> --repo <issue-owner/repo> --json title,body,state
-```
-
-### 2. Fetch the diff and classify the change
-
-```bash
-git fetch <repo-remote> <baseRefName> refs/pull/<N>/head
-git diff <baseRefOid>...<headRefOid> --stat
-```
-
-Resolve `<repo-remote>` to the PR repository; do not assume the current checkout's `origin` or `main` matches. Record the exact base/head used. If either moved during fetching, refresh the snapshot before reviewing. Classify using the repository review policy; instruction changes that affect agent execution are mechanical, not exempt.
-
-### 3. Review the changed behavior
-
-Group files by functional area to trace callers and invariants. Use the root risk tier's review shape and only matching lenses. File count does not authorize delegation. When delegation is explicitly authorized, high-risk/substantial reviews use exactly two independent reviewers with the applicable lenses split between them; otherwise use two fresh sequential passes. Reviewers do not spawn further agents.
-
-Findings need a concrete failure scenario with `file:line`; a null verdict briefly names the relevant probes. Reuse existing evidence and choose local checks from the final diff under the root verification policy.
-
-### 4. Check CI status
-
-```bash
-gh pr checks <N> --repo <owner/repo>
-```
-
-Investigate a failed check when it bears on a finding or the user requested CI diagnosis/merge readiness:
-```bash
-gh run view --repo <owner/repo> --log-failed --job=<JOB_ID>
-```
-
-Use current evidence to distinguish pre-existing, flaky, and PR-caused failures. Do not classify them by guesswork or turn a code-only review into unrelated CI repair.
-
-### 5. Synthesize findings
-
-Report the PR, reviewed base/head, and risk tier, then summarize the assessment.
-Use the selected review's P0–P3 ratings and root finding standard: supported
-findings with `file:line`, failure scenario, and fix, or `No findings`.
-State the observed check status, including pending or unavailable checks, and
-the verdict (`APPROVE`, `REQUEST_CHANGES`, or `COMMENT`). Do not infer a pass
-from missing checks or add style nits to populate a clean review.
-
-### 6. Post the review
-
-Only when posting is authorized, write the review body to a temp file and post via CLI. Refresh the PR head first; if it changed, review the delta and update the verdict before posting:
-```bash
-# Request changes
-gh pr review <N> --repo <owner/repo> --request-changes --body-file /tmp/pr_review.md
-
-# Approve
-gh pr review <N> --repo <owner/repo> --approve --body-file /tmp/pr_review.md
-
-# Comment only (no verdict)
-gh pr review <N> --repo <owner/repo> --comment --body-file /tmp/pr_review.md
-```
-
-For authorized inline comments, use [the submission example](references/posting.md).
-
-Always use `--body-file` or `--input`, never inline multiline `--body`.
-
-### 7. Handle follow-up
-
-Follow the [PR lifecycle](../../references/pull-requests.md) and any explicit monitoring request. For follow-up, fetch the new head and compare the recorded reviewed SHA with the new SHA; revisit affected callers and findings. Never use an unfetched `origin/pull/<N>/head` ref as evidence. Update the posted review or resolve addressed threads only within existing authorization.
-
-## Notes
-
-- The user may ask for review in Chinese; respond in the same language but keep the review body in English per AGENTS.md rules.
-- When the user asks for "多角色对抗 review", run the full adversarial validation protocol — this skill's step 3 covers that.
-- If the PR is from a fork, check `maintainerCanModify` before attempting to push fixes.
-- For very large PRs, batch the review by functional area while keeping the same bounded review shape.
+   Record both SHAs; refresh if either moved during fetching. Follow relevant [implementation boundaries](../../references/implementation.md).
+2. Apply the [risk policy](../../references/adversarial-validation.md). Ordinary reviews use the root finding standard directly. For explicit adversarial, substantial, or high-risk reviews, use [the domain probes](../adversarial-validation/SKILL.md). File count does not authorize delegation; use exactly two independent reviewers only when authorized, otherwise two fresh sequential passes for that risk tier. Reviewers do not spawn further agents.
+3. Check `gh pr checks <N> --repo <owner/repo>`. Investigate failures when relevant to a finding or a requested CI/readiness diagnosis. Distinguish pending, unavailable, pre-existing, flaky, and PR-caused states using evidence; a code review does not authorize unrelated repairs.
+4. Report the reviewed SHAs, risk tier, supported findings or `No findings`, material verification limits, check state, and verdict. P0/P1 findings block approval. Severity: P0 = demonstrated data loss, security breach, remote crash, or deadlock; P1 = correctness, compatibility, or material hot-path regression; P2 = a concrete maintainability defect; P3 = optional style, only when requested.
+5. When posting is authorized, follow [Git and PR rules](../../references/pull-requests.md), recheck the remote head, and review any delta before submitting. Use `gh pr review` with `--approve`, `--request-changes`, or `--comment` and `--body-file`; use [the API example](references/posting.md) for authorized inline comments. Reuse existing authorization.
+6. Follow the repository PR lifecycle and explicit monitoring scope. Fetch new heads before comparing to the reviewed SHA; never rely on an unfetched `origin/pull/<N>/head`. Push fixes or resolve threads only when authorized; check `maintainerCanModify` before a fork push.

.agents/skills/rust-code-quality/SKILL.md

@@ -1,116 +0,0 @@
----
-name: rust-code-quality
-description: Run a focused Rust quality review when the user requests one or a selected review workflow needs Rust-specific checks for changed behavior. Do not auto-load for every implementation edit, comment-only or formatting-only Rust diff, or repeat an already completed review.
----
-
-# Rust Code Quality Gate
-
-Use this skill for a dedicated Rust review to cover rules that `cargo clippy`
-does not catch.
-
-Search matches and checklist items are candidates, not findings. Apply the root
-finding standard; distinguish a demonstrated bug, an explicit rule violation,
-and an optional preference. P2/P3 suggestions do not need to be invented or
-included in an otherwise clean correctness review.
-
-## Quick Start
-
-1. Identify changed `.rs` files.
-2. Run the matching candidate searches on changed files.
-3. Apply the manual checklist sections whose behavior the diff touches.
-4. Report or rebut every finding with evidence; P0/P1 findings block approval.
-   Fix them when implementation is authorized; a read-only review reports them.
-
-## Automated Checks
-
-Use these searches to find candidates in changed `.rs` files. Inspect syntax,
-`#[cfg(test)]` scope, and the changed hunk before reporting a finding; text
-filters do not reliably distinguish production code from tests.
-
-```bash
-# 1. unwrap/expect candidates
-rg -n '\.unwrap\(\)|\.expect\(' <changed-files>
-
-# 2. Silent type truncation via `as` cast
-rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
-
-# 3. String as error type
-rg -n 'Result<.*String>' <changed-files>
-
-# 4. Box<dyn Error> in public APIs
-rg -n 'Box<dyn.*Error' <changed-files>
-
-# 5. println/eprintln in production
-rg -n 'println!|eprintln!' <changed-files>
-
-# 6. Ordering::Relaxed usage (verify each is intentional)
-rg -n 'Ordering::Relaxed' <changed-files>
-
-# 7. Default substituted for a possibly-required value (judge each: is the value optional by domain?)
-rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
-```
-
-## Manual Review Checklist
-
-For the Rust diff under review, verify:
-
-### Error Handling
-- [ ] Every production `unwrap()` or `expect()` is infallible by type or a checked invariant; explain only non-obvious invariants, using an existing type, a useful `expect` message, or a concise comment
-- [ ] No `Result<_, String>` in public API signatures
-- [ ] Public library APIs use domain errors unless deliberate error erasure at a boundary is part of the contract
-- [ ] `Error::source()` is overridden when inner error is stored
-- [ ] Error messages are actionable without exposing secret input
-
-### Type Safety
-- [ ] No silent `as` truncation (negative→unsigned, large→small)
-- [ ] Fallible numeric conversions use `TryFrom`/`try_into()` and return a typed error; clamp or saturate only when the domain explicitly requires it
-- [ ] Floating-point to integer conversion validates finiteness, sign, and range before conversion
-
-### Concurrency
-- [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
-- [ ] No `tokio::sync` lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
-- [ ] Atomic read-modify-write uses the direct `fetch_*` operation when possible; use `compare_exchange` only for conditional updates
-- [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
-
-### Memory and Performance
-- [ ] On an identified hot path, report cloning or allocation only with a concrete per-request/per-object cost or benchmark signal
-- [ ] Prefer borrowing, moving, `Bytes`/`Arc`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs
-
-### Recursion Safety
-- [ ] Recursion over untrusted, persisted, or otherwise unbounded input has a depth limit or uses iterative traversal
-- [ ] Tree/cache traversals handle corrupted/cyclic input safely
-
-### Testing
-- [ ] Tests have an observable failure criterion; delegated assertions, `#[should_panic]`, snapshot/property checks, and meaningful `Result` failures do not need a redundant `assert!`
-- [ ] Use `expect` only when its message improves failure diagnosis; do not add boilerplate to self-evident test setup
-- [ ] Test volume and line count are never treated as production-code growth
-
-### Serde
-- [ ] Structs from untrusted input reject unknown fields where the compatibility contract permits; otherwise validate security-critical fields explicitly and test the supported input shape
-- [ ] `#[serde(default)]` not used on security-critical fields without validation
-
-### Code Hygiene
-- [ ] No `#![allow(dead_code)]` at crate root
-- [ ] No camelCase statics or Hungarian notation
-- [ ] New string literals don't duplicate existing constants
-
-### Reuse and Necessity
-- [ ] No new helper duplicates `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, a relevant direct dependency, or plain std/tokio behavior; reused helpers match the call site's semantics
-- [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
-- [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
-- [ ] Comments avoid narration and change history while completely stating non-obvious lock, `SAFETY`, durability, compatibility, and unwrap invariants
-- [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
-
-## Severity Classification
-
-- **P0 (Block merge)**: demonstrated data loss, security breach, remote crash, or deadlock
-- **P1 (Must fix)**: concrete correctness, compatibility, or material hot-path regression
-- **P2 (Should fix)**: avoidable duplication or maintainability issue with a concrete simpler replacement
-- **P3 (Nice to fix)**: local style or clarity issue with no behavioral risk
-
-## Output Template
-
-Use the calling review's output format. For a standalone review, report supported
-findings with severity, location, impact, fix, and validation, or `No findings`.
-Include only material unverified checks. Candidate counts are not a quality
-metric and do not need a separate scan report.

.agents/skills/rustfs-logging-governance/SKILL.md

@@ -1,34 +0,0 @@
----
-name: rustfs-logging-governance
-description: Add or review RustFS `tracing` events with the repository field shape, level policy, privacy boundaries, and guardrails. Use when a change adds or edits a tracing macro/instrumentation site or the logging guardrail script.
----
-
-# RustFS Logging Governance
-
-Apply this skill only to changed logging sites; do not turn a local log edit into
-a broad logging cleanup.
-
-## Workflow
-
-1. Read the changed function/module context and classify the site as lifecycle,
-   request/hot path, fallback, external fetch, or summary.
-2. Match neighboring structured events and reuse existing `EVENT_*`,
-   `LOG_COMPONENT_*`, and `LOG_SUBSYSTEM_*` constants.
-3. Put stable fields first (`event`, `component`, `subsystem`, `state`/`result`,
-   then context) and a short label last.
-4. Select the level by operational meaning:
-   - `error`: behavior/security-affecting failure;
-   - `warn`: degraded/fallback/operator-actionable state;
-   - `info`: low-frequency lifecycle/mode change;
-   - `debug`: targeted diagnostics;
-   - `trace`: repetitive request/object/shard success paths.
-5. Never log secrets, tokens, auth headers, credential payloads, raw
-   attacker-controlled bodies, or merged config dumps. Error strings and
-   `Debug` output are log surfaces too.
-6. Prefer one aggregate summary over inventories or startup banners.
-7. Run `./scripts/check_logging_guardrails.sh` and the checks selected by root
-   `AGENTS.md`.
-
-Read [logging-governance.md](references/logging-governance.md) only for a broad
-logging audit, event-model migration, or guardrail expansion. Ordinary single-
-site edits do not require the full workspace scope map.

.agents/skills/rustfs-logging-governance/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "RustFS Logging Governance"
-  short_description: "Standardize RustFS logs with structured events and guardrails."
-  default_prompt: "Use $rustfs-logging-governance to standardize or review RustFS logging, reduce noise, and update guardrails."

.agents/skills/rustfs-release-publish/SKILL.md

@@ -1,165 +1,42 @@
 ---
 name: rustfs-release-publish
-description: "Run the RustFS console gate, source version bump, preview validation, human confirmation, final-tag publication, and post-release installation and website updates. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
+description: Publish a RustFS version through Console, release-branch CI, preview acceptance, human confirmation, and final publication. Use only for an explicit release request.
 ---
-# RustFS Release Publish (preview-validated pipeline)
+# RustFS Release
 
-This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (invoked here with the authorized commit/push/PR scope) with a mandatory preview-tag validation loop before the final tag is published.
+Require the exact final SemVer target before version edits or publication. If unspecified, inspect current releases/tags and ask with concrete candidates; do not guess the channel or bump. A named bump type resolves to an exact version that still needs confirmation. Report the confirmed target verbatim; resolve any later mismatch before proceeding. Independent read-only preflight may continue.
 
-The binary reports its build tag (`build::TAG` via shadow_rs; `SHORT_VERSION` in
-`rustfs/src/config/cli.rs`), and `build.yml` derives asset names and preview
-classification from that tag. Cargo.toml supplies only the no-tag fallback.
-Preview and final tags must therefore share the validated source commit;
-their tag-dependent version and asset names differ. The channel and cleanup
-constraints are defined once under Preview tag naming and Hard rules below.
+## Invariants
 
-Pipeline shape:
+- Cargo stores `<target>`, never `-preview.N`. Before publication, installation defaults retain the previous available deliverable. Final and preview binaries embed their own tag names, so version/asset names differ despite identical source.
+- Use annotated tags without `v`. Preview names are exactly `<target>-preview.<digits>`; choose the next unused iteration after fetching tags. Preview Releases are prereleases, never Latest; publish only versioned assets/exact Docker tags, never latest assets, R2, Helm, moving Docker channels, or installation/banner updates.
+- `release/<target>` owns the candidate. Retain its history, backport only reviewed fixes, and never merge main wholesale or force-push it. Pin `PREVIEW_HASH` to its exact commit with complete passing CI. Preview and final tags both point there; never tag a later main/branch HEAD or insert another bump.
+- Phase failures block dependents. Keep preview Releases until final publication and notes generation finish; CI removes Releases while retaining tags. Never move an existing preview or final tag.
 
-```
-check console main against its latest Release
-  -> if ahead: publish console -> wait for Release asset + latest API
-  -> bump Cargo.toml and Cargo.lock to <target> -> merge
-  -> tag <preview-tag> at that commit -> CI green
-  -> verify preview Release assets -> run binary locally + console checks
-  -> validate with latest rc client
-  -> report preview acceptance results -> STOP for explicit human confirmation
-  -> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
-  -> CI deletes the <target>-preview.N Releases (tags kept)
-  -> verify published images -> publish Helm chart from the validated source
-  -> update installation references on main -> update rustfs.com announcement
-```
+## Read by Phase
 
-On preview validation failure: fix lands on main via normal PR (Cargo versions are already at `<target>`, no new source bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2. Installation references remain on the previous published deliverable throughout preview validation.
+Read only the current phase's procedure; retain completed evidence across turns. Follow root verification, Git/PR, and authorization rules.
 
-## Required inputs
+| Phase | Procedure |
+|---|---|
+| 0 | Clean tree, fetch main/tags, verify GitHub access, then complete the mandatory [Console release gate](references/console-gate.md). A green Console build alone is insufficient. |
+| 1–2 | [Prepare/reuse the release candidate and run exact-SHA branch CI](references/release-candidate.md). Source version changes use `rustfs-release-version-bump` with authorized delivery scope. |
+| 3 | [Verify the preview build, assets, channels, and notes](references/preview-release.md). |
+| 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. |
+| Confirmation | Complete the gate below and end the turn. |
+| 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. |
+| 7 | [Maintain milestones, installation references, and the website](references/post-release-updates.md) only after complete publication. |
 
-- Final target version, for example `1.0.0-beta.10`.
-- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` after `git fetch --tags`).
+## Human Confirmation
 
-If the target version is missing or ambiguous, collect the current release/tag baseline and ask before version edits or publication. Continue independent read-only preflight while the answer is pending (see the semver gate below).
+After Phases 3–5 pass, recheck that the release branch still equals `PREVIEW_HASH`. Report the target, branch, preview tag/hash, branch CI run, preview Release URL, Console result, and rc matrix. Explicitly ask whether to publish the final tag and end the turn without creating/pushing it.
 
-## Semver gate — resolve the target before version edits or publication
+Only a new affirmative user reply to that exact report authorizes Phase 6. The original release request, earlier confirmation, silence, or automated follow-up does not count. Confirmation binds the target, preview tag, and hash; a changed value or repeated/failed acceptance cycle invalidates it. Revalidate and obtain fresh confirmation before continuing.
 
-Versions follow [SemVer 2.0.0](https://semver.org/). Precedence reminder:
+## Recovery
 
-```
-1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0 < 1.0.1 < 1.1.0 < 2.0.0
-```
+- Before the final tag exists, land fixes on main, backport to the candidate, verify target versions, and restart branch CI/acceptance. Use a new preview iteration once a preview was tagged; main may advance independently.
+- Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases.
+- If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting candidate branches or preview tags requires an explicit decision.
 
-Numeric prerelease identifiers compare numerically (`beta.9 < beta.10`), not lexically — see [semver.org spec item 11](https://semver.org/#spec-item-11). Preview tags are internal validation tags layered on top of the target's prerelease channel — they are never themselves a deliverable version and never appear in version files.
-
-Rules:
-
-- A request like "发个版" / "release the next version" without an exact version string is ALWAYS ambiguous. Derive the current latest tag (`git tag --sort=-v:refname | head`), then ask the user to choose with concrete candidates, e.g. from `1.0.0-beta.10`: next prerelease `1.0.0-beta.11`, promote to `1.0.0-rc.1`, promote to stable `1.0.0`. Never guess between these — they have very different meanings (channel promotion vs. iteration) and different CI classification consequences.
-- After a stable `X.Y.Z` exists, the next version must state which component bumps: patch `X.Y.(Z+1)` for fixes only, minor `X.(Y+1).0` for backward-compatible features, major `(X+1).0.0` for breaking changes. If the user names a bump type but not a number, compute it from the latest stable tag and echo the exact resulting version back for confirmation.
-- Echo the final confirmed version string verbatim in your first status report; every later phase must use exactly that string. If at any point the user's wording and the confirmed version diverge, stop and re-confirm.
-
-## Preview tag naming
-
-- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
-- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
-- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
-- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
-
-## Hard rules
-
-- Before preview, bump only Cargo.toml (workspace package and internal dependency versions) and the corresponding workspace members in Cargo.lock, directly to `<target>`. Keep README installation examples, flake.nix, Chart.yaml, and rustfs.spec on the previous published deliverable until Phase 7. Never write a `-preview.N` suffix into these files; preview identifiers belong to validation tags and artifacts.
-- Publishing a tag alone does not make an installation target available. Verify the exact image manifest, Release assets, and source archive before advancing references that consume them. Helm packaging derives its versions from the final tag and waits for that image; it does not require an early Chart.yaml bump on main.
-- Preview Release assets are versioned and intentionally visible on the Releases page for the duration of validation. Do not label them Latest or use them to update any latest distribution channel.
-- Never delete a preview Release by hand before Phase 6 finishes — Phase 4 downloads its assets and the final Release notes are generated while it still exists. Cleanup is CI's job; only step in manually (`gh release delete "<preview-tag>" --yes`, never `--cleanup-tag`) if `cleanup-preview-releases` failed.
-- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
-- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final. Phase 7 updates installation references on main after publication; neither tag moves to that follow-up commit.
-- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
-- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
-- Phases run in order; a failure blocks dependent steps. For a preview/source acceptance failure before the final tag exists, land the fix, verify Cargo still matches the confirmed target, and restart from Phase 2 with the next preview at the fixed commit. If main has advanced to another target, resolve that source/version mismatch before another preview.
-- Once the final tag exists, preserve its commit. Retry failed publication jobs for that exact tag; do not recreate the tag or restart preview on newer main. If a source change is required after final-tag publication, report the failure and obtain a new target version. Phase 7 failures resume only the failed follow-up after rechecking artifact availability and whether a newer release has superseded it.
-- Completing preview acceptance does not authorize the final tag. After Phases 3–5 pass, report the acceptance evidence and stop until the user explicitly confirms continuation. The original release request, an earlier confirmation, silence, or an automated follow-up does not satisfy this gate.
-- Confirmation is scoped to the reported `<target>`, `<preview-tag>`, and `PREVIEW_HASH`. A failed or repeated acceptance cycle, including any new preview iteration, invalidates prior confirmation and requires a new one.
-- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
-- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
-
-## Phase 0 — Preflight
-
-- `git status --short` clean; `git fetch origin main --tags`.
-- `gh auth status` works; confirm you can view `gh release list -L 3`.
-- Confirm the exact final target version with the user if not explicit.
-
-### Console release gate
-
-Read and complete [the Console gate](references/console-gate.md) before Phase 1. Verify the latest published Console asset and exact commit; if Console main is ahead, complete its release and asset verification first. A successful build alone does not satisfy this gate.
-
-## Phase 1 — Source version bump to the final target (once)
-
-- If Cargo.toml and all workspace members in Cargo.lock already read `<target>` (e.g. this is a restart after a failed preview), verify both and skip the source bump. Installation versions intentionally differ during preview; do not treat them as leftovers. If a previous preparation advanced installation references to an unavailable target, restore those references to the verified published baseline before selecting the next preview commit.
-- Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
-- Get the PR merged into main. Record the resulting main commit:
-
-```bash
-git fetch origin main
-PREVIEW_HASH=$(git rev-parse origin/main)   # must contain the bump PR
-```
-
-`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
-
-## Phase 2 — Publish the preview tag
-
-```bash
-git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
-git push origin "<preview-tag>"
-```
-
-Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
-
-The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
-
-On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
-
-## Phase 3 — CI and preview Release verification
-
-- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
-- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
-- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
-- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
-- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
-
-## Phases 4–5 — Local artifact, Console, and rc acceptance
-
-Read and complete [preview acceptance](references/preview-acceptance.md): verify the downloaded binary's tag/SHA and readiness, exercise Console CRUD with byte-identical download, and pass the full latest-rc command matrix. Any failure blocks final publication. Retain the results for the confirmation gate below.
-
-### Manual confirmation gate
-
-After every Phase 3–5 check passes, report the target, preview tag, `PREVIEW_HASH`, preview Release URL, console result, and rc matrix, then explicitly ask the user whether to publish the final tag. End the turn without creating or pushing `<target>`.
-
-Continue to Phase 6 only after a new user reply explicitly confirms the reported target, preview tag, and commit. A clear affirmative reply to that exact report, such as `确认继续`, is sufficient; if the reply is ambiguous or any reported value changed, ask again.
-
-## Phase 6 — Publish the final tag on the validated commit
-
-No second source version bump. The final tag goes on the exact commit the preview validated:
-
-```bash
-git fetch origin --tags
-git rev-parse "<preview-tag>^{commit}"          # must equal PREVIEW_HASH — abort if not
-git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
-git push origin "<target>"
-```
-
-- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
-- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
-- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
-- Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
-- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
-
-## Phase 7 — Installation references and website announcement
-
-Only after Phase 6 succeeds, complete [post-release updates](references/post-release-updates.md): align installation references on main, then update the existing top banner in `rustfs/rustfs.com`. A failed or incomplete publication leaves both on the previous available version. These follow-up commits never change `PREVIEW_HASH` or either release tag. Track their PR, merge, and deployment states separately from artifact publication; an open PR is not a live website update.
-
-## Output contract
-
-Always report:
-
-- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
-- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
-- Manual confirmation gate status (`WAITING_FOR_CONFIRMATION` or `CONFIRMED`) and its exact target, preview tag, and `PREVIEW_HASH`.
-- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, the rc command matrix, and the preview-Release cleanup result (deleted Releases plus surviving tags).
-- Post-release installation PR and verification results; website banner target, text, link, PR, and observed deployment state. Report any remaining merge authorization or failed deployment explicitly rather than claiming the banner is live.
-- Any deviation from this pipeline and why the user approved it.
+Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization.

.agents/skills/rustfs-release-publish/references/final-release.md

@@ -0,0 +1,22 @@
+# Final Release Publication
+
+Read only after fresh user confirmation of the preview acceptance report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
+
+## Phase 6 — Publish the final tag on the validated commit
+
+No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH`. If it moved after confirmation, the approval no longer applies; validate and obtain confirmation for a new preview. The final tag goes on the exact commit the preview validated:
+
+```bash
+git fetch origin --tags
+git rev-parse "<preview-tag>^{commit}"          # must equal PREVIEW_HASH — abort if not
+git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
+git push origin "<target>"
+```
+
+- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
+- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
+- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
+- Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
+- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
+
+Never pass `--cleanup-tag` when removing preview Releases. Remove them manually only if `cleanup-preview-releases` failed, after final publication and notes generation. Record deleted Releases and surviving tags.

.agents/skills/rustfs-release-publish/references/post-release-updates.md

@@ -1,7 +1,15 @@
-# Post-release Installation and Website Updates
+# Post-release Milestone, Installation, and Website Updates
 
 Read only after the final non-preview tag has passed Phase 6. Tag creation or a green binary build alone is insufficient: the release assets, source archive, container images, Helm package, and applicable latest channels must be available. Preview tags never enter this phase.
 
+## Version milestones
+
+1. Query all pages of both open and closed milestones in the RustFS release repository, matching titles exactly to version strings without a `v` prefix. A failed or incomplete query is not evidence that a milestone is absent; resolve the lookup before making changes.
+2. Close the milestone matching the published `<target>` if it is open; an already closed milestone needs no change. If it is missing, report the gap and continue checking the next version. Do not create a substitute for the released milestone or close/reassign its issues or PRs.
+3. Use an explicitly planned next version when provided; otherwise, for a stable `X.Y.Z`, use `X.Y.(Z+1)`. For example, after `1.0.1` is published, close milestone `1.0.1` and ensure milestone `1.0.2` exists. For an alpha/beta/rc target with a trailing numeric counter, increment that counter within the same channel unless a different next version was specified. If the next version cannot be determined, ask for that version and report this follow-up as blocked. Creating a planning milestone does not confirm the target of a future release or bypass the semver gate.
+4. Reuse an existing next-version milestone, whether open or closed; do not reopen it or create a duplicate. Only when the complete query confirms it is absent, create an open milestone with the exact next-version title. Do not invent a due date or description. After an uncertain creation result, query again before retrying.
+5. Read back both milestones and report their titles, URLs, and actual states, including whether the next milestone was reused or created. A failed close, create, or verification leaves milestone maintenance incomplete; retry only the unfinished operation after refreshing live state. Continue independent installation and website work without moving or republishing the release tag.
+
 ## Installation references
 
 1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.

.agents/skills/rustfs-release-publish/references/preview-release.md

@@ -0,0 +1,24 @@
+# Preview Release Verification
+
+Read for Phase 3 after the candidate passed exact-SHA CI. The [parent release invariants](../SKILL.md) apply.
+
+## Preview channels
+
+- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
+- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
+- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
+- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
+
+## Phase 3 — CI and preview Release verification
+
+- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
+- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
+- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
+- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
+- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
+- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
+
+## Release notes
+
+- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
+- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.

.agents/skills/rustfs-release-publish/references/release-candidate.md

@@ -0,0 +1,37 @@
+# Release Candidate Preparation
+
+Read for Phases 1–2. The [parent release invariants](../SKILL.md) and Console gate apply.
+
+## Candidate Resume Preflight
+
+- `git status --short` clean; `git fetch origin main --tags`.
+- `gh auth status` works; confirm you can view `gh release list -L 3`.
+- Confirm the exact final target version with the user if not explicit.
+- Check whether `release/<target>` already exists before creating it; a resumed release reuses the existing branch and its history. For an in-flight release that already has a preview tag but no branch, anchor the new branch to that tag's commit only after verifying its target versions and prior acceptance evidence.
+
+## Phase 1 — Source version bump to the final target (once)
+
+- If `release/<target>` exists, verify its Cargo.toml and workspace members in Cargo.lock read `<target>` and skip the source bump, even if main has advanced. If the branch versions differ, stop and resolve the mismatch without downgrading main.
+- If no branch exists but an unfinished preview tag does, verify its commit has `<target>` in both version files and skip the bump; Phase 2 anchors the branch to that commit. Stop if those files disagree. Otherwise, if main already has `<target>` in both files, skip the bump. Installation references intentionally retain the previous published deliverable during preview; if they were advanced to an unavailable target, restore that baseline before choosing the next candidate.
+- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
+- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
+
+Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
+
+## Phase 2 — Pin and validate the release candidate, then publish the preview tag
+
+- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
+- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
+- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
+- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
+
+```bash
+git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
+git push origin "<preview-tag>"
+```
+
+Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
+
+The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
+
+On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -1,118 +1,24 @@
 ---
 name: rustfs-release-version-bump
-description: "Prepare Cargo versions for an exact RustFS target, or align installation references after its artifacts are published, with verification and optional commit/push/PR delivery. Use for an explicit version bump or when invoked by the release-publish workflow."
+description: Prepare an exact RustFS Cargo version or align installation references after publication. Use for requested version bumps or the release workflow.
 ---
-# RustFS Release Version Bump
+# Release Version Files
 
-Use this skill to prepare and verify release version files. Commit, push, and PR steps apply only when included in the user's delivery scope; publishing release tags belongs to `rustfs-release-publish`.
+Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those are tag-only. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
 
-## Required inputs
+| Stage | Files and constraints |
+|---|---|
+| `prepare` | Only `Cargo.toml` and `Cargo.lock`: workspace package, internal dependency, and workspace member versions become `<target>`. External dependencies stay unchanged. |
+| `post-release` | Only `README.md`, `README_ZH.md`, `flake.nix`, `helm/rustfs/Chart.yaml`, and `rustfs.spec`. Cargo may already target the next release; leave it untouched. |
 
-- Exact target version, for example `1.0.0-beta.4`.
-- Stage: `prepare` (default) or `post-release`. Infer `post-release` only when the user or parent release workflow explicitly requests installation updates after publication; a normal version bump means `prepare`.
-- Delivery scope: local (`edit/verify`), git (`commit/push`), or GitHub
-  (`commit/push/PR`). Derive it from the conversation; when unspecified, prepare
-  and verify locally without blocking on a delivery question.
+Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Preview preparation retains the previous published installation references; follow-up commits never replace either validated tag.
 
-If target version is missing or ambiguous, stop and ask before editing.
+Packaging policy:
 
-Reject any target version containing `-preview`: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
+- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>` (`beta.N -> 0.N.0`; other versions unchanged). Helm CI derives versions from the final tag, so no early chart bump is required.
+- RPM `Version` is numeric; `Release` is the prerelease suffix or `1` for stable. Expanded `Source0` and the unpack directory must resolve to the exact target including its suffix. Add the changelog entry using current git identity, date, and target; never copy an example identity/date.
+- Changing these mappings requires explicit confirmation.
 
-## Read before editing
+Verification follows root tiers: `prepare` validates Cargo metadata with the updated lockfile and checks every workspace/internal version with no unrelated lockfile updates. `post-release` renders Helm with the verified image, runs `scripts/test_helm_chart_version.sh` for chart version edits, and checks README, Nix, and RPM references. Run `git diff --check`; report unresolved required checks as `BLOCKED` without unrelated fixes.
 
-- `AGENTS.md` (root and nearest path-specific files).
-- `.github/pull_request_template.md` only when preparing a PR.
-- Current branch status and diff against `origin/main`.
-
-## Stage boundaries
-
-`prepare` updates only:
-
-- `Cargo.toml`
-- `Cargo.lock`
-
-`post-release` updates installation and packaging references only:
-
-- `README.md`
-- `README_ZH.md`
-- `flake.nix`
-- `helm/rustfs/Chart.yaml`
-- `rustfs.spec`
-
-During preview, installation references intentionally retain the previous published deliverable. Do not sweep them into a source bump to make every version string match. `flake.nix` builds local source, but its package label is aligned with the other packaging references after publication.
-
-Before `post-release` edits, verify that the exact non-preview target has a published GitHub Release, downloadable assets and source archive, and a pullable `rustfs/rustfs:<target>` image manifest. If any prerequisite is missing, report `BLOCKED` and leave installation references unchanged. If a newer deliverable has already superseded this target, do not downgrade installation defaults during a retry.
-
-Helm CI derives chart versions from the triggering tag, so the final tag can retain the previous Chart.yaml values. Update the repository copy only after the target image and published chart are available. Neither post-release changes nor their merge commit may replace the preview-validated final tag.
-
-## Hard release policy
-
-- Docker doc tags use `<version>` (for example `rustfs/rustfs:1.0.0-beta.4`), not `v<version>`.
-- Derive Helm chart and app versions with `scripts/helm_chart_version.sh <target>`; the existing mapping is `beta.N -> 0.N.0`, with other target versions retained.
-- `rustfs.spec` `Release` uses the prerelease suffix (for example `beta.4`), or `1` for a stable release.
-- Do not change these rules without explicit confirmation.
-
-## Step-by-step workflow
-
-1. Confirm intent and isolate scope
-- Use the exact target and delivery scope already supplied; ask only for a missing or ambiguous target or a material release-policy choice.
-- Inspect current branch and ensure only release-related files are touched for this task.
-
-2. Update workspace versions (`prepare` only)
-- Bump `[workspace.package].version` in `Cargo.toml`.
-- Bump internal workspace crate dependency versions in `Cargo.toml`.
-- Update `Cargo.lock` so workspace package versions match target version.
-- Re-scan Cargo.toml and workspace members in Cargo.lock for partial leftovers; leave external dependency versions unchanged.
-
-3. Update installation references (`post-release` only)
-- `README.md` and `README_ZH.md`: update versioned Docker examples to target version.
-- `flake.nix`: update package version to target version.
-- `helm/rustfs/Chart.yaml`: use the app and chart versions returned by `scripts/helm_chart_version.sh <target>`.
-- `rustfs.spec`:
-- Set `Version` to the numeric version and `Release` to the prerelease suffix (example `beta.4`), or `1` for a stable release. Verify that `Source0` and the unpacked source directory resolve to the exact published target, including its prerelease suffix when present.
-- Add/update top changelog entry with exact format:
-- `* Thu May 20 2026 houseme <housemecn@gmail.com>`
-- `- Update RPM package to RustFS 1.0.0-beta.4`
-- Changelog identity and time must come from current environment:
-- `git config --get user.name`
-- `git config --get user.email`
-- `date '+%a %b %d %Y'`
-- Changelog version text must match target release version exactly.
-
-4. Verify before shipping
-- Follow the root verification tiers for the final diff instead of running a full-workspace gate for version strings.
-- For `prepare`, validate Cargo metadata with the updated lockfile and confirm workspace package/internal dependency versions agree. Do not accept a lockfile containing unrelated dependency updates.
-- For `post-release`, render the Helm chart and confirm its default image is the verified target; run `scripts/test_helm_chart_version.sh` when chart versions change. Check the README image tags, Nix package version, and expanded RPM source URL against that same target.
-- Run `git diff --check` for either stage. Report unresolved required checks as `BLOCKED`; do not silently widen scope to fix unrelated issues.
-
-5. Commit strategy (only when committing is authorized)
-- Use `chore(release): prepare <version>` for `prepare` and `chore(release): align installation references for <version>` for `post-release`.
-- These stages happen on opposite sides of publication; do not combine them in a preview-preparation commit or PR.
-- Stage only intended release files; do not include unrelated working tree changes.
-
-6. Push and PR (only for the authorized delivery scope)
-- Push branch:
-- Use the user-requested or configured push remote: `git push -u <push-remote> <branch>` (first push), or `git push` when tracking is already configured.
-- Create PR with template headings unchanged:
-- `gh pr create --base main --head <branch> --title ... --body-file ...`
-- PR title/body must be English.
-- Use `N/A` for non-applicable template sections.
-- Include verification commands and any `BLOCKED` reason clearly.
-
-## Recommended check commands
-
-- `git status --short --branch`
-- `git diff --name-only origin/main...HEAD`
-- `git diff --stat origin/main...HEAD`
-- `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
-
-## Output contract
-
-When using this skill, always report:
-
-- Target version and stage.
-- Files changed.
-- Any assumptions or uncertainties requiring confirmation.
-- Verification result (`PASSED` or `BLOCKED`) with key evidence.
-- Commit message(s) used.
-- Push status and PR URL when GitHub flow is requested.
+For authorized commits, keep stages separate: `chore(release): prepare <target>` or `chore(release): align installation references for <target>`. Report target/stage, changed files, verification, material uncertainty, and actual commit/push/PR state.

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -50,6 +50,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-g8w9-qw9q-fghr` and `GHSA-xm99-m3gq-83g8`: a valid presigned `PutObject` accepted extra `x-amz-*` headers omitted from `SignedHeaders`; `x-amz-copy-source` could turn that upload capability into a cross-bucket read performed as the signer. Lesson: a presigned URL is a bounded capability; inspect all received security-sensitive headers and reject unsigned ones before selecting the S3 operation or reaching storage.
 - `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
+- `GHSA-gjph-xcgw-4p7w`: `PutObject` and browser POST accepted a caller-supplied `versionId` and replaced an unlocked historical version with only `s3:PutObject`. Lesson: only replication-authorized writes may choose an existing version ID; ordinary uploads must not rewrite version history.
 - `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
 - `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
 - `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
@@ -60,12 +61,14 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 - `GHSA-3g29-xff2-92vp`: FTP `RETR` and `SIZE`/`MDTM` read paths authenticated the user but skipped IAM before calling storage. Lesson: non-HTTP protocol frontends must enforce the same per-operation authorization as the S3 API before backend access.
 - `GHSA-g3vq-vv42-f647`: FTPS `MKD` called `create_bucket` without checking `s3:CreateBucket`. Lesson: protocol command handlers need action-specific checks even when sibling handlers already authorize correctly.
 - `GHSA-3p3x-734c-h5vx`: FTPS and WebDAV compared secret keys with early-return string equality, while FTPS also returned distinguishable invalid-user and invalid-password failures. Lesson: password-style protocol auth needs constant-time secret comparison, indistinguishable failures where practical, and rate limiting.
+- `GHSA-rhg6-jrf2-29c4`: WebDAV, FTP/FTPS, and SFTP used identity-policy checks with empty conditions and bypassed S3 bucket-policy explicit denies and condition-key denies. Lesson: alternate listeners must evaluate the same effective policy and trustworthy request context before storage access.
 
 ### Filesystem paths and object key traversal
 
 - `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
 - `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
 - `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
+- `GHSA-g7fq-6xjm-mxf4`: versioned `DeleteObjects` accepted dot-segment keys, authorized the literal prefix, then wrote delete markers under a resolved sibling prefix. Lesson: validate each batch key before authorization and storage normalization, using the same object-name rules as single delete.
 
 ### Secrets, defaults, and cryptographic misuse
 
@@ -106,6 +109,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 ### Object Lock and retention invariants
 
 - `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
+- `GHSA-28w4-rp2p-636w` and `GHSA-c7hx-vr97-45rg`: an authorized `DeleteBucket` with `x-rustfs-force-delete: true` removed active COMPLIANCE-retained versions even though ordinary version deletion was denied. Lesson: bucket force-delete must enforce retention across every version, including against root/owner callers.
 
 ### Serde deserialization and input validation
 
@@ -135,16 +139,17 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 ## Minimum Regression Test Expectations
 
 - Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
-- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
+- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases; test bucket-policy explicit denies and condition-key denies on each enabled listener against an S3 control, and assert denied paths do not reach storage.
 - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
 - Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, storage-class, and cross-bucket copy-source headers; require rejection before source or destination storage access, and verify explicitly signed equivalents still work.
 - Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
+- Version-write fixes: a `PutObject`-only principal must not replace an existing version through `PutObject` or browser POST `?versionId=`; verify replication-authorized writes and versioned, suspended, and unversioned buckets separately.
 - Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
-- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
+- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks; test per-key `DeleteObjects` errors and prefix-scoped denials on versioned buckets.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
 - IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
 - RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
-- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.
+- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions.

.agents/skills/test-coverage-improver/SKILL.md

@@ -1,57 +0,0 @@
----
-name: test-coverage-improver
-description: Analyze a supplied coverage report or perform an explicitly requested RustFS coverage assessment, rank uncovered risks, and propose focused tests. Do not trigger for ordinary implementation verification, a single regression test, documentation wording, or release preparation without a coverage request.
----
-
-# Test Coverage Improver
-
-Use this skill when you need a prioritized, risk-aware plan to improve tests from coverage results.
-
-## Usage assumptions
-- Focus scope is either changed lines/files, a module, or the whole repository.
-- Reuse a supplied coverage artifact when its revision, scope, and format match.
-- If required context is missing, call out assumptions explicitly before proposing work.
-
-## Workflow
-
-1. Define scope and baseline
-   - Derive the revision and scope from the request, diff, or supplied report.
-   - Default to the affected files/module; whole-workspace coverage requires that
-     scope in the request. Ask only if a wrong scope would change the result.
-
-2. Obtain coverage evidence
-   - First inspect a matching existing artifact; do not regenerate it merely
-     because this skill was selected.
-   - If measurement is needed, read the Coverage section of
-     [the testing guide](../../../docs/testing/README.md#coverage), check disk
-     space/tool availability, and select package/test-scoped `cargo llvm-cov`
-     using the repository's nextest configuration. `make coverage` measures the
-     whole workspace (excluding E2E) and is only for that requested scope.
-   - Collect only metrics the report supports. Missing branch/changed-line
-     coverage is unknown, not zero.
-   - If measurement cannot run, continue with code-based test proposals and
-     mark measured coverage unverified; do not invent a coverage percentage.
-
-3. Rank highest-risk gaps
-   - Prioritize changed code, branch coverage gaps, and low-confidence boundaries.
-   - Apply the risk rubric in [coverage-prioritization.md](references/coverage-prioritization.md).
-   - Report up to 5–8 evidenced gaps; do not pad a small scope.
-   - For each gap, capture: file, lines, uncovered branches, and estimated risk score.
-
-4. Propose high-impact tests
-   - For each gap, name the behavior and regression, distinguishing assertions,
-     relevant normal/edge/failure cases, necessary setup, and estimated effort.
-   - Include only scenarios and setup that apply; reuse shared fixture details.
-
-5. Close with validation plan
-   - State which gaps remain after proposals.
-   - Give a scoped verification command and behavior-based acceptance criterion;
-     use a coverage threshold only when the task or repository requires one.
-   - List assumptions or blockers (environment, fixtures, flaky dependencies).
-
-## Report
-
-Summarize the supported metrics, then combine each ranked gap with its proposed
-test and validation. Include source lines only when supplied or inspected;
-mark missing metrics or locations as unknown. Do not duplicate gaps and tests
-in separate templates or fill empty categories for an otherwise small report.

.agents/skills/test-coverage-improver/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "Test Coverage Improver"
-  short_description: "Find top uncovered risk areas and propose high-impact tests."
-  default_prompt: "Use $test-coverage-improver to analyze coverage for the requested scope, reuse matching reports, and propose tests for evidenced risks."

.agents/skills/test-coverage-improver/references/coverage-prioritization.md

@@ -1,25 +0,0 @@
-# Coverage Gap Prioritization Guide
-
-Use this rubric for each uncovered area.
-
-Score = (Criticality × 2) + CoverageDebt + (Volatility × 0.5)
-
-- Criticality:
-  - 5: authz/authn, data-loss, payment/consistency path
-  - 4: state mutation, cache invalidation, scheduling
-  - 3: error handling + fallbacks in user-visible flows
-  - 2: parsing/format conversion paths
-  - 1: logging-only or low-impact utilities
-
-- CoverageDebt:
-  - 0: 0–5 uncovered lines
-  - 1: 6–20 uncovered lines
-  - 2: 21–40 uncovered lines
-  - 3: 41+ uncovered lines
-
-- Volatility:
-  - 1: stable legacy code with few recent edits
-  - 2: changed in last 2 releases
-  - 3: touched in last 30 days or currently in active PR
-
-Sort by score descending, then by business impact.

.config/e2e-distributed-selection.txt

@@ -1,2 +1,2 @@
-sha256-linux=21ab66fd172018682ce825a53f26aa8f286d57f3f687d8616d1850e56474dcd8
-sha256-darwin=21ab66fd172018682ce825a53f26aa8f286d57f3f687d8616d1850e56474dcd8
+sha256-linux=233bd7a68777eda1f044b6cb77128056be2b6492583c0dfd114b21e98a6102ef
+sha256-darwin=233bd7a68777eda1f044b6cb77128056be2b6492583c0dfd114b21e98a6102ef

.config/e2e-full-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=9e9687ded961aa4e619a15def6e1ee0a5153be05423d009ee7a58126624c83e7
-sha256-linux=364666693c2bf258997643c9d87e07e7b7501e58175f190d201d53c81946eb20
+sha256-darwin=98dff0961c6eb21d6128bfd2d4a6100a624ddf1fb7db05dba360a5d3d0a310c2
+sha256-linux=890f5925b87e1bf3710c356253f3b2276d901e156df9ccc04ec9315374de6d22

.config/e2e-nightly-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=83a7dcaffd5a789517ae9f02a224f66a9713937885cff96fca2ad7e216f197ae
-sha256-linux=626c10f8c964507ff987b6c86069e9019dc6d2ae7fb02db9be5df5aa8cc5145b
+sha256-darwin=efe68675f4676a0d2901242c5815b1f7af5cdec8231f7cccd408f6dacf5c75db
+sha256-linux=003ed6480935a0a95680cc061be143722fbd37bd477b8ff98607b1a064f3aef1

.config/e2e-protocols-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=55534a97fbd376f64c8f6c341d319017d11ff77cad6da8629a1a7f6a874e0315
-sha256-linux=c06fb8c19aed6f388b9dc61cb8251b7a44f8561a9bf764ad2b9e635598f8dc17
+sha256-darwin=a2cb67d91eb0d093928371d6e84a84b2dd41b2ba975a28094e81879bd65838ad
+sha256-linux=82db0561cf9b111f11782d647d0859c9d2a44c5c68481b5e39c48505828bd392

.config/e2e-smoke-selection.txt

@@ -1 +1 @@
-sha256=548a58d74f3cb3b5a1ab9dcd4d3f8625b9ddebe847a2e14de1d08c3b2e6292eb
+sha256=ba8f392b617cf9f2338142233a417e43927b28096635e628233ce8a60cf49d10

.config/ecstore-required-tests.json

@@ -55,6 +55,281 @@
       "invariant": "odm-source-contract-gcs",
       "suite": "rustfs",
       "name": "on_demand_migration::gcs::tests::gcs_native_backend_satisfies_the_shared_backend_contract"
+    },
+    {
+      "invariant": "rollback-inspection",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::io_primitives::tests::rename_rollback_incomplete_preserves_overwrite_data_dirs_and_staging"
+    },
+    {
+      "invariant": "rollback-closed-inspection-channel",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::io_primitives::tests::rename_rollback_closed_heal_receiver_preserves_recovery_material"
+    },
+    {
+      "invariant": "rollback-cancellation",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::io_primitives::tests::rename_rollback_incomplete_cancelled_task_is_not_success"
+    },
+    {
+      "invariant": "rollback-completion-equivalence",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::io_primitives::tests::rename_rollback_incomplete_matches_early_ack_and_full_wait_after_reopen"
+    },
+    {
+      "invariant": "rollback-physical-owner",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::io_primitives::tests::rename_rollback_children_keep_namespace_ownership_after_coordinator_panic"
+    },
+    {
+      "invariant": "late-ack-retains-commit",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::io_primitives::tests::rename_data_early_ack_post_mutation_tail_error_never_rolls_back_commit"
+    },
+    {
+      "invariant": "tail-drained-cas",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::tail_drained_put_waits_for_tail_and_allows_immediate_cas"
+    },
+    {
+      "invariant": "seeded-put-tail-pressure",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::seeded_put_completion_pressure_drains_every_real_tail"
+    },
+    {
+      "invariant": "borrowed-lock-tail",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::no_lock_put_waits_for_rename_tail_under_outer_guard"
+    },
+    {
+      "invariant": "cancelled-writer-owner",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::tail_drained_put_owned_commit_survives_waiter_cancellation"
+    },
+    {
+      "invariant": "get-body-reconstruction",
+      "suite": "rustfs-ecstore",
+      "name": "ecstore_validation_blackbox::blackbox_get_restores_body_after_one_shard_file_is_removed"
+    },
+    {
+      "invariant": "range-reconstruction",
+      "suite": "rustfs-ecstore",
+      "name": "ecstore_validation_blackbox::blackbox_range_read_restores_exact_slice_with_one_offline_disk"
+    },
+    {
+      "invariant": "multipart-abort-linearization",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::multipart::tests::abort_and_complete_linearize_for_plain_sse_and_legacy_layouts"
+    },
+    {
+      "invariant": "multipart-write-quorum",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::multipart::tests::abort_enforces_delete_write_quorum_boundary"
+    },
+    {
+      "invariant": "multipart-physical-owner",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::multipart::tests::complete_multipart_timeout_keeps_namespace_owner_until_physical_publication"
+    },
+    {
+      "invariant": "restore-abort-cleanup",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::object::transition_commit_failure_tests::multipart_restore_aborts_every_post_create_failure"
+    },
+    {
+      "invariant": "legacy-metadata",
+      "suite": "rustfs-filemeta",
+      "name": "filemeta::test::test_issue_2288_legacy_xlmeta_compatibility"
+    },
+    {
+      "invariant": "legacy-nil-version",
+      "suite": "rustfs-filemeta",
+      "name": "filemeta::test::test_into_fileinfo_reads_legacy_nil_uuid_inline_key"
+    },
+    {
+      "invariant": "commit-phase-schedule",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::io_primitives::tests::commit_fault_schedule_keeps_multiple_disk_phases_independent"
+    },
+    {
+      "invariant": "commit-phase-ownership",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::io_primitives::tests::commit_fault_schedule_rejects_duplicate_checkpoint_without_disarming_owner"
+    },
+    {
+      "invariant": "metadata-observation-all-four-slot-states-and-arrival-orders-preserve-reduction",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::metadata_quorum::tests::metadata_observation_all_four_slot_states_and_arrival_orders_preserve_reduction"
+    },
+    {
+      "invariant": "metadata-observation-mixed-current-identity-arrivals-preserve-selection",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::metadata_quorum::tests::metadata_observation_mixed_current_identity_arrivals_preserve_selection"
+    },
+    {
+      "invariant": "metadata-observation-pending-newer-version-and-same-time-different-directory-force-full-wait",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::metadata_quorum::tests::metadata_observation_pending_newer_version_and_same_time_different_directory_force_full_wait"
+    },
+    {
+      "invariant": "metadata-observation-duplicate-shard-cannot-supply-a-read-reserve",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::metadata_quorum::tests::metadata_observation_duplicate_shard_cannot_supply_a_read_reserve"
+    },
+    {
+      "invariant": "metadata-observation-null-versions-markers-and-invalid-success-keep-their-meaning",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::metadata_quorum::tests::metadata_observation_null_versions_markers_and_invalid_success_keep_their_meaning"
+    },
+    {
+      "invariant": "metadata-cache-real-get-and-range-preserve-body-across-read-modes",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::read::metadata_cache_tests::metadata_cache_real_get_and_range_preserve_body_across_read_modes"
+    },
+    {
+      "invariant": "late-metadata-refresh-rejects-changed-generation-and-mapped-shards",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::read::metadata_cache_tests::late_metadata_refresh_rejects_changed_generation_and_mapped_shards"
+    },
+    {
+      "invariant": "late-full-get-rejects-changed-object-generation",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::prepared_get_object_metadata_tests::non_inline_two_phase_get_rejects_generation_change_before_late_refresh"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-static-namespace",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_static_namespace"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-cross-subdirectory-limit",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_cross_subdirectory_limit"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-max-and-max-plus-one",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_max_and_max_plus_one"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-fixed-seed-marker-and-prefix",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_fixed_seed_marker_and_prefix"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-offline-corrupt-and-minority-stale",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_offline_corrupt_and_minority_stale"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-versions-delete-markers-and-free-versions",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_versions_delete_markers_and_free_versions"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-orphan-cleanup-requires-authoritative-evidence",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_orphan_cleanup_requires_authoritative_evidence"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-consumer-drop-joins-real-disk-producer",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_consumer_drop_joins_real_disk_producer"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-cancelled-quorum-walk-releases-senders-and-disk-lock",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_cancelled_quorum_walk_releases_senders_and_disk_lock"
+    },
+    {
+      "invariant": "list-objects-shared-corpus-enabled-index-contracts",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_enabled_index_contracts"
+    },
+    {
+      "invariant": "borrowed-context-boundary",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::borrowed_write_context_rejects_wrong_namespace_and_cache_flag"
+    },
+    {
+      "invariant": "borrowed-put-cancellation",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::borrowed_tail_drained_put_keeps_owner_after_caller_cancel"
+    },
+    {
+      "invariant": "borrowed-mpu-cancellation",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::multipart::tests::borrowed_complete_keeps_namespace_owner_after_caller_cancel"
+    },
+    {
+      "invariant": "pool-cas-owner",
+      "suite": "rustfs-ecstore",
+      "name": "core::pools::pools_tests::pool_meta_cas_context_retains_actual_owner_for_exact_target"
+    },
+    {
+      "invariant": "pool-cas-released-owner",
+      "suite": "rustfs-ecstore",
+      "name": "core::pools::pools_tests::pool_meta_cas_rejects_released_owner"
+    },
+    {
+      "invariant": "copy-tail-drained",
+      "suite": "rustfs-ecstore",
+      "name": "store::object::tests::borrowed_copy_tail_drained_waits_for_rename_tail"
+    },
+    {
+      "invariant": "catalog-borrowed-publication",
+      "suite": "rustfs",
+      "name": "admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition"
+    },
+    {
+      "invariant": "directory-marker-quorum",
+      "suite": "rustfs-filemeta",
+      "name": "metacache::tests::resolve_directory_marker_and_prefix_preserve_quorum_in_both_orders"
+    },
+    {
+      "invariant": "logical-prefix-limit",
+      "suite": "rustfs-ecstore",
+      "name": "store::list_objects::test::list_path_gather_results_counts_common_prefixes_before_page_limit"
+    },
+    {
+      "invariant": "quota-same-name-owner-and-disk-authority",
+      "suite": "rustfs-ecstore",
+      "name": "bucket::metadata_sys::tests::quota_snapshot_reuses_same_name_owner_and_rereads_disk_authority"
+    },
+    {
+      "invariant": "quota-snapshot-scope-and-owner",
+      "suite": "rustfs-ecstore",
+      "name": "bucket::metadata_sys::tests::quota_snapshot_rejects_wrong_scope_incarnation_and_transaction_guard"
+    },
+    {
+      "invariant": "quota-snapshot-lifecycle-and-incarnation",
+      "suite": "rustfs-ecstore",
+      "name": "bucket::metadata_sys::tests::quota_snapshot_rejects_lifecycle_loss_during_read_and_changed_disk_incarnation"
+    },
+    {
+      "invariant": "marker-purge-receipt-reopen",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::object::marker_purge_receipt_tests::quorum_marker_purge_receipt_survives_reopen_with_stale_member"
+    },
+    {
+      "invariant": "external-minio-shards",
+      "suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
+      "name": "pinned_minio_shards_decode_and_reconstruct_plaintext"
+    },
+    {
+      "invariant": "external-legacy-shards",
+      "suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
+      "name": "pinned_legacy_shards_decode_and_reconstruct_plaintext"
+    },
+    {
+      "invariant": "external-shard-bitrot",
+      "suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
+      "name": "pinned_erasure_shards_reject_bitrot_corruption"
+    },
+    {
+      "invariant": "external-shard-quorum-codec",
+      "suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
+      "name": "pinned_erasure_shards_reject_insufficient_quorum_and_wrong_codec"
     }
   ],
   "fixtures": [
@@ -82,6 +357,36 @@
       "path": "crates/ecstore/tests/fixtures/minio/bucket_metadata_full.xlmeta.hex",
       "sha256": "3b6de589519c08a1614c8bd409bb8199c17d42043861b07bce513075e6fbfc12",
       "source": "MinIO RELEASE.2025-07-23T15-54-02Z; crates/ecstore/tests/fixtures/minio/README.md"
+    },
+    {
+      "path": "crates/filemeta/tests/fixtures/issue_2288_legacy_xlmeta.hex",
+      "sha256": "b3e8d32f9950a38b5dc71f679519cc830a2b2a807e8b0b39f78840b238d8eacf",
+      "source": "Legacy xl.meta captured for rustfs/backlog#2288; filemeta::test::test_issue_2288_legacy_xlmeta_compatibility"
+    },
+    {
+      "path": "crates/filemeta/tests/fixtures/issue_2265_legacy_meta_v2_object.hex",
+      "sha256": "89dd7ec92d16e54b5d31e0c49c9edffff8a7aa58890aaeff59bc2ff84da0a62f",
+      "source": "Legacy metadata v2 object captured for rustfs/backlog#2265; crates/filemeta/src/test_data.rs"
+    },
+    {
+      "path": "crates/ecstore/tests/fixtures/list_namespace_keys.json",
+      "sha256": "5afddb60565deba0938e617470255563f0e15a15fab699645c59bb7ccc84898e",
+      "source": "Shared static LIST namespace corpus; ECStore differential tests and ODM list_through_static_namespace_boundary_matrix"
+    },
+    {
+      "path": "crates/ecstore/tests/fixtures/erasure-shards/minio/manifest.json",
+      "sha256": "c398f8dcb76903f6cf8eee07931c76592895ced201ff9d59ed4ff9b0ead2b13c",
+      "source": "MinIO RELEASE.2025-07-23T15-54-02Z; crates/ecstore/tests/fixtures/erasure-shards/README.md"
+    },
+    {
+      "path": "crates/ecstore/tests/fixtures/erasure-shards/legacy/manifest.json",
+      "sha256": "dce397d5b2f42748eb5091b4355bb9d133a50cbba003626fafac28095dbe540b",
+      "source": "RustFS 1.0.0-alpha.37; crates/ecstore/tests/fixtures/erasure-shards/README.md"
+    },
+    {
+      "path": "crates/ecstore/tests/fixtures/erasure-shards/plaintext.bin",
+      "sha256": "b5a83332327961a31b65a8731c64559fd04ff8ac9f6124022201dbbdba5ffa01",
+      "source": "Independent fixed SHA256 counter-stream oracle; crates/ecstore/tests/fixtures/erasure-shards/README.md"
     }
   ]
 }

.config/make/tests.mak

@@ -52,6 +52,7 @@ script-tests: ## Run shell script tests
 	./scripts/check_embedded_secrets.sh --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
+	$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_scheduled_validation_freshness.py --self-test
@@ -61,6 +62,7 @@ script-tests: ## Run shell script tests
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain_health.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_report_compat.py
+	$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_runner_tools.py
 	bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_object_data_cache_follower_samples.py --self-test
 	./scripts/validate_object_data_cache_cold_stampede.sh --self-test

.config/nextest.toml

@@ -64,6 +64,26 @@ command = ['sh', '-c', 'echo RUST_MIN_STACK=33554432 >> "$NEXTEST_ENV"']
 command = ['sh', '-c', 'echo RUST_MIN_STACK=33554432 >> "$NEXTEST_ENV"']
 
 # --- default profile (local): serialize the flaky groups, never retry --------
+# These real-disk progress oracles exercise fsync, quorum IO, and crash/replay
+# boundaries. Reserve the run's capacity so unrelated fixtures cannot starve
+# their existing deadlines and typed-failure assertions (rustfs/backlog#2706).
+# Keep each test's internal concurrency and zero-retry policy unchanged.
+[[profile.default.overrides]]
+filter = 'package(rustfs) & test(/^connect::diagnostics::schedule::tests::/)'
+threads-required = "num-test-threads"
+
+[[profile.default.overrides]]
+filter = 'package(rustfs-ecstore) & (test(/^set_disk::tests::streaming_get_blocks_concurrent_/) | test(=store::object::tests::retired_marker_converges_after_bucket_recreation_on_sixteen_disks))'
+threads-required = "num-test-threads"
+
+[[profile.default.overrides]]
+filter = 'package(rustfs-heal) & (binary(mrf_partial_write_test) | (binary(mrf_pipeline_test) & test(/^journal_replay_survives_/)) | test(=admin_selector_survives_sigkill_with_original_scope_and_receipts))'
+threads-required = "num-test-threads"
+
+[[profile.default.overrides]]
+filter = 'package(rustfs-scanner) & (test(/^scanner::backlog::tests::native_(writer|retirement)_/) | test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(/^scanner_io::tests::service_cohort::/))'
+threads-required = "num-test-threads"
+
 [[profile.default.scripts]]
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
@@ -72,6 +92,12 @@ setup = 'ecstore-large-stack'
 filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
 setup = 'ecstore-base-stack'
 
+# These handler/store fixtures and the relocated-pool GET compose deep storage
+# futures in libtest threads; on Linux they overflow the default 2 MiB stack.
+[[profile.default.scripts]]
+filter = 'package(rustfs) & (test(/^admin::handlers::table_catalog::tests::/) | test(/^table_catalog::store::tests::/) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body))'
+setup = 'ecstore-base-stack'
+
 [[profile.default.scripts]]
 filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
 setup = 'lifecycle-large-stack'
@@ -82,6 +108,13 @@ setup = 'lifecycle-large-stack'
 filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
 setup = 'ecstore-large-stack'
 
+# These real-disk scanner fixtures assert quiet-workload publication and
+# durable MRF admission. Reserve the pool while their bounded CAS/fsync
+# handoffs run so unrelated disk fanouts cannot consume that fixture budget.
+[[profile.default.overrides]]
+filter = 'package(rustfs-scanner) & (test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(=scanner_io::tests::multi_pool_scanner_cycle_publishes_combined_usage) | test(=scanner_io::tests::pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage) | test(=scanner_folder::tests::mrf_ownership::checkpoint_fixture_metadata_heal_reaches_budgeted_tail))'
+threads-required = "num-test-threads"
+
 [[profile.default.overrides]]
 filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
 test-group = 'ecstore-serial-flaky'
@@ -107,6 +140,16 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::heal::heal_result_report_tests::/)'
 test-group = 'ecstore-serial-flaky'
 
+# Serialize the replacement bucket metadata heal test. It builds a real-disk
+# hermetic erasure set, writes bucket metadata, then asserts byte-identical
+# xl.meta readback after a rejected delayed repair. Under parallel load,
+# concurrent ecstore tests mutate the process-wide disk registry and alter
+# the serialized timestamp/disk-identity fields, flipping the assertion.
+# Preventive serialization only, no retries.
+[[profile.default.overrides]]
+filter = 'package(rustfs-ecstore) & test(replacement_bucket_metadata_respects_pool_ownership_and_required_records)'
+test-group = 'ecstore-serial-flaky'
+
 # Serialize the metadata-cache generation-retirement pair. Both carry
 # #[serial(metadata_cache_invalidation_probe)] — a no-op across nextest's
 # process boundary — and assert get_object_metadata_cache generation
@@ -132,11 +175,38 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
 test-group = 'embedded-test-ports'
 
-# Inventory delivery and real drive/object probes include durable filesystem IO
-# in short deadlines. Reserve capacity so unrelated storage fixtures cannot
-# exhaust those budgets. Keep the deadlines and assertions unchanged.
+# Registration, inventory delivery, and real drive/object probes include
+# filesystem or HTTP IO in short deadlines. Reserve capacity so sibling test
+# processes cannot exhaust those budgets. Keep deadlines and assertions intact.
+# The short HTTP fixtures are tracked in rustfs/backlog#2714.
 [[profile.default.overrides]]
-filter = 'package(rustfs) & (binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
+filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
+threads-required = "num-test-threads"
+
+# This cancellation probe hashes the serving executable before acquiring its
+# collector lease. Keep competing test processes out of its 30s startup budget.
+[[profile.default.overrides]]
+filter = 'package(rustfs) & test(=connect::diagnostics::trace_runtime::tests::local_runtime_profile_cancellation_waits_for_lease_release)'
+threads-required = "num-test-threads"
+
+# These real IAM/state fixtures keep production lock-acquisition limits while
+# peers complete durable IO. Reserve capacity from unrelated test processes;
+# the shared fixture's durable_iam_state_ prefix keeps the whole family together.
+[[profile.default.overrides]]
+filter = 'package(rustfs) & test(/^admin::handlers::site_replication::tests::durable_iam_state_/)'
+threads-required = "num-test-threads"
+
+# Real target-repair and rebalance-stop fixtures commit durable metadata while
+# peers wait on production locks or bounded cancellation checks. Reserve IO
+# capacity for each fixture; its internal races and deadlines remain intact.
+[[profile.default.overrides]]
+filter = 'package(rustfs) & test(/^admin::handlers::(replication::target_repair_tests::|rebalance::rebalance_handler_tests::real_admin_stop_)/)'
+threads-required = "num-test-threads"
+
+# Reserve capacity for these short heartbeat TLS/fsync probes (rustfs#8129).
+# Their existing deadlines, assertions, and retry policy remain in force.
+[[profile.default.overrides]]
+filter = 'package(rustfs) & binary(connect_heartbeat) & test(/^(restart_replays_pending_request_then_advances_sequence|restart_replays_exact_legacy_capabilities_with_and_without_jobs)$/)'
 threads-required = "num-test-threads"
 
 # Serialize the durable manual-transition checkpoint test across nextest's
@@ -219,6 +289,13 @@ test-group = 'e2e-cluster-nightly'
 filter = 'package(e2e_test) & (test(/^kms::kms_vault_test::/) | test(/^kms::kms_rekey_sweep_test::/) | test(/^kms::configured_roundtrip_test::test_configured_vault_kms_admin_and_versioned_cleanup$/))'
 test-group = 'e2e-vault'
 
+# These server-heavy smoke cases perform 1k+ sequential S3 writes or run the
+# scanner and a remote tier server together. Reserve the full test budget so
+# unrelated E2E processes cannot starve their storage and polling deadlines.
+[[profile.default.overrides]]
+filter = 'package(e2e_test) & (test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/) | test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry))'
+threads-required = "num-test-threads"
+
 # This four-disk, 65-member rollback probe already drives up to 32 concurrent
 # durable deletions. Reserve this nextest run's capacity for its progress oracle.
 [[profile.default.overrides]]
@@ -240,6 +317,24 @@ fail-fast = false
 # marker is the observable signal the flake policy is built around.
 path = "junit.xml"
 
+# Match the local durable-progress reservations. All members, deadlines,
+# assertions, and internal concurrency remain enabled without retries.
+[[profile.ci.overrides]]
+filter = 'package(rustfs) & test(/^connect::diagnostics::schedule::tests::/)'
+threads-required = "num-test-threads"
+
+[[profile.ci.overrides]]
+filter = 'package(rustfs-ecstore) & (test(/^set_disk::tests::streaming_get_blocks_concurrent_/) | test(=store::object::tests::retired_marker_converges_after_bucket_recreation_on_sixteen_disks))'
+threads-required = "num-test-threads"
+
+[[profile.ci.overrides]]
+filter = 'package(rustfs-heal) & (binary(mrf_partial_write_test) | (binary(mrf_pipeline_test) & test(/^journal_replay_survives_/)) | test(=admin_selector_survives_sigkill_with_original_scope_and_receipts))'
+threads-required = "num-test-threads"
+
+[[profile.ci.overrides]]
+filter = 'package(rustfs-scanner) & (test(/^scanner::backlog::tests::native_(writer|retirement)_/) | test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(/^scanner_io::tests::service_cohort::/))'
+threads-required = "num-test-threads"
+
 [[profile.ci.scripts]]
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
@@ -248,6 +343,10 @@ setup = 'ecstore-large-stack'
 filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
 setup = 'ecstore-base-stack'
 
+[[profile.ci.scripts]]
+filter = 'package(rustfs) & (test(/^admin::handlers::table_catalog::tests::/) | test(/^table_catalog::store::tests::/) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body))'
+setup = 'ecstore-base-stack'
+
 [[profile.ci.scripts]]
 filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
 setup = 'lifecycle-large-stack'
@@ -258,6 +357,13 @@ setup = 'lifecycle-large-stack'
 filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
 setup = 'ecstore-large-stack'
 
+# These real-disk scanner fixtures assert quiet-workload publication and
+# durable MRF admission. Reserve the pool while their bounded CAS/fsync
+# handoffs run so unrelated disk fanouts cannot consume that fixture budget.
+[[profile.ci.overrides]]
+filter = 'package(rustfs-scanner) & (test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(=scanner_io::tests::multi_pool_scanner_cycle_publishes_combined_usage) | test(=scanner_io::tests::pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage) | test(=scanner_folder::tests::mrf_ownership::checkpoint_fixture_metadata_heal_reaches_budgeted_tail))'
+threads-required = "num-test-threads"
+
 # ===========================================================================
 # QUARANTINE — flaky tests granted retries = 2 under the ci profile ONLY.
 #
@@ -283,13 +389,6 @@ retries = 2
 filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes))'
 test-group = 'ecstore-serial-flaky'
 
-# QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
-# on producer/consumer timing windows that stretch past the budget on loaded
-# CI runners (regression test for rustfs#4644; failed on a zero-Rust-diff PR).
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(walk_dir_does_not_charge_consumer_backpressure_to_the_stall_budget)'
-retries = 2
-
 # Serialize the relocated-pool GET resume regression under the ci profile too
 # (see the matching default-profile override near the top). No longer a
 # quarantine: the fixture race (rustfs#6701/rustfs#6703) was fixed by #6707,
@@ -321,6 +420,11 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::heal::heal_result_report_tests::/)'
 test-group = 'ecstore-serial-flaky'
 
+# Serialize the replacement bucket metadata heal test under the ci profile too.
+[[profile.ci.overrides]]
+filter = 'package(rustfs-ecstore) & test(replacement_bucket_metadata_respects_pool_ownership_and_required_records)'
+test-group = 'ecstore-serial-flaky'
+
 # Serialize the metadata-cache generation-retirement pair under the ci
 # profile too (see the matching default-profile override near the top). Not a
 # quarantine: no retries.
@@ -335,7 +439,30 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
 test-group = 'embedded-test-ports'
 
 [[profile.ci.overrides]]
-filter = 'package(rustfs) & (binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
+filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
+threads-required = "num-test-threads"
+
+# Preserve the cancellation probe's startup budget and assertions in CI too.
+[[profile.ci.overrides]]
+filter = 'package(rustfs) & test(=connect::diagnostics::trace_runtime::tests::local_runtime_profile_cancellation_waits_for_lease_release)'
+threads-required = "num-test-threads"
+
+# Keep the same IAM/state fixture capacity reservation in CI without changing the
+# production lock deadline, internal concurrency, assertions, or retry policy.
+[[profile.ci.overrides]]
+filter = 'package(rustfs) & test(/^admin::handlers::site_replication::tests::durable_iam_state_/)'
+threads-required = "num-test-threads"
+
+# Match the real admin fixture capacity reservation without retries or changes
+# to production lock deadlines, cancellation barriers, or assertions.
+[[profile.ci.overrides]]
+filter = 'package(rustfs) & test(/^admin::handlers::(replication::target_repair_tests::|rebalance::rebalance_handler_tests::real_admin_stop_)/)'
+threads-required = "num-test-threads"
+
+# Reserve capacity for these short heartbeat TLS/fsync probes (rustfs#8129).
+# Their existing deadlines, assertions, and retry policy remain in force.
+[[profile.ci.overrides]]
+filter = 'package(rustfs) & binary(connect_heartbeat) & test(/^(restart_replays_pending_request_then_advances_sequence|restart_replays_exact_legacy_capabilities_with_and_without_jobs)$/)'
 threads-required = "num-test-threads"
 
 # Serialize the durable manual-transition checkpoint test under the ci profile
@@ -469,6 +596,7 @@ default-filter = """
         | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
         | test(/^reliant::lifecycle::/)
         | test(/^reliant::tiering::/)
+        | test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
         | test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
     )
 """
@@ -477,13 +605,16 @@ fail-fast = false
 [profile.e2e-smoke.junit]
 path = "junit.xml"
 
-# The pagination boundary cases can stall when a server/listing regression
-# prevents the continuation request from completing. Keep the timeout scoped
-# to those known failure modes so legitimate lifecycle/tiering waits retain
-# their test-level timing budget.
+# Bound the whole pagination fixture, including startup, seeding, and cleanup;
+# its LIST deadline begins only after the seed objects have been written.
 [[profile.e2e-smoke.overrides]]
 filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
 slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
+threads-required = "num-test-threads"
+
+[[profile.e2e-smoke.overrides]]
+filter = 'package(e2e_test) & test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry)'
+threads-required = "num-test-threads"
 
 # ---------------------------------------------------------------------------
 # e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
@@ -693,6 +824,7 @@ default-filter = """
     & !test(/^replication_extension_test::/)
     & !test(/^replication_target_matrix_test::/)
     & !test(/^on_demand_migration::(concurrency_test|fault_test|interop_test|real_source_test)::/)
+    & !test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
 """
 fail-fast = false
 
@@ -721,3 +853,9 @@ test-group = 'e2e-cluster-nightly'
 [[profile.e2e-full.overrides]]
 filter = 'package(e2e_test) & (test(/^kms::kms_vault_test::/) | test(/^kms::kms_rekey_sweep_test::/) | test(/^kms::configured_roundtrip_test::test_configured_vault_kms_admin_and_versioned_cleanup$/))'
 test-group = 'e2e-vault'
+
+# Keep the storage-heavy smoke cases isolated in the full suite too. They
+# start multiple server fixtures and issue many sequential S3 operations.
+[[profile.e2e-full.overrides]]
+filter = 'package(e2e_test) & (test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/) | test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry))'
+threads-required = "num-test-threads"

.github/actions/quick-checks/action.yml

@@ -98,14 +98,13 @@ runs:
       shell: bash
       run: make script-tests
 
+    - name: Check object generation protocol model
+      shell: bash
+      run: python3 ./scripts/test_object_generation_protocol_model.py
+
     - name: Check test wiring
       shell: bash
-      run: |
-        python3 ./scripts/check_test_wiring.py --self-test
-        python3 ./scripts/check_scheduled_validation_freshness.py --self-test
-        python3 ./scripts/test_security_workflow.py
-        python3 ./scripts/test_nightly_candidate.py
-        python3 ./scripts/check_test_wiring.py
+      run: python3 ./scripts/check_test_wiring.py
 
     - name: Check no planning docs committed
       shell: bash

.github/actions/setup/action.yml

@@ -107,6 +107,8 @@ runs:
     - name: Install cargo-nextest
       if: inputs.install-test-tools == 'true'
       uses: taiki-e/install-action@96c7780c1d8a2b8723e12031def873a434d39d8d # nextest
+      with:
+        tool: cargo-nextest@0.9.138
 
     - name: Setup Rust cache
       uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2

.github/workflows/architecture-migration-rules.yml

@@ -1,61 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-name: Architecture Migration Rules
-
-on:
-  pull_request:
-    types: [ opened, synchronize, reopened, closed ]
-    branches: [ main ]
-    paths:
-      - "ARCHITECTURE.md"
-      - "docs/architecture/**"
-      - "scripts/check_architecture_migration_rules.sh"
-      - ".github/workflows/architecture-migration-rules.yml"
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  cancel-closed-pr-runs:
-    name: Cancel Closed PR Runs
-    if: github.event_name == 'pull_request' && github.event.action == 'closed'
-    runs-on: sm-standard-2
-    timeout-minutes: 10
-    steps:
-      - name: Explain cancellation run
-        run: echo "PR closed; this run only cancels older runs in the same concurrency group."
-
-  architecture-migration-rules:
-    name: Architecture Migration Rules
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    runs-on: sm-standard-2
-    timeout-minutes: 10
-    steps:
-      - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
-        with:
-          persist-credentials: false
-
-      - name: Install ripgrep
-        uses: taiki-e/install-action@7623a79cdfecb99d681017af368ca353d9f49bb5 # v2
-        with:
-          tool: ripgrep@15.2.0
-
-      - name: Check architecture migration rules
-        run: ./scripts/check_architecture_migration_rules.sh

.github/workflows/audit.yml

@@ -88,7 +88,7 @@ jobs:
   # database lookup with a second ignore list to maintain.
   cargo-deny:
     name: Cargo Deny
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
     runs-on: sm-standard-2
     timeout-minutes: 20
     steps:
@@ -131,7 +131,7 @@ jobs:
 
   workflow-pin-report:
     name: Workflow Pin Report
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
     runs-on: sm-standard-2
     timeout-minutes: 5
     steps:
@@ -203,8 +203,8 @@ jobs:
     # workflow produces, and until now it was only visible to whoever happened to
     # open the Actions tab. Same ci-8 mechanism coverage.yml and
     # e2e-replication-nightly.yml already use.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/build.yml

@@ -440,6 +440,17 @@ jobs:
             fi
           done
 
+      - name: Verify macOS binary has no dynamic liblzma dependency
+        if: matrix.platform == 'macos'
+        shell: bash
+        run: |
+          binary="target/${{ matrix.target }}/release/rustfs"
+          if otool -L "$binary" | grep -E 'liblzma.*\.dylib'; then
+            echo "macOS release binary must not depend on a dynamic liblzma library" >&2
+            exit 1
+          fi
+          "$binary" --version
+
       - name: Generate the CPU symbol catalog
         if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
         shell: bash
@@ -1167,7 +1178,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/cache-warm.yml

@@ -15,16 +15,16 @@
 # Sole writer of the Rust dependency caches that ci.yml restores.
 #
 # Why this is a separate workflow rather than steps inside ci.yml: ci.yml's
-# concurrency group cancels in-progress runs on main pushes, and merges land far
-# faster than its 70-minute pipeline. Measured over 15 consecutive main pushes:
+# concurrency group originally cancelled in-progress main runs, while merges
+# landed faster than its 70-minute pipeline. Over 15 consecutive main pushes:
 # 12 cancelled, 2 failed, 0 succeeded. A cancelled run never reaches
 # Swatinem/rust-cache's post step (cache-on-failure does not cover cancellation),
 # so the writer lanes were saving nothing and every PR paid a cold restore —
 # 11.8-20.9 minutes of "Setup Rust environment" against 0.7-3.4 warm.
 #
-# Splitting cache writing out of the test pipeline lets ci.yml keep cancelling
-# superseded runs (which is correct — nobody needs test results for a commit
-# that is already three merges behind) while the caches still get written.
+# Keep cache writing separate from validation: only these jobs publish the
+# complete feature closure for each key. Main validation now also finishes its
+# running baseline, while superseded PR attempts can still be cancelled.
 #
 # The group below deliberately does NOT cancel in progress; see the comment on
 # it for how that bounds concurrency and why it is scoped by event.
@@ -256,7 +256,7 @@ jobs:
   # belonged to it.
   warm-ci-uring:
     name: Warm ci-uring
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 60
     steps:
       - name: Checkout repository

.github/workflows/ci.yml

@@ -36,7 +36,9 @@ on:
       - "flake.lock"
   pull_request:
     types: [ opened, synchronize, reopened, closed ]
-    branches: [ main ]
+    # gateway/integration carries main's ruleset (rustfs/backlog#2734), so its
+    # pull requests must produce the same required checks.
+    branches: [ main, gateway/integration ]
   merge_group:
     types: [ checks_requested ]
   schedule:
@@ -47,12 +49,12 @@ permissions:
   contents: read
 
 # Concurrency groups are scoped per event so different triggers never cancel
-# each other: PR pushes cancel the previous run of that PR, main pushes keep
-# latest-wins semantics among themselves, and scheduled runs always complete
-# (a shared group used to let every merge kill the weekly scheduled run).
+# each other. PR pushes cancel superseded attempts; main and scheduled runs
+# finish so a busy merge stream cannot starve the complete baseline. For main,
+# GitHub retains one running run and replaces the pending run with the latest.
 concurrency:
   group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
-  cancel-in-progress: ${{ github.event_name != 'schedule' }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 env:
   CARGO_TERM_COLOR: always
@@ -128,7 +130,7 @@ jobs:
 
   test-and-lint:
     name: Workspace Test and Lint
-    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
+    if: contains(fromJSON('["full", "e2e"]'), needs.classify-changes.outputs.mode) && (github.event_name != 'pull_request' || github.event.action != 'closed')
     needs: [ quick-checks, classify-changes ]
     runs-on: sm-standard-4
     timeout-minutes: 90
@@ -146,11 +148,9 @@ jobs:
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          # Every lane in this workflow reads its cache and none writes it.
-          # cache-warm.yml is the sole writer for all four keys: this workflow
-          # cancels superseded runs on main, and a cancelled run never reaches
-          # rust-cache's post step, so writing from here saved nothing (12 of 15
-          # consecutive main-push runs were cancelled). See rustfs/backlog#1600.
+          # Every lane reads its cache; cache-warm.yml remains the sole writer
+          # with the matching feature closure. Tests never compete to replace
+          # shared caches, including when a PR attempt is cancelled.
           cache-shared-key: ci-dev
           cache-save-if: 'false'
           install-build-packaging-tools: 'false'
@@ -195,13 +195,10 @@ jobs:
           # Raised to 3 on main pushes and manual dispatches; PRs keep 2 so the
           # merge path is untouched while the experiment runs.
           #
-          # Dispatch is included because push alone cannot supply the samples:
-          # this workflow cancels superseded runs on main, and only 4 of the last
-          # 20 push-triggered Test and Lint jobs reached a terminal state — at
-          # that rate ten samples would take roughly fifty merges. The
-          # concurrency group is scoped by event_name, so a dispatched run has
-          # its own group and is not cancelled by merge traffic, which makes the
-          # sample collectable on demand rather than by waiting.
+          # During baseline collection, main pushes cancelled superseded runs:
+          # only 4 of 20 Test and Lint jobs reached a terminal state. Running
+          # main validations now finish; dispatch remains an independent group
+          # for collecting additional samples on demand.
           #
           # Baseline over 17 samples at 2:
           # median nextest/clippy step ratio 1.95, spread 1.85-2.06. The gate-2
@@ -250,11 +247,11 @@ jobs:
           mkdir -p artifacts/test-and-lint
           set +e
           timeout --verbose --signal=TERM --kill-after=30s 15m \
-            cargo test --all --doc \
+            cargo test --all --exclude e2e_test --doc \
             2>&1 | tee artifacts/test-and-lint/doctest.log
           status=${PIPESTATUS[0]}
           {
-            echo "command=cargo test --all --doc"
+            echo "command=cargo test --all --exclude e2e_test --doc"
             echo "exit_status=${status}"
             echo "finished_at=$(date --utc --iso-8601=seconds)"
             echo
@@ -281,21 +278,10 @@ jobs:
       - name: Check log-analyzer rule anchors
         run: ./scripts/check_log_analyzer_rules.sh
 
-      # Explicit gate for migration-critical suites. These tests already ran in
-      # the full nextest pass above; a single filtered nextest invocation keeps
-      # the named gate without rebuilding or re-running them one package at a time.
-      #
-      # The gate selects tests by name substring (data_movement / rebalance /
-      # decommission / source_cleanup / delete_marker), so renames can silently
-      # thin it. The script owns the filter expression and first verifies the
-      # selected-test count against the committed floor in
-      # .config/migration-gate-floor.txt before running the gate; renames or
-      # removals must update that file consciously (see the script header).
-      # Kept on the default profile (no --profile ci): a second --profile ci run
-      # would clobber target/nextest/ci/junit.xml, and none of these tests are
-      # quarantined so they gain nothing from the ci profile's retry overrides.
-      - name: Run rebalance/decommission migration proofs
-        run: ./scripts/check_migration_gate_count.sh
+      # Preserve the migration floor and require successful execution evidence
+      # from the workspace run, without compiling or executing its subset again.
+      - name: Verify rebalance/decommission migration proofs
+        run: ./scripts/check_migration_gate_count.sh evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml
 
   # This gate builds into a fresh target directory to isolate the E2E root.
   # Give its cold build a separate budget from workspace tests and migration proofs.
@@ -431,6 +417,7 @@ jobs:
         run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
 
       - name: Run rio-v2 feature tests
+        shell: bash
         env:
           # Match the main nextest lane's #5394 link-I/O guard. A cold feature
           # cache otherwise fans out enough rust-lld processes to exhaust this
@@ -440,9 +427,25 @@ jobs:
           # --profile ci so the quarantine list (and its junit flaky markers)
           # covers this leg too; the default profile is the local no-retry
           # profile and silently ignored quarantined flakes here (rustfs#6703).
-          cargo nextest run --profile ci -p rustfs -p rustfs-ecstore --features rio-v2
+          mkdir -p artifacts/test-and-lint
+          rm -f target/nextest/ci/junit.xml
+          ./scripts/ci/resource_sampler.sh start nextest-rio-v2
+          trap './scripts/ci/resource_sampler.sh stop' EXIT
+          cargo nextest run --profile ci -p rustfs -p rustfs-ecstore --features rio-v2 \
+            2>&1 | tee artifacts/test-and-lint/nextest.log
           cargo test -p rustfs --doc --features rio-v2
 
+      - name: Upload rio-v2 test reports and diagnostics
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: junit-test-and-lint-rio-v2-${{ github.run_number }}
+          path: |
+            target/nextest/ci/junit.xml
+            artifacts/test-and-lint
+          retention-days: 3
+          if-no-files-found: error
+
   connect-short-credential-boundary:
     name: Connect Short Credential Boundary
     if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
@@ -508,19 +511,17 @@ jobs:
     runs-on: sm-standard-4
     timeout-minutes: 90
     strategy:
-      # On a PR, one failing protocol leg is enough to know the PR is not ready,
-      # so stop the sibling leg instead of paying another ~40 minutes for it.
-      # Everywhere else (main pushes, the merge queue, the weekly schedule) keep
-      # the full signal: there we want to know whether swift AND sftp are broken,
-      # not just whichever failed first. This is the only part of the early-stop
-      # work that also covers fork PRs, since it needs no token.
-      fail-fast: ${{ github.event_name == 'pull_request' }}
+      # Preserve independent results when one protocol fails, so diagnosis
+      # and a focused fix do not require rebuilding an interrupted sibling.
+      fail-fast: false
       matrix:
         features:
           - name: swift
             flags: "--features swift"
           - name: sftp
             flags: "--features sftp"
+          - name: http3
+            flags: "--features http3,swift"
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
@@ -537,27 +538,50 @@ jobs:
           cache-save-if: 'false'
           install-build-packaging-tools: 'false'
 
-      - name: Protect Connect test home
-        run: chmod go-w "$(realpath "$HOME")"
+      - name: Prepare protocol test state
+        run: |
+          rm -f target/nextest/ci/junit.xml
+          chmod go-w "$(realpath "$HOME")"
 
       - name: Run clippy with ${{ matrix.features.name }}
         run: |
+          ./scripts/ci/resource_sampler.sh start clippy
+          trap './scripts/ci/resource_sampler.sh stop' EXIT
           cargo clippy -p rustfs -p rustfs-protocols --all-targets ${{ matrix.features.flags }} -- -D warnings
 
       - name: Run tests with ${{ matrix.features.name }}
+        id: protocol-tests
+        shell: bash
         env:
           # Keep feature-test linking under the same bounded concurrency as the
           # main nextest lane; Clippy is metadata-only and needs no such limit.
           CARGO_BUILD_JOBS: "2"
+          SAMPLER_INTERVAL_SECS: "15"
         run: |
           # --profile ci so the quarantine list (and its junit flaky markers)
           # covers this leg too; the default profile is the local no-retry
           # profile and silently ignored quarantined flakes here (rustfs#6703).
-          cargo nextest run --profile ci -p rustfs -p rustfs-protocols ${{ matrix.features.flags }}
+          mkdir -p artifacts/protocol-tests
+          ./scripts/ci/resource_sampler.sh start nextest
+          trap './scripts/ci/resource_sampler.sh stop' EXIT
+          cargo nextest run --profile ci -p rustfs -p rustfs-protocols ${{ matrix.features.flags }} \
+            2>&1 | tee artifacts/protocol-tests/nextest.log
+
+      - name: Upload protocol test reports and diagnostics
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: junit-test-and-lint-${{ matrix.features.name }}-${{ github.run_number }}-${{ github.run_attempt }}
+          path: |
+            target/nextest/ci/junit.xml
+            artifacts/protocol-tests
+            artifacts/test-and-lint
+          retention-days: 3
+          if-no-files-found: warn
 
   build-rustfs-debug-binary:
     name: Build RustFS Debug Binary
-    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
+    if: contains(fromJSON('["full", "e2e"]'), needs.classify-changes.outputs.mode) && (github.event_name != 'pull_request' || github.event.action != 'closed')
     needs: [ quick-checks, classify-changes ]
     runs-on: sm-standard-4
     timeout-minutes: 30
@@ -810,6 +834,10 @@ jobs:
           python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}"
           ./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
 
+      # Keep E2E documentation coverage with its already-built package graph.
+      - name: Run E2E documentation tests
+        run: cargo test -p e2e_test --doc
+
       # PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
       # profile.e2e-smoke default-filter in .config/nextest.toml is the single
       # wiring mechanism for e2e tests in CI — extend that filter instead of
@@ -819,18 +847,22 @@ jobs:
         env:
           NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
           RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
+          SAMPLER_INTERVAL_SECS: "15"
         run: |
+          ./scripts/ci/resource_sampler.sh start e2e-smoke
+          trap './scripts/ci/resource_sampler.sh stop' EXIT
           python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" \
             --status-level all --final-status-level all --failure-output final
 
       - name: Upload e2e smoke diagnostics
-        if: failure()
+        if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: e2e-smoke-diagnostics-${{ github.run_number }}
           path: |
             ${{ runner.temp }}/rustfs-e2e-smoke-logs/
             ${{ runner.temp }}/rustfs-e2e-smoke-list.json
+            artifacts/test-and-lint
           if-no-files-found: warn
 
       - name: Upload e2e smoke JUnit report
@@ -1005,6 +1037,7 @@ jobs:
           RUSTFS_E2E_STARTUP_CAS_BINARY: ${{ runner.temp }}/rustfs-startup-cas-input/rustfs
           RUSTFS_E2E_STARTUP_CAS_BUILD_MANIFEST: ${{ runner.temp }}/rustfs-startup-cas-input/rustfs.e2e-startup-cas-build.json
           RUSTFS_E2E_STARTUP_CAS_ARTIFACT_DIR: ${{ runner.temp }}/rustfs-startup-cas-evidence
+          RUSTFS_HEAL_CHAOS_LOG_ROOT: ${{ runner.temp }}/rustfs-heal-chaos-logs
         run: python3 scripts/e2e_binary.py run --binary "$RUSTFS_E2E_STARTUP_CAS_BINARY" --features e2e-test-hooks -- cargo nextest run --profile e2e-full -p e2e_test
 
       - name: Upload junit
@@ -1028,6 +1061,15 @@ jobs:
           if-no-files-found: warn
           retention-days: 7
 
+      - name: Upload heal cluster logs
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: e2e-full-heal-logs-${{ github.run_number }}
+          path: ${{ runner.temp }}/rustfs-heal-chaos-logs
+          if-no-files-found: warn
+          retention-days: 7
+
   e2e-tests-rio-v2:
     name: End-to-End Tests (rio-v2)
     # Inherits the schedule/dispatch-only gate through needs: on every other
@@ -1270,7 +1312,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/connect-health-service-acceptance.yml

@@ -172,7 +172,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli'} and len(entries) == 2
+                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 4
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None
                   pathlib.Path('binary').mkdir()

.github/workflows/coverage.yml

@@ -128,7 +128,7 @@ jobs:
     # manual workflow_dispatch runs stay quiet so a debugging run never files a
     # spurious alert.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/docker.yml

@@ -551,54 +551,6 @@ jobs:
           if-no-files-found: error
           retention-days: 30
 
-  # Docker Hub's overview is a separate `full_description` field that
-  # `docker push` never touches, so it drifted from README.md and kept
-  # linking to pages that no longer exist (#7995). Republish it from the
-  # commit the images were built from, only when images were pushed.
-  sync-dockerhub-description:
-    name: Sync Docker Hub Description
-    needs: [ build-check, build-docker ]
-    if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
-    runs-on: sm-standard-2
-    timeout-minutes: 10
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
-        with:
-          persist-credentials: false
-          ref: ${{ needs.build-check.outputs.source_ref }}
-          sparse-checkout: README.md
-          sparse-checkout-cone-mode: false
-
-      - name: Sync Docker Hub description
-        uses: peter-evans/dockerhub-description@1b9a80c056b620d92cedb9d9b5a223409c68ddfa # v5.0.0
-        with:
-          username: ${{ env.DOCKERHUB_USERNAME }}
-          password: ${{ secrets.DOCKERHUB_TOKEN }}
-          repository: ${{ env.REGISTRY_DOCKERHUB }}
-          readme-filepath: ./README.md
-          # README.md links to docs/ and CONTRIBUTING.md by relative path;
-          # rewrite those to github.com URLs so they resolve on Docker Hub.
-          enable-url-completion: true
-
-      # Docker Hub caps `full_description` at 25,000 bytes and the action
-      # truncates to fit with only a warning, which would publish a README
-      # cut off mid-sentence. README.md is already near the cap, so read the
-      # published overview back and fail if it hit the limit.
-      - name: Check published description was not truncated
-        env:
-          REPOSITORY: ${{ env.REGISTRY_DOCKERHUB }}
-        run: |
-          limit=25000
-          published=$(curl -sSf "https://hub.docker.com/v2/repositories/${REPOSITORY}/" \
-            | jq -j '.full_description' | wc -c | tr -d ' ')
-          echo "Published overview: ${published} bytes (Docker Hub limit: ${limit})"
-          # Truncation lands within one UTF-8 character of the cap.
-          if [ "$published" -ge $((limit - 4)) ]; then
-            echo "::error file=README.md::Docker Hub overview was truncated at ${limit} bytes; trim README.md before the next release."
-            exit 1
-          fi
-
   # Docker build summary
   docker-summary:
     name: Docker Build Summary

.github/workflows/e2e-distributed.yml

@@ -38,6 +38,7 @@ on:
       - "Cargo.lock"
       - "Cargo.toml"
       - ".config/nextest.toml"
+      - ".config/e2e-distributed-selection.txt"
       - ".github/workflows/e2e-distributed.yml"
       - "crates/audit/**"
       - "crates/common/**"
@@ -77,6 +78,7 @@ concurrency:
 
 jobs:
   distributed:
+    if: github.repository == 'rustfs/rustfs'
     name: Distributed 4-node 4-disk e2e
     # GitHub-hosted VM: loop and tmpfs mounts work here. sm-standard-4 is an
     # ARC pod and rejects both (`mount -o loop` ENOENT, tmpfs "read-only").
@@ -166,11 +168,13 @@ jobs:
           FILTER: ${{ inputs.filter }}
         run: |
           set -euo pipefail
+          rm -f target/nextest/e2e-distributed/junit.xml
           if [ -n "${FILTER}" ]; then
             python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-distributed -p e2e_test -E "${FILTER}"
           else
             python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-distributed -p e2e_test --no-tests=fail
           fi
+          test -s target/nextest/e2e-distributed/junit.xml
 
       - name: Upload distributed e2e diagnostics
         if: always()
@@ -199,8 +203,8 @@ jobs:
   alert-on-failure:
     name: Alert on scheduled failure
     needs: [distributed]
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/e2e-replication-nightly.yml

@@ -53,6 +53,7 @@ concurrency:
 
 jobs:
   repl-nightly:
+    if: github.repository == 'rustfs/rustfs'
     name: Replication e2e (nightly)
     # dual-node tests spawn two full rustfs servers each; the extra headroom
     # over the per-PR sm-standard-2 e2e job keeps the parallel process fan-out
@@ -119,6 +120,7 @@ jobs:
           if-no-files-found: ignore
 
   cluster-nightly:
+    if: github.repository == 'rustfs/rustfs'
     name: Cluster fault e2e (nightly)
     runs-on: sm-standard-4
     timeout-minutes: 90
@@ -168,6 +170,7 @@ jobs:
           if-no-files-found: warn
 
   protocols-nightly:
+    if: github.repository == 'rustfs/rustfs'
     name: Protocol e2e (nightly)
     runs-on: sm-standard-4
     timeout-minutes: 90
@@ -229,8 +232,8 @@ jobs:
     # Only scheduled runs open/append the tracking issue (backlog#1149 ci-8);
     # manual workflow_dispatch runs stay quiet so a debugging run never files a
     # spurious alert.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/e2e-s3tests.yml

@@ -453,7 +453,7 @@ jobs:
     # job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
     # dispatch failures are already watched by a human.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/e2e-upgrade.yml

@@ -51,8 +51,53 @@ env:
   UPGRADE_SOURCE_SHA256: 3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d
 
 jobs:
+  build:
+    name: Build upgrade candidate
+    runs-on: sm-standard-2
+    timeout-minutes: 60
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          cache-shared-key: e2e-upgrade-server
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          install-build-packaging-tools: "false"
+          install-test-tools: "false"
+
+      - name: Download pinned previous release
+        run: |
+          mkdir -p target/debug
+          archive="target/debug/$UPGRADE_SOURCE_ASSET"
+          curl --fail --location --retry 3 --output "$archive" \
+            "https://github.com/${GITHUB_REPOSITORY}/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}"
+          echo "$UPGRADE_SOURCE_SHA256  $archive" | sha256sum --check --strict
+
+      - name: Build current RustFS binary
+        run: python3 scripts/e2e_binary.py build
+
+      - name: Upload verified upgrade candidate
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: upgrade-candidate-${{ github.run_id }}
+          path: |
+            target/debug/rustfs
+            target/debug/rustfs.e2e.json
+            target/debug/${{ env.UPGRADE_SOURCE_ASSET }}
+          if-no-files-found: error
+          retention-days: 3
+          compression-level: 0
+          overwrite: true
+
   upgrade:
     name: ${{ matrix.name }}
+    needs: build
     strategy:
       fail-fast: false
       matrix:
@@ -63,31 +108,24 @@ jobs:
           # the CI required-check names. UPGRADE_SOURCE_VERSION above is the
           # single source of truth for which release they actually run against.
           - name: Direct upgrade from the previous release
-            cache_key: e2e-direct-upgrade
             test: direct_upgrade_from_rc2_preserves_object_contracts
             artifact: direct-upgrade
           - name: Mixed-version rolling upgrade from the previous release
-            cache_key: e2e-mixed-version-upgrade
             test: rolling_upgrade_from_rc2_preserves_mixed_version_contracts
             artifact: mixed-version-upgrade
           - name: Bucket configuration survives the upgrade
-            cache_key: e2e-bucket-config-upgrade
             test: direct_upgrade_from_previous_release_preserves_bucket_configuration
             artifact: bucket-config-upgrade
           - name: Rollback reads current bucket metadata
-            cache_key: e2e-bucket-config-rollback
             test: rollback_to_previous_release_reads_current_bucket_metadata
             artifact: bucket-config-rollback
           - name: ODM configuration recovery after rc.5 rollback
-            cache_key: e2e-odm-config-rollback
             test: rc5_rollback_requires_restoring_odm_configuration
             artifact: odm-config-rollback
           - name: Multipart layouts survive the rc.5 upgrade
-            cache_key: e2e-multipart-layout-upgrade
             test: direct_upgrade_from_rc5_preserves_multipart_layouts
             artifact: multipart-layout-upgrade
           - name: rc.5 multipart replication baseline
-            cache_key: e2e-multipart-layout-baseline
             test: rc5_baseline_replicates_multipart_layouts
             artifact: multipart-layout-baseline
     runs-on: sm-standard-2
@@ -103,19 +141,28 @@ jobs:
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
-          cache-shared-key: ${{ matrix.cache_key }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          cache-shared-key: e2e-upgrade-tests
+          # One main-branch consumer warms the shared test-harness dependencies.
+          cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.artifact == 'direct-upgrade' }}
           install-build-packaging-tools: "false"
+          install-test-tools: "false"
 
-      - name: Download pinned previous release
+      - name: Download verified upgrade candidate
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: upgrade-candidate-${{ github.run_id }}
+          path: target/debug
+
+      - name: Restore candidate executable bit
+        run: chmod +x target/debug/rustfs
+
+      - name: Verify and unpack pinned previous release
         env:
           SOURCE_DIR: ${{ runner.temp }}/rustfs-upgrade-source
         run: |
           set -euo pipefail
           mkdir -p "$SOURCE_DIR"
-          archive="$SOURCE_DIR/$UPGRADE_SOURCE_ASSET"
-          curl --fail --location --retry 3 --output "$archive" \
-            "https://github.com/${GITHUB_REPOSITORY}/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}"
+          archive="target/debug/$UPGRADE_SOURCE_ASSET"
           echo "$UPGRADE_SOURCE_SHA256  $archive" | sha256sum --check --strict
           unzip -q "$archive" -d "$SOURCE_DIR"
           chmod +x "$SOURCE_DIR/rustfs"
@@ -123,14 +170,12 @@ jobs:
           echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV"
           echo "RUSTFS_E2E_LOG_DIR=$RUNNER_TEMP/rustfs-upgrade-logs" >> "$GITHUB_ENV"
 
-      - name: Build current RustFS binary
-        run: |
-          python3 scripts/e2e_binary.py build
-
       - name: Run upgrade compatibility test
         env:
           RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
+          NO_PROXY: 127.0.0.1,localhost,::1
         run: |
+          export no_proxy="$NO_PROXY"
           python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
             "upgrade_compatibility_test::${{ matrix.test }}" \
             -- --ignored --exact --nocapture

.github/workflows/functional-chain-health.yml

@@ -11,6 +11,7 @@ concurrency:
   cancel-in-progress: false
 jobs:
   collect:
+    if: github.repository == 'rustfs/rustfs'
     runs-on: smoke-testing
     timeout-minutes: 15
     steps:

.github/workflows/fuzz.yml

@@ -72,9 +72,11 @@ jobs:
   fuzz-build:
     name: Build Fuzz Harness
     if: >
-      (github.event_name == 'pull_request' && github.event.action != 'closed') ||
-      github.event_name == 'schedule' ||
-      github.event_name == 'workflow_dispatch'
+      github.repository == 'rustfs/rustfs' && (
+        (github.event_name == 'pull_request' && github.event.action != 'closed') ||
+        github.event_name == 'schedule' ||
+        github.event_name == 'workflow_dispatch'
+      )
     runs-on: sm-standard-4
     timeout-minutes: 60
     env:
@@ -239,8 +241,8 @@ jobs:
     # `always()` is required: without it this job is skipped when a needed
     # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
     # ci-8); PR and manual dispatch failures are already watched by a human.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/gateway-integration-sync.yml

@@ -0,0 +1,197 @@
+# Copyright 2026 RustFS Team
+# SPDX-License-Identifier: Apache-2.0
+
+name: Gateway Integration Sync
+
+# Merges `main` into the long-lived `gateway/integration` branch every weekday
+# so the s3s -> gateway migration (rustfs/backlog#2734, task T0.1 in #2736)
+# never accumulates merge debt. A clean merge lands as a `--no-ff` merge commit
+# pushed straight to the branch. A conflicting merge is aborted and surfaced as
+# a pull request from `sync/<source>-<date>`, whose head is the un-merged
+# source commit, so the conflict is resolved the same day, in review, with no
+# other change riding along. The run itself stays green in that case: the pull
+# request is the signal. Any other failure (push rejected, pull request not
+# created) fails the run and, on the schedule, files the shared failure issue.
+#
+# Resolving a conflict pull request: merge `origin/gateway/integration` into
+# the sync branch, resolve, push, get the review. Land it with a fast-forward
+# push of the resolved merge commit to `gateway/integration`, not with the
+# squash button: a squash drops the merge ancestry and the same hunks conflict
+# again on the next run.
+#
+# Write scope: only `gateway/integration` and `sync/*` are ever pushed, and
+# both push targets below are literal refs. The pushes use a one-hour
+# installation token of the organisation's own GitHub App (contents: write)
+# because that app is the only non-admin bypass actor of the
+# `gateway/integration` ruleset; GITHUB_TOKEN cannot be a ruleset bypass actor,
+# so it cannot push through the required reviews and checks. The conflict pull
+# request is opened with FLAKE_UPDATE_TOKEN, the PAT that already opens the
+# weekly flake.lock pull requests, because this repository keeps "Allow GitHub
+# Actions to create and approve pull requests" switched off. GITHUB_TOKEN is
+# therefore read-only here.
+#
+# `schedule` fires only from the default branch, so the cron is live once this
+# file exists on `main`. `workflow_dispatch` runs from any ref that carries the
+# file and always acts on `gateway/integration`.
+
+on:
+  schedule:
+    - cron: "0 2 * * 1-5" # Weekdays 02:00 UTC
+  workflow_dispatch:
+    inputs:
+      source_ref:
+        description: >-
+          Branch to merge into gateway/integration. The schedule always merges
+          main; only the conflict acceptance test (rustfs/backlog#2736) sets
+          another branch.
+        required: false
+        default: main
+        type: string
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}
+  cancel-in-progress: false
+
+env:
+  TARGET_BRANCH: gateway/integration
+
+jobs:
+  sync:
+    name: Merge main into gateway/integration
+    if: github.repository == 'rustfs/rustfs'
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - name: Create the branch push token
+        id: push-token
+        uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3
+        with:
+          app-id: ${{ vars.CLA_BOT_APP_ID }}
+          private-key: ${{ secrets.CLA_BOT_APP_PRIVATE_KEY }}
+          owner: ${{ github.repository_owner }}
+          repositories: rustfs
+          permission-contents: write
+
+      - name: Checkout gateway/integration
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          # persist-credentials-exempt: this checkout's token IS the push
+          # credential — the merge commit and the sync branch are pushed with it
+          # below. It is a one-hour installation token scoped to contents.
+          ref: gateway/integration
+          fetch-depth: 0
+          token: ${{ steps.push-token.outputs.token }}
+
+      - name: Configure the bot identity
+        run: |
+          git config user.name "rustfs-cla-assistant[bot]"
+          git config user.email "273121464+rustfs-cla-assistant[bot]@users.noreply.github.com"
+
+      - name: Merge the source branch
+        id: merge
+        env:
+          SOURCE_REF: ${{ inputs.source_ref || 'main' }}
+        run: |
+          set -euo pipefail
+          if ! [[ "$SOURCE_REF" =~ ^[A-Za-z0-9][A-Za-z0-9._/-]*$ ]] \
+            || [[ "$SOURCE_REF" == *..* ]] \
+            || [[ "$SOURCE_REF" == "$TARGET_BRANCH" ]]; then
+            echo "::error::refusing source_ref '${SOURCE_REF}'"
+            exit 1
+          fi
+          git fetch origin "refs/heads/${SOURCE_REF}:refs/remotes/origin/${SOURCE_REF}"
+          source_sha=$(git rev-parse "origin/${SOURCE_REF}")
+          target_sha=$(git rev-parse HEAD)
+          {
+            echo "source_ref=${SOURCE_REF}"
+            echo "source_sha=${source_sha}"
+            echo "target_sha=${target_sha}"
+          } >> "$GITHUB_OUTPUT"
+          if git merge --no-ff --no-edit "origin/${SOURCE_REF}"; then
+            if [[ "$(git rev-parse HEAD)" == "$target_sha" ]]; then
+              echo "result=up-to-date" >> "$GITHUB_OUTPUT"
+              echo "\`${TARGET_BRANCH}\` already contains \`${SOURCE_REF}\` (${source_sha}); nothing to push." >> "$GITHUB_STEP_SUMMARY"
+            else
+              echo "result=merged" >> "$GITHUB_OUTPUT"
+            fi
+            exit 0
+          fi
+          git diff --name-only --diff-filter=U > "$RUNNER_TEMP/conflicts.txt"
+          git merge --abort
+          if [[ ! -s "$RUNNER_TEMP/conflicts.txt" ]]; then
+            echo "::error::git merge failed without reporting conflicting paths; this is not a conflict"
+            exit 1
+          fi
+          echo "result=conflict" >> "$GITHUB_OUTPUT"
+
+      - name: Push the merge commit
+        if: steps.merge.outputs.result == 'merged'
+        env:
+          SOURCE_REF: ${{ steps.merge.outputs.source_ref }}
+          SOURCE_SHA: ${{ steps.merge.outputs.source_sha }}
+        run: |
+          set -euo pipefail
+          git push origin HEAD:refs/heads/gateway/integration
+          echo "Merged \`${SOURCE_REF}\` (${SOURCE_SHA}) into \`${TARGET_BRANCH}\` as $(git rev-parse HEAD)." >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Open the conflict pull request
+        if: steps.merge.outputs.result == 'conflict'
+        env:
+          GH_TOKEN: ${{ secrets.FLAKE_UPDATE_TOKEN }}
+          SOURCE_REF: ${{ steps.merge.outputs.source_ref }}
+          SOURCE_SHA: ${{ steps.merge.outputs.source_sha }}
+          TARGET_SHA: ${{ steps.merge.outputs.target_sha }}
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+        run: |
+          set -euo pipefail
+          date_utc=$(date -u +%Y-%m-%d)
+          sync_branch="sync/${SOURCE_REF//\//-}-${date_utc}"
+          git push origin "refs/remotes/origin/${SOURCE_REF}:refs/heads/${sync_branch}"
+          {
+            echo "The scheduled merge of \`${SOURCE_REF}\` into \`${TARGET_BRANCH}\` stopped on conflicts and was aborted; \`${TARGET_BRANCH}\` is unchanged."
+            echo
+            echo "This branch is the un-merged \`${SOURCE_REF}\` head. Merge \`origin/${TARGET_BRANCH}\` into it, resolve the files below, push, and get the review. Land the resolved merge commit with a fast-forward push to \`${TARGET_BRANCH}\`; a squash merge drops the merge ancestry and the same hunks conflict again on the next run. Do not include any change that is not conflict resolution."
+            echo
+            echo "- Run: ${RUN_URL}"
+            echo "- Source: \`${SOURCE_REF}\` at ${SOURCE_SHA}"
+            echo "- Target: \`${TARGET_BRANCH}\` at ${TARGET_SHA}"
+            echo
+            echo "## Conflicting files"
+            echo
+            echo "Output of \`git diff --name-only --diff-filter=U\` captured before \`git merge --abort\`:"
+            echo
+            sed 's/^/- /' "$RUNNER_TEMP/conflicts.txt"
+          } > "$RUNNER_TEMP/pr-body.md"
+          title="sync: merge ${SOURCE_REF} ${date_utc} (conflicts)"
+          existing=$(gh pr list --repo "$GITHUB_REPOSITORY" --base "$TARGET_BRANCH" --head "$sync_branch" --state open --json number --jq '.[0].number // empty')
+          if [[ -n "$existing" ]]; then
+            gh pr edit "$existing" --repo "$GITHUB_REPOSITORY" --title "$title" --body-file "$RUNNER_TEMP/pr-body.md"
+            pr_url=$(gh pr view "$existing" --repo "$GITHUB_REPOSITORY" --json url --jq .url)
+          else
+            pr_url=$(gh pr create --repo "$GITHUB_REPOSITORY" --base "$TARGET_BRANCH" --head "$sync_branch" --title "$title" --body-file "$RUNNER_TEMP/pr-body.md")
+          fi
+          echo "Conflicts in \`${SOURCE_REF}\` -> \`${TARGET_BRANCH}\`; pull request: ${pr_url}" >> "$GITHUB_STEP_SUMMARY"
+
+  alert-on-failure:
+    name: Alert on scheduled failure
+    needs: [sync]
+    # Same ci-8 mechanism audit.yml and coverage.yml use: a scheduled run that
+    # could not push or could not open the conflict pull request must not stay
+    # visible only in the Actions tab.
+    if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+      - name: Open or update failure-tracking issue
+        uses: ./.github/actions/schedule-failure-issue
+        with:
+          github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}

.github/workflows/helm-package.yml

@@ -130,6 +130,21 @@ jobs:
       - name: Set up Helm
         uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
 
+      - name: Install yq
+        run: |
+          set -euo pipefail
+          yq_version="4.54.1"
+          yq_sha256="8e34fc298390875de416e6a4afcb8cabeceb25d9aa8506c1a2f9353cf702ea5f"
+          tools_dir="$RUNNER_TEMP/rustfs-helm-tools"
+          mkdir -p "$tools_dir"
+          curl --fail --silent --show-error --location --retry 3 \
+            "https://github.com/mikefarah/yq/releases/download/v${yq_version}/yq_linux_amd64" \
+            --output "$tools_dir/yq"
+          printf '%s  %s\n' "$yq_sha256" "$tools_dir/yq" | sha256sum --check -
+          chmod +x "$tools_dir/yq"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+          "$tools_dir/yq" --version
+
       - name: Test Helm Chart Templates
         run: ./scripts/test_helm_templates.sh
 
@@ -145,7 +160,9 @@ jobs:
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: helm-package
-          path: helm/rustfs/*.tgz
+          path: |
+            helm/rustfs/*.tgz
+            helm/rustfs/README.md
           retention-days: 1
 
   publish-helm-package:

.github/workflows/minio-interop.yml

@@ -60,9 +60,9 @@ permissions:
 jobs:
   minio-interop:
     name: MinIO interop (EC + SSE read parity)
-    # Skip on forks: needs the repo's runners and is not a contributor gate.
+    # Keep this scheduled interoperability lane scoped to the upstream repo.
     if: github.repository == 'rustfs/rustfs'
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 40
     env:
       # Fixed 32-byte test KMS key baked into the fixture lab; not a secret.
@@ -131,7 +131,7 @@ jobs:
     name: Alert on scheduled failure
     needs: [minio-interop]
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/mint.yml

@@ -258,7 +258,7 @@ jobs:
     # job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
     # dispatch failures are already watched by a human.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/nightly-gnu.yml

@@ -43,6 +43,7 @@ env:
 
 jobs:
   resolve-source:
+    if: github.repository == 'rustfs/rustfs'
     name: Resolve nightly source
     runs-on: sm-standard-2
     timeout-minutes: 10
@@ -496,9 +497,11 @@ jobs:
     name: Alert on scheduled failure
     needs: [build, kms-vault-lane, kms-vault-ha-failover]
     if: >-
-      always() && github.event_name == 'schedule' &&
-      (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+      github.repository == 'rustfs/rustfs' && (
+        always() && github.event_name == 'schedule' &&
+        (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
+      )
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/nix-flake-update.yml

@@ -32,6 +32,7 @@ concurrency:
 
 jobs:
   update-flake:
+    if: github.repository == 'rustfs/rustfs'
     name: Update flake.lock
     runs-on: sm-standard-2
     timeout-minutes: 90

.github/workflows/oidc-keycloak.yml

@@ -51,7 +51,7 @@ concurrency:
 jobs:
   oidc-keycloak-live:
     name: OIDC Keycloak live gate
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 60
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
@@ -72,6 +72,11 @@ jobs:
       - name: Build RustFS
         run: cargo build --locked -p rustfs --bin rustfs
 
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.12"
+
       - name: Install pinned request signer
         run: |
           python3 -m pip install --user --upgrade pip "awscurl==0.44"
@@ -95,7 +100,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (needs.oidc-keycloak-live.result == 'failure' || needs.oidc-keycloak-live.result == 'cancelled')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/on-demand-migration-interop.yml

@@ -69,10 +69,9 @@ env:
 jobs:
   minio-source:
     name: MinIO source (read-through, list-through, backfill)
-    # Skip on forks: needs this repository's runners and is not a contributor
-    # gate.
+    # Keep this scheduled interoperability lane scoped to the upstream repo.
     if: github.repository == 'rustfs/rustfs'
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 90
     env:
       NO_PROXY: 127.0.0.1,localhost
@@ -108,11 +107,15 @@ jobs:
         run: |
           set -euo pipefail
           mkdir -p artifacts/odm-interop/minio
+          docker build -t rustfs-odm-interop-minio \
+            -f crates/rio-v2/tests/minio_fixture_lab/Dockerfile \
+            crates/rio-v2/tests/minio_fixture_lab
           docker run -d --name rustfs-odm-interop-minio \
+            --entrypoint /usr/local/bin/minio \
             -e "MINIO_ROOT_USER=${MINIO_ROOT_USER}" \
             -e "MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD}" \
             -p 9100:9000 \
-            minio/minio:RELEASE.2025-09-07T16-13-09Z server /data
+            rustfs-odm-interop-minio server /data
           for _ in $(seq 1 120); do
             curl -fsS http://127.0.0.1:9100/minio/health/live >/dev/null 2>&1 && break
             sleep 1
@@ -300,7 +303,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/package.yml

@@ -123,7 +123,7 @@ jobs:
           semver_core='(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)'
           prerelease_id='(alpha|beta|rc)\.(0|[1-9][0-9]*)'
           if [[ -n "$TAG" ]]; then
-            if [[ "$TAG" =~ ^${semver_core}-${prerelease_id}-preview\.(0|[1-9][0-9]*)$ ]]; then
+            if [[ "$TAG" =~ ^${semver_core}(-${prerelease_id})?-preview\.(0|[1-9][0-9]*)$ ]]; then
               BUILD_TYPE=preview
             elif [[ "$TAG" =~ ^${semver_core}-${prerelease_id}$ ]]; then
               BUILD_TYPE=prerelease
@@ -504,7 +504,7 @@ jobs:
           retention-days: 30
 
       - name: Upload packages to Cloudflare R2
-        if: env.R2_ACCESS_KEY_ID != ''
+        if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''
         env:
           R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
           R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}

.github/workflows/performance-ab.yml

@@ -49,6 +49,7 @@ env:
 
 jobs:
   warp-ab:
+    if: github.repository == 'rustfs/rustfs'
     name: Warp A/B budget gate
     runs-on: sm-standard-2
     # A normal nightly restores the last successful binary and builds only the
@@ -364,9 +365,11 @@ jobs:
     # composite action already reports cancelled/timed-out jobs in the issue
     # body.
     if: >-
-      always() && github.event_name == 'schedule' &&
-      (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+      github.repository == 'rustfs/rustfs' && (
+        always() && github.event_name == 'schedule' &&
+        (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
+      )
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/runner-hygiene.yml

@@ -42,6 +42,7 @@ concurrency:
 
 jobs:
   check-ephemerality:
+    if: github.repository == 'rustfs/rustfs'
     name: Check runner ephemerality
     runs-on: sm-standard-2
     timeout-minutes: 15
@@ -65,8 +66,8 @@ jobs:
     # Same ci-8 mechanism as coverage.yml, audit.yml and the nightly lanes:
     # scheduled runs file a tracking issue, manual dispatch stays quiet so
     # debugging never produces a spurious alert.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/rustfs-pool-expand-test.yml

@@ -266,7 +266,10 @@ jobs:
         # the previous 45m step cap killed four consecutive runs mid-rebalance
         # with steps 1-7 green. 240m covers the observed pace (rebalance 1/4
         # in ~21m) plus one decommission pass; override per dispatch.
-        timeout-minutes: ${{ inputs.pool_timeout_minutes || 240 }}
+        # fromJSON coerces the string dispatch input to the number that
+        # timeout-minutes requires; the bare || form yielded the string '240'
+        # and the runner rejected the template (Unexpected value '240').
+        timeout-minutes: ${{ fromJSON(inputs.pool_timeout_minutes || '240') }}
         # Case failures keep the run green: the report and the backlog
         # issue manager carry the product signal.
         continue-on-error: true

.github/workflows/rustfs-security-test.yml

@@ -106,7 +106,7 @@ jobs:
         if: ${{ inputs.chain_manifest != '' }}
         env:
           CHAIN_MANIFEST: ${{ inputs.chain_manifest }}
-        run: python3 scripts/functional_chain_evidence.py consume
+        run: python3 rustfs-repo/scripts/functional_chain_evidence.py consume
 
       - name: Checkout auto-testing scripts
         uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -389,7 +389,7 @@ jobs:
           CHAIN_TEST_OUTCOME: ${{ steps.test.outcome }}
           CHAIN_REPORT_OUTCOME: ${{ steps.report.outcome }}
         run: >-
-          python3 scripts/functional_chain_evidence.py record --suite security
+          python3 rustfs-repo/scripts/functional_chain_evidence.py record --suite security
           --report "${SECURITY_ARTIFACTS_DIR}/suite-report.md"
           --output "${RUNNER_TEMP}/chain-security-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/security.json"
 

.github/workflows/schedule-failure-alert-drill.yml

@@ -58,7 +58,7 @@ jobs:
     # Mirrors the consumer wiring, minus the schedule-event guard (this
     # workflow is dispatch-only by design).
     if: always() && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/scheduled-validation-freshness.yml

@@ -28,6 +28,7 @@ concurrency:
 
 jobs:
   check-freshness:
+    if: github.repository == 'rustfs/rustfs'
     name: Check scheduled validation freshness
     # Stay on GitHub-hosted runners: this monitor must keep running when the
     # self-hosted sm-standard-* pool is down, which is exactly when the

.github/workflows/scheduled-validation-watchdog.yml

@@ -39,6 +39,7 @@ jobs:
   alert-on-incomplete-run:
     name: Alert on incomplete scheduled run
     if: >-
+      github.repository == 'rustfs/rustfs' &&
       github.event.workflow_run.event == 'schedule' &&
       github.event.workflow_run.conclusion != 'success' &&
       github.event.workflow_run.conclusion != 'failure'

.github/workflows/stale.yml

@@ -26,6 +26,7 @@ on:
 
 jobs:
   stale:
+    if: github.repository == 'rustfs/rustfs'
     runs-on: sm-standard-2
     timeout-minutes: 30
     steps:

.github/workflows/star-history.yml

@@ -14,6 +14,7 @@ concurrency:
 
 jobs:
   update:
+    if: github.repository == 'rustfs/rustfs'
     runs-on: sm-standard-2
     timeout-minutes: 10
     steps:

.github/workflows/targets-integration.yml

@@ -40,6 +40,7 @@ env:
 
 jobs:
   targets-live:
+    if: github.repository == 'rustfs/rustfs'
     name: PostgreSQL, MySQL, AMQP, and NATS
     runs-on: sm-standard-2
     timeout-minutes: 90
@@ -175,9 +176,11 @@ jobs:
     name: Alert on scheduled failure
     needs: [targets-live]
     if: >-
-      always() && github.event_name == 'schedule' &&
-      (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+      github.repository == 'rustfs/rustfs' && (
+        always() && github.event_name == 'schedule' &&
+        (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
+      )
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
       contents: read

AGENTS.md

@@ -17,6 +17,8 @@ rules. A skill cannot expand the user's requested scope or grant authorization.
 
 - Inquiry, diagnosis, review, and planning tasks are read-only unless the user
   explicitly requests changes.
+- Ordinary diff/commit reviews use the finding standard below directly; no
+  generic review skill is required. PR and high-risk reviews retain their workflows.
 - For implementation, read the relevant code, tests, and local guidance, then
   make the smallest change that satisfies the request.
 - State assumptions only when they affect behavior or verification. Ask only
@@ -88,6 +90,9 @@ runtime/build output:
 
 - Run `cargo fmt --all --check` for Rust changes.
 - Run the narrowest test that exercises the changed behavior.
+- For an explicitly requested coverage assessment, use the
+  [coverage guide](docs/testing/README.md#coverage); a regression test alone does
+  not require coverage measurement.
 - Add package-scoped `cargo check` or Clippy only for targets, features, public
   APIs, error handling, or control flow not compiled by the focused test.
 - Use `make pre-commit` only when its repository-wide fast checks add confidence
@@ -150,13 +155,18 @@ For every added or edited `tracing` call:
 
 - Reuse the module's `EVENT_*`, `LOG_COMPONENT_*`, and `LOG_SUBSYSTEM_*`
   constants and field shape.
-- Put fields first and a short label last.
+- Put stable fields first (`event`, `component`, `subsystem`, `state`/`result`,
+  then context when available) and a short label last.
 - Use `error` for behavior/security failure, `warn` for degradation/fallback,
   `info` for low-frequency lifecycle, `debug` for diagnostics, and `trace` for
   repetitive request/object success paths.
-- Never log secrets, credential payloads, or merged configs.
+- Never log secrets, credential payloads, raw untrusted bodies, or merged configs,
+  including through error strings and `Debug` output.
+- Prefer one aggregate summary over inventories or startup banners.
 
-Use `.agents/skills/rustfs-logging-governance/SKILL.md` for logging changes.
+Run `./scripts/check_logging_guardrails.sh` for changed logging sites. For a broad
+logging audit, event-model migration, or guardrail expansion, read
+[logging governance](docs/operations/logging-governance.md).
 
 ## Cross-Cutting Storage Invariants
 

CHANGELOG.md

@@ -25,6 +25,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Presigned URLs honour only signed headers** (GHSA-g8w9-qw9q-fghr): a SigV4 presigned request that carries an `x-amz-*` request header not listed in `X-Amz-SignedHeaders` is now rejected with `403 AccessDenied` ("There were headers present in the request which were not signed"), matching AWS S3. Previously the holder of a presigned `PutObject` URL could add unsigned `x-amz-tagging`, `x-amz-storage-class`, `x-amz-website-redirect-location`, ACL, metadata, Object Lock or SSE headers and have them applied. Presigners that intend a property must set it before signing so the SDK lists the header in `SignedHeaders`; `x-amz-cf-id` (CloudFront) remains tolerated unsigned. Header-signed SigV4 and SigV2 requests are unchanged.
 
 ### Fixed
+- **SSE-C TLS requirement now covers copy-source keys**: with `RUSTFS_SSE_C_REQUIRE_TLS=true`, a `CopyObject` or `UploadPartCopy` that carried only the `x-amz-copy-source-server-side-encryption-customer-*` headers (an SSE-C source copied to a destination that is not SSE-C) was accepted on a plaintext transport and was not counted in `rustfs_ssec_plaintext_requests_total`, although those headers carry the source object's customer key. They now count as SSE-C headers: such requests are counted, and refused with `400 InvalidRequest` when the switch is on. Requests over TLS are unaffected.
+- **SSE-KMS encryption context could replace the location entry**: data keys are wrapped under an encryption-context entry, keyed by the bucket name, that ties them to the object's location. A client `x-amz-server-side-encryption-context` entry under the same key replaced it, and because the client context is stored with the object, the replacement also applied on every read. SSE-KMS writes now refuse such a context with `400 InvalidArgument`; other keys, including the names of other buckets, are unaffected. Objects stored with such an entry by an earlier release remain readable. See `docs/operations/kms-backend-security.md`.
+- **Two admin handlers reported a missing policy or user as `500 InternalError`**: `GET /rustfs/admin/v3/info-canned-policy` and `DELETE /rustfs/admin/v3/remove-user` wrapped every IAM error as an internal fault instead of classifying it, so a lookup for something that does not exist answered `500 InternalError` where the rest of the admin API answers `404 NoSuchResource` — `GET /rustfs/admin/v3/user-info` already did, and `iam_error_to_s3_error` already mapped `NoSuchPolicy`, `NoSuchUser`, `NoSuchServiceAccount` and `NoSuchTempAccount` to `NoSuchResource`. A client could not tell "does not exist" from a real server failure, and a controller that checks existence before creating (a Kubernetes operator reconciling policies or users) could never create the object: the missing-policy lookup failed with a `500` and it retried forever. Both handlers now map their IAM errors through the shared `iam_error_to_s3_error` helper, so a missing policy or user is `404 NoSuchResource` and the `error` is preserved as the S3 error's source; the three IAM calls in the removal path (temporary-user lookup, service-account lookup, delete) are all reclassified, and every other IAM error stays `500 InternalError`. Fixes #8117.
 - **Vault backends accepted key names that escape the key prefix**: only the Local backend refused a key identifier containing a path separator, so on Vault KV2 `POST /rustfs/admin/v3/kms/keys` with a name such as `bad/name` succeeded and created a nested KV2 path that the key listing then reported as a directory rather than a key, and an identifier containing `..` addressed a record outside the configured key prefix once the HTTP client normalised the URL. Both Vault backends now apply the Local backend's containment rule at the point where the identifier becomes a path or a Transit key name: an empty identifier, one containing `/`, `\\` or NUL, or the dot segments `.` and `..` is refused with `400` (`InvalidKey`) before any request reaches Vault. The AWS backend is unaffected, since it addresses keys by ARN and alias. A Vault deployment that created such a key on an earlier build must re-create it under a plain name; objects encrypted under the old name stay readable only until that key is refused, so migrate them first. Refs rustfs/backlog#2474.
 - **Fresh multi-pool bootstrap with distinct format creators**: a new deployment whose pools have their first endpoint on different nodes (for example two single-node pools) could never publish its initial `pool.bin`: each node held fresh-bootstrap proof only for the pool it formatted, the deployment-wide proof collapsed to none, and every node died with `pool metadata recovery required: no durable bootstrap identity or pool.bin replica is available` after the startup retry budget. The first pool's creator now mints the pending cluster identity on its own pool, every other creator copies that nonce-bound identity onto the pool it formatted first-hand, and the elected writer publishes `pool.bin` once every pool replica carries the same pending identity. Corrupt or disagreeing replicas, pools that merely have a format, expansion pools joining an initialized deployment, and restarts without first-hand proof still fail closed. Non-elected nodes that start before `pool.bin` exists, and the elected writer while it waits for the other creators, no longer latch their pool-metadata write gate for the life of the process. Refs rustfs/backlog#2338, rustfs/backlog#2375.
 - **Lock RPC timeout storms** (#7363): the remote lock client no longer evicts and re-dials the shared internode HTTP/2 channel on every request deadline. A timeout evicts only when the peer has not completed any lock RPC for two deadlines, evictions and transport-failure re-dials are rate limited per peer (`RUSTFS_OBJECT_LOCK_RPC_EVICTION_COOLDOWN_MS`, default 5 s), and a timed-out request is left running instead of being reset (bounded per peer by `RUSTFS_OBJECT_LOCK_RPC_DETACHED_LIMIT`, default 256), so a slow lock endpoint can no longer drive the `RST_STREAM`/`GOAWAY too_many_resets`/reconnect loop. A lock granted after its caller timed out is released immediately, and unlocks that fail the quick retries continue on a deferred 1/2/4/8/16 s schedule before the server lease reclaims them. New `rustfs_remote_lock_*` metrics cover timeouts, evictions, suppressed evictions, detached streams, late completions and late releases per peer. Operator guide at `docs/operations/lock-rpc-storm-protection.md`.

Cargo.toml

@@ -18,6 +18,7 @@ members = [
     "crates/audit", # Audit target management system with multi-target fan-out
     "crates/checksums", # client checksums
     "crates/common", # Shared utilities and data structures
+    "crates/capture-authority", # Durable capture authority storage
     "crates/data-usage", # Shared data usage models and algorithms
     "crates/config", # Configuration management
     "crates/connect-symbol-catalog", # Release-only CPU profile symbol catalog generator
@@ -73,7 +74,7 @@ resolver = "3"
 edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
-rust-version = "1.98.1"
+rust-version = "1.99.0"
 version = "1.0.1"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
@@ -146,8 +147,7 @@ rustfs-zip = { path = "./crates/zip", version = "1.0.1" }
 async-channel = "2.5.0"
 async_zip = { default-features = false, version = "0.0.19" }
 mysql_async = { default-features = false, version = "0.37.1" }
-async-compression = { version = "0.4.48" }
-async-recursion = "1.1.1"
+async-compression = { version = "0.4.50" }
 async-trait = "0.1.92"
 async-nats = { version = "0.50.0", default-features = false }
 axum = "0.8.9"
@@ -158,27 +158,34 @@ futures-util = "0.3.34"
 pollster = "1.0.1"
 pulsar = { default-features = false, version = "6.9.0" }
 lapin = { default-features = false, version = "4.12.0" }
-hyper = { version = "1.11.1" }
+hyper = { version = "1.12.0" }
 hyper-rustls = { default-features = false, version = "0.27.10" }
-hyper-util = { version = "0.1.20" }
+hyper-util = { version = "0.1.21" }
 http = "1.5.0"
 http-body = "1.1.0"
 http-body-util = "0.1.5"
 memchr = "2.8.3"
 minlz = "1.2.3"
 reqwest = "0.13.5"
-rustfs-kafka-async = { version = "1.3.1" }
+rustfs-kafka-async = { version = "1.4.0" }
 socket2 = { version = "0.6.5" }
-tokio = { version = "1.53.1" }
-tokio-rustls = { default-features = false, version = "0.26.5" }
+tokio = { version = "1.53.2" }
+tokio-rustls = { default-features = false, version = "0.26.6" }
 tokio-stream = { version = "0.1.19" }
-tokio-test = "0.4.5"
+tokio-test = "0.4.6"
 tokio-util = { version = "0.7.19" }
 tonic = { version = "0.14.6" }
 tonic-prost = { version = "0.14.6" }
 tonic-prost-build = { version = "0.14.6" }
 tower = { version = "0.5.3" }
 tower-http = { version = "0.7.1" }
+h3 = "0.0.8"
+h3-quinn = "0.0.10"
+quinn = { version = "0.11.11", default-features = false, features = [
+      "futures-io",
+      "runtime-tokio",
+      "rustls-aws-lc-rs",
+] }
 
 # Serialization and Data Formats
 apache-avro = { version = "0.22.0", features = ["snappy", "zstandard"] }
@@ -220,13 +227,13 @@ x509-parser = "0.18.1"
 sha1 = "0.11.0"
 sha2 = "0.11.0"
 subtle = "2.6"
-zeroize = { version = "1.9.0" }
+zeroize = { version = "1.9.1" }
 proptest = "1"
 
 # Time and Date
 chrono = { version = "0.4.45" }
 humantime = "2.4.0"
-jiff = { version = "0.2.37" }
+jiff = { version = "0.2.38" }
 time = { version = "0.3.55" }
 
 # Database
@@ -245,13 +252,13 @@ atoi = "3.1.0"
 atomic_enum = "0.3.0"
 aws-config = { version = "1.12.0" }
 aws-credential-types = { version = "1.3.0" }
-aws-sdk-kms = { default-features = false, version = "1.121.0" }
-aws-sdk-s3 = { default-features = false, version = "1.149.0" }
-aws-sdk-sts = { default-features = false, version = "1.117.0" }
+aws-sdk-kms = { default-features = false, version = "1.123.0" }
+aws-sdk-s3 = { default-features = false, version = "1.152.0" }
+aws-sdk-sts = { default-features = false, version = "1.119.0" }
 aws-smithy-async = { version = "1.3.0" }
-aws-smithy-http-client = { default-features = false, version = "1.4.2" }
-aws-smithy-runtime-api = { version = "1.17.0" }
-aws-smithy-types = { version = "1.8.0" }
+aws-smithy-http-client = { default-features = false, version = "1.5.0" }
+aws-smithy-runtime-api = { version = "1.19.0" }
+aws-smithy-types = { version = "1.8.1" }
 base64-simd = "0.8.0"
 brotli = "9.0.0"
 clap = { version = "4.6.7" }
@@ -269,8 +276,8 @@ enumset = "1.1.14"
 faster-hex = "1.0.0"
 flate2 = "1.1.10"
 glob = "0.3.4"
-google-cloud-storage = "1.18.0"
-google-cloud-auth = "1.16.0"
+google-cloud-storage = "1.20.0"
+google-cloud-auth = "1.17.0"
 hashbrown = { version = "0.17.1" }
 # Base32 for RFC 6238 TOTP shared secrets (RFC 4648 unpadded, the alphabet
 # every authenticator app expects). Already in the graph transitively.
@@ -279,8 +286,8 @@ hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
 hostname = "0.4.2"
 ipnetwork = { version = "0.21.1" }
-lazy_static = "1.5.0"
-libc = "0.2.189"
+lazy_static = "1.5.1"
+libc = "0.2.190"
 libsystemd = "0.7.2"
 local-ip-address = "0.6.13"
 log = "0.4"
@@ -293,13 +300,14 @@ moka = { version = "0.12.16" }
 netif = "0.1.6"
 num_cpus = { version = "1.17.0" }
 nvml-wrapper = "0.13.0"
+object = { version = "=0.40.0", default-features = false }
 parking_lot = "0.12.5"
 path-absolutize = "4.0.1"
 percent-encoding = "2.3.2"
 # Server-side QR rendering for TOTP enrollment, so neither the console nor the
 # CLI needs its own QR encoder. No default features: the image/render backends
 # pull in an image stack this only needs SVG and text output from.
-qrcode-rs = { version = "2.1.1", default-features = false, features = ["std", "svg"] }
+qrcode-rs = { version = "2.2.0", default-features = false, features = ["std", "svg"] }
 pin-project-lite = "0.2.17"
 pretty_assertions = "1.4.1"
 rand = { version = "0.10.3" }
@@ -307,22 +315,24 @@ ratelimit = "2.0.0"
 rayon = "1.12.0"
 rustfs-erasure-codec = { version = "9.0.0" }
 reed-solomon-simd = "3.1.0"
+openraft = { version = "=0.9.25", default-features = false, features = ["serde"] }
+redb = "=4.3.0"
 regex = { version = "1.13.1" }
 rumqttc = { package = "rumqttc-next", version = "0.34.0" }
-redis = { version = "1.7.0" }
+redis = { version = "1.7.1" }
 rustify = { version = "0.7", default-features = false }
 rustix = { version = "1.1.5" }
 rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }
-s3s = { version = "0.17.0", features = ["minio"] }
-s3s-sigv4 = "0.17.0"
+s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
+s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0" }
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
-siphasher = "1.0.3"
-smallvec = { version = "1.16.1" }
+siphasher = "1.0.4"
+smallvec = { version = "1.16.2" }
 compact_str = "0.10.0"
 snap = "1.1.2"
-starshard = { version = "2.3.0" }
+starshard = { version = "2.4.0" }
 strum = { version = "0.28.0" }
 sysinfo = "0.39.6"
 temp-env = "0.3.6"
@@ -333,42 +343,42 @@ tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-core = "0.1.36"
 tracing-error = "0.2.1"
-tracing-opentelemetry = { version = "0.33" }
+tracing-opentelemetry = { version = "0.34" }
 tracing-subscriber = { version = "0.3.23" }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.26.1" }
+uuid = { version = "1.27.0" }
 vaultrs = { version = "0.8.0" }
 tar = "0.4.46"
 walkdir = "2.5.0"
 winapi-util = "0.1.11"
 windows = { version = "0.62.2" }
 windows-sys = "0.61.2"
-xxhash-rust = { version = "0.8.18" }
+xxhash-rust = { version = "0.8.19" }
 zip = "8.6.0"
 zstd = "0.14.0"
 
 # Observability and Metrics
 metrics = "0.24.6"
 metrics-util = "0.20"
-dial9 = { version = "0.5.2", default-features = false }
-opentelemetry = { version = "0.32.0" }
-opentelemetry-appender-tracing = { version = "0.32.0" }
-opentelemetry-otlp = { version = "0.32.0" }
-opentelemetry-proto = { version = "0.32.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
-opentelemetry_sdk = { version = "0.32.1" }
-opentelemetry-semantic-conventions = { version = "0.32.1" }
-opentelemetry-stdout = { version = "0.32.0" }
+dial9 = { version = "0.5.3", default-features = false }
+opentelemetry = { version = "0.33.0" }
+opentelemetry-appender-tracing = { version = "0.33.0" }
+opentelemetry-otlp = { version = "0.33.0" }
+opentelemetry-proto = { version = "0.33.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
+opentelemetry_sdk = { version = "0.33.0" }
+opentelemetry-semantic-conventions = { version = "0.33.0" }
+opentelemetry-stdout = { version = "0.33.0" }
 pyroscope = { version = "2.1.1" }
 
 # FTP and SFTP
-libunftp = { version = "0.23.0" }
+libunftp = { version = "0.23.1" }
 unftp-core = "0.1.0"
-suppaftp = { version = "12.0.1" }
+suppaftp = { version = "12.1.1" }
 rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
-russh = { version = "0.63.3" }
-russh-sftp = "3.0.0"
+russh = { version = "0.64.1" }
+russh-sftp = "3.0.1"
 
 # WebDAV
 dav-server = "0.11.0"
@@ -376,9 +386,9 @@ dav-server = "0.11.0"
 # Performance Analysis and Memory Profiling
 rustfs-mimalloc = { version = "0.5.6" }
 # Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
-hotpath = { version = "0.26.0", default-features = false }
+hotpath = { version = "0.28.5", default-features = false }
 # Snapshot testing for output format regression detection
-insta = { version = "1.48" }
+insta = { version = "1.49" }
 
 [workspace.metadata.cargo-shear]
 ignored = ["hotpath", "rustfs"]

Dockerfile

@@ -106,7 +106,9 @@ RUN chmod +x /usr/bin/rustfs /entrypoint.sh
 
 RUN addgroup -g 10001 -S rustfs && \
     adduser -u 10001 -G rustfs -S rustfs -D && \
-    mkdir -p /data /logs && \
+    mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs && \
+    ln -s /data/.rustfs/events /opt/rustfs/events && \
+    ln -s /data/.rustfs/audit /opt/rustfs/audit && \
     chown -R rustfs:rustfs /data /logs && \
     chmod 0750 /data /logs
 

Dockerfile.decommission-local

@@ -44,7 +44,9 @@ RUN set -eux; \
 WORKDIR /app
 
 RUN set -eux; \
-    mkdir -p /data /logs; \
+    mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs; \
+    ln -s /data/.rustfs/events /opt/rustfs/events; \
+    ln -s /data/.rustfs/audit /opt/rustfs/audit; \
     chown -R rustfs:rustfs /data /logs /app; \
     chmod 0750 /data /logs
 

Dockerfile.glibc

@@ -110,7 +110,9 @@ RUN chmod +x /usr/bin/rustfs /entrypoint.sh
 
 RUN groupadd -g 10001 rustfs && \
     useradd -u 10001 -g rustfs -m -s /sbin/nologin rustfs && \
-    mkdir -p /data /logs && \
+    mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs && \
+    ln -s /data/.rustfs/events /opt/rustfs/events && \
+    ln -s /data/.rustfs/audit /opt/rustfs/audit && \
     chown -R rustfs:rustfs /data /logs && \
     chmod 0750 /data /logs
 

Dockerfile.source

@@ -240,7 +240,9 @@ WORKDIR /app
 
 # Prepare data/log directories with sane defaults
 RUN set -eux; \
-    mkdir -p /data /logs; \
+    mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs; \
+    ln -s /data/.rustfs/events /opt/rustfs/events; \
+    ln -s /data/.rustfs/audit /opt/rustfs/audit; \
     chown -R rustfs:rustfs /data /logs /app; \
     chmod 0750 /data /logs
 

README.md

@@ -141,7 +141,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.1
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -195,7 +195,7 @@ docker run -d --name rustfs -p 9000:9000 \
 Notes:
 - `RUSTFS_NOTIFY_ENABLE=true` enables the global notify module switch.
 - For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
-- If queue dir is omitted, default is `/opt/rustfs/events`; ensure it is writable by the container runtime user.
+- If queue dir is omitted, the official image maps `/opt/rustfs/events` into the persistent `/data` volume so it is writable by the runtime user and pending events survive container recreation. Override `queue_dir` when using another deployment layout.
 - `RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
 - Since `1.0.0-beta.11`, webhook endpoints on private or container networks
   (`Docker Compose service names`, `host.docker.internal`, RFC 1918 addresses) are
@@ -220,7 +220,7 @@ For developers who want to build RustFS Docker images from source with multi-arc
 ./docker-buildx.sh --push
 
 # Build specific version
-./docker-buildx.sh --release v1.0.0 --push
+./docker-buildx.sh --release 1.0.1 --push
 
 # Build for custom registry
 ./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -238,7 +238,7 @@ You can also use Make targets for convenience:
 ```bash
 make docker-buildx                    # Build locally
 make docker-buildx-push               # Build and push
-make docker-buildx-version VERSION=v1.0.0  # Build specific version
+make docker-buildx-version VERSION=1.0.1  # Build specific version
 make help-docker                      # Show all Docker-related commands
 ```
 

README_ZH.md

@@ -138,7 +138,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # 使用指定版本运行
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.1
 ```
 
 如果您通过绑定挂载启用 TLS 证书目录，也请用同样方式准备该目录：
@@ -176,7 +176,7 @@ docker compose -f docker-compose-simple.yml up -d
 ./docker-buildx.sh --push
 
 # 构建指定版本
-./docker-buildx.sh --release v1.0.0 --push
+./docker-buildx.sh --release 1.0.1 --push
 
 # 构建并推送到自定义仓库
 ./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -193,7 +193,7 @@ docker compose -f docker-compose-simple.yml up -d
 ```bash
 make docker-buildx                    # 本地构建
 make docker-buildx-push               # 构建并推送
-make docker-buildx-version VERSION=v1.0.0  # 构建指定版本
+make docker-buildx-version VERSION=1.0.1  # 构建指定版本
 make help-docker                      # 显示所有 Docker 相关命令
 ```
 

crates/capture-authority/Cargo.toml

@@ -0,0 +1,24 @@
+[package]
+name = "rustfs-capture-authority"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+rust-version.workspace = true
+repository.workspace = true
+description = "Durable local storage for capture authority"
+
+[dependencies]
+openraft = { workspace = true }
+redb = { workspace = true }
+sha2 = { workspace = true }
+serde = { workspace = true, features = ["derive"] }
+thiserror = { workspace = true }
+tokio = { workspace = true }
+url = { workspace = true }
+
+[dev-dependencies]
+serde_json = { workspace = true }
+tempfile = { workspace = true }
+
+[lints]
+workspace = true

crates/capture-authority/src/application.rs

@@ -0,0 +1,941 @@
+fn read_native_range_in<R: std::ops::RangeBounds<u64>>(
+    tx: &redb::ReadTransaction,
+    range: R,
+) -> Result<Vec<crate::CaptureRaftEntry>, StoreError> {
+    use std::ops::Bound::{Excluded, Included, Unbounded};
+    let logs = tx.open_table(LOGS)?;
+    let start = match range.start_bound() {
+        Included(value) => *value,
+        Excluded(value) => match value.checked_add(1) {
+            Some(value) => value,
+            None => return Ok(Vec::new()),
+        },
+        Unbounded => 0,
+    };
+    let end = match range.end_bound() {
+        Included(value) => *value,
+        Excluded(value) => match value.checked_sub(1) {
+            Some(value) => value,
+            None => return Ok(Vec::new()),
+        },
+        Unbounded => u64::MAX,
+    };
+    if start > end {
+        return Ok(Vec::new());
+    }
+    let mut entries = Vec::new();
+    for row in logs.range(start..=end)? {
+        let (key, value) = row?;
+        let entry = decode_log(value.value())?;
+        if entry.id.index != key.value() {
+            return Err(StoreError::CorruptRecord);
+        }
+        entries.push(decode_native_entry(entry)?);
+    }
+    Ok(entries)
+}
+
+use crate::{
+    LOGS, LogId, META, Store, StoreError, committed::validate_committed, created::*, decode_log, encode_id, encode_log,
+    next_index, number,
+};
+use redb::{ReadTransaction, ReadableDatabase, ReadableTable, ReadableTableMetadata, TableDefinition};
+use std::collections::BTreeMap;
+
+pub(crate) const HEADS: TableDefinition<&[u8], &[u8]> = TableDefinition::new("heads");
+pub(crate) const VERSIONS: TableDefinition<&[u8], &[u8]> = TableDefinition::new("versions");
+pub(crate) const EVENTS: TableDefinition<&[u8], &[u8]> = TableDefinition::new("events");
+pub(crate) const OPERATIONS: TableDefinition<&[u8], &[u8]> = TableDefinition::new("operations");
+pub(crate) const APPLY_RESULTS: TableDefinition<u64, &[u8]> = TableDefinition::new("apply_results");
+
+fn version_key(object: &ObjectIdentity, version: [u8; 16]) -> Result<Vec<u8>, StoreError> {
+    let mut b = object_bytes(object)?;
+    b.extend(version);
+    Ok(b)
+}
+fn decode_version(b: &[u8]) -> Result<CreatedVersion, StoreError> {
+    let mut d = Decoder::new(b);
+    let v = CreatedVersion {
+        prepared: d.prepared()?,
+        decision_log_id: d.id()?,
+        binding_revision: d.u64()?,
+    };
+    d.finish()?;
+    Ok(v)
+}
+fn decode_event(b: &[u8]) -> Result<CreatedEvent, StoreError> {
+    let mut d = Decoder::new(b);
+    let e = CreatedEvent {
+        object: d.object()?,
+        prepared: d.prepared()?,
+        decision_log_id: d.id()?,
+        binding: d.binding()?,
+    };
+    d.finish()?;
+    Ok(e)
+}
+fn decode_operation(b: &[u8]) -> Result<CreatedOperation, StoreError> {
+    let mut d = Decoder::new(b);
+    let len = usize::try_from(d.u64()?).map_err(|_| StoreError::CorruptRecord)?;
+    let command = b
+        .get(8..8usize.checked_add(len).ok_or(StoreError::CorruptRecord)?)
+        .ok_or(StoreError::CorruptRecord)?;
+    DecideCreated::decode(command)?;
+    let mut d = Decoder::new(&b[8 + len..]);
+    let result = d.result()?;
+    d.finish()?;
+    Ok(CreatedOperation {
+        canonical_command: command.to_vec(),
+        result,
+    })
+}
+fn decode_apply(b: &[u8]) -> Result<CreatedApplyResult, StoreError> {
+    let mut d = Decoder::new(b);
+    let r = CreatedApplyResult {
+        log_id: d.id()?,
+        result: d.result()?,
+    };
+    d.finish()?;
+    Ok(r)
+}
+pub(crate) fn read_binding(meta: &impl ReadableTable<&'static str, &'static [u8]>) -> Result<CaptureBinding, StoreError> {
+    let b = meta.get("capture_binding")?.ok_or(StoreError::BindingNotPinned)?;
+    let mut d = Decoder::new(b.value());
+    let binding = d.binding()?;
+    d.finish()?;
+    Ok(binding)
+}
+fn read_applied(meta: &impl ReadableTable<&'static str, &'static [u8]>) -> Result<Option<LogId>, StoreError> {
+    let b = meta.get("last_applied")?.ok_or(StoreError::CorruptRecord)?;
+    let mut d = Decoder::new(b.value());
+    let id = match d.take::<1>()?[0] {
+        0 => None,
+        1 => Some(d.id()?),
+        _ => return Err(StoreError::CorruptRecord),
+    };
+    d.finish()?;
+    Ok(id)
+}
+fn required<T>(value: Option<T>) -> Result<T, StoreError> {
+    value.ok_or(StoreError::CorruptRecord)
+}
+fn encode_native_entry(entry: &crate::CaptureRaftEntry) -> Result<Vec<u8>, StoreError> {
+    let id: LogId = entry.log_id.into();
+    let (kind, body) = match &entry.payload {
+        openraft::EntryPayload::Blank => (0, vec![]),
+        openraft::EntryPayload::Normal(command) => (1, command.created().encode()?),
+        openraft::EntryPayload::Membership(membership) => {
+            let stored = openraft::StoredMembership::new(Some(entry.log_id), membership.clone());
+            let checked = crate::CaptureMembership::try_from(&stored)?;
+            (2, checked.encode()?)
+        }
+    };
+    let mut payload = b"CENR\x01".to_vec();
+    payload.push(kind);
+    payload.extend(
+        u64::try_from(body.len())
+            .map_err(|_| StoreError::CorruptRecord)?
+            .to_be_bytes(),
+    );
+    payload.extend(body);
+    encode_log(&crate::LogEntry { id, payload })
+}
+
+pub(crate) fn decode_native_entry(entry: crate::LogEntry) -> Result<crate::CaptureRaftEntry, StoreError> {
+    let bytes = &entry.payload;
+    if bytes.len() < 14 || &bytes[..5] != b"CENR\x01" {
+        return Err(StoreError::CorruptRecord);
+    }
+    let length = usize::try_from(number(&bytes[6..14])?).map_err(|_| StoreError::CorruptRecord)?;
+    let body = &bytes[14..];
+    if length != body.len() {
+        return Err(StoreError::CorruptRecord);
+    }
+    let payload = match bytes[5] {
+        0 if body.is_empty() => openraft::EntryPayload::Blank,
+        1 => openraft::EntryPayload::Normal(crate::CaptureCommand::try_created(DecideCreated::decode(body)?)?),
+        2 => {
+            let membership = crate::CaptureMembership::decode(body).map_err(|_| StoreError::CorruptRecord)?;
+            if *membership.log_id() != Some(entry.id) {
+                return Err(StoreError::CorruptRecord);
+            }
+            let stored = openraft::StoredMembership::try_from(&membership).map_err(|_| StoreError::CorruptRecord)?;
+            openraft::EntryPayload::Membership(stored.membership().clone())
+        }
+        _ => return Err(StoreError::CorruptRecord),
+    };
+    Ok(crate::CaptureRaftEntry {
+        log_id: entry.id.into(),
+        payload,
+    })
+}
+
+fn apply_created(
+    tx: &redb::WriteTransaction,
+    id: LogId,
+    command: &DecideCreated,
+    canonical: &[u8],
+    binding: &CaptureBinding,
+) -> Result<CreatedResult, StoreError> {
+    let mut operations = tx.open_table(OPERATIONS)?;
+    let existing = operations
+        .get(command.operation_id.as_slice())?
+        .map(|v| decode_operation(v.value()))
+        .transpose()?;
+    let result = if let Some(existing) = existing {
+        if existing.canonical_command == canonical {
+            existing.result
+        } else {
+            CreatedResult::OperationConflict
+        }
+    } else {
+        let mut heads = tx.open_table(HEADS)?;
+        let mut versions = tx.open_table(VERSIONS)?;
+        let object = object_bytes(&command.object)?;
+        let key = version_key(&command.object, command.prepared.version_id)?;
+        let actual = heads
+            .get(object.as_slice())?
+            .map(|v| {
+                let mut d = Decoder::new(v.value());
+                let id = d.uuid()?;
+                d.finish()?;
+                Ok::<_, StoreError>(id)
+            })
+            .transpose()?;
+        let outcome = if command.binding_revision != binding.revision {
+            CreatedResult::BindingMismatch
+        } else if actual != command.expected_head {
+            CreatedResult::HeadMismatch { actual_head: actual }
+        } else if versions.get(key.as_slice())?.is_some() {
+            CreatedResult::VersionConflict
+        } else {
+            let mut version = Vec::new();
+            put_prepared(&mut version, &command.prepared);
+            version.extend(encode_id(id));
+            version.extend(binding.revision.to_be_bytes());
+            versions.insert(key.as_slice(), version.as_slice())?;
+            let mut event = object;
+            put_prepared(&mut event, &command.prepared);
+            event.extend(encode_id(id));
+            event.extend(binding_bytes(&binding)?);
+            tx.open_table(EVENTS)?
+                .insert(command.operation_id.as_slice(), event.as_slice())?;
+            heads.insert(object_bytes(&command.object)?.as_slice(), command.prepared.version_id.as_slice())?;
+            CreatedResult::Created {
+                version_id: command.prepared.version_id,
+                event_id: command.operation_id,
+                decision_log_id: id,
+            }
+        };
+        let mut op = u64::try_from(canonical.len())
+            .map_err(|_| StoreError::CorruptRecord)?
+            .to_be_bytes()
+            .to_vec();
+        op.extend(canonical);
+        op.extend(result_bytes(&outcome));
+        operations.insert(command.operation_id.as_slice(), op.as_slice())?;
+        outcome
+    };
+    Ok(result)
+}
+
+fn validate_response_kind(entry: &crate::CaptureRaftEntry, response: &crate::CaptureResponse) -> Result<(), StoreError> {
+    let kind = match entry.payload {
+        openraft::EntryPayload::Blank => crate::CaptureResponseKind::Blank,
+        openraft::EntryPayload::Normal(_) => crate::CaptureResponseKind::Created,
+        openraft::EntryPayload::Membership(_) => crate::CaptureResponseKind::Membership,
+    };
+    if response.kind() != kind || *response.log_id() != LogId::from(entry.log_id) {
+        return Err(StoreError::CorruptRecord);
+    }
+    Ok(())
+}
+fn encode_response(response: &crate::CaptureResponse) -> Result<Vec<u8>, StoreError> {
+    let (kind, body) = match response.kind() {
+        crate::CaptureResponseKind::Blank => (0, vec![]),
+        crate::CaptureResponseKind::Created => (1, result_bytes(response.created_result().ok_or(StoreError::CorruptRecord)?)),
+        crate::CaptureResponseKind::Membership => (2, vec![]),
+    };
+    let mut bytes = b"CRSP\x01".to_vec();
+    bytes.push(kind);
+    bytes.extend(encode_id(*response.log_id()));
+    bytes.extend(
+        u64::try_from(body.len())
+            .map_err(|_| StoreError::CorruptRecord)?
+            .to_be_bytes(),
+    );
+    bytes.extend(body);
+    Ok(bytes)
+}
+fn decode_response(bytes: &[u8]) -> Result<crate::CaptureResponse, StoreError> {
+    if bytes.len() < 38
+        || &bytes[..5] != b"CRSP\x01"
+        || number(&bytes[30..38])? != u64::try_from(bytes.len() - 38).map_err(|_| StoreError::CorruptRecord)?
+    {
+        return Err(StoreError::CorruptRecord);
+    }
+    let id = crate::decode_id(&bytes[6..30])?;
+    match bytes[5] {
+        0 if bytes.len() == 38 => crate::CaptureResponse::blank(id),
+        1 => {
+            let mut body = encode_id(id);
+            body.extend(&bytes[38..]);
+            crate::CaptureResponse::created(decode_apply(&body)?)
+        }
+        2 if bytes.len() == 38 => crate::CaptureResponse::membership(id),
+        _ => Err(StoreError::CorruptRecord),
+    }
+}
+fn entry_command(entry: &crate::LogEntry, native: bool) -> Result<DecideCreated, StoreError> {
+    if native {
+        match decode_native_entry(entry.clone())?.payload {
+            openraft::EntryPayload::Normal(command) => Ok(command.created().clone()),
+            _ => Err(StoreError::CorruptRecord),
+        }
+    } else {
+        DecideCreated::decode(&entry.payload)
+    }
+}
+
+pub(crate) fn native_application_state(tx: &ReadTransaction) -> Result<(Option<LogId>, crate::CaptureMembership), StoreError> {
+    native_application_metadata(&tx.open_table(META)?)
+}
+
+fn native_application_metadata(
+    meta: &impl ReadableTable<&'static str, &'static [u8]>,
+) -> Result<(Option<LogId>, crate::CaptureMembership), StoreError> {
+    if number(required(meta.get("schema")?)?.value())? != 4 {
+        return Err(StoreError::UnsupportedSchema);
+    }
+    let last = read_applied(meta)?;
+    let membership = crate::CaptureMembership::decode(required(meta.get("native_membership")?)?.value())
+        .map_err(|_| StoreError::CorruptRecord)?;
+    if membership
+        .log_id()
+        .is_some_and(|id| last.is_none_or(|last| id.index > last.index))
+    {
+        return Err(StoreError::CorruptRecord);
+    }
+    Ok((last, membership))
+}
+
+impl Store {
+    pub fn migrate_legacy_created_to_native(&mut self) -> Result<(), StoreError> {
+        let tx = self.write_transaction()?;
+        {
+            let mut meta = tx.open_table(META)?;
+            let schema = number(required(meta.get("schema")?)?.value())?;
+            if !matches!(schema, 3 | 4) {
+                return Err(StoreError::UnsupportedSchema);
+            }
+            let mut logs = tx.open_table(LOGS)?;
+            let heads = tx.open_table(HEADS)?;
+            let versions = tx.open_table(VERSIONS)?;
+            let events = tx.open_table(EVENTS)?;
+            let operations = tx.open_table(OPERATIONS)?;
+            let mut results = tx.open_table(APPLY_RESULTS)?;
+            validate_application_tables(&meta, &logs, &heads, &versions, &events, &operations, &results)?;
+            if schema == 4 {
+                return Ok(());
+            }
+            if meta.get("native_membership")?.is_some() {
+                return Err(StoreError::CorruptRecord);
+            }
+            // Validate the entire retained suffix before replacing any record.
+            let mut native_logs = Vec::new();
+            for row in logs.iter()? {
+                let (index, bytes) = row?;
+                let entry = decode_log(bytes.value())?;
+                let command = DecideCreated::decode(&entry.payload)?;
+                if entry.id.index != index.value() || command.encode()? != entry.payload {
+                    return Err(StoreError::CorruptRecord);
+                }
+                let native = crate::CaptureRaftEntry {
+                    log_id: entry.id.into(),
+                    payload: openraft::EntryPayload::Normal(crate::CaptureCommand::try_created(command)?),
+                };
+                native_logs.push((index.value(), encode_native_entry(&native)?));
+            }
+            let mut native_results = Vec::new();
+            for row in results.iter()? {
+                let (index, bytes) = row?;
+                let response = crate::CaptureResponse::created(decode_apply(bytes.value())?)?;
+                native_results.push((index.value(), encode_response(&response)?));
+            }
+            for (index, bytes) in native_logs {
+                logs.insert(index, bytes.as_slice())?;
+            }
+            for (index, bytes) in native_results {
+                results.insert(index, bytes.as_slice())?;
+            }
+            let membership = crate::CaptureMembership::new(None, vec![], BTreeMap::new())?;
+            meta.insert("native_membership", membership.encode()?.as_slice())?;
+            meta.insert("schema", 4u64.to_be_bytes().as_slice())?;
+            // Replay native decisions inside the same durable writer boundary.
+            validate_application_tables(&meta, &logs, &heads, &versions, &events, &operations, &results)?;
+        }
+        self.commit(tx)
+    }
+
+    /// Explicitly opt an empty, bound store into native entry persistence.
+    pub fn enable_native_empty(&mut self) -> Result<(), StoreError> {
+        let tx = self.write_transaction()?;
+        {
+            let mut meta = tx.open_table(META)?;
+            let schema = number(required(meta.get("schema")?)?.value())?;
+            if !matches!(schema, 3 | 4) {
+                return Err(StoreError::UnsupportedSchema);
+            }
+            read_binding(&meta)?;
+            if schema == 4 {
+                let membership = crate::CaptureMembership::decode(required(meta.get("native_membership")?)?.value())
+                    .map_err(|_| StoreError::CorruptRecord)?;
+                let last = read_applied(&meta)?;
+                if membership
+                    .log_id()
+                    .is_some_and(|id| last.is_none_or(|last| id.index > last.index))
+                    || (last.is_none() && (!membership.configs().is_empty() || !membership.nodes().is_empty()))
+                {
+                    return Err(StoreError::CorruptRecord);
+                }
+                return Ok(());
+            }
+            if read_applied(&meta)?.is_some() || !tx.open_table(LOGS)?.is_empty()? {
+                return Err(StoreError::UnsupportedSchema);
+            }
+            for table in [HEADS, VERSIONS, EVENTS, OPERATIONS] {
+                if !tx.open_table(table)?.is_empty()? {
+                    return Err(StoreError::UnsupportedSchema);
+                }
+            }
+            if !tx.open_table(APPLY_RESULTS)?.is_empty()? || meta.get("committed")?.is_some() || meta.get("tail")?.is_some() {
+                return Err(StoreError::UnsupportedSchema);
+            }
+            if meta.get("native_membership")?.is_some() {
+                return Err(StoreError::CorruptRecord);
+            }
+            let membership = crate::CaptureMembership::new(None, vec![], BTreeMap::new())?;
+            meta.insert("native_membership", membership.encode()?.as_slice())?;
+            meta.insert("schema", 4u64.to_be_bytes().as_slice())?;
+        }
+        self.commit(tx)
+    }
+    pub fn append_native(&mut self, entries: &[crate::CaptureRaftEntry]) -> Result<(), StoreError> {
+        let tx = self.write_transaction()?;
+        {
+            let mut meta = tx.open_table(META)?;
+            if number(required(meta.get("schema")?)?.value())? != 4 {
+                return Err(StoreError::UnsupportedSchema);
+            }
+            if entries.is_empty() {
+                return Ok(());
+            }
+            let tail = meta.get("tail")?.map(|value| crate::decode_id(value.value())).transpose()?;
+            let mut next = next_index(tail.map(|id| id.index))?;
+            let mut encoded = Vec::with_capacity(entries.len());
+            for (position, entry) in entries.iter().enumerate() {
+                if entry.log_id.index != next {
+                    return Err(StoreError::InvalidLogSequence);
+                }
+                encoded.push(encode_native_entry(entry)?);
+                if position + 1 < entries.len() {
+                    next = next_index(Some(next))?;
+                }
+            }
+            let mut logs = tx.open_table(LOGS)?;
+            for (entry, bytes) in entries.iter().zip(&encoded) {
+                logs.insert(entry.log_id.index, bytes.as_slice())?;
+            }
+            let last = entries.last().ok_or(StoreError::InvalidLogSequence)?;
+            meta.insert("tail", encode_id(last.log_id.into()).as_slice())?;
+        }
+        self.commit(tx)
+    }
+    pub fn read_native_entry(&self, index: u64) -> Result<Option<crate::CaptureRaftEntry>, StoreError> {
+        let tx = self.database.begin_read()?;
+        if number(required(tx.open_table(META)?.get("schema")?)?.value())? != 4 {
+            return Err(StoreError::UnsupportedSchema);
+        }
+        tx.open_table(LOGS)?
+            .get(index)?
+            .map(|value| {
+                let entry = decode_log(value.value())?;
+                if entry.id.index != index {
+                    return Err(StoreError::CorruptRecord);
+                }
+                decode_native_entry(entry)
+            })
+            .transpose()
+    }
+    /// Read typed native entries within the requested bounds.
+    pub fn read_native_range<R: std::ops::RangeBounds<u64>>(&self, range: R) -> Result<Vec<crate::CaptureRaftEntry>, StoreError> {
+        let tx = self.database.begin_read()?;
+        if number(required(tx.open_table(META)?.get("schema")?)?.value())? != 4 {
+            return Err(StoreError::UnsupportedSchema);
+        }
+        read_native_range_in(&tx, range)
+    }
+    pub fn read_native_entries(&self, start: u64, end: u64) -> Result<Vec<crate::CaptureRaftEntry>, StoreError> {
+        let tx = self.database.begin_read()?;
+        if number(required(tx.open_table(META)?.get("schema")?)?.value())? != 4 {
+            return Err(StoreError::UnsupportedSchema);
+        }
+        if start > end {
+            return Err(StoreError::InvalidLogSequence);
+        }
+        read_native_range_in(&tx, start..end)
+    }
+
+    pub fn apply_next_native(&mut self, id: LogId) -> Result<crate::CaptureResponse, StoreError> {
+        let tx = self.write_transaction()?;
+        let response;
+        {
+            let mut meta = tx.open_table(META)?;
+            if number(required(meta.get("schema")?)?.value())? != 4 {
+                return Err(StoreError::UnsupportedSchema);
+            }
+            let binding = read_binding(&meta)?;
+            let logs = tx.open_table(LOGS)?;
+            let committed = validate_committed(&meta, &logs)?.ok_or(StoreError::InvalidCommittedLog)?;
+            let entry = decode_log(required(logs.get(id.index)?)?.value())?;
+            if entry.id != id || id.index > committed.index {
+                return Err(StoreError::InvalidCommittedLog);
+            }
+            let native = decode_native_entry(entry)?;
+            let last = read_applied(&meta)?;
+            let mut results = tx.open_table(APPLY_RESULTS)?;
+            if last.is_some_and(|last| id.index <= last.index) {
+                let saved = decode_response(required(results.get(id.index)?)?.value())?;
+                if *saved.log_id() != id {
+                    return Err(StoreError::InvalidApplySequence);
+                }
+                validate_response_kind(&native, &saved)?;
+                return Ok(saved);
+            }
+            if id.index != next_index(last.map(|last| last.index))? {
+                return Err(StoreError::InvalidApplySequence);
+            }
+            response = match native.payload {
+                openraft::EntryPayload::Blank => crate::CaptureResponse::blank(id)?,
+                openraft::EntryPayload::Normal(command) => {
+                    let canonical = command.created().encode()?;
+                    let result = apply_created(&tx, id, command.created(), &canonical, &binding)?;
+                    crate::CaptureResponse::created(CreatedApplyResult { log_id: id, result })?
+                }
+                openraft::EntryPayload::Membership(membership) => {
+                    let stored = openraft::StoredMembership::new(Some(id.into()), membership);
+                    let checked = crate::CaptureMembership::try_from(&stored)?;
+                    meta.insert("native_membership", checked.encode()?.as_slice())?;
+                    crate::CaptureResponse::membership(id)?
+                }
+            };
+            results.insert(id.index, encode_response(&response)?.as_slice())?;
+            let mut last = vec![1];
+            last.extend(encode_id(id));
+            meta.insert("last_applied", last.as_slice())?;
+        }
+        self.commit(tx)?;
+        Ok(response)
+    }
+    pub fn native_apply_response(&self, index: u64) -> Result<Option<crate::CaptureResponse>, StoreError> {
+        let tx = self.database.begin_read()?;
+        if number(required(tx.open_table(META)?.get("schema")?)?.value())? != 4 {
+            return Err(StoreError::UnsupportedSchema);
+        }
+        let results = tx.open_table(APPLY_RESULTS)?;
+        results.get(index)?.map(|v| decode_response(v.value())).transpose()
+    }
+    pub fn native_application_state(&self) -> Result<(Option<LogId>, crate::CaptureMembership), StoreError> {
+        native_application_state(&self.database.begin_read()?)
+    }
+    /// Bootstrap the immutable capture policy. The caller must be the trusted
+    /// deployment provisioning owner. This crate does not authenticate callers;
+    /// do not expose this public method through an ordinary object request API.
+    /// Prepared identities assert identity only, not physical data durability.
+    pub fn pin_capture_binding(&mut self, binding: CaptureBinding) -> Result<(), StoreError> {
+        let encoded = binding_bytes(&binding)?;
+        let tx = self.write_transaction()?;
+        {
+            let mut meta = tx.open_table(META)?;
+            if matches!(number(required(meta.get("schema")?)?.value())?, 3 | 4) {
+                if read_binding(&meta)? == binding {
+                    return Ok(());
+                }
+                return Err(StoreError::BindingConflict);
+            }
+            if meta.get("capture_binding")?.is_some() || meta.get("last_applied")?.is_some() {
+                return Err(StoreError::CorruptRecord);
+            }
+            for table in [HEADS, VERSIONS, EVENTS, OPERATIONS] {
+                if !tx.open_table(table)?.is_empty()? {
+                    return Err(StoreError::CorruptRecord);
+                }
+            }
+            if !tx.open_table(APPLY_RESULTS)?.is_empty()? {
+                return Err(StoreError::CorruptRecord);
+            }
+            meta.insert("capture_binding", encoded.as_slice())?;
+            meta.insert("last_applied", &[0][..])?;
+            meta.insert("schema", 3u64.to_be_bytes().as_slice())?;
+        }
+        self.commit(tx)
+    }
+    /// Apply the next exact persisted committed Created command atomically.
+    pub fn apply_next_created(&mut self, id: LogId) -> Result<CreatedResult, StoreError> {
+        let tx = self.write_transaction()?;
+        let result;
+        {
+            let mut meta = tx.open_table(META)?;
+            if number(required(meta.get("schema")?)?.value())? != 3 {
+                return Err(StoreError::UnsupportedSchema);
+            }
+            let binding = read_binding(&meta)?;
+            let logs = tx.open_table(LOGS)?;
+            let committed = validate_committed(&meta, &logs)?.ok_or(StoreError::InvalidCommittedLog)?;
+            let entry = decode_log(required(logs.get(id.index)?)?.value())?;
+            if entry.id != id || id.index > committed.index {
+                return Err(StoreError::InvalidCommittedLog);
+            }
+            let last = read_applied(&meta)?;
+            let mut results = tx.open_table(APPLY_RESULTS)?;
+            if last.is_some_and(|last| id.index <= last.index) {
+                let saved = decode_apply(required(results.get(id.index)?)?.value())?;
+                if saved.log_id != id {
+                    return Err(StoreError::InvalidApplySequence);
+                }
+                return Ok(saved.result);
+            }
+            if id.index != next_index(last.map(|last| last.index))? {
+                return Err(StoreError::InvalidApplySequence);
+            }
+            let command = DecideCreated::decode(&entry.payload)?;
+            result = apply_created(&tx, id, &command, &entry.payload, &binding)?;
+            let mut applied = encode_id(id);
+            applied.extend(result_bytes(&result));
+            results.insert(id.index, applied.as_slice())?;
+            let mut last = vec![1];
+            last.extend(encode_id(id));
+            meta.insert("last_applied", last.as_slice())?;
+        }
+        self.commit(tx)?;
+        Ok(result)
+    }
+    pub fn capture_binding(&self) -> Result<Option<CaptureBinding>, StoreError> {
+        let tx = self.database.begin_read()?;
+        let meta = tx.open_table(META)?;
+        if !matches!(number(required(meta.get("schema")?)?.value())?, 3 | 4) {
+            return Ok(None);
+        }
+        Ok(Some(read_binding(&meta)?))
+    }
+    pub fn last_applied(&self) -> Result<Option<LogId>, StoreError> {
+        let tx = self.database.begin_read()?;
+        let meta = tx.open_table(META)?;
+        if !matches!(number(required(meta.get("schema")?)?.value())?, 3 | 4) {
+            return Ok(None);
+        }
+        read_applied(&meta)
+    }
+    pub fn created_head(&self, object: &ObjectIdentity) -> Result<Option<[u8; 16]>, StoreError> {
+        let tx = self.database.begin_read()?;
+        let t = tx.open_table(HEADS)?;
+        t.get(object_bytes(object)?.as_slice())?
+            .map(|v| {
+                let mut d = Decoder::new(v.value());
+                let id = d.uuid()?;
+                d.finish()?;
+                Ok(id)
+            })
+            .transpose()
+    }
+    pub fn created_version(&self, object: &ObjectIdentity, version: [u8; 16]) -> Result<Option<CreatedVersion>, StoreError> {
+        let tx = self.database.begin_read()?;
+        let t = tx.open_table(VERSIONS)?;
+        t.get(version_key(object, version)?.as_slice())?
+            .map(|v| decode_version(v.value()))
+            .transpose()
+    }
+    pub fn created_event(&self, operation: [u8; 16]) -> Result<Option<CreatedEvent>, StoreError> {
+        let tx = self.database.begin_read()?;
+        let t = tx.open_table(EVENTS)?;
+        t.get(operation.as_slice())?.map(|v| decode_event(v.value())).transpose()
+    }
+    pub fn created_operation(&self, operation: [u8; 16]) -> Result<Option<CreatedOperation>, StoreError> {
+        let tx = self.database.begin_read()?;
+        let t = tx.open_table(OPERATIONS)?;
+        t.get(operation.as_slice())?.map(|v| decode_operation(v.value())).transpose()
+    }
+    pub fn created_apply_result(&self, index: u64) -> Result<Option<CreatedApplyResult>, StoreError> {
+        let tx = self.database.begin_read()?;
+        let t = tx.open_table(APPLY_RESULTS)?;
+        if number(required(tx.open_table(META)?.get("schema")?)?.value())? == 4 {
+            t.get(index)?
+                .map(|v| {
+                    decode_response(v.value()).map(|r| {
+                        r.created_result().cloned().map(|result| CreatedApplyResult {
+                            log_id: *r.log_id(),
+                            result,
+                        })
+                    })
+                })
+                .transpose()
+                .map(Option::flatten)
+        } else {
+            t.get(index)?.map(|v| decode_apply(v.value())).transpose()
+        }
+    }
+}
+
+pub(crate) fn validate_application(tx: &ReadTransaction) -> Result<(), StoreError> {
+    let meta = tx.open_table(META)?;
+    if !matches!(number(required(meta.get("schema")?)?.value())?, 3 | 4) {
+        return Ok(());
+    }
+    validate_application_tables(
+        &tx.open_table(META)?,
+        &tx.open_table(LOGS)?,
+        &tx.open_table(HEADS)?,
+        &tx.open_table(VERSIONS)?,
+        &tx.open_table(EVENTS)?,
+        &tx.open_table(OPERATIONS)?,
+        &tx.open_table(APPLY_RESULTS)?,
+    )
+}
+
+fn validate_application_tables(
+    meta: &impl ReadableTable<&'static str, &'static [u8]>,
+    logs: &impl ReadableTable<u64, &'static [u8]>,
+    heads: &impl ReadableTable<&'static [u8], &'static [u8]>,
+    versions: &impl ReadableTable<&'static [u8], &'static [u8]>,
+    events: &impl ReadableTable<&'static [u8], &'static [u8]>,
+    operations: &impl ReadableTable<&'static [u8], &'static [u8]>,
+    results: &impl ReadableTable<u64, &'static [u8]>,
+) -> Result<(), StoreError> {
+    let schema = number(required(meta.get("schema")?)?.value())?;
+    let native = schema == 4;
+    if !matches!(schema, 3 | 4) {
+        return Ok(());
+    }
+    let binding = read_binding(meta).map_err(|error| match error {
+        StoreError::BindingNotPinned => StoreError::CorruptRecord,
+        other => other,
+    })?;
+    let last = read_applied(meta)?;
+    let committed = validate_committed(meta, logs)?;
+    if last.is_some_and(|last| committed.is_none_or(|c| last.index > c.index)) {
+        return Err(StoreError::CorruptRecord);
+    }
+    let mut next = Some(0);
+    let mut observed = None;
+    let mut first_operations = BTreeMap::new();
+    let mut expected_membership = crate::CaptureMembership::new(None, vec![], BTreeMap::new())?;
+    let mut replay_operations = BTreeMap::<[u8; 16], (Vec<u8>, CreatedResult)>::new();
+    let mut replay_heads = BTreeMap::<Vec<u8>, [u8; 16]>::new();
+    let mut replay_versions = std::collections::BTreeSet::new();
+    for row in results.iter()? {
+        let (k, v) = row?;
+        let entry = decode_log(required(logs.get(k.value())?)?.value())?;
+        let r = if native {
+            let response = decode_response(v.value())?;
+            let native_entry = decode_native_entry(entry.clone())?;
+            validate_response_kind(&native_entry, &response)?;
+            if next != Some(k.value()) || response.log_id().index != k.value() {
+                return Err(StoreError::CorruptRecord);
+            }
+            observed = Some(*response.log_id());
+            next = k.value().checked_add(1);
+            match native_entry.payload {
+                openraft::EntryPayload::Membership(membership) => {
+                    expected_membership =
+                        crate::CaptureMembership::try_from(&openraft::StoredMembership::new(Some(entry.id.into()), membership))?;
+                    continue;
+                }
+                openraft::EntryPayload::Blank => continue,
+                openraft::EntryPayload::Normal(_) => CreatedApplyResult {
+                    log_id: *response.log_id(),
+                    result: response.created_result().cloned().ok_or(StoreError::CorruptRecord)?,
+                },
+            }
+        } else {
+            decode_apply(v.value())?
+        };
+        if !native && (next != Some(k.value()) || r.log_id.index != k.value()) {
+            return Err(StoreError::CorruptRecord);
+        }
+        if entry.id != r.log_id {
+            return Err(StoreError::CorruptRecord);
+        }
+        let c = entry_command(&entry, native)?;
+        let canonical = c.encode()?;
+        let op = decode_operation(required(operations.get(c.operation_id.as_slice())?)?.value())?;
+        let canonical_matches = op.canonical_command == canonical;
+        let expected = if canonical_matches {
+            op.result
+        } else {
+            CreatedResult::OperationConflict
+        };
+        if native {
+            let replay = if let Some((previous, result)) = replay_operations.get(&c.operation_id) {
+                if *previous == canonical {
+                    result.clone()
+                } else {
+                    CreatedResult::OperationConflict
+                }
+            } else {
+                let object = object_bytes(&c.object)?;
+                let key = version_key(&c.object, c.prepared.version_id)?;
+                let actual = replay_heads.get(&object).copied();
+                let result = if c.binding_revision != binding.revision {
+                    CreatedResult::BindingMismatch
+                } else if actual != c.expected_head {
+                    CreatedResult::HeadMismatch { actual_head: actual }
+                } else if replay_versions.contains(&key) {
+                    CreatedResult::VersionConflict
+                } else {
+                    replay_heads.insert(object, c.prepared.version_id);
+                    replay_versions.insert(key);
+                    CreatedResult::Created {
+                        version_id: c.prepared.version_id,
+                        event_id: c.operation_id,
+                        decision_log_id: r.log_id,
+                    }
+                };
+                replay_operations.insert(c.operation_id, (canonical.clone(), result.clone()));
+                result
+            };
+            if replay != r.result {
+                return Err(StoreError::CorruptRecord);
+            }
+        }
+        if expected != r.result {
+            return Err(StoreError::CorruptRecord);
+        }
+        if canonical_matches {
+            first_operations.entry(c.operation_id).or_insert(r.log_id);
+        }
+        observed = Some(r.log_id);
+        next = k.value().checked_add(1)
+    }
+    if observed != last {
+        return Err(StoreError::CorruptRecord);
+    }
+    if native {
+        if native_application_metadata(meta)?.1 != expected_membership
+            || heads.len()? != u64::try_from(replay_heads.len()).map_err(|_| StoreError::CorruptRecord)?
+            || versions.len()? != u64::try_from(replay_versions.len()).map_err(|_| StoreError::CorruptRecord)?
+        {
+            return Err(StoreError::CorruptRecord);
+        }
+        for (object, id) in replay_heads {
+            if required(heads.get(object.as_slice())?)?.value() != id {
+                return Err(StoreError::CorruptRecord);
+            }
+        }
+    }
+    for row in operations.iter()? {
+        let (k, v) = row?;
+        let op = decode_operation(v.value())?;
+        let c = DecideCreated::decode(&op.canonical_command)?;
+        if k.value() != c.operation_id {
+            return Err(StoreError::CorruptRecord);
+        }
+        let first = required(first_operations.get(&c.operation_id))?;
+        if let CreatedResult::Created {
+            version_id,
+            event_id,
+            decision_log_id,
+        } = op.result
+        {
+            if version_id != c.prepared.version_id
+                || event_id != c.operation_id
+                || c.binding_revision != binding.revision
+                || decision_log_id != *first
+            {
+                return Err(StoreError::CorruptRecord);
+            }
+            let event = decode_event(required(events.get(event_id.as_slice())?)?.value())?;
+            if event
+                != (CreatedEvent {
+                    object: c.object.clone(),
+                    prepared: c.prepared.clone(),
+                    decision_log_id,
+                    binding: binding.clone(),
+                })
+            {
+                return Err(StoreError::CorruptRecord);
+            }
+        } else if events.get(c.operation_id.as_slice())?.is_some() {
+            return Err(StoreError::CorruptRecord);
+        }
+    }
+    for row in events.iter()? {
+        let (k, v) = row?;
+        let e = decode_event(v.value())?;
+        if e.binding != binding {
+            return Err(StoreError::CorruptRecord);
+        }
+        let op = decode_operation(required(operations.get(k.value())?)?.value())?;
+        let c = DecideCreated::decode(&op.canonical_command)?;
+        if c.object != e.object
+            || c.prepared != e.prepared
+            || op.result
+                != (CreatedResult::Created {
+                    version_id: e.prepared.version_id,
+                    event_id: c.operation_id,
+                    decision_log_id: e.decision_log_id,
+                })
+        {
+            return Err(StoreError::CorruptRecord);
+        }
+        let version =
+            decode_version(required(versions.get(version_key(&e.object, e.prepared.version_id)?.as_slice())?)?.value())?;
+        if version
+            != (CreatedVersion {
+                prepared: e.prepared.clone(),
+                decision_log_id: e.decision_log_id,
+                binding_revision: binding.revision,
+            })
+        {
+            return Err(StoreError::CorruptRecord);
+        }
+    }
+    for row in versions.iter()? {
+        let (k, v) = row?;
+        let version = decode_version(v.value())?;
+        let mut d = Decoder::new(k.value());
+        let object = d.object()?;
+        let id = d.uuid()?;
+        d.finish()?;
+        if id != version.prepared.version_id || version.binding_revision != binding.revision {
+            return Err(StoreError::CorruptRecord);
+        }
+        let entry = decode_log(required(logs.get(version.decision_log_id.index)?)?.value())?;
+        let c = entry_command(&entry, native)?;
+        if entry.id != version.decision_log_id
+            || c.object != object
+            || c.prepared != version.prepared
+            || events.get(c.operation_id.as_slice())?.is_none()
+        {
+            return Err(StoreError::CorruptRecord);
+        }
+        let head = required(heads.get(object_bytes(&object)?.as_slice())?)?;
+        let mut d = Decoder::new(head.value());
+        let head_id = d.uuid()?;
+        d.finish()?;
+        let current = decode_version(required(versions.get(version_key(&object, head_id)?.as_slice())?)?.value())?;
+        if current.decision_log_id.index < version.decision_log_id.index {
+            return Err(StoreError::CorruptRecord);
+        }
+    }
+    for row in heads.iter()? {
+        let (k, v) = row?;
+        let mut d = Decoder::new(k.value());
+        let object = d.object()?;
+        d.finish()?;
+        let mut d = Decoder::new(v.value());
+        let id = d.uuid()?;
+        d.finish()?;
+        decode_version(required(versions.get(version_key(&object, id)?.as_slice())?)?.value())?;
+    }
+    Ok(())
+}

crates/capture-authority/src/application_tests.rs

@@ -0,0 +1,531 @@
+use super::*;
+use crate::application::{APPLY_RESULTS, EVENTS, OPERATIONS, VERSIONS};
+use crate::tests::CrashBackend;
+use redb::ReadableTableMetadata;
+use std::{
+    io::{BufRead, BufReader, Write},
+    process::{Command, Stdio},
+    sync::mpsc,
+    time::Duration,
+};
+fn identity() -> NodeIdentity {
+    NodeIdentity {
+        cluster_id: [7; 16],
+        node_id: 0,
+    }
+}
+fn binding() -> CaptureBinding {
+    CaptureBinding {
+        revision: 9,
+        binding_id: [8; 16],
+        target_id: [9; 16],
+    }
+}
+fn command() -> DecideCreated {
+    DecideCreated {
+        operation_id: [1; 16],
+        object: ObjectIdentity {
+            bucket_incarnation: [2; 16],
+            key: vec![0, 255, 3],
+        },
+        expected_head: None,
+        prepared: PreparedIdentity {
+            version_id: [3; 16],
+            preparation_id: [4; 16],
+            content_digest: [5; 32],
+            content_length: 123,
+        },
+        binding_revision: 9,
+    }
+}
+fn id(index: u64) -> LogId {
+    LogId {
+        term: 8,
+        leader_node: 0,
+        index,
+    }
+}
+fn append(store: &mut Store, index: u64, c: &DecideCreated) {
+    store
+        .append(&[LogEntry {
+            id: id(index),
+            payload: c.encode().expect("encode command"),
+        }])
+        .expect("append command");
+}
+fn apply(store: &mut Store, index: u64, c: &DecideCreated) -> CreatedResult {
+    append(store, index, c);
+    store.save_committed(id(index)).expect("commit log");
+    store.apply_next_created(id(index)).expect("apply command")
+}
+fn success(c: &DecideCreated, index: u64) -> CreatedResult {
+    CreatedResult::Created {
+        version_id: c.prepared.version_id,
+        event_id: c.operation_id,
+        decision_log_id: id(index),
+    }
+}
+fn complete(store: &Store) {
+    let c = command();
+    assert_eq!(store.capture_binding().expect("binding"), Some(binding()));
+    assert_eq!(
+        store.created_head(&c.object).expect("head"),
+        Some(c.prepared.version_id),
+        "U11 acknowledged head lost"
+    );
+    assert_eq!(
+        store.created_version(&c.object, c.prepared.version_id).expect("version"),
+        Some(CreatedVersion {
+            prepared: c.prepared.clone(),
+            decision_log_id: id(0),
+            binding_revision: 9
+        })
+    );
+    assert_eq!(
+        store.created_event(c.operation_id).expect("event"),
+        Some(CreatedEvent {
+            object: c.object.clone(),
+            prepared: c.prepared.clone(),
+            decision_log_id: id(0),
+            binding: binding()
+        }),
+        "U11 complete event missing"
+    );
+    assert_eq!(
+        store.created_operation(c.operation_id).expect("operation"),
+        Some(CreatedOperation {
+            canonical_command: c.encode().expect("encode"),
+            result: success(&c, 0)
+        })
+    );
+    assert_eq!(
+        store.created_apply_result(0).expect("result"),
+        Some(CreatedApplyResult {
+            log_id: id(0),
+            result: success(&c, 0)
+        })
+    );
+    assert_eq!(store.last_applied().expect("last applied"), Some(id(0)));
+}
+fn fixture() -> (tempfile::TempDir, Store) {
+    let dir = tempfile::tempdir().expect("tempdir");
+    let mut store = Store::initialize(dir.path().join("db"), identity()).expect("initialize");
+    store.pin_capture_binding(binding()).expect("pin");
+    (dir, store)
+}
+fn reopen(path: &std::path::Path) -> Store {
+    let result = Store::open(path, identity());
+    assert!(result.is_ok(), "U11 complete reopen: {:?}", result.as_ref().err());
+    result.expect("asserted successful reopen")
+}
+#[test]
+fn created_codec_is_canonical_and_strict() {
+    let c = command();
+    let b = c.encode().expect("encode");
+    let mut golden = b"CRTD\x01".to_vec();
+    golden.extend([1; 16]);
+    golden.extend([2; 16]);
+    golden.extend(3u32.to_be_bytes());
+    golden.extend([0, 255, 3]);
+    golden.push(0);
+    golden.extend([3; 16]);
+    golden.extend([4; 16]);
+    golden.extend([5; 32]);
+    golden.extend(123u64.to_be_bytes());
+    golden.extend(9u64.to_be_bytes());
+    assert_eq!(b, golden);
+    assert_eq!(DecideCreated::decode(&b).expect("decode"), c);
+    let mut variants = Vec::new();
+    let mut v = c.clone();
+    v.operation_id = [6; 16];
+    variants.push(v);
+    let mut v = c.clone();
+    v.object.bucket_incarnation = [6; 16];
+    variants.push(v);
+    let mut v = c.clone();
+    v.object.key.push(4);
+    variants.push(v);
+    let mut v = c.clone();
+    v.expected_head = Some([6; 16]);
+    variants.push(v);
+    let mut v = c.clone();
+    v.prepared.version_id = [6; 16];
+    variants.push(v);
+    let mut v = c.clone();
+    v.prepared.preparation_id = [6; 16];
+    variants.push(v);
+    let mut v = c.clone();
+    v.prepared.content_digest = [6; 32];
+    variants.push(v);
+    let mut v = c.clone();
+    v.prepared.content_length += 1;
+    variants.push(v);
+    let mut v = c.clone();
+    v.binding_revision += 1;
+    variants.push(v);
+    for v in variants {
+        assert_ne!(v.encode().expect("variant"), b)
+    }
+    for n in 0..b.len() {
+        assert!(DecideCreated::decode(&b[..n]).is_err())
+    }
+    let mut bad = b.clone();
+    bad[4] = 2;
+    assert!(DecideCreated::decode(&bad).is_err());
+    let mut bad = b.clone();
+    bad[44] = 2;
+    assert!(DecideCreated::decode(&bad).is_err());
+    let mut bad = b.clone();
+    bad.push(0);
+    assert!(DecideCreated::decode(&bad).is_err());
+    let mut bad = b;
+    bad[37..41].copy_from_slice(&4097u32.to_be_bytes());
+    assert!(DecideCreated::decode(&bad).is_err());
+    let mut bad = c.clone();
+    bad.operation_id = [0; 16];
+    assert!(bad.encode().is_err());
+    bad = c;
+    bad.object.key = vec![0; MAX_OBJECT_KEY_LENGTH + 1];
+    assert!(bad.encode().is_err());
+}
+#[test]
+fn created_committed_order_and_identity_are_enforced() {
+    let (_dir, mut s) = fixture();
+    let c = command();
+    append(&mut s, 0, &c);
+    append(&mut s, 1, &c);
+    assert!(s.apply_next_created(id(0)).is_err());
+    assert_eq!(s.last_applied().expect("no apply"), None);
+    s.save_committed(id(1)).expect("committed");
+    assert!(matches!(s.apply_next_created(id(1)), Err(StoreError::InvalidApplySequence)));
+    let mut wrong = id(0);
+    wrong.term += 1;
+    assert!(s.apply_next_created(wrong).is_err());
+    wrong = id(0);
+    wrong.leader_node += 1;
+    assert!(s.apply_next_created(wrong).is_err());
+    assert_eq!(s.apply_next_created(id(0)).expect("first"), success(&c, 0));
+    complete(&s);
+    assert_eq!(s.apply_next_created(id(0)).expect("retry"), success(&c, 0));
+    complete(&s);
+    assert!(s.apply_next_created(wrong).is_err());
+    assert_eq!(s.apply_next_created(id(1)).expect("duplicate log"), success(&c, 0));
+    assert_eq!(s.last_applied().expect("last"), Some(id(1)));
+    assert_eq!(
+        s.created_apply_result(1).expect("result"),
+        Some(CreatedApplyResult {
+            log_id: id(1),
+            result: success(&c, 0)
+        })
+    );
+    let tx = s.database.begin_read().expect("read");
+    assert_eq!(tx.open_table(EVENTS).expect("events").len().expect("len"), 1);
+}
+#[test]
+fn created_head_version_event_and_dedup_reopen_together() {
+    let (dir, mut s) = fixture();
+    let c = command();
+    assert_eq!(apply(&mut s, 0, &c), success(&c, 0));
+    drop(s);
+    let mut s = reopen(&dir.path().join("db"));
+    complete(&s);
+    assert_eq!(apply(&mut s, 1, &c), success(&c, 0));
+    let mut changed = c.clone();
+    changed.prepared.content_digest = [6; 32];
+    assert_eq!(
+        apply(&mut s, 2, &changed),
+        CreatedResult::OperationConflict,
+        "U11 full canonical operation conflict"
+    );
+    assert_eq!(
+        s.created_operation(c.operation_id)
+            .expect("original")
+            .expect("operation")
+            .result,
+        success(&c, 0)
+    );
+    let mut stale = c.clone();
+    stale.operation_id = [6; 16];
+    stale.prepared.version_id = [6; 16];
+    assert_eq!(
+        apply(&mut s, 3, &stale),
+        CreatedResult::HeadMismatch {
+            actual_head: Some(c.prepared.version_id)
+        }
+    );
+    let mut reuse = c.clone();
+    reuse.operation_id = [7; 16];
+    reuse.expected_head = Some(c.prepared.version_id);
+    assert_eq!(apply(&mut s, 4, &reuse), CreatedResult::VersionConflict);
+    let mut replacement = stale.clone();
+    replacement.operation_id = [10; 16];
+    replacement.expected_head = Some(c.prepared.version_id);
+    assert_eq!(apply(&mut s, 5, &replacement), success(&replacement, 5));
+    assert_eq!(s.created_head(&c.object).expect("head"), Some(replacement.prepared.version_id));
+    assert_eq!(
+        s.created_version(&c.object, c.prepared.version_id)
+            .expect("old version")
+            .expect("version")
+            .decision_log_id,
+        id(0)
+    );
+    assert!(s.created_event(c.operation_id).expect("old event").is_some());
+    assert!(s.created_event(replacement.operation_id).expect("new event").is_some());
+    assert!(s.created_event(stale.operation_id).expect("rejection event").is_none());
+    assert_eq!(
+        apply(&mut s, 6, &stale),
+        CreatedResult::HeadMismatch {
+            actual_head: Some(c.prepared.version_id)
+        }
+    );
+    drop(s);
+    let s = reopen(&dir.path().join("db"));
+    assert_eq!(s.last_applied().expect("last"), Some(id(6)));
+    let tx = s.database.begin_read().expect("read");
+    assert_eq!(tx.open_table(EVENTS).expect("events").len().expect("len"), 2);
+}
+#[test]
+fn created_binding_is_pinned_and_not_caller_selected() {
+    let (dir, mut s) = fixture();
+    let mut bad = command();
+    bad.binding_revision = 10;
+    assert_eq!(apply(&mut s, 0, &bad), CreatedResult::BindingMismatch);
+    assert!(s.created_head(&bad.object).expect("head").is_none());
+    assert!(s.created_event(bad.operation_id).expect("event").is_none());
+    assert_eq!(
+        s.created_operation(bad.operation_id)
+            .expect("operation")
+            .expect("saved")
+            .result,
+        CreatedResult::BindingMismatch
+    );
+    s.pin_capture_binding(binding()).expect("idempotent pin");
+    let mut changed = binding();
+    changed.target_id = [10; 16];
+    assert!(matches!(s.pin_capture_binding(changed), Err(StoreError::BindingConflict)));
+    drop(s);
+    let s = reopen(&dir.path().join("db"));
+    assert_eq!(s.capture_binding().expect("binding"), Some(binding()));
+}
+#[test]
+fn created_schema_compatibility_and_corruption_are_checked() {
+    let dir = tempfile::tempdir().expect("dir");
+    let path = dir.path().join("db");
+    let s = Store::initialize(&path, identity()).expect("schema1");
+    assert_eq!(s.capture_binding().expect("unmigrated"), None);
+    drop(s);
+    let mut s = reopen(&path);
+    assert_eq!(s.last_applied().expect("no apply"), None);
+    append(&mut s, 0, &command());
+    s.save_committed(id(0)).expect("schema2");
+    drop(s);
+    let mut s = reopen(&path);
+    assert_eq!(s.capture_binding().expect("schema2 unchanged"), None);
+    s.pin_capture_binding(binding()).expect("explicit upgrade");
+    drop(s);
+    let mut s = reopen(&path);
+    assert_eq!(s.capture_binding().expect("schema3"), Some(binding()));
+    s.apply_next_created(id(0)).expect("apply");
+    drop(s);
+    let db = Database::open(&path).expect("raw db");
+    let tx = db.begin_write().expect("write corruption");
+    tx.open_table(EVENTS)
+        .expect("events")
+        .remove(command().operation_id.as_slice())
+        .expect("remove reference");
+    tx.commit().expect("commit corruption");
+    drop(db);
+    let result = Store::open(&path, identity());
+    assert!(
+        matches!(result, Err(StoreError::CorruptRecord)),
+        "U11 missing event rejected before expect"
+    );
+    let (dir, mut s) = fixture();
+    assert_eq!(s.read_committed().expect("empty committed"), None);
+    append(&mut s, 0, &command());
+    s.save_committed(id(0)).expect("committed schema3");
+    assert_eq!(s.capture_binding().expect("no downgrade"), Some(binding()));
+    drop(s);
+    let path = dir.path().join("db");
+    let db = Database::open(&path).expect("raw db");
+    let tx = db.begin_write().expect("write");
+    let mut marker = vec![1];
+    marker.extend(encode_id(id(0)));
+    tx.open_table(META)
+        .expect("meta")
+        .insert("last_applied", marker.as_slice())
+        .expect("corrupt applied");
+    tx.commit().expect("commit");
+    drop(db);
+    assert!(
+        matches!(Store::open(&path, identity()), Err(StoreError::CorruptRecord)),
+        "U11 last applied missing result rejected"
+    );
+    let (dir, s) = fixture();
+    drop(s);
+    let path = dir.path().join("db");
+    let db = Database::open(&path).expect("raw pinned db");
+    let tx = db.begin_write().expect("missing binding write");
+    tx.open_table(META)
+        .expect("meta")
+        .remove("capture_binding")
+        .expect("remove binding");
+    tx.commit().expect("binding corruption commit");
+    drop(db);
+    assert!(
+        matches!(Store::open(&path, identity()), Err(StoreError::CorruptRecord)),
+        "U11 missing pinned binding rejected"
+    );
+
+    let (dir, mut s) = fixture();
+    let c = command();
+    assert_eq!(apply(&mut s, 0, &c), success(&c, 0));
+    assert_eq!(apply(&mut s, 1, &c), success(&c, 0));
+    drop(s);
+    let path = dir.path().join("db");
+    let db = Database::open(&path).expect("raw duplicate db");
+    let tx = db.begin_write().expect("retarget decision write");
+    let mut key = created::object_bytes(&c.object).expect("object key");
+    key.extend(c.prepared.version_id);
+    let mut version = Vec::new();
+    created::put_prepared(&mut version, &c.prepared);
+    version.extend(encode_id(id(1)));
+    version.extend(binding().revision.to_be_bytes());
+    tx.open_table(VERSIONS)
+        .expect("versions")
+        .insert(key.as_slice(), version.as_slice())
+        .expect("retarget version");
+    let mut event = created::object_bytes(&c.object).expect("event object");
+    created::put_prepared(&mut event, &c.prepared);
+    event.extend(encode_id(id(1)));
+    event.extend(created::binding_bytes(&binding()).expect("binding bytes"));
+    tx.open_table(EVENTS)
+        .expect("events")
+        .insert(c.operation_id.as_slice(), event.as_slice())
+        .expect("retarget event");
+    let canonical = c.encode().expect("canonical");
+    let mut operation = u64::try_from(canonical.len()).expect("length").to_be_bytes().to_vec();
+    operation.extend(canonical);
+    operation.extend(created::result_bytes(&success(&c, 1)));
+    tx.open_table(OPERATIONS)
+        .expect("operations")
+        .insert(c.operation_id.as_slice(), operation.as_slice())
+        .expect("retarget original response");
+    for index in 0..=1 {
+        let mut result = encode_id(id(index));
+        result.extend(created::result_bytes(&success(&c, 1)));
+        tx.open_table(APPLY_RESULTS)
+            .expect("results")
+            .insert(index, result.as_slice())
+            .expect("retarget duplicate response");
+    }
+    tx.commit().expect("coherent retarget commit");
+    drop(db);
+    assert!(
+        matches!(Store::open(&path, identity()), Err(StoreError::CorruptRecord)),
+        "U11 original decision cannot retarget duplicate log"
+    );
+}
+#[test]
+fn created_process_child() {
+    if let Some(path) = std::env::var_os("U11_CHILD_DB") {
+        let mut s = Store::initialize(path, identity()).expect("child init");
+        s.pin_capture_binding(binding()).expect("child pin");
+        assert_eq!(apply(&mut s, 0, &command()), success(&command(), 0));
+        complete(&s);
+        println!("U11_ACK");
+        std::io::stdout().flush().expect("flush ACK");
+        loop {
+            std::thread::park()
+        }
+    }
+    let (_dir, mut s) = fixture();
+    assert_eq!(apply(&mut s, 0, &command()), success(&command(), 0));
+    complete(&s);
+}
+#[test]
+fn created_acknowledged_apply_survives_child_kill() {
+    let dir = tempfile::tempdir().expect("dir");
+    let path = dir.path().join("db");
+    let mut child = Command::new(std::env::current_exe().expect("test executable"))
+        .args(["--exact", "application_tests::created_process_child", "--nocapture"])
+        .env("U11_CHILD_DB", &path)
+        .stdout(Stdio::piped())
+        .stderr(Stdio::inherit())
+        .spawn()
+        .expect("spawn");
+    let stdout = child.stdout.take().expect("stdout");
+    let (tx, rx) = mpsc::channel();
+    let reader = std::thread::spawn(move || {
+        for line in BufReader::new(stdout).lines() {
+            let line = line.expect("child line");
+            if line.ends_with("U11_ACK") {
+                let _ = tx.send(());
+                break;
+            }
+        }
+    });
+    let acknowledged = rx.recv_timeout(Duration::from_secs(20));
+    child.kill().expect("kill child");
+    child.wait().expect("wait child");
+    reader.join().expect("reader cleanup");
+    assert!(acknowledged.is_ok(), "U11 bounded child ACK missing");
+    complete(&reopen(&path));
+}
+#[test]
+fn created_power_cut_before_and_after_apply_flush() {
+    let (dir, mut s) = fixture();
+    append(&mut s, 0, &command());
+    s.save_committed(id(0)).expect("committed");
+    drop(s);
+    let path = dir.path().join("db");
+    let backend = CrashBackend::new(&path);
+    let mut s = Store::from_database(backend.database(), identity()).expect("backend");
+    assert_eq!(s.apply_next_created(id(0)).expect("apply ACK"), success(&command(), 0));
+    backend.cut();
+    drop(s);
+    complete(&reopen(&path));
+    for sync in 1..=2 {
+        for after in [false, true] {
+            let (dir, mut s) = fixture();
+            append(&mut s, 0, &command());
+            s.save_committed(id(0)).expect("committed");
+            drop(s);
+            let path = dir.path().join("db");
+            let backend = CrashBackend::new(&path);
+            let mut s = Store::from_database(backend.database(), identity()).expect("backend");
+            let (entered, resume) = backend.pause_sync(sync, after);
+            let writer = std::thread::spawn(move || {
+                let result = s.apply_next_created(id(0));
+                (s, result)
+            });
+            entered.wait();
+            backend.cut();
+            resume.wait();
+            let (s, result) = writer.join().expect("writer");
+            drop(s);
+            let reopened = Store::open(&path, identity());
+            assert!(
+                reopened.is_ok(),
+                "U11 apply cut must reopen whole old/new state sync={sync} after={after}: {:?}",
+                reopened.as_ref().err()
+            );
+            let s = reopened.expect("asserted reopen");
+            if s.last_applied().expect("last").is_some() {
+                complete(&s)
+            } else {
+                assert!(result.is_err());
+                assert_eq!(s.created_head(&command().object).expect("old head"), None);
+                assert_eq!(
+                    s.created_version(&command().object, command().prepared.version_id)
+                        .expect("old version"),
+                    None
+                );
+                assert_eq!(s.created_event(command().operation_id).expect("old event"), None);
+                assert_eq!(s.created_operation(command().operation_id).expect("old op"), None);
+                assert_eq!(s.created_apply_result(0).expect("old result"), None)
+            }
+            eprintln!("U11 cut boundary sync={sync} after={after}");
+        }
+    }
+}

crates/capture-authority/src/committed.rs

@@ -0,0 +1,63 @@
+use crate::{LOGS, LogId, META, Store, StoreError, decode_id, decode_log, encode_id, number};
+use redb::{ReadableDatabase, ReadableTable};
+
+pub(crate) fn validate_committed(
+    meta: &impl ReadableTable<&'static str, &'static [u8]>,
+    logs: &impl ReadableTable<u64, &'static [u8]>,
+) -> Result<Option<LogId>, StoreError> {
+    let schema = number(meta.get("schema")?.ok_or(StoreError::CorruptRecord)?.value())?;
+    let marker = meta.get("committed")?;
+    match (schema, marker) {
+        (1 | 3 | 4, None) => {
+            if matches!(schema, 3 | 4) && meta.get("last_applied")?.is_none_or(|v| v.value() != [0]) {
+                return Err(StoreError::CorruptRecord);
+            }
+            Ok(None)
+        }
+        (2 | 3 | 4, Some(marker)) => {
+            let id = decode_id(marker.value())?;
+            let record = logs.get(id.index)?.ok_or(StoreError::CorruptRecord)?;
+            if decode_log(record.value())?.id != id {
+                return Err(StoreError::CorruptRecord);
+            }
+            Ok(Some(id))
+        }
+        (1 | 2, _) => Err(StoreError::CorruptRecord),
+        _ => Err(StoreError::UnsupportedSchema),
+    }
+}
+
+impl Store {
+    /// Read the exact committed boundary, validating it against the stored log.
+    pub fn read_committed(&self) -> Result<Option<LogId>, StoreError> {
+        let tx = self.database.begin_read()?;
+        validate_committed(&tx.open_table(META)?, &tx.open_table(LOGS)?)
+    }
+
+    /// Persist a monotonic committed boundary supplied by the authority owner.
+    pub fn save_committed(&mut self, id: LogId) -> Result<(), StoreError> {
+        let tx = self.write_transaction()?;
+        {
+            let mut meta = tx.open_table(META)?;
+            let logs = tx.open_table(LOGS)?;
+            let previous = validate_committed(&meta, &logs)?;
+            let record = logs.get(id.index)?.ok_or(StoreError::InvalidCommittedLog)?;
+            if decode_log(record.value())?.id != id {
+                return Err(StoreError::InvalidCommittedLog);
+            }
+            if let Some(previous) = previous {
+                if id.index < previous.index {
+                    return Err(StoreError::CommittedRegression);
+                }
+                if id == previous {
+                    return Ok(());
+                }
+            }
+            meta.insert("committed", encode_id(id).as_slice())?;
+            if previous.is_none() && !matches!(number(meta.get("schema")?.ok_or(StoreError::CorruptRecord)?.value())?, 3 | 4) {
+                meta.insert("schema", 2u64.to_be_bytes().as_slice())?;
+            }
+        }
+        self.commit(tx)
+    }
+}

crates/capture-authority/src/created.rs

@@ -0,0 +1,242 @@
+use crate::{LogId, StoreError, decode_id, encode_id};
+
+pub const MAX_OBJECT_KEY_LENGTH: usize = 4096;
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct ObjectIdentity {
+    pub bucket_incarnation: [u8; 16],
+    pub key: Vec<u8>,
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct PreparedIdentity {
+    pub version_id: [u8; 16],
+    pub preparation_id: [u8; 16],
+    pub content_digest: [u8; 32],
+    pub content_length: u64,
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct DecideCreated {
+    pub operation_id: [u8; 16],
+    pub object: ObjectIdentity,
+    pub expected_head: Option<[u8; 16]>,
+    pub prepared: PreparedIdentity,
+    pub binding_revision: u64,
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CaptureBinding {
+    pub revision: u64,
+    pub binding_id: [u8; 16],
+    pub target_id: [u8; 16],
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum CreatedResult {
+    Created {
+        version_id: [u8; 16],
+        event_id: [u8; 16],
+        decision_log_id: LogId,
+    },
+    HeadMismatch {
+        actual_head: Option<[u8; 16]>,
+    },
+    OperationConflict,
+    BindingMismatch,
+    VersionConflict,
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CreatedVersion {
+    pub prepared: PreparedIdentity,
+    pub decision_log_id: LogId,
+    pub binding_revision: u64,
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CreatedEvent {
+    pub object: ObjectIdentity,
+    pub prepared: PreparedIdentity,
+    pub decision_log_id: LogId,
+    pub binding: CaptureBinding,
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CreatedOperation {
+    pub canonical_command: Vec<u8>,
+    pub result: CreatedResult,
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CreatedApplyResult {
+    pub log_id: LogId,
+    pub result: CreatedResult,
+}
+
+pub(crate) struct Decoder<'a> {
+    bytes: &'a [u8],
+}
+impl<'a> Decoder<'a> {
+    pub(crate) fn new(bytes: &'a [u8]) -> Self {
+        Self { bytes }
+    }
+    pub(crate) fn take<const N: usize>(&mut self) -> Result<[u8; N], StoreError> {
+        let a = self
+            .bytes
+            .get(..N)
+            .ok_or(StoreError::CorruptRecord)?
+            .try_into()
+            .map_err(|_| StoreError::CorruptRecord)?;
+        self.bytes = &self.bytes[N..];
+        Ok(a)
+    }
+    pub(crate) fn u64(&mut self) -> Result<u64, StoreError> {
+        Ok(u64::from_be_bytes(self.take()?))
+    }
+    pub(crate) fn uuid(&mut self) -> Result<[u8; 16], StoreError> {
+        let id = self.take()?;
+        if id == [0; 16] {
+            return Err(StoreError::CorruptRecord);
+        }
+        Ok(id)
+    }
+    pub(crate) fn option(&mut self) -> Result<Option<[u8; 16]>, StoreError> {
+        match self.take::<1>()?[0] {
+            0 => Ok(None),
+            1 => Ok(Some(self.uuid()?)),
+            _ => Err(StoreError::CorruptRecord),
+        }
+    }
+    pub(crate) fn object(&mut self) -> Result<ObjectIdentity, StoreError> {
+        let bucket_incarnation = self.uuid()?;
+        let len = usize::try_from(u32::from_be_bytes(self.take()?)).map_err(|_| StoreError::CorruptRecord)?;
+        if len > MAX_OBJECT_KEY_LENGTH {
+            return Err(StoreError::CorruptRecord);
+        }
+        let key = self.bytes.get(..len).ok_or(StoreError::CorruptRecord)?.to_vec();
+        self.bytes = &self.bytes[len..];
+        Ok(ObjectIdentity { bucket_incarnation, key })
+    }
+    pub(crate) fn prepared(&mut self) -> Result<PreparedIdentity, StoreError> {
+        Ok(PreparedIdentity {
+            version_id: self.uuid()?,
+            preparation_id: self.uuid()?,
+            content_digest: self.take()?,
+            content_length: self.u64()?,
+        })
+    }
+    pub(crate) fn id(&mut self) -> Result<LogId, StoreError> {
+        decode_id(&self.take::<24>()?)
+    }
+    pub(crate) fn binding(&mut self) -> Result<CaptureBinding, StoreError> {
+        Ok(CaptureBinding {
+            revision: self.u64()?,
+            binding_id: self.uuid()?,
+            target_id: self.uuid()?,
+        })
+    }
+    pub(crate) fn finish(self) -> Result<(), StoreError> {
+        if self.bytes.is_empty() {
+            Ok(())
+        } else {
+            Err(StoreError::CorruptRecord)
+        }
+    }
+    pub(crate) fn result(&mut self) -> Result<CreatedResult, StoreError> {
+        match self.take::<1>()?[0] {
+            0 => Ok(CreatedResult::Created {
+                version_id: self.uuid()?,
+                event_id: self.uuid()?,
+                decision_log_id: self.id()?,
+            }),
+            1 => Ok(CreatedResult::HeadMismatch {
+                actual_head: self.option()?,
+            }),
+            2 => Ok(CreatedResult::OperationConflict),
+            3 => Ok(CreatedResult::BindingMismatch),
+            4 => Ok(CreatedResult::VersionConflict),
+            _ => Err(StoreError::CorruptRecord),
+        }
+    }
+}
+pub(crate) fn put_option(b: &mut Vec<u8>, id: Option<[u8; 16]>) {
+    b.push(u8::from(id.is_some()));
+    if let Some(id) = id {
+        b.extend(id)
+    }
+}
+pub(crate) fn object_bytes(o: &ObjectIdentity) -> Result<Vec<u8>, StoreError> {
+    if o.bucket_incarnation == [0; 16] || o.key.len() > MAX_OBJECT_KEY_LENGTH {
+        return Err(StoreError::CorruptRecord);
+    }
+    let capacity = 20usize.checked_add(o.key.len()).ok_or(StoreError::CorruptRecord)?;
+    let mut b = Vec::with_capacity(capacity);
+    b.extend(o.bucket_incarnation);
+    b.extend(
+        u32::try_from(o.key.len())
+            .map_err(|_| StoreError::CorruptRecord)?
+            .to_be_bytes(),
+    );
+    b.extend(&o.key);
+    Ok(b)
+}
+pub(crate) fn put_prepared(b: &mut Vec<u8>, p: &PreparedIdentity) {
+    b.extend(p.version_id);
+    b.extend(p.preparation_id);
+    b.extend(p.content_digest);
+    b.extend(p.content_length.to_be_bytes())
+}
+pub(crate) fn binding_bytes(v: &CaptureBinding) -> Result<Vec<u8>, StoreError> {
+    let mut b = v.revision.to_be_bytes().to_vec();
+    b.extend(v.binding_id);
+    b.extend(v.target_id);
+    let mut d = Decoder::new(&b);
+    d.binding()?;
+    d.finish()?;
+    Ok(b)
+}
+pub(crate) fn result_bytes(r: &CreatedResult) -> Vec<u8> {
+    let mut b = Vec::new();
+    match r {
+        CreatedResult::Created {
+            version_id,
+            event_id,
+            decision_log_id,
+        } => {
+            b.push(0);
+            b.extend(version_id);
+            b.extend(event_id);
+            b.extend(encode_id(*decision_log_id))
+        }
+        CreatedResult::HeadMismatch { actual_head } => {
+            b.push(1);
+            put_option(&mut b, *actual_head)
+        }
+        CreatedResult::OperationConflict => b.push(2),
+        CreatedResult::BindingMismatch => b.push(3),
+        CreatedResult::VersionConflict => b.push(4),
+    }
+    b
+}
+impl DecideCreated {
+    pub fn encode(&self) -> Result<Vec<u8>, StoreError> {
+        let object = object_bytes(&self.object)?;
+        let capacity = object.len().checked_add(134).ok_or(StoreError::CorruptRecord)?;
+        let mut b = Vec::with_capacity(capacity);
+        b.extend(b"CRTD\x01");
+        b.extend(self.operation_id);
+        b.extend(object);
+        put_option(&mut b, self.expected_head);
+        put_prepared(&mut b, &self.prepared);
+        b.extend(self.binding_revision.to_be_bytes());
+        Self::decode(&b)?;
+        Ok(b)
+    }
+    pub fn decode(bytes: &[u8]) -> Result<Self, StoreError> {
+        let mut d = Decoder::new(bytes);
+        if d.take::<5>()? != *b"CRTD\x01" {
+            return Err(StoreError::CorruptRecord);
+        }
+        let c = Self {
+            operation_id: d.uuid()?,
+            object: d.object()?,
+            expected_head: d.option()?,
+            prepared: d.prepared()?,
+            binding_revision: d.u64()?,
+        };
+        d.finish()?;
+        Ok(c)
+    }
+}

crates/capture-authority/src/native_snapshot.rs

@@ -0,0 +1,95 @@
+//! Owned consistent native application snapshot view.
+use crate::{Store, StoreError, application, snapshot};
+use redb::{ReadTransaction, ReadableDatabase, ReadableTable, ReadableTableMetadata};
+use std::{
+    io::Write,
+    sync::{Arc, Mutex},
+};
+
+#[derive(Debug, thiserror::Error)]
+pub enum NativeSnapshotError {
+    #[error(transparent)]
+    Store(#[from] StoreError),
+    #[error(transparent)]
+    Snapshot(#[from] snapshot::SnapshotError),
+    #[error("native snapshot Store lock poisoned")]
+    Poisoned,
+    #[error("native snapshot worker failed")]
+    Worker(#[source] tokio::task::JoinError),
+}
+
+pub struct NativeSnapshotView {
+    // Drop the transaction before the owner so the Database stays open throughout the view.
+    transaction: ReadTransaction,
+    header: snapshot::Header,
+    counts: [u64; 4],
+    _owner: Arc<Mutex<Store>>,
+}
+impl NativeSnapshotView {
+    pub(crate) fn acquire(owner: Arc<Mutex<Store>>) -> Result<Self, NativeSnapshotError> {
+        let transaction = {
+            let store = owner.lock().map_err(|_| NativeSnapshotError::Poisoned)?;
+            if !store.writable {
+                return Err(StoreError::WriteDisabled.into());
+            }
+            store.database.begin_read().map_err(StoreError::from)?
+        };
+        let (applied, membership) = application::native_application_state(&transaction)?;
+        let header = {
+            let meta = transaction.open_table(crate::META).map_err(StoreError::from)?;
+            let identity = meta
+                .get("identity")
+                .map_err(StoreError::from)?
+                .ok_or(StoreError::CorruptRecord)?;
+            snapshot::Header {
+                cluster_id: crate::decode_identity(identity.value())?.cluster_id,
+                applied,
+                membership,
+                binding: application::read_binding(&meta)?,
+            }
+        };
+        let mut counts = [0; 4];
+        for (index, definition) in [
+            application::HEADS,
+            application::VERSIONS,
+            application::EVENTS,
+            application::OPERATIONS,
+        ]
+        .into_iter()
+        .enumerate()
+        {
+            counts[index] = transaction
+                .open_table(definition)
+                .map_err(StoreError::from)?
+                .len()
+                .map_err(StoreError::from)?;
+        }
+        Ok(Self {
+            transaction,
+            header,
+            counts,
+            _owner: owner,
+        })
+    }
+
+    pub fn stream<W: Write>(self, writer: W, limits: snapshot::Limits) -> Result<W, NativeSnapshotError> {
+        let mut encoder = snapshot::Encoder::new(writer, &self.header, self.counts, limits)?;
+        for (definition, section) in [
+            (application::HEADS, snapshot::Section::Heads),
+            (application::VERSIONS, snapshot::Section::Versions),
+            (application::EVENTS, snapshot::Section::Events),
+            (application::OPERATIONS, snapshot::Section::Operations),
+        ] {
+            let table = self.transaction.open_table(definition).map_err(StoreError::from)?;
+            for row in table.iter().map_err(StoreError::from)? {
+                let (key, value) = row.map_err(StoreError::from)?;
+                encoder.row(snapshot::Row {
+                    section,
+                    key: key.value(),
+                    value: value.value(),
+                })?;
+            }
+        }
+        Ok(encoder.finish()?)
+    }
+}

crates/capture-authority/src/node.rs

@@ -0,0 +1,371 @@
+//! Validated capture peer metadata and its independent CNOD v1 codec.
+use serde::{Deserialize, Deserializer, Serialize, Serializer, ser::SerializeStruct};
+
+#[derive(Clone, Debug, Default, PartialEq, Eq)]
+pub struct CaptureNode {
+    rpc_uri: String,
+}
+
+#[derive(Debug, thiserror::Error)]
+pub enum CaptureNodeError {
+    #[error("invalid RPC URI")]
+    InvalidRpcUri,
+    #[error("invalid RPC URI")]
+    InvalidRpcUriParse(#[source] url::ParseError),
+    #[error("RPC URI too long")]
+    RpcUriTooLong,
+    #[error("invalid node magic")]
+    InvalidNodeMagic,
+    #[error("unsupported node version")]
+    UnsupportedNodeVersion,
+    #[error("invalid node length")]
+    InvalidNodeLength,
+    #[error("truncated node record")]
+    TruncatedNodeRecord,
+    #[error("invalid node UTF-8")]
+    InvalidNodeUtf8(#[source] std::str::Utf8Error),
+    #[error("trailing node bytes")]
+    TrailingNodeBytes,
+}
+impl CaptureNodeError {
+    pub fn code(&self) -> &'static str {
+        match self {
+            Self::InvalidRpcUri | Self::InvalidRpcUriParse(_) => "invalid_rpc_uri",
+            Self::RpcUriTooLong => "rpc_uri_too_long",
+            Self::InvalidNodeMagic => "invalid_node_magic",
+            Self::UnsupportedNodeVersion => "unsupported_node_version",
+            Self::InvalidNodeLength => "invalid_node_length",
+            Self::TruncatedNodeRecord => "truncated_node_record",
+            Self::InvalidNodeUtf8(_) => "invalid_node_utf8",
+            Self::TrailingNodeBytes => "trailing_node_bytes",
+        }
+    }
+}
+impl CaptureNode {
+    pub fn new(rpc_uri: String) -> Result<Self, CaptureNodeError> {
+        let node = Self { rpc_uri };
+        node.validate()?;
+        Ok(node)
+    }
+
+    pub fn rpc_uri(&self) -> &str {
+        &self.rpc_uri
+    }
+
+    pub fn validate(&self) -> Result<(), CaptureNodeError> {
+        let uri = &self.rpc_uri;
+        if uri.len() > 4096 {
+            return Err(CaptureNodeError::RpcUriTooLong);
+        }
+        if uri.is_empty() || uri.chars().any(|c| c.is_whitespace() || c.is_control() || c == '\\') {
+            return Err(CaptureNodeError::InvalidRpcUri);
+        }
+        let rest = uri
+            .strip_prefix("http://")
+            .or_else(|| uri.strip_prefix("https://"))
+            .ok_or(CaptureNodeError::InvalidRpcUri)?;
+        if uri.contains('?') || uri.contains('#') {
+            return Err(CaptureNodeError::InvalidRpcUri);
+        }
+        let (authority, path) = rest.split_once('/').map_or((rest, ""), |(a, p)| (a, p));
+        // An empty suffix after the first slash is the only allowed path.
+        if !path.is_empty() || authority.contains('@') {
+            return Err(CaptureNodeError::InvalidRpcUri);
+        }
+        let (host, port) = if authority.starts_with('[') {
+            let end = authority.find(']').ok_or(CaptureNodeError::InvalidRpcUri)?;
+            let port = authority
+                .get(end + 1..)
+                .and_then(|suffix| suffix.strip_prefix(':'))
+                .ok_or(CaptureNodeError::InvalidRpcUri)?;
+            (&authority[..=end], port)
+        } else {
+            authority.split_once(':').ok_or(CaptureNodeError::InvalidRpcUri)?
+        };
+        if host.is_empty() || port.is_empty() || !port.bytes().all(|b| b.is_ascii_digit()) {
+            return Err(CaptureNodeError::InvalidRpcUri);
+        }
+        let port = port.parse::<u16>().map_err(|_| CaptureNodeError::InvalidRpcUri)?;
+        if port == 0 {
+            return Err(CaptureNodeError::InvalidRpcUri);
+        }
+        let parsed_url = url::Url::parse(uri).map_err(CaptureNodeError::InvalidRpcUriParse)?;
+        if parsed_url.host().is_none() {
+            return Err(CaptureNodeError::InvalidRpcUri);
+        }
+        Ok(())
+    }
+
+    pub fn encode(&self) -> Result<Vec<u8>, CaptureNodeError> {
+        self.validate()?;
+        let length = u32::try_from(self.rpc_uri.len()).map_err(|_| CaptureNodeError::InvalidNodeLength)?;
+        let mut bytes = b"CNOD\x01".to_vec();
+        bytes.extend_from_slice(&length.to_be_bytes());
+        bytes.extend_from_slice(self.rpc_uri.as_bytes());
+        Ok(bytes)
+    }
+
+    pub fn decode(bytes: &[u8]) -> Result<Self, CaptureNodeError> {
+        if bytes.len() < 9 {
+            return Err(CaptureNodeError::TruncatedNodeRecord);
+        }
+        if &bytes[..4] != b"CNOD" {
+            return Err(CaptureNodeError::InvalidNodeMagic);
+        }
+        if bytes[4] != 1 {
+            return Err(CaptureNodeError::UnsupportedNodeVersion);
+        }
+        let length = u32::from_be_bytes(bytes[5..9].try_into().map_err(|_| CaptureNodeError::InvalidNodeLength)?);
+        if !(1..=4096).contains(&length) {
+            return Err(CaptureNodeError::InvalidNodeLength);
+        }
+        let length = usize::try_from(length).map_err(|_| CaptureNodeError::InvalidNodeLength)?;
+        let end = 9_usize.checked_add(length).ok_or(CaptureNodeError::InvalidNodeLength)?;
+        if bytes.len() < end {
+            return Err(CaptureNodeError::TruncatedNodeRecord);
+        }
+        if bytes.len() != end {
+            return Err(CaptureNodeError::TrailingNodeBytes);
+        }
+        let uri = std::str::from_utf8(&bytes[9..end]).map_err(CaptureNodeError::InvalidNodeUtf8)?;
+        Self::new(uri.to_owned())
+    }
+}
+impl Serialize for CaptureNode {
+    fn serialize<S: Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> {
+        self.validate().map_err(serde::ser::Error::custom)?;
+        let mut state = serializer.serialize_struct("CaptureNode", 1)?;
+        state.serialize_field("rpc_uri", &self.rpc_uri)?;
+        state.end()
+    }
+}
+impl<'de> Deserialize<'de> for CaptureNode {
+    fn deserialize<D: Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> {
+        #[derive(Deserialize)]
+        #[serde(deny_unknown_fields)]
+        struct Fields {
+            rpc_uri: String,
+        }
+        let fields = Fields::deserialize(deserializer).map_err(|_| serde::de::Error::custom("invalid capture node fields"))?;
+        Self::new(fields.rpc_uri).map_err(serde::de::Error::custom)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::error::Error;
+
+    fn record(payload: &[u8]) -> Vec<u8> {
+        let mut bytes = b"CNOD\x01".to_vec();
+        bytes.extend_from_slice(&u32::try_from(payload.len()).expect("test payload length").to_be_bytes());
+        bytes.extend_from_slice(payload);
+        bytes
+    }
+    fn reject_uri(uri: &str) {
+        let result = CaptureNode::new(uri.to_owned());
+        assert!(result.is_err(), "URI should reject: {uri:?}");
+        assert_eq!(result.expect_err("rejected URI").code(), "invalid_rpc_uri");
+    }
+    fn reject_record(bytes: &[u8], code: &str) {
+        let result = CaptureNode::decode(bytes);
+        assert!(result.is_err(), "record should reject with {code}");
+        assert_eq!(result.expect_err("rejected record").code(), code);
+    }
+    #[test]
+    fn actual_openraft_node_bound() {
+        fn require_node<T: openraft::Node>() {}
+        require_node::<CaptureNode>();
+        assert_eq!(CaptureNode::default().rpc_uri(), "");
+    }
+    #[test]
+    fn default_and_empty_rejected_at_every_boundary() {
+        let node = CaptureNode::default();
+        assert_eq!(node.rpc_uri(), "");
+        reject_uri("");
+        assert_eq!(node.validate().expect_err("empty validation").code(), "invalid_rpc_uri");
+        assert!(serde_json::to_string(&node).is_err());
+        assert!(serde_json::from_str::<CaptureNode>(r#"{"rpc_uri":""}"#).is_err());
+        assert_eq!(node.encode().expect_err("empty encoding").code(), "invalid_rpc_uri");
+        reject_record(&record(b""), "invalid_node_length");
+    }
+    #[test]
+    fn accepted_uris_preserve_exact_bytes() {
+        for uri in [
+            "http://node:80",
+            "https://node:443",
+            "http://node:1",
+            "https://node:65535/",
+            "http://127.0.0.1:9000",
+            "https://[::1]:443/",
+            "http://[2001:db8::1]:1",
+            "http://NoDe:00080/",
+        ] {
+            let result = CaptureNode::new(uri.to_owned());
+            assert!(result.is_ok(), "valid URI construction: {uri}");
+            let node = result.expect("valid URI");
+            assert_eq!(node.rpc_uri(), uri, "preserve explicit default port and raw URI");
+            assert_eq!(node.encode().expect("encode URI"), record(uri.as_bytes()));
+            assert_eq!(CaptureNode::decode(&record(uri.as_bytes())).expect("decode URI"), node);
+        }
+    }
+    #[test]
+    fn golden_cnod_bytes() {
+        let node = CaptureNode {
+            rpc_uri: "http://node:80".to_owned(),
+        };
+        let golden = b"CNOD\x01\x00\x00\x00\x0ehttp://node:80";
+        assert_eq!(node.encode().ok().as_deref(), Some(golden.as_slice()), "golden CNOD bytes");
+    }
+    #[test]
+    fn valid_cnod_decode() {
+        let golden = b"CNOD\x01\x00\x00\x00\x0ehttp://node:80";
+        let decoded = CaptureNode::decode(golden);
+        assert!(decoded.is_ok(), "golden decode succeeds");
+        assert_eq!(decoded.expect("golden decoded node").rpc_uri(), "http://node:80");
+    }
+    #[test]
+    fn serde_shape_roundtrip_and_strict_fields() {
+        let node = CaptureNode::new("https://node:443/".to_owned()).expect("serde node");
+        let json = r#"{"rpc_uri":"https://node:443/"}"#;
+        assert_eq!(serde_json::to_string(&node).expect("serialize node"), json);
+        assert_eq!(serde_json::from_str::<CaptureNode>(json).expect("deserialize node"), node);
+        for json in [
+            r#"{}"#,
+            r#"{"rpc_uri":1}"#,
+            r#"{"rpc_uri":null}"#,
+            r#"{"rpc_uri":"http://node:1","extra":0}"#,
+            r#"{"rpc_uri":"http://node:1","rpc_uri":"http://node:2"}"#,
+        ] {
+            assert!(serde_json::from_str::<CaptureNode>(json).is_err(), "strict fields: {json}");
+        }
+    }
+    #[test]
+    fn credentials_and_raw_uri_shapes_reject() {
+        for uri in [
+            "http://user@node:1",
+            "http://user:password@node:1",
+            "http://@node:1",
+            "HTTP://node:1",
+            "Https://node:1",
+            "http://node",
+            "http://node:",
+            "http://node:0",
+            "http://node:-1",
+            "http://node:+1",
+            "http://node:65536",
+            "http://node:999999999999999999999",
+            "http://[::1:1",
+            "http://[::1]extra:1",
+            "http://[invalid]:1",
+            "http://::1:1",
+            "http://:1",
+            "/disk/path",
+            "file:///disk/path",
+            "http://node:1/./",
+            "http://node:1/../",
+            "http://node:1/%2e/",
+            "http://node:1/%2E%2e/",
+            "http://node:1/path",
+            "http://node:1\\",
+            " http://node:1",
+            "http://node:1\n",
+            "http://no\tde:1",
+            "http://node:1\u{00a0}",
+            "http://node:1\u{0085}",
+            "http://node:1\0",
+        ] {
+            reject_uri(uri);
+        }
+    }
+    #[test]
+    fn query_and_fragment_markers_reject() {
+        for uri in [
+            "http://node:1?",
+            "http://node:1?x=y",
+            "http://node:1/?",
+            "http://node:1#",
+            "http://node:1#fragment",
+            "http://node:1/#",
+        ] {
+            reject_uri(uri);
+        }
+    }
+    #[test]
+    fn utf8_byte_length_boundaries() {
+        for length in [4095, 4096] {
+            let prefix = "http://node:";
+            let uri = format!("{prefix}{}1", "0".repeat(length - prefix.len() - 1));
+            assert_eq!(uri.len(), length);
+            assert!(url::Url::parse(&uri).is_ok(), "boundary fixture accepted by actual URL parser");
+            let node = CaptureNode::new(uri.clone()).expect("boundary node");
+            assert_eq!(node.rpc_uri(), uri);
+            assert_eq!(
+                CaptureNode::decode(&node.encode().expect("boundary encoding")).expect("boundary decoding"),
+                node
+            );
+        }
+        let uri = format!("http://node:{}1", "0".repeat(4097 - "http://node:".len() - 1));
+        assert_eq!(CaptureNode::new(uri).expect_err("oversize URI").code(), "rpc_uri_too_long");
+        let unicode = format!("http://{}:1", "é".repeat(2044));
+        assert!(unicode.chars().count() < 4096);
+        assert!(unicode.len() > 4096);
+        assert_eq!(CaptureNode::new(unicode).expect_err("UTF-8 byte limit").code(), "rpc_uri_too_long");
+    }
+    #[test]
+    fn cnod_headers_and_lengths_reject_exact_codes() {
+        let valid = record(b"http://node:80");
+        for end in 0..9 {
+            reject_record(&valid[..end], "truncated_node_record");
+        }
+        let mut bad = valid.clone();
+        bad[0] = b'X';
+        reject_record(&bad, "invalid_node_magic");
+        bad = valid.clone();
+        bad[4] = 2;
+        reject_record(&bad, "unsupported_node_version");
+        for length in [0_u32, 4097, u32::MAX] {
+            bad = b"CNOD\x01".to_vec();
+            bad.extend_from_slice(&length.to_be_bytes());
+            reject_record(&bad, "invalid_node_length");
+        }
+        for end in 9..valid.len() {
+            reject_record(&valid[..end], "truncated_node_record");
+        }
+        reject_record(&record(&[0xff]), "invalid_node_utf8");
+        reject_record(&record(&[0xc3, 0x28]), "invalid_node_utf8");
+        reject_record(&record(b"invalid URI"), "invalid_rpc_uri");
+    }
+    #[test]
+    fn trailing_bytes_reject() {
+        for tail in [&b"x"[..], &b"xyz"[..]] {
+            let mut bytes = record(b"http://node:80");
+            bytes.extend_from_slice(tail);
+            reject_record(&bytes, "trailing_node_bytes");
+        }
+    }
+    #[test]
+    fn errors_and_sources_redact_input() {
+        let sentinel = "http://SECRET_USER:SECRET_PASSWORD@node:1";
+        let error = CaptureNode::new(sentinel.to_owned()).expect_err("credentials reject");
+        let mut current: &dyn Error = &error;
+        loop {
+            let rendered = format!("{current} {current:?}");
+            assert!(!rendered.contains("SECRET"));
+            match current.source() {
+                Some(source) => current = source,
+                None => break,
+            }
+        }
+        let json = format!(r#"{{"rpc_uri":"{sentinel}"}}"#);
+        let error = serde_json::from_str::<CaptureNode>(&json).expect_err("serde credentials reject");
+        assert!(!format!("{error} {error:?}").contains("SECRET"));
+        let json = format!(r#"{{"{sentinel}":0}}"#);
+        let error = serde_json::from_str::<CaptureNode>(&json).expect_err("unknown fields reject");
+        assert!(!format!("{error} {error:?}").contains("SECRET"));
+        let error = CaptureNode::new("http://[bad]:1".to_owned()).expect_err("parser rejection");
+        assert!(error.source().is_some(), "retain URL parser source");
+        let error = CaptureNode::decode(&record(&[0xff])).expect_err("UTF-8 rejection");
+        assert!(error.source().is_some(), "retain UTF-8 source");
+    }
+}

crates/capture-authority/src/truncate.rs

@@ -0,0 +1,43 @@
+use crate::{LOGS, META, Store, StoreError, decode_id, decode_log, encode_id};
+use redb::ReadableTable;
+
+impl Store {
+    /// Atomically remove every log entry at or after `from`.
+    pub fn truncate_suffix(&mut self, from: u64) -> Result<(), StoreError> {
+        let tx = self.write_transaction()?;
+        {
+            let mut meta = tx.open_table(META)?;
+            let logs = tx.open_table(LOGS)?;
+            if crate::committed::validate_committed(&meta, &logs)?.is_some_and(|id| from <= id.index) {
+                return Err(StoreError::CommittedLogProtected);
+            }
+            drop(logs);
+            let tail = meta.get("tail")?.map(|v| decode_id(v.value())).transpose()?;
+            if tail.is_none_or(|id| from > id.index) {
+                return Ok(());
+            }
+            let mut logs = tx.open_table(LOGS)?;
+            let new_tail = if from == 0 {
+                None
+            } else {
+                let index = from - 1;
+                let record = logs.get(index)?.ok_or(StoreError::InvalidLogSequence)?;
+                let id = decode_log(record.value())?.id;
+                if id.index != index {
+                    return Err(StoreError::CorruptRecord);
+                }
+                Some(id)
+            };
+            logs.retain(|index, _| index < from)?;
+            match new_tail {
+                Some(id) => {
+                    meta.insert("tail", encode_id(id).as_slice())?;
+                }
+                None => {
+                    meta.remove("tail")?;
+                }
+            }
+        }
+        self.commit(tx)
+    }
+}

crates/common/src/mrf_channel.rs

@@ -163,6 +163,15 @@ pub struct MrfDurableRepairAnchor {
     pub bucket_incarnation_id: Uuid,
 }
 
+/// A completed deep check proved that a legacy object is present and healthy,
+/// but the format has no independent identity commitment that can discharge a
+/// durable partial-write responsibility. Keep the journal entry and pause
+/// automatic retries until an operator has repaired the source of truth.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct MrfUnverifiedLegacyEvent {
+    pub anchor: MrfDurableRepairAnchor,
+}
+
 impl MrfDurableRepairAnchor {
     /// Build a dischargeable anchor only when the caller supplies the storage
     /// incarnation and the original ingress lease. Legacy replay records lack
@@ -324,6 +333,9 @@ pub enum MrfDurableAdmissionError {
 /// cancel repair.
 pub struct MrfDurableSubmission {
     pub intent: MrfIntent,
+    /// Bucket incarnation observed by the committed-write caller, when
+    /// available. Old journal records and legacy callers remain unbound.
+    pub source_bucket_incarnation_id: Option<Uuid>,
     pub response: oneshot::Sender<Result<(), MrfDurableAdmissionError>>,
 }
 
@@ -344,6 +356,19 @@ pub async fn persist_partial_write_intent(
     version_id: Option<Uuid>,
     scope: MrfScope,
 ) -> Result<(), MrfDurableAdmissionError> {
+    persist_partial_write_intent_with_incarnation(bucket, object, version_id, scope, None).await
+}
+
+pub async fn persist_partial_write_intent_with_incarnation(
+    bucket: &str,
+    object: &str,
+    version_id: Option<Uuid>,
+    scope: MrfScope,
+    source_bucket_incarnation_id: Option<Uuid>,
+) -> Result<(), MrfDurableAdmissionError> {
+    if source_bucket_incarnation_id.is_some_and(|incarnation| incarnation.is_nil()) {
+        return Err(MrfDurableAdmissionError::InvalidIdentity);
+    }
     let intent = MrfIntent {
         bucket: Arc::from(bucket),
         object: Arc::from(object),
@@ -355,7 +380,7 @@ pub async fn persist_partial_write_intent(
         enqueued_at_ms: unix_now_ms(),
         attempts: 0,
     };
-    persist_durable_intent(intent).await
+    persist_durable_intent(intent, source_bucket_incarnation_id).await
 }
 
 pub async fn persist_delete_marker_purge_intent(
@@ -368,6 +393,7 @@ pub async fn persist_delete_marker_purge_intent(
     if version_id.is_nil() {
         return Err(MrfDurableAdmissionError::InvalidIdentity);
     }
+    let source_bucket_incarnation_id = delete_marker_purge.bucket_incarnation_id;
     let intent = MrfIntent {
         bucket: Arc::from(bucket),
         object: Arc::from(object),
@@ -379,17 +405,23 @@ pub async fn persist_delete_marker_purge_intent(
         enqueued_at_ms: unix_now_ms(),
         attempts: 0,
     };
-    persist_durable_intent_unconditionally(intent).await
+    persist_durable_intent_unconditionally(intent, Some(source_bucket_incarnation_id)).await
 }
 
-async fn persist_durable_intent(intent: MrfIntent) -> Result<(), MrfDurableAdmissionError> {
+async fn persist_durable_intent(
+    intent: MrfIntent,
+    source_bucket_incarnation_id: Option<Uuid>,
+) -> Result<(), MrfDurableAdmissionError> {
     if !mrf_delivery_enabled() {
         return Err(MrfDurableAdmissionError::Disabled);
     }
-    persist_durable_intent_unconditionally(intent).await
+    persist_durable_intent_unconditionally(intent, source_bucket_incarnation_id).await
 }
 
-async fn persist_durable_intent_unconditionally(mut intent: MrfIntent) -> Result<(), MrfDurableAdmissionError> {
+async fn persist_durable_intent_unconditionally(
+    mut intent: MrfIntent,
+    source_bucket_incarnation_id: Option<Uuid>,
+) -> Result<(), MrfDurableAdmissionError> {
     if intent.bucket.is_empty()
         || intent.object.is_empty()
         || intent.bucket.len() > MRF_MAX_IDENTITY_COMPONENT
@@ -404,7 +436,11 @@ async fn persist_durable_intent_unconditionally(mut intent: MrfIntent) -> Result
     }
     let (response, receipt) = oneshot::channel();
     sender
-        .try_send(MrfDurableSubmission { intent, response })
+        .try_send(MrfDurableSubmission {
+            intent,
+            source_bucket_incarnation_id,
+            response,
+        })
         .map_err(|err| match err {
             mpsc::error::TrySendError::Full(_) => MrfDurableAdmissionError::Full,
             mpsc::error::TrySendError::Closed(_) => MrfDurableAdmissionError::Unavailable,
@@ -782,6 +818,8 @@ const MRF_REPAIRED_EVENT_CAP: usize = 4096;
 static MRF_REPAIRED_EVENTS: OnceLock<std::sync::Mutex<std::collections::VecDeque<MrfRepairedEvent>>> = OnceLock::new();
 static MRF_VERIFIED_REPAIR_EVENTS: OnceLock<std::sync::Mutex<std::collections::VecDeque<MrfVerifiedRepairEvent>>> =
     OnceLock::new();
+static MRF_UNVERIFIED_LEGACY_EVENTS: OnceLock<std::sync::Mutex<std::collections::VecDeque<MrfUnverifiedLegacyEvent>>> =
+    OnceLock::new();
 
 /// Record a legacy notification for compatibility. This is not an
 /// acknowledgement of storage verification or durable repair completion.
@@ -859,6 +897,44 @@ pub fn take_mrf_verified_repair_events_for(bucket: &str) -> Vec<MrfVerifiedRepai
     taken
 }
 
+/// Record a bounded, exact-identity notice that a legacy durable partial-write
+/// was checked but cannot be proven from its on-disk format. This is not a
+/// repair proof and must never release the durable responsibility.
+pub fn note_mrf_unverified_legacy(event: MrfUnverifiedLegacyEvent) {
+    let registry = MRF_UNVERIFIED_LEGACY_EVENTS.get_or_init(|| std::sync::Mutex::new(std::collections::VecDeque::new()));
+    let Ok(mut events) = registry.lock() else {
+        return;
+    };
+    if events.contains(&event) {
+        return;
+    }
+    if events.len() >= MRF_REPAIRED_EVENT_CAP {
+        events.pop_front();
+    }
+    events.push_back(event);
+}
+
+/// Take held-legacy notices for one bucket. The durable ledger still verifies
+/// the full anchor before changing the journal entry to its held state.
+pub fn take_mrf_unverified_legacy_events_for(bucket: &str) -> Vec<MrfUnverifiedLegacyEvent> {
+    let Some(registry) = MRF_UNVERIFIED_LEGACY_EVENTS.get() else {
+        return Vec::new();
+    };
+    let Ok(mut events) = registry.lock() else {
+        return Vec::new();
+    };
+    let mut taken = Vec::new();
+    events.retain(|event| {
+        if event.anchor.bucket.as_ref() == bucket {
+            taken.push(event.clone());
+            false
+        } else {
+            true
+        }
+    });
+    taken
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -1233,6 +1309,53 @@ mod tests {
         assert!(take_mrf_verified_repair_events_for("verified-bucket-a").is_empty());
         assert_eq!(take_mrf_verified_repair_events_for("verified-bucket-b").len(), 1);
     }
+
+    #[test]
+    fn unverified_legacy_events_are_bucket_scoped_and_bounded() {
+        let bucket = Arc::<str>::from("unverified-legacy-event-bucket");
+        let _ = take_mrf_unverified_legacy_events_for(bucket.as_ref());
+        let event = MrfUnverifiedLegacyEvent {
+            anchor: MrfDurableRepairAnchor {
+                kind: MrfKind::PartialWrite,
+                bucket: bucket.clone(),
+                object: Arc::from("unverified-object"),
+                version_id: None,
+                scope: Some(MrfScope {
+                    pool_index: 1,
+                    set_index: 2,
+                }),
+                delete_marker_purge: None,
+                lease: MrfIngressLease::new(912),
+                bucket_incarnation_id: Uuid::new_v4(),
+            },
+        };
+        note_mrf_unverified_legacy(event.clone());
+        note_mrf_unverified_legacy(event.clone());
+        assert_eq!(take_mrf_unverified_legacy_events_for(bucket.as_ref()), vec![event]);
+        assert!(take_mrf_unverified_legacy_events_for(bucket.as_ref()).is_empty());
+
+        let cap_bucket = Arc::<str>::from("unverified-legacy-event-cap-bucket");
+        for index in 0..MRF_REPAIRED_EVENT_CAP + 4 {
+            note_mrf_unverified_legacy(MrfUnverifiedLegacyEvent {
+                anchor: MrfDurableRepairAnchor {
+                    kind: MrfKind::PartialWrite,
+                    bucket: cap_bucket.clone(),
+                    object: Arc::from(format!("object-{index}")),
+                    version_id: None,
+                    scope: Some(MrfScope {
+                        pool_index: 1,
+                        set_index: 2,
+                    }),
+                    delete_marker_purge: None,
+                    lease: MrfIngressLease::new(u64::try_from(index).expect("test index fits")),
+                    bucket_incarnation_id: Uuid::new_v4(),
+                },
+            });
+        }
+        let events = take_mrf_unverified_legacy_events_for(cap_bucket.as_ref());
+        assert_eq!(events.len(), MRF_REPAIRED_EVENT_CAP);
+        assert_eq!(events.first().expect("bounded ring is non-empty").anchor.object.as_ref(), "object-4");
+    }
 }
 #[tokio::test]
 async fn partial_write_durable_admission_waits_for_persistence_and_renews_lease() {

crates/config/README.md

@@ -168,9 +168,10 @@ Scanner cycle budget controls:
 
 ## Foreground write admission environment variables
 
-Large direct `PutObject` requests and multipart `UploadPart` requests share one
-per-process permit pool that bounds concurrent body ingest and storage writes.
-Small direct PUTs stay on the legacy path.
+Large direct `PutObject` requests, multipart `UploadPart` requests, and
+server-side multipart `UploadPartCopy` requests share one per-process permit
+pool that bounds concurrent body ingest and storage writes. Small direct PUTs
+stay on the legacy path.
 
 - `RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE`
   - enables the default-on pool; `false` keeps only the soft request counter.
@@ -190,11 +191,11 @@ Small direct PUTs stay on the legacy path.
   - smallest `UploadPart` that takes a permit; `0` gates every part.
   - default is `0`.
 - `RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS`
-  - how long an `UploadPart` waits in the bounded queue for a permit before returning S3 `SlowDown`; `0` rejects immediately when the pool is full.
-  - default is `10000`. Parts wait before body ingest, so SDK-default clients that send every part of an upload concurrently drain through the pool instead of failing on a full pool.
-  - RustFS does not read the request body while a part is queued, so the client's socket write stalls for the whole wait and whatever timeout the client or an intermediary has configured competes with this value. Keep it with margin below the shortest such timeout in use (botocore applies its 60 s `connect_timeout` to the body write; the AWS SDK for Java v2 has a 30 s socket write timeout; reverse proxies add their own body timeouts); a wait that outlives the client timeout surfaces as a dropped connection instead of `SlowDown`.
+  - how long an `UploadPart` or `UploadPartCopy` waits in the bounded queue for a permit before returning S3 `SlowDown`; `0` rejects immediately when the pool is full.
+  - default is `10000`. Parts wait before body ingest or source-object reads, so SDK-default clients that send every part of an upload concurrently drain through the pool instead of failing on a full pool.
+  - RustFS does not read the request body while a part is queued, so the client's socket write stalls for the whole wait and whatever timeout the client or an intermediary has configured competes with this value. `UploadPartCopy` waits before taking bucket lifecycle locks or opening source readers, and uses the unknown-size weight because the authoritative copy length is available only after source metadata and range validation. Keep the wait with margin below the shortest client/proxy timeout in use (botocore applies its 60 s `connect_timeout` to the body write; the AWS SDK for Java v2 has a 30 s socket write timeout; reverse proxies add their own body timeouts); a wait that outlives the client timeout surfaces as a dropped connection instead of `SlowDown`.
 - `RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MAX_PENDING`
-  - maximum `UploadPart` requests waiting for a permit at once; parts beyond it return `SlowDown` without waiting.
+  - maximum `UploadPart` and `UploadPartCopy` requests waiting for a permit at once; parts beyond it return `SlowDown` without waiting.
   - default is `0`, which derives 16 times the large-write slot count, or the explicit request-count limit (512 at stock settings). Automatic subdivision does not enlarge this queue.
   - each queued HTTP/1 part holds whatever unread body the client already pushed into the connection's kernel receive buffer (an HTTP/2 part holds up to its flow-control window in process memory). Autotuned receive buffers can remain large on reused connections; queue depth does not imply a fixed per-connection memory cost.
 - `RUSTFS_PUT_FOREGROUND_ADMISSION_ENABLE`, `RUSTFS_PUT_FOREGROUND_ADMISSION_LIMIT`, `RUSTFS_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS`

crates/config/src/constants/app.rs

@@ -18,10 +18,8 @@ use const_str::concat;
 /// Default value: RustFS
 /// Environment variable: RUSTFS_APP_NAME
 pub const APP_NAME: &str = "RustFS";
-/// Application version
-/// Default value: 1.0.0
-/// Environment variable: RUSTFS_VERSION
-pub const VERSION: &str = "1.0.0";
+/// Cargo package version for this workspace member.
+pub const VERSION: &str = env!("CARGO_PKG_VERSION");
 
 /// Default configuration logger level
 /// Default value: error
@@ -39,12 +37,6 @@ pub const SAMPLE_RATIO: f64 = 1.0;
 /// Default value: 30
 pub const METER_INTERVAL: u64 = 30;
 
-/// Default configuration service version
-/// Default value: 1.0.0
-/// Environment variable: RUSTFS_OBS_SERVICE_VERSION
-/// Uses the same value as VERSION constant
-pub const SERVICE_VERSION: &str = "1.0.0";
-
 /// Default configuration environment
 /// Default value: production
 pub const ENVIRONMENT: &str = "production";
@@ -410,7 +402,6 @@ mod tests {
             APP_NAME,
             VERSION,
             DEFAULT_LOG_LEVEL,
-            SERVICE_VERSION,
             ENVIRONMENT,
             RUSTFS_TLS_KEY,
             RUSTFS_TLS_CERT,
@@ -446,8 +437,7 @@ mod tests {
     fn test_configuration_consistency() {
         // Test configuration consistency
 
-        // Version consistency
-        assert_eq!(VERSION, SERVICE_VERSION, "Application version should match service version");
+        assert_eq!(VERSION, env!("CARGO_PKG_VERSION"));
 
         // Port conflict check
         let ports = [DEFAULT_PORT, DEFAULT_CONSOLE_PORT];

crates/config/src/constants/object.rs

@@ -387,7 +387,9 @@ pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 32 * 10
 /// Multipart pressure is often many moderate-sized parts rather than one very
 /// large request. The default gates every multipart part through the same permit
 /// pool as large/unknown-size PutObject while keeping small direct PUTs on the
-/// legacy path.
+/// legacy path. Server-side UploadPartCopy requests use this same multipart
+/// admission path with unknown-size weighting because source metadata and range
+/// validation happen after admission.
 pub const ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: &str =
     "RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES";
 pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 0;
@@ -415,7 +417,9 @@ pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 250;
 /// wait with a 10 s request deadline. The wait must leave margin under the
 /// shortest of those, not merely fall below an SDK default, so the part
 /// receives S3 `SlowDown`/503 for the client to retry instead of losing its
-/// connection (issue #7385). `0` rejects immediately when the pool is full.
+/// connection (issue #7385). Server-side UploadPartCopy waits in the same queue
+/// before taking bucket lifecycle locks or opening source readers. `0` rejects
+/// immediately when the pool is full.
 pub const ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: &str =
     "RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS";
 pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 10_000;
@@ -427,7 +431,7 @@ pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 10_0
 // the wait past any client timeout.
 const _: () = assert!(DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS * 3 <= 30_000);
 
-/// Maximum multipart UploadPart requests waiting for a foreground write permit per process.
+/// Maximum multipart UploadPart or UploadPartCopy requests waiting for a foreground write permit per process.
 ///
 /// Parts beyond this queue depth are rejected with S3 `SlowDown`/503 without
 /// waiting, so a genuinely saturated node still fails fast instead of holding

crates/config/src/constants/tls.rs

@@ -207,3 +207,9 @@ pub const ENV_TLS_RELOAD_INTERVAL: &str = "RUSTFS_TLS_RELOAD_INTERVAL";
 
 /// Default interval for TLS certificate reload check
 pub const DEFAULT_TLS_RELOAD_INTERVAL: u64 = 30;
+
+/// Environment variable to enable experimental HTTP/3 support.
+pub const ENV_HTTP3_ENABLE: &str = "RUSTFS_HTTP3_ENABLE";
+
+/// HTTP/3 is disabled unless explicitly enabled.
+pub const DEFAULT_HTTP3_ENABLE: bool = false;

crates/connect-symbol-catalog/Cargo.toml

@@ -8,7 +8,7 @@ publish = false
 
 [dependencies]
 hex-simd.workspace = true
-object = { version = "=0.39.1", default-features = false, features = ["read", "std", "elf"] }
+object = { workspace = true, features = ["read", "std", "elf"] }
 serde = { workspace = true, features = ["derive"] }
 serde_json.workspace = true
 sha2.workspace = true
@@ -16,7 +16,7 @@ sha2.workspace = true
 symbolic-demangle = { version = "=13.9.0", default-features = false, features = ["rust"] }
 
 [dev-dependencies]
-object = { version = "=0.39.1", default-features = false, features = ["write", "elf"] }
+object = { workspace = true, features = ["write", "elf"] }
 
 [lints]
 workspace = true

crates/e2e_test/src/chaos.rs

@@ -384,17 +384,17 @@ pub(crate) fn is_cluster_heal_coordination_unavailable(error: &(dyn std::error::
     message.contains("500 Internal Server Error") && message.contains("cluster heal coordination unavailable")
 }
 
-/// Wait for the restarted cluster to admit the first root heal request.
+/// Wait for the restarted cluster to admit a root heal request and return its response.
 pub(crate) async fn start_root_heal_when_control_ready(
     heal_url: &str,
     heal_body: &str,
     access_key: &str,
     secret_key: &str,
-) -> ChaosResult<()> {
+) -> ChaosResult<String> {
     let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(45);
     loop {
         match signed_admin_post(heal_url, Some(heal_body), access_key, secret_key).await {
-            Ok(_) => return Ok(()),
+            Ok(body) => return Ok(body),
             Err(error) if is_cluster_heal_coordination_unavailable(error.as_ref()) && tokio::time::Instant::now() < deadline => {
                 tokio::time::sleep(std::time::Duration::from_millis(250)).await;
             }

crates/e2e_test/src/cluster_multidrive_pool_test.rs

@@ -33,6 +33,10 @@
 //! tracked separately.
 
 use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
+use crate::fault_proxy::FaultMode;
+use aws_sdk_s3::Client;
+use std::time::Duration;
+use tokio::time::{Instant, timeout};
 
 type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
 
@@ -144,7 +148,99 @@ async fn cluster_volume_fault_proxy_pass_smoke() -> TestResult {
         cluster.start().await?;
         cluster.create_test_bucket(BUCKET).await?;
         let payload = vec![0x6Du8; 256 * 1024];
-        put_get_roundtrip(&cluster, "volume-proxy/object", &payload).await
+        put_get_roundtrip(&cluster, "volume-proxy/object", &payload).await?;
+
+        // Node 1 owns only two disks; a successful three-vote write must
+        // authenticate at least one disk mutation through node 0's proxy.
+        let client = cluster.create_s3_client(1)?;
+        let client = Client::from_conf(
+            client
+                .config()
+                .to_builder()
+                .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
+                .build(),
+        );
+        let forwarded_before = proxy.forwarded_bytes();
+        client
+            .put_object()
+            .bucket(BUCKET)
+            .key("volume-proxy/authenticated")
+            .body(bytes::Bytes::copy_from_slice(&payload).into())
+            .send()
+            .await?;
+        assert!(
+            proxy.forwarded_bytes() > forwarded_before,
+            "successful peer-disk publication must traverse the proxy"
+        );
+
+        let dropped_before = proxy.dropped_bytes();
+        proxy.set_mode(FaultMode::Blackhole);
+        assert_eq!(proxy.mode(), FaultMode::Blackhole);
+        let error = timeout(
+            Duration::from_secs(30),
+            client
+                .put_object()
+                .bucket(BUCKET)
+                .key("volume-proxy/rejected")
+                .body(bytes::Bytes::copy_from_slice(&payload).into())
+                .send(),
+        )
+        .await?
+        .expect_err("two local disks cannot commit a write while the peer is blackholed");
+        assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
+        assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("ServiceUnavailable"));
+        assert!(
+            proxy.dropped_bytes() > dropped_before,
+            "the injected blackhole must discard actual peer traffic"
+        );
+
+        proxy.set_mode(FaultMode::Pass);
+        assert_eq!(proxy.mode(), FaultMode::Pass);
+        let http = reqwest::Client::builder()
+            .no_proxy()
+            .timeout(Duration::from_secs(3))
+            .build()?;
+        let deadline = Instant::now() + Duration::from_secs(30);
+        loop {
+            let response = http.get(format!("{}/health/ready", cluster.nodes[1].url)).send().await?;
+            let status = response.status();
+            let body: serde_json::Value = response.json().await?;
+            // Reachable peers can still have Returning disk handles that reject
+            // writes. Wait for this node's live writer inventory to be Online.
+            if status.as_u16() == 200
+                && body["details"]["storage"]["source"] == "local_runtime"
+                && body["details"]["storage"]["writeQuorum"] == true
+                && body["details"]["storage"]["unavailableDrives"]
+                    .as_array()
+                    .is_some_and(Vec::is_empty)
+                && body["details"]["lock"]["ready"] == true
+            {
+                break;
+            }
+            assert!(Instant::now() < deadline, "peer write quorum did not recover: HTTP {status}, {body}");
+            tokio::time::sleep(Duration::from_millis(200)).await;
+        }
+        client
+            .put_object()
+            .bucket(BUCKET)
+            .key("volume-proxy/recovered")
+            .body(bytes::Bytes::copy_from_slice(&payload).into())
+            .send()
+            .await?;
+        for key in ["volume-proxy/object", "volume-proxy/authenticated", "volume-proxy/recovered"] {
+            let got = client.get_object().bucket(BUCKET).key(key).send().await?;
+            let body = got.body.collect().await?.into_bytes();
+            assert_eq!(body.as_ref(), payload, "proxy recovery must preserve the full object body for {key}");
+        }
+        let rejected = client
+            .head_object()
+            .bucket(BUCKET)
+            .key("volume-proxy/rejected")
+            .send()
+            .await
+            .expect_err("a rejected proxied write must never become visible after recovery");
+        assert_eq!(rejected.raw_response().map(|response| response.status().as_u16()), Some(404));
+        Ok(())
     }
     .await;
 

crates/e2e_test/src/common.rs

@@ -22,6 +22,8 @@
 //! - Common test constants and utilities
 
 use aws_sdk_s3::config::{Credentials, Region};
+use aws_sdk_s3::error::{ProvideErrorMetadata, SdkError};
+use aws_sdk_s3::operation::list_buckets::ListBucketsError;
 use aws_sdk_s3::{Client, Config};
 use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
 use http::header::{CONTENT_TYPE, HOST};
@@ -33,7 +35,7 @@ use s3s::Body;
 use serde_json;
 use std::fs as stdfs;
 use std::io::ErrorKind;
-use std::net::SocketAddr;
+use std::net::{Ipv6Addr, SocketAddr};
 use std::path::{Path, PathBuf};
 use std::process::{Child, Command, Stdio};
 use std::sync::Once;
@@ -61,6 +63,41 @@ const TEST_PORT_COUNTER_PATH: &str = "/tmp/rustfs_e2e_next_port";
 const TEST_PORT_LOCK_DIR: &str = "/tmp/rustfs_e2e_port_allocator.lock";
 const TEST_PORT_LOCK_STALE_AFTER: Duration = Duration::from_secs(30);
 
+fn list_buckets_readiness_error(err: &SdkError<ListBucketsError>) -> String {
+    // SDK Display reports only the category. Do not expose raw response bodies or headers.
+    let service_code = err.as_service_error().and_then(|error| error.code()).filter(|code| {
+        matches!(
+            *code,
+            "ServiceUnavailable"
+                | "ServerNotInitialized"
+                | "InternalError"
+                | "AccessDenied"
+                | "InvalidAccessKeyId"
+                | "SignatureDoesNotMatch"
+        )
+    });
+    let transport = match err {
+        SdkError::DispatchFailure(failure) if failure.is_timeout() => Some("timeout"),
+        SdkError::DispatchFailure(failure) if failure.is_io() => Some("io"),
+        SdkError::DispatchFailure(failure) if failure.is_user() => Some("user"),
+        SdkError::DispatchFailure(failure) if failure.is_other() => Some("other"),
+        _ => None,
+    };
+    let mut source: Option<&(dyn std::error::Error + 'static)> = Some(err);
+    let mut io_error = None;
+    while let Some(error) = source {
+        if let Some(error) = error.downcast_ref::<std::io::Error>() {
+            io_error = Some((error.kind(), error.raw_os_error()));
+            break;
+        }
+        source = error.source();
+    }
+    format!(
+        "{err}; http_status={:?}; service_code={service_code:?}; transport={transport:?}; io_error={io_error:?}",
+        err.raw_response().map(|response| response.status().as_u16())
+    )
+}
+
 fn capture_log_path(log_dir: &Path, temp_dir: &str) -> Option<PathBuf> {
     let temp_name = Path::new(temp_dir).file_name()?.to_string_lossy();
     Some(log_dir.join(format!("{temp_name}.log")))
@@ -1405,7 +1442,11 @@ impl RustFSTestClusterEnvironment {
             return Err(format!("a volume proxy is already configured for node {node_idx}").into());
         }
         let target = self.nodes[node_idx].address.parse::<SocketAddr>()?;
-        let proxy = crate::fault_proxy::FaultProxy::start(target).await?;
+        // Endpoint locality and RPC audience both use the advertised authority.
+        // Keep the node's port while the second loopback IP separates the proxy
+        // listener from the direct IPv4 server and S3 client address.
+        let listen = SocketAddr::from((Ipv6Addr::LOCALHOST, target.port()));
+        let proxy = crate::fault_proxy::FaultProxy::start_on(listen, target).await?;
         self.volume_proxy_addresses[node_idx] = Some(proxy.local_addr());
         Ok(proxy)
     }
@@ -1575,8 +1616,9 @@ impl RustFSTestClusterEnvironment {
     ///
     /// Verifies service availability by calling the S3 `list_buckets` API against the requested node,
     /// retries up to 120 times with a 1-second interval between attempts.
-    async fn wait_for_node_service_ready(&self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    async fn wait_for_node_service_ready(&mut self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         let client = self.create_s3_client(node_idx)?;
+        let mut last_error = None;
 
         for attempt in 0..120 {
             match client.list_buckets().send().await {
@@ -1584,13 +1626,28 @@ impl RustFSTestClusterEnvironment {
                     info!("Cluster node {} service ready after {} attempts", node_idx, attempt + 1);
                     return Ok(());
                 }
-                Err(_) => {
+                Err(err) => {
+                    last_error = Some(err);
                     sleep(Duration::from_secs(1)).await;
                 }
             }
         }
 
-        Err(format!("Cluster node {} service failed to become ready", node_idx).into())
+        let last_error = last_error.as_ref().map(list_buckets_readiness_error);
+        // Observe liveness only after the existing readiness budget is exhausted, before cleanup.
+        let process_status = match self.nodes[node_idx].process.as_mut() {
+            Some(process) => match process.try_wait() {
+                Ok(Some(status)) => format!("exited ({status})"),
+                Ok(None) => "running".to_string(),
+                Err(error) => format!("unavailable (kind={:?}, os_code={:?})", error.kind(), error.raw_os_error()),
+            },
+            None => "not started".to_string(),
+        };
+        Err(format!(
+            "Cluster node {node_idx} service failed to become ready; last ListBuckets error={last_error:?}; process_status={process_status}; capture_log_path={:?}",
+            self.node_capture_log_paths[node_idx]
+        )
+        .into())
     }
 
     /// Create an S3 client configured to communicate with a specific cluster node.
@@ -2101,6 +2158,34 @@ mod tests {
         verify_awscurl_path(&executable).expect("an available client with a working help command should pass");
     }
 
+    #[test]
+    fn readiness_error_reports_transport_without_sensitive_source_text() {
+        use aws_sdk_s3::error::ConnectorError;
+
+        let sensitive = "request credentials must not appear in diagnostics";
+        for (connector, category) in [
+            (
+                ConnectorError::io(std::io::Error::new(ErrorKind::ConnectionReset, sensitive).into()),
+                "io",
+            ),
+            (ConnectorError::timeout(sensitive.into()), "timeout"),
+            (ConnectorError::user(sensitive.into()), "user"),
+            (ConnectorError::other(sensitive.into(), None), "other"),
+        ] {
+            let diagnostic = list_buckets_readiness_error(&SdkError::dispatch_failure(connector));
+            assert!(diagnostic.contains(&format!("transport=Some(\"{category}\")")), "{diagnostic}");
+            assert!(diagnostic.contains("http_status=None"), "{diagnostic}");
+            assert!(!diagnostic.contains(sensitive), "{diagnostic}");
+            if category == "io" {
+                assert!(diagnostic.contains("io_error=Some((ConnectionReset, None))"), "{diagnostic}");
+            }
+        }
+        let os_error = std::io::Error::from_raw_os_error(13);
+        let expected = format!("io_error=Some(({:?}, Some(13)))", os_error.kind());
+        let diagnostic = list_buckets_readiness_error(&SdkError::dispatch_failure(ConnectorError::io(os_error.into())));
+        assert!(diagnostic.contains(&expected), "{diagnostic}");
+    }
+
     #[test]
     fn capture_log_path_uses_temp_directory_basename() {
         assert_eq!(
@@ -2386,6 +2471,20 @@ mod tests {
 
         assert!(volumes.contains(&proxied), "volumes must use the proxy address: {volumes}");
         assert!(!volumes.contains(&direct), "volumes must not retain the direct address: {volumes}");
+        assert_eq!(proxy.local_addr().port(), proxy.target_addr().port());
+        assert_ne!(proxy.local_addr().ip(), proxy.target_addr().ip());
+        assert_eq!(env.nodes[0].address, direct, "S3 clients must retain the direct address");
+        let mut endpoint =
+            crate::storage_api::Endpoint::try_from(volumes.as_str()).expect("proxied volume should be a valid endpoint");
+        endpoint
+            .update_is_local(proxy.target_addr().port())
+            .expect("proxied endpoint locality should resolve");
+        assert!(endpoint.is_local, "the proxied node must keep its local disk and first-disk authority");
+        assert_eq!(
+            endpoint.host_port(),
+            proxied,
+            "the local RPC audience must be the advertised proxy authority"
+        );
 
         proxy.shutdown().await;
     }

crates/e2e_test/src/delete_authorization_test.rs

@@ -282,7 +282,7 @@ async fn sdk_list_bucket_and_list_bucket_versions_permissions_are_independent()
 }
 
 #[tokio::test]
-async fn console_admin_force_delete_removes_prefix_versions_and_delete_markers() -> TestResult {
+async fn explicit_force_delete_permission_removes_prefix_versions_and_delete_markers() -> TestResult {
     init_logging();
     let mut env = RustFSTestEnvironment::new().await?;
     env.start_rustfs_server(vec![]).await?;
@@ -300,7 +300,14 @@ async fn console_admin_force_delete_removes_prefix_versions_and_delete_markers()
     put(&root, bucket, "folder-sibling/keep.txt").await?;
     let keep = versions(&root, bucket, "keep.txt").await?;
     let sibling = versions(&root, bucket, "folder-sibling/").await?;
-    let user = policy_user(&env, "consoleAdmin", None).await?;
+    let user = policy_user(
+        &env,
+        "explicit-force-delete",
+        Some(json!({"Version":"2012-10-17","Statement":[
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")}
+        ]})),
+    )
+    .await?;
 
     force_delete(&user, bucket, "folder/").await?;
     assert!(
@@ -322,6 +329,126 @@ async fn console_admin_force_delete_removes_prefix_versions_and_delete_markers()
     Ok(())
 }
 
+#[tokio::test]
+async fn force_delete_header_requires_explicit_permission() -> TestResult {
+    init_logging();
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+    let root = env.create_s3_client();
+    let admin = policy_user(&env, "consoleAdmin", None).await?;
+
+    let owner_bucket = "force-owner-bucket";
+    root.create_bucket().bucket(owner_bucket).send().await?;
+    put(&root, owner_bucket, "kept-until-force.txt").await?;
+    force_delete_bucket(&root, owner_bucket, "x-rustfs-force-delete").await?;
+    assert!(
+        head_bucket_missing(&root, owner_bucket).await?,
+        "root force-delete must remove a non-empty bucket"
+    );
+
+    let denied_bucket = "force-admin-denied";
+    root.create_bucket().bucket(denied_bucket).send().await?;
+    put(&root, denied_bucket, "folder/child.txt").await?;
+    put(&root, denied_bucket, "plain.txt").await?;
+    assert_boxed_denied(force_delete(&admin, denied_bucket, "folder/").await);
+    assert!(
+        !versions(&root, denied_bucket, "folder/").await?.is_empty(),
+        "consoleAdmin without s3:ForceDeleteObject must not recursively delete"
+    );
+    assert_boxed_denied(force_delete_bucket(&admin, denied_bucket, "x-rustfs-force-delete").await);
+    assert_boxed_denied(force_delete_bucket(&admin, denied_bucket, "x-minio-force-delete").await);
+    assert!(
+        !head_bucket_missing(&root, denied_bucket).await?,
+        "consoleAdmin without s3:ForceDeleteBucket must not force-delete a non-empty bucket"
+    );
+    let not_empty = admin.delete_bucket().bucket(denied_bucket).send().await;
+    let not_empty = not_empty.expect_err("plain delete of a non-empty bucket must fail");
+    assert_eq!(
+        not_empty.as_service_error().and_then(|error| error.code()),
+        Some("BucketNotEmpty"),
+        "plain delete must keep the emptiness check, got {not_empty:?}"
+    );
+    admin.delete_object().bucket(denied_bucket).key("plain.txt").send().await?;
+    assert!(
+        versions(&root, denied_bucket, "plain.txt").await?.is_empty(),
+        "plain DeleteObject without the force header must stay allowed for consoleAdmin"
+    );
+
+    let object_bucket = "force-explicit-object";
+    root.create_bucket().bucket(object_bucket).send().await?;
+    put(&root, object_bucket, "folder/child.txt").await?;
+    put(&root, object_bucket, "keep.txt").await?;
+    let keep = versions(&root, object_bucket, "keep.txt").await?;
+    let object_user = policy_user(
+        &env,
+        "explicit-object-force",
+        Some(json!({"Version":"2012-10-17","Statement":[
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{object_bucket}/*")}
+        ]})),
+    )
+    .await?;
+    force_delete(&object_user, object_bucket, "folder/").await?;
+    assert!(versions(&root, object_bucket, "folder/").await?.is_empty());
+    assert_eq!(versions(&root, object_bucket, "keep.txt").await?, keep);
+
+    let allowed_bucket = "force-explicit-bucket";
+    root.create_bucket().bucket(allowed_bucket).send().await?;
+    put(&root, allowed_bucket, "inside.txt").await?;
+    let bucket_user = policy_user(
+        &env,
+        "explicit-bucket-force",
+        Some(json!({"Version":"2012-10-17","Statement":[
+            {"Effect":"Allow","Action":["s3:DeleteBucket","s3:ForceDeleteBucket"],"Resource":[
+                format!("arn:aws:s3:::{allowed_bucket}"), format!("arn:aws:s3:::{allowed_bucket}/*")
+            ]}
+        ]})),
+    )
+    .await?;
+    force_delete_bucket(&bucket_user, allowed_bucket, "x-minio-force-delete").await?;
+    assert!(
+        head_bucket_missing(&root, allowed_bucket).await?,
+        "an explicit s3:ForceDeleteBucket grant must remove a non-empty bucket"
+    );
+    Ok(())
+}
+
+async fn force_delete_bucket(
+    client: &Client,
+    bucket: &str,
+    header: &str,
+) -> Result<(), Box<SdkError<aws_sdk_s3::operation::delete_bucket::DeleteBucketError>>> {
+    let header: &'static str = match header {
+        "x-rustfs-force-delete" => "x-rustfs-force-delete",
+        "x-minio-force-delete" => "x-minio-force-delete",
+        other => panic!("unexpected force-delete header {other}"),
+    };
+    client
+        .delete_bucket()
+        .bucket(bucket)
+        .customize()
+        .mutate_request(move |request| {
+            request.headers_mut().insert(header, "true");
+        })
+        .send()
+        .await
+        .map(|_| ())
+        .map_err(Box::new)
+}
+
+async fn head_bucket_missing(client: &Client, bucket: &str) -> TestResult<bool> {
+    match client.head_bucket().bucket(bucket).send().await {
+        Ok(_) => Ok(false),
+        Err(error) => {
+            let code = error.as_service_error().and_then(|service| service.code());
+            if matches!(code, Some("NotFound" | "NoSuchBucket")) {
+                Ok(true)
+            } else {
+                Err(error.into())
+            }
+        }
+    }
+}
+
 #[tokio::test]
 async fn force_delete_authorizes_only_its_path_scope_without_list_permissions() -> TestResult {
     init_logging();
@@ -339,7 +466,7 @@ async fn force_delete_authorizes_only_its_path_scope_without_list_permissions()
         &env,
         "delete-only",
         Some(json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":[
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":[
                 format!("arn:aws:s3:::{bucket}/selected.txt"), format!("arn:aws:s3:::{bucket}/selected.txt/*")
             ]},
             {"Effect":"Deny","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/selected.txt-sibling")}
@@ -376,7 +503,7 @@ async fn force_directory_delete_cannot_remove_an_unauthorized_colliding_parent()
         &env,
         "parent-denier",
         Some(json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
             {"Effect":"Deny","Action":"s3:DeleteObjectVersion","Resource":format!("arn:aws:s3:::{bucket}/collision.txt"),
              "Condition":{"StringEquals":{"s3:VersionId":protected_parent_version}}}
         ]})),
@@ -409,7 +536,7 @@ async fn force_unversioned_directory_requires_only_delete_object() -> TestResult
         &env,
         "unversioned-deleter",
         Some(json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":"s3:DeleteObject","Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
             {"Effect":"Deny","Action":"s3:DeleteObjectVersion","Resource":format!("arn:aws:s3:::{bucket}/*")}
         ]})),
     )
@@ -438,7 +565,7 @@ async fn force_delete_denied_child_preserves_every_object_despite_bucket_allow()
         &env,
         "child-denier",
         Some(json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ReplicateDelete"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject","s3:ReplicateDelete"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
             {"Effect":"Deny","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ReplicateDelete"],"Resource":format!("arn:aws:s3:::{bucket}/folder/z-denied.txt")}
         ]})),
     )
@@ -468,7 +595,7 @@ async fn force_delete_denied_child_preserves_every_object_despite_bucket_allow()
         &env,
         "replica-deleter",
         Some(json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":"s3:DeleteObject","Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
             {"Effect":"Allow","Action":"s3:ReplicateDelete","Resource":format!("arn:aws:s3:::{bucket}/*")}
         ]})),
     )
@@ -500,7 +627,7 @@ async fn force_delete_denied_historical_version_preserves_versions_and_markers()
     put(&root, bucket, "folder/a-allowed.txt").await?;
     let policy = |version: &str| {
         json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
             {"Effect":"Deny","Action":"s3:DeleteObjectVersion","Resource":format!("arn:aws:s3:::{bucket}/folder/*"),
              "Condition":{"StringEquals":{"s3:VersionId":version}}}
         ]})
@@ -628,7 +755,7 @@ async fn force_delete_checks_every_version_page_before_mutation() -> TestResult
         .try_collect::<Vec<_>>()
         .await?;
     put(&root, bucket, "folder/z-denied.txt").await?;
-    let allow = json!({"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/*")});
+    let allow = json!({"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")});
     let user = policy_user(
         &env,
         "paged-deleter",

crates/e2e_test/src/distributed/harness.rs

@@ -34,6 +34,9 @@ use crate::common::{
 };
 use crate::replication_extension_test::LOOPBACK_REPLICATION_TARGET_ENV;
 use aws_sdk_s3::Client;
+use aws_sdk_s3::config::retry::RetryConfig;
+use aws_sdk_s3::error::{ProvideErrorMetadata, SdkError};
+use aws_sdk_s3::operation::{get_object::GetObjectError, put_object::PutObjectError};
 use aws_sdk_s3::primitives::ByteStream;
 use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
 use http::{Method, StatusCode};
@@ -41,7 +44,7 @@ use sha2::{Digest, Sha256};
 use std::collections::BTreeMap;
 use std::path::{Path, PathBuf};
 use std::time::Duration;
-use tokio::time::{Instant, sleep};
+use tokio::time::{Instant, sleep, timeout_at};
 use uuid::Uuid;
 
 pub(crate) type TestResult<T = ()> = Result<T, Box<dyn std::error::Error + Send + Sync>>;
@@ -400,9 +403,9 @@ pub(crate) async fn put_inventory(
     Ok(inventory)
 }
 
-/// Retry only transport-level service availability failures while a data
-/// movement operation changes the pool map. Generic InternalError responses
-/// remain fatal because accepting them would hide server defects.
+/// Retry only explicit S3 availability responses while a data movement
+/// operation changes the pool map. Transport failures and InternalError
+/// responses remain fatal because callers already require a ready endpoint.
 pub(crate) async fn put_inventory_retrying(
     client: &Client,
     bucket: &str,
@@ -449,19 +452,48 @@ where
     let deadline = Instant::now() + timeout;
     let mut delay = Duration::from_millis(50);
     loop {
-        let last_error = match probe().await {
-            Ok(true) => return Ok(()),
-            Ok(false) => format!("{label} still false"),
-            Err(error) => error.to_string(),
-        };
         if Instant::now() >= deadline {
-            return Err(format!("{label} did not become true within {timeout:?}: {last_error}").into());
+            return Err(format!("{label} did not become true within {timeout:?}").into());
         }
-        sleep(delay).await;
+        // A pending request must consume the same budget as unsuccessful probes.
+        match timeout_at(deadline, probe()).await {
+            Err(_) => return Err(format!("{label} did not become true within {timeout:?}").into()),
+            Ok(Err(error)) => return Err(error),
+            Ok(Ok(true)) => return Ok(()),
+            Ok(Ok(false)) => {}
+        }
+        tokio::time::sleep_until((Instant::now() + delay).min(deadline)).await;
         delay = (delay * 2).min(Duration::from_secs(1));
     }
 }
 
+// Boxed SDK errors retain the operation's service code; Display only says
+// "service error" and cannot distinguish convergence from a server defect.
+pub(crate) fn s3_probe_error_code<'a>(error: &'a (dyn std::error::Error + Send + Sync + 'static)) -> Option<&'a str> {
+    error
+        .downcast_ref::<SdkError<GetObjectError>>()
+        .and_then(|error| error.as_service_error())
+        .and_then(ProvideErrorMetadata::code)
+        .or_else(|| {
+            error
+                .downcast_ref::<SdkError<PutObjectError>>()
+                .and_then(|error| error.as_service_error())
+                .and_then(ProvideErrorMetadata::code)
+        })
+}
+
+pub(crate) fn s3_probe_client(client: &Client) -> Client {
+    // The probe owns the retry policy. SDK retries must not hide a response
+    // that the probe would classify as fatal.
+    Client::from_conf(
+        client
+            .config()
+            .to_builder()
+            .retry_config(RetryConfig::standard().with_max_attempts(1))
+            .build(),
+    )
+}
+
 pub(crate) async fn cluster_admin(
     cluster: &RustFSTestClusterEnvironment,
     method: Method,
@@ -598,15 +630,20 @@ pub(crate) async fn wait_for_replicated_bytes(
     expected: &[u8],
     timeout: Duration,
 ) -> TestResult {
+    let client = s3_probe_client(client);
     wait_until(
         timeout,
         || async {
-            match get_object_bytes(client, bucket, key).await {
+            match get_object_bytes(&client, bucket, key).await {
                 Ok(got) if got.as_slice() == expected => Ok(true),
-                Ok(_) => Ok(false),
+                Ok(got) => Err(format!(
+                    "replicated object {bucket}/{key} bytes mismatch: expected sha256={} got sha256={}",
+                    sha256_hex(expected),
+                    sha256_hex(&got)
+                )
+                .into()),
                 Err(error) => {
-                    let message = error.to_string();
-                    if message.contains("NoSuchKey") || message.contains("NotFound") {
+                    if matches!(s3_probe_error_code(error.as_ref()), Some("NoSuchKey" | "NotFound")) {
                         Ok(false)
                     } else {
                         Err(error)
@@ -988,6 +1025,7 @@ pub(crate) async fn list_pools_json(cluster: &RustFSTestClusterEnvironment) -> T
 }
 
 pub(crate) async fn retrying_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, timeout: Duration) -> TestResult {
+    let client = s3_probe_client(client);
     wait_until(
         timeout,
         || {
@@ -999,8 +1037,7 @@ pub(crate) async fn retrying_put(client: &Client, bucket: &str, key: &str, body:
                 match put_object(&client, &bucket, &key, body).await {
                     Ok(()) => Ok(true),
                     Err(error) => {
-                        let message = error.to_string();
-                        if message.contains("SlowDown") || message.contains("ServiceUnavailable") || message.contains("503") {
+                        if matches!(s3_probe_error_code(error.as_ref()), Some("SlowDown" | "ServiceUnavailable")) {
                             Ok(false)
                         } else {
                             Err(error)
@@ -1021,19 +1058,20 @@ pub(crate) async fn retrying_get_equals(
     expected: &[u8],
     timeout: Duration,
 ) -> TestResult {
+    let client = s3_probe_client(client);
     wait_until(
         timeout,
         || async {
-            match get_object_bytes(client, bucket, key).await {
+            match get_object_bytes(&client, bucket, key).await {
                 Ok(got) if got.as_slice() == expected => Ok(true),
-                Ok(_) => Ok(false),
+                Ok(got) => Err(format!(
+                    "object {bucket}/{key} bytes mismatch: expected sha256={} got sha256={}",
+                    sha256_hex(expected),
+                    sha256_hex(&got)
+                )
+                .into()),
                 Err(error) => {
-                    let message = error.to_string();
-                    if message.contains("NoSuchKey")
-                        || message.contains("SlowDown")
-                        || message.contains("ServiceUnavailable")
-                        || message.contains("503")
-                    {
+                    if matches!(s3_probe_error_code(error.as_ref()), Some("SlowDown" | "ServiceUnavailable")) {
                         Ok(false)
                     } else {
                         Err(error)
@@ -1046,6 +1084,162 @@ pub(crate) async fn retrying_get_equals(
     .await
 }
 
+#[cfg(test)]
+mod retry_tests {
+    use super::*;
+    use crate::fake_s3_target::{FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction, Operation, SeedMetadata};
+    use std::cell::Cell;
+
+    fn client(target: &FakeS3Target) -> Client {
+        Client::from_conf(build_test_s3_config(
+            target.endpoint(),
+            FAKE_ACCESS_KEY,
+            FAKE_SECRET_KEY,
+            None,
+            "distributed-retry-test",
+        ))
+    }
+
+    #[tokio::test]
+    async fn wait_until_preserves_first_error() {
+        let attempts = Cell::new(0);
+        let error = wait_until(
+            Duration::from_secs(5),
+            || {
+                attempts.set(attempts.get() + 1);
+                std::future::ready(if attempts.get() == 1 {
+                    Err(std::io::Error::new(std::io::ErrorKind::PermissionDenied, "permanent probe failure").into())
+                } else {
+                    Ok(true)
+                })
+            },
+            "permanent failure",
+        )
+        .await
+        .expect_err("a later success must not hide a permanent failure");
+        assert_eq!(attempts.get(), 1);
+        assert_eq!(
+            error
+                .downcast_ref::<std::io::Error>()
+                .expect("preserve the original error")
+                .kind(),
+            std::io::ErrorKind::PermissionDenied
+        );
+    }
+
+    #[tokio::test]
+    async fn wait_until_retries_explicit_pending_state() -> TestResult {
+        let attempts = Cell::new(0);
+        wait_until(
+            Duration::from_secs(5),
+            || {
+                attempts.set(attempts.get() + 1);
+                std::future::ready(Ok(attempts.get() == 2))
+            },
+            "eventual readiness",
+        )
+        .await?;
+        assert_eq!(attempts.get(), 2);
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn wait_until_bounds_a_pending_probe() {
+        let error = tokio::time::timeout(
+            Duration::from_secs(5),
+            wait_until(Duration::from_millis(10), std::future::pending, "hung probe"),
+        )
+        .await
+        .expect("the probe deadline must finish before the test watchdog")
+        .expect_err("a permanently pending probe must time out");
+        assert!(error.to_string().contains("hung probe did not become true"));
+    }
+
+    #[tokio::test]
+    async fn wait_until_does_not_start_a_probe_after_its_deadline() {
+        let attempts = Cell::new(0);
+        wait_until(
+            Duration::ZERO,
+            || {
+                attempts.set(attempts.get() + 1);
+                std::future::ready(Ok(true))
+            },
+            "expired budget",
+        )
+        .await
+        .expect_err("an expired budget must not admit another probe");
+        assert_eq!(attempts.get(), 0);
+    }
+
+    #[tokio::test]
+    async fn retrying_put_does_not_hide_internal_error() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket("retry-probe");
+        target.inject(Operation::PutObject, FaultAction::ResponseStatus(500), 1);
+        let error = retrying_put(&client(&target), "retry-probe", "key", b"body".to_vec(), Duration::from_secs(5))
+            .await
+            .expect_err("InternalError must remain fatal even if the next PUT would succeed");
+        assert_eq!(s3_probe_error_code(error.as_ref()), Some("InternalError"));
+        assert_eq!(target.requests().len(), 1);
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn retrying_put_allows_transient_availability_errors() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket("retry-probe");
+        for status in [429, 503] {
+            target.inject(Operation::PutObject, FaultAction::ResponseStatus(status), 1);
+            retrying_put(&client(&target), "retry-probe", "key", b"body".to_vec(), Duration::from_secs(5)).await?;
+            assert_eq!(target.take_requests().len(), 2);
+        }
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn object_polls_reject_wrong_object_bytes() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket("retry-probe");
+        target.put_seed_object("retry-probe", "key", "wrong bytes", &SeedMetadata::default());
+        let error = retrying_get_equals(&client(&target), "retry-probe", "key", b"expected bytes", Duration::from_secs(5))
+            .await
+            .expect_err("a successful response with wrong bytes must fail immediately");
+        assert!(error.to_string().contains("bytes mismatch"));
+        assert_eq!(target.take_requests().len(), 1);
+        let error = wait_for_replicated_bytes(&client(&target), "retry-probe", "key", b"expected bytes", Duration::from_secs(5))
+            .await
+            .expect_err("replication may lag, but it must not return corrupt bytes");
+        assert!(error.to_string().contains("bytes mismatch"));
+        assert_eq!(target.requests().len(), 1);
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn object_polls_distinguish_committed_and_replicated_keys() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket("retry-probe");
+        target.put_seed_object("retry-probe", "key", "expected bytes", &SeedMetadata::default());
+        let client = client(&target);
+        for (status, expected_code) in [(404, "NoSuchKey"), (500, "InternalError"), (403, "AccessDenied")] {
+            target.inject(Operation::GetObject, FaultAction::ResponseStatus(status), 1);
+            let error = retrying_get_equals(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5))
+                .await
+                .expect_err("an acknowledged object must not disappear or fail before a later successful read");
+            assert_eq!(s3_probe_error_code(error.as_ref()), Some(expected_code));
+            assert_eq!(target.take_requests().len(), 1);
+        }
+        for status in [429, 503] {
+            target.inject(Operation::GetObject, FaultAction::ResponseStatus(status), 1);
+            retrying_get_equals(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5)).await?;
+            assert_eq!(target.take_requests().len(), 2);
+        }
+        target.inject(Operation::GetObject, FaultAction::ResponseStatus(404), 1);
+        wait_for_replicated_bytes(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5)).await?;
+        assert_eq!(target.requests().len(), 2);
+        Ok(())
+    }
+}
+
 #[tokio::test]
 async fn append_single_node_pool_extends_ellipses_volumes() {
     let mut env = RustFSTestClusterEnvironment::with_topology(ClusterTopology::per_node_pools(2, vec![vec![0], vec![1]]))

crates/e2e_test/src/distributed/s3_basic_test.rs

@@ -20,6 +20,60 @@ use aws_sdk_s3::primitives::ByteStream;
 use aws_sdk_s3::types::{Delete, MetadataDirective, ObjectIdentifier};
 use std::time::Duration;
 
+#[tokio::test]
+async fn four_node_bucket_tags_opa_refreshes_warm_peer_caches() -> TestResult {
+    use super::harness::cluster_admin_ok;
+    use crate::sts_query_compat_test::{OPA_AUTH_TOKEN, OpaMock, set_department};
+
+    init_logging();
+    let mut opa = OpaMock::start().await?;
+    let dist = DistCluster::start_with_env(
+        DistLayout::FourNodeFourDisk,
+        &[
+            ("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str()),
+            ("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", OPA_AUTH_TOKEN),
+        ],
+    )
+    .await?;
+    let secret = uuid::Uuid::new_v4().to_string();
+    cluster_admin_ok(
+        &dist.cluster,
+        http::Method::PUT,
+        "/rustfs/admin/v3/add-user?accessKey=opabuckettags",
+        Some(serde_json::json!({"secretKey": secret, "status": "enabled"}).to_string()),
+    )
+    .await?;
+    let bucket = unique_bucket("opa-tags");
+    dist.create_bucket(&bucket).await?;
+    let admin = dist.client(0)?;
+    put_object(&admin, &bucket, "report", b"report".to_vec()).await?;
+    let clients = (0..4)
+        .map(|node| dist.client_with_credentials(node, "opabuckettags", &secret))
+        .collect::<Result<Vec<_>, _>>()?;
+
+    // Warm every peer, then verify the acknowledged updates/removal without
+    // polling or restarting nodes. This exercises healthy-peer metadata reload.
+    for department in [Some("finance"), Some("engineering"), None, Some("finance")] {
+        match department {
+            Some(value) => set_department(&admin, &bucket, value).await?,
+            None => {
+                admin.delete_bucket_tagging().bucket(&bucket).send().await?;
+            }
+        }
+        for client in &clients {
+            let result = client.get_object().bucket(&bucket).key("report").send().await;
+            if department == Some("finance") {
+                assert_eq!(result?.body.collect().await?.into_bytes().as_ref(), b"report");
+            } else {
+                let error = result.expect_err("updated or removed bucket tags must revoke this policy's read access");
+                assert_eq!(error.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
+            }
+            opa.expect_bucket_tags("s3:GetObject", &bucket, department).await?;
+        }
+    }
+    Ok(())
+}
+
 #[tokio::test]
 async fn four_node_four_drive_s3_put_get_head_list_copy_rename_delete_and_presign() -> TestResult {
     init_logging();

crates/e2e_test/src/distributed/upgrade_test.rs

@@ -26,11 +26,12 @@
 
 use super::harness::{
     DistCluster, DistLayout, TestResult, assert_object_bytes, cluster_admin_ok, enable_versioning, get_object_bytes, put_object,
-    unique_bucket, wait_until,
+    s3_probe_client, s3_probe_error_code, sha256_hex, unique_bucket, wait_until,
 };
 use crate::common::{
     AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, init_logging,
 };
+use crate::upgrade_compatibility_test::wait_for_upgrade_write_readiness;
 use aws_sdk_s3::Client;
 use aws_sdk_s3::error::ProvideErrorMetadata;
 use std::path::{Path, PathBuf};
@@ -111,6 +112,7 @@ async fn create_iam_user(dist: &DistCluster, user: &str, secret: &str, policy_na
 }
 
 async fn wait_for_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, label: &str) -> TestResult {
+    let client = s3_probe_client(client);
     wait_until(
         CREDENTIAL_TIMEOUT,
         || {
@@ -119,8 +121,18 @@ async fn wait_for_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, l
             let key = key.to_string();
             let body = body.clone();
             async move {
-                put_object(&client, &bucket, &key, body).await?;
-                Ok(true)
+                match put_object(&client, &bucket, &key, body).await {
+                    Ok(()) => Ok(true),
+                    Err(error)
+                        if matches!(
+                            s3_probe_error_code(error.as_ref()),
+                            Some("AccessDenied" | "InvalidAccessKeyId" | "SlowDown" | "ServiceUnavailable")
+                        ) =>
+                    {
+                        Ok(false)
+                    }
+                    Err(error) => Err(error),
+                }
             }
         },
         label,
@@ -129,6 +141,7 @@ async fn wait_for_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, l
 }
 
 async fn wait_for_bytes(client: &Client, bucket: &str, key: &str, expected: &[u8], label: &str) -> TestResult {
+    let client = s3_probe_client(client);
     wait_until(
         CREDENTIAL_TIMEOUT,
         || {
@@ -137,8 +150,24 @@ async fn wait_for_bytes(client: &Client, bucket: &str, key: &str, expected: &[u8
             let key = key.to_string();
             let expected = expected.to_vec();
             async move {
-                let got = get_object_bytes(&client, &bucket, &key).await?;
-                Ok(got == expected)
+                match get_object_bytes(&client, &bucket, &key).await {
+                    Ok(got) if got == expected => Ok(true),
+                    Ok(got) => Err(format!(
+                        "{label}: object {bucket}/{key} bytes mismatch: expected sha256={} got sha256={}",
+                        sha256_hex(&expected),
+                        sha256_hex(&got)
+                    )
+                    .into()),
+                    Err(error)
+                        if matches!(
+                            s3_probe_error_code(error.as_ref()),
+                            Some("AccessDenied" | "InvalidAccessKeyId" | "SlowDown" | "ServiceUnavailable")
+                        ) =>
+                    {
+                        Ok(false)
+                    }
+                    Err(error) => Err(error),
+                }
             }
         },
         label,
@@ -146,6 +175,57 @@ async fn wait_for_bytes(client: &Client, bucket: &str, key: &str, expected: &[u8
     .await
 }
 
+#[cfg(test)]
+mod retry_tests {
+    use super::*;
+    use crate::common::build_test_s3_config;
+    use crate::fake_s3_target::{FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction, Operation};
+
+    #[tokio::test]
+    async fn iam_polls_only_retry_auth_propagation_and_availability() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket("upgrade-retry-probe");
+        let client = Client::from_conf(build_test_s3_config(
+            target.endpoint(),
+            FAKE_ACCESS_KEY,
+            FAKE_SECRET_KEY,
+            None,
+            "upgrade-retry-test",
+        ));
+
+        for status in [403, 503] {
+            target.inject(Operation::PutObject, FaultAction::ResponseStatus(status), 1);
+            wait_for_put(&client, "upgrade-retry-probe", "key", b"body".to_vec(), "IAM PUT").await?;
+            assert_eq!(target.take_requests().len(), 2);
+
+            target.inject(Operation::GetObject, FaultAction::ResponseStatus(status), 1);
+            wait_for_bytes(&client, "upgrade-retry-probe", "key", b"body", "IAM GET").await?;
+            assert_eq!(target.take_requests().len(), 2);
+        }
+
+        target.inject(Operation::PutObject, FaultAction::ResponseStatus(500), 1);
+        let error = wait_for_put(&client, "upgrade-retry-probe", "key", b"body".to_vec(), "IAM PUT")
+            .await
+            .expect_err("IAM convergence must not hide InternalError on PUT");
+        assert_eq!(s3_probe_error_code(error.as_ref()), Some("InternalError"));
+        assert_eq!(target.take_requests().len(), 1);
+
+        target.inject(Operation::GetObject, FaultAction::ResponseStatus(500), 1);
+        let error = wait_for_bytes(&client, "upgrade-retry-probe", "key", b"body", "IAM GET")
+            .await
+            .expect_err("IAM convergence must not hide InternalError on GET");
+        assert_eq!(s3_probe_error_code(error.as_ref()), Some("InternalError"));
+        assert_eq!(target.take_requests().len(), 1);
+
+        let error = wait_for_bytes(&client, "upgrade-retry-probe", "key", b"wrong bytes", "IAM GET")
+            .await
+            .expect_err("IAM convergence must not hide a payload mismatch");
+        assert!(error.to_string().contains("bytes mismatch"));
+        assert_eq!(target.requests().len(), 1);
+        Ok(())
+    }
+}
+
 async fn seed_history_and_iam(dist: &DistCluster) -> TestResult<UpgradeSeed> {
     let history_bucket = unique_bucket("upg-hist");
     let versioned_bucket = unique_bucket("upg-ver");
@@ -321,6 +401,19 @@ async fn four_node_direct_upgrade_preserves_history_and_iam_credentials() -> Tes
     Ok(())
 }
 
+async fn replace_node_and_wait_for_write_readiness(dist: &mut DistCluster, seed: &UpgradeSeed, node_idx: usize) -> TestResult {
+    dist.replace_node_with_current_binary(node_idx).await?;
+    // Node readiness proves reads can succeed; a rolling writer must also
+    // recover before replacing the next peer or checking upgrade contracts.
+    wait_for_upgrade_write_readiness(
+        &dist.clients()?,
+        &seed.history_bucket,
+        &format!("replaced-node-{node_idx}"),
+        CREDENTIAL_TIMEOUT,
+    )
+    .await
+}
+
 #[tokio::test]
 async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> TestResult {
     init_logging();
@@ -331,15 +424,15 @@ async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> Te
 
     let seed = seed_history_and_iam(&dist).await?;
 
-    dist.replace_node_with_current_binary(0).await?;
+    replace_node_and_wait_for_write_readiness(&mut dist, &seed, 0).await?;
     assert_history_and_iam(&dist, &seed, "one-current-node").await?;
 
     for node_idx in [1, 2] {
-        dist.replace_node_with_current_binary(node_idx).await?;
+        replace_node_and_wait_for_write_readiness(&mut dist, &seed, node_idx).await?;
     }
     assert_history_and_iam(&dist, &seed, "one-previous-node").await?;
 
-    dist.replace_node_with_current_binary(3).await?;
+    replace_node_and_wait_for_write_readiness(&mut dist, &seed, 3).await?;
     assert_history_and_iam(&dist, &seed, "homogeneous-current").await?;
     Ok(())
 }

crates/e2e_test/src/distributed_startup_regression_test.rs

@@ -215,6 +215,8 @@ mod tests {
         probe_startup_cas_binary(&binary, &nonce, &artifact).await?;
 
         let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
+        // Isolate bootstrap CAS and its S3 readback from periodic usage publication.
+        cluster.set_env("RUSTFS_SCANNER_ENABLED", "false");
         let mut logs = Vec::new();
         let mut disks = Vec::new();
         let mut endpoints = Vec::new();

crates/e2e_test/src/fake_s3_target/mod.rs

@@ -50,8 +50,8 @@ use std::collections::{BTreeMap, HashMap, VecDeque};
 use std::error::Error;
 use std::io;
 use std::net::SocketAddr;
-use std::sync::{Arc, Mutex, MutexGuard};
-use std::time::{Duration, SystemTime};
+use std::sync::{Arc, Mutex, MutexGuard, OnceLock};
+use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
 use tokio::net::TcpListener;
 use tokio::sync::{OwnedSemaphorePermit, Semaphore, watch};
 use tokio::task::JoinHandle;
@@ -363,6 +363,14 @@ pub struct RequestRecord {
     pub upload_id: Option<String>,
     pub part_number: Option<i32>,
     pub content_length: Option<u64>,
+    pub source_version_id: Option<String>,
+    pub source_mtime: Option<String>,
+    pub source_etag: Option<String>,
+    pub source_replication_request: bool,
+    pub journaled_at_unix_millis: Option<u64>,
+    /// Computed by the target, before response stall/disconnect faults. This
+    /// does not prove that any response bytes reached the client.
+    pub prepared_response: Option<PreparedResponse>,
     pub consumed_bytes: Option<usize>,
     pub replication_timestamps: ReplicationTimestampHeaders,
     pub proxy_headers: ProxyHeaderSnapshot,
@@ -381,6 +389,20 @@ pub struct RequestRecord {
     pub fault: Option<FaultAction>,
 }
 
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct PreparedResponse {
+    pub status: u16,
+    pub version_id: Option<String>,
+    pub etag: Option<String>,
+    pub last_modified: Option<String>,
+    pub content_length: Option<u64>,
+    pub prepared_at_unix_millis: Option<u64>,
+    pub elapsed: Duration,
+}
+
+#[derive(Clone, Default)]
+struct RequestSequence(Arc<OnceLock<u64>>);
+
 /// Integrity and framing headers of an upload. A remote target's acceptance
 /// rules key off exactly these (rustfs#6853: `aws-chunked` framing stored
 /// verbatim; rustfs#7082: Object Lock PUTs need `Content-MD5` or
@@ -430,6 +452,10 @@ impl TransportSnapshot {
 #[derive(Debug, Clone, Default, PartialEq, Eq)]
 struct JournaledHeaders {
     content_length: Option<u64>,
+    source_version_id: Option<String>,
+    source_mtime: Option<String>,
+    source_etag: Option<String>,
+    source_replication_request: bool,
     replication_timestamps: ReplicationTimestampHeaders,
     proxy_headers: ProxyHeaderSnapshot,
     transport: TransportSnapshot,
@@ -444,6 +470,10 @@ impl JournaledHeaders {
                 .get(CONTENT_LENGTH)
                 .and_then(|value| value.to_str().ok())
                 .and_then(|value| value.parse().ok()),
+            source_version_id: header_value(headers, &SOURCE_VERSION_ID_HEADERS).map(bounded_journal_value),
+            source_mtime: header_value(headers, &SOURCE_MTIME_HEADERS).map(bounded_journal_value),
+            source_etag: header_value(headers, &SOURCE_ETAG_HEADERS).map(bounded_journal_value),
+            source_replication_request: header_value(headers, &SOURCE_REPLICATION_REQUEST_HEADERS).as_deref() == Some("true"),
             replication_timestamps: ReplicationTimestampHeaders::from_headers(headers),
             proxy_headers: ProxyHeaderSnapshot::from_headers(headers),
             transport: TransportSnapshot::from_headers(headers),
@@ -511,6 +541,10 @@ struct StoreState {
     /// version id the target never had answers 404 `NoSuchVersion` instead of
     /// the idempotent 204 RustFS/MinIO give.
     reject_unknown_version_deletes: bool,
+    /// Models MinIO and its derivatives (Storj): `X-Minio-Internal-` is a reserved
+    /// metadata prefix, so a PutObject or UploadPart carrying any such request
+    /// header is refused with `InvalidArgument` (rustfs#8362).
+    reject_reserved_internal_headers: bool,
     limits: StoreLimits,
     buckets: HashMap<String, BucketState>,
     uploads: HashMap<String, MultipartState>,
@@ -738,6 +772,7 @@ impl FakeS3Target {
         let connection_limit = Arc::new(Semaphore::new(MAX_CONNECTIONS));
         let (active_connections_tx, active_connections) = watch::channel(0usize);
         let task_connection_limit = Arc::clone(&connection_limit);
+        let task_control = Arc::clone(&control);
 
         let task = tokio::spawn(async move {
             let mut connections = tokio::task::JoinSet::new();
@@ -751,6 +786,7 @@ impl FakeS3Target {
                         };
                         active_connections_tx.send_modify(|count| *count += 1);
                         let service = service.clone();
+                        let control = Arc::clone(&task_control);
                         let active_connections = active_connections_tx.clone();
                         let mut connection_gate = connection_gate.clone();
                         connections.spawn(async move {
@@ -764,7 +800,7 @@ impl FakeS3Target {
                             {
                                 return;
                             }
-                            let handler = service_fn(move |request| handle_request(request, service.clone()));
+                            let handler = service_fn(move |request| handle_request(request, service.clone(), Arc::clone(&control)));
                             let mut connection = http1::Builder::new();
                             connection
                                 .keep_alive(false)
@@ -975,6 +1011,43 @@ impl FakeS3Target {
         lock(&self.backend.store).reject_unknown_version_deletes = enabled;
     }
 
+    /// Refuse every PutObject / UploadPart that carries an `x-minio-internal-*`
+    /// request header, the way MinIO and Storj do.
+    pub fn reject_reserved_internal_headers(&self, enabled: bool) {
+        lock(&self.backend.store).reject_reserved_internal_headers = enabled;
+    }
+
+    /// Keys of every stored, non-delete object in the bucket, sorted.
+    pub fn stored_keys(&self, bucket: &str) -> Vec<String> {
+        let state = lock(&self.backend.store);
+        let mut keys: Vec<String> = state
+            .buckets
+            .get(bucket)
+            .map(|bucket| {
+                bucket
+                    .objects
+                    .iter()
+                    .filter(|(_, versions)| versions.last().is_some_and(|version| !version.delete_marker))
+                    .map(|(key, _)| key.clone())
+                    .collect()
+            })
+            .unwrap_or_default();
+        keys.sort();
+        keys
+    }
+
+    /// Body and user metadata (names as stored, without the `x-amz-meta-` prefix)
+    /// of the latest non-delete version of an object.
+    pub fn stored_object(&self, bucket: &str, key: &str) -> Option<(Bytes, HashMap<String, String>)> {
+        lock(&self.backend.store)
+            .buckets
+            .get(bucket)
+            .and_then(|bucket| bucket.objects.get(key))
+            .and_then(|versions| versions.last())
+            .filter(|version| !version.delete_marker)
+            .map(|version| (version.body.clone(), version.metadata.clone().unwrap_or_default()))
+    }
+
     pub fn require_checksum_for_object_lock(&self, enabled: bool) {
         lock(&self.backend.store).require_checksum_for_object_lock = enabled;
     }
@@ -1161,7 +1234,10 @@ impl S3Access for FaultAccess {
         let operation = operation_from_s3_name(context.s3_op().name());
         let journaled = JournaledHeaders::from_headers(context.headers());
         let aws_chunked = journaled.transport.aws_chunked;
-        let fault = record_request(&self.control, operation, context.method().clone(), parsed, journaled);
+        let (sequence, fault) = record_request(&self.control, operation, context.method().clone(), parsed, journaled);
+        if let Some(slot) = context.extensions_mut().get::<RequestSequence>() {
+            let _ = slot.0.set(sequence);
+        }
         if let Some(status) = fault.as_ref().and_then(|fault| scripted_status(&fault.action)) {
             return Err(scripted_status_error(status));
         }
@@ -1174,6 +1250,15 @@ impl S3Access for FaultAccess {
                 "this target does not decode aws-chunked request bodies; send a plain signed payload with an exact Content-Length"
             ));
         }
+        if matches!(operation, Operation::PutObject | Operation::UploadPart)
+            && lock(&self.store).reject_reserved_internal_headers
+            && context
+                .headers()
+                .keys()
+                .any(|name| name.as_str().starts_with("x-minio-internal-"))
+        {
+            return Err(s3s::s3_error!(InvalidArgument, "Your metadata headers are not supported."));
+        }
         let prebody_permit = if operation == Operation::CompleteMultipartUpload {
             Some(
                 timeout(
@@ -1228,7 +1313,7 @@ fn record_request(
     method: Method,
     parsed: ParsedRequest,
     headers: JournaledHeaders,
-) -> Option<RequestFault> {
+) -> (u64, Option<RequestFault>) {
     let mut state = lock(control);
     let action = parsed
         .key
@@ -1251,6 +1336,12 @@ fn record_request(
         upload_id: parsed.upload_id.map(bounded_journal_value),
         part_number: parsed.part_number,
         content_length: headers.content_length,
+        source_version_id: headers.source_version_id,
+        source_mtime: headers.source_mtime,
+        source_etag: headers.source_etag,
+        source_replication_request: headers.source_replication_request,
+        journaled_at_unix_millis: unix_millis(),
+        prepared_response: None,
         consumed_bytes: None,
         replication_timestamps: headers.replication_timestamps,
         proxy_headers: headers.proxy_headers,
@@ -1261,7 +1352,14 @@ fn record_request(
         continuation_token: parsed.continuation_token.map(bounded_journal_value),
         fault: action.clone(),
     });
-    action.map(|action| RequestFault { sequence, action })
+    (sequence, action.map(|action| RequestFault { sequence, action }))
+}
+
+fn unix_millis() -> Option<u64> {
+    SystemTime::now()
+        .duration_since(UNIX_EPOCH)
+        .ok()
+        .and_then(|elapsed| u64::try_from(elapsed.as_millis()).ok())
 }
 
 fn bounded_journal_value(mut value: String) -> String {
@@ -1269,14 +1367,31 @@ fn bounded_journal_value(mut value: String) -> String {
     while !value.is_char_boundary(end) {
         end -= 1;
     }
-    value.truncate(end);
-    value
+    if value.capacity() > MAX_RETAINED_IDENTIFIER_BYTES {
+        value[..end].to_owned()
+    } else {
+        value.truncate(end);
+        value
+    }
 }
 
-async fn handle_request(request: Request<Incoming>, service: S3Service) -> Result<Response<Body>, BoxError> {
-    let mut response = timeout(MAX_REQUEST_DURATION, call_s3(service, request.map(Body::from)))
-        .await
-        .map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "fake target request exceeded 65 seconds"))??;
+async fn handle_request(
+    mut request: Request<Incoming>,
+    service: S3Service,
+    control: Arc<Mutex<ControlState>>,
+) -> Result<Response<Body>, BoxError> {
+    let started = Instant::now();
+    let sequence = RequestSequence::default();
+    request.extensions_mut().insert(sequence.clone());
+    let mut response = timeout(MAX_REQUEST_DURATION, async {
+        let result = call_s3(service, request.map(Body::from)).await;
+        if let (Ok(response), Some(sequence)) = (&result, sequence.0.get()) {
+            update_prepared_response(&control, *sequence, response, started.elapsed());
+        }
+        result
+    })
+    .await
+    .map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "fake target request exceeded 65 seconds"))??;
     if response.headers_mut().remove(DISCONNECT_HEADER).is_some() {
         return Err(io::Error::new(io::ErrorKind::ConnectionAborted, "fake target scripted disconnect").into());
     }
@@ -1531,6 +1646,25 @@ fn update_consumed(control: &Mutex<ControlState>, sequence: u64, consumed: usize
     }
 }
 
+fn update_prepared_response(control: &Mutex<ControlState>, sequence: u64, response: &Response<Body>, elapsed: Duration) {
+    let prepared = PreparedResponse {
+        status: response.status().as_u16(),
+        version_id: header_value(response.headers(), &["x-amz-version-id"]).map(bounded_journal_value),
+        etag: header_value(response.headers(), &[ETAG.as_str()]).map(bounded_journal_value),
+        last_modified: header_value(response.headers(), &[LAST_MODIFIED.as_str()]).map(bounded_journal_value),
+        content_length: response
+            .headers()
+            .get(CONTENT_LENGTH)
+            .and_then(|value| value.to_str().ok())
+            .and_then(|value| value.parse().ok()),
+        prepared_at_unix_millis: unix_millis(),
+        elapsed,
+    };
+    if let Some(record) = lock(control).requests.iter_mut().find(|record| record.sequence == sequence) {
+        record.prepared_response = Some(prepared);
+    }
+}
+
 async fn apply_non_body_fault(fault: Option<&RequestFault>, control: &Mutex<ControlState>) -> S3Result<()> {
     match fault.map(|fault| &fault.action) {
         Some(action @ (FaultAction::Status(_) | FaultAction::ResponseStatus(_))) => {
@@ -3230,10 +3364,38 @@ mod tests {
         let retry = client.clone();
         requests.spawn(async move { get_bytes(&retry, bucket, "held", None).await });
         timeout(Duration::from_secs(2), gate.entered.wait_for(|count| *count == 2)).await??;
+        let held_retry = target
+            .requests()
+            .into_iter()
+            .rev()
+            .find(|record| record.operation == Operation::GetObject && record.key.as_deref() == Some("held"))
+            .expect("the held retry must be journaled");
+        assert_eq!(held_retry.prepared_response, None);
         assert_eq!(
             timeout(Duration::from_secs(2), get_bytes(&client, bucket, "unrelated", None)).await??,
             Bytes::from_static(b"payload")
         );
+        let unrelated = target
+            .requests()
+            .into_iter()
+            .find(|record| record.operation == Operation::GetObject && record.key.as_deref() == Some("unrelated"))
+            .expect("the unrelated GET must be journaled");
+        let unrelated_response = unrelated.prepared_response.expect("the unrelated GET prepared a response");
+        assert_eq!(unrelated_response.status, 200);
+        assert_eq!(
+            unrelated_response.version_id.as_deref(),
+            Some(target.stored_versions(bucket, "unrelated")[0].0.as_str())
+        );
+        assert!(held_retry.sequence < unrelated.sequence);
+        assert_eq!(
+            target
+                .requests()
+                .into_iter()
+                .find(|record| record.sequence == held_retry.sequence)
+                .expect("the held retry remains journaled")
+                .prepared_response,
+            None
+        );
         assert!(requests.try_join_next().is_none(), "the retry must remain behind the gate");
         drop(gate);
         assert_eq!(
@@ -3242,6 +3404,18 @@ mod tests {
                 .expect("retried GET task")??,
             Bytes::from_static(b"payload")
         );
+        let held_response = target
+            .requests()
+            .into_iter()
+            .find(|record| record.sequence == held_retry.sequence)
+            .and_then(|record| record.prepared_response)
+            .expect("the released retry prepared its own response");
+        assert_eq!(held_response.status, 200);
+        assert_eq!(
+            held_response.version_id.as_deref(),
+            Some(target.stored_versions(bucket, "held")[0].0.as_str())
+        );
+        assert_ne!(held_response.version_id, unrelated_response.version_id);
         assert_eq!(get_bytes(&client, bucket, "held", None).await?, Bytes::from_static(b"payload"));
         assert_eq!(target.count_requests(Operation::GetObject, "held"), 3);
         Ok(())
@@ -3586,6 +3760,23 @@ mod tests {
             .await?;
         assert_eq!(first_head.content_length(), Some(5));
         assert_eq!(first_head.version_id(), first.version_id.as_deref());
+        let journaled_head = target
+            .requests()
+            .into_iter()
+            .rev()
+            .find(|record| record.operation == Operation::HeadObject)
+            .expect("the version HEAD must be journaled");
+        assert_eq!(journaled_head.version_id, first.version_id);
+        assert!(journaled_head.journaled_at_unix_millis.is_some());
+        let prepared_head = journaled_head
+            .prepared_response
+            .expect("the version HEAD prepared a response");
+        assert_eq!(prepared_head.status, 200);
+        assert_eq!(prepared_head.version_id.as_deref(), first_head.version_id());
+        assert_eq!(prepared_head.etag.as_deref(), first_head.e_tag());
+        assert_eq!(prepared_head.content_length, Some(5));
+        assert!(prepared_head.last_modified.is_some());
+        assert!(prepared_head.prepared_at_unix_millis.is_some());
         assert_eq!(
             get_bytes(&client, "target-bucket", "nested/key", None).await?,
             Bytes::from_static(b"second")
@@ -3611,6 +3802,20 @@ mod tests {
             .await
             .expect_err("deleted version must not remain readable");
         assert_sdk_error!(deleted, 404, "NoSuchVersion");
+        let deleted_record = target
+            .requests()
+            .into_iter()
+            .rev()
+            .find(|record| record.operation == Operation::GetObject)
+            .expect("the deleted version GET must be journaled");
+        assert_eq!(deleted_record.version_id, first.version_id);
+        assert_eq!(
+            deleted_record
+                .prepared_response
+                .expect("the error response was prepared")
+                .status,
+            404
+        );
         let marker = client
             .delete_object()
             .bucket("target-bucket")
@@ -4042,7 +4247,21 @@ mod tests {
             .await
             .expect_err("wrong part ETag must fail without consuming the upload");
         assert_sdk_error!(invalid_complete, 400, "InvalidPart");
-        client
+        let rejected_complete = target
+            .requests()
+            .into_iter()
+            .rev()
+            .find(|record| record.operation == Operation::CompleteMultipartUpload)
+            .expect("the rejected completion must be journaled");
+        assert_eq!(rejected_complete.upload_id.as_deref(), Some(upload_id.as_str()));
+        assert_eq!(
+            rejected_complete
+                .prepared_response
+                .expect("the rejection was prepared")
+                .status,
+            400
+        );
+        let successful_complete = client
             .complete_multipart_upload()
             .bucket("target-bucket")
             .key("multipart")
@@ -4050,6 +4269,18 @@ mod tests {
             .multipart_upload(CompletedMultipartUpload::builder().set_parts(Some(parts)).build())
             .send()
             .await?;
+        let completed_record = target
+            .requests()
+            .into_iter()
+            .rev()
+            .find(|record| record.operation == Operation::CompleteMultipartUpload)
+            .expect("the successful completion must be journaled");
+        assert_eq!(completed_record.upload_id.as_deref(), Some(upload_id.as_str()));
+        let prepared_complete = completed_record
+            .prepared_response
+            .expect("the completion response was prepared");
+        assert_eq!(prepared_complete.status, 200);
+        assert_eq!(prepared_complete.version_id.as_deref(), successful_complete.version_id());
         let completed = get_bytes(&client, "target-bucket", "multipart", None).await?;
         assert_eq!(completed.len(), MIN_MULTIPART_PART_BYTES + 5);
         assert_eq!(&completed[..MIN_MULTIPART_PART_BYTES], first_body.as_ref());
@@ -4430,6 +4661,32 @@ mod tests {
             .send()
             .await?;
         assert_eq!(completed.version_id(), Some(completed_version.as_str()));
+        let receipt_records = target.requests();
+        let receipt_create = receipt_records
+            .iter()
+            .find(|record| {
+                record.operation == Operation::CreateMultipartUpload && record.key.as_deref() == Some("multipart-receipt-time")
+            })
+            .expect("the source version must be retained on MPU creation");
+        assert_eq!(receipt_create.source_version_id.as_deref(), Some(completed_version.as_str()));
+        let receipt_complete = receipt_records
+            .iter()
+            .find(|record| {
+                record.operation == Operation::CompleteMultipartUpload
+                    && record.upload_id.as_deref() == Some(receipt_upload.as_str())
+            })
+            .expect("the source mtime must be retained on completion");
+        assert_eq!(receipt_complete.source_mtime.as_deref(), Some("1970-01-01T00:00:00Z"));
+        assert!(receipt_complete.source_replication_request);
+        assert_eq!(
+            receipt_complete
+                .prepared_response
+                .as_ref()
+                .expect("the receipt-time response was prepared")
+                .version_id
+                .as_deref(),
+            completed.version_id()
+        );
         assert_eq!(
             get_bytes(&client, "target-bucket", "multipart-receipt-time", None).await?,
             Bytes::from_static(b"completed"),
@@ -4648,6 +4905,23 @@ mod tests {
             .expect_err("disconnected request must not store an object");
         assert_sdk_error!(disconnected, 404, "NotFound");
 
+        target.inject_for_key(Operation::PutObject, "lost-response", FaultAction::DisconnectAfterResponse, 1);
+        client
+            .put_object()
+            .bucket("target-bucket")
+            .key("lost-response")
+            .body(ByteStream::from_static(b"stored"))
+            .send()
+            .await
+            .expect_err("a prepared response does not prove client delivery");
+        let lost_response_head = client
+            .head_object()
+            .bucket("target-bucket")
+            .key("lost-response")
+            .send()
+            .await?;
+        assert_eq!(lost_response_head.content_length(), Some(6));
+
         assert_eq!(
             parse_request(&Method::PUT, &"/target-bucket/key?uploadId=upload&partNumber=bad".parse::<Uri>()?).operation,
             Operation::Unknown
@@ -4661,6 +4935,28 @@ mod tests {
                 .count(),
             4
         );
+        assert!(
+            requests
+                .iter()
+                .filter(|request| request.fault == Some(FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE)))
+                .all(|request| {
+                    request
+                        .prepared_response
+                        .as_ref()
+                        .is_some_and(|response| response.status == 503)
+                })
+        );
+        let lost_response = requests
+            .iter()
+            .find(|request| request.operation == Operation::PutObject && request.key.as_deref() == Some("lost-response"))
+            .expect("the disconnected response must be journaled");
+        assert_eq!(lost_response.fault, Some(FaultAction::DisconnectAfterResponse));
+        let prepared_lost_response = lost_response
+            .prepared_response
+            .as_ref()
+            .expect("the target prepared the successful response");
+        assert_eq!(prepared_lost_response.status, 200);
+        assert_eq!(prepared_lost_response.version_id.as_deref(), lost_response_head.version_id());
         assert!(
             requests
                 .iter()
@@ -4851,8 +5147,44 @@ mod tests {
             assert_eq!(records.len(), MAX_REQUEST_RECORDS);
             assert_eq!(records.front().map(|record| record.sequence), Some(2));
             assert_eq!(records.back().map(|record| record.sequence), Some((MAX_REQUEST_RECORDS + 1) as u64));
+            let mut oversized_allocation = String::with_capacity(MAX_RETAINED_IDENTIFIER_BYTES * 16);
+            oversized_allocation.push_str("short");
+            let short_value = bounded_journal_value(oversized_allocation);
+            assert_eq!(short_value, "short");
+            assert!(short_value.capacity() <= MAX_RETAINED_IDENTIFIER_BYTES);
+            let mut response = Response::new(Body::empty());
+            for name in ["x-amz-version-id", "etag", "last-modified"] {
+                response.headers_mut().insert(
+                    name,
+                    HeaderValue::from_str(&"r".repeat(MAX_RETAINED_IDENTIFIER_BYTES + 1)).expect("header"),
+                );
+            }
+            response.headers_mut().insert(CONTENT_LENGTH, HeaderValue::from_static("17"));
+            response
+                .headers_mut()
+                .insert("authorization", HeaderValue::from_static("unrecorded-response-authorization"));
+            response.headers_mut().insert(
+                "x-amz-server-side-encryption-customer-key",
+                HeaderValue::from_static("unrecorded-response-customer-key"),
+            );
+            update_prepared_response(&control, 1, &response, Duration::ZERO);
+            update_prepared_response(&control, u64::MAX, &response, Duration::ZERO);
+            assert_eq!(lock(&control).requests, records, "misses and evicted records must not be resurrected");
             let bounded_control = Mutex::new(ControlState::default());
             let utf8_boundary = format!("{}é", "a".repeat(MAX_RETAINED_IDENTIFIER_BYTES - 1));
+            let mut source_headers = HeaderMap::new();
+            for name in ["x-rustfs-source-version-id", "x-minio-source-mtime", "x-rustfs-source-etag"] {
+                source_headers.insert(
+                    name,
+                    HeaderValue::from_str(&"s".repeat(MAX_RETAINED_IDENTIFIER_BYTES + 1)).expect("header"),
+                );
+            }
+            source_headers.insert("x-minio-source-replication-request", HeaderValue::from_static("true"));
+            source_headers.insert("authorization", HeaderValue::from_static("unrecorded-request-authorization"));
+            source_headers.insert(
+                "x-amz-server-side-encryption-customer-key",
+                HeaderValue::from_static("unrecorded-request-customer-key"),
+            );
             record_request(
                 &bounded_control,
                 Operation::GetObject,
@@ -4865,8 +5197,9 @@ mod tests {
                     upload_id: Some("u".repeat(MAX_RETAINED_IDENTIFIER_BYTES + 1)),
                     ..Default::default()
                 },
-                JournaledHeaders::default(),
+                JournaledHeaders::from_headers(&source_headers),
             );
+            update_prepared_response(&bounded_control, 1, &response, Duration::from_millis(7));
             {
                 let bounded_records = lock(&bounded_control);
                 let bounded = &bounded_records.requests[0];
@@ -4874,7 +5207,39 @@ mod tests {
                 assert_eq!(bounded.key.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES - 1));
                 assert_eq!(bounded.version_id.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
                 assert_eq!(bounded.upload_id.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
+                assert_eq!(bounded.source_version_id.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
+                assert_eq!(bounded.source_mtime.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
+                assert_eq!(bounded.source_etag.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
+                for value in [&bounded.source_version_id, &bounded.source_mtime, &bounded.source_etag]
+                    .into_iter()
+                    .flatten()
+                {
+                    assert!(value.capacity() <= MAX_RETAINED_IDENTIFIER_BYTES);
+                }
+                assert!(bounded.source_replication_request);
+                let prepared = bounded
+                    .prepared_response
+                    .as_ref()
+                    .expect("the retained record receives its response");
+                assert_eq!(prepared.version_id.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
+                assert_eq!(prepared.etag.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
+                assert_eq!(prepared.last_modified.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
+                for value in [&prepared.version_id, &prepared.etag, &prepared.last_modified]
+                    .into_iter()
+                    .flatten()
+                {
+                    assert!(value.capacity() <= MAX_RETAINED_IDENTIFIER_BYTES);
+                }
+                assert_eq!(prepared.content_length, Some(17));
+                assert_eq!(prepared.elapsed, Duration::from_millis(7));
+                assert!(
+                    !format!("{bounded:?}").contains("unrecorded-"),
+                    "authorization and customer keys must not enter the journal"
+                );
             }
+            lock(&bounded_control).requests.clear();
+            update_prepared_response(&bounded_control, 1, &response, Duration::ZERO);
+            assert!(lock(&bounded_control).requests.is_empty(), "a drained record must not be resurrected");
             let target = FakeS3Target::start().await.expect("target");
             target.create_bucket("b".repeat(63));
             let empty_bucket = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {