fix(ecstore): collapse ILM expiry worker env knobs to canonical name

Co-authored-by: heihutu <heihutu@gmail.com>

.agents/skills/adversarial-validation/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: adversarial-validation
-description: Execute the Adversarial Validation policy from the root AGENTS.md — run the seven reviewer roles (correctness, simplicity, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
+description: Execute the Adversarial Validation policy from the root AGENTS.md — run the applicable reviewer roles with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, design proposal, or agent-instruction change that alters execution before declaring it done.
 ---
 
 # Adversarial Validation Playbooks
@@ -61,14 +61,15 @@ Null report example: "Attacked quorum-1 error reduction, exact max-keys listing
 
 ### Simplicity adversary
 
-- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
+- Smaller-diff attack: inspect production growth separately from tests, fixtures, generated code, and documentation; test additions have no growth budget. Rewrite the production diff mentally (or in scratch) as the minimal equivalent edit. Report a finding only with a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries; fewer lines alone are not evidence.
   - Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
-  - Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Reuse Before You Write' (constants clause); the Adversarial Validation roles list charters the simplicity adversary with exactly this attack.
-- Reuse-and-necessity attack: for each new helper the diff introduces, run `ls crates/utils/src crates/common/src` and `rg -i 'fn \w*<term>'` over those dirs plus the touched crate (snake_case signatures — a full-text single-word grep drowns, a multi-word phrase returns nothing). A reimplementation of an existing workspace utility, or of plain std/tokio behavior no wrapper refines, is a finding — but so is forced reuse with mismatched semantics (normalization such as `clean` resolving `.`/`..` against raw S3 keys, error type, backoff, durability gating). For each new defensive branch, demand the nameable trigger and flag re-validation of what a validated upstream layer on the SAME path already guarantees — excluding the Cross-Cutting Domain Invariant patterns (nil/empty/absent UUID, dual metadata keys, unversioned-tier versionId) and re-checks before destructive actions, which are load-bearing even when redundant on the happy path. For each new test, flag near-duplicates pinning the same code path AND poison-value class as an existing test — boundary companions (n==max vs max+1, absent vs empty vs nil UUID, MetaObject vs MetaDeleteMarker) are never near-duplicates; the test-coverage skeptic playbook below mandates them.
-  - Where: Any diff adding helpers, branches on decoded/peer data, or tests; helper checks against crates/utils, crates/common, and the touched crate
+  - Evidence: AGENTS.md 'Change Style for Existing Logic' (conditional extraction rule, preserve sensitive control flow, canonical modules) and 'Reuse Before You Write'; the Adversarial Validation roles list charters this attack.
+- Reuse-and-necessity attack: for each new helper, search `crates/utils`, `crates/common`, the touched crate, the likely domain owner, and relevant direct dependencies. A reimplementation is a finding, but forced reuse with mismatched normalization, error, backoff, or durability semantics is also a finding. Demand a nameable trigger for new defensive branches. Tests remain subject to validity and near-duplicate coverage review, never a size limit.
+  - Where: Any diff adding helpers, branches on decoded/peer data, or tests
   - Evidence: AGENTS.md 'Reuse Before You Write' and 'Necessary Code Only'; GHSA-f4vq-9ffr-m8m3 (normalization-asymmetry traversal — why forced reuse of normalizing helpers on raw keys is itself an attack); docs/operations/tier-ilm-debugging.md nil-versionId incident (why boundary re-checks are load-bearing).
+- Replacement-and-comment attack: when the diff introduces a replacement path or representation, trace all callers and flag a superseded in-scope path left behind without a compatibility requirement. Keep one canonical core behind compatibility adapters. Comments must state non-obvious invariants completely without narration or change history. Never demand unrelated deletion or trade away correctness, compatibility, or readability to reduce the diff.
 
-Null report example: "Rewrote the diff as an in-place edit (no smaller equivalent exists), grepped both new helpers against crates/utils, crates/common, and the touched crate (no existing equivalent; call-site semantics checked), verified the two new defensive branches name concrete corrupt-input triggers, and checked the added tests against the existing suite (each pins a distinct poison-value class) — no break found."
+Null report example: "Separated production growth from tests/docs, tested a smaller equivalent, checked helper reuse and superseded paths, and found no break."
 
 ### Security reviewer
 
@@ -195,9 +196,9 @@ Null report example: "Attacked dual-key metadata writes/removals against MinIO-o
 
 ### Performance reviewer
 
-- For every `.clone()` the diff adds or moves onto a per-request/per-object path, open the cloned type and count heap fields (String, Vec, HashMap, Bytes). If >5 heap fields or it contains an EC block buffer, construct the cost: N concurrent PUTs x M objects -> N*M deep copies per second. Demand Arc-wrapping of heavy fields or pass-by-reference; also flag new `String` allocations in header/path/signature parsing where `&str`/`Cow<str>` suffices.
+- For each `.clone()` or allocation added to a per-request/per-object path, identify the copied data and execution frequency. Report a finding only for a concrete repeated cost or benchmark regression. Recommend borrowing, moving, `Bytes`/`Arc`, `Cow`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs.
   - Where: crates/ecstore/src/set_disk/**, crates/ecstore/src/store*.rs, rustfs/src/storage/, crates/filemeta/, request handlers in rustfs/src/
-  - Evidence: crates/ecstore/AGENTS.md 'Allocation Discipline in Hot Paths' (no Clone on >5-heap-field structs, Arc for large buffers, &str/Cow for temporary computations); .agents/skills/rust-code-quality/SKILL.md ranks 'unnecessary clone in hot path' as P1 must-fix
+  - Evidence: crates/ecstore/AGENTS.md 'Allocation Discipline in Hot Paths'; .agents/skills/rust-code-quality/SKILL.md requires a concrete hot-path cost rather than a proxy metric
 - For every new sync_all/sync_data/fdatasync/flush/File::sync call in the diff, trace the call chain to DurabilityMode / RUSTFS_DRIVE_SYNC_ENABLE resolution (crates/ecstore/src/disk/local.rs:291 DurabilityMode, :347 resolve_durability_mode) and to per-bucket durability overrides. Construct the run where the operator sets mode=none (or legacy RUSTFS_DRIVE_SYNC_ENABLE=false) and the new fsync still fires — that is an ungated durability cost and a regression on 4KiB writes.
   - Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/bucket/durability.rs, crates/ecstore/src/set_disk/** (rename_data/commit paths), any crate doing tokio::fs or std::fs writes
   - Evidence: #4221 fsync work caused a measured -10% 4KiB write regression (#814 investigation), later gated; durability modes added in eaff17cad (#4397), per-bucket tier overrides in 13e48d93a (#4407); 2df315baf (#4493) shows even ancestor-dir fsyncs are routed through the gate
@@ -230,12 +231,12 @@ Null report example: "Attacked the new rename_data commit-section work, durabili
 
 ### Test-coverage skeptic
 
-- For every behavior claim in the PR description, revert that hunk (git stash / manual undo of the changed lines) and name the exact test (`cargo test -p <crate> <test_name>`) that fails. If no test fails on revert, the behavior is untested — file a finding, not a note. Especially verify the test exercises the REAL production call path, not a lookalike helper.
+- For every testable behavior claim in the PR description, revert that hunk and name the focused test or executable check that detects the revert. If no reasonable check exists, require the reason and residual risk from the validation floor. Especially verify the check exercises the real production path, not a lookalike helper.
   - Where: All crates; highest value in crates/ecstore, rustfs/src/storage, crates/heal
-  - Evidence: AGENTS.md exit criterion 'Every behavior change has a test that fails without it'. Real bug: PR #4220 (ghost-directory cleanup) merged with green tests but its fix never executed on the real delete path — required follow-up rustfs#4307, backlog#798 stayed OPEN. The tests exercised a path the production flow never took.
+  - Evidence: AGENTS.md testable-behavior exit criterion. Real bug: PR #4220 (ghost-directory cleanup) merged with green tests but its fix never executed on the real delete path — required follow-up rustfs#4307, backlog#798 stayed OPEN. The tests exercised a path the production flow never took.
 - Read each added/modified test and confirm it asserts the real outcome (returned value, stored bytes, error variant), not merely 'call succeeded' or 'no panic'. Flag any test whose only observable is that the function returned, and any `assert!(result.is_err())` that never checks WHICH error. Then check: does the test prove the exploit/failure form is denied, or only that the intended form still works?
   - Where: crates/e2e_test (security_boundary_test.rs pattern), and every #[cfg(test)] module in the diff
-  - Evidence: Commit dee8e4e63 (#4466) had to rewrite 277 lines of crates/e2e_test/src/security_boundary_test.rs because 'security boundary tests' passed without asserting real outcomes. .agents/skills/rust-code-quality/SKILL.md checklist: 'Every test function has at least one assert!'; .agents/skills/security-advisory-lessons/SKILL.md: 'Does the test prove the exploit form is denied, or only that the intended form still works?'
+  - Evidence: Commit dee8e4e63 (#4466) had to rewrite 277 lines of crates/e2e_test/src/security_boundary_test.rs because 'security boundary tests' passed without asserting real outcomes. .agents/skills/rust-code-quality/SKILL.md requires an observable failure criterion; .agents/skills/security-advisory-lessons/SKILL.md asks whether the exploit form is denied.
 - When the diff adds a boolean/mode parameter or config flag, find the test that fails if the flag's effect is INVERTED inside the changed function. Tests that were mechanically updated to pass `false`/default at every call site assert nothing about the new behavior. Execute the check: flip the flag's branch in the source and confirm at least one test goes red for each branch.
   - Where: crates/ecstore/src/set_disk/ (e.g. build_codec_streaming_part_reader), any function gaining a parameter
   - Evidence: Commit 05890d6e2 (#4573): PR #4560 added a 15th param allow_inplace_legacy_fallback; the arity tests were fixed by passing `false` everywhere — they assert Err outcomes independent of the flag, so the fallback behavior itself has no revert-detecting test at those sites.
@@ -260,7 +261,7 @@ Null report example: "Attacked the new rename_data commit-section work, durabili
 - For any pagination/limit/truncation change, construct the exact-boundary test: result count == max (page exactly full), max+1, and a delimiter re-fold that lands precisely on the page boundary — assert both the item count AND the is_truncated/continuation marker. Off-by-one at the page boundary is a recurring shipped bug here.
   - Where: crates/ecstore listing paths (list_objects, ListMultipartUploads, metacache), S3 handlers in rustfs/src/storage
   - Evidence: Two shipped boundary bugs: fefa70b31 (#4447) ListMultipartUploads returned one upload past max-uploads; d91f4d455 (#4538) delimiter re-fold of a full page lost the truncation flag. Both survived existing tests because no test pinned n == max exactly.
-- Green `cargo test -p <crate>` on the touched crate is not a coverage verdict for the diff's test code itself: run `cargo clippy --all-targets -p <crate>` and a workspace-wide test BUILD (`cargo check --workspace --all-targets` at minimum) before accepting the tests as evidence. Test-only code that doesn't compile workspace-wide or fails clippy has repeatedly broken main and masked whether tests ran at all.
+- A green focused test is evidence only for the targets it builds. Follow the `AGENTS.md` validation tier: add package-scoped Clippy or broader test-target compilation only when changed targets, features, or dependents remain uncovered; do not require a workspace-wide build by default.
   - Where: All crates; especially concurrent-branch merges into crates/ecstore
   - Evidence: #4322 broke main because only cargo test ran (field_reassign_with_default is clippy-only). b06f3df6b (#4441) and 05890d6e2 (#4573): test code broke the workspace test build (E0061) on main after textually-clean merges, failing CI for every open PR.
 
@@ -271,7 +272,6 @@ Null report example: "Attacked revert-detection for all 3 claimed behaviors (eac
 Probes are distilled from shipped bugs in git history (commit/PR references
 above), GitHub security advisories (see the security-advisory-lessons
 skill), scoped `AGENTS.md` rules, and invariants under `docs/architecture/`
-and `docs/operations/`. Line numbers drift; when a cited location no longer
-matches, trust the invariant and re-locate the code. When a new bug class
-ships, add a probe with its evidence here rather than growing the policy
-section in `AGENTS.md`.
+and `docs/operations/`. Line numbers drift; re-locate the invariant. Merge
+new incidents into an existing probe when they share a failure class; add a
+new probe only for a distinct attack, rather than growing the root policy.

.agents/skills/pr-creation-checker/SKILL.md

@@ -24,15 +24,17 @@ Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whe
 
 2. Inspect change scope
 - Review the diff and summarize what changed.
+- Inspect `git diff --stat` and `git diff --numstat`; assess production-code growth separately. Tests, fixtures, generated code, and documentation have no growth budget. Treat line counts as signals, not quotas.
 - Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
 - Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
+- Use the simplicity-adversary verdict instead of producing a per-symbol inventory. Block growth only when the review identifies duplication or gives a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries.
+- Confirm replacement implementations remove the superseded in-scope path or adapt compatibility at the boundary to one canonical core.
 - Scan the diff for newly added string literals and confirm whether they duplicate values already defined as constants/enums/typed wrappers in the same module or shared modules.
 - Treat introducing a new hardcoded literal where a project constant already exists as a likely regression risk; require either a refactor to reuse the constant or an explicit exception explanation in the PR body.
 
 3. Verify readiness requirements
-- Require `make pre-commit` before marking PRs ready when the diff changes Rust code, product behavior, CI behavior, runtime configuration, security-sensitive logic, migrations, storage, auth, networking, or other high-risk paths.
-- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, allow focused verification instead of `make pre-commit` when it directly validates the changed surface.
-- For focused verification, explain why the full gate was not run and list the scope-specific commands in the PR body.
+- Select checks from `AGENTS.md` "Verification Before PR" based on the final diff's risk tier. Do not replace a focused behavioral test with `make pre-commit`, or a required high-risk `make pre-pr` with a narrower gate.
+- For focused verification, state why the selected tier is sufficient and list the scope-specific commands in the PR body.
 - If `make` is unavailable, use the equivalent commands from `.config/make/`.
 - Add scope-specific verification commands when the changed area needs more than the baseline.
 - If required checks fail, stop and return `BLOCKED`.
@@ -81,13 +83,14 @@ Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whe
 
 ## Blocker rules
 
-- Return `BLOCKED` if a code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk change has not passed `make pre-commit`.
+- Return `BLOCKED` if the checks required by the `AGENTS.md` validation tier have not passed.
 - Return `BLOCKED` if a documentation-only, agent-instruction-only, or local developer-tooling-only change lacks focused verification for the changed surface.
 - Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
 - Return `BLOCKED` if required template sections are missing.
 - Return `BLOCKED` if the title/body is not in English.
 - Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
 - Return `BLOCKED` if the diff introduces string literals that should use existing constants but did not.
+- Return `BLOCKED` for production-code growth only when the review identifies a duplicated or superseded implementation, or supplies a concrete smaller design with equivalent semantics. Fewer lines alone are not evidence.
 
 ## Reference
 

.agents/skills/pr-creation-checker/references/pr-readiness-checklist.md

@@ -3,8 +3,8 @@
 - Confirm the branch is based on current `main`.
 - Confirm the diff matches the stated scope.
 - Confirm no secrets, logs, temp files, or unrelated refactors are included.
-- Confirm `make pre-commit` passed for code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk changes.
-- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, confirm focused verification covered the changed surface and the PR body explains why the full gate was not run.
+- Confirm the checks required by the `AGENTS.md` validation tier passed.
+- For focused verification, confirm it covered the changed surface and the PR body explains why the selected tier is sufficient.
 - Confirm extra verification commands are listed for risky changes.
 - Confirm the PR title uses Conventional Commits and stays within 72 characters.
 - Confirm the PR title does not use tool-specific prefixes such as `[codex]`.

.agents/skills/rust-code-quality/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: rust-code-quality
-description: Enforce Rust-specific code quality rules on every code change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
+description: Enforce Rust-specific code quality rules on every Rust change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
 ---
 
 # Rust Code Quality Gate
@@ -12,27 +12,29 @@ Use this skill on every Rust code change to enforce quality rules that `cargo cl
 1. Identify changed `.rs` files.
 2. Run automated checks on changed files.
 3. Run manual review checklist on the diff.
-4. Report findings; block merge if P0/P1 issues exist.
+4. Resolve or rebut every finding with evidence; P0/P1 findings cannot be deferred.
 
 ## Automated Checks
 
-Run these on every changed `.rs` file (excluding test modules):
+Use these searches to find candidates in changed `.rs` files. Inspect syntax,
+`#[cfg(test)]` scope, and the changed hunk before reporting a finding; text
+filters do not reliably distinguish production code from tests.
 
 ```bash
-# 1. unwrap/expect in production code
-rg -n '\.unwrap\(\)|\.expect\(' <changed-files> | grep -v '#\[cfg(test)\]' | grep -v 'test' | grep -v 'bench'
+# 1. unwrap/expect candidates
+rg -n '\.unwrap\(\)|\.expect\(' <changed-files>
 
 # 2. Silent type truncation via `as` cast
 rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
 
 # 3. String as error type
-rg -n 'Result<.*String>' <changed-files> | grep -v test
+rg -n 'Result<.*String>' <changed-files>
 
 # 4. Box<dyn Error> in public APIs
-rg -n 'Box<dyn.*Error' <changed-files> | grep -v test
+rg -n 'Box<dyn.*Error' <changed-files>
 
 # 5. println/eprintln in production
-rg -n 'println!\|eprintln!' <changed-files> | grep -v test
+rg -n 'println!\|eprintln!' <changed-files>
 
 # 6. Ordering::Relaxed usage (verify each is intentional)
 rg -n 'Ordering::Relaxed' <changed-files>
@@ -46,37 +48,35 @@ rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
 For every Rust code change, verify:
 
 ### Error Handling
-- [ ] No `unwrap()` or `expect()` in production code without justification comment
+- [ ] Every production `unwrap()` or `expect()` is infallible by type or a checked invariant; explain only non-obvious invariants, using an existing type, a useful `expect` message, or a concise comment
 - [ ] No `Result<_, String>` in public API signatures
-- [ ] No `Box<dyn Error>` in public trait/struct methods
+- [ ] Public library APIs use domain errors unless deliberate error erasure at a boundary is part of the contract
 - [ ] `Error::source()` is overridden when inner error is stored
-- [ ] Error messages are actionable (what failed, with what input)
+- [ ] Error messages are actionable without exposing secret input
 
 ### Type Safety
 - [ ] No silent `as` truncation (negative→unsigned, large→small)
-- [ ] `try_into()` or explicit clamping used for numeric conversions
-- [ ] No `f64 as usize` without prior clamping
+- [ ] Fallible numeric conversions use `TryFrom`/`try_into()` and return a typed error; clamp or saturate only when the domain explicitly requires it
+- [ ] Floating-point to integer conversion validates finiteness, sign, and range before conversion
 
 ### Concurrency
 - [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
 - [ ] No `tokio::sync` lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
-- [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
+- [ ] Atomic read-modify-write uses the direct `fetch_*` operation when possible; use `compare_exchange` only for conditional updates
 - [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
 
 ### Memory and Performance
-- [ ] No `.clone()` on structs with >5 heap-allocated fields in hot paths
-- [ ] `HashMap::with_capacity()` / `Vec::with_capacity()` used when size is known
-- [ ] Large buffers wrapped in `Arc` rather than cloned
-- [ ] Temporary string computations use `&str` or `Cow<str>` instead of `String`
+- [ ] On an identified hot path, report cloning or allocation only with a concrete per-request/per-object cost or benchmark signal
+- [ ] Prefer borrowing, moving, `Bytes`/`Arc`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs
 
 ### Recursion Safety
-- [ ] Recursive functions have a depth limit or use iterative traversal
+- [ ] Recursion over untrusted, persisted, or otherwise unbounded input has a depth limit or uses iterative traversal
 - [ ] Tree/cache traversals handle corrupted/cyclic input safely
 
 ### Testing
-- [ ] Every test function has at least one `assert!`
-- [ ] Tests use `.expect("context")` not bare `.unwrap()`
-- [ ] No `println!`/`eprintln!` in production code (use `tracing`)
+- [ ] Tests have an observable failure criterion; delegated assertions, `#[should_panic]`, snapshot/property checks, and meaningful `Result` failures do not need a redundant `assert!`
+- [ ] Use `expect` only when its message improves failure diagnosis; do not add boilerplate to self-evident test setup
+- [ ] Test volume and line count are never treated as production-code growth
 
 ### Serde
 - [ ] Structs from untrusted input have `#[serde(deny_unknown_fields)]`
@@ -88,18 +88,18 @@ For every Rust code change, verify:
 - [ ] New string literals don't duplicate existing constants
 
 ### Reuse and Necessity
-- [ ] No new helper duplicating an existing workspace utility (`crates/utils`, `crates/common`, the touched crate) or plain std/tokio behavior no wrapper refines; reused helpers match the call site's semantics (normalization, error type, backoff, durability gating)
+- [ ] No new helper duplicates `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, a relevant direct dependency, or plain std/tokio behavior; reused helpers match the call site's semantics
 - [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
 - [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
-- [ ] No comments narrating the next line, restating a signature, or describing the change itself (invariant comments — lock ordering, `SAFETY`, unwrap justification — are not narration)
+- [ ] Comments avoid narration and change history while completely stating non-obvious lock, `SAFETY`, durability, compatibility, and unwrap invariants
 - [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
 
 ## Severity Classification
 
-- **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
-- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method, `unwrap_or_default()` on a domain-required value (metadata, quorum, version id)
-- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`, new helper duplicating an existing workspace utility, defensive branch with no nameable trigger (corrupt or stale persisted/peer data is always a nameable trigger for boundary-crossing values), near-duplicate test, redundant error re-wrapping
-- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`, narrating comment
+- **P0 (Block merge)**: demonstrated data loss, security breach, remote crash, or deadlock
+- **P1 (Must fix)**: concrete correctness, compatibility, or material hot-path regression
+- **P2 (Should fix)**: avoidable duplication or maintainability issue with a concrete simpler replacement
+- **P3 (Nice to fix)**: local style or clarity issue with no behavioral risk
 
 ## Output Template
 
@@ -107,10 +107,10 @@ For every Rust code change, verify:
 ## Rust Code Quality Report
 
 ### Automated Scan
-- unwrap/expect in production: N found
-- as casts: N found
-- String errors: N found
-- println/eprintln: N found
+- unwrap/expect candidates inspected: N
+- numeric-cast candidates inspected: N
+- error-type candidates inspected: N
+- output-macro candidates inspected: N
 
 ### Findings
 - [P1] `path:line` — description

.agents/skills/rust-code-quality/references/checklist.md

@@ -1,52 +0,0 @@
-# Rust Code Quality Checklist
-
-Use this as a quick pre-merge checklist for every Rust code change.
-
-## Critical (P0 — block merge)
-
-| Check | Command |
-|-------|---------|
-| No `unwrap()` in request/storage hot path | `rg '\.unwrap\(\)' <files> \| grep -v test` |
-| No `as` truncation on user input | `rg ' as (u32\|usize\|i32)' <files>` |
-| Lock order consistent across call sites | Manual: trace all lock acquisitions |
-| Recursive functions have depth limit | Manual: check for `max_depth` or iterative pattern |
-| No `panic!`/`unwrap_or_else(panic!)` in production | `rg 'panic!\|unwrap_or_else.*panic' <files> \| grep -v test` |
-
-## High (P1 — must fix)
-
-| Check | Command |
-|-------|---------|
-| No `Result<_, String>` in public API | `rg 'Result<.*String>' <files> \| grep -v test` |
-| No `Box<dyn Error>` in public trait | `rg 'Box<dyn.*Error' <files> \| grep -v test` |
-| No unnecessary `.clone()` in hot path | Manual: check loops and per-request paths |
-| `Error::source()` implemented when inner error stored | Manual: check `impl Error` |
-| No `eprintln!`/`println!` in production | `rg 'println!\|eprintln!' <files> \| grep -v test` |
-
-## Medium (P2 — should fix)
-
-| Check | Command |
-|-------|---------|
-| Tests have assertions | Manual: check for `assert` in test functions |
-| `HashMap`/`Vec` use `with_capacity` when size known | Manual: check `::new()` in loops |
-| No `#![allow(dead_code)]` at crate root | `rg 'allow.dead_code' <files> \| grep 'lib.rs'` |
-| Serde structs from untrusted input have `deny_unknown_fields` | Manual: check `#[derive(Deserialize)]` |
-
-## Low (P3 — nice to fix)
-
-| Check | Command |
-|-------|---------|
-| No camelCase statics | `rg 'static ref [a-z]' <files>` |
-| `Arc::ptr_eq` instead of `as_ptr + ptr::eq` | `rg 'as_ptr\|ptr::eq' <files>` |
-| Public functions have doc comments | `rg 'pub fn' <files> \| grep -v '///'` |
-
-## Quick One-Liner
-
-```bash
-# Run all automated checks on changed files
-CHANGED=$(git diff --name-only HEAD~1 -- '*.rs' | grep -v test | grep -v bench)
-echo "=== unwrap/expect ===" && rg -c '\.unwrap\(\)|\.expect\(' $CHANGED 2>/dev/null
-echo "=== as casts ===" && rg -c ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' $CHANGED 2>/dev/null
-echo "=== String errors ===" && rg -c 'Result<.*String>' $CHANGED 2>/dev/null
-echo "=== println ===" && rg -c 'println!|eprintln!' $CHANGED 2>/dev/null
-echo "=== Ordering::Relaxed ===" && rg -c 'Ordering::Relaxed' $CHANGED 2>/dev/null
-```

.agents/skills/rustfs-logging-governance/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: rustfs-logging-governance
-description: Standardize and review RustFS logging with structured `tracing` events, lower noise on hot paths, preserve security-sensitive diagnostics, and extend guardrails to prevent legacy logging patterns from returning. Use when editing or reviewing RustFS logs, startup/config diagnostics, cloud metadata logs, request validation logs, or `scripts/check_logging_guardrails.sh`.
+description: Standardize and review RustFS logging with structured `tracing` events, lower noise on hot paths, preserve security-sensitive diagnostics, and extend guardrails to prevent legacy logging patterns from returning. Use whenever a change adds or edits any `tracing` macro call (`error!`/`warn!`/`info!`/`debug!`/`trace!`/`#[instrument]`) — including a single log line added in passing while fixing unrelated logic, which is how most new log sites enter the repo — and when reviewing RustFS logs, startup/config diagnostics, cloud metadata logs, request validation logs, or `scripts/check_logging_guardrails.sh`.
 ---
 
 # RustFS Logging Governance

.agents/skills/security-advisory-lessons/SKILL.md

@@ -66,14 +66,23 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 
 ### STS, OIDC, and federation flows
 - Every STS endpoint must have an explicit authentication story: SigV4 where required, OIDC token verification for web identity, and role/session policy validation before issuing credentials.
+- For web identity, the JWT is the credential; exemption from SigV4 is not itself an authentication bypass. Treat pre-verification claims only as untrusted routing hints, bound token size, normalize public failures, rate-limit discovery, and issue credentials only after signature, issuer, audience, and expiration checks.
 - JWT session tokens must be signed and verified by a trusted issuer/key path, not by service-account-controlled material or a reused root secret.
 - JWT verification must enforce required claims and expiration for every bearer token path; "allow missing exp" is never acceptable for user-presented credentials.
 - Public OIDC bootstrap and callback routes must treat `Host`, `X-Forwarded-Proto`, redirect targets, `state`, and callback parameters as untrusted; credential-bearing redirects require a configured, allowlisted origin.
 - OIDC discovery and validation URLs are SSRF sinks. Resolve and classify hostnames at connection time, reject rebinding to loopback/private/link-local ranges, and do not rely on literal string checks.
 
-### S3 copy, multipart, and presigned POST
+### IAM policy conditions and plugins
+- Treat request headers as attacker-controlled even after SigV4; callers sign their own spoofed headers. Do not merge them into server-derived condition keys such as identity, groups, version ID, signature version, JWT, or LDAP claims.
+- Keep the condition-key namespace explicit. Reserved server-derived keys must reject or ignore colliding headers, while intentional request-header keys such as `s3:x-amz-*` remain available.
+- Quantified IAM condition tests need partially overlapping multi-value sets. Fully contained and fully disjoint sets cannot distinguish `ForAllValues` from `ForAnyValue` bugs.
+- External policy plugins must receive the same security context as built-in policy evaluation. If OPA or another plugin depends on existing object tags, load and pass `ExistingObjectTag/*` before the plugin decision.
+
+### S3 object actions, copy, multipart, and presigned POST
+- Version-aware object requests need version-aware actions. Explicit `versionId` reads and copy sources must authorize `s3:GetObjectVersion`, not only `s3:GetObject`.
 - Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
 - Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
+- Fallbacks from version actions to non-version actions must still pass the same public-access-block, anonymous-deny, and post-authorization gates as a direct allow.
 - Presigned POST policies are server-side contracts. Enforce `content-length-range`, key prefix, exact metadata/content-type, and all signed policy conditions.
 
 ### Protocol frontends and IAM parity
@@ -132,6 +141,11 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 - When touching reader/writer wrappers such as hashing, encryption, compression, or warp readers, verify wrapper order and inspect stored bytes in regression tests.
 - Avoid helper shortcuts that unwrap nested readers and accidentally bypass encryption or integrity layers.
 
+### Object Lock and retention invariants
+- Object Lock state must fail closed when bucket metadata is unreadable, fabricated, or unparsable. Only a confirmed absence of Object Lock configuration may permit unprotected deletes or writes.
+- Do not collapse metadata read faults, missing persisted metadata, parse failures, and genuinely absent Object Lock config into one "not configured" result.
+- Retention enforcement must cover foreground deletes, batch deletes, force-delete helpers, default-retention materialization on PUT, lifecycle expiry, scanner sweeps, and all-versions expiry.
+
 ## Review Prompts
 
 Use these prompts while reviewing a diff:
@@ -148,5 +162,9 @@ Use these prompts while reviewing a diff:
 - Does this outbound validation path resolve attacker-supplied hostnames and reject private, loopback, link-local, and rebound addresses at the actual connection boundary?
 - Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
 - Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
+- Does an explicit object version, fallback action, or plugin authorization path pass through the same action and post-authorization gates as the direct S3 path?
+- Can a caller-controlled header populate a condition key that should be derived only by the server?
+- Do condition tests include partially overlapping multi-value inputs for quantified operators?
+- Does unreadable bucket metadata make Object Lock or retention enforcement fail closed rather than disappear?
 - Does a preview or browser-surface fix preserve the original security invariant when adding alternate viewers or file-type detection?
 - Does the test prove the exploit form is denied, or only that the intended form still works?

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -35,12 +35,21 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 ### STS, OIDC, and federation flows
 
 - `GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
-- `GHSA-ccrv-v8v9-ch9q` and `GHSA-48rf-7j3q-3hfv`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
+- `GHSA-jxrr-r6pv-h958`: unsigned JWT issuer data was decoded before verification to select an OIDC provider, and distinguishable failures could expose provider configuration. Lesson: web-identity routing may be unauthenticated, but pre-verification claims are untrusted routing hints; bound and rate-limit the request, normalize public errors, and verify signature, issuer, audience, and expiration before issuing credentials.
+- `GHSA-ccrv-v8v9-ch9q`, `GHSA-48rf-7j3q-3hfv`, and `GHSA-xvfh-7c9g-hpw2`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
 - `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
 - `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
 
-### S3 copy, multipart, and upload policy validation
+### IAM policy conditions and external policy plugins
 
+- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
+- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
+- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
+
+### S3 object actions, copy, multipart, and upload policy validation
+
+- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
+- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
 - `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
 - `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
 - `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
@@ -59,7 +68,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 ### Secrets, defaults, and cryptographic misuse
 
-- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, and `GHSA-63xc-c3w3-m2cf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
+- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, `GHSA-63xc-c3w3-m2cf`, and `GHSA-ch63-6q4v-hwp5`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
 - `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
 - `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
 - `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
@@ -92,6 +101,10 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
 
+### Object Lock and retention invariants
+
+- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
+
 ### Serde deserialization and input validation
 
 - No `#[serde(deny_unknown_fields)]` found across the entire codebase. Lesson: all structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs) should have `#[serde(deny_unknown_fields)]` to reject malformed or adversarial payloads.
@@ -107,11 +120,13 @@ Use these targeted searches when a diff touches security-sensitive code:
 rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
 rg -n "authorize_operation|FtpsDriver|SftpDriver|RETR|MKD|SIZE|MDTM|CreateBucket|GetObject|HeadObject" crates/protocols rustfs
 rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
+rg -n "ListBucketVersions|GetObjectVersion|versionId|VersionId|ExistingObjectTag|ForAllValues|ForAnyValue|POLICY_PLUGIN|opa" rustfs crates
 rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
 rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
 rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
 rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
 rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
+rg -n "ObjectLock|object_lock|retention|COMPLIANCE|GOVERNANCE|delete_prefix|lifecycle|scanner" rustfs crates
 rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 ```
 
@@ -121,9 +136,12 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 - Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
 - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
+- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
+- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
 - Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
 - IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
 - RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
+- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.

.config/make/lint-fmt.mak

@@ -60,6 +60,11 @@ body-cache-whitelist-check: ## Check the body-cache eligibility gate stays a fai
 	@echo "🧱 Checking body-cache whitelist guard..."
 	./scripts/check_body_cache_whitelist.sh
 
+.PHONY: s3s-footprint-check
+s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
+	@echo "📦 Checking s3s footprint ratchet..."
+	./scripts/check_s3s_footprint.sh
+
 .PHONY: fips-wording-check
 fips-wording-check: ## Check outward docs do not make unsupported FIPS claims
 	@echo "📣 Checking FIPS wording guard..."

.config/make/pre-commit.mak

@@ -19,13 +19,13 @@ planning-docs-check: ## Check that no planning-type documents are committed
 	./scripts/check_no_planning_docs.sh
 
 .PHONY: pre-commit
-pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check fips-wording-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
+pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
 	@echo "✅ All pre-commit checks passed!"
 
 .PHONY: pre-pr
-pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check fips-wording-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
+pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
 	@echo "✅ All pre-PR checks passed!"
 
 .PHONY: dev-check
-dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check fips-wording-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
+dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
 	@echo "✅ Fast development checks passed!"

.config/make/tests.mak

@@ -25,6 +25,7 @@ TEST_THREADS ?= 1
 script-tests: ## Run shell script tests
 	@echo "Running script tests..."
 	./scripts/test_build_rustfs_options.sh
+	./scripts/test_docker_runtime_timezone.sh
 	./scripts/test_entrypoint_credentials.sh
 	./scripts/test_internode_grpc_ab_bench.sh
 	./scripts/test_object_batch_bench_enhanced.sh

.config/nextest.toml

@@ -29,10 +29,13 @@
 
 [test-groups]
 ecstore-serial-flaky = { max-threads = 1 }
+embedded-test-ports = { max-threads = 1 }
+e2e-vault = { max-threads = 1 }
 
 # Reliability / fault-injection e2e tests each spawn a single-node 4-disk RustFS
 # server and manipulate its disk directories at runtime (crates/e2e_test:
-# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13). They
+# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13, and
+# replacement_privileged_e2e_test when explicitly run as root on Linux). They
 # are correct in isolation but resource-heavy; serialize them under nextest's
 # process boundary (serial_test's #[serial] does not cross it) so several 4-disk
 # servers never run at once. ci-7's nightly picks these up via the e2e suite;
@@ -54,6 +57,20 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
 test-group = 'ecstore-serial-flaky'
 
+# The production-handler relocation regression builds an isolated 8-disk,
+# 2-pool store and commits a 72 MiB multipart object. Keep that cross-disk IO
+# from overlapping the ecstore commit fixtures above.
+[[profile.default.overrides]]
+filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
+test-group = 'ecstore-serial-flaky'
+
+# Embedded integration-test binaries discover an ephemeral port and release
+# the probe listener before RustFS binds it. Serialize that cross-process
+# TOCTOU window; retries would only hide real startup failures.
+[[profile.default.overrides]]
+filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
+test-group = 'embedded-test-ports'
+
 # Serialize the durable manual-transition checkpoint test across nextest's
 # process boundary; it mutates bucket lifecycle metadata and is not quarantined.
 [[profile.default.overrides]]
@@ -74,13 +91,19 @@ test-group = 'ecstore-serial-flaky'
 # e2e-reliability test-group note above). The matching ci-profile override is at
 # the end of the file, after [profile.ci] is declared.
 [[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
+filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
 test-group = 'e2e-reliability'
 
 [[profile.default.overrides]]
 filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
 test-group = 'e2e-inline-boundaries'
 
+# Vault KMS tests share the fixed dev-server port 8200. serial_test's #[serial]
+# does not cross nextest process boundaries, so keep these tests in one group.
+[[profile.default.overrides]]
+filter = 'package(e2e_test) & test(/^kms::kms_vault_test::/)'
+test-group = 'e2e-vault'
+
 # ---------------------------------------------------------------------------
 # ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
 # ---------------------------------------------------------------------------
@@ -133,7 +156,7 @@ retries = 2
 # quarantine: no retries, just single-threaded so several 4-disk servers never
 # run concurrently when ci-7's nightly runs the full e2e suite.
 [[profile.ci.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
+filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
 test-group = 'e2e-reliability'
 
 # Serialize the multipart crash-consistency scenarios under the ci profile too
@@ -143,6 +166,16 @@ test-group = 'e2e-reliability'
 filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
 test-group = 'ecstore-serial-flaky'
 
+[[profile.ci.overrides]]
+filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
+test-group = 'ecstore-serial-flaky'
+
+# Match the default-profile embedded test isolation without quarantining or
+# retrying failures in CI.
+[[profile.ci.overrides]]
+filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
+test-group = 'embedded-test-ports'
+
 # Serialize the durable manual-transition checkpoint test under the ci profile
 # too. No retries: failures stay visible.
 [[profile.ci.overrides]]
@@ -186,7 +219,7 @@ test-group = 'ecstore-serial-flaky'
 # the nightly profile derives its set as "the replication module MINUS this
 # allowlist", so any new replication test lands in nightly by default (never
 # silently unrun) until it is explicitly blessed as fast here. Keep the two
-# regexes byte-identical. Count invariant: 20 here + 28 nightly = 48 total
+# regexes byte-identical. Count invariant: 20 here + 49 nightly = 69 total
 # (authority: `cargo nextest list`; docs/testing/e2e-suite-inventory.md).
 # HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
 # (#4724) because they set a loopback (127.0.0.1) replication target that the
@@ -222,7 +255,7 @@ test-group = 'ecstore-serial-flaky'
 [profile.e2e-smoke]
 default-filter = """
     package(e2e_test) & (
-        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
+        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
         | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
         | test(/^reliant::lifecycle::/)
         | test(/^reliant::tiering::/)
@@ -248,10 +281,12 @@ slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
 # tests that are unfit for the per-PR e2e-smoke gate:
 #
 #   * 2 remote-target TLS validation tests.
-#   * 12 bucket-replication data-plane/helper tests — they PUT/delete objects
-#     and poll until source and target converge; two replicate over HTTPS, two
-#     pin active SSE failure contracts, and one guards event/history observers.
-#     The SSE-S3 contract remains ignored under backlog#1291.
+#   * 15 bucket-replication data-plane/helper tests — they PUT/delete objects
+#     and poll until source and target converge; two replicate over HTTPS,
+#     six pin SSE replication contracts (managed SSE-S3/SSE-KMS re-encrypt on
+#     the target incl. multipart and the resync path, SSE-C and
+#     target-without-KMS stay fail-closed), and one guards event/history
+#     observers.
 #   * 12 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
 #     servers and drives the cross-process site-replication control plane.
 #   * 1 `_real_three_node` site-replication test.
@@ -310,16 +345,16 @@ path = "junit.xml"
 #     object_lambda) — too heavy for the merge budget; they run in ci-7's
 #     nightly 4-node lane.
 #   * replication_extension_test — repl-1 already splits it into the PR
-#     `e2e-smoke` (20 fast) and `e2e-repl-nightly` (27 slow) lanes and reserves
+#     `e2e-smoke` (20 fast) and `e2e-repl-nightly` (49 slow) lanes and reserves
 #     it for those, so e2e-full does not double-run it.
 #   * #[ignore]d tests — nextest skips them by default (no --run-ignored); the
 #     manual-localhost:9000 reliant/policy tests are ci-13's migration.
 #
 # Each e2e test spawns its own single-node rustfs server on a random port with
 # an isolated temp dir (crates/e2e_test/src/common.rs), so the set is
-# parallel-safe — the same property e2e-smoke relies on. The exception is the
-# 4-disk reliability / degraded-read fault-injection tests, serialized below
-# (identical to the ci profile) so several 4-disk servers never run at once.
+# parallel-safe — the same property e2e-smoke relies on. The exceptions are the
+# 4-disk reliability / degraded-read fault-injection tests and the fixed-port
+# Vault tests, both serialized below.
 # KNOWN-FAILURE EXCLUSIONS (characterization run 29381309848, 2026-07-15:
 # 341 ran / 32 failed on the suites' first automated run ever). Deterministic
 # product failures cannot be quarantined away with retries, so each family is
@@ -349,9 +384,13 @@ path = "junit.xml"
 # quarantine: no retries, just single-threaded so several 4-disk servers never
 # run concurrently.
 [[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
+filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
 test-group = 'e2e-reliability'
 
 [[profile.e2e-full.overrides]]
 filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
 test-group = 'e2e-inline-boundaries'
+
+[[profile.e2e-full.overrides]]
+filter = 'package(e2e_test) & test(/^kms::kms_vault_test::/)'
+test-group = 'e2e-vault'

.docker/observability/README.md

@@ -170,6 +170,10 @@ Important behavior notes:
 
 - Logs and metrics usually appear during startup, so seeing those two signals
   first is expected.
+- The OpenTelemetry bridge sends `tracing` fields as log attributes. Loki stores
+  those attributes as structured metadata, and the Collector also mirrors the
+  common troubleshooting fields into the log line so simple line filters can
+  find them.
 - Visible trace data usually requires real HTTP/S3/gRPC request traffic after
   startup, because request-path spans are created on demand.
 - `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
@@ -195,6 +199,17 @@ curl -I http://127.0.0.1:9000/health/ready
 # Jaeger:  http://localhost:16686
 ```
 
+For a structured RustFS log such as an inter-node RPC authentication failure,
+the Loki line now includes fields such as `event`, `component`, `subsystem`,
+`failure_reason`, `rpc_service`, `rpc_method`, and `expected_audience`. Useful
+LogQL checks:
+
+```logql
+{service_name="RustFS"} |= "RPC signature verification failed"
+{service_name="RustFS"} |= "failure_reason="
+{service_name="RustFS"} | failure_reason != ""
+```
+
 If logs and metrics are present but traces are sparse, the most common cause is
 "no real request traffic yet" or "`info` level filtered nested spans", not an
 OTLP routing failure.

.docker/observability/README_ZH.md

@@ -169,6 +169,7 @@ RustFS 会自动在该基础 URL 后补全：
 需要注意：
 
 - 启动阶段通常会先看到日志和指标，因此“先有日志/指标、后有 trace”是正常现象。
+- OpenTelemetry bridge 会把 `tracing` 字段作为日志 attributes 发送。Loki 会将这些 attributes 存为 structured metadata，同时 Collector 会把常用排障字段镜像进日志行，方便用简单的行内容过滤直接查到。
 - 可见的 trace 数据通常依赖启动后的真实 HTTP/S3/gRPC 请求流量，因为请求路径上的 span 是按需创建的。
 - `RUSTFS_OBS_LOGGER_LEVEL=info` 会保留顶层请求 span，但会过滤掉很多 `debug` 级别的嵌套 span。
   如果 Tempo 或 Jaeger 中的 trace 看起来很稀疏，建议先改成 `RUSTFS_OBS_LOGGER_LEVEL=debug`，再判断是否是 collector 或 Tempo 问题。
@@ -192,6 +193,14 @@ curl -I http://127.0.0.1:9000/health/ready
 # Jaeger:  http://localhost:16686
 ```
 
+对于 RustFS 结构化日志，例如节点间 RPC 鉴权失败，Loki 日志行现在会包含 `event`、`component`、`subsystem`、`failure_reason`、`rpc_service`、`rpc_method`、`expected_audience` 等字段。常用 LogQL 检查：
+
+```logql
+{service_name="RustFS"} |= "RPC signature verification failed"
+{service_name="RustFS"} |= "failure_reason="
+{service_name="RustFS"} | failure_reason != ""
+```
+
 如果日志和指标已经正常，但 trace 仍然稀疏，最常见的原因通常是
 “还没有真实请求流量”或“`info` 级别过滤了嵌套 span”，而不是 OTLP 路由失败。
 

.docker/observability/otel-collector-config.yaml

@@ -29,11 +29,27 @@ processors:
     limit_mib: 1024
     spike_limit_mib: 256
   transform/logs:
+    error_mode: ignore
     log_statements:
       - context: log
         statements:
-          - set(attributes["message"], body.string)
-          - set(attributes["log.body"], body.string)
+          - set(attributes["message"], body.string) where IsString(body)
+          - set(attributes["log.body"], body.string) where IsString(body)
+          - set(body, Concat([body, " event=", attributes["event"]], "")) where IsString(body) and attributes["event"] != nil
+          - set(body, Concat([body, " component=", attributes["component"]], "")) where IsString(body) and attributes["component"] != nil
+          - set(body, Concat([body, " subsystem=", attributes["subsystem"]], "")) where IsString(body) and attributes["subsystem"] != nil
+          - set(body, Concat([body, " state=", attributes["state"]], "")) where IsString(body) and attributes["state"] != nil
+          - set(body, Concat([body, " result=", attributes["result"]], "")) where IsString(body) and attributes["result"] != nil
+          - set(body, Concat([body, " reason=", attributes["reason"]], "")) where IsString(body) and attributes["reason"] != nil
+          - set(body, Concat([body, " failure_reason=", attributes["failure_reason"]], "")) where IsString(body) and attributes["failure_reason"] != nil
+          - set(body, Concat([body, " rpc_path=", attributes["rpc_path"]], "")) where IsString(body) and attributes["rpc_path"] != nil
+          - set(body, Concat([body, " rpc_service=", attributes["rpc_service"]], "")) where IsString(body) and attributes["rpc_service"] != nil
+          - set(body, Concat([body, " rpc_method=", attributes["rpc_method"]], "")) where IsString(body) and attributes["rpc_method"] != nil
+          - set(body, Concat([body, " expected_audience=", attributes["expected_audience"]], "")) where IsString(body) and attributes["expected_audience"] != nil
+          - set(body, Concat([body, " peer_addr=", attributes["peer_addr"]], "")) where IsString(body) and attributes["peer_addr"] != nil
+          - set(body, Concat([body, " replay_scope_bootstrap_allowed=", attributes["replay_scope_bootstrap_allowed"]], "")) where IsString(body) and attributes["replay_scope_bootstrap_allowed"] != nil
+          - set(body, Concat([body, " error=", attributes["error"]], "")) where IsString(body) and attributes["error"] != nil
+          - set(body, Concat([body, " exception_message=", attributes["exception.message"]], "")) where IsString(body) and attributes["exception.message"] != nil
 
 exporters:
   otlp/tempo:

.docker/observability/prometheus-rules/rustfs-kms-alerts.yml

@@ -17,9 +17,11 @@
 # =============================================================================
 #
 # Metric source: the KMS operation-policy choke point in
-# crates/kms/src/policy.rs. All label values are bounded static strings
-# (operation, op_class, outcome, error_class, backend, scope); key identifiers,
-# key material, and tokens never appear in labels.
+# crates/kms/src/policy.rs, except KmsKeyRotationOverdue, which reads the
+# label-less key-lifecycle gauge published by the deletion worker's sweep
+# (crates/kms/src/deletion_worker.rs). All label values are bounded static
+# strings (operation, op_class, outcome, error_class, backend, scope); key
+# identifiers, key material, and tokens never appear in labels.
 #
 # Response procedures: docs/operations/kms-observability-runbook.md
 #
@@ -212,3 +214,38 @@ groups:
             circuit_open until the half-open probe succeeds or returns
             a non-retryable failure.
           runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendcircuitopen"
+
+      # ------------------------------------------------------------------
+      # 7. KmsKeyRotationOverdue
+      #    The least recently rotated usable key has gone more than 400
+      #    days without a rotation (measured from creation for keys with
+      #    no recorded rotation). Direct gauge state published by the
+      #    deletion worker's sweep, so no traffic guard applies; the
+      #    one-hour hold only bridges scrape gaps. The worker runs only
+      #    on backends with the schedule_deletion capability, so on the
+      #    Static backend the series never exists and this alert cannot
+      #    fire — that backend cannot rotate either; see the rotation
+      #    driver matrix in docs/operations/kms-backend-security.md.
+      #    Threshold: 400 days — conservative default sitting above a
+      #    one-year rotation policy. Align it with the rotation period
+      #    your compliance policy requires, and with
+      #    RUSTFS_KMS_ROTATION_MAX_AGE_SECS so the per-key rotation_due
+      #    verdict and this aggregate alert agree.
+      # ------------------------------------------------------------------
+      - alert: KmsKeyRotationOverdue
+        expr: |
+          rustfs_kms_oldest_key_rotation_age_seconds > (400 * 86400)
+        for: 1h
+        labels:
+          severity: warning
+          component: kms
+        annotations:
+          summary: "Oldest KMS key unrotated for more than 400 days"
+          description: >-
+            The least recently rotated usable KMS key was last rotated
+            {{ $value | humanizeDuration }} ago (measured from creation
+            for keys with no recorded rotation). List keys through the
+            admin API and read rotation_due / rotation_due_reason for
+            the per-key verdict; an "unsupported" reason means the
+            backend cannot rotate at all.
+          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmskeyrotationoverdue"

.github/actions/setup/action.yml

@@ -57,7 +57,7 @@ runs:
   using: "composite"
   steps:
     # protobuf-compiler is deliberately absent: the setup-protoc step below
-    # installs 34.1 into the tool cache and prepends it to PATH, so the apt
+    # installs 35.1 into the tool cache and prepends it to PATH, so the apt
     # build (older, and never version-matched) was shadowed on every run and
     # simply never used.
     - name: Install system dependencies (Ubuntu)
@@ -81,7 +81,7 @@ runs:
     - name: Install protoc
       uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
       with:
-        version: "34.1"
+        version: "35.1"
         repo-token: ${{ github.token }}
 
     - name: Install flatc

.github/workflows/ci-docs-only.yml

@@ -102,6 +102,9 @@ jobs:
       - name: Check architecture migration rules
         run: ./scripts/check_architecture_migration_rules.sh
 
+      - name: Check logging guardrails
+        run: ./scripts/check_logging_guardrails.sh
+
       - name: Check tokio io-uring feature guard
         run: ./scripts/check_no_tokio_io_uring.sh
 
@@ -111,6 +114,9 @@ jobs:
       - name: Check body-cache whitelist guard
         run: ./scripts/check_body_cache_whitelist.sh
 
+      - name: Check s3s footprint ratchet
+        run: ./scripts/check_s3s_footprint.sh
+
       - name: Check no planning docs committed
         run: ./scripts/check_no_planning_docs.sh
 

.github/workflows/ci.yml

@@ -137,6 +137,9 @@ jobs:
       - name: Check architecture migration rules
         run: ./scripts/check_architecture_migration_rules.sh
 
+      - name: Check logging guardrails
+        run: ./scripts/check_logging_guardrails.sh
+
       - name: Check tokio io-uring feature guard
         run: ./scripts/check_no_tokio_io_uring.sh
 
@@ -146,6 +149,9 @@ jobs:
       - name: Check body-cache whitelist guard
         run: ./scripts/check_body_cache_whitelist.sh
 
+      - name: Check s3s footprint ratchet
+        run: ./scripts/check_s3s_footprint.sh
+
       - name: Check no planning docs committed
         run: ./scripts/check_no_planning_docs.sh
 
@@ -764,6 +770,32 @@ jobs:
           cache-save-if: 'false'
           install-build-packaging-tools: 'false'
 
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: "3.12"
+
+      - name: Install awscurl
+        run: |
+          python3 -m pip install --user --upgrade pip "awscurl==0.44"
+          echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
+
+      - name: Verify awscurl
+        run: test -x "$AWSCURL_PATH"
+
+      - name: Install Vault
+        run: |
+          VAULT_VERSION="1.17.6"
+          VAULT_ARCHIVE="vault_${VAULT_VERSION}_linux_amd64.zip"
+          curl -fsSLo "$RUNNER_TEMP/$VAULT_ARCHIVE" "https://releases.hashicorp.com/vault/${VAULT_VERSION}/${VAULT_ARCHIVE}"
+          echo "0cddc1fbbb88583b5ba5b845f9f8fae47c6fb39a6d48cd543c6ba6fd3ac1a669  $RUNNER_TEMP/$VAULT_ARCHIVE" | sha256sum --check --status
+          unzip -q "$RUNNER_TEMP/$VAULT_ARCHIVE" -d "$RUNNER_TEMP/vault-bin"
+          echo "RUSTFS_TEST_VAULT_BIN=$RUNNER_TEMP/vault-bin/vault" >> "$GITHUB_ENV"
+
+      - name: Verify Vault
+        run: |
+          "$RUSTFS_TEST_VAULT_BIN" version
+
       # Download after the cache restore so the freshly built binary from the
       # build job always wins over anything restored into target/debug.
       - name: Download debug binary

.github/workflows/e2e-replication-nightly.yml

@@ -14,25 +14,24 @@
 
 # Nightly full replication e2e lane (backlog#1147 repl-1, deps: ci-4).
 #
-# The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the 20
-# FAST replication tests. This scheduled lane runs the remaining 27
-# heavier replication e2e tests that are unfit for a per-PR gate:
-#
-#   * 2 remote-target TLS validation tests.
-#   * 12 bucket-replication data-plane/helper tests (PUT/delete + poll for
-#     convergence; two replicate over HTTPS, two pin active SSE failure
-#     contracts, and one guards event/history observers). The SSE-S3 contract
-#     remains ignored under backlog#1291.
-#   * 11 `_real_dual_node` site-replication tests (each spawns TWO rustfs
-#     servers and drives the cross-process site-replication control plane).
-#   * 1 `_real_three_node` site-replication test.
-#   * 1 `_real_single_node` service-account round-trip test.
+# The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the
+# FAST replication tests. This scheduled lane runs the remaining heavier
+# replication e2e tests that are unfit for a per-PR gate: remote-target TLS
+# validation, bucket-replication data-plane/helper tests (PUT/delete + poll
+# for convergence, HTTPS targets, active SSE failure contracts, event/history
+# observers), and the `_real_dual_node` / `_real_three_node` /
+# `_real_single_node` site-replication tests that each spawn full rustfs
+# server processes.
 #
 # The selection is the [profile.e2e-repl-nightly] default-filter in
 # .config/nextest.toml — the single wiring mechanism (repl-1 / ci-4). Do NOT
-# add ad-hoc cargo-test steps here; change the filterset instead.
+# add ad-hoc cargo-test steps here; change the filterset instead. The
+# authoritative membership and count come from
+# `cargo nextest list -p e2e_test --profile e2e-repl-nightly`; the PR/nightly
+# count invariant is maintained next to the filtersets in .config/nextest.toml
+# (deliberately not duplicated here).
 #
-# Explicit division of labor: these 27 tests run ONLY here, never double-run
+# Explicit division of labor: the nightly subset runs ONLY here, never double-run
 # in ci-5's future e2e-full merge gate. TODO(ci-7): once the ci domain's
 # consolidated scheduled e2e workflow exists, fold this interim repl-owned lane
 # into it rather than growing a second scheduled entrypoint.

.github/workflows/nightly-gnu.yml

@@ -55,3 +55,142 @@ jobs:
 
       - name: Build RustFS
         run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins
+
+  # Live-Vault lane for the rustfs-kms suite (rustfs/backlog#1774).
+  #
+  # RUSTFS_KMS_VAULT_TOKEN is the single switch that adds the Vault KV2 and
+  # Vault Transit backends to every for_each_backend spec in
+  # crates/kms/tests/behavior_*.rs (see crates/kms/AGENTS.md). rotate and
+  # versioning are advertised only by the Vault backends, so without this lane
+  # no CI run ever asserts the working half of behavior_rotation.rs — a
+  # rotation that silently dropped historical key versions would stay green.
+  # The same lane runs the dev-Vault #[ignore] tests and the two self-hosting
+  # live scripts (AppRole login, three-node Raft leader failover).
+  #
+  # GitHub-hosted ubuntu-latest, deliberately not the self-hosted sm-standard
+  # fleet: the HA failover script needs a working Docker daemon, and the
+  # self-hosted fleet is heterogeneous — a docker-dependent workflow has been
+  # burned by it before (see the banner in e2e-s3tests.yml, rustfs/backlog#1149).
+  kms-vault-lane:
+    name: KMS live Vault lane
+    runs-on: ubuntu-latest
+    timeout-minutes: 90
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+      # Root token of the ephemeral loopback dev server. Not a secret: the
+      # server lives only for this job, listens on 127.0.0.1, and holds only
+      # keys the tests create. The literal value matters — the dev-Vault
+      # #[ignore] fixtures in crates/kms/src/backends/vault.rs hardcode it.
+      VAULT_LANE_TOKEN: dev-only-token
+      VAULT_LANE_ADDR: http://127.0.0.1:8200
+      # Keeps a runner-level proxy from swallowing the loopback dev-server
+      # traffic (see crates/kms/AGENTS.md). Actions env keys are
+      # case-insensitive, so only the uppercase form is set; reqwest reads
+      # either casing.
+      NO_PROXY: 127.0.0.1,localhost
+    steps:
+      - name: Checkout main branch
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+        with:
+          persist-credentials: false
+          ref: main
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          # Dedicated key: rust-cache cannot tell runner images apart, so
+          # sharing a key with an sm-standard lane would let two different
+          # system images overwrite each other's artifacts (same reasoning as
+          # ci.yml's ci-uring lane). Saved from this nightly job itself so the
+          # next night starts warm.
+          cache-shared-key: kms-vault-lane
+          cache-save-if: 'true'
+          install-build-packaging-tools: 'false'
+          install-test-tools: 'false'
+
+      - name: Install Vault CLI
+        run: |
+          set -euo pipefail
+          wget -qO- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
+          echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list >/dev/null
+          sudo apt-get update -qq
+          sudo apt-get install -y -qq vault
+          vault version
+
+      - name: Start Vault dev server with KV2 and Transit engines
+        run: |
+          set -euo pipefail
+          nohup vault server -dev \
+            -dev-root-token-id="${VAULT_LANE_TOKEN}" \
+            -dev-listen-address=127.0.0.1:8200 >/tmp/vault-dev.log 2>&1 &
+          for _ in $(seq 1 60); do
+            if curl -fsS "${VAULT_LANE_ADDR}/v1/sys/health" >/dev/null 2>&1; then
+              break
+            fi
+            sleep 1
+          done
+          curl -fsS "${VAULT_LANE_ADDR}/v1/sys/health"
+          export VAULT_ADDR="${VAULT_LANE_ADDR}" VAULT_TOKEN="${VAULT_LANE_TOKEN}"
+          # Dev mode mounts KV v2 at secret/ by default; Transit is explicit.
+          # Prove both engines actually work rather than assuming the defaults.
+          vault secrets enable transit
+          vault kv put secret/rustfs-ci-lane-probe value=ok >/dev/null
+          vault kv get secret/rustfs-ci-lane-probe >/dev/null
+          vault write -f transit/keys/rustfs-ci-lane-probe >/dev/null
+
+      - name: Run rustfs-kms suite with the Vault lane on
+        env:
+          RUSTFS_KMS_VAULT_TOKEN: ${{ env.VAULT_LANE_TOKEN }}
+          RUSTFS_KMS_VAULT_ADDR: ${{ env.VAULT_LANE_ADDR }}
+        run: cargo test -p rustfs-kms --locked
+
+      - name: Run dev-Vault ignored tests
+        env:
+          RUSTFS_KMS_VAULT_TOKEN: ${{ env.VAULT_LANE_TOKEN }}
+          RUSTFS_KMS_VAULT_ADDR: ${{ env.VAULT_LANE_ADDR }}
+        # Filters select the dev-Vault-only #[ignore] tests. The AWS #[ignore]
+        # tests (backends::aws, service_manager) stay excluded — they need real
+        # AWS credentials and create billable keys. The AppRole and HA #[ignore]
+        # tests are excluded here because their own scripts below provision the
+        # Vault topology they need.
+        run: |
+          set -euo pipefail
+          cargo test -p rustfs-kms --locked --lib backends::contract_tests -- --ignored
+          cargo test -p rustfs-kms --locked --lib backends::vault -- --ignored
+          cargo test -p rustfs-kms --locked --test vault_fault_injection -- --ignored
+
+      - name: Run AppRole live checks (self-hosting ephemeral Vault)
+        run: bash scripts/test/vault_approle_kms_live.sh
+
+      - name: Show Vault dev server log on failure
+        if: failure()
+        run: tail -n 200 /tmp/vault-dev.log || true
+
+  # Three-node Raft leader failover (crates/kms/tests/vault_ha_failover_live.rs,
+  # first validated by rustfs/rustfs#5653). Its own job so an election-timing
+  # flake cannot mask the main lane's verdict, and vice versa. The script
+  # provisions and tears down its own Docker cluster.
+  kms-vault-ha-failover:
+    name: KMS Vault HA failover lane
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+      NO_PROXY: 127.0.0.1,localhost
+    steps:
+      - name: Checkout main branch
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+        with:
+          persist-credentials: false
+          ref: main
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          cache-shared-key: kms-vault-lane
+          cache-save-if: 'false'
+          install-build-packaging-tools: 'false'
+          install-test-tools: 'false'
+
+      - name: Run HA leader failover live checks (three-node Raft cluster in Docker)
+        run: bash scripts/test/vault_ha_kms_live.sh

.github/workflows/package.yml

@@ -0,0 +1,477 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Package Workflow - Build DEB/RPM packages
+#
+# This workflow builds DEB and RPM packages from pre-built Linux binaries
+# and uploads them to Cloudflare R2.
+#
+# Trigger:
+# - release published: automatically package when a GitHub release is published
+# - workflow_dispatch: manual trigger with optional tag/run_id
+#
+# Flow:
+# 1. Find the Build workflow run for the release tag
+# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu) from build artifacts
+# 3. Build DEB packages for amd64 and arm64
+# 4. Build RPM packages for x86_64 and aarch64
+# 5. Upload all packages to Cloudflare R2
+
+name: Package DEB/RPM
+
+permissions:
+  contents: read
+  actions: read
+
+on:
+  release:
+    types: [ published ]
+  workflow_dispatch:
+    inputs:
+      tag:
+        description: "Release tag to package (e.g. 1.0.0-beta.12). Leave empty for latest main build."
+        required: false
+        type: string
+      build_run_id:
+        description: "Build workflow run ID (overrides tag lookup)"
+        required: false
+        type: string
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.release.tag_name || github.event.inputs.tag || github.run_id }}
+  cancel-in-progress: true
+
+jobs:
+  # Resolve which build run to use and extract version info
+  resolve:
+    name: Resolve Build
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    outputs:
+      version: ${{ steps.resolve.outputs.version }}
+      build_type: ${{ steps.resolve.outputs.build_type }}
+      build_run_id: ${{ steps.resolve.outputs.build_run_id }}
+      tag: ${{ steps.resolve.outputs.tag }}
+    steps:
+      - name: Resolve build run
+        id: resolve
+        shell: bash
+        env:
+          GH_TOKEN: ${{ github.token }}
+          INPUT_TAG: ${{ github.event.inputs.tag }}
+          INPUT_RUN_ID: ${{ github.event.inputs.build_run_id }}
+        run: |
+          set -euo pipefail
+
+          # Determine tag
+          if [[ "${{ github.event_name }}" == "release" ]]; then
+            TAG="${{ github.event.release.tag_name }}"
+          elif [[ -n "$INPUT_TAG" ]]; then
+            TAG="$INPUT_TAG"
+          else
+            TAG=""
+          fi
+
+          echo "Tag: ${TAG:-<none>}"
+
+          # Determine build run ID
+          BUILD_RUN_ID=""
+
+          if [[ -n "$INPUT_RUN_ID" ]]; then
+            # Explicit run ID takes priority
+            BUILD_RUN_ID="$INPUT_RUN_ID"
+            echo "Using explicit build run ID: $BUILD_RUN_ID"
+
+          elif [[ -n "$TAG" ]]; then
+            # Find the build run that produced this tag
+            echo "Looking for build run for tag: $TAG"
+            BUILD_RUN_ID=$(gh api \
+              "repos/${{ github.repository }}/actions/workflows/build.yml/runs?branch=${TAG}&status=success&per_page=1" \
+              --jq '.workflow_runs[0].id' 2>/dev/null || echo "")
+
+            if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
+              # Tag might not be a branch; try event=push with head_branch matching
+              BUILD_RUN_ID=$(gh api \
+                "repos/${{ github.repository }}/actions/workflows/build.yml/runs?event=push&status=success&per_page=100" \
+                --jq ".workflow_runs[] | select(.head_branch == \"$TAG\") | .id" 2>/dev/null | head -1 || echo "")
+            fi
+
+            if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
+              echo "❌ No successful build run found for tag: $TAG"
+              exit 1
+            fi
+            echo "Found build run: $BUILD_RUN_ID"
+
+          else
+            # No tag — latest successful main build
+            echo "No tag specified, looking for latest main build"
+            BUILD_RUN_ID=$(gh api \
+              "repos/${{ github.repository }}/actions/workflows/build.yml/runs?branch=main&status=success&per_page=1" \
+              --jq '.workflow_runs[0].id' 2>/dev/null || echo "")
+
+            if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
+              echo "❌ No successful main build found"
+              exit 1
+            fi
+            echo "Latest main build: $BUILD_RUN_ID"
+          fi
+
+          # Determine version and build type
+          if [[ -n "$TAG" ]]; then
+            VERSION="$TAG"
+            if [[ "$TAG" == *"-preview"* ]]; then
+              BUILD_TYPE="preview"
+            elif [[ "$TAG" == *"alpha"* || "$TAG" == *"beta"* || "$TAG" == *"rc"* ]]; then
+              BUILD_TYPE="prerelease"
+            else
+              BUILD_TYPE="release"
+            fi
+          else
+            SHORT_SHA=$(gh api "repos/${{ github.repository }}/actions/runs/${BUILD_RUN_ID}" \
+              --jq '.head_sha' 2>/dev/null | head -c 7)
+            VERSION="dev-${SHORT_SHA}"
+            BUILD_TYPE="development"
+          fi
+
+          {
+            echo "version=$VERSION"
+            echo "build_type=$BUILD_TYPE"
+            echo "build_run_id=$BUILD_RUN_ID"
+            echo "tag=${TAG}"
+          } >> "$GITHUB_OUTPUT"
+
+          echo "📊 Resolved:"
+          echo "  Version: $VERSION"
+          echo "  Build type: $BUILD_TYPE"
+          echo "  Build run ID: $BUILD_RUN_ID"
+
+  # Build DEB and RPM packages for each architecture
+  package:
+    name: Package (${{ matrix.arch }})
+    needs: resolve
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - arch: x86_64
+            deb_arch: amd64
+            rpm_arch: x86_64
+            artifact_name: "rustfs-linux-x86_64-gnu"
+          - arch: aarch64
+            deb_arch: arm64
+            rpm_arch: aarch64
+            artifact_name: "rustfs-linux-aarch64-gnu"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+        with:
+          persist-credentials: false
+
+      - name: Download binary artifact from build run
+        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
+        with:
+          pattern: ${{ matrix.artifact_name }}*
+          path: ./binary-artifact
+          run-id: ${{ needs.resolve.outputs.build_run_id }}
+          github-token: ${{ github.token }}
+          merge-multiple: true
+
+      - name: Extract binary
+        id: binary
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
+          if [[ -z "$ZIP_FILE" ]]; then
+            echo "❌ No binary artifact found"
+            ls -la ./binary-artifact/ || true
+            exit 1
+          fi
+
+          echo "Found artifact: $ZIP_FILE"
+
+          mkdir -p ./bin
+          unzip -o "$ZIP_FILE" -d ./bin
+
+          if [[ ! -f ./bin/rustfs ]]; then
+            echo "❌ rustfs binary not found in archive"
+            exit 1
+          fi
+
+          chmod +x ./bin/rustfs
+          ls -lh ./bin/rustfs
+          echo "✅ Binary extracted"
+
+      - name: Build DEB package
+        id: deb
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          VERSION="${{ needs.resolve.outputs.version }}"
+          DEB_ARCH="${{ matrix.deb_arch }}"
+          # DEB version: replace - with ~ (1.0.0-beta.12 -> 1.0.0~beta.12)
+          # Use a variable for ~ to prevent tilde expansion by bash
+          TILDE='~'
+          DEB_VERSION="${VERSION/-/$TILDE}"
+          PKG_DIR="rustfs_${DEB_VERSION}_${DEB_ARCH}"
+
+          echo "Building DEB: ${PKG_DIR}.deb"
+
+          mkdir -p "${PKG_DIR}/DEBIAN"
+          mkdir -p "${PKG_DIR}/usr/bin"
+          mkdir -p "${PKG_DIR}/etc/default"
+          mkdir -p "${PKG_DIR}/lib/systemd/system"
+          mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
+
+          cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
+          chmod 755 "${PKG_DIR}/usr/bin/rustfs"
+
+          cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
+
+          cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
+          # RustFS Environment Configuration
+          # See https://rustfs.com/docs/ for more information
+          # RUSTFS_VOLUMES=""
+          # RUSTFS_ROOT_USER=""
+          # RUSTFS_ROOT_PASSWORD=""
+          ENVEOF
+
+          cat > "${PKG_DIR}/DEBIAN/control" << EOF
+          Package: rustfs
+          Version: ${DEB_VERSION}
+          Section: utils
+          Priority: optional
+          Architecture: ${DEB_ARCH}
+          Depends: libc6 (>= 2.31)
+          Maintainer: RustFS Team <support@rustfs.com>
+          Description: High-performance distributed object storage
+           RustFS is a high-performance distributed object storage software
+           built using Rust. It is compatible with MinIO and S3 API.
+          Homepage: https://rustfs.com
+          EOF
+
+          cat > "${PKG_DIR}/DEBIAN/postinst" << 'POSTINST'
+          #!/bin/bash
+          set -e
+          if ! getent passwd rustfs > /dev/null 2>&1; then
+              useradd -r -s /bin/false -d /opt/rustfs rustfs
+          fi
+          mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
+          chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
+          if [ -d /run/systemd/system ]; then
+              systemctl daemon-reload
+          fi
+          echo "RustFS installed. Configure /etc/default/rustfs then: systemctl start rustfs"
+          POSTINST
+          chmod 755 "${PKG_DIR}/DEBIAN/postinst"
+
+          cat > "${PKG_DIR}/DEBIAN/prerm" << 'PRERM'
+          #!/bin/bash
+          set -e
+          if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
+              systemctl stop rustfs
+          fi
+          PRERM
+          chmod 755 "${PKG_DIR}/DEBIAN/prerm"
+
+          cat > "${PKG_DIR}/DEBIAN/postrm" << 'POSTRM'
+          #!/bin/bash
+          set -e
+          if [ -d /run/systemd/system ]; then
+              systemctl daemon-reload
+          fi
+          POSTRM
+          chmod 755 "${PKG_DIR}/DEBIAN/postrm"
+
+          cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
+          cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
+
+          fakeroot dpkg-deb --build "${PKG_DIR}"
+
+          DEB_FILE="${PKG_DIR}.deb"
+          ls -lh "$DEB_FILE"
+          echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
+          echo "✅ DEB built: $DEB_FILE"
+
+      - name: Build RPM package
+        id: rpm
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          VERSION="${{ needs.resolve.outputs.version }}"
+          RPM_ARCH="${{ matrix.rpm_arch }}"
+
+          echo "Building RPM for ${RPM_ARCH}"
+
+          sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential
+          sudo gem install fpm
+
+          # Create config file for fpm (DEB build creates it in its package dir structure,
+          # but fpm needs the file to exist before packaging)
+          mkdir -p ./tmp-pkg/etc/default
+          cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
+          # RustFS Environment Configuration
+          # See https://rustfs.com/docs/ for more information
+          # RUSTFS_VOLUMES=""
+          # RUSTFS_ROOT_USER=""
+          # RUSTFS_ROOT_PASSWORD=""
+          ENVEOF
+
+          fpm -s dir -t rpm \
+            --name rustfs \
+            --version "$VERSION" \
+            --architecture "$RPM_ARCH" \
+            --depends "glibc >= 2.31" \
+            --maintainer "RustFS Team <support@rustfs.com>" \
+            --description "High-performance distributed object storage" \
+            --url "https://rustfs.com" \
+            --license "Apache-2.0" \
+            --after-install <(cat <<'POSTINST'
+          #!/bin/bash
+          set -e
+          if ! getent passwd rustfs > /dev/null 2>&1; then
+              useradd -r -s /bin/false -d /opt/rustfs rustfs
+          fi
+          mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
+          chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
+          if [ -d /run/systemd/system ]; then
+              systemctl daemon-reload
+          fi
+          POSTINST
+            ) \
+            --before-remove <(cat <<'PRERM'
+          #!/bin/bash
+          set -e
+          if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
+              systemctl stop rustfs
+          fi
+          PRERM
+            ) \
+            --after-remove <(cat <<'POSTRM'
+          #!/bin/bash
+          set -e
+          if [ -d /run/systemd/system ]; then
+              systemctl daemon-reload
+          fi
+          POSTRM
+            ) \
+            --config-files /etc/default/rustfs \
+            ./bin/rustfs=/usr/bin/rustfs \
+            ./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
+            deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
+            LICENSE=/usr/share/doc/rustfs/LICENSE \
+            README.md=/usr/share/doc/rustfs/README.md
+
+          RPM_FILE=$(ls -1 rustfs-*.rpm 2>/dev/null | head -1)
+          if [[ -z "$RPM_FILE" ]]; then
+            echo "❌ RPM build failed"
+            exit 1
+          fi
+
+          ls -lh "$RPM_FILE"
+          echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
+          echo "✅ RPM built: $RPM_FILE"
+
+      - name: Upload packages to artifacts
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: packages-${{ matrix.arch }}
+          path: |
+            *.deb
+            *.rpm
+          retention-days: 30
+
+      - name: Upload packages to Cloudflare R2
+        if: env.R2_ACCESS_KEY_ID != ''
+        env:
+          R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
+          R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
+          R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
+          R2_BUCKET: ${{ secrets.R2_BUCKET }}
+          AWS_EC2_METADATA_DISABLED: true
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
+            echo "⚠️ R2 credentials missing, skipping upload"
+            exit 0
+          fi
+
+          if ! command -v aws >/dev/null 2>&1; then
+            sudo apt-get update && sudo apt-get install -y awscli
+          fi
+
+          export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
+          export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
+          export AWS_DEFAULT_REGION="auto"
+
+          BUILD_TYPE="${{ needs.resolve.outputs.build_type }}"
+          if [[ "$BUILD_TYPE" == "development" ]]; then
+            R2_PREFIX="artifacts/rustfs/packages/dev"
+          else
+            R2_PREFIX="artifacts/rustfs/packages/release"
+          fi
+          R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
+
+          echo "📤 Uploading to $R2_PATH"
+
+          DEB_FILE="${{ steps.deb.outputs.deb_file }}"
+          RPM_FILE="${{ steps.rpm.outputs.rpm_file }}"
+
+          for f in "$DEB_FILE" "$RPM_FILE"; do
+            if [[ -n "$f" && -f "$f" ]]; then
+              echo "Uploading: $f"
+              aws s3 cp "$f" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
+            fi
+          done
+
+          echo "✅ Upload complete"
+
+          # Also upload as latest for release/prerelease
+          if [[ "$BUILD_TYPE" == "release" || "$BUILD_TYPE" == "prerelease" ]]; then
+            LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
+            for f in "$DEB_FILE" "$RPM_FILE"; do
+              if [[ -n "$f" && -f "$f" ]]; then
+                echo "Uploading latest: $(basename "$f")"
+                aws s3 cp "$f" "$LATEST_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
+              fi
+            done
+            echo "✅ Latest packages updated"
+          fi
+
+  # Summary
+  summary:
+    name: Summary
+    needs: [ resolve, package ]
+    if: always()
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - name: Print summary
+        shell: bash
+        run: |
+          echo "## 📦 Package Summary" >> "$GITHUB_STEP_SUMMARY"
+          echo "" >> "$GITHUB_STEP_SUMMARY"
+          echo "| Item | Value |" >> "$GITHUB_STEP_SUMMARY"
+          echo "|------|-------|" >> "$GITHUB_STEP_SUMMARY"
+          echo "| Version | \`${{ needs.resolve.outputs.version }}\` |" >> "$GITHUB_STEP_SUMMARY"
+          echo "| Build Type | ${{ needs.resolve.outputs.build_type }} |" >> "$GITHUB_STEP_SUMMARY"
+          echo "| Build Run | #${{ needs.resolve.outputs.build_run_id }} |" >> "$GITHUB_STEP_SUMMARY"
+          echo "| Package Status | ${{ needs.package.result }} |" >> "$GITHUB_STEP_SUMMARY"

.github/workflows/windows-filesystem.yml

@@ -69,6 +69,10 @@ jobs:
           install-build-packaging-tools: 'false'
           install-test-tools: 'false'
 
+      - name: Check production Windows dependencies
+        shell: pwsh
+        run: cargo check -p rustfs-ecstore --lib
+
       - name: Test guarded rename publication
         shell: pwsh
         run: cargo test -p rustfs-ecstore --lib rename_all_ -- --nocapture

AGENTS.md

@@ -51,26 +51,25 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 
 ## Change Style for Existing Logic
 
-- Prefer direct, local code over extracting one-off helpers.
-- Extract a helper only when logic is reused or the extraction materially clarifies a non-trivial flow.
+- Start with the smallest direct, local edit. Add production files, types, traits, helpers, wrappers, or abstraction layers only when current behavior requires them. Extraction must remove present duplication, enforce a real boundary, or materially clarify a non-trivial flow; anticipated reuse is not enough.
 - Use Rust's default module file layout (`mod foo;` with `foo.rs` or `foo/mod.rs`/`foo/*.rs`).
   Avoid `#[path = "..."]` for module inclusion; move files into the canonical module tree instead.
   If an unavoidable generated-code, FFI, or test-fixture exception remains, keep it local and document why the canonical layout cannot work.
 - Solve only the requested problem; do not add speculative features, configurability, or adjacent improvements.
 - Prefer editing existing code over rewriting files or reshaping unrelated logic.
-- Modify only what is required and remove only artifacts introduced by your own changes.
+- Modify only what is required. Remove any in-scope path or representation superseded by the change. If compatibility or rollback requires retention, adapt at the boundary to one canonical core and follow the repository's `RUSTFS_COMPAT_TODO` removal policy; never delete unrelated code merely to improve addition/deletion statistics.
 - Preserve the existing control-flow and logic shape when fixing bugs or addressing review comments, especially in init, distributed coordination, locking, metadata, and concurrency paths.
 - Do not refactor existing code only to make it easier to unit test.
 - Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths.
-- Keep code elegant, concise, and direct. Prefer minimal, readable implementations over over-engineering and excessive abstraction. Use comments to clarify non-obvious intent and invariants, not to compensate for unclear code.
-- Do not write comments that narrate what the next line does, restate a signature, or describe the change you just made — that commentary belongs in the PR description, not the code. Required invariant comments — lock ordering, `SAFETY`, unwrap justification, `#[allow(dead_code)]` rationale, `RUSTFS_COMPAT_TODO` — are never narration.
+- Keep code elegant, concise, and direct. Prefer the smallest readable design and existing abstractions over parallel managers, factories, adapters, or wrappers added only to make the design look extensible.
+- Comments state non-obvious reasons, assumptions, and invariants in the shortest complete form. Their length follows the invariant's complexity: `SAFETY`, lock ordering, durability, and compatibility contracts may need a short list of conditions. Never narrate the next line, restate a signature, or record change history; move durable design rationale to architecture or operations documentation.
 - Mention unrelated issues when useful, but do not fix them as part of a narrow task.
 
 ## Reuse Before You Write
 
 Search for an existing implementation before writing a new one; extend what exists instead of duplicating it:
 
-- **Helpers and utilities** (path/string handling, hashing, retry, env parsing, IO wrappers): check `ls crates/utils/src` first — file names map to operations (`retry.rs`, `envs.rs`, `hash.rs`, `path.rs`, `string.rs`, `io.rs`) — plus `crates/common` (shared structures/globals), then `rg -i 'fn \w*<term>' crates/utils/src crates/common/src <touched-crate>/src` for signatures. Helpers are snake_case: a full-text single-word grep over a large crate drowns you and a multi-word phrase returns nothing. Reimplementing an existing workspace helper — or hand-rolling what `std`, `tokio`, or an existing workspace dependency already provides — is a review finding, not a style preference.
+- **Helpers and utilities** (path/string handling, hashing, retry, env parsing, IO wrappers): check `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, and relevant direct workspace dependencies from `Cargo.toml`. Search snake_case signatures with a focused term. Reimplementing an existing workspace helper — or hand-rolling what `std`, `tokio`, or an existing dependency already provides — is a review finding, not a style preference.
 - **Reuse requires matching semantics, not a matching name**: before adopting a helper, check its normalization (`clean` resolves `.`/`..` — never apply it to raw S3 object keys), error type, backoff/deadline behavior, and durability gating against the call site. When semantics differ, a new narrowly-named helper with a comment naming the rejected lookalike is the correct outcome. The inverse also holds: workspace wrappers exist because raw `std`/`tokio` semantics were insufficient (durability gates, retries) — prefer the wrapper over the raw call.
 - **Constants and fixed tokens** (protocol labels, error identifiers, header keys, event names, metric names, command tags): search for existing constants/enums that already represent the same semantic value and reuse them. If a value is truly new, define one local constant near related logic; never scatter the literal across sites. When changing existing behavior, align naming and format with the established constants.
 - **Test scaffolding**: reuse existing test utilities and fixtures (the touched crate's own `test_util` module and `tests/fixtures`, or `crates/test-utils`) instead of writing new setup code — run `rg -l '<fn-under-test>' <crate>/src <crate>/tests` before writing a test. A new test must pin a failure mode no existing test covers. Near-duplicate means same code path AND same poison-value class: this repo's boundary companions (n==max vs max+1, absent vs empty vs nil UUID bytes, MetaObject vs MetaDeleteMarker) are distinct by definition and must all be written.
@@ -79,6 +78,7 @@ Search for an existing implementation before writing a new one; extend what exis
 
 Net-new code — files, types, branches, comments — is cost to justify, not progress:
 
+- Inspect production-code additions separately. Tests, fixtures, generated code, and documentation do not count as production-code growth. Line counts are signals, not quotas: new production structures must map to a current requirement, and a blocker requires a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries.
 - Validate at the trust boundary — untrusted client input, bytes read from disk, RPC payloads, config (see Serde Safety and Cross-Cutting Domain Invariants) — then trust the type: do not re-check what the type system or a validated upstream layer already guarantees, and cite the establishing check (`file:line`) when the guarantee is not obvious.
 - The exception is load-bearing: a value that crossed a persistence, RPC, or version boundary is never guaranteed by the code on the other side — a peer may be older or buggy, disk bytes may be corrupt — so the Cross-Cutting Domain Invariant patterns apply at every consumer, and re-checks immediately before a destructive action (delete, overwrite, quorum decision) stay. Deleting an existing guard is a behavior change requiring adversarial review, not cleanup.
 - Every new branch needs a nameable trigger: a concrete input, state, or failure that reaches it — for boundary-crossing values, corrupt or stale persisted/peer data is always nameable. If you cannot name one, do not write the branch. If the case is truly unreachable, encode the invariant in the type; where that is impossible, return a typed internal error (fail closed). `debug_assert!` is acceptable only for pure internal arithmetic on values that never crossed a disk/RPC/config boundary — never as the sole guard on decoded or peer-supplied data.
@@ -218,9 +218,10 @@ not to bless it.
 
 Pick the tier from the riskiest file touched; when in doubt, pick the higher.
 
-- **Exempt:** docs/comments/instruction-only changes, formatting, typos with
-  no runtime surface. Skip this section.
-- **Mechanical:** pure renames, file moves, test-only or tooling changes —
+- **Exempt:** docs/comments, formatting, and typos that cannot affect runtime,
+  builds, tests, or agent execution. Skip this section.
+- **Mechanical:** pure renames, file moves, test-only or tooling changes, and
+  agent-instruction changes that alter execution —
   correctness and simplicity adversaries only.
 - **Standard (the default):** any change that affects behavior.
 - **High risk:** touches locking, erasure coding, quorum/heal, replication,
@@ -242,7 +243,7 @@ encode this repo's shipped bugs.
 - **Correctness adversary** — construct a concrete input/state/interleaving
   that yields wrong output, data loss, or a crash. Probe error paths and edge
   values (empty, nil UUID, zero-length, quorum−1, missing version).
-- **Simplicity adversary** — same behavior, less code. Hunt the materially smaller or more idiomatic diff (see Change Style for Existing Logic, Reuse Before You Write, and Necessary Code Only): reimplemented workspace helpers, one-caller extractions, rewrites where an in-place edit suffices, defensive branches with no nameable trigger, redundant error wrapping, near-duplicate tests, narration comments. A smaller diff achieving identical behavior is a finding, reported with the concrete replacement; forced reuse of a helper with mismatched semantics is equally a finding.
+- **Simplicity adversary** — same behavior, less code. Hunt reimplemented helpers, rewrites where an in-place edit suffices, speculative abstractions, defensive branches with no nameable trigger, redundant error wrapping, near-duplicate tests, and narration comments. A one-caller helper is a finding only when it merely forwards or splits a short linear flow without adding domain naming, boundary isolation, an invariant, or useful error context. Report a concrete smaller replacement; fewer lines alone are not evidence.
 - **Security reviewer** — authn/authz bypass, injection, secret leakage,
   untrusted deserialization (see Serde Safety), path traversal, timing leaks.
 - **Concurrency/durability reviewer** — lock ordering, races, cancellation,
@@ -253,10 +254,11 @@ encode this repo's shipped bugs.
   time across IO, sync or CPU-heavy work on async runtime threads, added
   fsync/flush outside the durability gate, hot-path logging noise. A
   measurable regression on a per-request or per-object path is a finding.
-- **Test-coverage skeptic** — for each claimed behavior, name the test that
-  fails if the change is reverted; then name a changed line that could be
-  wrong while all tests stay green — if one exists, coverage is insufficient.
-  A missing test is a finding, not a note.
+- **Test-coverage skeptic** — for each testable behavior claim, name the test
+  or executable check that detects a revert; then name a changed line that
+  could be wrong while all checks stay green. If a focused check is not
+  reasonable, require the reason and residual risk from the validation floor.
+  Test additions have no line-count or growth budget.
 
 Standard tier: correctness adversary + simplicity adversary + test-coverage
 skeptic, plus every role whose domain the diff touches (async or
@@ -282,7 +284,9 @@ High risk: all seven roles.
 
 - Every applicable role has run; every finding is fixed or rebutted with
   evidence.
-- Every behavior change has a test that fails without it.
+- Every testable behavior change has a focused regression check. Exceptions
+  follow the validation floor and state why a check is impractical and what
+  risk remains.
 - The Verification Before PR gates pass — adversarial review supplements
   those gates, never replaces them.
 - High risk only: record a one-line verdict per role in the PR description.
@@ -322,6 +326,28 @@ High risk: all seven roles.
 - Use environment variables or vault tooling for sensitive configuration.
 - For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
 
+## Logging
+
+Applies to **every** `tracing` macro you add or edit, including a single line
+added in passing while fixing something else — not only to log-focused changes.
+
+- Fields first, message second: `event`, `component`, `subsystem`,
+  `result`/`state`, then key context. The message is a short label, not a
+  sentence with values interpolated into it.
+- Reuse the existing `EVENT_*` / `LOG_COMPONENT_*` / `LOG_SUBSYSTEM_*`
+  constants of the module you are editing; match the shape of the log sites
+  already in that file rather than introducing a second style next to them.
+- Level policy: `error` for behavior/security-affecting failures, `warn` for
+  degraded or fallback paths, `info` for low-frequency lifecycle, `debug` for
+  targeted diagnostics, `trace` for hot paths. Per-object and per-request
+  success paths are `trace`.
+- Never log secrets, tokens, credential payloads, or merged config dumps.
+- `scripts/check_logging_guardrails.sh` enforces a subset of this on the files
+  it lists; passing it is a floor, not evidence the log matches the house style.
+
+See `.agents/skills/rustfs-logging-governance/SKILL.md` for the full event
+model, level policy, and guardrail-update checklist.
+
 ## Tools
 
 ### xl.meta decode tool Quick Use

ARCHITECTURE.md

@@ -1,6 +1,6 @@
 # ARCHITECTURE.md
 
-> Last updated: 2026-07-02 · Revision: 2
+> Last updated: 2026-08-12 · Revision: 3
 >
 > This document describes the high-level architecture of RustFS.
 > If you want to familiarize yourself with the code base, you are in the right place!
@@ -119,19 +119,44 @@ module split is tracked under `docs/architecture/`.
 
 3. **Each type has exactly one definition.** Types shared across crates must be defined
    in one crate and re-exported or imported by others.
-   - ⚠️ VIOLATED: `ReplicationStats` (4 copies), `LastMinuteLatency` (3 copies),
-     `BackpressureConfig` (3 copies), `DataUsageInfo` (2 copies).
+   - ⚠️ VIOLATED: `ReplicationStats` names three unrelated types
+     (`crates/data-usage/src/data_usage.rs`,
+     `crates/obs/src/metrics/collectors/replication.rs`,
+     `crates/ecstore/src/bucket/replication/replication_state.rs`) — a naming
+     collision, not copies; renaming is tracked in rustfs/backlog#1847.
+   - `LastMinuteLatency` has two deliberately different implementations: the
+     per-second bucketed accumulator in `crates/common/src/last_minute.rs` and
+     the in-memory endpoint-health sample tracker in
+     `crates/ecstore/src/bucket/bucket_target_sys.rs` (its doc comment explains
+     why it stays local).
+   - ✅ RESOLVED: `BackpressureConfig` and `DataUsageInfo` each have exactly one
+     definition (`crates/io-core/src/backpressure.rs`,
+     `crates/data-usage/src/data_usage.rs`). A zero-consumer
+     `BackpressureSettings` copy lingers in `crates/io-metrics/src/config.rs`;
+     its removal is tracked in rustfs/backlog#1833.
 
 4. **ecstore does not know about HTTP or S3 protocol details.** It operates on
    storage-level abstractions (objects, buckets, disks, pools).
+   - ⚠️ VIOLATED: 58 files under `crates/ecstore/src` reference `s3s`
+     (`rg -l 's3s' crates/ecstore/src | wc -l`), `crates/ecstore/src/client/`
+     is a ~9.4K-line embedded S3 HTTP client, and `crates/ecstore/Cargo.toml`
+     depends on `s3s`, `http`, `hyper`/`hyper-util`/`hyper-rustls`, and
+     `reqwest`. Target state: the engine's need to act as an S3 client
+     (tiering, replication targets) is served by an extracted client crate,
+     and ecstore holds no wire or DTO types.
 
 5. **The `rustfs` binary crate is the only place that wires everything together.**
    Individual crates should be testable in isolation.
 
 6. **Error types use `thiserror` with descriptive names** (e.g., `StorageError`,
    not bare `Error`).
-   - ⚠️ VIOLATED: 6 crates use `pub enum Error`; 2 crates use `snafu`;
-      `heal` use `anyhow` in library code.
+   - ✅ RESOLVED (strategy): `snafu` is gone from source
+     (`rg -l snafu crates/ rustfs/` is empty) and library code no longer uses
+     `anyhow` (remaining hits are test code and the `e2e_test` crate; `heal`
+     uses `thiserror`).
+   - ⚠️ VIOLATED (naming): 6 crates still export a bare `pub enum Error`:
+     `crypto`, `filemeta`, `heal`, `iam`, `policy`, and `replication`
+     (`src/resync.rs`) — all `thiserror`-derived.
 
 ## Known Structural Issues
 
@@ -140,13 +165,25 @@ module split is tracked under `docs/architecture/`.
 
 ### Critical
 
-- **common/scanner code duplication (~3K lines).** `scanner` depends on `common`
-  but maintains its own copies of `DataUsageInfo`, `LastMinuteLatency`, and related
-  types instead of importing them.
+- **scanner/data-usage duplicate `.usage-cache.bin` serialization types.** The
+  original finding ("common/scanner code duplication, ~3K lines") is resolved:
+  `scanner` imports the shared data-usage types from `rustfs-data-usage` (see
+  the `pub use rustfs_data_usage::…` re-exports at the top of
+  `crates/scanner/src/data_usage_define.rs`). What remains: `scanner` and
+  `data-usage` each hold their own serialization types for the scanner cache
+  file (`DataUsageCacheInfo`/`DataUsageEntryInfo` in
+  `crates/scanner/src/data_usage_define.rs` vs
+  `DataUsageCacheInfo`/`DataUsageEntry` in
+  `crates/data-usage/src/data_usage.rs`); convergence is tracked in
+  rustfs/backlog#1828.
 
-- **ecstore is a monolith (87K lines, 163 files).** It contains disk management,
-  bucket management, erasure coding, replication, lifecycle, RPC, and configuration
-  — all in one crate. It should be decomposed along its existing subdirectories.
+- **ecstore is a monolith (265 files, ~288K lines — roughly half is inline
+  `#[cfg(test)]` code).** Measured with
+  `find crates/ecstore/src -name '*.rs' | xargs wc -l`. It contains disk
+  management, bucket management, erasure coding, replication, lifecycle, RPC,
+  and configuration — all in one crate. It should be decomposed along its
+  existing subdirectories; the split plan lives in
+  [docs/architecture/ecstore-module-split-plan.md](docs/architecture/ecstore-module-split-plan.md).
 
 ### High
 
@@ -154,19 +191,26 @@ module split is tracked under `docs/architecture/`.
   `common → filemeta/madmin` edges must stay removed so leaf/helper crates do
   not regain upward dependencies.
 
-- **Three-layer BackpressureConfig/DeadlockConfig duplication** across io-core,
-  concurrency, and `rustfs/src/storage`. Storage policies now expose and consume
-  explicit projections into the concurrency/io-core policy shapes, and workload
+- **Three-layer backpressure/deadlock policy bridging** across io-core,
+  concurrency, and `rustfs/src/storage`. The config types are no longer
+  duplicated (`BackpressureConfig` and `DeadlockDetectorConfig` are each
+  defined once, in io-core). Storage policies expose and consume explicit
+  projections into the concurrency/io-core policy shapes, and workload
   admission snapshots are composed through provider registries; later work
   should use those bridges before deleting compatibility wrappers.
 
 ### Medium
 
-- **Inconsistent error handling.** Three strategies (thiserror/snafu/anyhow) and
-  mixed naming (bare `Error` vs descriptive names).
+- **Bare `Error` naming.** Error-handling strategy has converged on `thiserror`
+  (no `snafu`, no `anyhow` in library code); the remaining inconsistency is the
+  bare `pub enum Error` naming in the 6 crates listed under Invariant 6.
 
-- **Ambiguous common vs utils boundary.** Both described as "utilities and data
-  structures." Need clear ownership rules.
+- **`common` is mostly parked domain code, not shared utilities.** Of its
+  6,724 lines, ~83% is scanner/heal domain code stranded there to break
+  dependency cycles (`metrics.rs`, ~4,810 lines of scanner-domain metrics;
+  `heal_channel.rs`, ~776 lines of heal-domain channel types). The
+  "common vs utils" naming ambiguity is secondary to moving that code to its
+  domain owners.
 
 ## Cross-Cutting Concerns
 
@@ -232,7 +276,7 @@ The binary (`main.rs`) boots in this order:
 
 ```
                             ┌─────────┐
-                            │  rustfs │  (binary + lib, 75K lines)
+                            │  rustfs │  (binary + lib)
                             │  main   │
                             └────┬────┘
                                  │
@@ -255,7 +299,7 @@ The binary (`main.rs`) boots in this order:
               │                  │                  │
         ┌─────▼──────┐    ┌──────▼──────┐    ┌──────▼──────┐
         │  ecstore   │    │     rio     │    │   io-core   │
-        │ (87K,core) │    │  (readers)  │    │ (zero-copy) │
+        │   (core)   │    │  (readers)  │    │ (zero-copy) │
         └─────┬──────┘    └─────────────┘    └─────────────┘
               │
      ┌─────┬──┼──┬─────┬──────┐

Cargo.toml

@@ -69,7 +69,7 @@ edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
 rust-version = "1.97.1"
-version = "1.0.0-beta.12"
+version = "1.0.0-rc.1"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -86,52 +86,52 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-beta.12" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.12" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.12" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.12" }
-rustfs-common = { path = "crates/common", version = "1.0.0-beta.12" }
-rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.12" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-beta.12" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.12" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.12" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.12" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.12" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.12" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.12" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.12" }
-rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.12" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.12" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.12" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.12" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.12" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.12" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.12" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.12" }
-rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.12" }
-rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-beta.12" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.12" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.12" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.12" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.12" }
-rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.12" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.12" }
-rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.12" }
-rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.12" }
-rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.12" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.12" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.12" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.12" }
-rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.12" }
-rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.12" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.12" }
-rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.12" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.12" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.12" }
-rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-beta.12" }
-rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.12" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.12" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.12" }
+rustfs = { path = "./rustfs", version = "1.0.0-rc.1" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-rc.1" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-rc.1" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-rc.1" }
+rustfs-common = { path = "crates/common", version = "1.0.0-rc.1" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-rc.1" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-rc.1" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-rc.1" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-rc.1" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-rc.1" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-rc.1" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-rc.1" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-rc.1" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-rc.1" }
+rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-rc.1" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-rc.1" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-rc.1" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-rc.1" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-rc.1" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-rc.1" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-rc.1" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-rc.1" }
+rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-rc.1", default-features = false }
+rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-rc.1" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-rc.1" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-rc.1" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-rc.1" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-rc.1" }
+rustfs-replication = { path = "crates/replication", version = "1.0.0-rc.1" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-rc.1" }
+rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-rc.1" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-rc.1" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-rc.1" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-rc.1" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-rc.1" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-rc.1" }
+rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-rc.1" }
+rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-rc.1" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-rc.1" }
+rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-rc.1" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-rc.1" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-rc.1" }
+rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-rc.1" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-rc.1" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-rc.1" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-rc.1" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
@@ -139,13 +139,13 @@ async_zip = { default-features = false, version = "0.0.18" }
 mysql_async = { default-features = false, version = "0.37" }
 async-compression = { version = "0.4.43" }
 async-recursion = "1.1.1"
-async-trait = "0.1.91"
+async-trait = "0.1.92"
 async-nats = { version = "0.50.0", default-features = false }
 axum = "0.8.9"
-futures = "0.3.33"
-futures-core = "0.3.33"
+futures = "0.3.34"
+futures-core = "0.3.34"
 futures-lite = "2.6.1"
-futures-util = "0.3.33"
+futures-util = "0.3.34"
 pollster = "1.0.1"
 pulsar = { default-features = false, version = "6.8.0" }
 lapin = { default-features = false, version = "4.10.0" }
@@ -171,7 +171,7 @@ tower = { version = "0.5.3" }
 tower-http = { version = "0.7.0" }
 
 # Serialization and Data Formats
-apache-avro = "0.21.0"
+apache-avro = "0.22.0"
 bytes = { version = "1.12.1" }
 bytesize = "2.7.0"
 byteorder = "1.5.0"
@@ -228,15 +228,15 @@ atomic_enum = "0.3.0"
 aws-config = { version = "1.10.1" }
 aws-credential-types = { version = "1.3.0" }
 aws-sdk-kms = { default-features = false, version = "1.114.0" }
-aws-sdk-s3 = { default-features = false, version = "1.140.0" }
+aws-sdk-s3 = { default-features = false, version = "1.141.0" }
 aws-sdk-sts = { default-features = false, version = "1.110.0" }
 aws-smithy-http-client = { default-features = false, version = "1.2.0" }
 aws-smithy-runtime-api = { version = "1.14.0" }
 aws-smithy-types = { version = "1.6.1" }
-base64 = "0.23.0"
+base64 = "0.23.1"
 base64-simd = "0.8.0"
 brotli = "8.0.4"
-clap = { version = "4.6.5" }
+clap = { version = "4.6.6" }
 const-str = { version = "1.1.0" }
 convert_case = "0.11.0"
 criterion = { version = "0.8" }
@@ -244,7 +244,7 @@ crossbeam-queue = "0.3.13"
 crossbeam-channel = "0.5.16"
 crossbeam-deque = "0.8.7"
 crossbeam-utils = "0.8.22"
-datafusion = { default-features = false, git = "https://github.com/apache/datafusion.git", rev = "dae03ee062b2abf986de8df12ea82fb1578a2d99" }
+datafusion = { default-features = false, git = "https://github.com/apache/datafusion.git", rev = "e08aed1e5de41dcf81d529140dae07723b942a5e" }
 #datafusion = { default-features = false, version = "54.1.0" }
 derive_builder = "0.20.2"
 enumset = "1.1.14"
@@ -268,7 +268,7 @@ lz4 = "1.28.1"
 matchit = "0.9.2"
 md-5 = "0.11.0"
 mime_guess = "2.0.5"
-moka = { version = "0.12.15" }
+moka = { version = "0.12.16" }
 netif = "0.1.6"
 num_cpus = { version = "1.17.0" }
 nvml-wrapper = "0.12.1"
@@ -278,7 +278,7 @@ percent-encoding = "2.3.2"
 pin-project-lite = "0.2.17"
 pretty_assertions = "1.4.1"
 rand = { version = "0.10.2" }
-ratelimit = "0.10.1"
+ratelimit = "2.0.0"
 rayon = "1.12.0"
 reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.2" }
 reed-solomon-simd = "3.1.0"
@@ -289,12 +289,12 @@ rustify = { version = "0.7", default-features = false }
 rustix = { version = "1.1.4" }
 rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }
-s3s = { git = "https://github.com/cxymds/s3s.git", rev = "fe3941d91fa1c69956f209a9145995c9f0235bff" }
+s3s = { git = "https://github.com/rustfs/s3s.git", rev = "d7028511a53f69d41ed3c69f36899f9b1aede647" }
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.3"
 smallvec = { version = "1.15.2" }
-smartstring = "1.0.1"
+compact_str = "0.10.0"
 snap = "1.1.2"
 starshard = { version = "2.2.2" }
 strum = { version = "0.28.0" }
@@ -302,7 +302,7 @@ sysinfo = "0.39.6"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
-thiserror = "2.0.19"
+thiserror = "2.0.20"
 tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-core = "0.1.36"
@@ -339,22 +339,22 @@ pyroscope = { version = "2.1.1" }
 libunftp = { version = "0.23.0" }
 unftp-core = "0.1.0"
 suppaftp = { version = "10.0.1" }
-rcgen = { version = "0.14.8", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
-russh = { version = "0.62.5" }
-russh-sftp = "2.3.0"
+rcgen = { version = "0.14.9", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
+russh = { version = "0.62.6" }
+russh-sftp = "2.4.0"
 
 # WebDAV
 dav-server = "0.11.0"
 
 # Performance Analysis and Memory Profiling
-mimalloc = { version = "0.1.52", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "1cdadea43e9c5a0f054b65be21200ce580e4eb13" }
-libmimalloc-sys = { version = "0.1.49", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "1cdadea43e9c5a0f054b65be21200ce580e4eb13", features = ["extended"] }
-hotpath = { version = "0.23.0", default-features = false }
+mimalloc = { version = "0.1.52", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "ce6338661179c8be22e516b00af7483f151485a7" }
+libmimalloc-sys = { version = "0.1.49", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "ce6338661179c8be22e516b00af7483f151485a7", features = ["extended"] }
+hotpath = { version = "0.23.2", default-features = false }
 # Snapshot testing for output format regression detection
 insta = { version = "1.48" }
 
 [workspace.metadata.cargo-shear]
-ignored = ["rustfs"]
+ignored = ["hotpath", "rustfs"]
 
 [profile.dev]
 # Full debuginfo roughly doubles compile+link time and produces multi-GB

Dockerfile

@@ -91,7 +91,12 @@ LABEL name="RustFS" \
 # Upgrade base-image packages so published images pick up security fixes
 # (e.g. openssl/libssl3 CVEs) without waiting for a new Alpine point release.
 RUN apk upgrade --no-cache && \
-    apk add --no-cache ca-certificates coreutils curl
+    apk add --no-cache \
+    ca-certificates \
+    coreutils \
+    curl \
+    tzdata \
+    && test "$(TZ=Asia/Kolkata date +%z)" = "+0530"
 
 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
 COPY --from=build /build/rustfs /usr/bin/rustfs

Dockerfile.glibc

@@ -96,9 +96,11 @@ LABEL name="RustFS" \
 # Upgrade base-image packages so published images pick up security fixes
 # (e.g. tar/gzip/perl CVEs) without waiting for a new Ubuntu point release.
 RUN apt-get update && apt-get upgrade -y \
-    && apt-get install -y --no-install-recommends \
+    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
     ca-certificates \
     curl \
+    tzdata \
+    && test "$(TZ=Asia/Kolkata date +%z)" = "+0530" \
     && rm -rf /var/lib/apt/lists/*
 
 COPY --from=build /build/rustfs /usr/bin/rustfs

MINIO_REPLICATION_COMPAT_REVIEW.md

@@ -1,314 +0,0 @@
-# RustFS 站点复制 / 桶复制 — MinIO 兼容性审查报告
-
-> 审查日期:2026-08-05
-> 审查对象:RustFS(worktree `reatang/minio-compatibility-review-03a7fb`)vs MinIO(`/Users/tang/Documents/GitHub/minio`)
-> 审查方式:白盒代码对比(5 个维度并行审查)+ P0 问题对抗性复核
-> 审查维度:站点复制白盒对比、桶复制白盒对比、mc 工具兼容性、S3 标准协议兼容性、代码结构与分层
-
----
-
-## 一、总体结论
-
-| 领域 | 兼容性评价 |
-|---|---|
-| **站点复制(RustFS↔RustFS + mc 管理)** | 良好。admin 端点全覆盖、JSON 结构对齐 madmin-go、请求体 DARE 加密兼容,mc admin replicate 全家桶基本可用 |
-| **站点复制(RustFS↔MinIO 混合组网)** | **断裂**。4 个 P0:出站 join 路径 404、metainfo 大小写解析失败、STS item 类型名不一致、policy-mapping userType 数值错位 |
-| **桶复制(控制面,S3 标准 API)** | 良好。Put/Get/DeleteBucketReplication、错误码、状态机字符串、xl.meta 内部键均对齐 |
-| **桶复制(数据面,RustFS→MinIO)** | **断裂**。复制 PUT 缺 `?versionId=` 导致目标端版本漂移(P0);CopyObject 完全不复制(P0) |
-| **mc 桶复制命令** | **部分断裂**。`mc replicate add` 默认参数即失败(P0);status/resync/backlog 响应结构不匹配导致静默空输出(P1) |
-| **代码结构** | 桶复制侧迁移架构有纪律但成本高;**站点复制侧无领域层,约 9500 行业务逻辑堆在 admin handler,且存在 3 处反向依赖违反项目分层不变量(P0)** |
-
-**做得好的地方**(已确认兼容,无需整改):复制状态机字符串(PENDING/COMPLETED/FAILED/REPLICA 含 legacy COMPLETE)、xl.meta 内部键双前缀(x-rustfs-internal- + x-minio-internal-)读写、ReplicateDecision 内部状态串格式、复制内部头主链路双前缀、Delete/VersionPurge 语义、Resync reset-id 判定、admin 路由 `/minio/admin/v3` 前缀别名、madmin DARE 加密流解密、站点复制 gob netperf 编码、`site-repl-<deploymentID>` 规则模板。
-
----
-
-## 二、P0 问题清单(8 项)
-
-| # | 问题 | 来源维度 | 断裂方向 |
-|---|---|---|---|
-| P0-1 | 出站 peer join 使用 MinIO 已移除的遗留路径 `/site-replication/join` → 404 | 站点复制 | RustFS→MinIO |
-| P0-2 | 解析 MinIO metainfo(SRInfo)字段大小写不匹配 → add preflight 失败 | 站点复制 | RustFS→MinIO |
-| P0-3 | STS 凭证复制 item 类型名 `sts-credential` vs `sts-account` | 站点复制 | 双向 |
-| P0-4 | policy-mapping `userType` 数值语义错位(RustFS: None=0/Svc=1/Sts=2/Reg=3;MinIO: reg=0/sts=1/svc=2)→ 权限静默漂移 | 站点复制 | 双向 |
-| P0-5 | 复制 PUT/CompleteMultipart 不携带 `?versionId=` query → MinIO 端版本号漂移、版本删除永久 no-op、双端静默发散(**功能视角复核:定级调整为 P1**,问题重述为"普通复制对象缺少可靠的源→目标版本身份策略";versionId query 是可行修复之一而非唯一正确方案) | 桶复制 | RustFS→MinIO |
-| P0-6 | CopyObject(含 metadata-replace 自拷贝)完全不触发复制调度,对象静默不复制(**功能视角复核:定级调整为 P1**;scanner 在 ExistingObjectReplication 启用+状态为空时可最终补齐,但同步复制语义失效,且继承 stale COMPLETED / 显式 Disabled 场景长期漏复制) | 桶复制 + S3 协议 | 所有方向 |
-| P0-7 | `mc replicate add` 默认参数(healthcheck-seconds=60)被硬拒 400;且字段单位按秒解析而 wire 为纳秒 | mc 兼容 | mc→RustFS |
-| P0-8 | 架构:站点复制约 9500 行业务逻辑堆在 admin handler 单文件;app/storage 层 3 处反向导入 admin 层,违反 ARCHITECTURE.md 分层不变量 #1(**对抗复核后降级为 P1**:反向边已被 arch 守卫棘轮基线锁死,属受控技术债) | 代码结构 | — |
-
-每项 P0 的对抗性复核结论、验证方案与解决方案见 **第五节**。
-
-**修复状态(2026-08-05)**:7 项确认 P0 已全部修复并创建 PR(红灯→绿灯 TDD):P0-1 [#5748](https://github.com/rustfs/rustfs/pull/5748)、P0-2 [#5749](https://github.com/rustfs/rustfs/pull/5749)、P0-3 [#5750](https://github.com/rustfs/rustfs/pull/5750)、P0-4 [#5751](https://github.com/rustfs/rustfs/pull/5751)、P0-5 [#5752](https://github.com/rustfs/rustfs/pull/5752)、P0-6+P1-10 [#5753](https://github.com/rustfs/rustfs/pull/5753)、P0-7 [#5754](https://github.com/rustfs/rustfs/pull/5754)。合并顺序:#5748+#5749 同批;#5752 先于 #5753。
-
----
-
-## 三、P1 问题清单
-
-### 站点复制
-
-| # | 问题 | 证据 | 影响 |
-|---|---|---|---|
-| P1-1 | ILM(lc-config)复制语义:对外开关限定 `replicateILMExpiry`,但发送端把**完整** lifecycle.xml 放入 `expiry_lc_config`,接收端整体覆盖/删除本地配置(功能视角复核:**确认,维持 P1**;更新时间检查只能拒旧,不能修复整体覆盖语义) | RustFS `bucket_meta.rs:948-951`、`site_replication.rs:7590-7683` vs MinIO `site-replication.go:1784-1810,6138` | lifecycle 同时含 expiry 与本地 transition 时,非 expiry 规则被错误传播或本地 transition 被覆盖。**缺"同步 expiry 后保留本地 transition"测试** |
-| ~~P1-2~~→**P2-25** | `SRInfo.ilmExpiryRules` 从不填充,ILM 一致性状态恒为空(功能视角复核:**降级 P2**——仅影响管理面可观测性,不改变对象数据) | `site_replication.rs:4152-4266,4855-4868` | `mc admin replicate status --ilm-expiry-rules` 恒空,ILM 漂移不可见 |
-| P1-3 | 无自动跨站元数据 heal(MinIO 有周期 heal 协程) | RustFS 仅 600s 本地 wiring 修复(`site_replication_reconcile.rs:34,59-81`)+ 手动 repair 端点 vs MinIO `site-replication.go:4257-4288` | 错过的 IAM/bucket 元数据更新持续漂移,须手工 repair |
-| P1-4(拆分) | ①`sync` 同步复制指控:功能视角复核**不成立/证据不足**——RustFS 自身契约明确将 `sync_state` 定义为站点可达性/配置完整性健康状态且有测试,不能以他家同名字段判其错误(属"RustFS 独特设计保持不变"项,撤销);②`defaultbandwidth`:**确认,降级 P2**——公共 API 接受并持久化,但建 site replication bucket target 时不应用,reconcile 只保留既有 `bandwidth_limit`,配置成功但不生效 | `site_replication.rs:6303-6357,5004-5027` | ②为用户可见的"配置成功但无效"能力缺口 |
-
-### 桶复制 / S3 协议
-
-| # | 问题 | 证据 | 影响 |
-|---|---|---|---|
-| P1-5 | 未复制完成对象的 GET/HEAD 远端 proxy 未实现;也不识别 MinIO 的 `X-Minio-Source-Proxy-Request` 防环头 | 仅指标占位(`storage_api.rs:799-804`);`SUFFIX_SOURCE_PROXY_REQUEST` 定义后无人使用 vs MinIO `bucket-replication.go:2334,2409,2534` | active-active 复制滞后窗口内 RustFS 端 404 |
-| P1-6 | `X-Minio-Source-Replication-{Tagging,Retention,LegalHold}-Timestamp` 三个时间戳头收发均缺失 | `replication_target_boundary.rs:251-297` 填了 options 但 `PutObjectOptions::header()` 不序列化;接收端不解析 vs MinIO `object-api-options.go:377-399` | active-active 下标签/retention/legal-hold 并发修改的 LWW 冲突解析退化,可能元数据回滚 |
-| P1-7 | ARN 前缀 `arn:rustfs:` 与 `arn:minio:` 不互认(解析侧强制 `arn:rustfs:`) | `crates/ecstore/src/bucket/target/arn.rs:43,51` vs MinIO `bucket-targets.go:709` | 存量 MinIO 复制配置迁移被 StaleTarget 拒;原生 madmin SDK 解析 RustFS ARN 失败 |
-| P1-8 | PutBucketReplication 校验缺口(规则数/Priority 唯一/ID 长度/Filter 互斥/2MB 上限全缺)+ 主动拒绝 `Destination.StorageClass` 等 MinIO/AWS 合法字段 | `bucket_usecase.rs:582-616`、`config.rs:143-232` vs MinIO `internal/bucket/replication/replication.go:29-90` | 非法配置被接受、优先级冲突行为不可预测;存量 AWS/Terraform 配置(含 StorageClass)直接 400 |
-| ~~P1-9~~→**P2-26** | GetObject 响应缺 `x-amz-replication-status` 头(HEAD 有 GET 无),且 GET 专门把它从 metadata 过滤掉(功能视角复核:**降级 P2**;GET/HEAD 不一致确认,缺 GET replication-status 回归测试) | `object_usecase.rs:5696-5735`、`options.rs:702` vs MinIO `api-headers.go:236-238` | 依赖 GET 判断复制状态的客户端/监控失效;修复约一行 |
-| P1-10 | Snowball auto-extract 解包对象不触发复制(功能视角复核:**确认,维持 P1**,但"全部永不复制"不准确——scanner 在状态空+ExistingObjectReplication 启用时可补齐;显式 Disabled 等场景长期遗漏,即时复制始终失效。带 REPLICA 状态的入站成员须继续避免回环)。**已随 [#5753](https://github.com/rustfs/rustfs/pull/5753) 修复**(含入站复制 PUT 不再被误派发 extract 的次生缺陷) | `object_usecase.rs:8201` vs MinIO `object-handlers.go:2452,2510-2511` | 批量导入对象不即时复制;缺普通解包成员复制结果的测试(已在 #5753 补充 e2e) |
-
-### mc 响应结构(静默空输出类)
-
-| # | 问题 | 证据 | 影响 |
-|---|---|---|---|
-| P1-11 | `?replication-metrics[=2]` 响应为 Rust snake_case,minio-go MetricsV2 期望 camelCase(`currStats`/`queueStats`/…) | `stats.rs:617-770`、`admin/router.rs:1583-1592` vs MinIO `bucket-stats.go:154-188` | `mc replicate status` 不报错但全零(静默错误) |
-| P1-12 | replication-reset(resync)响应壳不匹配:`{"Targets":[{"Arn","ResetID",...}]}` vs `{"target":[{"arn","resetid","resyncStatus",...}]}` | `router.rs:126-198,1735-1803` vs MinIO `bucket-replication-utils.go:613-636` | `mc replicate resync start/status` 输出空;仅响应壳问题,修复成本低 |
-| P1-13 | `/v3/replication/mrf` 与 `/v3/replication/diff` 返回单个聚合对象而非条目流(代码自述 deliberate) | `replication.rs:695-725,879-911,998-1047` vs madmin-go `replication-api.go:104-176` | `mc replicate backlog` 输出空;`node`/`arn`/`verbose` 参数被忽略 |
-| P1-14 | set-remote-target 请求体 `deny_unknown_fields` + 字段名偏差(期望 `bandwidth_limit`,madmin 发 `bandwidthlimit`;`session_token` vs `sessionToken` 等) | `handlers/replication.rs:88-95,108-163` vs madmin-go `bucket-targets.go:76` | `mc replicate add/update --bandwidth` 整请求失败;凡 omitempty 字段一旦出现即 400 |
-
-### 代码结构
-
-| # | 问题 | 证据 | 影响 |
-|---|---|---|---|
-| P1-15 | 站点复制状态两套归一化实现(handler 类型化 vs service 无类型 JSON),且 reload 的 read→normalize→save 全程无共同分布式对象锁,存在 lost-update 竞争;repair state 已用 `with_config_object_write_lock` 包住完整 RMW,主 state 未采用同等保护(功能视角复核:**确认,维持 P1**;进程内 `SITE_REPLICATION_STATE_LOCK` 与单次 read/save 各自的对象锁均不能保护跨调用 RMW:A 读旧→B 另节点写入→A 用旧快照覆盖,B 丢失) | `handlers/site_replication.rs:114,347,1039-1130` vs `service/site_replication.rs:26-135` | 归一化语义可 drift;多节点/RPC 并发写状态互相覆盖。**缺多节点/双写者 lost-update 回归测试** |
-| P1-16 | 复制状态机类型双份定义:`rustfs-filemeta` 与 `rustfs-replication` 各持一份(ReplicationStatusType/VersionPurgeStatusType/ReplicationState/MrfReplicateEntry/ReplicateObjectInfo),靠 boundary 双向转换 | `crates/filemeta/src/replication.rs` vs `crates/replication/src/filemeta.rs` | 状态机语义修改须同步两处+转换层,漏一处即静默数据语义错误;建议加 enum 对账测试 |
-| P1-17 | 桶复制逻辑分裂:`crates/replication` 仅契约,执行引擎(pool 5947 行、resyncer 4090 行)仍在 ecstore,中间 20+ 个 boundary/bridge 微文件;迁移无完成判据,脚手架有固化风险 | `crates/ecstore/src/bucket/replication/README.md`、`mod.rs:15-45` | 可读性/可维护性成本;需设定迁移里程碑 |
-| P1-18 | 超长函数集中在复制热路径:`resync_bucket` 536 行、`replicate_all` 403 行、`start_mrf_processor` 305 行、`apply_iam_item` 248 行 | `replication_resyncer.rs:546`、`replication_pool.rs`、`site_replication.rs:7806` | 正确性审查与修改风险高 |
-
-### 第三方复审新增与调整项(功能视角二次复核后)
-
-| # | 问题 | 来源 | 影响 |
-|---|---|---|---|
-| P1-19 | 普通复制对象缺少可靠的源→目标版本身份策略:PUT 响应的目标版本 ID 未捕获/持久化,对不支持 versionId query 的目标(原生 AWS S3 等),后续版本删除复制落空;MRF 只会重试同一个错误身份,HEAD ETag fallback 不能修复删除 | P0-5 复审 | 非 MinIO 系目标的版本化复制双端发散。缺"目标自行分配版本 ID"场景测试 |
-| P1-20 | 缺少 scanner 补偿边界的 e2e:ExistingObjectReplication Enabled/Disabled × 空状态/继承状态 组合下的补齐与不补齐行为无回归覆盖(Copy 与 Snowball 两路径) | P0-6/P1-10 复审 | scanner 兜底语义变化不可见 |
-| P1-21 | delete-marker 延迟 purge 失败静默丢弃(由 P2-20① 升级):目标删除失败无日志/状态/MRF,目标端 marker/版本可能永久残留 | P2-20 复核升级 | 数据一致性;缺失败注入测试 |
-| P1-22 | 桶复制整体 SSE 支持能力缺口(替代原 P2-23):SSE-S3/SSE-KMS 所有复制模式统一 fail closed,SSE-C 失败被 e2e 钉为当前行为,无 encrypted-object resync e2e | P2-23 复核改写 | 加密对象跨站不复制;需覆盖普通复制/Heal/Resync/Multipart 四模式 |
-
-### 功能视角二次复核采纳记录(backlog#1675,基于 main f0c4fbd28)
-
-复核共 10 项,判定依据为 RustFS 自身功能契约与实际调用链,不以对齐 MinIO 为正确性标准。采纳结果:
-
-| 原编号 | 复核结论 | 采纳动作 |
-|---|---|---|
-| P0-5 | 确认,P0→P1,问题重述为"源→目标版本身份策略缺失" | 定级调整;修复已合 [#5752](https://github.com/rustfs/rustfs/pull/5752);残留缺口 P1-19 |
-| P0-6 | 确认,P0→P1,scanner 描述纠正 | 定级调整;修复已合 [#5753](https://github.com/rustfs/rustfs/pull/5753);测试缺口 P1-20 |
-| P1-1 | 确认,维持 P1 | 补记"expiry 同步后保留本地 transition"测试缺口 |
-| P1-2 | 确认,P1→P2(仅管理面可观测性) | 改编号 P2-25 |
-| P1-4 | 拆分:`sync` 指控不成立(RustFS 自身契约定义为健康状态,有测试);`defaultbandwidth` 确认为 P2 能力缺口 | `sync` 撤销并归入"独特设计保持不变";`defaultbandwidth` 降 P2 |
-| P1-9 | 确认,P1→P2 | 改编号 P2-26;补记缺 GET 回归测试 |
-| P1-10 | 确认,维持 P1,"全部永不复制"改为"即时复制失效+部分场景长期遗漏" | 已随 [#5753](https://github.com/rustfs/rustfs/pull/5753) 修复(含回环防护) |
-| P1-15 | 确认,维持 P1(竞争机理精确化:跨调用 RMW 无共同分布式锁) | 补记缺双写者 lost-update 测试 |
-| P2-20① | 确认,P2→P1(延迟 purge 失败静默丢弃部分) | 升级为 P1-21;②③维持 P2 |
-| P2-23 | resync 专属指控不成立;暴露桶复制整体 SSE 能力缺口 | 撤销原表述,改立 P1-22 |
-
-**复核指出的测试补齐清单**(均未运行跨实例集成验证,需落地):目标自行分配版本 ID、Copy/Snowball scanner 补偿边界、lifecycle expiry/transition 保留、site state 双写竞争、delayed purge 失败注入、encrypted-object resync。
-
----
-
-## 四、P2 问题清单
-
-### 站点复制
-- **P2-1** `showDeleted` 选项与 `bucketDeletedTimestamp` 未实现(`site_replication.rs:1364-1381`)
-- **P2-2** 错误码泛化:统一 `InvalidRequest`/`InternalError`,无 MinIO 的 9 个 `XMinioSiteReplication*` 专用码(400/503 语义丢失)
-- **P2-3** `make-with-versioning` 忽略 `versioningEnabled`/`forceCreate` 参数,恒 true(`site_replication.rs:8597-8627`)
-- **P2-4** netperf 返回"不支持"占位(gob 格式兼容不会崩);devnull 有请求体大小上限(MinIO 无限 discard)
-- **P2-5** Metrics 摘要仅含本站,无 per-peer 链路统计(downtime/latency/失败窗口)
-- **P2-6** `external-user`/`credential` IAM item 未实现——与本仓 MinIO 版本等价缺失,结构已预留;对接新版 MinIO 时会成缺口
-- **P2-7** 本地 deploymentID 缺失时回退 endpoint 哈希(16 位 hex,非 UUID 形态)
-
-### 桶复制 / S3 协议
-- **P2-8** 遗留内部 client 头名错误:`X-Source-DeleteMarker`/`X-Check-Replication-Ready` 缺 `X-Minio-` 前缀(`client/api_stat.rs:191-231`,当前路径未激活,潜伏缺陷)
-- **P2-9** Remote target admin 错误码扁平化(MinIO 有 404/503 专用码,RustFS 统一 400/500)
-- **P2-10** Remote target 拒绝 `disableProxy`/`edge`/`edgeSyncBeforeExpiry` 等 madmin 字段(非默认参数,影响小)
-- **P2-11** `list-remote-targets` 序列化偏差:`bandwidth_limit`/`storage_class`/`deployment_id`/`reset_id`/`session_token` vs madmin 的 `bandwidthlimit`/`storageclass`/`deploymentID`/`resetID`/`sessionToken`;`healthCheckDuration`/`totalDowntime` 按秒序列化而 Go 按纳秒解;`type` 过滤参数被忽略
-- **P2-12** set-remote-target?update=true 忽略 madmin 的 op 标志(creds/sync/proxy/…),固定整体覆盖
-- **P2-13** XML 反序列化:Rule 内未知元素严格报 MalformedXML(顶层却跳过,行为不一致);缺 `<Role>` 报 MalformedXML(Go 容忍)——向前兼容性差,当前主流客户端不受影响
-- **P2-14** `ReplicaModifications` 默认 Disabled(与 AWS 一致、与 MinIO 的注入 Enabled 分歧);PUT 时不像 MinIO 那样注入默认元素回写
-- **P2-15** PutBucketReplication 要求预先注册 remote target(与 MinIO 同构、与纯 AWS 流程分歧),报错未指引先建 target
-- **P2-16** GetBucketReplication 响应无 xmlns(与 MinIO 一致,极少数严格 SDK 可能拒收)
-- **P2-17** 站点复制启用时不阻止普通用户直接改桶复制配置(MinIO 非 root 报 `ErrReplicationDenyEditError`)
-- **P2-18** Prometheus 指标名对齐 metrics-v3 但注册前缀为 rustfs 体系;versioning 错误文案与 MinIO 不同(code 一致)
-
-### 代码结构
-- **P2-19** `apply_iam_item` / bucket-ops 用裸字符串 match 分发,无法穷尽检查;建议改 `#[serde(tag)]` 枚举
-- **P2-20(拆分)** 静默吞错:①`replication_resyncer.rs:1693` delete-marker 延迟 purge 失败被 `let _ =` 丢弃,target client 缺失时直接跳过——**功能视角复核:升级为 P1-21**(失败后无日志、无状态更新、不入 MRF,目标 delete marker/版本可能永久残留;启动前的 5 次循环只是等源 marker 消失,不是对目标删除失败的重试。缺注入目标删除失败并验证重试/状态/MRF 的测试);②`site_replication.rs:8661` purge-deleted-bucket 吞掉非 NotFound 错误、`:9227` cancel resync 失败无痕迹——维持 P2
-- **P2-21** `MrfV2` 全套机制(Error/Capabilities/Readiness/Reader/Envelope)未接线,生产只用 v1,属投机代码
-- **P2-22** `persist_site_replication_state` 双重 clone + 双重 normalize(`site_replication.rs:1143-1152` → `:1116-1122`)
-- **P2-23(撤销并改写)** 原"resync 不处理 SSE"指控不成立——`ReplicationType::Resync` 与普通复制/Heal 最终走同一 `replication_put_object_options`,`// TODO: SSE` 不构成 resync 独立行为差异。真实状态:SSE-S3/SSE-KMS 在**所有复制模式**下统一 fail closed,SSE-C 普通桶复制失败已被现有 e2e 钉为当前行为,且无 encrypted-object resync e2e → 改立能力项 **P1-22"桶复制整体 SSE 支持"**(需分别覆盖普通复制、Heal、手动 Resync、Multipart)
-- **P2-24** `crates/replication` 命名误导(名为复制引擎实为契约库),建议 lib.rs 顶部文档说明
-- 正面确认:生产代码 unwrap/expect 纪律良好(几乎全在测试模块);MinIO 概念映射(ReplicationPool/Resyncer/MRF/TargetClient)桶复制侧清晰,站点复制侧缺 `SiteReplicationSys` 聚合体
-
----
-
-## 五、P0 问题对抗性分析(复核结论 + 验证方案 + 解决方案)
-
-### P0-1 出站 peer join 路径 — **CONFIRMED(比原指控更严重)**
-
-**复核结论**:指控全部成立,且加重三点:
-1. `/minio/admin/v3/site-replication/join` 在 MinIO 历史上**从未存在过**(`git log -S` 追到功能诞生的 2021 年首个提交,注册的就是 `peer/join`)。RustFS 实现者疑似被 MinIO `admin-handlers-site-replication.go:76` 一条过时的文档注释误导。
-2. 无任何 404 回退、版本探测或 feature flag;唯一的重试逻辑只针对 secret 不匹配(`site_replication.rs:3036-3082`),404 直接失败。
-3. 现有单测 `:13683-13696` 正在**固化错误行为**(测试名声称匹配 MinIO 路由,断言的却是不存在的路由)。RustFS↔RustFS 之所以不暴雷,是因为 RustFS 入站自己注册了该错误路径的兼容别名,掩盖了 bug。
-
-**影响面**:RustFS 发起的 add(含 MinIO 站点)、服务账号轮换通知 MinIO peer 均断;MinIO→RustFS 与 RustFS↔RustFS 不受影响;其余 peer/* 端点走通用前缀改写,路径正确。
-
-**修路径还不够,还有三处 join 协议分歧须同批修**:①加密判定 `site_replication_peer_payload_encrypted`(:2899-2901)只对旧路径加密,MinIO `SRPeerJoin` 强制解密,须跟随路径改;②MinIO join 成功返回**空 body**,RustFS `:8163` 强制解析 `SRPeerJoinResponse` 会失败,须容忍空 body(peer 身份回退用 preflight 已取得的数据合成);③`deferSyncStateEnable`/`bootstrapToken` 对 MinIO 无效但不阻断(行为差异,建议日志标注)。
-
-**验证方案**:
-- 单测:翻转 `:13683`/`:13699` 两个测试断言为 `peer/join`(把固化 bug 的测试变成回归防护)。
-- 集成测:测试内起 axum stub 精确复刻 `admin-router.go` 路由(仅注册 `PUT .../peer/join`,其余 404),handler 内用 `decrypt_stream_io` 验证 body 是 madmin 兼容密文,返回 200 空 body;断言修复前 404、修复后全链路成功。
-- e2e:docker compose(rustfs+minio),RustFS 侧 `mc admin replicate add`,MinIO 侧 `mc admin trace -a` 断言 `PUT .../peer/join` 200。注意:**e2e 会先被 P0-2 的 preflight 挡住,两问题必须同批修复才能全链路验证**。
-
-**解决方案**(均在 `handlers/site_replication.rs`):删除 :2885-2886 的 join 特判使其落入通用前缀改写;:2899-2901 加密判定改为对 `peer/join` 返回 true;:8163 响应解析容忍空 body;更新两个单测。
-**滚动升级风险**:必须保留入站的 `/v3/site-replication/join` 旧路径路由(旧版 RustFS 出站仍发它);发版前对最近 release tag 复核旧版入站已注册 `peer/join`。
-
-### P0-2 SRInfo 大小写不匹配 — **CONFIRMED(范围精确化)**
-
-**复核结论**:成立。madmin-go v3.0.109(minio go.mod 锁定版)`SRInfo` 除 `APIVersion` 外 12 个顶层字段**全部无 json tag**,Go 按 PascalCase 序列化;RustFS `SRInfo` serde 大小写敏感、全字段 `#[serde(default)]` → 解析 MinIO 输出**不报错而是静默全空**。精确化:**不兼容仅限 SRInfo 顶层 12 个字段**,嵌套结构(SRBucketInfo/SRStateInfo/SRIAMPolicy 等)madmin 本就带小写 tag,不受影响。`:5581` 的 `"buckets"|"Buckets"` 手写双读证明作者已知 MinIO 输出 PascalCase,只是未系统化修复。
-
-**影响面**:RustFS 发起 add 时 preflight 硬失败("site did not report deploymentID")——**触发顺序先于 P0-1 的 join**;`mc admin replicate status` 对 MinIO peer 静默显示全空/全 mismatch(HTTP 200,无报错)。MinIO 读 RustFS 方向因 Go unmarshal 大小写不敏感而无恙。
-
-**验证方案**:
-- 单测(crates/madmin):用 Go `json.Marshal(madmin.SRInfo{...})` 真实生成的 PascalCase JSON 作 fixture,断言反序列化后字段非空;再加序列化回归断言输出仍为 camelCase(保证 RustFS↔RustFS 不回归)。
-- 集成测:stub 在 metainfo 端点返回 PascalCase body,走 `remote_add_preflight_info`,断言不再报错。
-- e2e:与 P0-1 同批,`mc admin replicate status --json` 断言 MinIO 站点条目完整。
-
-**解决方案**:`crates/madmin/src/site_replication.rs:642-670` 为 12 个顶层字段逐一加 `#[serde(alias = "...")]`(精确取 Go 字段名,注意是 `ILMExpiryRules` 不是 `IlmExpiryRules`)。alias 只影响反序列化,出站格式零变化,风险几乎为零。**只加顶层、不扩散到嵌套结构**,并留注释说明原因。回归防护关键是把 Go 真实输出固化为测试 fixture。
-
-### P0-7 `mc replicate add` 默认参数被拒 + 单位错误 — **CONFIRMED**
-
-**复核结论**:全部反驳方向反向坐实(本地有 mc 源码,非推断):
-- mc `replicate-add.go:93-95` 默认 `healthcheck-seconds=60`,`:301-303` 无条件调用 `SetRemoteTarget`,失败即终止,无跳过路径;
-- madmin `bucket-targets.go:79` `HealthCheckDuration time.Duration` 无自定义 Marshal → wire 上是纳秒整数 `60000000000`;
-- RustFS `handlers/replication.rs:213-225` 对非零值必拒 400;`mc replicate update` 同样失败;无老端点绕过。
-- **单位错误独立成立且双向**:请求侧按 `Duration::from_secs` 解析(60e9 ns 会被当 60e9 秒 ≈ 1900 年);响应/持久化侧 `bucket_target.rs:195-197` 按秒序列化,mc 按纳秒解(60s 显示为 60ns),同时构成与 MinIO `bucket-targets.json` 的持久化格式偏差。
-- **为何没被发现**:这是刻意的"能力契约式拒绝"策略,且有单测 `replication.rs:1353-1379` 固化拒绝行为;e2e 全部自行构造 JSON、不含该字段,测的是"RustFS 自己的请求形态"而非"mc 默认请求形态"。缓解:`--healthcheck-seconds 0` 时字段 omitempty 被省略可通过,但默认路径必失败,P0 成立。
-
-**验证方案**:复现——`mc replicate add rustfs/src --remote-bucket http://ak:sk@target/dst` 预期 400;修复后——madmin 形态 payload(60e9 ns)单测断言内部 Duration==60s;set→list 往返断言响应为纳秒;e2e 增加"mc 默认 payload"用例;持久化防御性读回归(旧秒格式升级后读取不变)。
-
-**解决方案(分阶段)**:
-1. **解阻塞**:从不支持清单移除 `healthCheckDuration`(能力契约版本号递增);请求按 `Duration::from_nanos` 解析(`total_downtime` 同步核查);调度上显式忽略并在契约/文档标注"接受但暂不生效";响应侧新增 DTO 按纳秒序列化(**勿直接改 `bucket_target.rs` 的 `duration_seconds`,它同时是持久化格式**);持久化读取加防御(≥10^7 视为纳秒),写入统一新格式。
-2. **落地语义**:`bucket_target_sys.rs:332-441` heartbeat 循环改为按 target 取值,对齐 MinIO(默认 5s、有下限)。
-3. **防复发**:建立容器内跑真 mc 命令的兼容 e2e 通道,覆盖 `replicate add/update/status`。
-
-### P0-8 站点复制架构 — **事实 CONFIRMED,定性部分 REFUTED,降级为 P1**
-
-**复核结论**:巨型文件(14614 行,非测试约 9533 行,24 个 handler)与三处反向导入全部属实;但"失察"定性被推翻:
-- `scripts/check_layer_dependencies.sh` **已建模并拦截**这些边,`layer-dependency-baseline.txt` 棘轮基线逐条列出全部 46 条存量反向边,**新增反向边 CI 必炸**;
-- `ecfs.rs` 被脚本刻意归类为 interface 层(有意的建模决策);
-- ARCHITECTURE.md 自己声明部分不变量 "currently violated... documenting them makes violations explicit and trackable";git 历史显示这是已知、受控、正在偿还的过渡态。
-- **结论:不构成正确性风险,从 P0 降为 P1(可维护性债务)**。真实成本:9.5k 行单文件的评审/合并冲突/增量编译负担,hook 直连使 app/storage 单测无法脱离 admin 层。
-
-**验证方案**:每阶段跑 `make pre-pr`;每消除一条反向边即**删除基线对应行**(而非重生成),使回归必炸;行为回归靠 site replication e2e + 路由快照测试 + `git diff --color-moved` 评审纯移动。
-
-**解决方案(分阶段)**:
-1. **解反向依赖(低风险,先做)**:复用 `site_replication_reconcile.rs` 已验证的 OnceLock 注册模式——bucket 三个 hook 在 app 层定义 fn-pointer 契约、admin 构建路由时注册;`node_service.rs` 的 reload 走 infra 层"运行时重载注册表"。注册缺失时显式降级(warn + no-op)。
-2. **文件拆分(纯移动)**:`site_replication.rs` → 模块目录:`transport`(peer client/DNS/TLS)、`gob`、`state`(注意 config key 路径不可变)、`iam_sync`、`heal`、`handlers`(24 个薄 handler)。
-3. **领域下沉(风险最高,最后做)**:hook 解耦后把 gob/transport/状态机移入独立 crate,注意全局状态清单(`docs/architecture/global-state-inventory.md:114`)。
-
-### P0-3 STS item 类型名不一致 — **CONFIRMED(双向硬断)**
-
-**复核结论**:成立,且两端都是**报错而非静默忽略**:MinIO 收到 `"sts-credential"` 走 default 分支返回 400 `errSRInvalidRequest`;RustFS 收到 `"sts-account"` 返回 NotImplemented。两端 heal/重试机制都会永久重试失败(MinIO 日志持续 "Unable to heal temporary credentials")。MinIO 当前版本 STS 复制发送面很广(AssumeRole/WebIdentity/ClientGrants/LDAPIdentity/Certificate 全系 + sftp/ftp + heal 路径)。除类型串外 `SRSTSCredential` 字段双方完全对齐——**只差这一个字符串**(推测 RustFS 实现时把 madmin 的 JSON 字段名 `stsCredential` 误当成了类型常量)。
-
-**影响面**:跨厂商 STS 临时凭证双向不复制(客户端在对端站点 `InvalidAccessKeyId`),纯可用性问题,无权限漂移;RustFS↔RustFS 自洽。
-
-**验证方案**:单测——出站产物断言 `type == "sts-account"`(改 `federated_identity.rs:497` 现有快照测试);入站构造 `"sts-account"` item 断言不落 NotImplemented。e2e——compose(RustFS+MinIO,root 凭证必须一致,否则 token 验签失败会误判修复无效):对 MinIO assume-role 拿临时凭证访问 RustFS,修复前 InvalidAccessKeyId、修复后成功;反向同测。
-
-**解决方案**:出站(`sts.rs:248`、`federated_identity.rs:241`)改发 `"sts-account"`(提常量集中定义);入站(`site_replication.rs:7857`)match 臂改 `"sts-account" | "sts-credential"`(**永久保留旧别名**兼容旧 RustFS peer)。滚动升级窗口内新→旧 RustFS 会降级(warn+重试,peer 升级后收敛);STS 凭证短生命周期,不建议为此拆两阶段发布。
-
-### P0-4 policy-mapping userType 数值错位 — **CONFIRMED(比指控更严重)**
-
-**复核结论**:数值表属实(RustFS: None=0/Svc=1/Sts=2/Reg=3;MinIO: unknown=-1/reg=0/sts=1/svc=2),wire 上确为数值、无翻译层。对抗复核修正与加重:
-- **RustFS→MinIO 方向今天"侥幸能用"**:RustFS 当前只出站 Reg=3 与组的 0,MinIO 对超范围值静默落 default 分支,恰好落对位置;
-- **MinIO→RustFS 方向三类断裂**:①**组映射硬失败(新发现)**——MinIO 组映射发 `UserType: -1`,RustFS `user_type: u64` 反序列化直接报错,整个 item 被拒,组→策略映射完全无法同步;②STS 用户映射(MinIO 发 1)被 RustFS 解释为 Svc,落错前缀/缓存,联邦用户在 RustFS 站点**静默丢权限**;③svc=2 被解释为 Sts,同类错位;
-- **低概率提权路径**:LDAP DN/OIDC 主体的映射被误存入常规用户缓存后,若本地恰有同名静态用户则继承本不属于它的策略——名字碰撞概率低但非零,这是保 P0 的理由。
-
-**验证方案**:单测——wire 编解码全矩阵(-1/0/1/2/3/非法值);e2e——MinIO 侧 `mc admin policy attach --group` 修复前 RustFS 查不到组实体、修复后可见;`mc idp ldap policy attach` 修复前落 `policydb/service-accounts/` 且访问被拒、修复后落 `sts-users/` 且放行;反向回归守住"侥幸兼容";混版本(旧+新 RustFS)双向 attach 互通。
-
-**解决方案(核心原则:不改 `UserType::to_u64/from_u64`)**——该编码被集群内部节点 RPC 使用(`node_service.rs:1513`),改动会破坏同集群滚动重启。只在站点复制 wire 边界加 MinIO 语义编解码:
-1. `SRPolicyMapping.user_type` 由 `u64` 改 `i64`(必须,才能收下 -1);
-2. 出站 `sr_wire_user_type`:Reg→0/Sts→1/Svc→2,组一律发 0(对 MinIO 与旧 RustFS 同时兼容);入站 `user_type_from_sr_wire`:-1→None/0→Reg/1→Sts/2→Svc/**3→Reg(旧 RustFS 别名,永久保留)**;
-3. 兼容矩阵已逐格验证:新↔旧 RustFS、MinIO↔新 RustFS 全通;唯一残余窗口(未来出站 Sts/Svc 映射对旧 RustFS 错读)当前不可达,在 doc comment 写明约束;
-4. 回归防护:编解码矩阵单测 + "wire 常量契约"字面值断言测试(防止将来被"顺手统一"回内部编码)+ e2e 进 P0 套件;顺带把 `SRCredInfo.iam_user_type` 一并改 `i64` 复用同一编解码,消除同族隐患。
-
-### P0-5 复制 PUT 缺 `?versionId=` query — **CONFIRMED**
-
-**复核结论**:所有反驳方向均失败,指控成立:
-- minio-go 官方复制端(v7.0.91)`api-put-object-streaming.go:767-776` 等三处全部是 `urlValues.Set("versionId", ...)`——**query,不是 header**;`x-minio-source-version-id` 这个 header 在 MinIO 全仓不存在,被静默忽略;
-- multipart 的版本在 **initiate 时**决定(`erasure-multipart.go:458-460`,为空即生成新 UUID),complete 不读 versionId;
-- aws-sdk-s3 `PutObjectInput` 无 versionId 成员属实,但 DELETE 路径已用 `.set_version_id()` 正确落 query,证明是遗漏而非不可行;
-- RustFS↔RustFS 不受影响的原因:RustFS 接收端有私有 header fallback(`options.rs:296-301`),恰好掩盖了 bug。
-
-**影响加重**:除版本漂移与按版本删除永久 no-op 外,目标校验/heal 用源 versionId `head_object` 永远 miss → **反复重传,目标端版本无限膨胀**。另有边缘缺陷:RustFS 内部 null 版本是 nil-UUID,直接发 query 会被 MinIO 当真实版本;minio-go 约定发字面 `"null"`。
-
-**验证方案**:L1 e2e(本仓可落地,红→绿)——复用 `crates/e2e_test/src/fake_s3_target/`(已解析 versionId query 并写 journal),断言 PutObject/CreateMultipartUpload 请求的 query == 源版本;L2 互操作(docker + 真 MinIO)`mc ls --versions` 断言目标 versionId == 源、删源版本目标同步消失;L3 单测 nil-UUID→`"null"` 映射。
-
-**解决方案**(`bucket_target_sys.rs`):`put_object`/`create_multipart_upload` 在 `map_request` 闭包内改写 URI 追加 `versionId` query(nil-UUID 映射 `"null"`);保留双 header 兼容旧版 RustFS 接收端;顺带核对 delete 路径的 nil-UUID 映射。**签名安全性已验证**:`map_request` 挂在 `modify_before_signing`,query 会进 canonical request,不会 SignatureDoesNotMatch。非版本化目标桶沿用"空则不发",`"null"` 值 MinIO 免检。
-
-### P0-6 CopyObject 不触发复制 — **CONFIRMED(附带加重发现)**
-
-**复核结论**:三个反驳方向全部不成立:
-- copy 直接调 `store.copy_object`,不经 put 路径;ecstore 层 copy 实现无任何调度;
-- **scanner 兜底不存在(关键)**:heal 入队条件是状态为 Pending/Failed 或手动 resync;而 copy 路径不 stamp PENDING(对照 put 路径 `object_usecase.rs:5255-5266`),状态为空 → heal 判定 Skip。
-- **加重发现**:copy 路径没有 MinIO `filterReplicationStatusMetadata` 的等价清理——COPY 指令下源对象的旧复制状态可能原样带到目的对象,**伪造 COMPLETED 假状态**。
-- 附带 P1(snowball `execute_put_object_extract`)同样确认:无 stamp 无 schedule。
-
-**影响面**:配复制规则的桶上,CopyObject 写入的对象(跨桶复制、rename 工作流、REPLACE 元数据更新)永不复制、scanner 不捞、仅手动 resync 可补;还可能带 stale 假状态。
-
-**验证方案**:e2e(参照 `replication_extension_test.rs` 双实例)——copy 后断言目的对象在目标桶超时内出现、源 COMPLETED、目标 REPLICA、无 stale 状态;snowball 参照 `snowball_auto_extract_test.rs` 加成员对象复制断言;usecase 单测用 `storage_api.rs:641` 现有 test-only 调用计数断言 copy/extract 触发决策与调度。
-
-**解决方案**(`object_usecase.rs`):
-1. `execute_copy_object` 在 `store.copy_object` 之前算一次 `dsc = must_replicate_object(...)`,`replicate_any` 时向 `dst_opts.user_defined` stamp pending + timestamp(严格镜像 put 路径,单一 dsc 决策贯穿两阶段);
-2. 同处清理源带来的复制状态 reserved 元数据;
-3. copy 成功、锁释放后 `schedule_object_replication`;
-4. `execute_put_object_extract` 对每个解出对象同样处理。
-风险已排除:replica 判定内置于 `must_replicate_object` 不会回环;self-copy 调度与 MinIO 一致。
-**落地顺序约束:先修 P0-5 再修 P0-6**——否则 copy 的失败重试经 heal 兜底后,只会在 MinIO 端制造更多漂移版本。
-
-### 第三方复审修正(2026-08-05,修复分支均已完成 review)
-
-**P0-5 修正**:问题的准确表述应为"**普通复制对象缺少可靠的源→目标版本身份策略**"——复制 PUT 只返回成功/失败,未捕获目标实际分配的版本 ID(已核实 `bucket_target_sys.rs` put 路径无 `res.version_id()` 捕获,delete 路径 :2030 有);multipart 只保留 upload ID。`fix/p0-5` 的 versionId query 方案对 MinIO/RustFS 目标成立(目标端沿用源版本 ID,身份问题消解),但对**忽略该私有 query 的目标(如原生 AWS S3)**身份问题仍在:目标自行生成版本 ID → 后续按源版本 ID 的删除复制落空。第三方建议定级 P1(修复已完成,残留缺口另行跟进):可选方案包括捕获 PUT 响应的 `x-amz-version-id` 并持久化源→目标映射。→ 记为 **P1-19(新增)**。
-
-**P0-6 修正**:scanner"兜底不存在"的表述过度。已核实 `crates/replication/src/operation.rs` `resync_target_for_object`:无 reset 记录且复制状态为 Empty 时返回 `replicate=true`,即 ExistingObjectReplication 启用时 scanner **可能最终补齐**空状态对象,无需手动 resync。准确结论:即时/同步复制语义失效(P0 定级依据),且以下场景**长期**漏复制——①源对象 COMPLETED 等复制元数据被 Copy 继承致误判(`fix/p0-6` 已修,清理先于决策);②显式 ExistingObjectReplication=Disabled;③其他无法进入 existing-object 补偿的场景。`fix/p0-6` 分支已含 copy 调度 e2e 与 stale 元数据白盒断言;**scanner 补偿边界的 e2e 仍缺** → 记为 **P1-20(新增)**。
-
-### 对抗性复核总览
-
-| 问题 | 复核结论 | 关键修正/加重 |
-|---|---|---|
-| P0-1 join 路径 | CONFIRMED,加重 | 路径在 MinIO 从未存在;现有单测固化错误;修复需同批改加密判定与空响应容忍 |
-| P0-2 SRInfo 大小写 | CONFIRMED,精确化 | 仅顶层 12 个无 tag 字段;preflight 失败先于 P0-1 触发 |
-| P0-3 STS 类型名 | CONFIRMED | 双向硬断、两端 heal 永久重试;只差一个字符串 |
-| P0-4 userType 错位 | CONFIRMED,加重 | MinIO 组映射发 -1 → RustFS u64 解析硬失败;存在低概率名字碰撞提权路径;修复不得触碰内部 RPC 编码 |
-| P0-5 versionId query | CONFIRMED,加重 | heal 反复重传致目标版本膨胀;nil-UUID 需映射 "null" |
-| P0-6 CopyObject | CONFIRMED,加重 | scanner 兜底不存在;stale COMPLETED 假状态;须在 P0-5 之后落地 |
-| P0-7 healthCheckDuration | CONFIRMED | 单位错误双向独立成立;有单测固化拒绝行为 |
-| P0-8 架构 | 事实 CONFIRMED,定性 REFUTED | 反向边被棘轮基线锁死,降级 P1(受控技术债) |
-
----
-
-## 六、修复路线图(2026-08-05 更新)
-
-**✅ 第一批已完成**:全部 7 项 P0 已修复并创建 PR(见第二节修复状态;P1-10 snowball 随 #5753 一并修复)。待合并,注意顺序约束:#5748+#5749 同批、#5752 先于 #5753。
-
-**第二批(数据一致性优先,采纳功能视角复核定级)**
-1. **P1-21** delete-marker 延迟 purge 失败静默丢弃(复核升级,数据一致性,建议单独小 PR + 失败注入测试)
-2. **P1-19** 源→目标版本身份策略(捕获 PUT 响应 `x-amz-version-id` / 持久化映射,覆盖非 MinIO 系目标)
-3. **P1-1** ILM expiry 同步语义(只传播 expiry、保留接收端本地 transition + 对应测试)
-4. **P1-15** site state RMW 分布式锁统一(对齐 repair state 的 `with_config_object_write_lock` 模式)+ 双写者回归测试
-5. **P1-22** 桶复制 SSE 能力(普通复制/Heal/Resync/Multipart 四模式,先补 encrypted-object e2e 钉现状)
-
-**第三批(mc 可观测性与互操作补齐)**
-6. P1-11/12/14 mc 响应结构 serde rename(改动小、消除静默空输出)
-7. P1-7 ARN 解析侧兼容 `arn:minio:` 前缀
-8. P1-5 GET/HEAD proxy、P1-6 时间戳头、P1-3 自动跨站 heal
-9. P1-20 scanner 补偿边界 e2e;P0-7 阶段 2(per-target 心跳 + healthcheck update op)
-10. P2-26 GET 补 `x-amz-replication-status`(约一行)+ 回归测试;P2 清单其余项
-
-**第四批(架构与长期)**
-11. P0-8(降级 P1)架构:先解 3 处反向依赖(复用 reconcile 注册模式),再拆分/下沉站点复制领域模块
-12. P1-16 类型对账测试、P1-17 迁移完成判据、P1-8 配置校验补齐

P1_REVIEW_AND_PLAN.md

@@ -1,195 +0,0 @@
-# P1 逐条复审订正与方案计划
-
-> 复审基线:main @ `77f2b948c`(7 个 P0 修复 #5748~#5754 已全部合入)
-> 复审方式:5 组对抗性复审 agent 并行,先怀疑后确认;以 RustFS 自身功能契约为正确性标准,不以"未对齐 MinIO"为根因;RustFS 更优/独特设计标注"保持不变"
-> 参照:MinIO 源码、mc@cf909e1063a9、madmin-go v3.0.109、minio-go v7.0.91
-> 日期:2026-08-06
-
----
-
-## 〇、复审总裁定表
-
-| 项 | 主题 | 复审结论 | 关键订正 | 工作量 |
-|---|---|---|---|---|
-| P1-1 | ILM expiry 复制语义 | CONFIRMED(范围扩大) | 发送点共 4 处非 1 处;接收端无门禁;修复重心移到接收端 merge | M |
-| P1-3 | 自动跨站元数据 heal | CONFIRMED(范围收窄) | 真实缺口="retry queue 有账本无消费者";不移植 MinIO 全量 heal | M |
-| P1-5 | GET/HEAD 远端 proxy | CONFIRMED | 同步复制模式是已实现的部分缓解(保持不变);proxy 指标语义被出站 HEAD 污染 | L(P0 段 M) |
-| P1-6 | 三类时间戳头收发 | CONFIRMED(缺口扩大) | 实为三段缺失:tagging 无本地写入方 + 不发头 + 接收端无 LWW 合并点 | M |
-| P1-7 | ARN 前缀不互认 | CONFIRMED+(加重) | 新发现 FromStr id/region 互换 bug;madmin ParseARN 硬校验实锤 → 生成侧必须改 | M |
-| P1-8 | 配置校验缺口 + StorageClass | 部分 CONFIRMED | 2MB 子项 REFUTED(MinIO 亦无);StorageClass 属刻意设计成立(MinIO 也不消费 rule 级,target 级 RustFS 已生效)| S |
-| P1-11 | replication-metrics snake_case | CONFIRMED | BucketStats 复用内部 RPC 线格式实锤 → 必须独立响应 DTO | M |
-| P1-12 | replication-reset 响应壳 | CONFIRMED(面缩小) | 致命键仅 5 个(壳 `Targets`≠`target` + 4 个字段名);其余靠 Go 大小写不敏感能对上 | S |
-| P1-13 | mrf/diff 聚合响应 | CONFIRMED(症状加重) | 实际输出**伪数据行**而非空;diff/mrf 数据源均可支撑逐条流 | diff S / mrf M |
-| P1-14 | set-remote-target 请求体 | 原缺口已缓解;**新 CONFIRMED 阻断** | #5754 后 26 字段已全覆盖;但**零值 `expiration` 恒被拒 → mc replicate add 仍 100% 失败**;latency 单位 round-trip 污染 | S(**建议立即修**) |
-| P1-15 | site state RMW 竞争 | CONFIRMED(加重) | hook 路径 enqueue/dequeue 同进程内绕过既有 Mutex → 单节点即可触发 | M-L |
-| P1-16 | 状态机类型双份定义 | CONFIRMED(加重+收窄) | drift 已发生(MrfOpKind 两侧不一致);但 filemeta 侧 worker DTO 是死代码,活跃双份仅 3 个 wire 类型;"抽公共 crate"否决 | S+M |
-| P1-17 | 桶复制逻辑分裂 | CONFIRMED;微文件合并子项 REFUTED | boundary 微文件是棘轮机制的机械接缝(守护脚本按文件名锚定),合并负收益;缺的是完成判据 | M0=S,整体 L |
-| P1-18 | 超长函数 | 行数 CONFIRMED;apply_iam_item 降级 | apply_iam_item 长而不复杂(6 臂 dispatch),不拆降 P2;其余 4 个给纯移动拆分草案 | M |
-| P1-19 | 源→目标版本身份策略 | CONFIRMED(范围收窄) | delete-marker 的"捕获+持久化映射"模式已落地(保持不变);推荐能力探测+显式拒绝而非全量映射 | M |
-| P1-20 | scanner 补偿边界 e2e | CONFIRMED(缺口收窄) | 决策函数单测与 Failed-heal e2e 已存在;缺 existing-object 矩阵与 Replica 防环 e2e;附完整入队真值表 | M |
-| P1-21 | delayed purge 静默丢弃 | CONFIRMED | 映射损坏防护已加固(保持不变);`let _ =` 与无 MRF 通道仍在;附带发现 MRF outcome 恒 false 滞留问题 | M |
-| P1-22 | 桶复制 SSE 能力 | CONFIRMED(前提订正) | SSE-S3 自 #5633 已 fail closed,被 ignore 的 e2e 理由过期(先摘 ignore);SSE-C 缺的是目标侧头摄取 | L(4 阶段) |
-
-**"保持不变"清单(复审确认的 RustFS 更优/刻意设计,不纳入修复)**:per-PUT 即时元数据传播 hook(优于 MinIO 纯周期 heal)、单向推送+stale 守卫收敛模型、delete 走 merge-with-empty(优于 MinIO 整删)、delete-marker 版本映射持久化+损坏拒猜、同步复制模式(partition_by_sync)、能力契约式显式拒绝+`deny_unknown_fields`(字段清单已与 madmin v3.0.109 同步)、StorageClass 显式拒绝非 STANDARD(target 级已真正生效)、replication-check 真实探针写删、响应中的 RustFS 增强字段(ResetBeforeDate/Error/可观测性键,Go 忽略未知键可共存)。
-
----
-
-## 一、紧急项(建议立即处理)
-
-### ⚡ P1-14 新阻断:零值 `expiration` 拒绝 → mc replicate add 仍 100% 失败
-
-- **证据**:Go `omitempty` 不省略零值 `time.Time`(已用 Go 程序按 madmin 逐字 tag 实测),mc/madmin marshal 恒输出 `"credentials":{"expiration":"0001-01-01T00:00:00Z"}` 与 `"resetBeforeDate":"0001-01-01T00:00:00Z"`;RustFS `handlers/replication.rs:286-291` 对 `expiration.is_some()` 一律 400。#5754 的测试全部用手写 payload(`expiration: None`),未被现网形状打中。
-- **修复(S)**:①`expiration` 改"非 Go 零值时间才拒"(与 `sessionToken` trim-empty 判断对称);②`latency` 请求字段直接忽略(消除 #5754 后纳秒响应 ↔ 毫秒请求的 round-trip 1e6 倍污染);③把"Go 真实 marshal 形状 payload"固化为测试夹具惯例。
-- **红灯测试**:用实测 Go marshal 全形状 body(含零值 expiration/resetBeforeDate/latency{0,0,0}/edge:false/healthCheckDuration:60000000000)打 set-remote-target,期望 200;非零 expiration 仍 400(能力契约保持)。
-
-### ⚡ P1-7 附带 bug:ARN FromStr 字段互换
-
-`arn.rs` Display 输出 `{type}:{region}:{id}:{bucket}`,FromStr 却读 `id=parts[3], region=parts[4]`——id 与 region 互换。当前仅因消费方只用 arn_type 而潜伏。随 P1-7 一并修。
-
----
-
-## 二、逐项方案计划
-
-### P1-1 ILM expiry 复制语义(M)
-
-**订正后事实**:发送完整 lifecycle XML 的路径 4 处——PUT hook(`bucket_usecase.rs:2177-2180`)、DELETE hook(`:1512-1514`,触发接收端**整删**)、import(`bucket_meta.rs:948-951`)、build_sr_info/bootstrap(`site_replication.rs:4190,2241-2249`);接收端 `apply_bucket_meta_item`(`:7669-7683`)整体覆盖/删除,且**无 `replicate_ilm_expiry` 门禁**。P0 后已有缓解(发送开关、bootstrap 跳过、stale 判定)只解决"发不发/新旧",不解决"发什么/怎么合"。
-
-**方案**:接收端 merge 为主(信任边界),发送端 expiry-only 提取为辅:
-1. 新增纯函数 `extract_expiry_only(cfg)` 与 `merge_expiry_rules(local, incoming)`——语义对齐 MinIO `mergeWithCurrentLCConfig`,两处 RustFS 改进:incoming 一律先剥 transition(防旧端);`None` 走 merge-with-empty 而非整删(**MinIO 整删连本地 transition 一起删是缺陷,不照抄**);
-2. 接收端 lc-config 分支改 读→merge→条件写/删,保留 stale 判定与 incarnation 守卫;补 `replicate_ilm_expiry` 门禁;
-3. 4 个发送点接 `extract_expiry_only`;expiry 判定用 RustFS 口径(含 `del_marker_expiration`)。
-
-**红灯测试**:L1 单测 5 例(提取剥离/合并保留 T/防御剥离/merge-with-empty/import 无 transition);L3 e2e——B 配本地 transition,A PUT expiry → B 两者共存;A DELETE lifecycle → B transition 仍在。
-**兼容**:旧端发完整 XML → 新接收端剥后 merge 正确;新端 expiry-only → 旧接收端仍整覆盖(不劣于现状)。规则按 ID 对齐,`rule-{idx}` 撞名同 MinIO 语义,文档注明。
-
-### P1-3 自动跨站 heal → 改为"retry queue 自动 drain"(M)
-
-**订正后事实**:retry queue 是现成增量账本(失败即入队 `:3243-3262`,持久化于 state,`retry_count` 字段存在)但**全库无消费者**;手动 repair 是本地快照单向推送,收敛方向依赖运维判断。即时 hook + 显式 repair 模型保持不变。
-
-**方案**:
-- 阶段 1(核心):周期任务挂进现有 reconcile ticker,per-event 重发(body 从本地当前元数据重建,复用 `SiteReplicationRepairTask::send`,天然发"当前值"+对端 stale 守卫幂等);指数退避(`retry_count`+上限转 failed);drain 全程包分布式锁去抖(先用 `with_config_object_write_lock` 专用对象,P1-15 落地后并入统一 state store);结构化 tracing 汇总一条。
-- 阶段 2(可选,默认关闭):每 N tick 比对 repair plan token,不同才自动 dry-run→execute。**不移植** MinIO 跨站取最新 pull 语义(各站各自 drain 即双向收敛)。
-
-**红灯测试**:L2——state 带 retry event,调 `drain_site_replication_retry_queue()`(现不存在),fake peer 成功后断言队列清空;退避断言。L3——停 B→A PUT policy 失败入队→起 B→drain 后 B 收到且 SRRetryStats 归零。
-
-### P1-5 GET/HEAD 远端 proxy(L;P0 段 M)
-
-**订正后事实**:`SUFFIX_SOURCE_PROXY_REQUEST` 零消费者;`ProxyMetric` 字段与 admin 汇总通路已就位,但 resyncer 把**出站** HEAD 计入 `head_total` 污染语义;`disable_proxy` 管道存在无人消费;同步复制模式(`partition_by_sync`,`replication_pool.rs:2667-2689`)是部分缓解但不等价(手动 per-target、失败仍 404、不覆盖兜底窗口)。防环头当前仅潜在问题,但 proxy 实现与防环识别**必须同 PR**(否则 RustFS↔RustFS 成环)。
-
-**方案**(P0 段):新增 `replication_proxy_boundary.rs`——`proxy_targets`(version_suspended/入站 proxy 头/disable_proxy 三重 gate)+ `proxy_get/head_to_replication_target`(走现有 TargetClient,range/条件头透传);触发点在 usecase 层 NotFound/VersionNotFound 分支;接收侧 options.rs 解析防环头,出站双前缀发送;`tokio::timeout`(~3s env 可调)、仅 2xx 采纳其余回落本地 404、复用离线标记短路;指标接 `record_replication_proxy` 并纠正 resyncer 计数语义。P1 段:tagging 三操作 proxy(依赖 P1-6)。
-**红灯测试**:e2e 双站断复制链路后从对端 GET/HEAD 应 200(现 404);防环负例(带头请求不转发、计数不增);降级负例(target 全离线时限时 404);disable_proxy 负例。
-
-### P1-6 时间戳头收发(M;三段修复)
-
-**订正后事实**:①`SUFFIX_TAGGING_TIMESTAMP` 全仓无写入方(retention/legalhold 已有双前缀写入);②`PutObjectOptions::header()` 只序列化 4 个内部头,三类时间戳被丢弃,multipart 同;③接收端不解析,且 replica PUT 是 verbatim 覆盖——解析后必须在写盘前与本地版本做 per-类别 LWW 合并才有效;④`AdvancedPutOptions` 默认 `now_utc()` 无法当"未设置"哨兵,需 Option 化。
-
-**方案**:阶段 0——`put/delete_object_tagging` 落 `SUFFIX_TAGGING_TIMESTAMP`(双前缀);阶段 1——新增三个 suffix 常量(对齐 MinIO headers.go:239-243),三字段 Option 化,`header()` 与 multipart 条件序列化;阶段 2——接收端解析(仅授权复制请求)+ PUT 路径 LWW 合并并持久化赢家时间戳(合并仅限三类元数据,不触碰数据与其余元数据,与 verbatim-replica 不变式共存)。
-**红灯测试**:单测 header 双前缀序列化断言/未设置缺席断言;接收端解析单测;e2e active-active tagging 并发收敛(晚者胜,现 main 旧值覆盖新值为红)。
-
-### P1-7 ARN 前缀(M)
-
-**订正后事实**:madmin `ParseARN` 硬校验 `arn:minio:` 前缀 + ID/bucket 非空(v3.0.109 remote-target-commands.go:50-63);mc 爆炸点仅 `replicate update`(fatalIf)与 `replicate ls`(软降级);`replicate add` 把 ARN 当不透明串不受影响——解释了"add 通 update 挂"。RustFS ARN 结构(`type::id:bucket`)与 madmin 兼容,仅 vendor token 障碍;另有 FromStr id/region 互换 bug(见紧急项)。
-
-**方案(推荐路线 A)**:生成侧默认改 `arn:minio:`(留常量可品牌化);解析侧接受双前缀(存量 `arn:rustfs:` 靠双前缀解析 + 现有字符串等值匹配继续工作);修字段序;改 `generate_arn`、`site_replication.rs:6329` 与相关测试断言。混合版本集群前缀不一致靠双前缀解析吸收;不做存量数据前缀归一化改写。
-**红灯测试**:单测 `from_str("arn:minio:replication:us-east-1:depl:bucket")` 成功且 id/region 正确(现双重红灯);round-trip 属性测试;e2e set-remote-target 返回 ARN 可被 madmin 语义解析、预置 `arn:minio:` 目标可 remove。
-
-### P1-8 配置校验(S)
-
-**订正后事实**:2MB 上限 REFUTED(MinIO 亦无显式检查,剔除);StorageClass 已缓解且刻意设计成立——MinIO 自己也不消费 rule 级 `Destination.StorageClass`(复制 PUT 用 target 级 `tgt.StorageClass`),RustFS target 级 storage_class 已真正生效(`bucket_target_sys.rs:1633-1634`),容忍显式 STANDARD 已实现。仍缺:规则数≤1000、≥1 条、Priority 唯一非负、ID≤255、Filter 互斥、Tag×DeleteMarkerReplication 互斥、sameTarget 拒绝。
-
-**方案**:`config.rs` 新增 `validate_replication_config_structure` 纯函数,`bucket_usecase.rs:2418` 接入;StorageClass 保持现状+契约文档化("rule 级请改用 remote target 的 storageclass 字段")。
-**红灯测试**:单测逐格(1001 规则/重复 Priority/256 字符 ID/Filter 并存/Tag+DMR)期望特定错误;e2e aws-sdk 形状 XML 断言 InvalidRequest。
-
-### P1-11 replication-metrics DTO(M)
-
-**订正后事实**:`BucketStats` 走内部 peer RPC 线格式(`rmp_serde::to_vec_named` 字段名入线,node_service.rs:1401 / peer_rest_client.rs:88-104)——**改原结构 serde 名会破坏混合版本集群 RPC,禁止**;必须走 #5754 的响应 DTO 模式(同文件先例 `remote_target_admin_json`)。
-
-**方案**:新增仅 Serialize 的 `MetricsV2Dto{uptime,currStats,queueStats,downtimeInfo}`/`MetricsDto`/`TargetMetricsDto`,显式映射(`q_stat`→`queued`、`bandwidth_limit_bytes_per_sec`→`limitInBits`、failed→TimedErrStats total-only);`queueStats.nodes` 先填本机一条;RustFS 可观测性扩展键保留(Go 忽略未知键,双栖零成本)。
-**红灯测试**:e2e 用镜像 minio-go MetricsV2 tag 的结构反序列化断言 `currStats.completedReplicationSize > 0`(现全零);DTO 键名 snapshot 单测。
-
-### P1-12 replication-reset 响应壳(S)
-
-**订正后事实**:致命键仅 5 个——壳 `Targets`≠`target`、`Status`≠`resyncStatus`、`ReplicatedSize`≠`completedReplicationSize`、`ReplicatedCount`≠`replicationCount`、`FailedSize/FailedCount`≠`failedReplicationSize/failedReplicationCount`;其余(Arn/ResetID/StartTime/...)靠 Go 大小写不敏感能对上;`ResetBeforeDate`/`Error` 是增强字段可保留。响应结构是 router.rs 独立 DTO 无内部复用,改名零风险。
-
-**方案**:纯 serde rename(建议全字段精确对齐 madmin 小写形态),保留增强键+文档标注。
-**红灯测试**:e2e 断言响应含 `target` 数组且 `target[0].resetid` 非空、status 侧 `resyncStatus`/`completedReplicationSize` 键存在。
-
-### P1-13 mrf/diff 流式响应(diff S / mrf M)
-
-**订正后事实**:症状比"输出空"更糟——聚合对象会被 madmin `json.Decoder` 成功解码一次,`mc replicate backlog` 输出一条 object 为空的**伪行**(静默伪数据);路线 A(保持聚合+文档化)无法消除伪行且与 madmin 同 path 无内容协商,**不可行**。数据源评估:diff 已逐条扫描只需去壳;mrf 的 durable backlog(`MrfReplicateEntry` 字段恰好覆盖 `ReplicationMRF` 所需)已可枚举。
-
-**方案(路线 B)**:diff 去壳输出 NDJSON `DiffInfo` 形状(仅 `IsDeleteMarker`/`ReplicationStatus` 需 rename;truncation 信息入日志不入流);mrf 遍历 durable entries 逐条输出 `ReplicationMRF` 形状(nodeName 填本机);聚合响应保留在 `?aggregate=true`(RustFS 扩展,deliberate 注释随迁)。条目量有 `REPLICATION_DIFF_MAX_SCAN` 封顶,内存拼 NDJSON 即可不必真流式。
-**红灯测试**:e2e 制造失败复制后逐行反序列化断言至少一条 `object` 非空(现为伪空行);diff 断言无 `Entries` 壳。
-
-### P1-14 set-remote-target(S,含紧急项)
-
-见"一、紧急项"。另:`deny_unknown_fields` **保留**(推荐)——字段清单已与 madmin v3.0.109 全同步,严格模式+显式清单兼得契约哲学与防静默;代价写进维护清单:"madmin 版本升级时同步字段清单"(加对照 madmin tag 列表的常量测试防漂移)。
-
-### P1-15 site state 统一 store(M-L,两 PR)
-
-**订正后事实**:主 state 有进程内 Mutex(`:347`)但两处不完备——①无分布式锁(多节点 RMW 丢更新);②**retry event enqueue/dequeue 不持锁**(挂在所有 hook 广播路径上,同进程即可丢更新);reload 路径完全无锁(稳态不写盘收窄窗口,迁移期可覆盖并发写)。repair state 的 `with_config_object_write_lock` + no-lock IO 是正确样板(`:1097-1114`);两套归一化的语义差异(JSON-level 容忍畸形 peer)是**有意的**,统一时必须保留。锁序注释 `:346` 可挂靠。
-
-**方案**:PR1——新建 `admin/site_replication_state.rs`:两阶段归一化合一(JSON 宽容清洗→类型化)、`read_state()/update_state(F)`(分布式锁包完整 RMW,锁内禁网络调用与嵌套配置锁)、常量收敛;service reload 接入;迁移 service 侧 5 个归一化测试保语义。PR2——迁移全部 ~30 个 RMW 调用点(含 enqueue/dequeue),**移除**进程内 Mutex(避免双锁新顺序约束);dequeue 热路径保留"先无锁读、命中才进 update_state"两段式;更新锁序注释。每个调用点做重入审查(现有 drop-reacquire 模式保持)。
-**红灯测试**:L2 单进程并发——持锁 RMW(mark_pending_rotation_peer_acked)×绕锁写者(enqueue_retry_event)注入交错,断言最终 state 两者共存(现必丢其一,确定性红灯);L1 归一化等价性测试迁移;L3 双节点并发(nice-to-have)。
-**风险**:盘上格式不变;锁超时从"静默丢更新"变"显式报错",hook 路径保持 warn 不阻断 S3 主路径。
-
-### P1-16 类型对账护栏(S)+ 死代码清理(M)
-
-**订正后事实**:drift 已发生(filemeta 侧 `MrfOpKind` 缺 Metadata/Heal/ExistingObject 三 variant、`MrfReplicateEntry` 缺 force_delete/target_arns)——但 filemeta 侧 8 个 worker DTO 全是**死代码**(零消费者);活跃双份仅 `ReplicationStatusType/VersionPurgeStatusType/ReplicationState` 三个 wire 类型(filemeta 绑 xl.meta 磁盘格式,replication 绑 MRF/resync 持久化格式);boundary 枚举转换 `as_str()` 兜底 `_ => Empty` 会静默降级。"抽公共 leaf crate"否决(两 wire 格式演进节奏不同,迁移规则 #12 本意是所有权独立)。
-
-**方案**:Step 1(S,即刻)——boundary 加对账测试:两侧枚举穷尽 match(新增 variant 即编译失败)+ as_str 双向 round-trip + ReplicationState 全字段往返;Step 2(M)——清理 filemeta 侧 ~600 行死代码 DTO,注意 crates.io semver(先 `#[deprecated]` 一版再删);Step 3(S)——replication 侧注释指向对账测试。
-
-### P1-17 迁移完成判据(M0=S;整体 L)
-
-**订正后事实**:"合并 boundary 微文件"REFUTED——守护脚本按具体文件名锚定每个 boundary,合并要同步改脚本+mod+导入点而功能收益为零;微文件是棘轮机制的机械接缝。唯一可退役:`datatypes.rs`(消费者迁完即删)。README 建议的第一步(event sink/runtime boundary)实际已部分落地,文档滞后。
-
-**方案**:M0(S)文档 PR——完成判据 = Required Contracts 表 "Current dependency to remove" 列清空;终态 = pool/resyncer/state 移入 crates/replication,boundary 随 crate 移动自然消解;更新 split-plan "Proposal only" 状态。M2(M)resyncer 纯决策逻辑下沉;M3(L)trait 稳定后移 worker 运行时(全计划唯一高危段,最后做);M4(S)统一退役 boundary 与守护条目。**不做**批量合并微文件。
-
-### P1-18 超长函数拆分(M;4 个 PR)
-
-**订正后事实**:行数确认(resync_bucket 537 / start_mrf_processor 306 / replicate_all 409 / delete 路径 replicate_object 299 / apply_iam_item 255);`apply_iam_item` **降级 P2 不拆**(6 臂 dispatch,每臂线性短小,拆分违反 "Prefer direct, local code");`replicate_object` 有两个同名体,原清单指 delete 路径 trait impl。
-
-**方案**(每函数独立 PR,纯移动,`git diff --color-moved=dimmed-zebra` 验证):
-1. `resync_bucket`(最优先,三处历史并发 bug 注释所在):acquire_resync_leadership / load_resync_replication_config / spawn workers+collector 三段抽出,并发 bug 注释随代码移动,每个 return 前的 mark_status 逐一保持;
-2. `start_mrf_processor`:抽 `reconstruct_mrf_delete/object` 纯函数(主循环 -150 行,重建逻辑可单测);
-3. `replicate_all` + delete 路径 `replicate_object`:各拆 3-4 个阶段 helper;**明确不合并两函数**(delete-marker 404/405 校验语义是刻意差异)。
-**排序依赖**:先 P1-18 拆分、后 P1-17 M2/M3 迁移(小函数降低搬运风险)。
-
-### P1-19 版本身份策略(M,推荐方案 B)
-
-**订正后事实**:#5752 已合入(PUT/multipart initiate 带 query,RustFS 目标侧也支持);PUT 响应 `x-amz-version-id` 仍被丢弃(`:1891 Ok(_)`);**delete-marker 子案已系统性缓解**——`remove_object` 捕获目标版本号→`target_delete_marker_version_ids` 持久化进 xl.meta(含上限与损坏标记)→延迟 purge 优先用映射、损坏拒猜(**保持不变**);RustFS 无"仅支持 MinIO 目标"契约声明;replication-check 探针已捕获响应版本号但不比对。MinIO 同样丢弃响应版本号(平价),RustFS 已有两点增强。
-
-**方案对比**:A 全量映射持久化(完整但 xl.meta 膨胀、全链路改造,L);**B(推荐)**:契约=仅支持"沿用源版本 ID"的目标,在 replication-check 增加 VersionFidelity phase(探针 PUT 带 versionId query,比对响应版本号)+ `validate_target` 复用同一探测,不镜像则新错误 `BucketRemoteTargetVersionMismatch` 显式拒绝/告警(M);C 混合(无需求支撑)。探针是主动写,进 validate_target 会扩 set-target 副作用面——可先只做 check phase + 运行期首次 PUT 抽查告警。
-**红灯测试**:FakeS3Target 加 `assign_own_version_ids` 开关模拟原生 S3,断言版本删除复制落空(现红)与探测后显式拒绝(修后绿)。
-
-### P1-20 scanner 补偿边界 e2e(M,纯测试)
-
-**订正后事实**:决策函数单测(queue.rs 7 例等)与 scanner 驱动的 Failed-heal e2e(target 断电恢复/源重启重放,FAST_SCANNER_ENV)已存在;真实缺口=无任何"先写对象→后配复制"的 existing-object 用例。完整入队真值表已梳理(见复审记录):Enabled×Empty 补齐、Pending/Failed 恒补(不受 existing 开关影响)、Disabled×Empty 永不补、Replica 恒不补(防环)、null-version 永不入队、reset_id 重置补齐。
-
-**方案**:e2e 矩阵 1-2 个用例(先 PUT 四种来源对象含 Copy/Snowball 产物→后配 Enabled/Disabled 规则→正例 wait_for_replicated_object / 负例 assert_failed_replication_stays_absent_for ≥3 周期,**"永不补齐"是契约必须显式断言**)+ Replica 防环变体 + queue.rs 补 2 格单测;null-version 跳过行为先写"记录现状"断言并注明出处。不改产品代码。
-
-### P1-21 delayed purge 失败处理(M)
-
-**订正后事实**:静默点两处——target client 缺失 `continue` 无日志(`:1673-1675`)、`let _ = remove_object`(`:1693-1700`);5 次循环是等源 marker 消失非重试;purge 调用后无条件 break;MRF 入队接口(`queue_replica_delete_task`,队满自动落盘)同 crate 可用无分层障碍;映射优先/损坏拒猜是已加固项保持不变。**附带发现**(建议单独跟进):`requires_delayed_purge` 恒真使 delete-marker 类 MRF 条目 outcome 恒 false → 重放永远 Missed 保留,可能永久滞留。
-
-**方案**:①purge 函数返回 per-target 成败,失败 warn(带 event 常量)+ metrics,client 缺失同样 warn(S);②循环内失败重试、轮次耗尽入 MRF、入队失败 warn+metric 兜底(S/M);③两层失败注入测试(mock 503 断言重试/状态/MRF;FakeS3Target inject 断言故障清除后最终收敛)(M)。风险:MRF 重放重发 DELETE marker 创建——mtime 幂等,风险低。
-
-### P1-22 SSE 能力(L,4 阶段)
-
-**订正后事实**:fail-closed 由 #5633 引入(`replication_target_boundary.rs:101-174`),普通/Heal/Resync/Multipart 全走同一函数;SSE-C 发送半边已建(内部头→`X-Rustfs-Replication-*` 映射+CRC),**目标侧摄取代码完全缺失**(链路必断,e2e 已钉 FAILED);SSE-S3 契约 e2e 的 `#[ignore]` 理由(backlog#1291 silently drops)已被 #5633 过期;直传托管 SSE 不可行(封存密钥绑本站 KMS),MinIO 是源解密+目标重加密;ecstore 已有 `ObjectEncryptionResolver` trait seam,解密不破分层。
-
-**方案**:阶段 0(S)摘 ignore + 补 encrypted resync/heal e2e 钉全矩阵 fail-closed 现状;阶段 1(M)SSE-C 目标侧头摄取+加密尺寸/CRC(MinIO :1670-1740 参照);阶段 2(M/L)SSE-S3 经 resolver 解密+目标 AES256 重加密(resolver 未注册必须继续 fail closed;multipart 按明文尺寸分片);阶段 3(L)SSE-KMS + key id 随行开关(目标站无同名 key 显式失败,禁止回退 SSE-S3)。过渡期全矩阵维持 fail closed,禁止明文降级。
-
----
-
-## 三、执行批次建议
-
-| 批次 | 内容 | 性质 |
-|---|---|---|
-| **B0 立即** | P1-14 零值 expiration + latency 忽略(S);P1-7 FromStr 字段互换(并入 P1-7 或先行) | mc 阻断修复 |
-| **B1 小改动高收益** | P1-12 响应壳 rename(S)、P1-13 diff 去壳(S)、P1-8 结构校验(S)、P1-16 Step1 对账测试(S)、P1-17 M0 文档判据(S)、P1-22 阶段 0 摘 ignore(S) | serde/校验/测试护栏 |
-| **B2 数据一致性** | P1-21 purge 失败处理(M)→ P1-20 scanner 矩阵 e2e(M,纯测试)→ P1-19 方案 B 能力探测(M)→ P1-15 state store PR1+PR2(M-L) | 一致性核心 |
-| **B3 互操作补齐** | P1-7 ARN 路线 A(M)、P1-11 MetricsV2 DTO(M)、P1-13 mrf 流(M)、P1-6 时间戳三段(M)、P1-1 ILM merge(M)、P1-3 retry drain(M) | mc/跨站语义 |
-| **B4 大功能与架构** | P1-5 proxy P0 段(M→L)、P1-22 阶段 1-3(L)、P1-18 四函数拆分(M)→ P1-17 M2-M4(L)、P1-16 Step2 死代码(M) | 长期 |
-
-**批内依赖**:P1-6 先于 P1-5 的 tagging proxy;P1-18 先于 P1-17 M2/M3;P1-15 PR1 的锁对象可先供 P1-3 drain 使用。

README.md

@@ -116,7 +116,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.12
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-rc.1
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command

README_ZH.md

@@ -113,7 +113,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # 使用指定版本运行
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.12
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-rc.1
 ```
 
 如果您通过绑定挂载启用 TLS 证书目录，也请用同样方式准备该目录：

crates/common/Cargo.toml

@@ -39,11 +39,15 @@ tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
 tonic = { workspace = true, features = ["gzip", "deflate"] }
 uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
 chrono = { workspace = true, features = ["serde"] }
+jiff = { workspace = true, features = ["serde"] }
 metrics = { workspace = true }
 serde = { workspace = true, features = ["derive"] }
 rmp-serde = { workspace = true }
 s3s = { workspace = true, features = ["minio"] }
 tracing = { workspace = true }
 
+[dev-dependencies]
+serde_json = { workspace = true }
+
 [lib]
 doctest = false

crates/common/src/lib.rs

@@ -19,6 +19,7 @@ pub mod heal_channel;
 pub mod last_minute;
 pub mod metrics;
 mod readiness;
+pub mod table_catalog;
 
 pub use globals::*;
 pub use readiness::{GlobalReadiness, SystemStage};

crates/common/src/metrics.rs

@@ -15,6 +15,7 @@
 use crate::heal_channel::HealScanMode;
 use crate::last_minute::{AccElem, LastMinuteLatency};
 use chrono::{DateTime, Utc};
+use jiff::Timestamp;
 use serde::{Deserialize, Serialize};
 use std::{
     collections::{BTreeSet, HashMap},
@@ -669,7 +670,7 @@ impl LockedLastMinuteLatency {
 #[derive(Clone, Debug)]
 struct CurrentPathState {
     path: String,
-    updated_at: DateTime<Utc>,
+    updated_at: Timestamp,
 }
 
 struct CurrentPathTracker {
@@ -678,10 +679,10 @@ struct CurrentPathTracker {
 
 impl CurrentPathTracker {
     fn new(initial_path: String) -> Self {
-        Self::new_at(initial_path, Utc::now())
+        Self::new_at(initial_path, Timestamp::now())
     }
 
-    fn new_at(initial_path: String, updated_at: DateTime<Utc>) -> Self {
+    fn new_at(initial_path: String, updated_at: Timestamp) -> Self {
         Self {
             state: Arc::new(RwLock::new(CurrentPathState {
                 path: initial_path,
@@ -693,7 +694,7 @@ impl CurrentPathTracker {
     async fn update_path(&self, path: String) {
         let mut state = self.state.write().await;
         state.path = path;
-        state.updated_at = Utc::now();
+        state.updated_at = Timestamp::now();
     }
 
     async fn get_state(&self) -> CurrentPathState {
@@ -701,6 +702,36 @@ impl CurrentPathTracker {
     }
 }
 
+fn chrono_to_jiff_timestamp(dt: DateTime<Utc>) -> Timestamp {
+    let seconds = dt.timestamp();
+    let nanoseconds = match i32::try_from(dt.timestamp_subsec_nanos()) {
+        Ok(nanoseconds) => nanoseconds,
+        Err(_) => {
+            return if seconds < 0 { Timestamp::MIN } else { Timestamp::MAX };
+        }
+    };
+
+    match Timestamp::new(seconds, nanoseconds) {
+        Ok(timestamp) => timestamp,
+        Err(_) => {
+            if seconds < 0 {
+                Timestamp::MIN
+            } else {
+                Timestamp::MAX
+            }
+        }
+    }
+}
+
+fn timestamp_elapsed_seconds_since(now: Timestamp, earlier: Timestamp) -> u64 {
+    let duration = now.duration_since(earlier);
+    if duration.is_negative() {
+        return 0;
+    }
+
+    u64::try_from(duration.as_secs()).map_or(u64::MAX, |seconds| seconds)
+}
+
 #[derive(Clone, Copy, Debug, Default)]
 struct ScannerDiskBucketScanState {
     concurrency_limit: u64,
@@ -884,11 +915,13 @@ const SCAN_CYCLE_RESULT_SUCCESS: u8 = 1;
 const SCAN_CYCLE_RESULT_ERROR: u8 = 2;
 const SCAN_CYCLE_RESULT_PARTIAL: u8 = 3;
 const SCAN_CYCLE_RESULT_SUPERSEDED: u8 = 4;
+const SCAN_CYCLE_RESULT_DEFERRED: u8 = 5;
 const SCAN_CYCLE_RESULT_UNKNOWN_LABEL: &str = "unknown";
 const SCAN_CYCLE_RESULT_SUCCESS_LABEL: &str = "success";
 const SCAN_CYCLE_RESULT_ERROR_LABEL: &str = "error";
 const SCAN_CYCLE_RESULT_PARTIAL_LABEL: &str = "partial";
 const SCAN_CYCLE_RESULT_SUPERSEDED_LABEL: &str = "superseded";
+const SCAN_CYCLE_RESULT_DEFERRED_LABEL: &str = "deferred";
 
 #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
 pub enum ScanCyclePartialReason {
@@ -1166,12 +1199,12 @@ pub struct ScannerLastMinute {
 
 #[derive(Clone, Debug, Default, Serialize, Deserialize)]
 pub struct ScannerMetricsReport {
-    pub collected_at: DateTime<Utc>,
+    pub collected_at: Timestamp,
     pub current_cycle: u64,
     #[serde(default)]
     pub current_cycle_active: bool,
-    pub current_started: DateTime<Utc>,
-    pub cycles_completed_at: Vec<DateTime<Utc>>,
+    pub current_started: Timestamp,
+    pub cycles_completed_at: Vec<Timestamp>,
     pub ongoing_buckets: usize,
     #[serde(default)]
     pub active_scan_paths: usize,
@@ -1393,6 +1426,7 @@ fn scan_cycle_result_label(result: u8) -> &'static str {
         SCAN_CYCLE_RESULT_ERROR => SCAN_CYCLE_RESULT_ERROR_LABEL,
         SCAN_CYCLE_RESULT_PARTIAL => SCAN_CYCLE_RESULT_PARTIAL_LABEL,
         SCAN_CYCLE_RESULT_SUPERSEDED => SCAN_CYCLE_RESULT_SUPERSEDED_LABEL,
+        SCAN_CYCLE_RESULT_DEFERRED => SCAN_CYCLE_RESULT_DEFERRED_LABEL,
         _ => SCAN_CYCLE_RESULT_UNKNOWN_LABEL,
     }
 }
@@ -1721,6 +1755,11 @@ pub fn emit_scan_cycle_superseded(duration: Duration) {
     metrics::counter!(OTEL_SCANNER_CYCLES, "result" => SCAN_CYCLE_RESULT_SUPERSEDED_LABEL).increment(1);
 }
 
+pub fn emit_scan_cycle_deferred(duration: Duration) {
+    global_metrics().record_scan_cycle_deferred(duration);
+    metrics::counter!(OTEL_SCANNER_CYCLES, "result" => SCAN_CYCLE_RESULT_DEFERRED_LABEL).increment(1);
+}
+
 pub fn emit_scan_bucket_drive_complete(success: bool, bucket: &str, disk: &str, duration: Duration) {
     let result = if success { "success" } else { "error" };
     global_metrics().record_scanner_bucket_drive_result(bucket, disk, result);
@@ -2518,6 +2557,17 @@ impl Metrics {
             .store(duration_millis_saturated(duration), Ordering::Relaxed);
     }
 
+    pub fn record_scan_cycle_deferred(&self, duration: Duration) {
+        self.record_scanner_cycle_end_time();
+        self.last_scan_cycle_result
+            .store(SCAN_CYCLE_RESULT_DEFERRED, Ordering::Relaxed);
+        self.last_scan_cycle_partial_reason
+            .store(ScanCyclePartialReason::Unknown as u8, Ordering::Relaxed);
+        self.last_scan_cycle_partial_source.store(0, Ordering::Relaxed);
+        self.last_scan_cycle_duration_millis
+            .store(duration_millis_saturated(duration), Ordering::Relaxed);
+    }
+
     pub fn record_scan_cycle_partial(&self, duration: Duration, reason: ScanCyclePartialReason) {
         self.record_scan_cycle_partial_with_source(duration, reason, None);
     }
@@ -2988,8 +3038,8 @@ impl Metrics {
             let cycle = self.cycle_info.read().await;
             let has_cycle = if let Some(cycle) = cycle.as_ref() {
                 m.current_cycle = cycle.current;
-                m.cycles_completed_at = cycle.cycle_completed.clone();
-                m.current_started = cycle.started;
+                m.cycles_completed_at = cycle.cycle_completed.iter().copied().map(chrono_to_jiff_timestamp).collect();
+                m.current_started = chrono_to_jiff_timestamp(cycle.started);
                 true
             } else {
                 false
@@ -3024,15 +3074,15 @@ impl Metrics {
         };
 
         if !has_cycle && let Some(init_time) = crate::get_global_init_time().await {
-            m.current_started = init_time;
+            m.current_started = chrono_to_jiff_timestamp(init_time);
         }
 
-        m.collected_at = Utc::now();
+        m.collected_at = Timestamp::now();
         let current_path_snapshots = self.current_path_snapshots().await;
         m.active_scan_paths = current_path_snapshots.len();
         m.oldest_active_path_age_seconds = current_path_snapshots
             .iter()
-            .map(|(_, state)| m.collected_at.signed_duration_since(state.updated_at).num_seconds().max(0) as u64)
+            .map(|(_, state)| timestamp_elapsed_seconds_since(m.collected_at, state.updated_at))
             .max()
             .unwrap_or_default();
         m.active_paths = current_path_snapshots
@@ -3308,6 +3358,22 @@ impl Drop for CloseDiskGuard {
 mod tests {
     use super::*;
 
+    #[test]
+    fn scanner_metrics_report_timestamps_serialize_as_rfc3339_utc() {
+        let report = ScannerMetricsReport {
+            collected_at: Timestamp::constant(1_700_000_000, 123_456_000),
+            current_started: Timestamp::constant(1_699_999_940, 0),
+            cycles_completed_at: vec![Timestamp::constant(1_700_000_060, 987_654_000)],
+            ..Default::default()
+        };
+
+        let value = serde_json::to_value(&report).expect("scanner metrics report should serialize");
+
+        assert_eq!(value["collected_at"].as_str(), Some("2023-11-14T22:13:20.123456Z"));
+        assert_eq!(value["current_started"].as_str(), Some("2023-11-14T22:12:20Z"));
+        assert_eq!(value["cycles_completed_at"][0].as_str(), Some("2023-11-14T22:14:20.987654Z"));
+    }
+
     #[tokio::test]
     async fn close_disk_guard_runs_cleanup_when_an_early_return_drops_it() {
         let (closed_tx, closed_rx) = tokio::sync::oneshot::channel();
@@ -3366,7 +3432,7 @@ mod tests {
     #[tokio::test]
     async fn report_counts_active_scan_paths() {
         let metrics = Metrics::new();
-        let updated_at = Utc::now() - chrono::Duration::seconds(12);
+        let updated_at = Timestamp::now() - jiff::SignedDuration::from_secs(12);
         metrics.current_paths.write().await.insert(
             "disk-a".to_string(),
             Arc::new(CurrentPathTracker::new_at("bucket-a".to_string(), updated_at)),
@@ -3388,7 +3454,7 @@ mod tests {
         let metrics = Metrics::new();
         let tracker = Arc::new(CurrentPathTracker::new_at(
             "bucket-a".to_string(),
-            Utc::now() - chrono::Duration::hours(1),
+            Timestamp::now() - jiff::SignedDuration::from_secs(60 * 60),
         ));
         metrics
             .current_paths
@@ -4161,7 +4227,7 @@ mod tests {
         let report = metrics.report().await;
         *crate::globals::GLOBAL_INIT_TIME.write().await = previous_init_time;
 
-        assert_eq!(report.current_started, cycle_started);
+        assert_eq!(report.current_started, chrono_to_jiff_timestamp(cycle_started));
     }
 
     #[tokio::test]
@@ -4217,6 +4283,21 @@ mod tests {
         assert_eq!(report.partial_cycles, 0);
     }
 
+    #[tokio::test]
+    async fn report_tracks_deferred_cycle_without_failed_increment() {
+        let metrics = Metrics::new();
+        metrics.record_scan_cycle_deferred(Duration::from_millis(250));
+
+        let report = metrics.report().await;
+
+        assert_eq!(report.last_cycle_result, SCAN_CYCLE_RESULT_DEFERRED_LABEL);
+        assert_eq!(report.last_cycle_result_code, u64::from(SCAN_CYCLE_RESULT_DEFERRED));
+        assert_eq!(report.last_cycle_duration_seconds, 0.25);
+        assert_eq!(report.failed_cycles, 0);
+        assert_eq!(report.superseded_cycles, 0);
+        assert_eq!(report.partial_cycles, 0);
+    }
+
     #[tokio::test]
     async fn report_tracks_successful_scan_cycle_without_failed_increment() {
         let metrics = Metrics::new();
@@ -4584,7 +4665,7 @@ mod tests {
         let active = metrics.report().await;
         assert!(active.current_cycle_active);
         assert_eq!(active.current_cycle, 12);
-        assert_eq!(active.current_started, cycle_started);
+        assert_eq!(active.current_started, chrono_to_jiff_timestamp(cycle_started));
 
         let idle_cycle = CurrentCycle {
             current: 0,

crates/common/src/table_catalog.rs

@@ -0,0 +1,17 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// Cross-crate lock identity used to fence table-bucket publication against
+/// object mutations that bypass the S3 request authorization layer.
+pub const TABLE_BUCKET_PUBLICATION_LOCK_PATH: &str = ".rustfs-table/warehouses/default/publication.lock";

crates/config/README.md

@@ -97,6 +97,14 @@ Current guidance:
   - enables minimal payload mode for GET health responses (`status`, `ready` only).
 - `RUSTFS_HEALTH_READINESS_CACHE_TTL_MS`
   - TTL for readiness cache evaluation.
+- `RUSTFS_HEALTH_OBJECT_PROGRESS_ENABLE`
+  - withdraws readiness when bounded object read/write stages stop completing while requests remain active.
+  - default is `true`.
+- `RUSTFS_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS`
+  - maximum time without completion in a bounded object stage before readiness is withdrawn.
+  - default is `30000`; `0` uses the default.
+  - the effective value is at least 5 seconds longer than `RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT`.
+  - this readiness SLO is independent of disk read/write failure deadlines and may withdraw traffic before those deadlines expire.
 - `RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE`
   - enables busy protection behavior for health probes.
   - default is `false`.

crates/config/src/constants/health.rs

@@ -22,6 +22,19 @@ pub const DEFAULT_HEALTH_ENDPOINT_ENABLE: bool = true;
 pub const ENV_HEALTH_READINESS_CACHE_TTL_MS: &str = "RUSTFS_HEALTH_READINESS_CACHE_TTL_MS";
 pub const DEFAULT_HEALTH_READINESS_CACHE_TTL_MS: u64 = 1000;
 
+/// Enable readiness withdrawal when bounded object read/write stages stop
+/// completing while requests remain active.
+pub const ENV_HEALTH_OBJECT_PROGRESS_ENABLE: &str = "RUSTFS_HEALTH_OBJECT_PROGRESS_ENABLE";
+pub const DEFAULT_HEALTH_OBJECT_PROGRESS_ENABLE: bool = true;
+
+/// Requested time without completion in a bounded object stage before local
+/// readiness is withdrawn (milliseconds). A value of `0` uses the default;
+/// runtime adds a safety floor based on the object-lock acquisition timeout.
+pub const ENV_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS: &str = "RUSTFS_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS";
+pub const DEFAULT_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS: u64 = 30_000;
+/// Additional time beyond the configured object-lock acquisition deadline.
+pub const HEALTH_OBJECT_PROGRESS_LOCK_MARGIN_MS: u64 = 5_000;
+
 /// Timeout for cluster health readiness collectors (milliseconds).
 /// This bounds expensive storage and lock quorum checks used by cluster probes.
 pub const ENV_HEALTH_CLUSTER_TIMEOUT_MS: &str = "RUSTFS_HEALTH_CLUSTER_TIMEOUT_MS";

crates/config/src/constants/internode.rs

@@ -177,10 +177,9 @@ const _: () = assert!(!DEFAULT_INTERNODE_RPC_REPLAY_SCOPE_STRICT);
 ///
 /// The cache retains each nonce for the ~10-minute signature freshness envelope. Once peers use
 /// replay-scoped v3 authentication, every authenticated RPC consumes one entry, so the steady
-/// state holds roughly `authenticated RPC RPS x 601s` entries. The default sustains about 1,700
-/// authenticated RPCs per second (about 120 MiB worst case, allocated only under sustained load);
-/// operators must size it for the node's aggregate peak RPC rate before enabling strict replay
-/// scope. Overflow fails closed — legitimate signed traffic is the only thing that can fill the
+/// state holds roughly `authenticated RPC RPS x 601s` entries. This default is the minimum floor:
+/// explicit operator values and resource-aware auto sizing both clamp upward to at least this
+/// value. Overflow fails closed — legitimate signed traffic is the only thing that can fill the
 /// cache (replays are rejected before insertion, and an attacker cannot mint valid nonces without
 /// the shared secret) — and increments
 /// `rustfs_system_network_internode_replay_cache_overflow_total`, so a sustained non-zero overflow

crates/config/src/constants/runtime.rs

@@ -81,6 +81,8 @@ pub const ENV_TEST_IAM_FAIL_INIT_ATTEMPTS: &str = "RUSTFS_TEST_IAM_FAIL_INIT_ATT
 pub const ENV_TEST_IAM_RETRY_INTERVAL_MS: &str = "RUSTFS_TEST_IAM_RETRY_INTERVAL_MS";
 /// Runtime env var controlling the transition worker count.
 pub const ENV_TRANSITION_WORKERS: &str = "RUSTFS_MAX_TRANSITION_WORKERS";
+/// Runtime env var controlling the expiry worker count.
+pub const ENV_MAX_EXPIRY_WORKERS: &str = "RUSTFS_MAX_EXPIRY_WORKERS";
 /// Runtime env var controlling the absolute maximum transition workers.
 pub const ENV_TRANSITION_WORKERS_ABSOLUTE_MAX: &str = "RUSTFS_ABSOLUTE_MAX_WORKERS";
 /// Runtime env var controlling the transition queue capacity.

crates/config/src/constants/tls.rs

@@ -36,6 +36,11 @@ pub const ENV_TRUST_SYSTEM_CA: &str = "RUSTFS_TRUST_SYSTEM_CA";
 /// To change this behavior, set the environment variable RUSTFS_TRUST_SYSTEM_CA=1
 pub const DEFAULT_TRUST_SYSTEM_CA: bool = false;
 
+/// Environment variable for an extra outbound root CA certificate bundle.
+/// Use this to trust an internal CA for outbound HTTPS clients without replacing
+/// the default operating-system/web PKI roots via SSL_CERT_FILE.
+pub const ENV_RUSTFS_EXTRA_CA_CERT: &str = "RUSTFS_EXTRA_CA_CERT";
+
 /// Environment variable to trust leaf certificates as CA
 /// When set to "1", RustFS will treat leaf certificates as CA certificates for trust validation.
 /// By default, this is disabled.

crates/data-usage/Cargo.toml

@@ -37,7 +37,6 @@ hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-filemeta/hotpath-cpu"]
 hotpath.workspace = true
 serde = { workspace = true, features = ["derive"] }
 rmp-serde = { workspace = true }
-async-trait = { workspace = true }
 rustfs-filemeta = { workspace = true }
 
 [lib]

crates/data-usage/src/data_usage.rs

@@ -37,6 +37,10 @@ pub const USAGE_LAST_UPDATE_FUTURE_TOLERANCE: Duration = Duration::from_secs(5 *
 /// Keeping the existing object name preserves rolling-upgrade and rollback
 /// compatibility without allowing an ambiguous snapshot to become authoritative.
 pub const DATA_USAGE_OBJECT_NAME: &str = ".usage.v2.json";
+/// Latest structurally complete scanner observation. Unlike
+/// [`DATA_USAGE_OBJECT_NAME`], this object is never authoritative for quota
+/// admission because namespace activity may have raced the scan.
+pub const DATA_USAGE_OBSERVED_OBJECT_NAME: &str = ".usage.observed.json";
 
 /// Usage snapshot written by scanner implementations predating distributed
 /// leadership fencing. It is read only when neither authoritative snapshot
@@ -218,6 +222,20 @@ pub struct DataUsageInfo {
     /// explicit entry for every bucket, including confirmed-empty buckets.
     #[serde(default)]
     pub usage_snapshot_complete: bool,
+    /// Whether no namespace activity or dirty-usage generation changed while
+    /// the coordinated snapshot was being produced.
+    ///
+    /// `false` still describes a structurally complete, useful point-in-time
+    /// usage view, but follow-up scanner work remains pending. `None` is kept
+    /// for snapshots written before this status became observable.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub usage_snapshot_converged: Option<bool>,
+    /// Identity of the authoritative snapshot from which a nonconverged
+    /// observation started. Admin readers require an exact match before using
+    /// the observation, so bucket namespace mutations fence old observations
+    /// without relying on synchronized clocks.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub usage_snapshot_authoritative_baseline: Option<DataUsageSnapshotIdentity>,
     /// Deprecated kept here for backward compatibility reasons
     pub bucket_sizes: HashMap<String, u64>,
     /// Per-disk snapshot information when available
@@ -225,6 +243,59 @@ pub struct DataUsageInfo {
     pub disk_usage_status: Vec<DiskUsageStatus>,
 }
 
+/// Stable identity fields changed by both coordinated scanner publication and
+/// backward-compatible bucket namespace cleanup.
+#[derive(Debug, Default, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
+pub struct DataUsageSnapshotIdentity {
+    pub last_update: Option<SystemTime>,
+    pub scanner_cycle: Option<u64>,
+    pub scanner_epoch: Option<u64>,
+}
+
+impl DataUsageInfo {
+    pub fn snapshot_identity(&self) -> DataUsageSnapshotIdentity {
+        DataUsageSnapshotIdentity {
+            last_update: self.last_update,
+            scanner_cycle: self.scanner_cycle,
+            scanner_epoch: self.scanner_epoch,
+        }
+    }
+}
+
+/// Return whether `candidate` was produced after `baseline`.
+///
+/// New coordinated snapshots are ordered by leadership epoch and scanner
+/// cycle. The timestamp fallback preserves ordering for legacy snapshots that
+/// predate those fields.
+pub fn data_usage_snapshot_is_newer(candidate: &DataUsageInfo, baseline: &DataUsageInfo) -> bool {
+    match (
+        candidate.scanner_epoch.zip(candidate.scanner_cycle),
+        baseline.scanner_epoch.zip(baseline.scanner_cycle),
+    ) {
+        (Some(candidate), Some(baseline)) => candidate > baseline,
+        (Some(_), None) => true,
+        (None, Some(_)) => false,
+        (None, None) => match (candidate.last_update, baseline.last_update) {
+            (Some(candidate), Some(baseline)) => candidate > baseline,
+            (Some(_), None) => true,
+            (None, Some(_) | None) => false,
+        },
+    }
+}
+
+/// Return whether a nonconverged observation may safely supersede the admin
+/// view of `authoritative`.
+///
+/// The exact baseline identity is independent of clock ordering. Older binaries
+/// already advance the authoritative timestamp when deleting a bucket, so a
+/// rollback delete/recreate fences the previous bucket incarnation too.
+pub fn observed_data_usage_is_newer(observed: &DataUsageInfo, authoritative: &DataUsageInfo) -> bool {
+    observed.usage_snapshot_converged == Some(false)
+        && observed.is_complete_bucket_usage_snapshot()
+        && observed.usage_snapshot_authoritative_baseline.as_ref() == Some(&authoritative.snapshot_identity())
+        && data_usage_snapshot_is_newer(observed, authoritative)
+}
+
 /// Metadata describing the status of a disk-level data usage snapshot.
 #[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
 pub struct DiskUsageStatus {
@@ -775,8 +846,15 @@ impl DataUsageEntry {
     }
 }
 
-/// Data usage cache info
-#[derive(Clone, Debug, Default, Serialize, Deserialize)]
+/// Read-only projection of the scanner's `.usage-cache.bin` info block.
+///
+/// The canonical wire format is written by the hand-written map-encoded
+/// `Serialize` on the scanner-side `DataUsageCacheInfo`
+/// (`crates/scanner/src/data_usage_define.rs`), which carries 16 fields.
+/// This type decodes only the shared subset and is deliberately not
+/// `Serialize`: a derived (array) encoding of this 6-field subset would
+/// corrupt the cache for scanner readers, so no write path may exist here.
+#[derive(Clone, Debug, Default, Deserialize)]
 pub struct DataUsageCacheInfo {
     pub name: String,
     pub next_cycle: u64,
@@ -792,8 +870,12 @@ pub struct DataUsageCacheInfo {
     pub snapshot_complete: bool,
 }
 
-/// Data usage cache
-#[derive(Clone, Debug, Default, Serialize, Deserialize)]
+/// Read-only projection of a scanner-written `.usage-cache.bin` file.
+///
+/// The scanner-side `DataUsageCache` (`crates/scanner/src/data_usage_define.rs`)
+/// owns the persisted format; this type only decodes it (see
+/// [`DataUsageCacheInfo`]) and must never grow a serialization path.
+#[derive(Clone, Debug, Default, Deserialize)]
 pub struct DataUsageCache {
     pub info: DataUsageCacheInfo,
     pub cache: HashMap<String, DataUsageEntry>,
@@ -1115,31 +1197,10 @@ impl DataUsageCache {
         }
     }
 
-    pub fn marshal_msg(&self) -> Result<Vec<u8>, Box<dyn std::error::Error + Send + Sync>> {
-        let mut buf = Vec::new();
-        self.serialize(&mut rmp_serde::Serializer::new(&mut buf))?;
-        Ok(buf)
-    }
-
     pub fn unmarshal(buf: &[u8]) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
         let t: Self = rmp_serde::from_slice(buf)?;
         Ok(t)
     }
-
-    // Note: load and save methods are storage-specific and should be implemented
-    // in the ecstore crate where storage access is available
-}
-
-/// Trait for storage-specific operations on DataUsageCache
-#[async_trait::async_trait]
-pub trait DataUsageCacheStorage {
-    /// Load data usage cache from backend storage
-    async fn load(store: &dyn std::any::Any, name: &str) -> Result<Self, Box<dyn std::error::Error + Send + Sync>>
-    where
-        Self: Sized;
-
-    /// Save data usage cache to backend storage
-    async fn save(&self, name: &str) -> Result<(), Box<dyn std::error::Error + Send + Sync>>;
 }
 
 // Helper structs and functions for cache operations
@@ -1761,6 +1822,82 @@ mod tests {
         assert!(decoded.all_tier_stats.is_none());
     }
 
+    /// Scanner-written `.usage-cache.bin` bytes: a 2-element array of the
+    /// canonical 16-field map-encoded info block and one map-encoded entry.
+    /// Captured from the canonical writer's `marshal_msg` — see
+    /// `usage_cache_wire_format_is_pinned` in
+    /// `crates/scanner/src/data_usage_define.rs`, which pins these exact
+    /// bytes and documents regeneration. Hardcoded here because a
+    /// dev-dependency on rustfs-scanner would pull the whole ecstore tree
+    /// into this crate's test build, and a fixture generated at test runtime
+    /// could not detect writer drift anyway.
+    const SCANNER_USAGE_CACHE_WIRE_FIXTURE: &[u8] = &[
+        0x92, 0xde, 0x00, 0x10, 0xa4, 0x6e, 0x61, 0x6d, 0x65, 0xab, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65,
+        0x74, 0xaa, 0x6e, 0x65, 0x78, 0x74, 0x5f, 0x63, 0x79, 0x63, 0x6c, 0x65, 0x07, 0xac, 0x6c, 0x65, 0x61, 0x64, 0x65, 0x72,
+        0x5f, 0x65, 0x70, 0x6f, 0x63, 0x68, 0x09, 0xab, 0x6c, 0x61, 0x73, 0x74, 0x5f, 0x75, 0x70, 0x64, 0x61, 0x74, 0x65, 0x92,
+        0xce, 0x65, 0x53, 0xf1, 0x00, 0x00, 0xac, 0x73, 0x6b, 0x69, 0x70, 0x5f, 0x68, 0x65, 0x61, 0x6c, 0x69, 0x6e, 0x67, 0xc3,
+        0xa9, 0x6c, 0x69, 0x66, 0x65, 0x63, 0x79, 0x63, 0x6c, 0x65, 0xc0, 0xab, 0x72, 0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74,
+        0x69, 0x6f, 0x6e, 0xc0, 0xae, 0x66, 0x61, 0x69, 0x6c, 0x65, 0x64, 0x5f, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x81,
+        0xb0, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2f, 0x6c, 0x6f, 0x73, 0x74, 0x0b, 0xb1, 0x73,
+        0x63, 0x61, 0x6e, 0x5f, 0x72, 0x65, 0x73, 0x75, 0x6d, 0x65, 0x5f, 0x61, 0x66, 0x74, 0x65, 0x72, 0xb2, 0x77, 0x69, 0x72,
+        0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2f, 0x72, 0x65, 0x73, 0x75, 0x6d, 0x65, 0xaf, 0x73, 0x63, 0x61, 0x6e,
+        0x5f, 0x63, 0x68, 0x65, 0x63, 0x6b, 0x70, 0x6f, 0x69, 0x6e, 0x74, 0xc0, 0xad, 0x70, 0x65, 0x6e, 0x64, 0x69, 0x6e, 0x67,
+        0x5f, 0x68, 0x65, 0x61, 0x6c, 0x73, 0x91, 0x9a, 0xa6, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0xab, 0x77, 0x69, 0x72, 0x65,
+        0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0xa6, 0x62, 0x72, 0x6f, 0x6b, 0x65, 0x6e, 0xc0, 0x01, 0x64, 0xcc, 0xc8, 0x03,
+        0xa8, 0x64, 0x65, 0x66, 0x65, 0x72, 0x72, 0x65, 0x64, 0xa6, 0x62, 0x75, 0x64, 0x67, 0x65, 0x74, 0xab, 0x6f, 0x62, 0x6a,
+        0x65, 0x63, 0x74, 0x5f, 0x6c, 0x6f, 0x63, 0x6b, 0xc0, 0xa6, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x92, 0x01, 0x02, 0xb1,
+        0x73, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x5f, 0x63, 0x6f, 0x6d, 0x70, 0x6c, 0x65, 0x74, 0x65, 0xc3, 0xb0, 0x73,
+        0x63, 0x61, 0x6e, 0x5f, 0x70, 0x6c, 0x61, 0x6e, 0x5f, 0x64, 0x69, 0x67, 0x65, 0x73, 0x74, 0xdc, 0x00, 0x20, 0x03, 0x03,
+        0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03,
+        0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0xb0, 0x63, 0x61, 0x63, 0x68, 0x65, 0x5f, 0x6b, 0x65, 0x79,
+        0x5f, 0x66, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x01, 0x81, 0xab, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65,
+        0x74, 0x8b, 0xa8, 0x63, 0x68, 0x69, 0x6c, 0x64, 0x72, 0x65, 0x6e, 0x90, 0xa4, 0x73, 0x69, 0x7a, 0x65, 0xcd, 0x10, 0x00,
+        0xa7, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x03, 0xa8, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x73, 0x05, 0xae,
+        0x64, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x5f, 0x6d, 0x61, 0x72, 0x6b, 0x65, 0x72, 0x73, 0x01, 0xa9, 0x6f, 0x62, 0x6a, 0x5f,
+        0x73, 0x69, 0x7a, 0x65, 0x73, 0x9b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xac, 0x6f, 0x62,
+        0x6a, 0x5f, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x73, 0x97, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xb1, 0x72,
+        0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x73, 0xc0, 0xa9, 0x63, 0x6f,
+        0x6d, 0x70, 0x61, 0x63, 0x74, 0x65, 0x64, 0xc3, 0xae, 0x66, 0x61, 0x69, 0x6c, 0x65, 0x64, 0x5f, 0x6f, 0x62, 0x6a, 0x65,
+        0x63, 0x74, 0x73, 0x02, 0xae, 0x61, 0x6c, 0x6c, 0x5f, 0x74, 0x69, 0x65, 0x72, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x73, 0x91,
+        0x81, 0xa4, 0x57, 0x41, 0x52, 0x4d, 0x93, 0xcd, 0x08, 0x00, 0x02, 0x01,
+    ];
+
+    #[test]
+    fn thin_usage_cache_decodes_scanner_wire_fixture() {
+        let decoded =
+            DataUsageCache::unmarshal(SCANNER_USAGE_CACHE_WIRE_FIXTURE).expect("thin projection decodes a scanner-written cache");
+
+        // The six fields shared with the scanner's 16-field info block; the
+        // remaining ten (lifecycle, replication, checkpoint, heals, ...) must
+        // be skipped, not error.
+        assert_eq!(decoded.info.name, "wire-bucket");
+        assert_eq!(decoded.info.next_cycle, 7);
+        assert_eq!(
+            decoded.info.last_update,
+            Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1_700_000_000))
+        );
+        assert!(decoded.info.skip_healing);
+        assert_eq!(decoded.info.failed_objects.get("wire-bucket/lost"), Some(&11));
+        assert!(decoded.info.snapshot_complete);
+
+        // Entries use the shared canonical map-encoded type end to end.
+        let entry = decoded.cache.get("wire-bucket").expect("fixture entry decodes");
+        assert_eq!(entry.size, 4096);
+        assert_eq!(entry.objects, 3);
+        assert_eq!(entry.versions, 5);
+        assert_eq!(entry.delete_markers, 1);
+        assert!(entry.compacted);
+        assert_eq!(entry.failed_objects, 2);
+        assert_eq!(
+            entry.all_tier_stats.as_ref().and_then(|tiers| tiers.tiers.get("WARM")),
+            Some(&TierStats {
+                total_size: 2048,
+                num_versions: 2,
+                num_objects: 1,
+            })
+        );
+    }
+
     #[test]
     fn hash_path_uses_portable_slash_semantics() {
         for (input, expected) in [
@@ -1783,6 +1920,8 @@ mod tests {
         let current = DataUsageInfo {
             last_update: Some(SystemTime::UNIX_EPOCH),
             usage_snapshot_complete: true,
+            usage_snapshot_converged: Some(false),
+            usage_snapshot_authoritative_baseline: Some(DataUsageSnapshotIdentity::default()),
             ..Default::default()
         };
         let encoded = rmp_serde::to_vec_named(&current).expect("encode current data usage snapshot");
@@ -1790,6 +1929,76 @@ mod tests {
 
         assert_eq!(legacy.buckets_count, 0);
         assert!(current.is_complete_bucket_usage_snapshot());
+        assert_eq!(current.usage_snapshot_converged, Some(false));
+    }
+
+    #[test]
+    fn convergence_marker_defaults_to_unknown_for_older_snapshots() {
+        let encoded = rmp_serde::to_vec_named(&DataUsageInfo {
+            last_update: Some(SystemTime::UNIX_EPOCH),
+            usage_snapshot_complete: true,
+            ..Default::default()
+        })
+        .expect("encode pre-convergence data usage snapshot");
+        let decoded: DataUsageInfo = rmp_serde::from_slice(&encoded).expect("decode older data usage snapshot");
+
+        assert!(decoded.is_complete_bucket_usage_snapshot());
+        assert_eq!(decoded.usage_snapshot_converged, None);
+    }
+
+    #[test]
+    fn observation_selection_is_clock_independent_and_baseline_fenced() {
+        let mut authoritative = DataUsageInfo {
+            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(600)),
+            scanner_epoch: Some(7),
+            scanner_cycle: Some(10),
+            usage_snapshot_complete: true,
+            ..Default::default()
+        };
+        let observed = DataUsageInfo {
+            // A newer leader may have a slower wall clock.
+            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(300)),
+            scanner_epoch: Some(8),
+            scanner_cycle: Some(1),
+            usage_snapshot_complete: true,
+            usage_snapshot_converged: Some(false),
+            usage_snapshot_authoritative_baseline: Some(authoritative.snapshot_identity()),
+            ..Default::default()
+        };
+
+        assert!(observed_data_usage_is_newer(&observed, &authoritative));
+
+        authoritative.last_update = Some(SystemTime::UNIX_EPOCH + Duration::from_secs(601));
+        assert!(
+            !observed_data_usage_is_newer(&observed, &authoritative),
+            "an old-binary namespace mutation must fence the prior bucket incarnation regardless of clock skew"
+        );
+    }
+
+    #[test]
+    fn observation_selection_requires_nonconverged_complete_newer_data() {
+        let authoritative = DataUsageInfo {
+            last_update: Some(SystemTime::UNIX_EPOCH),
+            scanner_epoch: Some(2),
+            scanner_cycle: Some(10),
+            usage_snapshot_complete: true,
+            ..Default::default()
+        };
+        let baseline = Some(authoritative.snapshot_identity());
+        let candidate = |epoch, cycle, converged, complete| DataUsageInfo {
+            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
+            scanner_epoch: Some(epoch),
+            scanner_cycle: Some(cycle),
+            usage_snapshot_complete: complete,
+            usage_snapshot_converged: converged,
+            usage_snapshot_authoritative_baseline: baseline,
+            ..Default::default()
+        };
+
+        assert!(observed_data_usage_is_newer(&candidate(2, 11, Some(false), true), &authoritative));
+        assert!(!observed_data_usage_is_newer(&candidate(2, 9, Some(false), true), &authoritative));
+        assert!(!observed_data_usage_is_newer(&candidate(2, 11, Some(true), true), &authoritative));
+        assert!(!observed_data_usage_is_newer(&candidate(2, 11, Some(false), false), &authoritative));
     }
 
     #[test]

crates/e2e_test/src/admin_iam_crud_test.rs

@@ -26,7 +26,9 @@
 //! Later batches tracked on backlog#1154: config get/set, info, pools status,
 //! group lifecycle, import/export IAM.
 
-use crate::common::{RustFSTestEnvironment, admin_ok, admin_request, init_logging};
+use crate::common::{
+    RustFSTestEnvironment, admin_ok, admin_request, admin_request_with_session_token, build_test_sts_client, init_logging,
+};
 use aws_sdk_s3::config::{Credentials, Region};
 use aws_sdk_s3::primitives::ByteStream;
 use aws_sdk_s3::{Client, Config};
@@ -87,6 +89,262 @@ fn bucket_rw_policy(bucket: &str) -> String {
     .to_string()
 }
 
+async fn create_user_with_service_account_update_policy(
+    env: &RustFSTestEnvironment,
+    user: &str,
+    secret: &str,
+    policy: &str,
+) -> TestResult {
+    admin_ok(
+        env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/add-user?accessKey={user}"),
+        Some(serde_json::json!({ "secretKey": secret, "status": "enabled" }).to_string()),
+    )
+    .await?;
+    admin_ok(
+        env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/add-canned-policy?name={policy}"),
+        Some(
+            serde_json::json!({
+                "Version": "2012-10-17",
+                "Statement": [
+                    {
+                        "Effect": "Allow",
+                        "Action": ["admin:UpdateServiceAccount"]
+                    },
+                    {
+                        "Effect": "Allow",
+                        "Action": ["sts:AssumeRole"],
+                        "Resource": ["arn:aws:s3:::*"]
+                    }
+                ]
+            })
+            .to_string(),
+        ),
+    )
+    .await?;
+    admin_ok(
+        env,
+        http::Method::POST,
+        "/rustfs/admin/v3/idp/builtin/policy/attach",
+        Some(serde_json::json!({ "policies": [policy], "user": user }).to_string()),
+    )
+    .await?;
+    Ok(())
+}
+
+async fn create_service_account_for(
+    env: &RustFSTestEnvironment,
+    parent: &str,
+) -> Result<(String, String), Box<dyn Error + Send + Sync>> {
+    let response = admin_ok(
+        env,
+        http::Method::PUT,
+        "/rustfs/admin/v3/add-service-accounts",
+        Some(serde_json::json!({ "targetUser": parent }).to_string()),
+    )
+    .await?;
+    let response: serde_json::Value = serde_json::from_str(&response)?;
+    let access_key = response["credentials"]["accessKey"]
+        .as_str()
+        .ok_or("service account response should contain credentials.accessKey")?
+        .to_owned();
+    let secret_key = response["credentials"]["secretKey"]
+        .as_str()
+        .ok_or("service account response should contain credentials.secretKey")?
+        .to_owned();
+    Ok((access_key, secret_key))
+}
+
+async fn assert_admin_status(
+    env: &RustFSTestEnvironment,
+    credentials: (&str, &str, Option<&str>),
+    path: &str,
+    body: String,
+    expected: StatusCode,
+    context: &str,
+) -> TestResult {
+    let (access_key, secret_key, session_token) = credentials;
+    let (status, response) =
+        admin_request_with_session_token(&env.url, http::Method::POST, path, Some(body), access_key, secret_key, session_token)
+            .await?;
+    assert_eq!(status, expected, "{context}: got {status}: {response}");
+    if expected == StatusCode::FORBIDDEN {
+        assert!(response.contains("AccessDenied"), "{context}: expected AccessDenied body, got {response}");
+    }
+    Ok(())
+}
+
+#[tokio::test]
+#[serial]
+async fn test_update_service_account_enforces_owner_and_parent_scope() -> TestResult {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+
+    let parent = "updateparent";
+    let parent_secret = "updateparentsecret";
+    let outsider = "updateoutsider";
+    let outsider_secret = "updateoutsidersecret";
+    let ordinary = "updateordinary";
+    let ordinary_secret = "updateordinarysecret";
+    create_user_with_service_account_update_policy(&env, parent, parent_secret, "update-parent-policy").await?;
+    create_user_with_service_account_update_policy(&env, outsider, outsider_secret, "update-outsider-policy").await?;
+    admin_ok(
+        &env,
+        http::Method::POST,
+        "/rustfs/admin/v3/idp/builtin/policy/attach",
+        Some(serde_json::json!({ "policies": ["consoleAdmin"], "user": outsider }).to_string()),
+    )
+    .await?;
+    admin_ok(
+        &env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/add-user?accessKey={ordinary}"),
+        Some(serde_json::json!({ "secretKey": ordinary_secret, "status": "enabled" }).to_string()),
+    )
+    .await?;
+
+    let (target_access_key, _) = create_service_account_for(&env, parent).await?;
+    let target_path = format!("/rustfs/admin/v3/update-service-account?accessKey={target_access_key}");
+
+    assert_admin_status(
+        &env,
+        (&env.access_key, &env.secret_key, None),
+        &target_path,
+        serde_json::json!({}).to_string(),
+        StatusCode::NO_CONTENT,
+        "root no-op update across parents must succeed",
+    )
+    .await?;
+
+    let custom_policy = serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [{
+            "Effect": "Allow",
+            "Action": ["s3:GetObject"],
+            "Resource": ["arn:aws:s3:::update-scope/*"]
+        }]
+    });
+    assert_admin_status(
+        &env,
+        (&env.access_key, &env.secret_key, None),
+        &target_path,
+        serde_json::json!({ "newPolicy": custom_policy }).to_string(),
+        StatusCode::NO_CONTENT,
+        "root implied-to-custom update across parents must succeed",
+    )
+    .await?;
+
+    assert_admin_status(
+        &env,
+        (parent, parent_secret, None),
+        &target_path,
+        serde_json::json!({ "newDescription": "updated by parent" }).to_string(),
+        StatusCode::NO_CONTENT,
+        "parent with UpdateServiceAccount may update its own service account",
+    )
+    .await?;
+
+    let takeover = serde_json::json!({
+        "newSecretKey": "cross-parent-takeover-secret",
+        "newDescription": "cross-parent takeover"
+    })
+    .to_string();
+    assert_admin_status(
+        &env,
+        (ordinary, ordinary_secret, None),
+        &target_path,
+        takeover.clone(),
+        StatusCode::FORBIDDEN,
+        "ordinary user must not update another parent's service account",
+    )
+    .await?;
+    assert_admin_status(
+        &env,
+        (outsider, outsider_secret, None),
+        &target_path,
+        takeover.clone(),
+        StatusCode::FORBIDDEN,
+        "non-owner consoleAdmin must not update across parents",
+    )
+    .await?;
+
+    let (derived_access_key, derived_secret_key) = create_service_account_for(&env, outsider).await?;
+    assert_admin_status(
+        &env,
+        (&derived_access_key, &derived_secret_key, None),
+        &target_path,
+        takeover.clone(),
+        StatusCode::FORBIDDEN,
+        "service-account credential must not update across parents",
+    )
+    .await?;
+
+    let assumed = build_test_sts_client(&env.url, outsider, outsider_secret, None, "e2e-admin-update-service-account")
+        .assume_role()
+        .role_arn("arn:aws:iam::123456789012:role/update-service-account")
+        .role_session_name("update-service-account-scope")
+        .send()
+        .await?;
+    let temporary = assumed
+        .credentials()
+        .ok_or("AssumeRole response should contain credentials")?;
+    assert_admin_status(
+        &env,
+        (temporary.access_key_id(), temporary.secret_access_key(), Some(temporary.session_token())),
+        &target_path,
+        takeover,
+        StatusCode::FORBIDDEN,
+        "temporary credential must not update across parents",
+    )
+    .await?;
+
+    let info = admin_ok(
+        &env,
+        http::Method::GET,
+        &format!("/rustfs/admin/v3/info-service-account?accessKey={target_access_key}"),
+        None,
+    )
+    .await?;
+    let info: serde_json::Value = serde_json::from_str(&info)?;
+    assert_eq!(
+        info["impliedPolicy"].as_bool(),
+        Some(false),
+        "root update must replace the implied policy with a custom policy"
+    );
+    assert!(
+        info["policy"].as_str().is_some_and(|policy| policy.contains("s3:GetObject")),
+        "custom policy must round-trip through the handler: {info}"
+    );
+    assert_eq!(
+        info["description"].as_str(),
+        Some("updated by parent"),
+        "denied takeover attempts must not mutate target"
+    );
+
+    let (missing_status, missing_body) = admin_request(
+        &env.url,
+        http::Method::POST,
+        "/rustfs/admin/v3/update-service-account?accessKey=missing-service-account",
+        Some(serde_json::json!({}).to_string()),
+        &env.access_key,
+        &env.secret_key,
+    )
+    .await?;
+    assert_eq!(missing_status, StatusCode::NOT_FOUND, "missing target must fail closed: {missing_body}");
+    assert!(
+        missing_body.contains("NoSuchResource"),
+        "missing target must preserve the lookup error: {missing_body}"
+    );
+
+    env.stop_server();
+    Ok(())
+}
+
 /// Full user -> policy -> service-account lifecycle, proving each management
 /// call takes effect on the data plane, not just that the endpoint answers 200.
 #[tokio::test]

crates/e2e_test/src/bucket_logging_test.rs

@@ -16,91 +16,17 @@
 
 #[cfg(test)]
 mod tests {
-    use crate::common::{RustFSTestEnvironment, init_logging};
+    use crate::common::{RustFSTestEnvironment, init_logging, signed_s3_request};
     use aws_sdk_s3::error::ProvideErrorMetadata;
     use aws_sdk_s3::types::{
         AccelerateConfiguration, BucketAccelerateStatus, BucketLoggingStatus, IndexDocument, LoggingEnabled, Payer,
         RequestPaymentConfiguration, WebsiteConfiguration,
     };
+    use http::Method;
+    use http::header::CONTENT_TYPE;
     use serial_test::serial;
-    use std::path::PathBuf;
-    use std::process::Command;
     use tracing::info;
 
-    fn awscurl_binary_path() -> PathBuf {
-        std::env::var_os("AWSCURL_PATH")
-            .map(PathBuf::from)
-            .unwrap_or_else(|| PathBuf::from("awscurl"))
-    }
-
-    fn awscurl_available() -> bool {
-        Command::new(awscurl_binary_path()).arg("--version").output().is_ok()
-    }
-
-    fn execute_s3_awscurl(
-        method: &str,
-        url: &str,
-        access_key: &str,
-        secret_key: &str,
-    ) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
-        let output = Command::new(awscurl_binary_path())
-            .args([
-                "--service",
-                "s3",
-                "--region",
-                "us-east-1",
-                "--access_key",
-                access_key,
-                "--secret_key",
-                secret_key,
-                "-i",
-                "-X",
-                method,
-                url,
-            ])
-            .output()?;
-        if !output.status.success() {
-            let stderr = String::from_utf8_lossy(&output.stderr);
-            let stdout = String::from_utf8_lossy(&output.stdout);
-            return Err(format!("awscurl failed: stderr='{stderr}', stdout='{stdout}'").into());
-        }
-        Ok(String::from_utf8_lossy(&output.stdout).to_string())
-    }
-
-    fn parse_status(raw: &str) -> Option<u16> {
-        raw.lines()
-            .filter_map(|line| {
-                if line.starts_with("HTTP/") {
-                    line.split_whitespace().nth(1)?.parse::<u16>().ok()
-                } else {
-                    None
-                }
-            })
-            .next_back()
-    }
-
-    fn parse_body(raw: &str) -> String {
-        if let Some(pos) = raw.rfind("\r\n\r\n") {
-            return raw[pos + 4..].to_string();
-        }
-        if let Some(pos) = raw.rfind("\n\n") {
-            return raw[pos + 2..].to_string();
-        }
-        String::new()
-    }
-
-    fn parse_headers(raw: &str) -> String {
-        let start = raw.rfind("HTTP/").unwrap_or(0);
-        let tail = &raw[start..];
-        if let Some(pos) = tail.find("\r\n\r\n") {
-            return tail[..pos].to_string();
-        }
-        if let Some(pos) = tail.find("\n\n") {
-            return tail[..pos].to_string();
-        }
-        tail.to_string()
-    }
-
     #[tokio::test]
     #[serial]
     async fn test_dummy_bucket_compatibility_endpoints() {
@@ -470,10 +396,6 @@ mod tests {
     async fn test_dummy_bucket_endpoints_http_contracts() {
         init_logging();
         info!("Starting test: dummy-compat bucket API HTTP contracts");
-        if !awscurl_available() {
-            info!("Skipping test_dummy_bucket_endpoints_http_contracts: awscurl binary not found");
-            return;
-        }
 
         let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
         env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
@@ -488,56 +410,112 @@ mod tests {
             .await
             .expect("Failed to create bucket");
 
-        let logging_raw = execute_s3_awscurl("GET", &format!("{}/{bucket}?logging=", env.url), &env.access_key, &env.secret_key)
-            .expect("GetBucketLogging HTTP request failed");
-        assert_eq!(parse_status(&logging_raw), Some(200), "GetBucketLogging should return 200");
-        let logging_body = parse_body(&logging_raw);
+        let logging_response = signed_s3_request(
+            Method::GET,
+            &format!("{}/{bucket}?logging=", env.url),
+            None,
+            None,
+            &env.access_key,
+            &env.secret_key,
+        )
+        .await
+        .expect("GetBucketLogging HTTP request failed");
+        assert_eq!(logging_response.status(), 200, "GetBucketLogging should return 200");
+        let logging_body = logging_response
+            .text()
+            .await
+            .expect("Failed to read GetBucketLogging response body");
         assert!(
             logging_body.contains("<BucketLoggingStatus"),
             "GetBucketLogging response should contain BucketLoggingStatus XML, got: {logging_body}"
         );
 
-        let accel_raw = execute_s3_awscurl("GET", &format!("{}/{bucket}?accelerate=", env.url), &env.access_key, &env.secret_key)
-            .expect("GetBucketAccelerateConfiguration HTTP request failed");
-        assert_eq!(parse_status(&accel_raw), Some(200), "GetBucketAccelerateConfiguration should return 200");
-        let accel_body = parse_body(&accel_raw);
+        let accel_response = signed_s3_request(
+            Method::GET,
+            &format!("{}/{bucket}?accelerate=", env.url),
+            None,
+            None,
+            &env.access_key,
+            &env.secret_key,
+        )
+        .await
+        .expect("GetBucketAccelerateConfiguration HTTP request failed");
+        assert_eq!(accel_response.status(), 200, "GetBucketAccelerateConfiguration should return 200");
+        let accel_body = accel_response
+            .text()
+            .await
+            .expect("Failed to read GetBucketAccelerateConfiguration response body");
         assert!(
             accel_body.contains("<AccelerateConfiguration"),
             "GetBucketAccelerateConfiguration response should contain AccelerateConfiguration XML, got: {accel_body}"
         );
 
-        let payment_raw =
-            execute_s3_awscurl("GET", &format!("{}/{bucket}?requestPayment=", env.url), &env.access_key, &env.secret_key)
-                .expect("GetBucketRequestPayment HTTP request failed");
-        assert_eq!(parse_status(&payment_raw), Some(200), "GetBucketRequestPayment should return 200");
-        let payment_body = parse_body(&payment_raw);
+        let payment_response = signed_s3_request(
+            Method::GET,
+            &format!("{}/{bucket}?requestPayment=", env.url),
+            None,
+            None,
+            &env.access_key,
+            &env.secret_key,
+        )
+        .await
+        .expect("GetBucketRequestPayment HTTP request failed");
+        assert_eq!(payment_response.status(), 200, "GetBucketRequestPayment should return 200");
+        let payment_body = payment_response
+            .text()
+            .await
+            .expect("Failed to read GetBucketRequestPayment response body");
         assert!(
             payment_body.contains("<Payer>BucketOwner</Payer>"),
             "GetBucketRequestPayment should return BucketOwner payer, got: {payment_body}"
         );
 
-        let website_raw = execute_s3_awscurl("GET", &format!("{}/{bucket}?website=", env.url), &env.access_key, &env.secret_key)
-            .expect("GetBucketWebsite HTTP request failed");
+        let website_response = signed_s3_request(
+            Method::GET,
+            &format!("{}/{bucket}?website=", env.url),
+            None,
+            None,
+            &env.access_key,
+            &env.secret_key,
+        )
+        .await
+        .expect("GetBucketWebsite HTTP request failed");
         assert_eq!(
-            parse_status(&website_raw),
-            Some(404),
+            website_response.status(),
+            404,
             "GetBucketWebsite should return 404 when website config is absent"
         );
-        let website_content_type = parse_headers(&website_raw).to_ascii_lowercase();
+        let website_content_type = website_response
+            .headers()
+            .get(CONTENT_TYPE)
+            .expect("GetBucketWebsite response should include Content-Type")
+            .to_str()
+            .expect("GetBucketWebsite Content-Type should be valid ASCII")
+            .to_ascii_lowercase();
         assert!(
-            website_content_type.contains("content-type:") && website_content_type.contains("xml"),
+            website_content_type.contains("xml"),
             "GetBucketWebsite error response should be XML, got content-type: {website_content_type}"
         );
-        let website_body = parse_body(&website_raw);
+        let website_body = website_response
+            .text()
+            .await
+            .expect("Failed to read GetBucketWebsite response body");
         assert!(
             website_body.contains("<Code>NoSuchWebsiteConfiguration</Code>"),
             "GetBucketWebsite should return NoSuchWebsiteConfiguration code, got: {website_body}"
         );
 
-        let delete_raw =
-            execute_s3_awscurl("DELETE", &format!("{}/{bucket}?website=", env.url), &env.access_key, &env.secret_key)
-                .expect("DeleteBucketWebsite HTTP request failed");
-        assert_eq!(parse_status(&delete_raw), Some(204), "DeleteBucketWebsite should return 204");
+        let delete_response = signed_s3_request(
+            Method::DELETE,
+            &format!("{}/{bucket}?website=", env.url),
+            None,
+            None,
+            &env.access_key,
+            &env.secret_key,
+        )
+        .await
+        .expect("DeleteBucketWebsite HTTP request failed");
+        assert_eq!(delete_response.status(), 204, "DeleteBucketWebsite should return 204");
 
         env.stop_server();
     }

crates/e2e_test/src/bucket_stats_regression_test.rs

@@ -31,7 +31,7 @@
 
 #[cfg(test)]
 mod tests {
-    use crate::common::{RustFSTestEnvironment, awscurl_get, init_logging};
+    use crate::common::{FAST_DATA_USAGE_SCANNER_ENV, RustFSTestEnvironment, awscurl_get, init_logging};
     use aws_sdk_s3::primitives::ByteStream;
     use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
     use rustfs_data_usage::DataUsageInfo;
@@ -65,7 +65,7 @@ mod tests {
         info!("RT-09: bucket object count updates after PUT");
 
         let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
+        env.start_rustfs_server_with_env(vec![], FAST_DATA_USAGE_SCANNER_ENV)
             .await
             .expect("start RustFS");
 
@@ -88,12 +88,21 @@ mod tests {
 
         // Wait for scanner to process (up to 90 seconds)
         let mut found_nonzero = false;
+        let mut last_query_error = None;
         for attempt in 0..18 {
             sleep(Duration::from_secs(5)).await;
 
-            if let Ok(usage) = get_data_usage(&env).await
-                && let Some(bucket_usage) = usage.buckets_usage.get(bucket)
-            {
+            let usage = match get_data_usage(&env).await {
+                Ok(usage) => {
+                    last_query_error = None;
+                    usage
+                }
+                Err(err) => {
+                    last_query_error = Some(err.to_string());
+                    continue;
+                }
+            };
+            if let Some(bucket_usage) = usage.buckets_usage.get(bucket) {
                 info!("  attempt {attempt}: objectsCount = {}", bucket_usage.objects_count);
                 if bucket_usage.objects_count >= 10 {
                     found_nonzero = true;
@@ -104,7 +113,8 @@ mod tests {
 
         assert!(
             found_nonzero,
-            "RT-09 FAIL: bucket object count did not update after PUT 10 objects (regression: stats stuck at 0)"
+            "RT-09 FAIL: bucket object count did not update after PUT 10 objects (regression: stats stuck at 0); last query error: {}",
+            last_query_error.as_deref().unwrap_or("none")
         );
 
         info!("RT-09 PASS: bucket object count updates after PUT");
@@ -122,7 +132,7 @@ mod tests {
         info!("RT-09b: bucket object count updates after DELETE");
 
         let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
+        env.start_rustfs_server_with_env(vec![], FAST_DATA_USAGE_SCANNER_ENV)
             .await
             .expect("start RustFS");
 
@@ -143,6 +153,22 @@ mod tests {
                 .expect("put object");
         }
 
+        let mut found_nonzero = false;
+        for attempt in 0..18 {
+            sleep(Duration::from_secs(5)).await;
+
+            if let Ok(usage) = get_data_usage(&env).await
+                && let Some(bucket_usage) = usage.buckets_usage.get(bucket)
+            {
+                info!("  baseline attempt {attempt}: objectsCount = {}", bucket_usage.objects_count);
+                if bucket_usage.objects_count >= 5 {
+                    found_nonzero = true;
+                    break;
+                }
+            }
+        }
+        assert!(found_nonzero, "RT-09b setup failed: scanner did not observe the 5 uploaded objects");
+
         // Delete all objects
         for i in 0..5 {
             client
@@ -156,12 +182,21 @@ mod tests {
 
         // Wait for scanner to update stats (up to 90 seconds)
         let mut found_zero = false;
+        let mut last_query_error = None;
         for attempt in 0..18 {
             sleep(Duration::from_secs(5)).await;
 
-            if let Ok(usage) = get_data_usage(&env).await
-                && let Some(bucket_usage) = usage.buckets_usage.get(bucket)
-            {
+            let usage = match get_data_usage(&env).await {
+                Ok(usage) => {
+                    last_query_error = None;
+                    usage
+                }
+                Err(err) => {
+                    last_query_error = Some(err.to_string());
+                    continue;
+                }
+            };
+            if let Some(bucket_usage) = usage.buckets_usage.get(bucket) {
                 info!("  attempt {attempt}: objectsCount = {}", bucket_usage.objects_count);
                 if bucket_usage.objects_count == 0 {
                     found_zero = true;
@@ -172,7 +207,8 @@ mod tests {
 
         assert!(
             found_zero,
-            "RT-09b FAIL: bucket object count did not update to 0 after deleting all objects (regression rustfs#5615)"
+            "RT-09b FAIL: bucket object count did not update to 0 after deleting all objects (regression rustfs#5615); last query error: {}",
+            last_query_error.as_deref().unwrap_or("none")
         );
 
         info!("RT-09b PASS: bucket object count updates to 0 after DELETE");

crates/e2e_test/src/chaos.rs

@@ -40,6 +40,8 @@ use http::header::{CONTENT_TYPE, HOST};
 use rustfs_signer::constants::UNSIGNED_PAYLOAD;
 use rustfs_signer::sign_v4;
 use s3s::Body;
+use sha2::{Digest, Sha256};
+use std::collections::{BTreeMap, BTreeSet};
 use std::error::Error;
 use std::path::{Path, PathBuf};
 use tracing::info;
@@ -48,6 +50,62 @@ use walkdir::WalkDir;
 
 type ChaosResult<T> = Result<T, Box<dyn Error + Send + Sync>>;
 
+/// Physical `xl.meta` and shard-file census for one object version on one disk.
+///
+/// A successful S3 GET only proves that a quorum can serve an object. Replacement
+/// tests need this lower-level record to prove that the rebuilt target holds the
+/// `xl.meta` selected for a specific version and every `part.N` it declares.
+#[derive(Clone, Debug, Eq, PartialEq)]
+pub(crate) struct VersionShardCensus {
+    pub version_id: Option<String>,
+    pub has_xl_meta: bool,
+    pub data_dir: Option<String>,
+    pub erasure_index: Option<usize>,
+    pub expected_part_numbers: BTreeSet<usize>,
+    pub present_part_fingerprints: BTreeMap<usize, PartShardFingerprint>,
+    pub inline_data_fingerprint: Option<PartShardFingerprint>,
+}
+
+#[derive(Clone, Debug, Eq, PartialEq)]
+pub(crate) struct PartShardFingerprint {
+    pub size: u64,
+    pub sha256: String,
+}
+
+impl VersionShardCensus {
+    pub(crate) fn is_complete(&self) -> bool {
+        self.has_xl_meta
+            && self.expected_part_numbers.len() == self.present_part_fingerprints.len()
+            && self
+                .expected_part_numbers
+                .iter()
+                .all(|part_number| self.present_part_fingerprints.contains_key(part_number))
+    }
+
+    pub(crate) fn matches_manifest(&self, manifest: &Self) -> bool {
+        self.version_id == manifest.version_id
+            && self.is_complete()
+            && manifest.is_complete()
+            && self.data_dir == manifest.data_dir
+            && self.erasure_index == manifest.erasure_index
+            && self.expected_part_numbers == manifest.expected_part_numbers
+            && self.present_part_fingerprints == manifest.present_part_fingerprints
+            && self.inline_data_fingerprint == manifest.inline_data_fingerprint
+    }
+}
+
+fn sha256_hex(data: &[u8]) -> String {
+    let digest = Sha256::digest(data);
+    digest.iter().map(|byte| format!("{byte:02x}")).collect()
+}
+
+fn shard_fingerprint(data: &[u8]) -> ChaosResult<PartShardFingerprint> {
+    Ok(PartShardFingerprint {
+        size: u64::try_from(data.len())?,
+        sha256: sha256_hex(data),
+    })
+}
+
 /// Single-node RustFS server with `disk_count` local volume directories that
 /// can be faulted individually while the server is running.
 pub struct DiskFaultHarness {
@@ -219,6 +277,93 @@ impl DiskFaultHarness {
     pub fn object_metadata_exists_on_disk(&self, disk_index: usize, bucket: &str, key: &str) -> bool {
         self.disks[disk_index].join(bucket).join(key).join("xl.meta").is_file()
     }
+
+    /// Census the physical files selected by `version_id` on one disk.
+    ///
+    /// Missing metadata and missing shard files are represented in the returned
+    /// census rather than as an error so callers can poll replacement progress.
+    /// Invalid metadata or an unknown requested version remains an error: treating
+    /// either as an incomplete rebuild would hide corruption or a wrong-version
+    /// recovery result.
+    pub(crate) fn census_object_version(
+        &self,
+        disk_index: usize,
+        bucket: &str,
+        key: &str,
+        version_id: Option<&str>,
+    ) -> ChaosResult<VersionShardCensus> {
+        census_object_version_on_disk(&self.disks[disk_index], bucket, key, version_id)
+    }
+}
+
+/// Census one physical object version without requiring a single-node harness.
+/// Cluster replacement tests use the same evidence as the disk-fault tests.
+pub(crate) fn census_object_version_on_disk(
+    disk: &Path,
+    bucket: &str,
+    key: &str,
+    version_id: Option<&str>,
+) -> ChaosResult<VersionShardCensus> {
+    let version_id = version_id.map(str::to_owned);
+    let object_dir = disk.join(bucket).join(key);
+    let meta_path = object_dir.join("xl.meta");
+    if !meta_path.is_file() {
+        return Ok(VersionShardCensus {
+            version_id,
+            has_xl_meta: false,
+            data_dir: None,
+            erasure_index: None,
+            expected_part_numbers: BTreeSet::new(),
+            present_part_fingerprints: BTreeMap::new(),
+            inline_data_fingerprint: None,
+        });
+    }
+
+    let metadata = rustfs_filemeta::FileMeta::load(&std::fs::read(&meta_path)?)?;
+    let file_info = metadata.into_fileinfo(bucket, key, version_id.as_deref().unwrap_or_default(), true, false, true)?;
+    let expected_part_numbers = if file_info.inline_data() {
+        BTreeSet::new()
+    } else {
+        file_info.parts.iter().map(|part| part.number).collect()
+    };
+    let data_dir = file_info.data_dir.map(|id| id.to_string());
+    let erasure_index = Some(file_info.erasure.index);
+    let inline_data_fingerprint = file_info.data.as_deref().map(shard_fingerprint).transpose()?;
+    let part_dir = data_dir.as_ref().map_or_else(|| object_dir.clone(), |id| object_dir.join(id));
+    let present_part_fingerprints = match std::fs::read_dir(&part_dir) {
+        Ok(entries) => {
+            let mut fingerprints = BTreeMap::new();
+            for entry in entries {
+                let entry = entry?;
+                if !entry.file_type()?.is_file() {
+                    continue;
+                }
+                let file_name = entry.file_name();
+                let Some(part_number) = file_name
+                    .to_str()
+                    .and_then(|name| name.strip_prefix("part."))
+                    .and_then(|number| number.parse::<usize>().ok())
+                else {
+                    continue;
+                };
+                let data = std::fs::read(entry.path())?;
+                fingerprints.insert(part_number, shard_fingerprint(&data)?);
+            }
+            fingerprints
+        }
+        Err(error) if error.kind() == std::io::ErrorKind::NotFound => BTreeMap::new(),
+        Err(error) => return Err(error.into()),
+    };
+
+    Ok(VersionShardCensus {
+        version_id,
+        has_xl_meta: true,
+        data_dir,
+        erasure_index,
+        expected_part_numbers,
+        present_part_fingerprints,
+        inline_data_fingerprint,
+    })
 }
 
 /// `POST` a signed (SigV4, service `s3`) admin request without relying on the
@@ -257,3 +402,43 @@ pub async fn signed_admin_post(url: &str, body: Option<&str>, access_key: &str,
 
     Ok(body)
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn complete_census() -> VersionShardCensus {
+        VersionShardCensus {
+            version_id: Some("version".to_string()),
+            has_xl_meta: true,
+            data_dir: Some("data-dir".to_string()),
+            erasure_index: Some(3),
+            expected_part_numbers: BTreeSet::from([1]),
+            present_part_fingerprints: BTreeMap::from([(1, shard_fingerprint(b"part").unwrap())]),
+            inline_data_fingerprint: None,
+        }
+    }
+
+    #[test]
+    fn shard_fingerprint_uses_physical_length_and_sha256() {
+        assert_eq!(
+            shard_fingerprint(b"abc").unwrap(),
+            PartShardFingerprint {
+                size: 3,
+                sha256: "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad".to_string(),
+            }
+        );
+    }
+
+    #[test]
+    fn manifest_requires_matching_inline_payload() {
+        let mut expected = complete_census();
+        expected.expected_part_numbers.clear();
+        expected.present_part_fingerprints.clear();
+        expected.inline_data_fingerprint = Some(shard_fingerprint(b"expected").unwrap());
+        let mut changed = expected.clone();
+        changed.inline_data_fingerprint = Some(shard_fingerprint(b"changed").unwrap());
+        assert!(expected.matches_manifest(&expected));
+        assert!(!changed.matches_manifest(&expected));
+    }
+}

crates/e2e_test/src/common.rs

@@ -47,6 +47,8 @@ use walkdir::WalkDir;
 pub const DEFAULT_ACCESS_KEY: &str = "rustfsadmin";
 pub const DEFAULT_SECRET_KEY: &str = "rustfsadmin";
 pub const ENV_RUSTFS_BUILD_FEATURES: &str = "RUSTFS_BUILD_FEATURES";
+pub(crate) const FAST_DATA_USAGE_SCANNER_ENV: &[(&str, &str)] =
+    &[("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_SCANNER_START_DELAY_SECS", "0")];
 pub const TEST_BUCKET: &str = "e2e-test-bucket";
 const RUSTFS_FULL_FEATURE: &str = "full";
 
@@ -65,8 +67,24 @@ fn configured_capture_log_path(temp_dir: &str) -> Option<String> {
     capture_log_path(Path::new(&log_dir), temp_dir).map(|path| path.to_string_lossy().into_owned())
 }
 
-fn build_test_s3_config(endpoint_url: &str, access_key: &str, secret_key: &str, provider_name: &'static str) -> Config {
-    let credentials = Credentials::new(access_key, secret_key, None, None, provider_name);
+fn capture_command_logs(command: &mut Command, log_path: Option<&str>) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    let Some(log_path) = log_path else {
+        return Ok(());
+    };
+    let file = stdfs::OpenOptions::new().create(true).append(true).open(log_path)?;
+    let stderr_file = file.try_clone()?;
+    command.stdout(Stdio::from(file)).stderr(Stdio::from(stderr_file));
+    Ok(())
+}
+
+pub(crate) fn build_test_s3_config(
+    endpoint_url: &str,
+    access_key: &str,
+    secret_key: &str,
+    session_token: Option<&str>,
+    provider_name: &'static str,
+) -> Config {
+    let credentials = Credentials::new(access_key, secret_key, session_token.map(str::to_owned), None, provider_name);
     let mut config = Config::builder()
         .credentials_provider(credentials)
         .region(Region::new("us-east-1"))
@@ -81,6 +99,33 @@ fn build_test_s3_config(endpoint_url: &str, access_key: &str, secret_key: &str,
     config.build()
 }
 
+pub(crate) fn build_test_sts_client(
+    endpoint_url: &str,
+    access_key: &str,
+    secret_key: &str,
+    session_token: Option<&str>,
+    provider_name: &'static str,
+) -> aws_sdk_sts::Client {
+    let mut config = aws_sdk_sts::Config::builder()
+        .credentials_provider(aws_sdk_sts::config::Credentials::new(
+            access_key,
+            secret_key,
+            session_token.map(str::to_owned),
+            None,
+            provider_name,
+        ))
+        .region(aws_sdk_sts::config::Region::new("us-east-1"))
+        .endpoint_url(endpoint_url)
+        .retry_config(aws_sdk_sts::config::retry::RetryConfig::standard().with_max_attempts(1))
+        .behavior_version_latest();
+
+    if endpoint_url.starts_with("http://") {
+        config = config.http_client(SmithyHttpClientBuilder::new().build_http());
+    }
+
+    aws_sdk_sts::Client::from_conf(config.build())
+}
+
 pub fn workspace_root() -> PathBuf {
     let mut path = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
     path.pop(); // e2e_test
@@ -95,6 +140,57 @@ pub fn local_http_client() -> HttpClient {
         .expect("failed to build local reqwest client")
 }
 
+pub(crate) async fn signed_s3_request(
+    method: http::Method,
+    url: &str,
+    body: Option<String>,
+    content_type: Option<&str>,
+    access_key: &str,
+    secret_key: &str,
+) -> Result<reqwest::Response, Box<dyn std::error::Error + Send + Sync>> {
+    signed_s3_request_with_session_token(method, url, body, content_type, access_key, secret_key, None).await
+}
+
+async fn signed_s3_request_with_session_token(
+    method: http::Method,
+    url: &str,
+    body: Option<String>,
+    content_type: Option<&str>,
+    access_key: &str,
+    secret_key: &str,
+    session_token: Option<&str>,
+) -> Result<reqwest::Response, Box<dyn std::error::Error + Send + Sync>> {
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri.authority().ok_or("S3 URL missing authority")?.to_string();
+    let mut request = http::Request::builder()
+        .method(method.clone())
+        .uri(uri)
+        .header(HOST, authority)
+        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
+    if let Some(content_type) = content_type {
+        request = request.header(CONTENT_TYPE, content_type);
+    }
+
+    let content_length = i64::try_from(body.as_ref().map_or(0, String::len)).map_err(|_| "S3 request body is too large")?;
+    let signed = sign_v4(
+        request.body(Body::empty())?,
+        content_length,
+        access_key,
+        secret_key,
+        session_token.unwrap_or_default(),
+        "us-east-1",
+    );
+
+    let mut request = local_http_client().request(method, url);
+    for (name, value) in signed.headers() {
+        request = request.header(name, value);
+    }
+    if let Some(body) = body {
+        request = request.body(body);
+    }
+    Ok(request.send().await?)
+}
+
 /// Signs and sends an admin HTTP request with the given credentials.
 pub(crate) async fn admin_request(
     base_url: &str,
@@ -103,30 +199,23 @@ pub(crate) async fn admin_request(
     body: Option<String>,
     access_key: &str,
     secret_key: &str,
+) -> Result<(StatusCode, String), Box<dyn std::error::Error + Send + Sync>> {
+    admin_request_with_session_token(base_url, method, path_and_query, body, access_key, secret_key, None).await
+}
+
+pub(crate) async fn admin_request_with_session_token(
+    base_url: &str,
+    method: http::Method,
+    path_and_query: &str,
+    body: Option<String>,
+    access_key: &str,
+    secret_key: &str,
+    session_token: Option<&str>,
 ) -> Result<(StatusCode, String), Box<dyn std::error::Error + Send + Sync>> {
     let url = format!("{base_url}{path_and_query}");
-    let uri = url.parse::<http::Uri>()?;
-    let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
-    let mut request = http::Request::builder()
-        .method(method.clone())
-        .uri(uri)
-        .header(HOST, authority)
-        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
-    if body.is_some() {
-        request = request.header(CONTENT_TYPE, "application/json");
-    }
-
-    let content_length = i64::try_from(body.as_ref().map_or(0, String::len)).map_err(|_| "admin request body is too large")?;
-    let signed = sign_v4(request.body(Body::empty())?, content_length, access_key, secret_key, "", "us-east-1");
-
-    let mut request = local_http_client().request(method, &url);
-    for (name, value) in signed.headers() {
-        request = request.header(name, value);
-    }
-    if let Some(body) = body {
-        request = request.body(body);
-    }
-    let response = request.send().await?;
+    let content_type = body.as_ref().map(|_| "application/json");
+    let response =
+        signed_s3_request_with_session_token(method, &url, body, content_type, access_key, secret_key, session_token).await?;
     let status = response.status();
     let body = response.text().await?;
     Ok((status, body))
@@ -478,13 +567,7 @@ impl RustFSTestEnvironment {
         for (key, value) in extra_env {
             command.env(key, value);
         }
-        // Optionally capture the child's stdout+stderr to a file so the test can
-        // grep server logs (e.g. to confirm which GET reader path was taken).
-        if let Some(log_path) = &self.capture_log_path {
-            let file = stdfs::OpenOptions::new().create(true).append(true).open(log_path)?;
-            let stderr_file = file.try_clone()?;
-            command.stdout(Stdio::from(file)).stderr(Stdio::from(stderr_file));
-        }
+        capture_command_logs(&mut command, self.capture_log_path.as_deref())?;
         let process = command.args(&args).spawn()?;
 
         self.process = Some(process);
@@ -564,7 +647,12 @@ impl RustFSTestEnvironment {
 
     /// Create an AWS S3 client configured for this RustFS instance
     pub fn create_s3_client(&self) -> Client {
-        Client::from_conf(build_test_s3_config(&self.url, &self.access_key, &self.secret_key, "e2e-test"))
+        self.create_s3_client_with_credentials(&self.access_key, &self.secret_key)
+    }
+
+    /// Create an AWS S3 client with explicit credentials for this RustFS instance.
+    pub fn create_s3_client_with_credentials(&self, access_key: &str, secret_key: &str) -> Client {
+        Client::from_conf(build_test_s3_config(&self.url, access_key, secret_key, None, "e2e-test"))
     }
 
     /// Create test bucket
@@ -967,6 +1055,7 @@ pub struct RustFSTestClusterEnvironment {
     pub secret_key: String,
     pub extra_env: Vec<(String, String)>,
     pub node_extra_env: Vec<Vec<(String, String)>>,
+    pub node_capture_log_paths: Vec<Option<String>>,
     pub topology: ClusterTopology,
 }
 
@@ -1066,6 +1155,7 @@ impl RustFSTestClusterEnvironment {
             secret_key: "rustfs-cluster-test-secret".to_string(),
             extra_env,
             node_extra_env: vec![Vec::new(); topology.node_count],
+            node_capture_log_paths: vec![None; topology.node_count],
             topology,
         })
     }
@@ -1095,6 +1185,20 @@ impl RustFSTestClusterEnvironment {
         Ok(())
     }
 
+    /// Capture stdout+stderr for a single cluster node process.
+    pub fn set_node_capture_log_path<P>(
+        &mut self,
+        node_idx: usize,
+        path: P,
+    ) -> Result<(), Box<dyn std::error::Error + Send + Sync>>
+    where
+        P: Into<String>,
+    {
+        self.ensure_node_index(node_idx)?;
+        self.node_capture_log_paths[node_idx] = Some(path.into());
+        Ok(())
+    }
+
     fn ensure_node_index(&self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         if node_idx >= self.nodes.len() {
             return Err(format!("node_idx {node_idx} is invalid").into());
@@ -1184,6 +1288,7 @@ impl RustFSTestClusterEnvironment {
             for (key, value) in &self.node_extra_env[i] {
                 command.env(key, value);
             }
+            capture_command_logs(&mut command, self.node_capture_log_paths[i].as_deref())?;
 
             let process = command.current_dir(&node.data_dir).spawn()?;
 
@@ -1210,6 +1315,7 @@ impl RustFSTestClusterEnvironment {
 
         let binary_path = rustfs_binary_path();
         let volumes_arg = self.build_volumes_arg();
+        let log_path = self.node_capture_log_paths[node_idx].clone();
         let node = &mut self.nodes[node_idx];
         info!("Starting cluster node {} on {}", node_idx, node.address);
 
@@ -1228,6 +1334,7 @@ impl RustFSTestClusterEnvironment {
         for (key, value) in &self.node_extra_env[node_idx] {
             command.env(key, value);
         }
+        capture_command_logs(&mut command, log_path.as_deref())?;
 
         let process = command.current_dir(&node.data_dir).spawn()?;
         node.process = Some(process);
@@ -1296,6 +1403,7 @@ impl RustFSTestClusterEnvironment {
             &self.nodes[node_idx].url,
             &self.access_key,
             &self.secret_key,
+            None,
             "cluster-test",
         )))
     }
@@ -1478,6 +1586,7 @@ mod tests {
             secret_key: DEFAULT_SECRET_KEY.to_string(),
             extra_env: Vec::new(),
             node_extra_env: vec![Vec::new(); topology.node_count],
+            node_capture_log_paths: vec![None; topology.node_count],
             topology,
         }
     }
@@ -1573,6 +1682,16 @@ mod tests {
         );
     }
 
+    #[test]
+    fn cluster_node_log_capture_supports_per_node_paths() {
+        let mut env = fake_cluster(ClusterTopology::single_pool(3));
+        env.set_node_capture_log_path(1, "/tmp/node1.log").unwrap();
+        assert_eq!(env.node_capture_log_paths[0], None);
+        assert_eq!(env.node_capture_log_paths[1], Some("/tmp/node1.log".to_string()));
+        assert_eq!(env.node_capture_log_paths[2], None);
+        assert!(env.set_node_capture_log_path(3, "/tmp/invalid.log").is_err());
+    }
+
     #[test]
     fn cluster_node_env_rejects_invalid_index() {
         let mut env = fake_cluster(ClusterTopology::single_pool(4));

crates/e2e_test/src/data_usage_test.rs

@@ -18,7 +18,7 @@ use rustfs_data_usage::DataUsageInfo;
 use serial_test::serial;
 use tokio::time::{Duration, sleep};
 
-use crate::common::{RustFSTestEnvironment, TEST_BUCKET, awscurl_get, init_logging};
+use crate::common::{FAST_DATA_USAGE_SCANNER_ENV, RustFSTestEnvironment, TEST_BUCKET, awscurl_get, init_logging};
 
 async fn get_data_usage_info(env: &RustFSTestEnvironment) -> Result<DataUsageInfo, Box<dyn std::error::Error + Send + Sync>> {
     let url = format!("{}/rustfs/admin/v3/datausageinfo", env.url);
@@ -35,16 +35,26 @@ where
     F: FnMut(&DataUsageInfo) -> bool,
 {
     let mut last_usage = DataUsageInfo::default();
+    let mut last_query_error = None;
     for _ in 0..45 {
-        let usage = get_data_usage_info(env).await?;
-        if usage.buckets_usage.contains_key(bucket) && predicate(&usage) {
-            return Ok(usage);
+        match get_data_usage_info(env).await {
+            Ok(usage) => {
+                last_query_error = None;
+                if usage.buckets_usage.contains_key(bucket) && predicate(&usage) {
+                    return Ok(usage);
+                }
+                last_usage = usage;
+            }
+            Err(err) => last_query_error = Some(err.to_string()),
         }
-        last_usage = usage;
         sleep(Duration::from_secs(2)).await;
     }
 
-    Err(format!("bucket usage did not converge for {bucket}; last usage: {last_usage:?}").into())
+    Err(format!(
+        "bucket usage did not converge for {bucket}; last usage: {last_usage:?}; last query error: {}",
+        last_query_error.as_deref().unwrap_or("none")
+    )
+    .into())
 }
 
 /// Regression test for data usage accuracy (issue #1012).
@@ -56,7 +66,7 @@ async fn data_usage_reports_all_objects() -> Result<(), Box<dyn std::error::Erro
     init_logging();
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    env.start_rustfs_server_with_env(vec![], FAST_DATA_USAGE_SCANNER_ENV).await?;
 
     let client = env.create_s3_client();
 
@@ -74,8 +84,14 @@ async fn data_usage_reports_all_objects() -> Result<(), Box<dyn std::error::Erro
             .await?;
     }
 
-    // Query admin data usage API
-    let usage = get_data_usage_info(&env).await?;
+    let usage = wait_for_bucket_usage(&env, TEST_BUCKET, |usage| {
+        usage
+            .buckets_usage
+            .get(TEST_BUCKET)
+            .map(|bucket_usage| usage.objects_total_count >= 1000 && bucket_usage.objects_count >= 1000)
+            .unwrap_or(false)
+    })
+    .await?;
 
     // Assert total object count and per-bucket count are not truncated
     let bucket_usage = usage
@@ -108,7 +124,7 @@ async fn data_usage_reports_versioned_objects_and_delete_markers() -> Result<(),
     init_logging();
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    env.start_rustfs_server_with_env(vec![], FAST_DATA_USAGE_SCANNER_ENV).await?;
 
     let client = env.create_s3_client();
     let bucket = "data-usage-versioned";
@@ -184,8 +200,8 @@ async fn data_usage_reports_versioned_objects_and_delete_markers() -> Result<(),
     assert_eq!(usage.versions_total_count, 3, "total version count should match bucket usage");
     assert_eq!(usage.delete_markers_total_count, 1, "total delete marker count should match bucket usage");
 
-    env.stop_server();
-    env.start_rustfs_server(vec![]).await?;
+    env.restart_server_preserving_data(vec![], FAST_DATA_USAGE_SCANNER_ENV)
+        .await?;
 
     let restarted_usage = wait_for_bucket_usage(&env, bucket, |usage| {
         usage

crates/e2e_test/src/fake_s3_target/mod.rs

@@ -30,10 +30,10 @@ use s3s::access::{S3Access, S3AccessContext};
 use s3s::auth::SimpleAuth;
 use s3s::dto::{
     AbortMultipartUploadInput, AbortMultipartUploadOutput, CompleteMultipartUploadInput, CompleteMultipartUploadOutput,
-    CreateMultipartUploadInput, CreateMultipartUploadOutput, DeleteObjectInput, DeleteObjectOutput, ETag,
+    CreateMultipartUploadInput, CreateMultipartUploadOutput, DeleteMarkerEntry, DeleteObjectInput, DeleteObjectOutput, ETag,
     GetBucketVersioningInput, GetBucketVersioningOutput, GetObjectInput, GetObjectOutput, HeadBucketInput, HeadBucketOutput,
-    HeadObjectInput, HeadObjectOutput, PutObjectInput, PutObjectOutput, StreamingBlob, Timestamp, TimestampFormat,
-    UploadPartInput, UploadPartOutput,
+    HeadObjectInput, HeadObjectOutput, ListObjectVersionsInput, ListObjectVersionsOutput, ObjectVersionId, PutObjectInput,
+    PutObjectOutput, StreamingBlob, Timestamp, TimestampFormat, UploadPartInput, UploadPartOutput,
 };
 use s3s::service::{S3Service, S3ServiceBuilder};
 use s3s::validation::{AwsNameValidation, NameValidation};
@@ -91,6 +91,7 @@ pub enum Operation {
     GetObject,
     HeadObject,
     DeleteObject,
+    ListObjectVersions,
     CreateMultipartUpload,
     UploadPart,
     CompleteMultipartUpload,
@@ -109,6 +110,8 @@ pub enum FaultAction {
     /// already have buffered the rest of the current frame; the journal reports
     /// the threshold, and the backend never receives or stores the request.
     DisconnectAfterBytes(usize),
+    /// Apply the request, then close the connection before returning its response.
+    DisconnectAfterResponse,
     /// Drain a request body in fixed-size slices, sleeping after every slice.
     SlowDrain { chunk_bytes: usize, delay: Duration },
     /// Store the request normally but replace the response ETag.
@@ -134,12 +137,15 @@ pub struct RequestRecord {
 #[derive(Default)]
 struct ControlState {
     scripts: HashMap<Operation, VecDeque<FaultAction>>,
+    keyed_scripts: HashMap<(Operation, String), VecDeque<FaultAction>>,
     requests: VecDeque<RequestRecord>,
     next_sequence: u64,
 }
 
 #[derive(Default)]
 struct StoreState {
+    assign_own_version_ids: bool,
+    assign_own_multipart_version_ids: bool,
     buckets: HashMap<String, BucketState>,
     uploads: HashMap<String, MultipartState>,
     total_bytes: usize,
@@ -352,25 +358,60 @@ impl FakeS3Target {
         state.buckets.entry(bucket).or_default();
     }
 
+    /// Remove all retained object versions while preserving the bucket.
+    pub fn clear_bucket_objects(&self, bucket: &str) {
+        let mut state = lock(&self.backend.store);
+        let (removed_versions, removed_bytes) = state
+            .buckets
+            .get_mut(bucket)
+            .expect("fake target bucket must exist")
+            .objects
+            .drain()
+            .flat_map(|(_, versions)| versions)
+            .fold((0usize, 0usize), |(count, bytes), version| (count + 1, bytes + version.body.len()));
+        state.total_versions = state
+            .total_versions
+            .checked_sub(removed_versions)
+            .expect("fake target version accounting must not underflow");
+        state.total_bytes = state
+            .total_bytes
+            .checked_sub(removed_bytes)
+            .expect("fake target byte accounting must not underflow");
+    }
+
+    pub fn has_object(&self, bucket: &str, key: &str) -> bool {
+        lock(&self.backend.store)
+            .buckets
+            .get(bucket)
+            .and_then(|bucket| bucket.objects.get(key))
+            .and_then(|versions| versions.last())
+            .is_some_and(|version| !version.delete_marker)
+    }
+
+    /// Make the target mint its own version ids instead of mirroring the
+    /// forwarded source version id — models a generic S3 service.
+    pub fn assign_own_version_ids(&self, enabled: bool) {
+        lock(&self.backend.store).assign_own_version_ids = enabled;
+    }
+
+    /// Mint own version ids for the multipart path only — models a target
+    /// that adopts PutObject version ids but not CreateMultipartUpload ones.
+    pub fn assign_own_multipart_version_ids(&self, enabled: bool) {
+        lock(&self.backend.store).assign_own_multipart_version_ids = enabled;
+    }
+
+    pub fn active_multipart_upload_count(&self) -> usize {
+        lock(&self.backend.store).uploads.len()
+    }
+
     /// Queue `times` copies of a fault for one operation.
     pub fn inject(&self, operation: Operation, action: FaultAction, times: usize) {
         if times == 0 {
             return;
         }
-        if let FaultAction::SlowDrain { chunk_bytes: 0, .. } = action {
-            panic!("slow-drain chunk size must be non-zero");
-        }
-        match &action {
-            FaultAction::Delay(duration) if *duration > MAX_FAULT_DURATION => {
-                panic!("fault delay must not exceed 30 seconds");
-            }
-            FaultAction::SlowDrain { delay, .. } if *delay >= MAX_FAULT_DURATION => {
-                panic!("slow-drain slice delay must be below 30 seconds");
-            }
-            _ => {}
-        }
+        validate_fault_action(&action);
         let mut state = lock(&self.control);
-        let queued = state.scripts.values().map(VecDeque::len).sum::<usize>();
+        let queued = queued_fault_count(&state);
         if queued.checked_add(times).is_none_or(|total| total > MAX_SCRIPTED_FAULTS) {
             panic!("fake target queues at most 4096 scripted faults");
         }
@@ -381,8 +422,28 @@ impl FakeS3Target {
             .extend(std::iter::repeat_n(action, times));
     }
 
+    /// Queue faults for one exact object key without affecting concurrent requests.
+    pub fn inject_for_key(&self, operation: Operation, key: impl Into<String>, action: FaultAction, times: usize) {
+        if times == 0 {
+            return;
+        }
+        validate_fault_action(&action);
+        let mut state = lock(&self.control);
+        let queued = queued_fault_count(&state);
+        if queued.checked_add(times).is_none_or(|total| total > MAX_SCRIPTED_FAULTS) {
+            panic!("fake target queues at most 4096 scripted faults");
+        }
+        state
+            .keyed_scripts
+            .entry((operation, key.into()))
+            .or_default()
+            .extend(std::iter::repeat_n(action, times));
+    }
+
     pub fn clear_faults(&self) {
-        lock(&self.control).scripts.clear();
+        let mut state = lock(&self.control);
+        state.scripts.clear();
+        state.keyed_scripts.clear();
     }
 
     pub fn requests(&self) -> Vec<RequestRecord> {
@@ -393,6 +454,25 @@ impl FakeS3Target {
         lock(&self.control).requests.drain(..).collect()
     }
 
+    /// Stored versions for one key as `(version_id, is_delete_marker)`, oldest
+    /// first. Empty when the bucket or key does not exist. Lets purge tests
+    /// assert on the target's actual state instead of inferring it from the
+    /// request journal (a versioned DELETE is a silent no-op for missing ids).
+    pub fn stored_versions(&self, bucket: &str, key: &str) -> Vec<(String, bool)> {
+        let state = lock(&self.backend.store);
+        state
+            .buckets
+            .get(bucket)
+            .and_then(|bucket_state| bucket_state.objects.get(key))
+            .map(|versions| {
+                versions
+                    .iter()
+                    .map(|version| (version.version_id.clone(), version.delete_marker))
+                    .collect()
+            })
+            .unwrap_or_default()
+    }
+
     pub async fn shutdown(mut self) {
         let _ = self.shutdown.send(true);
         if let Some(task) = self.task.take() {
@@ -420,6 +500,25 @@ fn lock<T>(mutex: &Mutex<T>) -> MutexGuard<'_, T> {
     mutex.lock().unwrap_or_else(std::sync::PoisonError::into_inner)
 }
 
+fn validate_fault_action(action: &FaultAction) {
+    if let FaultAction::SlowDrain { chunk_bytes: 0, .. } = action {
+        panic!("slow-drain chunk size must be non-zero");
+    }
+    match action {
+        FaultAction::Delay(duration) if *duration > MAX_FAULT_DURATION => {
+            panic!("fault delay must not exceed 30 seconds");
+        }
+        FaultAction::SlowDrain { delay, .. } if *delay >= MAX_FAULT_DURATION => {
+            panic!("slow-drain slice delay must be below 30 seconds");
+        }
+        _ => {}
+    }
+}
+
+fn queued_fault_count(state: &ControlState) -> usize {
+    state.scripts.values().map(VecDeque::len).sum::<usize>() + state.keyed_scripts.values().map(VecDeque::len).sum::<usize>()
+}
+
 #[async_trait]
 impl S3Access for FaultAccess {
     async fn check(&self, context: &mut S3AccessContext<'_>) -> S3Result<()> {
@@ -492,7 +591,12 @@ fn record_request(
     content_length: Option<u64>,
 ) -> Option<RequestFault> {
     let mut state = lock(control);
-    let action = state.scripts.get_mut(&operation).and_then(VecDeque::pop_front);
+    let action = parsed
+        .key
+        .as_ref()
+        .and_then(|key| state.keyed_scripts.get_mut(&(operation, key.clone())))
+        .and_then(VecDeque::pop_front)
+        .or_else(|| state.scripts.get_mut(&operation).and_then(VecDeque::pop_front));
     state.next_sequence += 1;
     let sequence = state.next_sequence;
     if state.requests.len() == MAX_REQUEST_RECORDS {
@@ -569,12 +673,14 @@ fn parse_request(method: &Method, uri: &Uri) -> ParsedRequest {
     let operation = match (method, key.is_some()) {
         (&Method::HEAD, false) => Operation::HeadBucket,
         (&Method::GET, false) if query.contains_key("versioning") => Operation::GetBucketVersioning,
+        (&Method::GET, false) if query.contains_key("versions") => Operation::ListObjectVersions,
         (&Method::PUT, true) if upload_id.is_some() && part_number.is_some() => Operation::UploadPart,
         (&Method::PUT, true) if upload_id.is_some() || query.contains_key("partNumber") => Operation::Unknown,
         (&Method::POST, true) if query.contains_key("uploads") => Operation::CreateMultipartUpload,
         (&Method::POST, true) if upload_id.is_some() => Operation::CompleteMultipartUpload,
         (&Method::DELETE, true) if upload_id.is_some() => Operation::AbortMultipartUpload,
-        (&Method::PUT, true) if only_query_keys(&[]) => Operation::PutObject,
+        // A replication PUT addresses the source version via `?versionId=`.
+        (&Method::PUT, true) if only_query_keys(&["versionId"]) => Operation::PutObject,
         (&Method::GET, true) if only_query_keys(&["versionId"]) => Operation::GetObject,
         (&Method::HEAD, true) if only_query_keys(&["versionId"]) => Operation::HeadObject,
         (&Method::DELETE, true) if only_query_keys(&["versionId"]) => Operation::DeleteObject,
@@ -624,10 +730,17 @@ fn validate_retained_identifier(value: String, field: &str) -> S3Result<String>
     }
 }
 
-fn new_version_id(headers: &HeaderMap) -> S3Result<String> {
+/// `assign_own` models a target that mints its own version ids (a generic S3
+/// service): the forwarded source-version-id header is validated but NOT
+/// mirrored into the stored version.
+fn new_version_id(headers: &HeaderMap, assign_own: bool) -> S3Result<String> {
     let Some(value) = header_value(headers, &SOURCE_VERSION_ID_HEADERS) else {
         return Ok(Uuid::new_v4().to_string());
     };
+    if assign_own {
+        validate_retained_identifier(value.trim().to_owned(), "source version ID")?;
+        return Ok(Uuid::new_v4().to_string());
+    }
     let value = validate_retained_identifier(value.trim().to_owned(), "source version ID")?;
     let version_id = Uuid::parse_str(&value).map_err(|_| s3s::s3_error!(InvalidArgument, "source version ID must be a UUID"))?;
     Ok(version_id.to_string())
@@ -712,7 +825,10 @@ async fn apply_non_body_fault(fault: Option<&RequestFault>, control: &Mutex<Cont
             update_consumed(control, fault.expect("matched fault").sequence, 0);
             Err(scripted_disconnect_error())
         }
-        Some(FaultAction::SlowDrain { .. }) | Some(FaultAction::WrongEtag) | None => Ok(()),
+        Some(FaultAction::SlowDrain { .. })
+        | Some(FaultAction::WrongEtag)
+        | Some(FaultAction::DisconnectAfterResponse)
+        | None => Ok(()),
     }
 }
 
@@ -753,7 +869,7 @@ async fn collect_stream(
             Some(FaultAction::SlowDrain { chunk_bytes, delay }) => {
                 return collect_stream_slow(body, capacity, *chunk_bytes, *delay).await;
             }
-            Some(FaultAction::WrongEtag) | None => {}
+            Some(FaultAction::WrongEtag) | Some(FaultAction::DisconnectAfterResponse) | None => {}
         }
 
         let mut output = BytesMut::with_capacity(capacity);
@@ -815,6 +931,9 @@ fn apply_response_fault<T>(mut response: S3Response<T>, fault: Option<&RequestFa
     if fault.is_some_and(|fault| fault.action == FaultAction::WrongEtag) {
         response.headers.insert(ETAG, HeaderValue::from_static(WRONG_ETAG));
     }
+    if fault.is_some_and(|fault| fault.action == FaultAction::DisconnectAfterResponse) {
+        response.headers.insert(DISCONNECT_HEADER, HeaderValue::from_static("true"));
+    }
     response
 }
 
@@ -1003,6 +1122,63 @@ impl S3 for FakeBackend {
         ))
     }
 
+    /// Prefix + max-keys subset only — enough for the replication-check probe
+    /// key allocation. No pagination markers or delimiter folding.
+    async fn list_object_versions(
+        &self,
+        req: S3Request<ListObjectVersionsInput>,
+    ) -> S3Result<S3Response<ListObjectVersionsOutput>> {
+        let fault = request_fault(&req);
+        apply_non_body_fault(fault.as_ref(), &self.control).await?;
+        let state = lock(&self.store);
+        let Some(bucket_state) = state.buckets.get(&req.input.bucket) else {
+            return Err(s3s::s3_error!(NoSuchBucket, "bucket does not exist"));
+        };
+        let prefix = req.input.prefix.as_deref().unwrap_or_default();
+        let max_keys = req.input.max_keys.unwrap_or(1000).max(0) as usize;
+
+        let mut keys: Vec<&String> = bucket_state.objects.keys().filter(|key| key.starts_with(prefix)).collect();
+        keys.sort();
+
+        let mut versions = Vec::new();
+        let mut delete_markers = Vec::new();
+        'keys: for key in keys {
+            for version in bucket_state.objects[key].iter().rev() {
+                if versions.len() + delete_markers.len() >= max_keys {
+                    break 'keys;
+                }
+                if version.delete_marker {
+                    delete_markers.push(DeleteMarkerEntry {
+                        key: Some(key.clone()),
+                        version_id: Some(ObjectVersionId::from(version.version_id.clone())),
+                        last_modified: Some(version.last_modified.clone()),
+                        ..Default::default()
+                    });
+                } else {
+                    versions.push(s3s::dto::ObjectVersion {
+                        key: Some(key.clone()),
+                        version_id: Some(ObjectVersionId::from(version.version_id.clone())),
+                        last_modified: Some(version.last_modified.clone()),
+                        e_tag: Some(ETag::Strong(version.e_tag.clone())),
+                        size: Some(version.body.len() as i64),
+                        ..Default::default()
+                    });
+                }
+            }
+        }
+        drop(state);
+
+        Ok(apply_response_fault(
+            S3Response::new(ListObjectVersionsOutput {
+                name: Some(req.input.bucket),
+                versions: Some(versions),
+                delete_markers: Some(delete_markers),
+                ..Default::default()
+            }),
+            fault.as_ref(),
+        ))
+    }
+
     async fn put_object(&self, req: S3Request<PutObjectInput>) -> S3Result<S3Response<PutObjectOutput>> {
         let fault = request_fault(&req);
         let _body_permit = timeout(MAX_FAULT_DURATION, Arc::clone(&self.body_limit).acquire_owned())
@@ -1013,7 +1189,8 @@ impl S3 for FakeBackend {
         let input = req.input;
         let body = collect_stream(input.body, input.content_length, fault.as_ref(), &self.control).await?;
         validate_stored_metadata(&input.content_type, &input.metadata)?;
-        let version_id = new_version_id(&headers)?;
+        let assign_own = lock(&self.store).assign_own_version_ids;
+        let version_id = new_version_id(&headers, assign_own)?;
         let e_tag = match source_etag(&headers)? {
             Some(value) => value,
             None => {
@@ -1056,7 +1233,7 @@ impl S3 for FakeBackend {
                 content_type: version.content_type,
                 metadata: version.metadata,
                 e_tag: Some(ETag::Strong(version.e_tag)),
-                last_modified: Some(version.last_modified),
+                last_modified: Some(version.last_modified.clone()),
                 version_id: Some(version.version_id),
                 ..Default::default()
             }),
@@ -1078,7 +1255,7 @@ impl S3 for FakeBackend {
                 content_type: version.content_type,
                 metadata: version.metadata,
                 e_tag: Some(ETag::Strong(version.e_tag)),
-                last_modified: Some(version.last_modified),
+                last_modified: Some(version.last_modified.clone()),
                 version_id: Some(version.version_id),
                 ..Default::default()
             }),
@@ -1147,7 +1324,9 @@ impl S3 for FakeBackend {
             ));
         }
 
-        let version_id = new_version_id(&headers)?;
+        // `state` is the live store guard: read the flag from it. Re-locking
+        // would self-deadlock (the store mutex is not reentrant).
+        let version_id = new_version_id(&headers, state.assign_own_version_ids)?;
         upsert_version(
             &mut state,
             &input.bucket,
@@ -1187,12 +1366,16 @@ impl S3 for FakeBackend {
         ensure_upload_budget(&state)?;
         validate_stored_metadata(&input.content_type, &input.metadata)?;
         let upload_id = Uuid::new_v4().to_string();
+        // Read the flag before the mutable borrow of `state.uploads` below
+        // (and never re-lock the store: the mutex is not reentrant).
+        let mint_own = state.assign_own_version_ids || state.assign_own_multipart_version_ids;
+        let version_id = new_version_id(&headers, mint_own)?;
         state.uploads.insert(
             upload_id.clone(),
             MultipartState {
                 bucket: input.bucket.clone(),
                 key: input.key.clone(),
-                version_id: new_version_id(&headers)?,
+                version_id,
                 content_type: input.content_type,
                 metadata: input.metadata,
                 parts: BTreeMap::new(),

crates/e2e_test/src/get_codec_streaming_compat_test.rs

@@ -189,8 +189,6 @@ mod tests {
             ("RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT", "100"),
             ("RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED", "true"),
             ("RUSTFS_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED", "true"),
-            // Lower the min-size floor so every non-inline object below is eligible.
-            ("RUSTFS_GET_CODEC_STREAMING_MIN_SIZE", "4096"),
             // Route multipart objects through per-part codec streaming too.
             ("RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE", "true"),
             // Lock optimization is on by default, but pin it so the gate's
@@ -315,6 +313,13 @@ mod tests {
                 },
                 payload(64 * 1024, 2),
             ),
+            (
+                Shape {
+                    key: "small-non-inline-256kib-plus",
+                    expect_large: true,
+                },
+                payload(256 * 1024 + 1, 6),
+            ),
             (
                 Shape {
                     key: "mid-1_5mib",

crates/e2e_test/src/group_delete_test.rs

@@ -14,7 +14,7 @@
 
 //! E2E tests for group management (fixes #2028).
 
-use crate::common::{RustFSTestEnvironment, awscurl_delete, awscurl_get, awscurl_put, init_logging};
+use crate::common::{RustFSTestEnvironment, admin_request, awscurl_delete, awscurl_get, awscurl_put, init_logging};
 use aws_sdk_s3::config::{Credentials, Region};
 use aws_sdk_s3::{Client, Config};
 use serial_test::serial;
@@ -32,6 +32,56 @@ fn create_user_s3_client(env: &RustFSTestEnvironment, access_key: &str, secret_k
     Client::from_conf(config)
 }
 
+#[tokio::test(flavor = "multi_thread")]
+async fn update_group_members_rejects_invalid_new_group_names() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+
+    let invalid_groups = [
+        ("test group", "group name contains whitespace"),
+        ("test=group", "group name contains reserved characters =,"),
+        ("test,group", "group name contains reserved characters =,"),
+    ];
+
+    for (group, expected_message) in invalid_groups {
+        let body = serde_json::json!({
+            "group": group,
+            "members": [],
+            "isRemove": false,
+            "groupStatus": "enabled"
+        })
+        .to_string();
+        let (status, response_body) = admin_request(
+            &env.url,
+            http::Method::PUT,
+            "/rustfs/admin/v3/update-group-members",
+            Some(body),
+            &env.access_key,
+            &env.secret_key,
+        )
+        .await?;
+
+        assert_eq!(
+            status,
+            reqwest::StatusCode::BAD_REQUEST,
+            "invalid group {group:?} must return HTTP 400, body: {response_body}"
+        );
+        assert!(
+            response_body.contains("<Code>InvalidArgument</Code>"),
+            "invalid group {group:?} must return InvalidArgument, body: {response_body}"
+        );
+        assert!(
+            response_body.contains(&format!("<Message>{expected_message}</Message>")),
+            "invalid group {group:?} returned an unexpected message: {response_body}"
+        );
+    }
+
+    env.stop_server();
+    Ok(())
+}
+
 /// Test that deleting a group with members fails, and deleting an empty group succeeds.
 #[tokio::test(flavor = "multi_thread")]
 #[serial]

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -431,4 +431,104 @@ mod tests {
         )
         .into())
     }
+
+    /// Issue #5850: `background-heal/status` must answer while a peer is down.
+    ///
+    /// Exercises the production path in `read_cluster_heal_status` end to end,
+    /// which the unit tests around `merge_peer_heal_statuses` cannot: with one
+    /// node stopped, the endpoint must return 200 with
+    /// `clusterStatusComplete: false` and an explicit `degraded` (or, when
+    /// heal work is known active, `active`) state — never the previous
+    /// cluster-wide 500 — and must return to a complete, non-degraded answer
+    /// once the node rejoins. Reverting either all-or-nothing gate (the
+    /// topology early-return or the merge hard-fail) turns the down-window
+    /// response into a 500 and fails this test.
+    #[tokio::test]
+    #[serial]
+    async fn test_background_heal_status_degrades_while_peer_down_and_recovers_after_rejoin()
+    -> Result<(), Box<dyn Error + Send + Sync>> {
+        init_logging();
+        info!("Issue #5850: background-heal/status must degrade, not 500, while a peer is down");
+
+        let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
+        cluster.set_env("RUSTFS_UNSAFE_BYPASS_DISK_CHECK", "true");
+        cluster.start().await?;
+
+        let status_url = format!("{}/rustfs/admin/v3/background-heal/status", cluster.nodes[0].url);
+        // Owned copies: the closure must not borrow `cluster`, which
+        // stop_node/start_node need mutably between polls.
+        let access_key = cluster.access_key.clone();
+        let secret_key = cluster.secret_key.clone();
+        let fetch_status = || async {
+            let body = signed_admin_post(&status_url, None, &access_key, &secret_key).await?;
+            let json: serde_json::Value =
+                serde_json::from_str(&body).map_err(|err| format!("heal status response is not JSON ({err}): {body}"))?;
+            Ok::<serde_json::Value, Box<dyn Error + Send + Sync>>(json)
+        };
+
+        // Healthy cluster: the answer must be definitive. Poll briefly — the
+        // peer grid may still be settling right after start().
+        let mut healthy = fetch_status().await?;
+        for _ in 0..30 {
+            if healthy["clusterStatusComplete"] == serde_json::Value::Bool(true) {
+                break;
+            }
+            sleep(Duration::from_secs(1)).await;
+            healthy = fetch_status().await?;
+        }
+        assert_eq!(
+            healthy["clusterStatusComplete"],
+            serde_json::Value::Bool(true),
+            "healthy cluster should report a complete heal status: {healthy}"
+        );
+
+        cluster.stop_node(1)?;
+
+        // While the peer is down every response must stay 200 (signed_admin_post
+        // fails on any non-2xx, so the old 500 fails the test immediately) and
+        // must degrade to an explicitly-partial answer. The peer query timeout
+        // is 5 s, so a couple of polls are enough for the dead peer to surface.
+        let mut degraded = serde_json::Value::Null;
+        for _ in 0..30 {
+            degraded = fetch_status().await?;
+            if degraded["clusterStatusComplete"] == serde_json::Value::Bool(false) {
+                break;
+            }
+            sleep(Duration::from_secs(1)).await;
+        }
+        assert_eq!(
+            degraded["clusterStatusComplete"],
+            serde_json::Value::Bool(false),
+            "heal status must mark itself partial while a peer is down: {degraded}"
+        );
+        let state = degraded["state"].as_str().unwrap_or_default();
+        assert!(
+            state == "degraded" || state == "active",
+            "a partial answer must be labeled degraded (or active for known work), got {state:?}: {degraded}"
+        );
+
+        cluster.start_node(1).await?;
+
+        // After the rejoin the endpoint must return to a definitive answer.
+        let mut recovered = serde_json::Value::Null;
+        for _ in 0..60 {
+            recovered = fetch_status().await?;
+            if recovered["clusterStatusComplete"] == serde_json::Value::Bool(true) {
+                break;
+            }
+            sleep(Duration::from_secs(1)).await;
+        }
+        assert_eq!(
+            recovered["clusterStatusComplete"],
+            serde_json::Value::Bool(true),
+            "heal status should be complete again after the node rejoined: {recovered}"
+        );
+        assert_ne!(
+            recovered["state"].as_str().unwrap_or_default(),
+            "degraded",
+            "a complete answer must not be labeled degraded: {recovered}"
+        );
+
+        Ok(())
+    }
 }

crates/e2e_test/src/inline_fast_path_cluster_test.rs

@@ -2211,11 +2211,6 @@ async fn four_node_manual_transition_distributed_admission_conflict_reports_stat
             "queue_snapshot.{field} must be readable in terminal status: {terminal}"
         );
     }
-    assert!(
-        cold_tier_object_count(&cold_client).await? < 64,
-        "queue pressure should leave at least one object untransitioned"
-    );
-
     Ok(())
 }
 

crates/e2e_test/src/kms/bucket_default_encryption_test.rs

@@ -21,9 +21,12 @@
 
 use super::common::LocalKMSTestEnvironment;
 use crate::common::{TEST_BUCKET, init_logging};
+use aws_sdk_s3::primitives::ByteStream;
 use aws_sdk_s3::types::{
-    ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule,
+    ChecksumAlgorithm, ChecksumMode, CompletedMultipartUpload, CompletedPart, ServerSideEncryption,
+    ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule,
 };
+use rustfs_rio::{Checksum, ChecksumType};
 use serial_test::serial;
 use tracing::{debug, info, warn};
 
@@ -273,7 +276,7 @@ async fn test_bucket_default_sse_kms_put_object() -> Result<(), Box<dyn std::err
 /// Test 3: When bucket is configured with default encryption, create_multipart_upload should inherit the configuration
 #[tokio::test]
 #[serial]
-async fn test_bucket_default_encryption_multipart_upload() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     init_logging();
     info!("Testing bucket default encryption impact on create_multipart_upload");
 
@@ -309,15 +312,16 @@ async fn test_bucket_default_encryption_multipart_upload() -> Result<(), Box<dyn
         .await
         .expect("Failed to set bucket encryption");
 
-    // Step 2: Create multipart upload (without specifying encryption parameters)
-    info!("Creating multipart upload (without specifying encryption parameters, should use bucket default configuration)");
-    let test_key = "test-multipart-bucket-default.txt";
+    // Step 2: Declare CRC32 without specifying encryption parameters. The AWS SDK
+    // calculates each UploadPart checksum and sends it as a flexible checksum.
+    info!("Creating CRC32 multipart upload that should use bucket default encryption");
+    let test_key = "test-multipart-bucket-default-crc32.bin";
 
     let create_multipart_response = s3_client
         .create_multipart_upload()
         .bucket(TEST_BUCKET)
         .key(test_key)
-        // Note: No encryption parameters specified here, should use bucket default configuration
+        .checksum_algorithm(ChecksumAlgorithm::Crc32)
         .send()
         .await
         .expect("Failed to create multipart upload");
@@ -343,28 +347,61 @@ async fn test_bucket_default_encryption_multipart_upload() -> Result<(), Box<dyn
         "create_multipart_upload response should contain correct KMS key ID"
     );
 
-    // Step 3: Upload a part and complete multipart upload
-    info!("Uploading part and completing multipart upload");
-    let test_data = b"test-multipart-bucket-default-encryption-data";
+    // Step 3: Upload two parts. The first is exactly the S3 minimum size so this
+    // follows the same managed SSE-KMS multipart path as issue #5756.
+    const PART_SIZE: usize = 5 * 1024 * 1024;
+    let part1: Vec<u8> = (0..PART_SIZE).map(|i| (i % 251) as u8).collect();
+    let part2: Vec<u8> = (0..1024 * 1024).map(|i| ((i + 17) % 251) as u8).collect();
+    let expected_body: Vec<u8> = part1.iter().chain(&part2).copied().collect();
 
-    // Upload part 1
-    let upload_part_response = s3_client
-        .upload_part()
-        .bucket(TEST_BUCKET)
-        .key(test_key)
-        .upload_id(upload_id)
-        .part_number(1)
-        .body(test_data.to_vec().into())
-        .send()
-        .await
-        .expect("Failed to upload part");
+    let upload_part = |part_number: i32, body: Vec<u8>| {
+        s3_client
+            .upload_part()
+            .bucket(TEST_BUCKET)
+            .key(test_key)
+            .upload_id(upload_id)
+            .part_number(part_number)
+            .checksum_algorithm(ChecksumAlgorithm::Crc32)
+            .body(ByteStream::from(body))
+            .send()
+    };
 
-    let etag = upload_part_response.e_tag().unwrap().to_string();
+    let expected_part1_crc32 = Checksum::new_from_data(ChecksumType::CRC32, &part1)
+        .expect("calculate part 1 CRC32")
+        .encoded;
+    let upload1 = upload_part(1, part1).await.expect("Failed to upload part 1 with CRC32");
+    assert_eq!(
+        upload1.checksum_crc32(),
+        Some(expected_part1_crc32.as_str()),
+        "UploadPart must return the CRC32 calculated over plaintext"
+    );
+
+    let expected_part2_crc32 = Checksum::new_from_data(ChecksumType::CRC32, &part2)
+        .expect("calculate part 2 CRC32")
+        .encoded;
+    let upload2 = upload_part(2, part2).await.expect("Failed to upload part 2 with CRC32");
+    assert_eq!(
+        upload2.checksum_crc32(),
+        Some(expected_part2_crc32.as_str()),
+        "UploadPart must return the CRC32 calculated over plaintext"
+    );
 
     // Complete multipart upload
-    let completed_part = aws_sdk_s3::types::CompletedPart::builder()
-        .part_number(1)
-        .e_tag(&etag)
+    let completed_upload = CompletedMultipartUpload::builder()
+        .parts(
+            CompletedPart::builder()
+                .part_number(1)
+                .e_tag(upload1.e_tag().expect("No ETag for part 1"))
+                .checksum_crc32(upload1.checksum_crc32().expect("No CRC32 for part 1"))
+                .build(),
+        )
+        .parts(
+            CompletedPart::builder()
+                .part_number(2)
+                .e_tag(upload2.e_tag().expect("No ETag for part 2"))
+                .checksum_crc32(upload2.checksum_crc32().expect("No CRC32 for part 2"))
+                .build(),
+        )
         .build();
 
     let complete_multipart_response = s3_client
@@ -372,11 +409,7 @@ async fn test_bucket_default_encryption_multipart_upload() -> Result<(), Box<dyn
         .bucket(TEST_BUCKET)
         .key(test_key)
         .upload_id(upload_id)
-        .multipart_upload(
-            aws_sdk_s3::types::CompletedMultipartUpload::builder()
-                .parts(completed_part)
-                .build(),
-        )
+        .multipart_upload(completed_upload)
         .send()
         .await
         .expect("Failed to complete multipart upload");
@@ -400,6 +433,7 @@ async fn test_bucket_default_encryption_multipart_upload() -> Result<(), Box<dyn
         .get_object()
         .bucket(TEST_BUCKET)
         .key(test_key)
+        .checksum_mode(ChecksumMode::Enabled)
         .send()
         .await
         .expect("Failed to get object");
@@ -410,6 +444,13 @@ async fn test_bucket_default_encryption_multipart_upload() -> Result<(), Box<dyn
         Some(&ServerSideEncryption::AwsKms),
         "Final object should contain SSE-KMS encryption information"
     );
+    if let Some(completed_crc32) = complete_multipart_response.checksum_crc32() {
+        assert_eq!(
+            get_response.checksum_crc32(),
+            Some(completed_crc32),
+            "GetObject should return the persisted composite CRC32 when completion reports it"
+        );
+    }
 
     // Verify data integrity
     let downloaded_data = get_response
@@ -418,7 +459,11 @@ async fn test_bucket_default_encryption_multipart_upload() -> Result<(), Box<dyn
         .await
         .expect("Failed to collect body")
         .into_bytes();
-    assert_eq!(&downloaded_data[..], test_data, "Downloaded data should match original data");
+    assert_eq!(
+        downloaded_data.as_ref(),
+        expected_body.as_slice(),
+        "Downloaded data should match the uploaded multipart body"
+    );
 
     // Cleanup is handled automatically when the test environment is dropped
     info!("Test passed: bucket default encryption correctly applied to multipart upload");

crates/e2e_test/src/lib.rs

@@ -39,6 +39,10 @@ pub mod fault_proxy;
 #[cfg(test)]
 mod reliability_disk_fault_test;
 
+// Privileged Linux-only 3x4 replacement rebuild proof for rustfs#5869/#1791.
+#[cfg(all(test, target_os = "linux"))]
+mod replacement_privileged_e2e_test;
+
 // dist-13 (backlog#1150/#1155): e2e regression net proving a large-object
 // degraded EC read never returns a silently truncated body (rustfs#4594/#4560/#4585).
 #[cfg(test)]
@@ -290,6 +294,14 @@ mod overwrite_cleanup_regression_test;
 #[cfg(test)]
 mod list_buckets_double_slash_test;
 
+// Regression coverage for bucket-scoped ListBuckets authorization fallback.
+#[cfg(test)]
+mod list_buckets_auth_test;
+
+// ListBuckets visibility follows IAM authorization, not bucket policy.
+#[cfg(test)]
+mod list_buckets_iam_filter_test;
+
 // Regression test for backlog#629(b): region-aware CreateBucket SigV4.
 #[cfg(test)]
 mod create_bucket_region_test;

crates/e2e_test/src/list_buckets_auth_test.rs

@@ -0,0 +1,88 @@
+// Copyright 2026 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Regression coverage for the MinIO-compatible filtered ListBuckets fallback.
+
+use crate::common::{RustFSTestEnvironment, admin_ok, init_logging};
+use std::error::Error;
+
+type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
+
+#[tokio::test]
+async fn bucket_scoped_policy_returns_only_authorized_bucket() -> TestResult {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+
+    let root_client = env.create_s3_client();
+    let allowed_bucket = "list-buckets-authorized";
+    let hidden_bucket = "list-buckets-hidden";
+    let user = "listbucketsuser";
+    let secret = "listbucketssecret";
+    let policy = "list-buckets-scoped";
+
+    root_client.create_bucket().bucket(allowed_bucket).send().await?;
+    root_client.create_bucket().bucket(hidden_bucket).send().await?;
+
+    admin_ok(
+        &env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/add-canned-policy?name={policy}"),
+        Some(
+            serde_json::json!({
+                "Version": "2012-10-17",
+                "Statement": [{
+                    "Effect": "Allow",
+                    "Action": ["s3:*"],
+                    "Resource": [
+                        format!("arn:aws:s3:::{allowed_bucket}"),
+                        format!("arn:aws:s3:::{allowed_bucket}/*")
+                    ]
+                }]
+            })
+            .to_string(),
+        ),
+    )
+    .await?;
+    admin_ok(
+        &env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/add-user?accessKey={user}"),
+        Some(serde_json::json!({ "secretKey": secret, "status": "enabled" }).to_string()),
+    )
+    .await?;
+    admin_ok(
+        &env,
+        http::Method::POST,
+        "/rustfs/admin/v3/idp/builtin/policy/attach",
+        Some(serde_json::json!({ "policies": [policy], "user": user }).to_string()),
+    )
+    .await?;
+
+    let client = env.create_s3_client_with_credentials(user, secret);
+    // Capture ListBuckets first so the direct-access control cannot warm bucket metadata and mask the regression.
+    let listed = client.list_buckets().send().await;
+    client.list_objects_v2().bucket(allowed_bucket).send().await?;
+
+    let listed = listed?;
+    let names = listed
+        .buckets()
+        .iter()
+        .filter_map(|bucket| bucket.name().map(ToOwned::to_owned))
+        .collect::<Vec<_>>();
+    assert_eq!(names, vec![allowed_bucket]);
+
+    Ok(())
+}

crates/e2e_test/src/list_buckets_iam_filter_test.rs

@@ -0,0 +1,459 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::{RustFSTestEnvironment, admin_ok, build_test_s3_config, build_test_sts_client, init_logging};
+use aws_sdk_s3::Client;
+use aws_sdk_s3::error::ProvideErrorMetadata;
+use serial_test::serial;
+use tokio::time::{Duration, Instant};
+
+fn user_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str, session_token: Option<&str>) -> Client {
+    Client::from_conf(build_test_s3_config(
+        &env.url,
+        access_key,
+        secret_key,
+        session_token,
+        "list-buckets-iam-filter",
+    ))
+}
+
+fn bucket_names(buckets: &[aws_sdk_s3::types::Bucket]) -> Vec<String> {
+    let mut names = buckets
+        .iter()
+        .filter_map(|bucket| bucket.name().map(str::to_owned))
+        .collect::<Vec<_>>();
+    names.sort();
+    names
+}
+
+async fn create_user(
+    env: &RustFSTestEnvironment,
+    access_key: &str,
+    secret_key: &str,
+) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    let body = serde_json::json!({ "secretKey": secret_key, "status": "enabled" }).to_string();
+    admin_ok(
+        env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/add-user?accessKey={access_key}"),
+        Some(body),
+    )
+    .await?;
+    Ok(())
+}
+
+async fn create_service_account(
+    env: &RustFSTestEnvironment,
+    target_user: &str,
+    policy: Option<&serde_json::Value>,
+) -> Result<(String, String), Box<dyn std::error::Error + Send + Sync>> {
+    let request = match policy {
+        Some(policy) => serde_json::json!({ "targetUser": target_user, "policy": policy }),
+        None => serde_json::json!({ "targetUser": target_user }),
+    };
+    let response = admin_ok(env, http::Method::PUT, "/rustfs/admin/v3/add-service-accounts", Some(request.to_string())).await?;
+    let response: serde_json::Value = serde_json::from_str(&response)?;
+    let access_key = response["credentials"]["accessKey"]
+        .as_str()
+        .ok_or("service account response should contain credentials.accessKey")?
+        .to_owned();
+    let secret_key = response["credentials"]["secretKey"]
+        .as_str()
+        .ok_or("service account response should contain credentials.secretKey")?
+        .to_owned();
+    Ok((access_key, secret_key))
+}
+
+#[tokio::test]
+#[serial]
+async fn list_buckets_filters_with_iam_bucket_resources() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    init_logging();
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.capture_log_path = Some(format!("{}/server.log", env.temp_dir));
+    env.start_rustfs_server_with_env(vec![], &[("RUST_LOG", "rustfs=debug,rustfs_notify=debug")])
+        .await?;
+
+    let admin_client = env.create_s3_client();
+    for bucket in [
+        "benchmark-artifacts",
+        "benchmark-denied",
+        "benchmark-location-only",
+        "benchmark-test1",
+        "testuser1-artifacts",
+    ] {
+        admin_client.create_bucket().bucket(bucket).send().await?;
+    }
+    assert_eq!(
+        bucket_names(admin_client.list_buckets().send().await?.buckets()),
+        vec![
+            "benchmark-artifacts",
+            "benchmark-denied",
+            "benchmark-location-only",
+            "benchmark-test1",
+            "testuser1-artifacts"
+        ]
+    );
+
+    let access_key = "benchmark";
+    let secret_key = "benchmark-secret-1234567890";
+    create_user(&env, access_key, secret_key).await?;
+
+    let policy_name = "benchmark-bucket-prefix";
+    let policy = serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [
+            {
+                "Effect": "Allow",
+                "Action": ["s3:*"],
+                "Resource": ["arn:aws:s3:::benchmark-*", "arn:aws:s3:::benchmark-*/*"],
+                "Condition": {
+                    "StringEquals": {
+                        "s3:prefix": [""],
+                        "s3:delimiter": ["/"]
+                    }
+                }
+            },
+            {
+                "Effect": "Deny",
+                "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
+                "Resource": ["arn:aws:s3:::benchmark-denied"]
+            },
+            {
+                "Effect": "Deny",
+                "Action": ["s3:ListBucket"],
+                "Resource": ["arn:aws:s3:::benchmark-location-only"]
+            },
+            {
+                "Effect": "Allow",
+                "Action": ["sts:AssumeRole"],
+                "Resource": ["arn:aws:s3:::*"]
+            }
+        ]
+    })
+    .to_string();
+    admin_ok(
+        &env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/add-canned-policy?name={policy_name}"),
+        Some(policy),
+    )
+    .await?;
+    admin_ok(
+        &env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/set-user-or-group-policy?policyName={policy_name}&userOrGroup={access_key}&isGroup=false"),
+        Some(String::new()),
+    )
+    .await?;
+
+    let bucket_policy_allow = serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [{
+            "Effect": "Allow",
+            "Principal": { "AWS": [access_key] },
+            "Action": ["s3:ListBucket"],
+            "Resource": ["arn:aws:s3:::testuser1-artifacts"]
+        }]
+    })
+    .to_string();
+    admin_client
+        .put_bucket_policy()
+        .bucket("testuser1-artifacts")
+        .policy(bucket_policy_allow)
+        .send()
+        .await?;
+
+    let bucket_policy_deny = serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [{
+            "Effect": "Deny",
+            "Principal": { "AWS": [access_key] },
+            "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
+            "Resource": ["arn:aws:s3:::benchmark-artifacts"]
+        }]
+    })
+    .to_string();
+    admin_client
+        .put_bucket_policy()
+        .bucket("benchmark-artifacts")
+        .policy(bucket_policy_deny)
+        .send()
+        .await?;
+
+    let benchmark_client = user_client(&env, access_key, secret_key, None);
+    benchmark_client
+        .list_objects_v2()
+        .bucket("testuser1-artifacts")
+        .send()
+        .await?;
+
+    assert_eq!(
+        bucket_names(benchmark_client.list_buckets().send().await?.buckets()),
+        vec!["benchmark-artifacts", "benchmark-location-only", "benchmark-test1"]
+    );
+    let log_path = env.capture_log_path.as_deref().expect("server log path should be configured");
+    let deadline = Instant::now() + Duration::from_secs(5);
+    let audit_log = loop {
+        let audit_log = tokio::fs::read_to_string(log_path).await?;
+        if [
+            "iam_implicit_deny",
+            "s3_authorization_denied",
+            "ListAllMyBucketsAction",
+            "benchmark",
+            "DEBUG",
+        ]
+        .iter()
+        .all(|field| audit_log.contains(field))
+            || Instant::now() >= deadline
+        {
+            break audit_log;
+        }
+        tokio::time::sleep(Duration::from_millis(50)).await;
+    };
+    assert_eq!(audit_log.matches("iam_implicit_deny").count(), 1, "{audit_log}");
+    for field in ["s3_authorization_denied", "ListAllMyBucketsAction", "benchmark", "DEBUG"] {
+        assert!(audit_log.contains(field), "missing {field} in {audit_log}");
+    }
+
+    let denied_access_key = "no-bucket-access";
+    let denied_secret_key = "no-bucket-access-secret-1234567890";
+    create_user(&env, denied_access_key, denied_secret_key).await?;
+    let denied = user_client(&env, denied_access_key, denied_secret_key, None)
+        .list_buckets()
+        .send()
+        .await
+        .expect_err("a user without IAM bucket permissions must be denied");
+    assert_eq!(denied.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
+
+    let put_only_policy_name = "put-only-no-bucket-discovery";
+    let put_only_policy = serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [{
+            "Effect": "Allow",
+            "Action": ["s3:PutObject"],
+            "Resource": ["arn:aws:s3:::benchmark-*/*"]
+        }]
+    })
+    .to_string();
+    admin_ok(
+        &env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/add-canned-policy?name={put_only_policy_name}"),
+        Some(put_only_policy),
+    )
+    .await?;
+    admin_ok(
+        &env,
+        http::Method::PUT,
+        &format!(
+            "/rustfs/admin/v3/set-user-or-group-policy?policyName={put_only_policy_name}&userOrGroup={denied_access_key}&isGroup=false"
+        ),
+        Some(String::new()),
+    )
+    .await?;
+    let denied = user_client(&env, denied_access_key, denied_secret_key, None)
+        .list_buckets()
+        .send()
+        .await
+        .expect_err("an unrelated IAM action must not reveal bucket names");
+    assert_eq!(denied.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
+
+    let list_all_policy_name = "list-all-buckets";
+    let list_all_policy = serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [{
+            "Effect": "Allow",
+            "Action": ["s3:ListAllMyBuckets"],
+            "Resource": ["arn:aws:s3:::*"]
+        }]
+    })
+    .to_string();
+    admin_ok(
+        &env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/add-canned-policy?name={list_all_policy_name}"),
+        Some(list_all_policy),
+    )
+    .await?;
+    admin_ok(
+        &env,
+        http::Method::PUT,
+        &format!(
+            "/rustfs/admin/v3/set-user-or-group-policy?policyName={list_all_policy_name}&userOrGroup={denied_access_key}&isGroup=false"
+        ),
+        Some(String::new()),
+    )
+    .await?;
+    assert_eq!(
+        bucket_names(
+            user_client(&env, denied_access_key, denied_secret_key, None)
+                .list_buckets()
+                .send()
+                .await?
+                .buckets()
+        ),
+        vec![
+            "benchmark-artifacts",
+            "benchmark-denied",
+            "benchmark-location-only",
+            "benchmark-test1",
+            "testuser1-artifacts"
+        ]
+    );
+
+    let group_user = "benchmark-group-user";
+    let group_secret = "benchmark-group-secret-1234567890";
+    let group_name = "benchmark-group";
+    create_user(&env, group_user, group_secret).await?;
+    admin_ok(
+        &env,
+        http::Method::PUT,
+        "/rustfs/admin/v3/update-group-members",
+        Some(
+            serde_json::json!({
+                "group": group_name,
+                "members": [group_user],
+                "isRemove": false,
+                "groupStatus": "enabled"
+            })
+            .to_string(),
+        ),
+    )
+    .await?;
+    admin_ok(
+        &env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/set-user-or-group-policy?policyName={policy_name}&userOrGroup={group_name}&isGroup=true"),
+        Some(String::new()),
+    )
+    .await?;
+
+    assert_eq!(
+        bucket_names(
+            user_client(&env, group_user, group_secret, None)
+                .list_buckets()
+                .send()
+                .await?
+                .buckets()
+        ),
+        vec!["benchmark-artifacts", "benchmark-location-only", "benchmark-test1"]
+    );
+
+    let (service_access_key, service_secret_key) = create_service_account(&env, group_user, None).await?;
+    assert_eq!(
+        bucket_names(
+            user_client(&env, &service_access_key, &service_secret_key, None)
+                .list_buckets()
+                .send()
+                .await?
+                .buckets()
+        ),
+        vec!["benchmark-artifacts", "benchmark-location-only", "benchmark-test1"]
+    );
+
+    let service_account_policy = serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [{
+            "Effect": "Allow",
+            "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
+            "Resource": ["arn:aws:s3:::benchmark-test1"],
+            "Condition": {
+                "StringEquals": {
+                    "s3:prefix": [""],
+                    "s3:delimiter": ["/"]
+                }
+            }
+        }]
+    });
+    let (restricted_service_access_key, restricted_service_secret_key) =
+        create_service_account(&env, group_user, Some(&service_account_policy)).await?;
+    assert_eq!(
+        bucket_names(
+            user_client(&env, &restricted_service_access_key, &restricted_service_secret_key, None,)
+                .list_buckets()
+                .send()
+                .await?
+                .buckets()
+        ),
+        vec!["benchmark-test1"]
+    );
+
+    let sts_client = build_test_sts_client(&env.url, group_user, group_secret, None, "list-buckets-iam-filter-sts");
+    let inherited = sts_client
+        .assume_role()
+        .role_arn("arn:aws:iam::123456789012:role/list-buckets")
+        .role_session_name("list-buckets-iam-filter-inherited")
+        .send()
+        .await?;
+    let inherited = inherited
+        .credentials()
+        .ok_or("AssumeRole response should contain inherited temporary credentials")?;
+    assert_eq!(
+        bucket_names(
+            user_client(
+                &env,
+                inherited.access_key_id(),
+                inherited.secret_access_key(),
+                Some(inherited.session_token()),
+            )
+            .list_buckets()
+            .send()
+            .await?
+            .buckets()
+        ),
+        vec!["benchmark-artifacts", "benchmark-location-only", "benchmark-test1"]
+    );
+
+    let session_policy = serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [{
+            "Effect": "Allow",
+            "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
+            "Resource": ["arn:aws:s3:::benchmark-test1"],
+            "Condition": {
+                "StringEquals": {
+                    "s3:prefix": [""],
+                    "s3:delimiter": ["/"]
+                }
+            }
+        }]
+    })
+    .to_string();
+    let assumed = sts_client
+        .assume_role()
+        .role_arn("arn:aws:iam::123456789012:role/list-buckets")
+        .role_session_name("list-buckets-iam-filter")
+        .policy(session_policy)
+        .send()
+        .await?;
+    let temporary = assumed
+        .credentials()
+        .ok_or("AssumeRole response should contain temporary credentials")?;
+    assert_eq!(
+        bucket_names(
+            user_client(
+                &env,
+                temporary.access_key_id(),
+                temporary.secret_access_key(),
+                Some(temporary.session_token()),
+            )
+            .list_buckets()
+            .send()
+            .await?
+            .buckets()
+        ),
+        vec!["benchmark-test1"]
+    );
+    Ok(())
+}

crates/e2e_test/src/multipart_auth_test.rs

@@ -285,6 +285,198 @@ async fn allow_anonymous_put_object(
     Ok(())
 }
 
+/// One rejected POST Object upload driven end-to-end (backlog#1838): starts a
+/// fresh server, allows anonymous PutObject on `bucket`, posts an anonymous
+/// POST Object form whose policy carries `policy_conditions` and whose form
+/// carries `form_fields` on top of the mandatory key+policy fields, then
+/// asserts the expected status, error code, and lowercase-body mention.
+/// `case` prefixes every assertion message so a failing table row is
+/// identifiable at a glance.
+#[allow(clippy::too_many_arguments)]
+async fn run_post_object_policy_case(
+    bucket: &str,
+    object_key: &str,
+    policy_conditions: Vec<serde_json::Value>,
+    form_fields: &[(&str, &str)],
+    file_body: &[u8],
+    expected_status: reqwest::StatusCode,
+    expected_code: &str,
+    expected_mention: &str,
+    case: &str,
+) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+
+    let admin_client = env.create_s3_client();
+    admin_client.create_bucket().bucket(bucket).send().await?;
+    allow_anonymous_put_object(&admin_client, bucket).await?;
+
+    let policy = encode_post_policy(policy_conditions);
+
+    let mut post_form = reqwest::multipart::Form::new()
+        .text("key", object_key.to_string())
+        .text("policy", policy);
+    for (name, value) in form_fields {
+        post_form = post_form.text(name.to_string(), value.to_string());
+    }
+    let post_form = post_form.part(
+        "file",
+        reqwest::multipart::Part::bytes(file_body.to_vec())
+            .file_name("upload.txt")
+            .mime_str("text/plain")?,
+    );
+
+    let post_resp = local_http_client()
+        .post(format!("{}/{}", env.url, bucket))
+        .multipart(post_form)
+        .send()
+        .await?;
+
+    let status = post_resp.status();
+    let response_body = post_resp.text().await?;
+    let response_body_lower = response_body.to_ascii_lowercase();
+
+    assert_eq!(status, expected_status, "[{case}] unexpected status, body: {response_body}");
+    assert!(
+        response_body.contains(expected_code),
+        "[{case}] response should contain {expected_code}, got: {response_body}"
+    );
+    assert!(
+        response_body_lower.contains(expected_mention),
+        "[{case}] response should mention {expected_mention}, got: {response_body}"
+    );
+
+    Ok(())
+}
+
+/// Table-driven fold of the nine `*_missing_from_policy_conditions` POST
+/// Object tests (backlog#1838 PR1). Every row keeps its original test's exact
+/// bucket, key, form field, file body, and expected error strings; the shared
+/// shape is: policy pins bucket + key + content-length-range only, the form
+/// smuggles one extra field the policy never declared, and the upload must be
+/// rejected with 403 AccessDenied naming the offending field.
+#[tokio::test]
+#[serial]
+async fn test_anonymous_post_object_rejects_fields_missing_from_policy_conditions()
+-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    init_logging();
+
+    // (case, bucket, object_key, form field, file body, expected code, expected mention)
+    type Case = (
+        &'static str,
+        &'static str,
+        &'static str,
+        (&'static str, &'static str),
+        &'static [u8],
+        &'static str,
+        &'static str,
+    );
+    let cases: &[Case] = &[
+        (
+            "cache-control",
+            "anon-post-policy-cache-control-missing",
+            "uploads/cache-control-missing.txt",
+            ("Cache-Control", "max-age=60"),
+            b"post-policy-cache-control-missing",
+            "AccessDenied",
+            "cache-control",
+        ),
+        (
+            "content-language",
+            "anon-post-policy-content-language-missing",
+            "uploads/content-language-missing.txt",
+            ("Content-Language", "en-US"),
+            b"post-policy-content-language-missing",
+            "AccessDenied",
+            "content-language",
+        ),
+        (
+            "content-encoding",
+            "anon-post-policy-content-encoding-missing",
+            "uploads/content-encoding-missing.txt",
+            ("Content-Encoding", "gzip"),
+            b"post-policy-content-encoding-missing",
+            "AccessDenied",
+            "content-encoding",
+        ),
+        (
+            "website-redirect-location",
+            "anon-post-policy-website-redirect-missing",
+            "uploads/website-redirect-missing.txt",
+            ("x-amz-website-redirect-location", "/docs/landing.html"),
+            b"post-policy-website-redirect-missing",
+            "AccessDenied",
+            "x-amz-website-redirect-location",
+        ),
+        (
+            "expires",
+            "anon-post-policy-expires-missing",
+            "uploads/expires-missing-object.txt",
+            ("Expires", "Wed, 21 Oct 2037 07:28:00 GMT"),
+            b"post-policy-expires-missing",
+            "AccessDenied",
+            "expires",
+        ),
+        (
+            "tagging",
+            "anon-post-policy-tagging-missing",
+            "uploads/tagging-missing-object.txt",
+            ("x-amz-tagging", "project=alpha&env=test"),
+            b"post-policy-tagging-missing",
+            "AccessDenied",
+            "x-amz-tagging",
+        ),
+        (
+            "metadata",
+            "anon-post-policy-meta-reject",
+            "uploads/meta-reject-object.txt",
+            ("x-amz-meta-project", "alpha-demo"),
+            b"post-policy-body",
+            "<Code>AccessDenied</Code>",
+            "x-amz-meta-project",
+        ),
+        (
+            "metadata-new-key",
+            "anon-post-policy-meta-name-missing",
+            "uploads/meta-name-missing.txt",
+            ("x-amz-meta-name", "demo-name"),
+            b"post-policy-meta-name-missing",
+            "<Code>AccessDenied</Code>",
+            "x-amz-meta-name",
+        ),
+        (
+            "content-type",
+            "anon-post-policy-content-type-missing",
+            "uploads/content-type-missing.txt",
+            ("Content-Type", "text/plain"),
+            b"post-policy-content-type-missing",
+            "AccessDenied",
+            "content-type",
+        ),
+    ];
+
+    for (case, bucket, object_key, form_field, file_body, expected_code, expected_mention) in cases {
+        run_post_object_policy_case(
+            bucket,
+            object_key,
+            vec![
+                serde_json::json!({ "bucket": bucket }),
+                serde_json::json!({ "key": object_key }),
+                serde_json::json!(["content-length-range", 0, 1024]),
+            ],
+            &[*form_field],
+            file_body,
+            reqwest::StatusCode::FORBIDDEN,
+            expected_code,
+            expected_mention,
+            case,
+        )
+        .await?;
+    }
+
+    Ok(())
+}
+
 #[tokio::test]
 #[serial]
 async fn test_anonymous_multipart_control_apis_require_auth() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
@@ -2869,59 +3061,6 @@ async fn test_anonymous_post_object_rejects_cache_control_policy_mismatch() -> R
     Ok(())
 }
 
-#[tokio::test]
-#[serial]
-async fn test_anonymous_post_object_rejects_cache_control_missing_from_policy_conditions()
--> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let bucket = "anon-post-policy-cache-control-missing";
-    let object_key = "uploads/cache-control-missing.txt";
-
-    let admin_client = env.create_s3_client();
-    admin_client.create_bucket().bucket(bucket).send().await?;
-    allow_anonymous_put_object(&admin_client, bucket).await?;
-
-    let policy = encode_post_policy(vec![
-        serde_json::json!({ "bucket": bucket }),
-        serde_json::json!({ "key": object_key }),
-        serde_json::json!(["content-length-range", 0, 1024]),
-    ]);
-
-    let post_form = reqwest::multipart::Form::new()
-        .text("key", object_key.to_string())
-        .text("policy", policy)
-        .text("Cache-Control", "max-age=60")
-        .part(
-            "file",
-            reqwest::multipart::Part::bytes(b"post-policy-cache-control-missing".to_vec())
-                .file_name("upload.txt")
-                .mime_str("text/plain")?,
-        );
-
-    let post_resp = local_http_client()
-        .post(format!("{}/{}", env.url, bucket))
-        .multipart(post_form)
-        .send()
-        .await?;
-
-    let status = post_resp.status();
-    let response_body = post_resp.text().await?;
-    let response_body_lower = response_body.to_ascii_lowercase();
-
-    assert_eq!(status, reqwest::StatusCode::FORBIDDEN);
-    assert!(response_body.contains("AccessDenied"));
-    assert!(
-        response_body_lower.contains("cache-control"),
-        "response should mention cache-control, got: {response_body}"
-    );
-
-    Ok(())
-}
-
 #[tokio::test]
 #[serial]
 async fn test_anonymous_post_object_accepts_content_language_field_exact_policy_match()
@@ -3034,59 +3173,6 @@ async fn test_anonymous_post_object_rejects_content_language_policy_mismatch()
     Ok(())
 }
 
-#[tokio::test]
-#[serial]
-async fn test_anonymous_post_object_rejects_content_language_missing_from_policy_conditions()
--> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let bucket = "anon-post-policy-content-language-missing";
-    let object_key = "uploads/content-language-missing.txt";
-
-    let admin_client = env.create_s3_client();
-    admin_client.create_bucket().bucket(bucket).send().await?;
-    allow_anonymous_put_object(&admin_client, bucket).await?;
-
-    let policy = encode_post_policy(vec![
-        serde_json::json!({ "bucket": bucket }),
-        serde_json::json!({ "key": object_key }),
-        serde_json::json!(["content-length-range", 0, 1024]),
-    ]);
-
-    let post_form = reqwest::multipart::Form::new()
-        .text("key", object_key.to_string())
-        .text("policy", policy)
-        .text("Content-Language", "en-US")
-        .part(
-            "file",
-            reqwest::multipart::Part::bytes(b"post-policy-content-language-missing".to_vec())
-                .file_name("upload.txt")
-                .mime_str("text/plain")?,
-        );
-
-    let post_resp = local_http_client()
-        .post(format!("{}/{}", env.url, bucket))
-        .multipart(post_form)
-        .send()
-        .await?;
-
-    let status = post_resp.status();
-    let response_body = post_resp.text().await?;
-    let response_body_lower = response_body.to_ascii_lowercase();
-
-    assert_eq!(status, reqwest::StatusCode::FORBIDDEN);
-    assert!(response_body.contains("AccessDenied"));
-    assert!(
-        response_body_lower.contains("content-language"),
-        "response should mention content-language, got: {response_body}"
-    );
-
-    Ok(())
-}
-
 #[tokio::test]
 #[serial]
 async fn test_anonymous_post_object_accepts_content_encoding_field_exact_policy_match()
@@ -3199,59 +3285,6 @@ async fn test_anonymous_post_object_rejects_content_encoding_policy_mismatch()
     Ok(())
 }
 
-#[tokio::test]
-#[serial]
-async fn test_anonymous_post_object_rejects_content_encoding_missing_from_policy_conditions()
--> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let bucket = "anon-post-policy-content-encoding-missing";
-    let object_key = "uploads/content-encoding-missing.txt";
-
-    let admin_client = env.create_s3_client();
-    admin_client.create_bucket().bucket(bucket).send().await?;
-    allow_anonymous_put_object(&admin_client, bucket).await?;
-
-    let policy = encode_post_policy(vec![
-        serde_json::json!({ "bucket": bucket }),
-        serde_json::json!({ "key": object_key }),
-        serde_json::json!(["content-length-range", 0, 1024]),
-    ]);
-
-    let post_form = reqwest::multipart::Form::new()
-        .text("key", object_key.to_string())
-        .text("policy", policy)
-        .text("Content-Encoding", "gzip")
-        .part(
-            "file",
-            reqwest::multipart::Part::bytes(b"post-policy-content-encoding-missing".to_vec())
-                .file_name("upload.txt")
-                .mime_str("text/plain")?,
-        );
-
-    let post_resp = local_http_client()
-        .post(format!("{}/{}", env.url, bucket))
-        .multipart(post_form)
-        .send()
-        .await?;
-
-    let status = post_resp.status();
-    let response_body = post_resp.text().await?;
-    let response_body_lower = response_body.to_ascii_lowercase();
-
-    assert_eq!(status, reqwest::StatusCode::FORBIDDEN);
-    assert!(response_body.contains("AccessDenied"));
-    assert!(
-        response_body_lower.contains("content-encoding"),
-        "response should mention content-encoding, got: {response_body}"
-    );
-
-    Ok(())
-}
-
 #[tokio::test]
 #[serial]
 async fn test_anonymous_post_object_accepts_website_redirect_location_exact_policy_match()
@@ -3310,59 +3343,6 @@ async fn test_anonymous_post_object_accepts_website_redirect_location_exact_poli
     Ok(())
 }
 
-#[tokio::test]
-#[serial]
-async fn test_anonymous_post_object_rejects_website_redirect_location_missing_from_policy_conditions()
--> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let bucket = "anon-post-policy-website-redirect-missing";
-    let object_key = "uploads/website-redirect-missing.txt";
-
-    let admin_client = env.create_s3_client();
-    admin_client.create_bucket().bucket(bucket).send().await?;
-    allow_anonymous_put_object(&admin_client, bucket).await?;
-
-    let policy = encode_post_policy(vec![
-        serde_json::json!({ "bucket": bucket }),
-        serde_json::json!({ "key": object_key }),
-        serde_json::json!(["content-length-range", 0, 1024]),
-    ]);
-
-    let post_form = reqwest::multipart::Form::new()
-        .text("key", object_key.to_string())
-        .text("policy", policy)
-        .text("x-amz-website-redirect-location", "/docs/landing.html")
-        .part(
-            "file",
-            reqwest::multipart::Part::bytes(b"post-policy-website-redirect-missing".to_vec())
-                .file_name("upload.txt")
-                .mime_str("text/plain")?,
-        );
-
-    let post_resp = local_http_client()
-        .post(format!("{}/{}", env.url, bucket))
-        .multipart(post_form)
-        .send()
-        .await?;
-
-    let status = post_resp.status();
-    let response_body = post_resp.text().await?;
-    let response_body_lower = response_body.to_ascii_lowercase();
-
-    assert_eq!(status, reqwest::StatusCode::FORBIDDEN);
-    assert!(response_body.contains("AccessDenied"));
-    assert!(
-        response_body_lower.contains("x-amz-website-redirect-location"),
-        "response should mention x-amz-website-redirect-location, got: {response_body}"
-    );
-
-    Ok(())
-}
-
 #[tokio::test]
 #[serial]
 async fn test_anonymous_post_object_rejects_website_redirect_location_policy_mismatch()
@@ -3529,59 +3509,6 @@ async fn test_anonymous_post_object_rejects_expires_field_policy_mismatch() -> R
     Ok(())
 }
 
-#[tokio::test]
-#[serial]
-async fn test_anonymous_post_object_rejects_expires_field_missing_from_policy_conditions()
--> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let bucket = "anon-post-policy-expires-missing";
-    let object_key = "uploads/expires-missing-object.txt";
-
-    let admin_client = env.create_s3_client();
-    admin_client.create_bucket().bucket(bucket).send().await?;
-    allow_anonymous_put_object(&admin_client, bucket).await?;
-
-    let policy = encode_post_policy(vec![
-        serde_json::json!({ "bucket": bucket }),
-        serde_json::json!({ "key": object_key }),
-        serde_json::json!(["content-length-range", 0, 1024]),
-    ]);
-
-    let post_form = reqwest::multipart::Form::new()
-        .text("key", object_key.to_string())
-        .text("policy", policy)
-        .text("Expires", "Wed, 21 Oct 2037 07:28:00 GMT")
-        .part(
-            "file",
-            reqwest::multipart::Part::bytes(b"post-policy-expires-missing".to_vec())
-                .file_name("upload.txt")
-                .mime_str("text/plain")?,
-        );
-
-    let post_resp = local_http_client()
-        .post(format!("{}/{}", env.url, bucket))
-        .multipart(post_form)
-        .send()
-        .await?;
-
-    let status = post_resp.status();
-    let response_body = post_resp.text().await?;
-    let response_body_lower = response_body.to_ascii_lowercase();
-
-    assert_eq!(status, reqwest::StatusCode::FORBIDDEN);
-    assert!(response_body.contains("AccessDenied"));
-    assert!(
-        response_body_lower.contains("expires"),
-        "response should mention Expires, got: {response_body}"
-    );
-
-    Ok(())
-}
-
 #[tokio::test]
 #[serial]
 async fn test_anonymous_post_object_rejects_object_lock_retention_without_permission()
@@ -4110,115 +4037,6 @@ async fn test_anonymous_post_object_rejects_tagging_field_policy_mismatch() -> R
     Ok(())
 }
 
-#[tokio::test]
-#[serial]
-async fn test_anonymous_post_object_rejects_tagging_field_missing_from_policy_conditions()
--> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let bucket = "anon-post-policy-tagging-missing";
-    let object_key = "uploads/tagging-missing-object.txt";
-
-    let admin_client = env.create_s3_client();
-    admin_client.create_bucket().bucket(bucket).send().await?;
-    allow_anonymous_put_object(&admin_client, bucket).await?;
-
-    let policy = encode_post_policy(vec![
-        serde_json::json!({ "bucket": bucket }),
-        serde_json::json!({ "key": object_key }),
-        serde_json::json!(["content-length-range", 0, 1024]),
-    ]);
-
-    let post_form = reqwest::multipart::Form::new()
-        .text("key", object_key.to_string())
-        .text("policy", policy)
-        .text("x-amz-tagging", "project=alpha&env=test")
-        .part(
-            "file",
-            reqwest::multipart::Part::bytes(b"post-policy-tagging-missing".to_vec())
-                .file_name("upload.txt")
-                .mime_str("text/plain")?,
-        );
-
-    let post_resp = local_http_client()
-        .post(format!("{}/{}", env.url, bucket))
-        .multipart(post_form)
-        .send()
-        .await?;
-
-    let status = post_resp.status();
-    let response_body = post_resp.text().await?;
-    let response_body_lower = response_body.to_ascii_lowercase();
-
-    assert_eq!(status, reqwest::StatusCode::FORBIDDEN);
-    assert!(response_body.contains("AccessDenied"));
-    assert!(
-        response_body_lower.contains("x-amz-tagging"),
-        "response should mention x-amz-tagging, got: {response_body}"
-    );
-
-    Ok(())
-}
-
-#[tokio::test]
-#[serial]
-async fn test_anonymous_post_object_rejects_metadata_field_missing_from_policy_conditions()
--> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let bucket = "anon-post-policy-meta-reject";
-    let object_key = "uploads/meta-reject-object.txt";
-
-    let admin_client = env.create_s3_client();
-    admin_client.create_bucket().bucket(bucket).send().await?;
-    allow_anonymous_put_object(&admin_client, bucket).await?;
-
-    let policy = encode_post_policy(vec![
-        serde_json::json!({ "bucket": bucket }),
-        serde_json::json!({ "key": object_key }),
-        serde_json::json!(["content-length-range", 0, 1024]),
-    ]);
-
-    let post_form = reqwest::multipart::Form::new()
-        .text("key", object_key.to_string())
-        .text("policy", policy)
-        .text("x-amz-meta-project", "alpha-demo")
-        .part(
-            "file",
-            reqwest::multipart::Part::bytes(b"post-policy-body".to_vec())
-                .file_name("upload.txt")
-                .mime_str("text/plain")?,
-        );
-
-    let post_resp = local_http_client()
-        .post(format!("{}/{}", env.url, bucket))
-        .multipart(post_form)
-        .send()
-        .await?;
-
-    let status = post_resp.status();
-    let response_body = post_resp.text().await?;
-    let response_body_lower = response_body.to_ascii_lowercase();
-
-    assert_eq!(status, reqwest::StatusCode::FORBIDDEN);
-    assert!(
-        response_body.contains("<Code>AccessDenied</Code>"),
-        "response should contain AccessDenied code, got: {response_body}"
-    );
-    assert!(
-        response_body_lower.contains("x-amz-meta-project"),
-        "response should mention the missing metadata field, got: {response_body}"
-    );
-
-    Ok(())
-}
-
 #[tokio::test]
 #[serial]
 async fn test_anonymous_post_object_rejects_metadata_field_exact_policy_mismatch()
@@ -4388,59 +4206,6 @@ async fn test_anonymous_post_object_allows_x_ignore_fields_outside_policy_condit
     Ok(())
 }
 
-#[tokio::test]
-#[serial]
-async fn test_anonymous_post_object_rejects_metadata_field_missing_from_policy_conditions_for_new_key()
--> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let bucket = "anon-post-policy-meta-name-missing";
-    let object_key = "uploads/meta-name-missing.txt";
-
-    let admin_client = env.create_s3_client();
-    admin_client.create_bucket().bucket(bucket).send().await?;
-    allow_anonymous_put_object(&admin_client, bucket).await?;
-
-    let policy = encode_post_policy(vec![
-        serde_json::json!({ "bucket": bucket }),
-        serde_json::json!({ "key": object_key }),
-        serde_json::json!(["content-length-range", 0, 1024]),
-    ]);
-
-    let post_form = reqwest::multipart::Form::new()
-        .text("key", object_key.to_string())
-        .text("policy", policy)
-        .text("x-amz-meta-name", "demo-name")
-        .part(
-            "file",
-            reqwest::multipart::Part::bytes(b"post-policy-meta-name-missing".to_vec())
-                .file_name("upload.txt")
-                .mime_str("text/plain")?,
-        );
-
-    let post_resp = local_http_client()
-        .post(format!("{}/{}", env.url, bucket))
-        .multipart(post_form)
-        .send()
-        .await?;
-
-    let status = post_resp.status();
-    let response_body = post_resp.text().await?;
-    let response_body_lower = response_body.to_ascii_lowercase();
-
-    assert_eq!(status, reqwest::StatusCode::FORBIDDEN);
-    assert!(response_body.contains("<Code>AccessDenied</Code>"));
-    assert!(
-        response_body_lower.contains("x-amz-meta-name"),
-        "response should mention x-amz-meta-name, got: {response_body}"
-    );
-
-    Ok(())
-}
-
 #[tokio::test]
 #[serial]
 async fn test_anonymous_post_object_rejects_metadata_uuid_exact_policy_mismatch()
@@ -4876,59 +4641,6 @@ async fn test_anonymous_post_object_rejects_content_type_policy_mismatch() -> Re
     Ok(())
 }
 
-#[tokio::test]
-#[serial]
-async fn test_anonymous_post_object_rejects_content_type_missing_from_policy_conditions()
--> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let bucket = "anon-post-policy-content-type-missing";
-    let object_key = "uploads/content-type-missing.txt";
-
-    let admin_client = env.create_s3_client();
-    admin_client.create_bucket().bucket(bucket).send().await?;
-    allow_anonymous_put_object(&admin_client, bucket).await?;
-
-    let policy = encode_post_policy(vec![
-        serde_json::json!({ "bucket": bucket }),
-        serde_json::json!({ "key": object_key }),
-        serde_json::json!(["content-length-range", 0, 1024]),
-    ]);
-
-    let post_form = reqwest::multipart::Form::new()
-        .text("key", object_key.to_string())
-        .text("policy", policy)
-        .text("Content-Type", "text/plain")
-        .part(
-            "file",
-            reqwest::multipart::Part::bytes(b"post-policy-content-type-missing".to_vec())
-                .file_name("upload.txt")
-                .mime_str("text/plain")?,
-        );
-
-    let post_resp = local_http_client()
-        .post(format!("{}/{}", env.url, bucket))
-        .multipart(post_form)
-        .send()
-        .await?;
-
-    let status = post_resp.status();
-    let response_body = post_resp.text().await?;
-    let response_body_lower = response_body.to_ascii_lowercase();
-
-    assert_eq!(status, reqwest::StatusCode::FORBIDDEN);
-    assert!(response_body.contains("AccessDenied"));
-    assert!(
-        response_body_lower.contains("content-type"),
-        "response should mention content-type, got: {response_body}"
-    );
-
-    Ok(())
-}
-
 #[tokio::test]
 #[serial]
 async fn test_signed_put_object_extract_expands_tar_entries_with_prefix_headers()
@@ -6028,6 +5740,33 @@ async fn test_signed_put_object_extract_authorizes_each_pax_privilege_and_retent
     }
 
     let version_condition_client = restricted_user_client(&env, version_condition_user, version_condition_secret);
+    let mismatching_version_pax = HashMap::from([("minio.versionId", Uuid::new_v4().to_string())]);
+    let archive = make_tar_with_pax_entry("version-mismatch-entry.txt", b"must-not-write", None, &mismatching_version_pax).await;
+    let err = version_condition_client
+        .put_object()
+        .bucket(bucket)
+        .key("version-mismatch.tar")
+        .body(ByteStream::from(archive))
+        .customize()
+        .mutate_request(|req| {
+            req.headers_mut().insert("x-amz-meta-snowball-auto-extract", "true");
+        })
+        .send()
+        .await
+        .expect_err("a mismatching PAX version ID must fail the replication condition");
+    assert_eq!(err.as_service_error().and_then(|error| error.meta().code()), Some("AccessDenied"));
+    let err = admin_client
+        .head_object()
+        .bucket(bucket)
+        .key("version-mismatch-entry.txt")
+        .send()
+        .await
+        .expect_err("a denied PAX entry must not be written");
+    assert!(matches!(
+        err.as_service_error().and_then(|error| error.meta().code()),
+        Some("NoSuchKey" | "NotFound")
+    ));
+
     let matching_version_pax = HashMap::from([("minio.versionId", conditional_version_id)]);
     let archive = make_tar_with_pax_entry("condition-entry.txt", b"condition-body", None, &matching_version_pax).await;
     version_condition_client
@@ -6041,6 +5780,13 @@ async fn test_signed_put_object_extract_authorizes_each_pax_privilege_and_retent
         })
         .send()
         .await?;
+    let stored = admin_client
+        .get_object()
+        .bucket(bucket)
+        .key("condition-entry.txt")
+        .send()
+        .await?;
+    assert_eq!(stored.body.collect().await?.into_bytes().as_ref(), b"condition-body");
 
     let pax_context_client = restricted_user_client(&env, pax_context_user, pax_context_secret);
     let tag_pax = HashMap::from([("minio.metadata.x-amz-tagging", "classification=public".to_string())]);

crates/e2e_test/src/quota_test.rs

@@ -12,9 +12,11 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::common::{RustFSTestEnvironment, awscurl_delete, awscurl_get, awscurl_post, awscurl_put, init_logging};
+use crate::common::{RustFSTestEnvironment, admin_request, awscurl_delete, awscurl_get, awscurl_post, awscurl_put, init_logging};
 use aws_sdk_s3::Client;
+use http::{Method, StatusCode};
 use serial_test::serial;
+use tokio::time::{Duration, sleep, timeout};
 use tracing::{debug, info};
 
 fn skip_without_awscurl() -> bool {
@@ -37,7 +39,8 @@ impl QuotaTestEnv {
     pub async fn new() -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
         let bucket_name = format!("quota-test-{}", uuid::Uuid::new_v4());
         let mut env = RustFSTestEnvironment::new().await?;
-        env.start_rustfs_server(vec![]).await?;
+        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_SCANNER_START_DELAY_SECS", "0")])
+            .await?;
         let client = env.create_s3_client();
 
         Ok(Self {
@@ -67,18 +70,7 @@ impl QuotaTestEnv {
     }
 
     pub async fn set_bucket_quota(&self, quota_bytes: u64) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        let url = format!("{}/rustfs/admin/v3/quota/{}", self.env.url, self.bucket_name);
-        let quota_config = serde_json::json!({
-            "quota": quota_bytes,
-            "quota_type": "HARD"
-        });
-
-        let response = awscurl_put(&url, &quota_config.to_string(), &self.env.access_key, &self.env.secret_key).await?;
-        if response.contains("error") {
-            Err(format!("Failed to set quota: {}", response).into())
-        } else {
-            Ok(())
-        }
+        self.set_bucket_quota_for(&self.bucket_name, quota_bytes).await
     }
 
     pub async fn get_bucket_quota(&self) -> Result<Option<u64>, Box<dyn std::error::Error + Send + Sync>> {
@@ -178,6 +170,29 @@ impl QuotaTestEnv {
         bucket: &str,
         quota_bytes: u64,
     ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+        let stats_path = format!("/rustfs/admin/v3/quota-stats/{bucket}");
+        let readiness = async {
+            loop {
+                let (status, response) =
+                    admin_request(&self.env.url, Method::GET, &stats_path, None, &self.env.access_key, &self.env.secret_key)
+                        .await?;
+                if status.is_success() {
+                    return Ok::<(), Box<dyn std::error::Error + Send + Sync>>(());
+                }
+                if status != StatusCode::SERVICE_UNAVAILABLE {
+                    return Err(format!("quota usage readiness failed for {bucket}: {status} {response}").into());
+                }
+
+                sleep(Duration::from_secs(1)).await;
+            }
+        };
+        match timeout(Duration::from_secs(30), readiness).await {
+            Ok(result) => result?,
+            Err(_) => {
+                return Err(format!("quota usage did not become authoritative for {bucket} within 30 seconds").into());
+            }
+        }
+
         let url = format!("{}/rustfs/admin/v3/quota/{}", self.env.url, bucket);
         let quota_config = serde_json::json!({
             "quota": quota_bytes,
@@ -237,6 +252,7 @@ impl QuotaTestEnv {
 #[cfg(test)]
 mod integration_tests {
     use super::*;
+    use aws_sdk_s3::error::ProvideErrorMetadata;
 
     #[tokio::test]
     #[serial]
@@ -948,9 +964,27 @@ mod integration_tests {
             .send()
             .await;
 
-        assert!(complete_result.is_err());
+        let complete_error = complete_result.expect_err("multipart completion above quota must be rejected");
+        assert_eq!(complete_error.as_service_error().and_then(|error| error.code()), Some("InvalidRequest"));
         assert!(!env.object_exists("over_quota.txt").await?);
 
+        let staged_parts = env
+            .client
+            .list_parts()
+            .bucket(&env.bucket_name)
+            .key("over_quota.txt")
+            .upload_id(upload_id2)
+            .send()
+            .await?;
+        assert_eq!(staged_parts.parts().len(), 2, "quota rejection must preserve the multipart upload");
+        env.client
+            .abort_multipart_upload()
+            .bucket(&env.bucket_name)
+            .key("over_quota.txt")
+            .upload_id(upload_id2)
+            .send()
+            .await?;
+
         env.cleanup_bucket().await?;
 
         Ok(())

crates/e2e_test/src/reliability_disk_fault_test.rs

@@ -22,15 +22,16 @@
 
 #[cfg(test)]
 mod tests {
-    use crate::chaos::{DiskFaultHarness, signed_admin_post};
+    use crate::chaos::{DiskFaultHarness, VersionShardCensus, signed_admin_post};
     use crate::common::init_logging;
     use aws_sdk_s3::Client;
     use aws_sdk_s3::primitives::ByteStream;
-    use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
+    use aws_sdk_s3::types::{BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, VersioningConfiguration};
     use serial_test::serial;
     use sha2::{Digest, Sha256};
     use std::collections::HashSet;
-    use tokio::time::{Duration, sleep, timeout};
+    use std::error::Error;
+    use tokio::time::{Duration, Instant, interval, timeout};
     use tracing::info;
 
     const GET_TIMEOUT: Duration = Duration::from_secs(60);
@@ -271,12 +272,17 @@ mod tests {
         put_and_record(&client, bucket, "heal/nested/large.bin", payload(2 * 1024 * 1024, 34), &mut manifest).await?;
 
         verify_manifest(&client, bucket, &manifest, "baseline before disk replacement").await?;
-        for (key, _) in &manifest {
-            assert!(
-                harness.object_metadata_exists_on_disk(0, bucket, key),
-                "disk0 should hold xl.meta for {key} before replacement"
-            );
-        }
+        let manifest_keys = manifest.iter().map(|(key, _)| key.clone()).collect::<Vec<_>>();
+        let target_manifest: Vec<(String, VersionShardCensus)> = manifest_keys
+            .iter()
+            .map(|key| {
+                let census = harness.census_object_version(0, bucket, key, None)?;
+                if !census.is_complete() {
+                    return Err(format!("disk 0 has incomplete physical census for {key}: {census:?}").into());
+                }
+                Ok((key.clone(), census))
+            })
+            .collect::<Result<_, Box<dyn Error + Send + Sync>>>()?;
 
         harness.kill_server();
         harness.replace_disk_with_empty(0)?;
@@ -287,21 +293,179 @@ mod tests {
         signed_admin_post(&heal_url, Some(heal_body), &harness.env.access_key, &harness.env.secret_key).await?;
 
         let client = harness.env.create_s3_client();
-        let mut remaining: HashSet<String> = manifest.iter().map(|(key, _)| key.clone()).collect();
+        let mut remaining: HashSet<String> = manifest_keys.iter().cloned().collect();
         let heal_timeout_secs = std::env::var("RUSTFS_RELIABILITY_HEAL_TIMEOUT_SECS")
             .ok()
             .and_then(|value| value.parse::<u64>().ok())
             .unwrap_or(120);
+        let deadline = Instant::now() + Duration::from_secs(heal_timeout_secs);
+        let mut retry = interval(Duration::from_secs(1));
 
-        for _ in 0..heal_timeout_secs {
-            remaining.retain(|key| !harness.object_metadata_exists_on_disk(0, bucket, key));
+        loop {
+            remaining.retain(|key| {
+                let expected = target_manifest
+                    .iter()
+                    .find(|(manifest_key, _)| manifest_key == key)
+                    .map(|(_, manifest)| manifest)
+                    .expect("every key has a physical manifest");
+                harness
+                    .census_object_version(0, bucket, key, None)
+                    .map(|census| !census.matches_manifest(expected))
+                    .unwrap_or(true)
+            });
             if remaining.is_empty() {
                 verify_manifest(&client, bucket, &manifest, "after fresh-disk heal completed").await?;
                 return Ok(());
             }
-            sleep(Duration::from_secs(1)).await;
+            if Instant::now() >= deadline {
+                break;
+            }
+            retry.tick().await;
         }
 
         Err(format!("fresh-disk heal did not rebuild {remaining:?} on the replaced disk within {heal_timeout_secs}s").into())
     }
+
+    #[tokio::test]
+    #[serial]
+    async fn test_versioned_shard_census_selects_each_version_data_dir() -> Result<(), Box<dyn Error + Send + Sync>> {
+        init_logging();
+        info!("Reliability: physical shard census selects the requested object version");
+
+        let mut harness = DiskFaultHarness::new(4).await?;
+        harness.start_server().await?;
+        let client = harness.env.create_s3_client();
+        let bucket = "reliability-versioned-census";
+        let key = "versions/large.bin";
+        client.create_bucket().bucket(bucket).send().await?;
+        client
+            .put_bucket_versioning()
+            .bucket(bucket)
+            .versioning_configuration(
+                VersioningConfiguration::builder()
+                    .status(BucketVersioningStatus::Enabled)
+                    .build(),
+            )
+            .send()
+            .await?;
+
+        let first_inline = client
+            .put_object()
+            .bucket(bucket)
+            .key("versions/inline.bin")
+            .body(ByteStream::from(payload(8 * 1024, 40)))
+            .send()
+            .await?;
+        let first_inline_version = first_inline
+            .version_id()
+            .ok_or("first inline PUT did not return a version ID")?;
+        let second_inline = client
+            .put_object()
+            .bucket(bucket)
+            .key("versions/inline.bin")
+            .body(ByteStream::from(payload(8 * 1024, 41)))
+            .send()
+            .await?;
+        let second_inline_version = second_inline
+            .version_id()
+            .ok_or("second inline PUT did not return a version ID")?;
+
+        let first = client
+            .put_object()
+            .bucket(bucket)
+            .key(key)
+            .body(ByteStream::from(payload(128 * 1024, 41)))
+            .send()
+            .await?;
+        let first_version = first.version_id().ok_or("first PUT did not return a version ID")?;
+        let second = client
+            .put_object()
+            .bucket(bucket)
+            .key(key)
+            .body(ByteStream::from(payload(3 * 1024 * 1024, 42)))
+            .send()
+            .await?;
+        let second_version = second.version_id().ok_or("second PUT did not return a version ID")?;
+        let delete = client.delete_object().bucket(bucket).key(key).send().await?;
+        let delete_version = delete.version_id().ok_or("delete marker did not return a version ID")?;
+
+        let first_inline_census = harness.census_object_version(0, bucket, "versions/inline.bin", Some(first_inline_version))?;
+        let second_inline_census =
+            harness.census_object_version(0, bucket, "versions/inline.bin", Some(second_inline_version))?;
+        let first_census = harness.census_object_version(0, bucket, key, Some(first_version))?;
+        let first_other_disk_census = harness.census_object_version(1, bucket, key, Some(first_version))?;
+        let second_census = harness.census_object_version(0, bucket, key, Some(second_version))?;
+        let delete_census = harness.census_object_version(0, bucket, key, Some(delete_version))?;
+        assert!(
+            first_inline_census.is_complete() && second_inline_census.is_complete(),
+            "inline version physical census is incomplete: first={first_inline_census:?} second={second_inline_census:?}"
+        );
+        assert!(
+            first_inline_census.present_part_fingerprints.is_empty() && second_inline_census.present_part_fingerprints.is_empty(),
+            "inline versions must not select external shard files: first={first_inline_census:?} second={second_inline_census:?}"
+        );
+        assert!(
+            first_inline_census.inline_data_fingerprint.is_some() && second_inline_census.inline_data_fingerprint.is_some(),
+            "inline versions must fingerprint payload bytes stored in xl.meta"
+        );
+        assert_ne!(
+            first_inline_census.inline_data_fingerprint, second_inline_census.inline_data_fingerprint,
+            "same-size inline versions with different payloads must retain distinct xl.meta fingerprints"
+        );
+        assert!(
+            first_census.is_complete(),
+            "first version physical census is incomplete: {first_census:?}"
+        );
+        assert!(
+            second_census.is_complete(),
+            "second version physical census is incomplete: {second_census:?}"
+        );
+        assert!(
+            first_other_disk_census.is_complete(),
+            "first version physical census on the second disk is incomplete: {first_other_disk_census:?}"
+        );
+        assert_ne!(
+            first_census.erasure_index, first_other_disk_census.erasure_index,
+            "physical census must preserve each disk's erasure index"
+        );
+        assert_ne!(
+            first_census.data_dir, second_census.data_dir,
+            "distinct object versions must select distinct physical data directories"
+        );
+        assert_eq!(
+            first_census.expected_part_numbers, second_census.expected_part_numbers,
+            "same single-part shape should expose the same part numbers"
+        );
+        let first_part = first_census
+            .present_part_fingerprints
+            .values()
+            .next()
+            .ok_or("first version did not expose a physical part fingerprint")?;
+        let second_part = second_census
+            .present_part_fingerprints
+            .values()
+            .next()
+            .ok_or("second version did not expose a physical part fingerprint")?;
+        assert_ne!(
+            first_part.size, second_part.size,
+            "different shard lengths must retain their physical sizes"
+        );
+        assert_ne!(
+            first_part.sha256, second_part.sha256,
+            "different shard contents must retain their physical hashes"
+        );
+        assert!(
+            delete_census.is_complete(),
+            "delete marker physical census is incomplete: {delete_census:?}"
+        );
+        assert!(
+            delete_census.expected_part_numbers.is_empty(),
+            "delete marker must not declare object shards: {delete_census:?}"
+        );
+        assert!(
+            delete_census.present_part_fingerprints.is_empty(),
+            "delete marker must not select stale object shards: {delete_census:?}"
+        );
+        Ok(())
+    }
 }

crates/e2e_test/src/reliant/grpc_lock_server.rs

@@ -848,6 +848,13 @@ impl NodeService for MinimalLockNodeService {
         Err(Status::unimplemented("lock-only test server"))
     }
 
+    async fn replacement_recovery_status(
+        &self,
+        _request: Request<rustfs_protos::proto_gen::node_service::ReplacementRecoveryStatusRequest>,
+    ) -> Result<Response<rustfs_protos::proto_gen::node_service::ReplacementRecoveryStatusResponse>, Status> {
+        Err(Status::unimplemented("lock-only test server"))
+    }
+
     async fn get_metacache_listing(
         &self,
         _request: Request<rustfs_protos::proto_gen::node_service::GetMetacacheListingRequest>,

crates/e2e_test/src/sts_query_compat_test.rs

@@ -12,13 +12,10 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::common::{RustFSTestEnvironment, admin_ok, init_logging};
-use aws_sdk_sts::config::retry::RetryConfig;
-use aws_sdk_sts::config::{Credentials, Region};
+use crate::common::{RustFSTestEnvironment, admin_ok, build_test_s3_config, build_test_sts_client, init_logging};
+use aws_sdk_sts::Client;
 use aws_sdk_sts::error::ProvideErrorMetadata;
 use aws_sdk_sts::operation::RequestId;
-use aws_sdk_sts::{Client, Config};
-use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
 use bytes::Bytes;
 use http::header::{AUTHORIZATION, CONTENT_TYPE};
 use http::{Request, Response};
@@ -32,9 +29,8 @@ use serial_test::serial;
 use std::collections::BTreeSet;
 use std::convert::Infallible;
 use std::error::Error;
-use std::sync::Arc;
 use tokio::net::TcpListener;
-use tokio::sync::{Notify, mpsc};
+use tokio::sync::mpsc;
 use tokio::task::{JoinHandle, JoinSet};
 use tokio::time::{Duration, timeout};
 
@@ -43,22 +39,7 @@ type TestResult = Result<(), BoxError>;
 const OPA_AUTH_TOKEN: &str = "sts-opa-token";
 
 fn sts_client(url: &str, access_key: &str, secret_key: &str, session_token: Option<&str>) -> Client {
-    let mut config = Config::builder()
-        .credentials_provider(Credentials::new(
-            access_key,
-            secret_key,
-            session_token.map(str::to_owned),
-            None,
-            "e2e-sts-query-compat",
-        ))
-        .region(Region::new("us-east-1"))
-        .endpoint_url(url)
-        .retry_config(RetryConfig::standard().with_max_attempts(1))
-        .behavior_version_latest();
-    if url.starts_with("http://") {
-        config = config.http_client(SmithyHttpClientBuilder::new().build_http());
-    }
-    Client::from_conf(config.build())
+    build_test_sts_client(url, access_key, secret_key, session_token, "e2e-sts-query-compat")
 }
 
 async fn create_root_service_account(env: &RustFSTestEnvironment) -> Result<(String, String), BoxError> {
@@ -145,6 +126,52 @@ async fn assert_access_denied(client: &Client, context: &str) -> TestResult {
     Ok(())
 }
 
+async fn assert_list_buckets_access_denied(
+    env: &RustFSTestEnvironment,
+    access_key: &str,
+    secret_key: &str,
+    context: &str,
+) -> TestResult {
+    let error = aws_sdk_s3::Client::from_conf(build_test_s3_config(
+        &env.url,
+        access_key,
+        secret_key,
+        None,
+        "e2e-list-buckets-opa-unavailable",
+    ))
+    .list_buckets()
+    .send()
+    .await
+    .expect_err("ListBuckets must be denied while OPA is unavailable");
+    let service_error = error
+        .as_service_error()
+        .ok_or_else(|| format!("{context} should deserialize as an S3 service error: {error:?}"))?;
+
+    assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(403));
+    assert_eq!(service_error.code(), Some("AccessDenied"));
+    Ok(())
+}
+
+async fn assert_opa_unavailable_denies_sts_and_list_buckets(env: &RustFSTestEnvironment, context: &str) -> TestResult {
+    let user = "opaunavailable";
+    let secret = "stsOpaUnavailableSecret123";
+    create_user_with_policy(
+        env,
+        user,
+        secret,
+        "sts-opa-unavailable-local-policy",
+        serde_json::json!([{
+            "Effect": "Allow",
+            "Action": ["s3:ListAllMyBuckets"],
+            "Resource": ["arn:aws:s3:::*"],
+        }]),
+    )
+    .await?;
+
+    assert_access_denied(&sts_client(&env.url, user, secret, None), context).await?;
+    assert_list_buckets_access_denied(env, user, secret, context).await
+}
+
 async fn handle_opa_request(
     request: Request<Incoming>,
     requests: mpsc::UnboundedSender<Value>,
@@ -186,12 +213,15 @@ async fn handle_opa_request(
     };
     if payload.is_none() {
         let _ = validation_started.send(());
-        if let OpaValidationMode::DelayedUnavailable(release) = validation_mode {
-            release.notified().await;
-            return Ok(Response::builder()
-                .status(503)
-                .body(Full::new(Bytes::new()))
-                .expect("static OPA unavailable response must be valid"));
+        match validation_mode {
+            OpaValidationMode::Blocked => std::future::pending::<()>().await,
+            OpaValidationMode::Unavailable => {
+                return Ok(Response::builder()
+                    .status(503)
+                    .body(Full::new(Bytes::new()))
+                    .expect("static OPA unavailable response must be valid"));
+            }
+            OpaValidationMode::Ready => {}
         }
     }
     let allow = match payload.as_ref().and_then(|value| value.pointer("/input/identity/account")) {
@@ -201,6 +231,25 @@ async fn handle_opa_request(
             .and_then(Value::as_bool)
             .unwrap_or(false),
         Some(Value::String(account)) if account == "opadeny" => false,
+        Some(Value::String(account))
+            if account == "opaunavailable" && matches!(validation_mode, OpaValidationMode::Unavailable) =>
+        {
+            true
+        }
+        Some(Value::String(account)) if account == "opalistbuckets" => {
+            let action = payload
+                .as_ref()
+                .and_then(|value| value.pointer("/input/action"))
+                .and_then(Value::as_str);
+            let bucket = payload
+                .as_ref()
+                .and_then(|value| value.pointer("/input/resource/bucket"))
+                .and_then(Value::as_str);
+            matches!(
+                (action, bucket),
+                (Some("s3:ListBucket"), Some("opa-list-visible")) | (Some("s3:GetBucketLocation"), Some("opa-list-location"))
+            )
+        }
         None => true,
         _ => false,
     };
@@ -215,17 +264,17 @@ async fn handle_opa_request(
         .expect("static OPA response must be valid"))
 }
 
-#[derive(Clone)]
+#[derive(Clone, Copy)]
 enum OpaValidationMode {
     Ready,
-    DelayedUnavailable(Arc<Notify>),
+    Blocked,
+    Unavailable,
 }
 
 struct OpaMock {
     url: String,
     requests: mpsc::UnboundedReceiver<Value>,
     validation_started: mpsc::UnboundedReceiver<()>,
-    validation_release: Option<Arc<Notify>>,
     task: JoinHandle<()>,
 }
 
@@ -234,9 +283,8 @@ impl OpaMock {
         Self::start_with_mode(OpaValidationMode::Ready, Some(OPA_AUTH_TOKEN)).await
     }
 
-    async fn start_delayed_unavailable() -> Result<Self, BoxError> {
-        let release = Arc::new(Notify::new());
-        Self::start_with_mode(OpaValidationMode::DelayedUnavailable(release), None).await
+    async fn start_blocked() -> Result<Self, BoxError> {
+        Self::start_with_mode(OpaValidationMode::Blocked, None).await
     }
 
     async fn start_with_mode(validation_mode: OpaValidationMode, auth_token: Option<&str>) -> Result<Self, BoxError> {
@@ -245,10 +293,6 @@ impl OpaMock {
         let (requests_tx, requests) = mpsc::unbounded_channel();
         let (validation_started_tx, validation_started) = mpsc::unbounded_channel();
         let expected_authorization = auth_token.map(|token| format!("Bearer {token}"));
-        let validation_release = match &validation_mode {
-            OpaValidationMode::Ready => None,
-            OpaValidationMode::DelayedUnavailable(release) => Some(Arc::clone(release)),
-        };
         let task = tokio::spawn(async move {
             let mut connections = JoinSet::new();
             loop {
@@ -257,7 +301,7 @@ impl OpaMock {
                         let Ok((stream, _)) = accepted else { break };
                         let requests = requests_tx.clone();
                         let validation_started = validation_started_tx.clone();
-                        let validation_mode = validation_mode.clone();
+                        let validation_mode = validation_mode;
                         let expected_authorization = expected_authorization.clone();
                         connections.spawn(async move {
                             let handler = service_fn(move |request| {
@@ -265,7 +309,7 @@ impl OpaMock {
                                     request,
                                     requests.clone(),
                                     validation_started.clone(),
-                                    validation_mode.clone(),
+                                    validation_mode,
                                     expected_authorization.clone(),
                                 )
                             });
@@ -282,7 +326,6 @@ impl OpaMock {
             url,
             requests,
             validation_started,
-            validation_release,
             task,
         })
     }
@@ -298,12 +341,6 @@ impl OpaMock {
             .await?
             .ok_or_else(|| "OPA validation channel closed".into())
     }
-
-    fn release_validation(&self) {
-        if let Some(release) = &self.validation_release {
-            release.notify_one();
-        }
-    }
 }
 
 impl Drop for OpaMock {
@@ -523,35 +560,119 @@ async fn test_sts_assume_role_opa_contract() -> TestResult {
 
 #[tokio::test]
 #[serial]
-async fn test_sts_assume_role_fails_closed_while_opa_is_unavailable() -> TestResult {
+async fn test_list_buckets_opa_contract() -> TestResult {
     init_logging();
 
-    let mut opa = OpaMock::start_delayed_unavailable().await?;
+    let mut opa = OpaMock::start().await?;
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_with_env(
+        vec![],
+        &[
+            ("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str()),
+            ("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", OPA_AUTH_TOKEN),
+        ],
+    )
+    .await?;
+
+    let admin_client = env.create_s3_client();
+    for bucket in ["opa-list-hidden", "opa-list-location", "opa-list-visible"] {
+        admin_client.create_bucket().bucket(bucket).send().await?;
+    }
+
+    let user = "opalistbuckets";
+    let secret = "opaListBucketsSecret123";
+    create_user(&env, user, secret).await?;
+
+    let output = aws_sdk_s3::Client::from_conf(build_test_s3_config(&env.url, user, secret, None, "e2e-list-buckets-opa"))
+        .list_buckets()
+        .send()
+        .await?;
+    let mut names = output
+        .buckets()
+        .iter()
+        .filter_map(|bucket| bucket.name().map(str::to_owned))
+        .collect::<Vec<_>>();
+    names.sort();
+    assert_eq!(names, ["opa-list-location", "opa-list-visible"]);
+
+    let mut evaluations = BTreeSet::new();
+    for _ in 0..6 {
+        let request = opa.next_request().await?;
+        assert_eq!(request.pointer("/input/identity/account").and_then(Value::as_str), Some(user));
+        assert_eq!(request.pointer("/input/context/deny_only").and_then(Value::as_bool), Some(false));
+
+        let action = request
+            .pointer("/input/action")
+            .and_then(Value::as_str)
+            .ok_or("OPA ListBuckets input should include action")?;
+        let bucket = request
+            .pointer("/input/resource/bucket")
+            .and_then(Value::as_str)
+            .ok_or("OPA ListBuckets input should include resource.bucket")?;
+        if bucket.is_empty() {
+            assert_eq!(action, "s3:ListAllMyBuckets");
+            assert!(request.pointer("/input/context/conditions/prefix").is_none());
+            assert!(request.pointer("/input/context/conditions/delimiter").is_none());
+        } else {
+            let expected_arn = format!("arn:aws:s3:::{bucket}");
+            assert_eq!(request.pointer("/input/context/conditions/prefix"), Some(&serde_json::json!([""])));
+            assert_eq!(request.pointer("/input/context/conditions/delimiter"), Some(&serde_json::json!(["/"])));
+            assert_eq!(
+                request.pointer("/input/resource/arn").and_then(Value::as_str),
+                Some(expected_arn.as_str())
+            );
+        }
+        evaluations.insert((action.to_owned(), bucket.to_owned()));
+    }
+    assert_eq!(
+        evaluations,
+        BTreeSet::from([
+            ("s3:GetBucketLocation".to_owned(), "opa-list-hidden".to_owned()),
+            ("s3:GetBucketLocation".to_owned(), "opa-list-location".to_owned()),
+            ("s3:ListAllMyBuckets".to_owned(), String::new()),
+            ("s3:ListBucket".to_owned(), "opa-list-hidden".to_owned()),
+            ("s3:ListBucket".to_owned(), "opa-list-location".to_owned()),
+            ("s3:ListBucket".to_owned(), "opa-list-visible".to_owned()),
+        ])
+    );
+    assert!(
+        matches!(opa.requests.try_recv(), Err(mpsc::error::TryRecvError::Empty)),
+        "ListBuckets should not make redundant OPA evaluations"
+    );
+
+    env.stop_server();
+    Ok(())
+}
+
+#[tokio::test]
+#[serial]
+async fn test_sts_and_list_buckets_fail_closed_while_opa_is_initializing() -> TestResult {
+    init_logging();
+
+    let mut opa = OpaMock::start_blocked().await?;
     let mut env = RustFSTestEnvironment::new().await?;
     env.start_rustfs_server_with_env(vec![], &[("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str())])
         .await?;
     opa.wait_for_validation().await?;
 
-    let user = "opaunavailable";
-    let secret = "stsOpaUnavailableSecret123";
-    create_user_with_policy(
-        &env,
-        user,
-        secret,
-        "sts-opa-unavailable-local-policy",
-        serde_json::json!([{
-            "Effect": "Allow",
-            "Action": ["s3:ListAllMyBuckets"],
-            "Resource": ["arn:aws:s3:::*"],
-        }]),
-    )
-    .await?;
+    assert_opa_unavailable_denies_sts_and_list_buckets(&env, "configured OPA initialization").await?;
 
-    assert_access_denied(&sts_client(&env.url, user, secret, None), "configured OPA initialization").await?;
+    env.stop_server();
+    Ok(())
+}
 
-    opa.release_validation();
-    tokio::time::sleep(Duration::from_millis(200)).await;
-    assert_access_denied(&sts_client(&env.url, user, secret, None), "configured OPA validation failure").await?;
+#[tokio::test]
+#[serial]
+async fn test_sts_and_list_buckets_fail_closed_after_opa_validation_failure() -> TestResult {
+    init_logging();
+
+    let mut opa = OpaMock::start_with_mode(OpaValidationMode::Unavailable, None).await?;
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str())])
+        .await?;
+    opa.wait_for_validation().await?;
+
+    assert_opa_unavailable_denies_sts_and_list_buckets(&env, "configured OPA validation failure").await?;
 
     env.stop_server();
     Ok(())

crates/ecstore/Cargo.toml

@@ -144,11 +144,13 @@ rustfs-lifecycle.workspace = true
 rustfs-s3-types = { workspace = true }
 rustfs-data-usage.workspace = true
 rustfs-object-capacity.workspace = true
+rustfs-object-data-cache = { workspace = true, features = ["runtime-memory"] }
 arc-swap.workspace = true
 async-trait.workspace = true
 bytes = { workspace = true, features = ["serde"] }
 byteorder = { workspace = true }
 chrono = { workspace = true, features = ["serde"] }
+jiff = { workspace = true, features = ["serde"] }
 glob = { workspace = true }
 thiserror.workspace = true
 flatbuffers.workspace = true
@@ -179,6 +181,7 @@ path-absolutize = { workspace = true }
 rmp.workspace = true
 rmp-serde.workspace = true
 tokio-util = { workspace = true, features = ["io", "compat"] }
+tokio-stream = { workspace = true, features = ["sync"] }
 base64 = { workspace = true }
 hmac = { workspace = true }
 sha1 = { workspace = true }
@@ -237,7 +240,19 @@ rustfs-uring = "0.2.1"
 
 [target.'cfg(windows)'.dependencies]
 winapi-util.workspace = true
-windows-sys = { workspace = true, features = ["Win32_Foundation", "Win32_Storage_FileSystem"] }
+windows-sys = { workspace = true, features = [
+    "Wdk_Foundation",
+    "Wdk_Storage_FileSystem",
+    "Win32_Foundation",
+    "Win32_Security",
+    "Win32_Storage_FileSystem",
+    "Win32_System_IO",
+    "Win32_System_SystemServices",
+    "Win32_System_WindowsProgramming",
+] }
+
+[target.'cfg(windows)'.dev-dependencies]
+windows-sys = { workspace = true, features = ["Win32_System_Ioctl"] }
 
 [dev-dependencies]
 tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util", "fs"] }

crates/ecstore/benches/single_block_non_inline_benchmark.rs

@@ -69,6 +69,7 @@ fn build_non_inline_writers(config: &BenchConfig) -> Vec<Option<BitrotWriterWrap
 fn bench_single_block_non_inline_fast_path(c: &mut Criterion) {
     let configs = vec![
         BenchConfig::new(4 * 1024, 4, 2, 128 * 1024),
+        BenchConfig::new(16 * 1024, 4, 2, 128 * 1024),
         BenchConfig::new(64 * 1024, 4, 2, 128 * 1024),
         BenchConfig::new(128 * 1024, 4, 2, 128 * 1024),
     ];
@@ -112,7 +113,12 @@ fn bench_single_block_non_inline_fast_path(c: &mut Criterion) {
                 rt.block_on(async {
                     erasure
                         .clone()
-                        .encode_single_block_non_inline(reader, &mut writers, config.data_shards)
+                        .encode_single_block_non_inline_with_size_hint(
+                            reader,
+                            &mut writers,
+                            config.data_shards,
+                            config.payload_size,
+                        )
                         .await
                         .expect("single block candidate benchmark");
                 });

crates/ecstore/src/api/mod.rs

@@ -32,7 +32,7 @@ pub mod bucket {
     pub mod bucket_target_sys {
         pub use crate::bucket::bucket_target_sys::{
             AdvancedPutOptions, BucketTargetError, BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError,
-            TargetClient,
+            TargetClient, append_version_id_query,
         };
     }
 
@@ -185,19 +185,19 @@ pub mod bucket {
             MustReplicateOptions, ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS,
             REMOTE_TARGET_WRITABLE_FIELDS, REPLICATE_INCOMING_DELETE, REPLICATION_CAPABILITY_CONTRACT_VERSION,
             REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS, ReplicateDecision, ReplicateObjectInfo,
-            ReplicationBatchAdmission, ReplicationConfig, ReplicationConfigurationExt, ReplicationDeleteScheduleInput,
-            ReplicationDeleteStateSource, ReplicationHealQueueResult, ReplicationObjectBridge, ReplicationObjectIO,
-            ReplicationOperation, ReplicationPoolTrait, ReplicationPriority, ReplicationQueueAdmission, ReplicationScannerBridge,
-            ReplicationState, ReplicationStats, ReplicationStatusType, ReplicationStorage, ReplicationTargetValidationError,
-            ReplicationType, ResyncOpts, ResyncStatusType, RuntimeReplicationTargetBacklog, TargetReplicationResyncStatus,
-            VersionPurgeStatusType, commit_force_delete_intent, complete_force_delete_intent,
+            ReplicationBatchAdmission, ReplicationConfig, ReplicationConfigStructureError, ReplicationConfigurationExt,
+            ReplicationDeleteScheduleInput, ReplicationDeleteStateSource, ReplicationHealQueueResult, ReplicationObjectBridge,
+            ReplicationObjectIO, ReplicationOperation, ReplicationPoolTrait, ReplicationPriority, ReplicationQueueAdmission,
+            ReplicationScannerBridge, ReplicationState, ReplicationStats, ReplicationStatusType, ReplicationStorage,
+            ReplicationTargetValidationError, ReplicationType, ResyncOpts, ResyncStatusType, RuntimeReplicationTargetBacklog,
+            TargetReplicationResyncStatus, VersionPurgeStatusType, commit_force_delete_intent, complete_force_delete_intent,
             delete_replication_state_from_config, delete_replication_version_id, get_global_replication_pool,
             get_global_replication_stats, init_background_replication, invalid_replication_config_status_field,
             persist_force_delete_intent, read_durable_mrf_backlog, replication_state_to_filemeta, replication_status_to_filemeta,
             replication_statuses_map, replication_target_arns, resync_start_conflict_id, should_remove_replication_target,
             should_schedule_delete_replication, should_use_existing_delete_replication_info,
             should_use_existing_delete_replication_source, unsupported_replication_config_field,
-            validate_replication_config_target_arns, version_purge_status_to_filemeta,
+            validate_replication_config_structure, validate_replication_config_target_arns, version_purge_status_to_filemeta,
         };
     }
 
@@ -206,7 +206,9 @@ pub mod bucket {
     }
 
     pub mod target {
-        pub use crate::bucket::target::{ARN, BucketTarget, BucketTargetType, BucketTargets, Credentials, LatencyStat};
+        pub use crate::bucket::target::{
+            ARN, BucketTarget, BucketTargetType, BucketTargets, Credentials, LatencyStat, duration_from_secs_or_nanos,
+        };
     }
 
     pub mod utils {
@@ -279,7 +281,7 @@ pub mod config {
     pub mod com {
         pub use crate::config::com::{
             COMMA_SEPARATED_LISTS, CONFIG_PREFIX, ENV_CONFIG_RECOVER_ON_CORRUPTION, STORAGE_CLASS_SUB_SYS,
-            ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config,
+            ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config, delete_config_no_lock,
             is_server_config_corrupt_error, lookup_configs, read_config, read_config_no_lock, read_config_with_metadata,
             read_config_without_migrate, read_config_without_migrate_no_lock, read_existing_server_config_no_lock,
             read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_server_config,
@@ -306,9 +308,12 @@ pub mod config {
 }
 
 pub mod data_usage {
+    #[cfg(feature = "test-util")]
+    pub use crate::data_usage::seed_bucket_usage_memory_for_test;
     pub use crate::data_usage::{
         DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
-        init_compression_total_memory_from_backend, invalidate_data_usage_snapshot_cache, live_bucket_usage_computations,
+        init_compression_total_memory_from_backend, invalidate_admin_data_usage_snapshot_cache,
+        invalidate_data_usage_snapshot_cache, live_bucket_usage_computations, load_admin_data_usage_from_backend_cached,
         load_compression_total_from_memory, load_data_usage_from_backend, load_data_usage_from_backend_cached,
         record_bucket_delete_marker_memory, record_bucket_object_delete_memory, record_bucket_object_version_write_memory,
         record_bucket_object_write_memory, record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
@@ -323,8 +328,8 @@ pub mod disk {
     pub use crate::disk::local::ScanGuard;
     pub use crate::disk::{
         BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
-        CheckPartsResp, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, DiskStore,
-        FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
+        CheckPartsResp, ConditionalFileUpdate, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
+        DiskStore, FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
         PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
         STORAGE_FORMAT_FILE, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk,
         validate_batch_read_version_item_count,
@@ -406,12 +411,15 @@ pub mod object {
     pub use crate::object_api::{
         BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
         GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, NamespaceLockFence, ObjectEncryptionResolver,
-        ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, RangedDecompressReader,
-        ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest, StreamConsumer, get_object_body_cache_plaintext_len,
-        lookup_get_object_body_cache_hook, register_get_object_body_cache_hook, register_object_mutation_hook,
-        unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
+        ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, QuotaAdmission,
+        RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest, StreamConsumer,
+        get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
+        register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
+    };
+    pub use crate::store::{
+        PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError,
+        SnapshotConsistencyError,
     };
-    pub use crate::store::PreparedGetObjectReader;
 }
 
 pub mod rebalance {
@@ -433,18 +441,29 @@ pub mod rpc {
     pub use crate::cluster::rpc::{
         AuthenticatedChannel, KMS_SIGNAL_SUBSYSTEM, LocalPeerS3Client, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS,
         PeerRestClient, PeerS3Client, S3PeerSys, SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC,
-        ScannerBucketListing, ScannerPeerActivity, TONIC_RPC_PREFIX, TonicInterceptor, gen_signature_headers,
-        gen_tonic_replay_scope_headers, gen_tonic_signature_headers, gen_tonic_signature_interceptor,
-        node_service_time_out_client, node_service_time_out_client_no_auth, normalize_tonic_rpc_audience,
-        set_tonic_canonical_body_digest, sign_ns_scanner_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge,
-        tonic_boot_epoch_response_headers, verify_rpc_signature, verify_tonic_boot_epoch_response,
-        verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest, verify_tonic_rpc_response_proof,
-        verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
+        ScannerBucketListing, ScannerPeerActivity, TONIC_RPC_PREFIX, TonicInterceptor, build_put_file_auth_trailer,
+        check_and_record_signed_rpc_nonce, gen_signature_headers, gen_tonic_replay_scope_headers, gen_tonic_signature_headers,
+        gen_tonic_signature_interceptor, node_service_time_out_client, node_service_time_out_client_no_auth,
+        normalize_tonic_rpc_audience, set_tonic_canonical_body_digest, sign_ns_scanner_capability, sign_put_file_capability,
+        sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers,
+        tonic_rpc_auth_failure_reason, verify_put_file_auth_trailer, verify_put_file_capability, verify_rpc_signature,
+        verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest,
+        verify_tonic_rpc_response_proof, verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
     };
 }
 
 pub mod set_disk {
     pub use crate::set_disk::{DEFAULT_READ_BUFFER_SIZE, SetDisks, get_lock_acquire_timeout, is_valid_storage_class};
+
+    /// Return the canonical object-metadata identity used for read-quorum grouping.
+    pub fn file_info_quorum_hash(meta: &rustfs_filemeta::FileInfo) -> [u8; 32] {
+        crate::set_disk::SetDisks::file_info_quorum_hash(meta)
+    }
+
+    #[cfg(feature = "test-util")]
+    pub mod test_util {
+        pub use crate::set_disk::{PutObjectCommitBarrier, PutObjectCommitPause};
+    }
 }
 
 pub mod store_list {

crates/ecstore/src/bucket/bandwidth/monitor.rs

@@ -13,7 +13,7 @@
 // limitations under the License.
 
 use crate::bucket::bandwidth::reader::BucketOptions;
-use ratelimit::{Error as RatelimitError, Ratelimiter};
+use ratelimit::{Clock, Error as RatelimitError, Ratelimiter};
 use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
 use std::sync::atomic::{AtomicU64, Ordering};
@@ -24,6 +24,33 @@ use tracing::warn;
 /// BETA_BUCKET is the weight used to calculate exponential moving average
 const BETA_BUCKET: f64 = 0.1;
 
+// ratelimit 2.0 stores tokens at six decimal places. Above this limit its
+// scaled capacity and token-cost calculations saturate instead of preserving
+// the configured bandwidth.
+const MAX_RATELIMIT_TOKENS: i64 = 18_446_744_073_709;
+
+fn consume_tokens<C: Clock>(limiter: &Ratelimiter<C>, n: u64) -> (u64, f64, u64) {
+    if n == 0 {
+        return (0, limiter.rate() as f64, 0);
+    }
+
+    let mut consumed = 0u64;
+    // Consuming one token also refills the bucket based on elapsed time, so
+    // the subsequent `available()` read reflects freshly accrued tokens.
+    if limiter.try_wait().is_ok() {
+        consumed = 1;
+    }
+    let available = limiter.available();
+    let to_consume = n - consumed;
+    let batch = to_consume.min(available);
+    if batch > 0 && limiter.try_wait_n(batch).is_ok() {
+        consumed += batch;
+    }
+    let deficit = n.saturating_sub(consumed);
+    let rate = limiter.rate() as f64;
+    (deficit, rate, consumed)
+}
+
 #[derive(Clone)]
 pub struct BucketThrottle {
     limiter: Arc<Mutex<Ratelimiter>>,
@@ -34,9 +61,9 @@ impl BucketThrottle {
     fn new(node_bandwidth_per_sec: i64) -> Result<Self, RatelimitError> {
         let node_bandwidth_per_sec = node_bandwidth_per_sec.max(1);
         let amount = node_bandwidth_per_sec as u64;
-        let limiter_inner = Ratelimiter::builder(amount, Duration::from_secs(1))
-            .max_tokens(amount)
-            .build()?;
+        // ratelimit 2.0's builder takes a per-second rate; the refill period
+        // defaults to one second, so `amount` tokens accrue per second.
+        let limiter_inner = Ratelimiter::builder(amount).max_tokens(amount).build()?;
         Ok(Self {
             limiter: Arc::new(Mutex::new(limiter_inner)),
             node_bandwidth_per_sec,
@@ -47,32 +74,21 @@ impl BucketThrottle {
         self.limiter.lock().unwrap_or_else(|e| e.into_inner()).max_tokens()
     }
 
-    /// The ratelimit crate (0.10.0) does not provide a bulk token consumption API.
-    /// try_wait() first to consume 1 token AND trigger the internal refill
-    /// mechanism (tokens are only refilled during try_wait/wait calls).
-    /// directly adjust available tokens via set_available() to consume the remaining amount.
+    /// Best-effort bulk token consumption: consume up to `n` tokens and report
+    /// how many were taken plus any shortfall.
+    ///
+    /// `try_wait_n` on the ratelimit crate is all-or-nothing, so we cannot ask
+    /// for `n` directly and still consume a partial amount. Instead we take one
+    /// token first (which also triggers the internal time-based refill), read
+    /// the now-current available count, and consume `min(remaining, available)`
+    /// in a single `try_wait_n` call — that batch never exceeds `available`, so
+    /// it always succeeds.
     pub(crate) fn consume(&self, n: u64) -> (u64, f64, u64) {
         let guard = self.limiter.lock().unwrap_or_else(|e| {
             warn!("bucket throttle mutex poisoned, recovering");
             e.into_inner()
         });
-        if n == 0 {
-            return (0, guard.rate(), 0);
-        }
-        let mut consumed = 0u64;
-        if guard.try_wait().is_ok() {
-            consumed = 1;
-        }
-        let available = guard.available();
-        let to_consume = n - consumed;
-        let batch = to_consume.min(available);
-        if batch > 0 {
-            let _ = guard.set_available(available - batch);
-            consumed += batch;
-        }
-        let deficit = n.saturating_sub(consumed);
-        let rate = guard.rate();
-        (deficit, rate, consumed)
+        consume_tokens(&guard, n)
     }
 }
 
@@ -329,6 +345,16 @@ impl Monitor {
                 "bandwidth limit too small for cluster size, per-node limit will clamp to 1 byte/s"
             );
         }
+        if limit_bytes > MAX_RATELIMIT_TOKENS {
+            warn!(
+                bucket = bucket,
+                arn = arn,
+                limit_bytes = limit_bytes,
+                max_limit_bytes = MAX_RATELIMIT_TOKENS,
+                "bandwidth limit exceeds ratelimiter capacity, throttling disabled for this target"
+            );
+            return;
+        }
         let opts = BucketOptions {
             name: bucket.to_string(),
             replication_arn: arn.to_string(),
@@ -375,6 +401,30 @@ mod tests {
     use super::*;
     use std::panic::{AssertUnwindSafe, catch_unwind};
 
+    #[derive(Clone)]
+    struct TestClock {
+        elapsed_ns: Arc<AtomicU64>,
+    }
+
+    impl TestClock {
+        fn new() -> Self {
+            Self {
+                elapsed_ns: Arc::new(AtomicU64::new(0)),
+            }
+        }
+
+        fn advance(&self, duration: Duration) {
+            let elapsed_ns = u64::try_from(duration.as_nanos()).unwrap_or(u64::MAX);
+            self.elapsed_ns.fetch_add(elapsed_ns, Ordering::Relaxed);
+        }
+    }
+
+    impl Clock for TestClock {
+        fn elapsed(&self) -> Duration {
+            Duration::from_nanos(self.elapsed_ns.load(Ordering::Relaxed))
+        }
+    }
+
     #[test]
     fn test_set_and_get_throttle_with_node_split() {
         let monitor = Monitor::new(4);
@@ -426,6 +476,15 @@ mod tests {
         assert!(!monitor.is_throttled("b1", "arn1"));
     }
 
+    #[test]
+    fn test_set_bandwidth_limit_rejects_unrepresentable_rate() {
+        let monitor = Monitor::new(1);
+
+        monitor.set_bandwidth_limit("b1", "arn1", MAX_RATELIMIT_TOKENS + 1);
+
+        assert!(!monitor.is_throttled("b1", "arn1"));
+    }
+
     #[test]
     fn test_consume_returns_deficit_when_tokens_exhausted() {
         let throttle = BucketThrottle::new(100).expect("test");
@@ -436,6 +495,23 @@ mod tests {
         assert!(rate > 0.0);
     }
 
+    #[test]
+    fn test_consume_refills_continuously() {
+        let clock = TestClock::new();
+        let limiter = Ratelimiter::with_clock(100, clock.clone());
+
+        assert_eq!(consume_tokens(&limiter, 100), (100, 100.0, 0));
+
+        clock.advance(Duration::from_millis(250));
+        assert_eq!(consume_tokens(&limiter, 100), (75, 100.0, 25));
+
+        clock.advance(Duration::from_millis(250));
+        assert_eq!(consume_tokens(&limiter, 100), (75, 100.0, 25));
+
+        clock.advance(Duration::from_millis(500));
+        assert_eq!(consume_tokens(&limiter, 100), (50, 100.0, 50));
+    }
+
     #[test]
     fn test_consume_no_deficit_when_tokens_sufficient() {
         let throttle = BucketThrottle::new(10000).expect("test");

crates/ecstore/src/bucket/bandwidth/reader.rs

@@ -306,7 +306,7 @@ mod tests {
     #[tokio::test]
     async fn test_monitored_reader_header_size_accounting() {
         let monitor = Monitor::new(1);
-        monitor.set_bandwidth_limit("b1", "arn1", 100);
+        monitor.set_bandwidth_limit("b1", "arn1", 1_000_000_000);
 
         let data = vec![0u8; 200];
         let inner = TestAsyncReader::new(&data);

crates/ecstore/src/bucket/bucket_target_sys.rs

@@ -1424,6 +1424,37 @@ fn resolve_delete_api_version_id(version_id: Option<String>, opts: &RemoveObject
     }
 }
 
+/// Resolve the S3 `versionId` query parameter for a replication PUT /
+/// CreateMultipartUpload against a remote target.
+///
+/// MinIO reads the replicated version only from the query string
+/// (`putOptsFromReq`); the internal `x-*-source-version-id` headers do not
+/// exist there, so without the query a MinIO target mints fresh version ids
+/// and the deployments drift apart. RustFS represents the null version
+/// internally as the nil UUID while the S3 API addresses it as the literal
+/// "null" (the delete path already maps it via `target_delete_version_id`),
+/// and an empty id means the source object carries no version: send no query
+/// so an unversioned target stays valid.
+fn resolve_put_api_version_id(source_version_id: &str) -> Option<&str> {
+    if source_version_id.is_empty() {
+        None
+    } else if Uuid::parse_str(source_version_id).is_ok_and(|uuid| uuid.is_nil()) {
+        Some(rustfs_filemeta::NULL_VERSION_ID)
+    } else {
+        Some(source_version_id)
+    }
+}
+
+/// Append `versionId=<id>` to an already-built request URI. aws-sdk-s3's
+/// `PutObjectInput` / `CreateMultipartUploadInput` expose no version id
+/// member, so the query is spliced in via `map_request`, which runs at
+/// `modify_before_signing`: the parameter becomes part of the SigV4 canonical
+/// request.
+pub fn append_version_id_query(uri: &str, version_id: &str) -> String {
+    let separator = if uri.contains('?') { '&' } else { '?' };
+    format!("{uri}{separator}versionId={}", urlencoding::encode(version_id))
+}
+
 #[derive(Debug, Clone)]
 pub struct AdvancedPutOptions {
     pub source_version_id: String,
@@ -1801,12 +1832,27 @@ impl TargetClient {
         object: &str,
         version_id: Option<String>,
     ) -> Result<HeadObjectOutput, SdkError<HeadObjectError>> {
+        // Announce the replication check so a RustFS target returns SSE-C
+        // object metadata (etag/size) without the customer key the replication
+        // worker cannot hold; otherwise SSE-C replicas never converge on HEAD.
+        let mut headers = HeaderMap::new();
+        insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_CHECK, "true");
         match self
             .client
             .head_object()
             .bucket(bucket)
             .key(object)
             .set_version_id(version_id)
+            .customize()
+            .map_request(move |mut req| {
+                for (k, v) in headers.clone().into_iter() {
+                    if let Some(key_str) = k.map(|k| k.as_str().to_string()) {
+                        let value_str = v.to_str().unwrap_or("").to_string();
+                        req.headers_mut().insert(key_str, value_str);
+                    }
+                }
+                Result::<_, std::convert::Infallible>::Ok(req)
+            })
             .send()
             .await
         {
@@ -1815,6 +1861,9 @@ impl TargetClient {
         }
     }
 
+    /// On success returns the version id the target assigned (from
+    /// `x-amz-version-id`), letting callers audit the version-identity
+    /// contract — a target that adopts the source version echoes it back.
     pub async fn put_object(
         &self,
         bucket: &str,
@@ -1822,7 +1871,7 @@ impl TargetClient {
         size: i64,
         body: ByteStream,
         opts: &PutObjectOptions,
-    ) -> Result<(), S3ClientError> {
+    ) -> Result<Option<String>, S3ClientError> {
         let mut headers = opts.header();
 
         let builder = self.client.put_object();
@@ -1831,6 +1880,7 @@ impl TargetClient {
         if !version_id.is_empty() {
             insert_header(&mut headers, SUFFIX_SOURCE_VERSION_ID, &version_id);
         }
+        let api_version_id = resolve_put_api_version_id(&version_id).map(ToOwned::to_owned);
 
         match builder
             .bucket(bucket)
@@ -1845,13 +1895,18 @@ impl TargetClient {
                         req.headers_mut().insert(key_str, value_str);
                     }
                 }
+                if let Some(version_id) = &api_version_id {
+                    let uri = append_version_id_query(req.uri(), version_id);
+                    req.set_uri(uri)
+                        .map_err(aws_smithy_types::error::operation::BuildError::other)?;
+                }
 
                 Result::<_, aws_smithy_types::error::operation::BuildError>::Ok(req)
             })
             .send()
             .await
         {
-            Ok(_) => Ok(()),
+            Ok(output) => Ok(output.version_id().map(ToOwned::to_owned)),
             Err(e) => match e {
                 SdkError::ServiceError(service_err) => {
                     let err = service_err.into_err();
@@ -1885,14 +1940,14 @@ impl TargetClient {
         object: &str,
         opts: &PutObjectOptions,
     ) -> Result<String, S3ClientError> {
-        let mut headers = HeaderMap::new();
+        // Object metadata belongs to CreateMultipartUpload in S3 semantics;
+        // building only the source-version headers here used to drop user
+        // metadata, content-type, and the SSE intent for multipart replicas.
+        let headers = opts.header();
         let version_id = opts.internal.source_version_id.clone();
-        if !version_id.is_empty() {
-            insert_header(&mut headers, SUFFIX_SOURCE_VERSION_ID, &version_id);
-        }
-        if opts.internal.replication_request {
-            insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
-        }
+        // The remote version of a multipart replication is decided at initiate
+        // time; CompleteMultipartUpload does not read a versionId.
+        let api_version_id = resolve_put_api_version_id(&version_id).map(ToOwned::to_owned);
 
         match self
             .client
@@ -1907,6 +1962,11 @@ impl TargetClient {
                         req.headers_mut().insert(key_str, value_str);
                     }
                 }
+                if let Some(version_id) = &api_version_id {
+                    let uri = append_version_id_query(req.uri(), version_id);
+                    req.set_uri(uri)
+                        .map_err(aws_smithy_types::error::operation::BuildError::other)?;
+                }
                 Result::<_, aws_smithy_types::error::operation::BuildError>::Ok(req)
             })
             .send()
@@ -2679,6 +2739,91 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    async fn put_object_sends_source_version_id_query_to_target() {
+        // MinIO reads the replicated version only from the `versionId` query
+        // parameter (its receive path ignores the x-*-source-version-id
+        // headers), so the query must carry the source version: a real UUID
+        // as-is, the internal nil-UUID null-version representation as the
+        // literal "null", and no query at all when the source object has no
+        // version (P0-5 RustFS->MinIO version drift).
+        let (client, request_uris) = recording_target_client();
+        let version_id = Uuid::new_v4().to_string();
+        let nil_version = Uuid::nil().to_string();
+        for source_version in [version_id.as_str(), nil_version.as_str(), ""] {
+            let mut opts = PutObjectOptions::default();
+            opts.internal.source_version_id = source_version.to_string();
+            opts.internal.replication_request = true;
+            client
+                .put_object("target-bucket", "object", 4, ByteStream::from_static(b"data"), &opts)
+                .await
+                .expect("recorded put_object should succeed");
+        }
+
+        let request_uris = request_uris.lock().expect("recorded request lock should not be poisoned");
+        assert_eq!(request_uris.len(), 3);
+        assert!(
+            request_uris[0].contains(&format!("versionId={version_id}")),
+            "replication put_object must carry the source version as a versionId query: {}",
+            request_uris[0]
+        );
+        assert!(
+            request_uris[1].contains("versionId=null"),
+            "a nil-UUID (null) source version must be sent as the literal null: {}",
+            request_uris[1]
+        );
+        assert!(
+            !request_uris[2].contains("versionId="),
+            "put_object without a source version must omit the versionId query: {}",
+            request_uris[2]
+        );
+    }
+
+    #[tokio::test]
+    async fn create_multipart_upload_sends_source_version_id_query_to_target() {
+        // The remote version of a multipart replication is decided at initiate
+        // time: CreateMultipartUpload must carry the source version in the
+        // `versionId` query (CompleteMultipartUpload does not read one).
+        let (client, request_uris) = recording_target_client();
+        let version_id = Uuid::new_v4().to_string();
+        let nil_version = Uuid::nil().to_string();
+        for source_version in [version_id.as_str(), nil_version.as_str()] {
+            let mut opts = PutObjectOptions::default();
+            opts.internal.source_version_id = source_version.to_string();
+            opts.internal.replication_request = true;
+            let _ = client.create_multipart_upload("target-bucket", "object", &opts).await;
+        }
+
+        let request_uris = request_uris.lock().expect("recorded request lock should not be poisoned");
+        assert_eq!(request_uris.len(), 2);
+        assert!(
+            request_uris[0].contains(&format!("versionId={version_id}")),
+            "replication create_multipart_upload must carry the source version as a versionId query: {}",
+            request_uris[0]
+        );
+        assert!(
+            request_uris[1].contains("versionId=null"),
+            "a nil-UUID (null) source version must be sent as the literal null: {}",
+            request_uris[1]
+        );
+    }
+
+    #[test]
+    fn put_object_headers_keep_source_version_id_for_legacy_receivers() {
+        // Older RustFS receivers have no versionId query support and fall back
+        // to the internal source-version-id headers (rolling-upgrade path);
+        // the query addition must never remove them.
+        let mut opts = PutObjectOptions::default();
+        let version_id = Uuid::new_v4().to_string();
+        opts.internal.source_version_id = version_id.clone();
+
+        assert_eq!(
+            rustfs_utils::http::get_header(&opts.header(), SUFFIX_SOURCE_VERSION_ID).as_deref(),
+            Some(version_id.as_str()),
+            "replication put requests must keep the internal source-version-id headers"
+        );
+    }
+
     #[test]
     fn put_object_headers_include_non_empty_source_etag_only() {
         let mut opts = PutObjectOptions::default();

crates/ecstore/src/bucket/durability.rs

@@ -64,10 +64,41 @@ impl BucketDurabilityConfig {
     }
 }
 
+/// Default durability tier seeded into a newly created bucket's metadata
+/// (rustfs/backlog#1811). `relaxed` aligns new buckets with MinIO's default
+/// posture: object data is still fdatasynced, while xl.meta and directory-entry
+/// fsyncs follow the relaxed durability gate.
+pub const ENV_NEW_BUCKET_DURABILITY_MODE: &str = "RUSTFS_NEW_BUCKET_DURABILITY_MODE";
+pub const DEFAULT_NEW_BUCKET_DURABILITY_MODE: &str = BUCKET_DURABILITY_MODE_RELAXED;
+
+/// The `durability.json` bytes to seed into a freshly created bucket's metadata.
+/// Empty means "no override" (the bucket then follows the global
+/// `RUSTFS_DURABILITY_MODE`); otherwise the serialized chosen tier. Operators
+/// can set `inherit` to disable the new-bucket override. Invalid values also
+/// fail closed to inherit the global mode instead of seeding a surprising tier.
+pub fn new_bucket_durability_config_json() -> Vec<u8> {
+    let raw = std::env::var(ENV_NEW_BUCKET_DURABILITY_MODE).unwrap_or_else(|_| DEFAULT_NEW_BUCKET_DURABILITY_MODE.to_string());
+    let mode = raw.trim();
+    if mode.eq_ignore_ascii_case("inherit") || mode.is_empty() || !BucketDurabilityConfig::is_valid_mode(mode) {
+        return Vec::new();
+    }
+    serde_json::to_vec(&BucketDurabilityConfig::new(mode)).expect("BucketDurabilityConfig serialization cannot fail")
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
 
+    fn new_bucket_seeded_mode() -> Option<String> {
+        let json = new_bucket_durability_config_json();
+        if json.is_empty() {
+            return None;
+        }
+        serde_json::from_slice::<BucketDurabilityConfig>(&json)
+            .expect("new-bucket durability config must serialize")
+            .normalized_mode()
+    }
+
     #[test]
     fn valid_modes_are_recognized() {
         assert!(BucketDurabilityConfig::is_valid_mode("strict"));
@@ -99,4 +130,33 @@ mod tests {
         let empty: BucketDurabilityConfig = serde_json::from_slice(b"{}").expect("deserialize empty");
         assert_eq!(empty.normalized_mode(), None);
     }
+
+    #[test]
+    fn new_bucket_default_seeds_relaxed_when_unset() {
+        temp_env::with_var_unset(ENV_NEW_BUCKET_DURABILITY_MODE, || {
+            assert_eq!(new_bucket_seeded_mode().as_deref(), Some(BUCKET_DURABILITY_MODE_RELAXED));
+        });
+    }
+
+    #[test]
+    fn new_bucket_default_honors_explicit_tiers() {
+        for mode in [
+            BUCKET_DURABILITY_MODE_STRICT,
+            BUCKET_DURABILITY_MODE_RELAXED,
+            BUCKET_DURABILITY_MODE_NONE,
+        ] {
+            temp_env::with_var(ENV_NEW_BUCKET_DURABILITY_MODE, Some(mode), || {
+                assert_eq!(new_bucket_seeded_mode().as_deref(), Some(mode));
+            });
+        }
+    }
+
+    #[test]
+    fn new_bucket_default_can_inherit_global_mode() {
+        for mode in ["inherit", "", "bogus"] {
+            temp_env::with_var(ENV_NEW_BUCKET_DURABILITY_MODE, Some(mode), || {
+                assert_eq!(new_bucket_seeded_mode(), None);
+            });
+        }
+    }
 }

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -78,8 +78,8 @@ use rustfs_common::metrics::{
 };
 use rustfs_config::{
     DEFAULT_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_QUEUE_SEND_TIMEOUT_MS, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX,
-    DEFAULT_TRANSITION_WORKERS_CAP, ENV_TRANSITION_QUEUE_CAPACITY, ENV_TRANSITION_QUEUE_SEND_TIMEOUT_MS, ENV_TRANSITION_WORKERS,
-    ENV_TRANSITION_WORKERS_ABSOLUTE_MAX,
+    DEFAULT_TRANSITION_WORKERS_CAP, ENV_MAX_EXPIRY_WORKERS, ENV_TRANSITION_QUEUE_CAPACITY, ENV_TRANSITION_QUEUE_SEND_TIMEOUT_MS,
+    ENV_TRANSITION_WORKERS, ENV_TRANSITION_WORKERS_ABSOLUTE_MAX,
 };
 use rustfs_data_usage::TierStats;
 use rustfs_filemeta::{
@@ -159,6 +159,8 @@ const TIER_FREE_VERSION_RECOVERY_MAX_IDLE_INTERVAL: StdDuration = StdDuration::f
 const TIER_FREE_VERSION_RECOVERY_JITTER_PERCENT: u64 = 10;
 const DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS: i64 = 5;
 const EXPIRY_WORKER_QUEUE_CAPACITY: usize = 1000;
+/// Maximum expiry workers used as a local fallback when runtime env is unset.
+const DEFAULT_EXPIRY_WORKERS_CAP: usize = 16;
 const DEFAULT_MANUAL_TRANSITION_JOB_RECOVERY_LIMIT: usize = 100;
 
 // Phase 5 (backlog#939): lifecycle expiry/transition state moved into the
@@ -204,6 +206,15 @@ fn resolve_transition_queue_send_timeout() -> StdDuration {
     )
 }
 
+fn resolve_expiry_worker_count() -> usize {
+    let fallback = std::cmp::min(num_cpus::get(), DEFAULT_EXPIRY_WORKERS_CAP);
+    env::var(ENV_MAX_EXPIRY_WORKERS)
+        .ok()
+        .and_then(|value| value.parse::<usize>().ok())
+        .filter(|value| *value > 0)
+        .unwrap_or(fallback)
+}
+
 fn is_immediate_transition_source(src: &LcEventSrc) -> bool {
     matches!(
         src,
@@ -2017,17 +2028,7 @@ fn is_slow_down(err: &Error) -> bool {
 }
 
 pub async fn init_background_expiry(api: Arc<ECStore>) {
-    let mut workers = get_env_usize("RUSTFS_MAX_EXPIRY_WORKERS", std::cmp::min(num_cpus::get(), 16));
-    //globalILMConfig.getExpirationWorkers()
-    if let Ok(env_expiration_workers) = env::var("_RUSTFS_ILM_EXPIRATION_WORKERS")
-        && let Ok(num_expirations) = env_expiration_workers.parse::<usize>()
-    {
-        workers = num_expirations;
-    }
-
-    if workers == 0 {
-        workers = get_env_usize("RUSTFS_DEFAULT_EXPIRY_WORKERS", 8);
-    }
+    let workers = resolve_expiry_worker_count();
 
     ExpiryState::resize_workers(workers, api.clone()).await;
     let _ = spawn_tier_free_version_recovery_once(api.clone(), &TIER_FREE_VERSION_RECOVERY_STARTED);
@@ -3318,6 +3319,9 @@ pub async fn enqueue_immediate_expiry(oi: &ObjectInfo, src: LcEventSrc) {
             return;
         }
     };
+    if configs.table_bucket_enabled {
+        return;
+    }
     let Some(lifecycle) = configs.lifecycle else {
         return;
     };
@@ -3978,6 +3982,9 @@ async fn enqueue_expiry_for_existing_object_group(
 
 pub async fn enqueue_expiry_for_existing_objects(api: Arc<ECStore>, bucket: &str) -> Result<(), Error> {
     let configs = metadata_boundary::get_expiry_configs(&api, bucket).await?;
+    if configs.table_bucket_enabled {
+        return Ok(());
+    }
     let Some(lc) = configs.lifecycle else {
         return Ok(());
     };
@@ -4194,12 +4201,16 @@ pub async fn expire_transitioned_object(
     _src: &LcEventSrc,
     bucket_incarnation_id: Uuid,
 ) -> Result<ObjectInfo, std::io::Error> {
+    let publication_guard = lifecycle_expiry_publication_guard(&api, oi, bucket_incarnation_id)
+        .await
+        .ok_or_else(|| std::io::Error::other("lifecycle expiry is not allowed for this bucket"))?;
     let snapshot = lifecycle_delete_config_snapshot(&api, oi)
         .await
         .map_err(std::io::Error::other)?;
     let (versioned, version_suspended) = snapshot.versioning_config().delete_state(&oi.name);
     let mut opts = transitioned_object_delete_opts(oi, lc_event.action, versioned, version_suspended, bucket_incarnation_id)
         .map_err(std::io::Error::other)?;
+    opts.add_namespace_lock_guard(&publication_guard);
     opts.delete_replication_config_snapshot = Some(Arc::new(snapshot));
     //let tags = LcAuditEvent::new(src, lcEvent).Tags();
     if lc_event.action.delete_restored() {
@@ -4789,6 +4800,43 @@ pub async fn apply_transition_rule(event: &lifecycle::Event, src: &LcEventSrc, o
         .await
 }
 
+async fn lifecycle_expiry_publication_guard(
+    api: &ECStore,
+    oi: &ObjectInfo,
+    bucket_incarnation_id: Uuid,
+) -> Option<rustfs_lock::NamespaceLockGuard> {
+    let result = async {
+        let lock = api
+            .new_ns_lock(&oi.bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH)
+            .await?;
+        let guard = lock.get_read_lock(get_lock_acquire_timeout()).await.map_err(Error::other)?;
+        if guard.is_lock_lost() {
+            return Err(Error::other("table-bucket publication lock was lost before lifecycle delete admission"));
+        }
+        if !metadata_boundary::lifecycle_expiry_allowed(api, &oi.bucket, bucket_incarnation_id).await? {
+            return Ok(None);
+        }
+        Ok(Some(guard))
+    }
+    .await;
+    match result {
+        Ok(guard) => guard,
+        Err(err) => {
+            warn!(
+                event = EVENT_LIFECYCLE_DELETE_FAILED,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_LIFECYCLE,
+                bucket = %oi.bucket,
+                object = %oi.name,
+                operation = "authorize_lifecycle_expiry",
+                error = %err,
+                "Lifecycle delete admission failed"
+            );
+            None
+        }
+    }
+}
+
 pub async fn apply_expiry_on_transitioned_object(
     api: Arc<ECStore>,
     oi: &ObjectInfo,
@@ -4812,6 +4860,9 @@ pub async fn apply_expiry_on_non_transitioned_objects(
     _src: &LcEventSrc,
     bucket_incarnation_id: Uuid,
 ) -> bool {
+    let Some(publication_guard) = lifecycle_expiry_publication_guard(&api, oi, bucket_incarnation_id).await else {
+        return false;
+    };
     let snapshot = match lifecycle_delete_config_snapshot(&api, oi).await {
         Ok(snapshot) => snapshot,
         Err(err) => {
@@ -4837,6 +4888,7 @@ pub async fn apply_expiry_on_non_transitioned_objects(
         expected_bucket_incarnation_id: Some(bucket_incarnation_id),
         ..Default::default()
     };
+    opts.add_namespace_lock_guard(&publication_guard);
 
     if lc_event.action.delete_versioned() {
         opts.version_id = oi.version_id.map(|v| v.to_string());
@@ -5024,26 +5076,27 @@ pub async fn apply_lifecycle_action(event: &lifecycle::Event, src: &LcEventSrc,
 #[cfg(test)]
 mod tests {
     use super::{
-        DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS, DEFAULT_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX,
-        DEFAULT_TRANSITION_WORKERS_CAP, EVENT_LIFECYCLE_EVALUATION_FAILED, EVENT_LIFECYCLE_EXPIRED_DETECTED,
-        EVENT_LIFECYCLE_NOT_ENQUEUED, ExpiryState, ExpiryTask, FreeVersionTask, ManualTransitionJobRecoveryOutcome,
-        ManualTransitionQueueSnapshot, ManualTransitionRunOptions, ManualTransitionRunReport, StaleMultipartUploadCandidate,
-        TIER_FREE_VERSION_RECOVERY_BASE_INTERVAL, TIER_FREE_VERSION_RECOVERY_MAX_IDLE_INTERVAL, TRANSITION_COMPLETE,
-        TierFreeVersionRecoverySchedule, TransitionEnqueueOutcome, TransitionState, TransitionedObject, VersionReplicationScan,
-        cleanup_empty_multipart_sha_dirs_on_local_disks, cleanup_stale_multipart_uploads_once_at,
+        DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS, DEFAULT_EXPIRY_WORKERS_CAP, DEFAULT_TRANSITION_QUEUE_CAPACITY,
+        DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX, DEFAULT_TRANSITION_WORKERS_CAP, EVENT_LIFECYCLE_EVALUATION_FAILED,
+        EVENT_LIFECYCLE_EXPIRED_DETECTED, EVENT_LIFECYCLE_NOT_ENQUEUED, ExpiryState, ExpiryTask, FreeVersionTask,
+        ManualTransitionJobRecoveryOutcome, ManualTransitionQueueSnapshot, ManualTransitionRunOptions, ManualTransitionRunReport,
+        StaleMultipartUploadCandidate, TIER_FREE_VERSION_RECOVERY_BASE_INTERVAL, TIER_FREE_VERSION_RECOVERY_MAX_IDLE_INTERVAL,
+        TRANSITION_COMPLETE, TierFreeVersionRecoverySchedule, TransitionEnqueueOutcome, TransitionState, TransitionedObject,
+        VersionReplicationScan, cleanup_empty_multipart_sha_dirs_on_local_disks, cleanup_stale_multipart_uploads_once_at,
         enqueue_recovered_free_version_with_state, enqueue_transition_for_existing_objects_scoped,
         enqueue_transition_with_lifecycle, enqueue_transition_with_lifecycle_report, eval_action_from_lifecycle,
-        jitter_tier_free_version_recovery_delay, lifecycle_action_blocked_by_replication,
+        get_lock_acquire_timeout, jitter_tier_free_version_recovery_delay, lifecycle_action_blocked_by_replication,
         lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object, lifecycle_replication_blocks_action,
         lifecycle_rule_has_date_expiration, manual_transition_duration_elapsed, manual_transition_has_more_after_limit,
         manual_transition_recovery_progress_sink, manual_transition_version_marker, manual_transition_worker_failure_reason,
         mark_delete_opts_skip_decommissioned_on_remote_success, merge_stale_multipart_candidate,
         persist_manual_transition_job_progress, persist_manual_transition_page_checkpoint, recover_manual_transition_job,
-        recover_manual_transition_jobs, resolve_tier_free_version_recovery_enabled, resolve_transition_queue_capacity,
-        resolve_transition_queue_send_timeout, resolve_transition_worker_count, resolve_transition_workers_absolute_max,
-        run_tier_free_version_recovery_loop, select_restore_s3_location, set_lifecycle_observability_observer,
-        set_recovered_free_version_enqueue_observer, should_defer_date_expiry_for_recent_config_update,
-        transitioned_cleanup_tuple, transitioned_object_delete_opts, wait_for_tier_free_version_recovery,
+        recover_manual_transition_jobs, resolve_expiry_worker_count, resolve_tier_free_version_recovery_enabled,
+        resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, resolve_transition_worker_count,
+        resolve_transition_workers_absolute_max, run_tier_free_version_recovery_loop, select_restore_s3_location,
+        set_lifecycle_observability_observer, set_recovered_free_version_enqueue_observer,
+        should_defer_date_expiry_for_recent_config_update, transitioned_cleanup_tuple, transitioned_object_delete_opts,
+        wait_for_tier_free_version_recovery,
     };
     #[cfg(feature = "test-util")]
     use super::{delete_free_version_remote_object_then, encode_dir_object, get_transitioned_object_reader_with_tier_manager};
@@ -5090,7 +5143,6 @@ mod tests {
     #[cfg(feature = "test-util")]
     use crate::services::tier::warm_backend::WarmBackend as _;
     use crate::set_disk::{RUSTFS_MULTIPART_BUCKET_KEY, RUSTFS_MULTIPART_OBJECT_KEY};
-    #[cfg(feature = "test-util")]
     use crate::storage_api_contracts::namespace::NamespaceLocking as _;
     use crate::storage_api_contracts::{
         bucket::{BucketOperations, BucketOptions, DeleteBucketOptions, MakeBucketOptions},
@@ -5105,7 +5157,7 @@ mod tests {
     #[cfg(feature = "test-util")]
     use http::HeaderMap;
     use rustfs_common::metrics::{IlmAction, global_metrics};
-    use rustfs_config::ENV_TRANSITION_WORKERS_ABSOLUTE_MAX;
+    use rustfs_config::{ENV_MAX_EXPIRY_WORKERS, ENV_TRANSITION_WORKERS_ABSOLUTE_MAX};
     use rustfs_data_usage::TierStats;
     use rustfs_filemeta::{FileInfo, FileMeta};
     use s3s::dto::{
@@ -7357,6 +7409,76 @@ mod tests {
         });
     }
 
+    // SAFETY: this helper is only used from `#[serial]` tests and those tests run under a
+    // single-thread runtime (`worker_threads = 1`), so no concurrent reader/writer can access
+    // process environment while `env::set_var`/`env::remove_var` is active.
+    #[allow(unsafe_code)]
+    fn with_expiry_worker_env<F>(value: Option<&str>, test_fn: F)
+    where
+        F: FnOnce(),
+    {
+        let original = env::var_os(ENV_MAX_EXPIRY_WORKERS);
+
+        match value {
+            Some(value) => unsafe {
+                env::set_var(ENV_MAX_EXPIRY_WORKERS, value);
+            },
+            None => unsafe {
+                env::remove_var(ENV_MAX_EXPIRY_WORKERS);
+            },
+        }
+
+        let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(test_fn));
+
+        match original {
+            Some(value) => unsafe {
+                env::set_var(ENV_MAX_EXPIRY_WORKERS, value);
+            },
+            None => unsafe {
+                env::remove_var(ENV_MAX_EXPIRY_WORKERS);
+            },
+        }
+
+        if let Err(e) = result {
+            std::panic::resume_unwind(e);
+        }
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_expiry_worker_count_uses_fallback_when_env_missing() {
+        with_expiry_worker_env(None, || {
+            let fallback = std::cmp::min(num_cpus::get(), DEFAULT_EXPIRY_WORKERS_CAP);
+            assert_eq!(resolve_expiry_worker_count(), fallback);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_expiry_worker_count_honors_positive_env_value() {
+        with_expiry_worker_env(Some("6"), || {
+            assert_eq!(resolve_expiry_worker_count(), 6);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_expiry_worker_count_falls_back_for_zero_value() {
+        with_expiry_worker_env(Some("0"), || {
+            let fallback = std::cmp::min(num_cpus::get(), DEFAULT_EXPIRY_WORKERS_CAP);
+            assert_eq!(resolve_expiry_worker_count(), fallback);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_expiry_worker_count_falls_back_for_invalid_value() {
+        with_expiry_worker_env(Some("not-a-number"), || {
+            let fallback = std::cmp::min(num_cpus::get(), DEFAULT_EXPIRY_WORKERS_CAP);
+            assert_eq!(resolve_expiry_worker_count(), fallback);
+        });
+    }
+
     #[test]
     #[serial]
     fn resolve_transition_queue_capacity_uses_default_when_env_missing() {
@@ -10319,6 +10441,85 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn queued_lifecycle_expiry_does_not_delete_from_table_bucket() {
+        let (_disk_paths, ecstore) = setup_test_env().await;
+        let bucket = format!("table-bucket-lifecycle-{}", Uuid::new_v4().simple());
+        let object = "tables/table-id/data/part-00001.parquet";
+        create_test_bucket(&ecstore, &bucket).await;
+
+        let mut reader = PutObjReader::from_vec(b"referenced table data".to_vec());
+        let object_info = ecstore
+            .put_object(&bucket, object, &mut reader, &ObjectOptions::default())
+            .await
+            .expect("table data object should be created");
+
+        let publication_lock = ecstore
+            .new_ns_lock(&bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH)
+            .await
+            .expect("table-bucket publication lock should be created");
+        let enable_guard = publication_lock
+            .get_write_lock(get_lock_acquire_timeout())
+            .await
+            .expect("table-bucket enablement should acquire the publication lock");
+        let expiry_store = ecstore.clone();
+        let expiry_object = object_info.clone();
+        let (expiry_started_tx, expiry_started_rx) = tokio::sync::oneshot::channel();
+        let mut expiry = tokio::spawn(async move {
+            let event = crate::bucket::lifecycle::lifecycle::Event {
+                action: IlmAction::DeleteAction,
+                ..Default::default()
+            };
+            let bucket_incarnation_id = expiry_store
+                .bucket_incarnation_id_from_disk(&expiry_object.bucket)
+                .await
+                .expect("bucket incarnation should be available");
+            expiry_started_tx.send(()).expect("lifecycle expiry start should be observed");
+            super::apply_expiry_on_non_transitioned_objects(
+                expiry_store,
+                &expiry_object,
+                &event,
+                &LcEventSrc::Scanner,
+                bucket_incarnation_id,
+            )
+            .await
+        });
+        expiry_started_rx.await.expect("lifecycle expiry should start");
+        assert!(
+            tokio::time::timeout(StdDuration::from_millis(100), &mut expiry)
+                .await
+                .is_err(),
+            "queued lifecycle expiry must wait for table-bucket enablement"
+        );
+
+        let sys = metadata_sys::bucket_metadata_sys_of(&ecstore.ctx).expect("metadata system should be initialized");
+        let sys = sys.read().await.clone();
+        let mut metadata = (*sys.get(&bucket).await.expect("bucket metadata should exist")).clone();
+        metadata.table_bucket_config_json = br#"{"enabled":true}"#.to_vec();
+        sys.persist_and_set(metadata)
+            .await
+            .expect("table bucket marker should be persisted");
+        sys.reload_from_store(&bucket)
+            .await
+            .expect("table bucket marker should become authoritative");
+        drop(enable_guard);
+        assert!(
+            !tokio::time::timeout(StdDuration::from_secs(2), expiry)
+                .await
+                .expect("queued lifecycle expiry should resume after enablement")
+                .expect("queued lifecycle expiry task should join"),
+            "a queued lifecycle task must be rejected after the bucket becomes table-enabled"
+        );
+        assert!(
+            ecstore
+                .get_object_info(&bucket, object, &ObjectOptions::default())
+                .await
+                .is_ok(),
+            "table data must remain readable after lifecycle admission rejects the delete"
+        );
+    }
+
     #[tokio::test]
     async fn existing_object_lifecycle_skips_current_expiration_for_explicit_legal_hold() {
         let lc = latest_expiration_lifecycle();
@@ -10774,7 +10975,18 @@ mod tests {
     #[serial]
     async fn tier_free_version_recovery_real_enqueue_failure_retries_same_object() {
         let (disk_paths, ecstore) = setup_test_env().await;
+        let bucket = format!("recovery-enqueue-failure-{}", Uuid::new_v4());
+        let object = "free-version-b";
+        let start_marker = "free-version-a0";
+        create_test_bucket(&ecstore, &bucket).await;
+        seed_recoverable_free_version(&disk_paths, &bucket, object, None, None).await;
+
         let runtime_state = install_unconsumed_runtime_expiry_worker(&ecstore, 1).await;
+        let recovery_rx = {
+            let state = runtime_state.read().await;
+            Arc::clone(&state.tasks_rx[0])
+        };
+        let mut recovery_rx = recovery_rx.lock().await;
         assert!(
             super::enqueue_recovered_free_version(ObjectInfo {
                 bucket: "prefill".to_string(),
@@ -10784,20 +10996,29 @@ mod tests {
             .await,
             "the production recovery queue should accept its first task"
         );
-        let bucket = format!("recovery-enqueue-failure-{}", Uuid::new_v4());
-        let object = "free-version";
-        create_test_bucket(&ecstore, &bucket).await;
-        seed_recoverable_free_version(&disk_paths, &bucket, object, None, None).await;
 
-        let first = recover_tier_free_versions_with_cancel(Arc::clone(&ecstore), 1, None, None, CancellationToken::new())
-            .await
-            .expect("queue failure should return retry markers");
+        let first = recover_tier_free_versions_with_cancel(
+            Arc::clone(&ecstore),
+            1,
+            Some(bucket.clone()),
+            Some(start_marker.to_string()),
+            CancellationToken::new(),
+        )
+        .await
+        .expect("queue failure should return retry markers");
         assert_eq!(first.scanned, 1);
         assert_eq!(first.enqueued, 0);
         assert_eq!(first.failed, 1);
         assert!(first.truncated);
         assert_eq!(first.next_bucket_marker.as_deref(), Some(bucket.as_str()));
-        assert!(first.next_object_marker.is_none());
+        assert_eq!(first.next_object_marker.as_deref(), Some(start_marker));
+
+        drop(
+            recovery_rx
+                .try_recv()
+                .expect("the failed recovery attempt must leave the prefilled task queued")
+                .expect("the prefilled recovery queue entry should contain a task"),
+        );
 
         let retried = recover_tier_free_versions_with_cancel(
             Arc::clone(&ecstore),
@@ -10809,7 +11030,19 @@ mod tests {
         .await
         .expect("retry markers should revisit the failed free version");
         assert_eq!(retried.scanned, 1);
-        assert_eq!(retried.failed, 1);
+        assert_eq!(retried.enqueued, 1);
+        assert_eq!(retried.failed, 0);
+
+        let retried_task = recovery_rx
+            .try_recv()
+            .expect("the retry should enqueue the recovered free-version task")
+            .expect("the recovered queue entry should contain a task");
+        let retried_task = retried_task
+            .as_any()
+            .downcast_ref::<FreeVersionTask>()
+            .expect("the recovered queue entry should be a free-version task");
+        assert_eq!(retried_task.0.bucket, bucket);
+        assert_eq!(retried_task.0.name, object);
 
         remove_seeded_free_version(&disk_paths, &bucket, object).await;
         ecstore

crates/ecstore/src/bucket/lifecycle/metadata_boundary.rs

@@ -18,6 +18,7 @@ use s3s::dto::{BucketLifecycleConfiguration, ObjectLockConfiguration};
 use time::OffsetDateTime;
 use uuid::Uuid;
 
+use crate::bucket::metadata::BucketMetadata;
 use crate::bucket::metadata_sys::{self, ObjectLockConfigState};
 use crate::error::{Error, Result};
 
@@ -26,16 +27,37 @@ pub(crate) struct LifecycleExpiryConfigs {
     pub(crate) lifecycle: Option<Arc<BucketLifecycleConfiguration>>,
     pub(crate) object_lock: Option<Arc<ObjectLockConfiguration>>,
     pub(crate) bucket_incarnation_id: Uuid,
+    pub(crate) table_bucket_enabled: bool,
 }
 
-pub(crate) async fn get_expiry_configs(api: &crate::store::ECStore, bucket: &str) -> Result<LifecycleExpiryConfigs> {
-    let bucket_incarnation_id = api.bucket_incarnation_id_from_disk(bucket).await?;
+async fn get_authoritative_metadata(
+    api: &crate::store::ECStore,
+    bucket: &str,
+    bucket_incarnation_id: Uuid,
+) -> Result<Arc<BucketMetadata>> {
     let sys = metadata_sys::bucket_metadata_sys_of(&api.ctx)?;
     let sys = sys.read().await.clone();
     let metadata = sys.get_authoritative_metadata(bucket).await?;
     if !metadata.bucket_incarnation_sidecar || metadata.bucket_incarnation_id != bucket_incarnation_id {
         return Err(Error::other(format!("bucket lifecycle metadata is not authoritative: {bucket}")));
     }
+    Ok(metadata)
+}
+
+pub(crate) async fn lifecycle_expiry_allowed(
+    api: &crate::store::ECStore,
+    bucket: &str,
+    bucket_incarnation_id: Uuid,
+) -> Result<bool> {
+    Ok(!get_authoritative_metadata(api, bucket, bucket_incarnation_id)
+        .await?
+        .table_bucket_enabled())
+}
+
+pub(crate) async fn get_expiry_configs(api: &crate::store::ECStore, bucket: &str) -> Result<LifecycleExpiryConfigs> {
+    let bucket_incarnation_id = api.bucket_incarnation_id_from_disk(bucket).await?;
+    let metadata = get_authoritative_metadata(api, bucket, bucket_incarnation_id).await?;
+    let table_bucket_enabled = metadata.table_bucket_enabled();
 
     let lifecycle = if metadata.lifecycle_config.is_none() && !metadata.lifecycle_config_xml.is_empty() {
         return Err(Error::other("persisted bucket lifecycle configuration is invalid"));
@@ -51,6 +73,7 @@ pub(crate) async fn get_expiry_configs(api: &crate::store::ECStore, bucket: &str
             lifecycle: None,
             object_lock: None,
             bucket_incarnation_id,
+            table_bucket_enabled,
         });
     }
     let object_lock = match metadata_sys::object_lock_config_state_from_authoritative_metadata(&metadata)? {
@@ -65,6 +88,7 @@ pub(crate) async fn get_expiry_configs(api: &crate::store::ECStore, bucket: &str
         lifecycle,
         object_lock,
         bucket_incarnation_id,
+        table_bucket_enabled,
     })
 }
 
@@ -125,6 +149,7 @@ mod tests {
         let lifecycle = lifecycle_config();
         metadata.lifecycle_config_xml = crate::bucket::utils::serialize(&lifecycle).unwrap();
         metadata.lifecycle_config = Some(lifecycle);
+        metadata.table_bucket_config_json = br#"{"enabled":true}"#.to_vec();
         metadata_sys::set_new_bucket_metadata_in(&store_a.ctx, metadata)
             .await
             .unwrap();
@@ -132,7 +157,14 @@ mod tests {
             .await
             .unwrap();
 
-        assert!(get_expiry_configs(&store_a, bucket).await.unwrap().lifecycle.is_some());
+        let configs = get_expiry_configs(&store_a, bucket).await.unwrap();
+        assert!(configs.lifecycle.is_some());
+        assert!(configs.table_bucket_enabled);
+        assert!(
+            !lifecycle_expiry_allowed(&store_a, bucket, configs.bucket_incarnation_id)
+                .await
+                .unwrap()
+        );
         assert!(get_expiry_configs(&store_b, bucket).await.unwrap().lifecycle.is_none());
     }
 }

crates/ecstore/src/bucket/metadata.rs

@@ -425,6 +425,15 @@ impl BucketMetadata {
         }
     }
 
+    /// Metadata for a physically new user bucket. Existing or fabricated legacy
+    /// metadata must use [`Self::new`] so upgrades do not rewrite their
+    /// durability posture.
+    pub fn new_with_default_durability(name: &str) -> Self {
+        let mut metadata = Self::new(name);
+        metadata.durability_config_json = super::durability::new_bucket_durability_config_json();
+        metadata
+    }
+
     pub fn save_file_path(&self) -> String {
         format!("{}/{}/{}", BUCKET_META_PREFIX, self.name.as_str(), BUCKET_METADATA_FILE)
     }
@@ -1302,7 +1311,7 @@ mod test {
         assert!(bm.object_locking(), "object lock active via parsed config");
     }
 
-    /// backlog#580: KNOWN GAP (weisd 2026-03-06 "inline_data 前缀不同"). RustFS's
+    /// backlog#580: KNOWN GAP (flagged 2026-03-06: "inline_data 前缀不同"). RustFS's
     /// inline-data extraction does not yet recover the object body from a
     /// MinIO-written bucket-metadata object: `into_fileinfo(read_data=true).data`
     /// returns bytes that are not the `.metadata.bin` blob (no `format|version`
@@ -1310,7 +1319,7 @@ mod test {
     /// inline-data framing is handled on the read path.
     /// backlog#580: prove RustFS reads a MinIO-written **inlined** bucket-metadata
     /// object end-to-end. MinIO stores inline data as `[bitrot hash][object body]`
-    /// (the "`inline_data` 前缀不同" that weisd flagged on 2026-03-06 is that
+    /// (the "`inline_data` 前缀不同" gap flagged on 2026-03-06 is that
     /// bitrot prefix, not a format incompatibility). Running the raw inline shard
     /// through RustFS's `BitrotReader` with the default `HighwayHash256S` must
     /// verify the checksum and yield the exact `.metadata.bin` blob.
@@ -1378,6 +1387,43 @@ mod test {
         assert_ne!(old.bucket_incarnation_id, new.bucket_incarnation_id);
     }
 
+    #[test]
+    fn regular_bucket_metadata_constructor_does_not_seed_durability() {
+        temp_env::with_var_unset(crate::bucket::durability::ENV_NEW_BUCKET_DURABILITY_MODE, || {
+            let metadata = BucketMetadata::new("legacy-or-fabricated");
+            assert!(metadata.durability_config_json.is_empty());
+            assert!(metadata.durability_config().is_none());
+        });
+    }
+
+    #[test]
+    fn new_bucket_metadata_constructor_seeds_default_durability() {
+        temp_env::with_var_unset(crate::bucket::durability::ENV_NEW_BUCKET_DURABILITY_MODE, || {
+            let metadata = BucketMetadata::new_with_default_durability("new-user-bucket");
+            assert_eq!(
+                metadata.durability_config().and_then(|cfg| cfg.normalized_mode()).as_deref(),
+                Some(crate::bucket::durability::BUCKET_DURABILITY_MODE_RELAXED)
+            );
+
+            let encoded = metadata.marshal_msg().expect("marshal metadata");
+            let decoded = BucketMetadata::unmarshal(&encoded).expect("unmarshal metadata");
+            assert_eq!(decoded.durability_config_json, metadata.durability_config_json);
+            assert_eq!(
+                decoded.durability_config().and_then(|cfg| cfg.normalized_mode()).as_deref(),
+                Some(crate::bucket::durability::BUCKET_DURABILITY_MODE_RELAXED)
+            );
+        });
+    }
+
+    #[test]
+    fn new_bucket_metadata_constructor_can_inherit_global_durability() {
+        temp_env::with_var(crate::bucket::durability::ENV_NEW_BUCKET_DURABILITY_MODE, Some("inherit"), || {
+            let metadata = BucketMetadata::new_with_default_durability("strict-fleet-new-bucket");
+            assert!(metadata.durability_config_json.is_empty());
+            assert!(metadata.durability_config().is_none());
+        });
+    }
+
     #[test]
     fn site_replication_config_updates_cannot_replace_bucket_incarnation() {
         let mut metadata = BucketMetadata::new("site-replication-update");

crates/ecstore/src/bucket/metadata_sys.rs

@@ -288,6 +288,13 @@ pub(crate) fn bucket_metadata_sys_of(ctx: &crate::runtime::instance::InstanceCon
     get_bucket_metadata_sys()
 }
 
+pub(crate) fn require_bucket_metadata_sys_in(
+    ctx: &crate::runtime::instance::InstanceContext,
+) -> Result<Arc<RwLock<BucketMetadataSys>>> {
+    ctx.bucket_metadata_sys()
+        .ok_or_else(|| Error::other("bucket metadata sys not initialized for this instance"))
+}
+
 pub(crate) async fn object_store_in(ctx: &crate::runtime::instance::InstanceContext) -> Result<Arc<ECStore>> {
     let sys = bucket_metadata_sys_of(ctx)?;
     Ok(sys.read().await.api.clone())
@@ -376,6 +383,15 @@ pub async fn update(bucket: &str, config_file: &str, data: Vec<u8>) -> Result<Of
     Box::pin(update_with_sys(get_bucket_metadata_sys()?, bucket, config_file, data)).await
 }
 
+pub(crate) async fn update_in(
+    ctx: &crate::runtime::instance::InstanceContext,
+    bucket: &str,
+    config_file: &str,
+    data: Vec<u8>,
+) -> Result<OffsetDateTime> {
+    Box::pin(update_with_sys(require_bucket_metadata_sys_in(ctx)?, bucket, config_file, data)).await
+}
+
 pub async fn delete(bucket: &str, config_file: &str) -> Result<OffsetDateTime> {
     delete_with_sys(get_bucket_metadata_sys()?, bucket, config_file).await
 }

crates/ecstore/src/bucket/object_lock/objectlock.rs

@@ -200,6 +200,29 @@ mod tests {
         assert!(retention.retain_until_date.is_some());
     }
 
+    /// backlog#1733 g-key-002: the persisted literal keys must still be read
+    /// through the current header constants, or WORM metadata fails open.
+    #[test]
+    fn persisted_compliance_lock_metadata_remains_effective() {
+        let mut meta = HashMap::new();
+        meta.insert("x-amz-object-lock-mode".to_string(), "COMPLIANCE".to_string());
+        meta.insert("x-amz-object-lock-retain-until-date".to_string(), "9999-01-01T00:00:00Z".to_string());
+        meta.insert("x-amz-object-lock-legal-hold".to_string(), "ON".to_string());
+
+        let retention = get_object_retention_meta(&meta);
+        assert_eq!(
+            retention.mode.as_ref().map(|mode| mode.as_str()),
+            Some(ObjectLockRetentionMode::COMPLIANCE)
+        );
+        assert!(retention.retain_until_date.is_some(), "persisted retention date must remain readable");
+
+        let legal_hold = get_object_legalhold_meta(&meta);
+        assert_eq!(
+            legal_hold.status.as_ref().map(|status| status.as_str()),
+            Some(ObjectLockLegalHoldStatus::ON)
+        );
+    }
+
     #[test]
     fn test_get_object_legalhold_meta_empty() {
         let meta = HashMap::new();

crates/ecstore/src/bucket/policy_sys.rs

@@ -14,6 +14,7 @@
 
 use super::metadata_sys::get_bucket_metadata_sys;
 use crate::error::{Result, StorageError};
+use crate::store::ECStore;
 use rustfs_policy::policy::{BucketPolicy, BucketPolicyArgs};
 
 pub struct PolicySys {}
@@ -27,6 +28,10 @@ impl PolicySys {
         Self::is_allowed_with_policy(args, Self::get(args.bucket).await).await
     }
 
+    pub async fn try_is_allowed_for_store(store: &ECStore, args: &BucketPolicyArgs<'_>) -> Result<bool> {
+        Self::is_allowed_with_policy(args, store.get_bucket_policy(args.bucket).await.map(|(policy, _)| policy)).await
+    }
+
     async fn is_allowed_with_policy(args: &BucketPolicyArgs<'_>, policy: Result<BucketPolicy>) -> Result<bool> {
         match policy {
             Ok(policy) => Ok(policy.is_allowed(args).await),

crates/ecstore/src/bucket/quota/checker.rs

@@ -198,11 +198,36 @@ impl QuotaChecker {
     }
 
     pub async fn get_real_time_usage(&self, bucket: &str) -> Result<u64, QuotaError> {
-        get_bucket_usage_memory(bucket)
-            .await
-            .ok_or_else(|| QuotaError::UsageUnavailable {
-                bucket: bucket.to_string(),
-            })
+        if let Some(usage) = get_bucket_usage_memory(bucket).await {
+            return Ok(usage);
+        }
+
+        // Degraded window (issue #5716): with no authoritative usage — most
+        // prominently after upgrading from a pre-v2 release, whose legacy
+        // `.usage.json` is demoted to non-authoritative until the scanner's
+        // first complete cycle persists `.usage.v2.json` — failing closed
+        // turned every write to a quota-enabled bucket into a retryable 503
+        // for the whole window. Quota admission instead degrades to the last
+        // persisted per-bucket size. That baseline is static between snapshot
+        // loads (live writes do not advance it), so hard-quota enforcement is
+        // advisory for the duration of the window: the overrun is bounded by
+        // the writes issued before the next complete scanner cycle. Buckets
+        // with no persisted baseline anywhere keep failing closed.
+        let store = self.metadata_sys.read().await.object_store();
+        // Box the fallback: it embeds the whole snapshot-load future, and every
+        // object write nests a quota check several futures deep, so keeping it
+        // inline would grow each write's state machine by the loader's full
+        // size — the debug-build 2MiB worker-stack overflow class fixed for
+        // bucket-config writes in #5648. The allocation only happens on the
+        // degraded path; the authoritative fast path returns above.
+        if let Some(baseline) = Box::pin(crate::data_usage::lookup_degraded_bucket_usage_baseline(store, bucket)).await {
+            debug!(bucket, baseline, "Bucket quota admission using degraded persisted usage baseline");
+            return Ok(baseline);
+        }
+
+        Err(QuotaError::UsageUnavailable {
+            bucket: bucket.to_string(),
+        })
     }
 }
 
@@ -232,6 +257,76 @@ mod tests {
         assert_eq!(result.quota_limit, None);
     }
 
+    /// Regression (issue #5716): an upgrade from a pre-v2 release leaves only
+    /// the legacy `.usage.json` snapshot, which has no completeness marker and
+    /// is demoted to non-authoritative, and the scanner's first complete cycle
+    /// can be a long way off. Quota admission must degrade to that persisted
+    /// baseline instead of failing every write to a quota-enabled bucket with
+    /// a retryable 503 for the whole window.
+    #[tokio::test]
+    #[serial]
+    async fn quota_admission_falls_back_to_legacy_snapshot_baseline() {
+        let (_dirs, ecstore) = isolated_store_over_temp_disks().await;
+        let sys = Arc::new(RwLock::new(BucketMetadataSys::new(ecstore.clone())));
+        let checker = QuotaChecker::new(sys);
+        let bucket = format!("quota-legacy-{}", Uuid::new_v4().simple());
+
+        let mut legacy = rustfs_data_usage::DataUsageInfo {
+            last_update: Some(std::time::SystemTime::now()),
+            buckets_count: 1,
+            ..Default::default()
+        };
+        legacy.buckets_usage.insert(
+            bucket.clone(),
+            rustfs_data_usage::BucketUsageInfo {
+                size: 1_234,
+                ..Default::default()
+            },
+        );
+        legacy.bucket_sizes.insert(bucket.clone(), 1_234);
+        // usage_snapshot_complete stays false: pre-v2 snapshots do not carry
+        // the field at all, so they always deserialize as incomplete.
+        let legacy_path = format!("{}/{}", crate::disk::BUCKET_META_PREFIX, rustfs_data_usage::LEGACY_DATA_USAGE_OBJECT_NAME);
+        crate::config::com::save_config(
+            ecstore.clone(),
+            &legacy_path,
+            serde_json::to_vec(&legacy).expect("legacy snapshot should encode"),
+        )
+        .await
+        .expect("legacy snapshot fixture should be stored");
+        crate::data_usage::invalidate_data_usage_snapshot_cache().await;
+
+        let usage = checker
+            .get_real_time_usage(&bucket)
+            .await
+            .expect("quota admission must degrade to the persisted legacy baseline");
+        assert_eq!(usage, 1_234);
+
+        // A bucket absent from every persisted snapshot still has no grounded
+        // baseline and must keep failing closed.
+        let unknown = format!("quota-unknown-{}", Uuid::new_v4().simple());
+        assert!(matches!(
+            checker.get_real_time_usage(&unknown).await,
+            Err(QuotaError::UsageUnavailable { .. })
+        ));
+
+        // Deleting the bucket's usage from the backend must purge the
+        // baseline: a recreated bucket may not inherit the dead incarnation's
+        // size, so with no persisted trace left it fails closed again.
+        crate::data_usage::remove_bucket_usage_from_backend(ecstore.clone(), &bucket)
+            .await
+            .expect("bucket usage removal should succeed");
+        assert!(matches!(
+            checker.get_real_time_usage(&bucket).await,
+            Err(QuotaError::UsageUnavailable { .. })
+        ));
+
+        crate::data_usage::prepare_bucket_usage_for_namespace_change(&bucket, None)
+            .await
+            .expect("test usage cache cleanup should succeed");
+        crate::data_usage::invalidate_data_usage_snapshot_cache().await;
+    }
+
     #[tokio::test]
     #[serial]
     async fn quota_usage_rejects_an_unknown_mutation_baseline() {

crates/ecstore/src/bucket/replication/README.md

@@ -100,11 +100,41 @@ paths.
     behind the ECStore replication facade; only `rustfs/src/app/storage_api.rs`
     may retain direct object/delete replication helper calls.
 
-## First Code-Bearing Step
+## Completion Criteria
 
-Start with `ReplicationRuntime` or `ReplicationEventSink`. Both can be added as
-narrow internal contracts while keeping current queue, MRF, resync, and target
-behavior unchanged. Do not start with a crate move.
+The split is complete when the "Current dependency to remove" column in the
+Required Contracts table above is empty: every row is either deleted because
+the dependency is gone, or reduced to "none". No other signal — file count,
+boundary count, line count — measures completion.
+
+Target end state:
+
+- `replication_pool.rs`, `replication_resyncer.rs`, and `replication_state.rs`
+  move into `crates/replication` behind the contracts above;
+- the `*_boundary.rs` and `*_bridge.rs` micro-files dissolve naturally as the
+  code they fence moves across the crate boundary. They are the mechanical
+  seams of the migration ratchet — the architecture guard scripts anchor on
+  their file names — so batch-merging them beforehand is explicitly rejected:
+  it forces synchronized guard-script/mod/import churn with zero functional
+  gain;
+- the only module that can retire early is `datatypes.rs`: delete it once its
+  facade consumers import the resync status enums through `rustfs-replication`
+  directly.
+
+## Milestones
+
+| Milestone | Scope | Status |
+|---|---|---|
+| M0 | Record the completion criteria and end state (this section). | Done |
+| M1 | Contract extraction: resync/queue/stats/object-decision/filemeta/storage wire contracts owned by `crates/replication`; ECStore imports concentrated in `*_boundary.rs`; event sink and runtime access behind local contracts. | Done — see Required Contracts |
+| M2 | Move resyncer pure decision logic (no IO) into `crates/replication`. | Pending; sequence after splitting the oversized resyncer/pool functions (`resync_bucket`, `replicate_all`, `start_mrf_processor`) so moves stay mechanical |
+| M3 | Move the worker runtime (`replication_pool.rs`, the IO paths of `replication_resyncer.rs`, `replication_state.rs`) once the contract traits are stable. Highest-risk step of the whole plan; do it last. | Pending |
+| M4 | Retire the boundary modules together with their guard-script entries; delete `datatypes.rs`. | Pending |
+
+The original first code-bearing step (narrow `ReplicationEventSink` /
+`ReplicationRuntime` contracts) has landed — `replication_event_sink.rs`
+exists and runtime access goes through local boundary aliases — so new work
+starts from M2.
 
 Current compatibility guard: `crates/ecstore/tests/replication_facade_compat_test.rs`
 keeps the ECStore replication facade types covered while architecture rules

crates/ecstore/src/bucket/replication/mod.rs

@@ -47,9 +47,9 @@ pub use datatypes::ResyncStatusType;
 pub use replication_config_boundary::{
     ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS,
     REPLICATION_CAPABILITY_CONTRACT_VERSION, REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS,
-    ReplicationConfigurationExt, ReplicationTargetValidationError, invalid_replication_config_status_field,
-    replication_target_arns, should_remove_replication_target, unsupported_replication_config_field,
-    validate_replication_config_target_arns,
+    ReplicationConfigStructureError, ReplicationConfigurationExt, ReplicationTargetValidationError,
+    invalid_replication_config_status_field, replication_target_arns, should_remove_replication_target,
+    unsupported_replication_config_field, validate_replication_config_structure, validate_replication_config_target_arns,
 };
 pub(crate) use replication_filemeta_boundary::version_purge_statuses_map;
 pub use replication_filemeta_boundary::{

crates/ecstore/src/bucket/replication/replication_config_boundary.rs

@@ -15,7 +15,7 @@
 pub use rustfs_replication::{
     ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS,
     REPLICATION_CAPABILITY_CONTRACT_VERSION, REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS,
-    ReplicationConfigurationExt, ReplicationRuleExt, ReplicationTargetValidationError, invalid_replication_config_status_field,
-    replication_target_arns, should_remove_replication_target, unsupported_replication_config_field,
-    validate_replication_config_target_arns,
+    ReplicationConfigStructureError, ReplicationConfigurationExt, ReplicationRuleExt, ReplicationTargetValidationError,
+    invalid_replication_config_status_field, replication_target_arns, should_remove_replication_target,
+    unsupported_replication_config_field, validate_replication_config_structure, validate_replication_config_target_arns,
 };

crates/ecstore/src/bucket/replication/replication_filemeta_boundary.rs

@@ -89,3 +89,116 @@ pub fn replication_state_to_filemeta(state: &ReplicationState) -> rustfs_filemet
         target_delete_marker_version_ids_corrupt: state.target_delete_marker_version_ids_corrupt,
     }
 }
+
+// Reconciliation tests for the deliberately duplicated wire types.
+//
+// `rustfs-filemeta` (xl.meta disk format) and `rustfs-replication` (MRF/resync
+// persistence format) each own a copy of `ReplicationStatusType`,
+// `VersionPurgeStatusType` and `ReplicationState`; the conversions above hop
+// between them via `as_str()`, whose `From<&str>` impls fall back to `Empty`
+// on any unknown token. That fallback silently degrades data the moment one
+// side gains a variant the other lacks, so these tests pin the two sides
+// together:
+//
+// - the `match` statements are exhaustive with no `_` arm — adding a variant
+//   on either side fails compilation here until the mapping is reconsidered;
+// - the round-trips assert the string token survives both directions — a
+//   variant whose token the other side does not recognize fails the assert
+//   instead of quietly becoming `Empty`.
+//
+// Struct-shaped drift on `ReplicationState` is already compile-guarded by the
+// exhaustive struct literals in the two conversion functions above; the
+// round-trip test below additionally pins value fidelity for every field.
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::collections::HashMap;
+
+    #[test]
+    fn replication_status_variants_round_trip_across_boundary() {
+        use rustfs_replication::ReplicationStatusType as Repl;
+
+        let all = [
+            Repl::Pending,
+            Repl::Completed,
+            Repl::CompletedLegacy,
+            Repl::Failed,
+            Repl::Replica,
+            Repl::Empty,
+        ];
+        for status in all {
+            // Exhaustive on the replication side: a new variant breaks this match.
+            match status {
+                Repl::Pending | Repl::Completed | Repl::CompletedLegacy | Repl::Failed | Repl::Replica | Repl::Empty => {}
+            }
+            let filemeta = replication_status_to_filemeta(status.clone());
+            assert_eq!(
+                filemeta.as_str(),
+                status.as_str(),
+                "replication->filemeta conversion must not degrade {status:?} (unknown tokens fall back to Empty)"
+            );
+            assert_eq!(replication_status_from_filemeta(filemeta), status);
+        }
+
+        // Exhaustive on the filemeta side: a new variant breaks this match.
+        fn _filemeta_side_is_covered(status: rustfs_filemeta::ReplicationStatusType) {
+            use rustfs_filemeta::ReplicationStatusType as Meta;
+            match status {
+                Meta::Pending | Meta::Completed | Meta::CompletedLegacy | Meta::Failed | Meta::Replica | Meta::Empty => {}
+            }
+        }
+    }
+
+    #[test]
+    fn version_purge_status_variants_round_trip_across_boundary() {
+        use rustfs_replication::VersionPurgeStatusType as Repl;
+
+        let all = [Repl::Pending, Repl::Complete, Repl::Failed, Repl::Empty];
+        for status in all {
+            // Exhaustive on the replication side: a new variant breaks this match.
+            match status {
+                Repl::Pending | Repl::Complete | Repl::Failed | Repl::Empty => {}
+            }
+            let filemeta = version_purge_status_to_filemeta(status.clone());
+            assert_eq!(
+                filemeta.as_str(),
+                status.as_str(),
+                "replication->filemeta conversion must not degrade {status:?} (unknown tokens fall back to Empty)"
+            );
+            assert_eq!(version_purge_status_from_filemeta(filemeta), status);
+        }
+
+        // Exhaustive on the filemeta side: a new variant breaks this match.
+        fn _filemeta_side_is_covered(status: rustfs_filemeta::VersionPurgeStatusType) {
+            use rustfs_filemeta::VersionPurgeStatusType as Meta;
+            match status {
+                Meta::Pending | Meta::Complete | Meta::Failed | Meta::Empty => {}
+            }
+        }
+    }
+
+    #[test]
+    fn replication_state_round_trips_every_field_across_boundary() {
+        let timestamp = time::OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("valid timestamp");
+        let state = ReplicationState {
+            replica_timestamp: Some(timestamp),
+            replica_status: ReplicationStatusType::Replica,
+            delete_marker: true,
+            replication_timestamp: Some(timestamp),
+            replication_status_internal: Some("arn:a=PENDING;".to_string()),
+            version_purge_status_internal: Some("arn:a=FAILED;".to_string()),
+            replicate_decision_str: "arn:a=true;false;;".to_string(),
+            targets: HashMap::from([
+                ("arn:a".to_string(), ReplicationStatusType::Completed),
+                ("arn:b".to_string(), ReplicationStatusType::Failed),
+            ]),
+            purge_targets: HashMap::from([("arn:a".to_string(), VersionPurgeStatusType::Pending)]),
+            reset_statuses_map: HashMap::from([("reset-arn:a".to_string(), "reset-id;ts".to_string())]),
+            target_delete_marker_version_ids: HashMap::from([("arn:a".to_string(), "version-1".to_string())]),
+            target_delete_marker_version_ids_corrupt: true,
+        };
+
+        let round_tripped = replication_state_from_filemeta(&replication_state_to_filemeta(&state));
+        assert_eq!(round_tripped, state);
+    }
+}

crates/ecstore/src/bucket/replication/replication_pool.rs

@@ -710,8 +710,8 @@ pub struct ReplicationPool<S: ReplicationStorage> {
     mrf_save_tx: Sender<MrfReplicateEntry>,
     mrf_save_rx: Mutex<Option<Receiver<MrfReplicateEntry>>>,
 
-    // Control channels
-    mrf_worker_kill_tx: Sender<()>,
+    // MRF worker lifecycle
+    mrf_worker_cancellations: Mutex<Vec<CancellationToken>>,
     mrf_stop_tx: Sender<()>,
 
     // Worker size tracking
@@ -734,7 +734,6 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
         // Create MRF channels
         let (mrf_replica_tx, mrf_replica_rx) = mpsc::channel(100000);
         let (mrf_save_tx, mrf_save_rx) = mpsc::channel(100000);
-        let (mrf_worker_kill_tx, _mrf_worker_kill_rx) = mpsc::channel(worker_counts.mrf_workers);
         let (mrf_stop_tx, _mrf_stop_rx) = mpsc::channel(1);
 
         let pool = Arc::new(Self {
@@ -752,7 +751,7 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
             mrf_replica_rx: Arc::new(Mutex::new(mrf_replica_rx)),
             mrf_save_tx,
             mrf_save_rx: Mutex::new(Some(mrf_save_rx)),
-            mrf_worker_kill_tx,
+            mrf_worker_cancellations: Mutex::new(Vec::with_capacity(worker_counts.mrf_workers)),
             mrf_stop_tx,
             mrf_worker_size: AtomicI32::new(0),
             task_handles: Mutex::new(Vec::new()),
@@ -896,12 +895,12 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
 
     /// Resizes the failed workers pool
     pub async fn resize_failed_workers(&self, n: i32) {
-        // Spawn workers up to n.  Each worker shares the receiver via Arc<Mutex<...>>.
-        // The mutex is held only while calling recv() — released before processing — so
-        // all workers process entries concurrently (the dequeue step is serialised but
-        // the replication I/O is not).
-        while self.mrf_worker_size.load(Ordering::SeqCst) < n {
-            self.mrf_worker_size.fetch_add(1, Ordering::SeqCst);
+        let target = mrf_worker_size_to_count(n);
+        let mut cancellations = self.mrf_worker_cancellations.lock().await;
+
+        while cancellations.len() < target {
+            let cancellation = CancellationToken::new();
+            cancellations.push(cancellation.clone());
 
             let active_counter = self.active_mrf_workers.clone();
             let stats = self.stats.clone();
@@ -910,7 +909,18 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
 
             let handle = tokio::spawn(async move {
                 loop {
-                    let operation = { mrf_rx.lock().await.recv().await };
+                    let operation = tokio::select! {
+                        biased;
+                        operation = async {
+                            let mut receiver = mrf_rx.lock().await;
+                            tokio::select! {
+                                biased;
+                                operation = receiver.recv() => operation,
+                                _ = cancellation.cancelled() => None,
+                            }
+                        } => operation,
+                        _ = cancellation.cancelled() => break,
+                    };
                     let Some(operation) = operation else { break };
 
                     let _active = ActiveWorkerGuard::new(active_counter.clone());
@@ -920,11 +930,13 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
             self.task_handles.lock().await.push(handle);
         }
 
-        // Remove workers if needed
-        while self.mrf_worker_size.load(Ordering::SeqCst) > n {
-            self.mrf_worker_size.fetch_sub(1, Ordering::SeqCst);
-            let _ = self.mrf_worker_kill_tx.try_send(());
+        while cancellations.len() > target {
+            if let Some(cancellation) = cancellations.pop() {
+                cancellation.cancel();
+            }
         }
+
+        self.mrf_worker_size.store(n.max(0), Ordering::SeqCst);
     }
 
     /// Resizes worker priority and counts
@@ -2555,6 +2567,12 @@ pub trait ReplicationPoolTrait: std::fmt::Debug {
     async fn queue_replica_task(&self, ri: ReplicateObjectInfo) -> ReplicationQueueAdmission;
     async fn queue_replica_delete_task(&self, ri: DeletedObjectReplicationInfo) -> ReplicationQueueAdmission;
     async fn queue_replica_delete_batch(&self, deletes: &[DeletedObjectReplicationInfo]) -> ReplicationBatchAdmission;
+    /// Persist one entry straight to the durable MRF journal, bypassing the
+    /// live worker queues. For failures whose source state is already gone —
+    /// e.g. exhausted delete-marker purges — where only a startup replay can
+    /// retry, and live re-dispatch would loop unboundedly against a down
+    /// target.
+    async fn persist_mrf_entry(&self, entry: MrfReplicateEntry) -> ReplicationQueueAdmission;
     async fn resize(&self, priority: ReplicationPriority, max_workers: usize, max_l_workers: usize);
     async fn get_bucket_resync_status(&self, bucket: &str) -> Result<BucketReplicationResyncStatus, EcstoreError>;
     async fn cancel_bucket_resync(&self, opts: ResyncOpts) -> Result<(), EcstoreError>;
@@ -2595,6 +2613,10 @@ impl<S: ReplicationStorage> ReplicationPoolTrait for ReplicationPool<S> {
         self.queue_replica_delete_batch(deletes).await
     }
 
+    async fn persist_mrf_entry(&self, entry: MrfReplicateEntry) -> ReplicationQueueAdmission {
+        self.queue_mrf_save_admission(entry, "delete_marker_purge").await
+    }
+
     async fn resize(&self, priority: ReplicationPriority, max_workers: usize, max_l_workers: usize) {
         self.resize(priority, max_workers, max_l_workers).await;
     }
@@ -3350,7 +3372,6 @@ mod tests {
     ) -> Arc<ReplicationPool<LoadResyncNodeStore>> {
         let (mrf_replica_tx, mrf_replica_rx) = mpsc::channel(1);
         let (mrf_save_tx, mrf_save_rx) = mpsc::channel(mrf_save_capacity);
-        let (mrf_worker_kill_tx, _) = mpsc::channel(1);
         let (mrf_stop_tx, _) = mpsc::channel(1);
 
         Arc::new(ReplicationPool {
@@ -3368,7 +3389,7 @@ mod tests {
             mrf_replica_rx: Arc::new(Mutex::new(mrf_replica_rx)),
             mrf_save_tx,
             mrf_save_rx: Mutex::new(Some(mrf_save_rx)),
-            mrf_worker_kill_tx,
+            mrf_worker_cancellations: Mutex::new(Vec::new()),
             mrf_stop_tx,
             mrf_worker_size: AtomicI32::new(0),
             task_handles: Mutex::new(Vec::new()),
@@ -3971,6 +3992,54 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    async fn resize_failed_workers_cancels_idle_workers() {
+        let shared = empty_resync_shared_state();
+        let pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("mrf-resize", shared))).await;
+
+        pool.resize_failed_workers(4).await;
+        assert_eq!(pool.mrf_worker_cancellations.lock().await.len(), 4);
+        assert_eq!(pool.mrf_worker_size.load(Ordering::SeqCst), 4);
+
+        pool.resize_failed_workers(1).await;
+
+        tokio::time::timeout(Duration::from_secs(10), async {
+            loop {
+                let finished = pool
+                    .task_handles
+                    .lock()
+                    .await
+                    .iter()
+                    .filter(|handle| handle.is_finished())
+                    .count();
+                if finished == 3 {
+                    break;
+                }
+                tokio::task::yield_now().await;
+            }
+        })
+        .await
+        .expect("canceled MRF workers should exit while the shared queue is idle");
+
+        assert_eq!(pool.mrf_worker_cancellations.lock().await.len(), 1);
+        assert_eq!(pool.mrf_worker_size.load(Ordering::SeqCst), 1);
+    }
+
+    #[tokio::test]
+    async fn resize_failed_workers_is_idempotent_across_growth_and_shrink() {
+        let shared = empty_resync_shared_state();
+        let pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("mrf-resize-repeat", shared))).await;
+
+        for target in [2, 4, 1, 4, 4] {
+            pool.resize_failed_workers(target).await;
+            assert_eq!(
+                pool.mrf_worker_cancellations.lock().await.len(),
+                usize::try_from(target).expect("test worker count should fit usize")
+            );
+            assert_eq!(pool.mrf_worker_size.load(Ordering::SeqCst), target);
+        }
+    }
+
     #[test]
     fn replicate_object_info_from_object_info_preserves_ssec_checksum() {
         let checksum = bytes::Bytes::from_static(b"ssec-checksum");

crates/ecstore/src/bucket/replication/replication_target_boundary.rs

@@ -28,7 +28,7 @@ use rustfs_utils::http::{
     AMZ_STORAGE_CLASS, AMZ_TAG_COUNT, CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE,
     HeaderExt as _, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP,
     SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_TAGGING_TIMESTAMP, get_str, insert_header_map,
-    is_internal_key,
+    is_internal_key, is_object_encryption_marker, is_replication_stripped_encryption_key, ssec_replication_transport_header,
 };
 use time::OffsetDateTime;
 use time::format_description::well_known::Rfc3339;
@@ -62,24 +62,6 @@ static STANDARD_HEADERS: &[&str] = &[
     AMZ_SERVER_SIDE_ENCRYPTION,
 ];
 
-static VALID_SSE_REPLICATION_HEADERS: &[(&str, &str)] = &[
-    (
-        "X-Rustfs-Internal-Server-Side-Encryption-Sealed-Key",
-        "X-Rustfs-Replication-Server-Side-Encryption-Sealed-Key",
-    ),
-    (
-        "X-Rustfs-Internal-Server-Side-Encryption-Seal-Algorithm",
-        "X-Rustfs-Replication-Server-Side-Encryption-Seal-Algorithm",
-    ),
-    (
-        "X-Rustfs-Internal-Server-Side-Encryption-Iv",
-        "X-Rustfs-Replication-Server-Side-Encryption-Iv",
-    ),
-    ("X-Rustfs-Internal-Encrypted-Multipart", "X-Rustfs-Replication-Encrypted-Multipart"),
-    ("X-Rustfs-Internal-Actual-Object-Size", "X-Rustfs-Replication-Actual-Object-Size"),
-];
-
-const ERR_REPLICATION_MANAGED_SSE_UNSUPPORTED: &str = "managed SSE replication requires target encryption support";
 const ERR_REPLICATION_ENCRYPTION_METADATA_UNSUPPORTED: &str = "replication source contains unsupported encryption metadata";
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -105,15 +87,29 @@ fn classify_replication_source_encryption(metadata: &HashMap<String, String>) ->
     let kms_context = metadata_value(metadata, AMZ_SERVER_SIDE_ENCRYPTION_KMS_CONTEXT);
 
     if is_ssec {
-        return if sse.is_some() || kms_key_id.is_some() || kms_context.is_some() {
-            ReplicationSourceEncryption::Unsupported
-        } else {
+        // Stored SSE-C objects always carry x-amz-server-side-encryption=AES256
+        // alongside the customer-algorithm key; only KMS evidence marks a
+        // mixed, unsupported state.
+        let sse_compatible = sse.map(str::trim).is_none_or(|value| value.eq_ignore_ascii_case("AES256"));
+        return if sse_compatible && kms_key_id.is_none() && kms_context.is_none() {
             ReplicationSourceEncryption::SseC
+        } else {
+            ReplicationSourceEncryption::Unsupported
         };
     }
 
     match sse.map(str::trim) {
-        None if kms_key_id.is_none() && kms_context.is_none() => ReplicationSourceEncryption::Plaintext,
+        None if kms_key_id.is_none() && kms_context.is_none() => {
+            // Sealed material without any recognizable SSE marker (e.g. an
+            // object written by MinIO, which does not persist the x-amz SSE
+            // intent header) must fail closed: replicating it as plaintext
+            // ships ciphertext the target can never decrypt.
+            if metadata.keys().any(|key| is_object_encryption_marker(key)) {
+                ReplicationSourceEncryption::Unsupported
+            } else {
+                ReplicationSourceEncryption::Plaintext
+            }
+        }
         Some(value) if value.eq_ignore_ascii_case("AES256") && kms_key_id.is_none() && kms_context.is_none() => {
             ReplicationSourceEncryption::SseS3
         }
@@ -163,28 +159,38 @@ pub(crate) fn replication_put_object_options(sc: &str, object_info: &ObjectInfo)
     let source_encryption = classify_replication_source_encryption(&object_info.user_defined);
     let is_ssec = matches!(source_encryption, ReplicationSourceEncryption::SseC);
 
-    match source_encryption {
-        ReplicationSourceEncryption::Plaintext | ReplicationSourceEncryption::SseC => {}
-        ReplicationSourceEncryption::SseS3 | ReplicationSourceEncryption::SseKms => {
-            return Err(Error::other(ERR_REPLICATION_MANAGED_SSE_UNSUPPORTED));
-        }
-        ReplicationSourceEncryption::Unsupported => {
-            return Err(Error::other(ERR_REPLICATION_ENCRYPTION_METADATA_UNSUPPORTED));
-        }
+    if matches!(source_encryption, ReplicationSourceEncryption::Unsupported) {
+        return Err(Error::other(ERR_REPLICATION_ENCRYPTION_METADATA_UNSUPPORTED));
     }
 
     for (key, value) in object_info.user_defined.iter() {
-        let has_valid_sse_header = valid_sse_replication_header(key).is_some();
-
-        if (!is_ssec || !has_valid_sse_header) && (is_internal_key(key) || is_standard_header(key)) {
+        if is_ssec && let Some(transport_header) = ssec_replication_transport_header(key) {
+            meta.insert(transport_header.to_string(), value.to_string());
             continue;
         }
 
-        if let Some(replication_header) = valid_sse_replication_header(key) {
-            meta.insert(replication_header.to_string(), value.to_string());
-        } else {
-            meta.insert(key.to_string(), value.to_string());
+        // Encryption metadata that is not remapped for SSE-C passthrough must
+        // never leave the source site: envelopes and intent headers are only
+        // meaningful to the source KMS.
+        if is_replication_stripped_encryption_key(key) {
+            continue;
         }
+
+        if is_internal_key(key) || is_standard_header(key) {
+            continue;
+        }
+
+        meta.insert(key.to_string(), value.to_string());
+    }
+
+    // Managed SSE replicates as plaintext (the replication reader decrypts via
+    // the object-encryption resolver) and re-encrypts on the target with the
+    // target's own KMS. Send only the encryption intent — never the source
+    // key id, whose meaning is local to the source site's KMS.
+    if matches!(source_encryption, ReplicationSourceEncryption::SseS3) {
+        meta.insert(AMZ_SERVER_SIDE_ENCRYPTION.to_string(), "AES256".to_string());
+    } else if matches!(source_encryption, ReplicationSourceEncryption::SseKms) {
+        meta.insert(AMZ_SERVER_SIDE_ENCRYPTION.to_string(), "aws:kms".to_string());
     }
 
     let mut is_multipart = object_info.is_multipart();
@@ -195,6 +201,11 @@ pub(crate) fn replication_put_object_options(sc: &str, object_info: &ObjectInfo)
         if is_ssec {
             let encoded = BASE64_STANDARD.encode(checksum_data);
             insert_header_map(&mut meta, SUFFIX_REPLICATION_SSEC_CRC, encoded);
+        } else if object_info.is_encrypted() {
+            // Encrypted checksums cannot be exposed as plaintext headers, and
+            // decrypt_checksums reports is_multipart=false for them (a value
+            // the response path relies on). Keep the object's own multipart
+            // flag so encrypted objects stay on the multipart route.
         } else {
             let (checksum_meta, is_mp) = object_info.decrypt_checksums(0, &HeaderMap::new())?;
             is_multipart = is_mp;
@@ -394,13 +405,22 @@ pub(crate) fn replication_force_delete_remove_options() -> RemoveObjectOptions {
     }
 }
 
-pub(crate) fn replication_complete_multipart_options(actual_size: String) -> PutObjectOptions {
+pub(crate) fn replication_complete_multipart_options(
+    actual_size: String,
+    source_etag: String,
+    source_mtime: Option<OffsetDateTime>,
+) -> PutObjectOptions {
     let mut user_metadata = HashMap::new();
     insert_header_map(&mut user_metadata, SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, actual_size);
 
     PutObjectOptions {
         user_metadata,
         internal: AdvancedPutOptions {
+            source_etag,
+            // AdvancedPutOptions::default() stamps now_utc(); an absent source
+            // mtime must degrade to epoch so header() suppresses the header
+            // instead of asserting the replication time as the object's mtime.
+            source_mtime: source_mtime.unwrap_or(OffsetDateTime::UNIX_EPOCH),
             replication_status: ReplicationStatusType::Replica,
             replication_request: true,
             ..Default::default()
@@ -413,20 +433,14 @@ fn is_standard_header(key: &str) -> bool {
     STANDARD_HEADERS.iter().any(|header| header.eq_ignore_ascii_case(key))
 }
 
-fn valid_sse_replication_header(key: &str) -> Option<&str> {
-    VALID_SSE_REPLICATION_HEADERS
-        .iter()
-        .find(|(internal, _)| key.eq_ignore_ascii_case(internal))
-        .map(|(_, replication)| *replication)
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;
     use aws_smithy_types::DateTime;
     use rustfs_replication::content_matches_by_etag;
     use rustfs_utils::http::{
-        SSEC_ALGORITHM_HEADER, SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC, get_header_map,
+        SSEC_ALGORITHM_HEADER, SSEC_KEY_MD5_HEADER, SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC,
+        get_header_map,
     };
     use std::sync::Arc;
     use time::Duration;
@@ -571,7 +585,21 @@ mod tests {
 
     #[test]
     fn replication_complete_multipart_options_sets_actual_size() {
-        let options = replication_complete_multipart_options("1024".to_string());
+        let source_mtime = OffsetDateTime::from_unix_timestamp(1_716_170_000).expect("valid test timestamp");
+        let options = replication_complete_multipart_options(
+            "1024".to_string(),
+            "0123456789abcdef0123456789abcdef-3".to_string(),
+            Some(source_mtime),
+        );
+        assert_eq!(options.internal.source_etag, "0123456789abcdef0123456789abcdef-3");
+        assert_eq!(options.internal.source_mtime, source_mtime);
+
+        // Absent source mtime must degrade to epoch (header suppressed), not
+        // the AdvancedPutOptions default of now_utc() — that default would
+        // stamp the replication time as the replica's mtime and break the
+        // multipart HEAD convergence.
+        let options_no_mtime = replication_complete_multipart_options("1024".to_string(), String::new(), None);
+        assert_eq!(options_no_mtime.internal.source_mtime.unix_timestamp(), 0);
 
         assert_eq!(
             get_header_map(&options.user_metadata, SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE).as_deref(),
@@ -583,11 +611,29 @@ mod tests {
 
     #[test]
     fn replication_put_options_filter_and_map_metadata() {
+        use rustfs_utils::http::object_encryption_keys::{
+            INTERNAL_ENCRYPTION_IV_HEADER, MINIO_INTERNAL_ENCRYPTION_ALGORITHM_HEADER, MINIO_INTERNAL_ENCRYPTION_IV_HEADER,
+            MINIO_INTERNAL_ENCRYPTION_MULTIPART_HEADER, MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER,
+            REPLICATION_ENCRYPTED_MULTIPART_HEADER, REPLICATION_ENCRYPTION_IV_HEADER, REPLICATION_SSE_IV_HEADER,
+            REPLICATION_SSE_SEAL_ALGORITHM_HEADER, REPLICATION_SSE_SEALED_KEY_HEADER, REPLICATION_SSEC_ALGORITHM_HEADER,
+            REPLICATION_SSEC_KEY_MD5_HEADER, REPLICATION_SSEC_ORIGINAL_SIZE_HEADER, SSEC_ORIGINAL_SIZE_HEADER,
+        };
+
+        // The stored shape of a real SSE-C object: SSE marker plus customer
+        // material, per encryption_material_to_metadata. Every transport-table
+        // source key is present so each mapping is pinned individually.
         let mut metadata = HashMap::new();
         metadata.insert(CONTENT_TYPE.to_string(), "text/plain".to_string());
         metadata.insert("x-user-meta".to_string(), "value".to_string());
+        metadata.insert(AMZ_SERVER_SIDE_ENCRYPTION.to_string(), "AES256".to_string());
         metadata.insert(SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string());
-        metadata.insert("X-Rustfs-Internal-Server-Side-Encryption-Sealed-Key".to_string(), "sealed".to_string());
+        metadata.insert(SSEC_KEY_MD5_HEADER.to_string(), "md5-value".to_string());
+        metadata.insert(SSEC_ORIGINAL_SIZE_HEADER.to_string(), "1024".to_string());
+        metadata.insert(INTERNAL_ENCRYPTION_IV_HEADER.to_string(), "iv-direct".to_string());
+        metadata.insert(MINIO_INTERNAL_ENCRYPTION_IV_HEADER.to_string(), "iv-minio".to_string());
+        metadata.insert(MINIO_INTERNAL_ENCRYPTION_ALGORITHM_HEADER.to_string(), "DAREv2-HMAC-SHA256".to_string());
+        metadata.insert(MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER.to_string(), "sealed".to_string());
+        metadata.insert(MINIO_INTERNAL_ENCRYPTION_MULTIPART_HEADER.to_string(), "true".to_string());
 
         let object_info = ObjectInfo {
             user_defined: Arc::new(metadata),
@@ -605,12 +651,40 @@ mod tests {
         assert!(!is_multipart);
         assert_eq!(options.user_metadata.get("x-user-meta"), Some(&"value".to_string()));
         assert!(!options.user_metadata.contains_key(CONTENT_TYPE));
+
+        // Every stored SSE-C material key is remapped onto its transport name.
+        assert_eq!(options.user_metadata.get(REPLICATION_SSEC_ALGORITHM_HEADER), Some(&"AES256".to_string()));
+        assert_eq!(options.user_metadata.get(REPLICATION_SSEC_KEY_MD5_HEADER), Some(&"md5-value".to_string()));
         assert_eq!(
-            options
-                .user_metadata
-                .get("X-Rustfs-Replication-Server-Side-Encryption-Sealed-Key"),
-            Some(&"sealed".to_string())
+            options.user_metadata.get(REPLICATION_SSEC_ORIGINAL_SIZE_HEADER),
+            Some(&"1024".to_string())
         );
+        assert_eq!(
+            options.user_metadata.get(REPLICATION_ENCRYPTION_IV_HEADER),
+            Some(&"iv-direct".to_string())
+        );
+        assert_eq!(options.user_metadata.get(REPLICATION_SSE_IV_HEADER), Some(&"iv-minio".to_string()));
+        assert_eq!(
+            options.user_metadata.get(REPLICATION_SSE_SEAL_ALGORITHM_HEADER),
+            Some(&"DAREv2-HMAC-SHA256".to_string())
+        );
+        assert_eq!(options.user_metadata.get(REPLICATION_SSE_SEALED_KEY_HEADER), Some(&"sealed".to_string()));
+        assert_eq!(
+            options.user_metadata.get(REPLICATION_ENCRYPTED_MULTIPART_HEADER),
+            Some(&"true".to_string())
+        );
+
+        // The stored keys themselves and the SSE intent header must not leave
+        // the source verbatim.
+        assert!(!options.user_metadata.contains_key(AMZ_SERVER_SIDE_ENCRYPTION));
+        assert!(!options.user_metadata.contains_key(SSEC_ALGORITHM_HEADER));
+        assert!(!options.user_metadata.contains_key(INTERNAL_ENCRYPTION_IV_HEADER));
+        assert!(
+            !options
+                .user_metadata
+                .contains_key(MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER)
+        );
+
         assert_eq!(options.content_type, "text/plain");
         assert_eq!(options.content_encoding, "gzip");
         assert_eq!(options.user_tags.get("env"), Some(&"prod".to_string()));
@@ -620,6 +694,68 @@ mod tests {
         assert!(options.internal.replication_request);
     }
 
+    #[test]
+    fn replication_put_options_strip_encryption_metadata_from_plaintext_objects() {
+        use rustfs_utils::http::object_encryption_keys::{INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER, SSEC_ORIGINAL_SIZE_HEADER};
+
+        // Migration leftovers: original-size metadata is not an encryption
+        // marker (older plaintext objects can retain it), so the object still
+        // classifies as plaintext — but the keys must be stripped, never
+        // forwarded as plain user metadata (backlog#1783 D2). The SSE-C
+        // original-size key is also a transport-table source key, so this
+        // doubles as the guard for the is_ssec gate: without SSE-C
+        // classification it must be stripped, not remapped.
+        let metadata = HashMap::from([
+            ("x-user-meta".to_string(), "value".to_string()),
+            (INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER.to_string(), "1024".to_string()),
+            (SSEC_ORIGINAL_SIZE_HEADER.to_string(), "1024".to_string()),
+        ]);
+        let object_info = ObjectInfo {
+            user_defined: Arc::new(metadata),
+            ..Default::default()
+        };
+
+        let (options, _) = replication_put_object_options("", &object_info).expect("build put options");
+
+        assert_eq!(options.user_metadata.get("x-user-meta"), Some(&"value".to_string()));
+        assert!(!options.user_metadata.contains_key(INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER));
+        assert!(!options.user_metadata.contains_key(SSEC_ORIGINAL_SIZE_HEADER));
+        assert!(
+            !options
+                .user_metadata
+                .keys()
+                .any(|key| key.to_ascii_lowercase().starts_with("x-rustfs-replication-")),
+            "non-SSE-C objects must never emit SSE replication transport keys"
+        );
+    }
+
+    #[test]
+    fn replication_put_options_fail_closed_on_sealed_material_without_sse_marker() {
+        use rustfs_utils::http::object_encryption_keys::{
+            INTERNAL_ENCRYPTION_KEY_HEADER, MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER,
+        };
+
+        // Sealed material without a recognizable SSE marker (MinIO-written
+        // objects, or corrupted metadata) must fail closed instead of
+        // replicating ciphertext as a plaintext object.
+        for sealed_key in [
+            INTERNAL_ENCRYPTION_KEY_HEADER,
+            MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER,
+        ] {
+            let object_info = ObjectInfo {
+                user_defined: Arc::new(HashMap::from([(sealed_key.to_string(), "sealed-envelope".to_string())])),
+                ..Default::default()
+            };
+
+            let err = match replication_put_object_options("", &object_info) {
+                Ok(_) => panic!("sealed material without an SSE marker must fail closed ({sealed_key})"),
+                Err(err) => err,
+            };
+            assert!(err.to_string().contains(ERR_REPLICATION_ENCRYPTION_METADATA_UNSUPPORTED));
+            assert!(!err.to_string().contains("sealed-envelope"));
+        }
+    }
+
     #[test]
     fn replication_put_options_adds_ssec_checksum_metadata() {
         let metadata = HashMap::from([(SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string())]);
@@ -658,6 +794,30 @@ mod tests {
             classify_replication_source_encryption(&HashMap::from([(SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string())])),
             ReplicationSourceEncryption::SseC
         );
+        // Real stored SSE-C objects carry the AES256 SSE marker alongside the
+        // customer algorithm (encryption_material_to_metadata writes both).
+        assert_eq!(
+            classify_replication_source_encryption(&HashMap::from([
+                (SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string()),
+                ("x-amz-server-side-encryption".to_string(), "AES256".to_string()),
+            ])),
+            ReplicationSourceEncryption::SseC
+        );
+        // SSE-C material mixed with KMS evidence stays unsupported.
+        assert_eq!(
+            classify_replication_source_encryption(&HashMap::from([
+                (SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string()),
+                (AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID.to_string(), "key-1".to_string()),
+            ])),
+            ReplicationSourceEncryption::Unsupported
+        );
+        assert_eq!(
+            classify_replication_source_encryption(&HashMap::from([
+                (SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string()),
+                ("x-amz-server-side-encryption".to_string(), "aws:kms".to_string()),
+            ])),
+            ReplicationSourceEncryption::Unsupported
+        );
         assert_eq!(
             classify_replication_source_encryption(&HashMap::from([(
                 "x-amz-server-side-encryption".to_string(),
@@ -675,36 +835,75 @@ mod tests {
     }
 
     #[test]
-    fn replication_put_options_rejects_sse_s3_until_target_encryption_is_supported() {
-        let object_info = ObjectInfo {
-            user_defined: Arc::new(HashMap::from([(AMZ_SERVER_SIDE_ENCRYPTION.to_string(), "AES256".to_string())])),
-            ..Default::default()
+    fn replication_put_options_sends_sse_s3_intent_without_source_material() {
+        use rustfs_utils::http::object_encryption_keys::{
+            INTERNAL_ENCRYPTION_ALGORITHM_HEADER, INTERNAL_ENCRYPTION_IV_HEADER, INTERNAL_ENCRYPTION_KEY_HEADER,
+            INTERNAL_ENCRYPTION_KEY_ID_HEADER, INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER,
         };
 
-        let err = match replication_put_object_options("", &object_info) {
-            Ok(_) => panic!("SSE-S3 replication should fail closed until target encryption headers are supported"),
-            Err(err) => err,
-        };
-
-        assert!(err.to_string().contains(ERR_REPLICATION_MANAGED_SSE_UNSUPPORTED));
-    }
-
-    #[test]
-    fn replication_put_options_rejects_sse_kms_until_target_encryption_is_supported() {
+        // The stored shape of a managed SSE-S3 object per
+        // encryption_material_to_metadata: SSE marker plus envelope material.
         let object_info = ObjectInfo {
             user_defined: Arc::new(HashMap::from([
-                (AMZ_SERVER_SIDE_ENCRYPTION.to_string(), "aws:kms".to_string()),
-                (AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID.to_string(), "key-1".to_string()),
+                (AMZ_SERVER_SIDE_ENCRYPTION.to_string(), "AES256".to_string()),
+                (INTERNAL_ENCRYPTION_KEY_ID_HEADER.to_string(), "default".to_string()),
+                (INTERNAL_ENCRYPTION_KEY_HEADER.to_string(), "sealed-envelope".to_string()),
+                (INTERNAL_ENCRYPTION_IV_HEADER.to_string(), "iv".to_string()),
+                (INTERNAL_ENCRYPTION_ALGORITHM_HEADER.to_string(), "AES256-GCM".to_string()),
+                (INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER.to_string(), "1024".to_string()),
+                ("x-user-meta".to_string(), "value".to_string()),
             ])),
             ..Default::default()
         };
 
-        let err = match replication_put_object_options("", &object_info) {
-            Ok(_) => panic!("SSE-KMS replication should fail closed until target encryption headers are supported"),
-            Err(err) => err,
+        let (options, _) = replication_put_object_options("", &object_info).expect("managed SSE-S3 must build put options");
+
+        assert_eq!(options.user_metadata.get(AMZ_SERVER_SIDE_ENCRYPTION), Some(&"AES256".to_string()));
+        assert_eq!(options.user_metadata.get("x-user-meta"), Some(&"value".to_string()));
+        // No envelope material and no key id may leave the source.
+        assert!(!options.user_metadata.contains_key(INTERNAL_ENCRYPTION_KEY_HEADER));
+        assert!(!options.user_metadata.contains_key(INTERNAL_ENCRYPTION_KEY_ID_HEADER));
+        assert!(!options.user_metadata.contains_key(INTERNAL_ENCRYPTION_IV_HEADER));
+        assert!(
+            !options.user_metadata.values().any(|value| value.contains("sealed-envelope")),
+            "source envelope material must never leave the source site"
+        );
+    }
+
+    #[test]
+    fn replication_put_options_sends_sse_kms_intent_without_source_key_id() {
+        use rustfs_utils::http::object_encryption_keys::{
+            INTERNAL_ENCRYPTION_KEY_HEADER, MINIO_INTERNAL_ENCRYPTION_KMS_CONTEXT_HEADER,
         };
 
-        assert!(err.to_string().contains(ERR_REPLICATION_MANAGED_SSE_UNSUPPORTED));
+        let object_info = ObjectInfo {
+            user_defined: Arc::new(HashMap::from([
+                (AMZ_SERVER_SIDE_ENCRYPTION.to_string(), "aws:kms".to_string()),
+                (AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID.to_string(), "source-key-1".to_string()),
+                (INTERNAL_ENCRYPTION_KEY_HEADER.to_string(), "sealed-envelope".to_string()),
+                (MINIO_INTERNAL_ENCRYPTION_KMS_CONTEXT_HEADER.to_string(), "ctx".to_string()),
+            ])),
+            ..Default::default()
+        };
+
+        let (options, _) = replication_put_object_options("", &object_info).expect("managed SSE-KMS must build put options");
+
+        // Intent only: the target encrypts with its own default KMS key.
+        assert_eq!(options.user_metadata.get(AMZ_SERVER_SIDE_ENCRYPTION), Some(&"aws:kms".to_string()));
+        assert!(!options.user_metadata.contains_key(AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID));
+        assert!(!options.user_metadata.contains_key(INTERNAL_ENCRYPTION_KEY_HEADER));
+        assert!(
+            !options
+                .user_metadata
+                .contains_key(MINIO_INTERNAL_ENCRYPTION_KMS_CONTEXT_HEADER)
+        );
+        assert!(
+            !options
+                .user_metadata
+                .values()
+                .any(|value| value.contains("sealed-envelope") || value.contains("source-key-1")),
+            "source KMS identifiers and envelopes must never leave the source site"
+        );
     }
 
     #[test]

crates/ecstore/src/bucket/target/arn.rs

@@ -56,11 +56,59 @@ impl FromStr for ARN {
         if parts.len() != 6 {
             return Err(std::io::Error::new(std::io::ErrorKind::InvalidInput, "Invalid ARN format"));
         }
+        // Display emits `arn:rustfs:{type}:{region}:{id}:{bucket}`; read the
+        // segments back in the same order so parse(display(a)) == a.
         Ok(ARN {
             arn_type: BucketTargetType::from_str(parts[2]).unwrap_or_default(),
-            id: parts[3].to_string(),
-            region: parts[4].to_string(),
+            region: parts[3].to_string(),
+            id: parts[4].to_string(),
             bucket: parts[5].to_string(),
         })
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// Display emits `arn:rustfs:{type}:{region}:{id}:{bucket}` (madmin layout);
+    /// FromStr must read the same positions back so parse(display(a)) == a.
+    #[test]
+    fn from_str_round_trips_display_with_region_and_id() {
+        let arn = ARN::new(
+            BucketTargetType::ReplicationService,
+            "depl-123".to_string(),
+            "us-east-1".to_string(),
+            "bucket-a".to_string(),
+        );
+
+        let parsed = ARN::from_str(&arn.to_string()).expect("display output must parse");
+
+        assert_eq!(parsed.arn_type, arn.arn_type);
+        assert_eq!(parsed.region, arn.region, "region must survive display->parse round-trip");
+        assert_eq!(parsed.id, arn.id, "id must survive display->parse round-trip");
+        assert_eq!(parsed.bucket, arn.bucket);
+    }
+
+    #[test]
+    fn from_str_reads_region_then_id_in_display_order() {
+        let parsed = ARN::from_str("arn:rustfs:replication:us-east-1:depl-123:bucket-a").expect("valid ARN must parse");
+
+        assert_eq!(parsed.arn_type, BucketTargetType::ReplicationService);
+        assert_eq!(parsed.region, "us-east-1");
+        assert_eq!(parsed.id, "depl-123");
+        assert_eq!(parsed.bucket, "bucket-a");
+    }
+
+    /// RustFS commonly generates ARNs with an empty region:
+    /// `arn:rustfs:replication::<deployment_id>:<bucket>`.
+    #[test]
+    fn from_str_handles_empty_region_segment() {
+        let parsed = ARN::from_str("arn:rustfs:replication::depl-123:bucket-a").expect("valid ARN must parse");
+
+        assert_eq!(parsed.arn_type, BucketTargetType::ReplicationService);
+        assert_eq!(parsed.region, "", "region segment is empty in this form");
+        assert_eq!(parsed.id, "depl-123");
+        assert_eq!(parsed.bucket, "bucket-a");
+    }
+}

crates/ecstore/src/bucket/target/bucket_target.rs

@@ -13,6 +13,7 @@
 // limitations under the License.
 
 use crate::error::{Error, Result};
+use jiff::Timestamp;
 use rmp_serde::Serializer as rmpSerializer;
 use serde::{Deserialize, Serialize};
 use std::{
@@ -32,7 +33,7 @@ pub struct Credentials {
     #[serde(rename = "secretKey")]
     pub secret_key: String,
     pub session_token: Option<String>,
-    pub expiration: Option<chrono::DateTime<chrono::Utc>>,
+    pub expiration: Option<Timestamp>,
 }
 
 impl Credentials {
@@ -93,6 +94,21 @@ mod duration_milliseconds {
     }
 }
 
+/// Defensive decode for the two integer wire encodings of these duration
+/// fields: RustFS persists (and legacy RustFS clients sent) plain seconds,
+/// while Go `time.Duration` JSON — madmin/mc requests and MinIO-written
+/// bucket-targets metadata — is nanoseconds. No meaningful interval lies
+/// between 10^7 seconds (~115 days) and 10^7 nanoseconds (10ms), so the
+/// magnitude disambiguates the unit.
+pub fn duration_from_secs_or_nanos(value: u64) -> Duration {
+    const NANOS_THRESHOLD: u64 = 10_000_000;
+    if value < NANOS_THRESHOLD {
+        Duration::from_secs(value)
+    } else {
+        Duration::from_nanos(value)
+    }
+}
+
 mod duration_seconds {
     use serde::{Deserialize, Deserializer, Serializer};
     use std::time::Duration;
@@ -108,8 +124,8 @@ mod duration_seconds {
     where
         D: Deserializer<'de>,
     {
-        let secs = u64::deserialize(deserializer)?;
-        Ok(Duration::from_secs(secs))
+        let value = u64::deserialize(deserializer)?;
+        Ok(super::duration_from_secs_or_nanos(value))
     }
 }
 
@@ -408,7 +424,11 @@ mod tests {
         assert_eq!(credentials.access_key, "test-access-key");
         assert_eq!(credentials.secret_key, "test-secret-key");
         assert_eq!(credentials.session_token, Some("test-session-token".to_string()));
-        assert!(credentials.expiration.is_some());
+        assert_eq!(
+            serde_json::to_value(credentials.expiration.expect("expiration should parse"))
+                .expect("expiration should serialize to JSON"),
+            serde_json::json!("2024-12-31T23:59:59Z")
+        );
 
         // Verify latency statistics
         assert_eq!(target.latency.curr, Duration::from_millis(100));
@@ -484,6 +504,29 @@ mod tests {
         assert_eq!(original.offline_count, deserialized.offline_count);
     }
 
+    #[test]
+    fn bucket_target_reads_go_nanosecond_durations_defensively() {
+        // MinIO-written bucket-targets metadata and madmin clients encode
+        // these fields as Go `time.Duration` nanoseconds; RustFS has always
+        // persisted seconds. Both encodings must decode to the same interval.
+        let target: BucketTarget = serde_json::from_value(serde_json::json!({
+            "endpoint": "localhost:9000",
+            "targetbucket": "target",
+            "type": "replication",
+            "healthCheckDuration": 60_000_000_000u64,
+            "totalDowntime": 90_000_000_000u64
+        }))
+        .expect("nanosecond durations should deserialize");
+
+        assert_eq!(target.health_check_duration, Duration::from_secs(60));
+        assert_eq!(target.total_downtime, Duration::from_secs(90));
+
+        // The persisted wire format stays seconds for existing RustFS readers.
+        let value = serde_json::to_value(&target).expect("target should serialize");
+        assert_eq!(value["healthCheckDuration"], 60);
+        assert_eq!(value["totalDowntime"], 90);
+    }
+
     #[test]
     fn test_bucket_target_debug_redacts_credentials() {
         let target = BucketTarget {
@@ -562,12 +605,15 @@ mod tests {
                 .and_then(|credentials| credentials.session_token.as_deref()),
             Some("legacy-session-token")
         );
-        assert!(
+        assert_eq!(
             target
                 .credentials
                 .as_ref()
                 .and_then(|credentials| credentials.expiration)
-                .is_some()
+                .map(serde_json::to_value)
+                .transpose()
+                .expect("expiration should serialize to JSON"),
+            Some(serde_json::json!("2024-12-31T23:59:59Z"))
         );
     }
 
@@ -609,7 +655,11 @@ mod tests {
             credentials.session_token,
             Some("AQoEXAMPLEH4aoAH0gNCAPyJxz4BlCFFxWNE1OPTgk5TthT".to_string())
         );
-        assert!(credentials.expiration.is_some());
+        assert_eq!(
+            serde_json::to_value(credentials.expiration.expect("expiration should parse"))
+                .expect("expiration should serialize to JSON"),
+            serde_json::json!("2024-12-31T23:59:59Z")
+        );
     }
 
     #[test]

crates/ecstore/src/bucket/utils.rs

@@ -269,6 +269,32 @@ pub fn check_del_obj_args(bucket: &str, object: &str) -> Result<()> {
     check_bucket_and_object_names(bucket, object)
 }
 
+/// Filesystem `NAME_MAX`: every object-key path segment becomes one on-disk
+/// directory entry, so a longer segment can never be stored and previously
+/// escaped as an `ENAMETOOLONG` io error → `InternalError` 500 (rustfs#5785).
+const MAX_OBJECT_KEY_SEGMENT_BYTES: usize = 255;
+
+/// Reject object keys whose on-disk directory names would exceed `NAME_MAX`.
+///
+/// Middle segments map to their raw bytes; the final segment of a
+/// directory-object key (trailing `/`) is stored with the `__XLDIR__` suffix
+/// appended, shrinking its budget accordingly.
+fn object_key_segments_fit_on_disk(object: &str) -> bool {
+    let trailing_dir = object.ends_with('/');
+    let segments: Vec<&str> = object.split('/').collect();
+    let last_nonempty = segments.iter().rposition(|s| !s.is_empty());
+    for (index, segment) in segments.iter().enumerate() {
+        let mut budget = MAX_OBJECT_KEY_SEGMENT_BYTES;
+        if trailing_dir && Some(index) == last_nonempty {
+            budget = budget.saturating_sub(rustfs_utils::path::GLOBAL_DIR_SUFFIX.len());
+        }
+        if segment.len() > budget {
+            return false;
+        }
+    }
+    true
+}
+
 pub fn check_bucket_and_object_names(bucket: &str, object: &str) -> Result<()> {
     if !is_meta_bucketname(bucket) && check_valid_bucket_name_strict(bucket).is_err() {
         return Err(StorageError::BucketNameInvalid(bucket.to_string()));
@@ -282,6 +308,10 @@ pub fn check_bucket_and_object_names(bucket: &str, object: &str) -> Result<()> {
         return Err(StorageError::ObjectNameInvalid(bucket.to_string(), object.to_string()));
     }
 
+    if !object_key_segments_fit_on_disk(object) {
+        return Err(StorageError::ObjectNameInvalid(bucket.to_string(), object.to_string()));
+    }
+
     // if cfg!(target_os = "windows") && object.contains('\\') {
     //     return Err(StorageError::ObjectNameInvalid(bucket.to_string(), object.to_string()));
     // }
@@ -379,6 +409,14 @@ pub fn check_put_object_args(bucket: &str, object: &str) -> Result<()> {
         return Err(StorageError::ObjectNameInvalid(bucket.to_string(), object.to_string()));
     }
 
+    // The write path validates arguments here rather than through
+    // check_bucket_and_object_names, so the on-disk segment budget has to be
+    // enforced in both places or an over-NAME_MAX key still reaches the disk
+    // layer and escapes as ENAMETOOLONG → InternalError 500 (rustfs#5785).
+    if !object_key_segments_fit_on_disk(object) {
+        return Err(StorageError::ObjectNameInvalid(bucket.to_string(), object.to_string()));
+    }
+
     Ok(())
 }
 
@@ -387,6 +425,62 @@ mod tests {
     use super::*;
     use proptest::prelude::*;
 
+    /// rustfs#5785: keys whose path segments exceed the on-disk NAME_MAX
+    /// budget must be rejected up front as ObjectNameInvalid (4xx), not leak
+    /// ENAMETOOLONG as InternalError 500 from the disk layer.
+    #[test]
+    fn object_key_segment_name_max_budget() {
+        // 255-byte single segment: exactly at the on-disk limit.
+        assert!(check_bucket_and_object_names("bucket", &"a".repeat(255)).is_ok());
+        // 256 bytes: one over.
+        assert!(matches!(
+            check_bucket_and_object_names("bucket", &"a".repeat(256)),
+            Err(StorageError::ObjectNameInvalid(_, _))
+        ));
+        // Long keys are fine as long as every segment fits.
+        let segmented = ["b".repeat(200), "c".repeat(200), "d".repeat(200)].join("/");
+        assert!(check_bucket_and_object_names("bucket", &segmented).is_ok());
+        // The budget counts bytes, not characters (100 CJK chars = 300 bytes).
+        assert!(matches!(
+            check_bucket_and_object_names("bucket", &"中".repeat(100)),
+            Err(StorageError::ObjectNameInvalid(_, _))
+        ));
+        assert!(check_bucket_and_object_names("bucket", &"中".repeat(85)).is_ok());
+        // Directory-object keys spend GLOBAL_DIR_SUFFIX bytes of the final
+        // segment's budget on the on-disk __XLDIR__ encoding.
+        let dir_budget = 255 - rustfs_utils::path::GLOBAL_DIR_SUFFIX.len();
+        assert!(check_bucket_and_object_names("bucket", &format!("{}/", "e".repeat(dir_budget))).is_ok());
+        assert!(matches!(
+            check_bucket_and_object_names("bucket", &format!("{}/", "e".repeat(dir_budget + 1))),
+            Err(StorageError::ObjectNameInvalid(_, _))
+        ));
+    }
+
+    /// rustfs#5785 follow-up: the write path validates through
+    /// check_put_object_args, not check_bucket_and_object_names, so the same
+    /// budget has to hold there — otherwise an over-NAME_MAX PUT still reached
+    /// the disk layer and came back as InternalError 500.
+    #[test]
+    fn put_object_args_enforce_the_same_segment_budget() {
+        assert!(check_put_object_args("bucket", &"a".repeat(255)).is_ok());
+        assert!(matches!(
+            check_put_object_args("bucket", &"a".repeat(256)),
+            Err(StorageError::ObjectNameInvalid(_, _))
+        ));
+        assert!(matches!(
+            check_put_object_args("bucket", &"\u{4e2d}".repeat(100)),
+            Err(StorageError::ObjectNameInvalid(_, _))
+        ));
+        let segmented = ["b".repeat(200), "c".repeat(200), "d".repeat(200)].join("/");
+        assert!(check_put_object_args("bucket", &segmented).is_ok());
+        let dir_budget = 255 - rustfs_utils::path::GLOBAL_DIR_SUFFIX.len();
+        assert!(check_put_object_args("bucket", &format!("{}/", "e".repeat(dir_budget))).is_ok());
+        assert!(matches!(
+            check_put_object_args("bucket", &format!("{}/", "e".repeat(dir_budget + 1))),
+            Err(StorageError::ObjectNameInvalid(_, _))
+        ));
+    }
+
     // Test validation functions
     #[test]
     fn test_is_valid_object_name() {

crates/ecstore/src/client/api_bucket_policy.rs

@@ -1,171 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-#![allow(unused_imports)]
-#![allow(unused_variables)]
-#![allow(unused_mut)]
-#![allow(unused_assignments)]
-#![allow(unused_must_use)]
-#![allow(clippy::all)]
-
-use http::{HeaderMap, StatusCode};
-use http_body_util::BodyExt;
-use hyper::body::Body;
-use hyper::body::Bytes;
-use std::collections::HashMap;
-
-use crate::client::{
-    api_error_response::http_resp_to_error_response,
-    transition_api::{ReaderImpl, RequestMetadata, TransitionClient},
-};
-use rustfs_utils::hash::EMPTY_STRING_SHA256_HASH;
-
-impl TransitionClient {
-    pub async fn set_bucket_policy(&self, bucket_name: &str, policy: &str) -> Result<(), std::io::Error> {
-        if policy == "" {
-            return self.remove_bucket_policy(bucket_name).await;
-        }
-
-        self.put_bucket_policy(bucket_name, policy).await
-    }
-
-    pub async fn put_bucket_policy(&self, bucket_name: &str, policy: &str) -> Result<(), std::io::Error> {
-        let mut url_values = HashMap::new();
-        url_values.insert("policy".to_string(), "".to_string());
-
-        let mut req_metadata = RequestMetadata {
-            bucket_name: bucket_name.to_string(),
-            query_values: url_values,
-            content_body: ReaderImpl::Body(Bytes::from(policy.as_bytes().to_vec())),
-            content_length: policy.len() as i64,
-            object_name: "".to_string(),
-            custom_header: HeaderMap::new(),
-            content_md5_base64: "".to_string(),
-            content_sha256_hex: "".to_string(),
-            stream_sha256: false,
-            trailer: HeaderMap::new(),
-            pre_sign_url: Default::default(),
-            add_crc: Default::default(),
-            extra_pre_sign_header: Default::default(),
-            bucket_location: Default::default(),
-            expires: Default::default(),
-        };
-
-        let resp = self.execute_method(http::Method::PUT, &mut req_metadata).await?;
-        //defer closeResponse(resp)
-
-        let resp_status = resp.status();
-        let h = resp.headers().clone();
-
-        //if resp != nil {
-        if resp_status != StatusCode::NO_CONTENT && resp.status() != StatusCode::OK {
-            return Err(std::io::Error::other(http_resp_to_error_response(
-                resp_status,
-                &h,
-                vec![],
-                bucket_name,
-                "",
-            )));
-        }
-        //}
-        Ok(())
-    }
-
-    pub async fn remove_bucket_policy(&self, bucket_name: &str) -> Result<(), std::io::Error> {
-        let mut url_values = HashMap::new();
-        url_values.insert("policy".to_string(), "".to_string());
-
-        let resp = self
-            .execute_method(
-                http::Method::DELETE,
-                &mut RequestMetadata {
-                    bucket_name: bucket_name.to_string(),
-                    query_values: url_values,
-                    content_sha256_hex: EMPTY_STRING_SHA256_HASH.to_string(),
-                    object_name: "".to_string(),
-                    custom_header: HeaderMap::new(),
-                    content_body: ReaderImpl::Body(Bytes::new()),
-                    content_length: 0,
-                    content_md5_base64: "".to_string(),
-                    stream_sha256: false,
-                    trailer: HeaderMap::new(),
-                    pre_sign_url: Default::default(),
-                    add_crc: Default::default(),
-                    extra_pre_sign_header: Default::default(),
-                    bucket_location: Default::default(),
-                    expires: Default::default(),
-                },
-            )
-            .await?;
-        //defer closeResponse(resp)
-
-        let resp_status = resp.status();
-        let h = resp.headers().clone();
-
-        if resp_status != StatusCode::NO_CONTENT {
-            return Err(std::io::Error::other(http_resp_to_error_response(
-                resp_status,
-                &h,
-                vec![],
-                bucket_name,
-                "",
-            )));
-        }
-
-        Ok(())
-    }
-
-    pub async fn get_bucket_policy(&self, bucket_name: &str) -> Result<String, std::io::Error> {
-        let bucket_policy = self.get_bucket_policy_inner(bucket_name).await?;
-        Ok(bucket_policy)
-    }
-
-    pub async fn get_bucket_policy_inner(&self, bucket_name: &str) -> Result<String, std::io::Error> {
-        let mut url_values = HashMap::new();
-        url_values.insert("policy".to_string(), "".to_string());
-
-        let resp = self
-            .execute_method(
-                http::Method::GET,
-                &mut RequestMetadata {
-                    bucket_name: bucket_name.to_string(),
-                    query_values: url_values,
-                    content_sha256_hex: EMPTY_STRING_SHA256_HASH.to_string(),
-                    object_name: "".to_string(),
-                    custom_header: HeaderMap::new(),
-                    content_body: ReaderImpl::Body(Bytes::new()),
-                    content_length: 0,
-                    content_md5_base64: "".to_string(),
-                    stream_sha256: false,
-                    trailer: HeaderMap::new(),
-                    pre_sign_url: Default::default(),
-                    add_crc: Default::default(),
-                    extra_pre_sign_header: Default::default(),
-                    bucket_location: Default::default(),
-                    expires: Default::default(),
-                },
-            )
-            .await?;
-
-        let mut body_vec = Vec::new();
-        let mut body = resp.into_body();
-        while let Some(frame) = body.frame().await {
-            let frame = frame.map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e.to_string()))?;
-            if let Some(data) = frame.data_ref() {
-                body_vec.extend_from_slice(data);
-            }
-        }
-        let policy = String::from_utf8_lossy(&body_vec).to_string();
-        Ok(policy)
-    }
-}

crates/ecstore/src/client/api_get_object_acl.rs

@@ -1,199 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-#![allow(unused_imports)]
-#![allow(unused_variables)]
-#![allow(unused_mut)]
-#![allow(unused_assignments)]
-#![allow(unused_must_use)]
-#![allow(clippy::all)]
-
-use crate::client::{
-    api_error_response::http_resp_to_error_response,
-    api_get_options::GetObjectOptions,
-    transition_api::{ObjectInfo, ReaderImpl, RequestMetadata, TransitionClient},
-};
-use bytes::Bytes;
-use http::{HeaderMap, HeaderValue};
-use http_body_util::BodyExt;
-use rustfs_config::MAX_S3_CLIENT_RESPONSE_SIZE;
-use rustfs_utils::EMPTY_STRING_SHA256_HASH;
-use s3s::dto::Owner;
-use std::collections::HashMap;
-
-#[derive(Clone, Debug, Default, serde::Serialize, serde::Deserialize)]
-pub struct Grantee {
-    pub id: String,
-    pub display_name: String,
-    pub uri: String,
-}
-
-#[derive(Clone, Debug, Default, serde::Serialize, serde::Deserialize)]
-pub struct Grant {
-    pub grantee: Grantee,
-    pub permission: String,
-}
-
-#[derive(Debug, Default, serde::Serialize, serde::Deserialize)]
-pub struct AccessControlList {
-    pub grant: Vec<Grant>,
-    pub permission: String,
-}
-
-#[derive(Debug, Default, serde::Deserialize)]
-pub struct AccessControlPolicy {
-    #[serde(skip)]
-    owner: Owner,
-    pub access_control_list: AccessControlList,
-}
-
-impl TransitionClient {
-    pub async fn get_object_acl(&self, bucket_name: &str, object_name: &str) -> Result<ObjectInfo, std::io::Error> {
-        let mut url_values = HashMap::new();
-        url_values.insert("acl".to_string(), "".to_string());
-        let mut resp = self
-            .execute_method(
-                http::Method::GET,
-                &mut RequestMetadata {
-                    bucket_name: bucket_name.to_string(),
-                    object_name: object_name.to_string(),
-                    query_values: url_values,
-                    custom_header: HeaderMap::new(),
-                    content_sha256_hex: EMPTY_STRING_SHA256_HASH.to_string(),
-                    content_body: ReaderImpl::Body(Bytes::new()),
-                    content_length: 0,
-                    content_md5_base64: "".to_string(),
-                    stream_sha256: false,
-                    trailer: HeaderMap::new(),
-                    pre_sign_url: Default::default(),
-                    add_crc: Default::default(),
-                    extra_pre_sign_header: Default::default(),
-                    bucket_location: Default::default(),
-                    expires: Default::default(),
-                },
-            )
-            .await?;
-
-        let resp_status = resp.status();
-        let h = resp.headers().clone();
-
-        let mut body_vec = Vec::new();
-        let mut body = resp.into_body();
-        while let Some(frame) = body.frame().await {
-            let frame = frame.map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e.to_string()))?;
-            if let Some(data) = frame.data_ref() {
-                body_vec.extend_from_slice(data);
-            }
-        }
-
-        if resp_status != http::StatusCode::OK {
-            return Err(std::io::Error::other(http_resp_to_error_response(
-                resp_status,
-                &h,
-                body_vec,
-                bucket_name,
-                object_name,
-            )));
-        }
-
-        let mut res = match quick_xml::de::from_str::<AccessControlPolicy>(&String::from_utf8(body_vec).unwrap()) {
-            Ok(result) => result,
-            Err(err) => {
-                return Err(std::io::Error::other(err.to_string()));
-            }
-        };
-
-        let mut obj_info = self
-            .stat_object(bucket_name, object_name, &GetObjectOptions::default())
-            .await?;
-
-        obj_info.owner.display_name = res.owner.display_name.clone();
-        obj_info.owner.id = res.owner.id.clone();
-
-        //obj_info.grant.extend(res.access_control_list.grant);
-
-        let canned_acl = get_canned_acl(&res);
-        if canned_acl != "" {
-            obj_info
-                .metadata
-                .insert("X-Amz-Acl", HeaderValue::from_str(&canned_acl).unwrap());
-            return Ok(obj_info);
-        }
-
-        let grant_acl = get_amz_grant_acl(&res);
-        /*for (k, v) in grant_acl {
-            obj_info.metadata.insert(HeaderName::from_bytes(k.as_bytes()).unwrap(), HeaderValue::from_str(&v.to_string()).unwrap());
-        }*/
-
-        Ok(obj_info)
-    }
-}
-
-fn get_canned_acl(ac_policy: &AccessControlPolicy) -> String {
-    let grants = ac_policy.access_control_list.grant.clone();
-
-    if grants.len() == 1 {
-        if grants[0].grantee.uri == "" && grants[0].permission == "FULL_CONTROL" {
-            return "private".to_string();
-        }
-    } else if grants.len() == 2 {
-        for g in grants {
-            if g.grantee.uri == "http://acs.amazonaws.com/groups/global/AuthenticatedUsers" && &g.permission == "READ" {
-                return "authenticated-read".to_string();
-            }
-            if g.grantee.uri == "http://acs.amazonaws.com/groups/global/AllUsers" && &g.permission == "READ" {
-                return "public-read".to_string();
-            }
-            if g.permission == "READ" && g.grantee.id == ac_policy.owner.id.clone().unwrap() {
-                return "bucket-owner-read".to_string();
-            }
-        }
-    } else if grants.len() == 3 {
-        for g in grants {
-            if g.grantee.uri == "http://acs.amazonaws.com/groups/global/AllUsers" && g.permission == "WRITE" {
-                return "public-read-write".to_string();
-            }
-        }
-    }
-    "".to_string()
-}
-
-pub fn get_amz_grant_acl(ac_policy: &AccessControlPolicy) -> HashMap<String, Vec<String>> {
-    let grants = ac_policy.access_control_list.grant.clone();
-    let mut res = HashMap::<String, Vec<String>>::new();
-
-    for g in grants {
-        let mut id = "id=".to_string();
-        id.push_str(&g.grantee.id);
-        let permission: &str = &g.permission;
-        match permission {
-            "READ" => {
-                res.entry("X-Amz-Grant-Read".to_string()).or_insert(vec![]).push(id);
-            }
-            "WRITE" => {
-                res.entry("X-Amz-Grant-Write".to_string()).or_insert(vec![]).push(id);
-            }
-            "READ_ACP" => {
-                res.entry("X-Amz-Grant-Read-Acp".to_string()).or_insert(vec![]).push(id);
-            }
-            "WRITE_ACP" => {
-                res.entry("X-Amz-Grant-Write-Acp".to_string()).or_insert(vec![]).push(id);
-            }
-            "FULL_CONTROL" => {
-                res.entry("X-Amz-Grant-Full-Control".to_string()).or_insert(vec![]).push(id);
-            }
-            _ => (),
-        }
-    }
-    res
-}

crates/ecstore/src/client/api_get_object_attributes.rs

@@ -1,266 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-#![allow(unused_imports)]
-#![allow(unused_variables)]
-#![allow(unused_mut)]
-#![allow(unused_assignments)]
-#![allow(unused_must_use)]
-#![allow(clippy::all)]
-
-use http::{HeaderMap, HeaderValue};
-use std::collections::HashMap;
-use time::OffsetDateTime;
-
-use crate::client::constants::{GET_OBJECT_ATTRIBUTES_MAX_PARTS, GET_OBJECT_ATTRIBUTES_TAGS, ISO8601_DATEFORMAT};
-use crate::client::{
-    api_get_object_acl::AccessControlPolicy,
-    transition_api::{ReaderImpl, RequestMetadata, TransitionClient},
-};
-use http_body_util::BodyExt;
-use hyper::body::Body;
-use hyper::body::Bytes;
-use hyper::body::Incoming;
-use rustfs_config::MAX_S3_CLIENT_RESPONSE_SIZE;
-use rustfs_utils::EMPTY_STRING_SHA256_HASH;
-use s3s::header::{X_AMZ_MAX_PARTS, X_AMZ_OBJECT_ATTRIBUTES, X_AMZ_PART_NUMBER_MARKER, X_AMZ_VERSION_ID};
-
-pub struct ObjectAttributesOptions {
-    pub max_parts: i64,
-    pub version_id: String,
-    pub part_number_marker: i64,
-    //server_side_encryption: encrypt::ServerSide,
-}
-
-pub struct ObjectAttributes {
-    pub version_id: String,
-    pub last_modified: OffsetDateTime,
-    pub object_attributes_response: ObjectAttributesResponse,
-}
-
-impl ObjectAttributes {
-    fn new() -> Self {
-        Self {
-            version_id: "".to_string(),
-            last_modified: OffsetDateTime::now_utc(),
-            object_attributes_response: ObjectAttributesResponse::new(),
-        }
-    }
-}
-
-#[derive(Debug, Default, serde::Deserialize)]
-pub struct Checksum {
-    checksum_crc32: String,
-    checksum_crc32c: String,
-    checksum_sha1: String,
-    checksum_sha256: String,
-}
-
-impl Checksum {
-    fn new() -> Self {
-        Self {
-            checksum_crc32: "".to_string(),
-            checksum_crc32c: "".to_string(),
-            checksum_sha1: "".to_string(),
-            checksum_sha256: "".to_string(),
-        }
-    }
-}
-
-#[derive(Debug, Default, serde::Deserialize)]
-pub struct ObjectParts {
-    pub parts_count: i64,
-    pub part_number_marker: i64,
-    pub next_part_number_marker: i64,
-    pub max_parts: i64,
-    is_truncated: bool,
-    parts: Vec<ObjectAttributePart>,
-}
-
-impl ObjectParts {
-    fn new() -> Self {
-        Self {
-            parts_count: 0,
-            part_number_marker: 0,
-            next_part_number_marker: 0,
-            max_parts: 0,
-            is_truncated: false,
-            parts: Vec::new(),
-        }
-    }
-}
-
-#[derive(Debug, Default, serde::Deserialize)]
-pub struct ObjectAttributesResponse {
-    pub etag: String,
-    pub storage_class: String,
-    pub object_size: i64,
-    pub checksum: Checksum,
-    pub object_parts: ObjectParts,
-}
-
-impl ObjectAttributesResponse {
-    fn new() -> Self {
-        Self {
-            etag: "".to_string(),
-            storage_class: "".to_string(),
-            object_size: 0,
-            checksum: Checksum::new(),
-            object_parts: ObjectParts::new(),
-        }
-    }
-}
-
-#[derive(Debug, Default, serde::Deserialize)]
-struct ObjectAttributePart {
-    checksum_crc32: String,
-    checksum_crc32c: String,
-    checksum_sha1: String,
-    checksum_sha256: String,
-    part_number: i64,
-    size: i64,
-}
-
-impl ObjectAttributes {
-    pub async fn parse_response(&mut self, h: &HeaderMap, body_vec: Vec<u8>) -> Result<(), std::io::Error> {
-        let last_modified = h
-            .get("Last-Modified")
-            .ok_or_else(|| std::io::Error::other("missing Last-Modified header"))?
-            .to_str()
-            .map_err(|e| std::io::Error::other(format!("invalid Last-Modified header: {e}")))?;
-        let mod_time = OffsetDateTime::parse(last_modified, ISO8601_DATEFORMAT)
-            .map_err(|e| std::io::Error::other(format!("invalid Last-Modified date: {e}")))?;
-        self.last_modified = mod_time;
-
-        let version_id = h
-            .get(X_AMZ_VERSION_ID)
-            .ok_or_else(|| std::io::Error::other("missing version ID header"))?
-            .to_str()
-            .map_err(|e| std::io::Error::other(format!("invalid version ID header: {e}")))?;
-        self.version_id = version_id.to_string();
-
-        let body_str = String::from_utf8(body_vec).map_err(|e| std::io::Error::other(format!("invalid UTF-8 body: {e}")))?;
-        let mut response = match quick_xml::de::from_str::<ObjectAttributesResponse>(&body_str) {
-            Ok(result) => result,
-            Err(err) => {
-                return Err(std::io::Error::other(err.to_string()));
-            }
-        };
-        self.object_attributes_response = response;
-
-        Ok(())
-    }
-}
-
-impl TransitionClient {
-    pub async fn get_object_attributes(
-        &self,
-        bucket_name: &str,
-        object_name: &str,
-        opts: ObjectAttributesOptions,
-    ) -> Result<ObjectAttributes, std::io::Error> {
-        let mut url_values = HashMap::new();
-        url_values.insert("attributes".to_string(), "".to_string());
-        if opts.version_id != "" {
-            url_values.insert("versionId".to_string(), opts.version_id);
-        }
-
-        let mut headers = HeaderMap::new();
-        headers.insert(
-            X_AMZ_OBJECT_ATTRIBUTES,
-            HeaderValue::from_str(GET_OBJECT_ATTRIBUTES_TAGS).expect("valid header value"),
-        );
-
-        if opts.part_number_marker > 0 {
-            headers.insert(
-                X_AMZ_PART_NUMBER_MARKER,
-                HeaderValue::from_str(&opts.part_number_marker.to_string()).expect("valid header value"),
-            );
-        }
-
-        if opts.max_parts > 0 {
-            headers.insert(
-                X_AMZ_MAX_PARTS,
-                HeaderValue::from_str(&opts.max_parts.to_string()).expect("valid header value"),
-            );
-        } else {
-            headers.insert(
-                X_AMZ_MAX_PARTS,
-                HeaderValue::from_str(&GET_OBJECT_ATTRIBUTES_MAX_PARTS.to_string()).expect("valid header value"),
-            );
-        }
-
-        /*if opts.server_side_encryption.is_some() {
-            opts.server_side_encryption.Marshal(headers);
-        }*/
-
-        let mut resp = self
-            .execute_method(
-                http::Method::HEAD,
-                &mut RequestMetadata {
-                    bucket_name: bucket_name.to_string(),
-                    object_name: object_name.to_string(),
-                    query_values: url_values,
-                    custom_header: headers,
-                    content_sha256_hex: EMPTY_STRING_SHA256_HASH.to_string(),
-                    content_md5_base64: "".to_string(),
-                    content_body: ReaderImpl::Body(Bytes::new()),
-                    content_length: 0,
-                    stream_sha256: false,
-                    trailer: HeaderMap::new(),
-                    pre_sign_url: Default::default(),
-                    add_crc: Default::default(),
-                    extra_pre_sign_header: Default::default(),
-                    bucket_location: Default::default(),
-                    expires: Default::default(),
-                },
-            )
-            .await?;
-
-        let resp_status = resp.status();
-        let h = resp.headers().clone();
-        let has_etag = h.get("ETag").and_then(|v| v.to_str().ok()).unwrap_or("");
-        if !has_etag.is_empty() {
-            return Err(std::io::Error::other(
-                "get_object_attributes is not supported by the current endpoint version",
-            ));
-        }
-
-        let mut body_vec = Vec::new();
-        let mut body = resp.into_body();
-        while let Some(frame) = body.frame().await {
-            let frame = frame.map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e.to_string()))?;
-            if let Some(data) = frame.data_ref() {
-                body_vec.extend_from_slice(data);
-            }
-        }
-
-        if resp_status != http::StatusCode::OK {
-            let err_body =
-                String::from_utf8(body_vec).map_err(|e| std::io::Error::other(format!("invalid UTF-8 error body: {e}")))?;
-            let mut er = match quick_xml::de::from_str::<AccessControlPolicy>(&err_body) {
-                Ok(result) => result,
-                Err(err) => {
-                    return Err(std::io::Error::other(err.to_string()));
-                }
-            };
-
-            return Err(std::io::Error::other(er.access_control_list.permission));
-        }
-
-        let mut oa = ObjectAttributes::new();
-        oa.parse_response(&h, body_vec).await?;
-
-        Ok(oa)
-    }
-}

crates/ecstore/src/client/api_get_object_file.rs

@@ -1,159 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use std::io;
-use std::path::{Path, PathBuf};
-
-#[cfg(not(windows))]
-use std::os::unix::fs::PermissionsExt;
-
-use tokio::fs::{self, OpenOptions};
-use tokio::io::{AsyncSeekExt, AsyncWriteExt, SeekFrom};
-
-use crate::client::{
-    api_error_response::err_invalid_argument, api_get_options::GetObjectOptions, transition_api::TransitionClient,
-};
-
-async fn prepare_download_target(file_path: &Path) -> io::Result<()> {
-    match fs::metadata(file_path).await {
-        Ok(metadata) if metadata.is_dir() => {
-            return Err(io::Error::other(err_invalid_argument("filename is a directory.")));
-        }
-        Ok(_) => {}
-        Err(err) if err.kind() == io::ErrorKind::NotFound => {}
-        Err(err) => return Err(err),
-    }
-
-    if let Some(parent) = file_path.parent()
-        && !parent.as_os_str().is_empty()
-    {
-        fs::create_dir_all(parent).await?;
-
-        #[cfg(not(windows))]
-        {
-            let mut permissions = fs::metadata(parent).await?.permissions();
-            permissions.set_mode(0o700);
-            fs::set_permissions(parent, permissions).await?;
-        }
-    }
-
-    Ok(())
-}
-
-fn build_part_path(file_path: &Path) -> PathBuf {
-    PathBuf::from(format!("{}.part.rustfs", file_path.display()))
-}
-
-async fn open_download_part_file(file_part_path: &Path) -> io::Result<tokio::fs::File> {
-    let mut options = OpenOptions::new();
-    options.create(true).read(true).write(true);
-
-    #[cfg(not(windows))]
-    options.mode(0o600);
-
-    options.open(file_part_path).await
-}
-
-async fn cleanup_part_file(file_part_path: &Path) {
-    let _ = fs::remove_file(file_part_path).await;
-}
-
-impl TransitionClient {
-    pub async fn fget_object(
-        &self,
-        bucket_name: &str,
-        object_name: &str,
-        file_path: &str,
-        mut opts: GetObjectOptions,
-    ) -> Result<(), io::Error> {
-        let file_path = Path::new(file_path);
-        prepare_download_target(file_path).await?;
-
-        let file_part_path = build_part_path(file_path);
-        let mut file_part = open_download_part_file(&file_part_path).await?;
-        let existing_len = file_part.metadata().await?.len();
-        if existing_len > 0 {
-            opts.set_range(existing_len as i64, 0)?;
-            file_part.seek(SeekFrom::Start(existing_len)).await?;
-        }
-
-        let (_object_info, _headers, mut object_reader) = self.get_object_inner(bucket_name, object_name, &opts).await?;
-        if let Err(err) = tokio::io::copy(&mut object_reader, &mut file_part).await {
-            cleanup_part_file(&file_part_path).await;
-            return Err(err);
-        }
-
-        if let Err(err) = file_part.flush().await {
-            cleanup_part_file(&file_part_path).await;
-            return Err(err);
-        }
-        drop(file_part);
-
-        if let Err(err) = fs::rename(&file_part_path, file_path).await {
-            cleanup_part_file(&file_part_path).await;
-            return Err(err);
-        }
-
-        Ok(())
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use tempfile::tempdir;
-
-    #[tokio::test]
-    async fn prepare_download_target_allows_missing_file_and_creates_parent_dirs() {
-        let dir = tempdir().expect("temp dir");
-        let target = dir.path().join("nested").join("object.bin");
-
-        prepare_download_target(&target)
-            .await
-            .expect("missing target should be accepted");
-
-        assert!(target.parent().expect("parent").exists(), "parent directory should be created");
-        assert!(
-            fs::metadata(&target).await.is_err(),
-            "preparing the target should not create the final file eagerly"
-        );
-    }
-
-    #[tokio::test]
-    async fn prepare_download_target_rejects_directory_paths() {
-        let dir = tempdir().expect("temp dir");
-        let target_dir = dir.path().join("download-dir");
-        fs::create_dir_all(&target_dir).await.expect("target dir");
-
-        let err = prepare_download_target(&target_dir)
-            .await
-            .expect_err("directory targets must be rejected");
-
-        assert!(err.to_string().contains("directory"), "unexpected error for directory target: {err}");
-    }
-
-    #[tokio::test]
-    async fn open_download_part_file_creates_part_file() {
-        let dir = tempdir().expect("temp dir");
-        let target = dir.path().join("object.bin");
-        let part_path = build_part_path(&target);
-
-        let file = open_download_part_file(&part_path)
-            .await
-            .expect("part file should be created");
-        drop(file);
-
-        assert!(part_path.exists(), "part file should exist after creation");
-    }
-}

crates/ecstore/src/client/api_restore.rs

@@ -1,134 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-#![allow(unused_imports)]
-#![allow(unused_variables)]
-#![allow(unused_mut)]
-#![allow(unused_assignments)]
-#![allow(unused_must_use)]
-#![allow(clippy::all)]
-
-use crate::client::{
-    api_error_response::{err_invalid_argument, http_resp_to_error_response},
-    api_get_object_acl::AccessControlList,
-    api_get_options::GetObjectOptions,
-    transition_api::{ObjectInfo, ReadCloser, ReaderImpl, RequestMetadata, TransitionClient, to_object_info},
-};
-use http::HeaderMap;
-use http_body_util::BodyExt;
-use hyper::body::Body;
-use hyper::body::Bytes;
-use s3s::dto::RestoreRequest;
-use std::collections::HashMap;
-use std::io::Cursor;
-use tokio::io::BufReader;
-
-const TIER_STANDARD: &str = "Standard";
-const TIER_BULK: &str = "Bulk";
-const TIER_EXPEDITED: &str = "Expedited";
-
-#[derive(Debug, Default, serde::Serialize, serde::Deserialize)]
-pub struct Encryption {
-    pub encryption_type: String,
-    pub kms_context: String,
-    pub kms_key_id: String,
-}
-
-#[derive(Debug, Default, serde::Serialize, serde::Deserialize)]
-pub struct MetadataEntry {
-    pub name: String,
-    pub value: String,
-}
-
-#[derive(Debug, Default, serde::Serialize)]
-pub struct S3 {
-    pub access_control_list: AccessControlList,
-    pub bucket_name: String,
-    pub prefix: String,
-    pub canned_acl: String,
-    pub encryption: Encryption,
-    pub storage_class: String,
-    //tagging:            Tags,
-    pub user_metadata: MetadataEntry,
-}
-
-impl TransitionClient {
-    pub async fn restore_object(
-        &self,
-        bucket_name: &str,
-        object_name: &str,
-        version_id: &str,
-        restore_req: &RestoreRequest,
-    ) -> Result<(), std::io::Error> {
-        /*let restore_request = match quick_xml::se::to_string(restore_req) {
-            Ok(buf) => buf,
-            Err(e) => {
-                return Err(std::io::Error::other(e));
-            }
-        };*/
-        let restore_request = "".to_string();
-        let restore_request_bytes = restore_request.as_bytes().to_vec();
-
-        let mut url_values = HashMap::new();
-        url_values.insert("restore".to_string(), "".to_string());
-        if version_id != "" {
-            url_values.insert("versionId".to_string(), version_id.to_string());
-        }
-
-        let restore_request_buffer = Bytes::from(restore_request_bytes.clone());
-        let resp = self
-            .execute_method(
-                http::Method::HEAD,
-                &mut RequestMetadata {
-                    bucket_name: bucket_name.to_string(),
-                    object_name: object_name.to_string(),
-                    query_values: url_values,
-                    custom_header: HeaderMap::new(),
-                    content_sha256_hex: "".to_string(), //sum_sha256_hex(&restore_request_bytes),
-                    content_md5_base64: "".to_string(), //sum_md5_base64(&restore_request_bytes),
-                    content_body: ReaderImpl::Body(restore_request_buffer),
-                    content_length: restore_request_bytes.len() as i64,
-                    stream_sha256: false,
-                    trailer: HeaderMap::new(),
-                    pre_sign_url: Default::default(),
-                    add_crc: Default::default(),
-                    extra_pre_sign_header: Default::default(),
-                    bucket_location: Default::default(),
-                    expires: Default::default(),
-                },
-            )
-            .await?;
-
-        let resp_status = resp.status();
-        let h = resp.headers().clone();
-
-        let mut body_vec = Vec::new();
-        let mut body = resp.into_body();
-        while let Some(frame) = body.frame().await {
-            let frame = frame.map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e.to_string()))?;
-            if let Some(data) = frame.data_ref() {
-                body_vec.extend_from_slice(data);
-            }
-        }
-        if resp_status != http::StatusCode::ACCEPTED && resp_status != http::StatusCode::OK {
-            return Err(std::io::Error::other(http_resp_to_error_response(
-                resp_status,
-                &h,
-                body_vec,
-                bucket_name,
-                "",
-            )));
-        }
-        Ok(())
-    }
-}

crates/ecstore/src/client/constants.rs

@@ -37,6 +37,3 @@ pub const TOTAL_WORKERS: i64 = 4;
 pub const SIGN_V4_ALGORITHM: &str = "AWS4-HMAC-SHA256";
 pub const ISO8601_DATEFORMAT: &[FormatItem<'_>] =
     format_description!("[year]-[month]-[day]T[hour]:[minute]:[second].[subsecond]Z");
-
-pub const GET_OBJECT_ATTRIBUTES_TAGS: &str = "ETag,Checksum,StorageClass,ObjectSize,ObjectParts";
-pub const GET_OBJECT_ATTRIBUTES_MAX_PARTS: i64 = 1000;

crates/ecstore/src/client/mod.rs

@@ -16,12 +16,8 @@
 #![allow(dead_code)]
 
 pub mod admin_handler_utils;
-pub mod api_bucket_policy;
 pub mod api_error_response;
 pub mod api_get_object;
-pub mod api_get_object_acl;
-pub mod api_get_object_attributes;
-pub mod api_get_object_file;
 pub mod api_get_options;
 pub mod api_list;
 pub mod api_put_object;
@@ -29,7 +25,6 @@ pub mod api_put_object_common;
 pub mod api_put_object_multipart;
 pub mod api_put_object_streaming;
 pub mod api_remove;
-pub mod api_restore;
 pub mod api_s3_datatypes;
 pub mod api_stat;
 pub mod bucket_cache;

crates/ecstore/src/client/transition_api.rs

@@ -1006,16 +1006,6 @@ impl TransitionCore {
         client.abort_multipart_upload(bucket_name, object, upload_id).await
     }
 
-    pub async fn get_bucket_policy(&self, bucket_name: &str) -> Result<String, std::io::Error> {
-        let client = self.0.clone();
-        client.get_bucket_policy(bucket_name).await
-    }
-
-    pub async fn put_bucket_policy(&self, bucket_name: &str, bucket_policy: &str) -> Result<(), std::io::Error> {
-        let client = self.0.clone();
-        client.put_bucket_policy(bucket_name, bucket_policy).await
-    }
-
     pub async fn get_object(
         &self,
         bucket_name: &str,

crates/ecstore/src/cluster/rpc/client.rs

@@ -15,12 +15,12 @@
 #[cfg(test)]
 use crate::cluster::rpc::http_auth::RPC_REPLAY_SCOPE_VERSION_HEADER;
 use crate::cluster::rpc::http_auth::{
-    RPC_AUTH_VERSION_HEADER, RPC_AUTH_VERSION_V2, RPC_BOOT_EPOCH_CHALLENGE_HEADER, RPC_BOOT_EPOCH_HEADER,
-    RPC_BOOT_EPOCH_PROOF_HEADER, RPC_CONTENT_SHA256_HEADER, TIMESTAMP_HEADER,
-};
-use crate::cluster::rpc::{
-    gen_tonic_replay_scope_headers, gen_tonic_signature_headers, normalize_tonic_rpc_audience, verify_tonic_boot_epoch_response,
+    AuthenticatedPeerReplayCapabilities, RPC_AUTH_VERSION_HEADER, RPC_AUTH_VERSION_V2, RPC_BOOT_EPOCH_CHALLENGE_HEADER,
+    RPC_CONTENT_SHA256_HEADER, RPC_REPLAY_CACHE_CAPABILITY_HEADER, RPC_REPLAY_CACHE_CAPABILITY_PROOF_HEADER,
+    RollingMutationBodyDigest, TIMESTAMP_HEADER, internode_rpc_body_digest_strict,
+    verify_tonic_peer_replay_capabilities_response,
 };
+use crate::cluster::rpc::{gen_tonic_replay_scope_headers, gen_tonic_signature_headers, normalize_tonic_rpc_audience};
 #[cfg(test)]
 use crate::cluster::rpc::{tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers};
 use crate::disk::error::{DiskError, Error as DiskErrorType, RpcStatusError};
@@ -233,7 +233,22 @@ pub struct ReplayScopeChannel<S> {
 /// The channel type used by internode clients after v2 authentication and replay-scope handling.
 pub type AuthenticatedChannel = ReplayScopeChannel<Channel>;
 
-static PEER_BOOT_EPOCHS: LazyLock<Mutex<HashMap<String, Uuid>>> = LazyLock::new(|| Mutex::new(HashMap::new()));
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+enum PeerReplayCapability {
+    Capable { boot_epoch: Uuid },
+    Revoked,
+}
+
+#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
+struct PeerReplayState {
+    boot_epoch: Option<Uuid>,
+    cache_capability: Option<PeerReplayCapability>,
+}
+
+#[derive(Clone, Copy, Debug)]
+struct PeerReplayStateSnapshot(PeerReplayState);
+
+static PEER_REPLAY_STATES: LazyLock<Mutex<HashMap<String, PeerReplayState>>> = LazyLock::new(|| Mutex::new(HashMap::new()));
 
 impl<S> ReplayScopeChannel<S> {
     fn new(inner: S, audience: Option<String>) -> Self {
@@ -241,13 +256,67 @@ impl<S> ReplayScopeChannel<S> {
     }
 }
 
-fn cached_peer_boot_epoch(audience: &str) -> Option<Uuid> {
-    PEER_BOOT_EPOCHS.lock().ok().and_then(|epochs| epochs.get(audience).copied())
+fn peer_replay_state(audience: &str) -> PeerReplayState {
+    PEER_REPLAY_STATES
+        .lock()
+        .ok()
+        .and_then(|states| states.get(audience).copied())
+        .unwrap_or_default()
 }
 
-fn remember_peer_boot_epoch(audience: String, epoch: Uuid) {
-    if let Ok(mut epochs) = PEER_BOOT_EPOCHS.lock() {
-        epochs.insert(audience, epoch);
+fn apply_peer_replay_response(
+    audience: String,
+    sent_state: PeerReplayState,
+    response: std::io::Result<AuthenticatedPeerReplayCapabilities>,
+) {
+    if let Ok(mut states) = PEER_REPLAY_STATES.lock() {
+        let current_state = states.get(&audience).copied().unwrap_or_default();
+        let mut next_state = current_state;
+        if let Ok(response) = &response
+            && sent_state.boot_epoch == current_state.boot_epoch
+        {
+            next_state.boot_epoch = Some(response.boot_epoch);
+        }
+
+        if sent_state.boot_epoch == current_state.boot_epoch {
+            let response_capability = response
+                .as_ref()
+                .ok()
+                .filter(|response| response.dynamic_replay_cache)
+                .map(|response| response.boot_epoch);
+            match (sent_state.cache_capability, current_state.cache_capability, response_capability) {
+                (None, None, Some(boot_epoch))
+                | (Some(PeerReplayCapability::Revoked), Some(PeerReplayCapability::Revoked), Some(boot_epoch)) => {
+                    next_state.cache_capability = Some(PeerReplayCapability::Capable { boot_epoch });
+                }
+                (
+                    Some(PeerReplayCapability::Capable {
+                        boot_epoch: sent_boot_epoch,
+                    }),
+                    Some(PeerReplayCapability::Capable {
+                        boot_epoch: current_boot_epoch,
+                    }),
+                    Some(response_boot_epoch),
+                ) if sent_boot_epoch == current_boot_epoch => {
+                    next_state.cache_capability = Some(PeerReplayCapability::Capable {
+                        boot_epoch: response_boot_epoch,
+                    });
+                }
+                (
+                    Some(PeerReplayCapability::Capable {
+                        boot_epoch: sent_boot_epoch,
+                    }),
+                    Some(PeerReplayCapability::Capable {
+                        boot_epoch: current_boot_epoch,
+                    }),
+                    None,
+                ) if sent_boot_epoch == current_boot_epoch => {
+                    next_state.cache_capability = Some(PeerReplayCapability::Revoked);
+                }
+                _ => {}
+            }
+        }
+        states.insert(audience, next_state);
     }
 }
 
@@ -276,6 +345,11 @@ where
                 == Some(RPC_AUTH_VERSION_V2)
         });
         let challenge = authenticated.then(Uuid::new_v4);
+        let sent_state = request
+            .extensions()
+            .get::<PeerReplayStateSnapshot>()
+            .map(|snapshot| snapshot.0)
+            .unwrap_or_default();
         if let (Some(audience), Some(challenge)) = (self.audience.as_deref(), challenge) {
             // The challenge is independently HMAC-authenticated by the response proof. It is not
             // part of v2 so old peers ignore it, while a new peer can safely advertise its epoch.
@@ -284,7 +358,7 @@ where
                 challenge.to_string().parse().expect("UUID must be a valid header value"),
             );
             if let (Some(boot_epoch), Some(timestamp), Some(content_sha256)) = (
-                cached_peer_boot_epoch(audience),
+                sent_state.boot_epoch,
                 request.headers().get(TIMESTAMP_HEADER).and_then(|value| value.to_str().ok()),
                 request
                     .headers()
@@ -303,16 +377,21 @@ where
         Box::pin(async move {
             let response = future.await?;
             if let (Some(audience), Some(challenge)) = (audience, challenge) {
-                match verify_tonic_boot_epoch_response(&audience, challenge, response.headers()) {
-                    Ok(epoch) => remember_peer_boot_epoch(audience, epoch),
-                    Err(error)
-                        if response.headers().contains_key(RPC_BOOT_EPOCH_HEADER)
-                            || response.headers().contains_key(RPC_BOOT_EPOCH_PROOF_HEADER) =>
-                    {
-                        debug!(error = %error, "peer boot epoch response proof was rejected")
-                    }
-                    Err(_) => {}
+                let response_state = verify_tonic_peer_replay_capabilities_response(&audience, challenge, response.headers());
+                if let Err(error) = &response_state
+                    && (response.headers().contains_key(RPC_REPLAY_CACHE_CAPABILITY_HEADER)
+                        || response.headers().contains_key(RPC_REPLAY_CACHE_CAPABILITY_PROOF_HEADER))
+                {
+                    debug!(
+                        event = "internode_rpc_capability_proof_rejected",
+                        component = "ecstore",
+                        subsystem = "rpc_client",
+                        result = "rejected",
+                        error = %error,
+                        "internode RPC capability proof rejected"
+                    )
                 }
+                apply_peer_replay_response(audience, sent_state, response_state);
             }
             Ok(response)
         })
@@ -321,6 +400,7 @@ where
 
 pub struct TonicSignatureInterceptor {
     audience: Option<String>,
+    body_digest_strict: bool,
 }
 
 impl tonic::service::Interceptor for TonicSignatureInterceptor {
@@ -337,9 +417,31 @@ impl tonic::service::Interceptor for TonicSignatureInterceptor {
             .metadata()
             .get(RPC_CONTENT_SHA256_HEADER)
             .and_then(|value| value.to_str().ok());
+        // RUSTFS_COMPAT_TODO(disk-mutation-body-digest): use cache-free v2 for peers without an authenticated boot epoch. Remove after every supported peer advertises the authenticated dynamic replay-cache capability and body-digest strict mode is the default.
+        // beta.11 verifies v2 body digests but stores their nonces in a fixed-size cache.
+        let rolling_mutation = req.extensions().get::<RollingMutationBodyDigest>().is_some();
+        let peer_state = PEER_REPLAY_STATES
+            .lock()
+            .map_err(|_| tonic::Status::unauthenticated("RPC peer capability state unavailable"))?
+            .get(audience)
+            .copied()
+            .unwrap_or_default();
+        let content_sha256 = if content_sha256.is_some() {
+            if peer_state.cache_capability == Some(PeerReplayCapability::Revoked) {
+                return Err(tonic::Status::unauthenticated("RPC peer replay capability changed"));
+            }
+            if rolling_mutation && !self.body_digest_strict && peer_state.boot_epoch.is_none() {
+                None
+            } else {
+                content_sha256
+            }
+        } else {
+            content_sha256
+        };
         let headers = gen_tonic_signature_headers(audience, method.service(), method.method(), content_sha256)
             .map_err(|_| tonic::Status::unauthenticated("No valid auth token"))?;
         req.metadata_mut().as_mut().extend(headers);
+        req.extensions_mut().insert(PeerReplayStateSnapshot(peer_state));
         inject_trace_context_into_metadata(req.metadata_mut());
         inject_request_id_into_metadata(req.metadata_mut());
         Ok(req)
@@ -347,7 +449,10 @@ impl tonic::service::Interceptor for TonicSignatureInterceptor {
 }
 
 pub fn gen_tonic_signature_interceptor() -> TonicSignatureInterceptor {
-    TonicSignatureInterceptor { audience: None }
+    TonicSignatureInterceptor {
+        audience: None,
+        body_digest_strict: internode_rpc_body_digest_strict(),
+    }
 }
 
 pub struct NoOpInterceptor;
@@ -409,6 +514,7 @@ mod tests {
     #[derive(Clone)]
     struct EpochProofService {
         audience: String,
+        include_capability: bool,
         seen_headers: std::sync::Arc<Mutex<Vec<http::HeaderMap>>>,
     }
 
@@ -430,29 +536,97 @@ mod tests {
                 .expect("client challenge must be syntactically valid")
                 .expect("authenticated client request must carry a boot epoch challenge");
             let mut response = HttpResponse::new(());
-            response.headers_mut().extend(
-                tonic_boot_epoch_response_headers(&self.audience, challenge)
-                    .expect("test server must be able to sign an epoch proof"),
-            );
+            let mut headers = tonic_boot_epoch_response_headers(&self.audience, challenge)
+                .expect("test server must be able to sign an epoch proof");
+            if !self.include_capability {
+                headers.remove(RPC_REPLAY_CACHE_CAPABILITY_HEADER);
+                headers.remove(RPC_REPLAY_CACHE_CAPABILITY_PROOF_HEADER);
+            }
+            response.headers_mut().extend(headers);
             std::future::ready(Ok(response))
         }
     }
 
+    #[derive(Clone)]
+    struct MissingProofService;
+
+    impl Service<HttpRequest<()>> for MissingProofService {
+        type Response = HttpResponse<()>;
+        type Error = std::convert::Infallible;
+        type Future = std::future::Ready<Result<Self::Response, Self::Error>>;
+
+        fn poll_ready(&mut self, _cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
+            Poll::Ready(Ok(()))
+        }
+
+        fn call(&mut self, _request: HttpRequest<()>) -> Self::Future {
+            std::future::ready(Ok(HttpResponse::new(())))
+        }
+    }
+
     fn ensure_test_rpc_secret() {
         runtime_sources::ensure_test_rpc_secret();
     }
 
     fn test_request() -> tonic::Request<()> {
+        test_request_for("Ping")
+    }
+
+    fn test_request_for(method: &'static str) -> tonic::Request<()> {
         let mut request = tonic::Request::new(());
         request
             .extensions_mut()
-            .insert(tonic::GrpcMethod::new("node_service.NodeService", "Ping"));
+            .insert(tonic::GrpcMethod::new("node_service.NodeService", method));
         request
     }
 
     fn test_interceptor() -> TonicSignatureInterceptor {
+        test_interceptor_for("node-a:9000", false)
+    }
+
+    fn test_interceptor_for(audience: &str, body_digest_strict: bool) -> TonicSignatureInterceptor {
         TonicSignatureInterceptor {
-            audience: Some("node-a:9000".to_string()),
+            audience: Some(audience.to_string()),
+            body_digest_strict,
+        }
+    }
+
+    fn clear_peer_capability(audience: &str) {
+        PEER_REPLAY_STATES
+            .lock()
+            .expect("peer capability cache lock must not be poisoned")
+            .remove(audience);
+    }
+
+    fn rolling_mutation_request(method: &'static str) -> tonic::Request<()> {
+        let mut request = tonic::Request::new(rustfs_protos::proto_gen::node_service::GenerallyLockRequest {
+            args: "canonical mutation request".to_string(),
+        });
+        request
+            .extensions_mut()
+            .insert(tonic::GrpcMethod::new("node_service.NodeService", method));
+        crate::cluster::rpc::set_tonic_rolling_mutation_body_digest(&mut request).expect("test mutation digest must be attached");
+        request.map(|_| ())
+    }
+
+    fn replay_scope_request(audience: &str, method: &'static str) -> HttpRequest<()> {
+        let mut request = HttpRequest::builder()
+            .uri(format!("/node_service.NodeService/{method}"))
+            .body(())
+            .expect("test RPC request must build");
+        request.headers_mut().extend(
+            gen_tonic_signature_headers(audience, "node_service.NodeService", method, None).expect("v2 test headers must mint"),
+        );
+        request
+            .extensions_mut()
+            .insert(PeerReplayStateSnapshot(peer_replay_state(audience)));
+        request
+    }
+
+    fn authenticated_peer_response(boot_epoch: Uuid, dynamic_replay_cache: bool) -> AuthenticatedPeerReplayCapabilities {
+        AuthenticatedPeerReplayCapabilities {
+            boot_epoch,
+            dynamic_replay_cache,
         }
     }
 
@@ -567,6 +741,431 @@ mod tests {
         );
     }
 
+    #[test]
+    fn unknown_peer_mutations_use_cache_free_unsigned_v2() {
+        ensure_test_rpc_secret();
+        let audience = "legacy-body-digest-client-test:9000";
+        clear_peer_capability(audience);
+        let mut interceptor = test_interceptor_for(audience, false);
+        for method in ["Lock", "WriteAll"] {
+            let request = interceptor
+                .call(rolling_mutation_request(method))
+                .expect("interceptor call should succeed");
+
+            assert_eq!(
+                request
+                    .metadata()
+                    .get("x-rustfs-content-sha256")
+                    .and_then(|value| value.to_str().ok()),
+                Some("UNSIGNED-PAYLOAD")
+            );
+            assert_eq!(
+                request
+                    .metadata()
+                    .get("x-rustfs-rpc-nonce")
+                    .and_then(|value| value.to_str().ok()),
+                Some("unsigned")
+            );
+            assert!(
+                crate::cluster::rpc::verify_tonic_rpc_signature(
+                    audience,
+                    &format!("/node_service.NodeService/{method}"),
+                    request.metadata().as_ref(),
+                )
+                .is_ok(),
+                "the cache-free request must retain valid audience- and method-bound v2 authentication"
+            );
+        }
+    }
+
+    #[test]
+    fn unknown_peer_exact_body_contract_remains_body_bound() {
+        ensure_test_rpc_secret();
+        let audience = "exact-body-contract-client-test:9000";
+        clear_peer_capability(audience);
+        let mut interceptor = test_interceptor_for(audience, false);
+        let mut request = test_request_for("ScannerActivity");
+        crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, b"exact scanner activity body")
+            .expect("test exact body digest must be attached");
+        let expected_digest = request
+            .metadata()
+            .get("x-rustfs-content-sha256")
+            .and_then(|value| value.to_str().ok())
+            .expect("test request must carry its digest")
+            .to_string();
+
+        let request = interceptor.call(request).expect("interceptor call should succeed");
+
+        assert_eq!(
+            request
+                .metadata()
+                .get("x-rustfs-content-sha256")
+                .and_then(|value| value.to_str().ok()),
+            Some(expected_digest.as_str())
+        );
+        assert!(
+            crate::cluster::rpc::verify_tonic_rpc_signature(
+                audience,
+                "/node_service.NodeService/ScannerActivity",
+                request.metadata().as_ref(),
+            )
+            .is_ok()
+        );
+    }
+
+    #[test]
+    fn unknown_peer_iam_mutation_helper_remains_body_bound() {
+        ensure_test_rpc_secret();
+        let audience = "exact-iam-mutation-client-test:9000";
+        clear_peer_capability(audience);
+        let mut interceptor = test_interceptor_for(audience, false);
+        let mut request = tonic::Request::new(rustfs_protos::proto_gen::node_service::DeleteUserRequest {
+            access_key: "target-access-key".to_string(),
+        });
+        request
+            .extensions_mut()
+            .insert(tonic::GrpcMethod::new("node_service.NodeService", "DeleteUser"));
+        crate::cluster::rpc::set_tonic_mutation_body_digest(&mut request).expect("test IAM mutation digest must be attached");
+        let request = request.map(|_| ());
+        let expected_digest = request
+            .metadata()
+            .get("x-rustfs-content-sha256")
+            .and_then(|value| value.to_str().ok())
+            .expect("test IAM mutation must carry its digest")
+            .to_string();
+
+        let request = interceptor.call(request).expect("interceptor call should succeed");
+
+        assert_eq!(
+            request
+                .metadata()
+                .get("x-rustfs-content-sha256")
+                .and_then(|value| value.to_str().ok()),
+            Some(expected_digest.as_str())
+        );
+        assert!(
+            crate::cluster::rpc::verify_tonic_rpc_signature(
+                audience,
+                "/node_service.NodeService/DeleteUser",
+                request.metadata().as_ref(),
+            )
+            .is_ok(),
+            "IAM mutations must remain body-bound before capability discovery"
+        );
+    }
+
+    #[test]
+    fn authenticated_replay_cache_capability_enables_body_binding() {
+        ensure_test_rpc_secret();
+        let audience = "body-digest-capable-client-test:9000";
+        clear_peer_capability(audience);
+        let seen_headers = std::sync::Arc::new(Mutex::new(Vec::new()));
+        let service = EpochProofService {
+            audience: audience.to_string(),
+            include_capability: true,
+            seen_headers,
+        };
+        let mut channel = ReplayScopeChannel::new(service, Some(audience.to_string()));
+        futures::executor::block_on(channel.call(replay_scope_request(audience, "Ping")))
+            .expect("authenticated capability probe must complete");
+
+        let mut interceptor = test_interceptor_for(audience, false);
+        let request = rolling_mutation_request("Lock");
+        let expected_digest = request
+            .metadata()
+            .get("x-rustfs-content-sha256")
+            .and_then(|value| value.to_str().ok())
+            .expect("test mutation must carry its digest")
+            .to_string();
+
+        let request = interceptor.call(request).expect("interceptor call should succeed");
+
+        assert_eq!(
+            request
+                .metadata()
+                .get("x-rustfs-content-sha256")
+                .and_then(|value| value.to_str().ok()),
+            Some(expected_digest.as_str())
+        );
+        let nonce = request
+            .metadata()
+            .get("x-rustfs-rpc-nonce")
+            .and_then(|value| value.to_str().ok())
+            .and_then(|value| Uuid::parse_str(value).ok())
+            .expect("capable peer body-bound mutation must carry a UUID nonce");
+        assert!(!nonce.is_nil());
+        assert!(
+            crate::cluster::rpc::verify_tonic_rpc_signature(
+                audience,
+                "/node_service.NodeService/Lock",
+                request.metadata().as_ref(),
+            )
+            .is_ok(),
+            "the body-bound request must retain valid audience- and method-bound v2 authentication"
+        );
+        clear_peer_capability(audience);
+    }
+
+    #[test]
+    fn invalid_capability_proof_does_not_enable_body_binding() {
+        ensure_test_rpc_secret();
+        let audience = "invalid-capability-client-test:9000";
+        clear_peer_capability(audience);
+        let service = EpochProofService {
+            audience: "wrong-capability-audience:9000".to_string(),
+            include_capability: true,
+            seen_headers: std::sync::Arc::new(Mutex::new(Vec::new())),
+        };
+        let mut channel = ReplayScopeChannel::new(service, Some(audience.to_string()));
+        futures::executor::block_on(channel.call(replay_scope_request(audience, "Ping")))
+            .expect("invalid capability response must still complete");
+
+        let mut interceptor = test_interceptor_for(audience, false);
+        let request = interceptor
+            .call(rolling_mutation_request("Lock"))
+            .expect("legacy-compatible mutation must still be signed");
+
+        assert_eq!(
+            request
+                .metadata()
+                .get("x-rustfs-content-sha256")
+                .and_then(|value| value.to_str().ok()),
+            Some("UNSIGNED-PAYLOAD")
+        );
+    }
+
+    #[test]
+    fn legacy_boot_proof_keeps_mutations_body_bound_and_enables_non_ping_v3() {
+        ensure_test_rpc_secret();
+        let audience = "legacy-boot-proof-client-test:9000";
+        clear_peer_capability(audience);
+        let seen_headers = std::sync::Arc::new(Mutex::new(Vec::new()));
+        let service = EpochProofService {
+            audience: audience.to_string(),
+            include_capability: false,
+            seen_headers: seen_headers.clone(),
+        };
+        let mut channel = ReplayScopeChannel::new(service, Some(audience.to_string()));
+        futures::executor::block_on(channel.call(replay_scope_request(audience, "Ping")))
+            .expect("legacy boot proof response must complete");
+
+        let state = peer_replay_state(audience);
+        assert!(state.boot_epoch.is_some(), "authenticated legacy proof must enable replay-scoped v3");
+        assert_eq!(state.cache_capability, None);
+
+        let mut interceptor = test_interceptor_for(audience, false);
+        let request = rolling_mutation_request("Lock");
+        let expected_digest = request
+            .metadata()
+            .get("x-rustfs-content-sha256")
+            .and_then(|value| value.to_str().ok())
+            .expect("test mutation must carry its digest")
+            .to_string();
+        let request = interceptor.call(request).expect("legacy-compatible mutation must be signed");
+        assert_eq!(
+            request
+                .metadata()
+                .get("x-rustfs-content-sha256")
+                .and_then(|value| value.to_str().ok()),
+            Some(expected_digest.as_str())
+        );
+        let (metadata, extensions, body) = request.into_parts();
+        let mut request = HttpRequest::new(body);
+        *request.uri_mut() = "/node_service.NodeService/Lock".parse().expect("test RPC URI must parse");
+        *request.headers_mut() = metadata.into_headers();
+        *request.extensions_mut() = extensions;
+        futures::executor::block_on(channel.call(request)).expect("legacy strict-compatible lock request must complete");
+
+        let headers = seen_headers.lock().expect("test header capture lock must not be poisoned");
+        assert!(
+            headers[1].contains_key(RPC_REPLAY_SCOPE_VERSION_HEADER),
+            "authenticated legacy boot proof must enable v3 on a non-Ping request"
+        );
+    }
+
+    #[test]
+    fn reordered_capability_responses_cannot_undo_newer_state() {
+        let audience = "reordered-capability-client-test:9000";
+        let epoch_one = Uuid::new_v4();
+        let epoch_two = Uuid::new_v4();
+        clear_peer_capability(audience);
+
+        let unknown = PeerReplayState::default();
+        apply_peer_replay_response(audience.to_string(), unknown, Ok(authenticated_peer_response(epoch_one, true)));
+        apply_peer_replay_response(audience.to_string(), unknown, Err(std::io::Error::other("delayed legacy response")));
+        let epoch_one_state = PeerReplayState {
+            boot_epoch: Some(epoch_one),
+            cache_capability: Some(PeerReplayCapability::Capable { boot_epoch: epoch_one }),
+        };
+        assert_eq!(peer_replay_state(audience), epoch_one_state);
+
+        apply_peer_replay_response(audience.to_string(), epoch_one_state, Err(std::io::Error::other("rollback response")));
+        apply_peer_replay_response(audience.to_string(), epoch_one_state, Ok(authenticated_peer_response(epoch_one, true)));
+        assert_eq!(
+            peer_replay_state(audience),
+            PeerReplayState {
+                boot_epoch: Some(epoch_one),
+                cache_capability: Some(PeerReplayCapability::Revoked),
+            }
+        );
+
+        let revoked = peer_replay_state(audience);
+        apply_peer_replay_response(audience.to_string(), revoked, Ok(authenticated_peer_response(epoch_two, true)));
+        apply_peer_replay_response(audience.to_string(), epoch_one_state, Ok(authenticated_peer_response(epoch_one, true)));
+        assert_eq!(
+            peer_replay_state(audience),
+            PeerReplayState {
+                boot_epoch: Some(epoch_two),
+                cache_capability: Some(PeerReplayCapability::Capable { boot_epoch: epoch_two }),
+            }
+        );
+        clear_peer_capability(audience);
+    }
+
+    #[test]
+    fn stale_capability_response_cannot_cross_a_new_boot_epoch() {
+        let audience = "cross-epoch-capability-client-test:9000";
+        let epoch_one = Uuid::new_v4();
+        let epoch_two = Uuid::new_v4();
+        let epoch_three = Uuid::new_v4();
+        clear_peer_capability(audience);
+
+        let revoked_epoch_one = PeerReplayState {
+            boot_epoch: Some(epoch_one),
+            cache_capability: Some(PeerReplayCapability::Revoked),
+        };
+        PEER_REPLAY_STATES
+            .lock()
+            .expect("peer replay state lock must not be poisoned")
+            .insert(audience.to_string(), revoked_epoch_one);
+
+        apply_peer_replay_response(
+            audience.to_string(),
+            revoked_epoch_one,
+            Ok(authenticated_peer_response(epoch_three, false)),
+        );
+        apply_peer_replay_response(audience.to_string(), revoked_epoch_one, Ok(authenticated_peer_response(epoch_two, true)));
+
+        assert_eq!(
+            peer_replay_state(audience),
+            PeerReplayState {
+                boot_epoch: Some(epoch_three),
+                cache_capability: Some(PeerReplayCapability::Revoked),
+            },
+            "a stale dynamic-cache proof must not cross a newer authenticated boot epoch"
+        );
+        clear_peer_capability(audience);
+    }
+
+    #[test]
+    fn interceptor_snapshot_prevents_delayed_legacy_response_from_revoking_capability() {
+        ensure_test_rpc_secret();
+        let audience = "capability-snapshot-client-test:9000";
+        clear_peer_capability(audience);
+        let boot_epoch = Uuid::new_v4();
+        let mut interceptor = test_interceptor_for(audience, false);
+        let request = interceptor
+            .call(rolling_mutation_request("Lock"))
+            .expect("legacy-compatible request must pass the interceptor");
+        assert_eq!(
+            request
+                .extensions()
+                .get::<PeerReplayStateSnapshot>()
+                .map(|snapshot| snapshot.0),
+            Some(PeerReplayState::default()),
+            "interceptor must preserve its unknown-state admission snapshot"
+        );
+
+        let capable_state = PeerReplayState {
+            boot_epoch: Some(boot_epoch),
+            cache_capability: Some(PeerReplayCapability::Capable { boot_epoch }),
+        };
+        PEER_REPLAY_STATES
+            .lock()
+            .expect("peer capability cache lock must not be poisoned")
+            .insert(audience.to_string(), capable_state);
+        let (metadata, extensions, body) = request.into_parts();
+        let mut request = HttpRequest::new(body);
+        *request.uri_mut() = "/node_service.NodeService/Lock".parse().expect("test RPC URI must parse");
+        *request.headers_mut() = metadata.into_headers();
+        *request.extensions_mut() = extensions;
+        let mut channel = ReplayScopeChannel::new(MissingProofService, Some(audience.to_string()));
+
+        futures::executor::block_on(channel.call(request)).expect("in-flight request response must complete");
+
+        assert_eq!(peer_replay_state(audience), capable_state);
+        clear_peer_capability(audience);
+    }
+
+    #[test]
+    fn strict_mode_keeps_unknown_peer_mutations_body_bound() {
+        ensure_test_rpc_secret();
+        let audience = "strict-body-digest-client-test:9000";
+        clear_peer_capability(audience);
+        let mut interceptor = test_interceptor_for(audience, true);
+        let request = rolling_mutation_request("WriteAll");
+        let expected_digest = request
+            .metadata()
+            .get("x-rustfs-content-sha256")
+            .and_then(|value| value.to_str().ok())
+            .expect("test mutation must carry its digest")
+            .to_string();
+
+        let request = interceptor.call(request).expect("interceptor call should succeed");
+
+        assert_eq!(
+            request
+                .metadata()
+                .get("x-rustfs-content-sha256")
+                .and_then(|value| value.to_str().ok()),
+            Some(expected_digest.as_str())
+        );
+        let nonce = request
+            .metadata()
+            .get("x-rustfs-rpc-nonce")
+            .and_then(|value| value.to_str().ok())
+            .and_then(|value| Uuid::parse_str(value).ok())
+            .expect("strict body-bound mutation must carry a UUID nonce");
+        assert!(!nonce.is_nil());
+        assert!(
+            crate::cluster::rpc::verify_tonic_rpc_signature(
+                audience,
+                "/node_service.NodeService/WriteAll",
+                request.metadata().as_ref(),
+            )
+            .is_ok()
+        );
+    }
+
+    #[test]
+    fn missing_capability_after_pin_fails_closed() {
+        ensure_test_rpc_secret();
+        let audience = "revoked-capability-client-test:9000";
+        let boot_epoch = Uuid::new_v4();
+        PEER_REPLAY_STATES
+            .lock()
+            .expect("peer capability cache lock must not be poisoned")
+            .insert(
+                audience.to_string(),
+                PeerReplayState {
+                    boot_epoch: Some(boot_epoch),
+                    cache_capability: Some(PeerReplayCapability::Capable { boot_epoch }),
+                },
+            );
+        let mut channel = ReplayScopeChannel::new(MissingProofService, Some(audience.to_string()));
+        futures::executor::block_on(channel.call(replay_scope_request(audience, "Ping")))
+            .expect("legacy response must complete before capability rejection");
+
+        let mut interceptor = test_interceptor_for(audience, false);
+        let error = interceptor
+            .call(rolling_mutation_request("Lock"))
+            .expect_err("a peer that loses its pinned capability must fail closed");
+
+        assert_eq!(error.code(), tonic::Code::Unauthenticated);
+        assert_eq!(error.message(), "RPC peer replay capability changed");
+        clear_peer_capability(audience);
+    }
+
     #[test]
     fn test_signature_interceptor_binds_audience_from_peer_uri() {
         let interceptor = TonicInterceptor::Signature(gen_tonic_signature_interceptor())
@@ -583,27 +1182,15 @@ mod tests {
     fn replay_scope_channel_uses_epoch_proof_before_sending_v3() {
         ensure_test_rpc_secret();
         let audience = "replay-scope-client-test:9000";
-        PEER_BOOT_EPOCHS
-            .lock()
-            .expect("peer epoch cache lock must not be poisoned")
-            .remove(audience);
+        clear_peer_capability(audience);
         let seen_headers = std::sync::Arc::new(Mutex::new(Vec::new()));
         let service = EpochProofService {
             audience: audience.to_string(),
+            include_capability: true,
             seen_headers: seen_headers.clone(),
         };
         let mut channel = ReplayScopeChannel::new(service, Some(audience.to_string()));
-        let make_request = || {
-            let mut request = HttpRequest::builder()
-                .uri("/node_service.NodeService/Ping")
-                .body(())
-                .expect("test RPC request must build");
-            request.headers_mut().extend(
-                gen_tonic_signature_headers(audience, "node_service.NodeService", "Ping", None)
-                    .expect("v2 test headers must mint"),
-            );
-            request
-        };
+        let make_request = || replay_scope_request(audience, "Ping");
 
         futures::executor::block_on(channel.call(make_request())).expect("first request must complete");
         futures::executor::block_on(channel.call(make_request())).expect("second request must complete");
@@ -619,10 +1206,7 @@ mod tests {
             headers[1].contains_key(RPC_REPLAY_SCOPE_VERSION_HEADER),
             "the second request must carry the replay-scoped v3 signature"
         );
-        PEER_BOOT_EPOCHS
-            .lock()
-            .expect("peer epoch cache lock must not be poisoned")
-            .remove(audience);
+        clear_peer_capability(audience);
     }
 
     #[test]