fix(kms): harden key list contract

.agents/skills/adversarial-validation/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: adversarial-validation
-description: Execute the Adversarial Validation policy from the root AGENTS.md — run the applicable reviewer roles with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, design proposal, or agent-instruction change that alters execution before declaring it done.
+description: Execute the Adversarial Validation policy from the root AGENTS.md — run the seven reviewer roles (correctness, simplicity, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
 ---
 
 # Adversarial Validation Playbooks
@@ -61,15 +61,14 @@ Null report example: "Attacked quorum-1 error reduction, exact max-keys listing
 
 ### Simplicity adversary
 
-- Smaller-diff attack: inspect production growth separately from tests, fixtures, generated code, and documentation; test additions have no growth budget. Rewrite the production diff mentally (or in scratch) as the minimal equivalent edit. Report a finding only with a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries; fewer lines alone are not evidence.
+- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
   - Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
-  - Evidence: AGENTS.md 'Change Style for Existing Logic' (conditional extraction rule, preserve sensitive control flow, canonical modules) and 'Reuse Before You Write'; the Adversarial Validation roles list charters this attack.
-- Reuse-and-necessity attack: for each new helper, search `crates/utils`, `crates/common`, the touched crate, the likely domain owner, and relevant direct dependencies. A reimplementation is a finding, but forced reuse with mismatched normalization, error, backoff, or durability semantics is also a finding. Demand a nameable trigger for new defensive branches. Tests remain subject to validity and near-duplicate coverage review, never a size limit.
-  - Where: Any diff adding helpers, branches on decoded/peer data, or tests
+  - Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Reuse Before You Write' (constants clause); the Adversarial Validation roles list charters the simplicity adversary with exactly this attack.
+- Reuse-and-necessity attack: for each new helper the diff introduces, run `ls crates/utils/src crates/common/src` and `rg -i 'fn \w*<term>'` over those dirs plus the touched crate (snake_case signatures — a full-text single-word grep drowns, a multi-word phrase returns nothing). A reimplementation of an existing workspace utility, or of plain std/tokio behavior no wrapper refines, is a finding — but so is forced reuse with mismatched semantics (normalization such as `clean` resolving `.`/`..` against raw S3 keys, error type, backoff, durability gating). For each new defensive branch, demand the nameable trigger and flag re-validation of what a validated upstream layer on the SAME path already guarantees — excluding the Cross-Cutting Domain Invariant patterns (nil/empty/absent UUID, dual metadata keys, unversioned-tier versionId) and re-checks before destructive actions, which are load-bearing even when redundant on the happy path. For each new test, flag near-duplicates pinning the same code path AND poison-value class as an existing test — boundary companions (n==max vs max+1, absent vs empty vs nil UUID, MetaObject vs MetaDeleteMarker) are never near-duplicates; the test-coverage skeptic playbook below mandates them.
+  - Where: Any diff adding helpers, branches on decoded/peer data, or tests; helper checks against crates/utils, crates/common, and the touched crate
   - Evidence: AGENTS.md 'Reuse Before You Write' and 'Necessary Code Only'; GHSA-f4vq-9ffr-m8m3 (normalization-asymmetry traversal — why forced reuse of normalizing helpers on raw keys is itself an attack); docs/operations/tier-ilm-debugging.md nil-versionId incident (why boundary re-checks are load-bearing).
-- Replacement-and-comment attack: when the diff introduces a replacement path or representation, trace all callers and flag a superseded in-scope path left behind without a compatibility requirement. Keep one canonical core behind compatibility adapters. Comments must state non-obvious invariants completely without narration or change history. Never demand unrelated deletion or trade away correctness, compatibility, or readability to reduce the diff.
 
-Null report example: "Separated production growth from tests/docs, tested a smaller equivalent, checked helper reuse and superseded paths, and found no break."
+Null report example: "Rewrote the diff as an in-place edit (no smaller equivalent exists), grepped both new helpers against crates/utils, crates/common, and the touched crate (no existing equivalent; call-site semantics checked), verified the two new defensive branches name concrete corrupt-input triggers, and checked the added tests against the existing suite (each pins a distinct poison-value class) — no break found."
 
 ### Security reviewer
 
@@ -196,9 +195,9 @@ Null report example: "Attacked dual-key metadata writes/removals against MinIO-o
 
 ### Performance reviewer
 
-- For each `.clone()` or allocation added to a per-request/per-object path, identify the copied data and execution frequency. Report a finding only for a concrete repeated cost or benchmark regression. Recommend borrowing, moving, `Bytes`/`Arc`, `Cow`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs.
+- For every `.clone()` the diff adds or moves onto a per-request/per-object path, open the cloned type and count heap fields (String, Vec, HashMap, Bytes). If >5 heap fields or it contains an EC block buffer, construct the cost: N concurrent PUTs x M objects -> N*M deep copies per second. Demand Arc-wrapping of heavy fields or pass-by-reference; also flag new `String` allocations in header/path/signature parsing where `&str`/`Cow<str>` suffices.
   - Where: crates/ecstore/src/set_disk/**, crates/ecstore/src/store*.rs, rustfs/src/storage/, crates/filemeta/, request handlers in rustfs/src/
-  - Evidence: crates/ecstore/AGENTS.md 'Allocation Discipline in Hot Paths'; .agents/skills/rust-code-quality/SKILL.md requires a concrete hot-path cost rather than a proxy metric
+  - Evidence: crates/ecstore/AGENTS.md 'Allocation Discipline in Hot Paths' (no Clone on >5-heap-field structs, Arc for large buffers, &str/Cow for temporary computations); .agents/skills/rust-code-quality/SKILL.md ranks 'unnecessary clone in hot path' as P1 must-fix
 - For every new sync_all/sync_data/fdatasync/flush/File::sync call in the diff, trace the call chain to DurabilityMode / RUSTFS_DRIVE_SYNC_ENABLE resolution (crates/ecstore/src/disk/local.rs:291 DurabilityMode, :347 resolve_durability_mode) and to per-bucket durability overrides. Construct the run where the operator sets mode=none (or legacy RUSTFS_DRIVE_SYNC_ENABLE=false) and the new fsync still fires — that is an ungated durability cost and a regression on 4KiB writes.
   - Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/bucket/durability.rs, crates/ecstore/src/set_disk/** (rename_data/commit paths), any crate doing tokio::fs or std::fs writes
   - Evidence: #4221 fsync work caused a measured -10% 4KiB write regression (#814 investigation), later gated; durability modes added in eaff17cad (#4397), per-bucket tier overrides in 13e48d93a (#4407); 2df315baf (#4493) shows even ancestor-dir fsyncs are routed through the gate
@@ -231,12 +230,12 @@ Null report example: "Attacked the new rename_data commit-section work, durabili
 
 ### Test-coverage skeptic
 
-- For every testable behavior claim in the PR description, revert that hunk and name the focused test or executable check that detects the revert. If no reasonable check exists, require the reason and residual risk from the validation floor. Especially verify the check exercises the real production path, not a lookalike helper.
+- For every behavior claim in the PR description, revert that hunk (git stash / manual undo of the changed lines) and name the exact test (`cargo test -p <crate> <test_name>`) that fails. If no test fails on revert, the behavior is untested — file a finding, not a note. Especially verify the test exercises the REAL production call path, not a lookalike helper.
   - Where: All crates; highest value in crates/ecstore, rustfs/src/storage, crates/heal
-  - Evidence: AGENTS.md testable-behavior exit criterion. Real bug: PR #4220 (ghost-directory cleanup) merged with green tests but its fix never executed on the real delete path — required follow-up rustfs#4307, backlog#798 stayed OPEN. The tests exercised a path the production flow never took.
+  - Evidence: AGENTS.md exit criterion 'Every behavior change has a test that fails without it'. Real bug: PR #4220 (ghost-directory cleanup) merged with green tests but its fix never executed on the real delete path — required follow-up rustfs#4307, backlog#798 stayed OPEN. The tests exercised a path the production flow never took.
 - Read each added/modified test and confirm it asserts the real outcome (returned value, stored bytes, error variant), not merely 'call succeeded' or 'no panic'. Flag any test whose only observable is that the function returned, and any `assert!(result.is_err())` that never checks WHICH error. Then check: does the test prove the exploit/failure form is denied, or only that the intended form still works?
   - Where: crates/e2e_test (security_boundary_test.rs pattern), and every #[cfg(test)] module in the diff
-  - Evidence: Commit dee8e4e63 (#4466) had to rewrite 277 lines of crates/e2e_test/src/security_boundary_test.rs because 'security boundary tests' passed without asserting real outcomes. .agents/skills/rust-code-quality/SKILL.md requires an observable failure criterion; .agents/skills/security-advisory-lessons/SKILL.md asks whether the exploit form is denied.
+  - Evidence: Commit dee8e4e63 (#4466) had to rewrite 277 lines of crates/e2e_test/src/security_boundary_test.rs because 'security boundary tests' passed without asserting real outcomes. .agents/skills/rust-code-quality/SKILL.md checklist: 'Every test function has at least one assert!'; .agents/skills/security-advisory-lessons/SKILL.md: 'Does the test prove the exploit form is denied, or only that the intended form still works?'
 - When the diff adds a boolean/mode parameter or config flag, find the test that fails if the flag's effect is INVERTED inside the changed function. Tests that were mechanically updated to pass `false`/default at every call site assert nothing about the new behavior. Execute the check: flip the flag's branch in the source and confirm at least one test goes red for each branch.
   - Where: crates/ecstore/src/set_disk/ (e.g. build_codec_streaming_part_reader), any function gaining a parameter
   - Evidence: Commit 05890d6e2 (#4573): PR #4560 added a 15th param allow_inplace_legacy_fallback; the arity tests were fixed by passing `false` everywhere — they assert Err outcomes independent of the flag, so the fallback behavior itself has no revert-detecting test at those sites.
@@ -261,7 +260,7 @@ Null report example: "Attacked the new rename_data commit-section work, durabili
 - For any pagination/limit/truncation change, construct the exact-boundary test: result count == max (page exactly full), max+1, and a delimiter re-fold that lands precisely on the page boundary — assert both the item count AND the is_truncated/continuation marker. Off-by-one at the page boundary is a recurring shipped bug here.
   - Where: crates/ecstore listing paths (list_objects, ListMultipartUploads, metacache), S3 handlers in rustfs/src/storage
   - Evidence: Two shipped boundary bugs: fefa70b31 (#4447) ListMultipartUploads returned one upload past max-uploads; d91f4d455 (#4538) delimiter re-fold of a full page lost the truncation flag. Both survived existing tests because no test pinned n == max exactly.
-- A green focused test is evidence only for the targets it builds. Follow the `AGENTS.md` validation tier: add package-scoped Clippy or broader test-target compilation only when changed targets, features, or dependents remain uncovered; do not require a workspace-wide build by default.
+- Green `cargo test -p <crate>` on the touched crate is not a coverage verdict for the diff's test code itself: run `cargo clippy --all-targets -p <crate>` and a workspace-wide test BUILD (`cargo check --workspace --all-targets` at minimum) before accepting the tests as evidence. Test-only code that doesn't compile workspace-wide or fails clippy has repeatedly broken main and masked whether tests ran at all.
   - Where: All crates; especially concurrent-branch merges into crates/ecstore
   - Evidence: #4322 broke main because only cargo test ran (field_reassign_with_default is clippy-only). b06f3df6b (#4441) and 05890d6e2 (#4573): test code broke the workspace test build (E0061) on main after textually-clean merges, failing CI for every open PR.
 
@@ -272,6 +271,7 @@ Null report example: "Attacked revert-detection for all 3 claimed behaviors (eac
 Probes are distilled from shipped bugs in git history (commit/PR references
 above), GitHub security advisories (see the security-advisory-lessons
 skill), scoped `AGENTS.md` rules, and invariants under `docs/architecture/`
-and `docs/operations/`. Line numbers drift; re-locate the invariant. Merge
-new incidents into an existing probe when they share a failure class; add a
-new probe only for a distinct attack, rather than growing the root policy.
+and `docs/operations/`. Line numbers drift; when a cited location no longer
+matches, trust the invariant and re-locate the code. When a new bug class
+ships, add a probe with its evidence here rather than growing the policy
+section in `AGENTS.md`.

.agents/skills/pr-creation-checker/SKILL.md

@@ -24,17 +24,15 @@ Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whe
 
 2. Inspect change scope
 - Review the diff and summarize what changed.
-- Inspect `git diff --stat` and `git diff --numstat`; assess production-code growth separately. Tests, fixtures, generated code, and documentation have no growth budget. Treat line counts as signals, not quotas.
 - Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
 - Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
-- Use the simplicity-adversary verdict instead of producing a per-symbol inventory. Block growth only when the review identifies duplication or gives a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries.
-- Confirm replacement implementations remove the superseded in-scope path or adapt compatibility at the boundary to one canonical core.
 - Scan the diff for newly added string literals and confirm whether they duplicate values already defined as constants/enums/typed wrappers in the same module or shared modules.
 - Treat introducing a new hardcoded literal where a project constant already exists as a likely regression risk; require either a refactor to reuse the constant or an explicit exception explanation in the PR body.
 
 3. Verify readiness requirements
-- Select checks from `AGENTS.md` "Verification Before PR" based on the final diff's risk tier. Do not replace a focused behavioral test with `make pre-commit`, or a required high-risk `make pre-pr` with a narrower gate.
-- For focused verification, state why the selected tier is sufficient and list the scope-specific commands in the PR body.
+- Require `make pre-commit` before marking PRs ready when the diff changes Rust code, product behavior, CI behavior, runtime configuration, security-sensitive logic, migrations, storage, auth, networking, or other high-risk paths.
+- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, allow focused verification instead of `make pre-commit` when it directly validates the changed surface.
+- For focused verification, explain why the full gate was not run and list the scope-specific commands in the PR body.
 - If `make` is unavailable, use the equivalent commands from `.config/make/`.
 - Add scope-specific verification commands when the changed area needs more than the baseline.
 - If required checks fail, stop and return `BLOCKED`.
@@ -83,14 +81,13 @@ Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whe
 
 ## Blocker rules
 
-- Return `BLOCKED` if the checks required by the `AGENTS.md` validation tier have not passed.
+- Return `BLOCKED` if a code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk change has not passed `make pre-commit`.
 - Return `BLOCKED` if a documentation-only, agent-instruction-only, or local developer-tooling-only change lacks focused verification for the changed surface.
 - Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
 - Return `BLOCKED` if required template sections are missing.
 - Return `BLOCKED` if the title/body is not in English.
 - Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
 - Return `BLOCKED` if the diff introduces string literals that should use existing constants but did not.
-- Return `BLOCKED` for production-code growth only when the review identifies a duplicated or superseded implementation, or supplies a concrete smaller design with equivalent semantics. Fewer lines alone are not evidence.
 
 ## Reference
 

.agents/skills/pr-creation-checker/references/pr-readiness-checklist.md

@@ -3,8 +3,8 @@
 - Confirm the branch is based on current `main`.
 - Confirm the diff matches the stated scope.
 - Confirm no secrets, logs, temp files, or unrelated refactors are included.
-- Confirm the checks required by the `AGENTS.md` validation tier passed.
-- For focused verification, confirm it covered the changed surface and the PR body explains why the selected tier is sufficient.
+- Confirm `make pre-commit` passed for code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk changes.
+- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, confirm focused verification covered the changed surface and the PR body explains why the full gate was not run.
 - Confirm extra verification commands are listed for risky changes.
 - Confirm the PR title uses Conventional Commits and stays within 72 characters.
 - Confirm the PR title does not use tool-specific prefixes such as `[codex]`.

.agents/skills/rust-code-quality/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: rust-code-quality
-description: Enforce Rust-specific code quality rules on every Rust change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
+description: Enforce Rust-specific code quality rules on every code change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
 ---
 
 # Rust Code Quality Gate
@@ -12,29 +12,27 @@ Use this skill on every Rust code change to enforce quality rules that `cargo cl
 1. Identify changed `.rs` files.
 2. Run automated checks on changed files.
 3. Run manual review checklist on the diff.
-4. Resolve or rebut every finding with evidence; P0/P1 findings cannot be deferred.
+4. Report findings; block merge if P0/P1 issues exist.
 
 ## Automated Checks
 
-Use these searches to find candidates in changed `.rs` files. Inspect syntax,
-`#[cfg(test)]` scope, and the changed hunk before reporting a finding; text
-filters do not reliably distinguish production code from tests.
+Run these on every changed `.rs` file (excluding test modules):
 
 ```bash
-# 1. unwrap/expect candidates
-rg -n '\.unwrap\(\)|\.expect\(' <changed-files>
+# 1. unwrap/expect in production code
+rg -n '\.unwrap\(\)|\.expect\(' <changed-files> | grep -v '#\[cfg(test)\]' | grep -v 'test' | grep -v 'bench'
 
 # 2. Silent type truncation via `as` cast
 rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
 
 # 3. String as error type
-rg -n 'Result<.*String>' <changed-files>
+rg -n 'Result<.*String>' <changed-files> | grep -v test
 
 # 4. Box<dyn Error> in public APIs
-rg -n 'Box<dyn.*Error' <changed-files>
+rg -n 'Box<dyn.*Error' <changed-files> | grep -v test
 
 # 5. println/eprintln in production
-rg -n 'println!\|eprintln!' <changed-files>
+rg -n 'println!\|eprintln!' <changed-files> | grep -v test
 
 # 6. Ordering::Relaxed usage (verify each is intentional)
 rg -n 'Ordering::Relaxed' <changed-files>
@@ -48,35 +46,37 @@ rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
 For every Rust code change, verify:
 
 ### Error Handling
-- [ ] Every production `unwrap()` or `expect()` is infallible by type or a checked invariant; explain only non-obvious invariants, using an existing type, a useful `expect` message, or a concise comment
+- [ ] No `unwrap()` or `expect()` in production code without justification comment
 - [ ] No `Result<_, String>` in public API signatures
-- [ ] Public library APIs use domain errors unless deliberate error erasure at a boundary is part of the contract
+- [ ] No `Box<dyn Error>` in public trait/struct methods
 - [ ] `Error::source()` is overridden when inner error is stored
-- [ ] Error messages are actionable without exposing secret input
+- [ ] Error messages are actionable (what failed, with what input)
 
 ### Type Safety
 - [ ] No silent `as` truncation (negative→unsigned, large→small)
-- [ ] Fallible numeric conversions use `TryFrom`/`try_into()` and return a typed error; clamp or saturate only when the domain explicitly requires it
-- [ ] Floating-point to integer conversion validates finiteness, sign, and range before conversion
+- [ ] `try_into()` or explicit clamping used for numeric conversions
+- [ ] No `f64 as usize` without prior clamping
 
 ### Concurrency
 - [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
 - [ ] No `tokio::sync` lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
-- [ ] Atomic read-modify-write uses the direct `fetch_*` operation when possible; use `compare_exchange` only for conditional updates
+- [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
 - [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
 
 ### Memory and Performance
-- [ ] On an identified hot path, report cloning or allocation only with a concrete per-request/per-object cost or benchmark signal
-- [ ] Prefer borrowing, moving, `Bytes`/`Arc`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs
+- [ ] No `.clone()` on structs with >5 heap-allocated fields in hot paths
+- [ ] `HashMap::with_capacity()` / `Vec::with_capacity()` used when size is known
+- [ ] Large buffers wrapped in `Arc` rather than cloned
+- [ ] Temporary string computations use `&str` or `Cow<str>` instead of `String`
 
 ### Recursion Safety
-- [ ] Recursion over untrusted, persisted, or otherwise unbounded input has a depth limit or uses iterative traversal
+- [ ] Recursive functions have a depth limit or use iterative traversal
 - [ ] Tree/cache traversals handle corrupted/cyclic input safely
 
 ### Testing
-- [ ] Tests have an observable failure criterion; delegated assertions, `#[should_panic]`, snapshot/property checks, and meaningful `Result` failures do not need a redundant `assert!`
-- [ ] Use `expect` only when its message improves failure diagnosis; do not add boilerplate to self-evident test setup
-- [ ] Test volume and line count are never treated as production-code growth
+- [ ] Every test function has at least one `assert!`
+- [ ] Tests use `.expect("context")` not bare `.unwrap()`
+- [ ] No `println!`/`eprintln!` in production code (use `tracing`)
 
 ### Serde
 - [ ] Structs from untrusted input have `#[serde(deny_unknown_fields)]`
@@ -88,18 +88,18 @@ For every Rust code change, verify:
 - [ ] New string literals don't duplicate existing constants
 
 ### Reuse and Necessity
-- [ ] No new helper duplicates `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, a relevant direct dependency, or plain std/tokio behavior; reused helpers match the call site's semantics
+- [ ] No new helper duplicating an existing workspace utility (`crates/utils`, `crates/common`, the touched crate) or plain std/tokio behavior no wrapper refines; reused helpers match the call site's semantics (normalization, error type, backoff, durability gating)
 - [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
 - [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
-- [ ] Comments avoid narration and change history while completely stating non-obvious lock, `SAFETY`, durability, compatibility, and unwrap invariants
+- [ ] No comments narrating the next line, restating a signature, or describing the change itself (invariant comments — lock ordering, `SAFETY`, unwrap justification — are not narration)
 - [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
 
 ## Severity Classification
 
-- **P0 (Block merge)**: demonstrated data loss, security breach, remote crash, or deadlock
-- **P1 (Must fix)**: concrete correctness, compatibility, or material hot-path regression
-- **P2 (Should fix)**: avoidable duplication or maintainability issue with a concrete simpler replacement
-- **P3 (Nice to fix)**: local style or clarity issue with no behavioral risk
+- **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
+- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method, `unwrap_or_default()` on a domain-required value (metadata, quorum, version id)
+- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`, new helper duplicating an existing workspace utility, defensive branch with no nameable trigger (corrupt or stale persisted/peer data is always a nameable trigger for boundary-crossing values), near-duplicate test, redundant error re-wrapping
+- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`, narrating comment
 
 ## Output Template
 
@@ -107,10 +107,10 @@ For every Rust code change, verify:
 ## Rust Code Quality Report
 
 ### Automated Scan
-- unwrap/expect candidates inspected: N
-- numeric-cast candidates inspected: N
-- error-type candidates inspected: N
-- output-macro candidates inspected: N
+- unwrap/expect in production: N found
+- as casts: N found
+- String errors: N found
+- println/eprintln: N found
 
 ### Findings
 - [P1] `path:line` — description

.agents/skills/rust-code-quality/references/checklist.md

@@ -0,0 +1,52 @@
+# Rust Code Quality Checklist
+
+Use this as a quick pre-merge checklist for every Rust code change.
+
+## Critical (P0 — block merge)
+
+| Check | Command |
+|-------|---------|
+| No `unwrap()` in request/storage hot path | `rg '\.unwrap\(\)' <files> \| grep -v test` |
+| No `as` truncation on user input | `rg ' as (u32\|usize\|i32)' <files>` |
+| Lock order consistent across call sites | Manual: trace all lock acquisitions |
+| Recursive functions have depth limit | Manual: check for `max_depth` or iterative pattern |
+| No `panic!`/`unwrap_or_else(panic!)` in production | `rg 'panic!\|unwrap_or_else.*panic' <files> \| grep -v test` |
+
+## High (P1 — must fix)
+
+| Check | Command |
+|-------|---------|
+| No `Result<_, String>` in public API | `rg 'Result<.*String>' <files> \| grep -v test` |
+| No `Box<dyn Error>` in public trait | `rg 'Box<dyn.*Error' <files> \| grep -v test` |
+| No unnecessary `.clone()` in hot path | Manual: check loops and per-request paths |
+| `Error::source()` implemented when inner error stored | Manual: check `impl Error` |
+| No `eprintln!`/`println!` in production | `rg 'println!\|eprintln!' <files> \| grep -v test` |
+
+## Medium (P2 — should fix)
+
+| Check | Command |
+|-------|---------|
+| Tests have assertions | Manual: check for `assert` in test functions |
+| `HashMap`/`Vec` use `with_capacity` when size known | Manual: check `::new()` in loops |
+| No `#![allow(dead_code)]` at crate root | `rg 'allow.dead_code' <files> \| grep 'lib.rs'` |
+| Serde structs from untrusted input have `deny_unknown_fields` | Manual: check `#[derive(Deserialize)]` |
+
+## Low (P3 — nice to fix)
+
+| Check | Command |
+|-------|---------|
+| No camelCase statics | `rg 'static ref [a-z]' <files>` |
+| `Arc::ptr_eq` instead of `as_ptr + ptr::eq` | `rg 'as_ptr\|ptr::eq' <files>` |
+| Public functions have doc comments | `rg 'pub fn' <files> \| grep -v '///'` |
+
+## Quick One-Liner
+
+```bash
+# Run all automated checks on changed files
+CHANGED=$(git diff --name-only HEAD~1 -- '*.rs' | grep -v test | grep -v bench)
+echo "=== unwrap/expect ===" && rg -c '\.unwrap\(\)|\.expect\(' $CHANGED 2>/dev/null
+echo "=== as casts ===" && rg -c ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' $CHANGED 2>/dev/null
+echo "=== String errors ===" && rg -c 'Result<.*String>' $CHANGED 2>/dev/null
+echo "=== println ===" && rg -c 'println!|eprintln!' $CHANGED 2>/dev/null
+echo "=== Ordering::Relaxed ===" && rg -c 'Ordering::Relaxed' $CHANGED 2>/dev/null
+```

.agents/skills/rustfs-logging-governance/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: rustfs-logging-governance
-description: Standardize and review RustFS logging with structured `tracing` events, lower noise on hot paths, preserve security-sensitive diagnostics, and extend guardrails to prevent legacy logging patterns from returning. Use whenever a change adds or edits any `tracing` macro call (`error!`/`warn!`/`info!`/`debug!`/`trace!`/`#[instrument]`) — including a single log line added in passing while fixing unrelated logic, which is how most new log sites enter the repo — and when reviewing RustFS logs, startup/config diagnostics, cloud metadata logs, request validation logs, or `scripts/check_logging_guardrails.sh`.
+description: Standardize and review RustFS logging with structured `tracing` events, lower noise on hot paths, preserve security-sensitive diagnostics, and extend guardrails to prevent legacy logging patterns from returning. Use when editing or reviewing RustFS logs, startup/config diagnostics, cloud metadata logs, request validation logs, or `scripts/check_logging_guardrails.sh`.
 ---
 
 # RustFS Logging Governance

.agents/skills/security-advisory-lessons/SKILL.md

@@ -66,23 +66,14 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 
 ### STS, OIDC, and federation flows
 - Every STS endpoint must have an explicit authentication story: SigV4 where required, OIDC token verification for web identity, and role/session policy validation before issuing credentials.
-- For web identity, the JWT is the credential; exemption from SigV4 is not itself an authentication bypass. Treat pre-verification claims only as untrusted routing hints, bound token size, normalize public failures, rate-limit discovery, and issue credentials only after signature, issuer, audience, and expiration checks.
 - JWT session tokens must be signed and verified by a trusted issuer/key path, not by service-account-controlled material or a reused root secret.
 - JWT verification must enforce required claims and expiration for every bearer token path; "allow missing exp" is never acceptable for user-presented credentials.
 - Public OIDC bootstrap and callback routes must treat `Host`, `X-Forwarded-Proto`, redirect targets, `state`, and callback parameters as untrusted; credential-bearing redirects require a configured, allowlisted origin.
 - OIDC discovery and validation URLs are SSRF sinks. Resolve and classify hostnames at connection time, reject rebinding to loopback/private/link-local ranges, and do not rely on literal string checks.
 
-### IAM policy conditions and plugins
-- Treat request headers as attacker-controlled even after SigV4; callers sign their own spoofed headers. Do not merge them into server-derived condition keys such as identity, groups, version ID, signature version, JWT, or LDAP claims.
-- Keep the condition-key namespace explicit. Reserved server-derived keys must reject or ignore colliding headers, while intentional request-header keys such as `s3:x-amz-*` remain available.
-- Quantified IAM condition tests need partially overlapping multi-value sets. Fully contained and fully disjoint sets cannot distinguish `ForAllValues` from `ForAnyValue` bugs.
-- External policy plugins must receive the same security context as built-in policy evaluation. If OPA or another plugin depends on existing object tags, load and pass `ExistingObjectTag/*` before the plugin decision.
-
-### S3 object actions, copy, multipart, and presigned POST
-- Version-aware object requests need version-aware actions. Explicit `versionId` reads and copy sources must authorize `s3:GetObjectVersion`, not only `s3:GetObject`.
+### S3 copy, multipart, and presigned POST
 - Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
 - Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
-- Fallbacks from version actions to non-version actions must still pass the same public-access-block, anonymous-deny, and post-authorization gates as a direct allow.
 - Presigned POST policies are server-side contracts. Enforce `content-length-range`, key prefix, exact metadata/content-type, and all signed policy conditions.
 
 ### Protocol frontends and IAM parity
@@ -141,11 +132,6 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 - When touching reader/writer wrappers such as hashing, encryption, compression, or warp readers, verify wrapper order and inspect stored bytes in regression tests.
 - Avoid helper shortcuts that unwrap nested readers and accidentally bypass encryption or integrity layers.
 
-### Object Lock and retention invariants
-- Object Lock state must fail closed when bucket metadata is unreadable, fabricated, or unparsable. Only a confirmed absence of Object Lock configuration may permit unprotected deletes or writes.
-- Do not collapse metadata read faults, missing persisted metadata, parse failures, and genuinely absent Object Lock config into one "not configured" result.
-- Retention enforcement must cover foreground deletes, batch deletes, force-delete helpers, default-retention materialization on PUT, lifecycle expiry, scanner sweeps, and all-versions expiry.
-
 ## Review Prompts
 
 Use these prompts while reviewing a diff:
@@ -162,9 +148,5 @@ Use these prompts while reviewing a diff:
 - Does this outbound validation path resolve attacker-supplied hostnames and reject private, loopback, link-local, and rebound addresses at the actual connection boundary?
 - Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
 - Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
-- Does an explicit object version, fallback action, or plugin authorization path pass through the same action and post-authorization gates as the direct S3 path?
-- Can a caller-controlled header populate a condition key that should be derived only by the server?
-- Do condition tests include partially overlapping multi-value inputs for quantified operators?
-- Does unreadable bucket metadata make Object Lock or retention enforcement fail closed rather than disappear?
 - Does a preview or browser-surface fix preserve the original security invariant when adding alternate viewers or file-type detection?
 - Does the test prove the exploit form is denied, or only that the intended form still works?

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -35,21 +35,12 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 ### STS, OIDC, and federation flows
 
 - `GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
-- `GHSA-jxrr-r6pv-h958`: unsigned JWT issuer data was decoded before verification to select an OIDC provider, and distinguishable failures could expose provider configuration. Lesson: web-identity routing may be unauthenticated, but pre-verification claims are untrusted routing hints; bound and rate-limit the request, normalize public errors, and verify signature, issuer, audience, and expiration before issuing credentials.
-- `GHSA-ccrv-v8v9-ch9q`, `GHSA-48rf-7j3q-3hfv`, and `GHSA-xvfh-7c9g-hpw2`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
+- `GHSA-ccrv-v8v9-ch9q` and `GHSA-48rf-7j3q-3hfv`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
 - `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
 - `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
 
-### IAM policy conditions and external policy plugins
+### S3 copy, multipart, and upload policy validation
 
-- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
-- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
-- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
-
-### S3 object actions, copy, multipart, and upload policy validation
-
-- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
-- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
 - `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
 - `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
 - `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
@@ -68,7 +59,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 ### Secrets, defaults, and cryptographic misuse
 
-- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, `GHSA-63xc-c3w3-m2cf`, and `GHSA-ch63-6q4v-hwp5`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
+- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, and `GHSA-63xc-c3w3-m2cf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
 - `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
 - `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
 - `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
@@ -101,10 +92,6 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
 
-### Object Lock and retention invariants
-
-- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
-
 ### Serde deserialization and input validation
 
 - No `#[serde(deny_unknown_fields)]` found across the entire codebase. Lesson: all structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs) should have `#[serde(deny_unknown_fields)]` to reject malformed or adversarial payloads.
@@ -120,13 +107,11 @@ Use these targeted searches when a diff touches security-sensitive code:
 rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
 rg -n "authorize_operation|FtpsDriver|SftpDriver|RETR|MKD|SIZE|MDTM|CreateBucket|GetObject|HeadObject" crates/protocols rustfs
 rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
-rg -n "ListBucketVersions|GetObjectVersion|versionId|VersionId|ExistingObjectTag|ForAllValues|ForAnyValue|POLICY_PLUGIN|opa" rustfs crates
 rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
 rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
 rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
 rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
 rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
-rg -n "ObjectLock|object_lock|retention|COMPLIANCE|GOVERNANCE|delete_prefix|lifecycle|scanner" rustfs crates
 rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 ```
 
@@ -136,12 +121,9 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 - Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
 - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
-- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
-- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
 - Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
 - IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
 - RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
-- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.

.config/make/lint-fmt.mak

@@ -60,16 +60,6 @@ body-cache-whitelist-check: ## Check the body-cache eligibility gate stays a fai
 	@echo "🧱 Checking body-cache whitelist guard..."
 	./scripts/check_body_cache_whitelist.sh
 
-.PHONY: s3s-footprint-check
-s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
-	@echo "📦 Checking s3s footprint ratchet..."
-	./scripts/check_s3s_footprint.sh
-
-.PHONY: fips-wording-check
-fips-wording-check: ## Check outward docs do not make unsupported FIPS claims
-	@echo "📣 Checking FIPS wording guard..."
-	./scripts/check_fips_wording.sh
-
 .PHONY: log-analyzer-rules-check
 log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
 	@echo "🩺 Checking log-analyzer rule anchors..."

.config/make/pre-commit.mak

@@ -19,13 +19,13 @@ planning-docs-check: ## Check that no planning-type documents are committed
 	./scripts/check_no_planning_docs.sh
 
 .PHONY: pre-commit
-pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
+pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
 	@echo "✅ All pre-commit checks passed!"
 
 .PHONY: pre-pr
-pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
+pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
 	@echo "✅ All pre-PR checks passed!"
 
 .PHONY: dev-check
-dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
+dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
 	@echo "✅ Fast development checks passed!"

.config/make/tests.mak

@@ -25,7 +25,6 @@ TEST_THREADS ?= 1
 script-tests: ## Run shell script tests
 	@echo "Running script tests..."
 	./scripts/test_build_rustfs_options.sh
-	./scripts/test_docker_runtime_timezone.sh
 	./scripts/test_entrypoint_credentials.sh
 	./scripts/test_internode_grpc_ab_bench.sh
 	./scripts/test_object_batch_bench_enhanced.sh

.config/nextest.toml

@@ -9,8 +9,6 @@
 #   * bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
 #     uses the shared multipart fixture and a deterministic uploadId-lock
 #     handoff, so it must not overlap another process mutating that fixture.
-#   * bucket::metadata_sys::tests::concurrent_config_writes_from_separate_nodes_do_not_lose_writes
-#     uses the shared transaction lock and must not overlap other ecstore tests.
 #
 # serial_test's #[serial] attribute does NOT serialize these across runs:
 # nextest executes each test in its own process, where the in-process
@@ -29,13 +27,10 @@
 
 [test-groups]
 ecstore-serial-flaky = { max-threads = 1 }
-embedded-test-ports = { max-threads = 1 }
-e2e-vault = { max-threads = 1 }
 
 # Reliability / fault-injection e2e tests each spawn a single-node 4-disk RustFS
 # server and manipulate its disk directories at runtime (crates/e2e_test:
-# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13, and
-# replacement_privileged_e2e_test when explicitly run as root on Linux). They
+# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13). They
 # are correct in isolation but resource-heavy; serialize them under nextest's
 # process boundary (serial_test's #[serial] does not cross it) so several 4-disk
 # servers never run at once. ci-7's nightly picks these up via the e2e suite;
@@ -45,7 +40,7 @@ e2e-inline-boundaries = { max-threads = 1 }
 
 # --- default profile (local): serialize the flaky groups, never retry --------
 [[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
+filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
 test-group = 'ecstore-serial-flaky'
 
 # Serialize the multipart crash-consistency scenarios (dist-2, backlog#1150):
@@ -57,53 +52,23 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
 test-group = 'ecstore-serial-flaky'
 
-# The production-handler relocation regression builds an isolated 8-disk,
-# 2-pool store and commits a 72 MiB multipart object. Keep that cross-disk IO
-# from overlapping the ecstore commit fixtures above.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
-test-group = 'ecstore-serial-flaky'
-
-# Embedded integration-test binaries discover an ephemeral port and release
-# the probe listener before RustFS binds it. Serialize that cross-process
-# TOCTOU window; retries would only hide real startup failures.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
-test-group = 'embedded-test-ports'
-
 # Serialize the durable manual-transition checkpoint test across nextest's
 # process boundary; it mutates bucket lifecycle metadata and is not quarantined.
 [[profile.default.overrides]]
 filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
 test-group = 'ecstore-serial-flaky'
 
-# Serialize the bucket-incarnation / lifecycle-fence tests. They drive
-# init_bucket_metadata_sys and bucket_metadata_sys_of, i.e. process-global
-# OnceLock state that serial_test's #[serial] cannot protect across nextest's
-# process boundary, and they delete+recreate buckets — the same shape that
-# raced into InsufficientWriteQuorum in backlog#937. Preventive only, no
-# retries. The matching ci-profile override is after [profile.ci].
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
-test-group = 'ecstore-serial-flaky'
-
 # Serialize the 4-disk reliability / degraded-read e2e tests (see the
 # e2e-reliability test-group note above). The matching ci-profile override is at
 # the end of the file, after [profile.ci] is declared.
 [[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
+filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
 test-group = 'e2e-reliability'
 
 [[profile.default.overrides]]
 filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
 test-group = 'e2e-inline-boundaries'
 
-# Vault KMS tests share the fixed dev-server port 8200. serial_test's #[serial]
-# does not cross nextest process boundaries, so keep these tests in one group.
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^kms::kms_vault_test::/)'
-test-group = 'e2e-vault'
-
 # ---------------------------------------------------------------------------
 # ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
 # ---------------------------------------------------------------------------
@@ -139,9 +104,9 @@ filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(
 test-group = 'ecstore-serial-flaky'
 retries = 2
 
-# Keep deterministic ECStore write handoffs isolated across nextest processes.
+# Keep the deterministic multipart handoff isolated across nextest processes.
 [[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes))'
+filter = 'package(rustfs-ecstore) & test(concurrent_resend_same_part_commits_one_generation)'
 test-group = 'ecstore-serial-flaky'
 
 # QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
@@ -156,7 +121,7 @@ retries = 2
 # quarantine: no retries, just single-threaded so several 4-disk servers never
 # run concurrently when ci-7's nightly runs the full e2e suite.
 [[profile.ci.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
+filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
 test-group = 'e2e-reliability'
 
 # Serialize the multipart crash-consistency scenarios under the ci profile too
@@ -166,28 +131,12 @@ test-group = 'e2e-reliability'
 filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
 test-group = 'ecstore-serial-flaky'
 
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
-test-group = 'ecstore-serial-flaky'
-
-# Match the default-profile embedded test isolation without quarantining or
-# retrying failures in CI.
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
-test-group = 'embedded-test-ports'
-
 # Serialize the durable manual-transition checkpoint test under the ci profile
 # too. No retries: failures stay visible.
 [[profile.ci.overrides]]
 filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
 test-group = 'ecstore-serial-flaky'
 
-# Serialize the bucket-incarnation / lifecycle-fence tests under the ci profile
-# too (see the matching default-profile override near the top). No retries.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
-test-group = 'ecstore-serial-flaky'
-
 # ---------------------------------------------------------------------------
 # e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
 # ---------------------------------------------------------------------------
@@ -219,7 +168,7 @@ test-group = 'ecstore-serial-flaky'
 # the nightly profile derives its set as "the replication module MINUS this
 # allowlist", so any new replication test lands in nightly by default (never
 # silently unrun) until it is explicitly blessed as fast here. Keep the two
-# regexes byte-identical. Count invariant: 20 here + 49 nightly = 69 total
+# regexes byte-identical. Count invariant: 20 here + 28 nightly = 48 total
 # (authority: `cargo nextest list`; docs/testing/e2e-suite-inventory.md).
 # HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
 # (#4724) because they set a loopback (127.0.0.1) replication target that the
@@ -255,7 +204,7 @@ test-group = 'ecstore-serial-flaky'
 [profile.e2e-smoke]
 default-filter = """
     package(e2e_test) & (
-        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
+        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
         | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
         | test(/^reliant::lifecycle::/)
         | test(/^reliant::tiering::/)
@@ -263,17 +212,6 @@ default-filter = """
 """
 fail-fast = false
 
-[profile.e2e-smoke.junit]
-path = "junit.xml"
-
-# The pagination boundary cases can stall when a server/listing regression
-# prevents the continuation request from completing. Keep the timeout scoped
-# to those known failure modes so legitimate lifecycle/tiering waits retain
-# their test-level timing budget.
-[[profile.e2e-smoke.overrides]]
-filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
-slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
-
 # ---------------------------------------------------------------------------
 # e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
 # ---------------------------------------------------------------------------
@@ -281,12 +219,10 @@ slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
 # tests that are unfit for the per-PR e2e-smoke gate:
 #
 #   * 2 remote-target TLS validation tests.
-#   * 15 bucket-replication data-plane/helper tests — they PUT/delete objects
-#     and poll until source and target converge; two replicate over HTTPS,
-#     six pin SSE replication contracts (managed SSE-S3/SSE-KMS re-encrypt on
-#     the target incl. multipart and the resync path, SSE-C and
-#     target-without-KMS stay fail-closed), and one guards event/history
-#     observers.
+#   * 12 bucket-replication data-plane/helper tests — they PUT/delete objects
+#     and poll until source and target converge; two replicate over HTTPS, two
+#     pin active SSE failure contracts, and one guards event/history observers.
+#     The SSE-S3 contract remains ignored under backlog#1291.
 #   * 12 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
 #     servers and drives the cross-process site-replication control plane.
 #   * 1 `_real_three_node` site-replication test.
@@ -345,16 +281,16 @@ path = "junit.xml"
 #     object_lambda) — too heavy for the merge budget; they run in ci-7's
 #     nightly 4-node lane.
 #   * replication_extension_test — repl-1 already splits it into the PR
-#     `e2e-smoke` (20 fast) and `e2e-repl-nightly` (49 slow) lanes and reserves
+#     `e2e-smoke` (20 fast) and `e2e-repl-nightly` (27 slow) lanes and reserves
 #     it for those, so e2e-full does not double-run it.
 #   * #[ignore]d tests — nextest skips them by default (no --run-ignored); the
 #     manual-localhost:9000 reliant/policy tests are ci-13's migration.
 #
 # Each e2e test spawns its own single-node rustfs server on a random port with
 # an isolated temp dir (crates/e2e_test/src/common.rs), so the set is
-# parallel-safe — the same property e2e-smoke relies on. The exceptions are the
-# 4-disk reliability / degraded-read fault-injection tests and the fixed-port
-# Vault tests, both serialized below.
+# parallel-safe — the same property e2e-smoke relies on. The exception is the
+# 4-disk reliability / degraded-read fault-injection tests, serialized below
+# (identical to the ci profile) so several 4-disk servers never run at once.
 # KNOWN-FAILURE EXCLUSIONS (characterization run 29381309848, 2026-07-15:
 # 341 ran / 32 failed on the suites' first automated run ever). Deterministic
 # product failures cannot be quarantined away with retries, so each family is
@@ -384,13 +320,9 @@ path = "junit.xml"
 # quarantine: no retries, just single-threaded so several 4-disk servers never
 # run concurrently.
 [[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
+filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
 test-group = 'e2e-reliability'
 
 [[profile.e2e-full.overrides]]
 filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
 test-group = 'e2e-inline-boundaries'
-
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^kms::kms_vault_test::/)'
-test-group = 'e2e-vault'

.docker/observability/README.md

@@ -170,10 +170,6 @@ Important behavior notes:
 
 - Logs and metrics usually appear during startup, so seeing those two signals
   first is expected.
-- The OpenTelemetry bridge sends `tracing` fields as log attributes. Loki stores
-  those attributes as structured metadata, and the Collector also mirrors the
-  common troubleshooting fields into the log line so simple line filters can
-  find them.
 - Visible trace data usually requires real HTTP/S3/gRPC request traffic after
   startup, because request-path spans are created on demand.
 - `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
@@ -199,17 +195,6 @@ curl -I http://127.0.0.1:9000/health/ready
 # Jaeger:  http://localhost:16686
 ```
 
-For a structured RustFS log such as an inter-node RPC authentication failure,
-the Loki line now includes fields such as `event`, `component`, `subsystem`,
-`failure_reason`, `rpc_service`, `rpc_method`, and `expected_audience`. Useful
-LogQL checks:
-
-```logql
-{service_name="RustFS"} |= "RPC signature verification failed"
-{service_name="RustFS"} |= "failure_reason="
-{service_name="RustFS"} | failure_reason != ""
-```
-
 If logs and metrics are present but traces are sparse, the most common cause is
 "no real request traffic yet" or "`info` level filtered nested spans", not an
 OTLP routing failure.

.docker/observability/README_ZH.md

@@ -169,7 +169,6 @@ RustFS 会自动在该基础 URL 后补全：
 需要注意：
 
 - 启动阶段通常会先看到日志和指标，因此“先有日志/指标、后有 trace”是正常现象。
-- OpenTelemetry bridge 会把 `tracing` 字段作为日志 attributes 发送。Loki 会将这些 attributes 存为 structured metadata，同时 Collector 会把常用排障字段镜像进日志行，方便用简单的行内容过滤直接查到。
 - 可见的 trace 数据通常依赖启动后的真实 HTTP/S3/gRPC 请求流量，因为请求路径上的 span 是按需创建的。
 - `RUSTFS_OBS_LOGGER_LEVEL=info` 会保留顶层请求 span，但会过滤掉很多 `debug` 级别的嵌套 span。
   如果 Tempo 或 Jaeger 中的 trace 看起来很稀疏，建议先改成 `RUSTFS_OBS_LOGGER_LEVEL=debug`，再判断是否是 collector 或 Tempo 问题。
@@ -193,14 +192,6 @@ curl -I http://127.0.0.1:9000/health/ready
 # Jaeger:  http://localhost:16686
 ```
 
-对于 RustFS 结构化日志，例如节点间 RPC 鉴权失败，Loki 日志行现在会包含 `event`、`component`、`subsystem`、`failure_reason`、`rpc_service`、`rpc_method`、`expected_audience` 等字段。常用 LogQL 检查：
-
-```logql
-{service_name="RustFS"} |= "RPC signature verification failed"
-{service_name="RustFS"} |= "failure_reason="
-{service_name="RustFS"} | failure_reason != ""
-```
-
 如果日志和指标已经正常，但 trace 仍然稀疏，最常见的原因通常是
 “还没有真实请求流量”或“`info` 级别过滤了嵌套 span”，而不是 OTLP 路由失败。
 

.docker/observability/grafana/dashboards/rustfs.json

@@ -11500,831 +11500,6 @@
       ],
       "title": "Compression Operations Rate",
       "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 332
-      },
-      "id": 531,
-      "panels": [],
-      "title": "Metrics Dimensions Drilldown",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${datasource}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisColorMode": "text",
-            "axisLabel": "",
-            "axisPlacement": "auto",
-            "barAlignment": 0,
-            "barWidthFactor": 0.6,
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "gradientMode": "none",
-            "hideFrom": {
-              "legend": false,
-              "tooltip": false,
-              "viz": false
-            },
-            "insertNulls": false,
-            "lineInterpolation": "smooth",
-            "lineWidth": 1,
-            "pointSize": 5,
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "showPoints": "never",
-            "showValues": false,
-            "spanNulls": false,
-            "stacking": {
-              "group": "A",
-              "mode": "none"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": 0
-              }
-            ]
-          },
-          "unit": "ops"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 333
-      },
-      "id": 532,
-      "options": {
-        "legend": {
-          "calcs": [
-            "lastNotNull",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "right",
-          "showLegend": true
-        },
-        "tooltip": {
-          "hideZeros": false,
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "pluginVersion": "12.3.2",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "A",
-          "expr": "sum by (server, name, type) (rate(rustfs_api_requests_requests_total_by_server{job=~\"$job\",server=~\"$server\",name=~\"$api\"}[$__rate_interval]))",
-          "legendFormat": "{{server}} | {{name}} | {{type}}"
-        }
-      ],
-      "title": "API Requests by Server and API",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${datasource}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisColorMode": "text",
-            "axisLabel": "",
-            "axisPlacement": "auto",
-            "barAlignment": 0,
-            "barWidthFactor": 0.6,
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "gradientMode": "none",
-            "hideFrom": {
-              "legend": false,
-              "tooltip": false,
-              "viz": false
-            },
-            "insertNulls": false,
-            "lineInterpolation": "smooth",
-            "lineWidth": 1,
-            "pointSize": 5,
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "showPoints": "never",
-            "showValues": false,
-            "spanNulls": false,
-            "stacking": {
-              "group": "A",
-              "mode": "none"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": 0
-              }
-            ]
-          },
-          "unit": "s"
-        },
-        "overrides": [
-          {
-            "matcher": {
-              "id": "byFrameRefID",
-              "options": "A"
-            },
-            "properties": [
-              {
-                "id": "unit",
-                "value": "none"
-              }
-            ]
-          }
-        ]
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 333
-      },
-      "id": 533,
-      "options": {
-        "legend": {
-          "calcs": [
-            "lastNotNull",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "right",
-          "showLegend": true
-        },
-        "tooltip": {
-          "hideZeros": false,
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "pluginVersion": "12.3.2",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "A",
-          "expr": "max by (server, drive, pool_index, set_index, drive_index, state) (rustfs_system_drive_runtime_state{job=~\"$job\",server=~\"$server\",drive=~\"$drive\"})",
-          "legendFormat": "{{server}} | {{drive}} | p{{pool_index}}/s{{set_index}}/d{{drive_index}} | {{state}}"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "B",
-          "expr": "max by (server, drive, pool_index, set_index, drive_index) (rustfs_system_drive_offline_duration_seconds{job=~\"$job\",server=~\"$server\",drive=~\"$drive\"})",
-          "legendFormat": "{{server}} | {{drive}} | offline seconds"
-        }
-      ],
-      "title": "Drive Runtime State and Offline Duration",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${datasource}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisColorMode": "text",
-            "axisLabel": "",
-            "axisPlacement": "auto",
-            "barAlignment": 0,
-            "barWidthFactor": 0.6,
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "gradientMode": "none",
-            "hideFrom": {
-              "legend": false,
-              "tooltip": false,
-              "viz": false
-            },
-            "insertNulls": false,
-            "lineInterpolation": "smooth",
-            "lineWidth": 1,
-            "pointSize": 5,
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "showPoints": "never",
-            "showValues": false,
-            "spanNulls": false,
-            "stacking": {
-              "group": "A",
-              "mode": "none"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": 0
-              }
-            ]
-          },
-          "unit": "ops"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 341
-      },
-      "id": 534,
-      "options": {
-        "legend": {
-          "calcs": [
-            "lastNotNull",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "right",
-          "showLegend": true
-        },
-        "tooltip": {
-          "hideZeros": false,
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "pluginVersion": "12.3.2",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "A",
-          "expr": "sum by (server, drive, pool_index, set_index, drive_index, api) (rate(rustfs_system_drive_api_calls_total{job=~\"$job\",server=~\"$server\",drive=~\"$drive\",api=~\"$drive_api\"}[$__rate_interval]))",
-          "legendFormat": "{{server}} | {{drive}} | p{{pool_index}}/s{{set_index}}/d{{drive_index}} | {{api}}"
-        }
-      ],
-      "title": "Drive API Calls by Operation",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${datasource}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisColorMode": "text",
-            "axisLabel": "",
-            "axisPlacement": "auto",
-            "barAlignment": 0,
-            "barWidthFactor": 0.6,
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "gradientMode": "none",
-            "hideFrom": {
-              "legend": false,
-              "tooltip": false,
-              "viz": false
-            },
-            "insertNulls": false,
-            "lineInterpolation": "smooth",
-            "lineWidth": 1,
-            "pointSize": 5,
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "showPoints": "never",
-            "showValues": false,
-            "spanNulls": false,
-            "stacking": {
-              "group": "A",
-              "mode": "none"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": 0
-              }
-            ]
-          },
-          "unit": "ops"
-        },
-        "overrides": [
-          {
-            "matcher": {
-              "id": "byFrameRefID",
-              "options": "B"
-            },
-            "properties": [
-              {
-                "id": "unit",
-                "value": "short"
-              },
-              {
-                "id": "custom.axisPlacement",
-                "value": "right"
-              }
-            ]
-          }
-        ]
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 341
-      },
-      "id": 535,
-      "options": {
-        "legend": {
-          "calcs": [
-            "lastNotNull",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "right",
-          "showLegend": true
-        },
-        "tooltip": {
-          "hideZeros": false,
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "pluginVersion": "12.3.2",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "A",
-          "expr": "sum by (server, source, state) (rate(rustfs_scanner_source_work_total{job=~\"$job\",server=~\"$server\",source=~\"$scanner_source\"}[$__rate_interval]))",
-          "legendFormat": "{{server}} | {{source}} | {{state}}"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "B",
-          "expr": "sum by (server, cycle_scope, source, state) (rustfs_scanner_cycle_source_work{job=~\"$job\",server=~\"$server\",source=~\"$scanner_source\"})",
-          "legendFormat": "{{server}} | {{cycle_scope}} | {{source}} | {{state}}"
-        }
-      ],
-      "title": "Scanner Source Work by Server",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${datasource}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisColorMode": "text",
-            "axisLabel": "",
-            "axisPlacement": "auto",
-            "barAlignment": 0,
-            "barWidthFactor": 0.6,
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "gradientMode": "none",
-            "hideFrom": {
-              "legend": false,
-              "tooltip": false,
-              "viz": false
-            },
-            "insertNulls": false,
-            "lineInterpolation": "smooth",
-            "lineWidth": 1,
-            "pointSize": 5,
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "showPoints": "never",
-            "showValues": false,
-            "spanNulls": false,
-            "stacking": {
-              "group": "A",
-              "mode": "none"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": 0
-              }
-            ]
-          },
-          "unit": "ops"
-        },
-        "overrides": [
-          {
-            "matcher": {
-              "id": "byFrameRefID",
-              "options": "B"
-            },
-            "properties": [
-              {
-                "id": "unit",
-                "value": "short"
-              },
-              {
-                "id": "custom.axisPlacement",
-                "value": "right"
-              }
-            ]
-          }
-        ]
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 349
-      },
-      "id": 536,
-      "options": {
-        "legend": {
-          "calcs": [
-            "lastNotNull",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "right",
-          "showLegend": true
-        },
-        "tooltip": {
-          "hideZeros": false,
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "pluginVersion": "12.3.2",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "A",
-          "expr": "sum by (server, bucket, drive, result) (rate(rustfs_scanner_bucket_drive_result_total{job=~\"$job\",server=~\"$server\",bucket=~\"$bucket\",drive=~\"$drive\",result=~\"$scanner_result\"}[$__rate_interval]))",
-          "legendFormat": "{{server}} | {{bucket}} | {{drive}} | {{result}}"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "B",
-          "expr": "sum by (server, cycle_scope, bucket, drive, result) (rustfs_scanner_cycle_bucket_drive_result{job=~\"$job\",server=~\"$server\",bucket=~\"$bucket\",drive=~\"$drive\",result=~\"$scanner_result\"})",
-          "legendFormat": "{{server}} | {{cycle_scope}} | {{bucket}} | {{drive}} | {{result}}"
-        }
-      ],
-      "title": "Scanner Bucket Drive Results",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${datasource}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisColorMode": "text",
-            "axisLabel": "",
-            "axisPlacement": "auto",
-            "barAlignment": 0,
-            "barWidthFactor": 0.6,
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "gradientMode": "none",
-            "hideFrom": {
-              "legend": false,
-              "tooltip": false,
-              "viz": false
-            },
-            "insertNulls": false,
-            "lineInterpolation": "smooth",
-            "lineWidth": 1,
-            "pointSize": 5,
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "showPoints": "never",
-            "showValues": false,
-            "spanNulls": false,
-            "stacking": {
-              "group": "A",
-              "mode": "none"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": 0
-              }
-            ]
-          },
-          "unit": "ops"
-        },
-        "overrides": [
-          {
-            "matcher": {
-              "id": "byFrameRefID",
-              "options": "B"
-            },
-            "properties": [
-              {
-                "id": "unit",
-                "value": "Bps"
-              }
-            ]
-          }
-        ]
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 349
-      },
-      "id": 537,
-      "options": {
-        "legend": {
-          "calcs": [
-            "lastNotNull",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "right",
-          "showLegend": true
-        },
-        "tooltip": {
-          "hideZeros": false,
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "pluginVersion": "12.3.2",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "A",
-          "expr": "sum by (bucket, target_arn) (rate(rustfs_bucket_replication_target_sent_count{job=~\"$job\",bucket=~\"$bucket\",target_arn=~\"$target_arn\"}[$__rate_interval]))",
-          "legendFormat": "sent objects | {{bucket}} | {{target_arn}}"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "B",
-          "expr": "sum by (bucket, target_arn) (rate(rustfs_bucket_replication_target_sent_bytes{job=~\"$job\",bucket=~\"$bucket\",target_arn=~\"$target_arn\"}[$__rate_interval]))",
-          "legendFormat": "sent bytes | {{bucket}} | {{target_arn}}"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "C",
-          "expr": "sum by (bucket, target_arn) (rate(rustfs_bucket_replication_target_total_failed_count{job=~\"$job\",bucket=~\"$bucket\",target_arn=~\"$target_arn\"}[$__rate_interval]))",
-          "legendFormat": "failed objects | {{bucket}} | {{target_arn}}"
-        }
-      ],
-      "title": "Bucket Replication Target Flow",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${datasource}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisColorMode": "text",
-            "axisLabel": "",
-            "axisPlacement": "auto",
-            "barAlignment": 0,
-            "barWidthFactor": 0.6,
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "gradientMode": "none",
-            "hideFrom": {
-              "legend": false,
-              "tooltip": false,
-              "viz": false
-            },
-            "insertNulls": false,
-            "lineInterpolation": "smooth",
-            "lineWidth": 1,
-            "pointSize": 5,
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "showPoints": "never",
-            "showValues": false,
-            "spanNulls": false,
-            "stacking": {
-              "group": "A",
-              "mode": "none"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": 0
-              }
-            ]
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 357
-      },
-      "id": 538,
-      "options": {
-        "legend": {
-          "calcs": [
-            "lastNotNull",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "right",
-          "showLegend": true
-        },
-        "tooltip": {
-          "hideZeros": false,
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "pluginVersion": "12.3.2",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "A",
-          "expr": "max by (server, target_id) (rustfs_audit_target_queue_length_by_server{job=~\"$job\",server=~\"$server\"})",
-          "legendFormat": "audit queue | {{server}} | {{target_id}}"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "B",
-          "expr": "max by (server, action, state) (rustfs_ilm_action_tasks{job=~\"$job\",server=~\"$server\"})",
-          "legendFormat": "ilm | {{server}} | {{action}} | {{state}}"
-        }
-      ],
-      "title": "Audit and ILM by Server",
-      "type": "timeseries"
     }
   ],
   "preload": false,
@@ -12376,32 +11551,6 @@
         "sort": 1,
         "type": "query"
       },
-      {
-        "allValue": ".*",
-        "current": {
-          "text": "All",
-          "value": "$__all"
-        },
-        "datasource": {
-          "type": "prometheus",
-          "uid": "${datasource}"
-        },
-        "definition": "label_values(rustfs_system_drive_api_calls_total,api)",
-        "includeAll": true,
-        "label": "Drive API",
-        "multi": true,
-        "name": "drive_api",
-        "options": [],
-        "query": {
-          "qryType": 1,
-          "query": "label_values(rustfs_system_drive_api_calls_total,api)",
-          "refId": "PrometheusVariableQueryEditor-drive_api"
-        },
-        "refresh": 2,
-        "regex": "",
-        "sort": 1,
-        "type": "query"
-      },
       {
         "allValue": ".*",
         "current": {
@@ -12521,136 +11670,6 @@
         "regex": "",
         "sort": 1,
         "type": "query"
-      },
-      {
-        "allValue": ".*",
-        "current": {
-          "text": "All",
-          "value": "$__all"
-        },
-        "datasource": {
-          "type": "prometheus",
-          "uid": "${datasource}"
-        },
-        "definition": "label_values(rustfs_api_requests_requests_total_by_server,server)",
-        "includeAll": true,
-        "label": "Server",
-        "multi": true,
-        "name": "server",
-        "options": [],
-        "query": {
-          "qryType": 1,
-          "query": "label_values(rustfs_api_requests_requests_total_by_server,server)",
-          "refId": "PrometheusVariableQueryEditor-server"
-        },
-        "refresh": 2,
-        "regex": "",
-        "sort": 1,
-        "type": "query"
-      },
-      {
-        "allValue": ".*",
-        "current": {
-          "text": "All",
-          "value": "$__all"
-        },
-        "datasource": {
-          "type": "prometheus",
-          "uid": "${datasource}"
-        },
-        "definition": "label_values(rustfs_api_requests_requests_total_by_server,name)",
-        "includeAll": true,
-        "label": "API",
-        "multi": true,
-        "name": "api",
-        "options": [],
-        "query": {
-          "qryType": 1,
-          "query": "label_values(rustfs_api_requests_requests_total_by_server,name)",
-          "refId": "PrometheusVariableQueryEditor-api"
-        },
-        "refresh": 2,
-        "regex": "",
-        "sort": 1,
-        "type": "query"
-      },
-      {
-        "allValue": ".*",
-        "current": {
-          "text": "All",
-          "value": "$__all"
-        },
-        "datasource": {
-          "type": "prometheus",
-          "uid": "${datasource}"
-        },
-        "definition": "label_values({__name__=\"rustfs_bucket_replication_target_sent_count\",bucket=~\"$bucket\"},target_arn)",
-        "includeAll": true,
-        "label": "Target ARN",
-        "multi": true,
-        "name": "target_arn",
-        "options": [],
-        "query": {
-          "qryType": 1,
-          "query": "label_values({__name__=\"rustfs_bucket_replication_target_sent_count\",bucket=~\"$bucket\"},target_arn)",
-          "refId": "PrometheusVariableQueryEditor-target_arn"
-        },
-        "refresh": 2,
-        "regex": "",
-        "sort": 1,
-        "type": "query"
-      },
-      {
-        "allValue": ".*",
-        "current": {
-          "text": "All",
-          "value": "$__all"
-        },
-        "datasource": {
-          "type": "prometheus",
-          "uid": "${datasource}"
-        },
-        "definition": "label_values(rustfs_scanner_source_work_total,source)",
-        "includeAll": true,
-        "label": "Scanner Source",
-        "multi": true,
-        "name": "scanner_source",
-        "options": [],
-        "query": {
-          "qryType": 1,
-          "query": "label_values(rustfs_scanner_source_work_total,source)",
-          "refId": "PrometheusVariableQueryEditor-scanner_source"
-        },
-        "refresh": 2,
-        "regex": "",
-        "sort": 1,
-        "type": "query"
-      },
-      {
-        "allValue": ".*",
-        "current": {
-          "text": "All",
-          "value": "$__all"
-        },
-        "datasource": {
-          "type": "prometheus",
-          "uid": "${datasource}"
-        },
-        "definition": "label_values(rustfs_scanner_bucket_drive_result_total,result)",
-        "includeAll": true,
-        "label": "Scanner Result",
-        "multi": true,
-        "name": "scanner_result",
-        "options": [],
-        "query": {
-          "qryType": 1,
-          "query": "label_values(rustfs_scanner_bucket_drive_result_total,result)",
-          "refId": "PrometheusVariableQueryEditor-scanner_result"
-        },
-        "refresh": 2,
-        "regex": "",
-        "sort": 1,
-        "type": "query"
       }
     ]
   },

.docker/observability/otel-collector-config.yaml

@@ -29,27 +29,11 @@ processors:
     limit_mib: 1024
     spike_limit_mib: 256
   transform/logs:
-    error_mode: ignore
     log_statements:
       - context: log
         statements:
-          - set(attributes["message"], body.string) where IsString(body)
-          - set(attributes["log.body"], body.string) where IsString(body)
-          - set(body, Concat([body, " event=", attributes["event"]], "")) where IsString(body) and attributes["event"] != nil
-          - set(body, Concat([body, " component=", attributes["component"]], "")) where IsString(body) and attributes["component"] != nil
-          - set(body, Concat([body, " subsystem=", attributes["subsystem"]], "")) where IsString(body) and attributes["subsystem"] != nil
-          - set(body, Concat([body, " state=", attributes["state"]], "")) where IsString(body) and attributes["state"] != nil
-          - set(body, Concat([body, " result=", attributes["result"]], "")) where IsString(body) and attributes["result"] != nil
-          - set(body, Concat([body, " reason=", attributes["reason"]], "")) where IsString(body) and attributes["reason"] != nil
-          - set(body, Concat([body, " failure_reason=", attributes["failure_reason"]], "")) where IsString(body) and attributes["failure_reason"] != nil
-          - set(body, Concat([body, " rpc_path=", attributes["rpc_path"]], "")) where IsString(body) and attributes["rpc_path"] != nil
-          - set(body, Concat([body, " rpc_service=", attributes["rpc_service"]], "")) where IsString(body) and attributes["rpc_service"] != nil
-          - set(body, Concat([body, " rpc_method=", attributes["rpc_method"]], "")) where IsString(body) and attributes["rpc_method"] != nil
-          - set(body, Concat([body, " expected_audience=", attributes["expected_audience"]], "")) where IsString(body) and attributes["expected_audience"] != nil
-          - set(body, Concat([body, " peer_addr=", attributes["peer_addr"]], "")) where IsString(body) and attributes["peer_addr"] != nil
-          - set(body, Concat([body, " replay_scope_bootstrap_allowed=", attributes["replay_scope_bootstrap_allowed"]], "")) where IsString(body) and attributes["replay_scope_bootstrap_allowed"] != nil
-          - set(body, Concat([body, " error=", attributes["error"]], "")) where IsString(body) and attributes["error"] != nil
-          - set(body, Concat([body, " exception_message=", attributes["exception.message"]], "")) where IsString(body) and attributes["exception.message"] != nil
+          - set(attributes["message"], body.string)
+          - set(attributes["log.body"], body.string)
 
 exporters:
   otlp/tempo:

.docker/observability/prometheus-rules/rustfs-kms-alerts.yml

@@ -17,11 +17,9 @@
 # =============================================================================
 #
 # Metric source: the KMS operation-policy choke point in
-# crates/kms/src/policy.rs, except KmsKeyRotationOverdue, which reads the
-# label-less key-lifecycle gauge published by the deletion worker's sweep
-# (crates/kms/src/deletion_worker.rs). All label values are bounded static
-# strings (operation, op_class, outcome, error_class, backend, scope); key
-# identifiers, key material, and tokens never appear in labels.
+# crates/kms/src/policy.rs. All label values are static enum strings
+# (operation, op_class, outcome, error_class); key identifiers, key material,
+# and tokens never appear in labels.
 #
 # Response procedures: docs/operations/kms-observability-runbook.md
 #
@@ -72,9 +70,8 @@ groups:
       # ------------------------------------------------------------------
       # 2. KmsBackendHighErrorRate
       #    Sustained share of operations terminating without success
-      #    (fatal, budget/deadline exhaustion, admission backpressure,
-      #    or an open circuit). The cancelled outcome is excluded because
-      #    shutdowns legitimately produce it.
+      #    (fatal, budget_exhausted, deadline_exceeded). The cancelled
+      #    outcome is excluded because shutdowns legitimately produce it.
       #    The traffic guard keeps a single failure on a near-idle
       #    cluster from firing the alert.
       #    Threshold: 5% for 10m — conservative default, calibrate
@@ -97,11 +94,9 @@ groups:
           summary: "KMS backend non-success ratio above 5% for 10m"
           description: >-
             {{ $value | humanizePercentage }} of KMS backend operations
-            are terminating in fatal, budget_exhausted,
-            deadline_exceeded, backpressure_timeout,
-            backpressure_rejected, or circuit_open. Object encryption
-            and decryption paths depending on the KMS are degraded or
-            failing.
+            are terminating in fatal, budget_exhausted, or
+            deadline_exceeded. Object encryption and decryption paths
+            depending on the KMS are degraded or failing.
           runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendhigherrorrate"
 
   # ==========================================================================
@@ -191,61 +186,3 @@ groups:
             Retryable failures are outlasting the retry budget, so
             callers are seeing hard failures.
           runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendretrybudgetexhausted"
-
-      # ------------------------------------------------------------------
-      # 6. KmsBackendCircuitOpen
-      #    Direct circuit-state signal, independent of operation traffic.
-      #    A transient open can recover on its first half-open probe; alert
-      #    only when the circuit remains open or half-open for one minute.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendCircuitOpen
-        expr: |
-          rustfs_kms_backend_circuit_open > 0
-        for: 1m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend circuit open ({{ $labels.backend }}/{{ $labels.scope }})"
-          description: >-
-            The KMS backend circuit for {{ $labels.backend }} scope
-            {{ $labels.scope }} has remained open or half-open for one
-            minute. Operations in this scope can terminate as
-            circuit_open until the half-open probe succeeds or returns
-            a non-retryable failure.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendcircuitopen"
-
-      # ------------------------------------------------------------------
-      # 7. KmsKeyRotationOverdue
-      #    The least recently rotated usable key has gone more than 400
-      #    days without a rotation (measured from creation for keys with
-      #    no recorded rotation). Direct gauge state published by the
-      #    deletion worker's sweep, so no traffic guard applies; the
-      #    one-hour hold only bridges scrape gaps. The worker runs only
-      #    on backends with the schedule_deletion capability, so on the
-      #    Static backend the series never exists and this alert cannot
-      #    fire — that backend cannot rotate either; see the rotation
-      #    driver matrix in docs/operations/kms-backend-security.md.
-      #    Threshold: 400 days — conservative default sitting above a
-      #    one-year rotation policy. Align it with the rotation period
-      #    your compliance policy requires, and with
-      #    RUSTFS_KMS_ROTATION_MAX_AGE_SECS so the per-key rotation_due
-      #    verdict and this aggregate alert agree.
-      # ------------------------------------------------------------------
-      - alert: KmsKeyRotationOverdue
-        expr: |
-          rustfs_kms_oldest_key_rotation_age_seconds > (400 * 86400)
-        for: 1h
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "Oldest KMS key unrotated for more than 400 days"
-          description: >-
-            The least recently rotated usable KMS key was last rotated
-            {{ $value | humanizeDuration }} ago (measured from creation
-            for keys with no recorded rotation). List keys through the
-            admin API and read rotation_due / rotation_due_reason for
-            the per-key verdict; an "unsupported" reason means the
-            backend cannot rotate at all.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmskeyrotationoverdue"

.github/actions/setup/action.yml

@@ -57,7 +57,7 @@ runs:
   using: "composite"
   steps:
     # protobuf-compiler is deliberately absent: the setup-protoc step below
-    # installs 35.1 into the tool cache and prepends it to PATH, so the apt
+    # installs 34.1 into the tool cache and prepends it to PATH, so the apt
     # build (older, and never version-matched) was shadowed on every run and
     # simply never used.
     - name: Install system dependencies (Ubuntu)
@@ -81,11 +81,11 @@ runs:
     - name: Install protoc
       uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
       with:
-        version: "35.1"
+        version: "34.1"
         repo-token: ${{ github.token }}
 
     - name: Install flatc
-      uses: Nugine/setup-flatc@698800de72a96bfb22cf60431dc21a2ff9a7e07b # v1
+      uses: Nugine/setup-flatc@e7855e994773ce90094a3f1626d4afc9080c23ae # v1
       with:
         version: "25.12.19"
 

.github/workflows/audit.yml

@@ -24,7 +24,6 @@ on:
       - '.github/actions/**'
       - '.github/workflows/**'
       - 'scripts/release/create_or_update_release.sh'
-      - 'scripts/security/check_performance_ab_workflow.sh'
       - 'scripts/security/check_preview_release_workflow.sh'
       - 'scripts/security/check_workflow_pins.sh'
   pull_request:
@@ -37,7 +36,6 @@ on:
       - '.github/actions/**'
       - '.github/workflows/**'
       - 'scripts/release/create_or_update_release.sh'
-      - 'scripts/security/check_performance_ab_workflow.sh'
       - 'scripts/security/check_preview_release_workflow.sh'
       - 'scripts/security/check_workflow_pins.sh'
   schedule:
@@ -143,9 +141,6 @@ jobs:
       - name: Check preview release workflow policy
         run: ./scripts/security/check_preview_release_workflow.sh
 
-      - name: Check performance A/B workflow trust boundary
-        run: ./scripts/security/check_performance_ab_workflow.sh
-
   dependency-review:
     name: Dependency Review
     runs-on: ubuntu-latest

.github/workflows/ci-docs-only.yml

@@ -102,9 +102,6 @@ jobs:
       - name: Check architecture migration rules
         run: ./scripts/check_architecture_migration_rules.sh
 
-      - name: Check logging guardrails
-        run: ./scripts/check_logging_guardrails.sh
-
       - name: Check tokio io-uring feature guard
         run: ./scripts/check_no_tokio_io_uring.sh
 
@@ -114,9 +111,6 @@ jobs:
       - name: Check body-cache whitelist guard
         run: ./scripts/check_body_cache_whitelist.sh
 
-      - name: Check s3s footprint ratchet
-        run: ./scripts/check_s3s_footprint.sh
-
       - name: Check no planning docs committed
         run: ./scripts/check_no_planning_docs.sh
 

.github/workflows/ci.yml

@@ -137,9 +137,6 @@ jobs:
       - name: Check architecture migration rules
         run: ./scripts/check_architecture_migration_rules.sh
 
-      - name: Check logging guardrails
-        run: ./scripts/check_logging_guardrails.sh
-
       - name: Check tokio io-uring feature guard
         run: ./scripts/check_no_tokio_io_uring.sh
 
@@ -149,9 +146,6 @@ jobs:
       - name: Check body-cache whitelist guard
         run: ./scripts/check_body_cache_whitelist.sh
 
-      - name: Check s3s footprint ratchet
-        run: ./scripts/check_s3s_footprint.sh
-
       - name: Check no planning docs committed
         run: ./scripts/check_no_planning_docs.sh
 
@@ -346,11 +340,9 @@ jobs:
       - name: Annotate early-stop reason
         if: failure() && github.event_name == 'pull_request'
         run: |
-          {
-            echo "## CI early-stop"
-            echo "Job \`${GITHUB_JOB}\` (Test and Lint) failed; cancelling run ${GITHUB_RUN_ID} to free runners."
-            echo "Sibling jobs showing **cancelled** were stopped by this job, not by their own failure."
-          } >> "$GITHUB_STEP_SUMMARY"
+          echo "## CI early-stop" >> "$GITHUB_STEP_SUMMARY"
+          echo "Job \`${GITHUB_JOB}\` (Test and Lint) failed; cancelling run ${GITHUB_RUN_ID} to free runners." >> "$GITHUB_STEP_SUMMARY"
+          echo "Sibling jobs showing **cancelled** were stopped by this job, not by their own failure." >> "$GITHUB_STEP_SUMMARY"
 
       # curl rather than `gh`: every existing `gh` call in this repo runs on
       # ubuntu-latest, and the sm-standard-* images are custom and trimmed (they
@@ -673,17 +665,15 @@ jobs:
       - name: Make binary executable
         run: chmod +x ./target/debug/rustfs
 
-      # Build the e2e test graph once. The archive is reused by the security
-      # count-floor check and the smoke run below, avoiding a second compile of
-      # the same e2e_test target on cold runners (backlog#1645).
-      - name: Archive e2e smoke test binaries
-        env:
-          NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
-          NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-smoke-list.json
-        run: |
-          cargo nextest archive --profile e2e-smoke -p e2e_test --archive-file "${NEXTEST_ARCHIVE}"
-          cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --message-format json > "${NEXTEST_LISTING}"
-          ./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
+      # Guard the security negative-auth smoke subset (backlog#1151 sec-5)
+      # against a rename or deletion silently dropping it out of the e2e-smoke
+      # filter. The script lists what the profile selects and fails if the count
+      # of security auth-rejection tests falls below the committed floor in
+      # .config/security-smoke-floor.txt (infra-12 count-floor mechanism). Run
+      # before the smoke suite so a thinned gate fails fast; the `nextest list`
+      # here compiles the e2e_test binaries the run below reuses.
+      - name: Check security smoke subset count floor
+        run: ./scripts/check_security_smoke_count.sh check
 
       # PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
       # profile.e2e-smoke default-filter in .config/nextest.toml is the single
@@ -691,30 +681,7 @@ jobs:
       # adding new e2e jobs here. Each test spawns its own rustfs server on a
       # random port and reuses the downloaded debug binary above.
       - name: Run e2e smoke suite
-        env:
-          NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
-          RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
-        run: |
-          cargo nextest run --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" \
-            --status-level all --final-status-level all --failure-output final
-
-      - name: Upload e2e smoke diagnostics
-        if: failure()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: e2e-smoke-diagnostics-${{ github.run_number }}
-          path: |
-            ${{ runner.temp }}/rustfs-e2e-smoke-logs/
-            ${{ runner.temp }}/rustfs-e2e-smoke-list.json
-          if-no-files-found: warn
-
-      - name: Upload e2e smoke JUnit report
-        if: always()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: e2e-smoke-junit-${{ github.run_number }}
-          path: target/nextest/e2e-smoke/junit.xml
-          if-no-files-found: warn
+        run: cargo nextest run --profile e2e-smoke -p e2e_test
 
       - name: Install s3s-e2e test tool
         uses: taiki-e/cache-cargo-install-action@7447f04c51f2ba27ca35e7f1e28fab848c5b3ba7 # v2
@@ -770,32 +737,6 @@ jobs:
           cache-save-if: 'false'
           install-build-packaging-tools: 'false'
 
-      - name: Set up Python
-        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
-        with:
-          python-version: "3.12"
-
-      - name: Install awscurl
-        run: |
-          python3 -m pip install --user --upgrade pip "awscurl==0.44"
-          echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
-
-      - name: Verify awscurl
-        run: test -x "$AWSCURL_PATH"
-
-      - name: Install Vault
-        run: |
-          VAULT_VERSION="1.17.6"
-          VAULT_ARCHIVE="vault_${VAULT_VERSION}_linux_amd64.zip"
-          curl -fsSLo "$RUNNER_TEMP/$VAULT_ARCHIVE" "https://releases.hashicorp.com/vault/${VAULT_VERSION}/${VAULT_ARCHIVE}"
-          echo "0cddc1fbbb88583b5ba5b845f9f8fae47c6fb39a6d48cd543c6ba6fd3ac1a669  $RUNNER_TEMP/$VAULT_ARCHIVE" | sha256sum --check --status
-          unzip -q "$RUNNER_TEMP/$VAULT_ARCHIVE" -d "$RUNNER_TEMP/vault-bin"
-          echo "RUSTFS_TEST_VAULT_BIN=$RUNNER_TEMP/vault-bin/vault" >> "$GITHUB_ENV"
-
-      - name: Verify Vault
-        run: |
-          "$RUSTFS_TEST_VAULT_BIN" version
-
       # Download after the cache restore so the freshly built binary from the
       # build job always wins over anything restored into target/debug.
       - name: Download debug binary

.github/workflows/e2e-replication-nightly.yml

@@ -14,24 +14,25 @@
 
 # Nightly full replication e2e lane (backlog#1147 repl-1, deps: ci-4).
 #
-# The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the
-# FAST replication tests. This scheduled lane runs the remaining heavier
-# replication e2e tests that are unfit for a per-PR gate: remote-target TLS
-# validation, bucket-replication data-plane/helper tests (PUT/delete + poll
-# for convergence, HTTPS targets, active SSE failure contracts, event/history
-# observers), and the `_real_dual_node` / `_real_three_node` /
-# `_real_single_node` site-replication tests that each spawn full rustfs
-# server processes.
+# The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the 20
+# FAST replication tests. This scheduled lane runs the remaining 27
+# heavier replication e2e tests that are unfit for a per-PR gate:
+#
+#   * 2 remote-target TLS validation tests.
+#   * 12 bucket-replication data-plane/helper tests (PUT/delete + poll for
+#     convergence; two replicate over HTTPS, two pin active SSE failure
+#     contracts, and one guards event/history observers). The SSE-S3 contract
+#     remains ignored under backlog#1291.
+#   * 11 `_real_dual_node` site-replication tests (each spawns TWO rustfs
+#     servers and drives the cross-process site-replication control plane).
+#   * 1 `_real_three_node` site-replication test.
+#   * 1 `_real_single_node` service-account round-trip test.
 #
 # The selection is the [profile.e2e-repl-nightly] default-filter in
 # .config/nextest.toml — the single wiring mechanism (repl-1 / ci-4). Do NOT
-# add ad-hoc cargo-test steps here; change the filterset instead. The
-# authoritative membership and count come from
-# `cargo nextest list -p e2e_test --profile e2e-repl-nightly`; the PR/nightly
-# count invariant is maintained next to the filtersets in .config/nextest.toml
-# (deliberately not duplicated here).
+# add ad-hoc cargo-test steps here; change the filterset instead.
 #
-# Explicit division of labor: the nightly subset runs ONLY here, never double-run
+# Explicit division of labor: these 27 tests run ONLY here, never double-run
 # in ci-5's future e2e-full merge gate. TODO(ci-7): once the ci domain's
 # consolidated scheduled e2e workflow exists, fold this interim repl-owned lane
 # into it rather than growing a second scheduled entrypoint.

.github/workflows/minio-interop.yml

@@ -75,7 +75,6 @@ jobs:
       INTEROP_PACKAGE: rustfs
       INTEROP_FEATURES: rio-v2
       INTEROP_FILTER: "test(minio_generated_read_test::)"
-      INTEROP_REQUIRED_TESTS: '["reads_minio_generated_sse_s3_multipart_fixture", "reads_minio_generated_sse_kms_multipart_fixture", "rejects_minio_generated_sse_s3_fixture_with_wrong_kms_key", "rejects_minio_generated_sse_s3_fixture_with_truncated_ciphertext"]'
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
@@ -96,23 +95,20 @@ jobs:
       # is a perfectly valid filterset that matches zero tests, so the next
       # rename or module move would leave this job selecting nothing and
       # reporting success without executing a single interop assertion. Count
-      # the selection and require every core reader test, while allowing new
-      # reader cases to be added without changing this guard.
+      # the selection and fail with a reason instead.
       #
       # Count only `filter-match.status == "matches"`: the top-level
       # `test-count` in the JSON is the package total and ignores `-E` entirely.
       - name: Assert the interop selector still matches tests
         run: |
           set -euo pipefail
-          selection="$(cargo nextest list --run-ignored ignored-only \
+          count="$(cargo nextest list --run-ignored all \
             -p "$INTEROP_PACKAGE" --features "$INTEROP_FEATURES" \
             -E "$INTEROP_FILTER" --message-format json \
-            | python3 -c 'import json,os,sys; d=json.load(sys.stdin); required=json.loads(os.environ["INTEROP_REQUIRED_TESTS"]); matched=[name for suite in d.get("rust-suites", {}).values() for name,test in suite.get("testcases", {}).items() if test.get("filter-match", {}).get("status") == "matches"]; missing=[test for test in required if not any(name.endswith("minio_generated_read_test::" + test) for name in matched)]; print(len(matched)); print(",".join(missing))')"
-          count="$(printf '%s\n' "$selection" | sed -n '1p')"
-          missing="$(printf '%s\n' "$selection" | sed -n '2p')"
+            | python3 -c 'import json,sys; d=json.load(sys.stdin); print(sum(1 for s in d.get("rust-suites", {}).values() for t in s.get("testcases", {}).values() if t.get("filter-match", {}).get("status") == "matches"))')"
           echo "interop tests selected: ${count}"
-          if [ -n "${missing}" ]; then
-            echo "::error::Selector '${INTEROP_FILTER}' in package '${INTEROP_PACKAGE}' is missing required tests: ${missing}. The MinIO interop reader tests have moved or been renamed; fix the selector instead of running an incomplete matrix. Context: rustfs/backlog#1638."
+          if [ "${count}" -eq 0 ]; then
+            echo "::error::Selector '${INTEROP_FILTER}' in package '${INTEROP_PACKAGE}' matched 0 tests. The MinIO interop reader tests have moved or been renamed again; fix the selector instead of letting this job pass without running them. Context: rustfs/backlog#1638."
             exit 1
           fi
 

.github/workflows/nightly-gnu.yml

@@ -55,142 +55,3 @@ jobs:
 
       - name: Build RustFS
         run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins
-
-  # Live-Vault lane for the rustfs-kms suite (rustfs/backlog#1774).
-  #
-  # RUSTFS_KMS_VAULT_TOKEN is the single switch that adds the Vault KV2 and
-  # Vault Transit backends to every for_each_backend spec in
-  # crates/kms/tests/behavior_*.rs (see crates/kms/AGENTS.md). rotate and
-  # versioning are advertised only by the Vault backends, so without this lane
-  # no CI run ever asserts the working half of behavior_rotation.rs — a
-  # rotation that silently dropped historical key versions would stay green.
-  # The same lane runs the dev-Vault #[ignore] tests and the two self-hosting
-  # live scripts (AppRole login, three-node Raft leader failover).
-  #
-  # GitHub-hosted ubuntu-latest, deliberately not the self-hosted sm-standard
-  # fleet: the HA failover script needs a working Docker daemon, and the
-  # self-hosted fleet is heterogeneous — a docker-dependent workflow has been
-  # burned by it before (see the banner in e2e-s3tests.yml, rustfs/backlog#1149).
-  kms-vault-lane:
-    name: KMS live Vault lane
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-      # Root token of the ephemeral loopback dev server. Not a secret: the
-      # server lives only for this job, listens on 127.0.0.1, and holds only
-      # keys the tests create. The literal value matters — the dev-Vault
-      # #[ignore] fixtures in crates/kms/src/backends/vault.rs hardcode it.
-      VAULT_LANE_TOKEN: dev-only-token
-      VAULT_LANE_ADDR: http://127.0.0.1:8200
-      # Keeps a runner-level proxy from swallowing the loopback dev-server
-      # traffic (see crates/kms/AGENTS.md). Actions env keys are
-      # case-insensitive, so only the uppercase form is set; reqwest reads
-      # either casing.
-      NO_PROXY: 127.0.0.1,localhost
-    steps:
-      - name: Checkout main branch
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-          ref: main
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          # Dedicated key: rust-cache cannot tell runner images apart, so
-          # sharing a key with an sm-standard lane would let two different
-          # system images overwrite each other's artifacts (same reasoning as
-          # ci.yml's ci-uring lane). Saved from this nightly job itself so the
-          # next night starts warm.
-          cache-shared-key: kms-vault-lane
-          cache-save-if: 'true'
-          install-build-packaging-tools: 'false'
-          install-test-tools: 'false'
-
-      - name: Install Vault CLI
-        run: |
-          set -euo pipefail
-          wget -qO- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
-          echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list >/dev/null
-          sudo apt-get update -qq
-          sudo apt-get install -y -qq vault
-          vault version
-
-      - name: Start Vault dev server with KV2 and Transit engines
-        run: |
-          set -euo pipefail
-          nohup vault server -dev \
-            -dev-root-token-id="${VAULT_LANE_TOKEN}" \
-            -dev-listen-address=127.0.0.1:8200 >/tmp/vault-dev.log 2>&1 &
-          for _ in $(seq 1 60); do
-            if curl -fsS "${VAULT_LANE_ADDR}/v1/sys/health" >/dev/null 2>&1; then
-              break
-            fi
-            sleep 1
-          done
-          curl -fsS "${VAULT_LANE_ADDR}/v1/sys/health"
-          export VAULT_ADDR="${VAULT_LANE_ADDR}" VAULT_TOKEN="${VAULT_LANE_TOKEN}"
-          # Dev mode mounts KV v2 at secret/ by default; Transit is explicit.
-          # Prove both engines actually work rather than assuming the defaults.
-          vault secrets enable transit
-          vault kv put secret/rustfs-ci-lane-probe value=ok >/dev/null
-          vault kv get secret/rustfs-ci-lane-probe >/dev/null
-          vault write -f transit/keys/rustfs-ci-lane-probe >/dev/null
-
-      - name: Run rustfs-kms suite with the Vault lane on
-        env:
-          RUSTFS_KMS_VAULT_TOKEN: ${{ env.VAULT_LANE_TOKEN }}
-          RUSTFS_KMS_VAULT_ADDR: ${{ env.VAULT_LANE_ADDR }}
-        run: cargo test -p rustfs-kms --locked
-
-      - name: Run dev-Vault ignored tests
-        env:
-          RUSTFS_KMS_VAULT_TOKEN: ${{ env.VAULT_LANE_TOKEN }}
-          RUSTFS_KMS_VAULT_ADDR: ${{ env.VAULT_LANE_ADDR }}
-        # Filters select the dev-Vault-only #[ignore] tests. The AWS #[ignore]
-        # tests (backends::aws, service_manager) stay excluded — they need real
-        # AWS credentials and create billable keys. The AppRole and HA #[ignore]
-        # tests are excluded here because their own scripts below provision the
-        # Vault topology they need.
-        run: |
-          set -euo pipefail
-          cargo test -p rustfs-kms --locked --lib backends::contract_tests -- --ignored
-          cargo test -p rustfs-kms --locked --lib backends::vault -- --ignored
-          cargo test -p rustfs-kms --locked --test vault_fault_injection -- --ignored
-
-      - name: Run AppRole live checks (self-hosting ephemeral Vault)
-        run: bash scripts/test/vault_approle_kms_live.sh
-
-      - name: Show Vault dev server log on failure
-        if: failure()
-        run: tail -n 200 /tmp/vault-dev.log || true
-
-  # Three-node Raft leader failover (crates/kms/tests/vault_ha_failover_live.rs,
-  # first validated by rustfs/rustfs#5653). Its own job so an election-timing
-  # flake cannot mask the main lane's verdict, and vice versa. The script
-  # provisions and tears down its own Docker cluster.
-  kms-vault-ha-failover:
-    name: KMS Vault HA failover lane
-    runs-on: ubuntu-latest
-    timeout-minutes: 60
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-      NO_PROXY: 127.0.0.1,localhost
-    steps:
-      - name: Checkout main branch
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-          ref: main
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          cache-shared-key: kms-vault-lane
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-          install-test-tools: 'false'
-
-      - name: Run HA leader failover live checks (three-node Raft cluster in Docker)
-        run: bash scripts/test/vault_ha_kms_live.sh

.github/workflows/package.yml

@@ -1,477 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Package Workflow - Build DEB/RPM packages
-#
-# This workflow builds DEB and RPM packages from pre-built Linux binaries
-# and uploads them to Cloudflare R2.
-#
-# Trigger:
-# - release published: automatically package when a GitHub release is published
-# - workflow_dispatch: manual trigger with optional tag/run_id
-#
-# Flow:
-# 1. Find the Build workflow run for the release tag
-# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu) from build artifacts
-# 3. Build DEB packages for amd64 and arm64
-# 4. Build RPM packages for x86_64 and aarch64
-# 5. Upload all packages to Cloudflare R2
-
-name: Package DEB/RPM
-
-permissions:
-  contents: read
-  actions: read
-
-on:
-  release:
-    types: [ published ]
-  workflow_dispatch:
-    inputs:
-      tag:
-        description: "Release tag to package (e.g. 1.0.0-beta.12). Leave empty for latest main build."
-        required: false
-        type: string
-      build_run_id:
-        description: "Build workflow run ID (overrides tag lookup)"
-        required: false
-        type: string
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event.release.tag_name || github.event.inputs.tag || github.run_id }}
-  cancel-in-progress: true
-
-jobs:
-  # Resolve which build run to use and extract version info
-  resolve:
-    name: Resolve Build
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    outputs:
-      version: ${{ steps.resolve.outputs.version }}
-      build_type: ${{ steps.resolve.outputs.build_type }}
-      build_run_id: ${{ steps.resolve.outputs.build_run_id }}
-      tag: ${{ steps.resolve.outputs.tag }}
-    steps:
-      - name: Resolve build run
-        id: resolve
-        shell: bash
-        env:
-          GH_TOKEN: ${{ github.token }}
-          INPUT_TAG: ${{ github.event.inputs.tag }}
-          INPUT_RUN_ID: ${{ github.event.inputs.build_run_id }}
-        run: |
-          set -euo pipefail
-
-          # Determine tag
-          if [[ "${{ github.event_name }}" == "release" ]]; then
-            TAG="${{ github.event.release.tag_name }}"
-          elif [[ -n "$INPUT_TAG" ]]; then
-            TAG="$INPUT_TAG"
-          else
-            TAG=""
-          fi
-
-          echo "Tag: ${TAG:-<none>}"
-
-          # Determine build run ID
-          BUILD_RUN_ID=""
-
-          if [[ -n "$INPUT_RUN_ID" ]]; then
-            # Explicit run ID takes priority
-            BUILD_RUN_ID="$INPUT_RUN_ID"
-            echo "Using explicit build run ID: $BUILD_RUN_ID"
-
-          elif [[ -n "$TAG" ]]; then
-            # Find the build run that produced this tag
-            echo "Looking for build run for tag: $TAG"
-            BUILD_RUN_ID=$(gh api \
-              "repos/${{ github.repository }}/actions/workflows/build.yml/runs?branch=${TAG}&status=success&per_page=1" \
-              --jq '.workflow_runs[0].id' 2>/dev/null || echo "")
-
-            if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
-              # Tag might not be a branch; try event=push with head_branch matching
-              BUILD_RUN_ID=$(gh api \
-                "repos/${{ github.repository }}/actions/workflows/build.yml/runs?event=push&status=success&per_page=100" \
-                --jq ".workflow_runs[] | select(.head_branch == \"$TAG\") | .id" 2>/dev/null | head -1 || echo "")
-            fi
-
-            if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
-              echo "❌ No successful build run found for tag: $TAG"
-              exit 1
-            fi
-            echo "Found build run: $BUILD_RUN_ID"
-
-          else
-            # No tag — latest successful main build
-            echo "No tag specified, looking for latest main build"
-            BUILD_RUN_ID=$(gh api \
-              "repos/${{ github.repository }}/actions/workflows/build.yml/runs?branch=main&status=success&per_page=1" \
-              --jq '.workflow_runs[0].id' 2>/dev/null || echo "")
-
-            if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
-              echo "❌ No successful main build found"
-              exit 1
-            fi
-            echo "Latest main build: $BUILD_RUN_ID"
-          fi
-
-          # Determine version and build type
-          if [[ -n "$TAG" ]]; then
-            VERSION="$TAG"
-            if [[ "$TAG" == *"-preview"* ]]; then
-              BUILD_TYPE="preview"
-            elif [[ "$TAG" == *"alpha"* || "$TAG" == *"beta"* || "$TAG" == *"rc"* ]]; then
-              BUILD_TYPE="prerelease"
-            else
-              BUILD_TYPE="release"
-            fi
-          else
-            SHORT_SHA=$(gh api "repos/${{ github.repository }}/actions/runs/${BUILD_RUN_ID}" \
-              --jq '.head_sha' 2>/dev/null | head -c 7)
-            VERSION="dev-${SHORT_SHA}"
-            BUILD_TYPE="development"
-          fi
-
-          {
-            echo "version=$VERSION"
-            echo "build_type=$BUILD_TYPE"
-            echo "build_run_id=$BUILD_RUN_ID"
-            echo "tag=${TAG}"
-          } >> "$GITHUB_OUTPUT"
-
-          echo "📊 Resolved:"
-          echo "  Version: $VERSION"
-          echo "  Build type: $BUILD_TYPE"
-          echo "  Build run ID: $BUILD_RUN_ID"
-
-  # Build DEB and RPM packages for each architecture
-  package:
-    name: Package (${{ matrix.arch }})
-    needs: resolve
-    runs-on: ubuntu-latest
-    timeout-minutes: 30
-    strategy:
-      fail-fast: false
-      matrix:
-        include:
-          - arch: x86_64
-            deb_arch: amd64
-            rpm_arch: x86_64
-            artifact_name: "rustfs-linux-x86_64-gnu"
-          - arch: aarch64
-            deb_arch: arm64
-            rpm_arch: aarch64
-            artifact_name: "rustfs-linux-aarch64-gnu"
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Download binary artifact from build run
-        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
-        with:
-          pattern: ${{ matrix.artifact_name }}*
-          path: ./binary-artifact
-          run-id: ${{ needs.resolve.outputs.build_run_id }}
-          github-token: ${{ github.token }}
-          merge-multiple: true
-
-      - name: Extract binary
-        id: binary
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
-          if [[ -z "$ZIP_FILE" ]]; then
-            echo "❌ No binary artifact found"
-            ls -la ./binary-artifact/ || true
-            exit 1
-          fi
-
-          echo "Found artifact: $ZIP_FILE"
-
-          mkdir -p ./bin
-          unzip -o "$ZIP_FILE" -d ./bin
-
-          if [[ ! -f ./bin/rustfs ]]; then
-            echo "❌ rustfs binary not found in archive"
-            exit 1
-          fi
-
-          chmod +x ./bin/rustfs
-          ls -lh ./bin/rustfs
-          echo "✅ Binary extracted"
-
-      - name: Build DEB package
-        id: deb
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          VERSION="${{ needs.resolve.outputs.version }}"
-          DEB_ARCH="${{ matrix.deb_arch }}"
-          # DEB version: replace - with ~ (1.0.0-beta.12 -> 1.0.0~beta.12)
-          # Use a variable for ~ to prevent tilde expansion by bash
-          TILDE='~'
-          DEB_VERSION="${VERSION/-/$TILDE}"
-          PKG_DIR="rustfs_${DEB_VERSION}_${DEB_ARCH}"
-
-          echo "Building DEB: ${PKG_DIR}.deb"
-
-          mkdir -p "${PKG_DIR}/DEBIAN"
-          mkdir -p "${PKG_DIR}/usr/bin"
-          mkdir -p "${PKG_DIR}/etc/default"
-          mkdir -p "${PKG_DIR}/lib/systemd/system"
-          mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
-
-          cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
-          chmod 755 "${PKG_DIR}/usr/bin/rustfs"
-
-          cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
-
-          cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
-          # RustFS Environment Configuration
-          # See https://rustfs.com/docs/ for more information
-          # RUSTFS_VOLUMES=""
-          # RUSTFS_ROOT_USER=""
-          # RUSTFS_ROOT_PASSWORD=""
-          ENVEOF
-
-          cat > "${PKG_DIR}/DEBIAN/control" << EOF
-          Package: rustfs
-          Version: ${DEB_VERSION}
-          Section: utils
-          Priority: optional
-          Architecture: ${DEB_ARCH}
-          Depends: libc6 (>= 2.31)
-          Maintainer: RustFS Team <support@rustfs.com>
-          Description: High-performance distributed object storage
-           RustFS is a high-performance distributed object storage software
-           built using Rust. It is compatible with MinIO and S3 API.
-          Homepage: https://rustfs.com
-          EOF
-
-          cat > "${PKG_DIR}/DEBIAN/postinst" << 'POSTINST'
-          #!/bin/bash
-          set -e
-          if ! getent passwd rustfs > /dev/null 2>&1; then
-              useradd -r -s /bin/false -d /opt/rustfs rustfs
-          fi
-          mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
-          chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          echo "RustFS installed. Configure /etc/default/rustfs then: systemctl start rustfs"
-          POSTINST
-          chmod 755 "${PKG_DIR}/DEBIAN/postinst"
-
-          cat > "${PKG_DIR}/DEBIAN/prerm" << 'PRERM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
-              systemctl stop rustfs
-          fi
-          PRERM
-          chmod 755 "${PKG_DIR}/DEBIAN/prerm"
-
-          cat > "${PKG_DIR}/DEBIAN/postrm" << 'POSTRM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          POSTRM
-          chmod 755 "${PKG_DIR}/DEBIAN/postrm"
-
-          cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
-          cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
-
-          fakeroot dpkg-deb --build "${PKG_DIR}"
-
-          DEB_FILE="${PKG_DIR}.deb"
-          ls -lh "$DEB_FILE"
-          echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
-          echo "✅ DEB built: $DEB_FILE"
-
-      - name: Build RPM package
-        id: rpm
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          VERSION="${{ needs.resolve.outputs.version }}"
-          RPM_ARCH="${{ matrix.rpm_arch }}"
-
-          echo "Building RPM for ${RPM_ARCH}"
-
-          sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential
-          sudo gem install fpm
-
-          # Create config file for fpm (DEB build creates it in its package dir structure,
-          # but fpm needs the file to exist before packaging)
-          mkdir -p ./tmp-pkg/etc/default
-          cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
-          # RustFS Environment Configuration
-          # See https://rustfs.com/docs/ for more information
-          # RUSTFS_VOLUMES=""
-          # RUSTFS_ROOT_USER=""
-          # RUSTFS_ROOT_PASSWORD=""
-          ENVEOF
-
-          fpm -s dir -t rpm \
-            --name rustfs \
-            --version "$VERSION" \
-            --architecture "$RPM_ARCH" \
-            --depends "glibc >= 2.31" \
-            --maintainer "RustFS Team <support@rustfs.com>" \
-            --description "High-performance distributed object storage" \
-            --url "https://rustfs.com" \
-            --license "Apache-2.0" \
-            --after-install <(cat <<'POSTINST'
-          #!/bin/bash
-          set -e
-          if ! getent passwd rustfs > /dev/null 2>&1; then
-              useradd -r -s /bin/false -d /opt/rustfs rustfs
-          fi
-          mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
-          chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          POSTINST
-            ) \
-            --before-remove <(cat <<'PRERM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
-              systemctl stop rustfs
-          fi
-          PRERM
-            ) \
-            --after-remove <(cat <<'POSTRM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          POSTRM
-            ) \
-            --config-files /etc/default/rustfs \
-            ./bin/rustfs=/usr/bin/rustfs \
-            ./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
-            deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
-            LICENSE=/usr/share/doc/rustfs/LICENSE \
-            README.md=/usr/share/doc/rustfs/README.md
-
-          RPM_FILE=$(ls -1 rustfs-*.rpm 2>/dev/null | head -1)
-          if [[ -z "$RPM_FILE" ]]; then
-            echo "❌ RPM build failed"
-            exit 1
-          fi
-
-          ls -lh "$RPM_FILE"
-          echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
-          echo "✅ RPM built: $RPM_FILE"
-
-      - name: Upload packages to artifacts
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: packages-${{ matrix.arch }}
-          path: |
-            *.deb
-            *.rpm
-          retention-days: 30
-
-      - name: Upload packages to Cloudflare R2
-        if: env.R2_ACCESS_KEY_ID != ''
-        env:
-          R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
-          R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
-          R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
-          R2_BUCKET: ${{ secrets.R2_BUCKET }}
-          AWS_EC2_METADATA_DISABLED: true
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
-            echo "⚠️ R2 credentials missing, skipping upload"
-            exit 0
-          fi
-
-          if ! command -v aws >/dev/null 2>&1; then
-            sudo apt-get update && sudo apt-get install -y awscli
-          fi
-
-          export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
-          export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
-          export AWS_DEFAULT_REGION="auto"
-
-          BUILD_TYPE="${{ needs.resolve.outputs.build_type }}"
-          if [[ "$BUILD_TYPE" == "development" ]]; then
-            R2_PREFIX="artifacts/rustfs/packages/dev"
-          else
-            R2_PREFIX="artifacts/rustfs/packages/release"
-          fi
-          R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
-
-          echo "📤 Uploading to $R2_PATH"
-
-          DEB_FILE="${{ steps.deb.outputs.deb_file }}"
-          RPM_FILE="${{ steps.rpm.outputs.rpm_file }}"
-
-          for f in "$DEB_FILE" "$RPM_FILE"; do
-            if [[ -n "$f" && -f "$f" ]]; then
-              echo "Uploading: $f"
-              aws s3 cp "$f" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
-            fi
-          done
-
-          echo "✅ Upload complete"
-
-          # Also upload as latest for release/prerelease
-          if [[ "$BUILD_TYPE" == "release" || "$BUILD_TYPE" == "prerelease" ]]; then
-            LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
-            for f in "$DEB_FILE" "$RPM_FILE"; do
-              if [[ -n "$f" && -f "$f" ]]; then
-                echo "Uploading latest: $(basename "$f")"
-                aws s3 cp "$f" "$LATEST_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
-              fi
-            done
-            echo "✅ Latest packages updated"
-          fi
-
-  # Summary
-  summary:
-    name: Summary
-    needs: [ resolve, package ]
-    if: always()
-    runs-on: ubuntu-latest
-    timeout-minutes: 5
-    steps:
-      - name: Print summary
-        shell: bash
-        run: |
-          echo "## 📦 Package Summary" >> "$GITHUB_STEP_SUMMARY"
-          echo "" >> "$GITHUB_STEP_SUMMARY"
-          echo "| Item | Value |" >> "$GITHUB_STEP_SUMMARY"
-          echo "|------|-------|" >> "$GITHUB_STEP_SUMMARY"
-          echo "| Version | \`${{ needs.resolve.outputs.version }}\` |" >> "$GITHUB_STEP_SUMMARY"
-          echo "| Build Type | ${{ needs.resolve.outputs.build_type }} |" >> "$GITHUB_STEP_SUMMARY"
-          echo "| Build Run | #${{ needs.resolve.outputs.build_run_id }} |" >> "$GITHUB_STEP_SUMMARY"
-          echo "| Package Status | ${{ needs.package.result }} |" >> "$GITHUB_STEP_SUMMARY"

.github/workflows/performance-ab.yml

@@ -17,10 +17,10 @@
 # Two entry points, honestly scoped:
 #   * schedule (nightly, on main): post-merge detection — catches a regression
 #     within 24h of landing, not before merge.
-#   * workflow_dispatch: an explicitly selected trusted ref.
-# The dispatch input can run the gate with --allow-regression so a deliberate
-# correctness cost (e.g. the #4221 fsync durability fix) is recorded, not
-# blocked (rustfs/backlog#935 correction 1).
+#   * pull_request labeled `perf-ab`: opt-in pre-merge gate for a specific PR.
+# The `perf-deliberate-tradeoff` label runs the gate with --allow-regression so
+# a deliberate correctness cost (e.g. the #4221 fsync durability fix) is
+# recorded but does not block (rustfs/backlog#935 correction 1).
 
 # DISABLED. This workflow is switched off in the repository's Actions settings
 # (state: disabled_manually) and does not run on any trigger, including its cron
@@ -46,6 +46,8 @@ on:
         required: false
         default: false
         type: boolean
+  pull_request:
+    types: [labeled, synchronize, reopened]
   push:
     # Every main commit pre-builds and caches its release binary (perf-3) so the
     # nightly A/B restores a ready baseline instead of paying the double build.
@@ -53,6 +55,14 @@ on:
 
 permissions:
   contents: read
+  pull-requests: write
+
+# Per-PR: a new push cancels the previous (up to 90-minute) A/B run instead of
+# stacking them. Nightly schedule and manual dispatch get a unique group and
+# always run to completion.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 env:
   CARGO_TERM_COLOR: always
@@ -60,8 +70,8 @@ env:
 
 jobs:
   # perf-3: on every push to main, build the release binary once and cache it
-  # keyed by commit SHA (rustfs-baseline-<sha>). The warp-ab measurements
-  # restore this instead of paying the ~32min-per-side source
+  # keyed by commit SHA (rustfs-baseline-<sha>). The nightly A/B (and, later, the
+  # perf-7 PR gate) restore this instead of paying the ~32min-per-side source
   # build. That double build is what pushed the expanded 24-cell nightly past its
   # ceiling — 2026-07-11..07-14 all cancelled on the 120min timeout. Incremental
   # builds off the shared cargo cache keep each push cheap, and building on the
@@ -116,11 +126,17 @@ jobs:
 
   warp-ab:
     name: Warp A/B budget gate
-    # Always run on schedule / manual dispatch. Never on push — that event only
-    # feeds build-baseline-cache above.
+    # Always run on schedule / manual dispatch. Opt-in on PRs: only when the
+    # `perf-ab` label is present, and for `labeled` events only when the label
+    # being added is `perf-ab` itself (adding an unrelated label to an opted-in
+    # PR must not re-run the gate). Never on push — that event only feeds
+    # build-baseline-cache above.
     if: >-
       github.event_name == 'schedule' ||
-      github.event_name == 'workflow_dispatch'
+      github.event_name == 'workflow_dispatch' ||
+      (github.event_name == 'pull_request' &&
+       contains(github.event.pull_request.labels.*.name, 'perf-ab') &&
+       (github.event.action != 'labeled' || github.event.label.name == 'perf-ab'))
     runs-on: sm-standard-2
     # With perf-3's cached baseline binary the common (cache-hit) nightly is
     # measurement-only and finishes well under 50min. This ceiling stays
@@ -158,6 +174,10 @@ jobs:
           INPUT_ALLOW_REGRESSION: ${{ github.event.inputs.allow_regression }}
         run: |
           allow="false"
+          if [[ "${{ github.event_name }}" == "pull_request" ]] \
+            && ${{ contains(github.event.pull_request.labels.*.name, 'perf-deliberate-tradeoff') }}; then
+            allow="true"
+          fi
           if [[ "$INPUT_ALLOW_REGRESSION" == "true" ]]; then
             allow="true"
           fi
@@ -294,10 +314,10 @@ jobs:
           echo "candidate binary: $cand_src"
 
           if [[ "${{ steps.exempt.outputs.allow_regression }}" == "true" ]]; then
-            args+=(--allow-regression --exemption-reason "workflow dispatch override")
+            args+=(--allow-regression --exemption-reason "labeled perf-deliberate-tradeoff / dispatch override")
           fi
           # Do not let a gate FAIL abort the job here; capture status and surface
-          # it after the step summary is written.
+          # it after the PR comment is posted.
           set +e
           bash scripts/run_hotpath_warp_abba.sh "${args[@]}"
           echo "status=$?" >> "$GITHUB_OUTPUT"
@@ -362,6 +382,13 @@ jobs:
             fi
           } >> "$GITHUB_STEP_SUMMARY"
 
+      - name: Comment gate result on PR
+        if: always() && github.event_name == 'pull_request' && steps.ab.outputs.gate_md != ''
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          gh pr comment "${{ github.event.pull_request.number }}" --body-file "${{ steps.ab.outputs.gate_md }}"
+
       # Scheduled failure alerting is handled by the alert-on-failure job below
       # (perf-2 consuming ci-8's schedule-failure-issue composite action).
 
@@ -370,7 +397,7 @@ jobs:
         run: |
           status="${{ steps.ab.outputs.status }}"
           if [[ "$status" != "0" ]]; then
-            echo "::error::warp A/B budget gate failed (exit $status). See the step summary / gate.md artifact." >&2
+            echo "::error::warp A/B budget gate failed (exit $status). See the step summary / PR comment / gate.md artifact." >&2
             exit "$status"
           fi
           echo "warp A/B budget gate passed."
@@ -380,12 +407,14 @@ jobs:
     needs: [warp-ab]
     # `always()` is required: without it this job is skipped when a needed
     # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
-    # ci-8); manual dispatch failures are already watched by a human.
+    # ci-8); PR and manual dispatch failures are already watched by a human.
     # `cancelled` is included alongside `failure` on purpose: a job that hits
     # timeout-minutes ends as `cancelled`, and the 2026-07-11..07-14 nightly
     # timeouts went silent precisely because the guard was failure-only. The
     # composite action already reports cancelled/timed-out jobs in the issue
-    # body.
+    # body. (Scheduled runs get a unique concurrency group with
+    # cancel-in-progress off, so a cancellation here means a timeout/manual
+    # abort, never a superseding run.)
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))

.github/workflows/windows-filesystem.yml

@@ -1,90 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-name: Windows Filesystem Tests
-
-on:
-  push:
-    branches: [ main ]
-    paths:
-      - "crates/ecstore/src/disk/**"
-      - "crates/ecstore/src/store/init_format.rs"
-      - "crates/ecstore/Cargo.toml"
-      - "Cargo.toml"
-      - "Cargo.lock"
-      - ".github/actions/setup/**"
-      - ".github/workflows/windows-filesystem.yml"
-  pull_request:
-    branches: [ main ]
-    paths:
-      - "crates/ecstore/src/disk/**"
-      - "crates/ecstore/src/store/init_format.rs"
-      - "crates/ecstore/Cargo.toml"
-      - "Cargo.toml"
-      - "Cargo.lock"
-      - ".github/actions/setup/**"
-      - ".github/workflows/windows-filesystem.yml"
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
-  cancel-in-progress: true
-
-env:
-  CARGO_TERM_COLOR: always
-  FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-  RUST_BACKTRACE: 1
-
-jobs:
-  rename-safety:
-    name: Rename Safety
-    runs-on: windows-latest
-    timeout-minutes: 60
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: build-x86_64-pc-windows-msvc
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-          install-test-tools: 'false'
-
-      - name: Check production Windows dependencies
-        shell: pwsh
-        run: cargo check -p rustfs-ecstore --lib
-
-      - name: Test guarded rename publication
-        shell: pwsh
-        run: cargo test -p rustfs-ecstore --lib rename_all_ -- --nocapture
-
-      - name: Test Windows handle guards
-        shell: pwsh
-        run: cargo test -p rustfs-ecstore --lib windows_ -- --nocapture
-
-      - name: Test startup temporary-directory cleanup
-        shell: pwsh
-        run: cargo test -p rustfs-ecstore --lib cleanup_tmp_on_startup_ -- --nocapture
-
-      - name: Test fresh format publication
-        shell: pwsh
-        run: cargo test -p rustfs-ecstore --lib fresh_format_load_initializes_all_disks -- --nocapture

.gitignore

@@ -83,7 +83,3 @@ worktrees/*
 
 # Local AI-agent review artifacts (omo evidence dumps)
 .omo/
-
-# insta scratch files; the accepted .snap files ARE the assertions and are committed
-*.snap.new
-*.pending-snap

.mimocode/skills/issue-triage/SKILL.md

@@ -1,116 +0,0 @@
----
-name: issue-triage
-description: Triage a GitHub issue — determine if it is already fixed, needs implementation, or should be closed. Searches related commits and PRs, verifies implementation status, and posts a triage comment or closes the issue. Use when the user provides an issue URL and asks whether it can be closed or needs work.
----
-
-# Issue Triage
-
-Use this skill when the user provides a GitHub issue URL and asks "can this be closed?", "is this already implemented?", "check completion status", or similar triage questions.
-
-## Workflow
-
-### 1. Fetch issue context
-
-```bash
-gh issue view <N> --repo <owner/repo> --json title,body,state,comments,labels,updatedAt
-```
-
-Read the issue body to understand what was requested. Extract:
-- The specific feature/fix/behavior described.
-- Any linked PRs or commits mentioned in the body or comments.
-- Any checklist items or sub-issues.
-
-### 2. Search for related work
-
-Search git history for commits referencing the issue:
-```bash
-git log --oneline --all --grep="<N>" | head -30
-```
-
-Search for related PRs:
-```bash
-gh pr list --search "fixes #<N> OR closes #<N> OR #<N>" --state all --json number,title,state,mergedAt
-```
-
-If the issue mentions specific PRs, check their status:
-```bash
-gh pr view <PR_N> --json state,mergedAt,title
-```
-
-### 3. Verify implementation
-
-For each linked or related PR that is merged, verify the fix is actually present on the current main branch:
-```bash
-git log --oneline main | grep -i "<keyword>"
-# or
-git log --oneline main --grep="<PR_N>"
-```
-
-If the issue describes a specific defect, check the relevant code to confirm the fix is in place:
-```bash
-grep -n "<pattern>" crates/<relevant>/src/<file>.rs
-```
-
-For issues with checklists, verify each item individually. If sub-items are tracked as separate issues, check those too:
-```bash
-gh issue view <SUB_N> --repo <owner/repo> --json state
-```
-
-### 4. Determine verdict
-
-- **All items fixed and merged**: Close with a summary comment listing what was fixed and which PRs.
-- **Some items fixed, some remaining**: Comment with status of each item. Do not close.
-- **Not yet implemented**: Comment with a summary of what remains. Do not close.
-- **Superseded or no longer relevant**: Close with explanation.
-
-### 5. Take action
-
-Close with comment:
-```bash
-gh issue close <N> --repo <owner/repo> --comment "<body>"
-```
-
-Comment without closing:
-```bash
-gh issue comment <N> --repo <owner/repo> --body-file /tmp/triage.md
-```
-
-Update issue labels if needed:
-```bash
-gh issue edit <N> --repo <owner/repo> --add-label "completed" --remove-label "needs-triage"
-```
-
-Always use `--body-file` for multiline content, never inline `--body`.
-
-### 6. Handle multi-issue batches
-
-When the user asks to check multiple issues (e.g., "check all issues by user X" or "scan backlog for closable issues"):
-1. List the issues: `gh issue list --repo <repo> --author <user> --state open --json number,title,updatedAt`
-2. For each issue, run steps 1-5 above.
-3. Report a summary table of all triaged issues with verdicts.
-
-## Output format
-
-### Issue Triage: #<N> — <title>
-
-**State**: OPEN / CLOSED
-**Linked PRs**: <list with merge status>
-
-#### Assessment
-<what was requested vs what is implemented>
-
-#### Verdict
-- Close — all items resolved by <PR list>
-- Keep open — <remaining items>
-- Not started — <what needs to be done>
-
-#### Action taken
-- Closed with comment / Commented / No action
-
-## Notes
-
-- The user may ask in Chinese ("是否可以关闭", "检查完成情况"); respond in the same language.
-- When closing, always include a summary of what was fixed and which PRs resolved it — this creates a useful audit trail.
-- For issues in `rustfs/backlog`, use `--repo rustfs/backlog`.
-- For issues in `rustfs/rustfs`, use `--repo rustfs/rustfs`.
-- If the issue has sub-issues (GitHub sub-issues API), check each one's state before declaring the parent complete.

.mimocode/skills/pr-review/SKILL.md

@@ -1,147 +0,0 @@
----
-name: pr-review
-description: Review a GitHub PR end-to-end from a URL or number — fetch metadata, inspect the diff, run multi-role adversarial review, check CI status, and post the review comment. Use when the user provides a PR link and asks to review it.
----
-
-# PR Review
-
-Use this skill when the user provides a GitHub PR URL or number and asks to review it. This covers the full review lifecycle: data gathering, code review, CI verification, and posting the result.
-
-## Prerequisites
-
-- Read `AGENTS.md` for the repository's adversarial validation policy and change-style rules.
-- The `adversarial-validation` skill handles the review role playbooks; this skill orchestrates the workflow around it.
-
-## Workflow
-
-### 1. Gather PR context
-
-```bash
-gh pr view <N> --json title,author,state,body,additions,deletions,changedFiles,commits,baseRefName,headRefName
-gh pr diff <N> --name-only
-```
-
-Read the PR body and linked issues to understand the change's purpose. If the PR references an issue, fetch that too:
-```bash
-gh issue view <ISSUE> --json title,body,state
-```
-
-### 2. Fetch the diff and classify the change
-
-```bash
-git fetch origin pull/<N>/head:pr-<N>
-git diff main...pr-<N> --stat
-```
-
-Classify the change by risk tier (per AGENTS.md):
-- **Exempt**: docs/comments/instruction-only, formatting, typos.
-- **Mechanical**: renames, file moves, test-only or tooling changes.
-- **Standard** (default): any behavior change.
-- **High risk**: locking, erasure coding, quorum/heal, replication, multipart, RPC, lifecycle/tiering, metadata formats, persistence/fsync, IAM/KMS/auth, on-disk/on-wire formats, S3 API-visible behavior.
-
-### 3. Cluster changed files and delegate review
-
-Group the changed files into logical clusters (by crate or functional area). For each cluster, spawn a subagent with a focused review prompt that includes:
-- The cluster's changed files and their diffs.
-- The applicable adversarial role probes (from the `adversarial-validation` skill).
-- The repository's AGENTS.md rules relevant to that domain.
-
-For standard-tier changes: correctness adversary + simplicity adversary + test-coverage skeptic, plus every role whose domain the diff touches.
-For high-risk changes: run all seven roles.
-
-Each subagent must produce findings (concrete failure scenario with file:line) or a null report ("attacked X, Y, Z — no break found").
-
-### 4. Check CI status
-
-```bash
-gh pr checks <N>
-```
-
-If any checks fail, investigate:
-```bash
-gh run view --log-failed --job=<JOB_ID>
-```
-
-Determine whether failures are pre-existing (on main), flaky, or caused by the PR.
-
-### 5. Synthesize findings
-
-Combine all subagent findings into a structured review:
-- **Summary**: one-paragraph overview of the change and overall assessment.
-- **Findings**: each finding with severity (critical/major/minor/nit), file:line, concrete failure scenario, and suggested fix.
-- **CI status**: pass/fail with notes on any failures.
-- **Verdict**: APPROVE, REQUEST_CHANGES, or COMMENT.
-
-### 6. Post the review
-
-Write the review body to a temp file and post via CLI:
-```bash
-# Request changes
-gh pr review <N> --request-changes --body-file /tmp/pr_review.md
-
-# Approve
-gh pr review <N> --approve --body-file /tmp/pr_review.md
-
-# Comment only (no verdict)
-gh pr review <N> --comment --body-file /tmp/pr_review.md
-```
-
-For inline comments on specific lines, use the GitHub API:
-```bash
-cat > /tmp/pr_review.json <<'EOF'
-{
-  "body": "review body",
-  "event": "REQUEST_CHANGES",
-  "comments": [
-    {
-      "path": "crates/foo/src/bar.rs",
-      "line": 42,
-      "body": "finding description"
-    }
-  ]
-}
-EOF
-gh api --method POST /repos/{owner}/{repo}/pulls/<N>/reviews --input /tmp/pr_review.json
-```
-
-Always use `--body-file` or `--input`, never inline multiline `--body`.
-
-### 7. Handle follow-up
-
-If the review requests changes:
-- Monitor for new commits: `gh pr view <N> --json commits`
-- Re-review changed files only: `git diff pr-<N>..origin/pull/<N>/head`
-- Update the review when findings are addressed.
-
-If CI was failing due to pre-existing main breakage:
-- Comment on the PR noting the failure is pre-existing.
-- Suggest updating the branch: `gh pr update-branch <N>`
-
-## Output format
-
-### PR Review: #<N> — <title>
-
-**Author**: <author>
-**Risk tier**: exempt | mechanical | standard | high-risk
-**Changed files**: <count> across <cluster count> clusters
-
-#### Summary
-<one-paragraph overview>
-
-#### Findings
-| Severity | Location | Finding |
-|----------|----------|---------|
-| critical | file:line | concrete failure scenario |
-
-#### CI Status
-- All checks pass / Failing: <details>
-
-#### Verdict
-APPROVE / REQUEST_CHANGES / COMMENT
-
-## Notes
-
-- The user may ask for review in Chinese; respond in the same language but keep the review body in English per AGENTS.md rules.
-- When the user asks for "多角色对抗 review", run the full adversarial validation protocol — this skill's step 3 covers that.
-- If the PR is from a fork, check `maintainerCanModify` before attempting to push fixes.
-- For very large PRs (>50 files), cluster aggressively and delegate in parallel to keep review time reasonable.

AGENTS.md

@@ -51,25 +51,26 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 
 ## Change Style for Existing Logic
 
-- Start with the smallest direct, local edit. Add production files, types, traits, helpers, wrappers, or abstraction layers only when current behavior requires them. Extraction must remove present duplication, enforce a real boundary, or materially clarify a non-trivial flow; anticipated reuse is not enough.
+- Prefer direct, local code over extracting one-off helpers.
+- Extract a helper only when logic is reused or the extraction materially clarifies a non-trivial flow.
 - Use Rust's default module file layout (`mod foo;` with `foo.rs` or `foo/mod.rs`/`foo/*.rs`).
   Avoid `#[path = "..."]` for module inclusion; move files into the canonical module tree instead.
   If an unavoidable generated-code, FFI, or test-fixture exception remains, keep it local and document why the canonical layout cannot work.
 - Solve only the requested problem; do not add speculative features, configurability, or adjacent improvements.
 - Prefer editing existing code over rewriting files or reshaping unrelated logic.
-- Modify only what is required. Remove any in-scope path or representation superseded by the change. If compatibility or rollback requires retention, adapt at the boundary to one canonical core and follow the repository's `RUSTFS_COMPAT_TODO` removal policy; never delete unrelated code merely to improve addition/deletion statistics.
+- Modify only what is required and remove only artifacts introduced by your own changes.
 - Preserve the existing control-flow and logic shape when fixing bugs or addressing review comments, especially in init, distributed coordination, locking, metadata, and concurrency paths.
 - Do not refactor existing code only to make it easier to unit test.
 - Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths.
-- Keep code elegant, concise, and direct. Prefer the smallest readable design and existing abstractions over parallel managers, factories, adapters, or wrappers added only to make the design look extensible.
-- Comments state non-obvious reasons, assumptions, and invariants in the shortest complete form. Their length follows the invariant's complexity: `SAFETY`, lock ordering, durability, and compatibility contracts may need a short list of conditions. Never narrate the next line, restate a signature, or record change history; move durable design rationale to architecture or operations documentation.
+- Keep code elegant, concise, and direct. Prefer minimal, readable implementations over over-engineering and excessive abstraction. Use comments to clarify non-obvious intent and invariants, not to compensate for unclear code.
+- Do not write comments that narrate what the next line does, restate a signature, or describe the change you just made — that commentary belongs in the PR description, not the code. Required invariant comments — lock ordering, `SAFETY`, unwrap justification, `#[allow(dead_code)]` rationale, `RUSTFS_COMPAT_TODO` — are never narration.
 - Mention unrelated issues when useful, but do not fix them as part of a narrow task.
 
 ## Reuse Before You Write
 
 Search for an existing implementation before writing a new one; extend what exists instead of duplicating it:
 
-- **Helpers and utilities** (path/string handling, hashing, retry, env parsing, IO wrappers): check `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, and relevant direct workspace dependencies from `Cargo.toml`. Search snake_case signatures with a focused term. Reimplementing an existing workspace helper — or hand-rolling what `std`, `tokio`, or an existing dependency already provides — is a review finding, not a style preference.
+- **Helpers and utilities** (path/string handling, hashing, retry, env parsing, IO wrappers): check `ls crates/utils/src` first — file names map to operations (`retry.rs`, `envs.rs`, `hash.rs`, `path.rs`, `string.rs`, `io.rs`) — plus `crates/common` (shared structures/globals), then `rg -i 'fn \w*<term>' crates/utils/src crates/common/src <touched-crate>/src` for signatures. Helpers are snake_case: a full-text single-word grep over a large crate drowns you and a multi-word phrase returns nothing. Reimplementing an existing workspace helper — or hand-rolling what `std`, `tokio`, or an existing workspace dependency already provides — is a review finding, not a style preference.
 - **Reuse requires matching semantics, not a matching name**: before adopting a helper, check its normalization (`clean` resolves `.`/`..` — never apply it to raw S3 object keys), error type, backoff/deadline behavior, and durability gating against the call site. When semantics differ, a new narrowly-named helper with a comment naming the rejected lookalike is the correct outcome. The inverse also holds: workspace wrappers exist because raw `std`/`tokio` semantics were insufficient (durability gates, retries) — prefer the wrapper over the raw call.
 - **Constants and fixed tokens** (protocol labels, error identifiers, header keys, event names, metric names, command tags): search for existing constants/enums that already represent the same semantic value and reuse them. If a value is truly new, define one local constant near related logic; never scatter the literal across sites. When changing existing behavior, align naming and format with the established constants.
 - **Test scaffolding**: reuse existing test utilities and fixtures (the touched crate's own `test_util` module and `tests/fixtures`, or `crates/test-utils`) instead of writing new setup code — run `rg -l '<fn-under-test>' <crate>/src <crate>/tests` before writing a test. A new test must pin a failure mode no existing test covers. Near-duplicate means same code path AND same poison-value class: this repo's boundary companions (n==max vs max+1, absent vs empty vs nil UUID bytes, MetaObject vs MetaDeleteMarker) are distinct by definition and must all be written.
@@ -78,7 +79,6 @@ Search for an existing implementation before writing a new one; extend what exis
 
 Net-new code — files, types, branches, comments — is cost to justify, not progress:
 
-- Inspect production-code additions separately. Tests, fixtures, generated code, and documentation do not count as production-code growth. Line counts are signals, not quotas: new production structures must map to a current requirement, and a blocker requires a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries.
 - Validate at the trust boundary — untrusted client input, bytes read from disk, RPC payloads, config (see Serde Safety and Cross-Cutting Domain Invariants) — then trust the type: do not re-check what the type system or a validated upstream layer already guarantees, and cite the establishing check (`file:line`) when the guarantee is not obvious.
 - The exception is load-bearing: a value that crossed a persistence, RPC, or version boundary is never guaranteed by the code on the other side — a peer may be older or buggy, disk bytes may be corrupt — so the Cross-Cutting Domain Invariant patterns apply at every consumer, and re-checks immediately before a destructive action (delete, overwrite, quorum decision) stay. Deleting an existing guard is a behavior change requiring adversarial review, not cleanup.
 - Every new branch needs a nameable trigger: a concrete input, state, or failure that reaches it — for boundary-crossing values, corrupt or stale persisted/peer data is always nameable. If you cannot name one, do not write the branch. If the case is truly unreachable, encode the invariant in the type; where that is impossible, return a typed internal error (fail closed). `debug_assert!` is acceptable only for pure internal arithmetic on values that never crossed a disk/RPC/config boundary — never as the sole guard on decoded or peer-supplied data.
@@ -218,10 +218,9 @@ not to bless it.
 
 Pick the tier from the riskiest file touched; when in doubt, pick the higher.
 
-- **Exempt:** docs/comments, formatting, and typos that cannot affect runtime,
-  builds, tests, or agent execution. Skip this section.
-- **Mechanical:** pure renames, file moves, test-only or tooling changes, and
-  agent-instruction changes that alter execution —
+- **Exempt:** docs/comments/instruction-only changes, formatting, typos with
+  no runtime surface. Skip this section.
+- **Mechanical:** pure renames, file moves, test-only or tooling changes —
   correctness and simplicity adversaries only.
 - **Standard (the default):** any change that affects behavior.
 - **High risk:** touches locking, erasure coding, quorum/heal, replication,
@@ -243,7 +242,7 @@ encode this repo's shipped bugs.
 - **Correctness adversary** — construct a concrete input/state/interleaving
   that yields wrong output, data loss, or a crash. Probe error paths and edge
   values (empty, nil UUID, zero-length, quorum−1, missing version).
-- **Simplicity adversary** — same behavior, less code. Hunt reimplemented helpers, rewrites where an in-place edit suffices, speculative abstractions, defensive branches with no nameable trigger, redundant error wrapping, near-duplicate tests, and narration comments. A one-caller helper is a finding only when it merely forwards or splits a short linear flow without adding domain naming, boundary isolation, an invariant, or useful error context. Report a concrete smaller replacement; fewer lines alone are not evidence.
+- **Simplicity adversary** — same behavior, less code. Hunt the materially smaller or more idiomatic diff (see Change Style for Existing Logic, Reuse Before You Write, and Necessary Code Only): reimplemented workspace helpers, one-caller extractions, rewrites where an in-place edit suffices, defensive branches with no nameable trigger, redundant error wrapping, near-duplicate tests, narration comments. A smaller diff achieving identical behavior is a finding, reported with the concrete replacement; forced reuse of a helper with mismatched semantics is equally a finding.
 - **Security reviewer** — authn/authz bypass, injection, secret leakage,
   untrusted deserialization (see Serde Safety), path traversal, timing leaks.
 - **Concurrency/durability reviewer** — lock ordering, races, cancellation,
@@ -254,11 +253,10 @@ encode this repo's shipped bugs.
   time across IO, sync or CPU-heavy work on async runtime threads, added
   fsync/flush outside the durability gate, hot-path logging noise. A
   measurable regression on a per-request or per-object path is a finding.
-- **Test-coverage skeptic** — for each testable behavior claim, name the test
-  or executable check that detects a revert; then name a changed line that
-  could be wrong while all checks stay green. If a focused check is not
-  reasonable, require the reason and residual risk from the validation floor.
-  Test additions have no line-count or growth budget.
+- **Test-coverage skeptic** — for each claimed behavior, name the test that
+  fails if the change is reverted; then name a changed line that could be
+  wrong while all tests stay green — if one exists, coverage is insufficient.
+  A missing test is a finding, not a note.
 
 Standard tier: correctness adversary + simplicity adversary + test-coverage
 skeptic, plus every role whose domain the diff touches (async or
@@ -284,9 +282,7 @@ High risk: all seven roles.
 
 - Every applicable role has run; every finding is fixed or rebutted with
   evidence.
-- Every testable behavior change has a focused regression check. Exceptions
-  follow the validation floor and state why a check is impractical and what
-  risk remains.
+- Every behavior change has a test that fails without it.
 - The Verification Before PR gates pass — adversarial review supplements
   those gates, never replaces them.
 - High risk only: record a one-line verdict per role in the PR description.
@@ -326,28 +322,6 @@ High risk: all seven roles.
 - Use environment variables or vault tooling for sensitive configuration.
 - For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
 
-## Logging
-
-Applies to **every** `tracing` macro you add or edit, including a single line
-added in passing while fixing something else — not only to log-focused changes.
-
-- Fields first, message second: `event`, `component`, `subsystem`,
-  `result`/`state`, then key context. The message is a short label, not a
-  sentence with values interpolated into it.
-- Reuse the existing `EVENT_*` / `LOG_COMPONENT_*` / `LOG_SUBSYSTEM_*`
-  constants of the module you are editing; match the shape of the log sites
-  already in that file rather than introducing a second style next to them.
-- Level policy: `error` for behavior/security-affecting failures, `warn` for
-  degraded or fallback paths, `info` for low-frequency lifecycle, `debug` for
-  targeted diagnostics, `trace` for hot paths. Per-object and per-request
-  success paths are `trace`.
-- Never log secrets, tokens, credential payloads, or merged config dumps.
-- `scripts/check_logging_guardrails.sh` enforces a subset of this on the files
-  it lists; passing it is a floor, not evidence the log matches the house style.
-
-See `.agents/skills/rustfs-logging-governance/SKILL.md` for the full event
-model, level policy, and guardrail-update checklist.
-
 ## Tools
 
 ### xl.meta decode tool Quick Use
@@ -373,11 +347,6 @@ cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/file/xl.meta"
   absent, empty, and nil all mean "no value", never `Uuid::nil()`.
 - A remote-tier version of `None`/`""` means the tier bucket is unversioned:
   send **no** `versionId` on tier GET/DELETE.
-- Structs persisted in the scanner data-usage cache (`DataUsageCacheInfo`,
-  `DataUsageEntry`) carry a hand-written map-encoded `Serialize`. MessagePack
-  encodes derived structs as arrays, where an appended field makes the whole
-  cache a decode error for older readers — keep new fields `#[serde(default)]`
-  and keep the map encoding rather than reverting to `derive(Serialize)`.
 
 ## Naming Conventions
 

ARCHITECTURE.md

@@ -1,6 +1,6 @@
 # ARCHITECTURE.md
 
-> Last updated: 2026-08-12 · Revision: 3
+> Last updated: 2026-07-02 · Revision: 2
 >
 > This document describes the high-level architecture of RustFS.
 > If you want to familiarize yourself with the code base, you are in the right place!
@@ -119,44 +119,19 @@ module split is tracked under `docs/architecture/`.
 
 3. **Each type has exactly one definition.** Types shared across crates must be defined
    in one crate and re-exported or imported by others.
-   - ⚠️ VIOLATED: `ReplicationStats` names three unrelated types
-     (`crates/data-usage/src/data_usage.rs`,
-     `crates/obs/src/metrics/collectors/replication.rs`,
-     `crates/ecstore/src/bucket/replication/replication_state.rs`) — a naming
-     collision, not copies; renaming is tracked in rustfs/backlog#1847.
-   - `LastMinuteLatency` has two deliberately different implementations: the
-     per-second bucketed accumulator in `crates/common/src/last_minute.rs` and
-     the in-memory endpoint-health sample tracker in
-     `crates/ecstore/src/bucket/bucket_target_sys.rs` (its doc comment explains
-     why it stays local).
-   - ✅ RESOLVED: `BackpressureConfig` and `DataUsageInfo` each have exactly one
-     definition (`crates/io-core/src/backpressure.rs`,
-     `crates/data-usage/src/data_usage.rs`). The zero-consumer
-     `BackpressureSettings` copy that lingered in io-metrics was removed
-     (rustfs/backlog#1833).
+   - ⚠️ VIOLATED: `ReplicationStats` (4 copies), `LastMinuteLatency` (3 copies),
+     `BackpressureConfig` (3 copies), `DataUsageInfo` (2 copies).
 
 4. **ecstore does not know about HTTP or S3 protocol details.** It operates on
    storage-level abstractions (objects, buckets, disks, pools).
-   - ⚠️ VIOLATED: 58 files under `crates/ecstore/src` reference `s3s`
-     (`rg -l 's3s' crates/ecstore/src | wc -l`), `crates/ecstore/src/client/`
-     is a ~9.4K-line embedded S3 HTTP client, and `crates/ecstore/Cargo.toml`
-     depends on `s3s`, `http`, `hyper`/`hyper-util`/`hyper-rustls`, and
-     `reqwest`. Target state: the engine's need to act as an S3 client
-     (tiering, replication targets) is served by an extracted client crate,
-     and ecstore holds no wire or DTO types.
 
 5. **The `rustfs` binary crate is the only place that wires everything together.**
    Individual crates should be testable in isolation.
 
 6. **Error types use `thiserror` with descriptive names** (e.g., `StorageError`,
    not bare `Error`).
-   - ✅ RESOLVED (strategy): `snafu` is gone from source
-     (`rg -l snafu crates/ rustfs/` is empty) and library code no longer uses
-     `anyhow` (remaining hits are test code and the `e2e_test` crate; `heal`
-     uses `thiserror`).
-   - ⚠️ VIOLATED (naming): 6 crates still export a bare `pub enum Error`:
-     `crypto`, `filemeta`, `heal`, `iam`, `policy`, and `replication`
-     (`src/resync.rs`) — all `thiserror`-derived.
+   - ⚠️ VIOLATED: 6 crates use `pub enum Error`; 2 crates use `snafu`;
+      `heal` use `anyhow` in library code.
 
 ## Known Structural Issues
 
@@ -165,25 +140,13 @@ module split is tracked under `docs/architecture/`.
 
 ### Critical
 
-- **scanner/data-usage duplicate `.usage-cache.bin` serialization types.** The
-  original finding ("common/scanner code duplication, ~3K lines") is resolved:
-  `scanner` imports the shared data-usage types from `rustfs-data-usage` (see
-  the `pub use rustfs_data_usage::…` re-exports at the top of
-  `crates/scanner/src/data_usage_define.rs`). What remains: `scanner` and
-  `data-usage` each hold their own serialization types for the scanner cache
-  file (`DataUsageCacheInfo`/`DataUsageEntryInfo` in
-  `crates/scanner/src/data_usage_define.rs` vs
-  `DataUsageCacheInfo`/`DataUsageEntry` in
-  `crates/data-usage/src/data_usage.rs`); convergence is tracked in
-  rustfs/backlog#1828.
+- **common/scanner code duplication (~3K lines).** `scanner` depends on `common`
+  but maintains its own copies of `DataUsageInfo`, `LastMinuteLatency`, and related
+  types instead of importing them.
 
-- **ecstore is a monolith (265 files, ~288K lines — roughly half is inline
-  `#[cfg(test)]` code).** Measured with
-  `find crates/ecstore/src -name '*.rs' | xargs wc -l`. It contains disk
-  management, bucket management, erasure coding, replication, lifecycle, RPC,
-  and configuration — all in one crate. It should be decomposed along its
-  existing subdirectories; the split plan lives in
-  [docs/architecture/ecstore-module-split-plan.md](docs/architecture/ecstore-module-split-plan.md).
+- **ecstore is a monolith (87K lines, 163 files).** It contains disk management,
+  bucket management, erasure coding, replication, lifecycle, RPC, and configuration
+  — all in one crate. It should be decomposed along its existing subdirectories.
 
 ### High
 
@@ -191,26 +154,19 @@ module split is tracked under `docs/architecture/`.
   `common → filemeta/madmin` edges must stay removed so leaf/helper crates do
   not regain upward dependencies.
 
-- **Three-layer backpressure/deadlock policy bridging** across io-core,
-  concurrency, and `rustfs/src/storage`. The config types are no longer
-  duplicated (`BackpressureConfig` and `DeadlockDetectorConfig` are each
-  defined once, in io-core). Storage policies expose and consume explicit
-  projections into the concurrency/io-core policy shapes, and workload
+- **Three-layer BackpressureConfig/DeadlockConfig duplication** across io-core,
+  concurrency, and `rustfs/src/storage`. Storage policies now expose and consume
+  explicit projections into the concurrency/io-core policy shapes, and workload
   admission snapshots are composed through provider registries; later work
   should use those bridges before deleting compatibility wrappers.
 
 ### Medium
 
-- **Bare `Error` naming.** Error-handling strategy has converged on `thiserror`
-  (no `snafu`, no `anyhow` in library code); the remaining inconsistency is the
-  bare `pub enum Error` naming in the 6 crates listed under Invariant 6.
+- **Inconsistent error handling.** Three strategies (thiserror/snafu/anyhow) and
+  mixed naming (bare `Error` vs descriptive names).
 
-- **`common` is mostly parked domain code, not shared utilities.** Of its
-  6,724 lines, ~83% is scanner/heal domain code stranded there to break
-  dependency cycles (`metrics.rs`, ~4,810 lines of scanner-domain metrics;
-  `heal_channel.rs`, ~776 lines of heal-domain channel types). The
-  "common vs utils" naming ambiguity is secondary to moving that code to its
-  domain owners.
+- **Ambiguous common vs utils boundary.** Both described as "utilities and data
+  structures." Need clear ownership rules.
 
 ## Cross-Cutting Concerns
 
@@ -276,7 +232,7 @@ The binary (`main.rs`) boots in this order:
 
 ```
                             ┌─────────┐
-                            │  rustfs │  (binary + lib)
+                            │  rustfs │  (binary + lib, 75K lines)
                             │  main   │
                             └────┬────┘
                                  │
@@ -299,7 +255,7 @@ The binary (`main.rs`) boots in this order:
               │                  │                  │
         ┌─────▼──────┐    ┌──────▼──────┐    ┌──────▼──────┐
         │  ecstore   │    │     rio     │    │   io-core   │
-        │   (core)   │    │  (readers)  │    │ (zero-copy) │
+        │ (87K,core) │    │  (readers)  │    │ (zero-copy) │
         └─────┬──────┘    └─────────────┘    └─────────────┘
               │
      ┌─────┬──┼──┬─────┬──────┐

Cargo.toml

@@ -69,7 +69,7 @@ edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
 rust-version = "1.97.1"
-version = "1.0.0-rc.1"
+version = "1.0.0-beta.12"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -86,52 +86,52 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-rc.1" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-rc.1" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-rc.1" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-rc.1" }
-rustfs-common = { path = "crates/common", version = "1.0.0-rc.1" }
-rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-rc.1" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-rc.1" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-rc.1" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-rc.1" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-rc.1" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-rc.1" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-rc.1" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-rc.1" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-rc.1" }
-rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-rc.1" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-rc.1" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-rc.1" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-rc.1" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-rc.1" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-rc.1" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-rc.1" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-rc.1" }
-rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-rc.1", default-features = false }
-rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-rc.1" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-rc.1" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-rc.1" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-rc.1" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-rc.1" }
-rustfs-replication = { path = "crates/replication", version = "1.0.0-rc.1" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-rc.1" }
-rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-rc.1" }
-rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-rc.1" }
-rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-rc.1" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-rc.1" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-rc.1" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-rc.1" }
-rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-rc.1" }
-rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-rc.1" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-rc.1" }
-rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-rc.1" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-rc.1" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-rc.1" }
-rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-rc.1" }
-rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-rc.1" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-rc.1" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-rc.1" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.12" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.12" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.12" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.12" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.12" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.12" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.12" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.12" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.12" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.12" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.12" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.12" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.12" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.12" }
+rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.12" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.12" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.12" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.12" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.12" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.12" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.12" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.12" }
+rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.12" }
+rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-beta.12" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.12" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.12" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.12" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.12" }
+rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.12" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.12" }
+rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.12" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.12" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.12" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.12" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.12" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.12" }
+rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.12" }
+rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.12" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.12" }
+rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.12" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.12" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.12" }
+rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-beta.12" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.12" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.12" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.12" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
@@ -139,13 +139,13 @@ async_zip = { default-features = false, version = "0.0.18" }
 mysql_async = { default-features = false, version = "0.37" }
 async-compression = { version = "0.4.43" }
 async-recursion = "1.1.1"
-async-trait = "0.1.92"
+async-trait = "0.1.91"
 async-nats = { version = "0.50.0", default-features = false }
 axum = "0.8.9"
-futures = "0.3.34"
-futures-core = "0.3.34"
+futures = "0.3.33"
+futures-core = "0.3.33"
 futures-lite = "2.6.1"
-futures-util = "0.3.34"
+futures-util = "0.3.33"
 pollster = "1.0.1"
 pulsar = { default-features = false, version = "6.8.0" }
 lapin = { default-features = false, version = "4.10.0" }
@@ -154,7 +154,7 @@ hyper-rustls = { default-features = false, version = "0.27.9" }
 hyper-util = { version = "0.1.20" }
 http = "1.5.0"
 http-body = "1.1.0"
-http-body-util = "0.1.5"
+http-body-util = "0.1.4"
 minlz = "1.2.3"
 reqwest = "0.13.4"
 rustfs-kafka-async = { version = "1.2.0" }
@@ -171,9 +171,9 @@ tower = { version = "0.5.3" }
 tower-http = { version = "0.7.0" }
 
 # Serialization and Data Formats
-apache-avro = "0.22.0"
+apache-avro = "0.21.0"
 bytes = { version = "1.12.1" }
-bytesize = "2.7.0"
+bytesize = "2.6.0"
 byteorder = "1.5.0"
 flatbuffers = "25.12.19"
 form_urlencoded = "1.2.2"
@@ -182,7 +182,6 @@ quick-xml = "0.41.0"
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.229" }
-serde_ignored = { version = "0.1" }
 serde_json = { version = "1.0.151" }
 serde_urlencoded = "0.7.1"
 
@@ -213,7 +212,7 @@ zeroize = { version = "1.9.0" }
 chrono = { version = "0.4.45" }
 humantime = "2.4.0"
 jiff = { version = "0.2.35" }
-time = { version = "0.3.55" }
+time = { version = "0.3.54" }
 
 # Database
 deadpool-postgres = { version = "0.14" }
@@ -229,15 +228,15 @@ atomic_enum = "0.3.0"
 aws-config = { version = "1.10.1" }
 aws-credential-types = { version = "1.3.0" }
 aws-sdk-kms = { default-features = false, version = "1.114.0" }
-aws-sdk-s3 = { default-features = false, version = "1.141.0" }
+aws-sdk-s3 = { default-features = false, version = "1.140.0" }
 aws-sdk-sts = { default-features = false, version = "1.110.0" }
-aws-smithy-http-client = { default-features = false, version = "1.3.0" }
+aws-smithy-http-client = { default-features = false, version = "1.2.0" }
 aws-smithy-runtime-api = { version = "1.14.0" }
-aws-smithy-types = { version = "1.6.2" }
-base64 = "0.23.1"
+aws-smithy-types = { version = "1.6.1" }
+base64 = "0.23.0"
 base64-simd = "0.8.0"
 brotli = "8.0.4"
-clap = { version = "4.6.6" }
+clap = { version = "4.6.5" }
 const-str = { version = "1.1.0" }
 convert_case = "0.11.0"
 criterion = { version = "0.8" }
@@ -245,7 +244,7 @@ crossbeam-queue = "0.3.13"
 crossbeam-channel = "0.5.16"
 crossbeam-deque = "0.8.7"
 crossbeam-utils = "0.8.22"
-datafusion = { default-features = false, git = "https://github.com/apache/datafusion.git", rev = "e08aed1e5de41dcf81d529140dae07723b942a5e" }
+datafusion = { default-features = false, git = "https://github.com/apache/datafusion.git", rev = "dae03ee062b2abf986de8df12ea82fb1578a2d99" }
 #datafusion = { default-features = false, version = "54.1.0" }
 derive_builder = "0.20.2"
 enumset = "1.1.14"
@@ -269,17 +268,18 @@ lz4 = "1.28.1"
 matchit = "0.9.2"
 md-5 = "0.11.0"
 mime_guess = "2.0.5"
-moka = { version = "0.12.16" }
+moka = { version = "0.12.15" }
 netif = "0.1.6"
 num_cpus = { version = "1.17.0" }
 nvml-wrapper = "0.12.1"
 parking_lot = "0.12.5"
 path-absolutize = "4.0.1"
+path-clean = "1.0.1"
 percent-encoding = "2.3.2"
 pin-project-lite = "0.2.17"
 pretty_assertions = "1.4.1"
 rand = { version = "0.10.2" }
-ratelimit = "2.0.0"
+ratelimit = "0.10.1"
 rayon = "1.12.0"
 reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.2" }
 reed-solomon-simd = "3.1.0"
@@ -290,12 +290,12 @@ rustify = { version = "0.7", default-features = false }
 rustix = { version = "1.1.4" }
 rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }
-s3s = { git = "https://github.com/rustfs/s3s.git", rev = "d7028511a53f69d41ed3c69f36899f9b1aede647" }
+s3s = { git = "https://github.com/cxymds/s3s.git", rev = "fe3941d91fa1c69956f209a9145995c9f0235bff" }
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.3"
 smallvec = { version = "1.15.2" }
-compact_str = "0.10.0"
+smartstring = "1.0.1"
 snap = "1.1.2"
 starshard = { version = "2.2.2" }
 strum = { version = "0.28.0" }
@@ -303,7 +303,7 @@ sysinfo = "0.39.6"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
-thiserror = "2.0.20"
+thiserror = "2.0.19"
 tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-core = "0.1.36"
@@ -340,22 +340,22 @@ pyroscope = { version = "2.1.1" }
 libunftp = { version = "0.23.0" }
 unftp-core = "0.1.0"
 suppaftp = { version = "10.0.1" }
-rcgen = { version = "0.14.9", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
-russh = { version = "0.62.6" }
-russh-sftp = "2.4.0"
+rcgen = { version = "0.14.8", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
+russh = { version = "0.62.5" }
+russh-sftp = "2.3.0"
 
 # WebDAV
 dav-server = "0.11.0"
 
 # Performance Analysis and Memory Profiling
-mimalloc = { version = "0.1.52", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11" }
-libmimalloc-sys = { version = "0.1.49", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11", features = ["extended"] }
-hotpath = { version = "0.23.2", default-features = false }
+mimalloc = { version = "0.1.52", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "1cdadea43e9c5a0f054b65be21200ce580e4eb13" }
+libmimalloc-sys = { version = "0.1.49", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "1cdadea43e9c5a0f054b65be21200ce580e4eb13", features = ["extended"] }
+hotpath = { version = "0.22.0", default-features = false }
 # Snapshot testing for output format regression detection
 insta = { version = "1.48" }
 
 [workspace.metadata.cargo-shear]
-ignored = ["hotpath", "rustfs"]
+ignored = ["rustfs"]
 
 [profile.dev]
 # Full debuginfo roughly doubles compile+link time and produces multi-GB

Dockerfile

@@ -91,12 +91,7 @@ LABEL name="RustFS" \
 # Upgrade base-image packages so published images pick up security fixes
 # (e.g. openssl/libssl3 CVEs) without waiting for a new Alpine point release.
 RUN apk upgrade --no-cache && \
-    apk add --no-cache \
-    ca-certificates \
-    coreutils \
-    curl \
-    tzdata \
-    && test "$(TZ=Asia/Kolkata date +%z)" = "+0530"
+    apk add --no-cache ca-certificates coreutils curl
 
 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
 COPY --from=build /build/rustfs /usr/bin/rustfs

Dockerfile.glibc

@@ -96,11 +96,9 @@ LABEL name="RustFS" \
 # Upgrade base-image packages so published images pick up security fixes
 # (e.g. tar/gzip/perl CVEs) without waiting for a new Ubuntu point release.
 RUN apt-get update && apt-get upgrade -y \
-    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    && apt-get install -y --no-install-recommends \
     ca-certificates \
     curl \
-    tzdata \
-    && test "$(TZ=Asia/Kolkata date +%z)" = "+0530" \
     && rm -rf /var/lib/apt/lists/*
 
 COPY --from=build /build/rustfs /usr/bin/rustfs

README.md

@@ -116,7 +116,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-rc.1
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.12
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command

README_ZH.md

@@ -113,7 +113,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # 使用指定版本运行
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-rc.1
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.12
 ```
 
 如果您通过绑定挂载启用 TLS 证书目录，也请用同样方式准备该目录：

crates/audit/Cargo.toml

@@ -55,10 +55,10 @@ hotpath.workspace = true
 rustfs-targets = { workspace = true }
 rustfs-config = { workspace = true, features = ["audit", "server-config-model"] }
 rustfs-s3-types = { workspace = true }
+chrono = { workspace = true, features = ["serde"] }
 const-str = { workspace = true, features = ["std", "proc"] }
 futures = { workspace = true }
 hashbrown = { workspace = true, features = ["serde", "rayon"] }
-jiff = { workspace = true, features = ["serde"] }
 metrics = { workspace = true }
 serde = { workspace = true, features = ["derive"] }
 serde_json = { workspace = true, features = ["raw_value"] }

crates/audit/src/entity.rs

@@ -12,8 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use chrono::{DateTime, Utc};
 use hashbrown::HashMap;
-use jiff::Timestamp;
 use rustfs_s3_types::EventName;
 use serde::{Deserialize, Serialize};
 use serde_json::Value;
@@ -151,8 +151,8 @@ pub struct AuditEntry {
     pub deployment_id: Option<String>,
     #[serde(rename = "siteName", skip_serializing_if = "Option::is_none")]
     pub site_name: Option<String>,
-    #[serde(with = "jiff::fmt::serde::timestamp::millisecond::required")]
-    pub time: Timestamp,
+    #[serde(with = "chrono::serde::ts_milliseconds")]
+    pub time: DateTime<Utc>,
     pub event: EventName,
     #[serde(rename = "type", skip_serializing_if = "Option::is_none")]
     pub entry_type: Option<String>,
@@ -198,7 +198,7 @@ impl AuditEntryBuilder {
     pub fn new(version: impl Into<String>, event: EventName, trigger: impl Into<String>, api: ApiDetails) -> Self {
         Self(AuditEntry {
             version: version.into(),
-            time: Timestamp::now(),
+            time: Utc::now(),
             event,
             trigger: trigger.into(),
             api,
@@ -232,7 +232,7 @@ impl AuditEntryBuilder {
         self
     }
 
-    pub fn time(mut self, time: Timestamp) -> Self {
+    pub fn time(mut self, time: DateTime<Utc>) -> Self {
         self.0.time = time;
         self
     }
@@ -342,23 +342,4 @@ mod tests {
         assert_eq!(value["requestID"], Value::String("req-audit-123".to_string()));
         assert!(value.get("request_id").is_none(), "historical audit contract must not expose request_id");
     }
-
-    #[test]
-    fn audit_entry_time_serializes_as_epoch_milliseconds() {
-        let entry = AuditEntryBuilder::new(
-            "1",
-            EventName::ObjectCreatedPut,
-            "s3",
-            ApiDetailsBuilder::new()
-                .name("PutObject")
-                .status("OK")
-                .status_code(200)
-                .build(),
-        )
-        .time(Timestamp::from_millisecond(1_711_423_698_870).expect("timestamp should be valid"))
-        .build();
-
-        let value = serde_json::to_value(entry).expect("audit entry should serialize");
-        assert_eq!(value["time"], Value::Number(1_711_423_698_870_i64.into()));
-    }
 }

crates/audit/tests/performance_test.rs

@@ -97,7 +97,7 @@ async fn test_audit_log_dispatch_performance() {
         return; // Alternatively: assert!(false, "AuditSystem failed to start");
     }
 
-    use jiff::Timestamp;
+    use chrono::Utc;
     use rustfs_targets::EventName;
     use serde_json::json;
     use std::collections::HashMap;
@@ -136,7 +136,7 @@ async fn test_audit_log_dispatch_performance() {
         version: "1".to_string(),
         deployment_id: Some(format!("test-deployment-{id}")),
         site_name: Some("test-site".to_string()),
-        time: Timestamp::now(),
+        time: Utc::now(),
         event: EventName::ObjectCreatedPut,
         entry_type: Some("object".to_string()),
         trigger: "api".to_string(),
@@ -298,7 +298,7 @@ fn test_performance_requirements() {
     for i in 0..3000 {
         // Simulate event name parsing and processing
         let _event_id = format!("s3:ObjectCreated:Put_{i}");
-        let _timestamp = jiff::Timestamp::now().to_string();
+        let _timestamp = chrono::Utc::now().to_rfc3339();
 
         // Simulate basic audit entry creation overhead
         let _entry_size = 512; // bytes

crates/audit/tests/system_integration_test.rs

@@ -264,7 +264,7 @@ fn create_sample_audit_entry() -> AuditEntry {
 }
 
 fn create_sample_audit_entry_with_id(id: u32) -> AuditEntry {
-    use jiff::Timestamp;
+    use chrono::Utc;
     use rustfs_targets::EventName;
     use serde_json::json;
 
@@ -301,7 +301,7 @@ fn create_sample_audit_entry_with_id(id: u32) -> AuditEntry {
         version: "1".to_string(),
         deployment_id: Some(format!("test-deployment-{id}")),
         site_name: Some("test-site".to_string()),
-        time: Timestamp::now(),
+        time: Utc::now(),
         event: EventName::ObjectCreatedPut,
         entry_type: Some("object".to_string()),
         trigger: "api".to_string(),

crates/checksums/src/http.rs

@@ -21,13 +21,6 @@ use crate::{
     Xxhash3, Xxhash64, Xxhash128,
 };
 
-// DELIBERATE DUPLICATION of the x-amz-checksum-* names that also exist as
-// AMZ_CHECKSUM_* in rustfs-utils' headers module (crates/utils/src/http/
-// headers.rs): this crate is a zero-internal-dependency leaf, so it cannot
-// import them, and it additionally owns the RustFS extension names
-// (sha512/xxhash*) that utils does not carry. Values are pinned by the S3
-// wire protocol; do not merge without a maintainer decision on the leaf
-// boundary (backlog#1833).
 pub const CRC_32_HEADER_NAME: &str = "x-amz-checksum-crc32";
 pub const CRC_32_C_HEADER_NAME: &str = "x-amz-checksum-crc32c";
 pub const SHA_1_HEADER_NAME: &str = "x-amz-checksum-sha1";

crates/checksums/src/lib.rs

@@ -41,14 +41,6 @@ pub const XXHASH_64_NAME: &str = "xxhash64";
 pub const XXHASH_128_NAME: &str = "xxhash128";
 pub const MD5_NAME: &str = "md5";
 
-/// One of three deliberately separate checksum registries (backlog#1833):
-/// this enum owns the **streaming-hash algorithm registry**, including the
-/// RustFS extensions (sha512, xxhash3/64/128). The on-disk xl.meta bitset
-/// lives in `rustfs_rio::ChecksumType` (crates/rio/src/checksum.rs, varint
-/// bits are append-only), and the MinIO-port client keeps its own
-/// `ChecksumMode` (crates/ecstore/src/client/checksum.rs). When adding an
-/// algorithm, extend all three (or record why not) — they do not derive from
-/// each other.
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
 #[non_exhaustive]
 pub enum ChecksumAlgorithm {

crates/common/Cargo.toml

@@ -39,15 +39,11 @@ tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
 tonic = { workspace = true, features = ["gzip", "deflate"] }
 uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
 chrono = { workspace = true, features = ["serde"] }
-jiff = { workspace = true, features = ["serde"] }
 metrics = { workspace = true }
 serde = { workspace = true, features = ["derive"] }
 rmp-serde = { workspace = true }
 s3s = { workspace = true, features = ["minio"] }
 tracing = { workspace = true }
 
-[dev-dependencies]
-serde_json = { workspace = true }
-
 [lib]
 doctest = false

crates/common/src/bucket_stats.rs

@@ -0,0 +1,87 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::last_minute::{self};
+use std::collections::HashMap;
+
+pub struct ReplicationLatency {
+    // Delays for single and multipart PUT requests
+    upload_histogram: last_minute::LastMinuteHistogram,
+}
+
+impl ReplicationLatency {
+    // Merge two ReplicationLatency
+    pub fn merge(&mut self, other: &mut ReplicationLatency) -> &ReplicationLatency {
+        self.upload_histogram.merge(&other.upload_histogram);
+        self
+    }
+
+    // Get upload delay (categorized by object size interval)
+    pub fn get_upload_latency(&mut self) -> HashMap<String, u64> {
+        let mut ret = HashMap::new();
+        let avg = self.upload_histogram.get_avg_data();
+        for (i, v) in avg.iter().enumerate() {
+            let avg_duration = v.avg();
+            ret.insert(self.size_tag_to_string(i), avg_duration.as_millis() as u64);
+        }
+        ret
+    }
+    pub fn update(&mut self, size: i64, during: std::time::Duration) {
+        self.upload_histogram.add(size, during);
+    }
+
+    // Simulate the conversion from size tag to string
+    fn size_tag_to_string(&self, tag: usize) -> String {
+        match tag {
+            0 => String::from("Size < 1 KiB"),
+            1 => String::from("Size < 1 MiB"),
+            2 => String::from("Size < 10 MiB"),
+            3 => String::from("Size < 100 MiB"),
+            4 => String::from("Size < 1 GiB"),
+            _ => String::from("Size > 1 GiB"),
+        }
+    }
+}
+
+// #[derive(Debug, Clone, Default)]
+// pub struct ReplicationLastMinute {
+//     pub last_minute: LastMinuteLatency,
+// }
+
+// impl ReplicationLastMinute {
+//     pub fn merge(&mut self, other: ReplicationLastMinute) -> ReplicationLastMinute {
+//         let mut nl = ReplicationLastMinute::default();
+//         nl.last_minute = self.last_minute.merge(&mut other.last_minute);
+//         nl
+//     }
+
+//     pub fn add_size(&mut self, n: i64) {
+//         let t = SystemTime::now()
+//             .duration_since(UNIX_EPOCH)
+//             .expect("Time went backwards")
+//             .as_secs();
+//         self.last_minute.add_all(t - 1, &AccElem { total: t - 1, size: n as u64, n: 1 });
+//     }
+
+//     pub fn get_total(&self) -> AccElem {
+//         self.last_minute.get_total()
+//     }
+// }
+
+// impl fmt::Display for ReplicationLastMinute {
+//     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+//         let t = self.last_minute.get_total();
+//         write!(f, "ReplicationLastMinute sz= {}, n= {}, dur= {}", t.size, t.n, t.total)
+//     }
+// }

crates/common/src/heal_channel.rs

@@ -356,8 +356,6 @@ pub struct HealChannelRequest {
     pub recursive: Option<bool>,
     /// Whether to dry run
     pub dry_run: Option<bool>,
-    /// Whether to skip namespace locking
-    pub no_lock: Option<bool>,
     /// Timeout in seconds (optional)
     pub timeout_seconds: Option<u64>,
     /// Origin of the request for operational status and queue accounting
@@ -562,7 +560,6 @@ pub fn create_heal_request(
         update_parity: None,
         recursive: None,
         dry_run: None,
-        no_lock: None,
         timeout_seconds: None,
         source: HealRequestSource::Internal,
         disk: None,
@@ -721,7 +718,6 @@ pub async fn send_heal_disk(set_disk_id: String, priority: Option<HealChannelPri
         update_parity: None,
         recursive: None,
         dry_run: None,
-        no_lock: None,
         timeout_seconds: None,
         source: HealRequestSource::AutoHeal,
     };

crates/common/src/last_minute.rs

@@ -572,3 +572,44 @@ mod tests {
         assert_eq!(total.n, 6);
     }
 }
+
+const SIZE_LAST_ELEM_MARKER: usize = 10; // Assumed marker size is 10, modify according to actual situation
+
+#[allow(dead_code)]
+#[derive(Debug, Default)]
+pub struct LastMinuteHistogram {
+    histogram: Vec<LastMinuteLatency>,
+    size: u32,
+}
+
+impl LastMinuteHistogram {
+    pub fn merge(&mut self, other: &LastMinuteHistogram) {
+        for i in 0..self.histogram.len() {
+            self.histogram[i].merge(&other.histogram[i]);
+        }
+    }
+
+    pub fn add(&mut self, size: i64, t: Duration) {
+        let index = size_to_tag(size);
+        self.histogram[index].add(&t);
+    }
+
+    pub fn get_avg_data(&mut self) -> [AccElem; SIZE_LAST_ELEM_MARKER] {
+        let mut res = [AccElem::default(); SIZE_LAST_ELEM_MARKER];
+        for (i, elem) in self.histogram.iter_mut().enumerate() {
+            res[i] = elem.get_total();
+        }
+        res
+    }
+}
+
+fn size_to_tag(size: i64) -> usize {
+    match size {
+        _ if size < 1024 => 0,               // sizeLessThan1KiB
+        _ if size < 1024 * 1024 => 1,        // sizeLessThan1MiB
+        _ if size < 10 * 1024 * 1024 => 2,   // sizeLessThan10MiB
+        _ if size < 100 * 1024 * 1024 => 3,  // sizeLessThan100MiB
+        _ if size < 1024 * 1024 * 1024 => 4, // sizeLessThan1GiB
+        _ => 5,                              // sizeGreaterThan1GiB
+    }
+}

crates/common/src/lib.rs

@@ -12,13 +12,13 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+pub mod bucket_stats;
 // pub mod error;
 pub mod globals;
 pub mod heal_channel;
 pub mod last_minute;
 pub mod metrics;
 mod readiness;
-pub mod table_catalog;
 
 pub use globals::*;
 pub use readiness::{GlobalReadiness, SystemStage};

crates/common/src/metrics.rs

@@ -15,10 +15,9 @@
 use crate::heal_channel::HealScanMode;
 use crate::last_minute::{AccElem, LastMinuteLatency};
 use chrono::{DateTime, Utc};
-use jiff::Timestamp;
 use serde::{Deserialize, Serialize};
 use std::{
-    collections::{BTreeSet, HashMap},
+    collections::HashMap,
     fmt::Display,
     future::Future,
     pin::Pin,
@@ -670,7 +669,7 @@ impl LockedLastMinuteLatency {
 #[derive(Clone, Debug)]
 struct CurrentPathState {
     path: String,
-    updated_at: Timestamp,
+    updated_at: DateTime<Utc>,
 }
 
 struct CurrentPathTracker {
@@ -679,10 +678,10 @@ struct CurrentPathTracker {
 
 impl CurrentPathTracker {
     fn new(initial_path: String) -> Self {
-        Self::new_at(initial_path, Timestamp::now())
+        Self::new_at(initial_path, Utc::now())
     }
 
-    fn new_at(initial_path: String, updated_at: Timestamp) -> Self {
+    fn new_at(initial_path: String, updated_at: DateTime<Utc>) -> Self {
         Self {
             state: Arc::new(RwLock::new(CurrentPathState {
                 path: initial_path,
@@ -694,7 +693,7 @@ impl CurrentPathTracker {
     async fn update_path(&self, path: String) {
         let mut state = self.state.write().await;
         state.path = path;
-        state.updated_at = Timestamp::now();
+        state.updated_at = Utc::now();
     }
 
     async fn get_state(&self) -> CurrentPathState {
@@ -702,36 +701,6 @@ impl CurrentPathTracker {
     }
 }
 
-fn chrono_to_jiff_timestamp(dt: DateTime<Utc>) -> Timestamp {
-    let seconds = dt.timestamp();
-    let nanoseconds = match i32::try_from(dt.timestamp_subsec_nanos()) {
-        Ok(nanoseconds) => nanoseconds,
-        Err(_) => {
-            return if seconds < 0 { Timestamp::MIN } else { Timestamp::MAX };
-        }
-    };
-
-    match Timestamp::new(seconds, nanoseconds) {
-        Ok(timestamp) => timestamp,
-        Err(_) => {
-            if seconds < 0 {
-                Timestamp::MIN
-            } else {
-                Timestamp::MAX
-            }
-        }
-    }
-}
-
-fn timestamp_elapsed_seconds_since(now: Timestamp, earlier: Timestamp) -> u64 {
-    let duration = now.duration_since(earlier);
-    if duration.is_negative() {
-        return 0;
-    }
-
-    u64::try_from(duration.as_secs()).map_or(u64::MAX, |seconds| seconds)
-}
-
 #[derive(Clone, Copy, Debug, Default)]
 struct ScannerDiskBucketScanState {
     concurrency_limit: u64,
@@ -739,48 +708,6 @@ struct ScannerDiskBucketScanState {
     active: u64,
 }
 
-type ScannerDiskBucketScanKey = (String, String);
-
-#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
-pub struct ScannerDiskBucketScanSnapshot {
-    pub pool: String,
-    pub set: String,
-    pub concurrency_limit: u64,
-    pub queued: u64,
-    pub active: u64,
-}
-
-#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
-struct ScannerBucketDriveResultKey {
-    bucket: String,
-    drive: String,
-    result: String,
-}
-
-impl ScannerBucketDriveResultKey {
-    fn new(bucket: impl Into<String>, drive: impl Into<String>, result: impl Into<String>) -> Self {
-        Self {
-            bucket: bucket.into(),
-            drive: drive.into(),
-            result: result.into(),
-        }
-    }
-}
-
-const MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS: usize = 4096;
-
-#[derive(Debug, Default)]
-struct ScannerBucketDriveResults {
-    counts: HashMap<ScannerBucketDriveResultKey, ScannerBucketDriveResultValue>,
-    eviction_index: BTreeSet<(u64, ScannerBucketDriveResultKey)>,
-}
-
-#[derive(Clone, Copy, Debug)]
-struct ScannerBucketDriveResultValue {
-    count: u64,
-    last_seen: u64,
-}
-
 // ---------------------------------------------------------------------------
 // Metrics
 // ---------------------------------------------------------------------------
@@ -811,11 +738,7 @@ pub struct Metrics {
     scanner_set_scan_concurrency_limit: AtomicU64,
     scanner_set_scans_queued: AtomicU64,
     scanner_set_scans_active: AtomicU64,
-    scanner_disk_bucket_scan_states: Mutex<HashMap<ScannerDiskBucketScanKey, ScannerDiskBucketScanState>>,
-    scanner_bucket_drive_results: Mutex<ScannerBucketDriveResults>,
-    scanner_bucket_drive_result_clock: AtomicU64,
-    current_scan_cycle_bucket_drive_results_start: Mutex<HashMap<ScannerBucketDriveResultKey, u64>>,
-    last_scan_cycle_bucket_drive_results: Mutex<Vec<ScannerBucketDriveResultSnapshot>>,
+    scanner_disk_bucket_scan_states: Mutex<HashMap<String, ScannerDiskBucketScanState>>,
     scanner_leader_lock_state: RwLock<String>,
     scanner_leader_lock_held: AtomicBool,
     scanner_leader_lock_last_error: RwLock<String>,
@@ -915,13 +838,11 @@ const SCAN_CYCLE_RESULT_SUCCESS: u8 = 1;
 const SCAN_CYCLE_RESULT_ERROR: u8 = 2;
 const SCAN_CYCLE_RESULT_PARTIAL: u8 = 3;
 const SCAN_CYCLE_RESULT_SUPERSEDED: u8 = 4;
-const SCAN_CYCLE_RESULT_DEFERRED: u8 = 5;
 const SCAN_CYCLE_RESULT_UNKNOWN_LABEL: &str = "unknown";
 const SCAN_CYCLE_RESULT_SUCCESS_LABEL: &str = "success";
 const SCAN_CYCLE_RESULT_ERROR_LABEL: &str = "error";
 const SCAN_CYCLE_RESULT_PARTIAL_LABEL: &str = "partial";
 const SCAN_CYCLE_RESULT_SUPERSEDED_LABEL: &str = "superseded";
-const SCAN_CYCLE_RESULT_DEFERRED_LABEL: &str = "deferred";
 
 #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
 pub enum ScanCyclePartialReason {
@@ -1037,14 +958,6 @@ pub struct ScannerSourceWorkSnapshot {
     pub missed: u64,
 }
 
-#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
-pub struct ScannerBucketDriveResultSnapshot {
-    pub bucket: String,
-    pub drive: String,
-    pub result: String,
-    pub count: u64,
-}
-
 #[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
 pub struct ScannerReplicationRepairSnapshot {
     pub source: String,
@@ -1199,12 +1112,12 @@ pub struct ScannerLastMinute {
 
 #[derive(Clone, Debug, Default, Serialize, Deserialize)]
 pub struct ScannerMetricsReport {
-    pub collected_at: Timestamp,
+    pub collected_at: DateTime<Utc>,
     pub current_cycle: u64,
     #[serde(default)]
     pub current_cycle_active: bool,
-    pub current_started: Timestamp,
-    pub cycles_completed_at: Vec<Timestamp>,
+    pub current_started: DateTime<Utc>,
+    pub cycles_completed_at: Vec<DateTime<Utc>>,
     pub ongoing_buckets: usize,
     #[serde(default)]
     pub active_scan_paths: usize,
@@ -1377,18 +1290,6 @@ pub struct ScannerMetricsReport {
     pub partial_cycles: u64,
 }
 
-#[derive(Clone, Debug, Default, Serialize, Deserialize)]
-pub struct ScannerRuntimeDetailsReport {
-    #[serde(default)]
-    pub disk_bucket_scan_states: Vec<ScannerDiskBucketScanSnapshot>,
-    #[serde(default)]
-    pub bucket_drive_results: Vec<ScannerBucketDriveResultSnapshot>,
-    #[serde(default)]
-    pub current_cycle_bucket_drive_results: Vec<ScannerBucketDriveResultSnapshot>,
-    #[serde(default)]
-    pub last_cycle_bucket_drive_results: Vec<ScannerBucketDriveResultSnapshot>,
-}
-
 impl CurrentCycle {
     pub fn unmarshal(&mut self, buf: &[u8]) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         *self = rmp_serde::from_slice(buf)?;
@@ -1426,7 +1327,6 @@ fn scan_cycle_result_label(result: u8) -> &'static str {
         SCAN_CYCLE_RESULT_ERROR => SCAN_CYCLE_RESULT_ERROR_LABEL,
         SCAN_CYCLE_RESULT_PARTIAL => SCAN_CYCLE_RESULT_PARTIAL_LABEL,
         SCAN_CYCLE_RESULT_SUPERSEDED => SCAN_CYCLE_RESULT_SUPERSEDED_LABEL,
-        SCAN_CYCLE_RESULT_DEFERRED => SCAN_CYCLE_RESULT_DEFERRED_LABEL,
         _ => SCAN_CYCLE_RESULT_UNKNOWN_LABEL,
     }
 }
@@ -1755,14 +1655,8 @@ pub fn emit_scan_cycle_superseded(duration: Duration) {
     metrics::counter!(OTEL_SCANNER_CYCLES, "result" => SCAN_CYCLE_RESULT_SUPERSEDED_LABEL).increment(1);
 }
 
-pub fn emit_scan_cycle_deferred(duration: Duration) {
-    global_metrics().record_scan_cycle_deferred(duration);
-    metrics::counter!(OTEL_SCANNER_CYCLES, "result" => SCAN_CYCLE_RESULT_DEFERRED_LABEL).increment(1);
-}
-
 pub fn emit_scan_bucket_drive_complete(success: bool, bucket: &str, disk: &str, duration: Duration) {
     let result = if success { "success" } else { "error" };
-    global_metrics().record_scanner_bucket_drive_result(bucket, disk, result);
     metrics::counter!(
         OTEL_SCANNER_BUCKETS_SCANNED,
         "result" => result,
@@ -1779,7 +1673,6 @@ pub fn emit_scan_bucket_drive_complete(success: bool, bucket: &str, disk: &str,
 }
 
 pub fn emit_scan_bucket_drive_partial(bucket: &str, disk: &str, duration: Duration) {
-    global_metrics().record_scanner_bucket_drive_result(bucket, disk, SCAN_CYCLE_RESULT_PARTIAL_LABEL);
     metrics::counter!(
         OTEL_SCANNER_BUCKETS_SCANNED,
         "result" => SCAN_CYCLE_RESULT_PARTIAL_LABEL,
@@ -1830,10 +1723,6 @@ impl Metrics {
             scanner_set_scans_queued: AtomicU64::new(0),
             scanner_set_scans_active: AtomicU64::new(0),
             scanner_disk_bucket_scan_states: Mutex::new(HashMap::new()),
-            scanner_bucket_drive_results: Mutex::new(ScannerBucketDriveResults::default()),
-            scanner_bucket_drive_result_clock: AtomicU64::new(0),
-            current_scan_cycle_bucket_drive_results_start: Mutex::new(HashMap::new()),
-            last_scan_cycle_bucket_drive_results: Mutex::new(Vec::new()),
             scanner_leader_lock_state: RwLock::new("unknown".to_string()),
             scanner_leader_lock_held: AtomicBool::new(false),
             scanner_leader_lock_last_error: RwLock::new(String::new()),
@@ -2404,7 +2293,7 @@ impl Metrics {
         queued: Option<usize>,
         active: Option<usize>,
     ) {
-        let key = (pool.to_string(), set.to_string());
+        let key = format!("{pool}/{set}");
         let mut states = self
             .scanner_disk_bucket_scan_states
             .lock()
@@ -2421,41 +2310,6 @@ impl Metrics {
         }
     }
 
-    pub fn record_scanner_bucket_drive_result(&self, bucket: &str, drive: &str, result: &str) {
-        if bucket.is_empty() || drive.is_empty() || result.is_empty() {
-            return;
-        }
-        let key = ScannerBucketDriveResultKey::new(bucket, drive, result);
-        let mut results = self
-            .scanner_bucket_drive_results
-            .lock()
-            .unwrap_or_else(|poisoned| poisoned.into_inner());
-        let last_seen = self.scanner_bucket_drive_result_clock.fetch_add(1, Ordering::Relaxed);
-        if let Some(previous_last_seen) = results.counts.get_mut(&key).map(|value| {
-            let previous_last_seen = value.last_seen;
-            value.count = value.count.saturating_add(1);
-            value.last_seen = last_seen;
-            previous_last_seen
-        }) {
-            results.eviction_index.remove(&(previous_last_seen, key.clone()));
-            results.eviction_index.insert((last_seen, key));
-            return;
-        }
-
-        if results.counts.len() >= MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS
-            && let Some((_, stale_key)) = results.eviction_index.pop_first()
-        {
-            results.counts.remove(&stale_key);
-        }
-
-        if results.counts.len() < MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS {
-            results
-                .counts
-                .insert(key.clone(), ScannerBucketDriveResultValue { count: 1, last_seen });
-            results.eviction_index.insert((last_seen, key));
-        }
-    }
-
     // -----------------------------------------------------------------------
     // Read-side helpers
     // -----------------------------------------------------------------------
@@ -2557,17 +2411,6 @@ impl Metrics {
             .store(duration_millis_saturated(duration), Ordering::Relaxed);
     }
 
-    pub fn record_scan_cycle_deferred(&self, duration: Duration) {
-        self.record_scanner_cycle_end_time();
-        self.last_scan_cycle_result
-            .store(SCAN_CYCLE_RESULT_DEFERRED, Ordering::Relaxed);
-        self.last_scan_cycle_partial_reason
-            .store(ScanCyclePartialReason::Unknown as u8, Ordering::Relaxed);
-        self.last_scan_cycle_partial_source.store(0, Ordering::Relaxed);
-        self.last_scan_cycle_duration_millis
-            .store(duration_millis_saturated(duration), Ordering::Relaxed);
-    }
-
     pub fn record_scan_cycle_partial(&self, duration: Duration, reason: ScanCyclePartialReason) {
         self.record_scan_cycle_partial_with_source(duration, reason, None);
     }
@@ -2638,11 +2481,6 @@ impl Metrics {
             &self.current_scan_cycle_replication_repair_work_start,
             &replication_repair_snapshot,
         );
-        let bucket_drive_results = self.scanner_bucket_drive_result_counts();
-        match self.current_scan_cycle_bucket_drive_results_start.lock() {
-            Ok(mut start) => *start = bucket_drive_results,
-            Err(poisoned) => *poisoned.into_inner() = bucket_drive_results,
-        }
         self.current_scan_cycle_work_active.store(true, Ordering::Release);
         snapshot
     }
@@ -2655,11 +2493,6 @@ impl Metrics {
         self.record_scan_cycle_work(work);
         self.record_scan_cycle_source_work(&source_work);
         self.record_scan_cycle_replication_repair_work(&replication_repair_work);
-        let bucket_drive_results = self.current_cycle_bucket_drive_result_snapshots();
-        match self.last_scan_cycle_bucket_drive_results.lock() {
-            Ok(mut last) => *last = bucket_drive_results,
-            Err(poisoned) => *poisoned.into_inner() = bucket_drive_results,
-        }
         self.current_scan_cycle_work_active.store(false, Ordering::Release);
     }
 
@@ -2743,105 +2576,6 @@ impl Metrics {
         }
     }
 
-    fn scanner_bucket_drive_result_counts(&self) -> HashMap<ScannerBucketDriveResultKey, u64> {
-        self.scanner_bucket_drive_results
-            .lock()
-            .unwrap_or_else(|poisoned| poisoned.into_inner())
-            .counts
-            .iter()
-            .map(|(key, value)| (key.clone(), value.count))
-            .collect()
-    }
-
-    fn scanner_bucket_drive_result_snapshots(
-        counts: impl IntoIterator<Item = (ScannerBucketDriveResultKey, u64)>,
-    ) -> Vec<ScannerBucketDriveResultSnapshot> {
-        let mut snapshots = counts
-            .into_iter()
-            .filter(|(_, count)| *count > 0)
-            .map(|(key, count)| ScannerBucketDriveResultSnapshot {
-                bucket: key.bucket,
-                drive: key.drive,
-                result: key.result,
-                count,
-            })
-            .collect::<Vec<_>>();
-        snapshots.sort_by(|left, right| {
-            left.bucket
-                .cmp(&right.bucket)
-                .then_with(|| left.drive.cmp(&right.drive))
-                .then_with(|| left.result.cmp(&right.result))
-        });
-        snapshots
-    }
-
-    fn scanner_bucket_drive_result_counter_snapshots(&self) -> Vec<ScannerBucketDriveResultSnapshot> {
-        Self::scanner_bucket_drive_result_snapshots(self.scanner_bucket_drive_result_counts())
-    }
-
-    fn current_cycle_bucket_drive_result_snapshots(&self) -> Vec<ScannerBucketDriveResultSnapshot> {
-        let current = self.scanner_bucket_drive_result_counts();
-        let start = self
-            .current_scan_cycle_bucket_drive_results_start
-            .lock()
-            .unwrap_or_else(|poisoned| poisoned.into_inner())
-            .clone();
-        Self::scanner_bucket_drive_result_snapshots(current.into_iter().filter_map(|(key, count)| {
-            let delta = count.saturating_sub(start.get(&key).copied().unwrap_or_default());
-            (delta > 0).then_some((key, delta))
-        }))
-    }
-
-    pub fn scanner_runtime_details_report(&self) -> ScannerRuntimeDetailsReport {
-        self.scanner_runtime_details_report_for_active(self.current_scan_cycle_work_active.load(Ordering::Acquire))
-    }
-
-    fn scanner_runtime_details_report_for_active(&self, current_cycle_active: bool) -> ScannerRuntimeDetailsReport {
-        let current_cycle_bucket_drive_results = if current_cycle_active {
-            self.current_cycle_bucket_drive_result_snapshots()
-        } else {
-            Vec::new()
-        };
-        ScannerRuntimeDetailsReport {
-            disk_bucket_scan_states: self.scanner_disk_bucket_scan_state_snapshots(),
-            bucket_drive_results: self.scanner_bucket_drive_result_counter_snapshots(),
-            current_cycle_bucket_drive_results,
-            last_cycle_bucket_drive_results: self
-                .last_scan_cycle_bucket_drive_results
-                .lock()
-                .unwrap_or_else(|poisoned| poisoned.into_inner())
-                .clone(),
-        }
-    }
-
-    fn scanner_disk_bucket_scan_state_snapshots(&self) -> Vec<ScannerDiskBucketScanSnapshot> {
-        let mut disk_bucket_scan_states = match self.scanner_disk_bucket_scan_states.lock() {
-            Ok(states) => states
-                .iter()
-                .map(|((pool, set), state)| ScannerDiskBucketScanSnapshot {
-                    pool: pool.clone(),
-                    set: set.clone(),
-                    concurrency_limit: state.concurrency_limit,
-                    queued: state.queued,
-                    active: state.active,
-                })
-                .collect::<Vec<_>>(),
-            Err(poisoned) => poisoned
-                .into_inner()
-                .iter()
-                .map(|((pool, set), state)| ScannerDiskBucketScanSnapshot {
-                    pool: pool.clone(),
-                    set: set.clone(),
-                    concurrency_limit: state.concurrency_limit,
-                    queued: state.queued,
-                    active: state.active,
-                })
-                .collect::<Vec<_>>(),
-        };
-        disk_bucket_scan_states.sort_by(|left, right| left.pool.cmp(&right.pool).then_with(|| left.set.cmp(&right.set)));
-        disk_bucket_scan_states
-    }
-
     fn scanner_source_work_values(&self) -> Vec<ScannerSourceWorkValues> {
         ScannerWorkSource::all()
             .iter()
@@ -3027,26 +2761,20 @@ impl Metrics {
 
     /// Build a full metrics report snapshot.
     pub async fn report(&self) -> ScannerMetricsReport {
-        self.report_with_runtime_details().await.0
-    }
-
-    pub async fn report_with_runtime_details(&self) -> (ScannerMetricsReport, ScannerRuntimeDetailsReport) {
         let mut m = ScannerMetricsReport::default();
-        let runtime_details;
 
         let has_cycle = {
             let cycle = self.cycle_info.read().await;
             let has_cycle = if let Some(cycle) = cycle.as_ref() {
                 m.current_cycle = cycle.current;
-                m.cycles_completed_at = cycle.cycle_completed.iter().copied().map(chrono_to_jiff_timestamp).collect();
-                m.current_started = chrono_to_jiff_timestamp(cycle.started);
+                m.cycles_completed_at = cycle.cycle_completed.clone();
+                m.current_started = cycle.started;
                 true
             } else {
                 false
             };
             m.current_cycle_active = self.current_scan_cycle_work_active.load(Ordering::Acquire);
             if m.current_cycle_active {
-                // Keep cycle_info before cycle-baseline locks so active scrapes cannot mix two cycle identities.
                 let current_work = self.scan_cycle_work_since(self.current_scan_cycle_work_start());
                 let current_source_work = self.scanner_source_work_since(&self.current_scan_cycle_source_work_start_values());
                 let current_replication_repair_work =
@@ -3069,20 +2797,19 @@ impl Metrics {
                 m.current_cycle_replication_repair =
                     self.scanner_replication_repair_work_snapshots(&current_replication_repair_work);
             }
-            runtime_details = self.scanner_runtime_details_report_for_active(m.current_cycle_active);
             has_cycle
         };
 
         if !has_cycle && let Some(init_time) = crate::get_global_init_time().await {
-            m.current_started = chrono_to_jiff_timestamp(init_time);
+            m.current_started = init_time;
         }
 
-        m.collected_at = Timestamp::now();
+        m.collected_at = Utc::now();
         let current_path_snapshots = self.current_path_snapshots().await;
         m.active_scan_paths = current_path_snapshots.len();
         m.oldest_active_path_age_seconds = current_path_snapshots
             .iter()
-            .map(|(_, state)| timestamp_elapsed_seconds_since(m.collected_at, state.updated_at))
+            .map(|(_, state)| m.collected_at.signed_duration_since(state.updated_at).num_seconds().max(0) as u64)
             .max()
             .unwrap_or_default();
         m.active_paths = current_path_snapshots
@@ -3099,11 +2826,15 @@ impl Metrics {
         m.current_set_scan_concurrency_limit = self.scanner_set_scan_concurrency_limit.load(Ordering::Relaxed);
         m.current_set_scans_queued = self.scanner_set_scans_queued.load(Ordering::Relaxed);
         m.current_set_scans_active = self.scanner_set_scans_active.load(Ordering::Relaxed);
-        let disk_bucket_scan_states = self.scanner_disk_bucket_scan_state_snapshots();
         let (disk_scan_concurrency_limit, disk_bucket_scans_queued, disk_bucket_scans_active) =
-            disk_bucket_scan_states.iter().fold((0, 0, 0), |acc, state| {
-                (acc.0 + state.concurrency_limit, acc.1 + state.queued, acc.2 + state.active)
-            });
+            match self.scanner_disk_bucket_scan_states.lock() {
+                Ok(states) => states.values().fold((0, 0, 0), |acc, state| {
+                    (acc.0 + state.concurrency_limit, acc.1 + state.queued, acc.2 + state.active)
+                }),
+                Err(poisoned) => poisoned.into_inner().values().fold((0, 0, 0), |acc, state| {
+                    (acc.0 + state.concurrency_limit, acc.1 + state.queued, acc.2 + state.active)
+                }),
+            };
         m.current_disk_scan_concurrency_limit = disk_scan_concurrency_limit;
         m.current_disk_bucket_scans_queued = disk_bucket_scans_queued;
         m.current_disk_bucket_scans_active = disk_bucket_scans_active;
@@ -3272,7 +3003,7 @@ impl Metrics {
         m.pacing_pressure = scanner_pacing_pressure(&m);
         m.maintenance_control = scanner_maintenance_control(&m);
 
-        (m, runtime_details)
+        m
     }
 }
 
@@ -3358,22 +3089,6 @@ impl Drop for CloseDiskGuard {
 mod tests {
     use super::*;
 
-    #[test]
-    fn scanner_metrics_report_timestamps_serialize_as_rfc3339_utc() {
-        let report = ScannerMetricsReport {
-            collected_at: Timestamp::constant(1_700_000_000, 123_456_000),
-            current_started: Timestamp::constant(1_699_999_940, 0),
-            cycles_completed_at: vec![Timestamp::constant(1_700_000_060, 987_654_000)],
-            ..Default::default()
-        };
-
-        let value = serde_json::to_value(&report).expect("scanner metrics report should serialize");
-
-        assert_eq!(value["collected_at"].as_str(), Some("2023-11-14T22:13:20.123456Z"));
-        assert_eq!(value["current_started"].as_str(), Some("2023-11-14T22:12:20Z"));
-        assert_eq!(value["cycles_completed_at"][0].as_str(), Some("2023-11-14T22:14:20.987654Z"));
-    }
-
     #[tokio::test]
     async fn close_disk_guard_runs_cleanup_when_an_early_return_drops_it() {
         let (closed_tx, closed_rx) = tokio::sync::oneshot::channel();
@@ -3432,7 +3147,7 @@ mod tests {
     #[tokio::test]
     async fn report_counts_active_scan_paths() {
         let metrics = Metrics::new();
-        let updated_at = Timestamp::now() - jiff::SignedDuration::from_secs(12);
+        let updated_at = Utc::now() - chrono::Duration::seconds(12);
         metrics.current_paths.write().await.insert(
             "disk-a".to_string(),
             Arc::new(CurrentPathTracker::new_at("bucket-a".to_string(), updated_at)),
@@ -3454,7 +3169,7 @@ mod tests {
         let metrics = Metrics::new();
         let tracker = Arc::new(CurrentPathTracker::new_at(
             "bucket-a".to_string(),
-            Timestamp::now() - jiff::SignedDuration::from_secs(60 * 60),
+            Utc::now() - chrono::Duration::hours(1),
         ));
         metrics
             .current_paths
@@ -4227,7 +3942,7 @@ mod tests {
         let report = metrics.report().await;
         *crate::globals::GLOBAL_INIT_TIME.write().await = previous_init_time;
 
-        assert_eq!(report.current_started, chrono_to_jiff_timestamp(cycle_started));
+        assert_eq!(report.current_started, cycle_started);
     }
 
     #[tokio::test]
@@ -4283,21 +3998,6 @@ mod tests {
         assert_eq!(report.partial_cycles, 0);
     }
 
-    #[tokio::test]
-    async fn report_tracks_deferred_cycle_without_failed_increment() {
-        let metrics = Metrics::new();
-        metrics.record_scan_cycle_deferred(Duration::from_millis(250));
-
-        let report = metrics.report().await;
-
-        assert_eq!(report.last_cycle_result, SCAN_CYCLE_RESULT_DEFERRED_LABEL);
-        assert_eq!(report.last_cycle_result_code, u64::from(SCAN_CYCLE_RESULT_DEFERRED));
-        assert_eq!(report.last_cycle_duration_seconds, 0.25);
-        assert_eq!(report.failed_cycles, 0);
-        assert_eq!(report.superseded_cycles, 0);
-        assert_eq!(report.partial_cycles, 0);
-    }
-
     #[tokio::test]
     async fn report_tracks_successful_scan_cycle_without_failed_increment() {
         let metrics = Metrics::new();
@@ -4400,137 +4100,6 @@ mod tests {
         assert_eq!(report.life_time_ops.get("scan_bucket_drive_failure"), Some(&1));
     }
 
-    #[tokio::test]
-    async fn report_includes_structured_bucket_drive_results() {
-        let metrics = Metrics::new();
-        metrics.record_scanner_bucket_drive_result("photos", "/data1", "success");
-
-        let cycle_start = metrics.start_scan_cycle_work();
-        metrics.record_scanner_bucket_drive_result("photos", "/data1", "partial");
-
-        let active_report = metrics.scanner_runtime_details_report();
-        assert_eq!(
-            active_report.current_cycle_bucket_drive_results,
-            vec![ScannerBucketDriveResultSnapshot {
-                bucket: "photos".to_string(),
-                drive: "/data1".to_string(),
-                result: "partial".to_string(),
-                count: 1,
-            }]
-        );
-
-        metrics.finish_scan_cycle_work(cycle_start);
-
-        let report = metrics.scanner_runtime_details_report();
-
-        assert_eq!(
-            report.bucket_drive_results,
-            vec![
-                ScannerBucketDriveResultSnapshot {
-                    bucket: "photos".to_string(),
-                    drive: "/data1".to_string(),
-                    result: "partial".to_string(),
-                    count: 1,
-                },
-                ScannerBucketDriveResultSnapshot {
-                    bucket: "photos".to_string(),
-                    drive: "/data1".to_string(),
-                    result: "success".to_string(),
-                    count: 1,
-                },
-            ]
-        );
-        assert!(report.current_cycle_bucket_drive_results.is_empty());
-        assert_eq!(
-            report.last_cycle_bucket_drive_results,
-            vec![ScannerBucketDriveResultSnapshot {
-                bucket: "photos".to_string(),
-                drive: "/data1".to_string(),
-                result: "partial".to_string(),
-                count: 1,
-            }]
-        );
-    }
-
-    #[tokio::test]
-    async fn scanner_bucket_drive_results_are_bounded() {
-        let metrics = Metrics::new();
-        for index in 0..MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS {
-            metrics.record_scanner_bucket_drive_result(&format!("bucket-{index}"), "/data1", "success");
-        }
-        metrics.record_scanner_bucket_drive_result("overflow", "/data1", "success");
-
-        let report = metrics.scanner_runtime_details_report();
-
-        assert_eq!(report.bucket_drive_results.len(), MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS);
-        assert!(
-            report
-                .bucket_drive_results
-                .iter()
-                .any(|snapshot| snapshot.bucket == "overflow")
-        );
-        assert!(
-            report
-                .bucket_drive_results
-                .iter()
-                .all(|snapshot| snapshot.bucket != "bucket-0")
-        );
-    }
-
-    #[tokio::test]
-    async fn scanner_bucket_drive_result_eviction_keeps_recent_keys() {
-        let metrics = Metrics::new();
-        for index in 0..MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS {
-            metrics.record_scanner_bucket_drive_result(&format!("bucket-{index}"), "/data1", "success");
-        }
-        metrics.record_scanner_bucket_drive_result("bucket-0", "/data1", "success");
-        metrics.record_scanner_bucket_drive_result("overflow", "/data1", "success");
-
-        let report = metrics.scanner_runtime_details_report();
-
-        assert_eq!(report.bucket_drive_results.len(), MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS);
-        assert!(
-            report
-                .bucket_drive_results
-                .iter()
-                .any(|snapshot| snapshot.bucket == "bucket-0" && snapshot.count == 2)
-        );
-        assert!(
-            report
-                .bucket_drive_results
-                .iter()
-                .all(|snapshot| snapshot.bucket != "bucket-1")
-        );
-    }
-
-    #[tokio::test]
-    async fn scanner_bucket_drive_result_eviction_survives_full_refresh() {
-        let metrics = Metrics::new();
-        for index in 0..MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS {
-            metrics.record_scanner_bucket_drive_result(&format!("bucket-{index}"), "/data1", "success");
-        }
-        for index in 0..MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS {
-            metrics.record_scanner_bucket_drive_result(&format!("bucket-{index}"), "/data1", "success");
-        }
-        metrics.record_scanner_bucket_drive_result("overflow", "/data1", "success");
-
-        let report = metrics.scanner_runtime_details_report();
-
-        assert_eq!(report.bucket_drive_results.len(), MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS);
-        assert!(
-            report
-                .bucket_drive_results
-                .iter()
-                .any(|snapshot| snapshot.bucket == "overflow")
-        );
-        assert!(
-            report
-                .bucket_drive_results
-                .iter()
-                .all(|snapshot| snapshot.bucket != "bucket-0")
-        );
-    }
-
     #[tokio::test]
     async fn report_includes_usage_freshness_status() {
         let metrics = Metrics::new();
@@ -4665,7 +4234,7 @@ mod tests {
         let active = metrics.report().await;
         assert!(active.current_cycle_active);
         assert_eq!(active.current_cycle, 12);
-        assert_eq!(active.current_started, chrono_to_jiff_timestamp(cycle_started));
+        assert_eq!(active.current_started, cycle_started);
 
         let idle_cycle = CurrentCycle {
             current: 0,
@@ -4696,10 +4265,9 @@ mod tests {
         };
         let cycle_ten_start = metrics.start_scan_cycle_work_with_cycle(cycle_ten.clone()).await;
         metrics.operations[Metric::ScanObject as usize].store(1, Ordering::Relaxed);
-        metrics.record_scanner_bucket_drive_result("cycle-ten", "/data1", "partial");
 
         let paths = metrics.current_paths.write().await;
-        let mut report = Box::pin(metrics.report_with_runtime_details());
+        let mut report = Box::pin(metrics.report());
         let waker = std::task::Waker::noop();
         let mut context = std::task::Context::from_waker(waker);
         assert!(report.as_mut().poll(&mut context).is_pending());
@@ -4716,22 +4284,12 @@ mod tests {
             })
             .await;
         metrics.operations[Metric::ScanObject as usize].store(101, Ordering::Relaxed);
-        metrics.record_scanner_bucket_drive_result("cycle-eleven", "/data1", "partial");
 
         drop(paths);
-        let (snapshot, runtime_details) = report.await;
+        let snapshot = report.await;
 
         assert_eq!(snapshot.current_cycle, 10);
         assert_eq!(snapshot.current_cycle_objects_scanned, 1);
-        assert_eq!(
-            runtime_details.current_cycle_bucket_drive_results,
-            vec![ScannerBucketDriveResultSnapshot {
-                bucket: "cycle-ten".to_string(),
-                drive: "/data1".to_string(),
-                result: "partial".to_string(),
-                count: 1,
-            }]
-        );
 
         metrics
             .finish_scan_cycle_work_with_cycle(cycle_eleven_start, CurrentCycle::default())

crates/common/src/table_catalog.rs

@@ -1,17 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-/// Cross-crate lock identity used to fence table-bucket publication against
-/// object mutations that bypass the S3 request authorization layer.
-pub const TABLE_BUCKET_PUBLICATION_LOCK_PATH: &str = ".rustfs-table/warehouses/default/publication.lock";

crates/config/README.md

@@ -97,14 +97,6 @@ Current guidance:
   - enables minimal payload mode for GET health responses (`status`, `ready` only).
 - `RUSTFS_HEALTH_READINESS_CACHE_TTL_MS`
   - TTL for readiness cache evaluation.
-- `RUSTFS_HEALTH_OBJECT_PROGRESS_ENABLE`
-  - withdraws readiness when bounded object read/write stages stop completing while requests remain active.
-  - default is `true`.
-- `RUSTFS_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS`
-  - maximum time without completion in a bounded object stage before readiness is withdrawn.
-  - default is `30000`; `0` uses the default.
-  - the effective value is at least 5 seconds longer than `RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT`.
-  - this readiness SLO is independent of disk read/write failure deadlines and may withdraw traffic before those deadlines expire.
 - `RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE`
   - enables busy protection behavior for health probes.
   - default is `false`.

crates/config/src/constants/app.rs

@@ -353,11 +353,6 @@ pub const DEFAULT_OBS_TRACES_EXPORT_ENABLED: bool = true;
 /// Environment variable: RUSTFS_OBS_METRICS_EXPORT_ENABLED
 pub const DEFAULT_OBS_METRICS_EXPORT_ENABLED: bool = true;
 
-/// Default detailed PUT stage metrics enabled
-/// Default value: false
-/// Environment variable: RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED
-pub const DEFAULT_OBS_PUT_STAGE_METRICS_ENABLED: bool = false;
-
 /// Default logs export enabled
 /// It is used to enable or disable exporting logs
 /// Default value: true

crates/config/src/constants/health.rs

@@ -22,19 +22,6 @@ pub const DEFAULT_HEALTH_ENDPOINT_ENABLE: bool = true;
 pub const ENV_HEALTH_READINESS_CACHE_TTL_MS: &str = "RUSTFS_HEALTH_READINESS_CACHE_TTL_MS";
 pub const DEFAULT_HEALTH_READINESS_CACHE_TTL_MS: u64 = 1000;
 
-/// Enable readiness withdrawal when bounded object read/write stages stop
-/// completing while requests remain active.
-pub const ENV_HEALTH_OBJECT_PROGRESS_ENABLE: &str = "RUSTFS_HEALTH_OBJECT_PROGRESS_ENABLE";
-pub const DEFAULT_HEALTH_OBJECT_PROGRESS_ENABLE: bool = true;
-
-/// Requested time without completion in a bounded object stage before local
-/// readiness is withdrawn (milliseconds). A value of `0` uses the default;
-/// runtime adds a safety floor based on the object-lock acquisition timeout.
-pub const ENV_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS: &str = "RUSTFS_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS";
-pub const DEFAULT_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS: u64 = 30_000;
-/// Additional time beyond the configured object-lock acquisition deadline.
-pub const HEALTH_OBJECT_PROGRESS_LOCK_MARGIN_MS: u64 = 5_000;
-
 /// Timeout for cluster health readiness collectors (milliseconds).
 /// This bounds expensive storage and lock quorum checks used by cluster probes.
 pub const ENV_HEALTH_CLUSTER_TIMEOUT_MS: &str = "RUSTFS_HEALTH_CLUSTER_TIMEOUT_MS";

crates/config/src/constants/internode.rs

@@ -177,9 +177,10 @@ const _: () = assert!(!DEFAULT_INTERNODE_RPC_REPLAY_SCOPE_STRICT);
 ///
 /// The cache retains each nonce for the ~10-minute signature freshness envelope. Once peers use
 /// replay-scoped v3 authentication, every authenticated RPC consumes one entry, so the steady
-/// state holds roughly `authenticated RPC RPS x 601s` entries. This default is the minimum floor:
-/// explicit operator values and resource-aware auto sizing both clamp upward to at least this
-/// value. Overflow fails closed — legitimate signed traffic is the only thing that can fill the
+/// state holds roughly `authenticated RPC RPS x 601s` entries. The default sustains about 1,700
+/// authenticated RPCs per second (about 120 MiB worst case, allocated only under sustained load);
+/// operators must size it for the node's aggregate peak RPC rate before enabling strict replay
+/// scope. Overflow fails closed — legitimate signed traffic is the only thing that can fill the
 /// cache (replays are rejected before insertion, and an attacker cannot mint valid nonces without
 /// the shared secret) — and increments
 /// `rustfs_system_network_internode_replay_cache_overflow_total`, so a sustained non-zero overflow

crates/config/src/constants/object.rs

@@ -137,21 +137,6 @@ pub const DEFAULT_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED: bool = false;
 const _: () = assert!(!DEFAULT_TIER_REMOTE_VERSION_STATE_WRITE);
 const _: () = assert!(!DEFAULT_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED);
 
-/// Request preserving legacy per-part checksum metadata during data movement.
-///
-/// This remains ineffective until
-/// [`ENV_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED`] is also enabled.
-pub const ENV_DATA_MOVEMENT_PART_CHECKSUMS_WRITE: &str = "RUSTFS_DATA_MOVEMENT_PART_CHECKSUMS_WRITE";
-pub const DEFAULT_DATA_MOVEMENT_PART_CHECKSUMS_WRITE: bool = false;
-
-/// Operator-attested confirmation that every serving node understands the
-/// data-movement per-part checksum sidecar.
-pub const ENV_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED: &str = "RUSTFS_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED";
-pub const DEFAULT_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED: bool = false;
-
-const _: () = assert!(!DEFAULT_DATA_MOVEMENT_PART_CHECKSUMS_WRITE);
-const _: () = assert!(!DEFAULT_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED);
-
 // =============================================================================
 // Concurrent Request Fix - Timeout and Backpressure Configuration
 // =============================================================================
@@ -664,13 +649,4 @@ mod remote_version_state_tests {
             "RUSTFS_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED"
         );
     }
-
-    #[test]
-    fn data_movement_part_checksum_gate_uses_stable_environment_names() {
-        assert_eq!(super::ENV_DATA_MOVEMENT_PART_CHECKSUMS_WRITE, "RUSTFS_DATA_MOVEMENT_PART_CHECKSUMS_WRITE");
-        assert_eq!(
-            super::ENV_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED,
-            "RUSTFS_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED"
-        );
-    }
 }

crates/config/src/constants/runtime.rs

@@ -81,9 +81,6 @@ pub const ENV_TEST_IAM_FAIL_INIT_ATTEMPTS: &str = "RUSTFS_TEST_IAM_FAIL_INIT_ATT
 pub const ENV_TEST_IAM_RETRY_INTERVAL_MS: &str = "RUSTFS_TEST_IAM_RETRY_INTERVAL_MS";
 /// Runtime env var controlling the transition worker count.
 pub const ENV_TRANSITION_WORKERS: &str = "RUSTFS_MAX_TRANSITION_WORKERS";
-/// Runtime env var controlling the ILM expiry worker count. A set, parsable,
-/// non-zero value wins; anything else falls back to `min(cpus, 16)`.
-pub const ENV_MAX_EXPIRY_WORKERS: &str = "RUSTFS_MAX_EXPIRY_WORKERS";
 /// Runtime env var controlling the absolute maximum transition workers.
 pub const ENV_TRANSITION_WORKERS_ABSOLUTE_MAX: &str = "RUSTFS_ABSOLUTE_MAX_WORKERS";
 /// Runtime env var controlling the transition queue capacity.

crates/config/src/constants/tls.rs

@@ -36,11 +36,6 @@ pub const ENV_TRUST_SYSTEM_CA: &str = "RUSTFS_TRUST_SYSTEM_CA";
 /// To change this behavior, set the environment variable RUSTFS_TRUST_SYSTEM_CA=1
 pub const DEFAULT_TRUST_SYSTEM_CA: bool = false;
 
-/// Environment variable for an extra outbound root CA certificate bundle.
-/// Use this to trust an internal CA for outbound HTTPS clients without replacing
-/// the default operating-system/web PKI roots via SSL_CERT_FILE.
-pub const ENV_RUSTFS_EXTRA_CA_CERT: &str = "RUSTFS_EXTRA_CA_CERT";
-
 /// Environment variable to trust leaf certificates as CA
 /// When set to "1", RustFS will treat leaf certificates as CA certificates for trust validation.
 /// By default, this is disabled.

crates/config/src/observability/mod.rs

@@ -44,10 +44,6 @@ pub const ENV_OBS_METRICS_EXPORT_ENABLED: &str = "RUSTFS_OBS_METRICS_EXPORT_ENAB
 pub const ENV_OBS_LOGS_EXPORT_ENABLED: &str = "RUSTFS_OBS_LOGS_EXPORT_ENABLED";
 pub const ENV_OBS_PROFILING_EXPORT_ENABLED: &str = "RUSTFS_OBS_PROFILING_EXPORT_ENABLED";
 
-/// Enables detailed per-stage PUT metrics. Disabled by default because each
-/// PUT records multiple timers and histograms when attribution is active.
-pub const ENV_OBS_PUT_STAGE_METRICS_ENABLED: &str = "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED";
-
 pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL";
 pub const ENV_OBS_LOG_STDOUT_ENABLED: &str = "RUSTFS_OBS_LOG_STDOUT_ENABLED";
 pub const ENV_OBS_LOG_DIRECTORY: &str = "RUSTFS_OBS_LOG_DIRECTORY";
@@ -145,7 +141,6 @@ mod tests {
         assert_eq!(ENV_OBS_METRICS_EXPORT_ENABLED, "RUSTFS_OBS_METRICS_EXPORT_ENABLED");
         assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED");
         assert_eq!(ENV_OBS_PROFILING_EXPORT_ENABLED, "RUSTFS_OBS_PROFILING_EXPORT_ENABLED");
-        assert_eq!(ENV_OBS_PUT_STAGE_METRICS_ENABLED, "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED");
         // Test log cleanup related env keys
         assert_eq!(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES");
         assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES");

crates/data-usage/Cargo.toml

@@ -37,6 +37,7 @@ hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-filemeta/hotpath-cpu"]
 hotpath.workspace = true
 serde = { workspace = true, features = ["derive"] }
 rmp-serde = { workspace = true }
+async-trait = { workspace = true }
 rustfs-filemeta = { workspace = true }
 
 [lib]

crates/data-usage/src/data_usage.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use serde::{Deserialize, Serialize, ser::SerializeMap as _};
+use serde::{Deserialize, Serialize};
 use std::{
     collections::{HashMap, HashSet},
     hash::{DefaultHasher, Hash, Hasher},
@@ -37,10 +37,6 @@ pub const USAGE_LAST_UPDATE_FUTURE_TOLERANCE: Duration = Duration::from_secs(5 *
 /// Keeping the existing object name preserves rolling-upgrade and rollback
 /// compatibility without allowing an ambiguous snapshot to become authoritative.
 pub const DATA_USAGE_OBJECT_NAME: &str = ".usage.v2.json";
-/// Latest structurally complete scanner observation. Unlike
-/// [`DATA_USAGE_OBJECT_NAME`], this object is never authoritative for quota
-/// admission because namespace activity may have raced the scan.
-pub const DATA_USAGE_OBSERVED_OBJECT_NAME: &str = ".usage.observed.json";
 
 /// Usage snapshot written by scanner implementations predating distributed
 /// leadership fencing. It is read only when neither authoritative snapshot
@@ -55,36 +51,24 @@ pub fn usage_last_update_is_untrusted_future(existing_last_update: SystemTime, n
     existing_last_update > now + USAGE_LAST_UPDATE_FUTURE_TOLERANCE
 }
 
-#[derive(Clone, Copy, Default, Debug, Serialize, Deserialize, PartialEq, Eq)]
+#[derive(Clone, Copy, Default, Debug, Serialize, Deserialize, PartialEq)]
 pub struct TierStats {
     pub total_size: u64,
-    pub num_versions: u64,
-    pub num_objects: u64,
+    pub num_versions: i32,
+    pub num_objects: i32,
 }
 
 impl TierStats {
     pub fn add(&self, u: &TierStats) -> TierStats {
         TierStats {
-            total_size: self.total_size.saturating_add(u.total_size),
-            num_versions: self.num_versions.saturating_add(u.num_versions),
-            num_objects: self.num_objects.saturating_add(u.num_objects),
+            total_size: self.total_size + u.total_size,
+            num_versions: self.num_versions + u.num_versions,
+            num_objects: self.num_objects + u.num_objects,
         }
     }
-
-    /// True when [`TierStats::add`] would report the exact sum instead of saturating.
-    pub fn fits_add(&self, u: &TierStats) -> bool {
-        self.total_size.checked_add(u.total_size).is_some()
-            && self.num_versions.checked_add(u.num_versions).is_some()
-            && self.num_objects.checked_add(u.num_objects).is_some()
-    }
-
-    /// True when this tier contributed nothing, i.e. merging it is a no-op.
-    pub fn is_empty(&self) -> bool {
-        self.total_size == 0 && self.num_versions == 0 && self.num_objects == 0
-    }
 }
 
-#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
+#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq)]
 pub struct AllTierStats {
     pub tiers: HashMap<String, TierStats>,
 }
@@ -94,35 +78,31 @@ impl AllTierStats {
         Self { tiers: HashMap::new() }
     }
 
-    pub fn is_empty(&self) -> bool {
-        self.tiers.is_empty()
-    }
-
-    /// Folds a scan summary's per-tier map in.
-    ///
-    /// Scanners seed the map with a zeroed entry for every configured tier, so
-    /// empty contributions are skipped to keep the persisted cache from growing
-    /// one key per tier on every folder that never held tiered data.
-    pub fn add_sizes(&mut self, tiers: &HashMap<String, TierStats>) {
+    pub fn add_sizes(&mut self, tiers: HashMap<String, TierStats>) {
         for (tier, st) in tiers {
-            if st.is_empty() {
-                continue;
-            }
-            let entry = self.tiers.entry(tier.clone()).or_default();
-            *entry = entry.add(st);
+            self.tiers
+                .insert(tier.clone(), self.tiers.get(&tier).copied().unwrap_or_default().add(&st));
         }
     }
 
-    pub fn merge(&mut self, other: &AllTierStats) {
-        self.add_sizes(&other.tiers);
+    pub fn merge(&mut self, other: AllTierStats) {
+        for (tier, st) in other.tiers {
+            self.tiers
+                .insert(tier.clone(), self.tiers.get(&tier).copied().unwrap_or_default().add(&st));
+        }
     }
 
-    /// True when [`AllTierStats::merge`] would report exact sums for every tier.
-    pub fn fits_merge(&self, other: &AllTierStats) -> bool {
-        other
-            .tiers
-            .iter()
-            .all(|(tier, right)| self.tiers.get(tier).is_none_or(|left| left.fits_add(right)))
+    pub fn populate_stats(&self, stats: &mut HashMap<String, TierStats>) {
+        for (tier, st) in &self.tiers {
+            stats.insert(
+                tier.clone(),
+                TierStats {
+                    total_size: st.total_size,
+                    num_versions: st.num_versions,
+                    num_objects: st.num_objects,
+                },
+            );
+        }
     }
 }
 
@@ -203,14 +183,6 @@ pub struct DataUsageInfo {
     pub objects_total_size: u64,
     /// Replication info across all buckets
     pub replication_info: HashMap<String, BucketTargetUsageInfo>,
-    /// Usage per storage class and remote tier across all buckets.
-    ///
-    /// Absent on snapshots written before per-tier accounting was published,
-    /// and on clusters with no remote tier configured: the scanner classifies
-    /// objects by tier (including `STANDARD`/`REDUCED_REDUNDANCY`) only once a
-    /// tier exists, so an absent value means "not accounted", never "zero".
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    pub tier_stats: Option<AllTierStats>,
 
     /// Total number of buckets in this cluster
     pub buckets_count: u64,
@@ -222,20 +194,6 @@ pub struct DataUsageInfo {
     /// explicit entry for every bucket, including confirmed-empty buckets.
     #[serde(default)]
     pub usage_snapshot_complete: bool,
-    /// Whether no namespace activity or dirty-usage generation changed while
-    /// the coordinated snapshot was being produced.
-    ///
-    /// `false` still describes a structurally complete, useful point-in-time
-    /// usage view, but follow-up scanner work remains pending. `None` is kept
-    /// for snapshots written before this status became observable.
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    pub usage_snapshot_converged: Option<bool>,
-    /// Identity of the authoritative snapshot from which a nonconverged
-    /// observation started. Admin readers require an exact match before using
-    /// the observation, so bucket namespace mutations fence old observations
-    /// without relying on synchronized clocks.
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    pub usage_snapshot_authoritative_baseline: Option<DataUsageSnapshotIdentity>,
     /// Deprecated kept here for backward compatibility reasons
     pub bucket_sizes: HashMap<String, u64>,
     /// Per-disk snapshot information when available
@@ -243,59 +201,6 @@ pub struct DataUsageInfo {
     pub disk_usage_status: Vec<DiskUsageStatus>,
 }
 
-/// Stable identity fields changed by both coordinated scanner publication and
-/// backward-compatible bucket namespace cleanup.
-#[derive(Debug, Default, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
-pub struct DataUsageSnapshotIdentity {
-    pub last_update: Option<SystemTime>,
-    pub scanner_cycle: Option<u64>,
-    pub scanner_epoch: Option<u64>,
-}
-
-impl DataUsageInfo {
-    pub fn snapshot_identity(&self) -> DataUsageSnapshotIdentity {
-        DataUsageSnapshotIdentity {
-            last_update: self.last_update,
-            scanner_cycle: self.scanner_cycle,
-            scanner_epoch: self.scanner_epoch,
-        }
-    }
-}
-
-/// Return whether `candidate` was produced after `baseline`.
-///
-/// New coordinated snapshots are ordered by leadership epoch and scanner
-/// cycle. The timestamp fallback preserves ordering for legacy snapshots that
-/// predate those fields.
-pub fn data_usage_snapshot_is_newer(candidate: &DataUsageInfo, baseline: &DataUsageInfo) -> bool {
-    match (
-        candidate.scanner_epoch.zip(candidate.scanner_cycle),
-        baseline.scanner_epoch.zip(baseline.scanner_cycle),
-    ) {
-        (Some(candidate), Some(baseline)) => candidate > baseline,
-        (Some(_), None) => true,
-        (None, Some(_)) => false,
-        (None, None) => match (candidate.last_update, baseline.last_update) {
-            (Some(candidate), Some(baseline)) => candidate > baseline,
-            (Some(_), None) => true,
-            (None, Some(_) | None) => false,
-        },
-    }
-}
-
-/// Return whether a nonconverged observation may safely supersede the admin
-/// view of `authoritative`.
-///
-/// The exact baseline identity is independent of clock ordering. Older binaries
-/// already advance the authoritative timestamp when deleting a bucket, so a
-/// rollback delete/recreate fences the previous bucket incarnation too.
-pub fn observed_data_usage_is_newer(observed: &DataUsageInfo, authoritative: &DataUsageInfo) -> bool {
-    observed.usage_snapshot_converged == Some(false)
-        && observed.is_complete_bucket_usage_snapshot()
-        && observed.usage_snapshot_authoritative_baseline.as_ref() == Some(&authoritative.snapshot_identity())
-        && data_usage_snapshot_is_newer(observed, authoritative)
-}
-
 /// Metadata describing the status of a disk-level data usage snapshot.
 #[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
 pub struct DiskUsageStatus {
@@ -657,7 +562,7 @@ impl ReplicationAllStats {
 }
 
 /// Data usage cache entry
-#[derive(Clone, Debug, Default, Deserialize)]
+#[derive(Clone, Debug, Default, Serialize, Deserialize)]
 pub struct DataUsageEntry {
     pub children: DataUsageHashMap,
     // These fields do not include any children.
@@ -672,34 +577,6 @@ pub struct DataUsageEntry {
     /// Number of objects that failed to scan (e.g., IO errors)
     #[serde(default)]
     pub failed_objects: usize,
-    /// Per-tier usage contributed by this entry, present only once a scan
-    /// observed tier-classified objects.
-    #[serde(default)]
-    pub all_tier_stats: Option<AllTierStats>,
-}
-
-impl Serialize for DataUsageEntry {
-    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
-    where
-        S: serde::Serializer,
-    {
-        // Keep entries map-encoded so older readers can ignore fields appended
-        // by newer scanner versions during rolling upgrades. The derived
-        // (array) encoding made any appended field a decode error for them.
-        let mut state = serializer.serialize_map(Some(11))?;
-        state.serialize_entry("children", &self.children)?;
-        state.serialize_entry("size", &self.size)?;
-        state.serialize_entry("objects", &self.objects)?;
-        state.serialize_entry("versions", &self.versions)?;
-        state.serialize_entry("delete_markers", &self.delete_markers)?;
-        state.serialize_entry("obj_sizes", &self.obj_sizes)?;
-        state.serialize_entry("obj_versions", &self.obj_versions)?;
-        state.serialize_entry("replication_stats", &self.replication_stats)?;
-        state.serialize_entry("compacted", &self.compacted)?;
-        state.serialize_entry("failed_objects", &self.failed_objects)?;
-        state.serialize_entry("all_tier_stats", &self.all_tier_stats)?;
-        state.end()
-    }
 }
 
 impl DataUsageEntry {
@@ -758,22 +635,10 @@ impl DataUsageEntry {
             }
         }
 
-        if let Some(o_tiers) = other.all_tier_stats.as_ref().filter(|tiers| !tiers.is_empty()) {
-            self.all_tier_stats.get_or_insert_with(AllTierStats::new).merge(o_tiers);
-        }
-
         self.obj_sizes.merge_from(&other.obj_sizes);
         self.obj_versions.merge_from(&other.obj_versions);
     }
 
-    /// Folds a scan summary's per-tier map into this entry.
-    pub fn add_tier_sizes(&mut self, tiers: &HashMap<String, TierStats>) {
-        if tiers.values().all(TierStats::is_empty) {
-            return;
-        }
-        self.all_tier_stats.get_or_insert_with(AllTierStats::new).add_sizes(tiers);
-    }
-
     pub fn checked_merge(&mut self, other: &DataUsageEntry) -> bool {
         let scalar_counts_fit = self.objects.checked_add(other.objects).is_some()
             && self.versions.checked_add(other.versions).is_some()
@@ -833,12 +698,7 @@ impl DataUsageEntry {
             }
         };
 
-        let tier_stats_fit = match (&self.all_tier_stats, &other.all_tier_stats) {
-            (_, None) | (None, Some(_)) => true,
-            (Some(left), Some(right)) => left.fits_merge(right),
-        };
-
-        if !scalar_counts_fit || !histograms_fit || !replication_fits || !tier_stats_fit {
+        if !scalar_counts_fit || !histograms_fit || !replication_fits {
             return false;
         }
         self.merge(other);
@@ -846,15 +706,8 @@ impl DataUsageEntry {
     }
 }
 
-/// Read-only projection of the scanner's `.usage-cache.bin` info block.
-///
-/// The canonical wire format is written by the hand-written map-encoded
-/// `Serialize` on the scanner-side `DataUsageCacheInfo`
-/// (`crates/scanner/src/data_usage_define.rs`), which carries 16 fields.
-/// This type decodes only the shared subset and is deliberately not
-/// `Serialize`: a derived (array) encoding of this 6-field subset would
-/// corrupt the cache for scanner readers, so no write path may exist here.
-#[derive(Clone, Debug, Default, Deserialize)]
+/// Data usage cache info
+#[derive(Clone, Debug, Default, Serialize, Deserialize)]
 pub struct DataUsageCacheInfo {
     pub name: String,
     pub next_cycle: u64,
@@ -870,12 +723,8 @@ pub struct DataUsageCacheInfo {
     pub snapshot_complete: bool,
 }
 
-/// Read-only projection of a scanner-written `.usage-cache.bin` file.
-///
-/// The scanner-side `DataUsageCache` (`crates/scanner/src/data_usage_define.rs`)
-/// owns the persisted format; this type only decodes it (see
-/// [`DataUsageCacheInfo`]) and must never grow a serialization path.
-#[derive(Clone, Debug, Default, Deserialize)]
+/// Data usage cache
+#[derive(Clone, Debug, Default, Serialize, Deserialize)]
 pub struct DataUsageCache {
     pub info: DataUsageCacheInfo,
     pub cache: HashMap<String, DataUsageEntry>,
@@ -1189,7 +1038,6 @@ impl DataUsageCache {
             versions_total_count: flat.versions as u64,
             delete_markers_total_count: flat.delete_markers as u64,
             objects_total_size: flat.size as u64,
-            tier_stats: flat.all_tier_stats.filter(|tiers| !tiers.is_empty()),
             buckets_count: u64::try_from(buckets.len()).unwrap_or(u64::MAX),
             buckets_usage,
             usage_snapshot_complete: self.info.snapshot_complete,
@@ -1197,10 +1045,31 @@ impl DataUsageCache {
         }
     }
 
+    pub fn marshal_msg(&self) -> Result<Vec<u8>, Box<dyn std::error::Error + Send + Sync>> {
+        let mut buf = Vec::new();
+        self.serialize(&mut rmp_serde::Serializer::new(&mut buf))?;
+        Ok(buf)
+    }
+
     pub fn unmarshal(buf: &[u8]) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
         let t: Self = rmp_serde::from_slice(buf)?;
         Ok(t)
     }
+
+    // Note: load and save methods are storage-specific and should be implemented
+    // in the ecstore crate where storage access is available
+}
+
+/// Trait for storage-specific operations on DataUsageCache
+#[async_trait::async_trait]
+pub trait DataUsageCacheStorage {
+    /// Load data usage cache from backend storage
+    async fn load(store: &dyn std::any::Any, name: &str) -> Result<Self, Box<dyn std::error::Error + Send + Sync>>
+    where
+        Self: Sized;
+
+    /// Save data usage cache to backend storage
+    async fn save(&self, name: &str) -> Result<(), Box<dyn std::error::Error + Send + Sync>>;
 }
 
 // Helper structs and functions for cache operations
@@ -1656,248 +1525,6 @@ mod tests {
         buckets_count: u64,
     }
 
-    fn tier_entry(tier: &str, stats: TierStats) -> DataUsageEntry {
-        let mut entry = DataUsageEntry::default();
-        entry.add_tier_sizes(&HashMap::from([(tier.to_string(), stats)]));
-        entry
-    }
-
-    #[test]
-    fn tier_stats_survive_entry_merge() {
-        let mut left = tier_entry(
-            "WARM",
-            TierStats {
-                total_size: 10,
-                num_versions: 2,
-                num_objects: 1,
-            },
-        );
-        let mut right = tier_entry(
-            "WARM",
-            TierStats {
-                total_size: 5,
-                num_versions: 1,
-                num_objects: 1,
-            },
-        );
-        right.add_tier_sizes(&HashMap::from([(
-            "COLD".to_string(),
-            TierStats {
-                total_size: 7,
-                num_versions: 1,
-                num_objects: 0,
-            },
-        )]));
-
-        assert!(left.checked_merge(&right), "merging exact tier totals must be accepted");
-
-        let tiers = &left.all_tier_stats.expect("merged entry keeps tier stats").tiers;
-        assert_eq!(
-            tiers.get("WARM"),
-            Some(&TierStats {
-                total_size: 15,
-                num_versions: 3,
-                num_objects: 2,
-            })
-        );
-        assert_eq!(
-            tiers.get("COLD"),
-            Some(&TierStats {
-                total_size: 7,
-                num_versions: 1,
-                num_objects: 0,
-            })
-        );
-    }
-
-    #[test]
-    fn tier_stats_merge_into_an_untiered_entry() {
-        let mut left = DataUsageEntry::default();
-        let right = tier_entry(
-            "WARM",
-            TierStats {
-                total_size: 10,
-                num_versions: 1,
-                num_objects: 1,
-            },
-        );
-
-        assert!(left.checked_merge(&right));
-
-        assert_eq!(
-            left.all_tier_stats.expect("tier stats adopted from the merged entry").tiers["WARM"],
-            TierStats {
-                total_size: 10,
-                num_versions: 1,
-                num_objects: 1,
-            }
-        );
-    }
-
-    #[test]
-    fn checked_merge_rejects_overflowing_tier_totals() {
-        let mut left = tier_entry(
-            "WARM",
-            TierStats {
-                total_size: u64::MAX,
-                num_versions: 1,
-                num_objects: 1,
-            },
-        );
-        let right = tier_entry(
-            "WARM",
-            TierStats {
-                total_size: 1,
-                num_versions: 1,
-                num_objects: 1,
-            },
-        );
-
-        assert!(!left.checked_merge(&right), "saturating tier totals must not be published");
-        assert_eq!(left.all_tier_stats.expect("left is untouched").tiers["WARM"].total_size, u64::MAX);
-    }
-
-    /// Entry shape released before per-tier accounting, using the derived
-    /// (array) encoding those writers produced.
-    #[derive(Serialize, Deserialize)]
-    struct LegacyEntry {
-        children: DataUsageHashMap,
-        size: usize,
-        objects: usize,
-        versions: usize,
-        delete_markers: usize,
-        obj_sizes: SizeHistogram,
-        obj_versions: VersionsHistogram,
-        replication_stats: Option<ReplicationAllStats>,
-        compacted: bool,
-        #[serde(default)]
-        failed_objects: usize,
-    }
-
-    #[test]
-    fn entries_are_map_encoded_so_appended_fields_stay_readable() {
-        // A derived (array) encoding turns every appended field into a decode
-        // error for readers built before it existed, which would cost a mixed
-        // -version cluster its whole scan cache. Entries must stay map-encoded.
-        let current = tier_entry(
-            "WARM",
-            TierStats {
-                total_size: 3,
-                num_versions: 1,
-                num_objects: 1,
-            },
-        );
-        let mut encoded = Vec::new();
-        current
-            .serialize(&mut rmp_serde::Serializer::new(&mut encoded))
-            .expect("encode current entry");
-
-        let legacy: LegacyEntry = rmp_serde::from_slice(&encoded).expect("legacy reader should ignore the appended field");
-        assert_eq!(legacy.objects, 0);
-    }
-
-    #[test]
-    fn legacy_array_encoded_entries_still_load() {
-        let legacy = LegacyEntry {
-            children: DataUsageHashMap::default(),
-            size: 12,
-            objects: 3,
-            versions: 4,
-            delete_markers: 1,
-            obj_sizes: SizeHistogram::default(),
-            obj_versions: VersionsHistogram::default(),
-            replication_stats: None,
-            compacted: false,
-            failed_objects: 2,
-        };
-        let mut encoded = Vec::new();
-        legacy
-            .serialize(&mut rmp_serde::Serializer::new(&mut encoded))
-            .expect("encode legacy entry");
-
-        let decoded: DataUsageEntry = rmp_serde::from_slice(&encoded).expect("current reader should default the missing field");
-
-        assert_eq!(decoded.size, 12);
-        assert_eq!(decoded.failed_objects, 2);
-        assert!(decoded.all_tier_stats.is_none());
-    }
-
-    /// Scanner-written `.usage-cache.bin` bytes: a 2-element array of the
-    /// canonical 16-field map-encoded info block and one map-encoded entry.
-    /// Captured from the canonical writer's `marshal_msg` — see
-    /// `usage_cache_wire_format_is_pinned` in
-    /// `crates/scanner/src/data_usage_define.rs`, which pins these exact
-    /// bytes and documents regeneration. Hardcoded here because a
-    /// dev-dependency on rustfs-scanner would pull the whole ecstore tree
-    /// into this crate's test build, and a fixture generated at test runtime
-    /// could not detect writer drift anyway.
-    const SCANNER_USAGE_CACHE_WIRE_FIXTURE: &[u8] = &[
-        0x92, 0xde, 0x00, 0x10, 0xa4, 0x6e, 0x61, 0x6d, 0x65, 0xab, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65,
-        0x74, 0xaa, 0x6e, 0x65, 0x78, 0x74, 0x5f, 0x63, 0x79, 0x63, 0x6c, 0x65, 0x07, 0xac, 0x6c, 0x65, 0x61, 0x64, 0x65, 0x72,
-        0x5f, 0x65, 0x70, 0x6f, 0x63, 0x68, 0x09, 0xab, 0x6c, 0x61, 0x73, 0x74, 0x5f, 0x75, 0x70, 0x64, 0x61, 0x74, 0x65, 0x92,
-        0xce, 0x65, 0x53, 0xf1, 0x00, 0x00, 0xac, 0x73, 0x6b, 0x69, 0x70, 0x5f, 0x68, 0x65, 0x61, 0x6c, 0x69, 0x6e, 0x67, 0xc3,
-        0xa9, 0x6c, 0x69, 0x66, 0x65, 0x63, 0x79, 0x63, 0x6c, 0x65, 0xc0, 0xab, 0x72, 0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74,
-        0x69, 0x6f, 0x6e, 0xc0, 0xae, 0x66, 0x61, 0x69, 0x6c, 0x65, 0x64, 0x5f, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x81,
-        0xb0, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2f, 0x6c, 0x6f, 0x73, 0x74, 0x0b, 0xb1, 0x73,
-        0x63, 0x61, 0x6e, 0x5f, 0x72, 0x65, 0x73, 0x75, 0x6d, 0x65, 0x5f, 0x61, 0x66, 0x74, 0x65, 0x72, 0xb2, 0x77, 0x69, 0x72,
-        0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2f, 0x72, 0x65, 0x73, 0x75, 0x6d, 0x65, 0xaf, 0x73, 0x63, 0x61, 0x6e,
-        0x5f, 0x63, 0x68, 0x65, 0x63, 0x6b, 0x70, 0x6f, 0x69, 0x6e, 0x74, 0xc0, 0xad, 0x70, 0x65, 0x6e, 0x64, 0x69, 0x6e, 0x67,
-        0x5f, 0x68, 0x65, 0x61, 0x6c, 0x73, 0x91, 0x9a, 0xa6, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0xab, 0x77, 0x69, 0x72, 0x65,
-        0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0xa6, 0x62, 0x72, 0x6f, 0x6b, 0x65, 0x6e, 0xc0, 0x01, 0x64, 0xcc, 0xc8, 0x03,
-        0xa8, 0x64, 0x65, 0x66, 0x65, 0x72, 0x72, 0x65, 0x64, 0xa6, 0x62, 0x75, 0x64, 0x67, 0x65, 0x74, 0xab, 0x6f, 0x62, 0x6a,
-        0x65, 0x63, 0x74, 0x5f, 0x6c, 0x6f, 0x63, 0x6b, 0xc0, 0xa6, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x92, 0x01, 0x02, 0xb1,
-        0x73, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x5f, 0x63, 0x6f, 0x6d, 0x70, 0x6c, 0x65, 0x74, 0x65, 0xc3, 0xb0, 0x73,
-        0x63, 0x61, 0x6e, 0x5f, 0x70, 0x6c, 0x61, 0x6e, 0x5f, 0x64, 0x69, 0x67, 0x65, 0x73, 0x74, 0xdc, 0x00, 0x20, 0x03, 0x03,
-        0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03,
-        0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0xb0, 0x63, 0x61, 0x63, 0x68, 0x65, 0x5f, 0x6b, 0x65, 0x79,
-        0x5f, 0x66, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x01, 0x81, 0xab, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65,
-        0x74, 0x8b, 0xa8, 0x63, 0x68, 0x69, 0x6c, 0x64, 0x72, 0x65, 0x6e, 0x90, 0xa4, 0x73, 0x69, 0x7a, 0x65, 0xcd, 0x10, 0x00,
-        0xa7, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x03, 0xa8, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x73, 0x05, 0xae,
-        0x64, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x5f, 0x6d, 0x61, 0x72, 0x6b, 0x65, 0x72, 0x73, 0x01, 0xa9, 0x6f, 0x62, 0x6a, 0x5f,
-        0x73, 0x69, 0x7a, 0x65, 0x73, 0x9b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xac, 0x6f, 0x62,
-        0x6a, 0x5f, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x73, 0x97, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xb1, 0x72,
-        0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x73, 0xc0, 0xa9, 0x63, 0x6f,
-        0x6d, 0x70, 0x61, 0x63, 0x74, 0x65, 0x64, 0xc3, 0xae, 0x66, 0x61, 0x69, 0x6c, 0x65, 0x64, 0x5f, 0x6f, 0x62, 0x6a, 0x65,
-        0x63, 0x74, 0x73, 0x02, 0xae, 0x61, 0x6c, 0x6c, 0x5f, 0x74, 0x69, 0x65, 0x72, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x73, 0x91,
-        0x81, 0xa4, 0x57, 0x41, 0x52, 0x4d, 0x93, 0xcd, 0x08, 0x00, 0x02, 0x01,
-    ];
-
-    #[test]
-    fn thin_usage_cache_decodes_scanner_wire_fixture() {
-        let decoded =
-            DataUsageCache::unmarshal(SCANNER_USAGE_CACHE_WIRE_FIXTURE).expect("thin projection decodes a scanner-written cache");
-
-        // The six fields shared with the scanner's 16-field info block; the
-        // remaining ten (lifecycle, replication, checkpoint, heals, ...) must
-        // be skipped, not error.
-        assert_eq!(decoded.info.name, "wire-bucket");
-        assert_eq!(decoded.info.next_cycle, 7);
-        assert_eq!(
-            decoded.info.last_update,
-            Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1_700_000_000))
-        );
-        assert!(decoded.info.skip_healing);
-        assert_eq!(decoded.info.failed_objects.get("wire-bucket/lost"), Some(&11));
-        assert!(decoded.info.snapshot_complete);
-
-        // Entries use the shared canonical map-encoded type end to end.
-        let entry = decoded.cache.get("wire-bucket").expect("fixture entry decodes");
-        assert_eq!(entry.size, 4096);
-        assert_eq!(entry.objects, 3);
-        assert_eq!(entry.versions, 5);
-        assert_eq!(entry.delete_markers, 1);
-        assert!(entry.compacted);
-        assert_eq!(entry.failed_objects, 2);
-        assert_eq!(
-            entry.all_tier_stats.as_ref().and_then(|tiers| tiers.tiers.get("WARM")),
-            Some(&TierStats {
-                total_size: 2048,
-                num_versions: 2,
-                num_objects: 1,
-            })
-        );
-    }
-
     #[test]
     fn hash_path_uses_portable_slash_semantics() {
         for (input, expected) in [
@@ -1920,8 +1547,6 @@ mod tests {
         let current = DataUsageInfo {
             last_update: Some(SystemTime::UNIX_EPOCH),
             usage_snapshot_complete: true,
-            usage_snapshot_converged: Some(false),
-            usage_snapshot_authoritative_baseline: Some(DataUsageSnapshotIdentity::default()),
             ..Default::default()
         };
         let encoded = rmp_serde::to_vec_named(&current).expect("encode current data usage snapshot");
@@ -1929,76 +1554,6 @@ mod tests {
 
         assert_eq!(legacy.buckets_count, 0);
         assert!(current.is_complete_bucket_usage_snapshot());
-        assert_eq!(current.usage_snapshot_converged, Some(false));
-    }
-
-    #[test]
-    fn convergence_marker_defaults_to_unknown_for_older_snapshots() {
-        let encoded = rmp_serde::to_vec_named(&DataUsageInfo {
-            last_update: Some(SystemTime::UNIX_EPOCH),
-            usage_snapshot_complete: true,
-            ..Default::default()
-        })
-        .expect("encode pre-convergence data usage snapshot");
-        let decoded: DataUsageInfo = rmp_serde::from_slice(&encoded).expect("decode older data usage snapshot");
-
-        assert!(decoded.is_complete_bucket_usage_snapshot());
-        assert_eq!(decoded.usage_snapshot_converged, None);
-    }
-
-    #[test]
-    fn observation_selection_is_clock_independent_and_baseline_fenced() {
-        let mut authoritative = DataUsageInfo {
-            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(600)),
-            scanner_epoch: Some(7),
-            scanner_cycle: Some(10),
-            usage_snapshot_complete: true,
-            ..Default::default()
-        };
-        let observed = DataUsageInfo {
-            // A newer leader may have a slower wall clock.
-            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(300)),
-            scanner_epoch: Some(8),
-            scanner_cycle: Some(1),
-            usage_snapshot_complete: true,
-            usage_snapshot_converged: Some(false),
-            usage_snapshot_authoritative_baseline: Some(authoritative.snapshot_identity()),
-            ..Default::default()
-        };
-
-        assert!(observed_data_usage_is_newer(&observed, &authoritative));
-
-        authoritative.last_update = Some(SystemTime::UNIX_EPOCH + Duration::from_secs(601));
-        assert!(
-            !observed_data_usage_is_newer(&observed, &authoritative),
-            "an old-binary namespace mutation must fence the prior bucket incarnation regardless of clock skew"
-        );
-    }
-
-    #[test]
-    fn observation_selection_requires_nonconverged_complete_newer_data() {
-        let authoritative = DataUsageInfo {
-            last_update: Some(SystemTime::UNIX_EPOCH),
-            scanner_epoch: Some(2),
-            scanner_cycle: Some(10),
-            usage_snapshot_complete: true,
-            ..Default::default()
-        };
-        let baseline = Some(authoritative.snapshot_identity());
-        let candidate = |epoch, cycle, converged, complete| DataUsageInfo {
-            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
-            scanner_epoch: Some(epoch),
-            scanner_cycle: Some(cycle),
-            usage_snapshot_complete: complete,
-            usage_snapshot_converged: converged,
-            usage_snapshot_authoritative_baseline: baseline,
-            ..Default::default()
-        };
-
-        assert!(observed_data_usage_is_newer(&candidate(2, 11, Some(false), true), &authoritative));
-        assert!(!observed_data_usage_is_newer(&candidate(2, 9, Some(false), true), &authoritative));
-        assert!(!observed_data_usage_is_newer(&candidate(2, 11, Some(true), true), &authoritative));
-        assert!(!observed_data_usage_is_newer(&candidate(2, 11, Some(false), false), &authoritative));
     }
 
     #[test]
@@ -2346,44 +1901,6 @@ mod tests {
         assert_eq!(info.buckets_count, 2);
         assert!(info.buckets_usage.is_empty());
         assert_eq!(info.objects_total_count, 3);
-        assert!(info.tier_stats.is_none());
-    }
-
-    #[test]
-    fn test_dui_reports_tier_usage_from_the_flattened_tree() {
-        let root_hash = hash_path("root");
-        let bucket_hash = hash_path("bucket-a");
-        let mut cache = DataUsageCache {
-            info: DataUsageCacheInfo {
-                name: "root".to_string(),
-                ..Default::default()
-            },
-            ..Default::default()
-        };
-        cache.replace_hashed(&root_hash, &None, &DataUsageEntry::default());
-        cache.replace_hashed(
-            &bucket_hash,
-            &Some(root_hash),
-            &tier_entry(
-                "WARM",
-                TierStats {
-                    total_size: 40,
-                    num_versions: 2,
-                    num_objects: 2,
-                },
-            ),
-        );
-
-        let info = cache.dui("root", &["bucket-a".to_string()]);
-
-        assert_eq!(
-            info.tier_stats.expect("child tier usage should roll up to the root").tiers["WARM"],
-            TierStats {
-                total_size: 40,
-                num_versions: 2,
-                num_objects: 2,
-            }
-        );
     }
 
     #[test]

crates/e2e_test/src/admin_iam_crud_test.rs

@@ -26,9 +26,7 @@
 //! Later batches tracked on backlog#1154: config get/set, info, pools status,
 //! group lifecycle, import/export IAM.
 
-use crate::common::{
-    RustFSTestEnvironment, admin_ok, admin_request, admin_request_with_session_token, build_test_sts_client, init_logging,
-};
+use crate::common::{RustFSTestEnvironment, admin_ok, admin_request, init_logging};
 use aws_sdk_s3::config::{Credentials, Region};
 use aws_sdk_s3::primitives::ByteStream;
 use aws_sdk_s3::{Client, Config};
@@ -89,262 +87,6 @@ fn bucket_rw_policy(bucket: &str) -> String {
     .to_string()
 }
 
-async fn create_user_with_service_account_update_policy(
-    env: &RustFSTestEnvironment,
-    user: &str,
-    secret: &str,
-    policy: &str,
-) -> TestResult {
-    admin_ok(
-        env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-user?accessKey={user}"),
-        Some(serde_json::json!({ "secretKey": secret, "status": "enabled" }).to_string()),
-    )
-    .await?;
-    admin_ok(
-        env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-canned-policy?name={policy}"),
-        Some(
-            serde_json::json!({
-                "Version": "2012-10-17",
-                "Statement": [
-                    {
-                        "Effect": "Allow",
-                        "Action": ["admin:UpdateServiceAccount"]
-                    },
-                    {
-                        "Effect": "Allow",
-                        "Action": ["sts:AssumeRole"],
-                        "Resource": ["arn:aws:s3:::*"]
-                    }
-                ]
-            })
-            .to_string(),
-        ),
-    )
-    .await?;
-    admin_ok(
-        env,
-        http::Method::POST,
-        "/rustfs/admin/v3/idp/builtin/policy/attach",
-        Some(serde_json::json!({ "policies": [policy], "user": user }).to_string()),
-    )
-    .await?;
-    Ok(())
-}
-
-async fn create_service_account_for(
-    env: &RustFSTestEnvironment,
-    parent: &str,
-) -> Result<(String, String), Box<dyn Error + Send + Sync>> {
-    let response = admin_ok(
-        env,
-        http::Method::PUT,
-        "/rustfs/admin/v3/add-service-accounts",
-        Some(serde_json::json!({ "targetUser": parent }).to_string()),
-    )
-    .await?;
-    let response: serde_json::Value = serde_json::from_str(&response)?;
-    let access_key = response["credentials"]["accessKey"]
-        .as_str()
-        .ok_or("service account response should contain credentials.accessKey")?
-        .to_owned();
-    let secret_key = response["credentials"]["secretKey"]
-        .as_str()
-        .ok_or("service account response should contain credentials.secretKey")?
-        .to_owned();
-    Ok((access_key, secret_key))
-}
-
-async fn assert_admin_status(
-    env: &RustFSTestEnvironment,
-    credentials: (&str, &str, Option<&str>),
-    path: &str,
-    body: String,
-    expected: StatusCode,
-    context: &str,
-) -> TestResult {
-    let (access_key, secret_key, session_token) = credentials;
-    let (status, response) =
-        admin_request_with_session_token(&env.url, http::Method::POST, path, Some(body), access_key, secret_key, session_token)
-            .await?;
-    assert_eq!(status, expected, "{context}: got {status}: {response}");
-    if expected == StatusCode::FORBIDDEN {
-        assert!(response.contains("AccessDenied"), "{context}: expected AccessDenied body, got {response}");
-    }
-    Ok(())
-}
-
-#[tokio::test]
-#[serial]
-async fn test_update_service_account_enforces_owner_and_parent_scope() -> TestResult {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let parent = "updateparent";
-    let parent_secret = "updateparentsecret";
-    let outsider = "updateoutsider";
-    let outsider_secret = "updateoutsidersecret";
-    let ordinary = "updateordinary";
-    let ordinary_secret = "updateordinarysecret";
-    create_user_with_service_account_update_policy(&env, parent, parent_secret, "update-parent-policy").await?;
-    create_user_with_service_account_update_policy(&env, outsider, outsider_secret, "update-outsider-policy").await?;
-    admin_ok(
-        &env,
-        http::Method::POST,
-        "/rustfs/admin/v3/idp/builtin/policy/attach",
-        Some(serde_json::json!({ "policies": ["consoleAdmin"], "user": outsider }).to_string()),
-    )
-    .await?;
-    admin_ok(
-        &env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-user?accessKey={ordinary}"),
-        Some(serde_json::json!({ "secretKey": ordinary_secret, "status": "enabled" }).to_string()),
-    )
-    .await?;
-
-    let (target_access_key, _) = create_service_account_for(&env, parent).await?;
-    let target_path = format!("/rustfs/admin/v3/update-service-account?accessKey={target_access_key}");
-
-    assert_admin_status(
-        &env,
-        (&env.access_key, &env.secret_key, None),
-        &target_path,
-        serde_json::json!({}).to_string(),
-        StatusCode::NO_CONTENT,
-        "root no-op update across parents must succeed",
-    )
-    .await?;
-
-    let custom_policy = serde_json::json!({
-        "Version": "2012-10-17",
-        "Statement": [{
-            "Effect": "Allow",
-            "Action": ["s3:GetObject"],
-            "Resource": ["arn:aws:s3:::update-scope/*"]
-        }]
-    });
-    assert_admin_status(
-        &env,
-        (&env.access_key, &env.secret_key, None),
-        &target_path,
-        serde_json::json!({ "newPolicy": custom_policy }).to_string(),
-        StatusCode::NO_CONTENT,
-        "root implied-to-custom update across parents must succeed",
-    )
-    .await?;
-
-    assert_admin_status(
-        &env,
-        (parent, parent_secret, None),
-        &target_path,
-        serde_json::json!({ "newDescription": "updated by parent" }).to_string(),
-        StatusCode::NO_CONTENT,
-        "parent with UpdateServiceAccount may update its own service account",
-    )
-    .await?;
-
-    let takeover = serde_json::json!({
-        "newSecretKey": "cross-parent-takeover-secret",
-        "newDescription": "cross-parent takeover"
-    })
-    .to_string();
-    assert_admin_status(
-        &env,
-        (ordinary, ordinary_secret, None),
-        &target_path,
-        takeover.clone(),
-        StatusCode::FORBIDDEN,
-        "ordinary user must not update another parent's service account",
-    )
-    .await?;
-    assert_admin_status(
-        &env,
-        (outsider, outsider_secret, None),
-        &target_path,
-        takeover.clone(),
-        StatusCode::FORBIDDEN,
-        "non-owner consoleAdmin must not update across parents",
-    )
-    .await?;
-
-    let (derived_access_key, derived_secret_key) = create_service_account_for(&env, outsider).await?;
-    assert_admin_status(
-        &env,
-        (&derived_access_key, &derived_secret_key, None),
-        &target_path,
-        takeover.clone(),
-        StatusCode::FORBIDDEN,
-        "service-account credential must not update across parents",
-    )
-    .await?;
-
-    let assumed = build_test_sts_client(&env.url, outsider, outsider_secret, None, "e2e-admin-update-service-account")
-        .assume_role()
-        .role_arn("arn:aws:iam::123456789012:role/update-service-account")
-        .role_session_name("update-service-account-scope")
-        .send()
-        .await?;
-    let temporary = assumed
-        .credentials()
-        .ok_or("AssumeRole response should contain credentials")?;
-    assert_admin_status(
-        &env,
-        (temporary.access_key_id(), temporary.secret_access_key(), Some(temporary.session_token())),
-        &target_path,
-        takeover,
-        StatusCode::FORBIDDEN,
-        "temporary credential must not update across parents",
-    )
-    .await?;
-
-    let info = admin_ok(
-        &env,
-        http::Method::GET,
-        &format!("/rustfs/admin/v3/info-service-account?accessKey={target_access_key}"),
-        None,
-    )
-    .await?;
-    let info: serde_json::Value = serde_json::from_str(&info)?;
-    assert_eq!(
-        info["impliedPolicy"].as_bool(),
-        Some(false),
-        "root update must replace the implied policy with a custom policy"
-    );
-    assert!(
-        info["policy"].as_str().is_some_and(|policy| policy.contains("s3:GetObject")),
-        "custom policy must round-trip through the handler: {info}"
-    );
-    assert_eq!(
-        info["description"].as_str(),
-        Some("updated by parent"),
-        "denied takeover attempts must not mutate target"
-    );
-
-    let (missing_status, missing_body) = admin_request(
-        &env.url,
-        http::Method::POST,
-        "/rustfs/admin/v3/update-service-account?accessKey=missing-service-account",
-        Some(serde_json::json!({}).to_string()),
-        &env.access_key,
-        &env.secret_key,
-    )
-    .await?;
-    assert_eq!(missing_status, StatusCode::NOT_FOUND, "missing target must fail closed: {missing_body}");
-    assert!(
-        missing_body.contains("NoSuchResource"),
-        "missing target must preserve the lookup error: {missing_body}"
-    );
-
-    env.stop_server();
-    Ok(())
-}
-
 /// Full user -> policy -> service-account lifecycle, proving each management
 /// call takes effect on the data plane, not just that the endpoint answers 200.
 #[tokio::test]

crates/e2e_test/src/bucket_logging_test.rs

@@ -16,17 +16,91 @@
 
 #[cfg(test)]
 mod tests {
-    use crate::common::{RustFSTestEnvironment, init_logging, signed_s3_request};
+    use crate::common::{RustFSTestEnvironment, init_logging};
     use aws_sdk_s3::error::ProvideErrorMetadata;
     use aws_sdk_s3::types::{
         AccelerateConfiguration, BucketAccelerateStatus, BucketLoggingStatus, IndexDocument, LoggingEnabled, Payer,
         RequestPaymentConfiguration, WebsiteConfiguration,
     };
-    use http::Method;
-    use http::header::CONTENT_TYPE;
     use serial_test::serial;
+    use std::path::PathBuf;
+    use std::process::Command;
     use tracing::info;
 
+    fn awscurl_binary_path() -> PathBuf {
+        std::env::var_os("AWSCURL_PATH")
+            .map(PathBuf::from)
+            .unwrap_or_else(|| PathBuf::from("awscurl"))
+    }
+
+    fn awscurl_available() -> bool {
+        Command::new(awscurl_binary_path()).arg("--version").output().is_ok()
+    }
+
+    fn execute_s3_awscurl(
+        method: &str,
+        url: &str,
+        access_key: &str,
+        secret_key: &str,
+    ) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
+        let output = Command::new(awscurl_binary_path())
+            .args([
+                "--service",
+                "s3",
+                "--region",
+                "us-east-1",
+                "--access_key",
+                access_key,
+                "--secret_key",
+                secret_key,
+                "-i",
+                "-X",
+                method,
+                url,
+            ])
+            .output()?;
+        if !output.status.success() {
+            let stderr = String::from_utf8_lossy(&output.stderr);
+            let stdout = String::from_utf8_lossy(&output.stdout);
+            return Err(format!("awscurl failed: stderr='{stderr}', stdout='{stdout}'").into());
+        }
+        Ok(String::from_utf8_lossy(&output.stdout).to_string())
+    }
+
+    fn parse_status(raw: &str) -> Option<u16> {
+        raw.lines()
+            .filter_map(|line| {
+                if line.starts_with("HTTP/") {
+                    line.split_whitespace().nth(1)?.parse::<u16>().ok()
+                } else {
+                    None
+                }
+            })
+            .next_back()
+    }
+
+    fn parse_body(raw: &str) -> String {
+        if let Some(pos) = raw.rfind("\r\n\r\n") {
+            return raw[pos + 4..].to_string();
+        }
+        if let Some(pos) = raw.rfind("\n\n") {
+            return raw[pos + 2..].to_string();
+        }
+        String::new()
+    }
+
+    fn parse_headers(raw: &str) -> String {
+        let start = raw.rfind("HTTP/").unwrap_or(0);
+        let tail = &raw[start..];
+        if let Some(pos) = tail.find("\r\n\r\n") {
+            return tail[..pos].to_string();
+        }
+        if let Some(pos) = tail.find("\n\n") {
+            return tail[..pos].to_string();
+        }
+        tail.to_string()
+    }
+
     #[tokio::test]
     #[serial]
     async fn test_dummy_bucket_compatibility_endpoints() {
@@ -396,6 +470,10 @@ mod tests {
     async fn test_dummy_bucket_endpoints_http_contracts() {
         init_logging();
         info!("Starting test: dummy-compat bucket API HTTP contracts");
+        if !awscurl_available() {
+            info!("Skipping test_dummy_bucket_endpoints_http_contracts: awscurl binary not found");
+            return;
+        }
 
         let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
         env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
@@ -410,112 +488,56 @@ mod tests {
             .await
             .expect("Failed to create bucket");
 
-        let logging_response = signed_s3_request(
-            Method::GET,
-            &format!("{}/{bucket}?logging=", env.url),
-            None,
-            None,
-            &env.access_key,
-            &env.secret_key,
-        )
-        .await
-        .expect("GetBucketLogging HTTP request failed");
-        assert_eq!(logging_response.status(), 200, "GetBucketLogging should return 200");
-        let logging_body = logging_response
-            .text()
-            .await
-            .expect("Failed to read GetBucketLogging response body");
+        let logging_raw = execute_s3_awscurl("GET", &format!("{}/{bucket}?logging=", env.url), &env.access_key, &env.secret_key)
+            .expect("GetBucketLogging HTTP request failed");
+        assert_eq!(parse_status(&logging_raw), Some(200), "GetBucketLogging should return 200");
+        let logging_body = parse_body(&logging_raw);
         assert!(
             logging_body.contains("<BucketLoggingStatus"),
             "GetBucketLogging response should contain BucketLoggingStatus XML, got: {logging_body}"
         );
 
-        let accel_response = signed_s3_request(
-            Method::GET,
-            &format!("{}/{bucket}?accelerate=", env.url),
-            None,
-            None,
-            &env.access_key,
-            &env.secret_key,
-        )
-        .await
-        .expect("GetBucketAccelerateConfiguration HTTP request failed");
-        assert_eq!(accel_response.status(), 200, "GetBucketAccelerateConfiguration should return 200");
-        let accel_body = accel_response
-            .text()
-            .await
-            .expect("Failed to read GetBucketAccelerateConfiguration response body");
+        let accel_raw = execute_s3_awscurl("GET", &format!("{}/{bucket}?accelerate=", env.url), &env.access_key, &env.secret_key)
+            .expect("GetBucketAccelerateConfiguration HTTP request failed");
+        assert_eq!(parse_status(&accel_raw), Some(200), "GetBucketAccelerateConfiguration should return 200");
+        let accel_body = parse_body(&accel_raw);
         assert!(
             accel_body.contains("<AccelerateConfiguration"),
             "GetBucketAccelerateConfiguration response should contain AccelerateConfiguration XML, got: {accel_body}"
         );
 
-        let payment_response = signed_s3_request(
-            Method::GET,
-            &format!("{}/{bucket}?requestPayment=", env.url),
-            None,
-            None,
-            &env.access_key,
-            &env.secret_key,
-        )
-        .await
-        .expect("GetBucketRequestPayment HTTP request failed");
-        assert_eq!(payment_response.status(), 200, "GetBucketRequestPayment should return 200");
-        let payment_body = payment_response
-            .text()
-            .await
-            .expect("Failed to read GetBucketRequestPayment response body");
+        let payment_raw =
+            execute_s3_awscurl("GET", &format!("{}/{bucket}?requestPayment=", env.url), &env.access_key, &env.secret_key)
+                .expect("GetBucketRequestPayment HTTP request failed");
+        assert_eq!(parse_status(&payment_raw), Some(200), "GetBucketRequestPayment should return 200");
+        let payment_body = parse_body(&payment_raw);
         assert!(
             payment_body.contains("<Payer>BucketOwner</Payer>"),
             "GetBucketRequestPayment should return BucketOwner payer, got: {payment_body}"
         );
 
-        let website_response = signed_s3_request(
-            Method::GET,
-            &format!("{}/{bucket}?website=", env.url),
-            None,
-            None,
-            &env.access_key,
-            &env.secret_key,
-        )
-        .await
-        .expect("GetBucketWebsite HTTP request failed");
+        let website_raw = execute_s3_awscurl("GET", &format!("{}/{bucket}?website=", env.url), &env.access_key, &env.secret_key)
+            .expect("GetBucketWebsite HTTP request failed");
         assert_eq!(
-            website_response.status(),
-            404,
+            parse_status(&website_raw),
+            Some(404),
             "GetBucketWebsite should return 404 when website config is absent"
         );
-        let website_content_type = website_response
-            .headers()
-            .get(CONTENT_TYPE)
-            .expect("GetBucketWebsite response should include Content-Type")
-            .to_str()
-            .expect("GetBucketWebsite Content-Type should be valid ASCII")
-            .to_ascii_lowercase();
+        let website_content_type = parse_headers(&website_raw).to_ascii_lowercase();
         assert!(
-            website_content_type.contains("xml"),
+            website_content_type.contains("content-type:") && website_content_type.contains("xml"),
             "GetBucketWebsite error response should be XML, got content-type: {website_content_type}"
         );
-        let website_body = website_response
-            .text()
-            .await
-            .expect("Failed to read GetBucketWebsite response body");
+        let website_body = parse_body(&website_raw);
         assert!(
             website_body.contains("<Code>NoSuchWebsiteConfiguration</Code>"),
             "GetBucketWebsite should return NoSuchWebsiteConfiguration code, got: {website_body}"
         );
 
-        let delete_response = signed_s3_request(
-            Method::DELETE,
-            &format!("{}/{bucket}?website=", env.url),
-            None,
-            None,
-            &env.access_key,
-            &env.secret_key,
-        )
-        .await
-        .expect("DeleteBucketWebsite HTTP request failed");
-        assert_eq!(delete_response.status(), 204, "DeleteBucketWebsite should return 204");
+        let delete_raw =
+            execute_s3_awscurl("DELETE", &format!("{}/{bucket}?website=", env.url), &env.access_key, &env.secret_key)
+                .expect("DeleteBucketWebsite HTTP request failed");
+        assert_eq!(parse_status(&delete_raw), Some(204), "DeleteBucketWebsite should return 204");
 
         env.stop_server();
     }

crates/e2e_test/src/bucket_stats_regression_test.rs

@@ -1,295 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression tests for bucket statistics and data usage accuracy.
-//!
-//! Covers the recurring pattern where bucket statistics (object count, size)
-//! show stale/incorrect values, remain at 0, or oscillate between complete,
-//! partial, and zero. This has regressed 10+ times.
-//!
-//! ## Regression Issues
-//!
-//! - rustfs#5615: bucket statistics remain unchanged after data expiration
-//! - rustfs#5008: Admin usage reports only one pool
-//! - rustfs#5116: Admin usage reports stale 0/0 for non-empty bucket after upgrade
-//! - rustfs#5055: console object count and size still loading
-//! - rustfs#5010: Storage usage info changed abnormally
-//! - rustfs#3662: Incorrect bucket, object count and size
-//! - rustfs#3898: DataUsageInfo undercounts versioned bucket versions
-//! - rustfs#1012: Object count in the console doesn't change
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{FAST_DATA_USAGE_SCANNER_ENV, RustFSTestEnvironment, awscurl_get, init_logging};
-    use aws_sdk_s3::primitives::ByteStream;
-    use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
-    use rustfs_data_usage::DataUsageInfo;
-    use serial_test::serial;
-    use std::error::Error;
-    use tokio::time::{Duration, sleep};
-    use tracing::info;
-
-    type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-    async fn get_data_usage(env: &RustFSTestEnvironment) -> Result<DataUsageInfo, Box<dyn Error + Send + Sync>> {
-        let url = format!("{}/rustfs/admin/v3/datausageinfo", env.url);
-        let resp = awscurl_get(&url, &env.access_key, &env.secret_key).await?;
-        Ok(serde_json::from_str(&resp)?)
-    }
-
-    /// RT-09: Verify bucket object count updates after PUT.
-    ///
-    /// Regression pattern: bucket stats remain at 0 after objects are uploaded
-    /// (rustfs#5055, rustfs#1012).
-    ///
-    /// Steps:
-    /// 1. Create a bucket
-    /// 2. Upload 10 objects
-    /// 3. Query admin data usage API
-    /// 4. Verify object count > 0
-    #[tokio::test]
-    #[serial]
-    async fn test_bucket_object_count_updates_after_put() -> TestResult {
-        init_logging();
-        info!("RT-09: bucket object count updates after PUT");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], FAST_DATA_USAGE_SCANNER_ENV)
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt09-stats-put";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Upload 10 objects
-        for i in 0..10 {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(format!("stat-obj-{i:04}.txt"))
-                .body(ByteStream::from_static(b"statistical data"))
-                .send()
-                .await
-                .expect("put object");
-        }
-
-        // Wait for scanner to process (up to 90 seconds)
-        let mut found_nonzero = false;
-        let mut last_query_error = None;
-        for attempt in 0..18 {
-            sleep(Duration::from_secs(5)).await;
-
-            let usage = match get_data_usage(&env).await {
-                Ok(usage) => {
-                    last_query_error = None;
-                    usage
-                }
-                Err(err) => {
-                    last_query_error = Some(err.to_string());
-                    continue;
-                }
-            };
-            if let Some(bucket_usage) = usage.buckets_usage.get(bucket) {
-                info!("  attempt {attempt}: objectsCount = {}", bucket_usage.objects_count);
-                if bucket_usage.objects_count >= 10 {
-                    found_nonzero = true;
-                    break;
-                }
-            }
-        }
-
-        assert!(
-            found_nonzero,
-            "RT-09 FAIL: bucket object count did not update after PUT 10 objects (regression: stats stuck at 0); last query error: {}",
-            last_query_error.as_deref().unwrap_or("none")
-        );
-
-        info!("RT-09 PASS: bucket object count updates after PUT");
-        Ok(())
-    }
-
-    /// RT-09b: Verify bucket stats update after DELETE.
-    ///
-    /// Regression pattern: stats remain unchanged after objects are deleted
-    /// (rustfs#5615).
-    #[tokio::test]
-    #[serial]
-    async fn test_bucket_object_count_updates_after_delete() -> TestResult {
-        init_logging();
-        info!("RT-09b: bucket object count updates after DELETE");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], FAST_DATA_USAGE_SCANNER_ENV)
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt09b-stats-delete";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Upload 5 objects
-        for i in 0..5 {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(format!("del-stat-{i}.txt"))
-                .body(ByteStream::from_static(b"data"))
-                .send()
-                .await
-                .expect("put object");
-        }
-
-        let mut found_nonzero = false;
-        for attempt in 0..18 {
-            sleep(Duration::from_secs(5)).await;
-
-            if let Ok(usage) = get_data_usage(&env).await
-                && let Some(bucket_usage) = usage.buckets_usage.get(bucket)
-            {
-                info!("  baseline attempt {attempt}: objectsCount = {}", bucket_usage.objects_count);
-                if bucket_usage.objects_count >= 5 {
-                    found_nonzero = true;
-                    break;
-                }
-            }
-        }
-        assert!(found_nonzero, "RT-09b setup failed: scanner did not observe the 5 uploaded objects");
-
-        // Delete all objects
-        for i in 0..5 {
-            client
-                .delete_object()
-                .bucket(bucket)
-                .key(format!("del-stat-{i}.txt"))
-                .send()
-                .await
-                .expect("delete object");
-        }
-
-        // Wait for scanner to update stats (up to 90 seconds)
-        let mut found_zero = false;
-        let mut last_query_error = None;
-        for attempt in 0..18 {
-            sleep(Duration::from_secs(5)).await;
-
-            let usage = match get_data_usage(&env).await {
-                Ok(usage) => {
-                    last_query_error = None;
-                    usage
-                }
-                Err(err) => {
-                    last_query_error = Some(err.to_string());
-                    continue;
-                }
-            };
-            if let Some(bucket_usage) = usage.buckets_usage.get(bucket) {
-                info!("  attempt {attempt}: objectsCount = {}", bucket_usage.objects_count);
-                if bucket_usage.objects_count == 0 {
-                    found_zero = true;
-                    break;
-                }
-            }
-        }
-
-        assert!(
-            found_zero,
-            "RT-09b FAIL: bucket object count did not update to 0 after deleting all objects (regression rustfs#5615); last query error: {}",
-            last_query_error.as_deref().unwrap_or("none")
-        );
-
-        info!("RT-09b PASS: bucket object count updates to 0 after DELETE");
-        Ok(())
-    }
-
-    /// RT-09c: Verify versioned bucket stats count all versions.
-    ///
-    /// Regression pattern: DataUsageInfo undercounts versioned bucket versions
-    /// and delete markers (rustfs#3898).
-    #[tokio::test]
-    #[serial]
-    async fn test_versioned_bucket_stats_count_all_versions() -> TestResult {
-        init_logging();
-        info!("RT-09c: versioned bucket stats count all versions");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt09c-versioned-stats";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        client
-            .put_bucket_versioning()
-            .bucket(bucket)
-            .versioning_configuration(
-                VersioningConfiguration::builder()
-                    .status(BucketVersioningStatus::Enabled)
-                    .build(),
-            )
-            .send()
-            .await
-            .expect("enable versioning");
-
-        // Create 3 versions of the same object
-        for i in 0..3 {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key("multi-version.txt")
-                .body(ByteStream::from(format!("version-{i}").into_bytes()))
-                .send()
-                .await
-                .expect("put version");
-        }
-
-        // Create a delete marker
-        client
-            .delete_object()
-            .bucket(bucket)
-            .key("multi-version.txt")
-            .send()
-            .await
-            .expect("create delete marker");
-
-        // Verify versions via API (immediate, no scanner wait)
-        let versions = client
-            .list_object_versions()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list versions");
-
-        assert_eq!(
-            versions.versions().len(),
-            3,
-            "RT-09c FAIL: expected 3 versions, found {}",
-            versions.versions().len()
-        );
-        assert_eq!(
-            versions.delete_markers().len(),
-            1,
-            "RT-09c FAIL: expected 1 delete marker, found {}",
-            versions.delete_markers().len()
-        );
-
-        info!("RT-09c PASS: versioned bucket correctly tracks all versions and delete markers");
-        Ok(())
-    }
-}

crates/e2e_test/src/chaos.rs

@@ -40,8 +40,6 @@ use http::header::{CONTENT_TYPE, HOST};
 use rustfs_signer::constants::UNSIGNED_PAYLOAD;
 use rustfs_signer::sign_v4;
 use s3s::Body;
-use sha2::{Digest, Sha256};
-use std::collections::{BTreeMap, BTreeSet};
 use std::error::Error;
 use std::path::{Path, PathBuf};
 use tracing::info;
@@ -50,62 +48,6 @@ use walkdir::WalkDir;
 
 type ChaosResult<T> = Result<T, Box<dyn Error + Send + Sync>>;
 
-/// Physical `xl.meta` and shard-file census for one object version on one disk.
-///
-/// A successful S3 GET only proves that a quorum can serve an object. Replacement
-/// tests need this lower-level record to prove that the rebuilt target holds the
-/// `xl.meta` selected for a specific version and every `part.N` it declares.
-#[derive(Clone, Debug, Eq, PartialEq)]
-pub(crate) struct VersionShardCensus {
-    pub version_id: Option<String>,
-    pub has_xl_meta: bool,
-    pub data_dir: Option<String>,
-    pub erasure_index: Option<usize>,
-    pub expected_part_numbers: BTreeSet<usize>,
-    pub present_part_fingerprints: BTreeMap<usize, PartShardFingerprint>,
-    pub inline_data_fingerprint: Option<PartShardFingerprint>,
-}
-
-#[derive(Clone, Debug, Eq, PartialEq)]
-pub(crate) struct PartShardFingerprint {
-    pub size: u64,
-    pub sha256: String,
-}
-
-impl VersionShardCensus {
-    pub(crate) fn is_complete(&self) -> bool {
-        self.has_xl_meta
-            && self.expected_part_numbers.len() == self.present_part_fingerprints.len()
-            && self
-                .expected_part_numbers
-                .iter()
-                .all(|part_number| self.present_part_fingerprints.contains_key(part_number))
-    }
-
-    pub(crate) fn matches_manifest(&self, manifest: &Self) -> bool {
-        self.version_id == manifest.version_id
-            && self.is_complete()
-            && manifest.is_complete()
-            && self.data_dir == manifest.data_dir
-            && self.erasure_index == manifest.erasure_index
-            && self.expected_part_numbers == manifest.expected_part_numbers
-            && self.present_part_fingerprints == manifest.present_part_fingerprints
-            && self.inline_data_fingerprint == manifest.inline_data_fingerprint
-    }
-}
-
-fn sha256_hex(data: &[u8]) -> String {
-    let digest = Sha256::digest(data);
-    digest.iter().map(|byte| format!("{byte:02x}")).collect()
-}
-
-fn shard_fingerprint(data: &[u8]) -> ChaosResult<PartShardFingerprint> {
-    Ok(PartShardFingerprint {
-        size: u64::try_from(data.len())?,
-        sha256: sha256_hex(data),
-    })
-}
-
 /// Single-node RustFS server with `disk_count` local volume directories that
 /// can be faulted individually while the server is running.
 pub struct DiskFaultHarness {
@@ -277,93 +219,6 @@ impl DiskFaultHarness {
     pub fn object_metadata_exists_on_disk(&self, disk_index: usize, bucket: &str, key: &str) -> bool {
         self.disks[disk_index].join(bucket).join(key).join("xl.meta").is_file()
     }
-
-    /// Census the physical files selected by `version_id` on one disk.
-    ///
-    /// Missing metadata and missing shard files are represented in the returned
-    /// census rather than as an error so callers can poll replacement progress.
-    /// Invalid metadata or an unknown requested version remains an error: treating
-    /// either as an incomplete rebuild would hide corruption or a wrong-version
-    /// recovery result.
-    pub(crate) fn census_object_version(
-        &self,
-        disk_index: usize,
-        bucket: &str,
-        key: &str,
-        version_id: Option<&str>,
-    ) -> ChaosResult<VersionShardCensus> {
-        census_object_version_on_disk(&self.disks[disk_index], bucket, key, version_id)
-    }
-}
-
-/// Census one physical object version without requiring a single-node harness.
-/// Cluster replacement tests use the same evidence as the disk-fault tests.
-pub(crate) fn census_object_version_on_disk(
-    disk: &Path,
-    bucket: &str,
-    key: &str,
-    version_id: Option<&str>,
-) -> ChaosResult<VersionShardCensus> {
-    let version_id = version_id.map(str::to_owned);
-    let object_dir = disk.join(bucket).join(key);
-    let meta_path = object_dir.join("xl.meta");
-    if !meta_path.is_file() {
-        return Ok(VersionShardCensus {
-            version_id,
-            has_xl_meta: false,
-            data_dir: None,
-            erasure_index: None,
-            expected_part_numbers: BTreeSet::new(),
-            present_part_fingerprints: BTreeMap::new(),
-            inline_data_fingerprint: None,
-        });
-    }
-
-    let metadata = rustfs_filemeta::FileMeta::load(&std::fs::read(&meta_path)?)?;
-    let file_info = metadata.into_fileinfo(bucket, key, version_id.as_deref().unwrap_or_default(), true, false, true)?;
-    let expected_part_numbers = if file_info.inline_data() {
-        BTreeSet::new()
-    } else {
-        file_info.parts.iter().map(|part| part.number).collect()
-    };
-    let data_dir = file_info.data_dir.map(|id| id.to_string());
-    let erasure_index = Some(file_info.erasure.index);
-    let inline_data_fingerprint = file_info.data.as_deref().map(shard_fingerprint).transpose()?;
-    let part_dir = data_dir.as_ref().map_or_else(|| object_dir.clone(), |id| object_dir.join(id));
-    let present_part_fingerprints = match std::fs::read_dir(&part_dir) {
-        Ok(entries) => {
-            let mut fingerprints = BTreeMap::new();
-            for entry in entries {
-                let entry = entry?;
-                if !entry.file_type()?.is_file() {
-                    continue;
-                }
-                let file_name = entry.file_name();
-                let Some(part_number) = file_name
-                    .to_str()
-                    .and_then(|name| name.strip_prefix("part."))
-                    .and_then(|number| number.parse::<usize>().ok())
-                else {
-                    continue;
-                };
-                let data = std::fs::read(entry.path())?;
-                fingerprints.insert(part_number, shard_fingerprint(&data)?);
-            }
-            fingerprints
-        }
-        Err(error) if error.kind() == std::io::ErrorKind::NotFound => BTreeMap::new(),
-        Err(error) => return Err(error.into()),
-    };
-
-    Ok(VersionShardCensus {
-        version_id,
-        has_xl_meta: true,
-        data_dir,
-        erasure_index,
-        expected_part_numbers,
-        present_part_fingerprints,
-        inline_data_fingerprint,
-    })
 }
 
 /// `POST` a signed (SigV4, service `s3`) admin request without relying on the
@@ -402,43 +257,3 @@ pub async fn signed_admin_post(url: &str, body: Option<&str>, access_key: &str,
 
     Ok(body)
 }
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    fn complete_census() -> VersionShardCensus {
-        VersionShardCensus {
-            version_id: Some("version".to_string()),
-            has_xl_meta: true,
-            data_dir: Some("data-dir".to_string()),
-            erasure_index: Some(3),
-            expected_part_numbers: BTreeSet::from([1]),
-            present_part_fingerprints: BTreeMap::from([(1, shard_fingerprint(b"part").unwrap())]),
-            inline_data_fingerprint: None,
-        }
-    }
-
-    #[test]
-    fn shard_fingerprint_uses_physical_length_and_sha256() {
-        assert_eq!(
-            shard_fingerprint(b"abc").unwrap(),
-            PartShardFingerprint {
-                size: 3,
-                sha256: "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad".to_string(),
-            }
-        );
-    }
-
-    #[test]
-    fn manifest_requires_matching_inline_payload() {
-        let mut expected = complete_census();
-        expected.expected_part_numbers.clear();
-        expected.present_part_fingerprints.clear();
-        expected.inline_data_fingerprint = Some(shard_fingerprint(b"expected").unwrap());
-        let mut changed = expected.clone();
-        changed.inline_data_fingerprint = Some(shard_fingerprint(b"changed").unwrap());
-        assert!(expected.matches_manifest(&expected));
-        assert!(!changed.matches_manifest(&expected));
-    }
-}

crates/e2e_test/src/common.rs

@@ -47,44 +47,11 @@ use walkdir::WalkDir;
 pub const DEFAULT_ACCESS_KEY: &str = "rustfsadmin";
 pub const DEFAULT_SECRET_KEY: &str = "rustfsadmin";
 pub const ENV_RUSTFS_BUILD_FEATURES: &str = "RUSTFS_BUILD_FEATURES";
-pub(crate) const FAST_DATA_USAGE_SCANNER_ENV: &[(&str, &str)] =
-    &[("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_SCANNER_START_DELAY_SECS", "0")];
 pub const TEST_BUCKET: &str = "e2e-test-bucket";
 const RUSTFS_FULL_FEATURE: &str = "full";
 
-fn capture_log_path(log_dir: &Path, temp_dir: &str) -> Option<PathBuf> {
-    let temp_name = Path::new(temp_dir).file_name()?.to_string_lossy();
-    Some(log_dir.join(format!("{temp_name}.log")))
-}
-
-fn configured_capture_log_path(temp_dir: &str) -> Option<String> {
-    let log_dir = std::env::var_os("RUSTFS_E2E_LOG_DIR")?;
-    if stdfs::create_dir_all(&log_dir).is_err() {
-        warn!(?log_dir, "failed to create configured E2E server log directory");
-        return None;
-    }
-
-    capture_log_path(Path::new(&log_dir), temp_dir).map(|path| path.to_string_lossy().into_owned())
-}
-
-fn capture_command_logs(command: &mut Command, log_path: Option<&str>) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    let Some(log_path) = log_path else {
-        return Ok(());
-    };
-    let file = stdfs::OpenOptions::new().create(true).append(true).open(log_path)?;
-    let stderr_file = file.try_clone()?;
-    command.stdout(Stdio::from(file)).stderr(Stdio::from(stderr_file));
-    Ok(())
-}
-
-pub(crate) fn build_test_s3_config(
-    endpoint_url: &str,
-    access_key: &str,
-    secret_key: &str,
-    session_token: Option<&str>,
-    provider_name: &'static str,
-) -> Config {
-    let credentials = Credentials::new(access_key, secret_key, session_token.map(str::to_owned), None, provider_name);
+fn build_test_s3_config(endpoint_url: &str, access_key: &str, secret_key: &str, provider_name: &'static str) -> Config {
+    let credentials = Credentials::new(access_key, secret_key, None, None, provider_name);
     let mut config = Config::builder()
         .credentials_provider(credentials)
         .region(Region::new("us-east-1"))
@@ -99,33 +66,6 @@ pub(crate) fn build_test_s3_config(
     config.build()
 }
 
-pub(crate) fn build_test_sts_client(
-    endpoint_url: &str,
-    access_key: &str,
-    secret_key: &str,
-    session_token: Option<&str>,
-    provider_name: &'static str,
-) -> aws_sdk_sts::Client {
-    let mut config = aws_sdk_sts::Config::builder()
-        .credentials_provider(aws_sdk_sts::config::Credentials::new(
-            access_key,
-            secret_key,
-            session_token.map(str::to_owned),
-            None,
-            provider_name,
-        ))
-        .region(aws_sdk_sts::config::Region::new("us-east-1"))
-        .endpoint_url(endpoint_url)
-        .retry_config(aws_sdk_sts::config::retry::RetryConfig::standard().with_max_attempts(1))
-        .behavior_version_latest();
-
-    if endpoint_url.starts_with("http://") {
-        config = config.http_client(SmithyHttpClientBuilder::new().build_http());
-    }
-
-    aws_sdk_sts::Client::from_conf(config.build())
-}
-
 pub fn workspace_root() -> PathBuf {
     let mut path = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
     path.pop(); // e2e_test
@@ -140,57 +80,6 @@ pub fn local_http_client() -> HttpClient {
         .expect("failed to build local reqwest client")
 }
 
-pub(crate) async fn signed_s3_request(
-    method: http::Method,
-    url: &str,
-    body: Option<String>,
-    content_type: Option<&str>,
-    access_key: &str,
-    secret_key: &str,
-) -> Result<reqwest::Response, Box<dyn std::error::Error + Send + Sync>> {
-    signed_s3_request_with_session_token(method, url, body, content_type, access_key, secret_key, None).await
-}
-
-async fn signed_s3_request_with_session_token(
-    method: http::Method,
-    url: &str,
-    body: Option<String>,
-    content_type: Option<&str>,
-    access_key: &str,
-    secret_key: &str,
-    session_token: Option<&str>,
-) -> Result<reqwest::Response, Box<dyn std::error::Error + Send + Sync>> {
-    let uri = url.parse::<http::Uri>()?;
-    let authority = uri.authority().ok_or("S3 URL missing authority")?.to_string();
-    let mut request = http::Request::builder()
-        .method(method.clone())
-        .uri(uri)
-        .header(HOST, authority)
-        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
-    if let Some(content_type) = content_type {
-        request = request.header(CONTENT_TYPE, content_type);
-    }
-
-    let content_length = i64::try_from(body.as_ref().map_or(0, String::len)).map_err(|_| "S3 request body is too large")?;
-    let signed = sign_v4(
-        request.body(Body::empty())?,
-        content_length,
-        access_key,
-        secret_key,
-        session_token.unwrap_or_default(),
-        "us-east-1",
-    );
-
-    let mut request = local_http_client().request(method, url);
-    for (name, value) in signed.headers() {
-        request = request.header(name, value);
-    }
-    if let Some(body) = body {
-        request = request.body(body);
-    }
-    Ok(request.send().await?)
-}
-
 /// Signs and sends an admin HTTP request with the given credentials.
 pub(crate) async fn admin_request(
     base_url: &str,
@@ -199,23 +88,30 @@ pub(crate) async fn admin_request(
     body: Option<String>,
     access_key: &str,
     secret_key: &str,
-) -> Result<(StatusCode, String), Box<dyn std::error::Error + Send + Sync>> {
-    admin_request_with_session_token(base_url, method, path_and_query, body, access_key, secret_key, None).await
-}
-
-pub(crate) async fn admin_request_with_session_token(
-    base_url: &str,
-    method: http::Method,
-    path_and_query: &str,
-    body: Option<String>,
-    access_key: &str,
-    secret_key: &str,
-    session_token: Option<&str>,
 ) -> Result<(StatusCode, String), Box<dyn std::error::Error + Send + Sync>> {
     let url = format!("{base_url}{path_and_query}");
-    let content_type = body.as_ref().map(|_| "application/json");
-    let response =
-        signed_s3_request_with_session_token(method, &url, body, content_type, access_key, secret_key, session_token).await?;
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
+    let mut request = http::Request::builder()
+        .method(method.clone())
+        .uri(uri)
+        .header(HOST, authority)
+        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
+    if body.is_some() {
+        request = request.header(CONTENT_TYPE, "application/json");
+    }
+
+    let content_length = i64::try_from(body.as_ref().map_or(0, String::len)).map_err(|_| "admin request body is too large")?;
+    let signed = sign_v4(request.body(Body::empty())?, content_length, access_key, secret_key, "", "us-east-1");
+
+    let mut request = local_http_client().request(method, &url);
+    for (name, value) in signed.headers() {
+        request = request.header(name, value);
+    }
+    if let Some(body) = body {
+        request = request.body(body);
+    }
+    let response = request.send().await?;
     let status = response.status();
     let body = response.text().await?;
     Ok((status, body))
@@ -465,7 +361,6 @@ impl RustFSTestEnvironment {
     pub async fn new() -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
         let temp_dir = format!("/tmp/rustfs_e2e_test_{}", Uuid::new_v4());
         fs::create_dir_all(&temp_dir).await?;
-        let capture_log_path = configured_capture_log_path(&temp_dir);
 
         // Use a unique port for each test environment
         let port = Self::find_available_port().await?;
@@ -479,7 +374,7 @@ impl RustFSTestEnvironment {
             access_key: DEFAULT_ACCESS_KEY.to_string(),
             secret_key: DEFAULT_SECRET_KEY.to_string(),
             process: None,
-            capture_log_path,
+            capture_log_path: None,
         })
     }
 
@@ -487,7 +382,6 @@ impl RustFSTestEnvironment {
     pub async fn with_address(address: &str) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
         let temp_dir = format!("/tmp/rustfs_e2e_test_{}", Uuid::new_v4());
         fs::create_dir_all(&temp_dir).await?;
-        let capture_log_path = configured_capture_log_path(&temp_dir);
 
         let url = format!("http://{address}");
 
@@ -498,7 +392,7 @@ impl RustFSTestEnvironment {
             access_key: DEFAULT_ACCESS_KEY.to_string(),
             secret_key: DEFAULT_SECRET_KEY.to_string(),
             process: None,
-            capture_log_path,
+            capture_log_path: None,
         })
     }
 
@@ -567,7 +461,13 @@ impl RustFSTestEnvironment {
         for (key, value) in extra_env {
             command.env(key, value);
         }
-        capture_command_logs(&mut command, self.capture_log_path.as_deref())?;
+        // Optionally capture the child's stdout+stderr to a file so the test can
+        // grep server logs (e.g. to confirm which GET reader path was taken).
+        if let Some(log_path) = &self.capture_log_path {
+            let file = stdfs::OpenOptions::new().create(true).append(true).open(log_path)?;
+            let stderr_file = file.try_clone()?;
+            command.stdout(Stdio::from(file)).stderr(Stdio::from(stderr_file));
+        }
         let process = command.args(&args).spawn()?;
 
         self.process = Some(process);
@@ -647,12 +547,7 @@ impl RustFSTestEnvironment {
 
     /// Create an AWS S3 client configured for this RustFS instance
     pub fn create_s3_client(&self) -> Client {
-        self.create_s3_client_with_credentials(&self.access_key, &self.secret_key)
-    }
-
-    /// Create an AWS S3 client with explicit credentials for this RustFS instance.
-    pub fn create_s3_client_with_credentials(&self, access_key: &str, secret_key: &str) -> Client {
-        Client::from_conf(build_test_s3_config(&self.url, access_key, secret_key, None, "e2e-test"))
+        Client::from_conf(build_test_s3_config(&self.url, &self.access_key, &self.secret_key, "e2e-test"))
     }
 
     /// Create test bucket
@@ -1055,7 +950,6 @@ pub struct RustFSTestClusterEnvironment {
     pub secret_key: String,
     pub extra_env: Vec<(String, String)>,
     pub node_extra_env: Vec<Vec<(String, String)>>,
-    pub node_capture_log_paths: Vec<Option<String>>,
     pub topology: ClusterTopology,
 }
 
@@ -1155,7 +1049,6 @@ impl RustFSTestClusterEnvironment {
             secret_key: "rustfs-cluster-test-secret".to_string(),
             extra_env,
             node_extra_env: vec![Vec::new(); topology.node_count],
-            node_capture_log_paths: vec![None; topology.node_count],
             topology,
         })
     }
@@ -1185,20 +1078,6 @@ impl RustFSTestClusterEnvironment {
         Ok(())
     }
 
-    /// Capture stdout+stderr for a single cluster node process.
-    pub fn set_node_capture_log_path<P>(
-        &mut self,
-        node_idx: usize,
-        path: P,
-    ) -> Result<(), Box<dyn std::error::Error + Send + Sync>>
-    where
-        P: Into<String>,
-    {
-        self.ensure_node_index(node_idx)?;
-        self.node_capture_log_paths[node_idx] = Some(path.into());
-        Ok(())
-    }
-
     fn ensure_node_index(&self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         if node_idx >= self.nodes.len() {
             return Err(format!("node_idx {node_idx} is invalid").into());
@@ -1288,7 +1167,6 @@ impl RustFSTestClusterEnvironment {
             for (key, value) in &self.node_extra_env[i] {
                 command.env(key, value);
             }
-            capture_command_logs(&mut command, self.node_capture_log_paths[i].as_deref())?;
 
             let process = command.current_dir(&node.data_dir).spawn()?;
 
@@ -1315,7 +1193,6 @@ impl RustFSTestClusterEnvironment {
 
         let binary_path = rustfs_binary_path();
         let volumes_arg = self.build_volumes_arg();
-        let log_path = self.node_capture_log_paths[node_idx].clone();
         let node = &mut self.nodes[node_idx];
         info!("Starting cluster node {} on {}", node_idx, node.address);
 
@@ -1334,7 +1211,6 @@ impl RustFSTestClusterEnvironment {
         for (key, value) in &self.node_extra_env[node_idx] {
             command.env(key, value);
         }
-        capture_command_logs(&mut command, log_path.as_deref())?;
 
         let process = command.current_dir(&node.data_dir).spawn()?;
         node.process = Some(process);
@@ -1403,7 +1279,6 @@ impl RustFSTestClusterEnvironment {
             &self.nodes[node_idx].url,
             &self.access_key,
             &self.secret_key,
-            None,
             "cluster-test",
         )))
     }
@@ -1517,14 +1392,6 @@ mod tests {
         assert_eq!(normalize_rustfs_build_features(" , "), None);
     }
 
-    #[test]
-    fn capture_log_path_uses_temp_directory_basename() {
-        assert_eq!(
-            capture_log_path(Path::new("/tmp/e2e-logs"), "/tmp/rustfs_e2e_test_abc"),
-            Some(PathBuf::from("/tmp/e2e-logs/rustfs_e2e_test_abc.log"))
-        );
-    }
-
     #[test]
     fn full_feature_enables_any_required_feature() {
         assert!(rustfs_build_feature_enabled(Some("full"), "sftp"));
@@ -1586,7 +1453,6 @@ mod tests {
             secret_key: DEFAULT_SECRET_KEY.to_string(),
             extra_env: Vec::new(),
             node_extra_env: vec![Vec::new(); topology.node_count],
-            node_capture_log_paths: vec![None; topology.node_count],
             topology,
         }
     }
@@ -1682,16 +1548,6 @@ mod tests {
         );
     }
 
-    #[test]
-    fn cluster_node_log_capture_supports_per_node_paths() {
-        let mut env = fake_cluster(ClusterTopology::single_pool(3));
-        env.set_node_capture_log_path(1, "/tmp/node1.log").unwrap();
-        assert_eq!(env.node_capture_log_paths[0], None);
-        assert_eq!(env.node_capture_log_paths[1], Some("/tmp/node1.log".to_string()));
-        assert_eq!(env.node_capture_log_paths[2], None);
-        assert!(env.set_node_capture_log_path(3, "/tmp/invalid.log").is_err());
-    }
-
     #[test]
     fn cluster_node_env_rejects_invalid_index() {
         let mut env = fake_cluster(ClusterTopology::single_pool(4));

crates/e2e_test/src/data_usage_test.rs

@@ -18,7 +18,7 @@ use rustfs_data_usage::DataUsageInfo;
 use serial_test::serial;
 use tokio::time::{Duration, sleep};
 
-use crate::common::{FAST_DATA_USAGE_SCANNER_ENV, RustFSTestEnvironment, TEST_BUCKET, awscurl_get, init_logging};
+use crate::common::{RustFSTestEnvironment, TEST_BUCKET, awscurl_get, init_logging};
 
 async fn get_data_usage_info(env: &RustFSTestEnvironment) -> Result<DataUsageInfo, Box<dyn std::error::Error + Send + Sync>> {
     let url = format!("{}/rustfs/admin/v3/datausageinfo", env.url);
@@ -35,26 +35,16 @@ where
     F: FnMut(&DataUsageInfo) -> bool,
 {
     let mut last_usage = DataUsageInfo::default();
-    let mut last_query_error = None;
     for _ in 0..45 {
-        match get_data_usage_info(env).await {
-            Ok(usage) => {
-                last_query_error = None;
-                if usage.buckets_usage.contains_key(bucket) && predicate(&usage) {
-                    return Ok(usage);
-                }
-                last_usage = usage;
-            }
-            Err(err) => last_query_error = Some(err.to_string()),
+        let usage = get_data_usage_info(env).await?;
+        if usage.buckets_usage.contains_key(bucket) && predicate(&usage) {
+            return Ok(usage);
         }
+        last_usage = usage;
         sleep(Duration::from_secs(2)).await;
     }
 
-    Err(format!(
-        "bucket usage did not converge for {bucket}; last usage: {last_usage:?}; last query error: {}",
-        last_query_error.as_deref().unwrap_or("none")
-    )
-    .into())
+    Err(format!("bucket usage did not converge for {bucket}; last usage: {last_usage:?}").into())
 }
 
 /// Regression test for data usage accuracy (issue #1012).
@@ -66,7 +56,7 @@ async fn data_usage_reports_all_objects() -> Result<(), Box<dyn std::error::Erro
     init_logging();
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server_with_env(vec![], FAST_DATA_USAGE_SCANNER_ENV).await?;
+    env.start_rustfs_server(vec![]).await?;
 
     let client = env.create_s3_client();
 
@@ -84,14 +74,8 @@ async fn data_usage_reports_all_objects() -> Result<(), Box<dyn std::error::Erro
             .await?;
     }
 
-    let usage = wait_for_bucket_usage(&env, TEST_BUCKET, |usage| {
-        usage
-            .buckets_usage
-            .get(TEST_BUCKET)
-            .map(|bucket_usage| usage.objects_total_count >= 1000 && bucket_usage.objects_count >= 1000)
-            .unwrap_or(false)
-    })
-    .await?;
+    // Query admin data usage API
+    let usage = get_data_usage_info(&env).await?;
 
     // Assert total object count and per-bucket count are not truncated
     let bucket_usage = usage
@@ -124,7 +108,7 @@ async fn data_usage_reports_versioned_objects_and_delete_markers() -> Result<(),
     init_logging();
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server_with_env(vec![], FAST_DATA_USAGE_SCANNER_ENV).await?;
+    env.start_rustfs_server(vec![]).await?;
 
     let client = env.create_s3_client();
     let bucket = "data-usage-versioned";
@@ -200,8 +184,8 @@ async fn data_usage_reports_versioned_objects_and_delete_markers() -> Result<(),
     assert_eq!(usage.versions_total_count, 3, "total version count should match bucket usage");
     assert_eq!(usage.delete_markers_total_count, 1, "total delete marker count should match bucket usage");
 
-    env.restart_server_preserving_data(vec![], FAST_DATA_USAGE_SCANNER_ENV)
-        .await?;
+    env.stop_server();
+    env.start_rustfs_server(vec![]).await?;
 
     let restarted_usage = wait_for_bucket_usage(&env, bucket, |usage| {
         usage

crates/e2e_test/src/delete_regression_test.rs

@@ -1,445 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression tests for object delete operations.
-//!
-//! Covers the recurring pattern where DELETE succeeds at the API level but the
-//! object remains visible in LIST, or deleted objects reappear after restart,
-//! or versioned delete operations fail with FileAccessDenied.
-//! This has regressed 15+ times across the entire release history.
-//!
-//! ## Regression Issues
-//!
-//! - rustfs#5375: delete object in a bucket list api also exist this object
-//! - rustfs#5349: The deleted bucket was rebuilt after some time
-//! - rustfs#5339: data not delete in Object Lock bucket
-//! - rustfs#5029: Node Does Not Remove Files After Reconnect to Cluster
-//! - rustfs#4978: DELETE fails with InternalError/FileAccessDenied on beta 10
-//! - rustfs#760: Cannot delete a versioned bucket
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestEnvironment, init_logging};
-    use aws_sdk_s3::primitives::ByteStream;
-    use aws_sdk_s3::types::{BucketVersioningStatus, Delete, ObjectIdentifier, VersioningConfiguration};
-    use serial_test::serial;
-    use std::error::Error;
-    use tracing::info;
-
-    type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-    /// RT-05: Verify DELETE → LIST → HEAD consistency.
-    ///
-    /// Regression pattern: DELETE returns 200 but the object remains in LIST.
-    /// Covers rustfs#5375.
-    ///
-    /// Steps:
-    /// 1. Create a bucket and upload an object
-    /// 2. Verify the object is in LIST
-    /// 3. DELETE the object
-    /// 4. Verify the object is NOT in LIST
-    /// 5. Verify HEAD returns 404
-    #[tokio::test]
-    #[serial]
-    async fn test_delete_removes_object_from_list() -> TestResult {
-        init_logging();
-        info!("RT-05: delete removes object from list");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt05-delete-consistency";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Upload an object
-        client
-            .put_object()
-            .bucket(bucket)
-            .key("to-delete.txt")
-            .body(ByteStream::from_static(b"will be deleted"))
-            .send()
-            .await
-            .expect("put object");
-
-        // Verify it appears in LIST
-        let list = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list objects before delete");
-
-        assert!(
-            list.contents()
-                .iter()
-                .map(|o| o.key().unwrap_or(""))
-                .any(|key| key == "to-delete.txt"),
-            "RT-05 FAIL: object not in LIST before delete"
-        );
-
-        // DELETE
-        client
-            .delete_object()
-            .bucket(bucket)
-            .key("to-delete.txt")
-            .send()
-            .await
-            .expect("delete object");
-
-        // Verify NOT in LIST
-        let list = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list objects after delete");
-
-        assert!(
-            !list
-                .contents()
-                .iter()
-                .map(|o| o.key().unwrap_or(""))
-                .any(|key| key == "to-delete.txt"),
-            "RT-05 FAIL: deleted object still in LIST (regression rustfs#5375)"
-        );
-
-        // Verify HEAD returns 404
-        let head = client.head_object().bucket(bucket).key("to-delete.txt").send().await;
-
-        assert!(head.is_err(), "RT-05 FAIL: HEAD on deleted object should return error, got success");
-
-        info!("RT-05 PASS: delete correctly removes object from LIST and HEAD");
-        Ok(())
-    }
-
-    /// RT-05c: Verify batch delete (DeleteObjects) consistency.
-    ///
-    /// Regression pattern: batch delete returns success but some objects
-    /// remain in LIST.
-    #[tokio::test]
-    #[serial]
-    async fn test_batch_delete_removes_all_objects() -> TestResult {
-        init_logging();
-        info!("RT-05c: batch delete removes all objects");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt05c-batch-delete";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Upload multiple objects
-        let keys: Vec<String> = (0..5).map(|i| format!("batch-{i:04}.txt")).collect();
-        for key in &keys {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(key)
-                .body(ByteStream::from_static(b"batch-delete-me"))
-                .send()
-                .await
-                .expect("put object");
-        }
-
-        // Verify all in LIST
-        let list = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list before batch delete");
-
-        assert_eq!(
-            list.contents().len(),
-            5,
-            "RT-05c FAIL: expected 5 objects before batch delete, found {}",
-            list.contents().len()
-        );
-
-        // Batch delete
-        let objects: Vec<ObjectIdentifier> = keys
-            .iter()
-            .map(|k| ObjectIdentifier::builder().key(k).build().expect("build object id"))
-            .collect();
-
-        client
-            .delete_objects()
-            .bucket(bucket)
-            .delete(Delete::builder().set_objects(Some(objects)).build().expect("build delete"))
-            .send()
-            .await
-            .expect("batch delete");
-
-        // Verify all removed
-        let list = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list after batch delete");
-
-        assert!(
-            list.contents().is_empty(),
-            "RT-05c FAIL: {} objects remain after batch delete (regression: delete objects not fully applied)",
-            list.contents().len()
-        );
-
-        info!("RT-05c PASS: batch delete removes all objects");
-        Ok(())
-    }
-
-    /// RT-05d: Verify versioned delete → permanent delete → object gone.
-    ///
-    /// Covers the pattern where permanent deletion of a specific version
-    /// fails with FileAccessDenied (rustfs#4978).
-    #[tokio::test]
-    #[serial]
-    async fn test_versioned_permanent_delete() -> TestResult {
-        init_logging();
-        info!("RT-05d: versioned permanent delete");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt05d-permanent-delete";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        client
-            .put_bucket_versioning()
-            .bucket(bucket)
-            .versioning_configuration(
-                VersioningConfiguration::builder()
-                    .status(BucketVersioningStatus::Enabled)
-                    .build(),
-            )
-            .send()
-            .await
-            .expect("enable versioning");
-
-        // Upload a single object (single version)
-        let put_resp = client
-            .put_object()
-            .bucket(bucket)
-            .key("single-version.txt")
-            .body(ByteStream::from_static(b"to-be-permanently-deleted"))
-            .send()
-            .await
-            .expect("put object");
-
-        let version_id = put_resp.version_id().expect("version ID should be present").to_string();
-
-        // Permanently delete the specific version (rustfs#4978: FileAccessDenied)
-        client
-            .delete_object()
-            .bucket(bucket)
-            .key("single-version.txt")
-            .version_id(&version_id)
-            .send()
-            .await
-            .expect("permanent delete should succeed (regression rustfs#4978)");
-
-        // Verify the object is completely gone
-        let versions = client
-            .list_object_versions()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list versions");
-
-        assert!(
-            versions.versions().is_empty(),
-            "RT-05d FAIL: version still present after permanent delete"
-        );
-
-        info!("RT-05d PASS: versioned permanent delete succeeds");
-        Ok(())
-    }
-
-    /// RT-05e: Verify delete marker + version history interaction.
-    ///
-    /// Covers the pattern where creating a delete marker and then listing
-    /// versions shows incorrect state (rustfs#760).
-    #[tokio::test]
-    #[serial]
-    async fn test_versioned_delete_marker_and_list_consistency() -> TestResult {
-        init_logging();
-        info!("RT-05e: versioned delete marker and list consistency");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt05e-dm-consistency";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        client
-            .put_bucket_versioning()
-            .bucket(bucket)
-            .versioning_configuration(
-                VersioningConfiguration::builder()
-                    .status(BucketVersioningStatus::Enabled)
-                    .build(),
-            )
-            .send()
-            .await
-            .expect("enable versioning");
-
-        // Create 3 versions
-        for i in 0..3 {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key("history.txt")
-                .body(ByteStream::from(format!("v{i}").into_bytes()))
-                .send()
-                .await
-                .expect("put version");
-        }
-
-        // Create a delete marker
-        let del = client
-            .delete_object()
-            .bucket(bucket)
-            .key("history.txt")
-            .send()
-            .await
-            .expect("delete (create marker)");
-
-        assert!(del.delete_marker().unwrap_or(false), "RT-05e FAIL: should have created a delete marker");
-
-        // ListObjectVersions should show 3 versions + 1 delete marker
-        let versions = client
-            .list_object_versions()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list versions");
-
-        assert_eq!(
-            versions.versions().len(),
-            3,
-            "RT-05e FAIL: expected 3 versions, found {}",
-            versions.versions().len()
-        );
-        assert_eq!(
-            versions.delete_markers().len(),
-            1,
-            "RT-05e FAIL: expected 1 delete marker, found {}",
-            versions.delete_markers().len()
-        );
-
-        // Now delete the delete marker (restore the object)
-        let dm_version = &versions.delete_markers()[0];
-        client
-            .delete_object()
-            .bucket(bucket)
-            .key("history.txt")
-            .version_id(dm_version.version_id().expect("dm version id"))
-            .send()
-            .await
-            .expect("delete delete-marker");
-
-        // HEAD should succeed now (latest version is accessible)
-        let head = client.head_object().bucket(bucket).key("history.txt").send().await;
-
-        assert!(head.is_ok(), "RT-05e FAIL: HEAD should succeed after removing delete marker");
-
-        info!("RT-05e PASS: versioned delete marker and list consistency");
-        Ok(())
-    }
-
-    /// RT-05f: Verify object deletion does not leave orphan data on disk.
-    ///
-    /// Regression pattern: after delete, the object data files remain on disk
-    /// (rustfs#5029: Node Does Not Remove Files After Reconnect).
-    #[tokio::test]
-    #[serial]
-    async fn test_delete_removes_object_head_returns_404() -> TestResult {
-        init_logging();
-        info!("RT-05f: delete → HEAD 404 consistency");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt05f-delete-head";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Upload, delete, verify HEAD returns 404
-        let keys = vec!["small.txt", "medium.txt", "with-slash.txt", "special+chars.txt"];
-
-        for key in &keys {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(*key)
-                .body(ByteStream::from_static(b"delete-me"))
-                .send()
-                .await
-                .expect("put object");
-        }
-
-        for key in &keys {
-            client
-                .delete_object()
-                .bucket(bucket)
-                .key(*key)
-                .send()
-                .await
-                .expect("delete object");
-        }
-
-        // All HEAD requests should return 404
-        for key in &keys {
-            let head = client.head_object().bucket(bucket).key(*key).send().await;
-
-            assert!(head.is_err(), "RT-05f FAIL: HEAD on deleted key '{key}' should return error");
-        }
-
-        // LIST should be empty
-        let list = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list after all deletes");
-
-        assert!(
-            list.contents().is_empty(),
-            "RT-05f FAIL: {} objects remain after deleting all",
-            list.contents().len()
-        );
-
-        info!("RT-05f PASS: all deleted objects return 404 on HEAD");
-        Ok(())
-    }
-}

crates/e2e_test/src/distributed_startup_regression_test.rs

@@ -1,202 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression tests for distributed cluster startup and quorum.
-//!
-//! Covers the recurring pattern where multi-node clusters fail to start due to
-//! lock quorum issues, DNS resolution delays, or erasure quorum deadlocks.
-//! This has regressed 7+ times.
-//!
-//! ## Regression Issues
-//!
-//! - rustfs#5416: RustFS cannot cold-start with 2/3 quorum when Pod DNS missing
-//! - rustfs#2945: Distributed mode fails on K8s: erasure quorum deadlock
-//! - rustfs#2794: distributed deployment does not become ready
-//! - rustfs#2601: fresh pod immediately enters FaultyDisk state
-//! - rustfs#4040: Distributed startup can fail lock quorum before AppContext initializes
-//! - rustfs#5655: fix(ecstore): bootstrap fresh four-node clusters reliably
-//! - rustfs#4954: S3/health endpoint unavailability after multi-pool scale-up
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestClusterEnvironment, init_logging};
-    use aws_sdk_s3::primitives::ByteStream;
-    use serial_test::serial;
-    use std::error::Error;
-    use tokio::time::{Duration, sleep};
-    use tracing::info;
-
-    type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-    /// RT-10: Verify 4-node cluster starts successfully and all nodes are ready.
-    ///
-    /// Regression pattern: distributed startup fails with quorum deadlock or
-    /// lock acquisition timeout (rustfs#2945, rustfs#5655).
-    ///
-    /// Steps:
-    /// 1. Create a 4-node cluster
-    /// 2. Start all nodes simultaneously
-    /// 3. Verify all nodes report healthy
-    /// 4. Verify S3 operations work through any node
-    #[tokio::test]
-    #[serial]
-    async fn test_four_node_cluster_startup_and_health() -> TestResult {
-        init_logging();
-        info!("RT-10: 4-node cluster startup and health");
-
-        let mut cluster = RustFSTestClusterEnvironment::new(4).await.expect("create 4-node cluster");
-
-        cluster.start().await.expect("start 4-node cluster");
-
-        // Create a bucket and verify it's accessible from all nodes
-        cluster
-            .create_test_bucket("rt10-startup")
-            .await
-            .expect("create bucket on cluster");
-
-        let clients = cluster.create_all_clients().expect("create per-node clients");
-
-        // Verify S3 operations work from every node
-        for (i, client) in clients.iter().enumerate() {
-            client
-                .put_object()
-                .bucket("rt10-startup")
-                .key(format!("from-node-{i}.txt"))
-                .body(ByteStream::from_static(b"hello from node"))
-                .send()
-                .await
-                .unwrap_or_else(|e| panic!("PUT from node {i} failed: {e}"));
-        }
-
-        // Verify all objects are visible from node 0
-        let list = clients[0]
-            .list_objects_v2()
-            .bucket("rt10-startup")
-            .send()
-            .await
-            .expect("list objects from node 0");
-
-        assert_eq!(
-            list.contents().len(),
-            4,
-            "RT-10 FAIL: expected 4 objects (one per node), found {}",
-            list.contents().len()
-        );
-
-        info!("RT-10 PASS: 4-node cluster starts and serves S3 from all nodes");
-        Ok(())
-    }
-
-    /// RT-10b: Verify cluster handles node restart gracefully.
-    ///
-    /// Regression pattern: after a node restart, it cannot rejoin the cluster
-    /// or enters a faulty state (rustfs#2601).
-    #[tokio::test]
-    #[serial]
-    async fn test_cluster_survives_node_restart() -> TestResult {
-        init_logging();
-        info!("RT-10b: cluster survives node restart");
-
-        let mut cluster = RustFSTestClusterEnvironment::new(4).await.expect("create 4-node cluster");
-
-        cluster.start().await.expect("start cluster");
-
-        cluster.create_test_bucket("rt10b-restart").await.expect("create bucket");
-
-        // Write data
-        let clients = cluster.create_all_clients()?;
-        clients[0]
-            .put_object()
-            .bucket("rt10b-restart")
-            .key("before-restart.txt")
-            .body(ByteStream::from_static(b"persistent data"))
-            .send()
-            .await
-            .expect("put object before restart");
-
-        // Stop node 3
-        cluster.stop_node(3).expect("stop node 3");
-        sleep(Duration::from_secs(2)).await;
-
-        // Verify cluster still works with 3/4 nodes (quorum)
-        clients[0]
-            .put_object()
-            .bucket("rt10b-restart")
-            .key("during-offline.txt")
-            .body(ByteStream::from_static(b"written while node 3 down"))
-            .send()
-            .await
-            .expect("PUT should succeed with 3/4 nodes");
-
-        // Restart node 3
-        cluster.start_node(3).await.expect("restart node 3");
-
-        // Wait for node to rejoin
-        sleep(Duration::from_secs(3)).await;
-
-        // Verify the restarted node can serve reads
-        let list = clients[3]
-            .list_objects_v2()
-            .bucket("rt10b-restart")
-            .send()
-            .await
-            .expect("list from restarted node");
-
-        assert!(
-            list.contents().len() >= 2,
-            "RT-10b FAIL: restarted node sees {} objects, expected >= 2",
-            list.contents().len()
-        );
-
-        info!("RT-10b PASS: cluster survives and recovers from node restart");
-        Ok(())
-    }
-
-    /// RT-10c: Verify bucket creation persists across all nodes.
-    ///
-    /// Regression pattern: bucket metadata is not replicated to all nodes,
-    /// causing NoSuchBucket errors on some nodes (rustfs#3191).
-    #[tokio::test]
-    #[serial]
-    async fn test_bucket_visible_from_all_nodes() -> TestResult {
-        init_logging();
-        info!("RT-10c: bucket visible from all nodes");
-
-        let mut cluster = RustFSTestClusterEnvironment::new(4).await.expect("create 4-node cluster");
-
-        cluster.start().await.expect("start cluster");
-
-        cluster
-            .create_test_bucket("rt10c-bucket-visibility")
-            .await
-            .expect("create bucket");
-
-        let clients = cluster.create_all_clients()?;
-
-        // Verify the bucket is visible from every node
-        for (i, client) in clients.iter().enumerate() {
-            let resp = client
-                .list_objects_v2()
-                .bucket("rt10c-bucket-visibility")
-                .send()
-                .await
-                .unwrap_or_else(|e| panic!("list from node {i} failed (NoSuchBucket?): {e}"));
-
-            assert!(resp.contents().is_empty(), "RT-10c: fresh bucket should be empty on node {i}");
-        }
-
-        info!("RT-10c PASS: bucket visible from all 4 nodes");
-        Ok(())
-    }
-}

crates/e2e_test/src/fake_s3_target/mod.rs

@@ -30,10 +30,10 @@ use s3s::access::{S3Access, S3AccessContext};
 use s3s::auth::SimpleAuth;
 use s3s::dto::{
     AbortMultipartUploadInput, AbortMultipartUploadOutput, CompleteMultipartUploadInput, CompleteMultipartUploadOutput,
-    CreateMultipartUploadInput, CreateMultipartUploadOutput, DeleteMarkerEntry, DeleteObjectInput, DeleteObjectOutput, ETag,
+    CreateMultipartUploadInput, CreateMultipartUploadOutput, DeleteObjectInput, DeleteObjectOutput, ETag,
     GetBucketVersioningInput, GetBucketVersioningOutput, GetObjectInput, GetObjectOutput, HeadBucketInput, HeadBucketOutput,
-    HeadObjectInput, HeadObjectOutput, ListObjectVersionsInput, ListObjectVersionsOutput, ObjectVersionId, PutObjectInput,
-    PutObjectOutput, StreamingBlob, Timestamp, TimestampFormat, UploadPartInput, UploadPartOutput,
+    HeadObjectInput, HeadObjectOutput, PutObjectInput, PutObjectOutput, StreamingBlob, Timestamp, TimestampFormat,
+    UploadPartInput, UploadPartOutput,
 };
 use s3s::service::{S3Service, S3ServiceBuilder};
 use s3s::validation::{AwsNameValidation, NameValidation};
@@ -91,7 +91,6 @@ pub enum Operation {
     GetObject,
     HeadObject,
     DeleteObject,
-    ListObjectVersions,
     CreateMultipartUpload,
     UploadPart,
     CompleteMultipartUpload,
@@ -110,8 +109,6 @@ pub enum FaultAction {
     /// already have buffered the rest of the current frame; the journal reports
     /// the threshold, and the backend never receives or stores the request.
     DisconnectAfterBytes(usize),
-    /// Apply the request, then close the connection before returning its response.
-    DisconnectAfterResponse,
     /// Drain a request body in fixed-size slices, sleeping after every slice.
     SlowDrain { chunk_bytes: usize, delay: Duration },
     /// Store the request normally but replace the response ETag.
@@ -137,15 +134,12 @@ pub struct RequestRecord {
 #[derive(Default)]
 struct ControlState {
     scripts: HashMap<Operation, VecDeque<FaultAction>>,
-    keyed_scripts: HashMap<(Operation, String), VecDeque<FaultAction>>,
     requests: VecDeque<RequestRecord>,
     next_sequence: u64,
 }
 
 #[derive(Default)]
 struct StoreState {
-    assign_own_version_ids: bool,
-    assign_own_multipart_version_ids: bool,
     buckets: HashMap<String, BucketState>,
     uploads: HashMap<String, MultipartState>,
     total_bytes: usize,
@@ -358,60 +352,25 @@ impl FakeS3Target {
         state.buckets.entry(bucket).or_default();
     }
 
-    /// Remove all retained object versions while preserving the bucket.
-    pub fn clear_bucket_objects(&self, bucket: &str) {
-        let mut state = lock(&self.backend.store);
-        let (removed_versions, removed_bytes) = state
-            .buckets
-            .get_mut(bucket)
-            .expect("fake target bucket must exist")
-            .objects
-            .drain()
-            .flat_map(|(_, versions)| versions)
-            .fold((0usize, 0usize), |(count, bytes), version| (count + 1, bytes + version.body.len()));
-        state.total_versions = state
-            .total_versions
-            .checked_sub(removed_versions)
-            .expect("fake target version accounting must not underflow");
-        state.total_bytes = state
-            .total_bytes
-            .checked_sub(removed_bytes)
-            .expect("fake target byte accounting must not underflow");
-    }
-
-    pub fn has_object(&self, bucket: &str, key: &str) -> bool {
-        lock(&self.backend.store)
-            .buckets
-            .get(bucket)
-            .and_then(|bucket| bucket.objects.get(key))
-            .and_then(|versions| versions.last())
-            .is_some_and(|version| !version.delete_marker)
-    }
-
-    /// Make the target mint its own version ids instead of mirroring the
-    /// forwarded source version id — models a generic S3 service.
-    pub fn assign_own_version_ids(&self, enabled: bool) {
-        lock(&self.backend.store).assign_own_version_ids = enabled;
-    }
-
-    /// Mint own version ids for the multipart path only — models a target
-    /// that adopts PutObject version ids but not CreateMultipartUpload ones.
-    pub fn assign_own_multipart_version_ids(&self, enabled: bool) {
-        lock(&self.backend.store).assign_own_multipart_version_ids = enabled;
-    }
-
-    pub fn active_multipart_upload_count(&self) -> usize {
-        lock(&self.backend.store).uploads.len()
-    }
-
     /// Queue `times` copies of a fault for one operation.
     pub fn inject(&self, operation: Operation, action: FaultAction, times: usize) {
         if times == 0 {
             return;
         }
-        validate_fault_action(&action);
+        if let FaultAction::SlowDrain { chunk_bytes: 0, .. } = action {
+            panic!("slow-drain chunk size must be non-zero");
+        }
+        match &action {
+            FaultAction::Delay(duration) if *duration > MAX_FAULT_DURATION => {
+                panic!("fault delay must not exceed 30 seconds");
+            }
+            FaultAction::SlowDrain { delay, .. } if *delay >= MAX_FAULT_DURATION => {
+                panic!("slow-drain slice delay must be below 30 seconds");
+            }
+            _ => {}
+        }
         let mut state = lock(&self.control);
-        let queued = queued_fault_count(&state);
+        let queued = state.scripts.values().map(VecDeque::len).sum::<usize>();
         if queued.checked_add(times).is_none_or(|total| total > MAX_SCRIPTED_FAULTS) {
             panic!("fake target queues at most 4096 scripted faults");
         }
@@ -422,28 +381,8 @@ impl FakeS3Target {
             .extend(std::iter::repeat_n(action, times));
     }
 
-    /// Queue faults for one exact object key without affecting concurrent requests.
-    pub fn inject_for_key(&self, operation: Operation, key: impl Into<String>, action: FaultAction, times: usize) {
-        if times == 0 {
-            return;
-        }
-        validate_fault_action(&action);
-        let mut state = lock(&self.control);
-        let queued = queued_fault_count(&state);
-        if queued.checked_add(times).is_none_or(|total| total > MAX_SCRIPTED_FAULTS) {
-            panic!("fake target queues at most 4096 scripted faults");
-        }
-        state
-            .keyed_scripts
-            .entry((operation, key.into()))
-            .or_default()
-            .extend(std::iter::repeat_n(action, times));
-    }
-
     pub fn clear_faults(&self) {
-        let mut state = lock(&self.control);
-        state.scripts.clear();
-        state.keyed_scripts.clear();
+        lock(&self.control).scripts.clear();
     }
 
     pub fn requests(&self) -> Vec<RequestRecord> {
@@ -454,25 +393,6 @@ impl FakeS3Target {
         lock(&self.control).requests.drain(..).collect()
     }
 
-    /// Stored versions for one key as `(version_id, is_delete_marker)`, oldest
-    /// first. Empty when the bucket or key does not exist. Lets purge tests
-    /// assert on the target's actual state instead of inferring it from the
-    /// request journal (a versioned DELETE is a silent no-op for missing ids).
-    pub fn stored_versions(&self, bucket: &str, key: &str) -> Vec<(String, bool)> {
-        let state = lock(&self.backend.store);
-        state
-            .buckets
-            .get(bucket)
-            .and_then(|bucket_state| bucket_state.objects.get(key))
-            .map(|versions| {
-                versions
-                    .iter()
-                    .map(|version| (version.version_id.clone(), version.delete_marker))
-                    .collect()
-            })
-            .unwrap_or_default()
-    }
-
     pub async fn shutdown(mut self) {
         let _ = self.shutdown.send(true);
         if let Some(task) = self.task.take() {
@@ -500,25 +420,6 @@ fn lock<T>(mutex: &Mutex<T>) -> MutexGuard<'_, T> {
     mutex.lock().unwrap_or_else(std::sync::PoisonError::into_inner)
 }
 
-fn validate_fault_action(action: &FaultAction) {
-    if let FaultAction::SlowDrain { chunk_bytes: 0, .. } = action {
-        panic!("slow-drain chunk size must be non-zero");
-    }
-    match action {
-        FaultAction::Delay(duration) if *duration > MAX_FAULT_DURATION => {
-            panic!("fault delay must not exceed 30 seconds");
-        }
-        FaultAction::SlowDrain { delay, .. } if *delay >= MAX_FAULT_DURATION => {
-            panic!("slow-drain slice delay must be below 30 seconds");
-        }
-        _ => {}
-    }
-}
-
-fn queued_fault_count(state: &ControlState) -> usize {
-    state.scripts.values().map(VecDeque::len).sum::<usize>() + state.keyed_scripts.values().map(VecDeque::len).sum::<usize>()
-}
-
 #[async_trait]
 impl S3Access for FaultAccess {
     async fn check(&self, context: &mut S3AccessContext<'_>) -> S3Result<()> {
@@ -591,12 +492,7 @@ fn record_request(
     content_length: Option<u64>,
 ) -> Option<RequestFault> {
     let mut state = lock(control);
-    let action = parsed
-        .key
-        .as_ref()
-        .and_then(|key| state.keyed_scripts.get_mut(&(operation, key.clone())))
-        .and_then(VecDeque::pop_front)
-        .or_else(|| state.scripts.get_mut(&operation).and_then(VecDeque::pop_front));
+    let action = state.scripts.get_mut(&operation).and_then(VecDeque::pop_front);
     state.next_sequence += 1;
     let sequence = state.next_sequence;
     if state.requests.len() == MAX_REQUEST_RECORDS {
@@ -673,14 +569,12 @@ fn parse_request(method: &Method, uri: &Uri) -> ParsedRequest {
     let operation = match (method, key.is_some()) {
         (&Method::HEAD, false) => Operation::HeadBucket,
         (&Method::GET, false) if query.contains_key("versioning") => Operation::GetBucketVersioning,
-        (&Method::GET, false) if query.contains_key("versions") => Operation::ListObjectVersions,
         (&Method::PUT, true) if upload_id.is_some() && part_number.is_some() => Operation::UploadPart,
         (&Method::PUT, true) if upload_id.is_some() || query.contains_key("partNumber") => Operation::Unknown,
         (&Method::POST, true) if query.contains_key("uploads") => Operation::CreateMultipartUpload,
         (&Method::POST, true) if upload_id.is_some() => Operation::CompleteMultipartUpload,
         (&Method::DELETE, true) if upload_id.is_some() => Operation::AbortMultipartUpload,
-        // A replication PUT addresses the source version via `?versionId=`.
-        (&Method::PUT, true) if only_query_keys(&["versionId"]) => Operation::PutObject,
+        (&Method::PUT, true) if only_query_keys(&[]) => Operation::PutObject,
         (&Method::GET, true) if only_query_keys(&["versionId"]) => Operation::GetObject,
         (&Method::HEAD, true) if only_query_keys(&["versionId"]) => Operation::HeadObject,
         (&Method::DELETE, true) if only_query_keys(&["versionId"]) => Operation::DeleteObject,
@@ -730,17 +624,10 @@ fn validate_retained_identifier(value: String, field: &str) -> S3Result<String>
     }
 }
 
-/// `assign_own` models a target that mints its own version ids (a generic S3
-/// service): the forwarded source-version-id header is validated but NOT
-/// mirrored into the stored version.
-fn new_version_id(headers: &HeaderMap, assign_own: bool) -> S3Result<String> {
+fn new_version_id(headers: &HeaderMap) -> S3Result<String> {
     let Some(value) = header_value(headers, &SOURCE_VERSION_ID_HEADERS) else {
         return Ok(Uuid::new_v4().to_string());
     };
-    if assign_own {
-        validate_retained_identifier(value.trim().to_owned(), "source version ID")?;
-        return Ok(Uuid::new_v4().to_string());
-    }
     let value = validate_retained_identifier(value.trim().to_owned(), "source version ID")?;
     let version_id = Uuid::parse_str(&value).map_err(|_| s3s::s3_error!(InvalidArgument, "source version ID must be a UUID"))?;
     Ok(version_id.to_string())
@@ -825,10 +712,7 @@ async fn apply_non_body_fault(fault: Option<&RequestFault>, control: &Mutex<Cont
             update_consumed(control, fault.expect("matched fault").sequence, 0);
             Err(scripted_disconnect_error())
         }
-        Some(FaultAction::SlowDrain { .. })
-        | Some(FaultAction::WrongEtag)
-        | Some(FaultAction::DisconnectAfterResponse)
-        | None => Ok(()),
+        Some(FaultAction::SlowDrain { .. }) | Some(FaultAction::WrongEtag) | None => Ok(()),
     }
 }
 
@@ -869,7 +753,7 @@ async fn collect_stream(
             Some(FaultAction::SlowDrain { chunk_bytes, delay }) => {
                 return collect_stream_slow(body, capacity, *chunk_bytes, *delay).await;
             }
-            Some(FaultAction::WrongEtag) | Some(FaultAction::DisconnectAfterResponse) | None => {}
+            Some(FaultAction::WrongEtag) | None => {}
         }
 
         let mut output = BytesMut::with_capacity(capacity);
@@ -931,9 +815,6 @@ fn apply_response_fault<T>(mut response: S3Response<T>, fault: Option<&RequestFa
     if fault.is_some_and(|fault| fault.action == FaultAction::WrongEtag) {
         response.headers.insert(ETAG, HeaderValue::from_static(WRONG_ETAG));
     }
-    if fault.is_some_and(|fault| fault.action == FaultAction::DisconnectAfterResponse) {
-        response.headers.insert(DISCONNECT_HEADER, HeaderValue::from_static("true"));
-    }
     response
 }
 
@@ -1122,63 +1003,6 @@ impl S3 for FakeBackend {
         ))
     }
 
-    /// Prefix + max-keys subset only — enough for the replication-check probe
-    /// key allocation. No pagination markers or delimiter folding.
-    async fn list_object_versions(
-        &self,
-        req: S3Request<ListObjectVersionsInput>,
-    ) -> S3Result<S3Response<ListObjectVersionsOutput>> {
-        let fault = request_fault(&req);
-        apply_non_body_fault(fault.as_ref(), &self.control).await?;
-        let state = lock(&self.store);
-        let Some(bucket_state) = state.buckets.get(&req.input.bucket) else {
-            return Err(s3s::s3_error!(NoSuchBucket, "bucket does not exist"));
-        };
-        let prefix = req.input.prefix.as_deref().unwrap_or_default();
-        let max_keys = req.input.max_keys.unwrap_or(1000).max(0) as usize;
-
-        let mut keys: Vec<&String> = bucket_state.objects.keys().filter(|key| key.starts_with(prefix)).collect();
-        keys.sort();
-
-        let mut versions = Vec::new();
-        let mut delete_markers = Vec::new();
-        'keys: for key in keys {
-            for version in bucket_state.objects[key].iter().rev() {
-                if versions.len() + delete_markers.len() >= max_keys {
-                    break 'keys;
-                }
-                if version.delete_marker {
-                    delete_markers.push(DeleteMarkerEntry {
-                        key: Some(key.clone()),
-                        version_id: Some(ObjectVersionId::from(version.version_id.clone())),
-                        last_modified: Some(version.last_modified.clone()),
-                        ..Default::default()
-                    });
-                } else {
-                    versions.push(s3s::dto::ObjectVersion {
-                        key: Some(key.clone()),
-                        version_id: Some(ObjectVersionId::from(version.version_id.clone())),
-                        last_modified: Some(version.last_modified.clone()),
-                        e_tag: Some(ETag::Strong(version.e_tag.clone())),
-                        size: Some(version.body.len() as i64),
-                        ..Default::default()
-                    });
-                }
-            }
-        }
-        drop(state);
-
-        Ok(apply_response_fault(
-            S3Response::new(ListObjectVersionsOutput {
-                name: Some(req.input.bucket),
-                versions: Some(versions),
-                delete_markers: Some(delete_markers),
-                ..Default::default()
-            }),
-            fault.as_ref(),
-        ))
-    }
-
     async fn put_object(&self, req: S3Request<PutObjectInput>) -> S3Result<S3Response<PutObjectOutput>> {
         let fault = request_fault(&req);
         let _body_permit = timeout(MAX_FAULT_DURATION, Arc::clone(&self.body_limit).acquire_owned())
@@ -1189,8 +1013,7 @@ impl S3 for FakeBackend {
         let input = req.input;
         let body = collect_stream(input.body, input.content_length, fault.as_ref(), &self.control).await?;
         validate_stored_metadata(&input.content_type, &input.metadata)?;
-        let assign_own = lock(&self.store).assign_own_version_ids;
-        let version_id = new_version_id(&headers, assign_own)?;
+        let version_id = new_version_id(&headers)?;
         let e_tag = match source_etag(&headers)? {
             Some(value) => value,
             None => {
@@ -1233,7 +1056,7 @@ impl S3 for FakeBackend {
                 content_type: version.content_type,
                 metadata: version.metadata,
                 e_tag: Some(ETag::Strong(version.e_tag)),
-                last_modified: Some(version.last_modified.clone()),
+                last_modified: Some(version.last_modified),
                 version_id: Some(version.version_id),
                 ..Default::default()
             }),
@@ -1255,7 +1078,7 @@ impl S3 for FakeBackend {
                 content_type: version.content_type,
                 metadata: version.metadata,
                 e_tag: Some(ETag::Strong(version.e_tag)),
-                last_modified: Some(version.last_modified.clone()),
+                last_modified: Some(version.last_modified),
                 version_id: Some(version.version_id),
                 ..Default::default()
             }),
@@ -1324,9 +1147,7 @@ impl S3 for FakeBackend {
             ));
         }
 
-        // `state` is the live store guard: read the flag from it. Re-locking
-        // would self-deadlock (the store mutex is not reentrant).
-        let version_id = new_version_id(&headers, state.assign_own_version_ids)?;
+        let version_id = new_version_id(&headers)?;
         upsert_version(
             &mut state,
             &input.bucket,
@@ -1366,16 +1187,12 @@ impl S3 for FakeBackend {
         ensure_upload_budget(&state)?;
         validate_stored_metadata(&input.content_type, &input.metadata)?;
         let upload_id = Uuid::new_v4().to_string();
-        // Read the flag before the mutable borrow of `state.uploads` below
-        // (and never re-lock the store: the mutex is not reentrant).
-        let mint_own = state.assign_own_version_ids || state.assign_own_multipart_version_ids;
-        let version_id = new_version_id(&headers, mint_own)?;
         state.uploads.insert(
             upload_id.clone(),
             MultipartState {
                 bucket: input.bucket.clone(),
                 key: input.key.clone(),
-                version_id,
+                version_id: new_version_id(&headers)?,
                 content_type: input.content_type,
                 metadata: input.metadata,
                 parts: BTreeMap::new(),

crates/e2e_test/src/get_codec_streaming_compat_test.rs

@@ -189,6 +189,8 @@ mod tests {
             ("RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT", "100"),
             ("RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED", "true"),
             ("RUSTFS_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED", "true"),
+            // Lower the min-size floor so every non-inline object below is eligible.
+            ("RUSTFS_GET_CODEC_STREAMING_MIN_SIZE", "4096"),
             // Route multipart objects through per-part codec streaming too.
             ("RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE", "true"),
             // Lock optimization is on by default, but pin it so the gate's
@@ -313,13 +315,6 @@ mod tests {
                 },
                 payload(64 * 1024, 2),
             ),
-            (
-                Shape {
-                    key: "small-non-inline-256kib-plus",
-                    expect_large: true,
-                },
-                payload(256 * 1024 + 1, 6),
-            ),
             (
                 Shape {
                     key: "mid-1_5mib",

crates/e2e_test/src/group_delete_test.rs

@@ -14,7 +14,7 @@
 
 //! E2E tests for group management (fixes #2028).
 
-use crate::common::{RustFSTestEnvironment, admin_request, awscurl_delete, awscurl_get, awscurl_put, init_logging};
+use crate::common::{RustFSTestEnvironment, awscurl_delete, awscurl_get, awscurl_put, init_logging};
 use aws_sdk_s3::config::{Credentials, Region};
 use aws_sdk_s3::{Client, Config};
 use serial_test::serial;
@@ -32,56 +32,6 @@ fn create_user_s3_client(env: &RustFSTestEnvironment, access_key: &str, secret_k
     Client::from_conf(config)
 }
 
-#[tokio::test(flavor = "multi_thread")]
-async fn update_group_members_rejects_invalid_new_group_names() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let invalid_groups = [
-        ("test group", "group name contains whitespace"),
-        ("test=group", "group name contains reserved characters =,"),
-        ("test,group", "group name contains reserved characters =,"),
-    ];
-
-    for (group, expected_message) in invalid_groups {
-        let body = serde_json::json!({
-            "group": group,
-            "members": [],
-            "isRemove": false,
-            "groupStatus": "enabled"
-        })
-        .to_string();
-        let (status, response_body) = admin_request(
-            &env.url,
-            http::Method::PUT,
-            "/rustfs/admin/v3/update-group-members",
-            Some(body),
-            &env.access_key,
-            &env.secret_key,
-        )
-        .await?;
-
-        assert_eq!(
-            status,
-            reqwest::StatusCode::BAD_REQUEST,
-            "invalid group {group:?} must return HTTP 400, body: {response_body}"
-        );
-        assert!(
-            response_body.contains("<Code>InvalidArgument</Code>"),
-            "invalid group {group:?} must return InvalidArgument, body: {response_body}"
-        );
-        assert!(
-            response_body.contains(&format!("<Message>{expected_message}</Message>")),
-            "invalid group {group:?} returned an unexpected message: {response_body}"
-        );
-    }
-
-    env.stop_server();
-    Ok(())
-}
-
 /// Test that deleting a group with members fails, and deleting an empty group succeeds.
 #[tokio::test(flavor = "multi_thread")]
 #[serial]

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -431,104 +431,4 @@ mod tests {
         )
         .into())
     }
-
-    /// Issue #5850: `background-heal/status` must answer while a peer is down.
-    ///
-    /// Exercises the production path in `read_cluster_heal_status` end to end,
-    /// which the unit tests around `merge_peer_heal_statuses` cannot: with one
-    /// node stopped, the endpoint must return 200 with
-    /// `clusterStatusComplete: false` and an explicit `degraded` (or, when
-    /// heal work is known active, `active`) state — never the previous
-    /// cluster-wide 500 — and must return to a complete, non-degraded answer
-    /// once the node rejoins. Reverting either all-or-nothing gate (the
-    /// topology early-return or the merge hard-fail) turns the down-window
-    /// response into a 500 and fails this test.
-    #[tokio::test]
-    #[serial]
-    async fn test_background_heal_status_degrades_while_peer_down_and_recovers_after_rejoin()
-    -> Result<(), Box<dyn Error + Send + Sync>> {
-        init_logging();
-        info!("Issue #5850: background-heal/status must degrade, not 500, while a peer is down");
-
-        let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
-        cluster.set_env("RUSTFS_UNSAFE_BYPASS_DISK_CHECK", "true");
-        cluster.start().await?;
-
-        let status_url = format!("{}/rustfs/admin/v3/background-heal/status", cluster.nodes[0].url);
-        // Owned copies: the closure must not borrow `cluster`, which
-        // stop_node/start_node need mutably between polls.
-        let access_key = cluster.access_key.clone();
-        let secret_key = cluster.secret_key.clone();
-        let fetch_status = || async {
-            let body = signed_admin_post(&status_url, None, &access_key, &secret_key).await?;
-            let json: serde_json::Value =
-                serde_json::from_str(&body).map_err(|err| format!("heal status response is not JSON ({err}): {body}"))?;
-            Ok::<serde_json::Value, Box<dyn Error + Send + Sync>>(json)
-        };
-
-        // Healthy cluster: the answer must be definitive. Poll briefly — the
-        // peer grid may still be settling right after start().
-        let mut healthy = fetch_status().await?;
-        for _ in 0..30 {
-            if healthy["clusterStatusComplete"] == serde_json::Value::Bool(true) {
-                break;
-            }
-            sleep(Duration::from_secs(1)).await;
-            healthy = fetch_status().await?;
-        }
-        assert_eq!(
-            healthy["clusterStatusComplete"],
-            serde_json::Value::Bool(true),
-            "healthy cluster should report a complete heal status: {healthy}"
-        );
-
-        cluster.stop_node(1)?;
-
-        // While the peer is down every response must stay 200 (signed_admin_post
-        // fails on any non-2xx, so the old 500 fails the test immediately) and
-        // must degrade to an explicitly-partial answer. The peer query timeout
-        // is 5 s, so a couple of polls are enough for the dead peer to surface.
-        let mut degraded = serde_json::Value::Null;
-        for _ in 0..30 {
-            degraded = fetch_status().await?;
-            if degraded["clusterStatusComplete"] == serde_json::Value::Bool(false) {
-                break;
-            }
-            sleep(Duration::from_secs(1)).await;
-        }
-        assert_eq!(
-            degraded["clusterStatusComplete"],
-            serde_json::Value::Bool(false),
-            "heal status must mark itself partial while a peer is down: {degraded}"
-        );
-        let state = degraded["state"].as_str().unwrap_or_default();
-        assert!(
-            state == "degraded" || state == "active",
-            "a partial answer must be labeled degraded (or active for known work), got {state:?}: {degraded}"
-        );
-
-        cluster.start_node(1).await?;
-
-        // After the rejoin the endpoint must return to a definitive answer.
-        let mut recovered = serde_json::Value::Null;
-        for _ in 0..60 {
-            recovered = fetch_status().await?;
-            if recovered["clusterStatusComplete"] == serde_json::Value::Bool(true) {
-                break;
-            }
-            sleep(Duration::from_secs(1)).await;
-        }
-        assert_eq!(
-            recovered["clusterStatusComplete"],
-            serde_json::Value::Bool(true),
-            "heal status should be complete again after the node rejoined: {recovered}"
-        );
-        assert_ne!(
-            recovered["state"].as_str().unwrap_or_default(),
-            "degraded",
-            "a complete answer must not be labeled degraded: {recovered}"
-        );
-
-        Ok(())
-    }
 }

crates/e2e_test/src/inline_fast_path_cluster_test.rs

@@ -1687,44 +1687,6 @@ async fn four_node_inline_storage_and_get_boundaries() -> TestResult {
     Ok(())
 }
 
-#[tokio::test]
-#[serial]
-async fn four_node_empty_legacy_volumes_start_as_fresh() -> TestResult {
-    init_logging();
-
-    let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
-    for data_dir in cluster.nodes.iter().flat_map(|node| &node.data_dirs) {
-        tokio::fs::create_dir_all(Path::new(data_dir).join(".minio.sys")).await?;
-    }
-
-    cluster.start().await?;
-
-    // Starting is not the assertion. The regression is that an empty legacy
-    // `.minio.sys` must be classified as a *fresh* volume, not as an existing
-    // MinIO deployment to adopt or migrate. Pin what that classification leaves
-    // on disk and in the namespace.
-    let buckets = cluster.create_s3_client(0)?.list_buckets().send().await?;
-    assert!(
-        buckets.buckets().is_empty(),
-        "a fresh classification must not adopt buckets from the pre-existing directories, got {:?}",
-        buckets.buckets().iter().filter_map(|b| b.name()).collect::<Vec<_>>()
-    );
-
-    for data_dir in cluster.nodes.iter().flat_map(|node| &node.data_dirs) {
-        assert!(
-            Path::new(data_dir).join(".rustfs.sys").join("format.json").is_file(),
-            "each drive must be formatted as fresh: {data_dir} has no .rustfs.sys/format.json"
-        );
-        let mut legacy = tokio::fs::read_dir(Path::new(data_dir).join(".minio.sys")).await?;
-        assert!(
-            legacy.next_entry().await?.is_none(),
-            "the empty legacy directory must be left untouched, not migrated into: {data_dir}"
-        );
-    }
-
-    Ok(())
-}
-
 #[tokio::test]
 #[serial]
 async fn four_node_inline_fallback_controls() -> TestResult {
@@ -2211,6 +2173,11 @@ async fn four_node_manual_transition_distributed_admission_conflict_reports_stat
             "queue_snapshot.{field} must be readable in terminal status: {terminal}"
         );
     }
+    assert!(
+        cold_tier_object_count(&cold_client).await? < 64,
+        "queue pressure should leave at least one object untransitioned"
+    );
+
     Ok(())
 }
 

crates/e2e_test/src/kms/bucket_default_encryption_test.rs

@@ -21,12 +21,9 @@
 
 use super::common::LocalKMSTestEnvironment;
 use crate::common::{TEST_BUCKET, init_logging};
-use aws_sdk_s3::primitives::ByteStream;
 use aws_sdk_s3::types::{
-    ChecksumAlgorithm, ChecksumMode, CompletedMultipartUpload, CompletedPart, ServerSideEncryption,
-    ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule,
+    ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule,
 };
-use rustfs_rio::{Checksum, ChecksumType};
 use serial_test::serial;
 use tracing::{debug, info, warn};
 
@@ -276,7 +273,7 @@ async fn test_bucket_default_sse_kms_put_object() -> Result<(), Box<dyn std::err
 /// Test 3: When bucket is configured with default encryption, create_multipart_upload should inherit the configuration
 #[tokio::test]
 #[serial]
-async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+async fn test_bucket_default_encryption_multipart_upload() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     init_logging();
     info!("Testing bucket default encryption impact on create_multipart_upload");
 
@@ -312,16 +309,15 @@ async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std
         .await
         .expect("Failed to set bucket encryption");
 
-    // Step 2: Declare CRC32 without specifying encryption parameters. The AWS SDK
-    // calculates each UploadPart checksum and sends it as a flexible checksum.
-    info!("Creating CRC32 multipart upload that should use bucket default encryption");
-    let test_key = "test-multipart-bucket-default-crc32.bin";
+    // Step 2: Create multipart upload (without specifying encryption parameters)
+    info!("Creating multipart upload (without specifying encryption parameters, should use bucket default configuration)");
+    let test_key = "test-multipart-bucket-default.txt";
 
     let create_multipart_response = s3_client
         .create_multipart_upload()
         .bucket(TEST_BUCKET)
         .key(test_key)
-        .checksum_algorithm(ChecksumAlgorithm::Crc32)
+        // Note: No encryption parameters specified here, should use bucket default configuration
         .send()
         .await
         .expect("Failed to create multipart upload");
@@ -347,61 +343,28 @@ async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std
         "create_multipart_upload response should contain correct KMS key ID"
     );
 
-    // Step 3: Upload two parts. The first is exactly the S3 minimum size so this
-    // follows the same managed SSE-KMS multipart path as issue #5756.
-    const PART_SIZE: usize = 5 * 1024 * 1024;
-    let part1: Vec<u8> = (0..PART_SIZE).map(|i| (i % 251) as u8).collect();
-    let part2: Vec<u8> = (0..1024 * 1024).map(|i| ((i + 17) % 251) as u8).collect();
-    let expected_body: Vec<u8> = part1.iter().chain(&part2).copied().collect();
+    // Step 3: Upload a part and complete multipart upload
+    info!("Uploading part and completing multipart upload");
+    let test_data = b"test-multipart-bucket-default-encryption-data";
 
-    let upload_part = |part_number: i32, body: Vec<u8>| {
-        s3_client
-            .upload_part()
-            .bucket(TEST_BUCKET)
-            .key(test_key)
-            .upload_id(upload_id)
-            .part_number(part_number)
-            .checksum_algorithm(ChecksumAlgorithm::Crc32)
-            .body(ByteStream::from(body))
-            .send()
-    };
+    // Upload part 1
+    let upload_part_response = s3_client
+        .upload_part()
+        .bucket(TEST_BUCKET)
+        .key(test_key)
+        .upload_id(upload_id)
+        .part_number(1)
+        .body(test_data.to_vec().into())
+        .send()
+        .await
+        .expect("Failed to upload part");
 
-    let expected_part1_crc32 = Checksum::new_from_data(ChecksumType::CRC32, &part1)
-        .expect("calculate part 1 CRC32")
-        .encoded;
-    let upload1 = upload_part(1, part1).await.expect("Failed to upload part 1 with CRC32");
-    assert_eq!(
-        upload1.checksum_crc32(),
-        Some(expected_part1_crc32.as_str()),
-        "UploadPart must return the CRC32 calculated over plaintext"
-    );
-
-    let expected_part2_crc32 = Checksum::new_from_data(ChecksumType::CRC32, &part2)
-        .expect("calculate part 2 CRC32")
-        .encoded;
-    let upload2 = upload_part(2, part2).await.expect("Failed to upload part 2 with CRC32");
-    assert_eq!(
-        upload2.checksum_crc32(),
-        Some(expected_part2_crc32.as_str()),
-        "UploadPart must return the CRC32 calculated over plaintext"
-    );
+    let etag = upload_part_response.e_tag().unwrap().to_string();
 
     // Complete multipart upload
-    let completed_upload = CompletedMultipartUpload::builder()
-        .parts(
-            CompletedPart::builder()
-                .part_number(1)
-                .e_tag(upload1.e_tag().expect("No ETag for part 1"))
-                .checksum_crc32(upload1.checksum_crc32().expect("No CRC32 for part 1"))
-                .build(),
-        )
-        .parts(
-            CompletedPart::builder()
-                .part_number(2)
-                .e_tag(upload2.e_tag().expect("No ETag for part 2"))
-                .checksum_crc32(upload2.checksum_crc32().expect("No CRC32 for part 2"))
-                .build(),
-        )
+    let completed_part = aws_sdk_s3::types::CompletedPart::builder()
+        .part_number(1)
+        .e_tag(&etag)
         .build();
 
     let complete_multipart_response = s3_client
@@ -409,7 +372,11 @@ async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std
         .bucket(TEST_BUCKET)
         .key(test_key)
         .upload_id(upload_id)
-        .multipart_upload(completed_upload)
+        .multipart_upload(
+            aws_sdk_s3::types::CompletedMultipartUpload::builder()
+                .parts(completed_part)
+                .build(),
+        )
         .send()
         .await
         .expect("Failed to complete multipart upload");
@@ -433,7 +400,6 @@ async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std
         .get_object()
         .bucket(TEST_BUCKET)
         .key(test_key)
-        .checksum_mode(ChecksumMode::Enabled)
         .send()
         .await
         .expect("Failed to get object");
@@ -444,13 +410,6 @@ async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std
         Some(&ServerSideEncryption::AwsKms),
         "Final object should contain SSE-KMS encryption information"
     );
-    if let Some(completed_crc32) = complete_multipart_response.checksum_crc32() {
-        assert_eq!(
-            get_response.checksum_crc32(),
-            Some(completed_crc32),
-            "GetObject should return the persisted composite CRC32 when completion reports it"
-        );
-    }
 
     // Verify data integrity
     let downloaded_data = get_response
@@ -459,11 +418,7 @@ async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std
         .await
         .expect("Failed to collect body")
         .into_bytes();
-    assert_eq!(
-        downloaded_data.as_ref(),
-        expected_body.as_slice(),
-        "Downloaded data should match the uploaded multipart body"
-    );
+    assert_eq!(&downloaded_data[..], test_data, "Downloaded data should match original data");
 
     // Cleanup is handled automatically when the test environment is dropped
     info!("Test passed: bucket default encryption correctly applied to multipart upload");

crates/e2e_test/src/kms/copy_object_self_copy_sse_test.rs

@@ -1,351 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression test: a same-key CopyObject that only rewrites metadata must never re-key a
-//! managed-SSE (SSE-S3 / SSE-KMS) object.
-//!
-//! On an **unversioned** bucket the handler marks a same-name copy `metadata_only`, and the
-//! store layer then updates `xl.meta` in place without touching the data blocks. The handler
-//! nevertheless strips the source encryption metadata and generates a *fresh* DEK for the
-//! destination. Combining the two writes "new DEK + old ciphertext": the object is permanently
-//! undecryptable. The fix forces a full data rewrite whenever the copy re-derives managed
-//! encryption material, so the stored bytes always match the key metadata beside them.
-//!
-//! Companion to `copy_object_version_restore_sse_test` (issue #4238), which pins the same
-//! invariant for the versioned historical-restore path.
-
-use super::common::{LocalKMSTestEnvironment, create_key_with_specific_id};
-use crate::common::init_logging;
-use aws_sdk_s3::primitives::ByteStream;
-use aws_sdk_s3::types::{
-    MetadataDirective, ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration,
-    ServerSideEncryptionRule,
-};
-use serial_test::serial;
-use tracing::info;
-
-#[tokio::test]
-#[serial]
-async fn test_metadata_replace_self_copy_of_sse_object_stays_decryptable() {
-    init_logging();
-    info!("same-key CopyObject with REPLACE metadata must not re-key an SSE-S3 object");
-
-    let mut kms_env = LocalKMSTestEnvironment::new().await.expect("failed to create local KMS env");
-    let default_key_id = "rustfs-e2e-test-default-key";
-    let keys_dir = kms_env.kms_keys_dir.clone();
-    create_key_with_specific_id(&keys_dir, default_key_id)
-        .await
-        .expect("failed to create local KMS key");
-    kms_env
-        .base_env
-        .start_rustfs_server_with_env(
-            vec![
-                "--kms-enable",
-                "--kms-backend",
-                "local",
-                "--kms-key-dir",
-                &keys_dir,
-                "--kms-default-key-id",
-                default_key_id,
-            ],
-            &[("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")],
-        )
-        .await
-        .expect("failed to start RustFS with local KMS");
-    tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;
-
-    let client = kms_env.base_env.create_s3_client();
-    // Deliberately an UNVERSIONED bucket: that is the branch where the store layer can service
-    // the self-copy as a pure metadata update.
-    let bucket = "copy-object-self-copy-sse-test";
-    let key = "secrets/report.txt";
-
-    client
-        .create_bucket()
-        .bucket(bucket)
-        .send()
-        .await
-        .expect("failed to create bucket");
-
-    // Content long enough that a truncated/garbled decrypt cannot coincidentally match.
-    let content = b"encrypted payload that must survive a metadata-only self copy -- 0123456789";
-    let put = client
-        .put_object()
-        .bucket(bucket)
-        .key(key)
-        .content_type("text/plain; charset=utf-8")
-        .metadata("stage", "before")
-        .server_side_encryption(ServerSideEncryption::Aes256)
-        .body(ByteStream::from_static(content))
-        .send()
-        .await
-        .expect("PUT failed");
-    assert_eq!(put.server_side_encryption(), Some(&ServerSideEncryption::Aes256));
-
-    // Copy the object onto itself, replacing user metadata. This is the `mc cp --attr` /
-    // "edit metadata in place" shape that AWS supports on an existing object.
-    let copy_out = client
-        .copy_object()
-        .bucket(bucket)
-        .key(key)
-        .copy_source(format!("{bucket}/{key}"))
-        .metadata_directive(MetadataDirective::Replace)
-        .content_type("text/plain; charset=utf-8")
-        .metadata("stage", "after")
-        .server_side_encryption(ServerSideEncryption::Aes256)
-        .send()
-        .await
-        .expect("same-key CopyObject with REPLACE metadata must succeed");
-    assert_eq!(copy_out.server_side_encryption(), Some(&ServerSideEncryption::Aes256));
-
-    // The object must still decrypt to the original plaintext. Before the fix the stored
-    // ciphertext was left untouched while the metadata carried a brand-new DEK, so this GET
-    // either failed outright or returned garbage.
-    let get = client
-        .get_object()
-        .bucket(bucket)
-        .key(key)
-        .send()
-        .await
-        .expect("GET after self-copy failed: the object was re-keyed without rewriting the ciphertext");
-    assert_eq!(get.server_side_encryption(), Some(&ServerSideEncryption::Aes256));
-    assert_eq!(
-        get.metadata().and_then(|m| m.get("stage")),
-        Some(&"after".to_string()),
-        "REPLACE metadata must take effect"
-    );
-    let body = get.body.collect().await.expect("collect body").into_bytes();
-    assert_eq!(
-        body.as_ref(),
-        content,
-        "object must still decrypt to the original plaintext after a metadata-only self copy"
-    );
-
-    kms_env.base_env.stop_server();
-}
-
-#[tokio::test]
-#[serial]
-async fn test_metadata_replace_self_copy_dropping_sse_rewrites_plaintext() {
-    init_logging();
-    info!("same-key CopyObject that drops SSE must rewrite the data, not orphan the ciphertext");
-
-    let mut kms_env = LocalKMSTestEnvironment::new().await.expect("failed to create local KMS env");
-    let default_key_id = "rustfs-e2e-test-default-key";
-    let keys_dir = kms_env.kms_keys_dir.clone();
-    create_key_with_specific_id(&keys_dir, default_key_id)
-        .await
-        .expect("failed to create local KMS key");
-    kms_env
-        .base_env
-        .start_rustfs_server_with_env(
-            vec![
-                "--kms-enable",
-                "--kms-backend",
-                "local",
-                "--kms-key-dir",
-                &keys_dir,
-                "--kms-default-key-id",
-                default_key_id,
-            ],
-            &[("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")],
-        )
-        .await
-        .expect("failed to start RustFS with local KMS");
-    tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;
-
-    let client = kms_env.base_env.create_s3_client();
-    // Unversioned, and deliberately WITHOUT a bucket default-encryption rule, so the copy below
-    // resolves to "no destination encryption".
-    let bucket = "copy-object-self-copy-drop-sse-test";
-    let key = "secrets/report.txt";
-
-    client
-        .create_bucket()
-        .bucket(bucket)
-        .send()
-        .await
-        .expect("failed to create bucket");
-
-    let content = b"encrypted payload whose ciphertext must not survive as bogus plaintext -- 0123456789";
-    client
-        .put_object()
-        .bucket(bucket)
-        .key(key)
-        .metadata("stage", "before")
-        .server_side_encryption(ServerSideEncryption::Aes256)
-        .body(ByteStream::from_static(content))
-        .send()
-        .await
-        .expect("PUT failed");
-
-    // Self-copy with REPLACE and no SSE header. Per AWS semantics the destination ends up
-    // unencrypted. The dangerous outcome is the silent one: the handler strips the source key
-    // metadata while a metadata-only copy leaves the ciphertext in place, so a later GET would
-    // hand back raw ciphertext as if it were plaintext — corruption with no error anywhere.
-    client
-        .copy_object()
-        .bucket(bucket)
-        .key(key)
-        .copy_source(format!("{bucket}/{key}"))
-        .metadata_directive(MetadataDirective::Replace)
-        .metadata("stage", "after")
-        .send()
-        .await
-        .expect("same-key CopyObject dropping SSE must succeed");
-
-    let get = client
-        .get_object()
-        .bucket(bucket)
-        .key(key)
-        .send()
-        .await
-        .expect("GET after self-copy failed");
-    assert_eq!(
-        get.server_side_encryption(),
-        None,
-        "destination must be unencrypted once the copy drops SSE"
-    );
-    assert_eq!(
-        get.metadata().and_then(|m| m.get("stage")),
-        Some(&"after".to_string()),
-        "REPLACE metadata must take effect"
-    );
-    let body = get.body.collect().await.expect("collect body").into_bytes();
-    assert_eq!(
-        body.as_ref(),
-        content,
-        "object must read back as the original plaintext, not the orphaned ciphertext"
-    );
-
-    kms_env.base_env.stop_server();
-}
-
-#[tokio::test]
-#[serial]
-async fn test_metadata_replace_self_copy_under_bucket_default_sse_stays_decryptable() {
-    init_logging();
-    info!("bucket default encryption must also keep a same-key copy off the metadata-only path");
-
-    let mut kms_env = LocalKMSTestEnvironment::new().await.expect("failed to create local KMS env");
-    let default_key_id = "rustfs-e2e-test-default-key";
-    let keys_dir = kms_env.kms_keys_dir.clone();
-    create_key_with_specific_id(&keys_dir, default_key_id)
-        .await
-        .expect("failed to create local KMS key");
-    kms_env
-        .base_env
-        .start_rustfs_server_with_env(
-            vec![
-                "--kms-enable",
-                "--kms-backend",
-                "local",
-                "--kms-key-dir",
-                &keys_dir,
-                "--kms-default-key-id",
-                default_key_id,
-            ],
-            &[("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")],
-        )
-        .await
-        .expect("failed to start RustFS with local KMS");
-    tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;
-
-    let client = kms_env.base_env.create_s3_client();
-    let bucket = "copy-object-self-copy-bucket-default-sse-test";
-    let key = "secrets/report.txt";
-
-    client
-        .create_bucket()
-        .bucket(bucket)
-        .send()
-        .await
-        .expect("failed to create bucket");
-
-    // Store the object as PLAINTEXT first: no SSE header and no bucket default rule yet. This is
-    // what makes the case sharp — at copy time the source metadata carries no encryption markers,
-    // so the source-side half of the guard cannot fire.
-    let content = b"plaintext payload that must not be orphaned under a new DEK -- 0123456789";
-    let put = client
-        .put_object()
-        .bucket(bucket)
-        .key(key)
-        .metadata("stage", "before")
-        .body(ByteStream::from_static(content))
-        .send()
-        .await
-        .expect("PUT failed");
-    assert_eq!(put.server_side_encryption(), None, "the object must start out unencrypted");
-
-    // Only NOW enable bucket default encryption. The destination's encryption therefore comes
-    // from the bucket rule and from nowhere else: the source is unencrypted and the copy request
-    // carries no SSE header. A guard that only inspects request headers (MinIO decides
-    // `isTargetEncrypted` from `crypto.S3.IsRequested(r.Header)`) would let this through, yet
-    // `sse_encryption` still mints a fresh DEK from the resolved bucket default — which is why
-    // the guard keys off the *effective* encryption rather than the requested one.
-    let encryption_config = ServerSideEncryptionConfiguration::builder()
-        .rules(
-            ServerSideEncryptionRule::builder()
-                .apply_server_side_encryption_by_default(
-                    ServerSideEncryptionByDefault::builder()
-                        .sse_algorithm(ServerSideEncryption::Aes256)
-                        .build()
-                        .unwrap(),
-                )
-                .build(),
-        )
-        .build()
-        .unwrap();
-    client
-        .put_bucket_encryption()
-        .bucket(bucket)
-        .server_side_encryption_configuration(encryption_config)
-        .send()
-        .await
-        .expect("failed to set bucket default encryption");
-
-    // No SSE header on the copy — the bucket default alone drives the destination encryption.
-    client
-        .copy_object()
-        .bucket(bucket)
-        .key(key)
-        .copy_source(format!("{bucket}/{key}"))
-        .metadata_directive(MetadataDirective::Replace)
-        .metadata("stage", "after")
-        .send()
-        .await
-        .expect("same-key CopyObject under bucket default encryption must succeed");
-
-    let get = client
-        .get_object()
-        .bucket(bucket)
-        .key(key)
-        .send()
-        .await
-        .expect("GET after self-copy failed: the object was re-keyed without rewriting the ciphertext");
-    assert_eq!(get.server_side_encryption(), Some(&ServerSideEncryption::Aes256));
-    assert_eq!(
-        get.metadata().and_then(|m| m.get("stage")),
-        Some(&"after".to_string()),
-        "REPLACE metadata must take effect"
-    );
-    let body = get.body.collect().await.expect("collect body").into_bytes();
-    assert_eq!(
-        body.as_ref(),
-        content,
-        "object must still decrypt to the original plaintext after a metadata-only self copy"
-    );
-
-    kms_env.base_env.stop_server();
-}

crates/e2e_test/src/kms/kms_ilm_sse_kms_test.rs

@@ -1,612 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! ILM on SSE-KMS buckets while per-key SSE authorization is enforced (backlog#1582).
-//!
-//! Per-key KMS authorization (`RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY=true`) scopes the
-//! SSE-KMS data path to the requesting principal's `kms:GenerateDataKey` /
-//! `kms:Decrypt` grants. Internal callers — the lifecycle scanner's expiry deletes
-//! and the tier transition worker's reads — carry no request principal, and
-//! `authorize_sse_kms_key` (rustfs/src/storage/sse.rs) exempts a `None` principal
-//! so background maintenance keeps working on encrypted buckets.
-//!
-//! These tests pin that exemption end to end. If enforcement ever starts applying
-//! to the scanner's internal operations, expiry stops happening on SSE-KMS buckets
-//! and [`ilm_expiration_on_sse_kms_bucket_under_enforcement`] times out; if it
-//! starts applying to the transition worker or the read-through path,
-//! [`ilm_transition_on_sse_kms_bucket_under_enforcement_reads_back`] fails at the
-//! transition wait or the plaintext round-trip.
-//!
-//! The replication half of the same acceptance item lives in
-//! `crates/e2e_test/src/replication_extension_test.rs`
-//! (`test_bucket_replication_sse_kms_failure_contract`); ILM had no coverage
-//! before this file.
-//!
-//! Deployment constraint pinned by the transition test's setup: the RustFS warm
-//! backend forwards the object's stored `x-amz-server-side-encryption*` metadata
-//! as raw headers on the tier data PUT (`build_transition_put_options` +
-//! `api_put_object.rs` header mapping), so a RustFS tier target must itself have
-//! KMS enabled and hold the named key or it rejects every transition upload with
-//! 400 InvalidRequest. That rejection is independent of the enforcement switch;
-//! the cold server here therefore runs its own Local KMS with the same key id.
-
-use super::common::{LocalKMSTestEnvironment, create_key_with_specific_id};
-use crate::common::{RustFSTestEnvironment, admin_request, init_logging};
-use aws_sdk_s3::Client;
-use aws_sdk_s3::primitives::ByteStream;
-use aws_sdk_s3::types::{
-    BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter, RestoreRequest,
-    ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule, Transition,
-    TransitionStorageClass,
-};
-use serde::Deserialize;
-use serial_test::serial;
-use std::time::{Duration as StdDuration, Instant};
-use tracing::info;
-
-type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
-
-const SSE_KEY: &str = "kms-ilm-sse-key";
-const PAYLOAD: &[u8] = b"kms ilm sse payload: survives enforcement, expires and transitions on schedule";
-
-const EXPIRY_BUCKET: &str = "kms-ilm-expiry";
-const EXPIRE_KEY: &str = "expire/object.bin";
-const SURVIVOR_KEY: &str = "keep/object.bin";
-
-const TIER_NAME: &str = "KMSCOLD";
-const TIER_BUCKET: &str = "kms-ilm-cold-tier";
-const TIER_PREFIX: &str = "tiered";
-const TRANSITION_BUCKET: &str = "kms-ilm-transition";
-const TRANSITION_KEY: &str = "tier/object.bin";
-
-/// Generous CI safety net; with a 1s scanner cycle and 2s lifecycle days the
-/// terminal state normally lands within a few seconds.
-const ILM_DEADLINE: StdDuration = StdDuration::from_secs(90);
-
-/// Start a Local-KMS server with per-key SSE authorization enforced and the
-/// lifecycle clock accelerated.
-///
-/// KMS wiring matches `kms_authorization_negative_matrix_test.rs` (local backend,
-/// `--kms-default-key-id`, insecure dev defaults). The lifecycle env matches
-/// `reliant/lifecycle.rs::fast_lifecycle_env` plus `RUSTFS_ILM_DEBUG_DAY_SECS=2`,
-/// so a `Days=1` rule is due about two seconds after the write.
-async fn start_enforcing_ilm_server(env: &mut LocalKMSTestEnvironment) -> TestResult {
-    create_key_with_specific_id(&env.kms_keys_dir, SSE_KEY).await?;
-
-    let key_dir = env.kms_keys_dir.clone();
-    let args = vec![
-        "--kms-enable",
-        "--kms-backend",
-        "local",
-        "--kms-key-dir",
-        key_dir.as_str(),
-        "--kms-default-key-id",
-        SSE_KEY,
-    ];
-
-    let envs = [
-        ("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true"),
-        ("RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY", "false"),
-        ("RUSTFS_SCANNER_CYCLE", "1"),
-        ("RUSTFS_ILM_PROCESS_TIME", "1"),
-        ("RUSTFS_ILM_DEBUG_DAY_SECS", "2"),
-    ];
-
-    env.base_env.start_rustfs_server_with_env(args, &envs).await?;
-    Ok(())
-}
-
-/// Set the bucket's default encryption to SSE-KMS under [`SSE_KEY`], so plain
-/// PUTs (and internal rewrites) are encrypted without per-request SSE headers.
-async fn set_bucket_default_sse_kms(client: &Client, bucket: &str) -> TestResult {
-    let encryption_config = ServerSideEncryptionConfiguration::builder()
-        .rules(
-            ServerSideEncryptionRule::builder()
-                .apply_server_side_encryption_by_default(
-                    ServerSideEncryptionByDefault::builder()
-                        .sse_algorithm(ServerSideEncryption::AwsKms)
-                        .kms_master_key_id(SSE_KEY)
-                        .build()?,
-                )
-                .build(),
-        )
-        .build()?;
-    client
-        .put_bucket_encryption()
-        .bucket(bucket)
-        .server_side_encryption_configuration(encryption_config)
-        .send()
-        .await?;
-    Ok(())
-}
-
-/// Assert via `HeadObject` that the stored object is SSE-KMS encrypted under
-/// [`SSE_KEY`]. Without this, a bucket-default misconfiguration would let the
-/// tests pass on an unencrypted object and prove nothing about KMS.
-async fn assert_head_sse_kms(client: &Client, bucket: &str, key: &str) -> TestResult {
-    let head = client.head_object().bucket(bucket).key(key).send().await?;
-    assert_eq!(
-        head.server_side_encryption(),
-        Some(&ServerSideEncryption::AwsKms),
-        "{bucket}/{key} must be SSE-KMS encrypted via the bucket default"
-    );
-    assert_eq!(
-        head.ssekms_key_id(),
-        Some(SSE_KEY),
-        "{bucket}/{key} must be wrapped under the configured KMS key"
-    );
-    Ok(())
-}
-
-/// Returns `true` once `GET bucket/key` fails with `NoSuchKey`, `false` while it
-/// still succeeds. Any other error is surfaced. (Copied from
-/// `reliant/lifecycle.rs`; that helper is private to the reliant module.)
-async fn object_is_gone(client: &Client, bucket: &str, key: &str) -> Result<bool, Box<dyn std::error::Error + Send + Sync>> {
-    match client.get_object().bucket(bucket).key(key).send().await {
-        Ok(output) => {
-            output.body.collect().await?;
-            Ok(false)
-        }
-        Err(e) => {
-            if let Some(service_error) = e.as_service_error() {
-                if service_error.is_no_such_key() {
-                    return Ok(true);
-                }
-                return Err(format!("expected NoSuchKey, got: {e:?}").into());
-            }
-            Err(format!("expected a service error, got: {e:?}").into())
-        }
-    }
-}
-
-/// Poll until `GET bucket/key` returns `NoSuchKey`, or fail after `deadline`.
-async fn wait_for_object_expired(client: &Client, bucket: &str, key: &str, deadline: StdDuration) -> TestResult {
-    let start = Instant::now();
-    loop {
-        if object_is_gone(client, bucket, key).await? {
-            return Ok(());
-        }
-        if start.elapsed() >= deadline {
-            return Err(format!(
-                "object {bucket}/{key} was not expired by the lifecycle scanner within {}s; \
-                 SSE key-policy enforcement may have started blocking the scanner's internal deletes",
-                deadline.as_secs()
-            )
-            .into());
-        }
-        tokio::time::sleep(StdDuration::from_millis(500)).await;
-    }
-}
-
-/// Install a prefix-scoped `Days`-based expiration rule.
-async fn put_expiration_rule(client: &Client, bucket: &str, id: &str, prefix: &str, days: i32) -> TestResult {
-    let rule = LifecycleRule::builder()
-        .id(id)
-        .filter(LifecycleRuleFilter::builder().prefix(prefix).build())
-        .expiration(LifecycleExpiration::builder().days(days).build())
-        .status(ExpirationStatus::Enabled)
-        .build()?;
-    let lifecycle = BucketLifecycleConfiguration::builder().rules(rule).build()?;
-    client
-        .put_bucket_lifecycle_configuration()
-        .bucket(bucket)
-        .lifecycle_configuration(lifecycle)
-        .send()
-        .await?;
-    Ok(())
-}
-
-/// Install a prefix-scoped `Days`-based transition rule targeting [`TIER_NAME`].
-async fn put_transition_rule(client: &Client, bucket: &str, id: &str, prefix: &str, days: i32) -> TestResult {
-    let rule = LifecycleRule::builder()
-        .id(id)
-        .filter(LifecycleRuleFilter::builder().prefix(prefix).build())
-        .transitions(
-            Transition::builder()
-                .days(days)
-                .storage_class(TransitionStorageClass::from(TIER_NAME))
-                .build(),
-        )
-        .status(ExpirationStatus::Enabled)
-        .build()?;
-    let lifecycle = BucketLifecycleConfiguration::builder().rules(rule).build()?;
-    client
-        .put_bucket_lifecycle_configuration()
-        .bucket(bucket)
-        .lifecycle_configuration(lifecycle)
-        .send()
-        .await?;
-    Ok(())
-}
-
-/// Start a plain Local-KMS server (no enforcement, no lifecycle acceleration)
-/// holding [`SSE_KEY`], to serve as the cold tier target.
-///
-/// The RustFS warm backend forwards the stored SSE-KMS headers on the tier data
-/// PUT, so the target re-applies managed SSE-KMS under the named key and must
-/// be able to resolve it; without KMS it answers 400 InvalidRequest and the
-/// transition can never complete. Enforcement stays off here: the tier writes
-/// arrive under `cold`'s root credentials, and one enforcing side is enough to
-/// pin the exemption.
-async fn start_cold_tier_kms_server(env: &mut LocalKMSTestEnvironment) -> TestResult {
-    create_key_with_specific_id(&env.kms_keys_dir, SSE_KEY).await?;
-
-    let key_dir = env.kms_keys_dir.clone();
-    let args = vec![
-        "--kms-enable",
-        "--kms-backend",
-        "local",
-        "--kms-key-dir",
-        key_dir.as_str(),
-        "--kms-default-key-id",
-        SSE_KEY,
-    ];
-
-    env.base_env
-        .start_rustfs_server_with_env(args, &[("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")])
-        .await?;
-    Ok(())
-}
-
-/// The subset of the manual transition run report these tests assert on.
-///
-/// Unknown fields are ignored, so this stays compatible with report growth; the
-/// full shape is pinned by `reliant/tiering.rs`.
-#[derive(Debug, Deserialize)]
-struct ManualTransitionRunReport {
-    #[serde(default)]
-    scanned: u64,
-    #[serde(default)]
-    enqueued: u64,
-    #[serde(default)]
-    skipped_already_in_flight: u64,
-    #[serde(default)]
-    skipped_tier: u64,
-}
-
-#[derive(Debug, Deserialize)]
-struct ManualTransitionRunResponse {
-    state: String,
-    report: ManualTransitionRunReport,
-}
-
-/// One synchronous (enqueue-only) manual transition run over `bucket/prefix`,
-/// via the same admin endpoint `reliant/tiering.rs` drives.
-async fn manual_transition_run(
-    hot: &RustFSTestEnvironment,
-    bucket: &str,
-    prefix: &str,
-) -> Result<ManualTransitionRunResponse, Box<dyn std::error::Error + Send + Sync>> {
-    let bucket = urlencoding::encode(bucket);
-    let prefix = urlencoding::encode(prefix);
-    let tier = urlencoding::encode(TIER_NAME);
-    let path =
-        format!("/rustfs/admin/v3/ilm/transition/run?bucket={bucket}&prefix={prefix}&tier={tier}&dryRun=false&maxObjects=10");
-    let (status, body) = admin_request(&hot.url, http::Method::POST, &path, None, &hot.access_key, &hot.secret_key).await?;
-    if !status.is_success() {
-        return Err(format!("manual transition run failed: status={status}, body={body}").into());
-    }
-    Ok(serde_json::from_str(&body)?)
-}
-
-/// Drive manual transition runs until one reports the object as processed.
-///
-/// The `Days=1` rule becomes due about two seconds after the write
-/// (`RUSTFS_ILM_DEBUG_DAY_SECS=2`), so early runs may legitimately report the
-/// object as not yet eligible; the loop keeps running the endpoint until it
-/// either enqueues the transition, sees it already in flight (the 1s scanner
-/// backstop got there first), or finds it already on the tier.
-async fn run_manual_transition_until_processed(
-    hot: &RustFSTestEnvironment,
-    bucket: &str,
-    prefix: &str,
-    deadline: StdDuration,
-) -> TestResult {
-    let start = Instant::now();
-    loop {
-        let run = manual_transition_run(hot, bucket, prefix).await?;
-        assert_eq!(run.report.scanned, 1, "manual transition run must scan the object: {run:#?}");
-        if run.report.enqueued + run.report.skipped_already_in_flight + run.report.skipped_tier >= 1 {
-            info!(state = %run.state, report = ?run.report, "manual transition run processed the SSE-KMS object");
-            return Ok(());
-        }
-        if start.elapsed() >= deadline {
-            return Err(format!(
-                "manual transition runs never processed {bucket}/{prefix} within {}s; last report: {run:#?}",
-                deadline.as_secs()
-            )
-            .into());
-        }
-        tokio::time::sleep(StdDuration::from_millis(500)).await;
-    }
-}
-
-/// Wire `hot` -> `cold` as a `TierType::RustFS` remote tier via `AddTier`.
-///
-/// No `force`, so the server runs the real connectivity probe against `cold`
-/// (the tier bucket must already exist there). Mirrors
-/// `reliant/tiering.rs::add_rustfs_tier`, which is private to that module.
-async fn add_rustfs_tier(hot: &RustFSTestEnvironment, cold: &RustFSTestEnvironment) -> TestResult {
-    let body = serde_json::json!({
-        "type": "rustfs",
-        "rustfs": {
-            "name": TIER_NAME,
-            "endpoint": cold.url.as_str(),
-            "accessKey": cold.access_key.as_str(),
-            "secretKey": cold.secret_key.as_str(),
-            "bucket": TIER_BUCKET,
-            "prefix": TIER_PREFIX,
-            "region": "us-east-1",
-            "storageClass": ""
-        }
-    })
-    .to_string();
-
-    let (status, resp) = admin_request(
-        &hot.url,
-        http::Method::PUT,
-        "/rustfs/admin/v3/tier",
-        Some(body),
-        &hot.access_key,
-        &hot.secret_key,
-    )
-    .await?;
-    if !status.is_success() {
-        return Err(format!("AddTier(RustFS) failed: status={status}, body={resp}").into());
-    }
-    Ok(())
-}
-
-/// Poll `HEAD` until the object's storage class is the tier name (transition
-/// complete), or fail after `deadline`. (From `reliant/tiering.rs`.)
-async fn wait_for_transition(client: &Client, bucket: &str, key: &str, deadline: StdDuration) -> TestResult {
-    let start = Instant::now();
-    loop {
-        let head = client.head_object().bucket(bucket).key(key).send().await?;
-        if head.storage_class().map(|sc| sc.as_str()) == Some(TIER_NAME) {
-            return Ok(());
-        }
-        if start.elapsed() >= deadline {
-            return Err(format!(
-                "object {bucket}/{key} was not transitioned to {TIER_NAME} within {}s (storage_class={:?}); \
-                 SSE key-policy enforcement may have started blocking the transition worker's internal reads",
-                deadline.as_secs(),
-                head.storage_class()
-            )
-            .into());
-        }
-        tokio::time::sleep(StdDuration::from_millis(500)).await;
-    }
-}
-
-/// Poll `HEAD` until `x-amz-restore` reports a finished restore
-/// (`ongoing-request="false"`), or fail after `deadline`.
-async fn wait_for_restore_complete(client: &Client, bucket: &str, key: &str, deadline: StdDuration) -> TestResult {
-    let start = Instant::now();
-    loop {
-        let head = client.head_object().bucket(bucket).key(key).send().await?;
-        if head.restore().is_some_and(|r| r.contains("ongoing-request=\"false\"")) {
-            return Ok(());
-        }
-        if start.elapsed() >= deadline {
-            return Err(format!(
-                "object {bucket}/{key} restore did not complete within {}s (restore={:?}); \
-                 SSE key-policy enforcement may have started blocking the restore copy-back's internal reads",
-                deadline.as_secs(),
-                head.restore()
-            )
-            .into());
-        }
-        tokio::time::sleep(StdDuration::from_millis(500)).await;
-    }
-}
-
-/// ILM expiration keeps working on an SSE-KMS bucket while per-key SSE
-/// authorization is enforced.
-///
-/// The lifecycle scanner deletes expired objects with an internal (no-principal)
-/// identity that holds no `kms` grant. If enforcement ever starts applying to
-/// those internal deletes (or to the scanner's metadata reads) on encrypted
-/// buckets, expiry stops happening and this test times out.
-///
-/// A survivor object under a non-matching prefix isolates the rule's prefix
-/// filter as the cause of the deletion and proves the encrypted bucket stays
-/// readable end to end after the scanner has run.
-#[tokio::test]
-#[serial]
-async fn ilm_expiration_on_sse_kms_bucket_under_enforcement() -> TestResult {
-    init_logging();
-
-    let mut env = LocalKMSTestEnvironment::new().await?;
-    start_enforcing_ilm_server(&mut env).await?;
-    env.base_env.create_test_bucket(EXPIRY_BUCKET).await?;
-
-    let client = env.base_env.create_s3_client();
-    set_bucket_default_sse_kms(&client, EXPIRY_BUCKET).await?;
-
-    for key in [EXPIRE_KEY, SURVIVOR_KEY] {
-        client
-            .put_object()
-            .bucket(EXPIRY_BUCKET)
-            .key(key)
-            .body(ByteStream::from_static(PAYLOAD))
-            .send()
-            .await?;
-        assert_head_sse_kms(&client, EXPIRY_BUCKET, key).await?;
-    }
-    info!("both objects stored SSE-KMS encrypted under enforcement");
-
-    put_expiration_rule(&client, EXPIRY_BUCKET, "kms-ilm-expire", "expire/", 1).await?;
-
-    // The regression this pins: the scanner's internal delete must stay exempt
-    // from per-key SSE authorization, so the encrypted object actually expires.
-    wait_for_object_expired(&client, EXPIRY_BUCKET, EXPIRE_KEY, ILM_DEADLINE).await?;
-    info!("SSE-KMS object expired by the lifecycle scanner under enforcement");
-
-    // Negative control: same bucket, same encryption, non-matching prefix. It
-    // must survive the scanner and still decrypt for the requesting principal.
-    assert!(
-        !object_is_gone(&client, EXPIRY_BUCKET, SURVIVOR_KEY).await?,
-        "non-matching-prefix object must not be expired by a prefix-scoped rule"
-    );
-    let survivor = client.get_object().bucket(EXPIRY_BUCKET).key(SURVIVOR_KEY).send().await?;
-    assert_eq!(
-        survivor.body.collect().await?.into_bytes().as_ref(),
-        PAYLOAD,
-        "surviving SSE-KMS object must still decrypt after the scanner has run"
-    );
-
-    Ok(())
-}
-
-/// ILM transition to a remote tier keeps working on an SSE-KMS bucket while
-/// per-key SSE authorization is enforced, and the transitioned object reads
-/// back as plaintext.
-///
-/// The transition worker moves the stored (encrypted) bytes to the cold tier
-/// with an internal (no-principal) identity; the read-through `GET` then
-/// decrypts the envelope for the requesting principal. If enforcement ever
-/// starts applying to the worker's internal reads, the transition wait times
-/// out; if the stored envelope is mishandled across the tier round trip, the
-/// plaintext comparison fails.
-///
-/// The transition is driven through the manual transition-run admin endpoint
-/// (the mechanism `reliant/tiering.rs` established), so the test does not
-/// depend on scanner scheduling; the 1s scanner cycle stays on as a backstop.
-#[tokio::test]
-#[serial]
-#[ignore = "pins rustfs/rustfs#6025: GET on a transitioned managed-SSE object silently returns corrupt bytes (fails with enforcement on AND off, so it is not an authorization regression); un-ignore with the fix"]
-async fn ilm_transition_on_sse_kms_bucket_under_enforcement_reads_back() -> TestResult {
-    init_logging();
-
-    // Cold-tier server: independent credentials, its own Local KMS holding the
-    // same key id (see the module docs for why the tier target needs KMS).
-    // Started first; each server's startup cleanup only matches its own unique
-    // address and temp dir, so the two instances coexist.
-    let mut cold = LocalKMSTestEnvironment::new().await?;
-    cold.base_env.access_key = "kmscoldtieradmin".to_string();
-    cold.base_env.secret_key = "kmscoldtiersecret".to_string();
-    start_cold_tier_kms_server(&mut cold).await?;
-    let cold_client = cold.base_env.create_s3_client();
-    cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
-
-    // Hot server: Local KMS + enforcement + accelerated lifecycle clock.
-    let mut env = LocalKMSTestEnvironment::new().await?;
-    start_enforcing_ilm_server(&mut env).await?;
-    let hot_client = env.base_env.create_s3_client();
-
-    add_rustfs_tier(&env.base_env, &cold.base_env).await?;
-
-    env.base_env.create_test_bucket(TRANSITION_BUCKET).await?;
-    set_bucket_default_sse_kms(&hot_client, TRANSITION_BUCKET).await?;
-
-    hot_client
-        .put_object()
-        .bucket(TRANSITION_BUCKET)
-        .key(TRANSITION_KEY)
-        .body(ByteStream::from_static(PAYLOAD))
-        .send()
-        .await?;
-    assert_head_sse_kms(&hot_client, TRANSITION_BUCKET, TRANSITION_KEY).await?;
-    info!("object stored SSE-KMS encrypted under enforcement");
-
-    // Days=1 is due ~2s after the write with RUSTFS_ILM_DEBUG_DAY_SECS=2.
-    put_transition_rule(&hot_client, TRANSITION_BUCKET, "kms-ilm-transition", "tier/", 1).await?;
-
-    // Drive the transition deterministically via the manual run endpoint, then
-    // wait for HEAD to report the tier as the object's storage class.
-    run_manual_transition_until_processed(&env.base_env, TRANSITION_BUCKET, "tier/", ILM_DEADLINE).await?;
-    wait_for_transition(&hot_client, TRANSITION_BUCKET, TRANSITION_KEY, ILM_DEADLINE).await?;
-    info!("SSE-KMS object transitioned to the remote tier under enforcement");
-
-    let head = hot_client
-        .head_object()
-        .bucket(TRANSITION_BUCKET)
-        .key(TRANSITION_KEY)
-        .send()
-        .await?;
-    assert!(
-        head.restore().is_none(),
-        "a freshly transitioned object must not advertise x-amz-restore, got {:?}",
-        head.restore()
-    );
-
-    // The remote copy exists on the cold tier. The payload the tier holds is the
-    // hot server's stored ciphertext, wrapped once more under the cold server's
-    // own managed SSE-KMS layer (the forwarded headers re-request encryption).
-    let remote = cold_client.list_objects_v2().bucket(TIER_BUCKET).send().await?;
-    assert!(!remote.contents().is_empty(), "cold-tier bucket must hold the transitioned object's data");
-
-    // Read-through GET under enforcement must succeed (not AccessDenied) and
-    // keep advertising SSE-KMS. Its BODY is deliberately not compared here:
-    // the transitioned read path skips managed-SSE decryption — a product gap
-    // unrelated to enforcement — so a direct GET streams the stored ciphertext
-    // (`new_getobjectreader` in crates/ecstore/src/client/object_api_utils.rs
-    // hardcodes `is_encrypted = false` and never applies the
-    // `ReadTransform::Encrypted` wrapping the hot-read path builds in
-    // crates/ecstore/src/object_api/readers.rs). Plaintext recovery is pinned
-    // through restore semantics below; when the read-through gap is fixed, a
-    // byte assertion can be added here too.
-    let read_through = hot_client
-        .get_object()
-        .bucket(TRANSITION_BUCKET)
-        .key(TRANSITION_KEY)
-        .send()
-        .await?;
-    assert_eq!(
-        read_through.server_side_encryption(),
-        Some(&ServerSideEncryption::AwsKms),
-        "transitioned object must still report SSE-KMS on read-through"
-    );
-    let read_through_body = read_through.body.collect().await?.into_bytes();
-    assert_eq!(
-        read_through_body.len(),
-        PAYLOAD.len(),
-        "read-through GET must stream the object's full logical size under enforcement"
-    );
-
-    // RestoreObject copies the ciphertext back from the tier under the original
-    // envelope metadata; the restored copy is then served by the normal
-    // decrypting read path. The copy-back runs with an internal (no-principal)
-    // identity, so this also pins the exemption on the restore path. Days=300
-    // because RUSTFS_ILM_DEBUG_DAY_SECS=2 accelerates the restored copy's
-    // expiry as well (300 accelerated days == 600s of validity).
-    hot_client
-        .restore_object()
-        .bucket(TRANSITION_BUCKET)
-        .key(TRANSITION_KEY)
-        .restore_request(RestoreRequest::builder().days(300).build())
-        .send()
-        .await?;
-    wait_for_restore_complete(&hot_client, TRANSITION_BUCKET, TRANSITION_KEY, ILM_DEADLINE).await?;
-    info!("SSE-KMS object restored from the remote tier under enforcement");
-
-    // The KMS-relevant half: the restored envelope decrypts back to the exact
-    // plaintext for the requesting principal.
-    let restored = hot_client
-        .get_object()
-        .bucket(TRANSITION_BUCKET)
-        .key(TRANSITION_KEY)
-        .send()
-        .await?;
-    assert_eq!(
-        restored.server_side_encryption(),
-        Some(&ServerSideEncryption::AwsKms),
-        "restored object must still report SSE-KMS"
-    );
-    let body = restored.body.collect().await?.into_bytes();
-    assert_eq!(body.as_ref(), PAYLOAD, "restored SSE-KMS object must round-trip byte-identical plaintext");
-
-    Ok(())
-}

crates/e2e_test/src/kms/mod.rs

@@ -48,9 +48,6 @@ mod bucket_default_encryption_test;
 #[cfg(test)]
 mod encryption_metadata_test;
 
-#[cfg(test)]
-mod copy_object_self_copy_sse_test;
-
 #[cfg(test)]
 mod copy_object_version_restore_sse_test;
 
@@ -59,6 +56,3 @@ mod configured_roundtrip_test;
 
 #[cfg(test)]
 mod kms_authorization_negative_matrix_test;
-
-#[cfg(test)]
-mod kms_ilm_sse_kms_test;

crates/e2e_test/src/lib.rs

@@ -39,10 +39,6 @@ pub mod fault_proxy;
 #[cfg(test)]
 mod reliability_disk_fault_test;
 
-// Privileged Linux-only 3x4 replacement rebuild proof for rustfs#5869/#1791.
-#[cfg(all(test, target_os = "linux"))]
-mod replacement_privileged_e2e_test;
-
 // dist-13 (backlog#1150/#1155): e2e regression net proving a large-object
 // degraded EC read never returns a silently truncated body (rustfs#4594/#4560/#4585).
 #[cfg(test)]
@@ -294,14 +290,6 @@ mod overwrite_cleanup_regression_test;
 #[cfg(test)]
 mod list_buckets_double_slash_test;
 
-// Regression coverage for bucket-scoped ListBuckets authorization fallback.
-#[cfg(test)]
-mod list_buckets_auth_test;
-
-// ListBuckets visibility follows IAM authorization, not bucket policy.
-#[cfg(test)]
-mod list_buckets_iam_filter_test;
-
 // Regression test for backlog#629(b): region-aware CreateBucket SigV4.
 #[cfg(test)]
 mod create_bucket_region_test;
@@ -310,32 +298,4 @@ mod create_bucket_region_test;
 #[cfg(test)]
 mod copy_source_invalid_date_test;
 
-// P0 regression: event notification startup race (rustfs#5387, #5681, #5401, #5183, #5115, #4796)
-#[cfg(test)]
-mod notification_startup_regression_test;
-
-// P0 regression: lifecycle/ILM object expiration (rustfs#5407, #5167, #4963, #5615, #4879)
-#[cfg(test)]
-mod lifecycle_regression_test;
-
-// P0 regression: delete operations consistency (rustfs#5375, #5349, #5339, #5029, #4978, #760)
-#[cfg(test)]
-mod delete_regression_test;
-
-// P1 regression: listing/metacache completeness (rustfs#5166, #5156, #5051, #4810, #4648, #3191)
-#[cfg(test)]
-mod listing_regression_test;
-
-// P1 regression: bucket statistics accuracy (rustfs#5615, #5008, #5116, #5055, #3898, #1012)
-#[cfg(test)]
-mod bucket_stats_regression_test;
-
-// P1 regression: distributed startup/quorum (rustfs#5416, #2945, #2794, #2601, #4040, #5655)
-#[cfg(test)]
-mod distributed_startup_regression_test;
-
-// P1 regression: tier/ILM transition (rustfs#5218, #5130, #5011, #4826, #5024)
-#[cfg(test)]
-mod tier_transition_regression_test;
-
 pub mod tls_gen;

crates/e2e_test/src/lifecycle_regression_test.rs

@@ -1,360 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression tests for lifecycle/ILM object expiration and transition.
-//!
-//! Covers the recurring pattern where ILM expiration rules do not actually
-//! delete objects, or lifecycle rule parameters are silently corrupted.
-//! This has regressed 6+ times.
-//!
-//! ## Regression Issues
-//!
-//! - rustfs#5407: lifecycle not delete any bucket object
-//! - rustfs#5167: lifecycle not delete object
-//! - rustfs#4963: lifecycle rule 3 days → effective value 0 days
-//! - rustfs#5615: bucket statistics remain unchanged after data expiration
-//! - rustfs#4879: ILM serial lane: restore transition never completes
-//! - rustfs#5442: Uncheck of Replicate Delete still deletes the file
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestEnvironment, init_logging};
-    use aws_sdk_s3::Client;
-    use aws_sdk_s3::primitives::ByteStream;
-    use aws_sdk_s3::types::{
-        BucketLifecycleConfiguration, BucketVersioningStatus, ExpirationStatus, LifecycleExpiration, LifecycleRule,
-        LifecycleRuleFilter, NoncurrentVersionExpiration, VersioningConfiguration,
-    };
-    use serial_test::serial;
-    use std::error::Error;
-    use tracing::info;
-
-    type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-    async fn setup_versioned_bucket(client: &Client, bucket: &str) -> TestResult {
-        client
-            .create_bucket()
-            .bucket(bucket)
-            .send()
-            .await
-            .map_err(|e| format!("create bucket: {e}"))?;
-
-        client
-            .put_bucket_versioning()
-            .bucket(bucket)
-            .versioning_configuration(
-                VersioningConfiguration::builder()
-                    .status(BucketVersioningStatus::Enabled)
-                    .build(),
-            )
-            .send()
-            .await
-            .map_err(|e| format!("enable versioning: {e}"))?;
-
-        Ok(())
-    }
-
-    /// RT-03: Verify that a lifecycle expiration rule actually deletes objects.
-    ///
-    /// Regression pattern: lifecycle rules are accepted but the scanner never
-    /// processes them, leaving expired objects in place.
-    ///
-    /// Steps:
-    /// 1. Create a versioned bucket
-    /// 2. Upload several objects
-    /// 3. Apply a lifecycle rule with 1-day expiration
-    /// 4. Wait for the scanner to process
-    /// 5. Verify objects are still present (they shouldn't expire yet — 1 day)
-    /// 6. Verify the lifecycle rule was persisted correctly (not corrupted to 0 days)
-    ///
-    /// This tests the rule persistence path (rustfs#4963: 3 days → 0 days).
-    #[tokio::test]
-    #[serial]
-    async fn test_lifecycle_expiration_rule_persists_correctly() -> TestResult {
-        init_logging();
-        info!("RT-03: lifecycle expiration rule persists correctly");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt03-lifecycle-persist";
-        setup_versioned_bucket(&client, bucket).await?;
-
-        // Apply a lifecycle rule with 1-day expiration on a prefix
-        let rule = LifecycleRule::builder()
-            .id("expire-after-1-day")
-            .status(ExpirationStatus::Enabled)
-            .filter(LifecycleRuleFilter::builder().prefix("logs/").build())
-            .expiration(LifecycleExpiration::builder().days(1).build())
-            .build()
-            .expect("build lifecycle rule");
-
-        client
-            .put_bucket_lifecycle_configuration()
-            .bucket(bucket)
-            .lifecycle_configuration(
-                BucketLifecycleConfiguration::builder()
-                    .rules(rule)
-                    .build()
-                    .expect("build lifecycle config"),
-            )
-            .send()
-            .await
-            .expect("put lifecycle configuration");
-
-        // Read back and verify the rule was not corrupted (rustfs#4963: days → 0)
-        let resp = client
-            .get_bucket_lifecycle_configuration()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("get lifecycle configuration");
-
-        let rules = resp.rules();
-        assert_eq!(rules.len(), 1, "RT-03 FAIL: expected exactly 1 lifecycle rule");
-
-        let retrieved = &rules[0];
-        assert_eq!(retrieved.id(), Some("expire-after-1-day"), "RT-03 FAIL: rule ID mismatch");
-        assert_eq!(retrieved.status(), &ExpirationStatus::Enabled, "RT-03 FAIL: rule should be Enabled");
-
-        let exp = retrieved.expiration().expect("expiration should be set");
-        assert_eq!(
-            exp.days(),
-            Some(1),
-            "RT-03 FAIL: expiration days corrupted (regression rustfs#4963: expected 1, got {:?})",
-            exp.days()
-        );
-
-        info!("RT-03 PASS: lifecycle expiration rule persists correctly");
-        Ok(())
-    }
-
-    /// RT-03b: Verify lifecycle rule with noncurrent version expiration.
-    ///
-    /// Covers the pattern where noncurrent version expiration rules are
-    /// accepted but old versions are never cleaned up.
-    #[tokio::test]
-    #[serial]
-    async fn test_lifecycle_noncurrent_version_expiration_rule_persists() -> TestResult {
-        init_logging();
-        info!("RT-03b: noncurrent version expiration rule persists");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt03b-noncurrent-expire";
-        setup_versioned_bucket(&client, bucket).await?;
-
-        // Create multiple versions of the same object
-        for i in 0..3 {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key("versioned-obj.txt")
-                .body(ByteStream::from(format!("version-{i}").into_bytes()))
-                .send()
-                .await
-                .expect("put object version");
-        }
-
-        // Verify we have 3 versions
-        let versions = client
-            .list_object_versions()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list versions");
-
-        let count = versions.versions().len();
-        assert_eq!(count, 3, "RT-03b FAIL: expected 3 versions, found {count}");
-
-        // Apply noncurrent version expiration rule
-        let rule = LifecycleRule::builder()
-            .id("expire-noncurrent-after-1-day")
-            .status(ExpirationStatus::Enabled)
-            .filter(LifecycleRuleFilter::builder().prefix("").build())
-            .noncurrent_version_expiration(NoncurrentVersionExpiration::builder().noncurrent_days(1).build())
-            .build()
-            .expect("build lifecycle rule");
-
-        client
-            .put_bucket_lifecycle_configuration()
-            .bucket(bucket)
-            .lifecycle_configuration(
-                BucketLifecycleConfiguration::builder()
-                    .rules(rule)
-                    .build()
-                    .expect("build lifecycle config"),
-            )
-            .send()
-            .await
-            .expect("put lifecycle configuration");
-
-        // Read back and verify
-        let resp = client
-            .get_bucket_lifecycle_configuration()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("get lifecycle configuration");
-
-        let rules = resp.rules();
-        assert_eq!(rules.len(), 1, "RT-03b FAIL: expected 1 rule");
-
-        let nc_exp = rules[0]
-            .noncurrent_version_expiration()
-            .expect("noncurrent expiration should be set");
-        assert_eq!(nc_exp.noncurrent_days(), Some(1), "RT-03b FAIL: noncurrent days corrupted");
-
-        info!("RT-03b PASS: noncurrent version expiration rule persists correctly");
-        Ok(())
-    }
-
-    /// RT-04: Verify lifecycle rule with prefix filter persists after restart.
-    ///
-    /// Covers the pattern where lifecycle rules are accepted but silently lost
-    /// after restart. Transition rules require a configured remote tier
-    /// (tested in reliant/tiering.rs), so this test uses expiration only.
-    #[tokio::test]
-    #[serial]
-    async fn test_lifecycle_prefix_rule_persists() -> TestResult {
-        init_logging();
-        info!("RT-04: lifecycle prefix rule persists");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt04-lifecycle-prefix";
-        setup_versioned_bucket(&client, bucket).await?;
-
-        let rule = LifecycleRule::builder()
-            .id("expire-archive-after-7-days")
-            .status(ExpirationStatus::Enabled)
-            .filter(LifecycleRuleFilter::builder().prefix("archive/").build())
-            .expiration(LifecycleExpiration::builder().days(7).build())
-            .build()
-            .expect("build lifecycle rule");
-
-        client
-            .put_bucket_lifecycle_configuration()
-            .bucket(bucket)
-            .lifecycle_configuration(
-                BucketLifecycleConfiguration::builder()
-                    .rules(rule)
-                    .build()
-                    .expect("build lifecycle config"),
-            )
-            .send()
-            .await
-            .expect("put lifecycle configuration");
-
-        // Restart server
-        env.restart_server_preserving_data(vec![], &[]).await.expect("restart RustFS");
-
-        // Verify the rule survived restart
-        let resp = client
-            .get_bucket_lifecycle_configuration()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("get lifecycle after restart");
-
-        let rules = resp.rules();
-        assert_eq!(rules.len(), 1, "RT-04 FAIL: expected 1 rule after restart");
-
-        let exp = rules[0].expiration().expect("expiration should be set");
-        assert_eq!(exp.days(), Some(7), "RT-04 FAIL: expiration days corrupted after restart");
-
-        info!("RT-04 PASS: lifecycle prefix rule persists after restart");
-        Ok(())
-    }
-
-    /// RT-05b: Verify delete marker creation in versioned bucket.
-    ///
-    /// Regression pattern: DELETE on a versioned object fails or does not
-    /// create a delete marker, or the delete marker is not visible in LIST.
-    #[tokio::test]
-    #[serial]
-    async fn test_delete_marker_creation_and_visibility() -> TestResult {
-        init_logging();
-        info!("RT-05b: delete marker creation and visibility");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt05b-delete-marker";
-        setup_versioned_bucket(&client, bucket).await?;
-
-        // Put an object
-        client
-            .put_object()
-            .bucket(bucket)
-            .key("marker-test.txt")
-            .body(ByteStream::from_static(b"to-be-deleted"))
-            .send()
-            .await
-            .expect("put object");
-
-        // Delete without specifying versionId → should create a delete marker
-        let del_resp = client
-            .delete_object()
-            .bucket(bucket)
-            .key("marker-test.txt")
-            .send()
-            .await
-            .expect("delete object");
-
-        // The response should indicate a delete marker was created
-        assert!(
-            del_resp.delete_marker().unwrap_or(false),
-            "RT-05b FAIL: DELETE on versioned object did not create a delete marker"
-        );
-
-        // ListObjectVersions should show both the original version and the delete marker
-        let versions = client
-            .list_object_versions()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list versions");
-
-        let delete_markers: Vec<_> = versions
-            .delete_markers()
-            .iter()
-            .filter(|dm| dm.key() == Some("marker-test.txt"))
-            .collect();
-
-        assert_eq!(
-            delete_markers.len(),
-            1,
-            "RT-05b FAIL: expected 1 delete marker, found {}",
-            delete_markers.len()
-        );
-
-        info!("RT-05b PASS: delete marker created and visible");
-        Ok(())
-    }
-}

crates/e2e_test/src/list_buckets_auth_test.rs

@@ -1,88 +0,0 @@
-// Copyright 2026 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression coverage for the MinIO-compatible filtered ListBuckets fallback.
-
-use crate::common::{RustFSTestEnvironment, admin_ok, init_logging};
-use std::error::Error;
-
-type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-#[tokio::test]
-async fn bucket_scoped_policy_returns_only_authorized_bucket() -> TestResult {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let root_client = env.create_s3_client();
-    let allowed_bucket = "list-buckets-authorized";
-    let hidden_bucket = "list-buckets-hidden";
-    let user = "listbucketsuser";
-    let secret = "listbucketssecret";
-    let policy = "list-buckets-scoped";
-
-    root_client.create_bucket().bucket(allowed_bucket).send().await?;
-    root_client.create_bucket().bucket(hidden_bucket).send().await?;
-
-    admin_ok(
-        &env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-canned-policy?name={policy}"),
-        Some(
-            serde_json::json!({
-                "Version": "2012-10-17",
-                "Statement": [{
-                    "Effect": "Allow",
-                    "Action": ["s3:*"],
-                    "Resource": [
-                        format!("arn:aws:s3:::{allowed_bucket}"),
-                        format!("arn:aws:s3:::{allowed_bucket}/*")
-                    ]
-                }]
-            })
-            .to_string(),
-        ),
-    )
-    .await?;
-    admin_ok(
-        &env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-user?accessKey={user}"),
-        Some(serde_json::json!({ "secretKey": secret, "status": "enabled" }).to_string()),
-    )
-    .await?;
-    admin_ok(
-        &env,
-        http::Method::POST,
-        "/rustfs/admin/v3/idp/builtin/policy/attach",
-        Some(serde_json::json!({ "policies": [policy], "user": user }).to_string()),
-    )
-    .await?;
-
-    let client = env.create_s3_client_with_credentials(user, secret);
-    // Capture ListBuckets first so the direct-access control cannot warm bucket metadata and mask the regression.
-    let listed = client.list_buckets().send().await;
-    client.list_objects_v2().bucket(allowed_bucket).send().await?;
-
-    let listed = listed?;
-    let names = listed
-        .buckets()
-        .iter()
-        .filter_map(|bucket| bucket.name().map(ToOwned::to_owned))
-        .collect::<Vec<_>>();
-    assert_eq!(names, vec![allowed_bucket]);
-
-    Ok(())
-}

crates/e2e_test/src/list_buckets_iam_filter_test.rs

@@ -1,459 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use crate::common::{RustFSTestEnvironment, admin_ok, build_test_s3_config, build_test_sts_client, init_logging};
-use aws_sdk_s3::Client;
-use aws_sdk_s3::error::ProvideErrorMetadata;
-use serial_test::serial;
-use tokio::time::{Duration, Instant};
-
-fn user_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str, session_token: Option<&str>) -> Client {
-    Client::from_conf(build_test_s3_config(
-        &env.url,
-        access_key,
-        secret_key,
-        session_token,
-        "list-buckets-iam-filter",
-    ))
-}
-
-fn bucket_names(buckets: &[aws_sdk_s3::types::Bucket]) -> Vec<String> {
-    let mut names = buckets
-        .iter()
-        .filter_map(|bucket| bucket.name().map(str::to_owned))
-        .collect::<Vec<_>>();
-    names.sort();
-    names
-}
-
-async fn create_user(
-    env: &RustFSTestEnvironment,
-    access_key: &str,
-    secret_key: &str,
-) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    let body = serde_json::json!({ "secretKey": secret_key, "status": "enabled" }).to_string();
-    admin_ok(
-        env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-user?accessKey={access_key}"),
-        Some(body),
-    )
-    .await?;
-    Ok(())
-}
-
-async fn create_service_account(
-    env: &RustFSTestEnvironment,
-    target_user: &str,
-    policy: Option<&serde_json::Value>,
-) -> Result<(String, String), Box<dyn std::error::Error + Send + Sync>> {
-    let request = match policy {
-        Some(policy) => serde_json::json!({ "targetUser": target_user, "policy": policy }),
-        None => serde_json::json!({ "targetUser": target_user }),
-    };
-    let response = admin_ok(env, http::Method::PUT, "/rustfs/admin/v3/add-service-accounts", Some(request.to_string())).await?;
-    let response: serde_json::Value = serde_json::from_str(&response)?;
-    let access_key = response["credentials"]["accessKey"]
-        .as_str()
-        .ok_or("service account response should contain credentials.accessKey")?
-        .to_owned();
-    let secret_key = response["credentials"]["secretKey"]
-        .as_str()
-        .ok_or("service account response should contain credentials.secretKey")?
-        .to_owned();
-    Ok((access_key, secret_key))
-}
-
-#[tokio::test]
-#[serial]
-async fn list_buckets_filters_with_iam_bucket_resources() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.capture_log_path = Some(format!("{}/server.log", env.temp_dir));
-    env.start_rustfs_server_with_env(vec![], &[("RUST_LOG", "rustfs=debug,rustfs_notify=debug")])
-        .await?;
-
-    let admin_client = env.create_s3_client();
-    for bucket in [
-        "benchmark-artifacts",
-        "benchmark-denied",
-        "benchmark-location-only",
-        "benchmark-test1",
-        "testuser1-artifacts",
-    ] {
-        admin_client.create_bucket().bucket(bucket).send().await?;
-    }
-    assert_eq!(
-        bucket_names(admin_client.list_buckets().send().await?.buckets()),
-        vec![
-            "benchmark-artifacts",
-            "benchmark-denied",
-            "benchmark-location-only",
-            "benchmark-test1",
-            "testuser1-artifacts"
-        ]
-    );
-
-    let access_key = "benchmark";
-    let secret_key = "benchmark-secret-1234567890";
-    create_user(&env, access_key, secret_key).await?;
-
-    let policy_name = "benchmark-bucket-prefix";
-    let policy = serde_json::json!({
-        "Version": "2012-10-17",
-        "Statement": [
-            {
-                "Effect": "Allow",
-                "Action": ["s3:*"],
-                "Resource": ["arn:aws:s3:::benchmark-*", "arn:aws:s3:::benchmark-*/*"],
-                "Condition": {
-                    "StringEquals": {
-                        "s3:prefix": [""],
-                        "s3:delimiter": ["/"]
-                    }
-                }
-            },
-            {
-                "Effect": "Deny",
-                "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
-                "Resource": ["arn:aws:s3:::benchmark-denied"]
-            },
-            {
-                "Effect": "Deny",
-                "Action": ["s3:ListBucket"],
-                "Resource": ["arn:aws:s3:::benchmark-location-only"]
-            },
-            {
-                "Effect": "Allow",
-                "Action": ["sts:AssumeRole"],
-                "Resource": ["arn:aws:s3:::*"]
-            }
-        ]
-    })
-    .to_string();
-    admin_ok(
-        &env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-canned-policy?name={policy_name}"),
-        Some(policy),
-    )
-    .await?;
-    admin_ok(
-        &env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/set-user-or-group-policy?policyName={policy_name}&userOrGroup={access_key}&isGroup=false"),
-        Some(String::new()),
-    )
-    .await?;
-
-    let bucket_policy_allow = serde_json::json!({
-        "Version": "2012-10-17",
-        "Statement": [{
-            "Effect": "Allow",
-            "Principal": { "AWS": [access_key] },
-            "Action": ["s3:ListBucket"],
-            "Resource": ["arn:aws:s3:::testuser1-artifacts"]
-        }]
-    })
-    .to_string();
-    admin_client
-        .put_bucket_policy()
-        .bucket("testuser1-artifacts")
-        .policy(bucket_policy_allow)
-        .send()
-        .await?;
-
-    let bucket_policy_deny = serde_json::json!({
-        "Version": "2012-10-17",
-        "Statement": [{
-            "Effect": "Deny",
-            "Principal": { "AWS": [access_key] },
-            "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
-            "Resource": ["arn:aws:s3:::benchmark-artifacts"]
-        }]
-    })
-    .to_string();
-    admin_client
-        .put_bucket_policy()
-        .bucket("benchmark-artifacts")
-        .policy(bucket_policy_deny)
-        .send()
-        .await?;
-
-    let benchmark_client = user_client(&env, access_key, secret_key, None);
-    benchmark_client
-        .list_objects_v2()
-        .bucket("testuser1-artifacts")
-        .send()
-        .await?;
-
-    assert_eq!(
-        bucket_names(benchmark_client.list_buckets().send().await?.buckets()),
-        vec!["benchmark-artifacts", "benchmark-location-only", "benchmark-test1"]
-    );
-    let log_path = env.capture_log_path.as_deref().expect("server log path should be configured");
-    let deadline = Instant::now() + Duration::from_secs(5);
-    let audit_log = loop {
-        let audit_log = tokio::fs::read_to_string(log_path).await?;
-        if [
-            "iam_implicit_deny",
-            "s3_authorization_denied",
-            "ListAllMyBucketsAction",
-            "benchmark",
-            "DEBUG",
-        ]
-        .iter()
-        .all(|field| audit_log.contains(field))
-            || Instant::now() >= deadline
-        {
-            break audit_log;
-        }
-        tokio::time::sleep(Duration::from_millis(50)).await;
-    };
-    assert_eq!(audit_log.matches("iam_implicit_deny").count(), 1, "{audit_log}");
-    for field in ["s3_authorization_denied", "ListAllMyBucketsAction", "benchmark", "DEBUG"] {
-        assert!(audit_log.contains(field), "missing {field} in {audit_log}");
-    }
-
-    let denied_access_key = "no-bucket-access";
-    let denied_secret_key = "no-bucket-access-secret-1234567890";
-    create_user(&env, denied_access_key, denied_secret_key).await?;
-    let denied = user_client(&env, denied_access_key, denied_secret_key, None)
-        .list_buckets()
-        .send()
-        .await
-        .expect_err("a user without IAM bucket permissions must be denied");
-    assert_eq!(denied.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
-
-    let put_only_policy_name = "put-only-no-bucket-discovery";
-    let put_only_policy = serde_json::json!({
-        "Version": "2012-10-17",
-        "Statement": [{
-            "Effect": "Allow",
-            "Action": ["s3:PutObject"],
-            "Resource": ["arn:aws:s3:::benchmark-*/*"]
-        }]
-    })
-    .to_string();
-    admin_ok(
-        &env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-canned-policy?name={put_only_policy_name}"),
-        Some(put_only_policy),
-    )
-    .await?;
-    admin_ok(
-        &env,
-        http::Method::PUT,
-        &format!(
-            "/rustfs/admin/v3/set-user-or-group-policy?policyName={put_only_policy_name}&userOrGroup={denied_access_key}&isGroup=false"
-        ),
-        Some(String::new()),
-    )
-    .await?;
-    let denied = user_client(&env, denied_access_key, denied_secret_key, None)
-        .list_buckets()
-        .send()
-        .await
-        .expect_err("an unrelated IAM action must not reveal bucket names");
-    assert_eq!(denied.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
-
-    let list_all_policy_name = "list-all-buckets";
-    let list_all_policy = serde_json::json!({
-        "Version": "2012-10-17",
-        "Statement": [{
-            "Effect": "Allow",
-            "Action": ["s3:ListAllMyBuckets"],
-            "Resource": ["arn:aws:s3:::*"]
-        }]
-    })
-    .to_string();
-    admin_ok(
-        &env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-canned-policy?name={list_all_policy_name}"),
-        Some(list_all_policy),
-    )
-    .await?;
-    admin_ok(
-        &env,
-        http::Method::PUT,
-        &format!(
-            "/rustfs/admin/v3/set-user-or-group-policy?policyName={list_all_policy_name}&userOrGroup={denied_access_key}&isGroup=false"
-        ),
-        Some(String::new()),
-    )
-    .await?;
-    assert_eq!(
-        bucket_names(
-            user_client(&env, denied_access_key, denied_secret_key, None)
-                .list_buckets()
-                .send()
-                .await?
-                .buckets()
-        ),
-        vec![
-            "benchmark-artifacts",
-            "benchmark-denied",
-            "benchmark-location-only",
-            "benchmark-test1",
-            "testuser1-artifacts"
-        ]
-    );
-
-    let group_user = "benchmark-group-user";
-    let group_secret = "benchmark-group-secret-1234567890";
-    let group_name = "benchmark-group";
-    create_user(&env, group_user, group_secret).await?;
-    admin_ok(
-        &env,
-        http::Method::PUT,
-        "/rustfs/admin/v3/update-group-members",
-        Some(
-            serde_json::json!({
-                "group": group_name,
-                "members": [group_user],
-                "isRemove": false,
-                "groupStatus": "enabled"
-            })
-            .to_string(),
-        ),
-    )
-    .await?;
-    admin_ok(
-        &env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/set-user-or-group-policy?policyName={policy_name}&userOrGroup={group_name}&isGroup=true"),
-        Some(String::new()),
-    )
-    .await?;
-
-    assert_eq!(
-        bucket_names(
-            user_client(&env, group_user, group_secret, None)
-                .list_buckets()
-                .send()
-                .await?
-                .buckets()
-        ),
-        vec!["benchmark-artifacts", "benchmark-location-only", "benchmark-test1"]
-    );
-
-    let (service_access_key, service_secret_key) = create_service_account(&env, group_user, None).await?;
-    assert_eq!(
-        bucket_names(
-            user_client(&env, &service_access_key, &service_secret_key, None)
-                .list_buckets()
-                .send()
-                .await?
-                .buckets()
-        ),
-        vec!["benchmark-artifacts", "benchmark-location-only", "benchmark-test1"]
-    );
-
-    let service_account_policy = serde_json::json!({
-        "Version": "2012-10-17",
-        "Statement": [{
-            "Effect": "Allow",
-            "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
-            "Resource": ["arn:aws:s3:::benchmark-test1"],
-            "Condition": {
-                "StringEquals": {
-                    "s3:prefix": [""],
-                    "s3:delimiter": ["/"]
-                }
-            }
-        }]
-    });
-    let (restricted_service_access_key, restricted_service_secret_key) =
-        create_service_account(&env, group_user, Some(&service_account_policy)).await?;
-    assert_eq!(
-        bucket_names(
-            user_client(&env, &restricted_service_access_key, &restricted_service_secret_key, None,)
-                .list_buckets()
-                .send()
-                .await?
-                .buckets()
-        ),
-        vec!["benchmark-test1"]
-    );
-
-    let sts_client = build_test_sts_client(&env.url, group_user, group_secret, None, "list-buckets-iam-filter-sts");
-    let inherited = sts_client
-        .assume_role()
-        .role_arn("arn:aws:iam::123456789012:role/list-buckets")
-        .role_session_name("list-buckets-iam-filter-inherited")
-        .send()
-        .await?;
-    let inherited = inherited
-        .credentials()
-        .ok_or("AssumeRole response should contain inherited temporary credentials")?;
-    assert_eq!(
-        bucket_names(
-            user_client(
-                &env,
-                inherited.access_key_id(),
-                inherited.secret_access_key(),
-                Some(inherited.session_token()),
-            )
-            .list_buckets()
-            .send()
-            .await?
-            .buckets()
-        ),
-        vec!["benchmark-artifacts", "benchmark-location-only", "benchmark-test1"]
-    );
-
-    let session_policy = serde_json::json!({
-        "Version": "2012-10-17",
-        "Statement": [{
-            "Effect": "Allow",
-            "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
-            "Resource": ["arn:aws:s3:::benchmark-test1"],
-            "Condition": {
-                "StringEquals": {
-                    "s3:prefix": [""],
-                    "s3:delimiter": ["/"]
-                }
-            }
-        }]
-    })
-    .to_string();
-    let assumed = sts_client
-        .assume_role()
-        .role_arn("arn:aws:iam::123456789012:role/list-buckets")
-        .role_session_name("list-buckets-iam-filter")
-        .policy(session_policy)
-        .send()
-        .await?;
-    let temporary = assumed
-        .credentials()
-        .ok_or("AssumeRole response should contain temporary credentials")?;
-    assert_eq!(
-        bucket_names(
-            user_client(
-                &env,
-                temporary.access_key_id(),
-                temporary.secret_access_key(),
-                Some(temporary.session_token()),
-            )
-            .list_buckets()
-            .send()
-            .await?
-            .buckets()
-        ),
-        vec!["benchmark-test1"]
-    );
-    Ok(())
-}

crates/e2e_test/src/listing_regression_test.rs

@@ -1,357 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression tests for object listing and metacache consistency.
-//!
-//! Covers the recurring pattern where ListObjectsV2 returns incomplete results,
-//! silently truncates with IsTruncated=false, or corrupts the metadata cache.
-//! This has regressed 8+ times.
-//!
-//! ## Regression Issues
-//!
-//! - rustfs#5166: Metacache listing quorum failed timeout after cluster startup
-//! - rustfs#5156: Metacache producer failed
-//! - rustfs#5051: ListObjectsV2 returns empty results for shallow prefixes
-//! - rustfs#4810: walk_dir timeout silently truncates listings (200, IsTruncated=false)
-//! - rustfs#4648: Object listing oscillates between complete, partial, and zero
-//! - rustfs#3191: ListObjectsV2 timeout corrupts metadata cache → NoSuchBucket
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestEnvironment, init_logging};
-    use aws_sdk_s3::primitives::ByteStream;
-    use serial_test::serial;
-    use std::collections::HashSet;
-    use std::error::Error;
-    use tracing::info;
-
-    type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-    /// RT-06: Verify ListObjectsV2 pagination completeness for medium-sized bucket.
-    ///
-    /// Regression pattern: listing returns 200 with IsTruncated=false but
-    /// misses objects (rustfs#4810: walk_dir timeout truncation).
-    ///
-    /// Steps:
-    /// 1. Upload 100 objects with known keys
-    /// 2. List all objects via pagination (max_keys=10)
-    /// 3. Verify all 100 keys are returned exactly once
-    /// 4. Verify no duplicates or skipped keys
-    #[tokio::test]
-    #[serial]
-    async fn test_list_objects_v2_completeness_100_objects() -> TestResult {
-        init_logging();
-        info!("RT-06: listing completeness with 100 objects");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt06-list-completeness";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Upload 100 objects
-        let expected_keys: Vec<String> = (0..100).map(|i| format!("obj-{i:04}.txt")).collect();
-        for key in &expected_keys {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(key)
-                .body(ByteStream::from_static(b"data"))
-                .send()
-                .await
-                .expect("put object");
-        }
-
-        // Paginate through all objects (small page size to force multiple pages)
-        let mut all_keys: Vec<String> = Vec::new();
-        let mut continuation_token: Option<String> = None;
-
-        loop {
-            let mut req = client.list_objects_v2().bucket(bucket).max_keys(10);
-
-            if let Some(ref token) = continuation_token {
-                req = req.continuation_token(token);
-            }
-
-            let resp = req.send().await.expect("list objects page");
-
-            for obj in resp.contents() {
-                all_keys.push(obj.key().unwrap_or("").to_string());
-            }
-
-            if !resp.is_truncated().unwrap_or(false) {
-                break;
-            }
-            continuation_token = resp.next_continuation_token().map(|s| s.to_string());
-        }
-
-        // Verify completeness and uniqueness
-        let unique_keys: HashSet<&str> = all_keys.iter().map(|s| s.as_str()).collect();
-
-        assert_eq!(
-            all_keys.len(),
-            100,
-            "RT-06 FAIL: expected 100 objects, listed {} (regression: walk_dir truncation)",
-            all_keys.len()
-        );
-        assert_eq!(
-            unique_keys.len(),
-            100,
-            "RT-06 FAIL: found {} unique keys but listed {} total (duplicates!)",
-            unique_keys.len(),
-            all_keys.len()
-        );
-
-        for key in &expected_keys {
-            assert!(
-                unique_keys.contains(key.as_str()),
-                "RT-06 FAIL: key '{key}' missing from listing (regression rustfs#4810)"
-            );
-        }
-
-        info!("RT-06 PASS: all 100 objects listed completely and uniquely");
-        Ok(())
-    }
-
-    /// RT-06b: Verify listing with prefix filter returns correct subset.
-    ///
-    /// Regression pattern: prefix filter returns empty or includes wrong keys
-    /// (rustfs#5051: empty results for shallow prefixes).
-    #[tokio::test]
-    #[serial]
-    async fn test_list_objects_v2_prefix_filter_correctness() -> TestResult {
-        init_logging();
-        info!("RT-06b: prefix filter correctness");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt06b-prefix-filter";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Upload objects with different prefixes
-        for i in 0..5 {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(format!("logs/app-{i:04}.log"))
-                .body(ByteStream::from_static(b"log data"))
-                .send()
-                .await
-                .expect("put log object");
-
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(format!("data/file-{i:04}.csv"))
-                .body(ByteStream::from_static(b"csv data"))
-                .send()
-                .await
-                .expect("put data object");
-        }
-
-        // List with prefix "logs/" — should return exactly 5
-        let resp = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .prefix("logs/")
-            .send()
-            .await
-            .expect("list with prefix");
-
-        assert_eq!(
-            resp.contents().len(),
-            5,
-            "RT-06b FAIL: expected 5 objects with prefix 'logs/', found {} (regression rustfs#5051)",
-            resp.contents().len()
-        );
-
-        for obj in resp.contents() {
-            assert!(
-                obj.key().unwrap_or("").starts_with("logs/"),
-                "RT-06b FAIL: object '{}' does not match prefix 'logs/'",
-                obj.key().unwrap_or("?")
-            );
-        }
-
-        // List with prefix "data/" — should return exactly 5
-        let resp = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .prefix("data/")
-            .send()
-            .await
-            .expect("list with data/ prefix");
-
-        assert_eq!(
-            resp.contents().len(),
-            5,
-            "RT-06b FAIL: expected 5 objects with prefix 'data/', found {}",
-            resp.contents().len()
-        );
-
-        // List with prefix "nonexistent/" — should return 0
-        let resp = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .prefix("nonexistent/")
-            .send()
-            .await
-            .expect("list with nonexistent prefix");
-
-        assert!(
-            resp.contents().is_empty(),
-            "RT-06b FAIL: expected 0 objects with prefix 'nonexistent/', found {}",
-            resp.contents().len()
-        );
-
-        info!("RT-06b PASS: prefix filter returns correct subset");
-        Ok(())
-    }
-
-    /// RT-06c: Verify listing with delimiter and CommonPrefixes.
-    ///
-    /// Regression pattern: delimiter handling produces incorrect CommonPrefixes
-    /// or misses objects at the delimiter boundary.
-    #[tokio::test]
-    #[serial]
-    async fn test_list_objects_v2_delimiter_common_prefixes() -> TestResult {
-        init_logging();
-        info!("RT-06c: delimiter and CommonPrefixes");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt06c-delimiter";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Create a hierarchical structure
-        let keys = vec!["a.txt", "dir1/b.txt", "dir1/sub1/c.txt", "dir1/sub2/d.txt", "dir2/e.txt"];
-
-        for key in &keys {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(*key)
-                .body(ByteStream::from_static(b"content"))
-                .send()
-                .await
-                .expect("put object");
-        }
-
-        // List with delimiter "/" at root level
-        let resp = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .delimiter("/")
-            .send()
-            .await
-            .expect("list with delimiter");
-
-        // Should have 1 object (a.txt) and 2 common prefixes (dir1/, dir2/)
-        let contents: Vec<_> = resp.contents().iter().map(|o| o.key().unwrap_or("")).collect();
-        let prefixes: Vec<_> = resp.common_prefixes().iter().map(|p| p.prefix().unwrap_or("")).collect();
-
-        assert!(contents.contains(&"a.txt"), "RT-06c FAIL: root object 'a.txt' missing from listing");
-        assert_eq!(contents.len(), 1, "RT-06c FAIL: expected 1 root-level object, found {}", contents.len());
-        assert_eq!(prefixes.len(), 2, "RT-06c FAIL: expected 2 common prefixes, found {:?}", prefixes);
-        assert!(prefixes.contains(&"dir1/"), "RT-06c FAIL: 'dir1/' missing from CommonPrefixes");
-        assert!(prefixes.contains(&"dir2/"), "RT-06c FAIL: 'dir2/' missing from CommonPrefixes");
-
-        info!("RT-06c PASS: delimiter and CommonPrefixes correct");
-        Ok(())
-    }
-
-    /// RT-06d: Verify listing returns correct IsTruncated flag.
-    ///
-    /// Regression pattern: IsTruncated=false when there are more objects
-    /// (rustfs#4810: walk_dir timeout truncation with false IsTruncated).
-    #[tokio::test]
-    #[serial]
-    async fn test_list_objects_v2_is_truncated_correctness() -> TestResult {
-        init_logging();
-        info!("RT-06d: IsTruncated correctness");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt06d-truncated";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Upload 15 objects
-        for i in 0..15 {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(format!("item-{i:04}.txt"))
-                .body(ByteStream::from_static(b"data"))
-                .send()
-                .await
-                .expect("put object");
-        }
-
-        // List with max_keys=5 — should be truncated
-        let resp = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .max_keys(5)
-            .send()
-            .await
-            .expect("list with max_keys=5");
-
-        assert!(
-            resp.is_truncated().unwrap_or(false),
-            "RT-06d FAIL: IsTruncated should be true with 15 objects and max_keys=5"
-        );
-        assert_eq!(resp.contents().len(), 5, "RT-06d FAIL: expected 5 objects in first page");
-        assert!(
-            resp.next_continuation_token().is_some(),
-            "RT-06d FAIL: NextContinuationToken should be present when truncated"
-        );
-
-        // List with max_keys=100 — should NOT be truncated
-        let resp = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .max_keys(100)
-            .send()
-            .await
-            .expect("list with max_keys=100");
-
-        assert!(
-            !resp.is_truncated().unwrap_or(false),
-            "RT-06d FAIL: IsTruncated should be false with 15 objects and max_keys=100"
-        );
-        assert_eq!(resp.contents().len(), 15, "RT-06d FAIL: expected 15 objects with max_keys=100");
-
-        info!("RT-06d PASS: IsTruncated flag is correct");
-        Ok(())
-    }
-}

crates/e2e_test/src/notification_startup_regression_test.rs

@@ -1,153 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression tests for the event notification startup race.
-//!
-//! Covers the recurring pattern where webhook/audit targets fail to load at boot
-//! due to startup ordering (notification runtime starts before server config is
-//! loaded). This has regressed 9+ times across beta.3 ~ beta.12.
-//!
-//! ## Regression Issues
-//!
-//! - rustfs#5387: webhook notifications broken again in beta.9+
-//! - rustfs#5681: Audit webhook targets are not loaded at boot
-//! - rustfs#5401: Event Destinations broken again
-//! - rustfs#5183: Audit webhooks stay offline after restart
-//! - rustfs#5115: init_event_notifier loses startup race against server config load
-//! - rustfs#4796: Pulsar event destinations offline after restart
-//! - rustfs#5428: MQTT bucket notifications stop on restarted cluster node
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestEnvironment, init_logging};
-    use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
-    use serial_test::serial;
-    use std::error::Error;
-    use tracing::info;
-
-    type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-    /// RT-01: Verify that the notification runtime initializes correctly at boot.
-    ///
-    /// Regression pattern: notification runtime initializes before server config
-    /// is fully loaded, causing webhook targets to never come online.
-    ///
-    /// This test verifies the startup ordering by checking that the server
-    /// starts successfully with notification enabled and can serve S3 requests.
-    /// A full webhook delivery test is in notification_webhook_test.rs.
-    #[tokio::test]
-    #[serial]
-    async fn test_notification_enabled_server_starts_cleanly() -> TestResult {
-        init_logging();
-        info!("RT-01: notification enabled server starts cleanly");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false"), ("RUSTFS_NOTIFY_ENABLE", "true")])
-            .await
-            .expect("start RustFS with notifications enabled");
-
-        let client = env.create_s3_client();
-        let bucket = "rt01-notify-startup";
-
-        // Server should be healthy and able to serve S3 requests
-        client
-            .create_bucket()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("create bucket with notifications enabled");
-
-        client
-            .put_object()
-            .bucket(bucket)
-            .key("test.txt")
-            .body(aws_sdk_s3::primitives::ByteStream::from_static(b"test"))
-            .send()
-            .await
-            .expect("put object with notifications enabled");
-
-        info!("RT-01 PASS: notification enabled server starts and serves S3");
-        Ok(())
-    }
-
-    /// RT-02: Verify notification config persists after server restart.
-    ///
-    /// Regression pattern: after a node restart, notification targets stay
-    /// offline permanently because the config is not re-loaded.
-    ///
-    /// Steps:
-    /// 1. Start server with notification enabled
-    /// 2. Create bucket and configure notification
-    /// 3. Restart server
-    /// 4. Verify notification config still exists
-    #[tokio::test]
-    #[serial]
-    async fn test_notification_config_survives_restart() -> TestResult {
-        init_logging();
-        info!("RT-02: notification config survives restart");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false"), ("RUSTFS_NOTIFY_ENABLE", "true")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt02-notify-restart";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Enable versioning (required for notification configuration)
-        client
-            .put_bucket_versioning()
-            .bucket(bucket)
-            .versioning_configuration(
-                VersioningConfiguration::builder()
-                    .status(BucketVersioningStatus::Enabled)
-                    .build(),
-            )
-            .send()
-            .await
-            .expect("enable versioning");
-
-        // Note: We can't fully test notification config persistence without a
-        // configured target. But we verify the server restarts cleanly with
-        // notification enabled, which is the core regression scenario.
-        env.restart_server_preserving_data(vec![], &[])
-            .await
-            .expect("restart RustFS with notifications enabled");
-
-        // Verify bucket still exists and is accessible after restart
-        let list = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list objects after restart");
-
-        assert!(list.contents().is_empty(), "RT-02: bucket should be empty after restart");
-
-        // Verify we can still write objects (notification runtime initialized)
-        client
-            .put_object()
-            .bucket(bucket)
-            .key("after-restart.txt")
-            .body(aws_sdk_s3::primitives::ByteStream::from_static(b"post-restart"))
-            .send()
-            .await
-            .expect("put object after restart — notification runtime must be initialized");
-
-        info!("RT-02 PASS: server with notifications survives restart");
-        Ok(())
-    }
-}

crates/e2e_test/src/notification_webhook_test.rs

@@ -24,7 +24,7 @@
 //!   * PUT / multipart-complete / DeleteObject / DeleteObjects each deliver one event with the correct
 //!     eventName, bucket, key, versionId and eTag.
 //!   * prefix/suffix filters drop non-matching keys (rule-engine gate).
-//!   * an event queued while the target endpoint rejects delivery is redelivered
+//!   * an event queued while the target endpoint is unreachable is redelivered
 //!     from the on-disk store once the endpoint recovers (store-and-forward).
 //!   * responseElements and the S3 response use the canonical request ID while
 //!     requestParameters preserve a conflicting client-supplied value.
@@ -897,10 +897,11 @@ async fn test_webhook_event_delivery_and_filtering() -> TestResult {
     Ok(())
 }
 
-/// An event queued while the target endpoint rejects delivery survives on the
+/// An event queued while the target endpoint is unreachable survives on the
 /// durable store and is redelivered once the endpoint comes back.
 #[tokio::test]
 #[serial]
+#[ignore = "FAILING deterministically on main since it landed (#4821): the target is created but never appears in /rustfs/admin/v3/target/arns, so wait_for_target_registered times out. Quarantined per the flake policy; remove with the fix for rustfs#4852"]
 async fn test_webhook_redelivers_event_after_target_recovers() -> TestResult {
     init_logging();
 
@@ -931,55 +932,28 @@ async fn test_webhook_redelivers_event_after_target_recovers() -> TestResult {
     wait_for_target_registered(&env, target).await?;
     put_notification_config(&client, bucket, target, "uploads/", ".dat").await?;
 
-    // Replace the healthy setup listener with one that rejects the first POST.
-    // Waiting for that response below proves the queued event reached a failed
-    // delivery attempt before the endpoint recovers.
+    // Take the endpoint down (drops the listener, so connections are refused —
+    // a retryable NotConnected), then PUT: the event cannot be delivered and
+    // must survive on the durable queue store.
     setup_handle.abort();
     let _ = setup_handle.await;
 
-    let listener = TcpListener::bind(("0.0.0.0", port)).await?;
     let key = "uploads/redeliver.dat";
     client
         .put_object()
         .bucket(bucket)
         .key(key)
-        .body(ByteStream::from_static(b"queued while target rejects"))
+        .body(ByteStream::from_static(b"queued while target down"))
         .send()
         .await?;
 
-    let mut failure_handle = tokio::spawn(async move {
-        loop {
-            let (mut stream, _) = listener.accept().await?;
-            let (method, _) = timeout(Duration::from_secs(5), read_http_message(&mut stream)).await??;
-            if method == "HEAD" {
-                stream
-                    .write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 0\r\nconnection: close\r\n\r\n")
-                    .await?;
-                stream.shutdown().await?;
-                continue;
-            }
-            if method == "POST" {
-                stream
-                    .write_all(b"HTTP/1.1 503 Service Unavailable\r\ncontent-length: 0\r\nconnection: close\r\n\r\n")
-                    .await?;
-                stream.shutdown().await?;
-                return Ok::<(), BoxError>(());
-            }
-        }
-    });
+    // Hold the endpoint down long enough for at least one replay attempt to
+    // fail (the replay worker scans the store every 500ms), so recovery below
+    // exercises real redelivery rather than a first-attempt success.
+    tokio::time::sleep(Duration::from_secs(2)).await;
 
-    let rejected = match timeout(Duration::from_secs(20), &mut failure_handle).await {
-        Ok(rejected) => rejected,
-        Err(_) => {
-            failure_handle.abort();
-            let _ = failure_handle.await;
-            return Err("webhook replay did not reach the rejecting endpoint".into());
-        }
-    };
-    rejected??;
-
-    // Bring the endpoint back on the same port; the replay worker rescans the
-    // durable queue and delivers the retained event.
+    // Bring the endpoint back on the same port; the replay worker retries with
+    // exponential backoff and delivers the queued event.
     let listener = TcpListener::bind(("0.0.0.0", port)).await?;
     let (tx, mut rx) = mpsc::unbounded_channel();
     let handle = serve_event_collector(listener, tx);

crates/e2e_test/src/protocols/sftp_compliance_tests.rs

@@ -2854,7 +2854,7 @@ pub(crate) mod cmptst_30 {
         result
     }
 
-    #[ignore = "timing-sensitive backend-pressure latency probe; run explicitly with --ignored"]
+    #[ignore]
     #[tokio::test]
     async fn regression() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         crate::common::init_logging();

crates/e2e_test/src/quota_test.rs

@@ -12,11 +12,9 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::common::{RustFSTestEnvironment, admin_request, awscurl_delete, awscurl_get, awscurl_post, awscurl_put, init_logging};
+use crate::common::{RustFSTestEnvironment, awscurl_delete, awscurl_get, awscurl_post, awscurl_put, init_logging};
 use aws_sdk_s3::Client;
-use http::{Method, StatusCode};
 use serial_test::serial;
-use tokio::time::{Duration, sleep, timeout};
 use tracing::{debug, info};
 
 fn skip_without_awscurl() -> bool {
@@ -39,8 +37,7 @@ impl QuotaTestEnv {
     pub async fn new() -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
         let bucket_name = format!("quota-test-{}", uuid::Uuid::new_v4());
         let mut env = RustFSTestEnvironment::new().await?;
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_SCANNER_START_DELAY_SECS", "0")])
-            .await?;
+        env.start_rustfs_server(vec![]).await?;
         let client = env.create_s3_client();
 
         Ok(Self {
@@ -70,7 +67,18 @@ impl QuotaTestEnv {
     }
 
     pub async fn set_bucket_quota(&self, quota_bytes: u64) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        self.set_bucket_quota_for(&self.bucket_name, quota_bytes).await
+        let url = format!("{}/rustfs/admin/v3/quota/{}", self.env.url, self.bucket_name);
+        let quota_config = serde_json::json!({
+            "quota": quota_bytes,
+            "quota_type": "HARD"
+        });
+
+        let response = awscurl_put(&url, &quota_config.to_string(), &self.env.access_key, &self.env.secret_key).await?;
+        if response.contains("error") {
+            Err(format!("Failed to set quota: {}", response).into())
+        } else {
+            Ok(())
+        }
     }
 
     pub async fn get_bucket_quota(&self) -> Result<Option<u64>, Box<dyn std::error::Error + Send + Sync>> {
@@ -170,29 +178,6 @@ impl QuotaTestEnv {
         bucket: &str,
         quota_bytes: u64,
     ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        let stats_path = format!("/rustfs/admin/v3/quota-stats/{bucket}");
-        let readiness = async {
-            loop {
-                let (status, response) =
-                    admin_request(&self.env.url, Method::GET, &stats_path, None, &self.env.access_key, &self.env.secret_key)
-                        .await?;
-                if status.is_success() {
-                    return Ok::<(), Box<dyn std::error::Error + Send + Sync>>(());
-                }
-                if status != StatusCode::SERVICE_UNAVAILABLE {
-                    return Err(format!("quota usage readiness failed for {bucket}: {status} {response}").into());
-                }
-
-                sleep(Duration::from_secs(1)).await;
-            }
-        };
-        match timeout(Duration::from_secs(30), readiness).await {
-            Ok(result) => result?,
-            Err(_) => {
-                return Err(format!("quota usage did not become authoritative for {bucket} within 30 seconds").into());
-            }
-        }
-
         let url = format!("{}/rustfs/admin/v3/quota/{}", self.env.url, bucket);
         let quota_config = serde_json::json!({
             "quota": quota_bytes,
@@ -252,7 +237,6 @@ impl QuotaTestEnv {
 #[cfg(test)]
 mod integration_tests {
     use super::*;
-    use aws_sdk_s3::error::ProvideErrorMetadata;
 
     #[tokio::test]
     #[serial]
@@ -964,27 +948,9 @@ mod integration_tests {
             .send()
             .await;
 
-        let complete_error = complete_result.expect_err("multipart completion above quota must be rejected");
-        assert_eq!(complete_error.as_service_error().and_then(|error| error.code()), Some("InvalidRequest"));
+        assert!(complete_result.is_err());
         assert!(!env.object_exists("over_quota.txt").await?);
 
-        let staged_parts = env
-            .client
-            .list_parts()
-            .bucket(&env.bucket_name)
-            .key("over_quota.txt")
-            .upload_id(upload_id2)
-            .send()
-            .await?;
-        assert_eq!(staged_parts.parts().len(), 2, "quota rejection must preserve the multipart upload");
-        env.client
-            .abort_multipart_upload()
-            .bucket(&env.bucket_name)
-            .key("over_quota.txt")
-            .upload_id(upload_id2)
-            .send()
-            .await?;
-
         env.cleanup_bucket().await?;
 
         Ok(())

crates/e2e_test/src/reliability_disk_fault_test.rs

@@ -22,16 +22,15 @@
 
 #[cfg(test)]
 mod tests {
-    use crate::chaos::{DiskFaultHarness, VersionShardCensus, signed_admin_post};
+    use crate::chaos::{DiskFaultHarness, signed_admin_post};
     use crate::common::init_logging;
     use aws_sdk_s3::Client;
     use aws_sdk_s3::primitives::ByteStream;
-    use aws_sdk_s3::types::{BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, VersioningConfiguration};
+    use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
     use serial_test::serial;
     use sha2::{Digest, Sha256};
     use std::collections::HashSet;
-    use std::error::Error;
-    use tokio::time::{Duration, Instant, interval, timeout};
+    use tokio::time::{Duration, sleep, timeout};
     use tracing::info;
 
     const GET_TIMEOUT: Duration = Duration::from_secs(60);
@@ -272,17 +271,12 @@ mod tests {
         put_and_record(&client, bucket, "heal/nested/large.bin", payload(2 * 1024 * 1024, 34), &mut manifest).await?;
 
         verify_manifest(&client, bucket, &manifest, "baseline before disk replacement").await?;
-        let manifest_keys = manifest.iter().map(|(key, _)| key.clone()).collect::<Vec<_>>();
-        let target_manifest: Vec<(String, VersionShardCensus)> = manifest_keys
-            .iter()
-            .map(|key| {
-                let census = harness.census_object_version(0, bucket, key, None)?;
-                if !census.is_complete() {
-                    return Err(format!("disk 0 has incomplete physical census for {key}: {census:?}").into());
-                }
-                Ok((key.clone(), census))
-            })
-            .collect::<Result<_, Box<dyn Error + Send + Sync>>>()?;
+        for (key, _) in &manifest {
+            assert!(
+                harness.object_metadata_exists_on_disk(0, bucket, key),
+                "disk0 should hold xl.meta for {key} before replacement"
+            );
+        }
 
         harness.kill_server();
         harness.replace_disk_with_empty(0)?;
@@ -293,179 +287,21 @@ mod tests {
         signed_admin_post(&heal_url, Some(heal_body), &harness.env.access_key, &harness.env.secret_key).await?;
 
         let client = harness.env.create_s3_client();
-        let mut remaining: HashSet<String> = manifest_keys.iter().cloned().collect();
+        let mut remaining: HashSet<String> = manifest.iter().map(|(key, _)| key.clone()).collect();
         let heal_timeout_secs = std::env::var("RUSTFS_RELIABILITY_HEAL_TIMEOUT_SECS")
             .ok()
             .and_then(|value| value.parse::<u64>().ok())
             .unwrap_or(120);
-        let deadline = Instant::now() + Duration::from_secs(heal_timeout_secs);
-        let mut retry = interval(Duration::from_secs(1));
 
-        loop {
-            remaining.retain(|key| {
-                let expected = target_manifest
-                    .iter()
-                    .find(|(manifest_key, _)| manifest_key == key)
-                    .map(|(_, manifest)| manifest)
-                    .expect("every key has a physical manifest");
-                harness
-                    .census_object_version(0, bucket, key, None)
-                    .map(|census| !census.matches_manifest(expected))
-                    .unwrap_or(true)
-            });
+        for _ in 0..heal_timeout_secs {
+            remaining.retain(|key| !harness.object_metadata_exists_on_disk(0, bucket, key));
             if remaining.is_empty() {
                 verify_manifest(&client, bucket, &manifest, "after fresh-disk heal completed").await?;
                 return Ok(());
             }
-            if Instant::now() >= deadline {
-                break;
-            }
-            retry.tick().await;
+            sleep(Duration::from_secs(1)).await;
         }
 
         Err(format!("fresh-disk heal did not rebuild {remaining:?} on the replaced disk within {heal_timeout_secs}s").into())
     }
-
-    #[tokio::test]
-    #[serial]
-    async fn test_versioned_shard_census_selects_each_version_data_dir() -> Result<(), Box<dyn Error + Send + Sync>> {
-        init_logging();
-        info!("Reliability: physical shard census selects the requested object version");
-
-        let mut harness = DiskFaultHarness::new(4).await?;
-        harness.start_server().await?;
-        let client = harness.env.create_s3_client();
-        let bucket = "reliability-versioned-census";
-        let key = "versions/large.bin";
-        client.create_bucket().bucket(bucket).send().await?;
-        client
-            .put_bucket_versioning()
-            .bucket(bucket)
-            .versioning_configuration(
-                VersioningConfiguration::builder()
-                    .status(BucketVersioningStatus::Enabled)
-                    .build(),
-            )
-            .send()
-            .await?;
-
-        let first_inline = client
-            .put_object()
-            .bucket(bucket)
-            .key("versions/inline.bin")
-            .body(ByteStream::from(payload(8 * 1024, 40)))
-            .send()
-            .await?;
-        let first_inline_version = first_inline
-            .version_id()
-            .ok_or("first inline PUT did not return a version ID")?;
-        let second_inline = client
-            .put_object()
-            .bucket(bucket)
-            .key("versions/inline.bin")
-            .body(ByteStream::from(payload(8 * 1024, 41)))
-            .send()
-            .await?;
-        let second_inline_version = second_inline
-            .version_id()
-            .ok_or("second inline PUT did not return a version ID")?;
-
-        let first = client
-            .put_object()
-            .bucket(bucket)
-            .key(key)
-            .body(ByteStream::from(payload(128 * 1024, 41)))
-            .send()
-            .await?;
-        let first_version = first.version_id().ok_or("first PUT did not return a version ID")?;
-        let second = client
-            .put_object()
-            .bucket(bucket)
-            .key(key)
-            .body(ByteStream::from(payload(3 * 1024 * 1024, 42)))
-            .send()
-            .await?;
-        let second_version = second.version_id().ok_or("second PUT did not return a version ID")?;
-        let delete = client.delete_object().bucket(bucket).key(key).send().await?;
-        let delete_version = delete.version_id().ok_or("delete marker did not return a version ID")?;
-
-        let first_inline_census = harness.census_object_version(0, bucket, "versions/inline.bin", Some(first_inline_version))?;
-        let second_inline_census =
-            harness.census_object_version(0, bucket, "versions/inline.bin", Some(second_inline_version))?;
-        let first_census = harness.census_object_version(0, bucket, key, Some(first_version))?;
-        let first_other_disk_census = harness.census_object_version(1, bucket, key, Some(first_version))?;
-        let second_census = harness.census_object_version(0, bucket, key, Some(second_version))?;
-        let delete_census = harness.census_object_version(0, bucket, key, Some(delete_version))?;
-        assert!(
-            first_inline_census.is_complete() && second_inline_census.is_complete(),
-            "inline version physical census is incomplete: first={first_inline_census:?} second={second_inline_census:?}"
-        );
-        assert!(
-            first_inline_census.present_part_fingerprints.is_empty() && second_inline_census.present_part_fingerprints.is_empty(),
-            "inline versions must not select external shard files: first={first_inline_census:?} second={second_inline_census:?}"
-        );
-        assert!(
-            first_inline_census.inline_data_fingerprint.is_some() && second_inline_census.inline_data_fingerprint.is_some(),
-            "inline versions must fingerprint payload bytes stored in xl.meta"
-        );
-        assert_ne!(
-            first_inline_census.inline_data_fingerprint, second_inline_census.inline_data_fingerprint,
-            "same-size inline versions with different payloads must retain distinct xl.meta fingerprints"
-        );
-        assert!(
-            first_census.is_complete(),
-            "first version physical census is incomplete: {first_census:?}"
-        );
-        assert!(
-            second_census.is_complete(),
-            "second version physical census is incomplete: {second_census:?}"
-        );
-        assert!(
-            first_other_disk_census.is_complete(),
-            "first version physical census on the second disk is incomplete: {first_other_disk_census:?}"
-        );
-        assert_ne!(
-            first_census.erasure_index, first_other_disk_census.erasure_index,
-            "physical census must preserve each disk's erasure index"
-        );
-        assert_ne!(
-            first_census.data_dir, second_census.data_dir,
-            "distinct object versions must select distinct physical data directories"
-        );
-        assert_eq!(
-            first_census.expected_part_numbers, second_census.expected_part_numbers,
-            "same single-part shape should expose the same part numbers"
-        );
-        let first_part = first_census
-            .present_part_fingerprints
-            .values()
-            .next()
-            .ok_or("first version did not expose a physical part fingerprint")?;
-        let second_part = second_census
-            .present_part_fingerprints
-            .values()
-            .next()
-            .ok_or("second version did not expose a physical part fingerprint")?;
-        assert_ne!(
-            first_part.size, second_part.size,
-            "different shard lengths must retain their physical sizes"
-        );
-        assert_ne!(
-            first_part.sha256, second_part.sha256,
-            "different shard contents must retain their physical hashes"
-        );
-        assert!(
-            delete_census.is_complete(),
-            "delete marker physical census is incomplete: {delete_census:?}"
-        );
-        assert!(
-            delete_census.expected_part_numbers.is_empty(),
-            "delete marker must not declare object shards: {delete_census:?}"
-        );
-        assert!(
-            delete_census.present_part_fingerprints.is_empty(),
-            "delete marker must not select stale object shards: {delete_census:?}"
-        );
-        Ok(())
-    }
 }

crates/e2e_test/src/reliant/grpc_lock_server.rs

@@ -848,13 +848,6 @@ impl NodeService for MinimalLockNodeService {
         Err(Status::unimplemented("lock-only test server"))
     }
 
-    async fn replacement_recovery_status(
-        &self,
-        _request: Request<rustfs_protos::proto_gen::node_service::ReplacementRecoveryStatusRequest>,
-    ) -> Result<Response<rustfs_protos::proto_gen::node_service::ReplacementRecoveryStatusResponse>, Status> {
-        Err(Status::unimplemented("lock-only test server"))
-    }
-
     async fn get_metacache_listing(
         &self,
         _request: Request<rustfs_protos::proto_gen::node_service::GetMetacacheListingRequest>,

crates/e2e_test/src/sts_query_compat_test.rs

@@ -12,10 +12,13 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::common::{RustFSTestEnvironment, admin_ok, build_test_s3_config, build_test_sts_client, init_logging};
-use aws_sdk_sts::Client;
+use crate::common::{RustFSTestEnvironment, admin_ok, init_logging};
+use aws_sdk_sts::config::retry::RetryConfig;
+use aws_sdk_sts::config::{Credentials, Region};
 use aws_sdk_sts::error::ProvideErrorMetadata;
 use aws_sdk_sts::operation::RequestId;
+use aws_sdk_sts::{Client, Config};
+use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
 use bytes::Bytes;
 use http::header::{AUTHORIZATION, CONTENT_TYPE};
 use http::{Request, Response};
@@ -29,8 +32,9 @@ use serial_test::serial;
 use std::collections::BTreeSet;
 use std::convert::Infallible;
 use std::error::Error;
+use std::sync::Arc;
 use tokio::net::TcpListener;
-use tokio::sync::mpsc;
+use tokio::sync::{Notify, mpsc};
 use tokio::task::{JoinHandle, JoinSet};
 use tokio::time::{Duration, timeout};
 
@@ -39,7 +43,22 @@ type TestResult = Result<(), BoxError>;
 const OPA_AUTH_TOKEN: &str = "sts-opa-token";
 
 fn sts_client(url: &str, access_key: &str, secret_key: &str, session_token: Option<&str>) -> Client {
-    build_test_sts_client(url, access_key, secret_key, session_token, "e2e-sts-query-compat")
+    let mut config = Config::builder()
+        .credentials_provider(Credentials::new(
+            access_key,
+            secret_key,
+            session_token.map(str::to_owned),
+            None,
+            "e2e-sts-query-compat",
+        ))
+        .region(Region::new("us-east-1"))
+        .endpoint_url(url)
+        .retry_config(RetryConfig::standard().with_max_attempts(1))
+        .behavior_version_latest();
+    if url.starts_with("http://") {
+        config = config.http_client(SmithyHttpClientBuilder::new().build_http());
+    }
+    Client::from_conf(config.build())
 }
 
 async fn create_root_service_account(env: &RustFSTestEnvironment) -> Result<(String, String), BoxError> {
@@ -126,52 +145,6 @@ async fn assert_access_denied(client: &Client, context: &str) -> TestResult {
     Ok(())
 }
 
-async fn assert_list_buckets_access_denied(
-    env: &RustFSTestEnvironment,
-    access_key: &str,
-    secret_key: &str,
-    context: &str,
-) -> TestResult {
-    let error = aws_sdk_s3::Client::from_conf(build_test_s3_config(
-        &env.url,
-        access_key,
-        secret_key,
-        None,
-        "e2e-list-buckets-opa-unavailable",
-    ))
-    .list_buckets()
-    .send()
-    .await
-    .expect_err("ListBuckets must be denied while OPA is unavailable");
-    let service_error = error
-        .as_service_error()
-        .ok_or_else(|| format!("{context} should deserialize as an S3 service error: {error:?}"))?;
-
-    assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(403));
-    assert_eq!(service_error.code(), Some("AccessDenied"));
-    Ok(())
-}
-
-async fn assert_opa_unavailable_denies_sts_and_list_buckets(env: &RustFSTestEnvironment, context: &str) -> TestResult {
-    let user = "opaunavailable";
-    let secret = "stsOpaUnavailableSecret123";
-    create_user_with_policy(
-        env,
-        user,
-        secret,
-        "sts-opa-unavailable-local-policy",
-        serde_json::json!([{
-            "Effect": "Allow",
-            "Action": ["s3:ListAllMyBuckets"],
-            "Resource": ["arn:aws:s3:::*"],
-        }]),
-    )
-    .await?;
-
-    assert_access_denied(&sts_client(&env.url, user, secret, None), context).await?;
-    assert_list_buckets_access_denied(env, user, secret, context).await
-}
-
 async fn handle_opa_request(
     request: Request<Incoming>,
     requests: mpsc::UnboundedSender<Value>,
@@ -213,15 +186,12 @@ async fn handle_opa_request(
     };
     if payload.is_none() {
         let _ = validation_started.send(());
-        match validation_mode {
-            OpaValidationMode::Blocked => std::future::pending::<()>().await,
-            OpaValidationMode::Unavailable => {
-                return Ok(Response::builder()
-                    .status(503)
-                    .body(Full::new(Bytes::new()))
-                    .expect("static OPA unavailable response must be valid"));
-            }
-            OpaValidationMode::Ready => {}
+        if let OpaValidationMode::DelayedUnavailable(release) = validation_mode {
+            release.notified().await;
+            return Ok(Response::builder()
+                .status(503)
+                .body(Full::new(Bytes::new()))
+                .expect("static OPA unavailable response must be valid"));
         }
     }
     let allow = match payload.as_ref().and_then(|value| value.pointer("/input/identity/account")) {
@@ -231,25 +201,6 @@ async fn handle_opa_request(
             .and_then(Value::as_bool)
             .unwrap_or(false),
         Some(Value::String(account)) if account == "opadeny" => false,
-        Some(Value::String(account))
-            if account == "opaunavailable" && matches!(validation_mode, OpaValidationMode::Unavailable) =>
-        {
-            true
-        }
-        Some(Value::String(account)) if account == "opalistbuckets" => {
-            let action = payload
-                .as_ref()
-                .and_then(|value| value.pointer("/input/action"))
-                .and_then(Value::as_str);
-            let bucket = payload
-                .as_ref()
-                .and_then(|value| value.pointer("/input/resource/bucket"))
-                .and_then(Value::as_str);
-            matches!(
-                (action, bucket),
-                (Some("s3:ListBucket"), Some("opa-list-visible")) | (Some("s3:GetBucketLocation"), Some("opa-list-location"))
-            )
-        }
         None => true,
         _ => false,
     };
@@ -264,17 +215,17 @@ async fn handle_opa_request(
         .expect("static OPA response must be valid"))
 }
 
-#[derive(Clone, Copy)]
+#[derive(Clone)]
 enum OpaValidationMode {
     Ready,
-    Blocked,
-    Unavailable,
+    DelayedUnavailable(Arc<Notify>),
 }
 
 struct OpaMock {
     url: String,
     requests: mpsc::UnboundedReceiver<Value>,
     validation_started: mpsc::UnboundedReceiver<()>,
+    validation_release: Option<Arc<Notify>>,
     task: JoinHandle<()>,
 }
 
@@ -283,8 +234,9 @@ impl OpaMock {
         Self::start_with_mode(OpaValidationMode::Ready, Some(OPA_AUTH_TOKEN)).await
     }
 
-    async fn start_blocked() -> Result<Self, BoxError> {
-        Self::start_with_mode(OpaValidationMode::Blocked, None).await
+    async fn start_delayed_unavailable() -> Result<Self, BoxError> {
+        let release = Arc::new(Notify::new());
+        Self::start_with_mode(OpaValidationMode::DelayedUnavailable(release), None).await
     }
 
     async fn start_with_mode(validation_mode: OpaValidationMode, auth_token: Option<&str>) -> Result<Self, BoxError> {
@@ -293,6 +245,10 @@ impl OpaMock {
         let (requests_tx, requests) = mpsc::unbounded_channel();
         let (validation_started_tx, validation_started) = mpsc::unbounded_channel();
         let expected_authorization = auth_token.map(|token| format!("Bearer {token}"));
+        let validation_release = match &validation_mode {
+            OpaValidationMode::Ready => None,
+            OpaValidationMode::DelayedUnavailable(release) => Some(Arc::clone(release)),
+        };
         let task = tokio::spawn(async move {
             let mut connections = JoinSet::new();
             loop {
@@ -301,7 +257,7 @@ impl OpaMock {
                         let Ok((stream, _)) = accepted else { break };
                         let requests = requests_tx.clone();
                         let validation_started = validation_started_tx.clone();
-                        let validation_mode = validation_mode;
+                        let validation_mode = validation_mode.clone();
                         let expected_authorization = expected_authorization.clone();
                         connections.spawn(async move {
                             let handler = service_fn(move |request| {
@@ -309,7 +265,7 @@ impl OpaMock {
                                     request,
                                     requests.clone(),
                                     validation_started.clone(),
-                                    validation_mode,
+                                    validation_mode.clone(),
                                     expected_authorization.clone(),
                                 )
                             });
@@ -326,6 +282,7 @@ impl OpaMock {
             url,
             requests,
             validation_started,
+            validation_release,
             task,
         })
     }
@@ -341,6 +298,12 @@ impl OpaMock {
             .await?
             .ok_or_else(|| "OPA validation channel closed".into())
     }
+
+    fn release_validation(&self) {
+        if let Some(release) = &self.validation_release {
+            release.notify_one();
+        }
+    }
 }
 
 impl Drop for OpaMock {
@@ -560,119 +523,35 @@ async fn test_sts_assume_role_opa_contract() -> TestResult {
 
 #[tokio::test]
 #[serial]
-async fn test_list_buckets_opa_contract() -> TestResult {
+async fn test_sts_assume_role_fails_closed_while_opa_is_unavailable() -> TestResult {
     init_logging();
 
-    let mut opa = OpaMock::start().await?;
+    let mut opa = OpaMock::start_delayed_unavailable().await?;
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server_with_env(
-        vec![],
-        &[
-            ("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str()),
-            ("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", OPA_AUTH_TOKEN),
-        ],
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str())])
+        .await?;
+    opa.wait_for_validation().await?;
+
+    let user = "opaunavailable";
+    let secret = "stsOpaUnavailableSecret123";
+    create_user_with_policy(
+        &env,
+        user,
+        secret,
+        "sts-opa-unavailable-local-policy",
+        serde_json::json!([{
+            "Effect": "Allow",
+            "Action": ["s3:ListAllMyBuckets"],
+            "Resource": ["arn:aws:s3:::*"],
+        }]),
     )
     .await?;
 
-    let admin_client = env.create_s3_client();
-    for bucket in ["opa-list-hidden", "opa-list-location", "opa-list-visible"] {
-        admin_client.create_bucket().bucket(bucket).send().await?;
-    }
+    assert_access_denied(&sts_client(&env.url, user, secret, None), "configured OPA initialization").await?;
 
-    let user = "opalistbuckets";
-    let secret = "opaListBucketsSecret123";
-    create_user(&env, user, secret).await?;
-
-    let output = aws_sdk_s3::Client::from_conf(build_test_s3_config(&env.url, user, secret, None, "e2e-list-buckets-opa"))
-        .list_buckets()
-        .send()
-        .await?;
-    let mut names = output
-        .buckets()
-        .iter()
-        .filter_map(|bucket| bucket.name().map(str::to_owned))
-        .collect::<Vec<_>>();
-    names.sort();
-    assert_eq!(names, ["opa-list-location", "opa-list-visible"]);
-
-    let mut evaluations = BTreeSet::new();
-    for _ in 0..6 {
-        let request = opa.next_request().await?;
-        assert_eq!(request.pointer("/input/identity/account").and_then(Value::as_str), Some(user));
-        assert_eq!(request.pointer("/input/context/deny_only").and_then(Value::as_bool), Some(false));
-
-        let action = request
-            .pointer("/input/action")
-            .and_then(Value::as_str)
-            .ok_or("OPA ListBuckets input should include action")?;
-        let bucket = request
-            .pointer("/input/resource/bucket")
-            .and_then(Value::as_str)
-            .ok_or("OPA ListBuckets input should include resource.bucket")?;
-        if bucket.is_empty() {
-            assert_eq!(action, "s3:ListAllMyBuckets");
-            assert!(request.pointer("/input/context/conditions/prefix").is_none());
-            assert!(request.pointer("/input/context/conditions/delimiter").is_none());
-        } else {
-            let expected_arn = format!("arn:aws:s3:::{bucket}");
-            assert_eq!(request.pointer("/input/context/conditions/prefix"), Some(&serde_json::json!([""])));
-            assert_eq!(request.pointer("/input/context/conditions/delimiter"), Some(&serde_json::json!(["/"])));
-            assert_eq!(
-                request.pointer("/input/resource/arn").and_then(Value::as_str),
-                Some(expected_arn.as_str())
-            );
-        }
-        evaluations.insert((action.to_owned(), bucket.to_owned()));
-    }
-    assert_eq!(
-        evaluations,
-        BTreeSet::from([
-            ("s3:GetBucketLocation".to_owned(), "opa-list-hidden".to_owned()),
-            ("s3:GetBucketLocation".to_owned(), "opa-list-location".to_owned()),
-            ("s3:ListAllMyBuckets".to_owned(), String::new()),
-            ("s3:ListBucket".to_owned(), "opa-list-hidden".to_owned()),
-            ("s3:ListBucket".to_owned(), "opa-list-location".to_owned()),
-            ("s3:ListBucket".to_owned(), "opa-list-visible".to_owned()),
-        ])
-    );
-    assert!(
-        matches!(opa.requests.try_recv(), Err(mpsc::error::TryRecvError::Empty)),
-        "ListBuckets should not make redundant OPA evaluations"
-    );
-
-    env.stop_server();
-    Ok(())
-}
-
-#[tokio::test]
-#[serial]
-async fn test_sts_and_list_buckets_fail_closed_while_opa_is_initializing() -> TestResult {
-    init_logging();
-
-    let mut opa = OpaMock::start_blocked().await?;
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str())])
-        .await?;
-    opa.wait_for_validation().await?;
-
-    assert_opa_unavailable_denies_sts_and_list_buckets(&env, "configured OPA initialization").await?;
-
-    env.stop_server();
-    Ok(())
-}
-
-#[tokio::test]
-#[serial]
-async fn test_sts_and_list_buckets_fail_closed_after_opa_validation_failure() -> TestResult {
-    init_logging();
-
-    let mut opa = OpaMock::start_with_mode(OpaValidationMode::Unavailable, None).await?;
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str())])
-        .await?;
-    opa.wait_for_validation().await?;
-
-    assert_opa_unavailable_denies_sts_and_list_buckets(&env, "configured OPA validation failure").await?;
+    opa.release_validation();
+    tokio::time::sleep(Duration::from_millis(200)).await;
+    assert_access_denied(&sts_client(&env.url, user, secret, None), "configured OPA validation failure").await?;
 
     env.stop_server();
     Ok(())

crates/e2e_test/src/tier_transition_regression_test.rs

@@ -1,172 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression tests for Tier/ILM transition operations.
-//!
-//! Covers the recurring pattern where tier transition fails silently, the
-//! free-version recovery task loops forever, or transitioned objects cannot
-//! be read back. This has regressed 6+ times.
-//!
-//! ## Regression Issues
-//!
-//! - rustfs#5218: Remote tier mutation commit failed
-//! - rustfs#5130: tier_free_version_recovery task loops forever
-//! - rustfs#5011: Idle tier free-version recovery rescans every 60 seconds
-//! - rustfs#4826: Full GET of multipart transitioned object fails
-//! - rustfs#5024: Some files succeeded in tier offloading, others failed
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestEnvironment, admin_ok, init_logging};
-    use serde_json::Value;
-    use serial_test::serial;
-    use std::error::Error;
-    use tracing::info;
-
-    type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-    /// RT-13: Verify lifecycle rule with transition persists and is retrievable.
-    ///
-    /// Note: Actual transition requires a configured remote tier. This test
-    /// validates that an expiration-only rule (the persistence path) survives
-    /// a server restart.
-    #[tokio::test]
-    #[serial]
-    async fn test_lifecycle_rule_persists_after_restart() -> TestResult {
-        init_logging();
-        info!("RT-13: lifecycle rule persists after restart");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt13-tier-persist";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Apply a lifecycle rule with expiration (transition needs a real tier)
-        let rule = aws_sdk_s3::types::LifecycleRule::builder()
-            .id("expire-after-90d")
-            .status(aws_sdk_s3::types::ExpirationStatus::Enabled)
-            .filter(aws_sdk_s3::types::LifecycleRuleFilter::builder().prefix("archive/").build())
-            .expiration(aws_sdk_s3::types::LifecycleExpiration::builder().days(90).build())
-            .build()
-            .expect("build rule");
-
-        client
-            .put_bucket_lifecycle_configuration()
-            .bucket(bucket)
-            .lifecycle_configuration(
-                aws_sdk_s3::types::BucketLifecycleConfiguration::builder()
-                    .rules(rule)
-                    .build()
-                    .expect("build config"),
-            )
-            .send()
-            .await
-            .expect("put lifecycle");
-
-        // Restart server
-        env.restart_server_preserving_data(vec![], &[]).await.expect("restart RustFS");
-
-        // Verify the rule survived restart
-        let resp = client
-            .get_bucket_lifecycle_configuration()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("get lifecycle after restart");
-
-        let rules = resp.rules();
-        assert_eq!(rules.len(), 1, "RT-13 FAIL: expected 1 rule after restart");
-
-        let exp = rules[0].expiration().expect("expiration should be set");
-        assert_eq!(exp.days(), Some(90), "RT-13 FAIL: expiration days corrupted after restart");
-
-        info!("RT-13 PASS: lifecycle rule persists after restart");
-        Ok(())
-    }
-
-    /// RT-13b: Verify admin tier configuration API is functional.
-    ///
-    /// Regression pattern: tier add/verify/delete API fails or the tier
-    /// configuration is not persisted (rustfs#5218).
-    #[tokio::test]
-    #[serial]
-    async fn test_admin_tier_list_endpoint_returns_json() -> TestResult {
-        init_logging();
-        info!("RT-13b: admin tier list endpoint returns JSON");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        // Query the tier list endpoint
-        let body = admin_ok(&env, http::Method::GET, "/rustfs/admin/v3/tier", None)
-            .await
-            .expect("list remote tiers");
-
-        let json: Value = serde_json::from_str(&body).expect("tier list response should be valid JSON");
-
-        // Should return an array (possibly empty)
-        assert!(json.is_array(), "RT-13b FAIL: tier list response is not an array: {json}");
-
-        info!("RT-13b PASS: admin tier list endpoint returns valid JSON array");
-        Ok(())
-    }
-
-    /// RT-13c: Verify scanner configuration persistence.
-    ///
-    /// Regression pattern: scanner admin config update reports success but
-    /// is not persisted (rustfs#5013), causing the scanner to not run or
-    /// use stale settings.
-    #[tokio::test]
-    #[serial]
-    async fn test_scanner_config_persists_after_restart() -> TestResult {
-        init_logging();
-        info!("RT-13c: scanner config persists after restart");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        // Get current scanner status
-        let body = admin_ok(&env, http::Method::GET, "/rustfs/admin/v3/scanner/status", None)
-            .await
-            .expect("get scanner status");
-
-        let json: Value = serde_json::from_str(&body).expect("scanner status should be valid JSON");
-
-        info!("  scanner status: {:?}", json.as_object().map(|o| o.keys().collect::<Vec<_>>()));
-
-        // Restart and verify config is still accessible
-        env.restart_server_preserving_data(vec![], &[]).await.expect("restart RustFS");
-
-        let body2 = admin_ok(&env, http::Method::GET, "/rustfs/admin/v3/scanner/status", None)
-            .await
-            .expect("get scanner status after restart");
-
-        let json2: Value = serde_json::from_str(&body2).expect("scanner status after restart should be valid JSON");
-
-        // Both should be valid JSON objects
-        assert!(json2.is_object(), "RT-13c FAIL: scanner status after restart is not a valid JSON object");
-
-        info!("RT-13c PASS: scanner/config persists across restart");
-        Ok(())
-    }
-}

crates/ecstore/Cargo.toml

@@ -32,11 +32,6 @@ workspace = true
 
 [features]
 default = []
-# Compiles the controlled list-objects namespace-journal chaos injector into a
-# production binary (it is always available to tests). Off by default so the
-# RUSTFS_LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_* env vars cannot rewrite journal
-# state in a stock build (backlog#1832).
-list-chaos = []
 rio-v2 = ["dep:rustfs-rio-v2"]
 hotpath = [
     "hotpath/hotpath",
@@ -149,13 +144,11 @@ rustfs-lifecycle.workspace = true
 rustfs-s3-types = { workspace = true }
 rustfs-data-usage.workspace = true
 rustfs-object-capacity.workspace = true
-rustfs-object-data-cache = { workspace = true, features = ["runtime-memory"] }
 arc-swap.workspace = true
 async-trait.workspace = true
 bytes = { workspace = true, features = ["serde"] }
 byteorder = { workspace = true }
 chrono = { workspace = true, features = ["serde"] }
-jiff = { workspace = true, features = ["serde"] }
 glob = { workspace = true }
 thiserror.workspace = true
 flatbuffers.workspace = true
@@ -186,7 +179,6 @@ path-absolutize = { workspace = true }
 rmp.workspace = true
 rmp-serde.workspace = true
 tokio-util = { workspace = true, features = ["io", "compat"] }
-tokio-stream = { workspace = true, features = ["sync"] }
 base64 = { workspace = true }
 hmac = { workspace = true }
 sha1 = { workspace = true }
@@ -245,19 +237,7 @@ rustfs-uring = "0.2.1"
 
 [target.'cfg(windows)'.dependencies]
 winapi-util.workspace = true
-windows-sys = { workspace = true, features = [
-    "Wdk_Foundation",
-    "Wdk_Storage_FileSystem",
-    "Win32_Foundation",
-    "Win32_Security",
-    "Win32_Storage_FileSystem",
-    "Win32_System_IO",
-    "Win32_System_SystemServices",
-    "Win32_System_WindowsProgramming",
-] }
-
-[target.'cfg(windows)'.dev-dependencies]
-windows-sys = { workspace = true, features = ["Win32_System_Ioctl"] }
+windows-sys = { workspace = true, features = ["Win32_Foundation", "Win32_Storage_FileSystem"] }
 
 [dev-dependencies]
 tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util", "fs"] }

crates/ecstore/benches/single_block_non_inline_benchmark.rs

@@ -69,7 +69,6 @@ fn build_non_inline_writers(config: &BenchConfig) -> Vec<Option<BitrotWriterWrap
 fn bench_single_block_non_inline_fast_path(c: &mut Criterion) {
     let configs = vec![
         BenchConfig::new(4 * 1024, 4, 2, 128 * 1024),
-        BenchConfig::new(16 * 1024, 4, 2, 128 * 1024),
         BenchConfig::new(64 * 1024, 4, 2, 128 * 1024),
         BenchConfig::new(128 * 1024, 4, 2, 128 * 1024),
     ];
@@ -113,12 +112,7 @@ fn bench_single_block_non_inline_fast_path(c: &mut Criterion) {
                 rt.block_on(async {
                     erasure
                         .clone()
-                        .encode_single_block_non_inline_with_size_hint(
-                            reader,
-                            &mut writers,
-                            config.data_shards,
-                            config.payload_size,
-                        )
+                        .encode_single_block_non_inline(reader, &mut writers, config.data_shards)
                         .await
                         .expect("single block candidate benchmark");
                 });

crates/ecstore/src/api/mod.rs

@@ -32,7 +32,7 @@ pub mod bucket {
     pub mod bucket_target_sys {
         pub use crate::bucket::bucket_target_sys::{
             AdvancedPutOptions, BucketTargetError, BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError,
-            TargetClient, append_version_id_query,
+            TargetClient,
         };
     }
 
@@ -61,11 +61,9 @@ pub mod bucket {
                 delete_manual_transition_scope_admission_if_current, load_manual_transition_job_record,
                 load_manual_transition_job_record_with_etag, load_manual_transition_scope_admission,
                 manual_transition_job_lease_expired, manual_transition_scope_admission_lease_expired,
-                manual_transition_scope_key, persist_manual_transition_job_progress,
-                persist_manual_transition_job_progress_if_owned, renew_manual_transition_job_lease,
-                renew_manual_transition_job_lease_if_owned, request_manual_transition_job_cancel,
-                save_manual_transition_job_record, save_manual_transition_job_record_if_current,
-                save_manual_transition_scope_admission_if_absent, update_manual_transition_job_record,
+                manual_transition_scope_key, persist_manual_transition_job_progress, renew_manual_transition_job_lease,
+                request_manual_transition_job_cancel, save_manual_transition_job_record,
+                save_manual_transition_job_record_if_current, save_manual_transition_scope_admission_if_absent,
             };
         }
 
@@ -132,15 +130,13 @@ pub mod bucket {
 
     pub mod metadata_sys {
         pub use crate::bucket::metadata_sys::{
-            BucketMetadataMutationGuard, BucketMetadataSys, ObjectLockConfigState, acquire_bucket_metadata_transaction_lock,
-            capture_bucket_metadata_incarnation, delete, delete_if_incarnation, get, get_accelerate_config, get_bucket_policy,
+            BucketMetadataSys, acquire_bucket_metadata_transaction_lock, delete, get, get_accelerate_config, get_bucket_policy,
             get_bucket_policy_raw, get_bucket_targets_config, get_config_from_disk, get_cors_config, get_durability_config,
             get_global_bucket_metadata_sys, get_lifecycle_config, get_logging_config, get_notification_config,
-            get_object_lock_config, get_object_lock_config_state, get_public_access_block_config, get_quota_config,
-            get_replication_config, get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config,
-            get_website_config, init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
-            set_bucket_metadata, update, update_bucket_targets_under_transaction_lock, update_config_with, update_if_incarnation,
-            update_under_transaction_lock,
+            get_object_lock_config, get_public_access_block_config, get_quota_config, get_replication_config,
+            get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
+            init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata, set_bucket_metadata,
+            update, update_bucket_targets_under_transaction_lock, update_config_with, update_under_transaction_lock,
         };
     }
 
@@ -182,24 +178,20 @@ pub mod bucket {
             mrf_backlog_observability_snapshot,
         };
         pub use crate::bucket::replication::{
-            BucketReplicationResyncStatus, BucketReplicationStats, BucketStats, DeleteReplicationConfigSnapshot,
-            DeletedObjectReplicationInfo, DurableMrfBacklog, DynReplicationPool, MrfOpKind, MrfReplicateEntry,
-            MustReplicateOptions, ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS,
-            REMOTE_TARGET_WRITABLE_FIELDS, REPLICATE_INCOMING_DELETE, REPLICATION_CAPABILITY_CONTRACT_VERSION,
-            REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS, ReplicateDecision, ReplicateObjectInfo,
-            ReplicationBatchAdmission, ReplicationConfig, ReplicationConfigStructureError, ReplicationConfigurationExt,
+            BucketReplicationResyncStatus, BucketStats, DeleteReplicationConfigSnapshot, DeletedObjectReplicationInfo,
+            DurableMrfBacklog, DynReplicationPool, MrfOpKind, MrfReplicateEntry, MustReplicateOptions, ObjectOpts,
+            REPLICATE_INCOMING_DELETE, ReplicateDecision, ReplicateObjectInfo, ReplicationConfig, ReplicationConfigurationExt,
             ReplicationDeleteScheduleInput, ReplicationDeleteStateSource, ReplicationHealQueueResult, ReplicationObjectBridge,
             ReplicationObjectIO, ReplicationOperation, ReplicationPoolTrait, ReplicationPriority, ReplicationQueueAdmission,
             ReplicationScannerBridge, ReplicationState, ReplicationStats, ReplicationStatusType, ReplicationStorage,
             ReplicationTargetValidationError, ReplicationType, ResyncOpts, ResyncStatusType, RuntimeReplicationTargetBacklog,
-            TargetReplicationResyncStatus, VersionPurgeStatusType, commit_force_delete_intent, complete_force_delete_intent,
-            delete_replication_state_from_config, delete_replication_version_id, get_global_replication_pool,
-            get_global_replication_stats, init_background_replication, invalid_replication_config_status_field,
-            persist_force_delete_intent, read_durable_mrf_backlog, replication_state_to_filemeta, replication_status_to_filemeta,
-            replication_statuses_map, replication_target_arns, resync_start_conflict_id, should_remove_replication_target,
-            should_schedule_delete_replication, should_use_existing_delete_replication_info,
-            should_use_existing_delete_replication_source, unsupported_replication_config_field,
-            validate_replication_config_structure, validate_replication_config_target_arns, version_purge_status_to_filemeta,
+            TargetReplicationResyncStatus, VersionPurgeStatusType, delete_replication_state_from_config,
+            delete_replication_version_id, get_global_replication_pool, get_global_replication_stats,
+            init_background_replication, invalid_replication_config_status_field, read_durable_mrf_backlog,
+            replication_state_to_filemeta, replication_status_to_filemeta, replication_statuses_map, replication_target_arns,
+            resync_start_conflict_id, should_remove_replication_target, should_schedule_delete_replication,
+            should_use_existing_delete_replication_info, should_use_existing_delete_replication_source,
+            unsupported_replication_config_field, validate_replication_config_target_arns, version_purge_status_to_filemeta,
         };
     }
 
@@ -208,9 +200,7 @@ pub mod bucket {
     }
 
     pub mod target {
-        pub use crate::bucket::target::{
-            ARN, BucketTarget, BucketTargetType, BucketTargets, Credentials, LatencyStat, duration_from_secs_or_nanos,
-        };
+        pub use crate::bucket::target::{ARN, BucketTarget, BucketTargetType, BucketTargets, Credentials, LatencyStat};
     }
 
     pub mod utils {
@@ -283,7 +273,7 @@ pub mod config {
     pub mod com {
         pub use crate::config::com::{
             COMMA_SEPARATED_LISTS, CONFIG_PREFIX, ENV_CONFIG_RECOVER_ON_CORRUPTION, STORAGE_CLASS_SUB_SYS,
-            ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config, delete_config_no_lock,
+            ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config,
             is_server_config_corrupt_error, lookup_configs, read_config, read_config_no_lock, read_config_with_metadata,
             read_config_without_migrate, read_config_without_migrate_no_lock, read_existing_server_config_no_lock,
             read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_server_config,
@@ -310,12 +300,9 @@ pub mod config {
 }
 
 pub mod data_usage {
-    #[cfg(feature = "test-util")]
-    pub use crate::data_usage::seed_bucket_usage_memory_for_test;
     pub use crate::data_usage::{
         DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
-        init_compression_total_memory_from_backend, invalidate_admin_data_usage_snapshot_cache,
-        invalidate_data_usage_snapshot_cache, live_bucket_usage_computations, load_admin_data_usage_from_backend_cached,
+        init_compression_total_memory_from_backend, invalidate_data_usage_snapshot_cache, live_bucket_usage_computations,
         load_compression_total_from_memory, load_data_usage_from_backend, load_data_usage_from_backend_cached,
         record_bucket_delete_marker_memory, record_bucket_object_delete_memory, record_bucket_object_version_write_memory,
         record_bucket_object_write_memory, record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
@@ -330,8 +317,8 @@ pub mod disk {
     pub use crate::disk::local::ScanGuard;
     pub use crate::disk::{
         BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
-        CheckPartsResp, ConditionalFileUpdate, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
-        DiskStore, FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
+        CheckPartsResp, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, DiskStore,
+        FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
         PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
         STORAGE_FORMAT_FILE, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk,
         validate_batch_read_version_item_count,
@@ -346,7 +333,7 @@ pub mod disk {
     }
 
     pub mod error {
-        pub use crate::disk::error::{DiskError, Error, FileAccessDeniedWithContext, Result};
+        pub use crate::disk::error::{BitrotErrorType, DiskError, Error, FileAccessDeniedWithContext, Result};
     }
 
     pub mod error_reduce {
@@ -412,16 +399,13 @@ pub mod notification {
 pub mod object {
     pub use crate::object_api::{
         BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
-        GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, NamespaceLockFence, ObjectEncryptionResolver,
-        ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, QuotaAdmission,
-        RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest, StreamConsumer,
-        get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
-        register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
-    };
-    pub use crate::store::{
-        PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError,
-        SnapshotConsistencyError,
+        GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, ObjectEncryptionResolver, ObjectInfo,
+        ObjectMutationHook, ObjectOptions, PutObjReader, RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode,
+        ReadEncryptionRequest, StreamConsumer, get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook,
+        register_get_object_body_cache_hook, register_object_mutation_hook, unregister_get_object_body_cache_hook,
+        unregister_object_mutation_hook,
     };
+    pub use crate::store::PreparedGetObjectReader;
 }
 
 pub mod rebalance {
@@ -443,29 +427,18 @@ pub mod rpc {
     pub use crate::cluster::rpc::{
         AuthenticatedChannel, KMS_SIGNAL_SUBSYSTEM, LocalPeerS3Client, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS,
         PeerRestClient, PeerS3Client, S3PeerSys, SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC,
-        ScannerBucketListing, ScannerPeerActivity, TONIC_RPC_PREFIX, TonicInterceptor, build_put_file_auth_trailer,
-        check_and_record_signed_rpc_nonce, gen_signature_headers, gen_tonic_replay_scope_headers, gen_tonic_signature_headers,
-        gen_tonic_signature_interceptor, node_service_time_out_client, node_service_time_out_client_no_auth,
-        normalize_tonic_rpc_audience, set_tonic_canonical_body_digest, sign_ns_scanner_capability, sign_put_file_capability,
-        sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers,
-        tonic_rpc_auth_failure_reason, verify_put_file_auth_trailer, verify_put_file_capability, verify_rpc_signature,
-        verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest,
-        verify_tonic_rpc_response_proof, verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
+        ScannerBucketListing, ScannerPeerActivity, TONIC_RPC_PREFIX, TonicInterceptor, gen_signature_headers,
+        gen_tonic_replay_scope_headers, gen_tonic_signature_headers, gen_tonic_signature_interceptor,
+        node_service_time_out_client, node_service_time_out_client_no_auth, normalize_tonic_rpc_audience,
+        set_tonic_canonical_body_digest, sign_ns_scanner_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge,
+        tonic_boot_epoch_response_headers, verify_rpc_signature, verify_tonic_boot_epoch_response,
+        verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest, verify_tonic_rpc_response_proof,
+        verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
     };
 }
 
 pub mod set_disk {
     pub use crate::set_disk::{DEFAULT_READ_BUFFER_SIZE, SetDisks, get_lock_acquire_timeout, is_valid_storage_class};
-
-    /// Return the canonical object-metadata identity used for read-quorum grouping.
-    pub fn file_info_quorum_hash(meta: &rustfs_filemeta::FileInfo) -> [u8; 32] {
-        crate::set_disk::SetDisks::file_info_quorum_hash(meta)
-    }
-
-    #[cfg(feature = "test-util")]
-    pub mod test_util {
-        pub use crate::set_disk::{PutObjectCommitBarrier, PutObjectCommitPause};
-    }
 }
 
 pub mod store_list {

crates/ecstore/src/bucket/bandwidth/monitor.rs

@@ -13,7 +13,7 @@
 // limitations under the License.
 
 use crate::bucket::bandwidth::reader::BucketOptions;
-use ratelimit::{Clock, Error as RatelimitError, Ratelimiter};
+use ratelimit::{Error as RatelimitError, Ratelimiter};
 use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
 use std::sync::atomic::{AtomicU64, Ordering};
@@ -24,33 +24,6 @@ use tracing::warn;
 /// BETA_BUCKET is the weight used to calculate exponential moving average
 const BETA_BUCKET: f64 = 0.1;
 
-// ratelimit 2.0 stores tokens at six decimal places. Above this limit its
-// scaled capacity and token-cost calculations saturate instead of preserving
-// the configured bandwidth.
-const MAX_RATELIMIT_TOKENS: i64 = 18_446_744_073_709;
-
-fn consume_tokens<C: Clock>(limiter: &Ratelimiter<C>, n: u64) -> (u64, f64, u64) {
-    if n == 0 {
-        return (0, limiter.rate() as f64, 0);
-    }
-
-    let mut consumed = 0u64;
-    // Consuming one token also refills the bucket based on elapsed time, so
-    // the subsequent `available()` read reflects freshly accrued tokens.
-    if limiter.try_wait().is_ok() {
-        consumed = 1;
-    }
-    let available = limiter.available();
-    let to_consume = n - consumed;
-    let batch = to_consume.min(available);
-    if batch > 0 && limiter.try_wait_n(batch).is_ok() {
-        consumed += batch;
-    }
-    let deficit = n.saturating_sub(consumed);
-    let rate = limiter.rate() as f64;
-    (deficit, rate, consumed)
-}
-
 #[derive(Clone)]
 pub struct BucketThrottle {
     limiter: Arc<Mutex<Ratelimiter>>,
@@ -61,9 +34,9 @@ impl BucketThrottle {
     fn new(node_bandwidth_per_sec: i64) -> Result<Self, RatelimitError> {
         let node_bandwidth_per_sec = node_bandwidth_per_sec.max(1);
         let amount = node_bandwidth_per_sec as u64;
-        // ratelimit 2.0's builder takes a per-second rate; the refill period
-        // defaults to one second, so `amount` tokens accrue per second.
-        let limiter_inner = Ratelimiter::builder(amount).max_tokens(amount).build()?;
+        let limiter_inner = Ratelimiter::builder(amount, Duration::from_secs(1))
+            .max_tokens(amount)
+            .build()?;
         Ok(Self {
             limiter: Arc::new(Mutex::new(limiter_inner)),
             node_bandwidth_per_sec,
@@ -74,21 +47,32 @@ impl BucketThrottle {
         self.limiter.lock().unwrap_or_else(|e| e.into_inner()).max_tokens()
     }
 
-    /// Best-effort bulk token consumption: consume up to `n` tokens and report
-    /// how many were taken plus any shortfall.
-    ///
-    /// `try_wait_n` on the ratelimit crate is all-or-nothing, so we cannot ask
-    /// for `n` directly and still consume a partial amount. Instead we take one
-    /// token first (which also triggers the internal time-based refill), read
-    /// the now-current available count, and consume `min(remaining, available)`
-    /// in a single `try_wait_n` call — that batch never exceeds `available`, so
-    /// it always succeeds.
+    /// The ratelimit crate (0.10.0) does not provide a bulk token consumption API.
+    /// try_wait() first to consume 1 token AND trigger the internal refill
+    /// mechanism (tokens are only refilled during try_wait/wait calls).
+    /// directly adjust available tokens via set_available() to consume the remaining amount.
     pub(crate) fn consume(&self, n: u64) -> (u64, f64, u64) {
         let guard = self.limiter.lock().unwrap_or_else(|e| {
             warn!("bucket throttle mutex poisoned, recovering");
             e.into_inner()
         });
-        consume_tokens(&guard, n)
+        if n == 0 {
+            return (0, guard.rate(), 0);
+        }
+        let mut consumed = 0u64;
+        if guard.try_wait().is_ok() {
+            consumed = 1;
+        }
+        let available = guard.available();
+        let to_consume = n - consumed;
+        let batch = to_consume.min(available);
+        if batch > 0 {
+            let _ = guard.set_available(available - batch);
+            consumed += batch;
+        }
+        let deficit = n.saturating_sub(consumed);
+        let rate = guard.rate();
+        (deficit, rate, consumed)
     }
 }
 
@@ -345,16 +329,6 @@ impl Monitor {
                 "bandwidth limit too small for cluster size, per-node limit will clamp to 1 byte/s"
             );
         }
-        if limit_bytes > MAX_RATELIMIT_TOKENS {
-            warn!(
-                bucket = bucket,
-                arn = arn,
-                limit_bytes = limit_bytes,
-                max_limit_bytes = MAX_RATELIMIT_TOKENS,
-                "bandwidth limit exceeds ratelimiter capacity, throttling disabled for this target"
-            );
-            return;
-        }
         let opts = BucketOptions {
             name: bucket.to_string(),
             replication_arn: arn.to_string(),
@@ -401,30 +375,6 @@ mod tests {
     use super::*;
     use std::panic::{AssertUnwindSafe, catch_unwind};
 
-    #[derive(Clone)]
-    struct TestClock {
-        elapsed_ns: Arc<AtomicU64>,
-    }
-
-    impl TestClock {
-        fn new() -> Self {
-            Self {
-                elapsed_ns: Arc::new(AtomicU64::new(0)),
-            }
-        }
-
-        fn advance(&self, duration: Duration) {
-            let elapsed_ns = u64::try_from(duration.as_nanos()).unwrap_or(u64::MAX);
-            self.elapsed_ns.fetch_add(elapsed_ns, Ordering::Relaxed);
-        }
-    }
-
-    impl Clock for TestClock {
-        fn elapsed(&self) -> Duration {
-            Duration::from_nanos(self.elapsed_ns.load(Ordering::Relaxed))
-        }
-    }
-
     #[test]
     fn test_set_and_get_throttle_with_node_split() {
         let monitor = Monitor::new(4);
@@ -476,15 +426,6 @@ mod tests {
         assert!(!monitor.is_throttled("b1", "arn1"));
     }
 
-    #[test]
-    fn test_set_bandwidth_limit_rejects_unrepresentable_rate() {
-        let monitor = Monitor::new(1);
-
-        monitor.set_bandwidth_limit("b1", "arn1", MAX_RATELIMIT_TOKENS + 1);
-
-        assert!(!monitor.is_throttled("b1", "arn1"));
-    }
-
     #[test]
     fn test_consume_returns_deficit_when_tokens_exhausted() {
         let throttle = BucketThrottle::new(100).expect("test");
@@ -495,23 +436,6 @@ mod tests {
         assert!(rate > 0.0);
     }
 
-    #[test]
-    fn test_consume_refills_continuously() {
-        let clock = TestClock::new();
-        let limiter = Ratelimiter::with_clock(100, clock.clone());
-
-        assert_eq!(consume_tokens(&limiter, 100), (100, 100.0, 0));
-
-        clock.advance(Duration::from_millis(250));
-        assert_eq!(consume_tokens(&limiter, 100), (75, 100.0, 25));
-
-        clock.advance(Duration::from_millis(250));
-        assert_eq!(consume_tokens(&limiter, 100), (75, 100.0, 25));
-
-        clock.advance(Duration::from_millis(500));
-        assert_eq!(consume_tokens(&limiter, 100), (50, 100.0, 50));
-    }
-
     #[test]
     fn test_consume_no_deficit_when_tokens_sufficient() {
         let throttle = BucketThrottle::new(10000).expect("test");

crates/ecstore/src/bucket/bandwidth/reader.rs

@@ -306,7 +306,7 @@ mod tests {
     #[tokio::test]
     async fn test_monitored_reader_header_size_accounting() {
         let monitor = Monitor::new(1);
-        monitor.set_bandwidth_limit("b1", "arn1", 1_000_000_000);
+        monitor.set_bandwidth_limit("b1", "arn1", 100);
 
         let data = vec![0u8; 200];
         let inner = TestAsyncReader::new(&data);