feat(obs): enhance observability with tracing spans and metrics integration

.agents/skills/adversarial-validation/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: adversarial-validation
-description: Execute the Adversarial Validation policy from the root AGENTS.md — run the seven reviewer roles (correctness, simplicity, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
+description: Execute the Adversarial Validation policy from the root AGENTS.md — run the six reviewer roles (correctness, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
 ---
 
 # Adversarial Validation Playbooks
@@ -53,22 +53,14 @@ shipped bug or rule that earns each probe its place.
 - Exercise the zero/empty end of every new size or count parameter: zero-length object PUT then GET (body must be empty, not error), part count 0, empty Vec of disks/entries into aggregation functions, and env/config values of 0 (must clamp or reject, never divide-by-zero or 'scan nothing and report zero usage'). Anywhere the diff computes a ratio, capacity, or progress percentage, plug in 0 and the max value.
   - Where: crates/ecstore aggregation and scanner paths; crates/object-capacity; config/env parsing in touched crates
   - Evidence: Commits 787cc77a7 'clamp zero capacity env values to safe defaults' (#4559) and 32b1094ec 'resolve a symlinked scan root instead of silently counting zero' (#4564) — zero-as-silent-wrong-answer is a recurring repo bug class.
+- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
+  - Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
+  - Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Constant and String Usage'; Adversarial Validation section names the smaller-diff clause as a correctness-adversary finding.
 - For any diff touching multipart or object commit paths, order the operations on paper and attack the failure point between them: kill the process (or return Err) after the commit rename but before cleanup, and after cleanup but before commit. Verify the earlier-failure case leaves the object readable and the later-failure case leaves no half-visible object; part meta files must never be deleted before the commit is durable.
   - Where: crates/ecstore multipart commit/cleanup (set_disk/ops); rustfs/src/storage multipart handlers
   - Evidence: Commit c77c5f047 'defer multipart part.N.meta cleanup until after commit' (#4548) — cleanup-before-commit ordering already caused a real data-loss window; the #4221 durability work shows fsync/ordering bugs are endemic here.
 
-Null report example: "Attacked quorum-1 error reduction, exact max-keys listing boundary, trailing-slash dir keys, nil-UUID tier versionId, and mid-stream reconstruct error propagation — no break found."
-
-### Simplicity adversary
-
-- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
-  - Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
-  - Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Reuse Before You Write' (constants clause); the Adversarial Validation roles list charters the simplicity adversary with exactly this attack.
-- Reuse-and-necessity attack: for each new helper the diff introduces, run `ls crates/utils/src crates/common/src` and `rg -i 'fn \w*<term>'` over those dirs plus the touched crate (snake_case signatures — a full-text single-word grep drowns, a multi-word phrase returns nothing). A reimplementation of an existing workspace utility, or of plain std/tokio behavior no wrapper refines, is a finding — but so is forced reuse with mismatched semantics (normalization such as `clean` resolving `.`/`..` against raw S3 keys, error type, backoff, durability gating). For each new defensive branch, demand the nameable trigger and flag re-validation of what a validated upstream layer on the SAME path already guarantees — excluding the Cross-Cutting Domain Invariant patterns (nil/empty/absent UUID, dual metadata keys, unversioned-tier versionId) and re-checks before destructive actions, which are load-bearing even when redundant on the happy path. For each new test, flag near-duplicates pinning the same code path AND poison-value class as an existing test — boundary companions (n==max vs max+1, absent vs empty vs nil UUID, MetaObject vs MetaDeleteMarker) are never near-duplicates; the test-coverage skeptic playbook below mandates them.
-  - Where: Any diff adding helpers, branches on decoded/peer data, or tests; helper checks against crates/utils, crates/common, and the touched crate
-  - Evidence: AGENTS.md 'Reuse Before You Write' and 'Necessary Code Only'; GHSA-f4vq-9ffr-m8m3 (normalization-asymmetry traversal — why forced reuse of normalizing helpers on raw keys is itself an attack); docs/operations/tier-ilm-debugging.md nil-versionId incident (why boundary re-checks are load-bearing).
-
-Null report example: "Rewrote the diff as an in-place edit (no smaller equivalent exists), grepped both new helpers against crates/utils, crates/common, and the touched crate (no existing equivalent; call-site semantics checked), verified the two new defensive branches name concrete corrupt-input triggers, and checked the added tests against the existing suite (each pins a distinct poison-value class) — no break found."
+Null report example: "Attacked quorum-1 error reduction, exact max-keys listing boundary, trailing-slash dir keys, nil-UUID tier versionId, mid-stream reconstruct error propagation, and a minimal-diff rewrite — no break found; diff is already the minimal in-place edit."
 
 ### Security reviewer
 
@@ -84,9 +76,6 @@ Null report example: "Rewrote the diff as an in-place edit (no smaller equivalen
 - For any secret/token/signature/password comparison in the diff, check it uses a constant-time compare (e.g. subtle/constant_time_eq), not == or early-return byte loops. Then check the failure-response paths: construct an invalid-user request and an invalid-secret request and confirm they are indistinguishable (same error, no early length short-circuit) so an attacker cannot enumerate valid users or time-side-channel the secret.
   - Where: crates/protocols/ (FTPS/WebDAV/FormPost auth), crates/credentials/, rustfs/src/auth.rs, RPC signature verification
   - Evidence: GHSA-3p3x-734c-h5vx (FTPS/WebDAV early-return string equality + distinguishable invalid-user vs invalid-password). Fix commits 3c3113619 (constant-time FTPS/WebDAV) and c41062f27 (constant-time FormPost signature). 3p3x was fixed by PR #4403.
-- If the diff parses or transports secret-bearing config (env vars, key files, connection strings), grep every error-construction and format site on that value's path (`format!` feeding `Error::other`/`configuration_error`/`panic!`/`expect`) for interpolation of the raw value or of variables named like secret material. Construct the likeliest misconfiguration: the operator supplies the bare secret without the expected `<name>:` prefix (or with a stray newline) — if the parse-failure hint echoes the input, the secret lands in startup logs. Error strings are log content; the hint may name the env var and expected format, never the value. If the diff re-implements an existing parse helper, diff the two error paths — the duplicate is where the leak hides.
-  - Where: rustfs/src/init.rs (env plumbing), crates/kms/src/config.rs, crates/credentials/, any from_env/parse on secret values; mechanical backstop in scripts/check_logging_guardrails.sh (secret-interpolation check)
-  - Evidence: PR #5222 introduced `got: {secret_str}` in build_static_kms_config's format-hint error — a bare base64 key (the secret itself) would have been echoed into startup logs; fixed by PR #5243. The parallel parse in KmsConfig::from_env already omitted the value: the leak lived only in the duplicated copy (AGENTS.md 'Reuse Before You Write').
 - If the diff touches internode/RPC auth secret handling, trace whether the RPC HMAC secret can fall back to a public default (e.g. 'rustfsadmin', 'rustfs rpc') or be derived deterministically from the S3 root credentials. Construct the case where RUSTFS_RPC_SECRET is unset and confirm the code fails closed rather than silently using a default or a root-derived key. Verify RPC signing keys are independent random secrets, not reused across S3-root/RPC-HMAC/STS-JWT roles.
   - Where: crates/credentials/, crates/ecstore/src/rpc/, internode auth setup
   - Evidence: GHSA-r5qv-rc46-hv8q (fell back to 'rustfsadmin'), GHSA-75fx/68cw (RPC secret derivable from root creds → forgeable signatures), GHSA-h956 (hard-coded 'rustfs rpc'), GHSA-m77q (STS JWT reused root secret). Fix commit 7b2055405 (fail closed when deriving RPC secret from default credentials, PR#4402).
@@ -254,7 +243,7 @@ Null report example: "Attacked the new rename_data commit-section work, durabili
 - If the diff writes internal object metadata, run the dual-key mutation: delete the `x-minio-internal-<suffix>` write (keeping only `x-rustfs-internal-`) and check whether any test fails. Because `get_bytes` prefers the RustFS key, every read-back test stays green while MinIO interop is silently broken — coverage must include an assertion that BOTH keys are present in the stored metadata map.
   - Where: crates/utils/src/http/metadata_compat.rs and all its callers in crates/ecstore and rustfs/src/storage
   - Evidence: CLAUDE.md domain convention: metadata must be written under both x-rustfs-internal- and x-minio-internal- keys for MinIO interop; get_bytes prefers the RustFS key, making the MinIO-key half of the invariant invisible to read-back tests.
-- For changed quorum/version/UUID logic, name the tests covering the specific poison values: quorum−1 disks, nil UUID, absent vs empty vs nil-serialized UUID bytes, remote-tier version_id of None/"" (unversioned tier bucket → no versionId sent), and the same metadata read on both MetaObject and MetaDeleteMarker version types. Mutation check: remove a `.filter(|u| !u.is_nil())` guard from the diff and confirm a test fails; if none does, the nil-UUID class is uncovered.
+- For changed quorum/version/UUID logic, name the tests covering the specific poison values: quorum−1 disks, nil UUID, absent vs empty vs nil-serialized UUID bytes, and remote-tier version_id of None/"" (unversioned tier bucket → no versionId sent). Mutation check: remove a `.filter(|u| !u.is_nil())` guard from the diff and confirm a test fails; if none does, the nil-UUID class is uncovered.
   - Where: crates/ecstore (tier recovery, heal, quorum paths), crates/filemeta, code reading UUIDs from xl.meta metadata
   - Evidence: Commit 726f3dc18 (#4552) fixed rejection of empty remote version_id in tier recovery. CLAUDE.md invariant: absent/empty/nil UUID all mean 'no value', not Uuid::nil(). docs/operations/tier-ilm-debugging.md: None/"" tier version means unversioned bucket. df9cbc4ed (#4427): unvalidated distribution values caused shuffle index panic — edge values reached production untested.
 - For any pagination/limit/truncation change, construct the exact-boundary test: result count == max (page exactly full), max+1, and a delimiter re-fold that lands precisely on the page boundary — assert both the item count AND the is_truncated/continuation marker. Off-by-one at the page boundary is a recurring shipped bug here.

.agents/skills/code-change-verification/SKILL.md

@@ -43,7 +43,16 @@ Use this skill to review code changes consistently before merge, before release,
 
 #### Rust-specific checks (apply to all Rust changes)
 
-Run the full checklist in [rust-code-quality](../rust-code-quality/SKILL.md) — the canonical Rust review checklist for the unwrap/casting/cloning/locking/recursion/error-type/serde/test rules and the reuse-and-necessity checks (duplicated helpers, defensive branches without a nameable trigger, redundant error wrapping). Do not restate those rules here; carry its P0–P3 ratings over unchanged and use this skill's output format.
+- **unwrap/expect in production**: Search changed files for `.unwrap()` and `.expect(` outside test modules. Every `unwrap()` in production code must have a justification comment or be replaced with `?`.
+- **Silent type truncation**: Search for `as u8/u16/u32/u64/usize/i8/i16/i32/i64/isize` casts. Every `as` cast must be justified; negative-to-unsigned and large-to-small are bugs by default. Use `try_into()` or explicit clamping.
+- **Unnecessary cloning**: Check `.clone()` calls in loops, per-request paths, and on structs with >5 heap-allocated fields. Consider `Arc`, references, or `Cow<str>`.
+- **Lock ordering**: If the change acquires multiple locks, verify the order matches all other call sites. Document the order in a comment.
+- **Locks across .await**: Flag any `tokio::sync::RwLock`/`Mutex` guard held across an `.await` point without bounded hold time.
+- **Recursion depth**: If the change adds or modifies a recursive function, verify it has a depth limit or uses iterative traversal with an explicit stack.
+- **Error types**: Flag `Result<_, String>`, `Box<dyn Error>`, and missing `Error::source()` implementations in public APIs.
+- **Test assertions**: Every test function must have at least one `assert!`. Flag tests that only call code without verifying results.
+- **println/eprintln**: Search changed files for `println!`/`eprintln!` outside test modules. Production code must use `tracing` macros.
+- **Serde safety**: Structs deserialized from untrusted input (S3 API, user config) should have `#[serde(deny_unknown_fields)]`.
 
 ### 4) Findings-first output
 - Order findings by severity:

.agents/skills/rust-code-quality/SKILL.md

@@ -36,9 +36,6 @@ rg -n 'println!\|eprintln!' <changed-files> | grep -v test
 
 # 6. Ordering::Relaxed usage (verify each is intentional)
 rg -n 'Ordering::Relaxed' <changed-files>
-
-# 7. Default substituted for a possibly-required value (judge each: is the value optional by domain?)
-rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
 ```
 
 ## Manual Review Checklist
@@ -58,8 +55,8 @@ For every Rust code change, verify:
 - [ ] No `f64 as usize` without prior clamping
 
 ### Concurrency
-- [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
-- [ ] No `tokio::sync` lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
+- [ ] Lock acquisition order is documented when multiple locks are used
+- [ ] No `tokio::sync` write guards held across `.await` without bounded hold time
 - [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
 - [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
 
@@ -87,19 +84,12 @@ For every Rust code change, verify:
 - [ ] No camelCase statics or Hungarian notation
 - [ ] New string literals don't duplicate existing constants
 
-### Reuse and Necessity
-- [ ] No new helper duplicating an existing workspace utility (`crates/utils`, `crates/common`, the touched crate) or plain std/tokio behavior no wrapper refines; reused helpers match the call site's semantics (normalization, error type, backoff, durability gating)
-- [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
-- [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
-- [ ] No comments narrating the next line, restating a signature, or describing the change itself (invariant comments — lock ordering, `SAFETY`, unwrap justification — are not narration)
-- [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
-
 ## Severity Classification
 
 - **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
-- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method, `unwrap_or_default()` on a domain-required value (metadata, quorum, version id)
-- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`, new helper duplicating an existing workspace utility, defensive branch with no nameable trigger (corrupt or stale persisted/peer data is always a nameable trigger for boundary-crossing values), near-duplicate test, redundant error re-wrapping
-- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`, narrating comment
+- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method
+- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`
+- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`
 
 ## Output Template
 

.agents/skills/rustfs-release-publish/SKILL.md

@@ -1,169 +0,0 @@
----
-name: rustfs-release-publish
-description: "End-to-end RustFS release pipeline: bump version files on main directly to the final target version, cut a preview tag on that commit, verify the CI build and release artifacts, run the downloaded binary locally and exercise the console, validate the server with the latest rc client, then publish the final tag on the SAME validated commit — never a new bump commit, never latest main. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
----
-# RustFS Release Publish (preview-validated pipeline)
-
-This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
-
-Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and prerelease classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
-
-Pipeline shape:
-
-```
-bump version files to <target> (final version, ONE commit) -> merge
-  -> tag <preview-tag> at that commit -> CI green
-  -> verify release artifacts -> run binary locally + console checks
-  -> validate with latest rc client
-  -> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
-```
-
-On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
-
-## Required inputs
-
-- Final target version, for example `1.0.0-beta.10`.
-- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` — and for stable targets `git tag -l '<target>-rc.*'` — after `git fetch --tags`).
-
-If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
-
-## Semver gate — confirm the target version before touching anything
-
-Versions follow [SemVer 2.0.0](https://semver.org/). Precedence reminder:
-
-```
-1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0 < 1.0.1 < 1.1.0 < 2.0.0
-```
-
-Numeric prerelease identifiers compare numerically (`beta.9 < beta.10`), not lexically — see [semver.org spec item 11](https://semver.org/#spec-item-11). Preview tags are internal validation tags layered on top of the target's prerelease channel — they are never themselves a deliverable version and never appear in version files.
-
-Rules:
-
-- A request like "发个版" / "release the next version" without an exact version string is ALWAYS ambiguous. Derive the current latest tag (`git tag --sort=-v:refname | head`), then ask the user to choose via AskUserQuestion with concrete candidates, e.g. from `1.0.0-beta.10`: next prerelease `1.0.0-beta.11`, promote to `1.0.0-rc.1`, promote to stable `1.0.0`. Never guess between these — they have very different meanings (channel promotion vs. iteration) and different CI classification consequences.
-- After a stable `X.Y.Z` exists, the next version must state which component bumps: patch `X.Y.(Z+1)` for fixes only, minor `X.(Y+1).0` for backward-compatible features, major `(X+1).0.0` for breaking changes. If the user names a bump type but not a number, compute it from the latest stable tag and echo the exact resulting version back for confirmation.
-- Echo the final confirmed version string verbatim in your first status report; every later phase must use exactly that string. If at any point the user's wording and the confirmed version diverge, stop and re-confirm.
-
-## Preview tag naming
-
-- Prerelease target (contains `alpha`/`beta`/`rc`): preview tag is `<target>-preview.N`, e.g. `1.0.0-beta.10-preview.3`. It contains `beta`, so `build.yml`'s substring-based classification marks it prerelease — safe.
-- **Stable** target (e.g. `1.1.0`): NEVER tag `1.1.0-preview.N` — `build.yml` marks a tag prerelease only if its name contains `alpha`, `beta`, or `rc`, so `1.1.0-preview.N` would be treated as a stable release and overwrite `latest.json` as stable. Use `1.1.0-rc.N` as the preview tag instead.
-
-## Hard rules
-
-- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
-- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
-- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
-- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
-- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
-- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
-
-## Phase 0 — Preflight
-
-- `git status --short` clean; `git fetch origin main --tags`.
-- `gh auth status` works; confirm you can view `gh release list -L 3`.
-- Confirm the exact final target version with the user if not explicit.
-
-## Phase 1 — Version bump to the final target (once)
-
-- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
-- Otherwise invoke the `rustfs-release-version-bump` skill with the final `<target>` (NOT a preview version), full GitHub flow (commit/push/PR).
-- Get the PR merged into main. Record the resulting main commit:
-
-```bash
-git fetch origin main
-PREVIEW_HASH=$(git rev-parse origin/main)   # must contain the bump PR
-```
-
-`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
-
-## Phase 2 — Publish the preview tag
-
-```bash
-git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
-git push origin "<preview-tag>"
-```
-
-Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
-
-On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
-
-## Phase 3 — CI and artifact verification
-
-- Watch the tag build: `gh run list --workflow build.yml --limit 5` then `gh run watch <run-id>`. Every matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64) plus the release and latest.json jobs.
-- Verify the GitHub release: `gh release view "<preview-tag>" --json isPrerelease,assets`
-  - `isPrerelease` must be `true`.
-  - Assets must include all 6 platform zips in both versioned (`rustfs-<platform>-v<tag>.zip`) and `-latest` forms, plus `SHA256SUMS`, `SHA512SUMS`, `rustfs-<tag>.sbom.cdx.json`, `rustfs-<tag>.provenance.json`.
-- Verify the chained Docker run succeeded: `gh run list --workflow docker.yml --limit 3`.
-- Checksum spot-check for the platform you will run locally: download the zip and `SHA256SUMS`, verify with `shasum -a 256 -c` (grep to one line).
-
-## Phase 4 — Run the artifact locally, verify the console
-
-Work inside the session scratchpad directory; never leave stray data dirs.
-
-```bash
-gh release download "<preview-tag>" -p "rustfs-macos-aarch64-v<preview-tag>.zip" -D "$SCRATCH"
-cd "$SCRATCH" && unzip -o rustfs-*.zip
-./rustfs --version        # must report the PREVIEW TAG (build::TAG), not the Cargo.toml version, plus expected short SHA
-mkdir -p data
-RUSTFS_ACCESS_KEY=rustfsadmin RUSTFS_SECRET_KEY=rustfsadmin ./rustfs ./data
-```
-
-Defaults: S3 endpoint `:9000`, embedded console `:9001`.
-
-Checks (all must pass):
-
-- `./rustfs --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
-- `curl -fsS http://localhost:9000/health/ready` returns ready.
-- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
-- Open `http://localhost:9001` in the browser: login with `rustfsadmin`/`rustfsadmin`; dashboard renders without JS console errors; create a bucket, upload a file, download it back (byte-identical), delete the object and bucket. Keep the server running for Phase 5.
-
-## Phase 5 — Validate with the latest rc client
-
-`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
-
-- Ensure the latest release is installed: compare `rc --version` against `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; update via `brew upgrade rustfs/tap/rc` (or download the release binary).
-- Point it at the preview server and run the command matrix, recording PASS/FAIL per command:
-
-```bash
-rc alias set preview http://localhost:9000 rustfsadmin rustfsadmin
-rc ls preview/
-rc mb preview/rel-check
-rc cp <local-file> preview/rel-check/
-rc stat preview/rel-check/<file>
-rc cat preview/rel-check/<file>          # matches source
-rc cp preview/rel-check/<file> ./out && cmp <local-file> ./out
-rc cp -r <local-dir>/ preview/rel-check/dir/
-rc find preview/rel-check --name "*"
-rc share download preview/rel-check/<file> --expire 1h   # presigned URL fetchable via curl
-rc rm preview/rel-check/<file> && rc rm -r --force preview/rel-check/dir
-rc rb preview/rel-check
-rc admin user list preview/
-rc admin user add preview/ relcheckuser relchecksecret12
-rc admin user remove preview/ relcheckuser
-rc alias remove preview
-```
-
-- Any FAIL blocks the release. Afterwards stop the server and delete the scratch data directory.
-
-## Phase 6 — Publish the final tag on the validated commit
-
-No second version bump, no release branch. The final tag goes on the exact commit the preview validated:
-
-```bash
-git fetch origin --tags
-git rev-parse "<preview-tag>^{commit}"          # must equal PREVIEW_HASH — abort if not
-git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
-git push origin "<target>"
-```
-
-- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
-- Re-run the Phase 3 verification against the final tag: all matrix jobs green; `gh release view "<target>"` shows the full asset set; for a prerelease target `isPrerelease` is `true`, for a stable target it must be `false` and `latest.json` must be updated.
-- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
-
-## Output contract
-
-Always report:
-
-- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
-- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: CI run URLs, release URLs, console check results, the rc command matrix.
-- Any deviation from this pipeline and why the user approved it.

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -18,8 +18,6 @@ Validated baseline: release pattern used in PR `#2957`.
 
 If target version is missing or ambiguous, stop and ask before editing.
 
-Reject any target version containing a `-preview.` suffix: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
-
 ## Read before editing
 
 - `AGENTS.md` (root and nearest path-specific files).

.agents/skills/security-advisory-lessons/SKILL.md

@@ -60,14 +60,12 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 ### IAM and service accounts
 - Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
 - For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims; an action permission alone must not allow choosing root or another user as `target_user`.
-- Treat IAM export packages as credential disclosure surfaces; never include plaintext user or service-account secret keys unless the caller is allowed to recover those secrets and the export format is intentionally sealed.
 - Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
 - Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
 
 ### STS, OIDC, and federation flows
 - Every STS endpoint must have an explicit authentication story: SigV4 where required, OIDC token verification for web identity, and role/session policy validation before issuing credentials.
 - JWT session tokens must be signed and verified by a trusted issuer/key path, not by service-account-controlled material or a reused root secret.
-- JWT verification must enforce required claims and expiration for every bearer token path; "allow missing exp" is never acceptable for user-presented credentials.
 - Public OIDC bootstrap and callback routes must treat `Host`, `X-Forwarded-Proto`, redirect targets, `state`, and callback parameters as untrusted; credential-bearing redirects require a configured, allowlisted origin.
 - OIDC discovery and validation URLs are SSRF sinks. Resolve and classify hostnames at connection time, reject rebinding to loopback/private/link-local ranges, and do not rely on literal string checks.
 
@@ -99,8 +97,6 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 ### Logging and debug output
 - Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials.
 - Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces.
-- Error and panic messages are log content: they propagate through `?` and get printed by `error!`/startup logging far from where they were constructed. Never interpolate a raw config or credential value into an error string.
-- A value that fails secret-format parsing is usually the secret itself (e.g. a bare base64 key missing its `<name>:` prefix), so a parse-failure hint must name the env var or file and the expected format, never echo the input. Redacting `Debug` impls does not cover this channel.
 - Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies.
 
 ### RPC, parsing, and panic safety
@@ -141,10 +137,7 @@ Use these prompts while reviewing a diff:
 - Does a public/default/empty config change security behavior from fail-closed to fail-open?
 - Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
 - Does this response contain stored replication, remote target, or service credentials that need redaction or stricter authorization?
-- Does any error constructor or `format!` interpolate a variable that can hold secret material, including a config parse error that echoes the raw input?
-- Does an IAM export/import path expose or trust plaintext credential secrets beyond the caller's intended authority?
 - Can this STS/OIDC path issue credentials without SigV4, trusted issuer validation, allowlisted redirects, or trusted-proxy host/scheme handling?
-- Can a service-account or STS token omit `exp`, forge `sessionPolicy`, or use a principal-controlled key as signing authority?
 - Does this outbound validation path resolve attacker-supplied hostnames and reject private, loopback, link-local, and rebound addresses at the actual connection boundary?
 - Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
 - Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -27,15 +27,14 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 ### IAM import, service accounts, and privilege boundaries
 
 - `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
-- `GHSA-3495-h8r9-gfqg`: `ExportIAM` wrote regular-user and service-account secret keys into exported ZIP data. Lesson: IAM export is a credential-disclosure boundary; redact, seal, or strictly justify every exported secret before treating export permission as safe.
 - `GHSA-5354-r3w2-34m8`: `AddServiceAccount` checked `CreateServiceAccountAdminAction` but trusted caller-supplied `target_user`, allowing service accounts under the root parent. Lesson: service-account create paths must validate parent ownership or root/admin authority, not only the create action.
 - `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
 - `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
 
 ### STS, OIDC, and federation flows
 
-- `GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
-- `GHSA-ccrv-v8v9-ch9q` and `GHSA-48rf-7j3q-3hfv`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
+- `GHSA-5qfg-mf7r-jp3w`: `AssumeRoleWithWebIdentity` was reachable without the required request authentication and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption.
+- `GHSA-ccrv-v8v9-ch9q`: service-account-controlled material could self-sign JWT session tokens with forged policy claims. Lesson: session tokens must be signed by a trusted issuer/key path and validation must reject self-signed or principal-controlled tokens.
 - `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
 - `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
 
@@ -59,7 +58,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 ### Secrets, defaults, and cryptographic misuse
 
-- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, and `GHSA-63xc-c3w3-m2cf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
+- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, and `GHSA-9gf3-jx4p-4xxf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
 - `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
 - `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
 - `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
@@ -123,7 +122,6 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
 - Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
-- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
 - RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.

.config/make/coverage.mak

@@ -1,26 +0,0 @@
-## —— Coverage --------------------------------------------------------------------------------------
-
-# Local equivalent of the weekly coverage workflow (.github/workflows/coverage.yml,
-# backlog#1153 infra-5): same measurement scope (--workspace --exclude e2e_test,
-# nextest `ci` profile) and the same per-crate table. Slow — the instrumented
-# build cannot reuse your normal target cache and then runs the whole suite.
-# Doctests are not measured (needs nightly). Outputs land in target/llvm-cov/.
-.PHONY: coverage
-coverage: core-deps ## Workspace line coverage (cargo-llvm-cov + nextest; slow, writes target/llvm-cov/)
-	@if ! command -v cargo-llvm-cov >/dev/null 2>&1; then \
-		echo >&2 "❌ cargo-llvm-cov is required for 'make coverage' but was not found."; \
-		echo >&2 "   Install it with:"; \
-		echo >&2 "     cargo install cargo-llvm-cov --locked"; \
-		echo >&2 "     rustup component add llvm-tools-preview"; \
-		exit 1; \
-	fi
-	@if ! command -v cargo-nextest >/dev/null 2>&1; then \
-		echo >&2 "❌ cargo-nextest is required for 'make coverage' (see 'make test')."; \
-		echo >&2 "   Install it with: cargo install cargo-nextest --locked"; \
-		exit 1; \
-	fi
-	NEXTEST_PROFILE=ci cargo llvm-cov nextest --workspace --exclude e2e_test --no-report
-	@mkdir -p target/llvm-cov
-	cargo llvm-cov report --lcov --output-path target/llvm-cov/lcov.info
-	cargo llvm-cov report --json --output-path target/llvm-cov/coverage.json
-	python3 scripts/coverage_per_crate.py target/llvm-cov/coverage.json

.config/make/lint-fmt.mak

@@ -60,11 +60,6 @@ body-cache-whitelist-check: ## Check the body-cache eligibility gate stays a fai
 	@echo "🧱 Checking body-cache whitelist guard..."
 	./scripts/check_body_cache_whitelist.sh
 
-.PHONY: log-analyzer-rules-check
-log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
-	@echo "🩺 Checking log-analyzer rule anchors..."
-	./scripts/check_log_analyzer_rules.sh
-
 .PHONY: compilation-check
 compilation-check: core-deps ## Run compilation check
 	@echo "🔨 Running compilation check..."

.config/make/pre-commit.mak

@@ -23,7 +23,7 @@ pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-gua
 	@echo "✅ All pre-commit checks passed!"
 
 .PHONY: pre-pr
-pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
+pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check clippy-check test ## Run full pre-PR checks with clippy and tests
 	@echo "✅ All pre-PR checks passed!"
 
 .PHONY: dev-check

.config/make/tests.mak

@@ -26,14 +26,6 @@ script-tests: ## Run shell script tests
 	@echo "Running script tests..."
 	./scripts/test_build_rustfs_options.sh
 	./scripts/test_entrypoint_credentials.sh
-	./scripts/test_internode_grpc_ab_bench.sh
-	./scripts/test_object_batch_bench_enhanced.sh
-	./scripts/test_exact_1mib_handoff_abba.sh
-	./scripts/test_pinned_paired_abba_bench.sh
-	./scripts/test_manual_transition_runbooks.sh
-	bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
-	python3 ./scripts/check_object_data_cache_follower_samples.py --self-test
-	./scripts/validate_object_data_cache_cold_stampede.sh --self-test
 
 .PHONY: test
 test: core-deps script-tests ## Run all tests (needs cargo-nextest; RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to override)

.config/nextest.toml

@@ -1,14 +1,17 @@
 # nextest configuration for RustFS.
 #
-# Serialize the ecstore tests that share the process-wide disk registry or
-# exercise a multi-disk commit handoff across nextest process boundaries.
+# Serialize two known load-sensitive / global-state-sharing ecstore test groups
+# so the full parallel nextest suite stops producing spurious failures
+# (backlog #937). These tests pass in isolation but flake under the loaded
+# parallel run for two distinct reasons:
 #
 #   * store::bucket::tests::bucket_delete_* share process/global state (disk
 #     registry, lock client) and race make_bucket into InsufficientWriteQuorum
 #     when run concurrently with other ecstore tests.
 #   * bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
-#     uses the shared multipart fixture and a deterministic uploadId-lock
-#     handoff, so it must not overlap another process mutating that fixture.
+#     asserts a lock-acquire correctness property whose serialized cross-disk
+#     commits exceed the (already max'd, 60s) acquire deadline only when the
+#     suite saturates disk I/O.
 #
 # serial_test's #[serial] attribute does NOT serialize these across runs:
 # nextest executes each test in its own process, where the in-process
@@ -36,26 +39,10 @@ ecstore-serial-flaky = { max-threads = 1 }
 # servers never run at once. ci-7's nightly picks these up via the e2e suite;
 # they are deliberately NOT in the fast PR `e2e-smoke` filter.
 e2e-reliability = { max-threads = 1 }
-e2e-inline-boundaries = { max-threads = 1 }
 
 # --- default profile (local): serialize the flaky groups, never retry --------
 [[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the multipart crash-consistency scenarios (dist-2, backlog#1150):
-# each spawns a 4-disk hermetic erasure set and drives full staged-upload +
-# commit + GET cycles — the same cross-disk-commit IO shape that made
-# concurrent_resend load-sensitive. Preventive serialization only, no retries.
-# The matching ci-profile override is after [profile.ci].
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the durable manual-transition checkpoint test across nextest's
-# process boundary; it mutates bucket lifecycle metadata and is not quarantined.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
+filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(/^store::bucket::tests::bucket_delete_(mark_delete_marks|purge_removes|default_s3_delete)/))'
 test-group = 'ecstore-serial-flaky'
 
 # Serialize the 4-disk reliability / degraded-read e2e tests (see the
@@ -65,10 +52,6 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
 test-group = 'e2e-reliability'
 
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
-test-group = 'e2e-inline-boundaries'
-
 # ---------------------------------------------------------------------------
 # ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
 # ---------------------------------------------------------------------------
@@ -97,17 +80,19 @@ path = "junit.xml"
 # profile's own overrides list, not the default profile's).
 # ===========================================================================
 
-# QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
-# make_bucket into InsufficientWriteQuorum via shared global state under load.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/)'
-test-group = 'ecstore-serial-flaky'
-retries = 2
-
-# Keep the deterministic multipart handoff isolated across nextest processes.
+# QUARANTINE: OPEN backlog#937 — concurrent_resend lock-acquire deadline flakes
+# under saturated disk I/O in the full parallel suite.
 [[profile.ci.overrides]]
 filter = 'package(rustfs-ecstore) & test(concurrent_resend_same_part_commits_one_generation)'
 test-group = 'ecstore-serial-flaky'
+retries = 2
+
+# QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
+# make_bucket into InsufficientWriteQuorum via shared global state under load.
+[[profile.ci.overrides]]
+filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete_marks|purge_removes|default_s3_delete)/)'
+test-group = 'ecstore-serial-flaky'
+retries = 2
 
 # QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
 # on producer/consumer timing windows that stretch past the budget on loaded
@@ -124,19 +109,6 @@ retries = 2
 filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
 test-group = 'e2e-reliability'
 
-# Serialize the multipart crash-consistency scenarios under the ci profile too
-# (see the matching default-profile override near the top). Not a quarantine:
-# no retries, just serialized 4-disk cross-disk-commit IO.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the durable manual-transition checkpoint test under the ci profile
-# too. No retries: failures stay visible.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
-test-group = 'ecstore-serial-flaky'
-
 # ---------------------------------------------------------------------------
 # e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
 # ---------------------------------------------------------------------------
@@ -150,10 +122,6 @@ test-group = 'ecstore-serial-flaky'
 # Each e2e test spawns its own rustfs server on a random port with an isolated
 # temp dir (crates/e2e_test/src/common.rs), so the subset is parallel-safe.
 #
-# Replication failure harness (backlog#1147 repl-8): the first clause admits
-# its four in-process fake-target self-tests. They bind random loopback ports,
-# use no external service, and finish in under a second.
-#
 # Replication PR subset (backlog#1147 repl-1): the second clause admits the 20
 # FAST bucket-replication tests from replication_extension_test — the
 # target-registration / replication-check / list / remove / delete admin paths
@@ -168,7 +136,7 @@ test-group = 'ecstore-serial-flaky'
 # the nightly profile derives its set as "the replication module MINUS this
 # allowlist", so any new replication test lands in nightly by default (never
 # silently unrun) until it is explicitly blessed as fast here. Keep the two
-# regexes byte-identical. Count invariant: 20 here + 28 nightly = 48 total
+# regexes byte-identical. Count invariant: 20 here + 18 nightly = 38 total
 # (authority: `cargo nextest list`; docs/testing/e2e-suite-inventory.md).
 # HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
 # (#4724) because they set a loopback (127.0.0.1) replication target that the
@@ -176,38 +144,12 @@ test-group = 'ecstore-serial-flaky'
 # the guard now honours an off-by-default opt-in and this suite's source servers
 # set it (RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) — so the allowlist below is
 # restored.
-#
-# Security negative-auth subset (backlog#1151 sec-5): the three attacker-facing
-# S3 auth-rejection suites join the first clause above by module name —
-# presigned_negative (sec-2), negative_sigv4 (sec-1, header SigV4), and
-# admin_auth (sec-4, admin gate + root-credential lifecycle). All three use
-# RustFSTestEnvironment on a random port and are parallel-safe, so they meet the
-# smoke admission criteria unchanged. This is the wiring step that makes those
-# merged suites actually execute on every PR (they were dead until listed here).
-# A rename that drops any of them out of this filter would silently thin the
-# security gate with no CI signal, so scripts/check_security_smoke_count.sh owns
-# a count-floor guard over exactly this subset (infra-12 mechanism, floor in
-# .config/security-smoke-floor.txt), invoked from the e2e-tests job in ci.yml.
-# NOT here by topology: the GHSA-3p3x FTPS/WebDAV constant-time e2e
-# (protocols::test_protocol_core_suite) binds fixed ports and needs the
-# ftps,webdav features, so it cannot join this random-port, default-feature
-# profile; its GHSA-r5qv sibling is a unit test that already runs in the
-# test-and-lint `--all --exclude e2e_test` pass. See
-# docs/testing/security-regressions.md for the full CI-execution map.
-#
-# ILM tiering main path (backlog#1148 ilm-7): the `reliant::tiering::` clause
-# admits the hermetic transition e2e. Like the fast replication pair checks it
-# spawns a second independent single-node server (the cold RustFS tier), not a
-# cluster, so it keeps the lane's parallel-safe / no-external-dependency
-# properties. The RustFS warm backend has no loopback guard (that guard is
-# replication-only), so it needs no opt-in env for its 127.0.0.1 tier target.
 [profile.e2e-smoke]
 default-filter = """
     package(e2e_test) & (
-        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
+        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_source_invalid_date|content_encoding|anonymous_access|bucket_policy_check|presigned_negative)_test::/)
         | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
         | test(/^reliant::lifecycle::/)
-        | test(/^reliant::tiering::/)
     )
 """
 fail-fast = false
@@ -218,14 +160,10 @@ fail-fast = false
 # backlog#1147 repl-1 (deps: ci-4). Runs the SLOW / cross-process replication
 # tests that are unfit for the per-PR e2e-smoke gate:
 #
-#   * 2 remote-target TLS validation tests.
-#   * 12 bucket-replication data-plane/helper tests — they PUT/delete objects
-#     and poll until source and target converge; two replicate over HTTPS, two
-#     pin active SSE failure contracts, and one guards event/history observers.
-#     The SSE-S3 contract remains ignored under backlog#1291.
-#   * 12 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
+#   * 8 bucket-replication data-plane tests — they PUT/delete objects and poll
+#     until source and target converge; two replicate over HTTPS.
+#   * 9 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
 #     servers and drives the cross-process site-replication control plane.
-#   * 1 `_real_three_node` site-replication test.
 #   * 1 `_real_single_node` service-account round-trip test.
 #
 # The set is defined as "everything in replication_extension_test that is NOT
@@ -261,72 +199,3 @@ fail-fast = false
 # Emitted to target/nextest/e2e-repl-nightly/junit.xml; uploaded by the nightly
 # workflow as the failure-triage artifact.
 path = "junit.xml"
-
-# ---------------------------------------------------------------------------
-# e2e-full profile — merge-gate full single-node e2e lane (backlog#1149 ci-5)
-# ---------------------------------------------------------------------------
-# The merge gate (ci.yml `e2e-full` job: push main + merge_group +
-# workflow_dispatch). Runs the never-automated user-visible suites — KMS (40),
-# object_lock (33), multipart_auth (109), quota, checksum, encryption,
-# security-boundary, ... — that the fast PR `e2e-smoke` subset deliberately
-# skips. Budget <= 45 min; authority for the suite count is `cargo nextest list
-# --profile e2e-full` (see docs/testing/e2e-suite-inventory.md).
-#
-# The filter is "the whole e2e_test crate MINUS the sets owned by other lanes":
-#   * protocols:: — FTPS/SFTP/WebDAV, still pinned to --test-threads=1 by fixed
-#     ports; they join a scheduled lane once ci-6 randomises the ports (ci-7).
-#   * the 7 cluster suites that spin up a RustFSTestClusterEnvironment
-#     (cluster_concurrency, cluster_multidrive_pool, stale_multipart_cleanup_cluster,
-#     namespace_lock_quorum, heal_erasure_disk_rebuild, admin_timeout_regression,
-#     object_lambda) — too heavy for the merge budget; they run in ci-7's
-#     nightly 4-node lane.
-#   * replication_extension_test — repl-1 already splits it into the PR
-#     `e2e-smoke` (20 fast) and `e2e-repl-nightly` (27 slow) lanes and reserves
-#     it for those, so e2e-full does not double-run it.
-#   * #[ignore]d tests — nextest skips them by default (no --run-ignored); the
-#     manual-localhost:9000 reliant/policy tests are ci-13's migration.
-#
-# Each e2e test spawns its own single-node rustfs server on a random port with
-# an isolated temp dir (crates/e2e_test/src/common.rs), so the set is
-# parallel-safe — the same property e2e-smoke relies on. The exception is the
-# 4-disk reliability / degraded-read fault-injection tests, serialized below
-# (identical to the ci profile) so several 4-disk servers never run at once.
-# KNOWN-FAILURE EXCLUSIONS (characterization run 29381309848, 2026-07-15:
-# 341 ran / 32 failed on the suites' first automated run ever). Deterministic
-# product failures cannot be quarantined away with retries, so each family is
-# excluded here with its tracking issue, under the same discipline as the
-# ci-profile quarantine (docs/testing/README.md): every entry MUST cite one
-# OPEN issue, and the fixing PR MUST delete the exclusion. The passing
-# negative-path siblings of each family stay in as regression guards.
-#   * rustfs#4843 — over-limit archive entry paths hard-reject the whole
-#     archive even under ignore-errors semantics.
-#   * rustfs#4846 — distributed-lock quorum tests misclassify as timeout
-#     under parallel load (multi-node in-process clusters; natural home is
-#     ci-7's nightly cluster lane).
-[profile.e2e-full]
-default-filter = """
-    package(e2e_test)
-    & !test(/^protocols::/)
-    & !test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
-    & !test(/^replication_extension_test::/)
-    & !test(/^multipart_auth_test::test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_enabled$/)
-    & !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(ignores_invalid_entries_when_requested|supports_standard_headers_with_combined_extract_options)$/)
-    & !test(/^reliant::lock::test_distributed_lock_(2_nodes_grpc_read_survives_failed_node|4_nodes_grpc_read_write_quorum_split_with_two_failed_nodes)$/)
-"""
-fail-fast = false
-
-[profile.e2e-full.junit]
-# Emitted to target/nextest/e2e-full/junit.xml; uploaded by the e2e-full job.
-path = "junit.xml"
-
-# Serialize the 4-disk reliability / degraded-read e2e tests under e2e-full too
-# (see the e2e-reliability test-group note near the top of this file). Not a
-# quarantine: no retries, just single-threaded so several 4-disk servers never
-# run concurrently.
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
-test-group = 'e2e-reliability'
-
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
-test-group = 'e2e-inline-boundaries'

.config/security-smoke-floor.txt

@@ -1,12 +0,0 @@
-# Committed floor for the number of security negative-auth tests selected by the
-# e2e-smoke PR profile (see scripts/check_security_smoke_count.sh, backlog#1151
-# sec-5).
-#
-# The floor equals the exact count of e2e_test cases whose name starts with a
-# security module prefix (negative_sigv4_test, presigned_negative_test,
-# admin_auth_test) that the [profile.e2e-smoke] default-filter in
-# .config/nextest.toml selects, at the time this file was last updated. CI fails
-# if the selected count drops below this number, so a rename or removal that
-# thins the security smoke gate must update this file in the same PR.
-# Adding tests does not require a bump, but bumping keeps the guard tight.
-16

.docker/observability/grafana/dashboards/rustfs.json

@@ -1744,7 +1744,7 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_objects_count{job=~\"$job\", bucket=~\"$bucket\"})",
+          "expr": "sum by (bucket) (rustfs_bucket_api_objects_total{job=~\"$job\", bucket=~\"$bucket\"})",
           "legendFormat": "{{bucket}}",
           "range": true,
           "refId": "A"
@@ -1844,7 +1844,7 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_total_bytes{job=~\"$job\", bucket=~\"$bucket\"})",
+          "expr": "sum by (bucket) (rustfs_bucket_api_usage_bytes{job=~\"$job\", bucket=~\"$bucket\"})",
           "legendFormat": "{{bucket}}",
           "range": true,
           "refId": "A"
@@ -11583,7 +11583,7 @@
           "text": "All",
           "value": "$__all"
         },
-        "definition": "label_values(rustfs_cluster_usage_buckets_objects_count,bucket)",
+        "definition": "label_values(rustfs_bucket_api_objects_total,bucket)",
         "includeAll": true,
         "label": "Bucket",
         "multi": true,
@@ -11591,7 +11591,7 @@
         "options": [],
         "query": {
           "qryType": 1,
-          "query": "label_values(rustfs_cluster_usage_buckets_objects_count,bucket)",
+          "query": "label_values(rustfs_bucket_api_objects_total,bucket)",
           "refId": "PrometheusVariableQueryEditor-VariableQuery"
         },
         "refresh": 2,

.docker/test/site-replication/run-object-flow-check.sh

@@ -18,7 +18,6 @@ set -eu
 ACCESS_KEY="${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}"
 SECRET_KEY="${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}"
 BUCKET="${RUSTFS_SITE_REPL_FLOW_BUCKET:-site-repl-flow-check}"
-DELETE_BUCKET="${RUSTFS_SITE_REPL_DELETE_BUCKET:-site-repl-delete-$(date +%Y%m%d-%H%M%S)-$$}"
 PREFIX="${RUSTFS_SITE_REPL_FLOW_PREFIX:-flow-$(date +%Y%m%d-%H%M%S)}"
 WAIT_ATTEMPTS="${RUSTFS_SITE_REPL_WAIT_ATTEMPTS:-90}"
 WAIT_SLEEP_SECONDS="${RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS:-2}"
@@ -86,39 +85,17 @@ wait_for_object() {
 
 wait_for_bucket() {
   site="$1"
-  bucket="${2:-$BUCKET}"
   attempt=1
 
   while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
-    if mc stat "$site/$bucket" >/dev/null 2>&1; then
+    if mc stat "$site/$BUCKET" >/dev/null 2>&1; then
       return 0
     fi
     sleep "$WAIT_SLEEP_SECONDS"
     attempt=$((attempt + 1))
   done
 
-  echo "bucket was not replicated in time: $site/$bucket" >&2
-  return 1
-}
-
-wait_for_bucket_delete() {
-  site="$1"
-  bucket="$2"
-  attempt=1
-
-  while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
-    if result="$(mc stat --json "$site/$bucket" 2>&1)"; then
-      :
-    else
-      case "$result" in
-        *NoSuchBucket*) return 0 ;;
-      esac
-    fi
-    sleep "$WAIT_SLEEP_SECONDS"
-    attempt=$((attempt + 1))
-  done
-
-  echo "bucket deletion was not replicated in time: $site/$bucket" >&2
+  echo "bucket was not replicated in time: $site/$BUCKET" >&2
   return 1
 }
 
@@ -209,20 +186,6 @@ EOF
   echo "verified replicated downloads for $object_name"
 done
 
-echo "creating empty bucket for replicated delete check: $DELETE_BUCKET"
-mc mb "site1/$DELETE_BUCKET" >/dev/null
-
-for site in site1 site2 site3; do
-  wait_for_bucket "$site" "$DELETE_BUCKET"
-done
-
-echo "deleting empty bucket on site1: $DELETE_BUCKET"
-mc rb "site1/$DELETE_BUCKET" >/dev/null
-
-for site in site1 site2 site3; do
-  wait_for_bucket_delete "$site" "$DELETE_BUCKET"
-done
-
 echo "site replication object flow check passed"
 echo "bucket: $BUCKET"
 echo "prefix: $PREFIX"

.github/actions/setup/action.yml

@@ -56,7 +56,6 @@ runs:
           libssl-dev \
           ripgrep \
           unzip \
-          zip \
           protobuf-compiler
 
     - name: Install protoc

.github/cla.yml

@@ -15,8 +15,8 @@
 enabled: true
 
 document:
-  version: v2
-  url: https://github.com/rustfs/cla/blob/main/cla/v2.md
+  version: v1
+  url: https://github.com/rustfs/cla/blob/main/cla/v1.md
 
 signing:
   mode: comment

.github/pull_request_template.md

@@ -33,4 +33,4 @@ documentation impact. Use N/A when there is no expected impact.
 
 ---
 
-Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v2.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.
+Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v1.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.

.github/workflows/build.yml

@@ -136,13 +136,11 @@ jobs:
             echo "⚡ Manual/scheduled build detected"
           fi
 
-          {
-            echo "should_build=$should_build"
-            echo "build_type=$build_type"
-            echo "version=$version"
-            echo "short_sha=$short_sha"
-            echo "is_prerelease=$is_prerelease"
-          } >> "$GITHUB_OUTPUT"
+          echo "should_build=$should_build" >> $GITHUB_OUTPUT
+          echo "build_type=$build_type" >> $GITHUB_OUTPUT
+          echo "version=$version" >> $GITHUB_OUTPUT
+          echo "short_sha=$short_sha" >> $GITHUB_OUTPUT
+          echo "is_prerelease=$is_prerelease" >> $GITHUB_OUTPUT
 
           echo "📊 Build Summary:"
           echo "  - Should build: $should_build"
@@ -190,6 +188,7 @@ jobs:
             {"target_id":"linux-x86_64-gnu","os":"sm-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
             {"target_id":"linux-aarch64-gnu","os":"sm-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
             {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"macos-x86_64","os":"macos-26-intel","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
             {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
           ]}'
 
@@ -221,8 +220,8 @@ jobs:
     name: Build RustFS
     needs: [ build-check, prepare-platform-matrix ]
     if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
-    runs-on: ${{ matrix.os }}
-    timeout-minutes: 150
+    runs-on: ${{ matrix.platform == 'linux' && fromJSON('["self-hosted","linux","sm-standard-2"]') || matrix.os }}
+    timeout-minutes: 90
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
       # Release binaries ship without dial9 telemetry and therefore do not need
@@ -289,7 +288,6 @@ jobs:
             local console_url
             local console_sha256
             local curl_auth_args=()
-            console_tag=""
 
             if [[ "${{ matrix.platform }}" == "windows" ]]; then
               curl_bin="curl.exe"
@@ -312,7 +310,7 @@ jobs:
             "$curl_bin" "${curl_auth_args[@]}" --fail -L "$console_api" \
               -o "$console_json" --retry 3 --retry-delay 5 --max-time 300 || return 1
 
-            read -r console_tag console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
+            read -r console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
           import json
           import re
           import sys
@@ -320,8 +318,6 @@ jobs:
           with open(sys.argv[1], encoding="utf-8") as handle:
               release = json.load(handle)
 
-          tag = release.get("tag_name", "") or "unknown"
-
           for asset in release.get("assets", []):
               name = asset.get("name", "")
               digest = asset.get("digest", "")
@@ -330,7 +326,7 @@ jobs:
                   sha256 = digest.split(":", 1)[1]
                   if not re.fullmatch(r"[0-9a-fA-F]{64}", sha256):
                       raise SystemExit(f"console zip asset has invalid sha256 digest: {sha256}")
-                  sys.stdout.buffer.write(f"{tag} {url} {sha256}\n".encode("utf-8"))
+                  sys.stdout.buffer.write(f"{url} {sha256}\n".encode("utf-8"))
                   break
           else:
               raise SystemExit("no console zip asset with sha256 digest found")
@@ -342,8 +338,6 @@ jobs:
               return 1
             fi
 
-            echo "Console release: ${console_tag}"
-            echo "Downloading console asset: ${console_url}"
             "$curl_bin" --fail -L "$console_url" -o console.zip --retry 3 --retry-delay 5 --max-time 300 || return 1
             verify_sha256 "$console_sha256" console.zip || return 2
             unzip -o console.zip -d ./rustfs/static || return 2
@@ -356,19 +350,12 @@ jobs:
             rm -f console.zip console-release.json
             if [[ "$status" -eq 2 ]]; then
               echo "Console asset integrity verification failed" >&2
+              exit 1
             fi
-            echo "Failed to download verified console assets" >&2
-            exit 1
+            echo "Warning: Failed to download verified console assets, continuing without them"
+            echo "// Static assets not available" > ./rustfs/static/empty.txt
           fi
 
-          if [[ ! -s ./rustfs/static/index.html ]]; then
-            echo "Console asset archive is missing static/index.html" >&2
-            exit 1
-          fi
-
-          asset_count=$(find ./rustfs/static -type f | wc -l | tr -d '[:space:]')
-          echo "Console assets ready: version=${console_tag:-unknown}, ${asset_count} files extracted to ./rustfs/static"
-
       - name: Build RustFS
         shell: bash
         run: |
@@ -450,11 +437,11 @@ jobs:
             # Release/Prerelease build: rustfs-${platform}-${arch}-${variant}-v${version}.zip
             PACKAGE_NAME="${PACKAGE_BASENAME}-v${PACKAGE_VERSION}"
           fi
-          
+
           # Create zip packages for all platforms
           # Ensure zip is available
           if ! command -v zip &> /dev/null; then
-            if [[ "${{ matrix.os }}" == "ubuntu-latest" || "${{ matrix.platform }}" == "linux" ]]; then
+            if [[ "${{ matrix.os }}" == "ubuntu-latest" ]]; then
               sudo apt-get update && sudo apt-get install -y zip
             fi
           fi
@@ -506,7 +493,7 @@ jobs:
             if [[ "${{ matrix.platform }}" == "windows" ]]; then
               dir
             else
-              ls -lh "${PACKAGE_NAME}.zip"
+              ls -lh ${PACKAGE_NAME}.zip
             fi
           else
             echo "❌ Failed to create package: ${PACKAGE_NAME}.zip"
@@ -555,13 +542,11 @@ jobs:
             fi
           fi
 
-          {
-            echo "package_name=${PACKAGE_NAME}"
-            echo "package_file=${PACKAGE_NAME}.zip"
-            echo "latest_files=${LATEST_FILES}"
-            echo "build_type=${BUILD_TYPE}"
-            echo "version=${VERSION}"
-          } >> "$GITHUB_OUTPUT"
+          echo "package_name=${PACKAGE_NAME}" >> $GITHUB_OUTPUT
+          echo "package_file=${PACKAGE_NAME}.zip" >> $GITHUB_OUTPUT
+          echo "latest_files=${LATEST_FILES}" >> $GITHUB_OUTPUT
+          echo "build_type=${BUILD_TYPE}" >> $GITHUB_OUTPUT
+          echo "version=${VERSION}" >> $GITHUB_OUTPUT
 
           echo "📦 Package created: ${PACKAGE_NAME}.zip"
           if [[ -n "$LATEST_FILES" ]]; then
@@ -570,60 +555,6 @@ jobs:
           echo "🔧 Build type: ${BUILD_TYPE}"
           echo "📊 Version: ${VERSION}"
 
-      - name: Verify packaged console
-        if: matrix.target == 'x86_64-unknown-linux-gnu'
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          find_free_port() {
-            python3 - <<'PY'
-          import socket
-
-          with socket.socket() as sock:
-              sock.bind(("127.0.0.1", 0))
-              print(sock.getsockname()[1])
-          PY
-          }
-
-          package_dir="$(mktemp -d)"
-          data_dir="$(mktemp -d)"
-          log_file="${RUNNER_TEMP}/rustfs-console-smoke.log"
-          server_pid=""
-          trap '
-            if [[ -n "${server_pid:-}" ]]; then
-              kill "$server_pid" 2>/dev/null || true
-              wait "$server_pid" 2>/dev/null || true
-            fi
-            rm -rf "$package_dir" "$data_dir"
-          ' EXIT
-
-          unzip -q "${{ steps.package.outputs.package_file }}" -d "$package_dir"
-          api_port="$(find_free_port)"
-          console_port="$(find_free_port)"
-          console_url="http://127.0.0.1:${console_port}/rustfs/console/"
-
-          "$package_dir/rustfs" server \
-            --address "127.0.0.1:${api_port}" \
-            --console-enable \
-            --console-address "127.0.0.1:${console_port}" \
-            --access-key console-smoke \
-            --secret-key console-smoke-secret \
-            "$data_dir" >"$log_file" 2>&1 &
-          server_pid=$!
-
-          for _ in {1..40}; do
-            response="$(curl --silent --output /dev/null --write-out '%{http_code} %{content_type}' "$console_url" || true)"
-            if [[ "$response" == 200\ text/html* ]]; then
-              exit 0
-            fi
-            sleep 0.25
-          done
-
-          echo "Console endpoint did not return 200 text/html: $response" >&2
-          cat "$log_file"
-          exit 1
-
       - name: Upload to GitHub artifacts
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
         with:
@@ -648,19 +579,9 @@ jobs:
             exit 0
           fi
 
-          # The self-hosted Linux runners do not ship the aws CLI (GitHub-hosted
-          # images did). Install it on demand so R2 uploads survive a fresh runner
-          # instead of hard-failing here.
           if ! command -v aws >/dev/null 2>&1; then
-            echo "aws CLI not found on runner; installing..."
-            if command -v apt-get >/dev/null 2>&1; then
-              sudo apt-get update && sudo apt-get install -y awscli
-            elif command -v brew >/dev/null 2>&1; then
-              brew install awscli
-            else
-              echo "❌ aws CLI missing and no apt-get/brew to install it; cannot upload to R2"
-              exit 1
-            fi
+            echo "❌ aws CLI not found on runner; cannot upload to R2"
+            exit 1
           fi
 
           export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
@@ -822,8 +743,8 @@ jobs:
             RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
           fi
 
-          echo "release_id=$RELEASE_ID" >> "$GITHUB_OUTPUT"
-          echo "release_url=$RELEASE_URL" >> "$GITHUB_OUTPUT"
+          echo "release_id=$RELEASE_ID" >> $GITHUB_OUTPUT
+          echo "release_url=$RELEASE_URL" >> $GITHUB_OUTPUT
           echo "Created release: $RELEASE_URL"
 
   # Prepare and upload release assets
@@ -872,9 +793,9 @@ jobs:
           cd ./release-assets
 
           # Generate checksums for all files (including latest versions)
-          if compgen -G "*.zip" >/dev/null; then
-            sha256sum -- *.zip > SHA256SUMS
-            sha512sum -- *.zip > SHA512SUMS
+          if ls *.zip >/dev/null 2>&1; then
+            sha256sum *.zip > SHA256SUMS
+            sha512sum *.zip > SHA512SUMS
           fi
 
           cd ..
@@ -914,14 +835,12 @@ jobs:
 
           echo "✅ All assets uploaded successfully"
 
-  # Update latest.json for every release tag (stable and prerelease): the
-  # project currently ships prerelease tags only, so gating this to stable
-  # left the version pointer permanently stale. release_type records whether
-  # the pointed-to version is a prerelease.
+  # Update latest.json for stable releases only: prerelease tags (alpha/beta/
+  # rc) must never overwrite the stable version pointer.
   update-latest-version:
     name: Update Latest Version
     needs: [ build-check, upload-release-assets ]
-    if: startsWith(github.ref, 'refs/tags/')
+    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type == 'release'
     runs-on: ubuntu-latest
     steps:
       - name: Update latest.json
@@ -940,12 +859,6 @@ jobs:
           VERSION="${{ needs.build-check.outputs.version }}"
           TAG="${{ needs.build-check.outputs.version }}"
 
-          if [[ "${{ needs.build-check.outputs.build_type }}" == "prerelease" ]]; then
-            RELEASE_TYPE="prerelease"
-          else
-            RELEASE_TYPE="stable"
-          fi
-
           # Install ossutil
           OSSUTIL_VERSION="2.1.1"
           OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-amd64.zip"
@@ -966,7 +879,7 @@ jobs:
             "version": "${VERSION}",
             "tag": "${TAG}",
             "release_date": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
-            "release_type": "${RELEASE_TYPE}",
+            "release_type": "stable",
             "download_url": "https://github.com/${{ github.repository }}/releases/tag/${TAG}"
           }
           EOF
@@ -974,7 +887,7 @@ jobs:
           # Upload to OSS
           "$OSSUTIL_BIN" cp latest.json oss://rustfs-version/latest.json --force
 
-          echo "✅ Updated latest.json for ${RELEASE_TYPE} release $VERSION"
+          echo "✅ Updated latest.json for stable release $VERSION"
 
   # Publish release (remove draft status)
   publish-release:

.github/workflows/ci.yml

@@ -141,7 +141,7 @@ jobs:
     name: Test and Lint
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
     runs-on: sm-standard-4
-    timeout-minutes: 90
+    timeout-minutes: 60
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
@@ -156,76 +156,25 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      - name: Prepare test evidence
-        run: |
-          mkdir -p artifacts/test-and-lint
-          {
-            echo "run_id=${GITHUB_RUN_ID}"
-            echo "job=${GITHUB_JOB}"
-            echo "runner=${RUNNER_NAME}"
-            echo "started_at=$(date --utc --iso-8601=seconds)"
-          } > artifacts/test-and-lint/run-metadata.txt
-
       # Clippy runs before the test pass: lint failures are the most common
       # CI-only breakage and should surface in minutes, not after 20+ minutes
       # of tests.
       - name: Run clippy lints
         run: cargo clippy --all-targets -- -D warnings
 
-      - name: Run nextest tests
+      - name: Run tests
         run: |
-          mkdir -p artifacts/test-and-lint
-          set +e
-          NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 75m \
-            cargo nextest run --profile ci --all --exclude e2e_test \
-            --status-level all --final-status-level all \
-            2>&1 | tee artifacts/test-and-lint/nextest.log
-          status=${PIPESTATUS[0]}
-          {
-            echo "command=cargo nextest run --profile ci --all --exclude e2e_test"
-            echo "exit_status=${status}"
-            echo "finished_at=$(date --utc --iso-8601=seconds)"
-            echo
-            echo "Remaining test-related processes:"
-            pgrep -af 'cargo|nextest|target/.*/deps/' || true
-          } > artifacts/test-and-lint/nextest-diagnostics.txt
-          exit "${status}"
+          cargo nextest run --profile ci --all --exclude e2e_test
+          cargo test --all --doc
 
-      - name: Run documentation tests
-        run: |
-          mkdir -p artifacts/test-and-lint
-          set +e
-          timeout --verbose --signal=TERM --kill-after=30s 15m \
-            cargo test --all --doc \
-            2>&1 | tee artifacts/test-and-lint/doctest.log
-          status=${PIPESTATUS[0]}
-          {
-            echo "command=cargo test --all --doc"
-            echo "exit_status=${status}"
-            echo "finished_at=$(date --utc --iso-8601=seconds)"
-            echo
-            echo "Remaining test-related processes:"
-            pgrep -af 'cargo|rustdoc|target/.*/deps/' || true
-          } > artifacts/test-and-lint/doctest-diagnostics.txt
-          exit "${status}"
-
-      - name: Upload test reports and diagnostics
+      - name: Upload test junit report
         if: always()
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
         with:
           name: junit-test-and-lint-${{ github.run_number }}
-          path: |
-            target/nextest/ci/junit.xml
-            artifacts/test-and-lint
+          path: target/nextest/ci/junit.xml
           retention-days: 3
-          if-no-files-found: error
-
-      # rustfs/backlog#1289: fail if a seed rule's log anchor no longer exists
-      # verbatim in the source tree (log message drifted without updating the
-      # rule). Placed here where the workspace — including the la-dump-anchors
-      # bin — is already built by the clippy/test steps above.
-      - name: Check log-analyzer rule anchors
-        run: ./scripts/check_log_analyzer_rules.sh
+          if-no-files-found: ignore
 
       # Explicit gate for migration-critical suites. These tests already ran in
       # the full nextest pass above; a single filtered nextest invocation keeps
@@ -272,23 +221,15 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
-      # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
-      # disk_index 0), not an EC metadata-distribution issue.
-      # restore_object_usecase_reports_ongoing_conflict_and_completion was
-      # re-enabled by backlog#1304 (restore accepts serialize on a short CAS
-      # guard; the copy-back no longer holds the #4877 whole-copy-back lock,
-      # so the mid-restore ongoing read and fast 409 rejection it asserts are
-      # the implemented contract). The remaining exclusions each hit a
-      # DIFFERENT, independent issue (all tracked under rustfs/backlog#1148;
-      # they keep #[ignore] with a backlog reference):
-      #   - test_noncurrent_{expiry,transition}_still_works_after_immediate_compensation_transition:
-      #     noncurrent transition/expiry after an immediate compensation transition.
+      # Three scanner tests fail on main independently of this lane (restore of
+      # a transitioned multipart object, noncurrent transition/expiry after an
+      # immediate compensation transition); they keep #[ignore] with a backlog
+      # reference and are excluded here by name until fixed (rustfs/backlog#1148).
       - name: Run ignored ILM integration tests serially
         run: |
           cargo nextest run -j1 --run-ignored ignored-only \
             -p rustfs-scanner -p rustfs \
-            -E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition))'
+            -E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_transition_and_restore_flows) or test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition))'
 
   test-and-lint-rio-v2:
     name: Test and Lint (rio-v2)
@@ -372,7 +313,7 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Build debug binary
-        run: cargo build -p rustfs --bins --features e2e-test-hooks
+        run: cargo build -p rustfs --bins
 
       - name: Upload debug binary
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
@@ -402,7 +343,7 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Build debug binary with rio-v2
-        run: cargo build -p rustfs --bins --features rio-v2,e2e-test-hooks
+        run: cargo build -p rustfs --bins --features rio-v2
 
       - name: Upload debug binary
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
@@ -494,16 +435,6 @@ jobs:
       - name: Make binary executable
         run: chmod +x ./target/debug/rustfs
 
-      # Guard the security negative-auth smoke subset (backlog#1151 sec-5)
-      # against a rename or deletion silently dropping it out of the e2e-smoke
-      # filter. The script lists what the profile selects and fails if the count
-      # of security auth-rejection tests falls below the committed floor in
-      # .config/security-smoke-floor.txt (infra-12 count-floor mechanism). Run
-      # before the smoke suite so a thinned gate fails fast; the `nextest list`
-      # here compiles the e2e_test binaries the run below reuses.
-      - name: Check security smoke subset count floor
-        run: ./scripts/check_security_smoke_count.sh check
-
       # PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
       # profile.e2e-smoke default-filter in .config/nextest.toml is the single
       # wiring mechanism for e2e tests in CI — extend that filter instead of
@@ -537,59 +468,6 @@ jobs:
           path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
           retention-days: 3
 
-  e2e-full:
-    name: End-to-End Tests (full merge gate)
-    # Merge gate only (backlog#1149 ci-5): the never-automated user-visible
-    # suites — KMS, object_lock, multipart_auth, quota, checksum, encryption,
-    # security-boundary, ... — via the e2e-full nextest profile. Too heavy for
-    # every PR, so it is gated to main pushes, the merge queue, and manual
-    # dispatch. protocols / the 6 cluster suites / replication / #[ignore] are
-    # owned by other lanes (see .config/nextest.toml profile.e2e-full).
-    if: >-
-      github.event_name == 'workflow_dispatch' ||
-      github.event_name == 'merge_group' ||
-      (github.event_name == 'push' && github.ref == 'refs/heads/main')
-    needs: [ build-rustfs-debug-binary ]
-    runs-on: sm-standard-2
-    timeout-minutes: 55
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-e2e
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-
-      # Download after the cache restore so the freshly built binary from the
-      # build job always wins over anything restored into target/debug.
-      - name: Download debug binary
-        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
-        with:
-          name: rustfs-debug-binary
-          path: target/debug
-
-      - name: Make binary executable
-        run: chmod +x ./target/debug/rustfs
-
-      # Full single-node e2e lane (backlog#1149 ci-5). The e2e-full
-      # default-filter in .config/nextest.toml is the single wiring mechanism —
-      # extend that filter, never add ad-hoc e2e jobs here. Reuses the downloaded
-      # debug binary; each test spawns its own rustfs server on a random port.
-      - name: Run e2e full suite
-        run: cargo nextest run --profile e2e-full -p e2e_test
-
-      - name: Upload junit
-        if: always()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: e2e-full-junit-${{ github.run_number }}
-          path: target/nextest/e2e-full/junit.xml
-          retention-days: 7
-
   e2e-tests-rio-v2:
     name: End-to-End Tests (rio-v2)
     needs: [ build-rustfs-debug-binary-rio-v2 ]
@@ -616,14 +494,6 @@ jobs:
       - name: Setup Rust toolchain for s3s-e2e installation
         uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
 
-      # The sm-standard-* custom runner images (introduced in #4884) ship no C
-      # toolchain, unlike GitHub-hosted ubuntu-latest. Installing s3s-e2e below
-      # compiles it from source on a cache miss, and build scripts need cc.
-      - name: Install build tools for s3s-e2e compilation
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential cmake pkg-config libssl-dev
-
       - name: Install s3s-e2e test tool
         uses: taiki-e/cache-cargo-install-action@7447f04c51f2ba27ca35e7f1e28fab848c5b3ba7 # v2
         with:

.github/workflows/coverage.yml

@@ -1,122 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Weekly workspace line-coverage baseline (backlog#1153 infra-5).
-#
-# NON-BLOCKING by design: this workflow only runs on schedule and manual
-# dispatch, so it never attaches a status to a PR and must never be made a
-# required check. It exists to give coverage a visible baseline and trend
-# (per-crate table in the job summary, lcov artifact kept 90 days) — the
-# per-crate ratchet for the security-critical crates builds on it later
-# (backlog#1153 infra-6, report-only first per the ci-11 ladder).
-#
-# Measurement scope matches the PR test gate (ci.yml "Run tests"):
-# `--workspace --exclude e2e_test` with the `ci` nextest profile. Doctests are
-# NOT measured (ci.yml runs them uninstrumented; `cargo llvm-cov` needs a
-# nightly toolchain to cover doctests). Trend-comparison workflow:
-# docs/testing/README.md "Coverage" section. `make coverage` is the local
-# equivalent. Scheduled failures alert via the ci-8 composite action.
-
-name: coverage
-
-on:
-  workflow_dispatch:
-  schedule:
-    # 07:00 UTC Sunday — staggered clear of the other Sunday crons: ci (00:00),
-    # build (01:00), e2e-s3tests (02:00), audit (03:00), nix-flake-update
-    # (05:00), mint (06:00), and the daily fuzz (02:00), minio-interop (03:17),
-    # e2e-replication-nightly (04:00) and performance-ab (06:00) lanes.
-    - cron: "0 7 * * 0"
-
-# Only alert-on-failure needs more than read access; it declares its own
-# job-level `issues: write`.
-permissions:
-  contents: read
-
-jobs:
-  coverage:
-    name: Workspace coverage (weekly)
-    runs-on: sm-standard-4
-    # The instrumented build cannot reuse the regular CI cache (different
-    # RUSTFLAGS), so a cold week rebuilds the workspace before running the
-    # full suite; give it double the test job's 60-minute budget.
-    timeout-minutes: 120
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-      # Match the PR gate's nextest semantics (ci.yml runs `--profile ci`):
-      # retries=0 plus the quarantine list and the ecstore-serial-flaky
-      # serialization. Set via env because `cargo llvm-cov`'s own --profile
-      # flag selects the *cargo build* profile, not the nextest profile.
-      NEXTEST_PROFILE: ci
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-coverage
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-
-      - name: Install cargo-llvm-cov
-        uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
-        with:
-          tool: cargo-llvm-cov
-
-      - name: Install llvm-tools component
-        run: rustup component add llvm-tools-preview
-
-      - name: Run instrumented test suite
-        run: cargo llvm-cov nextest --workspace --exclude e2e_test --no-report
-
-      - name: Generate lcov and JSON reports
-        run: |
-          mkdir -p target/llvm-cov
-          cargo llvm-cov report --lcov --output-path target/llvm-cov/lcov.info
-          cargo llvm-cov report --json --output-path target/llvm-cov/coverage.json
-
-      - name: Write per-crate summary
-        run: python3 scripts/coverage_per_crate.py target/llvm-cov/coverage.json >> "$GITHUB_STEP_SUMMARY"
-
-      - name: Upload coverage artifact
-        if: always()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: coverage-lcov-${{ github.run_number }}
-          path: |
-            target/llvm-cov/lcov.info
-            target/llvm-cov/coverage.json
-          retention-days: 90
-          if-no-files-found: ignore
-
-  alert-on-failure:
-    name: Alert on scheduled failure
-    needs: [coverage]
-    # Only scheduled runs open/append the tracking issue (backlog#1149 ci-8);
-    # manual workflow_dispatch runs stay quiet so a debugging run never files a
-    # spurious alert.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-      - name: Open or update failure-tracking issue
-        uses: ./.github/actions/schedule-failure-issue
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/docker.yml

@@ -66,7 +66,7 @@ env:
   CARGO_TERM_COLOR: always
   REGISTRY_DOCKERHUB: rustfs/rustfs
   REGISTRY_GHCR: ghcr.io/${{ github.repository }}
-  REGISTRY_QUAY: quay.io/rustfs/rustfs
+  REGISTRY_QUAY: quay.io/${{ secrets.QUAY_USERNAME }}/rustfs
   DOCKER_PLATFORMS: linux/amd64,linux/arm64
 
 jobs:
@@ -247,15 +247,13 @@ jobs:
             esac
           fi
 
-          {
-            echo "should_build=$should_build"
-            echo "should_push=$should_push"
-            echo "build_type=$build_type"
-            echo "version=$version"
-            echo "short_sha=$short_sha"
-            echo "is_prerelease=$is_prerelease"
-            echo "create_latest=$create_latest"
-          } >> "$GITHUB_OUTPUT"
+          echo "should_build=$should_build" >> $GITHUB_OUTPUT
+          echo "should_push=$should_push" >> $GITHUB_OUTPUT
+          echo "build_type=$build_type" >> $GITHUB_OUTPUT
+          echo "version=$version" >> $GITHUB_OUTPUT
+          echo "short_sha=$short_sha" >> $GITHUB_OUTPUT
+          echo "is_prerelease=$is_prerelease" >> $GITHUB_OUTPUT
+          echo "create_latest=$create_latest" >> $GITHUB_OUTPUT
 
           echo "🐳 Docker Build Summary:"
           echo "  - Should build: $should_build"
@@ -322,6 +320,7 @@ jobs:
         run: |
           BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
           VERSION="${{ needs.build-check.outputs.version }}"
+          SHORT_SHA="${{ needs.build-check.outputs.short_sha }}"
           CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
           VARIANT_SUFFIX="${{ matrix.suffix }}"
 
@@ -344,8 +343,8 @@ jobs:
               ;;
           esac
 
-          echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
-          echo "docker_channel=$DOCKER_CHANNEL" >> "$GITHUB_OUTPUT"
+          echo "docker_release=$DOCKER_RELEASE" >> $GITHUB_OUTPUT
+          echo "docker_channel=$DOCKER_CHANNEL" >> $GITHUB_OUTPUT
 
           echo "🐳 Docker build parameters:"
           echo "  - Original version: $VERSION"
@@ -377,7 +376,7 @@ jobs:
           fi
 
           # Output tags
-          echo "tags=$TAGS" >> "$GITHUB_OUTPUT"
+          echo "tags=$TAGS" >> $GITHUB_OUTPUT
 
           # Generate labels
           LABELS="org.opencontainers.image.title=RustFS"
@@ -388,7 +387,7 @@ jobs:
           LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
           LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
 
-          echo "labels=$LABELS" >> "$GITHUB_OUTPUT"
+          echo "labels=$LABELS" >> $GITHUB_OUTPUT
 
           echo "🐳 Generated Docker tags:"
           echo "$TAGS" | tr ',' '\n' | sed 's/^/  - /'

.github/workflows/e2e-replication-nightly.yml

@@ -15,24 +15,20 @@
 # Nightly full replication e2e lane (backlog#1147 repl-1, deps: ci-4).
 #
 # The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the 20
-# FAST replication tests. This scheduled lane runs the remaining 27
+# FAST bucket-replication tests. This scheduled lane runs the remaining 18
 # heavier replication e2e tests that are unfit for a per-PR gate:
 #
-#   * 2 remote-target TLS validation tests.
-#   * 12 bucket-replication data-plane/helper tests (PUT/delete + poll for
-#     convergence; two replicate over HTTPS, two pin active SSE failure
-#     contracts, and one guards event/history observers). The SSE-S3 contract
-#     remains ignored under backlog#1291.
-#   * 11 `_real_dual_node` site-replication tests (each spawns TWO rustfs
+#   * 8 bucket-replication data-plane tests (PUT/delete + poll for convergence;
+#     two replicate over HTTPS).
+#   * 9 `_real_dual_node` site-replication tests (each spawns TWO rustfs
 #     servers and drives the cross-process site-replication control plane).
-#   * 1 `_real_three_node` site-replication test.
 #   * 1 `_real_single_node` service-account round-trip test.
 #
 # The selection is the [profile.e2e-repl-nightly] default-filter in
 # .config/nextest.toml — the single wiring mechanism (repl-1 / ci-4). Do NOT
 # add ad-hoc cargo-test steps here; change the filterset instead.
 #
-# Explicit division of labor: these 27 tests run ONLY here, never double-run
+# Explicit division of labor: these 18 tests run ONLY here, never double-run
 # in ci-5's future e2e-full merge gate. TODO(ci-7): once the ci domain's
 # consolidated scheduled e2e workflow exists, fold this interim repl-owned lane
 # into it rather than growing a second scheduled entrypoint.
@@ -84,12 +80,7 @@ jobs:
           python-version: "3.12"
 
       - name: Install awscurl
-        run: |
-          python3 -m pip install --user --upgrade pip awscurl
-          echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
-
-      - name: Verify awscurl
-        run: test -x "$AWSCURL_PATH"
+        run: python3 -m pip install --user --upgrade pip awscurl
 
       # Build the rustfs binary once up front. The e2e tests spawn it as a
       # child process (crates/e2e_test/src/common.rs) and will build it on

.github/workflows/minio-interop.yml

@@ -1,70 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# MinIO on-disk interop: prove RustFS reads MinIO-written erasure-coded SSE
-# objects with byte-identical data and correct logical size.
-#
-# This is NOT a PR gate. The fixtures are real MinIO backend trees generated on
-# the fly (they are gitignored, never committed), so the job regenerates them
-# each run with Docker and then runs the `#[ignore]` reader tests in
-# crates/ecstore/tests/minio_generated_read_test.rs.
-#
-# Runner: GitHub-hosted `ubuntu-latest`. It reliably ships Docker + Python,
-# unlike the self-hosted fleet, whose pods drift in Docker/pip availability
-# (see the infra note in e2e-s3tests.yml). Nightly + manual only.
-name: minio-interop
-
-on:
-  workflow_dispatch:
-  schedule:
-    # Nightly at 03:17 UTC (offset from other nightly jobs).
-    - cron: "17 3 * * *"
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
-
-permissions:
-  contents: read
-
-jobs:
-  minio-interop:
-    name: MinIO interop (EC + SSE read parity)
-    # Skip on forks: needs the repo's runners and is not a contributor gate.
-    if: github.repository == 'rustfs/rustfs'
-    runs-on: ubuntu-latest
-    timeout-minutes: 40
-    env:
-      # Fixed 32-byte test KMS key baked into the fixture lab; not a secret.
-      RUSTFS_MINIO_STATIC_KMS_KEY_B64: IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g=
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-minio-interop
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-
-      - name: Generate real MinIO fixtures via Docker
-        run: bash crates/rio-v2/tests/minio_fixture_lab/capture_via_docker.sh
-
-      - name: Run MinIO interop reader tests
-        run: |
-          cargo nextest run --run-ignored ignored-only \
-            -p rustfs-ecstore --features rio-v2 \
-            -E 'binary(minio_generated_read_test)'

.github/workflows/mint.yml

@@ -135,10 +135,6 @@ jobs:
         run: |
           docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net
           docker rm -f rustfs-mint >/dev/null 2>&1 || true
-          # The four disks share one physical device on the runner (a single
-          # loopback filesystem), so the local physical-disk-independence guard
-          # would refuse to start. Bypass it — this is the CI use case the guard
-          # explicitly sanctions via RUSTFS_UNSAFE_BYPASS_DISK_CHECK.
           docker run -d --name rustfs-mint \
             --network rustfs-net \
             -p 9000:9000 \
@@ -146,7 +142,6 @@ jobs:
             -e RUSTFS_ACCESS_KEY="${S3_ACCESS_KEY}" \
             -e RUSTFS_SECRET_KEY="${S3_SECRET_KEY}" \
             -e RUSTFS_VOLUMES="/data/rustfs{0...3}" \
-            -e RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true \
             -v /tmp/rustfs-mint:/data \
             rustfs-ci
 

.github/workflows/performance-ab.yml

@@ -41,10 +41,6 @@ on:
         type: boolean
   pull_request:
     types: [labeled, synchronize, reopened]
-  push:
-    # Every main commit pre-builds and caches its release binary (perf-3) so the
-    # nightly A/B restores a ready baseline instead of paying the double build.
-    branches: [main]
 
 permissions:
   contents: read
@@ -62,81 +58,22 @@ env:
   RUST_BACKTRACE: 1
 
 jobs:
-  # perf-3: on every push to main, build the release binary once and cache it
-  # keyed by commit SHA (rustfs-baseline-<sha>). The nightly A/B (and, later, the
-  # perf-7 PR gate) restore this instead of paying the ~32min-per-side source
-  # build. That double build is what pushed the expanded 24-cell nightly past its
-  # ceiling — 2026-07-11..07-14 all cancelled on the 120min timeout. Incremental
-  # builds off the shared cargo cache keep each push cheap, and building on the
-  # same sm-standard-2 runner the A/B measures on guarantees the cached binary is
-  # ABI-identical. Do NOT source this from build.yml's per-merge artifact: those
-  # are cancelled ~7/8 of the time and are not a reliable baseline.
-  build-baseline-cache:
-    name: Build + cache baseline binary
-    if: github.event_name == 'push'
-    runs-on: sm-standard-2
-    # Latest-wins: consumers only ever restore the binary for the *current*
-    # origin/main tip, so when pushes land faster than the ~65min build, a
-    # superseded build's output is dead weight — cancel it instead of stacking
-    # hour-long jobs on the shared runner pool. A skipped intermediate SHA at
-    # most costs one same-commit self-heal in the A/B job.
-    concurrency:
-      group: perf-baseline-build-main
-      cancel-in-progress: true
-    # #4806 put thin LTO + codegen-units=1 on [profile.release], pushing a
-    # single release build past 60min on this runner — every cache build on
-    # 2026-07-15 died on the old 60min ceiling ("exceeded the maximum execution
-    # time of 1h0m0s") and the cache never populated. The measured binary must
-    # keep the production profile, so the budget absorbs the build instead.
-    timeout-minutes: 100
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: warp-ab-${{ hashFiles('**/Cargo.lock') }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-
-      - name: Build release rustfs
-        run: cargo build --release --bin rustfs
-
-      - name: Stage binary for cache
-        run: |
-          set -euo pipefail
-          mkdir -p baseline-bin
-          cp target/release/rustfs baseline-bin/rustfs
-
-      - name: Cache baseline binary by SHA
-        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
-        with:
-          path: baseline-bin/rustfs
-          key: rustfs-baseline-${{ github.sha }}
-
   warp-ab:
     name: Warp A/B budget gate
-    # Always run on schedule / manual dispatch. Opt-in on PRs: only when the
-    # `perf-ab` label is present, and for `labeled` events only when the label
-    # being added is `perf-ab` itself (adding an unrelated label to an opted-in
-    # PR must not re-run the gate). Never on push — that event only feeds
-    # build-baseline-cache above.
+    # Opt-in on PRs: only run when the `perf-ab` label is present, and for
+    # `labeled` events only when the label being added is `perf-ab` itself —
+    # adding an unrelated label to an opted-in PR must not re-run the gate.
+    # Always run on schedule / manual dispatch.
     if: >-
-      github.event_name == 'schedule' ||
-      github.event_name == 'workflow_dispatch' ||
-      (github.event_name == 'pull_request' &&
-       contains(github.event.pull_request.labels.*.name, 'perf-ab') &&
+      github.event_name != 'pull_request' ||
+      (contains(github.event.pull_request.labels.*.name, 'perf-ab') &&
        (github.event.action != 'labeled' || github.event.label.name == 'perf-ab'))
     runs-on: sm-standard-2
-    # With perf-3's cached baseline binary the common (cache-hit) nightly is
-    # measurement-only and finishes well under 50min. This ceiling stays
-    # generous only to absorb the same-commit cache-miss self-heal (~65min
-    # single build with the post-#4806 LTO profile + measurement). A timeout
-    # surfaces via the alert-on-failure job (it fires on cancelled/timed-out,
-    # not just failure). perf-6 recalibrates the budget once the noise study
-    # lands.
+    # Phase-0 stopgap: the baseline+candidate release double-build alone is
+    # ~65min on this runner, so 90min left no room for a real full-matrix
+    # measurement (the earlier nightly runs only ever failed *before*
+    # measuring). 120min gives the 24-cell short matrix headroom until perf-3
+    # caches the baseline binary and restores a tighter budget.
     timeout-minutes: 120
     steps:
       - name: Checkout repository
@@ -173,106 +110,21 @@ jobs:
           fi
           echo "allow_regression=$allow" >> "$GITHUB_OUTPUT"
 
-      # perf-3: resolve the commits so the cache can be keyed by SHA. The
-      # baseline is origin/main; the candidate is the checked-out ref. On the
-      # nightly (checkout == main) they are the same commit, so one cached binary
-      # serves both phases and the run does zero source builds.
-      - name: Resolve baseline / candidate commits
-        id: commits
-        run: |
-          set -euo pipefail
-          baseline_sha="$(git rev-parse origin/main)"
-          candidate_sha="$(git rev-parse HEAD)"
-          echo "baseline_sha=$baseline_sha" >> "$GITHUB_OUTPUT"
-          echo "candidate_sha=$candidate_sha" >> "$GITHUB_OUTPUT"
-          echo "baseline commit: $baseline_sha"
-          echo "candidate commit: $candidate_sha"
-
-      # Exact-key restore of the baseline binary built by build-baseline-cache
-      # when origin/main last landed. A miss (binary evicted or not built yet)
-      # leaves cache-hit unset and the rig falls back to a source build.
-      - name: Restore cached baseline binary
-        id: baseline_cache
-        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
-        with:
-          path: baseline-bin/rustfs
-          key: rustfs-baseline-${{ steps.commits.outputs.baseline_sha }}
-
-      # Self-heal: on a nightly/dispatch run where the candidate commit IS the
-      # baseline commit, a cache miss would make the rig build the same commit
-      # twice (~65min per side with the post-#4806 LTO profile — no job budget
-      # fits that). Build it once here, reuse it for both phases, and save it
-      # back to the cache so the next run hits.
-      - name: Build baseline on cache miss (same-commit self-heal)
-        id: selfheal
-        if: >-
-          steps.baseline_cache.outputs.cache-hit != 'true' &&
-          steps.commits.outputs.baseline_sha == steps.commits.outputs.candidate_sha
-        run: |
-          set -euo pipefail
-          cargo build --release --bin rustfs
-          mkdir -p baseline-bin
-          cp target/release/rustfs baseline-bin/rustfs
-          echo "built=true" >> "$GITHUB_OUTPUT"
-
-      - name: Save self-healed baseline to cache
-        if: steps.selfheal.outputs.built == 'true'
-        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
-        with:
-          path: baseline-bin/rustfs
-          key: rustfs-baseline-${{ steps.commits.outputs.baseline_sha }}
-
       - name: Run warp A/B and gate
         id: ab
         run: |
           set -euo pipefail
-          # Budget note: with perf-3's cached baseline the nightly does no source
-          # build on a cache hit, so the wall-clock is dominated by the short warp
-          # matrix — duration/rounds/cooldown are kept small to fit all 24 cells
-          # (6 workloads x 2 phases x 2 drive-sync) rather than dropping cells.
-          # --health-timeout 180 outlasts the server's own 120s startup-readiness
-          # budget, which the rig's previous 60s health poll undershot (the first
-          # two nightly failures). perf-6 recalibrates these once the noise study
-          # lands.
+          # Budget note: the baseline+candidate release double-build (~65 min,
+          # cached away later by perf-3) dominates the 90-min job, so the
+          # measurement runs a short warp matrix — duration/rounds/cooldown are
+          # kept small to fit all 24 cells (6 workloads x 2 phases x 2 drive-sync)
+          # under budget rather than dropping cells. --health-timeout 180 outlasts
+          # the server's own 120s startup-readiness budget, which is what the
+          # rig's previous 60s health poll undershot (the first two nightly
+          # failures). perf-6 will recalibrate these once the pipeline is green.
           duration="${{ github.event.inputs.duration || '12s' }}"
-          baseline_sha="${{ steps.commits.outputs.baseline_sha }}"
-          candidate_sha="${{ steps.commits.outputs.candidate_sha }}"
-          baseline_hit="${{ steps.baseline_cache.outputs.cache-hit }}"
-          selfheal_built="${{ steps.selfheal.outputs.built }}"
-
-          args=(--duration "$duration" --rounds 2 --cooldown 5 --health-timeout 180)
-
-          if [[ "$baseline_hit" == "true" || "$selfheal_built" == "true" ]]; then
-            chmod +x baseline-bin/rustfs
-            base_bin="$PWD/baseline-bin/rustfs"
-            args+=(--baseline-bin "$base_bin")
-            if [[ "$baseline_hit" == "true" ]]; then
-              base_src="actions-cache (rustfs-baseline-$baseline_sha)"
-            else
-              base_src="source build (cache self-heal, saved as rustfs-baseline-$baseline_sha)"
-            fi
-            if [[ "$candidate_sha" == "$baseline_sha" ]]; then
-              # Nightly on main: the candidate is the same commit as the baseline,
-              # so reuse the one binary for both phases and skip all builds.
-              args+=(--candidate-bin "$base_bin" --skip-build)
-              cand_src="same binary as baseline (same commit)"
-            else
-              cand_src="source build of the checked-out ref"
-            fi
-          else
-            # Cache miss with candidate != baseline (opt-in PR gate only): fall
-            # back to the source double-build. With the post-#4806 LTO profile
-            # this will overrun the job budget and alert; rerun once the push
-            # cache build for origin/main has completed, or wait for perf-7's
-            # merge-base caching.
-            args+=(--baseline-ref origin/main)
-            base_src="source build of origin/main (cache miss)"
-            cand_src="source build of the checked-out ref"
-          fi
-
-          args+=(--provenance-note "baseline commit: $baseline_sha - $base_src")
-          args+=(--provenance-note "candidate commit: $candidate_sha - $cand_src")
-
+          args=(--baseline-ref origin/main
+            --duration "$duration" --rounds 2 --cooldown 5 --health-timeout 180)
           if [[ "${{ steps.exempt.outputs.allow_regression }}" == "true" ]]; then
             args+=(--allow-regression --exemption-reason "labeled perf-deliberate-tradeoff / dispatch override")
           fi
@@ -349,8 +201,12 @@ jobs:
         run: |
           gh pr comment "${{ github.event.pull_request.number }}" --body-file "${{ steps.ab.outputs.gate_md }}"
 
-      # Scheduled failure alerting is handled by the alert-on-failure job below
-      # (perf-2 consuming ci-8's schedule-failure-issue composite action).
+      # TODO(ci-8/perf-2): scheduled/dispatch failures are currently silent. Once
+      # ci-8 lands the .github/actions/schedule-failure-issue composite action,
+      # perf-2 adds a step here guarded by
+      #   if: failure() && github.event_name != 'pull_request'
+      # that calls it (label perf-nightly-failure, append to an existing open
+      # issue) instead of hand-rolling gh CLI dedup. Do not implement it here.
 
       - name: Enforce gate
         if: always()
@@ -368,16 +224,7 @@ jobs:
     # `always()` is required: without it this job is skipped when a needed
     # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
     # ci-8); PR and manual dispatch failures are already watched by a human.
-    # `cancelled` is included alongside `failure` on purpose: a job that hits
-    # timeout-minutes ends as `cancelled`, and the 2026-07-11..07-14 nightly
-    # timeouts went silent precisely because the guard was failure-only. The
-    # composite action already reports cancelled/timed-out jobs in the issue
-    # body. (Scheduled runs get a unique concurrency group with
-    # cancel-in-progress off, so a cancellation here means a timeout/manual
-    # abort, never a superseding run.)
-    if: >-
-      always() && github.event_name == 'schedule' &&
-      (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
+    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
     runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:

.github/workflows/star-history.yml

@@ -1,26 +0,0 @@
-name: Star History
-
-on:
-  schedule:
-    - cron: "17 3 * * *"
-  workflow_dispatch:
-
-permissions:
-  contents: write
-
-concurrency:
-  group: star-history
-  cancel-in-progress: false
-
-jobs:
-  update:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: overtrue/repo-visuals-action@72f34d24769ff5d341956da2f23952594ef2f1e2 # v1.3.0
-        with:
-          github-token: ${{ github.token }}
-          output-branch: star-history
-          output-path: .
-          chart-style: gradient
-          animate: "true"
-          contributors: "true"

.vscode/launch.json

@@ -1,7 +1,45 @@
 {
     "version": "0.2.0",
     "configurations": [
-          {
+        {
+            "name": "Debug RustFS observability (OTLP)",
+            "type": "lldb",
+            "request": "launch",
+            "cargo": {
+                "args": [
+                    "build",
+                    "--bin=rustfs",
+                    "--package=rustfs"
+                ],
+                "filter": {
+                    "name": "rustfs",
+                    "kind": "bin"
+                }
+            },
+            "args": [],
+            "cwd": "${workspaceFolder}",
+            "env": {
+                "RUST_LOG": "rustfs=debug,ecstore=info,s3s=info,iam=info",
+                "RUST_BACKTRACE": "full",
+                "RUSTFS_ACCESS_KEY": "rustfsadmin",
+                "RUSTFS_SECRET_KEY": "rustfsadmin",
+                "RUSTFS_VOLUMES": "./target/observability/data{1...4}",
+                "RUSTFS_ADDRESS": ":9000",
+                "RUSTFS_CONSOLE_ENABLE": "true",
+                "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
+                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
+                "RUSTFS_OBS_ENDPOINT": "http://127.0.0.1:4318",
+                "RUSTFS_OBS_TRACES_EXPORT_ENABLED": "true",
+                "RUSTFS_OBS_METRICS_EXPORT_ENABLED": "true",
+                "RUSTFS_OBS_LOGS_EXPORT_ENABLED": "true",
+                "RUSTFS_OBS_USE_STDOUT": "true",
+                "RUSTFS_OBS_LOG_DIRECTORY": "./target/observability/logs",
+                "RUSTFS_OBS_METER_INTERVAL": "5",
+                "RUSTFS_OBS_SERVICE_NAME": "rustfs-observability-local",
+                "RUSTFS_OBS_ENVIRONMENT": "development"
+            }
+        },
+        {
             "type": "lldb",
             "request": "launch",
             "name": "Debug(only) executable 'rustfs'",
@@ -172,7 +210,7 @@
             ],
         },
         {
-            "name": "Debug executable target/debug/rustfs with sse kms",
+            "name": "Debug executable target/debug/rustfs with sse",
             "type":  "lldb",
             "request": "launch",
             "program": "${workspaceFolder}/target/debug/rustfs",
@@ -200,7 +238,7 @@
                 // 2. kms local backend test key
                 // "RUSTFS_KMS_ENABLE": "true",
                 // "RUSTFS_KMS_BACKEND": "local",
-                // "RUSTFS_KMS_KEY_DIR": "/tmp/kms-key-dir",
+                // "RUSTFS_KMS_KEY_DIR": "./target/kms-key-dir",
                 // "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
                 // "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
 
@@ -212,40 +250,13 @@
                 // "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
 
                 // 4. kms vault transit backend test key
-                // "RUSTFS_KMS_ENABLE": "true",
-                // "RUSTFS_KMS_BACKEND": "vault-transit",
-                // "RUSTFS_KMS_VAULT_ADDRESS": "http://127.0.0.1:8200",
-                // "RUSTFS_KMS_VAULT_TOKEN": "Dev Token",
-                // "RUSTFS_KMS_VAULT_MOUNT_PATH": "transit",
-                // "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
-                
-                // 5、kms static backend test key
                 "RUSTFS_KMS_ENABLE": "true",
-                "RUSTFS_KMS_BACKEND": "static",
-                "RUSTFS_KMS_STATIC_SECRET_KEY": "rustfs-master-key:2dfNXGHlsEflGVCxb+5DIdGEl1sIvtwX+QfmYasi5QM="
-            },
-            "sourceLanguages": [
-                "rust"
-            ],
-        },
-        {
-            "name": "Debug executable target/debug/rustfs with local sse",
-            "type": "lldb",
-            "request": "launch",
-            "program": "${workspaceFolder}/target/debug/rustfs",
-            "args": [],
-            "cwd": "${workspaceFolder}",
-            "env": {
-                "RUSTFS_ACCESS_KEY": "rustfsadmin",
-                "RUSTFS_SECRET_KEY": "rustfsadmin",
-                "RUSTFS_VOLUMES": "./target/volumes/test{1...4}",
-                "RUSTFS_ADDRESS": ":9000",
-                "RUSTFS_CONSOLE_ENABLE": "true",
-                "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
-                "RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
-                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
-                "RUSTFS_SSE_S3_MASTER_KEY": "xGb3aYSp825j2tPpg8JrUzghiXsIkfdOtmrsJ/iafiM=",
-                "RUST_LOG": "rustfs=debug,ecstore=debug,s3s=debug,iam=debug",
+                "RUSTFS_KMS_BACKEND": "vault-transit",
+                "RUSTFS_KMS_VAULT_ADDRESS": "http://127.0.0.1:8200",
+                "RUSTFS_KMS_VAULT_TOKEN": "Dev Token",
+                "RUSTFS_KMS_VAULT_MOUNT_PATH": "transit",
+                "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
+
             },
             "sourceLanguages": [
                 "rust"

AGENTS.md

@@ -14,19 +14,13 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 
 ## Execution Discipline
 
-- Read the relevant existing code, tests, and local guidance before changing behavior. For new helpers or test setup, that read includes `crates/utils`, `crates/common`, and the touched crate's own `test_util`/fixtures (see Reuse Before You Write).
+- Read the relevant existing code, tests, and local guidance before changing behavior.
 - State assumptions when they affect the implementation or verification path.
 - If a task has multiple plausible interpretations, list the options briefly and choose the narrowest reasonable path; ask when the ambiguity would make the change risky.
 - For multi-step work, keep the plan minimal and tied to verifiable outcomes.
 - Avoid redundant file reads, repeated commands, and unnecessary exploratory work once enough context is available.
 - A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification that survives Adversarial Validation (below).
 
-## Autonomy and Approval Boundaries
-
-- Inquiry tasks (answer, explain, review, diagnose, plan): report findings; do not change files unless a fix is explicitly requested.
-- Action tasks (change, build, fix): make in-scope local changes without asking for approval.
-- Ask for confirmation before destructive or hard-to-reverse operations (force-pushes, history rewrites, deleting data or branches), merging a PR (reviewer approval required), or any material expansion of the requested scope.
-
 ## Communication and Language
 
 - Respond in the same language used by the requester.
@@ -47,27 +41,14 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 - Do not refactor existing code only to make it easier to unit test.
 - Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths.
 - Keep code elegant, concise, and direct. Prefer minimal, readable implementations over over-engineering and excessive abstraction. Use comments to clarify non-obvious intent and invariants, not to compensate for unclear code.
-- Do not write comments that narrate what the next line does, restate a signature, or describe the change you just made — that commentary belongs in the PR description, not the code. Required invariant comments — lock ordering, `SAFETY`, unwrap justification, `#[allow(dead_code)]` rationale, `RUSTFS_COMPAT_TODO` — are never narration.
 - Mention unrelated issues when useful, but do not fix them as part of a narrow task.
 
-## Reuse Before You Write
+## Constant and String Usage
 
-Search for an existing implementation before writing a new one; extend what exists instead of duplicating it:
-
-- **Helpers and utilities** (path/string handling, hashing, retry, env parsing, IO wrappers): check `ls crates/utils/src` first — file names map to operations (`retry.rs`, `envs.rs`, `hash.rs`, `path.rs`, `string.rs`, `io.rs`) — plus `crates/common` (shared structures/globals), then `rg -i 'fn \w*<term>' crates/utils/src crates/common/src <touched-crate>/src` for signatures. Helpers are snake_case: a full-text single-word grep over a large crate drowns you and a multi-word phrase returns nothing. Reimplementing an existing workspace helper — or hand-rolling what `std`, `tokio`, or an existing workspace dependency already provides — is a review finding, not a style preference.
-- **Reuse requires matching semantics, not a matching name**: before adopting a helper, check its normalization (`clean` resolves `.`/`..` — never apply it to raw S3 object keys), error type, backoff/deadline behavior, and durability gating against the call site. When semantics differ, a new narrowly-named helper with a comment naming the rejected lookalike is the correct outcome. The inverse also holds: workspace wrappers exist because raw `std`/`tokio` semantics were insufficient (durability gates, retries) — prefer the wrapper over the raw call.
-- **Constants and fixed tokens** (protocol labels, error identifiers, header keys, event names, metric names, command tags): search for existing constants/enums that already represent the same semantic value and reuse them. If a value is truly new, define one local constant near related logic; never scatter the literal across sites. When changing existing behavior, align naming and format with the established constants.
-- **Test scaffolding**: reuse existing test utilities and fixtures (the touched crate's own `test_util` module and `tests/fixtures`, or `crates/test-utils`) instead of writing new setup code — run `rg -l '<fn-under-test>' <crate>/src <crate>/tests` before writing a test. A new test must pin a failure mode no existing test covers. Near-duplicate means same code path AND same poison-value class: this repo's boundary companions (n==max vs max+1, absent vs empty vs nil UUID bytes, MetaObject vs MetaDeleteMarker) are distinct by definition and must all be written.
-
-## Necessary Code Only
-
-Net-new code — files, types, branches, comments — is cost to justify, not progress:
-
-- Validate at the trust boundary — untrusted client input, bytes read from disk, RPC payloads, config (see Serde Safety and Cross-Cutting Domain Invariants) — then trust the type: do not re-check what the type system or a validated upstream layer already guarantees, and cite the establishing check (`file:line`) when the guarantee is not obvious.
-- The exception is load-bearing: a value that crossed a persistence, RPC, or version boundary is never guaranteed by the code on the other side — a peer may be older or buggy, disk bytes may be corrupt — so the Cross-Cutting Domain Invariant patterns apply at every consumer, and re-checks immediately before a destructive action (delete, overwrite, quorum decision) stay. Deleting an existing guard is a behavior change requiring adversarial review, not cleanup.
-- Every new branch needs a nameable trigger: a concrete input, state, or failure that reaches it — for boundary-crossing values, corrupt or stale persisted/peer data is always nameable. If you cannot name one, do not write the branch. If the case is truly unreachable, encode the invariant in the type; where that is impossible, return a typed internal error (fail closed). `debug_assert!` is acceptable only for pure internal arithmetic on values that never crossed a disk/RPC/config boundary — never as the sole guard on decoded or peer-supplied data.
-- Never substitute a default where the value is required (e.g. `unwrap_or_default()` on metadata that must exist) — that converts corruption into a wrong answer. Return the typed error instead: explicit failure over implicit success.
-- Attach error context once, at the layer where it is actionable: re-wrapping equivalent context at every hop is noise, and expanding a fallible chain into nested `match` blocks where `?` or a combinator suffices is a finding. Never add context by converting a typed error into a generic variant below an error-aggregation or quorum layer (`reduce_errs` classifies by variant equality) — context there belongs in a `tracing` event, not the error value.
+- Before introducing new string literals, search for existing constants/enums that already represent the same semantic value.
+- Reuse existing constants for protocol labels, error identifiers, header keys, event names, metric names, command tags, and similar fixed tokens.
+- If a new string is truly unique, define a local constant near related logic and avoid scattering the literal across multiple sites.
+- When changing existing behavior, keep naming and format consistency by aligning with established project constants.
 
 ## Sources of Truth
 
@@ -160,7 +141,7 @@ Pick the tier from the riskiest file touched; when in doubt, pick the higher.
 - **Exempt:** docs/comments/instruction-only changes, formatting, typos with
   no runtime surface. Skip this section.
 - **Mechanical:** pure renames, file moves, test-only or tooling changes —
-  correctness and simplicity adversaries only.
+  correctness adversary only.
 - **Standard (the default):** any change that affects behavior.
 - **High risk:** touches locking, erasure coding, quorum/heal, replication,
   multipart, RPC, lifecycle/tiering, metadata formats (`xl.meta`),
@@ -180,8 +161,9 @@ encode this repo's shipped bugs.
 
 - **Correctness adversary** — construct a concrete input/state/interleaving
   that yields wrong output, data loss, or a crash. Probe error paths and edge
-  values (empty, nil UUID, zero-length, quorum−1, missing version).
-- **Simplicity adversary** — same behavior, less code. Hunt the materially smaller or more idiomatic diff (see Change Style for Existing Logic, Reuse Before You Write, and Necessary Code Only): reimplemented workspace helpers, one-caller extractions, rewrites where an in-place edit suffices, defensive branches with no nameable trigger, redundant error wrapping, near-duplicate tests, narration comments. A smaller diff achieving identical behavior is a finding, reported with the concrete replacement; forced reuse of a helper with mismatched semantics is equally a finding.
+  values (empty, nil UUID, zero-length, quorum−1, missing version). For code
+  diffs, a materially smaller or more idiomatic diff achieving the same
+  behavior is also a finding (see Change Style for Existing Logic).
 - **Security reviewer** — authn/authz bypass, injection, secret leakage,
   untrusted deserialization (see Serde Safety), path traversal, timing leaks.
 - **Concurrency/durability reviewer** — lock ordering, races, cancellation,
@@ -197,12 +179,11 @@ encode this repo's shipped bugs.
   wrong while all tests stay green — if one exists, coverage is insufficient.
   A missing test is a finding, not a note.
 
-Standard tier: correctness adversary + simplicity adversary + test-coverage
-skeptic, plus every role whose domain the diff touches (async or
-shared-state code → concurrency; parsing of untrusted input → security;
-public crate API shape → compatibility; per-request or per-object hot paths
-→ performance).
-High risk: all seven roles.
+Standard tier: correctness adversary + test-coverage skeptic, plus every
+role whose domain the diff touches (async or shared-state code →
+concurrency; parsing of untrusted input → security; public crate API shape
+→ compatibility; per-request or per-object hot paths → performance).
+High risk: all six roles.
 
 ### Protocol
 

ARCHITECTURE.md

@@ -41,7 +41,7 @@ The repository is a Cargo workspace with a flat `crates/` layout:
 
 ```
 rustfs/                      # Workspace root (virtual manifest)
-├── rustfs/                  # Main binary + library crate
+├── rustfs/                  # Main binary + library crate (75K lines)
 │   └── src/
 │       ├── main.rs          # Entry point, startup sequence
 │       ├── lib.rs           # Module tree root
@@ -53,7 +53,7 @@ rustfs/                      # Workspace root (virtual manifest)
 │       ├── config/          # CLI args, config parsing, workload profiles
 │       └── ...
 ├── crates/                  # library crates (authoritative list: Cargo.toml [workspace].members)
-│   ├── ecstore/             # Erasure-coded storage engine
+│   ├── ecstore/             # Erasure-coded storage engine (⚠️ 87K lines)
 │   ├── rio/                 # Reader I/O pipeline (encrypt, compress, hash)
 │   ├── io-core/             # Zero-copy I/O, scheduling, buffer pool
 │   ├── io-metrics/          # I/O metrics collection
@@ -83,25 +83,124 @@ A request flows **downward** through the layers. No layer should reach upward
 
 ### Crate Reference
 
-`Cargo.toml` is the authoritative workspace membership and `cargo tree` is the
-authoritative dependency graph. This overview deliberately avoids line-count
-and dependency-depth snapshots because both quickly become stale during
-refactors.
+> Depth levels, line counts, and crate counts in this section are a
+> point-in-time snapshot and drift with refactors. Treat them as orders of
+> magnitude; `Cargo.toml` and `cargo tree` are the source of truth.
+
+Crates are organized in a dependency DAG with 9 depth levels (0 = leaf, 8 = top):
+
+```
+Depth 0 — LEAF (no internal deps):
+  appauth, checksums, config, credentials, crypto, io-metrics,
+  madmin, s3-common, workers, zip
+
+Depth 1:
+  io-core (→ io-metrics)
+  policy (→ config, credentials, crypto)
+  utils                                   (historical → config edge removed; now effectively leaf)
+
+Depth 2:
+  concurrency, filemeta, keystone, kms, lock, obs,
+  signer, targets, trusted-proxies
+
+Depth 3:
+  common                                  (historical → filemeta/madmin edges removed; now effectively leaf)
+
+Depth 4:
+  object-capacity, protos, rio
+
+Depth 5 — CORE:
+  ecstore (16 internal deps, 11 dependents — the architectural heart)
+
+Depth 6:
+  audit, heal, iam, metrics, notify, s3select-api, scanner
+
+Depth 7:
+  object-io, protocols, s3select-query
+
+Depth 8 — TOP:
+  rustfs (35 internal deps — the binary, depends on almost everything)
+```
 
 #### By Domain
 
-| Domain | Current workspace crates | Responsibility |
-|--------|--------------------------|----------------|
-| Foundation | `checksums`, `common`, `config`, `data-usage`, `utils` | Shared configuration, data-usage models, utilities, and checksums. |
-| I/O and storage | `concurrency`, `ecstore`, `filemeta`, `heal`, `io-core`, `io-metrics`, `lifecycle`, `lock`, `object-capacity`, `object-data-cache`, `replication`, `rio`, `rio-v2`, `scanner`, `storage-api` | Erasure-coded object storage, metadata, recovery, lifecycle, replication, locking, cache, and I/O pipelines. |
-| Security and identity | `credentials`, `crypto`, `iam`, `keystone`, `kms`, `policy`, `security-governance`, `signer`, `tls-runtime`, `trusted-proxies` | Credentials, authentication, authorization, encryption, key management, TLS, and security contracts. |
-| Protocols and contracts | `extension-schema`, `madmin`, `protos`, `protocols`, `s3-ops`, `s3-types`, `s3select-api`, `s3select-query` | Admin, inter-node, S3, S3 Select, and optional protocol contracts. |
-| Operations and integration | `audit`, `notify`, `obs`, `targets`, `zip` | Auditing, observability, event delivery, notification targets, and archive support. |
-| Test support | `e2e_test`, `test-utils` | End-to-end validation and shared test bootstrap utilities. |
+**Core Infrastructure:**
 
-The `rustfs` binary crate composes these libraries into the running server.
-`ecstore` remains the storage engine at the architectural center; its internal
-module split is tracked under `docs/architecture/`.
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `config` | 3.3K | Configuration types and environment parsing |
+| `utils` | 8.7K | Pure utilities (paths, compression, network, retry) |
+| `common` | 4.4K | Shared runtime state, globals, data usage types, metrics |
+| `madmin` | 5.5K | Admin API request/response types |
+
+**I/O Pipeline:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `io-core` | 6.5K | Zero-copy I/O, buffer pool, direct I/O, scheduling, backpressure |
+| `io-metrics` | 4.5K | I/O operation metrics and counters |
+| `rio` | 6.9K | Composable reader chain (encrypt → compress → hash → limit) |
+| `object-io` | 2.4K | High-level object read/write using rio + ecstore |
+| `concurrency` | 0.8K | Shared concurrency contract types: workload admission snapshots, worker-slot pool, policy types (runtime control lives in `rustfs/src/storage`) |
+
+**Storage Engine:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `ecstore` | 87K | ⚠️ Erasure-coded storage: disks, pools, buckets, replication, lifecycle |
+| `filemeta` | 10K | File/object metadata types and versioning |
+| `checksums` | 732 | Checksum computation |
+| `lock` | 7.1K | Distributed lock manager |
+| `heal` | 5.9K | Data healing / bitrot repair |
+| `scanner` | 5.4K | Background data usage scanner |
+| `object-capacity` | 2.5K | Capacity tracking and management |
+
+**Security & Auth:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `crypto` | 1.6K | Encryption primitives |
+| `credentials` | 713 | Credential types (access key / secret key) |
+| `signer` | 1.4K | S3 v4 request signing |
+| `iam` | 9.0K | Identity and access management |
+| `policy` | 8.8K | Policy engine (S3 bucket/IAM policies) |
+| `kms` | 8.1K | Key management service integration |
+| `keystone` | 1.9K | OpenStack Keystone auth |
+| `appauth` | 143 | Application-level auth tokens |
+
+**Protocol & API:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `protos` | 5.7K | Protobuf/gRPC definitions for inter-node RPC |
+| `protocols` | 18K | FTP/FTPS, WebDAV, Swift API support |
+| `s3-common` | 738 | Shared S3 types |
+| `s3select-api` | 1.9K | S3 Select interface |
+| `s3select-query` | 3.6K | S3 Select query engine |
+
+**Observability:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `metrics` | 8.4K | Prometheus metric collectors |
+| `io-metrics` | 4.5K | I/O-specific metrics |
+| `obs` | 5.6K | OpenTelemetry tracing and telemetry |
+| `audit` | 2.4K | Audit logging |
+
+**Events:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `notify` | 5.5K | Event notification system |
+| `targets` | 3.2K | Notification targets (Kafka, AMQP, webhook, etc.) |
+
+**Other:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `trusted-proxies` | 4.0K | Trusted proxy / IP forwarding |
+| `zip` | 986 | ZIP archive support for bulk downloads |
+| `workers` | 136 | Simple worker abstraction |
 
 ## Architecture Invariants
 
@@ -113,7 +212,7 @@ module split is tracked under `docs/architecture/`.
    No upward imports.
 
 2. **Leaf crates have zero internal dependencies.** `config`, `credentials`, `crypto`,
-   `io-metrics`, and `madmin` should depend only on external crates.
+   `io-metrics`, `madmin`, `s3-common` should depend only on external crates.
    - ✅ RESOLVED: the historical `utils → config` and `common → filemeta`/`madmin`
      edges were removed; do not reintroduce them (see Known Structural Issues).
 

CHANGELOG.md

@@ -9,14 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 - **Helm Ingress**: `customAnnotations` are now merged with class-specific annotations (nginx/traefik) instead of being ignored when `ingress.className` is set.
-- **Per-pool erasure parity**: Erasure parity (STANDARD and reduced-redundancy) is now resolved independently for every pool instead of reusing the first pool's value. A heterogeneous topology — for example a 4-drive pool plus a 2-drive pool created during expansion — previously inherited the first pool's parity and could resolve to zero data shards in the smaller pool, panicking Reed-Solomon construction on write. Automatic parity now resolves per pool (for example `2+2` in the 4-drive pool and `1+1` in the 2-drive pool). Fixes #4801.
 
 ### Added
-- **NATS JetStream Publish Path**: Opt-in at-least-once delivery for the NATS notify and audit targets. A NATS Core publish flushes to the connection without awaiting a broker acknowledgement, so an event can be lost across a broker restart or a reconnect after the send queue has already cleared it. A queued event now clears only after the JetStream `PublishAck`, so bucket notifications survive those interruptions. Off by default and byte-identical to the NATS Core path when disabled.
-  - Three configuration keys per target: `JETSTREAM_ENABLE`, `JETSTREAM_STREAM_NAME`, and `JETSTREAM_ACK_TIMEOUT_SECS`, under the `RUSTFS_NOTIFY_NATS_` and `RUSTFS_AUDIT_NATS_` prefixes
-  - Durable store-and-forward with a stable dedup id sent as the `Nats-Msg-Id` header, so a replay after a crash is collapsed by the server duplicate window
-  - Pre-flight stream validation, and a bounded failed-events store (count and TTL). Only a non-retryable rejection is recorded in the failed-events store. A retryable condition keeps the entry on the live queue until it is delivered
-  - Operator guide at `docs/operations/nats-jetstream.md`
 - **OpenStack Keystone Authentication Integration**: Full support for OpenStack Keystone authentication via X-Auth-Token headers
   - Tower-based middleware (`KeystoneAuthLayer`) self-contained within `rustfs-keystone` crate
   - Task-local storage for async-safe credential passing between middleware and auth handlers
@@ -39,7 +33,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 - **HTTP Server Stack**: Integrated `KeystoneAuthLayer` middleware from `rustfs-keystone` crate into service stack (positioned after ReadinessGateLayer)
-- **Storage-class validation on startup (upgrade note)**: A persisted explicit storage class (`RUSTFS_STORAGE_CLASS_STANDARD` / `RUSTFS_STORAGE_CLASS_RRS`, for example `EC:2`) is now validated against the actual per-pool drive counts at startup and rejected when a pool cannot satisfy it. This is fail-closed and correct, but a cluster that persisted a storage class larger than a small or heterogeneous pool can hold (for example `EC:2` alongside a 2-drive pool), which earlier releases accepted and silently resolved to an invalid layout, will now refuse to start after upgrade. To recover, unset `RUSTFS_STORAGE_CLASS_STANDARD` so the server derives a valid per-pool default automatically, or set it to a value every pool can satisfy.
 - **IAMAuth**: Enhanced `get_secret_key()` to return empty secret for Keystone credentials (bypasses signature validation)
 - **Auth Module**: Modified `check_key_valid()` to retrieve Keystone credentials from task-local storage and determine admin status
 - **`StorageBackend` trait**: extended with multipart upload methods (`create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`) plus `upload_part_copy`. Streaming-upload code path is now available to FTPS, WebDAV, and Swift drivers as well.

CLAUDE.md

@@ -31,8 +31,6 @@ make build-docker BUILD_OS=ubuntu22.04
 - Architecture, layering, crate map: [ARCHITECTURE.md](ARCHITECTURE.md)
 - Migration guardrails & readiness contracts: [docs/architecture/](docs/architecture/README.md)
 - CI gates: `.github/workflows/ci.yml` (source of truth; never copy its steps into docs)
-- Test-layer taxonomy, per-layer entry commands, serial/nextest rules, flake
-  policy: [docs/testing/README.md](docs/testing/README.md)
 - Tier/ILM transition debugging (xl.meta inspection, versionId tracing):
   [docs/operations/tier-ilm-debugging.md](docs/operations/tier-ilm-debugging.md)
 

CONTRIBUTING.md

@@ -68,8 +68,6 @@ make pre-pr
 
 > `make test` requires [cargo-nextest](https://nexte.st) (CI runs it and only nextest honours `.config/nextest.toml` test-groups). Install it with `cargo install cargo-nextest --locked` or a prebuilt binary (see https://nexte.st/docs/installation/). To run the plain `cargo test` fallback anyway (results not authoritative — serialization semantics differ from CI), set `RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1`.
 
-> For the full test-layer taxonomy (unit / ecstore black-box / e2e / s3s-e2e / S3 compatibility / chaos / fuzz / bench), each layer's entry command, the naming conventions the migration gate depends on, and the serial/nextest rules, see [docs/testing/README.md](docs/testing/README.md).
-
 ### 🔒 Automated Pre-commit Hooks
 #### What `make pre-commit` and `make pre-pr` actually run
 

Cargo.toml

@@ -31,7 +31,6 @@ members = [
     "crates/lifecycle", # Lifecycle rule evaluation contracts
     "crates/kms", # Key Management Service
     "crates/lock", # Distributed locking implementation
-    "crates/log-analyzer", # Offline log fault-analysis core (rustfs diagnose)
     "crates/madmin", # Management dashboard and admin API interface
     "crates/notify", # Notification system for events
     "crates/obs", # Observability utilities
@@ -53,7 +52,6 @@ members = [
     "crates/extension-schema", # Extension schema contracts
     "crates/signer", # client signer
     "crates/storage-api", # Storage API contracts
-    "crates/test-utils", # Shared test bootstrap helpers (dev-dependency only)
     "crates/targets", # Target-specific configurations and utilities
     "crates/trusted-proxies", # Trusted proxies management
     "crates/tls-runtime", # Project-wide TLS runtime foundation
@@ -68,8 +66,8 @@ resolver = "3"
 edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
-rust-version = "1.97.1"
-version = "1.0.0-beta.11"
+rust-version = "1.96.0"
+version = "1.0.0-beta.8"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -86,93 +84,91 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-beta.11" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.11" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.11" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.11" }
-rustfs-common = { path = "crates/common", version = "1.0.0-beta.11" }
-rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.11" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-beta.11" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.11" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.11" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.11" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.11" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.11" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.11" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.11" }
-rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.11" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.11" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.11" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.11" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.11" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.11" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.11" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.11" }
-rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.11" }
-rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-beta.11" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.11" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.11" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.11" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.11" }
-rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.11" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.11" }
-rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.11" }
-rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.11" }
-rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.11" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.11" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.11" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.11" }
-rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.11" }
-rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.11" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.11" }
-rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.11" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.11" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.11" }
-rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-beta.11" }
-rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.11" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.11" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.11" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.8" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.8" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.8" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.8" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.8" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.8" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.8" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.8" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.8" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.8" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.8" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.8" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.8" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.8" }
+rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.8" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.8" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.8" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.8" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.8" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.8" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.8" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.8" }
+rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.8" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.8" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.8" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.8" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.8" }
+rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.8" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.8" }
+rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.8" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.8" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.8" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.8" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.8" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.8" }
+rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.8" }
+rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.8" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.8" }
+rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.8" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.8" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.8" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.8" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.8" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.8" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
-async_zip = { default-features = false, version = "0.0.18" }
-mysql_async = { default-features = false, version = "0.37" }
+async_zip = { version = "0.0.18", default-features = false, features = ["tokio", "deflate"] }
+mysql_async = { version = "0.37", default-features = false, features = ["default-rustls", "tracing"] }
 async-compression = { version = "0.4.42" }
 async-recursion = "1.1.1"
-async-trait = "0.1.91"
-async-nats = { version = "0.50.0", default-features = false }
+async-trait = "0.1.89"
+async-nats = "0.49.1"
 axum = "0.8.9"
-futures = "0.3.33"
-futures-core = "0.3.33"
+futures = "0.3.32"
+futures-core = "0.3.32"
 futures-lite = "2.6.1"
-futures-util = "0.3.33"
+futures-util = "0.3.32"
 pollster = "1.0.1"
-pulsar = { default-features = false, version = "6.8.0" }
-lapin = { default-features = false, version = "4.10.0" }
-hyper = { version = "1.11.0" }
-hyper-rustls = { default-features = false, version = "0.27.9" }
-hyper-util = { version = "0.1.20" }
+pulsar = { version = "6.8.0", default-features = false, features = ["tokio-rustls-runtime", "telemetry"] }
+lapin = { version = "4.10.0", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
+hyper = { version = "1.10.1", features = ["http2", "http1", "server"] }
+hyper-rustls = { version = "0.27.9", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
+hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
 http = "1.4.2"
-http-body = "1.1.0"
-http-body-util = "0.1.4"
+http-body = "1.0.1"
+http-body-util = "0.1.3"
 minlz = "1.2.3"
-reqwest = "0.13.4"
+reqwest = { version = "0.13.4", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
 rustfs-kafka-async = { version = "1.2.0" }
-socket2 = { version = "0.6.5" }
-tokio = { version = "1.53.1" }
-tokio-rustls = { default-features = false, version = "0.26.4" }
-tokio-stream = { version = "0.1.19" }
+socket2 = { version = "0.6.4", features = ["all"] }
+tokio = { version = "1.52.3", features = ["fs", "rt-multi-thread"] }
+tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
+tokio-stream = { version = "0.1.18" }
 tokio-test = "0.4.5"
-tokio-util = { version = "0.7.19" }
-tonic = { version = "0.14.6" }
+tokio-util = { version = "0.7.18", features = ["io", "compat"] }
+tonic = { version = "0.14.6", features = ["gzip", "deflate"] }
 tonic-prost = { version = "0.14.6" }
 tonic-prost-build = { version = "0.14.6" }
-tower = { version = "0.5.3" }
-tower-http = { version = "0.7.0" }
+tower = { version = "0.5.3", features = ["timeout"] }
+tower-http = { version = "0.7.0", features = ["cors"] }
 
 # Serialization and Data Formats
 apache-avro = "0.21.0"
-bytes = { version = "1.12.1" }
+bytes = { version = "1.12.1", features = ["serde"] }
 bytesize = "2.4.2"
 byteorder = "1.5.0"
 flatbuffers = "25.12.19"
@@ -181,8 +177,8 @@ prost = "0.14.4"
 quick-xml = "0.41.0"
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
-serde = { version = "1.0.229" }
-serde_json = { version = "1.0.151" }
+serde = { version = "1.0.228", features = ["derive"] }
+serde_json = { version = "1.0.150", features = ["raw_value"] }
 serde_urlencoded = "0.7.1"
 
 # Cryptography and Security
@@ -190,75 +186,73 @@ serde_urlencoded = "0.7.1"
 # matching stable releases are not available yet, while previous stable lines
 # have incompatible APIs. Keep them exact-pinned and monitor upstream for stable
 # releases.
-aes-gcm = { version = "=0.11.0" }
+aes-gcm = { version = "=0.11.0", features = ["rand_core"] }
 argon2 = { version = "=0.6.0-rc.8" }
 blake2 = "=0.11.0-rc.6"
 chacha20poly1305 = { version = "=0.11.0" }
 crc-fast = "1.10.0"
 hmac = { version = "0.13.0" }
-jsonwebtoken = { version = "11.0.0" }
-openidconnect = { default-features = false, version = "4.0" }
+jsonwebtoken = { version = "10.4.0", features = ["aws_lc_rs"] }
+openidconnect = { version = "4.0", default-features = false, features = ["accept-rfc3339-timestamps"] }
 pbkdf2 = "0.13.0"
 rsa = { version = "=0.10.0-rc.18" }
-rustls = { default-features = false, version = "0.23.42" }
+rustls = { version = "0.23.41", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
 rustls-native-certs = "0.8"
-rustls-pki-types = "1.15.1"
+rustls-pki-types = "1.15.0"
 sha1 = "0.11.0"
 sha2 = "0.11.0"
 subtle = "2.6"
-zeroize = { version = "1.9.0" }
+zeroize = { version = "1.9.0", features = ["derive"] }
 
 # Time and Date
-chrono = { version = "0.4.45" }
+chrono = { version = "0.4.45", features = ["serde"] }
 humantime = "2.4.0"
-jiff = { version = "0.2.35" }
-time = { version = "0.3.54" }
+jiff = { version = "0.2.32", features = ["serde"] }
+time = { version = "0.3.53", features = ["std", "parsing", "formatting", "macros", "serde"] }
 
 # Database
-deadpool-postgres = { version = "0.14" }
-tokio-postgres = { default-features = false, version = "0.7.18" }
+deadpool-postgres = { version = "0.14", features = ["rt_tokio_1"] }
+tokio-postgres = { version = "0.7.18", default-features = false, features = ["runtime", "with-serde_json-1"] }
 tokio-postgres-rustls = "0.14.0"
 
 # Utilities and Tools
-anyhow = "1.0.104"
+anyhow = "1.0.103"
 arc-swap = "1.9.2"
-astral-tokio-tar = "0.6.4"
+astral-tokio-tar = "0.6.3"
 atoi = "3.1.0"
 atomic_enum = "0.3.0"
-aws-config = { version = "1.10.1" }
+aws-config = { version = "1.9.0" }
 aws-credential-types = { version = "1.3.0" }
-aws-sdk-s3 = { default-features = false, version = "1.140.0" }
-aws-sdk-sts = { default-features = false, version = "1.110.0" }
-aws-smithy-http-client = { default-features = false, version = "1.2.0" }
-aws-smithy-runtime-api = { version = "1.14.0" }
+aws-sdk-s3 = { version = "1.138.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+aws-smithy-http-client = { version = "1.2.0", default-features = false, features = ["default-client", "rustls-aws-lc"] }
+aws-smithy-runtime-api = { version = "1.13.0", features = ["http-1x"] }
 aws-smithy-types = { version = "1.6.1" }
-base64 = "0.23.0"
+base64 = "0.22.1"
 base64-simd = "0.8.0"
 brotli = "8.0.4"
-clap = { version = "4.6.4" }
-const-str = { version = "1.1.0" }
+clap = { version = "4.6.1", features = ["derive", "env"] }
+const-str = { version = "1.1.0", features = ["std", "proc"] }
 convert_case = "0.11.0"
-criterion = { version = "0.8" }
+criterion = { version = "0.8", features = ["html_reports"] }
 crossbeam-queue = "0.3.13"
 crossbeam-channel = "0.5.16"
 crossbeam-deque = "0.8.7"
 crossbeam-utils = "0.8.22"
-datafusion = { default-features = false, git = "https://github.com/apache/datafusion.git", rev = "dae03ee062b2abf986de8df12ea82fb1578a2d99" }
-#datafusion = { default-features = false, version = "54.1.0" }
+datafusion = { git = "https://github.com/apache/datafusion.git", rev = "dae03ee062b2abf986de8df12ea82fb1578a2d99" }
 derive_builder = "0.20.2"
-enumset = "1.1.14"
+enumset = "1.1.13"
 faster-hex = "0.10.0"
 flate2 = "1.1.9"
-glob = "0.3.4"
-google-cloud-storage = "1.16.0"
-google-cloud-auth = "1.14.0"
-hashbrown = { version = "0.17.1" }
+glob = "0.3.3"
+google-cloud-storage = "1.15.0"
+google-cloud-auth = "1.13.0"
+hashbrown = { version = "0.17.1", features = ["serde", "rayon"] }
 hex = "0.4.3"
 hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
-ipnetwork = { version = "0.21.1" }
+ipnetwork = { version = "0.21.1", features = ["serde"] }
 lazy_static = "1.5.0"
-libc = "0.2.189"
+libc = "0.2.186"
 libsystemd = "0.7.2"
 local-ip-address = "0.6.13"
 memmap2 = "0.9.11"
@@ -267,7 +261,7 @@ matchit = "0.9.2"
 md-5 = "0.11.0"
 md5 = "0.8.1"
 mime_guess = "2.0.5"
-moka = { version = "0.12.15" }
+moka = { version = "0.12.15", features = ["future"] }
 netif = "0.1.6"
 num_cpus = { version = "1.17.0" }
 nvml-wrapper = "0.12.1"
@@ -277,45 +271,45 @@ path-clean = "1.0.1"
 percent-encoding = "2.3.2"
 pin-project-lite = "0.2.17"
 pretty_assertions = "1.4.1"
-rand = { version = "0.10.2" }
+rand = { version = "0.10.2", features = ["serde"] }
 ratelimit = "0.10.1"
 rayon = "1.12.0"
-reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.2" }
+reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "7.0.1", features = ["simd-accel"] }
+#reed-solomon-erasure = { version = "6.0", features = ["simd-accel"], git = "https://github.com/houseme/reed-solomon-erasure",rev = "main" }
 reed-solomon-simd = "3.1.0"
-regex = { version = "1.13.1" }
-rumqttc = { package = "rumqttc-next", version = "0.33.3" }
-redis = { version = "1.4.1" }
-rustix = { version = "1.1.4" }
+regex = { version = "1.13.0" }
+rumqttc = { package = "rumqttc-next", version = "0.33.2", features = ["websocket"] }
+redis = { version = "1.3.0", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
+rustix = { version = "1.1.4", features = ["fs"] }
 rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }
-s3s = { git = "https://github.com/cxymds/s3s.git", rev = "fe3941d91fa1c69956f209a9145995c9f0235bff" }
-serial_test = "4.0.1"
-shadow-rs = { default-features = false, version = "2.0.0" }
+s3s = { version = "0.14.1", features = ["minio"] }
+serial_test = "3.5.0"
+shadow-rs = { version = "2.0.0", default-features = false }
 siphasher = "1.0.3"
-smallvec = { version = "1.15.2" }
+smallvec = { version = "1.15.2", features = ["serde"] }
 smartstring = "1.0.1"
-snap = "1.1.2"
-starshard = { version = "2.2.2" }
-strum = { version = "0.28.0" }
+snap = "1.1.1"
+starshard = { version = "2.2.1", features = ["rayon", "async", "serde"] }
+strum = { version = "0.28.0", features = ["derive"] }
 sysinfo = "0.39.6"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
-thiserror = "2.0.19"
+thiserror = "2.0.18"
 tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-error = "0.2.1"
 tracing-opentelemetry = { version = "0.33" }
-tracing-subscriber = { version = "0.3.23" }
+tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.24.0" }
+uuid = { version = "1.23.5", features = ["v4", "fast-rng", "macro-diagnostics"] }
 vaultrs = { version = "0.8.0" }
-tar = "0.4.46"
 walkdir = "2.5.0"
 windows = { version = "0.62.2" }
-xxhash-rust = { version = "0.8.18" }
+xxhash-rust = { version = "0.8.16", features = ["xxh64", "xxh3"] }
 zip = "8.6.0"
 zstd = "0.13.3"
 
@@ -323,30 +317,29 @@ zstd = "0.13.3"
 metrics = "0.24.6"
 dial9-tokio-telemetry = "0.3"
 opentelemetry = { version = "0.32.0" }
-opentelemetry-appender-tracing = { version = "0.32.0" }
-opentelemetry-otlp = { version = "0.32.0" }
-opentelemetry-proto = { version = "0.32.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
-opentelemetry_sdk = { version = "0.32.1" }
-opentelemetry-semantic-conventions = { version = "0.32.1" }
+opentelemetry-appender-tracing = { version = "0.32.0", features = ["experimental_span_attributes", "experimental_metadata_attributes"] }
+opentelemetry-otlp = { version = "0.32.0", features = ["gzip-http", "reqwest-rustls"] }
+opentelemetry_sdk = { version = "0.32.1", features = ["rt-tokio"] }
+opentelemetry-semantic-conventions = { version = "0.32.1", features = ["semconv_experimental"] }
 opentelemetry-stdout = { version = "0.32.0" }
-pyroscope = { version = "2.1.1" }
+pyroscope = { version = "2.1.0", features = ["backend-pprof-rs"] }
 
 # FTP and SFTP
-libunftp = { version = "0.23.0" }
+libunftp = { version = "0.23.0", features = ["experimental"] }
 unftp-core = "0.1.0"
-suppaftp = { version = "10.0.1" }
-rcgen = { version = "0.14.8", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
-russh = { version = "0.62.4" }
+suppaftp = { version = "10.0.0", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
+rcgen = "0.14.8"
+russh = { version = "0.62.2", features = ["serde"] }
 russh-sftp = "2.3.0"
 
 # WebDAV
 dav-server = "0.11.0"
 
 # Performance Analysis and Memory Profiling
-mimalloc = "0.1.52"
-hotpath = "0.22.0"
+mimalloc = "0.1"
+hotpath = "0.21"
 # Snapshot testing for output format regression detection
-insta = { version = "1.48" }
+insta = { version = "1.48", features = ["yaml", "json"] }
 
 [workspace.metadata.cargo-shear]
 ignored = ["rustfs"]
@@ -360,16 +353,12 @@ debug = "line-tables-only"
 
 [profile.release]
 opt-level = 3
-lto = "thin"
-codegen-units = 1
-debug = 0
-split-debuginfo = "off"
-strip = "symbols"
 
 [profile.production]
 inherits = "release"
+lto = "fat"
+codegen-units = 1
 
 [profile.profiling]
 inherits = "release"
 debug = true
-strip = "none"

Dockerfile

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM alpine:3.24.1 AS build
+FROM alpine:3.23.4 AS build
 
 ARG TARGETARCH
 ARG RELEASE=latest
@@ -70,7 +70,7 @@ RUN set -eux; \
     rm -rf rustfs.zip /build/.tmp || true
 
 
-FROM alpine:3.24.1
+FROM alpine:3.23.4
 
 ARG RELEASE=latest
 ARG BUILD_DATE
@@ -88,9 +88,7 @@ LABEL name="RustFS" \
       url="https://rustfs.com" \
       license="Apache-2.0"
 
-# Upgrade base-image packages so published images pick up security fixes
-# (e.g. openssl/libssl3 CVEs) without waiting for a new Alpine point release.
-RUN apk upgrade --no-cache && \
+RUN apk update && \
     apk add --no-cache ca-certificates coreutils curl
 
 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

Dockerfile.decommission-local

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM rust:1.97.1-trixie
+FROM rust:1.95-trixie
 
 RUN set -eux; \
     export DEBIAN_FRONTEND=noninteractive; \

Dockerfile.glibc

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM ubuntu:26.04 AS build
+FROM ubuntu:24.04 AS build
 
 ARG TARGETARCH
 ARG RELEASE=latest
@@ -76,7 +76,7 @@ RUN set -eux; \
     chmod +x /build/rustfs; \
     rm -rf rustfs.zip /build/.tmp || true
 
-FROM ubuntu:26.04
+FROM ubuntu:24.04
 
 ARG RELEASE=latest
 ARG BUILD_DATE
@@ -93,10 +93,7 @@ LABEL name="RustFS" \
       url="https://rustfs.com" \
       license="Apache-2.0"
 
-# Upgrade base-image packages so published images pick up security fixes
-# (e.g. tar/gzip/perl CVEs) without waiting for a new Ubuntu point release.
-RUN apt-get update && apt-get upgrade -y \
-    && apt-get install -y --no-install-recommends \
+RUN apt-get update && apt-get install -y --no-install-recommends \
     ca-certificates \
     curl \
     && rm -rf /var/lib/apt/lists/*

Dockerfile.source

@@ -32,7 +32,7 @@ ARG RUSTFS_BUILD_FEATURES=""
 # -----------------------------
 # Build stage
 # -----------------------------
-FROM rust:1.97.1-trixie AS builder
+FROM rust:1.95-trixie AS builder
 
 # Re-declare args after FROM
 ARG TARGETPLATFORM
@@ -208,7 +208,7 @@ CMD ["cargo", "run", "--bin", "rustfs", "--"]
 # -----------------------------
 # Runtime stage (Ubuntu minimal)
 # -----------------------------
-FROM ubuntu:26.04
+FROM ubuntu:24.04
 
 ARG BUILD_DATE
 ARG VCS_REF
@@ -223,7 +223,6 @@ LABEL name="RustFS (dev-local)" \
 RUN set -eux; \
     export DEBIAN_FRONTEND=noninteractive; \
     apt-get update; \
-    apt-get upgrade -y; \
     apt-get install -y --no-install-recommends \
     ca-certificates \
     curl \

README.md

@@ -116,7 +116,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.11
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.8
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -163,7 +163,6 @@ docker run -d --name rustfs -p 9000:9000 \
   -e RUSTFS_NOTIFY_WEBHOOK_ENABLE_PRIMARY=on \
   -e RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY=http://<host-ip>:3020/webhook \
   -e RUSTFS_NOTIFY_WEBHOOK_QUEUE_DIR_PRIMARY=/tmp/rustfs-events \
-  -e RUSTFS_OUTBOUND_ALLOW_ORIGINS=http://<host-ip>:3020 \
   rustfs/rustfs:latest
 ```
 
@@ -172,11 +171,6 @@ Notes:
 - For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
 - If queue dir is omitted, default is `/opt/rustfs/events`; ensure it is writable by the container runtime user.
 - `RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
-- Since `1.0.0-beta.11`, webhook endpoints on private or container networks
-  (`Docker Compose service names`, `host.docker.internal`, RFC 1918 addresses) are
-  blocked unless their exact `scheme://host:port` origin is listed in
-  `RUSTFS_OUTBOUND_ALLOW_ORIGINS` (the origin only, without the path). See
-  [Outbound Connection Policy](docs/operations/outbound-connection-policy.md).
 
 **NOTE**: We recommend reviewing the `docker-compose.yml` file before running. It defines several services including Grafana, Prometheus, and Jaeger, which are helpful for RustFS observability. If you wish to start Redis or Nginx containers, you can specify the corresponding profiles.
 
@@ -224,10 +218,7 @@ For scanner pacing, cycle budgets, bitrot cadence, lifecycle transition status,
 and single-node single-disk idle CPU tuning, see
 [Scanner Runtime Controls](docs/operations/scanner-runtime-controls.md). For
 repeatable scanner-pressure validation, see
-[Scanner Benchmark Runbook](docs/operations/scanner-benchmark-runbook.md). For
-drive timeout knobs on slow storage — including the walk stall budget that
-governs `ListObjects` on large prefixes — see
-[Drive Timeout Tuning](docs/operations/drive-timeout-tuning.md).
+[Scanner Benchmark Runbook](docs/operations/scanner-benchmark-runbook.md).
 
 ### 5\. Nix Flake (Option 5)
 
@@ -268,7 +259,7 @@ rustfs --help
 2. **Create a Bucket**: Use the console to create a new bucket for your objects.
 3. **Upload Objects**: You can upload files directly through the console or use S3-compatible APIs/clients to interact with your RustFS instance.
 
-**NOTE**: To access the RustFS instance via `https`, please refer to the [TLS Configuration Docs](https://docs.rustfs.com/integration/tls-configured).
+**NOTE**: To access the RustFS instance via `https`, please refer to the [TLS Configuration Docs](https://docs.rustfs.com/integration/tls-configured.html).
 
 ### OIDC Roles Claim (Microsoft Entra ID)
 
@@ -344,18 +335,12 @@ If you have any questions or need assistance:
 RustFS is a community-driven project, and we appreciate all contributions. Check out the [Contributors](https://github.com/rustfs/rustfs/graphs/contributors) page to see the amazing people who have helped make RustFS better.
 
 <a href="https://github.com/rustfs/rustfs/graphs/contributors">
-<picture>
-  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rustfs/rustfs/star-history/contributors-dark.svg">
-  <img src="https://raw.githubusercontent.com/rustfs/rustfs/star-history/contributors-light.svg" alt="RustFS contributors">
-</picture>
+<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" alt="Contributors" />
 </a>
 
 ## Star History
 
-<picture>
-  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rustfs/rustfs/star-history/star-history-dark.svg">
-  <img src="https://raw.githubusercontent.com/rustfs/rustfs/star-history/star-history-light.svg" alt="RustFS star history chart">
-</picture>
+[![Star History Chart](https://api.star-history.com/svg?repos=rustfs/rustfs&type=date&legend=top-left)](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
 
 ## License
 

README_ZH.md

@@ -113,7 +113,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # 使用指定版本运行
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.11
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.8
 ```
 
 如果您通过绑定挂载启用 TLS 证书目录，也请用同样方式准备该目录：
@@ -214,7 +214,7 @@ rustfs --help
 2. **创建存储桶**: 使用控制台为您​​的对象创建一个新的存储桶 (Bucket)。
 3. **上传对象**: 您可以直接通过控制台上传文件，或使用 S3 兼容的 API/客户端与您的 RustFS 实例进行交互。
 
-**注意**: 如果您希望通过 `https` 访问 RustFS 实例，请参考 [TLS 配置文档](https://docs.rustfs.com/integration/tls-configured)。
+**注意**: 如果您希望通过 `https` 访问 RustFS 实例，请参考 [TLS 配置文档](https://docs.rustfs.com/integration/tls-configured.html)。
 
 ## 文档
 
@@ -247,18 +247,12 @@ rustfs --help
 RustFS 是一个社区驱动的项目，我们感谢所有的贡献。请查看 [贡献者](https://github.com/rustfs/rustfs/graphs/contributors) 页面，看看那些让 RustFS 变得更好的了不起的人们。
 
 <a href="https://github.com/rustfs/rustfs/graphs/contributors">
-<picture>
-  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rustfs/rustfs/star-history/contributors-dark.svg">
-  <img src="https://raw.githubusercontent.com/rustfs/rustfs/star-history/contributors-light.svg" alt="RustFS 贡献者">
-</picture>
+<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" alt="Contributors" />
 </a>
 
 ## Star 历史
 
-<picture>
-  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rustfs/rustfs/star-history/star-history-dark.svg">
-  <img src="https://raw.githubusercontent.com/rustfs/rustfs/star-history/star-history-light.svg" alt="RustFS Star 历史图表">
-</picture>
+[![Star History Chart](https://api.star-history.com/svg?repos=rustfs/rustfs&type=date&legend=top-left)](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
 
 ## 许可证
 

build-rustfs.sh

@@ -217,7 +217,7 @@ setup_rust_environment() {
     # Set up environment variables for musl targets
     if [[ "$PLATFORM" == *"musl"* ]]; then
         print_message $YELLOW "Setting up environment for musl target..."
-        export RUSTFLAGS="${RUSTFLAGS:+$RUSTFLAGS }-C target-feature=-crt-static"
+        export RUSTFLAGS="--cfg tokio_unstable -C target-feature=-crt-static"
 
         # For cargo-zigbuild, set up additional environment variables
         if command -v cargo-zigbuild &> /dev/null; then
@@ -434,7 +434,7 @@ build_binary() {
         fi
     else
         # Native compilation
-        build_cmd="RUSTFLAGS='${RUSTFLAGS:+$RUSTFLAGS }-Clink-arg=-lm' cargo build"
+        build_cmd="RUSTFLAGS='--cfg tokio_unstable -Clink-arg=-lm' cargo build"
     fi
 
     if [ "$BUILD_TYPE" = "release" ]; then

crates/audit/Cargo.toml

@@ -27,17 +27,17 @@ categories = ["web-programming", "development-tools", "asynchronous", "api-bindi
 
 [dependencies]
 rustfs-targets = { workspace = true }
-rustfs-config = { workspace = true, features = ["audit", "server-config-model"] }
+rustfs-config = { workspace = true, features = ["audit", "constants", "server-config-model"] }
 rustfs-s3-types = { workspace = true }
-chrono = { workspace = true, features = ["serde"] }
-const-str = { workspace = true, features = ["std", "proc"] }
+chrono = { workspace = true }
+const-str = { workspace = true }
 futures = { workspace = true }
-hashbrown = { workspace = true, features = ["serde", "rayon"] }
+hashbrown = { workspace = true }
 metrics = { workspace = true }
-serde = { workspace = true, features = ["derive"] }
-serde_json = { workspace = true, features = ["raw_value"] }
+serde = { workspace = true }
+serde_json = { workspace = true }
 thiserror = { workspace = true }
-tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "time", "macros"] }
+tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "rt", "time", "macros"] }
 tracing = { workspace = true, features = ["std", "attributes"] }
 
 [dev-dependencies]

crates/audit/src/pipeline.rs

@@ -32,7 +32,6 @@ const EVENT_AUDIT_BATCH_DISPATCH_COMPLETED: &str = "audit_batch_dispatch_complet
 const EVENT_AUDIT_TARGET_STATE_CHANGED: &str = "audit_target_state_changed";
 const EVENT_AUDIT_REPLAY_DELIVERED: &str = "audit_replay_delivered";
 const EVENT_AUDIT_REPLAY_RETRY_SCHEDULED: &str = "audit_replay_retry_scheduled";
-const EVENT_AUDIT_REPLAY_RETRY_EXHAUSTED: &str = "audit_replay_retry_exhausted";
 const EVENT_AUDIT_REPLAY_DROPPED: &str = "audit_replay_dropped";
 const EVENT_AUDIT_REPLAY_STREAM_STATUS: &str = "audit_replay_stream_status";
 
@@ -282,7 +281,6 @@ impl AuditPipeline {
                 let delivery = target.delivery_snapshot();
                 AuditTargetMetricSnapshot {
                     failed_messages: delivery.failed_messages,
-                    failed_store_length: delivery.failed_store_length,
                     queue_length: delivery.queue_length,
                     target_id: target.id().to_string(),
                     total_messages: delivery.total_messages,
@@ -292,8 +290,8 @@ impl AuditPipeline {
     }
 
     pub async fn snapshot_target_health(&self) -> Vec<rustfs_targets::RuntimeTargetHealthSnapshot> {
-        let targets = self.registry.lock().await.list_target_values();
-        rustfs_targets::health_snapshots_for_targets(targets).await
+        let registry = self.registry.lock().await;
+        registry.runtime_manager().health_snapshots().await
     }
 }
 
@@ -465,16 +463,18 @@ impl AuditRuntimeFacade {
                             target.record_final_failure();
                             observability::record_target_failure();
                         }
-                        ReplayEvent::RetryExhausted { detail, key, target } => {
+                        ReplayEvent::RetryExhausted { key, target } => {
                             warn!(
-                                event = EVENT_AUDIT_REPLAY_RETRY_EXHAUSTED,
+                                event = EVENT_AUDIT_REPLAY_DROPPED,
                                 component = LOG_COMPONENT_AUDIT,
                                 subsystem = LOG_SUBSYSTEM_PIPELINE,
                                 target_id = %target.id(),
                                 replay_key = %key,
-                                error = %detail,
-                                "audit replay retry budget exhausted, entry stays queued and retries"
+                                reason = "retry_exhausted",
+                                "audit replay delivery"
                             );
+                            target.record_final_failure();
+                            observability::record_target_failure();
                         }
                         ReplayEvent::UnreadableEntry { key, error, target } => {
                             warn!(
@@ -570,7 +570,7 @@ mod tests {
     use rustfs_targets::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta};
     use rustfs_targets::{StoreError, Target, TargetError};
     use std::sync::Arc;
-    use tokio::sync::{Mutex, Notify};
+    use tokio::sync::Mutex;
 
     /// Mock target whose `save()` outcome is fixed at construction so tests can
     /// force full-success / full-failure / partial-failure fan-outs.
@@ -578,7 +578,6 @@ mod tests {
     struct MockTarget {
         id: TargetID,
         fail: bool,
-        health_gate: Option<(Arc<Notify>, Arc<Notify>)>,
     }
 
     impl MockTarget {
@@ -586,14 +585,8 @@ mod tests {
             Self {
                 id: TargetID::new(id.to_string(), "webhook".to_string()),
                 fail,
-                health_gate: None,
             }
         }
-
-        fn with_health_gate(mut self, started: Arc<Notify>, release: Arc<Notify>) -> Self {
-            self.health_gate = Some((started, release));
-            self
-        }
     }
 
     #[async_trait]
@@ -606,10 +599,6 @@ mod tests {
         }
 
         async fn is_active(&self) -> Result<bool, TargetError> {
-            if let Some((started, release)) = &self.health_gate {
-                started.notify_one();
-                release.notified().await;
-            }
             Ok(true)
         }
 
@@ -684,24 +673,6 @@ mod tests {
         pipeline.dispatch(entry()).await.expect("no targets should return Ok");
     }
 
-    #[tokio::test]
-    async fn health_probe_does_not_hold_the_registry_lock() {
-        let started = Arc::new(Notify::new());
-        let release = Arc::new(Notify::new());
-        let pipeline = pipeline_with(vec![MockTarget::new("blocked", false).with_health_gate(started.clone(), release.clone())]);
-        let registry = Arc::clone(&pipeline.registry);
-        let snapshot_task = tokio::spawn(async move { pipeline.snapshot_target_health().await });
-        started.notified().await;
-
-        let guard = tokio::time::timeout(std::time::Duration::from_secs(1), registry.lock())
-            .await
-            .expect("network health probe must not retain the audit registry lock");
-        drop(guard);
-        release.notify_one();
-
-        assert_eq!(snapshot_task.await.expect("snapshot task should finish").len(), 1);
-    }
-
     // backlog#962: dispatch_batch must mirror dispatch and propagate a
     // whole-batch loss instead of returning Ok.
     #[tokio::test]

crates/audit/src/system.rs

@@ -30,7 +30,6 @@ const EVENT_AUDIT_CONFIG_RELOADED: &str = "audit_config_reloaded";
 #[derive(Debug, Clone, Default, PartialEq, Eq)]
 pub struct AuditTargetMetricSnapshot {
     pub failed_messages: u64,
-    pub failed_store_length: u64,
     pub queue_length: u64,
     pub target_id: String,
     pub total_messages: u64,

crates/checksums/Cargo.toml

@@ -25,18 +25,14 @@ keywords = ["checksum-calculation", "verification", "integrity", "authenticity",
 categories = ["web-programming", "development-tools", "network-programming"]
 documentation = "https://docs.rs/rustfs-checksums/latest/rustfs_checksum/"
 
-[lints]
-workspace = true
-
 [dependencies]
-bytes = { workspace = true, features = ["serde"] }
+bytes = { workspace = true }
 crc-fast = { workspace = true }
 http = { workspace = true }
 base64-simd = { workspace = true }
 md-5 = { workspace = true }
 sha1 = { workspace = true }
 sha2 = { workspace = true }
-xxhash-rust = { workspace = true, features = ["xxh64", "xxh3"] }
 
 [dev-dependencies]
 pretty_assertions = { workspace = true }

crates/checksums/src/error.rs

@@ -36,7 +36,7 @@ impl fmt::Display for UnknownChecksumAlgorithmError {
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
         write!(
             f,
-            r#"unknown checksum algorithm "{}", please pass a known algorithm name ("crc32", "crc32c", "crc64nvme", "sha1", "sha256", "sha512", "xxhash3", "xxhash64", "xxhash128")"#,
+            r#"unknown checksum algorithm "{}", please pass a known algorithm name ("crc32", "crc32c", "crc64nvme", "sha1", "sha256")"#,
             self.checksum_algorithm
         )
     }

crates/checksums/src/http.rs

@@ -16,20 +16,13 @@ use crate::base64;
 use http::header::{HeaderMap, HeaderValue};
 
 use crate::Crc64Nvme;
-use crate::{
-    CRC_32_C_NAME, CRC_32_NAME, CRC_64_NVME_NAME, Checksum, Crc32, Crc32c, Md5, SHA_1_NAME, SHA_256_NAME, Sha1, Sha256, Sha512,
-    Xxhash3, Xxhash64, Xxhash128,
-};
+use crate::{CRC_32_C_NAME, CRC_32_NAME, CRC_64_NVME_NAME, Checksum, Crc32, Crc32c, Md5, SHA_1_NAME, SHA_256_NAME, Sha1, Sha256};
 
 pub const CRC_32_HEADER_NAME: &str = "x-amz-checksum-crc32";
 pub const CRC_32_C_HEADER_NAME: &str = "x-amz-checksum-crc32c";
 pub const SHA_1_HEADER_NAME: &str = "x-amz-checksum-sha1";
 pub const SHA_256_HEADER_NAME: &str = "x-amz-checksum-sha256";
 pub const CRC_64_NVME_HEADER_NAME: &str = "x-amz-checksum-crc64nvme";
-pub const SHA_512_HEADER_NAME: &str = "x-amz-checksum-sha512";
-pub const XXHASH_3_HEADER_NAME: &str = "x-amz-checksum-xxhash3";
-pub const XXHASH_64_HEADER_NAME: &str = "x-amz-checksum-xxhash64";
-pub const XXHASH_128_HEADER_NAME: &str = "x-amz-checksum-xxhash128";
 
 #[allow(dead_code)]
 pub(crate) static MD5_HEADER_NAME: &str = "content-md5";
@@ -92,30 +85,6 @@ impl HttpChecksum for Sha256 {
     }
 }
 
-impl HttpChecksum for Sha512 {
-    fn header_name(&self) -> &'static str {
-        SHA_512_HEADER_NAME
-    }
-}
-
-impl HttpChecksum for Xxhash3 {
-    fn header_name(&self) -> &'static str {
-        XXHASH_3_HEADER_NAME
-    }
-}
-
-impl HttpChecksum for Xxhash64 {
-    fn header_name(&self) -> &'static str {
-        XXHASH_64_HEADER_NAME
-    }
-}
-
-impl HttpChecksum for Xxhash128 {
-    fn header_name(&self) -> &'static str {
-        XXHASH_128_HEADER_NAME
-    }
-}
-
 impl HttpChecksum for Md5 {
     fn header_name(&self) -> &'static str {
         MD5_HEADER_NAME

crates/checksums/src/lib.rs

@@ -35,10 +35,6 @@ pub const CRC_32_C_NAME: &str = "crc32c";
 pub const CRC_64_NVME_NAME: &str = "crc64nvme";
 pub const SHA_1_NAME: &str = "sha1";
 pub const SHA_256_NAME: &str = "sha256";
-pub const SHA_512_NAME: &str = "sha512";
-pub const XXHASH_3_NAME: &str = "xxhash3";
-pub const XXHASH_64_NAME: &str = "xxhash64";
-pub const XXHASH_128_NAME: &str = "xxhash128";
 pub const MD5_NAME: &str = "md5";
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
@@ -50,10 +46,6 @@ pub enum ChecksumAlgorithm {
     Sha1,
     Sha256,
     Crc64Nvme,
-    Sha512,
-    Xxhash3,
-    Xxhash64,
-    Xxhash128,
 }
 
 impl FromStr for ChecksumAlgorithm {
@@ -70,14 +62,6 @@ impl FromStr for ChecksumAlgorithm {
             Ok(Self::Sha256)
         } else if checksum_algorithm.eq_ignore_ascii_case(CRC_64_NVME_NAME) {
             Ok(Self::Crc64Nvme)
-        } else if checksum_algorithm.eq_ignore_ascii_case(SHA_512_NAME) {
-            Ok(Self::Sha512)
-        } else if checksum_algorithm.eq_ignore_ascii_case(XXHASH_3_NAME) {
-            Ok(Self::Xxhash3)
-        } else if checksum_algorithm.eq_ignore_ascii_case(XXHASH_64_NAME) {
-            Ok(Self::Xxhash64)
-        } else if checksum_algorithm.eq_ignore_ascii_case(XXHASH_128_NAME) {
-            Ok(Self::Xxhash128)
         } else {
             Err(UnknownChecksumAlgorithmError::new(checksum_algorithm))
         }
@@ -92,10 +76,6 @@ impl ChecksumAlgorithm {
             Self::Crc64Nvme => Box::<Crc64Nvme>::default(),
             Self::Sha1 => Box::<Sha1>::default(),
             Self::Sha256 => Box::<Sha256>::default(),
-            Self::Sha512 => Box::<Sha512>::default(),
-            Self::Xxhash3 => Box::<Xxhash3>::default(),
-            Self::Xxhash64 => Box::<Xxhash64>::default(),
-            Self::Xxhash128 => Box::<Xxhash128>::default(),
         }
     }
 
@@ -106,10 +86,6 @@ impl ChecksumAlgorithm {
             Self::Crc64Nvme => CRC_64_NVME_NAME,
             Self::Sha1 => SHA_1_NAME,
             Self::Sha256 => SHA_256_NAME,
-            Self::Sha512 => SHA_512_NAME,
-            Self::Xxhash3 => XXHASH_3_NAME,
-            Self::Xxhash64 => XXHASH_64_NAME,
-            Self::Xxhash128 => XXHASH_128_NAME,
         }
     }
 }
@@ -309,165 +285,6 @@ impl Checksum for Sha256 {
         Self::size()
     }
 }
-
-#[derive(Debug, Default)]
-struct Sha512 {
-    hasher: sha2::Sha512,
-}
-
-impl Sha512 {
-    fn update(&mut self, bytes: &[u8]) {
-        use sha2::Digest;
-        self.hasher.update(bytes);
-    }
-
-    fn finalize(self) -> Bytes {
-        use sha2::Digest;
-        Bytes::copy_from_slice(self.hasher.finalize().as_slice())
-    }
-
-    fn size() -> u64 {
-        use sha2::Digest;
-        sha2::Sha512::output_size() as u64
-    }
-}
-
-impl Checksum for Sha512 {
-    fn update(&mut self, bytes: &[u8]) {
-        Self::update(self, bytes);
-    }
-    fn finalize(self: Box<Self>) -> Bytes {
-        Self::finalize(*self)
-    }
-    fn size(&self) -> u64 {
-        Self::size()
-    }
-}
-
-/// XXH3 (64-bit) hasher with the canonical seed of 0.
-///
-/// The raw digest is a `u64` serialized as 8 big-endian bytes so that the value
-/// matches the server-side (`rustfs-rio`) computation for the same algorithm.
-struct Xxhash3 {
-    hasher: xxhash_rust::xxh3::Xxh3,
-}
-
-impl Default for Xxhash3 {
-    fn default() -> Self {
-        Self {
-            hasher: xxhash_rust::xxh3::Xxh3::new(),
-        }
-    }
-}
-
-impl Xxhash3 {
-    fn update(&mut self, bytes: &[u8]) {
-        self.hasher.update(bytes);
-    }
-
-    fn finalize(self) -> Bytes {
-        Bytes::copy_from_slice(self.hasher.digest().to_be_bytes().as_slice())
-    }
-
-    fn size() -> u64 {
-        8
-    }
-}
-
-impl Checksum for Xxhash3 {
-    fn update(&mut self, bytes: &[u8]) {
-        Self::update(self, bytes)
-    }
-    fn finalize(self: Box<Self>) -> Bytes {
-        Self::finalize(*self)
-    }
-    fn size(&self) -> u64 {
-        Self::size()
-    }
-}
-
-/// XXH3 (128-bit) hasher with the canonical seed of 0.
-///
-/// The raw digest is a `u128` serialized as 16 big-endian bytes.
-struct Xxhash128 {
-    hasher: xxhash_rust::xxh3::Xxh3,
-}
-
-impl Default for Xxhash128 {
-    fn default() -> Self {
-        Self {
-            hasher: xxhash_rust::xxh3::Xxh3::new(),
-        }
-    }
-}
-
-impl Xxhash128 {
-    fn update(&mut self, bytes: &[u8]) {
-        self.hasher.update(bytes);
-    }
-
-    fn finalize(self) -> Bytes {
-        Bytes::copy_from_slice(self.hasher.digest128().to_be_bytes().as_slice())
-    }
-
-    fn size() -> u64 {
-        16
-    }
-}
-
-impl Checksum for Xxhash128 {
-    fn update(&mut self, bytes: &[u8]) {
-        Self::update(self, bytes)
-    }
-    fn finalize(self: Box<Self>) -> Bytes {
-        Self::finalize(*self)
-    }
-    fn size(&self) -> u64 {
-        Self::size()
-    }
-}
-
-/// XXH64 hasher with the canonical seed of 0.
-///
-/// The raw digest is a `u64` serialized as 8 big-endian bytes.
-struct Xxhash64 {
-    hasher: xxhash_rust::xxh64::Xxh64,
-}
-
-impl Default for Xxhash64 {
-    fn default() -> Self {
-        Self {
-            hasher: xxhash_rust::xxh64::Xxh64::new(0),
-        }
-    }
-}
-
-impl Xxhash64 {
-    fn update(&mut self, bytes: &[u8]) {
-        self.hasher.update(bytes);
-    }
-
-    fn finalize(self) -> Bytes {
-        Bytes::copy_from_slice(self.hasher.digest().to_be_bytes().as_slice())
-    }
-
-    fn size() -> u64 {
-        8
-    }
-}
-
-impl Checksum for Xxhash64 {
-    fn update(&mut self, bytes: &[u8]) {
-        Self::update(self, bytes)
-    }
-    fn finalize(self: Box<Self>) -> Bytes {
-        Self::finalize(*self)
-    }
-    fn size(&self) -> u64 {
-        Self::size()
-    }
-}
-
 #[allow(dead_code)]
 #[derive(Debug, Default)]
 struct Md5 {
@@ -638,97 +455,4 @@ mod tests {
         let error = "MD5".parse::<ChecksumAlgorithm>().expect_err("md5 should not parse");
         assert_eq!("MD5", error.checksum_algorithm());
     }
-
-    #[test]
-    fn test_additional_algorithms_parse_and_round_trip() {
-        // The AWS 2026-04 additional checksum algorithms must be recognised
-        // (case-insensitively) and round-trip through as_str().
-        for (name, expected) in [
-            ("sha512", ChecksumAlgorithm::Sha512),
-            ("SHA512", ChecksumAlgorithm::Sha512),
-            ("xxhash3", ChecksumAlgorithm::Xxhash3),
-            ("XXHASH3", ChecksumAlgorithm::Xxhash3),
-            ("xxhash64", ChecksumAlgorithm::Xxhash64),
-            ("xxhash128", ChecksumAlgorithm::Xxhash128),
-        ] {
-            let parsed = name.parse::<ChecksumAlgorithm>().expect("algorithm should parse");
-            assert_eq!(parsed, expected);
-            assert_eq!(expected.as_str().parse::<ChecksumAlgorithm>().unwrap(), expected);
-        }
-    }
-
-    #[test]
-    fn test_unknown_algorithm_never_panics_and_fails_closed() {
-        // Fail-closed contract: an unknown or garbage algorithm name must return
-        // an error instead of panicking or silently substituting another hasher.
-        for name in ["", "xxhash", "sha3", "crc16", "not-a-real-algo", "🦀"] {
-            assert!(name.parse::<ChecksumAlgorithm>().is_err(), "unknown algorithm {name:?} must fail closed");
-        }
-    }
-
-    #[test]
-    fn test_sha512_matches_direct_computation() {
-        use crate::Sha512;
-        use crate::http::SHA_512_HEADER_NAME;
-        use sha2::{Digest, Sha512 as Sha512Ref};
-
-        let mut checksum = Sha512::default();
-        checksum.update(TEST_DATA.as_bytes());
-        let header = Box::new(checksum).headers();
-        let encoded = header.get(SHA_512_HEADER_NAME).expect("sha512 header present");
-        let got = base64_encoded_checksum_to_hex_string(encoded);
-
-        let mut reference = Sha512Ref::new();
-        reference.update(TEST_DATA.as_bytes());
-        let expected = reference.finalize().iter().fold(String::from("0x"), |mut acc, b| {
-            write!(acc, "{b:02X?}").unwrap();
-            acc
-        });
-        assert_eq!(got, expected);
-    }
-
-    #[test]
-    fn test_xxhash3_matches_direct_computation_big_endian_seed0() {
-        use crate::Xxhash3;
-        use xxhash_rust::xxh3::Xxh3;
-
-        let mut checksum = Xxhash3::default();
-        checksum.update(TEST_DATA.as_bytes());
-        let raw = Box::new(checksum).finalize();
-
-        let mut reference = Xxh3::new();
-        reference.update(TEST_DATA.as_bytes());
-        assert_eq!(raw.len(), 8);
-        assert_eq!(&raw[..], reference.digest().to_be_bytes().as_slice());
-    }
-
-    #[test]
-    fn test_xxhash128_matches_direct_computation_big_endian_seed0() {
-        use crate::Xxhash128;
-        use xxhash_rust::xxh3::Xxh3;
-
-        let mut checksum = Xxhash128::default();
-        checksum.update(TEST_DATA.as_bytes());
-        let raw = Box::new(checksum).finalize();
-
-        let mut reference = Xxh3::new();
-        reference.update(TEST_DATA.as_bytes());
-        assert_eq!(raw.len(), 16);
-        assert_eq!(&raw[..], reference.digest128().to_be_bytes().as_slice());
-    }
-
-    #[test]
-    fn test_xxhash64_matches_direct_computation_big_endian_seed0() {
-        use crate::Xxhash64;
-        use xxhash_rust::xxh64::Xxh64;
-
-        let mut checksum = Xxhash64::default();
-        checksum.update(TEST_DATA.as_bytes());
-        let raw = Box::new(checksum).finalize();
-
-        let mut reference = Xxh64::new(0);
-        reference.update(TEST_DATA.as_bytes());
-        assert_eq!(raw.len(), 8);
-        assert_eq!(&raw[..], reference.digest().to_be_bytes().as_slice());
-    }
 }

crates/common/Cargo.toml

@@ -28,14 +28,14 @@ categories = ["web-programming", "development-tools", "data-structures"]
 workspace = true
 
 [dependencies]
-tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
-tonic = { workspace = true, features = ["gzip", "deflate"] }
-uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
-chrono = { workspace = true, features = ["serde"] }
+tokio = { workspace = true }
+tonic = { workspace = true }
+uuid = { workspace = true }
+chrono = { workspace = true }
 metrics = { workspace = true }
-serde = { workspace = true, features = ["derive"] }
+serde = { workspace = true }
 rmp-serde = { workspace = true }
-s3s = { workspace = true, features = ["minio"] }
+s3s = { workspace = true }
 tracing = { workspace = true }
 
 [lib]

crates/common/src/globals.rs

@@ -54,15 +54,6 @@ pub async fn get_global_local_node_name() -> String {
     GLOBAL_LOCAL_NODE_NAME.read().await.clone()
 }
 
-/// Read the local node name without waiting for initialization or a writer.
-pub fn try_get_global_local_node_name() -> Option<String> {
-    GLOBAL_LOCAL_NODE_NAME
-        .try_read()
-        .ok()
-        .map(|name| name.clone())
-        .filter(|name| !name.is_empty())
-}
-
 /// Set the global RustFS initialization time to the current UTC time.
 pub async fn set_global_init_time_now() {
     let now = Utc::now();

crates/common/src/heal_channel.rs

@@ -243,19 +243,6 @@ pub enum HealAdmissionResult {
     Dropped(HealAdmissionDropReason),
 }
 
-/// Admission decision together with the canonical task identifier.
-///
-/// A merged request must return the identifier of the task that already owns
-/// the work instead of exposing the discarded request identifier as a new
-/// client token.
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub struct HealAdmissionReceipt {
-    /// Admission decision for the submitted request.
-    pub result: HealAdmissionResult,
-    /// Canonical identifier of the accepted or merged task.
-    pub task_id: String,
-}
-
 impl HealAdmissionResult {
     pub fn result_label(self) -> &'static str {
         match self {
@@ -395,25 +382,8 @@ pub type HealChannelSender = mpsc::UnboundedSender<HealChannelCommand>;
 /// Heal channel receiver
 pub type HealChannelReceiver = mpsc::UnboundedReceiver<HealChannelCommand>;
 
-/// Canonical-receipt start command kept separate from the legacy public enum.
-#[derive(Debug)]
-pub struct HealReceiptCommand {
-    /// Heal request to admit.
-    pub request: HealChannelRequest,
-    /// Completion channel for the admission receipt.
-    pub response_tx: oneshot::Sender<Result<HealAdmissionReceipt, String>>,
-}
-
-/// Canonical-receipt command receiver.
-pub type HealReceiptReceiver = mpsc::UnboundedReceiver<HealReceiptCommand>;
-
-struct HealChannelSenders {
-    command: HealChannelSender,
-    receipt: mpsc::UnboundedSender<HealReceiptCommand>,
-}
-
 /// Global heal channel sender
-static GLOBAL_HEAL_CHANNEL_SENDERS: OnceLock<HealChannelSenders> = OnceLock::new();
+static GLOBAL_HEAL_CHANNEL_SENDER: OnceLock<HealChannelSender> = OnceLock::new();
 
 type HealResponseSender = broadcast::Sender<HealChannelResponse>;
 
@@ -422,24 +392,17 @@ static GLOBAL_HEAL_RESPONSE_SENDER: OnceLock<HealResponseSender> = OnceLock::new
 
 /// Initialize global heal channel
 pub fn init_heal_channel() -> Result<HealChannelReceiver, &'static str> {
-    let (receiver, receipt_receiver) = init_heal_channels()?;
-    drop(receipt_receiver);
-    Ok(receiver)
-}
-
-/// Initialize the legacy command and canonical-receipt channels atomically.
-pub fn init_heal_channels() -> Result<(HealChannelReceiver, HealReceiptReceiver), &'static str> {
-    let (command, command_receiver) = mpsc::unbounded_channel();
-    let (receipt, receipt_receiver) = mpsc::unbounded_channel();
-    GLOBAL_HEAL_CHANNEL_SENDERS
-        .set(HealChannelSenders { command, receipt })
-        .map_err(|_| "Heal channel sender already initialized")?;
-    Ok((command_receiver, receipt_receiver))
+    let (tx, rx) = mpsc::unbounded_channel();
+    if GLOBAL_HEAL_CHANNEL_SENDER.set(tx).is_ok() {
+        Ok(rx)
+    } else {
+        Err("Heal channel sender already initialized")
+    }
 }
 
 /// Get global heal channel sender
 pub fn get_heal_channel_sender() -> Option<&'static HealChannelSender> {
-    GLOBAL_HEAL_CHANNEL_SENDERS.get().map(|senders| &senders.command)
+    GLOBAL_HEAL_CHANNEL_SENDER.get()
 }
 
 /// Send heal command through global channel
@@ -473,21 +436,6 @@ pub fn subscribe_heal_responses() -> broadcast::Receiver<HealChannelResponse> {
     heal_response_sender().subscribe()
 }
 
-/// Send heal start request and wait for structured admission feedback.
-pub async fn send_heal_request_with_receipt(request: HealChannelRequest) -> Result<HealAdmissionReceipt, String> {
-    let (response_tx, response_rx) = oneshot::channel();
-    let senders = GLOBAL_HEAL_CHANNEL_SENDERS
-        .get()
-        .ok_or_else(|| "Heal channel not initialized".to_string())?;
-    senders
-        .receipt
-        .send(HealReceiptCommand { request, response_tx })
-        .map_err(|err| format!("Failed to send heal receipt command: {err}"))?;
-    response_rx
-        .await
-        .map_err(|e| format!("Failed to receive heal admission response: {e}"))?
-}
-
 /// Send heal start request and wait for structured admission feedback.
 pub async fn send_heal_request_with_admission(request: HealChannelRequest) -> Result<HealAdmissionResult, String> {
     let (response_tx, response_rx) = oneshot::channel();

crates/common/src/metrics.rs

@@ -768,7 +768,6 @@ pub struct Metrics {
     last_scan_cycle_replication_checks: AtomicU64,
     last_scan_cycle_usage_saves: AtomicU64,
     failed_scan_cycles: AtomicU64,
-    superseded_scan_cycles: AtomicU64,
     partial_scan_cycles_unknown: AtomicU64,
     partial_scan_cycles_runtime: AtomicU64,
     partial_scan_cycles_objects: AtomicU64,
@@ -786,9 +785,6 @@ pub struct Metrics {
     scanner_expiry_queue_missed: AtomicU64,
     scanner_expiry_queued_total: AtomicU64,
     scanner_expiry_missed_total: AtomicU64,
-    scanner_expiry_blocked_total: AtomicU64,
-    scanner_expiry_not_enqueued_total: AtomicU64,
-    scanner_expiry_delete_failed_total: AtomicU64,
     scanner_transition_queue_capacity: AtomicU64,
     scanner_transition_queued: AtomicU64,
     scanner_transition_active: AtomicU64,
@@ -837,12 +833,10 @@ const SCAN_CYCLE_RESULT_UNKNOWN: u8 = 0;
 const SCAN_CYCLE_RESULT_SUCCESS: u8 = 1;
 const SCAN_CYCLE_RESULT_ERROR: u8 = 2;
 const SCAN_CYCLE_RESULT_PARTIAL: u8 = 3;
-const SCAN_CYCLE_RESULT_SUPERSEDED: u8 = 4;
 const SCAN_CYCLE_RESULT_UNKNOWN_LABEL: &str = "unknown";
 const SCAN_CYCLE_RESULT_SUCCESS_LABEL: &str = "success";
 const SCAN_CYCLE_RESULT_ERROR_LABEL: &str = "error";
 const SCAN_CYCLE_RESULT_PARTIAL_LABEL: &str = "partial";
-const SCAN_CYCLE_RESULT_SUPERSEDED_LABEL: &str = "superseded";
 
 #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
 pub enum ScanCyclePartialReason {
@@ -1054,12 +1048,6 @@ pub struct ScannerLifecycleExpirySnapshot {
     pub queue_missed: u64,
     pub scanner_queued: u64,
     pub scanner_missed: u64,
-    #[serde(default)]
-    pub scanner_blocked: u64,
-    #[serde(default)]
-    pub scanner_not_enqueued: u64,
-    #[serde(default)]
-    pub delete_failed: u64,
 }
 
 #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
@@ -1220,8 +1208,6 @@ pub struct ScannerMetricsReport {
     pub last_cycle_usage_saves: u64,
     pub failed_cycles: u64,
     #[serde(default)]
-    pub superseded_cycles: u64,
-    #[serde(default)]
     pub partial_cycles_unknown: u64,
     #[serde(default)]
     pub partial_cycles_runtime: u64,
@@ -1324,7 +1310,6 @@ fn scan_cycle_result_label(result: u8) -> &'static str {
         SCAN_CYCLE_RESULT_SUCCESS => SCAN_CYCLE_RESULT_SUCCESS_LABEL,
         SCAN_CYCLE_RESULT_ERROR => SCAN_CYCLE_RESULT_ERROR_LABEL,
         SCAN_CYCLE_RESULT_PARTIAL => SCAN_CYCLE_RESULT_PARTIAL_LABEL,
-        SCAN_CYCLE_RESULT_SUPERSEDED => SCAN_CYCLE_RESULT_SUPERSEDED_LABEL,
         _ => SCAN_CYCLE_RESULT_UNKNOWN_LABEL,
     }
 }
@@ -1648,11 +1633,6 @@ pub fn emit_scan_cycle_partial_with_source(
     metrics::counter!(OTEL_SCANNER_CYCLES, "result" => SCAN_CYCLE_RESULT_PARTIAL_LABEL).increment(1);
 }
 
-pub fn emit_scan_cycle_superseded(duration: Duration) {
-    global_metrics().record_scan_cycle_superseded(duration);
-    metrics::counter!(OTEL_SCANNER_CYCLES, "result" => SCAN_CYCLE_RESULT_SUPERSEDED_LABEL).increment(1);
-}
-
 pub fn emit_scan_bucket_drive_complete(success: bool, bucket: &str, disk: &str, duration: Duration) {
     let result = if success { "success" } else { "error" };
     metrics::counter!(
@@ -1746,7 +1726,6 @@ impl Metrics {
             last_scan_cycle_replication_checks: AtomicU64::new(0),
             last_scan_cycle_usage_saves: AtomicU64::new(0),
             failed_scan_cycles: AtomicU64::new(0),
-            superseded_scan_cycles: AtomicU64::new(0),
             partial_scan_cycles_unknown: AtomicU64::new(0),
             partial_scan_cycles_runtime: AtomicU64::new(0),
             partial_scan_cycles_objects: AtomicU64::new(0),
@@ -1764,9 +1743,6 @@ impl Metrics {
             scanner_expiry_queue_missed: AtomicU64::new(0),
             scanner_expiry_queued_total: AtomicU64::new(0),
             scanner_expiry_missed_total: AtomicU64::new(0),
-            scanner_expiry_blocked_total: AtomicU64::new(0),
-            scanner_expiry_not_enqueued_total: AtomicU64::new(0),
-            scanner_expiry_delete_failed_total: AtomicU64::new(0),
             scanner_transition_queue_capacity: AtomicU64::new(0),
             scanner_transition_queued: AtomicU64::new(0),
             scanner_transition_active: AtomicU64::new(0),
@@ -1997,18 +1973,9 @@ impl Metrics {
             self.scanner_expiry_queued_total.fetch_add(count, Ordering::Relaxed);
         } else {
             self.scanner_expiry_missed_total.fetch_add(count, Ordering::Relaxed);
-            self.scanner_expiry_not_enqueued_total.fetch_add(count, Ordering::Relaxed);
         }
     }
 
-    pub fn record_scanner_expiry_blocked(&self, count: u64) {
-        self.scanner_expiry_blocked_total.fetch_add(count, Ordering::Relaxed);
-    }
-
-    pub fn record_scanner_expiry_delete_failed(&self, count: u64) {
-        self.scanner_expiry_delete_failed_total.fetch_add(count, Ordering::Relaxed);
-    }
-
     pub fn record_scanner_transition_enqueue_result(&self, count: u64, queued: bool) {
         self.record_scanner_ilm_enqueue_result(count, queued);
         if queued {
@@ -2382,18 +2349,6 @@ impl Metrics {
             .store(duration_millis_saturated(duration), Ordering::Relaxed);
     }
 
-    pub fn record_scan_cycle_superseded(&self, duration: Duration) {
-        self.record_scanner_cycle_end_time();
-        self.superseded_scan_cycles.fetch_add(1, Ordering::Relaxed);
-        self.last_scan_cycle_result
-            .store(SCAN_CYCLE_RESULT_SUPERSEDED, Ordering::Relaxed);
-        self.last_scan_cycle_partial_reason
-            .store(ScanCyclePartialReason::Unknown as u8, Ordering::Relaxed);
-        self.last_scan_cycle_partial_source.store(0, Ordering::Relaxed);
-        self.last_scan_cycle_duration_millis
-            .store(duration_millis_saturated(duration), Ordering::Relaxed);
-    }
-
     pub fn record_scan_cycle_partial(&self, duration: Duration, reason: ScanCyclePartialReason) {
         self.record_scan_cycle_partial_with_source(duration, reason, None);
     }
@@ -2479,18 +2434,6 @@ impl Metrics {
         self.current_scan_cycle_work_active.store(false, Ordering::Relaxed);
     }
 
-    pub fn current_scan_cycle_has_unresolved_heal_work(&self) -> bool {
-        if !self.current_scan_cycle_work_active.load(Ordering::Relaxed) {
-            return false;
-        }
-
-        let source_work = self.scanner_source_work_since(&self.current_scan_cycle_source_work_start_values());
-        [ScannerWorkSource::Heal, ScannerWorkSource::Bitrot]
-            .into_iter()
-            .filter_map(|source| source_work.get(source.index()))
-            .any(|work| work.queued > 0 || work.skipped > 0 || work.failed > 0 || work.missed > 0)
-    }
-
     fn scan_cycle_work_snapshot(&self) -> ScanCycleWorkSnapshot {
         ScanCycleWorkSnapshot {
             objects_scanned: self.lifetime(Metric::ScanObject),
@@ -2847,7 +2790,6 @@ impl Metrics {
         m.last_cycle_replication_repair =
             self.scanner_replication_repair_work_counter_snapshots(&self.last_scan_cycle_replication_repair_work);
         m.failed_cycles = self.failed_scan_cycles.load(Ordering::Relaxed);
-        m.superseded_cycles = self.superseded_scan_cycles.load(Ordering::Relaxed);
         m.partial_cycles_unknown = self.partial_scan_cycles_unknown.load(Ordering::Relaxed);
         m.partial_cycles_runtime = self.partial_scan_cycles_runtime.load(Ordering::Relaxed);
         m.partial_cycles_objects = self.partial_scan_cycles_objects.load(Ordering::Relaxed);
@@ -2871,9 +2813,6 @@ impl Metrics {
             queue_missed: self.scanner_expiry_queue_missed.load(Ordering::Relaxed),
             scanner_queued: self.scanner_expiry_queued_total.load(Ordering::Relaxed),
             scanner_missed: self.scanner_expiry_missed_total.load(Ordering::Relaxed),
-            scanner_blocked: self.scanner_expiry_blocked_total.load(Ordering::Relaxed),
-            scanner_not_enqueued: self.scanner_expiry_not_enqueued_total.load(Ordering::Relaxed),
-            delete_failed: self.scanner_expiry_delete_failed_total.load(Ordering::Relaxed),
         };
         m.lifecycle_transition = ScannerLifecycleTransitionSnapshot {
             current_queue_capacity: self.scanner_transition_queue_capacity.load(Ordering::Relaxed),
@@ -2999,15 +2938,19 @@ pub type CloseDiskFn = Arc<dyn Fn() -> Pin<Box<dyn Future<Output = ()> + Send>>
 
 /// Register a new disk in the global path tracker and return two callbacks:
 /// one to update the current path and one to deregister the disk when done.
-pub async fn current_path_updater(disk: &str, initial: &str) -> (UpdateCurrentPathFn, CloseDiskFn) {
+pub fn current_path_updater(disk: &str, initial: &str) -> (UpdateCurrentPathFn, CloseDiskFn) {
     let tracker = Arc::new(CurrentPathTracker::new(initial.to_string()));
     let disk_name = disk.to_string();
 
-    global_metrics()
-        .current_paths
-        .write()
-        .await
-        .insert(disk_name.clone(), Arc::clone(&tracker));
+    let tracker_clone = Arc::clone(&tracker);
+    let disk_insert = disk_name.clone();
+    tokio::spawn(async move {
+        global_metrics()
+            .current_paths
+            .write()
+            .await
+            .insert(disk_insert, tracker_clone);
+    });
 
     let update_fn: UpdateCurrentPathFn = {
         let tracker = Arc::clone(&tracker);
@@ -3035,28 +2978,23 @@ pub async fn current_path_updater(disk: &str, initial: &str) -> (UpdateCurrentPa
 // CloseDiskGuard
 // ---------------------------------------------------------------------------
 
-pub struct CloseDiskGuard(Option<CloseDiskFn>);
+pub struct CloseDiskGuard(CloseDiskFn);
 
 impl CloseDiskGuard {
     pub fn new(close_disk: CloseDiskFn) -> Self {
-        Self(Some(close_disk))
+        Self(close_disk)
     }
 
-    pub async fn close(&mut self) {
-        let Some(close_disk) = self.0.clone() else {
-            return;
-        };
-        close_disk().await;
-        self.0 = None;
+    pub async fn close(&self) {
+        self.0().await;
     }
 }
 
 impl Drop for CloseDiskGuard {
     fn drop(&mut self) {
-        if let Some(close_disk) = self.0.take()
-            && let Ok(handle) = tokio::runtime::Handle::try_current()
-        {
-            handle.spawn(close_disk());
+        if let Ok(handle) = tokio::runtime::Handle::try_current() {
+            let close_fn = self.0.clone();
+            handle.spawn(async move { close_fn().await });
         }
         // If there is no runtime we are in a test or shutdown path; skip cleanup.
     }
@@ -3066,61 +3004,6 @@ impl Drop for CloseDiskGuard {
 mod tests {
     use super::*;
 
-    #[tokio::test]
-    async fn close_disk_guard_runs_cleanup_when_an_early_return_drops_it() {
-        let (closed_tx, closed_rx) = tokio::sync::oneshot::channel();
-        let closed_tx = Arc::new(std::sync::Mutex::new(Some(closed_tx)));
-        let close_disk: CloseDiskFn = {
-            let closed_tx = Arc::clone(&closed_tx);
-            Arc::new(move || {
-                let closed_tx = closed_tx.lock().expect("close callback lock").take();
-                Box::pin(async move {
-                    if let Some(closed_tx) = closed_tx {
-                        let _ = closed_tx.send(());
-                    }
-                })
-            })
-        };
-
-        let guard = CloseDiskGuard::new(close_disk);
-        drop(guard);
-
-        tokio::time::timeout(std::time::Duration::from_secs(1), closed_rx)
-            .await
-            .expect("drop cleanup should run")
-            .expect("drop cleanup should signal");
-    }
-
-    #[tokio::test]
-    async fn close_disk_guard_runs_explicit_cleanup_once() {
-        let close_count = Arc::new(std::sync::atomic::AtomicUsize::new(0));
-        let close_disk: CloseDiskFn = {
-            let close_count = Arc::clone(&close_count);
-            Arc::new(move || {
-                close_count.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
-                Box::pin(std::future::ready(()))
-            })
-        };
-
-        let mut guard = CloseDiskGuard::new(close_disk);
-        guard.close().await;
-        drop(guard);
-        tokio::time::sleep(std::time::Duration::from_millis(10)).await;
-
-        assert_eq!(close_count.load(std::sync::atomic::Ordering::Relaxed), 1);
-    }
-
-    #[tokio::test]
-    async fn current_path_updater_registers_before_return() {
-        let disk = format!("test-disk-{}", uuid::Uuid::new_v4());
-        let (_update_path, close_disk) = current_path_updater(&disk, "bucket-a").await;
-
-        assert!(global_metrics().current_paths.read().await.contains_key(&disk));
-
-        close_disk().await;
-        assert!(!global_metrics().current_paths.read().await.contains_key(&disk));
-    }
-
     #[tokio::test]
     async fn report_counts_active_scan_paths() {
         let metrics = Metrics::new();
@@ -3409,8 +3292,6 @@ mod tests {
         });
         metrics.record_scanner_expiry_enqueue_result(6, true);
         metrics.record_scanner_expiry_enqueue_result(2, false);
-        metrics.record_scanner_expiry_blocked(4);
-        metrics.record_scanner_expiry_delete_failed(1);
 
         let report = metrics.report().await;
 
@@ -3421,9 +3302,6 @@ mod tests {
         assert_eq!(report.lifecycle_expiry.queue_missed, 3);
         assert_eq!(report.lifecycle_expiry.scanner_queued, 6);
         assert_eq!(report.lifecycle_expiry.scanner_missed, 2);
-        assert_eq!(report.lifecycle_expiry.scanner_blocked, 4);
-        assert_eq!(report.lifecycle_expiry.scanner_not_enqueued, 2);
-        assert_eq!(report.lifecycle_expiry.delete_failed, 1);
     }
 
     #[tokio::test]
@@ -3483,40 +3361,6 @@ mod tests {
         metrics.finish_scan_cycle_work(start);
     }
 
-    #[test]
-    fn unresolved_heal_work_only_reflects_the_active_cycle() {
-        let metrics = Metrics::new();
-        assert!(!metrics.current_scan_cycle_has_unresolved_heal_work());
-
-        let start = metrics.start_scan_cycle_work();
-        metrics.record_scanner_source_missed(ScannerWorkSource::Heal, 1);
-        assert!(metrics.current_scan_cycle_has_unresolved_heal_work());
-
-        metrics.finish_scan_cycle_work(start);
-        assert!(!metrics.current_scan_cycle_has_unresolved_heal_work());
-
-        let start = metrics.start_scan_cycle_work();
-        metrics.record_scanner_source_queued(ScannerWorkSource::Heal, 1);
-        assert!(metrics.current_scan_cycle_has_unresolved_heal_work());
-        metrics.finish_scan_cycle_work(start);
-
-        let start = metrics.start_scan_cycle_work();
-        metrics.record_scanner_source_work(
-            ScannerWorkSource::Bitrot,
-            ScannerSourceWorkUpdate {
-                skipped: 1,
-                ..Default::default()
-            },
-        );
-        assert!(metrics.current_scan_cycle_has_unresolved_heal_work());
-        metrics.finish_scan_cycle_work(start);
-
-        let start = metrics.start_scan_cycle_work();
-        metrics.record_scanner_source_failed(ScannerWorkSource::Bitrot, 1);
-        assert!(metrics.current_scan_cycle_has_unresolved_heal_work());
-        metrics.finish_scan_cycle_work(start);
-    }
-
     #[tokio::test]
     async fn report_marks_transition_failures_as_blocked_lifecycle_control() {
         let metrics = Metrics::new();
@@ -3960,21 +3804,6 @@ mod tests {
         assert_eq!(report.failed_cycles, 1);
     }
 
-    #[tokio::test]
-    async fn report_tracks_superseded_cycle_without_failed_increment() {
-        let metrics = Metrics::new();
-        metrics.record_scan_cycle_superseded(Duration::from_millis(750));
-
-        let report = metrics.report().await;
-
-        assert_eq!(report.last_cycle_result, SCAN_CYCLE_RESULT_SUPERSEDED_LABEL);
-        assert_eq!(report.last_cycle_result_code, u64::from(SCAN_CYCLE_RESULT_SUPERSEDED));
-        assert_eq!(report.last_cycle_duration_seconds, 0.75);
-        assert_eq!(report.failed_cycles, 0);
-        assert_eq!(report.superseded_cycles, 1);
-        assert_eq!(report.partial_cycles, 0);
-    }
-
     #[tokio::test]
     async fn report_tracks_successful_scan_cycle_without_failed_increment() {
         let metrics = Metrics::new();

crates/concurrency/Cargo.toml

@@ -10,21 +10,18 @@ description = "Shared concurrency contract types for RustFS - workload admission
 keywords = ["rustfs", "concurrency", "admission", "backpressure", "workers"]
 categories = ["concurrency", "filesystem"]
 
-[lints]
-workspace = true
-
 [dependencies]
 # Internal crates
 rustfs-io-core = { workspace = true }
-serde = { workspace = true, features = ["derive"] }
+serde = { workspace = true }
 
 # Async runtime
-tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread"] }
+tokio = { workspace = true, features = ["sync"] }
 
 # Logging
 tracing = { workspace = true }
 
 [dev-dependencies]
-insta = { workspace = true, features = ["yaml", "json"] }
-serde_json = { workspace = true, features = ["raw_value"] }
-tokio = { workspace = true, features = ["test-util", "macros", "rt-multi-thread", "fs"] }
+insta = { workspace = true }
+serde_json = { workspace = true }
+tokio = { workspace = true, features = ["test-util", "macros", "rt-multi-thread"] }

crates/config/Cargo.toml

@@ -25,9 +25,9 @@ keywords = ["configuration", "settings", "management", "rustfs", "Minio"]
 categories = ["web-programming", "development-tools", "config"]
 
 [dependencies]
-const-str = { workspace = true, optional = true, features = ["std", "proc"] }
-serde = { workspace = true, optional = true, features = ["derive"] }
-serde_json = { workspace = true, optional = true, features = ["raw_value"] }
+const-str = { workspace = true, optional = true }
+serde = { workspace = true, optional = true }
+serde_json = { workspace = true, optional = true }
 
 [lints]
 workspace = true

crates/config/src/audit/nats.rs

@@ -25,11 +25,8 @@ pub const ENV_AUDIT_NATS_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_NATS_TLS_CLIENT_KE
 pub const ENV_AUDIT_NATS_TLS_REQUIRED: &str = "RUSTFS_AUDIT_NATS_TLS_REQUIRED";
 pub const ENV_AUDIT_NATS_QUEUE_DIR: &str = "RUSTFS_AUDIT_NATS_QUEUE_DIR";
 pub const ENV_AUDIT_NATS_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_NATS_QUEUE_LIMIT";
-pub const ENV_AUDIT_NATS_JETSTREAM_ENABLE: &str = "RUSTFS_AUDIT_NATS_JETSTREAM_ENABLE";
-pub const ENV_AUDIT_NATS_JETSTREAM_STREAM_NAME: &str = "RUSTFS_AUDIT_NATS_JETSTREAM_STREAM_NAME";
-pub const ENV_AUDIT_NATS_JETSTREAM_ACK_TIMEOUT_SECS: &str = "RUSTFS_AUDIT_NATS_JETSTREAM_ACK_TIMEOUT_SECS";
 
-pub const ENV_AUDIT_NATS_KEYS: &[&str; 16] = &[
+pub const ENV_AUDIT_NATS_KEYS: &[&str; 13] = &[
     ENV_AUDIT_NATS_ENABLE,
     ENV_AUDIT_NATS_ADDRESS,
     ENV_AUDIT_NATS_SUBJECT,
@@ -43,9 +40,6 @@ pub const ENV_AUDIT_NATS_KEYS: &[&str; 16] = &[
     ENV_AUDIT_NATS_TLS_REQUIRED,
     ENV_AUDIT_NATS_QUEUE_DIR,
     ENV_AUDIT_NATS_QUEUE_LIMIT,
-    ENV_AUDIT_NATS_JETSTREAM_ENABLE,
-    ENV_AUDIT_NATS_JETSTREAM_STREAM_NAME,
-    ENV_AUDIT_NATS_JETSTREAM_ACK_TIMEOUT_SECS,
 ];
 
 pub const AUDIT_NATS_KEYS: &[&str] = &[
@@ -62,8 +56,5 @@ pub const AUDIT_NATS_KEYS: &[&str] = &[
     crate::NATS_TLS_REQUIRED,
     crate::NATS_QUEUE_DIR,
     crate::NATS_QUEUE_LIMIT,
-    crate::NATS_JETSTREAM_ENABLE,
-    crate::NATS_JETSTREAM_STREAM_NAME,
-    crate::NATS_JETSTREAM_ACK_TIMEOUT_SECS,
     crate::COMMENT_KEY,
 ];

crates/config/src/constants/api.rs

@@ -1,92 +0,0 @@
-//  Copyright 2024 RustFS Team
-//
-//  Licensed under the Apache License, Version 2.0 (the "License");
-//  you may not use this file except in compliance with the License.
-//  You may obtain a copy of the License at
-//
-//      http://www.apache.org/licenses/LICENSE-2.0
-//
-//  Unless required by applicable law or agreed to in writing, software
-//  distributed under the License is distributed on an "AS IS" BASIS,
-//  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-//  See the License for the specific language governing permissions and
-//  limitations under the License.
-
-/// Enable or disable per-client rate limiting for the S3 API.
-///
-/// When enabled (and `RUSTFS_API_RATE_LIMIT_RPM` > 0), requests are throttled
-/// per client IP using a token bucket; over-limit requests receive
-/// `429 Too Many Requests` with a `Retry-After` header. Internode RPC/gRPC,
-/// health probes, and the console (which has its own limiter) are exempt.
-/// Environment variable: RUSTFS_API_RATE_LIMIT_ENABLE
-/// Example: RUSTFS_API_RATE_LIMIT_ENABLE=true
-pub const ENV_API_RATE_LIMIT_ENABLE: &str = "RUSTFS_API_RATE_LIMIT_ENABLE";
-
-/// Default for `RUSTFS_API_RATE_LIMIT_ENABLE`.
-///
-/// Disabled by default: RustFS ships permissive and operators opt in to
-/// abuse-protection hardening. When disabled the request path is unchanged.
-pub const DEFAULT_API_RATE_LIMIT_ENABLE: bool = false;
-
-/// Sustained S3 API request budget per client IP, in requests per minute.
-///
-/// `0` means unlimited (rate limiting stays inert even when enabled).
-/// Environment variable: RUSTFS_API_RATE_LIMIT_RPM
-/// Example: RUSTFS_API_RATE_LIMIT_RPM=6000
-pub const ENV_API_RATE_LIMIT_RPM: &str = "RUSTFS_API_RATE_LIMIT_RPM";
-
-/// Default for `RUSTFS_API_RATE_LIMIT_RPM`.
-///
-/// `0` (unlimited) so that setting only the enable switch cannot throttle
-/// traffic by surprise; operators must choose an explicit budget.
-pub const DEFAULT_API_RATE_LIMIT_RPM: u32 = 0;
-
-/// Burst capacity per client IP (maximum tokens in the bucket).
-///
-/// Allows short spikes above the sustained rate. `0` means "same as RPM".
-/// Environment variable: RUSTFS_API_RATE_LIMIT_BURST
-/// Example: RUSTFS_API_RATE_LIMIT_BURST=200
-pub const ENV_API_RATE_LIMIT_BURST: &str = "RUSTFS_API_RATE_LIMIT_BURST";
-
-/// Default for `RUSTFS_API_RATE_LIMIT_BURST` (`0` = same as RPM).
-pub const DEFAULT_API_RATE_LIMIT_BURST: u32 = 0;
-
-/// Sustained S3 API request budget per addressed bucket, in requests per
-/// minute — a collective ceiling shared by all clients of that bucket.
-///
-/// Complements the per-client-IP dimension: it protects the server from one
-/// hot bucket regardless of how many client IPs the traffic comes from. `0`
-/// disables the bucket dimension. Requires `RUSTFS_API_RATE_LIMIT_ENABLE`.
-/// Environment variable: RUSTFS_API_RATE_LIMIT_BUCKET_RPM
-/// Example: RUSTFS_API_RATE_LIMIT_BUCKET_RPM=60000
-pub const ENV_API_RATE_LIMIT_BUCKET_RPM: &str = "RUSTFS_API_RATE_LIMIT_BUCKET_RPM";
-
-/// Default for `RUSTFS_API_RATE_LIMIT_BUCKET_RPM` (`0` = dimension disabled).
-pub const DEFAULT_API_RATE_LIMIT_BUCKET_RPM: u32 = 0;
-
-/// Burst capacity per bucket (maximum tokens in the bucket-dimension bucket).
-///
-/// `0` means "same as `RUSTFS_API_RATE_LIMIT_BUCKET_RPM`".
-/// Environment variable: RUSTFS_API_RATE_LIMIT_BUCKET_BURST
-/// Example: RUSTFS_API_RATE_LIMIT_BUCKET_BURST=2000
-pub const ENV_API_RATE_LIMIT_BUCKET_BURST: &str = "RUSTFS_API_RATE_LIMIT_BUCKET_BURST";
-
-/// Default for `RUSTFS_API_RATE_LIMIT_BUCKET_BURST` (`0` = same as bucket RPM).
-pub const DEFAULT_API_RATE_LIMIT_BUCKET_BURST: u32 = 0;
-
-/// Maximum concurrently served connections on the main API listener.
-///
-/// `0` (the default) means unlimited. When set, the accept loop stops
-/// accepting once the cap is reached and lets the kernel backlog absorb
-/// bursts, releasing capacity as connections close. This bounds file
-/// descriptor and memory usage under a connection flood.
-///
-/// The cap covers everything on the main listener — S3, admin, console,
-/// and internode gRPC — so size it well above peer-node count plus the
-/// expected client concurrency.
-/// Environment variable: RUSTFS_API_MAX_CONNECTIONS
-/// Example: RUSTFS_API_MAX_CONNECTIONS=10000
-pub const ENV_API_MAX_CONNECTIONS: &str = "RUSTFS_API_MAX_CONNECTIONS";
-
-/// Default for `RUSTFS_API_MAX_CONNECTIONS` (`0` = unlimited).
-pub const DEFAULT_API_MAX_CONNECTIONS: usize = 0;

crates/config/src/constants/body_limits.rs

@@ -28,15 +28,6 @@ pub const MAX_ADMIN_REQUEST_BODY_SIZE: usize = 1024 * 1024; // 1 MB
 /// Rationale: ZIP archives with hundreds of IAM entities. 10MB allows ~10,000 small configs.
 pub const MAX_IAM_IMPORT_SIZE: usize = 10 * 1024 * 1024; // 10 MB
 
-/// Maximum total size the members of an IAM import ZIP may expand to (100 MB).
-/// Used for: bounding decompression of `ImportIam` archive members.
-/// Rationale: `MAX_IAM_IMPORT_SIZE` caps the *compressed* upload only. Deflate
-/// reaches ratios far above 100:1, so without a separate budget a 10 MB archive
-/// can expand without bound. 100 MB keeps a 10x headroom over the compressed cap
-/// — ample for legitimate IAM exports, which are small JSON documents — while
-/// keeping the worst case bounded.
-pub const MAX_IAM_IMPORT_EXPANDED_SIZE: u64 = 100 * 1024 * 1024; // 100 MB
-
 /// Maximum size for bucket metadata import operations (100 MB)
 /// Used for: Bucket metadata import containing configurations for many buckets
 /// Rationale: Large deployments may have thousands of buckets with various configs.
@@ -63,12 +54,3 @@ pub const MAX_HEAL_REQUEST_SIZE: usize = 1024 * 1024; // 1 MB
 /// 10MB provides generous headroom for legitimate responses while preventing
 /// memory exhaustion from malicious or misconfigured remote services.
 pub const MAX_S3_CLIENT_RESPONSE_SIZE: usize = 10 * 1024 * 1024; // 10 MB
-
-/// Maximum size for OIDC provider response bodies (1 MB)
-/// Used for: discovery documents, JWKS documents and token endpoint responses
-/// Rationale: a hostile or compromised identity provider must not be able to exhaust
-/// memory through an arbitrarily large or endless response body.
-/// - Discovery documents: typically < 10KB
-/// - JWKS documents: typically < 50KB
-/// - Token responses: typically < 10KB
-pub const MAX_OIDC_RESPONSE_SIZE: usize = 1024 * 1024; // 1 MB

crates/config/src/constants/drive.rs

@@ -39,11 +39,6 @@ pub const DEFAULT_DRIVE_WALKDIR_TIMEOUT_SECS: u64 = 5;
 pub const ENV_DRIVE_WALKDIR_STALL_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS";
 pub const DEFAULT_DRIVE_WALKDIR_STALL_TIMEOUT_SECS: u64 = 5;
 
-/// Maximum time the metacache merge consumer waits for the next visible
-/// `walk_dir()` entry from a reader before detaching it from the merge.
-pub const ENV_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS";
-pub const DEFAULT_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS: u64 = 10;
-
 /// Interval in seconds between active health probes for local and remote drives.
 pub const ENV_DRIVE_ACTIVE_CHECK_INTERVAL_SECS: &str = "RUSTFS_DRIVE_ACTIVE_CHECK_INTERVAL_SECS";
 pub const DEFAULT_DRIVE_ACTIVE_CHECK_INTERVAL_SECS: u64 = 15;

crates/config/src/constants/internode.rs

@@ -97,80 +97,19 @@ pub const ENV_INTERNODE_RPC_MAX_MESSAGE_SIZE: &str = "RUSTFS_INTERNODE_RPC_MAX_M
 pub const ENV_INTERNODE_RPC_LARGE_PAYLOAD_WARN_BYTES: &str = "RUSTFS_INTERNODE_RPC_LARGE_PAYLOAD_WARN_BYTES";
 pub const DEFAULT_INTERNODE_RPC_LARGE_PAYLOAD_WARN_BYTES: usize = 8 * 1024 * 1024;
 
-/// Request stopping the JSON compatibility strings on internode metadata RPCs and sending only the
+/// Stop dual-writing the JSON compatibility strings on internode metadata RPCs and send only the
 /// msgpack `_bin` payloads (grpc-optimization P2-1).
 ///
-/// Defaults to `false` (dual-write, byte-for-byte legacy behavior). This is only a request; RustFS
-/// keeps JSON compatibility fields unless [`ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED`] is also
-/// true after the release-window convergence and rollback gates pass. See
+/// Defaults to `false` (dual-write, byte-for-byte legacy behavior). This is a rollout lever, not a
+/// wire-format change: it may only be enabled **after** the JSON-fallback counter
+/// (`rustfs_system_network_internode_msgpack_json_fallback_total`) has read zero across a release
+/// window fleet-wide, confirming every peer decodes `_bin` first. Single-env rollback. See
 /// `docs/operations/internode-msgpack-json-convergence-runbook.md`.
 pub const ENV_INTERNODE_RPC_MSGPACK_ONLY: &str = "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY";
 pub const DEFAULT_INTERNODE_RPC_MSGPACK_ONLY: bool = false;
 
-/// Explicit fleet-wide confirmation gate for [`ENV_INTERNODE_RPC_MSGPACK_ONLY`].
-///
-/// This separate default-off guard prevents a single legacy flag from accidentally emptying JSON
-/// fields in a mixed-version fleet where an older peer still reads the JSON field.
-pub const ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED: &str = "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED";
-pub const DEFAULT_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED: bool = false;
-
-// Compile-time invariants: dual-write by default so the base build is byte-for-byte legacy behavior.
+// Compile-time invariant: dual-write by default so the base build is byte-for-byte legacy behavior.
 const _: () = assert!(!DEFAULT_INTERNODE_RPC_MSGPACK_ONLY);
-const _: () = assert!(!DEFAULT_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED);
-
-/// Require target-bound v2 signatures on every internode gRPC request, rejecting the legacy
-/// constant-target fallback instead of accepting it (<https://github.com/rustfs/backlog/issues/1327>).
-///
-/// Defaults to `false` (fail-open): a request without any v2 auth headers keeps authenticating
-/// through the legacy signature, so legacy-only peers survive rolling upgrades with byte-for-byte
-/// the pre-gate acceptance behavior. This is a rollout lever, not a wire-format change: it may only
-/// be enabled **after** the v1-fallback counter
-/// (`rustfs_system_network_internode_signature_v1_fallback_total`) has read zero across a release
-/// window fleet-wide, confirming every peer already sends v2 authentication on every internode gRPC
-/// request. Single-env rollback. Requests that do carry v2 headers are unaffected by this switch:
-/// they are always verified as v2 with no downgrade, strict or not.
-pub const ENV_INTERNODE_RPC_SIGNATURE_STRICT: &str = "RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT";
-pub const DEFAULT_INTERNODE_RPC_SIGNATURE_STRICT: bool = false;
-
-// Compile-time invariant: fail-open by default so legacy-only peers keep authenticating during
-// rolling upgrades until the fleet-wide v1-fallback counter reads zero.
-const _: () = assert!(!DEFAULT_INTERNODE_RPC_SIGNATURE_STRICT);
-
-/// Require a signature-bound canonical body digest on every mutating internode disk RPC
-/// (RenameData, DeleteVersion, DeleteVersions, WriteMetadata, UpdateMetadata, WriteAll, Delete,
-/// DeletePaths, RenameFile, RenamePart, DeleteVolume, MakeVolume, MakeVolumes), rejecting requests
-/// that authenticate without one (<https://github.com/rustfs/backlog/issues/1327>).
-///
-/// Defaults to `false` (fail-open): a mutating request without a body digest keeps authenticating
-/// through the method-bound v2 (or legacy) signature, so peers from releases that predate
-/// body-digest signing survive rolling upgrades unchanged. Requests that do carry a digest are
-/// always verified with no downgrade, strict or not — the digest value is part of the signed v2
-/// scope, so an on-path attacker cannot strip it without invalidating the signature. This is a
-/// rollout lever gated on the body-digest fallback counter
-/// (`rustfs_system_network_internode_body_digest_fallback_total`) reading zero across a release
-/// window fleet-wide. Single-env rollback. It is deliberately separate from
-/// [`ENV_INTERNODE_RPC_SIGNATURE_STRICT`]: the two enforcement flips converge on different
-/// counters and must not gate each other.
-pub const ENV_INTERNODE_RPC_BODY_DIGEST_STRICT: &str = "RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT";
-pub const DEFAULT_INTERNODE_RPC_BODY_DIGEST_STRICT: bool = false;
-
-// Compile-time invariant: fail-open by default so digestless peers keep authenticating during
-// rolling upgrades until the fleet-wide body-digest fallback counter reads zero.
-const _: () = assert!(!DEFAULT_INTERNODE_RPC_BODY_DIGEST_STRICT);
-
-/// Capacity (distinct nonces) of the process-local internode RPC replay cache that enforces
-/// one-time consumption of body-bound v2 signatures.
-///
-/// The cache retains each nonce for the ~10-minute signature freshness envelope, so the steady
-/// state holds roughly `mutating RPS x 601s` entries; the default sustains ~1,700 body-bound
-/// mutating RPCs per second (about 120 MiB worst case, allocated only under sustained load).
-/// Overflow fails closed — legitimate signed traffic is the only thing that can fill the cache
-/// (replays are rejected before insertion, and an attacker cannot mint valid nonces without the
-/// shared secret) — and increments
-/// `rustfs_system_network_internode_replay_cache_overflow_total`, so a sustained non-zero overflow
-/// counter means this capacity is undersized for the node's peak mutation rate.
-pub const ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: &str = "RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY";
-pub const DEFAULT_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: usize = 1_048_576;
 
 /// Consecutive-failure threshold after which an internode peer is marked offline (grpc-optimization
 /// P3 observability).
@@ -334,30 +273,8 @@ mod tests {
 
     #[test]
     fn internode_msgpack_only_env_name_is_stable() {
-        // The dual-write-by-default invariants are asserted at compile time next to the definitions.
+        // The dual-write-by-default invariant is asserted at compile time next to the definition.
         assert_eq!(ENV_INTERNODE_RPC_MSGPACK_ONLY, "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY");
-        assert_eq!(
-            ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED,
-            "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED"
-        );
-    }
-
-    #[test]
-    fn internode_signature_strict_env_name_is_stable() {
-        // The fail-open default invariant is asserted at compile time next to the definition.
-        assert_eq!(ENV_INTERNODE_RPC_SIGNATURE_STRICT, "RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT");
-    }
-
-    #[test]
-    fn internode_body_digest_strict_env_name_is_stable() {
-        // The fail-open default invariant is asserted at compile time next to the definition.
-        assert_eq!(ENV_INTERNODE_RPC_BODY_DIGEST_STRICT, "RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT");
-    }
-
-    #[test]
-    fn internode_replay_cache_capacity_defaults_and_env_name() {
-        assert_eq!(ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY, "RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY");
-        assert_eq!(DEFAULT_INTERNODE_RPC_REPLAY_CACHE_CAPACITY, 1_048_576);
     }
 
     #[test]

crates/config/src/constants/mod.rs

@@ -12,7 +12,6 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-pub(crate) mod api;
 pub(crate) mod app;
 pub(crate) mod body_limits;
 pub(crate) mod capacity;

crates/config/src/constants/object.rs

@@ -73,36 +73,15 @@ pub const DEFAULT_OBJECT_MAX_CONCURRENT_DISK_READS: usize = 64;
 /// - Example: `export RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT=5`
 pub const ENV_OBJECT_DISK_PERMIT_WAIT_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT";
 
-/// Maximum time a GET request waits for a primary disk read permit (seconds).
+/// Maximum time a GET request waits for a disk read permit (seconds).
 ///
 /// Permits are held for the whole response body transfer, so slow clients can
 /// occupy all of them while the disks sit idle. Instead of stalling until the
-/// request-level timeout fires, a GET that waits longer than this falls through
-/// to a bounded degraded admission lane; if that lane is also full the request
-/// is rejected with `SlowDown`/503 rather than proceeding without any permit.
-/// Set to 0 to wait on the primary lane indefinitely (never degrade or reject).
+/// request-level timeout fires, a GET that waits longer than this proceeds
+/// without a permit (degraded pass-through) and the bypass is counted in
+/// metrics/logs. Set to 0 to wait indefinitely (previous behavior).
 pub const DEFAULT_OBJECT_DISK_PERMIT_WAIT_TIMEOUT: u64 = 5;
 
-/// Environment variable for the bounded degraded disk-read admission lane size.
-/// - Purpose: Cap how many GETs may proceed after the primary disk-read permit
-///   pool is saturated, giving a hard upper bound on concurrent disk-active
-///   reads (primary cap + degraded cap) instead of an unbounded pass-through.
-/// - Unit: request count (usize). `0` means "mirror the primary cap", so the
-///   absolute hard cap defaults to twice the primary disk-read cap.
-/// - Example: `export RUSTFS_OBJECT_DISK_DEGRADED_READ_CAP=16`
-pub const ENV_OBJECT_DISK_DEGRADED_READ_CAP: &str = "RUSTFS_OBJECT_DISK_DEGRADED_READ_CAP";
-
-/// Size of the bounded degraded disk-read admission lane.
-///
-/// When the primary disk-read permit pool is saturated and a GET exceeds
-/// [`DEFAULT_OBJECT_DISK_PERMIT_WAIT_TIMEOUT`], it may take one permit from this
-/// bounded overflow lane instead of reading without any admission token. The
-/// total number of GETs performing disk-active reads is therefore hard-capped at
-/// `primary_cap + degraded_cap`; beyond that a GET is rejected with `SlowDown`.
-/// The default `0` mirrors the primary cap, so the hard cap is twice the primary
-/// disk-read concurrency.
-pub const DEFAULT_OBJECT_DISK_DEGRADED_READ_CAP: usize = 0;
-
 /// Skip bitrot hash verification on GetObject reads.
 ///
 /// When enabled, GetObject reads skip the per-shard hash
@@ -116,27 +95,6 @@ pub const ENV_OBJECT_GET_SKIP_BITROT_VERIFY: &str = "RUSTFS_OBJECT_GET_SKIP_BITR
 /// Default: bitrot verification is enabled on GetObject reads (do not skip).
 pub const DEFAULT_OBJECT_GET_SKIP_BITROT_VERIFY: bool = false;
 
-/// Request writing the complete remote-tier version state into object metadata.
-///
-/// This remains ineffective until
-/// [`ENV_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED`] is also enabled.
-pub const ENV_TIER_REMOTE_VERSION_STATE_WRITE: &str = "RUSTFS_TIER_REMOTE_VERSION_STATE_WRITE";
-pub const DEFAULT_TIER_REMOTE_VERSION_STATE_WRITE: bool = false;
-
-/// Operator-attested fleet-wide confirmation for
-/// [`ENV_TIER_REMOTE_VERSION_STATE_WRITE`].
-///
-/// This flag is an operational contract, not automatic capability discovery.
-/// Operators may enable it only after every node that can write or read
-/// transitioned object metadata supports the remote version-state schema and
-/// semantics. Keeping the confirmation separate makes a single-node request or
-/// a writer whose local opt-in is removed fail closed.
-pub const ENV_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED: &str = "RUSTFS_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED";
-pub const DEFAULT_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED: bool = false;
-
-const _: () = assert!(!DEFAULT_TIER_REMOTE_VERSION_STATE_WRITE);
-const _: () = assert!(!DEFAULT_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED);
-
 // =============================================================================
 // Concurrent Request Fix - Timeout and Backpressure Configuration
 // =============================================================================
@@ -170,39 +128,6 @@ pub const ENV_OBJECT_DISK_READ_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_READ_TIMEOUT"
 /// Default disk read timeout in seconds.
 pub const DEFAULT_OBJECT_DISK_READ_TIMEOUT: u64 = 10;
 
-/// Environment variable for the per-shard erasure write stall timeout (seconds).
-///
-/// A single shard write (or shard-writer shutdown) that makes no forward
-/// progress for longer than this budget is failed and its disk is dropped
-/// before commit, so a black-hole peer that accepts the connection but never
-/// drains the body cannot pin an otherwise-healthy write quorum forever
-/// (see `MultiWriter` in `erasure/coding/encode.rs`). The budget is re-armed on
-/// every shard write, so it bounds a *stall* rather than the total transfer
-/// time of a large object.
-///
-/// Unit: seconds (u64). `0` disables the stall deadline (previous behavior:
-/// wait indefinitely). Default: 30 seconds.
-pub const ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_WRITE_STALL_TIMEOUT";
-
-/// Default per-shard erasure write stall timeout in seconds.
-pub const DEFAULT_OBJECT_DISK_WRITE_STALL_TIMEOUT: u64 = 30;
-
-/// Environment variable for the absolute per-object erasure write cap (seconds).
-///
-/// Optional administrator backstop against a "slow-drip" peer that produces
-/// just enough forward progress to reset the per-shard stall timeout on every
-/// block while never converging. When set, the shard writers for one object are
-/// engaged for at most this long in aggregate before a stalled writer is failed
-/// and dropped. It is disabled by default because a legitimate large upload
-/// over a slow-but-honest link must not be killed on total time alone; the
-/// per-shard stall timeout is the primary guarantee.
-///
-/// Unit: seconds (u64). `0` (default) disables the absolute cap.
-pub const ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP: &str = "RUSTFS_OBJECT_DISK_WRITE_ABSOLUTE_CAP";
-
-/// Default absolute per-object erasure write cap in seconds (`0` = disabled).
-pub const DEFAULT_OBJECT_DISK_WRITE_ABSOLUTE_CAP: u64 = 0;
-
 /// Environment variable for minimum GetObject timeout in seconds.
 ///
 /// When dynamic timeout calculation is enabled, this is the minimum timeout
@@ -638,15 +563,3 @@ pub const ENV_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY: &str = "RUSTFS_OBJ
 
 /// Default read-ahead disable concurrency threshold: 4.
 pub const DEFAULT_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY: usize = 4;
-
-#[cfg(test)]
-mod remote_version_state_tests {
-    #[test]
-    fn remote_version_state_gate_uses_stable_environment_names() {
-        assert_eq!(super::ENV_TIER_REMOTE_VERSION_STATE_WRITE, "RUSTFS_TIER_REMOTE_VERSION_STATE_WRITE");
-        assert_eq!(
-            super::ENV_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED,
-            "RUSTFS_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED"
-        );
-    }
-}

crates/config/src/constants/oidc.rs

@@ -14,7 +14,6 @@
 
 // OIDC configuration field keys (used in KVS)
 pub const OIDC_CONFIG_URL: &str = "config_url";
-pub const OIDC_ISSUER: &str = "issuer";
 pub const OIDC_CLIENT_ID: &str = "client_id";
 pub const OIDC_CLIENT_SECRET: &str = "client_secret";
 pub const OIDC_SCOPES: &str = "scopes";
@@ -34,7 +33,6 @@ pub const OIDC_HIDE_FROM_UI: &str = "hide_from_ui";
 // Environment variable names for OIDC
 pub const ENV_IDENTITY_OPENID_ENABLE: &str = "RUSTFS_IDENTITY_OPENID_ENABLE";
 pub const ENV_IDENTITY_OPENID_CONFIG_URL: &str = "RUSTFS_IDENTITY_OPENID_CONFIG_URL";
-pub const ENV_IDENTITY_OPENID_ISSUER: &str = "RUSTFS_IDENTITY_OPENID_ISSUER";
 pub const ENV_IDENTITY_OPENID_CLIENT_ID: &str = "RUSTFS_IDENTITY_OPENID_CLIENT_ID";
 pub const ENV_IDENTITY_OPENID_CLIENT_SECRET: &str = "RUSTFS_IDENTITY_OPENID_CLIENT_SECRET";
 pub const ENV_IDENTITY_OPENID_SCOPES: &str = "RUSTFS_IDENTITY_OPENID_SCOPES";
@@ -52,10 +50,9 @@ pub const ENV_IDENTITY_OPENID_USERNAME_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_USE
 pub const ENV_IDENTITY_OPENID_HIDE_FROM_UI: &str = "RUSTFS_IDENTITY_OPENID_HIDE_FROM_UI";
 
 /// List of all environment variable keys for an OIDC provider.
-pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 18] = &[
+pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 17] = &[
     ENV_IDENTITY_OPENID_ENABLE,
     ENV_IDENTITY_OPENID_CONFIG_URL,
-    ENV_IDENTITY_OPENID_ISSUER,
     ENV_IDENTITY_OPENID_CLIENT_ID,
     ENV_IDENTITY_OPENID_CLIENT_SECRET,
     ENV_IDENTITY_OPENID_SCOPES,
@@ -77,7 +74,6 @@ pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 18] = &[
 pub const IDENTITY_OPENID_KEYS: &[&str] = &[
     crate::ENABLE_KEY,
     OIDC_CONFIG_URL,
-    OIDC_ISSUER,
     OIDC_CLIENT_ID,
     OIDC_CLIENT_SECRET,
     OIDC_SCOPES,

crates/config/src/constants/protocols.rs

@@ -57,7 +57,6 @@ pub const ENV_WEBDAV_CERTS_DIR: &str = "RUSTFS_WEBDAV_CERTS_DIR";
 pub const ENV_WEBDAV_CA_FILE: &str = "RUSTFS_WEBDAV_CA_FILE";
 pub const ENV_WEBDAV_MAX_BODY_SIZE: &str = "RUSTFS_WEBDAV_MAX_BODY_SIZE";
 pub const ENV_WEBDAV_REQUEST_TIMEOUT: &str = "RUSTFS_WEBDAV_REQUEST_TIMEOUT";
-pub const ENV_WEBDAV_MAX_CONNECTIONS: &str = "RUSTFS_WEBDAV_MAX_CONNECTIONS";
 
 /// Default SFTP server bind address.
 pub const DEFAULT_SFTP_ADDRESS: &str = "0.0.0.0:2222";

crates/config/src/constants/scanner.rs

@@ -220,10 +220,12 @@ pub const ENV_SCANNER_YIELD_EVERY_N_OBJECTS: &str = "RUSTFS_SCANNER_YIELD_EVERY_
 pub const DEFAULT_SCANNER_IDLE_MODE: bool = true;
 
 /// Default set scan concurrency budget.
-pub const DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS: usize = 4;
+/// `0` means no additional limit beyond deployment topology.
+pub const DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS: usize = 0;
 
 /// Default disk scan concurrency budget.
-pub const DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS: usize = 4;
+/// `0` means no additional limit beyond available disks in the set.
+pub const DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS: usize = 0;
 
 /// Default object interval for cooperative scanner yields.
 pub const DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS: u64 = 128;

crates/config/src/constants/targets.rs

@@ -79,13 +79,6 @@ pub const NATS_TLS_CLIENT_KEY: &str = "tls_client_key";
 pub const NATS_TLS_REQUIRED: &str = "tls_required";
 pub const NATS_QUEUE_DIR: &str = "queue_dir";
 pub const NATS_QUEUE_LIMIT: &str = "queue_limit";
-pub const NATS_JETSTREAM_ENABLE: &str = "jetstream_enable";
-pub const NATS_JETSTREAM_STREAM_NAME: &str = "jetstream_stream_name";
-pub const NATS_JETSTREAM_ACK_TIMEOUT_SECS: &str = "jetstream_ack_timeout_secs";
-
-pub const NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS: u64 = 30;
-pub const NATS_JETSTREAM_ACK_TIMEOUT_MIN_SECS: u64 = 10;
-pub const NATS_JETSTREAM_ACK_TIMEOUT_MAX_SECS: u64 = 120;
 
 pub const PULSAR_BROKER: &str = "broker";
 pub const PULSAR_TOPIC: &str = "topic";

crates/config/src/constants/tls.rs

@@ -142,10 +142,6 @@ pub const DEFAULT_H2_KEEP_ALIVE_TIMEOUT: u64 = 10;
 /// proxy's upstream idle-keepalive, or lower the proxy's keepalive below this
 /// value. Environments that expose RustFS directly to untrusted slow clients and
 /// want tighter slowloris protection can lower it via the env var below.
-///
-/// The same budget bounds the TLS handshake on the listener, so an unauthenticated
-/// peer cannot park an accept task and its socket indefinitely by opening a
-/// connection and then stalling the handshake.
 pub const ENV_HTTP1_HEADER_READ_TIMEOUT: &str = "RUSTFS_HTTP1_HEADER_READ_TIMEOUT";
 pub const DEFAULT_HTTP1_HEADER_READ_TIMEOUT: u64 = 75;
 

crates/config/src/constants/zero_copy.rs

@@ -56,33 +56,3 @@ pub const DEFAULT_OBJECT_MMAP_READ_ENABLE: bool = true;
 ///
 /// Prefer [`DEFAULT_OBJECT_MMAP_READ_ENABLE`].
 pub const DEFAULT_OBJECT_ZERO_COPY_ENABLE: bool = DEFAULT_OBJECT_MMAP_READ_ENABLE;
-
-/// Environment variable capping the byte length a single mmap-copy read may
-/// materialize in memory.
-///
-/// The mmap-copy read path returns the whole requested range as one owned
-/// allocation before the first byte is served. GET/heal shard reads request
-/// the entire part span in one call, so for a large single-part object
-/// (e.g. a multi-gigabyte non-multipart upload) an uncapped mmap-copy read
-/// allocates the whole shard in memory — stalling first-byte latency past the
-/// disk-read timeout and OOM-killing memory-limited deployments
-/// (<https://github.com/rustfs/rustfs/issues/5123>). Reads longer than this
-/// cap fall back to the bounded streaming reader instead.
-///
-/// - Purpose: Bound per-shard-read memory for mmap-based reads
-/// - Acceptable values: byte count as an unsigned integer; `0` disables
-///   mmap-copy for all non-empty reads (every read streams)
-/// - Example: `export RUSTFS_OBJECT_MMAP_READ_MAX_LENGTH=8388608`
-pub const ENV_OBJECT_MMAP_READ_MAX_LENGTH: &str = "RUSTFS_OBJECT_MMAP_READ_MAX_LENGTH";
-
-/// Default mmap-copy read length cap: 32 MiB per shard read.
-///
-/// Large enough that typical multipart part shards (parts up to a few hundred
-/// megabytes across the erasure set) keep the mmap fast path, small enough
-/// that whole-part reads of huge single-part objects stream instead of
-/// materializing gigabytes per shard.
-///
-/// The cap bounds memory per shard reader, so a single part read can still
-/// materialize up to `data_shards x cap` bytes; raising the cap raises that
-/// per-request bound proportionally.
-pub const DEFAULT_OBJECT_MMAP_READ_MAX_LENGTH: usize = 32 * 1024 * 1024;

crates/config/src/lib.rs

@@ -15,8 +15,6 @@
 #[cfg(feature = "constants")]
 pub mod constants;
 #[cfg(feature = "constants")]
-pub use constants::api::*;
-#[cfg(feature = "constants")]
 pub use constants::app::*;
 #[cfg(feature = "constants")]
 pub use constants::body_limits::*;

crates/config/src/notify/nats.rs

@@ -26,9 +26,6 @@ pub const NOTIFY_NATS_KEYS: &[&str] = &[
     crate::NATS_TLS_REQUIRED,
     crate::NATS_QUEUE_DIR,
     crate::NATS_QUEUE_LIMIT,
-    crate::NATS_JETSTREAM_ENABLE,
-    crate::NATS_JETSTREAM_STREAM_NAME,
-    crate::NATS_JETSTREAM_ACK_TIMEOUT_SECS,
     crate::COMMENT_KEY,
 ];
 
@@ -45,11 +42,8 @@ pub const ENV_NOTIFY_NATS_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_NATS_TLS_CLIENT_
 pub const ENV_NOTIFY_NATS_TLS_REQUIRED: &str = "RUSTFS_NOTIFY_NATS_TLS_REQUIRED";
 pub const ENV_NOTIFY_NATS_QUEUE_DIR: &str = "RUSTFS_NOTIFY_NATS_QUEUE_DIR";
 pub const ENV_NOTIFY_NATS_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_NATS_QUEUE_LIMIT";
-pub const ENV_NOTIFY_NATS_JETSTREAM_ENABLE: &str = "RUSTFS_NOTIFY_NATS_JETSTREAM_ENABLE";
-pub const ENV_NOTIFY_NATS_JETSTREAM_STREAM_NAME: &str = "RUSTFS_NOTIFY_NATS_JETSTREAM_STREAM_NAME";
-pub const ENV_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS: &str = "RUSTFS_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS";
 
-pub const ENV_NOTIFY_NATS_KEYS: &[&str; 16] = &[
+pub const ENV_NOTIFY_NATS_KEYS: &[&str; 13] = &[
     ENV_NOTIFY_NATS_ENABLE,
     ENV_NOTIFY_NATS_ADDRESS,
     ENV_NOTIFY_NATS_SUBJECT,
@@ -63,7 +57,4 @@ pub const ENV_NOTIFY_NATS_KEYS: &[&str; 16] = &[
     ENV_NOTIFY_NATS_TLS_REQUIRED,
     ENV_NOTIFY_NATS_QUEUE_DIR,
     ENV_NOTIFY_NATS_QUEUE_LIMIT,
-    ENV_NOTIFY_NATS_JETSTREAM_ENABLE,
-    ENV_NOTIFY_NATS_JETSTREAM_STREAM_NAME,
-    ENV_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS,
 ];

crates/config/src/observability/mod.rs

@@ -37,6 +37,10 @@ pub const ENV_OBS_METER_INTERVAL: &str = "RUSTFS_OBS_METER_INTERVAL";
 pub const ENV_OBS_SERVICE_NAME: &str = "RUSTFS_OBS_SERVICE_NAME";
 pub const ENV_OBS_SERVICE_VERSION: &str = "RUSTFS_OBS_SERVICE_VERSION";
 pub const ENV_OBS_ENVIRONMENT: &str = "RUSTFS_OBS_ENVIRONMENT";
+/// Stable OpenTelemetry service instance identity for this RustFS process.
+pub const ENV_OBS_INSTANCE_ID: &str = "RUSTFS_OBS_INSTANCE_ID";
+/// Optional stable RustFS deployment identity used to correlate cluster telemetry.
+pub const ENV_OBS_CLUSTER_ID: &str = "RUSTFS_OBS_CLUSTER_ID";
 
 /// Per-signal enable/disable flags (default: true)
 pub const ENV_OBS_TRACES_EXPORT_ENABLED: &str = "RUSTFS_OBS_TRACES_EXPORT_ENABLED";
@@ -131,6 +135,8 @@ mod tests {
         assert_eq!(ENV_OBS_SERVICE_NAME, "RUSTFS_OBS_SERVICE_NAME");
         assert_eq!(ENV_OBS_SERVICE_VERSION, "RUSTFS_OBS_SERVICE_VERSION");
         assert_eq!(ENV_OBS_ENVIRONMENT, "RUSTFS_OBS_ENVIRONMENT");
+        assert_eq!(ENV_OBS_INSTANCE_ID, "RUSTFS_OBS_INSTANCE_ID");
+        assert_eq!(ENV_OBS_CLUSTER_ID, "RUSTFS_OBS_CLUSTER_ID");
         assert_eq!(ENV_OBS_LOGGER_LEVEL, "RUSTFS_OBS_LOGGER_LEVEL");
         assert_eq!(ENV_OBS_LOG_STDOUT_ENABLED, "RUSTFS_OBS_LOG_STDOUT_ENABLED");
         assert_eq!(ENV_OBS_LOG_DIRECTORY, "RUSTFS_OBS_LOG_DIRECTORY");

crates/credentials/Cargo.toml

@@ -27,9 +27,9 @@ categories = ["web-programming", "development-tools", "data-structures", "securi
 [dependencies]
 base64-simd = { workspace = true }
 hmac = { workspace = true }
-rand = { workspace = true, features = ["serde"] }
-serde = { workspace = true, features = ["derive"] }
-serde_json = { workspace = true, features = ["raw_value"] }
+rand = { workspace = true }
+serde = { workspace = true }
+serde_json.workspace = true
 sha2 = { workspace = true }
 time = { workspace = true, features = ["serde", "parsing", "formatting", "macros"] }
 

crates/credentials/src/credentials.rs

@@ -260,10 +260,13 @@ fn resolve_rpc_secret(env_secret: Option<&str>, global_access: Option<&str>, glo
 
     match (global_access, global_secret) {
         (Some(access_key), Some(secret_key)) => {
-            // Fail closed when either half of the active credential pair still
-            // uses the public default. Operators must configure both custom
-            // credentials or provide RUSTFS_RPC_SECRET explicitly.
-            if access_key.trim() == DEFAULT_ACCESS_KEY || secret_key.trim() == DEFAULT_SECRET_KEY {
+            // Fail closed: never derive the RPC secret while the default secret
+            // key is in effect. The derivation uses `secret_key` as the HMAC key,
+            // so a public default secret yields a publicly computable RPC secret
+            // that any network peer can use to forge internode RPC signatures.
+            // Operators running with default credentials must configure
+            // RUSTFS_RPC_SECRET (or set a non-default RUSTFS_SECRET_KEY) instead.
+            if secret_key.trim() == DEFAULT_SECRET_KEY {
                 return None;
             }
             derive_rpc_secret(access_key, secret_key)
@@ -586,11 +589,18 @@ mod tests {
     fn test_resolve_rpc_secret_rejects_default_credentials_for_derivation() {
         assert!(resolve_rpc_secret(None, None, None).is_none());
 
-        // Fail closed when either half of the credential pair uses the public
-        // default.
+        // Fail closed: the default secret key must not yield a derivable RPC
+        // secret, otherwise the derived value is publicly computable and any
+        // network peer can forge internode RPC signatures.
         assert!(resolve_rpc_secret(None, Some(DEFAULT_ACCESS_KEY), Some(DEFAULT_SECRET_KEY)).is_none());
-        assert!(resolve_rpc_secret(None, Some(DEFAULT_ACCESS_KEY), Some("custom-global-secret")).is_none());
-        assert!(resolve_rpc_secret(None, Some("custom-access"), Some(DEFAULT_SECRET_KEY)).is_none());
+
+        // A default access key paired with a non-default secret key is still
+        // safe to derive: the HMAC key (the secret key) is not public.
+        let expected = derive_rpc_secret(DEFAULT_ACCESS_KEY, "custom-global-secret").expect("secret should derive");
+        assert_eq!(
+            resolve_rpc_secret(None, Some(DEFAULT_ACCESS_KEY), Some("custom-global-secret")).as_deref(),
+            Some(expected.as_str())
+        );
 
         assert!(resolve_rpc_secret(Some(DEFAULT_SECRET_KEY), Some("custom-access"), Some("custom-global-secret")).is_none());
     }
@@ -625,10 +635,6 @@ mod tests {
             resolve_rpc_secret(Some("custom-rpc-secret"), None, None).as_deref(),
             Some("custom-rpc-secret")
         );
-        assert_eq!(
-            resolve_rpc_secret(Some("custom-rpc-secret"), Some(DEFAULT_ACCESS_KEY), Some(DEFAULT_SECRET_KEY)).as_deref(),
-            Some("custom-rpc-secret")
-        );
         let expected = derive_rpc_secret("custom-access", "custom-global-secret").expect("secret should derive");
         assert_eq!(
             resolve_rpc_secret(None, Some("custom-access"), Some("custom-global-secret")).as_deref(),

crates/crypto/Cargo.toml

@@ -29,23 +29,23 @@ documentation = "https://docs.rs/rustfs-crypto/latest/rustfs_crypto/"
 workspace = true
 
 [dependencies]
-aes-gcm = { workspace = true, optional = true, features = ["rand_core"] }
+aes-gcm = { workspace = true, optional = true }
 argon2 = { workspace = true, optional = true }
 chacha20poly1305 = { workspace = true, optional = true }
-jsonwebtoken = { workspace = true, features = ["aws_lc_rs"] }
+jsonwebtoken = { workspace = true }
 base64-simd = { workspace = true }
 pbkdf2 = { workspace = true, optional = true }
-rand = { workspace = true, features = ["serde"] }
+rand = { workspace = true }
 rsa = { workspace = true, features = ["sha2"] }
 serde = { workspace = true, features = ["derive"] }
 sha2 = { workspace = true, optional = true }
 thiserror.workspace = true
-serde_json = { workspace = true, features = ["raw_value"] }
+serde_json.workspace = true
 
 
 [dev-dependencies]
 test-case.workspace = true
-time = { workspace = true, features = ["parsing", "formatting", "macros", "serde"] }
+time.workspace = true
 
 [features]
 default = ["crypto", "fips"]

crates/data-usage/Cargo.toml

@@ -28,7 +28,7 @@ categories = ["data-structures", "filesystem"]
 workspace = true
 
 [dependencies]
-serde = { workspace = true, features = ["derive"] }
+serde = { workspace = true }
 path-clean = { workspace = true }
 rmp-serde = { workspace = true }
 async-trait = { workspace = true }

crates/data-usage/src/data_usage.rs

@@ -18,41 +18,9 @@ use std::{
     collections::{HashMap, HashSet},
     hash::{DefaultHasher, Hash, Hasher},
     path::Path,
-    time::{Duration, SystemTime},
+    time::SystemTime,
 };
 
-/// Maximum amount a persisted `last_update` may lead the local wall clock before the
-/// persisted timestamp is treated as untrustworthy.
-///
-/// Invariant: the "skip stale usage update" monotonicity check (incoming `last_update`
-/// <= existing `last_update` => skip persisting) is only valid while the existing
-/// timestamp could plausibly have been produced by a healthy clock. If the on-disk
-/// snapshot is future-dated beyond this tolerance (NTP step-back, or scanner
-/// leadership moving to a node with a slower clock), the comparison would skip every
-/// save forever and freeze admin usage stats; callers must bypass the skip instead.
-pub const USAGE_LAST_UPDATE_FUTURE_TOLERANCE: Duration = Duration::from_secs(5 * 60);
-
-/// Cluster-wide usage snapshot written by coordinated scanners.
-///
-/// `usage_snapshot_complete` is an additive JSON field: older readers ignore
-/// it, while current readers treat snapshots from older writers as unknown.
-/// Keeping the existing object name preserves rolling-upgrade and rollback
-/// compatibility without allowing an ambiguous snapshot to become authoritative.
-pub const DATA_USAGE_OBJECT_NAME: &str = ".usage.v2.json";
-
-/// Usage snapshot written by scanner implementations predating distributed
-/// leadership fencing. It is read only when neither authoritative snapshot
-/// copy exists.
-// RUSTFS_COMPAT_TODO(scanner-usage-v2): keep .usage.json readable and removable during rolling upgrades from pre-v2 scanners. Remove after supported direct-upgrade sources all write .usage.v2.json.
-pub const LEGACY_DATA_USAGE_OBJECT_NAME: &str = ".usage.json";
-
-/// Returns true when `existing_last_update` is ahead of `now` by more than
-/// [`USAGE_LAST_UPDATE_FUTURE_TOLERANCE`], i.e. the persisted timestamp cannot be
-/// trusted for staleness comparisons and a fresh snapshot save must be allowed.
-pub fn usage_last_update_is_untrusted_future(existing_last_update: SystemTime, now: SystemTime) -> bool {
-    existing_last_update > now + USAGE_LAST_UPDATE_FUTURE_TOLERANCE
-}
-
 #[derive(Clone, Copy, Default, Debug, Serialize, Deserialize, PartialEq)]
 pub struct TierStats {
     pub total_size: u64,
@@ -109,7 +77,7 @@ impl AllTierStats {
 }
 
 /// Bucket target usage info provides replication statistics
-#[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
+#[derive(Debug, Default, Clone, Serialize, Deserialize)]
 pub struct BucketTargetUsageInfo {
     pub replication_pending_size: u64,
     pub replication_failed_size: u64,
@@ -121,7 +89,7 @@ pub struct BucketTargetUsageInfo {
 }
 
 /// Bucket usage info provides bucket-level statistics
-#[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
+#[derive(Debug, Default, Clone, Serialize, Deserialize)]
 pub struct BucketUsageInfo {
     pub size: u64,
     // Following five fields suffixed with V1 are here for backward compatibility
@@ -147,7 +115,7 @@ pub struct BucketUsageInfo {
 }
 
 /// DataUsageInfo represents data usage stats of the underlying storage
-#[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
+#[derive(Debug, Default, Clone, Serialize, Deserialize)]
 pub struct DataUsageInfo {
     /// Total capacity
     pub total_capacity: u64,
@@ -159,22 +127,6 @@ pub struct DataUsageInfo {
     /// LastUpdate is the timestamp of when the data usage info was last updated
     pub last_update: Option<SystemTime>,
 
-    /// Monotonic scanner cycle that produced this complete snapshot.
-    ///
-    /// Older snapshots omit this field and continue to use `last_update` for
-    /// compatibility. New scanner snapshots use the cycle to fence stale
-    /// leaders independently of wall-clock skew.
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    pub scanner_cycle: Option<u64>,
-
-    /// Persisted scanner leadership epoch that produced this snapshot.
-    ///
-    /// The epoch is claimed through the cycle-state CAS before scanning. It
-    /// orders snapshots from different leaders even when their wall clocks or
-    /// cycle counters coincide.
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    pub scanner_epoch: Option<u64>,
-
     /// Objects total count across all buckets
     pub objects_total_count: u64,
     /// Versions total count across all buckets
@@ -190,12 +142,6 @@ pub struct DataUsageInfo {
     pub buckets_count: u64,
     /// Buckets usage info provides following information across all buckets
     pub buckets_usage: HashMap<String, BucketUsageInfo>,
-    /// Whether this snapshot covers the complete bucket namespace.
-    ///
-    /// Legacy snapshots default to `false`. A complete snapshot contains an
-    /// explicit entry for every bucket, including confirmed-empty buckets.
-    #[serde(default)]
-    pub usage_snapshot_complete: bool,
     /// Deprecated kept here for backward compatibility reasons
     pub bucket_sizes: HashMap<String, u64>,
     /// Per-disk snapshot information when available
@@ -204,7 +150,7 @@ pub struct DataUsageInfo {
 }
 
 /// Metadata describing the status of a disk-level data usage snapshot.
-#[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
+#[derive(Debug, Default, Clone, Serialize, Deserialize)]
 pub struct DiskUsageStatus {
     pub disk_id: String,
     pub pool_index: Option<usize>,
@@ -304,30 +250,12 @@ impl DataUsageHash {
 pub type DataUsageHashMap = HashSet<String>;
 
 /// Size histogram for object size distribution
-const SIZE_HISTOGRAM_LEN: usize = 11;
-
-#[derive(Clone, Debug, Serialize)]
+#[derive(Clone, Debug, Serialize, Deserialize)]
 pub struct SizeHistogram(Vec<u64>);
 
 impl Default for SizeHistogram {
     fn default() -> Self {
-        Self(vec![0; SIZE_HISTOGRAM_LEN])
-    }
-}
-
-impl<'de> Deserialize<'de> for SizeHistogram {
-    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
-    where
-        D: serde::Deserializer<'de>,
-    {
-        let values = Vec::<u64>::deserialize(deserializer)?;
-        if values.len() != SIZE_HISTOGRAM_LEN {
-            return Err(serde::de::Error::invalid_length(
-                values.len(),
-                &"exactly 11 object-size histogram buckets",
-            ));
-        }
-        Ok(Self(values))
+        Self(vec![0; 11]) // DATA_USAGE_BUCKET_LEN = 11
     }
 }
 
@@ -397,7 +325,7 @@ impl SizeHistogram {
             .zip(names.iter())
             .filter(|((_, (start, end)), name)| name != &&"BETWEEN_1024B_AND_1_MB" && *start >= 1024 && *end < ONE_MIB)
             .map(|((count, _), _)| *count)
-            .fold(0, u64::saturating_add);
+            .sum();
 
         let mut res = HashMap::new();
         for (count, name) in self.0.iter().zip(names.iter()) {
@@ -418,30 +346,12 @@ impl SizeHistogram {
 }
 
 /// Versions histogram for version count distribution
-const VERSIONS_HISTOGRAM_LEN: usize = 7;
-
-#[derive(Clone, Debug, Serialize)]
+#[derive(Clone, Debug, Serialize, Deserialize)]
 pub struct VersionsHistogram(Vec<u64>);
 
 impl Default for VersionsHistogram {
     fn default() -> Self {
-        Self(vec![0; VERSIONS_HISTOGRAM_LEN])
-    }
-}
-
-impl<'de> Deserialize<'de> for VersionsHistogram {
-    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
-    where
-        D: serde::Deserializer<'de>,
-    {
-        let values = Vec::<u64>::deserialize(deserializer)?;
-        if values.len() != VERSIONS_HISTOGRAM_LEN {
-            return Err(serde::de::Error::invalid_length(
-                values.len(),
-                &"exactly 7 object-version histogram buckets",
-            ));
-        }
-        Ok(Self(values))
+        Self(vec![0; 7]) // DATA_USAGE_VERSION_LEN = 7
     }
 }
 
@@ -506,22 +416,8 @@ pub struct ReplicationStats {
 }
 
 impl ReplicationStats {
-    pub fn is_empty(&self) -> bool {
-        self.pending_size == 0
-            && self.replicated_size == 0
-            && self.failed_size == 0
-            && self.failed_count == 0
-            && self.pending_count == 0
-            && self.missed_threshold_size == 0
-            && self.after_threshold_size == 0
-            && self.missed_threshold_count == 0
-            && self.after_threshold_count == 0
-            && self.replicated_count == 0
-    }
-
-    #[deprecated(note = "use is_empty instead")]
     pub fn empty(&self) -> bool {
-        self.is_empty()
+        self.replicated_size == 0 && self.failed_size == 0 && self.failed_count == 0
     }
 }
 
@@ -534,13 +430,16 @@ pub struct ReplicationAllStats {
 }
 
 impl ReplicationAllStats {
-    pub fn is_empty(&self) -> bool {
-        self.replica_size == 0 && self.replica_count == 0 && self.targets.values().all(ReplicationStats::is_empty)
-    }
-
-    #[deprecated(note = "use is_empty instead")]
     pub fn empty(&self) -> bool {
-        self.is_empty()
+        if self.replica_size != 0 && self.replica_count != 0 {
+            return false;
+        }
+        for v in self.targets.values() {
+            if !v.empty() {
+                return false;
+            }
+        }
+        true
     }
 }
 
@@ -618,74 +517,13 @@ impl DataUsageEntry {
             }
         }
 
-        self.obj_sizes.merge_from(&other.obj_sizes);
-        self.obj_versions.merge_from(&other.obj_versions);
-    }
-
-    pub fn checked_merge(&mut self, other: &DataUsageEntry) -> bool {
-        let scalar_counts_fit = self.objects.checked_add(other.objects).is_some()
-            && self.versions.checked_add(other.versions).is_some()
-            && self.delete_markers.checked_add(other.delete_markers).is_some()
-            && self.size.checked_add(other.size).is_some()
-            && self.failed_objects.checked_add(other.failed_objects).is_some();
-        let histograms_fit = self.obj_sizes.0.len() == SIZE_HISTOGRAM_LEN
-            && other.obj_sizes.0.len() == SIZE_HISTOGRAM_LEN
-            && self.obj_versions.0.len() == VERSIONS_HISTOGRAM_LEN
-            && other.obj_versions.0.len() == VERSIONS_HISTOGRAM_LEN
-            && self
-                .obj_sizes
-                .0
-                .iter()
-                .zip(other.obj_sizes.0.iter())
-                .all(|(left, right)| left.checked_add(*right).is_some())
-            && self
-                .obj_versions
-                .0
-                .iter()
-                .zip(other.obj_versions.0.iter())
-                .all(|(left, right)| left.checked_add(*right).is_some());
-        let replication_fits = match (&self.replication_stats, &other.replication_stats) {
-            (_, None) | (None, Some(_)) => true,
-            (Some(left), Some(right)) => {
-                left.replica_size.checked_add(right.replica_size).is_some()
-                    && left.replica_count.checked_add(right.replica_count).is_some()
-                    && right.targets.iter().all(|(target, right_stats)| {
-                        left.targets.get(target).is_none_or(|left_stats| {
-                            left_stats.pending_size.checked_add(right_stats.pending_size).is_some()
-                                && left_stats.replicated_size.checked_add(right_stats.replicated_size).is_some()
-                                && left_stats.failed_size.checked_add(right_stats.failed_size).is_some()
-                                && left_stats.failed_count.checked_add(right_stats.failed_count).is_some()
-                                && left_stats.pending_count.checked_add(right_stats.pending_count).is_some()
-                                && left_stats
-                                    .missed_threshold_size
-                                    .checked_add(right_stats.missed_threshold_size)
-                                    .is_some()
-                                && left_stats
-                                    .after_threshold_size
-                                    .checked_add(right_stats.after_threshold_size)
-                                    .is_some()
-                                && left_stats
-                                    .missed_threshold_count
-                                    .checked_add(right_stats.missed_threshold_count)
-                                    .is_some()
-                                && left_stats
-                                    .after_threshold_count
-                                    .checked_add(right_stats.after_threshold_count)
-                                    .is_some()
-                                && left_stats
-                                    .replicated_count
-                                    .checked_add(right_stats.replicated_count)
-                                    .is_some()
-                        })
-                    })
-            }
-        };
-
-        if !scalar_counts_fit || !histograms_fit || !replication_fits {
-            return false;
+        for (i, v) in other.obj_sizes.0.iter().enumerate() {
+            self.obj_sizes.0[i] += v;
+        }
+
+        for (i, v) in other.obj_versions.0.iter().enumerate() {
+            self.obj_versions.0[i] += v;
         }
-        self.merge(other);
-        true
     }
 }
 
@@ -698,12 +536,6 @@ pub struct DataUsageCacheInfo {
     pub skip_healing: bool,
     #[serde(default)]
     pub failed_objects: HashMap<String, u64>,
-    /// Whether this per-set cache was produced by a completed scanner pass.
-    ///
-    /// Older cache writers omit this field and therefore deserialize as
-    /// incomplete instead of exposing partial set totals as confirmed zeros.
-    #[serde(default)]
-    pub snapshot_complete: bool,
 }
 
 /// Data usage cache
@@ -794,7 +626,7 @@ impl DataUsageCache {
                     return Some(root);
                 }
                 let mut flat = self.flatten(&root);
-                if flat.replication_stats.as_ref().is_some_and(ReplicationAllStats::is_empty) {
+                if flat.replication_stats.as_ref().is_some_and(|stats| stats.empty()) {
                     flat.replication_stats = None;
                 }
                 Some(flat)
@@ -1023,7 +855,6 @@ impl DataUsageCache {
             objects_total_size: flat.size as u64,
             buckets_count: u64::try_from(buckets.len()).unwrap_or(u64::MAX),
             buckets_usage,
-            usage_snapshot_complete: self.info.snapshot_complete,
             ..Default::default()
         }
     }
@@ -1102,13 +933,6 @@ impl DataUsageInfo {
         Self::default()
     }
 
-    /// Whether this snapshot authoritatively covers every reported bucket.
-    pub fn is_complete_bucket_usage_snapshot(&self) -> bool {
-        self.usage_snapshot_complete
-            && self.last_update.is_some()
-            && u64::try_from(self.buckets_usage.len()).ok() == Some(self.buckets_count)
-    }
-
     /// Add object metadata to data usage statistics
     pub fn add_object(&mut self, object_path: &str, meta_object: &rustfs_filemeta::MetaObject) {
         // This method is kept for backward compatibility
@@ -1473,51 +1297,6 @@ pub struct CompressionTotalInfo {
 mod tests {
     use super::*;
 
-    #[derive(Deserialize)]
-    struct LegacyUsageReader {
-        buckets_count: u64,
-    }
-
-    #[test]
-    fn completeness_marker_is_additive_for_legacy_named_readers() {
-        let current = DataUsageInfo {
-            last_update: Some(SystemTime::UNIX_EPOCH),
-            usage_snapshot_complete: true,
-            ..Default::default()
-        };
-        let encoded = rmp_serde::to_vec_named(&current).expect("encode current data usage snapshot");
-        let legacy: LegacyUsageReader = rmp_serde::from_slice(&encoded).expect("legacy reader should ignore additive fields");
-
-        assert_eq!(legacy.buckets_count, 0);
-        assert!(current.is_complete_bucket_usage_snapshot());
-    }
-
-    #[test]
-    fn completeness_marker_requires_a_snapshot_timestamp() {
-        let untimestamped = DataUsageInfo {
-            usage_snapshot_complete: true,
-            ..Default::default()
-        };
-
-        assert!(!untimestamped.is_complete_bucket_usage_snapshot());
-    }
-
-    #[test]
-    fn test_usage_last_update_future_tolerance_boundary() {
-        let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
-
-        // Within tolerance (including the exact boundary) the timestamp is trusted.
-        assert!(!usage_last_update_is_untrusted_future(now, now));
-        assert!(!usage_last_update_is_untrusted_future(now - Duration::from_secs(60), now));
-        assert!(!usage_last_update_is_untrusted_future(now + USAGE_LAST_UPDATE_FUTURE_TOLERANCE, now));
-
-        // Beyond tolerance the persisted timestamp is untrustworthy.
-        assert!(usage_last_update_is_untrusted_future(
-            now + USAGE_LAST_UPDATE_FUTURE_TOLERANCE + Duration::from_secs(1),
-            now
-        ));
-    }
-
     #[test]
     fn test_data_usage_info_creation() {
         let mut info = DataUsageInfo::new();
@@ -1582,139 +1361,6 @@ mod tests {
         assert_eq!(map["BETWEEN_512_KB_AND_1_MB"], 1);
     }
 
-    #[test]
-    fn test_size_histogram_compat_rollup_saturates_on_corrupt_counts() {
-        let mut hist = SizeHistogram::default();
-        hist.0[1] = u64::MAX;
-        hist.0[2] = 1;
-
-        let map = hist.to_map();
-
-        assert_eq!(map["BETWEEN_1024B_AND_1_MB"], u64::MAX);
-    }
-
-    #[test]
-    #[allow(deprecated)]
-    fn replication_stats_empty_checks_every_field() {
-        type SetField = fn(&mut ReplicationStats);
-
-        let cases: [(&str, SetField); 10] = [
-            ("pending_size", |stats| stats.pending_size = 1),
-            ("replicated_size", |stats| stats.replicated_size = 1),
-            ("failed_size", |stats| stats.failed_size = 1),
-            ("failed_count", |stats| stats.failed_count = 1),
-            ("pending_count", |stats| stats.pending_count = 1),
-            ("missed_threshold_size", |stats| stats.missed_threshold_size = 1),
-            ("after_threshold_size", |stats| stats.after_threshold_size = 1),
-            ("missed_threshold_count", |stats| stats.missed_threshold_count = 1),
-            ("after_threshold_count", |stats| stats.after_threshold_count = 1),
-            ("replicated_count", |stats| stats.replicated_count = 1),
-        ];
-
-        assert!(ReplicationStats::default().empty());
-        for (field, set_nonzero) in cases {
-            let mut stats = ReplicationStats::default();
-            set_nonzero(&mut stats);
-            assert!(!stats.empty(), "{field} must make replication stats non-empty");
-        }
-    }
-
-    #[test]
-    #[allow(deprecated)]
-    fn replication_all_stats_empty_checks_aggregate_fields_independently() {
-        let cases = [
-            (
-                "replica_size",
-                ReplicationAllStats {
-                    replica_size: 1,
-                    ..Default::default()
-                },
-            ),
-            (
-                "replica_count",
-                ReplicationAllStats {
-                    replica_count: 1,
-                    ..Default::default()
-                },
-            ),
-        ];
-
-        assert!(ReplicationAllStats::default().empty());
-        for (field, stats) in cases {
-            assert!(!stats.empty(), "{field} must make aggregate replication stats non-empty");
-        }
-
-        let empty_targets = ReplicationAllStats {
-            targets: HashMap::from([("arn:test:empty".to_string(), ReplicationStats::default())]),
-            ..Default::default()
-        };
-        assert!(empty_targets.empty(), "all-empty targets must keep aggregate stats empty");
-
-        let stats = ReplicationAllStats {
-            targets: HashMap::from([
-                ("arn:test:empty".to_string(), ReplicationStats::default()),
-                (
-                    "arn:test:non-empty".to_string(),
-                    ReplicationStats {
-                        pending_count: 1,
-                        ..Default::default()
-                    },
-                ),
-            ]),
-            ..Default::default()
-        };
-        assert!(!stats.empty(), "a non-empty target must make aggregate replication stats non-empty");
-    }
-
-    #[test]
-    fn size_recursive_prunes_empty_and_preserves_pending_replication_stats() {
-        let root = hash_path("bucket");
-        let child = hash_path("bucket/child");
-        let mut cache = DataUsageCache::default();
-        cache.replace_hashed(&root, &None, &DataUsageEntry::default());
-        cache.replace_hashed(
-            &child,
-            &Some(root.clone()),
-            &DataUsageEntry {
-                replication_stats: Some(ReplicationAllStats::default()),
-                ..Default::default()
-            },
-        );
-
-        assert!(
-            cache
-                .size_recursive("bucket")
-                .expect("bucket usage should flatten")
-                .replication_stats
-                .is_none()
-        );
-
-        cache.replace_hashed(
-            &child,
-            &Some(root.clone()),
-            &DataUsageEntry {
-                replication_stats: Some(ReplicationAllStats {
-                    targets: HashMap::from([(
-                        "arn:test:pending".to_string(),
-                        ReplicationStats {
-                            pending_count: 1,
-                            ..Default::default()
-                        },
-                    )]),
-                    ..Default::default()
-                }),
-                ..Default::default()
-            },
-        );
-
-        let flattened = cache.size_recursive("bucket").expect("bucket usage should flatten");
-        let replication = flattened
-            .replication_stats
-            .expect("pending-only replication stats must survive pruning");
-
-        assert_eq!(replication.targets["arn:test:pending"].pending_count, 1);
-    }
-
     #[test]
     fn test_data_usage_cache_merge_adds_missing_child() {
         let mut base = DataUsageCache::default();
@@ -2028,86 +1674,4 @@ mod tests {
         assert!(cache.find("bucket/large/a").is_some());
         assert!(cache.find("bucket/large/b").is_some());
     }
-
-    #[test]
-    fn checked_merge_rejects_scalar_and_replication_overflow_without_mutation() {
-        let mut entry = DataUsageEntry {
-            objects: usize::MAX,
-            replication_stats: Some(ReplicationAllStats {
-                replica_size: 7,
-                ..Default::default()
-            }),
-            ..Default::default()
-        };
-        let other = DataUsageEntry {
-            objects: 1,
-            replication_stats: Some(ReplicationAllStats {
-                replica_size: u64::MAX,
-                ..Default::default()
-            }),
-            ..Default::default()
-        };
-
-        assert!(!entry.checked_merge(&other));
-        assert_eq!(entry.objects, usize::MAX);
-        assert_eq!(entry.replication_stats.as_ref().map(|stats| stats.replica_size), Some(7));
-    }
-
-    #[test]
-    fn checked_merge_accepts_valid_usage() {
-        let mut entry = DataUsageEntry {
-            objects: 2,
-            size: 20,
-            ..Default::default()
-        };
-        let other = DataUsageEntry {
-            objects: 3,
-            size: 30,
-            ..Default::default()
-        };
-
-        assert!(entry.checked_merge(&other));
-        assert_eq!(entry.objects, 5);
-        assert_eq!(entry.size, 50);
-    }
-
-    #[test]
-    fn histogram_deserialization_rejects_noncanonical_lengths() {
-        let invalid_sizes =
-            rmp_serde::to_vec(&vec![0_u64; SIZE_HISTOGRAM_LEN + 1]).expect("encode invalid object-size histogram fixture");
-        let invalid_versions =
-            rmp_serde::to_vec(&vec![0_u64; VERSIONS_HISTOGRAM_LEN - 1]).expect("encode invalid object-version histogram fixture");
-
-        assert!(rmp_serde::from_slice::<SizeHistogram>(&invalid_sizes).is_err());
-        assert!(rmp_serde::from_slice::<VersionsHistogram>(&invalid_versions).is_err());
-    }
-
-    #[test]
-    fn replication_target_deserialization_preserves_large_historical_maps() {
-        let mut stats = ReplicationAllStats::default();
-        for index in 0..=1024 {
-            stats.targets.insert(format!("target-{index}"), ReplicationStats::default());
-        }
-        let encoded = rmp_serde::to_vec_named(&stats).expect("large replication target fixture should encode");
-        let decoded = rmp_serde::from_slice::<ReplicationAllStats>(&encoded)
-            .expect("historical replication target maps must remain readable");
-
-        assert_eq!(decoded.targets.len(), stats.targets.len());
-    }
-
-    #[test]
-    fn checked_merge_rejects_noncanonical_histograms_without_mutation() {
-        let mut entry = DataUsageEntry {
-            objects: 2,
-            ..Default::default()
-        };
-        let other = DataUsageEntry {
-            objects: 3,
-            obj_sizes: SizeHistogram(vec![0; SIZE_HISTOGRAM_LEN + 1]),
-            ..Default::default()
-        };
-
-        assert!(!entry.checked_merge(&other));
-        assert_eq!(entry.objects, 2);
-    }
 }

crates/e2e_test/Cargo.toml

@@ -30,60 +30,52 @@ sftp = []
 
 [dependencies]
 rustfs-config = { workspace = true, features = ["constants"] }
-rustfs-credentials.workspace = true
 rustfs-ecstore.workspace = true
 rustfs-data-usage.workspace = true
 rustfs-rio.workspace = true
-rustfs-utils = { workspace = true, features = ["egress"] }
 flatbuffers.workspace = true
 futures.workspace = true
 rustfs-lock.workspace = true
 rustfs-protos.workspace = true
 rmp-serde.workspace = true
-serde = { workspace = true, features = ["derive"] }
-serde_json = { workspace = true, features = ["raw_value"] }
-tonic = { workspace = true, features = ["gzip", "deflate"] }
-tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
+serde.workspace = true
+serde_json.workspace = true
+tonic = { workspace = true }
+tokio = { workspace = true }
 tokio-stream = { workspace = true }
 rustfs-madmin.workspace = true
 rustfs-filemeta.workspace = true
-bytes = { workspace = true, features = ["serde"] }
+bytes.workspace = true
 serial_test = { workspace = true }
-aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
-aws-sdk-sts = { workspace = true, default-features = false, features = ["default-https-client", "rt-tokio"] }
+aws-sdk-s3.workspace = true
 aws-config = { workspace = true }
-aws-smithy-http-client = { workspace = true, default-features = false, features = ["rustls-aws-lc"] }
+aws-smithy-http-client.workspace = true
 async-compression = { workspace = true, features = ["tokio", "bzip2", "xz"] }
 async-trait = { workspace = true }
 flate2.workspace = true
 http.workspace = true
-http-body-util.workspace = true
-hyper = { workspace = true, features = ["http2", "http1", "server"] }
-hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] }
-reqwest = { workspace = true, features = ["json", "multipart", "stream"] }
+reqwest = { workspace = true }
 rustfs-signer.workspace = true
 tracing = { workspace = true }
-tracing-subscriber = { workspace = true, features = ["env-filter", "time"] }
-uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
+tracing-subscriber = { workspace = true }
+uuid = { workspace = true }
 urlencoding.workspace = true
 walkdir.workspace = true
 base64 = { workspace = true }
-rand = { workspace = true, features = ["serde"] }
-chrono = { workspace = true, features = ["serde"] }
+rand = { workspace = true }
+chrono = { workspace = true }
 md5 = { workspace = true }
-opentelemetry-proto = { workspace = true }
-prost.workspace = true
 sha2 = { workspace = true }
 astral-tokio-tar = { workspace = true }
-s3s = { workspace = true, features = ["minio"] }
+s3s = { workspace = true }
 zstd.workspace = true
-time = { workspace = true, features = ["parsing", "formatting", "macros", "serde"] }
-suppaftp = { workspace = true, features = ["rustls-aws-lc-rs", "tokio-rustls-aws-lc-rs"] }
+time.workspace = true
+suppaftp = { workspace = true, features = ["tokio", "rustls-aws-lc-rs"] }
 rcgen.workspace = true
 local-ip-address.workspace = true
 anyhow.workspace = true
-rustls = { workspace = true, default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
-russh = { workspace = true, features = ["serde"] }
+rustls.workspace = true
+russh = { workspace = true }
 russh-sftp = { workspace = true }
 zip.workspace = true
-clap = { workspace = true, features = ["derive", "env"] }
+clap.workspace = true

crates/e2e_test/src/admin_auth_test.rs

@@ -46,45 +46,6 @@ mod tests {
     use std::time::{Duration, Instant};
 
     const ADMIN_INFO_PATH: &str = "/rustfs/admin/v3/info";
-    const ADMIN_MANUAL_TRANSITION_BUCKET: &str = "auth-deny-manual-transition";
-    const ADMIN_MANUAL_TRANSITION_PATH: &str =
-        "/rustfs/admin/v3/ilm/transition/run?bucket=auth-deny-manual-transition&maxObjects=1&mode=async";
-
-    fn assert_no_raw_manual_transition_markers(body: &str, context: &str) {
-        assert!(
-            !body.contains("\"marker\"") && !body.contains("\"versionMarker\"") && !body.contains("\"version_marker\""),
-            "{context} must not expose raw manual transition resume markers, body: {body}"
-        );
-    }
-
-    async fn wait_for_terminal_manual_transition_job(
-        env: &RustFSTestEnvironment,
-        status_endpoint: &str,
-    ) -> Result<String, Box<dyn Error + Send + Sync>> {
-        let deadline = Instant::now() + Duration::from_secs(30);
-        loop {
-            let (status, body) =
-                signed_request(&env.url, http::Method::GET, status_endpoint, None, &env.access_key, &env.secret_key).await?;
-            assert_eq!(
-                status,
-                reqwest::StatusCode::OK,
-                "root credential must query manual transition job status, body: {body}"
-            );
-            assert_no_raw_manual_transition_markers(&body, "manual transition status response");
-            let value: serde_json::Value = serde_json::from_str(&body)?;
-            let job_status = value
-                .get("status")
-                .and_then(serde_json::Value::as_str)
-                .ok_or("manual transition job status response must include status")?;
-            if matches!(job_status, "completed" | "partial" | "cancelled" | "failed" | "unknown") {
-                return Ok(body);
-            }
-            if Instant::now() >= deadline {
-                return Err(format!("manual transition job did not reach terminal status within 30s; last={body}").into());
-            }
-            tokio::time::sleep(Duration::from_millis(250)).await;
-        }
-    }
 
     /// Send a SigV4-signed request to `path` (optionally with a JSON `body`) and
     /// return `(status, body)`. Uses the `UNSIGNED_PAYLOAD` content hash so a
@@ -197,130 +158,6 @@ mod tests {
         Ok(())
     }
 
-    #[tokio::test(flavor = "multi_thread")]
-    #[serial]
-    async fn non_admin_credential_denied_on_manual_transition_run() -> Result<(), Box<dyn Error + Send + Sync>> {
-        init_logging();
-        let mut env = RustFSTestEnvironment::new().await?;
-        env.start_rustfs_server(vec![]).await?;
-
-        let user_ak = "ilmtransitionlimited";
-        let user_sk = "ilmtransitionlimitedsecret";
-        create_limited_user(&env, user_ak, user_sk).await?;
-        env.create_s3_client()
-            .create_bucket()
-            .bucket(ADMIN_MANUAL_TRANSITION_BUCKET)
-            .send()
-            .await?;
-
-        let (root_status, root_body) = signed_request(
-            &env.url,
-            http::Method::POST,
-            ADMIN_MANUAL_TRANSITION_PATH,
-            None,
-            &env.access_key,
-            &env.secret_key,
-        )
-        .await?;
-        assert_eq!(
-            root_status,
-            reqwest::StatusCode::ACCEPTED,
-            "root credential must reach the manual transition handler, body: {root_body}"
-        );
-        assert!(
-            root_body.contains("\"mode\":\"durable_job\""),
-            "root response should be the durable manual transition JSON contract, body: {root_body}"
-        );
-        assert_no_raw_manual_transition_markers(&root_body, "manual transition run response");
-        let root_value: serde_json::Value = serde_json::from_str(&root_body)?;
-        let job_id = root_value
-            .get("job_id")
-            .and_then(serde_json::Value::as_str)
-            .ok_or("manual transition async response must include job_id")?;
-        let status_endpoint = root_value
-            .get("status_endpoint")
-            .and_then(serde_json::Value::as_str)
-            .ok_or("manual transition async response must include status_endpoint")?;
-        let cancel_endpoint = root_value
-            .get("cancel_endpoint")
-            .and_then(serde_json::Value::as_str)
-            .ok_or("manual transition async response must include cancel_endpoint")?;
-        assert_eq!(
-            cancel_endpoint, status_endpoint,
-            "manual transition durable jobs currently use the same status/cancel endpoint"
-        );
-        assert!(
-            status_endpoint.ends_with(job_id),
-            "status endpoint must address the returned job id, job_id={job_id}, status_endpoint={status_endpoint}"
-        );
-
-        let terminal_body = wait_for_terminal_manual_transition_job(&env, status_endpoint).await?;
-        let terminal: serde_json::Value = serde_json::from_str(&terminal_body)?;
-        assert_eq!(terminal.get("job_id").and_then(serde_json::Value::as_str), Some(job_id));
-        assert_eq!(
-            terminal
-                .get("report")
-                .and_then(|report| report.get("bucket"))
-                .and_then(serde_json::Value::as_str),
-            Some(ADMIN_MANUAL_TRANSITION_BUCKET)
-        );
-
-        let (root_status, root_body) =
-            signed_request(&env.url, http::Method::DELETE, status_endpoint, None, &env.access_key, &env.secret_key).await?;
-        assert_eq!(
-            root_status,
-            reqwest::StatusCode::OK,
-            "root credential must cancel/query a terminal manual transition job idempotently, body: {root_body}"
-        );
-        assert_no_raw_manual_transition_markers(&root_body, "manual transition cancel response");
-        let root_cancel: serde_json::Value = serde_json::from_str(&root_body)?;
-        assert_eq!(root_cancel.get("job_id").and_then(serde_json::Value::as_str), Some(job_id));
-        assert!(
-            matches!(
-                root_cancel.get("status").and_then(serde_json::Value::as_str),
-                Some("completed" | "partial" | "failed" | "unknown")
-            ),
-            "terminal cancel must not rewrite the job into cancelled state, body: {root_body}"
-        );
-
-        let (status, body) =
-            signed_request(&env.url, http::Method::POST, ADMIN_MANUAL_TRANSITION_PATH, None, user_ak, user_sk).await?;
-        assert_eq!(
-            status,
-            reqwest::StatusCode::FORBIDDEN,
-            "non-admin credential must get 403 on manual transition run, body: {body}"
-        );
-        assert!(
-            body.contains("AccessDenied"),
-            "manual transition rejection must carry the AccessDenied S3 error code, body: {body}"
-        );
-        let (status, body) = signed_request(&env.url, http::Method::GET, status_endpoint, None, user_ak, user_sk).await?;
-        assert_eq!(
-            status,
-            reqwest::StatusCode::FORBIDDEN,
-            "non-admin credential must get 403 on manual transition status, body: {body}"
-        );
-        assert_no_raw_manual_transition_markers(&body, "manual transition status rejection");
-        assert!(
-            body.contains("AccessDenied"),
-            "manual transition status rejection must carry the AccessDenied S3 error code, body: {body}"
-        );
-        let (status, body) = signed_request(&env.url, http::Method::DELETE, status_endpoint, None, user_ak, user_sk).await?;
-        assert_eq!(
-            status,
-            reqwest::StatusCode::FORBIDDEN,
-            "non-admin credential must get 403 on manual transition cancel, body: {body}"
-        );
-        assert_no_raw_manual_transition_markers(&body, "manual transition cancel rejection");
-        assert!(
-            body.contains("AccessDenied"),
-            "manual transition cancel rejection must carry the AccessDenied S3 error code, body: {body}"
-        );
-
-        env.stop_server();
-        Ok(())
-    }
-
     /// Rotating the root credentials (restart with new `--access-key` /
     /// `--secret-key` on the same data directory) takes effect: the new
     /// credential is accepted and the old one is rejected, on both the S3 data

crates/e2e_test/src/admin_iam_crud_test.rs

@@ -1,448 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Systematic HTTP e2e for the admin IAM management surface (backlog#1154
-//! peri-2, first batch): the full user / canned-policy / service-account CRUD
-//! lifecycle over real signed HTTP against a real binary, plus a non-admin
-//! denial probe for every management endpoint (reusing the sec-4 assertion
-//! pattern — the authorization gate itself is pinned by `admin_auth_test`).
-//!
-//! Before this suite the ~40 admin handler modules only had helper-level unit
-//! tests; no test proved that `mc admin user add` style flows work end to end
-//! (create -> attach policy -> the credential actually gains S3 access ->
-//! service account inherits it -> deletion revokes it).
-//!
-//! Later batches tracked on backlog#1154: config get/set, info, pools status,
-//! group lifecycle, import/export IAM.
-
-use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
-use aws_sdk_s3::config::{Credentials, Region};
-use aws_sdk_s3::primitives::ByteStream;
-use aws_sdk_s3::{Client, Config};
-use http::header::{CONTENT_TYPE, HOST};
-use reqwest::StatusCode;
-use rustfs_signer::constants::UNSIGNED_PAYLOAD;
-use rustfs_signer::sign_v4;
-use s3s::Body;
-use serial_test::serial;
-use std::error::Error;
-use tokio::time::{Duration, sleep};
-
-type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-type BoxError = Box<dyn Error + Send + Sync>;
-
-/// Signs and sends an admin HTTP request with the given credential, returning
-/// status and body. Native `/rustfs/admin/v3` requests and responses are plain
-/// JSON (the MinIO-compat encryption applies only to `/minio/admin/v3` paths).
-async fn admin_request(
-    base_url: &str,
-    method: http::Method,
-    path_and_query: &str,
-    body: Option<String>,
-    access_key: &str,
-    secret_key: &str,
-) -> Result<(StatusCode, String), BoxError> {
-    let url = format!("{base_url}{path_and_query}");
-    let uri = url.parse::<http::Uri>()?;
-    let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
-    let mut builder = http::Request::builder()
-        .method(method.clone())
-        .uri(uri)
-        .header(HOST, authority)
-        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
-    if body.is_some() {
-        builder = builder.header(CONTENT_TYPE, "application/json");
-    }
-
-    let content_len = body.as_ref().map(|b| b.len() as i64).unwrap_or_default();
-    let signed = sign_v4(builder.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1");
-
-    let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
-    let mut request = local_http_client().request(reqwest_method, &url);
-    for (name, value) in signed.headers() {
-        request = request.header(name, value);
-    }
-    if let Some(body) = body {
-        request = request.body(body);
-    }
-    let response = request.send().await?;
-    let status = response.status();
-    let text = response.text().await.unwrap_or_default();
-    Ok((status, text))
-}
-
-/// Root-credential admin request that must succeed; returns the response body.
-async fn admin_ok(
-    env: &RustFSTestEnvironment,
-    method: http::Method,
-    path_and_query: &str,
-    body: Option<String>,
-) -> Result<String, BoxError> {
-    let (status, text) = admin_request(&env.url, method.clone(), path_and_query, body, &env.access_key, &env.secret_key).await?;
-    if !status.is_success() {
-        return Err(format!("{method} {path_and_query} failed: {status} {text}").into());
-    }
-    Ok(text)
-}
-
-fn build_s3_client(url: &str, access_key: &str, secret_key: &str) -> Client {
-    let config = Config::builder()
-        .credentials_provider(Credentials::new(access_key, secret_key, None, None, "e2e-admin-crud"))
-        .region(Region::new("us-east-1"))
-        .endpoint_url(url)
-        .force_path_style(true)
-        .behavior_version_latest()
-        .build();
-    Client::from_conf(config)
-}
-
-/// Retries an S3 PUT until IAM propagation makes it succeed (bounded).
-async fn wait_for_s3_put(client: &Client, bucket: &str, key: &str, within: Duration) -> TestResult {
-    let deadline = tokio::time::Instant::now() + within;
-    loop {
-        match client
-            .put_object()
-            .bucket(bucket)
-            .key(key)
-            .body(ByteStream::from_static(b"peri-2 probe"))
-            .send()
-            .await
-        {
-            Ok(_) => return Ok(()),
-            Err(err) => {
-                if tokio::time::Instant::now() >= deadline {
-                    return Err(format!("PUT {bucket}/{key} never succeeded: {err:?}").into());
-                }
-            }
-        }
-        sleep(Duration::from_millis(500)).await;
-    }
-}
-
-/// A canned policy granting full S3 access to one bucket.
-fn bucket_rw_policy(bucket: &str) -> String {
-    serde_json::json!({
-        "Version": "2012-10-17",
-        "Statement": [{
-            "Effect": "Allow",
-            "Action": ["s3:*"],
-            "Resource": [
-                format!("arn:aws:s3:::{bucket}"),
-                format!("arn:aws:s3:::{bucket}/*")
-            ]
-        }]
-    })
-    .to_string()
-}
-
-/// Full user -> policy -> service-account lifecycle, proving each management
-/// call takes effect on the data plane, not just that the endpoint answers 200.
-#[tokio::test]
-#[serial]
-async fn test_admin_user_policy_service_account_crud_lifecycle() -> TestResult {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let bucket = "peri2-crud";
-    let user = "peri2user";
-    let user_secret = "peri2usersecret";
-    let policy = "peri2-rw";
-
-    let root_client = env.create_s3_client();
-    root_client.create_bucket().bucket(bucket).send().await?;
-
-    // --- canned policy CRUD ---------------------------------------------------
-    admin_ok(
-        &env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-canned-policy?name={policy}"),
-        Some(bucket_rw_policy(bucket)),
-    )
-    .await?;
-
-    let info = admin_ok(
-        &env,
-        http::Method::GET,
-        &format!("/rustfs/admin/v3/info-canned-policy?name={policy}"),
-        None,
-    )
-    .await?;
-    let info_json: serde_json::Value = serde_json::from_str(&info)?;
-    let statement_json = info_json.get("Policy").cloned().unwrap_or(info_json.clone());
-    assert!(
-        statement_json.to_string().contains("s3:*"),
-        "info-canned-policy must round-trip the policy document: {info}"
-    );
-
-    let listed = admin_ok(&env, http::Method::GET, "/rustfs/admin/v3/list-canned-policies", None).await?;
-    assert!(listed.contains(policy), "list-canned-policies must contain {policy}: {listed}");
-
-    // --- user CRUD --------------------------------------------------------------
-    admin_ok(
-        &env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-user?accessKey={user}"),
-        Some(serde_json::json!({ "secretKey": user_secret, "status": "enabled" }).to_string()),
-    )
-    .await?;
-
-    let users = admin_ok(&env, http::Method::GET, "/rustfs/admin/v3/list-users", None).await?;
-    assert!(users.contains(user), "list-users must contain {user}: {users}");
-
-    let user_info = admin_ok(&env, http::Method::GET, &format!("/rustfs/admin/v3/user-info?accessKey={user}"), None).await?;
-    let user_info: rustfs_madmin::UserInfo = serde_json::from_str(&user_info)
-        .map_err(|e| format!("user-info response must decode as rustfs_madmin::UserInfo: {e}"))?;
-    assert_eq!(user_info.status, rustfs_madmin::AccountStatus::Enabled);
-
-    // The fresh user is authenticatable but unauthorized for the bucket.
-    let user_client = build_s3_client(&env.url, user, user_secret);
-    let denied = user_client
-        .put_object()
-        .bucket(bucket)
-        .key("before-attach")
-        .body(ByteStream::from_static(b"x"))
-        .send()
-        .await;
-    assert!(denied.is_err(), "user without a policy must not be able to write to {bucket}");
-
-    // --- attach policy: the credential actually gains S3 access -----------------
-    admin_ok(
-        &env,
-        http::Method::POST,
-        "/rustfs/admin/v3/idp/builtin/policy/attach",
-        Some(serde_json::json!({ "policies": [policy], "user": user }).to_string()),
-    )
-    .await?;
-
-    wait_for_s3_put(&user_client, bucket, "after-attach.txt", Duration::from_secs(10)).await?;
-
-    let user_info = admin_ok(&env, http::Method::GET, &format!("/rustfs/admin/v3/user-info?accessKey={user}"), None).await?;
-    let user_info: rustfs_madmin::UserInfo = serde_json::from_str(&user_info)?;
-    assert!(
-        user_info.policy_name.as_deref().is_some_and(|p| p.contains(policy)),
-        "user-info must reflect the attached policy, got {:?}",
-        user_info.policy_name
-    );
-
-    // --- service account: create for the user, credential works, info/list pin it
-    let sa_resp = admin_ok(
-        &env,
-        http::Method::PUT,
-        "/rustfs/admin/v3/add-service-accounts",
-        Some(serde_json::json!({ "targetUser": user }).to_string()),
-    )
-    .await?;
-    let sa_json: serde_json::Value = serde_json::from_str(&sa_resp)?;
-    let sa_ak = sa_json["credentials"]["accessKey"]
-        .as_str()
-        .ok_or(format!("add-service-accounts response missing credentials.accessKey: {sa_resp}"))?
-        .to_string();
-    let sa_sk = sa_json["credentials"]["secretKey"]
-        .as_str()
-        .ok_or(format!("add-service-accounts response missing credentials.secretKey: {sa_resp}"))?
-        .to_string();
-
-    let sa_client = build_s3_client(&env.url, &sa_ak, &sa_sk);
-    wait_for_s3_put(&sa_client, bucket, "via-service-account.txt", Duration::from_secs(10)).await?;
-
-    let sa_info = admin_ok(
-        &env,
-        http::Method::GET,
-        &format!("/rustfs/admin/v3/info-service-account?accessKey={sa_ak}"),
-        None,
-    )
-    .await?;
-    let sa_info: serde_json::Value = serde_json::from_str(&sa_info)?;
-    assert_eq!(
-        sa_info["parentUser"].as_str(),
-        Some(user),
-        "info-service-account must name the parent user: {sa_info}"
-    );
-
-    let sa_list = admin_ok(
-        &env,
-        http::Method::GET,
-        &format!("/rustfs/admin/v3/list-service-accounts?user={user}"),
-        None,
-    )
-    .await?;
-    let sa_list: rustfs_madmin::ListServiceAccountsResp = serde_json::from_str(&sa_list)
-        .map_err(|e| format!("list-service-accounts must decode as rustfs_madmin::ListServiceAccountsResp: {e}"))?;
-    assert!(
-        sa_list.accounts.iter().any(|a| a.access_key == sa_ak),
-        "list-service-accounts must contain {sa_ak}"
-    );
-
-    // --- deletion revokes access -------------------------------------------------
-    admin_ok(
-        &env,
-        http::Method::DELETE,
-        &format!("/rustfs/admin/v3/delete-service-account?accessKey={sa_ak}"),
-        None,
-    )
-    .await?;
-    let deadline = tokio::time::Instant::now() + Duration::from_secs(10);
-    loop {
-        let revoked = sa_client
-            .put_object()
-            .bucket(bucket)
-            .key("after-sa-delete")
-            .body(ByteStream::from_static(b"x"))
-            .send()
-            .await;
-        if revoked.is_err() {
-            break;
-        }
-        if tokio::time::Instant::now() >= deadline {
-            return Err("deleted service account credential still works".into());
-        }
-        sleep(Duration::from_millis(500)).await;
-    }
-
-    // Disable then remove the user; the credential must stop working.
-    admin_ok(
-        &env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/set-user-status?accessKey={user}&status=disabled"),
-        None,
-    )
-    .await?;
-    let deadline = tokio::time::Instant::now() + Duration::from_secs(10);
-    loop {
-        let disabled = user_client
-            .put_object()
-            .bucket(bucket)
-            .key("after-disable")
-            .body(ByteStream::from_static(b"x"))
-            .send()
-            .await;
-        if disabled.is_err() {
-            break;
-        }
-        if tokio::time::Instant::now() >= deadline {
-            return Err("disabled user credential still works".into());
-        }
-        sleep(Duration::from_millis(500)).await;
-    }
-
-    admin_ok(
-        &env,
-        http::Method::DELETE,
-        &format!("/rustfs/admin/v3/remove-user?accessKey={user}"),
-        None,
-    )
-    .await?;
-    let users = admin_ok(&env, http::Method::GET, "/rustfs/admin/v3/list-users", None).await?;
-    assert!(!users.contains(user), "removed user must disappear from list-users: {users}");
-
-    admin_ok(
-        &env,
-        http::Method::DELETE,
-        &format!("/rustfs/admin/v3/remove-canned-policy?name={policy}"),
-        None,
-    )
-    .await?;
-    let (status, _) = admin_request(
-        &env.url,
-        http::Method::GET,
-        &format!("/rustfs/admin/v3/info-canned-policy?name={policy}"),
-        None,
-        &env.access_key,
-        &env.secret_key,
-    )
-    .await?;
-    assert!(!status.is_success(), "info-canned-policy must fail after remove-canned-policy");
-
-    env.stop_server();
-    Ok(())
-}
-
-/// Every management endpoint in the first batch must deny an authenticated but
-/// non-admin credential with 403 AccessDenied (sec-4 assertion pattern; the
-/// gate implementation itself is owned by sec-4 / admin_auth_test).
-#[tokio::test]
-#[serial]
-async fn test_admin_iam_endpoints_deny_non_admin_credential() -> TestResult {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let limited = "peri2limited";
-    let limited_secret = "peri2limitedsecret";
-    let other = "peri2other";
-    for (user, secret) in [(limited, limited_secret), (other, "peri2othersecret")] {
-        admin_ok(
-            &env,
-            http::Method::PUT,
-            &format!("/rustfs/admin/v3/add-user?accessKey={user}"),
-            Some(serde_json::json!({ "secretKey": secret, "status": "enabled" }).to_string()),
-        )
-        .await?;
-    }
-
-    // Admin-only management endpoints. Targeted probes aim at a DIFFERENT user
-    // (`other`): operations a principal performs on itself run in `deny_only`
-    // mode (see should_check_deny_only in rustfs/src/admin/handlers/user.rs)
-    // and would not be denied. Also deliberately excludes the self-service
-    // account endpoints (add/list service accounts), which any authenticated
-    // user may call for themselves.
-    let probes: Vec<(http::Method, String, Option<String>)> = vec![
-        (
-            http::Method::PUT,
-            "/rustfs/admin/v3/add-user?accessKey=peri2evil".to_string(),
-            Some(serde_json::json!({ "secretKey": "evilsecret123", "status": "enabled" }).to_string()),
-        ),
-        (http::Method::GET, "/rustfs/admin/v3/list-users".to_string(), None),
-        (http::Method::GET, format!("/rustfs/admin/v3/user-info?accessKey={other}"), None),
-        (
-            http::Method::PUT,
-            format!("/rustfs/admin/v3/set-user-status?accessKey={other}&status=disabled"),
-            None,
-        ),
-        (http::Method::DELETE, format!("/rustfs/admin/v3/remove-user?accessKey={other}"), None),
-        (
-            http::Method::PUT,
-            "/rustfs/admin/v3/add-canned-policy?name=peri2evilpolicy".to_string(),
-            Some(bucket_rw_policy("peri2-any")),
-        ),
-        (http::Method::GET, "/rustfs/admin/v3/list-canned-policies".to_string(), None),
-        (http::Method::GET, "/rustfs/admin/v3/info-canned-policy?name=readwrite".to_string(), None),
-        (
-            http::Method::DELETE,
-            "/rustfs/admin/v3/remove-canned-policy?name=readwrite".to_string(),
-            None,
-        ),
-        (
-            http::Method::POST,
-            "/rustfs/admin/v3/idp/builtin/policy/attach".to_string(),
-            Some(serde_json::json!({ "policies": ["readwrite"], "user": other }).to_string()),
-        ),
-    ];
-
-    for (method, path, body) in probes {
-        let (status, text) = admin_request(&env.url, method.clone(), &path, body, limited, limited_secret).await?;
-        assert_eq!(
-            status,
-            StatusCode::FORBIDDEN,
-            "non-admin credential must get 403 on {method} {path}, got {status}: {text}"
-        );
-        assert!(text.contains("AccessDenied"), "denial on {method} {path} must carry AccessDenied: {text}");
-    }
-
-    env.stop_server();
-    Ok(())
-}

crates/e2e_test/src/admin_pools_test.rs

@@ -1,79 +0,0 @@
-// Copyright 2026 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
-use http::header::HOST;
-use reqwest::StatusCode;
-use rustfs_signer::constants::UNSIGNED_PAYLOAD;
-use rustfs_signer::sign_v4;
-use s3s::Body;
-use serde::Deserialize;
-use std::error::Error;
-
-#[derive(Debug, Deserialize)]
-struct PoolListItem {
-    id: usize,
-    cmdline: String,
-    status: String,
-}
-
-async fn signed_admin_get(env: &RustFSTestEnvironment, path: &str) -> Result<reqwest::Response, Box<dyn Error + Send + Sync>> {
-    let url = format!("{}{path}", env.url);
-    let uri = url.parse::<http::Uri>()?;
-    let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
-    let request = http::Request::builder()
-        .method(http::Method::GET)
-        .uri(uri)
-        .header(HOST, authority)
-        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD)
-        .body(Body::empty())?;
-    let signed = sign_v4(request, 0, &env.access_key, &env.secret_key, "", "us-east-1");
-
-    let mut request = local_http_client().get(&url);
-    for (name, value) in signed.headers() {
-        request = request.header(name, value);
-    }
-    Ok(request.send().await?)
-}
-
-#[tokio::test]
-async fn single_drive_pools_list_succeeds_without_enabling_decommission_status() -> Result<(), Box<dyn Error + Send + Sync>> {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let response = signed_admin_get(&env, "/rustfs/admin/v3/pools/list").await?;
-    let status = response.status();
-    let body = response.bytes().await?;
-
-    assert_eq!(status, StatusCode::OK, "pools list failed: {}", String::from_utf8_lossy(&body));
-    let pools: Vec<PoolListItem> = serde_json::from_slice(&body)?;
-    assert_eq!(pools.len(), 1);
-    assert_eq!(pools[0].id, 0);
-    assert_eq!(pools[0].cmdline, env.temp_dir);
-    assert_eq!(pools[0].status, "active");
-
-    let response = signed_admin_get(&env, "/rustfs/admin/v3/decommission/status").await?;
-    let status = response.status();
-    let body = response.text().await?;
-    assert_eq!(
-        status,
-        StatusCode::NOT_IMPLEMENTED,
-        "decommission status changed for a single pool: {body}"
-    );
-    assert!(body.contains("NotImplemented"), "unexpected decommission error body: {body}");
-
-    Ok(())
-}

crates/e2e_test/src/anonymous_access_test.rs

@@ -170,81 +170,3 @@ async fn test_anonymous_access_allowed_when_restrict_public_buckets_disabled()
     info!("Test passed: anonymous access allowed with RestrictPublicBuckets=false");
     Ok(())
 }
-
-/// A policy granting anonymous `s3:ListBucket` also permits ListObjectVersions.
-/// That grant must still be subject to RestrictPublicBuckets: the versions listing
-/// reaches authorization through a fallback branch, and that branch has to apply the
-/// same public-access gate as a direct grant.
-#[tokio::test]
-#[serial]
-async fn ghsa_x298_anonymous_list_object_versions_denied_when_restrict_public_buckets_enabled()
--> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-    info!("Starting test: anonymous ListObjectVersions denied with RestrictPublicBuckets=true...");
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let bucket_name = "anon-test-restrict-versions";
-    let admin_client = env.create_s3_client();
-    admin_client.create_bucket().bucket(bucket_name).send().await?;
-
-    let policy_json = serde_json::json!({
-        "Version": "2012-10-17",
-        "Statement": [
-            {
-                "Sid": "AllowAnonymousListBucket",
-                "Effect": "Allow",
-                "Principal": "*",
-                "Action": ["s3:ListBucket"],
-                "Resource": [format!("arn:aws:s3:::{}", bucket_name)]
-            }
-        ]
-    })
-    .to_string();
-
-    admin_client
-        .put_bucket_policy()
-        .bucket(bucket_name)
-        .policy(&policy_json)
-        .send()
-        .await?;
-
-    admin_client
-        .put_object()
-        .bucket(bucket_name)
-        .key("test.txt")
-        .body(aws_sdk_s3::primitives::ByteStream::from_static(b"hello anonymous"))
-        .send()
-        .await?;
-
-    // Without the public-access block the fallback grant is expected to work.
-    let versions_url = format!("{}/{}?versions=", env.url, bucket_name);
-    let resp = local_http_client().get(&versions_url).send().await?;
-    assert_eq!(
-        resp.status().as_u16(),
-        200,
-        "Anonymous ListObjectVersions should succeed via the s3:ListBucket grant"
-    );
-
-    admin_client
-        .put_public_access_block()
-        .bucket(bucket_name)
-        .public_access_block_configuration(
-            PublicAccessBlockConfiguration::builder()
-                .restrict_public_buckets(true)
-                .build(),
-        )
-        .send()
-        .await?;
-
-    let resp = local_http_client().get(&versions_url).send().await?;
-    assert_eq!(
-        resp.status().as_u16(),
-        403,
-        "Anonymous ListObjectVersions must be denied when RestrictPublicBuckets is true"
-    );
-
-    info!("Test passed: anonymous ListObjectVersions denied with RestrictPublicBuckets=true");
-    Ok(())
-}

crates/e2e_test/src/api_rate_limit_test.rs

@@ -1,155 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! E2E coverage for the opt-in per-client S3 API rate limit (backlog#1191):
-//! the layer must be wired into the real server stack, reject over-limit
-//! clients with `429` + `Retry-After`, keep health probes exempt, and stay
-//! completely inert with default configuration.
-
-use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
-use serial_test::serial;
-use tracing::info;
-
-type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
-
-#[tokio::test]
-#[serial]
-async fn api_rate_limit_enforces_429_with_retry_after_when_enabled() -> TestResult {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    // Burst 50 leaves headroom for the readiness-poll ListBuckets calls that
-    // share the loopback client IP; refill (60 rpm = 1/s) is slow enough that
-    // a rapid burst below reliably exhausts the bucket.
-    env.start_rustfs_server_with_env(
-        vec![],
-        &[
-            ("RUSTFS_API_RATE_LIMIT_ENABLE", "true"),
-            ("RUSTFS_API_RATE_LIMIT_RPM", "60"),
-            ("RUSTFS_API_RATE_LIMIT_BURST", "50"),
-        ],
-    )
-    .await?;
-
-    let client = local_http_client();
-    let list_buckets_url = format!("{}/", env.url);
-
-    let mut throttled = None;
-    let mut allowed = 0usize;
-    for _ in 0..80 {
-        let response = client.get(&list_buckets_url).send().await?;
-        if response.status() == reqwest::StatusCode::TOO_MANY_REQUESTS {
-            throttled = Some(response);
-            break;
-        }
-        allowed += 1;
-    }
-
-    let throttled = throttled.unwrap_or_else(|| panic!("no 429 within 80 rapid requests ({allowed} allowed) at burst 50"));
-    assert!(allowed > 0, "healthy traffic below the burst must not be throttled");
-    info!("rate limit engaged after {allowed} allowed requests");
-
-    let retry_after = throttled
-        .headers()
-        .get(reqwest::header::RETRY_AFTER)
-        .and_then(|v| v.to_str().ok())
-        .and_then(|v| v.parse::<u64>().ok())
-        .expect("429 must carry a numeric Retry-After header");
-    assert!(retry_after >= 1, "Retry-After must be at least one second, got {retry_after}");
-    assert_eq!(
-        throttled.headers().get("x-ratelimit-limit").and_then(|v| v.to_str().ok()),
-        Some("60"),
-        "429 must expose the configured limit"
-    );
-
-    let body = throttled.text().await?;
-    assert!(body.contains("<Code>TooManyRequests</Code>"), "S3-style error body expected: {body}");
-
-    // Health probes stay exempt even while the client budget is exhausted.
-    let health = client.get(format!("{}/health", env.url)).send().await?;
-    assert_ne!(
-        health.status(),
-        reqwest::StatusCode::TOO_MANY_REQUESTS,
-        "health probes must never be rate limited"
-    );
-
-    Ok(())
-}
-
-#[tokio::test]
-#[serial]
-async fn api_rate_limit_bucket_dimension_throttles_per_bucket() -> TestResult {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    // Bucket dimension only: the readiness-poll ListBuckets calls hit "/"
-    // (no bucket) and therefore do not consume any budget.
-    env.start_rustfs_server_with_env(
-        vec![],
-        &[
-            ("RUSTFS_API_RATE_LIMIT_ENABLE", "true"),
-            ("RUSTFS_API_RATE_LIMIT_BUCKET_RPM", "60"),
-            ("RUSTFS_API_RATE_LIMIT_BUCKET_BURST", "5"),
-        ],
-    )
-    .await?;
-
-    let client = local_http_client();
-
-    // Unauthenticated GETs are still counted arrivals (403, not 429, while
-    // within budget); the sixth rapid hit on the same bucket must throttle.
-    let mut throttled = false;
-    for i in 0..6 {
-        let response = client.get(format!("{}/hot-bucket/object-{i}", env.url)).send().await?;
-        if response.status() == reqwest::StatusCode::TOO_MANY_REQUESTS {
-            throttled = true;
-            assert!(
-                response.headers().contains_key(reqwest::header::RETRY_AFTER),
-                "bucket-dimension 429 must carry Retry-After"
-            );
-            break;
-        }
-    }
-    assert!(throttled, "6 rapid requests against burst 5 must trip the bucket dimension");
-
-    // A different bucket has its own budget.
-    let other = client.get(format!("{}/cold-bucket/object", env.url)).send().await?;
-    assert_ne!(
-        other.status(),
-        reqwest::StatusCode::TOO_MANY_REQUESTS,
-        "an unrelated bucket must not be throttled"
-    );
-
-    Ok(())
-}
-
-#[tokio::test]
-#[serial]
-async fn api_rate_limit_stays_inert_by_default() -> TestResult {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let client = local_http_client();
-    let list_buckets_url = format!("{}/", env.url);
-
-    for i in 0..80 {
-        let response = client.get(&list_buckets_url).send().await?;
-        assert_ne!(
-            response.status(),
-            reqwest::StatusCode::TOO_MANY_REQUESTS,
-            "request {i} was throttled although rate limiting is disabled by default"
-        );
-    }
-
-    Ok(())
-}

crates/e2e_test/src/checksum_upload_test.rs

@@ -19,10 +19,8 @@
 mod tests {
     use crate::common::{RustFSTestEnvironment, init_logging};
     use aws_sdk_s3::Client;
-    use aws_sdk_s3::config::{Credentials, Region, RequestChecksumCalculation};
     use aws_sdk_s3::primitives::ByteStream;
     use aws_sdk_s3::types::{ChecksumAlgorithm, ChecksumMode, CompletedMultipartUpload, CompletedPart};
-    use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
     use base64::Engine;
     use rustfs_rio::{Checksum, ChecksumType as RioChecksumType};
     use serial_test::serial;
@@ -33,25 +31,6 @@ mod tests {
         env.create_s3_client()
     }
 
-    /// Client with boto/SDK automatic checksum calculation disabled, so the ONLY
-    /// checksum on the wire is the one the test injects. Needed because the SDK's
-    /// default (crc32) would otherwise collide with the additional-algorithm header
-    /// we inject via `mutate_request` for XXHash/SHA-512/MD5 (which have no typed
-    /// SDK builder). Mirrors the boto3 `request_checksum_calculation=when_required`.
-    fn create_s3_client_no_auto_checksum(env: &RustFSTestEnvironment) -> Client {
-        let creds = Credentials::new(&env.access_key, &env.secret_key, None, None, "e2e-additional-checksum");
-        let config = aws_sdk_s3::Config::builder()
-            .credentials_provider(creds)
-            .region(Region::new("us-east-1"))
-            .endpoint_url(format!("http://{}", env.address))
-            .force_path_style(true)
-            .behavior_version_latest()
-            .request_checksum_calculation(RequestChecksumCalculation::WhenRequired)
-            .http_client(SmithyHttpClientBuilder::new().build_http())
-            .build();
-        Client::from_conf(config)
-    }
-
     async fn create_bucket(client: &Client, bucket: &str) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         match client.create_bucket().bucket(bucket).send().await {
             Ok(_) => {
@@ -480,87 +459,4 @@ mod tests {
             "Multipart object should report the same full-object CRC64NVME as direct upload"
         );
     }
-
-    /// Integration test for the AWS 2026-04 additional checksum algorithms
-    /// (XXHash3/64/128, SHA-512, MD5). aws_sdk_s3 has no typed builder for these, so the
-    /// `x-amz-checksum-<algo>` header is injected via `mutate_request` (value computed by
-    /// rustfs-rio, which is byte-for-byte identical to awscrt). Verifies the server
-    /// verifies-on-write: a correct value is accepted and the object stored; a wrong
-    /// value is rejected with BadDigest and nothing is stored. Full HEAD/GET header
-    /// echo round-trip is additionally exercised by the boto3+awscrt e2e.
-    #[tokio::test]
-    #[serial]
-    async fn test_additional_checksums_verify_on_write() {
-        init_logging();
-        info!("TEST: additional checksums (XXHash3/64/128, SHA-512, MD5) verify-on-write");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
-        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
-
-        let client = create_s3_client_no_auto_checksum(&env);
-        let bucket = "test-additional-checksums";
-        create_bucket(&client, bucket).await.expect("Failed to create bucket");
-
-        let content: &[u8] = b"additional-checksum verify-on-write payload for xxhash/sha512/md5";
-
-        for (ty, header) in [
-            (RioChecksumType::XXHASH3, "x-amz-checksum-xxhash3"),
-            (RioChecksumType::XXHASH64, "x-amz-checksum-xxhash64"),
-            (RioChecksumType::XXHASH128, "x-amz-checksum-xxhash128"),
-            (RioChecksumType::SHA512, "x-amz-checksum-sha512"),
-            (RioChecksumType::MD5, "x-amz-checksum-md5"),
-        ] {
-            // Correct checksum -> accepted, and the object is stored intact.
-            let good = Checksum::new_from_data(ty, content).expect("compute checksum").encoded;
-            let ok_key = format!("ok-{header}");
-            let put = client
-                .put_object()
-                .bucket(bucket)
-                .key(&ok_key)
-                .body(ByteStream::from_static(content))
-                .customize()
-                .mutate_request(move |req| {
-                    req.headers_mut().insert(header, good.clone());
-                })
-                .send()
-                .await;
-            assert!(put.is_ok(), "{header}: correct checksum must be accepted: {:?}", put.err());
-            let got = client
-                .get_object()
-                .bucket(bucket)
-                .key(&ok_key)
-                .send()
-                .await
-                .expect("GetObject");
-            let body = got.body.collect().await.expect("collect body").into_bytes();
-            assert_eq!(body.as_ref(), content, "{header}: stored body must match uploaded content");
-
-            // Wrong checksum -> rejected with BadDigest, and nothing is stored.
-            let bad = Checksum::new_from_data(ty, b"a totally different payload")
-                .expect("compute checksum")
-                .encoded;
-            let bad_key = format!("bad-{header}");
-            let put_bad = client
-                .put_object()
-                .bucket(bucket)
-                .key(&bad_key)
-                .body(ByteStream::from_static(content))
-                .customize()
-                .mutate_request(move |req| {
-                    req.headers_mut().insert(header, bad.clone());
-                })
-                .send()
-                .await;
-            assert!(put_bad.is_err(), "{header}: a mismatched checksum must be rejected");
-            let msg = format!("{:?}", put_bad.err().unwrap());
-            assert!(
-                msg.contains("BadDigest") || msg.to_lowercase().contains("digest") || msg.to_lowercase().contains("checksum"),
-                "{header}: expected a BadDigest/checksum error, got: {msg}"
-            );
-            let head = client.head_object().bucket(bucket).key(&bad_key).send().await;
-            assert!(head.is_err(), "{header}: nothing must be stored after a rejected PutObject");
-
-            info!("PASSED additional-checksum verify-on-write: {header}");
-        }
-    }
 }

crates/e2e_test/src/cluster_multidrive_pool_test.rs

@@ -1,108 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Smoke tests for the multi-drive / multi-pool cluster harness.
-//!
-//! These belong to the nightly 4-node cluster lane (they spin up real RustFS
-//! processes, so they are excluded from the merge-gate `e2e-full` profile in
-//! `.config/nextest.toml`). They assert that:
-//!
-//! * `drivesPerNode > 1` produces a bootable single-pool cluster whose
-//!   `RUSTFS_VOLUMES` string enumerates every drive, and that a PUT/GET
-//!   round-trips through the multi-drive erasure layout.
-//! * A two-pool topology (one node per pool, `drivesPerNode` drives each) boots
-//!   with the ellipses `RUSTFS_VOLUMES` form and round-trips a PUT/GET.
-//!
-//! Readiness is established by the harness's `start()` handshake (TCP reachability
-//! plus an S3 `ListBuckets` poll) — there are no fixed sleeps.
-//!
-//! Out of scope for this block (tracked separately): network fault injection
-//! (toxiproxy / socket proxy) and 5GiB large-object budgets.
-
-use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
-use serial_test::serial;
-
-type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
-
-const BUCKET: &str = "cluster-multidrive-pool-bucket";
-
-/// PUT an object then GET it back and assert the bytes round-trip.
-async fn put_get_roundtrip(cluster: &RustFSTestClusterEnvironment, key: &str, payload: &[u8]) -> TestResult {
-    let writer = cluster.create_s3_client(0)?;
-    writer
-        .put_object()
-        .bucket(BUCKET)
-        .key(key)
-        .body(bytes::Bytes::copy_from_slice(payload).into())
-        .send()
-        .await?;
-
-    // Read back from the last node to exercise the cross-node/cross-pool path.
-    let reader = cluster.create_s3_client(cluster.nodes.len() - 1)?;
-    let got = reader.get_object().bucket(BUCKET).key(key).send().await?;
-    let body = got.body.collect().await?.into_bytes();
-    assert_eq!(body.as_ref(), payload, "round-tripped object bytes must match");
-    Ok(())
-}
-
-/// 4 nodes x 2 drives, single pool: the multi-drive layout boots and round-trips.
-#[tokio::test]
-#[serial]
-async fn cluster_multidrive_single_pool_smoke() -> TestResult {
-    crate::common::init_logging();
-
-    let mut cluster = RustFSTestClusterEnvironment::with_topology(ClusterTopology::single_pool_multidrive(4, 2)).await?;
-
-    // The single-pool multi-drive layout must list every (node, drive) endpoint
-    // explicitly (8 endpoints, no ellipses) so the server keeps one legacy pool.
-    let volumes = cluster.rustfs_volumes_arg();
-    assert_eq!(volumes.split(' ').count(), 8, "expected 8 explicit endpoints, got: {volumes}");
-    assert!(!volumes.contains('{'), "single-pool layout must not use ellipses: {volumes}");
-
-    cluster.start().await?;
-    cluster.create_test_bucket(BUCKET).await?;
-
-    let payload = vec![0xA5u8; 512 * 1024];
-    put_get_roundtrip(&cluster, "multidrive/object", &payload).await?;
-    Ok(())
-}
-
-/// Two single-node pools, 2 drives each: the multi-pool layout boots and
-/// round-trips. Every pool is a distinct erasure pool (`pool_idx` 0 and 1).
-#[tokio::test]
-#[serial]
-async fn cluster_two_pool_smoke() -> TestResult {
-    crate::common::init_logging();
-
-    let mut cluster =
-        RustFSTestClusterEnvironment::with_topology(ClusterTopology::per_node_pools(2, vec![vec![0], vec![1]])).await?;
-
-    // The two-pool layout must emit one ellipses argument per pool.
-    let volumes = cluster.rustfs_volumes_arg();
-    let args: Vec<&str> = volumes.split(' ').collect();
-    assert_eq!(args.len(), 2, "expected two pool arguments, got: {volumes}");
-    assert!(
-        args.iter().all(|a| a.contains("/drive{0...1}")),
-        "each pool arg must use the drive ellipses form: {volumes}"
-    );
-    assert_eq!(cluster.nodes[0].pool_idx, 0);
-    assert_eq!(cluster.nodes[1].pool_idx, 1);
-
-    cluster.start().await?;
-    cluster.create_test_bucket(BUCKET).await?;
-
-    let payload = vec![0x5Au8; 256 * 1024];
-    put_get_roundtrip(&cluster, "twopool/object", &payload).await?;
-    Ok(())
-}

crates/e2e_test/src/common.rs

@@ -446,13 +446,6 @@ impl RustFSTestEnvironment {
         self.start_rustfs_server_inner(extra_args, &[], false).await
     }
 
-    pub async fn start_rustfs_server_without_cleanup_with_env(
-        &mut self,
-        extra_env: &[(&str, &str)],
-    ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        self.start_rustfs_server_inner(vec![], extra_env, false).await
-    }
-
     /// Wait for RustFS server to be ready.
     ///
     /// A listening TCP port is not sufficient here: the process may accept
@@ -521,30 +514,6 @@ impl RustFSTestEnvironment {
             }
         }
     }
-
-    /// Restart this server in place, preserving its on-disk data directory,
-    /// listen address, and credentials.
-    ///
-    /// Failure-recovery tests (backlog#1147 repl-5) need the *same* instance to
-    /// come back up after a crash/stop with its persisted state intact:
-    /// per-object replication status in `xl.meta`, the MRF recovery file, and
-    /// resync metadata all live under `temp_dir`, which this method keeps. The
-    /// address is reused too (the server sets `SO_REUSEADDR`/`SO_REUSEPORT`), so
-    /// existing S3 clients keep working without reconfiguration.
-    ///
-    /// The previous process is stopped first. `cleanup_existing` is false so a
-    /// sibling server sharing the harness (e.g. a replication target on another
-    /// port) is left untouched — only this instance is recycled. Pass the same
-    /// `extra_args` / `extra_env` the instance was originally started with;
-    /// background tasks read their env once at startup.
-    pub async fn restart_server_preserving_data(
-        &mut self,
-        extra_args: Vec<&str>,
-        extra_env: &[(&str, &str)],
-    ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        self.stop_server();
-        self.start_rustfs_server_inner(extra_args, extra_env, false).await
-    }
 }
 
 impl Drop for RustFSTestEnvironment {
@@ -717,166 +686,15 @@ pub async fn awscurl_delete(
     execute_awscurl(url, "DELETE", None, access_key, secret_key).await
 }
 
-/// Cluster topology declaration for a `RustFSTestClusterEnvironment`.
-///
-/// Describes how many nodes to launch, how many data drives each node exposes,
-/// and how those nodes are grouped into erasure pools. The topology drives both
-/// on-disk directory creation and the `RUSTFS_VOLUMES` string assembled for the
-/// node processes.
-///
-/// # Single-host expressibility constraints
-///
-/// The harness runs every node on `127.0.0.1` with a distinct port, so the
-/// `RUSTFS_VOLUMES` string can only use the two forms the server parser accepts
-/// on a single machine (verified against `ecstore` `DisksLayout::from_volumes`):
-///
-/// * **Single pool** — every `(node, drive)` endpoint listed explicitly, no
-///   ellipses. This is the legacy form and yields exactly one pool spanning all
-///   nodes and drives (`DistErasure`). Any `drives_per_node >= 1` works.
-/// * **Multiple pools** — one ellipses arg per pool. A pool argument is a single
-///   URL template, so it can enumerate several drives on *one* host but cannot
-///   enumerate multiple distinct-port hosts. A pool that spanned two localhost
-///   nodes would need a host ellipses (`127.0.0.1:900{0...1}`), which forces the
-///   *same* on-disk path across those ports and collides physically. Therefore
-///   every pool in a multi-pool topology owns exactly one node. In addition, the
-///   parser rejects a single-drive ellipses pool (`drive{0...0}`), so a
-///   multi-pool topology requires `drives_per_node >= 2`.
-///
-/// Genuine multi-node pools (a pool striped across several hosts) require real
-/// multi-host infrastructure and are deferred to the nightly cluster CI lane
-/// (backlog #1313 / #1314); they are intentionally not expressible here.
-#[derive(Clone, Debug)]
-pub struct ClusterTopology {
-    /// Number of node processes to launch.
-    pub node_count: usize,
-    /// Number of independent data drives (directories) each node exposes.
-    pub drives_per_node: usize,
-    /// Pool membership as a list of node-index groups. An empty vector means a
-    /// single pool spanning every node.
-    pub pools: Vec<Vec<usize>>,
-}
-
-impl ClusterTopology {
-    /// Single pool, one drive per node — identical to the historical
-    /// `RustFSTestClusterEnvironment::new` layout.
-    pub fn single_pool(node_count: usize) -> Self {
-        Self {
-            node_count,
-            drives_per_node: 1,
-            pools: Vec::new(),
-        }
-    }
-
-    /// Single pool with `drives_per_node` drives on every node. The pool spans
-    /// all nodes and all drives (one `DistErasure` pool).
-    pub fn single_pool_multidrive(node_count: usize, drives_per_node: usize) -> Self {
-        Self {
-            node_count,
-            drives_per_node,
-            pools: Vec::new(),
-        }
-    }
-
-    /// Multi-pool topology where every pool owns exactly one node. `pools` lists
-    /// the node index backing each pool (e.g. `vec![vec![0], vec![1]]` for a
-    /// two-pool cluster). Requires `drives_per_node >= 2` (see the type-level
-    /// note on single-host expressibility).
-    pub fn per_node_pools(drives_per_node: usize, pools: Vec<Vec<usize>>) -> Self {
-        let node_count = pools.iter().flatten().copied().max().map(|m| m + 1).unwrap_or(0);
-        Self {
-            node_count,
-            drives_per_node,
-            pools,
-        }
-    }
-
-    /// Normalized pool membership: an empty `pools` becomes a single pool over
-    /// all node indices.
-    fn normalized_pools(&self) -> Vec<Vec<usize>> {
-        if self.pools.is_empty() {
-            vec![(0..self.node_count).collect()]
-        } else {
-            self.pools.clone()
-        }
-    }
-
-    /// Validate that the topology is expressible on a single localhost machine.
-    fn validate(&self) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        if self.node_count == 0 {
-            return Err("Node count must be greater than zero".into());
-        }
-        if self.drives_per_node == 0 {
-            return Err("drives_per_node must be greater than zero".into());
-        }
-
-        let pools = self.normalized_pools();
-
-        // Every referenced node index must be in range.
-        for (pool_idx, nodes) in pools.iter().enumerate() {
-            if nodes.is_empty() {
-                return Err(format!("pool {pool_idx} has no nodes").into());
-            }
-            for &n in nodes {
-                if n >= self.node_count {
-                    return Err(format!("pool {pool_idx} references node {n} but node_count is {}", self.node_count).into());
-                }
-            }
-        }
-
-        // Each node must belong to exactly one pool.
-        let mut seen = vec![0usize; self.node_count];
-        for nodes in &pools {
-            for &n in nodes {
-                seen[n] += 1;
-            }
-        }
-        for (n, count) in seen.iter().enumerate() {
-            match count {
-                0 => return Err(format!("node {n} is not assigned to any pool").into()),
-                1 => {}
-                _ => return Err(format!("node {n} is assigned to more than one pool").into()),
-            }
-        }
-
-        // Multi-pool constraints imposed by the single-host RUSTFS_VOLUMES syntax.
-        if pools.len() > 1 {
-            if self.drives_per_node < 2 {
-                return Err(
-                    "multi-pool topology requires drives_per_node >= 2 (the server parser rejects a single-drive ellipses pool)"
-                        .into(),
-                );
-            }
-            for (pool_idx, nodes) in pools.iter().enumerate() {
-                if nodes.len() != 1 {
-                    return Err(format!(
-                        "pool {pool_idx} spans {} nodes; a pool striped across multiple localhost nodes is not expressible (needs host ellipses, which collides on shared paths). Use one node per pool, or the nightly multi-host lane (backlog #1313/#1314).",
-                        nodes.len()
-                    )
-                    .into());
-                }
-            }
-        }
-
-        Ok(())
-    }
-}
-
 /// Represents a single RustFS server instance in a test cluster.
 ///
 /// Each `ClusterNode` tracks the node's network address, base URL for
-/// S3-compatible requests, on-disk data directories, and the underlying
+/// S3-compatible requests, on-disk data directory, and the underlying
 /// child process handle when the node is running.
 pub struct ClusterNode {
     pub address: String,
     pub url: String,
-    /// Primary data directory for the node. For single-drive nodes this is the
-    /// node's only drive; for multi-drive nodes it is the first drive. Kept as a
-    /// stable field so existing single-drive tests continue to compile.
     pub data_dir: String,
-    /// All data drives exposed by this node (`data_dirs[0] == data_dir`).
-    pub data_dirs: Vec<String>,
-    /// Index of the pool this node belongs to.
-    pub pool_idx: usize,
     pub process: Option<Child>,
 }
 
@@ -892,8 +710,6 @@ pub struct RustFSTestClusterEnvironment {
     pub access_key: String,
     pub secret_key: String,
     pub extra_env: Vec<(String, String)>,
-    pub node_extra_env: Vec<Vec<(String, String)>>,
-    pub topology: ClusterTopology,
 }
 
 impl RustFSTestClusterEnvironment {
@@ -915,84 +731,34 @@ impl RustFSTestClusterEnvironment {
     /// * `Err(Box<dyn Error + Send + Sync>)` - An error if any step fails, such as temporary
     ///   directory creation failure or available port lookup failure.
     pub async fn new(node_count: usize) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
-        Self::with_topology(ClusterTopology::single_pool(node_count)).await
-    }
-
-    /// Create a RustFS test cluster environment from an explicit topology.
-    ///
-    /// Allocates a unique temporary root directory, an available TCP port per
-    /// node, and one data directory per drive. The topology is validated up front
-    /// (node/pool assignment, single-host multi-pool constraints); node processes
-    /// are not started at this stage.
-    ///
-    /// When the topology exposes more than one local drive on a node (multi-drive
-    /// or multi-pool layouts), `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true` is added to
-    /// every node's environment: those drives live on the same temp filesystem, so
-    /// the server's distinct-physical-disk safety check would otherwise reject
-    /// startup. Single-drive single-pool clusters keep the historical environment
-    /// unchanged.
-    pub async fn with_topology(topology: ClusterTopology) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
-        topology.validate()?;
-
+        if node_count == 0 {
+            return Err("Node count must be greater than zero".into());
+        }
         let temp_dir = format!("/tmp/rustfs_cluster_test_{}", Uuid::new_v4());
         fs::create_dir_all(&temp_dir).await?;
 
-        // Map every node to its owning pool index.
-        let pools = topology.normalized_pools();
-        let mut pool_of_node = vec![0usize; topology.node_count];
-        for (pool_idx, nodes) in pools.iter().enumerate() {
-            for &n in nodes {
-                pool_of_node[n] = pool_idx;
-            }
-        }
-
-        let multidrive = topology.drives_per_node > 1;
-
-        let mut nodes = Vec::with_capacity(topology.node_count);
-        for (i, &pool_idx) in pool_of_node.iter().enumerate() {
+        let mut nodes = Vec::with_capacity(node_count);
+        for i in 0..node_count {
             let port = RustFSTestEnvironment::find_available_port().await?;
             let address = format!("127.0.0.1:{}", port);
             let url = format!("http://{}", address);
-
-            // Single-drive nodes keep the historical `{temp}/node{i}` path so
-            // existing tests (and their on-disk assertions) are unaffected.
-            // Multi-drive nodes nest one `drive{d}` directory per drive so the
-            // ellipses form `drive{0...N-1}` can address them.
-            let data_dirs: Vec<String> = if multidrive {
-                (0..topology.drives_per_node)
-                    .map(|d| format!("{}/node{}/drive{}", temp_dir, i, d))
-                    .collect()
-            } else {
-                vec![format!("{}/node{}", temp_dir, i)]
-            };
-            for dir in &data_dirs {
-                fs::create_dir_all(dir).await?;
-            }
+            let data_dir = format!("{}/node{}", temp_dir, i);
+            fs::create_dir_all(&data_dir).await?;
 
             nodes.push(ClusterNode {
                 address,
                 url,
-                data_dir: data_dirs[0].clone(),
-                data_dirs,
-                pool_idx,
+                data_dir,
                 process: None,
             });
         }
 
-        let mut extra_env = Vec::new();
-        extra_env.push(("RUSTFS_RPC_SECRET".to_string(), String::new()));
-        if multidrive {
-            extra_env.push(("RUSTFS_UNSAFE_BYPASS_DISK_CHECK".to_string(), "true".to_string()));
-        }
-
         Ok(Self {
             nodes,
             temp_dir,
-            access_key: "rustfs-cluster-test-access".to_string(),
-            secret_key: "rustfs-cluster-test-secret".to_string(),
-            extra_env,
-            node_extra_env: vec![Vec::new(); topology.node_count],
-            topology,
+            access_key: DEFAULT_ACCESS_KEY.to_string(),
+            secret_key: DEFAULT_SECRET_KEY.to_string(),
+            extra_env: Vec::new(),
         })
     }
 
@@ -1005,22 +771,6 @@ impl RustFSTestClusterEnvironment {
         self.extra_env.push((key.into(), value.into()));
     }
 
-    /// Add an extra environment variable applied to a single cluster node.
-    pub fn set_node_env<K, V>(
-        &mut self,
-        node_idx: usize,
-        key: K,
-        value: V,
-    ) -> Result<(), Box<dyn std::error::Error + Send + Sync>>
-    where
-        K: Into<String>,
-        V: Into<String>,
-    {
-        self.ensure_node_index(node_idx)?;
-        self.node_extra_env[node_idx].push((key.into(), value.into()));
-        Ok(())
-    }
-
     fn ensure_node_index(&self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         if node_idx >= self.nodes.len() {
             return Err(format!("node_idx {node_idx} is invalid").into());
@@ -1028,51 +778,14 @@ impl RustFSTestClusterEnvironment {
         Ok(())
     }
 
-    /// Build the `RUSTFS_VOLUMES` argument string for the cluster's topology.
+    /// Build the volumes argument string for RustFS binary (internal helper method).
     ///
-    /// * **Single pool** — every `(node, drive)` endpoint is listed explicitly and
-    ///   space-joined. With no ellipses the server parser collapses them into one
-    ///   legacy pool spanning all nodes and drives.
-    /// * **Multiple pools** — each pool (exactly one node) contributes one ellipses
-    ///   argument `http://<addr><node-base>/drive{0...N-1}`; the space-separated
-    ///   arguments become one pool each.
-    ///
-    /// The server splits `RUSTFS_VOLUMES` on spaces (`value_delimiter = ' '`), which
-    /// this assembly matches, and the resulting layout is verified against the
-    /// `ecstore` parser in the unit tests below.
-    /// Public view of the assembled `RUSTFS_VOLUMES` string, so tests can assert
-    /// the pool/drive layout without starting node processes.
-    pub fn rustfs_volumes_arg(&self) -> String {
-        self.build_volumes_arg()
-    }
-
+    /// Concatenates the address and data directory of all cluster nodes into a single string
+    /// used as the `RUSTFS_VOLUMES` environment variable for RustFS node processes.
     fn build_volumes_arg(&self) -> String {
-        let pools = self.topology.normalized_pools();
-
-        if pools.len() <= 1 {
-            // Single pool: explicit enumeration of every drive on every node.
-            return self
-                .nodes
-                .iter()
-                .flat_map(|n| n.data_dirs.iter().map(move |dir| format!("http://{}{}", n.address, dir)))
-                .collect::<Vec<_>>()
-                .join(" ");
-        }
-
-        // Multi-pool: one ellipses argument per single-node pool. The drive
-        // directories are `<temp>/node{i}/drive{0..N-1}`, so the ellipses base is
-        // the shared parent of the node's drives.
-        pools
+        self.nodes
             .iter()
-            .map(|nodes| {
-                let node = &self.nodes[nodes[0]];
-                let base = node
-                    .data_dirs
-                    .first()
-                    .and_then(|d| d.rsplit_once('/').map(|(parent, _)| parent))
-                    .unwrap_or(&node.data_dir);
-                format!("http://{}{}/drive{{0...{}}}", node.address, base, self.topology.drives_per_node - 1)
-            })
+            .map(|n| format!("http://{}{}", n.address, n.data_dir))
             .collect::<Vec<_>>()
             .join(" ")
     }
@@ -1107,9 +820,6 @@ impl RustFSTestClusterEnvironment {
             for (key, value) in &self.extra_env {
                 command.env(key, value);
             }
-            for (key, value) in &self.node_extra_env[i] {
-                command.env(key, value);
-            }
 
             let process = command.current_dir(&node.data_dir).spawn()?;
 
@@ -1151,9 +861,6 @@ impl RustFSTestClusterEnvironment {
         for (key, value) in &self.extra_env {
             command.env(key, value);
         }
-        for (key, value) in &self.node_extra_env[node_idx] {
-            command.env(key, value);
-        }
 
         let process = command.current_dir(&node.data_dir).spawn()?;
         node.process = Some(process);
@@ -1353,151 +1060,4 @@ mod tests {
 
         stdfs::remove_file(stamp_path).ok();
     }
-
-    /// Build a cluster environment struct in-memory (no ports, no processes) so
-    /// that `build_volumes_arg` can be exercised as a pure string builder. Node
-    /// directories mirror what `with_topology` would create for the topology.
-    fn fake_cluster(topology: ClusterTopology) -> RustFSTestClusterEnvironment {
-        let temp_dir = "/tmp/rustfs_cluster_test_FAKE".to_string();
-        let pools = topology.normalized_pools();
-        let mut pool_of_node = vec![0usize; topology.node_count];
-        for (pool_idx, nodes) in pools.iter().enumerate() {
-            for &n in nodes {
-                pool_of_node[n] = pool_idx;
-            }
-        }
-        let multidrive = topology.drives_per_node > 1;
-
-        let nodes = (0..topology.node_count)
-            .map(|i| {
-                let address = format!("127.0.0.1:{}", 9000 + i);
-                let data_dirs: Vec<String> = if multidrive {
-                    (0..topology.drives_per_node)
-                        .map(|d| format!("{}/node{}/drive{}", temp_dir, i, d))
-                        .collect()
-                } else {
-                    vec![format!("{}/node{}", temp_dir, i)]
-                };
-                ClusterNode {
-                    url: format!("http://{}", address),
-                    address,
-                    data_dir: data_dirs[0].clone(),
-                    data_dirs,
-                    pool_idx: pool_of_node[i],
-                    process: None,
-                }
-            })
-            .collect();
-
-        RustFSTestClusterEnvironment {
-            nodes,
-            temp_dir,
-            access_key: DEFAULT_ACCESS_KEY.to_string(),
-            secret_key: DEFAULT_SECRET_KEY.to_string(),
-            extra_env: Vec::new(),
-            node_extra_env: vec![Vec::new(); topology.node_count],
-            topology,
-        }
-    }
-
-    #[test]
-    fn volumes_single_pool_single_drive_is_backward_compatible() {
-        // The historical layout: one explicit endpoint per node, space-joined,
-        // no ellipses, no `drive` sub-directory.
-        let env = fake_cluster(ClusterTopology::single_pool(4));
-        assert_eq!(
-            env.build_volumes_arg(),
-            "http://127.0.0.1:9000/tmp/rustfs_cluster_test_FAKE/node0 \
-             http://127.0.0.1:9001/tmp/rustfs_cluster_test_FAKE/node1 \
-             http://127.0.0.1:9002/tmp/rustfs_cluster_test_FAKE/node2 \
-             http://127.0.0.1:9003/tmp/rustfs_cluster_test_FAKE/node3"
-        );
-    }
-
-    #[test]
-    fn volumes_single_pool_multidrive_enumerates_every_drive() {
-        // 4 nodes x 2 drives -> 8 explicit endpoints, one legacy pool. No
-        // ellipses, so the server parser keeps this as a single DistErasure pool.
-        let env = fake_cluster(ClusterTopology::single_pool_multidrive(4, 2));
-        let expected = (0..4)
-            .flat_map(|i| {
-                (0..2).map(move |d| format!("http://127.0.0.1:{}/tmp/rustfs_cluster_test_FAKE/node{}/drive{}", 9000 + i, i, d))
-            })
-            .collect::<Vec<_>>()
-            .join(" ");
-        assert_eq!(env.build_volumes_arg(), expected);
-        assert_eq!(env.build_volumes_arg().split(' ').count(), 8);
-    }
-
-    #[test]
-    fn volumes_two_pool_uses_one_ellipses_arg_per_pool() {
-        // Two single-node pools, 2 drives each -> two ellipses arguments. The
-        // server parser treats each space-separated ellipses arg as its own pool.
-        let env = fake_cluster(ClusterTopology::per_node_pools(2, vec![vec![0], vec![1]]));
-        assert_eq!(
-            env.build_volumes_arg(),
-            "http://127.0.0.1:9000/tmp/rustfs_cluster_test_FAKE/node0/drive{0...1} \
-             http://127.0.0.1:9001/tmp/rustfs_cluster_test_FAKE/node1/drive{0...1}"
-        );
-    }
-
-    #[test]
-    fn topology_rejects_multi_node_pool() {
-        // A pool striped across two localhost nodes is not expressible.
-        let err = ClusterTopology::per_node_pools(2, vec![vec![0, 1], vec![2, 3]])
-            .validate()
-            .unwrap_err()
-            .to_string();
-        assert!(err.contains("not expressible"), "unexpected error: {err}");
-    }
-
-    #[test]
-    fn topology_rejects_single_drive_multi_pool() {
-        // The server parser rejects a single-drive ellipses pool (`drive{0...0}`).
-        let err = ClusterTopology::per_node_pools(1, vec![vec![0], vec![1]])
-            .validate()
-            .unwrap_err()
-            .to_string();
-        assert!(err.contains("drives_per_node >= 2"), "unexpected error: {err}");
-    }
-
-    #[test]
-    fn topology_rejects_unassigned_and_duplicated_nodes() {
-        // Node 2 is never assigned to a pool.
-        let mut t = ClusterTopology::single_pool_multidrive(3, 2);
-        t.pools = vec![vec![0], vec![1]];
-        assert!(t.validate().unwrap_err().to_string().contains("not assigned"));
-
-        // Node 0 assigned twice.
-        let mut t = ClusterTopology::single_pool_multidrive(2, 2);
-        t.pools = vec![vec![0], vec![0]];
-        assert!(t.validate().unwrap_err().to_string().contains("more than one pool"));
-    }
-
-    #[test]
-    fn topology_single_pool_accepts_any_drive_count() {
-        assert!(ClusterTopology::single_pool(4).validate().is_ok());
-        assert!(ClusterTopology::single_pool_multidrive(4, 4).validate().is_ok());
-        assert!(ClusterTopology::single_pool_multidrive(1, 1).validate().is_ok());
-    }
-
-    #[test]
-    fn cluster_node_env_supports_per_node_overrides() {
-        let mut env = fake_cluster(ClusterTopology::single_pool(4));
-        env.set_node_env(2, "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY", "true").unwrap();
-        assert_eq!(
-            env.node_extra_env[2].as_slice(),
-            [("RUSTFS_INTERNODE_RPC_MSGPACK_ONLY".to_string(), "true".to_string())]
-        );
-    }
-
-    #[test]
-    fn cluster_node_env_rejects_invalid_index() {
-        let mut env = fake_cluster(ClusterTopology::single_pool(4));
-        let err = env
-            .set_node_env(4, "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY", "true")
-            .unwrap_err()
-            .to_string();
-        assert!(err.contains("invalid"), "unexpected error: {err}");
-    }
 }

crates/e2e_test/src/connection_cap_test.rs

@@ -1,127 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! E2E coverage for the opt-in global connection cap on the main API listener
-//! (backlog#1191 follow-up, `RUSTFS_API_MAX_CONNECTIONS`): permits must be
-//! released when connections close (no leak), and the cap must actually bound
-//! concurrency — a queued connection is served only after a held one closes.
-
-use crate::common::{RustFSTestEnvironment, init_logging};
-use serial_test::serial;
-use std::time::Duration;
-use tokio::io::{AsyncReadExt, AsyncWriteExt};
-use tokio::net::TcpStream;
-use tokio::time::timeout;
-
-type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
-
-/// Open a TCP connection and write one unauthenticated `GET /` (any response,
-/// e.g. 403, proves the connection was accepted and served).
-async fn open_and_request(addr: &str, connection: &str) -> std::io::Result<TcpStream> {
-    let mut stream = TcpStream::connect(addr).await?;
-    let request = format!("GET / HTTP/1.1\r\nHost: {addr}\r\nConnection: {connection}\r\n\r\n");
-    stream.write_all(request.as_bytes()).await?;
-    Ok(stream)
-}
-
-/// Read until the response head is complete, or `None` on timeout/close —
-/// a `None` on an open socket means the connection sits unaccepted in the
-/// kernel backlog behind the cap.
-async fn read_response_head(stream: &mut TcpStream, dur: Duration) -> Option<String> {
-    let deadline = tokio::time::Instant::now() + dur;
-    let mut buf = vec![0u8; 4096];
-    let mut collected = String::new();
-    loop {
-        let remaining = deadline.checked_duration_since(tokio::time::Instant::now())?;
-        match timeout(remaining, stream.read(&mut buf)).await {
-            Ok(Ok(0)) | Ok(Err(_)) | Err(_) => return None,
-            Ok(Ok(n)) => {
-                collected.push_str(&String::from_utf8_lossy(&buf[..n]));
-                if collected.contains("\r\n\r\n") {
-                    return Some(collected);
-                }
-            }
-        }
-    }
-}
-
-#[tokio::test]
-#[serial]
-async fn connection_cap_releases_permits_on_close() -> TestResult {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_API_MAX_CONNECTIONS", "2")])
-        .await?;
-
-    // Ten sequential connections against cap 2: if permits leaked, the third
-    // request would already hang in the backlog and time out.
-    for i in 0..10 {
-        let mut stream = open_and_request(&env.address, "close").await?;
-        let head = read_response_head(&mut stream, Duration::from_secs(10))
-            .await
-            .unwrap_or_else(|| panic!("request {i} got no response — a connection permit leaked"));
-        assert!(head.starts_with("HTTP/1.1"), "request {i} unexpected response: {head}");
-    }
-
-    Ok(())
-}
-
-/// Open a TCP connection and send an INCOMPLETE request head. Once accepted
-/// it pins a connection permit: hyper waits for the rest of the head (75s
-/// default header timeout) until we close the socket.
-async fn open_and_stall(addr: &str) -> std::io::Result<TcpStream> {
-    let mut stream = TcpStream::connect(addr).await?;
-    stream
-        .write_all(format!("GET / HTTP/1.1\r\nHost: {addr}\r\n").as_bytes())
-        .await?;
-    Ok(stream)
-}
-
-#[tokio::test]
-#[serial]
-async fn connection_cap_blocks_excess_connections_until_permits_free() -> TestResult {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_API_MAX_CONNECTIONS", "2")])
-        .await?;
-    // Let the readiness poller's pooled connection close and free its permit.
-    tokio::time::sleep(Duration::from_secs(1)).await;
-
-    // Two stalled connections saturate cap 2 (a served-and-closed connection
-    // would release its permit immediately, so stalling is what makes the
-    // occupancy deterministic).
-    let stalled_a = open_and_stall(&env.address).await?;
-    let stalled_b = open_and_stall(&env.address).await?;
-    tokio::time::sleep(Duration::from_millis(300)).await;
-
-    // A complete request now sits in the kernel backlog: connect() succeeds
-    // but no permit is available, so no response arrives.
-    let mut blocked = open_and_request(&env.address, "close").await?;
-    assert!(
-        read_response_head(&mut blocked, Duration::from_secs(3)).await.is_none(),
-        "cap 2 with two stalled connections must leave the third unserved"
-    );
-
-    // Dropping the stalled connections releases their permits (hyper sees
-    // EOF while reading the head); the queued request's bytes already sit in
-    // the socket buffer, so it must now be accepted and served.
-    drop(stalled_a);
-    drop(stalled_b);
-    let head = read_response_head(&mut blocked, Duration::from_secs(10))
-        .await
-        .expect("queued connection must be served after permits are released");
-    assert!(head.starts_with("HTTP/1.1"), "unexpected response: {head}");
-
-    Ok(())
-}

crates/e2e_test/src/console_smoke_test.rs

@@ -1,191 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Over-the-wire smoke net for the embedded console listener (backlog#1154
-//! peri-4). The console is a second HTTP listener that serves the full S3/admin
-//! surface plus the unauthenticated console routes under `/rustfs/console`;
-//! before this test its behaviour was only covered by in-process router tests
-//! (`rustfs/src/admin/console.rs`), so a misconfiguration that exposed
-//! credentials through the public console endpoints or fail-opened the
-//! protected surface on the console port had no regression net.
-//!
-//! Pinned wire contract (real binary, real TCP):
-//!   * `/rustfs/console/version` and `/rustfs/console/license` answer 200 JSON
-//!     without authentication, with complete fields and no credential material.
-//!   * `/rustfs/console/license` exposes only the coarse `licensed` flag.
-//!   * the console SPA prefix dispatches to the static handler, never to the
-//!     S3 API (no S3 error XML), whether or not console assets are embedded.
-//!   * unauthenticated requests to the admin API and the S3 root on the console
-//!     listener are denied (403 AccessDenied) — the extra listener does not
-//!     fail-open the protected surface.
-//!   * a listener with the console disabled (the main S3 port here) does not
-//!     serve the unauthenticated console endpoints at all.
-
-use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
-use serial_test::serial;
-use std::error::Error;
-use tokio::time::{Duration, sleep};
-
-type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-/// Polls the console version endpoint until the console listener accepts
-/// requests. The harness readiness check only covers the S3 listener; the
-/// console listener of the same process may come up moments later.
-async fn wait_for_console_ready(console_base: &str) -> Result<reqwest::Response, Box<dyn Error + Send + Sync>> {
-    let client = local_http_client();
-    let url = format!("{console_base}/rustfs/console/version");
-    let mut last_err = String::new();
-    for _ in 0..40 {
-        match client.get(&url).send().await {
-            Ok(response) if response.status() == reqwest::StatusCode::OK => return Ok(response),
-            Ok(response) => last_err = format!("status {}", response.status()),
-            Err(err) => last_err = err.to_string(),
-        }
-        sleep(Duration::from_millis(250)).await;
-    }
-    Err(format!("console listener at {console_base} never became ready: {last_err}").into())
-}
-
-#[tokio::test]
-#[serial]
-async fn test_console_over_the_wire_smoke() -> TestResult {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    // A unique secret makes the credential-leak assertions below sharp: any
-    // occurrence of it in a console response body is a genuine leak, not a
-    // collision with a common default string.
-    env.access_key = format!("peri4ak{}", uuid::Uuid::new_v4().simple());
-    env.secret_key = format!("peri4sk{}", uuid::Uuid::new_v4().simple());
-
-    let console_port = RustFSTestEnvironment::find_available_port().await?;
-    let console_address = format!("127.0.0.1:{console_port}");
-    let console_base = format!("http://{console_address}");
-    env.start_rustfs_server_with_env(
-        vec![],
-        &[
-            ("RUSTFS_CONSOLE_ENABLE", "true"),
-            ("RUSTFS_CONSOLE_ADDRESS", console_address.as_str()),
-        ],
-    )
-    .await?;
-
-    let client = local_http_client();
-
-    // --- /rustfs/console/version: 200 JSON, complete fields, no secrets ------
-    let version_response = wait_for_console_ready(&console_base).await?;
-    let content_type = version_response
-        .headers()
-        .get(reqwest::header::CONTENT_TYPE)
-        .and_then(|v| v.to_str().ok())
-        .unwrap_or_default()
-        .to_string();
-    assert!(content_type.starts_with("application/json"), "version content-type: {content_type}");
-    let version_body = version_response.text().await?;
-    let version_json: serde_json::Value = serde_json::from_str(&version_body)?;
-    for field in ["version", "version_info", "date"] {
-        assert!(
-            version_json[field].as_str().is_some_and(|v| !v.is_empty()),
-            "console version field {field} missing or empty: {version_body}"
-        );
-    }
-    assert!(
-        !version_body.contains(&env.secret_key) && !version_body.contains(&env.access_key),
-        "console version response leaks credentials: {version_body}"
-    );
-
-    // --- /rustfs/console/license: coarse licensed flag only ------------------
-    let license_response = client.get(format!("{console_base}/rustfs/console/license")).send().await?;
-    assert_eq!(license_response.status(), reqwest::StatusCode::OK);
-    let license_body = license_response.text().await?;
-    let license_json: serde_json::Value = serde_json::from_str(&license_body)?;
-    assert!(
-        license_json["licensed"].is_boolean(),
-        "license payload must expose a licensed bool: {license_body}"
-    );
-    let license_keys: Vec<&String> = license_json.as_object().map(|o| o.keys().collect()).unwrap_or_default();
-    assert_eq!(
-        license_keys,
-        vec!["licensed"],
-        "public license endpoint must not expose license metadata: {license_body}"
-    );
-    assert!(
-        !license_body.contains(&env.secret_key) && !license_body.contains(&env.access_key),
-        "console license response leaks credentials: {license_body}"
-    );
-
-    // --- console SPA prefix dispatches to the static handler, not the S3 API -
-    // With release console assets embedded this is 200 text/html; a from-source
-    // binary embeds an empty static dir and serves the handler's 404 fallback.
-    // Either way it must never fall through to an S3 handler (S3 error XML).
-    let spa_response = client.get(format!("{console_base}/rustfs/console/")).send().await?;
-    let spa_status = spa_response.status();
-    let spa_content_type = spa_response
-        .headers()
-        .get(reqwest::header::CONTENT_TYPE)
-        .and_then(|v| v.to_str().ok())
-        .unwrap_or_default()
-        .to_string();
-    let spa_body = spa_response.text().await?;
-    assert!(
-        !spa_body.contains("<Error>"),
-        "console SPA route fell through to the S3 API: status {spa_status}, body {spa_body}"
-    );
-    match spa_status {
-        reqwest::StatusCode::OK => {
-            assert!(
-                spa_content_type.starts_with("text/html"),
-                "embedded console index must be html, got {spa_content_type}"
-            );
-        }
-        reqwest::StatusCode::NOT_FOUND => {
-            assert!(
-                spa_body.contains("RustFS"),
-                "asset-less console 404 must come from the console static handler: {spa_body}"
-            );
-        }
-        other => panic!("console SPA route returned unexpected status {other}: {spa_body}"),
-    }
-
-    // --- protected surface on the console listener stays authenticated -------
-    let admin_response = client.get(format!("{console_base}/rustfs/admin/v3/info")).send().await?;
-    assert_eq!(
-        admin_response.status(),
-        reqwest::StatusCode::FORBIDDEN,
-        "unauthenticated admin API on the console listener must be denied"
-    );
-    let admin_body = admin_response.text().await?;
-    assert!(admin_body.contains("AccessDenied"), "admin denial must be AccessDenied: {admin_body}");
-
-    let s3_root_response = client.get(format!("{console_base}/")).send().await?;
-    assert_eq!(
-        s3_root_response.status(),
-        reqwest::StatusCode::FORBIDDEN,
-        "unauthenticated S3 root on the console listener must be denied"
-    );
-
-    // --- console disabled on a listener means no console surface at all ------
-    // The main S3 listener of this same process runs with the console disabled,
-    // so its console paths must not answer with the unauthenticated console
-    // payloads (they fall through to the authenticated S3 surface instead).
-    let s3_listener_console = client.get(format!("{}/rustfs/console/version", env.url)).send().await?;
-    assert_ne!(
-        s3_listener_console.status(),
-        reqwest::StatusCode::OK,
-        "console endpoints must not be served on a console-disabled listener"
-    );
-
-    env.stop_server();
-    Ok(())
-}

crates/e2e_test/src/content_encoding_test.rs

@@ -80,25 +80,6 @@ mod tests {
         assert_eq!(head_resp.content_encoding(), Some("zstd"), "HEAD should return Content-Encoding: zstd");
         assert_eq!(head_resp.content_type(), Some("text/plain"), "HEAD should return correct Content-Type");
 
-        client
-            .delete_object()
-            .bucket(bucket)
-            .key(key)
-            .send()
-            .await
-            .expect("DELETE object failed");
-        client
-            .delete_bucket()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("DELETE bucket failed");
-        client
-            .list_buckets()
-            .send()
-            .await
-            .expect("RustFS must remain available after deleting a bucket");
-
         env.stop_server();
     }
 

crates/e2e_test/src/copy_object_checksum_test.rs

@@ -1,708 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! CopyObject checksum compatibility tests. Covers all supported algorithms,
-//! source-checksum preservation, explicit override, and fail-closed handling of
-//! unsupported algorithms before destination mutation.
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestEnvironment, init_logging};
-    use aws_sdk_s3::config::{Credentials, Region, RequestChecksumCalculation};
-    use aws_sdk_s3::error::ProvideErrorMetadata;
-    use aws_sdk_s3::primitives::ByteStream;
-    use aws_sdk_s3::types::{
-        BucketVersioningStatus, ChecksumAlgorithm, ChecksumMode, ChecksumType, CompletedMultipartUpload, CompletedPart,
-        VersioningConfiguration,
-    };
-    use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
-    use base64::Engine as _;
-    use base64::engine::general_purpose::STANDARD as BASE64;
-    use rustfs_rio::{Checksum, ChecksumType as RioChecksumType};
-    use serial_test::serial;
-    use sha2::{Digest, Sha256};
-    use tracing::info;
-
-    async fn create_versioned_bucket(client: &aws_sdk_s3::Client, bucket: &str) {
-        client
-            .create_bucket()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("Failed to create bucket");
-        client
-            .put_bucket_versioning()
-            .bucket(bucket)
-            .versioning_configuration(
-                VersioningConfiguration::builder()
-                    .status(BucketVersioningStatus::Enabled)
-                    .build(),
-            )
-            .send()
-            .await
-            .expect("Failed to enable versioning");
-    }
-
-    fn create_s3_client_no_auto_checksum(env: &RustFSTestEnvironment) -> aws_sdk_s3::Client {
-        let credentials = Credentials::new(&env.access_key, &env.secret_key, None, None, "copy-checksum-e2e");
-        let config = aws_sdk_s3::Config::builder()
-            .credentials_provider(credentials)
-            .region(Region::new("us-east-1"))
-            .endpoint_url(format!("http://{}", env.address))
-            .force_path_style(true)
-            .behavior_version_latest()
-            .request_checksum_calculation(RequestChecksumCalculation::WhenRequired)
-            .http_client(SmithyHttpClientBuilder::new().build_http())
-            .build();
-        aws_sdk_s3::Client::from_conf(config)
-    }
-
-    fn algorithms() -> [(ChecksumAlgorithm, RioChecksumType); 10] {
-        [
-            (ChecksumAlgorithm::Crc32, RioChecksumType::CRC32),
-            (ChecksumAlgorithm::Crc32C, RioChecksumType::CRC32C),
-            (ChecksumAlgorithm::Crc64Nvme, RioChecksumType::CRC64_NVME),
-            (ChecksumAlgorithm::Sha1, RioChecksumType::SHA1),
-            (ChecksumAlgorithm::Sha256, RioChecksumType::SHA256),
-            (ChecksumAlgorithm::Md5, RioChecksumType::MD5),
-            (ChecksumAlgorithm::Sha512, RioChecksumType::SHA512),
-            (ChecksumAlgorithm::Xxhash3, RioChecksumType::XXHASH3),
-            (ChecksumAlgorithm::Xxhash64, RioChecksumType::XXHASH64),
-            (ChecksumAlgorithm::Xxhash128, RioChecksumType::XXHASH128),
-        ]
-    }
-
-    fn result_checksums(result: &aws_sdk_s3::types::CopyObjectResult) -> [Option<&str>; 10] {
-        [
-            result.checksum_crc32(),
-            result.checksum_crc32_c(),
-            result.checksum_crc64_nvme(),
-            result.checksum_sha1(),
-            result.checksum_sha256(),
-            result.checksum_md5(),
-            result.checksum_sha512(),
-            result.checksum_xxhash3(),
-            result.checksum_xxhash64(),
-            result.checksum_xxhash128(),
-        ]
-    }
-
-    fn head_checksums(output: &aws_sdk_s3::operation::head_object::HeadObjectOutput) -> [Option<&str>; 10] {
-        [
-            output.checksum_crc32(),
-            output.checksum_crc32_c(),
-            output.checksum_crc64_nvme(),
-            output.checksum_sha1(),
-            output.checksum_sha256(),
-            output.checksum_md5(),
-            output.checksum_sha512(),
-            output.checksum_xxhash3(),
-            output.checksum_xxhash64(),
-            output.checksum_xxhash128(),
-        ]
-    }
-
-    #[tokio::test]
-    #[serial]
-    async fn test_copy_supports_all_checksum_algorithms() {
-        init_logging();
-
-        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
-        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
-
-        let client = create_s3_client_no_auto_checksum(&env);
-        let src_bucket = "copy-all-checksums-src";
-        let dst_bucket = "copy-all-checksums-dst";
-        let src_key = "objects/source.bin";
-        let content = b"deterministic CopyObject payload for all ten checksum algorithms";
-
-        create_versioned_bucket(&client, src_bucket).await;
-        create_versioned_bucket(&client, dst_bucket).await;
-        client
-            .put_object()
-            .bucket(src_bucket)
-            .key(src_key)
-            .body(ByteStream::from_static(content))
-            .send()
-            .await
-            .expect("PUT source failed");
-
-        for (index, (sdk_algorithm, rio_algorithm)) in algorithms().into_iter().enumerate() {
-            let expected = Checksum::new_from_data(rio_algorithm, content)
-                .expect("supported checksum must be computable")
-                .encoded;
-            let dst_key = format!("objects/destination-{index}.bin");
-            let copy = client
-                .copy_object()
-                .bucket(dst_bucket)
-                .key(&dst_key)
-                .copy_source(format!("{src_bucket}/{src_key}"))
-                .checksum_algorithm(sdk_algorithm)
-                .send()
-                .await
-                .expect("CopyObject with supported checksum must succeed");
-            let result = copy.copy_object_result().expect("CopyObject result");
-            let checksums = result_checksums(result);
-            assert_eq!(checksums[index], Some(expected.as_str()), "{rio_algorithm}: response checksum");
-            assert_eq!(
-                checksums.iter().filter(|checksum| checksum.is_some()).count(),
-                1,
-                "{rio_algorithm}: only the requested checksum may be returned"
-            );
-
-            let head = client
-                .head_object()
-                .bucket(dst_bucket)
-                .key(&dst_key)
-                .checksum_mode(ChecksumMode::Enabled)
-                .send()
-                .await
-                .expect("HEAD destination failed");
-            let checksums = head_checksums(&head);
-            assert_eq!(checksums[index], Some(expected.as_str()), "{rio_algorithm}: persisted checksum");
-            assert_eq!(
-                checksums.iter().filter(|checksum| checksum.is_some()).count(),
-                1,
-                "{rio_algorithm}: destination must persist only the requested checksum"
-            );
-
-            let body = client
-                .get_object()
-                .bucket(dst_bucket)
-                .key(&dst_key)
-                .send()
-                .await
-                .expect("GET destination failed")
-                .body
-                .collect()
-                .await
-                .expect("collect destination body")
-                .into_bytes();
-            assert_eq!(body.as_ref(), content, "{rio_algorithm}: full copied body");
-        }
-
-        env.stop_server();
-    }
-
-    #[tokio::test]
-    #[serial]
-    async fn test_copy_without_algorithm_preserves_every_supported_source_checksum() {
-        init_logging();
-
-        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
-        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
-
-        let client = create_s3_client_no_auto_checksum(&env);
-        let src_bucket = "copy-preserve-all-src";
-        let dst_bucket = "copy-preserve-all-dst";
-        let content = b"source checksum preservation payload for all ten algorithms";
-
-        create_versioned_bucket(&client, src_bucket).await;
-        create_versioned_bucket(&client, dst_bucket).await;
-
-        for (index, (_sdk_algorithm, rio_algorithm)) in algorithms().into_iter().enumerate() {
-            let expected = Checksum::new_from_data(rio_algorithm, content)
-                .expect("supported checksum must be computable")
-                .encoded;
-            let checksum_header = rio_algorithm.key().expect("supported checksum header");
-            let request_checksum = expected.clone();
-            let src_key = format!("objects/source-{index}.bin");
-            let dst_key = format!("objects/destination-{index}.bin");
-            client
-                .put_object()
-                .bucket(src_bucket)
-                .key(&src_key)
-                .body(ByteStream::from_static(content))
-                .customize()
-                .mutate_request(move |request| {
-                    request.headers_mut().insert(checksum_header, request_checksum.clone());
-                })
-                .send()
-                .await
-                .expect("PUT checksummed source failed");
-
-            let copy = client
-                .copy_object()
-                .bucket(dst_bucket)
-                .key(&dst_key)
-                .copy_source(format!("{src_bucket}/{src_key}"))
-                .send()
-                .await
-                .expect("CopyObject without algorithm must succeed");
-            let result = copy.copy_object_result().expect("CopyObject result");
-            let checksums = result_checksums(result);
-            assert_eq!(checksums[index], Some(expected.as_str()), "{rio_algorithm}: preserved response checksum");
-            assert_eq!(checksums.iter().filter(|checksum| checksum.is_some()).count(), 1);
-
-            let head = client
-                .head_object()
-                .bucket(dst_bucket)
-                .key(&dst_key)
-                .checksum_mode(ChecksumMode::Enabled)
-                .send()
-                .await
-                .expect("HEAD destination failed");
-            let checksums = head_checksums(&head);
-            assert_eq!(checksums[index], Some(expected.as_str()), "{rio_algorithm}: preserved stored checksum");
-            assert_eq!(checksums.iter().filter(|checksum| checksum.is_some()).count(), 1);
-        }
-
-        env.stop_server();
-    }
-
-    #[tokio::test]
-    #[serial]
-    async fn test_copy_without_algorithm_preserves_composite_checksum_type() {
-        init_logging();
-
-        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
-        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
-
-        let client = create_s3_client_no_auto_checksum(&env);
-        let bucket = "copy-preserve-composite";
-        let source_key = "objects/multipart-source.bin";
-        let destination_key = "objects/copied-multipart.bin";
-        let content = b"multipart source checksum must remain composite";
-
-        create_versioned_bucket(&client, bucket).await;
-        let created = client
-            .create_multipart_upload()
-            .bucket(bucket)
-            .key(source_key)
-            .checksum_algorithm(ChecksumAlgorithm::Sha256)
-            .send()
-            .await
-            .expect("CreateMultipartUpload failed");
-        let upload_id = created.upload_id().expect("multipart upload ID");
-        let checksum = Checksum::new_from_data(RioChecksumType::SHA256, content)
-            .expect("SHA256 checksum")
-            .encoded;
-        let uploaded = client
-            .upload_part()
-            .bucket(bucket)
-            .key(source_key)
-            .upload_id(upload_id)
-            .part_number(1)
-            .checksum_sha256(&checksum)
-            .body(ByteStream::from_static(content))
-            .send()
-            .await
-            .expect("UploadPart failed");
-        let completed_part = CompletedPart::builder()
-            .part_number(1)
-            .e_tag(uploaded.e_tag().expect("part ETag"))
-            .checksum_sha256(uploaded.checksum_sha256().expect("part checksum"))
-            .build();
-        client
-            .complete_multipart_upload()
-            .bucket(bucket)
-            .key(source_key)
-            .upload_id(upload_id)
-            .multipart_upload(CompletedMultipartUpload::builder().parts(completed_part).build())
-            .send()
-            .await
-            .expect("CompleteMultipartUpload failed");
-
-        let source_head = client
-            .head_object()
-            .bucket(bucket)
-            .key(source_key)
-            .checksum_mode(ChecksumMode::Enabled)
-            .send()
-            .await
-            .expect("HEAD multipart source failed");
-        let source_checksum = source_head.checksum_sha256().expect("multipart source checksum");
-        assert_eq!(source_head.checksum_type(), Some(&ChecksumType::Composite));
-
-        let copied = client
-            .copy_object()
-            .bucket(bucket)
-            .key(destination_key)
-            .copy_source(format!("{bucket}/{source_key}"))
-            .send()
-            .await
-            .expect("CopyObject without algorithm failed");
-        let result = copied.copy_object_result().expect("CopyObject result");
-        assert_eq!(result.checksum_sha256(), Some(source_checksum));
-        assert_eq!(result.checksum_type(), Some(&ChecksumType::Composite));
-
-        let destination_head = client
-            .head_object()
-            .bucket(bucket)
-            .key(destination_key)
-            .checksum_mode(ChecksumMode::Enabled)
-            .send()
-            .await
-            .expect("HEAD copied multipart object failed");
-        assert_eq!(destination_head.checksum_sha256(), Some(source_checksum));
-        assert_eq!(destination_head.checksum_type(), Some(&ChecksumType::Composite));
-
-        env.stop_server();
-    }
-
-    #[tokio::test]
-    #[serial]
-    async fn test_copy_rejects_unknown_algorithm_without_destination_mutation() {
-        init_logging();
-
-        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
-        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "copy-reject-unknown-checksum";
-        let src_key = "objects/source.bin";
-        let dst_key = "objects/destination.bin";
-        let source = b"source must never replace destination";
-        let destination = b"pre-existing destination must remain byte-for-byte unchanged";
-        let expected = Checksum::new_from_data(RioChecksumType::SHA256, destination)
-            .expect("SHA256 checksum")
-            .encoded;
-
-        create_versioned_bucket(&client, bucket).await;
-        client
-            .put_object()
-            .bucket(bucket)
-            .key(src_key)
-            .body(ByteStream::from_static(source))
-            .send()
-            .await
-            .expect("PUT source failed");
-        let original = client
-            .put_object()
-            .bucket(bucket)
-            .key(dst_key)
-            .metadata("state", "original")
-            .checksum_algorithm(ChecksumAlgorithm::Sha256)
-            .body(ByteStream::from_static(destination))
-            .send()
-            .await
-            .expect("PUT destination failed");
-        let original_version = original.version_id().expect("versioned PUT must return a version id");
-
-        let missing_source_error = client
-            .copy_object()
-            .bucket(bucket)
-            .key(dst_key)
-            .copy_source(format!("{bucket}/objects/missing-source.bin"))
-            .checksum_algorithm(ChecksumAlgorithm::from("BLAKE3"))
-            .send()
-            .await
-            .expect_err("checksum validation must precede source lookup");
-        assert_eq!(
-            missing_source_error.as_service_error().and_then(|value| value.code()),
-            Some("InvalidArgument")
-        );
-        assert_eq!(missing_source_error.raw_response().map(|response| response.status().as_u16()), Some(400));
-
-        let error = client
-            .copy_object()
-            .bucket(bucket)
-            .key(dst_key)
-            .copy_source(format!("{bucket}/{src_key}"))
-            .checksum_algorithm(ChecksumAlgorithm::from("BLAKE3"))
-            .send()
-            .await
-            .expect_err("unsupported checksum algorithm must fail");
-        assert_eq!(error.as_service_error().and_then(|value| value.code()), Some("InvalidArgument"));
-        assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(400));
-
-        let head = client
-            .head_object()
-            .bucket(bucket)
-            .key(dst_key)
-            .checksum_mode(ChecksumMode::Enabled)
-            .send()
-            .await
-            .expect("HEAD unchanged destination");
-        assert_eq!(head.version_id(), Some(original_version));
-        assert_eq!(
-            head.metadata().and_then(|metadata| metadata.get("state").map(String::as_str)),
-            Some("original")
-        );
-        assert_eq!(head.checksum_sha256(), Some(expected.as_str()));
-
-        let body = client
-            .get_object()
-            .bucket(bucket)
-            .key(dst_key)
-            .send()
-            .await
-            .expect("GET unchanged destination")
-            .body
-            .collect()
-            .await
-            .expect("collect unchanged destination")
-            .into_bytes();
-        assert_eq!(body.as_ref(), destination);
-
-        env.stop_server();
-    }
-
-    /// Requested algorithm: a CopyObject asking for SHA256 must compute it over the copied
-    /// bytes, return it in `CopyObjectResult.ChecksumSHA256`, and persist it so a checksum-mode
-    /// HEAD on the destination returns the identical value.
-    #[tokio::test]
-    #[serial]
-    async fn test_copy_with_checksum_algorithm_returns_and_persists_sha256() {
-        init_logging();
-        info!("Issue #4996: CopyObject with ChecksumAlgorithm=SHA256 must return and persist the checksum");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
-        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
-
-        let client = env.create_s3_client();
-        let src_bucket = "copy-checksum-req-src";
-        let dst_bucket = "copy-checksum-req-dst";
-        let src_key = "objects/source.bin";
-        let dst_key = "objects/dest.bin";
-
-        create_versioned_bucket(&client, src_bucket).await;
-        create_versioned_bucket(&client, dst_bucket).await;
-
-        let content = b"deterministic synthetic payload for copy-object checksum #4996";
-        let expected_sha256 = BASE64.encode(Sha256::digest(content));
-
-        client
-            .put_object()
-            .bucket(src_bucket)
-            .key(src_key)
-            .body(ByteStream::from_static(content))
-            .send()
-            .await
-            .expect("PUT source failed");
-
-        let copy_out = client
-            .copy_object()
-            .bucket(dst_bucket)
-            .key(dst_key)
-            .copy_source(format!("{src_bucket}/{src_key}"))
-            .checksum_algorithm(ChecksumAlgorithm::Sha256)
-            .send()
-            .await
-            .expect("CopyObject with ChecksumAlgorithm must succeed");
-
-        // (3) The response must carry the freshly computed SHA-256 of the copied bytes.
-        let result = copy_out
-            .copy_object_result()
-            .expect("issue #4996: CopyObject must return a CopyObjectResult");
-        assert_eq!(
-            result.checksum_sha256(),
-            Some(expected_sha256.as_str()),
-            "issue #4996: CopyObjectResult.ChecksumSHA256 must equal the SHA-256 of the copied bytes"
-        );
-
-        // (4) A checksum-mode HEAD on the destination must return the same SHA-256.
-        let head = client
-            .head_object()
-            .bucket(dst_bucket)
-            .key(dst_key)
-            .checksum_mode(ChecksumMode::Enabled)
-            .send()
-            .await
-            .expect("HEAD destination failed");
-        assert_eq!(
-            head.checksum_sha256(),
-            Some(expected_sha256.as_str()),
-            "issue #4996: destination checksum-mode HEAD must return the same SHA-256 the copy reported"
-        );
-
-        env.stop_server();
-    }
-
-    /// No algorithm requested: when the source object already carries a checksum, the copy must
-    /// preserve it on the destination (AWS default), visible via a checksum-mode HEAD.
-    #[tokio::test]
-    #[serial]
-    async fn test_copy_without_algorithm_preserves_source_checksum() {
-        init_logging();
-        info!("Issue #4996: CopyObject without ChecksumAlgorithm must preserve the source object's checksum");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
-        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
-
-        let client = env.create_s3_client();
-        let src_bucket = "copy-checksum-preserve-src";
-        let dst_bucket = "copy-checksum-preserve-dst";
-        let src_key = "objects/source.bin";
-        let dst_key = "objects/dest.bin";
-
-        create_versioned_bucket(&client, src_bucket).await;
-        create_versioned_bucket(&client, dst_bucket).await;
-
-        let content = b"another deterministic payload whose source checksum must survive the copy";
-        let expected_sha256 = BASE64.encode(Sha256::digest(content));
-
-        // Store the source WITH a SHA-256 checksum so it has one to preserve.
-        let put_src = client
-            .put_object()
-            .bucket(src_bucket)
-            .key(src_key)
-            .checksum_algorithm(ChecksumAlgorithm::Sha256)
-            .body(ByteStream::from_static(content))
-            .send()
-            .await
-            .expect("PUT source with checksum failed");
-        assert_eq!(
-            put_src.checksum_sha256(),
-            Some(expected_sha256.as_str()),
-            "source PUT must report the SHA-256 it stored"
-        );
-
-        // Copy WITHOUT specifying a checksum algorithm.
-        let copy_out = client
-            .copy_object()
-            .bucket(dst_bucket)
-            .key(dst_key)
-            .copy_source(format!("{src_bucket}/{src_key}"))
-            .send()
-            .await
-            .expect("CopyObject without ChecksumAlgorithm must succeed");
-
-        // The response should echo the preserved source checksum.
-        let result = copy_out
-            .copy_object_result()
-            .expect("issue #4996: CopyObject must return a CopyObjectResult");
-        assert_eq!(
-            result.checksum_sha256(),
-            Some(expected_sha256.as_str()),
-            "issue #4996: a no-algorithm copy must preserve and report the source object's SHA-256"
-        );
-
-        // And a checksum-mode HEAD on the destination must return that same preserved SHA-256.
-        let head = client
-            .head_object()
-            .bucket(dst_bucket)
-            .key(dst_key)
-            .checksum_mode(ChecksumMode::Enabled)
-            .send()
-            .await
-            .expect("HEAD destination failed");
-        assert_eq!(
-            head.checksum_sha256(),
-            Some(expected_sha256.as_str()),
-            "issue #4996: destination checksum-mode HEAD must return the preserved source SHA-256"
-        );
-
-        env.stop_server();
-    }
-
-    /// Requested algorithm differs from the source's: a source stored with SHA256, copied while
-    /// requesting CRC32, must return/persist the freshly computed CRC32 and must NOT carry the
-    /// source's SHA256 through. Guards the request-over-source precedence and the destination's
-    /// checksum-not-inherited path, and exercises the CRC32 code path (a different branch of
-    /// ChecksumType::from_string than SHA256).
-    #[tokio::test]
-    #[serial]
-    async fn test_copy_requested_algorithm_overrides_source_checksum() {
-        init_logging();
-        info!("Issue #4996: a requested CopyObject checksum algorithm must override the source object's algorithm");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
-        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
-
-        let client = env.create_s3_client();
-        let src_bucket = "copy-checksum-override-src";
-        let dst_bucket = "copy-checksum-override-dst";
-        let src_key = "objects/source.bin";
-        let ref_key = "objects/reference-crc32.bin";
-        let dst_key = "objects/dest.bin";
-
-        create_versioned_bucket(&client, src_bucket).await;
-        create_versioned_bucket(&client, dst_bucket).await;
-
-        let content = b"payload whose copy must be re-checksummed with a different algorithm";
-        let expected_sha256 = BASE64.encode(Sha256::digest(content));
-
-        // Source is stored WITH a SHA-256 checksum.
-        client
-            .put_object()
-            .bucket(src_bucket)
-            .key(src_key)
-            .checksum_algorithm(ChecksumAlgorithm::Sha256)
-            .body(ByteStream::from_static(content))
-            .send()
-            .await
-            .expect("PUT source with SHA256 failed");
-
-        // Establish the canonical CRC32 the server computes for this content via a reference PUT,
-        // so the copy's CRC32 can be asserted against an exact server-computed value.
-        let ref_put = client
-            .put_object()
-            .bucket(src_bucket)
-            .key(ref_key)
-            .checksum_algorithm(ChecksumAlgorithm::Crc32)
-            .body(ByteStream::from_static(content))
-            .send()
-            .await
-            .expect("reference PUT with CRC32 failed");
-        let expected_crc32 = ref_put
-            .checksum_crc32()
-            .expect("reference PUT must report a CRC32")
-            .to_string();
-
-        // Copy the SHA256 source while requesting CRC32.
-        let copy_out = client
-            .copy_object()
-            .bucket(dst_bucket)
-            .key(dst_key)
-            .copy_source(format!("{src_bucket}/{src_key}"))
-            .checksum_algorithm(ChecksumAlgorithm::Crc32)
-            .send()
-            .await
-            .expect("CopyObject requesting a different algorithm must succeed");
-
-        let result = copy_out
-            .copy_object_result()
-            .expect("issue #4996: CopyObject must return a CopyObjectResult");
-        // The requested CRC32 must be computed and returned.
-        assert_eq!(
-            result.checksum_crc32(),
-            Some(expected_crc32.as_str()),
-            "issue #4996: a requested CRC32 must be computed fresh over the copied bytes"
-        );
-        // The source's SHA256 must NOT leak through — the requested algorithm wins.
-        assert_eq!(
-            result.checksum_sha256(),
-            None,
-            "issue #4996: the source object's SHA256 must not be inherited when a different algorithm is requested"
-        );
-        assert_ne!(
-            result.checksum_crc32(),
-            Some(expected_sha256.as_str()),
-            "sanity: CRC32 field must not carry the SHA256 value"
-        );
-
-        // The destination must persist CRC32 (and only CRC32) for a checksum-mode HEAD.
-        let head = client
-            .head_object()
-            .bucket(dst_bucket)
-            .key(dst_key)
-            .checksum_mode(ChecksumMode::Enabled)
-            .send()
-            .await
-            .expect("HEAD destination failed");
-        assert_eq!(
-            head.checksum_crc32(),
-            Some(expected_crc32.as_str()),
-            "issue #4996: destination checksum-mode HEAD must return the requested CRC32"
-        );
-        assert_eq!(
-            head.checksum_sha256(),
-            None,
-            "issue #4996: destination must not report the source's SHA256 after an override copy"
-        );
-
-        env.stop_server();
-    }
-}

crates/e2e_test/src/copy_object_metadata_test.rs

@@ -17,17 +17,14 @@
 #[cfg(test)]
 mod tests {
     use crate::common::{RustFSTestEnvironment, init_logging};
-    use aws_sdk_s3::error::ProvideErrorMetadata;
-    use aws_sdk_s3::primitives::{ByteStream, DateTime, DateTimeFormat};
-    use aws_sdk_s3::types::{
-        BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, MetadataDirective, StorageClass, VersioningConfiguration,
-    };
+    use aws_sdk_s3::primitives::ByteStream;
+    use aws_sdk_s3::types::MetadataDirective;
     use serial_test::serial;
     use tracing::info;
 
     #[tokio::test]
     #[serial]
-    async fn copy_object_standard_metadata_copy_replace_and_clear() {
+    async fn test_self_copy_replace_metadata_preserves_readable_object() {
         init_logging();
         info!("Issue #2789: self-copy metadata replacement must preserve object data");
 
@@ -38,14 +35,6 @@ mod tests {
         let bucket = "self-copy-metadata-replace-test";
         let key = "assets/chunk-2F3R7JUG.js";
         let content = b"console.log('metadata replacement should keep object data readable');";
-        let source_expires = DateTime::from_secs(1_893_456_000);
-        let source_expires_http_date = source_expires
-            .fmt(DateTimeFormat::HttpDate)
-            .expect("Test timestamp should format as an HTTP date");
-        let replacement_expires = DateTime::from_secs(1_924_992_000);
-        let replacement_expires_http_date = replacement_expires
-            .fmt(DateTimeFormat::HttpDate)
-            .expect("Test timestamp should format as an HTTP date");
 
         client
             .create_bucket()
@@ -58,14 +47,7 @@ mod tests {
             .put_object()
             .bucket(bucket)
             .key(key)
-            .cache_control("max-age=60")
-            .content_disposition("inline; filename=source.js")
-            .content_encoding("br")
-            .content_language("en-US")
             .content_type("text/javascript; charset=utf-8")
-            .expires(source_expires)
-            .website_redirect_location("/source.html")
-            .storage_class(StorageClass::ReducedRedundancy)
             .metadata("mtime", "1777992333")
             .metadata("stale", "must-be-removed")
             .body(ByteStream::from_static(content))
@@ -73,120 +55,13 @@ mod tests {
             .await
             .expect("PUT failed");
 
-        let copied_key = "assets/default-copy.js";
-        client
-            .copy_object()
-            .bucket(bucket)
-            .key(copied_key)
-            .copy_source(format!("{bucket}/{key}"))
-            .send()
-            .await
-            .expect("default CopyObject failed");
-
-        let copied_head = client
-            .head_object()
-            .bucket(bucket)
-            .key(copied_key)
-            .send()
-            .await
-            .expect("HEAD failed after default copy");
-        assert_eq!(copied_head.cache_control(), Some("max-age=60"));
-        assert_eq!(copied_head.content_disposition(), Some("inline; filename=source.js"));
-        assert_eq!(copied_head.content_encoding(), Some("br"));
-        assert_eq!(copied_head.content_language(), Some("en-US"));
-        assert_eq!(copied_head.content_type(), Some("text/javascript; charset=utf-8"));
-        assert_eq!(copied_head.expires_string(), Some(source_expires_http_date.as_str()));
-        assert_eq!(
-            copied_head.storage_class(),
-            None,
-            "CopyObject without a storage class should write STANDARD"
-        );
-        assert_eq!(
-            copied_head.website_redirect_location(),
-            Some("/source.html"),
-            "default CopyObject should preserve source metadata"
-        );
-        assert_eq!(
-            copied_head.metadata().and_then(|metadata| metadata.get("stale")),
-            Some(&"must-be-removed".to_string())
-        );
-
-        client
-            .copy_object()
-            .bucket(bucket)
-            .key("assets/explicit-copy.js")
-            .copy_source(format!("{bucket}/{key}"))
-            .metadata_directive(MetadataDirective::Copy)
-            .send()
-            .await
-            .expect("explicit COPY directive failed");
-        let explicit_copy_head = client
-            .head_object()
-            .bucket(bucket)
-            .key("assets/explicit-copy.js")
-            .send()
-            .await
-            .expect("HEAD failed after explicit COPY");
-        assert_eq!(explicit_copy_head.cache_control(), Some("max-age=60"));
-        assert_eq!(
-            explicit_copy_head.website_redirect_location(),
-            None,
-            "explicit COPY does not inherit website redirect metadata"
-        );
-
-        client
-            .copy_object()
-            .bucket(bucket)
-            .key("assets/explicit-copy-redirect.js")
-            .copy_source(format!("{bucket}/{key}"))
-            .metadata_directive(MetadataDirective::Copy)
-            .website_redirect_location("/explicit-copy.html")
-            .send()
-            .await
-            .expect("explicit COPY with redirect failed");
-        let explicit_redirect_head = client
-            .head_object()
-            .bucket(bucket)
-            .key("assets/explicit-copy-redirect.js")
-            .send()
-            .await
-            .expect("HEAD failed after explicit COPY with redirect");
-        assert_eq!(explicit_redirect_head.website_redirect_location(), Some("/explicit-copy.html"));
-
-        client
-            .copy_object()
-            .bucket(bucket)
-            .key("assets/explicit-storage-class.js")
-            .copy_source(format!("{bucket}/{key}"))
-            .storage_class(StorageClass::ReducedRedundancy)
-            .send()
-            .await
-            .expect("CopyObject with an explicit storage class failed");
-        let explicit_storage_class_head = client
-            .head_object()
-            .bucket(bucket)
-            .key("assets/explicit-storage-class.js")
-            .send()
-            .await
-            .expect("HEAD failed after explicit storage class copy");
-        assert_eq!(
-            explicit_storage_class_head.storage_class().map(StorageClass::as_str),
-            Some("REDUCED_REDUNDANCY")
-        );
-
         client
             .copy_object()
             .bucket(bucket)
             .key(key)
             .copy_source(format!("{bucket}/{key}"))
             .metadata_directive(MetadataDirective::Replace)
-            .cache_control("no-cache")
-            .content_disposition("attachment; filename=replaced.js")
-            .content_encoding("gzip")
-            .content_language("fr-FR")
-            .content_type("application/javascript")
-            .expires(replacement_expires)
-            .website_redirect_location("/replaced.html")
+            .content_type("text/javascript; charset=utf-8")
             .metadata("mtime", "1777992348")
             .send()
             .await
@@ -210,14 +85,6 @@ mod tests {
             None,
             "HEAD should not return metadata omitted by REPLACE"
         );
-        assert_eq!(head_resp.cache_control(), Some("no-cache"));
-        assert_eq!(head_resp.content_disposition(), Some("attachment; filename=replaced.js"));
-        assert_eq!(head_resp.content_encoding(), Some("gzip"));
-        assert_eq!(head_resp.content_language(), Some("fr-FR"));
-        assert_eq!(head_resp.content_type(), Some("application/javascript"));
-        assert_eq!(head_resp.expires_string(), Some(replacement_expires_http_date.as_str()));
-        assert_eq!(head_resp.website_redirect_location(), Some("/replaced.html"));
-        assert_eq!(head_resp.storage_class(), None, "REPLACE without a storage class should write STANDARD");
 
         let get_resp = client
             .get_object()
@@ -256,13 +123,6 @@ mod tests {
             None,
             "HEAD should not return metadata omitted by empty REPLACE"
         );
-        assert_eq!(empty_head_resp.cache_control(), None);
-        assert_eq!(empty_head_resp.content_disposition(), None);
-        assert_eq!(empty_head_resp.content_encoding(), None);
-        assert_eq!(empty_head_resp.content_language(), None);
-        assert_eq!(empty_head_resp.content_type(), None);
-        assert_eq!(empty_head_resp.expires_string(), None);
-        assert_eq!(empty_head_resp.website_redirect_location(), None);
 
         let empty_get_resp = client
             .get_object()
@@ -281,333 +141,4 @@ mod tests {
 
         env.stop_server();
     }
-
-    #[tokio::test]
-    #[serial]
-    async fn copy_object_replace_accepts_each_standard_field_independently() {
-        init_logging();
-        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
-        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "copy-object-metadata-fields";
-        let source = "source.txt";
-        client
-            .create_bucket()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("Failed to create bucket");
-        client
-            .put_object()
-            .bucket(bucket)
-            .key(source)
-            .cache_control("source-cache")
-            .content_disposition("inline")
-            .content_encoding("br")
-            .content_language("en")
-            .content_type("text/source")
-            .expires(DateTime::from_secs(1_893_456_000))
-            .body(ByteStream::from_static(b"field-by-field"))
-            .send()
-            .await
-            .expect("PUT failed");
-        let replacement_expires = DateTime::from_secs(1_924_992_000);
-        let replacement_expires_http_date = replacement_expires
-            .fmt(DateTimeFormat::HttpDate)
-            .expect("Test timestamp should format as an HTTP date");
-
-        for field in [
-            "cache-control",
-            "content-disposition",
-            "content-encoding",
-            "content-language",
-            "content-type",
-            "expires",
-            "website-redirect",
-        ] {
-            let destination = format!("{field}.txt");
-            let request = client
-                .copy_object()
-                .bucket(bucket)
-                .key(&destination)
-                .copy_source(format!("{bucket}/{source}"))
-                .metadata_directive(MetadataDirective::Replace);
-            let request = match field {
-                "cache-control" => request.cache_control("field-cache"),
-                "content-disposition" => request.content_disposition("attachment"),
-                "content-encoding" => request.content_encoding("gzip"),
-                "content-language" => request.content_language("de"),
-                "content-type" => request.content_type("text/field"),
-                "expires" => request.expires(replacement_expires),
-                "website-redirect" => request.website_redirect_location("/field.html"),
-                _ => unreachable!("field table contains only supported entries"),
-            };
-            request.send().await.expect("field-specific CopyObject failed");
-
-            let head = client
-                .head_object()
-                .bucket(bucket)
-                .key(&destination)
-                .send()
-                .await
-                .expect("HEAD failed");
-            assert_eq!(head.cache_control(), (field == "cache-control").then_some("field-cache"));
-            assert_eq!(head.content_disposition(), (field == "content-disposition").then_some("attachment"));
-            assert_eq!(head.content_encoding(), (field == "content-encoding").then_some("gzip"));
-            assert_eq!(head.content_language(), (field == "content-language").then_some("de"));
-            assert_eq!(head.content_type(), (field == "content-type").then_some("text/field"));
-            assert_eq!(
-                head.expires_string(),
-                (field == "expires").then_some(replacement_expires_http_date.as_str())
-            );
-            assert_eq!(head.website_redirect_location(), (field == "website-redirect").then_some("/field.html"));
-        }
-
-        client
-            .copy_object()
-            .bucket(bucket)
-            .key("user-metadata-collision.txt")
-            .copy_source(format!("{bucket}/{source}"))
-            .metadata_directive(MetadataDirective::Replace)
-            .metadata("content-type", "user-content-type")
-            .metadata("content-encoding", "user-content-encoding")
-            .send()
-            .await
-            .expect("CopyObject should preserve user metadata namespaces");
-        let collision_head = client
-            .head_object()
-            .bucket(bucket)
-            .key("user-metadata-collision.txt")
-            .send()
-            .await
-            .expect("HEAD failed for metadata collision case");
-        assert_eq!(collision_head.content_type(), None);
-        assert_eq!(collision_head.content_encoding(), None);
-        assert_eq!(
-            collision_head.metadata().and_then(|metadata| metadata.get("content-type")),
-            Some(&"user-content-type".to_string())
-        );
-        assert_eq!(
-            collision_head
-                .metadata()
-                .and_then(|metadata| metadata.get("content-encoding")),
-            Some(&"user-content-encoding".to_string())
-        );
-
-        env.stop_server();
-    }
-
-    #[tokio::test]
-    #[serial]
-    async fn copy_object_replace_handles_versioned_multipart_source() {
-        init_logging();
-        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
-        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "copy-object-metadata-multipart";
-        let source = "source.bin";
-        let multipart_body = b"multipart historical source";
-        client
-            .create_bucket()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("Failed to create bucket");
-        client
-            .put_bucket_versioning()
-            .bucket(bucket)
-            .versioning_configuration(
-                VersioningConfiguration::builder()
-                    .status(BucketVersioningStatus::Enabled)
-                    .build(),
-            )
-            .send()
-            .await
-            .expect("Failed to enable versioning");
-
-        let upload = client
-            .create_multipart_upload()
-            .bucket(bucket)
-            .key(source)
-            .content_type("application/source")
-            .send()
-            .await
-            .expect("Failed to create multipart upload");
-        let upload_id = upload.upload_id().expect("Multipart upload should return an ID");
-        let part = client
-            .upload_part()
-            .bucket(bucket)
-            .key(source)
-            .upload_id(upload_id)
-            .part_number(1)
-            .body(ByteStream::from_static(multipart_body))
-            .send()
-            .await
-            .expect("Failed to upload multipart part");
-        let completed = client
-            .complete_multipart_upload()
-            .bucket(bucket)
-            .key(source)
-            .upload_id(upload_id)
-            .multipart_upload(
-                CompletedMultipartUpload::builder()
-                    .parts(
-                        CompletedPart::builder()
-                            .part_number(1)
-                            .e_tag(part.e_tag().expect("Uploaded part should return an ETag"))
-                            .build(),
-                    )
-                    .build(),
-            )
-            .send()
-            .await
-            .expect("Failed to complete multipart upload");
-        let historical_version = completed
-            .version_id()
-            .expect("Versioned multipart upload should return a version ID")
-            .to_string();
-
-        client
-            .put_object()
-            .bucket(bucket)
-            .key(source)
-            .body(ByteStream::from_static(b"new current version"))
-            .send()
-            .await
-            .expect("Failed to write current version");
-
-        let copy = client
-            .copy_object()
-            .bucket(bucket)
-            .key("restored.bin")
-            .copy_source(format!("{bucket}/{source}?versionId={historical_version}"))
-            .metadata_directive(MetadataDirective::Replace)
-            .content_type("application/replaced")
-            .send()
-            .await
-            .expect("Failed to copy historical multipart version");
-        assert_eq!(copy.copy_source_version_id(), Some(historical_version.as_str()));
-
-        let restored = client
-            .get_object()
-            .bucket(bucket)
-            .key("restored.bin")
-            .send()
-            .await
-            .expect("Failed to read copied multipart source");
-        assert_eq!(restored.content_type(), Some("application/replaced"));
-        assert_eq!(
-            restored
-                .body
-                .collect()
-                .await
-                .expect("Failed to collect restored body")
-                .into_bytes()
-                .as_ref(),
-            multipart_body
-        );
-
-        env.stop_server();
-    }
-
-    #[tokio::test]
-    #[serial]
-    async fn invalid_replacement_metadata_does_not_mutate_destination() {
-        init_logging();
-        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_REJECT_ARCHIVE_CONTENT_ENCODING", "true")])
-            .await
-            .expect("Failed to start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "copy-object-invalid-metadata";
-        let key = "destination.zip";
-        client
-            .create_bucket()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("Failed to create bucket");
-        client
-            .put_object()
-            .bucket(bucket)
-            .key(key)
-            .content_type("application/zip")
-            .metadata("state", "original")
-            .body(ByteStream::from_static(b"original destination"))
-            .send()
-            .await
-            .expect("Failed to write destination");
-
-        let error = client
-            .copy_object()
-            .bucket(bucket)
-            .key(key)
-            .copy_source(format!("{bucket}/{key}"))
-            .metadata_directive(MetadataDirective::Replace)
-            .content_type("application/zip")
-            .content_encoding("gzip")
-            .send()
-            .await
-            .expect_err("Invalid replacement metadata should be rejected");
-        assert_eq!(error.as_service_error().and_then(|err| err.code()), Some("InvalidArgument"));
-
-        let invalid_directive = client
-            .copy_object()
-            .bucket(bucket)
-            .key(key)
-            .copy_source(format!("{bucket}/{key}"))
-            .customize()
-            .mutate_request(|request| {
-                request.headers_mut().insert("x-amz-metadata-directive", "UNKNOWN");
-            })
-            .send()
-            .await
-            .expect_err("Unknown metadata directives should be rejected");
-        assert_eq!(
-            invalid_directive.as_service_error().and_then(|error| error.code()),
-            Some("InvalidArgument")
-        );
-
-        let ignored_replacement = client
-            .copy_object()
-            .bucket(bucket)
-            .key(key)
-            .copy_source(format!("{bucket}/{key}"))
-            .content_type("application/ignored")
-            .send()
-            .await
-            .expect_err("Replacement fields without REPLACE should be rejected");
-        assert_eq!(
-            ignored_replacement.as_service_error().and_then(|error| error.code()),
-            Some("InvalidRequest")
-        );
-
-        let unchanged = client
-            .get_object()
-            .bucket(bucket)
-            .key(key)
-            .send()
-            .await
-            .expect("Destination should remain readable");
-        assert_eq!(unchanged.content_type(), Some("application/zip"));
-        assert_eq!(
-            unchanged.metadata().and_then(|metadata| metadata.get("state")),
-            Some(&"original".to_string())
-        );
-        assert_eq!(
-            unchanged
-                .body
-                .collect()
-                .await
-                .expect("Failed to collect destination body")
-                .into_bytes()
-                .as_ref(),
-            b"original destination"
-        );
-
-        env.stop_server();
-    }
 }

crates/e2e_test/src/copy_object_tagging_test.rs

@@ -1,468 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! CopyObject tagging directive regression tests.
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
-    use aws_sdk_s3::Client;
-    use aws_sdk_s3::error::ProvideErrorMetadata;
-    use aws_sdk_s3::primitives::ByteStream;
-    use aws_sdk_s3::types::{BucketVersioningStatus, MetadataDirective, TaggingDirective, VersioningConfiguration};
-    use serial_test::serial;
-    use std::collections::BTreeMap;
-
-    async fn object_tags(client: &Client, bucket: &str, key: &str) -> BTreeMap<String, String> {
-        client
-            .get_object_tagging()
-            .bucket(bucket)
-            .key(key)
-            .send()
-            .await
-            .expect("GetObjectTagging should succeed")
-            .tag_set()
-            .iter()
-            .map(|tag| (tag.key().to_string(), tag.value().to_string()))
-            .collect()
-    }
-
-    #[tokio::test]
-    #[serial]
-    async fn copy_object_applies_copy_replace_and_empty_tagging_directives() {
-        init_logging();
-        let mut env = RustFSTestEnvironment::new()
-            .await
-            .expect("test environment should initialize");
-        env.start_rustfs_server(vec![]).await.expect("RustFS should start");
-
-        let client = env.create_s3_client();
-        let bucket = "copy-object-tagging-directive";
-        let source = "source.txt";
-
-        client
-            .create_bucket()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("bucket creation should succeed");
-        client
-            .put_bucket_versioning()
-            .bucket(bucket)
-            .versioning_configuration(
-                VersioningConfiguration::builder()
-                    .status(BucketVersioningStatus::Enabled)
-                    .build(),
-            )
-            .send()
-            .await
-            .expect("versioning should be enabled");
-
-        let first_version = client
-            .put_object()
-            .bucket(bucket)
-            .key(source)
-            .tagging("project=rustfs&stage=first")
-            .body(ByteStream::from_static(b"first"))
-            .send()
-            .await
-            .expect("first source version should be written")
-            .version_id()
-            .expect("versioned PUT should return a version ID")
-            .to_string();
-
-        client
-            .put_object()
-            .bucket(bucket)
-            .key(source)
-            .tagging("project=rustfs&stage=current")
-            .body(ByteStream::from_static(b"current"))
-            .send()
-            .await
-            .expect("current source version should be written");
-
-        client
-            .copy_object()
-            .bucket(bucket)
-            .key("default-copy.txt")
-            .copy_source(format!("{bucket}/{source}"))
-            .send()
-            .await
-            .expect("default CopyObject should preserve current source tags");
-        assert_eq!(
-            object_tags(&client, bucket, "default-copy.txt").await,
-            BTreeMap::from([
-                ("project".to_string(), "rustfs".to_string()),
-                ("stage".to_string(), "current".to_string()),
-            ])
-        );
-
-        client
-            .copy_object()
-            .bucket(bucket)
-            .key("explicit-copy.txt")
-            .copy_source(format!("{bucket}/{source}?versionId={first_version}"))
-            .tagging_directive(TaggingDirective::Copy)
-            .send()
-            .await
-            .expect("COPY should preserve the selected historical version's tags");
-        assert_eq!(
-            object_tags(&client, bucket, "explicit-copy.txt").await,
-            BTreeMap::from([
-                ("project".to_string(), "rustfs".to_string()),
-                ("stage".to_string(), "first".to_string()),
-            ])
-        );
-
-        client
-            .copy_object()
-            .bucket(bucket)
-            .key("replace.txt")
-            .copy_source(format!("{bucket}/{source}"))
-            .tagging_directive(TaggingDirective::Replace)
-            .tagging("project=cli&label=copy%20test")
-            .send()
-            .await
-            .expect("REPLACE should atomically apply requested tags");
-        assert_eq!(
-            object_tags(&client, bucket, "replace.txt").await,
-            BTreeMap::from([
-                ("label".to_string(), "copy test".to_string()),
-                ("project".to_string(), "cli".to_string()),
-            ])
-        );
-        let replace_head = client
-            .head_object()
-            .bucket(bucket)
-            .key("replace.txt")
-            .send()
-            .await
-            .expect("HEAD should succeed after tag replacement");
-        assert_eq!(replace_head.tag_count(), Some(2));
-
-        client
-            .copy_object()
-            .bucket(bucket)
-            .key("empty-replace.txt")
-            .copy_source(format!("{bucket}/{source}"))
-            .tagging_directive(TaggingDirective::Replace)
-            .send()
-            .await
-            .expect("REPLACE without Tagging should clear the destination tag set");
-        assert!(object_tags(&client, bucket, "empty-replace.txt").await.is_empty());
-        let empty_head = client
-            .head_object()
-            .bucket(bucket)
-            .key("empty-replace.txt")
-            .send()
-            .await
-            .expect("HEAD should succeed after empty tag replacement");
-        assert_eq!(empty_head.tag_count(), None);
-
-        client
-            .copy_object()
-            .bucket(bucket)
-            .key("metadata-replace-tag-copy.txt")
-            .copy_source(format!("{bucket}/{source}"))
-            .metadata_directive(MetadataDirective::Replace)
-            .metadata("updated", "true")
-            .send()
-            .await
-            .expect("metadata REPLACE must preserve tags under the default COPY directive");
-        assert_eq!(
-            object_tags(&client, bucket, "metadata-replace-tag-copy.txt").await,
-            BTreeMap::from([
-                ("project".to_string(), "rustfs".to_string()),
-                ("stage".to_string(), "current".to_string()),
-            ])
-        );
-
-        client
-            .copy_object()
-            .bucket(bucket)
-            .key("combined-replace.txt")
-            .copy_source(format!("{bucket}/{source}"))
-            .metadata_directive(MetadataDirective::Replace)
-            .metadata("updated", "true")
-            .tagging_directive(TaggingDirective::Replace)
-            .tagging("project=combined")
-            .send()
-            .await
-            .expect("metadata and tagging REPLACE directives must be independent");
-        assert_eq!(
-            object_tags(&client, bucket, "combined-replace.txt").await,
-            BTreeMap::from([("project".to_string(), "combined".to_string())])
-        );
-
-        client
-            .copy_object()
-            .bucket(bucket)
-            .key(source)
-            .copy_source(format!("{bucket}/{source}"))
-            .tagging_directive(TaggingDirective::Replace)
-            .tagging("project=self-copy")
-            .send()
-            .await
-            .expect("self-copy with tag replacement should update tags atomically");
-        assert_eq!(
-            object_tags(&client, bucket, source).await,
-            BTreeMap::from([("project".to_string(), "self-copy".to_string())])
-        );
-        let self_copy_body = client
-            .get_object()
-            .bucket(bucket)
-            .key(source)
-            .send()
-            .await
-            .expect("self-copy destination should remain readable")
-            .body
-            .collect()
-            .await
-            .expect("self-copy body should be complete")
-            .into_bytes();
-        assert_eq!(self_copy_body.as_ref(), b"current", "tag-only self-copy must preserve the object body");
-
-        client
-            .put_object()
-            .bucket(bucket)
-            .key("malformed.txt")
-            .tagging("state=original")
-            .body(ByteStream::from_static(b"original destination"))
-            .send()
-            .await
-            .expect("preexisting malformed-test destination should be written");
-
-        let malformed = client
-            .copy_object()
-            .bucket(bucket)
-            .key("malformed.txt")
-            .copy_source(format!("{bucket}/{source}"))
-            .tagging_directive(TaggingDirective::Replace)
-            .tagging("project=rustfs%ZZ")
-            .send()
-            .await
-            .expect_err("malformed tags must fail CopyObject");
-        assert_eq!(malformed.as_service_error().and_then(ProvideErrorMetadata::code), Some("InvalidTag"));
-        assert_eq!(
-            object_tags(&client, bucket, "malformed.txt").await,
-            BTreeMap::from([("state".to_string(), "original".to_string())])
-        );
-        let preserved_body = client
-            .get_object()
-            .bucket(bucket)
-            .key("malformed.txt")
-            .send()
-            .await
-            .expect("malformed tags must not replace an existing destination")
-            .body
-            .collect()
-            .await
-            .expect("preserved destination body should be readable")
-            .into_bytes();
-        assert_eq!(
-            preserved_body.as_ref(),
-            b"original destination",
-            "malformed tags must leave destination data unchanged"
-        );
-
-        let discarded = client
-            .copy_object()
-            .bucket(bucket)
-            .key("discarded.txt")
-            .copy_source(format!("{bucket}/{source}"))
-            .tagging("project=must-not-be-discarded")
-            .send()
-            .await
-            .expect_err("Tagging without REPLACE must fail instead of discarding requested tags");
-        assert_eq!(discarded.as_service_error().and_then(ProvideErrorMetadata::code), Some("InvalidRequest"));
-
-        let invalid_directive = client
-            .copy_object()
-            .bucket(bucket)
-            .key("invalid-directive.txt")
-            .copy_source(format!("{bucket}/{source}"))
-            .tagging_directive(TaggingDirective::from("UNKNOWN"))
-            .send()
-            .await
-            .expect_err("an unknown TaggingDirective must fail");
-        assert_eq!(
-            invalid_directive.as_service_error().and_then(ProvideErrorMetadata::code),
-            Some("InvalidArgument")
-        );
-
-        env.stop_server();
-    }
-
-    #[tokio::test]
-    #[serial]
-    async fn copy_object_tag_replacement_honors_request_tag_policy_denial() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
-    {
-        init_logging();
-        let source_bucket = "copy-tags-policy-source";
-        let destination_bucket = "copy-tags-policy-destination";
-
-        let mut env = RustFSTestEnvironment::new().await?;
-        env.start_rustfs_server(vec![]).await?;
-        let admin = env.create_s3_client();
-        admin.create_bucket().bucket(source_bucket).send().await?;
-        admin.create_bucket().bucket(destination_bucket).send().await?;
-        admin
-            .put_object()
-            .bucket(source_bucket)
-            .key("source.txt")
-            .tagging("source=allowed")
-            .body(ByteStream::from_static(b"source"))
-            .send()
-            .await?;
-        admin
-            .put_object()
-            .bucket(source_bucket)
-            .key("conditioned.txt")
-            .body(ByteStream::from_static(b"conditioned source"))
-            .send()
-            .await?;
-
-        let source_policy = serde_json::json!({
-            "Version": "2012-10-17",
-            "Statement": [
-                {
-                    "Effect": "Allow",
-                    "Principal": "*",
-                    "Action": ["s3:GetObject"],
-                    "Resource": [format!("arn:aws:s3:::{source_bucket}/source.txt")]
-                },
-                {
-                    "Effect": "Allow",
-                    "Principal": "*",
-                    "Action": ["s3:GetObject"],
-                    "Resource": [format!("arn:aws:s3:::{source_bucket}/conditioned.txt")],
-                    "Condition": {
-                        "StringEquals": {
-                            "s3:RequestObjectTag/classification": "public"
-                        }
-                    }
-                }
-            ]
-        })
-        .to_string();
-        admin
-            .put_bucket_policy()
-            .bucket(source_bucket)
-            .policy(source_policy)
-            .send()
-            .await?;
-
-        let destination_policy = serde_json::json!({
-            "Version": "2012-10-17",
-            "Statement": [
-                {
-                    "Effect": "Allow",
-                    "Principal": "*",
-                    "Action": ["s3:PutObject"],
-                    "Resource": [format!("arn:aws:s3:::{destination_bucket}/*")]
-                },
-                {
-                    "Effect": "Deny",
-                    "Principal": "*",
-                    "Action": ["s3:PutObject"],
-                    "Resource": [format!("arn:aws:s3:::{destination_bucket}/*")],
-                    "Condition": {
-                        "StringEquals": {
-                            "s3:RequestObjectTag/classification": "restricted"
-                        }
-                    }
-                }
-            ]
-        })
-        .to_string();
-        admin
-            .put_bucket_policy()
-            .bucket(destination_bucket)
-            .policy(destination_policy)
-            .send()
-            .await?;
-
-        let copy_source = format!("/{source_bucket}/source.txt");
-        let allowed = local_http_client()
-            .put(format!("{}/{destination_bucket}/allowed.txt", env.url))
-            .header("x-amz-copy-source", &copy_source)
-            .header("x-amz-tagging-directive", "REPLACE")
-            .header("x-amz-tagging", "classification=public")
-            .send()
-            .await?;
-        assert_eq!(
-            allowed.status(),
-            reqwest::StatusCode::OK,
-            "a tag set allowed by the request-tag policy should copy successfully"
-        );
-        assert_eq!(
-            object_tags(&admin, destination_bucket, "allowed.txt").await,
-            BTreeMap::from([("classification".to_string(), "public".to_string())])
-        );
-
-        let denied = local_http_client()
-            .put(format!("{}/{destination_bucket}/denied.txt", env.url))
-            .header("x-amz-copy-source", copy_source)
-            .header("x-amz-tagging-directive", "REPLACE")
-            .header("x-amz-tagging", "classification=restricted")
-            .send()
-            .await?;
-        assert_eq!(
-            denied.status(),
-            reqwest::StatusCode::FORBIDDEN,
-            "CopyObject must honor a request-tag policy Deny"
-        );
-
-        let source_condition_bypass = local_http_client()
-            .put(format!("{}/{destination_bucket}/source-condition.txt", env.url))
-            .header("x-amz-copy-source", format!("/{source_bucket}/conditioned.txt"))
-            .header("x-amz-tagging-directive", "REPLACE")
-            .header("x-amz-tagging", "classification=public")
-            .send()
-            .await?;
-        assert_eq!(
-            source_condition_bypass.status(),
-            reqwest::StatusCode::FORBIDDEN,
-            "destination request tags must not satisfy source GetObject policy conditions"
-        );
-
-        let missing_destination = admin
-            .head_object()
-            .bucket(destination_bucket)
-            .key("denied.txt")
-            .send()
-            .await
-            .expect_err("an access-denied copy must not create a destination object");
-        assert_eq!(
-            missing_destination.as_service_error().and_then(ProvideErrorMetadata::code),
-            Some("NotFound")
-        );
-        let missing_bypass_destination = admin
-            .head_object()
-            .bucket(destination_bucket)
-            .key("source-condition.txt")
-            .send()
-            .await
-            .expect_err("a source authorization denial must not create a destination object");
-        assert_eq!(
-            missing_bypass_destination
-                .as_service_error()
-                .and_then(ProvideErrorMetadata::code),
-            Some("NotFound")
-        );
-
-        env.stop_server();
-        Ok(())
-    }
-}

crates/e2e_test/src/copy_object_version_restore_test.rs

@@ -160,146 +160,4 @@ mod tests {
 
         env.stop_server();
     }
-
-    /// Regression test for Issue #4976: a versioned-source CopyObject must echo the exact source
-    /// version copied via `x-amz-copy-source-version-id` (SDK `CopySourceVersionId`), kept distinct
-    /// from the newly created destination `x-amz-version-id`.
-    #[tokio::test]
-    #[serial]
-    async fn test_copy_of_non_latest_source_version_returns_copy_source_version_id() {
-        init_logging();
-        info!("Issue #4976: versioned CopyObject must return x-amz-copy-source-version-id for the exact source version");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
-        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
-
-        let client = env.create_s3_client();
-        let src_bucket = "copy-source-version-header-src";
-        let dst_bucket = "copy-source-version-header-dst";
-        let src_key = "reports/quarantined.bin";
-        let dst_key = "reports/promoted.bin";
-
-        for bucket in [src_bucket, dst_bucket] {
-            client
-                .create_bucket()
-                .bucket(bucket)
-                .send()
-                .await
-                .expect("Failed to create bucket");
-            client
-                .put_bucket_versioning()
-                .bucket(bucket)
-                .versioning_configuration(
-                    VersioningConfiguration::builder()
-                        .status(BucketVersioningStatus::Enabled)
-                        .build(),
-                )
-                .send()
-                .await
-                .expect("Failed to enable versioning");
-        }
-
-        // Source version 1: the exact (non-latest) version we will copy.
-        let v1_content = b"quarantined payload -- source version one (target of the copy)";
-        let put_v1 = client
-            .put_object()
-            .bucket(src_bucket)
-            .key(src_key)
-            .body(ByteStream::from_static(v1_content))
-            .send()
-            .await
-            .expect("PUT source v1 failed");
-        let v1_id = put_v1.version_id().expect("source v1 must have a version id").to_string();
-
-        // Source version 2: becomes the latest, so v1 is deliberately NOT the current version.
-        let v2_content = b"quarantined payload -- source version two (now current, must be ignored)";
-        let put_v2 = client
-            .put_object()
-            .bucket(src_bucket)
-            .key(src_key)
-            .body(ByteStream::from_static(v2_content))
-            .send()
-            .await
-            .expect("PUT source v2 failed");
-        let v2_id = put_v2.version_id().expect("source v2 must have a version id").to_string();
-        assert_ne!(v1_id, v2_id, "the two source puts must produce distinct versions");
-
-        // Copy the exact NON-LATEST source version into the destination bucket.
-        let copy_out = client
-            .copy_object()
-            .bucket(dst_bucket)
-            .key(dst_key)
-            .copy_source(format!("{src_bucket}/{src_key}?versionId={v1_id}"))
-            .send()
-            .await
-            .expect("CopyObject of a versioned source must succeed");
-
-        // (3) The response must echo the exact source version copied.
-        let copy_source_version_id = copy_out
-            .copy_source_version_id()
-            .expect("issue #4976: response must include x-amz-copy-source-version-id for a versioned source");
-        assert_eq!(
-            copy_source_version_id, v1_id,
-            "x-amz-copy-source-version-id must equal the exact source version requested, not the latest source version"
-        );
-        assert_ne!(
-            copy_source_version_id, v2_id,
-            "x-amz-copy-source-version-id must not be the latest source version"
-        );
-
-        // (4) The destination header must identify a distinct, newly created version.
-        let dst_version_id = copy_out
-            .version_id()
-            .expect("destination copy must create a new version id")
-            .to_string();
-        assert!(!dst_version_id.is_empty(), "destination version id must be present");
-        assert_ne!(dst_version_id, v1_id, "destination version must be distinct from the source version");
-        assert_ne!(
-            dst_version_id, v2_id,
-            "destination version must be distinct from the source latest version"
-        );
-
-        // (5) The destination must hold the exact bytes/size of the copied (v1) source version.
-        let head = client
-            .head_object()
-            .bucket(dst_bucket)
-            .key(dst_key)
-            .send()
-            .await
-            .expect("HEAD destination failed");
-        assert_eq!(
-            head.content_length(),
-            Some(v1_content.len() as i64),
-            "destination size must equal the copied source version (v1)"
-        );
-
-        let get_dst = client
-            .get_object()
-            .bucket(dst_bucket)
-            .key(dst_key)
-            .version_id(&dst_version_id)
-            .send()
-            .await
-            .expect("GET destination failed");
-        let dst_body = get_dst.body.collect().await.expect("collect destination body").into_bytes();
-        assert_eq!(
-            dst_body.as_ref(),
-            v1_content,
-            "destination bytes must exactly equal the copied source version (v1), not the latest (v2)"
-        );
-
-        // (6) The source version copied from must remain present and independently readable.
-        let get_src_v1 = client
-            .get_object()
-            .bucket(src_bucket)
-            .key(src_key)
-            .version_id(&v1_id)
-            .send()
-            .await
-            .expect("GET source v1 failed after copy");
-        let src_v1_body = get_src_v1.body.collect().await.expect("collect source v1 body").into_bytes();
-        assert_eq!(src_v1_body.as_ref(), v1_content, "source v1 must remain intact after the copy");
-
-        env.stop_server();
-    }
 }

crates/e2e_test/src/fake_s3_target/README.md

@@ -1,11 +0,0 @@
-# Programmable fake S3 target
-
-This module is the shared failure-injection boundary for replication end-to-end tests. It runs an in-process, path-style S3 endpoint backed by `s3s`; no production crate depends on it.
-
-`FakeS3Target::start()` creates the listener. Add target buckets with `create_bucket`, point a RustFS remote target at `address()`, use `FAKE_ACCESS_KEY` / `FAKE_SECRET_KEY`, then enqueue per-operation faults with `inject`. Faults for one operation are consumed in FIFO order and do not consume faults queued for another operation. A fault is consumed only after `s3s` verifies the full request signature, so anonymous, other-access-key, and bad-signature traffic cannot disturb a script.
-
-Supported data operations are HeadBucket, GetBucketVersioning, PUT/GET/HEAD/DELETE Object, and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB.
-
-Fault actions cover HTTP 401/403/503 responses, pre-dispatch delay, connection abort when a logical request-body threshold is reached, streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions.
-
-The listener is loopback-only. It admits at most 64 active connections and two concurrently buffered request bodies; authenticated multipart-complete XML collection and assembly take both body permits. Keep-alive is disabled, request-header reads are bounded to 30 seconds, a parsed request is bounded to 65 seconds, and the complete connection lifetime is bounded to 100 seconds. It retains at most 256 buckets, 4,096 journal entries, 4,096 scripted faults, 4,096 object versions, 256 multipart uploads, and 10,000 multipart parts. Retained identifiers are capped at 1 KiB, user metadata at 2 KiB, and content type at 1 KiB. A PUT or uploaded part is capped at 64 MiB; a completed multipart object and all stored object/part data are capped at 128 MiB. Body drain, body-permit waits, delay, and slow-drain execution are bounded to 30 seconds; each slow-drain slice delay must be below that bound.

crates/e2e_test/src/fault_proxy.rs

@@ -1,498 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! In-process, socket-level network fault-injection proxy for black-box
-//! cluster E2E tests (backlog#1325 network fault-injection block).
-//!
-//! `FaultProxy` binds a TCP listener on a random local port, forwards every
-//! accepted connection to a fixed target address, and lets a test flip the
-//! forwarding behaviour at runtime. It is the black-box counterpart to the
-//! in-process white-box hooks (rename barrier, disk call counters): those
-//! `#[cfg(test)]` primitives cannot reach across the process boundary into a
-//! spawned RustFS server, whereas this proxy sits on the wire between two
-//! nodes and needs no cooperation from the peer.
-//!
-//! Service targets:
-//! - **#1312 / #1319** lock-plane one-way partition and accept-then-blackhole
-//!   peer: run a node's lock/RPC port through the proxy, then
-//!   [`FaultMode::Partition`] one direction (data plane stays reachable via a
-//!   separate direct port) or [`FaultMode::Blackhole`] the whole connection.
-//! - **#1327** cross-process replay / tamper harness: the proxy is the wire
-//!   seam a replay/tamper filter can later hook into.
-//!
-//! Supported fault modes:
-//! - [`FaultMode::Pass`]: transparent byte forwarding (round-trip identical).
-//! - [`FaultMode::Latency`]: delay every forwarded chunk by a fixed duration.
-//! - [`FaultMode::Blackhole`]: accept the connection but forward nothing in
-//!   either direction and never respond (models an accepted-then-dead peer);
-//!   the client observes a read timeout, the proxy never panics.
-//! - [`FaultMode::Partition`]: block exactly one direction while the other
-//!   keeps flowing (models a lock-RPC one-way partition with a live data
-//!   plane).
-//!
-//! Wiring this into the cluster harness ("expose node port N through a proxy")
-//! is intentionally left as a follow-up: the harness multi-drive / 2-pool
-//! changes land separately (rustfs#4937), so this block ships the proxy tool
-//! plus its local self-tests against a loopback echo server and stays
-//! independent of that harness work.
-//!
-//! # Example
-//!
-//! ```ignore
-//! let proxy = FaultProxy::start(target_addr).await?;
-//! let via = proxy.local_addr(); // point the client here instead of `target_addr`
-//! proxy.set_mode(FaultMode::Latency(Duration::from_millis(50)));
-//! // ... exercise the client ...
-//! proxy.set_mode(FaultMode::Partition(Direction::ClientToServer));
-//! // ... assert the blocked direction is dead ...
-//! proxy.shutdown().await; // releases the listener port
-//! ```
-
-use std::io;
-use std::net::{Ipv4Addr, SocketAddr};
-use std::time::Duration;
-
-use tokio::io::{AsyncReadExt, AsyncWriteExt};
-use tokio::net::{TcpListener, TcpStream};
-use tokio::sync::watch;
-use tokio::task::JoinHandle;
-
-/// A single logical direction of a proxied connection.
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub enum Direction {
-    /// Bytes travelling from the connecting client towards the target server
-    /// (the forward / request path).
-    ClientToServer,
-    /// Bytes travelling from the target server back to the client (the return
-    /// / response path).
-    ServerToClient,
-}
-
-/// Runtime-switchable forwarding behaviour of a [`FaultProxy`].
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub enum FaultMode {
-    /// Forward bytes transparently in both directions.
-    Pass,
-    /// Forward bytes in both directions, but delay every chunk by this
-    /// duration before it is written onward.
-    Latency(Duration),
-    /// Accept connections but forward nothing in either direction and never
-    /// respond. Bytes read from either side are drained and discarded so the
-    /// proxy never blocks or panics; the peer simply never hears back.
-    Blackhole,
-    /// Block exactly one direction (drop its bytes) while the other direction
-    /// keeps forwarding normally.
-    Partition(Direction),
-}
-
-/// An async TCP proxy that forwards a random local port to a fixed target and
-/// can inject network faults at runtime.
-///
-/// The proxy owns a background accept loop; each accepted connection is
-/// handled by its own task pair (one per direction). [`FaultProxy::shutdown`]
-/// stops the accept loop and drops the listener, releasing the bound port.
-pub struct FaultProxy {
-    listen_addr: SocketAddr,
-    target: SocketAddr,
-    mode_tx: watch::Sender<FaultMode>,
-    shutdown_tx: watch::Sender<bool>,
-    accept_task: JoinHandle<()>,
-}
-
-impl FaultProxy {
-    /// Bind a listener on `127.0.0.1:0` and start forwarding accepted
-    /// connections to `target`. Starts in [`FaultMode::Pass`].
-    pub async fn start(target: SocketAddr) -> io::Result<Self> {
-        let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
-        let listen_addr = listener.local_addr()?;
-
-        let (mode_tx, mode_rx) = watch::channel(FaultMode::Pass);
-        let (shutdown_tx, shutdown_rx) = watch::channel(false);
-
-        let accept_task = tokio::spawn(accept_loop(listener, target, mode_rx, shutdown_rx));
-
-        Ok(Self {
-            listen_addr,
-            target,
-            mode_tx,
-            shutdown_tx,
-            accept_task,
-        })
-    }
-
-    /// Local address the proxy is listening on. Point clients here instead of
-    /// the real target to route their traffic through the proxy.
-    pub fn local_addr(&self) -> SocketAddr {
-        self.listen_addr
-    }
-
-    /// The fixed target address every connection is forwarded to.
-    pub fn target_addr(&self) -> SocketAddr {
-        self.target
-    }
-
-    /// The mode currently in effect.
-    pub fn mode(&self) -> FaultMode {
-        *self.mode_tx.borrow()
-    }
-
-    /// Switch the fault mode at runtime. Takes effect on the next chunk read by
-    /// each direction of every live and future connection.
-    pub fn set_mode(&self, mode: FaultMode) {
-        // A send only fails if every receiver has dropped, i.e. the accept loop
-        // and all connections have already ended; there is nothing to steer.
-        let _ = self.mode_tx.send(mode);
-    }
-
-    /// Stop the accept loop, signal live connections to wind down, and drop the
-    /// listener so the bound port is released. Awaits the accept loop so the
-    /// port is free once this returns.
-    pub async fn shutdown(self) {
-        let _ = self.shutdown_tx.send(true);
-        let _ = self.accept_task.await;
-    }
-}
-
-/// Accept loop: takes new connections until shutdown is signalled, then returns
-/// (dropping `listener`, which releases the port).
-async fn accept_loop(
-    listener: TcpListener,
-    target: SocketAddr,
-    mode_rx: watch::Receiver<FaultMode>,
-    mut shutdown_rx: watch::Receiver<bool>,
-) {
-    loop {
-        tokio::select! {
-            biased;
-            _ = shutdown_rx.changed() => {
-                if *shutdown_rx.borrow() {
-                    break;
-                }
-            }
-            accepted = listener.accept() => {
-                match accepted {
-                    Ok((client, _peer)) => {
-                        tokio::spawn(handle_connection(
-                            client,
-                            target,
-                            mode_rx.clone(),
-                            shutdown_rx.clone(),
-                        ));
-                    }
-                    // A transient accept error should not tear the proxy down.
-                    Err(_) => continue,
-                }
-            }
-        }
-    }
-}
-
-/// Bridge one accepted client connection to a fresh connection to the target,
-/// running an independent pump per direction.
-async fn handle_connection(
-    client: TcpStream,
-    target: SocketAddr,
-    mode_rx: watch::Receiver<FaultMode>,
-    shutdown_rx: watch::Receiver<bool>,
-) {
-    let server = match TcpStream::connect(target).await {
-        Ok(s) => s,
-        // Target unreachable: nothing to bridge; drop the client.
-        Err(_) => return,
-    };
-
-    let (client_rd, client_wr) = client.into_split();
-    let (server_rd, server_wr) = server.into_split();
-
-    let up = tokio::spawn(pump(
-        Direction::ClientToServer,
-        client_rd,
-        server_wr,
-        mode_rx.clone(),
-        shutdown_rx.clone(),
-    ));
-    let down = tokio::spawn(pump(Direction::ServerToClient, server_rd, client_wr, mode_rx, shutdown_rx));
-
-    let _ = up.await;
-    let _ = down.await;
-}
-
-/// Copy bytes from `from` to `to` for a single direction, applying the current
-/// [`FaultMode`]. Returns when the source hits EOF/error, a write fails, or
-/// shutdown is signalled.
-async fn pump<R, W>(
-    dir: Direction,
-    mut from: R,
-    mut to: W,
-    mut mode_rx: watch::Receiver<FaultMode>,
-    mut shutdown_rx: watch::Receiver<bool>,
-) where
-    R: AsyncReadExt + Unpin,
-    W: AsyncWriteExt + Unpin,
-{
-    let mut buf = vec![0u8; 16 * 1024];
-    loop {
-        let n = tokio::select! {
-            biased;
-            _ = shutdown_rx.changed() => break,
-            read = from.read(&mut buf) => match read {
-                Ok(0) => break,        // clean EOF
-                Ok(n) => n,
-                Err(_) => break,       // reset / error: end this direction
-            },
-        };
-
-        // Snapshot the mode without holding the borrow across an await.
-        let mode = *mode_rx.borrow_and_update();
-        match mode {
-            // Whole connection is a black hole: drain and drop.
-            FaultMode::Blackhole => continue,
-            // This direction is partitioned off: drop its bytes; the peer keeps
-            // reading nothing while the other direction may still flow.
-            FaultMode::Partition(blocked) if blocked == dir => continue,
-            FaultMode::Latency(delay) => {
-                tokio::time::sleep(delay).await;
-                if to.write_all(&buf[..n]).await.is_err() {
-                    break;
-                }
-                if to.flush().await.is_err() {
-                    break;
-                }
-            }
-            FaultMode::Pass | FaultMode::Partition(_) => {
-                if to.write_all(&buf[..n]).await.is_err() {
-                    break;
-                }
-                if to.flush().await.is_err() {
-                    break;
-                }
-            }
-        }
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use std::sync::Arc;
-    use std::time::Instant;
-    use tokio::sync::Mutex;
-
-    /// A loopback echo server that records every byte it receives, so tests can
-    /// distinguish "the server never got it" (client->server blocked) from "the
-    /// server got it but the reply never came back" (server->client blocked).
-    struct EchoServer {
-        addr: SocketAddr,
-        received: Arc<Mutex<Vec<u8>>>,
-        _task: JoinHandle<()>,
-    }
-
-    impl EchoServer {
-        async fn start() -> io::Result<Self> {
-            let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
-            let addr = listener.local_addr()?;
-            let received = Arc::new(Mutex::new(Vec::new()));
-            let received_task = received.clone();
-            let task = tokio::spawn(async move {
-                loop {
-                    let Ok((mut sock, _)) = listener.accept().await else {
-                        break;
-                    };
-                    let received_conn = received_task.clone();
-                    tokio::spawn(async move {
-                        let mut buf = vec![0u8; 16 * 1024];
-                        loop {
-                            match sock.read(&mut buf).await {
-                                Ok(0) | Err(_) => break,
-                                Ok(n) => {
-                                    received_conn.lock().await.extend_from_slice(&buf[..n]);
-                                    if sock.write_all(&buf[..n]).await.is_err() {
-                                        break;
-                                    }
-                                    let _ = sock.flush().await;
-                                }
-                            }
-                        }
-                    });
-                }
-            });
-            Ok(Self {
-                addr,
-                received,
-                _task: task,
-            })
-        }
-
-        async fn received(&self) -> Vec<u8> {
-            self.received.lock().await.clone()
-        }
-    }
-
-    /// Write `payload`, then try to read up to `want` bytes with a deadline.
-    /// Returns the bytes actually received before the timeout (possibly empty).
-    async fn write_then_read(stream: &mut TcpStream, payload: &[u8], want: usize, timeout: Duration) -> Vec<u8> {
-        stream.write_all(payload).await.expect("client write");
-        stream.flush().await.expect("client flush");
-
-        let mut out = Vec::new();
-        let mut buf = vec![0u8; want.max(1)];
-        let deadline = tokio::time::Instant::now() + timeout;
-        while out.len() < want {
-            let remaining = deadline.saturating_duration_since(tokio::time::Instant::now());
-            if remaining.is_zero() {
-                break;
-            }
-            match tokio::time::timeout(remaining, stream.read(&mut buf)).await {
-                Ok(Ok(0)) => break, // peer closed
-                Ok(Ok(n)) => out.extend_from_slice(&buf[..n]),
-                Ok(Err(_)) => break, // connection error
-                Err(_) => break,     // read timed out
-            }
-        }
-        out
-    }
-
-    #[tokio::test]
-    async fn pass_mode_round_trips_bytes() {
-        let echo = EchoServer::start().await.unwrap();
-        let proxy = FaultProxy::start(echo.addr).await.unwrap();
-        assert_eq!(proxy.mode(), FaultMode::Pass);
-
-        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
-        let payload = b"hello-fault-proxy";
-        let got = write_then_read(&mut client, payload, payload.len(), Duration::from_secs(5)).await;
-
-        assert_eq!(got, payload, "pass mode must forward bytes unchanged");
-        assert_eq!(echo.received().await, payload, "server must have received the bytes");
-
-        proxy.shutdown().await;
-    }
-
-    #[tokio::test]
-    async fn latency_mode_delays_forwarding() {
-        // Loose lower bound only: sleep() guarantees *at least* the delay, so
-        // the assertion cannot flake on a slow machine, and we never assert an
-        // upper bound.
-        let delay = Duration::from_millis(200);
-        let echo = EchoServer::start().await.unwrap();
-        let proxy = FaultProxy::start(echo.addr).await.unwrap();
-        proxy.set_mode(FaultMode::Latency(delay));
-
-        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
-        let payload = b"delayed";
-        let start = Instant::now();
-        let got = write_then_read(&mut client, payload, payload.len(), Duration::from_secs(5)).await;
-        let elapsed = start.elapsed();
-
-        assert_eq!(got, payload, "latency mode must still deliver the bytes");
-        // One delay on the request path plus one on the echo return path: the
-        // round trip must exceed at least a single configured delay.
-        assert!(elapsed >= delay, "round trip {elapsed:?} shorter than configured delay {delay:?}");
-
-        proxy.shutdown().await;
-    }
-
-    #[tokio::test]
-    async fn blackhole_mode_yields_no_response_and_no_panic() {
-        let echo = EchoServer::start().await.unwrap();
-        let proxy = FaultProxy::start(echo.addr).await.unwrap();
-        proxy.set_mode(FaultMode::Blackhole);
-
-        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
-        let got = write_then_read(&mut client, b"into-the-void", 1, Duration::from_millis(300)).await;
-
-        assert!(got.is_empty(), "blackhole mode must not return any bytes, got {got:?}");
-        assert!(echo.received().await.is_empty(), "blackhole must not forward to the server");
-
-        // Proxy is still alive and steerable after a blackholed connection.
-        proxy.set_mode(FaultMode::Pass);
-        proxy.shutdown().await;
-    }
-
-    #[tokio::test]
-    async fn partition_client_to_server_blocks_request_path() {
-        let echo = EchoServer::start().await.unwrap();
-        let proxy = FaultProxy::start(echo.addr).await.unwrap();
-        proxy.set_mode(FaultMode::Partition(Direction::ClientToServer));
-
-        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
-        let got = write_then_read(&mut client, b"blocked-request", 1, Duration::from_millis(300)).await;
-
-        assert!(got.is_empty(), "client->server partition must yield no echo");
-        assert!(
-            echo.received().await.is_empty(),
-            "client->server partition must stop bytes from reaching the server"
-        );
-
-        proxy.shutdown().await;
-    }
-
-    #[tokio::test]
-    async fn partition_server_to_client_blocks_only_return_path() {
-        let echo = EchoServer::start().await.unwrap();
-        let proxy = FaultProxy::start(echo.addr).await.unwrap();
-        proxy.set_mode(FaultMode::Partition(Direction::ServerToClient));
-
-        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
-        let payload = b"one-way";
-        let got = write_then_read(&mut client, payload, 1, Duration::from_millis(300)).await;
-
-        // Client hears nothing back...
-        assert!(got.is_empty(), "server->client partition must block the reply");
-        // ...but the request path is live, so the server did receive the bytes.
-        // Poll briefly to avoid racing the forward direction.
-        let mut server_saw = Vec::new();
-        for _ in 0..30 {
-            server_saw = echo.received().await;
-            if !server_saw.is_empty() {
-                break;
-            }
-            tokio::time::sleep(Duration::from_millis(10)).await;
-        }
-        assert_eq!(server_saw, payload, "server->client partition must leave the request path intact");
-
-        proxy.shutdown().await;
-    }
-
-    #[tokio::test]
-    async fn set_mode_switches_behaviour_at_runtime() {
-        let echo = EchoServer::start().await.unwrap();
-        let proxy = FaultProxy::start(echo.addr).await.unwrap();
-
-        // Start passing: first exchange round-trips.
-        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
-        let first = write_then_read(&mut client, b"first", 5, Duration::from_secs(5)).await;
-        assert_eq!(first, b"first", "initial pass exchange must round-trip");
-
-        // Flip to blackhole on the same live connection: the next write gets no reply.
-        proxy.set_mode(FaultMode::Blackhole);
-        let second = write_then_read(&mut client, b"second", 1, Duration::from_millis(300)).await;
-        assert!(second.is_empty(), "after switching to blackhole the live connection must go silent");
-
-        proxy.shutdown().await;
-    }
-
-    #[tokio::test]
-    async fn shutdown_releases_the_listener_port() {
-        let echo = EchoServer::start().await.unwrap();
-        let proxy = FaultProxy::start(echo.addr).await.unwrap();
-        let addr = proxy.local_addr();
-
-        proxy.shutdown().await;
-
-        // The port must be free to bind again once shutdown returns.
-        let rebind = TcpListener::bind(addr).await;
-        assert!(rebind.is_ok(), "shutdown must release the listener port {addr}");
-    }
-}

crates/e2e_test/src/get_codec_streaming_compat_test.rs

@@ -45,10 +45,10 @@
 //!   * Parity reconstruction: one data disk is taken offline
 //!     (`take_disk_offline`) and the SAME object matrix is GET both ways while
 //!     the EC 2+2 set rebuilds each large object from the surviving shards. The
-//!     eager first/single-part setup may keep its conservative whole-request
-//!     fallback when shard placement makes codec streaming unsafe, so this phase
-//!     asserts byte- and header-equality vs the legacy path rather than requiring
-//!     zero duplex fallbacks under degraded drive health.
+//!     codec-streaming reader gate never inspects drive health, so the codec
+//!     fast path is exercised end-to-end through reconstruction; the test
+//!     asserts byte- and header-equality vs the legacy path AND that the codec
+//!     phase never fell back to a duplex pipe while reconstructing.
 //!   * Missing object: a GET for an absent key is compared across both phases
 //!     to prove the error semantics (HTTP status + S3 error code) are identical
 //!     — the codec env must not perturb the NoSuchKey negative path.
@@ -475,14 +475,15 @@ mod tests {
             "ranged GET length diverged with codec streaming enabled"
         );
 
-        // ---- Phase B degraded: the same reconstruction, with codec gates open ----
-        // Re-run the reconstruction A/B with codec-streaming enabled. If eager
-        // first/single-part setup cannot prove the codec path is safe for the
-        // surviving shards, the implementation intentionally preserves the
-        // whole-request legacy fallback; later multipart parts can degrade in
-        // place. This phase verifies parity-reconstructed bytes and headers,
-        // while the healthy phase above remains the strict zero-duplex path
-        // confirmation.
+        // ---- Phase B degraded: the same reconstruction, now on the codec path ----
+        // Re-run the reconstruction A/B with the codec-streaming gates still
+        // open. The reader gate decision is independent of drive health (it
+        // never inspects disk state), so the codec fast path is exercised
+        // end-to-end while the EC set rebuilds each large object from the
+        // surviving shards — this is a real codec-vs-legacy reconstruction test,
+        // not legacy-vs-legacy. Snapshot the duplex count first (the range GET
+        // above already used the duplex path) so we can measure only the markers
+        // these degraded codec GETs add.
         let dup_codec_before_degraded = count_marker(&codec_log, DUPLEX_MARKER);
         harness.take_disk_offline(0)?;
         let mut codec_degraded: BTreeMap<String, GetView> = BTreeMap::new();
@@ -510,11 +511,16 @@ mod tests {
             );
         }
 
-        // Keep degraded duplex markers as diagnostic evidence only: eager setup
-        // may fall back before streaming when shard safety cannot be proven.
+        // Path confirmation under reconstruction: the codec fast path must have
+        // served the reconstructed large objects without ever falling back to
+        // the legacy duplex pipe. Without this, the equivalence above could be
+        // legacy-vs-legacy and prove nothing about codec reconstruction.
         sleep(Duration::from_millis(300)).await;
         let dup_codec_degraded = count_marker(&codec_log, DUPLEX_MARKER).saturating_sub(dup_codec_before_degraded);
-        info!(dup_codec_degraded, "codec phase degraded-read legacy duplex marker count");
+        assert_eq!(
+            dup_codec_degraded, 0,
+            "codec phase created {dup_codec_degraded} duplex pipe(s) while reconstructing large objects with disk0 offline; the codec fast path was not exercised under degraded reads (see {codec_log})"
+        );
 
         info!(
             objects = baseline.len(),

crates/e2e_test/src/internode_rpc_signature_e2e_test.rs

@@ -1,573 +0,0 @@
-#![cfg(test)]
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Cross-process replay / tamper acceptance for the internode NodeService v2 RPC
-//! signature (<https://github.com/rustfs/backlog/issues/1327>).
-//!
-//! # Why this exists on top of the in-process tests
-//!
-//! `http_auth.rs` unit-tests the signature algebra by calling the verifier
-//! directly. That proves the crypto, but it cannot prove that a *deployed*
-//! server actually reaches it: the request has to survive the hybrid HTTP/gRPC
-//! router, `check_auth`, tonic's own metadata handling, and finally the
-//! per-handler body-digest gate. A handler that forgets its
-//! `verify_disk_mutation_digest` call, or a router change that bypasses
-//! `check_auth`, is invisible in-process and wide open in production. These
-//! tests drive a real `rustfs` child process over a real TCP socket, so every
-//! one of those layers is in the path.
-//!
-//! # Attacker model
-//!
-//! The adversary is on-path: it observed one legitimately signed request and
-//! can resend, retarget, or edit those bytes — including individual headers.
-//! It does **not** hold the RPC secret. The test process does hold the secret,
-//! but uses it for exactly one purpose: minting the request that stands in for
-//! the captured one. Every attack then only *reuses or edits* an already-minted
-//! header set; no attack step ever re-signs. If any of these tests could pass
-//! by re-signing, it would be testing nothing.
-//!
-//! # Isolating one variable at a time
-//!
-//! Each rejection is paired with an acceptance that differs in exactly one
-//! respect, because a misconfigured harness (wrong audience, dead server,
-//! ambient strict env) would otherwise make every "rejected" assertion pass
-//! vacuously. Two pairings carry most of the weight:
-//!
-//! - Editing the body alone is caught by the *handler* (`PermissionDenied`);
-//!   editing the body **and** repairing the digest header to match is caught by
-//!   the *signature* (`Unauthenticated`). The second only fails closed if the
-//!   digest is genuinely inside the signed scope, so the pair pins both layers.
-//! - Replaying a captured nonce is caught by the replay cache; swapping in a
-//!   fresh nonce is caught by the signature. Again, only the pair proves the
-//!   nonce is signed rather than merely cached.
-//!
-//! # Why `MakeVolume` against a non-existent disk
-//!
-//! Every covered handler checks the digest before touching storage, and
-//! `MakeVolume` resolves its disk *after* that check. Aiming at a disk that
-//! cannot exist gives three cleanly separable outcomes with zero side effects
-//! on the server's real data:
-//!
-//! - `Err(Unauthenticated)` — rejected by `check_auth` (signature layer).
-//! - `Err(PermissionDenied)` — rejected by the handler's body-digest gate.
-//! - `Ok(success: false)` — **authentication passed**; the request reached
-//!   handler logic and only then failed on the bogus disk.
-//!
-//! # Coverage of the issue's acceptance matrix
-//!
-//! | Acceptance item | Test |
-//! |---|---|
-//! | replay a signature onto another method → reject | [`cross_method_signature_transplant_is_rejected`] |
-//! | replay same method + body after nonce consumed → reject | [`nonce_replay_of_a_captured_mutation_is_rejected`] |
-//! | nonce is signed, not just cached → reject a swapped nonce | [`swapping_in_a_fresh_nonce_is_rejected`] |
-//! | tamper one byte of the body → reject | [`tampered_mutation_body_is_rejected`] |
-//! | body digest is inside the signed scope → reject a repaired digest | [`rewriting_the_digest_to_match_a_tampered_body_is_rejected`] |
-//! | wrong destination node identity → reject | [`signature_minted_for_another_node_is_rejected`] |
-//! | mixed version: legacy-only still served, not blocked | [`legacy_only_signature_is_accepted_in_default_posture`] |
-//! | strict flip closes the signature downgrade | [`signature_strict_rejects_legacy_only_downgrade`] |
-//! | strict flip closes the body-digest downgrade, incl. v1 | [`body_digest_strict_rejects_digestless_mutation`] |
-//!
-//! Two acceptance items are deliberately left to the in-process tests. A stale
-//! timestamp cannot be forged from outside — it is inside the HMAC — so
-//! observing it would mean idling out the full freshness window. And the
-//! `signature_v1_fallback_total` / `body_digest_fallback_total` counter deltas
-//! that gate the strict flips are asserted directly in `http_auth.rs`; the
-//! legacy test below proves only the *accepted* half of that behaviour.
-
-use crate::common::{RustFSTestEnvironment, init_logging};
-use crate::storage_api::internode_rpc_signature::{
-    TONIC_RPC_PREFIX, gen_signature_headers, gen_tonic_signature_headers, node_service_time_out_client_no_auth,
-};
-use http::{HeaderMap, Method};
-use rustfs_config::{
-    ENV_INTERNODE_RPC_BODY_DIGEST_STRICT, ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY, ENV_INTERNODE_RPC_SIGNATURE_STRICT,
-};
-use rustfs_protos::canonical_make_volume_request_body;
-use rustfs_protos::proto_gen::node_service::{MakeVolumeRequest, MakeVolumeResponse};
-use serial_test::serial;
-use sha2::{Digest, Sha256};
-use std::error::Error;
-use tonic::{Code, Request, Status};
-use uuid::Uuid;
-
-type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-/// Shared internode secret handed to both the child server and this process.
-///
-/// Must not be the default credential: `resolve_rpc_secret` fails closed on
-/// defaults (GHSA-r5qv), so a default here would break every request rather
-/// than test anything.
-const TEST_RPC_SECRET: &str = "rustfs-internode-signature-e2e-secret";
-
-/// A disk path the server cannot possibly have configured, so a request that
-/// clears authentication stops harmlessly at `find_disk`.
-const ABSENT_DISK: &str = "/nonexistent/rustfs-signature-e2e-disk";
-
-/// Wire names of the two v2 headers these tests edit. They are `pub(crate)` in
-/// ecstore, so they are repeated here rather than imported — [`overwrite_header`]
-/// asserts the header it replaces was actually present, which turns a rename
-/// into a loud failure instead of silently reducing an attack to a no-op.
-const CONTENT_SHA256_HEADER: &str = "x-rustfs-content-sha256";
-const NONCE_HEADER: &str = "x-rustfs-rpc-nonce";
-
-/// gRPC service name carried in the signed scope, i.e. `TONIC_RPC_PREFIX`
-/// without its leading `/`.
-fn node_service_name() -> &'static str {
-    TONIC_RPC_PREFIX.trim_start_matches('/')
-}
-
-/// Make the RPC secret of this test process match the child server's.
-///
-/// The secret lands in a process-wide `OnceLock`, so the first writer wins for
-/// the whole test binary. Every test here uses the same constant, and the
-/// assertion turns a cross-test collision into an explicit failure instead of a
-/// confusing wall of signature rejections.
-fn align_rpc_secret_with_server() {
-    let _ = rustfs_credentials::set_global_rpc_secret(TEST_RPC_SECRET.to_string());
-    let effective = rustfs_credentials::try_get_rpc_token().expect("RPC secret must resolve in the test process");
-    assert_eq!(
-        effective, TEST_RPC_SECRET,
-        "another test in this binary already fixed a different process-wide RPC secret; \
-         the signature tests cannot mint requests the child server will accept"
-    );
-}
-
-/// Start a `rustfs` child process sharing [`TEST_RPC_SECRET`], with the rollout
-/// posture pinned explicitly.
-///
-/// The child inherits the ambient environment, so the strict gates and the
-/// replay-cache capacity are set here rather than assumed: a developer or CI
-/// runner exporting `RUSTFS_INTERNODE_RPC_*` would otherwise silently flip the
-/// posture and fail these tests for a non-security reason. `extra_env` is
-/// applied last so the strict tests can still override.
-///
-/// Uses the no-cleanup spawn so a `pkill` pattern cannot reap servers belonging
-/// to other tests running in the same binary.
-async fn start_server(extra_env: &[(&str, &str)]) -> Result<RustFSTestEnvironment, Box<dyn Error + Send + Sync>> {
-    let mut env = RustFSTestEnvironment::new().await?;
-    let mut child_env = vec![
-        ("RUSTFS_RPC_SECRET", TEST_RPC_SECRET),
-        (ENV_INTERNODE_RPC_SIGNATURE_STRICT, "false"),
-        (ENV_INTERNODE_RPC_BODY_DIGEST_STRICT, "false"),
-        (ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY, "1048576"),
-    ];
-    child_env.extend_from_slice(extra_env);
-    env.start_rustfs_server_without_cleanup_with_env(&child_env).await?;
-    Ok(env)
-}
-
-/// Stop the child and drop the cached gRPC channel for its address.
-///
-/// `node_service_time_out_client_no_auth` memoises channels in a process-global
-/// map keyed by URL, and ports handed out by `find_available_port` can recur
-/// within one test binary. Evicting here keeps a later test from inheriting a
-/// channel aimed at this test's dead server.
-async fn stop_server(mut env: RustFSTestEnvironment, url: &str) {
-    env.stop_server();
-    rustfs_protos::evict_failed_connection(url).await;
-}
-
-/// The audience the server binds into the v2 signature: its own node authority.
-///
-/// A single-node server started with `--address 127.0.0.1:PORT` over filesystem
-/// endpoints has no URL peer set, so `init_local_peer` falls back to
-/// `host:port` — exactly the address we dialed. The positive controls below
-/// fail loudly if that ever stops holding.
-fn audience_of(env: &RustFSTestEnvironment) -> String {
-    env.address.clone()
-}
-
-fn hex_sha256(bytes: &[u8]) -> String {
-    Sha256::digest(bytes).iter().fold(String::new(), |mut acc, byte| {
-        use std::fmt::Write as _;
-        let _ = write!(acc, "{byte:02x}");
-        acc
-    })
-}
-
-fn make_volume_request(volume: &str) -> MakeVolumeRequest {
-    MakeVolumeRequest {
-        disk: ABSENT_DISK.to_string(),
-        volume: volume.to_string(),
-    }
-}
-
-fn canonical_digest(request: &MakeVolumeRequest) -> String {
-    hex_sha256(&canonical_make_volume_request_body(request).expect("canonical body must encode"))
-}
-
-/// Mint a full v2 header set for `(audience, rpc_method, content_sha256)`.
-///
-/// This is the only place a signature is produced. Tests treat the returned map
-/// as an opaque captured artifact.
-fn mint_v2_headers(audience: &str, rpc_method: &str, content_sha256: Option<&str>) -> HeaderMap {
-    gen_tonic_signature_headers(audience, node_service_name(), rpc_method, content_sha256)
-        .expect("minting a v2 signature must succeed once the RPC secret is aligned")
-}
-
-/// Mint the pre-v2 header set: a signature over the fixed
-/// `TONIC_RPC_PREFIX|GET|timestamp` constant, with no v2 headers at all. This is
-/// both what an un-upgraded peer sends and what an attacker sends to force a
-/// downgrade.
-fn mint_legacy_only_headers() -> HeaderMap {
-    gen_signature_headers(TONIC_RPC_PREFIX, &Method::GET).expect("minting a legacy signature must succeed")
-}
-
-/// Replace one header of a captured set, asserting it was there to begin with.
-fn overwrite_header(headers: &mut HeaderMap, name: &'static str, value: &str) {
-    assert!(
-        headers.contains_key(name),
-        "minted headers must carry {name}; the wire contract changed and this attack would edit nothing"
-    );
-    headers.insert(name, value.parse().expect("header value must be valid"));
-}
-
-/// Send `request` to the server's NodeService with exactly `headers` attached
-/// and nothing else — no interceptor adds or rewrites auth metadata, so the
-/// bytes on the wire are the ones the test chose.
-async fn call_make_volume(url: &str, request: MakeVolumeRequest, headers: HeaderMap) -> Result<MakeVolumeResponse, Status> {
-    let mut client = node_service_time_out_client_no_auth(&url.to_string())
-        .await
-        .map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
-    let mut rpc_request = Request::new(request);
-    rpc_request.metadata_mut().as_mut().extend(headers);
-    client.make_volume(rpc_request).await.map(|response| response.into_inner())
-}
-
-/// Assert a call cleared authentication.
-///
-/// Receiving *any* `Ok` response is the load-bearing signal: both auth layers
-/// reject with a `Status`, so an `Ok` means the request reached handler logic.
-/// The failed disk lookup underneath is what keeps it side-effect free.
-fn assert_authenticated(result: Result<MakeVolumeResponse, Status>, context: &str) {
-    match result {
-        Ok(response) => {
-            assert!(
-                !response.success,
-                "{context}: the absent disk {ABSENT_DISK} must not yield a successful volume creation"
-            );
-            assert!(
-                response.error.is_some(),
-                "{context}: expected the request to reach disk lookup and fail there, got no error"
-            );
-        }
-        Err(status) => panic!(
-            "{context}: the request must clear authentication, but was rejected with {:?}: {}",
-            status.code(),
-            status.message()
-        ),
-    }
-}
-
-/// Assert a call was rejected, optionally pinning which check spoke.
-///
-/// `PermissionDenied` responses carry the reason on the wire, so the digest
-/// tests pin it and cannot be satisfied by an unrelated digest-gate failure.
-/// `Unauthenticated` is deliberately generic on the wire; those tests pin their
-/// cause structurally instead, by differing from a passing request in exactly
-/// one respect.
-fn assert_rejected(result: Result<MakeVolumeResponse, Status>, expected: Code, expected_message: Option<&str>, context: &str) {
-    match result {
-        Ok(response) => panic!(
-            "{context}: the request must be rejected, but the server accepted it and ran the handler \
-             (success={}, error={:?})",
-            response.success, response.error
-        ),
-        Err(status) => {
-            assert_eq!(
-                status.code(),
-                expected,
-                "{context}: expected {expected:?}, got {:?}: {}",
-                status.code(),
-                status.message()
-            );
-            if let Some(needle) = expected_message {
-                assert!(
-                    status.message().contains(needle),
-                    "{context}: expected the rejection to cite {needle:?}, got {:?}",
-                    status.message()
-                );
-            }
-        }
-    }
-}
-
-/// Default posture (both strict gates off): the protections that hold without
-/// any operator flip.
-///
-/// Grouped into one server start because each case is independent and spawning
-/// a `rustfs` process per assertion would dominate the runtime.
-#[tokio::test]
-#[serial]
-async fn internode_rpc_signature_default_posture_e2e() -> TestResult {
-    init_logging();
-    align_rpc_secret_with_server();
-    let env = start_server(&[]).await?;
-    let url = env.url.clone();
-    let audience = audience_of(&env);
-
-    signed_mutations_are_accepted(&url, &audience).await;
-    unsigned_request_is_rejected(&url).await;
-    cross_method_signature_transplant_is_rejected(&url, &audience).await;
-    nonce_replay_of_a_captured_mutation_is_rejected(&url, &audience).await;
-    swapping_in_a_fresh_nonce_is_rejected(&url, &audience).await;
-    tampered_mutation_body_is_rejected(&url, &audience).await;
-    rewriting_the_digest_to_match_a_tampered_body_is_rejected(&url, &audience).await;
-    signature_minted_for_another_node_is_rejected(&url).await;
-    legacy_only_signature_is_accepted_in_default_posture(&url).await;
-
-    stop_server(env, &url).await;
-    Ok(())
-}
-
-/// Baseline: correctly signed mutations are accepted, both with and without a
-/// body digest.
-///
-/// These anchor every rejection below. The body-bound case proves the audience
-/// the server verifies against really is the address we dialed. The digestless
-/// case is the control the transplant test needs: without it, a regression that
-/// rejected every `UNSIGNED-PAYLOAD` request would make the transplant
-/// assertion pass for entirely the wrong reason. It also documents that the
-/// default posture still serves digestless mutations.
-async fn signed_mutations_are_accepted(url: &str, audience: &str) {
-    let bound = make_volume_request("signature-e2e-control-bound");
-    let bound_headers = mint_v2_headers(audience, "MakeVolume", Some(&canonical_digest(&bound)));
-    assert_authenticated(
-        call_make_volume(url, bound, bound_headers).await,
-        "a correctly signed body-bound mutation",
-    );
-
-    let digestless = make_volume_request("signature-e2e-control-digestless");
-    let digestless_headers = mint_v2_headers(audience, "MakeVolume", None);
-    assert_authenticated(
-        call_make_volume(url, digestless, digestless_headers).await,
-        "a correctly signed digestless mutation in the default posture",
-    );
-}
-
-/// A request with no auth metadata at all must never reach a handler.
-async fn unsigned_request_is_rejected(url: &str) {
-    let result = call_make_volume(url, make_volume_request("signature-e2e-unsigned"), HeaderMap::new()).await;
-    assert_rejected(result, Code::Unauthenticated, None, "an entirely unsigned mutation");
-}
-
-/// GHSA-c667 class: a signature captured from one gRPC method must not be
-/// replayable onto another.
-///
-/// Before method-path binding every NodeService call signed the same constant,
-/// so a captured `Ping` — the cheapest, least privileged call on the service —
-/// authenticated a `MakeVolume` just as well. The captured `Ping` signature is
-/// transplanted verbatim; the server recomputes the scope with
-/// `rpc_method = MakeVolume` and the HMAC no longer matches. It differs from the
-/// accepted digestless control above only in the method it was minted for.
-async fn cross_method_signature_transplant_is_rejected(url: &str, audience: &str) {
-    let captured_ping = mint_v2_headers(audience, "Ping", None);
-    let result = call_make_volume(url, make_volume_request("signature-e2e-transplant"), captured_ping).await;
-    assert_rejected(
-        result,
-        Code::Unauthenticated,
-        None,
-        "a Ping signature transplanted onto a MakeVolume mutation",
-    );
-}
-
-/// A body-bound mutation must be consumable exactly once.
-///
-/// The first send establishes that the captured artifact is genuinely valid —
-/// without it, the second rejection could just mean the headers were malformed
-/// all along. The replay reuses the identical `(signature, timestamp, nonce)`
-/// well inside the freshness window, so only the server's replay cache can
-/// stop it.
-async fn nonce_replay_of_a_captured_mutation_is_rejected(url: &str, audience: &str) {
-    let request = make_volume_request("signature-e2e-replay");
-    let captured = mint_v2_headers(audience, "MakeVolume", Some(&canonical_digest(&request)));
-
-    let first = call_make_volume(url, request.clone(), captured.clone()).await;
-    assert_authenticated(first, "the captured mutation on its first delivery");
-
-    let replayed = call_make_volume(url, request, captured).await;
-    assert_rejected(
-        replayed,
-        Code::Unauthenticated,
-        None,
-        "the same captured mutation replayed after its nonce was consumed",
-    );
-}
-
-/// The nonce must be *signed*, not merely remembered.
-///
-/// A replay cache alone would be trivially defeated: swap in a fresh UUID and
-/// the cache has never seen it. This request is byte-identical to one the server
-/// would accept apart from that one header, so it can only be stopped by the
-/// nonce being inside the signed scope.
-async fn swapping_in_a_fresh_nonce_is_rejected(url: &str, audience: &str) {
-    let request = make_volume_request("signature-e2e-nonce-swap");
-    let mut captured = mint_v2_headers(audience, "MakeVolume", Some(&canonical_digest(&request)));
-    overwrite_header(&mut captured, NONCE_HEADER, &Uuid::new_v4().to_string());
-
-    let result = call_make_volume(url, request, captured).await;
-    assert_rejected(
-        result,
-        Code::Unauthenticated,
-        None,
-        "a captured mutation resent under a freshly minted nonce",
-    );
-}
-
-/// Editing the body of a captured request must invalidate it, in the default
-/// posture, with no operator flip required.
-///
-/// The headers are left byte-identical — including the signed digest of the
-/// original body — so `check_auth` still passes. Only the handler, recomputing
-/// the canonical body from the fields it actually received, can catch this. It
-/// is the test that fails if a handler ever loses its digest gate.
-async fn tampered_mutation_body_is_rejected(url: &str, audience: &str) {
-    let signed = make_volume_request("signature-e2e-tamper-a");
-    let captured = mint_v2_headers(audience, "MakeVolume", Some(&canonical_digest(&signed)));
-
-    // Exactly one byte of the volume name differs from what the digest covers.
-    let tampered = make_volume_request("signature-e2e-tamper-b");
-    let result = call_make_volume(url, tampered, captured).await;
-    assert_rejected(
-        result,
-        Code::PermissionDenied,
-        Some("RPC content SHA-256 mismatch"),
-        "a mutation whose body was edited after signing",
-    );
-}
-
-/// The body digest must be *inside the signed scope*, not merely cross-checked
-/// by the handler.
-///
-/// This is the same tampered body as above, except the attacker also repairs the
-/// digest header so it matches what it sends — defeating the handler's
-/// comparison. The only thing left standing is the signature, which covers the
-/// digest header itself. Drop `content_sha256` from `update_signature_v2` and
-/// this is the test that goes green when it should not.
-async fn rewriting_the_digest_to_match_a_tampered_body_is_rejected(url: &str, audience: &str) {
-    let signed = make_volume_request("signature-e2e-scope-a");
-    let mut captured = mint_v2_headers(audience, "MakeVolume", Some(&canonical_digest(&signed)));
-
-    let tampered = make_volume_request("signature-e2e-scope-b");
-    overwrite_header(&mut captured, CONTENT_SHA256_HEADER, &canonical_digest(&tampered));
-
-    let result = call_make_volume(url, tampered, captured).await;
-    assert_rejected(
-        result,
-        Code::Unauthenticated,
-        None,
-        "a tampered mutation whose digest header was repaired to match",
-    );
-}
-
-/// A signature is bound to its destination node, so a request captured against
-/// one node cannot be aimed at another.
-///
-/// `127.0.0.1:1` stands in for a different peer; the audience is inside the
-/// HMAC, so the server's own authority no longer reproduces it.
-async fn signature_minted_for_another_node_is_rejected(url: &str) {
-    let request = make_volume_request("signature-e2e-wrong-node");
-    let headers = mint_v2_headers("127.0.0.1:1", "MakeVolume", Some(&canonical_digest(&request)));
-    let result = call_make_volume(url, request, headers).await;
-    assert_rejected(result, Code::Unauthenticated, None, "a signature minted for a different node");
-}
-
-/// Rolling-upgrade compatibility: a peer that predates v2 must still be served
-/// while the strict gates are off.
-///
-/// This is the case the issue insists must not fail closed during an upgrade.
-/// It is also, honestly, the open downgrade window: an attacker can strip the
-/// v2 headers and land here too. That window is what
-/// [`signature_strict_rejects_legacy_only_downgrade`] closes.
-async fn legacy_only_signature_is_accepted_in_default_posture(url: &str) {
-    let result = call_make_volume(url, make_volume_request("signature-e2e-legacy"), mint_legacy_only_headers()).await;
-    assert_authenticated(result, "a legacy-only signature in the default posture");
-}
-
-/// With `RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT` on, the legacy downgrade lane is
-/// closed: the exact request accepted in the default posture is now refused.
-///
-/// The paired v2 positive control rules out "strict simply breaks everything".
-#[tokio::test]
-#[serial]
-async fn signature_strict_rejects_legacy_only_downgrade() -> TestResult {
-    init_logging();
-    align_rpc_secret_with_server();
-    let env = start_server(&[(ENV_INTERNODE_RPC_SIGNATURE_STRICT, "true")]).await?;
-    let url = env.url.clone();
-    let audience = audience_of(&env);
-
-    let downgraded = call_make_volume(&url, make_volume_request("signature-e2e-strict-legacy"), mint_legacy_only_headers()).await;
-    assert_rejected(
-        downgraded,
-        Code::Unauthenticated,
-        None,
-        "a legacy-only signature once signature-strict is enabled",
-    );
-
-    let request = make_volume_request("signature-e2e-strict-v2");
-    let signed = mint_v2_headers(&audience, "MakeVolume", Some(&canonical_digest(&request)));
-    assert_authenticated(
-        call_make_volume(&url, request, signed).await,
-        "a v2-signed mutation under signature-strict",
-    );
-
-    stop_server(env, &url).await;
-    Ok(())
-}
-
-/// With `RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT` on, any mutation that arrives
-/// without a body digest is refused — including one that downgraded all the way
-/// to the legacy signature.
-///
-/// This gate converges independently of the signature gate, so it is exercised
-/// on its own server with signature-strict left off. Both rejected requests
-/// clear `check_auth` on their own terms (one is properly v2-signed, the other
-/// takes the still-open legacy lane), which is what pins the rejection to the
-/// handler's digest gate; the cited message confirms which check spoke.
-#[tokio::test]
-#[serial]
-async fn body_digest_strict_rejects_digestless_mutation() -> TestResult {
-    init_logging();
-    align_rpc_secret_with_server();
-    let env = start_server(&[(ENV_INTERNODE_RPC_BODY_DIGEST_STRICT, "true")]).await?;
-    let url = env.url.clone();
-    let audience = audience_of(&env);
-
-    let digestless = mint_v2_headers(&audience, "MakeVolume", None);
-    assert_rejected(
-        call_make_volume(&url, make_volume_request("signature-e2e-digestless"), digestless).await,
-        Code::PermissionDenied,
-        Some("RPC mutation requires a body-bound v2 signature"),
-        "a v2-signed but digestless mutation once body-digest-strict is enabled",
-    );
-
-    assert_rejected(
-        call_make_volume(&url, make_volume_request("signature-e2e-digestless-legacy"), mint_legacy_only_headers()).await,
-        Code::PermissionDenied,
-        Some("RPC mutation requires a body-bound v2 signature"),
-        "a v1-downgraded mutation once body-digest-strict is enabled",
-    );
-
-    let request = make_volume_request("signature-e2e-digest-bound");
-    let bound = mint_v2_headers(&audience, "MakeVolume", Some(&canonical_digest(&request)));
-    assert_authenticated(
-        call_make_volume(&url, request, bound).await,
-        "a body-bound mutation under body-digest-strict",
-    );
-
-    stop_server(env, &url).await;
-    Ok(())
-}

crates/e2e_test/src/kms/common.rs

@@ -29,10 +29,6 @@ use aws_sdk_s3::Client;
 use aws_sdk_s3::primitives::ByteStream;
 use aws_sdk_s3::types::ServerSideEncryption;
 use base64::{Engine, engine::general_purpose::STANDARD as BASE64};
-use http::header::{CONTENT_TYPE, HOST};
-use rustfs_signer::constants::UNSIGNED_PAYLOAD;
-use rustfs_signer::sign_v4;
-use s3s::Body;
 use serde_json;
 use std::process::{Child, Command};
 use std::time::Duration;
@@ -71,49 +67,6 @@ pub fn sse_customer_key_md5_base64(key: &str) -> String {
     BASE64.encode(md5::compute(key).0)
 }
 
-pub async fn kms_admin_request(
-    base_url: &str,
-    method: http::Method,
-    path_and_query: &str,
-    body: Option<&str>,
-    access_key: &str,
-    secret_key: &str,
-) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
-    let url = format!("{base_url}{path_and_query}");
-    let uri = url.parse::<http::Uri>()?;
-    let authority = uri.authority().ok_or("KMS admin URL missing authority")?.to_string();
-    let mut builder = http::Request::builder()
-        .method(method.clone())
-        .uri(uri)
-        .header(HOST, authority)
-        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
-    if body.is_some() {
-        builder = builder.header(CONTENT_TYPE, "application/json");
-    }
-
-    let content_len = match body {
-        Some(value) => i64::try_from(value.len())?,
-        None => 0,
-    };
-    let signed = sign_v4(builder.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1");
-
-    let mut request = local_http_client().request(method.clone(), &url);
-    for (name, value) in signed.headers() {
-        request = request.header(name, value);
-    }
-    if let Some(value) = body {
-        request = request.body(value.to_owned());
-    }
-
-    let response = request.send().await?;
-    let status = response.status();
-    let response_body = response.text().await?;
-    if !status.is_success() {
-        return Err(format!("{method} {path_and_query} failed with {status}: {response_body}").into());
-    }
-    Ok(response_body)
-}
-
 // KMS-specific helper functions
 /// Configure KMS backend via admin API
 pub async fn configure_kms(
@@ -122,19 +75,8 @@ pub async fn configure_kms(
     access_key: &str,
     secret_key: &str,
 ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    let response = kms_admin_request(
-        base_url,
-        http::Method::POST,
-        "/rustfs/admin/v3/kms/configure",
-        Some(config_json),
-        access_key,
-        secret_key,
-    )
-    .await?;
-    let response: serde_json::Value = serde_json::from_str(&response)?;
-    if response["success"] != true {
-        return Err(format!("KMS configuration failed: {}", response["message"].as_str().unwrap_or("unknown error")).into());
-    }
+    let url = format!("{base_url}/rustfs/admin/v3/kms/configure");
+    awscurl_post(&url, config_json, access_key, secret_key).await?;
     info!("KMS configured successfully");
     Ok(())
 }
@@ -145,19 +87,8 @@ pub async fn start_kms(
     access_key: &str,
     secret_key: &str,
 ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    let response = kms_admin_request(
-        base_url,
-        http::Method::POST,
-        "/rustfs/admin/v3/kms/start",
-        Some("{}"),
-        access_key,
-        secret_key,
-    )
-    .await?;
-    let response: serde_json::Value = serde_json::from_str(&response)?;
-    if response["success"] != true {
-        return Err(format!("KMS start failed: {}", response["message"].as_str().unwrap_or("unknown error")).into());
-    }
+    let url = format!("{base_url}/rustfs/admin/v3/kms/start");
+    awscurl_post(&url, "{}", access_key, secret_key).await?;
     info!("KMS started successfully");
     Ok(())
 }
@@ -168,8 +99,8 @@ pub async fn get_kms_status(
     access_key: &str,
     secret_key: &str,
 ) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
-    let status =
-        kms_admin_request(base_url, http::Method::GET, "/rustfs/admin/v3/kms/status", None, access_key, secret_key).await?;
+    let url = format!("{base_url}/rustfs/admin/v3/kms/status");
+    let status = awscurl_get(&url, access_key, secret_key).await?;
     info!("KMS status retrieved: {}", status);
     Ok(status)
 }
@@ -577,8 +508,7 @@ impl VaultTestEnvironment {
             },
             "mount_path": VAULT_TRANSIT_PATH,
             "default_key_id": VAULT_KEY_NAME,
-            "skip_tls_verify": true,
-            "allow_insecure_dev_defaults": true
+            "skip_tls_verify": true
         })
         .to_string();
 
@@ -727,19 +657,14 @@ pub async fn test_multipart_upload_with_config(
         .build();
 
     info!("🔗 Completing multipart upload");
-    let mut complete_request = s3_client
+    let complete_output = s3_client
         .complete_multipart_upload()
         .bucket(bucket)
         .key(&config.object_key)
         .upload_id(upload_id)
-        .multipart_upload(completed_multipart_upload);
-    if let EncryptionType::SSEC { .. } = &config.encryption_type {
-        complete_request = complete_request
-            .sse_customer_algorithm("AES256")
-            .sse_customer_key(sse_c_key_b64.as_ref().unwrap())
-            .sse_customer_key_md5(sse_c_key_md5.as_ref().unwrap());
-    }
-    let complete_output = complete_request.send().await?;
+        .multipart_upload(completed_multipart_upload)
+        .send()
+        .await?;
 
     debug!("Multipart upload finalized with ETag {:?}", complete_output.e_tag());
 
@@ -871,8 +796,7 @@ impl LocalKMSTestEnvironment {
             "backend_type": "Local",
             "key_dir": self.kms_keys_dir,
             "file_permissions": 0o600,
-            "default_key_id": default_key_id,
-            "allow_insecure_dev_defaults": true
+            "default_key_id": default_key_id
         })
         .to_string();