fix(replication): honor disabled version deletes

.agents/skills/rustfs-release-publish/SKILL.md

@@ -1,21 +1,19 @@
 ---
 name: rustfs-release-publish
-description: "End-to-end RustFS release pipeline: first publish any merged-but-unreleased rustfs/console changes and wait for its latest Release asset, then bump RustFS version files on main directly to the final target, publish a visible GitHub prerelease from a preview tag without updating latest channels, validate it, and publish the final tag on the SAME commit. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
+description: "End-to-end RustFS release pipeline: bump version files on main directly to the final target version, cut a preview tag on that commit, verify the CI build and release artifacts, run the downloaded binary locally and exercise the console, validate the server with the latest rc client, then publish the final tag on the SAME validated commit — never a new bump commit, never latest main. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
 ---
 # RustFS Release Publish (preview-validated pipeline)
 
 This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
 
-Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
+Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and prerelease classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
 
 Pipeline shape:
 
 ```
-check console main against its latest Release
-  -> if ahead: publish console -> wait for Release asset + latest API
-  -> bump RustFS version files to <target> (final version, ONE commit) -> merge
+bump version files to <target> (final version, ONE commit) -> merge
   -> tag <preview-tag> at that commit -> CI green
-  -> verify preview Release assets -> run binary locally + console checks
+  -> verify release artifacts -> run binary locally + console checks
   -> validate with latest rc client
   -> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
 ```
@@ -25,7 +23,7 @@ On validation failure: fix lands on main via normal PR (version files are alread
 ## Required inputs
 
 - Final target version, for example `1.0.0-beta.10`.
-- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` after `git fetch --tags`).
+- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` — and for stable targets `git tag -l '<target>-rc.*'` — after `git fetch --tags`).
 
 If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
 
@@ -47,18 +45,14 @@ Rules:
 
 ## Preview tag naming
 
-- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
-- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
-- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
+- Prerelease target (contains `alpha`/`beta`/`rc`): preview tag is `<target>-preview.N`, e.g. `1.0.0-beta.10-preview.3`. It contains `beta`, so `build.yml`'s substring-based classification marks it prerelease — safe.
+- **Stable** target (e.g. `1.1.0`): NEVER tag `1.1.0-preview.N` — `build.yml` marks a tag prerelease only if its name contains `alpha`, `beta`, or `rc`, so `1.1.0-preview.N` would be treated as a stable release and overwrite `latest.json` as stable. Use `1.1.0-rc.N` as the preview tag instead.
 
 ## Hard rules
 
 - Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
-- Preview Release assets are versioned and intentionally visible on the Releases page. Do not label them Latest or use them to update any latest distribution channel.
 - Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
 - The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
-- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
-- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
 - Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
 - If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
 - User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
@@ -69,61 +63,6 @@ Rules:
 - `gh auth status` works; confirm you can view `gh release list -L 3`.
 - Confirm the exact final target version with the user if not explicit.
 
-### Console release gate
-
-Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
-
-1. Read the latest published Console tag and compare it with Console `main`:
-
-```bash
-CONSOLE_REPO="rustfs/console"
-CONSOLE_LATEST=$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)
-gh api "repos/${CONSOLE_REPO}/compare/${CONSOLE_LATEST}...main" \
-  --jq '{status, ahead_by, behind_by, commits: [.commits[] | {sha, message: .commit.message}]}'
-```
-
-- `ahead_by == 0`: no merged Console change is waiting for release. Still verify the current latest asset using step 4, then continue to Phase 1.
-- `ahead_by > 0` and `behind_by == 0`: publish Console before continuing. Report the merged commits and select the next unused `vX.Y.Z` tag. Default to the next patch version when the changes are fixes or backward-compatible UI work; stop for confirmation if a minor/major bump is plausible.
-- Any diverged history or `behind_by > 0`: stop and resolve the Console release baseline explicitly. Do not guess a range or publish RustFS.
-
-2. Clone/fetch `rustfs/console` into a scratch directory and record its exact `main` commit. Before creating a tag, check for a `v*` tag or Release workflow already associated with that hash. If one is in progress, wait for it instead of creating another version:
-
-```bash
-CONSOLE_SCRATCH=$(mktemp -d)
-gh repo clone "$CONSOLE_REPO" "$CONSOLE_SCRATCH/console"
-git -C "$CONSOLE_SCRATCH/console" fetch origin main --tags
-CONSOLE_HASH=$(git -C "$CONSOLE_SCRATCH/console" rev-parse origin/main)
-git -C "$CONSOLE_SCRATCH/console" tag --points-at "$CONSOLE_HASH" 'v*'
-gh run list -R "$CONSOLE_REPO" --workflow release.yml --commit "$CONSOLE_HASH" --limit 5
-```
-
-If no release exists or is running for `CONSOLE_HASH`, create the selected annotated tag at that exact hash and push it:
-
-```bash
-git -C "$CONSOLE_SCRATCH/console" tag -a "<console-tag>" -m "Release <console-tag>" "$CONSOLE_HASH"
-git -C "$CONSOLE_SCRATCH/console" push origin "<console-tag>"
-```
-
-Console tags include the `v` prefix. Pushing the tag triggers `.github/workflows/release.yml` (`🚀 Release`). Remove `CONSOLE_SCRATCH` after the gate completes.
-
-3. Find the exact tag run and wait for completion:
-
-```bash
-gh run list -R "$CONSOLE_REPO" --workflow release.yml --branch "<console-tag>" --limit 1
-gh run watch -R "$CONSOLE_REPO" "<console-run-id>" --exit-status
-```
-
-4. Block until the published Release is non-draft, the latest endpoint returns the expected tag, and `rustfs-console-<console-tag>.zip` is uploaded, non-empty, and carries a `sha256:` digest:
-
-```bash
-gh release view -R "$CONSOLE_REPO" "<console-tag>" --json isDraft,isPrerelease,assets,url
-test "$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)" = "<console-tag>"
-test "$(gh api "repos/${CONSOLE_REPO}/releases/tags/<console-tag>" \
-  --jq '[.assets[] | select(.name == "rustfs-console-<console-tag>.zip" and .state == "uploaded" and .size > 0 and (.digest | startswith("sha256:")))] | length')" -eq 1
-```
-
-Treat a missing/mismatched asset, digest, latest tag, or failed/cancelled workflow as BLOCKED. Do not start Phase 1 until the Console gate passes. Record `CONSOLE_TAG`, `CONSOLE_HASH`, Console run URL, and Release URL for the final report.
-
 ## Phase 1 — Version bump to the final target (once)
 
 - If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
@@ -146,17 +85,16 @@ git push origin "<preview-tag>"
 
 Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
 
-The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Its latest-channel, R2, Docker, and Helm jobs must be skipped. Those publication paths run only after the final tag is pushed.
-
 On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
 
-## Phase 3 — CI and preview Release verification
+## Phase 3 — CI and artifact verification
 
-- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
-- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
-- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
-- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
-- Confirm preview-triggered Docker and Helm jobs are skipped. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; Docker image construction and Helm publication are deferred to the final tag because the Dockerfiles consume GitHub Release assets.
+- Watch the tag build: `gh run list --workflow build.yml --limit 5` then `gh run watch <run-id>`. Every matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64) plus the release and latest.json jobs.
+- Verify the GitHub release: `gh release view "<preview-tag>" --json isPrerelease,assets`
+  - `isPrerelease` must be `true`.
+  - Assets must include all 6 platform zips in both versioned (`rustfs-<platform>-v<tag>.zip`) and `-latest` forms, plus `SHA256SUMS`, `SHA512SUMS`, `rustfs-<tag>.sbom.cdx.json`, `rustfs-<tag>.provenance.json`.
+- Verify the chained Docker run succeeded: `gh run list --workflow docker.yml --limit 3`.
+- Checksum spot-check for the platform you will run locally: download the zip and `SHA256SUMS`, verify with `shasum -a 256 -c` (grep to one line).
 
 ## Phase 4 — Run the artifact locally, verify the console
 
@@ -219,15 +157,13 @@ git push origin "<target>"
 ```
 
 - CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
-- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
-- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
+- Re-run the Phase 3 verification against the final tag: all matrix jobs green; `gh release view "<target>"` shows the full asset set; for a prerelease target `isPrerelease` is `true`, for a stable target it must be `false` and `latest.json` must be updated.
 - Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
 
 ## Output contract
 
 Always report:
 
-- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
 - Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
-- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, and the rc command matrix.
+- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: CI run URLs, release URLs, console check results, the rc command matrix.
 - Any deviation from this pipeline and why the user approved it.

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -18,7 +18,7 @@ Validated baseline: release pattern used in PR `#2957`.
 
 If target version is missing or ambiguous, stop and ask before editing.
 
-Reject any target version containing `-preview`: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
+Reject any target version containing a `-preview.` suffix: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
 
 ## Read before editing
 

.config/make/lint-fmt.mak

@@ -60,11 +60,6 @@ body-cache-whitelist-check: ## Check the body-cache eligibility gate stays a fai
 	@echo "🧱 Checking body-cache whitelist guard..."
 	./scripts/check_body_cache_whitelist.sh
 
-.PHONY: fips-wording-check
-fips-wording-check: ## Check outward docs do not make unsupported FIPS claims
-	@echo "📣 Checking FIPS wording guard..."
-	./scripts/check_fips_wording.sh
-
 .PHONY: log-analyzer-rules-check
 log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
 	@echo "🩺 Checking log-analyzer rule anchors..."

.config/make/pre-commit.mak

@@ -19,13 +19,13 @@ planning-docs-check: ## Check that no planning-type documents are committed
 	./scripts/check_no_planning_docs.sh
 
 .PHONY: pre-commit
-pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check fips-wording-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
+pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
 	@echo "✅ All pre-commit checks passed!"
 
 .PHONY: pre-pr
-pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check fips-wording-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
+pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
 	@echo "✅ All pre-PR checks passed!"
 
 .PHONY: dev-check
-dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check fips-wording-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
+dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
 	@echo "✅ Fast development checks passed!"

.config/make/tests.mak

@@ -28,8 +28,6 @@ script-tests: ## Run shell script tests
 	./scripts/test_entrypoint_credentials.sh
 	./scripts/test_internode_grpc_ab_bench.sh
 	./scripts/test_object_batch_bench_enhanced.sh
-	./scripts/test_hotpath_warp_ab_gate.sh
-	./scripts/test_hotpath_warp_abba.sh
 	./scripts/test_exact_1mib_handoff_abba.sh
 	./scripts/test_pinned_paired_abba_bench.sh
 	./scripts/test_manual_transition_runbooks.sh

.config/nextest.toml

@@ -9,8 +9,6 @@
 #   * bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
 #     uses the shared multipart fixture and a deterministic uploadId-lock
 #     handoff, so it must not overlap another process mutating that fixture.
-#   * bucket::metadata_sys::tests::concurrent_config_writes_from_separate_nodes_do_not_lose_writes
-#     uses the shared transaction lock and must not overlap other ecstore tests.
 #
 # serial_test's #[serial] attribute does NOT serialize these across runs:
 # nextest executes each test in its own process, where the in-process
@@ -42,7 +40,7 @@ e2e-inline-boundaries = { max-threads = 1 }
 
 # --- default profile (local): serialize the flaky groups, never retry --------
 [[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
+filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
 test-group = 'ecstore-serial-flaky'
 
 # Serialize the multipart crash-consistency scenarios (dist-2, backlog#1150):
@@ -60,16 +58,6 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
 test-group = 'ecstore-serial-flaky'
 
-# Serialize the bucket-incarnation / lifecycle-fence tests. They drive
-# init_bucket_metadata_sys and bucket_metadata_sys_of, i.e. process-global
-# OnceLock state that serial_test's #[serial] cannot protect across nextest's
-# process boundary, and they delete+recreate buckets — the same shape that
-# raced into InsufficientWriteQuorum in backlog#937. Preventive only, no
-# retries. The matching ci-profile override is after [profile.ci].
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
-test-group = 'ecstore-serial-flaky'
-
 # Serialize the 4-disk reliability / degraded-read e2e tests (see the
 # e2e-reliability test-group note above). The matching ci-profile override is at
 # the end of the file, after [profile.ci] is declared.
@@ -116,9 +104,9 @@ filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(
 test-group = 'ecstore-serial-flaky'
 retries = 2
 
-# Keep deterministic ECStore write handoffs isolated across nextest processes.
+# Keep the deterministic multipart handoff isolated across nextest processes.
 [[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes))'
+filter = 'package(rustfs-ecstore) & test(concurrent_resend_same_part_commits_one_generation)'
 test-group = 'ecstore-serial-flaky'
 
 # QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
@@ -149,12 +137,6 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
 test-group = 'ecstore-serial-flaky'
 
-# Serialize the bucket-incarnation / lifecycle-fence tests under the ci profile
-# too (see the matching default-profile override near the top). No retries.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
-test-group = 'ecstore-serial-flaky'
-
 # ---------------------------------------------------------------------------
 # e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
 # ---------------------------------------------------------------------------
@@ -230,17 +212,6 @@ default-filter = """
 """
 fail-fast = false
 
-[profile.e2e-smoke.junit]
-path = "junit.xml"
-
-# The pagination boundary cases can stall when a server/listing regression
-# prevents the continuation request from completing. Keep the timeout scoped
-# to those known failure modes so legitimate lifecycle/tiering waits retain
-# their test-level timing budget.
-[[profile.e2e-smoke.overrides]]
-filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
-slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
-
 # ---------------------------------------------------------------------------
 # e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
 # ---------------------------------------------------------------------------
@@ -329,6 +300,9 @@ path = "junit.xml"
 # negative-path siblings of each family stay in as regression guards.
 #   * rustfs#4843 — over-limit archive entry paths hard-reject the whole
 #     archive even under ignore-errors semantics.
+#   * rustfs#4846 — distributed-lock quorum tests misclassify as timeout
+#     under parallel load (multi-node in-process clusters; natural home is
+#     ci-7's nightly cluster lane).
 [profile.e2e-full]
 default-filter = """
     package(e2e_test)
@@ -337,6 +311,7 @@ default-filter = """
     & !test(/^replication_extension_test::/)
     & !test(/^multipart_auth_test::test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_enabled$/)
     & !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(ignores_invalid_entries_when_requested|supports_standard_headers_with_combined_extract_options)$/)
+    & !test(/^reliant::lock::test_distributed_lock_(2_nodes_grpc_read_survives_failed_node|4_nodes_grpc_read_write_quorum_split_with_two_failed_nodes)$/)
 """
 fail-fast = false
 

.docker/observability/README.md

@@ -60,16 +60,6 @@ The file `prometheus-rules/rustfs-get-optimization-alerts.yaml` contains pre-con
 | `CodecStreamingFallbackSpike` | Warning | Codec streaming fallback > 10x baseline for 10m |
 | `IoQueueSaturation` | Warning | IO queue utilization > 90% for 5m |
 
-The file `prometheus-rules/rustfs-kms-alerts.yml` contains alerting rules for the KMS backend operation metrics. Thresholds are conservative defaults pending staging baseline calibration; response procedures live in `docs/operations/kms-observability-runbook.md`, and the matching dashboard is `deploy/observability/grafana/rustfs-kms-observability.json`.
-
-| Alert | Severity | Condition |
-|-------|----------|-----------|
-| `KmsBackendFatalErrors` | Critical | Fatal (non-retryable) attempt failures > 0 for 5m |
-| `KmsBackendHighErrorRate` | Critical | Non-success operation ratio > 5% for 10m (with traffic guard) |
-| `KmsBackendP99LatencyHigh` | Warning | Operation p99 duration (incl. retries) > 2s for 10m |
-| `KmsBackendAttemptFailureSpike` | Warning | Attempt failure rate > 0.5/s for 10m |
-| `KmsBackendRetryBudgetExhausted` | Warning | budget_exhausted / deadline_exceeded outcomes > 0.05/s for 10m |
-
 ### Enabling Alert Rules
 
 Add the alert rules file to your Prometheus configuration:

.docker/observability/README_ZH.md

@@ -60,16 +60,6 @@
 | `CodecStreamingFallbackSpike` | 警告 | Codec streaming 回退 > 10x 基线，持续 10 分钟 |
 | `IoQueueSaturation` | 警告 | IO 队列利用率 > 90%，持续 5 分钟 |
 
-文件 `prometheus-rules/rustfs-kms-alerts.yml` 包含 KMS 后端操作指标的告警规则。阈值为保守默认值，待 staging 基线校准；响应流程见 `docs/operations/kms-observability-runbook.md`，配套仪表盘为 `deploy/observability/grafana/rustfs-kms-observability.json`。
-
-| 告警 | 级别 | 条件 |
-|------|------|------|
-| `KmsBackendFatalErrors` | 严重 | fatal（不可重试）尝试失败 > 0，持续 5 分钟 |
-| `KmsBackendHighErrorRate` | 严重 | 非 success 操作占比 > 5%，持续 10 分钟（含流量下限保护） |
-| `KmsBackendP99LatencyHigh` | 警告 | 操作 p99 耗时（含重试）> 2s，持续 10 分钟 |
-| `KmsBackendAttemptFailureSpike` | 警告 | 尝试失败率 > 0.5/s，持续 10 分钟 |
-| `KmsBackendRetryBudgetExhausted` | 警告 | budget_exhausted / deadline_exceeded 结果 > 0.05/s，持续 10 分钟 |
-
 ### 启用告警规则
 
 在 Prometheus 配置中添加告警规则文件：

.docker/observability/grafana/dashboards/rustfs.json

@@ -11500,831 +11500,6 @@
       ],
       "title": "Compression Operations Rate",
       "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 332
-      },
-      "id": 531,
-      "panels": [],
-      "title": "Metrics Dimensions Drilldown",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${datasource}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisColorMode": "text",
-            "axisLabel": "",
-            "axisPlacement": "auto",
-            "barAlignment": 0,
-            "barWidthFactor": 0.6,
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "gradientMode": "none",
-            "hideFrom": {
-              "legend": false,
-              "tooltip": false,
-              "viz": false
-            },
-            "insertNulls": false,
-            "lineInterpolation": "smooth",
-            "lineWidth": 1,
-            "pointSize": 5,
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "showPoints": "never",
-            "showValues": false,
-            "spanNulls": false,
-            "stacking": {
-              "group": "A",
-              "mode": "none"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": 0
-              }
-            ]
-          },
-          "unit": "ops"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 333
-      },
-      "id": 532,
-      "options": {
-        "legend": {
-          "calcs": [
-            "lastNotNull",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "right",
-          "showLegend": true
-        },
-        "tooltip": {
-          "hideZeros": false,
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "pluginVersion": "12.3.2",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "A",
-          "expr": "sum by (server, name, type) (rate(rustfs_api_requests_requests_total_by_server{job=~\"$job\",server=~\"$server\",name=~\"$api\"}[$__rate_interval]))",
-          "legendFormat": "{{server}} | {{name}} | {{type}}"
-        }
-      ],
-      "title": "API Requests by Server and API",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${datasource}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisColorMode": "text",
-            "axisLabel": "",
-            "axisPlacement": "auto",
-            "barAlignment": 0,
-            "barWidthFactor": 0.6,
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "gradientMode": "none",
-            "hideFrom": {
-              "legend": false,
-              "tooltip": false,
-              "viz": false
-            },
-            "insertNulls": false,
-            "lineInterpolation": "smooth",
-            "lineWidth": 1,
-            "pointSize": 5,
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "showPoints": "never",
-            "showValues": false,
-            "spanNulls": false,
-            "stacking": {
-              "group": "A",
-              "mode": "none"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": 0
-              }
-            ]
-          },
-          "unit": "s"
-        },
-        "overrides": [
-          {
-            "matcher": {
-              "id": "byFrameRefID",
-              "options": "A"
-            },
-            "properties": [
-              {
-                "id": "unit",
-                "value": "none"
-              }
-            ]
-          }
-        ]
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 333
-      },
-      "id": 533,
-      "options": {
-        "legend": {
-          "calcs": [
-            "lastNotNull",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "right",
-          "showLegend": true
-        },
-        "tooltip": {
-          "hideZeros": false,
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "pluginVersion": "12.3.2",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "A",
-          "expr": "max by (server, drive, pool_index, set_index, drive_index, state) (rustfs_system_drive_runtime_state{job=~\"$job\",server=~\"$server\",drive=~\"$drive\"})",
-          "legendFormat": "{{server}} | {{drive}} | p{{pool_index}}/s{{set_index}}/d{{drive_index}} | {{state}}"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "B",
-          "expr": "max by (server, drive, pool_index, set_index, drive_index) (rustfs_system_drive_offline_duration_seconds{job=~\"$job\",server=~\"$server\",drive=~\"$drive\"})",
-          "legendFormat": "{{server}} | {{drive}} | offline seconds"
-        }
-      ],
-      "title": "Drive Runtime State and Offline Duration",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${datasource}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisColorMode": "text",
-            "axisLabel": "",
-            "axisPlacement": "auto",
-            "barAlignment": 0,
-            "barWidthFactor": 0.6,
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "gradientMode": "none",
-            "hideFrom": {
-              "legend": false,
-              "tooltip": false,
-              "viz": false
-            },
-            "insertNulls": false,
-            "lineInterpolation": "smooth",
-            "lineWidth": 1,
-            "pointSize": 5,
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "showPoints": "never",
-            "showValues": false,
-            "spanNulls": false,
-            "stacking": {
-              "group": "A",
-              "mode": "none"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": 0
-              }
-            ]
-          },
-          "unit": "ops"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 341
-      },
-      "id": 534,
-      "options": {
-        "legend": {
-          "calcs": [
-            "lastNotNull",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "right",
-          "showLegend": true
-        },
-        "tooltip": {
-          "hideZeros": false,
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "pluginVersion": "12.3.2",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "A",
-          "expr": "sum by (server, drive, pool_index, set_index, drive_index, api) (rate(rustfs_system_drive_api_calls_total{job=~\"$job\",server=~\"$server\",drive=~\"$drive\",api=~\"$drive_api\"}[$__rate_interval]))",
-          "legendFormat": "{{server}} | {{drive}} | p{{pool_index}}/s{{set_index}}/d{{drive_index}} | {{api}}"
-        }
-      ],
-      "title": "Drive API Calls by Operation",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${datasource}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisColorMode": "text",
-            "axisLabel": "",
-            "axisPlacement": "auto",
-            "barAlignment": 0,
-            "barWidthFactor": 0.6,
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "gradientMode": "none",
-            "hideFrom": {
-              "legend": false,
-              "tooltip": false,
-              "viz": false
-            },
-            "insertNulls": false,
-            "lineInterpolation": "smooth",
-            "lineWidth": 1,
-            "pointSize": 5,
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "showPoints": "never",
-            "showValues": false,
-            "spanNulls": false,
-            "stacking": {
-              "group": "A",
-              "mode": "none"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": 0
-              }
-            ]
-          },
-          "unit": "ops"
-        },
-        "overrides": [
-          {
-            "matcher": {
-              "id": "byFrameRefID",
-              "options": "B"
-            },
-            "properties": [
-              {
-                "id": "unit",
-                "value": "short"
-              },
-              {
-                "id": "custom.axisPlacement",
-                "value": "right"
-              }
-            ]
-          }
-        ]
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 341
-      },
-      "id": 535,
-      "options": {
-        "legend": {
-          "calcs": [
-            "lastNotNull",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "right",
-          "showLegend": true
-        },
-        "tooltip": {
-          "hideZeros": false,
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "pluginVersion": "12.3.2",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "A",
-          "expr": "sum by (server, source, state) (rate(rustfs_scanner_source_work_total{job=~\"$job\",server=~\"$server\",source=~\"$scanner_source\"}[$__rate_interval]))",
-          "legendFormat": "{{server}} | {{source}} | {{state}}"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "B",
-          "expr": "sum by (server, cycle_scope, source, state) (rustfs_scanner_cycle_source_work{job=~\"$job\",server=~\"$server\",source=~\"$scanner_source\"})",
-          "legendFormat": "{{server}} | {{cycle_scope}} | {{source}} | {{state}}"
-        }
-      ],
-      "title": "Scanner Source Work by Server",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${datasource}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisColorMode": "text",
-            "axisLabel": "",
-            "axisPlacement": "auto",
-            "barAlignment": 0,
-            "barWidthFactor": 0.6,
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "gradientMode": "none",
-            "hideFrom": {
-              "legend": false,
-              "tooltip": false,
-              "viz": false
-            },
-            "insertNulls": false,
-            "lineInterpolation": "smooth",
-            "lineWidth": 1,
-            "pointSize": 5,
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "showPoints": "never",
-            "showValues": false,
-            "spanNulls": false,
-            "stacking": {
-              "group": "A",
-              "mode": "none"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": 0
-              }
-            ]
-          },
-          "unit": "ops"
-        },
-        "overrides": [
-          {
-            "matcher": {
-              "id": "byFrameRefID",
-              "options": "B"
-            },
-            "properties": [
-              {
-                "id": "unit",
-                "value": "short"
-              },
-              {
-                "id": "custom.axisPlacement",
-                "value": "right"
-              }
-            ]
-          }
-        ]
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 349
-      },
-      "id": 536,
-      "options": {
-        "legend": {
-          "calcs": [
-            "lastNotNull",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "right",
-          "showLegend": true
-        },
-        "tooltip": {
-          "hideZeros": false,
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "pluginVersion": "12.3.2",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "A",
-          "expr": "sum by (server, bucket, drive, result) (rate(rustfs_scanner_bucket_drive_result_total{job=~\"$job\",server=~\"$server\",bucket=~\"$bucket\",drive=~\"$drive\",result=~\"$scanner_result\"}[$__rate_interval]))",
-          "legendFormat": "{{server}} | {{bucket}} | {{drive}} | {{result}}"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "B",
-          "expr": "sum by (server, cycle_scope, bucket, drive, result) (rustfs_scanner_cycle_bucket_drive_result{job=~\"$job\",server=~\"$server\",bucket=~\"$bucket\",drive=~\"$drive\",result=~\"$scanner_result\"})",
-          "legendFormat": "{{server}} | {{cycle_scope}} | {{bucket}} | {{drive}} | {{result}}"
-        }
-      ],
-      "title": "Scanner Bucket Drive Results",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${datasource}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisColorMode": "text",
-            "axisLabel": "",
-            "axisPlacement": "auto",
-            "barAlignment": 0,
-            "barWidthFactor": 0.6,
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "gradientMode": "none",
-            "hideFrom": {
-              "legend": false,
-              "tooltip": false,
-              "viz": false
-            },
-            "insertNulls": false,
-            "lineInterpolation": "smooth",
-            "lineWidth": 1,
-            "pointSize": 5,
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "showPoints": "never",
-            "showValues": false,
-            "spanNulls": false,
-            "stacking": {
-              "group": "A",
-              "mode": "none"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": 0
-              }
-            ]
-          },
-          "unit": "ops"
-        },
-        "overrides": [
-          {
-            "matcher": {
-              "id": "byFrameRefID",
-              "options": "B"
-            },
-            "properties": [
-              {
-                "id": "unit",
-                "value": "Bps"
-              }
-            ]
-          }
-        ]
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 349
-      },
-      "id": 537,
-      "options": {
-        "legend": {
-          "calcs": [
-            "lastNotNull",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "right",
-          "showLegend": true
-        },
-        "tooltip": {
-          "hideZeros": false,
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "pluginVersion": "12.3.2",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "A",
-          "expr": "sum by (bucket, target_arn) (rate(rustfs_bucket_replication_target_sent_count{job=~\"$job\",bucket=~\"$bucket\",target_arn=~\"$target_arn\"}[$__rate_interval]))",
-          "legendFormat": "sent objects | {{bucket}} | {{target_arn}}"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "B",
-          "expr": "sum by (bucket, target_arn) (rate(rustfs_bucket_replication_target_sent_bytes{job=~\"$job\",bucket=~\"$bucket\",target_arn=~\"$target_arn\"}[$__rate_interval]))",
-          "legendFormat": "sent bytes | {{bucket}} | {{target_arn}}"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "C",
-          "expr": "sum by (bucket, target_arn) (rate(rustfs_bucket_replication_target_total_failed_count{job=~\"$job\",bucket=~\"$bucket\",target_arn=~\"$target_arn\"}[$__rate_interval]))",
-          "legendFormat": "failed objects | {{bucket}} | {{target_arn}}"
-        }
-      ],
-      "title": "Bucket Replication Target Flow",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${datasource}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisColorMode": "text",
-            "axisLabel": "",
-            "axisPlacement": "auto",
-            "barAlignment": 0,
-            "barWidthFactor": 0.6,
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "gradientMode": "none",
-            "hideFrom": {
-              "legend": false,
-              "tooltip": false,
-              "viz": false
-            },
-            "insertNulls": false,
-            "lineInterpolation": "smooth",
-            "lineWidth": 1,
-            "pointSize": 5,
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "showPoints": "never",
-            "showValues": false,
-            "spanNulls": false,
-            "stacking": {
-              "group": "A",
-              "mode": "none"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "mappings": [],
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": 0
-              }
-            ]
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 357
-      },
-      "id": 538,
-      "options": {
-        "legend": {
-          "calcs": [
-            "lastNotNull",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "right",
-          "showLegend": true
-        },
-        "tooltip": {
-          "hideZeros": false,
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "pluginVersion": "12.3.2",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "A",
-          "expr": "max by (server, target_id) (rustfs_audit_target_queue_length_by_server{job=~\"$job\",server=~\"$server\"})",
-          "legendFormat": "audit queue | {{server}} | {{target_id}}"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${datasource}"
-          },
-          "editorMode": "code",
-          "range": true,
-          "refId": "B",
-          "expr": "max by (server, action, state) (rustfs_ilm_action_tasks{job=~\"$job\",server=~\"$server\"})",
-          "legendFormat": "ilm | {{server}} | {{action}} | {{state}}"
-        }
-      ],
-      "title": "Audit and ILM by Server",
-      "type": "timeseries"
     }
   ],
   "preload": false,
@@ -12376,32 +11551,6 @@
         "sort": 1,
         "type": "query"
       },
-      {
-        "allValue": ".*",
-        "current": {
-          "text": "All",
-          "value": "$__all"
-        },
-        "datasource": {
-          "type": "prometheus",
-          "uid": "${datasource}"
-        },
-        "definition": "label_values(rustfs_system_drive_api_calls_total,api)",
-        "includeAll": true,
-        "label": "Drive API",
-        "multi": true,
-        "name": "drive_api",
-        "options": [],
-        "query": {
-          "qryType": 1,
-          "query": "label_values(rustfs_system_drive_api_calls_total,api)",
-          "refId": "PrometheusVariableQueryEditor-drive_api"
-        },
-        "refresh": 2,
-        "regex": "",
-        "sort": 1,
-        "type": "query"
-      },
       {
         "allValue": ".*",
         "current": {
@@ -12521,136 +11670,6 @@
         "regex": "",
         "sort": 1,
         "type": "query"
-      },
-      {
-        "allValue": ".*",
-        "current": {
-          "text": "All",
-          "value": "$__all"
-        },
-        "datasource": {
-          "type": "prometheus",
-          "uid": "${datasource}"
-        },
-        "definition": "label_values(rustfs_api_requests_requests_total_by_server,server)",
-        "includeAll": true,
-        "label": "Server",
-        "multi": true,
-        "name": "server",
-        "options": [],
-        "query": {
-          "qryType": 1,
-          "query": "label_values(rustfs_api_requests_requests_total_by_server,server)",
-          "refId": "PrometheusVariableQueryEditor-server"
-        },
-        "refresh": 2,
-        "regex": "",
-        "sort": 1,
-        "type": "query"
-      },
-      {
-        "allValue": ".*",
-        "current": {
-          "text": "All",
-          "value": "$__all"
-        },
-        "datasource": {
-          "type": "prometheus",
-          "uid": "${datasource}"
-        },
-        "definition": "label_values(rustfs_api_requests_requests_total_by_server,name)",
-        "includeAll": true,
-        "label": "API",
-        "multi": true,
-        "name": "api",
-        "options": [],
-        "query": {
-          "qryType": 1,
-          "query": "label_values(rustfs_api_requests_requests_total_by_server,name)",
-          "refId": "PrometheusVariableQueryEditor-api"
-        },
-        "refresh": 2,
-        "regex": "",
-        "sort": 1,
-        "type": "query"
-      },
-      {
-        "allValue": ".*",
-        "current": {
-          "text": "All",
-          "value": "$__all"
-        },
-        "datasource": {
-          "type": "prometheus",
-          "uid": "${datasource}"
-        },
-        "definition": "label_values({__name__=\"rustfs_bucket_replication_target_sent_count\",bucket=~\"$bucket\"},target_arn)",
-        "includeAll": true,
-        "label": "Target ARN",
-        "multi": true,
-        "name": "target_arn",
-        "options": [],
-        "query": {
-          "qryType": 1,
-          "query": "label_values({__name__=\"rustfs_bucket_replication_target_sent_count\",bucket=~\"$bucket\"},target_arn)",
-          "refId": "PrometheusVariableQueryEditor-target_arn"
-        },
-        "refresh": 2,
-        "regex": "",
-        "sort": 1,
-        "type": "query"
-      },
-      {
-        "allValue": ".*",
-        "current": {
-          "text": "All",
-          "value": "$__all"
-        },
-        "datasource": {
-          "type": "prometheus",
-          "uid": "${datasource}"
-        },
-        "definition": "label_values(rustfs_scanner_source_work_total,source)",
-        "includeAll": true,
-        "label": "Scanner Source",
-        "multi": true,
-        "name": "scanner_source",
-        "options": [],
-        "query": {
-          "qryType": 1,
-          "query": "label_values(rustfs_scanner_source_work_total,source)",
-          "refId": "PrometheusVariableQueryEditor-scanner_source"
-        },
-        "refresh": 2,
-        "regex": "",
-        "sort": 1,
-        "type": "query"
-      },
-      {
-        "allValue": ".*",
-        "current": {
-          "text": "All",
-          "value": "$__all"
-        },
-        "datasource": {
-          "type": "prometheus",
-          "uid": "${datasource}"
-        },
-        "definition": "label_values(rustfs_scanner_bucket_drive_result_total,result)",
-        "includeAll": true,
-        "label": "Scanner Result",
-        "multi": true,
-        "name": "scanner_result",
-        "options": [],
-        "query": {
-          "qryType": 1,
-          "query": "label_values(rustfs_scanner_bucket_drive_result_total,result)",
-          "refId": "PrometheusVariableQueryEditor-scanner_result"
-        },
-        "refresh": 2,
-        "regex": "",
-        "sort": 1,
-        "type": "query"
       }
     ]
   },

.docker/observability/prometheus-rules/rustfs-kms-alerts.yml

@@ -1,214 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# =============================================================================
-# RustFS KMS backend — Prometheus alerting rules
-# =============================================================================
-#
-# Metric source: the KMS operation-policy choke point in
-# crates/kms/src/policy.rs. All label values are bounded static strings
-# (operation, op_class, outcome, error_class, backend, scope); key identifiers,
-# key material, and tokens never appear in labels.
-#
-# Response procedures: docs/operations/kms-observability-runbook.md
-#
-# IMPORTANT — threshold status: every numeric threshold below is a
-# conservative default chosen without a production baseline. Calibrate against
-# a staging baseline before relying on these alerts for paging, and prefer
-# loosening over tightening until the baseline exists. Formal SLO targets are
-# deliberately not encoded here (see rustfs/backlog#1584).
-#
-# NOTE: prometheus.yml loads /etc/prometheus/rules/*.yml — keep the .yml
-# extension or the file is silently ignored by the docker-compose stack.
-#
-# Validate: promtool check rules rustfs-kms-alerts.yml
-# =============================================================================
-
-groups:
-  # ==========================================================================
-  # Critical alerts — immediate action required
-  # ==========================================================================
-  - name: rustfs-kms-critical
-    interval: 30s
-    rules:
-      # ------------------------------------------------------------------
-      # 1. KmsBackendFatalErrors
-      #    Any attempt failure classified as fatal (non-retryable): auth
-      #    or permission errors, malformed requests, missing keys. The
-      #    policy never retries these, so even a low rate means real
-      #    operations are failing right now.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendFatalErrors
-        expr: |
-          sum by (operation) (rate(rustfs_kms_backend_attempt_failures_total{error_class="fatal"}[5m])) > 0
-        for: 5m
-        labels:
-          severity: critical
-          component: kms
-        annotations:
-          summary: "KMS backend fatal errors on operation {{ $labels.operation }}"
-          description: >-
-            Attempt failures classified as fatal are occurring at
-            {{ $value | printf "%.3f" }}/s on operation
-            {{ $labels.operation }}. Fatal failures are not retried:
-            each one is a KMS backend call that failed permanently
-            (authentication, permissions, malformed request, or a
-            missing key/version).
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendfatalerrors"
-
-      # ------------------------------------------------------------------
-      # 2. KmsBackendHighErrorRate
-      #    Sustained share of operations terminating without success
-      #    (fatal, budget/deadline exhaustion, admission backpressure,
-      #    or an open circuit). The cancelled outcome is excluded because
-      #    shutdowns legitimately produce it.
-      #    The traffic guard keeps a single failure on a near-idle
-      #    cluster from firing the alert.
-      #    Threshold: 5% for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendHighErrorRate
-        expr: |
-          (
-            sum(rate(rustfs_kms_backend_operations_total{outcome!~"success|cancelled"}[5m]))
-            /
-            clamp_min(sum(rate(rustfs_kms_backend_operations_total[5m])), 1e-9)
-          ) > 0.05
-          and
-          sum(rate(rustfs_kms_backend_operations_total[5m])) > 0.02
-        for: 10m
-        labels:
-          severity: critical
-          component: kms
-        annotations:
-          summary: "KMS backend non-success ratio above 5% for 10m"
-          description: >-
-            {{ $value | humanizePercentage }} of KMS backend operations
-            are terminating in fatal, budget_exhausted,
-            deadline_exceeded, backpressure_timeout,
-            backpressure_rejected, or circuit_open. Object encryption
-            and decryption paths depending on the KMS are degraded or
-            failing.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendhigherrorrate"
-
-  # ==========================================================================
-  # Warning alerts — investigation needed
-  # ==========================================================================
-  - name: rustfs-kms-warning
-    interval: 30s
-    rules:
-      # ------------------------------------------------------------------
-      # 3. KmsBackendP99LatencyHigh
-      #    p99 wall-clock duration of whole operations (attempts plus
-      #    backoff) is sustained above 2 seconds. Because the histogram
-      #    includes retries, a high p99 usually means the retry policy
-      #    is absorbing backend failures, not that every call is slow.
-      #    Threshold: 2s for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendP99LatencyHigh
-        expr: |
-          histogram_quantile(0.99,
-            sum by (le) (rate(rustfs_kms_backend_operation_duration_seconds_bucket[5m]))
-          ) > 2
-        for: 10m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend operation p99 latency above 2s for 10m"
-          description: >-
-            The 99th-percentile KMS backend operation duration is
-            {{ $value | humanizeDuration }}, including retries and
-            backoff. Encryption and decryption latency is leaking into
-            S3 request latency.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendp99latencyhigh"
-
-      # ------------------------------------------------------------------
-      # 4. KmsBackendAttemptFailureSpike
-      #    Aggregate attempt-failure rate (all error classes) sustained
-      #    above an absolute floor. An absolute threshold is used instead
-      #    of an offset-1d baseline ratio because fresh deployments have
-      #    no baseline and an empty offset vector would keep a ratio
-      #    alert from ever firing; switch to a baseline-relative form
-      #    (see rustfs-get-optimization-alerts.yaml for the pattern)
-      #    once a stable staging baseline exists.
-      #    Threshold: 0.5/s for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendAttemptFailureSpike
-        expr: |
-          sum(rate(rustfs_kms_backend_attempt_failures_total[5m])) > 0.5
-        for: 10m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend attempt failures above 0.5/s for 10m"
-          description: >-
-            KMS backend attempts are failing at
-            {{ $value | printf "%.2f" }}/s across all error classes.
-            The retry policy may still be masking these from callers —
-            check the error-class breakdown before it stops absorbing
-            them.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendattemptfailurespike"
-
-      # ------------------------------------------------------------------
-      # 5. KmsBackendRetryBudgetExhausted
-      #    Operations are running out of retry budget (budget_exhausted)
-      #    or operation deadline (deadline_exceeded). These surface to
-      #    callers as failed KMS operations even though every individual
-      #    failure was retryable — the backend is unhealthy for longer
-      #    than the policy can bridge.
-      #    Threshold: 0.05/s for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendRetryBudgetExhausted
-        expr: |
-          sum by (outcome) (rate(rustfs_kms_backend_operations_total{outcome=~"budget_exhausted|deadline_exceeded"}[5m])) > 0.05
-        for: 10m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend operations exhausting retry budget ({{ $labels.outcome }})"
-          description: >-
-            KMS backend operations are terminating as
-            {{ $labels.outcome }} at {{ $value | printf "%.3f" }}/s.
-            Retryable failures are outlasting the retry budget, so
-            callers are seeing hard failures.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendretrybudgetexhausted"
-
-      # ------------------------------------------------------------------
-      # 6. KmsBackendCircuitOpen
-      #    Direct circuit-state signal, independent of operation traffic.
-      #    A transient open can recover on its first half-open probe; alert
-      #    only when the circuit remains open or half-open for one minute.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendCircuitOpen
-        expr: |
-          rustfs_kms_backend_circuit_open > 0
-        for: 1m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend circuit open ({{ $labels.backend }}/{{ $labels.scope }})"
-          description: >-
-            The KMS backend circuit for {{ $labels.backend }} scope
-            {{ $labels.scope }} has remained open or half-open for one
-            minute. Operations in this scope can terminate as
-            circuit_open until the half-open probe succeeds or returns
-            a non-retryable failure.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendcircuitopen"

.github/actions/setup/action.yml

@@ -25,13 +25,9 @@ inputs:
     required: false
     default: "rustfs-deps"
   cache-save-if:
-    description: >-
-      Whether to save the cache. The fail-safe default is 'false': a caller that
-      wants to populate a cache must opt in explicitly, so a forgotten input
-      costs a cold cache (minutes) rather than silently consuming the
-      repository-wide 10GB Actions cache quota and evicting other lanes.
+    description: "Condition for saving cache"
     required: false
-    default: "false"
+    default: "true"
   install-cross-tools:
     description: "Install cross-compilation tools"
     required: false
@@ -40,43 +36,28 @@ inputs:
     description: "Target architecture to add"
     required: false
     default: ""
-  install-build-packaging-tools:
-    description: >-
-      Install musl-tools/zip/unzip, needed for musl linking and release
-      packaging. Off for CI test lanes, which use none of them.
+  github-token:
+    description: "GitHub token for API access"
     required: false
-    default: "true"
-  install-test-tools:
-    description: >-
-      Install cargo-nextest and the rustfmt/clippy components. Off for release
-      and audit lanes, which run no tests and no lints.
-    required: false
-    default: "true"
+    default: ""
 
 runs:
   using: "composite"
   steps:
-    # protobuf-compiler is deliberately absent: the setup-protoc step below
-    # installs 34.1 into the tool cache and prepends it to PATH, so the apt
-    # build (older, and never version-matched) was shadowed on every run and
-    # simply never used.
     - name: Install system dependencies (Ubuntu)
       if: runner.os == 'Linux'
       shell: bash
       run: |
         sudo apt-get update
         sudo apt-get install -y \
+          musl-tools \
           build-essential \
           pkg-config \
           libssl-dev \
-          ripgrep
-
-    # musl-gcc is needed by the native musl release leg, and zip/unzip by the
-    # release packaging steps. No CI test lane touches any of them.
-    - name: Install packaging and cross-linking dependencies (Ubuntu)
-      if: runner.os == 'Linux' && inputs.install-build-packaging-tools == 'true'
-      shell: bash
-      run: sudo apt-get install -y musl-tools zip unzip
+          ripgrep \
+          unzip \
+          zip \
+          protobuf-compiler
 
     - name: Install protoc
       uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
@@ -94,7 +75,7 @@ runs:
       with:
         toolchain: ${{ inputs.rust-version }}
         targets: ${{ inputs.target }}
-        components: ${{ inputs.install-test-tools == 'true' && 'rustfmt, clippy' || '' }}
+        components: rustfmt, clippy
 
     - name: Install Zig
       if: inputs.install-cross-tools == 'true'
@@ -105,24 +86,12 @@ runs:
       uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild
 
     - name: Install cargo-nextest
-      if: inputs.install-test-tools == 'true'
       uses: taiki-e/install-action@96c7780c1d8a2b8723e12031def873a434d39d8d # nextest
 
     - name: Setup Rust cache
       uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
       with:
-        # false is rust-cache's own default. With true, cleanup.ts returns
-        # *before* pruning ~/.cargo/registry/src, and config.ts archives the
-        # whole registry — so every cache carried the unpacked source tree of
-        # every dependency, not just "a few extra crates".
-        #
-        # No coverage is lost: getPackages runs `cargo metadata --all-features`,
-        # a strict superset of any single lane's feature closure, and -sys crates
-        # are explicitly exempted from pruning (their src timestamps would
-        # otherwise trigger rebuilds). Anything pruned is re-unpacked from the
-        # .crate files still in registry/cache, whose mtimes crates.io
-        # normalises, so cargo fingerprints stay valid.
-        cache-all-crates: false
+        cache-all-crates: true
         cache-on-failure: true
         shared-key: ${{ inputs.cache-shared-key }}
         save-if: ${{ inputs.cache-save-if }}

.github/workflows/architecture-migration-rules.yml

@@ -37,7 +37,6 @@ jobs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request' && github.event.action == 'closed'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Explain cancellation run
         run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -46,11 +45,8 @@ jobs:
     name: Architecture Migration Rules
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Install ripgrep
         run: |

.github/workflows/audit.yml

@@ -23,9 +23,6 @@ on:
       - 'deny.toml'
       - '.github/actions/**'
       - '.github/workflows/**'
-      - 'scripts/release/create_or_update_release.sh'
-      - 'scripts/security/check_performance_ab_workflow.sh'
-      - 'scripts/security/check_preview_release_workflow.sh'
       - 'scripts/security/check_workflow_pins.sh'
   pull_request:
     types: [ opened, synchronize, reopened, closed ]
@@ -36,17 +33,9 @@ on:
       - 'deny.toml'
       - '.github/actions/**'
       - '.github/workflows/**'
-      - 'scripts/release/create_or_update_release.sh'
-      - 'scripts/security/check_performance_ab_workflow.sh'
-      - 'scripts/security/check_preview_release_workflow.sh'
       - 'scripts/security/check_workflow_pins.sh'
   schedule:
-    # Daily, not weekly. This schedule exists to catch RustSec advisories
-    # published against an unchanged dependency tree; at weekly cadence a new
-    # advisory could sit unnoticed for seven days. The check list is unchanged —
-    # splitting it into a light daily advisories-only run and a weekly full run
-    # would create runs where sources/bans/licenses go unverified.
-    - cron: '0 3 * * *' # Daily 03:00 UTC (staggered after the midnight ci/build crons)
+    - cron: '0 3 * * 0' # Weekly on Sunday 03:00 UTC (staggered after the midnight ci/build crons)
   workflow_dispatch:
 
 permissions:
@@ -66,7 +55,6 @@ jobs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request' && github.event.action == 'closed'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Explain cancellation run
         run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -82,32 +70,11 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
-      # cargo-deny compiles nothing, so the full setup composite (apt packages,
-      # protoc, flatc, nextest, rustfmt/clippy) was pure overhead here. It does
-      # still need a real cargo: `cargo deny check` runs `cargo metadata`, and
-      # Cargo.toml pins datafusion and s3s as git dependencies, which must be
-      # materialised into ~/.cargo/git — a cold clone is hundreds of MB, so the
-      # cache stays.
-      - name: Install Rust toolchain
-        uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
-
-      # Was relying on the composite's default, which used to be "true": every
-      # PR touching Cargo.toml/Cargo.lock saved a second, PR-scoped copy of this
-      # cache and pushed the main-scoped lanes out of the 10GB quota. The
-      # default is now "false", but state it explicitly — see
-      # scripts/security/check_cache_save_if.sh.
-      - name: Setup Rust cache
-        uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
         with:
-          # Same reasoning as the setup composite: true archives every
-          # dependency's unpacked source tree.
-          cache-all-crates: false
-          cache-on-failure: true
-          shared-key: rustfs-cargo-deny
-          save-if: ${{ github.ref == 'refs/heads/main' }}
+          cache-shared-key: rustfs-cargo-deny
 
       - name: Install cargo-deny
         uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
@@ -125,31 +92,13 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Report unpinned GitHub Actions
         run: ./scripts/security/check_workflow_pins.sh --enforce
 
-      - name: Check setup cache-save-if is explicit
-        run: ./scripts/security/check_cache_save_if.sh
-
-      - name: Check every job declares a timeout
-        run: ./scripts/security/check_job_timeouts.sh
-
-      - name: Check checkouts clear their credentials
-        run: ./scripts/security/check_persist_credentials.sh
-
-      - name: Check preview release workflow policy
-        run: ./scripts/security/check_preview_release_workflow.sh
-
-      - name: Check performance A/B workflow trust boundary
-        run: ./scripts/security/check_performance_ab_workflow.sh
-
   dependency-review:
     name: Dependency Review
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     if: github.event_name == 'pull_request' && github.event.action != 'closed'
     permissions:
       contents: read
@@ -157,8 +106,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Dependency Review
         uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5
@@ -171,28 +118,3 @@ jobs:
           # conscious re-review of the license/provenance claim (backlog#1181).
           allow-dependencies-licenses: pkg:cargo/rustfs-uring@0.1.0
           comment-summary-in-pr: always
-
-  alert-on-failure:
-    name: Alert on scheduled failure
-    # dependency-review is deliberately excluded: it only runs on pull_request,
-    # so it can never contribute a failure to a scheduled run.
-    needs: [cargo-deny, workflow-pin-report]
-    # A scheduled cargo-deny failure usually means the dependency tree just
-    # matched a newly published advisory — the single most important signal this
-    # workflow produces, and until now it was only visible to whoever happened to
-    # open the Actions tab. Same ci-8 mechanism coverage.yml and
-    # e2e-replication-nightly.yml already use.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-      - name: Open or update failure-tracking issue
-        uses: ./.github/actions/schedule-failure-issue
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/build.yml

@@ -50,18 +50,12 @@ on:
       - "**/*.svg"
       - ".gitignore"
       - ".dockerignore"
-      - "flake.lock"
   schedule:
     - cron: "0 1 * * 0" # Weekly on Sunday 01:00 UTC (staggered after the ci.yml midnight cron)
   workflow_dispatch:
     inputs:
       build_docker:
-        # Advisory only. docker.yml triggers on workflow_run and its job-level
-        # condition requires the triggering event to be a tag push, so a manual
-        # dispatch of this workflow never produces images regardless of this
-        # value. Kept because the summary step reports it; wiring it up would
-        # mean teaching docker.yml's version parser a second event shape.
-        description: "Build and push Docker images after binary build (ignored: dispatch runs never reach docker.yml)"
+        description: "Build and push Docker images after binary build"
         required: false
         default: true
         type: boolean
@@ -89,7 +83,6 @@ jobs:
   build-check:
     name: Build Strategy Check
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
       build_type: ${{ steps.check.outputs.build_type }}
@@ -99,8 +92,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Determine build strategy
         id: check
@@ -116,21 +107,13 @@ jobs:
 
           # Determine build type based on trigger
           if [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
-            # Tag push - preview, release, or prerelease
+            # Tag push - release or prerelease
             should_build=true
             tag_name="${GITHUB_REF#refs/tags/}"
             version="${tag_name}"
 
-            # Preview tags publish a GitHub prerelease for validation, but
-            # must not update any latest channel.
-            if [[ "$tag_name" =~ -preview\.[0-9]+$ ]]; then
-              build_type="preview"
-              is_prerelease=true
-              echo "🔍 Preview build detected: $tag_name"
-            elif [[ "$tag_name" == *"-preview"* ]]; then
-              echo "❌ Invalid preview tag: $tag_name (expected suffix: -preview.<number>)" >&2
-              exit 1
-            elif [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
+            # Check if this is a prerelease
+            if [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
               build_type="prerelease"
               is_prerelease=true
               echo "🚀 Prerelease build detected: $tag_name"
@@ -173,7 +156,6 @@ jobs:
     name: Prepare Platform Matrix
     needs: build-check
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     outputs:
       matrix: ${{ steps.select.outputs.matrix }}
       selected: ${{ steps.select.outputs.selected }}
@@ -181,14 +163,10 @@ jobs:
       - name: Select target platforms
         id: select
         shell: bash
-        env:
-          # via env, not interpolation: a dispatch input is free-form text and
-          # would otherwise be pasted into the script for bash to evaluate.
-          RAW_PLATFORMS: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}
         run: |
           set -euo pipefail
 
-          selected="$RAW_PLATFORMS"
+          selected="${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}"
           selected="$(echo "${selected}" | tr -d '[:space:]')"
           if [[ -z "${selected}" ]]; then
             selected="all"
@@ -259,7 +237,6 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
         with:
-          persist-credentials: false
           fetch-depth: 0
 
       - name: Setup Rust environment
@@ -268,17 +245,9 @@ jobs:
           rust-version: stable
           target: ${{ matrix.target }}
           cache-shared-key: build-${{ matrix.target }}
-          # main only. A cache saved on refs/tags/X is scoped to that tag: no
-          # other tag, no main run and no PR can restore it, so every release
-          # cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag,
-          # six legs each) that nobody could read, evicting the hot lanes from
-          # the repo-wide 10GB quota. Tag builds still restore the main-scoped
-          # cache, since default-branch caches are readable from every ref.
-          # The one real cost: re-running a failed leg of the same tag no longer
-          # finds that tag's own warm cache and falls back to main's.
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/') }}
           install-cross-tools: ${{ matrix.cross }}
-          install-test-tools: 'false'
 
       - name: Download static console assets
         shell: bash
@@ -725,14 +694,9 @@ jobs:
     needs: [ build-check, build-rustfs ]
     if: always() && needs.build-check.outputs.should_build == 'true'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Build completion summary
         shell: bash
-        env:
-          # dispatch input via env: free-form text must not be pasted into the
-          # script for bash to evaluate.
-          INPUT_BUILD_DOCKER: ${{ github.event.inputs.build_docker }}
         run: |
           BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
           VERSION="${{ needs.build-check.outputs.version }}"
@@ -750,10 +714,6 @@ jobs:
           echo ""
 
           case "$BUILD_TYPE" in
-            "preview")
-              echo "🔍 Preview artifacts are published in a GitHub prerelease"
-              echo "⏭️  Preview releases do not update latest channels"
-              ;;
             "development")
               echo "🛠️  Development build artifacts have been uploaded to OSS dev directory"
               echo "⚠️  This is a development build - not suitable for production use"
@@ -772,9 +732,7 @@ jobs:
 
           echo ""
           echo "🐳 Docker Images:"
-          if [[ "$BUILD_TYPE" == "preview" ]]; then
-            echo "⏭️  Preview tags do not publish Docker images"
-          elif [[ "$INPUT_BUILD_DOCKER" == "false" ]]; then
+          if [[ "${{ github.event.inputs.build_docker }}" == "false" ]]; then
             echo "⏭️  Docker image build was skipped (binary only build)"
           elif [[ "$BUILD_STATUS" == "success" ]]; then
             echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
@@ -782,13 +740,12 @@ jobs:
             echo "❌ Docker image build will be skipped due to build failure"
           fi
 
-  # Create GitHub Release for every valid release tag, including previews
+  # Create GitHub Release (only for tag pushes)
   create-release:
     name: Create GitHub Release
     needs: [ build-check, build-rustfs ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     permissions:
       contents: write
     outputs:
@@ -798,7 +755,6 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
         with:
-          persist-credentials: false
           fetch-depth: 0
 
       - name: Create GitHub Release
@@ -811,12 +767,9 @@ jobs:
           VERSION="${{ needs.build-check.outputs.version }}"
           IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
           BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
-          TARGET_COMMITISH=$(git rev-parse --verify "refs/tags/${TAG}^{commit}")
 
           # Determine release type for title
-          if [[ "$BUILD_TYPE" == "preview" ]]; then
-            RELEASE_TYPE="preview"
-          elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
+          if [[ "$BUILD_TYPE" == "prerelease" ]]; then
             if [[ "$TAG" == *"alpha"* ]]; then
               RELEASE_TYPE="alpha"
             elif [[ "$TAG" == *"beta"* ]]; then
@@ -830,34 +783,61 @@ jobs:
             RELEASE_TYPE="release"
           fi
 
-          # Create release title
-          if [[ "$IS_PRERELEASE" == "true" ]]; then
-            TITLE="RustFS $VERSION (${RELEASE_TYPE})"
+          # Check if release already exists
+          if gh release view "$TAG" >/dev/null 2>&1; then
+            echo "Release $TAG already exists"
+            RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
+            RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
           else
-            TITLE="RustFS $VERSION"
+            # Get release notes from tag message
+            RELEASE_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
+            if [[ -z "$RELEASE_NOTES" || "$RELEASE_NOTES" =~ ^[[:space:]]*$ ]]; then
+              if [[ "$IS_PRERELEASE" == "true" ]]; then
+                RELEASE_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
+              else
+                RELEASE_NOTES="Release ${VERSION}"
+              fi
+            fi
+
+            # Create release title
+            if [[ "$IS_PRERELEASE" == "true" ]]; then
+              TITLE="RustFS $VERSION (${RELEASE_TYPE})"
+            else
+              TITLE="RustFS $VERSION"
+            fi
+
+            # Create the release
+            PRERELEASE_FLAG=""
+            if [[ "$IS_PRERELEASE" == "true" ]]; then
+              PRERELEASE_FLAG="--prerelease"
+            fi
+
+            gh release create "$TAG" \
+              --title "$TITLE" \
+              --notes "$RELEASE_NOTES" \
+              $PRERELEASE_FLAG \
+              --draft
+
+            RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
+            RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
           fi
 
-          ./scripts/release/create_or_update_release.sh \
-            "$TAG" \
-            "$TARGET_COMMITISH" \
-            "$TITLE" \
-            "$IS_PRERELEASE"
+          echo "release_id=$RELEASE_ID" >> "$GITHUB_OUTPUT"
+          echo "release_url=$RELEASE_URL" >> "$GITHUB_OUTPUT"
+          echo "Created release: $RELEASE_URL"
 
   # Prepare and upload release assets
   upload-release-assets:
     name: Upload Release Assets
     needs: [ build-check, build-rustfs, create-release ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     permissions:
       contents: write
       actions: read
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Download all build artifacts
         uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -940,10 +920,9 @@ jobs:
   # the pointed-to version is a prerelease.
   update-latest-version:
     name: Update Latest Version
-    needs: [ build-check, publish-release ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    needs: [ build-check, upload-release-assets ]
+    if: startsWith(github.ref, 'refs/tags/')
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     steps:
       - name: Update latest.json
         env:
@@ -1001,34 +980,51 @@ jobs:
   publish-release:
     name: Publish Release
     needs: [ build-check, create-release, upload-release-assets ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     permissions:
       contents: write
     steps:
-      - name: Publish release
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Update release notes and publish
         env:
           GH_TOKEN: ${{ github.token }}
         shell: bash
         run: |
           TAG="${{ needs.build-check.outputs.version }}"
+          VERSION="${{ needs.build-check.outputs.version }}"
+          IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
           BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
-          RELEASE_ID="${{ needs.create-release.outputs.release_id }}"
 
-          # Publish the release and correct its channel state on retries.
-          # Only a stable final release may become GitHub Latest.
-          if [[ "$BUILD_TYPE" == "release" ]]; then
-            gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-              -F draft=false \
-              -F prerelease=false \
-              -f make_latest=true >/dev/null
+          # Determine release type
+          if [[ "$BUILD_TYPE" == "prerelease" ]]; then
+            if [[ "$TAG" == *"alpha"* ]]; then
+              RELEASE_TYPE="alpha"
+            elif [[ "$TAG" == *"beta"* ]]; then
+              RELEASE_TYPE="beta"
+            elif [[ "$TAG" == *"rc"* ]]; then
+              RELEASE_TYPE="rc"
+            else
+              RELEASE_TYPE="prerelease"
+            fi
           else
-            gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-              -F draft=false \
-              -F prerelease=true \
-              -f make_latest=false >/dev/null
+            RELEASE_TYPE="release"
           fi
 
+          # Get original release notes from tag
+          ORIGINAL_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
+          if [[ -z "$ORIGINAL_NOTES" || "$ORIGINAL_NOTES" =~ ^[[:space:]]*$ ]]; then
+            if [[ "$IS_PRERELEASE" == "true" ]]; then
+              ORIGINAL_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
+            else
+              ORIGINAL_NOTES="Release ${VERSION}"
+            fi
+          fi
+
+          # Publish the release (remove draft status)
+          gh release edit "$TAG" --draft=false
+
           echo "🎉 Released $TAG successfully!"
           echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"

.github/workflows/cache-warm.yml

@@ -1,265 +0,0 @@
-# Copyright 2026 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Sole writer of the Rust dependency caches that ci.yml restores.
-#
-# Why this is a separate workflow rather than steps inside ci.yml: ci.yml's
-# concurrency group cancels in-progress runs on main pushes, and merges land far
-# faster than its 70-minute pipeline. Measured over 15 consecutive main pushes:
-# 12 cancelled, 2 failed, 0 succeeded. A cancelled run never reaches
-# Swatinem/rust-cache's post step (cache-on-failure does not cover cancellation),
-# so the writer lanes were saving nothing and every PR paid a cold restore —
-# 11.8-20.9 minutes of "Setup Rust environment" against 0.7-3.4 warm.
-#
-# Splitting cache writing out of the test pipeline lets ci.yml keep cancelling
-# superseded runs (which is correct — nobody needs test results for a commit
-# that is already three merges behind) while the caches still get written.
-#
-# The group below deliberately does NOT cancel in progress; see the comment on
-# it for how that bounds concurrency and why it is scoped by event.
-#
-# Each job below owns exactly one shared-key and is the only place that sets
-# cache-save-if to anything but 'false' for it; every lane in ci.yml reads.
-# scripts/security/check_cache_save_if.sh keeps the declarations explicit.
-#
-# The builds are supersets of what the reading lanes compile, because a reader
-# restores only what the writer saved. Feature resolution matters here: a lane
-# built with e2e-test-hooks resolves dependency features differently, which
-# changes -Cmetadata, so the plain build does not cover it. See
-# rustfs/backlog#1600.
-
-name: Cache Warm
-
-on:
-  push:
-    branches: [ main ]
-    # Mirrors ci.yml's push paths-ignore: if a commit cannot change what ci.yml
-    # compiles, it cannot change what ci.yml needs restored either.
-    paths-ignore:
-      - "**.md"
-      - "docs/**"
-      - "deploy/**"
-      - "scripts/dev_*.sh"
-      - "scripts/probe.sh"
-      - "LICENSE*"
-      - ".gitignore"
-      - ".dockerignore"
-      - "README*"
-      - "**/*.png"
-      - "**/*.jpg"
-      - "**/*.svg"
-      - ".github/workflows/build.yml"
-      - ".github/workflows/docker.yml"
-      - ".github/workflows/audit.yml"
-  workflow_dispatch:
-    inputs:
-      emit_timings:
-        description: >-
-          Also emit cargo --timings for the ci-dev build and upload it. Used to
-          decide whether sccache is worth adopting (rustfs/backlog#1601 gate).
-        required: false
-        default: false
-        type: boolean
-
-permissions:
-  contents: read
-
-# Scoped by event. A push run and a dispatch run do not compete: GitHub keeps
-# one running plus one pending per group, so with a single shared group a
-# manually dispatched run was displaced as pending by the next merge and
-# cancelled — observed three times in a row, which made the --timings gate in
-# rustfs/backlog#1601 effectively impossible to trigger while main was busy.
-#
-# Still no cancel-in-progress: a burst of merges collapses into "current run
-# finishes, newest queued run follows" rather than a pile-up, which is what
-# bounds this workflow to one self-hosted runner per event type.
-#
-# The two paths can now overlap and race to save the same key. That is benign:
-# the loser finds the key already present and skips, and both builds produce the
-# same artifacts from the same commit.
-concurrency:
-  group: cache-warm-${{ github.event_name }}
-  cancel-in-progress: false
-
-env:
-  CARGO_TERM_COLOR: always
-
-jobs:
-  # Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary,
-  # e2e-tests, e2e-full.
-  warm-ci-dev:
-    name: Warm ci-dev
-    runs-on: sm-standard-4
-    timeout-minutes: 90
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-dev
-          cache-save-if: 'true'
-          install-build-packaging-tools: 'false'
-
-      # rustfs/backlog#1601 gate. sccache can only cache compilation units whose
-      # --emit includes link, so it covers workspace rlibs and nothing else:
-      # clippy is metadata-only, and the ~100 test binaries, the rustfs bin and
-      # every build script invoke the system linker. Before spending a bucket,
-      # credentials and a supply-chain boundary on it, measure how much of the
-      # build is actually rlib codegen.
-      #
-      # Read from the report: workspace lib codegen as a share of the build, and
-      # s3select-query's own rlib as a share. The plan adopts sccache only above
-      # 50% and 25% respectively; if linking dominates instead, the answer is
-      # mold/lld plus split-debuginfo, which is exactly the part sccache cannot
-      # touch. Off by default — this doubles the ci-dev build.
-      - name: Build ci-dev superset (with --timings)
-        if: inputs.emit_timings
-        env:
-          CARGO_BUILD_JOBS: "2"
-        run: cargo build --workspace --all-targets --timings
-
-      - name: Upload cargo timings report
-        if: inputs.emit_timings
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: cargo-timings-ci-dev
-          path: target/cargo-timings/
-          retention-days: 30
-          if-no-files-found: error
-
-      # --all-targets covers the test binaries nextest builds, including
-      # e2e_test, which test-and-lint's own run excludes. The second build adds
-      # the e2e-test-hooks feature resolution that build-rustfs-debug-binary uses
-      # and that no lint lane enables.
-      - name: Build ci-dev superset
-        env:
-          # Same limit ci.yml puts on its nextest step: this builds the same
-          # ~100 workspace test binaries, and three concurrent links saturate the
-          # self-hosted runner's overlay I/O and can wedge Cargo (#5394).
-          CARGO_BUILD_JOBS: "2"
-        run: |
-          cargo build --workspace --all-targets
-          cargo build -p rustfs --bins --features e2e-test-hooks
-
-      # Runs before rust-cache's post step, so these are the sizes it is about
-      # to archive. Reported so the cache-all-crates decision stays evidence-led:
-      # registry/src is what that flag prunes, registry/cache is what the pruned
-      # sources are re-unpacked from. See rustfs/backlog#1600.
-      - name: Report cache input sizes
-        if: always()
-        run: |
-          # tee, not a plain redirect: sent only to $GITHUB_STEP_SUMMARY these
-          # numbers are readable in the UI but absent from the job log, and the
-          # REST API exposes the log, not the summary — which made the figures
-          # unreachable for exactly the scripted comparison they exist for.
-          sizes="$(du -sh ~/.cargo/registry/src ~/.cargo/registry/cache \
-                          ~/.cargo/registry/index ~/.cargo/git target 2>/dev/null || true)"
-          echo "cache-input-sizes-begin"
-          printf '%s\n' "$sizes"
-          echo "cache-input-sizes-end"
-          {
-            echo "### Cache input sizes (ci-dev)"
-            echo '```'
-            printf '%s\n' "$sizes"
-            echo '```'
-          } >> "$GITHUB_STEP_SUMMARY"
-
-  # Readers: test-and-lint-rio-v2, build-rustfs-debug-binary-rio-v2.
-  warm-ci-feat-rio:
-    name: Warm ci-feat-rio
-    runs-on: sm-standard-4
-    timeout-minutes: 90
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-feat-rio
-          cache-save-if: 'true'
-          install-build-packaging-tools: 'false'
-
-      - name: Build ci-feat-rio superset
-        run: |
-          cargo build -p rustfs -p rustfs-ecstore --all-targets --features rio-v2
-          cargo build -p rustfs --bins --features rio-v2,e2e-test-hooks
-
-  # Readers: the swift and sftp legs of test-and-lint-protocols. Built in
-  # sequence rather than as `--features swift,sftp`, which is a combination no
-  # lane actually compiles; running both leaves the union in target/.
-  warm-ci-feat-proto:
-    name: Warm ci-feat-proto
-    runs-on: sm-standard-4
-    timeout-minutes: 90
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-feat-proto
-          cache-save-if: 'true'
-          install-build-packaging-tools: 'false'
-
-      - name: Build ci-feat-proto superset
-        run: |
-          cargo build -p rustfs -p rustfs-protocols --all-targets --features swift
-          cargo build -p rustfs -p rustfs-protocols --all-targets --features sftp
-
-  # Reader: uring-integration. Runs on ubuntu-latest to match it: rust-cache's
-  # key covers runner.os and arch but not the runner label or image, so a cache
-  # written on sm-standard-4 would be restored by the hosted runner as if it
-  # belonged to it.
-  warm-ci-uring:
-    name: Warm ci-uring
-    runs-on: ubuntu-latest
-    timeout-minutes: 60
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-uring
-          cache-save-if: 'true'
-          install-build-packaging-tools: 'false'
-
-      - name: Install build dependencies
-        run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
-
-      - name: Build ci-uring superset
-        run: cargo build -p rustfs-ecstore --all-targets

.github/workflows/ci-docs-only.yml

@@ -12,24 +12,18 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Companion to ci.yml for required status checks.
+# Companion to ci.yml for the required "Test and Lint" status check.
 #
-# ci.yml skips docs-only pull requests via paths-ignore, but the branch ruleset
-# requires a check named "Test and Lint" — without this workflow a docs-only PR
-# would wait on it forever. This workflow triggers on exactly the paths ci.yml
-# ignores and reports success under the same job name. Mixed PRs trigger both
-# workflows and the real check still gates: a required check with any failing
-# run blocks the merge.
+# ci.yml skips docs-only pull requests via paths-ignore, but the branch
+# ruleset requires a check named "Test and Lint" — without this workflow a
+# docs-only PR would wait on that check forever. This workflow triggers on
+# exactly the paths ci.yml ignores and reports an instant success under the
+# same job name. Mixed PRs trigger both workflows and the real check still
+# gates: a required check with any failing run blocks the merge.
 # https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/troubleshooting-required-status-checks#handling-skipped-but-required-checks
 #
-# "Quick Checks" is mirrored here ahead of the ruleset change that will make it
-# required too (rustfs/backlog#1599). Until that change lands this job is
-# inert; mirroring it first is what lets the ruleset change happen without
-# stranding docs-only PRs on a check nobody reports.
-#
 # Keep the paths list below in sync with the pull_request paths-ignore list
-# in ci.yml, and keep the quick-checks steps below byte-identical to the
-# quick-checks job in ci.yml.
+# in ci.yml.
 
 name: Continuous Integration (docs only)
 
@@ -53,82 +47,17 @@ on:
       - ".github/workflows/build.yml"
       - ".github/workflows/docker.yml"
       - ".github/workflows/audit.yml"
-      - "flake.lock"
 
 permissions:
   contents: read
 
 jobs:
-  # Deliberately NOT a bare `echo`. Once "Quick Checks" becomes a required
-  # check, ci.yml gates every expensive job behind it, so a mixed PR reports
-  # two check runs with this name: the real one (45-51s) and this companion.
-  # GitHub has no written contract for how it picks between same-named
-  # required check runs ("latest wins" vs "any failure blocks"), so instead of
-  # relying on ordering we make both runs execute the same commands against
-  # the same merge ref — their conclusions are then necessarily identical and
-  # the choice does not matter. Keep these steps byte-identical to the
-  # quick-checks job in ci.yml (a guard script that asserts this, and the paths
-  # sync below, is tracked in rustfs/backlog#1603).
-  #
-  # For a genuinely docs-only PR this adds no strictness (no code changed, so
-  # fmt and the guards always pass) and costs ~50s of ubuntu-latest.
-  quick-checks:
-    name: Quick Checks
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Install ripgrep
-        run: sudo apt-get update && sudo apt-get install -y ripgrep
-
-      - name: Install Rust toolchain
-        uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
-        with:
-          components: rustfmt
-
-      - name: Check code formatting
-        run: cargo fmt --all --check
-
-      - name: Check unsafe code allowances
-        run: ./scripts/check_unsafe_code_allowances.sh
-
-      - name: Check layered dependencies
-        run: ./scripts/check_layer_dependencies.sh
-
-      - name: Check architecture migration rules
-        run: ./scripts/check_architecture_migration_rules.sh
-
-      - name: Check tokio io-uring feature guard
-        run: ./scripts/check_no_tokio_io_uring.sh
-
-      - name: Check extension schema boundaries
-        run: ./scripts/check_extension_schema_boundaries.sh
-
-      - name: Check body-cache whitelist guard
-        run: ./scripts/check_body_cache_whitelist.sh
-
-      - name: Check no planning docs committed
-        run: ./scripts/check_no_planning_docs.sh
-
-      - name: Check CI paths stay in sync
-        run: ./scripts/check_ci_paths_sync.sh
-
-      - name: Check io_uring lane --lib precondition
-        run: ./scripts/check_uring_lane_lib_only.sh
-
   test-and-lint:
     name: Test and Lint
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       # Docs-only PRs skip the full code CI, but they are exactly where a
       # planning-type document could be slipped in (git add -f bypasses

.github/workflows/ci.yml

@@ -33,7 +33,6 @@ on:
       - ".github/workflows/build.yml"
       - ".github/workflows/docker.yml"
       - ".github/workflows/audit.yml"
-      - "flake.lock"
   pull_request:
     types: [ opened, synchronize, reopened, closed ]
     branches: [ main ]
@@ -55,7 +54,6 @@ on:
       - ".github/workflows/build.yml"
       - ".github/workflows/docker.yml"
       - ".github/workflows/audit.yml"
-      - "flake.lock"
   merge_group:
     types: [ checks_requested ]
   schedule:
@@ -83,7 +81,6 @@ jobs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request' && github.event.action == 'closed'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Explain cancellation run
         run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -92,20 +89,13 @@ jobs:
     name: Typos
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
       - name: Typos check with custom config file
         uses: crate-ci/typos@37bb98842b0d8c4ffebdb75301a13db0267cef89 # master
 
   # Fast, compile-free checks that fail early so contributors get feedback in
   # ~1 minute instead of waiting for the full test job.
-  #
-  # These steps are mirrored byte-for-byte in ci-docs-only.yml so that a mixed
-  # PR, which reports two check runs named "Quick Checks", cannot get one red
-  # and one green. Edit both jobs together.
   quick-checks:
     name: Quick Checks
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
@@ -114,8 +104,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Install ripgrep
         run: sudo apt-get update && sudo apt-get install -y ripgrep
@@ -149,48 +137,24 @@ jobs:
       - name: Check no planning docs committed
         run: ./scripts/check_no_planning_docs.sh
 
-      - name: Check CI paths stay in sync
-        run: ./scripts/check_ci_paths_sync.sh
-
-      - name: Check io_uring lane --lib precondition
-        run: ./scripts/check_uring_lane_lib_only.sh
-
   test-and-lint:
     name: Test and Lint
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    needs: [ quick-checks ]
     runs-on: sm-standard-4
     timeout-minutes: 90
-    # Both lines are required. Job-level `permissions` replaces the workflow
-    # block rather than merging with it, so declaring only `actions: write`
-    # would drop `contents: read` and break this job's checkout and the
-    # repo-token the setup action hands to setup-protoc.
-    permissions:
-      contents: read
-      actions: write
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          # This job's token can cancel runs and delete Actions caches. Checkout
-          # otherwise writes it into .git/config, where a PR's own build.rs or
-          # proc-macro could read it back out.
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          # Every lane in this workflow reads its cache and none writes it.
-          # cache-warm.yml is the sole writer for all four keys: this workflow
-          # cancels superseded runs on main, and a cancelled run never reaches
-          # rust-cache's post step, so writing from here saved nothing (12 of 15
-          # consecutive main-push runs were cancelled). See rustfs/backlog#1600.
-          cache-shared-key: ci-dev
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
+          cache-shared-key: ci-test
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
       - name: Prepare test evidence
         run: |
@@ -205,54 +169,43 @@ jobs:
       # Clippy runs before the test pass: lint failures are the most common
       # CI-only breakage and should surface in minutes, not after 20+ minutes
       # of tests.
-      # Sampled too: clippy is the natural control arm for any CARGO_BUILD_JOBS
-      # experiment, since --all-targets is check-only for workspace members and
-      # never links the ~100 test binaries the limit exists to throttle.
       - name: Run clippy lints
-        run: |
-          ./scripts/ci/resource_sampler.sh start clippy
-          trap './scripts/ci/resource_sampler.sh stop' EXIT
-          cargo clippy --all-targets -- -D warnings
+        run: cargo clippy --all-targets -- -D warnings
 
       - name: Run nextest tests
         env:
-          # #5394 mitigation, now under a measured experiment (backlog#1601).
-          #
-          # 2 was chosen when three concurrent workspace test links were believed
-          # to saturate the runner's overlay I/O and wedge Cargo until the 75m
-          # timeout. cgroup v2 readings from the sampler show the pod actually
-          # has 14 CPUs and 28GB (peak use 2.1GB), so 2 throttles compilation to
-          # a seventh of what is available and memory was never the constraint —
-          # the label name "sm-standard-4" had led everyone, including the
-          # original mitigation, to assume 4 cores.
-          #
-          # Raised to 3 on main pushes and manual dispatches; PRs keep 2 so the
-          # merge path is untouched while the experiment runs.
-          #
-          # Dispatch is included because push alone cannot supply the samples:
-          # this workflow cancels superseded runs on main, and only 4 of the last
-          # 20 push-triggered Test and Lint jobs reached a terminal state — at
-          # that rate ten samples would take roughly fifty merges. The
-          # concurrency group is scoped by event_name, so a dispatched run has
-          # its own group and is not cancelled by merge traffic, which makes the
-          # sample collectable on demand rather than by waiting.
-          #
-          # Baseline over 17 samples at 2:
-          # median nextest/clippy step ratio 1.95, spread 1.85-2.06. The gate-2
-          # criterion is that ratio dropping at least 10% (below ~1.76) with no
-          # 75m timeout and no run showing three consecutive samples of
-          # rustc/collect2/rust-lld in D state. If it does not, the conclusion is
-          # "this limit is not the bottleneck" — fix it back at 2 and record the
-          # experiment, which is a result, not a failure.
-          #
-          # Must stay step-level: rust-cache hashes CARGO/CC/CFLAGS/CXX/CMAKE/RUST
-          # prefixed variables from process.env into the cache key, so promoting
-          # this to job level would rotate every key on this lane.
-          CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
+          # Three concurrent workspace test links saturate the self-hosted
+          # runner's overlay I/O and can wedge Cargo until the 75m timeout.
+          CARGO_BUILD_JOBS: "2"
         run: |
           mkdir -p artifacts/test-and-lint
-          ./scripts/ci/resource_sampler.sh start nextest
-          trap './scripts/ci/resource_sampler.sh stop' EXIT
+          # Evidence sampler for issue #5394: the post-mortem pgrep below runs
+          # only after `timeout` has already TERM'd the whole cargo process
+          # group, so it cannot name a wedged process. Sample system and
+          # process state every 60s instead; the last samples before the
+          # timeout show what was stuck (rustc, linker, build script, memory
+          # pressure, ...). The log rides along in the existing artifact.
+          (
+            while true; do
+              {
+                echo "=== $(date --utc --iso-8601=seconds)"
+                echo "--- load";  cat /proc/loadavg
+                echo "--- psi";   grep -H . /proc/pressure/* 2>/dev/null || true
+                echo "--- mem";   free -m
+                echo "--- disk";  df -h / /home/runner 2>/dev/null || df -h /
+                echo "--- top-rss"
+                ps -eo pid,ppid,stat,etime,rss,pcpu,args --sort=-rss | head -15
+                echo "--- build/test processes"
+                ps -eo pid,ppid,stat,etime,rss,pcpu,args | grep -E '[c]argo|[r]ustc|[n]extest|[c]ollect2|rust-ll[d]|[b]uild-script|deps[/]' || true
+                echo "--- d-state (uninterruptible IO)"
+                ps -eo pid,stat,etime,args | awk 'NR > 1 && $2 ~ /D/' || true
+                echo
+              } >> artifacts/test-and-lint/sampler.log 2>&1 || true
+              sleep 60
+            done
+          ) &
+          sampler_pid=$!
+          trap 'kill "${sampler_pid}" 2>/dev/null || true' EXIT
           set +e
           NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 75m \
             cargo nextest run --profile ci --all --exclude e2e_test \
@@ -324,50 +277,6 @@ jobs:
       - name: Run rebalance/decommission migration proofs
         run: ./scripts/check_migration_gate_count.sh
 
-      # Early stop. Once this job has failed the PR cannot merge, so the sibling
-      # lanes are burning runners on a result nobody can act on: on run
-      # 30674613104 three lanes had already failed while Test and Lint and the
-      # rio-v2 variant kept going past 70 minutes.
-      #
-      # Only this job may cancel. The lanes that are NOT required checks
-      # (protocols, ILM, e2e, s3-tests) must never hold that power: a flake in
-      # one of them would turn the required "Test and Lint" into `cancelled`,
-      # which blocks the merge. Today a maintainer can merge with sftp red, and
-      # that has to stay true.
-      #
-      # These steps run last so the `if: always()` artifact upload above still
-      # captures logs and diagnostics before the run goes away.
-      - name: Annotate early-stop reason
-        if: failure() && github.event_name == 'pull_request'
-        run: |
-          {
-            echo "## CI early-stop"
-            echo "Job \`${GITHUB_JOB}\` (Test and Lint) failed; cancelling run ${GITHUB_RUN_ID} to free runners."
-            echo "Sibling jobs showing **cancelled** were stopped by this job, not by their own failure."
-          } >> "$GITHUB_STEP_SUMMARY"
-
-      # curl rather than `gh`: every existing `gh` call in this repo runs on
-      # ubuntu-latest, and the sm-standard-* images are custom and trimmed (they
-      # ship no C toolchain, see the e2e job below), so `gh` is not known to
-      # exist here.
-      #
-      # Fork PRs are excluded explicitly instead of relying on the error path:
-      # their GITHUB_TOKEN is forced read-only and job-level permissions cannot
-      # raise it, so the call would always 403. Skipping keeps their logs clean.
-      - name: Cancel run on failure (same-repo PR only)
-        if: >-
-          failure() && github.event_name == 'pull_request'
-          && github.event.pull_request.head.repo.full_name == github.repository
-        continue-on-error: true
-        env:
-          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        run: |
-          curl -fsS -X POST \
-            -H "Authorization: Bearer ${GH_TOKEN}" \
-            -H "Accept: application/vnd.github+json" \
-            -H "X-GitHub-Api-Version: 2022-11-28" \
-            "${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/cancel" || true
-
   # Dedicated serial lane for the ILM / lifecycle integration tests. These tests
   # drive the object layer through process-global singletons (the GLOBAL_ENV
   # ECStore, the global tier-config manager, background-expiry workers) and bind
@@ -381,7 +290,6 @@ jobs:
   test-ilm-integration-serial:
     name: ILM Integration (serial)
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    needs: [ quick-checks ]
     runs-on: sm-standard-4
     timeout-minutes: 45
     env:
@@ -389,16 +297,14 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          cache-shared-key: ci-dev
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
+          cache-shared-key: ci-ilm-serial
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
       # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
       # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
@@ -421,7 +327,6 @@ jobs:
   test-and-lint-rio-v2:
     name: Test and Lint (rio-v2)
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    needs: [ quick-checks ]
     runs-on: sm-standard-4
     timeout-minutes: 60
     env:
@@ -429,16 +334,14 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          cache-shared-key: ci-feat-rio
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
+          cache-shared-key: ci-test-rio-v2
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
       - name: Run rio-v2 clippy lints
         run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
@@ -451,17 +354,10 @@ jobs:
   test-and-lint-protocols:
     name: "Test and Lint (${{ matrix.features.name }})"
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    needs: [ quick-checks ]
     runs-on: sm-standard-4
     timeout-minutes: 60
     strategy:
-      # On a PR, one failing protocol leg is enough to know the PR is not ready,
-      # so stop the sibling leg instead of paying another ~40 minutes for it.
-      # Everywhere else (main pushes, the merge queue, the weekly schedule) keep
-      # the full signal: there we want to know whether swift AND sftp are broken,
-      # not just whichever failed first. This is the only part of the early-stop
-      # work that also covers fork PRs, since it needs no token.
-      fail-fast: ${{ github.event_name == 'pull_request' }}
+      fail-fast: false
       matrix:
         features:
           - name: swift
@@ -473,16 +369,14 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          cache-shared-key: ci-feat-proto
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
+          cache-shared-key: ci-test-${{ matrix.features.name }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
       - name: Run clippy with ${{ matrix.features.name }}
         run: |
@@ -495,7 +389,6 @@ jobs:
   build-rustfs-debug-binary:
     name: Build RustFS Debug Binary
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    needs: [ quick-checks ]
     runs-on: sm-standard-4
     timeout-minutes: 30
     env:
@@ -503,16 +396,14 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          cache-shared-key: ci-dev
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
+          cache-shared-key: ci-rustfs-debug-binary
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Build debug binary
         run: cargo build -p rustfs --bins --features e2e-test-hooks
@@ -528,7 +419,6 @@ jobs:
   build-rustfs-debug-binary-rio-v2:
     name: Build RustFS Debug Binary (rio-v2)
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    needs: [ quick-checks ]
     runs-on: sm-standard-4
     timeout-minutes: 30
     env:
@@ -536,16 +426,14 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          cache-shared-key: ci-feat-rio
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
+          cache-shared-key: ci-rustfs-debug-binary-rio-v2
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Build debug binary with rio-v2
         run: cargo build -p rustfs --bins --features rio-v2,e2e-test-hooks
@@ -560,14 +448,6 @@ jobs:
 
   uring-integration:
     name: io_uring Integration (real)
-    # The pull_request trigger includes `closed` purely so the concurrency
-    # group cancels in-flight runs of a closed PR; every other job opts out of
-    # that run with this guard (or is skipped through its `needs` chain). This
-    # job had neither, so each closed/merged PR really ran the whole io_uring
-    # suite (measured 4m17s / 7m19s / 7m31s on runs 30678272341 / 30678117601 /
-    # 30662728539) and kept the cancellation run in progress for minutes.
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    needs: [ quick-checks ]
     # GitHub-hosted ubuntu-latest runs a recent kernel with io_uring and, unlike
     # a container, applies no seccomp filter that would block io_uring_setup — so
     # the probe succeeds and the tests exercise the real UringBackend/FdCache/
@@ -578,24 +458,17 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          # Keeps its own key rather than joining ci-dev. rust-cache's key is
-          # built from runner.os/arch plus rustc and lockfile fingerprints — it
-          # does NOT include the runner label or image. ubuntu-latest and
-          # sm-standard-4 are therefore indistinguishable to it, so sharing a key
-          # would let two different system images overwrite each other's
-          # artifacts, and would make a 2-core hosted runner unpack ci-dev's ~3GB
-          # instead of this lane's ~1.3GB. cache-warm.yml warms this key on
-          # ubuntu-latest for the same reason.
           cache-shared-key: ci-uring
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Install build dependencies
+        run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
 
       # ext4 supports O_DIRECT; the runner's default TMPDIR may sit on tmpfs or
       # overlayfs, where open(O_DIRECT) returns EINVAL/EOPNOTSUPP and the native
@@ -622,17 +495,7 @@ jobs:
           RUSTFS_IO_URING_READ_ENABLE: "true"
           RUSTFS_URING_TESTS_MUST_RUN: "1"
           TMPDIR: /mnt/rustfs-odirect
-        # --lib narrows what gets compiled, not what gets run: every selected
-        # test lives in the lib target. The 7 integration binaries under
-        # crates/ecstore/tests/ each reported "running 0 tests" here, so they
-        # were compiled and linked for nothing.
-        #
-        # The `uring_` filter must stay exactly as it is. libtest matches on
-        # substring, so it also selects names containing `during_` — 6 of the 18
-        # selected tests are such incidental matches. Narrowing the filter to
-        # `io_uring` would silently drop them, which is a coverage change.
-        # scripts/check_uring_lane_lib_only.sh guards the --lib precondition.
-        run: cargo test -p rustfs-ecstore --lib uring_ -- --test-threads=1 --nocapture
+        run: cargo test -p rustfs-ecstore uring_ -- --test-threads=1 --nocapture
 
   e2e-tests:
     name: End-to-End Tests
@@ -642,8 +505,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       # Full setup with dependency caching: the smoke-suite step below
       # compiles the e2e_test crate, which pulls in most of the workspace.
@@ -652,9 +513,9 @@ jobs:
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          cache-shared-key: ci-dev
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
+          cache-shared-key: ci-e2e
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
       # Download after the cache restore so the freshly built binary from the
       # build job always wins over anything restored into target/debug.
@@ -667,17 +528,15 @@ jobs:
       - name: Make binary executable
         run: chmod +x ./target/debug/rustfs
 
-      # Build the e2e test graph once. The archive is reused by the security
-      # count-floor check and the smoke run below, avoiding a second compile of
-      # the same e2e_test target on cold runners (backlog#1645).
-      - name: Archive e2e smoke test binaries
-        env:
-          NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
-          NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-smoke-list.json
-        run: |
-          cargo nextest archive --profile e2e-smoke -p e2e_test --archive-file "${NEXTEST_ARCHIVE}"
-          cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --message-format json > "${NEXTEST_LISTING}"
-          ./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
+      # Guard the security negative-auth smoke subset (backlog#1151 sec-5)
+      # against a rename or deletion silently dropping it out of the e2e-smoke
+      # filter. The script lists what the profile selects and fails if the count
+      # of security auth-rejection tests falls below the committed floor in
+      # .config/security-smoke-floor.txt (infra-12 count-floor mechanism). Run
+      # before the smoke suite so a thinned gate fails fast; the `nextest list`
+      # here compiles the e2e_test binaries the run below reuses.
+      - name: Check security smoke subset count floor
+        run: ./scripts/check_security_smoke_count.sh check
 
       # PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
       # profile.e2e-smoke default-filter in .config/nextest.toml is the single
@@ -685,30 +544,7 @@ jobs:
       # adding new e2e jobs here. Each test spawns its own rustfs server on a
       # random port and reuses the downloaded debug binary above.
       - name: Run e2e smoke suite
-        env:
-          NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
-          RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
-        run: |
-          cargo nextest run --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" \
-            --status-level all --final-status-level all --failure-output final
-
-      - name: Upload e2e smoke diagnostics
-        if: failure()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: e2e-smoke-diagnostics-${{ github.run_number }}
-          path: |
-            ${{ runner.temp }}/rustfs-e2e-smoke-logs/
-            ${{ runner.temp }}/rustfs-e2e-smoke-list.json
-          if-no-files-found: warn
-
-      - name: Upload e2e smoke JUnit report
-        if: always()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: e2e-smoke-junit-${{ github.run_number }}
-          path: target/nextest/e2e-smoke/junit.xml
-          if-no-files-found: warn
+        run: cargo nextest run --profile e2e-smoke -p e2e_test
 
       - name: Install s3s-e2e test tool
         uses: taiki-e/cache-cargo-install-action@7447f04c51f2ba27ca35e7f1e28fab848c5b3ba7 # v2
@@ -753,16 +589,14 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          cache-shared-key: ci-dev
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
+          cache-shared-key: ci-e2e
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
       # Download after the cache restore so the freshly built binary from the
       # build job always wins over anything restored into target/debug.
@@ -798,8 +632,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Clean up previous test run
         run: |
@@ -854,8 +686,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Download debug binary
         uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -916,20 +746,12 @@ jobs:
   # evaluates ILM within ~2s of the due time, well inside the poll window.
   s3-lifecycle-behavior-tests:
     name: S3 Lifecycle Behavior Tests
-    # Also gated on e2e-tests, matching s3-implemented-tests: when the e2e smoke
-    # suite is already red this lane cannot tell us anything new, and it holds a
-    # sm-standard-4 for up to 30 minutes doing so. Both lanes only download the
-    # prebuilt debug binary (no cargo build), and s3-implemented-tests — which
-    # already waits on e2e-tests — finishes later anyway, so a green PR's total
-    # wall clock is unchanged.
-    needs: [ build-rustfs-debug-binary, e2e-tests ]
+    needs: [ build-rustfs-debug-binary ]
     runs-on: sm-standard-4
     timeout-minutes: 30
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Download debug binary
         uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7

.github/workflows/cla.yml

@@ -22,18 +22,11 @@ on:
   issue_comment:
     types: [created, edited]
 
-# Least privilege at the top, widened per job below. This workflow runs on
-# pull_request_target and issue_comment, so it holds full secrets on every fork
-# PR and on any comment anyone writes — the one place in this repository where a
-# compromised action would be handed a repo-write token. It does not check out
-# or execute PR code, so there is no pwn-request path today, but the blast
-# radius should not depend on that staying true.
-#
-# contents: write in particular was never used: the signature records are
-# written to rustfs/cla through the scoped app token created below, and nothing
-# here writes to this repository's contents.
 permissions:
-  contents: read
+  contents: write
+  pull-requests: write
+  issues: write
+  checks: write
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.event.issue.number || github.ref }}
@@ -43,26 +36,14 @@ jobs:
   cancel-closed-pr-runs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request_target' && github.event.action == 'closed'
-    # Echoes one line; the run exists only so the concurrency group cancels the
-    # in-flight run of a closed PR.
-    permissions: {}
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Explain cancellation run
         run: echo "PR closed; this run only cancels older runs in the same concurrency group."
 
   cla:
     if: ${{ (github.event_name != 'issue_comment' || github.event.issue.pull_request) && (github.event_name != 'pull_request_target' || github.event.action != 'closed') }}
-    # checks: write reports the merge-queue check run; pull-requests and issues
-    # let cla-bot comment and label. contents stays read — see the note above.
-    permissions:
-      contents: read
-      checks: write
-      issues: write
-      pull-requests: write
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     steps:
       - name: Report CLA result for merge queue
         if: github.event_name == 'merge_group'

.github/workflows/coverage.yml

@@ -62,16 +62,14 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
           cache-shared-key: ci-coverage
+          github-token: ${{ secrets.GITHUB_TOKEN }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-          install-build-packaging-tools: 'false'
 
       - name: Install cargo-llvm-cov
         uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
@@ -118,8 +116,6 @@ jobs:
       issues: write
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
       - name: Open or update failure-tracking issue
         uses: ./.github/actions/schedule-failure-issue
         with:

.github/workflows/docker.yml

@@ -82,10 +82,8 @@ jobs:
       github.event_name == 'workflow_dispatch' ||
       (github.event.workflow_run.conclusion == 'success' &&
        github.event.workflow_run.event == 'push' &&
-       github.event.workflow_run.head_branch != 'main' &&
-       !contains(github.event.workflow_run.head_branch, '-preview'))
+       github.event.workflow_run.head_branch != 'main')
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
       should_push: ${{ steps.check.outputs.should_push }}
@@ -98,18 +96,11 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
         with:
-          persist-credentials: false
           # For workflow_run events, checkout the specific commit that triggered the workflow
           ref: ${{ github.event.workflow_run.head_sha || github.sha }}
 
       - name: Check build conditions
         id: check
-        env:
-          # dispatch inputs via env, not `${{ }}` interpolation: they are
-          # free-form strings and would otherwise be evaluated by bash.
-          INPUT_VERSION: ${{ github.event.inputs.version }}
-          INPUT_PUSH_IMAGES: ${{ github.event.inputs.push_images }}
-          INPUT_FORCE_REBUILD: ${{ github.event.inputs.force_rebuild }}
         run: |
           should_build=false
           should_push=false
@@ -210,9 +201,9 @@ jobs:
 
           elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
             # Manual trigger
-            input_version="$INPUT_VERSION"
+            input_version="${{ github.event.inputs.version }}"
             version="${input_version}"
-            should_push="$INPUT_PUSH_IMAGES"
+            should_push="${{ github.event.inputs.push_images }}"
             should_build=true
 
             # Get short SHA
@@ -220,7 +211,7 @@ jobs:
 
             echo "🎯 Manual Docker build triggered:"
             echo "   📋 Requested version: $input_version"
-            echo "   🔧 Force rebuild: $INPUT_FORCE_REBUILD"
+            echo "   🔧 Force rebuild: ${{ github.event.inputs.force_rebuild }}"
             echo "   🚀 Push images: $should_push"
 
             case "$input_version" in
@@ -229,13 +220,6 @@ jobs:
                 create_latest=true
                 echo "🚀 Building with latest stable release version"
                 ;;
-              *-preview*)
-                build_type="preview"
-                is_prerelease=true
-                should_build=false
-                should_push=false
-                echo "⏭️  Preview tags do not publish Docker images"
-                ;;
               # Prerelease versions (must match first, more specific)
               v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
                 build_type="prerelease"
@@ -306,8 +290,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Login to Docker Hub
         uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
@@ -343,28 +325,32 @@ jobs:
           CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
           VARIANT_SUFFIX="${{ matrix.suffix }}"
 
-          # Convert version format for Dockerfile compatibility. The former
-          # DOCKER_CHANNEL was "release" down every branch and was passed as a
-          # build-arg no Dockerfile declares, so it is gone.
+          # Convert version format for Dockerfile compatibility
           case "$VERSION" in
             "latest")
+              # For stable latest, use RELEASE=latest + release CHANNEL
               DOCKER_RELEASE="latest"
+              DOCKER_CHANNEL="release"
               ;;
             v*)
               # For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
               DOCKER_RELEASE="${VERSION#v}"
+              DOCKER_CHANNEL="release"
               ;;
             *)
               # For other versions, pass as-is
               DOCKER_RELEASE="${VERSION}"
+              DOCKER_CHANNEL="release"
               ;;
           esac
 
           echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
+          echo "docker_channel=$DOCKER_CHANNEL" >> "$GITHUB_OUTPUT"
 
           echo "🐳 Docker build parameters:"
           echo "  - Original version: $VERSION"
           echo "  - Docker RELEASE: $DOCKER_RELEASE"
+          echo "  - Docker CHANNEL: $DOCKER_CHANNEL"
 
           # Generate tags based on build type
           # Only support release and prerelease builds (no development builds)
@@ -418,24 +404,18 @@ jobs:
           push: ${{ needs.build-check.outputs.should_push == 'true' }}
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
-          # No layer cache. This build compiles nothing — it downloads a
-          # release zip and runs apk/apt — so the cache could only save the
-          # minute or two those take, while creating a correctness problem: with
-          # RELEASE=latest the binary URL is resolved by curl *inside* a RUN
-          # layer, and the layer key does not include what that resolved to. A
-          # rebuild at the same RELEASE value (dispatch with version=latest, or
-          # a re-run of the same version) would hit the old layer and ship the
-          # previous release's binary. mode=max also consumed the same 10GB
-          # Actions cache quota the Rust lanes are fighting over.
-          #
-          # Only RELEASE is passed: it is the sole build-arg the Dockerfiles
-          # declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION
-          # and CHANNEL were never read by any stage (and BUILDTIME's $(date ...)
-          # was a literal here, not a shell substitution). BUILD_DATE and VCS_REF
-          # are declared by the Dockerfiles but deliberately left unset —
-          # supplying them would change the published image labels.
+          cache-from: |
+            type=gha,scope=docker-${{ matrix.variant }}
+          cache-to: |
+            type=gha,mode=max,scope=docker-${{ matrix.variant }}
           build-args: |
+            BUILDTIME=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
+            VERSION=${{ needs.build-check.outputs.version }}
+            BUILD_TYPE=${{ needs.build-check.outputs.build_type }}
+            REVISION=${{ github.sha }}
             RELEASE=${{ steps.meta.outputs.docker_release }}
+            CHANNEL=${{ steps.meta.outputs.docker_channel }}
+            BUILDKIT_INLINE_CACHE=1
           provenance: true
           sbom: true
           # Add retry mechanism by splitting the build process
@@ -451,7 +431,6 @@ jobs:
     needs: [ build-check, build-docker ]
     if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     permissions:
       contents: read
       security-events: write
@@ -506,7 +485,6 @@ jobs:
     needs: [ build-check, build-docker ]
     if: always() && needs.build-check.outputs.should_build == 'true'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Docker build completion summary
         run: |

.github/workflows/e2e-replication-nightly.yml

@@ -64,16 +64,14 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
           cache-shared-key: ci-e2e-repl
+          github-token: ${{ secrets.GITHUB_TOKEN }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-          install-build-packaging-tools: 'false'
 
       # awscurl lets the STS dual-node test actually exercise its path. Without
       # it the test skips gracefully with a visible log line
@@ -126,8 +124,6 @@ jobs:
       issues: write
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
       - name: Open or update failure-tracking issue
         uses: ./.github/actions/schedule-failure-issue
         with:

.github/workflows/e2e-s3tests.yml

@@ -45,13 +45,6 @@
 # The PR gate (ci.yml s3-implemented-tests) is unaffected: it avoids Docker
 # via DEPLOY_MODE=binary and defers all pip setup to run.sh's self-bootstrap.
 
-# DISABLED. This workflow is switched off in the repository's Actions settings
-# (state: disabled_manually) and does not run on any trigger, including its cron
-# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
-# reading this file, which has already misled at least one audit — hence this
-# banner. Re-enabling is a UI action; anyone doing so should first check that the
-# workflow still matches the current CI layout. See rustfs/backlog#1603.
-#
 name: e2e-s3tests
 
 on:
@@ -142,8 +135,6 @@ jobs:
       TEST_MODE: ${{ matrix.test-mode }}
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       # Provision Python explicitly rather than trusting the runner image to
       # ship a working pip (ci-1: a bare python3 without pip is what broke the
@@ -363,8 +354,6 @@ jobs:
       issues: write
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
       - name: Open or update failure-tracking issue
         uses: ./.github/actions/schedule-failure-issue
         with:

.github/workflows/fuzz.yml

@@ -12,13 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# DISABLED. This workflow is switched off in the repository's Actions settings
-# (state: disabled_manually) and does not run on any trigger, including its cron
-# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
-# reading this file, which has already misled at least one audit — hence this
-# banner. Re-enabling is a UI action; anyone doing so should first check that the
-# workflow still matches the current CI layout. See rustfs/backlog#1603.
-#
 name: Fuzz
 
 on:
@@ -66,7 +59,6 @@ jobs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request' && github.event.action == 'closed'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Explain cancellation run
         run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -87,14 +79,13 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: nightly
           cache-shared-key: fuzz-${{ hashFiles('fuzz/Cargo.lock') }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' || github.event_name == 'schedule' }}
 
       - name: Install cargo-fuzz
@@ -154,8 +145,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Download prebuilt fuzz binaries
         uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -211,8 +200,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Download prebuilt fuzz binaries
         uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -260,8 +247,6 @@ jobs:
       issues: write
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
       - name: Open or update failure-tracking issue
         uses: ./.github/actions/schedule-failure-issue
         with:

.github/workflows/helm-package.yml

@@ -32,14 +32,12 @@ permissions:
 jobs:
   build-helm-package:
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     if: |
-      (github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
+      github.event_name == 'workflow_dispatch' ||
       (
         github.event.workflow_run.conclusion == 'success' &&
         github.event.workflow_run.event == 'push' &&
-        contains(github.event.workflow_run.head_branch, '.') &&
-        !contains(github.event.workflow_run.head_branch, '-preview')
+        contains(github.event.workflow_run.head_branch, '.')
       )
 
     outputs:
@@ -50,26 +48,16 @@ jobs:
     steps:
       - name: Checkout helm chart repo
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
-      # Both inputs reach the shell through env rather than `${{ }}`
-      # interpolation. A git ref name may contain `$(...)` — anything without a
-      # space is a legal tag — and interpolation pastes it into the script
-      # verbatim, where bash would run it. Reading "$RAW_INPUT" instead makes it
-      # data.
       - name: Normalize release version
         id: version
-        env:
-          RAW_INPUT: ${{ github.event.inputs.version }}
-          RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
         run: |
           set -eux
 
           if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
-            RAW="$RAW_INPUT"
+            RAW="${{ github.event.inputs.version }}"
           else
-            RAW="$RAW_BRANCH"
+            RAW="${{ github.event.workflow_run.head_branch }}"
           fi
 
           case "$RAW" in
@@ -84,13 +72,10 @@ jobs:
           ./scripts/helm_chart_version.sh "$RAW_TAG"
 
       - name: Replace chart version and app version
-        env:
-          CHART_VERSION: ${{ steps.version.outputs.chart_version }}
-          APP_VERSION: ${{ steps.version.outputs.app_version }}
         run: |
           set -eux
-          sed -i -E "s/^version:.*/version: \"${CHART_VERSION}\"/" helm/rustfs/Chart.yaml
-          sed -i -E "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" helm/rustfs/Chart.yaml
+          sed -i -E 's/^version:.*/version: "${{ steps.version.outputs.chart_version }}"/' helm/rustfs/Chart.yaml
+          sed -i -E 's/^appVersion:.*/appVersion: "${{ steps.version.outputs.app_version }}"/' helm/rustfs/Chart.yaml
 
       - name: Set up Helm
         uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
@@ -115,7 +100,6 @@ jobs:
 
   publish-helm-package:
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     needs: [ build-helm-package ]
     if: needs.build-helm-package.result == 'success'
 
@@ -123,8 +107,6 @@ jobs:
       - name: Checkout helm package repo
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
         with:
-          # persist-credentials-exempt: this checkout's token IS the push credential —
-          # the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
           repository: rustfs/helm
           token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
 
@@ -140,19 +122,11 @@ jobs:
       - name: Generate index
         run: helm repo index . --url https://charts.rustfs.com
 
-      # app_version is derived from the triggering tag name, and this job holds
-      # the cross-repository push token with rustfs/helm already checked out —
-      # the worst place in the repo to paste an attacker-influenced string into
-      # a shell line. Passed through env so bash treats it as data.
       - name: Push helm package and index file
-        env:
-          GIT_USERNAME: ${{ secrets.USERNAME }}
-          GIT_EMAIL: ${{ secrets.EMAIL_ADDRESS }}
-          APP_VERSION: ${{ needs.build-helm-package.outputs.app_version }}
         run: |
           set -eux
-          git config --global user.name "${GIT_USERNAME}"
-          git config --global user.email "${GIT_EMAIL}"
+          git config --global user.name "${{ secrets.USERNAME }}"
+          git config --global user.email "${{ secrets.EMAIL_ADDRESS }}"
           git add .
-          git commit -m "Update rustfs helm package with ${APP_VERSION}." || echo "No changes to commit"
+          git commit -m "Update rustfs helm package with ${{ needs.build-helm-package.outputs.app_version }}." || echo "No changes to commit"
           git push origin main

.github/workflows/issue-translator.yml

@@ -12,13 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# DISABLED. This workflow is switched off in the repository's Actions settings
-# (state: disabled_manually) and does not run on any trigger, including its cron
-# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
-# reading this file, which has already misled at least one audit — hence this
-# banner. Re-enabling is a UI action; anyone doing so should first check that the
-# workflow still matches the current CI layout. See rustfs/backlog#1603.
-#
 name: "issue-translator"
 on:
   issue_comment:
@@ -33,7 +26,6 @@ permissions:
 jobs:
   build:
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - uses: usthe/issues-translate-action@b41f55ddc81d7d54bd542a4f289fe28ec081898e # v2.7
         with:

.github/workflows/minio-interop.yml

@@ -18,25 +18,11 @@
 # This is NOT a PR gate. The fixtures are real MinIO backend trees generated on
 # the fly (they are gitignored, never committed), so the job regenerates them
 # each run with Docker and then runs the `#[ignore]` reader tests in
-# rustfs/src/storage/minio_generated_read_test.rs.
-#
-# Scope: end-to-end MinIO-to-RustFS SSE interop is NOT implemented yet. Both
-# envelope parsers reject MinIO's own wrapped-DEK shape — see
-# `is_data_key_envelope` in crates/kms/src/encryption/dek.rs and the
-# `deny_unknown_fields` `LocalSseDekEnvelope` in rustfs/src/storage/sse.rs — and
-# closing that gap is tracked in rustfs/backlog#1638. Treat this job as the
-# harness for #1638, not as standing evidence that a MinIO migration reads back.
+# crates/ecstore/tests/minio_generated_read_test.rs.
 #
 # Runner: GitHub-hosted `ubuntu-latest`. It reliably ships Docker + Python,
 # unlike the self-hosted fleet, whose pods drift in Docker/pip availability
 # (see the infra note in e2e-s3tests.yml). Nightly + manual only.
-# DISABLED. This workflow is switched off in the repository's Actions settings
-# (state: disabled_manually) and does not run on any trigger, including its cron
-# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
-# reading this file, which has already misled at least one audit — hence this
-# banner. Re-enabling is a UI action; anyone doing so should first check that the
-# workflow still matches the current CI layout. See rustfs/backlog#1603.
-#
 name: minio-interop
 
 on:
@@ -62,62 +48,23 @@ jobs:
     env:
       # Fixed 32-byte test KMS key baked into the fixture lab; not a secret.
       RUSTFS_MINIO_STATIC_KMS_KEY_B64: IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g=
-      # Single definition of "the interop tests", shared by the guard step and
-      # the run step so the two cannot drift apart.
-      #
-      # These used to live in crates/ecstore/tests/minio_generated_read_test.rs
-      # and were selected with `-p rustfs-ecstore -E
-      # 'binary(minio_generated_read_test)'`. #5435 moved them into the `rustfs`
-      # crate as a `#[cfg(test)] mod`, which deleted that test binary; the
-      # selector was never updated and has selected zero interop tests ever
-      # since (cargo-nextest 0.9.140 now rejects it outright: "operator didn't
-      # match any binary names", exit 94).
-      INTEROP_PACKAGE: rustfs
-      INTEROP_FEATURES: rio-v2
-      INTEROP_FILTER: "test(minio_generated_read_test::)"
-      INTEROP_REQUIRED_TESTS: '["reads_minio_generated_sse_s3_multipart_fixture", "reads_minio_generated_sse_kms_multipart_fixture", "rejects_minio_generated_sse_s3_fixture_with_wrong_kms_key", "rejects_minio_generated_sse_s3_fixture_with_truncated_ciphertext"]'
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
           cache-shared-key: ci-minio-interop
+          github-token: ${{ secrets.GITHUB_TOKEN }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
       - name: Generate real MinIO fixtures via Docker
         run: bash crates/rio-v2/tests/minio_fixture_lab/capture_via_docker.sh
 
-      # `binary(...)` at least dies loudly when nothing matches, but `test(...)`
-      # is a perfectly valid filterset that matches zero tests, so the next
-      # rename or module move would leave this job selecting nothing and
-      # reporting success without executing a single interop assertion. Count
-      # the selection and require every core reader test, while allowing new
-      # reader cases to be added without changing this guard.
-      #
-      # Count only `filter-match.status == "matches"`: the top-level
-      # `test-count` in the JSON is the package total and ignores `-E` entirely.
-      - name: Assert the interop selector still matches tests
-        run: |
-          set -euo pipefail
-          selection="$(cargo nextest list --run-ignored ignored-only \
-            -p "$INTEROP_PACKAGE" --features "$INTEROP_FEATURES" \
-            -E "$INTEROP_FILTER" --message-format json \
-            | python3 -c 'import json,os,sys; d=json.load(sys.stdin); required=json.loads(os.environ["INTEROP_REQUIRED_TESTS"]); matched=[name for suite in d.get("rust-suites", {}).values() for name,test in suite.get("testcases", {}).items() if test.get("filter-match", {}).get("status") == "matches"]; missing=[test for test in required if not any(name.endswith("minio_generated_read_test::" + test) for name in matched)]; print(len(matched)); print(",".join(missing))')"
-          count="$(printf '%s\n' "$selection" | sed -n '1p')"
-          missing="$(printf '%s\n' "$selection" | sed -n '2p')"
-          echo "interop tests selected: ${count}"
-          if [ -n "${missing}" ]; then
-            echo "::error::Selector '${INTEROP_FILTER}' in package '${INTEROP_PACKAGE}' is missing required tests: ${missing}. The MinIO interop reader tests have moved or been renamed; fix the selector instead of running an incomplete matrix. Context: rustfs/backlog#1638."
-            exit 1
-          fi
-
       - name: Run MinIO interop reader tests
         run: |
-          cargo nextest run --run-ignored ignored-only --no-tests=fail \
-            -p "$INTEROP_PACKAGE" --features "$INTEROP_FEATURES" \
-            -E "$INTEROP_FILTER"
+          cargo nextest run --run-ignored ignored-only \
+            -p rustfs-ecstore --features rio-v2 \
+            -E 'binary(minio_generated_read_test)'

.github/workflows/mint.yml

@@ -45,13 +45,6 @@
 # docker-capable self-hosted `dind-sm-standard-2` label was the alternative but
 # has fewer cores and reintroduces fleet-state risk for no reliability gain.
 
-# DISABLED. This workflow is switched off in the repository's Actions settings
-# (state: disabled_manually) and does not run on any trigger, including its cron
-# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
-# reading this file, which has already misled at least one audit — hence this
-# banner. Re-enabling is a UI action; anyone doing so should first check that the
-# workflow still matches the current CI layout. See rustfs/backlog#1603.
-#
 name: mint
 
 on:
@@ -125,8 +118,6 @@ jobs:
     timeout-minutes: 120
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Enable buildx
         uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
@@ -272,8 +263,6 @@ jobs:
       issues: write
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
       - name: Open or update failure-tracking issue
         uses: ./.github/actions/schedule-failure-issue
         with:

.github/workflows/nightly-gnu.yml

@@ -1,57 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-name: Nightly GNU Build
-
-on:
-  schedule:
-    - cron: "0 0 * * *"
-      timezone: "Asia/Shanghai"
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-concurrency:
-  group: nightly-gnu-build-main-${{ github.event_name }}
-  cancel-in-progress: ${{ github.event_name == 'workflow_dispatch' }}
-
-env:
-  CARGO_TERM_COLOR: always
-  RUST_BACKTRACE: 1
-
-jobs:
-  build:
-    name: Build x86_64 GNU
-    runs-on: sm-standard-2
-    timeout-minutes: 150
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-    steps:
-      - name: Checkout main branch
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-          ref: main
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          cache-shared-key: build-x86_64-unknown-linux-gnu
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-          install-test-tools: 'false'
-
-      - name: Build RustFS
-        run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins

.github/workflows/nix-flake-update.yml

@@ -19,12 +19,9 @@ on:
   schedule:
     - cron: '0 5 * * 0' # Weekly on Sunday 05:00 UTC (staggered after the midnight ci/build crons)
 
-# GITHUB_TOKEN only needs to read the repository here: the branch push and the
-# pull request are both created by update-flake-lock using the
-# FLAKE_UPDATE_TOKEN PAT below, not by this token. Leaving write on it hands a
-# repo-write credential to an unattended weekly job that does not use it.
 permissions:
-  contents: read
+  contents: write
+  pull-requests: write
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
@@ -40,10 +37,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        # persist-credentials-exempt: update-flake-lock pushes the branch and opens
-        # the PR. It passes FLAKE_UPDATE_TOKEN to create-pull-request itself rather
-        # than reusing .git/config, but that is unverified — exempt until a
-        # workflow_dispatch run confirms it (rustfs/backlog#1602).
 
       - name: Install Nix
         uses: DeterminateSystems/determinate-nix-action@629b284231c2a82554b724e357e47fc6020833c8 # v3

.github/workflows/nix.yml

@@ -12,13 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# DISABLED. This workflow is switched off in the repository's Actions settings
-# (state: disabled_manually) and does not run on any trigger, including its cron
-# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
-# reading this file, which has already misled at least one audit — hence this
-# banner. Re-enabling is a UI action; anyone doing so should first check that the
-# workflow still matches the current CI layout. See rustfs/backlog#1603.
-#
 name: Nix CI
 
 on:
@@ -53,7 +46,6 @@ jobs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request' && github.event.action == 'closed'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Explain cancellation run
         run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -71,8 +63,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Install Nix
         uses: DeterminateSystems/determinate-nix-action@4eea0b33e3d1f02ecfe37cf16e7204c424009606 # v3.21.0

.github/workflows/performance-ab.yml

@@ -17,18 +17,11 @@
 # Two entry points, honestly scoped:
 #   * schedule (nightly, on main): post-merge detection — catches a regression
 #     within 24h of landing, not before merge.
-#   * workflow_dispatch: an explicitly selected trusted ref.
-# The dispatch input can run the gate with --allow-regression so a deliberate
-# correctness cost (e.g. the #4221 fsync durability fix) is recorded, not
-# blocked (rustfs/backlog#935 correction 1).
+#   * pull_request labeled `perf-ab`: opt-in pre-merge gate for a specific PR.
+# The `perf-deliberate-tradeoff` label runs the gate with --allow-regression so
+# a deliberate correctness cost (e.g. the #4221 fsync durability fix) is
+# recorded but does not block (rustfs/backlog#935 correction 1).
 
-# DISABLED. This workflow is switched off in the repository's Actions settings
-# (state: disabled_manually) and does not run on any trigger, including its cron
-# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
-# reading this file, which has already misled at least one audit — hence this
-# banner. Re-enabling is a UI action; anyone doing so should first check that the
-# workflow still matches the current CI layout. See rustfs/backlog#1603.
-#
 name: Performance A/B
 
 on:
@@ -46,6 +39,8 @@ on:
         required: false
         default: false
         type: boolean
+  pull_request:
+    types: [labeled, synchronize, reopened]
   push:
     # Every main commit pre-builds and caches its release binary (perf-3) so the
     # nightly A/B restores a ready baseline instead of paying the double build.
@@ -53,6 +48,14 @@ on:
 
 permissions:
   contents: read
+  pull-requests: write
+
+# Per-PR: a new push cancels the previous (up to 90-minute) A/B run instead of
+# stacking them. Nightly schedule and manual dispatch get a unique group and
+# always run to completion.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 env:
   CARGO_TERM_COLOR: always
@@ -60,8 +63,8 @@ env:
 
 jobs:
   # perf-3: on every push to main, build the release binary once and cache it
-  # keyed by commit SHA (rustfs-baseline-<sha>). The warp-ab measurements
-  # restore this instead of paying the ~32min-per-side source
+  # keyed by commit SHA (rustfs-baseline-<sha>). The nightly A/B (and, later, the
+  # perf-7 PR gate) restore this instead of paying the ~32min-per-side source
   # build. That double build is what pushed the expanded 24-cell nightly past its
   # ceiling — 2026-07-11..07-14 all cancelled on the 120min timeout. Incremental
   # builds off the shared cargo cache keep each push cheap, and building on the
@@ -89,8 +92,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
@@ -98,6 +99,7 @@ jobs:
           rust-version: stable
           cache-shared-key: warp-ab-${{ hashFiles('**/Cargo.lock') }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Build release rustfs
         run: cargo build --release --bin rustfs
@@ -116,11 +118,17 @@ jobs:
 
   warp-ab:
     name: Warp A/B budget gate
-    # Always run on schedule / manual dispatch. Never on push — that event only
-    # feeds build-baseline-cache above.
+    # Always run on schedule / manual dispatch. Opt-in on PRs: only when the
+    # `perf-ab` label is present, and for `labeled` events only when the label
+    # being added is `perf-ab` itself (adding an unrelated label to an opted-in
+    # PR must not re-run the gate). Never on push — that event only feeds
+    # build-baseline-cache above.
     if: >-
       github.event_name == 'schedule' ||
-      github.event_name == 'workflow_dispatch'
+      github.event_name == 'workflow_dispatch' ||
+      (github.event_name == 'pull_request' &&
+       contains(github.event.pull_request.labels.*.name, 'perf-ab') &&
+       (github.event.action != 'labeled' || github.event.label.name == 'perf-ab'))
     runs-on: sm-standard-2
     # With perf-3's cached baseline binary the common (cache-hit) nightly is
     # measurement-only and finishes well under 50min. This ceiling stays
@@ -134,7 +142,6 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
         with:
-          persist-credentials: false
           fetch-depth: 0 # baseline is built from origin/main
 
       - name: Setup Rust environment
@@ -143,6 +150,7 @@ jobs:
           rust-version: stable
           cache-shared-key: warp-ab-${{ hashFiles('**/Cargo.lock') }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Install warp
         run: |
@@ -154,11 +162,13 @@ jobs:
 
       - name: Decide exemption
         id: exempt
-        env:
-          INPUT_ALLOW_REGRESSION: ${{ github.event.inputs.allow_regression }}
         run: |
           allow="false"
-          if [[ "$INPUT_ALLOW_REGRESSION" == "true" ]]; then
+          if [[ "${{ github.event_name }}" == "pull_request" ]] \
+            && ${{ contains(github.event.pull_request.labels.*.name, 'perf-deliberate-tradeoff') }}; then
+            allow="true"
+          fi
+          if [[ "${{ github.event.inputs.allow_regression }}" == "true" ]]; then
             allow="true"
           fi
           echo "allow_regression=$allow" >> "$GITHUB_OUTPUT"
@@ -205,34 +215,8 @@ jobs:
           cp target/release/rustfs baseline-bin/rustfs
           echo "built=true" >> "$GITHUB_OUTPUT"
 
-      - name: Build baseline on cache miss (different candidate)
-        id: baseline_build
-        if: >-
-          steps.baseline_cache.outputs.cache-hit != 'true' &&
-          steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
-        run: |
-          set -euo pipefail
-          baseline_root="$RUNNER_TEMP/rustfs-baseline-${{ github.run_id }}"
-          baseline_target="$RUNNER_TEMP/rustfs-baseline-target-${{ github.run_id }}"
-          git worktree add --detach "$baseline_root" "${{ steps.commits.outputs.baseline_sha }}"
-          cargo build --release --manifest-path "$baseline_root/Cargo.toml" --bin rustfs --target-dir "$baseline_target"
-          mkdir -p baseline-bin
-          cp "$baseline_target/release/rustfs" baseline-bin/rustfs
-          git worktree remove --force "$baseline_root"
-          echo "built=true" >> "$GITHUB_OUTPUT"
-
-      - name: Build candidate binary
-        id: candidate_build
-        if: steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
-        run: |
-          set -euo pipefail
-          cargo build --release --bin rustfs
-          mkdir -p candidate-bin
-          cp target/release/rustfs candidate-bin/rustfs
-          echo "built=true" >> "$GITHUB_OUTPUT"
-
       - name: Save self-healed baseline to cache
-        if: steps.selfheal.outputs.built == 'true' || steps.baseline_build.outputs.built == 'true'
+        if: steps.selfheal.outputs.built == 'true'
         uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
         with:
           path: baseline-bin/rustfs
@@ -240,66 +224,62 @@ jobs:
 
       - name: Run warp A/B and gate
         id: ab
-        env:
-          INPUT_DURATION: ${{ github.event.inputs.duration }}
         run: |
           set -euo pipefail
-          # The formal runner executes A1 baseline -> B1 candidate -> B2 candidate
-          # -> A2 baseline for each workload and drive-sync cell. It requires three
-          # rounds per leg to emit tail latency and error-rate evidence.
+          # Budget note: with perf-3's cached baseline the nightly does no source
+          # build on a cache hit, so the wall-clock is dominated by the short warp
+          # matrix — duration/rounds/cooldown are kept small to fit all 24 cells
+          # (6 workloads x 2 phases x 2 drive-sync) rather than dropping cells.
           # --health-timeout 180 outlasts the server's own 120s startup-readiness
           # budget, which the rig's previous 60s health poll undershot (the first
           # two nightly failures). perf-6 recalibrates these once the noise study
           # lands.
-          duration="${INPUT_DURATION:-12s}"
+          duration="${{ github.event.inputs.duration || '12s' }}"
           baseline_sha="${{ steps.commits.outputs.baseline_sha }}"
           candidate_sha="${{ steps.commits.outputs.candidate_sha }}"
           baseline_hit="${{ steps.baseline_cache.outputs.cache-hit }}"
           selfheal_built="${{ steps.selfheal.outputs.built }}"
-          baseline_built="${{ steps.baseline_build.outputs.built }}"
-          candidate_built="${{ steps.candidate_build.outputs.built }}"
 
-          args=(--duration "$duration" --rounds 3 --cooldown 5 --health-timeout 180 --baseline-revision "$baseline_sha" --candidate-revision "$candidate_sha")
+          args=(--duration "$duration" --rounds 2 --cooldown 5 --health-timeout 180)
 
-          if [[ "$baseline_hit" == "true" || "$selfheal_built" == "true" || "$baseline_built" == "true" ]]; then
+          if [[ "$baseline_hit" == "true" || "$selfheal_built" == "true" ]]; then
             chmod +x baseline-bin/rustfs
             base_bin="$PWD/baseline-bin/rustfs"
             args+=(--baseline-bin "$base_bin")
             if [[ "$baseline_hit" == "true" ]]; then
               base_src="actions-cache (rustfs-baseline-$baseline_sha)"
-            elif [[ "$selfheal_built" == "true" ]]; then
-              base_src="source build (cache self-heal, saved as rustfs-baseline-$baseline_sha)"
             else
-              base_src="isolated origin/main source build (saved as rustfs-baseline-$baseline_sha)"
+              base_src="source build (cache self-heal, saved as rustfs-baseline-$baseline_sha)"
             fi
             if [[ "$candidate_sha" == "$baseline_sha" ]]; then
               # Nightly on main: the candidate is the same commit as the baseline,
               # so reuse the one binary for both phases and skip all builds.
-              args+=(--candidate-bin "$base_bin")
+              args+=(--candidate-bin "$base_bin" --skip-build)
               cand_src="same binary as baseline (same commit)"
-            elif [[ "$candidate_built" == "true" ]]; then
-              chmod +x candidate-bin/rustfs
-              args+=(--candidate-bin "$PWD/candidate-bin/rustfs")
-              cand_src="source build of the checked-out ref"
             else
-              echo "::error::candidate binary was not built" >&2
-              exit 2
+              cand_src="source build of the checked-out ref"
             fi
           else
-            echo "::error::baseline binary was not restored or built" >&2
-            exit 2
+            # Cache miss with candidate != baseline (opt-in PR gate only): fall
+            # back to the source double-build. With the post-#4806 LTO profile
+            # this will overrun the job budget and alert; rerun once the push
+            # cache build for origin/main has completed, or wait for perf-7's
+            # merge-base caching.
+            args+=(--baseline-ref origin/main)
+            base_src="source build of origin/main (cache miss)"
+            cand_src="source build of the checked-out ref"
           fi
 
-          echo "baseline binary: $base_src"
-          echo "candidate binary: $cand_src"
+          args+=(--provenance-note "baseline commit: $baseline_sha - $base_src")
+          args+=(--provenance-note "candidate commit: $candidate_sha - $cand_src")
 
           if [[ "${{ steps.exempt.outputs.allow_regression }}" == "true" ]]; then
-            args+=(--allow-regression --exemption-reason "workflow dispatch override")
+            args+=(--allow-regression --exemption-reason "labeled perf-deliberate-tradeoff / dispatch override")
           fi
           # Do not let a gate FAIL abort the job here; capture status and surface
-          # it after the step summary is written.
+          # it after the PR comment is posted.
           set +e
-          bash scripts/run_hotpath_warp_abba.sh "${args[@]}"
+          bash scripts/run_hotpath_warp_ab.sh "${args[@]}"
           echo "status=$?" >> "$GITHUB_OUTPUT"
           set -e
           # Locate the newest run dir + gate.md for the summary/comment/artifact
@@ -307,10 +287,10 @@ jobs:
           # holds server-logs/ for diagnosis.
           # Run dirs are UTC-timestamp names (no special chars); ls is safe here.
           # shellcheck disable=SC2012
-          run_dir="$(ls -td target/hotpath-abba/*/ 2>/dev/null | head -n1 || true)"
+          run_dir="$(ls -td target/hotpath-ab/*/ 2>/dev/null | head -n1 || true)"
           echo "run_dir=${run_dir%/}" >> "$GITHUB_OUTPUT"
           # shellcheck disable=SC2012
-          gate_md="$(ls -t target/hotpath-abba/*/candidate_gate.md 2>/dev/null | head -n1 || true)"
+          gate_md="$(ls -t target/hotpath-ab/*/gate.md 2>/dev/null | head -n1 || true)"
           echo "gate_md=$gate_md" >> "$GITHUB_OUTPUT"
 
       - name: Upload A/B results
@@ -318,10 +298,10 @@ jobs:
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
         with:
           name: hotpath-warp-ab-${{ github.run_number }}
-          # Includes per-cell median_summary.csv / baseline_compare.csv, both gates,
+          # Includes per-cell median_summary.csv / baseline_compare.csv, gate.md,
           # and server-logs/ (rustfs.log + startup env per phase) so a failed run
           # is diagnosable. Short retention: this is churny nightly debug data.
-          path: target/hotpath-abba/
+          path: target/hotpath-ab/
           if-no-files-found: warn
           retention-days: 14
 
@@ -362,6 +342,13 @@ jobs:
             fi
           } >> "$GITHUB_STEP_SUMMARY"
 
+      - name: Comment gate result on PR
+        if: always() && github.event_name == 'pull_request' && steps.ab.outputs.gate_md != ''
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          gh pr comment "${{ github.event.pull_request.number }}" --body-file "${{ steps.ab.outputs.gate_md }}"
+
       # Scheduled failure alerting is handled by the alert-on-failure job below
       # (perf-2 consuming ci-8's schedule-failure-issue composite action).
 
@@ -370,7 +357,7 @@ jobs:
         run: |
           status="${{ steps.ab.outputs.status }}"
           if [[ "$status" != "0" ]]; then
-            echo "::error::warp A/B budget gate failed (exit $status). See the step summary / gate.md artifact." >&2
+            echo "::error::warp A/B budget gate failed (exit $status). See the step summary / PR comment / gate.md artifact." >&2
             exit "$status"
           fi
           echo "warp A/B budget gate passed."
@@ -380,12 +367,14 @@ jobs:
     needs: [warp-ab]
     # `always()` is required: without it this job is skipped when a needed
     # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
-    # ci-8); manual dispatch failures are already watched by a human.
+    # ci-8); PR and manual dispatch failures are already watched by a human.
     # `cancelled` is included alongside `failure` on purpose: a job that hits
     # timeout-minutes ends as `cancelled`, and the 2026-07-11..07-14 nightly
     # timeouts went silent precisely because the guard was failure-only. The
     # composite action already reports cancelled/timed-out jobs in the issue
-    # body.
+    # body. (Scheduled runs get a unique concurrency group with
+    # cancel-in-progress off, so a cancellation here means a timeout/manual
+    # abort, never a superseding run.)
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
@@ -396,8 +385,6 @@ jobs:
       issues: write
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
       - name: Open or update failure-tracking issue
         uses: ./.github/actions/schedule-failure-issue
         with:

.github/workflows/runner-hygiene.yml

@@ -1,81 +0,0 @@
-# Copyright 2026 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Asserts that the self-hosted runners are still ephemeral — one job per pod.
-#
-# This repository is public and its pull_request jobs run on those runners,
-# executing the PR's own build.rs, proc-macros and tests. The only thing keeping
-# that code from reaching a later job is that each ARC pod handles exactly one
-# job and is then destroyed. That guarantee lives in the ARC scale-set
-# configuration, outside this repository, where it can be changed without any PR
-# — so it is asserted here from the outside, against real run data, instead of
-# being assumed.
-#
-# Monthly rather than per-PR: the property changes only when someone
-# reconfigures the scale set, and the check costs a few dozen API calls.
-# See docs/ci/runners.md and rustfs/backlog#1602.
-
-name: Runner Hygiene
-
-on:
-  schedule:
-    - cron: "0 6 1 * *" # Monthly, 1st at 06:00 UTC (after the daily audit cron)
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-concurrency:
-  group: runner-hygiene
-  cancel-in-progress: false
-
-jobs:
-  check-ephemerality:
-    name: Check runner ephemerality
-    runs-on: ubuntu-latest
-    timeout-minutes: 15
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      # Exit 2 (inconclusive / broken) is deliberately not a pass: a window
-      # where every sm-* job was still queued would otherwise look identical to
-      # a clean bill of health.
-      - name: Assert one job per self-hosted runner
-        env:
-          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        run: ./scripts/ci/check_runner_ephemerality.sh 40
-
-  alert-on-failure:
-    name: Alert on scheduled failure
-    needs: [check-ephemerality]
-    # Same ci-8 mechanism as coverage.yml, audit.yml and the nightly lanes:
-    # scheduled runs file a tracking issue, manual dispatch stays quiet so
-    # debugging never produces a spurious alert.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-      - name: Open or update failure-tracking issue
-        uses: ./.github/actions/schedule-failure-issue
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/schedule-failure-alert-drill.yml

@@ -24,13 +24,6 @@
 # The run itself is expected to end red (the forced failure); only the
 # alert-on-failure job result matters.
 
-# DISABLED. This workflow is switched off in the repository's Actions settings
-# (state: disabled_manually) and does not run on any trigger, including its cron
-# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
-# reading this file, which has already misled at least one audit — hence this
-# banner. Re-enabling is a UI action; anyone doing so should first check that the
-# workflow still matches the current CI layout. See rustfs/backlog#1603.
-#
 name: Schedule Failure Alert Drill
 
 on:
@@ -63,8 +56,6 @@ jobs:
       issues: write
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
       - name: Open or update failure-tracking issue
         uses: ./.github/actions/schedule-failure-issue
         with:

.github/workflows/stale.yml

@@ -12,13 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# DISABLED. This workflow is switched off in the repository's Actions settings
-# (state: disabled_manually) and does not run on any trigger, including its cron
-# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
-# reading this file, which has already misled at least one audit — hence this
-# banner. Re-enabling is a UI action; anyone doing so should first check that the
-# workflow still matches the current CI layout. See rustfs/backlog#1603.
-#
 name: "Mark stale issues"
 on:
   schedule:
@@ -27,7 +20,6 @@ on:
 jobs:
   stale:
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     steps:
       - uses: actions/stale@5bef64f19d7facfb25b37b414482c7164d639639 # v9
         with:

.github/workflows/star-history.yml

@@ -15,7 +15,6 @@ concurrency:
 jobs:
   update:
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - uses: overtrue/repo-visuals-action@72f34d24769ff5d341956da2f23952594ef2f1e2 # v1.3.0
         with:

.github/workflows/windows-filesystem.yml

@@ -1,86 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-name: Windows Filesystem Tests
-
-on:
-  push:
-    branches: [ main ]
-    paths:
-      - "crates/ecstore/src/disk/**"
-      - "crates/ecstore/src/store/init_format.rs"
-      - "crates/ecstore/Cargo.toml"
-      - "Cargo.toml"
-      - "Cargo.lock"
-      - ".github/actions/setup/**"
-      - ".github/workflows/windows-filesystem.yml"
-  pull_request:
-    branches: [ main ]
-    paths:
-      - "crates/ecstore/src/disk/**"
-      - "crates/ecstore/src/store/init_format.rs"
-      - "crates/ecstore/Cargo.toml"
-      - "Cargo.toml"
-      - "Cargo.lock"
-      - ".github/actions/setup/**"
-      - ".github/workflows/windows-filesystem.yml"
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
-  cancel-in-progress: true
-
-env:
-  CARGO_TERM_COLOR: always
-  FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-  RUST_BACKTRACE: 1
-
-jobs:
-  rename-safety:
-    name: Rename Safety
-    runs-on: windows-latest
-    timeout-minutes: 60
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: build-x86_64-pc-windows-msvc
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-          install-test-tools: 'false'
-
-      - name: Test guarded rename publication
-        shell: pwsh
-        run: cargo test -p rustfs-ecstore --lib rename_all_ -- --nocapture
-
-      - name: Test Windows handle guards
-        shell: pwsh
-        run: cargo test -p rustfs-ecstore --lib windows_ -- --nocapture
-
-      - name: Test startup temporary-directory cleanup
-        shell: pwsh
-        run: cargo test -p rustfs-ecstore --lib cleanup_tmp_on_startup_ -- --nocapture
-
-      - name: Test fresh format publication
-        shell: pwsh
-        run: cargo test -p rustfs-ecstore --lib fresh_format_load_initializes_all_disks -- --nocapture

.gitignore

@@ -83,7 +83,3 @@ worktrees/*
 
 # Local AI-agent review artifacts (omo evidence dumps)
 .omo/
-
-# insta scratch files; the accepted .snap files ARE the assertions and are committed
-*.snap.new
-*.pending-snap

.mimocode/skills/issue-triage/SKILL.md

@@ -1,116 +0,0 @@
----
-name: issue-triage
-description: Triage a GitHub issue — determine if it is already fixed, needs implementation, or should be closed. Searches related commits and PRs, verifies implementation status, and posts a triage comment or closes the issue. Use when the user provides an issue URL and asks whether it can be closed or needs work.
----
-
-# Issue Triage
-
-Use this skill when the user provides a GitHub issue URL and asks "can this be closed?", "is this already implemented?", "check completion status", or similar triage questions.
-
-## Workflow
-
-### 1. Fetch issue context
-
-```bash
-gh issue view <N> --repo <owner/repo> --json title,body,state,comments,labels,updatedAt
-```
-
-Read the issue body to understand what was requested. Extract:
-- The specific feature/fix/behavior described.
-- Any linked PRs or commits mentioned in the body or comments.
-- Any checklist items or sub-issues.
-
-### 2. Search for related work
-
-Search git history for commits referencing the issue:
-```bash
-git log --oneline --all --grep="<N>" | head -30
-```
-
-Search for related PRs:
-```bash
-gh pr list --search "fixes #<N> OR closes #<N> OR #<N>" --state all --json number,title,state,mergedAt
-```
-
-If the issue mentions specific PRs, check their status:
-```bash
-gh pr view <PR_N> --json state,mergedAt,title
-```
-
-### 3. Verify implementation
-
-For each linked or related PR that is merged, verify the fix is actually present on the current main branch:
-```bash
-git log --oneline main | grep -i "<keyword>"
-# or
-git log --oneline main --grep="<PR_N>"
-```
-
-If the issue describes a specific defect, check the relevant code to confirm the fix is in place:
-```bash
-grep -n "<pattern>" crates/<relevant>/src/<file>.rs
-```
-
-For issues with checklists, verify each item individually. If sub-items are tracked as separate issues, check those too:
-```bash
-gh issue view <SUB_N> --repo <owner/repo> --json state
-```
-
-### 4. Determine verdict
-
-- **All items fixed and merged**: Close with a summary comment listing what was fixed and which PRs.
-- **Some items fixed, some remaining**: Comment with status of each item. Do not close.
-- **Not yet implemented**: Comment with a summary of what remains. Do not close.
-- **Superseded or no longer relevant**: Close with explanation.
-
-### 5. Take action
-
-Close with comment:
-```bash
-gh issue close <N> --repo <owner/repo> --comment "<body>"
-```
-
-Comment without closing:
-```bash
-gh issue comment <N> --repo <owner/repo> --body-file /tmp/triage.md
-```
-
-Update issue labels if needed:
-```bash
-gh issue edit <N> --repo <owner/repo> --add-label "completed" --remove-label "needs-triage"
-```
-
-Always use `--body-file` for multiline content, never inline `--body`.
-
-### 6. Handle multi-issue batches
-
-When the user asks to check multiple issues (e.g., "check all issues by user X" or "scan backlog for closable issues"):
-1. List the issues: `gh issue list --repo <repo> --author <user> --state open --json number,title,updatedAt`
-2. For each issue, run steps 1-5 above.
-3. Report a summary table of all triaged issues with verdicts.
-
-## Output format
-
-### Issue Triage: #<N> — <title>
-
-**State**: OPEN / CLOSED
-**Linked PRs**: <list with merge status>
-
-#### Assessment
-<what was requested vs what is implemented>
-
-#### Verdict
-- Close — all items resolved by <PR list>
-- Keep open — <remaining items>
-- Not started — <what needs to be done>
-
-#### Action taken
-- Closed with comment / Commented / No action
-
-## Notes
-
-- The user may ask in Chinese ("是否可以关闭", "检查完成情况"); respond in the same language.
-- When closing, always include a summary of what was fixed and which PRs resolved it — this creates a useful audit trail.
-- For issues in `rustfs/backlog`, use `--repo rustfs/backlog`.
-- For issues in `rustfs/rustfs`, use `--repo rustfs/rustfs`.
-- If the issue has sub-issues (GitHub sub-issues API), check each one's state before declaring the parent complete.

.mimocode/skills/pr-review/SKILL.md

@@ -1,147 +0,0 @@
----
-name: pr-review
-description: Review a GitHub PR end-to-end from a URL or number — fetch metadata, inspect the diff, run multi-role adversarial review, check CI status, and post the review comment. Use when the user provides a PR link and asks to review it.
----
-
-# PR Review
-
-Use this skill when the user provides a GitHub PR URL or number and asks to review it. This covers the full review lifecycle: data gathering, code review, CI verification, and posting the result.
-
-## Prerequisites
-
-- Read `AGENTS.md` for the repository's adversarial validation policy and change-style rules.
-- The `adversarial-validation` skill handles the review role playbooks; this skill orchestrates the workflow around it.
-
-## Workflow
-
-### 1. Gather PR context
-
-```bash
-gh pr view <N> --json title,author,state,body,additions,deletions,changedFiles,commits,baseRefName,headRefName
-gh pr diff <N> --name-only
-```
-
-Read the PR body and linked issues to understand the change's purpose. If the PR references an issue, fetch that too:
-```bash
-gh issue view <ISSUE> --json title,body,state
-```
-
-### 2. Fetch the diff and classify the change
-
-```bash
-git fetch origin pull/<N>/head:pr-<N>
-git diff main...pr-<N> --stat
-```
-
-Classify the change by risk tier (per AGENTS.md):
-- **Exempt**: docs/comments/instruction-only, formatting, typos.
-- **Mechanical**: renames, file moves, test-only or tooling changes.
-- **Standard** (default): any behavior change.
-- **High risk**: locking, erasure coding, quorum/heal, replication, multipart, RPC, lifecycle/tiering, metadata formats, persistence/fsync, IAM/KMS/auth, on-disk/on-wire formats, S3 API-visible behavior.
-
-### 3. Cluster changed files and delegate review
-
-Group the changed files into logical clusters (by crate or functional area). For each cluster, spawn a subagent with a focused review prompt that includes:
-- The cluster's changed files and their diffs.
-- The applicable adversarial role probes (from the `adversarial-validation` skill).
-- The repository's AGENTS.md rules relevant to that domain.
-
-For standard-tier changes: correctness adversary + simplicity adversary + test-coverage skeptic, plus every role whose domain the diff touches.
-For high-risk changes: run all seven roles.
-
-Each subagent must produce findings (concrete failure scenario with file:line) or a null report ("attacked X, Y, Z — no break found").
-
-### 4. Check CI status
-
-```bash
-gh pr checks <N>
-```
-
-If any checks fail, investigate:
-```bash
-gh run view --log-failed --job=<JOB_ID>
-```
-
-Determine whether failures are pre-existing (on main), flaky, or caused by the PR.
-
-### 5. Synthesize findings
-
-Combine all subagent findings into a structured review:
-- **Summary**: one-paragraph overview of the change and overall assessment.
-- **Findings**: each finding with severity (critical/major/minor/nit), file:line, concrete failure scenario, and suggested fix.
-- **CI status**: pass/fail with notes on any failures.
-- **Verdict**: APPROVE, REQUEST_CHANGES, or COMMENT.
-
-### 6. Post the review
-
-Write the review body to a temp file and post via CLI:
-```bash
-# Request changes
-gh pr review <N> --request-changes --body-file /tmp/pr_review.md
-
-# Approve
-gh pr review <N> --approve --body-file /tmp/pr_review.md
-
-# Comment only (no verdict)
-gh pr review <N> --comment --body-file /tmp/pr_review.md
-```
-
-For inline comments on specific lines, use the GitHub API:
-```bash
-cat > /tmp/pr_review.json <<'EOF'
-{
-  "body": "review body",
-  "event": "REQUEST_CHANGES",
-  "comments": [
-    {
-      "path": "crates/foo/src/bar.rs",
-      "line": 42,
-      "body": "finding description"
-    }
-  ]
-}
-EOF
-gh api --method POST /repos/{owner}/{repo}/pulls/<N>/reviews --input /tmp/pr_review.json
-```
-
-Always use `--body-file` or `--input`, never inline multiline `--body`.
-
-### 7. Handle follow-up
-
-If the review requests changes:
-- Monitor for new commits: `gh pr view <N> --json commits`
-- Re-review changed files only: `git diff pr-<N>..origin/pull/<N>/head`
-- Update the review when findings are addressed.
-
-If CI was failing due to pre-existing main breakage:
-- Comment on the PR noting the failure is pre-existing.
-- Suggest updating the branch: `gh pr update-branch <N>`
-
-## Output format
-
-### PR Review: #<N> — <title>
-
-**Author**: <author>
-**Risk tier**: exempt | mechanical | standard | high-risk
-**Changed files**: <count> across <cluster count> clusters
-
-#### Summary
-<one-paragraph overview>
-
-#### Findings
-| Severity | Location | Finding |
-|----------|----------|---------|
-| critical | file:line | concrete failure scenario |
-
-#### CI Status
-- All checks pass / Failing: <details>
-
-#### Verdict
-APPROVE / REQUEST_CHANGES / COMMENT
-
-## Notes
-
-- The user may ask for review in Chinese; respond in the same language but keep the review body in English per AGENTS.md rules.
-- When the user asks for "多角色对抗 review", run the full adversarial validation protocol — this skill's step 3 covers that.
-- If the PR is from a fork, check `maintainerCanModify` before attempting to push fixes.
-- For very large PRs (>50 files), cluster aggressively and delegate in parallel to keep review time reasonable.

AGENTS.md

@@ -21,22 +21,6 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 - Avoid redundant file reads, repeated commands, and unnecessary exploratory work once enough context is available.
 - A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification that survives Adversarial Validation (below).
 
-## Worktree and Disk Hygiene
-
-- Unless the requester explicitly says otherwise, treat every new implementation task as isolated work: fetch the latest `origin/main`, confirm the requested change is not already present there, and create a dedicated feature branch and worktree from that exact upstream commit before editing. Do not implement new work directly in the primary checkout or reuse a worktree from another task.
-- Check available disk space before creating the worktree or starting dependency downloads, builds, tests, coverage, or other artifact-heavy commands. For long-running or artifact-heavy work, re-check disk usage at natural phase boundaries and before broad validation; if remaining space may not safely accommodate the next command, stop and reclaim task-owned artifacts before continuing.
-- Keep cleanup scoped and safe: remove generated build/test/coverage artifacts and temporary files created by the task when they are no longer needed, and never delete another task's worktree or uncommitted files. Prefer shared dependency caches where supported instead of duplicating large artifacts across worktrees.
-- At handoff, report the disk-space checks, cleanup performed, and any retained worktree or artifacts with the reason they are still needed.
-
-## PR Lifecycle Monitoring
-
-- Creating or updating a PR is not the terminal state. Unless the requester explicitly limits the task to PR creation, monitor the PR through its terminal state: merged, closed, or explicitly handed off because progress requires user or maintainer action.
-- While the task is active, monitor CI/check runs, review decisions and unresolved threads, mergeability and conflicts, and unexpected head/base changes. Prefer event-driven or bounded waits provided by the current environment over frequent polling; report only state changes, actionable failures, or meaningful prolonged delays.
-- Investigate every failing check and review comment before changing code. Fix failures attributable to the task, run the verification required for the new diff, push the update, respond to or resolve the corresponding review threads, and resume monitoring. Do not weaken checks, dismiss valid feedback, or retry flaky failures merely to obtain a green result.
-- Treat opening, green CI, approval, and mergeability as intermediate states. Never merge without the required reviewer approval or explicit authority. If progress depends on credentials, infrastructure, a maintainer decision, or another external action, report the exact blocker and the evidence already collected.
-- If the current execution environment cannot remain active until the next PR event, use a supported automation, monitor, or thread wakeup when available and within scope. Otherwise leave an explicit handoff containing the PR, current state, next event to observe, and pending cleanup; do not imply that background monitoring exists when none is scheduled.
-- After observing a merge, verify the commits are preserved on the upstream base, ensure the worktree is clean, remove the dedicated worktree, prune stale worktree metadata, and delete the local task branch when it is no longer in use. For a closed or abandoned PR, preserve any unmerged work unless deletion was explicitly authorized. Do not delete remote branches unless explicitly requested or repository automation owns that cleanup.
-
 ## Autonomy and Approval Boundaries
 
 - Inquiry tasks (answer, explain, review, diagnose, plan): report findings; do not change files unless a fix is explicitly requested.
@@ -121,78 +105,33 @@ CI) fails the build if anything is committed under `docs/superpowers/`, even via
 
 ## Verification Before PR
 
-Convert changes into independently verifiable outcomes. This section controls
-agent-run local validation; preparing a commit or PR does not by itself require
-the broadest gate. Inspect only the final task-owned diff, classify it by
-behavioral impact rather than line count or path alone, and run the smallest
-set of checks that provides meaningful coverage. Do not let unrelated
-worktree changes or a generic contributor checklist expand the scope.
-Non-exempt changes must also pass Adversarial Validation (next section) before
-the checks below count as completion.
+Convert changes into independently verifiable outcomes. Prefer focused tests for behavior changes and run the relevant checks before declaring completion.
+Non-exempt changes must also pass Adversarial Validation (next section) before the checks below count as completion.
 
-### Validation floor
+For code changes, run and pass the following before opening a PR:
 
-- Every change that is not documentation-only must finish with
-  `cargo fmt --all --check` passing. An umbrella gate that runs this exact
-  check satisfies the requirement; do not run it twice. Use `cargo fmt --all`
-  only when formatting needs to be fixed. Run the configured formatter or
-  validator for other changed languages when one exists.
-- Documentation-only or instruction-only means all task-owned changes are
-  prose or documentation assets and cannot affect runtime, builds, CI,
-  dependencies, generated code, or tests. Run `git diff --check` and any
-  relevant documentation guard, but skip Cargo formatting, compilation,
-  Clippy, tests, `make pre-commit`, and `make pre-pr`.
-- Behavior changes require relevant existing or new tests. Prefer the most
-  focused test or affected package. A passing targeted test can also provide
-  sufficient compilation coverage when it builds every changed target and
-  feature involved; do not add a redundant `cargo check` in that case.
-- `cargo check` supplements compilation coverage; it never substitutes for a
-  behavioral test. If a relevant test cannot reasonably be added or run, use
-  the narrowest compilation check and report the reason and remaining risk.
+```bash
+make pre-pr
+```
 
-### Validation tiers
+Before committing code changes, prefer focused verification for the touched
+surface and use the faster local gate when a broad smoke check is needed:
 
-1. **Documentation/instruction-only:** Apply the exemption above. Run a guard
-   such as `make doc-paths-check` only when it is relevant to the edited text.
-2. **Non-behavioral source change:** For comments, formatting, or another
-   demonstrably non-executable change, run the formatting floor. Compilation,
-   Clippy, and tests may be skipped only when the edit cannot affect
-   compilation or runtime behavior; run targeted doctests if executable
-   documentation examples changed.
-3. **Localized or bounded behavior change:** Run the formatting floor and the
-   narrowest relevant tests. Add package-scoped `cargo check` or Clippy only
-   for changed targets, features, APIs, error handling, async behavior, or
-   control flow not already covered. When several crates are affected but the
-   dependency set is identifiable, validate those packages and known
-   dependents instead of the whole workspace. Use `make pre-commit` only when
-   a repository-wide fast gate adds useful confidence beyond those checks.
-4. **Broad or high-risk change:** Run `make pre-pr` only when targeted coverage
-   cannot bound the impact, including:
-   - dependency, feature, build-script, procedural-macro, code-generation,
-     toolchain, or CI changes that alter compilation or the test matrix;
-   - cross-crate public APIs, shared foundational code, or broad refactors with
-     an unbounded dependent set;
-   - locking, storage durability or formats, erasure coding, replication,
-     RPC/protocol compatibility, IAM/KMS/auth, cryptography, or other
-     security-sensitive behavior;
-   - a targeted check that reveals wider impact, an explicit user request, or
-     a release policy that requires the full gate.
+```bash
+make pre-commit
+```
 
-Documentation-only and non-behavioral classifications take precedence over
-path-based triggers. A small diff can still be high-risk, while a CI comment,
-manifest comment, or release-note edit does not require full validation.
+For migration batches, do not run the full `make pre-pr` gate before every
+intermediate commit. Use focused tests and `make pre-commit` during
+development, then reserve `make pre-pr` for the final PR-ready branch.
 
-`make pre-pr` includes `make pre-commit` coverage. Never run both for the same
-unchanged diff, and do not repeat equivalent checks during PR preparation or
-because a local hook already ran them. Rerun only checks whose scope is affected
-by later edits. Full workspace checks do not replace a relevant integration or
-E2E test for changed behavior; run that focused test when required and
-available, or report why it was not run and the remaining risk.
+Before pushing code changes, make sure formatting is clean:
 
-If `make` is unavailable, run the equivalent checks defined under
-`.config/make/`. At handoff, list the checks actually run, checks intentionally
-skipped, and the reason for the selected tier.
+- Run `cargo fmt --all`.
+- Run `cargo fmt --all --check` and ensure no files are modified unexpectedly.
 
+If `make` is unavailable, run the equivalent checks defined under `.config/make/`.
+Documentation-only or instruction-only changes are exempt from the verification commands above (including the `.config/make/` equivalents), though any locally installed git pre-commit hooks may still run on commit unless explicitly skipped.
 After build-based verification completes, clean generated build artifacts before wrapping up to avoid unnecessary disk usage.
 Do not open a PR with code changes when the required checks fail.
 Make a failing check pass by fixing the cause, never by weakening the gate:
@@ -347,11 +286,6 @@ cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/file/xl.meta"
   absent, empty, and nil all mean "no value", never `Uuid::nil()`.
 - A remote-tier version of `None`/`""` means the tier bucket is unversioned:
   send **no** `versionId` on tier GET/DELETE.
-- Structs persisted in the scanner data-usage cache (`DataUsageCacheInfo`,
-  `DataUsageEntry`) carry a hand-written map-encoded `Serialize`. MessagePack
-  encodes derived structs as arrays, where an appended field makes the whole
-  cache a decode error for older readers — keep new fields `#[serde(default)]`
-  and keep the map encoding rather than reverting to `derive(Serialize)`.
 
 ## Naming Conventions
 

Cargo.lock

@@ -290,11 +290,11 @@ dependencies = [
 
 [[package]]
 name = "ar_archive_writer"
-version = "0.5.3"
+version = "0.5.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "73cd58deff2140a0a8eae87e417bd01db68a33e148aa93d1e8cd837e55e312b6"
+checksum = "4087686b4b0a3427190bae57a1d9a478dbb2d40c5dc1bd6e2b6d797913bdd348"
 dependencies = [
- "object 0.39.1",
+ "object 0.37.3",
 ]
 
 [[package]]
@@ -599,16 +599,6 @@ dependencies = [
  "syn 2.0.119",
 ]
 
-[[package]]
-name = "assert-json-diff"
-version = "2.0.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "47e4f2b81832e72834d7518d8487a0396a28cc408186a2e8854c0f98011faf12"
-dependencies = [
- "serde",
- "serde_json",
-]
-
 [[package]]
 name = "astral-tokio-tar"
 version = "0.6.4"
@@ -953,32 +943,6 @@ dependencies = [
  "uuid",
 ]
 
-[[package]]
-name = "aws-sdk-kms"
-version = "1.114.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c0b7d906608ee41e7ddea9983577ba82200435644d567d63dc34e822e088b453"
-dependencies = [
- "arc-swap",
- "aws-credential-types",
- "aws-runtime",
- "aws-smithy-async",
- "aws-smithy-http",
- "aws-smithy-json",
- "aws-smithy-observability",
- "aws-smithy-runtime",
- "aws-smithy-runtime-api",
- "aws-smithy-schema",
- "aws-smithy-types",
- "aws-types",
- "bytes",
- "fastrand",
- "http 0.2.12",
- "http 1.5.0",
- "regex-lite",
- "tracing",
-]
-
 [[package]]
 name = "aws-sdk-s3"
 version = "1.140.0"
@@ -1194,23 +1158,17 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "635d23afda0a6ab48d666c4d447c4873e8d1e83518a2be2093122397e50b838e"
 dependencies = [
  "aws-smithy-async",
- "aws-smithy-protocol-test",
  "aws-smithy-runtime-api",
  "aws-smithy-types",
- "bytes",
  "h2",
  "http 1.5.0",
- "http-body 1.1.0",
  "hyper",
  "hyper-rustls",
  "hyper-util",
- "indexmap 2.14.0",
  "pin-project-lite",
  "rustls",
  "rustls-native-certs",
  "rustls-pki-types",
- "serde",
- "serde_json",
  "tokio",
  "tokio-rustls",
  "tower",
@@ -1237,25 +1195,6 @@ dependencies = [
  "aws-smithy-runtime-api",
 ]
 
-[[package]]
-name = "aws-smithy-protocol-test"
-version = "0.64.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f76511a0e223ce78deb6a78b8afebda99cb737cfbc8a58d96dcb190f012dd40a"
-dependencies = [
- "assert-json-diff",
- "aws-smithy-runtime-api",
- "base64-simd",
- "cbor-diag",
- "ciborium",
- "http 0.2.12",
- "pretty_assertions",
- "regex-lite",
- "roxmltree",
- "serde_json",
- "thiserror 2.0.19",
-]
-
 [[package]]
 name = "aws-smithy-query"
 version = "0.62.0"
@@ -1740,9 +1679,9 @@ dependencies = [
 
 [[package]]
 name = "bytesize"
-version = "2.7.0"
+version = "2.4.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7354288c522e7e980fafd2075d63d1285794c3a6a16cdd492f189ea406e5f18b"
+checksum = "3d7c8918969267b2932ffd5655509bbbea0833823058c378876953217f5fc50e"
 
 [[package]]
 name = "bytestring"
@@ -1819,25 +1758,6 @@ dependencies = [
  "cipher 0.5.2",
 ]
 
-[[package]]
-name = "cbor-diag"
-version = "0.1.12"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "dc245b6ecd09b23901a4fbad1ad975701fd5061ceaef6afa93a2d70605a64429"
-dependencies = [
- "bs58",
- "chrono",
- "data-encoding",
- "half",
- "nom 7.1.3",
- "num-bigint",
- "num-rational",
- "num-traits",
- "separator",
- "url",
- "uuid",
-]
-
 [[package]]
 name = "cc"
 version = "1.4.0"
@@ -1968,9 +1888,9 @@ dependencies = [
 
 [[package]]
 name = "clap"
-version = "4.6.5"
+version = "4.6.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "301b56658598e48f3648647ac6fc887be7e7108eddfa4e9b63fcf3ec58c0cadf"
+checksum = "d91e0c145792ef73a6ad36d27c75ac09f1832222a3c209689d90f534685ee5b7"
 dependencies = [
  "clap_builder",
  "clap_derive",
@@ -1978,9 +1898,9 @@ dependencies = [
 
 [[package]]
 name = "clap_builder"
-version = "4.6.5"
+version = "4.6.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "94a65403d1a1bd28f7dc68eb8506e8874808ee5eecb59298de588e2e1407a078"
+checksum = "f09628afdcc538b57f3c6341e9c8e9970f18e4a481690a64974d7023bd33548b"
 dependencies = [
  "anstream",
  "anstyle",
@@ -3752,7 +3672,6 @@ dependencies = [
  "flate2",
  "futures",
  "hex",
- "hotpath",
  "http 1.5.0",
  "http-body-util",
  "hyper",
@@ -4453,9 +4372,9 @@ dependencies = [
 
 [[package]]
 name = "google-cloud-auth"
-version = "1.15.0"
+version = "1.14.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f54aab44c16b8463ae11b165a87c3d484780231f157bb1ed65843d591beb5abd"
+checksum = "a3494870d06f3cbbb3561ada6f234982549e3a2fb31e719ef258e6eadb9ae09a"
 dependencies = [
  "async-trait",
  "aws-lc-rs",
@@ -4482,9 +4401,9 @@ dependencies = [
 
 [[package]]
 name = "google-cloud-gax"
-version = "1.13.0"
+version = "1.12.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b9a46dd0fd026bbc4a5d84e6ab0c941cee6e3b057976a0bb107fdb5238ce598f"
+checksum = "3103a4a9013f1aed573ca56e19a9680b0211643a99ea85caf524b397d6be8be3"
 dependencies = [
  "bytes",
  "futures",
@@ -4501,9 +4420,9 @@ dependencies = [
 
 [[package]]
 name = "google-cloud-gax-internal"
-version = "0.7.16"
+version = "0.7.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fb04c54317ace06d489213f761797240b3046142a9b7ce6b9a82a9d134e193d1"
+checksum = "c0df265fba091ed7e00ecd0755009423310163f8b52820f007b6b4d97f4c6617"
 dependencies = [
  "bytes",
  "futures",
@@ -4605,9 +4524,9 @@ dependencies = [
 
 [[package]]
 name = "google-cloud-storage"
-version = "1.17.0"
+version = "1.16.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9227f65175fa91a6e41f246797917697efdadfe09dd8ea84ad8b737a71efbd28"
+checksum = "dc4b1d78c88db5c2530b12461e373a7d0d3a6caa3f6c1fc14e5d824cf2aeb307"
 dependencies = [
  "async-trait",
  "base64 0.22.1",
@@ -4658,9 +4577,9 @@ dependencies = [
 
 [[package]]
 name = "google-cloud-wkt"
-version = "1.7.0"
+version = "1.6.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7fccf98cfd5481a5f5a285181ab0c62123d7d47cd2bb7299448440649349e4e7"
+checksum = "46df1fcc3ab69164af3f4199ed21f45b5dbc56d9f03211eb4fa20116d442364b"
 dependencies = [
  "base64 0.22.1",
  "bytes",
@@ -4992,64 +4911,37 @@ dependencies = [
 
 [[package]]
 name = "hotpath"
-version = "0.23.0"
+version = "0.22.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ab303f15e2bbd9633a577338c9813a86bc1aef74beb8b536e27f28c80e84befc"
+checksum = "66750a77f4f6b408a148be5102ef1f3ba7172def7ee92b1cfc75d9f7a3870453"
 dependencies = [
  "arc-swap",
- "async-channel",
- "async-trait",
  "cfg-if",
  "crossbeam-channel",
- "flate2",
- "futures-channel",
  "futures-util",
  "hdrhistogram",
  "hotpath-macros",
- "hotpath-meta",
- "http 1.5.0",
  "libc",
- "object 0.36.7",
- "parking_lot",
  "pin-project-lite",
  "prettytable-rs",
  "quanta",
  "regex",
- "reqwest",
- "reqwest-middleware",
- "rustc-demangle",
  "serde",
  "serde_json",
  "tiny_http",
- "tokio",
 ]
 
 [[package]]
 name = "hotpath-macros"
-version = "0.23.0"
+version = "0.22.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4777d4dd3474c9b9c9391be713c6b570da0ac49e992a8cbfed67f60ca0f7e33d"
+checksum = "afe0e1900d2dbe2e2df8e9522b97ebd7a5598ba18478f57e247957022dffedbe"
 dependencies = [
  "proc-macro2",
  "quote",
  "syn 2.0.119",
 ]
 
-[[package]]
-name = "hotpath-macros-meta"
-version = "0.23.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "833200923e0ba8150fb91d6a3a39643eef95e604c7394c2f2679905cae13862c"
-
-[[package]]
-name = "hotpath-meta"
-version = "0.23.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "eca34dbbafc05f5da2a2696ce2640018fbe29e9932736309efbf3a990f0b2832"
-dependencies = [
- "hotpath-macros-meta",
-]
-
 [[package]]
 name = "htmlescape"
 version = "0.3.1"
@@ -5131,9 +5023,9 @@ checksum = "15cdd26707701c53297e2fa6afb323d55fbc1d0810c3aec078ae3ef0424c3c15"
 
 [[package]]
 name = "hybrid-array"
-version = "0.4.14"
+version = "0.4.13"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "707114b52a152fa7bdb290cd7cd5912d9467273b6d74e21b8d81aca1f8533f6b"
+checksum = "818356c5132c1fede50f837ca96afbe78ff42413047f4abb886217845e1b6c8c"
 dependencies = [
  "ctutils",
  "subtle",
@@ -5445,9 +5337,9 @@ dependencies = [
 
 [[package]]
 name = "ipnet"
-version = "2.12.1"
+version = "2.12.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6a756c3fac73139e83f14c2d742155dd2b78d3ee56597b419a0579b7bdd6dd78"
+checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2"
 dependencies = [
  "serde",
 ]
@@ -5915,7 +5807,8 @@ checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981"
 [[package]]
 name = "libmimalloc-sys"
 version = "0.1.49"
-source = "git+https://github.com/xonatius/mimalloc_rust.git?rev=1cdadea43e9c5a0f054b65be21200ce580e4eb13#1cdadea43e9c5a0f054b65be21200ce580e4eb13"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6a45a52f43e1c16f667ccfe4dd8c85b7f7c204fd5e3bf46c5b0db9a5c3c0b8e9"
 dependencies = [
  "cc",
  "cty",
@@ -5923,9 +5816,9 @@ dependencies = [
 
 [[package]]
 name = "libredox"
-version = "0.1.19"
+version = "0.1.18"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2026a5056764a10b2bf5d56488cba40da507f5493a6a429340e2004d9ed085fa"
+checksum = "c943259e342f1e06ff2da7a83eabdfe7f92ce10262688dbf1895ff0b3e6e4652"
 dependencies = [
  "libc",
 ]
@@ -6324,7 +6217,8 @@ dependencies = [
 [[package]]
 name = "mimalloc"
 version = "0.1.52"
-source = "git+https://github.com/xonatius/mimalloc_rust.git?rev=1cdadea43e9c5a0f054b65be21200ce580e4eb13#1cdadea43e9c5a0f054b65be21200ce580e4eb13"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2d4139bb28d14ad1facf21d5eb8825051b326e172d216b39f6d31df53cc97862"
 dependencies = [
  "libmimalloc-sys",
 ]
@@ -6746,17 +6640,6 @@ dependencies = [
  "num-traits",
 ]
 
-[[package]]
-name = "num-rational"
-version = "0.4.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824"
-dependencies = [
- "num-bigint",
- "num-integer",
- "num-traits",
-]
-
 [[package]]
 name = "num-traits"
 version = "0.2.19"
@@ -6821,7 +6704,7 @@ version = "5.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "51e219e79014df21a225b1860a479e2dcd7cbd9130f4defd4bd0e191ea31d67d"
 dependencies = [
- "base64 0.22.1",
+ "base64 0.21.7",
  "chrono",
  "getrandom 0.2.17",
  "http 1.5.0",
@@ -6900,15 +6783,6 @@ dependencies = [
  "objc2-core-foundation",
 ]
 
-[[package]]
-name = "object"
-version = "0.36.7"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "62948e14d923ea95ea2c7c86c71013138b66525b86bdc08d2dcc262bdb497b87"
-dependencies = [
- "memchr",
-]
-
 [[package]]
 name = "object"
 version = "0.37.3"
@@ -7946,7 +7820,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
 dependencies = [
  "heck",
- "itertools 0.14.0",
+ "itertools 0.10.5",
  "log",
  "multimap",
  "once_cell",
@@ -7966,7 +7840,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042"
 dependencies = [
  "heck",
- "itertools 0.14.0",
+ "itertools 0.10.5",
  "log",
  "multimap",
  "petgraph 0.8.3",
@@ -7987,7 +7861,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d"
 dependencies = [
  "anyhow",
- "itertools 0.14.0",
+ "itertools 0.10.5",
  "proc-macro2",
  "quote",
  "syn 2.0.119",
@@ -8000,7 +7874,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf"
 dependencies = [
  "anyhow",
- "itertools 0.14.0",
+ "itertools 0.10.5",
  "proc-macro2",
  "quote",
  "syn 2.0.119",
@@ -8036,9 +7910,9 @@ dependencies = [
 
 [[package]]
 name = "psm"
-version = "0.1.32"
+version = "0.1.31"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4dcd034599e63b970727f70d79e02d62390a4a84f7c6b827c27c46d5ac3fa622"
+checksum = "645dbe486e346d9b5de3ef16ede18c26e6c70ad97418f4874b8b1889d6e761ea"
 dependencies = [
  "ar_archive_writer",
  "cc",
@@ -8640,21 +8514,6 @@ dependencies = [
  "web-sys",
 ]
 
-[[package]]
-name = "reqwest-middleware"
-version = "0.5.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "07bc3f1384cffa4f274dad2d4ddd73aed32fed8f786d96c6be8aa4e5fd3c3b58"
-dependencies = [
- "anyhow",
- "async-trait",
- "http 1.5.0",
- "reqwest",
- "serde",
- "thiserror 2.0.19",
- "tower-service",
-]
-
 [[package]]
 name = "resolv-conf"
 version = "0.7.6"
@@ -8720,15 +8579,6 @@ dependencies = [
  "serde",
 ]
 
-[[package]]
-name = "roxmltree"
-version = "0.14.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "921904a62e410e37e215c40381b7117f830d9d89ba60ab5236170541dd25646b"
-dependencies = [
- "xmlparser",
-]
-
 [[package]]
 name = "rsa"
 version = "0.9.10"
@@ -8822,9 +8672,9 @@ dependencies = [
 
 [[package]]
 name = "russh"
-version = "0.62.5"
+version = "0.62.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "da7c230e0ed9cbeb92fbad6c8848985d6df2a1464c0dc247a021abd666e9005e"
+checksum = "b8b67b5a0d8068c89dcbe9d95df986af7a851d1f3c604525274c37468e60464f"
 dependencies = [
  "aes 0.9.2",
  "aws-lc-rs",
@@ -9125,7 +8975,6 @@ dependencies = [
  "url",
  "urlencoding",
  "uuid",
- "zeroize",
  "zip",
  "zstd",
 ]
@@ -9135,11 +8984,10 @@ name = "rustfs-audit"
 version = "1.0.0-beta.12"
 dependencies = [
  "async-trait",
+ "chrono",
  "const-str",
  "futures",
  "hashbrown 0.17.1",
- "hotpath",
- "jiff",
  "metrics",
  "rustfs-config",
  "rustfs-s3-types",
@@ -9160,7 +9008,6 @@ dependencies = [
  "base64-simd",
  "bytes",
  "crc-fast",
- "hotpath",
  "http 1.5.0",
  "md-5 0.11.0",
  "pretty_assertions",
@@ -9174,7 +9021,6 @@ name = "rustfs-common"
 version = "1.0.0-beta.12"
 dependencies = [
  "chrono",
- "hotpath",
  "metrics",
  "rmp-serde",
  "s3s",
@@ -9189,7 +9035,6 @@ dependencies = [
 name = "rustfs-concurrency"
 version = "1.0.0-beta.12"
 dependencies = [
- "hotpath",
  "insta",
  "rustfs-io-core",
  "serde",
@@ -9203,7 +9048,6 @@ name = "rustfs-config"
 version = "1.0.0-beta.12"
 dependencies = [
  "const-str",
- "hotpath",
  "serde",
  "serde_json",
 ]
@@ -9214,7 +9058,6 @@ version = "1.0.0-beta.12"
 dependencies = [
  "base64-simd",
  "hmac 0.13.0",
- "hotpath",
  "rand 0.10.2",
  "serde",
  "serde_json",
@@ -9230,7 +9073,6 @@ dependencies = [
  "argon2",
  "base64-simd",
  "chacha20poly1305",
- "hotpath",
  "jsonwebtoken 11.0.0",
  "pbkdf2 0.13.0",
  "rand 0.10.2",
@@ -9248,7 +9090,6 @@ name = "rustfs-data-usage"
 version = "1.0.0-beta.12"
 dependencies = [
  "async-trait",
- "hotpath",
  "rmp-serde",
  "rustfs-filemeta",
  "serde",
@@ -9258,6 +9099,7 @@ dependencies = [
 name = "rustfs-ecstore"
 version = "1.0.0-beta.12"
 dependencies = [
+ "aes-gcm",
  "arc-swap",
  "async-channel",
  "async-recursion",
@@ -9273,6 +9115,7 @@ dependencies = [
  "byteorder",
  "bytes",
  "bytesize",
+ "chacha20poly1305",
  "chrono",
  "criterion",
  "enumset",
@@ -9326,6 +9169,7 @@ dependencies = [
  "rustfs-erasure-codec",
  "rustfs-filemeta",
  "rustfs-io-metrics",
+ "rustfs-kms",
  "rustfs-lifecycle",
  "rustfs-lock",
  "rustfs-madmin",
@@ -9394,7 +9238,6 @@ dependencies = [
 name = "rustfs-extension-schema"
 version = "1.0.0-beta.12"
 dependencies = [
- "hotpath",
  "serde",
  "serde_json",
  "thiserror 2.0.19",
@@ -9417,7 +9260,6 @@ dependencies = [
  "rustfs-utils",
  "s3s",
  "serde",
- "serde_json",
  "tempfile",
  "thiserror 2.0.19",
  "time",
@@ -9434,7 +9276,6 @@ dependencies = [
  "async-trait",
  "base64 0.23.0",
  "futures",
- "hotpath",
  "http 1.5.0",
  "metrics",
  "rustfs-common",
@@ -9466,7 +9307,6 @@ dependencies = [
  "async-trait",
  "base64-simd",
  "futures",
- "hotpath",
  "http 1.5.0",
  "jsonwebtoken 11.0.0",
  "moka",
@@ -9501,7 +9341,6 @@ name = "rustfs-io-core"
 version = "1.0.0-beta.12"
 dependencies = [
  "bytes",
- "hotpath",
  "memmap2",
  "rustfs-io-metrics",
  "thiserror 2.0.19",
@@ -9514,13 +9353,10 @@ name = "rustfs-io-metrics"
 version = "1.0.0-beta.12"
 dependencies = [
  "criterion",
- "hotpath",
  "metrics",
  "metrics-util",
  "num_cpus",
- "rustfs-common",
  "rustfs-s3-ops",
- "rustfs-utils",
  "sysinfo",
  "thiserror 2.0.19",
  "tokio",
@@ -9581,7 +9417,6 @@ version = "1.0.0-beta.12"
 dependencies = [
  "bytes",
  "futures",
- "hotpath",
  "http 1.5.0",
  "http-body 1.1.0",
  "http-body-util",
@@ -9607,32 +9442,20 @@ name = "rustfs-kms"
 version = "1.0.0-beta.12"
 dependencies = [
  "aes-gcm",
- "anyhow",
  "arc-swap",
  "argon2",
  "async-trait",
- "aws-config",
- "aws-sdk-kms",
- "aws-smithy-http-client",
- "aws-smithy-runtime-api",
- "aws-smithy-types",
  "base64 0.23.0",
  "chacha20poly1305",
  "hex",
- "hotpath",
- "http 1.5.0",
  "insta",
  "jiff",
  "md-5 0.11.0",
- "metrics",
- "metrics-util",
  "moka",
  "rand 0.10.2",
  "reqwest",
- "rustfs-s3-types",
  "rustfs-security-governance",
  "rustfs-utils",
- "rustify",
  "serde",
  "serde_json",
  "sha2 0.11.0",
@@ -9641,9 +9464,7 @@ dependencies = [
  "tempfile",
  "thiserror 2.0.19",
  "tokio",
- "tokio-util",
  "tracing",
- "tracing-subscriber",
  "url",
  "uuid",
  "vaultrs",
@@ -9655,7 +9476,6 @@ name = "rustfs-lifecycle"
 version = "1.0.0-beta.12"
 dependencies = [
  "async-trait",
- "hotpath",
  "metrics",
  "metrics-util",
  "proptest",
@@ -9680,7 +9500,6 @@ dependencies = [
  "async-trait",
  "crossbeam-queue",
  "futures",
- "hotpath",
  "parking_lot",
  "rand 0.10.2",
  "rustfs-io-metrics",
@@ -9702,7 +9521,6 @@ version = "1.0.0-beta.12"
 dependencies = [
  "chrono",
  "flate2",
- "hotpath",
  "regex",
  "serde",
  "serde_json",
@@ -9720,7 +9538,6 @@ name = "rustfs-madmin"
 version = "1.0.0-beta.12"
 dependencies = [
  "chrono",
- "hotpath",
  "humantime",
  "hyper",
  "rmp-serde",
@@ -9737,11 +9554,10 @@ dependencies = [
  "arc-swap",
  "async-trait",
  "axum",
+ "chrono",
  "criterion",
  "form_urlencoded",
  "hashbrown 0.17.1",
- "hotpath",
- "jiff",
  "metrics",
  "percent-encoding",
  "quick-xml",
@@ -9771,7 +9587,6 @@ version = "1.0.0-beta.12"
 dependencies = [
  "criterion",
  "futures",
- "hotpath",
  "rustfs-config",
  "rustfs-io-metrics",
  "rustfs-utils",
@@ -9790,7 +9605,6 @@ version = "1.0.0-beta.12"
 dependencies = [
  "bytes",
  "criterion",
- "hotpath",
  "metrics",
  "metrics-util",
  "moka",
@@ -9813,10 +9627,8 @@ dependencies = [
  "flate2",
  "futures-util",
  "glob",
- "hotpath",
  "jiff",
  "libc",
- "log",
  "metrics",
  "num_cpus",
  "nvml-wrapper",
@@ -9852,7 +9664,6 @@ dependencies = [
  "tracing-error",
  "tracing-opentelemetry",
  "tracing-subscriber",
- "url",
  "zstd",
 ]
 
@@ -9862,10 +9673,9 @@ version = "1.0.0-beta.12"
 dependencies = [
  "async-trait",
  "base64-simd",
+ "chrono",
  "futures",
- "hotpath",
  "ipnetwork",
- "jiff",
  "jsonwebtoken 11.0.0",
  "moka",
  "pollster",
@@ -9884,7 +9694,6 @@ dependencies = [
  "time",
  "tokio",
  "tracing",
- "tracing-subscriber",
 ]
 
 [[package]]
@@ -9902,7 +9711,6 @@ dependencies = [
  "futures-util",
  "hex",
  "hmac 0.13.0",
- "hotpath",
  "http 1.5.0",
  "http-body-util",
  "hyper",
@@ -9955,7 +9763,6 @@ name = "rustfs-protos"
 version = "1.0.0-beta.12"
 dependencies = [
  "flatbuffers",
- "hotpath",
  "prost 0.14.4",
  "rmp-serde",
  "rustfs-common",
@@ -9980,7 +9787,6 @@ version = "1.0.0-beta.12"
 dependencies = [
  "byteorder",
  "bytes",
- "hotpath",
  "regex",
  "rmp",
  "rmp-serde",
@@ -10039,7 +9845,6 @@ dependencies = [
  "chacha20poly1305",
  "hex",
  "hmac 0.13.0",
- "hotpath",
  "minlz",
  "pin-project-lite",
  "rand 0.10.2",
@@ -10057,7 +9862,6 @@ dependencies = [
 name = "rustfs-s3-ops"
 version = "1.0.0-beta.12"
 dependencies = [
- "hotpath",
  "rustfs-s3-types",
 ]
 
@@ -10065,7 +9869,6 @@ dependencies = [
 name = "rustfs-s3-types"
 version = "1.0.0-beta.12"
 dependencies = [
- "hotpath",
  "serde",
  "serde_json",
 ]
@@ -10080,7 +9883,6 @@ dependencies = [
  "datafusion",
  "futures",
  "futures-core",
- "hotpath",
  "http 1.5.0",
  "metrics",
  "parking_lot",
@@ -10091,6 +9893,7 @@ dependencies = [
  "s3s",
  "serde_json",
  "serial_test",
+ "tempfile",
  "thiserror 2.0.19",
  "tokio",
  "tokio-util",
@@ -10108,7 +9911,6 @@ dependencies = [
  "datafusion",
  "derive_builder",
  "futures",
- "hotpath",
  "parking_lot",
  "rustfs-s3select-api",
  "s3s",
@@ -10126,7 +9928,6 @@ dependencies = [
  "futures",
  "hex-simd",
  "hmac 0.13.0",
- "hotpath",
  "http 1.5.0",
  "metrics",
  "rand 0.10.2",
@@ -10137,7 +9938,6 @@ dependencies = [
  "rustfs-data-usage",
  "rustfs-ecstore",
  "rustfs-filemeta",
- "rustfs-lock",
  "rustfs-storage-api",
  "rustfs-utils",
  "s3s",
@@ -10160,7 +9960,6 @@ dependencies = [
 name = "rustfs-security-governance"
 version = "1.0.0-beta.12"
 dependencies = [
- "hotpath",
  "thiserror 2.0.19",
 ]
 
@@ -10170,7 +9969,6 @@ version = "1.0.0-beta.12"
 dependencies = [
  "base64-simd",
  "bytes",
- "hotpath",
  "http 1.5.0",
  "hyper",
  "rustfs-utils",
@@ -10187,7 +9985,6 @@ name = "rustfs-storage-api"
 version = "1.0.0-beta.12"
 dependencies = [
  "async-trait",
- "hotpath",
  "insta",
  "rustfs-filemeta",
  "serde",
@@ -10204,14 +10001,13 @@ dependencies = [
  "arc-swap",
  "async-nats",
  "async-trait",
+ "chrono",
  "criterion",
  "deadpool-postgres",
  "futures-util",
  "hashbrown 0.17.1",
- "hotpath",
  "hyper",
  "hyper-rustls",
- "jiff",
  "lapin",
  "libc",
  "metrics",
@@ -10255,7 +10051,6 @@ dependencies = [
 name = "rustfs-test-utils"
 version = "1.0.0-beta.12"
 dependencies = [
- "hotpath",
  "rustfs-data-usage",
  "rustfs-ecstore",
  "rustfs-storage-api",
@@ -10272,7 +10067,6 @@ name = "rustfs-tls-runtime"
 version = "1.0.0-beta.12"
 dependencies = [
  "arc-swap",
- "hotpath",
  "metrics",
  "rcgen",
  "rustfs-common",
@@ -10294,7 +10088,6 @@ version = "1.0.0-beta.12"
 dependencies = [
  "async-trait",
  "axum",
- "hotpath",
  "http 1.5.0",
  "ipnetwork",
  "metrics",
@@ -10341,7 +10134,6 @@ dependencies = [
  "hex-simd",
  "highway",
  "hmac 0.13.0",
- "hotpath",
  "http 1.5.0",
  "hyper",
  "local-ip-address",
@@ -10374,7 +10166,6 @@ dependencies = [
  "astral-tokio-tar",
  "async-compression",
  "criterion",
- "hotpath",
  "tempfile",
  "thiserror 2.0.19",
  "tokio",
@@ -10771,12 +10562,6 @@ dependencies = [
  "serde_core",
 ]
 
-[[package]]
-name = "separator"
-version = "0.4.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f97841a747eef040fcd2e7b3b9a220a7205926e60488e673d9e4926d27772ce5"
-
 [[package]]
 name = "seq-macro"
 version = "0.3.6"
@@ -11405,9 +11190,9 @@ checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596"
 
 [[package]]
 name = "stacker"
-version = "0.1.25"
+version = "0.1.24"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "707f49d46706bacf8a2b00d51dace3f9de527c13eec3778f570c411f89e69967"
+checksum = "640c8cdd92b6b12f5bcb1803ca3bbf5ab96e5e6b6b96b9ab77dabe9e880b3190"
 dependencies = [
  "cc",
  "cfg-if",
@@ -11714,7 +11499,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
 dependencies = [
  "fastrand",
- "getrandom 0.4.3",
+ "getrandom 0.3.4",
  "once_cell",
  "rustix",
  "windows-sys 0.61.2",
@@ -11824,9 +11609,9 @@ dependencies = [
 
 [[package]]
 name = "time"
-version = "0.3.55"
+version = "0.3.54"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cdb87b95ec50ddfa440816d227a17b2ccbdda963a316a727fda0fc4334f7d134"
+checksum = "3e1d5e639ff6bab73cb6885cc7e7b1de96c3f32c68ec55f3952614bec1092244"
 dependencies = [
  "deranged",
  "js-sys",

Cargo.toml

@@ -173,7 +173,7 @@ tower-http = { version = "0.7.0" }
 # Serialization and Data Formats
 apache-avro = "0.21.0"
 bytes = { version = "1.12.1" }
-bytesize = "2.7.0"
+bytesize = "2.4.2"
 byteorder = "1.5.0"
 flatbuffers = "25.12.19"
 form_urlencoded = "1.2.2"
@@ -212,7 +212,7 @@ zeroize = { version = "1.9.0" }
 chrono = { version = "0.4.45" }
 humantime = "2.4.0"
 jiff = { version = "0.2.35" }
-time = { version = "0.3.55" }
+time = { version = "0.3.54" }
 
 # Database
 deadpool-postgres = { version = "0.14" }
@@ -227,7 +227,6 @@ atoi = "3.1.0"
 atomic_enum = "0.3.0"
 aws-config = { version = "1.10.1" }
 aws-credential-types = { version = "1.3.0" }
-aws-sdk-kms = { default-features = false, version = "1.114.0" }
 aws-sdk-s3 = { default-features = false, version = "1.140.0" }
 aws-sdk-sts = { default-features = false, version = "1.110.0" }
 aws-smithy-http-client = { default-features = false, version = "1.2.0" }
@@ -236,7 +235,7 @@ aws-smithy-types = { version = "1.6.1" }
 base64 = "0.23.0"
 base64-simd = "0.8.0"
 brotli = "8.0.4"
-clap = { version = "4.6.5" }
+clap = { version = "4.6.4" }
 const-str = { version = "1.1.0" }
 convert_case = "0.11.0"
 criterion = { version = "0.8" }
@@ -251,8 +250,8 @@ enumset = "1.1.14"
 faster-hex = "0.10.0"
 flate2 = "1.1.9"
 glob = "0.3.4"
-google-cloud-storage = "1.17.0"
-google-cloud-auth = "1.15.0"
+google-cloud-storage = "1.16.0"
+google-cloud-auth = "1.14.0"
 hashbrown = { version = "0.17.1" }
 hex = "0.4.3"
 hex-simd = "0.8.0"
@@ -274,6 +273,7 @@ num_cpus = { version = "1.17.0" }
 nvml-wrapper = "0.12.1"
 parking_lot = "0.12.5"
 path-absolutize = "4.0.1"
+path-clean = "1.0.1"
 percent-encoding = "2.3.2"
 pin-project-lite = "0.2.17"
 pretty_assertions = "1.4.1"
@@ -285,7 +285,6 @@ reed-solomon-simd = "3.1.0"
 regex = { version = "1.13.1" }
 rumqttc = { package = "rumqttc-next", version = "0.33.3" }
 redis = { version = "1.5.0" }
-rustify = { version = "0.7", default-features = false }
 rustix = { version = "1.1.4" }
 rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }
@@ -340,16 +339,15 @@ libunftp = { version = "0.23.0" }
 unftp-core = "0.1.0"
 suppaftp = { version = "10.0.1" }
 rcgen = { version = "0.14.8", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
-russh = { version = "0.62.5" }
+russh = { version = "0.62.4" }
 russh-sftp = "2.3.0"
 
 # WebDAV
 dav-server = "0.11.0"
 
 # Performance Analysis and Memory Profiling
-mimalloc = { version = "0.1.52", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "1cdadea43e9c5a0f054b65be21200ce580e4eb13" }
-libmimalloc-sys = { version = "0.1.49", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "1cdadea43e9c5a0f054b65be21200ce580e4eb13", features = ["extended"] }
-hotpath = { version = "0.23.0", default-features = false }
+mimalloc = "0.1.52"
+hotpath = "0.22.0"
 # Snapshot testing for output format regression detection
 insta = { version = "1.48" }
 

MINIO_REPLICATION_COMPAT_REVIEW.md

@@ -1,314 +0,0 @@
-# RustFS 站点复制 / 桶复制 — MinIO 兼容性审查报告
-
-> 审查日期:2026-08-05
-> 审查对象:RustFS(worktree `reatang/minio-compatibility-review-03a7fb`)vs MinIO(`/Users/tang/Documents/GitHub/minio`)
-> 审查方式:白盒代码对比(5 个维度并行审查)+ P0 问题对抗性复核
-> 审查维度:站点复制白盒对比、桶复制白盒对比、mc 工具兼容性、S3 标准协议兼容性、代码结构与分层
-
----
-
-## 一、总体结论
-
-| 领域 | 兼容性评价 |
-|---|---|
-| **站点复制(RustFS↔RustFS + mc 管理)** | 良好。admin 端点全覆盖、JSON 结构对齐 madmin-go、请求体 DARE 加密兼容,mc admin replicate 全家桶基本可用 |
-| **站点复制(RustFS↔MinIO 混合组网)** | **断裂**。4 个 P0:出站 join 路径 404、metainfo 大小写解析失败、STS item 类型名不一致、policy-mapping userType 数值错位 |
-| **桶复制(控制面,S3 标准 API)** | 良好。Put/Get/DeleteBucketReplication、错误码、状态机字符串、xl.meta 内部键均对齐 |
-| **桶复制(数据面,RustFS→MinIO)** | **断裂**。复制 PUT 缺 `?versionId=` 导致目标端版本漂移(P0);CopyObject 完全不复制(P0) |
-| **mc 桶复制命令** | **部分断裂**。`mc replicate add` 默认参数即失败(P0);status/resync/backlog 响应结构不匹配导致静默空输出(P1) |
-| **代码结构** | 桶复制侧迁移架构有纪律但成本高;**站点复制侧无领域层,约 9500 行业务逻辑堆在 admin handler,且存在 3 处反向依赖违反项目分层不变量(P0)** |
-
-**做得好的地方**(已确认兼容,无需整改):复制状态机字符串(PENDING/COMPLETED/FAILED/REPLICA 含 legacy COMPLETE)、xl.meta 内部键双前缀(x-rustfs-internal- + x-minio-internal-)读写、ReplicateDecision 内部状态串格式、复制内部头主链路双前缀、Delete/VersionPurge 语义、Resync reset-id 判定、admin 路由 `/minio/admin/v3` 前缀别名、madmin DARE 加密流解密、站点复制 gob netperf 编码、`site-repl-<deploymentID>` 规则模板。
-
----
-
-## 二、P0 问题清单(8 项)
-
-| # | 问题 | 来源维度 | 断裂方向 |
-|---|---|---|---|
-| P0-1 | 出站 peer join 使用 MinIO 已移除的遗留路径 `/site-replication/join` → 404 | 站点复制 | RustFS→MinIO |
-| P0-2 | 解析 MinIO metainfo(SRInfo)字段大小写不匹配 → add preflight 失败 | 站点复制 | RustFS→MinIO |
-| P0-3 | STS 凭证复制 item 类型名 `sts-credential` vs `sts-account` | 站点复制 | 双向 |
-| P0-4 | policy-mapping `userType` 数值语义错位(RustFS: None=0/Svc=1/Sts=2/Reg=3;MinIO: reg=0/sts=1/svc=2)→ 权限静默漂移 | 站点复制 | 双向 |
-| P0-5 | 复制 PUT/CompleteMultipart 不携带 `?versionId=` query → MinIO 端版本号漂移、版本删除永久 no-op、双端静默发散(**功能视角复核:定级调整为 P1**,问题重述为"普通复制对象缺少可靠的源→目标版本身份策略";versionId query 是可行修复之一而非唯一正确方案) | 桶复制 | RustFS→MinIO |
-| P0-6 | CopyObject(含 metadata-replace 自拷贝)完全不触发复制调度,对象静默不复制(**功能视角复核:定级调整为 P1**;scanner 在 ExistingObjectReplication 启用+状态为空时可最终补齐,但同步复制语义失效,且继承 stale COMPLETED / 显式 Disabled 场景长期漏复制) | 桶复制 + S3 协议 | 所有方向 |
-| P0-7 | `mc replicate add` 默认参数(healthcheck-seconds=60)被硬拒 400;且字段单位按秒解析而 wire 为纳秒 | mc 兼容 | mc→RustFS |
-| P0-8 | 架构:站点复制约 9500 行业务逻辑堆在 admin handler 单文件;app/storage 层 3 处反向导入 admin 层,违反 ARCHITECTURE.md 分层不变量 #1(**对抗复核后降级为 P1**:反向边已被 arch 守卫棘轮基线锁死,属受控技术债) | 代码结构 | — |
-
-每项 P0 的对抗性复核结论、验证方案与解决方案见 **第五节**。
-
-**修复状态(2026-08-05)**:7 项确认 P0 已全部修复并创建 PR(红灯→绿灯 TDD):P0-1 [#5748](https://github.com/rustfs/rustfs/pull/5748)、P0-2 [#5749](https://github.com/rustfs/rustfs/pull/5749)、P0-3 [#5750](https://github.com/rustfs/rustfs/pull/5750)、P0-4 [#5751](https://github.com/rustfs/rustfs/pull/5751)、P0-5 [#5752](https://github.com/rustfs/rustfs/pull/5752)、P0-6+P1-10 [#5753](https://github.com/rustfs/rustfs/pull/5753)、P0-7 [#5754](https://github.com/rustfs/rustfs/pull/5754)。合并顺序:#5748+#5749 同批;#5752 先于 #5753。
-
----
-
-## 三、P1 问题清单
-
-### 站点复制
-
-| # | 问题 | 证据 | 影响 |
-|---|---|---|---|
-| P1-1 | ILM(lc-config)复制语义:对外开关限定 `replicateILMExpiry`,但发送端把**完整** lifecycle.xml 放入 `expiry_lc_config`,接收端整体覆盖/删除本地配置(功能视角复核:**确认,维持 P1**;更新时间检查只能拒旧,不能修复整体覆盖语义) | RustFS `bucket_meta.rs:948-951`、`site_replication.rs:7590-7683` vs MinIO `site-replication.go:1784-1810,6138` | lifecycle 同时含 expiry 与本地 transition 时,非 expiry 规则被错误传播或本地 transition 被覆盖。**缺"同步 expiry 后保留本地 transition"测试** |
-| ~~P1-2~~→**P2-25** | `SRInfo.ilmExpiryRules` 从不填充,ILM 一致性状态恒为空(功能视角复核:**降级 P2**——仅影响管理面可观测性,不改变对象数据) | `site_replication.rs:4152-4266,4855-4868` | `mc admin replicate status --ilm-expiry-rules` 恒空,ILM 漂移不可见 |
-| P1-3 | 无自动跨站元数据 heal(MinIO 有周期 heal 协程) | RustFS 仅 600s 本地 wiring 修复(`site_replication_reconcile.rs:34,59-81`)+ 手动 repair 端点 vs MinIO `site-replication.go:4257-4288` | 错过的 IAM/bucket 元数据更新持续漂移,须手工 repair |
-| P1-4(拆分) | ①`sync` 同步复制指控:功能视角复核**不成立/证据不足**——RustFS 自身契约明确将 `sync_state` 定义为站点可达性/配置完整性健康状态且有测试,不能以他家同名字段判其错误(属"RustFS 独特设计保持不变"项,撤销);②`defaultbandwidth`:**确认,降级 P2**——公共 API 接受并持久化,但建 site replication bucket target 时不应用,reconcile 只保留既有 `bandwidth_limit`,配置成功但不生效 | `site_replication.rs:6303-6357,5004-5027` | ②为用户可见的"配置成功但无效"能力缺口 |
-
-### 桶复制 / S3 协议
-
-| # | 问题 | 证据 | 影响 |
-|---|---|---|---|
-| P1-5 | 未复制完成对象的 GET/HEAD 远端 proxy 未实现;也不识别 MinIO 的 `X-Minio-Source-Proxy-Request` 防环头 | 仅指标占位(`storage_api.rs:799-804`);`SUFFIX_SOURCE_PROXY_REQUEST` 定义后无人使用 vs MinIO `bucket-replication.go:2334,2409,2534` | active-active 复制滞后窗口内 RustFS 端 404 |
-| P1-6 | `X-Minio-Source-Replication-{Tagging,Retention,LegalHold}-Timestamp` 三个时间戳头收发均缺失 | `replication_target_boundary.rs:251-297` 填了 options 但 `PutObjectOptions::header()` 不序列化;接收端不解析 vs MinIO `object-api-options.go:377-399` | active-active 下标签/retention/legal-hold 并发修改的 LWW 冲突解析退化,可能元数据回滚 |
-| P1-7 | ARN 前缀 `arn:rustfs:` 与 `arn:minio:` 不互认(解析侧强制 `arn:rustfs:`) | `crates/ecstore/src/bucket/target/arn.rs:43,51` vs MinIO `bucket-targets.go:709` | 存量 MinIO 复制配置迁移被 StaleTarget 拒;原生 madmin SDK 解析 RustFS ARN 失败 |
-| P1-8 | PutBucketReplication 校验缺口(规则数/Priority 唯一/ID 长度/Filter 互斥/2MB 上限全缺)+ 主动拒绝 `Destination.StorageClass` 等 MinIO/AWS 合法字段 | `bucket_usecase.rs:582-616`、`config.rs:143-232` vs MinIO `internal/bucket/replication/replication.go:29-90` | 非法配置被接受、优先级冲突行为不可预测;存量 AWS/Terraform 配置(含 StorageClass)直接 400 |
-| ~~P1-9~~→**P2-26** | GetObject 响应缺 `x-amz-replication-status` 头(HEAD 有 GET 无),且 GET 专门把它从 metadata 过滤掉(功能视角复核:**降级 P2**;GET/HEAD 不一致确认,缺 GET replication-status 回归测试) | `object_usecase.rs:5696-5735`、`options.rs:702` vs MinIO `api-headers.go:236-238` | 依赖 GET 判断复制状态的客户端/监控失效;修复约一行 |
-| P1-10 | Snowball auto-extract 解包对象不触发复制(功能视角复核:**确认,维持 P1**,但"全部永不复制"不准确——scanner 在状态空+ExistingObjectReplication 启用时可补齐;显式 Disabled 等场景长期遗漏,即时复制始终失效。带 REPLICA 状态的入站成员须继续避免回环)。**已随 [#5753](https://github.com/rustfs/rustfs/pull/5753) 修复**(含入站复制 PUT 不再被误派发 extract 的次生缺陷) | `object_usecase.rs:8201` vs MinIO `object-handlers.go:2452,2510-2511` | 批量导入对象不即时复制;缺普通解包成员复制结果的测试(已在 #5753 补充 e2e) |
-
-### mc 响应结构(静默空输出类)
-
-| # | 问题 | 证据 | 影响 |
-|---|---|---|---|
-| P1-11 | `?replication-metrics[=2]` 响应为 Rust snake_case,minio-go MetricsV2 期望 camelCase(`currStats`/`queueStats`/…) | `stats.rs:617-770`、`admin/router.rs:1583-1592` vs MinIO `bucket-stats.go:154-188` | `mc replicate status` 不报错但全零(静默错误) |
-| P1-12 | replication-reset(resync)响应壳不匹配:`{"Targets":[{"Arn","ResetID",...}]}` vs `{"target":[{"arn","resetid","resyncStatus",...}]}` | `router.rs:126-198,1735-1803` vs MinIO `bucket-replication-utils.go:613-636` | `mc replicate resync start/status` 输出空;仅响应壳问题,修复成本低 |
-| P1-13 | `/v3/replication/mrf` 与 `/v3/replication/diff` 返回单个聚合对象而非条目流(代码自述 deliberate) | `replication.rs:695-725,879-911,998-1047` vs madmin-go `replication-api.go:104-176` | `mc replicate backlog` 输出空;`node`/`arn`/`verbose` 参数被忽略 |
-| P1-14 | set-remote-target 请求体 `deny_unknown_fields` + 字段名偏差(期望 `bandwidth_limit`,madmin 发 `bandwidthlimit`;`session_token` vs `sessionToken` 等) | `handlers/replication.rs:88-95,108-163` vs madmin-go `bucket-targets.go:76` | `mc replicate add/update --bandwidth` 整请求失败;凡 omitempty 字段一旦出现即 400 |
-
-### 代码结构
-
-| # | 问题 | 证据 | 影响 |
-|---|---|---|---|
-| P1-15 | 站点复制状态两套归一化实现(handler 类型化 vs service 无类型 JSON),且 reload 的 read→normalize→save 全程无共同分布式对象锁,存在 lost-update 竞争;repair state 已用 `with_config_object_write_lock` 包住完整 RMW,主 state 未采用同等保护(功能视角复核:**确认,维持 P1**;进程内 `SITE_REPLICATION_STATE_LOCK` 与单次 read/save 各自的对象锁均不能保护跨调用 RMW:A 读旧→B 另节点写入→A 用旧快照覆盖,B 丢失) | `handlers/site_replication.rs:114,347,1039-1130` vs `service/site_replication.rs:26-135` | 归一化语义可 drift;多节点/RPC 并发写状态互相覆盖。**缺多节点/双写者 lost-update 回归测试** |
-| P1-16 | 复制状态机类型双份定义:`rustfs-filemeta` 与 `rustfs-replication` 各持一份(ReplicationStatusType/VersionPurgeStatusType/ReplicationState/MrfReplicateEntry/ReplicateObjectInfo),靠 boundary 双向转换 | `crates/filemeta/src/replication.rs` vs `crates/replication/src/filemeta.rs` | 状态机语义修改须同步两处+转换层,漏一处即静默数据语义错误;建议加 enum 对账测试 |
-| P1-17 | 桶复制逻辑分裂:`crates/replication` 仅契约,执行引擎(pool 5947 行、resyncer 4090 行)仍在 ecstore,中间 20+ 个 boundary/bridge 微文件;迁移无完成判据,脚手架有固化风险 | `crates/ecstore/src/bucket/replication/README.md`、`mod.rs:15-45` | 可读性/可维护性成本;需设定迁移里程碑 |
-| P1-18 | 超长函数集中在复制热路径:`resync_bucket` 536 行、`replicate_all` 403 行、`start_mrf_processor` 305 行、`apply_iam_item` 248 行 | `replication_resyncer.rs:546`、`replication_pool.rs`、`site_replication.rs:7806` | 正确性审查与修改风险高 |
-
-### 第三方复审新增与调整项(功能视角二次复核后)
-
-| # | 问题 | 来源 | 影响 |
-|---|---|---|---|
-| P1-19 | 普通复制对象缺少可靠的源→目标版本身份策略:PUT 响应的目标版本 ID 未捕获/持久化,对不支持 versionId query 的目标(原生 AWS S3 等),后续版本删除复制落空;MRF 只会重试同一个错误身份,HEAD ETag fallback 不能修复删除 | P0-5 复审 | 非 MinIO 系目标的版本化复制双端发散。缺"目标自行分配版本 ID"场景测试 |
-| P1-20 | 缺少 scanner 补偿边界的 e2e:ExistingObjectReplication Enabled/Disabled × 空状态/继承状态 组合下的补齐与不补齐行为无回归覆盖(Copy 与 Snowball 两路径) | P0-6/P1-10 复审 | scanner 兜底语义变化不可见 |
-| P1-21 | delete-marker 延迟 purge 失败静默丢弃(由 P2-20① 升级):目标删除失败无日志/状态/MRF,目标端 marker/版本可能永久残留 | P2-20 复核升级 | 数据一致性;缺失败注入测试 |
-| P1-22 | 桶复制整体 SSE 支持能力缺口(替代原 P2-23):SSE-S3/SSE-KMS 所有复制模式统一 fail closed,SSE-C 失败被 e2e 钉为当前行为,无 encrypted-object resync e2e | P2-23 复核改写 | 加密对象跨站不复制;需覆盖普通复制/Heal/Resync/Multipart 四模式 |
-
-### 功能视角二次复核采纳记录(backlog#1675,基于 main f0c4fbd28)
-
-复核共 10 项,判定依据为 RustFS 自身功能契约与实际调用链,不以对齐 MinIO 为正确性标准。采纳结果:
-
-| 原编号 | 复核结论 | 采纳动作 |
-|---|---|---|
-| P0-5 | 确认,P0→P1,问题重述为"源→目标版本身份策略缺失" | 定级调整;修复已合 [#5752](https://github.com/rustfs/rustfs/pull/5752);残留缺口 P1-19 |
-| P0-6 | 确认,P0→P1,scanner 描述纠正 | 定级调整;修复已合 [#5753](https://github.com/rustfs/rustfs/pull/5753);测试缺口 P1-20 |
-| P1-1 | 确认,维持 P1 | 补记"expiry 同步后保留本地 transition"测试缺口 |
-| P1-2 | 确认,P1→P2(仅管理面可观测性) | 改编号 P2-25 |
-| P1-4 | 拆分:`sync` 指控不成立(RustFS 自身契约定义为健康状态,有测试);`defaultbandwidth` 确认为 P2 能力缺口 | `sync` 撤销并归入"独特设计保持不变";`defaultbandwidth` 降 P2 |
-| P1-9 | 确认,P1→P2 | 改编号 P2-26;补记缺 GET 回归测试 |
-| P1-10 | 确认,维持 P1,"全部永不复制"改为"即时复制失效+部分场景长期遗漏" | 已随 [#5753](https://github.com/rustfs/rustfs/pull/5753) 修复(含回环防护) |
-| P1-15 | 确认,维持 P1(竞争机理精确化:跨调用 RMW 无共同分布式锁) | 补记缺双写者 lost-update 测试 |
-| P2-20① | 确认,P2→P1(延迟 purge 失败静默丢弃部分) | 升级为 P1-21;②③维持 P2 |
-| P2-23 | resync 专属指控不成立;暴露桶复制整体 SSE 能力缺口 | 撤销原表述,改立 P1-22 |
-
-**复核指出的测试补齐清单**(均未运行跨实例集成验证,需落地):目标自行分配版本 ID、Copy/Snowball scanner 补偿边界、lifecycle expiry/transition 保留、site state 双写竞争、delayed purge 失败注入、encrypted-object resync。
-
----
-
-## 四、P2 问题清单
-
-### 站点复制
-- **P2-1** `showDeleted` 选项与 `bucketDeletedTimestamp` 未实现(`site_replication.rs:1364-1381`)
-- **P2-2** 错误码泛化:统一 `InvalidRequest`/`InternalError`,无 MinIO 的 9 个 `XMinioSiteReplication*` 专用码(400/503 语义丢失)
-- **P2-3** `make-with-versioning` 忽略 `versioningEnabled`/`forceCreate` 参数,恒 true(`site_replication.rs:8597-8627`)
-- **P2-4** netperf 返回"不支持"占位(gob 格式兼容不会崩);devnull 有请求体大小上限(MinIO 无限 discard)
-- **P2-5** Metrics 摘要仅含本站,无 per-peer 链路统计(downtime/latency/失败窗口)
-- **P2-6** `external-user`/`credential` IAM item 未实现——与本仓 MinIO 版本等价缺失,结构已预留;对接新版 MinIO 时会成缺口
-- **P2-7** 本地 deploymentID 缺失时回退 endpoint 哈希(16 位 hex,非 UUID 形态)
-
-### 桶复制 / S3 协议
-- **P2-8** 遗留内部 client 头名错误:`X-Source-DeleteMarker`/`X-Check-Replication-Ready` 缺 `X-Minio-` 前缀(`client/api_stat.rs:191-231`,当前路径未激活,潜伏缺陷)
-- **P2-9** Remote target admin 错误码扁平化(MinIO 有 404/503 专用码,RustFS 统一 400/500)
-- **P2-10** Remote target 拒绝 `disableProxy`/`edge`/`edgeSyncBeforeExpiry` 等 madmin 字段(非默认参数,影响小)
-- **P2-11** `list-remote-targets` 序列化偏差:`bandwidth_limit`/`storage_class`/`deployment_id`/`reset_id`/`session_token` vs madmin 的 `bandwidthlimit`/`storageclass`/`deploymentID`/`resetID`/`sessionToken`;`healthCheckDuration`/`totalDowntime` 按秒序列化而 Go 按纳秒解;`type` 过滤参数被忽略
-- **P2-12** set-remote-target?update=true 忽略 madmin 的 op 标志(creds/sync/proxy/…),固定整体覆盖
-- **P2-13** XML 反序列化:Rule 内未知元素严格报 MalformedXML(顶层却跳过,行为不一致);缺 `<Role>` 报 MalformedXML(Go 容忍)——向前兼容性差,当前主流客户端不受影响
-- **P2-14** `ReplicaModifications` 默认 Disabled(与 AWS 一致、与 MinIO 的注入 Enabled 分歧);PUT 时不像 MinIO 那样注入默认元素回写
-- **P2-15** PutBucketReplication 要求预先注册 remote target(与 MinIO 同构、与纯 AWS 流程分歧),报错未指引先建 target
-- **P2-16** GetBucketReplication 响应无 xmlns(与 MinIO 一致,极少数严格 SDK 可能拒收)
-- **P2-17** 站点复制启用时不阻止普通用户直接改桶复制配置(MinIO 非 root 报 `ErrReplicationDenyEditError`)
-- **P2-18** Prometheus 指标名对齐 metrics-v3 但注册前缀为 rustfs 体系;versioning 错误文案与 MinIO 不同(code 一致)
-
-### 代码结构
-- **P2-19** `apply_iam_item` / bucket-ops 用裸字符串 match 分发,无法穷尽检查;建议改 `#[serde(tag)]` 枚举
-- **P2-20(拆分)** 静默吞错:①`replication_resyncer.rs:1693` delete-marker 延迟 purge 失败被 `let _ =` 丢弃,target client 缺失时直接跳过——**功能视角复核:升级为 P1-21**(失败后无日志、无状态更新、不入 MRF,目标 delete marker/版本可能永久残留;启动前的 5 次循环只是等源 marker 消失,不是对目标删除失败的重试。缺注入目标删除失败并验证重试/状态/MRF 的测试);②`site_replication.rs:8661` purge-deleted-bucket 吞掉非 NotFound 错误、`:9227` cancel resync 失败无痕迹——维持 P2
-- **P2-21** `MrfV2` 全套机制(Error/Capabilities/Readiness/Reader/Envelope)未接线,生产只用 v1,属投机代码
-- **P2-22** `persist_site_replication_state` 双重 clone + 双重 normalize(`site_replication.rs:1143-1152` → `:1116-1122`)
-- **P2-23(撤销并改写)** 原"resync 不处理 SSE"指控不成立——`ReplicationType::Resync` 与普通复制/Heal 最终走同一 `replication_put_object_options`,`// TODO: SSE` 不构成 resync 独立行为差异。真实状态:SSE-S3/SSE-KMS 在**所有复制模式**下统一 fail closed,SSE-C 普通桶复制失败已被现有 e2e 钉为当前行为,且无 encrypted-object resync e2e → 改立能力项 **P1-22"桶复制整体 SSE 支持"**(需分别覆盖普通复制、Heal、手动 Resync、Multipart)
-- **P2-24** `crates/replication` 命名误导(名为复制引擎实为契约库),建议 lib.rs 顶部文档说明
-- 正面确认:生产代码 unwrap/expect 纪律良好(几乎全在测试模块);MinIO 概念映射(ReplicationPool/Resyncer/MRF/TargetClient)桶复制侧清晰,站点复制侧缺 `SiteReplicationSys` 聚合体
-
----
-
-## 五、P0 问题对抗性分析(复核结论 + 验证方案 + 解决方案)
-
-### P0-1 出站 peer join 路径 — **CONFIRMED(比原指控更严重)**
-
-**复核结论**:指控全部成立,且加重三点:
-1. `/minio/admin/v3/site-replication/join` 在 MinIO 历史上**从未存在过**(`git log -S` 追到功能诞生的 2021 年首个提交,注册的就是 `peer/join`)。RustFS 实现者疑似被 MinIO `admin-handlers-site-replication.go:76` 一条过时的文档注释误导。
-2. 无任何 404 回退、版本探测或 feature flag;唯一的重试逻辑只针对 secret 不匹配(`site_replication.rs:3036-3082`),404 直接失败。
-3. 现有单测 `:13683-13696` 正在**固化错误行为**(测试名声称匹配 MinIO 路由,断言的却是不存在的路由)。RustFS↔RustFS 之所以不暴雷,是因为 RustFS 入站自己注册了该错误路径的兼容别名,掩盖了 bug。
-
-**影响面**:RustFS 发起的 add(含 MinIO 站点)、服务账号轮换通知 MinIO peer 均断;MinIO→RustFS 与 RustFS↔RustFS 不受影响;其余 peer/* 端点走通用前缀改写,路径正确。
-
-**修路径还不够,还有三处 join 协议分歧须同批修**:①加密判定 `site_replication_peer_payload_encrypted`(:2899-2901)只对旧路径加密,MinIO `SRPeerJoin` 强制解密,须跟随路径改;②MinIO join 成功返回**空 body**,RustFS `:8163` 强制解析 `SRPeerJoinResponse` 会失败,须容忍空 body(peer 身份回退用 preflight 已取得的数据合成);③`deferSyncStateEnable`/`bootstrapToken` 对 MinIO 无效但不阻断(行为差异,建议日志标注)。
-
-**验证方案**:
-- 单测:翻转 `:13683`/`:13699` 两个测试断言为 `peer/join`(把固化 bug 的测试变成回归防护)。
-- 集成测:测试内起 axum stub 精确复刻 `admin-router.go` 路由(仅注册 `PUT .../peer/join`,其余 404),handler 内用 `decrypt_stream_io` 验证 body 是 madmin 兼容密文,返回 200 空 body;断言修复前 404、修复后全链路成功。
-- e2e:docker compose(rustfs+minio),RustFS 侧 `mc admin replicate add`,MinIO 侧 `mc admin trace -a` 断言 `PUT .../peer/join` 200。注意:**e2e 会先被 P0-2 的 preflight 挡住,两问题必须同批修复才能全链路验证**。
-
-**解决方案**(均在 `handlers/site_replication.rs`):删除 :2885-2886 的 join 特判使其落入通用前缀改写;:2899-2901 加密判定改为对 `peer/join` 返回 true;:8163 响应解析容忍空 body;更新两个单测。
-**滚动升级风险**:必须保留入站的 `/v3/site-replication/join` 旧路径路由(旧版 RustFS 出站仍发它);发版前对最近 release tag 复核旧版入站已注册 `peer/join`。
-
-### P0-2 SRInfo 大小写不匹配 — **CONFIRMED(范围精确化)**
-
-**复核结论**:成立。madmin-go v3.0.109(minio go.mod 锁定版)`SRInfo` 除 `APIVersion` 外 12 个顶层字段**全部无 json tag**,Go 按 PascalCase 序列化;RustFS `SRInfo` serde 大小写敏感、全字段 `#[serde(default)]` → 解析 MinIO 输出**不报错而是静默全空**。精确化:**不兼容仅限 SRInfo 顶层 12 个字段**,嵌套结构(SRBucketInfo/SRStateInfo/SRIAMPolicy 等)madmin 本就带小写 tag,不受影响。`:5581` 的 `"buckets"|"Buckets"` 手写双读证明作者已知 MinIO 输出 PascalCase,只是未系统化修复。
-
-**影响面**:RustFS 发起 add 时 preflight 硬失败("site did not report deploymentID")——**触发顺序先于 P0-1 的 join**;`mc admin replicate status` 对 MinIO peer 静默显示全空/全 mismatch(HTTP 200,无报错)。MinIO 读 RustFS 方向因 Go unmarshal 大小写不敏感而无恙。
-
-**验证方案**:
-- 单测(crates/madmin):用 Go `json.Marshal(madmin.SRInfo{...})` 真实生成的 PascalCase JSON 作 fixture,断言反序列化后字段非空;再加序列化回归断言输出仍为 camelCase(保证 RustFS↔RustFS 不回归)。
-- 集成测:stub 在 metainfo 端点返回 PascalCase body,走 `remote_add_preflight_info`,断言不再报错。
-- e2e:与 P0-1 同批,`mc admin replicate status --json` 断言 MinIO 站点条目完整。
-
-**解决方案**:`crates/madmin/src/site_replication.rs:642-670` 为 12 个顶层字段逐一加 `#[serde(alias = "...")]`(精确取 Go 字段名,注意是 `ILMExpiryRules` 不是 `IlmExpiryRules`)。alias 只影响反序列化,出站格式零变化,风险几乎为零。**只加顶层、不扩散到嵌套结构**,并留注释说明原因。回归防护关键是把 Go 真实输出固化为测试 fixture。
-
-### P0-7 `mc replicate add` 默认参数被拒 + 单位错误 — **CONFIRMED**
-
-**复核结论**:全部反驳方向反向坐实(本地有 mc 源码,非推断):
-- mc `replicate-add.go:93-95` 默认 `healthcheck-seconds=60`,`:301-303` 无条件调用 `SetRemoteTarget`,失败即终止,无跳过路径;
-- madmin `bucket-targets.go:79` `HealthCheckDuration time.Duration` 无自定义 Marshal → wire 上是纳秒整数 `60000000000`;
-- RustFS `handlers/replication.rs:213-225` 对非零值必拒 400;`mc replicate update` 同样失败;无老端点绕过。
-- **单位错误独立成立且双向**:请求侧按 `Duration::from_secs` 解析(60e9 ns 会被当 60e9 秒 ≈ 1900 年);响应/持久化侧 `bucket_target.rs:195-197` 按秒序列化,mc 按纳秒解(60s 显示为 60ns),同时构成与 MinIO `bucket-targets.json` 的持久化格式偏差。
-- **为何没被发现**:这是刻意的"能力契约式拒绝"策略,且有单测 `replication.rs:1353-1379` 固化拒绝行为;e2e 全部自行构造 JSON、不含该字段,测的是"RustFS 自己的请求形态"而非"mc 默认请求形态"。缓解:`--healthcheck-seconds 0` 时字段 omitempty 被省略可通过,但默认路径必失败,P0 成立。
-
-**验证方案**:复现——`mc replicate add rustfs/src --remote-bucket http://ak:sk@target/dst` 预期 400;修复后——madmin 形态 payload(60e9 ns)单测断言内部 Duration==60s;set→list 往返断言响应为纳秒;e2e 增加"mc 默认 payload"用例;持久化防御性读回归(旧秒格式升级后读取不变)。
-
-**解决方案(分阶段)**:
-1. **解阻塞**:从不支持清单移除 `healthCheckDuration`(能力契约版本号递增);请求按 `Duration::from_nanos` 解析(`total_downtime` 同步核查);调度上显式忽略并在契约/文档标注"接受但暂不生效";响应侧新增 DTO 按纳秒序列化(**勿直接改 `bucket_target.rs` 的 `duration_seconds`,它同时是持久化格式**);持久化读取加防御(≥10^7 视为纳秒),写入统一新格式。
-2. **落地语义**:`bucket_target_sys.rs:332-441` heartbeat 循环改为按 target 取值,对齐 MinIO(默认 5s、有下限)。
-3. **防复发**:建立容器内跑真 mc 命令的兼容 e2e 通道,覆盖 `replicate add/update/status`。
-
-### P0-8 站点复制架构 — **事实 CONFIRMED,定性部分 REFUTED,降级为 P1**
-
-**复核结论**:巨型文件(14614 行,非测试约 9533 行,24 个 handler)与三处反向导入全部属实;但"失察"定性被推翻:
-- `scripts/check_layer_dependencies.sh` **已建模并拦截**这些边,`layer-dependency-baseline.txt` 棘轮基线逐条列出全部 46 条存量反向边,**新增反向边 CI 必炸**;
-- `ecfs.rs` 被脚本刻意归类为 interface 层(有意的建模决策);
-- ARCHITECTURE.md 自己声明部分不变量 "currently violated... documenting them makes violations explicit and trackable";git 历史显示这是已知、受控、正在偿还的过渡态。
-- **结论:不构成正确性风险,从 P0 降为 P1(可维护性债务)**。真实成本:9.5k 行单文件的评审/合并冲突/增量编译负担,hook 直连使 app/storage 单测无法脱离 admin 层。
-
-**验证方案**:每阶段跑 `make pre-pr`;每消除一条反向边即**删除基线对应行**(而非重生成),使回归必炸;行为回归靠 site replication e2e + 路由快照测试 + `git diff --color-moved` 评审纯移动。
-
-**解决方案(分阶段)**:
-1. **解反向依赖(低风险,先做)**:复用 `site_replication_reconcile.rs` 已验证的 OnceLock 注册模式——bucket 三个 hook 在 app 层定义 fn-pointer 契约、admin 构建路由时注册;`node_service.rs` 的 reload 走 infra 层"运行时重载注册表"。注册缺失时显式降级(warn + no-op)。
-2. **文件拆分(纯移动)**:`site_replication.rs` → 模块目录:`transport`(peer client/DNS/TLS)、`gob`、`state`(注意 config key 路径不可变)、`iam_sync`、`heal`、`handlers`(24 个薄 handler)。
-3. **领域下沉(风险最高,最后做)**:hook 解耦后把 gob/transport/状态机移入独立 crate,注意全局状态清单(`docs/architecture/global-state-inventory.md:114`)。
-
-### P0-3 STS item 类型名不一致 — **CONFIRMED(双向硬断)**
-
-**复核结论**:成立,且两端都是**报错而非静默忽略**:MinIO 收到 `"sts-credential"` 走 default 分支返回 400 `errSRInvalidRequest`;RustFS 收到 `"sts-account"` 返回 NotImplemented。两端 heal/重试机制都会永久重试失败(MinIO 日志持续 "Unable to heal temporary credentials")。MinIO 当前版本 STS 复制发送面很广(AssumeRole/WebIdentity/ClientGrants/LDAPIdentity/Certificate 全系 + sftp/ftp + heal 路径)。除类型串外 `SRSTSCredential` 字段双方完全对齐——**只差这一个字符串**(推测 RustFS 实现时把 madmin 的 JSON 字段名 `stsCredential` 误当成了类型常量)。
-
-**影响面**:跨厂商 STS 临时凭证双向不复制(客户端在对端站点 `InvalidAccessKeyId`),纯可用性问题,无权限漂移;RustFS↔RustFS 自洽。
-
-**验证方案**:单测——出站产物断言 `type == "sts-account"`(改 `federated_identity.rs:497` 现有快照测试);入站构造 `"sts-account"` item 断言不落 NotImplemented。e2e——compose(RustFS+MinIO,root 凭证必须一致,否则 token 验签失败会误判修复无效):对 MinIO assume-role 拿临时凭证访问 RustFS,修复前 InvalidAccessKeyId、修复后成功;反向同测。
-
-**解决方案**:出站(`sts.rs:248`、`federated_identity.rs:241`)改发 `"sts-account"`(提常量集中定义);入站(`site_replication.rs:7857`)match 臂改 `"sts-account" | "sts-credential"`(**永久保留旧别名**兼容旧 RustFS peer)。滚动升级窗口内新→旧 RustFS 会降级(warn+重试,peer 升级后收敛);STS 凭证短生命周期,不建议为此拆两阶段发布。
-
-### P0-4 policy-mapping userType 数值错位 — **CONFIRMED(比指控更严重)**
-
-**复核结论**:数值表属实(RustFS: None=0/Svc=1/Sts=2/Reg=3;MinIO: unknown=-1/reg=0/sts=1/svc=2),wire 上确为数值、无翻译层。对抗复核修正与加重:
-- **RustFS→MinIO 方向今天"侥幸能用"**:RustFS 当前只出站 Reg=3 与组的 0,MinIO 对超范围值静默落 default 分支,恰好落对位置;
-- **MinIO→RustFS 方向三类断裂**:①**组映射硬失败(新发现)**——MinIO 组映射发 `UserType: -1`,RustFS `user_type: u64` 反序列化直接报错,整个 item 被拒,组→策略映射完全无法同步;②STS 用户映射(MinIO 发 1)被 RustFS 解释为 Svc,落错前缀/缓存,联邦用户在 RustFS 站点**静默丢权限**;③svc=2 被解释为 Sts,同类错位;
-- **低概率提权路径**:LDAP DN/OIDC 主体的映射被误存入常规用户缓存后,若本地恰有同名静态用户则继承本不属于它的策略——名字碰撞概率低但非零,这是保 P0 的理由。
-
-**验证方案**:单测——wire 编解码全矩阵(-1/0/1/2/3/非法值);e2e——MinIO 侧 `mc admin policy attach --group` 修复前 RustFS 查不到组实体、修复后可见;`mc idp ldap policy attach` 修复前落 `policydb/service-accounts/` 且访问被拒、修复后落 `sts-users/` 且放行;反向回归守住"侥幸兼容";混版本(旧+新 RustFS)双向 attach 互通。
-
-**解决方案(核心原则:不改 `UserType::to_u64/from_u64`)**——该编码被集群内部节点 RPC 使用(`node_service.rs:1513`),改动会破坏同集群滚动重启。只在站点复制 wire 边界加 MinIO 语义编解码:
-1. `SRPolicyMapping.user_type` 由 `u64` 改 `i64`(必须,才能收下 -1);
-2. 出站 `sr_wire_user_type`:Reg→0/Sts→1/Svc→2,组一律发 0(对 MinIO 与旧 RustFS 同时兼容);入站 `user_type_from_sr_wire`:-1→None/0→Reg/1→Sts/2→Svc/**3→Reg(旧 RustFS 别名,永久保留)**;
-3. 兼容矩阵已逐格验证:新↔旧 RustFS、MinIO↔新 RustFS 全通;唯一残余窗口(未来出站 Sts/Svc 映射对旧 RustFS 错读)当前不可达,在 doc comment 写明约束;
-4. 回归防护:编解码矩阵单测 + "wire 常量契约"字面值断言测试(防止将来被"顺手统一"回内部编码)+ e2e 进 P0 套件;顺带把 `SRCredInfo.iam_user_type` 一并改 `i64` 复用同一编解码,消除同族隐患。
-
-### P0-5 复制 PUT 缺 `?versionId=` query — **CONFIRMED**
-
-**复核结论**:所有反驳方向均失败,指控成立:
-- minio-go 官方复制端(v7.0.91)`api-put-object-streaming.go:767-776` 等三处全部是 `urlValues.Set("versionId", ...)`——**query,不是 header**;`x-minio-source-version-id` 这个 header 在 MinIO 全仓不存在,被静默忽略;
-- multipart 的版本在 **initiate 时**决定(`erasure-multipart.go:458-460`,为空即生成新 UUID),complete 不读 versionId;
-- aws-sdk-s3 `PutObjectInput` 无 versionId 成员属实,但 DELETE 路径已用 `.set_version_id()` 正确落 query,证明是遗漏而非不可行;
-- RustFS↔RustFS 不受影响的原因:RustFS 接收端有私有 header fallback(`options.rs:296-301`),恰好掩盖了 bug。
-
-**影响加重**:除版本漂移与按版本删除永久 no-op 外,目标校验/heal 用源 versionId `head_object` 永远 miss → **反复重传,目标端版本无限膨胀**。另有边缘缺陷:RustFS 内部 null 版本是 nil-UUID,直接发 query 会被 MinIO 当真实版本;minio-go 约定发字面 `"null"`。
-
-**验证方案**:L1 e2e(本仓可落地,红→绿)——复用 `crates/e2e_test/src/fake_s3_target/`(已解析 versionId query 并写 journal),断言 PutObject/CreateMultipartUpload 请求的 query == 源版本;L2 互操作(docker + 真 MinIO)`mc ls --versions` 断言目标 versionId == 源、删源版本目标同步消失;L3 单测 nil-UUID→`"null"` 映射。
-
-**解决方案**(`bucket_target_sys.rs`):`put_object`/`create_multipart_upload` 在 `map_request` 闭包内改写 URI 追加 `versionId` query(nil-UUID 映射 `"null"`);保留双 header 兼容旧版 RustFS 接收端;顺带核对 delete 路径的 nil-UUID 映射。**签名安全性已验证**:`map_request` 挂在 `modify_before_signing`,query 会进 canonical request,不会 SignatureDoesNotMatch。非版本化目标桶沿用"空则不发",`"null"` 值 MinIO 免检。
-
-### P0-6 CopyObject 不触发复制 — **CONFIRMED(附带加重发现)**
-
-**复核结论**:三个反驳方向全部不成立:
-- copy 直接调 `store.copy_object`,不经 put 路径;ecstore 层 copy 实现无任何调度;
-- **scanner 兜底不存在(关键)**:heal 入队条件是状态为 Pending/Failed 或手动 resync;而 copy 路径不 stamp PENDING(对照 put 路径 `object_usecase.rs:5255-5266`),状态为空 → heal 判定 Skip。
-- **加重发现**:copy 路径没有 MinIO `filterReplicationStatusMetadata` 的等价清理——COPY 指令下源对象的旧复制状态可能原样带到目的对象,**伪造 COMPLETED 假状态**。
-- 附带 P1(snowball `execute_put_object_extract`)同样确认:无 stamp 无 schedule。
-
-**影响面**:配复制规则的桶上,CopyObject 写入的对象(跨桶复制、rename 工作流、REPLACE 元数据更新)永不复制、scanner 不捞、仅手动 resync 可补;还可能带 stale 假状态。
-
-**验证方案**:e2e(参照 `replication_extension_test.rs` 双实例)——copy 后断言目的对象在目标桶超时内出现、源 COMPLETED、目标 REPLICA、无 stale 状态;snowball 参照 `snowball_auto_extract_test.rs` 加成员对象复制断言;usecase 单测用 `storage_api.rs:641` 现有 test-only 调用计数断言 copy/extract 触发决策与调度。
-
-**解决方案**(`object_usecase.rs`):
-1. `execute_copy_object` 在 `store.copy_object` 之前算一次 `dsc = must_replicate_object(...)`,`replicate_any` 时向 `dst_opts.user_defined` stamp pending + timestamp(严格镜像 put 路径,单一 dsc 决策贯穿两阶段);
-2. 同处清理源带来的复制状态 reserved 元数据;
-3. copy 成功、锁释放后 `schedule_object_replication`;
-4. `execute_put_object_extract` 对每个解出对象同样处理。
-风险已排除:replica 判定内置于 `must_replicate_object` 不会回环;self-copy 调度与 MinIO 一致。
-**落地顺序约束:先修 P0-5 再修 P0-6**——否则 copy 的失败重试经 heal 兜底后,只会在 MinIO 端制造更多漂移版本。
-
-### 第三方复审修正(2026-08-05,修复分支均已完成 review)
-
-**P0-5 修正**:问题的准确表述应为"**普通复制对象缺少可靠的源→目标版本身份策略**"——复制 PUT 只返回成功/失败,未捕获目标实际分配的版本 ID(已核实 `bucket_target_sys.rs` put 路径无 `res.version_id()` 捕获,delete 路径 :2030 有);multipart 只保留 upload ID。`fix/p0-5` 的 versionId query 方案对 MinIO/RustFS 目标成立(目标端沿用源版本 ID,身份问题消解),但对**忽略该私有 query 的目标(如原生 AWS S3)**身份问题仍在:目标自行生成版本 ID → 后续按源版本 ID 的删除复制落空。第三方建议定级 P1(修复已完成,残留缺口另行跟进):可选方案包括捕获 PUT 响应的 `x-amz-version-id` 并持久化源→目标映射。→ 记为 **P1-19(新增)**。
-
-**P0-6 修正**:scanner"兜底不存在"的表述过度。已核实 `crates/replication/src/operation.rs` `resync_target_for_object`:无 reset 记录且复制状态为 Empty 时返回 `replicate=true`,即 ExistingObjectReplication 启用时 scanner **可能最终补齐**空状态对象,无需手动 resync。准确结论:即时/同步复制语义失效(P0 定级依据),且以下场景**长期**漏复制——①源对象 COMPLETED 等复制元数据被 Copy 继承致误判(`fix/p0-6` 已修,清理先于决策);②显式 ExistingObjectReplication=Disabled;③其他无法进入 existing-object 补偿的场景。`fix/p0-6` 分支已含 copy 调度 e2e 与 stale 元数据白盒断言;**scanner 补偿边界的 e2e 仍缺** → 记为 **P1-20(新增)**。
-
-### 对抗性复核总览
-
-| 问题 | 复核结论 | 关键修正/加重 |
-|---|---|---|
-| P0-1 join 路径 | CONFIRMED,加重 | 路径在 MinIO 从未存在;现有单测固化错误;修复需同批改加密判定与空响应容忍 |
-| P0-2 SRInfo 大小写 | CONFIRMED,精确化 | 仅顶层 12 个无 tag 字段;preflight 失败先于 P0-1 触发 |
-| P0-3 STS 类型名 | CONFIRMED | 双向硬断、两端 heal 永久重试;只差一个字符串 |
-| P0-4 userType 错位 | CONFIRMED,加重 | MinIO 组映射发 -1 → RustFS u64 解析硬失败;存在低概率名字碰撞提权路径;修复不得触碰内部 RPC 编码 |
-| P0-5 versionId query | CONFIRMED,加重 | heal 反复重传致目标版本膨胀;nil-UUID 需映射 "null" |
-| P0-6 CopyObject | CONFIRMED,加重 | scanner 兜底不存在;stale COMPLETED 假状态;须在 P0-5 之后落地 |
-| P0-7 healthCheckDuration | CONFIRMED | 单位错误双向独立成立;有单测固化拒绝行为 |
-| P0-8 架构 | 事实 CONFIRMED,定性 REFUTED | 反向边被棘轮基线锁死,降级 P1(受控技术债) |
-
----
-
-## 六、修复路线图(2026-08-05 更新)
-
-**✅ 第一批已完成**:全部 7 项 P0 已修复并创建 PR(见第二节修复状态;P1-10 snowball 随 #5753 一并修复)。待合并,注意顺序约束:#5748+#5749 同批、#5752 先于 #5753。
-
-**第二批(数据一致性优先,采纳功能视角复核定级)**
-1. **P1-21** delete-marker 延迟 purge 失败静默丢弃(复核升级,数据一致性,建议单独小 PR + 失败注入测试)
-2. **P1-19** 源→目标版本身份策略(捕获 PUT 响应 `x-amz-version-id` / 持久化映射,覆盖非 MinIO 系目标)
-3. **P1-1** ILM expiry 同步语义(只传播 expiry、保留接收端本地 transition + 对应测试)
-4. **P1-15** site state RMW 分布式锁统一(对齐 repair state 的 `with_config_object_write_lock` 模式)+ 双写者回归测试
-5. **P1-22** 桶复制 SSE 能力(普通复制/Heal/Resync/Multipart 四模式,先补 encrypted-object e2e 钉现状)
-
-**第三批(mc 可观测性与互操作补齐)**
-6. P1-11/12/14 mc 响应结构 serde rename(改动小、消除静默空输出)
-7. P1-7 ARN 解析侧兼容 `arn:minio:` 前缀
-8. P1-5 GET/HEAD proxy、P1-6 时间戳头、P1-3 自动跨站 heal
-9. P1-20 scanner 补偿边界 e2e;P0-7 阶段 2(per-target 心跳 + healthcheck update op)
-10. P2-26 GET 补 `x-amz-replication-status`(约一行)+ 回归测试;P2 清单其余项
-
-**第四批(架构与长期)**
-11. P0-8(降级 P1)架构:先解 3 处反向依赖(复用 reconcile 注册模式),再拆分/下沉站点复制领域模块
-12. P1-16 类型对账测试、P1-17 迁移完成判据、P1-8 配置校验补齐

P1_REVIEW_AND_PLAN.md

@@ -1,195 +0,0 @@
-# P1 逐条复审订正与方案计划
-
-> 复审基线:main @ `77f2b948c`(7 个 P0 修复 #5748~#5754 已全部合入)
-> 复审方式:5 组对抗性复审 agent 并行,先怀疑后确认;以 RustFS 自身功能契约为正确性标准,不以"未对齐 MinIO"为根因;RustFS 更优/独特设计标注"保持不变"
-> 参照:MinIO 源码、mc@cf909e1063a9、madmin-go v3.0.109、minio-go v7.0.91
-> 日期:2026-08-06
-
----
-
-## 〇、复审总裁定表
-
-| 项 | 主题 | 复审结论 | 关键订正 | 工作量 |
-|---|---|---|---|---|
-| P1-1 | ILM expiry 复制语义 | CONFIRMED(范围扩大) | 发送点共 4 处非 1 处;接收端无门禁;修复重心移到接收端 merge | M |
-| P1-3 | 自动跨站元数据 heal | CONFIRMED(范围收窄) | 真实缺口="retry queue 有账本无消费者";不移植 MinIO 全量 heal | M |
-| P1-5 | GET/HEAD 远端 proxy | CONFIRMED | 同步复制模式是已实现的部分缓解(保持不变);proxy 指标语义被出站 HEAD 污染 | L(P0 段 M) |
-| P1-6 | 三类时间戳头收发 | CONFIRMED(缺口扩大) | 实为三段缺失:tagging 无本地写入方 + 不发头 + 接收端无 LWW 合并点 | M |
-| P1-7 | ARN 前缀不互认 | CONFIRMED+(加重) | 新发现 FromStr id/region 互换 bug;madmin ParseARN 硬校验实锤 → 生成侧必须改 | M |
-| P1-8 | 配置校验缺口 + StorageClass | 部分 CONFIRMED | 2MB 子项 REFUTED(MinIO 亦无);StorageClass 属刻意设计成立(MinIO 也不消费 rule 级,target 级 RustFS 已生效)| S |
-| P1-11 | replication-metrics snake_case | CONFIRMED | BucketStats 复用内部 RPC 线格式实锤 → 必须独立响应 DTO | M |
-| P1-12 | replication-reset 响应壳 | CONFIRMED(面缩小) | 致命键仅 5 个(壳 `Targets`≠`target` + 4 个字段名);其余靠 Go 大小写不敏感能对上 | S |
-| P1-13 | mrf/diff 聚合响应 | CONFIRMED(症状加重) | 实际输出**伪数据行**而非空;diff/mrf 数据源均可支撑逐条流 | diff S / mrf M |
-| P1-14 | set-remote-target 请求体 | 原缺口已缓解;**新 CONFIRMED 阻断** | #5754 后 26 字段已全覆盖;但**零值 `expiration` 恒被拒 → mc replicate add 仍 100% 失败**;latency 单位 round-trip 污染 | S(**建议立即修**) |
-| P1-15 | site state RMW 竞争 | CONFIRMED(加重) | hook 路径 enqueue/dequeue 同进程内绕过既有 Mutex → 单节点即可触发 | M-L |
-| P1-16 | 状态机类型双份定义 | CONFIRMED(加重+收窄) | drift 已发生(MrfOpKind 两侧不一致);但 filemeta 侧 worker DTO 是死代码,活跃双份仅 3 个 wire 类型;"抽公共 crate"否决 | S+M |
-| P1-17 | 桶复制逻辑分裂 | CONFIRMED;微文件合并子项 REFUTED | boundary 微文件是棘轮机制的机械接缝(守护脚本按文件名锚定),合并负收益;缺的是完成判据 | M0=S,整体 L |
-| P1-18 | 超长函数 | 行数 CONFIRMED;apply_iam_item 降级 | apply_iam_item 长而不复杂(6 臂 dispatch),不拆降 P2;其余 4 个给纯移动拆分草案 | M |
-| P1-19 | 源→目标版本身份策略 | CONFIRMED(范围收窄) | delete-marker 的"捕获+持久化映射"模式已落地(保持不变);推荐能力探测+显式拒绝而非全量映射 | M |
-| P1-20 | scanner 补偿边界 e2e | CONFIRMED(缺口收窄) | 决策函数单测与 Failed-heal e2e 已存在;缺 existing-object 矩阵与 Replica 防环 e2e;附完整入队真值表 | M |
-| P1-21 | delayed purge 静默丢弃 | CONFIRMED | 映射损坏防护已加固(保持不变);`let _ =` 与无 MRF 通道仍在;附带发现 MRF outcome 恒 false 滞留问题 | M |
-| P1-22 | 桶复制 SSE 能力 | CONFIRMED(前提订正) | SSE-S3 自 #5633 已 fail closed,被 ignore 的 e2e 理由过期(先摘 ignore);SSE-C 缺的是目标侧头摄取 | L(4 阶段) |
-
-**"保持不变"清单(复审确认的 RustFS 更优/刻意设计,不纳入修复)**:per-PUT 即时元数据传播 hook(优于 MinIO 纯周期 heal)、单向推送+stale 守卫收敛模型、delete 走 merge-with-empty(优于 MinIO 整删)、delete-marker 版本映射持久化+损坏拒猜、同步复制模式(partition_by_sync)、能力契约式显式拒绝+`deny_unknown_fields`(字段清单已与 madmin v3.0.109 同步)、StorageClass 显式拒绝非 STANDARD(target 级已真正生效)、replication-check 真实探针写删、响应中的 RustFS 增强字段(ResetBeforeDate/Error/可观测性键,Go 忽略未知键可共存)。
-
----
-
-## 一、紧急项(建议立即处理)
-
-### ⚡ P1-14 新阻断:零值 `expiration` 拒绝 → mc replicate add 仍 100% 失败
-
-- **证据**:Go `omitempty` 不省略零值 `time.Time`(已用 Go 程序按 madmin 逐字 tag 实测),mc/madmin marshal 恒输出 `"credentials":{"expiration":"0001-01-01T00:00:00Z"}` 与 `"resetBeforeDate":"0001-01-01T00:00:00Z"`;RustFS `handlers/replication.rs:286-291` 对 `expiration.is_some()` 一律 400。#5754 的测试全部用手写 payload(`expiration: None`),未被现网形状打中。
-- **修复(S)**:①`expiration` 改"非 Go 零值时间才拒"(与 `sessionToken` trim-empty 判断对称);②`latency` 请求字段直接忽略(消除 #5754 后纳秒响应 ↔ 毫秒请求的 round-trip 1e6 倍污染);③把"Go 真实 marshal 形状 payload"固化为测试夹具惯例。
-- **红灯测试**:用实测 Go marshal 全形状 body(含零值 expiration/resetBeforeDate/latency{0,0,0}/edge:false/healthCheckDuration:60000000000)打 set-remote-target,期望 200;非零 expiration 仍 400(能力契约保持)。
-
-### ⚡ P1-7 附带 bug:ARN FromStr 字段互换
-
-`arn.rs` Display 输出 `{type}:{region}:{id}:{bucket}`,FromStr 却读 `id=parts[3], region=parts[4]`——id 与 region 互换。当前仅因消费方只用 arn_type 而潜伏。随 P1-7 一并修。
-
----
-
-## 二、逐项方案计划
-
-### P1-1 ILM expiry 复制语义(M)
-
-**订正后事实**:发送完整 lifecycle XML 的路径 4 处——PUT hook(`bucket_usecase.rs:2177-2180`)、DELETE hook(`:1512-1514`,触发接收端**整删**)、import(`bucket_meta.rs:948-951`)、build_sr_info/bootstrap(`site_replication.rs:4190,2241-2249`);接收端 `apply_bucket_meta_item`(`:7669-7683`)整体覆盖/删除,且**无 `replicate_ilm_expiry` 门禁**。P0 后已有缓解(发送开关、bootstrap 跳过、stale 判定)只解决"发不发/新旧",不解决"发什么/怎么合"。
-
-**方案**:接收端 merge 为主(信任边界),发送端 expiry-only 提取为辅:
-1. 新增纯函数 `extract_expiry_only(cfg)` 与 `merge_expiry_rules(local, incoming)`——语义对齐 MinIO `mergeWithCurrentLCConfig`,两处 RustFS 改进:incoming 一律先剥 transition(防旧端);`None` 走 merge-with-empty 而非整删(**MinIO 整删连本地 transition 一起删是缺陷,不照抄**);
-2. 接收端 lc-config 分支改 读→merge→条件写/删,保留 stale 判定与 incarnation 守卫;补 `replicate_ilm_expiry` 门禁;
-3. 4 个发送点接 `extract_expiry_only`;expiry 判定用 RustFS 口径(含 `del_marker_expiration`)。
-
-**红灯测试**:L1 单测 5 例(提取剥离/合并保留 T/防御剥离/merge-with-empty/import 无 transition);L3 e2e——B 配本地 transition,A PUT expiry → B 两者共存;A DELETE lifecycle → B transition 仍在。
-**兼容**:旧端发完整 XML → 新接收端剥后 merge 正确;新端 expiry-only → 旧接收端仍整覆盖(不劣于现状)。规则按 ID 对齐,`rule-{idx}` 撞名同 MinIO 语义,文档注明。
-
-### P1-3 自动跨站 heal → 改为"retry queue 自动 drain"(M)
-
-**订正后事实**:retry queue 是现成增量账本(失败即入队 `:3243-3262`,持久化于 state,`retry_count` 字段存在)但**全库无消费者**;手动 repair 是本地快照单向推送,收敛方向依赖运维判断。即时 hook + 显式 repair 模型保持不变。
-
-**方案**:
-- 阶段 1(核心):周期任务挂进现有 reconcile ticker,per-event 重发(body 从本地当前元数据重建,复用 `SiteReplicationRepairTask::send`,天然发"当前值"+对端 stale 守卫幂等);指数退避(`retry_count`+上限转 failed);drain 全程包分布式锁去抖(先用 `with_config_object_write_lock` 专用对象,P1-15 落地后并入统一 state store);结构化 tracing 汇总一条。
-- 阶段 2(可选,默认关闭):每 N tick 比对 repair plan token,不同才自动 dry-run→execute。**不移植** MinIO 跨站取最新 pull 语义(各站各自 drain 即双向收敛)。
-
-**红灯测试**:L2——state 带 retry event,调 `drain_site_replication_retry_queue()`(现不存在),fake peer 成功后断言队列清空;退避断言。L3——停 B→A PUT policy 失败入队→起 B→drain 后 B 收到且 SRRetryStats 归零。
-
-### P1-5 GET/HEAD 远端 proxy(L;P0 段 M)
-
-**订正后事实**:`SUFFIX_SOURCE_PROXY_REQUEST` 零消费者;`ProxyMetric` 字段与 admin 汇总通路已就位,但 resyncer 把**出站** HEAD 计入 `head_total` 污染语义;`disable_proxy` 管道存在无人消费;同步复制模式(`partition_by_sync`,`replication_pool.rs:2667-2689`)是部分缓解但不等价(手动 per-target、失败仍 404、不覆盖兜底窗口)。防环头当前仅潜在问题,但 proxy 实现与防环识别**必须同 PR**(否则 RustFS↔RustFS 成环)。
-
-**方案**(P0 段):新增 `replication_proxy_boundary.rs`——`proxy_targets`(version_suspended/入站 proxy 头/disable_proxy 三重 gate)+ `proxy_get/head_to_replication_target`(走现有 TargetClient,range/条件头透传);触发点在 usecase 层 NotFound/VersionNotFound 分支;接收侧 options.rs 解析防环头,出站双前缀发送;`tokio::timeout`(~3s env 可调)、仅 2xx 采纳其余回落本地 404、复用离线标记短路;指标接 `record_replication_proxy` 并纠正 resyncer 计数语义。P1 段:tagging 三操作 proxy(依赖 P1-6)。
-**红灯测试**:e2e 双站断复制链路后从对端 GET/HEAD 应 200(现 404);防环负例(带头请求不转发、计数不增);降级负例(target 全离线时限时 404);disable_proxy 负例。
-
-### P1-6 时间戳头收发(M;三段修复)
-
-**订正后事实**:①`SUFFIX_TAGGING_TIMESTAMP` 全仓无写入方(retention/legalhold 已有双前缀写入);②`PutObjectOptions::header()` 只序列化 4 个内部头,三类时间戳被丢弃,multipart 同;③接收端不解析,且 replica PUT 是 verbatim 覆盖——解析后必须在写盘前与本地版本做 per-类别 LWW 合并才有效;④`AdvancedPutOptions` 默认 `now_utc()` 无法当"未设置"哨兵,需 Option 化。
-
-**方案**:阶段 0——`put/delete_object_tagging` 落 `SUFFIX_TAGGING_TIMESTAMP`(双前缀);阶段 1——新增三个 suffix 常量(对齐 MinIO headers.go:239-243),三字段 Option 化,`header()` 与 multipart 条件序列化;阶段 2——接收端解析(仅授权复制请求)+ PUT 路径 LWW 合并并持久化赢家时间戳(合并仅限三类元数据,不触碰数据与其余元数据,与 verbatim-replica 不变式共存)。
-**红灯测试**:单测 header 双前缀序列化断言/未设置缺席断言;接收端解析单测;e2e active-active tagging 并发收敛(晚者胜,现 main 旧值覆盖新值为红)。
-
-### P1-7 ARN 前缀(M)
-
-**订正后事实**:madmin `ParseARN` 硬校验 `arn:minio:` 前缀 + ID/bucket 非空(v3.0.109 remote-target-commands.go:50-63);mc 爆炸点仅 `replicate update`(fatalIf)与 `replicate ls`(软降级);`replicate add` 把 ARN 当不透明串不受影响——解释了"add 通 update 挂"。RustFS ARN 结构(`type::id:bucket`)与 madmin 兼容,仅 vendor token 障碍;另有 FromStr id/region 互换 bug(见紧急项)。
-
-**方案(推荐路线 A)**:生成侧默认改 `arn:minio:`(留常量可品牌化);解析侧接受双前缀(存量 `arn:rustfs:` 靠双前缀解析 + 现有字符串等值匹配继续工作);修字段序;改 `generate_arn`、`site_replication.rs:6329` 与相关测试断言。混合版本集群前缀不一致靠双前缀解析吸收;不做存量数据前缀归一化改写。
-**红灯测试**:单测 `from_str("arn:minio:replication:us-east-1:depl:bucket")` 成功且 id/region 正确(现双重红灯);round-trip 属性测试;e2e set-remote-target 返回 ARN 可被 madmin 语义解析、预置 `arn:minio:` 目标可 remove。
-
-### P1-8 配置校验(S)
-
-**订正后事实**:2MB 上限 REFUTED(MinIO 亦无显式检查,剔除);StorageClass 已缓解且刻意设计成立——MinIO 自己也不消费 rule 级 `Destination.StorageClass`(复制 PUT 用 target 级 `tgt.StorageClass`),RustFS target 级 storage_class 已真正生效(`bucket_target_sys.rs:1633-1634`),容忍显式 STANDARD 已实现。仍缺:规则数≤1000、≥1 条、Priority 唯一非负、ID≤255、Filter 互斥、Tag×DeleteMarkerReplication 互斥、sameTarget 拒绝。
-
-**方案**:`config.rs` 新增 `validate_replication_config_structure` 纯函数,`bucket_usecase.rs:2418` 接入;StorageClass 保持现状+契约文档化("rule 级请改用 remote target 的 storageclass 字段")。
-**红灯测试**:单测逐格(1001 规则/重复 Priority/256 字符 ID/Filter 并存/Tag+DMR)期望特定错误;e2e aws-sdk 形状 XML 断言 InvalidRequest。
-
-### P1-11 replication-metrics DTO(M)
-
-**订正后事实**:`BucketStats` 走内部 peer RPC 线格式(`rmp_serde::to_vec_named` 字段名入线,node_service.rs:1401 / peer_rest_client.rs:88-104)——**改原结构 serde 名会破坏混合版本集群 RPC,禁止**;必须走 #5754 的响应 DTO 模式(同文件先例 `remote_target_admin_json`)。
-
-**方案**:新增仅 Serialize 的 `MetricsV2Dto{uptime,currStats,queueStats,downtimeInfo}`/`MetricsDto`/`TargetMetricsDto`,显式映射(`q_stat`→`queued`、`bandwidth_limit_bytes_per_sec`→`limitInBits`、failed→TimedErrStats total-only);`queueStats.nodes` 先填本机一条;RustFS 可观测性扩展键保留(Go 忽略未知键,双栖零成本)。
-**红灯测试**:e2e 用镜像 minio-go MetricsV2 tag 的结构反序列化断言 `currStats.completedReplicationSize > 0`(现全零);DTO 键名 snapshot 单测。
-
-### P1-12 replication-reset 响应壳(S)
-
-**订正后事实**:致命键仅 5 个——壳 `Targets`≠`target`、`Status`≠`resyncStatus`、`ReplicatedSize`≠`completedReplicationSize`、`ReplicatedCount`≠`replicationCount`、`FailedSize/FailedCount`≠`failedReplicationSize/failedReplicationCount`;其余(Arn/ResetID/StartTime/...)靠 Go 大小写不敏感能对上;`ResetBeforeDate`/`Error` 是增强字段可保留。响应结构是 router.rs 独立 DTO 无内部复用,改名零风险。
-
-**方案**:纯 serde rename(建议全字段精确对齐 madmin 小写形态),保留增强键+文档标注。
-**红灯测试**:e2e 断言响应含 `target` 数组且 `target[0].resetid` 非空、status 侧 `resyncStatus`/`completedReplicationSize` 键存在。
-
-### P1-13 mrf/diff 流式响应(diff S / mrf M)
-
-**订正后事实**:症状比"输出空"更糟——聚合对象会被 madmin `json.Decoder` 成功解码一次,`mc replicate backlog` 输出一条 object 为空的**伪行**(静默伪数据);路线 A(保持聚合+文档化)无法消除伪行且与 madmin 同 path 无内容协商,**不可行**。数据源评估:diff 已逐条扫描只需去壳;mrf 的 durable backlog(`MrfReplicateEntry` 字段恰好覆盖 `ReplicationMRF` 所需)已可枚举。
-
-**方案(路线 B)**:diff 去壳输出 NDJSON `DiffInfo` 形状(仅 `IsDeleteMarker`/`ReplicationStatus` 需 rename;truncation 信息入日志不入流);mrf 遍历 durable entries 逐条输出 `ReplicationMRF` 形状(nodeName 填本机);聚合响应保留在 `?aggregate=true`(RustFS 扩展,deliberate 注释随迁)。条目量有 `REPLICATION_DIFF_MAX_SCAN` 封顶,内存拼 NDJSON 即可不必真流式。
-**红灯测试**:e2e 制造失败复制后逐行反序列化断言至少一条 `object` 非空(现为伪空行);diff 断言无 `Entries` 壳。
-
-### P1-14 set-remote-target(S,含紧急项)
-
-见"一、紧急项"。另:`deny_unknown_fields` **保留**(推荐)——字段清单已与 madmin v3.0.109 全同步,严格模式+显式清单兼得契约哲学与防静默;代价写进维护清单:"madmin 版本升级时同步字段清单"(加对照 madmin tag 列表的常量测试防漂移)。
-
-### P1-15 site state 统一 store(M-L,两 PR)
-
-**订正后事实**:主 state 有进程内 Mutex(`:347`)但两处不完备——①无分布式锁(多节点 RMW 丢更新);②**retry event enqueue/dequeue 不持锁**(挂在所有 hook 广播路径上,同进程即可丢更新);reload 路径完全无锁(稳态不写盘收窄窗口,迁移期可覆盖并发写)。repair state 的 `with_config_object_write_lock` + no-lock IO 是正确样板(`:1097-1114`);两套归一化的语义差异(JSON-level 容忍畸形 peer)是**有意的**,统一时必须保留。锁序注释 `:346` 可挂靠。
-
-**方案**:PR1——新建 `admin/site_replication_state.rs`:两阶段归一化合一(JSON 宽容清洗→类型化)、`read_state()/update_state(F)`(分布式锁包完整 RMW,锁内禁网络调用与嵌套配置锁)、常量收敛;service reload 接入;迁移 service 侧 5 个归一化测试保语义。PR2——迁移全部 ~30 个 RMW 调用点(含 enqueue/dequeue),**移除**进程内 Mutex(避免双锁新顺序约束);dequeue 热路径保留"先无锁读、命中才进 update_state"两段式;更新锁序注释。每个调用点做重入审查(现有 drop-reacquire 模式保持)。
-**红灯测试**:L2 单进程并发——持锁 RMW(mark_pending_rotation_peer_acked)×绕锁写者(enqueue_retry_event)注入交错,断言最终 state 两者共存(现必丢其一,确定性红灯);L1 归一化等价性测试迁移;L3 双节点并发(nice-to-have)。
-**风险**:盘上格式不变;锁超时从"静默丢更新"变"显式报错",hook 路径保持 warn 不阻断 S3 主路径。
-
-### P1-16 类型对账护栏(S)+ 死代码清理(M)
-
-**订正后事实**:drift 已发生(filemeta 侧 `MrfOpKind` 缺 Metadata/Heal/ExistingObject 三 variant、`MrfReplicateEntry` 缺 force_delete/target_arns)——但 filemeta 侧 8 个 worker DTO 全是**死代码**(零消费者);活跃双份仅 `ReplicationStatusType/VersionPurgeStatusType/ReplicationState` 三个 wire 类型(filemeta 绑 xl.meta 磁盘格式,replication 绑 MRF/resync 持久化格式);boundary 枚举转换 `as_str()` 兜底 `_ => Empty` 会静默降级。"抽公共 leaf crate"否决(两 wire 格式演进节奏不同,迁移规则 #12 本意是所有权独立)。
-
-**方案**:Step 1(S,即刻)——boundary 加对账测试:两侧枚举穷尽 match(新增 variant 即编译失败)+ as_str 双向 round-trip + ReplicationState 全字段往返;Step 2(M)——清理 filemeta 侧 ~600 行死代码 DTO,注意 crates.io semver(先 `#[deprecated]` 一版再删);Step 3(S)——replication 侧注释指向对账测试。
-
-### P1-17 迁移完成判据(M0=S;整体 L)
-
-**订正后事实**:"合并 boundary 微文件"REFUTED——守护脚本按具体文件名锚定每个 boundary,合并要同步改脚本+mod+导入点而功能收益为零;微文件是棘轮机制的机械接缝。唯一可退役:`datatypes.rs`(消费者迁完即删)。README 建议的第一步(event sink/runtime boundary)实际已部分落地,文档滞后。
-
-**方案**:M0(S)文档 PR——完成判据 = Required Contracts 表 "Current dependency to remove" 列清空;终态 = pool/resyncer/state 移入 crates/replication,boundary 随 crate 移动自然消解;更新 split-plan "Proposal only" 状态。M2(M)resyncer 纯决策逻辑下沉;M3(L)trait 稳定后移 worker 运行时(全计划唯一高危段,最后做);M4(S)统一退役 boundary 与守护条目。**不做**批量合并微文件。
-
-### P1-18 超长函数拆分(M;4 个 PR)
-
-**订正后事实**:行数确认(resync_bucket 537 / start_mrf_processor 306 / replicate_all 409 / delete 路径 replicate_object 299 / apply_iam_item 255);`apply_iam_item` **降级 P2 不拆**(6 臂 dispatch,每臂线性短小,拆分违反 "Prefer direct, local code");`replicate_object` 有两个同名体,原清单指 delete 路径 trait impl。
-
-**方案**(每函数独立 PR,纯移动,`git diff --color-moved=dimmed-zebra` 验证):
-1. `resync_bucket`(最优先,三处历史并发 bug 注释所在):acquire_resync_leadership / load_resync_replication_config / spawn workers+collector 三段抽出,并发 bug 注释随代码移动,每个 return 前的 mark_status 逐一保持;
-2. `start_mrf_processor`:抽 `reconstruct_mrf_delete/object` 纯函数(主循环 -150 行,重建逻辑可单测);
-3. `replicate_all` + delete 路径 `replicate_object`:各拆 3-4 个阶段 helper;**明确不合并两函数**(delete-marker 404/405 校验语义是刻意差异)。
-**排序依赖**:先 P1-18 拆分、后 P1-17 M2/M3 迁移(小函数降低搬运风险)。
-
-### P1-19 版本身份策略(M,推荐方案 B)
-
-**订正后事实**:#5752 已合入(PUT/multipart initiate 带 query,RustFS 目标侧也支持);PUT 响应 `x-amz-version-id` 仍被丢弃(`:1891 Ok(_)`);**delete-marker 子案已系统性缓解**——`remove_object` 捕获目标版本号→`target_delete_marker_version_ids` 持久化进 xl.meta(含上限与损坏标记)→延迟 purge 优先用映射、损坏拒猜(**保持不变**);RustFS 无"仅支持 MinIO 目标"契约声明;replication-check 探针已捕获响应版本号但不比对。MinIO 同样丢弃响应版本号(平价),RustFS 已有两点增强。
-
-**方案对比**:A 全量映射持久化(完整但 xl.meta 膨胀、全链路改造,L);**B(推荐)**:契约=仅支持"沿用源版本 ID"的目标,在 replication-check 增加 VersionFidelity phase(探针 PUT 带 versionId query,比对响应版本号)+ `validate_target` 复用同一探测,不镜像则新错误 `BucketRemoteTargetVersionMismatch` 显式拒绝/告警(M);C 混合(无需求支撑)。探针是主动写,进 validate_target 会扩 set-target 副作用面——可先只做 check phase + 运行期首次 PUT 抽查告警。
-**红灯测试**:FakeS3Target 加 `assign_own_version_ids` 开关模拟原生 S3,断言版本删除复制落空(现红)与探测后显式拒绝(修后绿)。
-
-### P1-20 scanner 补偿边界 e2e(M,纯测试)
-
-**订正后事实**:决策函数单测(queue.rs 7 例等)与 scanner 驱动的 Failed-heal e2e(target 断电恢复/源重启重放,FAST_SCANNER_ENV)已存在;真实缺口=无任何"先写对象→后配复制"的 existing-object 用例。完整入队真值表已梳理(见复审记录):Enabled×Empty 补齐、Pending/Failed 恒补(不受 existing 开关影响)、Disabled×Empty 永不补、Replica 恒不补(防环)、null-version 永不入队、reset_id 重置补齐。
-
-**方案**:e2e 矩阵 1-2 个用例(先 PUT 四种来源对象含 Copy/Snowball 产物→后配 Enabled/Disabled 规则→正例 wait_for_replicated_object / 负例 assert_failed_replication_stays_absent_for ≥3 周期,**"永不补齐"是契约必须显式断言**)+ Replica 防环变体 + queue.rs 补 2 格单测;null-version 跳过行为先写"记录现状"断言并注明出处。不改产品代码。
-
-### P1-21 delayed purge 失败处理(M)
-
-**订正后事实**:静默点两处——target client 缺失 `continue` 无日志(`:1673-1675`)、`let _ = remove_object`(`:1693-1700`);5 次循环是等源 marker 消失非重试;purge 调用后无条件 break;MRF 入队接口(`queue_replica_delete_task`,队满自动落盘)同 crate 可用无分层障碍;映射优先/损坏拒猜是已加固项保持不变。**附带发现**(建议单独跟进):`requires_delayed_purge` 恒真使 delete-marker 类 MRF 条目 outcome 恒 false → 重放永远 Missed 保留,可能永久滞留。
-
-**方案**:①purge 函数返回 per-target 成败,失败 warn(带 event 常量)+ metrics,client 缺失同样 warn(S);②循环内失败重试、轮次耗尽入 MRF、入队失败 warn+metric 兜底(S/M);③两层失败注入测试(mock 503 断言重试/状态/MRF;FakeS3Target inject 断言故障清除后最终收敛)(M)。风险:MRF 重放重发 DELETE marker 创建——mtime 幂等,风险低。
-
-### P1-22 SSE 能力(L,4 阶段)
-
-**订正后事实**:fail-closed 由 #5633 引入(`replication_target_boundary.rs:101-174`),普通/Heal/Resync/Multipart 全走同一函数;SSE-C 发送半边已建(内部头→`X-Rustfs-Replication-*` 映射+CRC),**目标侧摄取代码完全缺失**(链路必断,e2e 已钉 FAILED);SSE-S3 契约 e2e 的 `#[ignore]` 理由(backlog#1291 silently drops)已被 #5633 过期;直传托管 SSE 不可行(封存密钥绑本站 KMS),MinIO 是源解密+目标重加密;ecstore 已有 `ObjectEncryptionResolver` trait seam,解密不破分层。
-
-**方案**:阶段 0(S)摘 ignore + 补 encrypted resync/heal e2e 钉全矩阵 fail-closed 现状;阶段 1(M)SSE-C 目标侧头摄取+加密尺寸/CRC(MinIO :1670-1740 参照);阶段 2(M/L)SSE-S3 经 resolver 解密+目标 AES256 重加密(resolver 未注册必须继续 fail closed;multipart 按明文尺寸分片);阶段 3(L)SSE-KMS + key id 随行开关(目标站无同名 key 显式失败,禁止回退 SSE-S3)。过渡期全矩阵维持 fail closed,禁止明文降级。
-
----
-
-## 三、执行批次建议
-
-| 批次 | 内容 | 性质 |
-|---|---|---|
-| **B0 立即** | P1-14 零值 expiration + latency 忽略(S);P1-7 FromStr 字段互换(并入 P1-7 或先行) | mc 阻断修复 |
-| **B1 小改动高收益** | P1-12 响应壳 rename(S)、P1-13 diff 去壳(S)、P1-8 结构校验(S)、P1-16 Step1 对账测试(S)、P1-17 M0 文档判据(S)、P1-22 阶段 0 摘 ignore(S) | serde/校验/测试护栏 |
-| **B2 数据一致性** | P1-21 purge 失败处理(M)→ P1-20 scanner 矩阵 e2e(M,纯测试)→ P1-19 方案 B 能力探测(M)→ P1-15 state store PR1+PR2(M-L) | 一致性核心 |
-| **B3 互操作补齐** | P1-7 ARN 路线 A(M)、P1-11 MetricsV2 DTO(M)、P1-13 mrf 流(M)、P1-6 时间戳三段(M)、P1-1 ILM merge(M)、P1-3 retry drain(M) | mc/跨站语义 |
-| **B4 大功能与架构** | P1-5 proxy P0 段(M→L)、P1-22 阶段 1-3(L)、P1-18 四函数拆分(M)→ P1-17 M2-M4(L)、P1-16 Step2 死代码(M) | 长期 |
-
-**批内依赖**:P1-6 先于 P1-5 的 tagging proxy;P1-18 先于 P1-17 M2/M3;P1-15 PR1 的锁对象可先供 P1-3 drain 使用。

_typos.toml

@@ -47,9 +47,6 @@ consts = "consts"
 Hashi = "Hashi"  # HashiCorp
 # Accept alternate spelling used in parser/XML comments.
 unparseable = "unparseable"
-# Disaster-recovery objectives: recovery time and recovery point.
-RTO = "RTO"
-rto = "rto"
 
 [files]
 extend-exclude = []

crates/audit/Cargo.toml

@@ -25,40 +25,14 @@ documentation = "https://docs.rs/rustfs-audit/latest/rustfs_audit/"
 keywords = ["audit", "target", "management", "fan-out", "RustFS"]
 categories = ["web-programming", "development-tools", "asynchronous", "api-bindings"]
 
-[features]
-default = []
-hotpath = [
-    "hotpath/hotpath",
-    "hotpath/tokio",
-    "hotpath/futures",
-    "rustfs-config/hotpath",
-    "rustfs-s3-types/hotpath",
-    "rustfs-targets/hotpath",
-]
-hotpath-alloc = [
-    "hotpath",
-    "hotpath/hotpath-alloc",
-    "rustfs-config/hotpath-alloc",
-    "rustfs-s3-types/hotpath-alloc",
-    "rustfs-targets/hotpath-alloc",
-]
-hotpath-cpu = [
-    "hotpath",
-    "hotpath/hotpath-cpu",
-    "rustfs-config/hotpath-cpu",
-    "rustfs-s3-types/hotpath-cpu",
-    "rustfs-targets/hotpath-cpu",
-]
-
 [dependencies]
-hotpath.workspace = true
 rustfs-targets = { workspace = true }
 rustfs-config = { workspace = true, features = ["audit", "server-config-model"] }
 rustfs-s3-types = { workspace = true }
+chrono = { workspace = true, features = ["serde"] }
 const-str = { workspace = true, features = ["std", "proc"] }
 futures = { workspace = true }
 hashbrown = { workspace = true, features = ["serde", "rayon"] }
-jiff = { workspace = true, features = ["serde"] }
 metrics = { workspace = true }
 serde = { workspace = true, features = ["derive"] }
 serde_json = { workspace = true, features = ["raw_value"] }

crates/audit/src/entity.rs

@@ -12,8 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use chrono::{DateTime, Utc};
 use hashbrown::HashMap;
-use jiff::Timestamp;
 use rustfs_s3_types::EventName;
 use serde::{Deserialize, Serialize};
 use serde_json::Value;
@@ -151,8 +151,8 @@ pub struct AuditEntry {
     pub deployment_id: Option<String>,
     #[serde(rename = "siteName", skip_serializing_if = "Option::is_none")]
     pub site_name: Option<String>,
-    #[serde(with = "jiff::fmt::serde::timestamp::millisecond::required")]
-    pub time: Timestamp,
+    #[serde(with = "chrono::serde::ts_milliseconds")]
+    pub time: DateTime<Utc>,
     pub event: EventName,
     #[serde(rename = "type", skip_serializing_if = "Option::is_none")]
     pub entry_type: Option<String>,
@@ -198,7 +198,7 @@ impl AuditEntryBuilder {
     pub fn new(version: impl Into<String>, event: EventName, trigger: impl Into<String>, api: ApiDetails) -> Self {
         Self(AuditEntry {
             version: version.into(),
-            time: Timestamp::now(),
+            time: Utc::now(),
             event,
             trigger: trigger.into(),
             api,
@@ -232,7 +232,7 @@ impl AuditEntryBuilder {
         self
     }
 
-    pub fn time(mut self, time: Timestamp) -> Self {
+    pub fn time(mut self, time: DateTime<Utc>) -> Self {
         self.0.time = time;
         self
     }
@@ -342,23 +342,4 @@ mod tests {
         assert_eq!(value["requestID"], Value::String("req-audit-123".to_string()));
         assert!(value.get("request_id").is_none(), "historical audit contract must not expose request_id");
     }
-
-    #[test]
-    fn audit_entry_time_serializes_as_epoch_milliseconds() {
-        let entry = AuditEntryBuilder::new(
-            "1",
-            EventName::ObjectCreatedPut,
-            "s3",
-            ApiDetailsBuilder::new()
-                .name("PutObject")
-                .status("OK")
-                .status_code(200)
-                .build(),
-        )
-        .time(Timestamp::from_millisecond(1_711_423_698_870).expect("timestamp should be valid"))
-        .build();
-
-        let value = serde_json::to_value(entry).expect("audit entry should serialize");
-        assert_eq!(value["time"], Value::Number(1_711_423_698_870_i64.into()));
-    }
 }

crates/audit/tests/performance_test.rs

@@ -97,7 +97,7 @@ async fn test_audit_log_dispatch_performance() {
         return; // Alternatively: assert!(false, "AuditSystem failed to start");
     }
 
-    use jiff::Timestamp;
+    use chrono::Utc;
     use rustfs_targets::EventName;
     use serde_json::json;
     use std::collections::HashMap;
@@ -136,7 +136,7 @@ async fn test_audit_log_dispatch_performance() {
         version: "1".to_string(),
         deployment_id: Some(format!("test-deployment-{id}")),
         site_name: Some("test-site".to_string()),
-        time: Timestamp::now(),
+        time: Utc::now(),
         event: EventName::ObjectCreatedPut,
         entry_type: Some("object".to_string()),
         trigger: "api".to_string(),
@@ -298,7 +298,7 @@ fn test_performance_requirements() {
     for i in 0..3000 {
         // Simulate event name parsing and processing
         let _event_id = format!("s3:ObjectCreated:Put_{i}");
-        let _timestamp = jiff::Timestamp::now().to_string();
+        let _timestamp = chrono::Utc::now().to_rfc3339();
 
         // Simulate basic audit entry creation overhead
         let _entry_size = 512; // bytes

crates/audit/tests/system_integration_test.rs

@@ -264,7 +264,7 @@ fn create_sample_audit_entry() -> AuditEntry {
 }
 
 fn create_sample_audit_entry_with_id(id: u32) -> AuditEntry {
-    use jiff::Timestamp;
+    use chrono::Utc;
     use rustfs_targets::EventName;
     use serde_json::json;
 
@@ -301,7 +301,7 @@ fn create_sample_audit_entry_with_id(id: u32) -> AuditEntry {
         version: "1".to_string(),
         deployment_id: Some(format!("test-deployment-{id}")),
         site_name: Some("test-site".to_string()),
-        time: Timestamp::now(),
+        time: Utc::now(),
         event: EventName::ObjectCreatedPut,
         entry_type: Some("object".to_string()),
         trigger: "api".to_string(),

crates/checksums/Cargo.toml

@@ -28,14 +28,7 @@ documentation = "https://docs.rs/rustfs-checksums/latest/rustfs_checksum/"
 [lints]
 workspace = true
 
-[features]
-default = []
-hotpath = ["hotpath/hotpath"]
-hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
-hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
-
 [dependencies]
-hotpath.workspace = true
 bytes = { workspace = true, features = ["serde"] }
 crc-fast = { workspace = true }
 http = { workspace = true }

crates/common/Cargo.toml

@@ -27,14 +27,7 @@ categories = ["web-programming", "development-tools", "data-structures"]
 [lints]
 workspace = true
 
-[features]
-default = []
-hotpath = ["hotpath/hotpath", "hotpath/tokio"]
-hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
-hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
-
 [dependencies]
-hotpath.workspace = true
 tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
 tonic = { workspace = true, features = ["gzip", "deflate"] }
 uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }

crates/common/src/heal_channel.rs

@@ -356,8 +356,6 @@ pub struct HealChannelRequest {
     pub recursive: Option<bool>,
     /// Whether to dry run
     pub dry_run: Option<bool>,
-    /// Whether to skip namespace locking
-    pub no_lock: Option<bool>,
     /// Timeout in seconds (optional)
     pub timeout_seconds: Option<u64>,
     /// Origin of the request for operational status and queue accounting
@@ -562,7 +560,6 @@ pub fn create_heal_request(
         update_parity: None,
         recursive: None,
         dry_run: None,
-        no_lock: None,
         timeout_seconds: None,
         source: HealRequestSource::Internal,
         disk: None,
@@ -721,7 +718,6 @@ pub async fn send_heal_disk(set_disk_id: String, priority: Option<HealChannelPri
         update_parity: None,
         recursive: None,
         dry_run: None,
-        no_lock: None,
         timeout_seconds: None,
         source: HealRequestSource::AutoHeal,
     };

crates/common/src/metrics.rs

@@ -17,7 +17,7 @@ use crate::last_minute::{AccElem, LastMinuteLatency};
 use chrono::{DateTime, Utc};
 use serde::{Deserialize, Serialize};
 use std::{
-    collections::{BTreeSet, HashMap},
+    collections::HashMap,
     fmt::Display,
     future::Future,
     pin::Pin,
@@ -708,48 +708,6 @@ struct ScannerDiskBucketScanState {
     active: u64,
 }
 
-type ScannerDiskBucketScanKey = (String, String);
-
-#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
-pub struct ScannerDiskBucketScanSnapshot {
-    pub pool: String,
-    pub set: String,
-    pub concurrency_limit: u64,
-    pub queued: u64,
-    pub active: u64,
-}
-
-#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
-struct ScannerBucketDriveResultKey {
-    bucket: String,
-    drive: String,
-    result: String,
-}
-
-impl ScannerBucketDriveResultKey {
-    fn new(bucket: impl Into<String>, drive: impl Into<String>, result: impl Into<String>) -> Self {
-        Self {
-            bucket: bucket.into(),
-            drive: drive.into(),
-            result: result.into(),
-        }
-    }
-}
-
-const MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS: usize = 4096;
-
-#[derive(Debug, Default)]
-struct ScannerBucketDriveResults {
-    counts: HashMap<ScannerBucketDriveResultKey, ScannerBucketDriveResultValue>,
-    eviction_index: BTreeSet<(u64, ScannerBucketDriveResultKey)>,
-}
-
-#[derive(Clone, Copy, Debug)]
-struct ScannerBucketDriveResultValue {
-    count: u64,
-    last_seen: u64,
-}
-
 // ---------------------------------------------------------------------------
 // Metrics
 // ---------------------------------------------------------------------------
@@ -780,11 +738,7 @@ pub struct Metrics {
     scanner_set_scan_concurrency_limit: AtomicU64,
     scanner_set_scans_queued: AtomicU64,
     scanner_set_scans_active: AtomicU64,
-    scanner_disk_bucket_scan_states: Mutex<HashMap<ScannerDiskBucketScanKey, ScannerDiskBucketScanState>>,
-    scanner_bucket_drive_results: Mutex<ScannerBucketDriveResults>,
-    scanner_bucket_drive_result_clock: AtomicU64,
-    current_scan_cycle_bucket_drive_results_start: Mutex<HashMap<ScannerBucketDriveResultKey, u64>>,
-    last_scan_cycle_bucket_drive_results: Mutex<Vec<ScannerBucketDriveResultSnapshot>>,
+    scanner_disk_bucket_scan_states: Mutex<HashMap<String, ScannerDiskBucketScanState>>,
     scanner_leader_lock_state: RwLock<String>,
     scanner_leader_lock_held: AtomicBool,
     scanner_leader_lock_last_error: RwLock<String>,
@@ -1004,14 +958,6 @@ pub struct ScannerSourceWorkSnapshot {
     pub missed: u64,
 }
 
-#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
-pub struct ScannerBucketDriveResultSnapshot {
-    pub bucket: String,
-    pub drive: String,
-    pub result: String,
-    pub count: u64,
-}
-
 #[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
 pub struct ScannerReplicationRepairSnapshot {
     pub source: String,
@@ -1344,18 +1290,6 @@ pub struct ScannerMetricsReport {
     pub partial_cycles: u64,
 }
 
-#[derive(Clone, Debug, Default, Serialize, Deserialize)]
-pub struct ScannerRuntimeDetailsReport {
-    #[serde(default)]
-    pub disk_bucket_scan_states: Vec<ScannerDiskBucketScanSnapshot>,
-    #[serde(default)]
-    pub bucket_drive_results: Vec<ScannerBucketDriveResultSnapshot>,
-    #[serde(default)]
-    pub current_cycle_bucket_drive_results: Vec<ScannerBucketDriveResultSnapshot>,
-    #[serde(default)]
-    pub last_cycle_bucket_drive_results: Vec<ScannerBucketDriveResultSnapshot>,
-}
-
 impl CurrentCycle {
     pub fn unmarshal(&mut self, buf: &[u8]) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         *self = rmp_serde::from_slice(buf)?;
@@ -1723,7 +1657,6 @@ pub fn emit_scan_cycle_superseded(duration: Duration) {
 
 pub fn emit_scan_bucket_drive_complete(success: bool, bucket: &str, disk: &str, duration: Duration) {
     let result = if success { "success" } else { "error" };
-    global_metrics().record_scanner_bucket_drive_result(bucket, disk, result);
     metrics::counter!(
         OTEL_SCANNER_BUCKETS_SCANNED,
         "result" => result,
@@ -1740,7 +1673,6 @@ pub fn emit_scan_bucket_drive_complete(success: bool, bucket: &str, disk: &str,
 }
 
 pub fn emit_scan_bucket_drive_partial(bucket: &str, disk: &str, duration: Duration) {
-    global_metrics().record_scanner_bucket_drive_result(bucket, disk, SCAN_CYCLE_RESULT_PARTIAL_LABEL);
     metrics::counter!(
         OTEL_SCANNER_BUCKETS_SCANNED,
         "result" => SCAN_CYCLE_RESULT_PARTIAL_LABEL,
@@ -1791,10 +1723,6 @@ impl Metrics {
             scanner_set_scans_queued: AtomicU64::new(0),
             scanner_set_scans_active: AtomicU64::new(0),
             scanner_disk_bucket_scan_states: Mutex::new(HashMap::new()),
-            scanner_bucket_drive_results: Mutex::new(ScannerBucketDriveResults::default()),
-            scanner_bucket_drive_result_clock: AtomicU64::new(0),
-            current_scan_cycle_bucket_drive_results_start: Mutex::new(HashMap::new()),
-            last_scan_cycle_bucket_drive_results: Mutex::new(Vec::new()),
             scanner_leader_lock_state: RwLock::new("unknown".to_string()),
             scanner_leader_lock_held: AtomicBool::new(false),
             scanner_leader_lock_last_error: RwLock::new(String::new()),
@@ -2365,7 +2293,7 @@ impl Metrics {
         queued: Option<usize>,
         active: Option<usize>,
     ) {
-        let key = (pool.to_string(), set.to_string());
+        let key = format!("{pool}/{set}");
         let mut states = self
             .scanner_disk_bucket_scan_states
             .lock()
@@ -2382,41 +2310,6 @@ impl Metrics {
         }
     }
 
-    pub fn record_scanner_bucket_drive_result(&self, bucket: &str, drive: &str, result: &str) {
-        if bucket.is_empty() || drive.is_empty() || result.is_empty() {
-            return;
-        }
-        let key = ScannerBucketDriveResultKey::new(bucket, drive, result);
-        let mut results = self
-            .scanner_bucket_drive_results
-            .lock()
-            .unwrap_or_else(|poisoned| poisoned.into_inner());
-        let last_seen = self.scanner_bucket_drive_result_clock.fetch_add(1, Ordering::Relaxed);
-        if let Some(previous_last_seen) = results.counts.get_mut(&key).map(|value| {
-            let previous_last_seen = value.last_seen;
-            value.count = value.count.saturating_add(1);
-            value.last_seen = last_seen;
-            previous_last_seen
-        }) {
-            results.eviction_index.remove(&(previous_last_seen, key.clone()));
-            results.eviction_index.insert((last_seen, key));
-            return;
-        }
-
-        if results.counts.len() >= MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS
-            && let Some((_, stale_key)) = results.eviction_index.pop_first()
-        {
-            results.counts.remove(&stale_key);
-        }
-
-        if results.counts.len() < MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS {
-            results
-                .counts
-                .insert(key.clone(), ScannerBucketDriveResultValue { count: 1, last_seen });
-            results.eviction_index.insert((last_seen, key));
-        }
-    }
-
     // -----------------------------------------------------------------------
     // Read-side helpers
     // -----------------------------------------------------------------------
@@ -2588,11 +2481,6 @@ impl Metrics {
             &self.current_scan_cycle_replication_repair_work_start,
             &replication_repair_snapshot,
         );
-        let bucket_drive_results = self.scanner_bucket_drive_result_counts();
-        match self.current_scan_cycle_bucket_drive_results_start.lock() {
-            Ok(mut start) => *start = bucket_drive_results,
-            Err(poisoned) => *poisoned.into_inner() = bucket_drive_results,
-        }
         self.current_scan_cycle_work_active.store(true, Ordering::Release);
         snapshot
     }
@@ -2605,11 +2493,6 @@ impl Metrics {
         self.record_scan_cycle_work(work);
         self.record_scan_cycle_source_work(&source_work);
         self.record_scan_cycle_replication_repair_work(&replication_repair_work);
-        let bucket_drive_results = self.current_cycle_bucket_drive_result_snapshots();
-        match self.last_scan_cycle_bucket_drive_results.lock() {
-            Ok(mut last) => *last = bucket_drive_results,
-            Err(poisoned) => *poisoned.into_inner() = bucket_drive_results,
-        }
         self.current_scan_cycle_work_active.store(false, Ordering::Release);
     }
 
@@ -2693,105 +2576,6 @@ impl Metrics {
         }
     }
 
-    fn scanner_bucket_drive_result_counts(&self) -> HashMap<ScannerBucketDriveResultKey, u64> {
-        self.scanner_bucket_drive_results
-            .lock()
-            .unwrap_or_else(|poisoned| poisoned.into_inner())
-            .counts
-            .iter()
-            .map(|(key, value)| (key.clone(), value.count))
-            .collect()
-    }
-
-    fn scanner_bucket_drive_result_snapshots(
-        counts: impl IntoIterator<Item = (ScannerBucketDriveResultKey, u64)>,
-    ) -> Vec<ScannerBucketDriveResultSnapshot> {
-        let mut snapshots = counts
-            .into_iter()
-            .filter(|(_, count)| *count > 0)
-            .map(|(key, count)| ScannerBucketDriveResultSnapshot {
-                bucket: key.bucket,
-                drive: key.drive,
-                result: key.result,
-                count,
-            })
-            .collect::<Vec<_>>();
-        snapshots.sort_by(|left, right| {
-            left.bucket
-                .cmp(&right.bucket)
-                .then_with(|| left.drive.cmp(&right.drive))
-                .then_with(|| left.result.cmp(&right.result))
-        });
-        snapshots
-    }
-
-    fn scanner_bucket_drive_result_counter_snapshots(&self) -> Vec<ScannerBucketDriveResultSnapshot> {
-        Self::scanner_bucket_drive_result_snapshots(self.scanner_bucket_drive_result_counts())
-    }
-
-    fn current_cycle_bucket_drive_result_snapshots(&self) -> Vec<ScannerBucketDriveResultSnapshot> {
-        let current = self.scanner_bucket_drive_result_counts();
-        let start = self
-            .current_scan_cycle_bucket_drive_results_start
-            .lock()
-            .unwrap_or_else(|poisoned| poisoned.into_inner())
-            .clone();
-        Self::scanner_bucket_drive_result_snapshots(current.into_iter().filter_map(|(key, count)| {
-            let delta = count.saturating_sub(start.get(&key).copied().unwrap_or_default());
-            (delta > 0).then_some((key, delta))
-        }))
-    }
-
-    pub fn scanner_runtime_details_report(&self) -> ScannerRuntimeDetailsReport {
-        self.scanner_runtime_details_report_for_active(self.current_scan_cycle_work_active.load(Ordering::Acquire))
-    }
-
-    fn scanner_runtime_details_report_for_active(&self, current_cycle_active: bool) -> ScannerRuntimeDetailsReport {
-        let current_cycle_bucket_drive_results = if current_cycle_active {
-            self.current_cycle_bucket_drive_result_snapshots()
-        } else {
-            Vec::new()
-        };
-        ScannerRuntimeDetailsReport {
-            disk_bucket_scan_states: self.scanner_disk_bucket_scan_state_snapshots(),
-            bucket_drive_results: self.scanner_bucket_drive_result_counter_snapshots(),
-            current_cycle_bucket_drive_results,
-            last_cycle_bucket_drive_results: self
-                .last_scan_cycle_bucket_drive_results
-                .lock()
-                .unwrap_or_else(|poisoned| poisoned.into_inner())
-                .clone(),
-        }
-    }
-
-    fn scanner_disk_bucket_scan_state_snapshots(&self) -> Vec<ScannerDiskBucketScanSnapshot> {
-        let mut disk_bucket_scan_states = match self.scanner_disk_bucket_scan_states.lock() {
-            Ok(states) => states
-                .iter()
-                .map(|((pool, set), state)| ScannerDiskBucketScanSnapshot {
-                    pool: pool.clone(),
-                    set: set.clone(),
-                    concurrency_limit: state.concurrency_limit,
-                    queued: state.queued,
-                    active: state.active,
-                })
-                .collect::<Vec<_>>(),
-            Err(poisoned) => poisoned
-                .into_inner()
-                .iter()
-                .map(|((pool, set), state)| ScannerDiskBucketScanSnapshot {
-                    pool: pool.clone(),
-                    set: set.clone(),
-                    concurrency_limit: state.concurrency_limit,
-                    queued: state.queued,
-                    active: state.active,
-                })
-                .collect::<Vec<_>>(),
-        };
-        disk_bucket_scan_states.sort_by(|left, right| left.pool.cmp(&right.pool).then_with(|| left.set.cmp(&right.set)));
-        disk_bucket_scan_states
-    }
-
     fn scanner_source_work_values(&self) -> Vec<ScannerSourceWorkValues> {
         ScannerWorkSource::all()
             .iter()
@@ -2977,12 +2761,7 @@ impl Metrics {
 
     /// Build a full metrics report snapshot.
     pub async fn report(&self) -> ScannerMetricsReport {
-        self.report_with_runtime_details().await.0
-    }
-
-    pub async fn report_with_runtime_details(&self) -> (ScannerMetricsReport, ScannerRuntimeDetailsReport) {
         let mut m = ScannerMetricsReport::default();
-        let runtime_details;
 
         let has_cycle = {
             let cycle = self.cycle_info.read().await;
@@ -2996,7 +2775,6 @@ impl Metrics {
             };
             m.current_cycle_active = self.current_scan_cycle_work_active.load(Ordering::Acquire);
             if m.current_cycle_active {
-                // Keep cycle_info before cycle-baseline locks so active scrapes cannot mix two cycle identities.
                 let current_work = self.scan_cycle_work_since(self.current_scan_cycle_work_start());
                 let current_source_work = self.scanner_source_work_since(&self.current_scan_cycle_source_work_start_values());
                 let current_replication_repair_work =
@@ -3019,7 +2797,6 @@ impl Metrics {
                 m.current_cycle_replication_repair =
                     self.scanner_replication_repair_work_snapshots(&current_replication_repair_work);
             }
-            runtime_details = self.scanner_runtime_details_report_for_active(m.current_cycle_active);
             has_cycle
         };
 
@@ -3049,11 +2826,15 @@ impl Metrics {
         m.current_set_scan_concurrency_limit = self.scanner_set_scan_concurrency_limit.load(Ordering::Relaxed);
         m.current_set_scans_queued = self.scanner_set_scans_queued.load(Ordering::Relaxed);
         m.current_set_scans_active = self.scanner_set_scans_active.load(Ordering::Relaxed);
-        let disk_bucket_scan_states = self.scanner_disk_bucket_scan_state_snapshots();
         let (disk_scan_concurrency_limit, disk_bucket_scans_queued, disk_bucket_scans_active) =
-            disk_bucket_scan_states.iter().fold((0, 0, 0), |acc, state| {
-                (acc.0 + state.concurrency_limit, acc.1 + state.queued, acc.2 + state.active)
-            });
+            match self.scanner_disk_bucket_scan_states.lock() {
+                Ok(states) => states.values().fold((0, 0, 0), |acc, state| {
+                    (acc.0 + state.concurrency_limit, acc.1 + state.queued, acc.2 + state.active)
+                }),
+                Err(poisoned) => poisoned.into_inner().values().fold((0, 0, 0), |acc, state| {
+                    (acc.0 + state.concurrency_limit, acc.1 + state.queued, acc.2 + state.active)
+                }),
+            };
         m.current_disk_scan_concurrency_limit = disk_scan_concurrency_limit;
         m.current_disk_bucket_scans_queued = disk_bucket_scans_queued;
         m.current_disk_bucket_scans_active = disk_bucket_scans_active;
@@ -3222,7 +3003,7 @@ impl Metrics {
         m.pacing_pressure = scanner_pacing_pressure(&m);
         m.maintenance_control = scanner_maintenance_control(&m);
 
-        (m, runtime_details)
+        m
     }
 }
 
@@ -4319,137 +4100,6 @@ mod tests {
         assert_eq!(report.life_time_ops.get("scan_bucket_drive_failure"), Some(&1));
     }
 
-    #[tokio::test]
-    async fn report_includes_structured_bucket_drive_results() {
-        let metrics = Metrics::new();
-        metrics.record_scanner_bucket_drive_result("photos", "/data1", "success");
-
-        let cycle_start = metrics.start_scan_cycle_work();
-        metrics.record_scanner_bucket_drive_result("photos", "/data1", "partial");
-
-        let active_report = metrics.scanner_runtime_details_report();
-        assert_eq!(
-            active_report.current_cycle_bucket_drive_results,
-            vec![ScannerBucketDriveResultSnapshot {
-                bucket: "photos".to_string(),
-                drive: "/data1".to_string(),
-                result: "partial".to_string(),
-                count: 1,
-            }]
-        );
-
-        metrics.finish_scan_cycle_work(cycle_start);
-
-        let report = metrics.scanner_runtime_details_report();
-
-        assert_eq!(
-            report.bucket_drive_results,
-            vec![
-                ScannerBucketDriveResultSnapshot {
-                    bucket: "photos".to_string(),
-                    drive: "/data1".to_string(),
-                    result: "partial".to_string(),
-                    count: 1,
-                },
-                ScannerBucketDriveResultSnapshot {
-                    bucket: "photos".to_string(),
-                    drive: "/data1".to_string(),
-                    result: "success".to_string(),
-                    count: 1,
-                },
-            ]
-        );
-        assert!(report.current_cycle_bucket_drive_results.is_empty());
-        assert_eq!(
-            report.last_cycle_bucket_drive_results,
-            vec![ScannerBucketDriveResultSnapshot {
-                bucket: "photos".to_string(),
-                drive: "/data1".to_string(),
-                result: "partial".to_string(),
-                count: 1,
-            }]
-        );
-    }
-
-    #[tokio::test]
-    async fn scanner_bucket_drive_results_are_bounded() {
-        let metrics = Metrics::new();
-        for index in 0..MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS {
-            metrics.record_scanner_bucket_drive_result(&format!("bucket-{index}"), "/data1", "success");
-        }
-        metrics.record_scanner_bucket_drive_result("overflow", "/data1", "success");
-
-        let report = metrics.scanner_runtime_details_report();
-
-        assert_eq!(report.bucket_drive_results.len(), MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS);
-        assert!(
-            report
-                .bucket_drive_results
-                .iter()
-                .any(|snapshot| snapshot.bucket == "overflow")
-        );
-        assert!(
-            report
-                .bucket_drive_results
-                .iter()
-                .all(|snapshot| snapshot.bucket != "bucket-0")
-        );
-    }
-
-    #[tokio::test]
-    async fn scanner_bucket_drive_result_eviction_keeps_recent_keys() {
-        let metrics = Metrics::new();
-        for index in 0..MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS {
-            metrics.record_scanner_bucket_drive_result(&format!("bucket-{index}"), "/data1", "success");
-        }
-        metrics.record_scanner_bucket_drive_result("bucket-0", "/data1", "success");
-        metrics.record_scanner_bucket_drive_result("overflow", "/data1", "success");
-
-        let report = metrics.scanner_runtime_details_report();
-
-        assert_eq!(report.bucket_drive_results.len(), MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS);
-        assert!(
-            report
-                .bucket_drive_results
-                .iter()
-                .any(|snapshot| snapshot.bucket == "bucket-0" && snapshot.count == 2)
-        );
-        assert!(
-            report
-                .bucket_drive_results
-                .iter()
-                .all(|snapshot| snapshot.bucket != "bucket-1")
-        );
-    }
-
-    #[tokio::test]
-    async fn scanner_bucket_drive_result_eviction_survives_full_refresh() {
-        let metrics = Metrics::new();
-        for index in 0..MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS {
-            metrics.record_scanner_bucket_drive_result(&format!("bucket-{index}"), "/data1", "success");
-        }
-        for index in 0..MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS {
-            metrics.record_scanner_bucket_drive_result(&format!("bucket-{index}"), "/data1", "success");
-        }
-        metrics.record_scanner_bucket_drive_result("overflow", "/data1", "success");
-
-        let report = metrics.scanner_runtime_details_report();
-
-        assert_eq!(report.bucket_drive_results.len(), MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS);
-        assert!(
-            report
-                .bucket_drive_results
-                .iter()
-                .any(|snapshot| snapshot.bucket == "overflow")
-        );
-        assert!(
-            report
-                .bucket_drive_results
-                .iter()
-                .all(|snapshot| snapshot.bucket != "bucket-0")
-        );
-    }
-
     #[tokio::test]
     async fn report_includes_usage_freshness_status() {
         let metrics = Metrics::new();
@@ -4615,10 +4265,9 @@ mod tests {
         };
         let cycle_ten_start = metrics.start_scan_cycle_work_with_cycle(cycle_ten.clone()).await;
         metrics.operations[Metric::ScanObject as usize].store(1, Ordering::Relaxed);
-        metrics.record_scanner_bucket_drive_result("cycle-ten", "/data1", "partial");
 
         let paths = metrics.current_paths.write().await;
-        let mut report = Box::pin(metrics.report_with_runtime_details());
+        let mut report = Box::pin(metrics.report());
         let waker = std::task::Waker::noop();
         let mut context = std::task::Context::from_waker(waker);
         assert!(report.as_mut().poll(&mut context).is_pending());
@@ -4635,22 +4284,12 @@ mod tests {
             })
             .await;
         metrics.operations[Metric::ScanObject as usize].store(101, Ordering::Relaxed);
-        metrics.record_scanner_bucket_drive_result("cycle-eleven", "/data1", "partial");
 
         drop(paths);
-        let (snapshot, runtime_details) = report.await;
+        let snapshot = report.await;
 
         assert_eq!(snapshot.current_cycle, 10);
         assert_eq!(snapshot.current_cycle_objects_scanned, 1);
-        assert_eq!(
-            runtime_details.current_cycle_bucket_drive_results,
-            vec![ScannerBucketDriveResultSnapshot {
-                bucket: "cycle-ten".to_string(),
-                drive: "/data1".to_string(),
-                result: "partial".to_string(),
-                count: 1,
-            }]
-        );
 
         metrics
             .finish_scan_cycle_work_with_cycle(cycle_eleven_start, CurrentCycle::default())

crates/concurrency/Cargo.toml

@@ -13,14 +13,7 @@ categories = ["concurrency", "filesystem"]
 [lints]
 workspace = true
 
-[features]
-default = []
-hotpath = ["hotpath/hotpath", "hotpath/tokio", "rustfs-io-core/hotpath"]
-hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-io-core/hotpath-alloc"]
-hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-io-core/hotpath-cpu"]
-
 [dependencies]
-hotpath.workspace = true
 # Internal crates
 rustfs-io-core = { workspace = true }
 serde = { workspace = true, features = ["derive"] }

crates/config/Cargo.toml

@@ -25,7 +25,6 @@ keywords = ["configuration", "settings", "management", "rustfs", "Minio"]
 categories = ["web-programming", "development-tools", "config"]
 
 [dependencies]
-hotpath.workspace = true
 const-str = { workspace = true, optional = true, features = ["std", "proc"] }
 serde = { workspace = true, optional = true, features = ["derive"] }
 serde_json = { workspace = true, optional = true, features = ["raw_value"] }
@@ -35,9 +34,6 @@ workspace = true
 
 [features]
 default = ["constants"]
-hotpath = ["hotpath/hotpath"]
-hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
-hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
 audit = ["dep:const-str", "constants"]
 constants = ["dep:const-str"]
 notify = ["dep:const-str", "constants"]

crates/config/src/constants/app.rs

@@ -230,19 +230,6 @@ pub const ENV_RUSTFS_KMS_ENABLE: &str = "RUSTFS_KMS_ENABLE";
 /// Default value: false
 pub const DEFAULT_KMS_ENABLE: bool = false;
 
-/// Environment variable enabling per-key KMS authorization on the SSE-KMS data path.
-///
-/// When enabled, an SSE-KMS write additionally requires `kms:GenerateDataKey` and an
-/// SSE-KMS read additionally requires `kms:Decrypt` on the resolved key, evaluated as
-/// the requesting identity. SSE-S3 and SSE-C are unaffected.
-pub const ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY: &str = "RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY";
-
-/// Default per-key KMS authorization mode for the SSE-KMS data path.
-///
-/// Off for now so deployments whose identity policies only grant s3 actions keep
-/// working; the default flips to on in a later release.
-pub const DEFAULT_KMS_ENFORCE_SSE_KEY_POLICY: bool = false;
-
 /// Environment variable for server KMS backend.
 pub const ENV_RUSTFS_KMS_BACKEND: &str = "RUSTFS_KMS_BACKEND";
 

crates/credentials/Cargo.toml

@@ -24,14 +24,7 @@ description = "Credentials management utilities for RustFS, enabling secure hand
 keywords = ["rustfs", "Minio", "credentials", "authentication", "authorization"]
 categories = ["web-programming", "development-tools", "data-structures", "security"]
 
-[features]
-default = []
-hotpath = ["hotpath/hotpath"]
-hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
-hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
-
 [dependencies]
-hotpath.workspace = true
 base64-simd = { workspace = true }
 hmac = { workspace = true }
 rand = { workspace = true, features = ["serde"] }

crates/crypto/Cargo.toml

@@ -29,7 +29,6 @@ documentation = "https://docs.rs/rustfs-crypto/latest/rustfs_crypto/"
 workspace = true
 
 [dependencies]
-hotpath.workspace = true
 aes-gcm = { workspace = true, optional = true, features = ["rand_core"] }
 argon2 = { workspace = true, optional = true }
 chacha20poly1305 = { workspace = true, optional = true }
@@ -50,9 +49,6 @@ time = { workspace = true, features = ["parsing", "formatting", "macros", "serde
 
 [features]
 default = ["crypto", "fips"]
-hotpath = ["hotpath/hotpath"]
-hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
-hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
 fips = []
 crypto = [
     "dep:aes-gcm",

crates/data-usage/Cargo.toml

@@ -27,14 +27,7 @@ categories = ["data-structures", "filesystem"]
 [lints]
 workspace = true
 
-[features]
-default = []
-hotpath = ["hotpath/hotpath", "rustfs-filemeta/hotpath"]
-hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-filemeta/hotpath-alloc"]
-hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-filemeta/hotpath-cpu"]
-
 [dependencies]
-hotpath.workspace = true
 serde = { workspace = true, features = ["derive"] }
 rmp-serde = { workspace = true }
 async-trait = { workspace = true }

crates/data-usage/src/data_usage.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use serde::{Deserialize, Serialize, ser::SerializeMap as _};
+use serde::{Deserialize, Serialize};
 use std::{
     collections::{HashMap, HashSet},
     hash::{DefaultHasher, Hash, Hasher},
@@ -51,36 +51,24 @@ pub fn usage_last_update_is_untrusted_future(existing_last_update: SystemTime, n
     existing_last_update > now + USAGE_LAST_UPDATE_FUTURE_TOLERANCE
 }
 
-#[derive(Clone, Copy, Default, Debug, Serialize, Deserialize, PartialEq, Eq)]
+#[derive(Clone, Copy, Default, Debug, Serialize, Deserialize, PartialEq)]
 pub struct TierStats {
     pub total_size: u64,
-    pub num_versions: u64,
-    pub num_objects: u64,
+    pub num_versions: i32,
+    pub num_objects: i32,
 }
 
 impl TierStats {
     pub fn add(&self, u: &TierStats) -> TierStats {
         TierStats {
-            total_size: self.total_size.saturating_add(u.total_size),
-            num_versions: self.num_versions.saturating_add(u.num_versions),
-            num_objects: self.num_objects.saturating_add(u.num_objects),
+            total_size: self.total_size + u.total_size,
+            num_versions: self.num_versions + u.num_versions,
+            num_objects: self.num_objects + u.num_objects,
         }
     }
-
-    /// True when [`TierStats::add`] would report the exact sum instead of saturating.
-    pub fn fits_add(&self, u: &TierStats) -> bool {
-        self.total_size.checked_add(u.total_size).is_some()
-            && self.num_versions.checked_add(u.num_versions).is_some()
-            && self.num_objects.checked_add(u.num_objects).is_some()
-    }
-
-    /// True when this tier contributed nothing, i.e. merging it is a no-op.
-    pub fn is_empty(&self) -> bool {
-        self.total_size == 0 && self.num_versions == 0 && self.num_objects == 0
-    }
 }
 
-#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
+#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq)]
 pub struct AllTierStats {
     pub tiers: HashMap<String, TierStats>,
 }
@@ -90,35 +78,31 @@ impl AllTierStats {
         Self { tiers: HashMap::new() }
     }
 
-    pub fn is_empty(&self) -> bool {
-        self.tiers.is_empty()
-    }
-
-    /// Folds a scan summary's per-tier map in.
-    ///
-    /// Scanners seed the map with a zeroed entry for every configured tier, so
-    /// empty contributions are skipped to keep the persisted cache from growing
-    /// one key per tier on every folder that never held tiered data.
-    pub fn add_sizes(&mut self, tiers: &HashMap<String, TierStats>) {
+    pub fn add_sizes(&mut self, tiers: HashMap<String, TierStats>) {
         for (tier, st) in tiers {
-            if st.is_empty() {
-                continue;
-            }
-            let entry = self.tiers.entry(tier.clone()).or_default();
-            *entry = entry.add(st);
+            self.tiers
+                .insert(tier.clone(), self.tiers.get(&tier).copied().unwrap_or_default().add(&st));
         }
     }
 
-    pub fn merge(&mut self, other: &AllTierStats) {
-        self.add_sizes(&other.tiers);
+    pub fn merge(&mut self, other: AllTierStats) {
+        for (tier, st) in other.tiers {
+            self.tiers
+                .insert(tier.clone(), self.tiers.get(&tier).copied().unwrap_or_default().add(&st));
+        }
     }
 
-    /// True when [`AllTierStats::merge`] would report exact sums for every tier.
-    pub fn fits_merge(&self, other: &AllTierStats) -> bool {
-        other
-            .tiers
-            .iter()
-            .all(|(tier, right)| self.tiers.get(tier).is_none_or(|left| left.fits_add(right)))
+    pub fn populate_stats(&self, stats: &mut HashMap<String, TierStats>) {
+        for (tier, st) in &self.tiers {
+            stats.insert(
+                tier.clone(),
+                TierStats {
+                    total_size: st.total_size,
+                    num_versions: st.num_versions,
+                    num_objects: st.num_objects,
+                },
+            );
+        }
     }
 }
 
@@ -199,14 +183,6 @@ pub struct DataUsageInfo {
     pub objects_total_size: u64,
     /// Replication info across all buckets
     pub replication_info: HashMap<String, BucketTargetUsageInfo>,
-    /// Usage per storage class and remote tier across all buckets.
-    ///
-    /// Absent on snapshots written before per-tier accounting was published,
-    /// and on clusters with no remote tier configured: the scanner classifies
-    /// objects by tier (including `STANDARD`/`REDUCED_REDUNDANCY`) only once a
-    /// tier exists, so an absent value means "not accounted", never "zero".
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    pub tier_stats: Option<AllTierStats>,
 
     /// Total number of buckets in this cluster
     pub buckets_count: u64,
@@ -586,7 +562,7 @@ impl ReplicationAllStats {
 }
 
 /// Data usage cache entry
-#[derive(Clone, Debug, Default, Deserialize)]
+#[derive(Clone, Debug, Default, Serialize, Deserialize)]
 pub struct DataUsageEntry {
     pub children: DataUsageHashMap,
     // These fields do not include any children.
@@ -601,34 +577,6 @@ pub struct DataUsageEntry {
     /// Number of objects that failed to scan (e.g., IO errors)
     #[serde(default)]
     pub failed_objects: usize,
-    /// Per-tier usage contributed by this entry, present only once a scan
-    /// observed tier-classified objects.
-    #[serde(default)]
-    pub all_tier_stats: Option<AllTierStats>,
-}
-
-impl Serialize for DataUsageEntry {
-    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
-    where
-        S: serde::Serializer,
-    {
-        // Keep entries map-encoded so older readers can ignore fields appended
-        // by newer scanner versions during rolling upgrades. The derived
-        // (array) encoding made any appended field a decode error for them.
-        let mut state = serializer.serialize_map(Some(11))?;
-        state.serialize_entry("children", &self.children)?;
-        state.serialize_entry("size", &self.size)?;
-        state.serialize_entry("objects", &self.objects)?;
-        state.serialize_entry("versions", &self.versions)?;
-        state.serialize_entry("delete_markers", &self.delete_markers)?;
-        state.serialize_entry("obj_sizes", &self.obj_sizes)?;
-        state.serialize_entry("obj_versions", &self.obj_versions)?;
-        state.serialize_entry("replication_stats", &self.replication_stats)?;
-        state.serialize_entry("compacted", &self.compacted)?;
-        state.serialize_entry("failed_objects", &self.failed_objects)?;
-        state.serialize_entry("all_tier_stats", &self.all_tier_stats)?;
-        state.end()
-    }
 }
 
 impl DataUsageEntry {
@@ -687,22 +635,10 @@ impl DataUsageEntry {
             }
         }
 
-        if let Some(o_tiers) = other.all_tier_stats.as_ref().filter(|tiers| !tiers.is_empty()) {
-            self.all_tier_stats.get_or_insert_with(AllTierStats::new).merge(o_tiers);
-        }
-
         self.obj_sizes.merge_from(&other.obj_sizes);
         self.obj_versions.merge_from(&other.obj_versions);
     }
 
-    /// Folds a scan summary's per-tier map into this entry.
-    pub fn add_tier_sizes(&mut self, tiers: &HashMap<String, TierStats>) {
-        if tiers.values().all(TierStats::is_empty) {
-            return;
-        }
-        self.all_tier_stats.get_or_insert_with(AllTierStats::new).add_sizes(tiers);
-    }
-
     pub fn checked_merge(&mut self, other: &DataUsageEntry) -> bool {
         let scalar_counts_fit = self.objects.checked_add(other.objects).is_some()
             && self.versions.checked_add(other.versions).is_some()
@@ -762,12 +698,7 @@ impl DataUsageEntry {
             }
         };
 
-        let tier_stats_fit = match (&self.all_tier_stats, &other.all_tier_stats) {
-            (_, None) | (None, Some(_)) => true,
-            (Some(left), Some(right)) => left.fits_merge(right),
-        };
-
-        if !scalar_counts_fit || !histograms_fit || !replication_fits || !tier_stats_fit {
+        if !scalar_counts_fit || !histograms_fit || !replication_fits {
             return false;
         }
         self.merge(other);
@@ -1107,7 +1038,6 @@ impl DataUsageCache {
             versions_total_count: flat.versions as u64,
             delete_markers_total_count: flat.delete_markers as u64,
             objects_total_size: flat.size as u64,
-            tier_stats: flat.all_tier_stats.filter(|tiers| !tiers.is_empty()),
             buckets_count: u64::try_from(buckets.len()).unwrap_or(u64::MAX),
             buckets_usage,
             usage_snapshot_complete: self.info.snapshot_complete,
@@ -1595,172 +1525,6 @@ mod tests {
         buckets_count: u64,
     }
 
-    fn tier_entry(tier: &str, stats: TierStats) -> DataUsageEntry {
-        let mut entry = DataUsageEntry::default();
-        entry.add_tier_sizes(&HashMap::from([(tier.to_string(), stats)]));
-        entry
-    }
-
-    #[test]
-    fn tier_stats_survive_entry_merge() {
-        let mut left = tier_entry(
-            "WARM",
-            TierStats {
-                total_size: 10,
-                num_versions: 2,
-                num_objects: 1,
-            },
-        );
-        let mut right = tier_entry(
-            "WARM",
-            TierStats {
-                total_size: 5,
-                num_versions: 1,
-                num_objects: 1,
-            },
-        );
-        right.add_tier_sizes(&HashMap::from([(
-            "COLD".to_string(),
-            TierStats {
-                total_size: 7,
-                num_versions: 1,
-                num_objects: 0,
-            },
-        )]));
-
-        assert!(left.checked_merge(&right), "merging exact tier totals must be accepted");
-
-        let tiers = &left.all_tier_stats.expect("merged entry keeps tier stats").tiers;
-        assert_eq!(
-            tiers.get("WARM"),
-            Some(&TierStats {
-                total_size: 15,
-                num_versions: 3,
-                num_objects: 2,
-            })
-        );
-        assert_eq!(
-            tiers.get("COLD"),
-            Some(&TierStats {
-                total_size: 7,
-                num_versions: 1,
-                num_objects: 0,
-            })
-        );
-    }
-
-    #[test]
-    fn tier_stats_merge_into_an_untiered_entry() {
-        let mut left = DataUsageEntry::default();
-        let right = tier_entry(
-            "WARM",
-            TierStats {
-                total_size: 10,
-                num_versions: 1,
-                num_objects: 1,
-            },
-        );
-
-        assert!(left.checked_merge(&right));
-
-        assert_eq!(
-            left.all_tier_stats.expect("tier stats adopted from the merged entry").tiers["WARM"],
-            TierStats {
-                total_size: 10,
-                num_versions: 1,
-                num_objects: 1,
-            }
-        );
-    }
-
-    #[test]
-    fn checked_merge_rejects_overflowing_tier_totals() {
-        let mut left = tier_entry(
-            "WARM",
-            TierStats {
-                total_size: u64::MAX,
-                num_versions: 1,
-                num_objects: 1,
-            },
-        );
-        let right = tier_entry(
-            "WARM",
-            TierStats {
-                total_size: 1,
-                num_versions: 1,
-                num_objects: 1,
-            },
-        );
-
-        assert!(!left.checked_merge(&right), "saturating tier totals must not be published");
-        assert_eq!(left.all_tier_stats.expect("left is untouched").tiers["WARM"].total_size, u64::MAX);
-    }
-
-    /// Entry shape released before per-tier accounting, using the derived
-    /// (array) encoding those writers produced.
-    #[derive(Serialize, Deserialize)]
-    struct LegacyEntry {
-        children: DataUsageHashMap,
-        size: usize,
-        objects: usize,
-        versions: usize,
-        delete_markers: usize,
-        obj_sizes: SizeHistogram,
-        obj_versions: VersionsHistogram,
-        replication_stats: Option<ReplicationAllStats>,
-        compacted: bool,
-        #[serde(default)]
-        failed_objects: usize,
-    }
-
-    #[test]
-    fn entries_are_map_encoded_so_appended_fields_stay_readable() {
-        // A derived (array) encoding turns every appended field into a decode
-        // error for readers built before it existed, which would cost a mixed
-        // -version cluster its whole scan cache. Entries must stay map-encoded.
-        let current = tier_entry(
-            "WARM",
-            TierStats {
-                total_size: 3,
-                num_versions: 1,
-                num_objects: 1,
-            },
-        );
-        let mut encoded = Vec::new();
-        current
-            .serialize(&mut rmp_serde::Serializer::new(&mut encoded))
-            .expect("encode current entry");
-
-        let legacy: LegacyEntry = rmp_serde::from_slice(&encoded).expect("legacy reader should ignore the appended field");
-        assert_eq!(legacy.objects, 0);
-    }
-
-    #[test]
-    fn legacy_array_encoded_entries_still_load() {
-        let legacy = LegacyEntry {
-            children: DataUsageHashMap::default(),
-            size: 12,
-            objects: 3,
-            versions: 4,
-            delete_markers: 1,
-            obj_sizes: SizeHistogram::default(),
-            obj_versions: VersionsHistogram::default(),
-            replication_stats: None,
-            compacted: false,
-            failed_objects: 2,
-        };
-        let mut encoded = Vec::new();
-        legacy
-            .serialize(&mut rmp_serde::Serializer::new(&mut encoded))
-            .expect("encode legacy entry");
-
-        let decoded: DataUsageEntry = rmp_serde::from_slice(&encoded).expect("current reader should default the missing field");
-
-        assert_eq!(decoded.size, 12);
-        assert_eq!(decoded.failed_objects, 2);
-        assert!(decoded.all_tier_stats.is_none());
-    }
-
     #[test]
     fn hash_path_uses_portable_slash_semantics() {
         for (input, expected) in [
@@ -2137,44 +1901,6 @@ mod tests {
         assert_eq!(info.buckets_count, 2);
         assert!(info.buckets_usage.is_empty());
         assert_eq!(info.objects_total_count, 3);
-        assert!(info.tier_stats.is_none());
-    }
-
-    #[test]
-    fn test_dui_reports_tier_usage_from_the_flattened_tree() {
-        let root_hash = hash_path("root");
-        let bucket_hash = hash_path("bucket-a");
-        let mut cache = DataUsageCache {
-            info: DataUsageCacheInfo {
-                name: "root".to_string(),
-                ..Default::default()
-            },
-            ..Default::default()
-        };
-        cache.replace_hashed(&root_hash, &None, &DataUsageEntry::default());
-        cache.replace_hashed(
-            &bucket_hash,
-            &Some(root_hash),
-            &tier_entry(
-                "WARM",
-                TierStats {
-                    total_size: 40,
-                    num_versions: 2,
-                    num_objects: 2,
-                },
-            ),
-        );
-
-        let info = cache.dui("root", &["bucket-a".to_string()]);
-
-        assert_eq!(
-            info.tier_stats.expect("child tier usage should roll up to the root").tiers["WARM"],
-            TierStats {
-                total_size: 40,
-                num_versions: 2,
-                num_objects: 2,
-            }
-        );
     }
 
     #[test]

crates/e2e_test/Cargo.toml

@@ -25,58 +25,10 @@ workspace = true
 
 [features]
 default = []
-hotpath = [
-    "hotpath/hotpath",
-    "hotpath/tokio",
-    "hotpath/futures",
-    "hotpath/reqwest-0-13",
-    "rustfs-config/hotpath",
-    "rustfs-credentials/hotpath",
-    "rustfs-data-usage/hotpath",
-    "rustfs-ecstore/hotpath",
-    "rustfs-filemeta/hotpath",
-    "rustfs-lock/hotpath",
-    "rustfs-madmin/hotpath",
-    "rustfs-protos/hotpath",
-    "rustfs-rio/hotpath",
-    "rustfs-signer/hotpath",
-    "rustfs-utils/hotpath",
-]
-hotpath-alloc = [
-    "hotpath",
-    "hotpath/hotpath-alloc",
-    "rustfs-config/hotpath-alloc",
-    "rustfs-credentials/hotpath-alloc",
-    "rustfs-data-usage/hotpath-alloc",
-    "rustfs-ecstore/hotpath-alloc",
-    "rustfs-filemeta/hotpath-alloc",
-    "rustfs-lock/hotpath-alloc",
-    "rustfs-madmin/hotpath-alloc",
-    "rustfs-protos/hotpath-alloc",
-    "rustfs-rio/hotpath-alloc",
-    "rustfs-signer/hotpath-alloc",
-    "rustfs-utils/hotpath-alloc",
-]
-hotpath-cpu = [
-    "hotpath",
-    "hotpath/hotpath-cpu",
-    "rustfs-config/hotpath-cpu",
-    "rustfs-credentials/hotpath-cpu",
-    "rustfs-data-usage/hotpath-cpu",
-    "rustfs-ecstore/hotpath-cpu",
-    "rustfs-filemeta/hotpath-cpu",
-    "rustfs-lock/hotpath-cpu",
-    "rustfs-madmin/hotpath-cpu",
-    "rustfs-protos/hotpath-cpu",
-    "rustfs-rio/hotpath-cpu",
-    "rustfs-signer/hotpath-cpu",
-    "rustfs-utils/hotpath-cpu",
-]
 ftps = []
 sftp = []
 
 [dependencies]
-hotpath.workspace = true
 rustfs-config = { workspace = true, features = ["constants"] }
 rustfs-credentials.workspace = true
 rustfs-ecstore.workspace = true

crates/e2e_test/src/admin_iam_crud_test.rs

@@ -26,16 +26,75 @@
 //! Later batches tracked on backlog#1154: config get/set, info, pools status,
 //! group lifecycle, import/export IAM.
 
-use crate::common::{RustFSTestEnvironment, admin_ok, admin_request, init_logging};
+use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
 use aws_sdk_s3::config::{Credentials, Region};
 use aws_sdk_s3::primitives::ByteStream;
 use aws_sdk_s3::{Client, Config};
+use http::header::{CONTENT_TYPE, HOST};
 use reqwest::StatusCode;
+use rustfs_signer::constants::UNSIGNED_PAYLOAD;
+use rustfs_signer::sign_v4;
+use s3s::Body;
 use serial_test::serial;
 use std::error::Error;
 use tokio::time::{Duration, sleep};
 
 type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
+type BoxError = Box<dyn Error + Send + Sync>;
+
+/// Signs and sends an admin HTTP request with the given credential, returning
+/// status and body. Native `/rustfs/admin/v3` requests and responses are plain
+/// JSON (the MinIO-compat encryption applies only to `/minio/admin/v3` paths).
+async fn admin_request(
+    base_url: &str,
+    method: http::Method,
+    path_and_query: &str,
+    body: Option<String>,
+    access_key: &str,
+    secret_key: &str,
+) -> Result<(StatusCode, String), BoxError> {
+    let url = format!("{base_url}{path_and_query}");
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
+    let mut builder = http::Request::builder()
+        .method(method.clone())
+        .uri(uri)
+        .header(HOST, authority)
+        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
+    if body.is_some() {
+        builder = builder.header(CONTENT_TYPE, "application/json");
+    }
+
+    let content_len = body.as_ref().map(|b| b.len() as i64).unwrap_or_default();
+    let signed = sign_v4(builder.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1");
+
+    let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
+    let mut request = local_http_client().request(reqwest_method, &url);
+    for (name, value) in signed.headers() {
+        request = request.header(name, value);
+    }
+    if let Some(body) = body {
+        request = request.body(body);
+    }
+    let response = request.send().await?;
+    let status = response.status();
+    let text = response.text().await.unwrap_or_default();
+    Ok((status, text))
+}
+
+/// Root-credential admin request that must succeed; returns the response body.
+async fn admin_ok(
+    env: &RustFSTestEnvironment,
+    method: http::Method,
+    path_and_query: &str,
+    body: Option<String>,
+) -> Result<String, BoxError> {
+    let (status, text) = admin_request(&env.url, method.clone(), path_and_query, body, &env.access_key, &env.secret_key).await?;
+    if !status.is_success() {
+        return Err(format!("{method} {path_and_query} failed: {status} {text}").into());
+    }
+    Ok(text)
+}
 
 fn build_s3_client(url: &str, access_key: &str, secret_key: &str) -> Client {
     let config = Config::builder()

crates/e2e_test/src/bucket_stats_regression_test.rs

@@ -1,259 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression tests for bucket statistics and data usage accuracy.
-//!
-//! Covers the recurring pattern where bucket statistics (object count, size)
-//! show stale/incorrect values, remain at 0, or oscillate between complete,
-//! partial, and zero. This has regressed 10+ times.
-//!
-//! ## Regression Issues
-//!
-//! - rustfs#5615: bucket statistics remain unchanged after data expiration
-//! - rustfs#5008: Admin usage reports only one pool
-//! - rustfs#5116: Admin usage reports stale 0/0 for non-empty bucket after upgrade
-//! - rustfs#5055: console object count and size still loading
-//! - rustfs#5010: Storage usage info changed abnormally
-//! - rustfs#3662: Incorrect bucket, object count and size
-//! - rustfs#3898: DataUsageInfo undercounts versioned bucket versions
-//! - rustfs#1012: Object count in the console doesn't change
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestEnvironment, awscurl_get, init_logging};
-    use aws_sdk_s3::primitives::ByteStream;
-    use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
-    use rustfs_data_usage::DataUsageInfo;
-    use serial_test::serial;
-    use std::error::Error;
-    use tokio::time::{Duration, sleep};
-    use tracing::info;
-
-    type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-    async fn get_data_usage(env: &RustFSTestEnvironment) -> Result<DataUsageInfo, Box<dyn Error + Send + Sync>> {
-        let url = format!("{}/rustfs/admin/v3/datausageinfo", env.url);
-        let resp = awscurl_get(&url, &env.access_key, &env.secret_key).await?;
-        Ok(serde_json::from_str(&resp)?)
-    }
-
-    /// RT-09: Verify bucket object count updates after PUT.
-    ///
-    /// Regression pattern: bucket stats remain at 0 after objects are uploaded
-    /// (rustfs#5055, rustfs#1012).
-    ///
-    /// Steps:
-    /// 1. Create a bucket
-    /// 2. Upload 10 objects
-    /// 3. Query admin data usage API
-    /// 4. Verify object count > 0
-    #[tokio::test]
-    #[serial]
-    async fn test_bucket_object_count_updates_after_put() -> TestResult {
-        init_logging();
-        info!("RT-09: bucket object count updates after PUT");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt09-stats-put";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Upload 10 objects
-        for i in 0..10 {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(format!("stat-obj-{i:04}.txt"))
-                .body(ByteStream::from_static(b"statistical data"))
-                .send()
-                .await
-                .expect("put object");
-        }
-
-        // Wait for scanner to process (up to 90 seconds)
-        let mut found_nonzero = false;
-        for attempt in 0..18 {
-            sleep(Duration::from_secs(5)).await;
-
-            if let Ok(usage) = get_data_usage(&env).await
-                && let Some(bucket_usage) = usage.buckets_usage.get(bucket)
-            {
-                info!("  attempt {attempt}: objectsCount = {}", bucket_usage.objects_count);
-                if bucket_usage.objects_count >= 10 {
-                    found_nonzero = true;
-                    break;
-                }
-            }
-        }
-
-        assert!(
-            found_nonzero,
-            "RT-09 FAIL: bucket object count did not update after PUT 10 objects (regression: stats stuck at 0)"
-        );
-
-        info!("RT-09 PASS: bucket object count updates after PUT");
-        Ok(())
-    }
-
-    /// RT-09b: Verify bucket stats update after DELETE.
-    ///
-    /// Regression pattern: stats remain unchanged after objects are deleted
-    /// (rustfs#5615).
-    #[tokio::test]
-    #[serial]
-    async fn test_bucket_object_count_updates_after_delete() -> TestResult {
-        init_logging();
-        info!("RT-09b: bucket object count updates after DELETE");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt09b-stats-delete";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Upload 5 objects
-        for i in 0..5 {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(format!("del-stat-{i}.txt"))
-                .body(ByteStream::from_static(b"data"))
-                .send()
-                .await
-                .expect("put object");
-        }
-
-        // Delete all objects
-        for i in 0..5 {
-            client
-                .delete_object()
-                .bucket(bucket)
-                .key(format!("del-stat-{i}.txt"))
-                .send()
-                .await
-                .expect("delete object");
-        }
-
-        // Wait for scanner to update stats (up to 90 seconds)
-        let mut found_zero = false;
-        for attempt in 0..18 {
-            sleep(Duration::from_secs(5)).await;
-
-            if let Ok(usage) = get_data_usage(&env).await
-                && let Some(bucket_usage) = usage.buckets_usage.get(bucket)
-            {
-                info!("  attempt {attempt}: objectsCount = {}", bucket_usage.objects_count);
-                if bucket_usage.objects_count == 0 {
-                    found_zero = true;
-                    break;
-                }
-            }
-        }
-
-        assert!(
-            found_zero,
-            "RT-09b FAIL: bucket object count did not update to 0 after deleting all objects (regression rustfs#5615)"
-        );
-
-        info!("RT-09b PASS: bucket object count updates to 0 after DELETE");
-        Ok(())
-    }
-
-    /// RT-09c: Verify versioned bucket stats count all versions.
-    ///
-    /// Regression pattern: DataUsageInfo undercounts versioned bucket versions
-    /// and delete markers (rustfs#3898).
-    #[tokio::test]
-    #[serial]
-    async fn test_versioned_bucket_stats_count_all_versions() -> TestResult {
-        init_logging();
-        info!("RT-09c: versioned bucket stats count all versions");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt09c-versioned-stats";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        client
-            .put_bucket_versioning()
-            .bucket(bucket)
-            .versioning_configuration(
-                VersioningConfiguration::builder()
-                    .status(BucketVersioningStatus::Enabled)
-                    .build(),
-            )
-            .send()
-            .await
-            .expect("enable versioning");
-
-        // Create 3 versions of the same object
-        for i in 0..3 {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key("multi-version.txt")
-                .body(ByteStream::from(format!("version-{i}").into_bytes()))
-                .send()
-                .await
-                .expect("put version");
-        }
-
-        // Create a delete marker
-        client
-            .delete_object()
-            .bucket(bucket)
-            .key("multi-version.txt")
-            .send()
-            .await
-            .expect("create delete marker");
-
-        // Verify versions via API (immediate, no scanner wait)
-        let versions = client
-            .list_object_versions()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list versions");
-
-        assert_eq!(
-            versions.versions().len(),
-            3,
-            "RT-09c FAIL: expected 3 versions, found {}",
-            versions.versions().len()
-        );
-        assert_eq!(
-            versions.delete_markers().len(),
-            1,
-            "RT-09c FAIL: expected 1 delete marker, found {}",
-            versions.delete_markers().len()
-        );
-
-        info!("RT-09c PASS: versioned bucket correctly tracks all versions and delete markers");
-        Ok(())
-    }
-}

crates/e2e_test/src/common.rs

@@ -24,12 +24,7 @@
 use aws_sdk_s3::config::{Credentials, Region};
 use aws_sdk_s3::{Client, Config};
 use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
-use http::header::{CONTENT_TYPE, HOST};
 use reqwest::Client as HttpClient;
-use reqwest::StatusCode;
-use rustfs_signer::constants::UNSIGNED_PAYLOAD;
-use rustfs_signer::sign_v4;
-use s3s::Body;
 use std::ffi::OsStr;
 use std::fs as stdfs;
 use std::path::{Path, PathBuf};
@@ -50,21 +45,6 @@ pub const ENV_RUSTFS_BUILD_FEATURES: &str = "RUSTFS_BUILD_FEATURES";
 pub const TEST_BUCKET: &str = "e2e-test-bucket";
 const RUSTFS_FULL_FEATURE: &str = "full";
 
-fn capture_log_path(log_dir: &Path, temp_dir: &str) -> Option<PathBuf> {
-    let temp_name = Path::new(temp_dir).file_name()?.to_string_lossy();
-    Some(log_dir.join(format!("{temp_name}.log")))
-}
-
-fn configured_capture_log_path(temp_dir: &str) -> Option<String> {
-    let log_dir = std::env::var_os("RUSTFS_E2E_LOG_DIR")?;
-    if stdfs::create_dir_all(&log_dir).is_err() {
-        warn!(?log_dir, "failed to create configured E2E server log directory");
-        return None;
-    }
-
-    capture_log_path(Path::new(&log_dir), temp_dir).map(|path| path.to_string_lossy().into_owned())
-}
-
 fn build_test_s3_config(endpoint_url: &str, access_key: &str, secret_key: &str, provider_name: &'static str) -> Config {
     let credentials = Credentials::new(access_key, secret_key, None, None, provider_name);
     let mut config = Config::builder()
@@ -95,58 +75,6 @@ pub fn local_http_client() -> HttpClient {
         .expect("failed to build local reqwest client")
 }
 
-/// Signs and sends an admin HTTP request with the given credentials.
-pub(crate) async fn admin_request(
-    base_url: &str,
-    method: http::Method,
-    path_and_query: &str,
-    body: Option<String>,
-    access_key: &str,
-    secret_key: &str,
-) -> Result<(StatusCode, String), Box<dyn std::error::Error + Send + Sync>> {
-    let url = format!("{base_url}{path_and_query}");
-    let uri = url.parse::<http::Uri>()?;
-    let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
-    let mut request = http::Request::builder()
-        .method(method.clone())
-        .uri(uri)
-        .header(HOST, authority)
-        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
-    if body.is_some() {
-        request = request.header(CONTENT_TYPE, "application/json");
-    }
-
-    let content_length = i64::try_from(body.as_ref().map_or(0, String::len)).map_err(|_| "admin request body is too large")?;
-    let signed = sign_v4(request.body(Body::empty())?, content_length, access_key, secret_key, "", "us-east-1");
-
-    let mut request = local_http_client().request(method, &url);
-    for (name, value) in signed.headers() {
-        request = request.header(name, value);
-    }
-    if let Some(body) = body {
-        request = request.body(body);
-    }
-    let response = request.send().await?;
-    let status = response.status();
-    let body = response.text().await?;
-    Ok((status, body))
-}
-
-/// Sends a root-credential admin request and returns its successful response body.
-pub(crate) async fn admin_ok(
-    env: &RustFSTestEnvironment,
-    method: http::Method,
-    path_and_query: &str,
-    body: Option<String>,
-) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
-    let (status, response_body) =
-        admin_request(&env.url, method.clone(), path_and_query, body, &env.access_key, &env.secret_key).await?;
-    if !status.is_success() {
-        return Err(format!("{method} {path_and_query} failed: {status} {response_body}").into());
-    }
-    Ok(response_body)
-}
-
 /// Resolve the RustFS binary relative to the workspace.
 pub fn rustfs_binary_path() -> PathBuf {
     rustfs_binary_path_with_features(requested_rustfs_build_features().as_deref())
@@ -376,7 +304,6 @@ impl RustFSTestEnvironment {
     pub async fn new() -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
         let temp_dir = format!("/tmp/rustfs_e2e_test_{}", Uuid::new_v4());
         fs::create_dir_all(&temp_dir).await?;
-        let capture_log_path = configured_capture_log_path(&temp_dir);
 
         // Use a unique port for each test environment
         let port = Self::find_available_port().await?;
@@ -390,7 +317,7 @@ impl RustFSTestEnvironment {
             access_key: DEFAULT_ACCESS_KEY.to_string(),
             secret_key: DEFAULT_SECRET_KEY.to_string(),
             process: None,
-            capture_log_path,
+            capture_log_path: None,
         })
     }
 
@@ -398,7 +325,6 @@ impl RustFSTestEnvironment {
     pub async fn with_address(address: &str) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
         let temp_dir = format!("/tmp/rustfs_e2e_test_{}", Uuid::new_v4());
         fs::create_dir_all(&temp_dir).await?;
-        let capture_log_path = configured_capture_log_path(&temp_dir);
 
         let url = format!("http://{address}");
 
@@ -409,7 +335,7 @@ impl RustFSTestEnvironment {
             access_key: DEFAULT_ACCESS_KEY.to_string(),
             secret_key: DEFAULT_SECRET_KEY.to_string(),
             process: None,
-            capture_log_path,
+            capture_log_path: None,
         })
     }
 
@@ -1409,14 +1335,6 @@ mod tests {
         assert_eq!(normalize_rustfs_build_features(" , "), None);
     }
 
-    #[test]
-    fn capture_log_path_uses_temp_directory_basename() {
-        assert_eq!(
-            capture_log_path(Path::new("/tmp/e2e-logs"), "/tmp/rustfs_e2e_test_abc"),
-            Some(PathBuf::from("/tmp/e2e-logs/rustfs_e2e_test_abc.log"))
-        );
-    }
-
     #[test]
     fn full_feature_enables_any_required_feature() {
         assert!(rustfs_build_feature_enabled(Some("full"), "sftp"));

crates/e2e_test/src/copy_object_metadata_test.rs

@@ -117,14 +117,9 @@ mod tests {
             .key("assets/explicit-copy.js")
             .copy_source(format!("{bucket}/{key}"))
             .metadata_directive(MetadataDirective::Copy)
-            .customize()
-            .mutate_request(|request| {
-                request.headers_mut().insert("content-type", "application/octet-stream");
-                request.headers_mut().insert("x-amz-meta-request-only", "ignored");
-            })
             .send()
             .await
-            .expect("explicit COPY directive with request metadata failed");
+            .expect("explicit COPY directive failed");
         let explicit_copy_head = client
             .head_object()
             .bucket(bucket)
@@ -133,18 +128,6 @@ mod tests {
             .await
             .expect("HEAD failed after explicit COPY");
         assert_eq!(explicit_copy_head.cache_control(), Some("max-age=60"));
-        assert_eq!(explicit_copy_head.content_type(), Some("text/javascript; charset=utf-8"));
-        assert_eq!(
-            explicit_copy_head.metadata().and_then(|metadata| metadata.get("mtime")),
-            Some(&"1777992333".to_string())
-        );
-        assert_eq!(
-            explicit_copy_head
-                .metadata()
-                .and_then(|metadata| metadata.get("request-only")),
-            None,
-            "COPY must ignore request metadata"
-        );
         assert_eq!(
             explicit_copy_head.website_redirect_location(),
             None,
@@ -588,6 +571,20 @@ mod tests {
             Some("InvalidArgument")
         );
 
+        let ignored_replacement = client
+            .copy_object()
+            .bucket(bucket)
+            .key(key)
+            .copy_source(format!("{bucket}/{key}"))
+            .content_type("application/ignored")
+            .send()
+            .await
+            .expect_err("Replacement fields without REPLACE should be rejected");
+        assert_eq!(
+            ignored_replacement.as_service_error().and_then(|error| error.code()),
+            Some("InvalidRequest")
+        );
+
         let unchanged = client
             .get_object()
             .bucket(bucket)

crates/e2e_test/src/delete_marker_migration_semantics_test.rs

@@ -56,21 +56,6 @@ mod tests {
         );
     }
 
-    async fn assert_current_list_hides_delete_marker(client: &Client, bucket: &str, key: &str) {
-        let listed = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .prefix(key)
-            .send()
-            .await
-            .expect("list current objects after delete marker");
-
-        assert!(
-            listed.contents().iter().all(|object| object.key() != Some(key)),
-            "ListObjectsV2 must hide an object whose latest version is a delete marker"
-        );
-    }
-
     #[tokio::test]
     #[serial]
     async fn test_versioning_only_delete_marker_has_minio_compatible_visibility_for_migration_proof() {
@@ -109,7 +94,6 @@ mod tests {
         assert_eq!(markers[0].version_id(), Some(delete_marker_version_id));
         assert_eq!(markers[0].is_latest(), Some(true));
         assert_current_get_is_delete_marker_not_found(&client, bucket, key).await;
-        assert_current_list_hides_delete_marker(&client, bucket, key).await;
     }
 
     #[tokio::test]
@@ -134,17 +118,6 @@ mod tests {
             .await
             .expect("put historical version");
         let data_version_id = put.version_id().expect("put should return data version id");
-        let listed_before_delete = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .prefix(key)
-            .send()
-            .await
-            .expect("list current object before creating delete marker");
-        assert!(
-            listed_before_delete.contents().iter().any(|object| object.key() == Some(key)),
-            "ListObjectsV2 must include the current object before it is deleted"
-        );
 
         let delete_marker = client
             .delete_object()
@@ -172,7 +145,6 @@ mod tests {
         assert_eq!(markers[0].version_id(), Some(delete_marker_version_id));
         assert_eq!(markers[0].is_latest(), Some(true));
         assert_current_get_is_delete_marker_not_found(&client, bucket, key).await;
-        assert_current_list_hides_delete_marker(&client, bucket, key).await;
 
         let historical = client
             .get_object()

crates/e2e_test/src/delete_regression_test.rs

@@ -1,445 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression tests for object delete operations.
-//!
-//! Covers the recurring pattern where DELETE succeeds at the API level but the
-//! object remains visible in LIST, or deleted objects reappear after restart,
-//! or versioned delete operations fail with FileAccessDenied.
-//! This has regressed 15+ times across the entire release history.
-//!
-//! ## Regression Issues
-//!
-//! - rustfs#5375: delete object in a bucket list api also exist this object
-//! - rustfs#5349: The deleted bucket was rebuilt after some time
-//! - rustfs#5339: data not delete in Object Lock bucket
-//! - rustfs#5029: Node Does Not Remove Files After Reconnect to Cluster
-//! - rustfs#4978: DELETE fails with InternalError/FileAccessDenied on beta 10
-//! - rustfs#760: Cannot delete a versioned bucket
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestEnvironment, init_logging};
-    use aws_sdk_s3::primitives::ByteStream;
-    use aws_sdk_s3::types::{BucketVersioningStatus, Delete, ObjectIdentifier, VersioningConfiguration};
-    use serial_test::serial;
-    use std::error::Error;
-    use tracing::info;
-
-    type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-    /// RT-05: Verify DELETE → LIST → HEAD consistency.
-    ///
-    /// Regression pattern: DELETE returns 200 but the object remains in LIST.
-    /// Covers rustfs#5375.
-    ///
-    /// Steps:
-    /// 1. Create a bucket and upload an object
-    /// 2. Verify the object is in LIST
-    /// 3. DELETE the object
-    /// 4. Verify the object is NOT in LIST
-    /// 5. Verify HEAD returns 404
-    #[tokio::test]
-    #[serial]
-    async fn test_delete_removes_object_from_list() -> TestResult {
-        init_logging();
-        info!("RT-05: delete removes object from list");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt05-delete-consistency";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Upload an object
-        client
-            .put_object()
-            .bucket(bucket)
-            .key("to-delete.txt")
-            .body(ByteStream::from_static(b"will be deleted"))
-            .send()
-            .await
-            .expect("put object");
-
-        // Verify it appears in LIST
-        let list = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list objects before delete");
-
-        assert!(
-            list.contents()
-                .iter()
-                .map(|o| o.key().unwrap_or(""))
-                .any(|key| key == "to-delete.txt"),
-            "RT-05 FAIL: object not in LIST before delete"
-        );
-
-        // DELETE
-        client
-            .delete_object()
-            .bucket(bucket)
-            .key("to-delete.txt")
-            .send()
-            .await
-            .expect("delete object");
-
-        // Verify NOT in LIST
-        let list = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list objects after delete");
-
-        assert!(
-            !list
-                .contents()
-                .iter()
-                .map(|o| o.key().unwrap_or(""))
-                .any(|key| key == "to-delete.txt"),
-            "RT-05 FAIL: deleted object still in LIST (regression rustfs#5375)"
-        );
-
-        // Verify HEAD returns 404
-        let head = client.head_object().bucket(bucket).key("to-delete.txt").send().await;
-
-        assert!(head.is_err(), "RT-05 FAIL: HEAD on deleted object should return error, got success");
-
-        info!("RT-05 PASS: delete correctly removes object from LIST and HEAD");
-        Ok(())
-    }
-
-    /// RT-05c: Verify batch delete (DeleteObjects) consistency.
-    ///
-    /// Regression pattern: batch delete returns success but some objects
-    /// remain in LIST.
-    #[tokio::test]
-    #[serial]
-    async fn test_batch_delete_removes_all_objects() -> TestResult {
-        init_logging();
-        info!("RT-05c: batch delete removes all objects");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt05c-batch-delete";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Upload multiple objects
-        let keys: Vec<String> = (0..5).map(|i| format!("batch-{i:04}.txt")).collect();
-        for key in &keys {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(key)
-                .body(ByteStream::from_static(b"batch-delete-me"))
-                .send()
-                .await
-                .expect("put object");
-        }
-
-        // Verify all in LIST
-        let list = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list before batch delete");
-
-        assert_eq!(
-            list.contents().len(),
-            5,
-            "RT-05c FAIL: expected 5 objects before batch delete, found {}",
-            list.contents().len()
-        );
-
-        // Batch delete
-        let objects: Vec<ObjectIdentifier> = keys
-            .iter()
-            .map(|k| ObjectIdentifier::builder().key(k).build().expect("build object id"))
-            .collect();
-
-        client
-            .delete_objects()
-            .bucket(bucket)
-            .delete(Delete::builder().set_objects(Some(objects)).build().expect("build delete"))
-            .send()
-            .await
-            .expect("batch delete");
-
-        // Verify all removed
-        let list = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list after batch delete");
-
-        assert!(
-            list.contents().is_empty(),
-            "RT-05c FAIL: {} objects remain after batch delete (regression: delete objects not fully applied)",
-            list.contents().len()
-        );
-
-        info!("RT-05c PASS: batch delete removes all objects");
-        Ok(())
-    }
-
-    /// RT-05d: Verify versioned delete → permanent delete → object gone.
-    ///
-    /// Covers the pattern where permanent deletion of a specific version
-    /// fails with FileAccessDenied (rustfs#4978).
-    #[tokio::test]
-    #[serial]
-    async fn test_versioned_permanent_delete() -> TestResult {
-        init_logging();
-        info!("RT-05d: versioned permanent delete");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt05d-permanent-delete";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        client
-            .put_bucket_versioning()
-            .bucket(bucket)
-            .versioning_configuration(
-                VersioningConfiguration::builder()
-                    .status(BucketVersioningStatus::Enabled)
-                    .build(),
-            )
-            .send()
-            .await
-            .expect("enable versioning");
-
-        // Upload a single object (single version)
-        let put_resp = client
-            .put_object()
-            .bucket(bucket)
-            .key("single-version.txt")
-            .body(ByteStream::from_static(b"to-be-permanently-deleted"))
-            .send()
-            .await
-            .expect("put object");
-
-        let version_id = put_resp.version_id().expect("version ID should be present").to_string();
-
-        // Permanently delete the specific version (rustfs#4978: FileAccessDenied)
-        client
-            .delete_object()
-            .bucket(bucket)
-            .key("single-version.txt")
-            .version_id(&version_id)
-            .send()
-            .await
-            .expect("permanent delete should succeed (regression rustfs#4978)");
-
-        // Verify the object is completely gone
-        let versions = client
-            .list_object_versions()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list versions");
-
-        assert!(
-            versions.versions().is_empty(),
-            "RT-05d FAIL: version still present after permanent delete"
-        );
-
-        info!("RT-05d PASS: versioned permanent delete succeeds");
-        Ok(())
-    }
-
-    /// RT-05e: Verify delete marker + version history interaction.
-    ///
-    /// Covers the pattern where creating a delete marker and then listing
-    /// versions shows incorrect state (rustfs#760).
-    #[tokio::test]
-    #[serial]
-    async fn test_versioned_delete_marker_and_list_consistency() -> TestResult {
-        init_logging();
-        info!("RT-05e: versioned delete marker and list consistency");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt05e-dm-consistency";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        client
-            .put_bucket_versioning()
-            .bucket(bucket)
-            .versioning_configuration(
-                VersioningConfiguration::builder()
-                    .status(BucketVersioningStatus::Enabled)
-                    .build(),
-            )
-            .send()
-            .await
-            .expect("enable versioning");
-
-        // Create 3 versions
-        for i in 0..3 {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key("history.txt")
-                .body(ByteStream::from(format!("v{i}").into_bytes()))
-                .send()
-                .await
-                .expect("put version");
-        }
-
-        // Create a delete marker
-        let del = client
-            .delete_object()
-            .bucket(bucket)
-            .key("history.txt")
-            .send()
-            .await
-            .expect("delete (create marker)");
-
-        assert!(del.delete_marker().unwrap_or(false), "RT-05e FAIL: should have created a delete marker");
-
-        // ListObjectVersions should show 3 versions + 1 delete marker
-        let versions = client
-            .list_object_versions()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list versions");
-
-        assert_eq!(
-            versions.versions().len(),
-            3,
-            "RT-05e FAIL: expected 3 versions, found {}",
-            versions.versions().len()
-        );
-        assert_eq!(
-            versions.delete_markers().len(),
-            1,
-            "RT-05e FAIL: expected 1 delete marker, found {}",
-            versions.delete_markers().len()
-        );
-
-        // Now delete the delete marker (restore the object)
-        let dm_version = &versions.delete_markers()[0];
-        client
-            .delete_object()
-            .bucket(bucket)
-            .key("history.txt")
-            .version_id(dm_version.version_id().expect("dm version id"))
-            .send()
-            .await
-            .expect("delete delete-marker");
-
-        // HEAD should succeed now (latest version is accessible)
-        let head = client.head_object().bucket(bucket).key("history.txt").send().await;
-
-        assert!(head.is_ok(), "RT-05e FAIL: HEAD should succeed after removing delete marker");
-
-        info!("RT-05e PASS: versioned delete marker and list consistency");
-        Ok(())
-    }
-
-    /// RT-05f: Verify object deletion does not leave orphan data on disk.
-    ///
-    /// Regression pattern: after delete, the object data files remain on disk
-    /// (rustfs#5029: Node Does Not Remove Files After Reconnect).
-    #[tokio::test]
-    #[serial]
-    async fn test_delete_removes_object_head_returns_404() -> TestResult {
-        init_logging();
-        info!("RT-05f: delete → HEAD 404 consistency");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt05f-delete-head";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Upload, delete, verify HEAD returns 404
-        let keys = vec!["small.txt", "medium.txt", "with-slash.txt", "special+chars.txt"];
-
-        for key in &keys {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(*key)
-                .body(ByteStream::from_static(b"delete-me"))
-                .send()
-                .await
-                .expect("put object");
-        }
-
-        for key in &keys {
-            client
-                .delete_object()
-                .bucket(bucket)
-                .key(*key)
-                .send()
-                .await
-                .expect("delete object");
-        }
-
-        // All HEAD requests should return 404
-        for key in &keys {
-            let head = client.head_object().bucket(bucket).key(*key).send().await;
-
-            assert!(head.is_err(), "RT-05f FAIL: HEAD on deleted key '{key}' should return error");
-        }
-
-        // LIST should be empty
-        let list = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list after all deletes");
-
-        assert!(
-            list.contents().is_empty(),
-            "RT-05f FAIL: {} objects remain after deleting all",
-            list.contents().len()
-        );
-
-        info!("RT-05f PASS: all deleted objects return 404 on HEAD");
-        Ok(())
-    }
-}

crates/e2e_test/src/distributed_startup_regression_test.rs

@@ -1,202 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression tests for distributed cluster startup and quorum.
-//!
-//! Covers the recurring pattern where multi-node clusters fail to start due to
-//! lock quorum issues, DNS resolution delays, or erasure quorum deadlocks.
-//! This has regressed 7+ times.
-//!
-//! ## Regression Issues
-//!
-//! - rustfs#5416: RustFS cannot cold-start with 2/3 quorum when Pod DNS missing
-//! - rustfs#2945: Distributed mode fails on K8s: erasure quorum deadlock
-//! - rustfs#2794: distributed deployment does not become ready
-//! - rustfs#2601: fresh pod immediately enters FaultyDisk state
-//! - rustfs#4040: Distributed startup can fail lock quorum before AppContext initializes
-//! - rustfs#5655: fix(ecstore): bootstrap fresh four-node clusters reliably
-//! - rustfs#4954: S3/health endpoint unavailability after multi-pool scale-up
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestClusterEnvironment, init_logging};
-    use aws_sdk_s3::primitives::ByteStream;
-    use serial_test::serial;
-    use std::error::Error;
-    use tokio::time::{Duration, sleep};
-    use tracing::info;
-
-    type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-    /// RT-10: Verify 4-node cluster starts successfully and all nodes are ready.
-    ///
-    /// Regression pattern: distributed startup fails with quorum deadlock or
-    /// lock acquisition timeout (rustfs#2945, rustfs#5655).
-    ///
-    /// Steps:
-    /// 1. Create a 4-node cluster
-    /// 2. Start all nodes simultaneously
-    /// 3. Verify all nodes report healthy
-    /// 4. Verify S3 operations work through any node
-    #[tokio::test]
-    #[serial]
-    async fn test_four_node_cluster_startup_and_health() -> TestResult {
-        init_logging();
-        info!("RT-10: 4-node cluster startup and health");
-
-        let mut cluster = RustFSTestClusterEnvironment::new(4).await.expect("create 4-node cluster");
-
-        cluster.start().await.expect("start 4-node cluster");
-
-        // Create a bucket and verify it's accessible from all nodes
-        cluster
-            .create_test_bucket("rt10-startup")
-            .await
-            .expect("create bucket on cluster");
-
-        let clients = cluster.create_all_clients().expect("create per-node clients");
-
-        // Verify S3 operations work from every node
-        for (i, client) in clients.iter().enumerate() {
-            client
-                .put_object()
-                .bucket("rt10-startup")
-                .key(format!("from-node-{i}.txt"))
-                .body(ByteStream::from_static(b"hello from node"))
-                .send()
-                .await
-                .unwrap_or_else(|e| panic!("PUT from node {i} failed: {e}"));
-        }
-
-        // Verify all objects are visible from node 0
-        let list = clients[0]
-            .list_objects_v2()
-            .bucket("rt10-startup")
-            .send()
-            .await
-            .expect("list objects from node 0");
-
-        assert_eq!(
-            list.contents().len(),
-            4,
-            "RT-10 FAIL: expected 4 objects (one per node), found {}",
-            list.contents().len()
-        );
-
-        info!("RT-10 PASS: 4-node cluster starts and serves S3 from all nodes");
-        Ok(())
-    }
-
-    /// RT-10b: Verify cluster handles node restart gracefully.
-    ///
-    /// Regression pattern: after a node restart, it cannot rejoin the cluster
-    /// or enters a faulty state (rustfs#2601).
-    #[tokio::test]
-    #[serial]
-    async fn test_cluster_survives_node_restart() -> TestResult {
-        init_logging();
-        info!("RT-10b: cluster survives node restart");
-
-        let mut cluster = RustFSTestClusterEnvironment::new(4).await.expect("create 4-node cluster");
-
-        cluster.start().await.expect("start cluster");
-
-        cluster.create_test_bucket("rt10b-restart").await.expect("create bucket");
-
-        // Write data
-        let clients = cluster.create_all_clients()?;
-        clients[0]
-            .put_object()
-            .bucket("rt10b-restart")
-            .key("before-restart.txt")
-            .body(ByteStream::from_static(b"persistent data"))
-            .send()
-            .await
-            .expect("put object before restart");
-
-        // Stop node 3
-        cluster.stop_node(3).expect("stop node 3");
-        sleep(Duration::from_secs(2)).await;
-
-        // Verify cluster still works with 3/4 nodes (quorum)
-        clients[0]
-            .put_object()
-            .bucket("rt10b-restart")
-            .key("during-offline.txt")
-            .body(ByteStream::from_static(b"written while node 3 down"))
-            .send()
-            .await
-            .expect("PUT should succeed with 3/4 nodes");
-
-        // Restart node 3
-        cluster.start_node(3).await.expect("restart node 3");
-
-        // Wait for node to rejoin
-        sleep(Duration::from_secs(3)).await;
-
-        // Verify the restarted node can serve reads
-        let list = clients[3]
-            .list_objects_v2()
-            .bucket("rt10b-restart")
-            .send()
-            .await
-            .expect("list from restarted node");
-
-        assert!(
-            list.contents().len() >= 2,
-            "RT-10b FAIL: restarted node sees {} objects, expected >= 2",
-            list.contents().len()
-        );
-
-        info!("RT-10b PASS: cluster survives and recovers from node restart");
-        Ok(())
-    }
-
-    /// RT-10c: Verify bucket creation persists across all nodes.
-    ///
-    /// Regression pattern: bucket metadata is not replicated to all nodes,
-    /// causing NoSuchBucket errors on some nodes (rustfs#3191).
-    #[tokio::test]
-    #[serial]
-    async fn test_bucket_visible_from_all_nodes() -> TestResult {
-        init_logging();
-        info!("RT-10c: bucket visible from all nodes");
-
-        let mut cluster = RustFSTestClusterEnvironment::new(4).await.expect("create 4-node cluster");
-
-        cluster.start().await.expect("start cluster");
-
-        cluster
-            .create_test_bucket("rt10c-bucket-visibility")
-            .await
-            .expect("create bucket");
-
-        let clients = cluster.create_all_clients()?;
-
-        // Verify the bucket is visible from every node
-        for (i, client) in clients.iter().enumerate() {
-            let resp = client
-                .list_objects_v2()
-                .bucket("rt10c-bucket-visibility")
-                .send()
-                .await
-                .unwrap_or_else(|e| panic!("list from node {i} failed (NoSuchBucket?): {e}"));
-
-            assert!(resp.contents().is_empty(), "RT-10c: fresh bucket should be empty on node {i}");
-        }
-
-        info!("RT-10c PASS: bucket visible from all 4 nodes");
-        Ok(())
-    }
-}

crates/e2e_test/src/inline_fast_path_cluster_test.rs

@@ -1687,44 +1687,6 @@ async fn four_node_inline_storage_and_get_boundaries() -> TestResult {
     Ok(())
 }
 
-#[tokio::test]
-#[serial]
-async fn four_node_empty_legacy_volumes_start_as_fresh() -> TestResult {
-    init_logging();
-
-    let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
-    for data_dir in cluster.nodes.iter().flat_map(|node| &node.data_dirs) {
-        tokio::fs::create_dir_all(Path::new(data_dir).join(".minio.sys")).await?;
-    }
-
-    cluster.start().await?;
-
-    // Starting is not the assertion. The regression is that an empty legacy
-    // `.minio.sys` must be classified as a *fresh* volume, not as an existing
-    // MinIO deployment to adopt or migrate. Pin what that classification leaves
-    // on disk and in the namespace.
-    let buckets = cluster.create_s3_client(0)?.list_buckets().send().await?;
-    assert!(
-        buckets.buckets().is_empty(),
-        "a fresh classification must not adopt buckets from the pre-existing directories, got {:?}",
-        buckets.buckets().iter().filter_map(|b| b.name()).collect::<Vec<_>>()
-    );
-
-    for data_dir in cluster.nodes.iter().flat_map(|node| &node.data_dirs) {
-        assert!(
-            Path::new(data_dir).join(".rustfs.sys").join("format.json").is_file(),
-            "each drive must be formatted as fresh: {data_dir} has no .rustfs.sys/format.json"
-        );
-        let mut legacy = tokio::fs::read_dir(Path::new(data_dir).join(".minio.sys")).await?;
-        assert!(
-            legacy.next_entry().await?.is_none(),
-            "the empty legacy directory must be left untouched, not migrated into: {data_dir}"
-        );
-    }
-
-    Ok(())
-}
-
 #[tokio::test]
 #[serial]
 async fn four_node_inline_fallback_controls() -> TestResult {
@@ -2174,20 +2136,11 @@ async fn four_node_manual_transition_distributed_admission_conflict_reports_stat
     assert_eq!(terminal["bucket"].as_str(), Some(bucket.as_str()));
     assert_eq!(terminal["prefix"].as_str(), Some(prefix));
     assert_eq!(terminal["dry_run"].as_bool(), Some(false));
-    let terminal_status = terminal["status"].as_str();
-    assert!(
-        matches!(terminal_status, Some("partial" | "unknown")),
+    assert_eq!(
+        terminal["status"].as_str(),
+        Some("partial"),
         "small transition queue should surface terminal backpressure: {terminal}"
     );
-    if terminal_status == Some("unknown") {
-        let failure_reason = terminal["failure_reason"]
-            .as_str()
-            .ok_or_else(|| format!("unknown terminal status omitted failure_reason: {terminal}"))?;
-        assert!(
-            failure_reason.contains("worker result was not persisted before the transition queue drained"),
-            "unknown terminal status should identify lost worker-result persistence: {terminal}"
-        );
-    }
     let skipped_queue_full = terminal["report"]["skipped_queue_full"]
         .as_u64()
         .ok_or_else(|| format!("terminal status omitted report.skipped_queue_full: {terminal}"))?;

crates/e2e_test/src/kms/configured_roundtrip_test.rs

@@ -126,55 +126,7 @@ async fn assert_key_deletion_lifecycle(base_url: &str, access_key: &str, secret_
     assert_eq!(cancelled["success"], true);
     assert_eq!(cancelled["key_metadata"]["key_state"], "Enabled");
 
-    // A window outside 7-30 days is refused at the endpoint, whatever the
-    // backend: the bound is enforced once in the service, so no backend can
-    // stretch or skip it (rustfs/backlog#1585).
-    for days in [6, 31] {
-        let refused = kms_admin_request(
-            base_url,
-            http::Method::DELETE,
-            "/rustfs/admin/v3/kms/keys/delete",
-            Some(
-                &serde_json::json!({
-                    "key_id": key_id,
-                    "pending_window_in_days": days
-                })
-                .to_string(),
-            ),
-            access_key,
-            secret_key,
-        )
-        .await
-        .err()
-        .ok_or_else(|| format!("a {days}-day deletion window must be refused"))?;
-        assert!(
-            refused.to_string().contains("400 Bad Request"),
-            "a {days}-day deletion window must report a client error: {refused}"
-        );
-    }
-
-    // Immediate deletion is no longer reachable through the query string, so it
-    // fails before the service gate is even consulted.
-    let refused = kms_admin_request(
-        base_url,
-        http::Method::DELETE,
-        &format!("/rustfs/admin/v3/kms/keys/delete?keyId={key_id}&force_immediate=true"),
-        None,
-        access_key,
-        secret_key,
-    )
-    .await
-    .err()
-    .ok_or("immediate KMS key deletion must not be reachable through the query string")?;
-    assert!(
-        refused.to_string().contains("400 Bad Request"),
-        "a query-string immediate deletion must report a client error: {refused}"
-    );
-
-    // A default server refuses to skip the waiting window (rustfs/backlog#1585):
-    // immediate deletion is unrecoverable and takes every object encrypted under
-    // the key with it, so the endpoint must reject it rather than honour it.
-    let refused = kms_admin_request(
+    let removed = kms_admin_request(
         base_url,
         http::Method::DELETE,
         "/rustfs/admin/v3/kms/keys/delete",
@@ -188,49 +140,37 @@ async fn assert_key_deletion_lifecycle(base_url: &str, access_key: &str, secret_
         access_key,
         secret_key,
     )
-    .await
-    .err()
-    .ok_or("immediate KMS key deletion must be refused on a default server")?;
-    assert!(
-        refused.to_string().contains("400 Bad Request"),
-        "refused immediate deletion must report a client error: {refused}"
-    );
-
-    // The refused requests left the key alone, so the window-bounded path still
-    // has something to schedule.
-    let described = kms_admin_request(
-        base_url,
-        http::Method::GET,
-        &format!("/rustfs/admin/v3/kms/keys/{key_id}"),
-        None,
-        access_key,
-        secret_key,
-    )
     .await?;
-    let described: serde_json::Value = serde_json::from_str(&described)?;
-    assert_eq!(
-        described["key_metadata"]["key_state"], "Enabled",
-        "a refused immediate deletion must leave the key usable"
-    );
+    let removed: serde_json::Value = serde_json::from_str(&removed)?;
+    assert_eq!(removed["success"], true);
 
-    let rescheduled = kms_admin_request(
-        base_url,
-        http::Method::DELETE,
-        "/rustfs/admin/v3/kms/keys/delete",
-        Some(
-            &serde_json::json!({
-                "key_id": key_id,
-                "pending_window_in_days": 7
-            })
-            .to_string(),
-        ),
-        access_key,
-        secret_key,
-    )
-    .await?;
-    let rescheduled: serde_json::Value = serde_json::from_str(&rescheduled)?;
-    assert_eq!(rescheduled["success"], true);
-    assert!(rescheduled["deletion_date"].is_string());
+    let listed =
+        kms_admin_request(base_url, http::Method::GET, "/rustfs/admin/v3/kms/keys", None, access_key, secret_key).await?;
+    let listed: serde_json::Value = serde_json::from_str(&listed)?;
+    assert_eq!(listed["success"], true);
+    let keys = listed["keys"]
+        .as_array()
+        .ok_or("list KMS keys response omitted keys after deletion")?;
+    if let Some(key) = keys.iter().find(|key| key["key_id"] == key_id) {
+        assert_eq!(key["status"], "PendingDeletion", "a retained force-deleted key must be pending deletion");
+        let removed = kms_admin_request(
+            base_url,
+            http::Method::DELETE,
+            "/rustfs/admin/v3/kms/keys/delete",
+            Some(
+                &serde_json::json!({
+                    "key_id": key_id,
+                    "force_immediate": true
+                })
+                .to_string(),
+            ),
+            access_key,
+            secret_key,
+        )
+        .await?;
+        let removed: serde_json::Value = serde_json::from_str(&removed)?;
+        assert_eq!(removed["success"], true);
+    }
 
     let listed =
         kms_admin_request(base_url, http::Method::GET, "/rustfs/admin/v3/kms/keys", None, access_key, secret_key).await?;
@@ -239,11 +179,10 @@ async fn assert_key_deletion_lifecycle(base_url: &str, access_key: &str, secret_
     let keys = listed["keys"]
         .as_array()
         .ok_or("final list KMS keys response omitted keys after deletion")?;
-    let key = keys
-        .iter()
-        .find(|key| key["key_id"] == key_id)
-        .ok_or("a key awaiting its deletion window must still be listed")?;
-    assert_eq!(key["status"], "PendingDeletion", "a scheduled key must be pending deletion");
+    assert!(
+        keys.iter().all(|key| key["key_id"] != key_id),
+        "force-deleted KMS key must no longer appear in list"
+    );
     Ok(())
 }
 

crates/e2e_test/src/kms/copy_object_self_copy_sse_test.rs

@@ -1,351 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression test: a same-key CopyObject that only rewrites metadata must never re-key a
-//! managed-SSE (SSE-S3 / SSE-KMS) object.
-//!
-//! On an **unversioned** bucket the handler marks a same-name copy `metadata_only`, and the
-//! store layer then updates `xl.meta` in place without touching the data blocks. The handler
-//! nevertheless strips the source encryption metadata and generates a *fresh* DEK for the
-//! destination. Combining the two writes "new DEK + old ciphertext": the object is permanently
-//! undecryptable. The fix forces a full data rewrite whenever the copy re-derives managed
-//! encryption material, so the stored bytes always match the key metadata beside them.
-//!
-//! Companion to `copy_object_version_restore_sse_test` (issue #4238), which pins the same
-//! invariant for the versioned historical-restore path.
-
-use super::common::{LocalKMSTestEnvironment, create_key_with_specific_id};
-use crate::common::init_logging;
-use aws_sdk_s3::primitives::ByteStream;
-use aws_sdk_s3::types::{
-    MetadataDirective, ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration,
-    ServerSideEncryptionRule,
-};
-use serial_test::serial;
-use tracing::info;
-
-#[tokio::test]
-#[serial]
-async fn test_metadata_replace_self_copy_of_sse_object_stays_decryptable() {
-    init_logging();
-    info!("same-key CopyObject with REPLACE metadata must not re-key an SSE-S3 object");
-
-    let mut kms_env = LocalKMSTestEnvironment::new().await.expect("failed to create local KMS env");
-    let default_key_id = "rustfs-e2e-test-default-key";
-    let keys_dir = kms_env.kms_keys_dir.clone();
-    create_key_with_specific_id(&keys_dir, default_key_id)
-        .await
-        .expect("failed to create local KMS key");
-    kms_env
-        .base_env
-        .start_rustfs_server_with_env(
-            vec![
-                "--kms-enable",
-                "--kms-backend",
-                "local",
-                "--kms-key-dir",
-                &keys_dir,
-                "--kms-default-key-id",
-                default_key_id,
-            ],
-            &[("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")],
-        )
-        .await
-        .expect("failed to start RustFS with local KMS");
-    tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;
-
-    let client = kms_env.base_env.create_s3_client();
-    // Deliberately an UNVERSIONED bucket: that is the branch where the store layer can service
-    // the self-copy as a pure metadata update.
-    let bucket = "copy-object-self-copy-sse-test";
-    let key = "secrets/report.txt";
-
-    client
-        .create_bucket()
-        .bucket(bucket)
-        .send()
-        .await
-        .expect("failed to create bucket");
-
-    // Content long enough that a truncated/garbled decrypt cannot coincidentally match.
-    let content = b"encrypted payload that must survive a metadata-only self copy -- 0123456789";
-    let put = client
-        .put_object()
-        .bucket(bucket)
-        .key(key)
-        .content_type("text/plain; charset=utf-8")
-        .metadata("stage", "before")
-        .server_side_encryption(ServerSideEncryption::Aes256)
-        .body(ByteStream::from_static(content))
-        .send()
-        .await
-        .expect("PUT failed");
-    assert_eq!(put.server_side_encryption(), Some(&ServerSideEncryption::Aes256));
-
-    // Copy the object onto itself, replacing user metadata. This is the `mc cp --attr` /
-    // "edit metadata in place" shape that AWS supports on an existing object.
-    let copy_out = client
-        .copy_object()
-        .bucket(bucket)
-        .key(key)
-        .copy_source(format!("{bucket}/{key}"))
-        .metadata_directive(MetadataDirective::Replace)
-        .content_type("text/plain; charset=utf-8")
-        .metadata("stage", "after")
-        .server_side_encryption(ServerSideEncryption::Aes256)
-        .send()
-        .await
-        .expect("same-key CopyObject with REPLACE metadata must succeed");
-    assert_eq!(copy_out.server_side_encryption(), Some(&ServerSideEncryption::Aes256));
-
-    // The object must still decrypt to the original plaintext. Before the fix the stored
-    // ciphertext was left untouched while the metadata carried a brand-new DEK, so this GET
-    // either failed outright or returned garbage.
-    let get = client
-        .get_object()
-        .bucket(bucket)
-        .key(key)
-        .send()
-        .await
-        .expect("GET after self-copy failed: the object was re-keyed without rewriting the ciphertext");
-    assert_eq!(get.server_side_encryption(), Some(&ServerSideEncryption::Aes256));
-    assert_eq!(
-        get.metadata().and_then(|m| m.get("stage")),
-        Some(&"after".to_string()),
-        "REPLACE metadata must take effect"
-    );
-    let body = get.body.collect().await.expect("collect body").into_bytes();
-    assert_eq!(
-        body.as_ref(),
-        content,
-        "object must still decrypt to the original plaintext after a metadata-only self copy"
-    );
-
-    kms_env.base_env.stop_server();
-}
-
-#[tokio::test]
-#[serial]
-async fn test_metadata_replace_self_copy_dropping_sse_rewrites_plaintext() {
-    init_logging();
-    info!("same-key CopyObject that drops SSE must rewrite the data, not orphan the ciphertext");
-
-    let mut kms_env = LocalKMSTestEnvironment::new().await.expect("failed to create local KMS env");
-    let default_key_id = "rustfs-e2e-test-default-key";
-    let keys_dir = kms_env.kms_keys_dir.clone();
-    create_key_with_specific_id(&keys_dir, default_key_id)
-        .await
-        .expect("failed to create local KMS key");
-    kms_env
-        .base_env
-        .start_rustfs_server_with_env(
-            vec![
-                "--kms-enable",
-                "--kms-backend",
-                "local",
-                "--kms-key-dir",
-                &keys_dir,
-                "--kms-default-key-id",
-                default_key_id,
-            ],
-            &[("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")],
-        )
-        .await
-        .expect("failed to start RustFS with local KMS");
-    tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;
-
-    let client = kms_env.base_env.create_s3_client();
-    // Unversioned, and deliberately WITHOUT a bucket default-encryption rule, so the copy below
-    // resolves to "no destination encryption".
-    let bucket = "copy-object-self-copy-drop-sse-test";
-    let key = "secrets/report.txt";
-
-    client
-        .create_bucket()
-        .bucket(bucket)
-        .send()
-        .await
-        .expect("failed to create bucket");
-
-    let content = b"encrypted payload whose ciphertext must not survive as bogus plaintext -- 0123456789";
-    client
-        .put_object()
-        .bucket(bucket)
-        .key(key)
-        .metadata("stage", "before")
-        .server_side_encryption(ServerSideEncryption::Aes256)
-        .body(ByteStream::from_static(content))
-        .send()
-        .await
-        .expect("PUT failed");
-
-    // Self-copy with REPLACE and no SSE header. Per AWS semantics the destination ends up
-    // unencrypted. The dangerous outcome is the silent one: the handler strips the source key
-    // metadata while a metadata-only copy leaves the ciphertext in place, so a later GET would
-    // hand back raw ciphertext as if it were plaintext — corruption with no error anywhere.
-    client
-        .copy_object()
-        .bucket(bucket)
-        .key(key)
-        .copy_source(format!("{bucket}/{key}"))
-        .metadata_directive(MetadataDirective::Replace)
-        .metadata("stage", "after")
-        .send()
-        .await
-        .expect("same-key CopyObject dropping SSE must succeed");
-
-    let get = client
-        .get_object()
-        .bucket(bucket)
-        .key(key)
-        .send()
-        .await
-        .expect("GET after self-copy failed");
-    assert_eq!(
-        get.server_side_encryption(),
-        None,
-        "destination must be unencrypted once the copy drops SSE"
-    );
-    assert_eq!(
-        get.metadata().and_then(|m| m.get("stage")),
-        Some(&"after".to_string()),
-        "REPLACE metadata must take effect"
-    );
-    let body = get.body.collect().await.expect("collect body").into_bytes();
-    assert_eq!(
-        body.as_ref(),
-        content,
-        "object must read back as the original plaintext, not the orphaned ciphertext"
-    );
-
-    kms_env.base_env.stop_server();
-}
-
-#[tokio::test]
-#[serial]
-async fn test_metadata_replace_self_copy_under_bucket_default_sse_stays_decryptable() {
-    init_logging();
-    info!("bucket default encryption must also keep a same-key copy off the metadata-only path");
-
-    let mut kms_env = LocalKMSTestEnvironment::new().await.expect("failed to create local KMS env");
-    let default_key_id = "rustfs-e2e-test-default-key";
-    let keys_dir = kms_env.kms_keys_dir.clone();
-    create_key_with_specific_id(&keys_dir, default_key_id)
-        .await
-        .expect("failed to create local KMS key");
-    kms_env
-        .base_env
-        .start_rustfs_server_with_env(
-            vec![
-                "--kms-enable",
-                "--kms-backend",
-                "local",
-                "--kms-key-dir",
-                &keys_dir,
-                "--kms-default-key-id",
-                default_key_id,
-            ],
-            &[("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")],
-        )
-        .await
-        .expect("failed to start RustFS with local KMS");
-    tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;
-
-    let client = kms_env.base_env.create_s3_client();
-    let bucket = "copy-object-self-copy-bucket-default-sse-test";
-    let key = "secrets/report.txt";
-
-    client
-        .create_bucket()
-        .bucket(bucket)
-        .send()
-        .await
-        .expect("failed to create bucket");
-
-    // Store the object as PLAINTEXT first: no SSE header and no bucket default rule yet. This is
-    // what makes the case sharp — at copy time the source metadata carries no encryption markers,
-    // so the source-side half of the guard cannot fire.
-    let content = b"plaintext payload that must not be orphaned under a new DEK -- 0123456789";
-    let put = client
-        .put_object()
-        .bucket(bucket)
-        .key(key)
-        .metadata("stage", "before")
-        .body(ByteStream::from_static(content))
-        .send()
-        .await
-        .expect("PUT failed");
-    assert_eq!(put.server_side_encryption(), None, "the object must start out unencrypted");
-
-    // Only NOW enable bucket default encryption. The destination's encryption therefore comes
-    // from the bucket rule and from nowhere else: the source is unencrypted and the copy request
-    // carries no SSE header. A guard that only inspects request headers (MinIO decides
-    // `isTargetEncrypted` from `crypto.S3.IsRequested(r.Header)`) would let this through, yet
-    // `sse_encryption` still mints a fresh DEK from the resolved bucket default — which is why
-    // the guard keys off the *effective* encryption rather than the requested one.
-    let encryption_config = ServerSideEncryptionConfiguration::builder()
-        .rules(
-            ServerSideEncryptionRule::builder()
-                .apply_server_side_encryption_by_default(
-                    ServerSideEncryptionByDefault::builder()
-                        .sse_algorithm(ServerSideEncryption::Aes256)
-                        .build()
-                        .unwrap(),
-                )
-                .build(),
-        )
-        .build()
-        .unwrap();
-    client
-        .put_bucket_encryption()
-        .bucket(bucket)
-        .server_side_encryption_configuration(encryption_config)
-        .send()
-        .await
-        .expect("failed to set bucket default encryption");
-
-    // No SSE header on the copy — the bucket default alone drives the destination encryption.
-    client
-        .copy_object()
-        .bucket(bucket)
-        .key(key)
-        .copy_source(format!("{bucket}/{key}"))
-        .metadata_directive(MetadataDirective::Replace)
-        .metadata("stage", "after")
-        .send()
-        .await
-        .expect("same-key CopyObject under bucket default encryption must succeed");
-
-    let get = client
-        .get_object()
-        .bucket(bucket)
-        .key(key)
-        .send()
-        .await
-        .expect("GET after self-copy failed: the object was re-keyed without rewriting the ciphertext");
-    assert_eq!(get.server_side_encryption(), Some(&ServerSideEncryption::Aes256));
-    assert_eq!(
-        get.metadata().and_then(|m| m.get("stage")),
-        Some(&"after".to_string()),
-        "REPLACE metadata must take effect"
-    );
-    let body = get.body.collect().await.expect("collect body").into_bytes();
-    assert_eq!(
-        body.as_ref(),
-        content,
-        "object must still decrypt to the original plaintext after a metadata-only self copy"
-    );
-
-    kms_env.base_env.stop_server();
-}

crates/e2e_test/src/kms/kms_authorization_negative_matrix_test.rs

@@ -1,483 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Negative authorization matrix for per-key KMS access control.
-//!
-//! Every case here is an end-to-end denial that the pre-`kms` resource server
-//! allowed, so a regression that reopens one of them fails this file rather than
-//! only a unit test. The matrix varies one dimension at a time:
-//!
-//! - **wrong identity**: a caller holding S3 rights but no `kms` grant at all
-//! - **wrong key**: a caller scoped to key A naming key B
-//! - **wrong action**: a caller holding `kms:GenerateDataKey` but not `kms:Decrypt`
-//!   (and, on the admin plane, `kms:DisableKey` but not `kms:RotateKey`)
-//! - **wrong context**: an explicit `Deny` beating a wildcard `Allow`, and SSE-S3
-//!   staying exempt from `kms` authorization
-//!
-//! Each matrix opens with a positive control. Without it a denial proves nothing:
-//! an identity whose policy has not propagated yet is denied everything.
-
-use super::common::{LocalKMSTestEnvironment, create_key_with_specific_id};
-use crate::common::{admin_ok, admin_request, init_logging};
-use aws_sdk_s3::Client;
-use aws_sdk_s3::config::{Config, Credentials, Region};
-use aws_sdk_s3::error::ProvideErrorMetadata;
-use aws_sdk_s3::primitives::ByteStream;
-use aws_sdk_s3::types::ServerSideEncryption;
-use serial_test::serial;
-use std::time::Duration;
-use tracing::info;
-
-type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
-
-const ALLOWED_KEY: &str = "kms-matrix-allowed-key";
-const OTHER_KEY: &str = "kms-matrix-other-key";
-const BUCKET: &str = "kms-authz-matrix";
-const SECRET: &str = "kms-matrix-secret";
-const PAYLOAD: &[u8] = b"kms authorization matrix payload";
-
-/// How long an identity change may take to reach the request path.
-const IAM_PROPAGATION: Duration = Duration::from_secs(20);
-
-fn s3_client(url: &str, access_key: &str, secret_key: &str) -> Client {
-    let config = Config::builder()
-        .credentials_provider(Credentials::new(access_key, secret_key, None, None, "kms-authz-matrix"))
-        .region(Region::new("us-east-1"))
-        .endpoint_url(url)
-        .force_path_style(true)
-        .behavior_version_latest()
-        .build();
-    Client::from_conf(config)
-}
-
-/// Start a server whose SSE-KMS data path authorizes against the named key.
-///
-/// The enforcement switch defaults to off for compatibility, so it has to be set
-/// explicitly; without it every negative case below would silently pass as an allow.
-async fn start_enforcing_server(env: &mut LocalKMSTestEnvironment, extra_env: &[(&str, &str)]) -> TestResult {
-    create_key_with_specific_id(&env.kms_keys_dir, ALLOWED_KEY).await?;
-    create_key_with_specific_id(&env.kms_keys_dir, OTHER_KEY).await?;
-
-    let key_dir = env.kms_keys_dir.clone();
-    let args = vec![
-        "--kms-enable",
-        "--kms-backend",
-        "local",
-        "--kms-key-dir",
-        key_dir.as_str(),
-        "--kms-default-key-id",
-        ALLOWED_KEY,
-    ];
-
-    let mut envs = vec![("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")];
-    envs.extend_from_slice(extra_env);
-
-    env.base_env.start_rustfs_server_with_env(args, &envs).await?;
-    Ok(())
-}
-
-/// Create `user` with `policy_document` attached under a canned policy of the same name.
-async fn provision_user(env: &LocalKMSTestEnvironment, user: &str, policy_document: &str) -> TestResult {
-    admin_ok(
-        &env.base_env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-canned-policy?name={user}"),
-        Some(policy_document.to_string()),
-    )
-    .await?;
-    provision_user_with_policy(env, user, user).await
-}
-
-/// Create `user` and attach an existing policy (built-in or canned) by name.
-async fn provision_user_with_policy(env: &LocalKMSTestEnvironment, user: &str, policy_name: &str) -> TestResult {
-    admin_ok(
-        &env.base_env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-user?accessKey={user}"),
-        Some(serde_json::json!({ "secretKey": SECRET, "status": "enabled" }).to_string()),
-    )
-    .await?;
-    admin_ok(
-        &env.base_env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/set-user-or-group-policy?policyName={policy_name}&userOrGroup={user}&isGroup=false"),
-        None,
-    )
-    .await?;
-    Ok(())
-}
-
-/// The S3 half of every data-path policy below: full object access, no KMS grant.
-fn s3_full_access_statement() -> serde_json::Value {
-    serde_json::json!({
-        "Effect": "Allow",
-        "Action": ["s3:*"],
-        "Resource": ["arn:aws:s3:::*"]
-    })
-}
-
-fn policy_document(statements: Vec<serde_json::Value>) -> String {
-    serde_json::json!({ "Version": "2012-10-17", "Statement": statements }).to_string()
-}
-
-async fn put_sse_kms(client: &Client, key: &str, kms_key_id: &str) -> Result<(), aws_sdk_s3::Error> {
-    client
-        .put_object()
-        .bucket(BUCKET)
-        .key(key)
-        .body(ByteStream::from_static(PAYLOAD))
-        .server_side_encryption(ServerSideEncryption::AwsKms)
-        .ssekms_key_id(kms_key_id)
-        .send()
-        .await
-        .map(|_| ())
-        .map_err(aws_sdk_s3::Error::from)
-}
-
-/// Assert the operation failed with `AccessDenied` rather than any other error.
-///
-/// A bare `is_err` would also accept `KMSKeyDisabled` or an internal error, which
-/// would hide both a leak of key state and an outage masquerading as a denial.
-fn assert_access_denied<T: std::fmt::Debug>(result: Result<T, aws_sdk_s3::Error>, what: &str) {
-    let error = result.expect_err(&format!("{what} must be denied"));
-    assert_eq!(error.code(), Some("AccessDenied"), "{what} must fail with AccessDenied: {error:?}");
-}
-
-/// Retry an SSE-KMS write until the identity's policy has reached the request path.
-async fn wait_for_sse_kms_write(client: &Client, key: &str, kms_key_id: &str) -> TestResult {
-    let deadline = tokio::time::Instant::now() + IAM_PROPAGATION;
-    loop {
-        match put_sse_kms(client, key, kms_key_id).await {
-            Ok(()) => return Ok(()),
-            Err(error) if tokio::time::Instant::now() >= deadline => {
-                return Err(format!("positive control never became authorized: {error:?}").into());
-            }
-            Err(_) => tokio::time::sleep(Duration::from_millis(500)).await,
-        }
-    }
-}
-
-/// Retry an admin call until it stops returning 403, i.e. the policy is live.
-async fn wait_for_admin_success(
-    env: &LocalKMSTestEnvironment,
-    user: &str,
-    method: http::Method,
-    path: &str,
-    body: Option<String>,
-) -> TestResult {
-    let deadline = tokio::time::Instant::now() + IAM_PROPAGATION;
-    loop {
-        let (status, response) = admin_request(&env.base_env.url, method.clone(), path, body.clone(), user, SECRET).await?;
-        if status.is_success() {
-            return Ok(());
-        }
-        if tokio::time::Instant::now() >= deadline {
-            return Err(format!("positive control never became authorized: {method} {path} -> {status} {response}").into());
-        }
-        tokio::time::sleep(Duration::from_millis(500)).await;
-    }
-}
-
-async fn assert_admin_denied(
-    env: &LocalKMSTestEnvironment,
-    user: &str,
-    method: http::Method,
-    path: &str,
-    body: Option<String>,
-    what: &str,
-) -> TestResult {
-    let (status, response) = admin_request(&env.base_env.url, method, path, body, user, SECRET).await?;
-    assert_eq!(status.as_u16(), 403, "{what} must be denied, got {status}: {response}");
-    assert!(response.contains("AccessDenied"), "{what} must carry AccessDenied: {response}");
-    Ok(())
-}
-
-fn disable_body(key_id: &str) -> String {
-    serde_json::json!({ "key_id": key_id }).to_string()
-}
-
-/// Data-path matrix: SSE-KMS writes and reads are authorized against the resolved key.
-#[tokio::test]
-#[serial]
-async fn sse_kms_per_key_authorization_negative_matrix() -> TestResult {
-    init_logging();
-
-    let mut env = LocalKMSTestEnvironment::new().await?;
-    start_enforcing_server(&mut env, &[("RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY", "true")]).await?;
-    env.base_env.create_test_bucket(BUCKET).await?;
-
-    // Scoped to ALLOWED_KEY only.
-    provision_user(
-        &env,
-        "kmsmatrixscoped",
-        &policy_document(vec![
-            s3_full_access_statement(),
-            serde_json::json!({
-                "Effect": "Allow",
-                "Action": ["kms:GenerateDataKey", "kms:Decrypt"],
-                "Resource": [format!("arn:aws:kms:::key/{ALLOWED_KEY}")]
-            }),
-        ]),
-    )
-    .await?;
-    // S3 rights only: the identity shape that existed before per-key authorization.
-    provision_user(&env, "kmsmatrixs3only", &policy_document(vec![s3_full_access_statement()])).await?;
-    // May wrap a data key but may never unwrap one.
-    provision_user(
-        &env,
-        "kmsmatrixwriter",
-        &policy_document(vec![
-            s3_full_access_statement(),
-            serde_json::json!({
-                "Effect": "Allow",
-                "Action": ["kms:GenerateDataKey"],
-                "Resource": ["arn:aws:kms:::*"]
-            }),
-        ]),
-    )
-    .await?;
-    // Wildcard allow, explicit deny on one key.
-    provision_user(
-        &env,
-        "kmsmatrixdenied",
-        &policy_document(vec![
-            s3_full_access_statement(),
-            serde_json::json!({
-                "Effect": "Allow",
-                "Action": ["kms:*"],
-                "Resource": ["arn:aws:kms:::*"]
-            }),
-            serde_json::json!({
-                "Effect": "Deny",
-                "Action": ["kms:*"],
-                "Resource": [format!("arn:aws:kms:::key/{OTHER_KEY}")]
-            }),
-        ]),
-    )
-    .await?;
-
-    let scoped = s3_client(&env.base_env.url, "kmsmatrixscoped", SECRET);
-    let s3_only = s3_client(&env.base_env.url, "kmsmatrixs3only", SECRET);
-    let writer = s3_client(&env.base_env.url, "kmsmatrixwriter", SECRET);
-    let denied = s3_client(&env.base_env.url, "kmsmatrixdenied", SECRET);
-
-    // --- positive control -----------------------------------------------------
-    wait_for_sse_kms_write(&scoped, "scoped/allowed", ALLOWED_KEY).await?;
-    let read = scoped.get_object().bucket(BUCKET).key("scoped/allowed").send().await?;
-    assert_eq!(read.body.collect().await?.into_bytes().as_ref(), PAYLOAD);
-    info!("positive control: scoped identity may write and read under its own key");
-
-    // --- wrong key ------------------------------------------------------------
-    assert_access_denied(
-        put_sse_kms(&scoped, "scoped/other", OTHER_KEY).await,
-        "SSE-KMS write under a key outside the identity's scope",
-    );
-
-    // --- wrong identity -------------------------------------------------------
-    assert_access_denied(
-        put_sse_kms(&s3_only, "s3only/allowed", ALLOWED_KEY).await,
-        "SSE-KMS write by an identity holding no kms grant",
-    );
-    // The object the scoped identity wrote is readable by its owner only.
-    assert_access_denied(
-        s3_only
-            .get_object()
-            .bucket(BUCKET)
-            .key("scoped/allowed")
-            .send()
-            .await
-            .map(|_| ())
-            .map_err(aws_sdk_s3::Error::from),
-        "SSE-KMS read by an identity holding no kms grant",
-    );
-
-    // --- wrong action ---------------------------------------------------------
-    wait_for_sse_kms_write(&writer, "writer/allowed", ALLOWED_KEY).await?;
-    assert_access_denied(
-        writer
-            .get_object()
-            .bucket(BUCKET)
-            .key("writer/allowed")
-            .send()
-            .await
-            .map(|_| ())
-            .map_err(aws_sdk_s3::Error::from),
-        "SSE-KMS read by an identity holding kms:GenerateDataKey but not kms:Decrypt",
-    );
-
-    // --- wrong context: explicit Deny beats a wildcard Allow -------------------
-    wait_for_sse_kms_write(&denied, "denied/allowed", ALLOWED_KEY).await?;
-    assert_access_denied(
-        put_sse_kms(&denied, "denied/other", OTHER_KEY).await,
-        "SSE-KMS write under a key covered by an explicit Deny",
-    );
-
-    // --- wrong context: SSE-S3 is out of scope --------------------------------
-    // SSE-S3 wraps its data key with a server-owned key the caller never names, so
-    // it must stay reachable for an identity with no kms grant at all.
-    s3_only
-        .put_object()
-        .bucket(BUCKET)
-        .key("s3only/sse-s3")
-        .body(ByteStream::from_static(PAYLOAD))
-        .server_side_encryption(ServerSideEncryption::Aes256)
-        .send()
-        .await?;
-    let sse_s3_read = s3_only.get_object().bucket(BUCKET).key("s3only/sse-s3").send().await?;
-    assert_eq!(sse_s3_read.body.collect().await?.into_bytes().as_ref(), PAYLOAD);
-
-    // ... and so must an unencrypted object.
-    s3_only
-        .put_object()
-        .bucket(BUCKET)
-        .key("s3only/plain")
-        .body(ByteStream::from_static(PAYLOAD))
-        .send()
-        .await?;
-    s3_only.get_object().bucket(BUCKET).key("s3only/plain").send().await?;
-
-    Ok(())
-}
-
-/// Admin-plane matrix: KMS key endpoints are authorized against the key they name.
-///
-/// Runs without the SSE enforcement switch: admin scoping is unconditional, and
-/// leaving the switch off proves the two planes are independent.
-#[tokio::test]
-#[serial]
-async fn kms_admin_per_key_authorization_negative_matrix() -> TestResult {
-    init_logging();
-
-    let mut env = LocalKMSTestEnvironment::new().await?;
-    start_enforcing_server(&mut env, &[]).await?;
-
-    // Built-in role templates, attached by name.
-    provision_user_with_policy(&env, "kmsmatrixkeyadmin", "KMSKeyAdministrator").await?;
-    provision_user_with_policy(&env, "kmsmatrixauditor", "KMSAuditor").await?;
-    // A narrowed copy of the administrator template, scoped to one key.
-    provision_user(
-        &env,
-        "kmsmatrixscopedadmin",
-        &policy_document(vec![serde_json::json!({
-            "Effect": "Allow",
-            "Action": ["kms:DisableKey", "kms:EnableKey"],
-            "Resource": [format!("arn:aws:kms:::key/{ALLOWED_KEY}")]
-        })]),
-    )
-    .await?;
-
-    // --- positive control -----------------------------------------------------
-    wait_for_admin_success(
-        &env,
-        "kmsmatrixkeyadmin",
-        http::Method::POST,
-        "/rustfs/admin/v3/kms/keys/disable",
-        Some(disable_body(OTHER_KEY)),
-    )
-    .await?;
-    admin_request(
-        &env.base_env.url,
-        http::Method::POST,
-        "/rustfs/admin/v3/kms/keys/enable",
-        Some(disable_body(OTHER_KEY)),
-        "kmsmatrixkeyadmin",
-        SECRET,
-    )
-    .await?;
-
-    // --- wrong action: the administrator template withholds service-wide powers -
-    assert_admin_denied(
-        &env,
-        "kmsmatrixkeyadmin",
-        http::Method::GET,
-        "/rustfs/admin/v3/kms/config",
-        None,
-        "KMSKeyAdministrator reading the KMS backend configuration (kms:Configure)",
-    )
-    .await?;
-    assert_admin_denied(
-        &env,
-        "kmsmatrixkeyadmin",
-        http::Method::GET,
-        "/rustfs/admin/v3/kms/backup",
-        None,
-        "KMSKeyAdministrator exporting a backup bundle (kms:Backup)",
-    )
-    .await?;
-    // Separation of duties: managing a key never implies using it.
-    assert_admin_denied(
-        &env,
-        "kmsmatrixkeyadmin",
-        http::Method::POST,
-        "/rustfs/admin/v3/kms/generate-data-key",
-        Some(serde_json::json!({ "key_id": ALLOWED_KEY }).to_string()),
-        "KMSKeyAdministrator generating a data key (kms:GenerateDataKey)",
-    )
-    .await?;
-
-    // --- wrong action: the auditor template is read-only ----------------------
-    wait_for_admin_success(&env, "kmsmatrixauditor", http::Method::GET, "/rustfs/admin/v3/kms/keys", None).await?;
-    assert_admin_denied(
-        &env,
-        "kmsmatrixauditor",
-        http::Method::POST,
-        "/rustfs/admin/v3/kms/keys/disable",
-        Some(disable_body(ALLOWED_KEY)),
-        "KMSAuditor disabling a key (kms:DisableKey)",
-    )
-    .await?;
-
-    // --- wrong key ------------------------------------------------------------
-    wait_for_admin_success(
-        &env,
-        "kmsmatrixscopedadmin",
-        http::Method::POST,
-        "/rustfs/admin/v3/kms/keys/disable",
-        Some(disable_body(ALLOWED_KEY)),
-    )
-    .await?;
-    assert_admin_denied(
-        &env,
-        "kmsmatrixscopedadmin",
-        http::Method::POST,
-        "/rustfs/admin/v3/kms/keys/disable",
-        Some(disable_body(OTHER_KEY)),
-        "key-scoped administrator disabling a key outside its scope",
-    )
-    .await?;
-    // --- wrong action, same key ----------------------------------------------
-    assert_admin_denied(
-        &env,
-        "kmsmatrixscopedadmin",
-        http::Method::POST,
-        "/rustfs/admin/v3/kms/keys/rotate",
-        Some(disable_body(ALLOWED_KEY)),
-        "key-scoped administrator rotating a key it may only enable and disable",
-    )
-    .await?;
-
-    // --- wrong identity -------------------------------------------------------
-    provision_user(&env, "kmsmatrixnokms", &policy_document(vec![s3_full_access_statement()])).await?;
-    assert_admin_denied(
-        &env,
-        "kmsmatrixnokms",
-        http::Method::POST,
-        "/rustfs/admin/v3/kms/keys/disable",
-        Some(disable_body(ALLOWED_KEY)),
-        "identity holding no kms grant disabling a key",
-    )
-    .await?;
-
-    Ok(())
-}

crates/e2e_test/src/kms/kms_vault_test.rs

@@ -417,22 +417,6 @@ async fn test_vault_kms_key_crud(
 
     info!("✅ Read: Successfully listed keys, found test key");
 
-    // A waiting window outside 7-30 days is refused at the endpoint for this
-    // backend too: the bound is enforced once in the service (rustfs/backlog#1585).
-    for days in [6, 31] {
-        let window_error = crate::common::execute_awscurl(
-            &format!("{base_url}/rustfs/admin/v3/kms/keys/delete?keyId={key_id}&pending_window_in_days={days}"),
-            "DELETE",
-            None,
-            access_key,
-            secret_key,
-        )
-        .await
-        .err()
-        .ok_or_else(|| format!("A {days}-day deletion window must be refused"))?;
-        info!("✅ Delete window {} correctly refused: {}", days, window_error);
-    }
-
     // Delete
     let delete_response = crate::common::execute_awscurl(
         &format!("{base_url}/rustfs/admin/v3/kms/keys/delete?keyId={key_id}"),
@@ -465,32 +449,29 @@ async fn test_vault_kms_key_crud(
 
     info!("✅ Delete verification: Key state correctly changed to: {}", key_state);
 
-    // Force Delete - the query string can no longer ask for immediate deletion,
-    // and a default server refuses it in any case (rustfs/backlog#1585):
-    // destroying the key material immediately would take every object encrypted
-    // under the key with it.
-    let force_delete_error = crate::common::execute_awscurl(
+    // Force Delete - Force immediate deletion for PendingDeletion key
+    let force_delete_response = crate::common::execute_awscurl(
         &format!("{base_url}/rustfs/admin/v3/kms/keys/delete?keyId={key_id}&force_immediate=true"),
         "DELETE",
         None,
         access_key,
         secret_key,
     )
-    .await
-    .expect_err("Immediate KMS key deletion must be refused on a default server");
-    info!("✅ Force Delete: correctly refused for key {}: {}", key_id, force_delete_error);
+    .await?;
 
-    // The refused request must leave the key exactly as it was: still present,
-    // still pending deletion, still recoverable through cancel-deletion.
-    let describe_after_refusal =
-        crate::common::awscurl_get(&format!("{base_url}/rustfs/admin/v3/kms/keys/{key_id}"), access_key, secret_key).await?;
-    let describe_after_refusal: serde_json::Value = serde_json::from_str(&describe_after_refusal)?;
-    assert_eq!(
-        describe_after_refusal["key_metadata"]["key_state"], "PendingDeletion",
-        "A refused immediate deletion must leave the key pending deletion"
-    );
+    // Parse and validate the force delete response
+    let force_delete_result: serde_json::Value = serde_json::from_str(&force_delete_response)?;
+    assert_eq!(force_delete_result["success"], true, "Force delete operation must return success=true");
+    info!("✅ Force Delete: Successfully force deleted key: {}", key_id);
 
-    info!("✅ Force Delete verification: Key survived the refused immediate deletion");
+    // Verify key no longer exists after force deletion (should return error)
+    let describe_force_deleted_result =
+        crate::common::awscurl_get(&format!("{base_url}/rustfs/admin/v3/kms/keys/{key_id}"), access_key, secret_key).await;
+
+    // After force deletion, key should not be found (GET should fail)
+    assert!(describe_force_deleted_result.is_err(), "Force deleted key should not be found");
+
+    info!("✅ Force Delete verification: Key was permanently deleted and is no longer accessible");
 
     info!("Vault KMS key CRUD operations completed successfully");
     Ok(())

crates/e2e_test/src/kms/mod.rs

@@ -48,14 +48,8 @@ mod bucket_default_encryption_test;
 #[cfg(test)]
 mod encryption_metadata_test;
 
-#[cfg(test)]
-mod copy_object_self_copy_sse_test;
-
 #[cfg(test)]
 mod copy_object_version_restore_sse_test;
 
 #[cfg(test)]
 mod configured_roundtrip_test;
-
-#[cfg(test)]
-mod kms_authorization_negative_matrix_test;

crates/e2e_test/src/lib.rs

@@ -298,32 +298,4 @@ mod create_bucket_region_test;
 #[cfg(test)]
 mod copy_source_invalid_date_test;
 
-// P0 regression: event notification startup race (rustfs#5387, #5681, #5401, #5183, #5115, #4796)
-#[cfg(test)]
-mod notification_startup_regression_test;
-
-// P0 regression: lifecycle/ILM object expiration (rustfs#5407, #5167, #4963, #5615, #4879)
-#[cfg(test)]
-mod lifecycle_regression_test;
-
-// P0 regression: delete operations consistency (rustfs#5375, #5349, #5339, #5029, #4978, #760)
-#[cfg(test)]
-mod delete_regression_test;
-
-// P1 regression: listing/metacache completeness (rustfs#5166, #5156, #5051, #4810, #4648, #3191)
-#[cfg(test)]
-mod listing_regression_test;
-
-// P1 regression: bucket statistics accuracy (rustfs#5615, #5008, #5116, #5055, #3898, #1012)
-#[cfg(test)]
-mod bucket_stats_regression_test;
-
-// P1 regression: distributed startup/quorum (rustfs#5416, #2945, #2794, #2601, #4040, #5655)
-#[cfg(test)]
-mod distributed_startup_regression_test;
-
-// P1 regression: tier/ILM transition (rustfs#5218, #5130, #5011, #4826, #5024)
-#[cfg(test)]
-mod tier_transition_regression_test;
-
 pub mod tls_gen;

crates/e2e_test/src/lifecycle_regression_test.rs

@@ -1,360 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression tests for lifecycle/ILM object expiration and transition.
-//!
-//! Covers the recurring pattern where ILM expiration rules do not actually
-//! delete objects, or lifecycle rule parameters are silently corrupted.
-//! This has regressed 6+ times.
-//!
-//! ## Regression Issues
-//!
-//! - rustfs#5407: lifecycle not delete any bucket object
-//! - rustfs#5167: lifecycle not delete object
-//! - rustfs#4963: lifecycle rule 3 days → effective value 0 days
-//! - rustfs#5615: bucket statistics remain unchanged after data expiration
-//! - rustfs#4879: ILM serial lane: restore transition never completes
-//! - rustfs#5442: Uncheck of Replicate Delete still deletes the file
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestEnvironment, init_logging};
-    use aws_sdk_s3::Client;
-    use aws_sdk_s3::primitives::ByteStream;
-    use aws_sdk_s3::types::{
-        BucketLifecycleConfiguration, BucketVersioningStatus, ExpirationStatus, LifecycleExpiration, LifecycleRule,
-        LifecycleRuleFilter, NoncurrentVersionExpiration, VersioningConfiguration,
-    };
-    use serial_test::serial;
-    use std::error::Error;
-    use tracing::info;
-
-    type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-    async fn setup_versioned_bucket(client: &Client, bucket: &str) -> TestResult {
-        client
-            .create_bucket()
-            .bucket(bucket)
-            .send()
-            .await
-            .map_err(|e| format!("create bucket: {e}"))?;
-
-        client
-            .put_bucket_versioning()
-            .bucket(bucket)
-            .versioning_configuration(
-                VersioningConfiguration::builder()
-                    .status(BucketVersioningStatus::Enabled)
-                    .build(),
-            )
-            .send()
-            .await
-            .map_err(|e| format!("enable versioning: {e}"))?;
-
-        Ok(())
-    }
-
-    /// RT-03: Verify that a lifecycle expiration rule actually deletes objects.
-    ///
-    /// Regression pattern: lifecycle rules are accepted but the scanner never
-    /// processes them, leaving expired objects in place.
-    ///
-    /// Steps:
-    /// 1. Create a versioned bucket
-    /// 2. Upload several objects
-    /// 3. Apply a lifecycle rule with 1-day expiration
-    /// 4. Wait for the scanner to process
-    /// 5. Verify objects are still present (they shouldn't expire yet — 1 day)
-    /// 6. Verify the lifecycle rule was persisted correctly (not corrupted to 0 days)
-    ///
-    /// This tests the rule persistence path (rustfs#4963: 3 days → 0 days).
-    #[tokio::test]
-    #[serial]
-    async fn test_lifecycle_expiration_rule_persists_correctly() -> TestResult {
-        init_logging();
-        info!("RT-03: lifecycle expiration rule persists correctly");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt03-lifecycle-persist";
-        setup_versioned_bucket(&client, bucket).await?;
-
-        // Apply a lifecycle rule with 1-day expiration on a prefix
-        let rule = LifecycleRule::builder()
-            .id("expire-after-1-day")
-            .status(ExpirationStatus::Enabled)
-            .filter(LifecycleRuleFilter::builder().prefix("logs/").build())
-            .expiration(LifecycleExpiration::builder().days(1).build())
-            .build()
-            .expect("build lifecycle rule");
-
-        client
-            .put_bucket_lifecycle_configuration()
-            .bucket(bucket)
-            .lifecycle_configuration(
-                BucketLifecycleConfiguration::builder()
-                    .rules(rule)
-                    .build()
-                    .expect("build lifecycle config"),
-            )
-            .send()
-            .await
-            .expect("put lifecycle configuration");
-
-        // Read back and verify the rule was not corrupted (rustfs#4963: days → 0)
-        let resp = client
-            .get_bucket_lifecycle_configuration()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("get lifecycle configuration");
-
-        let rules = resp.rules();
-        assert_eq!(rules.len(), 1, "RT-03 FAIL: expected exactly 1 lifecycle rule");
-
-        let retrieved = &rules[0];
-        assert_eq!(retrieved.id(), Some("expire-after-1-day"), "RT-03 FAIL: rule ID mismatch");
-        assert_eq!(retrieved.status(), &ExpirationStatus::Enabled, "RT-03 FAIL: rule should be Enabled");
-
-        let exp = retrieved.expiration().expect("expiration should be set");
-        assert_eq!(
-            exp.days(),
-            Some(1),
-            "RT-03 FAIL: expiration days corrupted (regression rustfs#4963: expected 1, got {:?})",
-            exp.days()
-        );
-
-        info!("RT-03 PASS: lifecycle expiration rule persists correctly");
-        Ok(())
-    }
-
-    /// RT-03b: Verify lifecycle rule with noncurrent version expiration.
-    ///
-    /// Covers the pattern where noncurrent version expiration rules are
-    /// accepted but old versions are never cleaned up.
-    #[tokio::test]
-    #[serial]
-    async fn test_lifecycle_noncurrent_version_expiration_rule_persists() -> TestResult {
-        init_logging();
-        info!("RT-03b: noncurrent version expiration rule persists");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt03b-noncurrent-expire";
-        setup_versioned_bucket(&client, bucket).await?;
-
-        // Create multiple versions of the same object
-        for i in 0..3 {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key("versioned-obj.txt")
-                .body(ByteStream::from(format!("version-{i}").into_bytes()))
-                .send()
-                .await
-                .expect("put object version");
-        }
-
-        // Verify we have 3 versions
-        let versions = client
-            .list_object_versions()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list versions");
-
-        let count = versions.versions().len();
-        assert_eq!(count, 3, "RT-03b FAIL: expected 3 versions, found {count}");
-
-        // Apply noncurrent version expiration rule
-        let rule = LifecycleRule::builder()
-            .id("expire-noncurrent-after-1-day")
-            .status(ExpirationStatus::Enabled)
-            .filter(LifecycleRuleFilter::builder().prefix("").build())
-            .noncurrent_version_expiration(NoncurrentVersionExpiration::builder().noncurrent_days(1).build())
-            .build()
-            .expect("build lifecycle rule");
-
-        client
-            .put_bucket_lifecycle_configuration()
-            .bucket(bucket)
-            .lifecycle_configuration(
-                BucketLifecycleConfiguration::builder()
-                    .rules(rule)
-                    .build()
-                    .expect("build lifecycle config"),
-            )
-            .send()
-            .await
-            .expect("put lifecycle configuration");
-
-        // Read back and verify
-        let resp = client
-            .get_bucket_lifecycle_configuration()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("get lifecycle configuration");
-
-        let rules = resp.rules();
-        assert_eq!(rules.len(), 1, "RT-03b FAIL: expected 1 rule");
-
-        let nc_exp = rules[0]
-            .noncurrent_version_expiration()
-            .expect("noncurrent expiration should be set");
-        assert_eq!(nc_exp.noncurrent_days(), Some(1), "RT-03b FAIL: noncurrent days corrupted");
-
-        info!("RT-03b PASS: noncurrent version expiration rule persists correctly");
-        Ok(())
-    }
-
-    /// RT-04: Verify lifecycle rule with prefix filter persists after restart.
-    ///
-    /// Covers the pattern where lifecycle rules are accepted but silently lost
-    /// after restart. Transition rules require a configured remote tier
-    /// (tested in reliant/tiering.rs), so this test uses expiration only.
-    #[tokio::test]
-    #[serial]
-    async fn test_lifecycle_prefix_rule_persists() -> TestResult {
-        init_logging();
-        info!("RT-04: lifecycle prefix rule persists");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt04-lifecycle-prefix";
-        setup_versioned_bucket(&client, bucket).await?;
-
-        let rule = LifecycleRule::builder()
-            .id("expire-archive-after-7-days")
-            .status(ExpirationStatus::Enabled)
-            .filter(LifecycleRuleFilter::builder().prefix("archive/").build())
-            .expiration(LifecycleExpiration::builder().days(7).build())
-            .build()
-            .expect("build lifecycle rule");
-
-        client
-            .put_bucket_lifecycle_configuration()
-            .bucket(bucket)
-            .lifecycle_configuration(
-                BucketLifecycleConfiguration::builder()
-                    .rules(rule)
-                    .build()
-                    .expect("build lifecycle config"),
-            )
-            .send()
-            .await
-            .expect("put lifecycle configuration");
-
-        // Restart server
-        env.restart_server_preserving_data(vec![], &[]).await.expect("restart RustFS");
-
-        // Verify the rule survived restart
-        let resp = client
-            .get_bucket_lifecycle_configuration()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("get lifecycle after restart");
-
-        let rules = resp.rules();
-        assert_eq!(rules.len(), 1, "RT-04 FAIL: expected 1 rule after restart");
-
-        let exp = rules[0].expiration().expect("expiration should be set");
-        assert_eq!(exp.days(), Some(7), "RT-04 FAIL: expiration days corrupted after restart");
-
-        info!("RT-04 PASS: lifecycle prefix rule persists after restart");
-        Ok(())
-    }
-
-    /// RT-05b: Verify delete marker creation in versioned bucket.
-    ///
-    /// Regression pattern: DELETE on a versioned object fails or does not
-    /// create a delete marker, or the delete marker is not visible in LIST.
-    #[tokio::test]
-    #[serial]
-    async fn test_delete_marker_creation_and_visibility() -> TestResult {
-        init_logging();
-        info!("RT-05b: delete marker creation and visibility");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt05b-delete-marker";
-        setup_versioned_bucket(&client, bucket).await?;
-
-        // Put an object
-        client
-            .put_object()
-            .bucket(bucket)
-            .key("marker-test.txt")
-            .body(ByteStream::from_static(b"to-be-deleted"))
-            .send()
-            .await
-            .expect("put object");
-
-        // Delete without specifying versionId → should create a delete marker
-        let del_resp = client
-            .delete_object()
-            .bucket(bucket)
-            .key("marker-test.txt")
-            .send()
-            .await
-            .expect("delete object");
-
-        // The response should indicate a delete marker was created
-        assert!(
-            del_resp.delete_marker().unwrap_or(false),
-            "RT-05b FAIL: DELETE on versioned object did not create a delete marker"
-        );
-
-        // ListObjectVersions should show both the original version and the delete marker
-        let versions = client
-            .list_object_versions()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list versions");
-
-        let delete_markers: Vec<_> = versions
-            .delete_markers()
-            .iter()
-            .filter(|dm| dm.key() == Some("marker-test.txt"))
-            .collect();
-
-        assert_eq!(
-            delete_markers.len(),
-            1,
-            "RT-05b FAIL: expected 1 delete marker, found {}",
-            delete_markers.len()
-        );
-
-        info!("RT-05b PASS: delete marker created and visible");
-        Ok(())
-    }
-}

crates/e2e_test/src/listing_regression_test.rs

@@ -1,357 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression tests for object listing and metacache consistency.
-//!
-//! Covers the recurring pattern where ListObjectsV2 returns incomplete results,
-//! silently truncates with IsTruncated=false, or corrupts the metadata cache.
-//! This has regressed 8+ times.
-//!
-//! ## Regression Issues
-//!
-//! - rustfs#5166: Metacache listing quorum failed timeout after cluster startup
-//! - rustfs#5156: Metacache producer failed
-//! - rustfs#5051: ListObjectsV2 returns empty results for shallow prefixes
-//! - rustfs#4810: walk_dir timeout silently truncates listings (200, IsTruncated=false)
-//! - rustfs#4648: Object listing oscillates between complete, partial, and zero
-//! - rustfs#3191: ListObjectsV2 timeout corrupts metadata cache → NoSuchBucket
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestEnvironment, init_logging};
-    use aws_sdk_s3::primitives::ByteStream;
-    use serial_test::serial;
-    use std::collections::HashSet;
-    use std::error::Error;
-    use tracing::info;
-
-    type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-    /// RT-06: Verify ListObjectsV2 pagination completeness for medium-sized bucket.
-    ///
-    /// Regression pattern: listing returns 200 with IsTruncated=false but
-    /// misses objects (rustfs#4810: walk_dir timeout truncation).
-    ///
-    /// Steps:
-    /// 1. Upload 100 objects with known keys
-    /// 2. List all objects via pagination (max_keys=10)
-    /// 3. Verify all 100 keys are returned exactly once
-    /// 4. Verify no duplicates or skipped keys
-    #[tokio::test]
-    #[serial]
-    async fn test_list_objects_v2_completeness_100_objects() -> TestResult {
-        init_logging();
-        info!("RT-06: listing completeness with 100 objects");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt06-list-completeness";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Upload 100 objects
-        let expected_keys: Vec<String> = (0..100).map(|i| format!("obj-{i:04}.txt")).collect();
-        for key in &expected_keys {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(key)
-                .body(ByteStream::from_static(b"data"))
-                .send()
-                .await
-                .expect("put object");
-        }
-
-        // Paginate through all objects (small page size to force multiple pages)
-        let mut all_keys: Vec<String> = Vec::new();
-        let mut continuation_token: Option<String> = None;
-
-        loop {
-            let mut req = client.list_objects_v2().bucket(bucket).max_keys(10);
-
-            if let Some(ref token) = continuation_token {
-                req = req.continuation_token(token);
-            }
-
-            let resp = req.send().await.expect("list objects page");
-
-            for obj in resp.contents() {
-                all_keys.push(obj.key().unwrap_or("").to_string());
-            }
-
-            if !resp.is_truncated().unwrap_or(false) {
-                break;
-            }
-            continuation_token = resp.next_continuation_token().map(|s| s.to_string());
-        }
-
-        // Verify completeness and uniqueness
-        let unique_keys: HashSet<&str> = all_keys.iter().map(|s| s.as_str()).collect();
-
-        assert_eq!(
-            all_keys.len(),
-            100,
-            "RT-06 FAIL: expected 100 objects, listed {} (regression: walk_dir truncation)",
-            all_keys.len()
-        );
-        assert_eq!(
-            unique_keys.len(),
-            100,
-            "RT-06 FAIL: found {} unique keys but listed {} total (duplicates!)",
-            unique_keys.len(),
-            all_keys.len()
-        );
-
-        for key in &expected_keys {
-            assert!(
-                unique_keys.contains(key.as_str()),
-                "RT-06 FAIL: key '{key}' missing from listing (regression rustfs#4810)"
-            );
-        }
-
-        info!("RT-06 PASS: all 100 objects listed completely and uniquely");
-        Ok(())
-    }
-
-    /// RT-06b: Verify listing with prefix filter returns correct subset.
-    ///
-    /// Regression pattern: prefix filter returns empty or includes wrong keys
-    /// (rustfs#5051: empty results for shallow prefixes).
-    #[tokio::test]
-    #[serial]
-    async fn test_list_objects_v2_prefix_filter_correctness() -> TestResult {
-        init_logging();
-        info!("RT-06b: prefix filter correctness");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt06b-prefix-filter";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Upload objects with different prefixes
-        for i in 0..5 {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(format!("logs/app-{i:04}.log"))
-                .body(ByteStream::from_static(b"log data"))
-                .send()
-                .await
-                .expect("put log object");
-
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(format!("data/file-{i:04}.csv"))
-                .body(ByteStream::from_static(b"csv data"))
-                .send()
-                .await
-                .expect("put data object");
-        }
-
-        // List with prefix "logs/" — should return exactly 5
-        let resp = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .prefix("logs/")
-            .send()
-            .await
-            .expect("list with prefix");
-
-        assert_eq!(
-            resp.contents().len(),
-            5,
-            "RT-06b FAIL: expected 5 objects with prefix 'logs/', found {} (regression rustfs#5051)",
-            resp.contents().len()
-        );
-
-        for obj in resp.contents() {
-            assert!(
-                obj.key().unwrap_or("").starts_with("logs/"),
-                "RT-06b FAIL: object '{}' does not match prefix 'logs/'",
-                obj.key().unwrap_or("?")
-            );
-        }
-
-        // List with prefix "data/" — should return exactly 5
-        let resp = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .prefix("data/")
-            .send()
-            .await
-            .expect("list with data/ prefix");
-
-        assert_eq!(
-            resp.contents().len(),
-            5,
-            "RT-06b FAIL: expected 5 objects with prefix 'data/', found {}",
-            resp.contents().len()
-        );
-
-        // List with prefix "nonexistent/" — should return 0
-        let resp = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .prefix("nonexistent/")
-            .send()
-            .await
-            .expect("list with nonexistent prefix");
-
-        assert!(
-            resp.contents().is_empty(),
-            "RT-06b FAIL: expected 0 objects with prefix 'nonexistent/', found {}",
-            resp.contents().len()
-        );
-
-        info!("RT-06b PASS: prefix filter returns correct subset");
-        Ok(())
-    }
-
-    /// RT-06c: Verify listing with delimiter and CommonPrefixes.
-    ///
-    /// Regression pattern: delimiter handling produces incorrect CommonPrefixes
-    /// or misses objects at the delimiter boundary.
-    #[tokio::test]
-    #[serial]
-    async fn test_list_objects_v2_delimiter_common_prefixes() -> TestResult {
-        init_logging();
-        info!("RT-06c: delimiter and CommonPrefixes");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt06c-delimiter";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Create a hierarchical structure
-        let keys = vec!["a.txt", "dir1/b.txt", "dir1/sub1/c.txt", "dir1/sub2/d.txt", "dir2/e.txt"];
-
-        for key in &keys {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(*key)
-                .body(ByteStream::from_static(b"content"))
-                .send()
-                .await
-                .expect("put object");
-        }
-
-        // List with delimiter "/" at root level
-        let resp = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .delimiter("/")
-            .send()
-            .await
-            .expect("list with delimiter");
-
-        // Should have 1 object (a.txt) and 2 common prefixes (dir1/, dir2/)
-        let contents: Vec<_> = resp.contents().iter().map(|o| o.key().unwrap_or("")).collect();
-        let prefixes: Vec<_> = resp.common_prefixes().iter().map(|p| p.prefix().unwrap_or("")).collect();
-
-        assert!(contents.contains(&"a.txt"), "RT-06c FAIL: root object 'a.txt' missing from listing");
-        assert_eq!(contents.len(), 1, "RT-06c FAIL: expected 1 root-level object, found {}", contents.len());
-        assert_eq!(prefixes.len(), 2, "RT-06c FAIL: expected 2 common prefixes, found {:?}", prefixes);
-        assert!(prefixes.contains(&"dir1/"), "RT-06c FAIL: 'dir1/' missing from CommonPrefixes");
-        assert!(prefixes.contains(&"dir2/"), "RT-06c FAIL: 'dir2/' missing from CommonPrefixes");
-
-        info!("RT-06c PASS: delimiter and CommonPrefixes correct");
-        Ok(())
-    }
-
-    /// RT-06d: Verify listing returns correct IsTruncated flag.
-    ///
-    /// Regression pattern: IsTruncated=false when there are more objects
-    /// (rustfs#4810: walk_dir timeout truncation with false IsTruncated).
-    #[tokio::test]
-    #[serial]
-    async fn test_list_objects_v2_is_truncated_correctness() -> TestResult {
-        init_logging();
-        info!("RT-06d: IsTruncated correctness");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt06d-truncated";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Upload 15 objects
-        for i in 0..15 {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(format!("item-{i:04}.txt"))
-                .body(ByteStream::from_static(b"data"))
-                .send()
-                .await
-                .expect("put object");
-        }
-
-        // List with max_keys=5 — should be truncated
-        let resp = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .max_keys(5)
-            .send()
-            .await
-            .expect("list with max_keys=5");
-
-        assert!(
-            resp.is_truncated().unwrap_or(false),
-            "RT-06d FAIL: IsTruncated should be true with 15 objects and max_keys=5"
-        );
-        assert_eq!(resp.contents().len(), 5, "RT-06d FAIL: expected 5 objects in first page");
-        assert!(
-            resp.next_continuation_token().is_some(),
-            "RT-06d FAIL: NextContinuationToken should be present when truncated"
-        );
-
-        // List with max_keys=100 — should NOT be truncated
-        let resp = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .max_keys(100)
-            .send()
-            .await
-            .expect("list with max_keys=100");
-
-        assert!(
-            !resp.is_truncated().unwrap_or(false),
-            "RT-06d FAIL: IsTruncated should be false with 15 objects and max_keys=100"
-        );
-        assert_eq!(resp.contents().len(), 15, "RT-06d FAIL: expected 15 objects with max_keys=100");
-
-        info!("RT-06d PASS: IsTruncated flag is correct");
-        Ok(())
-    }
-}

crates/e2e_test/src/multipart_auth_test.rs

@@ -62,33 +62,6 @@ fn md5_hex(input: impl AsRef<[u8]>) -> String {
     hex::encode(hasher.finalize())
 }
 
-async fn create_restricted_user(
-    env: &RustFSTestEnvironment,
-    username: &str,
-    secret_key: &str,
-) -> Result<(), Box<dyn Error + Send + Sync>> {
-    let url = format!("{}/rustfs/admin/v3/add-user?accessKey={username}", env.url);
-    let body = serde_json::json!({
-        "secretKey": secret_key,
-        "status": "enabled"
-    })
-    .to_string();
-    crate::common::awscurl_put(&url, &body, &env.access_key, &env.secret_key).await?;
-    Ok(())
-}
-
-fn restricted_user_client(env: &RustFSTestEnvironment, username: &str, secret_key: &str) -> aws_sdk_s3::Client {
-    let credentials = aws_sdk_s3::config::Credentials::new(username, secret_key, None, None, "snowball-pax-auth-test");
-    let config = aws_sdk_s3::Config::builder()
-        .credentials_provider(credentials)
-        .region(aws_sdk_s3::config::Region::new("us-east-1"))
-        .endpoint_url(&env.url)
-        .force_path_style(true)
-        .behavior_version_latest()
-        .build();
-    aws_sdk_s3::Client::from_conf(config)
-}
-
 /// Env var consumed by the local SSE-S3 DEK provider when KMS is not configured.
 ///
 /// Since rustfs#3564 the server fails closed on managed SSE (SSE-S3 or
@@ -3584,8 +3557,8 @@ async fn test_anonymous_post_object_rejects_expires_field_missing_from_policy_co
 
 #[tokio::test]
 #[serial]
-async fn test_anonymous_post_object_rejects_object_lock_retention_without_permission()
--> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+async fn test_anonymous_post_object_accepts_object_lock_retention_fields() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
+{
     init_logging();
 
     let mut env = RustFSTestEnvironment::new().await?;
@@ -3594,6 +3567,8 @@ async fn test_anonymous_post_object_rejects_object_lock_retention_without_permis
     let bucket = "anon-post-policy-object-lock-retention";
     let object_key = "uploads/object-lock-retention.txt";
     let retain_until = "2037-10-21T07:28:00Z";
+    let expected_body = b"post-policy-object-lock-retention-body".to_vec();
+
     let admin_client = env.create_s3_client();
     admin_client
         .create_bucket()
@@ -3618,7 +3593,7 @@ async fn test_anonymous_post_object_rejects_object_lock_retention_without_permis
         .text("x-amz-object-lock-retain-until-date", retain_until)
         .part(
             "file",
-            reqwest::multipart::Part::bytes(b"post-policy-object-lock-retention-body".to_vec())
+            reqwest::multipart::Part::bytes(expected_body.clone())
                 .file_name("upload.txt")
                 .mime_str("text/plain")?,
         );
@@ -3632,8 +3607,26 @@ async fn test_anonymous_post_object_rejects_object_lock_retention_without_permis
     let status = post_resp.status();
     let response_body = post_resp.text().await?;
 
-    assert_eq!(status, reqwest::StatusCode::FORBIDDEN);
-    assert!(response_body.contains("AccessDenied"));
+    assert_eq!(status, reqwest::StatusCode::NO_CONTENT);
+    assert!(response_body.is_empty(), "204 response should not contain a body, got: {response_body}");
+
+    let retention = admin_client
+        .get_object_retention()
+        .bucket(bucket)
+        .key(object_key)
+        .send()
+        .await?;
+    let retention = retention.retention().expect("retention should be present");
+    assert_eq!(retention.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    let retain_until_out = retention
+        .retain_until_date()
+        .expect("retain_until_date should be present")
+        .fmt(aws_sdk_s3::primitives::DateTimeFormat::DateTime)?;
+    assert_eq!(retain_until_out, retain_until);
+
+    let get_out = admin_client.get_object().bucket(bucket).key(object_key).send().await?;
+    let uploaded = get_out.body.collect().await?.into_bytes();
+    assert_eq!(uploaded.as_ref(), expected_body.as_slice());
 
     Ok(())
 }
@@ -3822,8 +3815,8 @@ async fn test_anonymous_post_object_rejects_object_lock_retention_missing_from_p
 
 #[tokio::test]
 #[serial]
-async fn test_anonymous_post_object_rejects_object_lock_legal_hold_without_permission()
--> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+async fn test_anonymous_post_object_accepts_object_lock_legal_hold_field() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
+{
     init_logging();
 
     let mut env = RustFSTestEnvironment::new().await?;
@@ -3831,6 +3824,8 @@ async fn test_anonymous_post_object_rejects_object_lock_legal_hold_without_permi
 
     let bucket = "anon-post-policy-object-lock-legal-hold";
     let object_key = "uploads/object-lock-legal-hold.txt";
+    let expected_body = b"post-policy-object-lock-legal-hold-body".to_vec();
+
     let admin_client = env.create_s3_client();
     admin_client
         .create_bucket()
@@ -3853,7 +3848,7 @@ async fn test_anonymous_post_object_rejects_object_lock_legal_hold_without_permi
         .text("x-amz-object-lock-legal-hold", "ON")
         .part(
             "file",
-            reqwest::multipart::Part::bytes(b"post-policy-object-lock-legal-hold-body".to_vec())
+            reqwest::multipart::Part::bytes(expected_body.clone())
                 .file_name("upload.txt")
                 .mime_str("text/plain")?,
         );
@@ -3867,8 +3862,26 @@ async fn test_anonymous_post_object_rejects_object_lock_legal_hold_without_permi
     let status = post_resp.status();
     let response_body = post_resp.text().await?;
 
-    assert_eq!(status, reqwest::StatusCode::FORBIDDEN);
-    assert!(response_body.contains("AccessDenied"));
+    assert_eq!(status, reqwest::StatusCode::NO_CONTENT);
+    assert!(response_body.is_empty(), "204 response should not contain a body, got: {response_body}");
+
+    let legal_hold = admin_client
+        .get_object_legal_hold()
+        .bucket(bucket)
+        .key(object_key)
+        .send()
+        .await?;
+    assert_eq!(
+        legal_hold
+            .legal_hold()
+            .and_then(|value| value.status())
+            .map(|value| value.as_str()),
+        Some("ON")
+    );
+
+    let get_out = admin_client.get_object().bucket(bucket).key(object_key).send().await?;
+    let uploaded = get_out.body.collect().await?.into_bytes();
+    assert_eq!(uploaded.as_ref(), expected_body.as_slice());
 
     Ok(())
 }
@@ -5645,70 +5658,6 @@ async fn test_signed_put_object_extract_preserves_object_lock_retention() -> Res
     Ok(())
 }
 
-#[tokio::test]
-#[serial]
-async fn test_signed_put_object_extract_pax_retention_overrides_request_retention()
--> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let bucket = "signed-extract-pax-retention-precedence";
-    let archive_key = "retention.tar";
-    let extracted_key = "alpha.txt";
-    let request_retain_until = aws_sdk_s3::primitives::DateTime::from_secs(2_114_380_800);
-    let pax_retain_until = "2040-01-01T00:00:00Z";
-
-    let client = env.create_s3_client();
-    client
-        .create_bucket()
-        .bucket(bucket)
-        .object_lock_enabled_for_bucket(true)
-        .send()
-        .await?;
-
-    let pax = HashMap::from([
-        ("minio.metadata.x-amz-object-lock-mode", "COMPLIANCE".to_string()),
-        ("minio.metadata.x-amz-object-lock-retain-until-date", pax_retain_until.to_string()),
-    ]);
-    let archive = make_tar_with_pax_entry(extracted_key, b"alpha-body", None, &pax).await;
-
-    client
-        .put_object()
-        .bucket(bucket)
-        .key(archive_key)
-        .object_lock_mode(aws_sdk_s3::types::ObjectLockMode::Governance)
-        .object_lock_retain_until_date(request_retain_until)
-        .body(ByteStream::from(archive))
-        .customize()
-        .mutate_request(|req| {
-            req.headers_mut().insert("x-amz-meta-snowball-auto-extract", "true");
-        })
-        .send()
-        .await?;
-
-    let retention = client
-        .get_object_retention()
-        .bucket(bucket)
-        .key(extracted_key)
-        .send()
-        .await?
-        .retention()
-        .expect("retention should be present")
-        .clone();
-    assert_eq!(retention.mode().map(|value| value.as_str()), Some("COMPLIANCE"));
-    assert_eq!(
-        retention
-            .retain_until_date()
-            .expect("retain_until_date should be present")
-            .fmt(aws_sdk_s3::primitives::DateTimeFormat::DateTime)?,
-        pax_retain_until
-    );
-
-    Ok(())
-}
-
 #[tokio::test]
 #[serial]
 async fn test_signed_put_object_extract_returns_archive_etag() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
@@ -5833,282 +5782,6 @@ async fn test_signed_put_object_extract_preserves_pax_metadata_and_version_id()
     Ok(())
 }
 
-#[tokio::test]
-#[serial]
-async fn test_signed_put_object_extract_authorizes_each_pax_privilege_and_retention_conditions()
--> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-    if !crate::common::awscurl_available() {
-        return Ok(());
-    }
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let bucket = "signed-extract-pax-auth";
-    let put_only_user = "snowball-put-only";
-    let put_only_secret = "snowball-put-only-secret";
-    let conditional_user = "snowball-retention-condition";
-    let conditional_secret = "snowball-retention-condition-secret";
-    let wrong_action_user = "snowball-wrong-action";
-    let wrong_action_secret = "snowball-wrong-action-secret";
-    let version_condition_user = "snowball-version-condition";
-    let version_condition_secret = "snowball-version-condition-secret";
-    let pax_context_user = "snowball-pax-context";
-    let pax_context_secret = "snowball-pax-context-secret";
-    let conditional_version_id = Uuid::new_v4().to_string();
-    let admin_client = env.create_s3_client();
-    admin_client
-        .create_bucket()
-        .bucket(bucket)
-        .object_lock_enabled_for_bucket(true)
-        .send()
-        .await?;
-    create_restricted_user(&env, put_only_user, put_only_secret).await?;
-    create_restricted_user(&env, conditional_user, conditional_secret).await?;
-    create_restricted_user(&env, wrong_action_user, wrong_action_secret).await?;
-    create_restricted_user(&env, version_condition_user, version_condition_secret).await?;
-    create_restricted_user(&env, pax_context_user, pax_context_secret).await?;
-
-    let object_resource = format!("arn:aws:s3:::{bucket}/*");
-    let context_archive_resources = [
-        format!("arn:aws:s3:::{bucket}/tag-context.tar"),
-        format!("arn:aws:s3:::{bucket}/lock-context.tar"),
-    ];
-    let tag_entry_resource = format!("arn:aws:s3:::{bucket}/tag-context-entry.txt");
-    let lock_entry_resource = format!("arn:aws:s3:::{bucket}/lock-context-entry.txt");
-    let policy = serde_json::json!({
-        "Version": "2012-10-17",
-        "Statement": [
-            {
-                "Sid": "PutOnly",
-                "Effect": "Allow",
-                "Principal": { "AWS": [put_only_user] },
-                "Action": ["s3:PutObject"],
-                "Resource": [object_resource.clone()]
-            },
-            {
-                "Sid": "RetentionWithLimit",
-                "Effect": "Allow",
-                "Principal": { "AWS": [conditional_user] },
-                "Action": ["s3:PutObject", "s3:PutObjectRetention"],
-                "Resource": [object_resource.clone()]
-            },
-            {
-                "Sid": "DenyRetentionBeyondCutoff",
-                "Effect": "Deny",
-                "Principal": { "AWS": [conditional_user] },
-                "Action": ["s3:PutObject"],
-                "Resource": [object_resource.clone()],
-                "Condition": {
-                    "DateGreaterThan": {
-                        "s3:object-lock-retain-until-date": "2030-01-01T00:00:00Z"
-                    }
-                }
-            },
-            {
-                "Sid": "WrongAdditionalAction",
-                "Effect": "Allow",
-                "Principal": { "AWS": [wrong_action_user] },
-                "Action": ["s3:PutObject", "s3:PutObjectLegalHold"],
-                "Resource": [object_resource.clone()]
-            },
-            {
-                "Sid": "VersionConditionPut",
-                "Effect": "Allow",
-                "Principal": { "AWS": [version_condition_user] },
-                "Action": ["s3:PutObject"],
-                "Resource": [object_resource.clone()]
-            },
-            {
-                "Sid": "VersionConditionReplicate",
-                "Effect": "Allow",
-                "Principal": { "AWS": [version_condition_user] },
-                "Action": ["s3:ReplicateObject"],
-                "Resource": [object_resource],
-                "Condition": {
-                    "StringEquals": {
-                        "s3:VersionId": conditional_version_id.clone()
-                    }
-                }
-            },
-            {
-                "Sid": "PaxContextArchives",
-                "Effect": "Allow",
-                "Principal": { "AWS": [pax_context_user] },
-                "Action": ["s3:PutObject", "s3:PutObjectRetention", "s3:PutObjectTagging"],
-                "Resource": context_archive_resources
-            },
-            {
-                "Sid": "PaxTagContextPut",
-                "Effect": "Allow",
-                "Principal": { "AWS": [pax_context_user] },
-                "Action": ["s3:PutObject"],
-                "Resource": [tag_entry_resource.clone()],
-                "Condition": {
-                    "StringEquals": {
-                        "s3:RequestObjectTag/classification": "public"
-                    }
-                }
-            },
-            {
-                "Sid": "PaxTagContextAction",
-                "Effect": "Allow",
-                "Principal": { "AWS": [pax_context_user] },
-                "Action": ["s3:PutObjectTagging"],
-                "Resource": [tag_entry_resource]
-            },
-            {
-                "Sid": "PaxLockContextPut",
-                "Effect": "Allow",
-                "Principal": { "AWS": [pax_context_user] },
-                "Action": ["s3:PutObject"],
-                "Resource": [lock_entry_resource.clone()],
-                "Condition": {
-                    "StringEquals": {
-                        "s3:object-lock-mode": "COMPLIANCE"
-                    }
-                }
-            },
-            {
-                "Sid": "PaxLockContextAction",
-                "Effect": "Allow",
-                "Principal": { "AWS": [pax_context_user] },
-                "Action": ["s3:PutObjectRetention"],
-                "Resource": [lock_entry_resource]
-            }
-        ]
-    })
-    .to_string();
-    admin_client.put_bucket_policy().bucket(bucket).policy(policy).send().await?;
-
-    let put_only_client = restricted_user_client(&env, put_only_user, put_only_secret);
-    let conditional_client = restricted_user_client(&env, conditional_user, conditional_secret);
-    let wrong_action_client = restricted_user_client(&env, wrong_action_user, wrong_action_secret);
-    let cases = [
-        (
-            "legal-hold.tar",
-            put_only_client,
-            HashMap::from([("minio.metadata.x-amz-object-lock-legal-hold", "ON".to_string())]),
-        ),
-        (
-            "retention-condition.tar",
-            conditional_client,
-            HashMap::from([
-                ("minio.metadata.x-amz-object-lock-mode", "COMPLIANCE".to_string()),
-                ("minio.metadata.x-amz-object-lock-retain-until-date", "2099-01-01T00:00:00Z".to_string()),
-            ]),
-        ),
-        (
-            "version-id.tar",
-            wrong_action_client,
-            HashMap::from([("minio.versionId", Uuid::new_v4().to_string())]),
-        ),
-    ];
-
-    for (archive_key, client, pax) in cases {
-        let archive = make_tar_with_pax_entry("entry.txt", b"must-not-write", None, &pax).await;
-        let err = client
-            .put_object()
-            .bucket(bucket)
-            .key(archive_key)
-            .body(ByteStream::from(archive))
-            .customize()
-            .mutate_request(|req| {
-                req.headers_mut().insert("x-amz-meta-snowball-auto-extract", "true");
-            })
-            .send()
-            .await
-            .expect_err("missing, conditional, or wrong PAX privilege must be rejected");
-        assert_eq!(
-            err.as_service_error().and_then(|error| error.meta().code()),
-            Some("AccessDenied"),
-            "{archive_key}"
-        );
-    }
-
-    let version_condition_client = restricted_user_client(&env, version_condition_user, version_condition_secret);
-    let matching_version_pax = HashMap::from([("minio.versionId", conditional_version_id)]);
-    let archive = make_tar_with_pax_entry("condition-entry.txt", b"condition-body", None, &matching_version_pax).await;
-    version_condition_client
-        .put_object()
-        .bucket(bucket)
-        .key("version-condition.tar")
-        .body(ByteStream::from(archive))
-        .customize()
-        .mutate_request(|req| {
-            req.headers_mut().insert("x-amz-meta-snowball-auto-extract", "true");
-        })
-        .send()
-        .await?;
-
-    let pax_context_client = restricted_user_client(&env, pax_context_user, pax_context_secret);
-    let tag_pax = HashMap::from([("minio.metadata.x-amz-tagging", "classification=public".to_string())]);
-    let archive = make_tar_with_pax_entry("tag-context-entry.txt", b"tag-context-body", None, &tag_pax).await;
-    pax_context_client
-        .put_object()
-        .bucket(bucket)
-        .key("tag-context.tar")
-        .tagging("classification=restricted")
-        .body(ByteStream::from(archive))
-        .customize()
-        .mutate_request(|req| {
-            req.headers_mut().insert("x-amz-meta-snowball-auto-extract", "true");
-        })
-        .send()
-        .await?;
-    let tags = admin_client
-        .get_object_tagging()
-        .bucket(bucket)
-        .key("tag-context-entry.txt")
-        .send()
-        .await?;
-    assert!(
-        tags.tag_set()
-            .iter()
-            .any(|tag| tag.key() == "classification" && tag.value() == "public")
-    );
-
-    let pax_retain_until = "2040-01-01T00:00:00Z";
-    let lock_pax = HashMap::from([
-        ("minio.metadata.x-amz-object-lock-mode", "COMPLIANCE".to_string()),
-        ("minio.metadata.x-amz-object-lock-retain-until-date", pax_retain_until.to_string()),
-    ]);
-    let archive = make_tar_with_pax_entry("lock-context-entry.txt", b"lock-context-body", None, &lock_pax).await;
-    pax_context_client
-        .put_object()
-        .bucket(bucket)
-        .key("lock-context.tar")
-        .object_lock_mode(aws_sdk_s3::types::ObjectLockMode::Governance)
-        .object_lock_retain_until_date(aws_sdk_s3::primitives::DateTime::from_secs(2_114_380_800))
-        .body(ByteStream::from(archive))
-        .customize()
-        .mutate_request(|req| {
-            req.headers_mut().insert("x-amz-meta-snowball-auto-extract", "true");
-        })
-        .send()
-        .await?;
-    let retention = admin_client
-        .get_object_retention()
-        .bucket(bucket)
-        .key("lock-context-entry.txt")
-        .send()
-        .await?
-        .retention()
-        .expect("PAX retention should be present")
-        .clone();
-    assert_eq!(retention.mode().map(|mode| mode.as_str()), Some("COMPLIANCE"));
-    assert_eq!(
-        retention
-            .retain_until_date()
-            .expect("PAX retain-until should be present")
-            .fmt(aws_sdk_s3::primitives::DateTimeFormat::DateTime)?,
-        pax_retain_until
-    );
-
-    Ok(())
-}
-
 #[tokio::test]
 #[serial]
 async fn test_signed_put_object_extract_accepts_compat_header() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {

crates/e2e_test/src/notification_startup_regression_test.rs

@@ -1,153 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression tests for the event notification startup race.
-//!
-//! Covers the recurring pattern where webhook/audit targets fail to load at boot
-//! due to startup ordering (notification runtime starts before server config is
-//! loaded). This has regressed 9+ times across beta.3 ~ beta.12.
-//!
-//! ## Regression Issues
-//!
-//! - rustfs#5387: webhook notifications broken again in beta.9+
-//! - rustfs#5681: Audit webhook targets are not loaded at boot
-//! - rustfs#5401: Event Destinations broken again
-//! - rustfs#5183: Audit webhooks stay offline after restart
-//! - rustfs#5115: init_event_notifier loses startup race against server config load
-//! - rustfs#4796: Pulsar event destinations offline after restart
-//! - rustfs#5428: MQTT bucket notifications stop on restarted cluster node
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestEnvironment, init_logging};
-    use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
-    use serial_test::serial;
-    use std::error::Error;
-    use tracing::info;
-
-    type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-    /// RT-01: Verify that the notification runtime initializes correctly at boot.
-    ///
-    /// Regression pattern: notification runtime initializes before server config
-    /// is fully loaded, causing webhook targets to never come online.
-    ///
-    /// This test verifies the startup ordering by checking that the server
-    /// starts successfully with notification enabled and can serve S3 requests.
-    /// A full webhook delivery test is in notification_webhook_test.rs.
-    #[tokio::test]
-    #[serial]
-    async fn test_notification_enabled_server_starts_cleanly() -> TestResult {
-        init_logging();
-        info!("RT-01: notification enabled server starts cleanly");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false"), ("RUSTFS_NOTIFY_ENABLE", "true")])
-            .await
-            .expect("start RustFS with notifications enabled");
-
-        let client = env.create_s3_client();
-        let bucket = "rt01-notify-startup";
-
-        // Server should be healthy and able to serve S3 requests
-        client
-            .create_bucket()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("create bucket with notifications enabled");
-
-        client
-            .put_object()
-            .bucket(bucket)
-            .key("test.txt")
-            .body(aws_sdk_s3::primitives::ByteStream::from_static(b"test"))
-            .send()
-            .await
-            .expect("put object with notifications enabled");
-
-        info!("RT-01 PASS: notification enabled server starts and serves S3");
-        Ok(())
-    }
-
-    /// RT-02: Verify notification config persists after server restart.
-    ///
-    /// Regression pattern: after a node restart, notification targets stay
-    /// offline permanently because the config is not re-loaded.
-    ///
-    /// Steps:
-    /// 1. Start server with notification enabled
-    /// 2. Create bucket and configure notification
-    /// 3. Restart server
-    /// 4. Verify notification config still exists
-    #[tokio::test]
-    #[serial]
-    async fn test_notification_config_survives_restart() -> TestResult {
-        init_logging();
-        info!("RT-02: notification config survives restart");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false"), ("RUSTFS_NOTIFY_ENABLE", "true")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt02-notify-restart";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Enable versioning (required for notification configuration)
-        client
-            .put_bucket_versioning()
-            .bucket(bucket)
-            .versioning_configuration(
-                VersioningConfiguration::builder()
-                    .status(BucketVersioningStatus::Enabled)
-                    .build(),
-            )
-            .send()
-            .await
-            .expect("enable versioning");
-
-        // Note: We can't fully test notification config persistence without a
-        // configured target. But we verify the server restarts cleanly with
-        // notification enabled, which is the core regression scenario.
-        env.restart_server_preserving_data(vec![], &[])
-            .await
-            .expect("restart RustFS with notifications enabled");
-
-        // Verify bucket still exists and is accessible after restart
-        let list = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("list objects after restart");
-
-        assert!(list.contents().is_empty(), "RT-02: bucket should be empty after restart");
-
-        // Verify we can still write objects (notification runtime initialized)
-        client
-            .put_object()
-            .bucket(bucket)
-            .key("after-restart.txt")
-            .body(aws_sdk_s3::primitives::ByteStream::from_static(b"post-restart"))
-            .send()
-            .await
-            .expect("put object after restart — notification runtime must be initialized");
-
-        info!("RT-02 PASS: server with notifications survives restart");
-        Ok(())
-    }
-}

crates/e2e_test/src/notification_webhook_test.rs

@@ -24,7 +24,7 @@
 //!   * PUT / multipart-complete / DeleteObject / DeleteObjects each deliver one event with the correct
 //!     eventName, bucket, key, versionId and eTag.
 //!   * prefix/suffix filters drop non-matching keys (rule-engine gate).
-//!   * an event queued while the target endpoint rejects delivery is redelivered
+//!   * an event queued while the target endpoint is unreachable is redelivered
 //!     from the on-disk store once the endpoint recovers (store-and-forward).
 //!   * responseElements and the S3 response use the canonical request ID while
 //!     requestParameters preserve a conflicting client-supplied value.
@@ -897,10 +897,11 @@ async fn test_webhook_event_delivery_and_filtering() -> TestResult {
     Ok(())
 }
 
-/// An event queued while the target endpoint rejects delivery survives on the
+/// An event queued while the target endpoint is unreachable survives on the
 /// durable store and is redelivered once the endpoint comes back.
 #[tokio::test]
 #[serial]
+#[ignore = "FAILING deterministically on main since it landed (#4821): the target is created but never appears in /rustfs/admin/v3/target/arns, so wait_for_target_registered times out. Quarantined per the flake policy; remove with the fix for rustfs#4852"]
 async fn test_webhook_redelivers_event_after_target_recovers() -> TestResult {
     init_logging();
 
@@ -931,55 +932,28 @@ async fn test_webhook_redelivers_event_after_target_recovers() -> TestResult {
     wait_for_target_registered(&env, target).await?;
     put_notification_config(&client, bucket, target, "uploads/", ".dat").await?;
 
-    // Replace the healthy setup listener with one that rejects the first POST.
-    // Waiting for that response below proves the queued event reached a failed
-    // delivery attempt before the endpoint recovers.
+    // Take the endpoint down (drops the listener, so connections are refused —
+    // a retryable NotConnected), then PUT: the event cannot be delivered and
+    // must survive on the durable queue store.
     setup_handle.abort();
     let _ = setup_handle.await;
 
-    let listener = TcpListener::bind(("0.0.0.0", port)).await?;
     let key = "uploads/redeliver.dat";
     client
         .put_object()
         .bucket(bucket)
         .key(key)
-        .body(ByteStream::from_static(b"queued while target rejects"))
+        .body(ByteStream::from_static(b"queued while target down"))
         .send()
         .await?;
 
-    let mut failure_handle = tokio::spawn(async move {
-        loop {
-            let (mut stream, _) = listener.accept().await?;
-            let (method, _) = timeout(Duration::from_secs(5), read_http_message(&mut stream)).await??;
-            if method == "HEAD" {
-                stream
-                    .write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 0\r\nconnection: close\r\n\r\n")
-                    .await?;
-                stream.shutdown().await?;
-                continue;
-            }
-            if method == "POST" {
-                stream
-                    .write_all(b"HTTP/1.1 503 Service Unavailable\r\ncontent-length: 0\r\nconnection: close\r\n\r\n")
-                    .await?;
-                stream.shutdown().await?;
-                return Ok::<(), BoxError>(());
-            }
-        }
-    });
+    // Hold the endpoint down long enough for at least one replay attempt to
+    // fail (the replay worker scans the store every 500ms), so recovery below
+    // exercises real redelivery rather than a first-attempt success.
+    tokio::time::sleep(Duration::from_secs(2)).await;
 
-    let rejected = match timeout(Duration::from_secs(20), &mut failure_handle).await {
-        Ok(rejected) => rejected,
-        Err(_) => {
-            failure_handle.abort();
-            let _ = failure_handle.await;
-            return Err("webhook replay did not reach the rejecting endpoint".into());
-        }
-    };
-    rejected??;
-
-    // Bring the endpoint back on the same port; the replay worker rescans the
-    // durable queue and delivers the retained event.
+    // Bring the endpoint back on the same port; the replay worker retries with
+    // exponential backoff and delivers the queued event.
     let listener = TcpListener::bind(("0.0.0.0", port)).await?;
     let (tx, mut rx) = mpsc::unbounded_channel();
     let handle = serve_event_collector(listener, tx);

crates/e2e_test/src/object_lock/object_lock_test.rs

@@ -26,7 +26,6 @@
 
 use super::common::*;
 use aws_sdk_s3::Client;
-use aws_sdk_s3::error::ProvideErrorMetadata;
 use aws_sdk_s3::primitives::{ByteStream, DateTimeFormat};
 use aws_sdk_s3::types::{
     CompletedMultipartUpload, CompletedPart, Delete, MetadataDirective, ObjectIdentifier, ObjectLockLegalHoldStatus,
@@ -2121,127 +2120,6 @@ async fn test_multipart_default_retention_fixed_at_create() {
 // Versioning Auto-Enable Tests
 // ============================================================================
 
-#[tokio::test]
-#[serial]
-async fn test_unretained_object_lock_object_delete_and_bucket_cleanup() {
-    init_logging();
-    info!("🧪 Test: Unretained Object Lock object delete and bucket cleanup (Issue #5339)");
-
-    let mut env = ObjectLockTestEnvironment::new()
-        .await
-        .expect("failed to create Object Lock test environment");
-    env.start_rustfs().await.expect("failed to start RustFS");
-
-    let bucket = "test-object-lock-delete-cleanup";
-    let key = "unretained-object";
-
-    env.create_object_lock_bucket(bucket)
-        .await
-        .expect("failed to create Object Lock bucket");
-    let client = env.s3_client();
-
-    let put_response = client
-        .put_object()
-        .bucket(bucket)
-        .key(key)
-        .body(ByteStream::from_static(b"unretained data"))
-        .send()
-        .await
-        .expect("failed to upload unretained object");
-    let object_version_id = put_response
-        .version_id()
-        .expect("Object Lock buckets must create versioned objects")
-        .to_string();
-
-    let delete_response = client
-        .delete_object()
-        .bucket(bucket)
-        .key(key)
-        .send()
-        .await
-        .expect("failed to create delete marker");
-    assert_eq!(delete_response.delete_marker(), Some(true));
-    let delete_marker_version_id = delete_response
-        .version_id()
-        .expect("Deleting without a version ID must create a delete marker")
-        .to_string();
-
-    let get_error = client
-        .get_object()
-        .bucket(bucket)
-        .key(key)
-        .send()
-        .await
-        .expect_err("GET must not return an object hidden by a delete marker");
-    assert_eq!(get_error.raw_response().map(|response| response.status().as_u16()), Some(404));
-    assert_eq!(get_error.as_service_error().and_then(|error| error.code()), Some("NoSuchKey"));
-
-    let listed_objects = client
-        .list_objects_v2()
-        .bucket(bucket)
-        .send()
-        .await
-        .expect("failed to list current objects");
-    assert!(
-        listed_objects.contents().iter().all(|object| object.key() != Some(key)),
-        "ListObjectsV2 must hide objects whose latest version is a delete marker"
-    );
-
-    let listed_versions = client
-        .list_object_versions()
-        .bucket(bucket)
-        .send()
-        .await
-        .expect("failed to list object versions");
-    assert!(
-        listed_versions
-            .versions()
-            .iter()
-            .any(|version| version.key() == Some(key) && version.version_id() == Some(object_version_id.as_str())),
-        "The data version must remain until it is explicitly deleted"
-    );
-    assert!(
-        listed_versions
-            .delete_markers()
-            .iter()
-            .any(|marker| marker.key() == Some(key) && marker.version_id() == Some(delete_marker_version_id.as_str())),
-        "ListObjectVersions must expose the delete marker"
-    );
-
-    client
-        .delete_object()
-        .bucket(bucket)
-        .key(key)
-        .version_id(object_version_id)
-        .send()
-        .await
-        .expect("failed to delete the data version");
-    client
-        .delete_object()
-        .bucket(bucket)
-        .key(key)
-        .version_id(delete_marker_version_id)
-        .send()
-        .await
-        .expect("failed to delete the delete marker");
-
-    let remaining_versions = client
-        .list_object_versions()
-        .bucket(bucket)
-        .send()
-        .await
-        .expect("failed to list versions after cleanup");
-    assert!(remaining_versions.versions().is_empty());
-    assert!(remaining_versions.delete_markers().is_empty());
-
-    client
-        .delete_bucket()
-        .bucket(bucket)
-        .send()
-        .await
-        .expect("Deleting every version must remove xl.meta so the bucket can be deleted normally");
-}
-
 #[tokio::test]
 #[serial]
 async fn test_versioning_auto_enabled_with_object_lock() {

crates/e2e_test/src/reliant/lock.rs

@@ -15,7 +15,9 @@
 
 use super::{grpc_lock_client::GrpcLockClient, grpc_lock_server::spawn_lock_server};
 use rustfs_lock::client::{LockClient, local::LocalClient};
-use rustfs_lock::{GlobalLockManager, LockInfo, LockRequest, LockResponse, LockStats, LockType, NamespaceLock, ObjectKey};
+use rustfs_lock::{
+    GlobalLockManager, LockError, LockInfo, LockRequest, LockResponse, LockStats, LockType, NamespaceLock, ObjectKey,
+};
 use std::sync::Arc;
 use std::time::Duration;
 
@@ -33,11 +35,7 @@ struct FailingClient;
 #[async_trait::async_trait]
 impl rustfs_lock::LockClient for FailingClient {
     async fn acquire_lock(&self, _request: &rustfs_lock::LockRequest) -> rustfs_lock::Result<LockResponse> {
-        // Match RemoteClient's transport-failure response so the coordinator can count this node toward quorum loss.
-        Ok(LockResponse::failure(
-            "Remote lock RPC failed: simulated gRPC node failure",
-            Duration::ZERO,
-        ))
+        Err(LockError::internal("simulated gRPC node failure"))
     }
 
     async fn release(&self, _lock_id: &rustfs_lock::LockId) -> rustfs_lock::Result<bool> {

crates/e2e_test/src/reliant/tiering.rs

@@ -382,6 +382,7 @@ struct ManualTransitionRunReport {
     skipped_delete_marker: u64,
     skipped_directory: u64,
     skipped_replication: u64,
+    skipped_already_transitioned: u64,
     skipped_already_in_flight: u64,
     skipped_queue_full: u64,
     skipped_queue_closed: u64,
@@ -407,6 +408,41 @@ fn assert_completed_or_in_flight_partial(state: &str, report: &ManualTransitionR
     }
 }
 
+fn assert_conflict_winner_report(state: &str, report: &ManualTransitionRunReport, expected_objects: u64, context: &str) {
+    assert_completed_or_in_flight_partial(state, report, context);
+    if report.skipped_already_in_flight > 0 {
+        assert!(
+            report.scanned <= expected_objects,
+            "{context}: scanned more objects than the conflict scope contains: {report:#?}"
+        );
+        assert!(
+            report.eligible <= expected_objects,
+            "{context}: marked more objects eligible than the conflict scope contains: {report:#?}"
+        );
+        assert_eq!(
+            report.enqueued + report.skipped_already_in_flight,
+            report.eligible,
+            "{context}: partial in-flight accounting must cover every eligible object: {report:#?}"
+        );
+    } else {
+        assert_eq!(report.scanned, expected_objects, "{context}: {report:#?}");
+        assert_eq!(
+            report.eligible + report.skipped_already_transitioned,
+            expected_objects,
+            "{context}: {report:#?}"
+        );
+        assert_eq!(
+            report.enqueued + report.skipped_already_in_flight,
+            expected_objects,
+            "{context}: {report:#?}"
+        );
+    }
+    assert_eq!(
+        report.transition_completed, report.enqueued,
+        "{context}: winner must wait for all queued transitions: {report:#?}"
+    );
+}
+
 #[derive(Debug, Deserialize)]
 struct ManualTransitionQueueSnapshot {
     queue_capacity: u64,
@@ -1240,15 +1276,8 @@ async fn test_manual_transition_async_scope_conflicts_report_active_job() -> Tes
     cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
 
     let mut hot = RustFSTestEnvironment::new().await?;
-    hot.start_rustfs_server_with_env(
-        vec![],
-        &[
-            ("RUSTFS_SCANNER_ENABLED", "false"),
-            ("RUSTFS_SCANNER_CYCLE", "3600"),
-            (MANUAL_TRANSITION_CANCEL_BARRIER_ENV, "1"),
-        ],
-    )
-    .await?;
+    hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
+        .await?;
     let hot_client = hot.create_s3_client();
     add_rustfs_tier(&hot, &cold).await?;
 
@@ -1318,21 +1347,20 @@ async fn test_manual_transition_async_scope_conflicts_report_active_job() -> Tes
     assert_eq!(conflict.cancel_endpoint, status_endpoint);
     assert!(!conflict.scope_key.is_empty());
 
-    manual_transition_job_cancel(&hot, cancel_endpoint).await?;
-
     let terminal = wait_for_manual_transition_job_terminal(&hot, status_endpoint, MANUAL_ASYNC_CONFLICT_TERMINAL_TIMEOUT).await?;
     assert_eq!(terminal.job_id, job_id);
-    assert_eq!(terminal.status, "cancelled", "terminal conflict winner response: {terminal:#?}");
     assert!(!terminal.report.dry_run);
     assert_eq!(terminal.report.bucket, MANUAL_ASYNC_CONFLICT_BUCKET);
     assert_eq!(terminal.report.prefix, accepted.report.prefix);
-    assert!(terminal.report.cancelled, "terminal conflict winner response: {terminal:#?}");
-    assert_eq!(terminal.report.scanned, 0, "terminal conflict winner response: {terminal:#?}");
-    assert_eq!(terminal.report.enqueued, 0, "terminal conflict winner response: {terminal:#?}");
-    assert_eq!(
-        terminal.report.transition_completed, 0,
-        "terminal conflict winner response: {terminal:#?}"
+    assert_conflict_winner_report(
+        &terminal.status,
+        &terminal.report,
+        MANUAL_ASYNC_CONFLICT_OBJECTS as u64,
+        "terminal conflict winner response",
     );
+    assert_eq!(terminal.report.dry_run_eligible, 0, "terminal conflict winner response: {terminal:#?}");
+    assert_eq!(terminal.report.transition_failed, 0, "terminal conflict winner response: {terminal:#?}");
+    assert_eq!(terminal.report.tier_failure, 0, "terminal conflict winner response: {terminal:#?}");
     let after_remote_count = cold_tier_object_count(&cold_client).await?;
     assert!(after_remote_count >= before_remote_count);
     assert!(after_remote_count <= before_remote_count + MANUAL_ASYNC_CONFLICT_OBJECTS);

crates/e2e_test/src/sts_query_compat_test.rs

@@ -12,35 +12,22 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::common::{RustFSTestEnvironment, admin_ok, init_logging};
+use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
 use aws_sdk_sts::config::retry::RetryConfig;
 use aws_sdk_sts::config::{Credentials, Region};
 use aws_sdk_sts::error::ProvideErrorMetadata;
 use aws_sdk_sts::operation::RequestId;
 use aws_sdk_sts::{Client, Config};
 use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
-use bytes::Bytes;
-use http::header::{AUTHORIZATION, CONTENT_TYPE};
-use http::{Request, Response};
-use http_body_util::{BodyExt, Full};
-use hyper::body::Incoming;
-use hyper::server::conn::http1;
-use hyper::service::service_fn;
-use hyper_util::rt::TokioIo;
-use serde_json::Value;
+use http::header::{CONTENT_TYPE, HOST};
+use rustfs_signer::constants::UNSIGNED_PAYLOAD;
+use rustfs_signer::sign_v4;
+use s3s::Body;
 use serial_test::serial;
-use std::collections::BTreeSet;
-use std::convert::Infallible;
 use std::error::Error;
-use std::sync::Arc;
-use tokio::net::TcpListener;
-use tokio::sync::{Notify, mpsc};
-use tokio::task::{JoinHandle, JoinSet};
-use tokio::time::{Duration, timeout};
 
 type BoxError = Box<dyn Error + Send + Sync>;
 type TestResult = Result<(), BoxError>;
-const OPA_AUTH_TOKEN: &str = "sts-opa-token";
 
 fn sts_client(url: &str, access_key: &str, secret_key: &str, session_token: Option<&str>) -> Client {
     let mut config = Config::builder()
@@ -62,14 +49,32 @@ fn sts_client(url: &str, access_key: &str, secret_key: &str, session_token: Opti
 }
 
 async fn create_root_service_account(env: &RustFSTestEnvironment) -> Result<(String, String), BoxError> {
-    let body = admin_ok(
-        env,
-        http::Method::PUT,
-        "/rustfs/admin/v3/add-service-accounts",
-        Some(serde_json::json!({ "targetUser": env.access_key.clone() }).to_string()),
-    )
-    .await?;
-    let response: Value = serde_json::from_str(&body)?;
+    let path = "/rustfs/admin/v3/add-service-accounts";
+    let url = format!("{}{path}", env.url);
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
+    let body = serde_json::json!({ "targetUser": env.access_key.clone() }).to_string();
+    let request = http::Request::builder()
+        .method(http::Method::PUT)
+        .uri(uri)
+        .header(HOST, authority)
+        .header(CONTENT_TYPE, "application/json")
+        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD)
+        .body(Body::empty())?;
+    let content_length = i64::try_from(body.len()).map_err(|_| "service account request body is too large")?;
+    let signed = sign_v4(request, content_length, &env.access_key, &env.secret_key, "", "us-east-1");
+    let mut request = local_http_client().put(&url);
+    for (name, value) in signed.headers() {
+        request = request.header(name, value);
+    }
+    let response = request.body(body).send().await?;
+    let status = response.status();
+    let body = response.text().await?;
+    if !status.is_success() {
+        return Err(format!("create service account failed: {status} {body}").into());
+    }
+
+    let response: serde_json::Value = serde_json::from_str(&body)?;
     let access_key = response["credentials"]["accessKey"]
         .as_str()
         .ok_or("service account response should contain credentials.accessKey")?
@@ -81,237 +86,28 @@ async fn create_root_service_account(env: &RustFSTestEnvironment) -> Result<(Str
     Ok((access_key, secret_key))
 }
 
-async fn create_user_with_policy(
-    env: &RustFSTestEnvironment,
-    user: &str,
-    secret: &str,
-    policy_name: &str,
-    statements: Value,
-) -> TestResult {
-    create_user(env, user, secret).await?;
-    admin_ok(
-        env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-canned-policy?name={policy_name}"),
-        Some(
-            serde_json::json!({
-                "Version": "2012-10-17",
-                "Statement": statements,
-            })
-            .to_string(),
-        ),
-    )
-    .await?;
-    admin_ok(
-        env,
-        http::Method::POST,
-        "/rustfs/admin/v3/idp/builtin/policy/attach",
-        Some(serde_json::json!({ "policies": [policy_name], "user": user }).to_string()),
-    )
-    .await?;
-    Ok(())
-}
-
-async fn create_user(env: &RustFSTestEnvironment, user: &str, secret: &str) -> TestResult {
-    admin_ok(
-        env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-user?accessKey={user}"),
-        Some(serde_json::json!({ "secretKey": secret, "status": "enabled" }).to_string()),
-    )
-    .await?;
-    Ok(())
-}
-
-async fn assert_access_denied(client: &Client, context: &str) -> TestResult {
+async fn assert_chaining_denied(client: &Client, credential_kind: &str) -> TestResult {
     let error = client
         .assume_role()
         .role_arn("arn:aws:iam::123456789012:role/test")
         .role_session_name("sts-query-compat-e2e")
         .send()
         .await
-        .expect_err("AssumeRole must be denied");
+        .expect_err("credential chaining must be denied");
     let service_error = error
         .as_service_error()
-        .ok_or_else(|| format!("{context} should deserialize as an STS service error: {error:?}"))?;
+        .ok_or_else(|| format!("{credential_kind} denial should deserialize as an STS service error: {error:?}"))?;
 
     assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(403));
     assert_eq!(service_error.code(), Some("AccessDenied"));
     assert_eq!(service_error.message(), Some("Access Denied"));
     assert!(
         error.request_id().is_some_and(|request_id| !request_id.is_empty()),
-        "{context} should include a request ID"
+        "{credential_kind} denial should include a request ID"
     );
     Ok(())
 }
 
-async fn handle_opa_request(
-    request: Request<Incoming>,
-    requests: mpsc::UnboundedSender<Value>,
-    validation_started: mpsc::UnboundedSender<()>,
-    validation_mode: OpaValidationMode,
-    expected_authorization: Option<String>,
-) -> Result<Response<Full<Bytes>>, Infallible> {
-    if let Some(expected_authorization) = expected_authorization
-        && request.headers().get(AUTHORIZATION).and_then(|value| value.to_str().ok()) != Some(expected_authorization.as_str())
-    {
-        return Ok(Response::builder()
-            .status(401)
-            .body(Full::new(Bytes::new()))
-            .expect("static OPA unauthorized response must be valid"));
-    }
-
-    let body = match request.into_body().collect().await {
-        Ok(body) => body.to_bytes(),
-        Err(error) => {
-            return Ok(Response::builder()
-                .status(400)
-                .body(Full::new(Bytes::from(error.to_string())))
-                .expect("static OPA error response must be valid"));
-        }
-    };
-
-    let payload = if body.is_empty() {
-        None
-    } else {
-        match serde_json::from_slice::<Value>(&body) {
-            Ok(payload) => Some(payload),
-            Err(error) => {
-                return Ok(Response::builder()
-                    .status(400)
-                    .body(Full::new(Bytes::from(error.to_string())))
-                    .expect("static OPA error response must be valid"));
-            }
-        }
-    };
-    if payload.is_none() {
-        let _ = validation_started.send(());
-        if let OpaValidationMode::DelayedUnavailable(release) = validation_mode {
-            release.notified().await;
-            return Ok(Response::builder()
-                .status(503)
-                .body(Full::new(Bytes::new()))
-                .expect("static OPA unavailable response must be valid"));
-        }
-    }
-    let allow = match payload.as_ref().and_then(|value| value.pointer("/input/identity/account")) {
-        Some(Value::String(account)) if account == "opaallow" => payload
-            .as_ref()
-            .and_then(|value| value.pointer("/input/context/deny_only"))
-            .and_then(Value::as_bool)
-            .unwrap_or(false),
-        Some(Value::String(account)) if account == "opadeny" => false,
-        None => true,
-        _ => false,
-    };
-    if let Some(payload) = payload {
-        let _ = requests.send(payload);
-    }
-    let body =
-        serde_json::to_vec(&serde_json::json!({ "result": { "allow": allow } })).expect("static OPA response must serialize");
-    Ok(Response::builder()
-        .header(CONTENT_TYPE, "application/json")
-        .body(Full::new(Bytes::from(body)))
-        .expect("static OPA response must be valid"))
-}
-
-#[derive(Clone)]
-enum OpaValidationMode {
-    Ready,
-    DelayedUnavailable(Arc<Notify>),
-}
-
-struct OpaMock {
-    url: String,
-    requests: mpsc::UnboundedReceiver<Value>,
-    validation_started: mpsc::UnboundedReceiver<()>,
-    validation_release: Option<Arc<Notify>>,
-    task: JoinHandle<()>,
-}
-
-impl OpaMock {
-    async fn start() -> Result<Self, BoxError> {
-        Self::start_with_mode(OpaValidationMode::Ready, Some(OPA_AUTH_TOKEN)).await
-    }
-
-    async fn start_delayed_unavailable() -> Result<Self, BoxError> {
-        let release = Arc::new(Notify::new());
-        Self::start_with_mode(OpaValidationMode::DelayedUnavailable(release), None).await
-    }
-
-    async fn start_with_mode(validation_mode: OpaValidationMode, auth_token: Option<&str>) -> Result<Self, BoxError> {
-        let listener = TcpListener::bind("127.0.0.1:0").await?;
-        let url = format!("http://{}/v1/data/rustfs/authz/allow", listener.local_addr()?);
-        let (requests_tx, requests) = mpsc::unbounded_channel();
-        let (validation_started_tx, validation_started) = mpsc::unbounded_channel();
-        let expected_authorization = auth_token.map(|token| format!("Bearer {token}"));
-        let validation_release = match &validation_mode {
-            OpaValidationMode::Ready => None,
-            OpaValidationMode::DelayedUnavailable(release) => Some(Arc::clone(release)),
-        };
-        let task = tokio::spawn(async move {
-            let mut connections = JoinSet::new();
-            loop {
-                tokio::select! {
-                    accepted = listener.accept() => {
-                        let Ok((stream, _)) = accepted else { break };
-                        let requests = requests_tx.clone();
-                        let validation_started = validation_started_tx.clone();
-                        let validation_mode = validation_mode.clone();
-                        let expected_authorization = expected_authorization.clone();
-                        connections.spawn(async move {
-                            let handler = service_fn(move |request| {
-                                handle_opa_request(
-                                    request,
-                                    requests.clone(),
-                                    validation_started.clone(),
-                                    validation_mode.clone(),
-                                    expected_authorization.clone(),
-                                )
-                            });
-                            let _ = http1::Builder::new()
-                                .serve_connection(TokioIo::new(stream), handler)
-                                .await;
-                        });
-                    }
-                    _ = connections.join_next(), if !connections.is_empty() => {}
-                }
-            }
-        });
-        Ok(Self {
-            url,
-            requests,
-            validation_started,
-            validation_release,
-            task,
-        })
-    }
-
-    async fn next_request(&mut self) -> Result<Value, BoxError> {
-        timeout(Duration::from_secs(5), self.requests.recv())
-            .await?
-            .ok_or_else(|| "OPA request channel closed".into())
-    }
-
-    async fn wait_for_validation(&mut self) -> TestResult {
-        timeout(Duration::from_secs(5), self.validation_started.recv())
-            .await?
-            .ok_or_else(|| "OPA validation channel closed".into())
-    }
-
-    fn release_validation(&self) {
-        if let Some(release) = &self.validation_release {
-            release.notify_one();
-        }
-    }
-}
-
-impl Drop for OpaMock {
-    fn drop(&mut self) {
-        self.task.abort();
-    }
-}
-
 #[tokio::test]
 #[serial]
 async fn test_sts_query_responses_are_aws_sdk_compatible() -> TestResult {
@@ -359,7 +155,7 @@ async fn test_sts_query_responses_are_aws_sdk_compatible() -> TestResult {
         "signature rejection should include a request ID"
     );
 
-    assert_access_denied(
+    assert_chaining_denied(
         &sts_client(
             &env.url,
             temporary.access_key_id(),
@@ -371,187 +167,7 @@ async fn test_sts_query_responses_are_aws_sdk_compatible() -> TestResult {
     .await?;
 
     let (service_access_key, service_secret_key) = create_root_service_account(&env).await?;
-    assert_access_denied(
-        &sts_client(&env.url, &service_access_key, &service_secret_key, None),
-        "service-account denial",
-    )
-    .await?;
-
-    let implicit_user = "stsimplicit";
-    let explicit_allow_user = "stsallow";
-    let explicit_deny_user = "stsdeny";
-    let policyless_user = "stspolicyless";
-    let secret = "stsAuthzSecret123";
-    create_user(&env, policyless_user, secret).await?;
-    assert_access_denied(&sts_client(&env.url, policyless_user, secret, None), "policyless user").await?;
-    create_user_with_policy(
-        &env,
-        implicit_user,
-        secret,
-        "sts-implicit-policy",
-        serde_json::json!([{
-            "Effect": "Allow",
-            "Action": ["s3:ListAllMyBuckets"],
-            "Resource": ["arn:aws:s3:::*"],
-        }]),
-    )
-    .await?;
-    create_user_with_policy(
-        &env,
-        explicit_allow_user,
-        secret,
-        "sts-allow-policy",
-        serde_json::json!([{
-            "Effect": "Allow",
-            "Action": ["sts:AssumeRole"],
-            "Resource": ["arn:aws:s3:::*"],
-        }]),
-    )
-    .await?;
-    create_user_with_policy(
-        &env,
-        explicit_deny_user,
-        secret,
-        "sts-deny-policy",
-        serde_json::json!([
-            {
-                "Effect": "Allow",
-                "Action": ["sts:AssumeRole"],
-                "Resource": ["arn:aws:s3:::*"],
-            },
-            {
-                "Effect": "Deny",
-                "Action": ["sts:AssumeRole"],
-                "Resource": ["arn:aws:s3:::*"],
-            }
-        ]),
-    )
-    .await?;
-
-    for user in [implicit_user, explicit_allow_user] {
-        let output = sts_client(&env.url, user, secret, None)
-            .assume_role()
-            .role_arn("arn:aws:iam::123456789012:role/test")
-            .role_session_name("sts-authz-e2e")
-            .send()
-            .await
-            .map_err(|error| format!("{user} should be allowed to call AssumeRole: {error:?}"))?;
-        let credentials = output
-            .credentials()
-            .ok_or_else(|| format!("{user} AssumeRole response should contain credentials"))?;
-        assert!(!credentials.access_key_id().is_empty());
-        assert!(!credentials.secret_access_key().is_empty());
-        assert!(!credentials.session_token().is_empty());
-    }
-    assert_access_denied(&sts_client(&env.url, explicit_deny_user, secret, None), "explicit sts:AssumeRole Deny").await?;
-
-    env.stop_server();
-    Ok(())
-}
-
-#[tokio::test]
-#[serial]
-async fn test_sts_assume_role_opa_contract() -> TestResult {
-    init_logging();
-
-    let mut opa = OpaMock::start().await?;
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server_with_env(
-        vec![],
-        &[
-            ("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str()),
-            ("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", OPA_AUTH_TOKEN),
-        ],
-    )
-    .await?;
-
-    let secret = "stsOpaSecret123";
-    create_user_with_policy(
-        &env,
-        "opaallow",
-        secret,
-        "sts-opa-local-deny-policy",
-        serde_json::json!([{
-            "Effect": "Deny",
-            "Action": ["sts:AssumeRole"],
-            "Resource": ["arn:aws:s3:::*"],
-        }]),
-    )
-    .await?;
-    create_user_with_policy(
-        &env,
-        "opadeny",
-        secret,
-        "sts-opa-local-allow-policy",
-        serde_json::json!([{
-            "Effect": "Allow",
-            "Action": ["sts:AssumeRole"],
-            "Resource": ["arn:aws:s3:::*"],
-        }]),
-    )
-    .await?;
-
-    sts_client(&env.url, "opaallow", secret, None)
-        .assume_role()
-        .role_arn("arn:aws:iam::123456789012:role/test")
-        .role_session_name("sts-opa-contract")
-        .send()
-        .await
-        .map_err(|error| format!("OPA allow should override the local explicit Deny: {error:?}"))?;
-    assert_access_denied(
-        &sts_client(&env.url, "opadeny", secret, None),
-        "OPA denial despite local sts:AssumeRole Allow",
-    )
-    .await?;
-
-    let mut accounts = BTreeSet::new();
-    for _ in 0..2 {
-        let request = opa.next_request().await?;
-        assert_eq!(request.pointer("/input/action").and_then(Value::as_str), Some("sts:AssumeRole"));
-        assert_eq!(request.pointer("/input/context/deny_only").and_then(Value::as_bool), Some(true));
-        let account = request
-            .pointer("/input/identity/account")
-            .and_then(Value::as_str)
-            .ok_or("OPA input should include identity.account")?;
-        accounts.insert(account.to_owned());
-    }
-    assert_eq!(accounts, BTreeSet::from(["opaallow".to_owned(), "opadeny".to_owned()]));
-
-    env.stop_server();
-    Ok(())
-}
-
-#[tokio::test]
-#[serial]
-async fn test_sts_assume_role_fails_closed_while_opa_is_unavailable() -> TestResult {
-    init_logging();
-
-    let mut opa = OpaMock::start_delayed_unavailable().await?;
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str())])
-        .await?;
-    opa.wait_for_validation().await?;
-
-    let user = "opaunavailable";
-    let secret = "stsOpaUnavailableSecret123";
-    create_user_with_policy(
-        &env,
-        user,
-        secret,
-        "sts-opa-unavailable-local-policy",
-        serde_json::json!([{
-            "Effect": "Allow",
-            "Action": ["s3:ListAllMyBuckets"],
-            "Resource": ["arn:aws:s3:::*"],
-        }]),
-    )
-    .await?;
-
-    assert_access_denied(&sts_client(&env.url, user, secret, None), "configured OPA initialization").await?;
-
-    opa.release_validation();
-    tokio::time::sleep(Duration::from_millis(200)).await;
-    assert_access_denied(&sts_client(&env.url, user, secret, None), "configured OPA validation failure").await?;
+    assert_chaining_denied(&sts_client(&env.url, &service_access_key, &service_secret_key, None), "service account").await?;
 
     env.stop_server();
     Ok(())

crates/e2e_test/src/tier_transition_regression_test.rs

@@ -1,172 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression tests for Tier/ILM transition operations.
-//!
-//! Covers the recurring pattern where tier transition fails silently, the
-//! free-version recovery task loops forever, or transitioned objects cannot
-//! be read back. This has regressed 6+ times.
-//!
-//! ## Regression Issues
-//!
-//! - rustfs#5218: Remote tier mutation commit failed
-//! - rustfs#5130: tier_free_version_recovery task loops forever
-//! - rustfs#5011: Idle tier free-version recovery rescans every 60 seconds
-//! - rustfs#4826: Full GET of multipart transitioned object fails
-//! - rustfs#5024: Some files succeeded in tier offloading, others failed
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestEnvironment, admin_ok, init_logging};
-    use serde_json::Value;
-    use serial_test::serial;
-    use std::error::Error;
-    use tracing::info;
-
-    type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-    /// RT-13: Verify lifecycle rule with transition persists and is retrievable.
-    ///
-    /// Note: Actual transition requires a configured remote tier. This test
-    /// validates that an expiration-only rule (the persistence path) survives
-    /// a server restart.
-    #[tokio::test]
-    #[serial]
-    async fn test_lifecycle_rule_persists_after_restart() -> TestResult {
-        init_logging();
-        info!("RT-13: lifecycle rule persists after restart");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        let client = env.create_s3_client();
-        let bucket = "rt13-tier-persist";
-
-        client.create_bucket().bucket(bucket).send().await.expect("create bucket");
-
-        // Apply a lifecycle rule with expiration (transition needs a real tier)
-        let rule = aws_sdk_s3::types::LifecycleRule::builder()
-            .id("expire-after-90d")
-            .status(aws_sdk_s3::types::ExpirationStatus::Enabled)
-            .filter(aws_sdk_s3::types::LifecycleRuleFilter::builder().prefix("archive/").build())
-            .expiration(aws_sdk_s3::types::LifecycleExpiration::builder().days(90).build())
-            .build()
-            .expect("build rule");
-
-        client
-            .put_bucket_lifecycle_configuration()
-            .bucket(bucket)
-            .lifecycle_configuration(
-                aws_sdk_s3::types::BucketLifecycleConfiguration::builder()
-                    .rules(rule)
-                    .build()
-                    .expect("build config"),
-            )
-            .send()
-            .await
-            .expect("put lifecycle");
-
-        // Restart server
-        env.restart_server_preserving_data(vec![], &[]).await.expect("restart RustFS");
-
-        // Verify the rule survived restart
-        let resp = client
-            .get_bucket_lifecycle_configuration()
-            .bucket(bucket)
-            .send()
-            .await
-            .expect("get lifecycle after restart");
-
-        let rules = resp.rules();
-        assert_eq!(rules.len(), 1, "RT-13 FAIL: expected 1 rule after restart");
-
-        let exp = rules[0].expiration().expect("expiration should be set");
-        assert_eq!(exp.days(), Some(90), "RT-13 FAIL: expiration days corrupted after restart");
-
-        info!("RT-13 PASS: lifecycle rule persists after restart");
-        Ok(())
-    }
-
-    /// RT-13b: Verify admin tier configuration API is functional.
-    ///
-    /// Regression pattern: tier add/verify/delete API fails or the tier
-    /// configuration is not persisted (rustfs#5218).
-    #[tokio::test]
-    #[serial]
-    async fn test_admin_tier_list_endpoint_returns_json() -> TestResult {
-        init_logging();
-        info!("RT-13b: admin tier list endpoint returns JSON");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        // Query the tier list endpoint
-        let body = admin_ok(&env, http::Method::GET, "/rustfs/admin/v3/tier", None)
-            .await
-            .expect("list remote tiers");
-
-        let json: Value = serde_json::from_str(&body).expect("tier list response should be valid JSON");
-
-        // Should return an array (possibly empty)
-        assert!(json.is_array(), "RT-13b FAIL: tier list response is not an array: {json}");
-
-        info!("RT-13b PASS: admin tier list endpoint returns valid JSON array");
-        Ok(())
-    }
-
-    /// RT-13c: Verify scanner configuration persistence.
-    ///
-    /// Regression pattern: scanner admin config update reports success but
-    /// is not persisted (rustfs#5013), causing the scanner to not run or
-    /// use stale settings.
-    #[tokio::test]
-    #[serial]
-    async fn test_scanner_config_persists_after_restart() -> TestResult {
-        init_logging();
-        info!("RT-13c: scanner config persists after restart");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("start RustFS");
-
-        // Get current scanner status
-        let body = admin_ok(&env, http::Method::GET, "/rustfs/admin/v3/scanner/status", None)
-            .await
-            .expect("get scanner status");
-
-        let json: Value = serde_json::from_str(&body).expect("scanner status should be valid JSON");
-
-        info!("  scanner status: {:?}", json.as_object().map(|o| o.keys().collect::<Vec<_>>()));
-
-        // Restart and verify config is still accessible
-        env.restart_server_preserving_data(vec![], &[]).await.expect("restart RustFS");
-
-        let body2 = admin_ok(&env, http::Method::GET, "/rustfs/admin/v3/scanner/status", None)
-            .await
-            .expect("get scanner status after restart");
-
-        let json2: Value = serde_json::from_str(&body2).expect("scanner status after restart should be valid JSON");
-
-        // Both should be valid JSON objects
-        assert!(json2.is_object(), "RT-13c FAIL: scanner status after restart is not a valid JSON object");
-
-        info!("RT-13c PASS: scanner/config persists across restart");
-        Ok(())
-    }
-}

crates/ecstore/Cargo.toml

@@ -33,98 +33,14 @@ workspace = true
 [features]
 default = []
 rio-v2 = ["dep:rustfs-rio-v2"]
-hotpath = [
-    "hotpath/hotpath",
-    "hotpath/tokio",
-    "hotpath/futures",
-    "hotpath/async-channel",
-    "hotpath/parking_lot",
-    "hotpath/reqwest-0-13",
-    "rustfs-checksums/hotpath",
-    "rustfs-common/hotpath",
-    "rustfs-concurrency/hotpath",
-    "rustfs-config/hotpath",
-    "rustfs-credentials/hotpath",
-    "rustfs-data-usage/hotpath",
-    "rustfs-filemeta/hotpath",
-    "rustfs-io-metrics/hotpath",
-    "rustfs-lifecycle/hotpath",
-    "rustfs-lock/hotpath",
-    "rustfs-madmin/hotpath",
-    "rustfs-object-capacity/hotpath",
-    "rustfs-policy/hotpath",
-    "rustfs-protos/hotpath",
-    "rustfs-replication/hotpath",
-    "rustfs-rio/hotpath",
-    "rustfs-rio-v2?/hotpath",
-    "rustfs-s3-types/hotpath",
-    "rustfs-signer/hotpath",
-    "rustfs-storage-api/hotpath",
-    "rustfs-tls-runtime/hotpath",
-    "rustfs-utils/hotpath",
-    "rustfs-crypto/hotpath",
-]
-hotpath-alloc = [
-    "hotpath",
-    "hotpath/hotpath-alloc",
-    "rustfs-checksums/hotpath-alloc",
-    "rustfs-common/hotpath-alloc",
-    "rustfs-concurrency/hotpath-alloc",
-    "rustfs-config/hotpath-alloc",
-    "rustfs-credentials/hotpath-alloc",
-    "rustfs-data-usage/hotpath-alloc",
-    "rustfs-filemeta/hotpath-alloc",
-    "rustfs-io-metrics/hotpath-alloc",
-    "rustfs-lifecycle/hotpath-alloc",
-    "rustfs-lock/hotpath-alloc",
-    "rustfs-madmin/hotpath-alloc",
-    "rustfs-object-capacity/hotpath-alloc",
-    "rustfs-policy/hotpath-alloc",
-    "rustfs-protos/hotpath-alloc",
-    "rustfs-replication/hotpath-alloc",
-    "rustfs-rio/hotpath-alloc",
-    "rustfs-rio-v2?/hotpath-alloc",
-    "rustfs-s3-types/hotpath-alloc",
-    "rustfs-signer/hotpath-alloc",
-    "rustfs-storage-api/hotpath-alloc",
-    "rustfs-tls-runtime/hotpath-alloc",
-    "rustfs-utils/hotpath-alloc",
-    "rustfs-crypto/hotpath-alloc",
-]
-hotpath-cpu = [
-    "hotpath",
-    "hotpath/hotpath-cpu",
-    "rustfs-checksums/hotpath-cpu",
-    "rustfs-common/hotpath-cpu",
-    "rustfs-concurrency/hotpath-cpu",
-    "rustfs-config/hotpath-cpu",
-    "rustfs-credentials/hotpath-cpu",
-    "rustfs-data-usage/hotpath-cpu",
-    "rustfs-filemeta/hotpath-cpu",
-    "rustfs-io-metrics/hotpath-cpu",
-    "rustfs-lifecycle/hotpath-cpu",
-    "rustfs-lock/hotpath-cpu",
-    "rustfs-madmin/hotpath-cpu",
-    "rustfs-object-capacity/hotpath-cpu",
-    "rustfs-policy/hotpath-cpu",
-    "rustfs-protos/hotpath-cpu",
-    "rustfs-replication/hotpath-cpu",
-    "rustfs-rio/hotpath-cpu",
-    "rustfs-rio-v2?/hotpath-cpu",
-    "rustfs-s3-types/hotpath-cpu",
-    "rustfs-signer/hotpath-cpu",
-    "rustfs-storage-api/hotpath-cpu",
-    "rustfs-tls-runtime/hotpath-cpu",
-    "rustfs-utils/hotpath-cpu",
-    "rustfs-crypto/hotpath-cpu",
-]
+hotpath = ["dep:hotpath", "hotpath/hotpath", "rustfs-filemeta/hotpath", "rustfs-rio/hotpath"]
 # Exposes shared lifecycle/tier test utilities (MockWarmBackend, fault
 # injection, xl.meta transition assertions) via `api::tier::test_util`.
 # Enable only from `[dev-dependencies]` (rustfs/backlog#1148 ilm-6).
 test-util = []
 
 [dependencies]
-hotpath.workspace = true
+hotpath = { workspace = true, optional = true }
 rustfs-filemeta.workspace = true
 rustfs-utils = { workspace = true, features = ["full"] }
 rustfs-rio.workspace = true
@@ -141,6 +57,7 @@ rustfs-policy.workspace = true
 rustfs-protos.workspace = true
 rustfs-replication.workspace = true
 rustfs-lifecycle.workspace = true
+rustfs-kms.workspace = true
 rustfs-s3-types = { workspace = true }
 rustfs-data-usage.workspace = true
 rustfs-object-capacity.workspace = true
@@ -208,6 +125,8 @@ libc.workspace = true
 rustix = { workspace = true, features = ["process", "fs"] }
 rustfs-madmin.workspace = true
 reqwest = { workspace = true }
+aes-gcm = { workspace = true, features = ["rand_core"] }
+chacha20poly1305.workspace = true
 aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
 urlencoding = { workspace = true }
 smallvec = { workspace = true, features = ["serde"] }

crates/ecstore/src/api/mod.rs

@@ -130,15 +130,13 @@ pub mod bucket {
 
     pub mod metadata_sys {
         pub use crate::bucket::metadata_sys::{
-            BucketMetadataMutationGuard, BucketMetadataSys, ObjectLockConfigState, acquire_bucket_metadata_transaction_lock,
-            capture_bucket_metadata_incarnation, delete, delete_if_incarnation, get, get_accelerate_config, get_bucket_policy,
+            BucketMetadataSys, acquire_bucket_metadata_transaction_lock, delete, get, get_accelerate_config, get_bucket_policy,
             get_bucket_policy_raw, get_bucket_targets_config, get_config_from_disk, get_cors_config, get_durability_config,
             get_global_bucket_metadata_sys, get_lifecycle_config, get_logging_config, get_notification_config,
-            get_object_lock_config, get_object_lock_config_state, get_public_access_block_config, get_quota_config,
-            get_replication_config, get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config,
-            get_website_config, init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
-            set_bucket_metadata, update, update_bucket_targets_under_transaction_lock, update_config_with, update_if_incarnation,
-            update_under_transaction_lock,
+            get_object_lock_config, get_public_access_block_config, get_quota_config, get_replication_config,
+            get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
+            init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata, set_bucket_metadata,
+            update, update_bucket_targets_under_transaction_lock, update_config_with, update_under_transaction_lock,
         };
     }
 
@@ -174,30 +172,20 @@ pub mod bucket {
     }
 
     pub mod replication {
-        pub use crate::bucket::replication::replication_pool::{
-            DurableMrfBacklogSummary, DurableMrfBucketBacklog, DurableMrfTargetBacklog, MrfBacklogObservabilitySummary,
-            MrfBucketBacklogObservability, durable_mrf_backlog_summary_snapshot, durable_mrf_target_backlog_snapshot,
-            mrf_backlog_observability_snapshot,
-        };
         pub use crate::bucket::replication::{
-            BucketReplicationResyncStatus, BucketReplicationStats, BucketStats, DeleteReplicationConfigSnapshot,
-            DeletedObjectReplicationInfo, DurableMrfBacklog, DynReplicationPool, MrfOpKind, MrfReplicateEntry,
-            MustReplicateOptions, ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS,
-            REMOTE_TARGET_WRITABLE_FIELDS, REPLICATE_INCOMING_DELETE, REPLICATION_CAPABILITY_CONTRACT_VERSION,
-            REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS, ReplicateDecision, ReplicateObjectInfo,
-            ReplicationBatchAdmission, ReplicationConfig, ReplicationConfigurationExt, ReplicationDeleteScheduleInput,
+            BucketReplicationResyncStatus, BucketStats, DeletedObjectReplicationInfo, DurableMrfBacklog, DynReplicationPool,
+            MrfOpKind, MrfReplicateEntry, MustReplicateOptions, ObjectOpts, REPLICATE_INCOMING_DELETE, ReplicateDecision,
+            ReplicateObjectInfo, ReplicationConfig, ReplicationConfigurationExt, ReplicationDeleteScheduleInput,
             ReplicationDeleteStateSource, ReplicationHealQueueResult, ReplicationObjectBridge, ReplicationObjectIO,
             ReplicationOperation, ReplicationPoolTrait, ReplicationPriority, ReplicationQueueAdmission, ReplicationScannerBridge,
             ReplicationState, ReplicationStats, ReplicationStatusType, ReplicationStorage, ReplicationTargetValidationError,
-            ReplicationType, ResyncOpts, ResyncStatusType, RuntimeReplicationTargetBacklog, TargetReplicationResyncStatus,
-            VersionPurgeStatusType, commit_force_delete_intent, complete_force_delete_intent,
+            ReplicationType, ResyncOpts, ResyncStatusType, TargetReplicationResyncStatus, VersionPurgeStatusType,
             delete_replication_state_from_config, delete_replication_version_id, get_global_replication_pool,
-            get_global_replication_stats, init_background_replication, invalid_replication_config_status_field,
-            persist_force_delete_intent, read_durable_mrf_backlog, replication_state_to_filemeta, replication_status_to_filemeta,
-            replication_statuses_map, replication_target_arns, resync_start_conflict_id, should_remove_replication_target,
-            should_schedule_delete_replication, should_use_existing_delete_replication_info,
-            should_use_existing_delete_replication_source, unsupported_replication_config_field,
-            validate_replication_config_target_arns, version_purge_status_to_filemeta,
+            get_global_replication_stats, init_background_replication, read_durable_mrf_backlog, replication_state_to_filemeta,
+            replication_status_to_filemeta, replication_statuses_map, replication_target_arns, resync_start_conflict_id,
+            should_remove_replication_target, should_schedule_delete_replication, should_use_existing_delete_replication_info,
+            should_use_existing_delete_replication_source, validate_replication_config_target_arns,
+            version_purge_status_to_filemeta,
         };
     }
 
@@ -279,13 +267,12 @@ pub mod config {
     pub mod com {
         pub use crate::config::com::{
             COMMA_SEPARATED_LISTS, CONFIG_PREFIX, ENV_CONFIG_RECOVER_ON_CORRUPTION, STORAGE_CLASS_SUB_SYS,
-            ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config,
-            is_server_config_corrupt_error, lookup_configs, read_config, read_config_no_lock, read_config_with_metadata,
-            read_config_without_migrate, read_config_without_migrate_no_lock, read_existing_server_config_no_lock,
-            read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_server_config,
-            save_server_config_no_lock, save_server_config_snapshot, save_server_config_snapshot_with_generation,
-            server_config_path, try_migrate_server_config, with_config_object_read_lock, with_config_object_write_lock,
-            with_server_config_read_lock, with_server_config_write_lock,
+            ServerConfigCorruptError, ServerConfigSnapshot, delete_config, is_server_config_corrupt_error, lookup_configs,
+            read_config, read_config_no_lock, read_config_with_metadata, read_config_without_migrate,
+            read_config_without_migrate_no_lock, read_existing_server_config_no_lock, read_server_config_snapshot, save_config,
+            save_config_no_lock, save_config_with_opts, save_server_config, save_server_config_no_lock,
+            save_server_config_snapshot, server_config_path, try_migrate_server_config, with_config_object_read_lock,
+            with_config_object_write_lock, with_server_config_read_lock, with_server_config_write_lock,
         };
     }
 
@@ -404,12 +391,10 @@ pub mod notification {
 
 pub mod object {
     pub use crate::object_api::{
-        BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
-        GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, NamespaceLockFence, ObjectEncryptionResolver,
-        ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, RangedDecompressReader,
-        ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest, StreamConsumer, get_object_body_cache_plaintext_len,
-        lookup_get_object_body_cache_hook, register_get_object_body_cache_hook, register_object_mutation_hook,
-        unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
+        BLOCK_SIZE_V2, ERASURE_ALGORITHM, GetObjectBodyCacheHook, GetObjectBodyCacheHookLookup, GetObjectBodySource,
+        GetObjectReader, ObjectInfo, ObjectMutationHook, ObjectOptions, PutObjReader, RangedDecompressReader, StreamConsumer,
+        get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
+        register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
     };
     pub use crate::store::PreparedGetObjectReader;
 }
@@ -431,15 +416,15 @@ pub mod rio {
 
 pub mod rpc {
     pub use crate::cluster::rpc::{
-        AuthenticatedChannel, KMS_SIGNAL_SUBSYSTEM, LocalPeerS3Client, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS,
-        PeerRestClient, PeerS3Client, S3PeerSys, SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC,
-        ScannerBucketListing, ScannerPeerActivity, TONIC_RPC_PREFIX, TonicInterceptor, gen_signature_headers,
-        gen_tonic_replay_scope_headers, gen_tonic_signature_headers, gen_tonic_signature_interceptor,
-        node_service_time_out_client, node_service_time_out_client_no_auth, normalize_tonic_rpc_audience,
-        set_tonic_canonical_body_digest, sign_ns_scanner_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge,
-        tonic_boot_epoch_response_headers, verify_rpc_signature, verify_tonic_boot_epoch_response,
-        verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest, verify_tonic_rpc_response_proof,
-        verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
+        AuthenticatedChannel, LocalPeerS3Client, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient,
+        PeerS3Client, S3PeerSys, SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerBucketListing,
+        ScannerPeerActivity, TONIC_RPC_PREFIX, TonicInterceptor, gen_signature_headers, gen_tonic_replay_scope_headers,
+        gen_tonic_signature_headers, gen_tonic_signature_interceptor, node_service_time_out_client,
+        node_service_time_out_client_no_auth, normalize_tonic_rpc_audience, set_tonic_canonical_body_digest,
+        sign_ns_scanner_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers,
+        verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
+        verify_tonic_mutation_body_digest, verify_tonic_rpc_response_proof, verify_tonic_rpc_signature,
+        verify_tonic_rpc_signature_with_bootstrap,
     };
 }
 

crates/ecstore/src/bucket/lifecycle/metadata_boundary.rs

@@ -12,127 +12,20 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use std::sync::Arc;
-
-use s3s::dto::{BucketLifecycleConfiguration, ObjectLockConfiguration};
+use s3s::dto::{BucketLifecycleConfiguration, ObjectLockConfiguration, ReplicationConfiguration};
 use time::OffsetDateTime;
-use uuid::Uuid;
 
-use crate::bucket::metadata_sys::{self, ObjectLockConfigState};
-use crate::error::{Error, Result};
-
-#[derive(Debug)]
-pub(crate) struct LifecycleExpiryConfigs {
-    pub(crate) lifecycle: Option<Arc<BucketLifecycleConfiguration>>,
-    pub(crate) object_lock: Option<Arc<ObjectLockConfiguration>>,
-    pub(crate) bucket_incarnation_id: Uuid,
-}
-
-pub(crate) async fn get_expiry_configs(api: &crate::store::ECStore, bucket: &str) -> Result<LifecycleExpiryConfigs> {
-    let bucket_incarnation_id = api.bucket_incarnation_id_from_disk(bucket).await?;
-    let sys = metadata_sys::bucket_metadata_sys_of(&api.ctx)?;
-    let sys = sys.read().await.clone();
-    let metadata = sys.get_authoritative_metadata(bucket).await?;
-    if !metadata.bucket_incarnation_sidecar || metadata.bucket_incarnation_id != bucket_incarnation_id {
-        return Err(Error::other(format!("bucket lifecycle metadata is not authoritative: {bucket}")));
-    }
-
-    let lifecycle = if metadata.lifecycle_config.is_none() && !metadata.lifecycle_config_xml.is_empty() {
-        return Err(Error::other("persisted bucket lifecycle configuration is invalid"));
-    } else {
-        metadata
-            .lifecycle_config
-            .clone()
-            .filter(|config| !config.rules.is_empty())
-            .map(Arc::new)
-    };
-    if lifecycle.is_none() {
-        return Ok(LifecycleExpiryConfigs {
-            lifecycle: None,
-            object_lock: None,
-            bucket_incarnation_id,
-        });
-    }
-    let object_lock = match metadata_sys::object_lock_config_state_from_authoritative_metadata(&metadata)? {
-        ObjectLockConfigState::Configured { config, .. } => Some(Arc::new(config)),
-        ObjectLockConfigState::ConfirmedAbsent => None,
-        ObjectLockConfigState::Fabricated => {
-            return Err(Error::other(format!("bucket Object Lock metadata is not authoritative: {bucket}")));
-        }
-    };
-
-    Ok(LifecycleExpiryConfigs {
-        lifecycle,
-        object_lock,
-        bucket_incarnation_id,
-    })
-}
+use crate::bucket::metadata_sys;
+use crate::error::Result;
 
 pub(crate) async fn get_lifecycle_config(bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime)> {
     metadata_sys::get_lifecycle_config(bucket).await
 }
 
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use crate::bucket::metadata::BucketMetadata;
-    use crate::bucket::metadata_sys::{self, test_support::isolated_store_over_temp_disks};
-    use crate::storage_api_contracts::bucket::MakeBucketOptions;
-    use s3s::dto::{ExpirationStatus, LifecycleExpiration, LifecycleRule};
-    use serial_test::serial;
-
-    fn lifecycle_config() -> BucketLifecycleConfiguration {
-        BucketLifecycleConfiguration {
-            expiry_updated_at: None,
-            rules: vec![LifecycleRule {
-                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
-                expiration: Some(LifecycleExpiration {
-                    days: Some(1),
-                    ..Default::default()
-                }),
-                abort_incomplete_multipart_upload: None,
-                del_marker_expiration: None,
-                filter: None,
-                id: Some("expire".to_string()),
-                noncurrent_version_expiration: None,
-                noncurrent_version_transitions: None,
-                prefix: None,
-                transitions: None,
-            }],
-        }
-    }
-
-    #[tokio::test]
-    #[serial]
-    async fn expiry_configs_are_resolved_from_the_owning_store() {
-        let (_dirs_a, store_a) = isolated_store_over_temp_disks().await;
-        let (_dirs_b, store_b) = isolated_store_over_temp_disks().await;
-        let bucket = "same-name-expiry-config";
-        store_a
-            .peer_sys
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .unwrap();
-        store_b
-            .peer_sys
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .unwrap();
-        metadata_sys::init_bucket_metadata_sys(store_a.clone(), vec![bucket.to_string()]).await;
-        metadata_sys::init_bucket_metadata_sys(store_b.clone(), vec![bucket.to_string()]).await;
-
-        let mut metadata = BucketMetadata::new(bucket);
-        let lifecycle = lifecycle_config();
-        metadata.lifecycle_config_xml = crate::bucket::utils::serialize(&lifecycle).unwrap();
-        metadata.lifecycle_config = Some(lifecycle);
-        metadata_sys::set_new_bucket_metadata_in(&store_a.ctx, metadata)
-            .await
-            .unwrap();
-        metadata_sys::set_new_bucket_metadata_in(&store_b.ctx, BucketMetadata::new(bucket))
-            .await
-            .unwrap();
-
-        assert!(get_expiry_configs(&store_a, bucket).await.unwrap().lifecycle.is_some());
-        assert!(get_expiry_configs(&store_b, bucket).await.unwrap().lifecycle.is_none());
-    }
+pub(crate) async fn get_object_lock_config(bucket: &str) -> Result<(ObjectLockConfiguration, OffsetDateTime)> {
+    metadata_sys::get_object_lock_config(bucket).await
+}
+
+pub(crate) async fn get_replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
+    metadata_sys::get_replication_config(bucket).await
 }

crates/ecstore/src/bucket/lifecycle/mod.rs

@@ -19,7 +19,6 @@ pub mod core;
 pub mod evaluator;
 pub mod manual_transition_job;
 mod metadata_boundary;
-pub(crate) use metadata_boundary::get_expiry_configs;
 mod object_lock_boundary;
 pub use self::core as lifecycle;
 mod replication_sink;

crates/ecstore/src/bucket/lifecycle/object_lock_boundary.rs

@@ -21,12 +21,12 @@ pub(crate) fn is_object_locked_by_metadata(user_defined: &HashMap<String, String
     rustfs_lifecycle::object_lock::is_object_locked_by_metadata(user_defined, is_delete_marker)
 }
 
-pub(crate) fn check_object_lock_for_deletion_with_config(
-    config: Option<&s3s::dto::ObjectLockConfiguration>,
+pub(crate) async fn check_object_lock_for_deletion(
+    bucket: &str,
     obj_info: &ObjectInfo,
     bypass_governance: bool,
-) -> crate::error::Result<Option<ObjectLockBlockReason>> {
-    objectlock_sys::check_object_lock_for_deletion_with_config(config, obj_info, bypass_governance)
+) -> Option<ObjectLockBlockReason> {
+    objectlock_sys::check_object_lock_for_deletion(bucket, obj_info, bypass_governance).await
 }
 
 #[cfg(test)]

crates/ecstore/src/bucket/lifecycle/replication_sink.rs

@@ -15,14 +15,15 @@
 use rustfs_common::metrics::IlmAction;
 
 use crate::bucket::lifecycle::lifecycle::ObjectOpts;
-pub(crate) use crate::bucket::replication::ReplicationStatusType;
 #[cfg(test)]
-pub(crate) use crate::bucket::replication::VersionPurgeStatusType;
+pub(crate) use crate::bucket::replication::ReplicateTargetDecision;
 pub(crate) use crate::bucket::replication::{
-    DeleteReplicationConfigSnapshot, ReplicationObjectBridge, replication_state_to_filemeta,
+    ReplicateDecision, ReplicationState, ReplicationStatusType, VersionPurgeStatusType, replication_state_to_filemeta,
+    replication_statuses_map, version_purge_statuses_map,
 };
 use crate::bucket::replication::{ReplicationLifecycleBridge, ReplicationLifecycleConfig};
-use crate::storage_api_contracts::object::DeletedObject;
+use crate::object_api::{ObjectInfo, ObjectOptions};
+use crate::storage_api_contracts::object::{DeletedObject, ObjectToDelete};
 
 pub(crate) type LifecycleReplicationConfig = ReplicationLifecycleConfig;
 
@@ -56,6 +57,15 @@ pub(crate) fn lifecycle_action_waits_for_replication(action: IlmAction) -> bool
     )
 }
 
+pub(crate) async fn check_delete_replication(
+    bucket: &str,
+    object: ObjectToDelete,
+    source: &ObjectInfo,
+    opts: &ObjectOptions,
+) -> ReplicateDecision {
+    ReplicationLifecycleBridge::check_delete_replication(bucket, &object, source, opts).await
+}
+
 pub(crate) async fn schedule_delete(bucket: String, delete_object: DeletedObject) {
     ReplicationLifecycleBridge::schedule_delete(bucket, delete_object).await;
 }
@@ -64,16 +74,7 @@ pub(crate) async fn schedule_delete(bucket: String, delete_object: DeletedObject
 mod tests {
     use std::collections::HashMap;
 
-    use crate::bucket::replication::{DeleteReplicationConfigSnapshot, ReplicationObjectBridge};
-    use crate::object_api::{ObjectInfo, ObjectOptions};
-    use crate::storage_api_contracts::object::ObjectToDelete;
     use rustfs_common::metrics::IlmAction;
-    use s3s::dto::{
-        BucketVersioningStatus, DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication,
-        DeleteReplicationStatus, Destination, ReplicationConfiguration, ReplicationRule, ReplicationRuleStatus,
-        VersioningConfiguration,
-    };
-    use uuid::Uuid;
 
     use super::*;
 
@@ -138,97 +139,4 @@ mod tests {
         assert!(lifecycle_action_waits_for_replication(IlmAction::TransitionVersionAction));
         assert!(!lifecycle_action_waits_for_replication(IlmAction::NoneAction));
     }
-
-    #[test]
-    fn lifecycle_delete_admission_uses_marker_and_version_switches_for_all_purges() {
-        for marker_enabled in [false, true] {
-            for purge_enabled in [false, true] {
-                let snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
-                    VersioningConfiguration {
-                        status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
-                        ..Default::default()
-                    },
-                    Some(ReplicationConfiguration {
-                        role: String::new(),
-                        rules: vec![ReplicationRule {
-                            delete_marker_replication: Some(DeleteMarkerReplication {
-                                status: Some(if marker_enabled {
-                                    DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)
-                                } else {
-                                    DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)
-                                }),
-                            }),
-                            delete_replication: Some(DeleteReplication {
-                                status: if purge_enabled {
-                                    DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED)
-                                } else {
-                                    DeleteReplicationStatus::from_static(DeleteReplicationStatus::DISABLED)
-                                },
-                            }),
-                            destination: Destination {
-                                bucket: "arn:rustfs:replication:target".to_string(),
-                                ..Default::default()
-                            },
-                            existing_object_replication: None,
-                            filter: None,
-                            id: Some("lifecycle-delete-switches".to_string()),
-                            prefix: Some(String::new()),
-                            priority: Some(1),
-                            source_selection_criteria: None,
-                            status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
-                        }],
-                    }),
-                );
-                let source = ObjectInfo {
-                    bucket: "bucket".to_string(),
-                    name: "logs/object".to_string(),
-                    ..Default::default()
-                };
-                let marker = ObjectToDelete {
-                    object_name: source.name.clone(),
-                    ..Default::default()
-                };
-                let marker_opts = ObjectOptions {
-                    versioned: true,
-                    ..Default::default()
-                };
-                assert_eq!(
-                    ReplicationObjectBridge::check_delete_with_snapshot(&marker, &source, &marker_opts, false, &snapshot)
-                        .replicate_any(),
-                    marker_enabled
-                );
-
-                for delete_marker in [false, true] {
-                    for version_id in [Uuid::new_v4(), Uuid::nil()] {
-                        let purge = ObjectToDelete {
-                            object_name: source.name.clone(),
-                            version_id: Some(version_id),
-                            ..Default::default()
-                        };
-                        let purge_source = ObjectInfo {
-                            delete_marker,
-                            ..source.clone()
-                        };
-                        let purge_opts = ObjectOptions {
-                            version_id: Some(version_id.to_string()),
-                            versioned: true,
-                            ..Default::default()
-                        };
-                        assert_eq!(
-                            ReplicationObjectBridge::check_delete_with_snapshot(
-                                &purge,
-                                &purge_source,
-                                &purge_opts,
-                                false,
-                                &snapshot,
-                            )
-                            .replicate_any(),
-                            purge_enabled,
-                            "delete marker={delete_marker}, version_id={version_id}"
-                        );
-                    }
-                }
-            }
-        }
-    }
 }

crates/ecstore/src/bucket/lifecycle/tier_delete_journal.rs

@@ -20,10 +20,8 @@ use tokio_util::sync::CancellationToken;
 use tracing::{debug, warn};
 
 use crate::bucket::lifecycle::config_boundary;
-use crate::bucket::lifecycle::runtime_boundary;
 use crate::bucket::lifecycle::tier_sweeper::{
-    Jentry, TierDeleteJournalState, TierDeleteSourceIdentity,
-    delete_confirmed_transition_candidate_exact_with_manager_and_identity,
+    Jentry, delete_confirmed_transition_candidate_exact_with_manager_and_identity,
     delete_object_from_remote_tier_idempotent_with_manager_and_identity,
 };
 use crate::disk::RUSTFS_META_BUCKET;
@@ -32,7 +30,7 @@ use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader
 use crate::services::tier::tier::tier_destination_id_from_metadata;
 use crate::storage_api_contracts::{
     list::ListOperations as _,
-    object::{DeletedObject, HTTPPreconditions, ObjectIO, ObjectOperations, ObjectToDelete},
+    object::{DeletedObject, ObjectIO, ObjectOperations, ObjectToDelete},
     range::HTTPRangeSpec,
 };
 use crate::store::ECStore;
@@ -48,7 +46,6 @@ const TIER_DELETE_JOURNAL_RECOVERY_TIMEOUT: Duration = Duration::from_secs(300);
 const TIER_DELETE_JOURNAL_VERSION: u8 = 2;
 const TIER_DELETE_JOURNAL_EXACT_VERSION: u8 = 3;
 const TIER_DELETE_JOURNAL_STATE_VERSION: u8 = 4;
-const TIER_DELETE_JOURNAL_TRANSACTION_VERSION: u8 = 5;
 pub(crate) const TIER_DELETE_JOURNAL_PREFIX: &str = "ilm/tier-delete-journal/";
 
 #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
@@ -64,22 +61,13 @@ struct PersistedTierDeleteJournalEntry {
     version_id_exact: Option<bool>,
     #[serde(default, skip_serializing_if = "Option::is_none")]
     version_state: Option<rustfs_filemeta::TransitionVersionState>,
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    state: Option<TierDeleteJournalState>,
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    source: Option<TierDeleteSourceIdentity>,
 }
 
 impl PersistedTierDeleteJournalEntry {
     fn from_jentry(je: &Jentry) -> Result<Self> {
         validate_version_state(je.version_state, &je.version_id, je.version_id_exact)?;
         let legacy_unknown = je.version_state == rustfs_filemeta::TransitionVersionState::Unknown;
-        let version = if je.source.is_some() || je.state == TierDeleteJournalState::Prepared {
-            if je.backend_identity.is_none() {
-                return Err(Error::other("tier delete transaction is missing its backend identity"));
-            }
-            TIER_DELETE_JOURNAL_TRANSACTION_VERSION
-        } else if legacy_unknown {
+        let version = if legacy_unknown {
             if je.backend_identity.is_some() {
                 TIER_DELETE_JOURNAL_VERSION
             } else {
@@ -99,10 +87,6 @@ impl PersistedTierDeleteJournalEntry {
             backend_identity: je.backend_identity,
             version_id_exact: je.version_id_exact.then_some(true),
             version_state: (!legacy_unknown).then_some(je.version_state),
-            state: (version == TIER_DELETE_JOURNAL_TRANSACTION_VERSION).then_some(je.state),
-            source: (version == TIER_DELETE_JOURNAL_TRANSACTION_VERSION)
-                .then(|| je.source.clone())
-                .flatten(),
         })
     }
 
@@ -117,21 +101,14 @@ impl PersistedTierDeleteJournalEntry {
         }
         if self.version != TIER_DELETE_JOURNAL_EXACT_VERSION
             && self.version != TIER_DELETE_JOURNAL_STATE_VERSION
-            && self.version != TIER_DELETE_JOURNAL_TRANSACTION_VERSION
             && self.version_id_exact.unwrap_or(false)
         {
             return Err(Error::other(
                 "legacy tier delete journal entry has an unsupported exact version constraint",
             ));
         }
-        let (backend_identity, version_id_exact, version_state, state, source) = match self.version {
-            1 => (
-                None,
-                false,
-                rustfs_filemeta::TransitionVersionState::Unknown,
-                TierDeleteJournalState::Committed,
-                None,
-            ),
+        let (backend_identity, version_id_exact, version_state) = match self.version {
+            1 => (None, false, rustfs_filemeta::TransitionVersionState::Unknown),
             TIER_DELETE_JOURNAL_VERSION => (
                 Some(
                     self.backend_identity
@@ -139,8 +116,6 @@ impl PersistedTierDeleteJournalEntry {
                 ),
                 false,
                 rustfs_filemeta::TransitionVersionState::Unknown,
-                TierDeleteJournalState::Committed,
-                None,
             ),
             TIER_DELETE_JOURNAL_EXACT_VERSION => {
                 if self.version_id.is_empty() || self.version_id_exact != Some(true) {
@@ -153,8 +128,6 @@ impl PersistedTierDeleteJournalEntry {
                     ),
                     true,
                     rustfs_filemeta::TransitionVersionState::Exact,
-                    TierDeleteJournalState::Committed,
-                    None,
                 )
             }
             TIER_DELETE_JOURNAL_STATE_VERSION => {
@@ -170,31 +143,6 @@ impl PersistedTierDeleteJournalEntry {
                     ),
                     exact,
                     state,
-                    TierDeleteJournalState::Committed,
-                    None,
-                )
-            }
-            TIER_DELETE_JOURNAL_TRANSACTION_VERSION => {
-                let state = self
-                    .state
-                    .ok_or_else(|| Error::other("tier delete journal v5 entry is missing its state"))?;
-                let source = self
-                    .source
-                    .ok_or_else(|| Error::other("tier delete journal v5 entry is missing its source identity"))?;
-                let exact = self.version_id_exact.unwrap_or(false);
-                let version_state = self
-                    .version_state
-                    .ok_or_else(|| Error::other("tier delete journal v5 entry is missing its version state"))?;
-                validate_version_state(version_state, &self.version_id, exact)?;
-                (
-                    Some(
-                        self.backend_identity
-                            .ok_or_else(|| Error::other("tier delete journal v5 entry is missing its backend identity"))?,
-                    ),
-                    exact,
-                    version_state,
-                    state,
-                    Some(source),
                 )
             }
             version => return Err(Error::other(format!("unsupported tier delete journal version {version}"))),
@@ -206,8 +154,6 @@ impl PersistedTierDeleteJournalEntry {
             backend_identity,
             version_id_exact,
             version_state,
-            state,
-            source,
         })
     }
 }
@@ -255,20 +201,6 @@ pub(crate) fn tier_delete_journal_object_name(je: &Jentry) -> String {
         hasher.update([0]);
         hasher.update(b"exact-version-id");
     }
-    if let Some(source) = &je.source {
-        hasher.update([0]);
-        hasher.update(source.bucket.as_bytes());
-        hasher.update([0]);
-        hasher.update(source.object.as_bytes());
-        hasher.update([0]);
-        hasher.update(source.version_id.as_deref().unwrap_or_default().as_bytes());
-        hasher.update([0]);
-        hasher.update(source.data_dir.as_deref().unwrap_or_default().as_bytes());
-        hasher.update([0]);
-        hasher.update(source.etag.as_deref().unwrap_or_default().as_bytes());
-        hasher.update([0]);
-        hasher.update(source.mod_time.as_deref().unwrap_or_default().as_bytes());
-    }
     format!(
         "{TIER_DELETE_JOURNAL_PREFIX}{}.json",
         rustfs_utils::crypto::hex(hasher.finalize().as_slice())
@@ -314,66 +246,6 @@ where
         .map_err(std::io::Error::other)
 }
 
-pub async fn commit_tier_delete_journal_entry<S>(api: Arc<S>, je: &Jentry) -> std::io::Result<()>
-where
-    S: ObjectIO<
-            Error = Error,
-            RangeSpec = HTTPRangeSpec,
-            HeaderMap = http::HeaderMap,
-            ObjectOptions = ObjectOptions,
-            ObjectInfo = ObjectInfo,
-            GetObjectReader = GetObjectReader,
-            PutObjectReader = PutObjReader,
-        >,
-{
-    let mut committed = je.clone();
-    committed.state = TierDeleteJournalState::Committed;
-    persist_tier_delete_journal_entry(api, &committed).await
-}
-
-pub async fn abort_tier_delete_journal_entry<S>(api: Arc<S>, je: &Jentry) -> std::io::Result<()>
-where
-    S: ObjectOperations<
-            Error = Error,
-            ObjectInfo = ObjectInfo,
-            ObjectOptions = ObjectOptions,
-            FileInfo = FileInfo,
-            ObjectToDelete = ObjectToDelete,
-            DeletedObject = DeletedObject,
-        >,
-{
-    remove_tier_delete_journal_entry(api, je).await
-}
-
-pub async fn abort_prepared_tier_delete_journal_entry(api: Arc<ECStore>, je: &Jentry) -> std::io::Result<()> {
-    let name = tier_delete_journal_object_name(je);
-    let (data, metadata) = match config_boundary::read_config_with_metadata(api.clone(), &name, &ObjectOptions::default()).await {
-        Ok(result) => result,
-        Err(Error::ConfigNotFound) | Err(Error::FileNotFound) => return Ok(()),
-        Err(err) => return Err(std::io::Error::other(err)),
-    };
-    let current = decode_tier_delete_journal_entry(&data).map_err(std::io::Error::other)?;
-    if current.state != TierDeleteJournalState::Prepared {
-        return Ok(());
-    }
-    let etag = metadata
-        .etag
-        .ok_or_else(|| std::io::Error::other("prepared tier delete journal has no entity tag"))?;
-    match config_boundary::delete_config_if_match(api, &name, &etag).await {
-        Ok(()) | Err(Error::ConfigNotFound) => Ok(()),
-        Err(Error::PreconditionFailed) => Err(std::io::Error::new(
-            std::io::ErrorKind::WouldBlock,
-            "prepared tier delete journal changed before abort",
-        )),
-        Err(err) => Err(std::io::Error::other(err)),
-    }
-}
-
-pub(crate) async fn enqueue_committed_tier_delete_journal_entry(je: &Jentry) -> std::io::Result<()> {
-    let expiry_state = runtime_boundary::expiry_state_handle();
-    expiry_state.write().await.enqueue_tier_journal_entry(je)
-}
-
 pub async fn remove_tier_delete_journal_entry<S>(api: Arc<S>, je: &Jentry) -> std::io::Result<()>
 where
     S: ObjectOperations<
@@ -392,13 +264,6 @@ where
 }
 
 pub async fn process_tier_delete_journal_entry(api: Arc<ECStore>, je: &Jentry) -> std::io::Result<()> {
-    if je.state == TierDeleteJournalState::Prepared {
-        return reconcile_prepared_tier_delete_journal_entry(api, je).await;
-    }
-    process_committed_tier_delete_journal_entry(api, je).await
-}
-
-async fn process_committed_tier_delete_journal_entry(api: Arc<ECStore>, je: &Jentry) -> std::io::Result<()> {
     if je.version_state == rustfs_filemeta::TransitionVersionState::Unknown {
         return Err(std::io::Error::new(
             std::io::ErrorKind::InvalidData,
@@ -431,87 +296,6 @@ async fn process_committed_tier_delete_journal_entry(api: Arc<ECStore>, je: &Jen
     remove_tier_delete_journal_entry(api, je).await
 }
 
-async fn reconcile_prepared_tier_delete_journal_entry(api: Arc<ECStore>, je: &Jentry) -> std::io::Result<()> {
-    let (data, metadata) =
-        config_boundary::read_config_with_metadata(api.clone(), &tier_delete_journal_object_name(je), &ObjectOptions::default())
-            .await
-            .map_err(std::io::Error::other)?;
-    let current = decode_tier_delete_journal_entry(&data).map_err(std::io::Error::other)?;
-    if current.state != TierDeleteJournalState::Prepared {
-        return Err(std::io::Error::new(
-            std::io::ErrorKind::WouldBlock,
-            "prepared tier delete journal changed before reconciliation",
-        ));
-    }
-    let Some(etag) = metadata.etag else {
-        return Err(std::io::Error::new(
-            std::io::ErrorKind::InvalidData,
-            "prepared tier delete journal has no entity tag",
-        ));
-    };
-    let source = je
-        .source
-        .as_ref()
-        .ok_or_else(|| std::io::Error::new(std::io::ErrorKind::InvalidData, "prepared tier delete journal has no source"))?;
-    match api
-        .get_object_info(&source.bucket, &source.object, &source.lookup_options())
-        .await
-    {
-        Ok(info) if source.matches(&info) => {
-            match config_boundary::delete_config_if_match(api, &tier_delete_journal_object_name(&current), &etag).await {
-                Ok(()) => Ok(()),
-                Err(Error::PreconditionFailed) => Err(std::io::Error::new(
-                    std::io::ErrorKind::WouldBlock,
-                    "prepared tier delete journal changed before abort",
-                )),
-                Err(err) => Err(std::io::Error::other(err)),
-            }
-        }
-        Ok(_info) if source.has_stable_identity() => {
-            commit_prepared_tier_delete_journal_entry_if_current(api, current, etag).await
-        }
-        Ok(_) => Err(std::io::Error::new(
-            std::io::ErrorKind::WouldBlock,
-            "prepared tier delete journal source identity is not sufficient to confirm deletion",
-        )),
-        Err(Error::ObjectNotFound(_, _)) | Err(Error::FileNotFound) | Err(Error::FileVersionNotFound) => {
-            commit_prepared_tier_delete_journal_entry_if_current(api, current, etag).await
-        }
-        Err(err) => Err(std::io::Error::other(err)),
-    }
-}
-
-async fn commit_prepared_tier_delete_journal_entry_if_current(
-    api: Arc<ECStore>,
-    mut committed: Jentry,
-    etag: String,
-) -> std::io::Result<()> {
-    committed.state = TierDeleteJournalState::Committed;
-    let data = encode_tier_delete_journal_entry(&committed).map_err(std::io::Error::other)?;
-    match config_boundary::save_config_with_opts(
-        api.clone(),
-        &tier_delete_journal_object_name(&committed),
-        data,
-        &ObjectOptions {
-            max_parity: true,
-            http_preconditions: Some(HTTPPreconditions {
-                if_match: Some(etag),
-                ..Default::default()
-            }),
-            ..Default::default()
-        },
-    )
-    .await
-    {
-        Ok(()) => process_committed_tier_delete_journal_entry(api, &committed).await,
-        Err(Error::PreconditionFailed) => Err(std::io::Error::new(
-            std::io::ErrorKind::WouldBlock,
-            "prepared tier delete journal changed before commit",
-        )),
-        Err(err) => Err(std::io::Error::other(err)),
-    }
-}
-
 pub async fn recover_tier_delete_journal_entries(
     api: Arc<ECStore>,
     limit: usize,
@@ -698,13 +482,10 @@ mod tests {
         decode_tier_delete_journal_entry, encode_tier_delete_journal_entry, record_tier_delete_journal_backend_identity,
         tier_delete_journal_object_name,
     };
-    use crate::bucket::lifecycle::tier_sweeper::{Jentry, TierDeleteJournalState, TierDeleteSourceIdentity};
+    use crate::bucket::lifecycle::tier_sweeper::Jentry;
     use crate::error::Result;
-    use crate::object_api::ObjectInfo;
     use std::time::Duration;
-    use time::OffsetDateTime;
     use tokio_util::sync::CancellationToken;
-    use uuid::Uuid;
 
     fn journal_entry() -> Jentry {
         Jentry {
@@ -714,8 +495,6 @@ mod tests {
             backend_identity: Some([7; 32]),
             version_id_exact: true,
             version_state: rustfs_filemeta::TransitionVersionState::Exact,
-            state: TierDeleteJournalState::Committed,
-            source: None,
         }
     }
 
@@ -734,55 +513,6 @@ mod tests {
         assert_eq!(decoded.version_state, je.version_state);
     }
 
-    #[test]
-    fn tier_delete_transaction_roundtrips_prepared_source_identity() {
-        let mut je = journal_entry();
-        je.state = TierDeleteJournalState::Prepared;
-        je.source = Some(TierDeleteSourceIdentity {
-            bucket: "bucket".to_string(),
-            object: "object".to_string(),
-            version_id: Some("version".to_string()),
-            versioned: true,
-            version_suspended: false,
-            data_dir: Some("data-dir".to_string()),
-            etag: Some("etag".to_string()),
-            mod_time: Some("mod-time".to_string()),
-        });
-
-        let encoded = encode_tier_delete_journal_entry(&je).expect("prepared transaction should encode");
-        let value: serde_json::Value = serde_json::from_slice(&encoded).expect("transaction should be JSON");
-        assert_eq!(value["version"], serde_json::json!(5));
-        assert_eq!(value["state"], serde_json::json!("Prepared"));
-        assert!(value["source"].is_object());
-
-        let decoded = decode_tier_delete_journal_entry(&encoded).expect("prepared transaction should decode");
-        assert_eq!(decoded.state, TierDeleteJournalState::Prepared);
-        assert_eq!(decoded.source, je.source);
-    }
-
-    #[test]
-    fn tier_delete_source_identity_rejects_recreated_object() {
-        let version_id = Uuid::from_u128(1);
-        let data_dir = Uuid::from_u128(2);
-        let mod_time = OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(1);
-        let info = ObjectInfo {
-            bucket: "bucket".to_string(),
-            name: "object".to_string(),
-            version_id: Some(version_id),
-            data_dir: Some(data_dir),
-            mod_time: Some(mod_time),
-            ..Default::default()
-        };
-        let source = TierDeleteSourceIdentity::from_object_info("bucket", "object", &info, true, false);
-        assert!(source.matches(&info));
-
-        let recreated = ObjectInfo {
-            data_dir: Some(Uuid::from_u128(3)),
-            ..info
-        };
-        assert!(!source.matches(&recreated));
-    }
-
     #[test]
     fn tier_delete_journal_roundtrips_exact_put_response_constraint() {
         let mut exact = journal_entry();

crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs

@@ -23,12 +23,10 @@ use crate::bucket::lifecycle::bucket_lifecycle_ops::ExpiryOp;
 use crate::bucket::lifecycle::lifecycle::{self, ObjectOpts};
 use crate::bucket::lifecycle::tier_delete_journal::persist_tier_delete_journal_entry;
 use crate::client::signer_error::error_chain_contains_signer_header_marker;
-use crate::object_api::ObjectInfo;
 use crate::services::tier::tier::{TierConfigMgr, TierDestinationId, TierOperationLease};
 use crate::storage_api_contracts::lifecycle::TransitionedObject;
 use crate::store::ECStore;
 use rustfs_utils::get_env_usize;
-use serde::{Deserialize, Serialize};
 use sha2::{Digest, Sha256};
 use std::any::Any;
 use std::collections::VecDeque;
@@ -259,8 +257,6 @@ impl ObjSweeper {
                     rustfs_filemeta::TransitionVersionState::SuspendedNull | rustfs_filemeta::TransitionVersionState::Exact
                 ),
                 version_state: self.transition_version_state,
-                state: TierDeleteJournalState::Committed,
-                source: None,
             });
         }
         None
@@ -289,76 +285,6 @@ impl ObjSweeper {
     }
 }
 
-#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
-pub(crate) enum TierDeleteJournalState {
-    Prepared,
-    Committed,
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
-#[serde(deny_unknown_fields)]
-pub(crate) struct TierDeleteSourceIdentity {
-    pub(crate) bucket: String,
-    pub(crate) object: String,
-    pub(crate) version_id: Option<String>,
-    pub(crate) versioned: bool,
-    pub(crate) version_suspended: bool,
-    pub(crate) data_dir: Option<String>,
-    pub(crate) etag: Option<String>,
-    pub(crate) mod_time: Option<String>,
-}
-
-impl TierDeleteSourceIdentity {
-    pub(crate) fn from_object_info(
-        bucket: &str,
-        object: &str,
-        info: &ObjectInfo,
-        versioned: bool,
-        version_suspended: bool,
-    ) -> Self {
-        Self {
-            bucket: bucket.to_string(),
-            object: object.to_string(),
-            version_id: info.version_id.map(|id| id.to_string()),
-            versioned,
-            version_suspended,
-            data_dir: info.data_dir.map(|id| id.to_string()),
-            etag: info.etag.clone(),
-            mod_time: info.mod_time.map(|time| time.to_string()),
-        }
-    }
-
-    pub(crate) fn lookup_options(&self) -> crate::object_api::ObjectOptions {
-        crate::object_api::ObjectOptions {
-            version_id: self.version_id.clone(),
-            versioned: self.versioned,
-            version_suspended: self.version_suspended,
-            ..Default::default()
-        }
-    }
-
-    pub(crate) fn matches(&self, info: &ObjectInfo) -> bool {
-        if self.bucket != info.bucket {
-            return false;
-        }
-        if let Some(version_id) = &self.version_id {
-            return info.version_id.map(|id| id.to_string()).as_deref() == Some(version_id.as_str())
-                && self.data_dir == info.data_dir.map(|id| id.to_string());
-        }
-        if self.data_dir.is_some() {
-            return self.data_dir == info.data_dir.map(|id| id.to_string());
-        }
-        self.etag.is_some()
-            && self.etag == info.etag
-            && self.mod_time.is_some()
-            && self.mod_time == info.mod_time.map(|time| time.to_string())
-    }
-
-    pub(crate) fn has_stable_identity(&self) -> bool {
-        self.version_id.is_some() || self.data_dir.is_some() || (self.etag.is_some() && self.mod_time.is_some())
-    }
-}
-
 #[derive(Debug, Clone)]
 #[allow(unused_assignments)]
 pub struct Jentry {
@@ -368,8 +294,6 @@ pub struct Jentry {
     pub(crate) backend_identity: Option<TierDestinationId>,
     pub(crate) version_id_exact: bool,
     pub(crate) version_state: rustfs_filemeta::TransitionVersionState,
-    pub(crate) state: TierDeleteJournalState,
-    pub(crate) source: Option<TierDeleteSourceIdentity>,
 }
 
 impl ExpiryOp for Jentry {
@@ -630,48 +554,9 @@ pub fn transitioned_force_delete_journal_entry(
             rustfs_filemeta::TransitionVersionState::SuspendedNull | rustfs_filemeta::TransitionVersionState::Exact
         ),
         version_state: transition_version_state,
-        state: TierDeleteJournalState::Committed,
-        source: None,
     })
 }
 
-pub(crate) fn attach_tier_delete_source(
-    je: &mut Jentry,
-    bucket: &str,
-    object: &str,
-    info: &ObjectInfo,
-    versioned: bool,
-    version_suspended: bool,
-) {
-    je.state = TierDeleteJournalState::Prepared;
-    je.source = Some(TierDeleteSourceIdentity::from_object_info(
-        bucket,
-        object,
-        info,
-        versioned,
-        version_suspended,
-    ));
-}
-
-pub(crate) fn transitioned_delete_journal_entry_for_source(
-    version_id: Option<Uuid>,
-    versioned: bool,
-    suspended: bool,
-    bucket: &str,
-    object: &str,
-    source: &ObjectInfo,
-) -> Option<Jentry> {
-    let mut je = transitioned_delete_journal_entry(
-        version_id,
-        versioned,
-        suspended,
-        &source.transitioned_object,
-        source.transition_version_state,
-    )?;
-    attach_tier_delete_source(&mut je, bucket, object, source, versioned, suspended);
-    Some(je)
-}
-
 #[cfg(test)]
 mod test {
     use crate::client::signer_error::invalid_utf8_header_error;

crates/ecstore/src/bucket/metadata.rs

@@ -16,9 +16,8 @@ use super::msgp_decode::{read_msgp_ext8_time, skip_msgp_value, write_msgp_time};
 use super::object_lock::ObjectLockApi;
 use super::versioning::VersioningApi;
 use super::{quota::BucketQuota, target::BucketTargets};
-use crate::bucket::replication::invalid_replication_config_status_field;
 use crate::bucket::utils::deserialize;
-use crate::config::com::{read_config, read_config_preserve_empty, save_config};
+use crate::config::com::{read_config, save_config};
 use crate::disk::BUCKET_META_PREFIX;
 use crate::error::{Error, Result};
 use crate::runtime::sources as runtime_sources;
@@ -26,9 +25,9 @@ use crate::store::ECStore;
 use byteorder::{BigEndian, ByteOrder, LittleEndian};
 use rustfs_policy::policy::BucketPolicy;
 use s3s::dto::{
-    AccelerateConfiguration, BucketLifecycleConfiguration, BucketLoggingStatus, BucketVersioningStatus, CORSConfiguration,
-    NotificationConfiguration, ObjectLockConfiguration, PublicAccessBlockConfiguration, ReplicationConfiguration,
-    RequestPaymentConfiguration, ServerSideEncryptionConfiguration, Tagging, VersioningConfiguration, WebsiteConfiguration,
+    AccelerateConfiguration, BucketLifecycleConfiguration, BucketLoggingStatus, CORSConfiguration, NotificationConfiguration,
+    ObjectLockConfiguration, PublicAccessBlockConfiguration, ReplicationConfiguration, RequestPaymentConfiguration,
+    ServerSideEncryptionConfiguration, Tagging, VersioningConfiguration, WebsiteConfiguration,
 };
 use serde::Serializer;
 use sha2::{Digest, Sha256};
@@ -37,7 +36,6 @@ use std::io::{Read, Write};
 use std::sync::Arc;
 use time::{Date, OffsetDateTime, PrimitiveDateTime, Time as CivilTime, UtcOffset};
 use tracing::error;
-use uuid::Uuid;
 
 fn read_msgp_str<R: Read>(rd: &mut R) -> Result<String> {
     let len = rmp::decode::read_str_len(rd)? as usize;
@@ -227,7 +225,6 @@ fn write_bin_field<W: Write>(wr: &mut W, key: &str, val: &[u8]) -> Result<()> {
 }
 
 pub const BUCKET_METADATA_FILE: &str = ".metadata.bin";
-pub const BUCKET_INCARNATION_FILE: &str = ".bucket-incarnation";
 pub const BUCKET_METADATA_FORMAT: u16 = 1;
 pub const BUCKET_METADATA_VERSION: u16 = 1;
 
@@ -279,8 +276,6 @@ pub struct BucketMetadata {
     pub name: String,
     pub created: OffsetDateTime,
     pub lock_enabled: bool, // While marked as unused, it may need to be retained
-    pub bucket_incarnation_id: Uuid,
-    pub(crate) bucket_incarnation_sidecar: bool,
     pub policy_config_json: Vec<u8>,
     pub notification_config_xml: Vec<u8>,
     pub lifecycle_config_xml: Vec<u8>,
@@ -351,8 +346,6 @@ impl Default for BucketMetadata {
             name: Default::default(),
             created: OffsetDateTime::UNIX_EPOCH,
             lock_enabled: Default::default(),
-            bucket_incarnation_id: Uuid::nil(),
-            bucket_incarnation_sidecar: false,
             policy_config_json: Default::default(),
             notification_config_xml: Default::default(),
             lifecycle_config_xml: Default::default(),
@@ -420,7 +413,6 @@ impl BucketMetadata {
     pub fn new(name: &str) -> Self {
         BucketMetadata {
             name: name.to_string(),
-            bucket_incarnation_id: Uuid::new_v4(),
             ..Default::default()
         }
     }
@@ -486,11 +478,6 @@ impl BucketMetadata {
                 "Name" => self.name = read_msgp_str(rd)?,
                 "Created" => self.created = read_msgp_time_value(rd)?,
                 "LockEnabled" => self.lock_enabled = read_msgp_bool(rd)?,
-                "BucketIncarnationID" => {
-                    let bytes = read_msgp_bin(rd)?;
-                    self.bucket_incarnation_id =
-                        Uuid::from_slice(&bytes).map_err(|err| Error::other(format!("invalid BucketIncarnationID: {err}")))?;
-                }
                 "PolicyConfigJSON" | "PolicyConfigJson" => self.policy_config_json = read_msgp_bin(rd)?,
                 "NotificationConfigXML" | "NotificationConfigXml" => self.notification_config_xml = read_msgp_bin(rd)?,
                 "LifecycleConfigXML" | "LifecycleConfigXml" => self.lifecycle_config_xml = read_msgp_bin(rd)?,
@@ -547,8 +534,8 @@ impl BucketMetadata {
 
     /// Encode to msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
     pub fn encode_to<W: Write>(&self, wr: &mut W) -> Result<()> {
-        // Map size: MinIO fields (25) + RustFS extensions (19)
-        let map_len: u32 = 44;
+        // Map size: MinIO fields (25) + RustFS extensions (18)
+        let map_len: u32 = 43;
         rmp::encode::write_map_len(wr, map_len)?;
 
         // MinIO field order (same as Go struct)
@@ -561,8 +548,6 @@ impl BucketMetadata {
         rmp::encode::write_str(wr, "LockEnabled")?;
         rmp::encode::write_bool(wr, self.lock_enabled)?;
 
-        write_bin_field(wr, "BucketIncarnationID", self.bucket_incarnation_id.as_bytes())?;
-
         write_bin_field(wr, "PolicyConfigJSON", &self.policy_config_json)?;
         write_bin_field(wr, "NotificationConfigXML", &self.notification_config_xml)?;
         write_bin_field(wr, "LifecycleConfigXML", &self.lifecycle_config_xml)?;
@@ -762,41 +747,15 @@ impl BucketMetadata {
                 self.quota_config_updated_at = updated;
             }
             OBJECT_LOCK_CONFIG => {
-                self.object_lock_config = None;
-                if !data.is_empty() {
-                    self.lock_enabled = true;
-                }
                 self.object_lock_config_xml = data;
                 self.object_lock_config_updated_at = updated;
             }
             BUCKET_VERSIONING_CONFIG => {
-                let config = if data.is_empty() {
-                    None
-                } else {
-                    let config = deserialize::<VersioningConfiguration>(&data)?;
-                    if config.status.as_ref().is_some_and(|status| {
-                        !matches!(status.as_str(), BucketVersioningStatus::ENABLED | BucketVersioningStatus::SUSPENDED)
-                    }) {
-                        return Err(Error::other("bucket versioning configuration has an invalid status"));
-                    }
-                    Some(config)
-                };
                 self.versioning_config_xml = data;
-                self.versioning_config = config;
                 self.versioning_config_updated_at = updated;
             }
             BUCKET_REPLICATION_CONFIG => {
-                let config = if data.is_empty() {
-                    None
-                } else {
-                    let config = deserialize::<ReplicationConfiguration>(&data)?;
-                    if let Some(field) = invalid_replication_config_status_field(&config) {
-                        return Err(Error::other(format!("replication field {field} has an invalid status")));
-                    }
-                    Some(config)
-                };
                 self.replication_config_xml = data;
-                self.replication_config = config;
                 self.replication_config_updated_at = updated;
             }
             BUCKET_TARGETS_FILE => {
@@ -866,6 +825,7 @@ impl BucketMetadata {
     /// ambient (first) one. [`BucketMetadata::save`] keeps the ambient default.
     pub async fn save_with_store(&mut self, store: std::sync::Arc<crate::store::ECStore>) -> Result<()> {
         self.parse_all_configs()?;
+
         let mut buf: Vec<u8> = vec![0; 4];
 
         LittleEndian::write_u16(&mut buf[0..2], BUCKET_METADATA_FORMAT);
@@ -946,7 +906,6 @@ impl BucketMetadata {
                 "Failed to parse bucket metadata config"
             );
         }
-        self.versioning_config = None;
         if !self.versioning_config_xml.is_empty()
             && let Err(e) =
                 deserialize::<VersioningConfiguration>(&self.versioning_config_xml).map(|c| self.versioning_config = Some(c))
@@ -1001,7 +960,6 @@ impl BucketMetadata {
                 "Failed to parse bucket metadata config"
             );
         }
-        self.replication_config = None;
         if !self.replication_config_xml.is_empty()
             && let Err(e) =
                 deserialize::<ReplicationConfiguration>(&self.replication_config_xml).map(|c| self.replication_config = Some(c))
@@ -1133,29 +1091,6 @@ impl BucketMetadata {
     }
 }
 
-pub(crate) async fn load_bucket_incarnation(api: Arc<ECStore>, bucket: &str) -> Result<Option<Uuid>> {
-    let path = format!("{BUCKET_META_PREFIX}/{bucket}/{BUCKET_INCARNATION_FILE}");
-    let data = match read_config_preserve_empty(api, &path).await {
-        Ok(data) => data,
-        Err(Error::ConfigNotFound) => return Ok(None),
-        Err(err) => return Err(err),
-    };
-    let incarnation =
-        Uuid::from_slice(&data).map_err(|err| Error::other(format!("persisted bucket incarnation is invalid: {err}")))?;
-    if incarnation.is_nil() {
-        return Err(Error::other("persisted bucket incarnation is nil"));
-    }
-    Ok(Some(incarnation))
-}
-
-pub(crate) async fn save_bucket_incarnation(api: Arc<ECStore>, bucket: &str, incarnation: Uuid) -> Result<()> {
-    if incarnation.is_nil() {
-        return Err(Error::other("cannot persist a nil bucket incarnation"));
-    }
-    let path = format!("{BUCKET_META_PREFIX}/{bucket}/{BUCKET_INCARNATION_FILE}");
-    save_config(api, &path, incarnation.as_bytes().to_vec()).await
-}
-
 pub async fn load_bucket_metadata(api: Arc<ECStore>, bucket: &str) -> Result<BucketMetadata> {
     load_bucket_metadata_parse(api, bucket, true).await
 }
@@ -1183,23 +1118,6 @@ pub(crate) async fn load_bucket_metadata_parse_with_presence(
         }
     };
 
-    let incarnation = load_bucket_incarnation(api, bucket).await?;
-    if persisted {
-        if let Some(incarnation) = incarnation {
-            if !bm.bucket_incarnation_id.is_nil() && bm.bucket_incarnation_id != incarnation {
-                return Err(Error::other("bucket incarnation sidecar does not match bucket metadata"));
-            }
-            bm.bucket_incarnation_id = incarnation;
-            bm.bucket_incarnation_sidecar = true;
-        } else if !bm.bucket_incarnation_id.is_nil() {
-            return Err(Error::other(format!(
-                "bucket incarnation sidecar is missing for new-format metadata: {bucket}"
-            )));
-        }
-    } else if incarnation.is_some() {
-        return Err(Error::other("bucket incarnation sidecar exists without bucket metadata"));
-    }
-
     bm.default_timestamps();
 
     if parse {
@@ -1267,10 +1185,6 @@ mod test {
         // Same 4-byte format|version header (1|1) and msgpack layout as MinIO.
         BucketMetadata::check_header(&blob).expect("valid .metadata.bin header");
         let mut bm = BucketMetadata::unmarshal(&blob[4..]).expect("unmarshal MinIO bucket metadata");
-        assert!(
-            bm.bucket_incarnation_id.is_nil(),
-            "legacy MinIO metadata has no RustFS bucket incarnation field"
-        );
 
         // Raw config fields survive the msgpack decode (PascalCase MinIO field names).
         assert_eq!(bm.name, "interop");
@@ -1353,42 +1267,6 @@ mod test {
         let new = BucketMetadata::unmarshal(&buf).unwrap();
 
         assert_eq!(bm.name, new.name);
-        assert!(!bm.bucket_incarnation_id.is_nil());
-        assert_eq!(bm.bucket_incarnation_id, new.bucket_incarnation_id);
-    }
-
-    #[test]
-    fn bucket_incarnation_msgpack_rejects_invalid_binary_length() {
-        let mut fixture = Vec::new();
-        rmp::encode::write_map_len(&mut fixture, 1).unwrap();
-        rmp::encode::write_str(&mut fixture, "BucketIncarnationID").unwrap();
-        rmp::encode::write_bin(&mut fixture, &[0_u8; 15]).unwrap();
-
-        let err = BucketMetadata::unmarshal(&fixture).expect_err("non-UUID incarnation bytes must fail closed");
-        assert!(err.to_string().contains("invalid BucketIncarnationID"));
-    }
-
-    #[test]
-    fn same_name_bucket_metadata_gets_a_new_incarnation() {
-        let old = BucketMetadata::new("recreated");
-        let new = BucketMetadata::new("recreated");
-
-        assert!(!old.bucket_incarnation_id.is_nil());
-        assert!(!new.bucket_incarnation_id.is_nil());
-        assert_ne!(old.bucket_incarnation_id, new.bucket_incarnation_id);
-    }
-
-    #[test]
-    fn site_replication_config_updates_cannot_replace_bucket_incarnation() {
-        let mut metadata = BucketMetadata::new("site-replication-update");
-        let incarnation = metadata.bucket_incarnation_id;
-
-        metadata
-            .update_config(BUCKET_POLICY_CONFIG, br#"{"Version":"2012-10-17","Statement":[]}"#.to_vec())
-            .unwrap();
-        metadata.update_config(OBJECT_LOCK_CONFIG, Vec::new()).unwrap();
-
-        assert_eq!(metadata.bucket_incarnation_id, incarnation);
     }
 
     #[test]
@@ -1467,66 +1345,6 @@ mod test {
         assert!(bm.tagging_config.is_none());
     }
 
-    #[test]
-    fn delete_admission_configs_update_parsed_state_atomically() {
-        let mut bm = BucketMetadata::new("test-bucket");
-        let versioning_xml = b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>";
-        let replication_xml = b"<ReplicationConfiguration><Role>arn:aws:s3:::target-bucket</Role><Rule><ID>rule1</ID><Status>Enabled</Status><Prefix></Prefix><Destination><Bucket>arn:aws:s3:::target-bucket</Bucket></Destination></Rule></ReplicationConfiguration>";
-
-        bm.update_config(BUCKET_VERSIONING_CONFIG, versioning_xml.to_vec())
-            .expect("valid versioning config should update parsed state");
-        bm.update_config(BUCKET_REPLICATION_CONFIG, replication_xml.to_vec())
-            .expect("valid replication config should update parsed state");
-
-        assert!(bm.versioning_config.as_ref().is_some_and(VersioningConfiguration::enabled));
-        assert_eq!(
-            bm.replication_config.as_ref().map(|config| config.role.as_str()),
-            Some("arn:aws:s3:::target-bucket")
-        );
-
-        assert!(
-            bm.update_config(BUCKET_VERSIONING_CONFIG, b"<VersioningConfiguration>".to_vec())
-                .is_err()
-        );
-        assert!(
-            bm.update_config(BUCKET_REPLICATION_CONFIG, b"<ReplicationConfiguration>".to_vec())
-                .is_err()
-        );
-
-        assert_eq!(bm.versioning_config_xml, versioning_xml);
-        assert_eq!(bm.replication_config_xml, replication_xml);
-        assert!(bm.versioning_config.as_ref().is_some_and(VersioningConfiguration::enabled));
-        assert_eq!(
-            bm.replication_config.as_ref().map(|config| config.role.as_str()),
-            Some("arn:aws:s3:::target-bucket")
-        );
-
-        assert!(
-            bm.update_config(
-                BUCKET_VERSIONING_CONFIG,
-                b"<VersioningConfiguration><Status>Enabld</Status></VersioningConfiguration>".to_vec(),
-            )
-            .is_err()
-        );
-        assert!(
-            bm.update_config(
-                BUCKET_REPLICATION_CONFIG,
-                b"<ReplicationConfiguration><Role>arn:aws:s3:::target-bucket</Role><Rule><ID>rule1</ID><Status>Enabld</Status><Prefix></Prefix><Destination><Bucket>arn:aws:s3:::target-bucket</Bucket></Destination></Rule></ReplicationConfiguration>".to_vec(),
-            )
-            .is_err()
-        );
-        assert_eq!(bm.versioning_config_xml, versioning_xml);
-        assert_eq!(bm.replication_config_xml, replication_xml);
-
-        bm.versioning_config_xml = b"<VersioningConfiguration>".to_vec();
-        bm.replication_config_xml = b"<ReplicationConfiguration>".to_vec();
-        bm.parse_all_configs()
-            .expect("bulk config parsing reports malformed fields through cleared typed state");
-
-        assert!(bm.versioning_config.is_none());
-        assert!(bm.replication_config.is_none());
-    }
-
     #[tokio::test]
     async fn marshal_msg_complete_example() {
         // Create a complete BucketMetadata with various configurations

crates/ecstore/src/bucket/object_lock/objectlock_sys.rs

@@ -12,12 +12,10 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::bucket::metadata_sys::{ObjectLockConfigState, get_object_lock_config, get_object_lock_config_state};
+use crate::bucket::metadata_sys::get_object_lock_config;
 use crate::bucket::object_lock::objectlock;
-use crate::error::{Error, Result, StorageError};
 use crate::object_api::ObjectInfo;
-use s3s::dto::{Date, DefaultRetention, ObjectLockConfiguration, ObjectLockLegalHoldStatus, ObjectLockRetentionMode};
-use s3s::header::{X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE};
+use s3s::dto::{DefaultRetention, ObjectLockLegalHoldStatus, ObjectLockRetentionMode};
 use std::sync::Arc;
 use time::OffsetDateTime;
 
@@ -39,20 +37,6 @@ impl BucketObjectLockSys {
     }
 }
 
-pub(crate) fn ensure_recursive_force_delete_allowed_for_state(bucket: &str, state: &ObjectLockConfigState) -> Result<()> {
-    match state {
-        ObjectLockConfigState::ConfirmedAbsent => Ok(()),
-        ObjectLockConfigState::Configured { .. } => Err(StorageError::InvalidArgument(
-            bucket.to_string(),
-            String::new(),
-            "force-delete is forbidden on Object Locking enabled buckets".to_string(),
-        )),
-        ObjectLockConfigState::Fabricated => {
-            Err(Error::other(format!("bucket Object Lock metadata is not authoritative: {bucket}")))
-        }
-    }
-}
-
 /// Check if a retention period is still active based on mode and retain_until_date
 pub fn is_retention_active(mode: &str, retain_until_date: Option<&s3s::dto::Date>) -> bool {
     if mode != ObjectLockRetentionMode::COMPLIANCE && mode != ObjectLockRetentionMode::GOVERNANCE {
@@ -221,122 +205,71 @@ fn check_retention_blocks_deletion(
     None
 }
 
-/// Check an object's lock metadata using an already resolved bucket Object
-/// Lock configuration. `None` means the configuration is confirmed absent.
-///
 /// # S3 Standard Behavior
 /// - COMPLIANCE mode: Cannot be deleted even with bypass header
 /// - GOVERNANCE mode: Can be deleted if bypass_governance is true (caller must verify s3:BypassGovernanceRetention permission)
 /// - Legal Hold: Cannot be bypassed regardless of mode
-pub(crate) fn check_object_lock_for_deletion_with_config(
-    config: Option<&ObjectLockConfiguration>,
-    obj_info: &ObjectInfo,
-    bypass_governance: bool,
-) -> Result<Option<ObjectLockBlockReason>> {
-    if obj_info.delete_marker {
-        return Ok(None);
-    }
-
-    if let Some(status) = obj_info.user_defined.get(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str()) {
-        if status.eq_ignore_ascii_case(ObjectLockLegalHoldStatus::ON) {
-            return Ok(Some(ObjectLockBlockReason::LegalHold));
-        }
-        if !status.eq_ignore_ascii_case(ObjectLockLegalHoldStatus::OFF) {
-            return Err(Error::other("persisted object legal-hold metadata is invalid"));
-        }
-    }
-
-    let mode = obj_info.user_defined.get(X_AMZ_OBJECT_LOCK_MODE.as_str());
-    let retain_until = obj_info.user_defined.get(X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str());
-    let explicit_ret = match (mode, retain_until) {
-        (None, None) => None,
-        (Some(mode), Some(retain_until)) => {
-            let mode =
-                objectlock::parse_ret_mode(mode).ok_or_else(|| Error::other("persisted object retention mode is invalid"))?;
-            let retain_until = OffsetDateTime::parse(retain_until, &time::format_description::well_known::Iso8601::DEFAULT)
-                .map(Date::from)
-                .map_err(|_| Error::other("persisted object retention date is invalid"))?;
-            Some((mode, retain_until))
-        }
-        _ => return Err(Error::other("persisted object retention metadata is incomplete")),
-    };
-
-    if let Some((mode, retain_until)) = &explicit_ret {
-        let mode_str = mode.as_str();
-        if is_retention_active(mode_str, Some(retain_until))
-            && let Some(reason) =
-                check_retention_blocks_deletion(mode_str, Some(OffsetDateTime::from(retain_until.clone())), bypass_governance)
-        {
-            return Ok(Some(reason));
-        }
-    }
-
-    if explicit_ret.is_none()
-        && let Some(default_retention) = config.and_then(|config| config.rule.as_ref()?.default_retention.as_ref())
-        && let Some(mode) = &default_retention.mode
-    {
-        let mode_str = mode.as_str();
-        if mode_str == ObjectLockRetentionMode::COMPLIANCE || mode_str == ObjectLockRetentionMode::GOVERNANCE {
-            // Calculate retention expiration date from object modification time
-            let mod_time = obj_info
-                .mod_time
-                .ok_or_else(|| Error::other("persisted object modification time is missing"))?;
-            let now = objectlock::utc_now_ntp();
-            let retain_until = if let Some(days) = default_retention.days {
-                mod_time.saturating_add(time::Duration::days(i64::from(days)))
-            } else {
-                let years = default_retention
-                    .years
-                    .ok_or_else(|| Error::other("persisted bucket Object Lock retention period is invalid"))?;
-                add_years(mod_time, years)
-            };
-
-            if retain_until.unix_timestamp() > now.unix_timestamp()
-                && let Some(reason) = check_retention_blocks_deletion(mode_str, Some(retain_until), bypass_governance)
-            {
-                return Ok(Some(reason));
-            }
-        }
-    }
-
-    Ok(None)
-}
-
-pub(crate) fn check_object_lock_for_deletion_with_state(
-    state: &ObjectLockConfigState,
-    obj_info: &ObjectInfo,
-    bypass_governance: bool,
-) -> Result<Option<ObjectLockBlockReason>> {
-    match state {
-        ObjectLockConfigState::Configured { config, .. } => {
-            check_object_lock_for_deletion_with_config(Some(config), obj_info, bypass_governance)
-        }
-        ObjectLockConfigState::ConfirmedAbsent => check_object_lock_for_deletion_with_config(None, obj_info, bypass_governance),
-        ObjectLockConfigState::Fabricated => Err(Error::other("bucket Object Lock metadata is not authoritative")),
-    }
-}
-
-/// Compatibility wrapper for callers that predate fallible metadata lookup.
-/// An authority/read/parse failure is represented as a blocking reason rather
-/// than the old fail-open `None` result.
 pub async fn check_object_lock_for_deletion(
     bucket: &str,
     obj_info: &ObjectInfo,
     bypass_governance: bool,
 ) -> Option<ObjectLockBlockReason> {
-    match get_object_lock_config_state(bucket)
-        .await
-        .and_then(|state| check_object_lock_for_deletion_with_state(&state, obj_info, bypass_governance))
-    {
-        Ok(reason) => reason,
-        Err(_) => Some(ObjectLockBlockReason::LegalHold),
+    if obj_info.delete_marker {
+        return None;
     }
+
+    // 1. Check legal hold - cannot be bypassed (reuse has_legal_hold)
+    if has_legal_hold(&obj_info.user_defined) {
+        return Some(ObjectLockBlockReason::LegalHold);
+    }
+
+    // 2. Check explicit retention
+    let explicit_ret = objectlock::get_object_retention_meta(&obj_info.user_defined);
+    if let Some(mode) = &explicit_ret.mode {
+        let mode_str = mode.as_str();
+        if is_retention_active(mode_str, explicit_ret.retain_until_date.as_ref())
+            && let Some(reason) = check_retention_blocks_deletion(
+                mode_str,
+                explicit_ret.retain_until_date.map(OffsetDateTime::from),
+                bypass_governance,
+            )
+        {
+            return Some(reason);
+        }
+    }
+
+    // 3. Check default retention only if no explicit retention is set
+    if explicit_ret.mode.is_none()
+        && let Some(default_retention) = BucketObjectLockSys::get(bucket).await
+        && let Some(mode) = &default_retention.mode
+    {
+        let mode_str = mode.as_str();
+        if mode_str == ObjectLockRetentionMode::COMPLIANCE || mode_str == ObjectLockRetentionMode::GOVERNANCE {
+            // Calculate retention expiration date from object modification time
+            if let Some(mod_time) = obj_info.mod_time {
+                let now = objectlock::utc_now_ntp();
+                let retain_until = if let Some(days) = default_retention.days {
+                    mod_time.saturating_add(time::Duration::days(days as i64))
+                } else {
+                    let years = default_retention.years?;
+                    add_years(mod_time, years)
+                };
+
+                if retain_until.unix_timestamp() > now.unix_timestamp()
+                    && let Some(reason) = check_retention_blocks_deletion(mode_str, Some(retain_until), bypass_governance)
+                {
+                    return Some(reason);
+                }
+            }
+        }
+    }
+
+    None
 }
 
 #[cfg(test)]
 mod tests {
     use super::*;
-    use s3s::dto::{ObjectLockEnabled, ObjectLockRule};
     use time::{Date, Month, PrimitiveDateTime, Time};
 
     fn make_datetime(year: i32, month: u8, day: u8) -> OffsetDateTime {
@@ -345,160 +278,6 @@ mod tests {
         PrimitiveDateTime::new(date, time).assume_utc()
     }
 
-    fn default_retention_config(mode: &'static str) -> ObjectLockConfiguration {
-        ObjectLockConfiguration {
-            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
-            rule: Some(ObjectLockRule {
-                default_retention: Some(DefaultRetention {
-                    mode: Some(ObjectLockRetentionMode::from_static(mode)),
-                    days: Some(30),
-                    years: None,
-                }),
-            }),
-        }
-    }
-
-    #[test]
-    fn deletion_with_config_blocks_active_default_compliance_even_with_bypass() {
-        let config = default_retention_config(ObjectLockRetentionMode::COMPLIANCE);
-        let obj_info = ObjectInfo {
-            mod_time: Some(OffsetDateTime::now_utc()),
-            ..Default::default()
-        };
-
-        let result = check_object_lock_for_deletion_with_config(Some(&config), &obj_info, true);
-
-        assert!(matches!(result, Ok(Some(ObjectLockBlockReason::Retention { .. }))));
-    }
-
-    #[test]
-    fn deletion_with_config_allows_active_default_governance_with_bypass() {
-        let config = default_retention_config(ObjectLockRetentionMode::GOVERNANCE);
-        let obj_info = ObjectInfo {
-            mod_time: Some(OffsetDateTime::now_utc()),
-            ..Default::default()
-        };
-
-        assert!(matches!(
-            check_object_lock_for_deletion_with_config(Some(&config), &obj_info, true),
-            Ok(None)
-        ));
-    }
-
-    #[test]
-    fn deletion_with_default_retention_rejects_missing_object_mod_time() {
-        let config = default_retention_config(ObjectLockRetentionMode::COMPLIANCE);
-
-        let err = check_object_lock_for_deletion_with_config(Some(&config), &ObjectInfo::default(), false)
-            .expect_err("default retention needs an authoritative object modification time");
-
-        assert!(err.to_string().contains("modification time"));
-    }
-
-    #[test]
-    fn deletion_with_confirmed_absence_still_blocks_explicit_compliance() {
-        let retain_until = OffsetDateTime::now_utc() + time::Duration::days(30);
-        let mut user_defined = std::collections::HashMap::new();
-        user_defined.insert("x-amz-object-lock-mode".to_string(), ObjectLockRetentionMode::COMPLIANCE.to_string());
-        user_defined.insert(
-            "x-amz-object-lock-retain-until-date".to_string(),
-            retain_until
-                .format(&time::format_description::well_known::Rfc3339)
-                .expect("retain-until date should format"),
-        );
-        let obj_info = ObjectInfo {
-            user_defined: Arc::new(user_defined),
-            ..Default::default()
-        };
-
-        let result = check_object_lock_for_deletion_with_config(None, &obj_info, true);
-
-        assert!(matches!(result, Ok(Some(ObjectLockBlockReason::Retention { .. }))));
-    }
-
-    #[test]
-    fn deletion_with_fabricated_bucket_metadata_fails_closed() {
-        let err = check_object_lock_for_deletion_with_state(&ObjectLockConfigState::Fabricated, &ObjectInfo::default(), false)
-            .expect_err("non-authoritative Object Lock metadata must block deletion");
-
-        assert!(err.to_string().contains("not authoritative"));
-    }
-
-    #[test]
-    fn recursive_force_delete_with_fabricated_bucket_metadata_fails_closed() {
-        let err = ensure_recursive_force_delete_allowed_for_state("bucket", &ObjectLockConfigState::Fabricated)
-            .expect_err("non-authoritative Object Lock metadata must block recursive deletion");
-
-        assert!(err.to_string().contains("not authoritative"));
-    }
-
-    #[test]
-    fn deletion_rejects_incomplete_persisted_retention_metadata() {
-        let mut user_defined = std::collections::HashMap::new();
-        user_defined.insert(
-            X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(),
-            ObjectLockRetentionMode::COMPLIANCE.to_string(),
-        );
-        let obj_info = ObjectInfo {
-            user_defined: Arc::new(user_defined),
-            ..Default::default()
-        };
-
-        let err = check_object_lock_for_deletion_with_config(None, &obj_info, false)
-            .expect_err("mode without retain-until date must fail closed");
-
-        assert!(err.to_string().contains("incomplete"));
-    }
-
-    #[test]
-    fn deletion_rejects_each_malformed_persisted_retention_shape() {
-        let valid_date = (OffsetDateTime::now_utc() + time::Duration::days(30))
-            .format(&time::format_description::well_known::Rfc3339)
-            .expect("retain-until date should format");
-        let cases = [
-            ("invalid mode", Some("INVALID"), Some(valid_date.as_str()), "retention mode"),
-            (
-                "invalid date",
-                Some(ObjectLockRetentionMode::COMPLIANCE),
-                Some("not-a-date"),
-                "retention date",
-            ),
-            ("date only", None, Some(valid_date.as_str()), "incomplete"),
-        ];
-
-        for (case, mode, retain_until, expected) in cases {
-            let mut user_defined = std::collections::HashMap::new();
-            if let Some(mode) = mode {
-                user_defined.insert(X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), mode.to_string());
-            }
-            if let Some(retain_until) = retain_until {
-                user_defined.insert(X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), retain_until.to_string());
-            }
-            let obj_info = ObjectInfo {
-                user_defined: Arc::new(user_defined),
-                ..Default::default()
-            };
-
-            let err = check_object_lock_for_deletion_with_config(None, &obj_info, false).expect_err(case);
-            assert!(err.to_string().contains(expected), "unexpected {case} error: {err}");
-        }
-    }
-
-    #[test]
-    fn deletion_rejects_invalid_persisted_legal_hold_metadata() {
-        let mut user_defined = std::collections::HashMap::new();
-        user_defined.insert(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str().to_string(), "INVALID".to_string());
-        let obj_info = ObjectInfo {
-            user_defined: Arc::new(user_defined),
-            ..Default::default()
-        };
-
-        let err = check_object_lock_for_deletion_with_config(None, &obj_info, false)
-            .expect_err("invalid legal-hold value must fail closed");
-
-        assert!(err.to_string().contains("legal-hold"));
-    }
-
     #[test]
     fn test_add_years_normal() {
         // Normal case: add 1 year to a regular date

crates/ecstore/src/bucket/quota/checker.rs

@@ -13,7 +13,7 @@
 // limitations under the License.
 
 use super::{BucketQuota, QuotaCheckResult, QuotaError, QuotaOperation};
-use crate::bucket::metadata_sys::{BucketMetadataSys, update, update_if_incarnation};
+use crate::bucket::metadata_sys::{BucketMetadataSys, update};
 use crate::data_usage::get_bucket_usage_memory;
 use rustfs_common::metrics::Metric;
 use rustfs_config::QUOTA_CONFIG_FILE;
@@ -145,35 +145,14 @@ impl QuotaChecker {
     }
 
     pub async fn set_quota_config(&mut self, bucket: &str, quota: BucketQuota) -> Result<OffsetDateTime, QuotaError> {
-        self.set_quota_config_for_incarnation(bucket, quota, None).await
-    }
-
-    pub async fn set_quota_config_if_incarnation(
-        &mut self,
-        bucket: &str,
-        quota: BucketQuota,
-        expected_incarnation_id: uuid::Uuid,
-    ) -> Result<OffsetDateTime, QuotaError> {
-        self.set_quota_config_for_incarnation(bucket, quota, Some(expected_incarnation_id))
-            .await
-    }
-
-    async fn set_quota_config_for_incarnation(
-        &mut self,
-        bucket: &str,
-        quota: BucketQuota,
-        expected_incarnation_id: Option<uuid::Uuid>,
-    ) -> Result<OffsetDateTime, QuotaError> {
         let json_data = serde_json::to_vec(&quota).map_err(|e| QuotaError::InvalidConfig {
             reason: format!("Failed to serialize quota config: {}", e),
         })?;
         let start_time = Instant::now();
 
-        let updated_at = match expected_incarnation_id {
-            Some(incarnation_id) => update_if_incarnation(bucket, QUOTA_CONFIG_FILE, json_data, incarnation_id).await,
-            None => update(bucket, QUOTA_CONFIG_FILE, json_data).await,
-        }
-        .map_err(QuotaError::StorageError)?;
+        let updated_at = update(bucket, QUOTA_CONFIG_FILE, json_data)
+            .await
+            .map_err(QuotaError::StorageError)?;
 
         rustfs_common::metrics::Metrics::inc_time(Metric::QuotaSync, start_time.elapsed());
         Ok(updated_at)