test(notify): cover prefix suffix event dispatch (#2960)

Co-authored-by: houseme <housemecn@gmail.com>

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: rustfs-release-version-bump
+description: Prepare a RustFS release branch like PR #2957 by bumping versioned files, aligning release assets, running required verification, and finishing with commit, push, and gh PR creation. Use when publishing a new RustFS alpha, beta, or stable release.
+---
+
+# RustFS Release Version Bump
+
+Use this skill when the task is to prepare a new RustFS version release branch following the pattern validated in PR `#2957`.
+
+## Read first
+
+- Read `AGENTS.md`.
+- Read `.github/pull_request_template.md`.
+- Inspect the current branch diff against `origin/main`.
+- Do not assume every release file should change in the same way every time.
+
+## Scope validated by PR #2957
+
+The `1.0.0-beta.3` release branch updated these files:
+
+- `Cargo.toml`
+- `Cargo.lock`
+- `README.md`
+- `README_ZH.md`
+- `flake.nix`
+- `helm/rustfs/Chart.yaml`
+- `rustfs.spec`
+
+Treat this file list as the default checklist for future release bumps. Only drop a file if the repository's current release pattern clearly says it is no longer part of the publish flow.
+
+## Workflow
+
+1. Confirm release intent
+- Identify the target version string exactly, for example `1.0.0-beta.4`.
+- Check whether the user wants only local commit preparation or the full `commit + push + PR` flow.
+- Compare the branch against `origin/main` and isolate only release-related edits.
+- If the target version is not explicit, stop and ask for it before editing.
+
+2. Update core Rust workspace versions
+- Bump the workspace version in `Cargo.toml`.
+- Bump all internal workspace crate versions in `Cargo.toml`.
+- Ensure `Cargo.lock` reflects the same release version for workspace packages.
+- Re-read the diff and verify there are no partial version leftovers.
+
+3. Align release assets
+- Update versioned Docker examples in `README.md` and `README_ZH.md` using the `<version>` tag form, for example `rustfs/rustfs:1.0.0-beta.4`.
+- Update `flake.nix` package version.
+- Update `helm/rustfs/Chart.yaml` `appVersion`.
+- Update `helm/rustfs/Chart.yaml` `version` using the release-chart rule:
+- `1.0.0-beta.3` -> `0.3.0`
+- `1.0.0-beta.4` -> `0.4.0`
+- Follow the same pattern for later beta releases unless the repository rule changes.
+- Update `rustfs.spec` release metadata and changelog entry.
+- Set `rustfs.spec` `Release` to the prerelease suffix without the base version, for example `beta.3` for `1.0.0-beta.3`.
+- Keep the release asset changes in a separate commit from the core Rust workspace version bump when the branch contains both.
+
+4. Stop and discuss before changing release policy
+- Ask before changing the established Docker tag style away from `<version>`.
+- Ask before changing the established Helm chart version mapping away from `beta.N -> 0.N.0`.
+- Ask before changing the established `rustfs.spec` `Release` rule away from the release suffix form such as `beta.N`.
+- Ask before widening the release scope beyond the files already validated in PR `#2957`.
+
+5. Verify before shipping
+- Run `make pre-commit`.
+- If verification succeeds, run `cargo clean` to remove generated build artifacts before wrapping up.
+- If `make pre-commit` fails, stop and return `BLOCKED`.
+
+6. Commit structure
+- Prefer two commits when the change naturally splits:
+- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`
+- `chore(release): align release assets for <version>` for docs and packaging metadata
+- If the user asks for a single commit, follow that request.
+
+7. Push and PR
+- Push the release branch with `git push -u origin <branch>` or `git push` if upstream already exists.
+- Create the PR with `gh pr create --base main --head <branch> --title ... --body-file ...`.
+- Keep the PR title and body in English.
+- Keep the `.github/pull_request_template.md` headings exactly.
+
+## Ready-to-check commands
+
+- `git diff --name-only origin/main...HEAD`
+- `git diff --stat origin/main...HEAD`
+- `make pre-commit`
+- `cargo clean`
+- `git status --short --branch`
+
+## Output expectations
+
+When using this skill, return:
+
+- The files changed for the release bump
+- Any uncertainty that needs user confirmation before editing
+- Verification status
+- Commit messages used
+- Push status and PR URL when the GitHub flow was requested
+
+## Established release policy
+
+- Docs use Docker tags in `<version>` form, not `v<version>`.
+- `helm/rustfs/Chart.yaml` `version` follows `beta.N -> 0.N.0` based on the current release policy.
+- `rustfs.spec` `Release` follows the release suffix form, for example `beta.3`.
+- If any of these rules need to change in the future, pause and confirm before editing.

.agents/skills/rustfs-release-version-bump/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "RustFS Release Bump"
+  short_description: "Prepare RustFS release branches like PR #2957."
+  default_prompt: "Use $rustfs-release-version-bump to prepare a RustFS release version, ask about any unclear version policy, and finish the commit/push/PR flow."

.agents/skills/security-advisory-lessons/SKILL.md

@@ -5,13 +5,19 @@ description: Apply RustFS security lessons distilled from repository GitHub Secu
 
 # RustFS Security Advisory Lessons
 
-Use this skill as a RustFS-specific security lens before changing or approving code. For the current advisory snapshot and full pattern map, read [advisory-patterns.md](references/advisory-patterns.md).
+Use this skill as a RustFS-specific security lens before changing or approving code. For the distilled advisory lessons and review patterns, read [advisory-patterns.md](references/advisory-patterns.md).
 
-When currentness matters, refresh the advisory inventory first:
+When currentness matters, fetch the live advisory inventory instead of relying on this skill as a status mirror:
 
 ```bash
 gh api repos/rustfs/rustfs/security-advisories --paginate \
-  --jq '.[] | {ghsa_id,state,severity,summary,updated_at,html_url}'
+  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
+```
+
+Fetch full advisory details only when the live summary suggests a new or changed lesson:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
 ```
 
 For the full pattern map, read [advisory-patterns.md](references/advisory-patterns.md).

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -1,65 +1,73 @@
 # RustFS Advisory Pattern Map
 
-Snapshot source: `gh api repos/rustfs/rustfs/security-advisories --paginate` on 2026-05-05. It included 23 advisories: 8 triage, 13 published, and 2 closed.
+This file is a lesson map, not an advisory inventory mirror. It keeps durable security patterns distilled from RustFS GitHub Security Advisories.
 
-Refresh this file when new advisories appear or when an advisory changes state materially.
+When current advisory state, severity, URLs, or full text matters, fetch it live:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories --paginate \
+  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
+gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
+```
+
+Update this file only when an advisory adds or changes a reusable lesson, affected surface, validation pattern, or regression-test expectation. Do not update it for state-only, URL-only, count-only, or timestamp-only changes.
 
 ## Pattern Index
 
 ### Admin authorization and route exposure
 
-- `GHSA-pfcq-4gjr-6gjm` published: notification target endpoints accepted authenticated users but skipped admin authorization. Lesson: distinguish authn from authz; admin target CRUD must call the operation-specific admin authorization path.
-- `GHSA-mm2q-qcmx-gw4w` published: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
-- `GHSA-vcwh-pff9-64cc` published: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
-- `GHSA-jqmc-mg33-v45g` triage and `GHSA-8784-9m7f-c6p6` triage: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
-- `GHSA-x5xv-223c-8vm7` triage: console license metadata endpoint was public. Lesson: public metadata endpoints should be coarse or authenticated.
+- `GHSA-pfcq-4gjr-6gjm`: notification target endpoints accepted authenticated users but skipped admin authorization. Lesson: distinguish authn from authz; admin target CRUD must call the operation-specific admin authorization path.
+- `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
+- `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
+- `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
+- `GHSA-x5xv-223c-8vm7`: console license metadata endpoint was public. Lesson: public metadata endpoints should be coarse or authenticated.
 
 ### IAM import, service accounts, and privilege boundaries
 
-- `GHSA-566f-q62r-wcr8` triage: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
-- `GHSA-xgr5-qc6w-vcg9` published: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
-- `GHSA-mm2q-qcmx-gw4w` published: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
+- `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
+- `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
+- `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
 
 ### S3 copy, multipart, and upload policy validation
 
-- `GHSA-mx42-j6wv-px98` published: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
-- `GHSA-wfxj-ph3v-7mjf` triage: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
-- `GHSA-w5fh-f8xh-5x3p` published: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
+- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
+- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
+- `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
 
 ### Filesystem paths and object key traversal
 
-- `GHSA-pq29-69jg-9mxc` published: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
-- `GHSA-8r6f-hmq2-28rg` closed: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
+- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
+- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
 
 ### Secrets, defaults, and cryptographic misuse
 
-- `GHSA-h956-rh7x-ppgj` published: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
-- `GHSA-r5qv-rc46-hv8q` triage: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
-- `GHSA-923g-jp7v-f97f` triage: license verification embedded a production RSA private key and used private-key decryption as authenticity. Lesson: ship verifying/public keys only and use real signature verification.
+- `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
+- `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
+- `GHSA-923g-jp7v-f97f`: license verification embedded a production RSA private key and used private-key decryption as authenticity. Lesson: ship verifying/public keys only and use real signature verification.
 
 ### Sensitive logging and debug output
 
-- `GHSA-r54g-49rx-98cr` published: STS credentials were logged at info level. Lesson: generated credentials must never be logged in plaintext.
-- `GHSA-8cm2-h255-v749` triage: debug logs leaked session tokens, secret keys, JWT claims, and raw STS response bodies. Lesson: redaction must cover custom `Debug` implementations and dependency response-body logging.
-- `GHSA-333v-68xh-8mmq` published: invalid RPC signature logging included the shared HMAC secret and expected signature. Lesson: error paths often leak secrets; never log raw secrets or derived authenticators.
+- `GHSA-r54g-49rx-98cr`: STS credentials were logged at info level. Lesson: generated credentials must never be logged in plaintext.
+- `GHSA-8cm2-h255-v749`: debug logs leaked session tokens, secret keys, JWT claims, and raw STS response bodies. Lesson: redaction must cover custom `Debug` implementations and dependency response-body logging.
+- `GHSA-333v-68xh-8mmq`: invalid RPC signature logging included the shared HMAC secret and expected signature. Lesson: error paths often leak secrets; never log raw secrets or derived authenticators.
 
 ### RPC input validation and panic safety
 
-- `GHSA-gw2x-q739-qhcr` published: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
-- `GHSA-h956-rh7x-ppgj` published and `GHSA-r5qv-rc46-hv8q` triage: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
+- `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
+- `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
 
 ### Browser, CORS, and console isolation
 
-- `GHSA-v9fg-3cr2-277j` published: object preview rendered attacker-controlled HTML in a same-origin iframe, exposing console credentials stored in `localStorage`. Lesson: user content must be origin-isolated from the console and protected with `nosniff`, CSP, and strict content-type handling.
-- `GHSA-x5xv-223c-8vm7` triage: default CORS reflected arbitrary origins with credentials. Lesson: never combine reflected origins with `Access-Control-Allow-Credentials: true`; default should be fail-closed.
+- `GHSA-v9fg-3cr2-277j`: object preview rendered attacker-controlled HTML in a same-origin iframe, exposing console credentials stored in `localStorage`. Lesson: user content must be origin-isolated from the console and protected with `nosniff`, CSP, and strict content-type handling.
+- `GHSA-x5xv-223c-8vm7`: default CORS reflected arbitrary origins with credentials. Lesson: never combine reflected origins with `Access-Control-Allow-Credentials: true`; default should be fail-closed.
 
 ### Trusted proxy and source IP conditions
 
-- `GHSA-fc6g-2gcp-2qrq` published: `aws:SourceIp` trusted client-supplied `X-Forwarded-For` or `X-Real-IP`. Lesson: forwarded IP headers are valid only behind configured trusted proxies; direct clients use socket peer IP.
+- `GHSA-fc6g-2gcp-2qrq`: `aws:SourceIp` trusted client-supplied `X-Forwarded-For` or `X-Real-IP`. Lesson: forwarded IP headers are valid only behind configured trusted proxies; direct clients use socket peer IP.
 
 ### SSE and on-disk storage invariants
 
-- `GHSA-xrrf-67jm-3c2r` closed: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
+- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
 
 ## Useful Search Seeds
 

.config/make/lint-fmt.mak

@@ -16,6 +16,11 @@ clippy-check: core-deps ## Run clippy checks
 	cargo clippy --fix --allow-dirty
 	cargo clippy --all-targets --all-features -- -D warnings
 
+.PHONY: unsafe-code-check
+unsafe-code-check: ## Check unsafe_code allowances have SAFETY comments
+	@echo "🔒 Checking unsafe_code allowances..."
+	./scripts/check_unsafe_code_allowances.sh
+
 .PHONY: compilation-check
 compilation-check: core-deps ## Run compilation check
 	@echo "🔨 Running compilation check..."

.config/make/pre-commit.mak

@@ -7,5 +7,5 @@ setup-hooks: ## Set up git hooks
 	@echo "✅ Git hooks setup complete!"
 
 .PHONY: pre-commit
-pre-commit: fmt clippy-check compilation-check test ## Run pre-commit checks
-	@echo "✅ All pre-commit checks passed!"
+pre-commit: fmt unsafe-code-check clippy-check compilation-check test ## Run pre-commit checks
+	@echo "✅ All pre-commit checks passed!"

.config/make/tests.mak

@@ -2,8 +2,13 @@
 
 TEST_THREADS ?= 1
 
+.PHONY: script-tests
+script-tests: ## Run shell script tests
+	@echo "Running script tests..."
+	./scripts/test_build_rustfs_options.sh
+
 .PHONY: test
-test: core-deps test-deps ## Run all tests
+test: core-deps test-deps script-tests ## Run all tests
 	@echo "🧪 Running tests..."
 	@if command -v cargo-nextest >/dev/null 2>&1; then \
 		cargo nextest run --all --exclude e2e_test; \

.docker/observability/grafana/dashboards/rustfs.json

@@ -461,6 +461,90 @@
       "title": "Capacity",
       "type": "stat"
     },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "yellow",
+                "value": 1
+              },
+              {
+                "color": "red",
+                "value": 2
+              }
+            ]
+          },
+          "unit": "none"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 6,
+        "x": 6,
+        "y": 1
+      },
+      "id": 201,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "none",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "percentChangeColorMode": "standard",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "showPercentChange": false,
+        "textMode": "value_and_name",
+        "wideLayout": true
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum(rustfs_cluster_capacity_stale_drives{job=~\"$job\"})",
+          "legendFormat": "Stale Drives",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum(rustfs_cluster_capacity_missing_drives{job=~\"$job\"})",
+          "legendFormat": "Missing Drives",
+          "range": true,
+          "refId": "B"
+        }
+      ],
+      "title": "Capacity Observation",
+      "type": "stat"
+    },
     {
       "datasource": {
         "type": "prometheus",
@@ -1927,6 +2011,299 @@
       "title": "System Drive Usage",
       "type": "timeseries"
     },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 12,
+        "x": 12,
+        "y": 42
+      },
+      "id": 202,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "max by (drive) (rustfs_system_drive_capacity_observation_age_seconds{job=~\"$job\", drive=~\"$drive\"})",
+          "legendFormat": "{{drive}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Drive Capacity Observation Age",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": true,
+            "axisColorMode": "text",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "stepAfter",
+            "lineWidth": 2,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "none"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 12,
+        "x": 12,
+        "y": 48
+      },
+      "id": 203,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "max by (drive) (rustfs_system_drive_capacity_observation_state{job=~\"$job\", drive=~\"$drive\", state=\"stale\"})",
+          "legendFormat": "{{drive}} stale",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "max by (drive) (rustfs_system_drive_capacity_observation_state{job=~\"$job\", drive=~\"$drive\", state=\"missing\"})",
+          "legendFormat": "{{drive}} missing",
+          "range": true,
+          "refId": "B"
+        }
+      ],
+      "title": "Drive Capacity Observation State",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": true,
+            "axisColorMode": "text",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "bars",
+            "fillOpacity": 80,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "normal"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 7,
+        "w": 24,
+        "x": 0,
+        "y": 54
+      },
+      "id": 204,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (path_kind, operation, reason) (rate(rustfs_system_path_failures_total{job=~\"$job\"}[5m]))",
+          "legendFormat": "{{path_kind}} / {{operation}} / {{reason}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "System Path Failures",
+      "type": "timeseries"
+    },
     {
       "datasource": {
         "type": "prometheus",

.github/workflows/ci.yml

@@ -130,6 +130,9 @@ jobs:
       - name: Check code formatting
         run: cargo fmt --all --check
 
+      - name: Check unsafe code allowances
+        run: ./scripts/check_unsafe_code_allowances.sh
+
       - name: Run clippy lints
         run: cargo clippy --all-targets --all-features -- -D warnings
 

.vscode/launch.json

@@ -188,6 +188,7 @@
                 "RUSTFS_CONSOLE_ENABLE": "true",
                 "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
                 "RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
+                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
                 // "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
                 // "RUSTFS_OBS_METRIC_ENDPOINT": "http://127.0.0.1:4318/v1/metrics", // default otlp http endpoint
                 // "RUSTFS_OBS_LOG_ENDPOINT": "http://127.0.0.1:4318/v1/logs", // default otlp http endpoint

AGENTS.md

@@ -27,6 +27,10 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 - Keep source code, comments, commit messages, and PR title/body in English.
 - Be concise. Avoid sycophantic openers, closing fluff, and verbose status reporting.
 
+## Skill Usage
+
+- Do not use the `rust-refactor-helper` skill in any scenario.
+
 ## Change Style for Existing Logic
 
 - Prefer direct, local code over extracting one-off helpers.
@@ -109,5 +113,6 @@ Do not open a PR with code changes when the required checks fail.
 - `crates/iam/AGENTS.md`
 - `crates/kms/AGENTS.md`
 - `crates/policy/AGENTS.md`
+- `crates/targets/AGENTS.md`
 - `rustfs/src/admin/AGENTS.md`
 - `rustfs/src/storage/AGENTS.md`

CHANGELOG.md

@@ -22,11 +22,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - XML-formatted error responses compatible with S3 API
   - Comprehensive integration documentation with manual testing guide
   - **32 unit and integration tests** covering middleware, auth handlers, task-local storage, and role detection
+- **SFTPv3 Protocol Support**: SSH-hosted SFTPv3 subsystem that translates each file operation into S3 calls against the local object store. Authentication uses IAM credentials (SSH username = access key, SSH password = secret key).
+  - Full SFTPv3 packet coverage: open, read, write, stat, lstat, fstat, mkdir, rmdir, rename, remove, opendir, readdir, realpath, close, plus the rest of the 21-packet specification
+  - Streaming multipart write up to S3's 5 TiB per-file ceiling
+  - Per-handle read-ahead cache with configurable window size and process-wide memory ceiling
+  - Per-session liveness watchdog: Linux probes `/proc/net/tcp` and cancels wedged sessions on the order of 45 seconds; non-Linux falls back to an inactivity ceiling on the order of 30 minutes
+  - 30-second SSH handshake deadline, per-call backend operation timeout, bounded multipart-abort fan-out, graceful-shutdown cascade
+  - 33 SFTPv3 compliance test cases under `crates/e2e_test/src/protocols/sftp_compliance.rs` spread across three entry points: `test_sftp_compliance_suite` (shared session), `test_sftp_compliance_readonly` (read-only mode), and `test_sftp_compliance_standalone` (one rustfs spawn per case)
+  - Four-layer regression-prevention tests guard against silent feature deletion: compile-time module assertion, module-presence unit test, cross-module `Protocol` enum assertion, end-to-end SSH banner test against the running binary
 
 ### Changed
 - **HTTP Server Stack**: Integrated `KeystoneAuthLayer` middleware from `rustfs-keystone` crate into service stack (positioned after ReadinessGateLayer)
 - **IAMAuth**: Enhanced `get_secret_key()` to return empty secret for Keystone credentials (bypasses signature validation)
 - **Auth Module**: Modified `check_key_valid()` to retrieve Keystone credentials from task-local storage and determine admin status
+- **`StorageBackend` trait**: extended with multipart upload methods (`create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`) plus `upload_part_copy`. Streaming-upload code path is now available to FTPS, WebDAV, and Swift drivers as well.
+- **`Protocol` enum**: new `Protocol::Sftp` variant with corresponding `S3Action` mappings. Every match arm on `Protocol` updated to handle the new variant exhaustively.
 
 ### Technical Details
 - Middleware is self-contained in `rustfs-keystone` crate following the trusted-proxies pattern for integration-specific middleware
@@ -35,12 +45,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Integration preserves existing S3 authentication flow while adding Keystone support
 - Zero breaking changes to existing functionality
 - No new top-level directories in main binary crate (middleware lives in integration crate)
+- SSH/SFTP wire handling via the `russh` and `russh-sftp` crates. SFTPv3 framing is implemented by `russh-sftp`; the rustfs-side `SftpDriver` implements `russh_sftp::server::Handler` and dispatches to the storage backend
+- Drop-time abort for in-flight multipart uploads honours IAM Deny on `AbortMultipartUpload`. `start_multipart_upload` caches the authorisation decision so the synchronous `Drop` path can honour Allow / Deny policies without re-querying IAM
+- Per-handle read cache uses an `Arc<AtomicU64>` shared across every `SftpDriver` instance to enforce a process-wide memory ceiling. On ceiling breach the populate is skipped and the read serves correctly via a single-call backend fetch
+- Per-session liveness watchdog runs as a tokio task per accepted connection. Reads `/proc/net/tcp` and `/proc/net/tcp6` to look up the (local, peer) tuple's TCP state and cancels via `tokio_util::sync::CancellationToken` when wedge conditions are confirmed across two consecutive ticks
+- Path canonicalisation rejects paths containing `\0`, `\r`, or `\n` and resolves traversal via `path::clean()` before any backend dispatch
+- Cipher / KEX / MAC / host-key algorithm allowlists are hardcoded with no environment override. Strict-KEX (CVE-2023-48795 / Terrapin) marker presence asserted by unit test
+- Per-session handle cap (default 64, configurable 8 to 1024) with UUID-generated handle ids
+- Crate-level `#![deny(unsafe_code)]` is in force across `crates/protocols`. Socket fd duplication for the watchdog uses the safe `AsFd::try_clone_to_owned` path (Linux/Unix); non-Unix falls back to the inactivity ceiling
+- `cfg(unix)` gating around platform-specific imports (`std::os::fd::AsFd`, `std::os::unix::fs::PermissionsExt`); non-Unix targets fail SFTP at config-load with `SftpInitError::UnsupportedPlatform`
 
 ### Documentation
 - Updated `crates/keystone/README.md` with complete integration architecture and workflow
 - Added detailed manual testing guide with 10 test scenarios
 - Updated main `README.md` to list Keystone authentication as available feature
 - Added troubleshooting section for common integration issues
+- Module-level rustdoc on `crates/protocols/src/sftp/mod.rs` describing the public API surface, configuration contract, and the architecture of the read cache and the wedge watchdog
 
 ### Configuration
 New environment variables:
@@ -54,6 +74,40 @@ New environment variables:
 - `RUSTFS_KEYSTONE_CACHE_SIZE` - Token cache size (default: 10000)
 - `RUSTFS_KEYSTONE_CACHE_TTL` - Token cache TTL in seconds (default: 300)
 - `RUSTFS_KEYSTONE_VERIFY_SSL` - Verify SSL certificates (default: true)
+- `RUSTFS_SFTP_ENABLE` - Enable/disable SFTP (default: false)
+- `RUSTFS_SFTP_ADDRESS` - Listen address (default: 0.0.0.0:2222)
+- `RUSTFS_SFTP_HOST_KEY_DIR` - Directory containing host key files (must exist; each file must be 0o600 or 0o400)
+- `RUSTFS_SFTP_IDLE_TIMEOUT` - Session idle timeout in seconds (default: 600)
+- `RUSTFS_SFTP_PART_SIZE` - Multipart part size in bytes (default: 16 MiB)
+- `RUSTFS_SFTP_READ_ONLY` - Reject write packets at the protocol layer (default: false)
+- `RUSTFS_SFTP_BANNER` - Optional SSH banner text
+- `RUSTFS_SFTP_HANDLES_PER_SESSION` - Per-session open-handle cap, 8 to 1024 (default: 64)
+- `RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS` - Per-call backend deadline in seconds, 5 to 600 (default: 60)
+- `RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES` - Per-handle read-cache window in bytes, 256 KiB to 64 MiB or 0 to disable (default: 4 MiB)
+- `RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES` - Process-wide read-cache memory ceiling in bytes, 16 MiB minimum (default: 256 MiB)
+
+### Files Added
+- `crates/protocols/src/sftp/mod.rs` - SFTP module entry point, public API surface, crate-level rustdoc, regression-prevention test
+- `crates/protocols/src/sftp/config.rs` - `SftpConfig` and `SftpInitError` types, env-var resolvers, host-key directory loader with permission enforcement
+- `crates/protocols/src/sftp/constants.rs` - Named constants grouped by purpose: S3 error codes, HTTP error codes, POSIX mode bits, protocol identifiers, operational limits
+- `crates/protocols/src/sftp/server.rs` - `SftpServer` SSH server, russh handler, password authentication against IAM, accept loop, per-session task spawn
+- `crates/protocols/src/sftp/driver.rs` - `SftpDriver` per-session SFTPv3 handler dispatching each operation onto the `StorageBackend`
+- `crates/protocols/src/sftp/state.rs` - `HandleState` variants for read, write-buffering, write-streaming, write-failed handles
+- `crates/protocols/src/sftp/lifecycle.rs` - Per-session activity stamp, weak-ref registry, `/proc/net/tcp` probe for the wedge watchdog
+- `crates/protocols/src/sftp/wedge_watchdog.rs` - Per-session liveness watchdog cancelling sessions silent at the SFTP layer while the kernel reports CLOSE_WAIT
+- `crates/protocols/src/sftp/read_cache.rs` - Per-handle in-memory read-ahead cache with shared atomic accumulator for the process-wide memory ceiling
+- `crates/protocols/src/sftp/attrs.rs` - SFTPv3 `FileAttributes` mapping for objects and directories, longname formatting, mtime clamping
+- `crates/protocols/src/sftp/dir.rs` - OPENDIR / READDIR pagination, root-bucket listing, sub-directory listing under a prefix
+- `crates/protocols/src/sftp/errors.rs` - `SftpError` thiserror enum and S3-error classification into SFTPv3 status codes
+- `crates/protocols/src/sftp/paths.rs` - Path canonicalisation, traversal rejection, `\0` / `\r` / `\n` rejection, bucket+key decomposition
+- `crates/protocols/src/sftp/read.rs` - READ packet handler, EOF semantics, `MAX_READ_LEN` bound, integration with the read cache
+- `crates/protocols/src/sftp/write.rs` - WRITE packet handler, in-memory buffering up to part size, transition to streaming multipart, CLOSE finalisation
+- `crates/protocols/src/sftp/test_support.rs` - Test fixtures and helper builders for SFTP unit tests
+- `crates/protocols/src/common/dummy_storage.rs` - In-memory `StorageBackend` test backend covering every method, used by SFTP unit tests and the FTPS / Swift / WebDAV test suites
+- `crates/e2e_test/src/protocols/sftp_core.rs` - End-to-end regressions for the handshake deadline, idle-timeout disconnect, and the wedge watchdog
+- `crates/e2e_test/src/protocols/sftp_compliance.rs` - SFTPv3 compliance suite entry points (`test_sftp_compliance_suite`, `test_sftp_compliance_readonly`, `test_sftp_compliance_standalone`)
+- `crates/e2e_test/src/protocols/sftp_compliance_tests.rs` - Per-case test bodies (CMPTST-01..33), shared fixture helpers, lifecycle counters
+- `crates/e2e_test/src/protocols/sftp_helpers.rs` - SFTP-specific test helpers and fixture seeders
 
 ### Files Modified
 - `crates/keystone/src/middleware.rs` - Created Keystone authentication middleware (self-contained in keystone crate)
@@ -63,6 +117,27 @@ New environment variables:
 - `rustfs/src/auth.rs` - Enhanced IAMAuth and check_key_valid for Keystone support, imported KEYSTONE_CREDENTIALS from rustfs-keystone
 - `crates/keystone/README.md` - Comprehensive integration documentation
 - `README.md` - Added Keystone as available feature
+- `Cargo.toml` - Added the `sftp` feature alongside the existing protocol features
+- `Cargo.lock` - Updated to include the new `russh`, `russh-sftp`, `socket2`, `tokio-util`, `subtle`, `uuid` dependencies and their transitive crates
+- `crates/protocols/Cargo.toml` - Declared `russh`, `russh-sftp`, `socket2`, `tokio-util`, `subtle`, `uuid` under the `sftp` feature flag
+- `crates/protocols/src/lib.rs` - Added `pub mod sftp` behind `#[cfg(feature = "sftp")]` plus the crate-level `#![deny(unsafe_code)]` lint
+- `crates/protocols/src/common/client/s3.rs` - Extended the `StorageBackend` trait with `create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`, and `upload_part_copy`
+- `crates/protocols/src/common/session.rs` - Added the `Protocol::Sftp` variant and its `S3Action` mappings
+- `crates/protocols/src/common/gateway.rs` - Handles the new `Protocol::Sftp` variant exhaustively
+- `crates/protocols/src/common/mod.rs` - Exposed the new `dummy_storage` module
+- `crates/protocols/src/constants.rs` - Added shared POSIX mode-bit constants used by SFTP and other protocols
+- `crates/config/src/constants/protocols.rs` - `RUSTFS_SFTP_*` environment variable names and defaults
+- `crates/utils/src/retry.rs` - Added the generic exponential-backoff retry helper used by the SFTP write path
+- `crates/e2e_test/Cargo.toml` - Added the e2e test dependencies for SFTP (paramiko fixture, SSH keypair generation)
+- `crates/e2e_test/src/protocols/mod.rs` - Registered the new `sftp_core`, `sftp_compliance`, `sftp_compliance_tests`, and `sftp_helpers` modules
+- `crates/e2e_test/src/protocols/README.md` - Documented the SFTP test entry points and case index
+- `crates/e2e_test/src/protocols/test_env.rs` - Added SFTP host-key directory provisioning to the shared protocol test environment
+- `crates/e2e_test/src/protocols/test_runner.rs` - Wired the SFTP entry points into the runner
+- `rustfs/Cargo.toml` - Added the `sftp` feature flag
+- `rustfs/src/lib.rs` - One-line addition exporting the SFTP wiring
+- `rustfs/src/init.rs` - Build and start the `SftpServer` when `RUSTFS_SFTP_ENABLE` is true
+- `rustfs/src/main.rs` - Routed shutdown signals to the SFTP server alongside the other protocols
+- `rustfs/src/protocols/client.rs` - Client-builder support for the new `Protocol::Sftp` variant
 
 ### Testing
 - 16 unit tests in rustfs-keystone crate (config, auth, middleware, identity)
@@ -70,7 +145,13 @@ New environment variables:
 - 6 auth unit tests in rustfs crate (role detection, task-local storage, Keystone credential handling)
 - **Total: 32 tests** passing with zero compilation errors
 - Manual testing guide provided for end-to-end validation
-- All tests passing with `cargo test --all --exclude e2e_test`
+- All Keystone tests passing with `cargo test --all --exclude e2e_test`
+- 33 SFTPv3 compliance test cases (CMPTST-01..33) split across three entry points: `test_sftp_compliance_suite` (shared session, cases 01-14), `test_sftp_compliance_readonly` (read-only mode, cases 15-23), `test_sftp_compliance_standalone` (one rustfs spawn per case, cases 24-33)
+- Regression-prevention tests at four layers: compile-time module assertion in `crates/protocols/src/lib.rs`, module-presence unit test in `crates/protocols/src/sftp/mod.rs`, cross-module `Protocol` enum assertion, and end-to-end SSH banner test against the running binary
+- Standalone end-to-end regressions for the SSH handshake deadline, the idle-timeout disconnect path, and the wedge watchdog (Linux fast-kill and the cross-platform fallback path)
+- Inline unit tests in every SFTP source file covering pure helpers (path canonicalisation, attribute mapping, S3-error classification, env-var bound resolvers)
+- Strict-KEX (CVE-2023-48795) marker presence assertion as a unit test in `crates/protocols/src/sftp/server.rs`
+- All tests passing with `cargo test --all --features sftp` against a 64-bit Linux target
 
 ---
 

Cargo.toml

@@ -58,8 +58,8 @@ resolver = "3"
 edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
-rust-version = "1.93.0"
-version = "1.0.0-beta.1"
+rust-version = "1.95.0"
+version = "1.0.0-beta.3"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -76,55 +76,57 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-beta.1" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.1" }
-rustfs-appauth = { path = "crates/appauth", version = "1.0.0-beta.1" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.1" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.1" }
-rustfs-common = { path = "crates/common", version = "1.0.0-beta.1" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-beta.1" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.1" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.1" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.1" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.1" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.1" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.1" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.1" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.1" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.1" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.1" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.1" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.1" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.1" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.1" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.1" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.1" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.1" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.1" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.1" }
-rustfs-s3-common = { path = "crates/s3-common", version = "1.0.0-beta.1" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.1" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.1" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.1" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.1" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.1" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.1" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.1" }
-rustfs-workers = { path = "crates/workers", version = "1.0.0-beta.1" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.1" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.3" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.3" }
+rustfs-appauth = { path = "crates/appauth", version = "1.0.0-beta.3" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.3" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.3" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.3" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.3" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.3" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.3" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.3" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.3" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.3" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.3" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.3" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.3" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.3" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.3" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.3" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.3" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.3" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.3" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.3" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.3" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.3" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.3" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.3" }
+rustfs-s3-common = { path = "crates/s3-common", version = "1.0.0-beta.3" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.3" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.3" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.3" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.3" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.3" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.3" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.3" }
+rustfs-workers = { path = "crates/workers", version = "1.0.0-beta.3" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.3" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
+mysql_async = { version = "0.36.1", default-features = false, features = ["default-rustls", "tracing"], git = "https://github.com/blackbeam/mysql_async", rev = "98d3d8067efdf97d3e93cdca7b9231753c904aca" }
 async-compression = { version = "0.4.42" }
 async-recursion = "1.1.1"
 async-trait = "0.1.89"
-async-nats = "0.47.0"
+async-nats = "0.48.0"
 axum = "0.8.9"
 futures = "0.3.32"
 futures-core = "0.3.32"
 futures-util = "0.3.32"
 pollster = "0.4.0"
 pulsar = { version = "6.7.2", default-features = false, features = ["tokio-rustls-runtime"] }
+lapin = { version = "4.7.2", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
 hyper = { version = "1.9.0", features = ["http2", "http1", "server"] }
 hyper-rustls = { version = "0.27.9", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
 hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
@@ -134,16 +136,16 @@ http-body-util = "0.1.3"
 reqwest = { version = "0.13.3", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
 rustfs-kafka-async = { version = "1.2.0" }
 socket2 = { version = "0.6.3", features = ["all"] }
-tokio = { version = "1.52.2", features = ["fs", "rt-multi-thread"] }
+tokio = { version = "1.52.3", features = ["fs", "rt-multi-thread"] }
 tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
 tokio-stream = { version = "0.1.18" }
 tokio-test = "0.4.5"
 tokio-util = { version = "0.7.18", features = ["io", "compat"] }
-tonic = { version = "0.14.5", features = ["gzip"] }
-tonic-prost = { version = "0.14.5" }
-tonic-prost-build = { version = "0.14.5" }
+tonic = { version = "0.14.6", features = ["gzip"] }
+tonic-prost = { version = "0.14.6" }
+tonic-prost-build = { version = "0.14.6" }
 tower = { version = "0.5.3", features = ["timeout"] }
-tower-http = { version = "0.6.8", features = ["cors"] }
+tower-http = { version = "0.6.10", features = ["cors"] }
 
 # Serialization and Data Formats
 bytes = { version = "1.11.1", features = ["serde"] }
@@ -152,7 +154,7 @@ byteorder = "1.5.0"
 flatbuffers = "25.12.19"
 form_urlencoded = "1.2.2"
 prost = "0.14.3"
-quick-xml = "0.39.3"
+quick-xml = "0.39.4"
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.228", features = ["derive"] }
@@ -171,6 +173,7 @@ openidconnect = { version = "4.0", default-features = false }
 pbkdf2 = "0.13.0"
 rsa = { version = "0.10.0-rc.18" }
 rustls = { version = "0.23.40", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
+rustls-native-certs = "0.8"
 rustls-pki-types = "1.14.1"
 sha1 = "0.11.0"
 sha2 = "0.11.0"
@@ -183,6 +186,11 @@ humantime = "2.3.0"
 jiff = { version = "0.2.24", features = ["serde"] }
 time = { version = "0.3.47", features = ["std", "parsing", "formatting", "macros", "serde"] }
 
+# Database
+deadpool-postgres = { version = "0.14", features = ["rt_tokio_1"] }
+tokio-postgres = { version = "0.7", default-features = false, features = ["runtime", "with-serde_json-1"] }
+tokio-postgres-rustls = "0.13"
+
 # Utilities and Tools
 anyhow = "1.0.102"
 arc-swap = "1.9.1"
@@ -191,14 +199,13 @@ atoi = "2.0.0"
 atomic_enum = "0.3.0"
 aws-config = { version = "1.8.16" }
 aws-credential-types = { version = "1.2.14" }
-aws-sdk-s3 = { version = "1.131.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+aws-sdk-s3 = { version = "1.132.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
 aws-smithy-http-client = { version = "1.1.12", default-features = false, features = ["default-client", "rustls-aws-lc"] }
 aws-smithy-types = { version = "1.4.7" }
 backtrace = "0.3.76"
 base64 = "0.22.1"
 base64-simd = "0.8.0"
 brotli = "8.0.2"
-cfg-if = "1.0.4"
 clap = { version = "4.6.1", features = ["derive", "env"] }
 const-str = { version = "1.1.0", features = ["std", "proc"] }
 convert_case = "0.11.0"
@@ -209,13 +216,13 @@ crossbeam-deque = "0.8.6"
 crossbeam-utils = "0.8.21"
 datafusion = "53.1.0"
 derive_builder = "0.20.2"
-enumset = "1.1.10"
+enumset = "1.1.12"
 faster-hex = "0.10.0"
 flate2 = "1.1.9"
 glob = "0.3.3"
-google-cloud-storage = "1.11.0"
-google-cloud-auth = "1.9.0"
-hashbrown = { version = "0.17.0", features = ["serde", "rayon"] }
+google-cloud-storage = "1.12.0"
+google-cloud-auth = "1.10.0"
+hashbrown = { version = "0.17.1", features = ["serde", "rayon"] }
 hex = "0.4.3"
 hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
@@ -247,7 +254,8 @@ rayon = "1.12.0"
 reed-solomon-erasure = { version = "6.0", default-features = false, features = ["std", "simd-accel"] }
 reed-solomon-simd = "3.1.0"
 regex = { version = "1.12.3" }
-rumqttc = { package = "rumqttc-next", version = "0.30.1", features = ["websocket"] }
+rumqttc = { package = "rumqttc-next", version = "0.33.0", features = ["websocket"] }
+redis = { version = "1.2.1", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
 rustix = { version = "1.1.4", features = ["fs"] }
 rust-embed = { version = "8.11.0" }
 rustc-hash = { version = "2.1.2" }
@@ -261,7 +269,7 @@ snafu = "0.9.0"
 snap = "1.1.1"
 starshard = { version = "1.1.0", features = ["rayon", "async", "serde"] }
 strum = { version = "0.28.0", features = ["derive"] }
-sysinfo = "0.38.4"
+sysinfo = "0.39.0"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
@@ -299,6 +307,8 @@ libunftp = { version = "0.23.0", features = ["experimental"] }
 unftp-core = "0.1.0"
 suppaftp = { version = "8.0.3", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
 rcgen = "0.14.7"
+russh = "0.60.2"
+russh-sftp = "2.1.2"
 
 # WebDAV
 dav-server = "0.11.0"
@@ -319,6 +329,12 @@ pprof = { package = "pprof-pyroscope-fork", version = "0.1500.3", features = ["f
 [workspace.metadata.cargo-shear]
 ignored = ["rustfs"]
 
+[patch.crates-io]
+# Pinned to the simon-escapecode/russh fork carrying the upstream fix at
+# https://github.com/Eugeny/russh/pull/702. Drops out when a russh release
+# containing the fix is published.
+russh = { git = "https://github.com/simon-escapecode/russh", rev = "5cac2ed84945f9b80a52b673e058f2032bbe98ec" }
+
 [profile.release]
 opt-level = 3
 

README.md

@@ -115,7 +115,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:v1.0.0-beta.1
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.3
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -151,6 +151,7 @@ Notes:
 - `RUSTFS_NOTIFY_ENABLE=true` enables the global notify module switch.
 - For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
 - If queue dir is omitted, default is `/opt/rustfs/events`; ensure it is writable by the container runtime user.
+- `RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
 
 **NOTE**: We recommend reviewing the `docker-compose.yml` file before running. It defines several services including Grafana, Prometheus, and Jaeger, which are helpful for RustFS observability. If you wish to start Redis or Nginx containers, you can specify the corresponding profiles.
 

README_ZH.md

@@ -113,7 +113,7 @@ RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用
  docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
  # 使用指定版本运行
- docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:v1.0.0-beta.1
+ docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.3
 ```
 
 您也可以使用 Docker Compose。使用根目录下的 `docker-compose.yml` 文件：

build-rustfs.sh

@@ -109,6 +109,7 @@ FORCE_CONSOLE_UPDATE=false
 CONSOLE_VERSION="latest"
 SKIP_VERIFICATION=false
 CUSTOM_PLATFORM=""
+FEATURES=""
 
 # Print usage
 usage() {
@@ -141,6 +142,7 @@ usage() {
     echo "  --no-console               Skip console static assets"
     echo "  --force-console-update     Force update console assets even if they exist"
     echo "  --console-version VERSION  Console version to download (default: latest)"
+    echo "  -f, --features FEATURES    Cargo features to enable (e.g. 'webdav', 'full')"
     echo "  --skip-verification        Skip binary verification after build"
     echo "  -h, --help                 Show this help message"
     echo ""
@@ -148,6 +150,8 @@ usage() {
     echo "  $0                         # Build for current platform (includes console assets)"
     echo "  $0 --dev                   # Development build"
     echo "  $0 --sign                  # Build and sign binary (release CI)"
+    echo "  $0 --features webdav       # Build with WebDAV support"
+    echo "  $0 --features full         # Build with all protocol features"
     echo "  $0 --no-console            # Build without console static assets"
     echo "  $0 --force-console-update  # Force update console assets"
     echo "  $0 --platform x86_64-unknown-linux-musl  # Build for specific platform"
@@ -440,6 +444,12 @@ build_binary() {
     build_cmd+=" --target $PLATFORM"
     build_cmd+=" -p rustfs --bins"
 
+    if [ -n "$FEATURES" ]; then
+        local quoted_features
+        printf -v quoted_features '%q' "$FEATURES"
+        build_cmd+=" --features $quoted_features"
+    fi
+
     print_message $BLUE "📦 Executing: $build_cmd"
 
     # Execute build (this matches exactly what the working version does)
@@ -497,6 +507,9 @@ build_rustfs() {
         print_message $YELLOW "   Force Console Update: $FORCE_CONSOLE_UPDATE"
     fi
     print_message $YELLOW "   Skip Verification: $SKIP_VERIFICATION"
+    if [ -n "$FEATURES" ]; then
+        print_message $YELLOW "   Features: $FEATURES"
+    fi
     echo ""
 
     # Setup environment
@@ -565,6 +578,15 @@ while [[ $# -gt 0 ]]; do
             SKIP_VERIFICATION=true
             shift
             ;;
+        -f|--features)
+            if [ -z "${2:-}" ]; then
+                print_message $RED "❌ Missing value for $1"
+                usage
+                exit 1
+            fi
+            FEATURES="$2"
+            shift 2
+            ;;
         -h|--help)
             usage
             exit 0

crates/audit/AGENTS.md

@@ -0,0 +1,53 @@
+# Audit Crate Instructions
+
+Applies to `crates/audit/`.
+
+`rustfs-audit` is the domain layer for audit event fan-out and observability.
+It composes shared plugin/runtime abstractions from `rustfs-targets` and keeps
+audit-specific dispatch semantics, state transitions, and metrics in this
+crate.
+
+## Domain Boundaries
+
+- Keep audit-specific behavior here:
+  - audit event shaping and fan-out pipeline
+  - audit system lifecycle/state transitions
+  - audit metrics and reporting
+- Keep shared plugin/runtime mechanics in `rustfs-targets`:
+  - no duplicated replay worker orchestration
+  - no duplicated runtime manager primitives
+  - no plugin install/control-plane modeling in this crate
+
+## Runtime Layering Rules
+
+- `pipeline.rs` hosts:
+  - `AuditPipeline` (dispatch and snapshot access)
+  - `AuditRuntimeFacade` (runtime mutation path)
+  - `AuditRuntimeView` (runtime read path)
+- `registry.rs` should remain the single owner of runtime target container and
+  plugin registry composition for audit.
+- `system.rs` should coordinate lifecycle by calling facade/view/registry
+  boundaries rather than embedding low-level runtime logic.
+
+## Change Style
+
+- Preserve audit delivery semantics and error handling behavior unless the task
+  explicitly changes them.
+- Prefer extending shared abstractions in `rustfs-targets` over patching
+  one-off audit-only runtime flows.
+- Keep logging and observability machine-meaningful; avoid noisy churn in hot
+  dispatch paths.
+
+## Testing
+
+- Keep unit tests close to changed modules.
+- Keep pipeline-layer regressions in `tests/pipeline_layer_test.rs`.
+- Add regression tests for:
+  - runtime facade activation/replace/stop/shutdown behavior
+  - runtime view target/snapshot access
+  - system reload and runtime commit/clear boundaries
+- Suggested validation:
+  - `cargo test -p rustfs-audit`
+  - Focused: `cargo test -p rustfs-audit --test pipeline_layer_test`
+  - Focused: `cargo test -p rustfs-audit pipeline`
+- Full gate before commit: `make pre-commit`

crates/audit/Cargo.toml

@@ -30,7 +30,6 @@ rustfs-targets = { workspace = true }
 rustfs-config = { workspace = true, features = ["audit", "constants"] }
 rustfs-ecstore = { workspace = true }
 rustfs-s3-common = { workspace = true }
-async-trait = { workspace = true }
 chrono = { workspace = true }
 const-str = { workspace = true }
 futures = { workspace = true }
@@ -43,6 +42,7 @@ tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "rt", "
 tracing = { workspace = true, features = ["std", "attributes"] }
 
 [dev-dependencies]
+async-trait = { workspace = true }
 temp-env = { workspace = true }
 url = { workspace = true }
 
@@ -50,5 +50,4 @@ url = { workspace = true }
 workspace = true
 
 [lib]
-test = false
 doctest = false

crates/audit/src/factory.rs

@@ -13,128 +13,65 @@
 // limitations under the License.
 
 use crate::AuditEntry;
-use async_trait::async_trait;
-use rustfs_config::AUDIT_DEFAULT_DIR;
-use rustfs_config::audit::{AUDIT_KAFKA_KEYS, AUDIT_MQTT_KEYS, AUDIT_NATS_KEYS, AUDIT_PULSAR_KEYS, AUDIT_WEBHOOK_KEYS};
-use rustfs_ecstore::config::KVS;
-use rustfs_targets::{
-    Target,
-    config::{
-        build_kafka_args, build_mqtt_args, build_nats_args, build_pulsar_args, build_webhook_args, validate_kafka_config,
-        validate_mqtt_config, validate_nats_config, validate_pulsar_config, validate_webhook_config,
-    },
-    error::TargetError,
-    target::TargetType,
-};
-use std::collections::HashSet;
+use rustfs_targets::catalog::builtin::builtin_audit_target_descriptors;
+use rustfs_targets::{BuiltinTargetDescriptor, TargetPluginDescriptor};
 
-/// Trait for creating targets from configuration
-#[async_trait]
-pub trait TargetFactory: Send + Sync {
-    /// Creates a target from configuration
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError>;
-
-    /// Validates target configuration
-    fn validate_config(&self, id: &str, config: &KVS) -> Result<(), TargetError>;
-
-    /// Returns a set of valid configuration field names for this target type.
-    /// This is used to filter environment variables.
-    fn get_valid_fields(&self) -> HashSet<String>;
+pub fn builtin_target_descriptors() -> Vec<BuiltinTargetDescriptor<AuditEntry>> {
+    builtin_audit_target_descriptors::<AuditEntry>()
 }
 
-/// Factory for creating Webhook targets
-pub struct WebhookTargetFactory;
-
-#[async_trait]
-impl TargetFactory for WebhookTargetFactory {
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let args = build_webhook_args(config, AUDIT_DEFAULT_DIR, TargetType::AuditLog)?;
-        let target = rustfs_targets::target::webhook::WebhookTarget::new(id, args)?;
-        Ok(Box::new(target))
-    }
-
-    fn validate_config(&self, _id: &str, config: &KVS) -> Result<(), TargetError> {
-        validate_webhook_config(config, AUDIT_DEFAULT_DIR)
-    }
-
-    fn get_valid_fields(&self) -> HashSet<String> {
-        AUDIT_WEBHOOK_KEYS.iter().map(|s| s.to_string()).collect()
-    }
+pub fn builtin_target_plugins() -> Vec<TargetPluginDescriptor<AuditEntry>> {
+    builtin_target_descriptors()
+        .into_iter()
+        .map(|descriptor| descriptor.plugin().clone())
+        .collect()
 }
 
-/// Factory for creating MQTT targets
-pub struct MQTTTargetFactory;
+#[cfg(test)]
+mod tests {
+    use super::builtin_target_descriptors;
+    use rustfs_config::audit::AUDIT_AMQP_KEYS;
+    use rustfs_config::{AMQP_EXCHANGE, AMQP_QUEUE_DIR, AMQP_ROUTING_KEY, AMQP_URL};
+    use rustfs_ecstore::config::KVS;
+    use rustfs_targets::target::ChannelTargetType;
 
-#[async_trait]
-impl TargetFactory for MQTTTargetFactory {
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let args = build_mqtt_args(config, AUDIT_DEFAULT_DIR, TargetType::AuditLog)?;
-        let target = rustfs_targets::target::mqtt::MQTTTarget::new(id, args)?;
-        Ok(Box::new(target))
+    fn amqp_base_config() -> KVS {
+        let mut config = KVS::new();
+        config.insert(AMQP_URL.to_string(), "amqp://127.0.0.1:5672/%2f".to_string());
+        config.insert(AMQP_EXCHANGE.to_string(), "rustfs.audit".to_string());
+        config.insert(AMQP_ROUTING_KEY.to_string(), "audit".to_string());
+        config.insert(AMQP_QUEUE_DIR.to_string(), String::new());
+        config
     }
 
-    fn validate_config(&self, _id: &str, config: &KVS) -> Result<(), TargetError> {
-        validate_mqtt_config(config)
+    #[test]
+    fn builtin_plugins_include_amqp_descriptor() {
+        let plugin = builtin_target_descriptors()
+            .into_iter()
+            .find(|plugin| plugin.plugin().target_type() == ChannelTargetType::Amqp.as_str())
+            .expect("amqp plugin should exist");
+
+        assert!(plugin.plugin().valid_fields().contains(&AMQP_URL));
+        assert!(plugin.plugin().valid_fields().contains(&AMQP_EXCHANGE));
+        assert!(plugin.plugin().valid_fields().contains(&AMQP_ROUTING_KEY));
+        assert_eq!(plugin.plugin().valid_fields().len(), AUDIT_AMQP_KEYS.len());
     }
 
-    fn get_valid_fields(&self) -> HashSet<String> {
-        AUDIT_MQTT_KEYS.iter().map(|s| s.to_string()).collect()
-    }
-}
-
-pub struct NATSTargetFactory;
-
-#[async_trait]
-impl TargetFactory for NATSTargetFactory {
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let args = build_nats_args(config, AUDIT_DEFAULT_DIR, TargetType::AuditLog)?;
-        let target = rustfs_targets::target::nats::NATSTarget::new(id, args)?;
-        Ok(Box::new(target))
-    }
-
-    fn validate_config(&self, _id: &str, config: &KVS) -> Result<(), TargetError> {
-        validate_nats_config(config, AUDIT_DEFAULT_DIR)
-    }
-
-    fn get_valid_fields(&self) -> HashSet<String> {
-        AUDIT_NATS_KEYS.iter().map(|s| s.to_string()).collect()
-    }
-}
-
-pub struct PulsarTargetFactory;
-
-#[async_trait]
-impl TargetFactory for PulsarTargetFactory {
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let args = build_pulsar_args(config, AUDIT_DEFAULT_DIR, TargetType::AuditLog)?;
-        let target = rustfs_targets::target::pulsar::PulsarTarget::new(id, args)?;
-        Ok(Box::new(target))
-    }
-
-    fn validate_config(&self, _id: &str, config: &KVS) -> Result<(), TargetError> {
-        validate_pulsar_config(config, AUDIT_DEFAULT_DIR)
-    }
-
-    fn get_valid_fields(&self) -> HashSet<String> {
-        AUDIT_PULSAR_KEYS.iter().map(|s| s.to_string()).collect()
-    }
-}
-
-pub struct KafkaTargetFactory;
-
-#[async_trait]
-impl TargetFactory for KafkaTargetFactory {
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let args = build_kafka_args(config, AUDIT_DEFAULT_DIR, TargetType::AuditLog)?;
-        let target = rustfs_targets::target::kafka::KafkaTarget::new(id, args)?;
-        Ok(Box::new(target))
-    }
-
-    fn validate_config(&self, _id: &str, config: &KVS) -> Result<(), TargetError> {
-        validate_kafka_config(config, AUDIT_DEFAULT_DIR)
-    }
-
-    fn get_valid_fields(&self) -> HashSet<String> {
-        AUDIT_KAFKA_KEYS.iter().map(|s| s.to_string()).collect()
+    #[test]
+    fn builtin_plugins_create_audit_amqp_target() {
+        let plugin = builtin_target_descriptors()
+            .into_iter()
+            .find(|plugin| plugin.plugin().target_type() == ChannelTargetType::Amqp.as_str())
+            .expect("amqp plugin should exist");
+
+        let target = plugin
+            .plugin()
+            .create_target("primary".to_string(), &amqp_base_config())
+            .expect("AMQP audit target should be created");
+
+        let target_id = target.id();
+        assert_eq!(target_id.id, "primary");
+        assert_eq!(target_id.name, "amqp");
+        assert!(target.store().is_none());
     }
 }

crates/audit/src/lib.rs

@@ -23,6 +23,7 @@ pub mod error;
 pub mod factory;
 pub mod global;
 pub mod observability;
+pub mod pipeline;
 pub mod registry;
 pub mod system;
 
@@ -30,5 +31,6 @@ pub use entity::{ApiDetails, AuditEntry, ObjectVersion};
 pub use error::{AuditError, AuditResult};
 pub use global::*;
 pub use observability::{AuditMetrics, AuditMetricsReport, PerformanceValidation};
+pub use pipeline::{AuditPipeline, AuditRuntimeFacade, AuditRuntimeView};
 pub use registry::AuditRegistry;
 pub use system::{AuditSystem, AuditTargetMetricSnapshot};

crates/audit/src/pipeline.rs

@@ -0,0 +1,355 @@
+//  Copyright 2024 RustFS Team
+//
+//  Licensed under the Apache License, Version 2.0 (the "License");
+//  you may not use this file except in compliance with the License.
+//  You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+//  Unless required by applicable law or agreed to in writing, software
+//  distributed under the License is distributed on an "AS IS" BASIS,
+//  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+//  See the License for the specific language governing permissions and
+//  limitations under the License.
+
+use crate::{AuditEntry, AuditResult, observability, system::AuditTargetMetricSnapshot};
+use rustfs_targets::{
+    BuiltinPluginRuntimeAdapter, PluginRuntimeAdapter, ReplayEvent, ReplayWorkerManager, RuntimeActivation, SharedTarget, Target,
+    target::EntityTarget,
+};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::sync::{Mutex, RwLock};
+use tracing::{error, info, warn};
+
+#[derive(Clone)]
+pub struct AuditPipeline {
+    registry: Arc<Mutex<crate::AuditRegistry>>,
+}
+
+impl AuditPipeline {
+    pub fn new(registry: Arc<Mutex<crate::AuditRegistry>>) -> Self {
+        Self { registry }
+    }
+
+    pub async fn dispatch(&self, entry: Arc<AuditEntry>) -> AuditResult<()> {
+        let start_time = std::time::Instant::now();
+
+        let targets: Vec<SharedTarget<AuditEntry>> = {
+            let registry = self.registry.lock().await;
+            let targets = registry.list_target_values();
+
+            if targets.is_empty() {
+                warn!("No audit targets configured for dispatch");
+                return Ok(());
+            }
+
+            targets
+        };
+
+        let mut tasks = Vec::new();
+
+        for target in targets {
+            let entity_target = EntityTarget {
+                object_name: entry.api.name.clone().unwrap_or_default(),
+                bucket_name: entry.api.bucket.clone().unwrap_or_default(),
+                event_name: entry.event,
+                data: (*entry).clone(),
+            };
+
+            let task = async move {
+                let result = target.save(Arc::new(entity_target)).await;
+                (target.id().to_string(), result)
+            };
+
+            tasks.push(task);
+        }
+
+        let results = futures::future::join_all(tasks).await;
+
+        let mut errors = Vec::new();
+        let mut success_count = 0;
+
+        for (target_key, result) in results {
+            match result {
+                Ok(_) => {
+                    success_count += 1;
+                    observability::record_target_success();
+                }
+                Err(e) => {
+                    error!(target_id = %target_key, error = %e, "Failed to dispatch audit log to target");
+                    errors.push(e);
+                    observability::record_target_failure();
+                }
+            }
+        }
+
+        let dispatch_time = start_time.elapsed();
+
+        if errors.is_empty() {
+            observability::record_audit_success(dispatch_time);
+        } else {
+            observability::record_audit_failure(dispatch_time);
+            warn!(
+                error_count = errors.len(),
+                success_count = success_count,
+                "Some audit targets failed to receive log entry"
+            );
+        }
+
+        Ok(())
+    }
+
+    pub async fn dispatch_batch(&self, entries: Vec<Arc<AuditEntry>>) -> AuditResult<()> {
+        let start_time = std::time::Instant::now();
+
+        let targets: Vec<SharedTarget<AuditEntry>> = {
+            let registry = self.registry.lock().await;
+            let targets = registry.list_target_values();
+
+            if targets.is_empty() {
+                warn!("No audit targets configured for batch dispatch");
+                return Ok(());
+            }
+
+            targets
+        };
+
+        let mut tasks = Vec::new();
+        for target in targets {
+            let entries_clone: Vec<_> = entries.iter().map(Arc::clone).collect();
+
+            let task = async move {
+                let mut success_count = 0;
+                let mut errors = Vec::new();
+                for entry in entries_clone {
+                    let entity_target = EntityTarget {
+                        object_name: entry.api.name.clone().unwrap_or_default(),
+                        bucket_name: entry.api.bucket.clone().unwrap_or_default(),
+                        event_name: entry.event,
+                        data: (*entry).clone(),
+                    };
+                    match target.save(Arc::new(entity_target)).await {
+                        Ok(_) => success_count += 1,
+                        Err(e) => errors.push(e),
+                    }
+                }
+                (target.id().to_string(), success_count, errors)
+            };
+            tasks.push(task);
+        }
+
+        let results = futures::future::join_all(tasks).await;
+        let mut total_success = 0;
+        let mut total_errors = 0;
+        for (_target_id, success_count, errors) in results {
+            total_success += success_count;
+            total_errors += errors.len();
+            for e in errors {
+                error!("Batch dispatch error: {:?}", e);
+            }
+        }
+
+        let dispatch_time = start_time.elapsed();
+        info!(
+            "Batch dispatched {} entries, success: {}, errors: {}, time: {:?}",
+            entries.len(),
+            total_success,
+            total_errors,
+            dispatch_time
+        );
+
+        Ok(())
+    }
+
+    pub async fn snapshot_target_metrics(&self) -> Vec<AuditTargetMetricSnapshot> {
+        let registry = self.registry.lock().await;
+        registry
+            .list_target_values()
+            .into_iter()
+            .map(|target| {
+                let delivery = target.delivery_snapshot();
+                AuditTargetMetricSnapshot {
+                    failed_messages: delivery.failed_messages,
+                    queue_length: delivery.queue_length,
+                    target_id: target.id().to_string(),
+                    total_messages: delivery.total_messages,
+                }
+            })
+            .collect()
+    }
+
+    pub async fn snapshot_target_health(&self) -> Vec<rustfs_targets::RuntimeTargetHealthSnapshot> {
+        let registry = self.registry.lock().await;
+        registry.runtime_manager().health_snapshots().await
+    }
+}
+
+#[derive(Clone)]
+pub struct AuditRuntimeView {
+    registry: Arc<Mutex<crate::AuditRegistry>>,
+}
+
+impl AuditRuntimeView {
+    pub fn new(registry: Arc<Mutex<crate::AuditRegistry>>) -> Self {
+        Self { registry }
+    }
+
+    pub async fn list_targets(&self) -> Vec<String> {
+        let registry = self.registry.lock().await;
+        registry.list_targets()
+    }
+
+    pub async fn get_target_values(&self) -> Vec<rustfs_targets::SharedTarget<AuditEntry>> {
+        let registry = self.registry.lock().await;
+        registry.list_target_values()
+    }
+
+    pub async fn get_target(&self, target_id: &str) -> Option<String> {
+        let registry = self.registry.lock().await;
+        registry.get_target(target_id).map(|target| target.id().to_string())
+    }
+
+    pub async fn enable_target(&self, target_id: &str) -> AuditResult<()> {
+        let registry = self.registry.lock().await;
+        if registry.get_target(target_id).is_some() {
+            info!(target_id = %target_id, "Target enabled");
+            Ok(())
+        } else {
+            Err(crate::AuditError::Configuration(format!("Target not found: {target_id}"), None))
+        }
+    }
+
+    pub async fn disable_target(&self, target_id: &str) -> AuditResult<()> {
+        let registry = self.registry.lock().await;
+        if registry.get_target(target_id).is_some() {
+            info!(target_id = %target_id, "Target disabled");
+            Ok(())
+        } else {
+            Err(crate::AuditError::Configuration(format!("Target not found: {target_id}"), None))
+        }
+    }
+
+    pub async fn remove_target(&self, target_id: &str) -> AuditResult<()> {
+        let mut registry = self.registry.lock().await;
+        if registry.remove_target(target_id).await.is_some() {
+            info!(target_id = %target_id, "Target removed");
+            Ok(())
+        } else {
+            Err(crate::AuditError::Configuration(format!("Target not found: {target_id}"), None))
+        }
+    }
+
+    pub async fn upsert_target(&self, target_id: String, target: Box<dyn Target<AuditEntry> + Send + Sync>) -> AuditResult<()> {
+        if let Err(err) = target.init().await {
+            return Err(crate::AuditError::Target(err));
+        }
+
+        let shared_target: SharedTarget<AuditEntry> = Arc::from(target);
+        let mut registry = self.registry.lock().await;
+        let _ = registry.remove_target(&target_id).await;
+        registry.add_shared_target(target_id.clone(), shared_target);
+        info!(target_id = %target_id, "Target upserted");
+        Ok(())
+    }
+}
+
+#[derive(Clone)]
+pub struct AuditRuntimeFacade {
+    registry: Arc<Mutex<crate::AuditRegistry>>,
+    replay_workers: Arc<RwLock<ReplayWorkerManager>>,
+    runtime_adapter: Arc<dyn PluginRuntimeAdapter<AuditEntry>>,
+}
+
+impl AuditRuntimeFacade {
+    pub fn new(registry: Arc<Mutex<crate::AuditRegistry>>, replay_workers: Arc<RwLock<ReplayWorkerManager>>) -> Self {
+        let runtime_adapter = BuiltinPluginRuntimeAdapter::new(
+            Arc::new(move |event: ReplayEvent<AuditEntry>| {
+                Box::pin(async move {
+                    match event {
+                        ReplayEvent::Delivered { key, target } => {
+                            info!("Successfully sent audit entry, target: {}, key: {}", target.id(), key.to_string());
+                            observability::record_target_success();
+                        }
+                        ReplayEvent::RetryableError { error, target, .. } => match error {
+                            rustfs_targets::TargetError::NotConnected => {
+                                warn!("Target {} not connected, retrying...", target.id());
+                            }
+                            rustfs_targets::TargetError::Timeout(_) => {
+                                warn!("Timeout sending to target {}, retrying...", target.id());
+                            }
+                            _ => {}
+                        },
+                        ReplayEvent::Dropped { reason, target, .. } => {
+                            warn!("Dropped queued payload for target {}: {}", target.id(), reason);
+                            observability::record_target_failure();
+                        }
+                        ReplayEvent::PermanentFailure { error, target, .. } => {
+                            error!("Permanent error for target {}: {}", target.id(), error);
+                            target.record_final_failure();
+                            observability::record_target_failure();
+                        }
+                        ReplayEvent::RetryExhausted { key, target } => {
+                            warn!("Max retries exceeded for key {}, target: {}, skipping", key.to_string(), target.id());
+                            target.record_final_failure();
+                            observability::record_target_failure();
+                        }
+                        ReplayEvent::UnreadableEntry { key, error, target } => {
+                            warn!("Skipping unreadable audit store entry {} for target {}: {}", key, target.id(), error);
+                        }
+                    }
+                })
+            }),
+            Arc::new(|target_id, has_replay| {
+                if has_replay {
+                    info!(target_id = %target_id, "Audit stream processing started");
+                } else {
+                    info!(target_id = %target_id, "No store configured, skip audit stream processing");
+                }
+            }),
+            None,
+            Duration::from_millis(500),
+            Duration::from_millis(500),
+            "Stopping audit stream",
+        );
+
+        Self {
+            registry,
+            replay_workers,
+            runtime_adapter: Arc::new(runtime_adapter),
+        }
+    }
+
+    pub async fn replace_targets(&self, activation: RuntimeActivation<AuditEntry>) -> AuditResult<()> {
+        let mut registry = self.registry.lock().await;
+        let mut replay_workers = self.replay_workers.write().await;
+        self.runtime_adapter
+            .replace_runtime_targets(registry.runtime_manager_mut(), &mut replay_workers, activation)
+            .await
+            .map_err(crate::AuditError::Target)?;
+        Ok(())
+    }
+
+    pub async fn shutdown_runtime(
+        &self,
+        registry: &mut crate::AuditRegistry,
+        replay_workers: &mut ReplayWorkerManager,
+    ) -> AuditResult<()> {
+        self.runtime_adapter
+            .shutdown(registry.runtime_manager_mut(), replay_workers)
+            .await
+            .map_err(crate::AuditError::Target)
+    }
+
+    pub async fn activate_targets_with_replay(
+        &self,
+        targets: Vec<Box<dyn Target<AuditEntry> + Send + Sync>>,
+    ) -> RuntimeActivation<AuditEntry> {
+        self.runtime_adapter.activate_with_replay(targets).await
+    }
+
+    pub async fn stop_replay_workers(&self) {
+        let mut replay_workers = self.replay_workers.write().await;
+        self.runtime_adapter.stop_replay_workers(&mut replay_workers).await;
+    }
+}

crates/audit/src/registry.rs

@@ -12,28 +12,19 @@
 //  See the License for the specific language governing permissions and
 //  limitations under the License.
 
-use crate::{
-    AuditEntry, AuditError, AuditResult,
-    factory::{
-        KafkaTargetFactory, MQTTTargetFactory, NATSTargetFactory, PulsarTargetFactory, TargetFactory, WebhookTargetFactory,
-    },
-};
-use futures::StreamExt;
-use futures::stream::FuturesUnordered;
-use hashbrown::HashMap;
+use crate::{AuditEntry, AuditError, AuditResult, factory::builtin_target_plugins};
 use rustfs_config::audit::AUDIT_ROUTE_PREFIX;
 use rustfs_ecstore::config::{Config, KVS};
 use rustfs_targets::arn::TargetID;
-use rustfs_targets::{Target, TargetError, config::collect_target_configs, target::ChannelTargetType};
-use std::sync::Arc;
-use tracing::{error, info};
+use rustfs_targets::{SharedTarget, Target, TargetError, TargetPluginRegistry, TargetRuntimeManager};
+use tracing::info;
 
 /// Registry for managing audit targets
 pub struct AuditRegistry {
     /// Storage for created targets
-    targets: HashMap<String, Box<dyn Target<AuditEntry> + Send + Sync>>,
-    /// Factories for creating targets
-    factories: HashMap<String, Box<dyn TargetFactory>>,
+    targets: TargetRuntimeManager<AuditEntry>,
+    /// Registered plugins for creating targets
+    plugins: TargetPluginRegistry<AuditEntry>,
 }
 
 impl Default for AuditRegistry {
@@ -45,28 +36,17 @@ impl Default for AuditRegistry {
 impl AuditRegistry {
     /// Creates a new AuditRegistry
     pub fn new() -> Self {
-        let mut registry = AuditRegistry {
-            factories: HashMap::new(),
-            targets: HashMap::new(),
-        };
+        let mut plugins = TargetPluginRegistry::new();
+        plugins.register_all(builtin_target_plugins());
 
-        // Register built-in factories
-        registry.register(ChannelTargetType::Webhook.as_str(), Box::new(WebhookTargetFactory));
-        registry.register(ChannelTargetType::Mqtt.as_str(), Box::new(MQTTTargetFactory));
-        registry.register(ChannelTargetType::Nats.as_str(), Box::new(NATSTargetFactory));
-        registry.register(ChannelTargetType::Pulsar.as_str(), Box::new(PulsarTargetFactory));
-        registry.register(ChannelTargetType::Kafka.as_str(), Box::new(KafkaTargetFactory));
-
-        registry
+        AuditRegistry {
+            targets: TargetRuntimeManager::new(),
+            plugins,
+        }
     }
 
-    /// Registers a new factory for a target type
-    ///
-    /// # Arguments
-    /// * `target_type` - The type of the target (e.g., "webhook", "mqtt").
-    /// * `factory` - The factory instance to create targets of this type.
-    pub fn register(&mut self, target_type: &str, factory: Box<dyn TargetFactory>) {
-        self.factories.insert(target_type.to_string(), factory);
+    pub fn supports_target_type(&self, target_type: &str) -> bool {
+        self.plugins.supports_target_type(target_type)
     }
 
     /// Creates a target of the specified type with the given ID and configuration
@@ -84,16 +64,7 @@ impl AuditRegistry {
         id: String,
         config: &KVS,
     ) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let factory = self
-            .factories
-            .get(target_type)
-            .ok_or_else(|| TargetError::Configuration(format!("Unknown target type: {target_type}")))?;
-
-        // Validate configuration before creating target
-        factory.validate_config(&id, config)?;
-
-        // Create target
-        factory.create_target(id, config).await
+        self.plugins.create_target(target_type, id, config)
     }
 
     /// Creates all targets from a configuration
@@ -109,37 +80,10 @@ impl AuditRegistry {
         &self,
         config: &Config,
     ) -> AuditResult<Vec<Box<dyn Target<AuditEntry> + Send + Sync>>> {
-        let mut tasks = FuturesUnordered::new();
-        for (target_type, factory) in &self.factories {
-            tracing::Span::current().record("target_type", target_type.as_str());
-            info!("Start working on target types...");
-            let valid_fields = factory.get_valid_fields();
-            for (id, merged_config) in collect_target_configs(config, AUDIT_ROUTE_PREFIX, target_type, &valid_fields) {
-                info!(instance_id = %id, "Target is enabled, ready to create a task");
-                let tid = id.clone();
-                let merged_config_arc = Arc::new(merged_config);
-                tasks.push(async move {
-                    let result = factory.create_target(tid.clone(), &merged_config_arc).await;
-                    (tid, result)
-                });
-            }
-        }
-
-        let mut successful_targets = Vec::new();
-        while let Some((id, result)) = tasks.next().await {
-            match result {
-                Ok(target) => {
-                    info!(target_type = %target.id().name, instance_id = %id, "Create a target successfully");
-                    successful_targets.push(target);
-                }
-                Err(e) => {
-                    error!(instance_id = %id, error = %e, "Failed to create a target");
-                }
-            }
-        }
-
-        info!(count = successful_targets.len(), "All target processing completed");
-        Ok(successful_targets)
+        self.plugins
+            .create_targets_from_config(config, AUDIT_ROUTE_PREFIX)
+            .await
+            .map_err(AuditError::from)
     }
 
     /// Adds a target to the registry
@@ -147,8 +91,14 @@ impl AuditRegistry {
     /// # Arguments
     /// * `id` - The identifier for the target.
     /// * `target` - The target instance to be added.
-    pub fn add_target(&mut self, id: String, target: Box<dyn Target<AuditEntry> + Send + Sync>) {
-        self.targets.insert(id, target);
+    pub fn add_target(&mut self, _id: String, target: Box<dyn Target<AuditEntry> + Send + Sync>) {
+        debug_assert_eq!(_id, target.id().to_string());
+        self.targets.add_boxed(target);
+    }
+
+    pub fn add_shared_target(&mut self, _id: String, target: SharedTarget<AuditEntry>) {
+        debug_assert_eq!(_id, target.id().to_string());
+        self.targets.add_arc(target);
     }
 
     /// Removes a target from the registry
@@ -158,8 +108,8 @@ impl AuditRegistry {
     ///
     /// # Returns
     /// * `Option<Box<dyn Target<AuditEntry> + Send + Sync>>` - The removed target if it existed.
-    pub fn remove_target(&mut self, id: &str) -> Option<Box<dyn Target<AuditEntry> + Send + Sync>> {
-        self.targets.remove(id)
+    pub async fn remove_target(&mut self, id: &str) -> Option<rustfs_targets::SharedTarget<AuditEntry>> {
+        self.targets.remove_and_close(id).await
     }
 
     /// Gets a target from the registry
@@ -169,13 +119,21 @@ impl AuditRegistry {
     ///
     /// # Returns
     /// * `Option<&(dyn Target<AuditEntry> + Send + Sync)>` - The target if it exists.
-    pub fn get_target(&self, id: &str) -> Option<&(dyn Target<AuditEntry> + Send + Sync)> {
-        self.targets.get(id).map(|t| t.as_ref())
+    pub fn get_target(&self, id: &str) -> Option<rustfs_targets::SharedTarget<AuditEntry>> {
+        self.targets.get(id)
     }
 
     /// Lists cloned target values for runtime inspection without exposing mutable registry access.
-    pub fn list_target_values(&self) -> Vec<Box<dyn Target<AuditEntry> + Send + Sync>> {
-        self.targets.values().map(|target| target.clone_dyn()).collect()
+    pub fn list_target_values(&self) -> Vec<rustfs_targets::SharedTarget<AuditEntry>> {
+        self.targets.values()
+    }
+
+    pub fn runtime_manager(&self) -> &TargetRuntimeManager<AuditEntry> {
+        &self.targets
+    }
+
+    pub fn runtime_manager_mut(&mut self) -> &mut TargetRuntimeManager<AuditEntry> {
+        &mut self.targets
     }
 
     /// Lists all target IDs
@@ -183,7 +141,7 @@ impl AuditRegistry {
     /// # Returns
     /// * `Vec<String>` - A vector of all target IDs in the registry.
     pub fn list_targets(&self) -> Vec<String> {
-        self.targets.keys().cloned().collect()
+        self.targets.keys()
     }
 
     /// Closes all targets and clears the registry
@@ -191,20 +149,23 @@ impl AuditRegistry {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure.
     pub async fn close_all(&mut self) -> AuditResult<()> {
-        let mut errors = Vec::new();
+        let mut first_error = None;
 
-        for (id, target) in self.targets.drain() {
-            if let Err(e) = target.close().await {
-                error!(target_id = %id, error = %e, "Failed to close audit target");
-                errors.push(e);
+        for target_id in self.targets.keys() {
+            if let Some(target) = self.targets.remove(&target_id)
+                && let Err(err) = target.close().await
+            {
+                tracing::error!(target_id = %target_id, error = %err, "Failed to close target during shutdown");
+                if first_error.is_none() {
+                    first_error = Some(err);
+                }
             }
         }
 
-        if let Some(error) = errors.into_iter().next() {
-            return Err(AuditError::Target(error));
+        match first_error {
+            Some(err) => Err(AuditError::Target(err)),
+            None => Ok(()),
         }
-
-        Ok(())
     }
 
     /// Creates a unique key for a target based on its type and ID
@@ -279,7 +240,107 @@ impl AuditRegistry {
         target: Box<dyn Target<AuditEntry> + Send + Sync>,
     ) -> AuditResult<()> {
         let key = self.create_key(target_type, target_id);
-        self.targets.insert(key, target);
+        debug_assert_eq!(key, target.id().to_string());
+        self.targets.add_boxed(target);
         Ok(())
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::AuditRegistry;
+    use crate::{AuditEntry, AuditError};
+    use rustfs_targets::arn::TargetID;
+    use rustfs_targets::store::{Key, Store};
+    use rustfs_targets::target::{ChannelTargetType, EntityTarget, QueuedPayload, QueuedPayloadMeta};
+    use rustfs_targets::{StoreError, Target, TargetError};
+    use std::sync::Arc;
+    use std::sync::atomic::{AtomicUsize, Ordering};
+
+    #[derive(Clone)]
+    struct CloseTestTarget {
+        id: TargetID,
+        close_calls: Arc<AtomicUsize>,
+        fail_on_close: bool,
+    }
+
+    impl CloseTestTarget {
+        fn new(id: TargetID, close_calls: Arc<AtomicUsize>, fail_on_close: bool) -> Self {
+            Self {
+                id,
+                close_calls,
+                fail_on_close,
+            }
+        }
+    }
+
+    #[async_trait::async_trait]
+    impl Target<AuditEntry> for CloseTestTarget {
+        fn id(&self) -> TargetID {
+            self.id.clone()
+        }
+
+        async fn is_active(&self) -> Result<bool, TargetError> {
+            Ok(true)
+        }
+
+        async fn save(&self, _event: Arc<EntityTarget<AuditEntry>>) -> Result<(), TargetError> {
+            Ok(())
+        }
+
+        async fn send_raw_from_store(&self, _key: Key, _body: Vec<u8>, _meta: QueuedPayloadMeta) -> Result<(), TargetError> {
+            Ok(())
+        }
+
+        async fn close(&self) -> Result<(), TargetError> {
+            self.close_calls.fetch_add(1, Ordering::SeqCst);
+            if self.fail_on_close {
+                Err(TargetError::Unknown("close failed".to_string()))
+            } else {
+                Ok(())
+            }
+        }
+
+        fn store(&self) -> Option<&(dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send + Sync)> {
+            None
+        }
+
+        fn clone_dyn(&self) -> Box<dyn Target<AuditEntry> + Send + Sync> {
+            Box::new(self.clone())
+        }
+
+        fn is_enabled(&self) -> bool {
+            true
+        }
+    }
+
+    #[test]
+    fn registry_registers_amqp_factory() {
+        let registry = AuditRegistry::new();
+
+        assert!(registry.supports_target_type(ChannelTargetType::Amqp.as_str()));
+    }
+
+    #[tokio::test]
+    async fn close_all_returns_first_error_and_clears_targets() {
+        let mut registry = AuditRegistry::new();
+        let ok_calls = Arc::new(AtomicUsize::new(0));
+        let fail_calls = Arc::new(AtomicUsize::new(0));
+
+        let ok_id = TargetID::new("ok".to_string(), "webhook".to_string());
+        let fail_id = TargetID::new("fail".to_string(), "webhook".to_string());
+
+        registry.add_target(ok_id.to_string(), Box::new(CloseTestTarget::new(ok_id, Arc::clone(&ok_calls), false)));
+        registry.add_target(
+            fail_id.to_string(),
+            Box::new(CloseTestTarget::new(fail_id, Arc::clone(&fail_calls), true)),
+        );
+
+        let result = registry.close_all().await;
+
+        assert!(matches!(result, Err(AuditError::Target(TargetError::Unknown(_)))));
+        assert_eq!(ok_calls.load(Ordering::SeqCst), 1);
+        assert_eq!(fail_calls.load(Ordering::SeqCst), 1);
+        assert!(registry.list_targets().is_empty());
+    }
+}

crates/audit/src/system.rs

@@ -12,16 +12,14 @@
 //  See the License for the specific language governing permissions and
 //  limitations under the License.
 
-use crate::{AuditEntry, AuditError, AuditRegistry, AuditResult, observability};
-use hashbrown::HashMap;
-use rustfs_ecstore::config::Config;
-use rustfs_targets::{
-    StoreError, Target, TargetError,
-    store::{Key, Store},
-    target::{EntityTarget, QueuedPayload},
+use crate::{
+    AuditEntry, AuditError, AuditRegistry, AuditResult, observability,
+    pipeline::{AuditPipeline, AuditRuntimeFacade, AuditRuntimeView},
 };
+use rustfs_ecstore::config::Config;
+use rustfs_targets::{ReplayWorkerManager, Target};
 use std::sync::Arc;
-use tokio::sync::{Mutex, RwLock, mpsc};
+use tokio::sync::{Mutex, RwLock};
 use tracing::{error, info, warn};
 
 #[derive(Debug, Clone, Default, PartialEq, Eq)]
@@ -49,7 +47,7 @@ pub struct AuditSystem {
     state: Arc<RwLock<AuditSystemState>>,
     config: Arc<RwLock<Option<Config>>>,
     /// Cancellation senders for active audit stream tasks (target_id -> cancel tx)
-    stream_cancellers: Arc<RwLock<HashMap<String, mpsc::Sender<()>>>>,
+    stream_cancellers: Arc<RwLock<ReplayWorkerManager>>,
 }
 
 impl Default for AuditSystem {
@@ -59,16 +57,68 @@ impl Default for AuditSystem {
 }
 
 impl AuditSystem {
+    fn pipeline(&self) -> AuditPipeline {
+        AuditPipeline::new(self.registry.clone())
+    }
+
+    fn runtime_view(&self) -> AuditRuntimeView {
+        AuditRuntimeView::new(self.registry.clone())
+    }
+
+    fn runtime_facade(&self) -> AuditRuntimeFacade {
+        AuditRuntimeFacade::new(self.registry.clone(), self.stream_cancellers.clone())
+    }
+
     /// Creates a new audit system
     pub fn new() -> Self {
         Self {
             registry: Arc::new(Mutex::new(AuditRegistry::new())),
             state: Arc::new(RwLock::new(AuditSystemState::Stopped)),
             config: Arc::new(RwLock::new(None)),
-            stream_cancellers: Arc::new(RwLock::new(HashMap::new())),
+            stream_cancellers: Arc::new(RwLock::new(ReplayWorkerManager::new())),
         }
     }
 
+    async fn create_targets_from_config(&self, config: &Config) -> AuditResult<Vec<Box<dyn Target<AuditEntry> + Send + Sync>>> {
+        let registry = self.registry.lock().await;
+        registry.create_audit_targets_from_config(config).await
+    }
+
+    async fn clear_runtime_targets(&self) -> AuditResult<()> {
+        {
+            let mut registry = self.registry.lock().await;
+            let mut replay_workers = self.stream_cancellers.write().await;
+            self.runtime_facade()
+                .shutdown_runtime(&mut registry, &mut replay_workers)
+                .await?;
+        }
+
+        let mut state = self.state.write().await;
+        *state = AuditSystemState::Stopped;
+        Ok(())
+    }
+
+    async fn commit_runtime_targets(
+        &self,
+        targets: Vec<Box<dyn Target<AuditEntry> + Send + Sync>>,
+        final_state: AuditSystemState,
+    ) -> AuditResult<()> {
+        if targets.is_empty() {
+            info!("No enabled audit targets found, keeping audit system stopped");
+            self.clear_runtime_targets().await?;
+            return Ok(());
+        }
+
+        info!(target_count = targets.len(), "Created audit targets successfully");
+
+        let activation = self.runtime_facade().activate_targets_with_replay(targets).await;
+        self.runtime_facade().replace_targets(activation).await?;
+
+        let mut state = self.state.write().await;
+        *state = final_state;
+        Ok(())
+    }
+
     /// Starts the audit system with the given configuration
     ///
     /// # Arguments
@@ -103,31 +153,14 @@ impl AuditSystem {
             *config_guard = Some(config.clone());
         }
 
-        // Create targets from configuration
-        let mut registry = self.registry.lock().await;
-        match registry.create_audit_targets_from_config(&config).await {
+        match self.create_targets_from_config(&config).await {
             Ok(targets) => {
-                if targets.is_empty() {
-                    info!("No enabled audit targets found, keeping audit system stopped");
-                    drop(registry);
-                    return Ok(());
-                }
-
                 {
                     let mut state = self.state.write().await;
                     *state = AuditSystemState::Starting;
                 }
 
-                info!(target_count = targets.len(), "Created audit targets successfully");
-
-                // Initialize all targets
-                for target in targets {
-                    self.init_and_register_target(target, &mut registry).await;
-                }
-
-                // Update state to running
-                let mut state = self.state.write().await;
-                *state = AuditSystemState::Running;
+                self.commit_runtime_targets(targets, AuditSystemState::Running).await?;
                 info!("Audit system started successfully");
                 Ok(())
             }
@@ -207,18 +240,10 @@ impl AuditSystem {
         info!("Stopping audit system");
 
         // Stop all stream tasks first
-        self.stop_all_streams().await;
-
-        // Close all targets
-        let mut registry = self.registry.lock().await;
-        if let Err(e) = registry.close_all().await {
+        if let Err(e) = self.clear_runtime_targets().await {
             error!(error = %e, "Failed to close some audit targets");
         }
 
-        // Update state to stopped
-        let mut state = self.state.write().await;
-        *state = AuditSystemState::Stopped;
-
         // Clear configuration
         let mut config_guard = self.config.write().await;
         *config_guard = None;
@@ -248,8 +273,6 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn dispatch(&self, entry: Arc<AuditEntry>) -> AuditResult<()> {
-        let start_time = std::time::Instant::now();
-
         let state = self.state.read().await;
 
         match *state {
@@ -262,77 +285,7 @@ impl AuditSystem {
             }
         }
         drop(state);
-
-        // Collect cloned targets under lock, then dispatch without holding it
-        let targets: Vec<(String, Box<dyn Target<AuditEntry> + Send + Sync>)> = {
-            let registry = self.registry.lock().await;
-            let target_keys = registry.list_targets();
-
-            if target_keys.is_empty() {
-                warn!("No audit targets configured for dispatch");
-                return Ok(());
-            }
-
-            target_keys
-                .into_iter()
-                .filter_map(|key| registry.get_target(&key).map(|t| (key, t.clone_dyn())))
-                .collect()
-        };
-
-        // Dispatch to all targets concurrently (no lock held)
-        let mut tasks = Vec::new();
-
-        for (target_key, target) in targets {
-            let entity_target = EntityTarget {
-                object_name: entry.api.name.clone().unwrap_or_default(),
-                bucket_name: entry.api.bucket.clone().unwrap_or_default(),
-                event_name: entry.event,
-                data: (*entry).clone(),
-            };
-
-            let task = async move {
-                let result = target.save(Arc::new(entity_target)).await;
-                (target_key, result)
-            };
-
-            tasks.push(task);
-        }
-
-        // Execute all dispatch tasks
-        let results = futures::future::join_all(tasks).await;
-
-        let mut errors = Vec::new();
-        let mut success_count = 0;
-
-        for (target_key, result) in results {
-            match result {
-                Ok(_) => {
-                    success_count += 1;
-                    observability::record_target_success();
-                }
-                Err(e) => {
-                    error!(target_id = %target_key, error = %e, "Failed to dispatch audit log to target");
-                    errors.push(e);
-                    observability::record_target_failure();
-                }
-            }
-        }
-
-        let dispatch_time = start_time.elapsed();
-
-        if errors.is_empty() {
-            observability::record_audit_success(dispatch_time);
-        } else {
-            observability::record_audit_failure(dispatch_time);
-            // Log errors but don't fail the entire dispatch
-            warn!(
-                error_count = errors.len(),
-                success_count = success_count,
-                "Some audit targets failed to receive log entry"
-            );
-        }
-
-        Ok(())
+        self.pipeline().dispatch(entry).await
     }
 
     /// Dispatches a batch of audit log entries to all active targets
@@ -343,239 +296,12 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn dispatch_batch(&self, entries: Vec<Arc<AuditEntry>>) -> AuditResult<()> {
-        let start_time = std::time::Instant::now();
-
         let state = self.state.read().await;
         if *state != AuditSystemState::Running {
             return Err(AuditError::NotInitialized("Audit system is not running".to_string()));
         }
         drop(state);
-
-        // Collect targets under lock, then dispatch without holding it
-        let targets: Vec<(String, Box<dyn Target<AuditEntry> + Send + Sync>)> = {
-            let registry = self.registry.lock().await;
-            let target_keys = registry.list_targets();
-
-            if target_keys.is_empty() {
-                warn!("No audit targets configured for batch dispatch");
-                return Ok(());
-            }
-
-            target_keys
-                .into_iter()
-                .filter_map(|key| registry.get_target(&key).map(|t| (key, t.clone_dyn())))
-                .collect()
-        };
-
-        let mut tasks = Vec::new();
-        for (target_key, target) in targets {
-            let entries_clone: Vec<_> = entries.iter().map(Arc::clone).collect();
-            let target_key_clone = target_key.clone();
-
-            let task = async move {
-                let mut success_count = 0;
-                let mut errors = Vec::new();
-                for entry in entries_clone {
-                    let entity_target = EntityTarget {
-                        object_name: entry.api.name.clone().unwrap_or_default(),
-                        bucket_name: entry.api.bucket.clone().unwrap_or_default(),
-                        event_name: entry.event,
-                        data: (*entry).clone(),
-                    };
-                    match target.save(Arc::new(entity_target)).await {
-                        Ok(_) => success_count += 1,
-                        Err(e) => errors.push(e),
-                    }
-                }
-                (target_key_clone, success_count, errors)
-            };
-            tasks.push(task);
-        }
-
-        let results = futures::future::join_all(tasks).await;
-        let mut total_success = 0;
-        let mut total_errors = 0;
-        for (_target_id, success_count, errors) in results {
-            total_success += success_count;
-            total_errors += errors.len();
-            for e in errors {
-                error!("Batch dispatch error: {:?}", e);
-            }
-        }
-
-        let dispatch_time = start_time.elapsed();
-        info!(
-            "Batch dispatched {} entries, success: {}, errors: {}, time: {:?}",
-            entries.len(),
-            total_success,
-            total_errors,
-            dispatch_time
-        );
-
-        Ok(())
-    }
-
-    /// Stops all active audit stream tasks by sending cancellation signals.
-    async fn stop_all_streams(&self) {
-        let mut cancellers = self.stream_cancellers.write().await;
-        for (target_id, cancel_tx) in cancellers.drain() {
-            info!(target_id = %target_id, "Stopping audit stream");
-            let _ = cancel_tx.send(()).await;
-        }
-    }
-
-    /// Initializes a single target: runs init(), starts stream if store is present,
-    /// and adds it to the registry. For store-backed targets, registration and stream
-    /// startup proceed even if init() fails so queued entries can be drained later.
-    async fn init_and_register_target(
-        &self,
-        target: Box<dyn Target<AuditEntry> + Send + Sync>,
-        registry: &mut AuditRegistry,
-    ) -> Option<String> {
-        let target_id = target.id().to_string();
-        let has_store = target.store().is_some();
-
-        if let Err(e) = target.init().await {
-            error!(target_id = %target_id, error = %e, "Failed to initialize audit target");
-            // Non-store targets: init failure is fatal.
-            if !has_store {
-                return None;
-            }
-            // Store-backed targets: still register and start the stream so queued
-            // entries can be drained when connectivity recovers.
-            warn!(
-                target_id = %target_id,
-                "Proceeding with store-backed audit target despite init failure"
-            );
-        }
-
-        if target.is_enabled() {
-            if let Some(store) = target.store() {
-                info!(target_id = %target_id, "Start audit stream processing for target");
-                let store_clone: Box<dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send> = store.boxed_clone();
-                let target_arc: Arc<dyn Target<AuditEntry> + Send + Sync> = Arc::from(target.clone_dyn());
-                let cancel_tx = self.start_audit_stream_with_batching(store_clone, target_arc);
-
-                self.stream_cancellers.write().await.insert(target_id.clone(), cancel_tx);
-                info!(target_id = %target_id, "Audit stream processing started");
-            } else {
-                info!(target_id = %target_id, "No store configured, skip audit stream processing");
-            }
-        } else {
-            info!(target_id = %target_id, "Target disabled, skip audit stream processing");
-        }
-
-        registry.add_target(target_id.clone(), target);
-        Some(target_id)
-    }
-
-    /// Starts the audit stream processing for a target with batching and retry logic
-    ///
-    /// # Arguments
-    /// * `store` - The store from which to read audit entries
-    /// * `target` - The target to which audit entries will be sent
-    ///
-    /// This function spawns a background task that continuously reads audit entries from the provided store
-    /// and attempts to send them to the specified target. It implements retry logic with exponential backoff
-    fn start_audit_stream_with_batching(
-        &self,
-        store: Box<dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send>,
-        target: Arc<dyn Target<AuditEntry> + Send + Sync>,
-    ) -> mpsc::Sender<()> {
-        let (cancel_tx, mut cancel_rx) = mpsc::channel(1);
-        let state = self.state.clone();
-
-        tokio::spawn(async move {
-            use std::time::Duration;
-            use tokio::time::sleep;
-
-            info!("Starting audit stream for target: {}", target.id());
-
-            const MAX_RETRIES: usize = 5;
-            const BASE_RETRY_DELAY: Duration = Duration::from_secs(2);
-
-            loop {
-                // Check for cancellation signal
-                if cancel_rx.try_recv().is_ok() {
-                    info!("Audit stream cancelled for target: {}", target.id());
-                    break;
-                }
-
-                match *state.read().await {
-                    AuditSystemState::Running | AuditSystemState::Paused | AuditSystemState::Starting => {}
-                    _ => {
-                        info!("Audit stream stopped for target: {}", target.id());
-                        break;
-                    }
-                }
-
-                let keys: Vec<Key> = store.list();
-                if keys.is_empty() {
-                    tokio::select! {
-                        _ = sleep(Duration::from_millis(500)) => {},
-                        _ = cancel_rx.recv() => {
-                            info!("Audit stream cancelled during idle for target: {}", target.id());
-                            return;
-                        }
-                    }
-                    continue;
-                }
-
-                for key in keys {
-                    if cancel_rx.try_recv().is_ok() {
-                        info!("Audit stream cancelled during processing for target: {}", target.id());
-                        return;
-                    }
-
-                    let mut retries = 0usize;
-                    let mut success = false;
-
-                    while retries < MAX_RETRIES && !success {
-                        match target.send_from_store(key.clone()).await {
-                            Ok(_) => {
-                                info!("Successfully sent audit entry, target: {}, key: {}", target.id(), key.to_string());
-                                observability::record_target_success();
-                                success = true;
-                            }
-                            Err(e) => {
-                                match &e {
-                                    TargetError::NotConnected => {
-                                        warn!("Target {} not connected, retrying...", target.id());
-                                    }
-                                    TargetError::Timeout(_) => {
-                                        warn!("Timeout sending to target {}, retrying...", target.id());
-                                    }
-                                    TargetError::Dropped(reason) => {
-                                        warn!("Dropped queued payload for target {}: {}", target.id(), reason);
-                                        observability::record_target_failure();
-                                        break;
-                                    }
-                                    _ => {
-                                        error!("Permanent error for target {}: {}", target.id(), e);
-                                        target.record_final_failure();
-                                        observability::record_target_failure();
-                                        break;
-                                    }
-                                }
-                                retries += 1;
-                                let backoff = BASE_RETRY_DELAY * (1 << retries);
-                                sleep(backoff).await;
-                            }
-                        }
-                    }
-
-                    if retries >= MAX_RETRIES && !success {
-                        warn!("Max retries exceeded for key {}, target: {}, skipping", key.to_string(), target.id());
-                        target.record_final_failure();
-                        observability::record_target_failure();
-                    }
-                }
-
-                sleep(Duration::from_millis(100)).await;
-            }
-        });
-
-        cancel_tx
+        self.pipeline().dispatch_batch(entries).await
     }
 
     /// Enables a specific target
@@ -586,15 +312,7 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn enable_target(&self, target_id: &str) -> AuditResult<()> {
-        // This would require storing enabled/disabled state per target
-        // For now, just check if target exists
-        let registry = self.registry.lock().await;
-        if registry.get_target(target_id).is_some() {
-            info!(target_id = %target_id, "Target enabled");
-            Ok(())
-        } else {
-            Err(AuditError::Configuration(format!("Target not found: {target_id}"), None))
-        }
+        self.runtime_view().enable_target(target_id).await
     }
 
     /// Disables a specific target
@@ -605,15 +323,7 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn disable_target(&self, target_id: &str) -> AuditResult<()> {
-        // This would require storing enabled/disabled state per target
-        // For now, just check if target exists
-        let registry = self.registry.lock().await;
-        if registry.get_target(target_id).is_some() {
-            info!(target_id = %target_id, "Target disabled");
-            Ok(())
-        } else {
-            Err(AuditError::Configuration(format!("Target not found: {target_id}"), None))
-        }
+        self.runtime_view().disable_target(target_id).await
     }
 
     /// Removes a target from the system
@@ -624,16 +334,7 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn remove_target(&self, target_id: &str) -> AuditResult<()> {
-        let mut registry = self.registry.lock().await;
-        if let Some(target) = registry.remove_target(target_id) {
-            if let Err(e) = target.close().await {
-                error!(target_id = %target_id, error = %e, "Failed to close removed target");
-            }
-            info!(target_id = %target_id, "Target removed");
-            Ok(())
-        } else {
-            Err(AuditError::Configuration(format!("Target not found: {target_id}"), None))
-        }
+        self.runtime_view().remove_target(target_id).await
     }
 
     /// Updates or inserts a target
@@ -645,23 +346,7 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn upsert_target(&self, target_id: String, target: Box<dyn Target<AuditEntry> + Send + Sync>) -> AuditResult<()> {
-        let mut registry = self.registry.lock().await;
-
-        // Initialize the target
-        if let Err(e) = target.init().await {
-            return Err(AuditError::Target(e));
-        }
-
-        // Remove existing target if present
-        if let Some(old_target) = registry.remove_target(&target_id)
-            && let Err(e) = old_target.close().await
-        {
-            error!(target_id = %target_id, error = %e, "Failed to close old target during upsert");
-        }
-
-        registry.add_target(target_id.clone(), target);
-        info!(target_id = %target_id, "Target upserted");
-        Ok(())
+        self.runtime_view().upsert_target(target_id, target).await
     }
 
     /// Lists all targets
@@ -669,33 +354,27 @@ impl AuditSystem {
     /// # Returns
     /// * `Vec<String>` - List of target IDs
     pub async fn list_targets(&self) -> Vec<String> {
-        let registry = self.registry.lock().await;
-        registry.list_targets()
+        self.runtime_view().list_targets().await
     }
 
     /// Returns cloned target values for read-only runtime inspection.
-    pub async fn get_target_values(&self) -> Vec<Box<dyn Target<AuditEntry> + Send + Sync>> {
-        let registry = self.registry.lock().await;
-        registry.list_target_values()
+    pub async fn get_target_values(&self) -> Vec<rustfs_targets::SharedTarget<AuditEntry>> {
+        self.runtime_view().get_target_values().await
     }
 
     /// Returns per-target delivery metrics for Prometheus collection.
     pub async fn snapshot_target_metrics(&self) -> Vec<AuditTargetMetricSnapshot> {
-        let targets = self.get_target_values().await;
-        let mut snapshots = Vec::with_capacity(targets.len());
+        self.pipeline().snapshot_target_metrics().await
+    }
 
-        for target in targets {
-            let delivery = target.delivery_snapshot();
-            snapshots.push(AuditTargetMetricSnapshot {
-                failed_messages: delivery.failed_messages,
-                queue_length: delivery.queue_length,
-                target_id: target.id().to_string(),
-                total_messages: delivery.total_messages,
-            });
-        }
+    pub async fn snapshot_target_health(&self) -> Vec<rustfs_targets::RuntimeTargetHealthSnapshot> {
+        self.pipeline().snapshot_target_health().await
+    }
 
-        snapshots.sort_by(|a, b| a.target_id.cmp(&b.target_id));
-        snapshots
+    pub async fn runtime_status_snapshot(&self) -> rustfs_targets::RuntimeStatusSnapshot {
+        let replay_workers = self.stream_cancellers.read().await;
+        let registry = self.registry.lock().await;
+        registry.runtime_manager().status_snapshot(&replay_workers)
     }
 
     /// Gets information about a specific target
@@ -706,8 +385,7 @@ impl AuditSystem {
     /// # Returns
     /// * `Option<String>` - Target ID if found
     pub async fn get_target(&self, target_id: &str) -> Option<String> {
-        let registry = self.registry.lock().await;
-        registry.get_target(target_id).map(|target| target.id().to_string())
+        self.runtime_view().get_target(target_id).await
     }
 
     /// Reloads configuration and updates targets
@@ -722,32 +400,20 @@ impl AuditSystem {
 
         observability::record_config_reload();
 
-        // Stop all existing stream tasks first
-        self.stop_all_streams().await;
-
         // Store new configuration
         {
             let mut config_guard = self.config.write().await;
             *config_guard = Some(new_config.clone());
         }
 
-        // Close all existing targets
-        let mut registry = self.registry.lock().await;
-        if let Err(e) = registry.close_all().await {
-            error!(error = %e, "Failed to close existing targets during reload");
-        }
+        let final_state = match self.get_state().await {
+            AuditSystemState::Paused => AuditSystemState::Paused,
+            _ => AuditSystemState::Running,
+        };
 
-        // Create new targets from updated configuration
-        match registry.create_audit_targets_from_config(&new_config).await {
+        match self.create_targets_from_config(&new_config).await {
             Ok(targets) => {
-                info!(target_count = targets.len(), "Reloaded audit targets successfully");
-
-                for target in targets {
-                    if let Some(target_id) = self.init_and_register_target(target, &mut registry).await {
-                        info!(target_id = %target_id, "Target initialized (reload)");
-                    }
-                }
-
+                self.commit_runtime_targets(targets, final_state).await?;
                 info!("Audit configuration reloaded successfully");
                 Ok(())
             }
@@ -779,3 +445,106 @@ impl AuditSystem {
         observability::reset_metrics().await;
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::{AuditSystem, AuditSystemState};
+    use async_trait::async_trait;
+    use rustfs_targets::ReplayWorkerManager;
+    use rustfs_targets::arn::TargetID;
+    use rustfs_targets::store::{Key, Store};
+    use rustfs_targets::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta};
+    use rustfs_targets::{StoreError, Target, TargetError};
+    use serde::{Serialize, de::DeserializeOwned};
+    use std::collections::HashMap;
+    use std::sync::Arc;
+    use std::sync::atomic::{AtomicUsize, Ordering};
+    use tokio::sync::mpsc;
+
+    #[derive(Clone)]
+    struct TestTarget {
+        close_calls: Arc<AtomicUsize>,
+        id: TargetID,
+    }
+
+    impl TestTarget {
+        fn new(id: &str, name: &str) -> Self {
+            Self {
+                close_calls: Arc::new(AtomicUsize::new(0)),
+                id: TargetID::new(id.to_string(), name.to_string()),
+            }
+        }
+    }
+
+    #[async_trait]
+    impl<E> Target<E> for TestTarget
+    where
+        E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned,
+    {
+        fn id(&self) -> TargetID {
+            self.id.clone()
+        }
+
+        async fn is_active(&self) -> Result<bool, TargetError> {
+            Ok(true)
+        }
+
+        async fn save(&self, _event: Arc<EntityTarget<E>>) -> Result<(), TargetError> {
+            Ok(())
+        }
+
+        async fn send_raw_from_store(&self, _key: Key, _body: Vec<u8>, _meta: QueuedPayloadMeta) -> Result<(), TargetError> {
+            Ok(())
+        }
+
+        async fn close(&self) -> Result<(), TargetError> {
+            self.close_calls.fetch_add(1, Ordering::SeqCst);
+            Ok(())
+        }
+
+        fn store(&self) -> Option<&(dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send + Sync)> {
+            None
+        }
+
+        fn clone_dyn(&self) -> Box<dyn Target<E> + Send + Sync> {
+            Box::new(self.clone())
+        }
+
+        fn is_enabled(&self) -> bool {
+            true
+        }
+    }
+
+    #[tokio::test]
+    async fn reload_with_empty_config_stops_existing_runtime() {
+        let system = AuditSystem::new();
+        let target = TestTarget::new("primary", "webhook");
+        let close_calls = Arc::clone(&target.close_calls);
+
+        {
+            let mut registry = system.registry.lock().await;
+            registry.add_target("primary:webhook".to_string(), Box::new(target));
+        }
+        {
+            let mut state = system.state.write().await;
+            *state = AuditSystemState::Running;
+        }
+        {
+            let mut replay_workers = system.stream_cancellers.write().await;
+            let (cancel_tx, _cancel_rx) = mpsc::channel(1);
+            replay_workers.insert("primary:webhook".to_string(), cancel_tx);
+            assert_eq!(replay_workers.len(), 1);
+        }
+
+        system
+            .reload_config(rustfs_ecstore::config::Config(HashMap::new()))
+            .await
+            .expect("reload with empty config should succeed");
+
+        assert_eq!(system.get_state().await, AuditSystemState::Stopped);
+        assert!(system.list_targets().await.is_empty());
+        assert_eq!(system.runtime_status_snapshot().await, ReplayWorkerManager::new().snapshot(0));
+        assert_eq!(close_calls.load(Ordering::SeqCst), 1);
+        assert_eq!(*system.config.read().await, Some(rustfs_ecstore::config::Config(HashMap::new())));
+    }
+}

crates/audit/tests/pipeline_layer_test.rs

@@ -0,0 +1,170 @@
+//  Copyright 2024 RustFS Team
+//
+//  Licensed under the Apache License, Version 2.0 (the "License");
+//  you may not use this file except in compliance with the License.
+//  You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+//  Unless required by applicable law or agreed to in writing, software
+//  distributed under the License is distributed on an "AS IS" BASIS,
+//  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+//  See the License for the specific language governing permissions and
+//  limitations under the License.
+
+use async_trait::async_trait;
+use rustfs_audit::{AuditPipeline, AuditRegistry, AuditRuntimeFacade, AuditRuntimeView};
+use rustfs_targets::arn::TargetID;
+use rustfs_targets::store::{Key, Store};
+use rustfs_targets::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta};
+use rustfs_targets::{StoreError, Target, TargetError};
+use serde::{Serialize, de::DeserializeOwned};
+use std::sync::Arc;
+use std::sync::atomic::{AtomicUsize, Ordering};
+use tokio::sync::{Mutex, RwLock};
+
+#[derive(Clone)]
+struct TestTarget {
+    close_calls: Arc<AtomicUsize>,
+    id: TargetID,
+    init_calls: Arc<AtomicUsize>,
+}
+
+impl TestTarget {
+    fn new(id: &str, name: &str) -> Self {
+        Self {
+            close_calls: Arc::new(AtomicUsize::new(0)),
+            id: TargetID::new(id.to_string(), name.to_string()),
+            init_calls: Arc::new(AtomicUsize::new(0)),
+        }
+    }
+}
+
+#[async_trait]
+impl<E> Target<E> for TestTarget
+where
+    E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned,
+{
+    fn id(&self) -> TargetID {
+        self.id.clone()
+    }
+
+    async fn is_active(&self) -> Result<bool, TargetError> {
+        Ok(true)
+    }
+
+    async fn save(&self, _event: Arc<EntityTarget<E>>) -> Result<(), TargetError> {
+        Ok(())
+    }
+
+    async fn send_raw_from_store(&self, _key: Key, _body: Vec<u8>, _meta: QueuedPayloadMeta) -> Result<(), TargetError> {
+        Ok(())
+    }
+
+    async fn close(&self) -> Result<(), TargetError> {
+        self.close_calls.fetch_add(1, Ordering::SeqCst);
+        Ok(())
+    }
+
+    fn store(&self) -> Option<&(dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send + Sync)> {
+        None
+    }
+
+    fn clone_dyn(&self) -> Box<dyn Target<E> + Send + Sync> {
+        Box::new(self.clone())
+    }
+
+    async fn init(&self) -> Result<(), TargetError> {
+        self.init_calls.fetch_add(1, Ordering::SeqCst);
+        Ok(())
+    }
+
+    fn is_enabled(&self) -> bool {
+        true
+    }
+}
+
+#[tokio::test]
+async fn audit_runtime_view_lists_empty_targets() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let runtime_view = AuditRuntimeView::new(registry);
+
+    assert!(runtime_view.list_targets().await.is_empty());
+    assert!(runtime_view.get_target_values().await.is_empty());
+    assert!(runtime_view.get_target("missing").await.is_none());
+}
+
+#[tokio::test]
+async fn audit_pipeline_reports_empty_runtime_snapshots() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let pipeline = AuditPipeline::new(registry);
+
+    assert!(pipeline.snapshot_target_metrics().await.is_empty());
+    assert!(pipeline.snapshot_target_health().await.is_empty());
+}
+
+#[tokio::test]
+async fn audit_runtime_facade_stops_empty_replay_workers() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let replay_workers = Arc::new(RwLock::new(rustfs_targets::ReplayWorkerManager::new()));
+    let facade = AuditRuntimeFacade::new(registry, replay_workers);
+
+    facade.stop_replay_workers().await;
+}
+
+#[tokio::test]
+async fn audit_runtime_facade_activates_empty_target_list() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let replay_workers = Arc::new(RwLock::new(rustfs_targets::ReplayWorkerManager::new()));
+    let facade = AuditRuntimeFacade::new(registry, replay_workers);
+
+    let activation = facade.activate_targets_with_replay(Vec::new()).await;
+    assert!(activation.targets.is_empty());
+    assert_eq!(activation.replay_workers.len(), 0);
+}
+
+#[tokio::test]
+async fn audit_runtime_view_upsert_and_remove_target() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let runtime_view = AuditRuntimeView::new(registry.clone());
+    let target = TestTarget::new("primary", "webhook");
+    let init_calls = Arc::clone(&target.init_calls);
+    let close_calls = Arc::clone(&target.close_calls);
+
+    runtime_view
+        .upsert_target("primary:webhook".to_string(), Box::new(target))
+        .await
+        .expect("upsert should succeed");
+
+    assert_eq!(runtime_view.list_targets().await, vec!["primary:webhook".to_string()]);
+    assert_eq!(init_calls.load(Ordering::SeqCst), 1);
+
+    runtime_view
+        .remove_target("primary:webhook")
+        .await
+        .expect("remove should succeed");
+
+    assert!(runtime_view.list_targets().await.is_empty());
+    assert_eq!(close_calls.load(Ordering::SeqCst), 1);
+}
+
+#[tokio::test]
+async fn audit_runtime_facade_replace_targets_commits_runtime_state() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let replay_workers = Arc::new(RwLock::new(rustfs_targets::ReplayWorkerManager::new()));
+    let facade = AuditRuntimeFacade::new(registry.clone(), replay_workers.clone());
+    let target = TestTarget::new("primary", "webhook");
+    let activation = rustfs_targets::RuntimeActivation {
+        replay_workers: rustfs_targets::ReplayWorkerManager::new(),
+        targets: vec![Arc::new(target) as rustfs_targets::SharedTarget<rustfs_audit::AuditEntry>],
+    };
+
+    facade
+        .replace_targets(activation)
+        .await
+        .expect("replace_targets should succeed");
+
+    let runtime_view = AuditRuntimeView::new(registry);
+    assert_eq!(runtime_view.list_targets().await, vec!["primary:webhook".to_string()]);
+    assert_eq!(replay_workers.read().await.len(), 0);
+}

crates/config/src/audit/amqp.rs

@@ -0,0 +1,60 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const AUDIT_AMQP_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::AMQP_URL,
+    crate::AMQP_EXCHANGE,
+    crate::AMQP_ROUTING_KEY,
+    crate::AMQP_MANDATORY,
+    crate::AMQP_PERSISTENT,
+    crate::AMQP_USERNAME,
+    crate::AMQP_PASSWORD,
+    crate::AMQP_TLS_CA,
+    crate::AMQP_TLS_CLIENT_CERT,
+    crate::AMQP_TLS_CLIENT_KEY,
+    crate::AMQP_QUEUE_DIR,
+    crate::AMQP_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_AUDIT_AMQP_ENABLE: &str = "RUSTFS_AUDIT_AMQP_ENABLE";
+pub const ENV_AUDIT_AMQP_URL: &str = "RUSTFS_AUDIT_AMQP_URL";
+pub const ENV_AUDIT_AMQP_EXCHANGE: &str = "RUSTFS_AUDIT_AMQP_EXCHANGE";
+pub const ENV_AUDIT_AMQP_ROUTING_KEY: &str = "RUSTFS_AUDIT_AMQP_ROUTING_KEY";
+pub const ENV_AUDIT_AMQP_MANDATORY: &str = "RUSTFS_AUDIT_AMQP_MANDATORY";
+pub const ENV_AUDIT_AMQP_PERSISTENT: &str = "RUSTFS_AUDIT_AMQP_PERSISTENT";
+pub const ENV_AUDIT_AMQP_USERNAME: &str = "RUSTFS_AUDIT_AMQP_USERNAME";
+pub const ENV_AUDIT_AMQP_PASSWORD: &str = "RUSTFS_AUDIT_AMQP_PASSWORD";
+pub const ENV_AUDIT_AMQP_TLS_CA: &str = "RUSTFS_AUDIT_AMQP_TLS_CA";
+pub const ENV_AUDIT_AMQP_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_AMQP_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_AMQP_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_AMQP_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_AMQP_QUEUE_DIR: &str = "RUSTFS_AUDIT_AMQP_QUEUE_DIR";
+pub const ENV_AUDIT_AMQP_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_AMQP_QUEUE_LIMIT";
+
+pub const ENV_AUDIT_AMQP_KEYS: &[&str; 13] = &[
+    ENV_AUDIT_AMQP_ENABLE,
+    ENV_AUDIT_AMQP_URL,
+    ENV_AUDIT_AMQP_EXCHANGE,
+    ENV_AUDIT_AMQP_ROUTING_KEY,
+    ENV_AUDIT_AMQP_MANDATORY,
+    ENV_AUDIT_AMQP_PERSISTENT,
+    ENV_AUDIT_AMQP_USERNAME,
+    ENV_AUDIT_AMQP_PASSWORD,
+    ENV_AUDIT_AMQP_TLS_CA,
+    ENV_AUDIT_AMQP_TLS_CLIENT_CERT,
+    ENV_AUDIT_AMQP_TLS_CLIENT_KEY,
+    ENV_AUDIT_AMQP_QUEUE_DIR,
+    ENV_AUDIT_AMQP_QUEUE_LIMIT,
+];

crates/config/src/audit/mod.rs

@@ -16,16 +16,24 @@
 //! This module defines the configuration for audit systems, including
 //! webhook and MQTT audit-related settings.
 
+mod amqp;
 mod kafka;
 mod mqtt;
+mod mysql;
 mod nats;
+mod postgres;
 mod pulsar;
+mod redis;
 mod webhook;
 
+pub use amqp::*;
 pub use kafka::*;
 pub use mqtt::*;
+pub use mysql::*;
 pub use nats::*;
+pub use postgres::*;
 pub use pulsar::*;
+pub use redis::*;
 pub use webhook::*;
 
 use crate::DEFAULT_DELIMITER;
@@ -35,17 +43,24 @@ pub const AUDIT_PREFIX: &str = "audit";
 pub const AUDIT_ROUTE_PREFIX: &str = const_str::concat!(AUDIT_PREFIX, DEFAULT_DELIMITER);
 
 pub const AUDIT_WEBHOOK_SUB_SYS: &str = "audit_webhook";
+pub const AUDIT_AMQP_SUB_SYS: &str = "audit_amqp";
 pub const AUDIT_KAFKA_SUB_SYS: &str = "audit_kafka";
 pub const AUDIT_MQTT_SUB_SYS: &str = "audit_mqtt";
+pub const AUDIT_MYSQL_SUB_SYS: &str = "audit_mysql";
 pub const AUDIT_NATS_SUB_SYS: &str = "audit_nats";
+pub const AUDIT_POSTGRES_SUB_SYS: &str = "audit_postgres";
 pub const AUDIT_PULSAR_SUB_SYS: &str = "audit_pulsar";
-
+pub const AUDIT_REDIS_SUB_SYS: &str = "audit_redis";
+pub const AUDIT_REDIS_DEFAULT_CHANNEL: &str = "rustfs_audit_channel";
 pub const AUDIT_STORE_EXTENSION: &str = ".audit";
-#[allow(dead_code)]
 pub const AUDIT_SUB_SYSTEMS: &[&str] = &[
+    AUDIT_AMQP_SUB_SYS,
     AUDIT_KAFKA_SUB_SYS,
     AUDIT_MQTT_SUB_SYS,
+    AUDIT_MYSQL_SUB_SYS,
     AUDIT_NATS_SUB_SYS,
+    AUDIT_POSTGRES_SUB_SYS,
     AUDIT_PULSAR_SUB_SYS,
+    AUDIT_REDIS_SUB_SYS,
     AUDIT_WEBHOOK_SUB_SYS,
 ];

crates/config/src/audit/mysql.rs

@@ -0,0 +1,53 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// A list of all valid configuration keys for an audit MySQL target.
+pub const AUDIT_MYSQL_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::MYSQL_DSN_STRING,
+    crate::MYSQL_TABLE,
+    crate::MYSQL_FORMAT,
+    crate::MYSQL_TLS_CA,
+    crate::MYSQL_TLS_CLIENT_CERT,
+    crate::MYSQL_TLS_CLIENT_KEY,
+    crate::MYSQL_QUEUE_DIR,
+    crate::MYSQL_QUEUE_LIMIT,
+    crate::MYSQL_MAX_OPEN_CONNECTIONS,
+    crate::COMMENT_KEY,
+];
+
+// MySQL environment variables for audit target.
+pub const ENV_AUDIT_MYSQL_ENABLE: &str = "RUSTFS_AUDIT_MYSQL_ENABLE";
+pub const ENV_AUDIT_MYSQL_DSN_STRING: &str = "RUSTFS_AUDIT_MYSQL_DSN_STRING";
+pub const ENV_AUDIT_MYSQL_TABLE: &str = "RUSTFS_AUDIT_MYSQL_TABLE";
+pub const ENV_AUDIT_MYSQL_FORMAT: &str = "RUSTFS_AUDIT_MYSQL_FORMAT";
+pub const ENV_AUDIT_MYSQL_TLS_CA: &str = "RUSTFS_AUDIT_MYSQL_TLS_CA";
+pub const ENV_AUDIT_MYSQL_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_MYSQL_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_MYSQL_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_MYSQL_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_MYSQL_QUEUE_DIR: &str = "RUSTFS_AUDIT_MYSQL_QUEUE_DIR";
+pub const ENV_AUDIT_MYSQL_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_MYSQL_QUEUE_LIMIT";
+pub const ENV_AUDIT_MYSQL_MAX_OPEN_CONNECTIONS: &str = "RUSTFS_AUDIT_MYSQL_MAX_OPEN_CONNECTIONS";
+
+pub const ENV_AUDIT_MYSQL_KEYS: &[&str; 10] = &[
+    ENV_AUDIT_MYSQL_ENABLE,
+    ENV_AUDIT_MYSQL_DSN_STRING,
+    ENV_AUDIT_MYSQL_TABLE,
+    ENV_AUDIT_MYSQL_FORMAT,
+    ENV_AUDIT_MYSQL_TLS_CA,
+    ENV_AUDIT_MYSQL_TLS_CLIENT_CERT,
+    ENV_AUDIT_MYSQL_TLS_CLIENT_KEY,
+    ENV_AUDIT_MYSQL_QUEUE_DIR,
+    ENV_AUDIT_MYSQL_QUEUE_LIMIT,
+    ENV_AUDIT_MYSQL_MAX_OPEN_CONNECTIONS,
+];

crates/config/src/audit/postgres.rs

@@ -0,0 +1,51 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const AUDIT_POSTGRES_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::POSTGRES_DSN_STRING,
+    crate::POSTGRES_TABLE,
+    crate::POSTGRES_FORMAT,
+    crate::POSTGRES_TLS_REQUIRED,
+    crate::POSTGRES_TLS_CA,
+    crate::POSTGRES_TLS_CLIENT_CERT,
+    crate::POSTGRES_TLS_CLIENT_KEY,
+    crate::POSTGRES_QUEUE_DIR,
+    crate::POSTGRES_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_AUDIT_POSTGRES_ENABLE: &str = "RUSTFS_AUDIT_POSTGRES_ENABLE";
+pub const ENV_AUDIT_POSTGRES_DSN_STRING: &str = "RUSTFS_AUDIT_POSTGRES_DSN_STRING";
+pub const ENV_AUDIT_POSTGRES_TABLE: &str = "RUSTFS_AUDIT_POSTGRES_TABLE";
+pub const ENV_AUDIT_POSTGRES_FORMAT: &str = "RUSTFS_AUDIT_POSTGRES_FORMAT";
+pub const ENV_AUDIT_POSTGRES_TLS_REQUIRED: &str = "RUSTFS_AUDIT_POSTGRES_TLS_REQUIRED";
+pub const ENV_AUDIT_POSTGRES_TLS_CA: &str = "RUSTFS_AUDIT_POSTGRES_TLS_CA";
+pub const ENV_AUDIT_POSTGRES_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_POSTGRES_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_POSTGRES_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_POSTGRES_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_POSTGRES_QUEUE_DIR: &str = "RUSTFS_AUDIT_POSTGRES_QUEUE_DIR";
+pub const ENV_AUDIT_POSTGRES_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_POSTGRES_QUEUE_LIMIT";
+
+pub const ENV_AUDIT_POSTGRES_KEYS: &[&str; 10] = &[
+    ENV_AUDIT_POSTGRES_ENABLE,
+    ENV_AUDIT_POSTGRES_DSN_STRING,
+    ENV_AUDIT_POSTGRES_TABLE,
+    ENV_AUDIT_POSTGRES_FORMAT,
+    ENV_AUDIT_POSTGRES_TLS_REQUIRED,
+    ENV_AUDIT_POSTGRES_TLS_CA,
+    ENV_AUDIT_POSTGRES_TLS_CLIENT_CERT,
+    ENV_AUDIT_POSTGRES_TLS_CLIENT_KEY,
+    ENV_AUDIT_POSTGRES_QUEUE_DIR,
+    ENV_AUDIT_POSTGRES_QUEUE_LIMIT,
+];

crates/config/src/audit/redis.rs

@@ -0,0 +1,81 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const ENV_AUDIT_REDIS_ENABLE: &str = "RUSTFS_AUDIT_REDIS_ENABLE";
+pub const ENV_AUDIT_REDIS_URL: &str = "RUSTFS_AUDIT_REDIS_URL";
+pub const ENV_AUDIT_REDIS_CHANNEL: &str = "RUSTFS_AUDIT_REDIS_CHANNEL";
+pub const ENV_AUDIT_REDIS_USERNAME: &str = "RUSTFS_AUDIT_REDIS_USERNAME";
+pub const ENV_AUDIT_REDIS_PASSWORD: &str = "RUSTFS_AUDIT_REDIS_PASSWORD";
+pub const ENV_AUDIT_REDIS_KEEP_ALIVE_INTERVAL: &str = "RUSTFS_AUDIT_REDIS_KEEP_ALIVE_INTERVAL";
+pub const ENV_AUDIT_REDIS_QUEUE_DIR: &str = "RUSTFS_AUDIT_REDIS_QUEUE_DIR";
+pub const ENV_AUDIT_REDIS_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_REDIS_QUEUE_LIMIT";
+pub const ENV_AUDIT_REDIS_MAX_RETRY_ATTEMPTS: &str = "RUSTFS_AUDIT_REDIS_MAX_RETRY_ATTEMPTS";
+pub const ENV_AUDIT_REDIS_RECONNECT_RETRY_ATTEMPTS: &str = "RUSTFS_AUDIT_REDIS_RECONNECT_RETRY_ATTEMPTS";
+pub const ENV_AUDIT_REDIS_MIN_RETRY_DELAY: &str = "RUSTFS_AUDIT_REDIS_MIN_RETRY_DELAY";
+pub const ENV_AUDIT_REDIS_MAX_RETRY_DELAY: &str = "RUSTFS_AUDIT_REDIS_MAX_RETRY_DELAY";
+pub const ENV_AUDIT_REDIS_CONNECTION_TIMEOUT: &str = "RUSTFS_AUDIT_REDIS_CONNECTION_TIMEOUT";
+pub const ENV_AUDIT_REDIS_RESPONSE_TIMEOUT: &str = "RUSTFS_AUDIT_REDIS_RESPONSE_TIMEOUT";
+pub const ENV_AUDIT_REDIS_PIPELINE_BUFFER_SIZE: &str = "RUSTFS_AUDIT_REDIS_PIPELINE_BUFFER_SIZE";
+pub const ENV_AUDIT_REDIS_TLS_POLICY: &str = "RUSTFS_AUDIT_REDIS_TLS_POLICY";
+pub const ENV_AUDIT_REDIS_TLS_CA: &str = "RUSTFS_AUDIT_REDIS_TLS_CA";
+pub const ENV_AUDIT_REDIS_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_REDIS_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_REDIS_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_REDIS_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_REDIS_TLS_ALLOW_INSECURE: &str = "RUSTFS_AUDIT_REDIS_TLS_ALLOW_INSECURE";
+
+pub const ENV_AUDIT_REDIS_KEYS: &[&str; 20] = &[
+    ENV_AUDIT_REDIS_ENABLE,
+    ENV_AUDIT_REDIS_URL,
+    ENV_AUDIT_REDIS_CHANNEL,
+    ENV_AUDIT_REDIS_USERNAME,
+    ENV_AUDIT_REDIS_PASSWORD,
+    ENV_AUDIT_REDIS_KEEP_ALIVE_INTERVAL,
+    ENV_AUDIT_REDIS_QUEUE_DIR,
+    ENV_AUDIT_REDIS_QUEUE_LIMIT,
+    ENV_AUDIT_REDIS_MAX_RETRY_ATTEMPTS,
+    ENV_AUDIT_REDIS_RECONNECT_RETRY_ATTEMPTS,
+    ENV_AUDIT_REDIS_MIN_RETRY_DELAY,
+    ENV_AUDIT_REDIS_MAX_RETRY_DELAY,
+    ENV_AUDIT_REDIS_CONNECTION_TIMEOUT,
+    ENV_AUDIT_REDIS_RESPONSE_TIMEOUT,
+    ENV_AUDIT_REDIS_PIPELINE_BUFFER_SIZE,
+    ENV_AUDIT_REDIS_TLS_POLICY,
+    ENV_AUDIT_REDIS_TLS_CA,
+    ENV_AUDIT_REDIS_TLS_CLIENT_CERT,
+    ENV_AUDIT_REDIS_TLS_CLIENT_KEY,
+    ENV_AUDIT_REDIS_TLS_ALLOW_INSECURE,
+];
+
+pub const AUDIT_REDIS_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::REDIS_URL,
+    crate::REDIS_CHANNEL,
+    crate::REDIS_USERNAME,
+    crate::REDIS_PASSWORD,
+    crate::REDIS_KEEP_ALIVE_INTERVAL,
+    crate::REDIS_QUEUE_DIR,
+    crate::REDIS_QUEUE_LIMIT,
+    crate::REDIS_MAX_RETRY_ATTEMPTS,
+    crate::REDIS_RECONNECT_RETRY_ATTEMPTS,
+    crate::REDIS_MIN_RETRY_DELAY,
+    crate::REDIS_MAX_RETRY_DELAY,
+    crate::REDIS_CONNECTION_TIMEOUT,
+    crate::REDIS_RESPONSE_TIMEOUT,
+    crate::REDIS_PIPELINE_BUFFER_SIZE,
+    crate::REDIS_TLS_POLICY,
+    crate::REDIS_TLS_CA,
+    crate::REDIS_TLS_CLIENT_CERT,
+    crate::REDIS_TLS_CLIENT_KEY,
+    crate::REDIS_TLS_ALLOW_INSECURE,
+    crate::COMMENT_KEY,
+];

crates/config/src/constants/env.rs

@@ -34,6 +34,12 @@ pub const ENV_NOTIFY_ENABLE: &str = "RUSTFS_NOTIFY_ENABLE";
 pub const DEFAULT_AUDIT_ENABLE: bool = false;
 /// Default global notify switch (disabled by default).
 pub const DEFAULT_NOTIFY_ENABLE: bool = false;
+/// Canonical ILM process boundary env key (seconds).
+pub const ENV_ILM_PROCESS_TIME: &str = "RUSTFS_ILM_PROCESS_TIME";
+/// Deprecated ILM process boundary env key kept for compatibility.
+pub const ENV_ILM_PROCESS_TIME_DEPRECATED: &str = "_RUSTFS_ILM_PROCESS_TIME";
+/// Default ILM process boundary in seconds (24h).
+pub const DEFAULT_ILM_PROCESS_TIME_SECS: i32 = 86400;
 
 /// Medium-drawn lines separator
 /// This is used to separate words in environment variable names.

crates/config/src/constants/protocols.rs

@@ -57,3 +57,114 @@ pub const ENV_WEBDAV_CERTS_DIR: &str = "RUSTFS_WEBDAV_CERTS_DIR";
 pub const ENV_WEBDAV_CA_FILE: &str = "RUSTFS_WEBDAV_CA_FILE";
 pub const ENV_WEBDAV_MAX_BODY_SIZE: &str = "RUSTFS_WEBDAV_MAX_BODY_SIZE";
 pub const ENV_WEBDAV_REQUEST_TIMEOUT: &str = "RUSTFS_WEBDAV_REQUEST_TIMEOUT";
+
+/// Default SFTP server bind address.
+pub const DEFAULT_SFTP_ADDRESS: &str = "0.0.0.0:2222";
+
+/// Default for SFTP host-key directory. None means no default. Operators
+/// must set RUSTFS_SFTP_HOST_KEY_DIR explicitly when SFTP is enabled.
+pub const DEFAULT_SFTP_HOST_KEY_DIR: Option<&str> = None;
+
+/// SFTP environment variable names.
+pub const ENV_SFTP_ENABLE: &str = "RUSTFS_SFTP_ENABLE";
+pub const ENV_SFTP_ADDRESS: &str = "RUSTFS_SFTP_ADDRESS";
+pub const ENV_SFTP_HOST_KEY_DIR: &str = "RUSTFS_SFTP_HOST_KEY_DIR";
+pub const ENV_SFTP_HOST_KEY_RELOAD_ENABLE: &str = "RUSTFS_SFTP_HOST_KEY_RELOAD_ENABLE";
+pub const ENV_SFTP_HOST_KEY_RELOAD_INTERVAL: &str = "RUSTFS_SFTP_HOST_KEY_RELOAD_INTERVAL";
+pub const ENV_SFTP_IDLE_TIMEOUT: &str = "RUSTFS_SFTP_IDLE_TIMEOUT";
+/// S3 multipart part size in bytes. Default DEFAULT_SFTP_PART_SIZE (16 MiB).
+/// Valid range 5 MiB to 5 GiB (S3 protocol bounds), enforced at startup.
+///
+/// The per-upload size ceiling is part_size * 10_000 (the S3 parts cap),
+/// so the default caps single uploads at 160 GiB. Deployments expecting
+/// larger single files must raise this: 64 MiB -> 640 GiB, 128 MiB ->
+/// 1.25 TiB, 512 MiB -> 5 TiB (S3 object max). Rename is not affected;
+/// multipart_copy scales the per-part size dynamically and handles up
+/// to the 5 TiB S3 object limit regardless of this setting.
+pub const ENV_SFTP_PART_SIZE: &str = "RUSTFS_SFTP_PART_SIZE";
+pub const ENV_SFTP_READ_ONLY: &str = "RUSTFS_SFTP_READ_ONLY";
+pub const ENV_SFTP_BANNER: &str = "RUSTFS_SFTP_BANNER";
+/// Optional environment variable. If RUSTFS_SFTP_HANDLES_PER_SESSION
+/// is not set in the process environment, the server uses the default
+/// of 64 handles per session and emits no warning. If set, the value
+/// must be in the inclusive range 8 to 1024. Out-of-range values fall
+/// back to the default of 64 with a warn-level log naming the
+/// requested value and the bounds.
+///
+/// Caps the maximum number of simultaneously-open SFTP handles per
+/// session. A handle is the server-side identifier returned by
+/// SSH_FXP_OPEN and SSH_FXP_OPENDIR. One client typically uses one
+/// handle per file in flight plus one per directory listing.
+/// Operators running clients with deep pipelining may raise this.
+pub const ENV_SFTP_HANDLES_PER_SESSION: &str = "RUSTFS_SFTP_HANDLES_PER_SESSION";
+
+/// Optional environment variable. If RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS
+/// is not set in the process environment, the server uses the default
+/// of 60 seconds and emits no warning. If set, the value must be in
+/// the inclusive range 5 to 600 seconds. Out-of-range values fall
+/// back to the default with a warn-level log naming the requested
+/// value and the bounds.
+///
+/// Bounds every storage backend call issued by the SFTP driver. A
+/// backend that does not respond within this many seconds returns
+/// Failure to the client and emits a warn log naming the backend
+/// method. This catches a backend that accepted the request and never
+/// returned a body, which the SSH keepalive cannot detect because the
+/// transport itself remains live.
+pub const ENV_SFTP_BACKEND_OP_TIMEOUT_SECS: &str = "RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS";
+
+/// Optional environment variable. If RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES
+/// is not set in the process environment, the server uses a 4 MiB
+/// default and emits no warning. If set, the value must be in the
+/// inclusive range MAX_READ_LEN (256 KiB) to 64 MiB. Out-of-range
+/// values fall back to the default with a warn-level log naming the
+/// requested value and the bounds.
+///
+/// Per-handle byte window the SFTP read path fetches in one backend
+/// call on a cache miss. Subsequent FXP_READs within that window are
+/// served from the buffer without a backend round trip. For
+/// sequential downloads the backend round-trip count drops by
+/// window_bytes / MAX_READ_LEN. Random-access workloads should set
+/// the window equal to MAX_READ_LEN to opt out of read-ahead.
+pub const ENV_SFTP_READ_CACHE_WINDOW_BYTES: &str = "RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES";
+
+/// Optional environment variable. If
+/// RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES is not set in the process
+/// environment, the server uses a 256 MiB default and emits no
+/// warning. If set, the value must be at least 16 MiB. Below-min
+/// values fall back to the default with a warn-level log naming the
+/// requested value and the bound.
+///
+/// Process-wide ceiling on cumulative read cache memory across every
+/// live SFTP handle. Once the accumulator plus a new window would
+/// exceed this value, the populate call on the per-handle cache is
+/// skipped. The read still completes from the freshly-fetched bytes
+/// without storing them in the cache, at the cost of one backend
+/// call per FXP_READ. High-concurrency deployments expecting many
+/// parallel downloads should raise this in step with the per-session
+/// handle cap.
+pub const ENV_SFTP_READ_CACHE_TOTAL_MEM_BYTES: &str = "RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES";
+
+/// Default idle session timeout in seconds.
+pub const DEFAULT_SFTP_IDLE_TIMEOUT: u64 = 600;
+
+/// Default SFTP host key hot reload enabled state.
+pub const DEFAULT_SFTP_HOST_KEY_RELOAD_ENABLE: bool = false;
+
+/// Default SFTP host key hot reload interval in seconds.
+pub const DEFAULT_SFTP_HOST_KEY_RELOAD_INTERVAL: u64 = 30;
+
+/// Default S3 multipart upload part size in bytes (16 MiB).
+///
+/// The per-upload size ceiling is part_size * 10_000 (the S3 parts cap),
+/// so the default gives a 160 GiB single-upload limit. Deployments that
+/// expect single files larger than this must raise part_size:
+/// 64 MiB -> 640 GiB, 128 MiB -> 1.25 TiB, 512 MiB -> 5 TiB (S3 max).
+/// The minimum is 5 MiB and the maximum is 5 GiB (S3 protocol bounds).
+pub const DEFAULT_SFTP_PART_SIZE: u64 = 16_777_216;
+
+/// Default read-only mode (disabled).
+pub const DEFAULT_SFTP_READ_ONLY: bool = false;
+
+/// Default SSH identification string (no version disclosure).
+pub const DEFAULT_SFTP_BANNER: &str = "SSH-2.0-RustFS";

crates/config/src/constants/targets.rs

@@ -50,6 +50,19 @@ pub const KAFKA_TLS_CA: &str = "tls_ca";
 pub const KAFKA_TLS_CLIENT_CERT: &str = "tls_client_cert";
 pub const KAFKA_TLS_CLIENT_KEY: &str = "tls_client_key";
 
+pub const AMQP_URL: &str = "url";
+pub const AMQP_EXCHANGE: &str = "exchange";
+pub const AMQP_ROUTING_KEY: &str = "routing_key";
+pub const AMQP_MANDATORY: &str = "mandatory";
+pub const AMQP_PERSISTENT: &str = "persistent";
+pub const AMQP_USERNAME: &str = "username";
+pub const AMQP_PASSWORD: &str = "password";
+pub const AMQP_TLS_CA: &str = "tls_ca";
+pub const AMQP_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const AMQP_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const AMQP_QUEUE_DIR: &str = "queue_dir";
+pub const AMQP_QUEUE_LIMIT: &str = "queue_limit";
+
 pub const NATS_ADDRESS: &str = "address";
 pub const NATS_SUBJECT: &str = "subject";
 pub const NATS_USERNAME: &str = "username";
@@ -74,6 +87,48 @@ pub const PULSAR_TLS_HOSTNAME_VERIFICATION: &str = "tls_hostname_verification";
 pub const PULSAR_QUEUE_DIR: &str = "queue_dir";
 pub const PULSAR_QUEUE_LIMIT: &str = "queue_limit";
 
+pub const BASE_DSN_STRING: &str = "dsn_string";
+
+pub const MYSQL_DSN_STRING: &str = BASE_DSN_STRING;
+pub const MYSQL_TABLE: &str = "table";
+pub const MYSQL_FORMAT: &str = "format";
+pub const MYSQL_TLS_CA: &str = "tls_ca";
+pub const MYSQL_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const MYSQL_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const MYSQL_QUEUE_DIR: &str = "queue_dir";
+pub const MYSQL_QUEUE_LIMIT: &str = "queue_limit";
+pub const MYSQL_MAX_OPEN_CONNECTIONS: &str = "max_open_connections";
+
+pub const REDIS_URL: &str = "url";
+pub const REDIS_CHANNEL: &str = "channel";
+pub const REDIS_USERNAME: &str = "username";
+pub const REDIS_PASSWORD: &str = "password";
+pub const REDIS_KEEP_ALIVE_INTERVAL: &str = "keep_alive_interval";
+pub const REDIS_QUEUE_DIR: &str = "queue_dir";
+pub const REDIS_QUEUE_LIMIT: &str = "queue_limit";
+pub const REDIS_MAX_RETRY_ATTEMPTS: &str = "max_retry_attempts";
+pub const REDIS_RECONNECT_RETRY_ATTEMPTS: &str = "reconnect_retry_attempts";
+pub const REDIS_MIN_RETRY_DELAY: &str = "min_retry_delay";
+pub const REDIS_MAX_RETRY_DELAY: &str = "max_retry_delay";
+pub const REDIS_CONNECTION_TIMEOUT: &str = "connection_timeout";
+pub const REDIS_RESPONSE_TIMEOUT: &str = "response_timeout";
+pub const REDIS_PIPELINE_BUFFER_SIZE: &str = "pipeline_buffer_size";
+pub const REDIS_TLS_POLICY: &str = "tls_policy";
+pub const REDIS_TLS_CA: &str = "tls_ca";
+pub const REDIS_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const REDIS_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const REDIS_TLS_ALLOW_INSECURE: &str = "tls_allow_insecure";
+
+pub const POSTGRES_DSN_STRING: &str = BASE_DSN_STRING;
+pub const POSTGRES_TABLE: &str = "table";
+pub const POSTGRES_FORMAT: &str = "format";
+pub const POSTGRES_TLS_REQUIRED: &str = "tls_required";
+pub const POSTGRES_TLS_CA: &str = "tls_ca";
+pub const POSTGRES_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const POSTGRES_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const POSTGRES_QUEUE_DIR: &str = "queue_dir";
+pub const POSTGRES_QUEUE_LIMIT: &str = "queue_limit";
+
 /// Environment variable controlling whether target queue files are Snappy-compressed.
 /// Applies to both notify and audit target queue stores.
 pub const ENV_TARGET_STORE_COMPRESS: &str = "RUSTFS_TARGET_STORE_COMPRESS";

crates/config/src/notify/amqp.rs

@@ -0,0 +1,60 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const NOTIFY_AMQP_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::AMQP_URL,
+    crate::AMQP_EXCHANGE,
+    crate::AMQP_ROUTING_KEY,
+    crate::AMQP_MANDATORY,
+    crate::AMQP_PERSISTENT,
+    crate::AMQP_USERNAME,
+    crate::AMQP_PASSWORD,
+    crate::AMQP_TLS_CA,
+    crate::AMQP_TLS_CLIENT_CERT,
+    crate::AMQP_TLS_CLIENT_KEY,
+    crate::AMQP_QUEUE_DIR,
+    crate::AMQP_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_NOTIFY_AMQP_ENABLE: &str = "RUSTFS_NOTIFY_AMQP_ENABLE";
+pub const ENV_NOTIFY_AMQP_URL: &str = "RUSTFS_NOTIFY_AMQP_URL";
+pub const ENV_NOTIFY_AMQP_EXCHANGE: &str = "RUSTFS_NOTIFY_AMQP_EXCHANGE";
+pub const ENV_NOTIFY_AMQP_ROUTING_KEY: &str = "RUSTFS_NOTIFY_AMQP_ROUTING_KEY";
+pub const ENV_NOTIFY_AMQP_MANDATORY: &str = "RUSTFS_NOTIFY_AMQP_MANDATORY";
+pub const ENV_NOTIFY_AMQP_PERSISTENT: &str = "RUSTFS_NOTIFY_AMQP_PERSISTENT";
+pub const ENV_NOTIFY_AMQP_USERNAME: &str = "RUSTFS_NOTIFY_AMQP_USERNAME";
+pub const ENV_NOTIFY_AMQP_PASSWORD: &str = "RUSTFS_NOTIFY_AMQP_PASSWORD";
+pub const ENV_NOTIFY_AMQP_TLS_CA: &str = "RUSTFS_NOTIFY_AMQP_TLS_CA";
+pub const ENV_NOTIFY_AMQP_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_AMQP_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_AMQP_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_AMQP_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_AMQP_QUEUE_DIR: &str = "RUSTFS_NOTIFY_AMQP_QUEUE_DIR";
+pub const ENV_NOTIFY_AMQP_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_AMQP_QUEUE_LIMIT";
+
+pub const ENV_NOTIFY_AMQP_KEYS: &[&str; 13] = &[
+    ENV_NOTIFY_AMQP_ENABLE,
+    ENV_NOTIFY_AMQP_URL,
+    ENV_NOTIFY_AMQP_EXCHANGE,
+    ENV_NOTIFY_AMQP_ROUTING_KEY,
+    ENV_NOTIFY_AMQP_MANDATORY,
+    ENV_NOTIFY_AMQP_PERSISTENT,
+    ENV_NOTIFY_AMQP_USERNAME,
+    ENV_NOTIFY_AMQP_PASSWORD,
+    ENV_NOTIFY_AMQP_TLS_CA,
+    ENV_NOTIFY_AMQP_TLS_CLIENT_CERT,
+    ENV_NOTIFY_AMQP_TLS_CLIENT_KEY,
+    ENV_NOTIFY_AMQP_QUEUE_DIR,
+    ENV_NOTIFY_AMQP_QUEUE_LIMIT,
+];

crates/config/src/notify/mod.rs

@@ -12,19 +12,27 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+mod amqp;
 mod arn;
 mod kafka;
 mod mqtt;
+mod mysql;
 mod nats;
+mod postgres;
 mod pulsar;
+mod redis;
 mod store;
 mod webhook;
 
+pub use amqp::*;
 pub use arn::*;
 pub use kafka::*;
 pub use mqtt::*;
+pub use mysql::*;
 pub use nats::*;
+pub use postgres::*;
 pub use pulsar::*;
+pub use redis::*;
 pub use store::*;
 pub use webhook::*;
 
@@ -69,30 +77,31 @@ pub const ENV_NOTIFY_SEND_CONCURRENCY: &str = "RUSTFS_NOTIFY_SEND_CONCURRENCY";
 /// Adjust this value based on your system's capabilities and expected load.
 pub const DEFAULT_NOTIFY_SEND_CONCURRENCY: usize = 64;
 
-#[allow(dead_code)]
 pub const NOTIFY_SUB_SYSTEMS: &[&str] = &[
+    NOTIFY_AMQP_SUB_SYS,
     NOTIFY_KAFKA_SUB_SYS,
     NOTIFY_MQTT_SUB_SYS,
+    NOTIFY_MYSQL_SUB_SYS,
     NOTIFY_NATS_SUB_SYS,
+    NOTIFY_POSTGRES_SUB_SYS,
     NOTIFY_PULSAR_SUB_SYS,
+    NOTIFY_REDIS_SUB_SYS,
     NOTIFY_WEBHOOK_SUB_SYS,
 ];
 
 pub const NOTIFY_KAFKA_SUB_SYS: &str = "notify_kafka";
 pub const NOTIFY_MQTT_SUB_SYS: &str = "notify_mqtt";
-#[allow(dead_code)]
-pub const NOTIFY_MY_SQL_SUB_SYS: &str = "notify_mysql";
+pub const NOTIFY_MYSQL_SUB_SYS: &str = "notify_mysql";
 #[allow(dead_code)]
 pub const NOTIFY_NATS_SUB_SYS: &str = "notify_nats";
 #[allow(dead_code)]
 pub const NOTIFY_NSQ_SUB_SYS: &str = "notify_nsq";
 #[allow(dead_code)]
 pub const NOTIFY_ES_SUB_SYS: &str = "notify_elasticsearch";
-#[allow(dead_code)]
 pub const NOTIFY_AMQP_SUB_SYS: &str = "notify_amqp";
-#[allow(dead_code)]
 pub const NOTIFY_POSTGRES_SUB_SYS: &str = "notify_postgres";
 #[allow(dead_code)]
 pub const NOTIFY_REDIS_SUB_SYS: &str = "notify_redis";
+pub const NOTIFY_REDIS_DEFAULT_CHANNEL: &str = "rustfs_notify_channel";
 pub const NOTIFY_PULSAR_SUB_SYS: &str = "notify_pulsar";
 pub const NOTIFY_WEBHOOK_SUB_SYS: &str = "notify_webhook";

crates/config/src/notify/mysql.rs

@@ -0,0 +1,53 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// A list of all valid configuration keys for a MySQL target.
+pub const NOTIFY_MYSQL_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::MYSQL_DSN_STRING,
+    crate::MYSQL_TABLE,
+    crate::MYSQL_FORMAT,
+    crate::MYSQL_TLS_CA,
+    crate::MYSQL_TLS_CLIENT_CERT,
+    crate::MYSQL_TLS_CLIENT_KEY,
+    crate::MYSQL_QUEUE_DIR,
+    crate::MYSQL_QUEUE_LIMIT,
+    crate::MYSQL_MAX_OPEN_CONNECTIONS,
+    crate::COMMENT_KEY,
+];
+
+// MySQL Environment Variables
+pub const ENV_NOTIFY_MYSQL_ENABLE: &str = "RUSTFS_NOTIFY_MYSQL_ENABLE";
+pub const ENV_NOTIFY_MYSQL_DSN_STRING: &str = "RUSTFS_NOTIFY_MYSQL_DSN_STRING";
+pub const ENV_NOTIFY_MYSQL_TABLE: &str = "RUSTFS_NOTIFY_MYSQL_TABLE";
+pub const ENV_NOTIFY_MYSQL_FORMAT: &str = "RUSTFS_NOTIFY_MYSQL_FORMAT";
+pub const ENV_NOTIFY_MYSQL_TLS_CA: &str = "RUSTFS_NOTIFY_MYSQL_TLS_CA";
+pub const ENV_NOTIFY_MYSQL_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_MYSQL_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_MYSQL_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_MYSQL_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_MYSQL_QUEUE_DIR: &str = "RUSTFS_NOTIFY_MYSQL_QUEUE_DIR";
+pub const ENV_NOTIFY_MYSQL_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_MYSQL_QUEUE_LIMIT";
+pub const ENV_NOTIFY_MYSQL_MAX_OPEN_CONNECTIONS: &str = "RUSTFS_NOTIFY_MYSQL_MAX_OPEN_CONNECTIONS";
+
+pub const ENV_NOTIFY_MYSQL_KEYS: &[&str; 10] = &[
+    ENV_NOTIFY_MYSQL_ENABLE,
+    ENV_NOTIFY_MYSQL_DSN_STRING,
+    ENV_NOTIFY_MYSQL_TABLE,
+    ENV_NOTIFY_MYSQL_FORMAT,
+    ENV_NOTIFY_MYSQL_TLS_CA,
+    ENV_NOTIFY_MYSQL_TLS_CLIENT_CERT,
+    ENV_NOTIFY_MYSQL_TLS_CLIENT_KEY,
+    ENV_NOTIFY_MYSQL_QUEUE_DIR,
+    ENV_NOTIFY_MYSQL_QUEUE_LIMIT,
+    ENV_NOTIFY_MYSQL_MAX_OPEN_CONNECTIONS,
+];

crates/config/src/notify/postgres.rs

@@ -0,0 +1,51 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const NOTIFY_POSTGRES_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::POSTGRES_DSN_STRING,
+    crate::POSTGRES_TABLE,
+    crate::POSTGRES_FORMAT,
+    crate::POSTGRES_TLS_REQUIRED,
+    crate::POSTGRES_TLS_CA,
+    crate::POSTGRES_TLS_CLIENT_CERT,
+    crate::POSTGRES_TLS_CLIENT_KEY,
+    crate::POSTGRES_QUEUE_DIR,
+    crate::POSTGRES_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_NOTIFY_POSTGRES_ENABLE: &str = "RUSTFS_NOTIFY_POSTGRES_ENABLE";
+pub const ENV_NOTIFY_POSTGRES_DSN_STRING: &str = "RUSTFS_NOTIFY_POSTGRES_DSN_STRING";
+pub const ENV_NOTIFY_POSTGRES_TABLE: &str = "RUSTFS_NOTIFY_POSTGRES_TABLE";
+pub const ENV_NOTIFY_POSTGRES_FORMAT: &str = "RUSTFS_NOTIFY_POSTGRES_FORMAT";
+pub const ENV_NOTIFY_POSTGRES_TLS_REQUIRED: &str = "RUSTFS_NOTIFY_POSTGRES_TLS_REQUIRED";
+pub const ENV_NOTIFY_POSTGRES_TLS_CA: &str = "RUSTFS_NOTIFY_POSTGRES_TLS_CA";
+pub const ENV_NOTIFY_POSTGRES_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_POSTGRES_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_POSTGRES_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_POSTGRES_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_POSTGRES_QUEUE_DIR: &str = "RUSTFS_NOTIFY_POSTGRES_QUEUE_DIR";
+pub const ENV_NOTIFY_POSTGRES_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_POSTGRES_QUEUE_LIMIT";
+
+pub const ENV_NOTIFY_POSTGRES_KEYS: &[&str; 10] = &[
+    ENV_NOTIFY_POSTGRES_ENABLE,
+    ENV_NOTIFY_POSTGRES_DSN_STRING,
+    ENV_NOTIFY_POSTGRES_TABLE,
+    ENV_NOTIFY_POSTGRES_FORMAT,
+    ENV_NOTIFY_POSTGRES_TLS_REQUIRED,
+    ENV_NOTIFY_POSTGRES_TLS_CA,
+    ENV_NOTIFY_POSTGRES_TLS_CLIENT_CERT,
+    ENV_NOTIFY_POSTGRES_TLS_CLIENT_KEY,
+    ENV_NOTIFY_POSTGRES_QUEUE_DIR,
+    ENV_NOTIFY_POSTGRES_QUEUE_LIMIT,
+];

crates/config/src/notify/redis.rs

@@ -0,0 +1,82 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// A list of all valid configuration keys for a Redis target.
+pub const NOTIFY_REDIS_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::REDIS_URL,
+    crate::REDIS_CHANNEL,
+    crate::REDIS_USERNAME,
+    crate::REDIS_PASSWORD,
+    crate::REDIS_KEEP_ALIVE_INTERVAL,
+    crate::REDIS_QUEUE_DIR,
+    crate::REDIS_QUEUE_LIMIT,
+    crate::REDIS_MAX_RETRY_ATTEMPTS,
+    crate::REDIS_RECONNECT_RETRY_ATTEMPTS,
+    crate::REDIS_MIN_RETRY_DELAY,
+    crate::REDIS_MAX_RETRY_DELAY,
+    crate::REDIS_CONNECTION_TIMEOUT,
+    crate::REDIS_RESPONSE_TIMEOUT,
+    crate::REDIS_PIPELINE_BUFFER_SIZE,
+    crate::REDIS_TLS_POLICY,
+    crate::REDIS_TLS_CA,
+    crate::REDIS_TLS_CLIENT_CERT,
+    crate::REDIS_TLS_CLIENT_KEY,
+    crate::REDIS_TLS_ALLOW_INSECURE,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_NOTIFY_REDIS_ENABLE: &str = "RUSTFS_NOTIFY_REDIS_ENABLE";
+pub const ENV_NOTIFY_REDIS_URL: &str = "RUSTFS_NOTIFY_REDIS_URL";
+pub const ENV_NOTIFY_REDIS_CHANNEL: &str = "RUSTFS_NOTIFY_REDIS_CHANNEL";
+pub const ENV_NOTIFY_REDIS_USERNAME: &str = "RUSTFS_NOTIFY_REDIS_USERNAME";
+pub const ENV_NOTIFY_REDIS_PASSWORD: &str = "RUSTFS_NOTIFY_REDIS_PASSWORD";
+pub const ENV_NOTIFY_REDIS_KEEP_ALIVE_INTERVAL: &str = "RUSTFS_NOTIFY_REDIS_KEEP_ALIVE_INTERVAL";
+pub const ENV_NOTIFY_REDIS_QUEUE_DIR: &str = "RUSTFS_NOTIFY_REDIS_QUEUE_DIR";
+pub const ENV_NOTIFY_REDIS_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_REDIS_QUEUE_LIMIT";
+pub const ENV_NOTIFY_REDIS_MAX_RETRY_ATTEMPTS: &str = "RUSTFS_NOTIFY_REDIS_MAX_RETRY_ATTEMPTS";
+pub const ENV_NOTIFY_REDIS_RECONNECT_RETRY_ATTEMPTS: &str = "RUSTFS_NOTIFY_REDIS_RECONNECT_RETRY_ATTEMPTS";
+pub const ENV_NOTIFY_REDIS_MIN_RETRY_DELAY: &str = "RUSTFS_NOTIFY_REDIS_MIN_RETRY_DELAY";
+pub const ENV_NOTIFY_REDIS_MAX_RETRY_DELAY: &str = "RUSTFS_NOTIFY_REDIS_MAX_RETRY_DELAY";
+pub const ENV_NOTIFY_REDIS_CONNECTION_TIMEOUT: &str = "RUSTFS_NOTIFY_REDIS_CONNECTION_TIMEOUT";
+pub const ENV_NOTIFY_REDIS_RESPONSE_TIMEOUT: &str = "RUSTFS_NOTIFY_REDIS_RESPONSE_TIMEOUT";
+pub const ENV_NOTIFY_REDIS_PIPELINE_BUFFER_SIZE: &str = "RUSTFS_NOTIFY_REDIS_PIPELINE_BUFFER_SIZE";
+pub const ENV_NOTIFY_REDIS_TLS_POLICY: &str = "RUSTFS_NOTIFY_REDIS_TLS_POLICY";
+pub const ENV_NOTIFY_REDIS_TLS_CA: &str = "RUSTFS_NOTIFY_REDIS_TLS_CA";
+pub const ENV_NOTIFY_REDIS_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_REDIS_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_REDIS_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_REDIS_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_REDIS_TLS_ALLOW_INSECURE: &str = "RUSTFS_NOTIFY_REDIS_TLS_ALLOW_INSECURE";
+
+pub const ENV_NOTIFY_REDIS_KEYS: &[&str; 20] = &[
+    ENV_NOTIFY_REDIS_ENABLE,
+    ENV_NOTIFY_REDIS_URL,
+    ENV_NOTIFY_REDIS_CHANNEL,
+    ENV_NOTIFY_REDIS_USERNAME,
+    ENV_NOTIFY_REDIS_PASSWORD,
+    ENV_NOTIFY_REDIS_KEEP_ALIVE_INTERVAL,
+    ENV_NOTIFY_REDIS_QUEUE_DIR,
+    ENV_NOTIFY_REDIS_QUEUE_LIMIT,
+    ENV_NOTIFY_REDIS_MAX_RETRY_ATTEMPTS,
+    ENV_NOTIFY_REDIS_RECONNECT_RETRY_ATTEMPTS,
+    ENV_NOTIFY_REDIS_MIN_RETRY_DELAY,
+    ENV_NOTIFY_REDIS_MAX_RETRY_DELAY,
+    ENV_NOTIFY_REDIS_CONNECTION_TIMEOUT,
+    ENV_NOTIFY_REDIS_RESPONSE_TIMEOUT,
+    ENV_NOTIFY_REDIS_PIPELINE_BUFFER_SIZE,
+    ENV_NOTIFY_REDIS_TLS_POLICY,
+    ENV_NOTIFY_REDIS_TLS_CA,
+    ENV_NOTIFY_REDIS_TLS_CLIENT_CERT,
+    ENV_NOTIFY_REDIS_TLS_CLIENT_KEY,
+    ENV_NOTIFY_REDIS_TLS_ALLOW_INSECURE,
+];

crates/crypto/Cargo.toml

@@ -31,7 +31,6 @@ workspace = true
 [dependencies]
 aes-gcm = { workspace = true, optional = true }
 argon2 = { workspace = true, optional = true }
-cfg-if = { workspace = true }
 chacha20poly1305 = { workspace = true, optional = true }
 jsonwebtoken = { workspace = true }
 pbkdf2 = { workspace = true, optional = true }

crates/crypto/src/encdec/aes.rs

@@ -13,19 +13,23 @@
 // limitations under the License.
 
 pub fn native_aes() -> bool {
-    cfg_if::cfg_if! {
-        if #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] {
+    cfg_select! {
+        any(target_arch = "x86", target_arch = "x86_64") => {
             std::is_x86_feature_detected!("aes") && std::is_x86_feature_detected!("pclmulqdq")
-        } else if #[cfg(target_arch = "aarch64")] {
+        }
+        target_arch = "aarch64" => {
             std::arch::is_aarch64_feature_detected!("aes")
-        } else if #[cfg(target_arch = "powerpc64")] {
+        }
+        target_arch = "powerpc64" => {
             false
-        } else if #[cfg(target_arch = "s390x")] {
+        }
+        target_arch = "s390x" => {
             std::is_s390x_feature_detected!("aes")
                 && std::is_s390x_feature_detected!("aescbc")
                 && std::is_s390x_feature_detected!("aesctr")
                 && (std::is_s390x_feature_detected!("aesgcm") || std::is_s390x_feature_detected!("ghash"))
-        } else {
+        }
+        _ => {
             false
         }
     }

crates/e2e_test/Cargo.toml

@@ -26,8 +26,10 @@ workspace = true
 [features]
 default = []
 ftps = []
+sftp = []
 
 [dependencies]
+rustfs-config = { workspace = true, features = ["constants"] }
 rustfs-ecstore.workspace = true
 rustfs-common.workspace = true
 rustfs-rio.workspace = true
@@ -72,5 +74,7 @@ suppaftp = { workspace = true, features = ["tokio", "rustls-aws-lc-rs"] }
 rcgen.workspace = true
 anyhow.workspace = true
 rustls.workspace = true
+russh = { workspace = true }
+russh-sftp = { workspace = true }
 zip.workspace = true
 clap.workspace = true

crates/e2e_test/src/common.rs

@@ -41,7 +41,9 @@ use walkdir::WalkDir;
 // Common constants for all E2E tests
 pub const DEFAULT_ACCESS_KEY: &str = "rustfsadmin";
 pub const DEFAULT_SECRET_KEY: &str = "rustfsadmin";
+pub const ENV_RUSTFS_BUILD_FEATURES: &str = "RUSTFS_BUILD_FEATURES";
 pub const TEST_BUCKET: &str = "e2e-test-bucket";
+const RUSTFS_FULL_FEATURE: &str = "full";
 
 fn build_test_s3_config(endpoint_url: &str, access_key: &str, secret_key: &str, provider_name: &'static str) -> Config {
     let credentials = Credentials::new(access_key, secret_key, None, None, provider_name);
@@ -83,6 +85,7 @@ pub fn rustfs_binary_path_with_features(requested_features: Option<&str>) -> Pat
     if let Some(path) = std::env::var_os("CARGO_BIN_EXE_rustfs") {
         return PathBuf::from(path);
     }
+    let requested_features = requested_features.and_then(normalize_rustfs_build_features);
 
     let mut binary_path = workspace_root();
     binary_path.push("target");
@@ -90,7 +93,7 @@ pub fn rustfs_binary_path_with_features(requested_features: Option<&str>) -> Pat
     binary_path.push(profile_dir);
     binary_path.push(format!("rustfs{}", std::env::consts::EXE_SUFFIX));
 
-    let features_match = binary_features_match(&binary_path, requested_features);
+    let features_match = binary_features_match(&binary_path, requested_features.as_deref());
     let source_is_newer = workspace_sources_newer_than_binary(&binary_path);
     let can_reuse_inside_e2e = running_inside_e2e_test_binary() && requested_features.is_none() && features_match;
     if binary_path.is_file() && features_match && (!source_is_newer || can_reuse_inside_e2e) {
@@ -105,7 +108,7 @@ pub fn rustfs_binary_path_with_features(requested_features: Option<&str>) -> Pat
     }
 
     info!("Building RustFS binary to ensure it's up to date...");
-    build_rustfs_binary(requested_features);
+    build_rustfs_binary(requested_features.as_deref());
 
     info!("Using RustFS binary at {:?}", binary_path);
     binary_path
@@ -134,11 +137,31 @@ fn running_inside_e2e_test_binary() -> bool {
     std::env::var("CARGO_PKG_NAME").is_ok_and(|value| value == "e2e_test")
 }
 
-fn requested_rustfs_build_features() -> Option<String> {
-    std::env::var("RUSTFS_BUILD_FEATURES")
+pub fn requested_rustfs_build_features() -> Option<String> {
+    std::env::var(ENV_RUSTFS_BUILD_FEATURES)
         .ok()
-        .map(|value| value.trim().to_string())
-        .filter(|value| !value.is_empty())
+        .and_then(|value| normalize_rustfs_build_features(&value))
+}
+
+pub fn normalize_rustfs_build_features(features: &str) -> Option<String> {
+    let features = features
+        .split(',')
+        .map(str::trim)
+        .filter(|feature| !feature.is_empty())
+        .map(str::to_ascii_lowercase)
+        .collect::<Vec<_>>();
+
+    if features.is_empty() { None } else { Some(features.join(",")) }
+}
+
+pub fn rustfs_build_feature_enabled(requested_features: Option<&str>, required_feature: &str) -> bool {
+    let Some(requested_features) = requested_features.and_then(normalize_rustfs_build_features) else {
+        return true;
+    };
+
+    requested_features
+        .split(',')
+        .any(|feature| feature.eq_ignore_ascii_case(RUSTFS_FULL_FEATURE) || feature.eq_ignore_ascii_case(required_feature))
 }
 
 fn rustfs_binary_features_stamp_path(binary_path: &Path) -> PathBuf {
@@ -147,11 +170,14 @@ fn rustfs_binary_features_stamp_path(binary_path: &Path) -> PathBuf {
 
 fn binary_features_match(binary_path: &Path, requested_features: Option<&str>) -> bool {
     let stamp_path = rustfs_binary_features_stamp_path(binary_path);
-    let recorded = stdfs::read_to_string(stamp_path).ok().map(|value| value.trim().to_string());
+    let recorded = stdfs::read_to_string(stamp_path)
+        .ok()
+        .and_then(|value| normalize_rustfs_build_features(&value));
+    let requested = requested_features.and_then(normalize_rustfs_build_features);
 
-    match requested_features {
+    match requested.as_deref() {
         Some(features) => recorded.as_deref() == Some(features),
-        None => recorded.as_deref().is_none_or(str::is_empty),
+        None => recorded.is_none(),
     }
 }
 
@@ -351,6 +377,7 @@ impl RustFSTestEnvironment {
     async fn start_rustfs_server_inner(
         &mut self,
         extra_args: Vec<&str>,
+        extra_env: &[(&str, &str)],
         cleanup_existing: bool,
     ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         if cleanup_existing {
@@ -362,10 +389,12 @@ impl RustFSTestEnvironment {
         info!("Starting RustFS server with args: {:?}", args);
 
         let binary_path = rustfs_binary_path();
-        let process = Command::new(&binary_path)
-            .env("RUST_LOG", "rustfs=info,rustfs_notify=debug")
-            .args(&args)
-            .spawn()?;
+        let mut command = Command::new(&binary_path);
+        command.env("RUST_LOG", "rustfs=info,rustfs_notify=debug");
+        for (key, value) in extra_env {
+            command.env(key, value);
+        }
+        let process = command.args(&args).spawn()?;
 
         self.process = Some(process);
 
@@ -377,7 +406,16 @@ impl RustFSTestEnvironment {
 
     /// Start RustFS server with basic configuration
     pub async fn start_rustfs_server(&mut self, extra_args: Vec<&str>) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        self.start_rustfs_server_inner(extra_args, true).await
+        self.start_rustfs_server_inner(extra_args, &[], true).await
+    }
+
+    /// Start RustFS server with extra child-process environment variables.
+    pub async fn start_rustfs_server_with_env(
+        &mut self,
+        extra_args: Vec<&str>,
+        extra_env: &[(&str, &str)],
+    ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+        self.start_rustfs_server_inner(extra_args, extra_env, true).await
     }
 
     /// Start RustFS server without cleaning up other running RustFS processes.
@@ -388,7 +426,7 @@ impl RustFSTestEnvironment {
         &mut self,
         extra_args: Vec<&str>,
     ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        self.start_rustfs_server_inner(extra_args, false).await
+        self.start_rustfs_server_inner(extra_args, &[], false).await
     }
 
     /// Wait for RustFS server to be ready.
@@ -888,3 +926,36 @@ impl Drop for RustFSTestClusterEnvironment {
         }
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn normalizes_rustfs_build_features() {
+        assert_eq!(
+            normalize_rustfs_build_features(" SFTP, ftps ,, WebDAV "),
+            Some("sftp,ftps,webdav".to_string())
+        );
+        assert_eq!(normalize_rustfs_build_features(" , "), None);
+    }
+
+    #[test]
+    fn full_feature_enables_any_required_feature() {
+        assert!(rustfs_build_feature_enabled(Some("full"), "sftp"));
+        assert!(rustfs_build_feature_enabled(Some("ftps, full"), "webdav"));
+    }
+
+    #[test]
+    fn binary_feature_stamp_matching_uses_normalized_features() {
+        let binary_path = std::env::temp_dir().join(format!("rustfs-feature-stamp-test-{}", Uuid::new_v4()));
+        let stamp_path = rustfs_binary_features_stamp_path(&binary_path);
+
+        stdfs::write(&stamp_path, " SFTP, ftps ").expect("write feature stamp");
+        assert!(binary_features_match(&binary_path, Some("sftp,ftps")));
+        assert!(binary_features_match(&binary_path, Some(" SFTP, FTPS ")));
+        assert!(!binary_features_match(&binary_path, Some("sftp")));
+
+        stdfs::remove_file(stamp_path).ok();
+    }
+}

crates/e2e_test/src/delete_object_no_content_length_test.rs

@@ -0,0 +1,165 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Regression coverage for signed `DELETE Object?versionId` requests sent with
+//! no body and no `Content-Length` header.
+
+#[cfg(test)]
+mod tests {
+    use crate::common::{RustFSTestEnvironment, init_logging};
+    use aws_sdk_s3::primitives::ByteStream;
+    use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
+    use http::header::HOST;
+    use rustfs_signer::constants::UNSIGNED_PAYLOAD;
+    use rustfs_signer::sign_v4;
+    use s3s::Body;
+    use serial_test::serial;
+    use std::error::Error;
+    use tokio::io::{AsyncReadExt, AsyncWriteExt};
+    use tokio::net::TcpStream;
+    use tokio::time::{Duration, timeout};
+    use tracing::info;
+
+    const RAW_RESPONSE_TIMEOUT: Duration = Duration::from_secs(10);
+
+    fn parse_status(raw_response: &str) -> Option<u16> {
+        raw_response.lines().next()?.split_whitespace().nth(1)?.parse().ok()
+    }
+
+    fn response_body(raw_response: &str) -> &str {
+        raw_response
+            .split_once("\r\n\r\n")
+            .map(|(_, body)| body)
+            .or_else(|| raw_response.split_once("\n\n").map(|(_, body)| body))
+            .unwrap_or("")
+    }
+
+    async fn signed_delete_without_content_length(
+        url: &str,
+        access_key: &str,
+        secret_key: &str,
+    ) -> Result<String, Box<dyn Error + Send + Sync>> {
+        let uri = url.parse::<http::Uri>()?;
+        let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
+        let path_and_query = uri.path_and_query().ok_or("request URL missing path")?.as_str().to_string();
+
+        let request = http::Request::builder()
+            .method(http::Method::DELETE)
+            .uri(uri)
+            .header(HOST, authority.clone())
+            .header("x-amz-content-sha256", UNSIGNED_PAYLOAD)
+            .body(Body::empty())?;
+
+        let signed = sign_v4(request, 0, access_key, secret_key, "", "us-east-1");
+
+        let mut raw_request = format!("DELETE {path_and_query} HTTP/1.1\r\nHost: {authority}\r\nConnection: close\r\n");
+        for (name, value) in signed.headers() {
+            if name.as_str().eq_ignore_ascii_case("host") || name.as_str().eq_ignore_ascii_case("content-length") {
+                continue;
+            }
+            raw_request.push_str(name.as_str());
+            raw_request.push_str(": ");
+            raw_request.push_str(value.to_str()?);
+            raw_request.push_str("\r\n");
+        }
+        raw_request.push_str("\r\n");
+
+        assert!(
+            !raw_request.to_ascii_lowercase().contains("\r\ncontent-length:"),
+            "raw regression request must omit Content-Length; request was:\n{raw_request}"
+        );
+
+        let mut stream = TcpStream::connect(&authority).await?;
+        stream.write_all(raw_request.as_bytes()).await?;
+        stream.flush().await?;
+
+        let mut response = Vec::new();
+        timeout(RAW_RESPONSE_TIMEOUT, stream.read_to_end(&mut response))
+            .await
+            .map_err(|_| std::io::Error::new(std::io::ErrorKind::TimedOut, "timed out reading raw DELETE response"))??;
+        Ok(String::from_utf8_lossy(&response).into_owned())
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn test_delete_object_version_without_content_length_succeeds() -> Result<(), Box<dyn Error + Send + Sync>> {
+        init_logging();
+        info!("🧪 TEST: signed DELETE Object?versionId succeeds without Content-Length");
+
+        let mut env = RustFSTestEnvironment::new().await?;
+        env.start_rustfs_server(vec![]).await?;
+
+        let client = env.create_s3_client();
+        let bucket = "delete-version-no-content-length";
+        let key = "versioned-delete-target.txt";
+
+        client.create_bucket().bucket(bucket).send().await?;
+
+        client
+            .put_object()
+            .bucket(bucket)
+            .key(key)
+            .body(ByteStream::from_static(b"delete me after versioning is enabled"))
+            .send()
+            .await?;
+
+        client
+            .put_bucket_versioning()
+            .bucket(bucket)
+            .versioning_configuration(
+                VersioningConfiguration::builder()
+                    .status(BucketVersioningStatus::Enabled)
+                    .build(),
+            )
+            .send()
+            .await?;
+
+        let listed_versions = client.list_object_versions().bucket(bucket).prefix(key).send().await?;
+        let version_id = listed_versions
+            .versions()
+            .iter()
+            .find(|version| version.key() == Some(key))
+            .and_then(|version| version.version_id())
+            .ok_or("ListObjectVersions did not return the pre-versioning object version")?;
+
+        let url = format!("{}/{}/{}?versionId={}", env.url, bucket, key, urlencoding::encode(version_id));
+        let raw_response = signed_delete_without_content_length(&url, &env.access_key, &env.secret_key).await?;
+        info!("raw DELETE response:\n{}", raw_response);
+
+        assert_eq!(
+            parse_status(&raw_response),
+            Some(204),
+            "DELETE Object?versionId without Content-Length should succeed, got:\n{raw_response}"
+        );
+        assert!(
+            !raw_response.contains("MissingContentLength"),
+            "DELETE Object?versionId without Content-Length regressed to MissingContentLength: {raw_response}"
+        );
+        assert!(
+            response_body(&raw_response).trim().is_empty(),
+            "successful DELETE Object?versionId should not return an error body: {raw_response}"
+        );
+
+        let get_deleted_version = client
+            .get_object()
+            .bucket(bucket)
+            .key(key)
+            .version_id(version_id)
+            .send()
+            .await;
+        assert!(get_deleted_version.is_err(), "explicitly deleted version should no longer be readable");
+
+        Ok(())
+    }
+}

crates/e2e_test/src/head_object_consistency_test.rs

@@ -0,0 +1,131 @@
+// Copyright 2026 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
+use aws_sdk_s3::presigning::PresigningConfig;
+use aws_sdk_s3::primitives::ByteStream;
+use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
+use serial_test::serial;
+use std::time::Duration;
+use tracing::info;
+
+const CONSISTENCY_BUCKET: &str = "head-consistency-bucket";
+const PUT_KEY: &str = "consistency-put-object.txt";
+const MPU_KEY: &str = "consistency-multipart-object.txt";
+
+fn list_contains_key(output: &aws_sdk_s3::operation::list_objects_v2::ListObjectsV2Output, key: &str) -> bool {
+    output.contents().iter().any(|obj| obj.key().is_some_and(|k| k == key))
+}
+
+#[tokio::test]
+#[serial]
+async fn head_object_consistency_after_write_and_multipart_and_presigned_head()
+-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    init_logging();
+    info!("Starting HeadObject consistency regression test");
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(Vec::new()).await?;
+
+    let client = env.create_s3_client();
+    env.create_test_bucket(CONSISTENCY_BUCKET).await?;
+
+    client
+        .put_object()
+        .bucket(CONSISTENCY_BUCKET)
+        .key(PUT_KEY)
+        .body(ByteStream::from_static(b"head-consistency-put-body"))
+        .send()
+        .await?;
+
+    client.get_object().bucket(CONSISTENCY_BUCKET).key(PUT_KEY).send().await?;
+    let put_list = client
+        .list_objects_v2()
+        .bucket(CONSISTENCY_BUCKET)
+        .prefix(PUT_KEY)
+        .send()
+        .await?;
+    assert!(list_contains_key(&put_list, PUT_KEY), "ListObjectsV2 should include the PutObject key");
+
+    client.head_object().bucket(CONSISTENCY_BUCKET).key(PUT_KEY).send().await?;
+
+    let create = client
+        .create_multipart_upload()
+        .bucket(CONSISTENCY_BUCKET)
+        .key(MPU_KEY)
+        .send()
+        .await?;
+    let upload_id = create.upload_id().ok_or("missing multipart upload id")?.to_string();
+
+    let part1 = client
+        .upload_part()
+        .bucket(CONSISTENCY_BUCKET)
+        .key(MPU_KEY)
+        .upload_id(&upload_id)
+        .part_number(1)
+        .body(ByteStream::from_static(b"head-consistency-multipart-part-1"))
+        .send()
+        .await?;
+    let completed = CompletedMultipartUpload::builder()
+        .parts(
+            CompletedPart::builder()
+                .part_number(1)
+                .set_e_tag(part1.e_tag().map(str::to_string))
+                .build(),
+        )
+        .build();
+
+    client
+        .complete_multipart_upload()
+        .bucket(CONSISTENCY_BUCKET)
+        .key(MPU_KEY)
+        .upload_id(&upload_id)
+        .multipart_upload(completed)
+        .send()
+        .await?;
+
+    client.get_object().bucket(CONSISTENCY_BUCKET).key(MPU_KEY).send().await?;
+    let mpu_list = client
+        .list_objects_v2()
+        .bucket(CONSISTENCY_BUCKET)
+        .prefix(MPU_KEY)
+        .send()
+        .await?;
+    assert!(
+        list_contains_key(&mpu_list, MPU_KEY),
+        "ListObjectsV2 should include the completed multipart key"
+    );
+
+    client.head_object().bucket(CONSISTENCY_BUCKET).key(MPU_KEY).send().await?;
+
+    let presigned = client
+        .head_object()
+        .bucket(CONSISTENCY_BUCKET)
+        .key(PUT_KEY)
+        .presigned(PresigningConfig::expires_in(Duration::from_secs(300))?)
+        .await?;
+    let presigned_resp = local_http_client().head(presigned.uri().to_string()).send().await?;
+    assert!(
+        presigned_resp.status().is_success(),
+        "Presigned HEAD should succeed, got status {}",
+        presigned_resp.status()
+    );
+
+    client.delete_object().bucket(CONSISTENCY_BUCKET).key(PUT_KEY).send().await?;
+    client.delete_object().bucket(CONSISTENCY_BUCKET).key(MPU_KEY).send().await?;
+    env.delete_test_bucket(CONSISTENCY_BUCKET).await?;
+    env.stop_server();
+
+    Ok(())
+}

crates/e2e_test/src/lib.rs

@@ -74,6 +74,10 @@ mod compression_test;
 #[cfg(test)]
 mod delete_objects_versioning_test;
 
+// Regression test for signed DELETE Object?versionId requests without Content-Length.
+#[cfg(test)]
+mod delete_object_no_content_length_test;
+
 // Regression test for Issue #2252: ListObjectVersions misses newest version after put -> delete -> put
 #[cfg(test)]
 mod list_object_versions_regression_test;
@@ -107,6 +111,9 @@ mod group_delete_test;
 #[cfg(test)]
 mod head_object_range_test;
 
+#[cfg(test)]
+mod head_object_consistency_test;
+
 #[cfg(test)]
 mod copy_object_metadata_test;
 

crates/e2e_test/src/list_objects_v2_pagination_test.rs

@@ -31,6 +31,7 @@ mod tests {
     use aws_sdk_s3::Client;
     use aws_sdk_s3::primitives::ByteStream;
     use serial_test::serial;
+    use std::collections::HashSet;
     use tracing::info;
 
     /// Helper function to create an S3 client for testing
@@ -57,6 +58,130 @@ mod tests {
         }
     }
 
+    /// Test for Issue #2775: continuation forwarding must not
+    /// skip a child directory when the prefix component repeats in the key.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_repeated_prefix_continuation() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 repeated-prefix continuation");
+
+        const PAGE_SIZE: i32 = 2;
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
+            .await
+            .expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-repeated-prefix-small";
+        let prefix = "engineering/";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        let expected_keys = vec![
+            format!("{prefix}alpha-000/artifact.txt"),
+            format!("{prefix}engineering/engineering/project-000/artifact.txt"),
+            format!("{prefix}engineering/engineering/project-001/artifact.txt"),
+            format!("{prefix}engineering/project-000/artifact.txt"),
+            format!("{prefix}engineering/project-001/artifact.txt"),
+            format!("{prefix}engineering/project-002/artifact.txt"),
+            format!("{prefix}zulu-000/artifact.txt"),
+        ];
+        let noise_keys = [
+            "different/prefix/prefix/project-000/artifact.txt",
+            "engineering-other/project-000/artifact.txt",
+            "unrelated/engineering/project-000/artifact.txt",
+        ];
+
+        for key in &expected_keys {
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(key)
+                .body(ByteStream::from_static(b"x"))
+                .send()
+                .await
+                .expect("Failed to put object");
+        }
+        for key in noise_keys {
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(key)
+                .body(ByteStream::from_static(b"x"))
+                .send()
+                .await
+                .expect("Failed to put noise object");
+        }
+
+        let mut listed_keys = Vec::new();
+        let mut continuation_token: Option<String> = None;
+        let mut last_key: Option<String> = None;
+        let mut page_count = 0;
+
+        loop {
+            let mut request = client.list_objects_v2().bucket(bucket).prefix(prefix).max_keys(PAGE_SIZE);
+
+            if let Some(token) = continuation_token.take() {
+                request = request.continuation_token(token);
+            }
+
+            let output = request.send().await.expect("Failed to list objects");
+
+            for obj in output.contents() {
+                if let Some(key) = obj.key() {
+                    if let Some(previous) = &last_key {
+                        assert!(
+                            key > previous.as_str(),
+                            "ListObjectsV2 did not preserve lexicographic order: {key} <= {previous}"
+                        );
+                    }
+
+                    last_key = Some(key.to_string());
+                    listed_keys.push(key.to_string());
+                }
+            }
+
+            page_count += 1;
+
+            if output.is_truncated().unwrap_or(false) {
+                continuation_token = output.next_continuation_token().map(|s| s.to_string());
+                assert!(
+                    continuation_token.is_some(),
+                    "BUG: NextContinuationToken must be present when IsTruncated is true"
+                );
+            } else {
+                break;
+            }
+
+            if page_count > 10 {
+                panic!("Too many pages, possible infinite loop due to pagination bug");
+            }
+        }
+
+        let seen: HashSet<String> = listed_keys.iter().cloned().collect();
+
+        assert_eq!(
+            listed_keys, expected_keys,
+            "Issue #2775 regression: repeated-prefix pagination must return exactly the expected keys in lexicographic order"
+        );
+        assert_eq!(
+            listed_keys.len(),
+            expected_keys.len(),
+            "Issue #2775 regression: expected all {} repeated-prefix objects under {prefix}, got {}",
+            expected_keys.len(),
+            listed_keys.len()
+        );
+        assert_eq!(seen.len(), expected_keys.len(), "Listed keys must be unique");
+
+        for key in &expected_keys {
+            assert!(seen.contains(key), "Missing expected key after repeated-prefix pagination: {key}");
+        }
+
+        env.stop_server();
+    }
+
     /// Test that IsTruncated is false when all objects fit within max_keys
     ///
     /// This is the core bug from issue #1596: the server was returning
@@ -368,6 +493,71 @@ mod tests {
         env.stop_server();
     }
 
+    /// Test ListObjectsV2 caps max_keys above the service limit and still paginates.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_max_keys_above_limit_returns_token() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 with max_keys above limit");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-max-keys-above-limit";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        let object_count = 1002;
+        for i in 0..object_count {
+            let key = format!("object{:04}.txt", i);
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(&key)
+                .body(ByteStream::from_static(b"test content"))
+                .send()
+                .await
+                .expect("Failed to put object");
+        }
+
+        let output = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .max_keys(1001)
+            .send()
+            .await
+            .expect("Failed to list objects");
+
+        assert_eq!(output.contents().len(), 1000);
+        assert_eq!(output.key_count(), Some(1000));
+        assert_eq!(output.max_keys(), Some(1000));
+        assert!(
+            output.is_truncated().unwrap_or(false),
+            "IsTruncated should be true when more objects remain after capped max_keys"
+        );
+
+        let next_token = output
+            .next_continuation_token()
+            .expect("NextContinuationToken should be present when capped response is truncated")
+            .to_string();
+
+        let output = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .max_keys(1001)
+            .continuation_token(next_token)
+            .send()
+            .await
+            .expect("Failed to list objects with continuation token");
+
+        assert_eq!(output.contents().len(), 2);
+        assert!(!output.is_truncated().unwrap_or(false));
+        assert!(output.next_continuation_token().is_none());
+
+        env.stop_server();
+    }
+
     /// Test ListObjectsV2 with max_keys=0
     ///
     /// S3 semantics: when max_keys is 0, the response should include no objects

crates/e2e_test/src/object_lambda_test.rs

@@ -25,7 +25,7 @@ use std::error::Error;
 use time::OffsetDateTime;
 use tokio::io::{AsyncReadExt, AsyncWriteExt};
 use tokio::net::TcpListener;
-use tokio::sync::oneshot;
+use tokio::sync::{mpsc, oneshot};
 use tokio::time::{Duration, timeout};
 
 #[derive(Debug)]
@@ -172,6 +172,28 @@ async fn spawn_object_lambda_webhook_server_with_response(
     Ok((webhook_url, request_rx, handle))
 }
 
+async fn read_request_path(stream: &mut tokio::net::TcpStream) -> Result<String, Box<dyn Error + Send + Sync>> {
+    let mut buffer = Vec::new();
+    let mut chunk = [0_u8; 4096];
+
+    let header_end = loop {
+        let read = stream.read(&mut chunk).await?;
+        if read == 0 {
+            return Err("request ended before headers were fully received".into());
+        }
+        buffer.extend_from_slice(&chunk[..read]);
+        if let Some(pos) = find_header_terminator(&buffer) {
+            break pos;
+        }
+    };
+
+    let header_text = std::str::from_utf8(&buffer[..header_end])?;
+    let request_line = header_text.lines().next().ok_or("missing request line")?;
+    let path = request_line.split_whitespace().nth(1).ok_or("missing request path")?;
+
+    Ok(path.to_string())
+}
+
 async fn presigned_get_request(
     url: &str,
     access_key: &str,
@@ -326,11 +348,15 @@ async fn delete_webhook_target(env: &RustFSTestEnvironment, target_name: &str) -
 }
 
 fn notification_target_is_listed(targets: &serde_json::Value, target_name: &str) -> bool {
+    notification_target_entry(targets, target_name).is_some()
+}
+
+fn notification_target_entry<'a>(targets: &'a serde_json::Value, target_name: &str) -> Option<&'a serde_json::Value> {
     targets["notification_endpoints"]
         .as_array()
         .into_iter()
         .flatten()
-        .any(|entry| {
+        .find(|entry| {
             entry["account_id"].as_str() == Some(target_name)
                 && entry["service"]
                     .as_str()
@@ -338,6 +364,10 @@ fn notification_target_is_listed(targets: &serde_json::Value, target_name: &str)
         })
 }
 
+fn notification_target_status<'a>(targets: &'a serde_json::Value, target_name: &str) -> Option<&'a str> {
+    notification_target_entry(targets, target_name).and_then(|entry| entry["status"].as_str())
+}
+
 async fn wait_for_target_visibility(
     env: &RustFSTestEnvironment,
     target_name: &str,
@@ -387,6 +417,34 @@ async fn restart_rustfs_server(env: &mut RustFSTestEnvironment) -> Result<(), Bo
     env.start_rustfs_server_without_cleanup(vec![]).await
 }
 
+async fn spawn_http_origin_probe_server() -> Result<
+    (
+        String,
+        mpsc::Receiver<String>,
+        tokio::task::JoinHandle<Result<(), Box<dyn Error + Send + Sync>>>,
+    ),
+    Box<dyn Error + Send + Sync>,
+> {
+    let listener = TcpListener::bind("127.0.0.1:0").await?;
+    let address = listener.local_addr()?;
+    let webhook_url = format!("http://{address}/hook");
+    let (path_tx, path_rx) = mpsc::channel(1);
+
+    let handle = tokio::spawn(async move {
+        loop {
+            let (mut stream, _) = listener.accept().await?;
+            let path = timeout(Duration::from_secs(2), read_request_path(&mut stream)).await??;
+            let _ = path_tx.try_send(path.clone());
+            if path == "/" {
+                let response = b"HTTP/1.1 200 OK\r\nContent-Length: 0\r\nConnection: close\r\n\r\n";
+                stream.write_all(response).await?;
+            }
+        }
+    });
+
+    Ok((webhook_url, path_rx, handle))
+}
+
 async fn read_persisted_server_config(env: &RustFSTestEnvironment) -> String {
     let path = format!("{}/.rustfs.sys/config/config.json", env.temp_dir);
     match tokio::fs::read_to_string(&path).await {
@@ -515,6 +573,40 @@ async fn test_notification_target_persists_across_restart_and_delete() -> Result
     Ok(())
 }
 
+#[tokio::test]
+#[serial]
+async fn test_notification_target_with_path_is_online_via_transport_probe() -> Result<(), Box<dyn Error + Send + Sync>> {
+    init_logging();
+
+    let (webhook_url, mut probe_rx, probe_handle) = spawn_http_origin_probe_server().await?;
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_NOTIFY_ENABLE", "true")])
+        .await?;
+
+    let target_name = "path-probe";
+    configure_webhook_target(&env, target_name, &webhook_url, "secret-token").await?;
+
+    let (visible_targets, visible_arns) = wait_for_target_visibility(&env, target_name).await?;
+    assert_eq!(notification_target_status(&visible_targets, target_name), Some("online"));
+    let observed_path = timeout(Duration::from_secs(10), probe_rx.recv())
+        .await
+        .map_err(|_| "probe server timed out waiting for a request")?
+        .ok_or("probe server did not observe a request")?;
+    assert_eq!(observed_path, "/");
+    assert!(
+        visible_arns
+            .iter()
+            .any(|arn| arn.ends_with(&format!(":{target_name}:webhook"))),
+        "target ARN missing for reachable path endpoint: {visible_arns:?}"
+    );
+
+    probe_handle.abort();
+    let _ = probe_handle.await;
+
+    Ok(())
+}
+
 #[tokio::test]
 #[serial]
 async fn test_get_object_lambda_accepts_presigned_requests() -> Result<(), Box<dyn Error + Send + Sync>> {
@@ -1058,6 +1150,41 @@ async fn test_listen_notification_emits_after_put_object() -> Result<(), Box<dyn
     Ok(())
 }
 
+#[tokio::test]
+#[serial]
+async fn test_listen_notification_emits_on_empty_bucket_when_notify_disabled() -> Result<(), Box<dyn Error + Send + Sync>> {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_NOTIFY_ENABLE", "false")])
+        .await?;
+
+    let bucket = "listen-empty-bucket-e2e";
+    let key = "seed/object.txt";
+    let client = env.create_s3_client();
+
+    client.create_bucket().bucket(bucket).send().await?;
+
+    let listen_url = format!("{}/{bucket}?events={}&ping=1", env.url, urlencoding::encode("s3:ObjectCreated:*"),);
+    let response = signed_request(http::Method::GET, &listen_url, &env.access_key, &env.secret_key, None, None).await?;
+    assert_eq!(response.status(), StatusCode::OK);
+
+    let read_task = tokio::spawn(read_listen_notification_event(response, key));
+
+    client
+        .put_object()
+        .bucket(bucket)
+        .key(key)
+        .body(ByteStream::from_static(b"empty bucket watch body"))
+        .send()
+        .await?;
+
+    let payload = timeout(Duration::from_secs(12), read_task).await???;
+    assert!(!payload.is_empty(), "listen_notification payload should not be empty");
+
+    Ok(())
+}
+
 #[tokio::test]
 #[serial]
 async fn test_listen_notification_fans_in_remote_node_events() -> Result<(), Box<dyn Error + Send + Sync>> {

crates/e2e_test/src/object_lock/object_lock_test.rs

@@ -26,10 +26,12 @@
 
 use super::common::*;
 use aws_sdk_s3::Client;
-use aws_sdk_s3::primitives::ByteStream;
+use aws_sdk_s3::primitives::{ByteStream, DateTimeFormat};
 use aws_sdk_s3::types::{
-    CompletedMultipartUpload, CompletedPart, Delete, ObjectIdentifier, ObjectLockLegalHoldStatus, ObjectLockRetentionMode,
+    CompletedMultipartUpload, CompletedPart, Delete, MetadataDirective, ObjectIdentifier, ObjectLockLegalHoldStatus,
+    ObjectLockMode, ObjectLockRetentionMode,
 };
+use chrono::{DateTime, Duration, Utc};
 use serial_test::serial;
 use tracing::info;
 
@@ -79,6 +81,26 @@ fn assert_access_denied<T, E: std::fmt::Debug>(result: Result<T, E>, context: &s
     );
 }
 
+fn assert_invalid_object_lock_retention_pair<T, E: std::fmt::Debug>(result: Result<T, E>, context: &str) {
+    let err = match result {
+        Ok(_) => panic!("{context}"),
+        Err(err) => format!("{err:?}"),
+    };
+    assert!(
+        err.contains("InvalidRequest") || err.contains("must both be supplied"),
+        "{context}: expected invalid paired retention headers, got: {err}"
+    );
+}
+
+fn parse_s3_datetime(value: &aws_sdk_s3::primitives::DateTime) -> DateTime<Utc> {
+    let formatted = value
+        .fmt(DateTimeFormat::DateTime)
+        .expect("S3 timestamp should format as RFC3339");
+    DateTime::parse_from_rfc3339(&formatted)
+        .expect("S3 timestamp should parse as RFC3339")
+        .with_timezone(&Utc)
+}
+
 // ============================================================================
 // DeleteObject Tests
 // ============================================================================
@@ -537,6 +559,100 @@ async fn test_copy_object_applies_requested_legal_hold() {
     );
 }
 
+#[tokio::test]
+#[serial]
+async fn test_copy_object_does_not_inherit_source_legal_hold() {
+    init_logging();
+    info!("🧪 Test: CopyObject does not inherit source Legal Hold");
+
+    let mut env = ObjectLockTestEnvironment::new().await.unwrap();
+    env.start_rustfs().await.unwrap();
+
+    let bucket = "test-copy-object-legal-hold-inherit";
+    let src_key = "held-source";
+
+    env.create_object_lock_bucket(bucket).await.unwrap();
+
+    let client = env.s3_client();
+    put_object_with_legal_hold(&client, bucket, src_key, b"copy-source", ObjectLockLegalHoldStatus::On)
+        .await
+        .unwrap();
+
+    client
+        .copy_object()
+        .copy_source(format!("{bucket}/{src_key}"))
+        .bucket(bucket)
+        .key("implicit-copy")
+        .send()
+        .await
+        .unwrap();
+
+    let implicit_legal_hold = client
+        .get_object_legal_hold()
+        .bucket(bucket)
+        .key("implicit-copy")
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(
+        implicit_legal_hold
+            .legal_hold()
+            .and_then(|value| value.status())
+            .map(|value| value.as_str()),
+        Some("OFF")
+    );
+
+    client
+        .copy_object()
+        .copy_source(format!("{bucket}/{src_key}"))
+        .bucket(bucket)
+        .key("explicit-on-copy")
+        .object_lock_legal_hold_status(ObjectLockLegalHoldStatus::On)
+        .send()
+        .await
+        .unwrap();
+
+    let explicit_on_legal_hold = client
+        .get_object_legal_hold()
+        .bucket(bucket)
+        .key("explicit-on-copy")
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(
+        explicit_on_legal_hold
+            .legal_hold()
+            .and_then(|value| value.status())
+            .map(|value| value.as_str()),
+        Some("ON")
+    );
+
+    client
+        .copy_object()
+        .copy_source(format!("{bucket}/{src_key}"))
+        .bucket(bucket)
+        .key("explicit-off-copy")
+        .object_lock_legal_hold_status(ObjectLockLegalHoldStatus::Off)
+        .send()
+        .await
+        .unwrap();
+
+    let explicit_off_legal_hold = client
+        .get_object_legal_hold()
+        .bucket(bucket)
+        .key("explicit-off-copy")
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(
+        explicit_off_legal_hold
+            .legal_hold()
+            .and_then(|value| value.status())
+            .map(|value| value.as_str()),
+        Some("OFF")
+    );
+}
+
 #[tokio::test]
 #[serial]
 async fn test_copy_object_overwrite_blocked_by_legal_hold() {
@@ -1388,9 +1504,457 @@ async fn test_default_retention_applied_to_new_objects() {
     let delete_result = delete_object_with_bypass(&client, bucket, key, Some(version_id), false).await;
     assert!(delete_result.is_err(), "Delete should fail for object with default retention applied");
 
+    let retention = client
+        .get_object_retention()
+        .bucket(bucket)
+        .key(key)
+        .version_id(version_id)
+        .send()
+        .await
+        .unwrap();
+    let retention = retention.retention().expect("default retention should be readable");
+    assert_eq!(retention.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    assert!(
+        retention.retain_until_date().is_some(),
+        "default retention should write a retain-until date"
+    );
+
+    let head = client
+        .head_object()
+        .bucket(bucket)
+        .key(key)
+        .version_id(version_id)
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(head.object_lock_mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    assert!(
+        head.object_lock_retain_until_date().is_some(),
+        "HeadObject should expose the default retention retain-until date"
+    );
+
     info!("✅ Test passed: Default retention is applied to new objects");
 }
 
+#[tokio::test]
+#[serial]
+async fn test_delete_object_creates_delete_marker_for_default_retained_current_version() {
+    init_logging();
+    info!("🧪 Test: DeleteObject creates delete marker for default-retained current version");
+
+    let mut env = ObjectLockTestEnvironment::new().await.unwrap();
+    env.start_rustfs().await.unwrap();
+
+    let bucket = "test-default-retention-delete-marker";
+    let key = "default-retained-object";
+    let data = b"test data for default-retained current version";
+
+    env.create_object_lock_bucket(bucket).await.unwrap();
+
+    let client = env.s3_client();
+    put_object_lock_configuration(&client, bucket, ObjectLockRetentionMode::Governance, Some(30), None)
+        .await
+        .unwrap();
+
+    let put_output = client
+        .put_object()
+        .bucket(bucket)
+        .key(key)
+        .body(ByteStream::from(data.to_vec()))
+        .send()
+        .await
+        .unwrap();
+    let retained_version_id = put_output
+        .version_id()
+        .expect("default-retained object should have a version id")
+        .to_string();
+
+    let retention = client
+        .get_object_retention()
+        .bucket(bucket)
+        .key(key)
+        .version_id(&retained_version_id)
+        .send()
+        .await
+        .unwrap();
+    let retention = retention.retention().expect("default retention should be readable");
+    assert_eq!(retention.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    assert!(
+        retention.retain_until_date().is_some(),
+        "default retention should write a retain-until date"
+    );
+
+    let delete_marker_output = client.delete_object().bucket(bucket).key(key).send().await.unwrap();
+    assert_eq!(delete_marker_output.delete_marker(), Some(true));
+    let delete_marker_version_id = delete_marker_output
+        .version_id()
+        .expect("delete marker should have a version id")
+        .to_string();
+
+    let protected_delete = delete_object_with_bypass(&client, bucket, key, Some(&retained_version_id), false).await;
+    assert!(protected_delete.is_err(), "Default-retained version should still reject direct deletion");
+
+    let retention_after_delete_marker = client
+        .get_object_retention()
+        .bucket(bucket)
+        .key(key)
+        .version_id(&retained_version_id)
+        .send()
+        .await
+        .unwrap();
+    let retention_after_delete_marker = retention_after_delete_marker
+        .retention()
+        .expect("default retention should remain readable by version id after delete marker creation");
+    assert_eq!(retention_after_delete_marker.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    assert!(
+        retention_after_delete_marker.retain_until_date().is_some(),
+        "retained version should keep its retain-until date after delete marker creation"
+    );
+
+    delete_object_with_bypass(&client, bucket, key, Some(&delete_marker_version_id), false)
+        .await
+        .unwrap();
+    delete_object_with_bypass(&client, bucket, key, Some(&retained_version_id), true)
+        .await
+        .unwrap();
+
+    info!("✅ Test passed: Delete marker is allowed while default-retained version stays protected");
+}
+
+#[tokio::test]
+#[serial]
+async fn test_put_copy_and_multipart_reject_incomplete_retention_headers() {
+    init_logging();
+    info!("🧪 Test: write paths reject incomplete Object Lock retention headers");
+
+    let mut env = ObjectLockTestEnvironment::new().await.unwrap();
+    env.start_rustfs().await.unwrap();
+
+    let bucket = "test-incomplete-retention";
+    let src_key = "copy-source";
+
+    env.create_object_lock_bucket(bucket).await.unwrap();
+
+    let client = env.s3_client();
+    client
+        .put_object()
+        .bucket(bucket)
+        .key(src_key)
+        .body(ByteStream::from(b"copy-source".to_vec()))
+        .send()
+        .await
+        .unwrap();
+
+    put_object_lock_configuration(&client, bucket, ObjectLockRetentionMode::Governance, Some(30), None)
+        .await
+        .unwrap();
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .put_object()
+            .bucket(bucket)
+            .key("put-mode-only")
+            .body(ByteStream::from(b"put-body".to_vec()))
+            .object_lock_mode(ObjectLockMode::Governance)
+            .send()
+            .await,
+        "PutObject with mode only should fail",
+    );
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .put_object()
+            .bucket(bucket)
+            .key("put-date-only")
+            .body(ByteStream::from(b"put-body".to_vec()))
+            .object_lock_retain_until_date(retention_timestamp(30))
+            .send()
+            .await,
+        "PutObject with retain-until-date only should fail",
+    );
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .copy_object()
+            .copy_source(format!("{bucket}/{src_key}"))
+            .bucket(bucket)
+            .key("copy-mode-only")
+            .object_lock_mode(ObjectLockMode::Governance)
+            .send()
+            .await,
+        "CopyObject with mode only should fail",
+    );
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .copy_object()
+            .copy_source(format!("{bucket}/{src_key}"))
+            .bucket(bucket)
+            .key("copy-date-only")
+            .object_lock_retain_until_date(retention_timestamp(30))
+            .send()
+            .await,
+        "CopyObject with retain-until-date only should fail",
+    );
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .create_multipart_upload()
+            .bucket(bucket)
+            .key("multipart-mode-only")
+            .object_lock_mode(ObjectLockMode::Governance)
+            .send()
+            .await,
+        "CreateMultipartUpload with mode only should fail",
+    );
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .create_multipart_upload()
+            .bucket(bucket)
+            .key("multipart-date-only")
+            .object_lock_retain_until_date(retention_timestamp(30))
+            .send()
+            .await,
+        "CreateMultipartUpload with retain-until-date only should fail",
+    );
+}
+
+#[tokio::test]
+#[serial]
+async fn test_copy_object_retention_uses_destination_policy() {
+    init_logging();
+    info!("🧪 Test: CopyObject retention follows destination policy");
+
+    let mut env = ObjectLockTestEnvironment::new().await.unwrap();
+    env.start_rustfs().await.unwrap();
+
+    let src_bucket = "test-copy-retention-src";
+    let dst_bucket = "test-copy-retention-dst";
+    let no_default_bucket = "test-copy-retention-nodef";
+    let src_key = "retained-source";
+
+    env.create_object_lock_bucket(src_bucket).await.unwrap();
+    env.create_object_lock_bucket(dst_bucket).await.unwrap();
+    env.create_object_lock_bucket(no_default_bucket).await.unwrap();
+
+    let client = env.s3_client();
+    put_object_lock_configuration(&client, dst_bucket, ObjectLockRetentionMode::Governance, Some(1), None)
+        .await
+        .unwrap();
+
+    put_object_with_retention(
+        &client,
+        src_bucket,
+        src_key,
+        b"copy-source",
+        ObjectLockRetentionMode::Compliance,
+        future_retain_until(30),
+    )
+    .await
+    .unwrap();
+
+    let copy_started = Utc::now();
+    client
+        .copy_object()
+        .copy_source(format!("{src_bucket}/{src_key}"))
+        .bucket(dst_bucket)
+        .key("default-copy")
+        .send()
+        .await
+        .unwrap();
+
+    let retention = client
+        .get_object_retention()
+        .bucket(dst_bucket)
+        .key("default-copy")
+        .send()
+        .await
+        .unwrap();
+    let retention = retention
+        .retention()
+        .expect("destination default retention should be present");
+    assert_eq!(retention.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    let retain_until = parse_s3_datetime(retention.retain_until_date().expect("retain-until date should be present"));
+    assert!(
+        retain_until < copy_started + Duration::days(3),
+        "destination default retention should not inherit the source's longer retention"
+    );
+
+    client
+        .copy_object()
+        .copy_source(format!("{src_bucket}/{src_key}"))
+        .bucket(dst_bucket)
+        .key("replace-copy")
+        .metadata_directive(MetadataDirective::Replace)
+        .send()
+        .await
+        .unwrap();
+
+    let replace_retention = client
+        .get_object_retention()
+        .bucket(dst_bucket)
+        .key("replace-copy")
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(
+        replace_retention
+            .retention()
+            .and_then(|value| value.mode())
+            .map(|value| value.as_str()),
+        Some("GOVERNANCE")
+    );
+
+    client
+        .copy_object()
+        .copy_source(format!("{src_bucket}/{src_key}"))
+        .bucket(dst_bucket)
+        .key("explicit-copy")
+        .object_lock_mode(ObjectLockMode::Compliance)
+        .object_lock_retain_until_date(retention_timestamp(30))
+        .send()
+        .await
+        .unwrap();
+
+    let explicit_retention = client
+        .get_object_retention()
+        .bucket(dst_bucket)
+        .key("explicit-copy")
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(
+        explicit_retention
+            .retention()
+            .and_then(|value| value.mode())
+            .map(|value| value.as_str()),
+        Some("COMPLIANCE")
+    );
+    let explicit_retain_until = parse_s3_datetime(
+        explicit_retention
+            .retention()
+            .and_then(|value| value.retain_until_date())
+            .expect("explicit retain-until date should be present"),
+    );
+    assert!(
+        explicit_retain_until > Utc::now() + Duration::days(20),
+        "explicit retention should override the shorter bucket default"
+    );
+
+    client
+        .copy_object()
+        .copy_source(format!("{src_bucket}/{src_key}"))
+        .bucket(no_default_bucket)
+        .key("no-default-copy")
+        .send()
+        .await
+        .unwrap();
+
+    let no_default_retention = client
+        .get_object_retention()
+        .bucket(no_default_bucket)
+        .key("no-default-copy")
+        .send()
+        .await
+        .unwrap();
+    let no_default_retention = no_default_retention
+        .retention()
+        .expect("retention response should be present");
+    assert!(no_default_retention.mode().is_none());
+    assert!(no_default_retention.retain_until_date().is_none());
+
+    put_object_with_retention(
+        &client,
+        dst_bucket,
+        "locked-destination",
+        b"locked-target",
+        ObjectLockRetentionMode::Compliance,
+        future_retain_until(30),
+    )
+    .await
+    .unwrap();
+
+    let overwrite_result = client
+        .copy_object()
+        .copy_source(format!("{src_bucket}/{src_key}"))
+        .bucket(dst_bucket)
+        .key("locked-destination")
+        .send()
+        .await;
+    assert!(
+        overwrite_result.is_err(),
+        "CopyObject overwrite should not bypass active destination retention"
+    );
+}
+
+#[tokio::test]
+#[serial]
+async fn test_multipart_default_retention_fixed_at_create() {
+    init_logging();
+    info!("🧪 Test: multipart default retention is fixed at CreateMultipartUpload");
+
+    let mut env = ObjectLockTestEnvironment::new().await.unwrap();
+    env.start_rustfs().await.unwrap();
+
+    let bucket = "test-multipart-default-drift";
+    let key = "multipart-object";
+
+    env.create_object_lock_bucket(bucket).await.unwrap();
+
+    let client = env.s3_client();
+    put_object_lock_configuration(&client, bucket, ObjectLockRetentionMode::Governance, Some(1), None)
+        .await
+        .unwrap();
+
+    let create_started = Utc::now();
+    let create_output = client.create_multipart_upload().bucket(bucket).key(key).send().await.unwrap();
+    let upload_id = create_output.upload_id().unwrap();
+
+    let upload_part_output = client
+        .upload_part()
+        .bucket(bucket)
+        .key(key)
+        .upload_id(upload_id)
+        .part_number(1)
+        .body(ByteStream::from(b"multipart-body".to_vec()))
+        .send()
+        .await
+        .unwrap();
+
+    put_object_lock_configuration(&client, bucket, ObjectLockRetentionMode::Governance, Some(10), None)
+        .await
+        .unwrap();
+
+    let completed_upload = CompletedMultipartUpload::builder()
+        .parts(
+            CompletedPart::builder()
+                .part_number(1)
+                .e_tag(upload_part_output.e_tag().unwrap_or_default())
+                .build(),
+        )
+        .build();
+
+    client
+        .complete_multipart_upload()
+        .bucket(bucket)
+        .key(key)
+        .upload_id(upload_id)
+        .multipart_upload(completed_upload)
+        .send()
+        .await
+        .unwrap();
+
+    let retention = client.get_object_retention().bucket(bucket).key(key).send().await.unwrap();
+    let retention = retention.retention().expect("multipart default retention should be present");
+    assert_eq!(retention.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    let retain_until = parse_s3_datetime(retention.retain_until_date().expect("retain-until date should be present"));
+    assert!(
+        retain_until < create_started + Duration::days(3),
+        "CompleteMultipartUpload should keep the default retention calculated at create time"
+    );
+}
+
 // ============================================================================
 // Versioning Auto-Enable Tests
 // ============================================================================

crates/e2e_test/src/protocols/README.md

@@ -1,38 +1,26 @@
 # Protocol E2E Tests
 
-FTPS and WebDAV protocol end-to-end tests for RustFS.
+FTPS, WebDAV, and SFTP protocol end-to-end tests for RustFS.
 
 ## Prerequisites
 
-### Required Tools
-
-```bash
-# Ubuntu/Debian
-sudo apt-get install sshpass ssh-keygen
-
-# RHEL/CentOS
-sudo yum install sshpass openssh-clients
-
-# macOS
-brew install sshpass openssh
-```
+No external SSH tooling is required. The test framework generates ed25519
+host keys in-process via russh::keys under the per-test temp directory
+before each SFTP server spawn, and russh-sftp drives the protocol from the
+test process directly.
 
 ## Running Tests
 
-Run all protocol tests (FTPS + WebDAV):
 ```bash
-RUSTFS_BUILD_FEATURES=ftps,webdav cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
+RUSTFS_BUILD_FEATURES=ftps,webdav,sftp cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
 ```
 
-Run FTPS tests only:
-```bash
-RUSTFS_BUILD_FEATURES=ftps cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
-```
-
-Run WebDAV tests only:
-```bash
-RUSTFS_BUILD_FEATURES=webdav cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
-```
+`RUSTFS_BUILD_FEATURES` controls which features the test rustfs binary is
+built with. When this variable is set, the protocol test runner schedules
+only entries whose protocol is present in the requested feature list. Leave
+it unset to run every protocol entry.
+`--test-threads=1` is required because every entry spawns a rustfs server
+on fixed bind ports.
 
 ## Test Coverage
 
@@ -55,6 +43,94 @@ RUSTFS_BUILD_FEATURES=webdav cargo test --package e2e_test test_protocol_core_su
 - GET (download file)
 - PROPFIND on bucket (list objects)
 - DELETE file
+- MOVE file (rename object)
 - DELETE bucket
 - Authentication failure test
 
+### SFTP Tests
+
+The SFTP suite lives in three entries plus a standalone idle-timeout case.
+Every assertion runs against a freshly spawned rustfs binary with
+`RUSTFS_SFTP_ENABLE=true`; the test framework also pins
+`RUSTFS_SFTP_PART_SIZE=5242880` so the multipart boundary is deterministic.
+
+#### sftp_core (`test_sftp_core_operations`)
+
+Bind ports 9022 (SFTP) and 9200 (S3). 22 in-suite assertions covering the
+core protocol surface plus cross-protocol consistency:
+
+- Subsystem canary: SFTPv3 version exchange completes after password auth
+- Bucket lifecycle: mkdir, root listing, rmdir, post-delete listing
+- Small-file round-trip with SHA256 compare
+- Stat on a file (size + file type) and on a bucket (directory)
+- SETSTAT on a path returns ok
+- Rename within bucket, listing reflects the rename
+- Multipart-sized round-trip (just over 2 × part_size) with SHA256 compare
+- Negative cases: symlink rejected, open of nonexistent file rejected,
+  read_dir of nonexistent bucket rejected, path traversal rejected
+- Spec-letter assertions: APPEND open returns an error, CREATE+EXCLUDE on an
+  existing path returns an error, bad-password authentication is rejected
+- Cross-protocol via aws-sdk-s3: SFTP write then S3 read with SHA256 match,
+  S3 write then SFTP read with SHA256 match
+- Cross-API directory visibility: SFTP-created sub-directory visible via S3
+  ListObjectsV2, S3-created `__XLDIR__` marker visible via SFTP readdir as a
+  directory entry
+
+#### sftp_compliance (`test_sftp_compliance_suite`)
+
+Bind ports 9024 (SFTP) and 9300 (S3). 14 compliance regression cases against
+one shared server spawn. Each case carries a stable CMPTST-NN identifier:
+
+- CMPTST-01: medium-binary upload then download with SHA256 compare
+  (single-shot PutObject path below the multipart boundary)
+- CMPTST-02: zero-byte upload, download, and stat-size match
+- CMPTST-03: rm against a bucket path is rejected; the bucket is preserved
+- CMPTST-04: rmdir against a non-empty bucket is rejected; the contained
+  object survives
+- CMPTST-05: rmdir against a non-empty sub-directory is rejected; the inner
+  object survives
+- CMPTST-06: open with a path-traversal pattern cannot leak a host file via
+  SFTP read
+- CMPTST-07: read_dir of `/..` either errors or returns a listing that
+  contains no host system entries
+- CMPTST-08: rename across buckets preserves payload and removes the source
+  object
+- CMPTST-09: paths with embedded spaces round-trip through the russh-sftp
+  client
+- CMPTST-10: read_link is rejected (S3 storage has no symlinks)
+- CMPTST-11: SETSTAT on a path and FSETSTAT on a separate open handle both
+  return ok (rsync, WinSCP transfer-success contract)
+- CMPTST-12: rename to the same path is a no-op; the file persists with the
+  original payload
+- CMPTST-13: implicit-directory round-trip; uploading to a nested key
+  creates the parent directory implicitly and three listing forms surface
+  the inner file
+- CMPTST-14: OPEN, WRITE, FSETSTAT, CLOSE on the same write handle all
+  return ok (WinSCP wire shape)
+
+#### sftp_compliance_readonly (`test_sftp_compliance_readonly`)
+
+Bind ports 9025 (SFTP) and 9301 (S3). Spawns a second rustfs binary with
+`RUSTFS_SFTP_READ_ONLY=true`; the S3 endpoint stays writable so the suite
+can seed a bucket and a fixture object via aws-sdk-s3 before opening the
+SFTP session. 7 compliance cases:
+
+- CMPTST-15: put through SFTP is rejected
+- CMPTST-16: rm through SFTP is rejected
+- CMPTST-17: mkdir through SFTP is rejected
+- CMPTST-18: rmdir through SFTP is rejected
+- CMPTST-19: rename through SFTP is rejected
+- CMPTST-20: ls through SFTP is allowed and lists the seeded bucket
+- CMPTST-21: get through SFTP is allowed and returns the seeded payload
+  byte-for-byte
+
+The full case index lives at the top of `sftp_compliance.rs`; each helper's
+log lines name its CMPTST-NN code so a failure in CI points at one named
+property without consulting any external doc.
+
+#### sftp_idle_timeout (`test_sftp_idle_timeout_disconnects`)
+
+Bind ports 9023 (SFTP) and 9100 (S3). Spawns rustfs with
+`RUSTFS_SFTP_IDLE_TIMEOUT=5`, sleeps 10 s past the timeout, then issues an
+SFTP request and asserts the server has closed the session.
+

crates/e2e_test/src/protocols/mod.rs

@@ -12,9 +12,13 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-//! Protocol tests for FTPS and WebDAV
+//! Protocol tests for FTPS, WebDAV, and SFTP
 
 pub mod ftps_core;
+pub mod sftp_compliance;
+mod sftp_compliance_tests;
+pub mod sftp_core;
+pub mod sftp_helpers;
 pub mod test_env;
 pub mod test_runner;
 pub mod webdav_core;

crates/e2e_test/src/protocols/sftp_compliance.rs

@@ -0,0 +1,217 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Public test entry points for the SFTP compliance suite.
+//!
+//! Three suite entries cover every CMPTST-NN identifier:
+//!
+//! - test_sftp_compliance_suite           CMPTST-01..14 (one shared SFTP session)
+//! - test_sftp_compliance_readonly        CMPTST-15..23 (one shared session, read-only mode)
+//! - test_sftp_compliance_standalone      CMPTST-24..33 (each case spawns its own rustfs)
+//!
+//! The first two reuse a single rustfs spawn for the whole bracket
+//! because every case in the bracket exercises the same protocol
+//! against the same server. The third aggregates cases that each need
+//! a different server configuration (idle timeout, read-cache window,
+//! console disabled) and therefore cannot share a binary.
+//!
+//! Each per-case module exposes a single descriptive entry called
+//! run_<what_it_tests>(). For example cmptst_24 exposes
+//! cmptst_24::run_concurrent_half_close_no_leak().
+//!
+//! The per-case bodies (one cmptst_NN module per case) and the
+//! cross-case infrastructure (spawn helpers, fixture seeders, the
+//! half-close / wedge / paused-drain stream wrappers, and the session
+//! lifecycle counters) live in sftp_compliance_tests.rs. Per-case
+//! marker comments and the full case-index doc live there too.
+
+use crate::protocols::sftp_compliance_tests::{
+    cmptst_01, cmptst_02, cmptst_03, cmptst_04, cmptst_05, cmptst_06, cmptst_07, cmptst_08, cmptst_09, cmptst_10, cmptst_11,
+    cmptst_12, cmptst_13, cmptst_14, cmptst_15, cmptst_16, cmptst_17, cmptst_18, cmptst_19, cmptst_20, cmptst_21, cmptst_22,
+    cmptst_23, cmptst_27, cmptst_28, cmptst_29, cmptst_32, cmptst_33, spawn_compliance_rustfs,
+};
+#[cfg(target_os = "linux")]
+use crate::protocols::sftp_compliance_tests::{cmptst_24, cmptst_25, cmptst_26};
+use crate::protocols::sftp_helpers::{build_test_s3_client, connect_sftp_to, wait_for_s3_ready};
+use crate::protocols::test_env::ProtocolTestEnvironment;
+use anyhow::{Result, anyhow};
+use aws_sdk_s3::primitives::ByteStream;
+use tracing::info;
+
+// Read-write compliance suite ports. Distinct from sftp_core (9022/9200)
+// and from test_sftp_idle_timeout_disconnects (9023/9100) so the SFTP
+// entries can run sequentially without leftover-listener contention.
+const COMPLIANCE_RW_SFTP_PORT: u16 = 9024;
+const COMPLIANCE_RW_SFTP_ADDRESS: &str = "127.0.0.1:9024";
+const COMPLIANCE_RW_S3_ADDRESS: &str = "127.0.0.1:9300";
+
+// Read-only compliance suite ports. The SFTP session opened against
+// this address runs against a server started with
+// RUSTFS_SFTP_READ_ONLY=true. The S3 endpoint stays writable so the
+// suite can seed a bucket and a fixture object before running the SFTP
+// rejection assertions.
+const COMPLIANCE_RO_SFTP_PORT: u16 = 9025;
+const COMPLIANCE_RO_SFTP_ADDRESS: &str = "127.0.0.1:9025";
+const COMPLIANCE_RO_S3_ADDRESS: &str = "127.0.0.1:9301";
+const COMPLIANCE_RO_S3_ENDPOINT: &str = "http://127.0.0.1:9301";
+const COMPLIANCE_RO_S3_READY_ATTEMPTS: u32 = 30;
+
+/// Compliance suite entry: spawn one rustfs server, run every per-case
+/// helper that closes a coverage gap not exercised by sftp_core. Runs
+/// CMPTST-01 through CMPTST-14 against the same SFTP session.
+pub async fn test_sftp_compliance_suite() -> Result<()> {
+    info!("Starting SFTP server for compliance suite on {}", COMPLIANCE_RW_SFTP_ADDRESS);
+    let (_env, mut server_process) = spawn_compliance_rustfs(COMPLIANCE_RW_SFTP_ADDRESS, COMPLIANCE_RW_S3_ADDRESS, false).await?;
+
+    let result = async {
+        ProtocolTestEnvironment::wait_for_port_ready(COMPLIANCE_RW_SFTP_PORT, 30)
+            .await
+            .map_err(|e| anyhow!("{}", e))?;
+
+        let (session, sftp) = connect_sftp_to(COMPLIANCE_RW_SFTP_ADDRESS).await?;
+
+        cmptst_01::run_medium_binary_round_trip(&sftp).await?;
+        cmptst_02::run_zero_byte_round_trip(&sftp).await?;
+        cmptst_03::run_rm_on_bucket_path_rejected(&sftp).await?;
+        cmptst_04::run_rmdir_nonempty_bucket_rejected(&sftp).await?;
+        cmptst_05::run_rmdir_nonempty_subdir_rejected(&sftp).await?;
+        cmptst_06::run_path_traversal_get_rejected(&sftp).await?;
+        cmptst_07::run_dotdot_collapses_to_root(&sftp).await?;
+        cmptst_08::run_rename_cross_bucket(&sftp).await?;
+        cmptst_09::run_path_with_spaces_round_trip(&sftp).await?;
+        cmptst_10::run_readlink_rejected(&sftp).await?;
+        cmptst_11::run_setstat_after_put_returns_ok(&sftp).await?;
+        cmptst_12::run_rename_same_path_keeps_file(&sftp).await?;
+        cmptst_13::run_implicit_dir_round_trip(&sftp).await?;
+        cmptst_14::run_winscp_setstat_shape_on_handle(&sftp).await?;
+
+        drop(sftp);
+        session.disconnect(russh::Disconnect::ByApplication, "", "en").await?;
+        info!("SFTP compliance suite passed");
+        Ok::<(), anyhow::Error>(())
+    }
+    .await;
+
+    // Discard kill/wait errors on the teardown path: the test result
+    // above is the binding outcome, and a server that has already
+    // exited produces an error here that carries no useful signal.
+    server_process.kill_and_wait().await;
+
+    result
+}
+
+/// Read-only compliance entry: CMPTST-15 through CMPTST-23. The SFTP
+/// server runs with RUSTFS_SFTP_READ_ONLY=true. Mutations through SFTP
+/// must error. Reads through SFTP must succeed. The test seeds a bucket
+/// and a file through the writable S3 endpoint before opening the SFTP
+/// session.
+pub async fn test_sftp_compliance_readonly() -> Result<()> {
+    info!("Starting SFTP server in read-only mode on {}", COMPLIANCE_RO_SFTP_ADDRESS);
+    let (_env, mut server_process) = spawn_compliance_rustfs(COMPLIANCE_RO_SFTP_ADDRESS, COMPLIANCE_RO_S3_ADDRESS, true).await?;
+
+    let result = async {
+        ProtocolTestEnvironment::wait_for_port_ready(COMPLIANCE_RO_SFTP_PORT, 30)
+            .await
+            .map_err(|e| anyhow!("{}", e))?;
+
+        let s3 = build_test_s3_client(COMPLIANCE_RO_S3_ENDPOINT);
+        wait_for_s3_ready(&s3, COMPLIANCE_RO_S3_READY_ATTEMPTS).await?;
+
+        let bucket = "robucket";
+        let seeded_key = "small.txt";
+        let seeded_content = b"read-only seed\n";
+
+        s3.create_bucket()
+            .bucket(bucket)
+            .send()
+            .await
+            .map_err(|e| anyhow!("S3 CreateBucket {} failed: {:?}", bucket, e))?;
+        s3.put_object()
+            .bucket(bucket)
+            .key(seeded_key)
+            .body(ByteStream::from_static(seeded_content))
+            .send()
+            .await
+            .map_err(|e| anyhow!("S3 PutObject {}/{} failed: {:?}", bucket, seeded_key, e))?;
+        info!("Seeded read-only fixture via S3: {}/{}", bucket, seeded_key);
+
+        let (session, sftp) = connect_sftp_to(COMPLIANCE_RO_SFTP_ADDRESS).await?;
+
+        cmptst_15::run_ro_put_rejected(&sftp, bucket).await?;
+        cmptst_16::run_ro_rm_rejected(&sftp, bucket, seeded_key).await?;
+        cmptst_17::run_ro_mkdir_rejected(&sftp).await?;
+        cmptst_18::run_ro_rmdir_rejected(&sftp, bucket).await?;
+        cmptst_19::run_ro_rename_rejected(&sftp, bucket, seeded_key).await?;
+        cmptst_20::run_ro_ls_allowed(&sftp, bucket).await?;
+        cmptst_21::run_ro_get_allowed(&sftp, bucket, seeded_key, seeded_content).await?;
+        cmptst_22::run_ro_setstat_rejected(&sftp, bucket, seeded_key).await?;
+        cmptst_23::run_ro_fsetstat_rejected(&sftp, bucket, seeded_key).await?;
+
+        drop(sftp);
+        // Discard the disconnect Result. A read-only session that has
+        // returned errors against every mutation can still be cleanly
+        // torn down, but a transient transport-level error here
+        // carries no useful signal beyond what the assertions above
+        // already pin.
+        let _ = session.disconnect(russh::Disconnect::ByApplication, "", "en").await;
+        info!("SFTP read-only compliance suite passed");
+        Ok::<(), anyhow::Error>(())
+    }
+    .await;
+
+    server_process.kill_and_wait().await;
+
+    result
+}
+
+/// Standalone-server compliance entry: runs CMPTST-24..33 in numerical
+/// order. Each case spawns and tears down its own rustfs because each
+/// exercises a different server configuration (idle timeout, console
+/// listener, read-cache window) that cannot share a process with the
+/// others.
+///
+/// CMPTST-30 is omitted by default. Its assertion (per-operation
+/// wall-clock latency under pipelined metadata ops) is bounded by the
+/// SSH SFTP subsystem's per-channel serial handler dispatch and is
+/// structurally infeasible against the production code. The
+/// test_sftp_handler_latency_regression #[tokio::test] entry remains
+/// runnable on demand via `--ignored`.
+///
+/// CMPTST-31 (paused-drain) is omitted from the default suite for
+/// runtime cost (200 MiB seed plus a 25 s pause window). The
+/// test_sftp_paused_drain_regression #[tokio::test] entry covers it
+/// for direct invocation.
+///
+/// CMPTST-24, 25, 26 verify kernel-level state via ss(8) and the
+/// procfs ESTABLISHED discriminator. They are skipped on non-Linux
+/// targets where those interfaces are absent.
+pub async fn test_sftp_compliance_standalone() -> Result<()> {
+    info!("Starting SFTP standalone-server compliance suite");
+
+    #[cfg(target_os = "linux")]
+    {
+        cmptst_24::run_concurrent_half_close_no_leak().await?;
+        cmptst_25::run_wedge_kill_after_silence_in_close_wait().await?;
+        cmptst_26::run_healthy_idle_session_above_fast_threshold().await?;
+    }
+
+    cmptst_27::run_multi_session_mixed_pipelining().await?;
+    cmptst_28::run_5mb_download_with_concurrent_metadata_ops().await?;
+    cmptst_29::run_read_past_eof_volume().await?;
+    cmptst_32::run_read_cache_enabled_round_trip().await?;
+    cmptst_33::run_read_cache_disabled_round_trip().await?;
+
+    info!("SFTP standalone-server compliance suite passed");
+    Ok(())
+}

crates/e2e_test/src/protocols/sftp_core.rs

@@ -0,0 +1,557 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Core SFTP tests
+
+use crate::common::rustfs_binary_path_with_features;
+use crate::protocols::sftp_helpers::{
+    AcceptAnyServerKey, ServerProcess, build_test_s3_client, connect_sftp_to, generate_host_key, sftp_read_full,
+    wait_for_s3_ready,
+};
+use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, ProtocolTestEnvironment};
+use anyhow::{Result, anyhow};
+use aws_sdk_s3::Client as S3Client;
+use aws_sdk_s3::primitives::ByteStream;
+use russh::client::{self, Handle};
+use russh_sftp::client::SftpSession;
+use russh_sftp::protocol::{FileAttributes, OpenFlags};
+use rustfs_config::{
+    ENV_RUSTFS_ADDRESS, ENV_SFTP_ADDRESS, ENV_SFTP_ENABLE, ENV_SFTP_HOST_KEY_DIR, ENV_SFTP_IDLE_TIMEOUT, ENV_SFTP_PART_SIZE,
+    ENV_SFTP_READ_ONLY,
+};
+use sha2::{Digest, Sha256};
+use std::path::PathBuf;
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::io::{AsyncReadExt, AsyncWriteExt};
+use tokio::process::Command;
+use tokio::time::sleep;
+use tracing::info;
+
+const SFTP_PORT: u16 = 9022;
+const SFTP_ADDRESS: &str = "127.0.0.1:9022";
+
+// The cross-protocol assertions reach the same server process the SFTP
+// session is connected to. The S3 endpoint is bound on a non-default port to
+// avoid contention with any other rustfs running on port 9000 (for example a
+// dev-testing harness container).
+const S3_ADDRESS: &str = "127.0.0.1:9200";
+const S3_ENDPOINT: &str = "http://127.0.0.1:9200";
+const S3_READY_ATTEMPTS: u32 = 30;
+
+// Mirrors GLOBAL_DIR_SUFFIX in rustfs_utils::path. The e2e_test crate does
+// not depend on rustfs-utils, so the suffix is repeated locally.
+const XLDIR_SUFFIX: &str = "__XLDIR__";
+
+// Idle-timeout test uses its own ports so it can run alongside the core suite
+// without clashing on the default S3 port or on the core SFTP port.
+const IDLE_SFTP_PORT: u16 = 9023;
+const IDLE_SFTP_ADDRESS: &str = "127.0.0.1:9023";
+const IDLE_S3_ADDRESS: &str = "127.0.0.1:9100";
+const IDLE_TIMEOUT_SECS: u64 = 5;
+const IDLE_WAIT_SECS: u64 = 10;
+
+// Pin the server's multipart part_size to the spec minimum (5 MiB) so the
+// multipart payload in this file is sized relative to a known value and the
+// Buffering to Streaming transition triggers deterministically regardless of
+// the server's default.
+const PART_SIZE_BYTES: usize = 5 * 1024 * 1024;
+const PART_SIZE_ENV: &str = "5242880";
+// Just over two part_size worth so the upload issues CreateMultipartUpload,
+// at least one UploadPart mid-stream, and CompleteMultipartUpload.
+const MULTIPART_SIZE: usize = PART_SIZE_BYTES * 2 + 1024;
+
+// Fixed deterministic payload for the S3-write, SFTP-read direction. 256 KiB
+// is well below part_size so the SFTP read returns the object as a single
+// GetObject response without invoking the streaming multipart path.
+const S3_WRITTEN_SIZE: usize = 256 * 1024;
+
+async fn connect_sftp() -> Result<(Handle<AcceptAnyServerKey>, SftpSession)> {
+    connect_sftp_to(SFTP_ADDRESS).await
+}
+
+/// Confirm that an object is byte-identical when fetched via S3 and via SFTP.
+/// Hashes the expected payload once, then compares both fetched payloads
+/// against that hash. Either mismatch returns an error naming the side that
+/// disagreed.
+async fn assert_cross_protocol_sha_match(
+    s3: &S3Client,
+    sftp: &SftpSession,
+    bucket: &str,
+    key: &str,
+    expected: &[u8],
+) -> Result<()> {
+    let expected_sha = Sha256::digest(expected);
+
+    let s3_get = s3
+        .get_object()
+        .bucket(bucket)
+        .key(key)
+        .send()
+        .await
+        .map_err(|e| anyhow!("S3 GetObject {}/{} failed: {:?}", bucket, key, e))?;
+    let s3_bytes = s3_get
+        .body
+        .collect()
+        .await
+        .map_err(|e| anyhow!("S3 body collect failed for {}/{}: {:?}", bucket, key, e))?
+        .into_bytes();
+    if s3_bytes.len() != expected.len() {
+        return Err(anyhow!(
+            "S3 GetObject byte count mismatch for {}/{}: expected {}, got {}",
+            bucket,
+            key,
+            expected.len(),
+            s3_bytes.len()
+        ));
+    }
+    let s3_sha = Sha256::digest(&s3_bytes);
+    if s3_sha != expected_sha {
+        return Err(anyhow!("S3 GetObject SHA256 mismatch for {}/{}", bucket, key));
+    }
+
+    let sftp_path = format!("/{bucket}/{key}");
+    let sftp_bytes = sftp_read_full(sftp, &sftp_path).await?;
+    if sftp_bytes.len() != expected.len() {
+        return Err(anyhow!(
+            "SFTP read byte count mismatch for {}: expected {}, got {}",
+            sftp_path,
+            expected.len(),
+            sftp_bytes.len()
+        ));
+    }
+    let sftp_sha = Sha256::digest(&sftp_bytes);
+    if sftp_sha != expected_sha {
+        return Err(anyhow!("SFTP read SHA256 mismatch for {}", sftp_path));
+    }
+
+    Ok(())
+}
+
+/// SFTP core protocol round-trip: banner, mkdir, put, get with SHA compare, rename, delete, rmdir.
+pub async fn test_sftp_core_operations() -> Result<()> {
+    let env = ProtocolTestEnvironment::new().map_err(|e| anyhow!("{}", e))?;
+    let host_key_dir = PathBuf::from(&env.temp_dir).join("sftp_host_keys");
+    generate_host_key(&host_key_dir).await?;
+
+    info!("Starting SFTP server on {}", SFTP_ADDRESS);
+    let binary_path = rustfs_binary_path_with_features(Some("ftps,webdav,sftp"));
+    let host_key_dir_str = host_key_dir
+        .to_str()
+        .ok_or_else(|| anyhow!("host key dir path is not utf-8"))?;
+    let mut server_process = ServerProcess::new(
+        Command::new(&binary_path)
+            .env(ENV_SFTP_ENABLE, "true")
+            .env(ENV_SFTP_ADDRESS, SFTP_ADDRESS)
+            .env(ENV_SFTP_HOST_KEY_DIR, host_key_dir_str)
+            .env(ENV_SFTP_READ_ONLY, "false")
+            .env(ENV_SFTP_PART_SIZE, PART_SIZE_ENV)
+            .env(ENV_RUSTFS_ADDRESS, S3_ADDRESS)
+            .arg(&env.temp_dir)
+            .spawn()?,
+    );
+
+    let result = async {
+        ProtocolTestEnvironment::wait_for_port_ready(SFTP_PORT, 30)
+            .await
+            .map_err(|e| anyhow!("{}", e))?;
+
+        let (session, sftp) = connect_sftp().await?;
+
+        // --- 1. Subsystem canary: SFTP session reachable after password auth ---
+        // SftpSession::new completes the SFTPv3 version exchange. The
+        // canonicalize call below is a cheap round-trip that confirms
+        // the session handles real wire traffic.
+        info!("Testing SFTP: subsystem canary, server resolves '.' to an absolute path");
+        let pwd = sftp.canonicalize(".").await?;
+        assert!(!pwd.is_empty(), "server must resolve '.' to a non-empty absolute path");
+        info!("PASS: subsystem canary: server resolved '.' to {}", pwd);
+
+        // --- 2. Bucket lifecycle: mkdir then root listing ---
+        let bucket = "coretestbucket";
+        let bucket_path = format!("/{bucket}");
+        info!("Testing SFTP: mkdir bucket {}", bucket_path);
+        sftp.create_dir(&bucket_path).await?;
+        info!("PASS: mkdir bucket {}", bucket_path);
+
+        info!("Testing SFTP: root listing includes the new bucket");
+        let root_entries: Vec<String> = sftp.read_dir("/").await?.map(|e| e.file_name()).collect();
+        assert!(root_entries.iter().any(|n| n == bucket), "root listing should contain the new bucket");
+        info!("PASS: bucket {} appeared in read_dir(\"/\")", bucket);
+
+        // --- 3. Small-file round-trip with SHA256 compare ---
+        info!("Testing SFTP: small-file round-trip with SHA256 compare");
+        let small_path = format!("/{bucket}/small.txt");
+        let small_content = b"hello rustfs sftp\n";
+        let mut wf = sftp
+            .open_with_flags(&small_path, OpenFlags::CREATE | OpenFlags::TRUNCATE | OpenFlags::WRITE)
+            .await?;
+        wf.write_all(small_content).await?;
+        wf.flush().await?;
+        wf.shutdown().await?;
+
+        let mut rf = sftp.open_with_flags(&small_path, OpenFlags::READ).await?;
+        let mut buf = Vec::new();
+        rf.read_to_end(&mut buf).await?;
+        rf.shutdown().await?;
+        assert_eq!(buf.as_slice(), small_content, "small-file round-trip content mismatch");
+        let sha_in = Sha256::digest(small_content);
+        let sha_out = Sha256::digest(&buf);
+        assert_eq!(sha_in, sha_out, "small-file SHA256 mismatch");
+        info!("PASS: small-file round-trip SHA256 match");
+
+        // --- 4. Path STAT on a file and on a bucket ---
+        info!("Testing SFTP: stat on file returns size and file type");
+        let file_meta = sftp.metadata(&small_path).await?;
+        assert_eq!(file_meta.size, Some(small_content.len() as u64), "stat size mismatch");
+        assert!(file_meta.file_type().is_file(), "stat on a file must report regular file");
+        info!("PASS: stat on file reports size {} and file type", small_content.len());
+
+        info!("Testing SFTP: stat on bucket reports directory");
+        let bucket_meta = sftp.metadata(&bucket_path).await?;
+        assert!(bucket_meta.file_type().is_dir(), "stat on a bucket must report directory");
+        info!("PASS: stat on bucket reports directory");
+
+        // --- 5. SETSTAT on a file path returns ok ---
+        // SETSTAT is a no-op on the server because S3 has no POSIX mtime or permission
+        // semantics, but it must still return ok. Clients that send SETSTAT after every
+        // transfer (rsync, WinSCP) treat a non-ok status as a transfer failure.
+        info!("Testing SFTP: setstat on a path returns ok");
+        let attrs = FileAttributes {
+            permissions: Some(0o644),
+            ..FileAttributes::default()
+        };
+        sftp.set_metadata(&small_path, attrs).await?;
+        info!("PASS: setstat returned ok");
+
+        // --- 6. Rename within bucket and listing reflects it ---
+        info!("Testing SFTP: rename within bucket");
+        let renamed = format!("/{bucket}/renamed.txt");
+        sftp.rename(&small_path, &renamed).await?;
+        info!("PASS: rename {} -> {}", small_path, renamed);
+
+        info!("Testing SFTP: listing reflects rename");
+        let bucket_entries: Vec<String> = sftp.read_dir(&bucket_path).await?.map(|e| e.file_name()).collect();
+        assert!(bucket_entries.iter().any(|n| n == "renamed.txt"), "renamed file must be listed");
+        assert!(!bucket_entries.iter().any(|n| n == "small.txt"), "pre-rename name must be gone");
+        info!("PASS: directory listing reflects the rename");
+
+        // --- 7. Multipart round-trip across the part-size boundary ---
+        // MULTIPART_SIZE is paired with RUSTFS_SFTP_PART_SIZE above so the upload crosses the
+        // multipart threshold regardless of server defaults.
+        info!("Testing SFTP: multipart-sized round-trip with SHA256 compare");
+        let big_path = format!("/{bucket}/big.bin");
+        let big_content: Vec<u8> = (0..MULTIPART_SIZE).map(|i| (i as u8).wrapping_mul(31)).collect();
+        let mut bwf = sftp
+            .open_with_flags(&big_path, OpenFlags::CREATE | OpenFlags::TRUNCATE | OpenFlags::WRITE)
+            .await?;
+        bwf.write_all(&big_content).await?;
+        bwf.flush().await?;
+        bwf.shutdown().await?;
+
+        let mut brf = sftp.open_with_flags(&big_path, OpenFlags::READ).await?;
+        let mut big_buf = Vec::with_capacity(MULTIPART_SIZE);
+        brf.read_to_end(&mut big_buf).await?;
+        brf.shutdown().await?;
+        assert_eq!(big_buf.len(), MULTIPART_SIZE, "multipart round-trip length mismatch");
+        let big_in = Sha256::digest(&big_content);
+        let big_out = Sha256::digest(&big_buf);
+        assert_eq!(big_in, big_out, "multipart SHA256 mismatch");
+        info!("PASS: multipart round-trip SHA256 match ({} bytes)", MULTIPART_SIZE);
+
+        // --- 8. Negative cases: symlink, open nonexistent, read_dir nonexistent, path escape ---
+        info!("Testing SFTP: symlink returns an error");
+        let symlink_err = sftp.symlink(&big_path, &format!("/{bucket}/shortcut")).await;
+        assert!(symlink_err.is_err(), "symlink must be rejected by the server");
+        info!("PASS: symlink rejected");
+
+        info!("Testing SFTP: open of nonexistent file returns an error");
+        let missing_path = format!("/{bucket}/not_here.txt");
+        let missing_err = sftp.open_with_flags(&missing_path, OpenFlags::READ).await;
+        assert!(missing_err.is_err(), "open of a nonexistent path must error");
+        info!("PASS: open of nonexistent file rejected");
+
+        info!("Testing SFTP: read_dir of nonexistent bucket returns an error");
+        let missing_bucket = sftp.read_dir("/nosuchbucket").await;
+        assert!(missing_bucket.is_err(), "read_dir of a nonexistent bucket must error");
+        info!("PASS: read_dir of nonexistent bucket rejected");
+
+        info!("Testing SFTP: path traversal cannot escape the storage root");
+        let traversal = sftp.read_dir("/../../../etc").await;
+        assert!(traversal.is_err(), "path traversal must be rejected or resolve to a nonexistent bucket");
+        info!("PASS: path traversal rejected");
+
+        // --- Spec-letter assertion: APPEND open-flag returns an error ---
+        // The driver maps APPEND to OpUnsupported because S3 has no append
+        // primitive. Open requests with APPEND must return a failure rather
+        // than allow a silently mistruncated upload.
+        info!("Testing SFTP: open with APPEND returns an error");
+        let append_err = sftp.open_with_flags(&renamed, OpenFlags::APPEND | OpenFlags::WRITE).await;
+        assert!(append_err.is_err(), "open with APPEND must error");
+        info!("PASS: open with APPEND rejected");
+
+        // --- Spec-letter assertion: O_EXCL on existing path returns an error ---
+        // CREATE | EXCLUDE on a key that already exists must fail. The
+        // existing renamed.txt is the target. EXCLUDE without WRITE is
+        // rejected by the russh-sftp client itself, so WRITE is included.
+        // TRUNCATE is included because the driver requires WRITE | CREATE
+        // | TRUNCATE on every accepted write OPEN.
+        info!("Testing SFTP: open with CREATE + EXCLUDE on existing path returns an error");
+        let excl_err = sftp
+            .open_with_flags(&renamed, OpenFlags::CREATE | OpenFlags::TRUNCATE | OpenFlags::EXCLUDE | OpenFlags::WRITE)
+            .await;
+        assert!(excl_err.is_err(), "CREATE+EXCLUDE on existing path must error");
+        info!("PASS: CREATE+EXCLUDE on existing path rejected");
+
+        // --- WRITE without CREATE or TRUNCATE is rejected at OPEN ---
+        // The streaming write path overwrites the entire object at
+        // close. A WRITE-only OPEN asks for partial-write semantics
+        // the server cannot honour against S3, so the OPEN is
+        // rejected before any handle is allocated.
+        info!("Testing SFTP: open with WRITE only returns an error");
+        let write_only_err = sftp.open_with_flags(&renamed, OpenFlags::WRITE).await;
+        assert!(write_only_err.is_err(), "WRITE without CREATE or TRUNCATE must be rejected at OPEN");
+        info!("PASS: WRITE only rejected");
+
+        // --- WRITE | CREATE without TRUNCATE is rejected at OPEN ---
+        // Without TRUNCATE the client is asking for create-or-modify-
+        // existing semantics. The server cannot deliver that against
+        // S3, so the OPEN is rejected before any handle is allocated.
+        info!("Testing SFTP: open with WRITE | CREATE without TRUNCATE returns an error");
+        let create_no_trunc_err = sftp.open_with_flags(&renamed, OpenFlags::WRITE | OpenFlags::CREATE).await;
+        assert!(create_no_trunc_err.is_err(), "WRITE | CREATE without TRUNCATE must be rejected at OPEN");
+        info!("PASS: WRITE | CREATE without TRUNCATE rejected");
+
+        // --- Spec-letter assertion: bad password is rejected (separate session) ---
+        // Fresh russh session with wrong credentials. The authenticated
+        // handle is left untouched. Bad auth must not succeed.
+        info!("Testing SFTP: second russh session with wrong password is rejected");
+        let bad_config = Arc::new(client::Config::default());
+        let mut bad_session = client::connect(bad_config, SFTP_ADDRESS, AcceptAnyServerKey).await?;
+        let bad_auth = bad_session.authenticate_password(DEFAULT_ACCESS_KEY, "wrong-secret").await?;
+        assert!(!bad_auth.success(), "bad-password authentication must not succeed");
+        // Discard the disconnect Result. A server that already rejected
+        // auth can return an error here, but the assert above already
+        // pins the auth outcome.
+        let _ = bad_session.disconnect(russh::Disconnect::ByApplication, "", "en").await;
+        info!("PASS: bad-password authentication rejected");
+
+        // --- Cross-protocol setup: aws-sdk-s3 client against the same server ---
+        // The rustfs binary spawned for this suite serves both SFTP on port
+        // 9022 and S3 on port 9000. The S3 stack may need a moment to finish
+        // initialising after TCP is listening, so list_buckets is polled
+        // until it succeeds before any cross-protocol assertion runs.
+        info!("Testing SFTP: prepare aws-sdk-s3 client and wait for S3 readiness");
+        let s3 = build_test_s3_client(S3_ENDPOINT);
+        wait_for_s3_ready(&s3, S3_READY_ATTEMPTS).await?;
+        info!("PASS: S3 endpoint reachable from cross-protocol client");
+
+        // --- SFTP write, S3 read: SHA256 round-trip ---
+        // SFTP creates the object, then assert_cross_protocol_sha_match
+        // fetches it via both S3 GetObject and SFTP READ and compares
+        // each result against the SHA256 of the original payload. Both
+        // sides must match byte-exact, which proves the storage layer
+        // returns the same bytes regardless of wire protocol.
+        info!("Testing SFTP: SFTP write then S3 read, SHA256 round-trip");
+        let sftp_to_s3_key = "sftp_written.bin";
+        let sftp_to_s3_path = format!("/{bucket}/{sftp_to_s3_key}");
+        let sftp_to_s3_content: Vec<u8> = (0..S3_WRITTEN_SIZE).map(|i| (i as u8).wrapping_mul(17)).collect();
+        let mut wf = sftp
+            .open_with_flags(&sftp_to_s3_path, OpenFlags::CREATE | OpenFlags::TRUNCATE | OpenFlags::WRITE)
+            .await?;
+        wf.write_all(&sftp_to_s3_content).await?;
+        wf.flush().await?;
+        wf.shutdown().await?;
+        assert_cross_protocol_sha_match(&s3, &sftp, bucket, sftp_to_s3_key, &sftp_to_s3_content).await?;
+        info!("PASS: SFTP-written object matches via S3 GetObject and SFTP READ");
+
+        // --- S3 write, SFTP read: SHA256 round-trip ---
+        // aws-sdk-s3 PutObject writes a fixed deterministic payload. Both
+        // sides then read it back and SHA-compare.
+        info!("Testing SFTP: S3 write then SFTP read, SHA256 round-trip");
+        let s3_to_sftp_key = "s3_written.bin";
+        let s3_to_sftp_content: Vec<u8> = (0..S3_WRITTEN_SIZE).map(|i| (i as u8).wrapping_mul(31)).collect();
+        s3.put_object()
+            .bucket(bucket)
+            .key(s3_to_sftp_key)
+            .body(ByteStream::from(s3_to_sftp_content.clone()))
+            .send()
+            .await
+            .map_err(|e| anyhow!("S3 PutObject {}/{} failed: {:?}", bucket, s3_to_sftp_key, e))?;
+        assert_cross_protocol_sha_match(&s3, &sftp, bucket, s3_to_sftp_key, &s3_to_sftp_content).await?;
+        info!("PASS: S3-written object matches via S3 GetObject and SFTP READ");
+
+        // --- Cross-API directory visibility: SFTP mkdir, S3 ListObjectsV2 ---
+        // SFTP mkdir writes a __XLDIR__ marker. The rustfs S3 listing path
+        // decodes that marker back to a trailing-slash key, so the asserted
+        // pattern is "subdir_sftp/".
+        info!("Testing SFTP: SFTP-created sub-directory visible via S3 ListObjectsV2");
+        let sftp_subdir_name = "subdir_sftp";
+        let sftp_subdir_path = format!("/{bucket}/{sftp_subdir_name}");
+        sftp.create_dir(&sftp_subdir_path).await?;
+        let listed = s3
+            .list_objects_v2()
+            .bucket(bucket)
+            .prefix(sftp_subdir_name)
+            .send()
+            .await
+            .map_err(|e| anyhow!("S3 ListObjectsV2 {} failed: {:?}", bucket, e))?;
+        let listed_keys: Vec<String> = listed
+            .contents()
+            .iter()
+            .filter_map(|obj| obj.key().map(|s| s.to_string()))
+            .collect();
+        let visible_via_s3 = listed_keys
+            .iter()
+            .any(|k| k == &format!("{sftp_subdir_name}/") || k == &format!("{sftp_subdir_name}{XLDIR_SUFFIX}"));
+        assert!(
+            visible_via_s3,
+            "SFTP-created sub-directory must appear in S3 ListObjectsV2: keys returned were {listed_keys:?}"
+        );
+        info!("PASS: SFTP mkdir visible to S3 ListObjectsV2");
+
+        // --- Cross-API directory visibility: S3 marker, SFTP readdir ---
+        // aws-sdk-s3 PutObject writes a zero-byte marker keyed with the
+        // __XLDIR__ suffix. SFTP readdir must decode the marker back to a
+        // bare directory entry whose file_type reports as a directory.
+        info!("Testing SFTP: S3-created __XLDIR__ marker visible via SFTP readdir");
+        let s3_subdir_name = "subdir_s3";
+        let s3_subdir_marker_key = format!("{s3_subdir_name}{XLDIR_SUFFIX}");
+        s3.put_object()
+            .bucket(bucket)
+            .key(&s3_subdir_marker_key)
+            .body(ByteStream::from_static(b""))
+            .send()
+            .await
+            .map_err(|e| anyhow!("S3 PutObject {}/{} failed: {:?}", bucket, s3_subdir_marker_key, e))?;
+        let bucket_entries: Vec<(String, bool)> = sftp
+            .read_dir(&bucket_path)
+            .await?
+            .map(|entry| (entry.file_name(), entry.file_type().is_dir()))
+            .collect();
+        let visible_via_sftp = bucket_entries.iter().any(|(name, is_dir)| name == s3_subdir_name && *is_dir);
+        assert!(
+            visible_via_sftp,
+            "S3-created marker must appear as a directory in SFTP readdir: entries were {bucket_entries:?}"
+        );
+        info!("PASS: S3 marker visible to SFTP readdir as a directory");
+
+        // --- Pre-cleanup of cross-protocol fixtures ---
+        // Removes the new files and sub-directories so the existing rmdir
+        // call below operates against an empty bucket.
+        info!("Testing SFTP: pre-cleanup of cross-protocol fixtures");
+        sftp.remove_file(&format!("/{bucket}/{sftp_to_s3_key}")).await?;
+        sftp.remove_file(&format!("/{bucket}/{s3_to_sftp_key}")).await?;
+        sftp.remove_dir(&sftp_subdir_path).await?;
+        sftp.remove_dir(&format!("/{bucket}/{s3_subdir_name}")).await?;
+        info!("PASS: cross-protocol fixtures removed");
+
+        // --- 9. Cleanup: delete objects, rmdir bucket, confirm root empty ---
+        info!("Testing SFTP: delete objects then rmdir bucket");
+        sftp.remove_file(&renamed).await?;
+        sftp.remove_file(&big_path).await?;
+        sftp.remove_dir(&bucket_path).await?;
+        info!("PASS: delete + rmdir leaves the root empty");
+
+        let final_entries: Vec<String> = sftp.read_dir("/").await?.map(|e| e.file_name()).collect();
+        assert!(!final_entries.iter().any(|n| n == bucket), "bucket must be gone after rmdir");
+        info!("PASS: root listing no longer includes the deleted bucket");
+
+        drop(sftp);
+        session.disconnect(russh::Disconnect::ByApplication, "", "en").await?;
+        info!("SFTP core tests passed");
+        Ok::<(), anyhow::Error>(())
+    }
+    .await;
+
+    // Discard kill/wait errors on the teardown path: the test result
+    // above is the binding outcome, and a server that has already
+    // exited produces an error here that carries no useful signal.
+    server_process.kill_and_wait().await;
+
+    result
+}
+
+/// Idle-timeout regression: the server must close an SFTP session that
+/// remains inactive past RUSTFS_SFTP_IDLE_TIMEOUT.
+///
+/// Spawns its own rustfs binary on dedicated SFTP and S3 ports so it can run
+/// independently of the core protocol suite. The disconnect check issues a
+/// cheap SFTP request after the wait window. The same error path runs in any
+/// client when the server-initiated SSH_MSG_DISCONNECT arrives. The assertion
+/// does not pin a specific russh error variant because the exact error
+/// returned on server-initiated disconnect depends on timing.
+pub async fn test_sftp_idle_timeout_disconnects() -> Result<()> {
+    let env = ProtocolTestEnvironment::new().map_err(|e| anyhow!("{}", e))?;
+    let host_key_dir = PathBuf::from(&env.temp_dir).join("sftp_host_keys");
+    generate_host_key(&host_key_dir).await?;
+
+    info!("Starting SFTP server with idle timeout {} s on {}", IDLE_TIMEOUT_SECS, IDLE_SFTP_ADDRESS);
+    let binary_path = rustfs_binary_path_with_features(Some("ftps,webdav,sftp"));
+    let host_key_dir_str = host_key_dir
+        .to_str()
+        .ok_or_else(|| anyhow!("host key dir path is not utf-8"))?;
+    let mut server_process = ServerProcess::new(
+        Command::new(&binary_path)
+            .env(ENV_SFTP_ENABLE, "true")
+            .env(ENV_SFTP_ADDRESS, IDLE_SFTP_ADDRESS)
+            .env(ENV_SFTP_HOST_KEY_DIR, host_key_dir_str)
+            .env(ENV_SFTP_READ_ONLY, "false")
+            .env(ENV_SFTP_PART_SIZE, PART_SIZE_ENV)
+            .env(ENV_SFTP_IDLE_TIMEOUT, IDLE_TIMEOUT_SECS.to_string())
+            .env(ENV_RUSTFS_ADDRESS, IDLE_S3_ADDRESS)
+            .arg(&env.temp_dir)
+            .spawn()?,
+    );
+
+    let result = async {
+        ProtocolTestEnvironment::wait_for_port_ready(IDLE_SFTP_PORT, 30)
+            .await
+            .map_err(|e| anyhow!("{}", e))?;
+
+        let (session, sftp) = connect_sftp_to(IDLE_SFTP_ADDRESS).await?;
+
+        // Confirm the session is live before the wait so a failure in the
+        // post-wait read can be attributed to the idle timer rather than to
+        // a setup defect.
+        let pwd = sftp.canonicalize(".").await?;
+        assert!(!pwd.is_empty(), "server must resolve '.' to a non-empty absolute path");
+
+        info!("Idle wait: sleeping {} s past idle timeout {} s", IDLE_WAIT_SECS, IDLE_TIMEOUT_SECS);
+        sleep(Duration::from_secs(IDLE_WAIT_SECS)).await;
+
+        let post_idle = sftp.read_dir("/").await;
+        assert!(
+            post_idle.is_err(),
+            "SFTP request after idle wait must error once the server has closed the session"
+        );
+        info!("PASS: SFTP request after idle wait returned an error");
+
+        drop(sftp);
+        // Discard the disconnect Result. The server has already closed the
+        // session via the idle-timeout path the test is probing. A client
+        // disconnect against a half-closed transport may itself return Err
+        // with no useful signal.
+        let _ = session.disconnect(russh::Disconnect::ByApplication, "", "en").await;
+        Ok::<(), anyhow::Error>(())
+    }
+    .await;
+
+    // Discard kill/wait errors on the teardown path: the test result above
+    // is the binding outcome, and a server that has already exited produces
+    // an error here that carries no useful signal.
+    server_process.kill_and_wait().await;
+
+    result
+}

crates/e2e_test/src/protocols/sftp_helpers.rs

@@ -0,0 +1,194 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Shared helpers for SFTP protocol tests
+//!
+//! An accept-any host-key handler, an ed25519 host-key generator that
+//! matches the rustfs config loader permission gates, a russh client
+//! connector that authenticates with the default access key, and an
+//! SFTP-read-to-vec helper.
+
+use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY};
+use anyhow::{Result, anyhow};
+use aws_sdk_s3::Client as S3Client;
+use aws_sdk_s3::config::{Credentials, Region};
+use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
+use russh::client::{self, Handle};
+use russh::keys::ssh_key::LineEnding;
+use russh::keys::{Algorithm, PrivateKey, PublicKey};
+use russh_sftp::client::SftpSession;
+use russh_sftp::protocol::OpenFlags;
+use std::path::Path;
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::io::{AsyncReadExt, AsyncWriteExt};
+use tokio::process::Child;
+use tokio::time::sleep;
+use tracing::info;
+
+/// Accept-any server-key client handler. The test server uses a host key
+/// generated fresh at the start of each run, so strict verification would
+/// always fail. The suite exercises the auth path, not the host-key-trust
+/// path, which is out of scope for this suite.
+pub struct AcceptAnyServerKey;
+
+impl client::Handler for AcceptAnyServerKey {
+    type Error = anyhow::Error;
+
+    async fn check_server_key(&mut self, _server_public_key: &PublicKey) -> Result<bool, Self::Error> {
+        Ok(true)
+    }
+}
+
+/// Generate an ed25519 host key pair in host_key_dir with mode 0600. The
+/// key is generated in-process via russh::keys so the test suite has no
+/// host-tooling dependency on ssh-keygen, which is absent on Alpine,
+/// distroless, scratch, and Windows images. The RustFS config loader
+/// accepts the OpenSSH private-key format that PrivateKey::to_openssh
+/// emits. Both the private key and the .pub file need 0600 because the
+/// loader scans every entry in the directory and rejects the whole
+/// directory as insecure unless each file is 0600 or 0400.
+pub async fn generate_host_key(host_key_dir: &Path) -> Result<()> {
+    tokio::fs::create_dir_all(host_key_dir).await?;
+    let key_path = host_key_dir.join("ssh_host_ed25519_key");
+    let pub_path = host_key_dir.join("ssh_host_ed25519_key.pub");
+
+    let private_key =
+        PrivateKey::random(&mut rand::rng(), Algorithm::Ed25519).map_err(|e| anyhow!("ed25519 key generation failed: {e}"))?;
+    let private_pem = private_key
+        .to_openssh(LineEnding::LF)
+        .map_err(|e| anyhow!("OpenSSH private-key encode failed: {e}"))?;
+    let public_text = private_key
+        .public_key()
+        .to_openssh()
+        .map_err(|e| anyhow!("OpenSSH public-key encode failed: {e}"))?;
+
+    tokio::fs::write(&key_path, private_pem.as_bytes()).await?;
+    tokio::fs::write(&pub_path, format!("{public_text}\n")).await?;
+
+    #[cfg(unix)]
+    {
+        use std::os::unix::fs::PermissionsExt;
+        for path in [&key_path, &pub_path] {
+            let mut perm = std::fs::metadata(path)?.permissions();
+            perm.set_mode(0o600);
+            std::fs::set_permissions(path, perm)?;
+        }
+    }
+    Ok(())
+}
+
+/// Owns a spawned rustfs server child process and guarantees the process
+/// is sent SIGKILL even if the test panics. The wrapper exists because
+/// tokio::process::Child does not kill on Drop on stable Rust, so a
+/// panicking test would otherwise leak a running rustfs binary that
+/// keeps its listener port held until the test runner exits.
+///
+/// Use kill_and_wait on the success and Err paths to reap the child
+/// cleanly; Drop only fires the synchronous SIGKILL when those paths
+/// were skipped (panic unwind, runtime abort).
+pub struct ServerProcess {
+    inner: Option<Child>,
+}
+
+impl ServerProcess {
+    pub fn new(child: Child) -> Self {
+        Self { inner: Some(child) }
+    }
+
+    /// Borrow the inner Child for callers that need stdout piping or
+    /// other tokio::process APIs.
+    pub fn child_mut(&mut self) -> &mut Child {
+        self.inner.as_mut().expect("ServerProcess: child already taken")
+    }
+
+    /// Async kill plus wait. Idempotent. Use on every success and Err
+    /// path. After this returns, Drop becomes a no-op.
+    pub async fn kill_and_wait(&mut self) {
+        if let Some(mut child) = self.inner.take() {
+            let _ = child.kill().await;
+            let _ = child.wait().await;
+        }
+    }
+}
+
+impl Drop for ServerProcess {
+    fn drop(&mut self) {
+        if let Some(child) = self.inner.as_mut() {
+            // Synchronous SIGKILL via the kernel. Runs even on panic
+            // unwind. wait() is skipped here (Drop cannot await), so
+            // the process becomes a zombie reaped by the runtime.
+            let _ = child.start_kill();
+        }
+    }
+}
+
+/// Open a russh client session against the given address, authenticate
+/// with the default access key, request the SFTP subsystem, and return
+/// the session handle plus the SFTP wrapper. The handle is returned so
+/// the caller can keep the underlying SSH transport alive for the full
+/// session and disconnect cleanly afterwards.
+pub async fn connect_sftp_to(address: &str) -> Result<(Handle<AcceptAnyServerKey>, SftpSession)> {
+    let config = Arc::new(client::Config::default());
+    let mut session = client::connect(config, address, AcceptAnyServerKey).await?;
+    let auth = session.authenticate_password(DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY).await?;
+    if !auth.success() {
+        return Err(anyhow!("SFTP password auth rejected"));
+    }
+    let channel = session.channel_open_session().await?;
+    channel.request_subsystem(true, "sftp").await?;
+    let sftp = SftpSession::new(channel.into_stream()).await?;
+    Ok((session, sftp))
+}
+
+/// Read an SFTP object into memory.
+pub async fn sftp_read_full(sftp: &SftpSession, path: &str) -> Result<Vec<u8>> {
+    let mut file = sftp.open_with_flags(path, OpenFlags::READ).await?;
+    let mut buf = Vec::new();
+    file.read_to_end(&mut buf).await?;
+    file.shutdown().await?;
+    Ok(buf)
+}
+
+/// Construct an aws-sdk-s3 client wired against an http rustfs endpoint.
+/// Uses the same credential constants the SFTP session authenticates with so
+/// both protocols see the same backend identity.
+pub fn build_test_s3_client(endpoint_url: &str) -> S3Client {
+    let credentials = Credentials::new(DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, None, None, "sftp-helpers");
+    let mut config = aws_sdk_s3::Config::builder()
+        .credentials_provider(credentials)
+        .region(Region::new("us-east-1"))
+        .endpoint_url(endpoint_url)
+        .force_path_style(true)
+        .behavior_version_latest();
+    if endpoint_url.starts_with("http://") {
+        config = config.http_client(SmithyHttpClientBuilder::new().build_http());
+    }
+    S3Client::from_conf(config.build())
+}
+
+/// Poll the S3 endpoint until ListBuckets returns successfully or the
+/// attempt budget is exhausted. The TCP-level wait_for_port_ready check is
+/// not enough on its own because rustfs accepts connections before the S3
+/// stack has finished initialising.
+pub async fn wait_for_s3_ready(client: &S3Client, max_attempts: u32) -> Result<()> {
+    for attempt in 0..max_attempts {
+        if client.list_buckets().send().await.is_ok() {
+            info!("S3 endpoint ready after {} attempts", attempt + 1);
+            return Ok(());
+        }
+        sleep(Duration::from_secs(1)).await;
+    }
+    Err(anyhow!("S3 endpoint did not become ready"))
+}

crates/e2e_test/src/protocols/test_env.rs

@@ -32,8 +32,13 @@ impl ProtocolTestEnvironment {
     /// Create a new test environment
     /// This environment won't stop any server when dropped
     pub fn new() -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
-        let temp_dir = format!("/tmp/rustfs_protocol_test_{}", uuid::Uuid::new_v4());
-        std::fs::create_dir_all(&temp_dir)?;
+        let mut path = std::env::temp_dir();
+        path.push(format!("rustfs_protocol_test_{}", uuid::Uuid::new_v4()));
+        std::fs::create_dir_all(&path)?;
+        let temp_dir = path
+            .to_str()
+            .ok_or_else(|| format!("temp dir path is not utf-8: {}", path.display()))?
+            .to_string();
 
         Ok(Self { temp_dir })
     }

crates/e2e_test/src/protocols/test_runner.rs

@@ -15,8 +15,14 @@
 //! Protocol test runner
 
 use crate::common::init_logging;
+use crate::common::{requested_rustfs_build_features, rustfs_build_feature_enabled};
 use crate::protocols::ftps_core::test_ftps_core_operations;
+use crate::protocols::sftp_compliance::{
+    test_sftp_compliance_readonly, test_sftp_compliance_standalone, test_sftp_compliance_suite,
+};
+use crate::protocols::sftp_core::{test_sftp_core_operations, test_sftp_idle_timeout_disconnects};
 use crate::protocols::webdav_core::test_webdav_core_operations;
+use serial_test::serial;
 use std::time::Instant;
 use tokio::time::{Duration, sleep};
 use tracing::{error, info};
@@ -54,21 +60,22 @@ pub struct ProtocolTestSuite {
 
 #[derive(Debug, Clone)]
 struct TestDefinition {
-    name: String,
+    name: &'static str,
+    required_feature: &'static str,
 }
 
 impl ProtocolTestSuite {
     /// Create default test suite
     pub fn new() -> Self {
-        let tests = vec![
-            TestDefinition {
-                name: "test_ftps_core_operations".to_string(),
-            },
-            TestDefinition {
-                name: "test_webdav_core_operations".to_string(),
-            },
-        ];
+        let requested_features = requested_rustfs_build_features();
+        Self::with_requested_features(requested_features.as_deref())
+    }
 
+    fn with_requested_features(requested_features: Option<&str>) -> Self {
+        let tests = all_protocol_tests()
+            .into_iter()
+            .filter(|test| rustfs_build_feature_enabled(requested_features, test.required_feature))
+            .collect();
         Self { tests }
     }
 
@@ -84,7 +91,7 @@ impl ProtocolTestSuite {
 
         // Run tests
         for (i, test_def) in self.tests.iter().enumerate() {
-            let test_description = match test_def.name.as_str() {
+            let test_description = match test_def.name {
                 "test_ftps_core_operations" => {
                     info!("=== Starting FTPS Module Test ===");
                     "FTPS core operations (put, ls, mkdir, rmdir, delete)"
@@ -93,6 +100,26 @@ impl ProtocolTestSuite {
                     info!("=== Starting WebDAV Core Test ===");
                     "WebDAV core operations (MKCOL, PUT, GET, DELETE, PROPFIND)"
                 }
+                "test_sftp_core_operations" => {
+                    info!("=== Starting SFTP Core Test ===");
+                    "SFTP core operations (banner, mkdir, put, get with SHA compare, rename, delete, rmdir)"
+                }
+                "test_sftp_compliance_suite" => {
+                    info!("=== Starting SFTP Compliance Suite ===");
+                    "SFTP compliance regression suite (zero-byte, mutation rejection, traversal, rename, implicit dirs, FSETSTAT)"
+                }
+                "test_sftp_compliance_readonly" => {
+                    info!("=== Starting SFTP Read-Only Compliance Suite ===");
+                    "SFTP read-only mode (RUSTFS_SFTP_READ_ONLY=true rejects mutations and allows reads)"
+                }
+                "test_sftp_idle_timeout_disconnects" => {
+                    info!("=== Starting SFTP Idle-Timeout Test ===");
+                    "SFTP idle-timeout disconnects (server closes the session past RUSTFS_SFTP_IDLE_TIMEOUT)"
+                }
+                "test_sftp_compliance_standalone" => {
+                    info!("=== Starting SFTP Standalone-Server Compliance Suite ===");
+                    "SFTP standalone-server compliance suite"
+                }
                 _ => "",
             };
 
@@ -107,11 +134,11 @@ impl ProtocolTestSuite {
             match result {
                 Ok(_) => {
                     info!("Test passed: {} ({:.2}s)", test_def.name, test_duration.as_secs_f64());
-                    results.push(TestResult::success(test_def.name.clone()));
+                    results.push(TestResult::success(test_def.name.to_string()));
                 }
                 Err(e) => {
                     error!("Test failed: {} ({:.2}s): {}", test_def.name, test_duration.as_secs_f64(), e);
-                    results.push(TestResult::failure(test_def.name.clone(), e.to_string()));
+                    results.push(TestResult::failure(test_def.name.to_string(), e.to_string()));
                 }
             }
 
@@ -129,9 +156,14 @@ impl ProtocolTestSuite {
 
     /// Run a single test
     async fn run_single_test(&self, test_def: &TestDefinition) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        match test_def.name.as_str() {
+        match test_def.name {
             "test_ftps_core_operations" => test_ftps_core_operations().await.map_err(|e| e.into()),
             "test_webdav_core_operations" => test_webdav_core_operations().await.map_err(|e| e.into()),
+            "test_sftp_core_operations" => test_sftp_core_operations().await.map_err(|e| e.into()),
+            "test_sftp_compliance_suite" => test_sftp_compliance_suite().await.map_err(|e| e.into()),
+            "test_sftp_compliance_readonly" => test_sftp_compliance_readonly().await.map_err(|e| e.into()),
+            "test_sftp_idle_timeout_disconnects" => test_sftp_idle_timeout_disconnects().await.map_err(|e| e.into()),
+            "test_sftp_compliance_standalone" => test_sftp_compliance_standalone().await.map_err(|e| e.into()),
             _ => Err(format!("Test {} not implemented", test_def.name).into()),
         }
     }
@@ -141,6 +173,10 @@ impl ProtocolTestSuite {
         info!("=== Test Suite Summary ===");
         info!("Total duration: {:.2}s", total_duration.as_secs_f64());
         info!("Total tests: {}", results.len());
+        if results.is_empty() {
+            info!("No protocol tests scheduled for the requested feature set");
+            return;
+        }
 
         let passed = results.iter().filter(|r| r.success).count();
         let failed = results.len() - passed;
@@ -158,8 +194,42 @@ impl ProtocolTestSuite {
     }
 }
 
+fn all_protocol_tests() -> Vec<TestDefinition> {
+    vec![
+        TestDefinition {
+            name: "test_ftps_core_operations",
+            required_feature: "ftps",
+        },
+        TestDefinition {
+            name: "test_webdav_core_operations",
+            required_feature: "webdav",
+        },
+        TestDefinition {
+            name: "test_sftp_core_operations",
+            required_feature: "sftp",
+        },
+        TestDefinition {
+            name: "test_sftp_compliance_suite",
+            required_feature: "sftp",
+        },
+        TestDefinition {
+            name: "test_sftp_compliance_readonly",
+            required_feature: "sftp",
+        },
+        TestDefinition {
+            name: "test_sftp_idle_timeout_disconnects",
+            required_feature: "sftp",
+        },
+        TestDefinition {
+            name: "test_sftp_compliance_standalone",
+            required_feature: "sftp",
+        },
+    ]
+}
+
 /// Test suite
 #[tokio::test]
+#[serial]
 async fn test_protocol_core_suite() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     let suite = ProtocolTestSuite::new();
     let results = suite.run_test_suite().await;
@@ -172,3 +242,72 @@ async fn test_protocol_core_suite() -> Result<(), Box<dyn std::error::Error + Se
     info!("All protocol tests passed");
     Ok(())
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn scheduled_names(suite: ProtocolTestSuite) -> Vec<&'static str> {
+        suite.tests.into_iter().map(|test| test.name).collect()
+    }
+
+    #[test]
+    fn schedules_all_protocol_tests_without_feature_filter() {
+        let names = scheduled_names(ProtocolTestSuite::with_requested_features(None));
+
+        assert_eq!(names.len(), 7);
+        assert!(names.contains(&"test_ftps_core_operations"));
+        assert!(names.contains(&"test_webdav_core_operations"));
+        assert!(names.contains(&"test_sftp_core_operations"));
+        assert!(names.contains(&"test_sftp_compliance_standalone"));
+    }
+
+    #[test]
+    fn schedules_only_requested_non_sftp_protocols() {
+        let names = scheduled_names(ProtocolTestSuite::with_requested_features(Some("ftps, webdav")));
+
+        assert_eq!(names, vec!["test_ftps_core_operations", "test_webdav_core_operations"]);
+    }
+
+    #[test]
+    fn schedules_all_sftp_entries_for_sftp_feature() {
+        let names = scheduled_names(ProtocolTestSuite::with_requested_features(Some("sftp")));
+
+        assert_eq!(
+            names,
+            vec![
+                "test_sftp_core_operations",
+                "test_sftp_compliance_suite",
+                "test_sftp_compliance_readonly",
+                "test_sftp_idle_timeout_disconnects",
+                "test_sftp_compliance_standalone",
+            ]
+        );
+    }
+
+    #[test]
+    fn feature_filter_is_case_insensitive() {
+        let names = scheduled_names(ProtocolTestSuite::with_requested_features(Some("SFTP")));
+
+        assert_eq!(names.len(), 5);
+        assert!(names.iter().all(|name| name.contains("sftp")));
+    }
+
+    #[test]
+    fn full_feature_schedules_all_protocol_tests() {
+        let names = scheduled_names(ProtocolTestSuite::with_requested_features(Some("full")));
+
+        assert_eq!(names.len(), 7);
+        assert!(names.contains(&"test_ftps_core_operations"));
+        assert!(names.contains(&"test_webdav_core_operations"));
+        assert!(names.contains(&"test_sftp_core_operations"));
+        assert!(names.contains(&"test_sftp_compliance_standalone"));
+    }
+
+    #[test]
+    fn schedules_no_tests_when_requested_features_have_no_protocols() {
+        let names = scheduled_names(ProtocolTestSuite::with_requested_features(Some("diagnostics")));
+
+        assert!(names.is_empty());
+    }
+}

crates/e2e_test/src/protocols/webdav_core.rs

@@ -14,17 +14,24 @@
 
 //! Core WebDAV tests
 
+use crate::common::local_http_client;
 use crate::common::rustfs_binary_path_with_features;
 use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, ProtocolTestEnvironment};
 use anyhow::Result;
 use base64::Engine;
+use http::header::{CONTENT_TYPE, HOST};
 use reqwest::Client;
+use rustfs_signer::constants::UNSIGNED_PAYLOAD;
+use rustfs_signer::sign_v4;
+use s3s::Body;
+use serial_test::serial;
 use tokio::process::Command;
 use tracing::info;
 
 // Fixed WebDAV port for testing
 const WEBDAV_PORT: u16 = 9080;
 const WEBDAV_ADDRESS: &str = "127.0.0.1:9080";
+const S3_TEST_ADDRESS: &str = "127.0.0.1:9010";
 
 /// Create HTTP client with basic auth
 fn create_client() -> Client {
@@ -36,19 +43,114 @@ fn create_client() -> Client {
 
 /// Get basic auth header value
 fn basic_auth_header() -> String {
-    let credentials = format!("{}:{}", DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY);
+    basic_auth_header_for(DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY)
+}
+
+fn basic_auth_header_for(access_key: &str, secret_key: &str) -> String {
+    let credentials = format!("{}:{}", access_key, secret_key);
     let encoded = base64::engine::general_purpose::STANDARD.encode(credentials);
     format!("Basic {}", encoded)
 }
 
+async fn signed_admin_request(
+    method: http::Method,
+    url: &str,
+    body: Option<Vec<u8>>,
+    content_type: Option<&str>,
+) -> Result<reqwest::Response> {
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri
+        .authority()
+        .ok_or_else(|| anyhow::anyhow!("request URL missing authority"))?
+        .to_string();
+    let mut request = http::Request::builder().method(method.clone()).uri(uri);
+    request = request.header(HOST, authority);
+    request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
+    if let Some(content_type) = content_type {
+        request = request.header(CONTENT_TYPE, content_type);
+    }
+
+    let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default();
+    let signed = sign_v4(
+        request.body(Body::empty())?,
+        content_len,
+        DEFAULT_ACCESS_KEY,
+        DEFAULT_SECRET_KEY,
+        "",
+        "us-east-1",
+    );
+
+    let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
+    let mut request_builder = local_http_client().request(reqwest_method, url);
+    for (name, value) in signed.headers() {
+        request_builder = request_builder.header(name, value);
+    }
+    if let Some(body) = body {
+        request_builder = request_builder.body(body);
+    }
+
+    Ok(request_builder.send().await?)
+}
+
+async fn admin_create_user(base_url: &str, username: &str, secret_key: &str) -> Result<()> {
+    let url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", base_url, username);
+    let body = serde_json::json!({
+        "secretKey": secret_key,
+        "status": "enabled"
+    });
+    let response =
+        signed_admin_request(http::Method::PUT, &url, Some(body.to_string().into_bytes()), Some("application/json")).await?;
+
+    if response.status() != reqwest::StatusCode::OK {
+        let status = response.status();
+        let body = response.text().await.unwrap_or_default();
+        anyhow::bail!("create user failed: {status} {body}");
+    }
+
+    Ok(())
+}
+
+async fn admin_add_canned_policy(base_url: &str, policy_name: &str, policy: &serde_json::Value) -> Result<()> {
+    let url = format!("{}/rustfs/admin/v3/add-canned-policy?name={}", base_url, policy_name);
+    let response =
+        signed_admin_request(http::Method::PUT, &url, Some(policy.to_string().into_bytes()), Some("application/json")).await?;
+
+    if response.status() != reqwest::StatusCode::OK {
+        let status = response.status();
+        let body = response.text().await.unwrap_or_default();
+        anyhow::bail!("add canned policy failed: {status} {body}");
+    }
+
+    Ok(())
+}
+
+async fn admin_attach_policy_to_user(base_url: &str, policy_name: &str, username: &str) -> Result<()> {
+    let url = format!(
+        "{}/rustfs/admin/v3/set-user-or-group-policy?policyName={}&userOrGroup={}&isGroup=false",
+        base_url, policy_name, username
+    );
+    let response = signed_admin_request(http::Method::PUT, &url, Some(Vec::new()), None).await?;
+
+    if response.status() != reqwest::StatusCode::OK {
+        let status = response.status();
+        let body = response.text().await.unwrap_or_default();
+        anyhow::bail!("attach policy failed: {status} {body}");
+    }
+
+    Ok(())
+}
+
 /// Test WebDAV: MKCOL (create bucket), PUT, GET, DELETE, PROPFIND operations
 pub async fn test_webdav_core_operations() -> Result<()> {
     let env = ProtocolTestEnvironment::new().map_err(|e| anyhow::anyhow!("{}", e))?;
+    let admin_base_url = format!("http://{}", S3_TEST_ADDRESS);
 
     // Start server manually
     info!("Starting WebDAV server on {}", WEBDAV_ADDRESS);
-    let binary_path = rustfs_binary_path_with_features(Some("ftps,webdav"));
+    let binary_path = rustfs_binary_path_with_features(Some("webdav"));
     let mut server_process = Command::new(&binary_path)
+        .arg("--address")
+        .arg(S3_TEST_ADDRESS)
         .env("RUSTFS_WEBDAV_ENABLE", "true")
         .env("RUSTFS_WEBDAV_ADDRESS", WEBDAV_ADDRESS)
         .env("RUSTFS_WEBDAV_TLS_ENABLED", "false") // No TLS for testing
@@ -170,6 +272,372 @@ pub async fn test_webdav_core_operations() -> Result<()> {
         );
         info!("PASS: Verified file '{}' is deleted", filename);
 
+        // Test MOVE (rename) file
+        info!("Testing WebDAV: PUT file for MOVE test");
+        let move_filename = "move-source.txt";
+        let move_dest_filename = "move-dest.txt";
+        let move_content = "File to be moved!";
+        let resp = client
+            .put(format!("{}/{}/{}", base_url, bucket_name, move_filename))
+            .header("Authorization", &auth_header)
+            .body(move_content)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "PUT for MOVE test should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: PUT file '{}' for MOVE test successful", move_filename);
+
+        // Execute MOVE request
+        info!("Testing WebDAV: MOVE file '{}' to '{}'", move_filename, move_dest_filename);
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MOVE").unwrap(),
+                format!("{}/{}/{}", base_url, bucket_name, move_filename),
+            )
+            .header("Authorization", &auth_header)
+            .header("Destination", format!("/{}/{}", bucket_name, move_dest_filename))
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 204 || resp.status().as_u16() == 201,
+            "MOVE should succeed, got: {}",
+            resp.status()
+        );
+        info!(
+            "PASS: MOVE file '{}' to '{}' successful (HTTP {})",
+            move_filename,
+            move_dest_filename,
+            resp.status()
+        );
+
+        // Verify source file is gone
+        info!("Testing WebDAV: Verify source '{}' is deleted after MOVE", move_filename);
+        let resp = client
+            .get(format!("{}/{}/{}", base_url, bucket_name, move_filename))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().as_u16() == 404,
+            "GET moved source should return 404, got: {}",
+            resp.status()
+        );
+        info!("PASS: Source '{}' is deleted after MOVE", move_filename);
+
+        // Verify destination file exists and content matches
+        info!("Testing WebDAV: Verify destination '{}' has correct content", move_dest_filename);
+        let resp = client
+            .get(format!("{}/{}/{}", base_url, bucket_name, move_dest_filename))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success(),
+            "GET destination after MOVE should succeed, got: {}",
+            resp.status()
+        );
+        let moved_content = resp.text().await?;
+        assert_eq!(moved_content, move_content, "Moved file content should match original");
+        info!("PASS: Destination '{}' has correct content after MOVE", move_dest_filename);
+
+        // Test directory creation and rename
+        info!("Testing WebDAV: MKCOL directory");
+        let dir_name = "test-directory";
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MKCOL").unwrap(),
+                format!("{}/{}/{}", base_url, bucket_name, dir_name),
+            )
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "MKCOL directory should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: MKCOL directory successful");
+
+        // Upload file into directory
+        let dir_filename = "dir-file.txt";
+        let dir_file_content = "File inside test directory!";
+        let resp = client
+            .put(format!("{}/{}/{}/{}", base_url, bucket_name, dir_name, dir_filename))
+            .header("Authorization", &auth_header)
+            .body(dir_file_content)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "PUT file into directory should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: PUT file into directory successful");
+
+        // Test PROPFIND on directory
+        info!("Testing WebDAV: PROPFIND directory");
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"PROPFIND").unwrap(),
+                format!("{}/{}/{}", base_url, bucket_name, dir_name),
+            )
+            .header("Authorization", &auth_header)
+            .header("Depth", "1")
+            .send()
+            .await?;
+        assert!(resp.status().is_success(), "PROPFIND directory should succeed, got: {}", resp.status());
+        let propfind_body = resp.text().await?;
+        assert!(propfind_body.contains(dir_filename), "PROPFIND should list file in directory");
+        info!("PASS: PROPFIND directory successful, file listed correctly");
+
+        // Current WebDAV support exposes collection listing via PROPFIND; GET on a collection is not implemented.
+        info!("Testing WebDAV: GET directory listing fallback behavior");
+        let resp = client
+            .get(format!("{}/{}/{}", base_url, bucket_name, dir_name))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert_eq!(
+            resp.status().as_u16(),
+            405,
+            "GET on a WebDAV collection should currently return 405, got: {}",
+            resp.status()
+        );
+        info!("PASS: GET collection correctly returned 405; PROPFIND remains the listing path");
+
+        // Rename directory
+        info!("Testing WebDAV: MOVE directory");
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MOVE").unwrap(),
+                format!("{}/{}/{}", base_url, bucket_name, dir_name),
+            )
+            .header("Authorization", &auth_header)
+            .header("Destination", format!("/{}/renamed-dir", bucket_name))
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 204 || resp.status().as_u16() == 201,
+            "MOVE directory should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: MOVE directory successful");
+
+        // Verify source directory is gone
+        info!("Testing WebDAV: Verify source directory is deleted after MOVE");
+        let resp = client
+            .get(format!("{}/{}/{}", base_url, bucket_name, dir_name))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().as_u16() == 404,
+            "GET moved directory should return 404, got: {}",
+            resp.status()
+        );
+        info!("PASS: Source directory is deleted after MOVE");
+
+        // Verify renamed directory exists and file content matches
+        info!("Testing WebDAV: Verify file in renamed directory");
+        let resp = client
+            .get(format!("{}/{}/renamed-dir/{}", base_url, bucket_name, dir_filename))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success(),
+            "GET file in renamed directory should succeed, got: {}",
+            resp.status()
+        );
+        let renamed_dir_content = resp.text().await?;
+        assert_eq!(renamed_dir_content, dir_file_content, "File content in renamed directory should match");
+        info!("PASS: File in renamed directory has correct content");
+
+        // Test nested directory creation and rename
+        info!("Testing WebDAV: MKCOL nested directory");
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MKCOL").unwrap(),
+                format!("{}/{}/renamed-dir/nested-dir", base_url, bucket_name),
+            )
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "MKCOL nested directory should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: MKCOL nested directory successful");
+
+        // Upload file into nested directory
+        let nested_file_content = "File in nested directory!";
+        let resp = client
+            .put(format!("{}/{}/renamed-dir/nested-dir/nested-file.txt", base_url, bucket_name))
+            .header("Authorization", &auth_header)
+            .body(nested_file_content)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "PUT file into nested directory should succeed, got: {}",
+            resp.status()
+        );
+
+        // Rename nested directory
+        info!("Testing WebDAV: MOVE nested directory");
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MOVE").unwrap(),
+                format!("{}/{}/renamed-dir/nested-dir", base_url, bucket_name),
+            )
+            .header("Authorization", &auth_header)
+            .header("Destination", format!("/{}/renamed-dir/new-nested-dir", bucket_name))
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 204 || resp.status().as_u16() == 201,
+            "MOVE nested directory should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: MOVE nested directory successful");
+
+        // Verify nested file after rename
+        let resp = client
+            .get(format!("{}/{}/renamed-dir/new-nested-dir/nested-file.txt", base_url, bucket_name))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success(),
+            "GET file in renamed nested directory should succeed, got: {}",
+            resp.status()
+        );
+        let nested_content = resp.text().await?;
+        assert_eq!(
+            nested_content, nested_file_content,
+            "File content in renamed nested directory should match"
+        );
+        info!("PASS: File in renamed nested directory has correct content");
+
+        // Test directory MOVE authz failure does not create partial destination writes
+        info!("Testing WebDAV: directory MOVE denied by missing DeleteObject must not create partial writes");
+        let restricted_bucket = "webdav-authz-bucket";
+        let restricted_dir = "restricted-src";
+        let restricted_dst = "restricted-dst";
+        let restricted_file = "locked.txt";
+        let restricted_content = "must remain only at source";
+        let restricted_user = "webdav-limited-user";
+        let restricted_secret = "webdav-limited-secret";
+        let restricted_policy_name = "webdav-move-no-delete";
+
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MKCOL").unwrap(),
+                format!("{}/{}", base_url, restricted_bucket),
+            )
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "MKCOL restricted bucket should succeed, got: {}",
+            resp.status()
+        );
+
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MKCOL").unwrap(),
+                format!("{}/{}/{}", base_url, restricted_bucket, restricted_dir),
+            )
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "MKCOL restricted directory should succeed, got: {}",
+            resp.status()
+        );
+
+        let resp = client
+            .put(format!("{}/{}/{}/{}", base_url, restricted_bucket, restricted_dir, restricted_file))
+            .header("Authorization", &auth_header)
+            .body(restricted_content)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "PUT restricted file should succeed, got: {}",
+            resp.status()
+        );
+
+        admin_create_user(&admin_base_url, restricted_user, restricted_secret).await?;
+        admin_add_canned_policy(
+            &admin_base_url,
+            restricted_policy_name,
+            &serde_json::json!({
+                "Version": "2012-10-17",
+                "Statement": [
+                    {
+                        "Effect": "Allow",
+                        "Action": ["s3:ListBucket"],
+                        "Resource": [format!("arn:aws:s3:::{}", restricted_bucket)]
+                    },
+                    {
+                        "Effect": "Allow",
+                        "Action": ["s3:GetObject", "s3:PutObject"],
+                        "Resource": [format!("arn:aws:s3:::{}/*", restricted_bucket)]
+                    }
+                ]
+            }),
+        )
+        .await?;
+        admin_attach_policy_to_user(&admin_base_url, restricted_policy_name, restricted_user).await?;
+
+        let restricted_auth = basic_auth_header_for(restricted_user, restricted_secret);
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MOVE").unwrap(),
+                format!("{}/{}/{}", base_url, restricted_bucket, restricted_dir),
+            )
+            .header("Authorization", &restricted_auth)
+            .header("Destination", format!("/{}/{}", restricted_bucket, restricted_dst))
+            .send()
+            .await?;
+        assert!(
+            !resp.status().is_success(),
+            "MOVE without DeleteObject permission should be rejected, got: {}",
+            resp.status()
+        );
+
+        let resp = client
+            .get(format!("{}/{}/{}/{}", base_url, restricted_bucket, restricted_dst, restricted_file))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert_eq!(
+            resp.status().as_u16(),
+            404,
+            "Denied MOVE must not create destination object, got: {}",
+            resp.status()
+        );
+
+        let resp = client
+            .get(format!("{}/{}/{}/{}", base_url, restricted_bucket, restricted_dir, restricted_file))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success(),
+            "Source object should remain after denied MOVE, got: {}",
+            resp.status()
+        );
+        assert_eq!(resp.text().await?, restricted_content, "Denied MOVE must leave source content untouched");
+        info!("PASS: denied directory MOVE left source intact and created no destination objects");
+
         // Test DELETE bucket
         info!("Testing WebDAV: DELETE bucket '{}'", bucket_name);
         let resp = client
@@ -205,3 +673,9 @@ pub async fn test_webdav_core_operations() -> Result<()> {
 
     result
 }
+
+#[tokio::test]
+#[serial]
+async fn test_webdav_core_operations_direct() -> Result<()> {
+    test_webdav_core_operations().await
+}

crates/ecstore/Cargo.toml

@@ -44,6 +44,7 @@ rustfs-credentials = { workspace = true }
 rustfs-common.workspace = true
 rustfs-policy.workspace = true
 rustfs-protos.workspace = true
+rustfs-kms.workspace = true
 rustfs-s3-common = { workspace = true }
 async-trait.workspace = true
 bytes.workspace = true
@@ -104,6 +105,7 @@ rustix = { workspace = true }
 rustfs-madmin.workspace = true
 rustfs-workers.workspace = true
 reqwest = { workspace = true }
+aes-gcm.workspace = true
 aws-sdk-s3 = { workspace = true }
 urlencoding = { workspace = true }
 smallvec = { workspace = true }

crates/ecstore/src/bucket/lifecycle/core.rs

@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use rustfs_config::{DEFAULT_ILM_PROCESS_TIME_SECS, ENV_ILM_PROCESS_TIME, ENV_ILM_PROCESS_TIME_DEPRECATED};
 use rustfs_filemeta::{ReplicationStatusType, VersionPurgeStatusType};
 use s3s::dto::{
     BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter,
@@ -19,7 +20,6 @@ use s3s::dto::{
 };
 use std::cmp::Ordering;
 use std::collections::HashMap;
-use std::env;
 use std::sync::Arc;
 use time::macros::offset;
 use time::{self, Duration, OffsetDateTime};
@@ -805,16 +805,29 @@ pub fn expected_expiry_time(mod_time: OffsetDateTime, days: i32) -> OffsetDateTi
         .to_offset(offset!(-0:00:00))
         .saturating_add(Duration::days(days as i64));
 
-    // Truncate to midnight UTC per S3 standard, unless overridden by env var.
-    // _RUSTFS_ILM_PROCESS_TIME controls the truncation granularity in seconds.
-    let truncation_secs = env::var("_RUSTFS_ILM_PROCESS_TIME")
-        .ok()
-        .and_then(|v| v.parse::<u32>().ok())
-        .unwrap_or(86400); // default: truncate to midnight (24h)
+    // Round up to the next processing boundary per S3-compatible Days semantics.
+    // Canonical key: RUSTFS_ILM_PROCESS_TIME; deprecated alias: _RUSTFS_ILM_PROCESS_TIME.
+    // TODO(GA): Remove ENV_ILM_PROCESS_TIME_DEPRECATED compatibility during GA release.
+    let process_interval_secs = rustfs_utils::get_env_i32_with_aliases(
+        ENV_ILM_PROCESS_TIME,
+        &[ENV_ILM_PROCESS_TIME_DEPRECATED],
+        DEFAULT_ILM_PROCESS_TIME_SECS,
+    );
+    let process_interval_secs = if process_interval_secs > 0 {
+        process_interval_secs as u32
+    } else {
+        DEFAULT_ILM_PROCESS_TIME_SECS as u32
+    };
 
-    let unix_secs = t.unix_timestamp();
-    let truncated_secs = (unix_secs / truncation_secs as i64) * truncation_secs as i64;
-    OffsetDateTime::from_unix_timestamp(truncated_secs).unwrap_or(t)
+    let boundary_nanos = i128::from(process_interval_secs) * 1_000_000_000;
+    let timestamp_nanos = t.unix_timestamp_nanos();
+    let remainder = timestamp_nanos.rem_euclid(boundary_nanos);
+    let rounded_nanos = if remainder == 0 {
+        timestamp_nanos
+    } else {
+        timestamp_nanos + (boundary_nanos - remainder)
+    };
+    OffsetDateTime::from_unix_timestamp_nanos(rounded_nanos).unwrap_or(t)
 }
 
 pub async fn abort_incomplete_multipart_upload_due(
@@ -945,6 +958,12 @@ mod tests {
     use std::sync::Arc;
     use time::macros::datetime;
 
+    fn with_default_ilm_process_time(test: impl FnOnce()) {
+        temp_env::with_var_unset(ENV_ILM_PROCESS_TIME, || {
+            temp_env::with_var_unset(ENV_ILM_PROCESS_TIME_DEPRECATED, test);
+        });
+    }
+
     #[tokio::test]
     #[serial]
     async fn validate_accepts_zero_expiration_days() {
@@ -1336,7 +1355,7 @@ mod tests {
     #[tokio::test]
     #[serial]
     async fn eval_inner_expires_latest_object_after_days_due() {
-        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let base_time = datetime!(2025-01-15 10:30:45 UTC);
         let lc = BucketLifecycleConfiguration {
             expiry_updated_at: None,
             rules: vec![LifecycleRule {
@@ -1362,11 +1381,11 @@ mod tests {
             is_latest: true,
             ..Default::default()
         };
-        let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
+        let event = lc.eval_inner(&opts, datetime!(2025-01-17 00:00:00 UTC), 0).await;
 
         assert_eq!(event.action, IlmAction::DeleteAction);
         assert_eq!(event.rule_id, "expire-days");
-        assert_eq!(event.due, Some(expected_expiry_time(base_time, 1)));
+        assert_eq!(event.due, Some(datetime!(2025-01-17 00:00:00 UTC)));
     }
 
     #[tokio::test]
@@ -2135,40 +2154,98 @@ mod tests {
             .expect("expected days-based expiration to pass on locked bucket");
     }
 
-    // --- TASK-003 tests: Midnight UTC truncation ---
+    // --- TASK-003 tests: Round up to next UTC processing boundary ---
 
     #[test]
-    fn expected_expiry_time_truncates_to_midnight_utc() {
-        // Object created at 2025-01-15T10:30:45Z, expire in 30 days
-        let mod_time = datetime!(2025-01-15 10:30:45 UTC);
-        let result = expected_expiry_time(mod_time, 30);
+    #[serial]
+    fn expected_expiry_time_rounds_up_to_next_midnight_utc() {
+        with_default_ilm_process_time(|| {
+            // Object created at 2025-01-15T10:30:45Z, expire in 30 days
+            let mod_time = datetime!(2025-01-15 10:30:45 UTC);
+            let result = expected_expiry_time(mod_time, 30);
 
-        // Should be truncated to midnight: 2025-02-14T00:00:00Z
-        assert_eq!(result.hour(), 0);
-        assert_eq!(result.minute(), 0);
-        assert_eq!(result.second(), 0);
-        assert_eq!(result, datetime!(2025-02-14 00:00:00 UTC));
+            // Should round up to the next midnight: 2025-02-15T00:00:00Z
+            assert_eq!(result.hour(), 0);
+            assert_eq!(result.minute(), 0);
+            assert_eq!(result.second(), 0);
+            assert_eq!(result, datetime!(2025-02-15 00:00:00 UTC));
+        });
     }
 
     #[test]
+    #[serial]
     fn expected_expiry_time_immediate_expiry_returns_epoch() {
-        let mod_time = datetime!(2025-06-01 12:00:00 UTC);
-        let result = expected_expiry_time(mod_time, 0);
-        assert_eq!(result, OffsetDateTime::UNIX_EPOCH);
+        with_default_ilm_process_time(|| {
+            let mod_time = datetime!(2025-06-01 12:00:00 UTC);
+            let result = expected_expiry_time(mod_time, 0);
+            assert_eq!(result, OffsetDateTime::UNIX_EPOCH);
+        });
     }
 
     #[test]
-    fn expected_expiry_time_truncates_already_midnight() {
-        let mod_time = datetime!(2025-03-01 00:00:00 UTC);
-        let result = expected_expiry_time(mod_time, 1);
-        assert_eq!(result, datetime!(2025-03-02 00:00:00 UTC));
+    #[serial]
+    fn expected_expiry_time_preserves_exact_midnight_boundary() {
+        with_default_ilm_process_time(|| {
+            let mod_time = datetime!(2025-03-01 00:00:00 UTC);
+            let result = expected_expiry_time(mod_time, 1);
+            assert_eq!(result, datetime!(2025-03-02 00:00:00 UTC));
+        });
     }
 
     #[test]
-    fn expected_expiry_time_truncates_end_of_day() {
-        let mod_time = datetime!(2025-06-15 23:59:59 UTC);
-        let result = expected_expiry_time(mod_time, 1);
-        assert_eq!(result, datetime!(2025-06-16 00:00:00 UTC));
+    #[serial]
+    fn expected_expiry_time_rounds_end_of_day_to_following_midnight() {
+        with_default_ilm_process_time(|| {
+            let mod_time = datetime!(2025-06-15 23:59:59 UTC);
+            let result = expected_expiry_time(mod_time, 1);
+            assert_eq!(result, datetime!(2025-06-17 00:00:00 UTC));
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn expected_expiry_time_uses_canonical_process_time_boundary() {
+        let mod_time = datetime!(2025-01-15 10:30:45 UTC);
+
+        temp_env::with_var(ENV_ILM_PROCESS_TIME, Some("3600"), || {
+            temp_env::with_var_unset(ENV_ILM_PROCESS_TIME_DEPRECATED, || {
+                let result = expected_expiry_time(mod_time, 1);
+                assert_eq!(result, datetime!(2025-01-16 11:00:00 UTC));
+            });
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn expected_expiry_time_uses_deprecated_process_time_alias() {
+        let mod_time = datetime!(2025-01-15 10:30:45 UTC);
+
+        temp_env::with_var_unset(ENV_ILM_PROCESS_TIME, || {
+            temp_env::with_var(ENV_ILM_PROCESS_TIME_DEPRECATED, Some("3600"), || {
+                let result = expected_expiry_time(mod_time, 1);
+                assert_eq!(result, datetime!(2025-01-16 11:00:00 UTC));
+            });
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn expected_expiry_time_uses_default_boundary_when_process_time_is_zero_or_invalid() {
+        let mod_time = datetime!(2025-01-15 10:30:45 UTC);
+
+        temp_env::with_var(ENV_ILM_PROCESS_TIME, Some("0"), || {
+            temp_env::with_var_unset(ENV_ILM_PROCESS_TIME_DEPRECATED, || {
+                let result = expected_expiry_time(mod_time, 30);
+                assert_eq!(result, datetime!(2025-02-15 00:00:00 UTC));
+            });
+        });
+
+        temp_env::with_var(ENV_ILM_PROCESS_TIME, Some("not-a-number"), || {
+            temp_env::with_var_unset(ENV_ILM_PROCESS_TIME_DEPRECATED, || {
+                let result = expected_expiry_time(mod_time, 30);
+                assert_eq!(result, datetime!(2025-02-15 00:00:00 UTC));
+            });
+        });
     }
 
     // --- TASK-007 tests: Legacy Prefix/Filter conflict ---

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -119,7 +119,14 @@ fn should_count_head_proxy_failure(is_not_found: bool, code: Option<&str>, raw_s
     if is_not_found || matches!(code, Some("MethodNotAllowed" | "405")) {
         return false;
     }
-    !matches!(raw_status, Some(404 | 405))
+    if matches!(raw_status, Some(404 | 405)) {
+        return false;
+    }
+    !is_version_id_mismatch(code, raw_status)
+}
+
+fn has_raw_status(err: &SdkError<HeadObjectError>, status: u16) -> bool {
+    err.raw_response().is_some_and(|r| r.status().as_u16() == status)
 }
 
 fn is_head_proxy_failure(err: &SdkError<HeadObjectError>) -> bool {
@@ -150,6 +157,42 @@ async fn head_object_with_proxy_stats(
     result
 }
 
+// AWS returns 400 for root callers and 403 for IAM users when a UUID version ID
+// is rejected. The 403 case is safe: a real auth failure also returns 403 on the
+// versionId-less fallback, propagating as a hard error instead of silently skipping.
+fn is_version_id_mismatch(code: Option<&str>, raw_status: Option<u16>) -> bool {
+    match code {
+        Some(c) if !c.is_empty() => c == "InvalidArgument",
+        _ => matches!(raw_status, Some(400) | Some(403)),
+    }
+}
+
+fn is_version_id_format_mismatch(err: &SdkError<HeadObjectError>) -> bool {
+    let code = err.as_service_error().and_then(|se| se.code());
+    let raw_status = err.raw_response().map(|r| r.status().as_u16());
+    is_version_id_mismatch(code, raw_status)
+}
+
+async fn head_object_fallback(
+    source_bucket: &str,
+    tgt_client: &TargetClient,
+    object: &str,
+) -> std::result::Result<Option<HeadObjectOutput>, SdkError<HeadObjectError>> {
+    match head_object_with_proxy_stats(source_bucket, tgt_client, &tgt_client.bucket, object, None).await {
+        Ok(oi) => Ok(Some(oi)),
+        Err(e) if e.as_service_error().is_some_and(|se| se.is_not_found()) || has_raw_status(&e, 404) => Ok(None),
+        Err(e) => Err(e),
+    }
+}
+
+// Version IDs differ by design on this path (RustFS UUID vs AWS alphanumeric), so
+// compare only ETags. Equal ETags mean identical content; version ID is irrelevant.
+fn content_matches(src: &ObjectInfo, tgt: &HeadObjectOutput) -> bool {
+    let src_etag = src.etag.as_deref().map(rustfs_utils::path::trim_etag);
+    let tgt_etag = tgt.e_tag.as_deref().map(rustfs_utils::path::trim_etag);
+    src_etag.is_some() && src_etag == tgt_etag
+}
+
 #[derive(Debug, Clone, Default)]
 pub struct ResyncOpts {
     pub bucket: String,
@@ -2536,16 +2579,28 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
                 }
             }
             Err(e) => {
-                if let Some(se) = e.as_service_error() {
-                    if !se.is_not_found() {
-                        rinfo.error = Some(e.to_string());
-                        warn!("replication head_object failed bucket:{} arn:{} error:{}", bucket, tgt_client.arn, e);
-                        return rinfo;
+                if e.as_service_error().is_some_and(|se| se.is_not_found()) || has_raw_status(&e, 404) {
+                    // Object not on target yet → fall through to PUT.
+                } else if is_version_id_format_mismatch(&e) {
+                    // Version-ID format mismatch: retry without versionId and compare ETags.
+                    match head_object_fallback(&bucket, &tgt_client, &object).await {
+                        Ok(Some(oi)) if content_matches(&object_info, &oi) => {
+                            rinfo.replication_status = ReplicationStatusType::Completed;
+                            rinfo.replication_resynced = true;
+                            rinfo.replication_action = ReplicationAction::None;
+                            rinfo.size = size;
+                            return rinfo;
+                        }
+                        Ok(_) => {}
+                        Err(e2) => {
+                            rinfo.error = Some(e2.to_string());
+                            warn!(
+                                "replication head_object fallback failed bucket:{} arn:{} error:{}",
+                                bucket, tgt_client.arn, e2
+                            );
+                            return rinfo;
+                        }
                     }
-                } else if e.raw_response().is_some_and(|resp| resp.status().as_u16() == 404) {
-                    // Some HEAD Object 404 responses are surfaced by the AWS SDK as `response error`
-                    // instead of `service error (NotFound)`. Treat raw HTTP 404 as object-not-found
-                    // so replication can proceed with PUT.
                 } else {
                     rinfo.error = Some(e.to_string());
                     warn!("replication head_object failed bucket:{} arn:{} error:{}", bucket, tgt_client.arn, e);
@@ -2807,25 +2862,39 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
                 }
             }
             Err(e) => {
-                if let Some(se) = e.as_service_error() {
-                    if se.is_not_found() {
-                        replication_action = ReplicationAction::All;
-                    } else {
-                        rinfo.error = Some(e.to_string());
-                        warn!("failed to head object for bucket:{} arn:{} error:{}", bucket, tgt_client.arn, e);
-
-                        send_event(EventArgs {
-                            event_name: EventName::ObjectReplicationNotTracked.to_string(),
-                            bucket_name: bucket.clone(),
-                            object: object_info,
-                            host: GLOBAL_LocalNodeName.to_string(),
-                            user_agent: "Internal: [Replication]".to_string(),
-                            ..Default::default()
-                        });
-
-                        rinfo.duration = (OffsetDateTime::now_utc() - start_time).unsigned_abs();
-                        return rinfo;
+                if is_version_id_format_mismatch(&e) {
+                    // Version-ID format mismatch: retry without versionId and compare ETags.
+                    match head_object_fallback(&bucket, &tgt_client, &object).await {
+                        Ok(Some(oi)) => {
+                            replication_action = if content_matches(&object_info, &oi) {
+                                ReplicationAction::None
+                            } else {
+                                ReplicationAction::All
+                            };
+                        }
+                        Ok(None) => {
+                            replication_action = ReplicationAction::All;
+                        }
+                        Err(e2) => {
+                            rinfo.error = Some(e2.to_string());
+                            warn!(
+                                "replication head_object fallback failed bucket:{} arn:{} error:{}",
+                                bucket, tgt_client.arn, e2
+                            );
+                            send_event(EventArgs {
+                                event_name: EventName::ObjectReplicationNotTracked.to_string(),
+                                bucket_name: bucket.clone(),
+                                object: object_info,
+                                host: GLOBAL_LocalNodeName.to_string(),
+                                user_agent: "Internal: [Replication]".to_string(),
+                                ..Default::default()
+                            });
+                            rinfo.duration = (OffsetDateTime::now_utc() - start_time).unsigned_abs();
+                            return rinfo;
+                        }
                     }
+                } else if e.as_service_error().is_some_and(|se| se.is_not_found()) {
+                    replication_action = ReplicationAction::All;
                 } else {
                     rinfo.error = Some(e.to_string());
                     warn!("failed to head object for bucket:{} arn:{} error:{}", bucket, tgt_client.arn, e);
@@ -3837,6 +3906,117 @@ mod tests {
         );
     }
 
+    #[test]
+    fn test_should_count_head_proxy_failure_ignores_version_id_format_rejections() {
+        assert!(
+            !should_count_head_proxy_failure(false, Some("InvalidArgument"), Some(400)),
+            "InvalidArgument/400 is a version-ID format rejection and must not be counted as a proxy failure"
+        );
+        assert!(
+            !should_count_head_proxy_failure(false, None, Some(400)),
+            "raw HTTP 400 without error code must not be counted as a proxy failure"
+        );
+        assert!(
+            !should_count_head_proxy_failure(false, None, Some(403)),
+            "raw HTTP 403 without error code must not be counted as a proxy failure (IAM user + invalid versionId)"
+        );
+    }
+
+    #[test]
+    fn test_is_version_id_mismatch_detects_invalid_argument() {
+        assert!(
+            is_version_id_mismatch(Some("InvalidArgument"), Some(400)),
+            "AWS S3 returns InvalidArgument/400 when a UUID versionId is passed to HeadObject"
+        );
+        assert!(
+            !is_version_id_mismatch(Some("AccessDenied"), Some(403)),
+            "AccessDenied must not trigger the version-ID fallback path"
+        );
+        assert!(
+            !is_version_id_mismatch(Some("NoSuchKey"), Some(404)),
+            "NoSuchKey is an object-not-found response, not a version-ID mismatch"
+        );
+    }
+
+    #[test]
+    fn test_is_version_id_mismatch_raw_status_without_service_code() {
+        assert!(
+            is_version_id_mismatch(None, Some(400)),
+            "no error code + HTTP 400 is treated as version-ID mismatch (HEAD response)"
+        );
+        assert!(
+            is_version_id_mismatch(Some(""), Some(400)),
+            "empty error code + HTTP 400 is treated as version-ID mismatch"
+        );
+        assert!(
+            is_version_id_mismatch(None, Some(403)),
+            "no error code + HTTP 403 is treated as version-ID mismatch (IAM user + invalid versionId)"
+        );
+        assert!(
+            is_version_id_mismatch(Some(""), Some(403)),
+            "empty error code + HTTP 403 is treated as version-ID mismatch"
+        );
+        assert!(
+            !is_version_id_mismatch(None, Some(500)),
+            "raw 5xx must not trigger the version-ID fallback path"
+        );
+        assert!(
+            !is_version_id_mismatch(None, Some(404)),
+            "raw 404 must not trigger the version-ID fallback path"
+        );
+    }
+
+    #[test]
+    fn test_is_version_id_mismatch_400_with_other_service_code() {
+        assert!(
+            !is_version_id_mismatch(Some("MalformedXML"), Some(400)),
+            "MalformedXML/400 is a real request error and must not trigger version-ID fallback"
+        );
+        assert!(
+            !is_version_id_mismatch(Some("EntityTooLarge"), Some(400)),
+            "EntityTooLarge/400 is a real request error and must not trigger version-ID fallback"
+        );
+    }
+
+    #[test]
+    fn test_content_matches_compares_etag_only() {
+        let src = ObjectInfo {
+            etag: Some("\"abc123\"".to_string()),
+            ..Default::default()
+        };
+
+        let tgt_match = HeadObjectOutput::builder().e_tag("\"abc123\"").build();
+        assert!(content_matches(&src, &tgt_match), "identical ETags must match");
+
+        let tgt_unquoted_match = HeadObjectOutput::builder().e_tag("abc123").build();
+        assert!(
+            content_matches(&src, &tgt_unquoted_match),
+            "quoted and unquoted ETags with identical values must match"
+        );
+
+        // version_id on the target is intentionally ignored
+        let tgt_different_version = HeadObjectOutput::builder()
+            .e_tag("\"abc123\"")
+            .version_id("aws-alphanumeric-id")
+            .build();
+        assert!(
+            content_matches(&src, &tgt_different_version),
+            "matching ETags with different version IDs must still match"
+        );
+
+        let tgt_different_content = HeadObjectOutput::builder().e_tag("\"def456\"").build();
+        assert!(!content_matches(&src, &tgt_different_content), "different ETags must not match");
+
+        let src_no_etag = ObjectInfo {
+            etag: None,
+            ..Default::default()
+        };
+        assert!(!content_matches(&src_no_etag, &tgt_match), "missing source ETag must not match");
+
+        let tgt_no_etag = HeadObjectOutput::builder().build();
+        assert!(!content_matches(&src, &tgt_no_etag), "missing target ETag must not match");
+    }
+
     #[test]
     fn test_should_count_head_proxy_failure_counts_unexpected_errors() {
         assert!(

crates/ecstore/src/cache_value/metacache_set.rs

@@ -18,7 +18,12 @@ use crate::disk::{self, DiskAPI, DiskStore, WalkDirOptions};
 use futures::future::join_all;
 use metrics::counter;
 use rustfs_filemeta::{MetaCacheEntries, MetaCacheEntry, MetacacheReader, is_io_eof};
-use std::{future::Future, pin::Pin, time::Duration};
+use std::{
+    future::Future,
+    pin::Pin,
+    sync::{Arc, Mutex},
+    time::Duration,
+};
 use tokio::io::AsyncRead;
 use tokio::spawn;
 use tokio::time::timeout;
@@ -45,6 +50,15 @@ async fn peek_with_timeout<R: AsyncRead + Unpin>(reader: &mut MetacacheReader<R>
     }
 }
 
+#[cfg(test)]
+#[derive(Clone)]
+pub(crate) enum TestReaderBehavior {
+    Eof,
+    Stall,
+    ProducerError(DiskError),
+    PartialThenTimeout(Vec<MetaCacheEntry>),
+}
+
 #[derive(Default)]
 pub struct ListPathRawOptions {
     pub disks: Vec<Option<DiskStore>>,
@@ -60,6 +74,10 @@ pub struct ListPathRawOptions {
     pub agreed: Option<AgreedFn>,
     pub partial: Option<PartialFn>,
     pub finished: Option<FinishedFn>,
+    #[cfg(test)]
+    pub(crate) test_reader_behaviors: Vec<TestReaderBehavior>,
+    #[cfg(test)]
+    pub(crate) peek_timeout: Option<Duration>,
     // pub agreed: Option<Arc<dyn Fn(MetaCacheEntry) + Send + Sync>>,
     // pub partial: Option<Arc<dyn Fn(MetaCacheEntries, &[Option<Error>]) + Send + Sync>>,
     // pub finished: Option<Arc<dyn Fn(&[Option<Error>]) + Send + Sync>>,
@@ -78,6 +96,10 @@ impl Clone for ListPathRawOptions {
             min_disks: self.min_disks,
             report_not_found: self.report_not_found,
             per_disk_limit: self.per_disk_limit,
+            #[cfg(test)]
+            test_reader_behaviors: self.test_reader_behaviors.clone(),
+            #[cfg(test)]
+            peek_timeout: self.peek_timeout,
             ..Default::default()
         }
     }
@@ -85,23 +107,52 @@ impl Clone for ListPathRawOptions {
 
 pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> disk::error::Result<()> {
     if opts.disks.is_empty() {
-        return Err(DiskError::other("list_path_raw: 0 drives provided"));
+        return Err(DiskError::ErasureReadQuorum);
     }
 
     let mut jobs: Vec<tokio::task::JoinHandle<std::result::Result<(), DiskError>>> = Vec::new();
     let mut readers = Vec::with_capacity(opts.disks.len());
     let fds = opts.fallback_disks.iter().flatten().cloned().collect::<Vec<_>>();
+    let max_disk_failures = opts.disks.len().saturating_sub(opts.min_disks);
+    let producer_errs = Arc::new(Mutex::new(vec![None; opts.disks.len()]));
 
     let cancel_rx = CancellationToken::new();
 
-    for disk in opts.disks.iter() {
+    for (disk_idx, disk) in opts.disks.iter().enumerate() {
         let opdisk = disk.clone();
         let opts_clone = opts.clone();
         let mut fds_clone = fds.clone();
         let cancel_rx_clone = cancel_rx.clone();
-        let (rd, mut wr) = tokio::io::duplex(64);
+        let producer_errs_clone = producer_errs.clone();
+        let (rd, wr) = tokio::io::duplex(64);
         readers.push(MetacacheReader::new(rd));
         jobs.push(spawn(async move {
+            #[cfg(test)]
+            if let Some(behavior) = opts_clone.test_reader_behaviors.get(disk_idx).cloned() {
+                match behavior {
+                    TestReaderBehavior::Eof => return Ok(()),
+                    TestReaderBehavior::Stall => {
+                        let _held_writer = wr;
+                        cancel_rx_clone.cancelled().await;
+                        return Ok(());
+                    }
+                    TestReaderBehavior::ProducerError(err) => {
+                        producer_errs_clone.lock().expect("producer error mutex poisoned")[disk_idx] = Some(err.clone());
+                        return Err(err);
+                    }
+                    TestReaderBehavior::PartialThenTimeout(entries) => {
+                        let mut wr = wr;
+                        let mut out = rustfs_filemeta::MetacacheWriter::new(&mut wr);
+                        let err = DiskError::Timeout;
+                        producer_errs_clone.lock().expect("producer error mutex poisoned")[disk_idx] = Some(err.clone());
+                        let _ = out.write(&entries).await;
+                        drop(out);
+                        return Err(err);
+                    }
+                }
+            }
+
+            let mut wr = wr;
             let wakl_opts = WalkDirOptions {
                 bucket: opts_clone.bucket.clone(),
                 base_dir: opts_clone.path.clone(),
@@ -114,15 +165,18 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
             };
 
             let mut need_fallback = false;
+            let mut last_err = None;
             if let Some(disk) = opdisk {
                 match disk.walk_dir(wakl_opts, &mut wr).await {
                     Ok(_res) => {}
                     Err(err) => {
                         info!("walk dir err {:?}", &err);
+                        last_err = Some(err);
                         need_fallback = true;
                     }
                 }
             } else {
+                last_err = Some(DiskError::DiskNotFound);
                 need_fallback = true;
             }
 
@@ -132,18 +186,20 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
             }
 
             while need_fallback {
-                let disk_op = {
-                    if fds_clone.is_empty() {
-                        None
-                    } else {
-                        let disk = fds_clone.remove(0);
-                        if disk.is_online().await { Some(disk.clone()) } else { None }
+                let mut disk_op = None;
+                while !fds_clone.is_empty() {
+                    let disk = fds_clone.remove(0);
+                    if disk.is_online().await {
+                        disk_op = Some(disk);
+                        break;
                     }
-                };
+                }
 
                 let Some(disk) = disk_op else {
                     warn!("list_path_raw: fallback disk is none");
-                    break;
+                    let err = last_err.unwrap_or(DiskError::DiskNotFound);
+                    producer_errs_clone.lock().expect("producer error mutex poisoned")[disk_idx] = Some(err.clone());
+                    return Err(err);
                 };
 
                 match disk
@@ -165,20 +221,28 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                 {
                     Ok(_r) => {
                         need_fallback = false;
+                        last_err = None;
                     }
                     Err(err) => {
                         error!("walk dir2 err {:?}", &err);
-                        break;
+                        last_err = Some(err);
                     }
                 }
             }
 
+            if need_fallback {
+                return Err(last_err.unwrap_or(DiskError::DiskNotFound));
+            }
+
             // warn!("list_path_raw: while need_fallback done");
             Ok(())
         }));
     }
 
     let revjob = spawn(async move {
+        #[cfg(test)]
+        let peek_timeout = opts.peek_timeout.unwrap_or_else(get_drive_walkdir_stall_timeout);
+        #[cfg(not(test))]
         let peek_timeout = get_drive_walkdir_stall_timeout();
         let mut errs: Vec<Option<DiskError>> = Vec::with_capacity(readers.len());
         for _ in 0..readers.len() {
@@ -217,6 +281,11 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                             // info!("read entry disk: {}, name: {}", i, entry.name);
                             entry
                         } else {
+                            if let Some(err) = producer_errs.lock().expect("producer error mutex poisoned")[i].clone() {
+                                has_err += 1;
+                                errs[i] = Some(err);
+                                continue;
+                            }
                             // eof
                             at_eof += 1;
                             // warn!("list_path_raw: peek eof, disk: {}", i);
@@ -224,6 +293,12 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                         }
                     }
                     PeekOutcome::Error(err) => {
+                        if let Some(err) = producer_errs.lock().expect("producer error mutex poisoned")[i].clone() {
+                            has_err += 1;
+                            errs[i] = Some(err);
+                            continue;
+                        }
+
                         if err == rustfs_filemeta::Error::Unexpected {
                             at_eof += 1;
                             // warn!("list_path_raw: peek err eof, disk: {}", i);
@@ -333,6 +408,15 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                 if let Some(finished_fn) = opts.finished.as_ref() {
                     finished_fn(&errs).await;
                 }
+                if errs.iter().flatten().any(|err| *err == DiskError::Timeout) {
+                    return Err(DiskError::Timeout);
+                }
+                let mut err_iter = errs.iter().flatten();
+                if let Some(err) = err_iter.next()
+                    && err_iter.next().is_none()
+                {
+                    return Err(err.clone());
+                }
                 let mut combined_err = Vec::new();
                 errs.iter().zip(opts.disks.iter()).for_each(|(err, disk)| match (err, disk) {
                     (Some(err), Some(disk)) => {
@@ -358,6 +442,9 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                 {
                     finished_fn(&errs).await;
                 }
+                if errs.iter().flatten().any(|err| *err == DiskError::Timeout) {
+                    return Err(DiskError::Timeout);
+                }
 
                 // error!("list_path_raw: at_eof + has_err == readers.len() break {:?}", &errs);
                 break;
@@ -400,12 +487,25 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
     }
 
     let results = join_all(jobs).await;
+    let mut job_errs = Vec::new();
     for result in results {
-        if let Err(err) = result {
-            error!("list_path_raw err {:?}", err);
+        match result {
+            Ok(Ok(())) => {}
+            Ok(Err(err)) => {
+                error!("list_path_raw producer err {:?}", err);
+                job_errs.push(err);
+            }
+            Err(err) => {
+                error!("list_path_raw join err {:?}", err);
+                job_errs.push(err.into());
+            }
         }
     }
 
+    if job_errs.len() > max_disk_failures {
+        return Err(job_errs.remove(0));
+    }
+
     // warn!("list_path_raw: done");
     Ok(())
 }
@@ -415,6 +515,65 @@ mod tests {
     use super::*;
     use rustfs_filemeta::MetacacheWriter;
 
+    #[tokio::test]
+    async fn list_path_raw_empty_disks_returns_read_quorum() {
+        let err = list_path_raw(CancellationToken::new(), ListPathRawOptions::default())
+            .await
+            .expect_err("empty drive list should fail");
+
+        assert_eq!(err, DiskError::ErasureReadQuorum);
+    }
+
+    #[tokio::test]
+    async fn list_path_raw_returns_timeout_when_reader_stalls_before_completion() {
+        let err = list_path_raw(
+            CancellationToken::new(),
+            ListPathRawOptions {
+                disks: vec![None, None],
+                min_disks: 1,
+                test_reader_behaviors: vec![TestReaderBehavior::Stall, TestReaderBehavior::Eof],
+                peek_timeout: Some(Duration::from_millis(20)),
+                ..Default::default()
+            },
+        )
+        .await
+        .expect_err("stalled reader should make listing fail explicitly");
+
+        assert_eq!(err, DiskError::Timeout);
+    }
+
+    #[tokio::test]
+    async fn list_path_raw_returns_timeout_when_producer_fails_after_partial_entry() {
+        let seen = Arc::new(Mutex::new(Vec::new()));
+        let seen_clone = seen.clone();
+
+        let err = list_path_raw(
+            CancellationToken::new(),
+            ListPathRawOptions {
+                disks: vec![None],
+                min_disks: 1,
+                test_reader_behaviors: vec![TestReaderBehavior::PartialThenTimeout(vec![MetaCacheEntry {
+                    name: "bucket/object".to_string(),
+                    metadata: vec![1, 2, 3],
+                    cached: None,
+                    reusable: false,
+                }])],
+                agreed: Some(Box::new(move |entry: MetaCacheEntry| {
+                    let seen = seen_clone.clone();
+                    Box::pin(async move {
+                        seen.lock().expect("seen mutex poisoned").push(entry.name);
+                    })
+                })),
+                ..Default::default()
+            },
+        )
+        .await
+        .expect_err("producer timeout after partial output must fail the listing");
+
+        assert_eq!(err, DiskError::Timeout);
+        assert_eq!(seen.lock().expect("seen mutex poisoned").as_slice(), &["bucket/object".to_string()]);
+    }
+
     #[tokio::test]
     async fn peek_with_timeout_times_out_on_silent_reader() {
         let (_writer, reader) = tokio::io::duplex(64);
@@ -447,4 +606,21 @@ mod tests {
             other => panic!("expected ready entry, got {other:?}"),
         }
     }
+
+    #[tokio::test]
+    async fn list_path_raw_propagates_producer_access_denied() {
+        let err = list_path_raw(
+            CancellationToken::new(),
+            ListPathRawOptions {
+                disks: vec![None],
+                min_disks: 1,
+                test_reader_behaviors: vec![TestReaderBehavior::ProducerError(DiskError::FileAccessDenied)],
+                ..Default::default()
+            },
+        )
+        .await
+        .expect_err("producer access failure must not be treated as an empty listing");
+
+        assert_eq!(err, DiskError::FileAccessDenied);
+    }
 }

crates/ecstore/src/client/api_remove.rs

@@ -340,7 +340,7 @@ impl TransitionClient {
                         content_body: ReaderImpl::Body(Bytes::from(remove_bytes.clone())),
                         content_length: remove_bytes.len() as i64,
                         content_md5_base64: base64_encode(&HashAlgorithm::Md5.hash_encode(&remove_bytes).as_ref()),
-                        content_sha256_hex: base64_encode(&HashAlgorithm::SHA256.hash_encode(&remove_bytes).as_ref()),
+                        content_sha256_hex: rustfs_utils::hex(HashAlgorithm::SHA256.hash_encode(&remove_bytes)),
                         custom_header: headers,
                         object_name: "".to_string(),
                         stream_sha256: false,
@@ -742,3 +742,115 @@ pub async fn process_remove_multi_objects_response(
 fn has_invalid_xml_char(str: &str) -> bool {
     false
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::client::{
+        credentials::{Credentials, SignatureType, Static, Value},
+        transition_api::{BucketLookupType, Options},
+    };
+    use tokio::{
+        io::{AsyncReadExt, AsyncWriteExt},
+        net::TcpListener,
+    };
+
+    async fn capture_delete_objects_sha256_header() -> (String, tokio::task::JoinHandle<String>) {
+        let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
+        let endpoint = listener.local_addr().unwrap().to_string();
+        let task = tokio::spawn(async move {
+            let (mut stream, _) = listener.accept().await.unwrap();
+            let mut request = Vec::new();
+            let mut buffer = [0; 1024];
+            loop {
+                let read = stream.read(&mut buffer).await.unwrap();
+                assert_ne!(read, 0, "connection closed before request headers were received");
+                request.extend_from_slice(&buffer[..read]);
+                if request.windows(4).any(|window| window == b"\r\n\r\n") {
+                    break;
+                }
+            }
+
+            let request = String::from_utf8_lossy(&request);
+            let sha256_header = request
+                .lines()
+                .find_map(|line| {
+                    let (name, value) = line.split_once(':')?;
+                    name.eq_ignore_ascii_case("x-amz-content-sha256")
+                        .then(|| value.trim().to_string())
+                })
+                .expect("delete objects request should include X-Amz-Content-Sha256");
+
+            let response_body = r#"<?xml version="1.0" encoding="UTF-8"?><DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Deleted><Key>object.txt</Key></Deleted></DeleteResult>"#;
+            let response = format!(
+                "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}",
+                response_body.len(),
+                response_body
+            );
+            stream.write_all(response.as_bytes()).await.unwrap();
+            sha256_header
+        });
+
+        (endpoint, task)
+    }
+
+    #[tokio::test]
+    async fn multi_object_delete_request_uses_lowercase_hex_sha256_header() {
+        let objects = vec![ObjectInfo {
+            bucket: "bucket".to_string(),
+            name: "object.txt".to_string(),
+            ..Default::default()
+        }];
+        let body = generate_remove_multi_objects_request(&objects);
+        let expected = rustfs_utils::hex(HashAlgorithm::SHA256.hash_encode(&body));
+        let (endpoint, header_task) = capture_delete_objects_sha256_header().await;
+        let client = TransitionClient::new(
+            &endpoint,
+            Options {
+                creds: Credentials::new(Static(Value {
+                    access_key_id: "access-key".to_string(),
+                    secret_access_key: "secret-key".to_string(),
+                    signer_type: SignatureType::SignatureV4,
+                    ..Default::default()
+                })),
+                region: "us-east-1".to_string(),
+                bucket_lookup: BucketLookupType::BucketLookupPath,
+                max_retries: 1,
+                ..Default::default()
+            },
+            "",
+        )
+        .await
+        .unwrap();
+        let (objects_tx, objects_rx) = mpsc::channel(1);
+        let (result_tx, mut result_rx) = mpsc::channel(1);
+
+        objects_tx.send(objects[0].clone()).await.unwrap();
+        drop(objects_tx);
+
+        client
+            .remove_objects_inner(
+                "bucket",
+                objects_rx,
+                &result_tx,
+                RemoveObjectsOptions {
+                    governance_bypass: false,
+                },
+            )
+            .await
+            .unwrap();
+        drop(result_tx);
+
+        let header = header_task.await.unwrap();
+
+        assert_eq!(header, expected);
+        assert_eq!(header.len(), 64);
+        assert!(
+            header
+                .bytes()
+                .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte))
+        );
+        assert_ne!(header, base64_encode(&HashAlgorithm::SHA256.hash_encode(&body).as_ref()));
+        assert!(result_rx.recv().await.is_some());
+    }
+}

crates/ecstore/src/config/audit.rs

@@ -13,17 +13,27 @@
 //  limitations under the License.
 
 use crate::config::{KV, KVS};
+use rustfs_config::audit::AUDIT_REDIS_DEFAULT_CHANNEL;
 use rustfs_config::{
-    COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY, EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR,
-    KAFKA_QUEUE_LIMIT, KAFKA_TLS_CA, KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY, KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER,
-    MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA,
-    MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME,
-    MQTT_WS_PATH_ALLOWLIST, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT,
-    NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, PULSAR_AUTH_TOKEN,
-    PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
-    PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA,
-    WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR,
-    WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL, WEBHOOK_SKIP_TLS_VERIFY,
+    AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_QUEUE_LIMIT, AMQP_ROUTING_KEY,
+    AMQP_TLS_CA, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY,
+    EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_TLS_CA,
+    KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY, KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL,
+    MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT,
+    MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST,
+    MYSQL_DSN_STRING, MYSQL_FORMAT, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA,
+    MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR,
+    NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN,
+    NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE,
+    POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER,
+    PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
+    PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT,
+    REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD,
+    REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT,
+    REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL,
+    REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY,
+    WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL,
+    WEBHOOK_SKIP_TLS_VERIFY,
 };
 use std::sync::LazyLock;
 
@@ -198,6 +208,81 @@ pub static DEFAULT_AUDIT_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
     ])
 });
 
+pub static DEFAULT_AUDIT_AMQP_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: ENABLE_KEY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_URL.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_EXCHANGE.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_ROUTING_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_MANDATORY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_PERSISTENT.to_owned(),
+            value: EnableState::On.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_USERNAME.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_PASSWORD.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: AMQP_TLS_CA.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: AMQP_TLS_CLIENT_CERT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: AMQP_TLS_CLIENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: AMQP_QUEUE_DIR.to_owned(),
+            value: EVENT_DEFAULT_DIR.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_QUEUE_LIMIT.to_owned(),
+            value: DEFAULT_LIMIT.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: COMMENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+    ])
+});
+
 pub static DEFAULT_AUDIT_NATS_KVS: LazyLock<KVS> = LazyLock::new(|| {
     KVS(vec![
         KV {
@@ -339,6 +424,176 @@ pub static DEFAULT_AUDIT_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(|| {
     ])
 });
 
+pub static DEFAULT_AUDIT_REDIS_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: ENABLE_KEY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_URL.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_CHANNEL.to_owned(),
+            value: AUDIT_REDIS_DEFAULT_CHANNEL.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_USERNAME.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_PASSWORD.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_KEEP_ALIVE_INTERVAL.to_owned(),
+            value: "15".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_QUEUE_DIR.to_owned(),
+            value: EVENT_DEFAULT_DIR.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_QUEUE_LIMIT.to_owned(),
+            value: DEFAULT_LIMIT.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_MAX_RETRY_ATTEMPTS.to_owned(),
+            value: "3".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_RECONNECT_RETRY_ATTEMPTS.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_MIN_RETRY_DELAY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_MAX_RETRY_DELAY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_CONNECTION_TIMEOUT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_RESPONSE_TIMEOUT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_PIPELINE_BUFFER_SIZE.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_TLS_POLICY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_TLS_CA.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_TLS_CLIENT_CERT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_TLS_CLIENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_TLS_ALLOW_INSECURE.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: COMMENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+    ])
+});
+
+pub static DEFAULT_AUDIT_POSTGRES_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: ENABLE_KEY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_DSN_STRING.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: POSTGRES_TABLE.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_FORMAT.to_owned(),
+            value: "namespace".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_TLS_REQUIRED.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_TLS_CA.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: POSTGRES_TLS_CLIENT_CERT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: POSTGRES_TLS_CLIENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: POSTGRES_QUEUE_DIR.to_owned(),
+            value: EVENT_DEFAULT_DIR.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_QUEUE_LIMIT.to_owned(),
+            value: DEFAULT_LIMIT.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: COMMENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+    ])
+});
+
 pub static DEFAULT_AUDIT_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(|| {
     KVS(vec![
         KV {
@@ -398,3 +653,63 @@ pub static DEFAULT_AUDIT_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(|| {
         },
     ])
 });
+
+pub static DEFAULT_AUDIT_MYSQL_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: ENABLE_KEY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_DSN_STRING.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: MYSQL_TABLE.to_owned(),
+            value: "rustfs_audit_logs".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_FORMAT.to_owned(),
+            value: "access".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_TLS_CA.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: MYSQL_TLS_CLIENT_CERT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: MYSQL_TLS_CLIENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: MYSQL_QUEUE_DIR.to_owned(),
+            value: EVENT_DEFAULT_DIR.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_QUEUE_LIMIT.to_owned(),
+            value: DEFAULT_LIMIT.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_MAX_OPEN_CONNECTIONS.to_owned(),
+            value: "2".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: COMMENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+    ])
+});

crates/ecstore/src/config/com.rs

@@ -19,12 +19,15 @@ use crate::global::is_first_cluster_node_local;
 use crate::store_api::{ObjectInfo, ObjectOptions, PutObjReader, StorageAPI};
 use http::HeaderMap;
 use rustfs_config::audit::{
-    AUDIT_KAFKA_KEYS, AUDIT_KAFKA_SUB_SYS, AUDIT_MQTT_KEYS, AUDIT_MQTT_SUB_SYS, AUDIT_NATS_KEYS, AUDIT_NATS_SUB_SYS,
-    AUDIT_PULSAR_KEYS, AUDIT_PULSAR_SUB_SYS, AUDIT_WEBHOOK_KEYS, AUDIT_WEBHOOK_SUB_SYS,
+    AUDIT_AMQP_KEYS, AUDIT_AMQP_SUB_SYS, AUDIT_KAFKA_KEYS, AUDIT_KAFKA_SUB_SYS, AUDIT_MQTT_KEYS, AUDIT_MQTT_SUB_SYS,
+    AUDIT_MYSQL_KEYS, AUDIT_MYSQL_SUB_SYS, AUDIT_NATS_KEYS, AUDIT_NATS_SUB_SYS, AUDIT_POSTGRES_KEYS, AUDIT_POSTGRES_SUB_SYS,
+    AUDIT_PULSAR_KEYS, AUDIT_PULSAR_SUB_SYS, AUDIT_REDIS_KEYS, AUDIT_REDIS_SUB_SYS, AUDIT_WEBHOOK_KEYS, AUDIT_WEBHOOK_SUB_SYS,
 };
 use rustfs_config::notify::{
-    NOTIFY_KAFKA_KEYS, NOTIFY_KAFKA_SUB_SYS, NOTIFY_MQTT_KEYS, NOTIFY_MQTT_SUB_SYS, NOTIFY_NATS_KEYS, NOTIFY_NATS_SUB_SYS,
-    NOTIFY_PULSAR_KEYS, NOTIFY_PULSAR_SUB_SYS, NOTIFY_WEBHOOK_KEYS, NOTIFY_WEBHOOK_SUB_SYS,
+    NOTIFY_AMQP_KEYS, NOTIFY_AMQP_SUB_SYS, NOTIFY_KAFKA_KEYS, NOTIFY_KAFKA_SUB_SYS, NOTIFY_MQTT_KEYS, NOTIFY_MQTT_SUB_SYS,
+    NOTIFY_MYSQL_KEYS, NOTIFY_MYSQL_SUB_SYS, NOTIFY_NATS_KEYS, NOTIFY_NATS_SUB_SYS, NOTIFY_POSTGRES_KEYS,
+    NOTIFY_POSTGRES_SUB_SYS, NOTIFY_PULSAR_KEYS, NOTIFY_PULSAR_SUB_SYS, NOTIFY_REDIS_KEYS, NOTIFY_REDIS_SUB_SYS,
+    NOTIFY_WEBHOOK_KEYS, NOTIFY_WEBHOOK_SUB_SYS,
 };
 use rustfs_config::oidc::{IDENTITY_OPENID_KEYS, IDENTITY_OPENID_SUB_SYS, OIDC_REDIRECT_URI_DYNAMIC};
 use rustfs_config::{COMMENT_KEY, DEFAULT_DELIMITER, ENABLE_KEY, EnableState, RUSTFS_REGION};
@@ -58,7 +61,7 @@ struct TargetConfigDescriptor {
     valid_keys: &'static [&'static str],
 }
 
-fn notify_target_descriptors() -> [TargetConfigDescriptor; 5] {
+fn notify_target_descriptors() -> [TargetConfigDescriptor; 9] {
     [
         TargetConfigDescriptor {
             external_key: "webhook",
@@ -66,6 +69,12 @@ fn notify_target_descriptors() -> [TargetConfigDescriptor; 5] {
             default_kvs: &notify::DEFAULT_NOTIFY_WEBHOOK_KVS,
             valid_keys: NOTIFY_WEBHOOK_KEYS,
         },
+        TargetConfigDescriptor {
+            external_key: "amqp",
+            subsystem_key: NOTIFY_AMQP_SUB_SYS,
+            default_kvs: &notify::DEFAULT_NOTIFY_AMQP_KVS,
+            valid_keys: NOTIFY_AMQP_KEYS,
+        },
         TargetConfigDescriptor {
             external_key: "kafka",
             subsystem_key: NOTIFY_KAFKA_SUB_SYS,
@@ -78,12 +87,30 @@ fn notify_target_descriptors() -> [TargetConfigDescriptor; 5] {
             default_kvs: &notify::DEFAULT_NOTIFY_MQTT_KVS,
             valid_keys: NOTIFY_MQTT_KEYS,
         },
+        TargetConfigDescriptor {
+            external_key: "mysql",
+            subsystem_key: NOTIFY_MYSQL_SUB_SYS,
+            default_kvs: &notify::DEFAULT_NOTIFY_MYSQL_KVS,
+            valid_keys: NOTIFY_MYSQL_KEYS,
+        },
         TargetConfigDescriptor {
             external_key: "nats",
             subsystem_key: NOTIFY_NATS_SUB_SYS,
             default_kvs: &notify::DEFAULT_NOTIFY_NATS_KVS,
             valid_keys: NOTIFY_NATS_KEYS,
         },
+        TargetConfigDescriptor {
+            external_key: "postgres",
+            subsystem_key: NOTIFY_POSTGRES_SUB_SYS,
+            default_kvs: &notify::DEFAULT_NOTIFY_POSTGRES_KVS,
+            valid_keys: NOTIFY_POSTGRES_KEYS,
+        },
+        TargetConfigDescriptor {
+            external_key: "redis",
+            subsystem_key: NOTIFY_REDIS_SUB_SYS,
+            default_kvs: &notify::DEFAULT_NOTIFY_REDIS_KVS,
+            valid_keys: NOTIFY_REDIS_KEYS,
+        },
         TargetConfigDescriptor {
             external_key: "pulsar",
             subsystem_key: NOTIFY_PULSAR_SUB_SYS,
@@ -93,7 +120,7 @@ fn notify_target_descriptors() -> [TargetConfigDescriptor; 5] {
     ]
 }
 
-fn audit_target_descriptors() -> [TargetConfigDescriptor; 5] {
+fn audit_target_descriptors() -> [TargetConfigDescriptor; 9] {
     [
         TargetConfigDescriptor {
             external_key: "webhook",
@@ -101,6 +128,12 @@ fn audit_target_descriptors() -> [TargetConfigDescriptor; 5] {
             default_kvs: &audit::DEFAULT_AUDIT_WEBHOOK_KVS,
             valid_keys: AUDIT_WEBHOOK_KEYS,
         },
+        TargetConfigDescriptor {
+            external_key: "amqp",
+            subsystem_key: AUDIT_AMQP_SUB_SYS,
+            default_kvs: &audit::DEFAULT_AUDIT_AMQP_KVS,
+            valid_keys: AUDIT_AMQP_KEYS,
+        },
         TargetConfigDescriptor {
             external_key: "kafka",
             subsystem_key: AUDIT_KAFKA_SUB_SYS,
@@ -113,18 +146,36 @@ fn audit_target_descriptors() -> [TargetConfigDescriptor; 5] {
             default_kvs: &audit::DEFAULT_AUDIT_MQTT_KVS,
             valid_keys: AUDIT_MQTT_KEYS,
         },
+        TargetConfigDescriptor {
+            external_key: "mysql",
+            subsystem_key: AUDIT_MYSQL_SUB_SYS,
+            default_kvs: &audit::DEFAULT_AUDIT_MYSQL_KVS,
+            valid_keys: AUDIT_MYSQL_KEYS,
+        },
         TargetConfigDescriptor {
             external_key: "nats",
             subsystem_key: AUDIT_NATS_SUB_SYS,
             default_kvs: &audit::DEFAULT_AUDIT_NATS_KVS,
             valid_keys: AUDIT_NATS_KEYS,
         },
+        TargetConfigDescriptor {
+            external_key: "postgres",
+            subsystem_key: AUDIT_POSTGRES_SUB_SYS,
+            default_kvs: &audit::DEFAULT_AUDIT_POSTGRES_KVS,
+            valid_keys: AUDIT_POSTGRES_KEYS,
+        },
         TargetConfigDescriptor {
             external_key: "pulsar",
             subsystem_key: AUDIT_PULSAR_SUB_SYS,
             default_kvs: &audit::DEFAULT_AUDIT_PULSAR_KVS,
             valid_keys: AUDIT_PULSAR_KEYS,
         },
+        TargetConfigDescriptor {
+            external_key: "redis",
+            subsystem_key: AUDIT_REDIS_SUB_SYS,
+            default_kvs: &audit::DEFAULT_AUDIT_REDIS_KVS,
+            valid_keys: AUDIT_REDIS_KEYS,
+        },
     ]
 }
 
@@ -679,6 +730,8 @@ fn is_target_bool_key(key: &str) -> bool {
     matches!(
         key,
         ENABLE_KEY
+            | rustfs_config::AMQP_MANDATORY
+            | rustfs_config::AMQP_PERSISTENT
             | rustfs_config::WEBHOOK_SKIP_TLS_VERIFY
             | rustfs_config::KAFKA_TLS_ENABLE
             | rustfs_config::MQTT_TLS_TRUST_LEAF_AS_CA
@@ -688,14 +741,31 @@ fn is_target_bool_key(key: &str) -> bool {
     )
 }
 
+fn parse_target_bool_scalar(value: &str) -> Option<bool> {
+    if let Ok(state) = value.parse::<EnableState>() {
+        return Some(state.is_enabled());
+    }
+    if let Ok(boolean) = value.parse::<bool>() {
+        return Some(boolean);
+    }
+    None
+}
+
+fn target_scalar_values_equal(key: &str, lhs: &str, rhs: &str) -> bool {
+    if is_target_bool_key(key)
+        && let (Some(lhs), Some(rhs)) = (parse_target_bool_scalar(lhs), parse_target_bool_scalar(rhs))
+    {
+        return lhs == rhs;
+    }
+
+    lhs == rhs
+}
+
 fn encode_target_scalar_value(key: &str, value: &str) -> Value {
-    if is_target_bool_key(key) {
-        if let Ok(state) = value.parse::<EnableState>() {
-            return Value::Bool(state.is_enabled());
-        }
-        if let Ok(boolean) = value.parse::<bool>() {
-            return Value::Bool(boolean);
-        }
+    if is_target_bool_key(key)
+        && let Some(boolean) = parse_target_bool_scalar(value)
+    {
+        return Value::Bool(boolean);
     }
 
     Value::String(value.to_string())
@@ -721,7 +791,7 @@ fn build_target_instance_diff_object(kvs: &KVS, baseline: &KVS, valid_keys: &[&s
         let baseline_value = baseline.lookup(key).unwrap_or_default();
         let effective_value = kvs.lookup(key).unwrap_or_else(|| baseline_value.clone());
 
-        if effective_value == baseline_value {
+        if target_scalar_values_equal(key, &effective_value, &baseline_value) {
             continue;
         }
 
@@ -1131,10 +1201,14 @@ mod tests {
         ObjectOptions, ObjectToDelete, PartInfo, PutObjReader, StorageAPI, WalkOptions,
     };
     use http::HeaderMap;
-    use rustfs_config::audit::{AUDIT_KAFKA_SUB_SYS, AUDIT_MQTT_SUB_SYS, AUDIT_WEBHOOK_SUB_SYS};
-    use rustfs_config::notify::{NOTIFY_KAFKA_SUB_SYS, NOTIFY_MQTT_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS};
+    use rustfs_config::audit::{AUDIT_AMQP_SUB_SYS, AUDIT_KAFKA_SUB_SYS, AUDIT_MQTT_SUB_SYS, AUDIT_WEBHOOK_SUB_SYS};
+    use rustfs_config::notify::{
+        NOTIFY_AMQP_SUB_SYS, NOTIFY_KAFKA_SUB_SYS, NOTIFY_MQTT_SUB_SYS, NOTIFY_MYSQL_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS,
+    };
     use rustfs_config::oidc::IDENTITY_OPENID_SUB_SYS;
-    use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY, EnableState};
+    use rustfs_config::{
+        DEFAULT_DELIMITER, ENABLE_KEY, EnableState, MYSQL_DSN_STRING, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_TABLE,
+    };
     use rustfs_filemeta::FileInfo;
     use rustfs_lock::client::LockClient;
     use rustfs_lock::client::local::LocalClient;
@@ -1775,6 +1849,24 @@ mod tests {
                 "acks":"all",
                 "tls_enable":true
               }
+            },
+            "amqp":{
+              "primary":{
+                "enable":true,
+                "url":"amqp://127.0.0.1:5672/%2f",
+                "exchange":"rustfs.events",
+                "routing_key":"objects",
+                "persistent":true
+              }
+            },
+            "mysql":{
+              "primary":{
+                "enable":true,
+                "dsn_string":"rustfs:password@tcp(127.0.0.1:3306)/rustfs_events",
+                "table":"rustfs_events",
+                "queue_dir":"/tmp/mysql-queue",
+                "max_open_connections":"2"
+              }
             }
           }
         }"#;
@@ -1811,6 +1903,24 @@ mod tests {
         assert_eq!(kafka.get(rustfs_config::KAFKA_TOPIC), "events-kafka");
         assert_eq!(kafka.get(rustfs_config::KAFKA_ACKS), "all");
         assert_eq!(kafka.get(rustfs_config::KAFKA_TLS_ENABLE), "true");
+
+        let amqp = cfg
+            .get_value(NOTIFY_AMQP_SUB_SYS, "primary")
+            .expect("amqp target should be decoded");
+        assert_eq!(amqp.get(ENABLE_KEY), EnableState::On.to_string());
+        assert_eq!(amqp.get(rustfs_config::AMQP_URL), "amqp://127.0.0.1:5672/%2f");
+        assert_eq!(amqp.get(rustfs_config::AMQP_EXCHANGE), "rustfs.events");
+        assert_eq!(amqp.get(rustfs_config::AMQP_ROUTING_KEY), "objects");
+        assert_eq!(amqp.get(rustfs_config::AMQP_PERSISTENT), "true");
+
+        let mysql = cfg
+            .get_value(NOTIFY_MYSQL_SUB_SYS, "primary")
+            .expect("mysql target should be decoded");
+        assert_eq!(mysql.get(ENABLE_KEY), EnableState::On.to_string());
+        assert_eq!(mysql.get(MYSQL_DSN_STRING), "rustfs:password@tcp(127.0.0.1:3306)/rustfs_events");
+        assert_eq!(mysql.get(MYSQL_TABLE), "rustfs_events");
+        assert_eq!(mysql.get(MYSQL_QUEUE_DIR), "/tmp/mysql-queue");
+        assert_eq!(mysql.get(MYSQL_MAX_OPEN_CONNECTIONS), "2");
     }
 
     #[test]
@@ -1870,6 +1980,15 @@ mod tests {
                 "queue_dir":"/tmp/audit-queue"
               }
             },
+            "amqp":{
+              "primary":{
+                "enable":true,
+                "url":"amqp://127.0.0.1:5672/%2f",
+                "exchange":"rustfs.audit",
+                "routing_key":"audit",
+                "persistent":true
+              }
+            },
             "mqtt":{
               "default":{
                 "enable":true,
@@ -1901,6 +2020,15 @@ mod tests {
         assert_eq!(webhook.get(rustfs_config::WEBHOOK_ENDPOINT), "https://example.com/audit-hook");
         assert_eq!(webhook.get(rustfs_config::WEBHOOK_QUEUE_DIR), "/tmp/audit-queue");
 
+        let amqp = cfg
+            .get_value(AUDIT_AMQP_SUB_SYS, "primary")
+            .expect("audit amqp target should be decoded");
+        assert_eq!(amqp.get(ENABLE_KEY), EnableState::On.to_string());
+        assert_eq!(amqp.get(rustfs_config::AMQP_URL), "amqp://127.0.0.1:5672/%2f");
+        assert_eq!(amqp.get(rustfs_config::AMQP_EXCHANGE), "rustfs.audit");
+        assert_eq!(amqp.get(rustfs_config::AMQP_ROUTING_KEY), "audit");
+        assert_eq!(amqp.get(rustfs_config::AMQP_PERSISTENT), "true");
+
         let mqtt_default = cfg
             .get_value(AUDIT_MQTT_SUB_SYS, DEFAULT_DELIMITER)
             .expect("audit mqtt default should be decoded");
@@ -2069,6 +2197,44 @@ mod tests {
         );
         cfg.0.insert(NOTIFY_KAFKA_SUB_SYS.to_string(), kafka_section);
 
+        let mut amqp_section = std::collections::HashMap::new();
+        amqp_section.insert(
+            "primary".to_string(),
+            crate::config::KVS(vec![
+                crate::config::KV {
+                    key: ENABLE_KEY.to_string(),
+                    value: EnableState::On.to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_URL.to_string(),
+                    value: "amqp://127.0.0.1:5672/%2f".to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_EXCHANGE.to_string(),
+                    value: "rustfs.events".to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_ROUTING_KEY.to_string(),
+                    value: "objects".to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_MANDATORY.to_string(),
+                    value: "false".to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_PERSISTENT.to_string(),
+                    value: "false".to_string(),
+                    hidden_if_empty: false,
+                },
+            ]),
+        );
+        cfg.0.insert(NOTIFY_AMQP_SUB_SYS.to_string(), amqp_section);
+
         let out = encode_server_config_blob(&cfg, None).expect("encode should succeed");
         let v: Value = serde_json::from_slice(&out).expect("output should be json");
         let notify = v
@@ -2117,6 +2283,21 @@ mod tests {
         );
         assert_eq!(kafka.get(rustfs_config::KAFKA_ACKS).and_then(Value::as_str), Some("all"));
         assert_eq!(kafka.get(rustfs_config::KAFKA_TLS_ENABLE).and_then(Value::as_bool), Some(true));
+
+        let amqp = notify
+            .get("amqp")
+            .and_then(Value::as_object)
+            .and_then(|targets| targets.get("primary"))
+            .and_then(Value::as_object)
+            .expect("amqp target should be encoded");
+        assert_eq!(
+            amqp.get(rustfs_config::AMQP_URL).and_then(Value::as_str),
+            Some("amqp://127.0.0.1:5672/%2f")
+        );
+        assert_eq!(amqp.get(rustfs_config::AMQP_EXCHANGE).and_then(Value::as_str), Some("rustfs.events"));
+        assert_eq!(amqp.get(rustfs_config::AMQP_ROUTING_KEY).and_then(Value::as_str), Some("objects"));
+        assert!(!amqp.contains_key(rustfs_config::AMQP_MANDATORY));
+        assert_eq!(amqp.get(rustfs_config::AMQP_PERSISTENT).and_then(Value::as_bool), Some(false));
     }
 
     #[test]
@@ -2146,6 +2327,44 @@ mod tests {
         );
         cfg.0.insert(AUDIT_WEBHOOK_SUB_SYS.to_string(), webhook_section);
 
+        let mut amqp_section = std::collections::HashMap::new();
+        amqp_section.insert(
+            "primary".to_string(),
+            crate::config::KVS(vec![
+                crate::config::KV {
+                    key: ENABLE_KEY.to_string(),
+                    value: EnableState::On.to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_URL.to_string(),
+                    value: "amqp://127.0.0.1:5672/%2f".to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_EXCHANGE.to_string(),
+                    value: "rustfs.audit".to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_ROUTING_KEY.to_string(),
+                    value: "audit".to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_MANDATORY.to_string(),
+                    value: "false".to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_PERSISTENT.to_string(),
+                    value: "false".to_string(),
+                    hidden_if_empty: false,
+                },
+            ]),
+        );
+        cfg.0.insert(AUDIT_AMQP_SUB_SYS.to_string(), amqp_section);
+
         let mut mqtt_default = audit::DEFAULT_AUDIT_MQTT_KVS.clone();
         mqtt_default.insert(ENABLE_KEY.to_string(), EnableState::On.to_string());
         mqtt_default.insert(rustfs_config::MQTT_TOPIC.to_string(), "audit-events".to_string());
@@ -2208,6 +2427,21 @@ mod tests {
         );
         assert_eq!(webhook.get(ENABLE_KEY).and_then(Value::as_bool), Some(true));
 
+        let amqp = logger
+            .get("amqp")
+            .and_then(Value::as_object)
+            .and_then(|targets| targets.get("primary"))
+            .and_then(Value::as_object)
+            .expect("audit amqp target should be encoded");
+        assert_eq!(
+            amqp.get(rustfs_config::AMQP_URL).and_then(Value::as_str),
+            Some("amqp://127.0.0.1:5672/%2f")
+        );
+        assert_eq!(amqp.get(rustfs_config::AMQP_EXCHANGE).and_then(Value::as_str), Some("rustfs.audit"));
+        assert_eq!(amqp.get(rustfs_config::AMQP_ROUTING_KEY).and_then(Value::as_str), Some("audit"));
+        assert!(!amqp.contains_key(rustfs_config::AMQP_MANDATORY));
+        assert_eq!(amqp.get(rustfs_config::AMQP_PERSISTENT).and_then(Value::as_bool), Some(false));
+
         let mqtt_default = logger
             .get("mqtt")
             .and_then(Value::as_object)

crates/ecstore/src/config/mod.rs

@@ -26,10 +26,12 @@ use com::{STORAGE_CLASS_SUB_SYS, lookup_configs, read_config_without_migrate};
 use rustfs_config::COMMENT_KEY;
 use rustfs_config::DEFAULT_DELIMITER;
 use rustfs_config::audit::{
-    AUDIT_KAFKA_SUB_SYS, AUDIT_MQTT_SUB_SYS, AUDIT_NATS_SUB_SYS, AUDIT_PULSAR_SUB_SYS, AUDIT_WEBHOOK_SUB_SYS,
+    AUDIT_AMQP_SUB_SYS, AUDIT_KAFKA_SUB_SYS, AUDIT_MQTT_SUB_SYS, AUDIT_MYSQL_SUB_SYS, AUDIT_NATS_SUB_SYS, AUDIT_POSTGRES_SUB_SYS,
+    AUDIT_PULSAR_SUB_SYS, AUDIT_REDIS_SUB_SYS, AUDIT_WEBHOOK_SUB_SYS,
 };
 use rustfs_config::notify::{
-    NOTIFY_KAFKA_SUB_SYS, NOTIFY_MQTT_SUB_SYS, NOTIFY_NATS_SUB_SYS, NOTIFY_PULSAR_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS,
+    NOTIFY_AMQP_SUB_SYS, NOTIFY_KAFKA_SUB_SYS, NOTIFY_MQTT_SUB_SYS, NOTIFY_MYSQL_SUB_SYS, NOTIFY_NATS_SUB_SYS,
+    NOTIFY_POSTGRES_SUB_SYS, NOTIFY_PULSAR_SUB_SYS, NOTIFY_REDIS_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS,
 };
 use rustfs_config::oidc::IDENTITY_OPENID_SUB_SYS;
 use serde::{Deserialize, Serialize};
@@ -241,12 +243,20 @@ pub fn init() {
     kvs.insert(AUDIT_WEBHOOK_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_WEBHOOK_KVS.clone());
     kvs.insert(NOTIFY_MQTT_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_MQTT_KVS.clone());
     kvs.insert(AUDIT_MQTT_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_MQTT_KVS.clone());
+    kvs.insert(NOTIFY_AMQP_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_AMQP_KVS.clone());
+    kvs.insert(AUDIT_AMQP_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_AMQP_KVS.clone());
     kvs.insert(NOTIFY_NATS_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_NATS_KVS.clone());
     kvs.insert(AUDIT_NATS_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_NATS_KVS.clone());
+    kvs.insert(NOTIFY_REDIS_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_REDIS_KVS.clone());
+    kvs.insert(AUDIT_REDIS_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_REDIS_KVS.clone());
+    kvs.insert(NOTIFY_POSTGRES_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_POSTGRES_KVS.clone());
+    kvs.insert(AUDIT_POSTGRES_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_POSTGRES_KVS.clone());
     kvs.insert(NOTIFY_PULSAR_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_PULSAR_KVS.clone());
     kvs.insert(AUDIT_PULSAR_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_PULSAR_KVS.clone());
     kvs.insert(NOTIFY_KAFKA_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_KAFKA_KVS.clone());
     kvs.insert(AUDIT_KAFKA_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_KAFKA_KVS.clone());
+    kvs.insert(NOTIFY_MYSQL_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_MYSQL_KVS.clone());
+    kvs.insert(AUDIT_MYSQL_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_MYSQL_KVS.clone());
     kvs.insert(IDENTITY_OPENID_SUB_SYS.to_owned(), oidc::DEFAULT_IDENTITY_OPENID_KVS.clone());
 
     // Register all default configurations

crates/ecstore/src/config/notify.rs

@@ -13,16 +13,26 @@
 // limitations under the License.
 
 use crate::config::{KV, KVS};
+use rustfs_config::notify::NOTIFY_REDIS_DEFAULT_CHANNEL;
 use rustfs_config::{
-    COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY, EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR,
-    KAFKA_QUEUE_LIMIT, KAFKA_TLS_CA, KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY, KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER,
-    MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA,
-    MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME,
-    MQTT_WS_PATH_ALLOWLIST, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT,
-    NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, PULSAR_AUTH_TOKEN,
-    PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
-    PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT,
-    WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY,
+    AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_QUEUE_LIMIT, AMQP_ROUTING_KEY,
+    AMQP_TLS_CA, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY,
+    EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_TLS_CA,
+    KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY, KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL,
+    MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT,
+    MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST,
+    MYSQL_DSN_STRING, MYSQL_FORMAT, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA,
+    MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR,
+    NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN,
+    NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE,
+    POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER,
+    PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
+    PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT,
+    REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD,
+    REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT,
+    REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL,
+    REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT,
+    WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY,
 };
 use std::sync::LazyLock;
 
@@ -176,6 +186,81 @@ pub static DEFAULT_NOTIFY_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
     ])
 });
 
+pub static DEFAULT_NOTIFY_AMQP_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: ENABLE_KEY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_URL.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_EXCHANGE.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_ROUTING_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_MANDATORY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_PERSISTENT.to_owned(),
+            value: EnableState::On.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_USERNAME.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_PASSWORD.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: AMQP_TLS_CA.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: AMQP_TLS_CLIENT_CERT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: AMQP_TLS_CLIENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: AMQP_QUEUE_DIR.to_owned(),
+            value: EVENT_DEFAULT_DIR.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_QUEUE_LIMIT.to_owned(),
+            value: DEFAULT_LIMIT.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: COMMENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+    ])
+});
+
 pub static DEFAULT_NOTIFY_NATS_KVS: LazyLock<KVS> = LazyLock::new(|| {
     KVS(vec![
         KV {
@@ -316,6 +401,176 @@ pub static DEFAULT_NOTIFY_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(|| {
     ])
 });
 
+pub static DEFAULT_NOTIFY_REDIS_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: ENABLE_KEY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_URL.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_CHANNEL.to_owned(),
+            value: NOTIFY_REDIS_DEFAULT_CHANNEL.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_USERNAME.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_PASSWORD.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_KEEP_ALIVE_INTERVAL.to_owned(),
+            value: "15".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_QUEUE_DIR.to_owned(),
+            value: EVENT_DEFAULT_DIR.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_QUEUE_LIMIT.to_owned(),
+            value: DEFAULT_LIMIT.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_MAX_RETRY_ATTEMPTS.to_owned(),
+            value: "3".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_RECONNECT_RETRY_ATTEMPTS.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_MIN_RETRY_DELAY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_MAX_RETRY_DELAY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_CONNECTION_TIMEOUT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_RESPONSE_TIMEOUT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_PIPELINE_BUFFER_SIZE.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_TLS_POLICY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_TLS_CA.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_TLS_CLIENT_CERT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_TLS_CLIENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_TLS_ALLOW_INSECURE.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: COMMENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+    ])
+});
+
+pub static DEFAULT_NOTIFY_POSTGRES_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: ENABLE_KEY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_DSN_STRING.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: POSTGRES_TABLE.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_FORMAT.to_owned(),
+            value: "namespace".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_TLS_REQUIRED.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_TLS_CA.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: POSTGRES_TLS_CLIENT_CERT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: POSTGRES_TLS_CLIENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: POSTGRES_QUEUE_DIR.to_owned(),
+            value: EVENT_DEFAULT_DIR.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_QUEUE_LIMIT.to_owned(),
+            value: DEFAULT_LIMIT.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: COMMENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+    ])
+});
+
 pub static DEFAULT_NOTIFY_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(|| {
     KVS(vec![
         KV {
@@ -375,3 +630,64 @@ pub static DEFAULT_NOTIFY_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(|| {
         },
     ])
 });
+
+/// MySQL notification target default configuration
+pub static DEFAULT_NOTIFY_MYSQL_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: ENABLE_KEY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_DSN_STRING.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: MYSQL_TABLE.to_owned(),
+            value: "rustfs_events".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_FORMAT.to_owned(),
+            value: "access".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_TLS_CA.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: MYSQL_TLS_CLIENT_CERT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: MYSQL_TLS_CLIENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: MYSQL_QUEUE_DIR.to_owned(),
+            value: EVENT_DEFAULT_DIR.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_QUEUE_LIMIT.to_owned(),
+            value: DEFAULT_LIMIT.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_MAX_OPEN_CONNECTIONS.to_owned(),
+            value: "2".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: COMMENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+    ])
+});

crates/ecstore/src/data_usage.rs

@@ -15,7 +15,11 @@
 pub mod local_snapshot;
 
 use crate::{
-    bucket::metadata_sys::get_replication_config, config::com::read_config, disk::DiskAPI, error::Error, store::ECStore,
+    bucket::metadata_sys::get_replication_config,
+    config::com::read_config,
+    disk::DiskAPI,
+    error::{Error, classify_system_path_failure_reason},
+    store::ECStore,
     store_api::ListOperations,
 };
 pub use local_snapshot::{
@@ -26,6 +30,7 @@ pub use local_snapshot::{
 use rustfs_common::data_usage::{
     BucketTargetUsageInfo, BucketUsageInfo, DataUsageCache, DataUsageEntry, DataUsageInfo, DiskUsageStatus, SizeSummary,
 };
+use rustfs_io_metrics::record_system_path_failure;
 use rustfs_utils::path::SLASH_SEPARATOR;
 use std::{
     collections::{HashMap, HashSet, hash_map::Entry},
@@ -109,7 +114,16 @@ pub async fn load_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataUsa
     let buf: Vec<u8> = match read_config(store.clone(), &DATA_USAGE_OBJ_NAME_PATH).await {
         Ok(data) => data,
         Err(e) => {
-            error!("Failed to read data usage info from backend: {}", e);
+            let reason = classify_system_path_failure_reason(&e);
+            record_system_path_failure("data_usage", "read_primary", reason);
+            error!(
+                path_kind = "data_usage",
+                operation = "read_primary",
+                reason,
+                object = %DATA_USAGE_OBJ_NAME_PATH.as_str(),
+                error = %e,
+                "system path read failed"
+            );
 
             match read_config(store.clone(), format!("{}.bkp", DATA_USAGE_OBJ_NAME_PATH.as_str()).as_str()).await {
                 Ok(data) => data,
@@ -117,7 +131,16 @@ pub async fn load_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataUsa
                     if e == Error::ConfigNotFound {
                         return Ok(DataUsageInfo::default());
                     }
-                    error!("Failed to read data usage info from backend: {}", e);
+                    let reason = classify_system_path_failure_reason(&e);
+                    record_system_path_failure("data_usage", "read_backup", reason);
+                    error!(
+                        path_kind = "data_usage",
+                        operation = "read_backup",
+                        reason,
+                        object = %format!("{}.bkp", DATA_USAGE_OBJ_NAME_PATH.as_str()),
+                        error = %e,
+                        "system path read failed"
+                    );
                     return Err(Error::other(e));
                 }
             }

crates/ecstore/src/disk/disk_store.rs

@@ -31,7 +31,7 @@ use std::{
     path::PathBuf,
     sync::{
         Arc,
-        atomic::{AtomicI64, AtomicU32, Ordering},
+        atomic::{AtomicI64, AtomicU32, AtomicU64, Ordering},
     },
     time::Duration,
 };
@@ -138,6 +138,14 @@ pub struct DiskHealthTracker {
     pub offline_since_unix_secs: AtomicI64,
     /// Last runtime state transition timestamp
     pub last_transition_unix_secs: AtomicI64,
+    /// Last successfully probed total space in bytes
+    pub last_capacity_total: AtomicU64,
+    /// Last successfully probed used space in bytes
+    pub last_capacity_used: AtomicU64,
+    /// Last successfully probed free space in bytes
+    pub last_capacity_free: AtomicU64,
+    /// Last successful capacity probe timestamp
+    pub last_capacity_probe_unix_secs: AtomicI64,
 }
 
 impl DiskHealthTracker {
@@ -158,6 +166,10 @@ impl DiskHealthTracker {
             consecutive_successes: AtomicU32::new(0),
             offline_since_unix_secs: AtomicI64::new(0),
             last_transition_unix_secs: AtomicI64::new(now / 1_000_000_000),
+            last_capacity_total: AtomicU64::new(0),
+            last_capacity_used: AtomicU64::new(0),
+            last_capacity_free: AtomicU64::new(0),
+            last_capacity_probe_unix_secs: AtomicI64::new(0),
         }
     }
 
@@ -170,6 +182,28 @@ impl DiskHealthTracker {
         self.last_success.store(now, Ordering::Relaxed);
     }
 
+    pub fn record_capacity_probe(&self, total: u64, used: u64, free: u64) {
+        self.last_capacity_total.store(total, Ordering::Release);
+        self.last_capacity_used.store(used, Ordering::Release);
+        self.last_capacity_free.store(free, Ordering::Release);
+        self.last_capacity_probe_unix_secs
+            .store(current_unix_secs() as i64, Ordering::Release);
+    }
+
+    pub fn last_capacity_snapshot(&self) -> Option<(u64, u64, u64, u64)> {
+        let ts = self.last_capacity_probe_unix_secs.load(Ordering::Acquire);
+        if ts <= 0 {
+            return None;
+        }
+
+        Some((
+            self.last_capacity_total.load(Ordering::Acquire),
+            self.last_capacity_used.load(Ordering::Acquire),
+            self.last_capacity_free.load(Ordering::Acquire),
+            ts as u64,
+        ))
+    }
+
     /// Check if disk is faulty
     pub fn is_faulty(&self) -> bool {
         self.status.load(Ordering::Acquire) == DISK_HEALTH_FAULTY
@@ -265,6 +299,26 @@ impl DiskHealthTracker {
         true
     }
 
+    /// Clear faulty/offline state so a store-init format load retry can issue RPC again.
+    ///
+    /// Remote disks are marked faulty on timeout/network errors; the init loop retries with the
+    /// same [`DiskStore`] handles, which would otherwise fail immediately at `is_faulty()`.
+    pub fn reset_for_store_init_retry(&self, endpoint: &Endpoint) {
+        self.status.store(DISK_HEALTH_OK, Ordering::Release);
+        self.runtime_state
+            .store(RuntimeDriveHealthState::Online as u32, Ordering::Release);
+        self.consecutive_failures.store(0, Ordering::Release);
+        self.consecutive_successes.store(0, Ordering::Release);
+        self.offline_since_unix_secs.store(0, Ordering::Release);
+        self.waiting.store(0, Ordering::Release);
+        let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap();
+        let now_nanos = now.as_nanos() as i64;
+        self.last_success.store(now_nanos, Ordering::Relaxed);
+        self.last_started.store(now_nanos, Ordering::Relaxed);
+        self.last_transition_unix_secs.store(now.as_secs() as i64, Ordering::Release);
+        record_drive_runtime_state(endpoint, RuntimeDriveHealthState::Online);
+    }
+
     pub fn mark_recovery_success(&self, endpoint: &Endpoint, reason: &'static str) -> bool {
         let current = self.runtime_state();
         let next = match current {
@@ -443,11 +497,24 @@ impl LocalDiskWrapper {
         self.health.offline_duration().map(|duration| duration.as_secs())
     }
 
+    pub fn last_capacity_snapshot(&self) -> Option<(u64, u64, u64, u64)> {
+        self.health.last_capacity_snapshot()
+    }
+
+    pub fn record_capacity_probe(&self, total: u64, used: u64, free: u64) {
+        self.health.record_capacity_probe(total, used, free);
+    }
+
     #[cfg(test)]
     pub fn force_runtime_state_for_test(&self, state: RuntimeDriveHealthState) {
         self.health.force_runtime_state_for_test(state);
     }
 
+    /// Same as [`DiskHealthTracker::reset_for_store_init_retry`]: undo a transient faulty mark before another format load attempt.
+    pub fn reset_health_for_store_init_retry(&self) {
+        self.health.reset_for_store_init_retry(&self.disk.endpoint());
+    }
+
     /// Enable health monitoring after disk creation.
     /// Used to defer health checks until after startup format loading completes.
     pub fn enable_health_check(&self) {
@@ -796,7 +863,7 @@ impl LocalDiskWrapper {
                     timeout_ms = timeout_duration.as_millis(),
                     "Local disk operation timed out"
                 );
-                Err(DiskError::other(format!("disk operation timeout after {timeout_duration:?}")))
+                Err(DiskError::Timeout)
             }
         }
     }
@@ -1255,4 +1322,21 @@ mod tests {
         assert!(!health.is_faulty());
         assert!(health.offline_duration().is_none());
     }
+
+    #[test]
+    fn reset_for_store_init_retry_clears_faulty_and_back_online() {
+        let endpoint = Endpoint::try_from("/tmp/reset-store-init-retry").expect("endpoint should parse");
+        let health = DiskHealthTracker::new();
+
+        assert!(health.mark_offline(&endpoint, "simulated_fault"));
+        assert!(health.is_faulty());
+        assert_eq!(health.runtime_state(), RuntimeDriveHealthState::Offline);
+
+        health.reset_for_store_init_retry(&endpoint);
+        assert!(!health.is_faulty());
+        assert_eq!(health.runtime_state(), RuntimeDriveHealthState::Online);
+
+        assert!(health.mark_offline(&endpoint, "again"));
+        assert!(health.is_faulty());
+    }
 }

crates/ecstore/src/disk/local.rs

@@ -422,7 +422,7 @@ impl LocalDisk {
                             total: info.total,
                             free: info.free,
                             used: info.used,
-                            used_inodes: info.files - info.ffree,
+                            used_inodes: info.files.saturating_sub(info.ffree),
                             free_inodes: info.ffree,
                             major: info.major,
                             minor: info.minor,
@@ -1247,29 +1247,16 @@ impl LocalDisk {
         W: AsyncWrite + Unpin + Send,
     {
         let forward = {
-            opts.forward_to.as_ref().filter(|v| v.starts_with(&*current)).map(|v| {
-                let forward = v.trim_start_matches(&*current);
-                if let Some(idx) = forward.find('/') {
-                    forward[..idx].to_owned()
-                } else {
-                    forward.to_owned()
-                }
-            })
-            // if let Some(forward_to) = &opts.forward_to {
-
-            // } else {
-            //     None
-            // }
-            // if !opts.forward_to.is_empty() && opts.forward_to.starts_with(&*current) {
-            //     let forward = opts.forward_to.trim_start_matches(&*current);
-            //     if let Some(idx) = forward.find('/') {
-            //         &forward[..idx]
-            //     } else {
-            //         forward
-            //     }
-            // } else {
-            //     ""
-            // }
+            opts.forward_to
+                .as_ref()
+                .and_then(|v| v.strip_prefix(&current))
+                .map(|forward| {
+                    if let Some(idx) = forward.find('/') {
+                        forward[..idx].to_owned()
+                    } else {
+                        forward.to_owned()
+                    }
+                })
         };
 
         if opts.limit > 0 && *objs_returned >= opts.limit {
@@ -1283,6 +1270,7 @@ impl LocalDisk {
             Err(e) => {
                 if e != DiskError::VolumeNotFound && e != Error::FileNotFound {
                     error!("scan list_dir {}, err {:?}", &current, &e);
+                    return Err(e);
                 }
 
                 if opts.report_notfound && e == Error::FileNotFound && current == opts.base_dir {
@@ -2160,6 +2148,8 @@ impl DiskAPI for LocalDisk {
 
     /// Zero-copy file read using memory mapping (Unix) or efficient read (non-Unix).
     /// Returns Bytes that can be shared without copying.
+    // SAFETY: Unix unsafe calls in this function only query page size and mmap
+    // a read-only file region after bounds and alignment are validated.
     #[allow(unsafe_code)]
     #[tracing::instrument(level = "debug", skip(self))]
     async fn read_file_zero_copy(&self, volume: &str, path: &str, offset: usize, length: usize) -> Result<Bytes> {
@@ -2212,6 +2202,8 @@ impl DiskAPI for LocalDisk {
                 // mmap offsets on Unix must be page-size aligned. Align the
                 // mapping down to the nearest page boundary, then slice out the
                 // originally requested logical range.
+                // SAFETY: `sysconf(_SC_PAGESIZE)` has no pointer arguments and
+                // only queries process-global OS configuration.
                 let page_size = unsafe { libc::sysconf(libc::_SC_PAGESIZE) };
                 if page_size <= 0 {
                     return Err(DiskError::other("failed to determine system page size"));
@@ -3233,6 +3225,109 @@ mod test {
         assert_eq!(names.iter().filter(|name| *name == "marker/file.txt").count(), 1);
     }
 
+    #[tokio::test]
+    async fn test_scan_dir_forward_to_repeated_prefix_component() {
+        use rustfs_filemeta::MetacacheReader;
+        use tempfile::tempdir;
+
+        let dir = tempdir().unwrap();
+        let bucket = "test-bucket";
+        let bucket_dir = dir.path().join(bucket);
+
+        for name in [
+            "different/prefix/prefix/repo-0000",
+            "different/prefix/prefix/repo-0001",
+            "different/prefix/prefix/repo-0002",
+            "engineering/alpha-0000",
+            "engineering/engineering/engineering/repo-0000",
+            "engineering/engineering/engineering/repo-0001",
+            "engineering/engineering/repo-0000",
+            "engineering/engineering/repo-0001",
+            "engineering/engineering/repo-0002",
+            "engineering/zulu-0000",
+            "unrelated/engineering/repo-0000",
+        ] {
+            let object_dir = bucket_dir.join(name);
+            fs::create_dir_all(&object_dir).await.unwrap();
+            fs::write(object_dir.join(STORAGE_FORMAT_FILE), b"meta").await.unwrap();
+        }
+
+        let endpoint = Endpoint::try_from(dir.path().to_str().unwrap()).unwrap();
+        let disk = LocalDisk::new(&endpoint, false).await.unwrap();
+
+        async fn scan_names(disk: &LocalDisk, bucket: &str, base_dir: &str, forward_to: &str) -> (Vec<String>, i32) {
+            let (reader, mut writer) = tokio::io::duplex(4096);
+            let mut out = MetacacheWriter::new(&mut writer);
+            let opts = WalkDirOptions {
+                bucket: bucket.to_string(),
+                base_dir: base_dir.to_string(),
+                recursive: true,
+                forward_to: Some(forward_to.to_string()),
+                ..Default::default()
+            };
+            let mut objs_returned = 0;
+
+            disk.scan_dir(base_dir.to_string(), "".to_string(), &opts, &mut out, &mut objs_returned, false)
+                .await
+                .unwrap();
+            out.close().await.unwrap();
+            drop(out);
+            drop(writer);
+
+            let mut reader = MetacacheReader::new(reader);
+            let entries = reader.read_all().await.unwrap();
+            let names: Vec<String> = entries
+                .into_iter()
+                .filter(|entry| !entry.metadata.is_empty())
+                .map(|entry| entry.name)
+                .collect();
+
+            (names, objs_returned)
+        }
+
+        let (engineering_names, engineering_count) =
+            scan_names(&disk, bucket, "engineering/", "engineering/engineering/engineering/repo-0001").await;
+
+        assert_eq!(
+            engineering_names,
+            vec![
+                "engineering/engineering/engineering/repo-0001".to_string(),
+                "engineering/engineering/repo-0000".to_string(),
+                "engineering/engineering/repo-0001".to_string(),
+                "engineering/engineering/repo-0002".to_string(),
+                "engineering/zulu-0000".to_string(),
+            ],
+            "forward_to must resume at the requested triply repeated prefix and preserve lexicographic order"
+        );
+        assert_eq!(engineering_count as usize, engineering_names.len());
+
+        let (different_names, different_count) =
+            scan_names(&disk, bucket, "different/", "different/prefix/prefix/repo-0001").await;
+
+        assert_eq!(
+            different_names,
+            vec![
+                "different/prefix/prefix/repo-0001".to_string(),
+                "different/prefix/prefix/repo-0002".to_string(),
+            ],
+            "forward_to must also work for repeated components unrelated to the engineering prefix"
+        );
+        assert_eq!(different_count as usize, different_names.len());
+
+        let (double_names, double_count) = scan_names(&disk, bucket, "engineering/", "engineering/engineering/repo-0001").await;
+
+        assert_eq!(
+            double_names,
+            vec![
+                "engineering/engineering/repo-0001".to_string(),
+                "engineering/engineering/repo-0002".to_string(),
+                "engineering/zulu-0000".to_string(),
+            ],
+            "forward_to must not skip a child directory whose name repeats the base prefix"
+        );
+        assert_eq!(double_count as usize, double_names.len());
+    }
+
     #[tokio::test]
     async fn test_make_volume() {
         let p = "./testv0";

crates/ecstore/src/disk/mod.rs

@@ -427,6 +427,20 @@ impl Disk {
         }
     }
 
+    pub fn last_capacity_snapshot(&self) -> Option<(u64, u64, u64, u64)> {
+        match self {
+            Disk::Local(local_disk) => local_disk.last_capacity_snapshot(),
+            Disk::Remote(remote_disk) => remote_disk.last_capacity_snapshot(),
+        }
+    }
+
+    pub fn record_capacity_probe(&self, total: u64, used: u64, free: u64) {
+        match self {
+            Disk::Local(local_disk) => local_disk.record_capacity_probe(total, used, free),
+            Disk::Remote(remote_disk) => remote_disk.record_capacity_probe(total, used, free),
+        }
+    }
+
     #[cfg(test)]
     pub fn force_runtime_state_for_test(&self, state: RuntimeDriveHealthState) {
         match self {
@@ -437,6 +451,15 @@ impl Disk {
 }
 
 impl Disk {
+    /// Reset drive health so `connect_load_init_formats` retries are not blocked by a prior
+    /// transient mark-faulty (same disk handles are reused across retries).
+    pub fn reset_health_for_store_init_retry(&self) {
+        match self {
+            Disk::Local(local_disk) => local_disk.reset_health_for_store_init_retry(),
+            Disk::Remote(remote_disk) => remote_disk.reset_health_for_store_init_retry(),
+        }
+    }
+
     /// Enable health monitoring on this disk.
     /// Called after startup format loading completes so that remote peers
     /// have time to come online before being marked as faulty.
@@ -1110,4 +1133,40 @@ mod tests {
         // Clean up the test directory
         let _ = fs::remove_dir_all(&test_dir).await;
     }
+
+    #[tokio::test]
+    async fn reset_health_for_store_init_retry_delegates_to_disk_variants() {
+        let local_dir = tempfile::tempdir().unwrap();
+        let local_path = local_dir.path().to_str().expect("tempdir path should be utf8");
+        let mut local_endpoint = Endpoint::try_from(local_path).expect("local endpoint should parse");
+        local_endpoint.set_pool_index(0);
+        local_endpoint.set_set_index(0);
+        local_endpoint.set_disk_index(0);
+        let local_disk = LocalDisk::new(&local_endpoint, false).await.unwrap();
+        let local_disk = Disk::Local(Box::new(LocalDiskWrapper::new(Arc::new(local_disk), false)));
+
+        let mut remote_endpoint = Endpoint::try_from("http://remote-server:9000/data").expect("remote endpoint should parse");
+        remote_endpoint.set_pool_index(0);
+        remote_endpoint.set_set_index(0);
+        remote_endpoint.set_disk_index(1);
+        let remote_disk = RemoteDisk::new(
+            &remote_endpoint,
+            &DiskOption {
+                cleanup: false,
+                health_check: false,
+            },
+        )
+        .await
+        .unwrap();
+        let remote_disk = Disk::Remote(Box::new(remote_disk));
+
+        for disk in [&local_disk, &remote_disk] {
+            disk.force_runtime_state_for_test(RuntimeDriveHealthState::Offline);
+            assert_eq!(disk.runtime_state(), RuntimeDriveHealthState::Offline);
+
+            disk.reset_health_for_store_init_retry();
+
+            assert_eq!(disk.runtime_state(), RuntimeDriveHealthState::Online);
+        }
+    }
 }

crates/ecstore/src/error.rs

@@ -678,6 +678,18 @@ pub fn is_err_read_quorum(err: &Error) -> bool {
     matches!(err, &StorageError::ErasureReadQuorum)
 }
 
+pub fn classify_system_path_failure_reason(err: &Error) -> &'static str {
+    match err {
+        StorageError::ConfigNotFound => "config_not_found",
+        StorageError::ErasureReadQuorum | StorageError::InsufficientReadQuorum(_, _) => "read_quorum",
+        StorageError::Io(io_err) => match io_err.kind() {
+            std::io::ErrorKind::TimedOut => "timeout",
+            _ => "io",
+        },
+        _ => "other",
+    }
+}
+
 pub fn is_err_invalid_upload_id(err: &Error) -> bool {
     matches!(err, &StorageError::InvalidUploadID(_, _, _))
 }
@@ -1024,6 +1036,28 @@ mod tests {
         assert!(!is_err_not_initialized(&StorageError::DecommissionAlreadyRunning));
     }
 
+    #[test]
+    fn test_classify_system_path_failure_reason() {
+        assert_eq!(classify_system_path_failure_reason(&StorageError::ConfigNotFound), "config_not_found");
+        assert_eq!(classify_system_path_failure_reason(&StorageError::ErasureReadQuorum), "read_quorum");
+        assert_eq!(
+            classify_system_path_failure_reason(&StorageError::InsufficientReadQuorum(
+                "bucket".to_string(),
+                "object".to_string()
+            )),
+            "read_quorum"
+        );
+        assert_eq!(
+            classify_system_path_failure_reason(&StorageError::Io(IoError::new(ErrorKind::TimedOut, "probe"))),
+            "timeout"
+        );
+        assert_eq!(
+            classify_system_path_failure_reason(&StorageError::Io(IoError::new(ErrorKind::PermissionDenied, "probe"))),
+            "io"
+        );
+        assert_eq!(classify_system_path_failure_reason(&StorageError::DiskFull), "other");
+    }
+
     #[test]
     fn test_storage_error_from_disk_error() {
         // Test conversion from DiskError

crates/ecstore/src/notification_sys.rs

@@ -920,4 +920,18 @@ mod tests {
         assert!(msg.contains("1 failure(s)"));
         assert!(msg.contains("peer[0]"));
     }
+
+    #[tokio::test]
+    async fn load_transition_tier_config_reports_unreachable_peers() {
+        let sys = NotificationSys {
+            peer_clients: vec![None],
+            all_peer_clients: Vec::new(),
+        };
+
+        let results = sys.load_transition_tier_config().await;
+        assert_eq!(results.len(), 1);
+        assert!(results[0].host.is_empty());
+        assert!(results[0].err.is_some());
+        assert!(results[0].err.as_ref().unwrap().to_string().contains("peer is not reachable"));
+    }
 }

crates/ecstore/src/pools.rs

@@ -142,6 +142,13 @@ fn ensure_decommission_not_rebalancing(rebalance_running: bool) -> Result<()> {
     Ok(())
 }
 
+fn decommission_meta_bucket_options() -> MakeBucketOptions {
+    MakeBucketOptions {
+        force_create: true,
+        ..Default::default()
+    }
+}
+
 fn is_decommission_active(complete: bool, failed: bool, canceled: bool) -> bool {
     !complete && !failed && !canceled
 }
@@ -2036,9 +2043,10 @@ impl ECStore {
             path_join(&[PathBuf::from(RUSTFS_META_BUCKET), PathBuf::from(BUCKET_META_PREFIX)]),
         ];
 
+        let meta_bucket_opts = decommission_meta_bucket_options();
         for bk in meta_buckets.iter() {
             if let Err(err) = self
-                .make_bucket(bk.to_string_lossy().to_string().as_str(), &MakeBucketOptions::default())
+                .make_bucket(bk.to_string_lossy().to_string().as_str(), &meta_bucket_opts)
                 .await
                 && !is_err_bucket_exists(&err)
             {
@@ -2737,12 +2745,13 @@ mod pools_tests {
     use super::{
         DecomBucketInfo, DecommissionTerminalState, PoolDecommissionInfo, PoolMeta, PoolStatus, bind_decommission_cancelers,
         cancel_decommission_canceler, classify_decommission_terminal_state, count_decommission_item,
-        decommission_cancel_signal_result, decommission_item_size, decommission_start_guard_state, dedup_indices,
-        ensure_decommission_cancel_allowed, ensure_decommission_listing_disks_available, ensure_decommission_not_rebalancing,
-        ensure_decommission_start_allowed, ensure_decommission_terminal_operation_supported,
-        ensure_valid_decommission_pool_index, get_by_index, has_active_decommission_canceler, is_decommission_active,
-        is_decommission_cancel_terminal, load_decommission_entry_versions, mark_decommission_bucket_done,
-        require_decommission_store, resolve_decommission_bucket_done_save_result, resolve_decommission_bucket_state,
+        decommission_cancel_signal_result, decommission_item_size, decommission_meta_bucket_options,
+        decommission_start_guard_state, dedup_indices, ensure_decommission_cancel_allowed,
+        ensure_decommission_listing_disks_available, ensure_decommission_not_rebalancing, ensure_decommission_start_allowed,
+        ensure_decommission_terminal_operation_supported, ensure_valid_decommission_pool_index, get_by_index,
+        has_active_decommission_canceler, is_decommission_active, is_decommission_cancel_terminal,
+        load_decommission_entry_versions, mark_decommission_bucket_done, require_decommission_store,
+        resolve_decommission_bucket_done_save_result, resolve_decommission_bucket_state,
         resolve_decommission_check_after_list_result, resolve_decommission_entry_cleanup_delete_result,
         resolve_decommission_entry_reload_result, resolve_decommission_listing_worker_result,
         resolve_decommission_optional_bucket_config_result, resolve_decommission_pool_meta_reload_result,
@@ -3376,6 +3385,13 @@ mod pools_tests {
         assert!(ensure_decommission_not_rebalancing(false).is_ok());
     }
 
+    #[test]
+    fn test_decommission_meta_bucket_options_are_idempotent() {
+        let opts = decommission_meta_bucket_options();
+
+        assert!(opts.force_create);
+    }
+
     #[test]
     fn test_is_decommission_active_true_only_when_not_terminal() {
         assert!(is_decommission_active(false, false, false));

crates/ecstore/src/rebalance.rs

@@ -780,6 +780,7 @@ impl ECStore {
 
         let cancel_tx = CancellationToken::new();
         let rx = cancel_tx.clone();
+        let mut meta_to_save = None;
 
         {
             let mut rebalance_meta = self.rebalance_meta.write().await;
@@ -792,11 +793,19 @@ impl ECStore {
                 info!("start_rebalance: already in progress, skip duplicate start");
                 return Ok(());
             }
+            if complete_rebalance_pools_at_goal(meta, OffsetDateTime::now_utc()) {
+                meta_to_save = Some(meta.clone());
+            }
             meta.cancel = Some(cancel_tx);
 
             drop(rebalance_meta);
         }
 
+        if let Some(meta) = meta_to_save {
+            let pool = clone_first_arc(self.pools.as_slice(), "start_rebalance: no pools available")?;
+            resolve_rebalance_meta_save_result(meta.save(pool).await, "start_rebalance complete pools at goal")?;
+        }
+
         let participants = if let Some(ref meta) = *self.rebalance_meta.read().await {
             resolve_rebalance_participants(meta.pool_stats.as_slice(), self.pools.len())
         } else {
@@ -1126,6 +1135,30 @@ fn should_pool_participate(init_free_space: u64, init_capacity: u64, percent_fre
     init_capacity > 0 && percent_free_ratio(init_free_space, init_capacity) < percent_free_goal
 }
 
+fn complete_rebalance_pools_at_goal(meta: &mut RebalanceMeta, now: OffsetDateTime) -> bool {
+    let mut changed = false;
+
+    for pool_stat in meta.pool_stats.iter_mut() {
+        if !is_rebalance_pool_started(pool_stat) {
+            continue;
+        }
+
+        if rebalance_goal_reached(
+            pool_stat.init_free_space,
+            pool_stat.init_capacity,
+            pool_stat.bytes,
+            meta.percent_free_goal,
+        ) {
+            pool_stat.info.status = RebalStatus::Completed;
+            pool_stat.info.end_time = Some(now);
+            pool_stat.info.last_error = None;
+            changed = true;
+        }
+    }
+
+    changed
+}
+
 fn resolve_rebalance_worker_result(
     set_idx: usize,
     worker_result: std::result::Result<Result<()>, tokio::task::JoinError>,
@@ -1838,12 +1871,12 @@ mod rebalance_unit_tests {
         GetObjectReader, HTTPRangeSpec, MigrationBackend, MigrationVersionResult, ObjectInfo, ObjectOptions, RebalSaveOpt,
         RebalStatus, RebalanceInfo, RebalanceMeta, RebalanceStats, RebalanceTerminalEvent, apply_rebalance_save_option,
         apply_rebalance_terminal_event, apply_stopped_at, classify_rebalance_terminal_event, clone_arc_by_index, clone_first_arc,
-        clone_rebalance_pool_stats, ensure_rebalance_listing_disks_available, ensure_rebalance_not_decommissioning,
-        ensure_valid_rebalance_pool_index, is_rebalance_stopped_terminal_event, load_rebalance_bucket_configs,
-        mark_rebalance_bucket_done, migrate_entry_version, next_rebal_bucket_from_stat, rebalance_delete_marker_opts,
-        rebalance_meta_load_no_data_error, rebalance_meta_load_unknown_format_error, rebalance_meta_load_unknown_version_error,
-        resolve_load_rebalance_stats_update_result, resolve_next_rebalance_bucket, resolve_rebalance_bucket_error,
-        resolve_rebalance_bucket_result, resolve_rebalance_entry_cleanup_delete_result,
+        clone_rebalance_pool_stats, complete_rebalance_pools_at_goal, ensure_rebalance_listing_disks_available,
+        ensure_rebalance_not_decommissioning, ensure_valid_rebalance_pool_index, is_rebalance_stopped_terminal_event,
+        load_rebalance_bucket_configs, mark_rebalance_bucket_done, migrate_entry_version, next_rebal_bucket_from_stat,
+        rebalance_delete_marker_opts, rebalance_meta_load_no_data_error, rebalance_meta_load_unknown_format_error,
+        rebalance_meta_load_unknown_version_error, resolve_load_rebalance_stats_update_result, resolve_next_rebalance_bucket,
+        resolve_rebalance_bucket_error, resolve_rebalance_bucket_result, resolve_rebalance_entry_cleanup_delete_result,
         resolve_rebalance_file_info_versions_result, resolve_rebalance_meta_load_result, resolve_rebalance_meta_save_result,
         resolve_rebalance_migrate_result_error, resolve_rebalance_optional_bucket_config_result, resolve_rebalance_participants,
         resolve_rebalance_save_task_result, resolve_rebalance_stats_update_result, resolve_rebalance_terminal_error,
@@ -3248,6 +3281,44 @@ mod rebalance_unit_tests {
         assert!(!should_pool_participate(300, 1_000, 0.3));
     }
 
+    #[test]
+    fn test_complete_rebalance_pools_at_goal_marks_started_participants_completed() {
+        let now = OffsetDateTime::from_unix_timestamp(1_000).unwrap();
+        let mut meta = RebalanceMeta {
+            percent_free_goal: 0.5,
+            pool_stats: vec![
+                RebalanceStats {
+                    participating: true,
+                    init_free_space: 400,
+                    init_capacity: 1_000,
+                    bytes: 50,
+                    info: RebalanceInfo {
+                        status: RebalStatus::Started,
+                        ..Default::default()
+                    },
+                    ..Default::default()
+                },
+                RebalanceStats {
+                    participating: true,
+                    init_free_space: 100,
+                    init_capacity: 1_000,
+                    bytes: 0,
+                    info: RebalanceInfo {
+                        status: RebalStatus::Started,
+                        ..Default::default()
+                    },
+                    ..Default::default()
+                },
+            ],
+            ..Default::default()
+        };
+
+        assert!(complete_rebalance_pools_at_goal(&mut meta, now));
+        assert_eq!(meta.pool_stats[0].info.status, RebalStatus::Completed);
+        assert_eq!(meta.pool_stats[0].info.end_time, Some(now));
+        assert_eq!(meta.pool_stats[1].info.status, RebalStatus::Started);
+    }
+
     #[test]
     fn test_should_skip_start_rebalance_only_when_running_and_cancel_attached() {
         assert!(should_skip_start_rebalance(true, true));

crates/ecstore/src/rpc/peer_s3_client.rs

@@ -49,6 +49,32 @@ use tracing::{debug, info, warn};
 
 type Client = Arc<Box<dyn PeerS3Client>>;
 
+fn pool_participant_errors(clients: &[Client], errors: &[Option<Error>], pool_idx: usize) -> Vec<Option<Error>> {
+    clients
+        .iter()
+        .zip(errors.iter())
+        .filter_map(|(client, err)| {
+            if client.get_pools().unwrap_or_default().contains(&pool_idx) {
+                Some(err.clone())
+            } else {
+                None
+            }
+        })
+        .collect()
+}
+
+fn pool_write_quorum(participant_count: usize) -> usize {
+    (participant_count / 2) + 1
+}
+
+fn reduce_pool_write_quorum_errs(per_pool_errs: &[Option<Error>]) -> Option<Error> {
+    if per_pool_errs.is_empty() {
+        return Some(Error::ErasureWriteQuorum);
+    }
+
+    reduce_write_quorum_errs(per_pool_errs, BUCKET_OP_IGNORED_ERRS, pool_write_quorum(per_pool_errs.len()))
+}
+
 #[async_trait]
 pub trait PeerS3Client: Debug + Sync + Send + 'static {
     async fn heal_bucket(&self, bucket: &str, opts: &HealOpts) -> Result<HealResultItem>;
@@ -190,18 +216,8 @@ impl S3PeerSys {
         }
 
         for i in 0..self.pools_count {
-            let mut per_pool_errs = vec![None; self.clients.len()];
-            for (j, cli) in self.clients.iter().enumerate() {
-                let pools = cli.get_pools();
-                let idx = i;
-                if pools.unwrap_or_default().contains(&idx) {
-                    per_pool_errs[j] = errors[j].clone();
-                }
-            }
-
-            if let Some(pool_err) =
-                reduce_write_quorum_errs(&per_pool_errs, BUCKET_OP_IGNORED_ERRS, (per_pool_errs.len() / 2) + 1)
-            {
+            let per_pool_errs = pool_participant_errors(&self.clients, &errors, i);
+            if let Some(pool_err) = reduce_pool_write_quorum_errs(&per_pool_errs) {
                 tracing::error!("make_bucket per_pool_errs: {per_pool_errs:?}");
                 tracing::error!("make_bucket reduce_write_quorum_errs: {pool_err}");
                 return Err(pool_err);
@@ -1031,6 +1047,50 @@ async fn clone_drives() -> Vec<Option<DiskStore>> {
 mod tests {
     use super::*;
 
+    #[derive(Debug)]
+    struct TestPeerS3Client {
+        pools: Option<Vec<usize>>,
+        make_bucket_result: Result<()>,
+    }
+
+    #[async_trait]
+    impl PeerS3Client for TestPeerS3Client {
+        async fn heal_bucket(&self, _bucket: &str, _opts: &HealOpts) -> Result<HealResultItem> {
+            unreachable!("not used by quorum tests")
+        }
+
+        async fn make_bucket(&self, _bucket: &str, _opts: &MakeBucketOptions) -> Result<()> {
+            self.make_bucket_result.clone()
+        }
+
+        async fn list_bucket(&self, _opts: &BucketOptions) -> Result<Vec<BucketInfo>> {
+            unreachable!("not used by quorum tests")
+        }
+
+        async fn delete_bucket(&self, _bucket: &str, _opts: &DeleteBucketOptions) -> Result<()> {
+            unreachable!("not used by quorum tests")
+        }
+
+        async fn get_bucket_info(&self, _bucket: &str, _opts: &BucketOptions) -> Result<BucketInfo> {
+            unreachable!("not used by quorum tests")
+        }
+
+        fn get_pools(&self) -> Option<Vec<usize>> {
+            self.pools.clone()
+        }
+    }
+
+    fn test_peer(pools: &[usize]) -> Client {
+        test_peer_with_make_bucket(pools, Ok(()))
+    }
+
+    fn test_peer_with_make_bucket(pools: &[usize], make_bucket_result: Result<()>) -> Client {
+        Arc::new(Box::new(TestPeerS3Client {
+            pools: Some(pools.to_vec()),
+            make_bucket_result,
+        }))
+    }
+
     fn test_remote_peer(addr: &str) -> RemotePeerS3Client {
         let node = Node {
             url: url::Url::parse(addr).expect("test peer URL should parse"),
@@ -1091,4 +1151,57 @@ mod tests {
 
         client.cancel_token.cancel();
     }
+
+    #[test]
+    fn test_reduce_pool_write_quorum_uses_only_pool_participants() {
+        let clients = vec![
+            test_peer(&[0]),
+            test_peer(&[0]),
+            test_peer(&[0]),
+            test_peer(&[0]),
+            test_peer(&[1]),
+            test_peer(&[1]),
+            test_peer(&[1]),
+            test_peer(&[1]),
+        ];
+        let errors = vec![
+            Some(Error::VolumeExists),
+            Some(Error::VolumeExists),
+            Some(Error::VolumeExists),
+            Some(Error::VolumeExists),
+            None,
+            None,
+            None,
+            None,
+        ];
+
+        let per_pool_errs = pool_participant_errors(&clients, &errors, 0);
+        let err = reduce_pool_write_quorum_errs(&per_pool_errs).expect("all pool participants returned VolumeExists");
+
+        assert_eq!(err, Error::VolumeExists);
+    }
+
+    #[tokio::test]
+    async fn test_make_bucket_reduces_quorum_by_pool_participants() {
+        let peer_sys = S3PeerSys {
+            clients: vec![
+                test_peer_with_make_bucket(&[0], Err(Error::VolumeExists)),
+                test_peer_with_make_bucket(&[0], Err(Error::VolumeExists)),
+                test_peer_with_make_bucket(&[0], Err(Error::VolumeExists)),
+                test_peer_with_make_bucket(&[0], Err(Error::VolumeExists)),
+                test_peer(&[1]),
+                test_peer(&[1]),
+                test_peer(&[1]),
+                test_peer(&[1]),
+            ],
+            pools_count: 2,
+        };
+
+        let err = peer_sys
+            .make_bucket("existing-bucket", &MakeBucketOptions::default())
+            .await
+            .expect_err("existing bucket should surface as VolumeExists, not quorum failure");
+
+        assert_eq!(err, Error::VolumeExists);
+    }
 }

crates/ecstore/src/rpc/remote_disk.rs

@@ -135,11 +135,24 @@ impl RemoteDisk {
         self.health.offline_duration().map(|duration| duration.as_secs())
     }
 
+    pub fn last_capacity_snapshot(&self) -> Option<(u64, u64, u64, u64)> {
+        self.health.last_capacity_snapshot()
+    }
+
+    pub fn record_capacity_probe(&self, total: u64, used: u64, free: u64) {
+        self.health.record_capacity_probe(total, used, free);
+    }
+
     #[cfg(test)]
     pub fn force_runtime_state_for_test(&self, state: RuntimeDriveHealthState) {
         self.health.force_runtime_state_for_test(state);
     }
 
+    /// Same as [`DiskHealthTracker::reset_for_store_init_retry`]: undo a transient faulty mark before another format load attempt.
+    pub fn reset_health_for_store_init_retry(&self) {
+        self.health.reset_for_store_init_retry(&self.endpoint);
+    }
+
     fn spawn_recovery_monitor_if_needed(&self) {
         if !self.health_check {
             return;
@@ -372,7 +385,7 @@ impl RemoteDisk {
                     timeout_ms = timeout_duration.as_millis(),
                     "Remote disk operation timed out"
                 );
-                Err(Error::other(format!("Remote disk operation timeout after {timeout_duration:?}")))
+                Err(DiskError::Timeout)
             }
         }
     }

crates/ecstore/src/set_disk.rs

@@ -88,8 +88,9 @@ use rustfs_utils::http::headers::{
     CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE, EXPIRES, HeaderExt as _,
 };
 use rustfs_utils::http::{
-    SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE, SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_REPLICATION_SSEC_CRC,
-    contains_key_str, get_header_map, get_str, insert_str, remove_header_map,
+    SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER, SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE,
+    SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_REPLICATION_SSEC_CRC, contains_key_str, get_header_map, get_str,
+    insert_str, is_encryption_metadata_key, remove_header_map,
 };
 use rustfs_utils::{
     HashAlgorithm,
@@ -101,7 +102,7 @@ use s3s::header::{X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OB
 use sha2::{Digest, Sha256};
 use std::hash::Hash;
 use std::mem::{self};
-use std::time::{Instant, SystemTime};
+use std::time::{Instant, SystemTime, UNIX_EPOCH};
 use std::{
     collections::{HashMap, HashSet},
     io::{Cursor, Write},
@@ -134,6 +135,13 @@ pub(crate) fn strip_internal_multipart_metadata(metadata: &mut HashMap<String, S
     metadata.remove(RUSTFS_MULTIPART_OBJECT_KEY);
 }
 
+fn should_persist_encryption_original_size(metadata: &HashMap<String, String>) -> bool {
+    metadata.keys().any(|key| is_encryption_metadata_key(key))
+        || metadata.contains_key(SSEC_ALGORITHM_HEADER)
+        || metadata.contains_key(SSEC_KEY_HEADER)
+        || metadata.contains_key(SSEC_KEY_MD5_HEADER)
+}
+
 fn capacity_scope_from_disks(disks: &[Option<DiskStore>]) -> CapacityScope {
     let mut unique = HashSet::with_capacity(disks.len());
     let mut scoped_disks = Vec::with_capacity(disks.len());
@@ -694,7 +702,7 @@ impl ObjectIO for SetDisks {
         let (rd, wd) = tokio::io::duplex(duplex_buffer_size);
         debug!(bucket, object, duplex_buffer_size, "Created duplex pipe for object data transfer");
 
-        let (reader, offset, length) = GetObjectReader::new(Box::new(rd), range, &object_info, opts, &h)?;
+        let (reader, offset, length) = GetObjectReader::new(Box::new(rd), range, &object_info, opts, &h).await?;
 
         // let disks = disks.clone();
         let bucket = bucket.to_owned();
@@ -1929,8 +1937,11 @@ impl ObjectOperations for SetDisks {
             None
         };
 
+        // Use the same full xl.meta read path as GetObject metadata resolution.
+        // This avoids HEAD/GetObject metadata visibility skew immediately after
+        // PutObject/CompleteMultipartUpload.
         let (fi, _, _) = self
-            .get_object_fileinfo(bucket, object, opts, false)
+            .get_object_fileinfo(bucket, object, opts, true)
             .await
             .map_err(|e| to_object_err(e, vec![bucket, object]))?;
 
@@ -3510,16 +3521,22 @@ impl MultipartOperations for SetDisks {
 
         fi.metadata.insert("etag".to_owned(), etag);
 
+        let persist_encryption_original_size = should_persist_encryption_original_size(&fi.metadata);
+
         if opts.replication_request {
             if let Some(actual_size) = get_str(&opts.user_defined, SUFFIX_ACTUAL_OBJECT_SIZE_CAP) {
                 insert_str(&mut fi.metadata, SUFFIX_ACTUAL_SIZE, actual_size.clone());
-                fi.metadata
-                    .insert("x-rustfs-encryption-original-size".to_string(), actual_size);
+                if persist_encryption_original_size {
+                    fi.metadata
+                        .insert("x-rustfs-encryption-original-size".to_string(), actual_size);
+                }
             }
         } else {
             insert_str(&mut fi.metadata, SUFFIX_ACTUAL_SIZE, object_actual_size.to_string());
-            fi.metadata
-                .insert("x-rustfs-encryption-original-size".to_string(), object_actual_size.to_string());
+            if persist_encryption_original_size {
+                fi.metadata
+                    .insert("x-rustfs-encryption-original-size".to_string(), object_actual_size.to_string());
+            }
         }
 
         if fi.is_compressed() {
@@ -4146,56 +4163,78 @@ async fn get_disks_info(disks: &[Option<DiskStore>], eps: &[Endpoint]) -> Vec<ru
         if let Some(disk) = pool {
             let runtime_state = disk.runtime_state();
             let offline_duration_seconds = disk.offline_duration_secs();
-            if runtime_state.should_probe_for_admin() {
+            let capacity_snapshot = disk.last_capacity_snapshot();
+            if runtime_state.should_probe_for_admin()
+                || runtime_state == crate::disk::health_state::RuntimeDriveHealthState::Suspect
+            {
                 match disk.disk_info(&DiskInfoOptions::default()).await {
-                    Ok(res) => ret.push(rustfs_madmin::Disk {
-                        endpoint: eps[i].to_string(),
-                        local: eps[i].is_local,
-                        pool_index: eps[i].pool_idx,
-                        set_index: eps[i].set_idx,
-                        disk_index: eps[i].disk_idx,
-                        state: "ok".to_owned(),
+                    Ok(res) => {
+                        disk.record_capacity_probe(res.total, res.used, res.free);
+                        ret.push(rustfs_madmin::Disk {
+                            endpoint: eps[i].to_string(),
+                            local: eps[i].is_local,
+                            pool_index: eps[i].pool_idx,
+                            set_index: eps[i].set_idx,
+                            disk_index: eps[i].disk_idx,
+                            state: "ok".to_owned(),
 
-                        root_disk: res.root_disk,
-                        drive_path: res.mount_path.clone(),
-                        healing: res.healing,
-                        scanning: res.scanning,
-                        runtime_state: Some(runtime_state.as_str().to_string()),
-                        offline_duration_seconds,
+                            root_disk: res.root_disk,
+                            drive_path: res.mount_path.clone(),
+                            healing: res.healing,
+                            scanning: res.scanning,
+                            runtime_state: Some(runtime_state.as_str().to_string()),
+                            offline_duration_seconds,
+                            capacity_observation_source: Some("live_probe".to_owned()),
+                            capacity_observation_age_seconds: Some(0),
 
-                        uuid: res.id.map_or_else(|| "".to_string(), |id| id.to_string()),
-                        major: res.major as u32,
-                        minor: res.minor as u32,
-                        model: None,
-                        total_space: res.total,
-                        used_space: res.used,
-                        available_space: res.free,
-                        physical_device_ids: (!res.physical_device_ids.is_empty()).then_some(res.physical_device_ids.clone()),
-                        utilization: {
-                            if res.total > 0 {
-                                res.used as f64 / res.total as f64 * 100_f64
-                            } else {
-                                0_f64
-                            }
-                        },
-                        used_inodes: res.used_inodes,
-                        free_inodes: res.free_inodes,
-                        ..Default::default()
-                    }),
-                    Err(err) => ret.push(rustfs_madmin::Disk {
-                        state: err.to_string(),
-                        endpoint: eps[i].to_string(),
-                        local: eps[i].is_local,
-                        pool_index: eps[i].pool_idx,
-                        set_index: eps[i].set_idx,
-                        disk_index: eps[i].disk_idx,
-                        runtime_state: Some(runtime_state.as_str().to_string()),
-                        offline_duration_seconds,
-                        ..Default::default()
-                    }),
+                            uuid: res.id.map_or_else(|| "".to_string(), |id| id.to_string()),
+                            major: res.major as u32,
+                            minor: res.minor as u32,
+                            model: None,
+                            total_space: res.total,
+                            used_space: res.used,
+                            available_space: res.free,
+                            physical_device_ids: (!res.physical_device_ids.is_empty()).then_some(res.physical_device_ids.clone()),
+                            utilization: utilization_percent(res.total, res.used),
+                            used_inodes: res.used_inodes,
+                            free_inodes: res.free_inodes,
+                            ..Default::default()
+                        });
+                    }
+                    Err(err) => {
+                        let mut disk_info = rustfs_madmin::Disk {
+                            state: err.to_string(),
+                            endpoint: eps[i].to_string(),
+                            local: eps[i].is_local,
+                            pool_index: eps[i].pool_idx,
+                            set_index: eps[i].set_idx,
+                            disk_index: eps[i].disk_idx,
+                            runtime_state: Some(runtime_state.as_str().to_string()),
+                            offline_duration_seconds,
+                            ..Default::default()
+                        };
+                        if let Some((total, used, free, _)) = capacity_snapshot {
+                            disk_info.total_space = total;
+                            disk_info.used_space = used;
+                            disk_info.available_space = free;
+                            disk_info.utilization = utilization_percent(total, used);
+                            disk_info.capacity_observation_source = Some("snapshot".to_owned());
+                            disk_info.capacity_observation_age_seconds = capacity_snapshot
+                                .map(|(_, _, _, probe_unix_secs)| capacity_snapshot_age_seconds(probe_unix_secs));
+                        } else {
+                            disk_info.capacity_observation_source = Some("missing".to_owned());
+                            disk_info.capacity_observation_age_seconds = Some(0);
+                        }
+                        ret.push(disk_info);
+                    }
                 }
             } else {
-                ret.push(build_runtime_snapshot_disk(&eps[i], runtime_state, offline_duration_seconds));
+                ret.push(build_runtime_snapshot_disk(
+                    &eps[i],
+                    runtime_state,
+                    offline_duration_seconds,
+                    capacity_snapshot,
+                ));
             }
         } else {
             ret.push(rustfs_madmin::Disk {
@@ -4207,6 +4246,8 @@ async fn get_disks_info(disks: &[Option<DiskStore>], eps: &[Endpoint]) -> Vec<ru
                 runtime_state: None,
                 offline_duration_seconds: None,
                 state: DiskError::DiskNotFound.to_string(),
+                capacity_observation_source: Some("missing".to_owned()),
+                capacity_observation_age_seconds: Some(0),
                 ..Default::default()
             })
         }
@@ -4219,8 +4260,9 @@ fn build_runtime_snapshot_disk(
     endpoint: &Endpoint,
     runtime_state: crate::disk::health_state::RuntimeDriveHealthState,
     offline_duration_seconds: Option<u64>,
+    capacity_snapshot: Option<(u64, u64, u64, u64)>,
 ) -> rustfs_madmin::Disk {
-    rustfs_madmin::Disk {
+    let mut disk = rustfs_madmin::Disk {
         endpoint: endpoint.to_string(),
         local: endpoint.is_local,
         pool_index: endpoint.pool_idx,
@@ -4230,7 +4272,38 @@ fn build_runtime_snapshot_disk(
         runtime_state: Some(runtime_state.as_str().to_string()),
         offline_duration_seconds,
         ..Default::default()
+    };
+
+    if let Some((total, used, free, _)) = capacity_snapshot {
+        disk.total_space = total;
+        disk.used_space = used;
+        disk.available_space = free;
+        disk.utilization = utilization_percent(total, used);
+        disk.capacity_observation_source = Some("snapshot".to_owned());
+        disk.capacity_observation_age_seconds =
+            capacity_snapshot.map(|(_, _, _, probe_unix_secs)| capacity_snapshot_age_seconds(probe_unix_secs));
+    } else {
+        disk.capacity_observation_source = Some("missing".to_owned());
+        disk.capacity_observation_age_seconds = Some(0);
     }
+
+    disk
+}
+
+fn utilization_percent(total: u64, used: u64) -> f64 {
+    if total > 0 {
+        used as f64 / total as f64 * 100_f64
+    } else {
+        0_f64
+    }
+}
+
+fn capacity_snapshot_age_seconds(probe_unix_secs: u64) -> u64 {
+    let now_unix_secs = SystemTime::now()
+        .duration_since(UNIX_EPOCH)
+        .map(|dur| dur.as_secs())
+        .unwrap_or(probe_unix_secs);
+    now_unix_secs.saturating_sub(probe_unix_secs)
 }
 async fn get_storage_info(disks: &[Option<DiskStore>], eps: &[Endpoint]) -> rustfs_madmin::StorageInfo {
     // let mut disks = get_disks_info(disks, eps).await;
@@ -5313,7 +5386,7 @@ mod tests {
     }
 
     #[tokio::test]
-    async fn test_get_disks_info_uses_runtime_snapshot_for_suspect_and_offline_disks() {
+    async fn test_get_disks_info_preserves_runtime_state_for_suspect_and_offline_disks() {
         let format = FormatV3::new(1, 3);
         let mut temp_dirs = Vec::new();
         let mut endpoints = Vec::new();
@@ -5342,15 +5415,87 @@ mod tests {
         assert_eq!(info[0].runtime_state.as_deref(), Some("online"));
         assert!(!info[0].drive_path.is_empty(), "online disk should keep immediate disk_info probe");
 
-        assert_eq!(info[1].state, "suspect");
+        assert_eq!(info[1].state, "ok");
         assert_eq!(info[1].runtime_state.as_deref(), Some("suspect"));
-        assert!(info[1].drive_path.is_empty(), "suspect disk should use runtime snapshot fallback");
+        assert!(!info[1].drive_path.is_empty(), "suspect disk should still probe for fresher disk info");
 
         assert_eq!(info[2].state, "offline");
         assert_eq!(info[2].runtime_state.as_deref(), Some("offline"));
         assert!(info[2].drive_path.is_empty(), "offline disk should use runtime snapshot fallback");
     }
 
+    #[tokio::test]
+    async fn test_get_disks_info_uses_capacity_snapshot_for_offline_disk() {
+        let format = FormatV3::new(1, 1);
+        let (temp_dir, endpoint, disk) = make_formatted_local_disk_for_info_test(0, &format).await;
+        disk.record_capacity_probe(100, 40, 60);
+        disk.force_runtime_state_for_test(RuntimeDriveHealthState::Offline);
+
+        let info = get_disks_info(&[Some(disk)], &[endpoint]).await;
+        assert_eq!(info.len(), 1);
+        assert_eq!(info[0].state, "offline");
+        assert_eq!(info[0].runtime_state.as_deref(), Some("offline"));
+        assert_eq!(info[0].capacity_observation_source.as_deref(), Some("snapshot"));
+        assert!(info[0].capacity_observation_age_seconds.unwrap_or(u64::MAX) <= 60);
+        assert_eq!(info[0].total_space, 100);
+        assert_eq!(info[0].used_space, 40);
+        assert_eq!(info[0].available_space, 60);
+        assert_eq!(info[0].utilization, 40.0);
+
+        drop(temp_dir);
+    }
+
+    #[tokio::test]
+    async fn list_path_returns_read_quorum_when_runtime_candidates_are_empty() {
+        let disk_count = 4;
+        let format = FormatV3::new(1, disk_count);
+        let mut temp_dirs = Vec::with_capacity(disk_count);
+        let mut endpoints = Vec::with_capacity(disk_count);
+        let mut disks = Vec::with_capacity(disk_count);
+
+        for disk_idx in 0..disk_count {
+            let (dir, endpoint, disk) = make_formatted_local_disk_for_info_test(disk_idx, &format).await;
+            temp_dirs.push(dir);
+            endpoints.push(endpoint);
+            disks.push(Some(disk));
+        }
+
+        let set_disks = SetDisks::new(
+            "test-owner".to_string(),
+            Arc::new(RwLock::new(disks)),
+            disk_count,
+            disk_count / 2,
+            0,
+            0,
+            endpoints,
+            format,
+            Vec::new(),
+        )
+        .await;
+
+        for disk in set_disks.get_disks_internal().await.iter().flatten() {
+            disk.force_runtime_state_for_test(RuntimeDriveHealthState::Offline);
+        }
+
+        let (tx, _rx) = tokio::sync::mpsc::channel(1);
+        let err = set_disks
+            .list_path(
+                CancellationToken::new(),
+                crate::store_list_objects::ListPathOptions {
+                    bucket: "bucket".to_string(),
+                    recursive: true,
+                    ..Default::default()
+                },
+                tx,
+            )
+            .await
+            .expect_err("empty runtime candidate set should fail before list_path_raw");
+
+        assert_eq!(err, StorageError::ErasureReadQuorum);
+
+        drop(temp_dirs);
+    }
+
     #[test]
     fn test_dangling_meta_errs_count() {
         // Test counting dangling metadata errors
@@ -5414,6 +5559,36 @@ mod tests {
         assert_eq!(result, None); // No UUID meets quorum of 2
     }
 
+    #[test]
+    fn test_object_quorum_from_meta_returns_not_found_when_all_metadata_is_missing() {
+        let errs = vec![
+            Some(DiskError::FileNotFound),
+            Some(DiskError::VolumeNotFound),
+            Some(DiskError::DiskNotFound),
+            Some(DiskError::FileNotFound),
+        ];
+
+        let err = SetDisks::object_quorum_from_meta(&vec![FileInfo::default(); errs.len()], &errs, 2)
+            .expect_err("missing metadata should map to FileNotFound");
+
+        assert_eq!(err, DiskError::FileNotFound);
+    }
+
+    #[test]
+    fn test_object_quorum_from_meta_preserves_read_quorum_for_mixed_failures() {
+        let errs = vec![
+            Some(DiskError::FileNotFound),
+            Some(DiskError::VolumeNotFound),
+            Some(DiskError::FileCorrupt),
+            Some(DiskError::DiskNotFound),
+        ];
+
+        let err = SetDisks::object_quorum_from_meta(&vec![FileInfo::default(); errs.len()], &errs, 2)
+            .expect_err("mixed metadata failures should keep quorum semantics");
+
+        assert_eq!(err, DiskError::ErasureReadQuorum);
+    }
+
     #[test]
     fn test_shuffle_parts_metadata() {
         // Test metadata shuffling
@@ -5593,6 +5768,20 @@ mod tests {
             .expect("GOVERNANCE shortening with bypass should remain allowed");
     }
 
+    #[test]
+    fn test_should_persist_encryption_original_size_rejects_plain_metadata() {
+        let metadata = HashMap::from([("content-type".to_string(), "application/octet-stream".to_string())]);
+
+        assert!(!should_persist_encryption_original_size(&metadata));
+    }
+
+    #[test]
+    fn test_should_persist_encryption_original_size_accepts_sse_c_metadata() {
+        let metadata = HashMap::from([(SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string())]);
+
+        assert!(should_persist_encryption_original_size(&metadata));
+    }
+
     #[test]
     fn test_should_prevent_write() {
         let oi = ObjectInfo {

crates/ecstore/src/set_disk/lock.rs

@@ -107,101 +107,111 @@ impl SetDisks {
         DriveMembershipSnapshot::from_optional_disks(&disks)
     }
 
+    fn reprobe_runtime_candidates_once(&self, disks: &[DiskStore]) {
+        for disk in disks {
+            if disk.runtime_state() != crate::disk::health_state::RuntimeDriveHealthState::Online {
+                disk.reset_health_for_store_init_retry();
+            }
+        }
+    }
+
     pub async fn get_online_disks_with_healing_and_info(&self, incl_healing: bool) -> (Vec<DiskStore>, Vec<DiskInfo>, usize) {
         let snapshot = self.drive_membership_snapshot().await;
-        let mut disks = snapshot.scanner_heal_candidates().into_iter().map(Some).collect::<Vec<_>>();
+        let mut membership_candidates = snapshot.scanner_heal_candidates();
+        let mut reprobed = false;
 
-        let mut infos: Vec<Option<DiskInfo>> = vec![None; disks.len()];
+        loop {
+            let mut disks = membership_candidates.clone();
+            let mut infos: Vec<Option<DiskInfo>> = vec![None; disks.len()];
 
-        let mut futures = Vec::with_capacity(disks.len());
-        {
-            let mut rng = rand::rng();
-            disks.shuffle(&mut rng);
-        }
+            let mut futures = Vec::with_capacity(disks.len());
+            {
+                let mut rng = rand::rng();
+                disks.shuffle(&mut rng);
+            }
 
-        for (i, disk) in disks.iter().cloned().enumerate() {
-            futures.push(async move {
-                let info = if let Some(disk) = disk {
-                    match disk.disk_info(&DiskInfoOptions::default()).await {
+            for (i, disk) in disks.iter().cloned().enumerate() {
+                futures.push(async move {
+                    let info = match disk.disk_info(&DiskInfoOptions::default()).await {
                         Ok(info) => info,
                         Err(err) => DiskInfo {
                             error: err.to_string(),
                             ..Default::default()
                         },
+                    };
+
+                    Ok((i, info))
+                });
+            }
+
+            let processor = get_global_processors().metadata_processor();
+            let results = processor.execute_batch(futures).await;
+
+            for (submitted_idx, result) in results.into_iter().enumerate() {
+                match result {
+                    Ok((disk_idx, info)) => {
+                        infos[disk_idx] = Some(info);
                     }
-                } else {
-                    DiskInfo {
-                        error: DiskError::DiskNotFound.to_string(),
-                        ..Default::default()
+                    Err(err) => {
+                        infos[submitted_idx] = Some(DiskInfo {
+                            error: err.to_string(),
+                            ..Default::default()
+                        });
                     }
+                }
+            }
+
+            let mut healing: usize = 0;
+
+            let mut scanning_disks = Vec::new();
+            let mut healing_disks = Vec::new();
+            let mut scanning_infos = Vec::new();
+            let mut healing_infos = Vec::new();
+
+            let mut new_disks = Vec::new();
+            let mut new_infos = Vec::new();
+
+            for (disk, info) in disks.into_iter().zip(infos) {
+                let Some(info) = info else {
+                    continue;
                 };
 
-                Ok((i, info))
-            });
-        }
-
-        // Use optimized batch processor for disk info retrieval
-        let processor = get_global_processors().metadata_processor();
-        let results = processor.execute_batch(futures).await;
-
-        for (submitted_idx, result) in results.into_iter().enumerate() {
-            match result {
-                Ok((disk_idx, info)) => {
-                    infos[disk_idx] = Some(info);
-                }
-                Err(err) => {
-                    infos[submitted_idx] = Some(DiskInfo {
-                        error: err.to_string(),
-                        ..Default::default()
-                    });
-                }
-            }
-        }
-
-        let mut healing: usize = 0;
-
-        let mut scanning_disks = Vec::new();
-        let mut healing_disks = Vec::new();
-        let mut scanning_infos = Vec::new();
-        let mut healing_infos = Vec::new();
-
-        let mut new_disks = Vec::new();
-        let mut new_infos = Vec::new();
-
-        for (disk, info) in disks.into_iter().zip(infos) {
-            let Some(info) = info else {
-                continue;
-            };
-
-            if !info.error.is_empty() || disk.is_none() {
-                continue;
-            }
-
-            if info.healing {
-                healing += 1;
-                if incl_healing {
-                    healing_disks.push(disk.unwrap());
-                    healing_infos.push(info);
+                if !info.error.is_empty() {
+                    continue;
                 }
 
-                continue;
+                if info.healing {
+                    healing += 1;
+                    if incl_healing {
+                        healing_disks.push(disk);
+                        healing_infos.push(info);
+                    }
+
+                    continue;
+                }
+
+                if !info.scanning {
+                    new_disks.push(disk);
+                    new_infos.push(info);
+                } else {
+                    scanning_disks.push(disk);
+                    scanning_infos.push(info);
+                }
             }
 
-            if !info.healing {
-                new_disks.push(disk.unwrap());
-                new_infos.push(info);
-            } else {
-                scanning_disks.push(disk.unwrap());
-                scanning_infos.push(info);
+            new_disks.extend(scanning_disks);
+            new_infos.extend(scanning_infos);
+            new_disks.extend(healing_disks);
+            new_infos.extend(healing_infos);
+
+            if !new_disks.is_empty() || membership_candidates.is_empty() || reprobed {
+                return (new_disks, new_infos, healing);
             }
+
+            reprobed = true;
+            self.reprobe_runtime_candidates_once(&membership_candidates);
+            membership_candidates = self.drive_membership_snapshot().await.scanner_heal_candidates();
         }
-
-        new_disks.extend(scanning_disks);
-        new_infos.extend(scanning_infos);
-        new_disks.extend(healing_disks);
-        new_infos.extend(healing_infos);
-
-        (new_disks, new_infos, healing)
     }
 
     pub(super) async fn _get_local_disks(&self) -> Vec<Option<DiskStore>> {
@@ -527,4 +537,50 @@ mod tests {
 
         drop(temp_dirs);
     }
+
+    #[tokio::test]
+    async fn get_online_disks_with_healing_and_info_reprobes_runtime_candidates_once() {
+        let disk_count = 4;
+        let format = FormatV3::new(1, disk_count);
+
+        let mut temp_dirs = Vec::with_capacity(disk_count);
+        let mut endpoints = Vec::with_capacity(disk_count);
+        let mut disks = Vec::with_capacity(disk_count);
+
+        for disk_idx in 0..disk_count {
+            let (temp_dir, endpoint, disk) = make_formatted_local_disk(disk_idx, &format).await;
+            temp_dirs.push(temp_dir);
+            endpoints.push(endpoint);
+            disks.push(Some(disk));
+        }
+
+        let set_disks = SetDisks::new(
+            "test-owner".to_string(),
+            Arc::new(RwLock::new(disks)),
+            disk_count,
+            disk_count / 2,
+            0,
+            0,
+            endpoints,
+            format,
+            Vec::new(),
+        )
+        .await;
+
+        let all_disks = set_disks.get_disks_internal().await;
+        for disk in all_disks.iter().flatten() {
+            disk.force_runtime_state_for_test(crate::disk::health_state::RuntimeDriveHealthState::Returning);
+        }
+
+        let (online_disks, infos, healing) = set_disks.get_online_disks_with_healing_and_info(false).await;
+        assert_eq!(healing, 0);
+        assert_eq!(online_disks.len(), disk_count);
+        assert_eq!(infos.len(), disk_count);
+        assert!(
+            infos.iter().all(|info| info.error.is_empty()),
+            "runtime reprobe should recover a usable candidate set without probe errors"
+        );
+
+        drop(temp_dirs);
+    }
 }

crates/ecstore/src/set_disk/metadata.rs

@@ -15,6 +15,28 @@
 use super::*;
 
 impl SetDisks {
+    pub(super) fn all_not_found_metadata(errs: &[Option<DiskError>]) -> bool {
+        !errs.is_empty()
+            && errs.iter().all(|err| match err {
+                Some(err) => {
+                    matches!(
+                        err,
+                        DiskError::FileNotFound
+                            | DiskError::FileVersionNotFound
+                            | DiskError::VolumeNotFound
+                            | DiskError::DiskNotFound
+                    ) || OBJECT_OP_IGNORED_ERRS.contains(err)
+                }
+                None => false,
+            })
+            && errs.iter().any(|err| {
+                matches!(
+                    err,
+                    Some(DiskError::FileNotFound | DiskError::FileVersionNotFound | DiskError::VolumeNotFound)
+                )
+            })
+    }
+
     pub(super) fn reduce_common_data_dir(data_dirs: &Vec<Option<Uuid>>, write_quorum: usize) -> Option<Uuid> {
         let mut data_dirs_count = HashMap::new();
 
@@ -237,6 +259,10 @@ impl SetDisks {
         errs: &[Option<DiskError>],
         default_parity_count: usize,
     ) -> disk::error::Result<(i32, i32)> {
+        if Self::all_not_found_metadata(errs) {
+            return Err(DiskError::FileNotFound);
+        }
+
         let expected_rquorum = if default_parity_count == 0 {
             parts_metadata.len()
         } else {

crates/ecstore/src/store/init.rs

@@ -16,6 +16,10 @@ use super::*;
 use crate::error::is_err_decommission_running;
 use crate::global::is_first_cluster_node_local;
 
+fn pool_first_endpoint_is_local(pool: &crate::endpoints::PoolEndpoints) -> bool {
+    pool.endpoints.as_ref().first().is_some_and(|endpoint| endpoint.is_local)
+}
+
 fn should_resume_local_decommission(endpoints: &EndpointServerPools, idx: usize) -> Result<bool> {
     let pool = endpoints.as_ref().get(idx).ok_or_else(|| {
         Error::other(format!(
@@ -104,8 +108,6 @@ impl ECStore {
         let mut pools = Vec::with_capacity(endpoint_pools.as_ref().len());
         let mut disk_map = HashMap::with_capacity(endpoint_pools.as_ref().len());
 
-        let first_is_local = endpoint_pools.first_local();
-
         let mut local_disks = Vec::new();
 
         info!("ECStore new address: {}", address.to_string());
@@ -125,6 +127,7 @@ impl ECStore {
         let mut common_parity_drives = 0;
 
         for (i, pool_eps) in endpoint_pools.as_ref().iter().enumerate() {
+            let pool_first_is_local = pool_first_endpoint_is_local(pool_eps);
             if common_parity_drives == 0 {
                 let parity_drives = ec_drives_no_config(pool_eps.drives_per_set)?;
                 storageclass::validate_parity(parity_drives, pool_eps.drives_per_set)?;
@@ -133,14 +136,15 @@ impl ECStore {
 
             // validate_parity(parity_count, pool_eps.drives_per_set)?;
 
-            // Initialize disks without health monitoring so that remote peers
-            // are not immediately marked as faulty before they have a chance to
-            // start up. Health monitoring is enabled after format loading succeeds.
+            // Build disks with health monitoring available, but do not start
+            // periodic monitoring until format loading succeeds. Startup RPC
+            // failures can still spawn recovery probes for peers that come up
+            // after this node.
             let (disks, errs) = store_init::init_disks(
                 &pool_eps.endpoints,
                 &DiskOption {
                     cleanup: true,
-                    health_check: false,
+                    health_check: true,
                 },
             )
             .await;
@@ -152,7 +156,7 @@ impl ECStore {
                 let mut interval = 1;
                 loop {
                     match store_init::connect_load_init_formats(
-                        first_is_local,
+                        pool_first_is_local,
                         &disks,
                         pool_eps.set_count,
                         pool_eps.drives_per_set,
@@ -181,6 +185,11 @@ impl ECStore {
                         _ = sleep(Duration::from_secs(interval)) => {
                         }
                     }
+                    // After waiting for peers, clear transient faulty marks so the next attempt can open RPCs again
+                    // (these `DiskStore` handles are reused; `is_faulty()` would otherwise short-circuit).
+                    for disk in disks.iter().flatten() {
+                        disk.reset_health_for_store_init_retry();
+                    }
                 }
             }?;
 
@@ -366,8 +375,8 @@ impl ECStore {
 #[cfg(test)]
 mod tests {
     use super::{
-        LOCAL_DECOMMISSION_RESUME_MAX_CONFIG_RETRIES, resolve_store_init_stage_result, should_resume_local_decommission,
-        should_retry_local_decommission_resume, wait_for_local_decommission_resume_delay,
+        LOCAL_DECOMMISSION_RESUME_MAX_CONFIG_RETRIES, pool_first_endpoint_is_local, resolve_store_init_stage_result,
+        should_resume_local_decommission, should_retry_local_decommission_resume, wait_for_local_decommission_resume_delay,
     };
     use crate::{
         disk::endpoint::Endpoint,
@@ -464,4 +473,38 @@ mod tests {
         rx.cancel();
         assert!(!wait_for_local_decommission_resume_delay(&rx, Duration::from_secs(1)).await);
     }
+
+    #[test]
+    fn test_pool_first_endpoint_is_local_uses_pool_scope_for_expansion() {
+        let mut remote_endpoint = Endpoint::try_from("http://127.0.0.2:9000/data1").expect("remote endpoint should parse");
+        remote_endpoint.is_local = false;
+
+        let mut local_endpoint = Endpoint::try_from("http://127.0.0.1:9000/data1").expect("local endpoint should parse");
+        local_endpoint.is_local = true;
+
+        let endpoints = EndpointServerPools::from(vec![
+            PoolEndpoints {
+                legacy: false,
+                set_count: 1,
+                drives_per_set: 1,
+                endpoints: Endpoints::from(vec![remote_endpoint]),
+                cmd_line: "pool-0".to_string(),
+                platform: String::new(),
+            },
+            PoolEndpoints {
+                legacy: false,
+                set_count: 1,
+                drives_per_set: 1,
+                endpoints: Endpoints::from(vec![local_endpoint]),
+                cmd_line: "pool-1".to_string(),
+                platform: String::new(),
+            },
+        ]);
+
+        assert!(!endpoints.first_local(), "cluster first endpoint is intentionally remote");
+        assert!(
+            pool_first_endpoint_is_local(endpoints.as_ref().get(1).expect("second pool should exist")),
+            "the expanded pool should be initialized by its own first local endpoint"
+        );
+    }
 }

crates/ecstore/src/store_api/readers.rs

@@ -1,4 +1,79 @@
 use super::*;
+use aes_gcm::{
+    Aes256Gcm, Key, Nonce,
+    aead::{Aead, KeyInit},
+};
+use base64::{Engine, engine::general_purpose::STANDARD as BASE64_STANDARD};
+use md5::{Digest, Md5};
+use rustfs_kms::{service_manager::get_global_encryption_service, types::ObjectEncryptionContext};
+use rustfs_rio::DecryptReader;
+use rustfs_utils::http::{SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER};
+use std::collections::HashMap;
+use std::env;
+
+const INTERNAL_ENCRYPTION_KEY_ID_HEADER: &str = "x-rustfs-encryption-key-id";
+const INTERNAL_ENCRYPTION_KEY_HEADER: &str = "x-rustfs-encryption-key";
+const INTERNAL_ENCRYPTION_IV_HEADER: &str = "x-rustfs-encryption-iv";
+const INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER: &str = "x-rustfs-encryption-original-size";
+const SSEC_ORIGINAL_SIZE_HEADER: &str = "x-amz-server-side-encryption-customer-original-size";
+const DEFAULT_SSE_ALGORITHM: &str = "AES256";
+
+fn part_plaintext_size(part: &ObjectPartInfo) -> i64 {
+    if part.actual_size > 0 {
+        part.actual_size
+    } else {
+        part.size as i64
+    }
+}
+
+fn restore_request_active(opts: &ObjectOptions) -> bool {
+    let restore = &opts.transition.restore_request;
+    restore.type_.is_some() || restore.days.is_some() || restore.output_location.is_some() || restore.select_parameters.is_some()
+}
+
+fn decode_compression_index(index: Option<&bytes::Bytes>) -> Option<rustfs_rio::Index> {
+    let bytes = index?;
+    let mut decoded = rustfs_rio::Index::new();
+    if decoded.load(bytes.as_ref()).is_ok() {
+        Some(decoded)
+    } else {
+        None
+    }
+}
+
+fn get_compressed_offsets(oi: &ObjectInfo, offset: i64) -> (i64, i64, usize, i64, u64) {
+    let mut skip_length = 0_i64;
+    let mut cumulative_actual_size = 0_i64;
+    let mut first_part_idx = 0_usize;
+    let mut compressed_offset = 0_i64;
+
+    for (i, part) in oi.parts.iter().enumerate() {
+        cumulative_actual_size += part_plaintext_size(part);
+        if cumulative_actual_size <= offset {
+            compressed_offset += part.size as i64;
+        } else {
+            first_part_idx = i;
+            skip_length = cumulative_actual_size - part_plaintext_size(part);
+            break;
+        }
+    }
+
+    let mut part_skip = offset - skip_length;
+    let decrypt_skip = 0_i64;
+    let seq_num = 0_u64;
+
+    if part_skip > 0
+        && let Some(part) = oi.parts.get(first_part_idx)
+        && let Some(index) = decode_compression_index(part.index.as_ref())
+        && let Ok((comp_off, uncomp_off)) = index.find(part_skip)
+        && comp_off > 0
+    {
+        compressed_offset += comp_off;
+        part_skip -= uncomp_off;
+    }
+
+    (compressed_offset, part_skip, first_part_idx, decrypt_skip, seq_num)
+}
 
 pub struct PutObjReader {
     pub stream: HashReader,
@@ -46,14 +121,19 @@ pub struct GetObjectReader {
     pub object_info: ObjectInfo,
 }
 
+#[derive(Debug, Clone, Copy)]
+struct EncryptionMaterial {
+    key_bytes: [u8; 32],
+    base_nonce: [u8; 12],
+}
+
 impl GetObjectReader {
-    #[tracing::instrument(level = "debug", skip(reader, rs, opts, _h))]
-    pub fn new(
+    pub async fn new(
         reader: Box<dyn AsyncRead + Unpin + Send + Sync>,
         rs: Option<HTTPRangeSpec>,
         oi: &ObjectInfo,
         opts: &ObjectOptions,
-        _h: &HeaderMap<HeaderValue>,
+        h: &HeaderMap<HeaderValue>,
     ) -> Result<(Self, usize, i64)> {
         let mut rs = rs;
 
@@ -63,25 +143,27 @@ impl GetObjectReader {
             rs = HTTPRangeSpec::from_object_info(oi, part_number);
         }
 
-        // TODO:Encrypted
+        let mut is_encrypted = oi.is_encrypted();
+        let (algo, mut is_compressed) = oi.is_compressed_ok()?;
 
-        let (algo, is_compressed) = oi.is_compressed_ok()?;
+        if restore_request_active(opts) {
+            is_encrypted = false;
+            is_compressed = false;
+        }
 
-        // TODO: check TRANSITION
-
-        if is_compressed {
+        if is_compressed && !is_encrypted {
             let actual_size = oi.get_actual_size()?;
             let (off, length, dec_off, dec_length) = if let Some(rs) = rs {
-                // Support range requests for compressed objects
-                let (dec_off, dec_length) = rs.get_offset_length(actual_size)?;
-                (0, oi.size, dec_off, dec_length)
+                let (req_off, req_length) = rs.get_offset_length(actual_size)?;
+                let (physical_off, decompressed_skip, _, _, _) = get_compressed_offsets(oi, req_off as i64);
+                (physical_off as usize, oi.size - physical_off, decompressed_skip as usize, req_length)
             } else {
                 (0, oi.size, 0, actual_size)
             };
 
             let dec_reader = DecompressReader::new(reader, algo);
 
-            let actual_size_usize = if actual_size > 0 {
+            let actual_size_usize = if actual_size >= 0 {
                 actual_size as usize
             } else {
                 return Err(Error::other(format!("invalid decompressed size {actual_size}")));
@@ -122,6 +204,65 @@ impl GetObjectReader {
             ));
         }
 
+        if is_encrypted {
+            let material = resolve_encryption_material(oi, h).await?;
+            let is_multipart = is_multipart_encrypted_object(&oi.parts, oi.etag.as_deref());
+            let plaintext_size = encrypted_plaintext_size(oi, is_multipart, is_compressed)?;
+            let plaintext_size_usize =
+                usize::try_from(plaintext_size).map_err(|_| Error::other(format!("invalid decrypted size {plaintext_size}")))?;
+            let (plain_offset, plain_length) = if let Some(rs) = rs {
+                rs.get_offset_length(plaintext_size)?
+            } else {
+                (0, plaintext_size)
+            };
+
+            let decrypted_reader: Box<dyn AsyncRead + Unpin + Send + Sync> = if is_multipart {
+                Box::new(DecryptReader::new_multipart(
+                    reader,
+                    material.key_bytes,
+                    material.base_nonce,
+                    multipart_part_numbers(&oi.parts),
+                ))
+            } else {
+                Box::new(DecryptReader::new(reader, material.key_bytes, material.base_nonce))
+            };
+
+            let final_reader: Box<dyn AsyncRead + Unpin + Send + Sync> = if is_compressed {
+                let decompressed_reader = DecompressReader::new(decrypted_reader, algo);
+                if plain_offset > 0 || plain_length != plaintext_size {
+                    Box::new(RangedDecompressReader::new(
+                        decompressed_reader,
+                        plain_offset,
+                        plain_length,
+                        plaintext_size_usize,
+                    )?)
+                } else {
+                    Box::new(LimitReader::new(decompressed_reader, plaintext_size_usize))
+                }
+            } else if plain_offset > 0 || plain_length != plaintext_size {
+                Box::new(RangedDecompressReader::new(
+                    decrypted_reader,
+                    plain_offset,
+                    plain_length,
+                    plaintext_size_usize,
+                )?)
+            } else {
+                Box::new(LimitReader::new(decrypted_reader, plaintext_size_usize))
+            };
+
+            let mut object_info = oi.clone();
+            object_info.size = plain_length;
+
+            return Ok((
+                GetObjectReader {
+                    stream: final_reader,
+                    object_info,
+                },
+                0,
+                oi.size,
+            ));
+        }
+
         if let Some(rs) = rs {
             let (off, length) = rs.get_offset_length(oi.size)?;
 
@@ -188,7 +329,7 @@ impl HTTPRangeSpec {
         for i in 0..part_number {
             let part = &oi.parts[i];
             start = end + 1;
-            end = start + (part.size as i64) - 1;
+            end = start + part_plaintext_size(part) - 1;
         }
 
         Some(HTTPRangeSpec {
@@ -481,12 +622,239 @@ impl<R: AsyncRead + Unpin + Send + 'static> Drop for StreamConsumer<R> {
     }
 }
 
+fn encrypted_plaintext_size(oi: &ObjectInfo, is_multipart: bool, is_compressed: bool) -> Result<i64> {
+    if is_compressed {
+        return oi.get_actual_size().map_err(Into::into);
+    }
+
+    if is_multipart {
+        return Ok(multipart_plaintext_size(&oi.parts, oi.decrypted_size()?));
+    }
+
+    oi.decrypted_size().map_err(Into::into)
+}
+
+fn is_multipart_encrypted_object(parts: &[rustfs_filemeta::ObjectPartInfo], etag: Option<&str>) -> bool {
+    if parts.len() > 1 {
+        return true;
+    }
+
+    etag.map(|etag| etag.trim_matches('"').len() != 32).unwrap_or(false)
+}
+
+fn multipart_plaintext_size(parts: &[rustfs_filemeta::ObjectPartInfo], fallback: i64) -> i64 {
+    let total: i64 = parts.iter().map(part_plaintext_size).sum();
+
+    if total > 0 { total } else { fallback }
+}
+
+fn multipart_part_numbers(parts: &[rustfs_filemeta::ObjectPartInfo]) -> Vec<usize> {
+    parts.iter().map(|part| part.number).collect()
+}
+
+async fn resolve_encryption_material(oi: &ObjectInfo, headers: &HeaderMap<HeaderValue>) -> Result<EncryptionMaterial> {
+    if oi.user_defined.contains_key(SSEC_ALGORITHM_HEADER) {
+        return resolve_ssec_material(oi, headers);
+    }
+
+    if oi.user_defined.contains_key(INTERNAL_ENCRYPTION_KEY_HEADER) {
+        return resolve_managed_material(&oi.user_defined).await;
+    }
+
+    Err(Error::other("encrypted object metadata is incomplete"))
+}
+
+fn resolve_ssec_material(oi: &ObjectInfo, headers: &HeaderMap<HeaderValue>) -> Result<EncryptionMaterial> {
+    let algorithm = headers
+        .get(SSEC_ALGORITHM_HEADER)
+        .ok_or_else(|| Error::other("missing SSE-C algorithm header"))?
+        .to_str()
+        .map_err(|_| Error::other("invalid SSE-C algorithm header"))?;
+    if algorithm != DEFAULT_SSE_ALGORITHM {
+        return Err(Error::other(format!("unsupported SSE-C algorithm {algorithm}")));
+    }
+
+    let key_b64 = headers
+        .get(SSEC_KEY_HEADER)
+        .ok_or_else(|| Error::other("missing SSE-C key header"))?
+        .to_str()
+        .map_err(|_| Error::other("invalid SSE-C key header"))?;
+    let key_md5 = headers
+        .get(SSEC_KEY_MD5_HEADER)
+        .ok_or_else(|| Error::other("missing SSE-C key md5 header"))?
+        .to_str()
+        .map_err(|_| Error::other("invalid SSE-C key md5 header"))?;
+
+    let key_bytes_vec = BASE64_STANDARD
+        .decode(key_b64)
+        .map_err(|_| Error::other("failed to decode SSE-C key"))?;
+    let key_bytes: [u8; 32] = key_bytes_vec
+        .try_into()
+        .map_err(|_| Error::other("SSE-C key must be 32 bytes"))?;
+
+    let expected_md5 = BASE64_STANDARD.encode(md5_bytes(key_bytes));
+    if expected_md5 != key_md5 {
+        return Err(Error::other("SSE-C key MD5 mismatch"));
+    }
+
+    let stored_md5 = oi
+        .user_defined
+        .get(SSEC_KEY_MD5_HEADER)
+        .ok_or_else(|| Error::other("missing stored SSE-C key md5"))?;
+    if stored_md5 != &expected_md5 {
+        return Err(Error::other("SSE-C key does not match object metadata"));
+    }
+
+    Ok(EncryptionMaterial {
+        key_bytes,
+        base_nonce: generate_ssec_nonce(&oi.bucket, &oi.name),
+    })
+}
+
+async fn resolve_managed_material(metadata: &HashMap<String, String>) -> Result<EncryptionMaterial> {
+    let encrypted_dek = metadata
+        .get(INTERNAL_ENCRYPTION_KEY_HEADER)
+        .ok_or_else(|| Error::other("missing managed encrypted DEK"))?;
+    let encrypted_dek = BASE64_STANDARD
+        .decode(encrypted_dek)
+        .map_err(|e| Error::other(format!("failed to decode managed encrypted DEK: {e}")))?;
+
+    let iv_b64 = metadata
+        .get(INTERNAL_ENCRYPTION_IV_HEADER)
+        .ok_or_else(|| Error::other("missing managed encryption IV"))?;
+    let iv = BASE64_STANDARD
+        .decode(iv_b64)
+        .map_err(|e| Error::other(format!("failed to decode managed encryption IV: {e}")))?;
+    let base_nonce: [u8; 12] = iv
+        .as_slice()
+        .try_into()
+        .map_err(|_| Error::other("managed encryption IV must be 12 bytes"))?;
+
+    let kms_key_id = metadata
+        .get(INTERNAL_ENCRYPTION_KEY_ID_HEADER)
+        .map(String::as_str)
+        .unwrap_or("default");
+
+    let key_bytes = if let Some(service) = get_global_encryption_service().await {
+        service
+            .decrypt_data_key(&encrypted_dek, &ObjectEncryptionContext::new(String::new(), String::new()))
+            .await
+            .map_err(|e| Error::other(format!("failed to decrypt managed data key: {e}")))?
+            .plaintext_key
+    } else {
+        decrypt_local_sse_dek(&encrypted_dek, kms_key_id)?
+    };
+
+    Ok(EncryptionMaterial { key_bytes, base_nonce })
+}
+
+fn decrypt_local_sse_dek(encrypted_dek: &[u8], _kms_key_id: &str) -> Result<[u8; 32]> {
+    let encrypted_dek = std::str::from_utf8(encrypted_dek).map_err(|_| Error::other("managed DEK is not valid UTF-8"))?;
+    let parts: Vec<&str> = encrypted_dek.split(':').collect();
+    if parts.len() != 2 {
+        return Err(Error::other("invalid managed DEK format"));
+    }
+
+    let nonce_vec = BASE64_STANDARD
+        .decode(parts[0])
+        .map_err(|_| Error::other("invalid managed DEK nonce"))?;
+    let ciphertext = BASE64_STANDARD
+        .decode(parts[1])
+        .map_err(|_| Error::other("invalid managed DEK ciphertext"))?;
+
+    let nonce_array: [u8; 12] = nonce_vec
+        .as_slice()
+        .try_into()
+        .map_err(|_| Error::other("invalid managed DEK nonce length"))?;
+
+    let key = Key::<Aes256Gcm>::from(local_sse_master_key()?);
+    let cipher = Aes256Gcm::new(&key);
+    let plaintext = cipher
+        .decrypt(&Nonce::from(nonce_array), ciphertext.as_slice())
+        .map_err(|e| Error::other(format!("failed to decrypt managed DEK: {e}")))?;
+
+    plaintext
+        .as_slice()
+        .try_into()
+        .map_err(|_| Error::other("managed DEK has invalid plaintext length"))
+}
+
+fn local_sse_master_key() -> Result<[u8; 32]> {
+    if let Some(key) = decode_master_key_env("__RUSTFS_SSE_SIMPLE_CMK")? {
+        return Ok(key);
+    }
+
+    if let Some(key) = decode_master_key_env("RUSTFS_SSE_S3_MASTER_KEY")? {
+        return Ok(key);
+    }
+
+    Ok([0u8; 32])
+}
+
+fn decode_master_key_env(name: &str) -> Result<Option<[u8; 32]>> {
+    let Ok(value) = env::var(name) else {
+        return Ok(None);
+    };
+
+    let value = value.trim();
+    if value.is_empty() {
+        return Ok(None);
+    }
+
+    let decoded = BASE64_STANDARD
+        .decode(value)
+        .map_err(|e| Error::other(format!("{name} is not valid base64: {e}")))?;
+    let key =
+        <[u8; 32]>::try_from(decoded.as_slice()).map_err(|_| Error::other(format!("{name} must decode to exactly 32 bytes")))?;
+
+    Ok(Some(key))
+}
+
+fn generate_ssec_nonce(bucket: &str, key: &str) -> [u8; 12] {
+    let digest = md5_bytes(format!("{bucket}-{key}").as_bytes());
+    let mut nonce = [0u8; 12];
+    nonce.copy_from_slice(&digest[..12]);
+    nonce
+}
+
+fn md5_bytes(data: impl AsRef<[u8]>) -> [u8; 16] {
+    let digest = Md5::digest(data.as_ref());
+    let mut out = [0u8; 16];
+    out.copy_from_slice(&digest);
+    out
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
+    use base64::Engine;
+    use base64::engine::general_purpose::STANDARD as BASE64_STANDARD;
+    use md5::{Digest, Md5};
     use std::io::Cursor;
+    use temp_env::async_with_vars;
     use tokio::io::AsyncReadExt;
 
+    fn md5_bytes(data: impl AsRef<[u8]>) -> [u8; 16] {
+        let digest = Md5::digest(data.as_ref());
+        let mut bytes = [0u8; 16];
+        bytes.copy_from_slice(&digest);
+        bytes
+    }
+
+    fn ssec_headers_from_key(key_bytes: [u8; 32]) -> HeaderMap<HeaderValue> {
+        let mut headers = HeaderMap::new();
+        headers.insert(rustfs_utils::http::SSEC_ALGORITHM_HEADER, HeaderValue::from_static("AES256"));
+        headers.insert(
+            rustfs_utils::http::SSEC_KEY_HEADER,
+            HeaderValue::from_str(&BASE64_STANDARD.encode(key_bytes)).expect("valid base64 header"),
+        );
+        headers.insert(
+            rustfs_utils::http::SSEC_KEY_MD5_HEADER,
+            HeaderValue::from_str(&BASE64_STANDARD.encode(md5_bytes(key_bytes))).expect("valid md5 header"),
+        );
+        headers
+    }
+
     #[tokio::test]
     async fn test_ranged_decompress_reader() {
         // Create test data
@@ -626,6 +994,76 @@ mod tests {
         assert!(HTTPRangeSpec::from_object_info(&object_info, 4).is_none());
     }
 
+    #[test]
+    fn test_http_range_spec_from_object_info_uses_actual_size() {
+        let object_info = ObjectInfo {
+            size: 90,
+            parts: vec![
+                ObjectPartInfo {
+                    etag: String::new(),
+                    number: 1,
+                    size: 20,
+                    actual_size: 30,
+                    ..Default::default()
+                },
+                ObjectPartInfo {
+                    etag: String::new(),
+                    number: 2,
+                    size: 30,
+                    actual_size: 40,
+                    ..Default::default()
+                },
+                ObjectPartInfo {
+                    etag: String::new(),
+                    number: 3,
+                    size: 40,
+                    actual_size: 50,
+                    ..Default::default()
+                },
+            ],
+            ..Default::default()
+        };
+
+        let spec = HTTPRangeSpec::from_object_info(&object_info, 2).unwrap();
+        assert_eq!(spec.start, 30);
+        assert_eq!(spec.end, 69);
+    }
+
+    #[test]
+    fn test_http_range_spec_from_object_info_falls_back_to_part_size_when_actual_size_missing() {
+        let object_info = ObjectInfo {
+            size: 90,
+            parts: vec![
+                ObjectPartInfo {
+                    etag: String::new(),
+                    number: 1,
+                    size: 20,
+                    actual_size: 0,
+                    ..Default::default()
+                },
+                ObjectPartInfo {
+                    etag: String::new(),
+                    number: 2,
+                    size: 30,
+                    actual_size: 40,
+                    ..Default::default()
+                },
+                ObjectPartInfo {
+                    etag: String::new(),
+                    number: 3,
+                    size: 40,
+                    actual_size: 0,
+                    ..Default::default()
+                },
+            ],
+            ..Default::default()
+        };
+
+        let spec = HTTPRangeSpec::from_object_info(&object_info, 3).unwrap();
+        assert_eq!(spec.start, 60);
+        assert_eq!(spec.end, 99);
+    }
+
     #[tokio::test]
     async fn test_ranged_decompress_reader_zero_length() {
         let original_data = b"Hello, World!";
@@ -676,11 +1114,22 @@ mod tests {
         assert_eq!(&buf2[..1], b"e");
     }
 
-    #[test]
-    fn test_get_object_reader_range_uses_stored_size_for_encrypted_metadata() {
+    fn encrypt_managed_dek_for_test(dek: [u8; 32], master_key: [u8; 32]) -> String {
+        let key = Key::<Aes256Gcm>::from(master_key);
+        let cipher = Aes256Gcm::new(&key);
+        let nonce = Nonce::from([0u8; 12]);
+        let ciphertext = cipher.encrypt(&nonce, dek.as_slice()).expect("encrypt managed dek");
+        format!("{}:{}", BASE64_STANDARD.encode(nonce), BASE64_STANDARD.encode(ciphertext))
+    }
+
+    #[tokio::test]
+    async fn test_get_object_reader_rejects_ssec_read_without_headers() {
         let object_info = ObjectInfo {
             size: 10,
-            user_defined: HashMap::from([("x-amz-server-side-encryption-customer-original-size".to_string(), "20".to_string())]),
+            user_defined: HashMap::from([
+                ("x-amz-server-side-encryption-customer-algorithm".to_string(), "AES256".to_string()),
+                ("x-amz-server-side-encryption-customer-original-size".to_string(), "20".to_string()),
+            ]),
             ..Default::default()
         };
 
@@ -690,24 +1139,26 @@ mod tests {
             end: -1,
         };
 
-        let (_, offset, length) = GetObjectReader::new(
+        let result = GetObjectReader::new(
             Box::new(Cursor::new(b"0123456789".to_vec())),
             Some(range),
             &object_info,
             &ObjectOptions::default(),
             &HeaderMap::new(),
         )
-        .unwrap();
+        .await;
 
-        assert_eq!(offset, 8);
-        assert_eq!(length, 2);
+        assert!(result.is_err());
     }
 
-    #[test]
-    fn test_get_object_reader_suffix_range_uses_stored_size_for_encrypted_metadata() {
+    #[tokio::test]
+    async fn test_get_object_reader_restore_request_bypasses_encryption_range_rewrite() {
         let object_info = ObjectInfo {
             size: 10,
-            user_defined: HashMap::from([("x-rustfs-encryption-original-size".to_string(), "20".to_string())]),
+            user_defined: HashMap::from([
+                ("x-rustfs-encryption-key".to_string(), "encrypted-key".to_string()),
+                ("x-rustfs-encryption-original-size".to_string(), "20".to_string()),
+            ]),
             ..Default::default()
         };
 
@@ -717,16 +1168,341 @@ mod tests {
             end: -1,
         };
 
+        let mut opts = ObjectOptions::default();
+        opts.transition.restore_request.days = Some(1);
+
         let (_, offset, length) = GetObjectReader::new(
             Box::new(Cursor::new(b"0123456789".to_vec())),
             Some(range),
             &object_info,
-            &ObjectOptions::default(),
+            &opts,
             &HeaderMap::new(),
         )
+        .await
         .unwrap();
 
         assert_eq!(offset, 6);
         assert_eq!(length, 4);
     }
+
+    #[tokio::test]
+    async fn test_get_object_reader_allows_encrypted_full_object_passthrough() {
+        async_with_vars([("__RUSTFS_SSE_SIMPLE_CMK", Some(BASE64_STANDARD.encode([0u8; 32])))], async {
+            let plaintext = b"managed-full-object".to_vec();
+            let data_key = [0x21; 32];
+            let base_nonce = [0x11; 12];
+            let encrypted_dek = encrypt_managed_dek_for_test(data_key, [0u8; 32]);
+
+            let mut encrypted = Vec::new();
+            rustfs_rio::EncryptReader::new(Cursor::new(plaintext.clone()), data_key, base_nonce)
+                .read_to_end(&mut encrypted)
+                .await
+                .expect("encrypt managed object");
+
+            let object_info = ObjectInfo {
+                size: encrypted.len() as i64,
+                user_defined: HashMap::from([
+                    ("x-amz-server-side-encryption".to_string(), "AES256".to_string()),
+                    ("x-rustfs-encryption-key".to_string(), BASE64_STANDARD.encode(encrypted_dek.as_bytes())),
+                    ("x-rustfs-encryption-iv".to_string(), BASE64_STANDARD.encode(base_nonce)),
+                    ("x-rustfs-encryption-original-size".to_string(), plaintext.len().to_string()),
+                ]),
+                ..Default::default()
+            };
+
+            let (mut reader, offset, length) = GetObjectReader::new(
+                Box::new(Cursor::new(encrypted.clone())),
+                None,
+                &object_info,
+                &ObjectOptions::default(),
+                &HeaderMap::new(),
+            )
+            .await
+            .expect("managed encrypted full-object reads should decrypt inside ecstore");
+
+            let mut actual = Vec::new();
+            reader.read_to_end(&mut actual).await.expect("read managed plaintext");
+
+            assert_eq!(offset, 0);
+            assert_eq!(length, object_info.size);
+            assert_eq!(reader.object_info.size, plaintext.len() as i64);
+            assert_eq!(actual, plaintext);
+        })
+        .await;
+    }
+
+    #[tokio::test]
+    async fn test_get_object_reader_uses_local_managed_fallback_without_env() {
+        async_with_vars(
+            [
+                ("__RUSTFS_SSE_SIMPLE_CMK", None::<String>),
+                ("RUSTFS_SSE_S3_MASTER_KEY", None::<String>),
+            ],
+            async {
+                let plaintext = b"managed-local-fallback".to_vec();
+                let data_key = [0x22; 32];
+                let base_nonce = [0x12; 12];
+                let encrypted_dek = encrypt_managed_dek_for_test(data_key, [0u8; 32]);
+
+                let mut encrypted = Vec::new();
+                rustfs_rio::EncryptReader::new(Cursor::new(plaintext.clone()), data_key, base_nonce)
+                    .read_to_end(&mut encrypted)
+                    .await
+                    .expect("encrypt managed object with local fallback key");
+
+                let object_info = ObjectInfo {
+                    size: encrypted.len() as i64,
+                    user_defined: HashMap::from([
+                        ("x-amz-server-side-encryption".to_string(), "AES256".to_string()),
+                        ("x-rustfs-encryption-key".to_string(), BASE64_STANDARD.encode(encrypted_dek.as_bytes())),
+                        ("x-rustfs-encryption-iv".to_string(), BASE64_STANDARD.encode(base_nonce)),
+                        ("x-rustfs-encryption-original-size".to_string(), plaintext.len().to_string()),
+                    ]),
+                    ..Default::default()
+                };
+
+                let (mut reader, _, _) = GetObjectReader::new(
+                    Box::new(Cursor::new(encrypted)),
+                    None,
+                    &object_info,
+                    &ObjectOptions::default(),
+                    &HeaderMap::new(),
+                )
+                .await
+                .expect("managed encrypted reads should fall back to the local SSE-S3 key");
+
+                let mut actual = Vec::new();
+                reader.read_to_end(&mut actual).await.expect("read managed plaintext");
+
+                assert_eq!(reader.object_info.size, plaintext.len() as i64);
+                assert_eq!(actual, plaintext);
+            },
+        )
+        .await;
+    }
+
+    #[tokio::test]
+    async fn test_get_object_reader_compressed_range_returns_physical_offset_from_index() {
+        let mut index = rustfs_rio::Index::new();
+        index.add(0, 0).unwrap();
+        index.add(1_048_576, 2_097_152).unwrap();
+
+        let object_info = ObjectInfo {
+            size: 3_000_000,
+            parts: vec![ObjectPartInfo {
+                etag: String::new(),
+                number: 1,
+                size: 3_000_000,
+                actual_size: 4_194_304,
+                index: Some(index.into_vec()),
+                ..Default::default()
+            }],
+            user_defined: HashMap::from([
+                ("x-minio-internal-compression".to_string(), "gzip".to_string()),
+                ("x-minio-internal-actual-size".to_string(), "4194304".to_string()),
+            ]),
+            ..Default::default()
+        };
+
+        let range = HTTPRangeSpec {
+            is_suffix_length: false,
+            start: 2_097_152,
+            end: 2_097_161,
+        };
+
+        let (reader, offset, length) = GetObjectReader::new(
+            Box::new(Cursor::new(Vec::<u8>::new())),
+            Some(range),
+            &object_info,
+            &ObjectOptions::default(),
+            &HeaderMap::new(),
+        )
+        .await
+        .unwrap();
+
+        assert!(offset > 0);
+        assert!(offset < 2_097_152);
+        assert_eq!(length, object_info.size - offset as i64);
+        assert_eq!(reader.object_info.size, 10);
+    }
+
+    #[tokio::test]
+    async fn test_get_object_reader_decrypts_ssec_full_object() {
+        let plaintext = b"ecstore-ssec-full-object".to_vec();
+        let key_bytes = [0x31; 32];
+        let bucket = "bucket";
+        let object = "object";
+        let nonce = md5_bytes(format!("{bucket}-{object}").as_bytes());
+        let mut base_nonce = [0u8; 12];
+        base_nonce.copy_from_slice(&nonce[..12]);
+
+        let mut encrypted = Vec::new();
+        rustfs_rio::EncryptReader::new(Cursor::new(plaintext.clone()), key_bytes, base_nonce)
+            .read_to_end(&mut encrypted)
+            .await
+            .expect("encrypt object");
+
+        let object_info = ObjectInfo {
+            bucket: bucket.to_string(),
+            name: object.to_string(),
+            size: encrypted.len() as i64,
+            user_defined: HashMap::from([
+                ("x-amz-server-side-encryption-customer-algorithm".to_string(), "AES256".to_string()),
+                (
+                    "x-amz-server-side-encryption-customer-key-md5".to_string(),
+                    BASE64_STANDARD.encode(md5_bytes(key_bytes)),
+                ),
+                (
+                    "x-amz-server-side-encryption-customer-original-size".to_string(),
+                    plaintext.len().to_string(),
+                ),
+            ]),
+            ..Default::default()
+        };
+
+        let (mut reader, offset, length) = GetObjectReader::new(
+            Box::new(Cursor::new(encrypted.clone())),
+            None,
+            &object_info,
+            &ObjectOptions::default(),
+            &ssec_headers_from_key(key_bytes),
+        )
+        .await
+        .expect("ssec read should be supported");
+
+        let mut actual = Vec::new();
+        reader.read_to_end(&mut actual).await.expect("read decrypted ssec object");
+
+        assert_eq!(offset, 0);
+        assert_eq!(length, encrypted.len() as i64);
+        assert_eq!(reader.object_info.size, plaintext.len() as i64);
+        assert_eq!(actual, plaintext);
+    }
+
+    #[tokio::test]
+    async fn test_get_object_reader_decrypts_ssec_range_on_plaintext_semantics() {
+        let plaintext = b"0123456789abcdefghijklmnopqrstuvwxyz".to_vec();
+        let key_bytes = [0x41; 32];
+        let bucket = "bucket";
+        let object = "range-object";
+        let nonce = md5_bytes(format!("{bucket}-{object}").as_bytes());
+        let mut base_nonce = [0u8; 12];
+        base_nonce.copy_from_slice(&nonce[..12]);
+
+        let mut encrypted = Vec::new();
+        rustfs_rio::EncryptReader::new(Cursor::new(plaintext.clone()), key_bytes, base_nonce)
+            .read_to_end(&mut encrypted)
+            .await
+            .expect("encrypt ranged object");
+
+        let object_info = ObjectInfo {
+            bucket: bucket.to_string(),
+            name: object.to_string(),
+            size: encrypted.len() as i64,
+            user_defined: HashMap::from([
+                ("x-amz-server-side-encryption-customer-algorithm".to_string(), "AES256".to_string()),
+                (
+                    "x-amz-server-side-encryption-customer-key-md5".to_string(),
+                    BASE64_STANDARD.encode(md5_bytes(key_bytes)),
+                ),
+                (
+                    "x-amz-server-side-encryption-customer-original-size".to_string(),
+                    plaintext.len().to_string(),
+                ),
+            ]),
+            ..Default::default()
+        };
+        let range = HTTPRangeSpec {
+            is_suffix_length: false,
+            start: 5,
+            end: 11,
+        };
+
+        let (mut reader, offset, length) = GetObjectReader::new(
+            Box::new(Cursor::new(encrypted.clone())),
+            Some(range),
+            &object_info,
+            &ObjectOptions::default(),
+            &ssec_headers_from_key(key_bytes),
+        )
+        .await
+        .expect("ssec range read should be supported");
+
+        let mut actual = Vec::new();
+        reader.read_to_end(&mut actual).await.expect("read ranged decrypted object");
+
+        assert_eq!(offset, 0);
+        assert_eq!(length, encrypted.len() as i64);
+        assert_eq!(reader.object_info.size, 7);
+        assert_eq!(actual, b"56789ab");
+    }
+
+    #[tokio::test]
+    async fn test_get_object_reader_decrypts_then_decompresses_before_applying_range() {
+        let plaintext = b"abcdefghijklmnopqrstuvwxyz".to_vec();
+        let key_bytes = [0x51; 32];
+        let bucket = "bucket";
+        let object = "compressed-object";
+        let nonce = md5_bytes(format!("{bucket}-{object}").as_bytes());
+        let mut base_nonce = [0u8; 12];
+        base_nonce.copy_from_slice(&nonce[..12]);
+
+        let mut compressed = Vec::new();
+        rustfs_rio::CompressReader::new(Cursor::new(plaintext.clone()), CompressionAlgorithm::default())
+            .read_to_end(&mut compressed)
+            .await
+            .expect("compress plaintext");
+
+        let mut encrypted = Vec::new();
+        rustfs_rio::EncryptReader::new(Cursor::new(compressed), key_bytes, base_nonce)
+            .read_to_end(&mut encrypted)
+            .await
+            .expect("encrypt compressed plaintext");
+
+        let object_info = ObjectInfo {
+            bucket: bucket.to_string(),
+            name: object.to_string(),
+            size: encrypted.len() as i64,
+            user_defined: HashMap::from([
+                ("x-amz-server-side-encryption-customer-algorithm".to_string(), "AES256".to_string()),
+                (
+                    "x-amz-server-side-encryption-customer-key-md5".to_string(),
+                    BASE64_STANDARD.encode(md5_bytes(key_bytes)),
+                ),
+                (
+                    "x-amz-server-side-encryption-customer-original-size".to_string(),
+                    plaintext.len().to_string(),
+                ),
+                ("x-minio-internal-compression".to_string(), CompressionAlgorithm::default().to_string()),
+                ("x-minio-internal-actual-size".to_string(), plaintext.len().to_string()),
+            ]),
+            ..Default::default()
+        };
+        let range = HTTPRangeSpec {
+            is_suffix_length: false,
+            start: 5,
+            end: 11,
+        };
+
+        let (mut reader, offset, length) = GetObjectReader::new(
+            Box::new(Cursor::new(encrypted.clone())),
+            Some(range),
+            &object_info,
+            &ObjectOptions::default(),
+            &ssec_headers_from_key(key_bytes),
+        )
+        .await
+        .expect("encrypted+compressed range read should be supported");
+
+        let mut actual = Vec::new();
+        reader
+            .read_to_end(&mut actual)
+            .await
+            .expect("read ranged decompressed plaintext");
+
+        assert_eq!(offset, 0);
+        assert_eq!(length, encrypted.len() as i64);
+        assert_eq!(reader.object_info.size, 7);
+        assert_eq!(actual, b"fghijkl");
+    }
 }

crates/ecstore/src/store_api/types.rs

@@ -383,6 +383,37 @@ impl ObjectInfo {
         self.etag.as_ref().is_some_and(|v| v.len() != 32)
     }
 
+    pub fn is_encrypted(&self) -> bool {
+        use rustfs_utils::http::{SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER};
+
+        self.user_defined
+            .keys()
+            .any(|key| rustfs_utils::http::is_encryption_metadata_key(key))
+            || self.user_defined.contains_key(SSEC_ALGORITHM_HEADER)
+            || self.user_defined.contains_key(SSEC_KEY_HEADER)
+            || self.user_defined.contains_key(SSEC_KEY_MD5_HEADER)
+    }
+
+    pub fn encryption_original_size(&self) -> std::io::Result<Option<i64>> {
+        if let Some(size_str) = self
+            .user_defined
+            .get("x-rustfs-encryption-original-size")
+            .or_else(|| self.user_defined.get("x-amz-server-side-encryption-customer-original-size"))
+            && !size_str.is_empty()
+        {
+            let size = size_str
+                .parse::<i64>()
+                .map_err(|e| std::io::Error::other(format!("Failed to parse encryption original size: {e}")))?;
+            return Ok(Some(size));
+        }
+
+        Ok(None)
+    }
+
+    pub fn decrypted_size(&self) -> std::io::Result<i64> {
+        Ok(self.encryption_original_size()?.unwrap_or(self.size))
+    }
+
     pub fn get_actual_size(&self) -> std::io::Result<i64> {
         if self.actual_size > 0 {
             return Ok(self.actual_size);
@@ -410,15 +441,7 @@ impl ObjectInfo {
         // Check if object is encrypted
         // Managed SSE stores original size in x-rustfs-encryption-original-size metadata
         // SSE-C stores original size in x-amz-server-side-encryption-customer-original-size
-        if let Some(size_str) = self
-            .user_defined
-            .get("x-rustfs-encryption-original-size")
-            .or_else(|| self.user_defined.get("x-amz-server-side-encryption-customer-original-size"))
-            && !size_str.is_empty()
-        {
-            let size = size_str
-                .parse::<i64>()
-                .map_err(|e| std::io::Error::other(format!("Failed to parse encryption original size: {e}")))?;
+        if let Some(size) = self.encryption_original_size()? {
             return Ok(size);
         }
 

crates/ecstore/src/store_list_objects.rs

@@ -40,7 +40,7 @@ use std::sync::Arc;
 use tokio::sync::broadcast::{self};
 use tokio::sync::mpsc::{self, Receiver, Sender};
 use tokio_util::sync::CancellationToken;
-use tracing::{error, info};
+use tracing::{error, info, warn};
 use uuid::Uuid;
 
 const MAX_OBJECT_LIST: i32 = 1000;
@@ -50,6 +50,43 @@ const MAX_OBJECT_LIST: i32 = 1000;
 
 const METACACHE_SHARE_PREFIX: bool = false;
 
+fn normalize_max_keys(max_keys: i32) -> i32 {
+    max_keys.min(MAX_OBJECT_LIST)
+}
+
+fn ensure_non_empty_listing_disks(bucket: &str, path: &str, disks: &[DiskStore]) -> Result<()> {
+    if disks.is_empty() {
+        warn!(
+            bucket = %bucket,
+            path = %path,
+            "listing candidate disks collapsed to empty set"
+        );
+        return Err(StorageError::ErasureReadQuorum);
+    }
+
+    Ok(())
+}
+
+fn walk_result_from_set_errors(errs: &[Option<Error>]) -> Result<()> {
+    if is_all_not_found(errs) {
+        if is_all_volume_not_found(errs) {
+            return Err(StorageError::VolumeNotFound);
+        }
+
+        return Ok(());
+    }
+
+    for err in errs.iter().flatten() {
+        if err == &Error::Unexpected || err.is_not_found() {
+            continue;
+        }
+
+        return Err(err.clone());
+    }
+
+    Ok(())
+}
+
 pub fn max_keys_plus_one(max_keys: i32, add_one: bool) -> i32 {
     let mut max_keys = max_keys;
     if !(0..=MAX_OBJECT_LIST).contains(&max_keys) {
@@ -200,7 +237,7 @@ impl ListPathOptions {
                 MARKER_TAG_VERSION,
                 id.to_owned(),
                 self.pool_idx.unwrap_or_default(),
-                self.pool_idx.unwrap_or_default(),
+                self.set_idx.unwrap_or_default(),
             )
         } else {
             format!("{marker}[rustfs_cache:{MARKER_TAG_VERSION},return:]")
@@ -254,6 +291,7 @@ impl ECStore {
         max_keys: i32,
         incl_deleted: bool,
     ) -> Result<ListObjectsInfo> {
+        let max_keys = normalize_max_keys(max_keys);
         let effective_max_keys = if max_keys <= 0 { 0 } else { max_keys_plus_one(max_keys, true) };
         let opts = ListPathOptions {
             bucket: bucket.to_owned(),
@@ -302,7 +340,7 @@ impl ECStore {
                 ..Default::default()
             });
 
-        if let Some(err) = list_result.err.clone()
+        if let Some(err) = list_result.err.take()
             && err != rustfs_filemeta::Error::Unexpected
         {
             return Err(to_object_err(err.into(), vec![bucket, prefix]));
@@ -377,6 +415,7 @@ impl ECStore {
         delimiter: Option<String>,
         max_keys: i32,
     ) -> Result<ListObjectVersionsInfo> {
+        let max_keys = normalize_max_keys(max_keys);
         if marker.is_none() && version_marker.is_some() {
             return Err(StorageError::NotImplemented);
         }
@@ -414,7 +453,7 @@ impl ECStore {
                 ..Default::default()
             });
 
-        if let Some(err) = list_result.err.clone()
+        if let Some(err) = list_result.err.take()
             && err != rustfs_filemeta::Error::Unexpected
         {
             return Err(to_object_err(err.into(), vec![bucket, prefix]));
@@ -599,6 +638,11 @@ impl ECStore {
         // wait spawns exit
         join_all(vec![job1, job2]).await;
 
+        if let Ok(err) = err_rx.try_recv() {
+            error!("list_path err_rx.try_recv() ok {:?}", &err);
+            result.err = Some(err.as_ref().clone().into());
+        }
+
         if result.err.is_some() {
             return Ok(result);
         }
@@ -761,6 +805,7 @@ impl ECStore {
                     };
 
                     let path = base_dir_from_prefix(prefix);
+                    ensure_non_empty_listing_disks(bucket, &path, &disks)?;
 
                     let mut filter_prefix = {
                         prefix
@@ -942,19 +987,26 @@ impl ECStore {
         tokio::spawn(async move { merge_entry_channels(rx, inputs, merge_tx, 1).await });
 
         let walk_results = join_all(futures).await;
-        for (idx, walk_result) in walk_results.into_iter().enumerate() {
-            if let Err(err) = walk_result {
-                error!(
-                    bucket = %bucket,
-                    prefix = %prefix,
-                    set_task_index = idx,
-                    error = ?err,
-                    "walk_internal list_path_raw task failed"
-                );
+        let mut errs = Vec::new();
+        for walk_result in walk_results {
+            match walk_result {
+                Ok(()) => errs.push(None),
+                Err(err) => errs.push(Some(err.into())),
             }
         }
 
-        Ok(())
+        let result = walk_result_from_set_errors(&errs);
+        if let Err(err) = &result {
+            error!(
+                bucket = %bucket,
+                prefix = %prefix,
+                error = ?err,
+                set_errors = ?errs,
+                "walk_internal list_path_raw tasks failed"
+            );
+        }
+
+        result
     }
 }
 
@@ -1253,6 +1305,7 @@ impl SetDisks {
         }
 
         let listing_quorum = ((ask_disks + 1) / 2) as usize;
+        ensure_non_empty_listing_disks(&opts.bucket, &opts.base_dir, &disks)?;
 
         let mut fallback_disks = Vec::new();
 
@@ -1325,7 +1378,7 @@ impl SetDisks {
             },
         )
         .await
-        .map_err(Error::other)
+        .map_err(Error::from)
     }
 }
 
@@ -1390,8 +1443,19 @@ fn calc_common_counter(infos: &[DiskInfo], read_quorum: usize) -> u64 {
 
 #[cfg(test)]
 mod test {
+    use super::{ListPathOptions, MAX_OBJECT_LIST, max_keys_plus_one, walk_result_from_set_errors};
+    use crate::error::StorageError;
     use uuid::Uuid;
 
+    #[test]
+    fn test_max_keys_plus_one_caps_before_lookahead() {
+        assert_eq!(max_keys_plus_one(999, true), 1000);
+        assert_eq!(max_keys_plus_one(MAX_OBJECT_LIST, true), MAX_OBJECT_LIST + 1);
+        assert_eq!(max_keys_plus_one(MAX_OBJECT_LIST + 1, true), MAX_OBJECT_LIST + 1);
+        assert_eq!(max_keys_plus_one(i32::MAX, true), MAX_OBJECT_LIST + 1);
+        assert_eq!(max_keys_plus_one(-1, true), MAX_OBJECT_LIST + 1);
+    }
+
     /// Test that "null" version marker is handled correctly
     /// AWS S3 API uses "null" string to represent non-versioned objects
     #[test]
@@ -1462,6 +1526,75 @@ mod test {
         assert_eq!(parsed.unwrap().to_string(), uuid_str);
     }
 
+    #[test]
+    fn list_path_marker_round_trip_preserves_set_index() {
+        let mut opts = ListPathOptions {
+            id: Some("list-cache-id".to_string()),
+            pool_idx: Some(3),
+            set_idx: Some(7),
+            ..Default::default()
+        };
+
+        let marker = opts.encode_marker("photos/2026/image.jpg");
+        let expected_marker = format!(
+            "photos/2026/image.jpg[rustfs_cache:{},id:list-cache-id,p:3,s:7]",
+            super::MARKER_TAG_VERSION
+        );
+        assert_eq!(marker, expected_marker);
+
+        let mut parsed = ListPathOptions {
+            marker: Some(marker),
+            ..Default::default()
+        };
+        parsed.parse_marker();
+
+        assert_eq!(parsed.marker.as_deref(), Some("photos/2026/image.jpg"));
+        assert_eq!(parsed.id.as_deref(), Some("list-cache-id"));
+        assert_eq!(parsed.pool_idx, Some(3));
+        assert_eq!(parsed.set_idx, Some(7));
+        assert!(!parsed.create);
+    }
+
+    #[test]
+    fn walk_result_from_set_errors_returns_non_eof_error() {
+        let err = walk_result_from_set_errors(&[Some(StorageError::Unexpected), Some(StorageError::FileAccessDenied)])
+            .expect_err("walk should fail when any set reports a real listing error");
+
+        assert_eq!(err, StorageError::FileAccessDenied);
+    }
+
+    #[test]
+    fn walk_result_from_set_errors_prefers_real_error_over_not_found() {
+        let err = walk_result_from_set_errors(&[
+            Some(StorageError::VolumeNotFound),
+            Some(StorageError::DiskNotFound),
+            Some(StorageError::FileAccessDenied),
+        ])
+        .expect_err("walk should report the real listing error");
+
+        assert_eq!(err, StorageError::FileAccessDenied);
+    }
+
+    #[test]
+    fn walk_result_from_set_errors_preserves_volume_not_found() {
+        let err = walk_result_from_set_errors(&[Some(StorageError::VolumeNotFound), Some(StorageError::VolumeNotFound)])
+            .expect_err("all volume-not-found set errors should remain visible");
+
+        assert_eq!(err, StorageError::VolumeNotFound);
+    }
+
+    #[test]
+    fn walk_result_from_set_errors_allows_missing_entries() {
+        walk_result_from_set_errors(&[Some(StorageError::FileNotFound), Some(StorageError::VolumeNotFound)])
+            .expect("missing objects under an existing listing path should not fail the walk");
+    }
+
+    #[test]
+    fn walk_result_from_set_errors_ignores_only_unexpected_and_successes() {
+        walk_result_from_set_errors(&[None, Some(StorageError::Unexpected)])
+            .expect("successful sets and unexpected EOF-style markers should not fail the walk");
+    }
+
     // use std::sync::Arc;
 
     // use crate::cache_value::metacache_set::list_path_raw;

crates/iam/Cargo.toml

@@ -48,6 +48,7 @@ jsonwebtoken = { workspace = true }
 tracing.workspace = true
 rustfs-madmin.workspace = true
 rustfs-utils = { workspace = true, features = ["path"] }
+rustfs-io-metrics.workspace = true
 tokio-util.workspace = true
 pollster.workspace = true
 reqwest = { workspace = true }

crates/iam/src/lib.rs

@@ -48,7 +48,7 @@ pub async fn init_iam_sys(ecstore: Arc<ECStore>) -> Result<()> {
 
     // 2. Create the cache manager.
     // The `new` method now performs a blocking initial load from disk.
-    let cache_manager = IamCache::new(storage_adapter).await;
+    let cache_manager = IamCache::new(storage_adapter).await?;
 
     // 3. Construct the system interface
     let iam_instance = Arc::new(IamSys::new(cache_manager));

crates/iam/src/manager.rs

@@ -58,6 +58,10 @@ use tracing::{error, info};
 
 const IAM_FORMAT_FILE: &str = "format.json";
 const IAM_FORMAT_VERSION_1: i32 = 1;
+#[cfg(not(test))]
+const INITIAL_LOAD_RETRY_DELAY: Duration = Duration::from_secs(1);
+#[cfg(test)]
+const INITIAL_LOAD_RETRY_DELAY: Duration = Duration::from_millis(1);
 
 #[derive(Serialize, Deserialize)]
 struct IAMFormat {
@@ -116,7 +120,7 @@ where
     ///
     /// # Returns
     /// An Arc-wrapped instance of IamSystem
-    pub(crate) async fn new(api: T) -> Arc<Self> {
+    pub(crate) async fn new(api: T) -> Result<Arc<Self>> {
         let (sender, receiver) = mpsc::channel::<i64>(100);
 
         let sys = Arc::new(Self {
@@ -132,8 +136,8 @@ where
             last_sync_duration_millis: AtomicU64::new(0),
         });
 
-        sys.clone().init(receiver).await.unwrap();
-        sys
+        sys.clone().init(receiver).await?;
+        Ok(sys)
     }
 
     /// Initialize the IAM system
@@ -144,19 +148,26 @@ where
 
         // Critical: Load all existing users/policies into memory cache
         const MAX_RETRIES: usize = 3;
+        let mut load_error = None;
         for attempt in 0..MAX_RETRIES {
             if let Err(e) = self.clone().load().await {
                 if attempt == MAX_RETRIES - 1 {
                     self.state.store(IamState::Error as u8, Ordering::SeqCst);
                     warn!("IAM failed to load initial data after {} attempts: {:?}", MAX_RETRIES, e);
+                    load_error = Some(e);
                 } else {
                     warn!("IAM load failed, retrying... attempt {}", attempt + 1);
-                    tokio::time::sleep(Duration::from_secs(1)).await;
+                    tokio::time::sleep(INITIAL_LOAD_RETRY_DELAY).await;
                 }
             } else {
                 break;
             }
         }
+
+        if let Some(err) = load_error {
+            return Err(err);
+        }
+
         self.state.store(IamState::Ready as u8, Ordering::SeqCst);
         info!("IAM System successfully initialized and marked as READY");
 
@@ -2024,6 +2035,156 @@ mod tests {
     use serde_json::json;
     use std::collections::HashMap;
 
+    #[derive(Clone)]
+    struct FailingInitialLoadStore;
+
+    #[async_trait::async_trait]
+    impl Store for FailingInitialLoadStore {
+        fn has_watcher(&self) -> bool {
+            false
+        }
+
+        async fn save_iam_config<Item: Serialize + Send>(&self, _item: Item, _path: impl AsRef<str> + Send) -> Result<()> {
+            Ok(())
+        }
+
+        async fn load_iam_config<Item: serde::de::DeserializeOwned>(&self, _path: impl AsRef<str> + Send) -> Result<Item> {
+            Err(Error::ConfigNotFound)
+        }
+
+        async fn delete_iam_config(&self, _path: impl AsRef<str> + Send) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn save_user_identity(
+            &self,
+            _name: &str,
+            _user_type: UserType,
+            _item: UserIdentity,
+            _ttl: Option<usize>,
+        ) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn delete_user_identity(&self, _name: &str, _user_type: UserType) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn load_user_identity(&self, _name: &str, _user_type: UserType) -> Result<UserIdentity> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn load_user(&self, _name: &str, _user_type: UserType, _m: &mut HashMap<String, UserIdentity>) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn load_users(&self, _user_type: UserType, _m: &mut HashMap<String, UserIdentity>) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn load_secret_key(&self, _name: &str, _user_type: UserType) -> Result<String> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn save_group_info(&self, _name: &str, _item: GroupInfo) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn delete_group_info(&self, _name: &str) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn load_group(&self, _name: &str, _m: &mut HashMap<String, GroupInfo>) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn load_groups(&self, _m: &mut HashMap<String, GroupInfo>) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn save_policy_doc(&self, _name: &str, _item: PolicyDoc) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn delete_policy_doc(&self, _name: &str) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn load_policy(&self, _name: &str) -> Result<PolicyDoc> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn load_policy_doc(&self, _name: &str, _m: &mut HashMap<String, PolicyDoc>) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn load_policy_docs(&self, _m: &mut HashMap<String, PolicyDoc>) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn save_mapped_policy(
+            &self,
+            _name: &str,
+            _user_type: UserType,
+            _is_group: bool,
+            _item: MappedPolicy,
+            _ttl: Option<usize>,
+        ) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn delete_mapped_policy(&self, _name: &str, _user_type: UserType, _is_group: bool) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn load_mapped_policy(
+            &self,
+            _name: &str,
+            _user_type: UserType,
+            _is_group: bool,
+            _m: &mut HashMap<String, MappedPolicy>,
+        ) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn load_mapped_policies(
+            &self,
+            _user_type: UserType,
+            _is_group: bool,
+            _m: &mut HashMap<String, MappedPolicy>,
+        ) -> Result<()> {
+            Err(Error::InvalidArgument)
+        }
+
+        async fn load_all(&self, _cache: &Cache) -> Result<()> {
+            Err(Error::Io(std::io::Error::other("initial load failed")))
+        }
+    }
+
+    #[tokio::test]
+    async fn test_init_keeps_error_state_when_initial_load_fails() {
+        let (sender, receiver) = mpsc::channel::<i64>(1);
+        let sys = Arc::new(IamCache {
+            api: FailingInitialLoadStore,
+            cache: Cache::default(),
+            state: Arc::new(AtomicU8::new(IamState::Uninitialized as u8)),
+            loading: Arc::new(AtomicBool::new(false)),
+            send_chan: sender,
+            roles: HashMap::new(),
+            last_timestamp: AtomicI64::new(0),
+            sync_failures: AtomicU64::new(0),
+            sync_successes: AtomicU64::new(0),
+            last_sync_duration_millis: AtomicU64::new(0),
+        });
+
+        let result = Arc::clone(&sys).init(receiver).await;
+
+        assert!(matches!(result, Err(Error::Io(_))));
+        assert!(!sys.is_ready());
+        assert_eq!(sys.state.load(Ordering::SeqCst), IamState::Error as u8);
+        assert_eq!(sys.sync_failures.load(Ordering::Relaxed), 3);
+    }
+
     #[test]
     fn test_iam_format_new_version_1() {
         let format = IAMFormat::new_version_1();

crates/iam/src/store/object.rs

@@ -22,7 +22,7 @@ use crate::{
 };
 use futures::future::join_all;
 use rustfs_credentials::get_global_action_cred;
-use rustfs_ecstore::error::StorageError;
+use rustfs_ecstore::error::{StorageError, classify_system_path_failure_reason};
 use rustfs_ecstore::store_api::{ListOperations as _, ObjectInfoOrErr, WalkOptions};
 use rustfs_ecstore::{
     config::{
@@ -32,6 +32,7 @@ use rustfs_ecstore::{
     store::ECStore,
     store_api::{HTTPPreconditions, ObjectInfo, ObjectOptions},
 };
+use rustfs_io_metrics::record_system_path_failure;
 use rustfs_policy::{auth::UserIdentity, policy::PolicyDoc};
 use rustfs_utils::path::{SLASH_SEPARATOR, path_join_buf};
 use serde::{Serialize, de::DeserializeOwned};
@@ -366,11 +367,16 @@ impl ObjectStore {
                 .walk(ctx.clone(), Self::BUCKET_NAME, &path, tx, WalkOptions::default())
                 .await
             {
+                let reason = classify_system_path_failure_reason(&err);
+                record_system_path_failure("iam_config", "walk", reason);
                 error!(
+                    path_kind = "iam_config",
+                    operation = "walk",
+                    reason,
                     bucket = Self::BUCKET_NAME,
                     prefix = %path,
-                    error = ?err,
-                    "list_iam_config_items walk task failed"
+                    error = %err,
+                    "system path walk failed"
                 );
             }
         });
@@ -418,7 +424,14 @@ impl ObjectStore {
 
         while let Some(v) = rx.recv().await {
             if let Some(err) = v.err {
-                warn!("list_iam_config_items {:?}", err);
+                warn!(
+                    path_kind = "iam_config",
+                    operation = "list_items",
+                    reason = "walk_result",
+                    error = %err,
+                    "system path list failed"
+                );
+                record_system_path_failure("iam_config", "list_items", "walk_result");
                 ctx.cancel();
 
                 return Err(err);

crates/iam/src/sys.rs

@@ -838,6 +838,32 @@ impl<T: Store> IamSys<T> {
         !policy.is_empty() && policy.chars().all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
     }
 
+    // JWT policy claims carry canned policy names only; policy documents are resolved by IAM store.
+    fn safe_claim_policy_names(claims: &HashMap<String, Value>, parent_user: &str) -> Vec<String> {
+        let Some(claim_policies) = claims.get(POLICYNAME).and_then(|v| v.as_str()) else {
+            return Vec::new();
+        };
+
+        claim_policies
+            .split(',')
+            .map(str::trim)
+            .filter(|policy_name| {
+                if policy_name.is_empty() {
+                    return false;
+                }
+                if !Self::is_safe_claim_policy_name(policy_name) {
+                    tracing::debug!(
+                        parent_user = %parent_user,
+                        "prepare_sts_auth: ignoring unsafe policy name in STS policy claim"
+                    );
+                    return false;
+                }
+                true
+            })
+            .map(ToOwned::to_owned)
+            .collect()
+    }
+
     /// Compatibility wrapper for service-account authorization entry points.
     /// The canonical evaluation path is `prepare_service_account_auth + eval_prepared`.
     pub async fn is_allowed_service_account(&self, args: &Args<'_>, parent_user: &str) -> bool {
@@ -998,45 +1024,14 @@ impl<T: Store> IamSys<T> {
             (effective_groups, groups_source, p)
         };
 
-        let mut combined_policy = Policy::default();
+        let mut policy_names = policies;
+        if !is_owner && policy_names.is_empty() {
+            policy_names = Self::safe_claim_policy_names(args.claims, parent_user);
+        }
 
-        if !is_owner && policies.is_empty() {
-            // For OIDC/STS users, policies may be specified in JWT claims rather than IAM DB.
-            if let Some(claim_policies) = args.claims.get("policy").and_then(|v| v.as_str()) {
-                use rustfs_policy::policy::default::DEFAULT_POLICIES;
-                let mut resolved = Vec::new();
-                for policy_name in claim_policies.split(',').map(|s| s.trim()).filter(|s| !s.is_empty()) {
-                    if !Self::is_safe_claim_policy_name(policy_name) {
-                        continue;
-                    }
-                    for (name, p) in DEFAULT_POLICIES.iter() {
-                        if *name == policy_name {
-                            resolved.push(p.clone());
-                            break;
-                        }
-                    }
-                }
-                if !resolved.is_empty() {
-                    combined_policy = Policy::merge_policies(resolved);
-                } else if args.deny_only {
-                    combined_policy = Policy::default();
-                } else {
-                    return PreparedIamAuth {
-                        needs_existing_object_tag: false,
-                        mode: PreparedIamMode::Deny,
-                    };
-                }
-            } else if args.deny_only {
-                combined_policy = Policy::default();
-            } else {
-                return PreparedIamAuth {
-                    needs_existing_object_tag: false,
-                    mode: PreparedIamMode::Deny,
-                };
-            }
-        } else if !is_owner {
-            let (a, c) = self.store.merge_policies(&policies.join(",")).await;
-            if a.is_empty() {
+        let mut combined_policy = Policy::default();
+        if !is_owner {
+            if policy_names.is_empty() {
                 if args.deny_only {
                     combined_policy = Policy::default();
                 } else {
@@ -1046,7 +1041,37 @@ impl<T: Store> IamSys<T> {
                     };
                 }
             } else {
-                combined_policy = c;
+                let requested_policies = policy_names.join(",");
+                let (resolved_policies, c) = self.store.merge_policies(&requested_policies).await;
+                if resolved_policies.is_empty() {
+                    tracing::warn!(
+                        parent_user = %parent_user,
+                        requested_policies = %requested_policies,
+                        "prepare_sts_auth: no STS policy names resolved"
+                    );
+                    if args.deny_only {
+                        combined_policy = Policy::default();
+                    } else {
+                        return PreparedIamAuth {
+                            needs_existing_object_tag: false,
+                            mode: PreparedIamMode::Deny,
+                        };
+                    }
+                } else {
+                    let resolved_policy_names = MappedPolicy::new(&resolved_policies).to_slice();
+                    let has_unresolved_policy_names = policy_names
+                        .iter()
+                        .any(|policy_name| !resolved_policy_names.iter().any(|resolved| resolved == policy_name));
+                    if has_unresolved_policy_names {
+                        tracing::debug!(
+                            parent_user = %parent_user,
+                            requested_policies = %requested_policies,
+                            resolved_policies = %resolved_policies,
+                            "prepare_sts_auth: some STS policy names were not resolved"
+                        );
+                    }
+                    combined_policy = c;
+                }
             }
         }
 
@@ -1328,6 +1353,19 @@ mod tests {
         assert!(prepared.combined_policy_for_view().is_none());
     }
 
+    const CUSTOM_STS_CLAIM_POLICY: &str = "custom-sts-claim-getobject";
+    const CUSTOM_STS_CLAIM_BUCKET: &str = "claim-bucket";
+    const CUSTOM_STS_CLAIM_POLICY_JSON: &str = r#"{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": ["s3:GetObject"],
+      "Resource": ["arn:aws:s3:::claim-bucket/allowed/*"]
+    }
+  ]
+}"#;
+
     /// Mock Store for STS tests: either group-attached policies via parent user, or no IAM policies.
     #[derive(Clone)]
     struct StsTestMockStore {
@@ -1470,7 +1508,10 @@ mod tests {
         }
 
         async fn load_all(&self, cache: &Cache) -> Result<()> {
-            let policy_docs = get_default_policyes();
+            let mut policy_docs = get_default_policyes();
+            let custom_claim_policy =
+                Policy::parse_config(CUSTOM_STS_CLAIM_POLICY_JSON.as_bytes()).expect("custom STS claim policy should parse");
+            policy_docs.insert(CUSTOM_STS_CLAIM_POLICY.to_string(), PolicyDoc::new(custom_claim_policy));
             cache
                 .policy_docs
                 .store(Arc::new(CacheEntity::new(policy_docs).update_load_time()));
@@ -1565,7 +1606,7 @@ mod tests {
         ensure_test_global_credentials();
 
         let store = StsTestMockStore { empty_policies: false };
-        let cache_manager = IamCache::new(store).await;
+        let cache_manager = IamCache::new(store).await.unwrap();
         let iam_sys = IamSys::new(cache_manager);
 
         let (cred, _) = iam_sys
@@ -1588,7 +1629,7 @@ mod tests {
         ensure_test_global_credentials();
 
         let store = StsTestMockStore { empty_policies: false };
-        let cache_manager = IamCache::new(store).await;
+        let cache_manager = IamCache::new(store).await.unwrap();
         let iam_sys = IamSys::new(cache_manager);
 
         let initial_expiration = OffsetDateTime::now_utc() + time::Duration::hours(2);
@@ -1641,7 +1682,7 @@ mod tests {
         ensure_test_global_credentials();
 
         let store = StsTestMockStore { empty_policies: false };
-        let cache_manager = IamCache::new(store).await;
+        let cache_manager = IamCache::new(store).await.unwrap();
         let iam_sys = IamSys::new(cache_manager);
 
         let (cred, _) = iam_sys
@@ -1682,7 +1723,7 @@ mod tests {
         ensure_test_global_credentials();
 
         let store = StsTestMockStore { empty_policies: false };
-        let cache_manager = IamCache::new(store).await;
+        let cache_manager = IamCache::new(store).await.unwrap();
         let iam_sys = IamSys::new(cache_manager);
 
         let parent_user = "sts-fallback-test-parent";
@@ -1763,7 +1804,7 @@ mod tests {
         ensure_test_global_credentials();
 
         let store = StsTestMockStore { empty_policies: false };
-        let cache_manager = IamCache::new(store).await;
+        let cache_manager = IamCache::new(store).await.unwrap();
         let iam_sys = IamSys::new(cache_manager);
 
         let parent_user = "sts-fallback-test-parent";
@@ -1867,7 +1908,7 @@ mod tests {
     #[tokio::test]
     async fn test_sts_groups_fallback_temp_creds_receive_parent_group_policies() {
         let store = StsTestMockStore { empty_policies: false };
-        let cache_manager = IamCache::new(store).await;
+        let cache_manager = IamCache::new(store).await.unwrap();
         let iam_sys = IamSys::new(cache_manager);
 
         let parent_user = "sts-fallback-test-parent";
@@ -1893,12 +1934,198 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    async fn test_sts_claim_policy_resolves_custom_canned_policy() {
+        let store = StsTestMockStore { empty_policies: true };
+        let cache_manager = IamCache::new(store).await.unwrap();
+        let iam_sys = IamSys::new(cache_manager);
+
+        let parent_user = "sts-empty-parent-policy-test";
+        let sts_access_key = "sts-custom-claim-policy-test-user";
+        let sts_user = UserIdentity::from(Credentials {
+            access_key: sts_access_key.to_string(),
+            secret_key: "longenoughsecret".to_string(),
+            session_token: "sts-token".to_string(),
+            status: ACCOUNT_ON.to_string(),
+            parent_user: parent_user.to_string(),
+            ..Default::default()
+        });
+        Cache::add_or_update(&iam_sys.store.cache.sts_accounts, sts_access_key, &sts_user, OffsetDateTime::now_utc());
+
+        let mut claims = HashMap::new();
+        claims.insert(POLICYNAME.to_string(), Value::String(CUSTOM_STS_CLAIM_POLICY.to_string()));
+        let groups: Option<Vec<String>> = None;
+        let args = Args {
+            account: sts_access_key,
+            groups: &groups,
+            action: Action::S3Action(S3Action::GetObjectAction),
+            bucket: CUSTOM_STS_CLAIM_BUCKET,
+            conditions: &HashMap::new(),
+            is_owner: false,
+            object: "allowed/object.txt",
+            claims: &claims,
+            deny_only: false,
+        };
+
+        let prepared = iam_sys.prepare_sts_auth(&args, parent_user).await;
+        assert!(matches!(prepared.mode, PreparedIamMode::Sts { .. }));
+        assert!(
+            iam_sys.eval_prepared(&prepared, &args).await,
+            "STS temp credentials should resolve custom canned policy names carried in JWT policy claims"
+        );
+    }
+
+    #[tokio::test]
+    async fn test_sts_claim_policy_ignores_unsafe_and_missing_policy_names() {
+        let store = StsTestMockStore { empty_policies: true };
+        let cache_manager = IamCache::new(store).await.unwrap();
+        let iam_sys = IamSys::new(cache_manager);
+
+        let parent_user = "sts-empty-parent-policy-test";
+        let sts_access_key = "sts-mixed-claim-policy-test-user";
+        let sts_user = UserIdentity::from(Credentials {
+            access_key: sts_access_key.to_string(),
+            secret_key: "longenoughsecret".to_string(),
+            session_token: "sts-token".to_string(),
+            status: ACCOUNT_ON.to_string(),
+            parent_user: parent_user.to_string(),
+            ..Default::default()
+        });
+        Cache::add_or_update(&iam_sys.store.cache.sts_accounts, sts_access_key, &sts_user, OffsetDateTime::now_utc());
+
+        let mut claims = HashMap::new();
+        claims.insert(
+            POLICYNAME.to_string(),
+            Value::String(format!("unsafe/policy, missing-sts-claim-policy, {CUSTOM_STS_CLAIM_POLICY}")),
+        );
+        let groups: Option<Vec<String>> = None;
+        let args = Args {
+            account: sts_access_key,
+            groups: &groups,
+            action: Action::S3Action(S3Action::GetObjectAction),
+            bucket: CUSTOM_STS_CLAIM_BUCKET,
+            conditions: &HashMap::new(),
+            is_owner: false,
+            object: "allowed/object.txt",
+            claims: &claims,
+            deny_only: false,
+        };
+
+        let prepared = iam_sys.prepare_sts_auth(&args, parent_user).await;
+        assert!(matches!(prepared.mode, PreparedIamMode::Sts { .. }));
+        assert!(
+            iam_sys.eval_prepared(&prepared, &args).await,
+            "STS policy claims should ignore unsafe or unresolved names without dropping a resolvable canned policy"
+        );
+    }
+
+    #[tokio::test]
+    async fn test_sts_claim_policy_custom_canned_policy_does_not_grant_other_actions() {
+        let store = StsTestMockStore { empty_policies: true };
+        let cache_manager = IamCache::new(store).await.unwrap();
+        let iam_sys = IamSys::new(cache_manager);
+
+        let parent_user = "sts-empty-parent-policy-test";
+        let sts_access_key = "sts-custom-claim-policy-deny-test-user";
+        let sts_user = UserIdentity::from(Credentials {
+            access_key: sts_access_key.to_string(),
+            secret_key: "longenoughsecret".to_string(),
+            session_token: "sts-token".to_string(),
+            status: ACCOUNT_ON.to_string(),
+            parent_user: parent_user.to_string(),
+            ..Default::default()
+        });
+        Cache::add_or_update(&iam_sys.store.cache.sts_accounts, sts_access_key, &sts_user, OffsetDateTime::now_utc());
+
+        let mut claims = HashMap::new();
+        claims.insert(POLICYNAME.to_string(), Value::String(CUSTOM_STS_CLAIM_POLICY.to_string()));
+        let groups: Option<Vec<String>> = None;
+        let args = Args {
+            account: sts_access_key,
+            groups: &groups,
+            action: Action::S3Action(S3Action::PutObjectAction),
+            bucket: CUSTOM_STS_CLAIM_BUCKET,
+            conditions: &HashMap::new(),
+            is_owner: false,
+            object: "allowed/object.txt",
+            claims: &claims,
+            deny_only: false,
+        };
+
+        let prepared = iam_sys.prepare_sts_auth(&args, parent_user).await;
+        assert!(matches!(prepared.mode, PreparedIamMode::Sts { .. }));
+        assert!(
+            !iam_sys.eval_prepared(&prepared, &args).await,
+            "custom claim policy must not grant S3 actions outside the resolved canned policy"
+        );
+    }
+
+    #[tokio::test]
+    async fn test_sts_claim_policy_builtin_policy_remains_compatible() {
+        let store = StsTestMockStore { empty_policies: true };
+        let cache_manager = IamCache::new(store).await.unwrap();
+        let iam_sys = IamSys::new(cache_manager);
+
+        let parent_user = "sts-empty-parent-policy-test";
+        let mut claims = HashMap::new();
+        claims.insert(POLICYNAME.to_string(), Value::String("readwrite".to_string()));
+        let groups: Option<Vec<String>> = None;
+        let args = Args {
+            account: "sts-builtin-claim-policy-test-user",
+            groups: &groups,
+            action: Action::S3Action(S3Action::ListBucketAction),
+            bucket: "mybucket",
+            conditions: &HashMap::new(),
+            is_owner: false,
+            object: "",
+            claims: &claims,
+            deny_only: false,
+        };
+
+        let prepared = iam_sys.prepare_sts_auth(&args, parent_user).await;
+        assert!(matches!(prepared.mode, PreparedIamMode::Sts { .. }));
+        assert!(
+            iam_sys.eval_prepared(&prepared, &args).await,
+            "built-in policy names in STS JWT claims must keep working through the unified policy store path"
+        );
+    }
+
+    #[tokio::test]
+    async fn test_sts_claim_policy_missing_policy_denies() {
+        let store = StsTestMockStore { empty_policies: true };
+        let cache_manager = IamCache::new(store).await.unwrap();
+        let iam_sys = IamSys::new(cache_manager);
+
+        let parent_user = "sts-empty-parent-policy-test";
+        let mut claims = HashMap::new();
+        claims.insert(POLICYNAME.to_string(), Value::String("missing-sts-claim-policy".to_string()));
+        let groups: Option<Vec<String>> = None;
+        let args = Args {
+            account: "sts-missing-claim-policy-test-user",
+            groups: &groups,
+            action: Action::S3Action(S3Action::GetObjectAction),
+            bucket: CUSTOM_STS_CLAIM_BUCKET,
+            conditions: &HashMap::new(),
+            is_owner: false,
+            object: "allowed/object.txt",
+            claims: &claims,
+            deny_only: false,
+        };
+
+        let prepared = iam_sys.prepare_sts_auth(&args, parent_user).await;
+        assert!(matches!(prepared.mode, PreparedIamMode::Deny));
+        assert!(
+            !iam_sys.eval_prepared(&prepared, &args).await,
+            "missing STS claim policy names must deny instead of silently allowing"
+        );
+    }
+
     /// Regression: `deny_only` with empty IAM policies must still evaluate `sessionPolicy-extracted`
     /// so session policy Deny cannot be bypassed (see PR #2250 review).
     #[tokio::test]
     async fn test_sts_deny_only_session_policy_deny_blocks_when_iam_policies_empty() {
         let store = StsTestMockStore { empty_policies: true };
-        let cache_manager = IamCache::new(store).await;
+        let cache_manager = IamCache::new(store).await.unwrap();
         let iam_sys = IamSys::new(cache_manager);
 
         let parent_user = "sts-empty-parent-policy-test";
@@ -1938,7 +2165,7 @@ mod tests {
     #[tokio::test]
     async fn test_sts_deny_only_session_policy_allow_when_no_deny_on_action() {
         let store = StsTestMockStore { empty_policies: true };
-        let cache_manager = IamCache::new(store).await;
+        let cache_manager = IamCache::new(store).await.unwrap();
         let iam_sys = IamSys::new(cache_manager);
 
         let parent_user = "sts-empty-parent-policy-test";
@@ -1982,7 +2209,7 @@ mod tests {
     #[tokio::test]
     async fn test_load_user_notification_populates_user_and_policy_caches() {
         let store = StsTestMockStore { empty_policies: false };
-        let cache_manager = IamCache::new(store).await;
+        let cache_manager = IamCache::new(store).await.unwrap();
         let iam_sys = IamSys::new(cache_manager);
 
         iam_sys.load_user("notify-user", UserType::Reg).await.unwrap();
@@ -2003,7 +2230,7 @@ mod tests {
     #[tokio::test]
     async fn test_check_key_propagates_cache_miss_load_failure() {
         let store = StsTestMockStore { empty_policies: false };
-        let cache_manager = IamCache::new(store).await;
+        let cache_manager = IamCache::new(store).await.unwrap();
         let iam_sys = IamSys::new(cache_manager);
 
         let result = iam_sys.check_key("load-failure-user").await;
@@ -2014,7 +2241,7 @@ mod tests {
     #[tokio::test]
     async fn test_prepare_auth_eval_matches_prepare_sts_auth_for_parent_policy_fallback() {
         let store = StsTestMockStore { empty_policies: false };
-        let cache_manager = IamCache::new(store).await;
+        let cache_manager = IamCache::new(store).await.unwrap();
         let iam_sys = IamSys::new(cache_manager);
 
         let parent_user = "sts-fallback-test-parent";
@@ -2042,7 +2269,7 @@ mod tests {
     #[tokio::test]
     async fn test_prepare_auth_detects_existing_object_tag_in_session_policy() {
         let store = StsTestMockStore { empty_policies: true };
-        let cache_manager = IamCache::new(store).await;
+        let cache_manager = IamCache::new(store).await.unwrap();
         let iam_sys = IamSys::new(cache_manager);
         let sts_access_key = "sts-session-tag-test-user";
 
@@ -2156,7 +2383,7 @@ mod tests {
     #[tokio::test]
     async fn test_prepare_auth_detects_existing_object_tag_in_encoded_session_policy() {
         let store = StsTestMockStore { empty_policies: true };
-        let cache_manager = IamCache::new(store).await;
+        let cache_manager = IamCache::new(store).await.unwrap();
         let iam_sys = IamSys::new(cache_manager);
         let sts_access_key = "sts-session-tag-encoded-test-user";
 
@@ -2203,7 +2430,7 @@ mod tests {
     #[tokio::test]
     async fn test_prepare_auth_service_account_inherited_ignores_session_policy_tag_hint() {
         let store = StsTestMockStore { empty_policies: false };
-        let cache_manager = IamCache::new(store).await;
+        let cache_manager = IamCache::new(store).await.unwrap();
         let iam_sys = IamSys::new(cache_manager);
 
         let service_account_access_key = "svc-inherited-tag-hint-test-user";
@@ -2266,7 +2493,7 @@ mod tests {
     #[tokio::test]
     async fn test_policy_db_get_skips_nonexistent_groups() {
         let store = StsTestMockStore { empty_policies: false };
-        let cache_manager = IamCache::new(store).await;
+        let cache_manager = IamCache::new(store).await.unwrap();
         let iam_sys = IamSys::new(cache_manager);
 
         // "testgroup" exists with "readwrite" policy; "nonexistent-group" does not exist in IAM.
@@ -2287,7 +2514,7 @@ mod tests {
     #[tokio::test]
     async fn test_info_policy_returns_policy_as_json_object() {
         let store = StsTestMockStore { empty_policies: false };
-        let cache_manager = IamCache::new(store).await;
+        let cache_manager = IamCache::new(store).await.unwrap();
         let iam_sys = IamSys::new(cache_manager);
 
         let policy_info = iam_sys

crates/io-core/src/pool.rs

@@ -443,11 +443,13 @@ impl PoolTier {
 }
 
 impl Drop for PooledBuffer {
+    // SAFETY: Drop has exclusive access to `self`; taking the `ManuallyDrop`
+    // buffer moves it exactly once into the pool when a tier still owns it.
     #[allow(unsafe_code)]
     fn drop(&mut self) {
         // Return buffer to pool if tier reference exists
         if let Some(ref tier) = self.tier {
-            // Safety: We're in drop(), so this is the last use of the buffer
+            // SAFETY: We're in drop(), so this is the last use of the buffer
             // ManuallyDrop allows us to take the value without running BytesMut's drop
             let buffer = unsafe { ManuallyDrop::take(&mut self.buffer) };
             tier.return_buffer(buffer);

crates/io-core/src/reader.rs

@@ -119,6 +119,9 @@ impl ZeroCopyObjectReader {
     /// let reader = ZeroCopyObjectReader::from_file_mmap_path("large_file.bin", 0, 1024).await?;
     /// ```
     #[cfg(unix)]
+    // SAFETY: The mmap is created from a read-only file handle for the
+    // caller-provided range, then copied into owned `Bytes` before the file and
+    // mapping are dropped.
     #[allow(unsafe_code)]
     pub async fn from_file_mmap_path(path: &std::path::Path, offset: u64, size: usize) -> Result<Self, ZeroCopyReadError> {
         use memmap2::MmapOptions;
@@ -130,7 +133,8 @@ impl ZeroCopyObjectReader {
             // Open the file in sync context
             let std_file = std::fs::File::open(&path).map_err(|e| ZeroCopyReadError::Io(e.to_string()))?;
 
-            // Create memory map
+            // SAFETY: `std_file` remains open while the mapping is created and
+            // copied, and the mapped bytes are not exposed beyond this closure.
             let mmap = unsafe { MmapOptions::new().offset(offset).len(size).map(&std_file) }
                 .map_err(|e| ZeroCopyReadError::Mmap(e.to_string()))?;
 

crates/io-metrics/README.md

@@ -4,9 +4,6 @@
   <a href="https://github.com/rustfs/rustfs/actions/workflows/ci.yml">
     <img src="https://github.com/rustfs/rustfs/actions/workflows/ci.yml/badge.svg" alt="CI Status" />
   </a>
-  <a href="https://docs.rs/rustfs-io-metrics">
-    <img src="https://docs.rs/rustfs-io-metrics/badge.svg" alt="Documentation" />
-  </a>
   <a href="https://crates.io/crates/rustfs-io-metrics">
     <img src="https://img.shields.io/crates/v/rustfs-io-metrics.svg" alt="Crates.io" />
   </a>
@@ -14,7 +11,7 @@
 
 <p align="center">
   · <a href="https://github.com/rustfs/rustfs">Home</a>
-  · <a href="https://docs.rs/rustfs-io-metrics">Docs</a>
+  · <a href="#documentation">Docs</a>
   · <a href="https://github.com/rustfs/rustfs/issues">Issues</a>
   · <a href="https://github.com/rustfs/rustfs/discussions">Discussions</a>
 </p>
@@ -205,10 +202,23 @@ cargo bench --package rustfs-io-metrics --bench metrics_pipeline
 
 ## Documentation
 
-- [API Documentation](https://docs.rs/rustfs-io-metrics)
-- [Adaptive TTL Design](./docs/adaptive-ttl-design.md)
-- [Metrics Guide](./docs/metrics-guide.md)
-- [Configuration Reference](./docs/config-reference.md)
+This crate records metrics through the Rust `metrics` crate and leaves
+exporting to `rustfs-obs` or the application-level observability pipeline. It
+does not expose Prometheus-compatible HTTP endpoints such as
+`/rustfs/v2/metrics/cluster` or `/rustfs/v2/metrics/node`.
+
+API documentation can be generated locally:
+
+```bash
+cargo doc --package rustfs-io-metrics --no-deps --open
+```
+
+Useful source references:
+
+- [Crate API overview](./src/lib.rs)
+- [Metrics example](./examples/metrics_example.rs)
+- [Configuration module](./src/config.rs)
+- [Adaptive TTL module](./src/adaptive_ttl.rs)
 
 ## Related Modules
 

crates/io-metrics/README_zh.md

@@ -4,9 +4,6 @@
   <a href="https://github.com/rustfs/rustfs/actions/workflows/ci.yml">
     <img src="https://github.com/rustfs/rustfs/actions/workflows/ci.yml/badge.svg" alt="CI Status" />
   </a>
-  <a href="https://docs.rs/rustfs-io-metrics">
-    <img src="https://docs.rs/rustfs-io-metrics/badge.svg" alt="Documentation" />
-  </a>
   <a href="https://crates.io/crates/rustfs-io-metrics">
     <img src="https://img.shields.io/crates/v/rustfs-io-metrics.svg" alt="Crates.io" />
   </a>
@@ -14,7 +11,7 @@
 
 <p align="center">
   · <a href="https://github.com/rustfs/rustfs">🏠 主页</a>
-  · <a href="https://docs.rs/rustfs-io-metrics">📚 文档</a>
+  · <a href="#-文档">📚 文档</a>
   · <a href="https://github.com/rustfs/rustfs/issues">🐛 问题</a>
   · <a href="https://github.com/rustfs/rustfs/discussions">💬 讨论</a>
 </p>
@@ -286,10 +283,22 @@ cargo bench --package rustfs-io-metrics --bench metrics_pipeline
 
 ## 📚 文档
 
-- [API 文档](https://docs.rs/rustfs-io-metrics)
-- [自适应 TTL 设计](./docs/adaptive-ttl-design.md)
-- [指标收集指南](./docs/metrics-guide.md)
-- [配置参考](./docs/config-reference.md)
+此 crate 通过 Rust `metrics` crate 记录指标，并由 `rustfs-obs` 或应用层可观测性管线负责导出。
+它本身不提供 Prometheus 兼容的 HTTP 端点，例如 `/rustfs/v2/metrics/cluster`
+或 `/rustfs/v2/metrics/node`。
+
+可以在本地生成 API 文档：
+
+```bash
+cargo doc --package rustfs-io-metrics --no-deps --open
+```
+
+相关源码入口：
+
+- [Crate API 概览](./src/lib.rs)
+- [指标示例](./examples/metrics_example.rs)
+- [配置模块](./src/config.rs)
+- [自适应 TTL 模块](./src/adaptive_ttl.rs)
 
 ## 🔗 相关模块
 

crates/io-metrics/src/lib.rs

@@ -65,6 +65,7 @@ pub mod lock_metrics;
 pub mod performance;
 pub mod process_lock_metrics;
 pub mod sampler;
+pub mod system_path_metrics;
 pub mod timeout_metrics;
 
 pub use autotuner::{AutoTuner, TunerConfig, TuningResult};
@@ -121,6 +122,7 @@ pub use sampler::{
     ProcessResourceSnapshot, ProcessStatusSnapshot, ProcessSystemSnapshot, snapshot_process_platform, snapshot_process_resource,
     snapshot_process_resource_and_system, snapshot_process_system,
 };
+pub use system_path_metrics::record_system_path_failure;
 
 // Timeout metrics exports
 pub use timeout_metrics::{

crates/io-metrics/src/system_path_metrics.rs

@@ -0,0 +1,23 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub fn record_system_path_failure(path_kind: &'static str, operation: &'static str, reason: &'static str) {
+    counter!(
+        "rustfs_system_path_failures_total",
+        "path_kind" => path_kind,
+        "operation" => operation,
+        "reason" => reason
+    )
+    .increment(1);
+}

crates/keystone/README.md

@@ -82,6 +82,10 @@ export RUSTFS_KEYSTONE_CACHE_SIZE=10000
 export RUSTFS_KEYSTONE_CACHE_TTL=300
 ```
 
+TLS certificate verification is enabled by default. Set
+`RUSTFS_KEYSTONE_VERIFY_SSL=false` only for an explicitly trusted hop; it allows
+MITM attacks against the Keystone connection and emits a startup warning.
+
 ## API Documentation
 
 ### KeystoneClient
@@ -628,8 +632,8 @@ time curl -X GET http://localhost:9000/ \
 - Verify token format is correct (no newlines, extra spaces)
 
 **Issue: "SSL verification failed"**
-- If using self-signed certificates, set `RUSTFS_KEYSTONE_VERIFY_SSL=false`
-- Or install Keystone's CA certificate in system trust store
+- Prefer installing Keystone's CA certificate in the system trust store
+- If using a trusted non-production hop, set `RUSTFS_KEYSTONE_VERIFY_SSL=false`; this allows MITM attacks and emits a startup warning
 
 **Issue: Slow performance**
 - Increase cache size: `RUSTFS_KEYSTONE_CACHE_SIZE=50000`

crates/keystone/src/client.rs

@@ -58,6 +58,13 @@ impl KeystoneClient {
         admin_domain: String,
         verify_ssl: bool,
     ) -> Self {
+        if !verify_ssl {
+            warn!(
+                "Keystone client for '{}' is configured to skip TLS certificate verification. This permits MITM attacks and should not be used in production.",
+                auth_url
+            );
+        }
+
         let client = Client::builder()
             .danger_accept_invalid_certs(!verify_ssl)
             .timeout(std::time::Duration::from_secs(30))

crates/madmin/src/info_commands.rs

@@ -97,6 +97,10 @@ pub struct Disk {
     pub runtime_state: Option<String>,
     #[serde(rename = "offlineDurationSeconds", default, skip_serializing_if = "Option::is_none")]
     pub offline_duration_seconds: Option<u64>,
+    #[serde(rename = "capacityObservationSource", default, skip_serializing_if = "Option::is_none")]
+    pub capacity_observation_source: Option<String>,
+    #[serde(rename = "capacityObservationAgeSeconds", default, skip_serializing_if = "Option::is_none")]
+    pub capacity_observation_age_seconds: Option<u64>,
     /// Leaf physical block devices backing this disk path when the platform can resolve them.
     #[serde(rename = "physicalDeviceIds", default, skip_serializing_if = "Option::is_none")]
     pub physical_device_ids: Option<Vec<String>>,
@@ -515,6 +519,8 @@ mod tests {
             metrics: Some(DiskMetrics::default()),
             runtime_state: Some("online".to_string()),
             offline_duration_seconds: Some(0),
+            capacity_observation_source: None,
+            capacity_observation_age_seconds: None,
             heal_info: None,
             used_inodes: 1000000,
             free_inodes: 9000000,
@@ -616,6 +622,8 @@ mod tests {
             physical_device_ids: Some(vec!["nvme0n1".to_string(), "nvme1n1".to_string()]),
             runtime_state: Some("online".to_string()),
             offline_duration_seconds: Some(0),
+            capacity_observation_source: None,
+            capacity_observation_age_seconds: None,
         };
 
         let mut encoded = Vec::new();

crates/notify/AGENTS.md

@@ -0,0 +1,51 @@
+# Notify Crate Instructions
+
+Applies to `crates/notify/`.
+
+`rustfs-notify` is the domain layer for bucket notification semantics. It
+builds rules, event dispatch flow, and config/runtime orchestration on top of
+shared plugin/runtime primitives from `rustfs-targets`.
+
+## Domain Boundaries
+
+- Keep notify-specific business logic here:
+  - bucket/rule evaluation
+  - event bridge and pipeline dispatch
+  - notify config reload orchestration
+- Keep shared runtime/plugin mechanics in `rustfs-targets`:
+  - do not duplicate replay worker lifecycle logic
+  - do not reimplement plugin descriptor/registry/catalog semantics
+  - do not move install/control-plane state into this crate
+
+## Runtime Layering Rules
+
+- `runtime_facade.rs` is the mutation/orchestration boundary:
+  activation, replace, stop workers, shutdown.
+- `runtime_view.rs` is read-only runtime observation:
+  active targets, metrics/health snapshots, runtime status snapshots.
+- `config_manager.rs` should map config to runtime updates through facade/view
+  and `runtime_target_id_for_subsystem`; avoid bypassing these boundaries.
+- `stream.rs` is a compatibility shim; new replay/runtime work should prefer
+  shared helpers in `rustfs-targets::runtime`.
+
+## Change Style
+
+- Preserve best-effort dispatch semantics and observability signals unless the
+  task explicitly requests behavior changes.
+- Reuse existing notify constants and subsystem mappings from `rustfs_config`.
+- Keep changes local and avoid cross-crate refactors from this crate unless
+  required by the task.
+
+## Testing
+
+- Keep unit tests close to changed modules.
+- Add regression tests for:
+  - rules to runtime target resolution
+  - runtime facade replace/shutdown behavior
+  - runtime view health/status/metrics snapshots
+- Suggested validation:
+  - `cargo test -p rustfs-notify`
+  - Focused: `cargo test -p rustfs-notify runtime_facade`
+  - Focused: `cargo test -p rustfs-notify runtime_view`
+  - Focused: `cargo test -p rustfs-notify config_manager`
+- Full gate before commit: `make pre-commit`

crates/notify/Cargo.toml

@@ -34,9 +34,9 @@ rustfs-utils = { workspace = true }
 arc-swap = { workspace = true }
 async-trait = { workspace = true }
 chrono = { workspace = true, features = ["serde"] }
-futures = { workspace = true }
 form_urlencoded = { workspace = true }
 hashbrown = { workspace = true }
+percent-encoding = { workspace = true }
 rayon = { workspace = true }
 rustc-hash = { workspace = true }
 serde = { workspace = true }
@@ -46,6 +46,7 @@ tokio = { workspace = true, features = ["rt-multi-thread", "sync", "time"] }
 tracing = { workspace = true }
 url = { workspace = true }
 wildmatch = { workspace = true, features = ["serde"] }
+metrics = { workspace = true }
 
 # quick-xml dependencies for custom S3KeyFilter XML deserialization
 # Custom deserializer implemented for S3KeyFilter to handle both XML structures:

crates/notify/src/bucket_config_manager.rs

@@ -0,0 +1,125 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::{
+    BucketNotificationConfig, NotificationError, config_manager::notify_configuration_hint,
+    notification_system_subscriber::NotificationSystemSubscriberView, notifier::EventNotifier, rule_engine::NotifyRuleEngine,
+    rules::ParseConfigError,
+};
+use rustfs_s3_common::EventName;
+use std::sync::Arc;
+use tracing::{debug, info, warn};
+
+#[derive(Clone)]
+pub struct NotifyBucketConfigManager {
+    notifier: Arc<EventNotifier>,
+    rule_engine: NotifyRuleEngine,
+    subscriber_view: Arc<NotificationSystemSubscriberView>,
+}
+
+impl NotifyBucketConfigManager {
+    pub fn new(
+        notifier: Arc<EventNotifier>,
+        rule_engine: NotifyRuleEngine,
+        subscriber_view: Arc<NotificationSystemSubscriberView>,
+    ) -> Self {
+        Self {
+            notifier,
+            rule_engine,
+            subscriber_view,
+        }
+    }
+
+    pub async fn has_subscriber(&self, bucket: &str, event: &EventName) -> bool {
+        if !self.subscriber_view.has_subscriber(bucket, event) {
+            return false;
+        }
+        self.rule_engine.has_subscriber(bucket, event).await
+    }
+
+    pub async fn load_bucket_notification_config(
+        &self,
+        bucket: &str,
+        cfg: &BucketNotificationConfig,
+    ) -> Result<(), NotificationError> {
+        let arn_list = self.notifier.get_arn_list(&cfg.region).await;
+        if arn_list.is_empty() {
+            return Err(NotificationError::Configuration(notify_configuration_hint()));
+        }
+        info!("Available ARNs: {:?}", arn_list);
+
+        if let Err(e) = cfg.validate(&cfg.region, &arn_list) {
+            debug!("Bucket notification config validation region:{} failed: {}", &cfg.region, e);
+            if !matches!(e, ParseConfigError::ArnNotFound(_)) {
+                return Err(NotificationError::BucketNotification(e.to_string()));
+            }
+            warn!(
+                bucket = %bucket,
+                region = %cfg.region,
+                error = %e,
+                "Bucket notification config references missing target ARN; keeping compatibility and loading remaining rules"
+            );
+        }
+
+        self.subscriber_view.apply_bucket_config(bucket, cfg);
+        self.rule_engine.set_bucket_rules(bucket, cfg.get_rules_map().clone()).await;
+        info!("Loaded notification config for bucket: {}", bucket);
+        Ok(())
+    }
+
+    pub async fn remove_bucket_notification_config(&self, bucket: &str) {
+        self.subscriber_view.clear_bucket(bucket);
+        self.rule_engine.clear_bucket_rules(bucket).await;
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::NotifyBucketConfigManager;
+    use crate::{
+        BucketNotificationConfig, integration::NotificationMetrics,
+        notification_system_subscriber::NotificationSystemSubscriberView, notifier::EventNotifier, rule_engine::NotifyRuleEngine,
+    };
+    use rustfs_s3_common::EventName;
+    use rustfs_targets::arn::TargetID;
+    use std::sync::Arc;
+
+    fn build_manager() -> NotifyBucketConfigManager {
+        let metrics = Arc::new(NotificationMetrics::new());
+        let rule_engine = NotifyRuleEngine::new();
+        let notifier = Arc::new(EventNotifier::new(metrics, rule_engine.clone()));
+        let subscriber_view = Arc::new(NotificationSystemSubscriberView::new());
+        NotifyBucketConfigManager::new(notifier, rule_engine, subscriber_view)
+    }
+
+    #[tokio::test]
+    async fn bucket_config_manager_reports_no_subscriber_for_empty_state() {
+        let manager = build_manager();
+        assert!(!manager.has_subscriber("bucket", &EventName::ObjectCreatedPut).await);
+    }
+
+    #[tokio::test]
+    async fn bucket_config_manager_clears_bucket_snapshot() {
+        let manager = build_manager();
+        let target_id = TargetID::new("primary".to_string(), "webhook".to_string());
+        let mut cfg = BucketNotificationConfig::new("us-east-1");
+        cfg.add_rule(&[EventName::ObjectCreatedPut], "*".to_string(), target_id);
+
+        manager.subscriber_view.apply_bucket_config("bucket", &cfg);
+        assert!(manager.subscriber_view.has_subscriber("bucket", &EventName::ObjectCreatedPut));
+
+        manager.remove_bucket_notification_config("bucket").await;
+        assert!(!manager.subscriber_view.has_subscriber("bucket", &EventName::ObjectCreatedPut));
+    }
+}