test(notify): cover prefix suffix event dispatch (#2960)

Co-authored-by: houseme <housemecn@gmail.com>

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: rustfs-release-version-bump
+description: Prepare a RustFS release branch like PR #2957 by bumping versioned files, aligning release assets, running required verification, and finishing with commit, push, and gh PR creation. Use when publishing a new RustFS alpha, beta, or stable release.
+---
+
+# RustFS Release Version Bump
+
+Use this skill when the task is to prepare a new RustFS version release branch following the pattern validated in PR `#2957`.
+
+## Read first
+
+- Read `AGENTS.md`.
+- Read `.github/pull_request_template.md`.
+- Inspect the current branch diff against `origin/main`.
+- Do not assume every release file should change in the same way every time.
+
+## Scope validated by PR #2957
+
+The `1.0.0-beta.3` release branch updated these files:
+
+- `Cargo.toml`
+- `Cargo.lock`
+- `README.md`
+- `README_ZH.md`
+- `flake.nix`
+- `helm/rustfs/Chart.yaml`
+- `rustfs.spec`
+
+Treat this file list as the default checklist for future release bumps. Only drop a file if the repository's current release pattern clearly says it is no longer part of the publish flow.
+
+## Workflow
+
+1. Confirm release intent
+- Identify the target version string exactly, for example `1.0.0-beta.4`.
+- Check whether the user wants only local commit preparation or the full `commit + push + PR` flow.
+- Compare the branch against `origin/main` and isolate only release-related edits.
+- If the target version is not explicit, stop and ask for it before editing.
+
+2. Update core Rust workspace versions
+- Bump the workspace version in `Cargo.toml`.
+- Bump all internal workspace crate versions in `Cargo.toml`.
+- Ensure `Cargo.lock` reflects the same release version for workspace packages.
+- Re-read the diff and verify there are no partial version leftovers.
+
+3. Align release assets
+- Update versioned Docker examples in `README.md` and `README_ZH.md` using the `<version>` tag form, for example `rustfs/rustfs:1.0.0-beta.4`.
+- Update `flake.nix` package version.
+- Update `helm/rustfs/Chart.yaml` `appVersion`.
+- Update `helm/rustfs/Chart.yaml` `version` using the release-chart rule:
+- `1.0.0-beta.3` -> `0.3.0`
+- `1.0.0-beta.4` -> `0.4.0`
+- Follow the same pattern for later beta releases unless the repository rule changes.
+- Update `rustfs.spec` release metadata and changelog entry.
+- Set `rustfs.spec` `Release` to the prerelease suffix without the base version, for example `beta.3` for `1.0.0-beta.3`.
+- Keep the release asset changes in a separate commit from the core Rust workspace version bump when the branch contains both.
+
+4. Stop and discuss before changing release policy
+- Ask before changing the established Docker tag style away from `<version>`.
+- Ask before changing the established Helm chart version mapping away from `beta.N -> 0.N.0`.
+- Ask before changing the established `rustfs.spec` `Release` rule away from the release suffix form such as `beta.N`.
+- Ask before widening the release scope beyond the files already validated in PR `#2957`.
+
+5. Verify before shipping
+- Run `make pre-commit`.
+- If verification succeeds, run `cargo clean` to remove generated build artifacts before wrapping up.
+- If `make pre-commit` fails, stop and return `BLOCKED`.
+
+6. Commit structure
+- Prefer two commits when the change naturally splits:
+- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`
+- `chore(release): align release assets for <version>` for docs and packaging metadata
+- If the user asks for a single commit, follow that request.
+
+7. Push and PR
+- Push the release branch with `git push -u origin <branch>` or `git push` if upstream already exists.
+- Create the PR with `gh pr create --base main --head <branch> --title ... --body-file ...`.
+- Keep the PR title and body in English.
+- Keep the `.github/pull_request_template.md` headings exactly.
+
+## Ready-to-check commands
+
+- `git diff --name-only origin/main...HEAD`
+- `git diff --stat origin/main...HEAD`
+- `make pre-commit`
+- `cargo clean`
+- `git status --short --branch`
+
+## Output expectations
+
+When using this skill, return:
+
+- The files changed for the release bump
+- Any uncertainty that needs user confirmation before editing
+- Verification status
+- Commit messages used
+- Push status and PR URL when the GitHub flow was requested
+
+## Established release policy
+
+- Docs use Docker tags in `<version>` form, not `v<version>`.
+- `helm/rustfs/Chart.yaml` `version` follows `beta.N -> 0.N.0` based on the current release policy.
+- `rustfs.spec` `Release` follows the release suffix form, for example `beta.3`.
+- If any of these rules need to change in the future, pause and confirm before editing.

.agents/skills/rustfs-release-version-bump/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "RustFS Release Bump"
+  short_description: "Prepare RustFS release branches like PR #2957."
+  default_prompt: "Use $rustfs-release-version-bump to prepare a RustFS release version, ask about any unclear version policy, and finish the commit/push/PR flow."

.agents/skills/security-advisory-lessons/SKILL.md

@@ -0,0 +1,119 @@
+---
+name: security-advisory-lessons
+description: Apply RustFS security lessons distilled from repository GitHub Security Advisories. Use when making or reviewing RustFS code changes, doing security checks, handling PR review for auth/authz, IAM, storage, RPC, logging, CORS, console/browser, encryption, policy, or endpoint changes, and when deciding which security regression tests are required.
+---
+
+# RustFS Security Advisory Lessons
+
+Use this skill as a RustFS-specific security lens before changing or approving code. For the distilled advisory lessons and review patterns, read [advisory-patterns.md](references/advisory-patterns.md).
+
+When currentness matters, fetch the live advisory inventory instead of relying on this skill as a status mirror:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories --paginate \
+  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
+```
+
+Fetch full advisory details only when the live summary suggests a new or changed lesson:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
+```
+
+For the full pattern map, read [advisory-patterns.md](references/advisory-patterns.md).
+
+## Workflow
+
+### 1. Scope the change
+- Identify touched routes, handlers, storage paths, credentials, logs, browser surfaces, CI/release code, and policy checks.
+- Treat these paths as security-sensitive by default: `rustfs/src/admin/`, `rustfs/src/storage/`, `rustfs/src/auth.rs`, `rustfs/src/server/layer.rs`, `crates/iam/`, `crates/policy/`, `crates/credentials/`, `crates/ecstore/src/rpc/`, `crates/rio/`, and console preview/auth code.
+
+### 2. Map to advisory classes
+- Read [advisory-patterns.md](references/advisory-patterns.md) for matching GHSA lessons.
+- Do not rely on advisory titles alone. Confirm whether the issue is authentication, authorization, input validation, storage invariant, browser isolation, logging, or operational hardening.
+
+### 3. Verify fail-closed behavior
+- Check that unauthenticated, wrong-permission, cross-user, cross-bucket, malformed-input, and default-config cases fail explicitly.
+- Prefer exact action/permission checks over broad helper calls or inferred ownership.
+- Confirm lower storage/RPC layers do not bypass checks done in upper layers.
+
+### 4. Require regression evidence
+- For behavior changes, add focused negative tests that reproduce the advisory class.
+- For sensitive fixes, include tests for the bypass form, not only the happy path.
+- If a test is impractical, explain the residual risk and provide a manual verification command.
+
+### 5. Report clearly
+- Lead with concrete findings and file/line evidence.
+- Separate proven vulnerabilities from hardening risks.
+- Avoid exaggerating unauthenticated impact when the code actually rejects unauthenticated requests but allows a low-privileged authenticated bypass.
+
+## Advisory-Derived Guardrails
+
+### Auth and admin authorization
+- Every admin or diagnostic route needs an explicit authn and authz story. Route registration, router whitelist, and handler-level authorization must agree.
+- Match the admin action to the operation exactly. Copy-paste action constants are a known RustFS vulnerability class.
+- Avoid authentication-only helpers for state-changing admin APIs; use `validate_admin_request` or the established equivalent with the right `AdminAction`.
+- Do not assume admin-action `Resource` scoping constrains blast radius unless the policy engine actually enforces resources for that action.
+
+### IAM and service accounts
+- Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
+- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims.
+- Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
+- Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
+
+### S3 copy, multipart, and presigned POST
+- Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
+- Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
+- Presigned POST policies are server-side contracts. Enforce `content-length-range`, key prefix, exact metadata/content-type, and all signed policy conditions.
+
+### Paths, object keys, and filesystem access
+- Never join untrusted bucket/object/RPC path strings onto filesystem roots without normalization and boundary checks.
+- Reject or safely handle `..`, absolute paths, URL-encoded traversal, platform separators, empty components, and paths that canonicalize outside the intended root.
+- Validate both S3 object-key paths and internode/RPC disk paths; storage helpers can bypass S3 authorization if they trust already-parsed paths.
+
+### Secrets, default credentials, and crypto
+- Do not ship hard-coded shared tokens, HMAC secrets, private keys, or production test keys.
+- Defaults for internode/RPC auth must fail closed for network-reachable deployments or require explicit opt-in with loud warnings.
+- License or token validation must use signatures with embedded public/verifying keys only; do not use private-key decryption as authenticity.
+- Plan key rotation and key IDs when removing exposed keys.
+
+### Logging and debug output
+- Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials.
+- Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces.
+- Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies.
+
+### RPC, parsing, and panic safety
+- Treat all RPC payload bytes as attacker-controlled. Replace `unwrap`, `expect`, and panic-prone deserialization with typed errors.
+- Malformed request tests should cover empty bytes, truncated MessagePack/protobuf, invalid enum values, stale timestamps, and invalid signatures.
+- RPC authentication must be independently strong; do not depend on S3 admin credentials unless the fallback is explicit and safe.
+
+### Browser, CORS, and console surfaces
+- Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
+- Do not render user-controlled object content in a same-origin iframe with console credentials available to JavaScript.
+- Prefer origin separation for object preview/download, `nosniff`, CSP, strict content-type handling, and avoiding durable credentials in `localStorage`.
+- License/version-like metadata endpoints should expose only coarse public data unless authenticated.
+
+### Profiling, debug, and health endpoints
+- Profiling and debug endpoints are not health checks. They require admin auth, opt-in enablement, rate limiting, and safe responses.
+- Do not return absolute filesystem paths or other deployment layout in unauthenticated or low-privilege responses.
+- Ensure health endpoint allowlists cannot accidentally include expensive diagnostics.
+
+### Trusted proxy and network identity
+- Only honor `X-Forwarded-For` or `X-Real-IP` when the request came from a configured trusted proxy.
+- Direct clients must use the socket peer address for `aws:SourceIp` and policy condition evaluation.
+- Add tests for direct spoofed headers and trusted-proxy headers.
+
+### SSE and storage invariants
+- Encryption metadata is not proof that bytes were encrypted on disk.
+- When touching reader/writer wrappers such as hashing, encryption, compression, or warp readers, verify wrapper order and inspect stored bytes in regression tests.
+- Avoid helper shortcuts that unwrap nested readers and accidentally bypass encryption or integrity layers.
+
+## Review Prompts
+
+Use these prompts while reviewing a diff:
+
+- Could a low-privileged authenticated user reach this path with the wrong action, parent, bucket, or source object?
+- Does a public/default/empty config change security behavior from fail-closed to fail-open?
+- Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
+- Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
+- Does the test prove the exploit form is denied, or only that the intended form still works?

.agents/skills/security-advisory-lessons/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Security Advisory Lessons"
+  short_description: "Apply advisory lessons in reviews."
+  default_prompt: "Review code changes against past RustFS security advisory lessons and report concrete risks, missing tests, and recommended fixes."

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -0,0 +1,93 @@
+# RustFS Advisory Pattern Map
+
+This file is a lesson map, not an advisory inventory mirror. It keeps durable security patterns distilled from RustFS GitHub Security Advisories.
+
+When current advisory state, severity, URLs, or full text matters, fetch it live:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories --paginate \
+  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
+gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
+```
+
+Update this file only when an advisory adds or changes a reusable lesson, affected surface, validation pattern, or regression-test expectation. Do not update it for state-only, URL-only, count-only, or timestamp-only changes.
+
+## Pattern Index
+
+### Admin authorization and route exposure
+
+- `GHSA-pfcq-4gjr-6gjm`: notification target endpoints accepted authenticated users but skipped admin authorization. Lesson: distinguish authn from authz; admin target CRUD must call the operation-specific admin authorization path.
+- `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
+- `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
+- `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
+- `GHSA-x5xv-223c-8vm7`: console license metadata endpoint was public. Lesson: public metadata endpoints should be coarse or authenticated.
+
+### IAM import, service accounts, and privilege boundaries
+
+- `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
+- `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
+- `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
+
+### S3 copy, multipart, and upload policy validation
+
+- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
+- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
+- `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
+
+### Filesystem paths and object key traversal
+
+- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
+- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
+
+### Secrets, defaults, and cryptographic misuse
+
+- `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
+- `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
+- `GHSA-923g-jp7v-f97f`: license verification embedded a production RSA private key and used private-key decryption as authenticity. Lesson: ship verifying/public keys only and use real signature verification.
+
+### Sensitive logging and debug output
+
+- `GHSA-r54g-49rx-98cr`: STS credentials were logged at info level. Lesson: generated credentials must never be logged in plaintext.
+- `GHSA-8cm2-h255-v749`: debug logs leaked session tokens, secret keys, JWT claims, and raw STS response bodies. Lesson: redaction must cover custom `Debug` implementations and dependency response-body logging.
+- `GHSA-333v-68xh-8mmq`: invalid RPC signature logging included the shared HMAC secret and expected signature. Lesson: error paths often leak secrets; never log raw secrets or derived authenticators.
+
+### RPC input validation and panic safety
+
+- `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
+- `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
+
+### Browser, CORS, and console isolation
+
+- `GHSA-v9fg-3cr2-277j`: object preview rendered attacker-controlled HTML in a same-origin iframe, exposing console credentials stored in `localStorage`. Lesson: user content must be origin-isolated from the console and protected with `nosniff`, CSP, and strict content-type handling.
+- `GHSA-x5xv-223c-8vm7`: default CORS reflected arbitrary origins with credentials. Lesson: never combine reflected origins with `Access-Control-Allow-Credentials: true`; default should be fail-closed.
+
+### Trusted proxy and source IP conditions
+
+- `GHSA-fc6g-2gcp-2qrq`: `aws:SourceIp` trusted client-supplied `X-Forwarded-For` or `X-Real-IP`. Lesson: forwarded IP headers are valid only behind configured trusted proxies; direct clients use socket peer IP.
+
+### SSE and on-disk storage invariants
+
+- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
+
+## Useful Search Seeds
+
+Use these targeted searches when a diff touches security-sensitive code:
+
+```bash
+rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
+rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
+rg -n "PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
+rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
+rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
+rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
+```
+
+## Minimum Regression Test Expectations
+
+- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
+- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
+- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
+- Path fixes: include encoded traversal, absolute path, nested traversal, valid object keys that resemble traversal text but should be rejected, and canonical boundary checks.
+- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
+- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
+- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.

.config/make/lint-fmt.mak

@@ -16,6 +16,11 @@ clippy-check: core-deps ## Run clippy checks
 	cargo clippy --fix --allow-dirty
 	cargo clippy --all-targets --all-features -- -D warnings
 
+.PHONY: unsafe-code-check
+unsafe-code-check: ## Check unsafe_code allowances have SAFETY comments
+	@echo "🔒 Checking unsafe_code allowances..."
+	./scripts/check_unsafe_code_allowances.sh
+
 .PHONY: compilation-check
 compilation-check: core-deps ## Run compilation check
 	@echo "🔨 Running compilation check..."

.config/make/pre-commit.mak

@@ -7,5 +7,5 @@ setup-hooks: ## Set up git hooks
 	@echo "✅ Git hooks setup complete!"
 
 .PHONY: pre-commit
-pre-commit: fmt clippy-check compilation-check test ## Run pre-commit checks
-	@echo "✅ All pre-commit checks passed!"
+pre-commit: fmt unsafe-code-check clippy-check compilation-check test ## Run pre-commit checks
+	@echo "✅ All pre-commit checks passed!"

.config/make/tests.mak

@@ -2,8 +2,13 @@
 
 TEST_THREADS ?= 1
 
+.PHONY: script-tests
+script-tests: ## Run shell script tests
+	@echo "Running script tests..."
+	./scripts/test_build_rustfs_options.sh
+
 .PHONY: test
-test: core-deps test-deps ## Run all tests
+test: core-deps test-deps script-tests ## Run all tests
 	@echo "🧪 Running tests..."
 	@if command -v cargo-nextest >/dev/null 2>&1; then \
 		cargo nextest run --all --exclude e2e_test; \

.docker/nginx/nginx.conf

@@ -74,6 +74,17 @@ http {
     #     ssl_certificate /etc/nginx/ssl/server.crt;
     #     ssl_certificate_key /etc/nginx/ssl/server.key;
     #
+    #     # Restrict to modern TLS versions and ciphers. Operators copying this
+    #     # example must keep at least these directives — without them, nginx
+    #     # may negotiate older protocol versions that have known weaknesses.
+    #     ssl_protocols TLSv1.2 TLSv1.3;
+    #     ssl_prefer_server_ciphers on;
+    #     ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305';
+    #     ssl_session_timeout 1d;
+    #     ssl_session_cache shared:SSL:10m;
+    #     ssl_session_tickets off;
+    #     # add_header Strict-Transport-Security "max-age=63072000" always;
+    #
     #     location / {
     #         proxy_pass http://rustfs:9000;
     #         ...

.docker/observability/grafana/dashboards/rustfs.json

@@ -461,6 +461,90 @@
       "title": "Capacity",
       "type": "stat"
     },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "yellow",
+                "value": 1
+              },
+              {
+                "color": "red",
+                "value": 2
+              }
+            ]
+          },
+          "unit": "none"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 6,
+        "x": 6,
+        "y": 1
+      },
+      "id": 201,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "none",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "percentChangeColorMode": "standard",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "showPercentChange": false,
+        "textMode": "value_and_name",
+        "wideLayout": true
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum(rustfs_cluster_capacity_stale_drives{job=~\"$job\"})",
+          "legendFormat": "Stale Drives",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum(rustfs_cluster_capacity_missing_drives{job=~\"$job\"})",
+          "legendFormat": "Missing Drives",
+          "range": true,
+          "refId": "B"
+        }
+      ],
+      "title": "Capacity Observation",
+      "type": "stat"
+    },
     {
       "datasource": {
         "type": "prometheus",
@@ -1927,6 +2011,299 @@
       "title": "System Drive Usage",
       "type": "timeseries"
     },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 12,
+        "x": 12,
+        "y": 42
+      },
+      "id": 202,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "max by (drive) (rustfs_system_drive_capacity_observation_age_seconds{job=~\"$job\", drive=~\"$drive\"})",
+          "legendFormat": "{{drive}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Drive Capacity Observation Age",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": true,
+            "axisColorMode": "text",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "stepAfter",
+            "lineWidth": 2,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "none"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 12,
+        "x": 12,
+        "y": 48
+      },
+      "id": 203,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "max by (drive) (rustfs_system_drive_capacity_observation_state{job=~\"$job\", drive=~\"$drive\", state=\"stale\"})",
+          "legendFormat": "{{drive}} stale",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "max by (drive) (rustfs_system_drive_capacity_observation_state{job=~\"$job\", drive=~\"$drive\", state=\"missing\"})",
+          "legendFormat": "{{drive}} missing",
+          "range": true,
+          "refId": "B"
+        }
+      ],
+      "title": "Drive Capacity Observation State",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": true,
+            "axisColorMode": "text",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "bars",
+            "fillOpacity": 80,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "normal"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 7,
+        "w": 24,
+        "x": 0,
+        "y": 54
+      },
+      "id": 204,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (path_kind, operation, reason) (rate(rustfs_system_path_failures_total{job=~\"$job\"}[5m]))",
+          "legendFormat": "{{path_kind}} / {{operation}} / {{reason}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "System Path Failures",
+      "type": "timeseries"
+    },
     {
       "datasource": {
         "type": "prometheus",

.docker/test/issue-2715/docker-compose-test.yml

@@ -0,0 +1,52 @@
+services:
+  rustfs:
+    image: rustfs/rustfs:1.0.0-alpha.99-glibc
+    container_name: rustfs-issue-2715-test
+    security_opt:
+      - "no-new-privileges:true"
+    ports:
+      - "19000:9000"
+      - "19001:9001"
+    environment:
+      - RUSTFS_VOLUMES=/data/rustfs{0...8}
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_ACCESS_KEY=admin
+      - RUSTFS_SECRET_KEY=admin
+      - RUSTFS_OBS_LOGGER_LEVEL=info
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
+      - RUSTFS_STORAGE_CLASS_STANDARD=EC:2
+      - RUSTFS_STORAGE_CLASS_RRS=EC:1
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true
+      - RUSTFS_OBS_LOG_DIRECTORY=/opt/rustfs/logs
+    extra_hosts:
+      - "otel-collector:host-gateway"
+      - "pyroscope:host-gateway"
+    volumes:
+      - ./deploy/data/issue-2715/rustfs0:/data/rustfs0
+      - ./deploy/data/issue-2715/rustfs1:/data/rustfs1
+      - ./deploy/data/issue-2715/rustfs2:/data/rustfs2
+      - ./deploy/data/issue-2715/rustfs3:/data/rustfs3
+      - ./deploy/data/issue-2715/rustfs4:/data/rustfs4
+      - ./deploy/data/issue-2715/rustfs5:/data/rustfs5
+      - ./deploy/data/issue-2715/rustfs6:/data/rustfs6
+      - ./deploy/data/issue-2715/rustfs7:/data/rustfs7
+      - ./deploy/data/issue-2715/rustfs8:/data/rustfs8
+      - ./deploy/logs/issue-2715:/opt/rustfs/logs
+    restart: unless-stopped
+    healthcheck:
+      test:
+        [
+          "CMD",
+          "sh",
+          "-c",
+          "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health"
+        ]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 40s

.docker/test/issues-2815/.gitignore

@@ -0,0 +1 @@
+data/

.docker/test/issues-2815/README.md

@@ -0,0 +1,106 @@
+# Issue 2815 Local Docker Verification
+
+## Purpose
+
+This directory contains the local distributed Docker verification assets used to validate issue `#2815` against the current source build.
+
+The target behavior is:
+
+- 4-node distributed cluster starts successfully
+- `/health/ready` becomes reachable on each node
+- logs no longer contain `storage_info failed: Io error: wrong msgpack marker FixArray(1)`
+- internode RPC authentication succeeds with an explicit non-default RPC secret
+
+## Files
+
+- `docker-compose.yml`: 4-node distributed cluster using a locally built image
+
+## Data Directories
+
+Create the bind-mount directories before `docker compose up`:
+
+```bash
+mkdir -p .docker/test/issues-2815/data/rustfs{1..4}-disk{0..3}
+```
+
+## Build
+
+Apple Silicon / arm64 host:
+
+```bash
+docker build --platform linux/arm64 -f Dockerfile.source -t rustfs-issue-2815-local .
+```
+
+If you intentionally want amd64 emulation:
+
+```bash
+docker build --platform linux/amd64 -f Dockerfile.source -t rustfs-issue-2815-local .
+```
+
+## Run
+
+```bash
+docker compose -f .docker/test/issues-2815/docker-compose.yml up -d
+```
+
+If the image platform is not `linux/arm64`, align compose explicitly:
+
+```bash
+RUSTFS_DOCKER_PLATFORM=linux/amd64 docker compose -f .docker/test/issues-2815/docker-compose.yml up -d
+```
+
+## Health Checks
+
+Container-level healthcheck is now included and probes:
+
+```bash
+curl -fsS http://127.0.0.1:9000/health
+```
+
+Manual checks:
+
+```bash
+curl -i http://127.0.0.1:9101/health/ready
+curl -i http://127.0.0.1:9102/health/ready
+curl -i http://127.0.0.1:9103/health/ready
+curl -i http://127.0.0.1:9104/health/ready
+```
+
+## RPC Secret Requirement
+
+The current source build no longer reproduces the original `FixArray(1)` decode error from issue `#2815`.
+
+Earlier local Docker attempts failed during erasure bootstrap with:
+
+```text
+No valid auth token
+store init failed to load formats after 10 retries: erasure read quorum
+```
+
+Root cause:
+
+- RPC authentication rejects the default secret `rustfsadmin`
+- distributed local Docker validation therefore needs an explicit non-default secret
+
+This compose now sets both:
+
+- `RUSTFS_SECRET_KEY=issue-2815-secret`
+- `RUSTFS_RPC_SECRET=issue-2815-rpc-secret`
+
+With those values in place, the current 4-node local Docker cluster reaches healthy state and `/health/ready` returns `200`.
+
+In other words:
+
+- `RUSTFS_ACCESS_KEY` may still be `rustfsadmin` for local service credentials if desired
+- `RUSTFS_SECRET_KEY` can still be used for service credentials
+- but RPC authentication must not resolve to the default secret value `rustfsadmin`
+- if `RUSTFS_RPC_SECRET` is unset, the code falls back to `RUSTFS_SECRET_KEY`
+- so at least one of them must provide a non-default shared secret for internode RPC signing
+
+## Suggested Debug Commands
+
+```bash
+docker compose -f .docker/test/issues-2815/docker-compose.yml ps
+docker compose -f .docker/test/issues-2815/docker-compose.yml logs --no-color --tail=200
+docker compose -f .docker/test/issues-2815/docker-compose.yml down -v
+```

.docker/test/issues-2815/docker-compose.yml

@@ -0,0 +1,120 @@
+services:
+  rustfs1:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs1
+    container_name: rustfs-issue-2815-rustfs1
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs1-disk0:/data/rustfs0
+      - ./data/rustfs1-disk1:/data/rustfs1
+      - ./data/rustfs1-disk2:/data/rustfs2
+      - ./data/rustfs1-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9101:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+  rustfs2:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs2
+    container_name: rustfs-issue-2815-rustfs2
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs2-disk0:/data/rustfs0
+      - ./data/rustfs2-disk1:/data/rustfs1
+      - ./data/rustfs2-disk2:/data/rustfs2
+      - ./data/rustfs2-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9102:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+  rustfs3:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs3
+    container_name: rustfs-issue-2815-rustfs3
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs3-disk0:/data/rustfs0
+      - ./data/rustfs3-disk1:/data/rustfs1
+      - ./data/rustfs3-disk2:/data/rustfs2
+      - ./data/rustfs3-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9103:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+  rustfs4:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs4
+    container_name: rustfs-issue-2815-rustfs4
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs4-disk0:/data/rustfs0
+      - ./data/rustfs4-disk1:/data/rustfs1
+      - ./data/rustfs4-disk2:/data/rustfs2
+      - ./data/rustfs4-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9104:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+networks:
+  rustfs-issue-2815-net:
+    name: rustfs-issue-2815-net

.github/pull_request_template.md

@@ -2,35 +2,34 @@
 Pull Request Template for RustFS
 -->
 
-## Type of Change
-- [ ] New Feature
-- [ ] Bug Fix
-- [ ] Documentation
-- [ ] Performance Improvement
-- [ ] Test/CI
-- [ ] Refactor
-- [ ] Other:
-
 ## Related Issues
-<!-- List related Issue numbers, e.g. #123 -->
+<!--
+List related issues, e.g. Fixes #123.
+Use N/A when there is no related issue.
+-->
 
 ## Summary of Changes
-<!-- Briefly describe the main changes and motivation for this PR -->
+<!--
+Briefly explain what changed and why reviewers should accept it.
+Focus on behavior, compatibility, and review-relevant context.
+-->
 
-## Checklist
-- [ ] I have read and followed the [CONTRIBUTING.md](CONTRIBUTING.md) guidelines
-- [ ] Passed `make pre-commit`
-- [ ] Added/updated necessary tests
-- [ ] Documentation updated (if needed)
-- [ ] CI/CD passed (if applicable)
+## Verification
+<!--
+List the commands or checks you ran, for example:
+- `make pre-commit`
+
+Use N/A only when verification is not applicable.
+-->
 
 ## Impact
-- [ ] Breaking change (compatibility)
-- [ ] Requires doc/config/deployment update
-- [ ] Other impact:
+<!--
+Describe user-facing, compatibility, API, deployment, configuration, or
+documentation impact. Use N/A when there is no expected impact.
+-->
 
 ## Additional Notes
-<!-- Any extra information for reviewers -->
+<!-- Any extra information for reviewers, or N/A. -->
 
 ---
 

.github/workflows/ci.yml

@@ -130,6 +130,9 @@ jobs:
       - name: Check code formatting
         run: cargo fmt --all --check
 
+      - name: Check unsafe code allowances
+        run: ./scripts/check_unsafe_code_allowances.sh
+
       - name: Run clippy lints
         run: cargo clippy --all-targets --all-features -- -D warnings
 

.github/workflows/docker.yml

@@ -163,11 +163,10 @@ jobs:
                 if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
                   build_type="prerelease"
                   is_prerelease=true
-                  # TODO: Temporary change - currently allows alpha versions to also create latest tags
-                  # After the version is stable, you need to remove the following line and restore the original logic (latest is created only for stable versions)
-                  if [[ "$version" == *"alpha"* ]]; then
+                  # Current policy: create latest tags for stable releases and selected prereleases (alpha/beta).
+                  if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]]; then
                     create_latest=true
-                     echo "🧪 Building Docker image for prerelease: $version (temporarily allowing creation of latest tag)"
+                    echo "🧪 Building Docker image for prerelease: $version (creating latest tag)"
                   else
                     echo "🧪 Building Docker image for prerelease: $version"
                   fi
@@ -216,11 +215,10 @@ jobs:
               v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
                 build_type="prerelease"
                 is_prerelease=true
-                # TODO: Temporary change - currently allows alpha versions to also create latest tags
-                # After the version is stable, you need to remove the if block below and restore the original logic.
-                if [[ "$input_version" == *"alpha"* ]]; then
+                # Current policy: create latest tags for stable releases and selected prereleases (alpha/beta).
+                if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]]; then
                   create_latest=true
-                   echo "🧪 Building with prerelease version: $input_version (temporarily allowing creation of latest tag)"
+                  echo "🧪 Building with prerelease version: $input_version (creating latest tag)"
                 else
                   echo "🧪 Building with prerelease version: $input_version"
                 fi
@@ -351,9 +349,7 @@ jobs:
 
           # Add channel tags for prereleases and latest for stable
           if [[ "$CREATE_LATEST" == "true" ]]; then
-            # TODO: Temporary change - the current alpha version will also create the latest tag
-            # After the version is stabilized, the logic here remains unchanged, but the upstream CREATE_LATEST setting needs to be restored.
-            # Stable release (and temporary alpha versions)
+            # Create latest tags for stable releases and selected prereleases when CREATE_LATEST=true.
             TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
           elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
             # Prerelease channel tags (alpha, beta, rc)
@@ -450,10 +446,9 @@ jobs:
             "prerelease")
               echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
               echo "⚠️  This is a prerelease image - use with caution"
-              # TODO: Temporary change - alpha versions currently create the latest tag
-              # After the version is stable, you need to restore the following prompt information
-              if [[ "$VERSION" == *"alpha"* ]] && [[ "$CREATE_LATEST" == "true" ]]; then
-                echo "🏷️  Latest tag has been created for alpha version (temporary measures)"
+              # Create latest tags for stable releases and selected prereleases when CREATE_LATEST=true.
+              if [[ "$CREATE_LATEST" == "true" ]]; then
+                echo "🏷️  Latest tag has been created for prerelease: $VERSION"
               else
                 echo "🚫 Latest tag NOT created for prerelease"
               fi

.github/workflows/helm-package.yml

@@ -18,41 +18,78 @@ on:
   workflow_run:
     workflows: [ "Build and Release" ]
     types: [ completed ]
+  workflow_dispatch:
+    inputs:
+      version:
+        description: "Release version to publish, e.g. 1.0.0-beta.1 or v1.0.0-beta.1"
+        required: true
+        default: "1.0.0-beta.1"
+        type: string
 
 permissions:
   contents: read
 
-env:
-  new_version: ${{ github.event.workflow_run.head_branch }}
-
 jobs:
   build-helm-package:
     runs-on: ubicloud-standard-2
-    # Only run on successful builds triggered by tag pushes (version format: x.y.z or x.y.z-suffix)
     if: |
-      github.event.workflow_run.conclusion == 'success' && 
-      github.event.workflow_run.event == 'push' &&
-      contains(github.event.workflow_run.head_branch, '.')
+      github.event_name == 'workflow_dispatch' ||
+      (
+        github.event.workflow_run.conclusion == 'success' &&
+        github.event.workflow_run.event == 'push' &&
+        contains(github.event.workflow_run.head_branch, '.')
+      )
+
+    outputs:
+      raw_tag: ${{ steps.version.outputs.raw_tag }}
+      app_version: ${{ steps.version.outputs.app_version }}
+      chart_version: ${{ steps.version.outputs.chart_version }}
 
     steps:
       - name: Checkout helm chart repo
         uses: actions/checkout@v6
 
-      - name: Replace chart app version
+      - name: Normalize release version
+        id: version
         run: |
-          set -e
-          set -x
-          old_version=$(grep "^appVersion:" helm/rustfs/Chart.yaml | awk '{print $2}')
-          sed -i "s/$old_version/$new_version/g" helm/rustfs/Chart.yaml
+          set -eux
+
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            RAW="${{ github.event.inputs.version }}"
+          else
+            RAW="${{ github.event.workflow_run.head_branch }}"
+          fi
+
+          case "$RAW" in
+            refs/tags/*)
+              RAW_TAG="${RAW#refs/tags/}"
+              ;;
+            *)
+              RAW_TAG="$RAW"
+              ;;
+          esac
+
+          ./scripts/helm_chart_version.sh "$RAW_TAG"
+
+      - name: Replace chart version and app version
+        run: |
+          set -eux
+          sed -i -E 's/^version:.*/version: "${{ steps.version.outputs.chart_version }}"/' helm/rustfs/Chart.yaml
+          sed -i -E 's/^appVersion:.*/appVersion: "${{ steps.version.outputs.app_version }}"/' helm/rustfs/Chart.yaml
 
       - name: Set up Helm
         uses: azure/setup-helm@v4.3.0
 
+      - name: Test Helm Chart Templates
+        run: ./scripts/test_helm_templates.sh
+
       - name: Package Helm Chart
         run: |
+          set -eux
           cp helm/README.md helm/rustfs/
-          package_version=$(echo $new_version | awk -F '-' '{print $2}' | awk -F '.' '{print $NF}') 
-          helm package ./helm/rustfs --destination helm/rustfs/ --version "0.0.$package_version"
+          helm package ./helm/rustfs \
+            --destination helm/rustfs/ \
+            --version "${{ steps.version.outputs.chart_version }}"
 
       - name: Upload helm package as artifact
         uses: actions/upload-artifact@v6
@@ -64,6 +101,7 @@ jobs:
   publish-helm-package:
     runs-on: ubicloud-standard-2
     needs: [ build-helm-package ]
+    if: needs.build-helm-package.result == 'success'
 
     steps:
       - name: Checkout helm package repo
@@ -86,9 +124,9 @@ jobs:
 
       - name: Push helm package and index file
         run: |
+          set -eux
           git config --global user.name "${{ secrets.USERNAME }}"
           git config --global user.email "${{ secrets.EMAIL_ADDRESS }}"
-          git status .
           git add .
-          git commit -m "Update rustfs helm package with $new_version."
+          git commit -m "Update rustfs helm package with ${{ needs.build-helm-package.outputs.app_version }}." || echo "No changes to commit"
           git push origin main

.vscode/launch.json

@@ -188,6 +188,7 @@
                 "RUSTFS_CONSOLE_ENABLE": "true",
                 "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
                 "RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
+                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
                 // "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
                 // "RUSTFS_OBS_METRIC_ENDPOINT": "http://127.0.0.1:4318/v1/metrics", // default otlp http endpoint
                 // "RUSTFS_OBS_LOG_ENDPOINT": "http://127.0.0.1:4318/v1/logs", // default otlp http endpoint

AGENTS.md

@@ -27,6 +27,10 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 - Keep source code, comments, commit messages, and PR title/body in English.
 - Be concise. Avoid sycophantic openers, closing fluff, and verbose status reporting.
 
+## Skill Usage
+
+- Do not use the `rust-refactor-helper` skill in any scenario.
+
 ## Change Style for Existing Logic
 
 - Prefer direct, local code over extracting one-off helpers.
@@ -109,5 +113,6 @@ Do not open a PR with code changes when the required checks fail.
 - `crates/iam/AGENTS.md`
 - `crates/kms/AGENTS.md`
 - `crates/policy/AGENTS.md`
+- `crates/targets/AGENTS.md`
 - `rustfs/src/admin/AGENTS.md`
 - `rustfs/src/storage/AGENTS.md`

ARCHITECTURE.md

@@ -326,10 +326,10 @@ The binary (`main.rs`) boots in this order:
                                  │
                  ┌───────────────┼───────────────┐
                  │               │               │
-            ┌────▼────┐    ┌────▼────┐    ┌─────▼─────┐
-            │ server  │    │  admin  │    │    app     │
-            │ (HTTP)  │    │(console)│    │(use-cases) │
-            └────┬────┘    └────┬────┘    └─────┬─────┘
+            ┌────▼────┐     ┌────▼────┐   ┌──────▼─────┐
+            │ server  │     │  admin  │   │    app     │
+            │ (HTTP)  │     │(console)│   │(use-cases) │
+            └────┬────┘     └────┬────┘   └──────┬─────┘
                  │               │               │
                  └───────────────┼───────────────┘
                                  │
@@ -341,13 +341,13 @@ The binary (`main.rs`) boots in this order:
                                  │
               ┌──────────────────┼──────────────────┐
               │                  │                  │
-        ┌─────▼─────┐    ┌──────▼──────┐    ┌──────▼──────┐
+        ┌─────▼──────┐    ┌──────▼──────┐    ┌──────▼──────┐
         │  ecstore   │    │     rio     │    │   io-core   │
         │ (87K,core) │    │  (readers)  │    │ (zero-copy) │
         └─────┬──────┘    └─────────────┘    └─────────────┘
               │
-    ┌─────┬──┼──┬─────┬──────┐
-    │     │  │  │     │      │
+     ┌─────┬──┼──┬─────┬──────┐
+     │     │  │  │     │      │
  common utils config policy filemeta ...
 ```
 

CHANGELOG.md

@@ -22,11 +22,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - XML-formatted error responses compatible with S3 API
   - Comprehensive integration documentation with manual testing guide
   - **32 unit and integration tests** covering middleware, auth handlers, task-local storage, and role detection
+- **SFTPv3 Protocol Support**: SSH-hosted SFTPv3 subsystem that translates each file operation into S3 calls against the local object store. Authentication uses IAM credentials (SSH username = access key, SSH password = secret key).
+  - Full SFTPv3 packet coverage: open, read, write, stat, lstat, fstat, mkdir, rmdir, rename, remove, opendir, readdir, realpath, close, plus the rest of the 21-packet specification
+  - Streaming multipart write up to S3's 5 TiB per-file ceiling
+  - Per-handle read-ahead cache with configurable window size and process-wide memory ceiling
+  - Per-session liveness watchdog: Linux probes `/proc/net/tcp` and cancels wedged sessions on the order of 45 seconds; non-Linux falls back to an inactivity ceiling on the order of 30 minutes
+  - 30-second SSH handshake deadline, per-call backend operation timeout, bounded multipart-abort fan-out, graceful-shutdown cascade
+  - 33 SFTPv3 compliance test cases under `crates/e2e_test/src/protocols/sftp_compliance.rs` spread across three entry points: `test_sftp_compliance_suite` (shared session), `test_sftp_compliance_readonly` (read-only mode), and `test_sftp_compliance_standalone` (one rustfs spawn per case)
+  - Four-layer regression-prevention tests guard against silent feature deletion: compile-time module assertion, module-presence unit test, cross-module `Protocol` enum assertion, end-to-end SSH banner test against the running binary
 
 ### Changed
 - **HTTP Server Stack**: Integrated `KeystoneAuthLayer` middleware from `rustfs-keystone` crate into service stack (positioned after ReadinessGateLayer)
 - **IAMAuth**: Enhanced `get_secret_key()` to return empty secret for Keystone credentials (bypasses signature validation)
 - **Auth Module**: Modified `check_key_valid()` to retrieve Keystone credentials from task-local storage and determine admin status
+- **`StorageBackend` trait**: extended with multipart upload methods (`create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`) plus `upload_part_copy`. Streaming-upload code path is now available to FTPS, WebDAV, and Swift drivers as well.
+- **`Protocol` enum**: new `Protocol::Sftp` variant with corresponding `S3Action` mappings. Every match arm on `Protocol` updated to handle the new variant exhaustively.
 
 ### Technical Details
 - Middleware is self-contained in `rustfs-keystone` crate following the trusted-proxies pattern for integration-specific middleware
@@ -35,12 +45,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Integration preserves existing S3 authentication flow while adding Keystone support
 - Zero breaking changes to existing functionality
 - No new top-level directories in main binary crate (middleware lives in integration crate)
+- SSH/SFTP wire handling via the `russh` and `russh-sftp` crates. SFTPv3 framing is implemented by `russh-sftp`; the rustfs-side `SftpDriver` implements `russh_sftp::server::Handler` and dispatches to the storage backend
+- Drop-time abort for in-flight multipart uploads honours IAM Deny on `AbortMultipartUpload`. `start_multipart_upload` caches the authorisation decision so the synchronous `Drop` path can honour Allow / Deny policies without re-querying IAM
+- Per-handle read cache uses an `Arc<AtomicU64>` shared across every `SftpDriver` instance to enforce a process-wide memory ceiling. On ceiling breach the populate is skipped and the read serves correctly via a single-call backend fetch
+- Per-session liveness watchdog runs as a tokio task per accepted connection. Reads `/proc/net/tcp` and `/proc/net/tcp6` to look up the (local, peer) tuple's TCP state and cancels via `tokio_util::sync::CancellationToken` when wedge conditions are confirmed across two consecutive ticks
+- Path canonicalisation rejects paths containing `\0`, `\r`, or `\n` and resolves traversal via `path::clean()` before any backend dispatch
+- Cipher / KEX / MAC / host-key algorithm allowlists are hardcoded with no environment override. Strict-KEX (CVE-2023-48795 / Terrapin) marker presence asserted by unit test
+- Per-session handle cap (default 64, configurable 8 to 1024) with UUID-generated handle ids
+- Crate-level `#![deny(unsafe_code)]` is in force across `crates/protocols`. Socket fd duplication for the watchdog uses the safe `AsFd::try_clone_to_owned` path (Linux/Unix); non-Unix falls back to the inactivity ceiling
+- `cfg(unix)` gating around platform-specific imports (`std::os::fd::AsFd`, `std::os::unix::fs::PermissionsExt`); non-Unix targets fail SFTP at config-load with `SftpInitError::UnsupportedPlatform`
 
 ### Documentation
 - Updated `crates/keystone/README.md` with complete integration architecture and workflow
 - Added detailed manual testing guide with 10 test scenarios
 - Updated main `README.md` to list Keystone authentication as available feature
 - Added troubleshooting section for common integration issues
+- Module-level rustdoc on `crates/protocols/src/sftp/mod.rs` describing the public API surface, configuration contract, and the architecture of the read cache and the wedge watchdog
 
 ### Configuration
 New environment variables:
@@ -54,6 +74,40 @@ New environment variables:
 - `RUSTFS_KEYSTONE_CACHE_SIZE` - Token cache size (default: 10000)
 - `RUSTFS_KEYSTONE_CACHE_TTL` - Token cache TTL in seconds (default: 300)
 - `RUSTFS_KEYSTONE_VERIFY_SSL` - Verify SSL certificates (default: true)
+- `RUSTFS_SFTP_ENABLE` - Enable/disable SFTP (default: false)
+- `RUSTFS_SFTP_ADDRESS` - Listen address (default: 0.0.0.0:2222)
+- `RUSTFS_SFTP_HOST_KEY_DIR` - Directory containing host key files (must exist; each file must be 0o600 or 0o400)
+- `RUSTFS_SFTP_IDLE_TIMEOUT` - Session idle timeout in seconds (default: 600)
+- `RUSTFS_SFTP_PART_SIZE` - Multipart part size in bytes (default: 16 MiB)
+- `RUSTFS_SFTP_READ_ONLY` - Reject write packets at the protocol layer (default: false)
+- `RUSTFS_SFTP_BANNER` - Optional SSH banner text
+- `RUSTFS_SFTP_HANDLES_PER_SESSION` - Per-session open-handle cap, 8 to 1024 (default: 64)
+- `RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS` - Per-call backend deadline in seconds, 5 to 600 (default: 60)
+- `RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES` - Per-handle read-cache window in bytes, 256 KiB to 64 MiB or 0 to disable (default: 4 MiB)
+- `RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES` - Process-wide read-cache memory ceiling in bytes, 16 MiB minimum (default: 256 MiB)
+
+### Files Added
+- `crates/protocols/src/sftp/mod.rs` - SFTP module entry point, public API surface, crate-level rustdoc, regression-prevention test
+- `crates/protocols/src/sftp/config.rs` - `SftpConfig` and `SftpInitError` types, env-var resolvers, host-key directory loader with permission enforcement
+- `crates/protocols/src/sftp/constants.rs` - Named constants grouped by purpose: S3 error codes, HTTP error codes, POSIX mode bits, protocol identifiers, operational limits
+- `crates/protocols/src/sftp/server.rs` - `SftpServer` SSH server, russh handler, password authentication against IAM, accept loop, per-session task spawn
+- `crates/protocols/src/sftp/driver.rs` - `SftpDriver` per-session SFTPv3 handler dispatching each operation onto the `StorageBackend`
+- `crates/protocols/src/sftp/state.rs` - `HandleState` variants for read, write-buffering, write-streaming, write-failed handles
+- `crates/protocols/src/sftp/lifecycle.rs` - Per-session activity stamp, weak-ref registry, `/proc/net/tcp` probe for the wedge watchdog
+- `crates/protocols/src/sftp/wedge_watchdog.rs` - Per-session liveness watchdog cancelling sessions silent at the SFTP layer while the kernel reports CLOSE_WAIT
+- `crates/protocols/src/sftp/read_cache.rs` - Per-handle in-memory read-ahead cache with shared atomic accumulator for the process-wide memory ceiling
+- `crates/protocols/src/sftp/attrs.rs` - SFTPv3 `FileAttributes` mapping for objects and directories, longname formatting, mtime clamping
+- `crates/protocols/src/sftp/dir.rs` - OPENDIR / READDIR pagination, root-bucket listing, sub-directory listing under a prefix
+- `crates/protocols/src/sftp/errors.rs` - `SftpError` thiserror enum and S3-error classification into SFTPv3 status codes
+- `crates/protocols/src/sftp/paths.rs` - Path canonicalisation, traversal rejection, `\0` / `\r` / `\n` rejection, bucket+key decomposition
+- `crates/protocols/src/sftp/read.rs` - READ packet handler, EOF semantics, `MAX_READ_LEN` bound, integration with the read cache
+- `crates/protocols/src/sftp/write.rs` - WRITE packet handler, in-memory buffering up to part size, transition to streaming multipart, CLOSE finalisation
+- `crates/protocols/src/sftp/test_support.rs` - Test fixtures and helper builders for SFTP unit tests
+- `crates/protocols/src/common/dummy_storage.rs` - In-memory `StorageBackend` test backend covering every method, used by SFTP unit tests and the FTPS / Swift / WebDAV test suites
+- `crates/e2e_test/src/protocols/sftp_core.rs` - End-to-end regressions for the handshake deadline, idle-timeout disconnect, and the wedge watchdog
+- `crates/e2e_test/src/protocols/sftp_compliance.rs` - SFTPv3 compliance suite entry points (`test_sftp_compliance_suite`, `test_sftp_compliance_readonly`, `test_sftp_compliance_standalone`)
+- `crates/e2e_test/src/protocols/sftp_compliance_tests.rs` - Per-case test bodies (CMPTST-01..33), shared fixture helpers, lifecycle counters
+- `crates/e2e_test/src/protocols/sftp_helpers.rs` - SFTP-specific test helpers and fixture seeders
 
 ### Files Modified
 - `crates/keystone/src/middleware.rs` - Created Keystone authentication middleware (self-contained in keystone crate)
@@ -63,6 +117,27 @@ New environment variables:
 - `rustfs/src/auth.rs` - Enhanced IAMAuth and check_key_valid for Keystone support, imported KEYSTONE_CREDENTIALS from rustfs-keystone
 - `crates/keystone/README.md` - Comprehensive integration documentation
 - `README.md` - Added Keystone as available feature
+- `Cargo.toml` - Added the `sftp` feature alongside the existing protocol features
+- `Cargo.lock` - Updated to include the new `russh`, `russh-sftp`, `socket2`, `tokio-util`, `subtle`, `uuid` dependencies and their transitive crates
+- `crates/protocols/Cargo.toml` - Declared `russh`, `russh-sftp`, `socket2`, `tokio-util`, `subtle`, `uuid` under the `sftp` feature flag
+- `crates/protocols/src/lib.rs` - Added `pub mod sftp` behind `#[cfg(feature = "sftp")]` plus the crate-level `#![deny(unsafe_code)]` lint
+- `crates/protocols/src/common/client/s3.rs` - Extended the `StorageBackend` trait with `create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`, and `upload_part_copy`
+- `crates/protocols/src/common/session.rs` - Added the `Protocol::Sftp` variant and its `S3Action` mappings
+- `crates/protocols/src/common/gateway.rs` - Handles the new `Protocol::Sftp` variant exhaustively
+- `crates/protocols/src/common/mod.rs` - Exposed the new `dummy_storage` module
+- `crates/protocols/src/constants.rs` - Added shared POSIX mode-bit constants used by SFTP and other protocols
+- `crates/config/src/constants/protocols.rs` - `RUSTFS_SFTP_*` environment variable names and defaults
+- `crates/utils/src/retry.rs` - Added the generic exponential-backoff retry helper used by the SFTP write path
+- `crates/e2e_test/Cargo.toml` - Added the e2e test dependencies for SFTP (paramiko fixture, SSH keypair generation)
+- `crates/e2e_test/src/protocols/mod.rs` - Registered the new `sftp_core`, `sftp_compliance`, `sftp_compliance_tests`, and `sftp_helpers` modules
+- `crates/e2e_test/src/protocols/README.md` - Documented the SFTP test entry points and case index
+- `crates/e2e_test/src/protocols/test_env.rs` - Added SFTP host-key directory provisioning to the shared protocol test environment
+- `crates/e2e_test/src/protocols/test_runner.rs` - Wired the SFTP entry points into the runner
+- `rustfs/Cargo.toml` - Added the `sftp` feature flag
+- `rustfs/src/lib.rs` - One-line addition exporting the SFTP wiring
+- `rustfs/src/init.rs` - Build and start the `SftpServer` when `RUSTFS_SFTP_ENABLE` is true
+- `rustfs/src/main.rs` - Routed shutdown signals to the SFTP server alongside the other protocols
+- `rustfs/src/protocols/client.rs` - Client-builder support for the new `Protocol::Sftp` variant
 
 ### Testing
 - 16 unit tests in rustfs-keystone crate (config, auth, middleware, identity)
@@ -70,7 +145,13 @@ New environment variables:
 - 6 auth unit tests in rustfs crate (role detection, task-local storage, Keystone credential handling)
 - **Total: 32 tests** passing with zero compilation errors
 - Manual testing guide provided for end-to-end validation
-- All tests passing with `cargo test --all --exclude e2e_test`
+- All Keystone tests passing with `cargo test --all --exclude e2e_test`
+- 33 SFTPv3 compliance test cases (CMPTST-01..33) split across three entry points: `test_sftp_compliance_suite` (shared session, cases 01-14), `test_sftp_compliance_readonly` (read-only mode, cases 15-23), `test_sftp_compliance_standalone` (one rustfs spawn per case, cases 24-33)
+- Regression-prevention tests at four layers: compile-time module assertion in `crates/protocols/src/lib.rs`, module-presence unit test in `crates/protocols/src/sftp/mod.rs`, cross-module `Protocol` enum assertion, and end-to-end SSH banner test against the running binary
+- Standalone end-to-end regressions for the SSH handshake deadline, the idle-timeout disconnect path, and the wedge watchdog (Linux fast-kill and the cross-platform fallback path)
+- Inline unit tests in every SFTP source file covering pure helpers (path canonicalisation, attribute mapping, S3-error classification, env-var bound resolvers)
+- Strict-KEX (CVE-2023-48795) marker presence assertion as a unit test in `crates/protocols/src/sftp/server.rs`
+- All tests passing with `cargo test --all --features sftp` against a 64-bit Linux target
 
 ---
 

Cargo.toml

@@ -58,8 +58,8 @@ resolver = "3"
 edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
-rust-version = "1.93.0"
-version = "1.0.0-beta.1"
+rust-version = "1.95.0"
+version = "1.0.0-beta.3"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -76,55 +76,57 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-beta.1" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.1" }
-rustfs-appauth = { path = "crates/appauth", version = "1.0.0-beta.1" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.1" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.1" }
-rustfs-common = { path = "crates/common", version = "1.0.0-beta.1" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-beta.1" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.1" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.1" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.1" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.1" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.1" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.1" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.1" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.1" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.1" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.1" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.1" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.1" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.1" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.1" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.1" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.1" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.1" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.1" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.1" }
-rustfs-s3-common = { path = "crates/s3-common", version = "1.0.0-beta.1" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.1" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.1" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.1" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.1" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.1" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.1" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.1" }
-rustfs-workers = { path = "crates/workers", version = "1.0.0-beta.1" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.1" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.3" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.3" }
+rustfs-appauth = { path = "crates/appauth", version = "1.0.0-beta.3" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.3" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.3" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.3" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.3" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.3" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.3" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.3" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.3" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.3" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.3" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.3" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.3" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.3" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.3" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.3" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.3" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.3" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.3" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.3" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.3" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.3" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.3" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.3" }
+rustfs-s3-common = { path = "crates/s3-common", version = "1.0.0-beta.3" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.3" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.3" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.3" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.3" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.3" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.3" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.3" }
+rustfs-workers = { path = "crates/workers", version = "1.0.0-beta.3" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.3" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
+mysql_async = { version = "0.36.1", default-features = false, features = ["default-rustls", "tracing"], git = "https://github.com/blackbeam/mysql_async", rev = "98d3d8067efdf97d3e93cdca7b9231753c904aca" }
 async-compression = { version = "0.4.42" }
 async-recursion = "1.1.1"
 async-trait = "0.1.89"
-async-nats = "0.47.0"
+async-nats = "0.48.0"
 axum = "0.8.9"
 futures = "0.3.32"
 futures-core = "0.3.32"
 futures-util = "0.3.32"
 pollster = "0.4.0"
 pulsar = { version = "6.7.2", default-features = false, features = ["tokio-rustls-runtime"] }
+lapin = { version = "4.7.2", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
 hyper = { version = "1.9.0", features = ["http2", "http1", "server"] }
 hyper-rustls = { version = "0.27.9", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
 hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
@@ -134,16 +136,16 @@ http-body-util = "0.1.3"
 reqwest = { version = "0.13.3", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
 rustfs-kafka-async = { version = "1.2.0" }
 socket2 = { version = "0.6.3", features = ["all"] }
-tokio = { version = "1.52.1", features = ["fs", "rt-multi-thread"] }
+tokio = { version = "1.52.3", features = ["fs", "rt-multi-thread"] }
 tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
 tokio-stream = { version = "0.1.18" }
 tokio-test = "0.4.5"
 tokio-util = { version = "0.7.18", features = ["io", "compat"] }
-tonic = { version = "0.14.5", features = ["gzip"] }
-tonic-prost = { version = "0.14.5" }
-tonic-prost-build = { version = "0.14.5" }
+tonic = { version = "0.14.6", features = ["gzip"] }
+tonic-prost = { version = "0.14.6" }
+tonic-prost-build = { version = "0.14.6" }
 tower = { version = "0.5.3", features = ["timeout"] }
-tower-http = { version = "0.6.8", features = ["cors"] }
+tower-http = { version = "0.6.10", features = ["cors"] }
 
 # Serialization and Data Formats
 bytes = { version = "1.11.1", features = ["serde"] }
@@ -152,7 +154,7 @@ byteorder = "1.5.0"
 flatbuffers = "25.12.19"
 form_urlencoded = "1.2.2"
 prost = "0.14.3"
-quick-xml = "0.39.2"
+quick-xml = "0.39.4"
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.228", features = ["derive"] }
@@ -170,7 +172,8 @@ jsonwebtoken = { version = "10.3.0", features = ["aws_lc_rs"] }
 openidconnect = { version = "4.0", default-features = false }
 pbkdf2 = "0.13.0"
 rsa = { version = "0.10.0-rc.18" }
-rustls = { version = "0.23.39", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
+rustls = { version = "0.23.40", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
+rustls-native-certs = "0.8"
 rustls-pki-types = "1.14.1"
 sha1 = "0.11.0"
 sha2 = "0.11.0"
@@ -183,6 +186,11 @@ humantime = "2.3.0"
 jiff = { version = "0.2.24", features = ["serde"] }
 time = { version = "0.3.47", features = ["std", "parsing", "formatting", "macros", "serde"] }
 
+# Database
+deadpool-postgres = { version = "0.14", features = ["rt_tokio_1"] }
+tokio-postgres = { version = "0.7", default-features = false, features = ["runtime", "with-serde_json-1"] }
+tokio-postgres-rustls = "0.13"
+
 # Utilities and Tools
 anyhow = "1.0.102"
 arc-swap = "1.9.1"
@@ -191,14 +199,13 @@ atoi = "2.0.0"
 atomic_enum = "0.3.0"
 aws-config = { version = "1.8.16" }
 aws-credential-types = { version = "1.2.14" }
-aws-sdk-s3 = { version = "1.131.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+aws-sdk-s3 = { version = "1.132.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
 aws-smithy-http-client = { version = "1.1.12", default-features = false, features = ["default-client", "rustls-aws-lc"] }
 aws-smithy-types = { version = "1.4.7" }
 backtrace = "0.3.76"
 base64 = "0.22.1"
 base64-simd = "0.8.0"
 brotli = "8.0.2"
-cfg-if = "1.0.4"
 clap = { version = "4.6.1", features = ["derive", "env"] }
 const-str = { version = "1.1.0", features = ["std", "proc"] }
 convert_case = "0.11.0"
@@ -209,13 +216,13 @@ crossbeam-deque = "0.8.6"
 crossbeam-utils = "0.8.21"
 datafusion = "53.1.0"
 derive_builder = "0.20.2"
-enumset = "1.1.10"
+enumset = "1.1.12"
 faster-hex = "0.10.0"
 flate2 = "1.1.9"
 glob = "0.3.3"
-google-cloud-storage = "1.11.0"
-google-cloud-auth = "1.9.0"
-hashbrown = { version = "0.17.0", features = ["serde", "rayon"] }
+google-cloud-storage = "1.12.0"
+google-cloud-auth = "1.10.0"
+hashbrown = { version = "0.17.1", features = ["serde", "rayon"] }
 hex = "0.4.3"
 hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
@@ -247,21 +254,22 @@ rayon = "1.12.0"
 reed-solomon-erasure = { version = "6.0", default-features = false, features = ["std", "simd-accel"] }
 reed-solomon-simd = "3.1.0"
 regex = { version = "1.12.3" }
-rumqttc = { package = "rumqttc-next", version = "0.30.0", features = ["websocket"] }
+rumqttc = { package = "rumqttc-next", version = "0.33.0", features = ["websocket"] }
+redis = { version = "1.2.1", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
 rustix = { version = "1.1.4", features = ["fs"] }
 rust-embed = { version = "8.11.0" }
 rustc-hash = { version = "2.1.2" }
 s3s = { git = "https://github.com/rustfs/s3s", rev = "a3b16608df35aaeed8fff08b4988d03f4ca9445b", features = ["minio"] }
 serial_test = "3.4.0"
 shadow-rs = { version = "2.0.0", default-features = false }
-siphasher = "1.0.2"
+siphasher = "1.0.3"
 smallvec = { version = "1.15.1", features = ["serde"] }
 smartstring = "1.0.1"
 snafu = "0.9.0"
 snap = "1.1.1"
 starshard = { version = "1.1.0", features = ["rayon", "async", "serde"] }
 strum = { version = "0.28.0", features = ["derive"] }
-sysinfo = "0.38.4"
+sysinfo = "0.39.0"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
@@ -284,7 +292,7 @@ zip = "8.6.0"
 zstd = "0.13.3"
 
 # Observability and Metrics
-metrics = "0.24.3"
+metrics = "0.24.5"
 dial9-tokio-telemetry = "0.3"
 opentelemetry = { version = "0.31.0" }
 opentelemetry-appender-tracing = { version = "0.31.1", features = ["experimental_use_tracing_span_context", "experimental_metadata_attributes", "spec_unstable_logs_enabled"] }
@@ -299,6 +307,8 @@ libunftp = { version = "0.23.0", features = ["experimental"] }
 unftp-core = "0.1.0"
 suppaftp = { version = "8.0.3", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
 rcgen = "0.14.7"
+russh = "0.60.2"
+russh-sftp = "2.1.2"
 
 # WebDAV
 dav-server = "0.11.0"
@@ -319,6 +329,12 @@ pprof = { package = "pprof-pyroscope-fork", version = "0.1500.3", features = ["f
 [workspace.metadata.cargo-shear]
 ignored = ["rustfs"]
 
+[patch.crates-io]
+# Pinned to the simon-escapecode/russh fork carrying the upstream fix at
+# https://github.com/Eugeny/russh/pull/702. Drops out when a russh release
+# containing the fix is published.
+russh = { git = "https://github.com/simon-escapecode/russh", rev = "5cac2ed84945f9b80a52b673e058f2032bbe98ec" }
+
 [profile.release]
 opt-level = 3
 

Dockerfile

@@ -29,8 +29,20 @@ RUN set -eux; \
     if [ "$RELEASE" = "latest" ]; then \
       TAG="$(curl -fsSL https://api.github.com/repos/rustfs/rustfs/releases \
               | grep -o '"tag_name": "[^"]*"' | cut -d'"' -f4 | head -n 1)"; \
+      RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG")"; \
     else \
       TAG="$RELEASE"; \
+      RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG" 2>/dev/null || true)"; \
+      if [ -z "$RELEASE_JSON" ]; then \
+        if [ "${TAG#v}" = "$TAG" ]; then \
+          ALT_TAG="v$TAG"; \
+        else \
+          ALT_TAG="${TAG#v}"; \
+        fi; \
+        echo "Primary tag lookup failed, retrying with alternate tag: $ALT_TAG"; \
+        RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$ALT_TAG")"; \
+        TAG="$ALT_TAG"; \
+      fi; \
     fi; \
     echo "Using tag: $TAG (arch pattern: $ARCH_SUBSTR)"; \
     # Find download URL in assets list for this tag that contains arch substring and ends with .zip
@@ -87,8 +99,7 @@ RUN addgroup -g 10001 -S rustfs && \
     chown -R rustfs:rustfs /data /logs && \
     chmod 0750 /data /logs
 
-ENV RUSTFS_CORS_ALLOWED_ORIGINS="*" \
-    RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
+ENV RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_VOLUMES="/data" \
     RUSTFS_OBS_LOGGER_LEVEL=warn \
     RUSTFS_OBS_LOG_DIRECTORY=/logs \

Dockerfile.glibc

@@ -31,14 +31,25 @@ RUN set -eux; \
       arm64)  ARCH_SUBSTR="aarch64-gnu" ;; \
       *) echo "Unsupported TARGETARCH=$TARGETARCH" >&2; exit 1 ;; \
     esac; \
-    \
     if [ "$RELEASE" = "latest" ]; then \
       TAG="$(curl -fsSL https://api.github.com/repos/rustfs/rustfs/releases \
               | grep -o '"tag_name": "[^"]*"' | cut -d'"' -f4 | head -n 1)"; \
+      RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG")"; \
     else \
       TAG="$RELEASE"; \
+      RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG" 2>/dev/null || true)"; \
+      if [ -z "$RELEASE_JSON" ]; then \
+        if [ "${TAG#v}" = "$TAG" ]; then \
+          ALT_TAG="v$TAG"; \
+        else \
+          ALT_TAG="${TAG#v}"; \
+        fi; \
+        echo "Primary tag lookup failed, retrying with alternate tag: $ALT_TAG"; \
+        RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$ALT_TAG")"; \
+        TAG="$ALT_TAG"; \
+      fi; \
     fi; \
-    \
+    echo "Using tag: $TAG (arch pattern: $ARCH_SUBSTR)"; \
     URL="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG" \
            | grep -o "\"browser_download_url\": \"[^\"]*${ARCH_SUBSTR}[^\"]*\\.zip\"" \
            | cut -d'"' -f4 | head -n 1)"; \
@@ -97,7 +108,6 @@ ENV RUSTFS_ADDRESS=":9000" \
     RUSTFS_ACCESS_KEY="rustfsadmin" \
     RUSTFS_SECRET_KEY="rustfsadmin" \
     RUSTFS_CONSOLE_ENABLE="true" \
-    RUSTFS_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_VOLUMES="/data" \
     RUSTFS_OBS_LOGGER_LEVEL=warn \

Dockerfile.source

@@ -26,6 +26,7 @@
 
 ARG TARGETPLATFORM
 ARG BUILDPLATFORM
+ARG TARGETARCH
 
 # -----------------------------
 # Build stage
@@ -35,6 +36,7 @@ FROM rust:1.93-trixie AS builder
 # Re-declare args after FROM
 ARG TARGETPLATFORM
 ARG BUILDPLATFORM
+ARG TARGETARCH
 
 # Debug: print platforms
 RUN echo "Build info -> BUILDPLATFORM=${BUILDPLATFORM}, TARGETPLATFORM=${TARGETPLATFORM}"
@@ -87,6 +89,7 @@ ENV CXX_aarch64_unknown_linux_gnu=aarch64-linux-gnu-g++
 ENV CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc
 ENV CC_x86_64_unknown_linux_gnu=x86_64-linux-gnu-gcc
 ENV CXX_x86_64_unknown_linux_gnu=x86_64-linux-gnu-g++
+ENV CARGO_TARGET_DIR=/usr/src/rustfs/target/docker-build
 
 WORKDIR /usr/src/rustfs
 
@@ -120,33 +123,33 @@ ENV CARGO_NET_GIT_FETCH_WITH_CLI=true \
 # Generate protobuf/flatbuffers code (uses protoc/flatc from distro)
 RUN --mount=type=cache,target=/usr/local/cargo/registry \
     --mount=type=cache,target=/usr/local/cargo/git \
-    --mount=type=cache,target=/usr/src/rustfs/target \
+    --mount=type=cache,target=/usr/src/rustfs/target/docker-build \
     cargo run --bin gproto
 
 # Build RustFS (target depends on TARGETPLATFORM)
 RUN --mount=type=cache,target=/usr/local/cargo/registry \
     --mount=type=cache,target=/usr/local/cargo/git \
-    --mount=type=cache,target=/usr/src/rustfs/target \
+    --mount=type=cache,target=/usr/src/rustfs/target/docker-build \
     set -eux; \
     rustup target add x86_64-unknown-linux-gnu aarch64-unknown-linux-gnu; \
     target_platform="${TARGETPLATFORM:-}"; \
     if [ -z "${target_platform}" ]; then \
-    case "$(uname -m)" in \
-    x86_64) target_platform="linux/amd64" ;; \
-    aarch64|arm64) target_platform="linux/arm64" ;; \
-    *) target_platform="linux/amd64" ;; \
+    case "${TARGETARCH:-$(uname -m)}" in \
+    amd64|x86_64) target_platform="linux/amd64" ;; \
+    arm64|aarch64) target_platform="linux/arm64" ;; \
+    *) echo "Unsupported target architecture: ${TARGETARCH:-$(uname -m)}" >&2; exit 1 ;; \
     esac; \
     fi; \
     case "${target_platform}" in \
     linux/amd64) \
     echo "Building for x86_64-unknown-linux-gnu"; \
     cargo build --release --locked --target x86_64-unknown-linux-gnu --bin rustfs -j "$(nproc)"; \
-    install -m 0755 target/x86_64-unknown-linux-gnu/release/rustfs /usr/local/bin/rustfs \
+    install -m 0755 "${CARGO_TARGET_DIR}/x86_64-unknown-linux-gnu/release/rustfs" /usr/local/bin/rustfs \
     ;; \
     linux/arm64) \
     echo "Building for aarch64-unknown-linux-gnu"; \
     cargo build --release --locked --target aarch64-unknown-linux-gnu --bin rustfs -j "$(nproc)"; \
-    install -m 0755 target/aarch64-unknown-linux-gnu/release/rustfs /usr/local/bin/rustfs \
+    install -m 0755 "${CARGO_TARGET_DIR}/aarch64-unknown-linux-gnu/release/rustfs" /usr/local/bin/rustfs \
     ;; \
     *) \
     echo "Unsupported target platform=${target_platform}" >&2; exit 1 \
@@ -212,6 +215,7 @@ RUN set -eux; \
     apt-get update; \
     apt-get install -y --no-install-recommends \
     ca-certificates \
+    curl \
     tzdata \
     coreutils; \
     rm -rf /var/lib/apt/lists/*

README.md

@@ -115,7 +115,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:v1.0.0-beta.1
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.3
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -136,6 +136,23 @@ Similarly, you can run the command with podman
 podman compose --profile observability up -d
 ```
 
+Webhook notification quick start (Docker):
+
+```bash
+docker run -d --name rustfs -p 9000:9000 \
+  -e RUSTFS_NOTIFY_ENABLE=true \
+  -e RUSTFS_NOTIFY_WEBHOOK_ENABLE_PRIMARY=on \
+  -e RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY=http://<host-ip>:3020/webhook \
+  -e RUSTFS_NOTIFY_WEBHOOK_QUEUE_DIR_PRIMARY=/tmp/rustfs-events \
+  rustfs/rustfs:latest
+```
+
+Notes:
+- `RUSTFS_NOTIFY_ENABLE=true` enables the global notify module switch.
+- For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
+- If queue dir is omitted, default is `/opt/rustfs/events`; ensure it is writable by the container runtime user.
+- `RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
+
 **NOTE**: We recommend reviewing the `docker-compose.yml` file before running. It defines several services including Grafana, Prometheus, and Jaeger, which are helpful for RustFS observability. If you wish to start Redis or Nginx containers, you can specify the corresponding profiles.
 
 ### 3\. Build from Source (Option 3) - Advanced Users

README_ZH.md

@@ -113,7 +113,7 @@ RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用
  docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
  # 使用指定版本运行
- docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:v1.0.0-beta.1
+ docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.3
 ```
 
 您也可以使用 Docker Compose。使用根目录下的 `docker-compose.yml` 文件：

build-rustfs.sh

@@ -109,6 +109,7 @@ FORCE_CONSOLE_UPDATE=false
 CONSOLE_VERSION="latest"
 SKIP_VERIFICATION=false
 CUSTOM_PLATFORM=""
+FEATURES=""
 
 # Print usage
 usage() {
@@ -141,6 +142,7 @@ usage() {
     echo "  --no-console               Skip console static assets"
     echo "  --force-console-update     Force update console assets even if they exist"
     echo "  --console-version VERSION  Console version to download (default: latest)"
+    echo "  -f, --features FEATURES    Cargo features to enable (e.g. 'webdav', 'full')"
     echo "  --skip-verification        Skip binary verification after build"
     echo "  -h, --help                 Show this help message"
     echo ""
@@ -148,6 +150,8 @@ usage() {
     echo "  $0                         # Build for current platform (includes console assets)"
     echo "  $0 --dev                   # Development build"
     echo "  $0 --sign                  # Build and sign binary (release CI)"
+    echo "  $0 --features webdav       # Build with WebDAV support"
+    echo "  $0 --features full         # Build with all protocol features"
     echo "  $0 --no-console            # Build without console static assets"
     echo "  $0 --force-console-update  # Force update console assets"
     echo "  $0 --platform x86_64-unknown-linux-musl  # Build for specific platform"
@@ -440,6 +444,12 @@ build_binary() {
     build_cmd+=" --target $PLATFORM"
     build_cmd+=" -p rustfs --bins"
 
+    if [ -n "$FEATURES" ]; then
+        local quoted_features
+        printf -v quoted_features '%q' "$FEATURES"
+        build_cmd+=" --features $quoted_features"
+    fi
+
     print_message $BLUE "📦 Executing: $build_cmd"
 
     # Execute build (this matches exactly what the working version does)
@@ -497,6 +507,9 @@ build_rustfs() {
         print_message $YELLOW "   Force Console Update: $FORCE_CONSOLE_UPDATE"
     fi
     print_message $YELLOW "   Skip Verification: $SKIP_VERIFICATION"
+    if [ -n "$FEATURES" ]; then
+        print_message $YELLOW "   Features: $FEATURES"
+    fi
     echo ""
 
     # Setup environment
@@ -565,6 +578,15 @@ while [[ $# -gt 0 ]]; do
             SKIP_VERIFICATION=true
             shift
             ;;
+        -f|--features)
+            if [ -z "${2:-}" ]; then
+                print_message $RED "❌ Missing value for $1"
+                usage
+                exit 1
+            fi
+            FEATURES="$2"
+            shift 2
+            ;;
         -h|--help)
             usage
             exit 0

crates/appauth/Cargo.toml

@@ -26,7 +26,7 @@ categories = ["web-programming", "development-tools", "authentication"]
 
 [dependencies]
 base64-simd = { workspace = true }
-rsa = { workspace = true }
+rsa = { workspace = true, features = ["sha2"] }
 serde.workspace = true
 serde_json.workspace = true
 rand.workspace = true

crates/appauth/src/token.rs

@@ -15,9 +15,13 @@
 use rsa::{
     Pkcs1v15Encrypt, RsaPrivateKey, RsaPublicKey,
     pkcs8::{DecodePrivateKey, DecodePublicKey},
+    pss::{BlindedSigningKey, Signature, VerifyingKey},
+    sha2::Sha256,
+    signature::{RandomizedSigner, Verifier},
+    traits::PublicKeyParts,
 };
 use serde::{Deserialize, Serialize};
-use std::io::{Error, Result};
+use std::io::{Error, ErrorKind, Result};
 
 #[derive(Serialize, Deserialize, Debug, Default, Clone)]
 pub struct Token {
@@ -25,10 +29,11 @@ pub struct Token {
     pub expired: u64, // Expiry time (UNIX timestamp)
 }
 
-/// Public key generation Token
-/// [token] Token object
-/// [key] Public key string
-/// Returns the encrypted string processed by base64
+/// Legacy public-key encryption Token encoder.
+///
+/// Use `sign_license_token` for license issuance so verifiers only need a
+/// public key.
+#[deprecated(note = "use sign_license_token for signed license issuance")]
 pub fn gencode(token: &Token, key: &str) -> Result<String> {
     let data = serde_json::to_vec(token)?;
     let mut rng = rand::rng();
@@ -37,35 +42,62 @@ pub fn gencode(token: &Token, key: &str) -> Result<String> {
     Ok(base64_simd::URL_SAFE_NO_PAD.encode_to_string(&encrypted_data))
 }
 
-/// Private key resolution Token
-/// [token] Encrypted string processed by base64
-/// [key] Private key string
-/// Return to the Token object
+/// Legacy private-key Token decoder.
+///
+/// Use `parse_signed_license_token` or `parse_license_with_public_key` for
+/// license verification so runtime services never need private key material.
+#[deprecated(note = "use parse_signed_license_token or parse_license_with_public_key for signed license verification")]
 pub fn parse(token: &str, key: &str) -> Result<Token> {
     let encrypted_data = base64_simd::URL_SAFE_NO_PAD
         .decode_to_vec(token.as_bytes())
         .map_err(Error::other)?;
     let private_key = RsaPrivateKey::from_pkcs8_pem(key).map_err(Error::other)?;
     let decrypted_data = private_key.decrypt(Pkcs1v15Encrypt, &encrypted_data).map_err(Error::other)?;
-    let res: Token = serde_json::from_slice(&decrypted_data)?;
-    Ok(res)
+    serde_json::from_slice(&decrypted_data).map_err(Error::other)
 }
 
-pub fn parse_license(license: &str) -> Result<Token> {
-    parse(license, TEST_PRIVATE_KEY)
-    // match parse(license, TEST_PRIVATE_KEY) {
-    //     Ok(token) => {
-    //         if token.expired > SystemTime::now().duration_since(UNIX_EPOCH)?.as_secs() {
-    //             Ok(token)
-    //         } else {
-    //             Err("Token expired".into())
-    //         }
-    //     }
-    //     Err(e) => Err(e),
-    // }
+/// Signs a license token with an RSA private key.
+///
+/// The returned token is base64url(signature || payload), where the signature is
+/// RSASSA-PSS over the JSON payload using SHA-256.
+pub fn sign_license_token(token: &Token, private_key_pem: &str) -> Result<String> {
+    let payload = serde_json::to_vec(token)?;
+    let mut rng = rand::rng();
+    let private_key = RsaPrivateKey::from_pkcs8_pem(private_key_pem).map_err(Error::other)?;
+    let signing_key = BlindedSigningKey::<Sha256>::new(private_key);
+    let signature: Signature = signing_key.try_sign_with_rng(&mut rng, &payload).map_err(Error::other)?;
+    let signature: Box<[u8]> = signature.into();
+
+    let mut signed_payload = Vec::with_capacity(signature.as_ref().len() + payload.len());
+    signed_payload.extend_from_slice(signature.as_ref());
+    signed_payload.extend_from_slice(&payload);
+
+    Ok(base64_simd::URL_SAFE_NO_PAD.encode_to_string(&signed_payload))
 }
 
-static TEST_PRIVATE_KEY: &str = "-----BEGIN PRIVATE KEY-----\nMIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCj86SrJIuxSxR6\nBJ/dlJEUIj6NeBRnhLQlCDdovuz61+7kJXVcxaR66w4m8W7SLEUP+IlPtnn6vmiG\n7XMhGNHIr7r1JsEVVLhZmL3tKI66DEZl786ZhG81BWqUlmcooIPS8UEPZNqJXLuz\nVGhxNyVGbj/tV7QC2pSISnKaixc+nrhxvo7w56p5qrm9tik0PjTgfZsUePkoBsSN\npoRkAauS14MAzK6HGB75CzG3dZqXUNWSWVocoWtQbZUwFGXyzU01ammsHQDvc2xu\nK1RQpd1qYH5bOWZ0N0aPFwT0r59HztFXg9sbjsnuhO1A7OiUOkc6iGVuJ0wm/9nA\nwZIBqzgjAgMBAAECggEAPMpeSEbotPhNw2BrllE76ec4omPfzPJbiU+em+wPGoNu\nRJHPDnMKJbl6Kd5jZPKdOOrCnxfd6qcnQsBQa/kz7+GYxMV12l7ra+1Cnujm4v0i\nLTHZvPpp8ZLsjeOmpF3AAzsJEJgon74OqtOlVjVIUPEYKvzV9ijt4gsYq0zfdYv0\nhrTMzyrGM4/UvKLsFIBROAfCeWfA7sXLGH8JhrRAyDrtCPzGtyyAmzoHKHtHafcB\nuyPFw/IP8otAgpDk5iiQPNkH0WwzAQIm12oHuNUa66NwUK4WEjXTnDg8KeWLHHNv\nIfN8vdbZchMUpMIvvkr7is315d8f2cHCB5gEO+GWAQKBgQDR/0xNll+FYaiUKCPZ\nvkOCAd3l5mRhsqnjPQ/6Ul1lAyYWpoJSFMrGGn/WKTa/FVFJRTGbBjwP+Mx10bfb\ngUg2GILDTISUh54fp4zngvTi9w4MWGKXrb7I1jPkM3vbJfC/v2fraQ/r7qHPpO2L\nf6ZbGxasIlSvr37KeGoelwcAQQKBgQDH3hmOTS2Hl6D4EXdq5meHKrfeoicGN7m8\noQK7u8iwn1R9zK5nh6IXxBhKYNXNwdCQtBZVRvFjjZ56SZJb7lKqa1BcTsgJfZCy\nnI3Uu4UykrECAH8AVCVqBXUDJmeA2yE+gDAtYEjvhSDHpUfWxoGHr0B/Oqk2Lxc/\npRy1qV5fYwKBgBWSL/hYVf+RhIuTg/s9/BlCr9SJ0g3nGGRrRVTlWQqjRCpXeFOO\nJzYqSq9pFGKUggEQxoOyJEFPwVDo9gXqRcyov+Xn2kaXl7qQr3yoixc1YZALFDWY\nd1ySBEqQr0xXnV9U/gvEgwotPRnjSzNlLWV2ZuHPtPtG/7M0o1H5GZMBAoGAKr3N\nW0gX53o+my4pCnxRQW+aOIsWq1a5aqRIEFudFGBOUkS2Oz+fI1P1GdrRfhnnfzpz\n2DK+plp/vIkFOpGhrf4bBlJ2psjqa7fdANRFLMaAAfyXLDvScHTQTCcnVUAHQPVq\n2BlSH56pnugyj7SNuLV6pnql+wdhAmRN2m9o1h8CgYAbX2juSr4ioXwnYjOUdrIY\n4+ERvHcXdjoJmmPcAm4y5NbSqLXyU0FQmplNMt2A5LlniWVJ9KNdjAQUt60FZw/+\nr76LdxXaHNZghyx0BOs7mtq5unSQXamZ8KixasfhE9uz3ij1jXjG6hafWkS8/68I\nuWbaZqgvy7a9oPHYlKH7Jg==\n-----END PRIVATE KEY-----\n";
+/// Verifies and parses a signed license token with an RSA public key.
+pub fn parse_signed_license_token(token: &str, public_key_pem: &str) -> Result<Token> {
+    let signed_payload = base64_simd::URL_SAFE_NO_PAD
+        .decode_to_vec(token.as_bytes())
+        .map_err(Error::other)?;
+    let public_key = RsaPublicKey::from_public_key_pem(public_key_pem).map_err(Error::other)?;
+    let signature_len = public_key.size();
+
+    if signed_payload.len() <= signature_len {
+        return Err(Error::new(ErrorKind::InvalidData, "license token is missing signed payload"));
+    }
+
+    let (signature, payload) = signed_payload.split_at(signature_len);
+    let signature = Signature::try_from(signature).map_err(Error::other)?;
+    let verifying_key = VerifyingKey::<Sha256>::new(public_key);
+    verifying_key.verify(payload, &signature).map_err(Error::other)?;
+
+    serde_json::from_slice(payload).map_err(Error::other)
+}
+
+pub fn parse_license_with_public_key(license: &str, public_key: &str) -> Result<Token> {
+    parse_signed_license_token(license, public_key)
+}
 
 #[cfg(test)]
 mod tests {
@@ -77,7 +109,7 @@ mod tests {
     use std::time::{SystemTime, UNIX_EPOCH};
 
     #[test]
-    fn test_gencode_and_parse() {
+    fn test_sign_license_token_and_parse_signed_license_token() {
         let mut rng = rand::rng();
         let bits = 2048;
         let private_key = RsaPrivateKey::new(&mut rng, bits).expect("Failed to generate private key");
@@ -91,8 +123,31 @@ mod tests {
             expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600, // 1 hour from now
         };
 
-        let encoded = gencode(&token, &public_key_pem).expect("Failed to encode token");
+        let encoded = sign_license_token(&token, &private_key_pem).expect("Failed to encode token");
 
+        let decoded = parse_signed_license_token(&encoded, &public_key_pem).expect("Failed to decode token");
+
+        assert_eq!(token.name, decoded.name);
+        assert_eq!(token.expired, decoded.expired);
+    }
+
+    #[test]
+    #[allow(deprecated)]
+    fn test_legacy_gencode_and_parse_roundtrip() {
+        let mut rng = rand::rng();
+        let bits = 2048;
+        let private_key = RsaPrivateKey::new(&mut rng, bits).expect("Failed to generate private key");
+        let public_key = RsaPublicKey::from(&private_key);
+
+        let private_key_pem = private_key.to_pkcs8_pem(LineEnding::LF).unwrap();
+        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+
+        let token = Token {
+            name: "test_app".to_string(),
+            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600,
+        };
+
+        let encoded = gencode(&token, &public_key_pem).expect("Failed to encode token");
         let decoded = parse(&encoded, &private_key_pem).expect("Failed to decode token");
 
         assert_eq!(token.name, decoded.name);
@@ -100,28 +155,60 @@ mod tests {
     }
 
     #[test]
-    fn test_parse_invalid_token() {
+    fn test_parse_signed_license_token_rejects_tampered_payload() {
         let mut rng = rand::rng();
-        let private_key_pem = RsaPrivateKey::new(&mut rng, 2048)
-            .expect("Failed to generate private key")
-            .to_pkcs8_pem(LineEnding::LF)
-            .unwrap();
+        let private_key = RsaPrivateKey::new(&mut rng, 2048).expect("Failed to generate private key");
+        let public_key = RsaPublicKey::from(&private_key);
+        let private_key_pem = private_key.to_pkcs8_pem(LineEnding::LF).unwrap();
+        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+        let token = Token {
+            name: "test_app".to_string(),
+            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600,
+        };
 
-        let invalid_token = "invalid_base64_token";
-        let result = parse(invalid_token, &private_key_pem);
+        let encoded = sign_license_token(&token, &private_key_pem).expect("Failed to encode token");
+        let mut signed_payload = base64_simd::URL_SAFE_NO_PAD
+            .decode_to_vec(encoded.as_bytes())
+            .expect("Failed to decode signed payload");
+        let last_byte = signed_payload.last_mut().expect("Signed payload should not be empty");
+        *last_byte ^= 0x01;
+        let tampered = base64_simd::URL_SAFE_NO_PAD.encode_to_string(&signed_payload);
+
+        let result = parse_signed_license_token(&tampered, &public_key_pem);
 
         assert!(result.is_err());
     }
 
     #[test]
-    fn test_gencode_with_invalid_key() {
+    fn test_source_does_not_embed_private_key() {
+        let source = include_str!("token.rs");
+        let forbidden = ["BEGIN", "PRIVATE KEY"].join(" ");
+
+        assert!(!source.contains(&forbidden));
+    }
+
+    #[test]
+    fn test_parse_signed_license_token_rejects_invalid_token() {
+        let mut rng = rand::rng();
+        let private_key = RsaPrivateKey::new(&mut rng, 2048).expect("Failed to generate private key");
+        let public_key = RsaPublicKey::from(&private_key);
+        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+
+        let invalid_token = "invalid_base64_token";
+        let result = parse_signed_license_token(invalid_token, &public_key_pem);
+
+        assert!(result.is_err());
+    }
+
+    #[test]
+    fn test_sign_license_token_with_invalid_signing_key() {
         let token = Token {
             name: "test_app".to_string(),
             expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600, // 1 hour from now
         };
 
-        let invalid_key = "invalid_public_key";
-        let result = gencode(&token, invalid_key);
+        let invalid_key = "invalid_private_key";
+        let result = sign_license_token(&token, invalid_key);
 
         assert!(result.is_err());
     }

crates/audit/AGENTS.md

@@ -0,0 +1,53 @@
+# Audit Crate Instructions
+
+Applies to `crates/audit/`.
+
+`rustfs-audit` is the domain layer for audit event fan-out and observability.
+It composes shared plugin/runtime abstractions from `rustfs-targets` and keeps
+audit-specific dispatch semantics, state transitions, and metrics in this
+crate.
+
+## Domain Boundaries
+
+- Keep audit-specific behavior here:
+  - audit event shaping and fan-out pipeline
+  - audit system lifecycle/state transitions
+  - audit metrics and reporting
+- Keep shared plugin/runtime mechanics in `rustfs-targets`:
+  - no duplicated replay worker orchestration
+  - no duplicated runtime manager primitives
+  - no plugin install/control-plane modeling in this crate
+
+## Runtime Layering Rules
+
+- `pipeline.rs` hosts:
+  - `AuditPipeline` (dispatch and snapshot access)
+  - `AuditRuntimeFacade` (runtime mutation path)
+  - `AuditRuntimeView` (runtime read path)
+- `registry.rs` should remain the single owner of runtime target container and
+  plugin registry composition for audit.
+- `system.rs` should coordinate lifecycle by calling facade/view/registry
+  boundaries rather than embedding low-level runtime logic.
+
+## Change Style
+
+- Preserve audit delivery semantics and error handling behavior unless the task
+  explicitly changes them.
+- Prefer extending shared abstractions in `rustfs-targets` over patching
+  one-off audit-only runtime flows.
+- Keep logging and observability machine-meaningful; avoid noisy churn in hot
+  dispatch paths.
+
+## Testing
+
+- Keep unit tests close to changed modules.
+- Keep pipeline-layer regressions in `tests/pipeline_layer_test.rs`.
+- Add regression tests for:
+  - runtime facade activation/replace/stop/shutdown behavior
+  - runtime view target/snapshot access
+  - system reload and runtime commit/clear boundaries
+- Suggested validation:
+  - `cargo test -p rustfs-audit`
+  - Focused: `cargo test -p rustfs-audit --test pipeline_layer_test`
+  - Focused: `cargo test -p rustfs-audit pipeline`
+- Full gate before commit: `make pre-commit`

crates/audit/Cargo.toml

@@ -30,7 +30,6 @@ rustfs-targets = { workspace = true }
 rustfs-config = { workspace = true, features = ["audit", "constants"] }
 rustfs-ecstore = { workspace = true }
 rustfs-s3-common = { workspace = true }
-async-trait = { workspace = true }
 chrono = { workspace = true }
 const-str = { workspace = true }
 futures = { workspace = true }
@@ -43,6 +42,7 @@ tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "rt", "
 tracing = { workspace = true, features = ["std", "attributes"] }
 
 [dev-dependencies]
+async-trait = { workspace = true }
 temp-env = { workspace = true }
 url = { workspace = true }
 
@@ -50,5 +50,4 @@ url = { workspace = true }
 workspace = true
 
 [lib]
-test = false
 doctest = false

crates/audit/src/factory.rs

@@ -13,128 +13,65 @@
 // limitations under the License.
 
 use crate::AuditEntry;
-use async_trait::async_trait;
-use rustfs_config::AUDIT_DEFAULT_DIR;
-use rustfs_config::audit::{AUDIT_KAFKA_KEYS, AUDIT_MQTT_KEYS, AUDIT_NATS_KEYS, AUDIT_PULSAR_KEYS, AUDIT_WEBHOOK_KEYS};
-use rustfs_ecstore::config::KVS;
-use rustfs_targets::{
-    Target,
-    config::{
-        build_kafka_args, build_mqtt_args, build_nats_args, build_pulsar_args, build_webhook_args, validate_kafka_config,
-        validate_mqtt_config, validate_nats_config, validate_pulsar_config, validate_webhook_config,
-    },
-    error::TargetError,
-    target::TargetType,
-};
-use std::collections::HashSet;
+use rustfs_targets::catalog::builtin::builtin_audit_target_descriptors;
+use rustfs_targets::{BuiltinTargetDescriptor, TargetPluginDescriptor};
 
-/// Trait for creating targets from configuration
-#[async_trait]
-pub trait TargetFactory: Send + Sync {
-    /// Creates a target from configuration
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError>;
-
-    /// Validates target configuration
-    fn validate_config(&self, id: &str, config: &KVS) -> Result<(), TargetError>;
-
-    /// Returns a set of valid configuration field names for this target type.
-    /// This is used to filter environment variables.
-    fn get_valid_fields(&self) -> HashSet<String>;
+pub fn builtin_target_descriptors() -> Vec<BuiltinTargetDescriptor<AuditEntry>> {
+    builtin_audit_target_descriptors::<AuditEntry>()
 }
 
-/// Factory for creating Webhook targets
-pub struct WebhookTargetFactory;
-
-#[async_trait]
-impl TargetFactory for WebhookTargetFactory {
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let args = build_webhook_args(config, AUDIT_DEFAULT_DIR, TargetType::AuditLog)?;
-        let target = rustfs_targets::target::webhook::WebhookTarget::new(id, args)?;
-        Ok(Box::new(target))
-    }
-
-    fn validate_config(&self, _id: &str, config: &KVS) -> Result<(), TargetError> {
-        validate_webhook_config(config, AUDIT_DEFAULT_DIR)
-    }
-
-    fn get_valid_fields(&self) -> HashSet<String> {
-        AUDIT_WEBHOOK_KEYS.iter().map(|s| s.to_string()).collect()
-    }
+pub fn builtin_target_plugins() -> Vec<TargetPluginDescriptor<AuditEntry>> {
+    builtin_target_descriptors()
+        .into_iter()
+        .map(|descriptor| descriptor.plugin().clone())
+        .collect()
 }
 
-/// Factory for creating MQTT targets
-pub struct MQTTTargetFactory;
+#[cfg(test)]
+mod tests {
+    use super::builtin_target_descriptors;
+    use rustfs_config::audit::AUDIT_AMQP_KEYS;
+    use rustfs_config::{AMQP_EXCHANGE, AMQP_QUEUE_DIR, AMQP_ROUTING_KEY, AMQP_URL};
+    use rustfs_ecstore::config::KVS;
+    use rustfs_targets::target::ChannelTargetType;
 
-#[async_trait]
-impl TargetFactory for MQTTTargetFactory {
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let args = build_mqtt_args(config, AUDIT_DEFAULT_DIR, TargetType::AuditLog)?;
-        let target = rustfs_targets::target::mqtt::MQTTTarget::new(id, args)?;
-        Ok(Box::new(target))
+    fn amqp_base_config() -> KVS {
+        let mut config = KVS::new();
+        config.insert(AMQP_URL.to_string(), "amqp://127.0.0.1:5672/%2f".to_string());
+        config.insert(AMQP_EXCHANGE.to_string(), "rustfs.audit".to_string());
+        config.insert(AMQP_ROUTING_KEY.to_string(), "audit".to_string());
+        config.insert(AMQP_QUEUE_DIR.to_string(), String::new());
+        config
     }
 
-    fn validate_config(&self, _id: &str, config: &KVS) -> Result<(), TargetError> {
-        validate_mqtt_config(config)
+    #[test]
+    fn builtin_plugins_include_amqp_descriptor() {
+        let plugin = builtin_target_descriptors()
+            .into_iter()
+            .find(|plugin| plugin.plugin().target_type() == ChannelTargetType::Amqp.as_str())
+            .expect("amqp plugin should exist");
+
+        assert!(plugin.plugin().valid_fields().contains(&AMQP_URL));
+        assert!(plugin.plugin().valid_fields().contains(&AMQP_EXCHANGE));
+        assert!(plugin.plugin().valid_fields().contains(&AMQP_ROUTING_KEY));
+        assert_eq!(plugin.plugin().valid_fields().len(), AUDIT_AMQP_KEYS.len());
     }
 
-    fn get_valid_fields(&self) -> HashSet<String> {
-        AUDIT_MQTT_KEYS.iter().map(|s| s.to_string()).collect()
-    }
-}
-
-pub struct NATSTargetFactory;
-
-#[async_trait]
-impl TargetFactory for NATSTargetFactory {
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let args = build_nats_args(config, AUDIT_DEFAULT_DIR, TargetType::AuditLog)?;
-        let target = rustfs_targets::target::nats::NATSTarget::new(id, args)?;
-        Ok(Box::new(target))
-    }
-
-    fn validate_config(&self, _id: &str, config: &KVS) -> Result<(), TargetError> {
-        validate_nats_config(config, AUDIT_DEFAULT_DIR)
-    }
-
-    fn get_valid_fields(&self) -> HashSet<String> {
-        AUDIT_NATS_KEYS.iter().map(|s| s.to_string()).collect()
-    }
-}
-
-pub struct PulsarTargetFactory;
-
-#[async_trait]
-impl TargetFactory for PulsarTargetFactory {
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let args = build_pulsar_args(config, AUDIT_DEFAULT_DIR, TargetType::AuditLog)?;
-        let target = rustfs_targets::target::pulsar::PulsarTarget::new(id, args)?;
-        Ok(Box::new(target))
-    }
-
-    fn validate_config(&self, _id: &str, config: &KVS) -> Result<(), TargetError> {
-        validate_pulsar_config(config, AUDIT_DEFAULT_DIR)
-    }
-
-    fn get_valid_fields(&self) -> HashSet<String> {
-        AUDIT_PULSAR_KEYS.iter().map(|s| s.to_string()).collect()
-    }
-}
-
-pub struct KafkaTargetFactory;
-
-#[async_trait]
-impl TargetFactory for KafkaTargetFactory {
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let args = build_kafka_args(config, AUDIT_DEFAULT_DIR, TargetType::AuditLog)?;
-        let target = rustfs_targets::target::kafka::KafkaTarget::new(id, args)?;
-        Ok(Box::new(target))
-    }
-
-    fn validate_config(&self, _id: &str, config: &KVS) -> Result<(), TargetError> {
-        validate_kafka_config(config, AUDIT_DEFAULT_DIR)
-    }
-
-    fn get_valid_fields(&self) -> HashSet<String> {
-        AUDIT_KAFKA_KEYS.iter().map(|s| s.to_string()).collect()
+    #[test]
+    fn builtin_plugins_create_audit_amqp_target() {
+        let plugin = builtin_target_descriptors()
+            .into_iter()
+            .find(|plugin| plugin.plugin().target_type() == ChannelTargetType::Amqp.as_str())
+            .expect("amqp plugin should exist");
+
+        let target = plugin
+            .plugin()
+            .create_target("primary".to_string(), &amqp_base_config())
+            .expect("AMQP audit target should be created");
+
+        let target_id = target.id();
+        assert_eq!(target_id.id, "primary");
+        assert_eq!(target_id.name, "amqp");
+        assert!(target.store().is_none());
     }
 }

crates/audit/src/lib.rs

@@ -23,6 +23,7 @@ pub mod error;
 pub mod factory;
 pub mod global;
 pub mod observability;
+pub mod pipeline;
 pub mod registry;
 pub mod system;
 
@@ -30,5 +31,6 @@ pub use entity::{ApiDetails, AuditEntry, ObjectVersion};
 pub use error::{AuditError, AuditResult};
 pub use global::*;
 pub use observability::{AuditMetrics, AuditMetricsReport, PerformanceValidation};
+pub use pipeline::{AuditPipeline, AuditRuntimeFacade, AuditRuntimeView};
 pub use registry::AuditRegistry;
 pub use system::{AuditSystem, AuditTargetMetricSnapshot};

crates/audit/src/pipeline.rs

@@ -0,0 +1,355 @@
+//  Copyright 2024 RustFS Team
+//
+//  Licensed under the Apache License, Version 2.0 (the "License");
+//  you may not use this file except in compliance with the License.
+//  You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+//  Unless required by applicable law or agreed to in writing, software
+//  distributed under the License is distributed on an "AS IS" BASIS,
+//  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+//  See the License for the specific language governing permissions and
+//  limitations under the License.
+
+use crate::{AuditEntry, AuditResult, observability, system::AuditTargetMetricSnapshot};
+use rustfs_targets::{
+    BuiltinPluginRuntimeAdapter, PluginRuntimeAdapter, ReplayEvent, ReplayWorkerManager, RuntimeActivation, SharedTarget, Target,
+    target::EntityTarget,
+};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::sync::{Mutex, RwLock};
+use tracing::{error, info, warn};
+
+#[derive(Clone)]
+pub struct AuditPipeline {
+    registry: Arc<Mutex<crate::AuditRegistry>>,
+}
+
+impl AuditPipeline {
+    pub fn new(registry: Arc<Mutex<crate::AuditRegistry>>) -> Self {
+        Self { registry }
+    }
+
+    pub async fn dispatch(&self, entry: Arc<AuditEntry>) -> AuditResult<()> {
+        let start_time = std::time::Instant::now();
+
+        let targets: Vec<SharedTarget<AuditEntry>> = {
+            let registry = self.registry.lock().await;
+            let targets = registry.list_target_values();
+
+            if targets.is_empty() {
+                warn!("No audit targets configured for dispatch");
+                return Ok(());
+            }
+
+            targets
+        };
+
+        let mut tasks = Vec::new();
+
+        for target in targets {
+            let entity_target = EntityTarget {
+                object_name: entry.api.name.clone().unwrap_or_default(),
+                bucket_name: entry.api.bucket.clone().unwrap_or_default(),
+                event_name: entry.event,
+                data: (*entry).clone(),
+            };
+
+            let task = async move {
+                let result = target.save(Arc::new(entity_target)).await;
+                (target.id().to_string(), result)
+            };
+
+            tasks.push(task);
+        }
+
+        let results = futures::future::join_all(tasks).await;
+
+        let mut errors = Vec::new();
+        let mut success_count = 0;
+
+        for (target_key, result) in results {
+            match result {
+                Ok(_) => {
+                    success_count += 1;
+                    observability::record_target_success();
+                }
+                Err(e) => {
+                    error!(target_id = %target_key, error = %e, "Failed to dispatch audit log to target");
+                    errors.push(e);
+                    observability::record_target_failure();
+                }
+            }
+        }
+
+        let dispatch_time = start_time.elapsed();
+
+        if errors.is_empty() {
+            observability::record_audit_success(dispatch_time);
+        } else {
+            observability::record_audit_failure(dispatch_time);
+            warn!(
+                error_count = errors.len(),
+                success_count = success_count,
+                "Some audit targets failed to receive log entry"
+            );
+        }
+
+        Ok(())
+    }
+
+    pub async fn dispatch_batch(&self, entries: Vec<Arc<AuditEntry>>) -> AuditResult<()> {
+        let start_time = std::time::Instant::now();
+
+        let targets: Vec<SharedTarget<AuditEntry>> = {
+            let registry = self.registry.lock().await;
+            let targets = registry.list_target_values();
+
+            if targets.is_empty() {
+                warn!("No audit targets configured for batch dispatch");
+                return Ok(());
+            }
+
+            targets
+        };
+
+        let mut tasks = Vec::new();
+        for target in targets {
+            let entries_clone: Vec<_> = entries.iter().map(Arc::clone).collect();
+
+            let task = async move {
+                let mut success_count = 0;
+                let mut errors = Vec::new();
+                for entry in entries_clone {
+                    let entity_target = EntityTarget {
+                        object_name: entry.api.name.clone().unwrap_or_default(),
+                        bucket_name: entry.api.bucket.clone().unwrap_or_default(),
+                        event_name: entry.event,
+                        data: (*entry).clone(),
+                    };
+                    match target.save(Arc::new(entity_target)).await {
+                        Ok(_) => success_count += 1,
+                        Err(e) => errors.push(e),
+                    }
+                }
+                (target.id().to_string(), success_count, errors)
+            };
+            tasks.push(task);
+        }
+
+        let results = futures::future::join_all(tasks).await;
+        let mut total_success = 0;
+        let mut total_errors = 0;
+        for (_target_id, success_count, errors) in results {
+            total_success += success_count;
+            total_errors += errors.len();
+            for e in errors {
+                error!("Batch dispatch error: {:?}", e);
+            }
+        }
+
+        let dispatch_time = start_time.elapsed();
+        info!(
+            "Batch dispatched {} entries, success: {}, errors: {}, time: {:?}",
+            entries.len(),
+            total_success,
+            total_errors,
+            dispatch_time
+        );
+
+        Ok(())
+    }
+
+    pub async fn snapshot_target_metrics(&self) -> Vec<AuditTargetMetricSnapshot> {
+        let registry = self.registry.lock().await;
+        registry
+            .list_target_values()
+            .into_iter()
+            .map(|target| {
+                let delivery = target.delivery_snapshot();
+                AuditTargetMetricSnapshot {
+                    failed_messages: delivery.failed_messages,
+                    queue_length: delivery.queue_length,
+                    target_id: target.id().to_string(),
+                    total_messages: delivery.total_messages,
+                }
+            })
+            .collect()
+    }
+
+    pub async fn snapshot_target_health(&self) -> Vec<rustfs_targets::RuntimeTargetHealthSnapshot> {
+        let registry = self.registry.lock().await;
+        registry.runtime_manager().health_snapshots().await
+    }
+}
+
+#[derive(Clone)]
+pub struct AuditRuntimeView {
+    registry: Arc<Mutex<crate::AuditRegistry>>,
+}
+
+impl AuditRuntimeView {
+    pub fn new(registry: Arc<Mutex<crate::AuditRegistry>>) -> Self {
+        Self { registry }
+    }
+
+    pub async fn list_targets(&self) -> Vec<String> {
+        let registry = self.registry.lock().await;
+        registry.list_targets()
+    }
+
+    pub async fn get_target_values(&self) -> Vec<rustfs_targets::SharedTarget<AuditEntry>> {
+        let registry = self.registry.lock().await;
+        registry.list_target_values()
+    }
+
+    pub async fn get_target(&self, target_id: &str) -> Option<String> {
+        let registry = self.registry.lock().await;
+        registry.get_target(target_id).map(|target| target.id().to_string())
+    }
+
+    pub async fn enable_target(&self, target_id: &str) -> AuditResult<()> {
+        let registry = self.registry.lock().await;
+        if registry.get_target(target_id).is_some() {
+            info!(target_id = %target_id, "Target enabled");
+            Ok(())
+        } else {
+            Err(crate::AuditError::Configuration(format!("Target not found: {target_id}"), None))
+        }
+    }
+
+    pub async fn disable_target(&self, target_id: &str) -> AuditResult<()> {
+        let registry = self.registry.lock().await;
+        if registry.get_target(target_id).is_some() {
+            info!(target_id = %target_id, "Target disabled");
+            Ok(())
+        } else {
+            Err(crate::AuditError::Configuration(format!("Target not found: {target_id}"), None))
+        }
+    }
+
+    pub async fn remove_target(&self, target_id: &str) -> AuditResult<()> {
+        let mut registry = self.registry.lock().await;
+        if registry.remove_target(target_id).await.is_some() {
+            info!(target_id = %target_id, "Target removed");
+            Ok(())
+        } else {
+            Err(crate::AuditError::Configuration(format!("Target not found: {target_id}"), None))
+        }
+    }
+
+    pub async fn upsert_target(&self, target_id: String, target: Box<dyn Target<AuditEntry> + Send + Sync>) -> AuditResult<()> {
+        if let Err(err) = target.init().await {
+            return Err(crate::AuditError::Target(err));
+        }
+
+        let shared_target: SharedTarget<AuditEntry> = Arc::from(target);
+        let mut registry = self.registry.lock().await;
+        let _ = registry.remove_target(&target_id).await;
+        registry.add_shared_target(target_id.clone(), shared_target);
+        info!(target_id = %target_id, "Target upserted");
+        Ok(())
+    }
+}
+
+#[derive(Clone)]
+pub struct AuditRuntimeFacade {
+    registry: Arc<Mutex<crate::AuditRegistry>>,
+    replay_workers: Arc<RwLock<ReplayWorkerManager>>,
+    runtime_adapter: Arc<dyn PluginRuntimeAdapter<AuditEntry>>,
+}
+
+impl AuditRuntimeFacade {
+    pub fn new(registry: Arc<Mutex<crate::AuditRegistry>>, replay_workers: Arc<RwLock<ReplayWorkerManager>>) -> Self {
+        let runtime_adapter = BuiltinPluginRuntimeAdapter::new(
+            Arc::new(move |event: ReplayEvent<AuditEntry>| {
+                Box::pin(async move {
+                    match event {
+                        ReplayEvent::Delivered { key, target } => {
+                            info!("Successfully sent audit entry, target: {}, key: {}", target.id(), key.to_string());
+                            observability::record_target_success();
+                        }
+                        ReplayEvent::RetryableError { error, target, .. } => match error {
+                            rustfs_targets::TargetError::NotConnected => {
+                                warn!("Target {} not connected, retrying...", target.id());
+                            }
+                            rustfs_targets::TargetError::Timeout(_) => {
+                                warn!("Timeout sending to target {}, retrying...", target.id());
+                            }
+                            _ => {}
+                        },
+                        ReplayEvent::Dropped { reason, target, .. } => {
+                            warn!("Dropped queued payload for target {}: {}", target.id(), reason);
+                            observability::record_target_failure();
+                        }
+                        ReplayEvent::PermanentFailure { error, target, .. } => {
+                            error!("Permanent error for target {}: {}", target.id(), error);
+                            target.record_final_failure();
+                            observability::record_target_failure();
+                        }
+                        ReplayEvent::RetryExhausted { key, target } => {
+                            warn!("Max retries exceeded for key {}, target: {}, skipping", key.to_string(), target.id());
+                            target.record_final_failure();
+                            observability::record_target_failure();
+                        }
+                        ReplayEvent::UnreadableEntry { key, error, target } => {
+                            warn!("Skipping unreadable audit store entry {} for target {}: {}", key, target.id(), error);
+                        }
+                    }
+                })
+            }),
+            Arc::new(|target_id, has_replay| {
+                if has_replay {
+                    info!(target_id = %target_id, "Audit stream processing started");
+                } else {
+                    info!(target_id = %target_id, "No store configured, skip audit stream processing");
+                }
+            }),
+            None,
+            Duration::from_millis(500),
+            Duration::from_millis(500),
+            "Stopping audit stream",
+        );
+
+        Self {
+            registry,
+            replay_workers,
+            runtime_adapter: Arc::new(runtime_adapter),
+        }
+    }
+
+    pub async fn replace_targets(&self, activation: RuntimeActivation<AuditEntry>) -> AuditResult<()> {
+        let mut registry = self.registry.lock().await;
+        let mut replay_workers = self.replay_workers.write().await;
+        self.runtime_adapter
+            .replace_runtime_targets(registry.runtime_manager_mut(), &mut replay_workers, activation)
+            .await
+            .map_err(crate::AuditError::Target)?;
+        Ok(())
+    }
+
+    pub async fn shutdown_runtime(
+        &self,
+        registry: &mut crate::AuditRegistry,
+        replay_workers: &mut ReplayWorkerManager,
+    ) -> AuditResult<()> {
+        self.runtime_adapter
+            .shutdown(registry.runtime_manager_mut(), replay_workers)
+            .await
+            .map_err(crate::AuditError::Target)
+    }
+
+    pub async fn activate_targets_with_replay(
+        &self,
+        targets: Vec<Box<dyn Target<AuditEntry> + Send + Sync>>,
+    ) -> RuntimeActivation<AuditEntry> {
+        self.runtime_adapter.activate_with_replay(targets).await
+    }
+
+    pub async fn stop_replay_workers(&self) {
+        let mut replay_workers = self.replay_workers.write().await;
+        self.runtime_adapter.stop_replay_workers(&mut replay_workers).await;
+    }
+}

crates/audit/src/registry.rs

@@ -12,28 +12,19 @@
 //  See the License for the specific language governing permissions and
 //  limitations under the License.
 
-use crate::{
-    AuditEntry, AuditError, AuditResult,
-    factory::{
-        KafkaTargetFactory, MQTTTargetFactory, NATSTargetFactory, PulsarTargetFactory, TargetFactory, WebhookTargetFactory,
-    },
-};
-use futures::StreamExt;
-use futures::stream::FuturesUnordered;
-use hashbrown::HashMap;
+use crate::{AuditEntry, AuditError, AuditResult, factory::builtin_target_plugins};
 use rustfs_config::audit::AUDIT_ROUTE_PREFIX;
 use rustfs_ecstore::config::{Config, KVS};
 use rustfs_targets::arn::TargetID;
-use rustfs_targets::{Target, TargetError, config::collect_target_configs, target::ChannelTargetType};
-use std::sync::Arc;
-use tracing::{error, info};
+use rustfs_targets::{SharedTarget, Target, TargetError, TargetPluginRegistry, TargetRuntimeManager};
+use tracing::info;
 
 /// Registry for managing audit targets
 pub struct AuditRegistry {
     /// Storage for created targets
-    targets: HashMap<String, Box<dyn Target<AuditEntry> + Send + Sync>>,
-    /// Factories for creating targets
-    factories: HashMap<String, Box<dyn TargetFactory>>,
+    targets: TargetRuntimeManager<AuditEntry>,
+    /// Registered plugins for creating targets
+    plugins: TargetPluginRegistry<AuditEntry>,
 }
 
 impl Default for AuditRegistry {
@@ -45,28 +36,17 @@ impl Default for AuditRegistry {
 impl AuditRegistry {
     /// Creates a new AuditRegistry
     pub fn new() -> Self {
-        let mut registry = AuditRegistry {
-            factories: HashMap::new(),
-            targets: HashMap::new(),
-        };
+        let mut plugins = TargetPluginRegistry::new();
+        plugins.register_all(builtin_target_plugins());
 
-        // Register built-in factories
-        registry.register(ChannelTargetType::Webhook.as_str(), Box::new(WebhookTargetFactory));
-        registry.register(ChannelTargetType::Mqtt.as_str(), Box::new(MQTTTargetFactory));
-        registry.register(ChannelTargetType::Nats.as_str(), Box::new(NATSTargetFactory));
-        registry.register(ChannelTargetType::Pulsar.as_str(), Box::new(PulsarTargetFactory));
-        registry.register(ChannelTargetType::Kafka.as_str(), Box::new(KafkaTargetFactory));
-
-        registry
+        AuditRegistry {
+            targets: TargetRuntimeManager::new(),
+            plugins,
+        }
     }
 
-    /// Registers a new factory for a target type
-    ///
-    /// # Arguments
-    /// * `target_type` - The type of the target (e.g., "webhook", "mqtt").
-    /// * `factory` - The factory instance to create targets of this type.
-    pub fn register(&mut self, target_type: &str, factory: Box<dyn TargetFactory>) {
-        self.factories.insert(target_type.to_string(), factory);
+    pub fn supports_target_type(&self, target_type: &str) -> bool {
+        self.plugins.supports_target_type(target_type)
     }
 
     /// Creates a target of the specified type with the given ID and configuration
@@ -84,16 +64,7 @@ impl AuditRegistry {
         id: String,
         config: &KVS,
     ) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let factory = self
-            .factories
-            .get(target_type)
-            .ok_or_else(|| TargetError::Configuration(format!("Unknown target type: {target_type}")))?;
-
-        // Validate configuration before creating target
-        factory.validate_config(&id, config)?;
-
-        // Create target
-        factory.create_target(id, config).await
+        self.plugins.create_target(target_type, id, config)
     }
 
     /// Creates all targets from a configuration
@@ -109,37 +80,10 @@ impl AuditRegistry {
         &self,
         config: &Config,
     ) -> AuditResult<Vec<Box<dyn Target<AuditEntry> + Send + Sync>>> {
-        let mut tasks = FuturesUnordered::new();
-        for (target_type, factory) in &self.factories {
-            tracing::Span::current().record("target_type", target_type.as_str());
-            info!("Start working on target types...");
-            let valid_fields = factory.get_valid_fields();
-            for (id, merged_config) in collect_target_configs(config, AUDIT_ROUTE_PREFIX, target_type, &valid_fields) {
-                info!(instance_id = %id, "Target is enabled, ready to create a task");
-                let tid = id.clone();
-                let merged_config_arc = Arc::new(merged_config);
-                tasks.push(async move {
-                    let result = factory.create_target(tid.clone(), &merged_config_arc).await;
-                    (tid, result)
-                });
-            }
-        }
-
-        let mut successful_targets = Vec::new();
-        while let Some((id, result)) = tasks.next().await {
-            match result {
-                Ok(target) => {
-                    info!(target_type = %target.id().name, instance_id = %id, "Create a target successfully");
-                    successful_targets.push(target);
-                }
-                Err(e) => {
-                    error!(instance_id = %id, error = %e, "Failed to create a target");
-                }
-            }
-        }
-
-        info!(count = successful_targets.len(), "All target processing completed");
-        Ok(successful_targets)
+        self.plugins
+            .create_targets_from_config(config, AUDIT_ROUTE_PREFIX)
+            .await
+            .map_err(AuditError::from)
     }
 
     /// Adds a target to the registry
@@ -147,8 +91,14 @@ impl AuditRegistry {
     /// # Arguments
     /// * `id` - The identifier for the target.
     /// * `target` - The target instance to be added.
-    pub fn add_target(&mut self, id: String, target: Box<dyn Target<AuditEntry> + Send + Sync>) {
-        self.targets.insert(id, target);
+    pub fn add_target(&mut self, _id: String, target: Box<dyn Target<AuditEntry> + Send + Sync>) {
+        debug_assert_eq!(_id, target.id().to_string());
+        self.targets.add_boxed(target);
+    }
+
+    pub fn add_shared_target(&mut self, _id: String, target: SharedTarget<AuditEntry>) {
+        debug_assert_eq!(_id, target.id().to_string());
+        self.targets.add_arc(target);
     }
 
     /// Removes a target from the registry
@@ -158,8 +108,8 @@ impl AuditRegistry {
     ///
     /// # Returns
     /// * `Option<Box<dyn Target<AuditEntry> + Send + Sync>>` - The removed target if it existed.
-    pub fn remove_target(&mut self, id: &str) -> Option<Box<dyn Target<AuditEntry> + Send + Sync>> {
-        self.targets.remove(id)
+    pub async fn remove_target(&mut self, id: &str) -> Option<rustfs_targets::SharedTarget<AuditEntry>> {
+        self.targets.remove_and_close(id).await
     }
 
     /// Gets a target from the registry
@@ -169,13 +119,21 @@ impl AuditRegistry {
     ///
     /// # Returns
     /// * `Option<&(dyn Target<AuditEntry> + Send + Sync)>` - The target if it exists.
-    pub fn get_target(&self, id: &str) -> Option<&(dyn Target<AuditEntry> + Send + Sync)> {
-        self.targets.get(id).map(|t| t.as_ref())
+    pub fn get_target(&self, id: &str) -> Option<rustfs_targets::SharedTarget<AuditEntry>> {
+        self.targets.get(id)
     }
 
     /// Lists cloned target values for runtime inspection without exposing mutable registry access.
-    pub fn list_target_values(&self) -> Vec<Box<dyn Target<AuditEntry> + Send + Sync>> {
-        self.targets.values().map(|target| target.clone_dyn()).collect()
+    pub fn list_target_values(&self) -> Vec<rustfs_targets::SharedTarget<AuditEntry>> {
+        self.targets.values()
+    }
+
+    pub fn runtime_manager(&self) -> &TargetRuntimeManager<AuditEntry> {
+        &self.targets
+    }
+
+    pub fn runtime_manager_mut(&mut self) -> &mut TargetRuntimeManager<AuditEntry> {
+        &mut self.targets
     }
 
     /// Lists all target IDs
@@ -183,7 +141,7 @@ impl AuditRegistry {
     /// # Returns
     /// * `Vec<String>` - A vector of all target IDs in the registry.
     pub fn list_targets(&self) -> Vec<String> {
-        self.targets.keys().cloned().collect()
+        self.targets.keys()
     }
 
     /// Closes all targets and clears the registry
@@ -191,20 +149,23 @@ impl AuditRegistry {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure.
     pub async fn close_all(&mut self) -> AuditResult<()> {
-        let mut errors = Vec::new();
+        let mut first_error = None;
 
-        for (id, target) in self.targets.drain() {
-            if let Err(e) = target.close().await {
-                error!(target_id = %id, error = %e, "Failed to close audit target");
-                errors.push(e);
+        for target_id in self.targets.keys() {
+            if let Some(target) = self.targets.remove(&target_id)
+                && let Err(err) = target.close().await
+            {
+                tracing::error!(target_id = %target_id, error = %err, "Failed to close target during shutdown");
+                if first_error.is_none() {
+                    first_error = Some(err);
+                }
             }
         }
 
-        if let Some(error) = errors.into_iter().next() {
-            return Err(AuditError::Target(error));
+        match first_error {
+            Some(err) => Err(AuditError::Target(err)),
+            None => Ok(()),
         }
-
-        Ok(())
     }
 
     /// Creates a unique key for a target based on its type and ID
@@ -279,7 +240,107 @@ impl AuditRegistry {
         target: Box<dyn Target<AuditEntry> + Send + Sync>,
     ) -> AuditResult<()> {
         let key = self.create_key(target_type, target_id);
-        self.targets.insert(key, target);
+        debug_assert_eq!(key, target.id().to_string());
+        self.targets.add_boxed(target);
         Ok(())
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::AuditRegistry;
+    use crate::{AuditEntry, AuditError};
+    use rustfs_targets::arn::TargetID;
+    use rustfs_targets::store::{Key, Store};
+    use rustfs_targets::target::{ChannelTargetType, EntityTarget, QueuedPayload, QueuedPayloadMeta};
+    use rustfs_targets::{StoreError, Target, TargetError};
+    use std::sync::Arc;
+    use std::sync::atomic::{AtomicUsize, Ordering};
+
+    #[derive(Clone)]
+    struct CloseTestTarget {
+        id: TargetID,
+        close_calls: Arc<AtomicUsize>,
+        fail_on_close: bool,
+    }
+
+    impl CloseTestTarget {
+        fn new(id: TargetID, close_calls: Arc<AtomicUsize>, fail_on_close: bool) -> Self {
+            Self {
+                id,
+                close_calls,
+                fail_on_close,
+            }
+        }
+    }
+
+    #[async_trait::async_trait]
+    impl Target<AuditEntry> for CloseTestTarget {
+        fn id(&self) -> TargetID {
+            self.id.clone()
+        }
+
+        async fn is_active(&self) -> Result<bool, TargetError> {
+            Ok(true)
+        }
+
+        async fn save(&self, _event: Arc<EntityTarget<AuditEntry>>) -> Result<(), TargetError> {
+            Ok(())
+        }
+
+        async fn send_raw_from_store(&self, _key: Key, _body: Vec<u8>, _meta: QueuedPayloadMeta) -> Result<(), TargetError> {
+            Ok(())
+        }
+
+        async fn close(&self) -> Result<(), TargetError> {
+            self.close_calls.fetch_add(1, Ordering::SeqCst);
+            if self.fail_on_close {
+                Err(TargetError::Unknown("close failed".to_string()))
+            } else {
+                Ok(())
+            }
+        }
+
+        fn store(&self) -> Option<&(dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send + Sync)> {
+            None
+        }
+
+        fn clone_dyn(&self) -> Box<dyn Target<AuditEntry> + Send + Sync> {
+            Box::new(self.clone())
+        }
+
+        fn is_enabled(&self) -> bool {
+            true
+        }
+    }
+
+    #[test]
+    fn registry_registers_amqp_factory() {
+        let registry = AuditRegistry::new();
+
+        assert!(registry.supports_target_type(ChannelTargetType::Amqp.as_str()));
+    }
+
+    #[tokio::test]
+    async fn close_all_returns_first_error_and_clears_targets() {
+        let mut registry = AuditRegistry::new();
+        let ok_calls = Arc::new(AtomicUsize::new(0));
+        let fail_calls = Arc::new(AtomicUsize::new(0));
+
+        let ok_id = TargetID::new("ok".to_string(), "webhook".to_string());
+        let fail_id = TargetID::new("fail".to_string(), "webhook".to_string());
+
+        registry.add_target(ok_id.to_string(), Box::new(CloseTestTarget::new(ok_id, Arc::clone(&ok_calls), false)));
+        registry.add_target(
+            fail_id.to_string(),
+            Box::new(CloseTestTarget::new(fail_id, Arc::clone(&fail_calls), true)),
+        );
+
+        let result = registry.close_all().await;
+
+        assert!(matches!(result, Err(AuditError::Target(TargetError::Unknown(_)))));
+        assert_eq!(ok_calls.load(Ordering::SeqCst), 1);
+        assert_eq!(fail_calls.load(Ordering::SeqCst), 1);
+        assert!(registry.list_targets().is_empty());
+    }
+}

crates/audit/src/system.rs

@@ -12,16 +12,14 @@
 //  See the License for the specific language governing permissions and
 //  limitations under the License.
 
-use crate::{AuditEntry, AuditError, AuditRegistry, AuditResult, observability};
-use hashbrown::HashMap;
-use rustfs_ecstore::config::Config;
-use rustfs_targets::{
-    StoreError, Target, TargetError,
-    store::{Key, Store},
-    target::{EntityTarget, QueuedPayload},
+use crate::{
+    AuditEntry, AuditError, AuditRegistry, AuditResult, observability,
+    pipeline::{AuditPipeline, AuditRuntimeFacade, AuditRuntimeView},
 };
+use rustfs_ecstore::config::Config;
+use rustfs_targets::{ReplayWorkerManager, Target};
 use std::sync::Arc;
-use tokio::sync::{Mutex, RwLock, mpsc};
+use tokio::sync::{Mutex, RwLock};
 use tracing::{error, info, warn};
 
 #[derive(Debug, Clone, Default, PartialEq, Eq)]
@@ -49,7 +47,7 @@ pub struct AuditSystem {
     state: Arc<RwLock<AuditSystemState>>,
     config: Arc<RwLock<Option<Config>>>,
     /// Cancellation senders for active audit stream tasks (target_id -> cancel tx)
-    stream_cancellers: Arc<RwLock<HashMap<String, mpsc::Sender<()>>>>,
+    stream_cancellers: Arc<RwLock<ReplayWorkerManager>>,
 }
 
 impl Default for AuditSystem {
@@ -59,16 +57,68 @@ impl Default for AuditSystem {
 }
 
 impl AuditSystem {
+    fn pipeline(&self) -> AuditPipeline {
+        AuditPipeline::new(self.registry.clone())
+    }
+
+    fn runtime_view(&self) -> AuditRuntimeView {
+        AuditRuntimeView::new(self.registry.clone())
+    }
+
+    fn runtime_facade(&self) -> AuditRuntimeFacade {
+        AuditRuntimeFacade::new(self.registry.clone(), self.stream_cancellers.clone())
+    }
+
     /// Creates a new audit system
     pub fn new() -> Self {
         Self {
             registry: Arc::new(Mutex::new(AuditRegistry::new())),
             state: Arc::new(RwLock::new(AuditSystemState::Stopped)),
             config: Arc::new(RwLock::new(None)),
-            stream_cancellers: Arc::new(RwLock::new(HashMap::new())),
+            stream_cancellers: Arc::new(RwLock::new(ReplayWorkerManager::new())),
         }
     }
 
+    async fn create_targets_from_config(&self, config: &Config) -> AuditResult<Vec<Box<dyn Target<AuditEntry> + Send + Sync>>> {
+        let registry = self.registry.lock().await;
+        registry.create_audit_targets_from_config(config).await
+    }
+
+    async fn clear_runtime_targets(&self) -> AuditResult<()> {
+        {
+            let mut registry = self.registry.lock().await;
+            let mut replay_workers = self.stream_cancellers.write().await;
+            self.runtime_facade()
+                .shutdown_runtime(&mut registry, &mut replay_workers)
+                .await?;
+        }
+
+        let mut state = self.state.write().await;
+        *state = AuditSystemState::Stopped;
+        Ok(())
+    }
+
+    async fn commit_runtime_targets(
+        &self,
+        targets: Vec<Box<dyn Target<AuditEntry> + Send + Sync>>,
+        final_state: AuditSystemState,
+    ) -> AuditResult<()> {
+        if targets.is_empty() {
+            info!("No enabled audit targets found, keeping audit system stopped");
+            self.clear_runtime_targets().await?;
+            return Ok(());
+        }
+
+        info!(target_count = targets.len(), "Created audit targets successfully");
+
+        let activation = self.runtime_facade().activate_targets_with_replay(targets).await;
+        self.runtime_facade().replace_targets(activation).await?;
+
+        let mut state = self.state.write().await;
+        *state = final_state;
+        Ok(())
+    }
+
     /// Starts the audit system with the given configuration
     ///
     /// # Arguments
@@ -103,31 +153,14 @@ impl AuditSystem {
             *config_guard = Some(config.clone());
         }
 
-        // Create targets from configuration
-        let mut registry = self.registry.lock().await;
-        match registry.create_audit_targets_from_config(&config).await {
+        match self.create_targets_from_config(&config).await {
             Ok(targets) => {
-                if targets.is_empty() {
-                    info!("No enabled audit targets found, keeping audit system stopped");
-                    drop(registry);
-                    return Ok(());
-                }
-
                 {
                     let mut state = self.state.write().await;
                     *state = AuditSystemState::Starting;
                 }
 
-                info!(target_count = targets.len(), "Created audit targets successfully");
-
-                // Initialize all targets
-                for target in targets {
-                    self.init_and_register_target(target, &mut registry).await;
-                }
-
-                // Update state to running
-                let mut state = self.state.write().await;
-                *state = AuditSystemState::Running;
+                self.commit_runtime_targets(targets, AuditSystemState::Running).await?;
                 info!("Audit system started successfully");
                 Ok(())
             }
@@ -207,18 +240,10 @@ impl AuditSystem {
         info!("Stopping audit system");
 
         // Stop all stream tasks first
-        self.stop_all_streams().await;
-
-        // Close all targets
-        let mut registry = self.registry.lock().await;
-        if let Err(e) = registry.close_all().await {
+        if let Err(e) = self.clear_runtime_targets().await {
             error!(error = %e, "Failed to close some audit targets");
         }
 
-        // Update state to stopped
-        let mut state = self.state.write().await;
-        *state = AuditSystemState::Stopped;
-
         // Clear configuration
         let mut config_guard = self.config.write().await;
         *config_guard = None;
@@ -248,8 +273,6 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn dispatch(&self, entry: Arc<AuditEntry>) -> AuditResult<()> {
-        let start_time = std::time::Instant::now();
-
         let state = self.state.read().await;
 
         match *state {
@@ -262,77 +285,7 @@ impl AuditSystem {
             }
         }
         drop(state);
-
-        // Collect cloned targets under lock, then dispatch without holding it
-        let targets: Vec<(String, Box<dyn Target<AuditEntry> + Send + Sync>)> = {
-            let registry = self.registry.lock().await;
-            let target_keys = registry.list_targets();
-
-            if target_keys.is_empty() {
-                warn!("No audit targets configured for dispatch");
-                return Ok(());
-            }
-
-            target_keys
-                .into_iter()
-                .filter_map(|key| registry.get_target(&key).map(|t| (key, t.clone_dyn())))
-                .collect()
-        };
-
-        // Dispatch to all targets concurrently (no lock held)
-        let mut tasks = Vec::new();
-
-        for (target_key, target) in targets {
-            let entity_target = EntityTarget {
-                object_name: entry.api.name.clone().unwrap_or_default(),
-                bucket_name: entry.api.bucket.clone().unwrap_or_default(),
-                event_name: entry.event,
-                data: (*entry).clone(),
-            };
-
-            let task = async move {
-                let result = target.save(Arc::new(entity_target)).await;
-                (target_key, result)
-            };
-
-            tasks.push(task);
-        }
-
-        // Execute all dispatch tasks
-        let results = futures::future::join_all(tasks).await;
-
-        let mut errors = Vec::new();
-        let mut success_count = 0;
-
-        for (target_key, result) in results {
-            match result {
-                Ok(_) => {
-                    success_count += 1;
-                    observability::record_target_success();
-                }
-                Err(e) => {
-                    error!(target_id = %target_key, error = %e, "Failed to dispatch audit log to target");
-                    errors.push(e);
-                    observability::record_target_failure();
-                }
-            }
-        }
-
-        let dispatch_time = start_time.elapsed();
-
-        if errors.is_empty() {
-            observability::record_audit_success(dispatch_time);
-        } else {
-            observability::record_audit_failure(dispatch_time);
-            // Log errors but don't fail the entire dispatch
-            warn!(
-                error_count = errors.len(),
-                success_count = success_count,
-                "Some audit targets failed to receive log entry"
-            );
-        }
-
-        Ok(())
+        self.pipeline().dispatch(entry).await
     }
 
     /// Dispatches a batch of audit log entries to all active targets
@@ -343,239 +296,12 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn dispatch_batch(&self, entries: Vec<Arc<AuditEntry>>) -> AuditResult<()> {
-        let start_time = std::time::Instant::now();
-
         let state = self.state.read().await;
         if *state != AuditSystemState::Running {
             return Err(AuditError::NotInitialized("Audit system is not running".to_string()));
         }
         drop(state);
-
-        // Collect targets under lock, then dispatch without holding it
-        let targets: Vec<(String, Box<dyn Target<AuditEntry> + Send + Sync>)> = {
-            let registry = self.registry.lock().await;
-            let target_keys = registry.list_targets();
-
-            if target_keys.is_empty() {
-                warn!("No audit targets configured for batch dispatch");
-                return Ok(());
-            }
-
-            target_keys
-                .into_iter()
-                .filter_map(|key| registry.get_target(&key).map(|t| (key, t.clone_dyn())))
-                .collect()
-        };
-
-        let mut tasks = Vec::new();
-        for (target_key, target) in targets {
-            let entries_clone: Vec<_> = entries.iter().map(Arc::clone).collect();
-            let target_key_clone = target_key.clone();
-
-            let task = async move {
-                let mut success_count = 0;
-                let mut errors = Vec::new();
-                for entry in entries_clone {
-                    let entity_target = EntityTarget {
-                        object_name: entry.api.name.clone().unwrap_or_default(),
-                        bucket_name: entry.api.bucket.clone().unwrap_or_default(),
-                        event_name: entry.event,
-                        data: (*entry).clone(),
-                    };
-                    match target.save(Arc::new(entity_target)).await {
-                        Ok(_) => success_count += 1,
-                        Err(e) => errors.push(e),
-                    }
-                }
-                (target_key_clone, success_count, errors)
-            };
-            tasks.push(task);
-        }
-
-        let results = futures::future::join_all(tasks).await;
-        let mut total_success = 0;
-        let mut total_errors = 0;
-        for (_target_id, success_count, errors) in results {
-            total_success += success_count;
-            total_errors += errors.len();
-            for e in errors {
-                error!("Batch dispatch error: {:?}", e);
-            }
-        }
-
-        let dispatch_time = start_time.elapsed();
-        info!(
-            "Batch dispatched {} entries, success: {}, errors: {}, time: {:?}",
-            entries.len(),
-            total_success,
-            total_errors,
-            dispatch_time
-        );
-
-        Ok(())
-    }
-
-    /// Stops all active audit stream tasks by sending cancellation signals.
-    async fn stop_all_streams(&self) {
-        let mut cancellers = self.stream_cancellers.write().await;
-        for (target_id, cancel_tx) in cancellers.drain() {
-            info!(target_id = %target_id, "Stopping audit stream");
-            let _ = cancel_tx.send(()).await;
-        }
-    }
-
-    /// Initializes a single target: runs init(), starts stream if store is present,
-    /// and adds it to the registry. For store-backed targets, registration and stream
-    /// startup proceed even if init() fails so queued entries can be drained later.
-    async fn init_and_register_target(
-        &self,
-        target: Box<dyn Target<AuditEntry> + Send + Sync>,
-        registry: &mut AuditRegistry,
-    ) -> Option<String> {
-        let target_id = target.id().to_string();
-        let has_store = target.store().is_some();
-
-        if let Err(e) = target.init().await {
-            error!(target_id = %target_id, error = %e, "Failed to initialize audit target");
-            // Non-store targets: init failure is fatal.
-            if !has_store {
-                return None;
-            }
-            // Store-backed targets: still register and start the stream so queued
-            // entries can be drained when connectivity recovers.
-            warn!(
-                target_id = %target_id,
-                "Proceeding with store-backed audit target despite init failure"
-            );
-        }
-
-        if target.is_enabled() {
-            if let Some(store) = target.store() {
-                info!(target_id = %target_id, "Start audit stream processing for target");
-                let store_clone: Box<dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send> = store.boxed_clone();
-                let target_arc: Arc<dyn Target<AuditEntry> + Send + Sync> = Arc::from(target.clone_dyn());
-                let cancel_tx = self.start_audit_stream_with_batching(store_clone, target_arc);
-
-                self.stream_cancellers.write().await.insert(target_id.clone(), cancel_tx);
-                info!(target_id = %target_id, "Audit stream processing started");
-            } else {
-                info!(target_id = %target_id, "No store configured, skip audit stream processing");
-            }
-        } else {
-            info!(target_id = %target_id, "Target disabled, skip audit stream processing");
-        }
-
-        registry.add_target(target_id.clone(), target);
-        Some(target_id)
-    }
-
-    /// Starts the audit stream processing for a target with batching and retry logic
-    ///
-    /// # Arguments
-    /// * `store` - The store from which to read audit entries
-    /// * `target` - The target to which audit entries will be sent
-    ///
-    /// This function spawns a background task that continuously reads audit entries from the provided store
-    /// and attempts to send them to the specified target. It implements retry logic with exponential backoff
-    fn start_audit_stream_with_batching(
-        &self,
-        store: Box<dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send>,
-        target: Arc<dyn Target<AuditEntry> + Send + Sync>,
-    ) -> mpsc::Sender<()> {
-        let (cancel_tx, mut cancel_rx) = mpsc::channel(1);
-        let state = self.state.clone();
-
-        tokio::spawn(async move {
-            use std::time::Duration;
-            use tokio::time::sleep;
-
-            info!("Starting audit stream for target: {}", target.id());
-
-            const MAX_RETRIES: usize = 5;
-            const BASE_RETRY_DELAY: Duration = Duration::from_secs(2);
-
-            loop {
-                // Check for cancellation signal
-                if cancel_rx.try_recv().is_ok() {
-                    info!("Audit stream cancelled for target: {}", target.id());
-                    break;
-                }
-
-                match *state.read().await {
-                    AuditSystemState::Running | AuditSystemState::Paused | AuditSystemState::Starting => {}
-                    _ => {
-                        info!("Audit stream stopped for target: {}", target.id());
-                        break;
-                    }
-                }
-
-                let keys: Vec<Key> = store.list();
-                if keys.is_empty() {
-                    tokio::select! {
-                        _ = sleep(Duration::from_millis(500)) => {},
-                        _ = cancel_rx.recv() => {
-                            info!("Audit stream cancelled during idle for target: {}", target.id());
-                            return;
-                        }
-                    }
-                    continue;
-                }
-
-                for key in keys {
-                    if cancel_rx.try_recv().is_ok() {
-                        info!("Audit stream cancelled during processing for target: {}", target.id());
-                        return;
-                    }
-
-                    let mut retries = 0usize;
-                    let mut success = false;
-
-                    while retries < MAX_RETRIES && !success {
-                        match target.send_from_store(key.clone()).await {
-                            Ok(_) => {
-                                info!("Successfully sent audit entry, target: {}, key: {}", target.id(), key.to_string());
-                                observability::record_target_success();
-                                success = true;
-                            }
-                            Err(e) => {
-                                match &e {
-                                    TargetError::NotConnected => {
-                                        warn!("Target {} not connected, retrying...", target.id());
-                                    }
-                                    TargetError::Timeout(_) => {
-                                        warn!("Timeout sending to target {}, retrying...", target.id());
-                                    }
-                                    TargetError::Dropped(reason) => {
-                                        warn!("Dropped queued payload for target {}: {}", target.id(), reason);
-                                        observability::record_target_failure();
-                                        break;
-                                    }
-                                    _ => {
-                                        error!("Permanent error for target {}: {}", target.id(), e);
-                                        target.record_final_failure();
-                                        observability::record_target_failure();
-                                        break;
-                                    }
-                                }
-                                retries += 1;
-                                let backoff = BASE_RETRY_DELAY * (1 << retries);
-                                sleep(backoff).await;
-                            }
-                        }
-                    }
-
-                    if retries >= MAX_RETRIES && !success {
-                        warn!("Max retries exceeded for key {}, target: {}, skipping", key.to_string(), target.id());
-                        target.record_final_failure();
-                        observability::record_target_failure();
-                    }
-                }
-
-                sleep(Duration::from_millis(100)).await;
-            }
-        });
-
-        cancel_tx
+        self.pipeline().dispatch_batch(entries).await
     }
 
     /// Enables a specific target
@@ -586,15 +312,7 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn enable_target(&self, target_id: &str) -> AuditResult<()> {
-        // This would require storing enabled/disabled state per target
-        // For now, just check if target exists
-        let registry = self.registry.lock().await;
-        if registry.get_target(target_id).is_some() {
-            info!(target_id = %target_id, "Target enabled");
-            Ok(())
-        } else {
-            Err(AuditError::Configuration(format!("Target not found: {target_id}"), None))
-        }
+        self.runtime_view().enable_target(target_id).await
     }
 
     /// Disables a specific target
@@ -605,15 +323,7 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn disable_target(&self, target_id: &str) -> AuditResult<()> {
-        // This would require storing enabled/disabled state per target
-        // For now, just check if target exists
-        let registry = self.registry.lock().await;
-        if registry.get_target(target_id).is_some() {
-            info!(target_id = %target_id, "Target disabled");
-            Ok(())
-        } else {
-            Err(AuditError::Configuration(format!("Target not found: {target_id}"), None))
-        }
+        self.runtime_view().disable_target(target_id).await
     }
 
     /// Removes a target from the system
@@ -624,16 +334,7 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn remove_target(&self, target_id: &str) -> AuditResult<()> {
-        let mut registry = self.registry.lock().await;
-        if let Some(target) = registry.remove_target(target_id) {
-            if let Err(e) = target.close().await {
-                error!(target_id = %target_id, error = %e, "Failed to close removed target");
-            }
-            info!(target_id = %target_id, "Target removed");
-            Ok(())
-        } else {
-            Err(AuditError::Configuration(format!("Target not found: {target_id}"), None))
-        }
+        self.runtime_view().remove_target(target_id).await
     }
 
     /// Updates or inserts a target
@@ -645,23 +346,7 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn upsert_target(&self, target_id: String, target: Box<dyn Target<AuditEntry> + Send + Sync>) -> AuditResult<()> {
-        let mut registry = self.registry.lock().await;
-
-        // Initialize the target
-        if let Err(e) = target.init().await {
-            return Err(AuditError::Target(e));
-        }
-
-        // Remove existing target if present
-        if let Some(old_target) = registry.remove_target(&target_id)
-            && let Err(e) = old_target.close().await
-        {
-            error!(target_id = %target_id, error = %e, "Failed to close old target during upsert");
-        }
-
-        registry.add_target(target_id.clone(), target);
-        info!(target_id = %target_id, "Target upserted");
-        Ok(())
+        self.runtime_view().upsert_target(target_id, target).await
     }
 
     /// Lists all targets
@@ -669,33 +354,27 @@ impl AuditSystem {
     /// # Returns
     /// * `Vec<String>` - List of target IDs
     pub async fn list_targets(&self) -> Vec<String> {
-        let registry = self.registry.lock().await;
-        registry.list_targets()
+        self.runtime_view().list_targets().await
     }
 
     /// Returns cloned target values for read-only runtime inspection.
-    pub async fn get_target_values(&self) -> Vec<Box<dyn Target<AuditEntry> + Send + Sync>> {
-        let registry = self.registry.lock().await;
-        registry.list_target_values()
+    pub async fn get_target_values(&self) -> Vec<rustfs_targets::SharedTarget<AuditEntry>> {
+        self.runtime_view().get_target_values().await
     }
 
     /// Returns per-target delivery metrics for Prometheus collection.
     pub async fn snapshot_target_metrics(&self) -> Vec<AuditTargetMetricSnapshot> {
-        let targets = self.get_target_values().await;
-        let mut snapshots = Vec::with_capacity(targets.len());
+        self.pipeline().snapshot_target_metrics().await
+    }
 
-        for target in targets {
-            let delivery = target.delivery_snapshot();
-            snapshots.push(AuditTargetMetricSnapshot {
-                failed_messages: delivery.failed_messages,
-                queue_length: delivery.queue_length,
-                target_id: target.id().to_string(),
-                total_messages: delivery.total_messages,
-            });
-        }
+    pub async fn snapshot_target_health(&self) -> Vec<rustfs_targets::RuntimeTargetHealthSnapshot> {
+        self.pipeline().snapshot_target_health().await
+    }
 
-        snapshots.sort_by(|a, b| a.target_id.cmp(&b.target_id));
-        snapshots
+    pub async fn runtime_status_snapshot(&self) -> rustfs_targets::RuntimeStatusSnapshot {
+        let replay_workers = self.stream_cancellers.read().await;
+        let registry = self.registry.lock().await;
+        registry.runtime_manager().status_snapshot(&replay_workers)
     }
 
     /// Gets information about a specific target
@@ -706,8 +385,7 @@ impl AuditSystem {
     /// # Returns
     /// * `Option<String>` - Target ID if found
     pub async fn get_target(&self, target_id: &str) -> Option<String> {
-        let registry = self.registry.lock().await;
-        registry.get_target(target_id).map(|target| target.id().to_string())
+        self.runtime_view().get_target(target_id).await
     }
 
     /// Reloads configuration and updates targets
@@ -722,32 +400,20 @@ impl AuditSystem {
 
         observability::record_config_reload();
 
-        // Stop all existing stream tasks first
-        self.stop_all_streams().await;
-
         // Store new configuration
         {
             let mut config_guard = self.config.write().await;
             *config_guard = Some(new_config.clone());
         }
 
-        // Close all existing targets
-        let mut registry = self.registry.lock().await;
-        if let Err(e) = registry.close_all().await {
-            error!(error = %e, "Failed to close existing targets during reload");
-        }
+        let final_state = match self.get_state().await {
+            AuditSystemState::Paused => AuditSystemState::Paused,
+            _ => AuditSystemState::Running,
+        };
 
-        // Create new targets from updated configuration
-        match registry.create_audit_targets_from_config(&new_config).await {
+        match self.create_targets_from_config(&new_config).await {
             Ok(targets) => {
-                info!(target_count = targets.len(), "Reloaded audit targets successfully");
-
-                for target in targets {
-                    if let Some(target_id) = self.init_and_register_target(target, &mut registry).await {
-                        info!(target_id = %target_id, "Target initialized (reload)");
-                    }
-                }
-
+                self.commit_runtime_targets(targets, final_state).await?;
                 info!("Audit configuration reloaded successfully");
                 Ok(())
             }
@@ -779,3 +445,106 @@ impl AuditSystem {
         observability::reset_metrics().await;
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::{AuditSystem, AuditSystemState};
+    use async_trait::async_trait;
+    use rustfs_targets::ReplayWorkerManager;
+    use rustfs_targets::arn::TargetID;
+    use rustfs_targets::store::{Key, Store};
+    use rustfs_targets::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta};
+    use rustfs_targets::{StoreError, Target, TargetError};
+    use serde::{Serialize, de::DeserializeOwned};
+    use std::collections::HashMap;
+    use std::sync::Arc;
+    use std::sync::atomic::{AtomicUsize, Ordering};
+    use tokio::sync::mpsc;
+
+    #[derive(Clone)]
+    struct TestTarget {
+        close_calls: Arc<AtomicUsize>,
+        id: TargetID,
+    }
+
+    impl TestTarget {
+        fn new(id: &str, name: &str) -> Self {
+            Self {
+                close_calls: Arc::new(AtomicUsize::new(0)),
+                id: TargetID::new(id.to_string(), name.to_string()),
+            }
+        }
+    }
+
+    #[async_trait]
+    impl<E> Target<E> for TestTarget
+    where
+        E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned,
+    {
+        fn id(&self) -> TargetID {
+            self.id.clone()
+        }
+
+        async fn is_active(&self) -> Result<bool, TargetError> {
+            Ok(true)
+        }
+
+        async fn save(&self, _event: Arc<EntityTarget<E>>) -> Result<(), TargetError> {
+            Ok(())
+        }
+
+        async fn send_raw_from_store(&self, _key: Key, _body: Vec<u8>, _meta: QueuedPayloadMeta) -> Result<(), TargetError> {
+            Ok(())
+        }
+
+        async fn close(&self) -> Result<(), TargetError> {
+            self.close_calls.fetch_add(1, Ordering::SeqCst);
+            Ok(())
+        }
+
+        fn store(&self) -> Option<&(dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send + Sync)> {
+            None
+        }
+
+        fn clone_dyn(&self) -> Box<dyn Target<E> + Send + Sync> {
+            Box::new(self.clone())
+        }
+
+        fn is_enabled(&self) -> bool {
+            true
+        }
+    }
+
+    #[tokio::test]
+    async fn reload_with_empty_config_stops_existing_runtime() {
+        let system = AuditSystem::new();
+        let target = TestTarget::new("primary", "webhook");
+        let close_calls = Arc::clone(&target.close_calls);
+
+        {
+            let mut registry = system.registry.lock().await;
+            registry.add_target("primary:webhook".to_string(), Box::new(target));
+        }
+        {
+            let mut state = system.state.write().await;
+            *state = AuditSystemState::Running;
+        }
+        {
+            let mut replay_workers = system.stream_cancellers.write().await;
+            let (cancel_tx, _cancel_rx) = mpsc::channel(1);
+            replay_workers.insert("primary:webhook".to_string(), cancel_tx);
+            assert_eq!(replay_workers.len(), 1);
+        }
+
+        system
+            .reload_config(rustfs_ecstore::config::Config(HashMap::new()))
+            .await
+            .expect("reload with empty config should succeed");
+
+        assert_eq!(system.get_state().await, AuditSystemState::Stopped);
+        assert!(system.list_targets().await.is_empty());
+        assert_eq!(system.runtime_status_snapshot().await, ReplayWorkerManager::new().snapshot(0));
+        assert_eq!(close_calls.load(Ordering::SeqCst), 1);
+        assert_eq!(*system.config.read().await, Some(rustfs_ecstore::config::Config(HashMap::new())));
+    }
+}

crates/audit/tests/pipeline_layer_test.rs

@@ -0,0 +1,170 @@
+//  Copyright 2024 RustFS Team
+//
+//  Licensed under the Apache License, Version 2.0 (the "License");
+//  you may not use this file except in compliance with the License.
+//  You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+//  Unless required by applicable law or agreed to in writing, software
+//  distributed under the License is distributed on an "AS IS" BASIS,
+//  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+//  See the License for the specific language governing permissions and
+//  limitations under the License.
+
+use async_trait::async_trait;
+use rustfs_audit::{AuditPipeline, AuditRegistry, AuditRuntimeFacade, AuditRuntimeView};
+use rustfs_targets::arn::TargetID;
+use rustfs_targets::store::{Key, Store};
+use rustfs_targets::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta};
+use rustfs_targets::{StoreError, Target, TargetError};
+use serde::{Serialize, de::DeserializeOwned};
+use std::sync::Arc;
+use std::sync::atomic::{AtomicUsize, Ordering};
+use tokio::sync::{Mutex, RwLock};
+
+#[derive(Clone)]
+struct TestTarget {
+    close_calls: Arc<AtomicUsize>,
+    id: TargetID,
+    init_calls: Arc<AtomicUsize>,
+}
+
+impl TestTarget {
+    fn new(id: &str, name: &str) -> Self {
+        Self {
+            close_calls: Arc::new(AtomicUsize::new(0)),
+            id: TargetID::new(id.to_string(), name.to_string()),
+            init_calls: Arc::new(AtomicUsize::new(0)),
+        }
+    }
+}
+
+#[async_trait]
+impl<E> Target<E> for TestTarget
+where
+    E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned,
+{
+    fn id(&self) -> TargetID {
+        self.id.clone()
+    }
+
+    async fn is_active(&self) -> Result<bool, TargetError> {
+        Ok(true)
+    }
+
+    async fn save(&self, _event: Arc<EntityTarget<E>>) -> Result<(), TargetError> {
+        Ok(())
+    }
+
+    async fn send_raw_from_store(&self, _key: Key, _body: Vec<u8>, _meta: QueuedPayloadMeta) -> Result<(), TargetError> {
+        Ok(())
+    }
+
+    async fn close(&self) -> Result<(), TargetError> {
+        self.close_calls.fetch_add(1, Ordering::SeqCst);
+        Ok(())
+    }
+
+    fn store(&self) -> Option<&(dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send + Sync)> {
+        None
+    }
+
+    fn clone_dyn(&self) -> Box<dyn Target<E> + Send + Sync> {
+        Box::new(self.clone())
+    }
+
+    async fn init(&self) -> Result<(), TargetError> {
+        self.init_calls.fetch_add(1, Ordering::SeqCst);
+        Ok(())
+    }
+
+    fn is_enabled(&self) -> bool {
+        true
+    }
+}
+
+#[tokio::test]
+async fn audit_runtime_view_lists_empty_targets() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let runtime_view = AuditRuntimeView::new(registry);
+
+    assert!(runtime_view.list_targets().await.is_empty());
+    assert!(runtime_view.get_target_values().await.is_empty());
+    assert!(runtime_view.get_target("missing").await.is_none());
+}
+
+#[tokio::test]
+async fn audit_pipeline_reports_empty_runtime_snapshots() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let pipeline = AuditPipeline::new(registry);
+
+    assert!(pipeline.snapshot_target_metrics().await.is_empty());
+    assert!(pipeline.snapshot_target_health().await.is_empty());
+}
+
+#[tokio::test]
+async fn audit_runtime_facade_stops_empty_replay_workers() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let replay_workers = Arc::new(RwLock::new(rustfs_targets::ReplayWorkerManager::new()));
+    let facade = AuditRuntimeFacade::new(registry, replay_workers);
+
+    facade.stop_replay_workers().await;
+}
+
+#[tokio::test]
+async fn audit_runtime_facade_activates_empty_target_list() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let replay_workers = Arc::new(RwLock::new(rustfs_targets::ReplayWorkerManager::new()));
+    let facade = AuditRuntimeFacade::new(registry, replay_workers);
+
+    let activation = facade.activate_targets_with_replay(Vec::new()).await;
+    assert!(activation.targets.is_empty());
+    assert_eq!(activation.replay_workers.len(), 0);
+}
+
+#[tokio::test]
+async fn audit_runtime_view_upsert_and_remove_target() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let runtime_view = AuditRuntimeView::new(registry.clone());
+    let target = TestTarget::new("primary", "webhook");
+    let init_calls = Arc::clone(&target.init_calls);
+    let close_calls = Arc::clone(&target.close_calls);
+
+    runtime_view
+        .upsert_target("primary:webhook".to_string(), Box::new(target))
+        .await
+        .expect("upsert should succeed");
+
+    assert_eq!(runtime_view.list_targets().await, vec!["primary:webhook".to_string()]);
+    assert_eq!(init_calls.load(Ordering::SeqCst), 1);
+
+    runtime_view
+        .remove_target("primary:webhook")
+        .await
+        .expect("remove should succeed");
+
+    assert!(runtime_view.list_targets().await.is_empty());
+    assert_eq!(close_calls.load(Ordering::SeqCst), 1);
+}
+
+#[tokio::test]
+async fn audit_runtime_facade_replace_targets_commits_runtime_state() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let replay_workers = Arc::new(RwLock::new(rustfs_targets::ReplayWorkerManager::new()));
+    let facade = AuditRuntimeFacade::new(registry.clone(), replay_workers.clone());
+    let target = TestTarget::new("primary", "webhook");
+    let activation = rustfs_targets::RuntimeActivation {
+        replay_workers: rustfs_targets::ReplayWorkerManager::new(),
+        targets: vec![Arc::new(target) as rustfs_targets::SharedTarget<rustfs_audit::AuditEntry>],
+    };
+
+    facade
+        .replace_targets(activation)
+        .await
+        .expect("replace_targets should succeed");
+
+    let runtime_view = AuditRuntimeView::new(registry);
+    assert_eq!(runtime_view.list_targets().await, vec!["primary:webhook".to_string()]);
+    assert_eq!(replay_workers.read().await.len(), 0);
+}

crates/config/README.md

@@ -41,6 +41,7 @@ Examples:
 - `RUSTFS_ADDRESS`
 - `RUSTFS_VOLUMES`
 - `RUSTFS_LICENSE`
+- `RUSTFS_LICENSE_PUBLIC_KEY`
 
 Current guidance:
 - Prefer module-specific names only when they are not top-level product configuration.
@@ -51,6 +52,16 @@ Current guidance:
   - `RUSTFS_ENABLE_HEAL` -> `RUSTFS_HEAL_ENABLED`
   - `RUSTFS_DATA_SCANNER_START_DELAY_SECS` -> `RUSTFS_SCANNER_START_DELAY_SECS`
 
+## License environment variables
+
+- `RUSTFS_LICENSE` contains the signed license token.
+- `RUSTFS_LICENSE_PUBLIC_KEY` contains the RSA public key used to verify signed license tokens.
+
+## CORS environment variables
+
+- `RUSTFS_CORS_ALLOWED_ORIGINS` defaults to empty, so the S3 endpoint emits no generic CORS headers unless configured. Set `*` for wildcard origins without credentials, or a comma-separated allow-list for credentialed explicit origins.
+- `RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS` defaults to `*` for the console service.
+
 ## Scanner environment aliases
 
 - `RUSTFS_SCANNER_SPEED` (canonical, also accepts `MINIO_SCANNER_SPEED`)

crates/config/src/audit/amqp.rs

@@ -0,0 +1,60 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const AUDIT_AMQP_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::AMQP_URL,
+    crate::AMQP_EXCHANGE,
+    crate::AMQP_ROUTING_KEY,
+    crate::AMQP_MANDATORY,
+    crate::AMQP_PERSISTENT,
+    crate::AMQP_USERNAME,
+    crate::AMQP_PASSWORD,
+    crate::AMQP_TLS_CA,
+    crate::AMQP_TLS_CLIENT_CERT,
+    crate::AMQP_TLS_CLIENT_KEY,
+    crate::AMQP_QUEUE_DIR,
+    crate::AMQP_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_AUDIT_AMQP_ENABLE: &str = "RUSTFS_AUDIT_AMQP_ENABLE";
+pub const ENV_AUDIT_AMQP_URL: &str = "RUSTFS_AUDIT_AMQP_URL";
+pub const ENV_AUDIT_AMQP_EXCHANGE: &str = "RUSTFS_AUDIT_AMQP_EXCHANGE";
+pub const ENV_AUDIT_AMQP_ROUTING_KEY: &str = "RUSTFS_AUDIT_AMQP_ROUTING_KEY";
+pub const ENV_AUDIT_AMQP_MANDATORY: &str = "RUSTFS_AUDIT_AMQP_MANDATORY";
+pub const ENV_AUDIT_AMQP_PERSISTENT: &str = "RUSTFS_AUDIT_AMQP_PERSISTENT";
+pub const ENV_AUDIT_AMQP_USERNAME: &str = "RUSTFS_AUDIT_AMQP_USERNAME";
+pub const ENV_AUDIT_AMQP_PASSWORD: &str = "RUSTFS_AUDIT_AMQP_PASSWORD";
+pub const ENV_AUDIT_AMQP_TLS_CA: &str = "RUSTFS_AUDIT_AMQP_TLS_CA";
+pub const ENV_AUDIT_AMQP_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_AMQP_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_AMQP_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_AMQP_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_AMQP_QUEUE_DIR: &str = "RUSTFS_AUDIT_AMQP_QUEUE_DIR";
+pub const ENV_AUDIT_AMQP_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_AMQP_QUEUE_LIMIT";
+
+pub const ENV_AUDIT_AMQP_KEYS: &[&str; 13] = &[
+    ENV_AUDIT_AMQP_ENABLE,
+    ENV_AUDIT_AMQP_URL,
+    ENV_AUDIT_AMQP_EXCHANGE,
+    ENV_AUDIT_AMQP_ROUTING_KEY,
+    ENV_AUDIT_AMQP_MANDATORY,
+    ENV_AUDIT_AMQP_PERSISTENT,
+    ENV_AUDIT_AMQP_USERNAME,
+    ENV_AUDIT_AMQP_PASSWORD,
+    ENV_AUDIT_AMQP_TLS_CA,
+    ENV_AUDIT_AMQP_TLS_CLIENT_CERT,
+    ENV_AUDIT_AMQP_TLS_CLIENT_KEY,
+    ENV_AUDIT_AMQP_QUEUE_DIR,
+    ENV_AUDIT_AMQP_QUEUE_LIMIT,
+];

crates/config/src/audit/mod.rs

@@ -16,16 +16,24 @@
 //! This module defines the configuration for audit systems, including
 //! webhook and MQTT audit-related settings.
 
+mod amqp;
 mod kafka;
 mod mqtt;
+mod mysql;
 mod nats;
+mod postgres;
 mod pulsar;
+mod redis;
 mod webhook;
 
+pub use amqp::*;
 pub use kafka::*;
 pub use mqtt::*;
+pub use mysql::*;
 pub use nats::*;
+pub use postgres::*;
 pub use pulsar::*;
+pub use redis::*;
 pub use webhook::*;
 
 use crate::DEFAULT_DELIMITER;
@@ -35,17 +43,24 @@ pub const AUDIT_PREFIX: &str = "audit";
 pub const AUDIT_ROUTE_PREFIX: &str = const_str::concat!(AUDIT_PREFIX, DEFAULT_DELIMITER);
 
 pub const AUDIT_WEBHOOK_SUB_SYS: &str = "audit_webhook";
+pub const AUDIT_AMQP_SUB_SYS: &str = "audit_amqp";
 pub const AUDIT_KAFKA_SUB_SYS: &str = "audit_kafka";
 pub const AUDIT_MQTT_SUB_SYS: &str = "audit_mqtt";
+pub const AUDIT_MYSQL_SUB_SYS: &str = "audit_mysql";
 pub const AUDIT_NATS_SUB_SYS: &str = "audit_nats";
+pub const AUDIT_POSTGRES_SUB_SYS: &str = "audit_postgres";
 pub const AUDIT_PULSAR_SUB_SYS: &str = "audit_pulsar";
-
+pub const AUDIT_REDIS_SUB_SYS: &str = "audit_redis";
+pub const AUDIT_REDIS_DEFAULT_CHANNEL: &str = "rustfs_audit_channel";
 pub const AUDIT_STORE_EXTENSION: &str = ".audit";
-#[allow(dead_code)]
 pub const AUDIT_SUB_SYSTEMS: &[&str] = &[
+    AUDIT_AMQP_SUB_SYS,
     AUDIT_KAFKA_SUB_SYS,
     AUDIT_MQTT_SUB_SYS,
+    AUDIT_MYSQL_SUB_SYS,
     AUDIT_NATS_SUB_SYS,
+    AUDIT_POSTGRES_SUB_SYS,
     AUDIT_PULSAR_SUB_SYS,
+    AUDIT_REDIS_SUB_SYS,
     AUDIT_WEBHOOK_SUB_SYS,
 ];

crates/config/src/audit/mysql.rs

@@ -0,0 +1,53 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// A list of all valid configuration keys for an audit MySQL target.
+pub const AUDIT_MYSQL_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::MYSQL_DSN_STRING,
+    crate::MYSQL_TABLE,
+    crate::MYSQL_FORMAT,
+    crate::MYSQL_TLS_CA,
+    crate::MYSQL_TLS_CLIENT_CERT,
+    crate::MYSQL_TLS_CLIENT_KEY,
+    crate::MYSQL_QUEUE_DIR,
+    crate::MYSQL_QUEUE_LIMIT,
+    crate::MYSQL_MAX_OPEN_CONNECTIONS,
+    crate::COMMENT_KEY,
+];
+
+// MySQL environment variables for audit target.
+pub const ENV_AUDIT_MYSQL_ENABLE: &str = "RUSTFS_AUDIT_MYSQL_ENABLE";
+pub const ENV_AUDIT_MYSQL_DSN_STRING: &str = "RUSTFS_AUDIT_MYSQL_DSN_STRING";
+pub const ENV_AUDIT_MYSQL_TABLE: &str = "RUSTFS_AUDIT_MYSQL_TABLE";
+pub const ENV_AUDIT_MYSQL_FORMAT: &str = "RUSTFS_AUDIT_MYSQL_FORMAT";
+pub const ENV_AUDIT_MYSQL_TLS_CA: &str = "RUSTFS_AUDIT_MYSQL_TLS_CA";
+pub const ENV_AUDIT_MYSQL_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_MYSQL_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_MYSQL_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_MYSQL_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_MYSQL_QUEUE_DIR: &str = "RUSTFS_AUDIT_MYSQL_QUEUE_DIR";
+pub const ENV_AUDIT_MYSQL_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_MYSQL_QUEUE_LIMIT";
+pub const ENV_AUDIT_MYSQL_MAX_OPEN_CONNECTIONS: &str = "RUSTFS_AUDIT_MYSQL_MAX_OPEN_CONNECTIONS";
+
+pub const ENV_AUDIT_MYSQL_KEYS: &[&str; 10] = &[
+    ENV_AUDIT_MYSQL_ENABLE,
+    ENV_AUDIT_MYSQL_DSN_STRING,
+    ENV_AUDIT_MYSQL_TABLE,
+    ENV_AUDIT_MYSQL_FORMAT,
+    ENV_AUDIT_MYSQL_TLS_CA,
+    ENV_AUDIT_MYSQL_TLS_CLIENT_CERT,
+    ENV_AUDIT_MYSQL_TLS_CLIENT_KEY,
+    ENV_AUDIT_MYSQL_QUEUE_DIR,
+    ENV_AUDIT_MYSQL_QUEUE_LIMIT,
+    ENV_AUDIT_MYSQL_MAX_OPEN_CONNECTIONS,
+];

crates/config/src/audit/postgres.rs

@@ -0,0 +1,51 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const AUDIT_POSTGRES_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::POSTGRES_DSN_STRING,
+    crate::POSTGRES_TABLE,
+    crate::POSTGRES_FORMAT,
+    crate::POSTGRES_TLS_REQUIRED,
+    crate::POSTGRES_TLS_CA,
+    crate::POSTGRES_TLS_CLIENT_CERT,
+    crate::POSTGRES_TLS_CLIENT_KEY,
+    crate::POSTGRES_QUEUE_DIR,
+    crate::POSTGRES_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_AUDIT_POSTGRES_ENABLE: &str = "RUSTFS_AUDIT_POSTGRES_ENABLE";
+pub const ENV_AUDIT_POSTGRES_DSN_STRING: &str = "RUSTFS_AUDIT_POSTGRES_DSN_STRING";
+pub const ENV_AUDIT_POSTGRES_TABLE: &str = "RUSTFS_AUDIT_POSTGRES_TABLE";
+pub const ENV_AUDIT_POSTGRES_FORMAT: &str = "RUSTFS_AUDIT_POSTGRES_FORMAT";
+pub const ENV_AUDIT_POSTGRES_TLS_REQUIRED: &str = "RUSTFS_AUDIT_POSTGRES_TLS_REQUIRED";
+pub const ENV_AUDIT_POSTGRES_TLS_CA: &str = "RUSTFS_AUDIT_POSTGRES_TLS_CA";
+pub const ENV_AUDIT_POSTGRES_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_POSTGRES_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_POSTGRES_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_POSTGRES_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_POSTGRES_QUEUE_DIR: &str = "RUSTFS_AUDIT_POSTGRES_QUEUE_DIR";
+pub const ENV_AUDIT_POSTGRES_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_POSTGRES_QUEUE_LIMIT";
+
+pub const ENV_AUDIT_POSTGRES_KEYS: &[&str; 10] = &[
+    ENV_AUDIT_POSTGRES_ENABLE,
+    ENV_AUDIT_POSTGRES_DSN_STRING,
+    ENV_AUDIT_POSTGRES_TABLE,
+    ENV_AUDIT_POSTGRES_FORMAT,
+    ENV_AUDIT_POSTGRES_TLS_REQUIRED,
+    ENV_AUDIT_POSTGRES_TLS_CA,
+    ENV_AUDIT_POSTGRES_TLS_CLIENT_CERT,
+    ENV_AUDIT_POSTGRES_TLS_CLIENT_KEY,
+    ENV_AUDIT_POSTGRES_QUEUE_DIR,
+    ENV_AUDIT_POSTGRES_QUEUE_LIMIT,
+];

crates/config/src/audit/redis.rs

@@ -0,0 +1,81 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const ENV_AUDIT_REDIS_ENABLE: &str = "RUSTFS_AUDIT_REDIS_ENABLE";
+pub const ENV_AUDIT_REDIS_URL: &str = "RUSTFS_AUDIT_REDIS_URL";
+pub const ENV_AUDIT_REDIS_CHANNEL: &str = "RUSTFS_AUDIT_REDIS_CHANNEL";
+pub const ENV_AUDIT_REDIS_USERNAME: &str = "RUSTFS_AUDIT_REDIS_USERNAME";
+pub const ENV_AUDIT_REDIS_PASSWORD: &str = "RUSTFS_AUDIT_REDIS_PASSWORD";
+pub const ENV_AUDIT_REDIS_KEEP_ALIVE_INTERVAL: &str = "RUSTFS_AUDIT_REDIS_KEEP_ALIVE_INTERVAL";
+pub const ENV_AUDIT_REDIS_QUEUE_DIR: &str = "RUSTFS_AUDIT_REDIS_QUEUE_DIR";
+pub const ENV_AUDIT_REDIS_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_REDIS_QUEUE_LIMIT";
+pub const ENV_AUDIT_REDIS_MAX_RETRY_ATTEMPTS: &str = "RUSTFS_AUDIT_REDIS_MAX_RETRY_ATTEMPTS";
+pub const ENV_AUDIT_REDIS_RECONNECT_RETRY_ATTEMPTS: &str = "RUSTFS_AUDIT_REDIS_RECONNECT_RETRY_ATTEMPTS";
+pub const ENV_AUDIT_REDIS_MIN_RETRY_DELAY: &str = "RUSTFS_AUDIT_REDIS_MIN_RETRY_DELAY";
+pub const ENV_AUDIT_REDIS_MAX_RETRY_DELAY: &str = "RUSTFS_AUDIT_REDIS_MAX_RETRY_DELAY";
+pub const ENV_AUDIT_REDIS_CONNECTION_TIMEOUT: &str = "RUSTFS_AUDIT_REDIS_CONNECTION_TIMEOUT";
+pub const ENV_AUDIT_REDIS_RESPONSE_TIMEOUT: &str = "RUSTFS_AUDIT_REDIS_RESPONSE_TIMEOUT";
+pub const ENV_AUDIT_REDIS_PIPELINE_BUFFER_SIZE: &str = "RUSTFS_AUDIT_REDIS_PIPELINE_BUFFER_SIZE";
+pub const ENV_AUDIT_REDIS_TLS_POLICY: &str = "RUSTFS_AUDIT_REDIS_TLS_POLICY";
+pub const ENV_AUDIT_REDIS_TLS_CA: &str = "RUSTFS_AUDIT_REDIS_TLS_CA";
+pub const ENV_AUDIT_REDIS_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_REDIS_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_REDIS_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_REDIS_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_REDIS_TLS_ALLOW_INSECURE: &str = "RUSTFS_AUDIT_REDIS_TLS_ALLOW_INSECURE";
+
+pub const ENV_AUDIT_REDIS_KEYS: &[&str; 20] = &[
+    ENV_AUDIT_REDIS_ENABLE,
+    ENV_AUDIT_REDIS_URL,
+    ENV_AUDIT_REDIS_CHANNEL,
+    ENV_AUDIT_REDIS_USERNAME,
+    ENV_AUDIT_REDIS_PASSWORD,
+    ENV_AUDIT_REDIS_KEEP_ALIVE_INTERVAL,
+    ENV_AUDIT_REDIS_QUEUE_DIR,
+    ENV_AUDIT_REDIS_QUEUE_LIMIT,
+    ENV_AUDIT_REDIS_MAX_RETRY_ATTEMPTS,
+    ENV_AUDIT_REDIS_RECONNECT_RETRY_ATTEMPTS,
+    ENV_AUDIT_REDIS_MIN_RETRY_DELAY,
+    ENV_AUDIT_REDIS_MAX_RETRY_DELAY,
+    ENV_AUDIT_REDIS_CONNECTION_TIMEOUT,
+    ENV_AUDIT_REDIS_RESPONSE_TIMEOUT,
+    ENV_AUDIT_REDIS_PIPELINE_BUFFER_SIZE,
+    ENV_AUDIT_REDIS_TLS_POLICY,
+    ENV_AUDIT_REDIS_TLS_CA,
+    ENV_AUDIT_REDIS_TLS_CLIENT_CERT,
+    ENV_AUDIT_REDIS_TLS_CLIENT_KEY,
+    ENV_AUDIT_REDIS_TLS_ALLOW_INSECURE,
+];
+
+pub const AUDIT_REDIS_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::REDIS_URL,
+    crate::REDIS_CHANNEL,
+    crate::REDIS_USERNAME,
+    crate::REDIS_PASSWORD,
+    crate::REDIS_KEEP_ALIVE_INTERVAL,
+    crate::REDIS_QUEUE_DIR,
+    crate::REDIS_QUEUE_LIMIT,
+    crate::REDIS_MAX_RETRY_ATTEMPTS,
+    crate::REDIS_RECONNECT_RETRY_ATTEMPTS,
+    crate::REDIS_MIN_RETRY_DELAY,
+    crate::REDIS_MAX_RETRY_DELAY,
+    crate::REDIS_CONNECTION_TIMEOUT,
+    crate::REDIS_RESPONSE_TIMEOUT,
+    crate::REDIS_PIPELINE_BUFFER_SIZE,
+    crate::REDIS_TLS_POLICY,
+    crate::REDIS_TLS_CA,
+    crate::REDIS_TLS_CLIENT_CERT,
+    crate::REDIS_TLS_CLIENT_KEY,
+    crate::REDIS_TLS_ALLOW_INSECURE,
+    crate::COMMENT_KEY,
+];

crates/config/src/constants/app.rs

@@ -149,18 +149,12 @@ pub const ENV_RUSTFS_ACCESS_KEY: &str = "RUSTFS_ACCESS_KEY";
 /// Environment variable for server access key file.
 pub const ENV_RUSTFS_ACCESS_KEY_FILE: &str = "RUSTFS_ACCESS_KEY_FILE";
 
-/// Environment variable for server root user.
-pub const ENV_RUSTFS_ROOT_USER: &str = "RUSTFS_ROOT_USER";
-
 /// Environment variable for server secret key.
 pub const ENV_RUSTFS_SECRET_KEY: &str = "RUSTFS_SECRET_KEY";
 
 /// Environment variable for server secret key file.
 pub const ENV_RUSTFS_SECRET_KEY_FILE: &str = "RUSTFS_SECRET_KEY_FILE";
 
-/// Environment variable for server root password.
-pub const ENV_RUSTFS_ROOT_PASSWORD: &str = "RUSTFS_ROOT_PASSWORD";
-
 /// Environment variable for server OBS endpoint.
 pub const ENV_RUSTFS_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";
 
@@ -235,6 +229,9 @@ pub const ENV_RUSTFS_REGION: &str = "RUSTFS_REGION";
 /// Environment variable for server license.
 pub const ENV_RUSTFS_LICENSE: &str = "RUSTFS_LICENSE";
 
+/// Environment variable for the RSA public key used to verify server licenses.
+pub const ENV_RUSTFS_LICENSE_PUBLIC_KEY: &str = "RUSTFS_LICENSE_PUBLIC_KEY";
+
 /// Default log filename for rustfs
 /// This is the default log filename for rustfs.
 /// It is used to store the logs of the application.
@@ -348,6 +345,7 @@ mod tests {
     fn test_environment_constants() {
         // Test environment related constants
         assert_eq!(ENVIRONMENT, "production");
+        assert_eq!(ENV_RUSTFS_LICENSE_PUBLIC_KEY, "RUSTFS_LICENSE_PUBLIC_KEY");
         assert!(
             ["development", "staging", "production", "test"].contains(&ENVIRONMENT),
             "Environment should be a standard environment name"

crates/config/src/constants/console.rs

@@ -17,16 +17,21 @@
 pub const ENV_CORS_ALLOWED_ORIGINS: &str = "RUSTFS_CORS_ALLOWED_ORIGINS";
 
 /// Default CORS allowed origins for the endpoint service
-/// Comes from the console service default
-/// See DEFAULT_CONSOLE_CORS_ALLOWED_ORIGINS
-pub const DEFAULT_CORS_ALLOWED_ORIGINS: &str = DEFAULT_CONSOLE_CORS_ALLOWED_ORIGINS;
+/// Empty means the S3 endpoint emits no generic CORS headers unless configured.
+pub const DEFAULT_CORS_ALLOWED_ORIGINS: &str = "";
 
 /// CORS allowed origins for the console service
 /// Comma-separated list of origins or "*" for all origins
 pub const ENV_CONSOLE_CORS_ALLOWED_ORIGINS: &str = "RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS";
 
-/// Default CORS allowed origins for the console service
-pub const DEFAULT_CONSOLE_CORS_ALLOWED_ORIGINS: &str = "*";
+/// Default CORS allowed origins for the console service.
+///
+/// Empty string means same-origin only — no `Access-Control-Allow-Origin`
+/// header is emitted, so browsers will not allow cross-origin reads of
+/// console responses by default. Operators that need cross-origin access set
+/// `RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS` to a comma-separated allow-list, or
+/// to `*` to keep the previous permissive behavior.
+pub const DEFAULT_CONSOLE_CORS_ALLOWED_ORIGINS: &str = "";
 
 /// Enable or disable the console service
 pub const ENV_CONSOLE_ENABLE: &str = "RUSTFS_CONSOLE_ENABLE";
@@ -89,3 +94,20 @@ pub const ENV_UPDATE_CHECK: &str = "RUSTFS_CHECK_UPDATE";
 
 /// Default value for update toggle
 pub const DEFAULT_UPDATE_CHECK: bool = true;
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn endpoint_cors_default_is_restrictive() {
+        assert_eq!(ENV_CORS_ALLOWED_ORIGINS, "RUSTFS_CORS_ALLOWED_ORIGINS");
+        assert_eq!(DEFAULT_CORS_ALLOWED_ORIGINS, "");
+    }
+
+    #[test]
+    fn console_cors_default_is_same_origin_only() {
+        assert_eq!(ENV_CONSOLE_CORS_ALLOWED_ORIGINS, "RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS");
+        assert_eq!(DEFAULT_CONSOLE_CORS_ALLOWED_ORIGINS, "");
+    }
+}

crates/config/src/constants/env.rs

@@ -34,6 +34,12 @@ pub const ENV_NOTIFY_ENABLE: &str = "RUSTFS_NOTIFY_ENABLE";
 pub const DEFAULT_AUDIT_ENABLE: bool = false;
 /// Default global notify switch (disabled by default).
 pub const DEFAULT_NOTIFY_ENABLE: bool = false;
+/// Canonical ILM process boundary env key (seconds).
+pub const ENV_ILM_PROCESS_TIME: &str = "RUSTFS_ILM_PROCESS_TIME";
+/// Deprecated ILM process boundary env key kept for compatibility.
+pub const ENV_ILM_PROCESS_TIME_DEPRECATED: &str = "_RUSTFS_ILM_PROCESS_TIME";
+/// Default ILM process boundary in seconds (24h).
+pub const DEFAULT_ILM_PROCESS_TIME_SECS: i32 = 86400;
 
 /// Medium-drawn lines separator
 /// This is used to separate words in environment variable names.

crates/config/src/constants/protocols.rs

@@ -57,3 +57,114 @@ pub const ENV_WEBDAV_CERTS_DIR: &str = "RUSTFS_WEBDAV_CERTS_DIR";
 pub const ENV_WEBDAV_CA_FILE: &str = "RUSTFS_WEBDAV_CA_FILE";
 pub const ENV_WEBDAV_MAX_BODY_SIZE: &str = "RUSTFS_WEBDAV_MAX_BODY_SIZE";
 pub const ENV_WEBDAV_REQUEST_TIMEOUT: &str = "RUSTFS_WEBDAV_REQUEST_TIMEOUT";
+
+/// Default SFTP server bind address.
+pub const DEFAULT_SFTP_ADDRESS: &str = "0.0.0.0:2222";
+
+/// Default for SFTP host-key directory. None means no default. Operators
+/// must set RUSTFS_SFTP_HOST_KEY_DIR explicitly when SFTP is enabled.
+pub const DEFAULT_SFTP_HOST_KEY_DIR: Option<&str> = None;
+
+/// SFTP environment variable names.
+pub const ENV_SFTP_ENABLE: &str = "RUSTFS_SFTP_ENABLE";
+pub const ENV_SFTP_ADDRESS: &str = "RUSTFS_SFTP_ADDRESS";
+pub const ENV_SFTP_HOST_KEY_DIR: &str = "RUSTFS_SFTP_HOST_KEY_DIR";
+pub const ENV_SFTP_HOST_KEY_RELOAD_ENABLE: &str = "RUSTFS_SFTP_HOST_KEY_RELOAD_ENABLE";
+pub const ENV_SFTP_HOST_KEY_RELOAD_INTERVAL: &str = "RUSTFS_SFTP_HOST_KEY_RELOAD_INTERVAL";
+pub const ENV_SFTP_IDLE_TIMEOUT: &str = "RUSTFS_SFTP_IDLE_TIMEOUT";
+/// S3 multipart part size in bytes. Default DEFAULT_SFTP_PART_SIZE (16 MiB).
+/// Valid range 5 MiB to 5 GiB (S3 protocol bounds), enforced at startup.
+///
+/// The per-upload size ceiling is part_size * 10_000 (the S3 parts cap),
+/// so the default caps single uploads at 160 GiB. Deployments expecting
+/// larger single files must raise this: 64 MiB -> 640 GiB, 128 MiB ->
+/// 1.25 TiB, 512 MiB -> 5 TiB (S3 object max). Rename is not affected;
+/// multipart_copy scales the per-part size dynamically and handles up
+/// to the 5 TiB S3 object limit regardless of this setting.
+pub const ENV_SFTP_PART_SIZE: &str = "RUSTFS_SFTP_PART_SIZE";
+pub const ENV_SFTP_READ_ONLY: &str = "RUSTFS_SFTP_READ_ONLY";
+pub const ENV_SFTP_BANNER: &str = "RUSTFS_SFTP_BANNER";
+/// Optional environment variable. If RUSTFS_SFTP_HANDLES_PER_SESSION
+/// is not set in the process environment, the server uses the default
+/// of 64 handles per session and emits no warning. If set, the value
+/// must be in the inclusive range 8 to 1024. Out-of-range values fall
+/// back to the default of 64 with a warn-level log naming the
+/// requested value and the bounds.
+///
+/// Caps the maximum number of simultaneously-open SFTP handles per
+/// session. A handle is the server-side identifier returned by
+/// SSH_FXP_OPEN and SSH_FXP_OPENDIR. One client typically uses one
+/// handle per file in flight plus one per directory listing.
+/// Operators running clients with deep pipelining may raise this.
+pub const ENV_SFTP_HANDLES_PER_SESSION: &str = "RUSTFS_SFTP_HANDLES_PER_SESSION";
+
+/// Optional environment variable. If RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS
+/// is not set in the process environment, the server uses the default
+/// of 60 seconds and emits no warning. If set, the value must be in
+/// the inclusive range 5 to 600 seconds. Out-of-range values fall
+/// back to the default with a warn-level log naming the requested
+/// value and the bounds.
+///
+/// Bounds every storage backend call issued by the SFTP driver. A
+/// backend that does not respond within this many seconds returns
+/// Failure to the client and emits a warn log naming the backend
+/// method. This catches a backend that accepted the request and never
+/// returned a body, which the SSH keepalive cannot detect because the
+/// transport itself remains live.
+pub const ENV_SFTP_BACKEND_OP_TIMEOUT_SECS: &str = "RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS";
+
+/// Optional environment variable. If RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES
+/// is not set in the process environment, the server uses a 4 MiB
+/// default and emits no warning. If set, the value must be in the
+/// inclusive range MAX_READ_LEN (256 KiB) to 64 MiB. Out-of-range
+/// values fall back to the default with a warn-level log naming the
+/// requested value and the bounds.
+///
+/// Per-handle byte window the SFTP read path fetches in one backend
+/// call on a cache miss. Subsequent FXP_READs within that window are
+/// served from the buffer without a backend round trip. For
+/// sequential downloads the backend round-trip count drops by
+/// window_bytes / MAX_READ_LEN. Random-access workloads should set
+/// the window equal to MAX_READ_LEN to opt out of read-ahead.
+pub const ENV_SFTP_READ_CACHE_WINDOW_BYTES: &str = "RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES";
+
+/// Optional environment variable. If
+/// RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES is not set in the process
+/// environment, the server uses a 256 MiB default and emits no
+/// warning. If set, the value must be at least 16 MiB. Below-min
+/// values fall back to the default with a warn-level log naming the
+/// requested value and the bound.
+///
+/// Process-wide ceiling on cumulative read cache memory across every
+/// live SFTP handle. Once the accumulator plus a new window would
+/// exceed this value, the populate call on the per-handle cache is
+/// skipped. The read still completes from the freshly-fetched bytes
+/// without storing them in the cache, at the cost of one backend
+/// call per FXP_READ. High-concurrency deployments expecting many
+/// parallel downloads should raise this in step with the per-session
+/// handle cap.
+pub const ENV_SFTP_READ_CACHE_TOTAL_MEM_BYTES: &str = "RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES";
+
+/// Default idle session timeout in seconds.
+pub const DEFAULT_SFTP_IDLE_TIMEOUT: u64 = 600;
+
+/// Default SFTP host key hot reload enabled state.
+pub const DEFAULT_SFTP_HOST_KEY_RELOAD_ENABLE: bool = false;
+
+/// Default SFTP host key hot reload interval in seconds.
+pub const DEFAULT_SFTP_HOST_KEY_RELOAD_INTERVAL: u64 = 30;
+
+/// Default S3 multipart upload part size in bytes (16 MiB).
+///
+/// The per-upload size ceiling is part_size * 10_000 (the S3 parts cap),
+/// so the default gives a 160 GiB single-upload limit. Deployments that
+/// expect single files larger than this must raise part_size:
+/// 64 MiB -> 640 GiB, 128 MiB -> 1.25 TiB, 512 MiB -> 5 TiB (S3 max).
+/// The minimum is 5 MiB and the maximum is 5 GiB (S3 protocol bounds).
+pub const DEFAULT_SFTP_PART_SIZE: u64 = 16_777_216;
+
+/// Default read-only mode (disabled).
+pub const DEFAULT_SFTP_READ_ONLY: bool = false;
+
+/// Default SSH identification string (no version disclosure).
+pub const DEFAULT_SFTP_BANNER: &str = "SSH-2.0-RustFS";

crates/config/src/constants/runtime.rs

@@ -77,10 +77,13 @@ pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_WRITE_ENABLE: bool = false;
 pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE: bool = false;
 pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD: usize = 4 * 1024 * 1024;
 
-/// Threshold for small object seek support in megabytes.
+/// Threshold for small object seek support in bytes.
 ///
-/// When an object is smaller than this size, rustfs will provide seek support.
+/// When an object response is smaller than this size, rustfs may provide
+/// in-memory seek support. Runtime GET logic also enforces a hard safety cap
+/// (`64 MiB`) to prevent large-download memory spikes even if this threshold
+/// is configured higher.
 ///
-/// Default is set to 10MB.
+/// Default is set to 10 MiB.
 pub const ENV_OBJECT_SEEK_SUPPORT_THRESHOLD: &str = "RUSTFS_OBJECT_SEEK_SUPPORT_THRESHOLD";
 pub const DEFAULT_OBJECT_SEEK_SUPPORT_THRESHOLD: usize = 10 * 1024 * 1024;

crates/config/src/constants/targets.rs

@@ -50,6 +50,19 @@ pub const KAFKA_TLS_CA: &str = "tls_ca";
 pub const KAFKA_TLS_CLIENT_CERT: &str = "tls_client_cert";
 pub const KAFKA_TLS_CLIENT_KEY: &str = "tls_client_key";
 
+pub const AMQP_URL: &str = "url";
+pub const AMQP_EXCHANGE: &str = "exchange";
+pub const AMQP_ROUTING_KEY: &str = "routing_key";
+pub const AMQP_MANDATORY: &str = "mandatory";
+pub const AMQP_PERSISTENT: &str = "persistent";
+pub const AMQP_USERNAME: &str = "username";
+pub const AMQP_PASSWORD: &str = "password";
+pub const AMQP_TLS_CA: &str = "tls_ca";
+pub const AMQP_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const AMQP_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const AMQP_QUEUE_DIR: &str = "queue_dir";
+pub const AMQP_QUEUE_LIMIT: &str = "queue_limit";
+
 pub const NATS_ADDRESS: &str = "address";
 pub const NATS_SUBJECT: &str = "subject";
 pub const NATS_USERNAME: &str = "username";
@@ -74,6 +87,48 @@ pub const PULSAR_TLS_HOSTNAME_VERIFICATION: &str = "tls_hostname_verification";
 pub const PULSAR_QUEUE_DIR: &str = "queue_dir";
 pub const PULSAR_QUEUE_LIMIT: &str = "queue_limit";
 
+pub const BASE_DSN_STRING: &str = "dsn_string";
+
+pub const MYSQL_DSN_STRING: &str = BASE_DSN_STRING;
+pub const MYSQL_TABLE: &str = "table";
+pub const MYSQL_FORMAT: &str = "format";
+pub const MYSQL_TLS_CA: &str = "tls_ca";
+pub const MYSQL_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const MYSQL_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const MYSQL_QUEUE_DIR: &str = "queue_dir";
+pub const MYSQL_QUEUE_LIMIT: &str = "queue_limit";
+pub const MYSQL_MAX_OPEN_CONNECTIONS: &str = "max_open_connections";
+
+pub const REDIS_URL: &str = "url";
+pub const REDIS_CHANNEL: &str = "channel";
+pub const REDIS_USERNAME: &str = "username";
+pub const REDIS_PASSWORD: &str = "password";
+pub const REDIS_KEEP_ALIVE_INTERVAL: &str = "keep_alive_interval";
+pub const REDIS_QUEUE_DIR: &str = "queue_dir";
+pub const REDIS_QUEUE_LIMIT: &str = "queue_limit";
+pub const REDIS_MAX_RETRY_ATTEMPTS: &str = "max_retry_attempts";
+pub const REDIS_RECONNECT_RETRY_ATTEMPTS: &str = "reconnect_retry_attempts";
+pub const REDIS_MIN_RETRY_DELAY: &str = "min_retry_delay";
+pub const REDIS_MAX_RETRY_DELAY: &str = "max_retry_delay";
+pub const REDIS_CONNECTION_TIMEOUT: &str = "connection_timeout";
+pub const REDIS_RESPONSE_TIMEOUT: &str = "response_timeout";
+pub const REDIS_PIPELINE_BUFFER_SIZE: &str = "pipeline_buffer_size";
+pub const REDIS_TLS_POLICY: &str = "tls_policy";
+pub const REDIS_TLS_CA: &str = "tls_ca";
+pub const REDIS_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const REDIS_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const REDIS_TLS_ALLOW_INSECURE: &str = "tls_allow_insecure";
+
+pub const POSTGRES_DSN_STRING: &str = BASE_DSN_STRING;
+pub const POSTGRES_TABLE: &str = "table";
+pub const POSTGRES_FORMAT: &str = "format";
+pub const POSTGRES_TLS_REQUIRED: &str = "tls_required";
+pub const POSTGRES_TLS_CA: &str = "tls_ca";
+pub const POSTGRES_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const POSTGRES_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const POSTGRES_QUEUE_DIR: &str = "queue_dir";
+pub const POSTGRES_QUEUE_LIMIT: &str = "queue_limit";
+
 /// Environment variable controlling whether target queue files are Snappy-compressed.
 /// Applies to both notify and audit target queue stores.
 pub const ENV_TARGET_STORE_COMPRESS: &str = "RUSTFS_TARGET_STORE_COMPRESS";

crates/config/src/notify/amqp.rs

@@ -0,0 +1,60 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const NOTIFY_AMQP_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::AMQP_URL,
+    crate::AMQP_EXCHANGE,
+    crate::AMQP_ROUTING_KEY,
+    crate::AMQP_MANDATORY,
+    crate::AMQP_PERSISTENT,
+    crate::AMQP_USERNAME,
+    crate::AMQP_PASSWORD,
+    crate::AMQP_TLS_CA,
+    crate::AMQP_TLS_CLIENT_CERT,
+    crate::AMQP_TLS_CLIENT_KEY,
+    crate::AMQP_QUEUE_DIR,
+    crate::AMQP_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_NOTIFY_AMQP_ENABLE: &str = "RUSTFS_NOTIFY_AMQP_ENABLE";
+pub const ENV_NOTIFY_AMQP_URL: &str = "RUSTFS_NOTIFY_AMQP_URL";
+pub const ENV_NOTIFY_AMQP_EXCHANGE: &str = "RUSTFS_NOTIFY_AMQP_EXCHANGE";
+pub const ENV_NOTIFY_AMQP_ROUTING_KEY: &str = "RUSTFS_NOTIFY_AMQP_ROUTING_KEY";
+pub const ENV_NOTIFY_AMQP_MANDATORY: &str = "RUSTFS_NOTIFY_AMQP_MANDATORY";
+pub const ENV_NOTIFY_AMQP_PERSISTENT: &str = "RUSTFS_NOTIFY_AMQP_PERSISTENT";
+pub const ENV_NOTIFY_AMQP_USERNAME: &str = "RUSTFS_NOTIFY_AMQP_USERNAME";
+pub const ENV_NOTIFY_AMQP_PASSWORD: &str = "RUSTFS_NOTIFY_AMQP_PASSWORD";
+pub const ENV_NOTIFY_AMQP_TLS_CA: &str = "RUSTFS_NOTIFY_AMQP_TLS_CA";
+pub const ENV_NOTIFY_AMQP_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_AMQP_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_AMQP_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_AMQP_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_AMQP_QUEUE_DIR: &str = "RUSTFS_NOTIFY_AMQP_QUEUE_DIR";
+pub const ENV_NOTIFY_AMQP_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_AMQP_QUEUE_LIMIT";
+
+pub const ENV_NOTIFY_AMQP_KEYS: &[&str; 13] = &[
+    ENV_NOTIFY_AMQP_ENABLE,
+    ENV_NOTIFY_AMQP_URL,
+    ENV_NOTIFY_AMQP_EXCHANGE,
+    ENV_NOTIFY_AMQP_ROUTING_KEY,
+    ENV_NOTIFY_AMQP_MANDATORY,
+    ENV_NOTIFY_AMQP_PERSISTENT,
+    ENV_NOTIFY_AMQP_USERNAME,
+    ENV_NOTIFY_AMQP_PASSWORD,
+    ENV_NOTIFY_AMQP_TLS_CA,
+    ENV_NOTIFY_AMQP_TLS_CLIENT_CERT,
+    ENV_NOTIFY_AMQP_TLS_CLIENT_KEY,
+    ENV_NOTIFY_AMQP_QUEUE_DIR,
+    ENV_NOTIFY_AMQP_QUEUE_LIMIT,
+];

crates/config/src/notify/mod.rs

@@ -12,19 +12,27 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+mod amqp;
 mod arn;
 mod kafka;
 mod mqtt;
+mod mysql;
 mod nats;
+mod postgres;
 mod pulsar;
+mod redis;
 mod store;
 mod webhook;
 
+pub use amqp::*;
 pub use arn::*;
 pub use kafka::*;
 pub use mqtt::*;
+pub use mysql::*;
 pub use nats::*;
+pub use postgres::*;
 pub use pulsar::*;
+pub use redis::*;
 pub use store::*;
 pub use webhook::*;
 
@@ -69,30 +77,31 @@ pub const ENV_NOTIFY_SEND_CONCURRENCY: &str = "RUSTFS_NOTIFY_SEND_CONCURRENCY";
 /// Adjust this value based on your system's capabilities and expected load.
 pub const DEFAULT_NOTIFY_SEND_CONCURRENCY: usize = 64;
 
-#[allow(dead_code)]
 pub const NOTIFY_SUB_SYSTEMS: &[&str] = &[
+    NOTIFY_AMQP_SUB_SYS,
     NOTIFY_KAFKA_SUB_SYS,
     NOTIFY_MQTT_SUB_SYS,
+    NOTIFY_MYSQL_SUB_SYS,
     NOTIFY_NATS_SUB_SYS,
+    NOTIFY_POSTGRES_SUB_SYS,
     NOTIFY_PULSAR_SUB_SYS,
+    NOTIFY_REDIS_SUB_SYS,
     NOTIFY_WEBHOOK_SUB_SYS,
 ];
 
 pub const NOTIFY_KAFKA_SUB_SYS: &str = "notify_kafka";
 pub const NOTIFY_MQTT_SUB_SYS: &str = "notify_mqtt";
-#[allow(dead_code)]
-pub const NOTIFY_MY_SQL_SUB_SYS: &str = "notify_mysql";
+pub const NOTIFY_MYSQL_SUB_SYS: &str = "notify_mysql";
 #[allow(dead_code)]
 pub const NOTIFY_NATS_SUB_SYS: &str = "notify_nats";
 #[allow(dead_code)]
 pub const NOTIFY_NSQ_SUB_SYS: &str = "notify_nsq";
 #[allow(dead_code)]
 pub const NOTIFY_ES_SUB_SYS: &str = "notify_elasticsearch";
-#[allow(dead_code)]
 pub const NOTIFY_AMQP_SUB_SYS: &str = "notify_amqp";
-#[allow(dead_code)]
 pub const NOTIFY_POSTGRES_SUB_SYS: &str = "notify_postgres";
 #[allow(dead_code)]
 pub const NOTIFY_REDIS_SUB_SYS: &str = "notify_redis";
+pub const NOTIFY_REDIS_DEFAULT_CHANNEL: &str = "rustfs_notify_channel";
 pub const NOTIFY_PULSAR_SUB_SYS: &str = "notify_pulsar";
 pub const NOTIFY_WEBHOOK_SUB_SYS: &str = "notify_webhook";

crates/config/src/notify/mysql.rs

@@ -0,0 +1,53 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// A list of all valid configuration keys for a MySQL target.
+pub const NOTIFY_MYSQL_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::MYSQL_DSN_STRING,
+    crate::MYSQL_TABLE,
+    crate::MYSQL_FORMAT,
+    crate::MYSQL_TLS_CA,
+    crate::MYSQL_TLS_CLIENT_CERT,
+    crate::MYSQL_TLS_CLIENT_KEY,
+    crate::MYSQL_QUEUE_DIR,
+    crate::MYSQL_QUEUE_LIMIT,
+    crate::MYSQL_MAX_OPEN_CONNECTIONS,
+    crate::COMMENT_KEY,
+];
+
+// MySQL Environment Variables
+pub const ENV_NOTIFY_MYSQL_ENABLE: &str = "RUSTFS_NOTIFY_MYSQL_ENABLE";
+pub const ENV_NOTIFY_MYSQL_DSN_STRING: &str = "RUSTFS_NOTIFY_MYSQL_DSN_STRING";
+pub const ENV_NOTIFY_MYSQL_TABLE: &str = "RUSTFS_NOTIFY_MYSQL_TABLE";
+pub const ENV_NOTIFY_MYSQL_FORMAT: &str = "RUSTFS_NOTIFY_MYSQL_FORMAT";
+pub const ENV_NOTIFY_MYSQL_TLS_CA: &str = "RUSTFS_NOTIFY_MYSQL_TLS_CA";
+pub const ENV_NOTIFY_MYSQL_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_MYSQL_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_MYSQL_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_MYSQL_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_MYSQL_QUEUE_DIR: &str = "RUSTFS_NOTIFY_MYSQL_QUEUE_DIR";
+pub const ENV_NOTIFY_MYSQL_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_MYSQL_QUEUE_LIMIT";
+pub const ENV_NOTIFY_MYSQL_MAX_OPEN_CONNECTIONS: &str = "RUSTFS_NOTIFY_MYSQL_MAX_OPEN_CONNECTIONS";
+
+pub const ENV_NOTIFY_MYSQL_KEYS: &[&str; 10] = &[
+    ENV_NOTIFY_MYSQL_ENABLE,
+    ENV_NOTIFY_MYSQL_DSN_STRING,
+    ENV_NOTIFY_MYSQL_TABLE,
+    ENV_NOTIFY_MYSQL_FORMAT,
+    ENV_NOTIFY_MYSQL_TLS_CA,
+    ENV_NOTIFY_MYSQL_TLS_CLIENT_CERT,
+    ENV_NOTIFY_MYSQL_TLS_CLIENT_KEY,
+    ENV_NOTIFY_MYSQL_QUEUE_DIR,
+    ENV_NOTIFY_MYSQL_QUEUE_LIMIT,
+    ENV_NOTIFY_MYSQL_MAX_OPEN_CONNECTIONS,
+];

crates/config/src/notify/postgres.rs

@@ -0,0 +1,51 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const NOTIFY_POSTGRES_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::POSTGRES_DSN_STRING,
+    crate::POSTGRES_TABLE,
+    crate::POSTGRES_FORMAT,
+    crate::POSTGRES_TLS_REQUIRED,
+    crate::POSTGRES_TLS_CA,
+    crate::POSTGRES_TLS_CLIENT_CERT,
+    crate::POSTGRES_TLS_CLIENT_KEY,
+    crate::POSTGRES_QUEUE_DIR,
+    crate::POSTGRES_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_NOTIFY_POSTGRES_ENABLE: &str = "RUSTFS_NOTIFY_POSTGRES_ENABLE";
+pub const ENV_NOTIFY_POSTGRES_DSN_STRING: &str = "RUSTFS_NOTIFY_POSTGRES_DSN_STRING";
+pub const ENV_NOTIFY_POSTGRES_TABLE: &str = "RUSTFS_NOTIFY_POSTGRES_TABLE";
+pub const ENV_NOTIFY_POSTGRES_FORMAT: &str = "RUSTFS_NOTIFY_POSTGRES_FORMAT";
+pub const ENV_NOTIFY_POSTGRES_TLS_REQUIRED: &str = "RUSTFS_NOTIFY_POSTGRES_TLS_REQUIRED";
+pub const ENV_NOTIFY_POSTGRES_TLS_CA: &str = "RUSTFS_NOTIFY_POSTGRES_TLS_CA";
+pub const ENV_NOTIFY_POSTGRES_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_POSTGRES_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_POSTGRES_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_POSTGRES_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_POSTGRES_QUEUE_DIR: &str = "RUSTFS_NOTIFY_POSTGRES_QUEUE_DIR";
+pub const ENV_NOTIFY_POSTGRES_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_POSTGRES_QUEUE_LIMIT";
+
+pub const ENV_NOTIFY_POSTGRES_KEYS: &[&str; 10] = &[
+    ENV_NOTIFY_POSTGRES_ENABLE,
+    ENV_NOTIFY_POSTGRES_DSN_STRING,
+    ENV_NOTIFY_POSTGRES_TABLE,
+    ENV_NOTIFY_POSTGRES_FORMAT,
+    ENV_NOTIFY_POSTGRES_TLS_REQUIRED,
+    ENV_NOTIFY_POSTGRES_TLS_CA,
+    ENV_NOTIFY_POSTGRES_TLS_CLIENT_CERT,
+    ENV_NOTIFY_POSTGRES_TLS_CLIENT_KEY,
+    ENV_NOTIFY_POSTGRES_QUEUE_DIR,
+    ENV_NOTIFY_POSTGRES_QUEUE_LIMIT,
+];

crates/config/src/notify/redis.rs

@@ -0,0 +1,82 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// A list of all valid configuration keys for a Redis target.
+pub const NOTIFY_REDIS_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::REDIS_URL,
+    crate::REDIS_CHANNEL,
+    crate::REDIS_USERNAME,
+    crate::REDIS_PASSWORD,
+    crate::REDIS_KEEP_ALIVE_INTERVAL,
+    crate::REDIS_QUEUE_DIR,
+    crate::REDIS_QUEUE_LIMIT,
+    crate::REDIS_MAX_RETRY_ATTEMPTS,
+    crate::REDIS_RECONNECT_RETRY_ATTEMPTS,
+    crate::REDIS_MIN_RETRY_DELAY,
+    crate::REDIS_MAX_RETRY_DELAY,
+    crate::REDIS_CONNECTION_TIMEOUT,
+    crate::REDIS_RESPONSE_TIMEOUT,
+    crate::REDIS_PIPELINE_BUFFER_SIZE,
+    crate::REDIS_TLS_POLICY,
+    crate::REDIS_TLS_CA,
+    crate::REDIS_TLS_CLIENT_CERT,
+    crate::REDIS_TLS_CLIENT_KEY,
+    crate::REDIS_TLS_ALLOW_INSECURE,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_NOTIFY_REDIS_ENABLE: &str = "RUSTFS_NOTIFY_REDIS_ENABLE";
+pub const ENV_NOTIFY_REDIS_URL: &str = "RUSTFS_NOTIFY_REDIS_URL";
+pub const ENV_NOTIFY_REDIS_CHANNEL: &str = "RUSTFS_NOTIFY_REDIS_CHANNEL";
+pub const ENV_NOTIFY_REDIS_USERNAME: &str = "RUSTFS_NOTIFY_REDIS_USERNAME";
+pub const ENV_NOTIFY_REDIS_PASSWORD: &str = "RUSTFS_NOTIFY_REDIS_PASSWORD";
+pub const ENV_NOTIFY_REDIS_KEEP_ALIVE_INTERVAL: &str = "RUSTFS_NOTIFY_REDIS_KEEP_ALIVE_INTERVAL";
+pub const ENV_NOTIFY_REDIS_QUEUE_DIR: &str = "RUSTFS_NOTIFY_REDIS_QUEUE_DIR";
+pub const ENV_NOTIFY_REDIS_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_REDIS_QUEUE_LIMIT";
+pub const ENV_NOTIFY_REDIS_MAX_RETRY_ATTEMPTS: &str = "RUSTFS_NOTIFY_REDIS_MAX_RETRY_ATTEMPTS";
+pub const ENV_NOTIFY_REDIS_RECONNECT_RETRY_ATTEMPTS: &str = "RUSTFS_NOTIFY_REDIS_RECONNECT_RETRY_ATTEMPTS";
+pub const ENV_NOTIFY_REDIS_MIN_RETRY_DELAY: &str = "RUSTFS_NOTIFY_REDIS_MIN_RETRY_DELAY";
+pub const ENV_NOTIFY_REDIS_MAX_RETRY_DELAY: &str = "RUSTFS_NOTIFY_REDIS_MAX_RETRY_DELAY";
+pub const ENV_NOTIFY_REDIS_CONNECTION_TIMEOUT: &str = "RUSTFS_NOTIFY_REDIS_CONNECTION_TIMEOUT";
+pub const ENV_NOTIFY_REDIS_RESPONSE_TIMEOUT: &str = "RUSTFS_NOTIFY_REDIS_RESPONSE_TIMEOUT";
+pub const ENV_NOTIFY_REDIS_PIPELINE_BUFFER_SIZE: &str = "RUSTFS_NOTIFY_REDIS_PIPELINE_BUFFER_SIZE";
+pub const ENV_NOTIFY_REDIS_TLS_POLICY: &str = "RUSTFS_NOTIFY_REDIS_TLS_POLICY";
+pub const ENV_NOTIFY_REDIS_TLS_CA: &str = "RUSTFS_NOTIFY_REDIS_TLS_CA";
+pub const ENV_NOTIFY_REDIS_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_REDIS_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_REDIS_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_REDIS_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_REDIS_TLS_ALLOW_INSECURE: &str = "RUSTFS_NOTIFY_REDIS_TLS_ALLOW_INSECURE";
+
+pub const ENV_NOTIFY_REDIS_KEYS: &[&str; 20] = &[
+    ENV_NOTIFY_REDIS_ENABLE,
+    ENV_NOTIFY_REDIS_URL,
+    ENV_NOTIFY_REDIS_CHANNEL,
+    ENV_NOTIFY_REDIS_USERNAME,
+    ENV_NOTIFY_REDIS_PASSWORD,
+    ENV_NOTIFY_REDIS_KEEP_ALIVE_INTERVAL,
+    ENV_NOTIFY_REDIS_QUEUE_DIR,
+    ENV_NOTIFY_REDIS_QUEUE_LIMIT,
+    ENV_NOTIFY_REDIS_MAX_RETRY_ATTEMPTS,
+    ENV_NOTIFY_REDIS_RECONNECT_RETRY_ATTEMPTS,
+    ENV_NOTIFY_REDIS_MIN_RETRY_DELAY,
+    ENV_NOTIFY_REDIS_MAX_RETRY_DELAY,
+    ENV_NOTIFY_REDIS_CONNECTION_TIMEOUT,
+    ENV_NOTIFY_REDIS_RESPONSE_TIMEOUT,
+    ENV_NOTIFY_REDIS_PIPELINE_BUFFER_SIZE,
+    ENV_NOTIFY_REDIS_TLS_POLICY,
+    ENV_NOTIFY_REDIS_TLS_CA,
+    ENV_NOTIFY_REDIS_TLS_CLIENT_CERT,
+    ENV_NOTIFY_REDIS_TLS_CLIENT_KEY,
+    ENV_NOTIFY_REDIS_TLS_ALLOW_INSECURE,
+];

crates/credentials/src/constants.rs

@@ -66,7 +66,7 @@ mod tests {
 
         // In production environment, access key and secret key should be different
         // These are default values, so being the same is acceptable, but should be warned in documentation
-        println!("Warning: Default access key and secret key are the same. Change them in production!");
+        assert_eq!(DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY);
     }
 
     #[test]
@@ -74,10 +74,8 @@ mod tests {
         // Test security best practices
 
         // These are default values, should be changed in production environments
-        println!("Security Warning: Default credentials detected!");
-        println!("Access Key: {DEFAULT_ACCESS_KEY}");
-        println!("Secret Key: {DEFAULT_SECRET_KEY}");
-        println!("These should be changed in production environments!");
+        assert_eq!(DEFAULT_ACCESS_KEY, "rustfsadmin");
+        assert_eq!(DEFAULT_SECRET_KEY, "rustfsadmin");
 
         // Verify that key lengths meet minimum security requirements
         assert!(DEFAULT_ACCESS_KEY.len() >= 8, "Access key should be at least 8 characters");

crates/credentials/src/credentials.rs

@@ -29,6 +29,12 @@ static GLOBAL_ACTIVE_CRED: OnceLock<Credentials> = OnceLock::new();
 /// Global RPC authentication token
 pub static GLOBAL_RUSTFS_RPC_SECRET: OnceLock<String> = OnceLock::new();
 
+/// Public error returned when RPC authentication is not safely configured.
+pub const RPC_SECRET_REQUIRED_MESSAGE: &str = "RPC authentication secret is not configured";
+
+/// Operator-facing guidance for configuring RPC authentication safely.
+pub const RPC_SECRET_REQUIRED_OPERATOR_MESSAGE: &str = "RUSTFS_RPC_SECRET must be set to a non-default value or RUSTFS_SECRET_KEY must be changed from the default for RPC authentication";
+
 /// Error type for credentials operations
 #[derive(Debug)]
 pub enum CredentialsError {
@@ -216,13 +222,39 @@ pub fn gen_secret_key(length: usize) -> std::io::Result<String> {
 /// # Returns
 /// * `String` - The RPC authentication token
 ///
+fn resolve_rpc_secret(env_secret: Option<&str>, global_secret: Option<&str>) -> Option<String> {
+    if let Some(secret) = env_secret.map(str::trim).filter(|secret| !secret.is_empty()) {
+        return (secret != DEFAULT_SECRET_KEY).then(|| secret.to_string());
+    }
+
+    global_secret
+        .map(str::trim)
+        .filter(|secret| !secret.is_empty() && *secret != DEFAULT_SECRET_KEY)
+        .map(ToOwned::to_owned)
+}
+
+pub fn try_get_rpc_token() -> std::io::Result<String> {
+    if let Some(secret) = GLOBAL_RUSTFS_RPC_SECRET.get() {
+        return resolve_rpc_secret(None, Some(secret)).ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE));
+    }
+
+    let env_secret = env::var(ENV_RPC_SECRET).ok();
+    let global_secret = get_global_secret_key_opt();
+    let secret = resolve_rpc_secret(env_secret.as_deref(), global_secret.as_deref())
+        .ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE))?;
+
+    match GLOBAL_RUSTFS_RPC_SECRET.set(secret.clone()) {
+        Ok(()) => Ok(secret),
+        Err(_) => GLOBAL_RUSTFS_RPC_SECRET
+            .get()
+            .and_then(|stored| resolve_rpc_secret(None, Some(stored)))
+            .ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE)),
+    }
+}
+
+#[deprecated(note = "use try_get_rpc_token to handle missing RPC secrets explicitly")]
 pub fn get_rpc_token() -> String {
-    GLOBAL_RUSTFS_RPC_SECRET
-        .get_or_init(|| {
-            env::var(ENV_RPC_SECRET)
-                .unwrap_or_else(|_| get_global_secret_key_opt().unwrap_or_else(|| DEFAULT_SECRET_KEY.to_string()))
-        })
-        .clone()
+    try_get_rpc_token().expect(RPC_SECRET_REQUIRED_MESSAGE)
 }
 
 /// A wrapper struct for masking sensitive strings in Debug implementations.
@@ -301,7 +333,7 @@ impl fmt::Debug for Credentials {
         f.debug_struct("Credentials")
             .field("access_key", &self.access_key)
             .field("secret_key", &Masked(Some(&self.secret_key)))
-            .field("session_token", &self.session_token)
+            .field("session_token", &Masked(Some(&self.session_token)))
             .field("expiration", &self.expiration)
             .field("status", &self.status)
             .field("parent_user", &self.parent_user)
@@ -495,6 +527,63 @@ mod tests {
         assert!(!key.contains('='));
     }
 
+    #[test]
+    fn test_resolve_rpc_secret_rejects_default_fallback() {
+        assert!(resolve_rpc_secret(None, None).is_none());
+        assert!(resolve_rpc_secret(None, Some(DEFAULT_SECRET_KEY)).is_none());
+        assert!(resolve_rpc_secret(Some(DEFAULT_SECRET_KEY), Some("custom-global-secret")).is_none());
+    }
+
+    #[test]
+    fn test_rpc_secret_public_error_omits_configuration_details() {
+        assert!(!RPC_SECRET_REQUIRED_MESSAGE.contains("RUSTFS_"));
+        assert!(!RPC_SECRET_REQUIRED_MESSAGE.contains(DEFAULT_SECRET_KEY));
+        assert!(RPC_SECRET_REQUIRED_OPERATOR_MESSAGE.contains("RUSTFS_RPC_SECRET"));
+    }
+
+    #[allow(deprecated)]
+    #[test]
+    fn test_get_rpc_token_preserves_string_return_type() {
+        fn assert_string_return(_: fn() -> String) {}
+
+        assert_string_return(get_rpc_token);
+    }
+
+    #[test]
+    fn test_resolve_rpc_secret_accepts_non_default_secret() {
+        assert_eq!(resolve_rpc_secret(Some("custom-rpc-secret"), None).as_deref(), Some("custom-rpc-secret"));
+        assert_eq!(
+            resolve_rpc_secret(None, Some("custom-global-secret")).as_deref(),
+            Some("custom-global-secret")
+        );
+    }
+
+    #[test]
+    fn test_resolve_rpc_secret_trims_and_falls_back_from_blank_env() {
+        assert_eq!(
+            resolve_rpc_secret(Some("  custom-rpc-secret  "), None).as_deref(),
+            Some("custom-rpc-secret")
+        );
+        assert_eq!(
+            resolve_rpc_secret(Some(""), Some("custom-global-secret")).as_deref(),
+            Some("custom-global-secret")
+        );
+        assert_eq!(
+            resolve_rpc_secret(Some("  "), Some("  custom-global-secret  ")).as_deref(),
+            Some("custom-global-secret")
+        );
+        assert_eq!(
+            resolve_rpc_secret(Some("  "), Some("custom-global-secret")).as_deref(),
+            Some("custom-global-secret")
+        );
+    }
+
+    #[test]
+    fn test_resolve_rpc_secret_returns_none_for_trimmed_default_secret() {
+        let padded_default_secret = format!("  {}  ", DEFAULT_SECRET_KEY);
+        assert!(resolve_rpc_secret(Some(padded_default_secret.as_str()), Some("custom-global-secret")).is_none());
+    }
+
     #[test]
     fn test_masked_debug() {
         // Test None
@@ -524,6 +613,24 @@ mod tests {
         assert_eq!(format!("{:?}", Masked(Some("中文测试"))), "中***试|4");
     }
 
+    #[test]
+    fn test_credentials_debug_masks_sensitive_fields() {
+        let cred = Credentials {
+            access_key: "debug-access-key".to_string(),
+            secret_key: "debug-secret-key".to_string(),
+            session_token: "debug-session-token".to_string(),
+            parent_user: "parent-user".to_string(),
+            ..Default::default()
+        };
+
+        let output = format!("{cred:?}");
+
+        assert!(output.contains("debug-access-key"));
+        assert!(output.contains("parent-user"));
+        assert!(!output.contains("debug-secret-key"));
+        assert!(!output.contains("debug-session-token"));
+    }
+
     #[test]
     fn test_credentials_expiration_serialize_as_rfc3339() {
         use time::OffsetDateTime;

crates/crypto/Cargo.toml

@@ -31,7 +31,6 @@ workspace = true
 [dependencies]
 aes-gcm = { workspace = true, optional = true }
 argon2 = { workspace = true, optional = true }
-cfg-if = { workspace = true }
 chacha20poly1305 = { workspace = true, optional = true }
 jsonwebtoken = { workspace = true }
 pbkdf2 = { workspace = true, optional = true }

crates/crypto/src/encdec/aes.rs

@@ -13,19 +13,23 @@
 // limitations under the License.
 
 pub fn native_aes() -> bool {
-    cfg_if::cfg_if! {
-        if #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] {
+    cfg_select! {
+        any(target_arch = "x86", target_arch = "x86_64") => {
             std::is_x86_feature_detected!("aes") && std::is_x86_feature_detected!("pclmulqdq")
-        } else if #[cfg(target_arch = "aarch64")] {
+        }
+        target_arch = "aarch64" => {
             std::arch::is_aarch64_feature_detected!("aes")
-        } else if #[cfg(target_arch = "powerpc64")] {
+        }
+        target_arch = "powerpc64" => {
             false
-        } else if #[cfg(target_arch = "s390x")] {
+        }
+        target_arch = "s390x" => {
             std::is_s390x_feature_detected!("aes")
                 && std::is_s390x_feature_detected!("aescbc")
                 && std::is_s390x_feature_detected!("aesctr")
                 && (std::is_s390x_feature_detected!("aesgcm") || std::is_s390x_feature_detected!("ghash"))
-        } else {
+        }
+        _ => {
             false
         }
     }

crates/e2e_test/Cargo.toml

@@ -26,8 +26,10 @@ workspace = true
 [features]
 default = []
 ftps = []
+sftp = []
 
 [dependencies]
+rustfs-config = { workspace = true, features = ["constants"] }
 rustfs-ecstore.workspace = true
 rustfs-common.workspace = true
 rustfs-rio.workspace = true
@@ -72,5 +74,7 @@ suppaftp = { workspace = true, features = ["tokio", "rustls-aws-lc-rs"] }
 rcgen.workspace = true
 anyhow.workspace = true
 rustls.workspace = true
+russh = { workspace = true }
+russh-sftp = { workspace = true }
 zip.workspace = true
 clap.workspace = true

crates/e2e_test/src/common.rs

@@ -41,7 +41,9 @@ use walkdir::WalkDir;
 // Common constants for all E2E tests
 pub const DEFAULT_ACCESS_KEY: &str = "rustfsadmin";
 pub const DEFAULT_SECRET_KEY: &str = "rustfsadmin";
+pub const ENV_RUSTFS_BUILD_FEATURES: &str = "RUSTFS_BUILD_FEATURES";
 pub const TEST_BUCKET: &str = "e2e-test-bucket";
+const RUSTFS_FULL_FEATURE: &str = "full";
 
 fn build_test_s3_config(endpoint_url: &str, access_key: &str, secret_key: &str, provider_name: &'static str) -> Config {
     let credentials = Credentials::new(access_key, secret_key, None, None, provider_name);
@@ -83,6 +85,7 @@ pub fn rustfs_binary_path_with_features(requested_features: Option<&str>) -> Pat
     if let Some(path) = std::env::var_os("CARGO_BIN_EXE_rustfs") {
         return PathBuf::from(path);
     }
+    let requested_features = requested_features.and_then(normalize_rustfs_build_features);
 
     let mut binary_path = workspace_root();
     binary_path.push("target");
@@ -90,7 +93,7 @@ pub fn rustfs_binary_path_with_features(requested_features: Option<&str>) -> Pat
     binary_path.push(profile_dir);
     binary_path.push(format!("rustfs{}", std::env::consts::EXE_SUFFIX));
 
-    let features_match = binary_features_match(&binary_path, requested_features);
+    let features_match = binary_features_match(&binary_path, requested_features.as_deref());
     let source_is_newer = workspace_sources_newer_than_binary(&binary_path);
     let can_reuse_inside_e2e = running_inside_e2e_test_binary() && requested_features.is_none() && features_match;
     if binary_path.is_file() && features_match && (!source_is_newer || can_reuse_inside_e2e) {
@@ -105,7 +108,7 @@ pub fn rustfs_binary_path_with_features(requested_features: Option<&str>) -> Pat
     }
 
     info!("Building RustFS binary to ensure it's up to date...");
-    build_rustfs_binary(requested_features);
+    build_rustfs_binary(requested_features.as_deref());
 
     info!("Using RustFS binary at {:?}", binary_path);
     binary_path
@@ -134,11 +137,31 @@ fn running_inside_e2e_test_binary() -> bool {
     std::env::var("CARGO_PKG_NAME").is_ok_and(|value| value == "e2e_test")
 }
 
-fn requested_rustfs_build_features() -> Option<String> {
-    std::env::var("RUSTFS_BUILD_FEATURES")
+pub fn requested_rustfs_build_features() -> Option<String> {
+    std::env::var(ENV_RUSTFS_BUILD_FEATURES)
         .ok()
-        .map(|value| value.trim().to_string())
-        .filter(|value| !value.is_empty())
+        .and_then(|value| normalize_rustfs_build_features(&value))
+}
+
+pub fn normalize_rustfs_build_features(features: &str) -> Option<String> {
+    let features = features
+        .split(',')
+        .map(str::trim)
+        .filter(|feature| !feature.is_empty())
+        .map(str::to_ascii_lowercase)
+        .collect::<Vec<_>>();
+
+    if features.is_empty() { None } else { Some(features.join(",")) }
+}
+
+pub fn rustfs_build_feature_enabled(requested_features: Option<&str>, required_feature: &str) -> bool {
+    let Some(requested_features) = requested_features.and_then(normalize_rustfs_build_features) else {
+        return true;
+    };
+
+    requested_features
+        .split(',')
+        .any(|feature| feature.eq_ignore_ascii_case(RUSTFS_FULL_FEATURE) || feature.eq_ignore_ascii_case(required_feature))
 }
 
 fn rustfs_binary_features_stamp_path(binary_path: &Path) -> PathBuf {
@@ -147,11 +170,14 @@ fn rustfs_binary_features_stamp_path(binary_path: &Path) -> PathBuf {
 
 fn binary_features_match(binary_path: &Path, requested_features: Option<&str>) -> bool {
     let stamp_path = rustfs_binary_features_stamp_path(binary_path);
-    let recorded = stdfs::read_to_string(stamp_path).ok().map(|value| value.trim().to_string());
+    let recorded = stdfs::read_to_string(stamp_path)
+        .ok()
+        .and_then(|value| normalize_rustfs_build_features(&value));
+    let requested = requested_features.and_then(normalize_rustfs_build_features);
 
-    match requested_features {
+    match requested.as_deref() {
         Some(features) => recorded.as_deref() == Some(features),
-        None => recorded.as_deref().is_none_or(str::is_empty),
+        None => recorded.is_none(),
     }
 }
 
@@ -351,6 +377,7 @@ impl RustFSTestEnvironment {
     async fn start_rustfs_server_inner(
         &mut self,
         extra_args: Vec<&str>,
+        extra_env: &[(&str, &str)],
         cleanup_existing: bool,
     ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         if cleanup_existing {
@@ -362,10 +389,12 @@ impl RustFSTestEnvironment {
         info!("Starting RustFS server with args: {:?}", args);
 
         let binary_path = rustfs_binary_path();
-        let process = Command::new(&binary_path)
-            .env("RUST_LOG", "rustfs=info,rustfs_notify=debug")
-            .args(&args)
-            .spawn()?;
+        let mut command = Command::new(&binary_path);
+        command.env("RUST_LOG", "rustfs=info,rustfs_notify=debug");
+        for (key, value) in extra_env {
+            command.env(key, value);
+        }
+        let process = command.args(&args).spawn()?;
 
         self.process = Some(process);
 
@@ -377,7 +406,16 @@ impl RustFSTestEnvironment {
 
     /// Start RustFS server with basic configuration
     pub async fn start_rustfs_server(&mut self, extra_args: Vec<&str>) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        self.start_rustfs_server_inner(extra_args, true).await
+        self.start_rustfs_server_inner(extra_args, &[], true).await
+    }
+
+    /// Start RustFS server with extra child-process environment variables.
+    pub async fn start_rustfs_server_with_env(
+        &mut self,
+        extra_args: Vec<&str>,
+        extra_env: &[(&str, &str)],
+    ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+        self.start_rustfs_server_inner(extra_args, extra_env, true).await
     }
 
     /// Start RustFS server without cleaning up other running RustFS processes.
@@ -388,7 +426,7 @@ impl RustFSTestEnvironment {
         &mut self,
         extra_args: Vec<&str>,
     ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        self.start_rustfs_server_inner(extra_args, false).await
+        self.start_rustfs_server_inner(extra_args, &[], false).await
     }
 
     /// Wait for RustFS server to be ready.
@@ -888,3 +926,36 @@ impl Drop for RustFSTestClusterEnvironment {
         }
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn normalizes_rustfs_build_features() {
+        assert_eq!(
+            normalize_rustfs_build_features(" SFTP, ftps ,, WebDAV "),
+            Some("sftp,ftps,webdav".to_string())
+        );
+        assert_eq!(normalize_rustfs_build_features(" , "), None);
+    }
+
+    #[test]
+    fn full_feature_enables_any_required_feature() {
+        assert!(rustfs_build_feature_enabled(Some("full"), "sftp"));
+        assert!(rustfs_build_feature_enabled(Some("ftps, full"), "webdav"));
+    }
+
+    #[test]
+    fn binary_feature_stamp_matching_uses_normalized_features() {
+        let binary_path = std::env::temp_dir().join(format!("rustfs-feature-stamp-test-{}", Uuid::new_v4()));
+        let stamp_path = rustfs_binary_features_stamp_path(&binary_path);
+
+        stdfs::write(&stamp_path, " SFTP, ftps ").expect("write feature stamp");
+        assert!(binary_features_match(&binary_path, Some("sftp,ftps")));
+        assert!(binary_features_match(&binary_path, Some(" SFTP, FTPS ")));
+        assert!(!binary_features_match(&binary_path, Some("sftp")));
+
+        stdfs::remove_file(stamp_path).ok();
+    }
+}

crates/e2e_test/src/copy_object_metadata_test.rs

@@ -0,0 +1,144 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! CopyObject metadata replacement regression tests.
+
+#[cfg(test)]
+mod tests {
+    use crate::common::{RustFSTestEnvironment, init_logging};
+    use aws_sdk_s3::primitives::ByteStream;
+    use aws_sdk_s3::types::MetadataDirective;
+    use serial_test::serial;
+    use tracing::info;
+
+    #[tokio::test]
+    #[serial]
+    async fn test_self_copy_replace_metadata_preserves_readable_object() {
+        init_logging();
+        info!("Issue #2789: self-copy metadata replacement must preserve object data");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = env.create_s3_client();
+        let bucket = "self-copy-metadata-replace-test";
+        let key = "assets/chunk-2F3R7JUG.js";
+        let content = b"console.log('metadata replacement should keep object data readable');";
+
+        client
+            .create_bucket()
+            .bucket(bucket)
+            .send()
+            .await
+            .expect("Failed to create bucket");
+
+        client
+            .put_object()
+            .bucket(bucket)
+            .key(key)
+            .content_type("text/javascript; charset=utf-8")
+            .metadata("mtime", "1777992333")
+            .metadata("stale", "must-be-removed")
+            .body(ByteStream::from_static(content))
+            .send()
+            .await
+            .expect("PUT failed");
+
+        client
+            .copy_object()
+            .bucket(bucket)
+            .key(key)
+            .copy_source(format!("{bucket}/{key}"))
+            .metadata_directive(MetadataDirective::Replace)
+            .content_type("text/javascript; charset=utf-8")
+            .metadata("mtime", "1777992348")
+            .send()
+            .await
+            .expect("self CopyObject with metadata replacement failed");
+
+        let head_resp = client
+            .head_object()
+            .bucket(bucket)
+            .key(key)
+            .send()
+            .await
+            .expect("HEAD failed after self-copy");
+        assert_eq!(head_resp.content_length(), Some(content.len() as i64));
+        assert_eq!(
+            head_resp.metadata().and_then(|metadata| metadata.get("mtime")),
+            Some(&"1777992348".to_string()),
+            "HEAD should return replaced metadata"
+        );
+        assert_eq!(
+            head_resp.metadata().and_then(|metadata| metadata.get("stale")),
+            None,
+            "HEAD should not return metadata omitted by REPLACE"
+        );
+
+        let get_resp = client
+            .get_object()
+            .bucket(bucket)
+            .key(key)
+            .send()
+            .await
+            .expect("GET failed after self-copy");
+        let body = get_resp
+            .body
+            .collect()
+            .await
+            .expect("Failed to collect GET body")
+            .into_bytes();
+        assert_eq!(body.as_ref(), content, "self-copy metadata replacement must not drop object data");
+
+        client
+            .copy_object()
+            .bucket(bucket)
+            .key(key)
+            .copy_source(format!("{bucket}/{key}"))
+            .metadata_directive(MetadataDirective::Replace)
+            .send()
+            .await
+            .expect("self CopyObject with empty metadata replacement failed");
+
+        let empty_head_resp = client
+            .head_object()
+            .bucket(bucket)
+            .key(key)
+            .send()
+            .await
+            .expect("HEAD failed after empty metadata replacement");
+        assert_eq!(
+            empty_head_resp.metadata().and_then(|metadata| metadata.get("mtime")),
+            None,
+            "HEAD should not return metadata omitted by empty REPLACE"
+        );
+
+        let empty_get_resp = client
+            .get_object()
+            .bucket(bucket)
+            .key(key)
+            .send()
+            .await
+            .expect("GET failed after empty metadata replacement");
+        let empty_body = empty_get_resp
+            .body
+            .collect()
+            .await
+            .expect("Failed to collect GET body after empty metadata replacement")
+            .into_bytes();
+        assert_eq!(empty_body.as_ref(), content, "empty metadata replacement must not drop object data");
+
+        env.stop_server();
+    }
+}

crates/e2e_test/src/delete_object_no_content_length_test.rs

@@ -0,0 +1,165 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Regression coverage for signed `DELETE Object?versionId` requests sent with
+//! no body and no `Content-Length` header.
+
+#[cfg(test)]
+mod tests {
+    use crate::common::{RustFSTestEnvironment, init_logging};
+    use aws_sdk_s3::primitives::ByteStream;
+    use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
+    use http::header::HOST;
+    use rustfs_signer::constants::UNSIGNED_PAYLOAD;
+    use rustfs_signer::sign_v4;
+    use s3s::Body;
+    use serial_test::serial;
+    use std::error::Error;
+    use tokio::io::{AsyncReadExt, AsyncWriteExt};
+    use tokio::net::TcpStream;
+    use tokio::time::{Duration, timeout};
+    use tracing::info;
+
+    const RAW_RESPONSE_TIMEOUT: Duration = Duration::from_secs(10);
+
+    fn parse_status(raw_response: &str) -> Option<u16> {
+        raw_response.lines().next()?.split_whitespace().nth(1)?.parse().ok()
+    }
+
+    fn response_body(raw_response: &str) -> &str {
+        raw_response
+            .split_once("\r\n\r\n")
+            .map(|(_, body)| body)
+            .or_else(|| raw_response.split_once("\n\n").map(|(_, body)| body))
+            .unwrap_or("")
+    }
+
+    async fn signed_delete_without_content_length(
+        url: &str,
+        access_key: &str,
+        secret_key: &str,
+    ) -> Result<String, Box<dyn Error + Send + Sync>> {
+        let uri = url.parse::<http::Uri>()?;
+        let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
+        let path_and_query = uri.path_and_query().ok_or("request URL missing path")?.as_str().to_string();
+
+        let request = http::Request::builder()
+            .method(http::Method::DELETE)
+            .uri(uri)
+            .header(HOST, authority.clone())
+            .header("x-amz-content-sha256", UNSIGNED_PAYLOAD)
+            .body(Body::empty())?;
+
+        let signed = sign_v4(request, 0, access_key, secret_key, "", "us-east-1");
+
+        let mut raw_request = format!("DELETE {path_and_query} HTTP/1.1\r\nHost: {authority}\r\nConnection: close\r\n");
+        for (name, value) in signed.headers() {
+            if name.as_str().eq_ignore_ascii_case("host") || name.as_str().eq_ignore_ascii_case("content-length") {
+                continue;
+            }
+            raw_request.push_str(name.as_str());
+            raw_request.push_str(": ");
+            raw_request.push_str(value.to_str()?);
+            raw_request.push_str("\r\n");
+        }
+        raw_request.push_str("\r\n");
+
+        assert!(
+            !raw_request.to_ascii_lowercase().contains("\r\ncontent-length:"),
+            "raw regression request must omit Content-Length; request was:\n{raw_request}"
+        );
+
+        let mut stream = TcpStream::connect(&authority).await?;
+        stream.write_all(raw_request.as_bytes()).await?;
+        stream.flush().await?;
+
+        let mut response = Vec::new();
+        timeout(RAW_RESPONSE_TIMEOUT, stream.read_to_end(&mut response))
+            .await
+            .map_err(|_| std::io::Error::new(std::io::ErrorKind::TimedOut, "timed out reading raw DELETE response"))??;
+        Ok(String::from_utf8_lossy(&response).into_owned())
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn test_delete_object_version_without_content_length_succeeds() -> Result<(), Box<dyn Error + Send + Sync>> {
+        init_logging();
+        info!("🧪 TEST: signed DELETE Object?versionId succeeds without Content-Length");
+
+        let mut env = RustFSTestEnvironment::new().await?;
+        env.start_rustfs_server(vec![]).await?;
+
+        let client = env.create_s3_client();
+        let bucket = "delete-version-no-content-length";
+        let key = "versioned-delete-target.txt";
+
+        client.create_bucket().bucket(bucket).send().await?;
+
+        client
+            .put_object()
+            .bucket(bucket)
+            .key(key)
+            .body(ByteStream::from_static(b"delete me after versioning is enabled"))
+            .send()
+            .await?;
+
+        client
+            .put_bucket_versioning()
+            .bucket(bucket)
+            .versioning_configuration(
+                VersioningConfiguration::builder()
+                    .status(BucketVersioningStatus::Enabled)
+                    .build(),
+            )
+            .send()
+            .await?;
+
+        let listed_versions = client.list_object_versions().bucket(bucket).prefix(key).send().await?;
+        let version_id = listed_versions
+            .versions()
+            .iter()
+            .find(|version| version.key() == Some(key))
+            .and_then(|version| version.version_id())
+            .ok_or("ListObjectVersions did not return the pre-versioning object version")?;
+
+        let url = format!("{}/{}/{}?versionId={}", env.url, bucket, key, urlencoding::encode(version_id));
+        let raw_response = signed_delete_without_content_length(&url, &env.access_key, &env.secret_key).await?;
+        info!("raw DELETE response:\n{}", raw_response);
+
+        assert_eq!(
+            parse_status(&raw_response),
+            Some(204),
+            "DELETE Object?versionId without Content-Length should succeed, got:\n{raw_response}"
+        );
+        assert!(
+            !raw_response.contains("MissingContentLength"),
+            "DELETE Object?versionId without Content-Length regressed to MissingContentLength: {raw_response}"
+        );
+        assert!(
+            response_body(&raw_response).trim().is_empty(),
+            "successful DELETE Object?versionId should not return an error body: {raw_response}"
+        );
+
+        let get_deleted_version = client
+            .get_object()
+            .bucket(bucket)
+            .key(key)
+            .version_id(version_id)
+            .send()
+            .await;
+        assert!(get_deleted_version.is_err(), "explicitly deleted version should no longer be readable");
+
+        Ok(())
+    }
+}

crates/e2e_test/src/existing_object_tag_policy_test.rs

@@ -21,7 +21,7 @@ use crate::common::{
 };
 use aws_sdk_s3::config::{Credentials, Region};
 use aws_sdk_s3::primitives::ByteStream;
-use aws_sdk_s3::types::{Tag, Tagging};
+use aws_sdk_s3::types::{Delete, ObjectIdentifier, Tag, Tagging};
 use aws_sdk_s3::{Client, Config};
 use serial_test::serial;
 use tracing::info;
@@ -318,11 +318,18 @@ async fn test_e2e_sts_assume_role_session_policy_existing_object_tag() -> Result
 
     let rw = serde_json::to_string(&serde_json::json!({
         "Version": "2012-10-17",
-        "Statement": [{
-            "Effect": "Allow",
-            "Action": ["s3:*"],
-            "Resource": ["arn:aws:s3:::*"]
-        }]
+        "Statement": [
+            {
+                "Effect": "Allow",
+                "Action": ["s3:*"],
+                "Resource": ["arn:aws:s3:::*"]
+            },
+            {
+                "Effect": "Allow",
+                "Action": ["sts:AssumeRole"],
+                "Resource": ["arn:aws:s3:::*"]
+            }
+        ]
     }))?;
     admin_add_canned_policy(&env, &policy_readwrite, &rw).await?;
     admin_attach_policy_to_user(&env, &policy_readwrite, &parent).await?;
@@ -362,3 +369,114 @@ async fn test_e2e_sts_assume_role_session_policy_existing_object_tag() -> Result
     info!("test_e2e_sts_assume_role_session_policy_existing_object_tag passed");
     Ok(())
 }
+
+/// STS inline session policy: DeleteObjects must evaluate `s3:DeleteObject` per requested object key.
+#[tokio::test]
+#[serial]
+async fn test_e2e_sts_session_policy_delete_objects_object_prefix_only() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    init_logging();
+    if !awscurl_available() {
+        info!("Skipping test_e2e_sts_session_policy_delete_objects_object_prefix_only: awscurl not available");
+        return Ok(());
+    }
+
+    let suffix = Uuid::new_v4();
+    let parent = format!("e2e-sts-del-par-{suffix}");
+    let parent_secret = "longSecretKeyForParentDelete99!";
+    let policy_readwrite = format!("e2e-sts-del-rw-{suffix}");
+    let bucket = format!("e2e-sts-del-bkt-{suffix}");
+    let allowed_key = "allowed/table/data.parquet";
+    let denied_key = "denied/table/data.parquet";
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+
+    let admin = env.create_s3_client();
+    admin_create_user(&env, &parent, parent_secret).await?;
+
+    let rw = serde_json::to_string(&serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [
+            {
+                "Effect": "Allow",
+                "Action": ["s3:*"],
+                "Resource": ["arn:aws:s3:::*"]
+            },
+            {
+                "Effect": "Allow",
+                "Action": ["sts:AssumeRole"],
+                "Resource": ["arn:aws:s3:::*"]
+            }
+        ]
+    }))?;
+    admin_add_canned_policy(&env, &policy_readwrite, &rw).await?;
+    admin_attach_policy_to_user(&env, &policy_readwrite, &parent).await?;
+
+    let parent_client = user_client(&env, &parent, parent_secret);
+    parent_client.create_bucket().bucket(&bucket).send().await?;
+    parent_client
+        .put_object()
+        .bucket(&bucket)
+        .key(allowed_key)
+        .body(ByteStream::from_static(b"allowed-delete-data"))
+        .send()
+        .await?;
+    parent_client
+        .put_object()
+        .bucket(&bucket)
+        .key(denied_key)
+        .body(ByteStream::from_static(b"denied-delete-data"))
+        .send()
+        .await?;
+
+    let session_policy = serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [{
+            "Effect": "Allow",
+            "Action": ["s3:DeleteObject"],
+            "Resource": [format!("arn:aws:s3:::{}/allowed/*", bucket)]
+        }]
+    })
+    .to_string();
+
+    let (ak, sk, token) = assume_role_with_session_policy(&env, &parent, parent_secret, &session_policy).await?;
+    let session_client = sts_session_client(&env, &ak, &sk, &token);
+
+    let delete = Delete::builder()
+        .objects(ObjectIdentifier::builder().key(allowed_key).build()?)
+        .objects(ObjectIdentifier::builder().key(denied_key).build()?)
+        .build()?;
+
+    let result = session_client.delete_objects().bucket(&bucket).delete(delete).send().await?;
+
+    assert_eq!(result.deleted().len(), 1, "only the allowed-prefix object should be deleted");
+    assert!(
+        result.deleted().iter().any(|deleted| deleted.key() == Some(allowed_key)),
+        "DeleteObjects response should report the allowed-prefix object as deleted"
+    );
+
+    assert_eq!(result.errors().len(), 1, "the out-of-prefix object should return one per-key error");
+    let error = &result.errors()[0];
+    assert_eq!(error.key(), Some(denied_key));
+    assert_eq!(error.code(), Some("AccessDenied"));
+
+    let allowed_head = parent_client.head_object().bucket(&bucket).key(allowed_key).send().await;
+    assert!(allowed_head.is_err(), "allowed-prefix object should have been deleted");
+
+    parent_client
+        .head_object()
+        .bucket(&bucket)
+        .key(denied_key)
+        .send()
+        .await
+        .expect("out-of-prefix object should remain after per-key AccessDenied");
+
+    let _ = admin.delete_object().bucket(&bucket).key(allowed_key).send().await;
+    let _ = admin.delete_object().bucket(&bucket).key(denied_key).send().await;
+    let _ = admin.delete_bucket().bucket(&bucket).send().await;
+    admin_remove_user(&env, &parent).await;
+    admin_remove_policy(&env, &policy_readwrite).await;
+
+    info!("test_e2e_sts_session_policy_delete_objects_object_prefix_only passed");
+    Ok(())
+}

crates/e2e_test/src/head_object_consistency_test.rs

@@ -0,0 +1,131 @@
+// Copyright 2026 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
+use aws_sdk_s3::presigning::PresigningConfig;
+use aws_sdk_s3::primitives::ByteStream;
+use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
+use serial_test::serial;
+use std::time::Duration;
+use tracing::info;
+
+const CONSISTENCY_BUCKET: &str = "head-consistency-bucket";
+const PUT_KEY: &str = "consistency-put-object.txt";
+const MPU_KEY: &str = "consistency-multipart-object.txt";
+
+fn list_contains_key(output: &aws_sdk_s3::operation::list_objects_v2::ListObjectsV2Output, key: &str) -> bool {
+    output.contents().iter().any(|obj| obj.key().is_some_and(|k| k == key))
+}
+
+#[tokio::test]
+#[serial]
+async fn head_object_consistency_after_write_and_multipart_and_presigned_head()
+-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    init_logging();
+    info!("Starting HeadObject consistency regression test");
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(Vec::new()).await?;
+
+    let client = env.create_s3_client();
+    env.create_test_bucket(CONSISTENCY_BUCKET).await?;
+
+    client
+        .put_object()
+        .bucket(CONSISTENCY_BUCKET)
+        .key(PUT_KEY)
+        .body(ByteStream::from_static(b"head-consistency-put-body"))
+        .send()
+        .await?;
+
+    client.get_object().bucket(CONSISTENCY_BUCKET).key(PUT_KEY).send().await?;
+    let put_list = client
+        .list_objects_v2()
+        .bucket(CONSISTENCY_BUCKET)
+        .prefix(PUT_KEY)
+        .send()
+        .await?;
+    assert!(list_contains_key(&put_list, PUT_KEY), "ListObjectsV2 should include the PutObject key");
+
+    client.head_object().bucket(CONSISTENCY_BUCKET).key(PUT_KEY).send().await?;
+
+    let create = client
+        .create_multipart_upload()
+        .bucket(CONSISTENCY_BUCKET)
+        .key(MPU_KEY)
+        .send()
+        .await?;
+    let upload_id = create.upload_id().ok_or("missing multipart upload id")?.to_string();
+
+    let part1 = client
+        .upload_part()
+        .bucket(CONSISTENCY_BUCKET)
+        .key(MPU_KEY)
+        .upload_id(&upload_id)
+        .part_number(1)
+        .body(ByteStream::from_static(b"head-consistency-multipart-part-1"))
+        .send()
+        .await?;
+    let completed = CompletedMultipartUpload::builder()
+        .parts(
+            CompletedPart::builder()
+                .part_number(1)
+                .set_e_tag(part1.e_tag().map(str::to_string))
+                .build(),
+        )
+        .build();
+
+    client
+        .complete_multipart_upload()
+        .bucket(CONSISTENCY_BUCKET)
+        .key(MPU_KEY)
+        .upload_id(&upload_id)
+        .multipart_upload(completed)
+        .send()
+        .await?;
+
+    client.get_object().bucket(CONSISTENCY_BUCKET).key(MPU_KEY).send().await?;
+    let mpu_list = client
+        .list_objects_v2()
+        .bucket(CONSISTENCY_BUCKET)
+        .prefix(MPU_KEY)
+        .send()
+        .await?;
+    assert!(
+        list_contains_key(&mpu_list, MPU_KEY),
+        "ListObjectsV2 should include the completed multipart key"
+    );
+
+    client.head_object().bucket(CONSISTENCY_BUCKET).key(MPU_KEY).send().await?;
+
+    let presigned = client
+        .head_object()
+        .bucket(CONSISTENCY_BUCKET)
+        .key(PUT_KEY)
+        .presigned(PresigningConfig::expires_in(Duration::from_secs(300))?)
+        .await?;
+    let presigned_resp = local_http_client().head(presigned.uri().to_string()).send().await?;
+    assert!(
+        presigned_resp.status().is_success(),
+        "Presigned HEAD should succeed, got status {}",
+        presigned_resp.status()
+    );
+
+    client.delete_object().bucket(CONSISTENCY_BUCKET).key(PUT_KEY).send().await?;
+    client.delete_object().bucket(CONSISTENCY_BUCKET).key(MPU_KEY).send().await?;
+    env.delete_test_bucket(CONSISTENCY_BUCKET).await?;
+    env.stop_server();
+
+    Ok(())
+}

crates/e2e_test/src/head_object_range_test.rs

@@ -0,0 +1,52 @@
+use crate::common::{RustFSTestEnvironment, init_logging};
+use aws_sdk_s3::primitives::ByteStream;
+use serial_test::serial;
+use tracing::info;
+
+const RANGE_HEAD_BUCKET: &str = "range-head-test-bucket";
+const RANGE_HEAD_KEY: &str = "range-head-object.bin";
+const ACCEPT_RANGES_BYTES: &str = "bytes";
+
+#[tokio::test]
+#[serial]
+async fn head_object_advertises_accept_ranges() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    init_logging();
+    info!("Starting HeadObject Accept-Ranges regression test");
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(Vec::new()).await?;
+
+    let client = env.create_s3_client();
+    env.create_test_bucket(RANGE_HEAD_BUCKET).await?;
+
+    client
+        .put_object()
+        .bucket(RANGE_HEAD_BUCKET)
+        .key(RANGE_HEAD_KEY)
+        .body(ByteStream::from_static(b"0123456789abcdef"))
+        .send()
+        .await?;
+
+    let head = client
+        .head_object()
+        .bucket(RANGE_HEAD_BUCKET)
+        .key(RANGE_HEAD_KEY)
+        .send()
+        .await?;
+    assert_eq!(
+        head.accept_ranges(),
+        Some(ACCEPT_RANGES_BYTES),
+        "HeadObject should advertise byte range support"
+    );
+
+    client
+        .delete_object()
+        .bucket(RANGE_HEAD_BUCKET)
+        .key(RANGE_HEAD_KEY)
+        .send()
+        .await?;
+    env.delete_test_bucket(RANGE_HEAD_BUCKET).await?;
+    env.stop_server();
+
+    Ok(())
+}

crates/e2e_test/src/lib.rs

@@ -74,6 +74,10 @@ mod compression_test;
 #[cfg(test)]
 mod delete_objects_versioning_test;
 
+// Regression test for signed DELETE Object?versionId requests without Content-Length.
+#[cfg(test)]
+mod delete_object_no_content_length_test;
+
 // Regression test for Issue #2252: ListObjectVersions misses newest version after put -> delete -> put
 #[cfg(test)]
 mod list_object_versions_regression_test;
@@ -104,6 +108,15 @@ mod checksum_upload_test;
 #[cfg(test)]
 mod group_delete_test;
 
+#[cfg(test)]
+mod head_object_range_test;
+
+#[cfg(test)]
+mod head_object_consistency_test;
+
+#[cfg(test)]
+mod copy_object_metadata_test;
+
 // S3 dummy-compat bucket API tests
 #[cfg(test)]
 mod bucket_logging_test;

crates/e2e_test/src/list_objects_v2_pagination_test.rs

@@ -31,6 +31,7 @@ mod tests {
     use aws_sdk_s3::Client;
     use aws_sdk_s3::primitives::ByteStream;
     use serial_test::serial;
+    use std::collections::HashSet;
     use tracing::info;
 
     /// Helper function to create an S3 client for testing
@@ -57,6 +58,130 @@ mod tests {
         }
     }
 
+    /// Test for Issue #2775: continuation forwarding must not
+    /// skip a child directory when the prefix component repeats in the key.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_repeated_prefix_continuation() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 repeated-prefix continuation");
+
+        const PAGE_SIZE: i32 = 2;
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
+            .await
+            .expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-repeated-prefix-small";
+        let prefix = "engineering/";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        let expected_keys = vec![
+            format!("{prefix}alpha-000/artifact.txt"),
+            format!("{prefix}engineering/engineering/project-000/artifact.txt"),
+            format!("{prefix}engineering/engineering/project-001/artifact.txt"),
+            format!("{prefix}engineering/project-000/artifact.txt"),
+            format!("{prefix}engineering/project-001/artifact.txt"),
+            format!("{prefix}engineering/project-002/artifact.txt"),
+            format!("{prefix}zulu-000/artifact.txt"),
+        ];
+        let noise_keys = [
+            "different/prefix/prefix/project-000/artifact.txt",
+            "engineering-other/project-000/artifact.txt",
+            "unrelated/engineering/project-000/artifact.txt",
+        ];
+
+        for key in &expected_keys {
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(key)
+                .body(ByteStream::from_static(b"x"))
+                .send()
+                .await
+                .expect("Failed to put object");
+        }
+        for key in noise_keys {
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(key)
+                .body(ByteStream::from_static(b"x"))
+                .send()
+                .await
+                .expect("Failed to put noise object");
+        }
+
+        let mut listed_keys = Vec::new();
+        let mut continuation_token: Option<String> = None;
+        let mut last_key: Option<String> = None;
+        let mut page_count = 0;
+
+        loop {
+            let mut request = client.list_objects_v2().bucket(bucket).prefix(prefix).max_keys(PAGE_SIZE);
+
+            if let Some(token) = continuation_token.take() {
+                request = request.continuation_token(token);
+            }
+
+            let output = request.send().await.expect("Failed to list objects");
+
+            for obj in output.contents() {
+                if let Some(key) = obj.key() {
+                    if let Some(previous) = &last_key {
+                        assert!(
+                            key > previous.as_str(),
+                            "ListObjectsV2 did not preserve lexicographic order: {key} <= {previous}"
+                        );
+                    }
+
+                    last_key = Some(key.to_string());
+                    listed_keys.push(key.to_string());
+                }
+            }
+
+            page_count += 1;
+
+            if output.is_truncated().unwrap_or(false) {
+                continuation_token = output.next_continuation_token().map(|s| s.to_string());
+                assert!(
+                    continuation_token.is_some(),
+                    "BUG: NextContinuationToken must be present when IsTruncated is true"
+                );
+            } else {
+                break;
+            }
+
+            if page_count > 10 {
+                panic!("Too many pages, possible infinite loop due to pagination bug");
+            }
+        }
+
+        let seen: HashSet<String> = listed_keys.iter().cloned().collect();
+
+        assert_eq!(
+            listed_keys, expected_keys,
+            "Issue #2775 regression: repeated-prefix pagination must return exactly the expected keys in lexicographic order"
+        );
+        assert_eq!(
+            listed_keys.len(),
+            expected_keys.len(),
+            "Issue #2775 regression: expected all {} repeated-prefix objects under {prefix}, got {}",
+            expected_keys.len(),
+            listed_keys.len()
+        );
+        assert_eq!(seen.len(), expected_keys.len(), "Listed keys must be unique");
+
+        for key in &expected_keys {
+            assert!(seen.contains(key), "Missing expected key after repeated-prefix pagination: {key}");
+        }
+
+        env.stop_server();
+    }
+
     /// Test that IsTruncated is false when all objects fit within max_keys
     ///
     /// This is the core bug from issue #1596: the server was returning
@@ -368,6 +493,71 @@ mod tests {
         env.stop_server();
     }
 
+    /// Test ListObjectsV2 caps max_keys above the service limit and still paginates.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_max_keys_above_limit_returns_token() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 with max_keys above limit");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-max-keys-above-limit";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        let object_count = 1002;
+        for i in 0..object_count {
+            let key = format!("object{:04}.txt", i);
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(&key)
+                .body(ByteStream::from_static(b"test content"))
+                .send()
+                .await
+                .expect("Failed to put object");
+        }
+
+        let output = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .max_keys(1001)
+            .send()
+            .await
+            .expect("Failed to list objects");
+
+        assert_eq!(output.contents().len(), 1000);
+        assert_eq!(output.key_count(), Some(1000));
+        assert_eq!(output.max_keys(), Some(1000));
+        assert!(
+            output.is_truncated().unwrap_or(false),
+            "IsTruncated should be true when more objects remain after capped max_keys"
+        );
+
+        let next_token = output
+            .next_continuation_token()
+            .expect("NextContinuationToken should be present when capped response is truncated")
+            .to_string();
+
+        let output = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .max_keys(1001)
+            .continuation_token(next_token)
+            .send()
+            .await
+            .expect("Failed to list objects with continuation token");
+
+        assert_eq!(output.contents().len(), 2);
+        assert!(!output.is_truncated().unwrap_or(false));
+        assert!(output.next_continuation_token().is_none());
+
+        env.stop_server();
+    }
+
     /// Test ListObjectsV2 with max_keys=0
     ///
     /// S3 semantics: when max_keys is 0, the response should include no objects

crates/e2e_test/src/object_lambda_test.rs

@@ -25,7 +25,7 @@ use std::error::Error;
 use time::OffsetDateTime;
 use tokio::io::{AsyncReadExt, AsyncWriteExt};
 use tokio::net::TcpListener;
-use tokio::sync::oneshot;
+use tokio::sync::{mpsc, oneshot};
 use tokio::time::{Duration, timeout};
 
 #[derive(Debug)]
@@ -172,6 +172,28 @@ async fn spawn_object_lambda_webhook_server_with_response(
     Ok((webhook_url, request_rx, handle))
 }
 
+async fn read_request_path(stream: &mut tokio::net::TcpStream) -> Result<String, Box<dyn Error + Send + Sync>> {
+    let mut buffer = Vec::new();
+    let mut chunk = [0_u8; 4096];
+
+    let header_end = loop {
+        let read = stream.read(&mut chunk).await?;
+        if read == 0 {
+            return Err("request ended before headers were fully received".into());
+        }
+        buffer.extend_from_slice(&chunk[..read]);
+        if let Some(pos) = find_header_terminator(&buffer) {
+            break pos;
+        }
+    };
+
+    let header_text = std::str::from_utf8(&buffer[..header_end])?;
+    let request_line = header_text.lines().next().ok_or("missing request line")?;
+    let path = request_line.split_whitespace().nth(1).ok_or("missing request path")?;
+
+    Ok(path.to_string())
+}
+
 async fn presigned_get_request(
     url: &str,
     access_key: &str,
@@ -326,11 +348,15 @@ async fn delete_webhook_target(env: &RustFSTestEnvironment, target_name: &str) -
 }
 
 fn notification_target_is_listed(targets: &serde_json::Value, target_name: &str) -> bool {
+    notification_target_entry(targets, target_name).is_some()
+}
+
+fn notification_target_entry<'a>(targets: &'a serde_json::Value, target_name: &str) -> Option<&'a serde_json::Value> {
     targets["notification_endpoints"]
         .as_array()
         .into_iter()
         .flatten()
-        .any(|entry| {
+        .find(|entry| {
             entry["account_id"].as_str() == Some(target_name)
                 && entry["service"]
                     .as_str()
@@ -338,6 +364,10 @@ fn notification_target_is_listed(targets: &serde_json::Value, target_name: &str)
         })
 }
 
+fn notification_target_status<'a>(targets: &'a serde_json::Value, target_name: &str) -> Option<&'a str> {
+    notification_target_entry(targets, target_name).and_then(|entry| entry["status"].as_str())
+}
+
 async fn wait_for_target_visibility(
     env: &RustFSTestEnvironment,
     target_name: &str,
@@ -387,6 +417,34 @@ async fn restart_rustfs_server(env: &mut RustFSTestEnvironment) -> Result<(), Bo
     env.start_rustfs_server_without_cleanup(vec![]).await
 }
 
+async fn spawn_http_origin_probe_server() -> Result<
+    (
+        String,
+        mpsc::Receiver<String>,
+        tokio::task::JoinHandle<Result<(), Box<dyn Error + Send + Sync>>>,
+    ),
+    Box<dyn Error + Send + Sync>,
+> {
+    let listener = TcpListener::bind("127.0.0.1:0").await?;
+    let address = listener.local_addr()?;
+    let webhook_url = format!("http://{address}/hook");
+    let (path_tx, path_rx) = mpsc::channel(1);
+
+    let handle = tokio::spawn(async move {
+        loop {
+            let (mut stream, _) = listener.accept().await?;
+            let path = timeout(Duration::from_secs(2), read_request_path(&mut stream)).await??;
+            let _ = path_tx.try_send(path.clone());
+            if path == "/" {
+                let response = b"HTTP/1.1 200 OK\r\nContent-Length: 0\r\nConnection: close\r\n\r\n";
+                stream.write_all(response).await?;
+            }
+        }
+    });
+
+    Ok((webhook_url, path_rx, handle))
+}
+
 async fn read_persisted_server_config(env: &RustFSTestEnvironment) -> String {
     let path = format!("{}/.rustfs.sys/config/config.json", env.temp_dir);
     match tokio::fs::read_to_string(&path).await {
@@ -515,6 +573,40 @@ async fn test_notification_target_persists_across_restart_and_delete() -> Result
     Ok(())
 }
 
+#[tokio::test]
+#[serial]
+async fn test_notification_target_with_path_is_online_via_transport_probe() -> Result<(), Box<dyn Error + Send + Sync>> {
+    init_logging();
+
+    let (webhook_url, mut probe_rx, probe_handle) = spawn_http_origin_probe_server().await?;
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_NOTIFY_ENABLE", "true")])
+        .await?;
+
+    let target_name = "path-probe";
+    configure_webhook_target(&env, target_name, &webhook_url, "secret-token").await?;
+
+    let (visible_targets, visible_arns) = wait_for_target_visibility(&env, target_name).await?;
+    assert_eq!(notification_target_status(&visible_targets, target_name), Some("online"));
+    let observed_path = timeout(Duration::from_secs(10), probe_rx.recv())
+        .await
+        .map_err(|_| "probe server timed out waiting for a request")?
+        .ok_or("probe server did not observe a request")?;
+    assert_eq!(observed_path, "/");
+    assert!(
+        visible_arns
+            .iter()
+            .any(|arn| arn.ends_with(&format!(":{target_name}:webhook"))),
+        "target ARN missing for reachable path endpoint: {visible_arns:?}"
+    );
+
+    probe_handle.abort();
+    let _ = probe_handle.await;
+
+    Ok(())
+}
+
 #[tokio::test]
 #[serial]
 async fn test_get_object_lambda_accepts_presigned_requests() -> Result<(), Box<dyn Error + Send + Sync>> {
@@ -1058,6 +1150,41 @@ async fn test_listen_notification_emits_after_put_object() -> Result<(), Box<dyn
     Ok(())
 }
 
+#[tokio::test]
+#[serial]
+async fn test_listen_notification_emits_on_empty_bucket_when_notify_disabled() -> Result<(), Box<dyn Error + Send + Sync>> {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_NOTIFY_ENABLE", "false")])
+        .await?;
+
+    let bucket = "listen-empty-bucket-e2e";
+    let key = "seed/object.txt";
+    let client = env.create_s3_client();
+
+    client.create_bucket().bucket(bucket).send().await?;
+
+    let listen_url = format!("{}/{bucket}?events={}&ping=1", env.url, urlencoding::encode("s3:ObjectCreated:*"),);
+    let response = signed_request(http::Method::GET, &listen_url, &env.access_key, &env.secret_key, None, None).await?;
+    assert_eq!(response.status(), StatusCode::OK);
+
+    let read_task = tokio::spawn(read_listen_notification_event(response, key));
+
+    client
+        .put_object()
+        .bucket(bucket)
+        .key(key)
+        .body(ByteStream::from_static(b"empty bucket watch body"))
+        .send()
+        .await?;
+
+    let payload = timeout(Duration::from_secs(12), read_task).await???;
+    assert!(!payload.is_empty(), "listen_notification payload should not be empty");
+
+    Ok(())
+}
+
 #[tokio::test]
 #[serial]
 async fn test_listen_notification_fans_in_remote_node_events() -> Result<(), Box<dyn Error + Send + Sync>> {

crates/e2e_test/src/object_lock/object_lock_test.rs

@@ -26,10 +26,12 @@
 
 use super::common::*;
 use aws_sdk_s3::Client;
-use aws_sdk_s3::primitives::ByteStream;
+use aws_sdk_s3::primitives::{ByteStream, DateTimeFormat};
 use aws_sdk_s3::types::{
-    CompletedMultipartUpload, CompletedPart, Delete, ObjectIdentifier, ObjectLockLegalHoldStatus, ObjectLockRetentionMode,
+    CompletedMultipartUpload, CompletedPart, Delete, MetadataDirective, ObjectIdentifier, ObjectLockLegalHoldStatus,
+    ObjectLockMode, ObjectLockRetentionMode,
 };
+use chrono::{DateTime, Duration, Utc};
 use serial_test::serial;
 use tracing::info;
 
@@ -79,6 +81,26 @@ fn assert_access_denied<T, E: std::fmt::Debug>(result: Result<T, E>, context: &s
     );
 }
 
+fn assert_invalid_object_lock_retention_pair<T, E: std::fmt::Debug>(result: Result<T, E>, context: &str) {
+    let err = match result {
+        Ok(_) => panic!("{context}"),
+        Err(err) => format!("{err:?}"),
+    };
+    assert!(
+        err.contains("InvalidRequest") || err.contains("must both be supplied"),
+        "{context}: expected invalid paired retention headers, got: {err}"
+    );
+}
+
+fn parse_s3_datetime(value: &aws_sdk_s3::primitives::DateTime) -> DateTime<Utc> {
+    let formatted = value
+        .fmt(DateTimeFormat::DateTime)
+        .expect("S3 timestamp should format as RFC3339");
+    DateTime::parse_from_rfc3339(&formatted)
+        .expect("S3 timestamp should parse as RFC3339")
+        .with_timezone(&Utc)
+}
+
 // ============================================================================
 // DeleteObject Tests
 // ============================================================================
@@ -537,6 +559,100 @@ async fn test_copy_object_applies_requested_legal_hold() {
     );
 }
 
+#[tokio::test]
+#[serial]
+async fn test_copy_object_does_not_inherit_source_legal_hold() {
+    init_logging();
+    info!("🧪 Test: CopyObject does not inherit source Legal Hold");
+
+    let mut env = ObjectLockTestEnvironment::new().await.unwrap();
+    env.start_rustfs().await.unwrap();
+
+    let bucket = "test-copy-object-legal-hold-inherit";
+    let src_key = "held-source";
+
+    env.create_object_lock_bucket(bucket).await.unwrap();
+
+    let client = env.s3_client();
+    put_object_with_legal_hold(&client, bucket, src_key, b"copy-source", ObjectLockLegalHoldStatus::On)
+        .await
+        .unwrap();
+
+    client
+        .copy_object()
+        .copy_source(format!("{bucket}/{src_key}"))
+        .bucket(bucket)
+        .key("implicit-copy")
+        .send()
+        .await
+        .unwrap();
+
+    let implicit_legal_hold = client
+        .get_object_legal_hold()
+        .bucket(bucket)
+        .key("implicit-copy")
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(
+        implicit_legal_hold
+            .legal_hold()
+            .and_then(|value| value.status())
+            .map(|value| value.as_str()),
+        Some("OFF")
+    );
+
+    client
+        .copy_object()
+        .copy_source(format!("{bucket}/{src_key}"))
+        .bucket(bucket)
+        .key("explicit-on-copy")
+        .object_lock_legal_hold_status(ObjectLockLegalHoldStatus::On)
+        .send()
+        .await
+        .unwrap();
+
+    let explicit_on_legal_hold = client
+        .get_object_legal_hold()
+        .bucket(bucket)
+        .key("explicit-on-copy")
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(
+        explicit_on_legal_hold
+            .legal_hold()
+            .and_then(|value| value.status())
+            .map(|value| value.as_str()),
+        Some("ON")
+    );
+
+    client
+        .copy_object()
+        .copy_source(format!("{bucket}/{src_key}"))
+        .bucket(bucket)
+        .key("explicit-off-copy")
+        .object_lock_legal_hold_status(ObjectLockLegalHoldStatus::Off)
+        .send()
+        .await
+        .unwrap();
+
+    let explicit_off_legal_hold = client
+        .get_object_legal_hold()
+        .bucket(bucket)
+        .key("explicit-off-copy")
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(
+        explicit_off_legal_hold
+            .legal_hold()
+            .and_then(|value| value.status())
+            .map(|value| value.as_str()),
+        Some("OFF")
+    );
+}
+
 #[tokio::test]
 #[serial]
 async fn test_copy_object_overwrite_blocked_by_legal_hold() {
@@ -1388,9 +1504,457 @@ async fn test_default_retention_applied_to_new_objects() {
     let delete_result = delete_object_with_bypass(&client, bucket, key, Some(version_id), false).await;
     assert!(delete_result.is_err(), "Delete should fail for object with default retention applied");
 
+    let retention = client
+        .get_object_retention()
+        .bucket(bucket)
+        .key(key)
+        .version_id(version_id)
+        .send()
+        .await
+        .unwrap();
+    let retention = retention.retention().expect("default retention should be readable");
+    assert_eq!(retention.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    assert!(
+        retention.retain_until_date().is_some(),
+        "default retention should write a retain-until date"
+    );
+
+    let head = client
+        .head_object()
+        .bucket(bucket)
+        .key(key)
+        .version_id(version_id)
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(head.object_lock_mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    assert!(
+        head.object_lock_retain_until_date().is_some(),
+        "HeadObject should expose the default retention retain-until date"
+    );
+
     info!("✅ Test passed: Default retention is applied to new objects");
 }
 
+#[tokio::test]
+#[serial]
+async fn test_delete_object_creates_delete_marker_for_default_retained_current_version() {
+    init_logging();
+    info!("🧪 Test: DeleteObject creates delete marker for default-retained current version");
+
+    let mut env = ObjectLockTestEnvironment::new().await.unwrap();
+    env.start_rustfs().await.unwrap();
+
+    let bucket = "test-default-retention-delete-marker";
+    let key = "default-retained-object";
+    let data = b"test data for default-retained current version";
+
+    env.create_object_lock_bucket(bucket).await.unwrap();
+
+    let client = env.s3_client();
+    put_object_lock_configuration(&client, bucket, ObjectLockRetentionMode::Governance, Some(30), None)
+        .await
+        .unwrap();
+
+    let put_output = client
+        .put_object()
+        .bucket(bucket)
+        .key(key)
+        .body(ByteStream::from(data.to_vec()))
+        .send()
+        .await
+        .unwrap();
+    let retained_version_id = put_output
+        .version_id()
+        .expect("default-retained object should have a version id")
+        .to_string();
+
+    let retention = client
+        .get_object_retention()
+        .bucket(bucket)
+        .key(key)
+        .version_id(&retained_version_id)
+        .send()
+        .await
+        .unwrap();
+    let retention = retention.retention().expect("default retention should be readable");
+    assert_eq!(retention.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    assert!(
+        retention.retain_until_date().is_some(),
+        "default retention should write a retain-until date"
+    );
+
+    let delete_marker_output = client.delete_object().bucket(bucket).key(key).send().await.unwrap();
+    assert_eq!(delete_marker_output.delete_marker(), Some(true));
+    let delete_marker_version_id = delete_marker_output
+        .version_id()
+        .expect("delete marker should have a version id")
+        .to_string();
+
+    let protected_delete = delete_object_with_bypass(&client, bucket, key, Some(&retained_version_id), false).await;
+    assert!(protected_delete.is_err(), "Default-retained version should still reject direct deletion");
+
+    let retention_after_delete_marker = client
+        .get_object_retention()
+        .bucket(bucket)
+        .key(key)
+        .version_id(&retained_version_id)
+        .send()
+        .await
+        .unwrap();
+    let retention_after_delete_marker = retention_after_delete_marker
+        .retention()
+        .expect("default retention should remain readable by version id after delete marker creation");
+    assert_eq!(retention_after_delete_marker.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    assert!(
+        retention_after_delete_marker.retain_until_date().is_some(),
+        "retained version should keep its retain-until date after delete marker creation"
+    );
+
+    delete_object_with_bypass(&client, bucket, key, Some(&delete_marker_version_id), false)
+        .await
+        .unwrap();
+    delete_object_with_bypass(&client, bucket, key, Some(&retained_version_id), true)
+        .await
+        .unwrap();
+
+    info!("✅ Test passed: Delete marker is allowed while default-retained version stays protected");
+}
+
+#[tokio::test]
+#[serial]
+async fn test_put_copy_and_multipart_reject_incomplete_retention_headers() {
+    init_logging();
+    info!("🧪 Test: write paths reject incomplete Object Lock retention headers");
+
+    let mut env = ObjectLockTestEnvironment::new().await.unwrap();
+    env.start_rustfs().await.unwrap();
+
+    let bucket = "test-incomplete-retention";
+    let src_key = "copy-source";
+
+    env.create_object_lock_bucket(bucket).await.unwrap();
+
+    let client = env.s3_client();
+    client
+        .put_object()
+        .bucket(bucket)
+        .key(src_key)
+        .body(ByteStream::from(b"copy-source".to_vec()))
+        .send()
+        .await
+        .unwrap();
+
+    put_object_lock_configuration(&client, bucket, ObjectLockRetentionMode::Governance, Some(30), None)
+        .await
+        .unwrap();
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .put_object()
+            .bucket(bucket)
+            .key("put-mode-only")
+            .body(ByteStream::from(b"put-body".to_vec()))
+            .object_lock_mode(ObjectLockMode::Governance)
+            .send()
+            .await,
+        "PutObject with mode only should fail",
+    );
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .put_object()
+            .bucket(bucket)
+            .key("put-date-only")
+            .body(ByteStream::from(b"put-body".to_vec()))
+            .object_lock_retain_until_date(retention_timestamp(30))
+            .send()
+            .await,
+        "PutObject with retain-until-date only should fail",
+    );
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .copy_object()
+            .copy_source(format!("{bucket}/{src_key}"))
+            .bucket(bucket)
+            .key("copy-mode-only")
+            .object_lock_mode(ObjectLockMode::Governance)
+            .send()
+            .await,
+        "CopyObject with mode only should fail",
+    );
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .copy_object()
+            .copy_source(format!("{bucket}/{src_key}"))
+            .bucket(bucket)
+            .key("copy-date-only")
+            .object_lock_retain_until_date(retention_timestamp(30))
+            .send()
+            .await,
+        "CopyObject with retain-until-date only should fail",
+    );
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .create_multipart_upload()
+            .bucket(bucket)
+            .key("multipart-mode-only")
+            .object_lock_mode(ObjectLockMode::Governance)
+            .send()
+            .await,
+        "CreateMultipartUpload with mode only should fail",
+    );
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .create_multipart_upload()
+            .bucket(bucket)
+            .key("multipart-date-only")
+            .object_lock_retain_until_date(retention_timestamp(30))
+            .send()
+            .await,
+        "CreateMultipartUpload with retain-until-date only should fail",
+    );
+}
+
+#[tokio::test]
+#[serial]
+async fn test_copy_object_retention_uses_destination_policy() {
+    init_logging();
+    info!("🧪 Test: CopyObject retention follows destination policy");
+
+    let mut env = ObjectLockTestEnvironment::new().await.unwrap();
+    env.start_rustfs().await.unwrap();
+
+    let src_bucket = "test-copy-retention-src";
+    let dst_bucket = "test-copy-retention-dst";
+    let no_default_bucket = "test-copy-retention-nodef";
+    let src_key = "retained-source";
+
+    env.create_object_lock_bucket(src_bucket).await.unwrap();
+    env.create_object_lock_bucket(dst_bucket).await.unwrap();
+    env.create_object_lock_bucket(no_default_bucket).await.unwrap();
+
+    let client = env.s3_client();
+    put_object_lock_configuration(&client, dst_bucket, ObjectLockRetentionMode::Governance, Some(1), None)
+        .await
+        .unwrap();
+
+    put_object_with_retention(
+        &client,
+        src_bucket,
+        src_key,
+        b"copy-source",
+        ObjectLockRetentionMode::Compliance,
+        future_retain_until(30),
+    )
+    .await
+    .unwrap();
+
+    let copy_started = Utc::now();
+    client
+        .copy_object()
+        .copy_source(format!("{src_bucket}/{src_key}"))
+        .bucket(dst_bucket)
+        .key("default-copy")
+        .send()
+        .await
+        .unwrap();
+
+    let retention = client
+        .get_object_retention()
+        .bucket(dst_bucket)
+        .key("default-copy")
+        .send()
+        .await
+        .unwrap();
+    let retention = retention
+        .retention()
+        .expect("destination default retention should be present");
+    assert_eq!(retention.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    let retain_until = parse_s3_datetime(retention.retain_until_date().expect("retain-until date should be present"));
+    assert!(
+        retain_until < copy_started + Duration::days(3),
+        "destination default retention should not inherit the source's longer retention"
+    );
+
+    client
+        .copy_object()
+        .copy_source(format!("{src_bucket}/{src_key}"))
+        .bucket(dst_bucket)
+        .key("replace-copy")
+        .metadata_directive(MetadataDirective::Replace)
+        .send()
+        .await
+        .unwrap();
+
+    let replace_retention = client
+        .get_object_retention()
+        .bucket(dst_bucket)
+        .key("replace-copy")
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(
+        replace_retention
+            .retention()
+            .and_then(|value| value.mode())
+            .map(|value| value.as_str()),
+        Some("GOVERNANCE")
+    );
+
+    client
+        .copy_object()
+        .copy_source(format!("{src_bucket}/{src_key}"))
+        .bucket(dst_bucket)
+        .key("explicit-copy")
+        .object_lock_mode(ObjectLockMode::Compliance)
+        .object_lock_retain_until_date(retention_timestamp(30))
+        .send()
+        .await
+        .unwrap();
+
+    let explicit_retention = client
+        .get_object_retention()
+        .bucket(dst_bucket)
+        .key("explicit-copy")
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(
+        explicit_retention
+            .retention()
+            .and_then(|value| value.mode())
+            .map(|value| value.as_str()),
+        Some("COMPLIANCE")
+    );
+    let explicit_retain_until = parse_s3_datetime(
+        explicit_retention
+            .retention()
+            .and_then(|value| value.retain_until_date())
+            .expect("explicit retain-until date should be present"),
+    );
+    assert!(
+        explicit_retain_until > Utc::now() + Duration::days(20),
+        "explicit retention should override the shorter bucket default"
+    );
+
+    client
+        .copy_object()
+        .copy_source(format!("{src_bucket}/{src_key}"))
+        .bucket(no_default_bucket)
+        .key("no-default-copy")
+        .send()
+        .await
+        .unwrap();
+
+    let no_default_retention = client
+        .get_object_retention()
+        .bucket(no_default_bucket)
+        .key("no-default-copy")
+        .send()
+        .await
+        .unwrap();
+    let no_default_retention = no_default_retention
+        .retention()
+        .expect("retention response should be present");
+    assert!(no_default_retention.mode().is_none());
+    assert!(no_default_retention.retain_until_date().is_none());
+
+    put_object_with_retention(
+        &client,
+        dst_bucket,
+        "locked-destination",
+        b"locked-target",
+        ObjectLockRetentionMode::Compliance,
+        future_retain_until(30),
+    )
+    .await
+    .unwrap();
+
+    let overwrite_result = client
+        .copy_object()
+        .copy_source(format!("{src_bucket}/{src_key}"))
+        .bucket(dst_bucket)
+        .key("locked-destination")
+        .send()
+        .await;
+    assert!(
+        overwrite_result.is_err(),
+        "CopyObject overwrite should not bypass active destination retention"
+    );
+}
+
+#[tokio::test]
+#[serial]
+async fn test_multipart_default_retention_fixed_at_create() {
+    init_logging();
+    info!("🧪 Test: multipart default retention is fixed at CreateMultipartUpload");
+
+    let mut env = ObjectLockTestEnvironment::new().await.unwrap();
+    env.start_rustfs().await.unwrap();
+
+    let bucket = "test-multipart-default-drift";
+    let key = "multipart-object";
+
+    env.create_object_lock_bucket(bucket).await.unwrap();
+
+    let client = env.s3_client();
+    put_object_lock_configuration(&client, bucket, ObjectLockRetentionMode::Governance, Some(1), None)
+        .await
+        .unwrap();
+
+    let create_started = Utc::now();
+    let create_output = client.create_multipart_upload().bucket(bucket).key(key).send().await.unwrap();
+    let upload_id = create_output.upload_id().unwrap();
+
+    let upload_part_output = client
+        .upload_part()
+        .bucket(bucket)
+        .key(key)
+        .upload_id(upload_id)
+        .part_number(1)
+        .body(ByteStream::from(b"multipart-body".to_vec()))
+        .send()
+        .await
+        .unwrap();
+
+    put_object_lock_configuration(&client, bucket, ObjectLockRetentionMode::Governance, Some(10), None)
+        .await
+        .unwrap();
+
+    let completed_upload = CompletedMultipartUpload::builder()
+        .parts(
+            CompletedPart::builder()
+                .part_number(1)
+                .e_tag(upload_part_output.e_tag().unwrap_or_default())
+                .build(),
+        )
+        .build();
+
+    client
+        .complete_multipart_upload()
+        .bucket(bucket)
+        .key(key)
+        .upload_id(upload_id)
+        .multipart_upload(completed_upload)
+        .send()
+        .await
+        .unwrap();
+
+    let retention = client.get_object_retention().bucket(bucket).key(key).send().await.unwrap();
+    let retention = retention.retention().expect("multipart default retention should be present");
+    assert_eq!(retention.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    let retain_until = parse_s3_datetime(retention.retain_until_date().expect("retain-until date should be present"));
+    assert!(
+        retain_until < create_started + Duration::days(3),
+        "CompleteMultipartUpload should keep the default retention calculated at create time"
+    );
+}
+
 // ============================================================================
 // Versioning Auto-Enable Tests
 // ============================================================================

crates/e2e_test/src/protocols/README.md

@@ -1,38 +1,26 @@
 # Protocol E2E Tests
 
-FTPS and WebDAV protocol end-to-end tests for RustFS.
+FTPS, WebDAV, and SFTP protocol end-to-end tests for RustFS.
 
 ## Prerequisites
 
-### Required Tools
-
-```bash
-# Ubuntu/Debian
-sudo apt-get install sshpass ssh-keygen
-
-# RHEL/CentOS
-sudo yum install sshpass openssh-clients
-
-# macOS
-brew install sshpass openssh
-```
+No external SSH tooling is required. The test framework generates ed25519
+host keys in-process via russh::keys under the per-test temp directory
+before each SFTP server spawn, and russh-sftp drives the protocol from the
+test process directly.
 
 ## Running Tests
 
-Run all protocol tests (FTPS + WebDAV):
 ```bash
-RUSTFS_BUILD_FEATURES=ftps,webdav cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
+RUSTFS_BUILD_FEATURES=ftps,webdav,sftp cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
 ```
 
-Run FTPS tests only:
-```bash
-RUSTFS_BUILD_FEATURES=ftps cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
-```
-
-Run WebDAV tests only:
-```bash
-RUSTFS_BUILD_FEATURES=webdav cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
-```
+`RUSTFS_BUILD_FEATURES` controls which features the test rustfs binary is
+built with. When this variable is set, the protocol test runner schedules
+only entries whose protocol is present in the requested feature list. Leave
+it unset to run every protocol entry.
+`--test-threads=1` is required because every entry spawns a rustfs server
+on fixed bind ports.
 
 ## Test Coverage
 
@@ -55,6 +43,94 @@ RUSTFS_BUILD_FEATURES=webdav cargo test --package e2e_test test_protocol_core_su
 - GET (download file)
 - PROPFIND on bucket (list objects)
 - DELETE file
+- MOVE file (rename object)
 - DELETE bucket
 - Authentication failure test
 
+### SFTP Tests
+
+The SFTP suite lives in three entries plus a standalone idle-timeout case.
+Every assertion runs against a freshly spawned rustfs binary with
+`RUSTFS_SFTP_ENABLE=true`; the test framework also pins
+`RUSTFS_SFTP_PART_SIZE=5242880` so the multipart boundary is deterministic.
+
+#### sftp_core (`test_sftp_core_operations`)
+
+Bind ports 9022 (SFTP) and 9200 (S3). 22 in-suite assertions covering the
+core protocol surface plus cross-protocol consistency:
+
+- Subsystem canary: SFTPv3 version exchange completes after password auth
+- Bucket lifecycle: mkdir, root listing, rmdir, post-delete listing
+- Small-file round-trip with SHA256 compare
+- Stat on a file (size + file type) and on a bucket (directory)
+- SETSTAT on a path returns ok
+- Rename within bucket, listing reflects the rename
+- Multipart-sized round-trip (just over 2 × part_size) with SHA256 compare
+- Negative cases: symlink rejected, open of nonexistent file rejected,
+  read_dir of nonexistent bucket rejected, path traversal rejected
+- Spec-letter assertions: APPEND open returns an error, CREATE+EXCLUDE on an
+  existing path returns an error, bad-password authentication is rejected
+- Cross-protocol via aws-sdk-s3: SFTP write then S3 read with SHA256 match,
+  S3 write then SFTP read with SHA256 match
+- Cross-API directory visibility: SFTP-created sub-directory visible via S3
+  ListObjectsV2, S3-created `__XLDIR__` marker visible via SFTP readdir as a
+  directory entry
+
+#### sftp_compliance (`test_sftp_compliance_suite`)
+
+Bind ports 9024 (SFTP) and 9300 (S3). 14 compliance regression cases against
+one shared server spawn. Each case carries a stable CMPTST-NN identifier:
+
+- CMPTST-01: medium-binary upload then download with SHA256 compare
+  (single-shot PutObject path below the multipart boundary)
+- CMPTST-02: zero-byte upload, download, and stat-size match
+- CMPTST-03: rm against a bucket path is rejected; the bucket is preserved
+- CMPTST-04: rmdir against a non-empty bucket is rejected; the contained
+  object survives
+- CMPTST-05: rmdir against a non-empty sub-directory is rejected; the inner
+  object survives
+- CMPTST-06: open with a path-traversal pattern cannot leak a host file via
+  SFTP read
+- CMPTST-07: read_dir of `/..` either errors or returns a listing that
+  contains no host system entries
+- CMPTST-08: rename across buckets preserves payload and removes the source
+  object
+- CMPTST-09: paths with embedded spaces round-trip through the russh-sftp
+  client
+- CMPTST-10: read_link is rejected (S3 storage has no symlinks)
+- CMPTST-11: SETSTAT on a path and FSETSTAT on a separate open handle both
+  return ok (rsync, WinSCP transfer-success contract)
+- CMPTST-12: rename to the same path is a no-op; the file persists with the
+  original payload
+- CMPTST-13: implicit-directory round-trip; uploading to a nested key
+  creates the parent directory implicitly and three listing forms surface
+  the inner file
+- CMPTST-14: OPEN, WRITE, FSETSTAT, CLOSE on the same write handle all
+  return ok (WinSCP wire shape)
+
+#### sftp_compliance_readonly (`test_sftp_compliance_readonly`)
+
+Bind ports 9025 (SFTP) and 9301 (S3). Spawns a second rustfs binary with
+`RUSTFS_SFTP_READ_ONLY=true`; the S3 endpoint stays writable so the suite
+can seed a bucket and a fixture object via aws-sdk-s3 before opening the
+SFTP session. 7 compliance cases:
+
+- CMPTST-15: put through SFTP is rejected
+- CMPTST-16: rm through SFTP is rejected
+- CMPTST-17: mkdir through SFTP is rejected
+- CMPTST-18: rmdir through SFTP is rejected
+- CMPTST-19: rename through SFTP is rejected
+- CMPTST-20: ls through SFTP is allowed and lists the seeded bucket
+- CMPTST-21: get through SFTP is allowed and returns the seeded payload
+  byte-for-byte
+
+The full case index lives at the top of `sftp_compliance.rs`; each helper's
+log lines name its CMPTST-NN code so a failure in CI points at one named
+property without consulting any external doc.
+
+#### sftp_idle_timeout (`test_sftp_idle_timeout_disconnects`)
+
+Bind ports 9023 (SFTP) and 9100 (S3). Spawns rustfs with
+`RUSTFS_SFTP_IDLE_TIMEOUT=5`, sleeps 10 s past the timeout, then issues an
+SFTP request and asserts the server has closed the session.
+

crates/e2e_test/src/protocols/mod.rs

@@ -12,9 +12,13 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-//! Protocol tests for FTPS and WebDAV
+//! Protocol tests for FTPS, WebDAV, and SFTP
 
 pub mod ftps_core;
+pub mod sftp_compliance;
+mod sftp_compliance_tests;
+pub mod sftp_core;
+pub mod sftp_helpers;
 pub mod test_env;
 pub mod test_runner;
 pub mod webdav_core;

crates/e2e_test/src/protocols/sftp_compliance.rs

@@ -0,0 +1,217 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Public test entry points for the SFTP compliance suite.
+//!
+//! Three suite entries cover every CMPTST-NN identifier:
+//!
+//! - test_sftp_compliance_suite           CMPTST-01..14 (one shared SFTP session)
+//! - test_sftp_compliance_readonly        CMPTST-15..23 (one shared session, read-only mode)
+//! - test_sftp_compliance_standalone      CMPTST-24..33 (each case spawns its own rustfs)
+//!
+//! The first two reuse a single rustfs spawn for the whole bracket
+//! because every case in the bracket exercises the same protocol
+//! against the same server. The third aggregates cases that each need
+//! a different server configuration (idle timeout, read-cache window,
+//! console disabled) and therefore cannot share a binary.
+//!
+//! Each per-case module exposes a single descriptive entry called
+//! run_<what_it_tests>(). For example cmptst_24 exposes
+//! cmptst_24::run_concurrent_half_close_no_leak().
+//!
+//! The per-case bodies (one cmptst_NN module per case) and the
+//! cross-case infrastructure (spawn helpers, fixture seeders, the
+//! half-close / wedge / paused-drain stream wrappers, and the session
+//! lifecycle counters) live in sftp_compliance_tests.rs. Per-case
+//! marker comments and the full case-index doc live there too.
+
+use crate::protocols::sftp_compliance_tests::{
+    cmptst_01, cmptst_02, cmptst_03, cmptst_04, cmptst_05, cmptst_06, cmptst_07, cmptst_08, cmptst_09, cmptst_10, cmptst_11,
+    cmptst_12, cmptst_13, cmptst_14, cmptst_15, cmptst_16, cmptst_17, cmptst_18, cmptst_19, cmptst_20, cmptst_21, cmptst_22,
+    cmptst_23, cmptst_27, cmptst_28, cmptst_29, cmptst_32, cmptst_33, spawn_compliance_rustfs,
+};
+#[cfg(target_os = "linux")]
+use crate::protocols::sftp_compliance_tests::{cmptst_24, cmptst_25, cmptst_26};
+use crate::protocols::sftp_helpers::{build_test_s3_client, connect_sftp_to, wait_for_s3_ready};
+use crate::protocols::test_env::ProtocolTestEnvironment;
+use anyhow::{Result, anyhow};
+use aws_sdk_s3::primitives::ByteStream;
+use tracing::info;
+
+// Read-write compliance suite ports. Distinct from sftp_core (9022/9200)
+// and from test_sftp_idle_timeout_disconnects (9023/9100) so the SFTP
+// entries can run sequentially without leftover-listener contention.
+const COMPLIANCE_RW_SFTP_PORT: u16 = 9024;
+const COMPLIANCE_RW_SFTP_ADDRESS: &str = "127.0.0.1:9024";
+const COMPLIANCE_RW_S3_ADDRESS: &str = "127.0.0.1:9300";
+
+// Read-only compliance suite ports. The SFTP session opened against
+// this address runs against a server started with
+// RUSTFS_SFTP_READ_ONLY=true. The S3 endpoint stays writable so the
+// suite can seed a bucket and a fixture object before running the SFTP
+// rejection assertions.
+const COMPLIANCE_RO_SFTP_PORT: u16 = 9025;
+const COMPLIANCE_RO_SFTP_ADDRESS: &str = "127.0.0.1:9025";
+const COMPLIANCE_RO_S3_ADDRESS: &str = "127.0.0.1:9301";
+const COMPLIANCE_RO_S3_ENDPOINT: &str = "http://127.0.0.1:9301";
+const COMPLIANCE_RO_S3_READY_ATTEMPTS: u32 = 30;
+
+/// Compliance suite entry: spawn one rustfs server, run every per-case
+/// helper that closes a coverage gap not exercised by sftp_core. Runs
+/// CMPTST-01 through CMPTST-14 against the same SFTP session.
+pub async fn test_sftp_compliance_suite() -> Result<()> {
+    info!("Starting SFTP server for compliance suite on {}", COMPLIANCE_RW_SFTP_ADDRESS);
+    let (_env, mut server_process) = spawn_compliance_rustfs(COMPLIANCE_RW_SFTP_ADDRESS, COMPLIANCE_RW_S3_ADDRESS, false).await?;
+
+    let result = async {
+        ProtocolTestEnvironment::wait_for_port_ready(COMPLIANCE_RW_SFTP_PORT, 30)
+            .await
+            .map_err(|e| anyhow!("{}", e))?;
+
+        let (session, sftp) = connect_sftp_to(COMPLIANCE_RW_SFTP_ADDRESS).await?;
+
+        cmptst_01::run_medium_binary_round_trip(&sftp).await?;
+        cmptst_02::run_zero_byte_round_trip(&sftp).await?;
+        cmptst_03::run_rm_on_bucket_path_rejected(&sftp).await?;
+        cmptst_04::run_rmdir_nonempty_bucket_rejected(&sftp).await?;
+        cmptst_05::run_rmdir_nonempty_subdir_rejected(&sftp).await?;
+        cmptst_06::run_path_traversal_get_rejected(&sftp).await?;
+        cmptst_07::run_dotdot_collapses_to_root(&sftp).await?;
+        cmptst_08::run_rename_cross_bucket(&sftp).await?;
+        cmptst_09::run_path_with_spaces_round_trip(&sftp).await?;
+        cmptst_10::run_readlink_rejected(&sftp).await?;
+        cmptst_11::run_setstat_after_put_returns_ok(&sftp).await?;
+        cmptst_12::run_rename_same_path_keeps_file(&sftp).await?;
+        cmptst_13::run_implicit_dir_round_trip(&sftp).await?;
+        cmptst_14::run_winscp_setstat_shape_on_handle(&sftp).await?;
+
+        drop(sftp);
+        session.disconnect(russh::Disconnect::ByApplication, "", "en").await?;
+        info!("SFTP compliance suite passed");
+        Ok::<(), anyhow::Error>(())
+    }
+    .await;
+
+    // Discard kill/wait errors on the teardown path: the test result
+    // above is the binding outcome, and a server that has already
+    // exited produces an error here that carries no useful signal.
+    server_process.kill_and_wait().await;
+
+    result
+}
+
+/// Read-only compliance entry: CMPTST-15 through CMPTST-23. The SFTP
+/// server runs with RUSTFS_SFTP_READ_ONLY=true. Mutations through SFTP
+/// must error. Reads through SFTP must succeed. The test seeds a bucket
+/// and a file through the writable S3 endpoint before opening the SFTP
+/// session.
+pub async fn test_sftp_compliance_readonly() -> Result<()> {
+    info!("Starting SFTP server in read-only mode on {}", COMPLIANCE_RO_SFTP_ADDRESS);
+    let (_env, mut server_process) = spawn_compliance_rustfs(COMPLIANCE_RO_SFTP_ADDRESS, COMPLIANCE_RO_S3_ADDRESS, true).await?;
+
+    let result = async {
+        ProtocolTestEnvironment::wait_for_port_ready(COMPLIANCE_RO_SFTP_PORT, 30)
+            .await
+            .map_err(|e| anyhow!("{}", e))?;
+
+        let s3 = build_test_s3_client(COMPLIANCE_RO_S3_ENDPOINT);
+        wait_for_s3_ready(&s3, COMPLIANCE_RO_S3_READY_ATTEMPTS).await?;
+
+        let bucket = "robucket";
+        let seeded_key = "small.txt";
+        let seeded_content = b"read-only seed\n";
+
+        s3.create_bucket()
+            .bucket(bucket)
+            .send()
+            .await
+            .map_err(|e| anyhow!("S3 CreateBucket {} failed: {:?}", bucket, e))?;
+        s3.put_object()
+            .bucket(bucket)
+            .key(seeded_key)
+            .body(ByteStream::from_static(seeded_content))
+            .send()
+            .await
+            .map_err(|e| anyhow!("S3 PutObject {}/{} failed: {:?}", bucket, seeded_key, e))?;
+        info!("Seeded read-only fixture via S3: {}/{}", bucket, seeded_key);
+
+        let (session, sftp) = connect_sftp_to(COMPLIANCE_RO_SFTP_ADDRESS).await?;
+
+        cmptst_15::run_ro_put_rejected(&sftp, bucket).await?;
+        cmptst_16::run_ro_rm_rejected(&sftp, bucket, seeded_key).await?;
+        cmptst_17::run_ro_mkdir_rejected(&sftp).await?;
+        cmptst_18::run_ro_rmdir_rejected(&sftp, bucket).await?;
+        cmptst_19::run_ro_rename_rejected(&sftp, bucket, seeded_key).await?;
+        cmptst_20::run_ro_ls_allowed(&sftp, bucket).await?;
+        cmptst_21::run_ro_get_allowed(&sftp, bucket, seeded_key, seeded_content).await?;
+        cmptst_22::run_ro_setstat_rejected(&sftp, bucket, seeded_key).await?;
+        cmptst_23::run_ro_fsetstat_rejected(&sftp, bucket, seeded_key).await?;
+
+        drop(sftp);
+        // Discard the disconnect Result. A read-only session that has
+        // returned errors against every mutation can still be cleanly
+        // torn down, but a transient transport-level error here
+        // carries no useful signal beyond what the assertions above
+        // already pin.
+        let _ = session.disconnect(russh::Disconnect::ByApplication, "", "en").await;
+        info!("SFTP read-only compliance suite passed");
+        Ok::<(), anyhow::Error>(())
+    }
+    .await;
+
+    server_process.kill_and_wait().await;
+
+    result
+}
+
+/// Standalone-server compliance entry: runs CMPTST-24..33 in numerical
+/// order. Each case spawns and tears down its own rustfs because each
+/// exercises a different server configuration (idle timeout, console
+/// listener, read-cache window) that cannot share a process with the
+/// others.
+///
+/// CMPTST-30 is omitted by default. Its assertion (per-operation
+/// wall-clock latency under pipelined metadata ops) is bounded by the
+/// SSH SFTP subsystem's per-channel serial handler dispatch and is
+/// structurally infeasible against the production code. The
+/// test_sftp_handler_latency_regression #[tokio::test] entry remains
+/// runnable on demand via `--ignored`.
+///
+/// CMPTST-31 (paused-drain) is omitted from the default suite for
+/// runtime cost (200 MiB seed plus a 25 s pause window). The
+/// test_sftp_paused_drain_regression #[tokio::test] entry covers it
+/// for direct invocation.
+///
+/// CMPTST-24, 25, 26 verify kernel-level state via ss(8) and the
+/// procfs ESTABLISHED discriminator. They are skipped on non-Linux
+/// targets where those interfaces are absent.
+pub async fn test_sftp_compliance_standalone() -> Result<()> {
+    info!("Starting SFTP standalone-server compliance suite");
+
+    #[cfg(target_os = "linux")]
+    {
+        cmptst_24::run_concurrent_half_close_no_leak().await?;
+        cmptst_25::run_wedge_kill_after_silence_in_close_wait().await?;
+        cmptst_26::run_healthy_idle_session_above_fast_threshold().await?;
+    }
+
+    cmptst_27::run_multi_session_mixed_pipelining().await?;
+    cmptst_28::run_5mb_download_with_concurrent_metadata_ops().await?;
+    cmptst_29::run_read_past_eof_volume().await?;
+    cmptst_32::run_read_cache_enabled_round_trip().await?;
+    cmptst_33::run_read_cache_disabled_round_trip().await?;
+
+    info!("SFTP standalone-server compliance suite passed");
+    Ok(())
+}

crates/e2e_test/src/protocols/sftp_core.rs

@@ -0,0 +1,557 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Core SFTP tests
+
+use crate::common::rustfs_binary_path_with_features;
+use crate::protocols::sftp_helpers::{
+    AcceptAnyServerKey, ServerProcess, build_test_s3_client, connect_sftp_to, generate_host_key, sftp_read_full,
+    wait_for_s3_ready,
+};
+use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, ProtocolTestEnvironment};
+use anyhow::{Result, anyhow};
+use aws_sdk_s3::Client as S3Client;
+use aws_sdk_s3::primitives::ByteStream;
+use russh::client::{self, Handle};
+use russh_sftp::client::SftpSession;
+use russh_sftp::protocol::{FileAttributes, OpenFlags};
+use rustfs_config::{
+    ENV_RUSTFS_ADDRESS, ENV_SFTP_ADDRESS, ENV_SFTP_ENABLE, ENV_SFTP_HOST_KEY_DIR, ENV_SFTP_IDLE_TIMEOUT, ENV_SFTP_PART_SIZE,
+    ENV_SFTP_READ_ONLY,
+};
+use sha2::{Digest, Sha256};
+use std::path::PathBuf;
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::io::{AsyncReadExt, AsyncWriteExt};
+use tokio::process::Command;
+use tokio::time::sleep;
+use tracing::info;
+
+const SFTP_PORT: u16 = 9022;
+const SFTP_ADDRESS: &str = "127.0.0.1:9022";
+
+// The cross-protocol assertions reach the same server process the SFTP
+// session is connected to. The S3 endpoint is bound on a non-default port to
+// avoid contention with any other rustfs running on port 9000 (for example a
+// dev-testing harness container).
+const S3_ADDRESS: &str = "127.0.0.1:9200";
+const S3_ENDPOINT: &str = "http://127.0.0.1:9200";
+const S3_READY_ATTEMPTS: u32 = 30;
+
+// Mirrors GLOBAL_DIR_SUFFIX in rustfs_utils::path. The e2e_test crate does
+// not depend on rustfs-utils, so the suffix is repeated locally.
+const XLDIR_SUFFIX: &str = "__XLDIR__";
+
+// Idle-timeout test uses its own ports so it can run alongside the core suite
+// without clashing on the default S3 port or on the core SFTP port.
+const IDLE_SFTP_PORT: u16 = 9023;
+const IDLE_SFTP_ADDRESS: &str = "127.0.0.1:9023";
+const IDLE_S3_ADDRESS: &str = "127.0.0.1:9100";
+const IDLE_TIMEOUT_SECS: u64 = 5;
+const IDLE_WAIT_SECS: u64 = 10;
+
+// Pin the server's multipart part_size to the spec minimum (5 MiB) so the
+// multipart payload in this file is sized relative to a known value and the
+// Buffering to Streaming transition triggers deterministically regardless of
+// the server's default.
+const PART_SIZE_BYTES: usize = 5 * 1024 * 1024;
+const PART_SIZE_ENV: &str = "5242880";
+// Just over two part_size worth so the upload issues CreateMultipartUpload,
+// at least one UploadPart mid-stream, and CompleteMultipartUpload.
+const MULTIPART_SIZE: usize = PART_SIZE_BYTES * 2 + 1024;
+
+// Fixed deterministic payload for the S3-write, SFTP-read direction. 256 KiB
+// is well below part_size so the SFTP read returns the object as a single
+// GetObject response without invoking the streaming multipart path.
+const S3_WRITTEN_SIZE: usize = 256 * 1024;
+
+async fn connect_sftp() -> Result<(Handle<AcceptAnyServerKey>, SftpSession)> {
+    connect_sftp_to(SFTP_ADDRESS).await
+}
+
+/// Confirm that an object is byte-identical when fetched via S3 and via SFTP.
+/// Hashes the expected payload once, then compares both fetched payloads
+/// against that hash. Either mismatch returns an error naming the side that
+/// disagreed.
+async fn assert_cross_protocol_sha_match(
+    s3: &S3Client,
+    sftp: &SftpSession,
+    bucket: &str,
+    key: &str,
+    expected: &[u8],
+) -> Result<()> {
+    let expected_sha = Sha256::digest(expected);
+
+    let s3_get = s3
+        .get_object()
+        .bucket(bucket)
+        .key(key)
+        .send()
+        .await
+        .map_err(|e| anyhow!("S3 GetObject {}/{} failed: {:?}", bucket, key, e))?;
+    let s3_bytes = s3_get
+        .body
+        .collect()
+        .await
+        .map_err(|e| anyhow!("S3 body collect failed for {}/{}: {:?}", bucket, key, e))?
+        .into_bytes();
+    if s3_bytes.len() != expected.len() {
+        return Err(anyhow!(
+            "S3 GetObject byte count mismatch for {}/{}: expected {}, got {}",
+            bucket,
+            key,
+            expected.len(),
+            s3_bytes.len()
+        ));
+    }
+    let s3_sha = Sha256::digest(&s3_bytes);
+    if s3_sha != expected_sha {
+        return Err(anyhow!("S3 GetObject SHA256 mismatch for {}/{}", bucket, key));
+    }
+
+    let sftp_path = format!("/{bucket}/{key}");
+    let sftp_bytes = sftp_read_full(sftp, &sftp_path).await?;
+    if sftp_bytes.len() != expected.len() {
+        return Err(anyhow!(
+            "SFTP read byte count mismatch for {}: expected {}, got {}",
+            sftp_path,
+            expected.len(),
+            sftp_bytes.len()
+        ));
+    }
+    let sftp_sha = Sha256::digest(&sftp_bytes);
+    if sftp_sha != expected_sha {
+        return Err(anyhow!("SFTP read SHA256 mismatch for {}", sftp_path));
+    }
+
+    Ok(())
+}
+
+/// SFTP core protocol round-trip: banner, mkdir, put, get with SHA compare, rename, delete, rmdir.
+pub async fn test_sftp_core_operations() -> Result<()> {
+    let env = ProtocolTestEnvironment::new().map_err(|e| anyhow!("{}", e))?;
+    let host_key_dir = PathBuf::from(&env.temp_dir).join("sftp_host_keys");
+    generate_host_key(&host_key_dir).await?;
+
+    info!("Starting SFTP server on {}", SFTP_ADDRESS);
+    let binary_path = rustfs_binary_path_with_features(Some("ftps,webdav,sftp"));
+    let host_key_dir_str = host_key_dir
+        .to_str()
+        .ok_or_else(|| anyhow!("host key dir path is not utf-8"))?;
+    let mut server_process = ServerProcess::new(
+        Command::new(&binary_path)
+            .env(ENV_SFTP_ENABLE, "true")
+            .env(ENV_SFTP_ADDRESS, SFTP_ADDRESS)
+            .env(ENV_SFTP_HOST_KEY_DIR, host_key_dir_str)
+            .env(ENV_SFTP_READ_ONLY, "false")
+            .env(ENV_SFTP_PART_SIZE, PART_SIZE_ENV)
+            .env(ENV_RUSTFS_ADDRESS, S3_ADDRESS)
+            .arg(&env.temp_dir)
+            .spawn()?,
+    );
+
+    let result = async {
+        ProtocolTestEnvironment::wait_for_port_ready(SFTP_PORT, 30)
+            .await
+            .map_err(|e| anyhow!("{}", e))?;
+
+        let (session, sftp) = connect_sftp().await?;
+
+        // --- 1. Subsystem canary: SFTP session reachable after password auth ---
+        // SftpSession::new completes the SFTPv3 version exchange. The
+        // canonicalize call below is a cheap round-trip that confirms
+        // the session handles real wire traffic.
+        info!("Testing SFTP: subsystem canary, server resolves '.' to an absolute path");
+        let pwd = sftp.canonicalize(".").await?;
+        assert!(!pwd.is_empty(), "server must resolve '.' to a non-empty absolute path");
+        info!("PASS: subsystem canary: server resolved '.' to {}", pwd);
+
+        // --- 2. Bucket lifecycle: mkdir then root listing ---
+        let bucket = "coretestbucket";
+        let bucket_path = format!("/{bucket}");
+        info!("Testing SFTP: mkdir bucket {}", bucket_path);
+        sftp.create_dir(&bucket_path).await?;
+        info!("PASS: mkdir bucket {}", bucket_path);
+
+        info!("Testing SFTP: root listing includes the new bucket");
+        let root_entries: Vec<String> = sftp.read_dir("/").await?.map(|e| e.file_name()).collect();
+        assert!(root_entries.iter().any(|n| n == bucket), "root listing should contain the new bucket");
+        info!("PASS: bucket {} appeared in read_dir(\"/\")", bucket);
+
+        // --- 3. Small-file round-trip with SHA256 compare ---
+        info!("Testing SFTP: small-file round-trip with SHA256 compare");
+        let small_path = format!("/{bucket}/small.txt");
+        let small_content = b"hello rustfs sftp\n";
+        let mut wf = sftp
+            .open_with_flags(&small_path, OpenFlags::CREATE | OpenFlags::TRUNCATE | OpenFlags::WRITE)
+            .await?;
+        wf.write_all(small_content).await?;
+        wf.flush().await?;
+        wf.shutdown().await?;
+
+        let mut rf = sftp.open_with_flags(&small_path, OpenFlags::READ).await?;
+        let mut buf = Vec::new();
+        rf.read_to_end(&mut buf).await?;
+        rf.shutdown().await?;
+        assert_eq!(buf.as_slice(), small_content, "small-file round-trip content mismatch");
+        let sha_in = Sha256::digest(small_content);
+        let sha_out = Sha256::digest(&buf);
+        assert_eq!(sha_in, sha_out, "small-file SHA256 mismatch");
+        info!("PASS: small-file round-trip SHA256 match");
+
+        // --- 4. Path STAT on a file and on a bucket ---
+        info!("Testing SFTP: stat on file returns size and file type");
+        let file_meta = sftp.metadata(&small_path).await?;
+        assert_eq!(file_meta.size, Some(small_content.len() as u64), "stat size mismatch");
+        assert!(file_meta.file_type().is_file(), "stat on a file must report regular file");
+        info!("PASS: stat on file reports size {} and file type", small_content.len());
+
+        info!("Testing SFTP: stat on bucket reports directory");
+        let bucket_meta = sftp.metadata(&bucket_path).await?;
+        assert!(bucket_meta.file_type().is_dir(), "stat on a bucket must report directory");
+        info!("PASS: stat on bucket reports directory");
+
+        // --- 5. SETSTAT on a file path returns ok ---
+        // SETSTAT is a no-op on the server because S3 has no POSIX mtime or permission
+        // semantics, but it must still return ok. Clients that send SETSTAT after every
+        // transfer (rsync, WinSCP) treat a non-ok status as a transfer failure.
+        info!("Testing SFTP: setstat on a path returns ok");
+        let attrs = FileAttributes {
+            permissions: Some(0o644),
+            ..FileAttributes::default()
+        };
+        sftp.set_metadata(&small_path, attrs).await?;
+        info!("PASS: setstat returned ok");
+
+        // --- 6. Rename within bucket and listing reflects it ---
+        info!("Testing SFTP: rename within bucket");
+        let renamed = format!("/{bucket}/renamed.txt");
+        sftp.rename(&small_path, &renamed).await?;
+        info!("PASS: rename {} -> {}", small_path, renamed);
+
+        info!("Testing SFTP: listing reflects rename");
+        let bucket_entries: Vec<String> = sftp.read_dir(&bucket_path).await?.map(|e| e.file_name()).collect();
+        assert!(bucket_entries.iter().any(|n| n == "renamed.txt"), "renamed file must be listed");
+        assert!(!bucket_entries.iter().any(|n| n == "small.txt"), "pre-rename name must be gone");
+        info!("PASS: directory listing reflects the rename");
+
+        // --- 7. Multipart round-trip across the part-size boundary ---
+        // MULTIPART_SIZE is paired with RUSTFS_SFTP_PART_SIZE above so the upload crosses the
+        // multipart threshold regardless of server defaults.
+        info!("Testing SFTP: multipart-sized round-trip with SHA256 compare");
+        let big_path = format!("/{bucket}/big.bin");
+        let big_content: Vec<u8> = (0..MULTIPART_SIZE).map(|i| (i as u8).wrapping_mul(31)).collect();
+        let mut bwf = sftp
+            .open_with_flags(&big_path, OpenFlags::CREATE | OpenFlags::TRUNCATE | OpenFlags::WRITE)
+            .await?;
+        bwf.write_all(&big_content).await?;
+        bwf.flush().await?;
+        bwf.shutdown().await?;
+
+        let mut brf = sftp.open_with_flags(&big_path, OpenFlags::READ).await?;
+        let mut big_buf = Vec::with_capacity(MULTIPART_SIZE);
+        brf.read_to_end(&mut big_buf).await?;
+        brf.shutdown().await?;
+        assert_eq!(big_buf.len(), MULTIPART_SIZE, "multipart round-trip length mismatch");
+        let big_in = Sha256::digest(&big_content);
+        let big_out = Sha256::digest(&big_buf);
+        assert_eq!(big_in, big_out, "multipart SHA256 mismatch");
+        info!("PASS: multipart round-trip SHA256 match ({} bytes)", MULTIPART_SIZE);
+
+        // --- 8. Negative cases: symlink, open nonexistent, read_dir nonexistent, path escape ---
+        info!("Testing SFTP: symlink returns an error");
+        let symlink_err = sftp.symlink(&big_path, &format!("/{bucket}/shortcut")).await;
+        assert!(symlink_err.is_err(), "symlink must be rejected by the server");
+        info!("PASS: symlink rejected");
+
+        info!("Testing SFTP: open of nonexistent file returns an error");
+        let missing_path = format!("/{bucket}/not_here.txt");
+        let missing_err = sftp.open_with_flags(&missing_path, OpenFlags::READ).await;
+        assert!(missing_err.is_err(), "open of a nonexistent path must error");
+        info!("PASS: open of nonexistent file rejected");
+
+        info!("Testing SFTP: read_dir of nonexistent bucket returns an error");
+        let missing_bucket = sftp.read_dir("/nosuchbucket").await;
+        assert!(missing_bucket.is_err(), "read_dir of a nonexistent bucket must error");
+        info!("PASS: read_dir of nonexistent bucket rejected");
+
+        info!("Testing SFTP: path traversal cannot escape the storage root");
+        let traversal = sftp.read_dir("/../../../etc").await;
+        assert!(traversal.is_err(), "path traversal must be rejected or resolve to a nonexistent bucket");
+        info!("PASS: path traversal rejected");
+
+        // --- Spec-letter assertion: APPEND open-flag returns an error ---
+        // The driver maps APPEND to OpUnsupported because S3 has no append
+        // primitive. Open requests with APPEND must return a failure rather
+        // than allow a silently mistruncated upload.
+        info!("Testing SFTP: open with APPEND returns an error");
+        let append_err = sftp.open_with_flags(&renamed, OpenFlags::APPEND | OpenFlags::WRITE).await;
+        assert!(append_err.is_err(), "open with APPEND must error");
+        info!("PASS: open with APPEND rejected");
+
+        // --- Spec-letter assertion: O_EXCL on existing path returns an error ---
+        // CREATE | EXCLUDE on a key that already exists must fail. The
+        // existing renamed.txt is the target. EXCLUDE without WRITE is
+        // rejected by the russh-sftp client itself, so WRITE is included.
+        // TRUNCATE is included because the driver requires WRITE | CREATE
+        // | TRUNCATE on every accepted write OPEN.
+        info!("Testing SFTP: open with CREATE + EXCLUDE on existing path returns an error");
+        let excl_err = sftp
+            .open_with_flags(&renamed, OpenFlags::CREATE | OpenFlags::TRUNCATE | OpenFlags::EXCLUDE | OpenFlags::WRITE)
+            .await;
+        assert!(excl_err.is_err(), "CREATE+EXCLUDE on existing path must error");
+        info!("PASS: CREATE+EXCLUDE on existing path rejected");
+
+        // --- WRITE without CREATE or TRUNCATE is rejected at OPEN ---
+        // The streaming write path overwrites the entire object at
+        // close. A WRITE-only OPEN asks for partial-write semantics
+        // the server cannot honour against S3, so the OPEN is
+        // rejected before any handle is allocated.
+        info!("Testing SFTP: open with WRITE only returns an error");
+        let write_only_err = sftp.open_with_flags(&renamed, OpenFlags::WRITE).await;
+        assert!(write_only_err.is_err(), "WRITE without CREATE or TRUNCATE must be rejected at OPEN");
+        info!("PASS: WRITE only rejected");
+
+        // --- WRITE | CREATE without TRUNCATE is rejected at OPEN ---
+        // Without TRUNCATE the client is asking for create-or-modify-
+        // existing semantics. The server cannot deliver that against
+        // S3, so the OPEN is rejected before any handle is allocated.
+        info!("Testing SFTP: open with WRITE | CREATE without TRUNCATE returns an error");
+        let create_no_trunc_err = sftp.open_with_flags(&renamed, OpenFlags::WRITE | OpenFlags::CREATE).await;
+        assert!(create_no_trunc_err.is_err(), "WRITE | CREATE without TRUNCATE must be rejected at OPEN");
+        info!("PASS: WRITE | CREATE without TRUNCATE rejected");
+
+        // --- Spec-letter assertion: bad password is rejected (separate session) ---
+        // Fresh russh session with wrong credentials. The authenticated
+        // handle is left untouched. Bad auth must not succeed.
+        info!("Testing SFTP: second russh session with wrong password is rejected");
+        let bad_config = Arc::new(client::Config::default());
+        let mut bad_session = client::connect(bad_config, SFTP_ADDRESS, AcceptAnyServerKey).await?;
+        let bad_auth = bad_session.authenticate_password(DEFAULT_ACCESS_KEY, "wrong-secret").await?;
+        assert!(!bad_auth.success(), "bad-password authentication must not succeed");
+        // Discard the disconnect Result. A server that already rejected
+        // auth can return an error here, but the assert above already
+        // pins the auth outcome.
+        let _ = bad_session.disconnect(russh::Disconnect::ByApplication, "", "en").await;
+        info!("PASS: bad-password authentication rejected");
+
+        // --- Cross-protocol setup: aws-sdk-s3 client against the same server ---
+        // The rustfs binary spawned for this suite serves both SFTP on port
+        // 9022 and S3 on port 9000. The S3 stack may need a moment to finish
+        // initialising after TCP is listening, so list_buckets is polled
+        // until it succeeds before any cross-protocol assertion runs.
+        info!("Testing SFTP: prepare aws-sdk-s3 client and wait for S3 readiness");
+        let s3 = build_test_s3_client(S3_ENDPOINT);
+        wait_for_s3_ready(&s3, S3_READY_ATTEMPTS).await?;
+        info!("PASS: S3 endpoint reachable from cross-protocol client");
+
+        // --- SFTP write, S3 read: SHA256 round-trip ---
+        // SFTP creates the object, then assert_cross_protocol_sha_match
+        // fetches it via both S3 GetObject and SFTP READ and compares
+        // each result against the SHA256 of the original payload. Both
+        // sides must match byte-exact, which proves the storage layer
+        // returns the same bytes regardless of wire protocol.
+        info!("Testing SFTP: SFTP write then S3 read, SHA256 round-trip");
+        let sftp_to_s3_key = "sftp_written.bin";
+        let sftp_to_s3_path = format!("/{bucket}/{sftp_to_s3_key}");
+        let sftp_to_s3_content: Vec<u8> = (0..S3_WRITTEN_SIZE).map(|i| (i as u8).wrapping_mul(17)).collect();
+        let mut wf = sftp
+            .open_with_flags(&sftp_to_s3_path, OpenFlags::CREATE | OpenFlags::TRUNCATE | OpenFlags::WRITE)
+            .await?;
+        wf.write_all(&sftp_to_s3_content).await?;
+        wf.flush().await?;
+        wf.shutdown().await?;
+        assert_cross_protocol_sha_match(&s3, &sftp, bucket, sftp_to_s3_key, &sftp_to_s3_content).await?;
+        info!("PASS: SFTP-written object matches via S3 GetObject and SFTP READ");
+
+        // --- S3 write, SFTP read: SHA256 round-trip ---
+        // aws-sdk-s3 PutObject writes a fixed deterministic payload. Both
+        // sides then read it back and SHA-compare.
+        info!("Testing SFTP: S3 write then SFTP read, SHA256 round-trip");
+        let s3_to_sftp_key = "s3_written.bin";
+        let s3_to_sftp_content: Vec<u8> = (0..S3_WRITTEN_SIZE).map(|i| (i as u8).wrapping_mul(31)).collect();
+        s3.put_object()
+            .bucket(bucket)
+            .key(s3_to_sftp_key)
+            .body(ByteStream::from(s3_to_sftp_content.clone()))
+            .send()
+            .await
+            .map_err(|e| anyhow!("S3 PutObject {}/{} failed: {:?}", bucket, s3_to_sftp_key, e))?;
+        assert_cross_protocol_sha_match(&s3, &sftp, bucket, s3_to_sftp_key, &s3_to_sftp_content).await?;
+        info!("PASS: S3-written object matches via S3 GetObject and SFTP READ");
+
+        // --- Cross-API directory visibility: SFTP mkdir, S3 ListObjectsV2 ---
+        // SFTP mkdir writes a __XLDIR__ marker. The rustfs S3 listing path
+        // decodes that marker back to a trailing-slash key, so the asserted
+        // pattern is "subdir_sftp/".
+        info!("Testing SFTP: SFTP-created sub-directory visible via S3 ListObjectsV2");
+        let sftp_subdir_name = "subdir_sftp";
+        let sftp_subdir_path = format!("/{bucket}/{sftp_subdir_name}");
+        sftp.create_dir(&sftp_subdir_path).await?;
+        let listed = s3
+            .list_objects_v2()
+            .bucket(bucket)
+            .prefix(sftp_subdir_name)
+            .send()
+            .await
+            .map_err(|e| anyhow!("S3 ListObjectsV2 {} failed: {:?}", bucket, e))?;
+        let listed_keys: Vec<String> = listed
+            .contents()
+            .iter()
+            .filter_map(|obj| obj.key().map(|s| s.to_string()))
+            .collect();
+        let visible_via_s3 = listed_keys
+            .iter()
+            .any(|k| k == &format!("{sftp_subdir_name}/") || k == &format!("{sftp_subdir_name}{XLDIR_SUFFIX}"));
+        assert!(
+            visible_via_s3,
+            "SFTP-created sub-directory must appear in S3 ListObjectsV2: keys returned were {listed_keys:?}"
+        );
+        info!("PASS: SFTP mkdir visible to S3 ListObjectsV2");
+
+        // --- Cross-API directory visibility: S3 marker, SFTP readdir ---
+        // aws-sdk-s3 PutObject writes a zero-byte marker keyed with the
+        // __XLDIR__ suffix. SFTP readdir must decode the marker back to a
+        // bare directory entry whose file_type reports as a directory.
+        info!("Testing SFTP: S3-created __XLDIR__ marker visible via SFTP readdir");
+        let s3_subdir_name = "subdir_s3";
+        let s3_subdir_marker_key = format!("{s3_subdir_name}{XLDIR_SUFFIX}");
+        s3.put_object()
+            .bucket(bucket)
+            .key(&s3_subdir_marker_key)
+            .body(ByteStream::from_static(b""))
+            .send()
+            .await
+            .map_err(|e| anyhow!("S3 PutObject {}/{} failed: {:?}", bucket, s3_subdir_marker_key, e))?;
+        let bucket_entries: Vec<(String, bool)> = sftp
+            .read_dir(&bucket_path)
+            .await?
+            .map(|entry| (entry.file_name(), entry.file_type().is_dir()))
+            .collect();
+        let visible_via_sftp = bucket_entries.iter().any(|(name, is_dir)| name == s3_subdir_name && *is_dir);
+        assert!(
+            visible_via_sftp,
+            "S3-created marker must appear as a directory in SFTP readdir: entries were {bucket_entries:?}"
+        );
+        info!("PASS: S3 marker visible to SFTP readdir as a directory");
+
+        // --- Pre-cleanup of cross-protocol fixtures ---
+        // Removes the new files and sub-directories so the existing rmdir
+        // call below operates against an empty bucket.
+        info!("Testing SFTP: pre-cleanup of cross-protocol fixtures");
+        sftp.remove_file(&format!("/{bucket}/{sftp_to_s3_key}")).await?;
+        sftp.remove_file(&format!("/{bucket}/{s3_to_sftp_key}")).await?;
+        sftp.remove_dir(&sftp_subdir_path).await?;
+        sftp.remove_dir(&format!("/{bucket}/{s3_subdir_name}")).await?;
+        info!("PASS: cross-protocol fixtures removed");
+
+        // --- 9. Cleanup: delete objects, rmdir bucket, confirm root empty ---
+        info!("Testing SFTP: delete objects then rmdir bucket");
+        sftp.remove_file(&renamed).await?;
+        sftp.remove_file(&big_path).await?;
+        sftp.remove_dir(&bucket_path).await?;
+        info!("PASS: delete + rmdir leaves the root empty");
+
+        let final_entries: Vec<String> = sftp.read_dir("/").await?.map(|e| e.file_name()).collect();
+        assert!(!final_entries.iter().any(|n| n == bucket), "bucket must be gone after rmdir");
+        info!("PASS: root listing no longer includes the deleted bucket");
+
+        drop(sftp);
+        session.disconnect(russh::Disconnect::ByApplication, "", "en").await?;
+        info!("SFTP core tests passed");
+        Ok::<(), anyhow::Error>(())
+    }
+    .await;
+
+    // Discard kill/wait errors on the teardown path: the test result
+    // above is the binding outcome, and a server that has already
+    // exited produces an error here that carries no useful signal.
+    server_process.kill_and_wait().await;
+
+    result
+}
+
+/// Idle-timeout regression: the server must close an SFTP session that
+/// remains inactive past RUSTFS_SFTP_IDLE_TIMEOUT.
+///
+/// Spawns its own rustfs binary on dedicated SFTP and S3 ports so it can run
+/// independently of the core protocol suite. The disconnect check issues a
+/// cheap SFTP request after the wait window. The same error path runs in any
+/// client when the server-initiated SSH_MSG_DISCONNECT arrives. The assertion
+/// does not pin a specific russh error variant because the exact error
+/// returned on server-initiated disconnect depends on timing.
+pub async fn test_sftp_idle_timeout_disconnects() -> Result<()> {
+    let env = ProtocolTestEnvironment::new().map_err(|e| anyhow!("{}", e))?;
+    let host_key_dir = PathBuf::from(&env.temp_dir).join("sftp_host_keys");
+    generate_host_key(&host_key_dir).await?;
+
+    info!("Starting SFTP server with idle timeout {} s on {}", IDLE_TIMEOUT_SECS, IDLE_SFTP_ADDRESS);
+    let binary_path = rustfs_binary_path_with_features(Some("ftps,webdav,sftp"));
+    let host_key_dir_str = host_key_dir
+        .to_str()
+        .ok_or_else(|| anyhow!("host key dir path is not utf-8"))?;
+    let mut server_process = ServerProcess::new(
+        Command::new(&binary_path)
+            .env(ENV_SFTP_ENABLE, "true")
+            .env(ENV_SFTP_ADDRESS, IDLE_SFTP_ADDRESS)
+            .env(ENV_SFTP_HOST_KEY_DIR, host_key_dir_str)
+            .env(ENV_SFTP_READ_ONLY, "false")
+            .env(ENV_SFTP_PART_SIZE, PART_SIZE_ENV)
+            .env(ENV_SFTP_IDLE_TIMEOUT, IDLE_TIMEOUT_SECS.to_string())
+            .env(ENV_RUSTFS_ADDRESS, IDLE_S3_ADDRESS)
+            .arg(&env.temp_dir)
+            .spawn()?,
+    );
+
+    let result = async {
+        ProtocolTestEnvironment::wait_for_port_ready(IDLE_SFTP_PORT, 30)
+            .await
+            .map_err(|e| anyhow!("{}", e))?;
+
+        let (session, sftp) = connect_sftp_to(IDLE_SFTP_ADDRESS).await?;
+
+        // Confirm the session is live before the wait so a failure in the
+        // post-wait read can be attributed to the idle timer rather than to
+        // a setup defect.
+        let pwd = sftp.canonicalize(".").await?;
+        assert!(!pwd.is_empty(), "server must resolve '.' to a non-empty absolute path");
+
+        info!("Idle wait: sleeping {} s past idle timeout {} s", IDLE_WAIT_SECS, IDLE_TIMEOUT_SECS);
+        sleep(Duration::from_secs(IDLE_WAIT_SECS)).await;
+
+        let post_idle = sftp.read_dir("/").await;
+        assert!(
+            post_idle.is_err(),
+            "SFTP request after idle wait must error once the server has closed the session"
+        );
+        info!("PASS: SFTP request after idle wait returned an error");
+
+        drop(sftp);
+        // Discard the disconnect Result. The server has already closed the
+        // session via the idle-timeout path the test is probing. A client
+        // disconnect against a half-closed transport may itself return Err
+        // with no useful signal.
+        let _ = session.disconnect(russh::Disconnect::ByApplication, "", "en").await;
+        Ok::<(), anyhow::Error>(())
+    }
+    .await;
+
+    // Discard kill/wait errors on the teardown path: the test result above
+    // is the binding outcome, and a server that has already exited produces
+    // an error here that carries no useful signal.
+    server_process.kill_and_wait().await;
+
+    result
+}

crates/e2e_test/src/protocols/sftp_helpers.rs

@@ -0,0 +1,194 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Shared helpers for SFTP protocol tests
+//!
+//! An accept-any host-key handler, an ed25519 host-key generator that
+//! matches the rustfs config loader permission gates, a russh client
+//! connector that authenticates with the default access key, and an
+//! SFTP-read-to-vec helper.
+
+use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY};
+use anyhow::{Result, anyhow};
+use aws_sdk_s3::Client as S3Client;
+use aws_sdk_s3::config::{Credentials, Region};
+use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
+use russh::client::{self, Handle};
+use russh::keys::ssh_key::LineEnding;
+use russh::keys::{Algorithm, PrivateKey, PublicKey};
+use russh_sftp::client::SftpSession;
+use russh_sftp::protocol::OpenFlags;
+use std::path::Path;
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::io::{AsyncReadExt, AsyncWriteExt};
+use tokio::process::Child;
+use tokio::time::sleep;
+use tracing::info;
+
+/// Accept-any server-key client handler. The test server uses a host key
+/// generated fresh at the start of each run, so strict verification would
+/// always fail. The suite exercises the auth path, not the host-key-trust
+/// path, which is out of scope for this suite.
+pub struct AcceptAnyServerKey;
+
+impl client::Handler for AcceptAnyServerKey {
+    type Error = anyhow::Error;
+
+    async fn check_server_key(&mut self, _server_public_key: &PublicKey) -> Result<bool, Self::Error> {
+        Ok(true)
+    }
+}
+
+/// Generate an ed25519 host key pair in host_key_dir with mode 0600. The
+/// key is generated in-process via russh::keys so the test suite has no
+/// host-tooling dependency on ssh-keygen, which is absent on Alpine,
+/// distroless, scratch, and Windows images. The RustFS config loader
+/// accepts the OpenSSH private-key format that PrivateKey::to_openssh
+/// emits. Both the private key and the .pub file need 0600 because the
+/// loader scans every entry in the directory and rejects the whole
+/// directory as insecure unless each file is 0600 or 0400.
+pub async fn generate_host_key(host_key_dir: &Path) -> Result<()> {
+    tokio::fs::create_dir_all(host_key_dir).await?;
+    let key_path = host_key_dir.join("ssh_host_ed25519_key");
+    let pub_path = host_key_dir.join("ssh_host_ed25519_key.pub");
+
+    let private_key =
+        PrivateKey::random(&mut rand::rng(), Algorithm::Ed25519).map_err(|e| anyhow!("ed25519 key generation failed: {e}"))?;
+    let private_pem = private_key
+        .to_openssh(LineEnding::LF)
+        .map_err(|e| anyhow!("OpenSSH private-key encode failed: {e}"))?;
+    let public_text = private_key
+        .public_key()
+        .to_openssh()
+        .map_err(|e| anyhow!("OpenSSH public-key encode failed: {e}"))?;
+
+    tokio::fs::write(&key_path, private_pem.as_bytes()).await?;
+    tokio::fs::write(&pub_path, format!("{public_text}\n")).await?;
+
+    #[cfg(unix)]
+    {
+        use std::os::unix::fs::PermissionsExt;
+        for path in [&key_path, &pub_path] {
+            let mut perm = std::fs::metadata(path)?.permissions();
+            perm.set_mode(0o600);
+            std::fs::set_permissions(path, perm)?;
+        }
+    }
+    Ok(())
+}
+
+/// Owns a spawned rustfs server child process and guarantees the process
+/// is sent SIGKILL even if the test panics. The wrapper exists because
+/// tokio::process::Child does not kill on Drop on stable Rust, so a
+/// panicking test would otherwise leak a running rustfs binary that
+/// keeps its listener port held until the test runner exits.
+///
+/// Use kill_and_wait on the success and Err paths to reap the child
+/// cleanly; Drop only fires the synchronous SIGKILL when those paths
+/// were skipped (panic unwind, runtime abort).
+pub struct ServerProcess {
+    inner: Option<Child>,
+}
+
+impl ServerProcess {
+    pub fn new(child: Child) -> Self {
+        Self { inner: Some(child) }
+    }
+
+    /// Borrow the inner Child for callers that need stdout piping or
+    /// other tokio::process APIs.
+    pub fn child_mut(&mut self) -> &mut Child {
+        self.inner.as_mut().expect("ServerProcess: child already taken")
+    }
+
+    /// Async kill plus wait. Idempotent. Use on every success and Err
+    /// path. After this returns, Drop becomes a no-op.
+    pub async fn kill_and_wait(&mut self) {
+        if let Some(mut child) = self.inner.take() {
+            let _ = child.kill().await;
+            let _ = child.wait().await;
+        }
+    }
+}
+
+impl Drop for ServerProcess {
+    fn drop(&mut self) {
+        if let Some(child) = self.inner.as_mut() {
+            // Synchronous SIGKILL via the kernel. Runs even on panic
+            // unwind. wait() is skipped here (Drop cannot await), so
+            // the process becomes a zombie reaped by the runtime.
+            let _ = child.start_kill();
+        }
+    }
+}
+
+/// Open a russh client session against the given address, authenticate
+/// with the default access key, request the SFTP subsystem, and return
+/// the session handle plus the SFTP wrapper. The handle is returned so
+/// the caller can keep the underlying SSH transport alive for the full
+/// session and disconnect cleanly afterwards.
+pub async fn connect_sftp_to(address: &str) -> Result<(Handle<AcceptAnyServerKey>, SftpSession)> {
+    let config = Arc::new(client::Config::default());
+    let mut session = client::connect(config, address, AcceptAnyServerKey).await?;
+    let auth = session.authenticate_password(DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY).await?;
+    if !auth.success() {
+        return Err(anyhow!("SFTP password auth rejected"));
+    }
+    let channel = session.channel_open_session().await?;
+    channel.request_subsystem(true, "sftp").await?;
+    let sftp = SftpSession::new(channel.into_stream()).await?;
+    Ok((session, sftp))
+}
+
+/// Read an SFTP object into memory.
+pub async fn sftp_read_full(sftp: &SftpSession, path: &str) -> Result<Vec<u8>> {
+    let mut file = sftp.open_with_flags(path, OpenFlags::READ).await?;
+    let mut buf = Vec::new();
+    file.read_to_end(&mut buf).await?;
+    file.shutdown().await?;
+    Ok(buf)
+}
+
+/// Construct an aws-sdk-s3 client wired against an http rustfs endpoint.
+/// Uses the same credential constants the SFTP session authenticates with so
+/// both protocols see the same backend identity.
+pub fn build_test_s3_client(endpoint_url: &str) -> S3Client {
+    let credentials = Credentials::new(DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, None, None, "sftp-helpers");
+    let mut config = aws_sdk_s3::Config::builder()
+        .credentials_provider(credentials)
+        .region(Region::new("us-east-1"))
+        .endpoint_url(endpoint_url)
+        .force_path_style(true)
+        .behavior_version_latest();
+    if endpoint_url.starts_with("http://") {
+        config = config.http_client(SmithyHttpClientBuilder::new().build_http());
+    }
+    S3Client::from_conf(config.build())
+}
+
+/// Poll the S3 endpoint until ListBuckets returns successfully or the
+/// attempt budget is exhausted. The TCP-level wait_for_port_ready check is
+/// not enough on its own because rustfs accepts connections before the S3
+/// stack has finished initialising.
+pub async fn wait_for_s3_ready(client: &S3Client, max_attempts: u32) -> Result<()> {
+    for attempt in 0..max_attempts {
+        if client.list_buckets().send().await.is_ok() {
+            info!("S3 endpoint ready after {} attempts", attempt + 1);
+            return Ok(());
+        }
+        sleep(Duration::from_secs(1)).await;
+    }
+    Err(anyhow!("S3 endpoint did not become ready"))
+}

crates/e2e_test/src/protocols/test_env.rs

@@ -32,8 +32,13 @@ impl ProtocolTestEnvironment {
     /// Create a new test environment
     /// This environment won't stop any server when dropped
     pub fn new() -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
-        let temp_dir = format!("/tmp/rustfs_protocol_test_{}", uuid::Uuid::new_v4());
-        std::fs::create_dir_all(&temp_dir)?;
+        let mut path = std::env::temp_dir();
+        path.push(format!("rustfs_protocol_test_{}", uuid::Uuid::new_v4()));
+        std::fs::create_dir_all(&path)?;
+        let temp_dir = path
+            .to_str()
+            .ok_or_else(|| format!("temp dir path is not utf-8: {}", path.display()))?
+            .to_string();
 
         Ok(Self { temp_dir })
     }

crates/e2e_test/src/protocols/test_runner.rs

@@ -15,8 +15,14 @@
 //! Protocol test runner
 
 use crate::common::init_logging;
+use crate::common::{requested_rustfs_build_features, rustfs_build_feature_enabled};
 use crate::protocols::ftps_core::test_ftps_core_operations;
+use crate::protocols::sftp_compliance::{
+    test_sftp_compliance_readonly, test_sftp_compliance_standalone, test_sftp_compliance_suite,
+};
+use crate::protocols::sftp_core::{test_sftp_core_operations, test_sftp_idle_timeout_disconnects};
 use crate::protocols::webdav_core::test_webdav_core_operations;
+use serial_test::serial;
 use std::time::Instant;
 use tokio::time::{Duration, sleep};
 use tracing::{error, info};
@@ -54,21 +60,22 @@ pub struct ProtocolTestSuite {
 
 #[derive(Debug, Clone)]
 struct TestDefinition {
-    name: String,
+    name: &'static str,
+    required_feature: &'static str,
 }
 
 impl ProtocolTestSuite {
     /// Create default test suite
     pub fn new() -> Self {
-        let tests = vec![
-            TestDefinition {
-                name: "test_ftps_core_operations".to_string(),
-            },
-            TestDefinition {
-                name: "test_webdav_core_operations".to_string(),
-            },
-        ];
+        let requested_features = requested_rustfs_build_features();
+        Self::with_requested_features(requested_features.as_deref())
+    }
 
+    fn with_requested_features(requested_features: Option<&str>) -> Self {
+        let tests = all_protocol_tests()
+            .into_iter()
+            .filter(|test| rustfs_build_feature_enabled(requested_features, test.required_feature))
+            .collect();
         Self { tests }
     }
 
@@ -84,7 +91,7 @@ impl ProtocolTestSuite {
 
         // Run tests
         for (i, test_def) in self.tests.iter().enumerate() {
-            let test_description = match test_def.name.as_str() {
+            let test_description = match test_def.name {
                 "test_ftps_core_operations" => {
                     info!("=== Starting FTPS Module Test ===");
                     "FTPS core operations (put, ls, mkdir, rmdir, delete)"
@@ -93,6 +100,26 @@ impl ProtocolTestSuite {
                     info!("=== Starting WebDAV Core Test ===");
                     "WebDAV core operations (MKCOL, PUT, GET, DELETE, PROPFIND)"
                 }
+                "test_sftp_core_operations" => {
+                    info!("=== Starting SFTP Core Test ===");
+                    "SFTP core operations (banner, mkdir, put, get with SHA compare, rename, delete, rmdir)"
+                }
+                "test_sftp_compliance_suite" => {
+                    info!("=== Starting SFTP Compliance Suite ===");
+                    "SFTP compliance regression suite (zero-byte, mutation rejection, traversal, rename, implicit dirs, FSETSTAT)"
+                }
+                "test_sftp_compliance_readonly" => {
+                    info!("=== Starting SFTP Read-Only Compliance Suite ===");
+                    "SFTP read-only mode (RUSTFS_SFTP_READ_ONLY=true rejects mutations and allows reads)"
+                }
+                "test_sftp_idle_timeout_disconnects" => {
+                    info!("=== Starting SFTP Idle-Timeout Test ===");
+                    "SFTP idle-timeout disconnects (server closes the session past RUSTFS_SFTP_IDLE_TIMEOUT)"
+                }
+                "test_sftp_compliance_standalone" => {
+                    info!("=== Starting SFTP Standalone-Server Compliance Suite ===");
+                    "SFTP standalone-server compliance suite"
+                }
                 _ => "",
             };
 
@@ -107,11 +134,11 @@ impl ProtocolTestSuite {
             match result {
                 Ok(_) => {
                     info!("Test passed: {} ({:.2}s)", test_def.name, test_duration.as_secs_f64());
-                    results.push(TestResult::success(test_def.name.clone()));
+                    results.push(TestResult::success(test_def.name.to_string()));
                 }
                 Err(e) => {
                     error!("Test failed: {} ({:.2}s): {}", test_def.name, test_duration.as_secs_f64(), e);
-                    results.push(TestResult::failure(test_def.name.clone(), e.to_string()));
+                    results.push(TestResult::failure(test_def.name.to_string(), e.to_string()));
                 }
             }
 
@@ -129,9 +156,14 @@ impl ProtocolTestSuite {
 
     /// Run a single test
     async fn run_single_test(&self, test_def: &TestDefinition) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        match test_def.name.as_str() {
+        match test_def.name {
             "test_ftps_core_operations" => test_ftps_core_operations().await.map_err(|e| e.into()),
             "test_webdav_core_operations" => test_webdav_core_operations().await.map_err(|e| e.into()),
+            "test_sftp_core_operations" => test_sftp_core_operations().await.map_err(|e| e.into()),
+            "test_sftp_compliance_suite" => test_sftp_compliance_suite().await.map_err(|e| e.into()),
+            "test_sftp_compliance_readonly" => test_sftp_compliance_readonly().await.map_err(|e| e.into()),
+            "test_sftp_idle_timeout_disconnects" => test_sftp_idle_timeout_disconnects().await.map_err(|e| e.into()),
+            "test_sftp_compliance_standalone" => test_sftp_compliance_standalone().await.map_err(|e| e.into()),
             _ => Err(format!("Test {} not implemented", test_def.name).into()),
         }
     }
@@ -141,6 +173,10 @@ impl ProtocolTestSuite {
         info!("=== Test Suite Summary ===");
         info!("Total duration: {:.2}s", total_duration.as_secs_f64());
         info!("Total tests: {}", results.len());
+        if results.is_empty() {
+            info!("No protocol tests scheduled for the requested feature set");
+            return;
+        }
 
         let passed = results.iter().filter(|r| r.success).count();
         let failed = results.len() - passed;
@@ -158,8 +194,42 @@ impl ProtocolTestSuite {
     }
 }
 
+fn all_protocol_tests() -> Vec<TestDefinition> {
+    vec![
+        TestDefinition {
+            name: "test_ftps_core_operations",
+            required_feature: "ftps",
+        },
+        TestDefinition {
+            name: "test_webdav_core_operations",
+            required_feature: "webdav",
+        },
+        TestDefinition {
+            name: "test_sftp_core_operations",
+            required_feature: "sftp",
+        },
+        TestDefinition {
+            name: "test_sftp_compliance_suite",
+            required_feature: "sftp",
+        },
+        TestDefinition {
+            name: "test_sftp_compliance_readonly",
+            required_feature: "sftp",
+        },
+        TestDefinition {
+            name: "test_sftp_idle_timeout_disconnects",
+            required_feature: "sftp",
+        },
+        TestDefinition {
+            name: "test_sftp_compliance_standalone",
+            required_feature: "sftp",
+        },
+    ]
+}
+
 /// Test suite
 #[tokio::test]
+#[serial]
 async fn test_protocol_core_suite() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     let suite = ProtocolTestSuite::new();
     let results = suite.run_test_suite().await;
@@ -172,3 +242,72 @@ async fn test_protocol_core_suite() -> Result<(), Box<dyn std::error::Error + Se
     info!("All protocol tests passed");
     Ok(())
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn scheduled_names(suite: ProtocolTestSuite) -> Vec<&'static str> {
+        suite.tests.into_iter().map(|test| test.name).collect()
+    }
+
+    #[test]
+    fn schedules_all_protocol_tests_without_feature_filter() {
+        let names = scheduled_names(ProtocolTestSuite::with_requested_features(None));
+
+        assert_eq!(names.len(), 7);
+        assert!(names.contains(&"test_ftps_core_operations"));
+        assert!(names.contains(&"test_webdav_core_operations"));
+        assert!(names.contains(&"test_sftp_core_operations"));
+        assert!(names.contains(&"test_sftp_compliance_standalone"));
+    }
+
+    #[test]
+    fn schedules_only_requested_non_sftp_protocols() {
+        let names = scheduled_names(ProtocolTestSuite::with_requested_features(Some("ftps, webdav")));
+
+        assert_eq!(names, vec!["test_ftps_core_operations", "test_webdav_core_operations"]);
+    }
+
+    #[test]
+    fn schedules_all_sftp_entries_for_sftp_feature() {
+        let names = scheduled_names(ProtocolTestSuite::with_requested_features(Some("sftp")));
+
+        assert_eq!(
+            names,
+            vec![
+                "test_sftp_core_operations",
+                "test_sftp_compliance_suite",
+                "test_sftp_compliance_readonly",
+                "test_sftp_idle_timeout_disconnects",
+                "test_sftp_compliance_standalone",
+            ]
+        );
+    }
+
+    #[test]
+    fn feature_filter_is_case_insensitive() {
+        let names = scheduled_names(ProtocolTestSuite::with_requested_features(Some("SFTP")));
+
+        assert_eq!(names.len(), 5);
+        assert!(names.iter().all(|name| name.contains("sftp")));
+    }
+
+    #[test]
+    fn full_feature_schedules_all_protocol_tests() {
+        let names = scheduled_names(ProtocolTestSuite::with_requested_features(Some("full")));
+
+        assert_eq!(names.len(), 7);
+        assert!(names.contains(&"test_ftps_core_operations"));
+        assert!(names.contains(&"test_webdav_core_operations"));
+        assert!(names.contains(&"test_sftp_core_operations"));
+        assert!(names.contains(&"test_sftp_compliance_standalone"));
+    }
+
+    #[test]
+    fn schedules_no_tests_when_requested_features_have_no_protocols() {
+        let names = scheduled_names(ProtocolTestSuite::with_requested_features(Some("diagnostics")));
+
+        assert!(names.is_empty());
+    }
+}

crates/e2e_test/src/protocols/webdav_core.rs

@@ -14,17 +14,24 @@
 
 //! Core WebDAV tests
 
+use crate::common::local_http_client;
 use crate::common::rustfs_binary_path_with_features;
 use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, ProtocolTestEnvironment};
 use anyhow::Result;
 use base64::Engine;
+use http::header::{CONTENT_TYPE, HOST};
 use reqwest::Client;
+use rustfs_signer::constants::UNSIGNED_PAYLOAD;
+use rustfs_signer::sign_v4;
+use s3s::Body;
+use serial_test::serial;
 use tokio::process::Command;
 use tracing::info;
 
 // Fixed WebDAV port for testing
 const WEBDAV_PORT: u16 = 9080;
 const WEBDAV_ADDRESS: &str = "127.0.0.1:9080";
+const S3_TEST_ADDRESS: &str = "127.0.0.1:9010";
 
 /// Create HTTP client with basic auth
 fn create_client() -> Client {
@@ -36,19 +43,114 @@ fn create_client() -> Client {
 
 /// Get basic auth header value
 fn basic_auth_header() -> String {
-    let credentials = format!("{}:{}", DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY);
+    basic_auth_header_for(DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY)
+}
+
+fn basic_auth_header_for(access_key: &str, secret_key: &str) -> String {
+    let credentials = format!("{}:{}", access_key, secret_key);
     let encoded = base64::engine::general_purpose::STANDARD.encode(credentials);
     format!("Basic {}", encoded)
 }
 
+async fn signed_admin_request(
+    method: http::Method,
+    url: &str,
+    body: Option<Vec<u8>>,
+    content_type: Option<&str>,
+) -> Result<reqwest::Response> {
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri
+        .authority()
+        .ok_or_else(|| anyhow::anyhow!("request URL missing authority"))?
+        .to_string();
+    let mut request = http::Request::builder().method(method.clone()).uri(uri);
+    request = request.header(HOST, authority);
+    request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
+    if let Some(content_type) = content_type {
+        request = request.header(CONTENT_TYPE, content_type);
+    }
+
+    let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default();
+    let signed = sign_v4(
+        request.body(Body::empty())?,
+        content_len,
+        DEFAULT_ACCESS_KEY,
+        DEFAULT_SECRET_KEY,
+        "",
+        "us-east-1",
+    );
+
+    let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
+    let mut request_builder = local_http_client().request(reqwest_method, url);
+    for (name, value) in signed.headers() {
+        request_builder = request_builder.header(name, value);
+    }
+    if let Some(body) = body {
+        request_builder = request_builder.body(body);
+    }
+
+    Ok(request_builder.send().await?)
+}
+
+async fn admin_create_user(base_url: &str, username: &str, secret_key: &str) -> Result<()> {
+    let url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", base_url, username);
+    let body = serde_json::json!({
+        "secretKey": secret_key,
+        "status": "enabled"
+    });
+    let response =
+        signed_admin_request(http::Method::PUT, &url, Some(body.to_string().into_bytes()), Some("application/json")).await?;
+
+    if response.status() != reqwest::StatusCode::OK {
+        let status = response.status();
+        let body = response.text().await.unwrap_or_default();
+        anyhow::bail!("create user failed: {status} {body}");
+    }
+
+    Ok(())
+}
+
+async fn admin_add_canned_policy(base_url: &str, policy_name: &str, policy: &serde_json::Value) -> Result<()> {
+    let url = format!("{}/rustfs/admin/v3/add-canned-policy?name={}", base_url, policy_name);
+    let response =
+        signed_admin_request(http::Method::PUT, &url, Some(policy.to_string().into_bytes()), Some("application/json")).await?;
+
+    if response.status() != reqwest::StatusCode::OK {
+        let status = response.status();
+        let body = response.text().await.unwrap_or_default();
+        anyhow::bail!("add canned policy failed: {status} {body}");
+    }
+
+    Ok(())
+}
+
+async fn admin_attach_policy_to_user(base_url: &str, policy_name: &str, username: &str) -> Result<()> {
+    let url = format!(
+        "{}/rustfs/admin/v3/set-user-or-group-policy?policyName={}&userOrGroup={}&isGroup=false",
+        base_url, policy_name, username
+    );
+    let response = signed_admin_request(http::Method::PUT, &url, Some(Vec::new()), None).await?;
+
+    if response.status() != reqwest::StatusCode::OK {
+        let status = response.status();
+        let body = response.text().await.unwrap_or_default();
+        anyhow::bail!("attach policy failed: {status} {body}");
+    }
+
+    Ok(())
+}
+
 /// Test WebDAV: MKCOL (create bucket), PUT, GET, DELETE, PROPFIND operations
 pub async fn test_webdav_core_operations() -> Result<()> {
     let env = ProtocolTestEnvironment::new().map_err(|e| anyhow::anyhow!("{}", e))?;
+    let admin_base_url = format!("http://{}", S3_TEST_ADDRESS);
 
     // Start server manually
     info!("Starting WebDAV server on {}", WEBDAV_ADDRESS);
-    let binary_path = rustfs_binary_path_with_features(Some("ftps,webdav"));
+    let binary_path = rustfs_binary_path_with_features(Some("webdav"));
     let mut server_process = Command::new(&binary_path)
+        .arg("--address")
+        .arg(S3_TEST_ADDRESS)
         .env("RUSTFS_WEBDAV_ENABLE", "true")
         .env("RUSTFS_WEBDAV_ADDRESS", WEBDAV_ADDRESS)
         .env("RUSTFS_WEBDAV_TLS_ENABLED", "false") // No TLS for testing
@@ -170,6 +272,372 @@ pub async fn test_webdav_core_operations() -> Result<()> {
         );
         info!("PASS: Verified file '{}' is deleted", filename);
 
+        // Test MOVE (rename) file
+        info!("Testing WebDAV: PUT file for MOVE test");
+        let move_filename = "move-source.txt";
+        let move_dest_filename = "move-dest.txt";
+        let move_content = "File to be moved!";
+        let resp = client
+            .put(format!("{}/{}/{}", base_url, bucket_name, move_filename))
+            .header("Authorization", &auth_header)
+            .body(move_content)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "PUT for MOVE test should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: PUT file '{}' for MOVE test successful", move_filename);
+
+        // Execute MOVE request
+        info!("Testing WebDAV: MOVE file '{}' to '{}'", move_filename, move_dest_filename);
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MOVE").unwrap(),
+                format!("{}/{}/{}", base_url, bucket_name, move_filename),
+            )
+            .header("Authorization", &auth_header)
+            .header("Destination", format!("/{}/{}", bucket_name, move_dest_filename))
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 204 || resp.status().as_u16() == 201,
+            "MOVE should succeed, got: {}",
+            resp.status()
+        );
+        info!(
+            "PASS: MOVE file '{}' to '{}' successful (HTTP {})",
+            move_filename,
+            move_dest_filename,
+            resp.status()
+        );
+
+        // Verify source file is gone
+        info!("Testing WebDAV: Verify source '{}' is deleted after MOVE", move_filename);
+        let resp = client
+            .get(format!("{}/{}/{}", base_url, bucket_name, move_filename))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().as_u16() == 404,
+            "GET moved source should return 404, got: {}",
+            resp.status()
+        );
+        info!("PASS: Source '{}' is deleted after MOVE", move_filename);
+
+        // Verify destination file exists and content matches
+        info!("Testing WebDAV: Verify destination '{}' has correct content", move_dest_filename);
+        let resp = client
+            .get(format!("{}/{}/{}", base_url, bucket_name, move_dest_filename))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success(),
+            "GET destination after MOVE should succeed, got: {}",
+            resp.status()
+        );
+        let moved_content = resp.text().await?;
+        assert_eq!(moved_content, move_content, "Moved file content should match original");
+        info!("PASS: Destination '{}' has correct content after MOVE", move_dest_filename);
+
+        // Test directory creation and rename
+        info!("Testing WebDAV: MKCOL directory");
+        let dir_name = "test-directory";
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MKCOL").unwrap(),
+                format!("{}/{}/{}", base_url, bucket_name, dir_name),
+            )
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "MKCOL directory should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: MKCOL directory successful");
+
+        // Upload file into directory
+        let dir_filename = "dir-file.txt";
+        let dir_file_content = "File inside test directory!";
+        let resp = client
+            .put(format!("{}/{}/{}/{}", base_url, bucket_name, dir_name, dir_filename))
+            .header("Authorization", &auth_header)
+            .body(dir_file_content)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "PUT file into directory should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: PUT file into directory successful");
+
+        // Test PROPFIND on directory
+        info!("Testing WebDAV: PROPFIND directory");
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"PROPFIND").unwrap(),
+                format!("{}/{}/{}", base_url, bucket_name, dir_name),
+            )
+            .header("Authorization", &auth_header)
+            .header("Depth", "1")
+            .send()
+            .await?;
+        assert!(resp.status().is_success(), "PROPFIND directory should succeed, got: {}", resp.status());
+        let propfind_body = resp.text().await?;
+        assert!(propfind_body.contains(dir_filename), "PROPFIND should list file in directory");
+        info!("PASS: PROPFIND directory successful, file listed correctly");
+
+        // Current WebDAV support exposes collection listing via PROPFIND; GET on a collection is not implemented.
+        info!("Testing WebDAV: GET directory listing fallback behavior");
+        let resp = client
+            .get(format!("{}/{}/{}", base_url, bucket_name, dir_name))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert_eq!(
+            resp.status().as_u16(),
+            405,
+            "GET on a WebDAV collection should currently return 405, got: {}",
+            resp.status()
+        );
+        info!("PASS: GET collection correctly returned 405; PROPFIND remains the listing path");
+
+        // Rename directory
+        info!("Testing WebDAV: MOVE directory");
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MOVE").unwrap(),
+                format!("{}/{}/{}", base_url, bucket_name, dir_name),
+            )
+            .header("Authorization", &auth_header)
+            .header("Destination", format!("/{}/renamed-dir", bucket_name))
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 204 || resp.status().as_u16() == 201,
+            "MOVE directory should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: MOVE directory successful");
+
+        // Verify source directory is gone
+        info!("Testing WebDAV: Verify source directory is deleted after MOVE");
+        let resp = client
+            .get(format!("{}/{}/{}", base_url, bucket_name, dir_name))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().as_u16() == 404,
+            "GET moved directory should return 404, got: {}",
+            resp.status()
+        );
+        info!("PASS: Source directory is deleted after MOVE");
+
+        // Verify renamed directory exists and file content matches
+        info!("Testing WebDAV: Verify file in renamed directory");
+        let resp = client
+            .get(format!("{}/{}/renamed-dir/{}", base_url, bucket_name, dir_filename))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success(),
+            "GET file in renamed directory should succeed, got: {}",
+            resp.status()
+        );
+        let renamed_dir_content = resp.text().await?;
+        assert_eq!(renamed_dir_content, dir_file_content, "File content in renamed directory should match");
+        info!("PASS: File in renamed directory has correct content");
+
+        // Test nested directory creation and rename
+        info!("Testing WebDAV: MKCOL nested directory");
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MKCOL").unwrap(),
+                format!("{}/{}/renamed-dir/nested-dir", base_url, bucket_name),
+            )
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "MKCOL nested directory should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: MKCOL nested directory successful");
+
+        // Upload file into nested directory
+        let nested_file_content = "File in nested directory!";
+        let resp = client
+            .put(format!("{}/{}/renamed-dir/nested-dir/nested-file.txt", base_url, bucket_name))
+            .header("Authorization", &auth_header)
+            .body(nested_file_content)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "PUT file into nested directory should succeed, got: {}",
+            resp.status()
+        );
+
+        // Rename nested directory
+        info!("Testing WebDAV: MOVE nested directory");
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MOVE").unwrap(),
+                format!("{}/{}/renamed-dir/nested-dir", base_url, bucket_name),
+            )
+            .header("Authorization", &auth_header)
+            .header("Destination", format!("/{}/renamed-dir/new-nested-dir", bucket_name))
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 204 || resp.status().as_u16() == 201,
+            "MOVE nested directory should succeed, got: {}",
+            resp.status()
+        );
+        info!("PASS: MOVE nested directory successful");
+
+        // Verify nested file after rename
+        let resp = client
+            .get(format!("{}/{}/renamed-dir/new-nested-dir/nested-file.txt", base_url, bucket_name))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success(),
+            "GET file in renamed nested directory should succeed, got: {}",
+            resp.status()
+        );
+        let nested_content = resp.text().await?;
+        assert_eq!(
+            nested_content, nested_file_content,
+            "File content in renamed nested directory should match"
+        );
+        info!("PASS: File in renamed nested directory has correct content");
+
+        // Test directory MOVE authz failure does not create partial destination writes
+        info!("Testing WebDAV: directory MOVE denied by missing DeleteObject must not create partial writes");
+        let restricted_bucket = "webdav-authz-bucket";
+        let restricted_dir = "restricted-src";
+        let restricted_dst = "restricted-dst";
+        let restricted_file = "locked.txt";
+        let restricted_content = "must remain only at source";
+        let restricted_user = "webdav-limited-user";
+        let restricted_secret = "webdav-limited-secret";
+        let restricted_policy_name = "webdav-move-no-delete";
+
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MKCOL").unwrap(),
+                format!("{}/{}", base_url, restricted_bucket),
+            )
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "MKCOL restricted bucket should succeed, got: {}",
+            resp.status()
+        );
+
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MKCOL").unwrap(),
+                format!("{}/{}/{}", base_url, restricted_bucket, restricted_dir),
+            )
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "MKCOL restricted directory should succeed, got: {}",
+            resp.status()
+        );
+
+        let resp = client
+            .put(format!("{}/{}/{}/{}", base_url, restricted_bucket, restricted_dir, restricted_file))
+            .header("Authorization", &auth_header)
+            .body(restricted_content)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success() || resp.status().as_u16() == 201,
+            "PUT restricted file should succeed, got: {}",
+            resp.status()
+        );
+
+        admin_create_user(&admin_base_url, restricted_user, restricted_secret).await?;
+        admin_add_canned_policy(
+            &admin_base_url,
+            restricted_policy_name,
+            &serde_json::json!({
+                "Version": "2012-10-17",
+                "Statement": [
+                    {
+                        "Effect": "Allow",
+                        "Action": ["s3:ListBucket"],
+                        "Resource": [format!("arn:aws:s3:::{}", restricted_bucket)]
+                    },
+                    {
+                        "Effect": "Allow",
+                        "Action": ["s3:GetObject", "s3:PutObject"],
+                        "Resource": [format!("arn:aws:s3:::{}/*", restricted_bucket)]
+                    }
+                ]
+            }),
+        )
+        .await?;
+        admin_attach_policy_to_user(&admin_base_url, restricted_policy_name, restricted_user).await?;
+
+        let restricted_auth = basic_auth_header_for(restricted_user, restricted_secret);
+        let resp = client
+            .request(
+                reqwest::Method::from_bytes(b"MOVE").unwrap(),
+                format!("{}/{}/{}", base_url, restricted_bucket, restricted_dir),
+            )
+            .header("Authorization", &restricted_auth)
+            .header("Destination", format!("/{}/{}", restricted_bucket, restricted_dst))
+            .send()
+            .await?;
+        assert!(
+            !resp.status().is_success(),
+            "MOVE without DeleteObject permission should be rejected, got: {}",
+            resp.status()
+        );
+
+        let resp = client
+            .get(format!("{}/{}/{}/{}", base_url, restricted_bucket, restricted_dst, restricted_file))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert_eq!(
+            resp.status().as_u16(),
+            404,
+            "Denied MOVE must not create destination object, got: {}",
+            resp.status()
+        );
+
+        let resp = client
+            .get(format!("{}/{}/{}/{}", base_url, restricted_bucket, restricted_dir, restricted_file))
+            .header("Authorization", &auth_header)
+            .send()
+            .await?;
+        assert!(
+            resp.status().is_success(),
+            "Source object should remain after denied MOVE, got: {}",
+            resp.status()
+        );
+        assert_eq!(resp.text().await?, restricted_content, "Denied MOVE must leave source content untouched");
+        info!("PASS: denied directory MOVE left source intact and created no destination objects");
+
         // Test DELETE bucket
         info!("Testing WebDAV: DELETE bucket '{}'", bucket_name);
         let resp = client
@@ -205,3 +673,9 @@ pub async fn test_webdav_core_operations() -> Result<()> {
 
     result
 }
+
+#[tokio::test]
+#[serial]
+async fn test_webdav_core_operations_direct() -> Result<()> {
+    test_webdav_core_operations().await
+}

crates/ecstore/Cargo.toml

@@ -44,6 +44,7 @@ rustfs-credentials = { workspace = true }
 rustfs-common.workspace = true
 rustfs-policy.workspace = true
 rustfs-protos.workspace = true
+rustfs-kms.workspace = true
 rustfs-s3-common = { workspace = true }
 async-trait.workspace = true
 bytes.workspace = true
@@ -104,6 +105,7 @@ rustix = { workspace = true }
 rustfs-madmin.workspace = true
 rustfs-workers.workspace = true
 reqwest = { workspace = true }
+aes-gcm.workspace = true
 aws-sdk-s3 = { workspace = true }
 urlencoding = { workspace = true }
 smallvec = { workspace = true }

crates/ecstore/src/admin_server_info.rs

@@ -183,11 +183,8 @@ pub async fn get_local_server_property() -> ServerProperties {
     };
 
     // let mut sensitive = HashSet::new();
-    // sensitive.insert(ENV_ACCESS_KEY.to_string());
-    // sensitive.insert(ENV_SECRET_KEY.to_string());
-    // sensitive.insert(ENV_ROOT_USER.to_string());
-    // sensitive.insert(ENV_ROOT_PASSWORD.to_string());
-
+    // sensitive.insert(rustfs_config::ENV_RUSTFS_ACCESS_KEY.to_string());
+    // sensitive.insert(rustfs_config::ENV_RUSTFS_SECRET_KEY.to_string());
     if let Some(store) = new_object_layer_fn() {
         let storage_info = store.local_storage_info().await;
         props.state = ITEM_ONLINE.to_string();

crates/ecstore/src/bucket/lifecycle/core.rs

@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use rustfs_config::{DEFAULT_ILM_PROCESS_TIME_SECS, ENV_ILM_PROCESS_TIME, ENV_ILM_PROCESS_TIME_DEPRECATED};
 use rustfs_filemeta::{ReplicationStatusType, VersionPurgeStatusType};
 use s3s::dto::{
     BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter,
@@ -19,7 +20,6 @@ use s3s::dto::{
 };
 use std::cmp::Ordering;
 use std::collections::HashMap;
-use std::env;
 use std::sync::Arc;
 use time::macros::offset;
 use time::{self, Duration, OffsetDateTime};
@@ -805,16 +805,29 @@ pub fn expected_expiry_time(mod_time: OffsetDateTime, days: i32) -> OffsetDateTi
         .to_offset(offset!(-0:00:00))
         .saturating_add(Duration::days(days as i64));
 
-    // Truncate to midnight UTC per S3 standard, unless overridden by env var.
-    // _RUSTFS_ILM_PROCESS_TIME controls the truncation granularity in seconds.
-    let truncation_secs = env::var("_RUSTFS_ILM_PROCESS_TIME")
-        .ok()
-        .and_then(|v| v.parse::<u32>().ok())
-        .unwrap_or(86400); // default: truncate to midnight (24h)
+    // Round up to the next processing boundary per S3-compatible Days semantics.
+    // Canonical key: RUSTFS_ILM_PROCESS_TIME; deprecated alias: _RUSTFS_ILM_PROCESS_TIME.
+    // TODO(GA): Remove ENV_ILM_PROCESS_TIME_DEPRECATED compatibility during GA release.
+    let process_interval_secs = rustfs_utils::get_env_i32_with_aliases(
+        ENV_ILM_PROCESS_TIME,
+        &[ENV_ILM_PROCESS_TIME_DEPRECATED],
+        DEFAULT_ILM_PROCESS_TIME_SECS,
+    );
+    let process_interval_secs = if process_interval_secs > 0 {
+        process_interval_secs as u32
+    } else {
+        DEFAULT_ILM_PROCESS_TIME_SECS as u32
+    };
 
-    let unix_secs = t.unix_timestamp();
-    let truncated_secs = (unix_secs / truncation_secs as i64) * truncation_secs as i64;
-    OffsetDateTime::from_unix_timestamp(truncated_secs).unwrap_or(t)
+    let boundary_nanos = i128::from(process_interval_secs) * 1_000_000_000;
+    let timestamp_nanos = t.unix_timestamp_nanos();
+    let remainder = timestamp_nanos.rem_euclid(boundary_nanos);
+    let rounded_nanos = if remainder == 0 {
+        timestamp_nanos
+    } else {
+        timestamp_nanos + (boundary_nanos - remainder)
+    };
+    OffsetDateTime::from_unix_timestamp_nanos(rounded_nanos).unwrap_or(t)
 }
 
 pub async fn abort_incomplete_multipart_upload_due(
@@ -945,6 +958,12 @@ mod tests {
     use std::sync::Arc;
     use time::macros::datetime;
 
+    fn with_default_ilm_process_time(test: impl FnOnce()) {
+        temp_env::with_var_unset(ENV_ILM_PROCESS_TIME, || {
+            temp_env::with_var_unset(ENV_ILM_PROCESS_TIME_DEPRECATED, test);
+        });
+    }
+
     #[tokio::test]
     #[serial]
     async fn validate_accepts_zero_expiration_days() {
@@ -1336,7 +1355,7 @@ mod tests {
     #[tokio::test]
     #[serial]
     async fn eval_inner_expires_latest_object_after_days_due() {
-        let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
+        let base_time = datetime!(2025-01-15 10:30:45 UTC);
         let lc = BucketLifecycleConfiguration {
             expiry_updated_at: None,
             rules: vec![LifecycleRule {
@@ -1362,11 +1381,11 @@ mod tests {
             is_latest: true,
             ..Default::default()
         };
-        let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
+        let event = lc.eval_inner(&opts, datetime!(2025-01-17 00:00:00 UTC), 0).await;
 
         assert_eq!(event.action, IlmAction::DeleteAction);
         assert_eq!(event.rule_id, "expire-days");
-        assert_eq!(event.due, Some(expected_expiry_time(base_time, 1)));
+        assert_eq!(event.due, Some(datetime!(2025-01-17 00:00:00 UTC)));
     }
 
     #[tokio::test]
@@ -2135,40 +2154,98 @@ mod tests {
             .expect("expected days-based expiration to pass on locked bucket");
     }
 
-    // --- TASK-003 tests: Midnight UTC truncation ---
+    // --- TASK-003 tests: Round up to next UTC processing boundary ---
 
     #[test]
-    fn expected_expiry_time_truncates_to_midnight_utc() {
-        // Object created at 2025-01-15T10:30:45Z, expire in 30 days
-        let mod_time = datetime!(2025-01-15 10:30:45 UTC);
-        let result = expected_expiry_time(mod_time, 30);
+    #[serial]
+    fn expected_expiry_time_rounds_up_to_next_midnight_utc() {
+        with_default_ilm_process_time(|| {
+            // Object created at 2025-01-15T10:30:45Z, expire in 30 days
+            let mod_time = datetime!(2025-01-15 10:30:45 UTC);
+            let result = expected_expiry_time(mod_time, 30);
 
-        // Should be truncated to midnight: 2025-02-14T00:00:00Z
-        assert_eq!(result.hour(), 0);
-        assert_eq!(result.minute(), 0);
-        assert_eq!(result.second(), 0);
-        assert_eq!(result, datetime!(2025-02-14 00:00:00 UTC));
+            // Should round up to the next midnight: 2025-02-15T00:00:00Z
+            assert_eq!(result.hour(), 0);
+            assert_eq!(result.minute(), 0);
+            assert_eq!(result.second(), 0);
+            assert_eq!(result, datetime!(2025-02-15 00:00:00 UTC));
+        });
     }
 
     #[test]
+    #[serial]
     fn expected_expiry_time_immediate_expiry_returns_epoch() {
-        let mod_time = datetime!(2025-06-01 12:00:00 UTC);
-        let result = expected_expiry_time(mod_time, 0);
-        assert_eq!(result, OffsetDateTime::UNIX_EPOCH);
+        with_default_ilm_process_time(|| {
+            let mod_time = datetime!(2025-06-01 12:00:00 UTC);
+            let result = expected_expiry_time(mod_time, 0);
+            assert_eq!(result, OffsetDateTime::UNIX_EPOCH);
+        });
     }
 
     #[test]
-    fn expected_expiry_time_truncates_already_midnight() {
-        let mod_time = datetime!(2025-03-01 00:00:00 UTC);
-        let result = expected_expiry_time(mod_time, 1);
-        assert_eq!(result, datetime!(2025-03-02 00:00:00 UTC));
+    #[serial]
+    fn expected_expiry_time_preserves_exact_midnight_boundary() {
+        with_default_ilm_process_time(|| {
+            let mod_time = datetime!(2025-03-01 00:00:00 UTC);
+            let result = expected_expiry_time(mod_time, 1);
+            assert_eq!(result, datetime!(2025-03-02 00:00:00 UTC));
+        });
     }
 
     #[test]
-    fn expected_expiry_time_truncates_end_of_day() {
-        let mod_time = datetime!(2025-06-15 23:59:59 UTC);
-        let result = expected_expiry_time(mod_time, 1);
-        assert_eq!(result, datetime!(2025-06-16 00:00:00 UTC));
+    #[serial]
+    fn expected_expiry_time_rounds_end_of_day_to_following_midnight() {
+        with_default_ilm_process_time(|| {
+            let mod_time = datetime!(2025-06-15 23:59:59 UTC);
+            let result = expected_expiry_time(mod_time, 1);
+            assert_eq!(result, datetime!(2025-06-17 00:00:00 UTC));
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn expected_expiry_time_uses_canonical_process_time_boundary() {
+        let mod_time = datetime!(2025-01-15 10:30:45 UTC);
+
+        temp_env::with_var(ENV_ILM_PROCESS_TIME, Some("3600"), || {
+            temp_env::with_var_unset(ENV_ILM_PROCESS_TIME_DEPRECATED, || {
+                let result = expected_expiry_time(mod_time, 1);
+                assert_eq!(result, datetime!(2025-01-16 11:00:00 UTC));
+            });
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn expected_expiry_time_uses_deprecated_process_time_alias() {
+        let mod_time = datetime!(2025-01-15 10:30:45 UTC);
+
+        temp_env::with_var_unset(ENV_ILM_PROCESS_TIME, || {
+            temp_env::with_var(ENV_ILM_PROCESS_TIME_DEPRECATED, Some("3600"), || {
+                let result = expected_expiry_time(mod_time, 1);
+                assert_eq!(result, datetime!(2025-01-16 11:00:00 UTC));
+            });
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn expected_expiry_time_uses_default_boundary_when_process_time_is_zero_or_invalid() {
+        let mod_time = datetime!(2025-01-15 10:30:45 UTC);
+
+        temp_env::with_var(ENV_ILM_PROCESS_TIME, Some("0"), || {
+            temp_env::with_var_unset(ENV_ILM_PROCESS_TIME_DEPRECATED, || {
+                let result = expected_expiry_time(mod_time, 30);
+                assert_eq!(result, datetime!(2025-02-15 00:00:00 UTC));
+            });
+        });
+
+        temp_env::with_var(ENV_ILM_PROCESS_TIME, Some("not-a-number"), || {
+            temp_env::with_var_unset(ENV_ILM_PROCESS_TIME_DEPRECATED, || {
+                let result = expected_expiry_time(mod_time, 30);
+                assert_eq!(result, datetime!(2025-02-15 00:00:00 UTC));
+            });
+        });
     }
 
     // --- TASK-007 tests: Legacy Prefix/Filter conflict ---

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -119,7 +119,14 @@ fn should_count_head_proxy_failure(is_not_found: bool, code: Option<&str>, raw_s
     if is_not_found || matches!(code, Some("MethodNotAllowed" | "405")) {
         return false;
     }
-    !matches!(raw_status, Some(404 | 405))
+    if matches!(raw_status, Some(404 | 405)) {
+        return false;
+    }
+    !is_version_id_mismatch(code, raw_status)
+}
+
+fn has_raw_status(err: &SdkError<HeadObjectError>, status: u16) -> bool {
+    err.raw_response().is_some_and(|r| r.status().as_u16() == status)
 }
 
 fn is_head_proxy_failure(err: &SdkError<HeadObjectError>) -> bool {
@@ -150,6 +157,42 @@ async fn head_object_with_proxy_stats(
     result
 }
 
+// AWS returns 400 for root callers and 403 for IAM users when a UUID version ID
+// is rejected. The 403 case is safe: a real auth failure also returns 403 on the
+// versionId-less fallback, propagating as a hard error instead of silently skipping.
+fn is_version_id_mismatch(code: Option<&str>, raw_status: Option<u16>) -> bool {
+    match code {
+        Some(c) if !c.is_empty() => c == "InvalidArgument",
+        _ => matches!(raw_status, Some(400) | Some(403)),
+    }
+}
+
+fn is_version_id_format_mismatch(err: &SdkError<HeadObjectError>) -> bool {
+    let code = err.as_service_error().and_then(|se| se.code());
+    let raw_status = err.raw_response().map(|r| r.status().as_u16());
+    is_version_id_mismatch(code, raw_status)
+}
+
+async fn head_object_fallback(
+    source_bucket: &str,
+    tgt_client: &TargetClient,
+    object: &str,
+) -> std::result::Result<Option<HeadObjectOutput>, SdkError<HeadObjectError>> {
+    match head_object_with_proxy_stats(source_bucket, tgt_client, &tgt_client.bucket, object, None).await {
+        Ok(oi) => Ok(Some(oi)),
+        Err(e) if e.as_service_error().is_some_and(|se| se.is_not_found()) || has_raw_status(&e, 404) => Ok(None),
+        Err(e) => Err(e),
+    }
+}
+
+// Version IDs differ by design on this path (RustFS UUID vs AWS alphanumeric), so
+// compare only ETags. Equal ETags mean identical content; version ID is irrelevant.
+fn content_matches(src: &ObjectInfo, tgt: &HeadObjectOutput) -> bool {
+    let src_etag = src.etag.as_deref().map(rustfs_utils::path::trim_etag);
+    let tgt_etag = tgt.e_tag.as_deref().map(rustfs_utils::path::trim_etag);
+    src_etag.is_some() && src_etag == tgt_etag
+}
+
 #[derive(Debug, Clone, Default)]
 pub struct ResyncOpts {
     pub bucket: String,
@@ -2536,16 +2579,28 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
                 }
             }
             Err(e) => {
-                if let Some(se) = e.as_service_error() {
-                    if !se.is_not_found() {
-                        rinfo.error = Some(e.to_string());
-                        warn!("replication head_object failed bucket:{} arn:{} error:{}", bucket, tgt_client.arn, e);
-                        return rinfo;
+                if e.as_service_error().is_some_and(|se| se.is_not_found()) || has_raw_status(&e, 404) {
+                    // Object not on target yet → fall through to PUT.
+                } else if is_version_id_format_mismatch(&e) {
+                    // Version-ID format mismatch: retry without versionId and compare ETags.
+                    match head_object_fallback(&bucket, &tgt_client, &object).await {
+                        Ok(Some(oi)) if content_matches(&object_info, &oi) => {
+                            rinfo.replication_status = ReplicationStatusType::Completed;
+                            rinfo.replication_resynced = true;
+                            rinfo.replication_action = ReplicationAction::None;
+                            rinfo.size = size;
+                            return rinfo;
+                        }
+                        Ok(_) => {}
+                        Err(e2) => {
+                            rinfo.error = Some(e2.to_string());
+                            warn!(
+                                "replication head_object fallback failed bucket:{} arn:{} error:{}",
+                                bucket, tgt_client.arn, e2
+                            );
+                            return rinfo;
+                        }
                     }
-                } else if e.raw_response().is_some_and(|resp| resp.status().as_u16() == 404) {
-                    // Some HEAD Object 404 responses are surfaced by the AWS SDK as `response error`
-                    // instead of `service error (NotFound)`. Treat raw HTTP 404 as object-not-found
-                    // so replication can proceed with PUT.
                 } else {
                     rinfo.error = Some(e.to_string());
                     warn!("replication head_object failed bucket:{} arn:{} error:{}", bucket, tgt_client.arn, e);
@@ -2807,25 +2862,39 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
                 }
             }
             Err(e) => {
-                if let Some(se) = e.as_service_error() {
-                    if se.is_not_found() {
-                        replication_action = ReplicationAction::All;
-                    } else {
-                        rinfo.error = Some(e.to_string());
-                        warn!("failed to head object for bucket:{} arn:{} error:{}", bucket, tgt_client.arn, e);
-
-                        send_event(EventArgs {
-                            event_name: EventName::ObjectReplicationNotTracked.to_string(),
-                            bucket_name: bucket.clone(),
-                            object: object_info,
-                            host: GLOBAL_LocalNodeName.to_string(),
-                            user_agent: "Internal: [Replication]".to_string(),
-                            ..Default::default()
-                        });
-
-                        rinfo.duration = (OffsetDateTime::now_utc() - start_time).unsigned_abs();
-                        return rinfo;
+                if is_version_id_format_mismatch(&e) {
+                    // Version-ID format mismatch: retry without versionId and compare ETags.
+                    match head_object_fallback(&bucket, &tgt_client, &object).await {
+                        Ok(Some(oi)) => {
+                            replication_action = if content_matches(&object_info, &oi) {
+                                ReplicationAction::None
+                            } else {
+                                ReplicationAction::All
+                            };
+                        }
+                        Ok(None) => {
+                            replication_action = ReplicationAction::All;
+                        }
+                        Err(e2) => {
+                            rinfo.error = Some(e2.to_string());
+                            warn!(
+                                "replication head_object fallback failed bucket:{} arn:{} error:{}",
+                                bucket, tgt_client.arn, e2
+                            );
+                            send_event(EventArgs {
+                                event_name: EventName::ObjectReplicationNotTracked.to_string(),
+                                bucket_name: bucket.clone(),
+                                object: object_info,
+                                host: GLOBAL_LocalNodeName.to_string(),
+                                user_agent: "Internal: [Replication]".to_string(),
+                                ..Default::default()
+                            });
+                            rinfo.duration = (OffsetDateTime::now_utc() - start_time).unsigned_abs();
+                            return rinfo;
+                        }
                     }
+                } else if e.as_service_error().is_some_and(|se| se.is_not_found()) {
+                    replication_action = ReplicationAction::All;
                 } else {
                     rinfo.error = Some(e.to_string());
                     warn!("failed to head object for bucket:{} arn:{} error:{}", bucket, tgt_client.arn, e);
@@ -3837,6 +3906,117 @@ mod tests {
         );
     }
 
+    #[test]
+    fn test_should_count_head_proxy_failure_ignores_version_id_format_rejections() {
+        assert!(
+            !should_count_head_proxy_failure(false, Some("InvalidArgument"), Some(400)),
+            "InvalidArgument/400 is a version-ID format rejection and must not be counted as a proxy failure"
+        );
+        assert!(
+            !should_count_head_proxy_failure(false, None, Some(400)),
+            "raw HTTP 400 without error code must not be counted as a proxy failure"
+        );
+        assert!(
+            !should_count_head_proxy_failure(false, None, Some(403)),
+            "raw HTTP 403 without error code must not be counted as a proxy failure (IAM user + invalid versionId)"
+        );
+    }
+
+    #[test]
+    fn test_is_version_id_mismatch_detects_invalid_argument() {
+        assert!(
+            is_version_id_mismatch(Some("InvalidArgument"), Some(400)),
+            "AWS S3 returns InvalidArgument/400 when a UUID versionId is passed to HeadObject"
+        );
+        assert!(
+            !is_version_id_mismatch(Some("AccessDenied"), Some(403)),
+            "AccessDenied must not trigger the version-ID fallback path"
+        );
+        assert!(
+            !is_version_id_mismatch(Some("NoSuchKey"), Some(404)),
+            "NoSuchKey is an object-not-found response, not a version-ID mismatch"
+        );
+    }
+
+    #[test]
+    fn test_is_version_id_mismatch_raw_status_without_service_code() {
+        assert!(
+            is_version_id_mismatch(None, Some(400)),
+            "no error code + HTTP 400 is treated as version-ID mismatch (HEAD response)"
+        );
+        assert!(
+            is_version_id_mismatch(Some(""), Some(400)),
+            "empty error code + HTTP 400 is treated as version-ID mismatch"
+        );
+        assert!(
+            is_version_id_mismatch(None, Some(403)),
+            "no error code + HTTP 403 is treated as version-ID mismatch (IAM user + invalid versionId)"
+        );
+        assert!(
+            is_version_id_mismatch(Some(""), Some(403)),
+            "empty error code + HTTP 403 is treated as version-ID mismatch"
+        );
+        assert!(
+            !is_version_id_mismatch(None, Some(500)),
+            "raw 5xx must not trigger the version-ID fallback path"
+        );
+        assert!(
+            !is_version_id_mismatch(None, Some(404)),
+            "raw 404 must not trigger the version-ID fallback path"
+        );
+    }
+
+    #[test]
+    fn test_is_version_id_mismatch_400_with_other_service_code() {
+        assert!(
+            !is_version_id_mismatch(Some("MalformedXML"), Some(400)),
+            "MalformedXML/400 is a real request error and must not trigger version-ID fallback"
+        );
+        assert!(
+            !is_version_id_mismatch(Some("EntityTooLarge"), Some(400)),
+            "EntityTooLarge/400 is a real request error and must not trigger version-ID fallback"
+        );
+    }
+
+    #[test]
+    fn test_content_matches_compares_etag_only() {
+        let src = ObjectInfo {
+            etag: Some("\"abc123\"".to_string()),
+            ..Default::default()
+        };
+
+        let tgt_match = HeadObjectOutput::builder().e_tag("\"abc123\"").build();
+        assert!(content_matches(&src, &tgt_match), "identical ETags must match");
+
+        let tgt_unquoted_match = HeadObjectOutput::builder().e_tag("abc123").build();
+        assert!(
+            content_matches(&src, &tgt_unquoted_match),
+            "quoted and unquoted ETags with identical values must match"
+        );
+
+        // version_id on the target is intentionally ignored
+        let tgt_different_version = HeadObjectOutput::builder()
+            .e_tag("\"abc123\"")
+            .version_id("aws-alphanumeric-id")
+            .build();
+        assert!(
+            content_matches(&src, &tgt_different_version),
+            "matching ETags with different version IDs must still match"
+        );
+
+        let tgt_different_content = HeadObjectOutput::builder().e_tag("\"def456\"").build();
+        assert!(!content_matches(&src, &tgt_different_content), "different ETags must not match");
+
+        let src_no_etag = ObjectInfo {
+            etag: None,
+            ..Default::default()
+        };
+        assert!(!content_matches(&src_no_etag, &tgt_match), "missing source ETag must not match");
+
+        let tgt_no_etag = HeadObjectOutput::builder().build();
+        assert!(!content_matches(&src, &tgt_no_etag), "missing target ETag must not match");
+    }
+
     #[test]
     fn test_should_count_head_proxy_failure_counts_unexpected_errors() {
         assert!(

crates/ecstore/src/cache_value/metacache_set.rs

@@ -18,7 +18,12 @@ use crate::disk::{self, DiskAPI, DiskStore, WalkDirOptions};
 use futures::future::join_all;
 use metrics::counter;
 use rustfs_filemeta::{MetaCacheEntries, MetaCacheEntry, MetacacheReader, is_io_eof};
-use std::{future::Future, pin::Pin, time::Duration};
+use std::{
+    future::Future,
+    pin::Pin,
+    sync::{Arc, Mutex},
+    time::Duration,
+};
 use tokio::io::AsyncRead;
 use tokio::spawn;
 use tokio::time::timeout;
@@ -45,6 +50,15 @@ async fn peek_with_timeout<R: AsyncRead + Unpin>(reader: &mut MetacacheReader<R>
     }
 }
 
+#[cfg(test)]
+#[derive(Clone)]
+pub(crate) enum TestReaderBehavior {
+    Eof,
+    Stall,
+    ProducerError(DiskError),
+    PartialThenTimeout(Vec<MetaCacheEntry>),
+}
+
 #[derive(Default)]
 pub struct ListPathRawOptions {
     pub disks: Vec<Option<DiskStore>>,
@@ -60,6 +74,10 @@ pub struct ListPathRawOptions {
     pub agreed: Option<AgreedFn>,
     pub partial: Option<PartialFn>,
     pub finished: Option<FinishedFn>,
+    #[cfg(test)]
+    pub(crate) test_reader_behaviors: Vec<TestReaderBehavior>,
+    #[cfg(test)]
+    pub(crate) peek_timeout: Option<Duration>,
     // pub agreed: Option<Arc<dyn Fn(MetaCacheEntry) + Send + Sync>>,
     // pub partial: Option<Arc<dyn Fn(MetaCacheEntries, &[Option<Error>]) + Send + Sync>>,
     // pub finished: Option<Arc<dyn Fn(&[Option<Error>]) + Send + Sync>>,
@@ -78,6 +96,10 @@ impl Clone for ListPathRawOptions {
             min_disks: self.min_disks,
             report_not_found: self.report_not_found,
             per_disk_limit: self.per_disk_limit,
+            #[cfg(test)]
+            test_reader_behaviors: self.test_reader_behaviors.clone(),
+            #[cfg(test)]
+            peek_timeout: self.peek_timeout,
             ..Default::default()
         }
     }
@@ -85,23 +107,52 @@ impl Clone for ListPathRawOptions {
 
 pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> disk::error::Result<()> {
     if opts.disks.is_empty() {
-        return Err(DiskError::other("list_path_raw: 0 drives provided"));
+        return Err(DiskError::ErasureReadQuorum);
     }
 
     let mut jobs: Vec<tokio::task::JoinHandle<std::result::Result<(), DiskError>>> = Vec::new();
     let mut readers = Vec::with_capacity(opts.disks.len());
     let fds = opts.fallback_disks.iter().flatten().cloned().collect::<Vec<_>>();
+    let max_disk_failures = opts.disks.len().saturating_sub(opts.min_disks);
+    let producer_errs = Arc::new(Mutex::new(vec![None; opts.disks.len()]));
 
     let cancel_rx = CancellationToken::new();
 
-    for disk in opts.disks.iter() {
+    for (disk_idx, disk) in opts.disks.iter().enumerate() {
         let opdisk = disk.clone();
         let opts_clone = opts.clone();
         let mut fds_clone = fds.clone();
         let cancel_rx_clone = cancel_rx.clone();
-        let (rd, mut wr) = tokio::io::duplex(64);
+        let producer_errs_clone = producer_errs.clone();
+        let (rd, wr) = tokio::io::duplex(64);
         readers.push(MetacacheReader::new(rd));
         jobs.push(spawn(async move {
+            #[cfg(test)]
+            if let Some(behavior) = opts_clone.test_reader_behaviors.get(disk_idx).cloned() {
+                match behavior {
+                    TestReaderBehavior::Eof => return Ok(()),
+                    TestReaderBehavior::Stall => {
+                        let _held_writer = wr;
+                        cancel_rx_clone.cancelled().await;
+                        return Ok(());
+                    }
+                    TestReaderBehavior::ProducerError(err) => {
+                        producer_errs_clone.lock().expect("producer error mutex poisoned")[disk_idx] = Some(err.clone());
+                        return Err(err);
+                    }
+                    TestReaderBehavior::PartialThenTimeout(entries) => {
+                        let mut wr = wr;
+                        let mut out = rustfs_filemeta::MetacacheWriter::new(&mut wr);
+                        let err = DiskError::Timeout;
+                        producer_errs_clone.lock().expect("producer error mutex poisoned")[disk_idx] = Some(err.clone());
+                        let _ = out.write(&entries).await;
+                        drop(out);
+                        return Err(err);
+                    }
+                }
+            }
+
+            let mut wr = wr;
             let wakl_opts = WalkDirOptions {
                 bucket: opts_clone.bucket.clone(),
                 base_dir: opts_clone.path.clone(),
@@ -114,15 +165,18 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
             };
 
             let mut need_fallback = false;
+            let mut last_err = None;
             if let Some(disk) = opdisk {
                 match disk.walk_dir(wakl_opts, &mut wr).await {
                     Ok(_res) => {}
                     Err(err) => {
                         info!("walk dir err {:?}", &err);
+                        last_err = Some(err);
                         need_fallback = true;
                     }
                 }
             } else {
+                last_err = Some(DiskError::DiskNotFound);
                 need_fallback = true;
             }
 
@@ -132,18 +186,20 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
             }
 
             while need_fallback {
-                let disk_op = {
-                    if fds_clone.is_empty() {
-                        None
-                    } else {
-                        let disk = fds_clone.remove(0);
-                        if disk.is_online().await { Some(disk.clone()) } else { None }
+                let mut disk_op = None;
+                while !fds_clone.is_empty() {
+                    let disk = fds_clone.remove(0);
+                    if disk.is_online().await {
+                        disk_op = Some(disk);
+                        break;
                     }
-                };
+                }
 
                 let Some(disk) = disk_op else {
                     warn!("list_path_raw: fallback disk is none");
-                    break;
+                    let err = last_err.unwrap_or(DiskError::DiskNotFound);
+                    producer_errs_clone.lock().expect("producer error mutex poisoned")[disk_idx] = Some(err.clone());
+                    return Err(err);
                 };
 
                 match disk
@@ -165,20 +221,28 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                 {
                     Ok(_r) => {
                         need_fallback = false;
+                        last_err = None;
                     }
                     Err(err) => {
                         error!("walk dir2 err {:?}", &err);
-                        break;
+                        last_err = Some(err);
                     }
                 }
             }
 
+            if need_fallback {
+                return Err(last_err.unwrap_or(DiskError::DiskNotFound));
+            }
+
             // warn!("list_path_raw: while need_fallback done");
             Ok(())
         }));
     }
 
     let revjob = spawn(async move {
+        #[cfg(test)]
+        let peek_timeout = opts.peek_timeout.unwrap_or_else(get_drive_walkdir_stall_timeout);
+        #[cfg(not(test))]
         let peek_timeout = get_drive_walkdir_stall_timeout();
         let mut errs: Vec<Option<DiskError>> = Vec::with_capacity(readers.len());
         for _ in 0..readers.len() {
@@ -217,6 +281,11 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                             // info!("read entry disk: {}, name: {}", i, entry.name);
                             entry
                         } else {
+                            if let Some(err) = producer_errs.lock().expect("producer error mutex poisoned")[i].clone() {
+                                has_err += 1;
+                                errs[i] = Some(err);
+                                continue;
+                            }
                             // eof
                             at_eof += 1;
                             // warn!("list_path_raw: peek eof, disk: {}", i);
@@ -224,6 +293,12 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                         }
                     }
                     PeekOutcome::Error(err) => {
+                        if let Some(err) = producer_errs.lock().expect("producer error mutex poisoned")[i].clone() {
+                            has_err += 1;
+                            errs[i] = Some(err);
+                            continue;
+                        }
+
                         if err == rustfs_filemeta::Error::Unexpected {
                             at_eof += 1;
                             // warn!("list_path_raw: peek err eof, disk: {}", i);
@@ -333,6 +408,15 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                 if let Some(finished_fn) = opts.finished.as_ref() {
                     finished_fn(&errs).await;
                 }
+                if errs.iter().flatten().any(|err| *err == DiskError::Timeout) {
+                    return Err(DiskError::Timeout);
+                }
+                let mut err_iter = errs.iter().flatten();
+                if let Some(err) = err_iter.next()
+                    && err_iter.next().is_none()
+                {
+                    return Err(err.clone());
+                }
                 let mut combined_err = Vec::new();
                 errs.iter().zip(opts.disks.iter()).for_each(|(err, disk)| match (err, disk) {
                     (Some(err), Some(disk)) => {
@@ -358,6 +442,9 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                 {
                     finished_fn(&errs).await;
                 }
+                if errs.iter().flatten().any(|err| *err == DiskError::Timeout) {
+                    return Err(DiskError::Timeout);
+                }
 
                 // error!("list_path_raw: at_eof + has_err == readers.len() break {:?}", &errs);
                 break;
@@ -400,12 +487,25 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
     }
 
     let results = join_all(jobs).await;
+    let mut job_errs = Vec::new();
     for result in results {
-        if let Err(err) = result {
-            error!("list_path_raw err {:?}", err);
+        match result {
+            Ok(Ok(())) => {}
+            Ok(Err(err)) => {
+                error!("list_path_raw producer err {:?}", err);
+                job_errs.push(err);
+            }
+            Err(err) => {
+                error!("list_path_raw join err {:?}", err);
+                job_errs.push(err.into());
+            }
         }
     }
 
+    if job_errs.len() > max_disk_failures {
+        return Err(job_errs.remove(0));
+    }
+
     // warn!("list_path_raw: done");
     Ok(())
 }
@@ -415,6 +515,65 @@ mod tests {
     use super::*;
     use rustfs_filemeta::MetacacheWriter;
 
+    #[tokio::test]
+    async fn list_path_raw_empty_disks_returns_read_quorum() {
+        let err = list_path_raw(CancellationToken::new(), ListPathRawOptions::default())
+            .await
+            .expect_err("empty drive list should fail");
+
+        assert_eq!(err, DiskError::ErasureReadQuorum);
+    }
+
+    #[tokio::test]
+    async fn list_path_raw_returns_timeout_when_reader_stalls_before_completion() {
+        let err = list_path_raw(
+            CancellationToken::new(),
+            ListPathRawOptions {
+                disks: vec![None, None],
+                min_disks: 1,
+                test_reader_behaviors: vec![TestReaderBehavior::Stall, TestReaderBehavior::Eof],
+                peek_timeout: Some(Duration::from_millis(20)),
+                ..Default::default()
+            },
+        )
+        .await
+        .expect_err("stalled reader should make listing fail explicitly");
+
+        assert_eq!(err, DiskError::Timeout);
+    }
+
+    #[tokio::test]
+    async fn list_path_raw_returns_timeout_when_producer_fails_after_partial_entry() {
+        let seen = Arc::new(Mutex::new(Vec::new()));
+        let seen_clone = seen.clone();
+
+        let err = list_path_raw(
+            CancellationToken::new(),
+            ListPathRawOptions {
+                disks: vec![None],
+                min_disks: 1,
+                test_reader_behaviors: vec![TestReaderBehavior::PartialThenTimeout(vec![MetaCacheEntry {
+                    name: "bucket/object".to_string(),
+                    metadata: vec![1, 2, 3],
+                    cached: None,
+                    reusable: false,
+                }])],
+                agreed: Some(Box::new(move |entry: MetaCacheEntry| {
+                    let seen = seen_clone.clone();
+                    Box::pin(async move {
+                        seen.lock().expect("seen mutex poisoned").push(entry.name);
+                    })
+                })),
+                ..Default::default()
+            },
+        )
+        .await
+        .expect_err("producer timeout after partial output must fail the listing");
+
+        assert_eq!(err, DiskError::Timeout);
+        assert_eq!(seen.lock().expect("seen mutex poisoned").as_slice(), &["bucket/object".to_string()]);
+    }
+
     #[tokio::test]
     async fn peek_with_timeout_times_out_on_silent_reader() {
         let (_writer, reader) = tokio::io::duplex(64);
@@ -447,4 +606,21 @@ mod tests {
             other => panic!("expected ready entry, got {other:?}"),
         }
     }
+
+    #[tokio::test]
+    async fn list_path_raw_propagates_producer_access_denied() {
+        let err = list_path_raw(
+            CancellationToken::new(),
+            ListPathRawOptions {
+                disks: vec![None],
+                min_disks: 1,
+                test_reader_behaviors: vec![TestReaderBehavior::ProducerError(DiskError::FileAccessDenied)],
+                ..Default::default()
+            },
+        )
+        .await
+        .expect_err("producer access failure must not be treated as an empty listing");
+
+        assert_eq!(err, DiskError::FileAccessDenied);
+    }
 }

crates/ecstore/src/client/api_remove.rs

@@ -340,7 +340,7 @@ impl TransitionClient {
                         content_body: ReaderImpl::Body(Bytes::from(remove_bytes.clone())),
                         content_length: remove_bytes.len() as i64,
                         content_md5_base64: base64_encode(&HashAlgorithm::Md5.hash_encode(&remove_bytes).as_ref()),
-                        content_sha256_hex: base64_encode(&HashAlgorithm::SHA256.hash_encode(&remove_bytes).as_ref()),
+                        content_sha256_hex: rustfs_utils::hex(HashAlgorithm::SHA256.hash_encode(&remove_bytes)),
                         custom_header: headers,
                         object_name: "".to_string(),
                         stream_sha256: false,
@@ -742,3 +742,115 @@ pub async fn process_remove_multi_objects_response(
 fn has_invalid_xml_char(str: &str) -> bool {
     false
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::client::{
+        credentials::{Credentials, SignatureType, Static, Value},
+        transition_api::{BucketLookupType, Options},
+    };
+    use tokio::{
+        io::{AsyncReadExt, AsyncWriteExt},
+        net::TcpListener,
+    };
+
+    async fn capture_delete_objects_sha256_header() -> (String, tokio::task::JoinHandle<String>) {
+        let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
+        let endpoint = listener.local_addr().unwrap().to_string();
+        let task = tokio::spawn(async move {
+            let (mut stream, _) = listener.accept().await.unwrap();
+            let mut request = Vec::new();
+            let mut buffer = [0; 1024];
+            loop {
+                let read = stream.read(&mut buffer).await.unwrap();
+                assert_ne!(read, 0, "connection closed before request headers were received");
+                request.extend_from_slice(&buffer[..read]);
+                if request.windows(4).any(|window| window == b"\r\n\r\n") {
+                    break;
+                }
+            }
+
+            let request = String::from_utf8_lossy(&request);
+            let sha256_header = request
+                .lines()
+                .find_map(|line| {
+                    let (name, value) = line.split_once(':')?;
+                    name.eq_ignore_ascii_case("x-amz-content-sha256")
+                        .then(|| value.trim().to_string())
+                })
+                .expect("delete objects request should include X-Amz-Content-Sha256");
+
+            let response_body = r#"<?xml version="1.0" encoding="UTF-8"?><DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Deleted><Key>object.txt</Key></Deleted></DeleteResult>"#;
+            let response = format!(
+                "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}",
+                response_body.len(),
+                response_body
+            );
+            stream.write_all(response.as_bytes()).await.unwrap();
+            sha256_header
+        });
+
+        (endpoint, task)
+    }
+
+    #[tokio::test]
+    async fn multi_object_delete_request_uses_lowercase_hex_sha256_header() {
+        let objects = vec![ObjectInfo {
+            bucket: "bucket".to_string(),
+            name: "object.txt".to_string(),
+            ..Default::default()
+        }];
+        let body = generate_remove_multi_objects_request(&objects);
+        let expected = rustfs_utils::hex(HashAlgorithm::SHA256.hash_encode(&body));
+        let (endpoint, header_task) = capture_delete_objects_sha256_header().await;
+        let client = TransitionClient::new(
+            &endpoint,
+            Options {
+                creds: Credentials::new(Static(Value {
+                    access_key_id: "access-key".to_string(),
+                    secret_access_key: "secret-key".to_string(),
+                    signer_type: SignatureType::SignatureV4,
+                    ..Default::default()
+                })),
+                region: "us-east-1".to_string(),
+                bucket_lookup: BucketLookupType::BucketLookupPath,
+                max_retries: 1,
+                ..Default::default()
+            },
+            "",
+        )
+        .await
+        .unwrap();
+        let (objects_tx, objects_rx) = mpsc::channel(1);
+        let (result_tx, mut result_rx) = mpsc::channel(1);
+
+        objects_tx.send(objects[0].clone()).await.unwrap();
+        drop(objects_tx);
+
+        client
+            .remove_objects_inner(
+                "bucket",
+                objects_rx,
+                &result_tx,
+                RemoveObjectsOptions {
+                    governance_bypass: false,
+                },
+            )
+            .await
+            .unwrap();
+        drop(result_tx);
+
+        let header = header_task.await.unwrap();
+
+        assert_eq!(header, expected);
+        assert_eq!(header.len(), 64);
+        assert!(
+            header
+                .bytes()
+                .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte))
+        );
+        assert_ne!(header, base64_encode(&HashAlgorithm::SHA256.hash_encode(&body).as_ref()));
+        assert!(result_rx.recv().await.is_some());
+    }
+}

crates/ecstore/src/config/audit.rs

@@ -13,17 +13,27 @@
 //  limitations under the License.
 
 use crate::config::{KV, KVS};
+use rustfs_config::audit::AUDIT_REDIS_DEFAULT_CHANNEL;
 use rustfs_config::{
-    COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY, EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR,
-    KAFKA_QUEUE_LIMIT, KAFKA_TLS_CA, KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY, KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER,
-    MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA,
-    MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME,
-    MQTT_WS_PATH_ALLOWLIST, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT,
-    NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, PULSAR_AUTH_TOKEN,
-    PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
-    PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA,
-    WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR,
-    WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL, WEBHOOK_SKIP_TLS_VERIFY,
+    AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_QUEUE_LIMIT, AMQP_ROUTING_KEY,
+    AMQP_TLS_CA, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY,
+    EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_TLS_CA,
+    KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY, KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL,
+    MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT,
+    MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST,
+    MYSQL_DSN_STRING, MYSQL_FORMAT, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA,
+    MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR,
+    NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN,
+    NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE,
+    POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER,
+    PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
+    PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT,
+    REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD,
+    REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT,
+    REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL,
+    REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY,
+    WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL,
+    WEBHOOK_SKIP_TLS_VERIFY,
 };
 use std::sync::LazyLock;
 
@@ -198,6 +208,81 @@ pub static DEFAULT_AUDIT_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
     ])
 });
 
+pub static DEFAULT_AUDIT_AMQP_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: ENABLE_KEY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_URL.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_EXCHANGE.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_ROUTING_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_MANDATORY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_PERSISTENT.to_owned(),
+            value: EnableState::On.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_USERNAME.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_PASSWORD.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: AMQP_TLS_CA.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: AMQP_TLS_CLIENT_CERT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: AMQP_TLS_CLIENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: AMQP_QUEUE_DIR.to_owned(),
+            value: EVENT_DEFAULT_DIR.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_QUEUE_LIMIT.to_owned(),
+            value: DEFAULT_LIMIT.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: COMMENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+    ])
+});
+
 pub static DEFAULT_AUDIT_NATS_KVS: LazyLock<KVS> = LazyLock::new(|| {
     KVS(vec![
         KV {
@@ -339,6 +424,176 @@ pub static DEFAULT_AUDIT_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(|| {
     ])
 });
 
+pub static DEFAULT_AUDIT_REDIS_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: ENABLE_KEY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_URL.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_CHANNEL.to_owned(),
+            value: AUDIT_REDIS_DEFAULT_CHANNEL.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_USERNAME.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_PASSWORD.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_KEEP_ALIVE_INTERVAL.to_owned(),
+            value: "15".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_QUEUE_DIR.to_owned(),
+            value: EVENT_DEFAULT_DIR.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_QUEUE_LIMIT.to_owned(),
+            value: DEFAULT_LIMIT.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_MAX_RETRY_ATTEMPTS.to_owned(),
+            value: "3".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_RECONNECT_RETRY_ATTEMPTS.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_MIN_RETRY_DELAY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_MAX_RETRY_DELAY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_CONNECTION_TIMEOUT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_RESPONSE_TIMEOUT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_PIPELINE_BUFFER_SIZE.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_TLS_POLICY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_TLS_CA.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_TLS_CLIENT_CERT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_TLS_CLIENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_TLS_ALLOW_INSECURE.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: COMMENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+    ])
+});
+
+pub static DEFAULT_AUDIT_POSTGRES_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: ENABLE_KEY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_DSN_STRING.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: POSTGRES_TABLE.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_FORMAT.to_owned(),
+            value: "namespace".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_TLS_REQUIRED.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_TLS_CA.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: POSTGRES_TLS_CLIENT_CERT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: POSTGRES_TLS_CLIENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: POSTGRES_QUEUE_DIR.to_owned(),
+            value: EVENT_DEFAULT_DIR.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_QUEUE_LIMIT.to_owned(),
+            value: DEFAULT_LIMIT.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: COMMENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+    ])
+});
+
 pub static DEFAULT_AUDIT_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(|| {
     KVS(vec![
         KV {
@@ -398,3 +653,63 @@ pub static DEFAULT_AUDIT_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(|| {
         },
     ])
 });
+
+pub static DEFAULT_AUDIT_MYSQL_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: ENABLE_KEY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_DSN_STRING.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: MYSQL_TABLE.to_owned(),
+            value: "rustfs_audit_logs".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_FORMAT.to_owned(),
+            value: "access".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_TLS_CA.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: MYSQL_TLS_CLIENT_CERT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: MYSQL_TLS_CLIENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: MYSQL_QUEUE_DIR.to_owned(),
+            value: EVENT_DEFAULT_DIR.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_QUEUE_LIMIT.to_owned(),
+            value: DEFAULT_LIMIT.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_MAX_OPEN_CONNECTIONS.to_owned(),
+            value: "2".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: COMMENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+    ])
+});

crates/ecstore/src/config/com.rs

@@ -19,12 +19,15 @@ use crate::global::is_first_cluster_node_local;
 use crate::store_api::{ObjectInfo, ObjectOptions, PutObjReader, StorageAPI};
 use http::HeaderMap;
 use rustfs_config::audit::{
-    AUDIT_KAFKA_KEYS, AUDIT_KAFKA_SUB_SYS, AUDIT_MQTT_KEYS, AUDIT_MQTT_SUB_SYS, AUDIT_NATS_KEYS, AUDIT_NATS_SUB_SYS,
-    AUDIT_PULSAR_KEYS, AUDIT_PULSAR_SUB_SYS, AUDIT_WEBHOOK_KEYS, AUDIT_WEBHOOK_SUB_SYS,
+    AUDIT_AMQP_KEYS, AUDIT_AMQP_SUB_SYS, AUDIT_KAFKA_KEYS, AUDIT_KAFKA_SUB_SYS, AUDIT_MQTT_KEYS, AUDIT_MQTT_SUB_SYS,
+    AUDIT_MYSQL_KEYS, AUDIT_MYSQL_SUB_SYS, AUDIT_NATS_KEYS, AUDIT_NATS_SUB_SYS, AUDIT_POSTGRES_KEYS, AUDIT_POSTGRES_SUB_SYS,
+    AUDIT_PULSAR_KEYS, AUDIT_PULSAR_SUB_SYS, AUDIT_REDIS_KEYS, AUDIT_REDIS_SUB_SYS, AUDIT_WEBHOOK_KEYS, AUDIT_WEBHOOK_SUB_SYS,
 };
 use rustfs_config::notify::{
-    NOTIFY_KAFKA_KEYS, NOTIFY_KAFKA_SUB_SYS, NOTIFY_MQTT_KEYS, NOTIFY_MQTT_SUB_SYS, NOTIFY_NATS_KEYS, NOTIFY_NATS_SUB_SYS,
-    NOTIFY_PULSAR_KEYS, NOTIFY_PULSAR_SUB_SYS, NOTIFY_WEBHOOK_KEYS, NOTIFY_WEBHOOK_SUB_SYS,
+    NOTIFY_AMQP_KEYS, NOTIFY_AMQP_SUB_SYS, NOTIFY_KAFKA_KEYS, NOTIFY_KAFKA_SUB_SYS, NOTIFY_MQTT_KEYS, NOTIFY_MQTT_SUB_SYS,
+    NOTIFY_MYSQL_KEYS, NOTIFY_MYSQL_SUB_SYS, NOTIFY_NATS_KEYS, NOTIFY_NATS_SUB_SYS, NOTIFY_POSTGRES_KEYS,
+    NOTIFY_POSTGRES_SUB_SYS, NOTIFY_PULSAR_KEYS, NOTIFY_PULSAR_SUB_SYS, NOTIFY_REDIS_KEYS, NOTIFY_REDIS_SUB_SYS,
+    NOTIFY_WEBHOOK_KEYS, NOTIFY_WEBHOOK_SUB_SYS,
 };
 use rustfs_config::oidc::{IDENTITY_OPENID_KEYS, IDENTITY_OPENID_SUB_SYS, OIDC_REDIRECT_URI_DYNAMIC};
 use rustfs_config::{COMMENT_KEY, DEFAULT_DELIMITER, ENABLE_KEY, EnableState, RUSTFS_REGION};
@@ -58,7 +61,7 @@ struct TargetConfigDescriptor {
     valid_keys: &'static [&'static str],
 }
 
-fn notify_target_descriptors() -> [TargetConfigDescriptor; 5] {
+fn notify_target_descriptors() -> [TargetConfigDescriptor; 9] {
     [
         TargetConfigDescriptor {
             external_key: "webhook",
@@ -66,6 +69,12 @@ fn notify_target_descriptors() -> [TargetConfigDescriptor; 5] {
             default_kvs: &notify::DEFAULT_NOTIFY_WEBHOOK_KVS,
             valid_keys: NOTIFY_WEBHOOK_KEYS,
         },
+        TargetConfigDescriptor {
+            external_key: "amqp",
+            subsystem_key: NOTIFY_AMQP_SUB_SYS,
+            default_kvs: &notify::DEFAULT_NOTIFY_AMQP_KVS,
+            valid_keys: NOTIFY_AMQP_KEYS,
+        },
         TargetConfigDescriptor {
             external_key: "kafka",
             subsystem_key: NOTIFY_KAFKA_SUB_SYS,
@@ -78,12 +87,30 @@ fn notify_target_descriptors() -> [TargetConfigDescriptor; 5] {
             default_kvs: &notify::DEFAULT_NOTIFY_MQTT_KVS,
             valid_keys: NOTIFY_MQTT_KEYS,
         },
+        TargetConfigDescriptor {
+            external_key: "mysql",
+            subsystem_key: NOTIFY_MYSQL_SUB_SYS,
+            default_kvs: &notify::DEFAULT_NOTIFY_MYSQL_KVS,
+            valid_keys: NOTIFY_MYSQL_KEYS,
+        },
         TargetConfigDescriptor {
             external_key: "nats",
             subsystem_key: NOTIFY_NATS_SUB_SYS,
             default_kvs: &notify::DEFAULT_NOTIFY_NATS_KVS,
             valid_keys: NOTIFY_NATS_KEYS,
         },
+        TargetConfigDescriptor {
+            external_key: "postgres",
+            subsystem_key: NOTIFY_POSTGRES_SUB_SYS,
+            default_kvs: &notify::DEFAULT_NOTIFY_POSTGRES_KVS,
+            valid_keys: NOTIFY_POSTGRES_KEYS,
+        },
+        TargetConfigDescriptor {
+            external_key: "redis",
+            subsystem_key: NOTIFY_REDIS_SUB_SYS,
+            default_kvs: &notify::DEFAULT_NOTIFY_REDIS_KVS,
+            valid_keys: NOTIFY_REDIS_KEYS,
+        },
         TargetConfigDescriptor {
             external_key: "pulsar",
             subsystem_key: NOTIFY_PULSAR_SUB_SYS,
@@ -93,7 +120,7 @@ fn notify_target_descriptors() -> [TargetConfigDescriptor; 5] {
     ]
 }
 
-fn audit_target_descriptors() -> [TargetConfigDescriptor; 5] {
+fn audit_target_descriptors() -> [TargetConfigDescriptor; 9] {
     [
         TargetConfigDescriptor {
             external_key: "webhook",
@@ -101,6 +128,12 @@ fn audit_target_descriptors() -> [TargetConfigDescriptor; 5] {
             default_kvs: &audit::DEFAULT_AUDIT_WEBHOOK_KVS,
             valid_keys: AUDIT_WEBHOOK_KEYS,
         },
+        TargetConfigDescriptor {
+            external_key: "amqp",
+            subsystem_key: AUDIT_AMQP_SUB_SYS,
+            default_kvs: &audit::DEFAULT_AUDIT_AMQP_KVS,
+            valid_keys: AUDIT_AMQP_KEYS,
+        },
         TargetConfigDescriptor {
             external_key: "kafka",
             subsystem_key: AUDIT_KAFKA_SUB_SYS,
@@ -113,18 +146,36 @@ fn audit_target_descriptors() -> [TargetConfigDescriptor; 5] {
             default_kvs: &audit::DEFAULT_AUDIT_MQTT_KVS,
             valid_keys: AUDIT_MQTT_KEYS,
         },
+        TargetConfigDescriptor {
+            external_key: "mysql",
+            subsystem_key: AUDIT_MYSQL_SUB_SYS,
+            default_kvs: &audit::DEFAULT_AUDIT_MYSQL_KVS,
+            valid_keys: AUDIT_MYSQL_KEYS,
+        },
         TargetConfigDescriptor {
             external_key: "nats",
             subsystem_key: AUDIT_NATS_SUB_SYS,
             default_kvs: &audit::DEFAULT_AUDIT_NATS_KVS,
             valid_keys: AUDIT_NATS_KEYS,
         },
+        TargetConfigDescriptor {
+            external_key: "postgres",
+            subsystem_key: AUDIT_POSTGRES_SUB_SYS,
+            default_kvs: &audit::DEFAULT_AUDIT_POSTGRES_KVS,
+            valid_keys: AUDIT_POSTGRES_KEYS,
+        },
         TargetConfigDescriptor {
             external_key: "pulsar",
             subsystem_key: AUDIT_PULSAR_SUB_SYS,
             default_kvs: &audit::DEFAULT_AUDIT_PULSAR_KVS,
             valid_keys: AUDIT_PULSAR_KEYS,
         },
+        TargetConfigDescriptor {
+            external_key: "redis",
+            subsystem_key: AUDIT_REDIS_SUB_SYS,
+            default_kvs: &audit::DEFAULT_AUDIT_REDIS_KVS,
+            valid_keys: AUDIT_REDIS_KEYS,
+        },
     ]
 }
 
@@ -679,6 +730,8 @@ fn is_target_bool_key(key: &str) -> bool {
     matches!(
         key,
         ENABLE_KEY
+            | rustfs_config::AMQP_MANDATORY
+            | rustfs_config::AMQP_PERSISTENT
             | rustfs_config::WEBHOOK_SKIP_TLS_VERIFY
             | rustfs_config::KAFKA_TLS_ENABLE
             | rustfs_config::MQTT_TLS_TRUST_LEAF_AS_CA
@@ -688,14 +741,31 @@ fn is_target_bool_key(key: &str) -> bool {
     )
 }
 
+fn parse_target_bool_scalar(value: &str) -> Option<bool> {
+    if let Ok(state) = value.parse::<EnableState>() {
+        return Some(state.is_enabled());
+    }
+    if let Ok(boolean) = value.parse::<bool>() {
+        return Some(boolean);
+    }
+    None
+}
+
+fn target_scalar_values_equal(key: &str, lhs: &str, rhs: &str) -> bool {
+    if is_target_bool_key(key)
+        && let (Some(lhs), Some(rhs)) = (parse_target_bool_scalar(lhs), parse_target_bool_scalar(rhs))
+    {
+        return lhs == rhs;
+    }
+
+    lhs == rhs
+}
+
 fn encode_target_scalar_value(key: &str, value: &str) -> Value {
-    if is_target_bool_key(key) {
-        if let Ok(state) = value.parse::<EnableState>() {
-            return Value::Bool(state.is_enabled());
-        }
-        if let Ok(boolean) = value.parse::<bool>() {
-            return Value::Bool(boolean);
-        }
+    if is_target_bool_key(key)
+        && let Some(boolean) = parse_target_bool_scalar(value)
+    {
+        return Value::Bool(boolean);
     }
 
     Value::String(value.to_string())
@@ -721,7 +791,7 @@ fn build_target_instance_diff_object(kvs: &KVS, baseline: &KVS, valid_keys: &[&s
         let baseline_value = baseline.lookup(key).unwrap_or_default();
         let effective_value = kvs.lookup(key).unwrap_or_else(|| baseline_value.clone());
 
-        if effective_value == baseline_value {
+        if target_scalar_values_equal(key, &effective_value, &baseline_value) {
             continue;
         }
 
@@ -1131,10 +1201,14 @@ mod tests {
         ObjectOptions, ObjectToDelete, PartInfo, PutObjReader, StorageAPI, WalkOptions,
     };
     use http::HeaderMap;
-    use rustfs_config::audit::{AUDIT_KAFKA_SUB_SYS, AUDIT_MQTT_SUB_SYS, AUDIT_WEBHOOK_SUB_SYS};
-    use rustfs_config::notify::{NOTIFY_KAFKA_SUB_SYS, NOTIFY_MQTT_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS};
+    use rustfs_config::audit::{AUDIT_AMQP_SUB_SYS, AUDIT_KAFKA_SUB_SYS, AUDIT_MQTT_SUB_SYS, AUDIT_WEBHOOK_SUB_SYS};
+    use rustfs_config::notify::{
+        NOTIFY_AMQP_SUB_SYS, NOTIFY_KAFKA_SUB_SYS, NOTIFY_MQTT_SUB_SYS, NOTIFY_MYSQL_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS,
+    };
     use rustfs_config::oidc::IDENTITY_OPENID_SUB_SYS;
-    use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY, EnableState};
+    use rustfs_config::{
+        DEFAULT_DELIMITER, ENABLE_KEY, EnableState, MYSQL_DSN_STRING, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_TABLE,
+    };
     use rustfs_filemeta::FileInfo;
     use rustfs_lock::client::LockClient;
     use rustfs_lock::client::local::LocalClient;
@@ -1775,6 +1849,24 @@ mod tests {
                 "acks":"all",
                 "tls_enable":true
               }
+            },
+            "amqp":{
+              "primary":{
+                "enable":true,
+                "url":"amqp://127.0.0.1:5672/%2f",
+                "exchange":"rustfs.events",
+                "routing_key":"objects",
+                "persistent":true
+              }
+            },
+            "mysql":{
+              "primary":{
+                "enable":true,
+                "dsn_string":"rustfs:password@tcp(127.0.0.1:3306)/rustfs_events",
+                "table":"rustfs_events",
+                "queue_dir":"/tmp/mysql-queue",
+                "max_open_connections":"2"
+              }
             }
           }
         }"#;
@@ -1811,6 +1903,24 @@ mod tests {
         assert_eq!(kafka.get(rustfs_config::KAFKA_TOPIC), "events-kafka");
         assert_eq!(kafka.get(rustfs_config::KAFKA_ACKS), "all");
         assert_eq!(kafka.get(rustfs_config::KAFKA_TLS_ENABLE), "true");
+
+        let amqp = cfg
+            .get_value(NOTIFY_AMQP_SUB_SYS, "primary")
+            .expect("amqp target should be decoded");
+        assert_eq!(amqp.get(ENABLE_KEY), EnableState::On.to_string());
+        assert_eq!(amqp.get(rustfs_config::AMQP_URL), "amqp://127.0.0.1:5672/%2f");
+        assert_eq!(amqp.get(rustfs_config::AMQP_EXCHANGE), "rustfs.events");
+        assert_eq!(amqp.get(rustfs_config::AMQP_ROUTING_KEY), "objects");
+        assert_eq!(amqp.get(rustfs_config::AMQP_PERSISTENT), "true");
+
+        let mysql = cfg
+            .get_value(NOTIFY_MYSQL_SUB_SYS, "primary")
+            .expect("mysql target should be decoded");
+        assert_eq!(mysql.get(ENABLE_KEY), EnableState::On.to_string());
+        assert_eq!(mysql.get(MYSQL_DSN_STRING), "rustfs:password@tcp(127.0.0.1:3306)/rustfs_events");
+        assert_eq!(mysql.get(MYSQL_TABLE), "rustfs_events");
+        assert_eq!(mysql.get(MYSQL_QUEUE_DIR), "/tmp/mysql-queue");
+        assert_eq!(mysql.get(MYSQL_MAX_OPEN_CONNECTIONS), "2");
     }
 
     #[test]
@@ -1870,6 +1980,15 @@ mod tests {
                 "queue_dir":"/tmp/audit-queue"
               }
             },
+            "amqp":{
+              "primary":{
+                "enable":true,
+                "url":"amqp://127.0.0.1:5672/%2f",
+                "exchange":"rustfs.audit",
+                "routing_key":"audit",
+                "persistent":true
+              }
+            },
             "mqtt":{
               "default":{
                 "enable":true,
@@ -1901,6 +2020,15 @@ mod tests {
         assert_eq!(webhook.get(rustfs_config::WEBHOOK_ENDPOINT), "https://example.com/audit-hook");
         assert_eq!(webhook.get(rustfs_config::WEBHOOK_QUEUE_DIR), "/tmp/audit-queue");
 
+        let amqp = cfg
+            .get_value(AUDIT_AMQP_SUB_SYS, "primary")
+            .expect("audit amqp target should be decoded");
+        assert_eq!(amqp.get(ENABLE_KEY), EnableState::On.to_string());
+        assert_eq!(amqp.get(rustfs_config::AMQP_URL), "amqp://127.0.0.1:5672/%2f");
+        assert_eq!(amqp.get(rustfs_config::AMQP_EXCHANGE), "rustfs.audit");
+        assert_eq!(amqp.get(rustfs_config::AMQP_ROUTING_KEY), "audit");
+        assert_eq!(amqp.get(rustfs_config::AMQP_PERSISTENT), "true");
+
         let mqtt_default = cfg
             .get_value(AUDIT_MQTT_SUB_SYS, DEFAULT_DELIMITER)
             .expect("audit mqtt default should be decoded");
@@ -2069,6 +2197,44 @@ mod tests {
         );
         cfg.0.insert(NOTIFY_KAFKA_SUB_SYS.to_string(), kafka_section);
 
+        let mut amqp_section = std::collections::HashMap::new();
+        amqp_section.insert(
+            "primary".to_string(),
+            crate::config::KVS(vec![
+                crate::config::KV {
+                    key: ENABLE_KEY.to_string(),
+                    value: EnableState::On.to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_URL.to_string(),
+                    value: "amqp://127.0.0.1:5672/%2f".to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_EXCHANGE.to_string(),
+                    value: "rustfs.events".to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_ROUTING_KEY.to_string(),
+                    value: "objects".to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_MANDATORY.to_string(),
+                    value: "false".to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_PERSISTENT.to_string(),
+                    value: "false".to_string(),
+                    hidden_if_empty: false,
+                },
+            ]),
+        );
+        cfg.0.insert(NOTIFY_AMQP_SUB_SYS.to_string(), amqp_section);
+
         let out = encode_server_config_blob(&cfg, None).expect("encode should succeed");
         let v: Value = serde_json::from_slice(&out).expect("output should be json");
         let notify = v
@@ -2117,6 +2283,21 @@ mod tests {
         );
         assert_eq!(kafka.get(rustfs_config::KAFKA_ACKS).and_then(Value::as_str), Some("all"));
         assert_eq!(kafka.get(rustfs_config::KAFKA_TLS_ENABLE).and_then(Value::as_bool), Some(true));
+
+        let amqp = notify
+            .get("amqp")
+            .and_then(Value::as_object)
+            .and_then(|targets| targets.get("primary"))
+            .and_then(Value::as_object)
+            .expect("amqp target should be encoded");
+        assert_eq!(
+            amqp.get(rustfs_config::AMQP_URL).and_then(Value::as_str),
+            Some("amqp://127.0.0.1:5672/%2f")
+        );
+        assert_eq!(amqp.get(rustfs_config::AMQP_EXCHANGE).and_then(Value::as_str), Some("rustfs.events"));
+        assert_eq!(amqp.get(rustfs_config::AMQP_ROUTING_KEY).and_then(Value::as_str), Some("objects"));
+        assert!(!amqp.contains_key(rustfs_config::AMQP_MANDATORY));
+        assert_eq!(amqp.get(rustfs_config::AMQP_PERSISTENT).and_then(Value::as_bool), Some(false));
     }
 
     #[test]
@@ -2146,6 +2327,44 @@ mod tests {
         );
         cfg.0.insert(AUDIT_WEBHOOK_SUB_SYS.to_string(), webhook_section);
 
+        let mut amqp_section = std::collections::HashMap::new();
+        amqp_section.insert(
+            "primary".to_string(),
+            crate::config::KVS(vec![
+                crate::config::KV {
+                    key: ENABLE_KEY.to_string(),
+                    value: EnableState::On.to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_URL.to_string(),
+                    value: "amqp://127.0.0.1:5672/%2f".to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_EXCHANGE.to_string(),
+                    value: "rustfs.audit".to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_ROUTING_KEY.to_string(),
+                    value: "audit".to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_MANDATORY.to_string(),
+                    value: "false".to_string(),
+                    hidden_if_empty: false,
+                },
+                crate::config::KV {
+                    key: rustfs_config::AMQP_PERSISTENT.to_string(),
+                    value: "false".to_string(),
+                    hidden_if_empty: false,
+                },
+            ]),
+        );
+        cfg.0.insert(AUDIT_AMQP_SUB_SYS.to_string(), amqp_section);
+
         let mut mqtt_default = audit::DEFAULT_AUDIT_MQTT_KVS.clone();
         mqtt_default.insert(ENABLE_KEY.to_string(), EnableState::On.to_string());
         mqtt_default.insert(rustfs_config::MQTT_TOPIC.to_string(), "audit-events".to_string());
@@ -2208,6 +2427,21 @@ mod tests {
         );
         assert_eq!(webhook.get(ENABLE_KEY).and_then(Value::as_bool), Some(true));
 
+        let amqp = logger
+            .get("amqp")
+            .and_then(Value::as_object)
+            .and_then(|targets| targets.get("primary"))
+            .and_then(Value::as_object)
+            .expect("audit amqp target should be encoded");
+        assert_eq!(
+            amqp.get(rustfs_config::AMQP_URL).and_then(Value::as_str),
+            Some("amqp://127.0.0.1:5672/%2f")
+        );
+        assert_eq!(amqp.get(rustfs_config::AMQP_EXCHANGE).and_then(Value::as_str), Some("rustfs.audit"));
+        assert_eq!(amqp.get(rustfs_config::AMQP_ROUTING_KEY).and_then(Value::as_str), Some("audit"));
+        assert!(!amqp.contains_key(rustfs_config::AMQP_MANDATORY));
+        assert_eq!(amqp.get(rustfs_config::AMQP_PERSISTENT).and_then(Value::as_bool), Some(false));
+
         let mqtt_default = logger
             .get("mqtt")
             .and_then(Value::as_object)

crates/ecstore/src/config/mod.rs

@@ -26,10 +26,12 @@ use com::{STORAGE_CLASS_SUB_SYS, lookup_configs, read_config_without_migrate};
 use rustfs_config::COMMENT_KEY;
 use rustfs_config::DEFAULT_DELIMITER;
 use rustfs_config::audit::{
-    AUDIT_KAFKA_SUB_SYS, AUDIT_MQTT_SUB_SYS, AUDIT_NATS_SUB_SYS, AUDIT_PULSAR_SUB_SYS, AUDIT_WEBHOOK_SUB_SYS,
+    AUDIT_AMQP_SUB_SYS, AUDIT_KAFKA_SUB_SYS, AUDIT_MQTT_SUB_SYS, AUDIT_MYSQL_SUB_SYS, AUDIT_NATS_SUB_SYS, AUDIT_POSTGRES_SUB_SYS,
+    AUDIT_PULSAR_SUB_SYS, AUDIT_REDIS_SUB_SYS, AUDIT_WEBHOOK_SUB_SYS,
 };
 use rustfs_config::notify::{
-    NOTIFY_KAFKA_SUB_SYS, NOTIFY_MQTT_SUB_SYS, NOTIFY_NATS_SUB_SYS, NOTIFY_PULSAR_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS,
+    NOTIFY_AMQP_SUB_SYS, NOTIFY_KAFKA_SUB_SYS, NOTIFY_MQTT_SUB_SYS, NOTIFY_MYSQL_SUB_SYS, NOTIFY_NATS_SUB_SYS,
+    NOTIFY_POSTGRES_SUB_SYS, NOTIFY_PULSAR_SUB_SYS, NOTIFY_REDIS_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS,
 };
 use rustfs_config::oidc::IDENTITY_OPENID_SUB_SYS;
 use serde::{Deserialize, Serialize};
@@ -42,10 +44,6 @@ pub static DEFAULT_KVS: LazyLock<OnceLock<HashMap<String, KVS>>> = LazyLock::new
 pub static GLOBAL_SERVER_CONFIG: LazyLock<OnceLock<Config>> = LazyLock::new(OnceLock::new);
 pub static GLOBAL_CONFIG_SYS: LazyLock<ConfigSys> = LazyLock::new(ConfigSys::new);
 
-pub const ENV_ACCESS_KEY: &str = "RUSTFS_ACCESS_KEY";
-pub const ENV_SECRET_KEY: &str = "RUSTFS_SECRET_KEY";
-pub const ENV_ROOT_USER: &str = "RUSTFS_ROOT_USER";
-pub const ENV_ROOT_PASSWORD: &str = "RUSTFS_ROOT_PASSWORD";
 pub static RUSTFS_CONFIG_PREFIX: &str = "config";
 
 pub struct ConfigSys {}
@@ -245,12 +243,20 @@ pub fn init() {
     kvs.insert(AUDIT_WEBHOOK_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_WEBHOOK_KVS.clone());
     kvs.insert(NOTIFY_MQTT_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_MQTT_KVS.clone());
     kvs.insert(AUDIT_MQTT_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_MQTT_KVS.clone());
+    kvs.insert(NOTIFY_AMQP_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_AMQP_KVS.clone());
+    kvs.insert(AUDIT_AMQP_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_AMQP_KVS.clone());
     kvs.insert(NOTIFY_NATS_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_NATS_KVS.clone());
     kvs.insert(AUDIT_NATS_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_NATS_KVS.clone());
+    kvs.insert(NOTIFY_REDIS_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_REDIS_KVS.clone());
+    kvs.insert(AUDIT_REDIS_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_REDIS_KVS.clone());
+    kvs.insert(NOTIFY_POSTGRES_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_POSTGRES_KVS.clone());
+    kvs.insert(AUDIT_POSTGRES_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_POSTGRES_KVS.clone());
     kvs.insert(NOTIFY_PULSAR_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_PULSAR_KVS.clone());
     kvs.insert(AUDIT_PULSAR_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_PULSAR_KVS.clone());
     kvs.insert(NOTIFY_KAFKA_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_KAFKA_KVS.clone());
     kvs.insert(AUDIT_KAFKA_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_KAFKA_KVS.clone());
+    kvs.insert(NOTIFY_MYSQL_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_MYSQL_KVS.clone());
+    kvs.insert(AUDIT_MYSQL_SUB_SYS.to_owned(), audit::DEFAULT_AUDIT_MYSQL_KVS.clone());
     kvs.insert(IDENTITY_OPENID_SUB_SYS.to_owned(), oidc::DEFAULT_IDENTITY_OPENID_KVS.clone());
 
     // Register all default configurations

crates/ecstore/src/config/notify.rs

@@ -13,16 +13,26 @@
 // limitations under the License.
 
 use crate::config::{KV, KVS};
+use rustfs_config::notify::NOTIFY_REDIS_DEFAULT_CHANNEL;
 use rustfs_config::{
-    COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY, EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR,
-    KAFKA_QUEUE_LIMIT, KAFKA_TLS_CA, KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY, KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER,
-    MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA,
-    MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME,
-    MQTT_WS_PATH_ALLOWLIST, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT,
-    NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, PULSAR_AUTH_TOKEN,
-    PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
-    PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT,
-    WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY,
+    AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_QUEUE_LIMIT, AMQP_ROUTING_KEY,
+    AMQP_TLS_CA, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY,
+    EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_TLS_CA,
+    KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY, KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL,
+    MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT,
+    MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST,
+    MYSQL_DSN_STRING, MYSQL_FORMAT, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA,
+    MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR,
+    NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN,
+    NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE,
+    POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER,
+    PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
+    PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT,
+    REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD,
+    REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT,
+    REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL,
+    REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT,
+    WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY,
 };
 use std::sync::LazyLock;
 
@@ -176,6 +186,81 @@ pub static DEFAULT_NOTIFY_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
     ])
 });
 
+pub static DEFAULT_NOTIFY_AMQP_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: ENABLE_KEY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_URL.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_EXCHANGE.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_ROUTING_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_MANDATORY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_PERSISTENT.to_owned(),
+            value: EnableState::On.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_USERNAME.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_PASSWORD.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: AMQP_TLS_CA.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: AMQP_TLS_CLIENT_CERT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: AMQP_TLS_CLIENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: AMQP_QUEUE_DIR.to_owned(),
+            value: EVENT_DEFAULT_DIR.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: AMQP_QUEUE_LIMIT.to_owned(),
+            value: DEFAULT_LIMIT.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: COMMENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+    ])
+});
+
 pub static DEFAULT_NOTIFY_NATS_KVS: LazyLock<KVS> = LazyLock::new(|| {
     KVS(vec![
         KV {
@@ -316,6 +401,176 @@ pub static DEFAULT_NOTIFY_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(|| {
     ])
 });
 
+pub static DEFAULT_NOTIFY_REDIS_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: ENABLE_KEY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_URL.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_CHANNEL.to_owned(),
+            value: NOTIFY_REDIS_DEFAULT_CHANNEL.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_USERNAME.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_PASSWORD.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_KEEP_ALIVE_INTERVAL.to_owned(),
+            value: "15".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_QUEUE_DIR.to_owned(),
+            value: EVENT_DEFAULT_DIR.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_QUEUE_LIMIT.to_owned(),
+            value: DEFAULT_LIMIT.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_MAX_RETRY_ATTEMPTS.to_owned(),
+            value: "3".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_RECONNECT_RETRY_ATTEMPTS.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_MIN_RETRY_DELAY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_MAX_RETRY_DELAY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_CONNECTION_TIMEOUT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_RESPONSE_TIMEOUT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_PIPELINE_BUFFER_SIZE.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: REDIS_TLS_POLICY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_TLS_CA.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_TLS_CLIENT_CERT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_TLS_CLIENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: REDIS_TLS_ALLOW_INSECURE.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: COMMENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+    ])
+});
+
+pub static DEFAULT_NOTIFY_POSTGRES_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: ENABLE_KEY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_DSN_STRING.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: POSTGRES_TABLE.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_FORMAT.to_owned(),
+            value: "namespace".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_TLS_REQUIRED.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_TLS_CA.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: POSTGRES_TLS_CLIENT_CERT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: POSTGRES_TLS_CLIENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: POSTGRES_QUEUE_DIR.to_owned(),
+            value: EVENT_DEFAULT_DIR.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: POSTGRES_QUEUE_LIMIT.to_owned(),
+            value: DEFAULT_LIMIT.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: COMMENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+    ])
+});
+
 pub static DEFAULT_NOTIFY_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(|| {
     KVS(vec![
         KV {
@@ -375,3 +630,64 @@ pub static DEFAULT_NOTIFY_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(|| {
         },
     ])
 });
+
+/// MySQL notification target default configuration
+pub static DEFAULT_NOTIFY_MYSQL_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: ENABLE_KEY.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_DSN_STRING.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: MYSQL_TABLE.to_owned(),
+            value: "rustfs_events".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_FORMAT.to_owned(),
+            value: "access".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_TLS_CA.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: MYSQL_TLS_CLIENT_CERT.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: MYSQL_TLS_CLIENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: MYSQL_QUEUE_DIR.to_owned(),
+            value: EVENT_DEFAULT_DIR.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_QUEUE_LIMIT.to_owned(),
+            value: DEFAULT_LIMIT.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: MYSQL_MAX_OPEN_CONNECTIONS.to_owned(),
+            value: "2".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: COMMENT_KEY.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+    ])
+});

crates/ecstore/src/data_usage.rs

@@ -15,7 +15,11 @@
 pub mod local_snapshot;
 
 use crate::{
-    bucket::metadata_sys::get_replication_config, config::com::read_config, disk::DiskAPI, error::Error, store::ECStore,
+    bucket::metadata_sys::get_replication_config,
+    config::com::read_config,
+    disk::DiskAPI,
+    error::{Error, classify_system_path_failure_reason},
+    store::ECStore,
     store_api::ListOperations,
 };
 pub use local_snapshot::{
@@ -26,6 +30,7 @@ pub use local_snapshot::{
 use rustfs_common::data_usage::{
     BucketTargetUsageInfo, BucketUsageInfo, DataUsageCache, DataUsageEntry, DataUsageInfo, DiskUsageStatus, SizeSummary,
 };
+use rustfs_io_metrics::record_system_path_failure;
 use rustfs_utils::path::SLASH_SEPARATOR;
 use std::{
     collections::{HashMap, HashSet, hash_map::Entry},
@@ -109,7 +114,16 @@ pub async fn load_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataUsa
     let buf: Vec<u8> = match read_config(store.clone(), &DATA_USAGE_OBJ_NAME_PATH).await {
         Ok(data) => data,
         Err(e) => {
-            error!("Failed to read data usage info from backend: {}", e);
+            let reason = classify_system_path_failure_reason(&e);
+            record_system_path_failure("data_usage", "read_primary", reason);
+            error!(
+                path_kind = "data_usage",
+                operation = "read_primary",
+                reason,
+                object = %DATA_USAGE_OBJ_NAME_PATH.as_str(),
+                error = %e,
+                "system path read failed"
+            );
 
             match read_config(store.clone(), format!("{}.bkp", DATA_USAGE_OBJ_NAME_PATH.as_str()).as_str()).await {
                 Ok(data) => data,
@@ -117,7 +131,16 @@ pub async fn load_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataUsa
                     if e == Error::ConfigNotFound {
                         return Ok(DataUsageInfo::default());
                     }
-                    error!("Failed to read data usage info from backend: {}", e);
+                    let reason = classify_system_path_failure_reason(&e);
+                    record_system_path_failure("data_usage", "read_backup", reason);
+                    error!(
+                        path_kind = "data_usage",
+                        operation = "read_backup",
+                        reason,
+                        object = %format!("{}.bkp", DATA_USAGE_OBJ_NAME_PATH.as_str()),
+                        error = %e,
+                        "system path read failed"
+                    );
                     return Err(Error::other(e));
                 }
             }

crates/ecstore/src/disk/disk_store.rs

@@ -31,7 +31,7 @@ use std::{
     path::PathBuf,
     sync::{
         Arc,
-        atomic::{AtomicI64, AtomicU32, Ordering},
+        atomic::{AtomicI64, AtomicU32, AtomicU64, Ordering},
     },
     time::Duration,
 };
@@ -138,6 +138,14 @@ pub struct DiskHealthTracker {
     pub offline_since_unix_secs: AtomicI64,
     /// Last runtime state transition timestamp
     pub last_transition_unix_secs: AtomicI64,
+    /// Last successfully probed total space in bytes
+    pub last_capacity_total: AtomicU64,
+    /// Last successfully probed used space in bytes
+    pub last_capacity_used: AtomicU64,
+    /// Last successfully probed free space in bytes
+    pub last_capacity_free: AtomicU64,
+    /// Last successful capacity probe timestamp
+    pub last_capacity_probe_unix_secs: AtomicI64,
 }
 
 impl DiskHealthTracker {
@@ -158,6 +166,10 @@ impl DiskHealthTracker {
             consecutive_successes: AtomicU32::new(0),
             offline_since_unix_secs: AtomicI64::new(0),
             last_transition_unix_secs: AtomicI64::new(now / 1_000_000_000),
+            last_capacity_total: AtomicU64::new(0),
+            last_capacity_used: AtomicU64::new(0),
+            last_capacity_free: AtomicU64::new(0),
+            last_capacity_probe_unix_secs: AtomicI64::new(0),
         }
     }
 
@@ -170,6 +182,28 @@ impl DiskHealthTracker {
         self.last_success.store(now, Ordering::Relaxed);
     }
 
+    pub fn record_capacity_probe(&self, total: u64, used: u64, free: u64) {
+        self.last_capacity_total.store(total, Ordering::Release);
+        self.last_capacity_used.store(used, Ordering::Release);
+        self.last_capacity_free.store(free, Ordering::Release);
+        self.last_capacity_probe_unix_secs
+            .store(current_unix_secs() as i64, Ordering::Release);
+    }
+
+    pub fn last_capacity_snapshot(&self) -> Option<(u64, u64, u64, u64)> {
+        let ts = self.last_capacity_probe_unix_secs.load(Ordering::Acquire);
+        if ts <= 0 {
+            return None;
+        }
+
+        Some((
+            self.last_capacity_total.load(Ordering::Acquire),
+            self.last_capacity_used.load(Ordering::Acquire),
+            self.last_capacity_free.load(Ordering::Acquire),
+            ts as u64,
+        ))
+    }
+
     /// Check if disk is faulty
     pub fn is_faulty(&self) -> bool {
         self.status.load(Ordering::Acquire) == DISK_HEALTH_FAULTY
@@ -265,6 +299,26 @@ impl DiskHealthTracker {
         true
     }
 
+    /// Clear faulty/offline state so a store-init format load retry can issue RPC again.
+    ///
+    /// Remote disks are marked faulty on timeout/network errors; the init loop retries with the
+    /// same [`DiskStore`] handles, which would otherwise fail immediately at `is_faulty()`.
+    pub fn reset_for_store_init_retry(&self, endpoint: &Endpoint) {
+        self.status.store(DISK_HEALTH_OK, Ordering::Release);
+        self.runtime_state
+            .store(RuntimeDriveHealthState::Online as u32, Ordering::Release);
+        self.consecutive_failures.store(0, Ordering::Release);
+        self.consecutive_successes.store(0, Ordering::Release);
+        self.offline_since_unix_secs.store(0, Ordering::Release);
+        self.waiting.store(0, Ordering::Release);
+        let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap();
+        let now_nanos = now.as_nanos() as i64;
+        self.last_success.store(now_nanos, Ordering::Relaxed);
+        self.last_started.store(now_nanos, Ordering::Relaxed);
+        self.last_transition_unix_secs.store(now.as_secs() as i64, Ordering::Release);
+        record_drive_runtime_state(endpoint, RuntimeDriveHealthState::Online);
+    }
+
     pub fn mark_recovery_success(&self, endpoint: &Endpoint, reason: &'static str) -> bool {
         let current = self.runtime_state();
         let next = match current {
@@ -443,11 +497,24 @@ impl LocalDiskWrapper {
         self.health.offline_duration().map(|duration| duration.as_secs())
     }
 
+    pub fn last_capacity_snapshot(&self) -> Option<(u64, u64, u64, u64)> {
+        self.health.last_capacity_snapshot()
+    }
+
+    pub fn record_capacity_probe(&self, total: u64, used: u64, free: u64) {
+        self.health.record_capacity_probe(total, used, free);
+    }
+
     #[cfg(test)]
     pub fn force_runtime_state_for_test(&self, state: RuntimeDriveHealthState) {
         self.health.force_runtime_state_for_test(state);
     }
 
+    /// Same as [`DiskHealthTracker::reset_for_store_init_retry`]: undo a transient faulty mark before another format load attempt.
+    pub fn reset_health_for_store_init_retry(&self) {
+        self.health.reset_for_store_init_retry(&self.disk.endpoint());
+    }
+
     /// Enable health monitoring after disk creation.
     /// Used to defer health checks until after startup format loading completes.
     pub fn enable_health_check(&self) {
@@ -796,7 +863,7 @@ impl LocalDiskWrapper {
                     timeout_ms = timeout_duration.as_millis(),
                     "Local disk operation timed out"
                 );
-                Err(DiskError::other(format!("disk operation timeout after {timeout_duration:?}")))
+                Err(DiskError::Timeout)
             }
         }
     }
@@ -1255,4 +1322,21 @@ mod tests {
         assert!(!health.is_faulty());
         assert!(health.offline_duration().is_none());
     }
+
+    #[test]
+    fn reset_for_store_init_retry_clears_faulty_and_back_online() {
+        let endpoint = Endpoint::try_from("/tmp/reset-store-init-retry").expect("endpoint should parse");
+        let health = DiskHealthTracker::new();
+
+        assert!(health.mark_offline(&endpoint, "simulated_fault"));
+        assert!(health.is_faulty());
+        assert_eq!(health.runtime_state(), RuntimeDriveHealthState::Offline);
+
+        health.reset_for_store_init_retry(&endpoint);
+        assert!(!health.is_faulty());
+        assert_eq!(health.runtime_state(), RuntimeDriveHealthState::Online);
+
+        assert!(health.mark_offline(&endpoint, "again"));
+        assert!(health.is_faulty());
+    }
 }

crates/ecstore/src/disk/local.rs

@@ -422,7 +422,7 @@ impl LocalDisk {
                             total: info.total,
                             free: info.free,
                             used: info.used,
-                            used_inodes: info.files - info.ffree,
+                            used_inodes: info.files.saturating_sub(info.ffree),
                             free_inodes: info.ffree,
                             major: info.major,
                             minor: info.minor,
@@ -1247,29 +1247,16 @@ impl LocalDisk {
         W: AsyncWrite + Unpin + Send,
     {
         let forward = {
-            opts.forward_to.as_ref().filter(|v| v.starts_with(&*current)).map(|v| {
-                let forward = v.trim_start_matches(&*current);
-                if let Some(idx) = forward.find('/') {
-                    forward[..idx].to_owned()
-                } else {
-                    forward.to_owned()
-                }
-            })
-            // if let Some(forward_to) = &opts.forward_to {
-
-            // } else {
-            //     None
-            // }
-            // if !opts.forward_to.is_empty() && opts.forward_to.starts_with(&*current) {
-            //     let forward = opts.forward_to.trim_start_matches(&*current);
-            //     if let Some(idx) = forward.find('/') {
-            //         &forward[..idx]
-            //     } else {
-            //         forward
-            //     }
-            // } else {
-            //     ""
-            // }
+            opts.forward_to
+                .as_ref()
+                .and_then(|v| v.strip_prefix(&current))
+                .map(|forward| {
+                    if let Some(idx) = forward.find('/') {
+                        forward[..idx].to_owned()
+                    } else {
+                        forward.to_owned()
+                    }
+                })
         };
 
         if opts.limit > 0 && *objs_returned >= opts.limit {
@@ -1283,6 +1270,7 @@ impl LocalDisk {
             Err(e) => {
                 if e != DiskError::VolumeNotFound && e != Error::FileNotFound {
                     error!("scan list_dir {}, err {:?}", &current, &e);
+                    return Err(e);
                 }
 
                 if opts.report_notfound && e == Error::FileNotFound && current == opts.base_dir {
@@ -2160,6 +2148,8 @@ impl DiskAPI for LocalDisk {
 
     /// Zero-copy file read using memory mapping (Unix) or efficient read (non-Unix).
     /// Returns Bytes that can be shared without copying.
+    // SAFETY: Unix unsafe calls in this function only query page size and mmap
+    // a read-only file region after bounds and alignment are validated.
     #[allow(unsafe_code)]
     #[tracing::instrument(level = "debug", skip(self))]
     async fn read_file_zero_copy(&self, volume: &str, path: &str, offset: usize, length: usize) -> Result<Bytes> {
@@ -2212,6 +2202,8 @@ impl DiskAPI for LocalDisk {
                 // mmap offsets on Unix must be page-size aligned. Align the
                 // mapping down to the nearest page boundary, then slice out the
                 // originally requested logical range.
+                // SAFETY: `sysconf(_SC_PAGESIZE)` has no pointer arguments and
+                // only queries process-global OS configuration.
                 let page_size = unsafe { libc::sysconf(libc::_SC_PAGESIZE) };
                 if page_size <= 0 {
                     return Err(DiskError::other("failed to determine system page size"));
@@ -2665,7 +2657,7 @@ impl DiskAPI for LocalDisk {
 
             let mut xl_meta = FileMeta::load(buf.as_ref())?;
 
-            xl_meta.update_object_version(fi)?;
+            xl_meta.update_object_version_with_opts(fi, opts.replace_user_metadata)?;
 
             let wbuf = xl_meta.marshal_msg()?;
 
@@ -3233,6 +3225,109 @@ mod test {
         assert_eq!(names.iter().filter(|name| *name == "marker/file.txt").count(), 1);
     }
 
+    #[tokio::test]
+    async fn test_scan_dir_forward_to_repeated_prefix_component() {
+        use rustfs_filemeta::MetacacheReader;
+        use tempfile::tempdir;
+
+        let dir = tempdir().unwrap();
+        let bucket = "test-bucket";
+        let bucket_dir = dir.path().join(bucket);
+
+        for name in [
+            "different/prefix/prefix/repo-0000",
+            "different/prefix/prefix/repo-0001",
+            "different/prefix/prefix/repo-0002",
+            "engineering/alpha-0000",
+            "engineering/engineering/engineering/repo-0000",
+            "engineering/engineering/engineering/repo-0001",
+            "engineering/engineering/repo-0000",
+            "engineering/engineering/repo-0001",
+            "engineering/engineering/repo-0002",
+            "engineering/zulu-0000",
+            "unrelated/engineering/repo-0000",
+        ] {
+            let object_dir = bucket_dir.join(name);
+            fs::create_dir_all(&object_dir).await.unwrap();
+            fs::write(object_dir.join(STORAGE_FORMAT_FILE), b"meta").await.unwrap();
+        }
+
+        let endpoint = Endpoint::try_from(dir.path().to_str().unwrap()).unwrap();
+        let disk = LocalDisk::new(&endpoint, false).await.unwrap();
+
+        async fn scan_names(disk: &LocalDisk, bucket: &str, base_dir: &str, forward_to: &str) -> (Vec<String>, i32) {
+            let (reader, mut writer) = tokio::io::duplex(4096);
+            let mut out = MetacacheWriter::new(&mut writer);
+            let opts = WalkDirOptions {
+                bucket: bucket.to_string(),
+                base_dir: base_dir.to_string(),
+                recursive: true,
+                forward_to: Some(forward_to.to_string()),
+                ..Default::default()
+            };
+            let mut objs_returned = 0;
+
+            disk.scan_dir(base_dir.to_string(), "".to_string(), &opts, &mut out, &mut objs_returned, false)
+                .await
+                .unwrap();
+            out.close().await.unwrap();
+            drop(out);
+            drop(writer);
+
+            let mut reader = MetacacheReader::new(reader);
+            let entries = reader.read_all().await.unwrap();
+            let names: Vec<String> = entries
+                .into_iter()
+                .filter(|entry| !entry.metadata.is_empty())
+                .map(|entry| entry.name)
+                .collect();
+
+            (names, objs_returned)
+        }
+
+        let (engineering_names, engineering_count) =
+            scan_names(&disk, bucket, "engineering/", "engineering/engineering/engineering/repo-0001").await;
+
+        assert_eq!(
+            engineering_names,
+            vec![
+                "engineering/engineering/engineering/repo-0001".to_string(),
+                "engineering/engineering/repo-0000".to_string(),
+                "engineering/engineering/repo-0001".to_string(),
+                "engineering/engineering/repo-0002".to_string(),
+                "engineering/zulu-0000".to_string(),
+            ],
+            "forward_to must resume at the requested triply repeated prefix and preserve lexicographic order"
+        );
+        assert_eq!(engineering_count as usize, engineering_names.len());
+
+        let (different_names, different_count) =
+            scan_names(&disk, bucket, "different/", "different/prefix/prefix/repo-0001").await;
+
+        assert_eq!(
+            different_names,
+            vec![
+                "different/prefix/prefix/repo-0001".to_string(),
+                "different/prefix/prefix/repo-0002".to_string(),
+            ],
+            "forward_to must also work for repeated components unrelated to the engineering prefix"
+        );
+        assert_eq!(different_count as usize, different_names.len());
+
+        let (double_names, double_count) = scan_names(&disk, bucket, "engineering/", "engineering/engineering/repo-0001").await;
+
+        assert_eq!(
+            double_names,
+            vec![
+                "engineering/engineering/repo-0001".to_string(),
+                "engineering/engineering/repo-0002".to_string(),
+                "engineering/zulu-0000".to_string(),
+            ],
+            "forward_to must not skip a child directory whose name repeats the base prefix"
+        );
+        assert_eq!(double_count as usize, double_names.len());
+    }
+
     #[tokio::test]
     async fn test_make_volume() {
         let p = "./testv0";

crates/ecstore/src/disk/mod.rs

@@ -427,6 +427,20 @@ impl Disk {
         }
     }
 
+    pub fn last_capacity_snapshot(&self) -> Option<(u64, u64, u64, u64)> {
+        match self {
+            Disk::Local(local_disk) => local_disk.last_capacity_snapshot(),
+            Disk::Remote(remote_disk) => remote_disk.last_capacity_snapshot(),
+        }
+    }
+
+    pub fn record_capacity_probe(&self, total: u64, used: u64, free: u64) {
+        match self {
+            Disk::Local(local_disk) => local_disk.record_capacity_probe(total, used, free),
+            Disk::Remote(remote_disk) => remote_disk.record_capacity_probe(total, used, free),
+        }
+    }
+
     #[cfg(test)]
     pub fn force_runtime_state_for_test(&self, state: RuntimeDriveHealthState) {
         match self {
@@ -437,6 +451,15 @@ impl Disk {
 }
 
 impl Disk {
+    /// Reset drive health so `connect_load_init_formats` retries are not blocked by a prior
+    /// transient mark-faulty (same disk handles are reused across retries).
+    pub fn reset_health_for_store_init_retry(&self) {
+        match self {
+            Disk::Local(local_disk) => local_disk.reset_health_for_store_init_retry(),
+            Disk::Remote(remote_disk) => remote_disk.reset_health_for_store_init_retry(),
+        }
+    }
+
     /// Enable health monitoring on this disk.
     /// Called after startup format loading completes so that remote peers
     /// have time to come online before being marked as faulty.
@@ -559,6 +582,7 @@ pub struct CheckPartsResp {
 #[derive(Debug, Serialize, Deserialize, Default)]
 pub struct UpdateMetadataOpts {
     pub no_persistence: bool,
+    pub replace_user_metadata: bool,
 }
 
 pub struct DiskLocation {
@@ -914,9 +938,13 @@ mod tests {
     /// Test UpdateMetadataOpts structure
     #[test]
     fn test_update_metadata_opts() {
-        let opts = UpdateMetadataOpts { no_persistence: true };
+        let opts = UpdateMetadataOpts {
+            no_persistence: true,
+            ..Default::default()
+        };
 
         assert!(opts.no_persistence);
+        assert!(!opts.replace_user_metadata);
     }
 
     /// Test DiskOption structure
@@ -1105,4 +1133,40 @@ mod tests {
         // Clean up the test directory
         let _ = fs::remove_dir_all(&test_dir).await;
     }
+
+    #[tokio::test]
+    async fn reset_health_for_store_init_retry_delegates_to_disk_variants() {
+        let local_dir = tempfile::tempdir().unwrap();
+        let local_path = local_dir.path().to_str().expect("tempdir path should be utf8");
+        let mut local_endpoint = Endpoint::try_from(local_path).expect("local endpoint should parse");
+        local_endpoint.set_pool_index(0);
+        local_endpoint.set_set_index(0);
+        local_endpoint.set_disk_index(0);
+        let local_disk = LocalDisk::new(&local_endpoint, false).await.unwrap();
+        let local_disk = Disk::Local(Box::new(LocalDiskWrapper::new(Arc::new(local_disk), false)));
+
+        let mut remote_endpoint = Endpoint::try_from("http://remote-server:9000/data").expect("remote endpoint should parse");
+        remote_endpoint.set_pool_index(0);
+        remote_endpoint.set_set_index(0);
+        remote_endpoint.set_disk_index(1);
+        let remote_disk = RemoteDisk::new(
+            &remote_endpoint,
+            &DiskOption {
+                cleanup: false,
+                health_check: false,
+            },
+        )
+        .await
+        .unwrap();
+        let remote_disk = Disk::Remote(Box::new(remote_disk));
+
+        for disk in [&local_disk, &remote_disk] {
+            disk.force_runtime_state_for_test(RuntimeDriveHealthState::Offline);
+            assert_eq!(disk.runtime_state(), RuntimeDriveHealthState::Offline);
+
+            disk.reset_health_for_store_init_retry();
+
+            assert_eq!(disk.runtime_state(), RuntimeDriveHealthState::Online);
+        }
+    }
 }

crates/ecstore/src/erasure_coding/decode.rs

@@ -166,8 +166,19 @@ async fn write_data_blocks<W>(
 where
     W: tokio::io::AsyncWrite + Send + Sync + Unpin,
 {
-    if get_data_block_len(en_blocks, data_blocks) < length {
-        error!("write_data_blocks get_data_block_len < length");
+    if en_blocks.len() < data_blocks {
+        return Err(io::Error::new(ErrorKind::InvalidInput, "data block count exceeds available shards"));
+    }
+
+    if length == 0 {
+        return Ok(0);
+    }
+
+    let Some(required_len) = offset.checked_add(length) else {
+        return Err(io::Error::new(ErrorKind::InvalidInput, "offset + length overflows"));
+    };
+    if get_data_block_len(en_blocks, data_blocks) < required_len {
+        error!("write_data_blocks not enough data after offset");
         return Err(io::Error::new(ErrorKind::UnexpectedEof, "Not enough data blocks to write"));
     }
 
@@ -188,29 +199,22 @@ where
         let block_slice = &block[offset..];
         offset = 0;
 
-        if write_left < block_slice.len() {
-            writer.write_all(&block_slice[..write_left]).await.map_err(|e| {
-                error!("write_data_blocks write_all err: {}", e);
-                e
-            })?;
-
-            total_written += write_left;
-            break;
-        }
-
-        let n = block_slice.len();
-
-        writer.write_all(block_slice).await.map_err(|e| {
-            error!("write_data_blocks write_all2 err: {}", e);
+        let write_len = write_left.min(block_slice.len());
+        writer.write_all(&block_slice[..write_len]).await.map_err(|e| {
+            error!("write_data_blocks write_all err: {}", e);
             e
         })?;
 
-        write_left -= n;
+        total_written += write_len;
+        write_left -= write_len;
 
-        total_written += n;
+        if write_left == 0 {
+            return Ok(total_written);
+        }
     }
 
-    Ok(total_written)
+    error!("write_data_blocks loop exhausted with write_left>0");
+    Err(io::Error::new(ErrorKind::UnexpectedEof, "Not enough data blocks to write"))
 }
 
 impl Erasure {
@@ -323,6 +327,112 @@ mod tests {
     use rustfs_utils::HashAlgorithm;
     use std::io::Cursor;
 
+    #[tokio::test]
+    async fn test_write_data_blocks_writes_range_across_blocks() {
+        let blocks = vec![Some(vec![1, 2, 3, 4]), Some(vec![5, 6, 7]), Some(vec![8, 9])];
+        let mut out = Vec::new();
+
+        let written = write_data_blocks(&mut out, &blocks, 3, 2, 5).await.unwrap();
+
+        assert_eq!(written, 5);
+        assert_eq!(out, vec![3, 4, 5, 6, 7]);
+    }
+
+    #[tokio::test]
+    async fn test_write_data_blocks_rejects_short_data_after_offset() {
+        let blocks = vec![Some(vec![1, 2, 3, 4]), Some(vec![5, 6, 7])];
+        let mut out = Vec::new();
+
+        let err = write_data_blocks(&mut out, &blocks, 2, 3, 5).await.unwrap_err();
+
+        assert_eq!(err.kind(), ErrorKind::UnexpectedEof);
+        assert!(out.is_empty());
+    }
+
+    #[tokio::test]
+    async fn test_write_data_blocks_rejects_invalid_data_block_count() {
+        let blocks = vec![Some(vec![1, 2, 3, 4])];
+        let mut out = Vec::new();
+
+        let err = write_data_blocks(&mut out, &blocks, 2, 0, 1).await.unwrap_err();
+
+        assert_eq!(err.kind(), ErrorKind::InvalidInput);
+        assert!(out.is_empty());
+    }
+
+    /// Regression for upstream issue #2716: ranged GETs going through
+    /// `Erasure::decode` must return the requested byte range without
+    /// panicking or truncating, including when the range starts at a
+    /// non-zero offset and crosses EC block boundaries.
+    #[tokio::test]
+    async fn test_erasure_decode_ranged_read_returns_correct_bytes() {
+        const DATA_SHARDS: usize = 4;
+        const PARITY_SHARDS: usize = 2;
+        const BLOCK_SIZE: usize = 64;
+
+        // 200 bytes spans 3 full blocks + 1 partial block, exercising
+        // the start/middle/end branches in `Erasure::decode`.
+        let total_data: Vec<u8> = (0..200u32).map(|i| i as u8).collect();
+        let total_len = total_data.len();
+
+        let erasure = Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE);
+        let total_shards = DATA_SHARDS + PARITY_SHARDS;
+        let shard_size = erasure.shard_size();
+        let hash_algo = HashAlgorithm::HighwayHash256;
+
+        let mut shard_writers: Vec<BitrotWriter<Cursor<Vec<u8>>>> = (0..total_shards)
+            .map(|_| BitrotWriter::new(Cursor::new(Vec::new()), shard_size, hash_algo.clone()))
+            .collect();
+
+        let mut offset = 0;
+        while offset < total_len {
+            let end = (offset + BLOCK_SIZE).min(total_len);
+            let shards = erasure.encode_data(&total_data[offset..end]).unwrap();
+            for (i, shard) in shards.iter().enumerate() {
+                shard_writers[i].write(shard).await.unwrap();
+            }
+            offset = end;
+        }
+
+        let shard_bufs: Vec<Vec<u8>> = shard_writers.into_iter().map(|w| w.into_inner().into_inner()).collect();
+
+        // `Erasure::decode` does not seek the readers; the production caller
+        // (`create_bitrot_reader`) positions each reader at the shard byte
+        // offset corresponding to the request's start block. Mirror that here.
+        let hash_size = hash_algo.size();
+        let make_readers = |off: usize| -> Vec<Option<BitrotReader<Cursor<Vec<u8>>>>> {
+            let start_block = off / BLOCK_SIZE;
+            let cursor_pos = start_block * (shard_size + hash_size);
+            shard_bufs
+                .iter()
+                .map(|buf| {
+                    let mut cursor = Cursor::new(buf.clone());
+                    cursor.set_position(cursor_pos as u64);
+                    Some(BitrotReader::new(cursor, shard_size, hash_algo.clone(), false))
+                })
+                .collect()
+        };
+
+        // (offset, length, description)
+        let cases: &[(usize, usize, &str)] = &[
+            (0, total_len, "full read"),
+            (0, 50, "head from start, partial block"),
+            (10, 30, "small range within first block"),
+            (60, 80, "range crossing two block boundaries"),
+            (128, 50, "range starting at block boundary"),
+            (130, 10, "small range deep in middle"),
+            (192, 8, "tail covering last partial block"),
+        ];
+
+        for &(off, len, desc) in cases {
+            let mut output = Vec::new();
+            let (written, err) = erasure.decode(&mut output, make_readers(off), off, len, total_len).await;
+            assert!(err.is_none(), "{}: unexpected error: {:?}", desc, err);
+            assert_eq!(written, len, "{}: written != length", desc);
+            assert_eq!(output, total_data[off..off + len], "{}: bytes mismatch", desc);
+        }
+    }
+
     #[tokio::test]
     async fn test_parallel_reader_normal() {
         const BLOCK_SIZE: usize = 64;

crates/ecstore/src/error.rs

@@ -678,6 +678,18 @@ pub fn is_err_read_quorum(err: &Error) -> bool {
     matches!(err, &StorageError::ErasureReadQuorum)
 }
 
+pub fn classify_system_path_failure_reason(err: &Error) -> &'static str {
+    match err {
+        StorageError::ConfigNotFound => "config_not_found",
+        StorageError::ErasureReadQuorum | StorageError::InsufficientReadQuorum(_, _) => "read_quorum",
+        StorageError::Io(io_err) => match io_err.kind() {
+            std::io::ErrorKind::TimedOut => "timeout",
+            _ => "io",
+        },
+        _ => "other",
+    }
+}
+
 pub fn is_err_invalid_upload_id(err: &Error) -> bool {
     matches!(err, &StorageError::InvalidUploadID(_, _, _))
 }
@@ -1024,6 +1036,28 @@ mod tests {
         assert!(!is_err_not_initialized(&StorageError::DecommissionAlreadyRunning));
     }
 
+    #[test]
+    fn test_classify_system_path_failure_reason() {
+        assert_eq!(classify_system_path_failure_reason(&StorageError::ConfigNotFound), "config_not_found");
+        assert_eq!(classify_system_path_failure_reason(&StorageError::ErasureReadQuorum), "read_quorum");
+        assert_eq!(
+            classify_system_path_failure_reason(&StorageError::InsufficientReadQuorum(
+                "bucket".to_string(),
+                "object".to_string()
+            )),
+            "read_quorum"
+        );
+        assert_eq!(
+            classify_system_path_failure_reason(&StorageError::Io(IoError::new(ErrorKind::TimedOut, "probe"))),
+            "timeout"
+        );
+        assert_eq!(
+            classify_system_path_failure_reason(&StorageError::Io(IoError::new(ErrorKind::PermissionDenied, "probe"))),
+            "io"
+        );
+        assert_eq!(classify_system_path_failure_reason(&StorageError::DiskFull), "other");
+    }
+
     #[test]
     fn test_storage_error_from_disk_error() {
         // Test conversion from DiskError

crates/ecstore/src/notification_sys.rs

@@ -22,7 +22,7 @@ use crate::rpc::PeerRestClient;
 use crate::{endpoints::EndpointServerPools, new_object_layer_fn};
 use futures::future::join_all;
 use lazy_static::lazy_static;
-use rustfs_madmin::health::{Cpus, MemInfo, OsInfo, Partitions, ProcInfo, SysConfig, SysErrors, SysService};
+use rustfs_madmin::health::{Cpus, MemInfo, OsInfo, Partitions, ProcInfo, SysConfig, SysErrors, SysServices};
 use rustfs_madmin::metrics::RealtimeMetrics;
 use rustfs_madmin::net::NetInfo;
 use rustfs_madmin::{ItemState, ServerProperties};
@@ -485,31 +485,29 @@ impl NotificationSys {
         Ok(())
     }
 
-    pub async fn load_bucket_metadata(&self, bucket: &str) -> Vec<NotificationPeerErr> {
+    pub async fn load_bucket_metadata(&self, bucket: &str) -> Result<()> {
+        let operation = format!("load_bucket_metadata({bucket})");
+        let mut failures = Vec::new();
         let mut futures = Vec::with_capacity(self.peer_clients.len());
-        for client in self.peer_clients.iter() {
-            let b = bucket.to_string();
-            futures.push(async move {
-                if let Some(client) = client {
-                    match client.load_bucket_metadata(&b).await {
-                        Ok(_) => NotificationPeerErr {
-                            host: client.host.to_string(),
-                            err: None,
-                        },
-                        Err(e) => NotificationPeerErr {
-                            host: client.host.to_string(),
-                            err: Some(e),
-                        },
-                    }
-                } else {
-                    NotificationPeerErr {
-                        host: "".to_string(),
-                        err: Some(Error::other("peer is not reachable")),
-                    }
-                }
-            });
+        for (idx, client) in self.peer_clients.iter().enumerate() {
+            if let Some(client) = client {
+                let host = client.host.to_string();
+                let b = bucket.to_string();
+                futures.push(async move { client.load_bucket_metadata(&b).await.map_err(|err| (host, err)) });
+            } else {
+                failures.push(format!("peer[{idx}] {operation} failed: peer is not reachable"));
+            }
         }
-        join_all(futures).await
+
+        for result in join_all(futures).await {
+            if let Err((host, err)) = result {
+                let failure = format!("peer {host} {operation} failed: {err}");
+                error!("notification {operation} err {failure}");
+                failures.push(failure);
+            }
+        }
+
+        aggregate_notification_failures(&operation, failures)
     }
 
     pub async fn delete_bucket_metadata(&self, bucket: &str) -> Vec<NotificationPeerErr> {
@@ -622,14 +620,14 @@ impl NotificationSys {
         join_all(futures).await
     }
 
-    pub async fn get_sys_services(&self) -> Vec<SysService> {
+    pub async fn get_sys_services(&self) -> Vec<SysServices> {
         let mut futures = Vec::with_capacity(self.peer_clients.len());
         for client in self.peer_clients.iter().cloned() {
             futures.push(async move {
                 if let Some(client) = client {
                     client.get_se_linux_info().await.unwrap_or_default()
                 } else {
-                    SysService::default()
+                    SysServices::default()
                 }
             });
         }
@@ -904,4 +902,36 @@ mod tests {
         assert!(msg.contains("peer-1 failed"));
         assert!(msg.contains("local save failed"));
     }
+
+    #[tokio::test]
+    async fn load_bucket_metadata_reports_unreachable_peers() {
+        let sys = NotificationSys {
+            peer_clients: vec![None],
+            all_peer_clients: Vec::new(),
+        };
+
+        let err = sys
+            .load_bucket_metadata("bucket-a")
+            .await
+            .expect_err("unreachable peers should fail bucket metadata reload");
+
+        let msg = err.to_string();
+        assert!(msg.contains("load_bucket_metadata(bucket-a)"));
+        assert!(msg.contains("1 failure(s)"));
+        assert!(msg.contains("peer[0]"));
+    }
+
+    #[tokio::test]
+    async fn load_transition_tier_config_reports_unreachable_peers() {
+        let sys = NotificationSys {
+            peer_clients: vec![None],
+            all_peer_clients: Vec::new(),
+        };
+
+        let results = sys.load_transition_tier_config().await;
+        assert_eq!(results.len(), 1);
+        assert!(results[0].host.is_empty());
+        assert!(results[0].err.is_some());
+        assert!(results[0].err.as_ref().unwrap().to_string().contains("peer is not reachable"));
+    }
 }

crates/ecstore/src/pools.rs

@@ -142,6 +142,13 @@ fn ensure_decommission_not_rebalancing(rebalance_running: bool) -> Result<()> {
     Ok(())
 }
 
+fn decommission_meta_bucket_options() -> MakeBucketOptions {
+    MakeBucketOptions {
+        force_create: true,
+        ..Default::default()
+    }
+}
+
 fn is_decommission_active(complete: bool, failed: bool, canceled: bool) -> bool {
     !complete && !failed && !canceled
 }
@@ -2036,9 +2043,10 @@ impl ECStore {
             path_join(&[PathBuf::from(RUSTFS_META_BUCKET), PathBuf::from(BUCKET_META_PREFIX)]),
         ];
 
+        let meta_bucket_opts = decommission_meta_bucket_options();
         for bk in meta_buckets.iter() {
             if let Err(err) = self
-                .make_bucket(bk.to_string_lossy().to_string().as_str(), &MakeBucketOptions::default())
+                .make_bucket(bk.to_string_lossy().to_string().as_str(), &meta_bucket_opts)
                 .await
                 && !is_err_bucket_exists(&err)
             {
@@ -2737,12 +2745,13 @@ mod pools_tests {
     use super::{
         DecomBucketInfo, DecommissionTerminalState, PoolDecommissionInfo, PoolMeta, PoolStatus, bind_decommission_cancelers,
         cancel_decommission_canceler, classify_decommission_terminal_state, count_decommission_item,
-        decommission_cancel_signal_result, decommission_item_size, decommission_start_guard_state, dedup_indices,
-        ensure_decommission_cancel_allowed, ensure_decommission_listing_disks_available, ensure_decommission_not_rebalancing,
-        ensure_decommission_start_allowed, ensure_decommission_terminal_operation_supported,
-        ensure_valid_decommission_pool_index, get_by_index, has_active_decommission_canceler, is_decommission_active,
-        is_decommission_cancel_terminal, load_decommission_entry_versions, mark_decommission_bucket_done,
-        require_decommission_store, resolve_decommission_bucket_done_save_result, resolve_decommission_bucket_state,
+        decommission_cancel_signal_result, decommission_item_size, decommission_meta_bucket_options,
+        decommission_start_guard_state, dedup_indices, ensure_decommission_cancel_allowed,
+        ensure_decommission_listing_disks_available, ensure_decommission_not_rebalancing, ensure_decommission_start_allowed,
+        ensure_decommission_terminal_operation_supported, ensure_valid_decommission_pool_index, get_by_index,
+        has_active_decommission_canceler, is_decommission_active, is_decommission_cancel_terminal,
+        load_decommission_entry_versions, mark_decommission_bucket_done, require_decommission_store,
+        resolve_decommission_bucket_done_save_result, resolve_decommission_bucket_state,
         resolve_decommission_check_after_list_result, resolve_decommission_entry_cleanup_delete_result,
         resolve_decommission_entry_reload_result, resolve_decommission_listing_worker_result,
         resolve_decommission_optional_bucket_config_result, resolve_decommission_pool_meta_reload_result,
@@ -3376,6 +3385,13 @@ mod pools_tests {
         assert!(ensure_decommission_not_rebalancing(false).is_ok());
     }
 
+    #[test]
+    fn test_decommission_meta_bucket_options_are_idempotent() {
+        let opts = decommission_meta_bucket_options();
+
+        assert!(opts.force_create);
+    }
+
     #[test]
     fn test_is_decommission_active_true_only_when_not_terminal() {
         assert!(is_decommission_active(false, false, false));

crates/ecstore/src/rebalance.rs

@@ -780,6 +780,7 @@ impl ECStore {
 
         let cancel_tx = CancellationToken::new();
         let rx = cancel_tx.clone();
+        let mut meta_to_save = None;
 
         {
             let mut rebalance_meta = self.rebalance_meta.write().await;
@@ -792,11 +793,19 @@ impl ECStore {
                 info!("start_rebalance: already in progress, skip duplicate start");
                 return Ok(());
             }
+            if complete_rebalance_pools_at_goal(meta, OffsetDateTime::now_utc()) {
+                meta_to_save = Some(meta.clone());
+            }
             meta.cancel = Some(cancel_tx);
 
             drop(rebalance_meta);
         }
 
+        if let Some(meta) = meta_to_save {
+            let pool = clone_first_arc(self.pools.as_slice(), "start_rebalance: no pools available")?;
+            resolve_rebalance_meta_save_result(meta.save(pool).await, "start_rebalance complete pools at goal")?;
+        }
+
         let participants = if let Some(ref meta) = *self.rebalance_meta.read().await {
             resolve_rebalance_participants(meta.pool_stats.as_slice(), self.pools.len())
         } else {
@@ -1126,6 +1135,30 @@ fn should_pool_participate(init_free_space: u64, init_capacity: u64, percent_fre
     init_capacity > 0 && percent_free_ratio(init_free_space, init_capacity) < percent_free_goal
 }
 
+fn complete_rebalance_pools_at_goal(meta: &mut RebalanceMeta, now: OffsetDateTime) -> bool {
+    let mut changed = false;
+
+    for pool_stat in meta.pool_stats.iter_mut() {
+        if !is_rebalance_pool_started(pool_stat) {
+            continue;
+        }
+
+        if rebalance_goal_reached(
+            pool_stat.init_free_space,
+            pool_stat.init_capacity,
+            pool_stat.bytes,
+            meta.percent_free_goal,
+        ) {
+            pool_stat.info.status = RebalStatus::Completed;
+            pool_stat.info.end_time = Some(now);
+            pool_stat.info.last_error = None;
+            changed = true;
+        }
+    }
+
+    changed
+}
+
 fn resolve_rebalance_worker_result(
     set_idx: usize,
     worker_result: std::result::Result<Result<()>, tokio::task::JoinError>,
@@ -1838,12 +1871,12 @@ mod rebalance_unit_tests {
         GetObjectReader, HTTPRangeSpec, MigrationBackend, MigrationVersionResult, ObjectInfo, ObjectOptions, RebalSaveOpt,
         RebalStatus, RebalanceInfo, RebalanceMeta, RebalanceStats, RebalanceTerminalEvent, apply_rebalance_save_option,
         apply_rebalance_terminal_event, apply_stopped_at, classify_rebalance_terminal_event, clone_arc_by_index, clone_first_arc,
-        clone_rebalance_pool_stats, ensure_rebalance_listing_disks_available, ensure_rebalance_not_decommissioning,
-        ensure_valid_rebalance_pool_index, is_rebalance_stopped_terminal_event, load_rebalance_bucket_configs,
-        mark_rebalance_bucket_done, migrate_entry_version, next_rebal_bucket_from_stat, rebalance_delete_marker_opts,
-        rebalance_meta_load_no_data_error, rebalance_meta_load_unknown_format_error, rebalance_meta_load_unknown_version_error,
-        resolve_load_rebalance_stats_update_result, resolve_next_rebalance_bucket, resolve_rebalance_bucket_error,
-        resolve_rebalance_bucket_result, resolve_rebalance_entry_cleanup_delete_result,
+        clone_rebalance_pool_stats, complete_rebalance_pools_at_goal, ensure_rebalance_listing_disks_available,
+        ensure_rebalance_not_decommissioning, ensure_valid_rebalance_pool_index, is_rebalance_stopped_terminal_event,
+        load_rebalance_bucket_configs, mark_rebalance_bucket_done, migrate_entry_version, next_rebal_bucket_from_stat,
+        rebalance_delete_marker_opts, rebalance_meta_load_no_data_error, rebalance_meta_load_unknown_format_error,
+        rebalance_meta_load_unknown_version_error, resolve_load_rebalance_stats_update_result, resolve_next_rebalance_bucket,
+        resolve_rebalance_bucket_error, resolve_rebalance_bucket_result, resolve_rebalance_entry_cleanup_delete_result,
         resolve_rebalance_file_info_versions_result, resolve_rebalance_meta_load_result, resolve_rebalance_meta_save_result,
         resolve_rebalance_migrate_result_error, resolve_rebalance_optional_bucket_config_result, resolve_rebalance_participants,
         resolve_rebalance_save_task_result, resolve_rebalance_stats_update_result, resolve_rebalance_terminal_error,
@@ -3248,6 +3281,44 @@ mod rebalance_unit_tests {
         assert!(!should_pool_participate(300, 1_000, 0.3));
     }
 
+    #[test]
+    fn test_complete_rebalance_pools_at_goal_marks_started_participants_completed() {
+        let now = OffsetDateTime::from_unix_timestamp(1_000).unwrap();
+        let mut meta = RebalanceMeta {
+            percent_free_goal: 0.5,
+            pool_stats: vec![
+                RebalanceStats {
+                    participating: true,
+                    init_free_space: 400,
+                    init_capacity: 1_000,
+                    bytes: 50,
+                    info: RebalanceInfo {
+                        status: RebalStatus::Started,
+                        ..Default::default()
+                    },
+                    ..Default::default()
+                },
+                RebalanceStats {
+                    participating: true,
+                    init_free_space: 100,
+                    init_capacity: 1_000,
+                    bytes: 0,
+                    info: RebalanceInfo {
+                        status: RebalStatus::Started,
+                        ..Default::default()
+                    },
+                    ..Default::default()
+                },
+            ],
+            ..Default::default()
+        };
+
+        assert!(complete_rebalance_pools_at_goal(&mut meta, now));
+        assert_eq!(meta.pool_stats[0].info.status, RebalStatus::Completed);
+        assert_eq!(meta.pool_stats[0].info.end_time, Some(now));
+        assert_eq!(meta.pool_stats[1].info.status, RebalStatus::Started);
+    }
+
     #[test]
     fn test_should_skip_start_rebalance_only_when_running_and_cancel_attached() {
         assert!(should_skip_start_rebalance(true, true));