test(notify): cover prefix suffix event dispatch (#2960)

Co-authored-by: houseme <housemecn@gmail.com>

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: rustfs-release-version-bump
+description: Prepare a RustFS release branch like PR #2957 by bumping versioned files, aligning release assets, running required verification, and finishing with commit, push, and gh PR creation. Use when publishing a new RustFS alpha, beta, or stable release.
+---
+
+# RustFS Release Version Bump
+
+Use this skill when the task is to prepare a new RustFS version release branch following the pattern validated in PR `#2957`.
+
+## Read first
+
+- Read `AGENTS.md`.
+- Read `.github/pull_request_template.md`.
+- Inspect the current branch diff against `origin/main`.
+- Do not assume every release file should change in the same way every time.
+
+## Scope validated by PR #2957
+
+The `1.0.0-beta.3` release branch updated these files:
+
+- `Cargo.toml`
+- `Cargo.lock`
+- `README.md`
+- `README_ZH.md`
+- `flake.nix`
+- `helm/rustfs/Chart.yaml`
+- `rustfs.spec`
+
+Treat this file list as the default checklist for future release bumps. Only drop a file if the repository's current release pattern clearly says it is no longer part of the publish flow.
+
+## Workflow
+
+1. Confirm release intent
+- Identify the target version string exactly, for example `1.0.0-beta.4`.
+- Check whether the user wants only local commit preparation or the full `commit + push + PR` flow.
+- Compare the branch against `origin/main` and isolate only release-related edits.
+- If the target version is not explicit, stop and ask for it before editing.
+
+2. Update core Rust workspace versions
+- Bump the workspace version in `Cargo.toml`.
+- Bump all internal workspace crate versions in `Cargo.toml`.
+- Ensure `Cargo.lock` reflects the same release version for workspace packages.
+- Re-read the diff and verify there are no partial version leftovers.
+
+3. Align release assets
+- Update versioned Docker examples in `README.md` and `README_ZH.md` using the `<version>` tag form, for example `rustfs/rustfs:1.0.0-beta.4`.
+- Update `flake.nix` package version.
+- Update `helm/rustfs/Chart.yaml` `appVersion`.
+- Update `helm/rustfs/Chart.yaml` `version` using the release-chart rule:
+- `1.0.0-beta.3` -> `0.3.0`
+- `1.0.0-beta.4` -> `0.4.0`
+- Follow the same pattern for later beta releases unless the repository rule changes.
+- Update `rustfs.spec` release metadata and changelog entry.
+- Set `rustfs.spec` `Release` to the prerelease suffix without the base version, for example `beta.3` for `1.0.0-beta.3`.
+- Keep the release asset changes in a separate commit from the core Rust workspace version bump when the branch contains both.
+
+4. Stop and discuss before changing release policy
+- Ask before changing the established Docker tag style away from `<version>`.
+- Ask before changing the established Helm chart version mapping away from `beta.N -> 0.N.0`.
+- Ask before changing the established `rustfs.spec` `Release` rule away from the release suffix form such as `beta.N`.
+- Ask before widening the release scope beyond the files already validated in PR `#2957`.
+
+5. Verify before shipping
+- Run `make pre-commit`.
+- If verification succeeds, run `cargo clean` to remove generated build artifacts before wrapping up.
+- If `make pre-commit` fails, stop and return `BLOCKED`.
+
+6. Commit structure
+- Prefer two commits when the change naturally splits:
+- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`
+- `chore(release): align release assets for <version>` for docs and packaging metadata
+- If the user asks for a single commit, follow that request.
+
+7. Push and PR
+- Push the release branch with `git push -u origin <branch>` or `git push` if upstream already exists.
+- Create the PR with `gh pr create --base main --head <branch> --title ... --body-file ...`.
+- Keep the PR title and body in English.
+- Keep the `.github/pull_request_template.md` headings exactly.
+
+## Ready-to-check commands
+
+- `git diff --name-only origin/main...HEAD`
+- `git diff --stat origin/main...HEAD`
+- `make pre-commit`
+- `cargo clean`
+- `git status --short --branch`
+
+## Output expectations
+
+When using this skill, return:
+
+- The files changed for the release bump
+- Any uncertainty that needs user confirmation before editing
+- Verification status
+- Commit messages used
+- Push status and PR URL when the GitHub flow was requested
+
+## Established release policy
+
+- Docs use Docker tags in `<version>` form, not `v<version>`.
+- `helm/rustfs/Chart.yaml` `version` follows `beta.N -> 0.N.0` based on the current release policy.
+- `rustfs.spec` `Release` follows the release suffix form, for example `beta.3`.
+- If any of these rules need to change in the future, pause and confirm before editing.

.agents/skills/rustfs-release-version-bump/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "RustFS Release Bump"
+  short_description: "Prepare RustFS release branches like PR #2957."
+  default_prompt: "Use $rustfs-release-version-bump to prepare a RustFS release version, ask about any unclear version policy, and finish the commit/push/PR flow."

.agents/skills/security-advisory-lessons/SKILL.md

@@ -0,0 +1,119 @@
+---
+name: security-advisory-lessons
+description: Apply RustFS security lessons distilled from repository GitHub Security Advisories. Use when making or reviewing RustFS code changes, doing security checks, handling PR review for auth/authz, IAM, storage, RPC, logging, CORS, console/browser, encryption, policy, or endpoint changes, and when deciding which security regression tests are required.
+---
+
+# RustFS Security Advisory Lessons
+
+Use this skill as a RustFS-specific security lens before changing or approving code. For the distilled advisory lessons and review patterns, read [advisory-patterns.md](references/advisory-patterns.md).
+
+When currentness matters, fetch the live advisory inventory instead of relying on this skill as a status mirror:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories --paginate \
+  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
+```
+
+Fetch full advisory details only when the live summary suggests a new or changed lesson:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
+```
+
+For the full pattern map, read [advisory-patterns.md](references/advisory-patterns.md).
+
+## Workflow
+
+### 1. Scope the change
+- Identify touched routes, handlers, storage paths, credentials, logs, browser surfaces, CI/release code, and policy checks.
+- Treat these paths as security-sensitive by default: `rustfs/src/admin/`, `rustfs/src/storage/`, `rustfs/src/auth.rs`, `rustfs/src/server/layer.rs`, `crates/iam/`, `crates/policy/`, `crates/credentials/`, `crates/ecstore/src/rpc/`, `crates/rio/`, and console preview/auth code.
+
+### 2. Map to advisory classes
+- Read [advisory-patterns.md](references/advisory-patterns.md) for matching GHSA lessons.
+- Do not rely on advisory titles alone. Confirm whether the issue is authentication, authorization, input validation, storage invariant, browser isolation, logging, or operational hardening.
+
+### 3. Verify fail-closed behavior
+- Check that unauthenticated, wrong-permission, cross-user, cross-bucket, malformed-input, and default-config cases fail explicitly.
+- Prefer exact action/permission checks over broad helper calls or inferred ownership.
+- Confirm lower storage/RPC layers do not bypass checks done in upper layers.
+
+### 4. Require regression evidence
+- For behavior changes, add focused negative tests that reproduce the advisory class.
+- For sensitive fixes, include tests for the bypass form, not only the happy path.
+- If a test is impractical, explain the residual risk and provide a manual verification command.
+
+### 5. Report clearly
+- Lead with concrete findings and file/line evidence.
+- Separate proven vulnerabilities from hardening risks.
+- Avoid exaggerating unauthenticated impact when the code actually rejects unauthenticated requests but allows a low-privileged authenticated bypass.
+
+## Advisory-Derived Guardrails
+
+### Auth and admin authorization
+- Every admin or diagnostic route needs an explicit authn and authz story. Route registration, router whitelist, and handler-level authorization must agree.
+- Match the admin action to the operation exactly. Copy-paste action constants are a known RustFS vulnerability class.
+- Avoid authentication-only helpers for state-changing admin APIs; use `validate_admin_request` or the established equivalent with the right `AdminAction`.
+- Do not assume admin-action `Resource` scoping constrains blast radius unless the policy engine actually enforces resources for that action.
+
+### IAM and service accounts
+- Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
+- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims.
+- Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
+- Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
+
+### S3 copy, multipart, and presigned POST
+- Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
+- Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
+- Presigned POST policies are server-side contracts. Enforce `content-length-range`, key prefix, exact metadata/content-type, and all signed policy conditions.
+
+### Paths, object keys, and filesystem access
+- Never join untrusted bucket/object/RPC path strings onto filesystem roots without normalization and boundary checks.
+- Reject or safely handle `..`, absolute paths, URL-encoded traversal, platform separators, empty components, and paths that canonicalize outside the intended root.
+- Validate both S3 object-key paths and internode/RPC disk paths; storage helpers can bypass S3 authorization if they trust already-parsed paths.
+
+### Secrets, default credentials, and crypto
+- Do not ship hard-coded shared tokens, HMAC secrets, private keys, or production test keys.
+- Defaults for internode/RPC auth must fail closed for network-reachable deployments or require explicit opt-in with loud warnings.
+- License or token validation must use signatures with embedded public/verifying keys only; do not use private-key decryption as authenticity.
+- Plan key rotation and key IDs when removing exposed keys.
+
+### Logging and debug output
+- Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials.
+- Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces.
+- Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies.
+
+### RPC, parsing, and panic safety
+- Treat all RPC payload bytes as attacker-controlled. Replace `unwrap`, `expect`, and panic-prone deserialization with typed errors.
+- Malformed request tests should cover empty bytes, truncated MessagePack/protobuf, invalid enum values, stale timestamps, and invalid signatures.
+- RPC authentication must be independently strong; do not depend on S3 admin credentials unless the fallback is explicit and safe.
+
+### Browser, CORS, and console surfaces
+- Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
+- Do not render user-controlled object content in a same-origin iframe with console credentials available to JavaScript.
+- Prefer origin separation for object preview/download, `nosniff`, CSP, strict content-type handling, and avoiding durable credentials in `localStorage`.
+- License/version-like metadata endpoints should expose only coarse public data unless authenticated.
+
+### Profiling, debug, and health endpoints
+- Profiling and debug endpoints are not health checks. They require admin auth, opt-in enablement, rate limiting, and safe responses.
+- Do not return absolute filesystem paths or other deployment layout in unauthenticated or low-privilege responses.
+- Ensure health endpoint allowlists cannot accidentally include expensive diagnostics.
+
+### Trusted proxy and network identity
+- Only honor `X-Forwarded-For` or `X-Real-IP` when the request came from a configured trusted proxy.
+- Direct clients must use the socket peer address for `aws:SourceIp` and policy condition evaluation.
+- Add tests for direct spoofed headers and trusted-proxy headers.
+
+### SSE and storage invariants
+- Encryption metadata is not proof that bytes were encrypted on disk.
+- When touching reader/writer wrappers such as hashing, encryption, compression, or warp readers, verify wrapper order and inspect stored bytes in regression tests.
+- Avoid helper shortcuts that unwrap nested readers and accidentally bypass encryption or integrity layers.
+
+## Review Prompts
+
+Use these prompts while reviewing a diff:
+
+- Could a low-privileged authenticated user reach this path with the wrong action, parent, bucket, or source object?
+- Does a public/default/empty config change security behavior from fail-closed to fail-open?
+- Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
+- Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
+- Does the test prove the exploit form is denied, or only that the intended form still works?

.agents/skills/security-advisory-lessons/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Security Advisory Lessons"
+  short_description: "Apply advisory lessons in reviews."
+  default_prompt: "Review code changes against past RustFS security advisory lessons and report concrete risks, missing tests, and recommended fixes."

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -0,0 +1,93 @@
+# RustFS Advisory Pattern Map
+
+This file is a lesson map, not an advisory inventory mirror. It keeps durable security patterns distilled from RustFS GitHub Security Advisories.
+
+When current advisory state, severity, URLs, or full text matters, fetch it live:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories --paginate \
+  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
+gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
+```
+
+Update this file only when an advisory adds or changes a reusable lesson, affected surface, validation pattern, or regression-test expectation. Do not update it for state-only, URL-only, count-only, or timestamp-only changes.
+
+## Pattern Index
+
+### Admin authorization and route exposure
+
+- `GHSA-pfcq-4gjr-6gjm`: notification target endpoints accepted authenticated users but skipped admin authorization. Lesson: distinguish authn from authz; admin target CRUD must call the operation-specific admin authorization path.
+- `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
+- `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
+- `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
+- `GHSA-x5xv-223c-8vm7`: console license metadata endpoint was public. Lesson: public metadata endpoints should be coarse or authenticated.
+
+### IAM import, service accounts, and privilege boundaries
+
+- `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
+- `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
+- `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
+
+### S3 copy, multipart, and upload policy validation
+
+- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
+- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
+- `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
+
+### Filesystem paths and object key traversal
+
+- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
+- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
+
+### Secrets, defaults, and cryptographic misuse
+
+- `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
+- `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
+- `GHSA-923g-jp7v-f97f`: license verification embedded a production RSA private key and used private-key decryption as authenticity. Lesson: ship verifying/public keys only and use real signature verification.
+
+### Sensitive logging and debug output
+
+- `GHSA-r54g-49rx-98cr`: STS credentials were logged at info level. Lesson: generated credentials must never be logged in plaintext.
+- `GHSA-8cm2-h255-v749`: debug logs leaked session tokens, secret keys, JWT claims, and raw STS response bodies. Lesson: redaction must cover custom `Debug` implementations and dependency response-body logging.
+- `GHSA-333v-68xh-8mmq`: invalid RPC signature logging included the shared HMAC secret and expected signature. Lesson: error paths often leak secrets; never log raw secrets or derived authenticators.
+
+### RPC input validation and panic safety
+
+- `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
+- `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
+
+### Browser, CORS, and console isolation
+
+- `GHSA-v9fg-3cr2-277j`: object preview rendered attacker-controlled HTML in a same-origin iframe, exposing console credentials stored in `localStorage`. Lesson: user content must be origin-isolated from the console and protected with `nosniff`, CSP, and strict content-type handling.
+- `GHSA-x5xv-223c-8vm7`: default CORS reflected arbitrary origins with credentials. Lesson: never combine reflected origins with `Access-Control-Allow-Credentials: true`; default should be fail-closed.
+
+### Trusted proxy and source IP conditions
+
+- `GHSA-fc6g-2gcp-2qrq`: `aws:SourceIp` trusted client-supplied `X-Forwarded-For` or `X-Real-IP`. Lesson: forwarded IP headers are valid only behind configured trusted proxies; direct clients use socket peer IP.
+
+### SSE and on-disk storage invariants
+
+- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
+
+## Useful Search Seeds
+
+Use these targeted searches when a diff touches security-sensitive code:
+
+```bash
+rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
+rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
+rg -n "PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
+rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
+rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
+rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
+```
+
+## Minimum Regression Test Expectations
+
+- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
+- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
+- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
+- Path fixes: include encoded traversal, absolute path, nested traversal, valid object keys that resemble traversal text but should be rejected, and canonical boundary checks.
+- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
+- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
+- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.

.config/make/lint-fmt.mak

@@ -16,6 +16,11 @@ clippy-check: core-deps ## Run clippy checks
 	cargo clippy --fix --allow-dirty
 	cargo clippy --all-targets --all-features -- -D warnings
 
+.PHONY: unsafe-code-check
+unsafe-code-check: ## Check unsafe_code allowances have SAFETY comments
+	@echo "🔒 Checking unsafe_code allowances..."
+	./scripts/check_unsafe_code_allowances.sh
+
 .PHONY: compilation-check
 compilation-check: core-deps ## Run compilation check
 	@echo "🔨 Running compilation check..."

.config/make/pre-commit.mak

@@ -7,5 +7,5 @@ setup-hooks: ## Set up git hooks
 	@echo "✅ Git hooks setup complete!"
 
 .PHONY: pre-commit
-pre-commit: fmt clippy-check compilation-check test ## Run pre-commit checks
-	@echo "✅ All pre-commit checks passed!"
+pre-commit: fmt unsafe-code-check clippy-check compilation-check test ## Run pre-commit checks
+	@echo "✅ All pre-commit checks passed!"

.config/make/tests.mak

@@ -2,8 +2,13 @@
 
 TEST_THREADS ?= 1
 
+.PHONY: script-tests
+script-tests: ## Run shell script tests
+	@echo "Running script tests..."
+	./scripts/test_build_rustfs_options.sh
+
 .PHONY: test
-test: core-deps test-deps ## Run all tests
+test: core-deps test-deps script-tests ## Run all tests
 	@echo "🧪 Running tests..."
 	@if command -v cargo-nextest >/dev/null 2>&1; then \
 		cargo nextest run --all --exclude e2e_test; \

.docker/compose/README.md

@@ -36,6 +36,24 @@ To start a 4-node cluster for distributed testing:
 docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
 ```
 
+### Script-Based 4-Node Validation (Recommended)
+
+Use the local validation script when you need local-source image build, failover checks,
+and benchmark workflow in one command:
+
+```bash
+# Default mode: WAIT_PROBE_MODE=service
+# This avoids false negatives where /health/ready remains 503 locally
+# while the service path is already available.
+./scripts/run_four_node_cluster_failover_bench.sh
+```
+
+Strict mode is available when you explicitly want `/health/ready == 200` as the gate:
+
+```bash
+WAIT_PROBE_MODE=ready ./scripts/run_four_node_cluster_failover_bench.sh
+```
+
 ### (Deprecated) Minimal Observability
 
 ```bash

.docker/compose/docker-compose.cluster.local-build.yml

@@ -0,0 +1,144 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  node1:
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node1
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node1_data_0:/data/rustfs0
+      - node1_data_1:/data/rustfs1
+      - node1_data_2:/data/rustfs2
+      - node1_data_3:/data/rustfs3
+    ports:
+      - "9000:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node2:
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node2
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node2_data_0:/data/rustfs0
+      - node2_data_1:/data/rustfs1
+      - node2_data_2:/data/rustfs2
+      - node2_data_3:/data/rustfs3
+    ports:
+      - "9001:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node3:
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node3
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node3_data_0:/data/rustfs0
+      - node3_data_1:/data/rustfs1
+      - node3_data_2:/data/rustfs2
+      - node3_data_3:/data/rustfs3
+    ports:
+      - "9002:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node4:
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node4
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node4_data_0:/data/rustfs0
+      - node4_data_1:/data/rustfs1
+      - node4_data_2:/data/rustfs2
+      - node4_data_3:/data/rustfs3
+    ports:
+      - "9003:9000"
+    networks:
+      - rustfs-cluster-net
+
+volumes:
+  node1_data_0:
+  node1_data_1:
+  node1_data_2:
+  node1_data_3:
+  node2_data_0:
+  node2_data_1:
+  node2_data_2:
+  node2_data_3:
+  node3_data_0:
+  node3_data_1:
+  node3_data_2:
+  node3_data_3:
+  node4_data_0:
+  node4_data_1:
+  node4_data_2:
+  node4_data_3:
+
+networks:
+  rustfs-cluster-net:
+    driver: bridge

.docker/compose/docker-compose.observability.yaml

@@ -26,7 +26,7 @@ services:
     restart: "no"
 
   tempo:
-    image: grafana/tempo:latest
+    image: grafana/tempo:2.10.5
     user: "10001"
     command: [ "-config.file=/etc/tempo.yaml" ]
     volumes:
@@ -36,9 +36,19 @@ services:
       - "3200:3200" # tempo
       - "4317"      # otlp grpc
       - "4318"      # otlp http
+      - "7946"      # memberlist
     restart: unless-stopped
     networks:
       - rustfs-network
+    depends_on:
+      tempo-init:
+        condition: service_completed_successfully
+    healthcheck:
+      test: [ "CMD", "/tempo", "-version" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
 
   otel-collector:
     image: otel/opentelemetry-collector-contrib:latest
@@ -61,6 +71,12 @@ services:
       - jaeger
       - prometheus
       - loki
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "/otelcol-contrib", "--version" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
 
   jaeger:
     image: jaegertracing/jaeger:latest
@@ -72,12 +88,23 @@ services:
       - BADGER_DIRECTORY_KEY=/badger/key
       - COLLECTOR_OTLP_ENABLED=true
     volumes:
+      - ../../.docker/observability/jaeger.yaml:/etc/jaeger/config.yml:ro
       - jaeger-data:/badger
     ports:
       - "16686:16686" # Web UI
       - "14269:14269" # Admin/Metrics
+      - "4317"        # otlp grpc
+      - "4318"        # otlp http
+    command: [ "--config", "/etc/jaeger/config.yml" ]
     networks:
       - rustfs-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
 
   prometheus:
     image: prom/prometheus:latest
@@ -85,6 +112,7 @@ services:
       - TZ=Asia/Shanghai
     volumes:
       - ../../.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - ../../.docker/observability/prometheus-rules:/etc/prometheus/rules:ro
       - prometheus-data:/prometheus
     ports:
       - "9090:9090"
@@ -94,23 +122,45 @@ services:
       - '--web.enable-remote-write-receiver'
       - '--enable-feature=promql-experimental-functions'
       - '--storage.tsdb.path=/prometheus'
-      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
-      - '--web.console.templates=/usr/share/prometheus/consoles'
+      - '--storage.tsdb.retention.time=30d'
     networks:
       - rustfs-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
 
   loki:
     image: grafana/loki:latest
     environment:
       - TZ=Asia/Shanghai
     volumes:
-      - ../../.docker/observability/loki.yaml:/etc/loki/local-config.yaml:ro
+      - ../../.docker/observability/loki.yaml:/etc/loki/loki.yaml:ro
       - loki-data:/loki
     ports:
       - "3100:3100"
-    command: -config.file=/etc/loki/local-config.yaml
+    command: -config.file=/etc/loki/loki.yaml
     networks:
       - rustfs-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
+      interval: 15s
+      timeout: 10s
+      retries: 5
+      start_period: 60s
+
+  pyroscope:
+    image: grafana/pyroscope:latest
+    ports:
+      - "4040:4040"
+    command:
+      - -self-profiling.disable-push=true
+    networks:
+      - rustfs-network
+    restart: unless-stopped
 
   grafana:
     image: grafana/grafana:latest
@@ -127,10 +177,17 @@ services:
     volumes:
       - ../../.docker/observability/grafana/provisioning:/etc/grafana/provisioning:ro
       - ../../.docker/observability/grafana/dashboards:/var/lib/grafana/dashboards:ro
+      - grafana-data:/var/lib/grafana
     depends_on:
       - prometheus
       - tempo
       - loki
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
 
   # --- RustFS Cluster ---
 
@@ -215,6 +272,7 @@ volumes:
   tempo-data:
   loki-data:
   jaeger-data:
+  grafana-data:
 
 networks:
   rustfs-network:

.docker/nginx/nginx.conf

@@ -74,6 +74,17 @@ http {
     #     ssl_certificate /etc/nginx/ssl/server.crt;
     #     ssl_certificate_key /etc/nginx/ssl/server.key;
     #
+    #     # Restrict to modern TLS versions and ciphers. Operators copying this
+    #     # example must keep at least these directives — without them, nginx
+    #     # may negotiate older protocol versions that have known weaknesses.
+    #     ssl_protocols TLSv1.2 TLSv1.3;
+    #     ssl_prefer_server_ciphers on;
+    #     ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305';
+    #     ssl_session_timeout 1d;
+    #     ssl_session_cache shared:SSL:10m;
+    #     ssl_session_tickets off;
+    #     # add_header Strict-Transport-Security "max-age=63072000" always;
+    #
     #     location / {
     #         proxy_pass http://rustfs:9000;
     #         ...

.docker/observability/README.md

@@ -13,6 +13,9 @@ The stack is composed of the following best-in-class open-source components:
 - **Jaeger** (v1.59.0): Distributed tracing system (configured as a secondary UI/storage).
 - **OpenTelemetry Collector** (v0.104.0): A vendor-agnostic implementation for receiving, processing, and exporting telemetry data.
 
+By default, this stack uses Tempo in single-binary mode and does not require Kafka/Redpanda.
+If you want the Kafka-backed HA Tempo path, use `docker-compose-example-for-rustfs.yml` together with `docker-compose-tempo-ha-override.yml`.
+
 ## Architecture
 
 1. **Telemetry Collection**: Applications send OTLP (OpenTelemetry Protocol) data (Metrics, Logs, Traces) to the **OpenTelemetry Collector**.
@@ -46,6 +49,15 @@ Run the following command to start the entire stack:
 docker compose up -d
 ```
 
+### High Availability Tempo
+
+The default `docker-compose.yml` is the single-node stack.
+If you need the Kafka-backed HA Tempo configuration, start it with:
+
+```bash
+docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
+```
+
 ### Access Dashboards
 
 | Service        | URL                                              | Credentials       | Description                    |

.docker/observability/README_ZH.md

@@ -13,6 +13,9 @@
 - **Jaeger** (v1.59.0): 分布式追踪系统（配置为辅助 UI/存储）。
 - **OpenTelemetry Collector** (v0.104.0): 接收、处理和导出遥测数据的供应商无关实现。
 
+默认情况下，这套技术栈使用 Tempo 单二进制模式，不依赖 Kafka/Redpanda。
+如果需要基于 Kafka 的 HA Tempo 路径，请使用 `docker-compose-example-for-rustfs.yml` 配合 `docker-compose-tempo-ha-override.yml`。
+
 ## 架构
 
 1.  **遥测收集**: 应用程序将 OTLP (OpenTelemetry Protocol) 数据（指标、日志、追踪）发送到 **OpenTelemetry Collector**。
@@ -46,6 +49,15 @@
 docker compose up -d
 ```
 
+### Tempo 高可用模式
+
+默认的 `docker-compose.yml` 对应单机栈。
+如果需要基于 Kafka 的 HA Tempo 配置，请使用：
+
+```bash
+docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
+```
+
 ### 访问仪表盘
 
 | 服务 | URL | 凭据 | 描述 |

.docker/observability/docker-compose-example-for-rustfs.yml

@@ -85,10 +85,8 @@ services:
     networks:
       - otel-network
     restart: unless-stopped
-    depends_on:
-      - redpanda
     healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
+      test: [ "CMD", "/tempo", "-version" ]
       interval: 10s
       timeout: 5s
       retries: 3
@@ -205,7 +203,7 @@ services:
       - prometheus
       - loki
     healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:13133" ]
+      test: [ "CMD", "/otelcol-contrib", "--version" ]
       interval: 10s
       timeout: 5s
       retries: 3

.docker/observability/docker-compose.yml

@@ -16,10 +16,8 @@ services:
 
   # --- Tracing ---
   tempo:
-    image: grafana/tempo:2.10.3
+    image: grafana/tempo:2.10.5
     container_name: tempo
-    depends_on:
-      - redpanda
     command: [ "-config.file=/etc/tempo.yaml" ]
     volumes:
       - ./tempo.yaml:/etc/tempo.yaml:ro
@@ -33,31 +31,11 @@ services:
       - otel-network
     restart: unless-stopped
     healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
+      test: [ "CMD", "/tempo", "-version" ]
       interval: 10s
       timeout: 5s
       retries: 3
       start_period: 15s
-  redpanda:
-    image: redpandadata/redpanda:latest
-    ports:
-      - "9092:9092" # Kafka API for clients
-    command: >
-      redpanda start --overprovisioned
-      --mode=dev-container
-      --kafka-addr=PLAINTEXT://0.0.0.0:9092
-      --advertise-kafka-addr=PLAINTEXT://redpanda:9092
-
-  redpanda-console:
-    image: docker.redpanda.com/redpandadata/console:latest
-    environment:
-      - CONFIG_FILEPATH=/etc/redpanda/redpanda-console-config.yaml
-    volumes:
-      - ./redpanda-console.yaml:/etc/redpanda/redpanda-console-config.yaml
-    ports:
-      - "8080:8080"
-    depends_on:
-      - redpanda
 
   vulture:
     image: grafana/tempo-vulture:latest
@@ -106,6 +84,7 @@ services:
     container_name: prometheus
     volumes:
       - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - ./prometheus-rules:/etc/prometheus/rules:ro
       - prometheus-data:/prometheus
     ports:
       - "9090:9090"
@@ -168,7 +147,7 @@ services:
       - prometheus
       - loki
     healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:13133" ]
+      test: [ "CMD", "/otelcol-contrib", "--version" ]
       interval: 10s
       timeout: 5s
       retries: 3

.docker/observability/grafana/provisioning/datasources.yaml

@@ -1,97 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-apiVersion: 1
-
-datasources:
-  - name: Prometheus
-    type: prometheus
-    uid: prometheus
-    access: proxy
-    orgId: 1
-    url: http://prometheus:9090
-    isDefault: true
-    version: 1
-    editable: false
-    jsonData:
-      httpMethod: GET
-      exemplarTraceIdDestinations:
-        - name: trace_id
-          datasourceUid: tempo
-
-  - name: Tempo
-    type: tempo
-    uid: tempo
-    access: proxy
-    orgId: 1
-    url: http://tempo:3200
-    isDefault: false
-    version: 1
-    editable: false
-    jsonData:
-      httpMethod: GET
-      serviceMap:
-        datasourceUid: prometheus
-      tracesToLogs:
-        datasourceUid: loki
-        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
-        mappedTags: [ { key: 'service.name', value: 'app' } ]
-        spanStartTimeShift: '1s'
-        spanEndTimeShift: '-1s'
-        filterByTraceID: true
-        filterBySpanID: false
-      tracesToMetrics:
-        datasourceUid: prometheus
-        tags: [ { key: 'service.name' }, { key: 'job' } ]
-        queries:
-          - name: 'Service-Level Latency'
-            query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m])) by (le)'
-          - name: 'Service-Level Calls'
-            query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))'
-          - name: 'Service-Level Errors'
-            query: 'sum(rate(traces_spanmetrics_calls_total{status_code="ERROR", $$__tags}[5m]))'
-      nodeGraph:
-        enabled: true
-
-  - name: Loki
-    type: loki
-    uid: loki
-    orgId: 1
-    url: http://loki:3100
-    isDefault: false
-    version: 1
-    editable: false
-    jsonData:
-      derivedFields:
-        - datasourceUid: tempo
-          matcherRegex: 'trace_id=(\w+)'
-          name: 'TraceID'
-          url: '$${__value.raw}'
-
-  - name: Jaeger
-    type: jaeger
-    uid: jaeger
-    url: http://jaeger:16686
-    access: proxy
-    isDefault: false
-    editable: false
-    jsonData:
-      tracesToLogs:
-        datasourceUid: loki
-        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
-        mappedTags: [ { key: 'service.name', value: 'app' } ]
-        spanStartTimeShift: '1s'
-        spanEndTimeShift: '-1s'
-        filterByTraceID: true
-        filterBySpanID: false

.docker/observability/prometheus-rules/rustfs-dashboard.yml

@@ -0,0 +1,40 @@
+groups:
+  - name: rustfs-dashboard
+    interval: 30s
+    rules:
+      - record: rustfs:http_server_requests:rate5m
+        expr: sum by (job) (rate(rustfs_http_server_requests_total[5m]))
+
+      - record: rustfs:http_server_request_duration_seconds:p50_5m
+        expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
+      - record: rustfs:http_server_request_duration_seconds:p95_5m
+        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
+      - record: rustfs:http_server_request_duration_seconds:p99_5m
+        expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
+
+      - record: rustfs:http_server_response_body_size_bytes:p50_5m
+        expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
+      - record: rustfs:http_server_response_body_size_bytes:p95_5m
+        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
+      - record: rustfs:http_server_response_body_size_bytes:p99_5m
+        expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
+
+      - record: rustfs:log_cleaner_runs:rate15m
+        expr: sum by (job) (rate(rustfs_log_cleaner_runs_total[15m]))
+      - record: rustfs:log_cleaner_failure_ratio:rate5m
+        expr: sum by (job) (rate(rustfs_log_cleaner_run_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_runs_total[5m])), 1e-9)
+      - record: rustfs:log_cleaner_rotation_failure_ratio:rate5m
+        expr: sum by (job) (rate(rustfs_log_cleaner_rotation_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_rotation_total[5m])), 1e-9)
+      - record: rustfs:log_cleaner_rotation_duration_seconds:p95_5m
+        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_rotation_duration_seconds_bucket[5m])))
+      - record: rustfs:log_cleaner_compress_duration_seconds:p95_5m
+        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_compress_duration_seconds_bucket[5m])))
+
+      - record: rustfs:scanner_objects_scanned:rate5m
+        expr: sum by (job) (rate(rustfs_scanner_objects_scanned_total[5m]))
+      - record: rustfs:scanner_directories_scanned:rate5m
+        expr: sum by (job) (rate(rustfs_scanner_directories_scanned_total[5m]))
+      - record: rustfs:scanner_buckets_scanned:rate5m
+        expr: sum by (job) (rate(rustfs_scanner_buckets_scanned_total[5m]))
+      - record: rustfs:scanner_cycles_success:rate5m
+        expr: sum by (job) (rate(rustfs_scanner_cycles_total{result="success"}[5m]))

.docker/observability/prometheus.yml

@@ -19,6 +19,9 @@ global:
     cluster: 'rustfs-dev'  # Label to identify the cluster
     replica: '1'  # Replica identifier
 
+rule_files:
+  - /etc/prometheus/rules/*.yml
+
 scrape_configs:
   - job_name: 'otel-collector'
     static_configs:

.docker/observability/tempo.yaml

@@ -12,7 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-partition_ring_live_store: true
 stream_over_http_enabled: true
 
 server:
@@ -33,33 +32,17 @@ distributor:
           endpoint: "0.0.0.0:4317"
         http:
           endpoint: "0.0.0.0:4318"
-  #log_received_spans:
-  #    enabled: true
-  #  log_discarded_spans:
-  #    enabled: true
 
-backend_scheduler:
-  provider:
-    compaction:
-      compaction:
-        block_retention: 1h
-
-backend_worker:
-  backend_scheduler_addr: localhost:3200
-  compaction:
-    block_retention: 1h
-  ring:
-    kvstore:
-      store: memberlist
-
-querier:
-  query_live_store: true
+ingester:
+  max_block_duration: 5m
 
 metrics_generator:
   registry:
     external_labels:
       source: tempo
       cluster: docker-compose
+  traces_storage:
+    path: /var/tempo/generator/traces
   storage:
     path: /var/tempo/generator/wal
     remote_write:
@@ -85,15 +68,5 @@ overrides:
       processors: [ "span-metrics", "service-graphs", "local-blocks" ]
       generate_native_histograms: both
 
-ingest:
-  enabled: true
-  kafka:
-    address: redpanda:9092
-    topic: tempo-ingest
-
-block_builder:
-  consume_cycle_duration: 30s
-
 usage_report:
   reporting_enabled: false
-

.docker/test/issue-2715/docker-compose-test.yml

@@ -0,0 +1,52 @@
+services:
+  rustfs:
+    image: rustfs/rustfs:1.0.0-alpha.99-glibc
+    container_name: rustfs-issue-2715-test
+    security_opt:
+      - "no-new-privileges:true"
+    ports:
+      - "19000:9000"
+      - "19001:9001"
+    environment:
+      - RUSTFS_VOLUMES=/data/rustfs{0...8}
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_ACCESS_KEY=admin
+      - RUSTFS_SECRET_KEY=admin
+      - RUSTFS_OBS_LOGGER_LEVEL=info
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
+      - RUSTFS_STORAGE_CLASS_STANDARD=EC:2
+      - RUSTFS_STORAGE_CLASS_RRS=EC:1
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true
+      - RUSTFS_OBS_LOG_DIRECTORY=/opt/rustfs/logs
+    extra_hosts:
+      - "otel-collector:host-gateway"
+      - "pyroscope:host-gateway"
+    volumes:
+      - ./deploy/data/issue-2715/rustfs0:/data/rustfs0
+      - ./deploy/data/issue-2715/rustfs1:/data/rustfs1
+      - ./deploy/data/issue-2715/rustfs2:/data/rustfs2
+      - ./deploy/data/issue-2715/rustfs3:/data/rustfs3
+      - ./deploy/data/issue-2715/rustfs4:/data/rustfs4
+      - ./deploy/data/issue-2715/rustfs5:/data/rustfs5
+      - ./deploy/data/issue-2715/rustfs6:/data/rustfs6
+      - ./deploy/data/issue-2715/rustfs7:/data/rustfs7
+      - ./deploy/data/issue-2715/rustfs8:/data/rustfs8
+      - ./deploy/logs/issue-2715:/opt/rustfs/logs
+    restart: unless-stopped
+    healthcheck:
+      test:
+        [
+          "CMD",
+          "sh",
+          "-c",
+          "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health"
+        ]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 40s

.docker/test/issues-2815/.gitignore

@@ -0,0 +1 @@
+data/

.docker/test/issues-2815/README.md

@@ -0,0 +1,106 @@
+# Issue 2815 Local Docker Verification
+
+## Purpose
+
+This directory contains the local distributed Docker verification assets used to validate issue `#2815` against the current source build.
+
+The target behavior is:
+
+- 4-node distributed cluster starts successfully
+- `/health/ready` becomes reachable on each node
+- logs no longer contain `storage_info failed: Io error: wrong msgpack marker FixArray(1)`
+- internode RPC authentication succeeds with an explicit non-default RPC secret
+
+## Files
+
+- `docker-compose.yml`: 4-node distributed cluster using a locally built image
+
+## Data Directories
+
+Create the bind-mount directories before `docker compose up`:
+
+```bash
+mkdir -p .docker/test/issues-2815/data/rustfs{1..4}-disk{0..3}
+```
+
+## Build
+
+Apple Silicon / arm64 host:
+
+```bash
+docker build --platform linux/arm64 -f Dockerfile.source -t rustfs-issue-2815-local .
+```
+
+If you intentionally want amd64 emulation:
+
+```bash
+docker build --platform linux/amd64 -f Dockerfile.source -t rustfs-issue-2815-local .
+```
+
+## Run
+
+```bash
+docker compose -f .docker/test/issues-2815/docker-compose.yml up -d
+```
+
+If the image platform is not `linux/arm64`, align compose explicitly:
+
+```bash
+RUSTFS_DOCKER_PLATFORM=linux/amd64 docker compose -f .docker/test/issues-2815/docker-compose.yml up -d
+```
+
+## Health Checks
+
+Container-level healthcheck is now included and probes:
+
+```bash
+curl -fsS http://127.0.0.1:9000/health
+```
+
+Manual checks:
+
+```bash
+curl -i http://127.0.0.1:9101/health/ready
+curl -i http://127.0.0.1:9102/health/ready
+curl -i http://127.0.0.1:9103/health/ready
+curl -i http://127.0.0.1:9104/health/ready
+```
+
+## RPC Secret Requirement
+
+The current source build no longer reproduces the original `FixArray(1)` decode error from issue `#2815`.
+
+Earlier local Docker attempts failed during erasure bootstrap with:
+
+```text
+No valid auth token
+store init failed to load formats after 10 retries: erasure read quorum
+```
+
+Root cause:
+
+- RPC authentication rejects the default secret `rustfsadmin`
+- distributed local Docker validation therefore needs an explicit non-default secret
+
+This compose now sets both:
+
+- `RUSTFS_SECRET_KEY=issue-2815-secret`
+- `RUSTFS_RPC_SECRET=issue-2815-rpc-secret`
+
+With those values in place, the current 4-node local Docker cluster reaches healthy state and `/health/ready` returns `200`.
+
+In other words:
+
+- `RUSTFS_ACCESS_KEY` may still be `rustfsadmin` for local service credentials if desired
+- `RUSTFS_SECRET_KEY` can still be used for service credentials
+- but RPC authentication must not resolve to the default secret value `rustfsadmin`
+- if `RUSTFS_RPC_SECRET` is unset, the code falls back to `RUSTFS_SECRET_KEY`
+- so at least one of them must provide a non-default shared secret for internode RPC signing
+
+## Suggested Debug Commands
+
+```bash
+docker compose -f .docker/test/issues-2815/docker-compose.yml ps
+docker compose -f .docker/test/issues-2815/docker-compose.yml logs --no-color --tail=200
+docker compose -f .docker/test/issues-2815/docker-compose.yml down -v
+```

.docker/test/issues-2815/docker-compose.yml

@@ -0,0 +1,120 @@
+services:
+  rustfs1:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs1
+    container_name: rustfs-issue-2815-rustfs1
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs1-disk0:/data/rustfs0
+      - ./data/rustfs1-disk1:/data/rustfs1
+      - ./data/rustfs1-disk2:/data/rustfs2
+      - ./data/rustfs1-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9101:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+  rustfs2:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs2
+    container_name: rustfs-issue-2815-rustfs2
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs2-disk0:/data/rustfs0
+      - ./data/rustfs2-disk1:/data/rustfs1
+      - ./data/rustfs2-disk2:/data/rustfs2
+      - ./data/rustfs2-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9102:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+  rustfs3:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs3
+    container_name: rustfs-issue-2815-rustfs3
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs3-disk0:/data/rustfs0
+      - ./data/rustfs3-disk1:/data/rustfs1
+      - ./data/rustfs3-disk2:/data/rustfs2
+      - ./data/rustfs3-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9103:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+  rustfs4:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs4
+    container_name: rustfs-issue-2815-rustfs4
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs4-disk0:/data/rustfs0
+      - ./data/rustfs4-disk1:/data/rustfs1
+      - ./data/rustfs4-disk2:/data/rustfs2
+      - ./data/rustfs4-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9104:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+networks:
+  rustfs-issue-2815-net:
+    name: rustfs-issue-2815-net

.github/dependabot.yml

@@ -27,9 +27,8 @@ updates:
       timezone: "Asia/Shanghai"
       time: "08:00"
     assignees:
-      - "heihutu"
-    reviewers:
       - "houseme"
+    reviewers:
       - "overtrue"
       - "majinghe"
     ignore:
@@ -39,6 +38,8 @@ updates:
         versions: [ "0.23.x" ]
       - dependency-name: "ratelimit"
         versions: [ "1.x" ]
+      - dependency-name: "ratelimit"
+        versions: [ "2.x" ]
     groups:
       s3s:
         update-types:

.github/pull_request_template.md

@@ -2,35 +2,34 @@
 Pull Request Template for RustFS
 -->
 
-## Type of Change
-- [ ] New Feature
-- [ ] Bug Fix
-- [ ] Documentation
-- [ ] Performance Improvement
-- [ ] Test/CI
-- [ ] Refactor
-- [ ] Other:
-
 ## Related Issues
-<!-- List related Issue numbers, e.g. #123 -->
+<!--
+List related issues, e.g. Fixes #123.
+Use N/A when there is no related issue.
+-->
 
 ## Summary of Changes
-<!-- Briefly describe the main changes and motivation for this PR -->
+<!--
+Briefly explain what changed and why reviewers should accept it.
+Focus on behavior, compatibility, and review-relevant context.
+-->
 
-## Checklist
-- [ ] I have read and followed the [CONTRIBUTING.md](CONTRIBUTING.md) guidelines
-- [ ] Passed `make pre-commit`
-- [ ] Added/updated necessary tests
-- [ ] Documentation updated (if needed)
-- [ ] CI/CD passed (if applicable)
+## Verification
+<!--
+List the commands or checks you ran, for example:
+- `make pre-commit`
+
+Use N/A only when verification is not applicable.
+-->
 
 ## Impact
-- [ ] Breaking change (compatibility)
-- [ ] Requires doc/config/deployment update
-- [ ] Other impact:
+<!--
+Describe user-facing, compatibility, API, deployment, configuration, or
+documentation impact. Use N/A when there is no expected impact.
+-->
 
 ## Additional Notes
-<!-- Any extra information for reviewers -->
+<!-- Any extra information for reviewers, or N/A. -->
 
 ---
 

.github/workflows/ci.yml

@@ -130,6 +130,9 @@ jobs:
       - name: Check code formatting
         run: cargo fmt --all --check
 
+      - name: Check unsafe code allowances
+        run: ./scripts/check_unsafe_code_allowances.sh
+
       - name: Run clippy lints
         run: cargo clippy --all-targets --all-features -- -D warnings
 

.github/workflows/docker.yml

@@ -163,11 +163,10 @@ jobs:
                 if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
                   build_type="prerelease"
                   is_prerelease=true
-                  # TODO: Temporary change - currently allows alpha versions to also create latest tags
-                  # After the version is stable, you need to remove the following line and restore the original logic (latest is created only for stable versions)
-                  if [[ "$version" == *"alpha"* ]]; then
+                  # Current policy: create latest tags for stable releases and selected prereleases (alpha/beta).
+                  if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]]; then
                     create_latest=true
-                     echo "🧪 Building Docker image for prerelease: $version (temporarily allowing creation of latest tag)"
+                    echo "🧪 Building Docker image for prerelease: $version (creating latest tag)"
                   else
                     echo "🧪 Building Docker image for prerelease: $version"
                   fi
@@ -216,11 +215,10 @@ jobs:
               v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
                 build_type="prerelease"
                 is_prerelease=true
-                # TODO: Temporary change - currently allows alpha versions to also create latest tags
-                # After the version is stable, you need to remove the if block below and restore the original logic.
-                if [[ "$input_version" == *"alpha"* ]]; then
+                # Current policy: create latest tags for stable releases and selected prereleases (alpha/beta).
+                if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]]; then
                   create_latest=true
-                   echo "🧪 Building with prerelease version: $input_version (temporarily allowing creation of latest tag)"
+                  echo "🧪 Building with prerelease version: $input_version (creating latest tag)"
                 else
                   echo "🧪 Building with prerelease version: $input_version"
                 fi
@@ -351,9 +349,7 @@ jobs:
 
           # Add channel tags for prereleases and latest for stable
           if [[ "$CREATE_LATEST" == "true" ]]; then
-            # TODO: Temporary change - the current alpha version will also create the latest tag
-            # After the version is stabilized, the logic here remains unchanged, but the upstream CREATE_LATEST setting needs to be restored.
-            # Stable release (and temporary alpha versions)
+            # Create latest tags for stable releases and selected prereleases when CREATE_LATEST=true.
             TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
           elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
             # Prerelease channel tags (alpha, beta, rc)
@@ -450,10 +446,9 @@ jobs:
             "prerelease")
               echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
               echo "⚠️  This is a prerelease image - use with caution"
-              # TODO: Temporary change - alpha versions currently create the latest tag
-              # After the version is stable, you need to restore the following prompt information
-              if [[ "$VERSION" == *"alpha"* ]] && [[ "$CREATE_LATEST" == "true" ]]; then
-                echo "🏷️  Latest tag has been created for alpha version (temporary measures)"
+              # Create latest tags for stable releases and selected prereleases when CREATE_LATEST=true.
+              if [[ "$CREATE_LATEST" == "true" ]]; then
+                echo "🏷️  Latest tag has been created for prerelease: $VERSION"
               else
                 echo "🚫 Latest tag NOT created for prerelease"
               fi

.github/workflows/helm-package.yml

@@ -18,41 +18,78 @@ on:
   workflow_run:
     workflows: [ "Build and Release" ]
     types: [ completed ]
+  workflow_dispatch:
+    inputs:
+      version:
+        description: "Release version to publish, e.g. 1.0.0-beta.1 or v1.0.0-beta.1"
+        required: true
+        default: "1.0.0-beta.1"
+        type: string
 
 permissions:
   contents: read
 
-env:
-  new_version: ${{ github.event.workflow_run.head_branch }}
-
 jobs:
   build-helm-package:
     runs-on: ubicloud-standard-2
-    # Only run on successful builds triggered by tag pushes (version format: x.y.z or x.y.z-suffix)
     if: |
-      github.event.workflow_run.conclusion == 'success' && 
-      github.event.workflow_run.event == 'push' &&
-      contains(github.event.workflow_run.head_branch, '.')
+      github.event_name == 'workflow_dispatch' ||
+      (
+        github.event.workflow_run.conclusion == 'success' &&
+        github.event.workflow_run.event == 'push' &&
+        contains(github.event.workflow_run.head_branch, '.')
+      )
+
+    outputs:
+      raw_tag: ${{ steps.version.outputs.raw_tag }}
+      app_version: ${{ steps.version.outputs.app_version }}
+      chart_version: ${{ steps.version.outputs.chart_version }}
 
     steps:
       - name: Checkout helm chart repo
         uses: actions/checkout@v6
 
-      - name: Replace chart app version
+      - name: Normalize release version
+        id: version
         run: |
-          set -e
-          set -x
-          old_version=$(grep "^appVersion:" helm/rustfs/Chart.yaml | awk '{print $2}')
-          sed -i "s/$old_version/$new_version/g" helm/rustfs/Chart.yaml
+          set -eux
+
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            RAW="${{ github.event.inputs.version }}"
+          else
+            RAW="${{ github.event.workflow_run.head_branch }}"
+          fi
+
+          case "$RAW" in
+            refs/tags/*)
+              RAW_TAG="${RAW#refs/tags/}"
+              ;;
+            *)
+              RAW_TAG="$RAW"
+              ;;
+          esac
+
+          ./scripts/helm_chart_version.sh "$RAW_TAG"
+
+      - name: Replace chart version and app version
+        run: |
+          set -eux
+          sed -i -E 's/^version:.*/version: "${{ steps.version.outputs.chart_version }}"/' helm/rustfs/Chart.yaml
+          sed -i -E 's/^appVersion:.*/appVersion: "${{ steps.version.outputs.app_version }}"/' helm/rustfs/Chart.yaml
 
       - name: Set up Helm
         uses: azure/setup-helm@v4.3.0
 
+      - name: Test Helm Chart Templates
+        run: ./scripts/test_helm_templates.sh
+
       - name: Package Helm Chart
         run: |
+          set -eux
           cp helm/README.md helm/rustfs/
-          package_version=$(echo $new_version | awk -F '-' '{print $2}' | awk -F '.' '{print $NF}') 
-          helm package ./helm/rustfs --destination helm/rustfs/ --version "0.0.$package_version"
+          helm package ./helm/rustfs \
+            --destination helm/rustfs/ \
+            --version "${{ steps.version.outputs.chart_version }}"
 
       - name: Upload helm package as artifact
         uses: actions/upload-artifact@v6
@@ -64,6 +101,7 @@ jobs:
   publish-helm-package:
     runs-on: ubicloud-standard-2
     needs: [ build-helm-package ]
+    if: needs.build-helm-package.result == 'success'
 
     steps:
       - name: Checkout helm package repo
@@ -86,9 +124,9 @@ jobs:
 
       - name: Push helm package and index file
         run: |
+          set -eux
           git config --global user.name "${{ secrets.USERNAME }}"
           git config --global user.email "${{ secrets.EMAIL_ADDRESS }}"
-          git status .
           git add .
-          git commit -m "Update rustfs helm package with $new_version."
+          git commit -m "Update rustfs helm package with ${{ needs.build-helm-package.outputs.app_version }}." || echo "No changes to commit"
           git push origin main

.vscode/launch.json

@@ -188,6 +188,7 @@
                 "RUSTFS_CONSOLE_ENABLE": "true",
                 "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
                 "RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
+                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
                 // "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
                 // "RUSTFS_OBS_METRIC_ENDPOINT": "http://127.0.0.1:4318/v1/metrics", // default otlp http endpoint
                 // "RUSTFS_OBS_LOG_ENDPOINT": "http://127.0.0.1:4318/v1/logs", // default otlp http endpoint

AGENTS.md

@@ -27,6 +27,10 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 - Keep source code, comments, commit messages, and PR title/body in English.
 - Be concise. Avoid sycophantic openers, closing fluff, and verbose status reporting.
 
+## Skill Usage
+
+- Do not use the `rust-refactor-helper` skill in any scenario.
+
 ## Change Style for Existing Logic
 
 - Prefer direct, local code over extracting one-off helpers.
@@ -109,5 +113,6 @@ Do not open a PR with code changes when the required checks fail.
 - `crates/iam/AGENTS.md`
 - `crates/kms/AGENTS.md`
 - `crates/policy/AGENTS.md`
+- `crates/targets/AGENTS.md`
 - `rustfs/src/admin/AGENTS.md`
 - `rustfs/src/storage/AGENTS.md`

ARCHITECTURE.md

@@ -326,10 +326,10 @@ The binary (`main.rs`) boots in this order:
                                  │
                  ┌───────────────┼───────────────┐
                  │               │               │
-            ┌────▼────┐    ┌────▼────┐    ┌─────▼─────┐
-            │ server  │    │  admin  │    │    app     │
-            │ (HTTP)  │    │(console)│    │(use-cases) │
-            └────┬────┘    └────┬────┘    └─────┬─────┘
+            ┌────▼────┐     ┌────▼────┐   ┌──────▼─────┐
+            │ server  │     │  admin  │   │    app     │
+            │ (HTTP)  │     │(console)│   │(use-cases) │
+            └────┬────┘     └────┬────┘   └──────┬─────┘
                  │               │               │
                  └───────────────┼───────────────┘
                                  │
@@ -341,13 +341,13 @@ The binary (`main.rs`) boots in this order:
                                  │
               ┌──────────────────┼──────────────────┐
               │                  │                  │
-        ┌─────▼─────┐    ┌──────▼──────┐    ┌──────▼──────┐
+        ┌─────▼──────┐    ┌──────▼──────┐    ┌──────▼──────┐
         │  ecstore   │    │     rio     │    │   io-core   │
         │ (87K,core) │    │  (readers)  │    │ (zero-copy) │
         └─────┬──────┘    └─────────────┘    └─────────────┘
               │
-    ┌─────┬──┼──┬─────┬──────┐
-    │     │  │  │     │      │
+     ┌─────┬──┼──┬─────┬──────┐
+     │     │  │  │     │      │
  common utils config policy filemeta ...
 ```
 

CHANGELOG.md

@@ -22,11 +22,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - XML-formatted error responses compatible with S3 API
   - Comprehensive integration documentation with manual testing guide
   - **32 unit and integration tests** covering middleware, auth handlers, task-local storage, and role detection
+- **SFTPv3 Protocol Support**: SSH-hosted SFTPv3 subsystem that translates each file operation into S3 calls against the local object store. Authentication uses IAM credentials (SSH username = access key, SSH password = secret key).
+  - Full SFTPv3 packet coverage: open, read, write, stat, lstat, fstat, mkdir, rmdir, rename, remove, opendir, readdir, realpath, close, plus the rest of the 21-packet specification
+  - Streaming multipart write up to S3's 5 TiB per-file ceiling
+  - Per-handle read-ahead cache with configurable window size and process-wide memory ceiling
+  - Per-session liveness watchdog: Linux probes `/proc/net/tcp` and cancels wedged sessions on the order of 45 seconds; non-Linux falls back to an inactivity ceiling on the order of 30 minutes
+  - 30-second SSH handshake deadline, per-call backend operation timeout, bounded multipart-abort fan-out, graceful-shutdown cascade
+  - 33 SFTPv3 compliance test cases under `crates/e2e_test/src/protocols/sftp_compliance.rs` spread across three entry points: `test_sftp_compliance_suite` (shared session), `test_sftp_compliance_readonly` (read-only mode), and `test_sftp_compliance_standalone` (one rustfs spawn per case)
+  - Four-layer regression-prevention tests guard against silent feature deletion: compile-time module assertion, module-presence unit test, cross-module `Protocol` enum assertion, end-to-end SSH banner test against the running binary
 
 ### Changed
 - **HTTP Server Stack**: Integrated `KeystoneAuthLayer` middleware from `rustfs-keystone` crate into service stack (positioned after ReadinessGateLayer)
 - **IAMAuth**: Enhanced `get_secret_key()` to return empty secret for Keystone credentials (bypasses signature validation)
 - **Auth Module**: Modified `check_key_valid()` to retrieve Keystone credentials from task-local storage and determine admin status
+- **`StorageBackend` trait**: extended with multipart upload methods (`create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`) plus `upload_part_copy`. Streaming-upload code path is now available to FTPS, WebDAV, and Swift drivers as well.
+- **`Protocol` enum**: new `Protocol::Sftp` variant with corresponding `S3Action` mappings. Every match arm on `Protocol` updated to handle the new variant exhaustively.
 
 ### Technical Details
 - Middleware is self-contained in `rustfs-keystone` crate following the trusted-proxies pattern for integration-specific middleware
@@ -35,12 +45,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Integration preserves existing S3 authentication flow while adding Keystone support
 - Zero breaking changes to existing functionality
 - No new top-level directories in main binary crate (middleware lives in integration crate)
+- SSH/SFTP wire handling via the `russh` and `russh-sftp` crates. SFTPv3 framing is implemented by `russh-sftp`; the rustfs-side `SftpDriver` implements `russh_sftp::server::Handler` and dispatches to the storage backend
+- Drop-time abort for in-flight multipart uploads honours IAM Deny on `AbortMultipartUpload`. `start_multipart_upload` caches the authorisation decision so the synchronous `Drop` path can honour Allow / Deny policies without re-querying IAM
+- Per-handle read cache uses an `Arc<AtomicU64>` shared across every `SftpDriver` instance to enforce a process-wide memory ceiling. On ceiling breach the populate is skipped and the read serves correctly via a single-call backend fetch
+- Per-session liveness watchdog runs as a tokio task per accepted connection. Reads `/proc/net/tcp` and `/proc/net/tcp6` to look up the (local, peer) tuple's TCP state and cancels via `tokio_util::sync::CancellationToken` when wedge conditions are confirmed across two consecutive ticks
+- Path canonicalisation rejects paths containing `\0`, `\r`, or `\n` and resolves traversal via `path::clean()` before any backend dispatch
+- Cipher / KEX / MAC / host-key algorithm allowlists are hardcoded with no environment override. Strict-KEX (CVE-2023-48795 / Terrapin) marker presence asserted by unit test
+- Per-session handle cap (default 64, configurable 8 to 1024) with UUID-generated handle ids
+- Crate-level `#![deny(unsafe_code)]` is in force across `crates/protocols`. Socket fd duplication for the watchdog uses the safe `AsFd::try_clone_to_owned` path (Linux/Unix); non-Unix falls back to the inactivity ceiling
+- `cfg(unix)` gating around platform-specific imports (`std::os::fd::AsFd`, `std::os::unix::fs::PermissionsExt`); non-Unix targets fail SFTP at config-load with `SftpInitError::UnsupportedPlatform`
 
 ### Documentation
 - Updated `crates/keystone/README.md` with complete integration architecture and workflow
 - Added detailed manual testing guide with 10 test scenarios
 - Updated main `README.md` to list Keystone authentication as available feature
 - Added troubleshooting section for common integration issues
+- Module-level rustdoc on `crates/protocols/src/sftp/mod.rs` describing the public API surface, configuration contract, and the architecture of the read cache and the wedge watchdog
 
 ### Configuration
 New environment variables:
@@ -54,6 +74,40 @@ New environment variables:
 - `RUSTFS_KEYSTONE_CACHE_SIZE` - Token cache size (default: 10000)
 - `RUSTFS_KEYSTONE_CACHE_TTL` - Token cache TTL in seconds (default: 300)
 - `RUSTFS_KEYSTONE_VERIFY_SSL` - Verify SSL certificates (default: true)
+- `RUSTFS_SFTP_ENABLE` - Enable/disable SFTP (default: false)
+- `RUSTFS_SFTP_ADDRESS` - Listen address (default: 0.0.0.0:2222)
+- `RUSTFS_SFTP_HOST_KEY_DIR` - Directory containing host key files (must exist; each file must be 0o600 or 0o400)
+- `RUSTFS_SFTP_IDLE_TIMEOUT` - Session idle timeout in seconds (default: 600)
+- `RUSTFS_SFTP_PART_SIZE` - Multipart part size in bytes (default: 16 MiB)
+- `RUSTFS_SFTP_READ_ONLY` - Reject write packets at the protocol layer (default: false)
+- `RUSTFS_SFTP_BANNER` - Optional SSH banner text
+- `RUSTFS_SFTP_HANDLES_PER_SESSION` - Per-session open-handle cap, 8 to 1024 (default: 64)
+- `RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS` - Per-call backend deadline in seconds, 5 to 600 (default: 60)
+- `RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES` - Per-handle read-cache window in bytes, 256 KiB to 64 MiB or 0 to disable (default: 4 MiB)
+- `RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES` - Process-wide read-cache memory ceiling in bytes, 16 MiB minimum (default: 256 MiB)
+
+### Files Added
+- `crates/protocols/src/sftp/mod.rs` - SFTP module entry point, public API surface, crate-level rustdoc, regression-prevention test
+- `crates/protocols/src/sftp/config.rs` - `SftpConfig` and `SftpInitError` types, env-var resolvers, host-key directory loader with permission enforcement
+- `crates/protocols/src/sftp/constants.rs` - Named constants grouped by purpose: S3 error codes, HTTP error codes, POSIX mode bits, protocol identifiers, operational limits
+- `crates/protocols/src/sftp/server.rs` - `SftpServer` SSH server, russh handler, password authentication against IAM, accept loop, per-session task spawn
+- `crates/protocols/src/sftp/driver.rs` - `SftpDriver` per-session SFTPv3 handler dispatching each operation onto the `StorageBackend`
+- `crates/protocols/src/sftp/state.rs` - `HandleState` variants for read, write-buffering, write-streaming, write-failed handles
+- `crates/protocols/src/sftp/lifecycle.rs` - Per-session activity stamp, weak-ref registry, `/proc/net/tcp` probe for the wedge watchdog
+- `crates/protocols/src/sftp/wedge_watchdog.rs` - Per-session liveness watchdog cancelling sessions silent at the SFTP layer while the kernel reports CLOSE_WAIT
+- `crates/protocols/src/sftp/read_cache.rs` - Per-handle in-memory read-ahead cache with shared atomic accumulator for the process-wide memory ceiling
+- `crates/protocols/src/sftp/attrs.rs` - SFTPv3 `FileAttributes` mapping for objects and directories, longname formatting, mtime clamping
+- `crates/protocols/src/sftp/dir.rs` - OPENDIR / READDIR pagination, root-bucket listing, sub-directory listing under a prefix
+- `crates/protocols/src/sftp/errors.rs` - `SftpError` thiserror enum and S3-error classification into SFTPv3 status codes
+- `crates/protocols/src/sftp/paths.rs` - Path canonicalisation, traversal rejection, `\0` / `\r` / `\n` rejection, bucket+key decomposition
+- `crates/protocols/src/sftp/read.rs` - READ packet handler, EOF semantics, `MAX_READ_LEN` bound, integration with the read cache
+- `crates/protocols/src/sftp/write.rs` - WRITE packet handler, in-memory buffering up to part size, transition to streaming multipart, CLOSE finalisation
+- `crates/protocols/src/sftp/test_support.rs` - Test fixtures and helper builders for SFTP unit tests
+- `crates/protocols/src/common/dummy_storage.rs` - In-memory `StorageBackend` test backend covering every method, used by SFTP unit tests and the FTPS / Swift / WebDAV test suites
+- `crates/e2e_test/src/protocols/sftp_core.rs` - End-to-end regressions for the handshake deadline, idle-timeout disconnect, and the wedge watchdog
+- `crates/e2e_test/src/protocols/sftp_compliance.rs` - SFTPv3 compliance suite entry points (`test_sftp_compliance_suite`, `test_sftp_compliance_readonly`, `test_sftp_compliance_standalone`)
+- `crates/e2e_test/src/protocols/sftp_compliance_tests.rs` - Per-case test bodies (CMPTST-01..33), shared fixture helpers, lifecycle counters
+- `crates/e2e_test/src/protocols/sftp_helpers.rs` - SFTP-specific test helpers and fixture seeders
 
 ### Files Modified
 - `crates/keystone/src/middleware.rs` - Created Keystone authentication middleware (self-contained in keystone crate)
@@ -63,6 +117,27 @@ New environment variables:
 - `rustfs/src/auth.rs` - Enhanced IAMAuth and check_key_valid for Keystone support, imported KEYSTONE_CREDENTIALS from rustfs-keystone
 - `crates/keystone/README.md` - Comprehensive integration documentation
 - `README.md` - Added Keystone as available feature
+- `Cargo.toml` - Added the `sftp` feature alongside the existing protocol features
+- `Cargo.lock` - Updated to include the new `russh`, `russh-sftp`, `socket2`, `tokio-util`, `subtle`, `uuid` dependencies and their transitive crates
+- `crates/protocols/Cargo.toml` - Declared `russh`, `russh-sftp`, `socket2`, `tokio-util`, `subtle`, `uuid` under the `sftp` feature flag
+- `crates/protocols/src/lib.rs` - Added `pub mod sftp` behind `#[cfg(feature = "sftp")]` plus the crate-level `#![deny(unsafe_code)]` lint
+- `crates/protocols/src/common/client/s3.rs` - Extended the `StorageBackend` trait with `create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`, and `upload_part_copy`
+- `crates/protocols/src/common/session.rs` - Added the `Protocol::Sftp` variant and its `S3Action` mappings
+- `crates/protocols/src/common/gateway.rs` - Handles the new `Protocol::Sftp` variant exhaustively
+- `crates/protocols/src/common/mod.rs` - Exposed the new `dummy_storage` module
+- `crates/protocols/src/constants.rs` - Added shared POSIX mode-bit constants used by SFTP and other protocols
+- `crates/config/src/constants/protocols.rs` - `RUSTFS_SFTP_*` environment variable names and defaults
+- `crates/utils/src/retry.rs` - Added the generic exponential-backoff retry helper used by the SFTP write path
+- `crates/e2e_test/Cargo.toml` - Added the e2e test dependencies for SFTP (paramiko fixture, SSH keypair generation)
+- `crates/e2e_test/src/protocols/mod.rs` - Registered the new `sftp_core`, `sftp_compliance`, `sftp_compliance_tests`, and `sftp_helpers` modules
+- `crates/e2e_test/src/protocols/README.md` - Documented the SFTP test entry points and case index
+- `crates/e2e_test/src/protocols/test_env.rs` - Added SFTP host-key directory provisioning to the shared protocol test environment
+- `crates/e2e_test/src/protocols/test_runner.rs` - Wired the SFTP entry points into the runner
+- `rustfs/Cargo.toml` - Added the `sftp` feature flag
+- `rustfs/src/lib.rs` - One-line addition exporting the SFTP wiring
+- `rustfs/src/init.rs` - Build and start the `SftpServer` when `RUSTFS_SFTP_ENABLE` is true
+- `rustfs/src/main.rs` - Routed shutdown signals to the SFTP server alongside the other protocols
+- `rustfs/src/protocols/client.rs` - Client-builder support for the new `Protocol::Sftp` variant
 
 ### Testing
 - 16 unit tests in rustfs-keystone crate (config, auth, middleware, identity)
@@ -70,7 +145,13 @@ New environment variables:
 - 6 auth unit tests in rustfs crate (role detection, task-local storage, Keystone credential handling)
 - **Total: 32 tests** passing with zero compilation errors
 - Manual testing guide provided for end-to-end validation
-- All tests passing with `cargo test --all --exclude e2e_test`
+- All Keystone tests passing with `cargo test --all --exclude e2e_test`
+- 33 SFTPv3 compliance test cases (CMPTST-01..33) split across three entry points: `test_sftp_compliance_suite` (shared session, cases 01-14), `test_sftp_compliance_readonly` (read-only mode, cases 15-23), `test_sftp_compliance_standalone` (one rustfs spawn per case, cases 24-33)
+- Regression-prevention tests at four layers: compile-time module assertion in `crates/protocols/src/lib.rs`, module-presence unit test in `crates/protocols/src/sftp/mod.rs`, cross-module `Protocol` enum assertion, and end-to-end SSH banner test against the running binary
+- Standalone end-to-end regressions for the SSH handshake deadline, the idle-timeout disconnect path, and the wedge watchdog (Linux fast-kill and the cross-platform fallback path)
+- Inline unit tests in every SFTP source file covering pure helpers (path canonicalisation, attribute mapping, S3-error classification, env-var bound resolvers)
+- Strict-KEX (CVE-2023-48795) marker presence assertion as a unit test in `crates/protocols/src/sftp/server.rs`
+- All tests passing with `cargo test --all --features sftp` against a 64-bit Linux target
 
 ---
 

Cargo.toml

@@ -58,8 +58,8 @@ resolver = "3"
 edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
-rust-version = "1.93.0"
-version = "0.0.5"
+rust-version = "1.95.0"
+version = "1.0.0-beta.3"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -76,73 +76,76 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "0.0.5" }
-rustfs-heal = { path = "crates/heal", version = "0.0.5" }
-rustfs-appauth = { path = "crates/appauth", version = "0.0.5" }
-rustfs-audit = { path = "crates/audit", version = "0.0.5" }
-rustfs-checksums = { path = "crates/checksums", version = "0.0.5" }
-rustfs-common = { path = "crates/common", version = "0.0.5" }
-rustfs-config = { path = "./crates/config", version = "0.0.5" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "0.0.5" }
-rustfs-credentials = { path = "crates/credentials", version = "0.0.5" }
-rustfs-crypto = { path = "crates/crypto", version = "0.0.5" }
-rustfs-ecstore = { path = "crates/ecstore", version = "0.0.5" }
-rustfs-filemeta = { path = "crates/filemeta", version = "0.0.5" }
-rustfs-iam = { path = "crates/iam", version = "0.0.5" }
-rustfs-keystone = { path = "crates/keystone", version = "0.0.5" }
-rustfs-kms = { path = "crates/kms", version = "0.0.5" }
-rustfs-lock = { path = "crates/lock", version = "0.0.5" }
-rustfs-madmin = { path = "crates/madmin", version = "0.0.5" }
-rustfs-notify = { path = "crates/notify", version = "0.0.5" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "0.0.5" }
-rustfs-io-core = { path = "crates/io-core", version = "0.0.5" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "0.0.5" }
-rustfs-obs = { path = "crates/obs", version = "0.0.5" }
-rustfs-policy = { path = "crates/policy", version = "0.0.5" }
-rustfs-protos = { path = "crates/protos", version = "0.0.5" }
-rustfs-protocols = { path = "crates/protocols", version = "0.0.5" }
-rustfs-rio = { path = "crates/rio", version = "0.0.5" }
-rustfs-s3-common = { path = "crates/s3-common", version = "0.0.5" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "0.0.5" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "0.0.5" }
-rustfs-scanner = { path = "crates/scanner", version = "0.0.5" }
-rustfs-signer = { path = "crates/signer", version = "0.0.5" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "0.0.5" }
-rustfs-targets = { path = "crates/targets", version = "0.0.5" }
-rustfs-utils = { path = "crates/utils", version = "0.0.5" }
-rustfs-workers = { path = "crates/workers", version = "0.0.5" }
-rustfs-zip = { path = "./crates/zip", version = "0.0.5" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.3" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.3" }
+rustfs-appauth = { path = "crates/appauth", version = "1.0.0-beta.3" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.3" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.3" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.3" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.3" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.3" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.3" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.3" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.3" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.3" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.3" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.3" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.3" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.3" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.3" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.3" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.3" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.3" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.3" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.3" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.3" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.3" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.3" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.3" }
+rustfs-s3-common = { path = "crates/s3-common", version = "1.0.0-beta.3" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.3" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.3" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.3" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.3" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.3" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.3" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.3" }
+rustfs-workers = { path = "crates/workers", version = "1.0.0-beta.3" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.3" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
-async-compression = { version = "0.4.41" }
+mysql_async = { version = "0.36.1", default-features = false, features = ["default-rustls", "tracing"], git = "https://github.com/blackbeam/mysql_async", rev = "98d3d8067efdf97d3e93cdca7b9231753c904aca" }
+async-compression = { version = "0.4.42" }
 async-recursion = "1.1.1"
 async-trait = "0.1.89"
-async-nats = "0.47.0"
+async-nats = "0.48.0"
 axum = "0.8.9"
 futures = "0.3.32"
 futures-core = "0.3.32"
 futures-util = "0.3.32"
 pollster = "0.4.0"
 pulsar = { version = "6.7.2", default-features = false, features = ["tokio-rustls-runtime"] }
+lapin = { version = "4.7.2", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
 hyper = { version = "1.9.0", features = ["http2", "http1", "server"] }
 hyper-rustls = { version = "0.27.9", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
 hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
 http = "1.4.0"
 http-body = "1.0.1"
 http-body-util = "0.1.3"
-reqwest = { version = "0.13.2", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
+reqwest = { version = "0.13.3", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
+rustfs-kafka-async = { version = "1.2.0" }
 socket2 = { version = "0.6.3", features = ["all"] }
-tokio = { version = "1.52.1", features = ["fs", "rt-multi-thread"] }
+tokio = { version = "1.52.3", features = ["fs", "rt-multi-thread"] }
 tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
 tokio-stream = { version = "0.1.18" }
 tokio-test = "0.4.5"
 tokio-util = { version = "0.7.18", features = ["io", "compat"] }
-tonic = { version = "0.14.5", features = ["gzip"] }
-tonic-prost = { version = "0.14.5" }
-tonic-prost-build = { version = "0.14.5" }
+tonic = { version = "0.14.6", features = ["gzip"] }
+tonic-prost = { version = "0.14.6" }
+tonic-prost-build = { version = "0.14.6" }
 tower = { version = "0.5.3", features = ["timeout"] }
-tower-http = { version = "0.6.8", features = ["cors"] }
+tower-http = { version = "0.6.10", features = ["cors"] }
 
 # Serialization and Data Formats
 bytes = { version = "1.11.1", features = ["serde"] }
@@ -151,7 +154,7 @@ byteorder = "1.5.0"
 flatbuffers = "25.12.19"
 form_urlencoded = "1.2.2"
 prost = "0.14.3"
-quick-xml = "0.39.2"
+quick-xml = "0.39.4"
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.228", features = ["derive"] }
@@ -167,10 +170,11 @@ crc-fast = "1.9.0"
 hmac = { version = "0.13.0" }
 jsonwebtoken = { version = "10.3.0", features = ["aws_lc_rs"] }
 openidconnect = { version = "4.0", default-features = false }
-pbkdf2 = "0.13.0-rc.10"
-rsa = { version = "0.10.0-rc.17" }
-rustls = { version = "0.23.38", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
-rustls-pki-types = "1.14.0"
+pbkdf2 = "0.13.0"
+rsa = { version = "0.10.0-rc.18" }
+rustls = { version = "0.23.40", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
+rustls-native-certs = "0.8"
+rustls-pki-types = "1.14.1"
 sha1 = "0.11.0"
 sha2 = "0.11.0"
 subtle = "2.6"
@@ -179,25 +183,29 @@ zeroize = { version = "1.8.2", features = ["derive"] }
 # Time and Date
 chrono = { version = "0.4.44", features = ["serde"] }
 humantime = "2.3.0"
-jiff = { version = "0.2.23", features = ["serde"] }
+jiff = { version = "0.2.24", features = ["serde"] }
 time = { version = "0.3.47", features = ["std", "parsing", "formatting", "macros", "serde"] }
 
+# Database
+deadpool-postgres = { version = "0.14", features = ["rt_tokio_1"] }
+tokio-postgres = { version = "0.7", default-features = false, features = ["runtime", "with-serde_json-1"] }
+tokio-postgres-rustls = "0.13"
+
 # Utilities and Tools
 anyhow = "1.0.102"
 arc-swap = "1.9.1"
-astral-tokio-tar = "0.6.0"
+astral-tokio-tar = "0.6.1"
 atoi = "2.0.0"
 atomic_enum = "0.3.0"
-aws-config = { version = "1.8.15" }
+aws-config = { version = "1.8.16" }
 aws-credential-types = { version = "1.2.14" }
-aws-sdk-s3 = { version = "1.129.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+aws-sdk-s3 = { version = "1.132.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
 aws-smithy-http-client = { version = "1.1.12", default-features = false, features = ["default-client", "rustls-aws-lc"] }
 aws-smithy-types = { version = "1.4.7" }
 backtrace = "0.3.76"
 base64 = "0.22.1"
 base64-simd = "0.8.0"
 brotli = "8.0.2"
-cfg-if = "1.0.4"
 clap = { version = "4.6.1", features = ["derive", "env"] }
 const-str = { version = "1.1.0", features = ["std", "proc"] }
 convert_case = "0.11.0"
@@ -208,21 +216,21 @@ crossbeam-deque = "0.8.6"
 crossbeam-utils = "0.8.21"
 datafusion = "53.1.0"
 derive_builder = "0.20.2"
-enumset = "1.1.10"
+enumset = "1.1.12"
 faster-hex = "0.10.0"
 flate2 = "1.1.9"
 glob = "0.3.3"
-google-cloud-storage = "1.11.0"
-google-cloud-auth = "1.9.0"
-hashbrown = { version = "0.17.0", features = ["serde", "rayon"] }
+google-cloud-storage = "1.12.0"
+google-cloud-auth = "1.10.0"
+hashbrown = { version = "0.17.1", features = ["serde", "rayon"] }
 hex = "0.4.3"
 hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
 ipnetwork = { version = "0.21.1", features = ["serde"] }
 lazy_static = "1.5.0"
-libc = "0.2.185"
+libc = "0.2.186"
 libsystemd = "0.7.2"
-local-ip-address = "0.6.11"
+local-ip-address = "0.6.12"
 memmap2 = "0.9.10"
 lz4 = "1.28.1"
 matchit = "0.9.2"
@@ -246,21 +254,22 @@ rayon = "1.12.0"
 reed-solomon-erasure = { version = "6.0", default-features = false, features = ["std", "simd-accel"] }
 reed-solomon-simd = "3.1.0"
 regex = { version = "1.12.3" }
-rumqttc = { package = "rumqttc-next", version = "0.30.0", features = ["websocket"] }
+rumqttc = { package = "rumqttc-next", version = "0.33.0", features = ["websocket"] }
+redis = { version = "1.2.1", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
 rustix = { version = "1.1.4", features = ["fs"] }
 rust-embed = { version = "8.11.0" }
 rustc-hash = { version = "2.1.2" }
 s3s = { git = "https://github.com/rustfs/s3s", rev = "a3b16608df35aaeed8fff08b4988d03f4ca9445b", features = ["minio"] }
 serial_test = "3.4.0"
-shadow-rs = { version = "1.7.1", default-features = false }
-siphasher = "1.0.2"
+shadow-rs = { version = "2.0.0", default-features = false }
+siphasher = "1.0.3"
 smallvec = { version = "1.15.1", features = ["serde"] }
 smartstring = "1.0.1"
 snafu = "0.9.0"
 snap = "1.1.1"
 starshard = { version = "1.1.0", features = ["rayon", "async", "serde"] }
 strum = { version = "0.28.0", features = ["derive"] }
-sysinfo = "0.38.4"
+sysinfo = "0.39.0"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
@@ -279,11 +288,11 @@ walkdir = "2.5.0"
 wildmatch = { version = "2.6.1", features = ["serde"] }
 windows = { version = "0.62.2" }
 xxhash-rust = { version = "0.8.15", features = ["xxh64", "xxh3"] }
-zip = "8.5.1"
+zip = "8.6.0"
 zstd = "0.13.3"
 
 # Observability and Metrics
-metrics = "0.24.3"
+metrics = "0.24.5"
 dial9-tokio-telemetry = "0.3"
 opentelemetry = { version = "0.31.0" }
 opentelemetry-appender-tracing = { version = "0.31.1", features = ["experimental_use_tracing_span_context", "experimental_metadata_attributes", "spec_unstable_logs_enabled"] }
@@ -291,13 +300,15 @@ opentelemetry-otlp = { version = "0.31.1", features = ["gzip-http", "reqwest-rus
 opentelemetry_sdk = { version = "0.31.0" }
 opentelemetry-semantic-conventions = { version = "0.31.0", features = ["semconv_experimental"] }
 opentelemetry-stdout = { version = "0.31.0" }
-pyroscope = { version = "2.0.0", features = ["backend-pprof-rs"] }
+pyroscope = { version = "2.0.3", features = ["backend-pprof-rs"] }
 
 # FTP and SFTP
 libunftp = { version = "0.23.0", features = ["experimental"] }
 unftp-core = "0.1.0"
-suppaftp = { version = "8.0.2", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
+suppaftp = { version = "8.0.3", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
 rcgen = "0.14.7"
+russh = "0.60.2"
+russh-sftp = "2.1.2"
 
 # WebDAV
 dav-server = "0.11.0"
@@ -318,6 +329,12 @@ pprof = { package = "pprof-pyroscope-fork", version = "0.1500.3", features = ["f
 [workspace.metadata.cargo-shear]
 ignored = ["rustfs"]
 
+[patch.crates-io]
+# Pinned to the simon-escapecode/russh fork carrying the upstream fix at
+# https://github.com/Eugeny/russh/pull/702. Drops out when a russh release
+# containing the fix is published.
+russh = { git = "https://github.com/simon-escapecode/russh", rev = "5cac2ed84945f9b80a52b673e058f2032bbe98ec" }
+
 [profile.release]
 opt-level = 3
 

Dockerfile

@@ -29,8 +29,20 @@ RUN set -eux; \
     if [ "$RELEASE" = "latest" ]; then \
       TAG="$(curl -fsSL https://api.github.com/repos/rustfs/rustfs/releases \
               | grep -o '"tag_name": "[^"]*"' | cut -d'"' -f4 | head -n 1)"; \
+      RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG")"; \
     else \
       TAG="$RELEASE"; \
+      RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG" 2>/dev/null || true)"; \
+      if [ -z "$RELEASE_JSON" ]; then \
+        if [ "${TAG#v}" = "$TAG" ]; then \
+          ALT_TAG="v$TAG"; \
+        else \
+          ALT_TAG="${TAG#v}"; \
+        fi; \
+        echo "Primary tag lookup failed, retrying with alternate tag: $ALT_TAG"; \
+        RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$ALT_TAG")"; \
+        TAG="$ALT_TAG"; \
+      fi; \
     fi; \
     echo "Using tag: $TAG (arch pattern: $ARCH_SUBSTR)"; \
     # Find download URL in assets list for this tag that contains arch substring and ends with .zip
@@ -87,8 +99,7 @@ RUN addgroup -g 10001 -S rustfs && \
     chown -R rustfs:rustfs /data /logs && \
     chmod 0750 /data /logs
 
-ENV RUSTFS_CORS_ALLOWED_ORIGINS="*" \
-    RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
+ENV RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_VOLUMES="/data" \
     RUSTFS_OBS_LOGGER_LEVEL=warn \
     RUSTFS_OBS_LOG_DIRECTORY=/logs \

Dockerfile.decommission-local

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM rust:1.91-trixie
+FROM rust:1.93-trixie
 
 RUN set -eux; \
     export DEBIAN_FRONTEND=noninteractive; \

Dockerfile.glibc

@@ -31,14 +31,25 @@ RUN set -eux; \
       arm64)  ARCH_SUBSTR="aarch64-gnu" ;; \
       *) echo "Unsupported TARGETARCH=$TARGETARCH" >&2; exit 1 ;; \
     esac; \
-    \
     if [ "$RELEASE" = "latest" ]; then \
       TAG="$(curl -fsSL https://api.github.com/repos/rustfs/rustfs/releases \
               | grep -o '"tag_name": "[^"]*"' | cut -d'"' -f4 | head -n 1)"; \
+      RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG")"; \
     else \
       TAG="$RELEASE"; \
+      RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG" 2>/dev/null || true)"; \
+      if [ -z "$RELEASE_JSON" ]; then \
+        if [ "${TAG#v}" = "$TAG" ]; then \
+          ALT_TAG="v$TAG"; \
+        else \
+          ALT_TAG="${TAG#v}"; \
+        fi; \
+        echo "Primary tag lookup failed, retrying with alternate tag: $ALT_TAG"; \
+        RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$ALT_TAG")"; \
+        TAG="$ALT_TAG"; \
+      fi; \
     fi; \
-    \
+    echo "Using tag: $TAG (arch pattern: $ARCH_SUBSTR)"; \
     URL="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG" \
            | grep -o "\"browser_download_url\": \"[^\"]*${ARCH_SUBSTR}[^\"]*\\.zip\"" \
            | cut -d'"' -f4 | head -n 1)"; \
@@ -97,7 +108,6 @@ ENV RUSTFS_ADDRESS=":9000" \
     RUSTFS_ACCESS_KEY="rustfsadmin" \
     RUSTFS_SECRET_KEY="rustfsadmin" \
     RUSTFS_CONSOLE_ENABLE="true" \
-    RUSTFS_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_VOLUMES="/data" \
     RUSTFS_OBS_LOGGER_LEVEL=warn \

Dockerfile.source

@@ -26,15 +26,17 @@
 
 ARG TARGETPLATFORM
 ARG BUILDPLATFORM
+ARG TARGETARCH
 
 # -----------------------------
 # Build stage
 # -----------------------------
-FROM rust:1.91-trixie AS builder
+FROM rust:1.93-trixie AS builder
 
 # Re-declare args after FROM
 ARG TARGETPLATFORM
 ARG BUILDPLATFORM
+ARG TARGETARCH
 
 # Debug: print platforms
 RUN echo "Build info -> BUILDPLATFORM=${BUILDPLATFORM}, TARGETPLATFORM=${TARGETPLATFORM}"
@@ -60,7 +62,15 @@ RUN set -eux; \
 
 # Optional: cross toolchain for aarch64 (only when targeting linux/arm64)
 RUN set -eux; \
-    if [ "${TARGETPLATFORM:-linux/amd64}" = "linux/arm64" ]; then \
+    target_platform="${TARGETPLATFORM:-}"; \
+    if [ -z "${target_platform}" ]; then \
+    case "$(uname -m)" in \
+    x86_64) target_platform="linux/amd64" ;; \
+    aarch64|arm64) target_platform="linux/arm64" ;; \
+    *) target_platform="linux/amd64" ;; \
+    esac; \
+    fi; \
+    if [ "${target_platform}" = "linux/arm64" ]; then \
     export DEBIAN_FRONTEND=noninteractive; \
     apt-get update; \
     apt-get install -y --no-install-recommends gcc-aarch64-linux-gnu; \
@@ -79,6 +89,7 @@ ENV CXX_aarch64_unknown_linux_gnu=aarch64-linux-gnu-g++
 ENV CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc
 ENV CC_x86_64_unknown_linux_gnu=x86_64-linux-gnu-gcc
 ENV CXX_x86_64_unknown_linux_gnu=x86_64-linux-gnu-g++
+ENV CARGO_TARGET_DIR=/usr/src/rustfs/target/docker-build
 
 WORKDIR /usr/src/rustfs
 
@@ -112,27 +123,36 @@ ENV CARGO_NET_GIT_FETCH_WITH_CLI=true \
 # Generate protobuf/flatbuffers code (uses protoc/flatc from distro)
 RUN --mount=type=cache,target=/usr/local/cargo/registry \
     --mount=type=cache,target=/usr/local/cargo/git \
-    --mount=type=cache,target=/usr/src/rustfs/target \
+    --mount=type=cache,target=/usr/src/rustfs/target/docker-build \
     cargo run --bin gproto
 
 # Build RustFS (target depends on TARGETPLATFORM)
 RUN --mount=type=cache,target=/usr/local/cargo/registry \
     --mount=type=cache,target=/usr/local/cargo/git \
-    --mount=type=cache,target=/usr/src/rustfs/target \
+    --mount=type=cache,target=/usr/src/rustfs/target/docker-build \
     set -eux; \
-    case "${TARGETPLATFORM:-linux/amd64}" in \
+    rustup target add x86_64-unknown-linux-gnu aarch64-unknown-linux-gnu; \
+    target_platform="${TARGETPLATFORM:-}"; \
+    if [ -z "${target_platform}" ]; then \
+    case "${TARGETARCH:-$(uname -m)}" in \
+    amd64|x86_64) target_platform="linux/amd64" ;; \
+    arm64|aarch64) target_platform="linux/arm64" ;; \
+    *) echo "Unsupported target architecture: ${TARGETARCH:-$(uname -m)}" >&2; exit 1 ;; \
+    esac; \
+    fi; \
+    case "${target_platform}" in \
     linux/amd64) \
     echo "Building for x86_64-unknown-linux-gnu"; \
     cargo build --release --locked --target x86_64-unknown-linux-gnu --bin rustfs -j "$(nproc)"; \
-    install -m 0755 target/x86_64-unknown-linux-gnu/release/rustfs /usr/local/bin/rustfs \
+    install -m 0755 "${CARGO_TARGET_DIR}/x86_64-unknown-linux-gnu/release/rustfs" /usr/local/bin/rustfs \
     ;; \
     linux/arm64) \
     echo "Building for aarch64-unknown-linux-gnu"; \
     cargo build --release --locked --target aarch64-unknown-linux-gnu --bin rustfs -j "$(nproc)"; \
-    install -m 0755 target/aarch64-unknown-linux-gnu/release/rustfs /usr/local/bin/rustfs \
+    install -m 0755 "${CARGO_TARGET_DIR}/aarch64-unknown-linux-gnu/release/rustfs" /usr/local/bin/rustfs \
     ;; \
     *) \
-    echo "Unsupported TARGETPLATFORM=${TARGETPLATFORM}" >&2; exit 1 \
+    echo "Unsupported target platform=${target_platform}" >&2; exit 1 \
     ;; \
     esac
 
@@ -178,7 +198,7 @@ CMD ["cargo", "run", "--bin", "rustfs", "--"]
 # -----------------------------
 # Runtime stage (Ubuntu minimal)
 # -----------------------------
-FROM ubuntu:22.04
+FROM ubuntu:24.04
 
 ARG BUILD_DATE
 ARG VCS_REF
@@ -195,6 +215,7 @@ RUN set -eux; \
     apt-get update; \
     apt-get install -y --no-install-recommends \
     ca-certificates \
+    curl \
     tzdata \
     coreutils; \
     rm -rf /var/lib/apt/lists/*

README.md

@@ -115,7 +115,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-alpha.76
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.3
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -136,6 +136,23 @@ Similarly, you can run the command with podman
 podman compose --profile observability up -d
 ```
 
+Webhook notification quick start (Docker):
+
+```bash
+docker run -d --name rustfs -p 9000:9000 \
+  -e RUSTFS_NOTIFY_ENABLE=true \
+  -e RUSTFS_NOTIFY_WEBHOOK_ENABLE_PRIMARY=on \
+  -e RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY=http://<host-ip>:3020/webhook \
+  -e RUSTFS_NOTIFY_WEBHOOK_QUEUE_DIR_PRIMARY=/tmp/rustfs-events \
+  rustfs/rustfs:latest
+```
+
+Notes:
+- `RUSTFS_NOTIFY_ENABLE=true` enables the global notify module switch.
+- For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
+- If queue dir is omitted, default is `/opt/rustfs/events`; ensure it is writable by the container runtime user.
+- `RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
+
 **NOTE**: We recommend reviewing the `docker-compose.yml` file before running. It defines several services including Grafana, Prometheus, and Jaeger, which are helpful for RustFS observability. If you wish to start Redis or Nginx containers, you can specify the corresponding profiles.
 
 ### 3\. Build from Source (Option 3) - Advanced Users

README_ZH.md

@@ -113,7 +113,7 @@ RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用
  docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
  # 使用指定版本运行
- docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0.alpha.68
+ docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.3
 ```
 
 您也可以使用 Docker Compose。使用根目录下的 `docker-compose.yml` 文件：

build-rustfs.sh

@@ -109,6 +109,7 @@ FORCE_CONSOLE_UPDATE=false
 CONSOLE_VERSION="latest"
 SKIP_VERIFICATION=false
 CUSTOM_PLATFORM=""
+FEATURES=""
 
 # Print usage
 usage() {
@@ -141,6 +142,7 @@ usage() {
     echo "  --no-console               Skip console static assets"
     echo "  --force-console-update     Force update console assets even if they exist"
     echo "  --console-version VERSION  Console version to download (default: latest)"
+    echo "  -f, --features FEATURES    Cargo features to enable (e.g. 'webdav', 'full')"
     echo "  --skip-verification        Skip binary verification after build"
     echo "  -h, --help                 Show this help message"
     echo ""
@@ -148,6 +150,8 @@ usage() {
     echo "  $0                         # Build for current platform (includes console assets)"
     echo "  $0 --dev                   # Development build"
     echo "  $0 --sign                  # Build and sign binary (release CI)"
+    echo "  $0 --features webdav       # Build with WebDAV support"
+    echo "  $0 --features full         # Build with all protocol features"
     echo "  $0 --no-console            # Build without console static assets"
     echo "  $0 --force-console-update  # Force update console assets"
     echo "  $0 --platform x86_64-unknown-linux-musl  # Build for specific platform"
@@ -440,6 +444,12 @@ build_binary() {
     build_cmd+=" --target $PLATFORM"
     build_cmd+=" -p rustfs --bins"
 
+    if [ -n "$FEATURES" ]; then
+        local quoted_features
+        printf -v quoted_features '%q' "$FEATURES"
+        build_cmd+=" --features $quoted_features"
+    fi
+
     print_message $BLUE "📦 Executing: $build_cmd"
 
     # Execute build (this matches exactly what the working version does)
@@ -497,6 +507,9 @@ build_rustfs() {
         print_message $YELLOW "   Force Console Update: $FORCE_CONSOLE_UPDATE"
     fi
     print_message $YELLOW "   Skip Verification: $SKIP_VERIFICATION"
+    if [ -n "$FEATURES" ]; then
+        print_message $YELLOW "   Features: $FEATURES"
+    fi
     echo ""
 
     # Setup environment
@@ -565,6 +578,15 @@ while [[ $# -gt 0 ]]; do
             SKIP_VERIFICATION=true
             shift
             ;;
+        -f|--features)
+            if [ -z "${2:-}" ]; then
+                print_message $RED "❌ Missing value for $1"
+                usage
+                exit 1
+            fi
+            FEATURES="$2"
+            shift 2
+            ;;
         -h|--help)
             usage
             exit 0

crates/appauth/Cargo.toml

@@ -26,7 +26,7 @@ categories = ["web-programming", "development-tools", "authentication"]
 
 [dependencies]
 base64-simd = { workspace = true }
-rsa = { workspace = true }
+rsa = { workspace = true, features = ["sha2"] }
 serde.workspace = true
 serde_json.workspace = true
 rand.workspace = true

crates/appauth/src/token.rs

@@ -15,9 +15,13 @@
 use rsa::{
     Pkcs1v15Encrypt, RsaPrivateKey, RsaPublicKey,
     pkcs8::{DecodePrivateKey, DecodePublicKey},
+    pss::{BlindedSigningKey, Signature, VerifyingKey},
+    sha2::Sha256,
+    signature::{RandomizedSigner, Verifier},
+    traits::PublicKeyParts,
 };
 use serde::{Deserialize, Serialize};
-use std::io::{Error, Result};
+use std::io::{Error, ErrorKind, Result};
 
 #[derive(Serialize, Deserialize, Debug, Default, Clone)]
 pub struct Token {
@@ -25,10 +29,11 @@ pub struct Token {
     pub expired: u64, // Expiry time (UNIX timestamp)
 }
 
-/// Public key generation Token
-/// [token] Token object
-/// [key] Public key string
-/// Returns the encrypted string processed by base64
+/// Legacy public-key encryption Token encoder.
+///
+/// Use `sign_license_token` for license issuance so verifiers only need a
+/// public key.
+#[deprecated(note = "use sign_license_token for signed license issuance")]
 pub fn gencode(token: &Token, key: &str) -> Result<String> {
     let data = serde_json::to_vec(token)?;
     let mut rng = rand::rng();
@@ -37,35 +42,62 @@ pub fn gencode(token: &Token, key: &str) -> Result<String> {
     Ok(base64_simd::URL_SAFE_NO_PAD.encode_to_string(&encrypted_data))
 }
 
-/// Private key resolution Token
-/// [token] Encrypted string processed by base64
-/// [key] Private key string
-/// Return to the Token object
+/// Legacy private-key Token decoder.
+///
+/// Use `parse_signed_license_token` or `parse_license_with_public_key` for
+/// license verification so runtime services never need private key material.
+#[deprecated(note = "use parse_signed_license_token or parse_license_with_public_key for signed license verification")]
 pub fn parse(token: &str, key: &str) -> Result<Token> {
     let encrypted_data = base64_simd::URL_SAFE_NO_PAD
         .decode_to_vec(token.as_bytes())
         .map_err(Error::other)?;
     let private_key = RsaPrivateKey::from_pkcs8_pem(key).map_err(Error::other)?;
     let decrypted_data = private_key.decrypt(Pkcs1v15Encrypt, &encrypted_data).map_err(Error::other)?;
-    let res: Token = serde_json::from_slice(&decrypted_data)?;
-    Ok(res)
+    serde_json::from_slice(&decrypted_data).map_err(Error::other)
 }
 
-pub fn parse_license(license: &str) -> Result<Token> {
-    parse(license, TEST_PRIVATE_KEY)
-    // match parse(license, TEST_PRIVATE_KEY) {
-    //     Ok(token) => {
-    //         if token.expired > SystemTime::now().duration_since(UNIX_EPOCH)?.as_secs() {
-    //             Ok(token)
-    //         } else {
-    //             Err("Token expired".into())
-    //         }
-    //     }
-    //     Err(e) => Err(e),
-    // }
+/// Signs a license token with an RSA private key.
+///
+/// The returned token is base64url(signature || payload), where the signature is
+/// RSASSA-PSS over the JSON payload using SHA-256.
+pub fn sign_license_token(token: &Token, private_key_pem: &str) -> Result<String> {
+    let payload = serde_json::to_vec(token)?;
+    let mut rng = rand::rng();
+    let private_key = RsaPrivateKey::from_pkcs8_pem(private_key_pem).map_err(Error::other)?;
+    let signing_key = BlindedSigningKey::<Sha256>::new(private_key);
+    let signature: Signature = signing_key.try_sign_with_rng(&mut rng, &payload).map_err(Error::other)?;
+    let signature: Box<[u8]> = signature.into();
+
+    let mut signed_payload = Vec::with_capacity(signature.as_ref().len() + payload.len());
+    signed_payload.extend_from_slice(signature.as_ref());
+    signed_payload.extend_from_slice(&payload);
+
+    Ok(base64_simd::URL_SAFE_NO_PAD.encode_to_string(&signed_payload))
 }
 
-static TEST_PRIVATE_KEY: &str = "-----BEGIN PRIVATE KEY-----\nMIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCj86SrJIuxSxR6\nBJ/dlJEUIj6NeBRnhLQlCDdovuz61+7kJXVcxaR66w4m8W7SLEUP+IlPtnn6vmiG\n7XMhGNHIr7r1JsEVVLhZmL3tKI66DEZl786ZhG81BWqUlmcooIPS8UEPZNqJXLuz\nVGhxNyVGbj/tV7QC2pSISnKaixc+nrhxvo7w56p5qrm9tik0PjTgfZsUePkoBsSN\npoRkAauS14MAzK6HGB75CzG3dZqXUNWSWVocoWtQbZUwFGXyzU01ammsHQDvc2xu\nK1RQpd1qYH5bOWZ0N0aPFwT0r59HztFXg9sbjsnuhO1A7OiUOkc6iGVuJ0wm/9nA\nwZIBqzgjAgMBAAECggEAPMpeSEbotPhNw2BrllE76ec4omPfzPJbiU+em+wPGoNu\nRJHPDnMKJbl6Kd5jZPKdOOrCnxfd6qcnQsBQa/kz7+GYxMV12l7ra+1Cnujm4v0i\nLTHZvPpp8ZLsjeOmpF3AAzsJEJgon74OqtOlVjVIUPEYKvzV9ijt4gsYq0zfdYv0\nhrTMzyrGM4/UvKLsFIBROAfCeWfA7sXLGH8JhrRAyDrtCPzGtyyAmzoHKHtHafcB\nuyPFw/IP8otAgpDk5iiQPNkH0WwzAQIm12oHuNUa66NwUK4WEjXTnDg8KeWLHHNv\nIfN8vdbZchMUpMIvvkr7is315d8f2cHCB5gEO+GWAQKBgQDR/0xNll+FYaiUKCPZ\nvkOCAd3l5mRhsqnjPQ/6Ul1lAyYWpoJSFMrGGn/WKTa/FVFJRTGbBjwP+Mx10bfb\ngUg2GILDTISUh54fp4zngvTi9w4MWGKXrb7I1jPkM3vbJfC/v2fraQ/r7qHPpO2L\nf6ZbGxasIlSvr37KeGoelwcAQQKBgQDH3hmOTS2Hl6D4EXdq5meHKrfeoicGN7m8\noQK7u8iwn1R9zK5nh6IXxBhKYNXNwdCQtBZVRvFjjZ56SZJb7lKqa1BcTsgJfZCy\nnI3Uu4UykrECAH8AVCVqBXUDJmeA2yE+gDAtYEjvhSDHpUfWxoGHr0B/Oqk2Lxc/\npRy1qV5fYwKBgBWSL/hYVf+RhIuTg/s9/BlCr9SJ0g3nGGRrRVTlWQqjRCpXeFOO\nJzYqSq9pFGKUggEQxoOyJEFPwVDo9gXqRcyov+Xn2kaXl7qQr3yoixc1YZALFDWY\nd1ySBEqQr0xXnV9U/gvEgwotPRnjSzNlLWV2ZuHPtPtG/7M0o1H5GZMBAoGAKr3N\nW0gX53o+my4pCnxRQW+aOIsWq1a5aqRIEFudFGBOUkS2Oz+fI1P1GdrRfhnnfzpz\n2DK+plp/vIkFOpGhrf4bBlJ2psjqa7fdANRFLMaAAfyXLDvScHTQTCcnVUAHQPVq\n2BlSH56pnugyj7SNuLV6pnql+wdhAmRN2m9o1h8CgYAbX2juSr4ioXwnYjOUdrIY\n4+ERvHcXdjoJmmPcAm4y5NbSqLXyU0FQmplNMt2A5LlniWVJ9KNdjAQUt60FZw/+\nr76LdxXaHNZghyx0BOs7mtq5unSQXamZ8KixasfhE9uz3ij1jXjG6hafWkS8/68I\nuWbaZqgvy7a9oPHYlKH7Jg==\n-----END PRIVATE KEY-----\n";
+/// Verifies and parses a signed license token with an RSA public key.
+pub fn parse_signed_license_token(token: &str, public_key_pem: &str) -> Result<Token> {
+    let signed_payload = base64_simd::URL_SAFE_NO_PAD
+        .decode_to_vec(token.as_bytes())
+        .map_err(Error::other)?;
+    let public_key = RsaPublicKey::from_public_key_pem(public_key_pem).map_err(Error::other)?;
+    let signature_len = public_key.size();
+
+    if signed_payload.len() <= signature_len {
+        return Err(Error::new(ErrorKind::InvalidData, "license token is missing signed payload"));
+    }
+
+    let (signature, payload) = signed_payload.split_at(signature_len);
+    let signature = Signature::try_from(signature).map_err(Error::other)?;
+    let verifying_key = VerifyingKey::<Sha256>::new(public_key);
+    verifying_key.verify(payload, &signature).map_err(Error::other)?;
+
+    serde_json::from_slice(payload).map_err(Error::other)
+}
+
+pub fn parse_license_with_public_key(license: &str, public_key: &str) -> Result<Token> {
+    parse_signed_license_token(license, public_key)
+}
 
 #[cfg(test)]
 mod tests {
@@ -77,7 +109,7 @@ mod tests {
     use std::time::{SystemTime, UNIX_EPOCH};
 
     #[test]
-    fn test_gencode_and_parse() {
+    fn test_sign_license_token_and_parse_signed_license_token() {
         let mut rng = rand::rng();
         let bits = 2048;
         let private_key = RsaPrivateKey::new(&mut rng, bits).expect("Failed to generate private key");
@@ -91,8 +123,31 @@ mod tests {
             expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600, // 1 hour from now
         };
 
-        let encoded = gencode(&token, &public_key_pem).expect("Failed to encode token");
+        let encoded = sign_license_token(&token, &private_key_pem).expect("Failed to encode token");
 
+        let decoded = parse_signed_license_token(&encoded, &public_key_pem).expect("Failed to decode token");
+
+        assert_eq!(token.name, decoded.name);
+        assert_eq!(token.expired, decoded.expired);
+    }
+
+    #[test]
+    #[allow(deprecated)]
+    fn test_legacy_gencode_and_parse_roundtrip() {
+        let mut rng = rand::rng();
+        let bits = 2048;
+        let private_key = RsaPrivateKey::new(&mut rng, bits).expect("Failed to generate private key");
+        let public_key = RsaPublicKey::from(&private_key);
+
+        let private_key_pem = private_key.to_pkcs8_pem(LineEnding::LF).unwrap();
+        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+
+        let token = Token {
+            name: "test_app".to_string(),
+            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600,
+        };
+
+        let encoded = gencode(&token, &public_key_pem).expect("Failed to encode token");
         let decoded = parse(&encoded, &private_key_pem).expect("Failed to decode token");
 
         assert_eq!(token.name, decoded.name);
@@ -100,28 +155,60 @@ mod tests {
     }
 
     #[test]
-    fn test_parse_invalid_token() {
+    fn test_parse_signed_license_token_rejects_tampered_payload() {
         let mut rng = rand::rng();
-        let private_key_pem = RsaPrivateKey::new(&mut rng, 2048)
-            .expect("Failed to generate private key")
-            .to_pkcs8_pem(LineEnding::LF)
-            .unwrap();
+        let private_key = RsaPrivateKey::new(&mut rng, 2048).expect("Failed to generate private key");
+        let public_key = RsaPublicKey::from(&private_key);
+        let private_key_pem = private_key.to_pkcs8_pem(LineEnding::LF).unwrap();
+        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+        let token = Token {
+            name: "test_app".to_string(),
+            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600,
+        };
 
-        let invalid_token = "invalid_base64_token";
-        let result = parse(invalid_token, &private_key_pem);
+        let encoded = sign_license_token(&token, &private_key_pem).expect("Failed to encode token");
+        let mut signed_payload = base64_simd::URL_SAFE_NO_PAD
+            .decode_to_vec(encoded.as_bytes())
+            .expect("Failed to decode signed payload");
+        let last_byte = signed_payload.last_mut().expect("Signed payload should not be empty");
+        *last_byte ^= 0x01;
+        let tampered = base64_simd::URL_SAFE_NO_PAD.encode_to_string(&signed_payload);
+
+        let result = parse_signed_license_token(&tampered, &public_key_pem);
 
         assert!(result.is_err());
     }
 
     #[test]
-    fn test_gencode_with_invalid_key() {
+    fn test_source_does_not_embed_private_key() {
+        let source = include_str!("token.rs");
+        let forbidden = ["BEGIN", "PRIVATE KEY"].join(" ");
+
+        assert!(!source.contains(&forbidden));
+    }
+
+    #[test]
+    fn test_parse_signed_license_token_rejects_invalid_token() {
+        let mut rng = rand::rng();
+        let private_key = RsaPrivateKey::new(&mut rng, 2048).expect("Failed to generate private key");
+        let public_key = RsaPublicKey::from(&private_key);
+        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+
+        let invalid_token = "invalid_base64_token";
+        let result = parse_signed_license_token(invalid_token, &public_key_pem);
+
+        assert!(result.is_err());
+    }
+
+    #[test]
+    fn test_sign_license_token_with_invalid_signing_key() {
         let token = Token {
             name: "test_app".to_string(),
             expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600, // 1 hour from now
         };
 
-        let invalid_key = "invalid_public_key";
-        let result = gencode(&token, invalid_key);
+        let invalid_key = "invalid_private_key";
+        let result = sign_license_token(&token, invalid_key);
 
         assert!(result.is_err());
     }

crates/audit/AGENTS.md

@@ -0,0 +1,53 @@
+# Audit Crate Instructions
+
+Applies to `crates/audit/`.
+
+`rustfs-audit` is the domain layer for audit event fan-out and observability.
+It composes shared plugin/runtime abstractions from `rustfs-targets` and keeps
+audit-specific dispatch semantics, state transitions, and metrics in this
+crate.
+
+## Domain Boundaries
+
+- Keep audit-specific behavior here:
+  - audit event shaping and fan-out pipeline
+  - audit system lifecycle/state transitions
+  - audit metrics and reporting
+- Keep shared plugin/runtime mechanics in `rustfs-targets`:
+  - no duplicated replay worker orchestration
+  - no duplicated runtime manager primitives
+  - no plugin install/control-plane modeling in this crate
+
+## Runtime Layering Rules
+
+- `pipeline.rs` hosts:
+  - `AuditPipeline` (dispatch and snapshot access)
+  - `AuditRuntimeFacade` (runtime mutation path)
+  - `AuditRuntimeView` (runtime read path)
+- `registry.rs` should remain the single owner of runtime target container and
+  plugin registry composition for audit.
+- `system.rs` should coordinate lifecycle by calling facade/view/registry
+  boundaries rather than embedding low-level runtime logic.
+
+## Change Style
+
+- Preserve audit delivery semantics and error handling behavior unless the task
+  explicitly changes them.
+- Prefer extending shared abstractions in `rustfs-targets` over patching
+  one-off audit-only runtime flows.
+- Keep logging and observability machine-meaningful; avoid noisy churn in hot
+  dispatch paths.
+
+## Testing
+
+- Keep unit tests close to changed modules.
+- Keep pipeline-layer regressions in `tests/pipeline_layer_test.rs`.
+- Add regression tests for:
+  - runtime facade activation/replace/stop/shutdown behavior
+  - runtime view target/snapshot access
+  - system reload and runtime commit/clear boundaries
+- Suggested validation:
+  - `cargo test -p rustfs-audit`
+  - Focused: `cargo test -p rustfs-audit --test pipeline_layer_test`
+  - Focused: `cargo test -p rustfs-audit pipeline`
+- Full gate before commit: `make pre-commit`

crates/audit/Cargo.toml

@@ -30,7 +30,6 @@ rustfs-targets = { workspace = true }
 rustfs-config = { workspace = true, features = ["audit", "constants"] }
 rustfs-ecstore = { workspace = true }
 rustfs-s3-common = { workspace = true }
-async-trait = { workspace = true }
 chrono = { workspace = true }
 const-str = { workspace = true }
 futures = { workspace = true }
@@ -41,15 +40,14 @@ serde_json = { workspace = true }
 thiserror = { workspace = true }
 tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "rt", "time", "macros"] }
 tracing = { workspace = true, features = ["std", "attributes"] }
-url = { workspace = true }
-rumqttc = { workspace = true }
 
 [dev-dependencies]
+async-trait = { workspace = true }
 temp-env = { workspace = true }
+url = { workspace = true }
 
 [lints]
 workspace = true
 
 [lib]
-test = false
 doctest = false

crates/audit/src/factory.rs

@@ -13,109 +13,65 @@
 // limitations under the License.
 
 use crate::AuditEntry;
-use async_trait::async_trait;
-use rustfs_config::AUDIT_DEFAULT_DIR;
-use rustfs_config::audit::{AUDIT_MQTT_KEYS, AUDIT_NATS_KEYS, AUDIT_PULSAR_KEYS, AUDIT_WEBHOOK_KEYS};
-use rustfs_ecstore::config::KVS;
-use rustfs_targets::{
-    Target,
-    config::{
-        build_mqtt_args, build_nats_args, build_pulsar_args, build_webhook_args, validate_mqtt_config, validate_nats_config,
-        validate_pulsar_config, validate_webhook_config,
-    },
-    error::TargetError,
-    target::TargetType,
-};
-use std::collections::HashSet;
+use rustfs_targets::catalog::builtin::builtin_audit_target_descriptors;
+use rustfs_targets::{BuiltinTargetDescriptor, TargetPluginDescriptor};
 
-/// Trait for creating targets from configuration
-#[async_trait]
-pub trait TargetFactory: Send + Sync {
-    /// Creates a target from configuration
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError>;
-
-    /// Validates target configuration
-    fn validate_config(&self, id: &str, config: &KVS) -> Result<(), TargetError>;
-
-    /// Returns a set of valid configuration field names for this target type.
-    /// This is used to filter environment variables.
-    fn get_valid_fields(&self) -> HashSet<String>;
+pub fn builtin_target_descriptors() -> Vec<BuiltinTargetDescriptor<AuditEntry>> {
+    builtin_audit_target_descriptors::<AuditEntry>()
 }
 
-/// Factory for creating Webhook targets
-pub struct WebhookTargetFactory;
-
-#[async_trait]
-impl TargetFactory for WebhookTargetFactory {
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let args = build_webhook_args(config, AUDIT_DEFAULT_DIR, TargetType::AuditLog)?;
-        let target = rustfs_targets::target::webhook::WebhookTarget::new(id, args)?;
-        Ok(Box::new(target))
-    }
-
-    fn validate_config(&self, _id: &str, config: &KVS) -> Result<(), TargetError> {
-        validate_webhook_config(config, AUDIT_DEFAULT_DIR)
-    }
-
-    fn get_valid_fields(&self) -> HashSet<String> {
-        AUDIT_WEBHOOK_KEYS.iter().map(|s| s.to_string()).collect()
-    }
+pub fn builtin_target_plugins() -> Vec<TargetPluginDescriptor<AuditEntry>> {
+    builtin_target_descriptors()
+        .into_iter()
+        .map(|descriptor| descriptor.plugin().clone())
+        .collect()
 }
 
-/// Factory for creating MQTT targets
-pub struct MQTTTargetFactory;
+#[cfg(test)]
+mod tests {
+    use super::builtin_target_descriptors;
+    use rustfs_config::audit::AUDIT_AMQP_KEYS;
+    use rustfs_config::{AMQP_EXCHANGE, AMQP_QUEUE_DIR, AMQP_ROUTING_KEY, AMQP_URL};
+    use rustfs_ecstore::config::KVS;
+    use rustfs_targets::target::ChannelTargetType;
 
-#[async_trait]
-impl TargetFactory for MQTTTargetFactory {
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let args = build_mqtt_args(config, AUDIT_DEFAULT_DIR, TargetType::AuditLog)?;
-        let target = rustfs_targets::target::mqtt::MQTTTarget::new(id, args)?;
-        Ok(Box::new(target))
+    fn amqp_base_config() -> KVS {
+        let mut config = KVS::new();
+        config.insert(AMQP_URL.to_string(), "amqp://127.0.0.1:5672/%2f".to_string());
+        config.insert(AMQP_EXCHANGE.to_string(), "rustfs.audit".to_string());
+        config.insert(AMQP_ROUTING_KEY.to_string(), "audit".to_string());
+        config.insert(AMQP_QUEUE_DIR.to_string(), String::new());
+        config
     }
 
-    fn validate_config(&self, _id: &str, config: &KVS) -> Result<(), TargetError> {
-        validate_mqtt_config(config)
+    #[test]
+    fn builtin_plugins_include_amqp_descriptor() {
+        let plugin = builtin_target_descriptors()
+            .into_iter()
+            .find(|plugin| plugin.plugin().target_type() == ChannelTargetType::Amqp.as_str())
+            .expect("amqp plugin should exist");
+
+        assert!(plugin.plugin().valid_fields().contains(&AMQP_URL));
+        assert!(plugin.plugin().valid_fields().contains(&AMQP_EXCHANGE));
+        assert!(plugin.plugin().valid_fields().contains(&AMQP_ROUTING_KEY));
+        assert_eq!(plugin.plugin().valid_fields().len(), AUDIT_AMQP_KEYS.len());
     }
 
-    fn get_valid_fields(&self) -> HashSet<String> {
-        AUDIT_MQTT_KEYS.iter().map(|s| s.to_string()).collect()
-    }
-}
-
-pub struct NATSTargetFactory;
-
-#[async_trait]
-impl TargetFactory for NATSTargetFactory {
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let args = build_nats_args(config, AUDIT_DEFAULT_DIR, TargetType::AuditLog)?;
-        let target = rustfs_targets::target::nats::NATSTarget::new(id, args)?;
-        Ok(Box::new(target))
-    }
-
-    fn validate_config(&self, _id: &str, config: &KVS) -> Result<(), TargetError> {
-        validate_nats_config(config, AUDIT_DEFAULT_DIR)
-    }
-
-    fn get_valid_fields(&self) -> HashSet<String> {
-        AUDIT_NATS_KEYS.iter().map(|s| s.to_string()).collect()
-    }
-}
-
-pub struct PulsarTargetFactory;
-
-#[async_trait]
-impl TargetFactory for PulsarTargetFactory {
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let args = build_pulsar_args(config, AUDIT_DEFAULT_DIR, TargetType::AuditLog)?;
-        let target = rustfs_targets::target::pulsar::PulsarTarget::new(id, args)?;
-        Ok(Box::new(target))
-    }
-
-    fn validate_config(&self, _id: &str, config: &KVS) -> Result<(), TargetError> {
-        validate_pulsar_config(config, AUDIT_DEFAULT_DIR)
-    }
-
-    fn get_valid_fields(&self) -> HashSet<String> {
-        AUDIT_PULSAR_KEYS.iter().map(|s| s.to_string()).collect()
+    #[test]
+    fn builtin_plugins_create_audit_amqp_target() {
+        let plugin = builtin_target_descriptors()
+            .into_iter()
+            .find(|plugin| plugin.plugin().target_type() == ChannelTargetType::Amqp.as_str())
+            .expect("amqp plugin should exist");
+
+        let target = plugin
+            .plugin()
+            .create_target("primary".to_string(), &amqp_base_config())
+            .expect("AMQP audit target should be created");
+
+        let target_id = target.id();
+        assert_eq!(target_id.id, "primary");
+        assert_eq!(target_id.name, "amqp");
+        assert!(target.store().is_none());
     }
 }

crates/audit/src/lib.rs

@@ -23,6 +23,7 @@ pub mod error;
 pub mod factory;
 pub mod global;
 pub mod observability;
+pub mod pipeline;
 pub mod registry;
 pub mod system;
 
@@ -30,5 +31,6 @@ pub use entity::{ApiDetails, AuditEntry, ObjectVersion};
 pub use error::{AuditError, AuditResult};
 pub use global::*;
 pub use observability::{AuditMetrics, AuditMetricsReport, PerformanceValidation};
+pub use pipeline::{AuditPipeline, AuditRuntimeFacade, AuditRuntimeView};
 pub use registry::AuditRegistry;
 pub use system::{AuditSystem, AuditTargetMetricSnapshot};

crates/audit/src/pipeline.rs

@@ -0,0 +1,355 @@
+//  Copyright 2024 RustFS Team
+//
+//  Licensed under the Apache License, Version 2.0 (the "License");
+//  you may not use this file except in compliance with the License.
+//  You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+//  Unless required by applicable law or agreed to in writing, software
+//  distributed under the License is distributed on an "AS IS" BASIS,
+//  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+//  See the License for the specific language governing permissions and
+//  limitations under the License.
+
+use crate::{AuditEntry, AuditResult, observability, system::AuditTargetMetricSnapshot};
+use rustfs_targets::{
+    BuiltinPluginRuntimeAdapter, PluginRuntimeAdapter, ReplayEvent, ReplayWorkerManager, RuntimeActivation, SharedTarget, Target,
+    target::EntityTarget,
+};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::sync::{Mutex, RwLock};
+use tracing::{error, info, warn};
+
+#[derive(Clone)]
+pub struct AuditPipeline {
+    registry: Arc<Mutex<crate::AuditRegistry>>,
+}
+
+impl AuditPipeline {
+    pub fn new(registry: Arc<Mutex<crate::AuditRegistry>>) -> Self {
+        Self { registry }
+    }
+
+    pub async fn dispatch(&self, entry: Arc<AuditEntry>) -> AuditResult<()> {
+        let start_time = std::time::Instant::now();
+
+        let targets: Vec<SharedTarget<AuditEntry>> = {
+            let registry = self.registry.lock().await;
+            let targets = registry.list_target_values();
+
+            if targets.is_empty() {
+                warn!("No audit targets configured for dispatch");
+                return Ok(());
+            }
+
+            targets
+        };
+
+        let mut tasks = Vec::new();
+
+        for target in targets {
+            let entity_target = EntityTarget {
+                object_name: entry.api.name.clone().unwrap_or_default(),
+                bucket_name: entry.api.bucket.clone().unwrap_or_default(),
+                event_name: entry.event,
+                data: (*entry).clone(),
+            };
+
+            let task = async move {
+                let result = target.save(Arc::new(entity_target)).await;
+                (target.id().to_string(), result)
+            };
+
+            tasks.push(task);
+        }
+
+        let results = futures::future::join_all(tasks).await;
+
+        let mut errors = Vec::new();
+        let mut success_count = 0;
+
+        for (target_key, result) in results {
+            match result {
+                Ok(_) => {
+                    success_count += 1;
+                    observability::record_target_success();
+                }
+                Err(e) => {
+                    error!(target_id = %target_key, error = %e, "Failed to dispatch audit log to target");
+                    errors.push(e);
+                    observability::record_target_failure();
+                }
+            }
+        }
+
+        let dispatch_time = start_time.elapsed();
+
+        if errors.is_empty() {
+            observability::record_audit_success(dispatch_time);
+        } else {
+            observability::record_audit_failure(dispatch_time);
+            warn!(
+                error_count = errors.len(),
+                success_count = success_count,
+                "Some audit targets failed to receive log entry"
+            );
+        }
+
+        Ok(())
+    }
+
+    pub async fn dispatch_batch(&self, entries: Vec<Arc<AuditEntry>>) -> AuditResult<()> {
+        let start_time = std::time::Instant::now();
+
+        let targets: Vec<SharedTarget<AuditEntry>> = {
+            let registry = self.registry.lock().await;
+            let targets = registry.list_target_values();
+
+            if targets.is_empty() {
+                warn!("No audit targets configured for batch dispatch");
+                return Ok(());
+            }
+
+            targets
+        };
+
+        let mut tasks = Vec::new();
+        for target in targets {
+            let entries_clone: Vec<_> = entries.iter().map(Arc::clone).collect();
+
+            let task = async move {
+                let mut success_count = 0;
+                let mut errors = Vec::new();
+                for entry in entries_clone {
+                    let entity_target = EntityTarget {
+                        object_name: entry.api.name.clone().unwrap_or_default(),
+                        bucket_name: entry.api.bucket.clone().unwrap_or_default(),
+                        event_name: entry.event,
+                        data: (*entry).clone(),
+                    };
+                    match target.save(Arc::new(entity_target)).await {
+                        Ok(_) => success_count += 1,
+                        Err(e) => errors.push(e),
+                    }
+                }
+                (target.id().to_string(), success_count, errors)
+            };
+            tasks.push(task);
+        }
+
+        let results = futures::future::join_all(tasks).await;
+        let mut total_success = 0;
+        let mut total_errors = 0;
+        for (_target_id, success_count, errors) in results {
+            total_success += success_count;
+            total_errors += errors.len();
+            for e in errors {
+                error!("Batch dispatch error: {:?}", e);
+            }
+        }
+
+        let dispatch_time = start_time.elapsed();
+        info!(
+            "Batch dispatched {} entries, success: {}, errors: {}, time: {:?}",
+            entries.len(),
+            total_success,
+            total_errors,
+            dispatch_time
+        );
+
+        Ok(())
+    }
+
+    pub async fn snapshot_target_metrics(&self) -> Vec<AuditTargetMetricSnapshot> {
+        let registry = self.registry.lock().await;
+        registry
+            .list_target_values()
+            .into_iter()
+            .map(|target| {
+                let delivery = target.delivery_snapshot();
+                AuditTargetMetricSnapshot {
+                    failed_messages: delivery.failed_messages,
+                    queue_length: delivery.queue_length,
+                    target_id: target.id().to_string(),
+                    total_messages: delivery.total_messages,
+                }
+            })
+            .collect()
+    }
+
+    pub async fn snapshot_target_health(&self) -> Vec<rustfs_targets::RuntimeTargetHealthSnapshot> {
+        let registry = self.registry.lock().await;
+        registry.runtime_manager().health_snapshots().await
+    }
+}
+
+#[derive(Clone)]
+pub struct AuditRuntimeView {
+    registry: Arc<Mutex<crate::AuditRegistry>>,
+}
+
+impl AuditRuntimeView {
+    pub fn new(registry: Arc<Mutex<crate::AuditRegistry>>) -> Self {
+        Self { registry }
+    }
+
+    pub async fn list_targets(&self) -> Vec<String> {
+        let registry = self.registry.lock().await;
+        registry.list_targets()
+    }
+
+    pub async fn get_target_values(&self) -> Vec<rustfs_targets::SharedTarget<AuditEntry>> {
+        let registry = self.registry.lock().await;
+        registry.list_target_values()
+    }
+
+    pub async fn get_target(&self, target_id: &str) -> Option<String> {
+        let registry = self.registry.lock().await;
+        registry.get_target(target_id).map(|target| target.id().to_string())
+    }
+
+    pub async fn enable_target(&self, target_id: &str) -> AuditResult<()> {
+        let registry = self.registry.lock().await;
+        if registry.get_target(target_id).is_some() {
+            info!(target_id = %target_id, "Target enabled");
+            Ok(())
+        } else {
+            Err(crate::AuditError::Configuration(format!("Target not found: {target_id}"), None))
+        }
+    }
+
+    pub async fn disable_target(&self, target_id: &str) -> AuditResult<()> {
+        let registry = self.registry.lock().await;
+        if registry.get_target(target_id).is_some() {
+            info!(target_id = %target_id, "Target disabled");
+            Ok(())
+        } else {
+            Err(crate::AuditError::Configuration(format!("Target not found: {target_id}"), None))
+        }
+    }
+
+    pub async fn remove_target(&self, target_id: &str) -> AuditResult<()> {
+        let mut registry = self.registry.lock().await;
+        if registry.remove_target(target_id).await.is_some() {
+            info!(target_id = %target_id, "Target removed");
+            Ok(())
+        } else {
+            Err(crate::AuditError::Configuration(format!("Target not found: {target_id}"), None))
+        }
+    }
+
+    pub async fn upsert_target(&self, target_id: String, target: Box<dyn Target<AuditEntry> + Send + Sync>) -> AuditResult<()> {
+        if let Err(err) = target.init().await {
+            return Err(crate::AuditError::Target(err));
+        }
+
+        let shared_target: SharedTarget<AuditEntry> = Arc::from(target);
+        let mut registry = self.registry.lock().await;
+        let _ = registry.remove_target(&target_id).await;
+        registry.add_shared_target(target_id.clone(), shared_target);
+        info!(target_id = %target_id, "Target upserted");
+        Ok(())
+    }
+}
+
+#[derive(Clone)]
+pub struct AuditRuntimeFacade {
+    registry: Arc<Mutex<crate::AuditRegistry>>,
+    replay_workers: Arc<RwLock<ReplayWorkerManager>>,
+    runtime_adapter: Arc<dyn PluginRuntimeAdapter<AuditEntry>>,
+}
+
+impl AuditRuntimeFacade {
+    pub fn new(registry: Arc<Mutex<crate::AuditRegistry>>, replay_workers: Arc<RwLock<ReplayWorkerManager>>) -> Self {
+        let runtime_adapter = BuiltinPluginRuntimeAdapter::new(
+            Arc::new(move |event: ReplayEvent<AuditEntry>| {
+                Box::pin(async move {
+                    match event {
+                        ReplayEvent::Delivered { key, target } => {
+                            info!("Successfully sent audit entry, target: {}, key: {}", target.id(), key.to_string());
+                            observability::record_target_success();
+                        }
+                        ReplayEvent::RetryableError { error, target, .. } => match error {
+                            rustfs_targets::TargetError::NotConnected => {
+                                warn!("Target {} not connected, retrying...", target.id());
+                            }
+                            rustfs_targets::TargetError::Timeout(_) => {
+                                warn!("Timeout sending to target {}, retrying...", target.id());
+                            }
+                            _ => {}
+                        },
+                        ReplayEvent::Dropped { reason, target, .. } => {
+                            warn!("Dropped queued payload for target {}: {}", target.id(), reason);
+                            observability::record_target_failure();
+                        }
+                        ReplayEvent::PermanentFailure { error, target, .. } => {
+                            error!("Permanent error for target {}: {}", target.id(), error);
+                            target.record_final_failure();
+                            observability::record_target_failure();
+                        }
+                        ReplayEvent::RetryExhausted { key, target } => {
+                            warn!("Max retries exceeded for key {}, target: {}, skipping", key.to_string(), target.id());
+                            target.record_final_failure();
+                            observability::record_target_failure();
+                        }
+                        ReplayEvent::UnreadableEntry { key, error, target } => {
+                            warn!("Skipping unreadable audit store entry {} for target {}: {}", key, target.id(), error);
+                        }
+                    }
+                })
+            }),
+            Arc::new(|target_id, has_replay| {
+                if has_replay {
+                    info!(target_id = %target_id, "Audit stream processing started");
+                } else {
+                    info!(target_id = %target_id, "No store configured, skip audit stream processing");
+                }
+            }),
+            None,
+            Duration::from_millis(500),
+            Duration::from_millis(500),
+            "Stopping audit stream",
+        );
+
+        Self {
+            registry,
+            replay_workers,
+            runtime_adapter: Arc::new(runtime_adapter),
+        }
+    }
+
+    pub async fn replace_targets(&self, activation: RuntimeActivation<AuditEntry>) -> AuditResult<()> {
+        let mut registry = self.registry.lock().await;
+        let mut replay_workers = self.replay_workers.write().await;
+        self.runtime_adapter
+            .replace_runtime_targets(registry.runtime_manager_mut(), &mut replay_workers, activation)
+            .await
+            .map_err(crate::AuditError::Target)?;
+        Ok(())
+    }
+
+    pub async fn shutdown_runtime(
+        &self,
+        registry: &mut crate::AuditRegistry,
+        replay_workers: &mut ReplayWorkerManager,
+    ) -> AuditResult<()> {
+        self.runtime_adapter
+            .shutdown(registry.runtime_manager_mut(), replay_workers)
+            .await
+            .map_err(crate::AuditError::Target)
+    }
+
+    pub async fn activate_targets_with_replay(
+        &self,
+        targets: Vec<Box<dyn Target<AuditEntry> + Send + Sync>>,
+    ) -> RuntimeActivation<AuditEntry> {
+        self.runtime_adapter.activate_with_replay(targets).await
+    }
+
+    pub async fn stop_replay_workers(&self) {
+        let mut replay_workers = self.replay_workers.write().await;
+        self.runtime_adapter.stop_replay_workers(&mut replay_workers).await;
+    }
+}

crates/audit/src/registry.rs

@@ -12,26 +12,19 @@
 //  See the License for the specific language governing permissions and
 //  limitations under the License.
 
-use crate::{
-    AuditEntry, AuditError, AuditResult,
-    factory::{MQTTTargetFactory, NATSTargetFactory, PulsarTargetFactory, TargetFactory, WebhookTargetFactory},
-};
-use futures::StreamExt;
-use futures::stream::FuturesUnordered;
-use hashbrown::HashMap;
+use crate::{AuditEntry, AuditError, AuditResult, factory::builtin_target_plugins};
 use rustfs_config::audit::AUDIT_ROUTE_PREFIX;
 use rustfs_ecstore::config::{Config, KVS};
 use rustfs_targets::arn::TargetID;
-use rustfs_targets::{Target, TargetError, config::collect_target_configs, target::ChannelTargetType};
-use std::sync::Arc;
-use tracing::{error, info};
+use rustfs_targets::{SharedTarget, Target, TargetError, TargetPluginRegistry, TargetRuntimeManager};
+use tracing::info;
 
 /// Registry for managing audit targets
 pub struct AuditRegistry {
     /// Storage for created targets
-    targets: HashMap<String, Box<dyn Target<AuditEntry> + Send + Sync>>,
-    /// Factories for creating targets
-    factories: HashMap<String, Box<dyn TargetFactory>>,
+    targets: TargetRuntimeManager<AuditEntry>,
+    /// Registered plugins for creating targets
+    plugins: TargetPluginRegistry<AuditEntry>,
 }
 
 impl Default for AuditRegistry {
@@ -43,27 +36,17 @@ impl Default for AuditRegistry {
 impl AuditRegistry {
     /// Creates a new AuditRegistry
     pub fn new() -> Self {
-        let mut registry = AuditRegistry {
-            factories: HashMap::new(),
-            targets: HashMap::new(),
-        };
+        let mut plugins = TargetPluginRegistry::new();
+        plugins.register_all(builtin_target_plugins());
 
-        // Register built-in factories
-        registry.register(ChannelTargetType::Webhook.as_str(), Box::new(WebhookTargetFactory));
-        registry.register(ChannelTargetType::Mqtt.as_str(), Box::new(MQTTTargetFactory));
-        registry.register(ChannelTargetType::Nats.as_str(), Box::new(NATSTargetFactory));
-        registry.register(ChannelTargetType::Pulsar.as_str(), Box::new(PulsarTargetFactory));
-
-        registry
+        AuditRegistry {
+            targets: TargetRuntimeManager::new(),
+            plugins,
+        }
     }
 
-    /// Registers a new factory for a target type
-    ///
-    /// # Arguments
-    /// * `target_type` - The type of the target (e.g., "webhook", "mqtt").
-    /// * `factory` - The factory instance to create targets of this type.
-    pub fn register(&mut self, target_type: &str, factory: Box<dyn TargetFactory>) {
-        self.factories.insert(target_type.to_string(), factory);
+    pub fn supports_target_type(&self, target_type: &str) -> bool {
+        self.plugins.supports_target_type(target_type)
     }
 
     /// Creates a target of the specified type with the given ID and configuration
@@ -81,16 +64,7 @@ impl AuditRegistry {
         id: String,
         config: &KVS,
     ) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let factory = self
-            .factories
-            .get(target_type)
-            .ok_or_else(|| TargetError::Configuration(format!("Unknown target type: {target_type}")))?;
-
-        // Validate configuration before creating target
-        factory.validate_config(&id, config)?;
-
-        // Create target
-        factory.create_target(id, config).await
+        self.plugins.create_target(target_type, id, config)
     }
 
     /// Creates all targets from a configuration
@@ -106,37 +80,10 @@ impl AuditRegistry {
         &self,
         config: &Config,
     ) -> AuditResult<Vec<Box<dyn Target<AuditEntry> + Send + Sync>>> {
-        let mut tasks = FuturesUnordered::new();
-        for (target_type, factory) in &self.factories {
-            tracing::Span::current().record("target_type", target_type.as_str());
-            info!("Start working on target types...");
-            let valid_fields = factory.get_valid_fields();
-            for (id, merged_config) in collect_target_configs(config, AUDIT_ROUTE_PREFIX, target_type, &valid_fields) {
-                info!(instance_id = %id, "Target is enabled, ready to create a task");
-                let tid = id.clone();
-                let merged_config_arc = Arc::new(merged_config);
-                tasks.push(async move {
-                    let result = factory.create_target(tid.clone(), &merged_config_arc).await;
-                    (tid, result)
-                });
-            }
-        }
-
-        let mut successful_targets = Vec::new();
-        while let Some((id, result)) = tasks.next().await {
-            match result {
-                Ok(target) => {
-                    info!(target_type = %target.id().name, instance_id = %id, "Create a target successfully");
-                    successful_targets.push(target);
-                }
-                Err(e) => {
-                    error!(instance_id = %id, error = %e, "Failed to create a target");
-                }
-            }
-        }
-
-        info!(count = successful_targets.len(), "All target processing completed");
-        Ok(successful_targets)
+        self.plugins
+            .create_targets_from_config(config, AUDIT_ROUTE_PREFIX)
+            .await
+            .map_err(AuditError::from)
     }
 
     /// Adds a target to the registry
@@ -144,8 +91,14 @@ impl AuditRegistry {
     /// # Arguments
     /// * `id` - The identifier for the target.
     /// * `target` - The target instance to be added.
-    pub fn add_target(&mut self, id: String, target: Box<dyn Target<AuditEntry> + Send + Sync>) {
-        self.targets.insert(id, target);
+    pub fn add_target(&mut self, _id: String, target: Box<dyn Target<AuditEntry> + Send + Sync>) {
+        debug_assert_eq!(_id, target.id().to_string());
+        self.targets.add_boxed(target);
+    }
+
+    pub fn add_shared_target(&mut self, _id: String, target: SharedTarget<AuditEntry>) {
+        debug_assert_eq!(_id, target.id().to_string());
+        self.targets.add_arc(target);
     }
 
     /// Removes a target from the registry
@@ -155,8 +108,8 @@ impl AuditRegistry {
     ///
     /// # Returns
     /// * `Option<Box<dyn Target<AuditEntry> + Send + Sync>>` - The removed target if it existed.
-    pub fn remove_target(&mut self, id: &str) -> Option<Box<dyn Target<AuditEntry> + Send + Sync>> {
-        self.targets.remove(id)
+    pub async fn remove_target(&mut self, id: &str) -> Option<rustfs_targets::SharedTarget<AuditEntry>> {
+        self.targets.remove_and_close(id).await
     }
 
     /// Gets a target from the registry
@@ -166,13 +119,21 @@ impl AuditRegistry {
     ///
     /// # Returns
     /// * `Option<&(dyn Target<AuditEntry> + Send + Sync)>` - The target if it exists.
-    pub fn get_target(&self, id: &str) -> Option<&(dyn Target<AuditEntry> + Send + Sync)> {
-        self.targets.get(id).map(|t| t.as_ref())
+    pub fn get_target(&self, id: &str) -> Option<rustfs_targets::SharedTarget<AuditEntry>> {
+        self.targets.get(id)
     }
 
     /// Lists cloned target values for runtime inspection without exposing mutable registry access.
-    pub fn list_target_values(&self) -> Vec<Box<dyn Target<AuditEntry> + Send + Sync>> {
-        self.targets.values().map(|target| target.clone_dyn()).collect()
+    pub fn list_target_values(&self) -> Vec<rustfs_targets::SharedTarget<AuditEntry>> {
+        self.targets.values()
+    }
+
+    pub fn runtime_manager(&self) -> &TargetRuntimeManager<AuditEntry> {
+        &self.targets
+    }
+
+    pub fn runtime_manager_mut(&mut self) -> &mut TargetRuntimeManager<AuditEntry> {
+        &mut self.targets
     }
 
     /// Lists all target IDs
@@ -180,7 +141,7 @@ impl AuditRegistry {
     /// # Returns
     /// * `Vec<String>` - A vector of all target IDs in the registry.
     pub fn list_targets(&self) -> Vec<String> {
-        self.targets.keys().cloned().collect()
+        self.targets.keys()
     }
 
     /// Closes all targets and clears the registry
@@ -188,20 +149,23 @@ impl AuditRegistry {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure.
     pub async fn close_all(&mut self) -> AuditResult<()> {
-        let mut errors = Vec::new();
+        let mut first_error = None;
 
-        for (id, target) in self.targets.drain() {
-            if let Err(e) = target.close().await {
-                error!(target_id = %id, error = %e, "Failed to close audit target");
-                errors.push(e);
+        for target_id in self.targets.keys() {
+            if let Some(target) = self.targets.remove(&target_id)
+                && let Err(err) = target.close().await
+            {
+                tracing::error!(target_id = %target_id, error = %err, "Failed to close target during shutdown");
+                if first_error.is_none() {
+                    first_error = Some(err);
+                }
             }
         }
 
-        if !errors.is_empty() {
-            return Err(AuditError::Target(errors.into_iter().next().unwrap()));
+        match first_error {
+            Some(err) => Err(AuditError::Target(err)),
+            None => Ok(()),
         }
-
-        Ok(())
     }
 
     /// Creates a unique key for a target based on its type and ID
@@ -276,7 +240,107 @@ impl AuditRegistry {
         target: Box<dyn Target<AuditEntry> + Send + Sync>,
     ) -> AuditResult<()> {
         let key = self.create_key(target_type, target_id);
-        self.targets.insert(key, target);
+        debug_assert_eq!(key, target.id().to_string());
+        self.targets.add_boxed(target);
         Ok(())
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::AuditRegistry;
+    use crate::{AuditEntry, AuditError};
+    use rustfs_targets::arn::TargetID;
+    use rustfs_targets::store::{Key, Store};
+    use rustfs_targets::target::{ChannelTargetType, EntityTarget, QueuedPayload, QueuedPayloadMeta};
+    use rustfs_targets::{StoreError, Target, TargetError};
+    use std::sync::Arc;
+    use std::sync::atomic::{AtomicUsize, Ordering};
+
+    #[derive(Clone)]
+    struct CloseTestTarget {
+        id: TargetID,
+        close_calls: Arc<AtomicUsize>,
+        fail_on_close: bool,
+    }
+
+    impl CloseTestTarget {
+        fn new(id: TargetID, close_calls: Arc<AtomicUsize>, fail_on_close: bool) -> Self {
+            Self {
+                id,
+                close_calls,
+                fail_on_close,
+            }
+        }
+    }
+
+    #[async_trait::async_trait]
+    impl Target<AuditEntry> for CloseTestTarget {
+        fn id(&self) -> TargetID {
+            self.id.clone()
+        }
+
+        async fn is_active(&self) -> Result<bool, TargetError> {
+            Ok(true)
+        }
+
+        async fn save(&self, _event: Arc<EntityTarget<AuditEntry>>) -> Result<(), TargetError> {
+            Ok(())
+        }
+
+        async fn send_raw_from_store(&self, _key: Key, _body: Vec<u8>, _meta: QueuedPayloadMeta) -> Result<(), TargetError> {
+            Ok(())
+        }
+
+        async fn close(&self) -> Result<(), TargetError> {
+            self.close_calls.fetch_add(1, Ordering::SeqCst);
+            if self.fail_on_close {
+                Err(TargetError::Unknown("close failed".to_string()))
+            } else {
+                Ok(())
+            }
+        }
+
+        fn store(&self) -> Option<&(dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send + Sync)> {
+            None
+        }
+
+        fn clone_dyn(&self) -> Box<dyn Target<AuditEntry> + Send + Sync> {
+            Box::new(self.clone())
+        }
+
+        fn is_enabled(&self) -> bool {
+            true
+        }
+    }
+
+    #[test]
+    fn registry_registers_amqp_factory() {
+        let registry = AuditRegistry::new();
+
+        assert!(registry.supports_target_type(ChannelTargetType::Amqp.as_str()));
+    }
+
+    #[tokio::test]
+    async fn close_all_returns_first_error_and_clears_targets() {
+        let mut registry = AuditRegistry::new();
+        let ok_calls = Arc::new(AtomicUsize::new(0));
+        let fail_calls = Arc::new(AtomicUsize::new(0));
+
+        let ok_id = TargetID::new("ok".to_string(), "webhook".to_string());
+        let fail_id = TargetID::new("fail".to_string(), "webhook".to_string());
+
+        registry.add_target(ok_id.to_string(), Box::new(CloseTestTarget::new(ok_id, Arc::clone(&ok_calls), false)));
+        registry.add_target(
+            fail_id.to_string(),
+            Box::new(CloseTestTarget::new(fail_id, Arc::clone(&fail_calls), true)),
+        );
+
+        let result = registry.close_all().await;
+
+        assert!(matches!(result, Err(AuditError::Target(TargetError::Unknown(_)))));
+        assert_eq!(ok_calls.load(Ordering::SeqCst), 1);
+        assert_eq!(fail_calls.load(Ordering::SeqCst), 1);
+        assert!(registry.list_targets().is_empty());
+    }
+}

crates/audit/src/system.rs

@@ -12,16 +12,14 @@
 //  See the License for the specific language governing permissions and
 //  limitations under the License.
 
-use crate::{AuditEntry, AuditError, AuditRegistry, AuditResult, observability};
-use hashbrown::HashMap;
-use rustfs_ecstore::config::Config;
-use rustfs_targets::{
-    StoreError, Target, TargetError,
-    store::{Key, Store},
-    target::{EntityTarget, QueuedPayload},
+use crate::{
+    AuditEntry, AuditError, AuditRegistry, AuditResult, observability,
+    pipeline::{AuditPipeline, AuditRuntimeFacade, AuditRuntimeView},
 };
+use rustfs_ecstore::config::Config;
+use rustfs_targets::{ReplayWorkerManager, Target};
 use std::sync::Arc;
-use tokio::sync::{Mutex, RwLock, mpsc};
+use tokio::sync::{Mutex, RwLock};
 use tracing::{error, info, warn};
 
 #[derive(Debug, Clone, Default, PartialEq, Eq)]
@@ -49,7 +47,7 @@ pub struct AuditSystem {
     state: Arc<RwLock<AuditSystemState>>,
     config: Arc<RwLock<Option<Config>>>,
     /// Cancellation senders for active audit stream tasks (target_id -> cancel tx)
-    stream_cancellers: Arc<RwLock<HashMap<String, mpsc::Sender<()>>>>,
+    stream_cancellers: Arc<RwLock<ReplayWorkerManager>>,
 }
 
 impl Default for AuditSystem {
@@ -59,16 +57,68 @@ impl Default for AuditSystem {
 }
 
 impl AuditSystem {
+    fn pipeline(&self) -> AuditPipeline {
+        AuditPipeline::new(self.registry.clone())
+    }
+
+    fn runtime_view(&self) -> AuditRuntimeView {
+        AuditRuntimeView::new(self.registry.clone())
+    }
+
+    fn runtime_facade(&self) -> AuditRuntimeFacade {
+        AuditRuntimeFacade::new(self.registry.clone(), self.stream_cancellers.clone())
+    }
+
     /// Creates a new audit system
     pub fn new() -> Self {
         Self {
             registry: Arc::new(Mutex::new(AuditRegistry::new())),
             state: Arc::new(RwLock::new(AuditSystemState::Stopped)),
             config: Arc::new(RwLock::new(None)),
-            stream_cancellers: Arc::new(RwLock::new(HashMap::new())),
+            stream_cancellers: Arc::new(RwLock::new(ReplayWorkerManager::new())),
         }
     }
 
+    async fn create_targets_from_config(&self, config: &Config) -> AuditResult<Vec<Box<dyn Target<AuditEntry> + Send + Sync>>> {
+        let registry = self.registry.lock().await;
+        registry.create_audit_targets_from_config(config).await
+    }
+
+    async fn clear_runtime_targets(&self) -> AuditResult<()> {
+        {
+            let mut registry = self.registry.lock().await;
+            let mut replay_workers = self.stream_cancellers.write().await;
+            self.runtime_facade()
+                .shutdown_runtime(&mut registry, &mut replay_workers)
+                .await?;
+        }
+
+        let mut state = self.state.write().await;
+        *state = AuditSystemState::Stopped;
+        Ok(())
+    }
+
+    async fn commit_runtime_targets(
+        &self,
+        targets: Vec<Box<dyn Target<AuditEntry> + Send + Sync>>,
+        final_state: AuditSystemState,
+    ) -> AuditResult<()> {
+        if targets.is_empty() {
+            info!("No enabled audit targets found, keeping audit system stopped");
+            self.clear_runtime_targets().await?;
+            return Ok(());
+        }
+
+        info!(target_count = targets.len(), "Created audit targets successfully");
+
+        let activation = self.runtime_facade().activate_targets_with_replay(targets).await;
+        self.runtime_facade().replace_targets(activation).await?;
+
+        let mut state = self.state.write().await;
+        *state = final_state;
+        Ok(())
+    }
+
     /// Starts the audit system with the given configuration
     ///
     /// # Arguments
@@ -103,31 +153,14 @@ impl AuditSystem {
             *config_guard = Some(config.clone());
         }
 
-        // Create targets from configuration
-        let mut registry = self.registry.lock().await;
-        match registry.create_audit_targets_from_config(&config).await {
+        match self.create_targets_from_config(&config).await {
             Ok(targets) => {
-                if targets.is_empty() {
-                    info!("No enabled audit targets found, keeping audit system stopped");
-                    drop(registry);
-                    return Ok(());
-                }
-
                 {
                     let mut state = self.state.write().await;
                     *state = AuditSystemState::Starting;
                 }
 
-                info!(target_count = targets.len(), "Created audit targets successfully");
-
-                // Initialize all targets
-                for target in targets {
-                    self.init_and_register_target(target, &mut registry).await;
-                }
-
-                // Update state to running
-                let mut state = self.state.write().await;
-                *state = AuditSystemState::Running;
+                self.commit_runtime_targets(targets, AuditSystemState::Running).await?;
                 info!("Audit system started successfully");
                 Ok(())
             }
@@ -207,18 +240,10 @@ impl AuditSystem {
         info!("Stopping audit system");
 
         // Stop all stream tasks first
-        self.stop_all_streams().await;
-
-        // Close all targets
-        let mut registry = self.registry.lock().await;
-        if let Err(e) = registry.close_all().await {
+        if let Err(e) = self.clear_runtime_targets().await {
             error!(error = %e, "Failed to close some audit targets");
         }
 
-        // Update state to stopped
-        let mut state = self.state.write().await;
-        *state = AuditSystemState::Stopped;
-
         // Clear configuration
         let mut config_guard = self.config.write().await;
         *config_guard = None;
@@ -248,8 +273,6 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn dispatch(&self, entry: Arc<AuditEntry>) -> AuditResult<()> {
-        let start_time = std::time::Instant::now();
-
         let state = self.state.read().await;
 
         match *state {
@@ -262,77 +285,7 @@ impl AuditSystem {
             }
         }
         drop(state);
-
-        // Collect cloned targets under lock, then dispatch without holding it
-        let targets: Vec<(String, Box<dyn Target<AuditEntry> + Send + Sync>)> = {
-            let registry = self.registry.lock().await;
-            let target_keys = registry.list_targets();
-
-            if target_keys.is_empty() {
-                warn!("No audit targets configured for dispatch");
-                return Ok(());
-            }
-
-            target_keys
-                .into_iter()
-                .filter_map(|key| registry.get_target(&key).map(|t| (key, t.clone_dyn())))
-                .collect()
-        };
-
-        // Dispatch to all targets concurrently (no lock held)
-        let mut tasks = Vec::new();
-
-        for (target_key, target) in targets {
-            let entity_target = EntityTarget {
-                object_name: entry.api.name.clone().unwrap_or_default(),
-                bucket_name: entry.api.bucket.clone().unwrap_or_default(),
-                event_name: entry.event,
-                data: (*entry).clone(),
-            };
-
-            let task = async move {
-                let result = target.save(Arc::new(entity_target)).await;
-                (target_key, result)
-            };
-
-            tasks.push(task);
-        }
-
-        // Execute all dispatch tasks
-        let results = futures::future::join_all(tasks).await;
-
-        let mut errors = Vec::new();
-        let mut success_count = 0;
-
-        for (target_key, result) in results {
-            match result {
-                Ok(_) => {
-                    success_count += 1;
-                    observability::record_target_success();
-                }
-                Err(e) => {
-                    error!(target_id = %target_key, error = %e, "Failed to dispatch audit log to target");
-                    errors.push(e);
-                    observability::record_target_failure();
-                }
-            }
-        }
-
-        let dispatch_time = start_time.elapsed();
-
-        if errors.is_empty() {
-            observability::record_audit_success(dispatch_time);
-        } else {
-            observability::record_audit_failure(dispatch_time);
-            // Log errors but don't fail the entire dispatch
-            warn!(
-                error_count = errors.len(),
-                success_count = success_count,
-                "Some audit targets failed to receive log entry"
-            );
-        }
-
-        Ok(())
+        self.pipeline().dispatch(entry).await
     }
 
     /// Dispatches a batch of audit log entries to all active targets
@@ -343,239 +296,12 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn dispatch_batch(&self, entries: Vec<Arc<AuditEntry>>) -> AuditResult<()> {
-        let start_time = std::time::Instant::now();
-
         let state = self.state.read().await;
         if *state != AuditSystemState::Running {
             return Err(AuditError::NotInitialized("Audit system is not running".to_string()));
         }
         drop(state);
-
-        // Collect targets under lock, then dispatch without holding it
-        let targets: Vec<(String, Box<dyn Target<AuditEntry> + Send + Sync>)> = {
-            let registry = self.registry.lock().await;
-            let target_keys = registry.list_targets();
-
-            if target_keys.is_empty() {
-                warn!("No audit targets configured for batch dispatch");
-                return Ok(());
-            }
-
-            target_keys
-                .into_iter()
-                .filter_map(|key| registry.get_target(&key).map(|t| (key, t.clone_dyn())))
-                .collect()
-        };
-
-        let mut tasks = Vec::new();
-        for (target_key, target) in targets {
-            let entries_clone: Vec<_> = entries.iter().map(Arc::clone).collect();
-            let target_key_clone = target_key.clone();
-
-            let task = async move {
-                let mut success_count = 0;
-                let mut errors = Vec::new();
-                for entry in entries_clone {
-                    let entity_target = EntityTarget {
-                        object_name: entry.api.name.clone().unwrap_or_default(),
-                        bucket_name: entry.api.bucket.clone().unwrap_or_default(),
-                        event_name: entry.event,
-                        data: (*entry).clone(),
-                    };
-                    match target.save(Arc::new(entity_target)).await {
-                        Ok(_) => success_count += 1,
-                        Err(e) => errors.push(e),
-                    }
-                }
-                (target_key_clone, success_count, errors)
-            };
-            tasks.push(task);
-        }
-
-        let results = futures::future::join_all(tasks).await;
-        let mut total_success = 0;
-        let mut total_errors = 0;
-        for (_target_id, success_count, errors) in results {
-            total_success += success_count;
-            total_errors += errors.len();
-            for e in errors {
-                error!("Batch dispatch error: {:?}", e);
-            }
-        }
-
-        let dispatch_time = start_time.elapsed();
-        info!(
-            "Batch dispatched {} entries, success: {}, errors: {}, time: {:?}",
-            entries.len(),
-            total_success,
-            total_errors,
-            dispatch_time
-        );
-
-        Ok(())
-    }
-
-    /// Stops all active audit stream tasks by sending cancellation signals.
-    async fn stop_all_streams(&self) {
-        let mut cancellers = self.stream_cancellers.write().await;
-        for (target_id, cancel_tx) in cancellers.drain() {
-            info!(target_id = %target_id, "Stopping audit stream");
-            let _ = cancel_tx.send(()).await;
-        }
-    }
-
-    /// Initializes a single target: runs init(), starts stream if store is present,
-    /// and adds it to the registry. For store-backed targets, registration and stream
-    /// startup proceed even if init() fails so queued entries can be drained later.
-    async fn init_and_register_target(
-        &self,
-        target: Box<dyn Target<AuditEntry> + Send + Sync>,
-        registry: &mut AuditRegistry,
-    ) -> Option<String> {
-        let target_id = target.id().to_string();
-        let has_store = target.store().is_some();
-
-        if let Err(e) = target.init().await {
-            error!(target_id = %target_id, error = %e, "Failed to initialize audit target");
-            // Non-store targets: init failure is fatal.
-            if !has_store {
-                return None;
-            }
-            // Store-backed targets: still register and start the stream so queued
-            // entries can be drained when connectivity recovers.
-            warn!(
-                target_id = %target_id,
-                "Proceeding with store-backed audit target despite init failure"
-            );
-        }
-
-        if target.is_enabled() {
-            if let Some(store) = target.store() {
-                info!(target_id = %target_id, "Start audit stream processing for target");
-                let store_clone: Box<dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send> = store.boxed_clone();
-                let target_arc: Arc<dyn Target<AuditEntry> + Send + Sync> = Arc::from(target.clone_dyn());
-                let cancel_tx = self.start_audit_stream_with_batching(store_clone, target_arc);
-
-                self.stream_cancellers.write().await.insert(target_id.clone(), cancel_tx);
-                info!(target_id = %target_id, "Audit stream processing started");
-            } else {
-                info!(target_id = %target_id, "No store configured, skip audit stream processing");
-            }
-        } else {
-            info!(target_id = %target_id, "Target disabled, skip audit stream processing");
-        }
-
-        registry.add_target(target_id.clone(), target);
-        Some(target_id)
-    }
-
-    /// Starts the audit stream processing for a target with batching and retry logic
-    ///
-    /// # Arguments
-    /// * `store` - The store from which to read audit entries
-    /// * `target` - The target to which audit entries will be sent
-    ///
-    /// This function spawns a background task that continuously reads audit entries from the provided store
-    /// and attempts to send them to the specified target. It implements retry logic with exponential backoff
-    fn start_audit_stream_with_batching(
-        &self,
-        store: Box<dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send>,
-        target: Arc<dyn Target<AuditEntry> + Send + Sync>,
-    ) -> mpsc::Sender<()> {
-        let (cancel_tx, mut cancel_rx) = mpsc::channel(1);
-        let state = self.state.clone();
-
-        tokio::spawn(async move {
-            use std::time::Duration;
-            use tokio::time::sleep;
-
-            info!("Starting audit stream for target: {}", target.id());
-
-            const MAX_RETRIES: usize = 5;
-            const BASE_RETRY_DELAY: Duration = Duration::from_secs(2);
-
-            loop {
-                // Check for cancellation signal
-                if cancel_rx.try_recv().is_ok() {
-                    info!("Audit stream cancelled for target: {}", target.id());
-                    break;
-                }
-
-                match *state.read().await {
-                    AuditSystemState::Running | AuditSystemState::Paused | AuditSystemState::Starting => {}
-                    _ => {
-                        info!("Audit stream stopped for target: {}", target.id());
-                        break;
-                    }
-                }
-
-                let keys: Vec<Key> = store.list();
-                if keys.is_empty() {
-                    tokio::select! {
-                        _ = sleep(Duration::from_millis(500)) => {},
-                        _ = cancel_rx.recv() => {
-                            info!("Audit stream cancelled during idle for target: {}", target.id());
-                            return;
-                        }
-                    }
-                    continue;
-                }
-
-                for key in keys {
-                    if cancel_rx.try_recv().is_ok() {
-                        info!("Audit stream cancelled during processing for target: {}", target.id());
-                        return;
-                    }
-
-                    let mut retries = 0usize;
-                    let mut success = false;
-
-                    while retries < MAX_RETRIES && !success {
-                        match target.send_from_store(key.clone()).await {
-                            Ok(_) => {
-                                info!("Successfully sent audit entry, target: {}, key: {}", target.id(), key.to_string());
-                                observability::record_target_success();
-                                success = true;
-                            }
-                            Err(e) => {
-                                match &e {
-                                    TargetError::NotConnected => {
-                                        warn!("Target {} not connected, retrying...", target.id());
-                                    }
-                                    TargetError::Timeout(_) => {
-                                        warn!("Timeout sending to target {}, retrying...", target.id());
-                                    }
-                                    TargetError::Dropped(reason) => {
-                                        warn!("Dropped queued payload for target {}: {}", target.id(), reason);
-                                        observability::record_target_failure();
-                                        break;
-                                    }
-                                    _ => {
-                                        error!("Permanent error for target {}: {}", target.id(), e);
-                                        target.record_final_failure();
-                                        observability::record_target_failure();
-                                        break;
-                                    }
-                                }
-                                retries += 1;
-                                let backoff = BASE_RETRY_DELAY * (1 << retries);
-                                sleep(backoff).await;
-                            }
-                        }
-                    }
-
-                    if retries >= MAX_RETRIES && !success {
-                        warn!("Max retries exceeded for key {}, target: {}, skipping", key.to_string(), target.id());
-                        target.record_final_failure();
-                        observability::record_target_failure();
-                    }
-                }
-
-                sleep(Duration::from_millis(100)).await;
-            }
-        });
-
-        cancel_tx
+        self.pipeline().dispatch_batch(entries).await
     }
 
     /// Enables a specific target
@@ -586,15 +312,7 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn enable_target(&self, target_id: &str) -> AuditResult<()> {
-        // This would require storing enabled/disabled state per target
-        // For now, just check if target exists
-        let registry = self.registry.lock().await;
-        if registry.get_target(target_id).is_some() {
-            info!(target_id = %target_id, "Target enabled");
-            Ok(())
-        } else {
-            Err(AuditError::Configuration(format!("Target not found: {target_id}"), None))
-        }
+        self.runtime_view().enable_target(target_id).await
     }
 
     /// Disables a specific target
@@ -605,15 +323,7 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn disable_target(&self, target_id: &str) -> AuditResult<()> {
-        // This would require storing enabled/disabled state per target
-        // For now, just check if target exists
-        let registry = self.registry.lock().await;
-        if registry.get_target(target_id).is_some() {
-            info!(target_id = %target_id, "Target disabled");
-            Ok(())
-        } else {
-            Err(AuditError::Configuration(format!("Target not found: {target_id}"), None))
-        }
+        self.runtime_view().disable_target(target_id).await
     }
 
     /// Removes a target from the system
@@ -624,16 +334,7 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn remove_target(&self, target_id: &str) -> AuditResult<()> {
-        let mut registry = self.registry.lock().await;
-        if let Some(target) = registry.remove_target(target_id) {
-            if let Err(e) = target.close().await {
-                error!(target_id = %target_id, error = %e, "Failed to close removed target");
-            }
-            info!(target_id = %target_id, "Target removed");
-            Ok(())
-        } else {
-            Err(AuditError::Configuration(format!("Target not found: {target_id}"), None))
-        }
+        self.runtime_view().remove_target(target_id).await
     }
 
     /// Updates or inserts a target
@@ -645,23 +346,7 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn upsert_target(&self, target_id: String, target: Box<dyn Target<AuditEntry> + Send + Sync>) -> AuditResult<()> {
-        let mut registry = self.registry.lock().await;
-
-        // Initialize the target
-        if let Err(e) = target.init().await {
-            return Err(AuditError::Target(e));
-        }
-
-        // Remove existing target if present
-        if let Some(old_target) = registry.remove_target(&target_id)
-            && let Err(e) = old_target.close().await
-        {
-            error!(target_id = %target_id, error = %e, "Failed to close old target during upsert");
-        }
-
-        registry.add_target(target_id.clone(), target);
-        info!(target_id = %target_id, "Target upserted");
-        Ok(())
+        self.runtime_view().upsert_target(target_id, target).await
     }
 
     /// Lists all targets
@@ -669,33 +354,27 @@ impl AuditSystem {
     /// # Returns
     /// * `Vec<String>` - List of target IDs
     pub async fn list_targets(&self) -> Vec<String> {
-        let registry = self.registry.lock().await;
-        registry.list_targets()
+        self.runtime_view().list_targets().await
     }
 
     /// Returns cloned target values for read-only runtime inspection.
-    pub async fn get_target_values(&self) -> Vec<Box<dyn Target<AuditEntry> + Send + Sync>> {
-        let registry = self.registry.lock().await;
-        registry.list_target_values()
+    pub async fn get_target_values(&self) -> Vec<rustfs_targets::SharedTarget<AuditEntry>> {
+        self.runtime_view().get_target_values().await
     }
 
     /// Returns per-target delivery metrics for Prometheus collection.
     pub async fn snapshot_target_metrics(&self) -> Vec<AuditTargetMetricSnapshot> {
-        let targets = self.get_target_values().await;
-        let mut snapshots = Vec::with_capacity(targets.len());
+        self.pipeline().snapshot_target_metrics().await
+    }
 
-        for target in targets {
-            let delivery = target.delivery_snapshot();
-            snapshots.push(AuditTargetMetricSnapshot {
-                failed_messages: delivery.failed_messages,
-                queue_length: delivery.queue_length,
-                target_id: target.id().to_string(),
-                total_messages: delivery.total_messages,
-            });
-        }
+    pub async fn snapshot_target_health(&self) -> Vec<rustfs_targets::RuntimeTargetHealthSnapshot> {
+        self.pipeline().snapshot_target_health().await
+    }
 
-        snapshots.sort_by(|a, b| a.target_id.cmp(&b.target_id));
-        snapshots
+    pub async fn runtime_status_snapshot(&self) -> rustfs_targets::RuntimeStatusSnapshot {
+        let replay_workers = self.stream_cancellers.read().await;
+        let registry = self.registry.lock().await;
+        registry.runtime_manager().status_snapshot(&replay_workers)
     }
 
     /// Gets information about a specific target
@@ -706,8 +385,7 @@ impl AuditSystem {
     /// # Returns
     /// * `Option<String>` - Target ID if found
     pub async fn get_target(&self, target_id: &str) -> Option<String> {
-        let registry = self.registry.lock().await;
-        registry.get_target(target_id).map(|target| target.id().to_string())
+        self.runtime_view().get_target(target_id).await
     }
 
     /// Reloads configuration and updates targets
@@ -722,32 +400,20 @@ impl AuditSystem {
 
         observability::record_config_reload();
 
-        // Stop all existing stream tasks first
-        self.stop_all_streams().await;
-
         // Store new configuration
         {
             let mut config_guard = self.config.write().await;
             *config_guard = Some(new_config.clone());
         }
 
-        // Close all existing targets
-        let mut registry = self.registry.lock().await;
-        if let Err(e) = registry.close_all().await {
-            error!(error = %e, "Failed to close existing targets during reload");
-        }
+        let final_state = match self.get_state().await {
+            AuditSystemState::Paused => AuditSystemState::Paused,
+            _ => AuditSystemState::Running,
+        };
 
-        // Create new targets from updated configuration
-        match registry.create_audit_targets_from_config(&new_config).await {
+        match self.create_targets_from_config(&new_config).await {
             Ok(targets) => {
-                info!(target_count = targets.len(), "Reloaded audit targets successfully");
-
-                for target in targets {
-                    if let Some(target_id) = self.init_and_register_target(target, &mut registry).await {
-                        info!(target_id = %target_id, "Target initialized (reload)");
-                    }
-                }
-
+                self.commit_runtime_targets(targets, final_state).await?;
                 info!("Audit configuration reloaded successfully");
                 Ok(())
             }
@@ -779,3 +445,106 @@ impl AuditSystem {
         observability::reset_metrics().await;
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::{AuditSystem, AuditSystemState};
+    use async_trait::async_trait;
+    use rustfs_targets::ReplayWorkerManager;
+    use rustfs_targets::arn::TargetID;
+    use rustfs_targets::store::{Key, Store};
+    use rustfs_targets::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta};
+    use rustfs_targets::{StoreError, Target, TargetError};
+    use serde::{Serialize, de::DeserializeOwned};
+    use std::collections::HashMap;
+    use std::sync::Arc;
+    use std::sync::atomic::{AtomicUsize, Ordering};
+    use tokio::sync::mpsc;
+
+    #[derive(Clone)]
+    struct TestTarget {
+        close_calls: Arc<AtomicUsize>,
+        id: TargetID,
+    }
+
+    impl TestTarget {
+        fn new(id: &str, name: &str) -> Self {
+            Self {
+                close_calls: Arc::new(AtomicUsize::new(0)),
+                id: TargetID::new(id.to_string(), name.to_string()),
+            }
+        }
+    }
+
+    #[async_trait]
+    impl<E> Target<E> for TestTarget
+    where
+        E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned,
+    {
+        fn id(&self) -> TargetID {
+            self.id.clone()
+        }
+
+        async fn is_active(&self) -> Result<bool, TargetError> {
+            Ok(true)
+        }
+
+        async fn save(&self, _event: Arc<EntityTarget<E>>) -> Result<(), TargetError> {
+            Ok(())
+        }
+
+        async fn send_raw_from_store(&self, _key: Key, _body: Vec<u8>, _meta: QueuedPayloadMeta) -> Result<(), TargetError> {
+            Ok(())
+        }
+
+        async fn close(&self) -> Result<(), TargetError> {
+            self.close_calls.fetch_add(1, Ordering::SeqCst);
+            Ok(())
+        }
+
+        fn store(&self) -> Option<&(dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send + Sync)> {
+            None
+        }
+
+        fn clone_dyn(&self) -> Box<dyn Target<E> + Send + Sync> {
+            Box::new(self.clone())
+        }
+
+        fn is_enabled(&self) -> bool {
+            true
+        }
+    }
+
+    #[tokio::test]
+    async fn reload_with_empty_config_stops_existing_runtime() {
+        let system = AuditSystem::new();
+        let target = TestTarget::new("primary", "webhook");
+        let close_calls = Arc::clone(&target.close_calls);
+
+        {
+            let mut registry = system.registry.lock().await;
+            registry.add_target("primary:webhook".to_string(), Box::new(target));
+        }
+        {
+            let mut state = system.state.write().await;
+            *state = AuditSystemState::Running;
+        }
+        {
+            let mut replay_workers = system.stream_cancellers.write().await;
+            let (cancel_tx, _cancel_rx) = mpsc::channel(1);
+            replay_workers.insert("primary:webhook".to_string(), cancel_tx);
+            assert_eq!(replay_workers.len(), 1);
+        }
+
+        system
+            .reload_config(rustfs_ecstore::config::Config(HashMap::new()))
+            .await
+            .expect("reload with empty config should succeed");
+
+        assert_eq!(system.get_state().await, AuditSystemState::Stopped);
+        assert!(system.list_targets().await.is_empty());
+        assert_eq!(system.runtime_status_snapshot().await, ReplayWorkerManager::new().snapshot(0));
+        assert_eq!(close_calls.load(Ordering::SeqCst), 1);
+        assert_eq!(*system.config.read().await, Some(rustfs_ecstore::config::Config(HashMap::new())));
+    }
+}

crates/audit/tests/pipeline_layer_test.rs

@@ -0,0 +1,170 @@
+//  Copyright 2024 RustFS Team
+//
+//  Licensed under the Apache License, Version 2.0 (the "License");
+//  you may not use this file except in compliance with the License.
+//  You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+//  Unless required by applicable law or agreed to in writing, software
+//  distributed under the License is distributed on an "AS IS" BASIS,
+//  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+//  See the License for the specific language governing permissions and
+//  limitations under the License.
+
+use async_trait::async_trait;
+use rustfs_audit::{AuditPipeline, AuditRegistry, AuditRuntimeFacade, AuditRuntimeView};
+use rustfs_targets::arn::TargetID;
+use rustfs_targets::store::{Key, Store};
+use rustfs_targets::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta};
+use rustfs_targets::{StoreError, Target, TargetError};
+use serde::{Serialize, de::DeserializeOwned};
+use std::sync::Arc;
+use std::sync::atomic::{AtomicUsize, Ordering};
+use tokio::sync::{Mutex, RwLock};
+
+#[derive(Clone)]
+struct TestTarget {
+    close_calls: Arc<AtomicUsize>,
+    id: TargetID,
+    init_calls: Arc<AtomicUsize>,
+}
+
+impl TestTarget {
+    fn new(id: &str, name: &str) -> Self {
+        Self {
+            close_calls: Arc::new(AtomicUsize::new(0)),
+            id: TargetID::new(id.to_string(), name.to_string()),
+            init_calls: Arc::new(AtomicUsize::new(0)),
+        }
+    }
+}
+
+#[async_trait]
+impl<E> Target<E> for TestTarget
+where
+    E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned,
+{
+    fn id(&self) -> TargetID {
+        self.id.clone()
+    }
+
+    async fn is_active(&self) -> Result<bool, TargetError> {
+        Ok(true)
+    }
+
+    async fn save(&self, _event: Arc<EntityTarget<E>>) -> Result<(), TargetError> {
+        Ok(())
+    }
+
+    async fn send_raw_from_store(&self, _key: Key, _body: Vec<u8>, _meta: QueuedPayloadMeta) -> Result<(), TargetError> {
+        Ok(())
+    }
+
+    async fn close(&self) -> Result<(), TargetError> {
+        self.close_calls.fetch_add(1, Ordering::SeqCst);
+        Ok(())
+    }
+
+    fn store(&self) -> Option<&(dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send + Sync)> {
+        None
+    }
+
+    fn clone_dyn(&self) -> Box<dyn Target<E> + Send + Sync> {
+        Box::new(self.clone())
+    }
+
+    async fn init(&self) -> Result<(), TargetError> {
+        self.init_calls.fetch_add(1, Ordering::SeqCst);
+        Ok(())
+    }
+
+    fn is_enabled(&self) -> bool {
+        true
+    }
+}
+
+#[tokio::test]
+async fn audit_runtime_view_lists_empty_targets() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let runtime_view = AuditRuntimeView::new(registry);
+
+    assert!(runtime_view.list_targets().await.is_empty());
+    assert!(runtime_view.get_target_values().await.is_empty());
+    assert!(runtime_view.get_target("missing").await.is_none());
+}
+
+#[tokio::test]
+async fn audit_pipeline_reports_empty_runtime_snapshots() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let pipeline = AuditPipeline::new(registry);
+
+    assert!(pipeline.snapshot_target_metrics().await.is_empty());
+    assert!(pipeline.snapshot_target_health().await.is_empty());
+}
+
+#[tokio::test]
+async fn audit_runtime_facade_stops_empty_replay_workers() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let replay_workers = Arc::new(RwLock::new(rustfs_targets::ReplayWorkerManager::new()));
+    let facade = AuditRuntimeFacade::new(registry, replay_workers);
+
+    facade.stop_replay_workers().await;
+}
+
+#[tokio::test]
+async fn audit_runtime_facade_activates_empty_target_list() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let replay_workers = Arc::new(RwLock::new(rustfs_targets::ReplayWorkerManager::new()));
+    let facade = AuditRuntimeFacade::new(registry, replay_workers);
+
+    let activation = facade.activate_targets_with_replay(Vec::new()).await;
+    assert!(activation.targets.is_empty());
+    assert_eq!(activation.replay_workers.len(), 0);
+}
+
+#[tokio::test]
+async fn audit_runtime_view_upsert_and_remove_target() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let runtime_view = AuditRuntimeView::new(registry.clone());
+    let target = TestTarget::new("primary", "webhook");
+    let init_calls = Arc::clone(&target.init_calls);
+    let close_calls = Arc::clone(&target.close_calls);
+
+    runtime_view
+        .upsert_target("primary:webhook".to_string(), Box::new(target))
+        .await
+        .expect("upsert should succeed");
+
+    assert_eq!(runtime_view.list_targets().await, vec!["primary:webhook".to_string()]);
+    assert_eq!(init_calls.load(Ordering::SeqCst), 1);
+
+    runtime_view
+        .remove_target("primary:webhook")
+        .await
+        .expect("remove should succeed");
+
+    assert!(runtime_view.list_targets().await.is_empty());
+    assert_eq!(close_calls.load(Ordering::SeqCst), 1);
+}
+
+#[tokio::test]
+async fn audit_runtime_facade_replace_targets_commits_runtime_state() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let replay_workers = Arc::new(RwLock::new(rustfs_targets::ReplayWorkerManager::new()));
+    let facade = AuditRuntimeFacade::new(registry.clone(), replay_workers.clone());
+    let target = TestTarget::new("primary", "webhook");
+    let activation = rustfs_targets::RuntimeActivation {
+        replay_workers: rustfs_targets::ReplayWorkerManager::new(),
+        targets: vec![Arc::new(target) as rustfs_targets::SharedTarget<rustfs_audit::AuditEntry>],
+    };
+
+    facade
+        .replace_targets(activation)
+        .await
+        .expect("replace_targets should succeed");
+
+    let runtime_view = AuditRuntimeView::new(registry);
+    assert_eq!(runtime_view.list_targets().await, vec!["primary:webhook".to_string()]);
+    assert_eq!(replay_workers.read().await.len(), 0);
+}

crates/common/src/internode_metrics.rs

@@ -51,7 +51,7 @@ impl InternodeMetrics {
             return;
         }
         self.sent_bytes_total.fetch_add(bytes, Ordering::Relaxed);
-        counter!("rustfs.internode.sent.bytes.total").increment(bytes);
+        counter!("rustfs_system_network_internode_sent_bytes_total").increment(bytes);
     }
 
     pub fn record_recv_bytes(&self, bytes: usize) {
@@ -60,22 +60,22 @@ impl InternodeMetrics {
             return;
         }
         self.recv_bytes_total.fetch_add(bytes, Ordering::Relaxed);
-        counter!("rustfs.internode.recv.bytes.total").increment(bytes);
+        counter!("rustfs_system_network_internode_recv_bytes_total").increment(bytes);
     }
 
     pub fn record_outgoing_request(&self) {
         self.outgoing_requests_total.fetch_add(1, Ordering::Relaxed);
-        counter!("rustfs.internode.requests.outgoing.total").increment(1);
+        counter!("rustfs_system_network_internode_requests_outgoing_total").increment(1);
     }
 
     pub fn record_incoming_request(&self) {
         self.incoming_requests_total.fetch_add(1, Ordering::Relaxed);
-        counter!("rustfs.internode.requests.incoming.total").increment(1);
+        counter!("rustfs_system_network_internode_requests_incoming_total").increment(1);
     }
 
     pub fn record_error(&self) {
         self.errors_total.fetch_add(1, Ordering::Relaxed);
-        counter!("rustfs.internode.errors.total").increment(1);
+        counter!("rustfs_system_network_internode_errors_total").increment(1);
     }
 
     pub fn record_dial_result(&self, duration: Duration, success: bool) {
@@ -83,11 +83,11 @@ impl InternodeMetrics {
         self.dial_total_time_nanos.fetch_add(elapsed_nanos, Ordering::Relaxed);
         let samples = self.dial_samples_total.fetch_add(1, Ordering::Relaxed) + 1;
         let total = self.dial_total_time_nanos.load(Ordering::Relaxed);
-        gauge!("rustfs.internode.dial.avg_time.nanos").set(total as f64 / samples as f64);
+        gauge!("rustfs_system_network_internode_dial_avg_time_nanos").set(total as f64 / samples as f64);
 
         if !success {
             self.dial_errors_total.fetch_add(1, Ordering::Relaxed);
-            counter!("rustfs.internode.dial.errors.total").increment(1);
+            counter!("rustfs_system_network_internode_dial_errors_total").increment(1);
         }
 
         let now_ms = SystemTime::now()

crates/config/README.md

@@ -41,6 +41,7 @@ Examples:
 - `RUSTFS_ADDRESS`
 - `RUSTFS_VOLUMES`
 - `RUSTFS_LICENSE`
+- `RUSTFS_LICENSE_PUBLIC_KEY`
 
 Current guidance:
 - Prefer module-specific names only when they are not top-level product configuration.
@@ -51,6 +52,16 @@ Current guidance:
   - `RUSTFS_ENABLE_HEAL` -> `RUSTFS_HEAL_ENABLED`
   - `RUSTFS_DATA_SCANNER_START_DELAY_SECS` -> `RUSTFS_SCANNER_START_DELAY_SECS`
 
+## License environment variables
+
+- `RUSTFS_LICENSE` contains the signed license token.
+- `RUSTFS_LICENSE_PUBLIC_KEY` contains the RSA public key used to verify signed license tokens.
+
+## CORS environment variables
+
+- `RUSTFS_CORS_ALLOWED_ORIGINS` defaults to empty, so the S3 endpoint emits no generic CORS headers unless configured. Set `*` for wildcard origins without credentials, or a comma-separated allow-list for credentialed explicit origins.
+- `RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS` defaults to `*` for the console service.
+
 ## Scanner environment aliases
 
 - `RUSTFS_SCANNER_SPEED` (canonical, also accepts `MINIO_SCANNER_SPEED`)

crates/config/src/audit/amqp.rs

@@ -0,0 +1,60 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const AUDIT_AMQP_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::AMQP_URL,
+    crate::AMQP_EXCHANGE,
+    crate::AMQP_ROUTING_KEY,
+    crate::AMQP_MANDATORY,
+    crate::AMQP_PERSISTENT,
+    crate::AMQP_USERNAME,
+    crate::AMQP_PASSWORD,
+    crate::AMQP_TLS_CA,
+    crate::AMQP_TLS_CLIENT_CERT,
+    crate::AMQP_TLS_CLIENT_KEY,
+    crate::AMQP_QUEUE_DIR,
+    crate::AMQP_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_AUDIT_AMQP_ENABLE: &str = "RUSTFS_AUDIT_AMQP_ENABLE";
+pub const ENV_AUDIT_AMQP_URL: &str = "RUSTFS_AUDIT_AMQP_URL";
+pub const ENV_AUDIT_AMQP_EXCHANGE: &str = "RUSTFS_AUDIT_AMQP_EXCHANGE";
+pub const ENV_AUDIT_AMQP_ROUTING_KEY: &str = "RUSTFS_AUDIT_AMQP_ROUTING_KEY";
+pub const ENV_AUDIT_AMQP_MANDATORY: &str = "RUSTFS_AUDIT_AMQP_MANDATORY";
+pub const ENV_AUDIT_AMQP_PERSISTENT: &str = "RUSTFS_AUDIT_AMQP_PERSISTENT";
+pub const ENV_AUDIT_AMQP_USERNAME: &str = "RUSTFS_AUDIT_AMQP_USERNAME";
+pub const ENV_AUDIT_AMQP_PASSWORD: &str = "RUSTFS_AUDIT_AMQP_PASSWORD";
+pub const ENV_AUDIT_AMQP_TLS_CA: &str = "RUSTFS_AUDIT_AMQP_TLS_CA";
+pub const ENV_AUDIT_AMQP_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_AMQP_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_AMQP_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_AMQP_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_AMQP_QUEUE_DIR: &str = "RUSTFS_AUDIT_AMQP_QUEUE_DIR";
+pub const ENV_AUDIT_AMQP_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_AMQP_QUEUE_LIMIT";
+
+pub const ENV_AUDIT_AMQP_KEYS: &[&str; 13] = &[
+    ENV_AUDIT_AMQP_ENABLE,
+    ENV_AUDIT_AMQP_URL,
+    ENV_AUDIT_AMQP_EXCHANGE,
+    ENV_AUDIT_AMQP_ROUTING_KEY,
+    ENV_AUDIT_AMQP_MANDATORY,
+    ENV_AUDIT_AMQP_PERSISTENT,
+    ENV_AUDIT_AMQP_USERNAME,
+    ENV_AUDIT_AMQP_PASSWORD,
+    ENV_AUDIT_AMQP_TLS_CA,
+    ENV_AUDIT_AMQP_TLS_CLIENT_CERT,
+    ENV_AUDIT_AMQP_TLS_CLIENT_KEY,
+    ENV_AUDIT_AMQP_QUEUE_DIR,
+    ENV_AUDIT_AMQP_QUEUE_LIMIT,
+];

crates/config/src/audit/kafka.rs

@@ -0,0 +1,53 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Kafka Environment Variables
+pub const ENV_AUDIT_KAFKA_ENABLE: &str = "RUSTFS_AUDIT_KAFKA_ENABLE";
+pub const ENV_AUDIT_KAFKA_BROKERS: &str = "RUSTFS_AUDIT_KAFKA_BROKERS";
+pub const ENV_AUDIT_KAFKA_TOPIC: &str = "RUSTFS_AUDIT_KAFKA_TOPIC";
+pub const ENV_AUDIT_KAFKA_ACKS: &str = "RUSTFS_AUDIT_KAFKA_ACKS";
+pub const ENV_AUDIT_KAFKA_TLS_ENABLE: &str = "RUSTFS_AUDIT_KAFKA_TLS_ENABLE";
+pub const ENV_AUDIT_KAFKA_TLS_CA: &str = "RUSTFS_AUDIT_KAFKA_TLS_CA";
+pub const ENV_AUDIT_KAFKA_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_KAFKA_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_KAFKA_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_KAFKA_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_KAFKA_QUEUE_DIR: &str = "RUSTFS_AUDIT_KAFKA_QUEUE_DIR";
+pub const ENV_AUDIT_KAFKA_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_KAFKA_QUEUE_LIMIT";
+
+pub const ENV_AUDIT_KAFKA_KEYS: &[&str; 10] = &[
+    ENV_AUDIT_KAFKA_ENABLE,
+    ENV_AUDIT_KAFKA_BROKERS,
+    ENV_AUDIT_KAFKA_TOPIC,
+    ENV_AUDIT_KAFKA_ACKS,
+    ENV_AUDIT_KAFKA_TLS_ENABLE,
+    ENV_AUDIT_KAFKA_TLS_CA,
+    ENV_AUDIT_KAFKA_TLS_CLIENT_CERT,
+    ENV_AUDIT_KAFKA_TLS_CLIENT_KEY,
+    ENV_AUDIT_KAFKA_QUEUE_DIR,
+    ENV_AUDIT_KAFKA_QUEUE_LIMIT,
+];
+
+/// A list of all valid configuration keys for a Kafka audit target.
+pub const AUDIT_KAFKA_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::KAFKA_BROKERS,
+    crate::KAFKA_TOPIC,
+    crate::KAFKA_ACKS,
+    crate::KAFKA_TLS_ENABLE,
+    crate::KAFKA_TLS_CA,
+    crate::KAFKA_TLS_CLIENT_CERT,
+    crate::KAFKA_TLS_CLIENT_KEY,
+    crate::KAFKA_QUEUE_DIR,
+    crate::KAFKA_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];

crates/config/src/audit/mod.rs

@@ -16,14 +16,24 @@
 //! This module defines the configuration for audit systems, including
 //! webhook and MQTT audit-related settings.
 
+mod amqp;
+mod kafka;
 mod mqtt;
+mod mysql;
 mod nats;
+mod postgres;
 mod pulsar;
+mod redis;
 mod webhook;
 
+pub use amqp::*;
+pub use kafka::*;
 pub use mqtt::*;
+pub use mysql::*;
 pub use nats::*;
+pub use postgres::*;
 pub use pulsar::*;
+pub use redis::*;
 pub use webhook::*;
 
 use crate::DEFAULT_DELIMITER;
@@ -33,15 +43,24 @@ pub const AUDIT_PREFIX: &str = "audit";
 pub const AUDIT_ROUTE_PREFIX: &str = const_str::concat!(AUDIT_PREFIX, DEFAULT_DELIMITER);
 
 pub const AUDIT_WEBHOOK_SUB_SYS: &str = "audit_webhook";
+pub const AUDIT_AMQP_SUB_SYS: &str = "audit_amqp";
+pub const AUDIT_KAFKA_SUB_SYS: &str = "audit_kafka";
 pub const AUDIT_MQTT_SUB_SYS: &str = "audit_mqtt";
+pub const AUDIT_MYSQL_SUB_SYS: &str = "audit_mysql";
 pub const AUDIT_NATS_SUB_SYS: &str = "audit_nats";
+pub const AUDIT_POSTGRES_SUB_SYS: &str = "audit_postgres";
 pub const AUDIT_PULSAR_SUB_SYS: &str = "audit_pulsar";
-
+pub const AUDIT_REDIS_SUB_SYS: &str = "audit_redis";
+pub const AUDIT_REDIS_DEFAULT_CHANNEL: &str = "rustfs_audit_channel";
 pub const AUDIT_STORE_EXTENSION: &str = ".audit";
-#[allow(dead_code)]
 pub const AUDIT_SUB_SYSTEMS: &[&str] = &[
+    AUDIT_AMQP_SUB_SYS,
+    AUDIT_KAFKA_SUB_SYS,
     AUDIT_MQTT_SUB_SYS,
+    AUDIT_MYSQL_SUB_SYS,
     AUDIT_NATS_SUB_SYS,
+    AUDIT_POSTGRES_SUB_SYS,
     AUDIT_PULSAR_SUB_SYS,
+    AUDIT_REDIS_SUB_SYS,
     AUDIT_WEBHOOK_SUB_SYS,
 ];

crates/config/src/audit/mysql.rs

@@ -0,0 +1,53 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// A list of all valid configuration keys for an audit MySQL target.
+pub const AUDIT_MYSQL_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::MYSQL_DSN_STRING,
+    crate::MYSQL_TABLE,
+    crate::MYSQL_FORMAT,
+    crate::MYSQL_TLS_CA,
+    crate::MYSQL_TLS_CLIENT_CERT,
+    crate::MYSQL_TLS_CLIENT_KEY,
+    crate::MYSQL_QUEUE_DIR,
+    crate::MYSQL_QUEUE_LIMIT,
+    crate::MYSQL_MAX_OPEN_CONNECTIONS,
+    crate::COMMENT_KEY,
+];
+
+// MySQL environment variables for audit target.
+pub const ENV_AUDIT_MYSQL_ENABLE: &str = "RUSTFS_AUDIT_MYSQL_ENABLE";
+pub const ENV_AUDIT_MYSQL_DSN_STRING: &str = "RUSTFS_AUDIT_MYSQL_DSN_STRING";
+pub const ENV_AUDIT_MYSQL_TABLE: &str = "RUSTFS_AUDIT_MYSQL_TABLE";
+pub const ENV_AUDIT_MYSQL_FORMAT: &str = "RUSTFS_AUDIT_MYSQL_FORMAT";
+pub const ENV_AUDIT_MYSQL_TLS_CA: &str = "RUSTFS_AUDIT_MYSQL_TLS_CA";
+pub const ENV_AUDIT_MYSQL_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_MYSQL_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_MYSQL_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_MYSQL_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_MYSQL_QUEUE_DIR: &str = "RUSTFS_AUDIT_MYSQL_QUEUE_DIR";
+pub const ENV_AUDIT_MYSQL_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_MYSQL_QUEUE_LIMIT";
+pub const ENV_AUDIT_MYSQL_MAX_OPEN_CONNECTIONS: &str = "RUSTFS_AUDIT_MYSQL_MAX_OPEN_CONNECTIONS";
+
+pub const ENV_AUDIT_MYSQL_KEYS: &[&str; 10] = &[
+    ENV_AUDIT_MYSQL_ENABLE,
+    ENV_AUDIT_MYSQL_DSN_STRING,
+    ENV_AUDIT_MYSQL_TABLE,
+    ENV_AUDIT_MYSQL_FORMAT,
+    ENV_AUDIT_MYSQL_TLS_CA,
+    ENV_AUDIT_MYSQL_TLS_CLIENT_CERT,
+    ENV_AUDIT_MYSQL_TLS_CLIENT_KEY,
+    ENV_AUDIT_MYSQL_QUEUE_DIR,
+    ENV_AUDIT_MYSQL_QUEUE_LIMIT,
+    ENV_AUDIT_MYSQL_MAX_OPEN_CONNECTIONS,
+];

crates/config/src/audit/postgres.rs

@@ -0,0 +1,51 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const AUDIT_POSTGRES_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::POSTGRES_DSN_STRING,
+    crate::POSTGRES_TABLE,
+    crate::POSTGRES_FORMAT,
+    crate::POSTGRES_TLS_REQUIRED,
+    crate::POSTGRES_TLS_CA,
+    crate::POSTGRES_TLS_CLIENT_CERT,
+    crate::POSTGRES_TLS_CLIENT_KEY,
+    crate::POSTGRES_QUEUE_DIR,
+    crate::POSTGRES_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_AUDIT_POSTGRES_ENABLE: &str = "RUSTFS_AUDIT_POSTGRES_ENABLE";
+pub const ENV_AUDIT_POSTGRES_DSN_STRING: &str = "RUSTFS_AUDIT_POSTGRES_DSN_STRING";
+pub const ENV_AUDIT_POSTGRES_TABLE: &str = "RUSTFS_AUDIT_POSTGRES_TABLE";
+pub const ENV_AUDIT_POSTGRES_FORMAT: &str = "RUSTFS_AUDIT_POSTGRES_FORMAT";
+pub const ENV_AUDIT_POSTGRES_TLS_REQUIRED: &str = "RUSTFS_AUDIT_POSTGRES_TLS_REQUIRED";
+pub const ENV_AUDIT_POSTGRES_TLS_CA: &str = "RUSTFS_AUDIT_POSTGRES_TLS_CA";
+pub const ENV_AUDIT_POSTGRES_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_POSTGRES_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_POSTGRES_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_POSTGRES_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_POSTGRES_QUEUE_DIR: &str = "RUSTFS_AUDIT_POSTGRES_QUEUE_DIR";
+pub const ENV_AUDIT_POSTGRES_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_POSTGRES_QUEUE_LIMIT";
+
+pub const ENV_AUDIT_POSTGRES_KEYS: &[&str; 10] = &[
+    ENV_AUDIT_POSTGRES_ENABLE,
+    ENV_AUDIT_POSTGRES_DSN_STRING,
+    ENV_AUDIT_POSTGRES_TABLE,
+    ENV_AUDIT_POSTGRES_FORMAT,
+    ENV_AUDIT_POSTGRES_TLS_REQUIRED,
+    ENV_AUDIT_POSTGRES_TLS_CA,
+    ENV_AUDIT_POSTGRES_TLS_CLIENT_CERT,
+    ENV_AUDIT_POSTGRES_TLS_CLIENT_KEY,
+    ENV_AUDIT_POSTGRES_QUEUE_DIR,
+    ENV_AUDIT_POSTGRES_QUEUE_LIMIT,
+];

crates/config/src/audit/redis.rs

@@ -0,0 +1,81 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const ENV_AUDIT_REDIS_ENABLE: &str = "RUSTFS_AUDIT_REDIS_ENABLE";
+pub const ENV_AUDIT_REDIS_URL: &str = "RUSTFS_AUDIT_REDIS_URL";
+pub const ENV_AUDIT_REDIS_CHANNEL: &str = "RUSTFS_AUDIT_REDIS_CHANNEL";
+pub const ENV_AUDIT_REDIS_USERNAME: &str = "RUSTFS_AUDIT_REDIS_USERNAME";
+pub const ENV_AUDIT_REDIS_PASSWORD: &str = "RUSTFS_AUDIT_REDIS_PASSWORD";
+pub const ENV_AUDIT_REDIS_KEEP_ALIVE_INTERVAL: &str = "RUSTFS_AUDIT_REDIS_KEEP_ALIVE_INTERVAL";
+pub const ENV_AUDIT_REDIS_QUEUE_DIR: &str = "RUSTFS_AUDIT_REDIS_QUEUE_DIR";
+pub const ENV_AUDIT_REDIS_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_REDIS_QUEUE_LIMIT";
+pub const ENV_AUDIT_REDIS_MAX_RETRY_ATTEMPTS: &str = "RUSTFS_AUDIT_REDIS_MAX_RETRY_ATTEMPTS";
+pub const ENV_AUDIT_REDIS_RECONNECT_RETRY_ATTEMPTS: &str = "RUSTFS_AUDIT_REDIS_RECONNECT_RETRY_ATTEMPTS";
+pub const ENV_AUDIT_REDIS_MIN_RETRY_DELAY: &str = "RUSTFS_AUDIT_REDIS_MIN_RETRY_DELAY";
+pub const ENV_AUDIT_REDIS_MAX_RETRY_DELAY: &str = "RUSTFS_AUDIT_REDIS_MAX_RETRY_DELAY";
+pub const ENV_AUDIT_REDIS_CONNECTION_TIMEOUT: &str = "RUSTFS_AUDIT_REDIS_CONNECTION_TIMEOUT";
+pub const ENV_AUDIT_REDIS_RESPONSE_TIMEOUT: &str = "RUSTFS_AUDIT_REDIS_RESPONSE_TIMEOUT";
+pub const ENV_AUDIT_REDIS_PIPELINE_BUFFER_SIZE: &str = "RUSTFS_AUDIT_REDIS_PIPELINE_BUFFER_SIZE";
+pub const ENV_AUDIT_REDIS_TLS_POLICY: &str = "RUSTFS_AUDIT_REDIS_TLS_POLICY";
+pub const ENV_AUDIT_REDIS_TLS_CA: &str = "RUSTFS_AUDIT_REDIS_TLS_CA";
+pub const ENV_AUDIT_REDIS_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_REDIS_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_REDIS_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_REDIS_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_REDIS_TLS_ALLOW_INSECURE: &str = "RUSTFS_AUDIT_REDIS_TLS_ALLOW_INSECURE";
+
+pub const ENV_AUDIT_REDIS_KEYS: &[&str; 20] = &[
+    ENV_AUDIT_REDIS_ENABLE,
+    ENV_AUDIT_REDIS_URL,
+    ENV_AUDIT_REDIS_CHANNEL,
+    ENV_AUDIT_REDIS_USERNAME,
+    ENV_AUDIT_REDIS_PASSWORD,
+    ENV_AUDIT_REDIS_KEEP_ALIVE_INTERVAL,
+    ENV_AUDIT_REDIS_QUEUE_DIR,
+    ENV_AUDIT_REDIS_QUEUE_LIMIT,
+    ENV_AUDIT_REDIS_MAX_RETRY_ATTEMPTS,
+    ENV_AUDIT_REDIS_RECONNECT_RETRY_ATTEMPTS,
+    ENV_AUDIT_REDIS_MIN_RETRY_DELAY,
+    ENV_AUDIT_REDIS_MAX_RETRY_DELAY,
+    ENV_AUDIT_REDIS_CONNECTION_TIMEOUT,
+    ENV_AUDIT_REDIS_RESPONSE_TIMEOUT,
+    ENV_AUDIT_REDIS_PIPELINE_BUFFER_SIZE,
+    ENV_AUDIT_REDIS_TLS_POLICY,
+    ENV_AUDIT_REDIS_TLS_CA,
+    ENV_AUDIT_REDIS_TLS_CLIENT_CERT,
+    ENV_AUDIT_REDIS_TLS_CLIENT_KEY,
+    ENV_AUDIT_REDIS_TLS_ALLOW_INSECURE,
+];
+
+pub const AUDIT_REDIS_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::REDIS_URL,
+    crate::REDIS_CHANNEL,
+    crate::REDIS_USERNAME,
+    crate::REDIS_PASSWORD,
+    crate::REDIS_KEEP_ALIVE_INTERVAL,
+    crate::REDIS_QUEUE_DIR,
+    crate::REDIS_QUEUE_LIMIT,
+    crate::REDIS_MAX_RETRY_ATTEMPTS,
+    crate::REDIS_RECONNECT_RETRY_ATTEMPTS,
+    crate::REDIS_MIN_RETRY_DELAY,
+    crate::REDIS_MAX_RETRY_DELAY,
+    crate::REDIS_CONNECTION_TIMEOUT,
+    crate::REDIS_RESPONSE_TIMEOUT,
+    crate::REDIS_PIPELINE_BUFFER_SIZE,
+    crate::REDIS_TLS_POLICY,
+    crate::REDIS_TLS_CA,
+    crate::REDIS_TLS_CLIENT_CERT,
+    crate::REDIS_TLS_CLIENT_KEY,
+    crate::REDIS_TLS_ALLOW_INSECURE,
+    crate::COMMENT_KEY,
+];

crates/config/src/constants/app.rs

@@ -131,24 +131,30 @@ pub const ENV_RUSTFS_ADDRESS: &str = "RUSTFS_ADDRESS";
 /// Environment variable for server volumes.
 pub const ENV_RUSTFS_VOLUMES: &str = "RUSTFS_VOLUMES";
 
+/// Environment variable to explicitly bypass local physical disk independence checks.
+pub const ENV_UNSAFE_BYPASS_DISK_CHECK: &str = "RUSTFS_UNSAFE_BYPASS_DISK_CHECK";
+
+/// Compatibility alias used by legacy MinIO CI pipelines.
+///
+/// RustFS keeps this alias for backward compatibility only. Prefer
+/// `ENV_UNSAFE_BYPASS_DISK_CHECK` for explicit bypass control.
+pub const ENV_MINIO_CI: &str = "MINIO_CI";
+
+/// Default flag value for bypassing local physical disk independence checks.
+pub const DEFAULT_UNSAFE_BYPASS_DISK_CHECK: bool = false;
+
 /// Environment variable for server access key.
 pub const ENV_RUSTFS_ACCESS_KEY: &str = "RUSTFS_ACCESS_KEY";
 
 /// Environment variable for server access key file.
 pub const ENV_RUSTFS_ACCESS_KEY_FILE: &str = "RUSTFS_ACCESS_KEY_FILE";
 
-/// Environment variable for server root user.
-pub const ENV_RUSTFS_ROOT_USER: &str = "RUSTFS_ROOT_USER";
-
 /// Environment variable for server secret key.
 pub const ENV_RUSTFS_SECRET_KEY: &str = "RUSTFS_SECRET_KEY";
 
 /// Environment variable for server secret key file.
 pub const ENV_RUSTFS_SECRET_KEY_FILE: &str = "RUSTFS_SECRET_KEY_FILE";
 
-/// Environment variable for server root password.
-pub const ENV_RUSTFS_ROOT_PASSWORD: &str = "RUSTFS_ROOT_PASSWORD";
-
 /// Environment variable for server OBS endpoint.
 pub const ENV_RUSTFS_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";
 
@@ -223,6 +229,9 @@ pub const ENV_RUSTFS_REGION: &str = "RUSTFS_REGION";
 /// Environment variable for server license.
 pub const ENV_RUSTFS_LICENSE: &str = "RUSTFS_LICENSE";
 
+/// Environment variable for the RSA public key used to verify server licenses.
+pub const ENV_RUSTFS_LICENSE_PUBLIC_KEY: &str = "RUSTFS_LICENSE_PUBLIC_KEY";
+
 /// Default log filename for rustfs
 /// This is the default log filename for rustfs.
 /// It is used to store the logs of the application.
@@ -284,9 +293,9 @@ pub const DEFAULT_OBS_LOGS_EXPORT_ENABLED: bool = true;
 
 /// Default profiling export enabled
 /// It is used to enable or disable exporting profiles
-/// Default value: true
+/// Default value: false
 /// Environment variable: RUSTFS_OBS_PROFILING_EXPORT_ENABLED
-pub const DEFAULT_OBS_PROFILING_EXPORT_ENABLED: bool = true;
+pub const DEFAULT_OBS_PROFILING_EXPORT_ENABLED: bool = false;
 
 /// Default log local logging enabled for rustfs
 /// This is the default log local logging enabled for rustfs.
@@ -336,6 +345,7 @@ mod tests {
     fn test_environment_constants() {
         // Test environment related constants
         assert_eq!(ENVIRONMENT, "production");
+        assert_eq!(ENV_RUSTFS_LICENSE_PUBLIC_KEY, "RUSTFS_LICENSE_PUBLIC_KEY");
         assert!(
             ["development", "staging", "production", "test"].contains(&ENVIRONMENT),
             "Environment should be a standard environment name"

crates/config/src/constants/console.rs

@@ -17,16 +17,21 @@
 pub const ENV_CORS_ALLOWED_ORIGINS: &str = "RUSTFS_CORS_ALLOWED_ORIGINS";
 
 /// Default CORS allowed origins for the endpoint service
-/// Comes from the console service default
-/// See DEFAULT_CONSOLE_CORS_ALLOWED_ORIGINS
-pub const DEFAULT_CORS_ALLOWED_ORIGINS: &str = DEFAULT_CONSOLE_CORS_ALLOWED_ORIGINS;
+/// Empty means the S3 endpoint emits no generic CORS headers unless configured.
+pub const DEFAULT_CORS_ALLOWED_ORIGINS: &str = "";
 
 /// CORS allowed origins for the console service
 /// Comma-separated list of origins or "*" for all origins
 pub const ENV_CONSOLE_CORS_ALLOWED_ORIGINS: &str = "RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS";
 
-/// Default CORS allowed origins for the console service
-pub const DEFAULT_CONSOLE_CORS_ALLOWED_ORIGINS: &str = "*";
+/// Default CORS allowed origins for the console service.
+///
+/// Empty string means same-origin only — no `Access-Control-Allow-Origin`
+/// header is emitted, so browsers will not allow cross-origin reads of
+/// console responses by default. Operators that need cross-origin access set
+/// `RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS` to a comma-separated allow-list, or
+/// to `*` to keep the previous permissive behavior.
+pub const DEFAULT_CONSOLE_CORS_ALLOWED_ORIGINS: &str = "";
 
 /// Enable or disable the console service
 pub const ENV_CONSOLE_ENABLE: &str = "RUSTFS_CONSOLE_ENABLE";
@@ -89,3 +94,20 @@ pub const ENV_UPDATE_CHECK: &str = "RUSTFS_CHECK_UPDATE";
 
 /// Default value for update toggle
 pub const DEFAULT_UPDATE_CHECK: bool = true;
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn endpoint_cors_default_is_restrictive() {
+        assert_eq!(ENV_CORS_ALLOWED_ORIGINS, "RUSTFS_CORS_ALLOWED_ORIGINS");
+        assert_eq!(DEFAULT_CORS_ALLOWED_ORIGINS, "");
+    }
+
+    #[test]
+    fn console_cors_default_is_same_origin_only() {
+        assert_eq!(ENV_CONSOLE_CORS_ALLOWED_ORIGINS, "RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS");
+        assert_eq!(DEFAULT_CONSOLE_CORS_ALLOWED_ORIGINS, "");
+    }
+}

crates/config/src/constants/drive.rs

@@ -39,6 +39,14 @@ pub const DEFAULT_DRIVE_WALKDIR_TIMEOUT_SECS: u64 = 5;
 pub const ENV_DRIVE_WALKDIR_STALL_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS";
 pub const DEFAULT_DRIVE_WALKDIR_STALL_TIMEOUT_SECS: u64 = 5;
 
+/// Interval in seconds between active health probes for local and remote drives.
+pub const ENV_DRIVE_ACTIVE_CHECK_INTERVAL_SECS: &str = "RUSTFS_DRIVE_ACTIVE_CHECK_INTERVAL_SECS";
+pub const DEFAULT_DRIVE_ACTIVE_CHECK_INTERVAL_SECS: u64 = 15;
+
+/// Timeout in seconds for a single active health probe.
+pub const ENV_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS";
+pub const DEFAULT_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS: u64 = 5;
+
 /// Number of consecutive failures before a suspect drive is classified as offline.
 pub const ENV_DRIVE_SUSPECT_FAILURE_THRESHOLD: &str = "RUSTFS_DRIVE_SUSPECT_FAILURE_THRESHOLD";
 pub const DEFAULT_DRIVE_SUSPECT_FAILURE_THRESHOLD: u64 = 2;

crates/config/src/constants/env.rs

@@ -26,6 +26,21 @@ pub const RUSTFS_WEBHOOK_SKIP_TLS_VERIFY_DEFAULT: bool = false;
 pub const ENABLE_KEY: &str = "enable";
 pub const COMMENT_KEY: &str = "comment";
 
+/// Global switch for enabling the audit module.
+pub const ENV_AUDIT_ENABLE: &str = "RUSTFS_AUDIT_ENABLE";
+/// Global switch for enabling the notify module.
+pub const ENV_NOTIFY_ENABLE: &str = "RUSTFS_NOTIFY_ENABLE";
+/// Default global audit switch (disabled by default).
+pub const DEFAULT_AUDIT_ENABLE: bool = false;
+/// Default global notify switch (disabled by default).
+pub const DEFAULT_NOTIFY_ENABLE: bool = false;
+/// Canonical ILM process boundary env key (seconds).
+pub const ENV_ILM_PROCESS_TIME: &str = "RUSTFS_ILM_PROCESS_TIME";
+/// Deprecated ILM process boundary env key kept for compatibility.
+pub const ENV_ILM_PROCESS_TIME_DEPRECATED: &str = "_RUSTFS_ILM_PROCESS_TIME";
+/// Default ILM process boundary in seconds (24h).
+pub const DEFAULT_ILM_PROCESS_TIME_SECS: i32 = 86400;
+
 /// Medium-drawn lines separator
 /// This is used to separate words in environment variable names.
 pub const ENV_WORD_DELIMITER_DASH: &str = "-";
@@ -290,4 +305,10 @@ mod tests {
             assert!(state.is_disabled());
         }
     }
+
+    #[test]
+    fn test_global_audit_notify_switch_constants() {
+        assert_eq!(ENV_AUDIT_ENABLE, "RUSTFS_AUDIT_ENABLE");
+        assert_eq!(ENV_NOTIFY_ENABLE, "RUSTFS_NOTIFY_ENABLE");
+    }
 }

crates/config/src/constants/health.rs

@@ -0,0 +1,28 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// Enable or disable public `/health` and `/health/ready` endpoints.
+/// When disabled, the routes are not registered and return 404.
+pub const ENV_HEALTH_ENDPOINT_ENABLE: &str = "RUSTFS_HEALTH_ENDPOINT_ENABLE";
+pub const DEFAULT_HEALTH_ENDPOINT_ENABLE: bool = true;
+
+/// Cache TTL for storage readiness runtime-state evaluation (milliseconds).
+/// This reduces storage-layer pressure when probes are called at high frequency.
+pub const ENV_HEALTH_READINESS_CACHE_TTL_MS: &str = "RUSTFS_HEALTH_READINESS_CACHE_TTL_MS";
+pub const DEFAULT_HEALTH_READINESS_CACHE_TTL_MS: u64 = 1000;
+
+/// Enable minimal health payload mode for GET `/health*` responses.
+/// When enabled, only `status` and `ready` fields are returned.
+pub const ENV_HEALTH_MINIMAL_RESPONSE_ENABLE: &str = "RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE";
+pub const DEFAULT_HEALTH_MINIMAL_RESPONSE_ENABLE: bool = false;

crates/config/src/constants/internode.rs

@@ -0,0 +1,62 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// Timeout for establishing a new internode gRPC connection.
+pub const ENV_INTERNODE_CONNECT_TIMEOUT_SECS: &str = "RUSTFS_INTERNODE_CONNECT_TIMEOUT_SECS";
+pub const DEFAULT_INTERNODE_CONNECT_TIMEOUT_SECS: u64 = 3;
+
+/// TCP keepalive interval for internode gRPC channels.
+pub const ENV_INTERNODE_TCP_KEEPALIVE_SECS: &str = "RUSTFS_INTERNODE_TCP_KEEPALIVE_SECS";
+pub const DEFAULT_INTERNODE_TCP_KEEPALIVE_SECS: u64 = 10;
+
+/// HTTP/2 keepalive interval for internode gRPC channels.
+pub const ENV_INTERNODE_HTTP2_KEEPALIVE_INTERVAL_SECS: &str = "RUSTFS_INTERNODE_HTTP2_KEEPALIVE_INTERVAL_SECS";
+pub const DEFAULT_INTERNODE_HTTP2_KEEPALIVE_INTERVAL_SECS: u64 = 5;
+
+/// HTTP/2 keepalive timeout for internode gRPC channels.
+pub const ENV_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS: &str = "RUSTFS_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS";
+pub const DEFAULT_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS: u64 = 3;
+
+/// Overall timeout for a single internode gRPC request.
+pub const ENV_INTERNODE_RPC_TIMEOUT_SECS: &str = "RUSTFS_INTERNODE_RPC_TIMEOUT_SECS";
+pub const DEFAULT_INTERNODE_RPC_TIMEOUT_SECS: u64 = 10;
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn internode_timeout_defaults_stay_in_expected_bounds() {
+        assert_eq!(DEFAULT_INTERNODE_CONNECT_TIMEOUT_SECS, 3);
+        assert_eq!(DEFAULT_INTERNODE_TCP_KEEPALIVE_SECS, 10);
+        assert_eq!(DEFAULT_INTERNODE_HTTP2_KEEPALIVE_INTERVAL_SECS, 5);
+        assert_eq!(DEFAULT_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS, 3);
+        assert_eq!(DEFAULT_INTERNODE_RPC_TIMEOUT_SECS, 10);
+    }
+
+    #[test]
+    fn internode_timeout_env_names_are_stable() {
+        assert_eq!(ENV_INTERNODE_CONNECT_TIMEOUT_SECS, "RUSTFS_INTERNODE_CONNECT_TIMEOUT_SECS");
+        assert_eq!(ENV_INTERNODE_TCP_KEEPALIVE_SECS, "RUSTFS_INTERNODE_TCP_KEEPALIVE_SECS");
+        assert_eq!(
+            ENV_INTERNODE_HTTP2_KEEPALIVE_INTERVAL_SECS,
+            "RUSTFS_INTERNODE_HTTP2_KEEPALIVE_INTERVAL_SECS"
+        );
+        assert_eq!(
+            ENV_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS,
+            "RUSTFS_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS"
+        );
+        assert_eq!(ENV_INTERNODE_RPC_TIMEOUT_SECS, "RUSTFS_INTERNODE_RPC_TIMEOUT_SECS");
+    }
+}

crates/config/src/constants/mod.rs

@@ -20,6 +20,8 @@ pub(crate) mod console;
 pub(crate) mod drive;
 pub(crate) mod env;
 pub(crate) mod heal;
+pub(crate) mod health;
+pub(crate) mod internode;
 pub(crate) mod object;
 pub(crate) mod oidc;
 pub(crate) mod profiler;

crates/config/src/constants/protocols.rs

@@ -57,3 +57,114 @@ pub const ENV_WEBDAV_CERTS_DIR: &str = "RUSTFS_WEBDAV_CERTS_DIR";
 pub const ENV_WEBDAV_CA_FILE: &str = "RUSTFS_WEBDAV_CA_FILE";
 pub const ENV_WEBDAV_MAX_BODY_SIZE: &str = "RUSTFS_WEBDAV_MAX_BODY_SIZE";
 pub const ENV_WEBDAV_REQUEST_TIMEOUT: &str = "RUSTFS_WEBDAV_REQUEST_TIMEOUT";
+
+/// Default SFTP server bind address.
+pub const DEFAULT_SFTP_ADDRESS: &str = "0.0.0.0:2222";
+
+/// Default for SFTP host-key directory. None means no default. Operators
+/// must set RUSTFS_SFTP_HOST_KEY_DIR explicitly when SFTP is enabled.
+pub const DEFAULT_SFTP_HOST_KEY_DIR: Option<&str> = None;
+
+/// SFTP environment variable names.
+pub const ENV_SFTP_ENABLE: &str = "RUSTFS_SFTP_ENABLE";
+pub const ENV_SFTP_ADDRESS: &str = "RUSTFS_SFTP_ADDRESS";
+pub const ENV_SFTP_HOST_KEY_DIR: &str = "RUSTFS_SFTP_HOST_KEY_DIR";
+pub const ENV_SFTP_HOST_KEY_RELOAD_ENABLE: &str = "RUSTFS_SFTP_HOST_KEY_RELOAD_ENABLE";
+pub const ENV_SFTP_HOST_KEY_RELOAD_INTERVAL: &str = "RUSTFS_SFTP_HOST_KEY_RELOAD_INTERVAL";
+pub const ENV_SFTP_IDLE_TIMEOUT: &str = "RUSTFS_SFTP_IDLE_TIMEOUT";
+/// S3 multipart part size in bytes. Default DEFAULT_SFTP_PART_SIZE (16 MiB).
+/// Valid range 5 MiB to 5 GiB (S3 protocol bounds), enforced at startup.
+///
+/// The per-upload size ceiling is part_size * 10_000 (the S3 parts cap),
+/// so the default caps single uploads at 160 GiB. Deployments expecting
+/// larger single files must raise this: 64 MiB -> 640 GiB, 128 MiB ->
+/// 1.25 TiB, 512 MiB -> 5 TiB (S3 object max). Rename is not affected;
+/// multipart_copy scales the per-part size dynamically and handles up
+/// to the 5 TiB S3 object limit regardless of this setting.
+pub const ENV_SFTP_PART_SIZE: &str = "RUSTFS_SFTP_PART_SIZE";
+pub const ENV_SFTP_READ_ONLY: &str = "RUSTFS_SFTP_READ_ONLY";
+pub const ENV_SFTP_BANNER: &str = "RUSTFS_SFTP_BANNER";
+/// Optional environment variable. If RUSTFS_SFTP_HANDLES_PER_SESSION
+/// is not set in the process environment, the server uses the default
+/// of 64 handles per session and emits no warning. If set, the value
+/// must be in the inclusive range 8 to 1024. Out-of-range values fall
+/// back to the default of 64 with a warn-level log naming the
+/// requested value and the bounds.
+///
+/// Caps the maximum number of simultaneously-open SFTP handles per
+/// session. A handle is the server-side identifier returned by
+/// SSH_FXP_OPEN and SSH_FXP_OPENDIR. One client typically uses one
+/// handle per file in flight plus one per directory listing.
+/// Operators running clients with deep pipelining may raise this.
+pub const ENV_SFTP_HANDLES_PER_SESSION: &str = "RUSTFS_SFTP_HANDLES_PER_SESSION";
+
+/// Optional environment variable. If RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS
+/// is not set in the process environment, the server uses the default
+/// of 60 seconds and emits no warning. If set, the value must be in
+/// the inclusive range 5 to 600 seconds. Out-of-range values fall
+/// back to the default with a warn-level log naming the requested
+/// value and the bounds.
+///
+/// Bounds every storage backend call issued by the SFTP driver. A
+/// backend that does not respond within this many seconds returns
+/// Failure to the client and emits a warn log naming the backend
+/// method. This catches a backend that accepted the request and never
+/// returned a body, which the SSH keepalive cannot detect because the
+/// transport itself remains live.
+pub const ENV_SFTP_BACKEND_OP_TIMEOUT_SECS: &str = "RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS";
+
+/// Optional environment variable. If RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES
+/// is not set in the process environment, the server uses a 4 MiB
+/// default and emits no warning. If set, the value must be in the
+/// inclusive range MAX_READ_LEN (256 KiB) to 64 MiB. Out-of-range
+/// values fall back to the default with a warn-level log naming the
+/// requested value and the bounds.
+///
+/// Per-handle byte window the SFTP read path fetches in one backend
+/// call on a cache miss. Subsequent FXP_READs within that window are
+/// served from the buffer without a backend round trip. For
+/// sequential downloads the backend round-trip count drops by
+/// window_bytes / MAX_READ_LEN. Random-access workloads should set
+/// the window equal to MAX_READ_LEN to opt out of read-ahead.
+pub const ENV_SFTP_READ_CACHE_WINDOW_BYTES: &str = "RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES";
+
+/// Optional environment variable. If
+/// RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES is not set in the process
+/// environment, the server uses a 256 MiB default and emits no
+/// warning. If set, the value must be at least 16 MiB. Below-min
+/// values fall back to the default with a warn-level log naming the
+/// requested value and the bound.
+///
+/// Process-wide ceiling on cumulative read cache memory across every
+/// live SFTP handle. Once the accumulator plus a new window would
+/// exceed this value, the populate call on the per-handle cache is
+/// skipped. The read still completes from the freshly-fetched bytes
+/// without storing them in the cache, at the cost of one backend
+/// call per FXP_READ. High-concurrency deployments expecting many
+/// parallel downloads should raise this in step with the per-session
+/// handle cap.
+pub const ENV_SFTP_READ_CACHE_TOTAL_MEM_BYTES: &str = "RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES";
+
+/// Default idle session timeout in seconds.
+pub const DEFAULT_SFTP_IDLE_TIMEOUT: u64 = 600;
+
+/// Default SFTP host key hot reload enabled state.
+pub const DEFAULT_SFTP_HOST_KEY_RELOAD_ENABLE: bool = false;
+
+/// Default SFTP host key hot reload interval in seconds.
+pub const DEFAULT_SFTP_HOST_KEY_RELOAD_INTERVAL: u64 = 30;
+
+/// Default S3 multipart upload part size in bytes (16 MiB).
+///
+/// The per-upload size ceiling is part_size * 10_000 (the S3 parts cap),
+/// so the default gives a 160 GiB single-upload limit. Deployments that
+/// expect single files larger than this must raise part_size:
+/// 64 MiB -> 640 GiB, 128 MiB -> 1.25 TiB, 512 MiB -> 5 TiB (S3 max).
+/// The minimum is 5 MiB and the maximum is 5 GiB (S3 protocol bounds).
+pub const DEFAULT_SFTP_PART_SIZE: u64 = 16_777_216;
+
+/// Default read-only mode (disabled).
+pub const DEFAULT_SFTP_READ_ONLY: bool = false;
+
+/// Default SSH identification string (no version disclosure).
+pub const DEFAULT_SFTP_BANNER: &str = "SSH-2.0-RustFS";

crates/config/src/constants/proxy.rs

@@ -20,6 +20,11 @@ pub const ENV_TRUSTED_PROXY_ENABLED: &str = "RUSTFS_TRUSTED_PROXY_ENABLED";
 /// Trusted proxy middleware is enabled by default.
 pub const DEFAULT_TRUSTED_PROXY_ENABLED: bool = true;
 
+/// Environment variable to select the trusted proxy implementation.
+pub const ENV_TRUSTED_PROXY_IMPLEMENTATION: &str = "RUSTFS_TRUSTED_PROXY_IMPLEMENTATION";
+/// The simplified implementation is used by default.
+pub const DEFAULT_TRUSTED_PROXY_IMPLEMENTATION: &str = "simple";
+
 /// Environment variable for the proxy validation mode.
 pub const ENV_TRUSTED_PROXY_VALIDATION_MODE: &str = "RUSTFS_TRUSTED_PROXY_VALIDATION_MODE";
 /// Default validation mode is "hop_by_hop".

crates/config/src/constants/runtime.rs

@@ -59,9 +59,31 @@ pub const DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT: usize = 10;
 pub const DEFAULT_RUNTIME_DIAL9_SAMPLING_RATE: f64 = 1.0; // 100% sampling
 // Note: S3 bucket/prefix have no default; absence means upload is disabled (modeled as Option<String>)
 
-/// Threshold for small object seek support in megabytes.
+// Allocator reclaim configuration
+pub const ENV_ALLOCATOR_RECLAIM_ENABLED: &str = "RUSTFS_ALLOCATOR_RECLAIM_ENABLED";
+pub const ENV_ALLOCATOR_RECLAIM_INTERVAL_SECS: &str = "RUSTFS_ALLOCATOR_RECLAIM_INTERVAL_SECS";
+pub const ENV_ALLOCATOR_RECLAIM_FORCE: &str = "RUSTFS_ALLOCATOR_RECLAIM_FORCE";
+pub const ENV_ALLOCATOR_RECLAIM_IDLE_INTERVALS: &str = "RUSTFS_ALLOCATOR_RECLAIM_IDLE_INTERVALS";
+pub const DEFAULT_ALLOCATOR_RECLAIM_ENABLED: bool = false;
+pub const DEFAULT_ALLOCATOR_RECLAIM_INTERVAL_SECS: u64 = 30;
+pub const DEFAULT_ALLOCATOR_RECLAIM_FORCE: bool = true;
+pub const DEFAULT_ALLOCATOR_RECLAIM_IDLE_INTERVALS: u64 = 3;
+
+// File page-cache reclaim configuration
+pub const ENV_OBJECT_FILE_CACHE_RECLAIM_WRITE_ENABLE: &str = "RUSTFS_OBJECT_FILE_CACHE_RECLAIM_WRITE_ENABLE";
+pub const ENV_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE: &str = "RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE";
+pub const ENV_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD: &str = "RUSTFS_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD";
+pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_WRITE_ENABLE: bool = false;
+pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE: bool = false;
+pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD: usize = 4 * 1024 * 1024;
+
+/// Threshold for small object seek support in bytes.
 ///
-/// When an object is smaller than this size, rustfs will provide seek support.
+/// When an object response is smaller than this size, rustfs may provide
+/// in-memory seek support. Runtime GET logic also enforces a hard safety cap
+/// (`64 MiB`) to prevent large-download memory spikes even if this threshold
+/// is configured higher.
 ///
-/// Default is set to 10MB.
+/// Default is set to 10 MiB.
+pub const ENV_OBJECT_SEEK_SUPPORT_THRESHOLD: &str = "RUSTFS_OBJECT_SEEK_SUPPORT_THRESHOLD";
 pub const DEFAULT_OBJECT_SEEK_SUPPORT_THRESHOLD: usize = 10 * 1024 * 1024;

crates/config/src/constants/targets.rs

@@ -40,6 +40,28 @@ pub const MQTT_TLS_CLIENT_CERT: &str = "tls_client_cert";
 pub const MQTT_TLS_CLIENT_KEY: &str = "tls_client_key";
 pub const MQTT_TLS_TRUST_LEAF_AS_CA: &str = "tls_trust_leaf_as_ca";
 pub const MQTT_WS_PATH_ALLOWLIST: &str = "ws_path_allowlist";
+pub const KAFKA_BROKERS: &str = "brokers";
+pub const KAFKA_TOPIC: &str = "topic";
+pub const KAFKA_ACKS: &str = "acks";
+pub const KAFKA_QUEUE_DIR: &str = "queue_dir";
+pub const KAFKA_QUEUE_LIMIT: &str = "queue_limit";
+pub const KAFKA_TLS_ENABLE: &str = "tls_enable";
+pub const KAFKA_TLS_CA: &str = "tls_ca";
+pub const KAFKA_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const KAFKA_TLS_CLIENT_KEY: &str = "tls_client_key";
+
+pub const AMQP_URL: &str = "url";
+pub const AMQP_EXCHANGE: &str = "exchange";
+pub const AMQP_ROUTING_KEY: &str = "routing_key";
+pub const AMQP_MANDATORY: &str = "mandatory";
+pub const AMQP_PERSISTENT: &str = "persistent";
+pub const AMQP_USERNAME: &str = "username";
+pub const AMQP_PASSWORD: &str = "password";
+pub const AMQP_TLS_CA: &str = "tls_ca";
+pub const AMQP_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const AMQP_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const AMQP_QUEUE_DIR: &str = "queue_dir";
+pub const AMQP_QUEUE_LIMIT: &str = "queue_limit";
 
 pub const NATS_ADDRESS: &str = "address";
 pub const NATS_SUBJECT: &str = "subject";
@@ -65,6 +87,48 @@ pub const PULSAR_TLS_HOSTNAME_VERIFICATION: &str = "tls_hostname_verification";
 pub const PULSAR_QUEUE_DIR: &str = "queue_dir";
 pub const PULSAR_QUEUE_LIMIT: &str = "queue_limit";
 
+pub const BASE_DSN_STRING: &str = "dsn_string";
+
+pub const MYSQL_DSN_STRING: &str = BASE_DSN_STRING;
+pub const MYSQL_TABLE: &str = "table";
+pub const MYSQL_FORMAT: &str = "format";
+pub const MYSQL_TLS_CA: &str = "tls_ca";
+pub const MYSQL_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const MYSQL_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const MYSQL_QUEUE_DIR: &str = "queue_dir";
+pub const MYSQL_QUEUE_LIMIT: &str = "queue_limit";
+pub const MYSQL_MAX_OPEN_CONNECTIONS: &str = "max_open_connections";
+
+pub const REDIS_URL: &str = "url";
+pub const REDIS_CHANNEL: &str = "channel";
+pub const REDIS_USERNAME: &str = "username";
+pub const REDIS_PASSWORD: &str = "password";
+pub const REDIS_KEEP_ALIVE_INTERVAL: &str = "keep_alive_interval";
+pub const REDIS_QUEUE_DIR: &str = "queue_dir";
+pub const REDIS_QUEUE_LIMIT: &str = "queue_limit";
+pub const REDIS_MAX_RETRY_ATTEMPTS: &str = "max_retry_attempts";
+pub const REDIS_RECONNECT_RETRY_ATTEMPTS: &str = "reconnect_retry_attempts";
+pub const REDIS_MIN_RETRY_DELAY: &str = "min_retry_delay";
+pub const REDIS_MAX_RETRY_DELAY: &str = "max_retry_delay";
+pub const REDIS_CONNECTION_TIMEOUT: &str = "connection_timeout";
+pub const REDIS_RESPONSE_TIMEOUT: &str = "response_timeout";
+pub const REDIS_PIPELINE_BUFFER_SIZE: &str = "pipeline_buffer_size";
+pub const REDIS_TLS_POLICY: &str = "tls_policy";
+pub const REDIS_TLS_CA: &str = "tls_ca";
+pub const REDIS_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const REDIS_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const REDIS_TLS_ALLOW_INSECURE: &str = "tls_allow_insecure";
+
+pub const POSTGRES_DSN_STRING: &str = BASE_DSN_STRING;
+pub const POSTGRES_TABLE: &str = "table";
+pub const POSTGRES_FORMAT: &str = "format";
+pub const POSTGRES_TLS_REQUIRED: &str = "tls_required";
+pub const POSTGRES_TLS_CA: &str = "tls_ca";
+pub const POSTGRES_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const POSTGRES_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const POSTGRES_QUEUE_DIR: &str = "queue_dir";
+pub const POSTGRES_QUEUE_LIMIT: &str = "queue_limit";
+
 /// Environment variable controlling whether target queue files are Snappy-compressed.
 /// Applies to both notify and audit target queue stores.
 pub const ENV_TARGET_STORE_COMPRESS: &str = "RUSTFS_TARGET_STORE_COMPRESS";

crates/config/src/lib.rs

@@ -31,6 +31,10 @@ pub use constants::env::*;
 #[cfg(feature = "constants")]
 pub use constants::heal::*;
 #[cfg(feature = "constants")]
+pub use constants::health::*;
+#[cfg(feature = "constants")]
+pub use constants::internode::*;
+#[cfg(feature = "constants")]
 pub use constants::object::*;
 #[cfg(feature = "constants")]
 pub use constants::profiler::*;

crates/config/src/notify/amqp.rs

@@ -0,0 +1,60 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const NOTIFY_AMQP_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::AMQP_URL,
+    crate::AMQP_EXCHANGE,
+    crate::AMQP_ROUTING_KEY,
+    crate::AMQP_MANDATORY,
+    crate::AMQP_PERSISTENT,
+    crate::AMQP_USERNAME,
+    crate::AMQP_PASSWORD,
+    crate::AMQP_TLS_CA,
+    crate::AMQP_TLS_CLIENT_CERT,
+    crate::AMQP_TLS_CLIENT_KEY,
+    crate::AMQP_QUEUE_DIR,
+    crate::AMQP_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_NOTIFY_AMQP_ENABLE: &str = "RUSTFS_NOTIFY_AMQP_ENABLE";
+pub const ENV_NOTIFY_AMQP_URL: &str = "RUSTFS_NOTIFY_AMQP_URL";
+pub const ENV_NOTIFY_AMQP_EXCHANGE: &str = "RUSTFS_NOTIFY_AMQP_EXCHANGE";
+pub const ENV_NOTIFY_AMQP_ROUTING_KEY: &str = "RUSTFS_NOTIFY_AMQP_ROUTING_KEY";
+pub const ENV_NOTIFY_AMQP_MANDATORY: &str = "RUSTFS_NOTIFY_AMQP_MANDATORY";
+pub const ENV_NOTIFY_AMQP_PERSISTENT: &str = "RUSTFS_NOTIFY_AMQP_PERSISTENT";
+pub const ENV_NOTIFY_AMQP_USERNAME: &str = "RUSTFS_NOTIFY_AMQP_USERNAME";
+pub const ENV_NOTIFY_AMQP_PASSWORD: &str = "RUSTFS_NOTIFY_AMQP_PASSWORD";
+pub const ENV_NOTIFY_AMQP_TLS_CA: &str = "RUSTFS_NOTIFY_AMQP_TLS_CA";
+pub const ENV_NOTIFY_AMQP_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_AMQP_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_AMQP_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_AMQP_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_AMQP_QUEUE_DIR: &str = "RUSTFS_NOTIFY_AMQP_QUEUE_DIR";
+pub const ENV_NOTIFY_AMQP_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_AMQP_QUEUE_LIMIT";
+
+pub const ENV_NOTIFY_AMQP_KEYS: &[&str; 13] = &[
+    ENV_NOTIFY_AMQP_ENABLE,
+    ENV_NOTIFY_AMQP_URL,
+    ENV_NOTIFY_AMQP_EXCHANGE,
+    ENV_NOTIFY_AMQP_ROUTING_KEY,
+    ENV_NOTIFY_AMQP_MANDATORY,
+    ENV_NOTIFY_AMQP_PERSISTENT,
+    ENV_NOTIFY_AMQP_USERNAME,
+    ENV_NOTIFY_AMQP_PASSWORD,
+    ENV_NOTIFY_AMQP_TLS_CA,
+    ENV_NOTIFY_AMQP_TLS_CLIENT_CERT,
+    ENV_NOTIFY_AMQP_TLS_CLIENT_KEY,
+    ENV_NOTIFY_AMQP_QUEUE_DIR,
+    ENV_NOTIFY_AMQP_QUEUE_LIMIT,
+];

crates/config/src/notify/kafka.rs

@@ -0,0 +1,53 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// A list of all valid configuration keys for a Kafka target.
+pub const NOTIFY_KAFKA_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::KAFKA_BROKERS,
+    crate::KAFKA_TOPIC,
+    crate::KAFKA_ACKS,
+    crate::KAFKA_TLS_ENABLE,
+    crate::KAFKA_TLS_CA,
+    crate::KAFKA_TLS_CLIENT_CERT,
+    crate::KAFKA_TLS_CLIENT_KEY,
+    crate::KAFKA_QUEUE_DIR,
+    crate::KAFKA_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+// Kafka Environment Variables
+pub const ENV_NOTIFY_KAFKA_ENABLE: &str = "RUSTFS_NOTIFY_KAFKA_ENABLE";
+pub const ENV_NOTIFY_KAFKA_BROKERS: &str = "RUSTFS_NOTIFY_KAFKA_BROKERS";
+pub const ENV_NOTIFY_KAFKA_TOPIC: &str = "RUSTFS_NOTIFY_KAFKA_TOPIC";
+pub const ENV_NOTIFY_KAFKA_ACKS: &str = "RUSTFS_NOTIFY_KAFKA_ACKS";
+pub const ENV_NOTIFY_KAFKA_TLS_ENABLE: &str = "RUSTFS_NOTIFY_KAFKA_TLS_ENABLE";
+pub const ENV_NOTIFY_KAFKA_TLS_CA: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CA";
+pub const ENV_NOTIFY_KAFKA_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_KAFKA_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_KAFKA_QUEUE_DIR: &str = "RUSTFS_NOTIFY_KAFKA_QUEUE_DIR";
+pub const ENV_NOTIFY_KAFKA_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_KAFKA_QUEUE_LIMIT";
+
+pub const ENV_NOTIFY_KAFKA_KEYS: &[&str; 10] = &[
+    ENV_NOTIFY_KAFKA_ENABLE,
+    ENV_NOTIFY_KAFKA_BROKERS,
+    ENV_NOTIFY_KAFKA_TOPIC,
+    ENV_NOTIFY_KAFKA_ACKS,
+    ENV_NOTIFY_KAFKA_TLS_ENABLE,
+    ENV_NOTIFY_KAFKA_TLS_CA,
+    ENV_NOTIFY_KAFKA_TLS_CLIENT_CERT,
+    ENV_NOTIFY_KAFKA_TLS_CLIENT_KEY,
+    ENV_NOTIFY_KAFKA_QUEUE_DIR,
+    ENV_NOTIFY_KAFKA_QUEUE_LIMIT,
+];

crates/config/src/notify/mod.rs

@@ -12,17 +12,27 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+mod amqp;
 mod arn;
+mod kafka;
 mod mqtt;
+mod mysql;
 mod nats;
+mod postgres;
 mod pulsar;
+mod redis;
 mod store;
 mod webhook;
 
+pub use amqp::*;
 pub use arn::*;
+pub use kafka::*;
 pub use mqtt::*;
+pub use mysql::*;
 pub use nats::*;
+pub use postgres::*;
 pub use pulsar::*;
+pub use redis::*;
 pub use store::*;
 pub use webhook::*;
 
@@ -67,30 +77,31 @@ pub const ENV_NOTIFY_SEND_CONCURRENCY: &str = "RUSTFS_NOTIFY_SEND_CONCURRENCY";
 /// Adjust this value based on your system's capabilities and expected load.
 pub const DEFAULT_NOTIFY_SEND_CONCURRENCY: usize = 64;
 
-#[allow(dead_code)]
 pub const NOTIFY_SUB_SYSTEMS: &[&str] = &[
+    NOTIFY_AMQP_SUB_SYS,
+    NOTIFY_KAFKA_SUB_SYS,
     NOTIFY_MQTT_SUB_SYS,
+    NOTIFY_MYSQL_SUB_SYS,
     NOTIFY_NATS_SUB_SYS,
+    NOTIFY_POSTGRES_SUB_SYS,
     NOTIFY_PULSAR_SUB_SYS,
+    NOTIFY_REDIS_SUB_SYS,
     NOTIFY_WEBHOOK_SUB_SYS,
 ];
 
-#[allow(dead_code)]
 pub const NOTIFY_KAFKA_SUB_SYS: &str = "notify_kafka";
 pub const NOTIFY_MQTT_SUB_SYS: &str = "notify_mqtt";
-#[allow(dead_code)]
-pub const NOTIFY_MY_SQL_SUB_SYS: &str = "notify_mysql";
+pub const NOTIFY_MYSQL_SUB_SYS: &str = "notify_mysql";
 #[allow(dead_code)]
 pub const NOTIFY_NATS_SUB_SYS: &str = "notify_nats";
 #[allow(dead_code)]
 pub const NOTIFY_NSQ_SUB_SYS: &str = "notify_nsq";
 #[allow(dead_code)]
 pub const NOTIFY_ES_SUB_SYS: &str = "notify_elasticsearch";
-#[allow(dead_code)]
 pub const NOTIFY_AMQP_SUB_SYS: &str = "notify_amqp";
-#[allow(dead_code)]
 pub const NOTIFY_POSTGRES_SUB_SYS: &str = "notify_postgres";
 #[allow(dead_code)]
 pub const NOTIFY_REDIS_SUB_SYS: &str = "notify_redis";
+pub const NOTIFY_REDIS_DEFAULT_CHANNEL: &str = "rustfs_notify_channel";
 pub const NOTIFY_PULSAR_SUB_SYS: &str = "notify_pulsar";
 pub const NOTIFY_WEBHOOK_SUB_SYS: &str = "notify_webhook";

crates/config/src/notify/mysql.rs

@@ -0,0 +1,53 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// A list of all valid configuration keys for a MySQL target.
+pub const NOTIFY_MYSQL_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::MYSQL_DSN_STRING,
+    crate::MYSQL_TABLE,
+    crate::MYSQL_FORMAT,
+    crate::MYSQL_TLS_CA,
+    crate::MYSQL_TLS_CLIENT_CERT,
+    crate::MYSQL_TLS_CLIENT_KEY,
+    crate::MYSQL_QUEUE_DIR,
+    crate::MYSQL_QUEUE_LIMIT,
+    crate::MYSQL_MAX_OPEN_CONNECTIONS,
+    crate::COMMENT_KEY,
+];
+
+// MySQL Environment Variables
+pub const ENV_NOTIFY_MYSQL_ENABLE: &str = "RUSTFS_NOTIFY_MYSQL_ENABLE";
+pub const ENV_NOTIFY_MYSQL_DSN_STRING: &str = "RUSTFS_NOTIFY_MYSQL_DSN_STRING";
+pub const ENV_NOTIFY_MYSQL_TABLE: &str = "RUSTFS_NOTIFY_MYSQL_TABLE";
+pub const ENV_NOTIFY_MYSQL_FORMAT: &str = "RUSTFS_NOTIFY_MYSQL_FORMAT";
+pub const ENV_NOTIFY_MYSQL_TLS_CA: &str = "RUSTFS_NOTIFY_MYSQL_TLS_CA";
+pub const ENV_NOTIFY_MYSQL_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_MYSQL_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_MYSQL_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_MYSQL_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_MYSQL_QUEUE_DIR: &str = "RUSTFS_NOTIFY_MYSQL_QUEUE_DIR";
+pub const ENV_NOTIFY_MYSQL_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_MYSQL_QUEUE_LIMIT";
+pub const ENV_NOTIFY_MYSQL_MAX_OPEN_CONNECTIONS: &str = "RUSTFS_NOTIFY_MYSQL_MAX_OPEN_CONNECTIONS";
+
+pub const ENV_NOTIFY_MYSQL_KEYS: &[&str; 10] = &[
+    ENV_NOTIFY_MYSQL_ENABLE,
+    ENV_NOTIFY_MYSQL_DSN_STRING,
+    ENV_NOTIFY_MYSQL_TABLE,
+    ENV_NOTIFY_MYSQL_FORMAT,
+    ENV_NOTIFY_MYSQL_TLS_CA,
+    ENV_NOTIFY_MYSQL_TLS_CLIENT_CERT,
+    ENV_NOTIFY_MYSQL_TLS_CLIENT_KEY,
+    ENV_NOTIFY_MYSQL_QUEUE_DIR,
+    ENV_NOTIFY_MYSQL_QUEUE_LIMIT,
+    ENV_NOTIFY_MYSQL_MAX_OPEN_CONNECTIONS,
+];

crates/config/src/notify/postgres.rs

@@ -0,0 +1,51 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const NOTIFY_POSTGRES_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::POSTGRES_DSN_STRING,
+    crate::POSTGRES_TABLE,
+    crate::POSTGRES_FORMAT,
+    crate::POSTGRES_TLS_REQUIRED,
+    crate::POSTGRES_TLS_CA,
+    crate::POSTGRES_TLS_CLIENT_CERT,
+    crate::POSTGRES_TLS_CLIENT_KEY,
+    crate::POSTGRES_QUEUE_DIR,
+    crate::POSTGRES_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_NOTIFY_POSTGRES_ENABLE: &str = "RUSTFS_NOTIFY_POSTGRES_ENABLE";
+pub const ENV_NOTIFY_POSTGRES_DSN_STRING: &str = "RUSTFS_NOTIFY_POSTGRES_DSN_STRING";
+pub const ENV_NOTIFY_POSTGRES_TABLE: &str = "RUSTFS_NOTIFY_POSTGRES_TABLE";
+pub const ENV_NOTIFY_POSTGRES_FORMAT: &str = "RUSTFS_NOTIFY_POSTGRES_FORMAT";
+pub const ENV_NOTIFY_POSTGRES_TLS_REQUIRED: &str = "RUSTFS_NOTIFY_POSTGRES_TLS_REQUIRED";
+pub const ENV_NOTIFY_POSTGRES_TLS_CA: &str = "RUSTFS_NOTIFY_POSTGRES_TLS_CA";
+pub const ENV_NOTIFY_POSTGRES_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_POSTGRES_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_POSTGRES_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_POSTGRES_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_POSTGRES_QUEUE_DIR: &str = "RUSTFS_NOTIFY_POSTGRES_QUEUE_DIR";
+pub const ENV_NOTIFY_POSTGRES_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_POSTGRES_QUEUE_LIMIT";
+
+pub const ENV_NOTIFY_POSTGRES_KEYS: &[&str; 10] = &[
+    ENV_NOTIFY_POSTGRES_ENABLE,
+    ENV_NOTIFY_POSTGRES_DSN_STRING,
+    ENV_NOTIFY_POSTGRES_TABLE,
+    ENV_NOTIFY_POSTGRES_FORMAT,
+    ENV_NOTIFY_POSTGRES_TLS_REQUIRED,
+    ENV_NOTIFY_POSTGRES_TLS_CA,
+    ENV_NOTIFY_POSTGRES_TLS_CLIENT_CERT,
+    ENV_NOTIFY_POSTGRES_TLS_CLIENT_KEY,
+    ENV_NOTIFY_POSTGRES_QUEUE_DIR,
+    ENV_NOTIFY_POSTGRES_QUEUE_LIMIT,
+];

crates/config/src/notify/redis.rs

@@ -0,0 +1,82 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// A list of all valid configuration keys for a Redis target.
+pub const NOTIFY_REDIS_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::REDIS_URL,
+    crate::REDIS_CHANNEL,
+    crate::REDIS_USERNAME,
+    crate::REDIS_PASSWORD,
+    crate::REDIS_KEEP_ALIVE_INTERVAL,
+    crate::REDIS_QUEUE_DIR,
+    crate::REDIS_QUEUE_LIMIT,
+    crate::REDIS_MAX_RETRY_ATTEMPTS,
+    crate::REDIS_RECONNECT_RETRY_ATTEMPTS,
+    crate::REDIS_MIN_RETRY_DELAY,
+    crate::REDIS_MAX_RETRY_DELAY,
+    crate::REDIS_CONNECTION_TIMEOUT,
+    crate::REDIS_RESPONSE_TIMEOUT,
+    crate::REDIS_PIPELINE_BUFFER_SIZE,
+    crate::REDIS_TLS_POLICY,
+    crate::REDIS_TLS_CA,
+    crate::REDIS_TLS_CLIENT_CERT,
+    crate::REDIS_TLS_CLIENT_KEY,
+    crate::REDIS_TLS_ALLOW_INSECURE,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_NOTIFY_REDIS_ENABLE: &str = "RUSTFS_NOTIFY_REDIS_ENABLE";
+pub const ENV_NOTIFY_REDIS_URL: &str = "RUSTFS_NOTIFY_REDIS_URL";
+pub const ENV_NOTIFY_REDIS_CHANNEL: &str = "RUSTFS_NOTIFY_REDIS_CHANNEL";
+pub const ENV_NOTIFY_REDIS_USERNAME: &str = "RUSTFS_NOTIFY_REDIS_USERNAME";
+pub const ENV_NOTIFY_REDIS_PASSWORD: &str = "RUSTFS_NOTIFY_REDIS_PASSWORD";
+pub const ENV_NOTIFY_REDIS_KEEP_ALIVE_INTERVAL: &str = "RUSTFS_NOTIFY_REDIS_KEEP_ALIVE_INTERVAL";
+pub const ENV_NOTIFY_REDIS_QUEUE_DIR: &str = "RUSTFS_NOTIFY_REDIS_QUEUE_DIR";
+pub const ENV_NOTIFY_REDIS_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_REDIS_QUEUE_LIMIT";
+pub const ENV_NOTIFY_REDIS_MAX_RETRY_ATTEMPTS: &str = "RUSTFS_NOTIFY_REDIS_MAX_RETRY_ATTEMPTS";
+pub const ENV_NOTIFY_REDIS_RECONNECT_RETRY_ATTEMPTS: &str = "RUSTFS_NOTIFY_REDIS_RECONNECT_RETRY_ATTEMPTS";
+pub const ENV_NOTIFY_REDIS_MIN_RETRY_DELAY: &str = "RUSTFS_NOTIFY_REDIS_MIN_RETRY_DELAY";
+pub const ENV_NOTIFY_REDIS_MAX_RETRY_DELAY: &str = "RUSTFS_NOTIFY_REDIS_MAX_RETRY_DELAY";
+pub const ENV_NOTIFY_REDIS_CONNECTION_TIMEOUT: &str = "RUSTFS_NOTIFY_REDIS_CONNECTION_TIMEOUT";
+pub const ENV_NOTIFY_REDIS_RESPONSE_TIMEOUT: &str = "RUSTFS_NOTIFY_REDIS_RESPONSE_TIMEOUT";
+pub const ENV_NOTIFY_REDIS_PIPELINE_BUFFER_SIZE: &str = "RUSTFS_NOTIFY_REDIS_PIPELINE_BUFFER_SIZE";
+pub const ENV_NOTIFY_REDIS_TLS_POLICY: &str = "RUSTFS_NOTIFY_REDIS_TLS_POLICY";
+pub const ENV_NOTIFY_REDIS_TLS_CA: &str = "RUSTFS_NOTIFY_REDIS_TLS_CA";
+pub const ENV_NOTIFY_REDIS_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_REDIS_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_REDIS_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_REDIS_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_REDIS_TLS_ALLOW_INSECURE: &str = "RUSTFS_NOTIFY_REDIS_TLS_ALLOW_INSECURE";
+
+pub const ENV_NOTIFY_REDIS_KEYS: &[&str; 20] = &[
+    ENV_NOTIFY_REDIS_ENABLE,
+    ENV_NOTIFY_REDIS_URL,
+    ENV_NOTIFY_REDIS_CHANNEL,
+    ENV_NOTIFY_REDIS_USERNAME,
+    ENV_NOTIFY_REDIS_PASSWORD,
+    ENV_NOTIFY_REDIS_KEEP_ALIVE_INTERVAL,
+    ENV_NOTIFY_REDIS_QUEUE_DIR,
+    ENV_NOTIFY_REDIS_QUEUE_LIMIT,
+    ENV_NOTIFY_REDIS_MAX_RETRY_ATTEMPTS,
+    ENV_NOTIFY_REDIS_RECONNECT_RETRY_ATTEMPTS,
+    ENV_NOTIFY_REDIS_MIN_RETRY_DELAY,
+    ENV_NOTIFY_REDIS_MAX_RETRY_DELAY,
+    ENV_NOTIFY_REDIS_CONNECTION_TIMEOUT,
+    ENV_NOTIFY_REDIS_RESPONSE_TIMEOUT,
+    ENV_NOTIFY_REDIS_PIPELINE_BUFFER_SIZE,
+    ENV_NOTIFY_REDIS_TLS_POLICY,
+    ENV_NOTIFY_REDIS_TLS_CA,
+    ENV_NOTIFY_REDIS_TLS_CLIENT_CERT,
+    ENV_NOTIFY_REDIS_TLS_CLIENT_KEY,
+    ENV_NOTIFY_REDIS_TLS_ALLOW_INSECURE,
+];

crates/config/src/observability/mod.rs

@@ -22,6 +22,14 @@ pub const ENV_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";
 pub const ENV_OBS_TRACE_ENDPOINT: &str = "RUSTFS_OBS_TRACE_ENDPOINT";
 pub const ENV_OBS_METRIC_ENDPOINT: &str = "RUSTFS_OBS_METRIC_ENDPOINT";
 pub const ENV_OBS_LOG_ENDPOINT: &str = "RUSTFS_OBS_LOG_ENDPOINT";
+pub const ENV_OBS_ENDPOINT_HEADERS: &str = "RUSTFS_OBS_ENDPOINT_HEADERS";
+pub const ENV_OBS_ENDPOINT_TRACES_HEADERS: &str = "RUSTFS_OBS_ENDPOINT_TRACES_HEADERS";
+pub const ENV_OBS_ENDPOINT_METRICS_HEADERS: &str = "RUSTFS_OBS_ENDPOINT_METRICS_HEADERS";
+pub const ENV_OBS_ENDPOINT_LOGS_HEADERS: &str = "RUSTFS_OBS_ENDPOINT_LOGS_HEADERS";
+pub const ENV_OBS_ENDPOINT_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_TIMEOUT_MILLIS";
+pub const ENV_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS";
+pub const ENV_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS";
+pub const ENV_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS";
 pub const ENV_OBS_PROFILING_ENDPOINT: &str = "RUSTFS_OBS_PROFILING_ENDPOINT";
 pub const ENV_OBS_USE_STDOUT: &str = "RUSTFS_OBS_USE_STDOUT";
 pub const ENV_OBS_SAMPLE_RATIO: &str = "RUSTFS_OBS_SAMPLE_RATIO";
@@ -108,6 +116,14 @@ mod tests {
         assert_eq!(ENV_OBS_TRACE_ENDPOINT, "RUSTFS_OBS_TRACE_ENDPOINT");
         assert_eq!(ENV_OBS_METRIC_ENDPOINT, "RUSTFS_OBS_METRIC_ENDPOINT");
         assert_eq!(ENV_OBS_LOG_ENDPOINT, "RUSTFS_OBS_LOG_ENDPOINT");
+        assert_eq!(ENV_OBS_ENDPOINT_HEADERS, "RUSTFS_OBS_ENDPOINT_HEADERS");
+        assert_eq!(ENV_OBS_ENDPOINT_TRACES_HEADERS, "RUSTFS_OBS_ENDPOINT_TRACES_HEADERS");
+        assert_eq!(ENV_OBS_ENDPOINT_METRICS_HEADERS, "RUSTFS_OBS_ENDPOINT_METRICS_HEADERS");
+        assert_eq!(ENV_OBS_ENDPOINT_LOGS_HEADERS, "RUSTFS_OBS_ENDPOINT_LOGS_HEADERS");
+        assert_eq!(ENV_OBS_ENDPOINT_TIMEOUT_MILLIS, "RUSTFS_OBS_ENDPOINT_TIMEOUT_MILLIS");
+        assert_eq!(ENV_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS, "RUSTFS_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS");
+        assert_eq!(ENV_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS, "RUSTFS_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS");
+        assert_eq!(ENV_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS, "RUSTFS_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS");
         assert_eq!(ENV_OBS_PROFILING_ENDPOINT, "RUSTFS_OBS_PROFILING_ENDPOINT");
         assert_eq!(ENV_OBS_USE_STDOUT, "RUSTFS_OBS_USE_STDOUT");
         assert_eq!(ENV_OBS_SAMPLE_RATIO, "RUSTFS_OBS_SAMPLE_RATIO");

crates/credentials/src/constants.rs

@@ -66,7 +66,7 @@ mod tests {
 
         // In production environment, access key and secret key should be different
         // These are default values, so being the same is acceptable, but should be warned in documentation
-        println!("Warning: Default access key and secret key are the same. Change them in production!");
+        assert_eq!(DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY);
     }
 
     #[test]
@@ -74,10 +74,8 @@ mod tests {
         // Test security best practices
 
         // These are default values, should be changed in production environments
-        println!("Security Warning: Default credentials detected!");
-        println!("Access Key: {DEFAULT_ACCESS_KEY}");
-        println!("Secret Key: {DEFAULT_SECRET_KEY}");
-        println!("These should be changed in production environments!");
+        assert_eq!(DEFAULT_ACCESS_KEY, "rustfsadmin");
+        assert_eq!(DEFAULT_SECRET_KEY, "rustfsadmin");
 
         // Verify that key lengths meet minimum security requirements
         assert!(DEFAULT_ACCESS_KEY.len() >= 8, "Access key should be at least 8 characters");

crates/credentials/src/credentials.rs

@@ -29,6 +29,12 @@ static GLOBAL_ACTIVE_CRED: OnceLock<Credentials> = OnceLock::new();
 /// Global RPC authentication token
 pub static GLOBAL_RUSTFS_RPC_SECRET: OnceLock<String> = OnceLock::new();
 
+/// Public error returned when RPC authentication is not safely configured.
+pub const RPC_SECRET_REQUIRED_MESSAGE: &str = "RPC authentication secret is not configured";
+
+/// Operator-facing guidance for configuring RPC authentication safely.
+pub const RPC_SECRET_REQUIRED_OPERATOR_MESSAGE: &str = "RUSTFS_RPC_SECRET must be set to a non-default value or RUSTFS_SECRET_KEY must be changed from the default for RPC authentication";
+
 /// Error type for credentials operations
 #[derive(Debug)]
 pub enum CredentialsError {
@@ -216,13 +222,39 @@ pub fn gen_secret_key(length: usize) -> std::io::Result<String> {
 /// # Returns
 /// * `String` - The RPC authentication token
 ///
+fn resolve_rpc_secret(env_secret: Option<&str>, global_secret: Option<&str>) -> Option<String> {
+    if let Some(secret) = env_secret.map(str::trim).filter(|secret| !secret.is_empty()) {
+        return (secret != DEFAULT_SECRET_KEY).then(|| secret.to_string());
+    }
+
+    global_secret
+        .map(str::trim)
+        .filter(|secret| !secret.is_empty() && *secret != DEFAULT_SECRET_KEY)
+        .map(ToOwned::to_owned)
+}
+
+pub fn try_get_rpc_token() -> std::io::Result<String> {
+    if let Some(secret) = GLOBAL_RUSTFS_RPC_SECRET.get() {
+        return resolve_rpc_secret(None, Some(secret)).ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE));
+    }
+
+    let env_secret = env::var(ENV_RPC_SECRET).ok();
+    let global_secret = get_global_secret_key_opt();
+    let secret = resolve_rpc_secret(env_secret.as_deref(), global_secret.as_deref())
+        .ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE))?;
+
+    match GLOBAL_RUSTFS_RPC_SECRET.set(secret.clone()) {
+        Ok(()) => Ok(secret),
+        Err(_) => GLOBAL_RUSTFS_RPC_SECRET
+            .get()
+            .and_then(|stored| resolve_rpc_secret(None, Some(stored)))
+            .ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE)),
+    }
+}
+
+#[deprecated(note = "use try_get_rpc_token to handle missing RPC secrets explicitly")]
 pub fn get_rpc_token() -> String {
-    GLOBAL_RUSTFS_RPC_SECRET
-        .get_or_init(|| {
-            env::var(ENV_RPC_SECRET)
-                .unwrap_or_else(|_| get_global_secret_key_opt().unwrap_or_else(|| DEFAULT_SECRET_KEY.to_string()))
-        })
-        .clone()
+    try_get_rpc_token().expect(RPC_SECRET_REQUIRED_MESSAGE)
 }
 
 /// A wrapper struct for masking sensitive strings in Debug implementations.
@@ -301,7 +333,7 @@ impl fmt::Debug for Credentials {
         f.debug_struct("Credentials")
             .field("access_key", &self.access_key)
             .field("secret_key", &Masked(Some(&self.secret_key)))
-            .field("session_token", &self.session_token)
+            .field("session_token", &Masked(Some(&self.session_token)))
             .field("expiration", &self.expiration)
             .field("status", &self.status)
             .field("parent_user", &self.parent_user)
@@ -495,6 +527,63 @@ mod tests {
         assert!(!key.contains('='));
     }
 
+    #[test]
+    fn test_resolve_rpc_secret_rejects_default_fallback() {
+        assert!(resolve_rpc_secret(None, None).is_none());
+        assert!(resolve_rpc_secret(None, Some(DEFAULT_SECRET_KEY)).is_none());
+        assert!(resolve_rpc_secret(Some(DEFAULT_SECRET_KEY), Some("custom-global-secret")).is_none());
+    }
+
+    #[test]
+    fn test_rpc_secret_public_error_omits_configuration_details() {
+        assert!(!RPC_SECRET_REQUIRED_MESSAGE.contains("RUSTFS_"));
+        assert!(!RPC_SECRET_REQUIRED_MESSAGE.contains(DEFAULT_SECRET_KEY));
+        assert!(RPC_SECRET_REQUIRED_OPERATOR_MESSAGE.contains("RUSTFS_RPC_SECRET"));
+    }
+
+    #[allow(deprecated)]
+    #[test]
+    fn test_get_rpc_token_preserves_string_return_type() {
+        fn assert_string_return(_: fn() -> String) {}
+
+        assert_string_return(get_rpc_token);
+    }
+
+    #[test]
+    fn test_resolve_rpc_secret_accepts_non_default_secret() {
+        assert_eq!(resolve_rpc_secret(Some("custom-rpc-secret"), None).as_deref(), Some("custom-rpc-secret"));
+        assert_eq!(
+            resolve_rpc_secret(None, Some("custom-global-secret")).as_deref(),
+            Some("custom-global-secret")
+        );
+    }
+
+    #[test]
+    fn test_resolve_rpc_secret_trims_and_falls_back_from_blank_env() {
+        assert_eq!(
+            resolve_rpc_secret(Some("  custom-rpc-secret  "), None).as_deref(),
+            Some("custom-rpc-secret")
+        );
+        assert_eq!(
+            resolve_rpc_secret(Some(""), Some("custom-global-secret")).as_deref(),
+            Some("custom-global-secret")
+        );
+        assert_eq!(
+            resolve_rpc_secret(Some("  "), Some("  custom-global-secret  ")).as_deref(),
+            Some("custom-global-secret")
+        );
+        assert_eq!(
+            resolve_rpc_secret(Some("  "), Some("custom-global-secret")).as_deref(),
+            Some("custom-global-secret")
+        );
+    }
+
+    #[test]
+    fn test_resolve_rpc_secret_returns_none_for_trimmed_default_secret() {
+        let padded_default_secret = format!("  {}  ", DEFAULT_SECRET_KEY);
+        assert!(resolve_rpc_secret(Some(padded_default_secret.as_str()), Some("custom-global-secret")).is_none());
+    }
+
     #[test]
     fn test_masked_debug() {
         // Test None
@@ -524,6 +613,24 @@ mod tests {
         assert_eq!(format!("{:?}", Masked(Some("中文测试"))), "中***试|4");
     }
 
+    #[test]
+    fn test_credentials_debug_masks_sensitive_fields() {
+        let cred = Credentials {
+            access_key: "debug-access-key".to_string(),
+            secret_key: "debug-secret-key".to_string(),
+            session_token: "debug-session-token".to_string(),
+            parent_user: "parent-user".to_string(),
+            ..Default::default()
+        };
+
+        let output = format!("{cred:?}");
+
+        assert!(output.contains("debug-access-key"));
+        assert!(output.contains("parent-user"));
+        assert!(!output.contains("debug-secret-key"));
+        assert!(!output.contains("debug-session-token"));
+    }
+
     #[test]
     fn test_credentials_expiration_serialize_as_rfc3339() {
         use time::OffsetDateTime;

crates/crypto/Cargo.toml

@@ -31,7 +31,6 @@ workspace = true
 [dependencies]
 aes-gcm = { workspace = true, optional = true }
 argon2 = { workspace = true, optional = true }
-cfg-if = { workspace = true }
 chacha20poly1305 = { workspace = true, optional = true }
 jsonwebtoken = { workspace = true }
 pbkdf2 = { workspace = true, optional = true }

crates/crypto/src/encdec/aes.rs

@@ -13,19 +13,23 @@
 // limitations under the License.
 
 pub fn native_aes() -> bool {
-    cfg_if::cfg_if! {
-        if #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] {
+    cfg_select! {
+        any(target_arch = "x86", target_arch = "x86_64") => {
             std::is_x86_feature_detected!("aes") && std::is_x86_feature_detected!("pclmulqdq")
-        } else if #[cfg(target_arch = "aarch64")] {
+        }
+        target_arch = "aarch64" => {
             std::arch::is_aarch64_feature_detected!("aes")
-        } else if #[cfg(target_arch = "powerpc64")] {
+        }
+        target_arch = "powerpc64" => {
             false
-        } else if #[cfg(target_arch = "s390x")] {
+        }
+        target_arch = "s390x" => {
             std::is_s390x_feature_detected!("aes")
                 && std::is_s390x_feature_detected!("aescbc")
                 && std::is_s390x_feature_detected!("aesctr")
                 && (std::is_s390x_feature_detected!("aesgcm") || std::is_s390x_feature_detected!("ghash"))
-        } else {
+        }
+        _ => {
             false
         }
     }

crates/e2e_test/Cargo.toml

@@ -26,8 +26,10 @@ workspace = true
 [features]
 default = []
 ftps = []
+sftp = []
 
 [dependencies]
+rustfs-config = { workspace = true, features = ["constants"] }
 rustfs-ecstore.workspace = true
 rustfs-common.workspace = true
 rustfs-rio.workspace = true
@@ -72,4 +74,7 @@ suppaftp = { workspace = true, features = ["tokio", "rustls-aws-lc-rs"] }
 rcgen.workspace = true
 anyhow.workspace = true
 rustls.workspace = true
+russh = { workspace = true }
+russh-sftp = { workspace = true }
 zip.workspace = true
+clap.workspace = true

crates/e2e_test/src/bin/tls_gen.rs

@@ -0,0 +1,22 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use clap::Parser;
+use e2e_test::tls_gen::{Args, run};
+
+fn main() -> anyhow::Result<()> {
+    let out_dir = run(Args::parse())?;
+    println!("Generated RustFS TLS bundle in {}", out_dir.display());
+    Ok(())
+}

crates/e2e_test/src/common.rs

@@ -41,7 +41,9 @@ use walkdir::WalkDir;
 // Common constants for all E2E tests
 pub const DEFAULT_ACCESS_KEY: &str = "rustfsadmin";
 pub const DEFAULT_SECRET_KEY: &str = "rustfsadmin";
+pub const ENV_RUSTFS_BUILD_FEATURES: &str = "RUSTFS_BUILD_FEATURES";
 pub const TEST_BUCKET: &str = "e2e-test-bucket";
+const RUSTFS_FULL_FEATURE: &str = "full";
 
 fn build_test_s3_config(endpoint_url: &str, access_key: &str, secret_key: &str, provider_name: &'static str) -> Config {
     let credentials = Credentials::new(access_key, secret_key, None, None, provider_name);
@@ -83,6 +85,7 @@ pub fn rustfs_binary_path_with_features(requested_features: Option<&str>) -> Pat
     if let Some(path) = std::env::var_os("CARGO_BIN_EXE_rustfs") {
         return PathBuf::from(path);
     }
+    let requested_features = requested_features.and_then(normalize_rustfs_build_features);
 
     let mut binary_path = workspace_root();
     binary_path.push("target");
@@ -90,7 +93,7 @@ pub fn rustfs_binary_path_with_features(requested_features: Option<&str>) -> Pat
     binary_path.push(profile_dir);
     binary_path.push(format!("rustfs{}", std::env::consts::EXE_SUFFIX));
 
-    let features_match = binary_features_match(&binary_path, requested_features);
+    let features_match = binary_features_match(&binary_path, requested_features.as_deref());
     let source_is_newer = workspace_sources_newer_than_binary(&binary_path);
     let can_reuse_inside_e2e = running_inside_e2e_test_binary() && requested_features.is_none() && features_match;
     if binary_path.is_file() && features_match && (!source_is_newer || can_reuse_inside_e2e) {
@@ -105,7 +108,7 @@ pub fn rustfs_binary_path_with_features(requested_features: Option<&str>) -> Pat
     }
 
     info!("Building RustFS binary to ensure it's up to date...");
-    build_rustfs_binary(requested_features);
+    build_rustfs_binary(requested_features.as_deref());
 
     info!("Using RustFS binary at {:?}", binary_path);
     binary_path
@@ -134,11 +137,31 @@ fn running_inside_e2e_test_binary() -> bool {
     std::env::var("CARGO_PKG_NAME").is_ok_and(|value| value == "e2e_test")
 }
 
-fn requested_rustfs_build_features() -> Option<String> {
-    std::env::var("RUSTFS_BUILD_FEATURES")
+pub fn requested_rustfs_build_features() -> Option<String> {
+    std::env::var(ENV_RUSTFS_BUILD_FEATURES)
         .ok()
-        .map(|value| value.trim().to_string())
-        .filter(|value| !value.is_empty())
+        .and_then(|value| normalize_rustfs_build_features(&value))
+}
+
+pub fn normalize_rustfs_build_features(features: &str) -> Option<String> {
+    let features = features
+        .split(',')
+        .map(str::trim)
+        .filter(|feature| !feature.is_empty())
+        .map(str::to_ascii_lowercase)
+        .collect::<Vec<_>>();
+
+    if features.is_empty() { None } else { Some(features.join(",")) }
+}
+
+pub fn rustfs_build_feature_enabled(requested_features: Option<&str>, required_feature: &str) -> bool {
+    let Some(requested_features) = requested_features.and_then(normalize_rustfs_build_features) else {
+        return true;
+    };
+
+    requested_features
+        .split(',')
+        .any(|feature| feature.eq_ignore_ascii_case(RUSTFS_FULL_FEATURE) || feature.eq_ignore_ascii_case(required_feature))
 }
 
 fn rustfs_binary_features_stamp_path(binary_path: &Path) -> PathBuf {
@@ -147,11 +170,14 @@ fn rustfs_binary_features_stamp_path(binary_path: &Path) -> PathBuf {
 
 fn binary_features_match(binary_path: &Path, requested_features: Option<&str>) -> bool {
     let stamp_path = rustfs_binary_features_stamp_path(binary_path);
-    let recorded = stdfs::read_to_string(stamp_path).ok().map(|value| value.trim().to_string());
+    let recorded = stdfs::read_to_string(stamp_path)
+        .ok()
+        .and_then(|value| normalize_rustfs_build_features(&value));
+    let requested = requested_features.and_then(normalize_rustfs_build_features);
 
-    match requested_features {
+    match requested.as_deref() {
         Some(features) => recorded.as_deref() == Some(features),
-        None => recorded.as_deref().is_none_or(str::is_empty),
+        None => recorded.is_none(),
     }
 }
 
@@ -351,6 +377,7 @@ impl RustFSTestEnvironment {
     async fn start_rustfs_server_inner(
         &mut self,
         extra_args: Vec<&str>,
+        extra_env: &[(&str, &str)],
         cleanup_existing: bool,
     ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         if cleanup_existing {
@@ -362,10 +389,12 @@ impl RustFSTestEnvironment {
         info!("Starting RustFS server with args: {:?}", args);
 
         let binary_path = rustfs_binary_path();
-        let process = Command::new(&binary_path)
-            .env("RUST_LOG", "rustfs=info,rustfs_notify=debug")
-            .args(&args)
-            .spawn()?;
+        let mut command = Command::new(&binary_path);
+        command.env("RUST_LOG", "rustfs=info,rustfs_notify=debug");
+        for (key, value) in extra_env {
+            command.env(key, value);
+        }
+        let process = command.args(&args).spawn()?;
 
         self.process = Some(process);
 
@@ -377,7 +406,16 @@ impl RustFSTestEnvironment {
 
     /// Start RustFS server with basic configuration
     pub async fn start_rustfs_server(&mut self, extra_args: Vec<&str>) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        self.start_rustfs_server_inner(extra_args, true).await
+        self.start_rustfs_server_inner(extra_args, &[], true).await
+    }
+
+    /// Start RustFS server with extra child-process environment variables.
+    pub async fn start_rustfs_server_with_env(
+        &mut self,
+        extra_args: Vec<&str>,
+        extra_env: &[(&str, &str)],
+    ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+        self.start_rustfs_server_inner(extra_args, extra_env, true).await
     }
 
     /// Start RustFS server without cleaning up other running RustFS processes.
@@ -388,7 +426,7 @@ impl RustFSTestEnvironment {
         &mut self,
         extra_args: Vec<&str>,
     ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        self.start_rustfs_server_inner(extra_args, false).await
+        self.start_rustfs_server_inner(extra_args, &[], false).await
     }
 
     /// Wait for RustFS server to be ready.
@@ -888,3 +926,36 @@ impl Drop for RustFSTestClusterEnvironment {
         }
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn normalizes_rustfs_build_features() {
+        assert_eq!(
+            normalize_rustfs_build_features(" SFTP, ftps ,, WebDAV "),
+            Some("sftp,ftps,webdav".to_string())
+        );
+        assert_eq!(normalize_rustfs_build_features(" , "), None);
+    }
+
+    #[test]
+    fn full_feature_enables_any_required_feature() {
+        assert!(rustfs_build_feature_enabled(Some("full"), "sftp"));
+        assert!(rustfs_build_feature_enabled(Some("ftps, full"), "webdav"));
+    }
+
+    #[test]
+    fn binary_feature_stamp_matching_uses_normalized_features() {
+        let binary_path = std::env::temp_dir().join(format!("rustfs-feature-stamp-test-{}", Uuid::new_v4()));
+        let stamp_path = rustfs_binary_features_stamp_path(&binary_path);
+
+        stdfs::write(&stamp_path, " SFTP, ftps ").expect("write feature stamp");
+        assert!(binary_features_match(&binary_path, Some("sftp,ftps")));
+        assert!(binary_features_match(&binary_path, Some(" SFTP, FTPS ")));
+        assert!(!binary_features_match(&binary_path, Some("sftp")));
+
+        stdfs::remove_file(stamp_path).ok();
+    }
+}

crates/e2e_test/src/copy_object_metadata_test.rs

@@ -0,0 +1,144 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! CopyObject metadata replacement regression tests.
+
+#[cfg(test)]
+mod tests {
+    use crate::common::{RustFSTestEnvironment, init_logging};
+    use aws_sdk_s3::primitives::ByteStream;
+    use aws_sdk_s3::types::MetadataDirective;
+    use serial_test::serial;
+    use tracing::info;
+
+    #[tokio::test]
+    #[serial]
+    async fn test_self_copy_replace_metadata_preserves_readable_object() {
+        init_logging();
+        info!("Issue #2789: self-copy metadata replacement must preserve object data");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = env.create_s3_client();
+        let bucket = "self-copy-metadata-replace-test";
+        let key = "assets/chunk-2F3R7JUG.js";
+        let content = b"console.log('metadata replacement should keep object data readable');";
+
+        client
+            .create_bucket()
+            .bucket(bucket)
+            .send()
+            .await
+            .expect("Failed to create bucket");
+
+        client
+            .put_object()
+            .bucket(bucket)
+            .key(key)
+            .content_type("text/javascript; charset=utf-8")
+            .metadata("mtime", "1777992333")
+            .metadata("stale", "must-be-removed")
+            .body(ByteStream::from_static(content))
+            .send()
+            .await
+            .expect("PUT failed");
+
+        client
+            .copy_object()
+            .bucket(bucket)
+            .key(key)
+            .copy_source(format!("{bucket}/{key}"))
+            .metadata_directive(MetadataDirective::Replace)
+            .content_type("text/javascript; charset=utf-8")
+            .metadata("mtime", "1777992348")
+            .send()
+            .await
+            .expect("self CopyObject with metadata replacement failed");
+
+        let head_resp = client
+            .head_object()
+            .bucket(bucket)
+            .key(key)
+            .send()
+            .await
+            .expect("HEAD failed after self-copy");
+        assert_eq!(head_resp.content_length(), Some(content.len() as i64));
+        assert_eq!(
+            head_resp.metadata().and_then(|metadata| metadata.get("mtime")),
+            Some(&"1777992348".to_string()),
+            "HEAD should return replaced metadata"
+        );
+        assert_eq!(
+            head_resp.metadata().and_then(|metadata| metadata.get("stale")),
+            None,
+            "HEAD should not return metadata omitted by REPLACE"
+        );
+
+        let get_resp = client
+            .get_object()
+            .bucket(bucket)
+            .key(key)
+            .send()
+            .await
+            .expect("GET failed after self-copy");
+        let body = get_resp
+            .body
+            .collect()
+            .await
+            .expect("Failed to collect GET body")
+            .into_bytes();
+        assert_eq!(body.as_ref(), content, "self-copy metadata replacement must not drop object data");
+
+        client
+            .copy_object()
+            .bucket(bucket)
+            .key(key)
+            .copy_source(format!("{bucket}/{key}"))
+            .metadata_directive(MetadataDirective::Replace)
+            .send()
+            .await
+            .expect("self CopyObject with empty metadata replacement failed");
+
+        let empty_head_resp = client
+            .head_object()
+            .bucket(bucket)
+            .key(key)
+            .send()
+            .await
+            .expect("HEAD failed after empty metadata replacement");
+        assert_eq!(
+            empty_head_resp.metadata().and_then(|metadata| metadata.get("mtime")),
+            None,
+            "HEAD should not return metadata omitted by empty REPLACE"
+        );
+
+        let empty_get_resp = client
+            .get_object()
+            .bucket(bucket)
+            .key(key)
+            .send()
+            .await
+            .expect("GET failed after empty metadata replacement");
+        let empty_body = empty_get_resp
+            .body
+            .collect()
+            .await
+            .expect("Failed to collect GET body after empty metadata replacement")
+            .into_bytes();
+        assert_eq!(empty_body.as_ref(), content, "empty metadata replacement must not drop object data");
+
+        env.stop_server();
+    }
+}

crates/e2e_test/src/delete_object_no_content_length_test.rs

@@ -0,0 +1,165 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Regression coverage for signed `DELETE Object?versionId` requests sent with
+//! no body and no `Content-Length` header.
+
+#[cfg(test)]
+mod tests {
+    use crate::common::{RustFSTestEnvironment, init_logging};
+    use aws_sdk_s3::primitives::ByteStream;
+    use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
+    use http::header::HOST;
+    use rustfs_signer::constants::UNSIGNED_PAYLOAD;
+    use rustfs_signer::sign_v4;
+    use s3s::Body;
+    use serial_test::serial;
+    use std::error::Error;
+    use tokio::io::{AsyncReadExt, AsyncWriteExt};
+    use tokio::net::TcpStream;
+    use tokio::time::{Duration, timeout};
+    use tracing::info;
+
+    const RAW_RESPONSE_TIMEOUT: Duration = Duration::from_secs(10);
+
+    fn parse_status(raw_response: &str) -> Option<u16> {
+        raw_response.lines().next()?.split_whitespace().nth(1)?.parse().ok()
+    }
+
+    fn response_body(raw_response: &str) -> &str {
+        raw_response
+            .split_once("\r\n\r\n")
+            .map(|(_, body)| body)
+            .or_else(|| raw_response.split_once("\n\n").map(|(_, body)| body))
+            .unwrap_or("")
+    }
+
+    async fn signed_delete_without_content_length(
+        url: &str,
+        access_key: &str,
+        secret_key: &str,
+    ) -> Result<String, Box<dyn Error + Send + Sync>> {
+        let uri = url.parse::<http::Uri>()?;
+        let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
+        let path_and_query = uri.path_and_query().ok_or("request URL missing path")?.as_str().to_string();
+
+        let request = http::Request::builder()
+            .method(http::Method::DELETE)
+            .uri(uri)
+            .header(HOST, authority.clone())
+            .header("x-amz-content-sha256", UNSIGNED_PAYLOAD)
+            .body(Body::empty())?;
+
+        let signed = sign_v4(request, 0, access_key, secret_key, "", "us-east-1");
+
+        let mut raw_request = format!("DELETE {path_and_query} HTTP/1.1\r\nHost: {authority}\r\nConnection: close\r\n");
+        for (name, value) in signed.headers() {
+            if name.as_str().eq_ignore_ascii_case("host") || name.as_str().eq_ignore_ascii_case("content-length") {
+                continue;
+            }
+            raw_request.push_str(name.as_str());
+            raw_request.push_str(": ");
+            raw_request.push_str(value.to_str()?);
+            raw_request.push_str("\r\n");
+        }
+        raw_request.push_str("\r\n");
+
+        assert!(
+            !raw_request.to_ascii_lowercase().contains("\r\ncontent-length:"),
+            "raw regression request must omit Content-Length; request was:\n{raw_request}"
+        );
+
+        let mut stream = TcpStream::connect(&authority).await?;
+        stream.write_all(raw_request.as_bytes()).await?;
+        stream.flush().await?;
+
+        let mut response = Vec::new();
+        timeout(RAW_RESPONSE_TIMEOUT, stream.read_to_end(&mut response))
+            .await
+            .map_err(|_| std::io::Error::new(std::io::ErrorKind::TimedOut, "timed out reading raw DELETE response"))??;
+        Ok(String::from_utf8_lossy(&response).into_owned())
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn test_delete_object_version_without_content_length_succeeds() -> Result<(), Box<dyn Error + Send + Sync>> {
+        init_logging();
+        info!("🧪 TEST: signed DELETE Object?versionId succeeds without Content-Length");
+
+        let mut env = RustFSTestEnvironment::new().await?;
+        env.start_rustfs_server(vec![]).await?;
+
+        let client = env.create_s3_client();
+        let bucket = "delete-version-no-content-length";
+        let key = "versioned-delete-target.txt";
+
+        client.create_bucket().bucket(bucket).send().await?;
+
+        client
+            .put_object()
+            .bucket(bucket)
+            .key(key)
+            .body(ByteStream::from_static(b"delete me after versioning is enabled"))
+            .send()
+            .await?;
+
+        client
+            .put_bucket_versioning()
+            .bucket(bucket)
+            .versioning_configuration(
+                VersioningConfiguration::builder()
+                    .status(BucketVersioningStatus::Enabled)
+                    .build(),
+            )
+            .send()
+            .await?;
+
+        let listed_versions = client.list_object_versions().bucket(bucket).prefix(key).send().await?;
+        let version_id = listed_versions
+            .versions()
+            .iter()
+            .find(|version| version.key() == Some(key))
+            .and_then(|version| version.version_id())
+            .ok_or("ListObjectVersions did not return the pre-versioning object version")?;
+
+        let url = format!("{}/{}/{}?versionId={}", env.url, bucket, key, urlencoding::encode(version_id));
+        let raw_response = signed_delete_without_content_length(&url, &env.access_key, &env.secret_key).await?;
+        info!("raw DELETE response:\n{}", raw_response);
+
+        assert_eq!(
+            parse_status(&raw_response),
+            Some(204),
+            "DELETE Object?versionId without Content-Length should succeed, got:\n{raw_response}"
+        );
+        assert!(
+            !raw_response.contains("MissingContentLength"),
+            "DELETE Object?versionId without Content-Length regressed to MissingContentLength: {raw_response}"
+        );
+        assert!(
+            response_body(&raw_response).trim().is_empty(),
+            "successful DELETE Object?versionId should not return an error body: {raw_response}"
+        );
+
+        let get_deleted_version = client
+            .get_object()
+            .bucket(bucket)
+            .key(key)
+            .version_id(version_id)
+            .send()
+            .await;
+        assert!(get_deleted_version.is_err(), "explicitly deleted version should no longer be readable");
+
+        Ok(())
+    }
+}

crates/e2e_test/src/existing_object_tag_policy_test.rs

@@ -21,7 +21,7 @@ use crate::common::{
 };
 use aws_sdk_s3::config::{Credentials, Region};
 use aws_sdk_s3::primitives::ByteStream;
-use aws_sdk_s3::types::{Tag, Tagging};
+use aws_sdk_s3::types::{Delete, ObjectIdentifier, Tag, Tagging};
 use aws_sdk_s3::{Client, Config};
 use serial_test::serial;
 use tracing::info;
@@ -318,11 +318,18 @@ async fn test_e2e_sts_assume_role_session_policy_existing_object_tag() -> Result
 
     let rw = serde_json::to_string(&serde_json::json!({
         "Version": "2012-10-17",
-        "Statement": [{
-            "Effect": "Allow",
-            "Action": ["s3:*"],
-            "Resource": ["arn:aws:s3:::*"]
-        }]
+        "Statement": [
+            {
+                "Effect": "Allow",
+                "Action": ["s3:*"],
+                "Resource": ["arn:aws:s3:::*"]
+            },
+            {
+                "Effect": "Allow",
+                "Action": ["sts:AssumeRole"],
+                "Resource": ["arn:aws:s3:::*"]
+            }
+        ]
     }))?;
     admin_add_canned_policy(&env, &policy_readwrite, &rw).await?;
     admin_attach_policy_to_user(&env, &policy_readwrite, &parent).await?;
@@ -362,3 +369,114 @@ async fn test_e2e_sts_assume_role_session_policy_existing_object_tag() -> Result
     info!("test_e2e_sts_assume_role_session_policy_existing_object_tag passed");
     Ok(())
 }
+
+/// STS inline session policy: DeleteObjects must evaluate `s3:DeleteObject` per requested object key.
+#[tokio::test]
+#[serial]
+async fn test_e2e_sts_session_policy_delete_objects_object_prefix_only() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    init_logging();
+    if !awscurl_available() {
+        info!("Skipping test_e2e_sts_session_policy_delete_objects_object_prefix_only: awscurl not available");
+        return Ok(());
+    }
+
+    let suffix = Uuid::new_v4();
+    let parent = format!("e2e-sts-del-par-{suffix}");
+    let parent_secret = "longSecretKeyForParentDelete99!";
+    let policy_readwrite = format!("e2e-sts-del-rw-{suffix}");
+    let bucket = format!("e2e-sts-del-bkt-{suffix}");
+    let allowed_key = "allowed/table/data.parquet";
+    let denied_key = "denied/table/data.parquet";
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+
+    let admin = env.create_s3_client();
+    admin_create_user(&env, &parent, parent_secret).await?;
+
+    let rw = serde_json::to_string(&serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [
+            {
+                "Effect": "Allow",
+                "Action": ["s3:*"],
+                "Resource": ["arn:aws:s3:::*"]
+            },
+            {
+                "Effect": "Allow",
+                "Action": ["sts:AssumeRole"],
+                "Resource": ["arn:aws:s3:::*"]
+            }
+        ]
+    }))?;
+    admin_add_canned_policy(&env, &policy_readwrite, &rw).await?;
+    admin_attach_policy_to_user(&env, &policy_readwrite, &parent).await?;
+
+    let parent_client = user_client(&env, &parent, parent_secret);
+    parent_client.create_bucket().bucket(&bucket).send().await?;
+    parent_client
+        .put_object()
+        .bucket(&bucket)
+        .key(allowed_key)
+        .body(ByteStream::from_static(b"allowed-delete-data"))
+        .send()
+        .await?;
+    parent_client
+        .put_object()
+        .bucket(&bucket)
+        .key(denied_key)
+        .body(ByteStream::from_static(b"denied-delete-data"))
+        .send()
+        .await?;
+
+    let session_policy = serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [{
+            "Effect": "Allow",
+            "Action": ["s3:DeleteObject"],
+            "Resource": [format!("arn:aws:s3:::{}/allowed/*", bucket)]
+        }]
+    })
+    .to_string();
+
+    let (ak, sk, token) = assume_role_with_session_policy(&env, &parent, parent_secret, &session_policy).await?;
+    let session_client = sts_session_client(&env, &ak, &sk, &token);
+
+    let delete = Delete::builder()
+        .objects(ObjectIdentifier::builder().key(allowed_key).build()?)
+        .objects(ObjectIdentifier::builder().key(denied_key).build()?)
+        .build()?;
+
+    let result = session_client.delete_objects().bucket(&bucket).delete(delete).send().await?;
+
+    assert_eq!(result.deleted().len(), 1, "only the allowed-prefix object should be deleted");
+    assert!(
+        result.deleted().iter().any(|deleted| deleted.key() == Some(allowed_key)),
+        "DeleteObjects response should report the allowed-prefix object as deleted"
+    );
+
+    assert_eq!(result.errors().len(), 1, "the out-of-prefix object should return one per-key error");
+    let error = &result.errors()[0];
+    assert_eq!(error.key(), Some(denied_key));
+    assert_eq!(error.code(), Some("AccessDenied"));
+
+    let allowed_head = parent_client.head_object().bucket(&bucket).key(allowed_key).send().await;
+    assert!(allowed_head.is_err(), "allowed-prefix object should have been deleted");
+
+    parent_client
+        .head_object()
+        .bucket(&bucket)
+        .key(denied_key)
+        .send()
+        .await
+        .expect("out-of-prefix object should remain after per-key AccessDenied");
+
+    let _ = admin.delete_object().bucket(&bucket).key(allowed_key).send().await;
+    let _ = admin.delete_object().bucket(&bucket).key(denied_key).send().await;
+    let _ = admin.delete_bucket().bucket(&bucket).send().await;
+    admin_remove_user(&env, &parent).await;
+    admin_remove_policy(&env, &policy_readwrite).await;
+
+    info!("test_e2e_sts_session_policy_delete_objects_object_prefix_only passed");
+    Ok(())
+}

crates/e2e_test/src/head_object_consistency_test.rs

@@ -0,0 +1,131 @@
+// Copyright 2026 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
+use aws_sdk_s3::presigning::PresigningConfig;
+use aws_sdk_s3::primitives::ByteStream;
+use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
+use serial_test::serial;
+use std::time::Duration;
+use tracing::info;
+
+const CONSISTENCY_BUCKET: &str = "head-consistency-bucket";
+const PUT_KEY: &str = "consistency-put-object.txt";
+const MPU_KEY: &str = "consistency-multipart-object.txt";
+
+fn list_contains_key(output: &aws_sdk_s3::operation::list_objects_v2::ListObjectsV2Output, key: &str) -> bool {
+    output.contents().iter().any(|obj| obj.key().is_some_and(|k| k == key))
+}
+
+#[tokio::test]
+#[serial]
+async fn head_object_consistency_after_write_and_multipart_and_presigned_head()
+-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    init_logging();
+    info!("Starting HeadObject consistency regression test");
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(Vec::new()).await?;
+
+    let client = env.create_s3_client();
+    env.create_test_bucket(CONSISTENCY_BUCKET).await?;
+
+    client
+        .put_object()
+        .bucket(CONSISTENCY_BUCKET)
+        .key(PUT_KEY)
+        .body(ByteStream::from_static(b"head-consistency-put-body"))
+        .send()
+        .await?;
+
+    client.get_object().bucket(CONSISTENCY_BUCKET).key(PUT_KEY).send().await?;
+    let put_list = client
+        .list_objects_v2()
+        .bucket(CONSISTENCY_BUCKET)
+        .prefix(PUT_KEY)
+        .send()
+        .await?;
+    assert!(list_contains_key(&put_list, PUT_KEY), "ListObjectsV2 should include the PutObject key");
+
+    client.head_object().bucket(CONSISTENCY_BUCKET).key(PUT_KEY).send().await?;
+
+    let create = client
+        .create_multipart_upload()
+        .bucket(CONSISTENCY_BUCKET)
+        .key(MPU_KEY)
+        .send()
+        .await?;
+    let upload_id = create.upload_id().ok_or("missing multipart upload id")?.to_string();
+
+    let part1 = client
+        .upload_part()
+        .bucket(CONSISTENCY_BUCKET)
+        .key(MPU_KEY)
+        .upload_id(&upload_id)
+        .part_number(1)
+        .body(ByteStream::from_static(b"head-consistency-multipart-part-1"))
+        .send()
+        .await?;
+    let completed = CompletedMultipartUpload::builder()
+        .parts(
+            CompletedPart::builder()
+                .part_number(1)
+                .set_e_tag(part1.e_tag().map(str::to_string))
+                .build(),
+        )
+        .build();
+
+    client
+        .complete_multipart_upload()
+        .bucket(CONSISTENCY_BUCKET)
+        .key(MPU_KEY)
+        .upload_id(&upload_id)
+        .multipart_upload(completed)
+        .send()
+        .await?;
+
+    client.get_object().bucket(CONSISTENCY_BUCKET).key(MPU_KEY).send().await?;
+    let mpu_list = client
+        .list_objects_v2()
+        .bucket(CONSISTENCY_BUCKET)
+        .prefix(MPU_KEY)
+        .send()
+        .await?;
+    assert!(
+        list_contains_key(&mpu_list, MPU_KEY),
+        "ListObjectsV2 should include the completed multipart key"
+    );
+
+    client.head_object().bucket(CONSISTENCY_BUCKET).key(MPU_KEY).send().await?;
+
+    let presigned = client
+        .head_object()
+        .bucket(CONSISTENCY_BUCKET)
+        .key(PUT_KEY)
+        .presigned(PresigningConfig::expires_in(Duration::from_secs(300))?)
+        .await?;
+    let presigned_resp = local_http_client().head(presigned.uri().to_string()).send().await?;
+    assert!(
+        presigned_resp.status().is_success(),
+        "Presigned HEAD should succeed, got status {}",
+        presigned_resp.status()
+    );
+
+    client.delete_object().bucket(CONSISTENCY_BUCKET).key(PUT_KEY).send().await?;
+    client.delete_object().bucket(CONSISTENCY_BUCKET).key(MPU_KEY).send().await?;
+    env.delete_test_bucket(CONSISTENCY_BUCKET).await?;
+    env.stop_server();
+
+    Ok(())
+}

crates/e2e_test/src/head_object_range_test.rs

@@ -0,0 +1,52 @@
+use crate::common::{RustFSTestEnvironment, init_logging};
+use aws_sdk_s3::primitives::ByteStream;
+use serial_test::serial;
+use tracing::info;
+
+const RANGE_HEAD_BUCKET: &str = "range-head-test-bucket";
+const RANGE_HEAD_KEY: &str = "range-head-object.bin";
+const ACCEPT_RANGES_BYTES: &str = "bytes";
+
+#[tokio::test]
+#[serial]
+async fn head_object_advertises_accept_ranges() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    init_logging();
+    info!("Starting HeadObject Accept-Ranges regression test");
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(Vec::new()).await?;
+
+    let client = env.create_s3_client();
+    env.create_test_bucket(RANGE_HEAD_BUCKET).await?;
+
+    client
+        .put_object()
+        .bucket(RANGE_HEAD_BUCKET)
+        .key(RANGE_HEAD_KEY)
+        .body(ByteStream::from_static(b"0123456789abcdef"))
+        .send()
+        .await?;
+
+    let head = client
+        .head_object()
+        .bucket(RANGE_HEAD_BUCKET)
+        .key(RANGE_HEAD_KEY)
+        .send()
+        .await?;
+    assert_eq!(
+        head.accept_ranges(),
+        Some(ACCEPT_RANGES_BYTES),
+        "HeadObject should advertise byte range support"
+    );
+
+    client
+        .delete_object()
+        .bucket(RANGE_HEAD_BUCKET)
+        .key(RANGE_HEAD_KEY)
+        .send()
+        .await?;
+    env.delete_test_bucket(RANGE_HEAD_BUCKET).await?;
+    env.stop_server();
+
+    Ok(())
+}

crates/e2e_test/src/lib.rs

@@ -74,6 +74,10 @@ mod compression_test;
 #[cfg(test)]
 mod delete_objects_versioning_test;
 
+// Regression test for signed DELETE Object?versionId requests without Content-Length.
+#[cfg(test)]
+mod delete_object_no_content_length_test;
+
 // Regression test for Issue #2252: ListObjectVersions misses newest version after put -> delete -> put
 #[cfg(test)]
 mod list_object_versions_regression_test;
@@ -104,6 +108,15 @@ mod checksum_upload_test;
 #[cfg(test)]
 mod group_delete_test;
 
+#[cfg(test)]
+mod head_object_range_test;
+
+#[cfg(test)]
+mod head_object_consistency_test;
+
+#[cfg(test)]
+mod copy_object_metadata_test;
+
 // S3 dummy-compat bucket API tests
 #[cfg(test)]
 mod bucket_logging_test;
@@ -125,3 +138,5 @@ mod replication_extension_test;
 
 #[cfg(test)]
 mod snowball_auto_extract_test;
+
+pub mod tls_gen;

crates/e2e_test/src/list_objects_v2_pagination_test.rs

@@ -31,6 +31,7 @@ mod tests {
     use aws_sdk_s3::Client;
     use aws_sdk_s3::primitives::ByteStream;
     use serial_test::serial;
+    use std::collections::HashSet;
     use tracing::info;
 
     /// Helper function to create an S3 client for testing
@@ -57,6 +58,130 @@ mod tests {
         }
     }
 
+    /// Test for Issue #2775: continuation forwarding must not
+    /// skip a child directory when the prefix component repeats in the key.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_repeated_prefix_continuation() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 repeated-prefix continuation");
+
+        const PAGE_SIZE: i32 = 2;
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
+            .await
+            .expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-repeated-prefix-small";
+        let prefix = "engineering/";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        let expected_keys = vec![
+            format!("{prefix}alpha-000/artifact.txt"),
+            format!("{prefix}engineering/engineering/project-000/artifact.txt"),
+            format!("{prefix}engineering/engineering/project-001/artifact.txt"),
+            format!("{prefix}engineering/project-000/artifact.txt"),
+            format!("{prefix}engineering/project-001/artifact.txt"),
+            format!("{prefix}engineering/project-002/artifact.txt"),
+            format!("{prefix}zulu-000/artifact.txt"),
+        ];
+        let noise_keys = [
+            "different/prefix/prefix/project-000/artifact.txt",
+            "engineering-other/project-000/artifact.txt",
+            "unrelated/engineering/project-000/artifact.txt",
+        ];
+
+        for key in &expected_keys {
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(key)
+                .body(ByteStream::from_static(b"x"))
+                .send()
+                .await
+                .expect("Failed to put object");
+        }
+        for key in noise_keys {
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(key)
+                .body(ByteStream::from_static(b"x"))
+                .send()
+                .await
+                .expect("Failed to put noise object");
+        }
+
+        let mut listed_keys = Vec::new();
+        let mut continuation_token: Option<String> = None;
+        let mut last_key: Option<String> = None;
+        let mut page_count = 0;
+
+        loop {
+            let mut request = client.list_objects_v2().bucket(bucket).prefix(prefix).max_keys(PAGE_SIZE);
+
+            if let Some(token) = continuation_token.take() {
+                request = request.continuation_token(token);
+            }
+
+            let output = request.send().await.expect("Failed to list objects");
+
+            for obj in output.contents() {
+                if let Some(key) = obj.key() {
+                    if let Some(previous) = &last_key {
+                        assert!(
+                            key > previous.as_str(),
+                            "ListObjectsV2 did not preserve lexicographic order: {key} <= {previous}"
+                        );
+                    }
+
+                    last_key = Some(key.to_string());
+                    listed_keys.push(key.to_string());
+                }
+            }
+
+            page_count += 1;
+
+            if output.is_truncated().unwrap_or(false) {
+                continuation_token = output.next_continuation_token().map(|s| s.to_string());
+                assert!(
+                    continuation_token.is_some(),
+                    "BUG: NextContinuationToken must be present when IsTruncated is true"
+                );
+            } else {
+                break;
+            }
+
+            if page_count > 10 {
+                panic!("Too many pages, possible infinite loop due to pagination bug");
+            }
+        }
+
+        let seen: HashSet<String> = listed_keys.iter().cloned().collect();
+
+        assert_eq!(
+            listed_keys, expected_keys,
+            "Issue #2775 regression: repeated-prefix pagination must return exactly the expected keys in lexicographic order"
+        );
+        assert_eq!(
+            listed_keys.len(),
+            expected_keys.len(),
+            "Issue #2775 regression: expected all {} repeated-prefix objects under {prefix}, got {}",
+            expected_keys.len(),
+            listed_keys.len()
+        );
+        assert_eq!(seen.len(), expected_keys.len(), "Listed keys must be unique");
+
+        for key in &expected_keys {
+            assert!(seen.contains(key), "Missing expected key after repeated-prefix pagination: {key}");
+        }
+
+        env.stop_server();
+    }
+
     /// Test that IsTruncated is false when all objects fit within max_keys
     ///
     /// This is the core bug from issue #1596: the server was returning
@@ -368,6 +493,71 @@ mod tests {
         env.stop_server();
     }
 
+    /// Test ListObjectsV2 caps max_keys above the service limit and still paginates.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_max_keys_above_limit_returns_token() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 with max_keys above limit");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-max-keys-above-limit";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        let object_count = 1002;
+        for i in 0..object_count {
+            let key = format!("object{:04}.txt", i);
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(&key)
+                .body(ByteStream::from_static(b"test content"))
+                .send()
+                .await
+                .expect("Failed to put object");
+        }
+
+        let output = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .max_keys(1001)
+            .send()
+            .await
+            .expect("Failed to list objects");
+
+        assert_eq!(output.contents().len(), 1000);
+        assert_eq!(output.key_count(), Some(1000));
+        assert_eq!(output.max_keys(), Some(1000));
+        assert!(
+            output.is_truncated().unwrap_or(false),
+            "IsTruncated should be true when more objects remain after capped max_keys"
+        );
+
+        let next_token = output
+            .next_continuation_token()
+            .expect("NextContinuationToken should be present when capped response is truncated")
+            .to_string();
+
+        let output = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .max_keys(1001)
+            .continuation_token(next_token)
+            .send()
+            .await
+            .expect("Failed to list objects with continuation token");
+
+        assert_eq!(output.contents().len(), 2);
+        assert!(!output.is_truncated().unwrap_or(false));
+        assert!(output.next_continuation_token().is_none());
+
+        env.stop_server();
+    }
+
     /// Test ListObjectsV2 with max_keys=0
     ///
     /// S3 semantics: when max_keys is 0, the response should include no objects

crates/e2e_test/src/object_lambda_test.rs

@@ -25,7 +25,7 @@ use std::error::Error;
 use time::OffsetDateTime;
 use tokio::io::{AsyncReadExt, AsyncWriteExt};
 use tokio::net::TcpListener;
-use tokio::sync::oneshot;
+use tokio::sync::{mpsc, oneshot};
 use tokio::time::{Duration, timeout};
 
 #[derive(Debug)]
@@ -172,6 +172,28 @@ async fn spawn_object_lambda_webhook_server_with_response(
     Ok((webhook_url, request_rx, handle))
 }
 
+async fn read_request_path(stream: &mut tokio::net::TcpStream) -> Result<String, Box<dyn Error + Send + Sync>> {
+    let mut buffer = Vec::new();
+    let mut chunk = [0_u8; 4096];
+
+    let header_end = loop {
+        let read = stream.read(&mut chunk).await?;
+        if read == 0 {
+            return Err("request ended before headers were fully received".into());
+        }
+        buffer.extend_from_slice(&chunk[..read]);
+        if let Some(pos) = find_header_terminator(&buffer) {
+            break pos;
+        }
+    };
+
+    let header_text = std::str::from_utf8(&buffer[..header_end])?;
+    let request_line = header_text.lines().next().ok_or("missing request line")?;
+    let path = request_line.split_whitespace().nth(1).ok_or("missing request path")?;
+
+    Ok(path.to_string())
+}
+
 async fn presigned_get_request(
     url: &str,
     access_key: &str,
@@ -326,11 +348,15 @@ async fn delete_webhook_target(env: &RustFSTestEnvironment, target_name: &str) -
 }
 
 fn notification_target_is_listed(targets: &serde_json::Value, target_name: &str) -> bool {
+    notification_target_entry(targets, target_name).is_some()
+}
+
+fn notification_target_entry<'a>(targets: &'a serde_json::Value, target_name: &str) -> Option<&'a serde_json::Value> {
     targets["notification_endpoints"]
         .as_array()
         .into_iter()
         .flatten()
-        .any(|entry| {
+        .find(|entry| {
             entry["account_id"].as_str() == Some(target_name)
                 && entry["service"]
                     .as_str()
@@ -338,6 +364,10 @@ fn notification_target_is_listed(targets: &serde_json::Value, target_name: &str)
         })
 }
 
+fn notification_target_status<'a>(targets: &'a serde_json::Value, target_name: &str) -> Option<&'a str> {
+    notification_target_entry(targets, target_name).and_then(|entry| entry["status"].as_str())
+}
+
 async fn wait_for_target_visibility(
     env: &RustFSTestEnvironment,
     target_name: &str,
@@ -387,6 +417,34 @@ async fn restart_rustfs_server(env: &mut RustFSTestEnvironment) -> Result<(), Bo
     env.start_rustfs_server_without_cleanup(vec![]).await
 }
 
+async fn spawn_http_origin_probe_server() -> Result<
+    (
+        String,
+        mpsc::Receiver<String>,
+        tokio::task::JoinHandle<Result<(), Box<dyn Error + Send + Sync>>>,
+    ),
+    Box<dyn Error + Send + Sync>,
+> {
+    let listener = TcpListener::bind("127.0.0.1:0").await?;
+    let address = listener.local_addr()?;
+    let webhook_url = format!("http://{address}/hook");
+    let (path_tx, path_rx) = mpsc::channel(1);
+
+    let handle = tokio::spawn(async move {
+        loop {
+            let (mut stream, _) = listener.accept().await?;
+            let path = timeout(Duration::from_secs(2), read_request_path(&mut stream)).await??;
+            let _ = path_tx.try_send(path.clone());
+            if path == "/" {
+                let response = b"HTTP/1.1 200 OK\r\nContent-Length: 0\r\nConnection: close\r\n\r\n";
+                stream.write_all(response).await?;
+            }
+        }
+    });
+
+    Ok((webhook_url, path_rx, handle))
+}
+
 async fn read_persisted_server_config(env: &RustFSTestEnvironment) -> String {
     let path = format!("{}/.rustfs.sys/config/config.json", env.temp_dir);
     match tokio::fs::read_to_string(&path).await {
@@ -515,6 +573,40 @@ async fn test_notification_target_persists_across_restart_and_delete() -> Result
     Ok(())
 }
 
+#[tokio::test]
+#[serial]
+async fn test_notification_target_with_path_is_online_via_transport_probe() -> Result<(), Box<dyn Error + Send + Sync>> {
+    init_logging();
+
+    let (webhook_url, mut probe_rx, probe_handle) = spawn_http_origin_probe_server().await?;
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_NOTIFY_ENABLE", "true")])
+        .await?;
+
+    let target_name = "path-probe";
+    configure_webhook_target(&env, target_name, &webhook_url, "secret-token").await?;
+
+    let (visible_targets, visible_arns) = wait_for_target_visibility(&env, target_name).await?;
+    assert_eq!(notification_target_status(&visible_targets, target_name), Some("online"));
+    let observed_path = timeout(Duration::from_secs(10), probe_rx.recv())
+        .await
+        .map_err(|_| "probe server timed out waiting for a request")?
+        .ok_or("probe server did not observe a request")?;
+    assert_eq!(observed_path, "/");
+    assert!(
+        visible_arns
+            .iter()
+            .any(|arn| arn.ends_with(&format!(":{target_name}:webhook"))),
+        "target ARN missing for reachable path endpoint: {visible_arns:?}"
+    );
+
+    probe_handle.abort();
+    let _ = probe_handle.await;
+
+    Ok(())
+}
+
 #[tokio::test]
 #[serial]
 async fn test_get_object_lambda_accepts_presigned_requests() -> Result<(), Box<dyn Error + Send + Sync>> {
@@ -1058,6 +1150,41 @@ async fn test_listen_notification_emits_after_put_object() -> Result<(), Box<dyn
     Ok(())
 }
 
+#[tokio::test]
+#[serial]
+async fn test_listen_notification_emits_on_empty_bucket_when_notify_disabled() -> Result<(), Box<dyn Error + Send + Sync>> {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_NOTIFY_ENABLE", "false")])
+        .await?;
+
+    let bucket = "listen-empty-bucket-e2e";
+    let key = "seed/object.txt";
+    let client = env.create_s3_client();
+
+    client.create_bucket().bucket(bucket).send().await?;
+
+    let listen_url = format!("{}/{bucket}?events={}&ping=1", env.url, urlencoding::encode("s3:ObjectCreated:*"),);
+    let response = signed_request(http::Method::GET, &listen_url, &env.access_key, &env.secret_key, None, None).await?;
+    assert_eq!(response.status(), StatusCode::OK);
+
+    let read_task = tokio::spawn(read_listen_notification_event(response, key));
+
+    client
+        .put_object()
+        .bucket(bucket)
+        .key(key)
+        .body(ByteStream::from_static(b"empty bucket watch body"))
+        .send()
+        .await?;
+
+    let payload = timeout(Duration::from_secs(12), read_task).await???;
+    assert!(!payload.is_empty(), "listen_notification payload should not be empty");
+
+    Ok(())
+}
+
 #[tokio::test]
 #[serial]
 async fn test_listen_notification_fans_in_remote_node_events() -> Result<(), Box<dyn Error + Send + Sync>> {

crates/e2e_test/src/object_lock/object_lock_test.rs

@@ -26,10 +26,12 @@
 
 use super::common::*;
 use aws_sdk_s3::Client;
-use aws_sdk_s3::primitives::ByteStream;
+use aws_sdk_s3::primitives::{ByteStream, DateTimeFormat};
 use aws_sdk_s3::types::{
-    CompletedMultipartUpload, CompletedPart, Delete, ObjectIdentifier, ObjectLockLegalHoldStatus, ObjectLockRetentionMode,
+    CompletedMultipartUpload, CompletedPart, Delete, MetadataDirective, ObjectIdentifier, ObjectLockLegalHoldStatus,
+    ObjectLockMode, ObjectLockRetentionMode,
 };
+use chrono::{DateTime, Duration, Utc};
 use serial_test::serial;
 use tracing::info;
 
@@ -79,6 +81,26 @@ fn assert_access_denied<T, E: std::fmt::Debug>(result: Result<T, E>, context: &s
     );
 }
 
+fn assert_invalid_object_lock_retention_pair<T, E: std::fmt::Debug>(result: Result<T, E>, context: &str) {
+    let err = match result {
+        Ok(_) => panic!("{context}"),
+        Err(err) => format!("{err:?}"),
+    };
+    assert!(
+        err.contains("InvalidRequest") || err.contains("must both be supplied"),
+        "{context}: expected invalid paired retention headers, got: {err}"
+    );
+}
+
+fn parse_s3_datetime(value: &aws_sdk_s3::primitives::DateTime) -> DateTime<Utc> {
+    let formatted = value
+        .fmt(DateTimeFormat::DateTime)
+        .expect("S3 timestamp should format as RFC3339");
+    DateTime::parse_from_rfc3339(&formatted)
+        .expect("S3 timestamp should parse as RFC3339")
+        .with_timezone(&Utc)
+}
+
 // ============================================================================
 // DeleteObject Tests
 // ============================================================================
@@ -537,6 +559,100 @@ async fn test_copy_object_applies_requested_legal_hold() {
     );
 }
 
+#[tokio::test]
+#[serial]
+async fn test_copy_object_does_not_inherit_source_legal_hold() {
+    init_logging();
+    info!("🧪 Test: CopyObject does not inherit source Legal Hold");
+
+    let mut env = ObjectLockTestEnvironment::new().await.unwrap();
+    env.start_rustfs().await.unwrap();
+
+    let bucket = "test-copy-object-legal-hold-inherit";
+    let src_key = "held-source";
+
+    env.create_object_lock_bucket(bucket).await.unwrap();
+
+    let client = env.s3_client();
+    put_object_with_legal_hold(&client, bucket, src_key, b"copy-source", ObjectLockLegalHoldStatus::On)
+        .await
+        .unwrap();
+
+    client
+        .copy_object()
+        .copy_source(format!("{bucket}/{src_key}"))
+        .bucket(bucket)
+        .key("implicit-copy")
+        .send()
+        .await
+        .unwrap();
+
+    let implicit_legal_hold = client
+        .get_object_legal_hold()
+        .bucket(bucket)
+        .key("implicit-copy")
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(
+        implicit_legal_hold
+            .legal_hold()
+            .and_then(|value| value.status())
+            .map(|value| value.as_str()),
+        Some("OFF")
+    );
+
+    client
+        .copy_object()
+        .copy_source(format!("{bucket}/{src_key}"))
+        .bucket(bucket)
+        .key("explicit-on-copy")
+        .object_lock_legal_hold_status(ObjectLockLegalHoldStatus::On)
+        .send()
+        .await
+        .unwrap();
+
+    let explicit_on_legal_hold = client
+        .get_object_legal_hold()
+        .bucket(bucket)
+        .key("explicit-on-copy")
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(
+        explicit_on_legal_hold
+            .legal_hold()
+            .and_then(|value| value.status())
+            .map(|value| value.as_str()),
+        Some("ON")
+    );
+
+    client
+        .copy_object()
+        .copy_source(format!("{bucket}/{src_key}"))
+        .bucket(bucket)
+        .key("explicit-off-copy")
+        .object_lock_legal_hold_status(ObjectLockLegalHoldStatus::Off)
+        .send()
+        .await
+        .unwrap();
+
+    let explicit_off_legal_hold = client
+        .get_object_legal_hold()
+        .bucket(bucket)
+        .key("explicit-off-copy")
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(
+        explicit_off_legal_hold
+            .legal_hold()
+            .and_then(|value| value.status())
+            .map(|value| value.as_str()),
+        Some("OFF")
+    );
+}
+
 #[tokio::test]
 #[serial]
 async fn test_copy_object_overwrite_blocked_by_legal_hold() {
@@ -1388,9 +1504,457 @@ async fn test_default_retention_applied_to_new_objects() {
     let delete_result = delete_object_with_bypass(&client, bucket, key, Some(version_id), false).await;
     assert!(delete_result.is_err(), "Delete should fail for object with default retention applied");
 
+    let retention = client
+        .get_object_retention()
+        .bucket(bucket)
+        .key(key)
+        .version_id(version_id)
+        .send()
+        .await
+        .unwrap();
+    let retention = retention.retention().expect("default retention should be readable");
+    assert_eq!(retention.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    assert!(
+        retention.retain_until_date().is_some(),
+        "default retention should write a retain-until date"
+    );
+
+    let head = client
+        .head_object()
+        .bucket(bucket)
+        .key(key)
+        .version_id(version_id)
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(head.object_lock_mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    assert!(
+        head.object_lock_retain_until_date().is_some(),
+        "HeadObject should expose the default retention retain-until date"
+    );
+
     info!("✅ Test passed: Default retention is applied to new objects");
 }
 
+#[tokio::test]
+#[serial]
+async fn test_delete_object_creates_delete_marker_for_default_retained_current_version() {
+    init_logging();
+    info!("🧪 Test: DeleteObject creates delete marker for default-retained current version");
+
+    let mut env = ObjectLockTestEnvironment::new().await.unwrap();
+    env.start_rustfs().await.unwrap();
+
+    let bucket = "test-default-retention-delete-marker";
+    let key = "default-retained-object";
+    let data = b"test data for default-retained current version";
+
+    env.create_object_lock_bucket(bucket).await.unwrap();
+
+    let client = env.s3_client();
+    put_object_lock_configuration(&client, bucket, ObjectLockRetentionMode::Governance, Some(30), None)
+        .await
+        .unwrap();
+
+    let put_output = client
+        .put_object()
+        .bucket(bucket)
+        .key(key)
+        .body(ByteStream::from(data.to_vec()))
+        .send()
+        .await
+        .unwrap();
+    let retained_version_id = put_output
+        .version_id()
+        .expect("default-retained object should have a version id")
+        .to_string();
+
+    let retention = client
+        .get_object_retention()
+        .bucket(bucket)
+        .key(key)
+        .version_id(&retained_version_id)
+        .send()
+        .await
+        .unwrap();
+    let retention = retention.retention().expect("default retention should be readable");
+    assert_eq!(retention.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    assert!(
+        retention.retain_until_date().is_some(),
+        "default retention should write a retain-until date"
+    );
+
+    let delete_marker_output = client.delete_object().bucket(bucket).key(key).send().await.unwrap();
+    assert_eq!(delete_marker_output.delete_marker(), Some(true));
+    let delete_marker_version_id = delete_marker_output
+        .version_id()
+        .expect("delete marker should have a version id")
+        .to_string();
+
+    let protected_delete = delete_object_with_bypass(&client, bucket, key, Some(&retained_version_id), false).await;
+    assert!(protected_delete.is_err(), "Default-retained version should still reject direct deletion");
+
+    let retention_after_delete_marker = client
+        .get_object_retention()
+        .bucket(bucket)
+        .key(key)
+        .version_id(&retained_version_id)
+        .send()
+        .await
+        .unwrap();
+    let retention_after_delete_marker = retention_after_delete_marker
+        .retention()
+        .expect("default retention should remain readable by version id after delete marker creation");
+    assert_eq!(retention_after_delete_marker.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    assert!(
+        retention_after_delete_marker.retain_until_date().is_some(),
+        "retained version should keep its retain-until date after delete marker creation"
+    );
+
+    delete_object_with_bypass(&client, bucket, key, Some(&delete_marker_version_id), false)
+        .await
+        .unwrap();
+    delete_object_with_bypass(&client, bucket, key, Some(&retained_version_id), true)
+        .await
+        .unwrap();
+
+    info!("✅ Test passed: Delete marker is allowed while default-retained version stays protected");
+}
+
+#[tokio::test]
+#[serial]
+async fn test_put_copy_and_multipart_reject_incomplete_retention_headers() {
+    init_logging();
+    info!("🧪 Test: write paths reject incomplete Object Lock retention headers");
+
+    let mut env = ObjectLockTestEnvironment::new().await.unwrap();
+    env.start_rustfs().await.unwrap();
+
+    let bucket = "test-incomplete-retention";
+    let src_key = "copy-source";
+
+    env.create_object_lock_bucket(bucket).await.unwrap();
+
+    let client = env.s3_client();
+    client
+        .put_object()
+        .bucket(bucket)
+        .key(src_key)
+        .body(ByteStream::from(b"copy-source".to_vec()))
+        .send()
+        .await
+        .unwrap();
+
+    put_object_lock_configuration(&client, bucket, ObjectLockRetentionMode::Governance, Some(30), None)
+        .await
+        .unwrap();
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .put_object()
+            .bucket(bucket)
+            .key("put-mode-only")
+            .body(ByteStream::from(b"put-body".to_vec()))
+            .object_lock_mode(ObjectLockMode::Governance)
+            .send()
+            .await,
+        "PutObject with mode only should fail",
+    );
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .put_object()
+            .bucket(bucket)
+            .key("put-date-only")
+            .body(ByteStream::from(b"put-body".to_vec()))
+            .object_lock_retain_until_date(retention_timestamp(30))
+            .send()
+            .await,
+        "PutObject with retain-until-date only should fail",
+    );
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .copy_object()
+            .copy_source(format!("{bucket}/{src_key}"))
+            .bucket(bucket)
+            .key("copy-mode-only")
+            .object_lock_mode(ObjectLockMode::Governance)
+            .send()
+            .await,
+        "CopyObject with mode only should fail",
+    );
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .copy_object()
+            .copy_source(format!("{bucket}/{src_key}"))
+            .bucket(bucket)
+            .key("copy-date-only")
+            .object_lock_retain_until_date(retention_timestamp(30))
+            .send()
+            .await,
+        "CopyObject with retain-until-date only should fail",
+    );
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .create_multipart_upload()
+            .bucket(bucket)
+            .key("multipart-mode-only")
+            .object_lock_mode(ObjectLockMode::Governance)
+            .send()
+            .await,
+        "CreateMultipartUpload with mode only should fail",
+    );
+
+    assert_invalid_object_lock_retention_pair(
+        client
+            .create_multipart_upload()
+            .bucket(bucket)
+            .key("multipart-date-only")
+            .object_lock_retain_until_date(retention_timestamp(30))
+            .send()
+            .await,
+        "CreateMultipartUpload with retain-until-date only should fail",
+    );
+}
+
+#[tokio::test]
+#[serial]
+async fn test_copy_object_retention_uses_destination_policy() {
+    init_logging();
+    info!("🧪 Test: CopyObject retention follows destination policy");
+
+    let mut env = ObjectLockTestEnvironment::new().await.unwrap();
+    env.start_rustfs().await.unwrap();
+
+    let src_bucket = "test-copy-retention-src";
+    let dst_bucket = "test-copy-retention-dst";
+    let no_default_bucket = "test-copy-retention-nodef";
+    let src_key = "retained-source";
+
+    env.create_object_lock_bucket(src_bucket).await.unwrap();
+    env.create_object_lock_bucket(dst_bucket).await.unwrap();
+    env.create_object_lock_bucket(no_default_bucket).await.unwrap();
+
+    let client = env.s3_client();
+    put_object_lock_configuration(&client, dst_bucket, ObjectLockRetentionMode::Governance, Some(1), None)
+        .await
+        .unwrap();
+
+    put_object_with_retention(
+        &client,
+        src_bucket,
+        src_key,
+        b"copy-source",
+        ObjectLockRetentionMode::Compliance,
+        future_retain_until(30),
+    )
+    .await
+    .unwrap();
+
+    let copy_started = Utc::now();
+    client
+        .copy_object()
+        .copy_source(format!("{src_bucket}/{src_key}"))
+        .bucket(dst_bucket)
+        .key("default-copy")
+        .send()
+        .await
+        .unwrap();
+
+    let retention = client
+        .get_object_retention()
+        .bucket(dst_bucket)
+        .key("default-copy")
+        .send()
+        .await
+        .unwrap();
+    let retention = retention
+        .retention()
+        .expect("destination default retention should be present");
+    assert_eq!(retention.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    let retain_until = parse_s3_datetime(retention.retain_until_date().expect("retain-until date should be present"));
+    assert!(
+        retain_until < copy_started + Duration::days(3),
+        "destination default retention should not inherit the source's longer retention"
+    );
+
+    client
+        .copy_object()
+        .copy_source(format!("{src_bucket}/{src_key}"))
+        .bucket(dst_bucket)
+        .key("replace-copy")
+        .metadata_directive(MetadataDirective::Replace)
+        .send()
+        .await
+        .unwrap();
+
+    let replace_retention = client
+        .get_object_retention()
+        .bucket(dst_bucket)
+        .key("replace-copy")
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(
+        replace_retention
+            .retention()
+            .and_then(|value| value.mode())
+            .map(|value| value.as_str()),
+        Some("GOVERNANCE")
+    );
+
+    client
+        .copy_object()
+        .copy_source(format!("{src_bucket}/{src_key}"))
+        .bucket(dst_bucket)
+        .key("explicit-copy")
+        .object_lock_mode(ObjectLockMode::Compliance)
+        .object_lock_retain_until_date(retention_timestamp(30))
+        .send()
+        .await
+        .unwrap();
+
+    let explicit_retention = client
+        .get_object_retention()
+        .bucket(dst_bucket)
+        .key("explicit-copy")
+        .send()
+        .await
+        .unwrap();
+    assert_eq!(
+        explicit_retention
+            .retention()
+            .and_then(|value| value.mode())
+            .map(|value| value.as_str()),
+        Some("COMPLIANCE")
+    );
+    let explicit_retain_until = parse_s3_datetime(
+        explicit_retention
+            .retention()
+            .and_then(|value| value.retain_until_date())
+            .expect("explicit retain-until date should be present"),
+    );
+    assert!(
+        explicit_retain_until > Utc::now() + Duration::days(20),
+        "explicit retention should override the shorter bucket default"
+    );
+
+    client
+        .copy_object()
+        .copy_source(format!("{src_bucket}/{src_key}"))
+        .bucket(no_default_bucket)
+        .key("no-default-copy")
+        .send()
+        .await
+        .unwrap();
+
+    let no_default_retention = client
+        .get_object_retention()
+        .bucket(no_default_bucket)
+        .key("no-default-copy")
+        .send()
+        .await
+        .unwrap();
+    let no_default_retention = no_default_retention
+        .retention()
+        .expect("retention response should be present");
+    assert!(no_default_retention.mode().is_none());
+    assert!(no_default_retention.retain_until_date().is_none());
+
+    put_object_with_retention(
+        &client,
+        dst_bucket,
+        "locked-destination",
+        b"locked-target",
+        ObjectLockRetentionMode::Compliance,
+        future_retain_until(30),
+    )
+    .await
+    .unwrap();
+
+    let overwrite_result = client
+        .copy_object()
+        .copy_source(format!("{src_bucket}/{src_key}"))
+        .bucket(dst_bucket)
+        .key("locked-destination")
+        .send()
+        .await;
+    assert!(
+        overwrite_result.is_err(),
+        "CopyObject overwrite should not bypass active destination retention"
+    );
+}
+
+#[tokio::test]
+#[serial]
+async fn test_multipart_default_retention_fixed_at_create() {
+    init_logging();
+    info!("🧪 Test: multipart default retention is fixed at CreateMultipartUpload");
+
+    let mut env = ObjectLockTestEnvironment::new().await.unwrap();
+    env.start_rustfs().await.unwrap();
+
+    let bucket = "test-multipart-default-drift";
+    let key = "multipart-object";
+
+    env.create_object_lock_bucket(bucket).await.unwrap();
+
+    let client = env.s3_client();
+    put_object_lock_configuration(&client, bucket, ObjectLockRetentionMode::Governance, Some(1), None)
+        .await
+        .unwrap();
+
+    let create_started = Utc::now();
+    let create_output = client.create_multipart_upload().bucket(bucket).key(key).send().await.unwrap();
+    let upload_id = create_output.upload_id().unwrap();
+
+    let upload_part_output = client
+        .upload_part()
+        .bucket(bucket)
+        .key(key)
+        .upload_id(upload_id)
+        .part_number(1)
+        .body(ByteStream::from(b"multipart-body".to_vec()))
+        .send()
+        .await
+        .unwrap();
+
+    put_object_lock_configuration(&client, bucket, ObjectLockRetentionMode::Governance, Some(10), None)
+        .await
+        .unwrap();
+
+    let completed_upload = CompletedMultipartUpload::builder()
+        .parts(
+            CompletedPart::builder()
+                .part_number(1)
+                .e_tag(upload_part_output.e_tag().unwrap_or_default())
+                .build(),
+        )
+        .build();
+
+    client
+        .complete_multipart_upload()
+        .bucket(bucket)
+        .key(key)
+        .upload_id(upload_id)
+        .multipart_upload(completed_upload)
+        .send()
+        .await
+        .unwrap();
+
+    let retention = client.get_object_retention().bucket(bucket).key(key).send().await.unwrap();
+    let retention = retention.retention().expect("multipart default retention should be present");
+    assert_eq!(retention.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
+    let retain_until = parse_s3_datetime(retention.retain_until_date().expect("retain-until date should be present"));
+    assert!(
+        retain_until < create_started + Duration::days(3),
+        "CompleteMultipartUpload should keep the default retention calculated at create time"
+    );
+}
+
 // ============================================================================
 // Versioning Auto-Enable Tests
 // ============================================================================

crates/e2e_test/src/protocols/README.md

@@ -1,38 +1,26 @@
 # Protocol E2E Tests
 
-FTPS and WebDAV protocol end-to-end tests for RustFS.
+FTPS, WebDAV, and SFTP protocol end-to-end tests for RustFS.
 
 ## Prerequisites
 
-### Required Tools
-
-```bash
-# Ubuntu/Debian
-sudo apt-get install sshpass ssh-keygen
-
-# RHEL/CentOS
-sudo yum install sshpass openssh-clients
-
-# macOS
-brew install sshpass openssh
-```
+No external SSH tooling is required. The test framework generates ed25519
+host keys in-process via russh::keys under the per-test temp directory
+before each SFTP server spawn, and russh-sftp drives the protocol from the
+test process directly.
 
 ## Running Tests
 
-Run all protocol tests (FTPS + WebDAV):
 ```bash
-RUSTFS_BUILD_FEATURES=ftps,webdav cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
+RUSTFS_BUILD_FEATURES=ftps,webdav,sftp cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
 ```
 
-Run FTPS tests only:
-```bash
-RUSTFS_BUILD_FEATURES=ftps cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
-```
-
-Run WebDAV tests only:
-```bash
-RUSTFS_BUILD_FEATURES=webdav cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
-```
+`RUSTFS_BUILD_FEATURES` controls which features the test rustfs binary is
+built with. When this variable is set, the protocol test runner schedules
+only entries whose protocol is present in the requested feature list. Leave
+it unset to run every protocol entry.
+`--test-threads=1` is required because every entry spawns a rustfs server
+on fixed bind ports.
 
 ## Test Coverage
 
@@ -55,6 +43,94 @@ RUSTFS_BUILD_FEATURES=webdav cargo test --package e2e_test test_protocol_core_su
 - GET (download file)
 - PROPFIND on bucket (list objects)
 - DELETE file
+- MOVE file (rename object)
 - DELETE bucket
 - Authentication failure test
 
+### SFTP Tests
+
+The SFTP suite lives in three entries plus a standalone idle-timeout case.
+Every assertion runs against a freshly spawned rustfs binary with
+`RUSTFS_SFTP_ENABLE=true`; the test framework also pins
+`RUSTFS_SFTP_PART_SIZE=5242880` so the multipart boundary is deterministic.
+
+#### sftp_core (`test_sftp_core_operations`)
+
+Bind ports 9022 (SFTP) and 9200 (S3). 22 in-suite assertions covering the
+core protocol surface plus cross-protocol consistency:
+
+- Subsystem canary: SFTPv3 version exchange completes after password auth
+- Bucket lifecycle: mkdir, root listing, rmdir, post-delete listing
+- Small-file round-trip with SHA256 compare
+- Stat on a file (size + file type) and on a bucket (directory)
+- SETSTAT on a path returns ok
+- Rename within bucket, listing reflects the rename
+- Multipart-sized round-trip (just over 2 × part_size) with SHA256 compare
+- Negative cases: symlink rejected, open of nonexistent file rejected,
+  read_dir of nonexistent bucket rejected, path traversal rejected
+- Spec-letter assertions: APPEND open returns an error, CREATE+EXCLUDE on an
+  existing path returns an error, bad-password authentication is rejected
+- Cross-protocol via aws-sdk-s3: SFTP write then S3 read with SHA256 match,
+  S3 write then SFTP read with SHA256 match
+- Cross-API directory visibility: SFTP-created sub-directory visible via S3
+  ListObjectsV2, S3-created `__XLDIR__` marker visible via SFTP readdir as a
+  directory entry
+
+#### sftp_compliance (`test_sftp_compliance_suite`)
+
+Bind ports 9024 (SFTP) and 9300 (S3). 14 compliance regression cases against
+one shared server spawn. Each case carries a stable CMPTST-NN identifier:
+
+- CMPTST-01: medium-binary upload then download with SHA256 compare
+  (single-shot PutObject path below the multipart boundary)
+- CMPTST-02: zero-byte upload, download, and stat-size match
+- CMPTST-03: rm against a bucket path is rejected; the bucket is preserved
+- CMPTST-04: rmdir against a non-empty bucket is rejected; the contained
+  object survives
+- CMPTST-05: rmdir against a non-empty sub-directory is rejected; the inner
+  object survives
+- CMPTST-06: open with a path-traversal pattern cannot leak a host file via
+  SFTP read
+- CMPTST-07: read_dir of `/..` either errors or returns a listing that
+  contains no host system entries
+- CMPTST-08: rename across buckets preserves payload and removes the source
+  object
+- CMPTST-09: paths with embedded spaces round-trip through the russh-sftp
+  client
+- CMPTST-10: read_link is rejected (S3 storage has no symlinks)
+- CMPTST-11: SETSTAT on a path and FSETSTAT on a separate open handle both
+  return ok (rsync, WinSCP transfer-success contract)
+- CMPTST-12: rename to the same path is a no-op; the file persists with the
+  original payload
+- CMPTST-13: implicit-directory round-trip; uploading to a nested key
+  creates the parent directory implicitly and three listing forms surface
+  the inner file
+- CMPTST-14: OPEN, WRITE, FSETSTAT, CLOSE on the same write handle all
+  return ok (WinSCP wire shape)
+
+#### sftp_compliance_readonly (`test_sftp_compliance_readonly`)
+
+Bind ports 9025 (SFTP) and 9301 (S3). Spawns a second rustfs binary with
+`RUSTFS_SFTP_READ_ONLY=true`; the S3 endpoint stays writable so the suite
+can seed a bucket and a fixture object via aws-sdk-s3 before opening the
+SFTP session. 7 compliance cases:
+
+- CMPTST-15: put through SFTP is rejected
+- CMPTST-16: rm through SFTP is rejected
+- CMPTST-17: mkdir through SFTP is rejected
+- CMPTST-18: rmdir through SFTP is rejected
+- CMPTST-19: rename through SFTP is rejected
+- CMPTST-20: ls through SFTP is allowed and lists the seeded bucket
+- CMPTST-21: get through SFTP is allowed and returns the seeded payload
+  byte-for-byte
+
+The full case index lives at the top of `sftp_compliance.rs`; each helper's
+log lines name its CMPTST-NN code so a failure in CI points at one named
+property without consulting any external doc.
+
+#### sftp_idle_timeout (`test_sftp_idle_timeout_disconnects`)
+
+Bind ports 9023 (SFTP) and 9100 (S3). Spawns rustfs with
+`RUSTFS_SFTP_IDLE_TIMEOUT=5`, sleeps 10 s past the timeout, then issues an
+SFTP request and asserts the server has closed the session.
+

crates/e2e_test/src/protocols/mod.rs

@@ -12,9 +12,13 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-//! Protocol tests for FTPS and WebDAV
+//! Protocol tests for FTPS, WebDAV, and SFTP
 
 pub mod ftps_core;
+pub mod sftp_compliance;
+mod sftp_compliance_tests;
+pub mod sftp_core;
+pub mod sftp_helpers;
 pub mod test_env;
 pub mod test_runner;
 pub mod webdav_core;

crates/e2e_test/src/protocols/sftp_compliance.rs

@@ -0,0 +1,217 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Public test entry points for the SFTP compliance suite.
+//!
+//! Three suite entries cover every CMPTST-NN identifier:
+//!
+//! - test_sftp_compliance_suite           CMPTST-01..14 (one shared SFTP session)
+//! - test_sftp_compliance_readonly        CMPTST-15..23 (one shared session, read-only mode)
+//! - test_sftp_compliance_standalone      CMPTST-24..33 (each case spawns its own rustfs)
+//!
+//! The first two reuse a single rustfs spawn for the whole bracket
+//! because every case in the bracket exercises the same protocol
+//! against the same server. The third aggregates cases that each need
+//! a different server configuration (idle timeout, read-cache window,
+//! console disabled) and therefore cannot share a binary.
+//!
+//! Each per-case module exposes a single descriptive entry called
+//! run_<what_it_tests>(). For example cmptst_24 exposes
+//! cmptst_24::run_concurrent_half_close_no_leak().
+//!
+//! The per-case bodies (one cmptst_NN module per case) and the
+//! cross-case infrastructure (spawn helpers, fixture seeders, the
+//! half-close / wedge / paused-drain stream wrappers, and the session
+//! lifecycle counters) live in sftp_compliance_tests.rs. Per-case
+//! marker comments and the full case-index doc live there too.
+
+use crate::protocols::sftp_compliance_tests::{
+    cmptst_01, cmptst_02, cmptst_03, cmptst_04, cmptst_05, cmptst_06, cmptst_07, cmptst_08, cmptst_09, cmptst_10, cmptst_11,
+    cmptst_12, cmptst_13, cmptst_14, cmptst_15, cmptst_16, cmptst_17, cmptst_18, cmptst_19, cmptst_20, cmptst_21, cmptst_22,
+    cmptst_23, cmptst_27, cmptst_28, cmptst_29, cmptst_32, cmptst_33, spawn_compliance_rustfs,
+};
+#[cfg(target_os = "linux")]
+use crate::protocols::sftp_compliance_tests::{cmptst_24, cmptst_25, cmptst_26};
+use crate::protocols::sftp_helpers::{build_test_s3_client, connect_sftp_to, wait_for_s3_ready};
+use crate::protocols::test_env::ProtocolTestEnvironment;
+use anyhow::{Result, anyhow};
+use aws_sdk_s3::primitives::ByteStream;
+use tracing::info;
+
+// Read-write compliance suite ports. Distinct from sftp_core (9022/9200)
+// and from test_sftp_idle_timeout_disconnects (9023/9100) so the SFTP
+// entries can run sequentially without leftover-listener contention.
+const COMPLIANCE_RW_SFTP_PORT: u16 = 9024;
+const COMPLIANCE_RW_SFTP_ADDRESS: &str = "127.0.0.1:9024";
+const COMPLIANCE_RW_S3_ADDRESS: &str = "127.0.0.1:9300";
+
+// Read-only compliance suite ports. The SFTP session opened against
+// this address runs against a server started with
+// RUSTFS_SFTP_READ_ONLY=true. The S3 endpoint stays writable so the
+// suite can seed a bucket and a fixture object before running the SFTP
+// rejection assertions.
+const COMPLIANCE_RO_SFTP_PORT: u16 = 9025;
+const COMPLIANCE_RO_SFTP_ADDRESS: &str = "127.0.0.1:9025";
+const COMPLIANCE_RO_S3_ADDRESS: &str = "127.0.0.1:9301";
+const COMPLIANCE_RO_S3_ENDPOINT: &str = "http://127.0.0.1:9301";
+const COMPLIANCE_RO_S3_READY_ATTEMPTS: u32 = 30;
+
+/// Compliance suite entry: spawn one rustfs server, run every per-case
+/// helper that closes a coverage gap not exercised by sftp_core. Runs
+/// CMPTST-01 through CMPTST-14 against the same SFTP session.
+pub async fn test_sftp_compliance_suite() -> Result<()> {
+    info!("Starting SFTP server for compliance suite on {}", COMPLIANCE_RW_SFTP_ADDRESS);
+    let (_env, mut server_process) = spawn_compliance_rustfs(COMPLIANCE_RW_SFTP_ADDRESS, COMPLIANCE_RW_S3_ADDRESS, false).await?;
+
+    let result = async {
+        ProtocolTestEnvironment::wait_for_port_ready(COMPLIANCE_RW_SFTP_PORT, 30)
+            .await
+            .map_err(|e| anyhow!("{}", e))?;
+
+        let (session, sftp) = connect_sftp_to(COMPLIANCE_RW_SFTP_ADDRESS).await?;
+
+        cmptst_01::run_medium_binary_round_trip(&sftp).await?;
+        cmptst_02::run_zero_byte_round_trip(&sftp).await?;
+        cmptst_03::run_rm_on_bucket_path_rejected(&sftp).await?;
+        cmptst_04::run_rmdir_nonempty_bucket_rejected(&sftp).await?;
+        cmptst_05::run_rmdir_nonempty_subdir_rejected(&sftp).await?;
+        cmptst_06::run_path_traversal_get_rejected(&sftp).await?;
+        cmptst_07::run_dotdot_collapses_to_root(&sftp).await?;
+        cmptst_08::run_rename_cross_bucket(&sftp).await?;
+        cmptst_09::run_path_with_spaces_round_trip(&sftp).await?;
+        cmptst_10::run_readlink_rejected(&sftp).await?;
+        cmptst_11::run_setstat_after_put_returns_ok(&sftp).await?;
+        cmptst_12::run_rename_same_path_keeps_file(&sftp).await?;
+        cmptst_13::run_implicit_dir_round_trip(&sftp).await?;
+        cmptst_14::run_winscp_setstat_shape_on_handle(&sftp).await?;
+
+        drop(sftp);
+        session.disconnect(russh::Disconnect::ByApplication, "", "en").await?;
+        info!("SFTP compliance suite passed");
+        Ok::<(), anyhow::Error>(())
+    }
+    .await;
+
+    // Discard kill/wait errors on the teardown path: the test result
+    // above is the binding outcome, and a server that has already
+    // exited produces an error here that carries no useful signal.
+    server_process.kill_and_wait().await;
+
+    result
+}
+
+/// Read-only compliance entry: CMPTST-15 through CMPTST-23. The SFTP
+/// server runs with RUSTFS_SFTP_READ_ONLY=true. Mutations through SFTP
+/// must error. Reads through SFTP must succeed. The test seeds a bucket
+/// and a file through the writable S3 endpoint before opening the SFTP
+/// session.
+pub async fn test_sftp_compliance_readonly() -> Result<()> {
+    info!("Starting SFTP server in read-only mode on {}", COMPLIANCE_RO_SFTP_ADDRESS);
+    let (_env, mut server_process) = spawn_compliance_rustfs(COMPLIANCE_RO_SFTP_ADDRESS, COMPLIANCE_RO_S3_ADDRESS, true).await?;
+
+    let result = async {
+        ProtocolTestEnvironment::wait_for_port_ready(COMPLIANCE_RO_SFTP_PORT, 30)
+            .await
+            .map_err(|e| anyhow!("{}", e))?;
+
+        let s3 = build_test_s3_client(COMPLIANCE_RO_S3_ENDPOINT);
+        wait_for_s3_ready(&s3, COMPLIANCE_RO_S3_READY_ATTEMPTS).await?;
+
+        let bucket = "robucket";
+        let seeded_key = "small.txt";
+        let seeded_content = b"read-only seed\n";
+
+        s3.create_bucket()
+            .bucket(bucket)
+            .send()
+            .await
+            .map_err(|e| anyhow!("S3 CreateBucket {} failed: {:?}", bucket, e))?;
+        s3.put_object()
+            .bucket(bucket)
+            .key(seeded_key)
+            .body(ByteStream::from_static(seeded_content))
+            .send()
+            .await
+            .map_err(|e| anyhow!("S3 PutObject {}/{} failed: {:?}", bucket, seeded_key, e))?;
+        info!("Seeded read-only fixture via S3: {}/{}", bucket, seeded_key);
+
+        let (session, sftp) = connect_sftp_to(COMPLIANCE_RO_SFTP_ADDRESS).await?;
+
+        cmptst_15::run_ro_put_rejected(&sftp, bucket).await?;
+        cmptst_16::run_ro_rm_rejected(&sftp, bucket, seeded_key).await?;
+        cmptst_17::run_ro_mkdir_rejected(&sftp).await?;
+        cmptst_18::run_ro_rmdir_rejected(&sftp, bucket).await?;
+        cmptst_19::run_ro_rename_rejected(&sftp, bucket, seeded_key).await?;
+        cmptst_20::run_ro_ls_allowed(&sftp, bucket).await?;
+        cmptst_21::run_ro_get_allowed(&sftp, bucket, seeded_key, seeded_content).await?;
+        cmptst_22::run_ro_setstat_rejected(&sftp, bucket, seeded_key).await?;
+        cmptst_23::run_ro_fsetstat_rejected(&sftp, bucket, seeded_key).await?;
+
+        drop(sftp);
+        // Discard the disconnect Result. A read-only session that has
+        // returned errors against every mutation can still be cleanly
+        // torn down, but a transient transport-level error here
+        // carries no useful signal beyond what the assertions above
+        // already pin.
+        let _ = session.disconnect(russh::Disconnect::ByApplication, "", "en").await;
+        info!("SFTP read-only compliance suite passed");
+        Ok::<(), anyhow::Error>(())
+    }
+    .await;
+
+    server_process.kill_and_wait().await;
+
+    result
+}
+
+/// Standalone-server compliance entry: runs CMPTST-24..33 in numerical
+/// order. Each case spawns and tears down its own rustfs because each
+/// exercises a different server configuration (idle timeout, console
+/// listener, read-cache window) that cannot share a process with the
+/// others.
+///
+/// CMPTST-30 is omitted by default. Its assertion (per-operation
+/// wall-clock latency under pipelined metadata ops) is bounded by the
+/// SSH SFTP subsystem's per-channel serial handler dispatch and is
+/// structurally infeasible against the production code. The
+/// test_sftp_handler_latency_regression #[tokio::test] entry remains
+/// runnable on demand via `--ignored`.
+///
+/// CMPTST-31 (paused-drain) is omitted from the default suite for
+/// runtime cost (200 MiB seed plus a 25 s pause window). The
+/// test_sftp_paused_drain_regression #[tokio::test] entry covers it
+/// for direct invocation.
+///
+/// CMPTST-24, 25, 26 verify kernel-level state via ss(8) and the
+/// procfs ESTABLISHED discriminator. They are skipped on non-Linux
+/// targets where those interfaces are absent.
+pub async fn test_sftp_compliance_standalone() -> Result<()> {
+    info!("Starting SFTP standalone-server compliance suite");
+
+    #[cfg(target_os = "linux")]
+    {
+        cmptst_24::run_concurrent_half_close_no_leak().await?;
+        cmptst_25::run_wedge_kill_after_silence_in_close_wait().await?;
+        cmptst_26::run_healthy_idle_session_above_fast_threshold().await?;
+    }
+
+    cmptst_27::run_multi_session_mixed_pipelining().await?;
+    cmptst_28::run_5mb_download_with_concurrent_metadata_ops().await?;
+    cmptst_29::run_read_past_eof_volume().await?;
+    cmptst_32::run_read_cache_enabled_round_trip().await?;
+    cmptst_33::run_read_cache_disabled_round_trip().await?;
+
+    info!("SFTP standalone-server compliance suite passed");
+    Ok(())
+}