test(notify): cover prefix suffix event dispatch (#2960)

Co-authored-by: houseme <housemecn@gmail.com>

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: rustfs-release-version-bump
+description: Prepare a RustFS release branch like PR #2957 by bumping versioned files, aligning release assets, running required verification, and finishing with commit, push, and gh PR creation. Use when publishing a new RustFS alpha, beta, or stable release.
+---
+
+# RustFS Release Version Bump
+
+Use this skill when the task is to prepare a new RustFS version release branch following the pattern validated in PR `#2957`.
+
+## Read first
+
+- Read `AGENTS.md`.
+- Read `.github/pull_request_template.md`.
+- Inspect the current branch diff against `origin/main`.
+- Do not assume every release file should change in the same way every time.
+
+## Scope validated by PR #2957
+
+The `1.0.0-beta.3` release branch updated these files:
+
+- `Cargo.toml`
+- `Cargo.lock`
+- `README.md`
+- `README_ZH.md`
+- `flake.nix`
+- `helm/rustfs/Chart.yaml`
+- `rustfs.spec`
+
+Treat this file list as the default checklist for future release bumps. Only drop a file if the repository's current release pattern clearly says it is no longer part of the publish flow.
+
+## Workflow
+
+1. Confirm release intent
+- Identify the target version string exactly, for example `1.0.0-beta.4`.
+- Check whether the user wants only local commit preparation or the full `commit + push + PR` flow.
+- Compare the branch against `origin/main` and isolate only release-related edits.
+- If the target version is not explicit, stop and ask for it before editing.
+
+2. Update core Rust workspace versions
+- Bump the workspace version in `Cargo.toml`.
+- Bump all internal workspace crate versions in `Cargo.toml`.
+- Ensure `Cargo.lock` reflects the same release version for workspace packages.
+- Re-read the diff and verify there are no partial version leftovers.
+
+3. Align release assets
+- Update versioned Docker examples in `README.md` and `README_ZH.md` using the `<version>` tag form, for example `rustfs/rustfs:1.0.0-beta.4`.
+- Update `flake.nix` package version.
+- Update `helm/rustfs/Chart.yaml` `appVersion`.
+- Update `helm/rustfs/Chart.yaml` `version` using the release-chart rule:
+- `1.0.0-beta.3` -> `0.3.0`
+- `1.0.0-beta.4` -> `0.4.0`
+- Follow the same pattern for later beta releases unless the repository rule changes.
+- Update `rustfs.spec` release metadata and changelog entry.
+- Set `rustfs.spec` `Release` to the prerelease suffix without the base version, for example `beta.3` for `1.0.0-beta.3`.
+- Keep the release asset changes in a separate commit from the core Rust workspace version bump when the branch contains both.
+
+4. Stop and discuss before changing release policy
+- Ask before changing the established Docker tag style away from `<version>`.
+- Ask before changing the established Helm chart version mapping away from `beta.N -> 0.N.0`.
+- Ask before changing the established `rustfs.spec` `Release` rule away from the release suffix form such as `beta.N`.
+- Ask before widening the release scope beyond the files already validated in PR `#2957`.
+
+5. Verify before shipping
+- Run `make pre-commit`.
+- If verification succeeds, run `cargo clean` to remove generated build artifacts before wrapping up.
+- If `make pre-commit` fails, stop and return `BLOCKED`.
+
+6. Commit structure
+- Prefer two commits when the change naturally splits:
+- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`
+- `chore(release): align release assets for <version>` for docs and packaging metadata
+- If the user asks for a single commit, follow that request.
+
+7. Push and PR
+- Push the release branch with `git push -u origin <branch>` or `git push` if upstream already exists.
+- Create the PR with `gh pr create --base main --head <branch> --title ... --body-file ...`.
+- Keep the PR title and body in English.
+- Keep the `.github/pull_request_template.md` headings exactly.
+
+## Ready-to-check commands
+
+- `git diff --name-only origin/main...HEAD`
+- `git diff --stat origin/main...HEAD`
+- `make pre-commit`
+- `cargo clean`
+- `git status --short --branch`
+
+## Output expectations
+
+When using this skill, return:
+
+- The files changed for the release bump
+- Any uncertainty that needs user confirmation before editing
+- Verification status
+- Commit messages used
+- Push status and PR URL when the GitHub flow was requested
+
+## Established release policy
+
+- Docs use Docker tags in `<version>` form, not `v<version>`.
+- `helm/rustfs/Chart.yaml` `version` follows `beta.N -> 0.N.0` based on the current release policy.
+- `rustfs.spec` `Release` follows the release suffix form, for example `beta.3`.
+- If any of these rules need to change in the future, pause and confirm before editing.

.agents/skills/rustfs-release-version-bump/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "RustFS Release Bump"
+  short_description: "Prepare RustFS release branches like PR #2957."
+  default_prompt: "Use $rustfs-release-version-bump to prepare a RustFS release version, ask about any unclear version policy, and finish the commit/push/PR flow."

.agents/skills/security-advisory-lessons/SKILL.md

@@ -0,0 +1,119 @@
+---
+name: security-advisory-lessons
+description: Apply RustFS security lessons distilled from repository GitHub Security Advisories. Use when making or reviewing RustFS code changes, doing security checks, handling PR review for auth/authz, IAM, storage, RPC, logging, CORS, console/browser, encryption, policy, or endpoint changes, and when deciding which security regression tests are required.
+---
+
+# RustFS Security Advisory Lessons
+
+Use this skill as a RustFS-specific security lens before changing or approving code. For the distilled advisory lessons and review patterns, read [advisory-patterns.md](references/advisory-patterns.md).
+
+When currentness matters, fetch the live advisory inventory instead of relying on this skill as a status mirror:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories --paginate \
+  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
+```
+
+Fetch full advisory details only when the live summary suggests a new or changed lesson:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
+```
+
+For the full pattern map, read [advisory-patterns.md](references/advisory-patterns.md).
+
+## Workflow
+
+### 1. Scope the change
+- Identify touched routes, handlers, storage paths, credentials, logs, browser surfaces, CI/release code, and policy checks.
+- Treat these paths as security-sensitive by default: `rustfs/src/admin/`, `rustfs/src/storage/`, `rustfs/src/auth.rs`, `rustfs/src/server/layer.rs`, `crates/iam/`, `crates/policy/`, `crates/credentials/`, `crates/ecstore/src/rpc/`, `crates/rio/`, and console preview/auth code.
+
+### 2. Map to advisory classes
+- Read [advisory-patterns.md](references/advisory-patterns.md) for matching GHSA lessons.
+- Do not rely on advisory titles alone. Confirm whether the issue is authentication, authorization, input validation, storage invariant, browser isolation, logging, or operational hardening.
+
+### 3. Verify fail-closed behavior
+- Check that unauthenticated, wrong-permission, cross-user, cross-bucket, malformed-input, and default-config cases fail explicitly.
+- Prefer exact action/permission checks over broad helper calls or inferred ownership.
+- Confirm lower storage/RPC layers do not bypass checks done in upper layers.
+
+### 4. Require regression evidence
+- For behavior changes, add focused negative tests that reproduce the advisory class.
+- For sensitive fixes, include tests for the bypass form, not only the happy path.
+- If a test is impractical, explain the residual risk and provide a manual verification command.
+
+### 5. Report clearly
+- Lead with concrete findings and file/line evidence.
+- Separate proven vulnerabilities from hardening risks.
+- Avoid exaggerating unauthenticated impact when the code actually rejects unauthenticated requests but allows a low-privileged authenticated bypass.
+
+## Advisory-Derived Guardrails
+
+### Auth and admin authorization
+- Every admin or diagnostic route needs an explicit authn and authz story. Route registration, router whitelist, and handler-level authorization must agree.
+- Match the admin action to the operation exactly. Copy-paste action constants are a known RustFS vulnerability class.
+- Avoid authentication-only helpers for state-changing admin APIs; use `validate_admin_request` or the established equivalent with the right `AdminAction`.
+- Do not assume admin-action `Resource` scoping constrains blast radius unless the policy engine actually enforces resources for that action.
+
+### IAM and service accounts
+- Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
+- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims.
+- Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
+- Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
+
+### S3 copy, multipart, and presigned POST
+- Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
+- Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
+- Presigned POST policies are server-side contracts. Enforce `content-length-range`, key prefix, exact metadata/content-type, and all signed policy conditions.
+
+### Paths, object keys, and filesystem access
+- Never join untrusted bucket/object/RPC path strings onto filesystem roots without normalization and boundary checks.
+- Reject or safely handle `..`, absolute paths, URL-encoded traversal, platform separators, empty components, and paths that canonicalize outside the intended root.
+- Validate both S3 object-key paths and internode/RPC disk paths; storage helpers can bypass S3 authorization if they trust already-parsed paths.
+
+### Secrets, default credentials, and crypto
+- Do not ship hard-coded shared tokens, HMAC secrets, private keys, or production test keys.
+- Defaults for internode/RPC auth must fail closed for network-reachable deployments or require explicit opt-in with loud warnings.
+- License or token validation must use signatures with embedded public/verifying keys only; do not use private-key decryption as authenticity.
+- Plan key rotation and key IDs when removing exposed keys.
+
+### Logging and debug output
+- Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials.
+- Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces.
+- Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies.
+
+### RPC, parsing, and panic safety
+- Treat all RPC payload bytes as attacker-controlled. Replace `unwrap`, `expect`, and panic-prone deserialization with typed errors.
+- Malformed request tests should cover empty bytes, truncated MessagePack/protobuf, invalid enum values, stale timestamps, and invalid signatures.
+- RPC authentication must be independently strong; do not depend on S3 admin credentials unless the fallback is explicit and safe.
+
+### Browser, CORS, and console surfaces
+- Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
+- Do not render user-controlled object content in a same-origin iframe with console credentials available to JavaScript.
+- Prefer origin separation for object preview/download, `nosniff`, CSP, strict content-type handling, and avoiding durable credentials in `localStorage`.
+- License/version-like metadata endpoints should expose only coarse public data unless authenticated.
+
+### Profiling, debug, and health endpoints
+- Profiling and debug endpoints are not health checks. They require admin auth, opt-in enablement, rate limiting, and safe responses.
+- Do not return absolute filesystem paths or other deployment layout in unauthenticated or low-privilege responses.
+- Ensure health endpoint allowlists cannot accidentally include expensive diagnostics.
+
+### Trusted proxy and network identity
+- Only honor `X-Forwarded-For` or `X-Real-IP` when the request came from a configured trusted proxy.
+- Direct clients must use the socket peer address for `aws:SourceIp` and policy condition evaluation.
+- Add tests for direct spoofed headers and trusted-proxy headers.
+
+### SSE and storage invariants
+- Encryption metadata is not proof that bytes were encrypted on disk.
+- When touching reader/writer wrappers such as hashing, encryption, compression, or warp readers, verify wrapper order and inspect stored bytes in regression tests.
+- Avoid helper shortcuts that unwrap nested readers and accidentally bypass encryption or integrity layers.
+
+## Review Prompts
+
+Use these prompts while reviewing a diff:
+
+- Could a low-privileged authenticated user reach this path with the wrong action, parent, bucket, or source object?
+- Does a public/default/empty config change security behavior from fail-closed to fail-open?
+- Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
+- Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
+- Does the test prove the exploit form is denied, or only that the intended form still works?

.agents/skills/security-advisory-lessons/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Security Advisory Lessons"
+  short_description: "Apply advisory lessons in reviews."
+  default_prompt: "Review code changes against past RustFS security advisory lessons and report concrete risks, missing tests, and recommended fixes."

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -0,0 +1,93 @@
+# RustFS Advisory Pattern Map
+
+This file is a lesson map, not an advisory inventory mirror. It keeps durable security patterns distilled from RustFS GitHub Security Advisories.
+
+When current advisory state, severity, URLs, or full text matters, fetch it live:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories --paginate \
+  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
+gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
+```
+
+Update this file only when an advisory adds or changes a reusable lesson, affected surface, validation pattern, or regression-test expectation. Do not update it for state-only, URL-only, count-only, or timestamp-only changes.
+
+## Pattern Index
+
+### Admin authorization and route exposure
+
+- `GHSA-pfcq-4gjr-6gjm`: notification target endpoints accepted authenticated users but skipped admin authorization. Lesson: distinguish authn from authz; admin target CRUD must call the operation-specific admin authorization path.
+- `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
+- `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
+- `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
+- `GHSA-x5xv-223c-8vm7`: console license metadata endpoint was public. Lesson: public metadata endpoints should be coarse or authenticated.
+
+### IAM import, service accounts, and privilege boundaries
+
+- `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
+- `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
+- `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
+
+### S3 copy, multipart, and upload policy validation
+
+- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
+- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
+- `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
+
+### Filesystem paths and object key traversal
+
+- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
+- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
+
+### Secrets, defaults, and cryptographic misuse
+
+- `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
+- `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
+- `GHSA-923g-jp7v-f97f`: license verification embedded a production RSA private key and used private-key decryption as authenticity. Lesson: ship verifying/public keys only and use real signature verification.
+
+### Sensitive logging and debug output
+
+- `GHSA-r54g-49rx-98cr`: STS credentials were logged at info level. Lesson: generated credentials must never be logged in plaintext.
+- `GHSA-8cm2-h255-v749`: debug logs leaked session tokens, secret keys, JWT claims, and raw STS response bodies. Lesson: redaction must cover custom `Debug` implementations and dependency response-body logging.
+- `GHSA-333v-68xh-8mmq`: invalid RPC signature logging included the shared HMAC secret and expected signature. Lesson: error paths often leak secrets; never log raw secrets or derived authenticators.
+
+### RPC input validation and panic safety
+
+- `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
+- `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
+
+### Browser, CORS, and console isolation
+
+- `GHSA-v9fg-3cr2-277j`: object preview rendered attacker-controlled HTML in a same-origin iframe, exposing console credentials stored in `localStorage`. Lesson: user content must be origin-isolated from the console and protected with `nosniff`, CSP, and strict content-type handling.
+- `GHSA-x5xv-223c-8vm7`: default CORS reflected arbitrary origins with credentials. Lesson: never combine reflected origins with `Access-Control-Allow-Credentials: true`; default should be fail-closed.
+
+### Trusted proxy and source IP conditions
+
+- `GHSA-fc6g-2gcp-2qrq`: `aws:SourceIp` trusted client-supplied `X-Forwarded-For` or `X-Real-IP`. Lesson: forwarded IP headers are valid only behind configured trusted proxies; direct clients use socket peer IP.
+
+### SSE and on-disk storage invariants
+
+- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
+
+## Useful Search Seeds
+
+Use these targeted searches when a diff touches security-sensitive code:
+
+```bash
+rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
+rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
+rg -n "PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
+rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
+rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
+rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
+```
+
+## Minimum Regression Test Expectations
+
+- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
+- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
+- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
+- Path fixes: include encoded traversal, absolute path, nested traversal, valid object keys that resemble traversal text but should be rejected, and canonical boundary checks.
+- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
+- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
+- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.

.config/make/lint-fmt.mak

@@ -16,6 +16,11 @@ clippy-check: core-deps ## Run clippy checks
 	cargo clippy --fix --allow-dirty
 	cargo clippy --all-targets --all-features -- -D warnings
 
+.PHONY: unsafe-code-check
+unsafe-code-check: ## Check unsafe_code allowances have SAFETY comments
+	@echo "🔒 Checking unsafe_code allowances..."
+	./scripts/check_unsafe_code_allowances.sh
+
 .PHONY: compilation-check
 compilation-check: core-deps ## Run compilation check
 	@echo "🔨 Running compilation check..."

.config/make/pre-commit.mak

@@ -7,5 +7,5 @@ setup-hooks: ## Set up git hooks
 	@echo "✅ Git hooks setup complete!"
 
 .PHONY: pre-commit
-pre-commit: fmt clippy-check compilation-check test ## Run pre-commit checks
-	@echo "✅ All pre-commit checks passed!"
+pre-commit: fmt unsafe-code-check clippy-check compilation-check test ## Run pre-commit checks
+	@echo "✅ All pre-commit checks passed!"

.config/make/tests.mak

@@ -2,8 +2,13 @@
 
 TEST_THREADS ?= 1
 
+.PHONY: script-tests
+script-tests: ## Run shell script tests
+	@echo "Running script tests..."
+	./scripts/test_build_rustfs_options.sh
+
 .PHONY: test
-test: core-deps test-deps ## Run all tests
+test: core-deps test-deps script-tests ## Run all tests
 	@echo "🧪 Running tests..."
 	@if command -v cargo-nextest >/dev/null 2>&1; then \
 		cargo nextest run --all --exclude e2e_test; \

.docker/compose/README.md

@@ -36,6 +36,24 @@ To start a 4-node cluster for distributed testing:
 docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
 ```
 
+### Script-Based 4-Node Validation (Recommended)
+
+Use the local validation script when you need local-source image build, failover checks,
+and benchmark workflow in one command:
+
+```bash
+# Default mode: WAIT_PROBE_MODE=service
+# This avoids false negatives where /health/ready remains 503 locally
+# while the service path is already available.
+./scripts/run_four_node_cluster_failover_bench.sh
+```
+
+Strict mode is available when you explicitly want `/health/ready == 200` as the gate:
+
+```bash
+WAIT_PROBE_MODE=ready ./scripts/run_four_node_cluster_failover_bench.sh
+```
+
 ### (Deprecated) Minimal Observability
 
 ```bash

.docker/compose/docker-compose.cluster.local-build.yml

@@ -0,0 +1,144 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  node1:
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node1
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node1_data_0:/data/rustfs0
+      - node1_data_1:/data/rustfs1
+      - node1_data_2:/data/rustfs2
+      - node1_data_3:/data/rustfs3
+    ports:
+      - "9000:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node2:
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node2
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node2_data_0:/data/rustfs0
+      - node2_data_1:/data/rustfs1
+      - node2_data_2:/data/rustfs2
+      - node2_data_3:/data/rustfs3
+    ports:
+      - "9001:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node3:
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node3
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node3_data_0:/data/rustfs0
+      - node3_data_1:/data/rustfs1
+      - node3_data_2:/data/rustfs2
+      - node3_data_3:/data/rustfs3
+    ports:
+      - "9002:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node4:
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node4
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node4_data_0:/data/rustfs0
+      - node4_data_1:/data/rustfs1
+      - node4_data_2:/data/rustfs2
+      - node4_data_3:/data/rustfs3
+    ports:
+      - "9003:9000"
+    networks:
+      - rustfs-cluster-net
+
+volumes:
+  node1_data_0:
+  node1_data_1:
+  node1_data_2:
+  node1_data_3:
+  node2_data_0:
+  node2_data_1:
+  node2_data_2:
+  node2_data_3:
+  node3_data_0:
+  node3_data_1:
+  node3_data_2:
+  node3_data_3:
+  node4_data_0:
+  node4_data_1:
+  node4_data_2:
+  node4_data_3:
+
+networks:
+  rustfs-cluster-net:
+    driver: bridge

.docker/compose/docker-compose.observability.yaml

@@ -26,7 +26,7 @@ services:
     restart: "no"
 
   tempo:
-    image: grafana/tempo:latest
+    image: grafana/tempo:2.10.5
     user: "10001"
     command: [ "-config.file=/etc/tempo.yaml" ]
     volumes:
@@ -36,9 +36,19 @@ services:
       - "3200:3200" # tempo
       - "4317"      # otlp grpc
       - "4318"      # otlp http
+      - "7946"      # memberlist
     restart: unless-stopped
     networks:
       - rustfs-network
+    depends_on:
+      tempo-init:
+        condition: service_completed_successfully
+    healthcheck:
+      test: [ "CMD", "/tempo", "-version" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
 
   otel-collector:
     image: otel/opentelemetry-collector-contrib:latest
@@ -61,6 +71,12 @@ services:
       - jaeger
       - prometheus
       - loki
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "/otelcol-contrib", "--version" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
 
   jaeger:
     image: jaegertracing/jaeger:latest
@@ -72,12 +88,23 @@ services:
       - BADGER_DIRECTORY_KEY=/badger/key
       - COLLECTOR_OTLP_ENABLED=true
     volumes:
+      - ../../.docker/observability/jaeger.yaml:/etc/jaeger/config.yml:ro
       - jaeger-data:/badger
     ports:
       - "16686:16686" # Web UI
       - "14269:14269" # Admin/Metrics
+      - "4317"        # otlp grpc
+      - "4318"        # otlp http
+    command: [ "--config", "/etc/jaeger/config.yml" ]
     networks:
       - rustfs-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
 
   prometheus:
     image: prom/prometheus:latest
@@ -85,6 +112,7 @@ services:
       - TZ=Asia/Shanghai
     volumes:
       - ../../.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - ../../.docker/observability/prometheus-rules:/etc/prometheus/rules:ro
       - prometheus-data:/prometheus
     ports:
       - "9090:9090"
@@ -94,23 +122,45 @@ services:
       - '--web.enable-remote-write-receiver'
       - '--enable-feature=promql-experimental-functions'
       - '--storage.tsdb.path=/prometheus'
-      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
-      - '--web.console.templates=/usr/share/prometheus/consoles'
+      - '--storage.tsdb.retention.time=30d'
     networks:
       - rustfs-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
 
   loki:
     image: grafana/loki:latest
     environment:
       - TZ=Asia/Shanghai
     volumes:
-      - ../../.docker/observability/loki.yaml:/etc/loki/local-config.yaml:ro
+      - ../../.docker/observability/loki.yaml:/etc/loki/loki.yaml:ro
       - loki-data:/loki
     ports:
       - "3100:3100"
-    command: -config.file=/etc/loki/local-config.yaml
+    command: -config.file=/etc/loki/loki.yaml
     networks:
       - rustfs-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
+      interval: 15s
+      timeout: 10s
+      retries: 5
+      start_period: 60s
+
+  pyroscope:
+    image: grafana/pyroscope:latest
+    ports:
+      - "4040:4040"
+    command:
+      - -self-profiling.disable-push=true
+    networks:
+      - rustfs-network
+    restart: unless-stopped
 
   grafana:
     image: grafana/grafana:latest
@@ -127,10 +177,17 @@ services:
     volumes:
       - ../../.docker/observability/grafana/provisioning:/etc/grafana/provisioning:ro
       - ../../.docker/observability/grafana/dashboards:/var/lib/grafana/dashboards:ro
+      - grafana-data:/var/lib/grafana
     depends_on:
       - prometheus
       - tempo
       - loki
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
 
   # --- RustFS Cluster ---
 
@@ -215,6 +272,7 @@ volumes:
   tempo-data:
   loki-data:
   jaeger-data:
+  grafana-data:
 
 networks:
   rustfs-network:

.docker/nginx/nginx.conf

@@ -74,6 +74,17 @@ http {
     #     ssl_certificate /etc/nginx/ssl/server.crt;
     #     ssl_certificate_key /etc/nginx/ssl/server.key;
     #
+    #     # Restrict to modern TLS versions and ciphers. Operators copying this
+    #     # example must keep at least these directives — without them, nginx
+    #     # may negotiate older protocol versions that have known weaknesses.
+    #     ssl_protocols TLSv1.2 TLSv1.3;
+    #     ssl_prefer_server_ciphers on;
+    #     ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305';
+    #     ssl_session_timeout 1d;
+    #     ssl_session_cache shared:SSL:10m;
+    #     ssl_session_tickets off;
+    #     # add_header Strict-Transport-Security "max-age=63072000" always;
+    #
     #     location / {
     #         proxy_pass http://rustfs:9000;
     #         ...

.docker/observability/README.md

@@ -13,6 +13,9 @@ The stack is composed of the following best-in-class open-source components:
 - **Jaeger** (v1.59.0): Distributed tracing system (configured as a secondary UI/storage).
 - **OpenTelemetry Collector** (v0.104.0): A vendor-agnostic implementation for receiving, processing, and exporting telemetry data.
 
+By default, this stack uses Tempo in single-binary mode and does not require Kafka/Redpanda.
+If you want the Kafka-backed HA Tempo path, use `docker-compose-example-for-rustfs.yml` together with `docker-compose-tempo-ha-override.yml`.
+
 ## Architecture
 
 1. **Telemetry Collection**: Applications send OTLP (OpenTelemetry Protocol) data (Metrics, Logs, Traces) to the **OpenTelemetry Collector**.
@@ -46,6 +49,15 @@ Run the following command to start the entire stack:
 docker compose up -d
 ```
 
+### High Availability Tempo
+
+The default `docker-compose.yml` is the single-node stack.
+If you need the Kafka-backed HA Tempo configuration, start it with:
+
+```bash
+docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
+```
+
 ### Access Dashboards
 
 | Service        | URL                                              | Credentials       | Description                    |

.docker/observability/README_ZH.md

@@ -13,6 +13,9 @@
 - **Jaeger** (v1.59.0): 分布式追踪系统（配置为辅助 UI/存储）。
 - **OpenTelemetry Collector** (v0.104.0): 接收、处理和导出遥测数据的供应商无关实现。
 
+默认情况下，这套技术栈使用 Tempo 单二进制模式，不依赖 Kafka/Redpanda。
+如果需要基于 Kafka 的 HA Tempo 路径，请使用 `docker-compose-example-for-rustfs.yml` 配合 `docker-compose-tempo-ha-override.yml`。
+
 ## 架构
 
 1.  **遥测收集**: 应用程序将 OTLP (OpenTelemetry Protocol) 数据（指标、日志、追踪）发送到 **OpenTelemetry Collector**。
@@ -46,6 +49,15 @@
 docker compose up -d
 ```
 
+### Tempo 高可用模式
+
+默认的 `docker-compose.yml` 对应单机栈。
+如果需要基于 Kafka 的 HA Tempo 配置，请使用：
+
+```bash
+docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
+```
+
 ### 访问仪表盘
 
 | 服务 | URL | 凭据 | 描述 |

.docker/observability/docker-compose-example-for-rustfs.yml

@@ -85,10 +85,8 @@ services:
     networks:
       - otel-network
     restart: unless-stopped
-    depends_on:
-      - redpanda
     healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
+      test: [ "CMD", "/tempo", "-version" ]
       interval: 10s
       timeout: 5s
       retries: 3
@@ -205,7 +203,7 @@ services:
       - prometheus
       - loki
     healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:13133" ]
+      test: [ "CMD", "/otelcol-contrib", "--version" ]
       interval: 10s
       timeout: 5s
       retries: 3

.docker/observability/docker-compose.yml

@@ -16,10 +16,8 @@ services:
 
   # --- Tracing ---
   tempo:
-    image: grafana/tempo:2.10.3
+    image: grafana/tempo:2.10.5
     container_name: tempo
-    depends_on:
-      - redpanda
     command: [ "-config.file=/etc/tempo.yaml" ]
     volumes:
       - ./tempo.yaml:/etc/tempo.yaml:ro
@@ -33,31 +31,11 @@ services:
       - otel-network
     restart: unless-stopped
     healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
+      test: [ "CMD", "/tempo", "-version" ]
       interval: 10s
       timeout: 5s
       retries: 3
       start_period: 15s
-  redpanda:
-    image: redpandadata/redpanda:latest
-    ports:
-      - "9092:9092" # Kafka API for clients
-    command: >
-      redpanda start --overprovisioned
-      --mode=dev-container
-      --kafka-addr=PLAINTEXT://0.0.0.0:9092
-      --advertise-kafka-addr=PLAINTEXT://redpanda:9092
-
-  redpanda-console:
-    image: docker.redpanda.com/redpandadata/console:latest
-    environment:
-      - CONFIG_FILEPATH=/etc/redpanda/redpanda-console-config.yaml
-    volumes:
-      - ./redpanda-console.yaml:/etc/redpanda/redpanda-console-config.yaml
-    ports:
-      - "8080:8080"
-    depends_on:
-      - redpanda
 
   vulture:
     image: grafana/tempo-vulture:latest
@@ -106,6 +84,7 @@ services:
     container_name: prometheus
     volumes:
       - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - ./prometheus-rules:/etc/prometheus/rules:ro
       - prometheus-data:/prometheus
     ports:
       - "9090:9090"
@@ -168,7 +147,7 @@ services:
       - prometheus
       - loki
     healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:13133" ]
+      test: [ "CMD", "/otelcol-contrib", "--version" ]
       interval: 10s
       timeout: 5s
       retries: 3

.docker/observability/grafana/provisioning/datasources.yaml

@@ -1,97 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-apiVersion: 1
-
-datasources:
-  - name: Prometheus
-    type: prometheus
-    uid: prometheus
-    access: proxy
-    orgId: 1
-    url: http://prometheus:9090
-    isDefault: true
-    version: 1
-    editable: false
-    jsonData:
-      httpMethod: GET
-      exemplarTraceIdDestinations:
-        - name: trace_id
-          datasourceUid: tempo
-
-  - name: Tempo
-    type: tempo
-    uid: tempo
-    access: proxy
-    orgId: 1
-    url: http://tempo:3200
-    isDefault: false
-    version: 1
-    editable: false
-    jsonData:
-      httpMethod: GET
-      serviceMap:
-        datasourceUid: prometheus
-      tracesToLogs:
-        datasourceUid: loki
-        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
-        mappedTags: [ { key: 'service.name', value: 'app' } ]
-        spanStartTimeShift: '1s'
-        spanEndTimeShift: '-1s'
-        filterByTraceID: true
-        filterBySpanID: false
-      tracesToMetrics:
-        datasourceUid: prometheus
-        tags: [ { key: 'service.name' }, { key: 'job' } ]
-        queries:
-          - name: 'Service-Level Latency'
-            query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m])) by (le)'
-          - name: 'Service-Level Calls'
-            query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))'
-          - name: 'Service-Level Errors'
-            query: 'sum(rate(traces_spanmetrics_calls_total{status_code="ERROR", $$__tags}[5m]))'
-      nodeGraph:
-        enabled: true
-
-  - name: Loki
-    type: loki
-    uid: loki
-    orgId: 1
-    url: http://loki:3100
-    isDefault: false
-    version: 1
-    editable: false
-    jsonData:
-      derivedFields:
-        - datasourceUid: tempo
-          matcherRegex: 'trace_id=(\w+)'
-          name: 'TraceID'
-          url: '$${__value.raw}'
-
-  - name: Jaeger
-    type: jaeger
-    uid: jaeger
-    url: http://jaeger:16686
-    access: proxy
-    isDefault: false
-    editable: false
-    jsonData:
-      tracesToLogs:
-        datasourceUid: loki
-        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
-        mappedTags: [ { key: 'service.name', value: 'app' } ]
-        spanStartTimeShift: '1s'
-        spanEndTimeShift: '-1s'
-        filterByTraceID: true
-        filterBySpanID: false

.docker/observability/prometheus-rules/rustfs-dashboard.yml

@@ -0,0 +1,40 @@
+groups:
+  - name: rustfs-dashboard
+    interval: 30s
+    rules:
+      - record: rustfs:http_server_requests:rate5m
+        expr: sum by (job) (rate(rustfs_http_server_requests_total[5m]))
+
+      - record: rustfs:http_server_request_duration_seconds:p50_5m
+        expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
+      - record: rustfs:http_server_request_duration_seconds:p95_5m
+        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
+      - record: rustfs:http_server_request_duration_seconds:p99_5m
+        expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
+
+      - record: rustfs:http_server_response_body_size_bytes:p50_5m
+        expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
+      - record: rustfs:http_server_response_body_size_bytes:p95_5m
+        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
+      - record: rustfs:http_server_response_body_size_bytes:p99_5m
+        expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
+
+      - record: rustfs:log_cleaner_runs:rate15m
+        expr: sum by (job) (rate(rustfs_log_cleaner_runs_total[15m]))
+      - record: rustfs:log_cleaner_failure_ratio:rate5m
+        expr: sum by (job) (rate(rustfs_log_cleaner_run_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_runs_total[5m])), 1e-9)
+      - record: rustfs:log_cleaner_rotation_failure_ratio:rate5m
+        expr: sum by (job) (rate(rustfs_log_cleaner_rotation_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_rotation_total[5m])), 1e-9)
+      - record: rustfs:log_cleaner_rotation_duration_seconds:p95_5m
+        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_rotation_duration_seconds_bucket[5m])))
+      - record: rustfs:log_cleaner_compress_duration_seconds:p95_5m
+        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_compress_duration_seconds_bucket[5m])))
+
+      - record: rustfs:scanner_objects_scanned:rate5m
+        expr: sum by (job) (rate(rustfs_scanner_objects_scanned_total[5m]))
+      - record: rustfs:scanner_directories_scanned:rate5m
+        expr: sum by (job) (rate(rustfs_scanner_directories_scanned_total[5m]))
+      - record: rustfs:scanner_buckets_scanned:rate5m
+        expr: sum by (job) (rate(rustfs_scanner_buckets_scanned_total[5m]))
+      - record: rustfs:scanner_cycles_success:rate5m
+        expr: sum by (job) (rate(rustfs_scanner_cycles_total{result="success"}[5m]))

.docker/observability/prometheus.yml

@@ -19,6 +19,9 @@ global:
     cluster: 'rustfs-dev'  # Label to identify the cluster
     replica: '1'  # Replica identifier
 
+rule_files:
+  - /etc/prometheus/rules/*.yml
+
 scrape_configs:
   - job_name: 'otel-collector'
     static_configs:

.docker/observability/tempo.yaml

@@ -12,7 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-partition_ring_live_store: true
 stream_over_http_enabled: true
 
 server:
@@ -33,33 +32,17 @@ distributor:
           endpoint: "0.0.0.0:4317"
         http:
           endpoint: "0.0.0.0:4318"
-  #log_received_spans:
-  #    enabled: true
-  #  log_discarded_spans:
-  #    enabled: true
 
-backend_scheduler:
-  provider:
-    compaction:
-      compaction:
-        block_retention: 1h
-
-backend_worker:
-  backend_scheduler_addr: localhost:3200
-  compaction:
-    block_retention: 1h
-  ring:
-    kvstore:
-      store: memberlist
-
-querier:
-  query_live_store: true
+ingester:
+  max_block_duration: 5m
 
 metrics_generator:
   registry:
     external_labels:
       source: tempo
       cluster: docker-compose
+  traces_storage:
+    path: /var/tempo/generator/traces
   storage:
     path: /var/tempo/generator/wal
     remote_write:
@@ -85,15 +68,5 @@ overrides:
       processors: [ "span-metrics", "service-graphs", "local-blocks" ]
       generate_native_histograms: both
 
-ingest:
-  enabled: true
-  kafka:
-    address: redpanda:9092
-    topic: tempo-ingest
-
-block_builder:
-  consume_cycle_duration: 30s
-
 usage_report:
   reporting_enabled: false
-

.docker/test/issue-2715/docker-compose-test.yml

@@ -0,0 +1,52 @@
+services:
+  rustfs:
+    image: rustfs/rustfs:1.0.0-alpha.99-glibc
+    container_name: rustfs-issue-2715-test
+    security_opt:
+      - "no-new-privileges:true"
+    ports:
+      - "19000:9000"
+      - "19001:9001"
+    environment:
+      - RUSTFS_VOLUMES=/data/rustfs{0...8}
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_ACCESS_KEY=admin
+      - RUSTFS_SECRET_KEY=admin
+      - RUSTFS_OBS_LOGGER_LEVEL=info
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
+      - RUSTFS_STORAGE_CLASS_STANDARD=EC:2
+      - RUSTFS_STORAGE_CLASS_RRS=EC:1
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true
+      - RUSTFS_OBS_LOG_DIRECTORY=/opt/rustfs/logs
+    extra_hosts:
+      - "otel-collector:host-gateway"
+      - "pyroscope:host-gateway"
+    volumes:
+      - ./deploy/data/issue-2715/rustfs0:/data/rustfs0
+      - ./deploy/data/issue-2715/rustfs1:/data/rustfs1
+      - ./deploy/data/issue-2715/rustfs2:/data/rustfs2
+      - ./deploy/data/issue-2715/rustfs3:/data/rustfs3
+      - ./deploy/data/issue-2715/rustfs4:/data/rustfs4
+      - ./deploy/data/issue-2715/rustfs5:/data/rustfs5
+      - ./deploy/data/issue-2715/rustfs6:/data/rustfs6
+      - ./deploy/data/issue-2715/rustfs7:/data/rustfs7
+      - ./deploy/data/issue-2715/rustfs8:/data/rustfs8
+      - ./deploy/logs/issue-2715:/opt/rustfs/logs
+    restart: unless-stopped
+    healthcheck:
+      test:
+        [
+          "CMD",
+          "sh",
+          "-c",
+          "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health"
+        ]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 40s

.docker/test/issues-2815/.gitignore

@@ -0,0 +1 @@
+data/

.docker/test/issues-2815/README.md

@@ -0,0 +1,106 @@
+# Issue 2815 Local Docker Verification
+
+## Purpose
+
+This directory contains the local distributed Docker verification assets used to validate issue `#2815` against the current source build.
+
+The target behavior is:
+
+- 4-node distributed cluster starts successfully
+- `/health/ready` becomes reachable on each node
+- logs no longer contain `storage_info failed: Io error: wrong msgpack marker FixArray(1)`
+- internode RPC authentication succeeds with an explicit non-default RPC secret
+
+## Files
+
+- `docker-compose.yml`: 4-node distributed cluster using a locally built image
+
+## Data Directories
+
+Create the bind-mount directories before `docker compose up`:
+
+```bash
+mkdir -p .docker/test/issues-2815/data/rustfs{1..4}-disk{0..3}
+```
+
+## Build
+
+Apple Silicon / arm64 host:
+
+```bash
+docker build --platform linux/arm64 -f Dockerfile.source -t rustfs-issue-2815-local .
+```
+
+If you intentionally want amd64 emulation:
+
+```bash
+docker build --platform linux/amd64 -f Dockerfile.source -t rustfs-issue-2815-local .
+```
+
+## Run
+
+```bash
+docker compose -f .docker/test/issues-2815/docker-compose.yml up -d
+```
+
+If the image platform is not `linux/arm64`, align compose explicitly:
+
+```bash
+RUSTFS_DOCKER_PLATFORM=linux/amd64 docker compose -f .docker/test/issues-2815/docker-compose.yml up -d
+```
+
+## Health Checks
+
+Container-level healthcheck is now included and probes:
+
+```bash
+curl -fsS http://127.0.0.1:9000/health
+```
+
+Manual checks:
+
+```bash
+curl -i http://127.0.0.1:9101/health/ready
+curl -i http://127.0.0.1:9102/health/ready
+curl -i http://127.0.0.1:9103/health/ready
+curl -i http://127.0.0.1:9104/health/ready
+```
+
+## RPC Secret Requirement
+
+The current source build no longer reproduces the original `FixArray(1)` decode error from issue `#2815`.
+
+Earlier local Docker attempts failed during erasure bootstrap with:
+
+```text
+No valid auth token
+store init failed to load formats after 10 retries: erasure read quorum
+```
+
+Root cause:
+
+- RPC authentication rejects the default secret `rustfsadmin`
+- distributed local Docker validation therefore needs an explicit non-default secret
+
+This compose now sets both:
+
+- `RUSTFS_SECRET_KEY=issue-2815-secret`
+- `RUSTFS_RPC_SECRET=issue-2815-rpc-secret`
+
+With those values in place, the current 4-node local Docker cluster reaches healthy state and `/health/ready` returns `200`.
+
+In other words:
+
+- `RUSTFS_ACCESS_KEY` may still be `rustfsadmin` for local service credentials if desired
+- `RUSTFS_SECRET_KEY` can still be used for service credentials
+- but RPC authentication must not resolve to the default secret value `rustfsadmin`
+- if `RUSTFS_RPC_SECRET` is unset, the code falls back to `RUSTFS_SECRET_KEY`
+- so at least one of them must provide a non-default shared secret for internode RPC signing
+
+## Suggested Debug Commands
+
+```bash
+docker compose -f .docker/test/issues-2815/docker-compose.yml ps
+docker compose -f .docker/test/issues-2815/docker-compose.yml logs --no-color --tail=200
+docker compose -f .docker/test/issues-2815/docker-compose.yml down -v
+```

.docker/test/issues-2815/docker-compose.yml

@@ -0,0 +1,120 @@
+services:
+  rustfs1:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs1
+    container_name: rustfs-issue-2815-rustfs1
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs1-disk0:/data/rustfs0
+      - ./data/rustfs1-disk1:/data/rustfs1
+      - ./data/rustfs1-disk2:/data/rustfs2
+      - ./data/rustfs1-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9101:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+  rustfs2:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs2
+    container_name: rustfs-issue-2815-rustfs2
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs2-disk0:/data/rustfs0
+      - ./data/rustfs2-disk1:/data/rustfs1
+      - ./data/rustfs2-disk2:/data/rustfs2
+      - ./data/rustfs2-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9102:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+  rustfs3:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs3
+    container_name: rustfs-issue-2815-rustfs3
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs3-disk0:/data/rustfs0
+      - ./data/rustfs3-disk1:/data/rustfs1
+      - ./data/rustfs3-disk2:/data/rustfs2
+      - ./data/rustfs3-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9103:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+  rustfs4:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs4
+    container_name: rustfs-issue-2815-rustfs4
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs4-disk0:/data/rustfs0
+      - ./data/rustfs4-disk1:/data/rustfs1
+      - ./data/rustfs4-disk2:/data/rustfs2
+      - ./data/rustfs4-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9104:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+networks:
+  rustfs-issue-2815-net:
+    name: rustfs-issue-2815-net

.github/dependabot.yml

@@ -27,9 +27,8 @@ updates:
       timezone: "Asia/Shanghai"
       time: "08:00"
     assignees:
-      - "heihutu"
-    reviewers:
       - "houseme"
+    reviewers:
       - "overtrue"
       - "majinghe"
     ignore:
@@ -39,6 +38,8 @@ updates:
         versions: [ "0.23.x" ]
       - dependency-name: "ratelimit"
         versions: [ "1.x" ]
+      - dependency-name: "ratelimit"
+        versions: [ "2.x" ]
     groups:
       s3s:
         update-types:

.github/pull_request_template.md

@@ -2,35 +2,34 @@
 Pull Request Template for RustFS
 -->
 
-## Type of Change
-- [ ] New Feature
-- [ ] Bug Fix
-- [ ] Documentation
-- [ ] Performance Improvement
-- [ ] Test/CI
-- [ ] Refactor
-- [ ] Other:
-
 ## Related Issues
-<!-- List related Issue numbers, e.g. #123 -->
+<!--
+List related issues, e.g. Fixes #123.
+Use N/A when there is no related issue.
+-->
 
 ## Summary of Changes
-<!-- Briefly describe the main changes and motivation for this PR -->
+<!--
+Briefly explain what changed and why reviewers should accept it.
+Focus on behavior, compatibility, and review-relevant context.
+-->
 
-## Checklist
-- [ ] I have read and followed the [CONTRIBUTING.md](CONTRIBUTING.md) guidelines
-- [ ] Passed `make pre-commit`
-- [ ] Added/updated necessary tests
-- [ ] Documentation updated (if needed)
-- [ ] CI/CD passed (if applicable)
+## Verification
+<!--
+List the commands or checks you ran, for example:
+- `make pre-commit`
+
+Use N/A only when verification is not applicable.
+-->
 
 ## Impact
-- [ ] Breaking change (compatibility)
-- [ ] Requires doc/config/deployment update
-- [ ] Other impact:
+<!--
+Describe user-facing, compatibility, API, deployment, configuration, or
+documentation impact. Use N/A when there is no expected impact.
+-->
 
 ## Additional Notes
-<!-- Any extra information for reviewers -->
+<!-- Any extra information for reviewers, or N/A. -->
 
 ---
 

.github/workflows/ci.yml

@@ -130,6 +130,9 @@ jobs:
       - name: Check code formatting
         run: cargo fmt --all --check
 
+      - name: Check unsafe code allowances
+        run: ./scripts/check_unsafe_code_allowances.sh
+
       - name: Run clippy lints
         run: cargo clippy --all-targets --all-features -- -D warnings
 

.github/workflows/docker.yml

@@ -163,11 +163,10 @@ jobs:
                 if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
                   build_type="prerelease"
                   is_prerelease=true
-                  # TODO: Temporary change - currently allows alpha versions to also create latest tags
-                  # After the version is stable, you need to remove the following line and restore the original logic (latest is created only for stable versions)
-                  if [[ "$version" == *"alpha"* ]]; then
+                  # Current policy: create latest tags for stable releases and selected prereleases (alpha/beta).
+                  if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]]; then
                     create_latest=true
-                     echo "🧪 Building Docker image for prerelease: $version (temporarily allowing creation of latest tag)"
+                    echo "🧪 Building Docker image for prerelease: $version (creating latest tag)"
                   else
                     echo "🧪 Building Docker image for prerelease: $version"
                   fi
@@ -216,11 +215,10 @@ jobs:
               v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
                 build_type="prerelease"
                 is_prerelease=true
-                # TODO: Temporary change - currently allows alpha versions to also create latest tags
-                # After the version is stable, you need to remove the if block below and restore the original logic.
-                if [[ "$input_version" == *"alpha"* ]]; then
+                # Current policy: create latest tags for stable releases and selected prereleases (alpha/beta).
+                if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]]; then
                   create_latest=true
-                   echo "🧪 Building with prerelease version: $input_version (temporarily allowing creation of latest tag)"
+                  echo "🧪 Building with prerelease version: $input_version (creating latest tag)"
                 else
                   echo "🧪 Building with prerelease version: $input_version"
                 fi
@@ -351,9 +349,7 @@ jobs:
 
           # Add channel tags for prereleases and latest for stable
           if [[ "$CREATE_LATEST" == "true" ]]; then
-            # TODO: Temporary change - the current alpha version will also create the latest tag
-            # After the version is stabilized, the logic here remains unchanged, but the upstream CREATE_LATEST setting needs to be restored.
-            # Stable release (and temporary alpha versions)
+            # Create latest tags for stable releases and selected prereleases when CREATE_LATEST=true.
             TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
           elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
             # Prerelease channel tags (alpha, beta, rc)
@@ -450,10 +446,9 @@ jobs:
             "prerelease")
               echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
               echo "⚠️  This is a prerelease image - use with caution"
-              # TODO: Temporary change - alpha versions currently create the latest tag
-              # After the version is stable, you need to restore the following prompt information
-              if [[ "$VERSION" == *"alpha"* ]] && [[ "$CREATE_LATEST" == "true" ]]; then
-                echo "🏷️  Latest tag has been created for alpha version (temporary measures)"
+              # Create latest tags for stable releases and selected prereleases when CREATE_LATEST=true.
+              if [[ "$CREATE_LATEST" == "true" ]]; then
+                echo "🏷️  Latest tag has been created for prerelease: $VERSION"
               else
                 echo "🚫 Latest tag NOT created for prerelease"
               fi

.github/workflows/helm-package.yml

@@ -18,41 +18,78 @@ on:
   workflow_run:
     workflows: [ "Build and Release" ]
     types: [ completed ]
+  workflow_dispatch:
+    inputs:
+      version:
+        description: "Release version to publish, e.g. 1.0.0-beta.1 or v1.0.0-beta.1"
+        required: true
+        default: "1.0.0-beta.1"
+        type: string
 
 permissions:
   contents: read
 
-env:
-  new_version: ${{ github.event.workflow_run.head_branch }}
-
 jobs:
   build-helm-package:
     runs-on: ubicloud-standard-2
-    # Only run on successful builds triggered by tag pushes (version format: x.y.z or x.y.z-suffix)
     if: |
-      github.event.workflow_run.conclusion == 'success' && 
-      github.event.workflow_run.event == 'push' &&
-      contains(github.event.workflow_run.head_branch, '.')
+      github.event_name == 'workflow_dispatch' ||
+      (
+        github.event.workflow_run.conclusion == 'success' &&
+        github.event.workflow_run.event == 'push' &&
+        contains(github.event.workflow_run.head_branch, '.')
+      )
+
+    outputs:
+      raw_tag: ${{ steps.version.outputs.raw_tag }}
+      app_version: ${{ steps.version.outputs.app_version }}
+      chart_version: ${{ steps.version.outputs.chart_version }}
 
     steps:
       - name: Checkout helm chart repo
         uses: actions/checkout@v6
 
-      - name: Replace chart app version
+      - name: Normalize release version
+        id: version
         run: |
-          set -e
-          set -x
-          old_version=$(grep "^appVersion:" helm/rustfs/Chart.yaml | awk '{print $2}')
-          sed -i "s/$old_version/$new_version/g" helm/rustfs/Chart.yaml
+          set -eux
+
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            RAW="${{ github.event.inputs.version }}"
+          else
+            RAW="${{ github.event.workflow_run.head_branch }}"
+          fi
+
+          case "$RAW" in
+            refs/tags/*)
+              RAW_TAG="${RAW#refs/tags/}"
+              ;;
+            *)
+              RAW_TAG="$RAW"
+              ;;
+          esac
+
+          ./scripts/helm_chart_version.sh "$RAW_TAG"
+
+      - name: Replace chart version and app version
+        run: |
+          set -eux
+          sed -i -E 's/^version:.*/version: "${{ steps.version.outputs.chart_version }}"/' helm/rustfs/Chart.yaml
+          sed -i -E 's/^appVersion:.*/appVersion: "${{ steps.version.outputs.app_version }}"/' helm/rustfs/Chart.yaml
 
       - name: Set up Helm
         uses: azure/setup-helm@v4.3.0
 
+      - name: Test Helm Chart Templates
+        run: ./scripts/test_helm_templates.sh
+
       - name: Package Helm Chart
         run: |
+          set -eux
           cp helm/README.md helm/rustfs/
-          package_version=$(echo $new_version | awk -F '-' '{print $2}' | awk -F '.' '{print $NF}') 
-          helm package ./helm/rustfs --destination helm/rustfs/ --version "0.0.$package_version"
+          helm package ./helm/rustfs \
+            --destination helm/rustfs/ \
+            --version "${{ steps.version.outputs.chart_version }}"
 
       - name: Upload helm package as artifact
         uses: actions/upload-artifact@v6
@@ -64,6 +101,7 @@ jobs:
   publish-helm-package:
     runs-on: ubicloud-standard-2
     needs: [ build-helm-package ]
+    if: needs.build-helm-package.result == 'success'
 
     steps:
       - name: Checkout helm package repo
@@ -86,9 +124,9 @@ jobs:
 
       - name: Push helm package and index file
         run: |
+          set -eux
           git config --global user.name "${{ secrets.USERNAME }}"
           git config --global user.email "${{ secrets.EMAIL_ADDRESS }}"
-          git status .
           git add .
-          git commit -m "Update rustfs helm package with $new_version."
+          git commit -m "Update rustfs helm package with ${{ needs.build-helm-package.outputs.app_version }}." || echo "No changes to commit"
           git push origin main

.vscode/launch.json

@@ -188,6 +188,7 @@
                 "RUSTFS_CONSOLE_ENABLE": "true",
                 "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
                 "RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
+                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
                 // "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
                 // "RUSTFS_OBS_METRIC_ENDPOINT": "http://127.0.0.1:4318/v1/metrics", // default otlp http endpoint
                 // "RUSTFS_OBS_LOG_ENDPOINT": "http://127.0.0.1:4318/v1/logs", // default otlp http endpoint

AGENTS.md

@@ -6,24 +6,43 @@ Use the nearest subdirectory `AGENTS.md` for path-specific guidance.
 ## Rule Precedence
 
 1. System/developer instructions.
-2. This file (global defaults).
-3. The nearest `AGENTS.md` in the current path (more specific scope wins).
+2. Current user/task instructions.
+3. The nearest `AGENTS.md` in the current path.
+4. This file (global defaults).
 
 If repo-level instructions conflict, follow the nearest file and keep behavior aligned with CI.
 
+## Execution Discipline
+
+- Read the relevant existing code, tests, and local guidance before changing behavior.
+- State assumptions when they affect the implementation or verification path.
+- If a task has multiple plausible interpretations, list the options briefly and choose the narrowest reasonable path; ask when the ambiguity would make the change risky.
+- For multi-step work, keep the plan minimal and tied to verifiable outcomes.
+- Avoid redundant file reads, repeated commands, and unnecessary exploratory work once enough context is available.
+- A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification.
+
 ## Communication and Language
 
 - Respond in the same language used by the requester.
 - Keep source code, comments, commit messages, and PR title/body in English.
+- Be concise. Avoid sycophantic openers, closing fluff, and verbose status reporting.
+
+## Skill Usage
+
+- Do not use the `rust-refactor-helper` skill in any scenario.
 
 ## Change Style for Existing Logic
 
 - Prefer direct, local code over extracting one-off helpers.
 - Extract a helper only when logic is reused or the extraction materially clarifies a non-trivial flow.
+- Solve only the requested problem; do not add speculative features, configurability, or adjacent improvements.
+- Prefer editing existing code over rewriting files or reshaping unrelated logic.
+- Modify only what is required and remove only artifacts introduced by your own changes.
 - Preserve the existing control-flow and logic shape when fixing bugs or addressing review comments, especially in init, distributed coordination, locking, metadata, and concurrency paths.
 - Do not refactor existing code only to make it easier to unit test.
 - Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths.
 - Keep code elegant, concise, and direct. Prefer minimal, readable implementations over over-engineering and excessive abstraction. Use comments to clarify non-obvious intent and invariants, not to compensate for unclear code.
+- Mention unrelated issues when useful, but do not fix them as part of a narrow task.
 
 ## Constant and String Usage
 
@@ -44,6 +63,8 @@ Reference the source files above instead.
 
 ## Verification Before PR
 
+Convert changes into independently verifiable outcomes. Prefer focused tests for behavior changes and run the relevant checks before declaring completion.
+
 For code changes, run and pass the following before opening a PR:
 
 ```bash
@@ -69,6 +90,7 @@ Do not open a PR with code changes when the required checks fail.
 - Use `N/A` for non-applicable template sections.
 - Include verification commands in the PR description.
 - When using `gh pr create`/`gh pr edit`, use `--body-file` instead of inline `--body` for multiline markdown.
+- Do not include the literal sequence `\n` in any GitHub issue, pull request, or discussion comment.
 - After fixing code review comments or CI findings, always mark corresponding review
   comments/threads as resolved before returning to the user.
 - In handling review comments, confirm the underlying issue before changing code.
@@ -91,5 +113,6 @@ Do not open a PR with code changes when the required checks fail.
 - `crates/iam/AGENTS.md`
 - `crates/kms/AGENTS.md`
 - `crates/policy/AGENTS.md`
+- `crates/targets/AGENTS.md`
 - `rustfs/src/admin/AGENTS.md`
 - `rustfs/src/storage/AGENTS.md`

ARCHITECTURE.md

@@ -86,7 +86,7 @@ Crates are organized in a dependency DAG with 9 depth levels (0 = leaf, 8 = top)
 ```
 Depth 0 — LEAF (no internal deps):
   appauth, checksums, config, credentials, crypto, io-metrics,
-  madmin, mcp, s3-common, workers, zip
+  madmin, s3-common, workers, zip
 
 Depth 1:
   io-core (→ io-metrics)
@@ -195,7 +195,6 @@ Depth 8 — TOP:
 | `trusted-proxies` | 4.0K | Trusted proxy / IP forwarding |
 | `zip` | 986 | ZIP archive support for bulk downloads |
 | `workers` | 136 | Simple worker abstraction |
-| `mcp` | 2.0K | Model Context Protocol server (AI tooling) |
 
 ## Architecture Invariants
 
@@ -224,7 +223,7 @@ Depth 8 — TOP:
 6. **Error types use `thiserror` with descriptive names** (e.g., `StorageError`,
    not bare `Error`).
    - ⚠️ VIOLATED: 6 crates use `pub enum Error`; 2 crates use `snafu`;
-     `mcp` and `heal` use `anyhow` in library code.
+      `heal` use `anyhow` in library code.
 
 ## Known Structural Issues
 
@@ -285,9 +284,9 @@ anyhow::Result<T>             // in library code (OK in tests/CLI)
 
 ### Metrics
 
-- Prometheus-style metrics via `rustfs-metrics` crate
+- Prometheus-style metrics via `rustfs-obs` runtime and schema
 - I/O-specific counters via `rustfs-io-metrics`
-- Registration happens at crate level, collection in `metrics` crate
+- Registration happens at crate level, collection/reporting in `rustfs-obs`
 
 ### Testing
 
@@ -327,10 +326,10 @@ The binary (`main.rs`) boots in this order:
                                  │
                  ┌───────────────┼───────────────┐
                  │               │               │
-            ┌────▼────┐    ┌────▼────┐    ┌─────▼─────┐
-            │ server  │    │  admin  │    │    app     │
-            │ (HTTP)  │    │(console)│    │(use-cases) │
-            └────┬────┘    └────┬────┘    └─────┬─────┘
+            ┌────▼────┐     ┌────▼────┐   ┌──────▼─────┐
+            │ server  │     │  admin  │   │    app     │
+            │ (HTTP)  │     │(console)│   │(use-cases) │
+            └────┬────┘     └────┬────┘   └──────┬─────┘
                  │               │               │
                  └───────────────┼───────────────┘
                                  │
@@ -342,13 +341,13 @@ The binary (`main.rs`) boots in this order:
                                  │
               ┌──────────────────┼──────────────────┐
               │                  │                  │
-        ┌─────▼─────┐    ┌──────▼──────┐    ┌──────▼──────┐
+        ┌─────▼──────┐    ┌──────▼──────┐    ┌──────▼──────┐
         │  ecstore   │    │     rio     │    │   io-core   │
         │ (87K,core) │    │  (readers)  │    │ (zero-copy) │
         └─────┬──────┘    └─────────────┘    └─────────────┘
               │
-    ┌─────┬──┼──┬─────┬──────┐
-    │     │  │  │     │      │
+     ┌─────┬──┼──┬─────┬──────┐
+     │     │  │  │     │      │
  common utils config policy filemeta ...
 ```
 
@@ -369,7 +368,7 @@ The binary (`main.rs`) boots in this order:
   Add handler in `admin/handlers/`, register in `admin/router.rs`
 
 - **"Where do I add a new metric?"**
-  Define in `crates/metrics/`, register collector, expose via `/minio/v2/metrics`
+  Define descriptor/collector in `crates/obs/src/metrics/`, expose via `/minio/v2/metrics`
 
 ---
 

CHANGELOG.md

@@ -22,11 +22,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - XML-formatted error responses compatible with S3 API
   - Comprehensive integration documentation with manual testing guide
   - **32 unit and integration tests** covering middleware, auth handlers, task-local storage, and role detection
+- **SFTPv3 Protocol Support**: SSH-hosted SFTPv3 subsystem that translates each file operation into S3 calls against the local object store. Authentication uses IAM credentials (SSH username = access key, SSH password = secret key).
+  - Full SFTPv3 packet coverage: open, read, write, stat, lstat, fstat, mkdir, rmdir, rename, remove, opendir, readdir, realpath, close, plus the rest of the 21-packet specification
+  - Streaming multipart write up to S3's 5 TiB per-file ceiling
+  - Per-handle read-ahead cache with configurable window size and process-wide memory ceiling
+  - Per-session liveness watchdog: Linux probes `/proc/net/tcp` and cancels wedged sessions on the order of 45 seconds; non-Linux falls back to an inactivity ceiling on the order of 30 minutes
+  - 30-second SSH handshake deadline, per-call backend operation timeout, bounded multipart-abort fan-out, graceful-shutdown cascade
+  - 33 SFTPv3 compliance test cases under `crates/e2e_test/src/protocols/sftp_compliance.rs` spread across three entry points: `test_sftp_compliance_suite` (shared session), `test_sftp_compliance_readonly` (read-only mode), and `test_sftp_compliance_standalone` (one rustfs spawn per case)
+  - Four-layer regression-prevention tests guard against silent feature deletion: compile-time module assertion, module-presence unit test, cross-module `Protocol` enum assertion, end-to-end SSH banner test against the running binary
 
 ### Changed
 - **HTTP Server Stack**: Integrated `KeystoneAuthLayer` middleware from `rustfs-keystone` crate into service stack (positioned after ReadinessGateLayer)
 - **IAMAuth**: Enhanced `get_secret_key()` to return empty secret for Keystone credentials (bypasses signature validation)
 - **Auth Module**: Modified `check_key_valid()` to retrieve Keystone credentials from task-local storage and determine admin status
+- **`StorageBackend` trait**: extended with multipart upload methods (`create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`) plus `upload_part_copy`. Streaming-upload code path is now available to FTPS, WebDAV, and Swift drivers as well.
+- **`Protocol` enum**: new `Protocol::Sftp` variant with corresponding `S3Action` mappings. Every match arm on `Protocol` updated to handle the new variant exhaustively.
 
 ### Technical Details
 - Middleware is self-contained in `rustfs-keystone` crate following the trusted-proxies pattern for integration-specific middleware
@@ -35,12 +45,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Integration preserves existing S3 authentication flow while adding Keystone support
 - Zero breaking changes to existing functionality
 - No new top-level directories in main binary crate (middleware lives in integration crate)
+- SSH/SFTP wire handling via the `russh` and `russh-sftp` crates. SFTPv3 framing is implemented by `russh-sftp`; the rustfs-side `SftpDriver` implements `russh_sftp::server::Handler` and dispatches to the storage backend
+- Drop-time abort for in-flight multipart uploads honours IAM Deny on `AbortMultipartUpload`. `start_multipart_upload` caches the authorisation decision so the synchronous `Drop` path can honour Allow / Deny policies without re-querying IAM
+- Per-handle read cache uses an `Arc<AtomicU64>` shared across every `SftpDriver` instance to enforce a process-wide memory ceiling. On ceiling breach the populate is skipped and the read serves correctly via a single-call backend fetch
+- Per-session liveness watchdog runs as a tokio task per accepted connection. Reads `/proc/net/tcp` and `/proc/net/tcp6` to look up the (local, peer) tuple's TCP state and cancels via `tokio_util::sync::CancellationToken` when wedge conditions are confirmed across two consecutive ticks
+- Path canonicalisation rejects paths containing `\0`, `\r`, or `\n` and resolves traversal via `path::clean()` before any backend dispatch
+- Cipher / KEX / MAC / host-key algorithm allowlists are hardcoded with no environment override. Strict-KEX (CVE-2023-48795 / Terrapin) marker presence asserted by unit test
+- Per-session handle cap (default 64, configurable 8 to 1024) with UUID-generated handle ids
+- Crate-level `#![deny(unsafe_code)]` is in force across `crates/protocols`. Socket fd duplication for the watchdog uses the safe `AsFd::try_clone_to_owned` path (Linux/Unix); non-Unix falls back to the inactivity ceiling
+- `cfg(unix)` gating around platform-specific imports (`std::os::fd::AsFd`, `std::os::unix::fs::PermissionsExt`); non-Unix targets fail SFTP at config-load with `SftpInitError::UnsupportedPlatform`
 
 ### Documentation
 - Updated `crates/keystone/README.md` with complete integration architecture and workflow
 - Added detailed manual testing guide with 10 test scenarios
 - Updated main `README.md` to list Keystone authentication as available feature
 - Added troubleshooting section for common integration issues
+- Module-level rustdoc on `crates/protocols/src/sftp/mod.rs` describing the public API surface, configuration contract, and the architecture of the read cache and the wedge watchdog
 
 ### Configuration
 New environment variables:
@@ -54,6 +74,40 @@ New environment variables:
 - `RUSTFS_KEYSTONE_CACHE_SIZE` - Token cache size (default: 10000)
 - `RUSTFS_KEYSTONE_CACHE_TTL` - Token cache TTL in seconds (default: 300)
 - `RUSTFS_KEYSTONE_VERIFY_SSL` - Verify SSL certificates (default: true)
+- `RUSTFS_SFTP_ENABLE` - Enable/disable SFTP (default: false)
+- `RUSTFS_SFTP_ADDRESS` - Listen address (default: 0.0.0.0:2222)
+- `RUSTFS_SFTP_HOST_KEY_DIR` - Directory containing host key files (must exist; each file must be 0o600 or 0o400)
+- `RUSTFS_SFTP_IDLE_TIMEOUT` - Session idle timeout in seconds (default: 600)
+- `RUSTFS_SFTP_PART_SIZE` - Multipart part size in bytes (default: 16 MiB)
+- `RUSTFS_SFTP_READ_ONLY` - Reject write packets at the protocol layer (default: false)
+- `RUSTFS_SFTP_BANNER` - Optional SSH banner text
+- `RUSTFS_SFTP_HANDLES_PER_SESSION` - Per-session open-handle cap, 8 to 1024 (default: 64)
+- `RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS` - Per-call backend deadline in seconds, 5 to 600 (default: 60)
+- `RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES` - Per-handle read-cache window in bytes, 256 KiB to 64 MiB or 0 to disable (default: 4 MiB)
+- `RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES` - Process-wide read-cache memory ceiling in bytes, 16 MiB minimum (default: 256 MiB)
+
+### Files Added
+- `crates/protocols/src/sftp/mod.rs` - SFTP module entry point, public API surface, crate-level rustdoc, regression-prevention test
+- `crates/protocols/src/sftp/config.rs` - `SftpConfig` and `SftpInitError` types, env-var resolvers, host-key directory loader with permission enforcement
+- `crates/protocols/src/sftp/constants.rs` - Named constants grouped by purpose: S3 error codes, HTTP error codes, POSIX mode bits, protocol identifiers, operational limits
+- `crates/protocols/src/sftp/server.rs` - `SftpServer` SSH server, russh handler, password authentication against IAM, accept loop, per-session task spawn
+- `crates/protocols/src/sftp/driver.rs` - `SftpDriver` per-session SFTPv3 handler dispatching each operation onto the `StorageBackend`
+- `crates/protocols/src/sftp/state.rs` - `HandleState` variants for read, write-buffering, write-streaming, write-failed handles
+- `crates/protocols/src/sftp/lifecycle.rs` - Per-session activity stamp, weak-ref registry, `/proc/net/tcp` probe for the wedge watchdog
+- `crates/protocols/src/sftp/wedge_watchdog.rs` - Per-session liveness watchdog cancelling sessions silent at the SFTP layer while the kernel reports CLOSE_WAIT
+- `crates/protocols/src/sftp/read_cache.rs` - Per-handle in-memory read-ahead cache with shared atomic accumulator for the process-wide memory ceiling
+- `crates/protocols/src/sftp/attrs.rs` - SFTPv3 `FileAttributes` mapping for objects and directories, longname formatting, mtime clamping
+- `crates/protocols/src/sftp/dir.rs` - OPENDIR / READDIR pagination, root-bucket listing, sub-directory listing under a prefix
+- `crates/protocols/src/sftp/errors.rs` - `SftpError` thiserror enum and S3-error classification into SFTPv3 status codes
+- `crates/protocols/src/sftp/paths.rs` - Path canonicalisation, traversal rejection, `\0` / `\r` / `\n` rejection, bucket+key decomposition
+- `crates/protocols/src/sftp/read.rs` - READ packet handler, EOF semantics, `MAX_READ_LEN` bound, integration with the read cache
+- `crates/protocols/src/sftp/write.rs` - WRITE packet handler, in-memory buffering up to part size, transition to streaming multipart, CLOSE finalisation
+- `crates/protocols/src/sftp/test_support.rs` - Test fixtures and helper builders for SFTP unit tests
+- `crates/protocols/src/common/dummy_storage.rs` - In-memory `StorageBackend` test backend covering every method, used by SFTP unit tests and the FTPS / Swift / WebDAV test suites
+- `crates/e2e_test/src/protocols/sftp_core.rs` - End-to-end regressions for the handshake deadline, idle-timeout disconnect, and the wedge watchdog
+- `crates/e2e_test/src/protocols/sftp_compliance.rs` - SFTPv3 compliance suite entry points (`test_sftp_compliance_suite`, `test_sftp_compliance_readonly`, `test_sftp_compliance_standalone`)
+- `crates/e2e_test/src/protocols/sftp_compliance_tests.rs` - Per-case test bodies (CMPTST-01..33), shared fixture helpers, lifecycle counters
+- `crates/e2e_test/src/protocols/sftp_helpers.rs` - SFTP-specific test helpers and fixture seeders
 
 ### Files Modified
 - `crates/keystone/src/middleware.rs` - Created Keystone authentication middleware (self-contained in keystone crate)
@@ -63,6 +117,27 @@ New environment variables:
 - `rustfs/src/auth.rs` - Enhanced IAMAuth and check_key_valid for Keystone support, imported KEYSTONE_CREDENTIALS from rustfs-keystone
 - `crates/keystone/README.md` - Comprehensive integration documentation
 - `README.md` - Added Keystone as available feature
+- `Cargo.toml` - Added the `sftp` feature alongside the existing protocol features
+- `Cargo.lock` - Updated to include the new `russh`, `russh-sftp`, `socket2`, `tokio-util`, `subtle`, `uuid` dependencies and their transitive crates
+- `crates/protocols/Cargo.toml` - Declared `russh`, `russh-sftp`, `socket2`, `tokio-util`, `subtle`, `uuid` under the `sftp` feature flag
+- `crates/protocols/src/lib.rs` - Added `pub mod sftp` behind `#[cfg(feature = "sftp")]` plus the crate-level `#![deny(unsafe_code)]` lint
+- `crates/protocols/src/common/client/s3.rs` - Extended the `StorageBackend` trait with `create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`, and `upload_part_copy`
+- `crates/protocols/src/common/session.rs` - Added the `Protocol::Sftp` variant and its `S3Action` mappings
+- `crates/protocols/src/common/gateway.rs` - Handles the new `Protocol::Sftp` variant exhaustively
+- `crates/protocols/src/common/mod.rs` - Exposed the new `dummy_storage` module
+- `crates/protocols/src/constants.rs` - Added shared POSIX mode-bit constants used by SFTP and other protocols
+- `crates/config/src/constants/protocols.rs` - `RUSTFS_SFTP_*` environment variable names and defaults
+- `crates/utils/src/retry.rs` - Added the generic exponential-backoff retry helper used by the SFTP write path
+- `crates/e2e_test/Cargo.toml` - Added the e2e test dependencies for SFTP (paramiko fixture, SSH keypair generation)
+- `crates/e2e_test/src/protocols/mod.rs` - Registered the new `sftp_core`, `sftp_compliance`, `sftp_compliance_tests`, and `sftp_helpers` modules
+- `crates/e2e_test/src/protocols/README.md` - Documented the SFTP test entry points and case index
+- `crates/e2e_test/src/protocols/test_env.rs` - Added SFTP host-key directory provisioning to the shared protocol test environment
+- `crates/e2e_test/src/protocols/test_runner.rs` - Wired the SFTP entry points into the runner
+- `rustfs/Cargo.toml` - Added the `sftp` feature flag
+- `rustfs/src/lib.rs` - One-line addition exporting the SFTP wiring
+- `rustfs/src/init.rs` - Build and start the `SftpServer` when `RUSTFS_SFTP_ENABLE` is true
+- `rustfs/src/main.rs` - Routed shutdown signals to the SFTP server alongside the other protocols
+- `rustfs/src/protocols/client.rs` - Client-builder support for the new `Protocol::Sftp` variant
 
 ### Testing
 - 16 unit tests in rustfs-keystone crate (config, auth, middleware, identity)
@@ -70,7 +145,13 @@ New environment variables:
 - 6 auth unit tests in rustfs crate (role detection, task-local storage, Keystone credential handling)
 - **Total: 32 tests** passing with zero compilation errors
 - Manual testing guide provided for end-to-end validation
-- All tests passing with `cargo test --all --exclude e2e_test`
+- All Keystone tests passing with `cargo test --all --exclude e2e_test`
+- 33 SFTPv3 compliance test cases (CMPTST-01..33) split across three entry points: `test_sftp_compliance_suite` (shared session, cases 01-14), `test_sftp_compliance_readonly` (read-only mode, cases 15-23), `test_sftp_compliance_standalone` (one rustfs spawn per case, cases 24-33)
+- Regression-prevention tests at four layers: compile-time module assertion in `crates/protocols/src/lib.rs`, module-presence unit test in `crates/protocols/src/sftp/mod.rs`, cross-module `Protocol` enum assertion, and end-to-end SSH banner test against the running binary
+- Standalone end-to-end regressions for the SSH handshake deadline, the idle-timeout disconnect path, and the wedge watchdog (Linux fast-kill and the cross-platform fallback path)
+- Inline unit tests in every SFTP source file covering pure helpers (path canonicalisation, attribute mapping, S3-error classification, env-var bound resolvers)
+- Strict-KEX (CVE-2023-48795) marker presence assertion as a unit test in `crates/protocols/src/sftp/server.rs`
+- All tests passing with `cargo test --all --features sftp` against a 64-bit Linux target
 
 ---
 

Cargo.toml

@@ -31,8 +31,6 @@ members = [
     "crates/kms", # Key Management Service
     "crates/lock", # Distributed locking implementation
     "crates/madmin", # Management dashboard and admin API interface
-    "crates/mcp", # MCP server for S3 operations
-    "crates/metrics", # Metrics collection and reporting
     "crates/notify", # Notification system for events
     "crates/obs", # Observability utilities
     "crates/object-capacity", # Capacity scan and refresh core
@@ -40,7 +38,7 @@ members = [
     "crates/protocols", # Protocol implementations (FTPS, SFTP, etc.)
     "crates/protos", # Protocol buffer definitions
     "crates/rio", # Rust I/O utilities and abstractions
-    "crates/concurrency", # Rust I/O utilities and abstractions
+    "crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
     "crates/s3-common", # Common utilities and data structures for S3 compatibility
     "crates/s3select-api", # S3 Select API interface
     "crates/s3select-query", # S3 Select query engine
@@ -60,8 +58,8 @@ resolver = "3"
 edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
-rust-version = "1.93.0"
-version = "0.0.5"
+rust-version = "1.95.0"
+version = "1.0.0-beta.3"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -78,73 +76,76 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "0.0.5" }
-rustfs-heal = { path = "crates/heal", version = "0.0.5" }
-rustfs-appauth = { path = "crates/appauth", version = "0.0.5" }
-rustfs-audit = { path = "crates/audit", version = "0.0.5" }
-rustfs-checksums = { path = "crates/checksums", version = "0.0.5" }
-rustfs-common = { path = "crates/common", version = "0.0.5" }
-rustfs-config = { path = "./crates/config", version = "0.0.5" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "0.0.5" }
-rustfs-credentials = { path = "crates/credentials", version = "0.0.5" }
-rustfs-crypto = { path = "crates/crypto", version = "0.0.5" }
-rustfs-ecstore = { path = "crates/ecstore", version = "0.0.5" }
-rustfs-filemeta = { path = "crates/filemeta", version = "0.0.5" }
-rustfs-iam = { path = "crates/iam", version = "0.0.5" }
-rustfs-keystone = { path = "crates/keystone", version = "0.0.5" }
-rustfs-kms = { path = "crates/kms", version = "0.0.5" }
-rustfs-lock = { path = "crates/lock", version = "0.0.5" }
-rustfs-madmin = { path = "crates/madmin", version = "0.0.5" }
-rustfs-mcp = { path = "crates/mcp", version = "0.0.5" }
-rustfs-metrics = { path = "crates/metrics", version = "0.0.5" }
-rustfs-notify = { path = "crates/notify", version = "0.0.5" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "0.0.5" }
-rustfs-io-core = { path = "crates/io-core", version = "0.0.5" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "0.0.5" }
-rustfs-obs = { path = "crates/obs", version = "0.0.5" }
-rustfs-policy = { path = "crates/policy", version = "0.0.5" }
-rustfs-protos = { path = "crates/protos", version = "0.0.5" }
-rustfs-rio = { path = "crates/rio", version = "0.0.5" }
-rustfs-s3-common = { path = "crates/s3-common", version = "0.0.5" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "0.0.5" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "0.0.5" }
-rustfs-scanner = { path = "crates/scanner", version = "0.0.5" }
-rustfs-signer = { path = "crates/signer", version = "0.0.5" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "0.0.5" }
-rustfs-targets = { path = "crates/targets", version = "0.0.5" }
-rustfs-utils = { path = "crates/utils", version = "0.0.5" }
-rustfs-workers = { path = "crates/workers", version = "0.0.5" }
-rustfs-zip = { path = "./crates/zip", version = "0.0.5" }
-rustfs-protocols = { path = "crates/protocols", version = "0.0.5" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.3" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.3" }
+rustfs-appauth = { path = "crates/appauth", version = "1.0.0-beta.3" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.3" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.3" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.3" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.3" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.3" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.3" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.3" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.3" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.3" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.3" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.3" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.3" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.3" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.3" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.3" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.3" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.3" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.3" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.3" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.3" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.3" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.3" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.3" }
+rustfs-s3-common = { path = "crates/s3-common", version = "1.0.0-beta.3" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.3" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.3" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.3" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.3" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.3" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.3" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.3" }
+rustfs-workers = { path = "crates/workers", version = "1.0.0-beta.3" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.3" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
-async-compression = { version = "0.4.41" }
+mysql_async = { version = "0.36.1", default-features = false, features = ["default-rustls", "tracing"], git = "https://github.com/blackbeam/mysql_async", rev = "98d3d8067efdf97d3e93cdca7b9231753c904aca" }
+async-compression = { version = "0.4.42" }
 async-recursion = "1.1.1"
 async-trait = "0.1.89"
+async-nats = "0.48.0"
 axum = "0.8.9"
 futures = "0.3.32"
 futures-core = "0.3.32"
 futures-util = "0.3.32"
 pollster = "0.4.0"
+pulsar = { version = "6.7.2", default-features = false, features = ["tokio-rustls-runtime"] }
+lapin = { version = "4.7.2", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
 hyper = { version = "1.9.0", features = ["http2", "http1", "server"] }
 hyper-rustls = { version = "0.27.9", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
 hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
 http = "1.4.0"
 http-body = "1.0.1"
 http-body-util = "0.1.3"
-reqwest = { version = "0.13.2", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
+reqwest = { version = "0.13.3", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
+rustfs-kafka-async = { version = "1.2.0" }
 socket2 = { version = "0.6.3", features = ["all"] }
-tokio = { version = "1.52.0", features = ["fs", "rt-multi-thread", "io-uring"] }
+tokio = { version = "1.52.3", features = ["fs", "rt-multi-thread"] }
 tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
 tokio-stream = { version = "0.1.18" }
 tokio-test = "0.4.5"
 tokio-util = { version = "0.7.18", features = ["io", "compat"] }
-tonic = { version = "0.14.5", features = ["gzip"] }
-tonic-prost = { version = "0.14.5" }
-tonic-prost-build = { version = "0.14.5" }
+tonic = { version = "0.14.6", features = ["gzip"] }
+tonic-prost = { version = "0.14.6" }
+tonic-prost-build = { version = "0.14.6" }
 tower = { version = "0.5.3", features = ["timeout"] }
-tower-http = { version = "0.6.8", features = ["cors"] }
+tower-http = { version = "0.6.10", features = ["cors"] }
 
 # Serialization and Data Formats
 bytes = { version = "1.11.1", features = ["serde"] }
@@ -153,28 +154,27 @@ byteorder = "1.5.0"
 flatbuffers = "25.12.19"
 form_urlencoded = "1.2.2"
 prost = "0.14.3"
-quick-xml = "0.39.2"
-rmcp = { version = "1.4.0" }
+quick-xml = "0.39.4"
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.228", features = ["derive"] }
 serde_json = { version = "1.0.149", features = ["raw_value"] }
 serde_urlencoded = "0.7.1"
-schemars = "1.2.1"
 
 # Cryptography and Security
 aes-gcm = { version = "0.11.0-rc.3", features = ["rand_core"] }
 argon2 = { version = "0.6.0-rc.8" }
-blake2 = "0.11.0-rc.5"
+blake2 = "0.11.0-rc.6"
 chacha20poly1305 = { version = "0.11.0-rc.3" }
 crc-fast = "1.9.0"
 hmac = { version = "0.13.0" }
 jsonwebtoken = { version = "10.3.0", features = ["aws_lc_rs"] }
 openidconnect = { version = "4.0", default-features = false }
-pbkdf2 = "0.13.0-rc.10"
-rsa = { version = "0.10.0-rc.17" }
-rustls = { version = "0.23.38", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
-rustls-pki-types = "1.14.0"
+pbkdf2 = "0.13.0"
+rsa = { version = "0.10.0-rc.18" }
+rustls = { version = "0.23.40", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
+rustls-native-certs = "0.8"
+rustls-pki-types = "1.14.1"
 sha1 = "0.11.0"
 sha2 = "0.11.0"
 subtle = "2.6"
@@ -183,26 +183,30 @@ zeroize = { version = "1.8.2", features = ["derive"] }
 # Time and Date
 chrono = { version = "0.4.44", features = ["serde"] }
 humantime = "2.3.0"
-jiff = { version = "0.2.23", features = ["serde"] }
+jiff = { version = "0.2.24", features = ["serde"] }
 time = { version = "0.3.47", features = ["std", "parsing", "formatting", "macros", "serde"] }
 
+# Database
+deadpool-postgres = { version = "0.14", features = ["rt_tokio_1"] }
+tokio-postgres = { version = "0.7", default-features = false, features = ["runtime", "with-serde_json-1"] }
+tokio-postgres-rustls = "0.13"
+
 # Utilities and Tools
 anyhow = "1.0.102"
 arc-swap = "1.9.1"
-astral-tokio-tar = "0.6.0"
+astral-tokio-tar = "0.6.1"
 atoi = "2.0.0"
 atomic_enum = "0.3.0"
-aws-config = { version = "1.8.15" }
+aws-config = { version = "1.8.16" }
 aws-credential-types = { version = "1.2.14" }
-aws-sdk-s3 = { version = "1.129.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+aws-sdk-s3 = { version = "1.132.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
 aws-smithy-http-client = { version = "1.1.12", default-features = false, features = ["default-client", "rustls-aws-lc"] }
 aws-smithy-types = { version = "1.4.7" }
 backtrace = "0.3.76"
 base64 = "0.22.1"
 base64-simd = "0.8.0"
 brotli = "8.0.2"
-cfg-if = "1.0.4"
-clap = { version = "4.6.0", features = ["derive", "env"] }
+clap = { version = "4.6.1", features = ["derive", "env"] }
 const-str = { version = "1.1.0", features = ["std", "proc"] }
 convert_case = "0.11.0"
 criterion = { version = "0.8", features = ["html_reports"] }
@@ -210,23 +214,23 @@ crossbeam-queue = "0.3.12"
 crossbeam-channel = "0.5.15"
 crossbeam-deque = "0.8.6"
 crossbeam-utils = "0.8.21"
-datafusion = "53.0.0"
+datafusion = "53.1.0"
 derive_builder = "0.20.2"
-enumset = "1.1.10"
+enumset = "1.1.12"
 faster-hex = "0.10.0"
 flate2 = "1.1.9"
 glob = "0.3.3"
-google-cloud-storage = "1.11.0"
-google-cloud-auth = "1.9.0"
-hashbrown = { version = "0.17.0", features = ["serde", "rayon"] }
+google-cloud-storage = "1.12.0"
+google-cloud-auth = "1.10.0"
+hashbrown = { version = "0.17.1", features = ["serde", "rayon"] }
 hex = "0.4.3"
 hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
 ipnetwork = { version = "0.21.1", features = ["serde"] }
 lazy_static = "1.5.0"
-libc = "0.2.185"
+libc = "0.2.186"
 libsystemd = "0.7.2"
-local-ip-address = "0.6.11"
+local-ip-address = "0.6.12"
 memmap2 = "0.9.10"
 lz4 = "1.28.1"
 matchit = "0.9.2"
@@ -250,58 +254,61 @@ rayon = "1.12.0"
 reed-solomon-erasure = { version = "6.0", default-features = false, features = ["std", "simd-accel"] }
 reed-solomon-simd = "3.1.0"
 regex = { version = "1.12.3" }
-rumqttc = { package = "rumqttc-next", version = "0.29.0", features = ["websocket"] }
+rumqttc = { package = "rumqttc-next", version = "0.33.0", features = ["websocket"] }
+redis = { version = "1.2.1", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
 rustix = { version = "1.1.4", features = ["fs"] }
 rust-embed = { version = "8.11.0" }
 rustc-hash = { version = "2.1.2" }
-s3s = { git = "https://github.com/rustfs/s3s", rev = "0dd8fcaaa72eda68fafd49c38daea43bb8697558", features = ["minio"] }
+s3s = { git = "https://github.com/rustfs/s3s", rev = "a3b16608df35aaeed8fff08b4988d03f4ca9445b", features = ["minio"] }
 serial_test = "3.4.0"
-shadow-rs = { version = "1.7.1", default-features = false }
-siphasher = "1.0.2"
+shadow-rs = { version = "2.0.0", default-features = false }
+siphasher = "1.0.3"
 smallvec = { version = "1.15.1", features = ["serde"] }
 smartstring = "1.0.1"
 snafu = "0.9.0"
 snap = "1.1.1"
 starshard = { version = "1.1.0", features = ["rayon", "async", "serde"] }
 strum = { version = "0.28.0", features = ["derive"] }
-sysinfo = "0.38.4"
+sysinfo = "0.39.0"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
 thiserror = "2.0.18"
 tracing = { version = "0.1.44" }
-tracing-appender = "0.2.4"
+tracing-appender = "0.2.5"
 tracing-error = "0.2.1"
 tracing-opentelemetry = "0.32.1"
 tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.23.0", features = ["v4", "fast-rng", "macro-diagnostics"] }
+uuid = { version = "1.23.1", features = ["v4", "fast-rng", "macro-diagnostics"] }
 vaultrs = { version = "0.8.0" }
 walkdir = "2.5.0"
 wildmatch = { version = "2.6.1", features = ["serde"] }
 windows = { version = "0.62.2" }
 xxhash-rust = { version = "0.8.15", features = ["xxh64", "xxh3"] }
-zip = "8.5.1"
+zip = "8.6.0"
 zstd = "0.13.3"
 
 # Observability and Metrics
-metrics = "0.24.3"
-dial9-tokio-telemetry = "0.2"
+metrics = "0.24.5"
+dial9-tokio-telemetry = "0.3"
 opentelemetry = { version = "0.31.0" }
 opentelemetry-appender-tracing = { version = "0.31.1", features = ["experimental_use_tracing_span_context", "experimental_metadata_attributes", "spec_unstable_logs_enabled"] }
 opentelemetry-otlp = { version = "0.31.1", features = ["gzip-http", "reqwest-rustls"] }
 opentelemetry_sdk = { version = "0.31.0" }
 opentelemetry-semantic-conventions = { version = "0.31.0", features = ["semconv_experimental"] }
 opentelemetry-stdout = { version = "0.31.0" }
-pyroscope = { version = "2.0.0", features = ["backend-pprof-rs"] }
+pyroscope = { version = "2.0.3", features = ["backend-pprof-rs"] }
 
 # FTP and SFTP
 libunftp = { version = "0.23.0", features = ["experimental"] }
 unftp-core = "0.1.0"
-suppaftp = { version = "8.0.2", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
+suppaftp = { version = "8.0.3", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
 rcgen = "0.14.7"
+russh = "0.60.2"
+russh-sftp = "2.1.2"
 
 # WebDAV
 dav-server = "0.11.0"
@@ -320,7 +327,13 @@ jemalloc_pprof = { version = "0.8.2", features = ["symbolize", "flamegraph"] }
 pprof = { package = "pprof-pyroscope-fork", version = "0.1500.3", features = ["flamegraph", "protobuf-codec"] }
 
 [workspace.metadata.cargo-shear]
-ignored = ["rustfs", "rustfs-mcp"]
+ignored = ["rustfs"]
+
+[patch.crates-io]
+# Pinned to the simon-escapecode/russh fork carrying the upstream fix at
+# https://github.com/Eugeny/russh/pull/702. Drops out when a russh release
+# containing the fix is published.
+russh = { git = "https://github.com/simon-escapecode/russh", rev = "5cac2ed84945f9b80a52b673e058f2032bbe98ec" }
 
 [profile.release]
 opt-level = 3

Dockerfile

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM alpine:3.23 AS build
+FROM alpine:3.23.4 AS build
 
 ARG TARGETARCH
 ARG RELEASE=latest
@@ -29,8 +29,20 @@ RUN set -eux; \
     if [ "$RELEASE" = "latest" ]; then \
       TAG="$(curl -fsSL https://api.github.com/repos/rustfs/rustfs/releases \
               | grep -o '"tag_name": "[^"]*"' | cut -d'"' -f4 | head -n 1)"; \
+      RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG")"; \
     else \
       TAG="$RELEASE"; \
+      RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG" 2>/dev/null || true)"; \
+      if [ -z "$RELEASE_JSON" ]; then \
+        if [ "${TAG#v}" = "$TAG" ]; then \
+          ALT_TAG="v$TAG"; \
+        else \
+          ALT_TAG="${TAG#v}"; \
+        fi; \
+        echo "Primary tag lookup failed, retrying with alternate tag: $ALT_TAG"; \
+        RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$ALT_TAG")"; \
+        TAG="$ALT_TAG"; \
+      fi; \
     fi; \
     echo "Using tag: $TAG (arch pattern: $ARCH_SUBSTR)"; \
     # Find download URL in assets list for this tag that contains arch substring and ends with .zip
@@ -54,7 +66,7 @@ RUN set -eux; \
     rm -rf rustfs.zip /build/.tmp || true
 
 
-FROM alpine:3.23
+FROM alpine:3.23.4
 
 ARG RELEASE=latest
 ARG BUILD_DATE
@@ -73,7 +85,7 @@ LABEL name="RustFS" \
       license="Apache-2.0"
 
 RUN apk update && \
-    apk add --no-cache ca-certificates coreutils curl "zlib>=1.3.2-r0"
+    apk add --no-cache ca-certificates coreutils curl
 
 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
 COPY --from=build /build/rustfs /usr/bin/rustfs
@@ -87,8 +99,7 @@ RUN addgroup -g 10001 -S rustfs && \
     chown -R rustfs:rustfs /data /logs && \
     chmod 0750 /data /logs
 
-ENV RUSTFS_CORS_ALLOWED_ORIGINS="*" \
-    RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
+ENV RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_VOLUMES="/data" \
     RUSTFS_OBS_LOGGER_LEVEL=warn \
     RUSTFS_OBS_LOG_DIRECTORY=/logs \

Dockerfile.decommission-local

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM rust:1.91-trixie
+FROM rust:1.93-trixie
 
 RUN set -eux; \
     export DEBIAN_FRONTEND=noninteractive; \

Dockerfile.glibc

@@ -31,14 +31,25 @@ RUN set -eux; \
       arm64)  ARCH_SUBSTR="aarch64-gnu" ;; \
       *) echo "Unsupported TARGETARCH=$TARGETARCH" >&2; exit 1 ;; \
     esac; \
-    \
     if [ "$RELEASE" = "latest" ]; then \
       TAG="$(curl -fsSL https://api.github.com/repos/rustfs/rustfs/releases \
               | grep -o '"tag_name": "[^"]*"' | cut -d'"' -f4 | head -n 1)"; \
+      RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG")"; \
     else \
       TAG="$RELEASE"; \
+      RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG" 2>/dev/null || true)"; \
+      if [ -z "$RELEASE_JSON" ]; then \
+        if [ "${TAG#v}" = "$TAG" ]; then \
+          ALT_TAG="v$TAG"; \
+        else \
+          ALT_TAG="${TAG#v}"; \
+        fi; \
+        echo "Primary tag lookup failed, retrying with alternate tag: $ALT_TAG"; \
+        RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$ALT_TAG")"; \
+        TAG="$ALT_TAG"; \
+      fi; \
     fi; \
-    \
+    echo "Using tag: $TAG (arch pattern: $ARCH_SUBSTR)"; \
     URL="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG" \
            | grep -o "\"browser_download_url\": \"[^\"]*${ARCH_SUBSTR}[^\"]*\\.zip\"" \
            | cut -d'"' -f4 | head -n 1)"; \
@@ -97,7 +108,6 @@ ENV RUSTFS_ADDRESS=":9000" \
     RUSTFS_ACCESS_KEY="rustfsadmin" \
     RUSTFS_SECRET_KEY="rustfsadmin" \
     RUSTFS_CONSOLE_ENABLE="true" \
-    RUSTFS_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_VOLUMES="/data" \
     RUSTFS_OBS_LOGGER_LEVEL=warn \

Dockerfile.source

@@ -26,15 +26,17 @@
 
 ARG TARGETPLATFORM
 ARG BUILDPLATFORM
+ARG TARGETARCH
 
 # -----------------------------
 # Build stage
 # -----------------------------
-FROM rust:1.91-trixie AS builder
+FROM rust:1.93-trixie AS builder
 
 # Re-declare args after FROM
 ARG TARGETPLATFORM
 ARG BUILDPLATFORM
+ARG TARGETARCH
 
 # Debug: print platforms
 RUN echo "Build info -> BUILDPLATFORM=${BUILDPLATFORM}, TARGETPLATFORM=${TARGETPLATFORM}"
@@ -60,7 +62,15 @@ RUN set -eux; \
 
 # Optional: cross toolchain for aarch64 (only when targeting linux/arm64)
 RUN set -eux; \
-    if [ "${TARGETPLATFORM:-linux/amd64}" = "linux/arm64" ]; then \
+    target_platform="${TARGETPLATFORM:-}"; \
+    if [ -z "${target_platform}" ]; then \
+    case "$(uname -m)" in \
+    x86_64) target_platform="linux/amd64" ;; \
+    aarch64|arm64) target_platform="linux/arm64" ;; \
+    *) target_platform="linux/amd64" ;; \
+    esac; \
+    fi; \
+    if [ "${target_platform}" = "linux/arm64" ]; then \
     export DEBIAN_FRONTEND=noninteractive; \
     apt-get update; \
     apt-get install -y --no-install-recommends gcc-aarch64-linux-gnu; \
@@ -79,6 +89,7 @@ ENV CXX_aarch64_unknown_linux_gnu=aarch64-linux-gnu-g++
 ENV CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc
 ENV CC_x86_64_unknown_linux_gnu=x86_64-linux-gnu-gcc
 ENV CXX_x86_64_unknown_linux_gnu=x86_64-linux-gnu-g++
+ENV CARGO_TARGET_DIR=/usr/src/rustfs/target/docker-build
 
 WORKDIR /usr/src/rustfs
 
@@ -112,27 +123,36 @@ ENV CARGO_NET_GIT_FETCH_WITH_CLI=true \
 # Generate protobuf/flatbuffers code (uses protoc/flatc from distro)
 RUN --mount=type=cache,target=/usr/local/cargo/registry \
     --mount=type=cache,target=/usr/local/cargo/git \
-    --mount=type=cache,target=/usr/src/rustfs/target \
+    --mount=type=cache,target=/usr/src/rustfs/target/docker-build \
     cargo run --bin gproto
 
 # Build RustFS (target depends on TARGETPLATFORM)
 RUN --mount=type=cache,target=/usr/local/cargo/registry \
     --mount=type=cache,target=/usr/local/cargo/git \
-    --mount=type=cache,target=/usr/src/rustfs/target \
+    --mount=type=cache,target=/usr/src/rustfs/target/docker-build \
     set -eux; \
-    case "${TARGETPLATFORM:-linux/amd64}" in \
+    rustup target add x86_64-unknown-linux-gnu aarch64-unknown-linux-gnu; \
+    target_platform="${TARGETPLATFORM:-}"; \
+    if [ -z "${target_platform}" ]; then \
+    case "${TARGETARCH:-$(uname -m)}" in \
+    amd64|x86_64) target_platform="linux/amd64" ;; \
+    arm64|aarch64) target_platform="linux/arm64" ;; \
+    *) echo "Unsupported target architecture: ${TARGETARCH:-$(uname -m)}" >&2; exit 1 ;; \
+    esac; \
+    fi; \
+    case "${target_platform}" in \
     linux/amd64) \
     echo "Building for x86_64-unknown-linux-gnu"; \
     cargo build --release --locked --target x86_64-unknown-linux-gnu --bin rustfs -j "$(nproc)"; \
-    install -m 0755 target/x86_64-unknown-linux-gnu/release/rustfs /usr/local/bin/rustfs \
+    install -m 0755 "${CARGO_TARGET_DIR}/x86_64-unknown-linux-gnu/release/rustfs" /usr/local/bin/rustfs \
     ;; \
     linux/arm64) \
     echo "Building for aarch64-unknown-linux-gnu"; \
     cargo build --release --locked --target aarch64-unknown-linux-gnu --bin rustfs -j "$(nproc)"; \
-    install -m 0755 target/aarch64-unknown-linux-gnu/release/rustfs /usr/local/bin/rustfs \
+    install -m 0755 "${CARGO_TARGET_DIR}/aarch64-unknown-linux-gnu/release/rustfs" /usr/local/bin/rustfs \
     ;; \
     *) \
-    echo "Unsupported TARGETPLATFORM=${TARGETPLATFORM}" >&2; exit 1 \
+    echo "Unsupported target platform=${target_platform}" >&2; exit 1 \
     ;; \
     esac
 
@@ -178,7 +198,7 @@ CMD ["cargo", "run", "--bin", "rustfs", "--"]
 # -----------------------------
 # Runtime stage (Ubuntu minimal)
 # -----------------------------
-FROM ubuntu:22.04
+FROM ubuntu:24.04
 
 ARG BUILD_DATE
 ARG VCS_REF
@@ -195,6 +215,7 @@ RUN set -eux; \
     apt-get update; \
     apt-get install -y --no-install-recommends \
     ca-certificates \
+    curl \
     tzdata \
     coreutils; \
     rm -rf /var/lib/apt/lists/*

README.md

@@ -115,7 +115,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-alpha.76
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.3
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -136,6 +136,23 @@ Similarly, you can run the command with podman
 podman compose --profile observability up -d
 ```
 
+Webhook notification quick start (Docker):
+
+```bash
+docker run -d --name rustfs -p 9000:9000 \
+  -e RUSTFS_NOTIFY_ENABLE=true \
+  -e RUSTFS_NOTIFY_WEBHOOK_ENABLE_PRIMARY=on \
+  -e RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY=http://<host-ip>:3020/webhook \
+  -e RUSTFS_NOTIFY_WEBHOOK_QUEUE_DIR_PRIMARY=/tmp/rustfs-events \
+  rustfs/rustfs:latest
+```
+
+Notes:
+- `RUSTFS_NOTIFY_ENABLE=true` enables the global notify module switch.
+- For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
+- If queue dir is omitted, default is `/opt/rustfs/events`; ensure it is writable by the container runtime user.
+- `RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
+
 **NOTE**: We recommend reviewing the `docker-compose.yml` file before running. It defines several services including Grafana, Prometheus, and Jaeger, which are helpful for RustFS observability. If you wish to start Redis or Nginx containers, you can specify the corresponding profiles.
 
 ### 3\. Build from Source (Option 3) - Advanced Users

README_ZH.md

@@ -113,7 +113,7 @@ RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用
  docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
  # 使用指定版本运行
- docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0.alpha.68
+ docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.3
 ```
 
 您也可以使用 Docker Compose。使用根目录下的 `docker-compose.yml` 文件：

_typos.toml

@@ -39,6 +39,7 @@ abd = "abd"
 mak = "mak"
 gae = "gae"
 GAE = "GAE"
+thr = "thr"
 # s3-tests original test names (cannot be changed)
 nonexisted = "nonexisted"
 consts = "consts"

build-rustfs.sh

@@ -109,6 +109,7 @@ FORCE_CONSOLE_UPDATE=false
 CONSOLE_VERSION="latest"
 SKIP_VERIFICATION=false
 CUSTOM_PLATFORM=""
+FEATURES=""
 
 # Print usage
 usage() {
@@ -141,6 +142,7 @@ usage() {
     echo "  --no-console               Skip console static assets"
     echo "  --force-console-update     Force update console assets even if they exist"
     echo "  --console-version VERSION  Console version to download (default: latest)"
+    echo "  -f, --features FEATURES    Cargo features to enable (e.g. 'webdav', 'full')"
     echo "  --skip-verification        Skip binary verification after build"
     echo "  -h, --help                 Show this help message"
     echo ""
@@ -148,6 +150,8 @@ usage() {
     echo "  $0                         # Build for current platform (includes console assets)"
     echo "  $0 --dev                   # Development build"
     echo "  $0 --sign                  # Build and sign binary (release CI)"
+    echo "  $0 --features webdav       # Build with WebDAV support"
+    echo "  $0 --features full         # Build with all protocol features"
     echo "  $0 --no-console            # Build without console static assets"
     echo "  $0 --force-console-update  # Force update console assets"
     echo "  $0 --platform x86_64-unknown-linux-musl  # Build for specific platform"
@@ -440,6 +444,12 @@ build_binary() {
     build_cmd+=" --target $PLATFORM"
     build_cmd+=" -p rustfs --bins"
 
+    if [ -n "$FEATURES" ]; then
+        local quoted_features
+        printf -v quoted_features '%q' "$FEATURES"
+        build_cmd+=" --features $quoted_features"
+    fi
+
     print_message $BLUE "📦 Executing: $build_cmd"
 
     # Execute build (this matches exactly what the working version does)
@@ -497,6 +507,9 @@ build_rustfs() {
         print_message $YELLOW "   Force Console Update: $FORCE_CONSOLE_UPDATE"
     fi
     print_message $YELLOW "   Skip Verification: $SKIP_VERIFICATION"
+    if [ -n "$FEATURES" ]; then
+        print_message $YELLOW "   Features: $FEATURES"
+    fi
     echo ""
 
     # Setup environment
@@ -565,6 +578,15 @@ while [[ $# -gt 0 ]]; do
             SKIP_VERIFICATION=true
             shift
             ;;
+        -f|--features)
+            if [ -z "${2:-}" ]; then
+                print_message $RED "❌ Missing value for $1"
+                usage
+                exit 1
+            fi
+            FEATURES="$2"
+            shift 2
+            ;;
         -h|--help)
             usage
             exit 0

crates/appauth/Cargo.toml

@@ -26,7 +26,7 @@ categories = ["web-programming", "development-tools", "authentication"]
 
 [dependencies]
 base64-simd = { workspace = true }
-rsa = { workspace = true }
+rsa = { workspace = true, features = ["sha2"] }
 serde.workspace = true
 serde_json.workspace = true
 rand.workspace = true

crates/appauth/src/token.rs

@@ -15,9 +15,13 @@
 use rsa::{
     Pkcs1v15Encrypt, RsaPrivateKey, RsaPublicKey,
     pkcs8::{DecodePrivateKey, DecodePublicKey},
+    pss::{BlindedSigningKey, Signature, VerifyingKey},
+    sha2::Sha256,
+    signature::{RandomizedSigner, Verifier},
+    traits::PublicKeyParts,
 };
 use serde::{Deserialize, Serialize};
-use std::io::{Error, Result};
+use std::io::{Error, ErrorKind, Result};
 
 #[derive(Serialize, Deserialize, Debug, Default, Clone)]
 pub struct Token {
@@ -25,10 +29,11 @@ pub struct Token {
     pub expired: u64, // Expiry time (UNIX timestamp)
 }
 
-/// Public key generation Token
-/// [token] Token object
-/// [key] Public key string
-/// Returns the encrypted string processed by base64
+/// Legacy public-key encryption Token encoder.
+///
+/// Use `sign_license_token` for license issuance so verifiers only need a
+/// public key.
+#[deprecated(note = "use sign_license_token for signed license issuance")]
 pub fn gencode(token: &Token, key: &str) -> Result<String> {
     let data = serde_json::to_vec(token)?;
     let mut rng = rand::rng();
@@ -37,35 +42,62 @@ pub fn gencode(token: &Token, key: &str) -> Result<String> {
     Ok(base64_simd::URL_SAFE_NO_PAD.encode_to_string(&encrypted_data))
 }
 
-/// Private key resolution Token
-/// [token] Encrypted string processed by base64
-/// [key] Private key string
-/// Return to the Token object
+/// Legacy private-key Token decoder.
+///
+/// Use `parse_signed_license_token` or `parse_license_with_public_key` for
+/// license verification so runtime services never need private key material.
+#[deprecated(note = "use parse_signed_license_token or parse_license_with_public_key for signed license verification")]
 pub fn parse(token: &str, key: &str) -> Result<Token> {
     let encrypted_data = base64_simd::URL_SAFE_NO_PAD
         .decode_to_vec(token.as_bytes())
         .map_err(Error::other)?;
     let private_key = RsaPrivateKey::from_pkcs8_pem(key).map_err(Error::other)?;
     let decrypted_data = private_key.decrypt(Pkcs1v15Encrypt, &encrypted_data).map_err(Error::other)?;
-    let res: Token = serde_json::from_slice(&decrypted_data)?;
-    Ok(res)
+    serde_json::from_slice(&decrypted_data).map_err(Error::other)
 }
 
-pub fn parse_license(license: &str) -> Result<Token> {
-    parse(license, TEST_PRIVATE_KEY)
-    // match parse(license, TEST_PRIVATE_KEY) {
-    //     Ok(token) => {
-    //         if token.expired > SystemTime::now().duration_since(UNIX_EPOCH)?.as_secs() {
-    //             Ok(token)
-    //         } else {
-    //             Err("Token expired".into())
-    //         }
-    //     }
-    //     Err(e) => Err(e),
-    // }
+/// Signs a license token with an RSA private key.
+///
+/// The returned token is base64url(signature || payload), where the signature is
+/// RSASSA-PSS over the JSON payload using SHA-256.
+pub fn sign_license_token(token: &Token, private_key_pem: &str) -> Result<String> {
+    let payload = serde_json::to_vec(token)?;
+    let mut rng = rand::rng();
+    let private_key = RsaPrivateKey::from_pkcs8_pem(private_key_pem).map_err(Error::other)?;
+    let signing_key = BlindedSigningKey::<Sha256>::new(private_key);
+    let signature: Signature = signing_key.try_sign_with_rng(&mut rng, &payload).map_err(Error::other)?;
+    let signature: Box<[u8]> = signature.into();
+
+    let mut signed_payload = Vec::with_capacity(signature.as_ref().len() + payload.len());
+    signed_payload.extend_from_slice(signature.as_ref());
+    signed_payload.extend_from_slice(&payload);
+
+    Ok(base64_simd::URL_SAFE_NO_PAD.encode_to_string(&signed_payload))
 }
 
-static TEST_PRIVATE_KEY: &str = "-----BEGIN PRIVATE KEY-----\nMIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCj86SrJIuxSxR6\nBJ/dlJEUIj6NeBRnhLQlCDdovuz61+7kJXVcxaR66w4m8W7SLEUP+IlPtnn6vmiG\n7XMhGNHIr7r1JsEVVLhZmL3tKI66DEZl786ZhG81BWqUlmcooIPS8UEPZNqJXLuz\nVGhxNyVGbj/tV7QC2pSISnKaixc+nrhxvo7w56p5qrm9tik0PjTgfZsUePkoBsSN\npoRkAauS14MAzK6HGB75CzG3dZqXUNWSWVocoWtQbZUwFGXyzU01ammsHQDvc2xu\nK1RQpd1qYH5bOWZ0N0aPFwT0r59HztFXg9sbjsnuhO1A7OiUOkc6iGVuJ0wm/9nA\nwZIBqzgjAgMBAAECggEAPMpeSEbotPhNw2BrllE76ec4omPfzPJbiU+em+wPGoNu\nRJHPDnMKJbl6Kd5jZPKdOOrCnxfd6qcnQsBQa/kz7+GYxMV12l7ra+1Cnujm4v0i\nLTHZvPpp8ZLsjeOmpF3AAzsJEJgon74OqtOlVjVIUPEYKvzV9ijt4gsYq0zfdYv0\nhrTMzyrGM4/UvKLsFIBROAfCeWfA7sXLGH8JhrRAyDrtCPzGtyyAmzoHKHtHafcB\nuyPFw/IP8otAgpDk5iiQPNkH0WwzAQIm12oHuNUa66NwUK4WEjXTnDg8KeWLHHNv\nIfN8vdbZchMUpMIvvkr7is315d8f2cHCB5gEO+GWAQKBgQDR/0xNll+FYaiUKCPZ\nvkOCAd3l5mRhsqnjPQ/6Ul1lAyYWpoJSFMrGGn/WKTa/FVFJRTGbBjwP+Mx10bfb\ngUg2GILDTISUh54fp4zngvTi9w4MWGKXrb7I1jPkM3vbJfC/v2fraQ/r7qHPpO2L\nf6ZbGxasIlSvr37KeGoelwcAQQKBgQDH3hmOTS2Hl6D4EXdq5meHKrfeoicGN7m8\noQK7u8iwn1R9zK5nh6IXxBhKYNXNwdCQtBZVRvFjjZ56SZJb7lKqa1BcTsgJfZCy\nnI3Uu4UykrECAH8AVCVqBXUDJmeA2yE+gDAtYEjvhSDHpUfWxoGHr0B/Oqk2Lxc/\npRy1qV5fYwKBgBWSL/hYVf+RhIuTg/s9/BlCr9SJ0g3nGGRrRVTlWQqjRCpXeFOO\nJzYqSq9pFGKUggEQxoOyJEFPwVDo9gXqRcyov+Xn2kaXl7qQr3yoixc1YZALFDWY\nd1ySBEqQr0xXnV9U/gvEgwotPRnjSzNlLWV2ZuHPtPtG/7M0o1H5GZMBAoGAKr3N\nW0gX53o+my4pCnxRQW+aOIsWq1a5aqRIEFudFGBOUkS2Oz+fI1P1GdrRfhnnfzpz\n2DK+plp/vIkFOpGhrf4bBlJ2psjqa7fdANRFLMaAAfyXLDvScHTQTCcnVUAHQPVq\n2BlSH56pnugyj7SNuLV6pnql+wdhAmRN2m9o1h8CgYAbX2juSr4ioXwnYjOUdrIY\n4+ERvHcXdjoJmmPcAm4y5NbSqLXyU0FQmplNMt2A5LlniWVJ9KNdjAQUt60FZw/+\nr76LdxXaHNZghyx0BOs7mtq5unSQXamZ8KixasfhE9uz3ij1jXjG6hafWkS8/68I\nuWbaZqgvy7a9oPHYlKH7Jg==\n-----END PRIVATE KEY-----\n";
+/// Verifies and parses a signed license token with an RSA public key.
+pub fn parse_signed_license_token(token: &str, public_key_pem: &str) -> Result<Token> {
+    let signed_payload = base64_simd::URL_SAFE_NO_PAD
+        .decode_to_vec(token.as_bytes())
+        .map_err(Error::other)?;
+    let public_key = RsaPublicKey::from_public_key_pem(public_key_pem).map_err(Error::other)?;
+    let signature_len = public_key.size();
+
+    if signed_payload.len() <= signature_len {
+        return Err(Error::new(ErrorKind::InvalidData, "license token is missing signed payload"));
+    }
+
+    let (signature, payload) = signed_payload.split_at(signature_len);
+    let signature = Signature::try_from(signature).map_err(Error::other)?;
+    let verifying_key = VerifyingKey::<Sha256>::new(public_key);
+    verifying_key.verify(payload, &signature).map_err(Error::other)?;
+
+    serde_json::from_slice(payload).map_err(Error::other)
+}
+
+pub fn parse_license_with_public_key(license: &str, public_key: &str) -> Result<Token> {
+    parse_signed_license_token(license, public_key)
+}
 
 #[cfg(test)]
 mod tests {
@@ -77,7 +109,7 @@ mod tests {
     use std::time::{SystemTime, UNIX_EPOCH};
 
     #[test]
-    fn test_gencode_and_parse() {
+    fn test_sign_license_token_and_parse_signed_license_token() {
         let mut rng = rand::rng();
         let bits = 2048;
         let private_key = RsaPrivateKey::new(&mut rng, bits).expect("Failed to generate private key");
@@ -91,8 +123,31 @@ mod tests {
             expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600, // 1 hour from now
         };
 
-        let encoded = gencode(&token, &public_key_pem).expect("Failed to encode token");
+        let encoded = sign_license_token(&token, &private_key_pem).expect("Failed to encode token");
 
+        let decoded = parse_signed_license_token(&encoded, &public_key_pem).expect("Failed to decode token");
+
+        assert_eq!(token.name, decoded.name);
+        assert_eq!(token.expired, decoded.expired);
+    }
+
+    #[test]
+    #[allow(deprecated)]
+    fn test_legacy_gencode_and_parse_roundtrip() {
+        let mut rng = rand::rng();
+        let bits = 2048;
+        let private_key = RsaPrivateKey::new(&mut rng, bits).expect("Failed to generate private key");
+        let public_key = RsaPublicKey::from(&private_key);
+
+        let private_key_pem = private_key.to_pkcs8_pem(LineEnding::LF).unwrap();
+        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+
+        let token = Token {
+            name: "test_app".to_string(),
+            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600,
+        };
+
+        let encoded = gencode(&token, &public_key_pem).expect("Failed to encode token");
         let decoded = parse(&encoded, &private_key_pem).expect("Failed to decode token");
 
         assert_eq!(token.name, decoded.name);
@@ -100,28 +155,60 @@ mod tests {
     }
 
     #[test]
-    fn test_parse_invalid_token() {
+    fn test_parse_signed_license_token_rejects_tampered_payload() {
         let mut rng = rand::rng();
-        let private_key_pem = RsaPrivateKey::new(&mut rng, 2048)
-            .expect("Failed to generate private key")
-            .to_pkcs8_pem(LineEnding::LF)
-            .unwrap();
+        let private_key = RsaPrivateKey::new(&mut rng, 2048).expect("Failed to generate private key");
+        let public_key = RsaPublicKey::from(&private_key);
+        let private_key_pem = private_key.to_pkcs8_pem(LineEnding::LF).unwrap();
+        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+        let token = Token {
+            name: "test_app".to_string(),
+            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600,
+        };
 
-        let invalid_token = "invalid_base64_token";
-        let result = parse(invalid_token, &private_key_pem);
+        let encoded = sign_license_token(&token, &private_key_pem).expect("Failed to encode token");
+        let mut signed_payload = base64_simd::URL_SAFE_NO_PAD
+            .decode_to_vec(encoded.as_bytes())
+            .expect("Failed to decode signed payload");
+        let last_byte = signed_payload.last_mut().expect("Signed payload should not be empty");
+        *last_byte ^= 0x01;
+        let tampered = base64_simd::URL_SAFE_NO_PAD.encode_to_string(&signed_payload);
+
+        let result = parse_signed_license_token(&tampered, &public_key_pem);
 
         assert!(result.is_err());
     }
 
     #[test]
-    fn test_gencode_with_invalid_key() {
+    fn test_source_does_not_embed_private_key() {
+        let source = include_str!("token.rs");
+        let forbidden = ["BEGIN", "PRIVATE KEY"].join(" ");
+
+        assert!(!source.contains(&forbidden));
+    }
+
+    #[test]
+    fn test_parse_signed_license_token_rejects_invalid_token() {
+        let mut rng = rand::rng();
+        let private_key = RsaPrivateKey::new(&mut rng, 2048).expect("Failed to generate private key");
+        let public_key = RsaPublicKey::from(&private_key);
+        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+
+        let invalid_token = "invalid_base64_token";
+        let result = parse_signed_license_token(invalid_token, &public_key_pem);
+
+        assert!(result.is_err());
+    }
+
+    #[test]
+    fn test_sign_license_token_with_invalid_signing_key() {
         let token = Token {
             name: "test_app".to_string(),
             expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600, // 1 hour from now
         };
 
-        let invalid_key = "invalid_public_key";
-        let result = gencode(&token, invalid_key);
+        let invalid_key = "invalid_private_key";
+        let result = sign_license_token(&token, invalid_key);
 
         assert!(result.is_err());
     }

crates/audit/AGENTS.md

@@ -0,0 +1,53 @@
+# Audit Crate Instructions
+
+Applies to `crates/audit/`.
+
+`rustfs-audit` is the domain layer for audit event fan-out and observability.
+It composes shared plugin/runtime abstractions from `rustfs-targets` and keeps
+audit-specific dispatch semantics, state transitions, and metrics in this
+crate.
+
+## Domain Boundaries
+
+- Keep audit-specific behavior here:
+  - audit event shaping and fan-out pipeline
+  - audit system lifecycle/state transitions
+  - audit metrics and reporting
+- Keep shared plugin/runtime mechanics in `rustfs-targets`:
+  - no duplicated replay worker orchestration
+  - no duplicated runtime manager primitives
+  - no plugin install/control-plane modeling in this crate
+
+## Runtime Layering Rules
+
+- `pipeline.rs` hosts:
+  - `AuditPipeline` (dispatch and snapshot access)
+  - `AuditRuntimeFacade` (runtime mutation path)
+  - `AuditRuntimeView` (runtime read path)
+- `registry.rs` should remain the single owner of runtime target container and
+  plugin registry composition for audit.
+- `system.rs` should coordinate lifecycle by calling facade/view/registry
+  boundaries rather than embedding low-level runtime logic.
+
+## Change Style
+
+- Preserve audit delivery semantics and error handling behavior unless the task
+  explicitly changes them.
+- Prefer extending shared abstractions in `rustfs-targets` over patching
+  one-off audit-only runtime flows.
+- Keep logging and observability machine-meaningful; avoid noisy churn in hot
+  dispatch paths.
+
+## Testing
+
+- Keep unit tests close to changed modules.
+- Keep pipeline-layer regressions in `tests/pipeline_layer_test.rs`.
+- Add regression tests for:
+  - runtime facade activation/replace/stop/shutdown behavior
+  - runtime view target/snapshot access
+  - system reload and runtime commit/clear boundaries
+- Suggested validation:
+  - `cargo test -p rustfs-audit`
+  - Focused: `cargo test -p rustfs-audit --test pipeline_layer_test`
+  - Focused: `cargo test -p rustfs-audit pipeline`
+- Full gate before commit: `make pre-commit`

crates/audit/Cargo.toml

@@ -30,7 +30,6 @@ rustfs-targets = { workspace = true }
 rustfs-config = { workspace = true, features = ["audit", "constants"] }
 rustfs-ecstore = { workspace = true }
 rustfs-s3-common = { workspace = true }
-async-trait = { workspace = true }
 chrono = { workspace = true }
 const-str = { workspace = true }
 futures = { workspace = true }
@@ -41,15 +40,14 @@ serde_json = { workspace = true }
 thiserror = { workspace = true }
 tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "rt", "time", "macros"] }
 tracing = { workspace = true, features = ["std", "attributes"] }
-url = { workspace = true }
-rumqttc = { workspace = true }
 
 [dev-dependencies]
+async-trait = { workspace = true }
 temp-env = { workspace = true }
+url = { workspace = true }
 
 [lints]
 workspace = true
 
 [lib]
-test = false
 doctest = false

crates/audit/src/factory.rs

@@ -13,233 +13,65 @@
 // limitations under the License.
 
 use crate::AuditEntry;
-use async_trait::async_trait;
-use hashbrown::HashSet;
-use rumqttc::QoS;
-use rustfs_config::audit::{AUDIT_MQTT_KEYS, AUDIT_WEBHOOK_KEYS, ENV_AUDIT_MQTT_KEYS, ENV_AUDIT_WEBHOOK_KEYS};
-use rustfs_config::{
-    AUDIT_DEFAULT_DIR, DEFAULT_LIMIT, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR,
-    MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY,
-    MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, RUSTFS_WEBHOOK_SKIP_TLS_VERIFY_DEFAULT,
-    WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR,
-    WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY,
-};
-use rustfs_ecstore::config::KVS;
-use rustfs_targets::{
-    Target,
-    error::TargetError,
-    target::{
-        mqtt::{MQTTArgs, MQTTTlsConfig, validate_mqtt_broker_url},
-        webhook::WebhookArgs,
-    },
-};
-use std::time::Duration;
-use tracing::{debug, warn};
-use url::Url;
+use rustfs_targets::catalog::builtin::builtin_audit_target_descriptors;
+use rustfs_targets::{BuiltinTargetDescriptor, TargetPluginDescriptor};
 
-/// Trait for creating targets from configuration
-#[async_trait]
-pub trait TargetFactory: Send + Sync {
-    /// Creates a target from configuration
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError>;
-
-    /// Validates target configuration
-    fn validate_config(&self, id: &str, config: &KVS) -> Result<(), TargetError>;
-
-    /// Returns a set of valid configuration field names for this target type.
-    /// This is used to filter environment variables.
-    fn get_valid_fields(&self) -> HashSet<String>;
-
-    /// Returns a set of valid configuration env field names for this target type.
-    /// This is used to filter environment variables.
-    fn get_valid_env_fields(&self) -> HashSet<String>;
+pub fn builtin_target_descriptors() -> Vec<BuiltinTargetDescriptor<AuditEntry>> {
+    builtin_audit_target_descriptors::<AuditEntry>()
 }
 
-/// Factory for creating Webhook targets
-pub struct WebhookTargetFactory;
-
-#[async_trait]
-impl TargetFactory for WebhookTargetFactory {
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        // All config values are now read directly from the merged `config` KVS.
-        let endpoint = config
-            .lookup(WEBHOOK_ENDPOINT)
-            .ok_or_else(|| TargetError::Configuration("Missing webhook endpoint".to_string()))?;
-        let parsed_endpoint = endpoint.trim();
-        let endpoint_url = Url::parse(parsed_endpoint)
-            .map_err(|e| TargetError::Configuration(format!("Invalid endpoint URL: {e} (value: '{parsed_endpoint}')")))?;
-
-        let args = WebhookArgs {
-            enable: true, // If we are here, it's already enabled.
-            endpoint: endpoint_url,
-            auth_token: config.lookup(WEBHOOK_AUTH_TOKEN).unwrap_or_default(),
-            queue_dir: config.lookup(WEBHOOK_QUEUE_DIR).unwrap_or(AUDIT_DEFAULT_DIR.to_string()),
-            queue_limit: config
-                .lookup(WEBHOOK_QUEUE_LIMIT)
-                .and_then(|v| v.parse::<u64>().ok())
-                .unwrap_or(DEFAULT_LIMIT),
-            client_cert: config.lookup(WEBHOOK_CLIENT_CERT).unwrap_or_default(),
-            client_key: config.lookup(WEBHOOK_CLIENT_KEY).unwrap_or_default(),
-            client_ca: config.lookup(WEBHOOK_CLIENT_CA).unwrap_or_default(),
-            skip_tls_verify: config
-                .lookup(WEBHOOK_SKIP_TLS_VERIFY)
-                .and_then(|v| v.parse::<bool>().ok())
-                .unwrap_or(RUSTFS_WEBHOOK_SKIP_TLS_VERIFY_DEFAULT),
-            target_type: rustfs_targets::target::TargetType::AuditLog,
-        };
-
-        let target = rustfs_targets::target::webhook::WebhookTarget::new(id, args)?;
-        Ok(Box::new(target))
-    }
-
-    fn validate_config(&self, _id: &str, config: &KVS) -> Result<(), TargetError> {
-        // Validation also uses the merged `config` KVS directly.
-        let endpoint = config
-            .lookup(WEBHOOK_ENDPOINT)
-            .ok_or_else(|| TargetError::Configuration("Missing webhook endpoint".to_string()))?;
-        debug!("endpoint: {}", endpoint);
-        let parsed_endpoint = endpoint.trim();
-        Url::parse(parsed_endpoint)
-            .map_err(|e| TargetError::Configuration(format!("Invalid endpoint URL: {e} (value: '{parsed_endpoint}')")))?;
-
-        let client_cert = config.lookup(WEBHOOK_CLIENT_CERT).unwrap_or_default();
-        let client_key = config.lookup(WEBHOOK_CLIENT_KEY).unwrap_or_default();
-
-        if client_cert.is_empty() != client_key.is_empty() {
-            return Err(TargetError::Configuration(
-                "Both client_cert and client_key must be specified together".to_string(),
-            ));
-        }
-
-        let queue_dir = config.lookup(WEBHOOK_QUEUE_DIR).unwrap_or(AUDIT_DEFAULT_DIR.to_string());
-        if !queue_dir.is_empty() && !std::path::Path::new(&queue_dir).is_absolute() {
-            return Err(TargetError::Configuration("Webhook queue directory must be an absolute path".to_string()));
-        }
-
-        Ok(())
-    }
-
-    fn get_valid_fields(&self) -> HashSet<String> {
-        AUDIT_WEBHOOK_KEYS.iter().map(|s| s.to_string()).collect()
-    }
-
-    fn get_valid_env_fields(&self) -> HashSet<String> {
-        ENV_AUDIT_WEBHOOK_KEYS.iter().map(|s| s.to_string()).collect()
-    }
+pub fn builtin_target_plugins() -> Vec<TargetPluginDescriptor<AuditEntry>> {
+    builtin_target_descriptors()
+        .into_iter()
+        .map(|descriptor| descriptor.plugin().clone())
+        .collect()
 }
 
-/// Factory for creating MQTT targets
-pub struct MQTTTargetFactory;
+#[cfg(test)]
+mod tests {
+    use super::builtin_target_descriptors;
+    use rustfs_config::audit::AUDIT_AMQP_KEYS;
+    use rustfs_config::{AMQP_EXCHANGE, AMQP_QUEUE_DIR, AMQP_ROUTING_KEY, AMQP_URL};
+    use rustfs_ecstore::config::KVS;
+    use rustfs_targets::target::ChannelTargetType;
 
-#[async_trait]
-impl TargetFactory for MQTTTargetFactory {
-    async fn create_target(&self, id: String, config: &KVS) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let broker = config
-            .lookup(MQTT_BROKER)
-            .ok_or_else(|| TargetError::Configuration("Missing MQTT broker".to_string()))?;
-        let broker_url = Url::parse(&broker)
-            .map_err(|e| TargetError::Configuration(format!("Invalid broker URL: {e} (value: '{broker}')")))?;
-
-        let topic = config
-            .lookup(MQTT_TOPIC)
-            .ok_or_else(|| TargetError::Configuration("Missing MQTT topic".to_string()))?;
-
-        let args = MQTTArgs {
-            enable: true, // Assumed enabled.
-            broker: broker_url,
-            topic,
-            qos: config
-                .lookup(MQTT_QOS)
-                .and_then(|v| v.parse::<u8>().ok())
-                .map(|q| match q {
-                    0 => QoS::AtMostOnce,
-                    1 => QoS::AtLeastOnce,
-                    2 => QoS::ExactlyOnce,
-                    _ => QoS::AtLeastOnce,
-                })
-                .unwrap_or(QoS::AtLeastOnce),
-            username: config.lookup(MQTT_USERNAME).unwrap_or_default(),
-            password: config.lookup(MQTT_PASSWORD).unwrap_or_default(),
-            max_reconnect_interval: config
-                .lookup(MQTT_RECONNECT_INTERVAL)
-                .and_then(|v| v.parse::<u64>().ok())
-                .map(Duration::from_secs)
-                .unwrap_or_else(|| Duration::from_secs(5)),
-            keep_alive: config
-                .lookup(MQTT_KEEP_ALIVE_INTERVAL)
-                .and_then(|v| v.parse::<u64>().ok())
-                .map(Duration::from_secs)
-                .unwrap_or_else(|| Duration::from_secs(30)),
-            tls: MQTTTlsConfig::from_values(
-                config.lookup(MQTT_TLS_POLICY).as_deref(),
-                config.lookup(MQTT_TLS_CA).as_deref(),
-                config.lookup(MQTT_TLS_CLIENT_CERT).as_deref(),
-                config.lookup(MQTT_TLS_CLIENT_KEY).as_deref(),
-                config.lookup(MQTT_TLS_TRUST_LEAF_AS_CA).as_deref(),
-                config.lookup(MQTT_WS_PATH_ALLOWLIST).as_deref(),
-            )?,
-            queue_dir: config.lookup(MQTT_QUEUE_DIR).unwrap_or(AUDIT_DEFAULT_DIR.to_string()),
-            queue_limit: config
-                .lookup(MQTT_QUEUE_LIMIT)
-                .and_then(|v| v.parse::<u64>().ok())
-                .unwrap_or(DEFAULT_LIMIT),
-            target_type: rustfs_targets::target::TargetType::AuditLog,
-        };
-
-        let target = rustfs_targets::target::mqtt::MQTTTarget::new(id, args)?;
-        Ok(Box::new(target))
+    fn amqp_base_config() -> KVS {
+        let mut config = KVS::new();
+        config.insert(AMQP_URL.to_string(), "amqp://127.0.0.1:5672/%2f".to_string());
+        config.insert(AMQP_EXCHANGE.to_string(), "rustfs.audit".to_string());
+        config.insert(AMQP_ROUTING_KEY.to_string(), "audit".to_string());
+        config.insert(AMQP_QUEUE_DIR.to_string(), String::new());
+        config
     }
 
-    fn validate_config(&self, _id: &str, config: &KVS) -> Result<(), TargetError> {
-        let broker = config
-            .lookup(MQTT_BROKER)
-            .ok_or_else(|| TargetError::Configuration("Missing MQTT broker".to_string()))?;
-        let url = Url::parse(&broker)
-            .map_err(|e| TargetError::Configuration(format!("Invalid broker URL: {e} (value: '{broker}')")))?;
+    #[test]
+    fn builtin_plugins_include_amqp_descriptor() {
+        let plugin = builtin_target_descriptors()
+            .into_iter()
+            .find(|plugin| plugin.plugin().target_type() == ChannelTargetType::Amqp.as_str())
+            .expect("amqp plugin should exist");
 
-        let tls = MQTTTlsConfig::from_values(
-            config.lookup(MQTT_TLS_POLICY).as_deref(),
-            config.lookup(MQTT_TLS_CA).as_deref(),
-            config.lookup(MQTT_TLS_CLIENT_CERT).as_deref(),
-            config.lookup(MQTT_TLS_CLIENT_KEY).as_deref(),
-            config.lookup(MQTT_TLS_TRUST_LEAF_AS_CA).as_deref(),
-            config.lookup(MQTT_WS_PATH_ALLOWLIST).as_deref(),
-        )?;
-        validate_mqtt_broker_url(&url, &tls)?;
-
-        if config.lookup(MQTT_TOPIC).is_none() {
-            return Err(TargetError::Configuration("Missing MQTT topic".to_string()));
-        }
-
-        if let Some(qos_str) = config.lookup(MQTT_QOS) {
-            let qos = qos_str
-                .parse::<u8>()
-                .map_err(|_| TargetError::Configuration("Invalid QoS value".to_string()))?;
-            if qos > 2 {
-                return Err(TargetError::Configuration("QoS must be 0, 1, or 2".to_string()));
-            }
-        }
-
-        let queue_dir = config.lookup(MQTT_QUEUE_DIR).unwrap_or_default();
-        if !queue_dir.is_empty() {
-            if !std::path::Path::new(&queue_dir).is_absolute() {
-                return Err(TargetError::Configuration("MQTT queue directory must be an absolute path".to_string()));
-            }
-            if let Some(qos_str) = config.lookup(MQTT_QOS)
-                && qos_str == "0"
-            {
-                warn!("Using queue_dir with QoS 0 may result in event loss");
-            }
-        }
-
-        Ok(())
+        assert!(plugin.plugin().valid_fields().contains(&AMQP_URL));
+        assert!(plugin.plugin().valid_fields().contains(&AMQP_EXCHANGE));
+        assert!(plugin.plugin().valid_fields().contains(&AMQP_ROUTING_KEY));
+        assert_eq!(plugin.plugin().valid_fields().len(), AUDIT_AMQP_KEYS.len());
     }
 
-    fn get_valid_fields(&self) -> HashSet<String> {
-        AUDIT_MQTT_KEYS.iter().map(|s| s.to_string()).collect()
-    }
+    #[test]
+    fn builtin_plugins_create_audit_amqp_target() {
+        let plugin = builtin_target_descriptors()
+            .into_iter()
+            .find(|plugin| plugin.plugin().target_type() == ChannelTargetType::Amqp.as_str())
+            .expect("amqp plugin should exist");
 
-    fn get_valid_env_fields(&self) -> HashSet<String> {
-        ENV_AUDIT_MQTT_KEYS.iter().map(|s| s.to_string()).collect()
+        let target = plugin
+            .plugin()
+            .create_target("primary".to_string(), &amqp_base_config())
+            .expect("AMQP audit target should be created");
+
+        let target_id = target.id();
+        assert_eq!(target_id.id, "primary");
+        assert_eq!(target_id.name, "amqp");
+        assert!(target.store().is_none());
     }
 }

crates/audit/src/global.rs

@@ -12,7 +12,7 @@
 //  See the License for the specific language governing permissions and
 //  limitations under the License.
 
-use crate::{AuditEntry, AuditResult, AuditSystem};
+use crate::{AuditEntry, AuditResult, AuditSystem, system::AuditTargetMetricSnapshot};
 use rustfs_ecstore::config::Config;
 use std::sync::{Arc, OnceLock};
 use tracing::{debug, error, trace, warn};
@@ -89,6 +89,15 @@ pub async fn reload_audit_config(config: Config) -> AuditResult<()> {
     with_audit_system!(|system: Arc<AuditSystem>| async move { system.reload_config(config).await })
 }
 
+/// Returns per-target audit delivery metrics for Prometheus collection.
+pub async fn audit_target_metrics() -> Vec<AuditTargetMetricSnapshot> {
+    if let Some(system) = audit_system() {
+        system.snapshot_target_metrics().await
+    } else {
+        Vec::new()
+    }
+}
+
 /// Check if the global audit system is running
 pub async fn is_audit_system_running() -> bool {
     if let Some(system) = audit_system() {

crates/audit/src/lib.rs

@@ -23,6 +23,7 @@ pub mod error;
 pub mod factory;
 pub mod global;
 pub mod observability;
+pub mod pipeline;
 pub mod registry;
 pub mod system;
 
@@ -30,5 +31,6 @@ pub use entity::{ApiDetails, AuditEntry, ObjectVersion};
 pub use error::{AuditError, AuditResult};
 pub use global::*;
 pub use observability::{AuditMetrics, AuditMetricsReport, PerformanceValidation};
+pub use pipeline::{AuditPipeline, AuditRuntimeFacade, AuditRuntimeView};
 pub use registry::AuditRegistry;
-pub use system::AuditSystem;
+pub use system::{AuditSystem, AuditTargetMetricSnapshot};

crates/audit/src/pipeline.rs

@@ -0,0 +1,355 @@
+//  Copyright 2024 RustFS Team
+//
+//  Licensed under the Apache License, Version 2.0 (the "License");
+//  you may not use this file except in compliance with the License.
+//  You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+//  Unless required by applicable law or agreed to in writing, software
+//  distributed under the License is distributed on an "AS IS" BASIS,
+//  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+//  See the License for the specific language governing permissions and
+//  limitations under the License.
+
+use crate::{AuditEntry, AuditResult, observability, system::AuditTargetMetricSnapshot};
+use rustfs_targets::{
+    BuiltinPluginRuntimeAdapter, PluginRuntimeAdapter, ReplayEvent, ReplayWorkerManager, RuntimeActivation, SharedTarget, Target,
+    target::EntityTarget,
+};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::sync::{Mutex, RwLock};
+use tracing::{error, info, warn};
+
+#[derive(Clone)]
+pub struct AuditPipeline {
+    registry: Arc<Mutex<crate::AuditRegistry>>,
+}
+
+impl AuditPipeline {
+    pub fn new(registry: Arc<Mutex<crate::AuditRegistry>>) -> Self {
+        Self { registry }
+    }
+
+    pub async fn dispatch(&self, entry: Arc<AuditEntry>) -> AuditResult<()> {
+        let start_time = std::time::Instant::now();
+
+        let targets: Vec<SharedTarget<AuditEntry>> = {
+            let registry = self.registry.lock().await;
+            let targets = registry.list_target_values();
+
+            if targets.is_empty() {
+                warn!("No audit targets configured for dispatch");
+                return Ok(());
+            }
+
+            targets
+        };
+
+        let mut tasks = Vec::new();
+
+        for target in targets {
+            let entity_target = EntityTarget {
+                object_name: entry.api.name.clone().unwrap_or_default(),
+                bucket_name: entry.api.bucket.clone().unwrap_or_default(),
+                event_name: entry.event,
+                data: (*entry).clone(),
+            };
+
+            let task = async move {
+                let result = target.save(Arc::new(entity_target)).await;
+                (target.id().to_string(), result)
+            };
+
+            tasks.push(task);
+        }
+
+        let results = futures::future::join_all(tasks).await;
+
+        let mut errors = Vec::new();
+        let mut success_count = 0;
+
+        for (target_key, result) in results {
+            match result {
+                Ok(_) => {
+                    success_count += 1;
+                    observability::record_target_success();
+                }
+                Err(e) => {
+                    error!(target_id = %target_key, error = %e, "Failed to dispatch audit log to target");
+                    errors.push(e);
+                    observability::record_target_failure();
+                }
+            }
+        }
+
+        let dispatch_time = start_time.elapsed();
+
+        if errors.is_empty() {
+            observability::record_audit_success(dispatch_time);
+        } else {
+            observability::record_audit_failure(dispatch_time);
+            warn!(
+                error_count = errors.len(),
+                success_count = success_count,
+                "Some audit targets failed to receive log entry"
+            );
+        }
+
+        Ok(())
+    }
+
+    pub async fn dispatch_batch(&self, entries: Vec<Arc<AuditEntry>>) -> AuditResult<()> {
+        let start_time = std::time::Instant::now();
+
+        let targets: Vec<SharedTarget<AuditEntry>> = {
+            let registry = self.registry.lock().await;
+            let targets = registry.list_target_values();
+
+            if targets.is_empty() {
+                warn!("No audit targets configured for batch dispatch");
+                return Ok(());
+            }
+
+            targets
+        };
+
+        let mut tasks = Vec::new();
+        for target in targets {
+            let entries_clone: Vec<_> = entries.iter().map(Arc::clone).collect();
+
+            let task = async move {
+                let mut success_count = 0;
+                let mut errors = Vec::new();
+                for entry in entries_clone {
+                    let entity_target = EntityTarget {
+                        object_name: entry.api.name.clone().unwrap_or_default(),
+                        bucket_name: entry.api.bucket.clone().unwrap_or_default(),
+                        event_name: entry.event,
+                        data: (*entry).clone(),
+                    };
+                    match target.save(Arc::new(entity_target)).await {
+                        Ok(_) => success_count += 1,
+                        Err(e) => errors.push(e),
+                    }
+                }
+                (target.id().to_string(), success_count, errors)
+            };
+            tasks.push(task);
+        }
+
+        let results = futures::future::join_all(tasks).await;
+        let mut total_success = 0;
+        let mut total_errors = 0;
+        for (_target_id, success_count, errors) in results {
+            total_success += success_count;
+            total_errors += errors.len();
+            for e in errors {
+                error!("Batch dispatch error: {:?}", e);
+            }
+        }
+
+        let dispatch_time = start_time.elapsed();
+        info!(
+            "Batch dispatched {} entries, success: {}, errors: {}, time: {:?}",
+            entries.len(),
+            total_success,
+            total_errors,
+            dispatch_time
+        );
+
+        Ok(())
+    }
+
+    pub async fn snapshot_target_metrics(&self) -> Vec<AuditTargetMetricSnapshot> {
+        let registry = self.registry.lock().await;
+        registry
+            .list_target_values()
+            .into_iter()
+            .map(|target| {
+                let delivery = target.delivery_snapshot();
+                AuditTargetMetricSnapshot {
+                    failed_messages: delivery.failed_messages,
+                    queue_length: delivery.queue_length,
+                    target_id: target.id().to_string(),
+                    total_messages: delivery.total_messages,
+                }
+            })
+            .collect()
+    }
+
+    pub async fn snapshot_target_health(&self) -> Vec<rustfs_targets::RuntimeTargetHealthSnapshot> {
+        let registry = self.registry.lock().await;
+        registry.runtime_manager().health_snapshots().await
+    }
+}
+
+#[derive(Clone)]
+pub struct AuditRuntimeView {
+    registry: Arc<Mutex<crate::AuditRegistry>>,
+}
+
+impl AuditRuntimeView {
+    pub fn new(registry: Arc<Mutex<crate::AuditRegistry>>) -> Self {
+        Self { registry }
+    }
+
+    pub async fn list_targets(&self) -> Vec<String> {
+        let registry = self.registry.lock().await;
+        registry.list_targets()
+    }
+
+    pub async fn get_target_values(&self) -> Vec<rustfs_targets::SharedTarget<AuditEntry>> {
+        let registry = self.registry.lock().await;
+        registry.list_target_values()
+    }
+
+    pub async fn get_target(&self, target_id: &str) -> Option<String> {
+        let registry = self.registry.lock().await;
+        registry.get_target(target_id).map(|target| target.id().to_string())
+    }
+
+    pub async fn enable_target(&self, target_id: &str) -> AuditResult<()> {
+        let registry = self.registry.lock().await;
+        if registry.get_target(target_id).is_some() {
+            info!(target_id = %target_id, "Target enabled");
+            Ok(())
+        } else {
+            Err(crate::AuditError::Configuration(format!("Target not found: {target_id}"), None))
+        }
+    }
+
+    pub async fn disable_target(&self, target_id: &str) -> AuditResult<()> {
+        let registry = self.registry.lock().await;
+        if registry.get_target(target_id).is_some() {
+            info!(target_id = %target_id, "Target disabled");
+            Ok(())
+        } else {
+            Err(crate::AuditError::Configuration(format!("Target not found: {target_id}"), None))
+        }
+    }
+
+    pub async fn remove_target(&self, target_id: &str) -> AuditResult<()> {
+        let mut registry = self.registry.lock().await;
+        if registry.remove_target(target_id).await.is_some() {
+            info!(target_id = %target_id, "Target removed");
+            Ok(())
+        } else {
+            Err(crate::AuditError::Configuration(format!("Target not found: {target_id}"), None))
+        }
+    }
+
+    pub async fn upsert_target(&self, target_id: String, target: Box<dyn Target<AuditEntry> + Send + Sync>) -> AuditResult<()> {
+        if let Err(err) = target.init().await {
+            return Err(crate::AuditError::Target(err));
+        }
+
+        let shared_target: SharedTarget<AuditEntry> = Arc::from(target);
+        let mut registry = self.registry.lock().await;
+        let _ = registry.remove_target(&target_id).await;
+        registry.add_shared_target(target_id.clone(), shared_target);
+        info!(target_id = %target_id, "Target upserted");
+        Ok(())
+    }
+}
+
+#[derive(Clone)]
+pub struct AuditRuntimeFacade {
+    registry: Arc<Mutex<crate::AuditRegistry>>,
+    replay_workers: Arc<RwLock<ReplayWorkerManager>>,
+    runtime_adapter: Arc<dyn PluginRuntimeAdapter<AuditEntry>>,
+}
+
+impl AuditRuntimeFacade {
+    pub fn new(registry: Arc<Mutex<crate::AuditRegistry>>, replay_workers: Arc<RwLock<ReplayWorkerManager>>) -> Self {
+        let runtime_adapter = BuiltinPluginRuntimeAdapter::new(
+            Arc::new(move |event: ReplayEvent<AuditEntry>| {
+                Box::pin(async move {
+                    match event {
+                        ReplayEvent::Delivered { key, target } => {
+                            info!("Successfully sent audit entry, target: {}, key: {}", target.id(), key.to_string());
+                            observability::record_target_success();
+                        }
+                        ReplayEvent::RetryableError { error, target, .. } => match error {
+                            rustfs_targets::TargetError::NotConnected => {
+                                warn!("Target {} not connected, retrying...", target.id());
+                            }
+                            rustfs_targets::TargetError::Timeout(_) => {
+                                warn!("Timeout sending to target {}, retrying...", target.id());
+                            }
+                            _ => {}
+                        },
+                        ReplayEvent::Dropped { reason, target, .. } => {
+                            warn!("Dropped queued payload for target {}: {}", target.id(), reason);
+                            observability::record_target_failure();
+                        }
+                        ReplayEvent::PermanentFailure { error, target, .. } => {
+                            error!("Permanent error for target {}: {}", target.id(), error);
+                            target.record_final_failure();
+                            observability::record_target_failure();
+                        }
+                        ReplayEvent::RetryExhausted { key, target } => {
+                            warn!("Max retries exceeded for key {}, target: {}, skipping", key.to_string(), target.id());
+                            target.record_final_failure();
+                            observability::record_target_failure();
+                        }
+                        ReplayEvent::UnreadableEntry { key, error, target } => {
+                            warn!("Skipping unreadable audit store entry {} for target {}: {}", key, target.id(), error);
+                        }
+                    }
+                })
+            }),
+            Arc::new(|target_id, has_replay| {
+                if has_replay {
+                    info!(target_id = %target_id, "Audit stream processing started");
+                } else {
+                    info!(target_id = %target_id, "No store configured, skip audit stream processing");
+                }
+            }),
+            None,
+            Duration::from_millis(500),
+            Duration::from_millis(500),
+            "Stopping audit stream",
+        );
+
+        Self {
+            registry,
+            replay_workers,
+            runtime_adapter: Arc::new(runtime_adapter),
+        }
+    }
+
+    pub async fn replace_targets(&self, activation: RuntimeActivation<AuditEntry>) -> AuditResult<()> {
+        let mut registry = self.registry.lock().await;
+        let mut replay_workers = self.replay_workers.write().await;
+        self.runtime_adapter
+            .replace_runtime_targets(registry.runtime_manager_mut(), &mut replay_workers, activation)
+            .await
+            .map_err(crate::AuditError::Target)?;
+        Ok(())
+    }
+
+    pub async fn shutdown_runtime(
+        &self,
+        registry: &mut crate::AuditRegistry,
+        replay_workers: &mut ReplayWorkerManager,
+    ) -> AuditResult<()> {
+        self.runtime_adapter
+            .shutdown(registry.runtime_manager_mut(), replay_workers)
+            .await
+            .map_err(crate::AuditError::Target)
+    }
+
+    pub async fn activate_targets_with_replay(
+        &self,
+        targets: Vec<Box<dyn Target<AuditEntry> + Send + Sync>>,
+    ) -> RuntimeActivation<AuditEntry> {
+        self.runtime_adapter.activate_with_replay(targets).await
+    }
+
+    pub async fn stop_replay_workers(&self) {
+        let mut replay_workers = self.replay_workers.write().await;
+        self.runtime_adapter.stop_replay_workers(&mut replay_workers).await;
+    }
+}

crates/audit/src/registry.rs

@@ -12,27 +12,19 @@
 //  See the License for the specific language governing permissions and
 //  limitations under the License.
 
-use crate::{
-    AuditEntry, AuditError, AuditResult,
-    factory::{MQTTTargetFactory, TargetFactory, WebhookTargetFactory},
-};
-use futures::StreamExt;
-use futures::stream::FuturesUnordered;
-use hashbrown::{HashMap, HashSet};
-use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY, ENV_PREFIX, EnableState, audit::AUDIT_ROUTE_PREFIX};
+use crate::{AuditEntry, AuditError, AuditResult, factory::builtin_target_plugins};
+use rustfs_config::audit::AUDIT_ROUTE_PREFIX;
 use rustfs_ecstore::config::{Config, KVS};
 use rustfs_targets::arn::TargetID;
-use rustfs_targets::{Target, TargetError, target::ChannelTargetType};
-use std::str::FromStr;
-use std::sync::Arc;
-use tracing::{debug, error, info, warn};
+use rustfs_targets::{SharedTarget, Target, TargetError, TargetPluginRegistry, TargetRuntimeManager};
+use tracing::info;
 
 /// Registry for managing audit targets
 pub struct AuditRegistry {
     /// Storage for created targets
-    targets: HashMap<String, Box<dyn Target<AuditEntry> + Send + Sync>>,
-    /// Factories for creating targets
-    factories: HashMap<String, Box<dyn TargetFactory>>,
+    targets: TargetRuntimeManager<AuditEntry>,
+    /// Registered plugins for creating targets
+    plugins: TargetPluginRegistry<AuditEntry>,
 }
 
 impl Default for AuditRegistry {
@@ -44,25 +36,17 @@ impl Default for AuditRegistry {
 impl AuditRegistry {
     /// Creates a new AuditRegistry
     pub fn new() -> Self {
-        let mut registry = AuditRegistry {
-            factories: HashMap::new(),
-            targets: HashMap::new(),
-        };
+        let mut plugins = TargetPluginRegistry::new();
+        plugins.register_all(builtin_target_plugins());
 
-        // Register built-in factories
-        registry.register(ChannelTargetType::Webhook.as_str(), Box::new(WebhookTargetFactory));
-        registry.register(ChannelTargetType::Mqtt.as_str(), Box::new(MQTTTargetFactory));
-
-        registry
+        AuditRegistry {
+            targets: TargetRuntimeManager::new(),
+            plugins,
+        }
     }
 
-    /// Registers a new factory for a target type
-    ///
-    /// # Arguments
-    /// * `target_type` - The type of the target (e.g., "webhook", "mqtt").
-    /// * `factory` - The factory instance to create targets of this type.
-    pub fn register(&mut self, target_type: &str, factory: Box<dyn TargetFactory>) {
-        self.factories.insert(target_type.to_string(), factory);
+    pub fn supports_target_type(&self, target_type: &str) -> bool {
+        self.plugins.supports_target_type(target_type)
     }
 
     /// Creates a target of the specified type with the given ID and configuration
@@ -80,16 +64,7 @@ impl AuditRegistry {
         id: String,
         config: &KVS,
     ) -> Result<Box<dyn Target<AuditEntry> + Send + Sync>, TargetError> {
-        let factory = self
-            .factories
-            .get(target_type)
-            .ok_or_else(|| TargetError::Configuration(format!("Unknown target type: {target_type}")))?;
-
-        // Validate configuration before creating target
-        factory.validate_config(&id, config)?;
-
-        // Create target
-        factory.create_target(id, config).await
+        self.plugins.create_target(target_type, id, config)
     }
 
     /// Creates all targets from a configuration
@@ -105,158 +80,10 @@ impl AuditRegistry {
         &self,
         config: &Config,
     ) -> AuditResult<Vec<Box<dyn Target<AuditEntry> + Send + Sync>>> {
-        // Collect only environment variables with the relevant prefix to reduce memory usage
-        let all_env: Vec<(String, String)> = std::env::vars().filter(|(key, _)| key.starts_with(ENV_PREFIX)).collect();
-        // A collection of asynchronous tasks for concurrently executing target creation
-        let mut tasks = FuturesUnordered::new();
-        // 1. Traverse all registered plants and process them by target type
-        for (target_type, factory) in &self.factories {
-            tracing::Span::current().record("target_type", target_type.as_str());
-            info!("Start working on target types...");
-
-            // 2. Prepare the configuration source
-            // 2.1. Get the configuration segment in the file, e.g. 'audit_webhook'
-            let section_name = format!("{AUDIT_ROUTE_PREFIX}{target_type}").to_lowercase();
-            let file_configs = config.0.get(&section_name).cloned().unwrap_or_default();
-            // 2.2. Get the default configuration for that type
-            let default_cfg = file_configs.get(DEFAULT_DELIMITER).cloned().unwrap_or_default();
-            debug!(?default_cfg, "Get the default configuration");
-
-            // *** Optimization point 1: Get all legitimate fields of the current target type ***
-            let valid_fields = factory.get_valid_fields();
-            debug!(?valid_fields, "Get the legitimate configuration fields");
-
-            // 3. Resolve instance IDs and configuration overrides from environment variables
-            let mut instance_ids_from_env = HashSet::new();
-            // 3.1. Instance discovery: Based on the '..._ENABLE_INSTANCEID' format
-            let enable_prefix =
-                format!("{ENV_PREFIX}{AUDIT_ROUTE_PREFIX}{target_type}{DEFAULT_DELIMITER}{ENABLE_KEY}{DEFAULT_DELIMITER}")
-                    .to_uppercase();
-            for (key, value) in &all_env {
-                if EnableState::from_str(value).ok().map(|s| s.is_enabled()).unwrap_or(false)
-                    && let Some(id) = key.strip_prefix(&enable_prefix)
-                    && !id.is_empty()
-                {
-                    instance_ids_from_env.insert(id.to_lowercase());
-                }
-            }
-
-            // 3.2. Parse all relevant environment variable configurations
-            // 3.2.1. Build environment variable prefixes such as 'RUSTFS_AUDIT_WEBHOOK_'
-            let env_prefix = format!("{ENV_PREFIX}{AUDIT_ROUTE_PREFIX}{target_type}{DEFAULT_DELIMITER}").to_uppercase();
-            // 3.2.2. 'env_overrides' is used to store configurations parsed from environment variables in the format: {instance id -> {field -> value}}
-            let mut env_overrides: HashMap<String, HashMap<String, String>> = HashMap::new();
-            for (key, value) in &all_env {
-                if let Some(rest) = key.strip_prefix(&env_prefix) {
-                    // Use rsplitn to split from the right side to properly extract the INSTANCE_ID at the end
-                    // Format: <FIELD_NAME>_<INSTANCE_ID> or <FIELD_NAME>
-                    let mut parts = rest.rsplitn(2, DEFAULT_DELIMITER);
-
-                    // The first part from the right is INSTANCE_ID
-                    let instance_id_part = parts.next().unwrap_or(DEFAULT_DELIMITER);
-                    // The remaining part is FIELD_NAME
-                    let field_name_part = parts.next();
-
-                    let (field_name, instance_id) = match field_name_part {
-                        // Case 1: The format is <FIELD_NAME>_<INSTANCE_ID>
-                        // e.g., rest = "ENDPOINT_PRIMARY" -> field_name="ENDPOINT", instance_id="PRIMARY"
-                        Some(field) => (field.to_lowercase(), instance_id_part.to_lowercase()),
-                        // Case 2: The format is <FIELD_NAME> (without INSTANCE_ID)
-                        // e.g., rest = "ENABLE" -> field_name="ENABLE", instance_id="" (Universal configuration `_ DEFAULT_DELIMITER`)
-                        None => (instance_id_part.to_lowercase(), DEFAULT_DELIMITER.to_string()),
-                    };
-
-                    // *** Optimization point 2: Verify whether the parsed field_name is legal ***
-                    if !field_name.is_empty() && valid_fields.contains(&field_name) {
-                        debug!(
-                            instance_id = %if instance_id.is_empty() { DEFAULT_DELIMITER } else { &instance_id },
-                            %field_name,
-                            %value,
-                            "Parsing to environment variables"
-                        );
-                        env_overrides
-                            .entry(instance_id)
-                            .or_default()
-                            .insert(field_name, value.clone());
-                    } else {
-                        // Ignore illegal field names
-                        warn!(
-                            field_name = %field_name,
-                            "Ignore environment variable fields, not found in the list of valid fields for target type {}",
-                            target_type
-                        );
-                    }
-                }
-            }
-            debug!(?env_overrides, "Complete the environment variable analysis");
-
-            // 4. Determine all instance IDs that need to be processed
-            let mut all_instance_ids: HashSet<String> =
-                file_configs.keys().filter(|k| *k != DEFAULT_DELIMITER).cloned().collect();
-            all_instance_ids.extend(instance_ids_from_env);
-            debug!(?all_instance_ids, "Determine all instance IDs");
-
-            // 5. Merge configurations and create tasks for each instance
-            for id in all_instance_ids {
-                // 5.1. Merge configuration, priority: Environment variables > File instance configuration > File default configuration
-                let mut merged_config = default_cfg.clone();
-                // Instance-specific configuration in application files
-                if let Some(file_instance_cfg) = file_configs.get(&id) {
-                    merged_config.extend(file_instance_cfg.clone());
-                }
-                // Application instance-specific environment variable configuration
-                if let Some(env_instance_cfg) = env_overrides.get(&id) {
-                    // Convert HashMap<String, String> to KVS
-                    let mut kvs_from_env = KVS::new();
-                    for (k, v) in env_instance_cfg {
-                        kvs_from_env.insert(k.clone(), v.clone());
-                    }
-                    merged_config.extend(kvs_from_env);
-                }
-                debug!(instance_id = %id, ?merged_config, "Complete configuration merge");
-
-                // 5.2. Check if the instance is enabled
-                let enabled = merged_config
-                    .lookup(ENABLE_KEY)
-                    .map(|v| {
-                        EnableState::from_str(v.as_str())
-                            .ok()
-                            .map(|s| s.is_enabled())
-                            .unwrap_or(false)
-                    })
-                    .unwrap_or(false);
-
-                if enabled {
-                    info!(instance_id = %id, "Target is enabled, ready to create a task");
-                    // 5.3. Create asynchronous tasks for enabled instances
-                    let tid = id.clone();
-                    let merged_config_arc = Arc::new(merged_config);
-                    tasks.push(async move {
-                        let result = factory.create_target(tid.clone(), &merged_config_arc).await;
-                        (tid, result)
-                    });
-                } else {
-                    info!(instance_id = %id, "Skip disabled target");
-                }
-            }
-        }
-
-        // 6. Concurrently execute all creation tasks and collect results
-        let mut successful_targets = Vec::new();
-        while let Some((id, result)) = tasks.next().await {
-            match result {
-                Ok(target) => {
-                    info!(target_type = %target.id().name, instance_id = %id, "Create a target successfully");
-                    successful_targets.push(target);
-                }
-                Err(e) => {
-                    error!(instance_id = %id, error = %e, "Failed to create a target");
-                }
-            }
-        }
-
-        info!(count = successful_targets.len(), "All target processing completed");
-        Ok(successful_targets)
+        self.plugins
+            .create_targets_from_config(config, AUDIT_ROUTE_PREFIX)
+            .await
+            .map_err(AuditError::from)
     }
 
     /// Adds a target to the registry
@@ -264,8 +91,14 @@ impl AuditRegistry {
     /// # Arguments
     /// * `id` - The identifier for the target.
     /// * `target` - The target instance to be added.
-    pub fn add_target(&mut self, id: String, target: Box<dyn Target<AuditEntry> + Send + Sync>) {
-        self.targets.insert(id, target);
+    pub fn add_target(&mut self, _id: String, target: Box<dyn Target<AuditEntry> + Send + Sync>) {
+        debug_assert_eq!(_id, target.id().to_string());
+        self.targets.add_boxed(target);
+    }
+
+    pub fn add_shared_target(&mut self, _id: String, target: SharedTarget<AuditEntry>) {
+        debug_assert_eq!(_id, target.id().to_string());
+        self.targets.add_arc(target);
     }
 
     /// Removes a target from the registry
@@ -275,8 +108,8 @@ impl AuditRegistry {
     ///
     /// # Returns
     /// * `Option<Box<dyn Target<AuditEntry> + Send + Sync>>` - The removed target if it existed.
-    pub fn remove_target(&mut self, id: &str) -> Option<Box<dyn Target<AuditEntry> + Send + Sync>> {
-        self.targets.remove(id)
+    pub async fn remove_target(&mut self, id: &str) -> Option<rustfs_targets::SharedTarget<AuditEntry>> {
+        self.targets.remove_and_close(id).await
     }
 
     /// Gets a target from the registry
@@ -286,13 +119,21 @@ impl AuditRegistry {
     ///
     /// # Returns
     /// * `Option<&(dyn Target<AuditEntry> + Send + Sync)>` - The target if it exists.
-    pub fn get_target(&self, id: &str) -> Option<&(dyn Target<AuditEntry> + Send + Sync)> {
-        self.targets.get(id).map(|t| t.as_ref())
+    pub fn get_target(&self, id: &str) -> Option<rustfs_targets::SharedTarget<AuditEntry>> {
+        self.targets.get(id)
     }
 
     /// Lists cloned target values for runtime inspection without exposing mutable registry access.
-    pub fn list_target_values(&self) -> Vec<Box<dyn Target<AuditEntry> + Send + Sync>> {
-        self.targets.values().map(|target| target.clone_dyn()).collect()
+    pub fn list_target_values(&self) -> Vec<rustfs_targets::SharedTarget<AuditEntry>> {
+        self.targets.values()
+    }
+
+    pub fn runtime_manager(&self) -> &TargetRuntimeManager<AuditEntry> {
+        &self.targets
+    }
+
+    pub fn runtime_manager_mut(&mut self) -> &mut TargetRuntimeManager<AuditEntry> {
+        &mut self.targets
     }
 
     /// Lists all target IDs
@@ -300,7 +141,7 @@ impl AuditRegistry {
     /// # Returns
     /// * `Vec<String>` - A vector of all target IDs in the registry.
     pub fn list_targets(&self) -> Vec<String> {
-        self.targets.keys().cloned().collect()
+        self.targets.keys()
     }
 
     /// Closes all targets and clears the registry
@@ -308,20 +149,23 @@ impl AuditRegistry {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure.
     pub async fn close_all(&mut self) -> AuditResult<()> {
-        let mut errors = Vec::new();
+        let mut first_error = None;
 
-        for (id, target) in self.targets.drain() {
-            if let Err(e) = target.close().await {
-                error!(target_id = %id, error = %e, "Failed to close audit target");
-                errors.push(e);
+        for target_id in self.targets.keys() {
+            if let Some(target) = self.targets.remove(&target_id)
+                && let Err(err) = target.close().await
+            {
+                tracing::error!(target_id = %target_id, error = %err, "Failed to close target during shutdown");
+                if first_error.is_none() {
+                    first_error = Some(err);
+                }
             }
         }
 
-        if !errors.is_empty() {
-            return Err(AuditError::Target(errors.into_iter().next().unwrap()));
+        match first_error {
+            Some(err) => Err(AuditError::Target(err)),
+            None => Ok(()),
         }
-
-        Ok(())
     }
 
     /// Creates a unique key for a target based on its type and ID
@@ -396,7 +240,107 @@ impl AuditRegistry {
         target: Box<dyn Target<AuditEntry> + Send + Sync>,
     ) -> AuditResult<()> {
         let key = self.create_key(target_type, target_id);
-        self.targets.insert(key, target);
+        debug_assert_eq!(key, target.id().to_string());
+        self.targets.add_boxed(target);
         Ok(())
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::AuditRegistry;
+    use crate::{AuditEntry, AuditError};
+    use rustfs_targets::arn::TargetID;
+    use rustfs_targets::store::{Key, Store};
+    use rustfs_targets::target::{ChannelTargetType, EntityTarget, QueuedPayload, QueuedPayloadMeta};
+    use rustfs_targets::{StoreError, Target, TargetError};
+    use std::sync::Arc;
+    use std::sync::atomic::{AtomicUsize, Ordering};
+
+    #[derive(Clone)]
+    struct CloseTestTarget {
+        id: TargetID,
+        close_calls: Arc<AtomicUsize>,
+        fail_on_close: bool,
+    }
+
+    impl CloseTestTarget {
+        fn new(id: TargetID, close_calls: Arc<AtomicUsize>, fail_on_close: bool) -> Self {
+            Self {
+                id,
+                close_calls,
+                fail_on_close,
+            }
+        }
+    }
+
+    #[async_trait::async_trait]
+    impl Target<AuditEntry> for CloseTestTarget {
+        fn id(&self) -> TargetID {
+            self.id.clone()
+        }
+
+        async fn is_active(&self) -> Result<bool, TargetError> {
+            Ok(true)
+        }
+
+        async fn save(&self, _event: Arc<EntityTarget<AuditEntry>>) -> Result<(), TargetError> {
+            Ok(())
+        }
+
+        async fn send_raw_from_store(&self, _key: Key, _body: Vec<u8>, _meta: QueuedPayloadMeta) -> Result<(), TargetError> {
+            Ok(())
+        }
+
+        async fn close(&self) -> Result<(), TargetError> {
+            self.close_calls.fetch_add(1, Ordering::SeqCst);
+            if self.fail_on_close {
+                Err(TargetError::Unknown("close failed".to_string()))
+            } else {
+                Ok(())
+            }
+        }
+
+        fn store(&self) -> Option<&(dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send + Sync)> {
+            None
+        }
+
+        fn clone_dyn(&self) -> Box<dyn Target<AuditEntry> + Send + Sync> {
+            Box::new(self.clone())
+        }
+
+        fn is_enabled(&self) -> bool {
+            true
+        }
+    }
+
+    #[test]
+    fn registry_registers_amqp_factory() {
+        let registry = AuditRegistry::new();
+
+        assert!(registry.supports_target_type(ChannelTargetType::Amqp.as_str()));
+    }
+
+    #[tokio::test]
+    async fn close_all_returns_first_error_and_clears_targets() {
+        let mut registry = AuditRegistry::new();
+        let ok_calls = Arc::new(AtomicUsize::new(0));
+        let fail_calls = Arc::new(AtomicUsize::new(0));
+
+        let ok_id = TargetID::new("ok".to_string(), "webhook".to_string());
+        let fail_id = TargetID::new("fail".to_string(), "webhook".to_string());
+
+        registry.add_target(ok_id.to_string(), Box::new(CloseTestTarget::new(ok_id, Arc::clone(&ok_calls), false)));
+        registry.add_target(
+            fail_id.to_string(),
+            Box::new(CloseTestTarget::new(fail_id, Arc::clone(&fail_calls), true)),
+        );
+
+        let result = registry.close_all().await;
+
+        assert!(matches!(result, Err(AuditError::Target(TargetError::Unknown(_)))));
+        assert_eq!(ok_calls.load(Ordering::SeqCst), 1);
+        assert_eq!(fail_calls.load(Ordering::SeqCst), 1);
+        assert!(registry.list_targets().is_empty());
+    }
+}

crates/audit/src/system.rs

@@ -12,18 +12,24 @@
 //  See the License for the specific language governing permissions and
 //  limitations under the License.
 
-use crate::{AuditEntry, AuditError, AuditRegistry, AuditResult, observability};
-use hashbrown::HashMap;
-use rustfs_ecstore::config::Config;
-use rustfs_targets::{
-    StoreError, Target, TargetError,
-    store::{Key, Store},
-    target::{EntityTarget, QueuedPayload},
+use crate::{
+    AuditEntry, AuditError, AuditRegistry, AuditResult, observability,
+    pipeline::{AuditPipeline, AuditRuntimeFacade, AuditRuntimeView},
 };
+use rustfs_ecstore::config::Config;
+use rustfs_targets::{ReplayWorkerManager, Target};
 use std::sync::Arc;
-use tokio::sync::{Mutex, RwLock, mpsc};
+use tokio::sync::{Mutex, RwLock};
 use tracing::{error, info, warn};
 
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+pub struct AuditTargetMetricSnapshot {
+    pub failed_messages: u64,
+    pub queue_length: u64,
+    pub target_id: String,
+    pub total_messages: u64,
+}
+
 /// State of the audit system
 #[derive(Debug, Clone, PartialEq, Eq)]
 pub enum AuditSystemState {
@@ -41,7 +47,7 @@ pub struct AuditSystem {
     state: Arc<RwLock<AuditSystemState>>,
     config: Arc<RwLock<Option<Config>>>,
     /// Cancellation senders for active audit stream tasks (target_id -> cancel tx)
-    stream_cancellers: Arc<RwLock<HashMap<String, mpsc::Sender<()>>>>,
+    stream_cancellers: Arc<RwLock<ReplayWorkerManager>>,
 }
 
 impl Default for AuditSystem {
@@ -51,16 +57,68 @@ impl Default for AuditSystem {
 }
 
 impl AuditSystem {
+    fn pipeline(&self) -> AuditPipeline {
+        AuditPipeline::new(self.registry.clone())
+    }
+
+    fn runtime_view(&self) -> AuditRuntimeView {
+        AuditRuntimeView::new(self.registry.clone())
+    }
+
+    fn runtime_facade(&self) -> AuditRuntimeFacade {
+        AuditRuntimeFacade::new(self.registry.clone(), self.stream_cancellers.clone())
+    }
+
     /// Creates a new audit system
     pub fn new() -> Self {
         Self {
             registry: Arc::new(Mutex::new(AuditRegistry::new())),
             state: Arc::new(RwLock::new(AuditSystemState::Stopped)),
             config: Arc::new(RwLock::new(None)),
-            stream_cancellers: Arc::new(RwLock::new(HashMap::new())),
+            stream_cancellers: Arc::new(RwLock::new(ReplayWorkerManager::new())),
         }
     }
 
+    async fn create_targets_from_config(&self, config: &Config) -> AuditResult<Vec<Box<dyn Target<AuditEntry> + Send + Sync>>> {
+        let registry = self.registry.lock().await;
+        registry.create_audit_targets_from_config(config).await
+    }
+
+    async fn clear_runtime_targets(&self) -> AuditResult<()> {
+        {
+            let mut registry = self.registry.lock().await;
+            let mut replay_workers = self.stream_cancellers.write().await;
+            self.runtime_facade()
+                .shutdown_runtime(&mut registry, &mut replay_workers)
+                .await?;
+        }
+
+        let mut state = self.state.write().await;
+        *state = AuditSystemState::Stopped;
+        Ok(())
+    }
+
+    async fn commit_runtime_targets(
+        &self,
+        targets: Vec<Box<dyn Target<AuditEntry> + Send + Sync>>,
+        final_state: AuditSystemState,
+    ) -> AuditResult<()> {
+        if targets.is_empty() {
+            info!("No enabled audit targets found, keeping audit system stopped");
+            self.clear_runtime_targets().await?;
+            return Ok(());
+        }
+
+        info!(target_count = targets.len(), "Created audit targets successfully");
+
+        let activation = self.runtime_facade().activate_targets_with_replay(targets).await;
+        self.runtime_facade().replace_targets(activation).await?;
+
+        let mut state = self.state.write().await;
+        *state = final_state;
+        Ok(())
+    }
+
     /// Starts the audit system with the given configuration
     ///
     /// # Arguments
@@ -95,31 +153,14 @@ impl AuditSystem {
             *config_guard = Some(config.clone());
         }
 
-        // Create targets from configuration
-        let mut registry = self.registry.lock().await;
-        match registry.create_audit_targets_from_config(&config).await {
+        match self.create_targets_from_config(&config).await {
             Ok(targets) => {
-                if targets.is_empty() {
-                    info!("No enabled audit targets found, keeping audit system stopped");
-                    drop(registry);
-                    return Ok(());
-                }
-
                 {
                     let mut state = self.state.write().await;
                     *state = AuditSystemState::Starting;
                 }
 
-                info!(target_count = targets.len(), "Created audit targets successfully");
-
-                // Initialize all targets
-                for target in targets {
-                    self.init_and_register_target(target, &mut registry).await;
-                }
-
-                // Update state to running
-                let mut state = self.state.write().await;
-                *state = AuditSystemState::Running;
+                self.commit_runtime_targets(targets, AuditSystemState::Running).await?;
                 info!("Audit system started successfully");
                 Ok(())
             }
@@ -199,18 +240,10 @@ impl AuditSystem {
         info!("Stopping audit system");
 
         // Stop all stream tasks first
-        self.stop_all_streams().await;
-
-        // Close all targets
-        let mut registry = self.registry.lock().await;
-        if let Err(e) = registry.close_all().await {
+        if let Err(e) = self.clear_runtime_targets().await {
             error!(error = %e, "Failed to close some audit targets");
         }
 
-        // Update state to stopped
-        let mut state = self.state.write().await;
-        *state = AuditSystemState::Stopped;
-
         // Clear configuration
         let mut config_guard = self.config.write().await;
         *config_guard = None;
@@ -240,8 +273,6 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn dispatch(&self, entry: Arc<AuditEntry>) -> AuditResult<()> {
-        let start_time = std::time::Instant::now();
-
         let state = self.state.read().await;
 
         match *state {
@@ -254,77 +285,7 @@ impl AuditSystem {
             }
         }
         drop(state);
-
-        // Collect cloned targets under lock, then dispatch without holding it
-        let targets: Vec<(String, Box<dyn Target<AuditEntry> + Send + Sync>)> = {
-            let registry = self.registry.lock().await;
-            let target_keys = registry.list_targets();
-
-            if target_keys.is_empty() {
-                warn!("No audit targets configured for dispatch");
-                return Ok(());
-            }
-
-            target_keys
-                .into_iter()
-                .filter_map(|key| registry.get_target(&key).map(|t| (key, t.clone_dyn())))
-                .collect()
-        };
-
-        // Dispatch to all targets concurrently (no lock held)
-        let mut tasks = Vec::new();
-
-        for (target_key, target) in targets {
-            let entity_target = EntityTarget {
-                object_name: entry.api.name.clone().unwrap_or_default(),
-                bucket_name: entry.api.bucket.clone().unwrap_or_default(),
-                event_name: entry.event,
-                data: (*entry).clone(),
-            };
-
-            let task = async move {
-                let result = target.save(Arc::new(entity_target)).await;
-                (target_key, result)
-            };
-
-            tasks.push(task);
-        }
-
-        // Execute all dispatch tasks
-        let results = futures::future::join_all(tasks).await;
-
-        let mut errors = Vec::new();
-        let mut success_count = 0;
-
-        for (target_key, result) in results {
-            match result {
-                Ok(_) => {
-                    success_count += 1;
-                    observability::record_target_success();
-                }
-                Err(e) => {
-                    error!(target_id = %target_key, error = %e, "Failed to dispatch audit log to target");
-                    errors.push(e);
-                    observability::record_target_failure();
-                }
-            }
-        }
-
-        let dispatch_time = start_time.elapsed();
-
-        if errors.is_empty() {
-            observability::record_audit_success(dispatch_time);
-        } else {
-            observability::record_audit_failure(dispatch_time);
-            // Log errors but don't fail the entire dispatch
-            warn!(
-                error_count = errors.len(),
-                success_count = success_count,
-                "Some audit targets failed to receive log entry"
-            );
-        }
-
-        Ok(())
+        self.pipeline().dispatch(entry).await
     }
 
     /// Dispatches a batch of audit log entries to all active targets
@@ -335,232 +296,12 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn dispatch_batch(&self, entries: Vec<Arc<AuditEntry>>) -> AuditResult<()> {
-        let start_time = std::time::Instant::now();
-
         let state = self.state.read().await;
         if *state != AuditSystemState::Running {
             return Err(AuditError::NotInitialized("Audit system is not running".to_string()));
         }
         drop(state);
-
-        // Collect targets under lock, then dispatch without holding it
-        let targets: Vec<(String, Box<dyn Target<AuditEntry> + Send + Sync>)> = {
-            let registry = self.registry.lock().await;
-            let target_keys = registry.list_targets();
-
-            if target_keys.is_empty() {
-                warn!("No audit targets configured for batch dispatch");
-                return Ok(());
-            }
-
-            target_keys
-                .into_iter()
-                .filter_map(|key| registry.get_target(&key).map(|t| (key, t.clone_dyn())))
-                .collect()
-        };
-
-        let mut tasks = Vec::new();
-        for (target_key, target) in targets {
-            let entries_clone: Vec<_> = entries.iter().map(Arc::clone).collect();
-            let target_key_clone = target_key.clone();
-
-            let task = async move {
-                let mut success_count = 0;
-                let mut errors = Vec::new();
-                for entry in entries_clone {
-                    let entity_target = EntityTarget {
-                        object_name: entry.api.name.clone().unwrap_or_default(),
-                        bucket_name: entry.api.bucket.clone().unwrap_or_default(),
-                        event_name: entry.event,
-                        data: (*entry).clone(),
-                    };
-                    match target.save(Arc::new(entity_target)).await {
-                        Ok(_) => success_count += 1,
-                        Err(e) => errors.push(e),
-                    }
-                }
-                (target_key_clone, success_count, errors)
-            };
-            tasks.push(task);
-        }
-
-        let results = futures::future::join_all(tasks).await;
-        let mut total_success = 0;
-        let mut total_errors = 0;
-        for (_target_id, success_count, errors) in results {
-            total_success += success_count;
-            total_errors += errors.len();
-            for e in errors {
-                error!("Batch dispatch error: {:?}", e);
-            }
-        }
-
-        let dispatch_time = start_time.elapsed();
-        info!(
-            "Batch dispatched {} entries, success: {}, errors: {}, time: {:?}",
-            entries.len(),
-            total_success,
-            total_errors,
-            dispatch_time
-        );
-
-        Ok(())
-    }
-
-    /// Stops all active audit stream tasks by sending cancellation signals.
-    async fn stop_all_streams(&self) {
-        let mut cancellers = self.stream_cancellers.write().await;
-        for (target_id, cancel_tx) in cancellers.drain() {
-            info!(target_id = %target_id, "Stopping audit stream");
-            let _ = cancel_tx.send(()).await;
-        }
-    }
-
-    /// Initializes a single target: runs init(), starts stream if store is present,
-    /// and adds it to the registry. For store-backed targets, registration and stream
-    /// startup proceed even if init() fails so queued entries can be drained later.
-    async fn init_and_register_target(
-        &self,
-        target: Box<dyn Target<AuditEntry> + Send + Sync>,
-        registry: &mut AuditRegistry,
-    ) -> Option<String> {
-        let target_id = target.id().to_string();
-        let has_store = target.store().is_some();
-
-        if let Err(e) = target.init().await {
-            error!(target_id = %target_id, error = %e, "Failed to initialize audit target");
-            // Non-store targets: init failure is fatal.
-            if !has_store {
-                return None;
-            }
-            // Store-backed targets: still register and start the stream so queued
-            // entries can be drained when connectivity recovers.
-            warn!(
-                target_id = %target_id,
-                "Proceeding with store-backed audit target despite init failure"
-            );
-        }
-
-        if target.is_enabled() {
-            if let Some(store) = target.store() {
-                info!(target_id = %target_id, "Start audit stream processing for target");
-                let store_clone: Box<dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send> = store.boxed_clone();
-                let target_arc: Arc<dyn Target<AuditEntry> + Send + Sync> = Arc::from(target.clone_dyn());
-                let cancel_tx = self.start_audit_stream_with_batching(store_clone, target_arc);
-
-                self.stream_cancellers.write().await.insert(target_id.clone(), cancel_tx);
-                info!(target_id = %target_id, "Audit stream processing started");
-            } else {
-                info!(target_id = %target_id, "No store configured, skip audit stream processing");
-            }
-        } else {
-            info!(target_id = %target_id, "Target disabled, skip audit stream processing");
-        }
-
-        registry.add_target(target_id.clone(), target);
-        Some(target_id)
-    }
-
-    /// Starts the audit stream processing for a target with batching and retry logic
-    ///
-    /// # Arguments
-    /// * `store` - The store from which to read audit entries
-    /// * `target` - The target to which audit entries will be sent
-    ///
-    /// This function spawns a background task that continuously reads audit entries from the provided store
-    /// and attempts to send them to the specified target. It implements retry logic with exponential backoff
-    fn start_audit_stream_with_batching(
-        &self,
-        store: Box<dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send>,
-        target: Arc<dyn Target<AuditEntry> + Send + Sync>,
-    ) -> mpsc::Sender<()> {
-        let (cancel_tx, mut cancel_rx) = mpsc::channel(1);
-        let state = self.state.clone();
-
-        tokio::spawn(async move {
-            use std::time::Duration;
-            use tokio::time::sleep;
-
-            info!("Starting audit stream for target: {}", target.id());
-
-            const MAX_RETRIES: usize = 5;
-            const BASE_RETRY_DELAY: Duration = Duration::from_secs(2);
-
-            loop {
-                // Check for cancellation signal
-                if cancel_rx.try_recv().is_ok() {
-                    info!("Audit stream cancelled for target: {}", target.id());
-                    break;
-                }
-
-                match *state.read().await {
-                    AuditSystemState::Running | AuditSystemState::Paused | AuditSystemState::Starting => {}
-                    _ => {
-                        info!("Audit stream stopped for target: {}", target.id());
-                        break;
-                    }
-                }
-
-                let keys: Vec<Key> = store.list();
-                if keys.is_empty() {
-                    tokio::select! {
-                        _ = sleep(Duration::from_millis(500)) => {},
-                        _ = cancel_rx.recv() => {
-                            info!("Audit stream cancelled during idle for target: {}", target.id());
-                            return;
-                        }
-                    }
-                    continue;
-                }
-
-                for key in keys {
-                    if cancel_rx.try_recv().is_ok() {
-                        info!("Audit stream cancelled during processing for target: {}", target.id());
-                        return;
-                    }
-
-                    let mut retries = 0usize;
-                    let mut success = false;
-
-                    while retries < MAX_RETRIES && !success {
-                        match target.send_from_store(key.clone()).await {
-                            Ok(_) => {
-                                info!("Successfully sent audit entry, target: {}, key: {}", target.id(), key.to_string());
-                                observability::record_target_success();
-                                success = true;
-                            }
-                            Err(e) => {
-                                match &e {
-                                    TargetError::NotConnected => {
-                                        warn!("Target {} not connected, retrying...", target.id());
-                                    }
-                                    TargetError::Timeout(_) => {
-                                        warn!("Timeout sending to target {}, retrying...", target.id());
-                                    }
-                                    _ => {
-                                        error!("Permanent error for target {}: {}", target.id(), e);
-                                        observability::record_target_failure();
-                                        break;
-                                    }
-                                }
-                                retries += 1;
-                                let backoff = BASE_RETRY_DELAY * (1 << retries);
-                                sleep(backoff).await;
-                            }
-                        }
-                    }
-
-                    if retries >= MAX_RETRIES && !success {
-                        warn!("Max retries exceeded for key {}, target: {}, skipping", key.to_string(), target.id());
-                        observability::record_target_failure();
-                    }
-                }
-
-                sleep(Duration::from_millis(100)).await;
-            }
-        });
-
-        cancel_tx
+        self.pipeline().dispatch_batch(entries).await
     }
 
     /// Enables a specific target
@@ -571,15 +312,7 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn enable_target(&self, target_id: &str) -> AuditResult<()> {
-        // This would require storing enabled/disabled state per target
-        // For now, just check if target exists
-        let registry = self.registry.lock().await;
-        if registry.get_target(target_id).is_some() {
-            info!(target_id = %target_id, "Target enabled");
-            Ok(())
-        } else {
-            Err(AuditError::Configuration(format!("Target not found: {target_id}"), None))
-        }
+        self.runtime_view().enable_target(target_id).await
     }
 
     /// Disables a specific target
@@ -590,15 +323,7 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn disable_target(&self, target_id: &str) -> AuditResult<()> {
-        // This would require storing enabled/disabled state per target
-        // For now, just check if target exists
-        let registry = self.registry.lock().await;
-        if registry.get_target(target_id).is_some() {
-            info!(target_id = %target_id, "Target disabled");
-            Ok(())
-        } else {
-            Err(AuditError::Configuration(format!("Target not found: {target_id}"), None))
-        }
+        self.runtime_view().disable_target(target_id).await
     }
 
     /// Removes a target from the system
@@ -609,16 +334,7 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn remove_target(&self, target_id: &str) -> AuditResult<()> {
-        let mut registry = self.registry.lock().await;
-        if let Some(target) = registry.remove_target(target_id) {
-            if let Err(e) = target.close().await {
-                error!(target_id = %target_id, error = %e, "Failed to close removed target");
-            }
-            info!(target_id = %target_id, "Target removed");
-            Ok(())
-        } else {
-            Err(AuditError::Configuration(format!("Target not found: {target_id}"), None))
-        }
+        self.runtime_view().remove_target(target_id).await
     }
 
     /// Updates or inserts a target
@@ -630,23 +346,7 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn upsert_target(&self, target_id: String, target: Box<dyn Target<AuditEntry> + Send + Sync>) -> AuditResult<()> {
-        let mut registry = self.registry.lock().await;
-
-        // Initialize the target
-        if let Err(e) = target.init().await {
-            return Err(AuditError::Target(e));
-        }
-
-        // Remove existing target if present
-        if let Some(old_target) = registry.remove_target(&target_id)
-            && let Err(e) = old_target.close().await
-        {
-            error!(target_id = %target_id, error = %e, "Failed to close old target during upsert");
-        }
-
-        registry.add_target(target_id.clone(), target);
-        info!(target_id = %target_id, "Target upserted");
-        Ok(())
+        self.runtime_view().upsert_target(target_id, target).await
     }
 
     /// Lists all targets
@@ -654,14 +354,27 @@ impl AuditSystem {
     /// # Returns
     /// * `Vec<String>` - List of target IDs
     pub async fn list_targets(&self) -> Vec<String> {
-        let registry = self.registry.lock().await;
-        registry.list_targets()
+        self.runtime_view().list_targets().await
     }
 
     /// Returns cloned target values for read-only runtime inspection.
-    pub async fn get_target_values(&self) -> Vec<Box<dyn Target<AuditEntry> + Send + Sync>> {
+    pub async fn get_target_values(&self) -> Vec<rustfs_targets::SharedTarget<AuditEntry>> {
+        self.runtime_view().get_target_values().await
+    }
+
+    /// Returns per-target delivery metrics for Prometheus collection.
+    pub async fn snapshot_target_metrics(&self) -> Vec<AuditTargetMetricSnapshot> {
+        self.pipeline().snapshot_target_metrics().await
+    }
+
+    pub async fn snapshot_target_health(&self) -> Vec<rustfs_targets::RuntimeTargetHealthSnapshot> {
+        self.pipeline().snapshot_target_health().await
+    }
+
+    pub async fn runtime_status_snapshot(&self) -> rustfs_targets::RuntimeStatusSnapshot {
+        let replay_workers = self.stream_cancellers.read().await;
         let registry = self.registry.lock().await;
-        registry.list_target_values()
+        registry.runtime_manager().status_snapshot(&replay_workers)
     }
 
     /// Gets information about a specific target
@@ -672,8 +385,7 @@ impl AuditSystem {
     /// # Returns
     /// * `Option<String>` - Target ID if found
     pub async fn get_target(&self, target_id: &str) -> Option<String> {
-        let registry = self.registry.lock().await;
-        registry.get_target(target_id).map(|target| target.id().to_string())
+        self.runtime_view().get_target(target_id).await
     }
 
     /// Reloads configuration and updates targets
@@ -688,32 +400,20 @@ impl AuditSystem {
 
         observability::record_config_reload();
 
-        // Stop all existing stream tasks first
-        self.stop_all_streams().await;
-
         // Store new configuration
         {
             let mut config_guard = self.config.write().await;
             *config_guard = Some(new_config.clone());
         }
 
-        // Close all existing targets
-        let mut registry = self.registry.lock().await;
-        if let Err(e) = registry.close_all().await {
-            error!(error = %e, "Failed to close existing targets during reload");
-        }
+        let final_state = match self.get_state().await {
+            AuditSystemState::Paused => AuditSystemState::Paused,
+            _ => AuditSystemState::Running,
+        };
 
-        // Create new targets from updated configuration
-        match registry.create_audit_targets_from_config(&new_config).await {
+        match self.create_targets_from_config(&new_config).await {
             Ok(targets) => {
-                info!(target_count = targets.len(), "Reloaded audit targets successfully");
-
-                for target in targets {
-                    if let Some(target_id) = self.init_and_register_target(target, &mut registry).await {
-                        info!(target_id = %target_id, "Target initialized (reload)");
-                    }
-                }
-
+                self.commit_runtime_targets(targets, final_state).await?;
                 info!("Audit configuration reloaded successfully");
                 Ok(())
             }
@@ -745,3 +445,106 @@ impl AuditSystem {
         observability::reset_metrics().await;
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::{AuditSystem, AuditSystemState};
+    use async_trait::async_trait;
+    use rustfs_targets::ReplayWorkerManager;
+    use rustfs_targets::arn::TargetID;
+    use rustfs_targets::store::{Key, Store};
+    use rustfs_targets::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta};
+    use rustfs_targets::{StoreError, Target, TargetError};
+    use serde::{Serialize, de::DeserializeOwned};
+    use std::collections::HashMap;
+    use std::sync::Arc;
+    use std::sync::atomic::{AtomicUsize, Ordering};
+    use tokio::sync::mpsc;
+
+    #[derive(Clone)]
+    struct TestTarget {
+        close_calls: Arc<AtomicUsize>,
+        id: TargetID,
+    }
+
+    impl TestTarget {
+        fn new(id: &str, name: &str) -> Self {
+            Self {
+                close_calls: Arc::new(AtomicUsize::new(0)),
+                id: TargetID::new(id.to_string(), name.to_string()),
+            }
+        }
+    }
+
+    #[async_trait]
+    impl<E> Target<E> for TestTarget
+    where
+        E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned,
+    {
+        fn id(&self) -> TargetID {
+            self.id.clone()
+        }
+
+        async fn is_active(&self) -> Result<bool, TargetError> {
+            Ok(true)
+        }
+
+        async fn save(&self, _event: Arc<EntityTarget<E>>) -> Result<(), TargetError> {
+            Ok(())
+        }
+
+        async fn send_raw_from_store(&self, _key: Key, _body: Vec<u8>, _meta: QueuedPayloadMeta) -> Result<(), TargetError> {
+            Ok(())
+        }
+
+        async fn close(&self) -> Result<(), TargetError> {
+            self.close_calls.fetch_add(1, Ordering::SeqCst);
+            Ok(())
+        }
+
+        fn store(&self) -> Option<&(dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send + Sync)> {
+            None
+        }
+
+        fn clone_dyn(&self) -> Box<dyn Target<E> + Send + Sync> {
+            Box::new(self.clone())
+        }
+
+        fn is_enabled(&self) -> bool {
+            true
+        }
+    }
+
+    #[tokio::test]
+    async fn reload_with_empty_config_stops_existing_runtime() {
+        let system = AuditSystem::new();
+        let target = TestTarget::new("primary", "webhook");
+        let close_calls = Arc::clone(&target.close_calls);
+
+        {
+            let mut registry = system.registry.lock().await;
+            registry.add_target("primary:webhook".to_string(), Box::new(target));
+        }
+        {
+            let mut state = system.state.write().await;
+            *state = AuditSystemState::Running;
+        }
+        {
+            let mut replay_workers = system.stream_cancellers.write().await;
+            let (cancel_tx, _cancel_rx) = mpsc::channel(1);
+            replay_workers.insert("primary:webhook".to_string(), cancel_tx);
+            assert_eq!(replay_workers.len(), 1);
+        }
+
+        system
+            .reload_config(rustfs_ecstore::config::Config(HashMap::new()))
+            .await
+            .expect("reload with empty config should succeed");
+
+        assert_eq!(system.get_state().await, AuditSystemState::Stopped);
+        assert!(system.list_targets().await.is_empty());
+        assert_eq!(system.runtime_status_snapshot().await, ReplayWorkerManager::new().snapshot(0));
+        assert_eq!(close_calls.load(Ordering::SeqCst), 1);
+        assert_eq!(*system.config.read().await, Some(rustfs_ecstore::config::Config(HashMap::new())));
+    }
+}

crates/audit/tests/pipeline_layer_test.rs

@@ -0,0 +1,170 @@
+//  Copyright 2024 RustFS Team
+//
+//  Licensed under the Apache License, Version 2.0 (the "License");
+//  you may not use this file except in compliance with the License.
+//  You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+//  Unless required by applicable law or agreed to in writing, software
+//  distributed under the License is distributed on an "AS IS" BASIS,
+//  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+//  See the License for the specific language governing permissions and
+//  limitations under the License.
+
+use async_trait::async_trait;
+use rustfs_audit::{AuditPipeline, AuditRegistry, AuditRuntimeFacade, AuditRuntimeView};
+use rustfs_targets::arn::TargetID;
+use rustfs_targets::store::{Key, Store};
+use rustfs_targets::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta};
+use rustfs_targets::{StoreError, Target, TargetError};
+use serde::{Serialize, de::DeserializeOwned};
+use std::sync::Arc;
+use std::sync::atomic::{AtomicUsize, Ordering};
+use tokio::sync::{Mutex, RwLock};
+
+#[derive(Clone)]
+struct TestTarget {
+    close_calls: Arc<AtomicUsize>,
+    id: TargetID,
+    init_calls: Arc<AtomicUsize>,
+}
+
+impl TestTarget {
+    fn new(id: &str, name: &str) -> Self {
+        Self {
+            close_calls: Arc::new(AtomicUsize::new(0)),
+            id: TargetID::new(id.to_string(), name.to_string()),
+            init_calls: Arc::new(AtomicUsize::new(0)),
+        }
+    }
+}
+
+#[async_trait]
+impl<E> Target<E> for TestTarget
+where
+    E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned,
+{
+    fn id(&self) -> TargetID {
+        self.id.clone()
+    }
+
+    async fn is_active(&self) -> Result<bool, TargetError> {
+        Ok(true)
+    }
+
+    async fn save(&self, _event: Arc<EntityTarget<E>>) -> Result<(), TargetError> {
+        Ok(())
+    }
+
+    async fn send_raw_from_store(&self, _key: Key, _body: Vec<u8>, _meta: QueuedPayloadMeta) -> Result<(), TargetError> {
+        Ok(())
+    }
+
+    async fn close(&self) -> Result<(), TargetError> {
+        self.close_calls.fetch_add(1, Ordering::SeqCst);
+        Ok(())
+    }
+
+    fn store(&self) -> Option<&(dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send + Sync)> {
+        None
+    }
+
+    fn clone_dyn(&self) -> Box<dyn Target<E> + Send + Sync> {
+        Box::new(self.clone())
+    }
+
+    async fn init(&self) -> Result<(), TargetError> {
+        self.init_calls.fetch_add(1, Ordering::SeqCst);
+        Ok(())
+    }
+
+    fn is_enabled(&self) -> bool {
+        true
+    }
+}
+
+#[tokio::test]
+async fn audit_runtime_view_lists_empty_targets() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let runtime_view = AuditRuntimeView::new(registry);
+
+    assert!(runtime_view.list_targets().await.is_empty());
+    assert!(runtime_view.get_target_values().await.is_empty());
+    assert!(runtime_view.get_target("missing").await.is_none());
+}
+
+#[tokio::test]
+async fn audit_pipeline_reports_empty_runtime_snapshots() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let pipeline = AuditPipeline::new(registry);
+
+    assert!(pipeline.snapshot_target_metrics().await.is_empty());
+    assert!(pipeline.snapshot_target_health().await.is_empty());
+}
+
+#[tokio::test]
+async fn audit_runtime_facade_stops_empty_replay_workers() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let replay_workers = Arc::new(RwLock::new(rustfs_targets::ReplayWorkerManager::new()));
+    let facade = AuditRuntimeFacade::new(registry, replay_workers);
+
+    facade.stop_replay_workers().await;
+}
+
+#[tokio::test]
+async fn audit_runtime_facade_activates_empty_target_list() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let replay_workers = Arc::new(RwLock::new(rustfs_targets::ReplayWorkerManager::new()));
+    let facade = AuditRuntimeFacade::new(registry, replay_workers);
+
+    let activation = facade.activate_targets_with_replay(Vec::new()).await;
+    assert!(activation.targets.is_empty());
+    assert_eq!(activation.replay_workers.len(), 0);
+}
+
+#[tokio::test]
+async fn audit_runtime_view_upsert_and_remove_target() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let runtime_view = AuditRuntimeView::new(registry.clone());
+    let target = TestTarget::new("primary", "webhook");
+    let init_calls = Arc::clone(&target.init_calls);
+    let close_calls = Arc::clone(&target.close_calls);
+
+    runtime_view
+        .upsert_target("primary:webhook".to_string(), Box::new(target))
+        .await
+        .expect("upsert should succeed");
+
+    assert_eq!(runtime_view.list_targets().await, vec!["primary:webhook".to_string()]);
+    assert_eq!(init_calls.load(Ordering::SeqCst), 1);
+
+    runtime_view
+        .remove_target("primary:webhook")
+        .await
+        .expect("remove should succeed");
+
+    assert!(runtime_view.list_targets().await.is_empty());
+    assert_eq!(close_calls.load(Ordering::SeqCst), 1);
+}
+
+#[tokio::test]
+async fn audit_runtime_facade_replace_targets_commits_runtime_state() {
+    let registry = Arc::new(Mutex::new(AuditRegistry::new()));
+    let replay_workers = Arc::new(RwLock::new(rustfs_targets::ReplayWorkerManager::new()));
+    let facade = AuditRuntimeFacade::new(registry.clone(), replay_workers.clone());
+    let target = TestTarget::new("primary", "webhook");
+    let activation = rustfs_targets::RuntimeActivation {
+        replay_workers: rustfs_targets::ReplayWorkerManager::new(),
+        targets: vec![Arc::new(target) as rustfs_targets::SharedTarget<rustfs_audit::AuditEntry>],
+    };
+
+    facade
+        .replace_targets(activation)
+        .await
+        .expect("replace_targets should succeed");
+
+    let runtime_view = AuditRuntimeView::new(registry);
+    assert_eq!(runtime_view.list_targets().await, vec!["primary:webhook".to_string()]);
+    assert_eq!(replay_workers.read().await.len(), 0);
+}

crates/common/src/data_usage.rs

@@ -51,14 +51,14 @@ impl AllTierStats {
     pub fn add_sizes(&mut self, tiers: HashMap<String, TierStats>) {
         for (tier, st) in tiers {
             self.tiers
-                .insert(tier.clone(), self.tiers.get(&tier).unwrap_or(&TierStats::default()).add(&st));
+                .insert(tier.clone(), self.tiers.get(&tier).copied().unwrap_or_default().add(&st));
         }
     }
 
     pub fn merge(&mut self, other: AllTierStats) {
         for (tier, st) in other.tiers {
             self.tiers
-                .insert(tier.clone(), self.tiers.get(&tier).unwrap_or(&TierStats::default()).add(&st));
+                .insert(tier.clone(), self.tiers.get(&tier).copied().unwrap_or_default().add(&st));
         }
     }
 
@@ -673,7 +673,7 @@ impl DataUsageCache {
         let mut leaves = Vec::new();
         let mut remove = total - limit;
         add(self, path, &mut leaves);
-        leaves.sort_by(|a, b| a.objects.cmp(&b.objects));
+        leaves.sort_by_key(|a| a.objects);
 
         while remove > 0 && !leaves.is_empty() {
             let Some(e) = leaves.first() else {

crates/common/src/heal_channel.rs

@@ -18,7 +18,7 @@ use std::{
     fmt::{self, Display},
     sync::OnceLock,
 };
-use tokio::sync::{broadcast, mpsc};
+use tokio::sync::{broadcast, mpsc, oneshot};
 use uuid::Uuid;
 
 pub const HEAL_DELETE_DANGLING: bool = true;
@@ -206,11 +206,59 @@ pub struct HealOpts {
     pub set: Option<usize>,
 }
 
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum HealAdmissionDropReason {
+    QueueFull,
+    PolicyDropped,
+}
+
+impl HealAdmissionDropReason {
+    pub fn as_str(self) -> &'static str {
+        match self {
+            Self::QueueFull => "queue_full",
+            Self::PolicyDropped => "policy_dropped",
+        }
+    }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum HealAdmissionResult {
+    Accepted,
+    Merged,
+    Full,
+    Dropped(HealAdmissionDropReason),
+}
+
+impl HealAdmissionResult {
+    pub fn result_label(self) -> &'static str {
+        match self {
+            Self::Accepted => "accepted",
+            Self::Merged => "merged",
+            Self::Full => "full",
+            Self::Dropped(_) => "dropped",
+        }
+    }
+
+    pub fn reason_label(self) -> &'static str {
+        match self {
+            Self::Dropped(reason) => reason.as_str(),
+            _ => "none",
+        }
+    }
+
+    pub fn is_admitted(self) -> bool {
+        matches!(self, Self::Accepted | Self::Merged)
+    }
+}
+
 /// Heal channel command type
-#[derive(Debug, Clone)]
+#[derive(Debug)]
 pub enum HealChannelCommand {
     /// Start a new heal task
-    Start(HealChannelRequest),
+    Start {
+        request: HealChannelRequest,
+        response_tx: oneshot::Sender<Result<HealAdmissionResult, String>>,
+    },
     /// Query heal task status
     Query { heal_path: String, client_token: String },
     /// Cancel heal task
@@ -339,9 +387,22 @@ pub fn subscribe_heal_responses() -> broadcast::Receiver<HealChannelResponse> {
     heal_response_sender().subscribe()
 }
 
+/// Send heal start request and wait for structured admission feedback.
+pub async fn send_heal_request_with_admission(request: HealChannelRequest) -> Result<HealAdmissionResult, String> {
+    let (response_tx, response_rx) = oneshot::channel();
+    send_heal_command(HealChannelCommand::Start { request, response_tx }).await?;
+    response_rx
+        .await
+        .map_err(|e| format!("Failed to receive heal admission response: {e}"))?
+}
+
 /// Send heal start request
 pub async fn send_heal_request(request: HealChannelRequest) -> Result<(), String> {
-    send_heal_command(HealChannelCommand::Start(request)).await
+    match send_heal_request_with_admission(request).await? {
+        HealAdmissionResult::Accepted | HealAdmissionResult::Merged => Ok(()),
+        HealAdmissionResult::Full => Err("Heal request queue is full".to_string()),
+        HealAdmissionResult::Dropped(reason) => Err(format!("Heal request dropped: {}", reason.as_str())),
+    }
 }
 
 /// Send heal query request
@@ -450,7 +511,7 @@ pub fn lc_has_active_rules(config: &BucketLifecycleConfiguration, prefix: &str)
         }
 
         if let Some(e) = &rule.noncurrent_version_expiration {
-            if let Some(true) = e.noncurrent_days.map(|d| d > 0) {
+            if e.noncurrent_days.is_some() {
                 return true;
             }
             if let Some(true) = e.newer_noncurrent_versions.map(|d| d > 0) {
@@ -542,6 +603,19 @@ pub async fn send_heal_disk(set_disk_id: String, priority: Option<HealChannelPri
 mod tests {
     use super::*;
 
+    #[test]
+    fn heal_admission_result_labels_are_stable() {
+        assert_eq!(HealAdmissionResult::Accepted.result_label(), "accepted");
+        assert_eq!(HealAdmissionResult::Merged.result_label(), "merged");
+        assert_eq!(HealAdmissionResult::Full.result_label(), "full");
+        assert_eq!(
+            HealAdmissionResult::Dropped(HealAdmissionDropReason::QueueFull).reason_label(),
+            "queue_full"
+        );
+        assert!(HealAdmissionResult::Merged.is_admitted());
+        assert!(!HealAdmissionResult::Full.is_admitted());
+    }
+
     #[tokio::test]
     async fn heal_response_broadcast_reaches_subscriber() {
         let mut receiver = subscribe_heal_responses();

crates/common/src/internode_metrics.rs

@@ -51,7 +51,7 @@ impl InternodeMetrics {
             return;
         }
         self.sent_bytes_total.fetch_add(bytes, Ordering::Relaxed);
-        counter!("rustfs.internode.sent.bytes.total").increment(bytes);
+        counter!("rustfs_system_network_internode_sent_bytes_total").increment(bytes);
     }
 
     pub fn record_recv_bytes(&self, bytes: usize) {
@@ -60,22 +60,22 @@ impl InternodeMetrics {
             return;
         }
         self.recv_bytes_total.fetch_add(bytes, Ordering::Relaxed);
-        counter!("rustfs.internode.recv.bytes.total").increment(bytes);
+        counter!("rustfs_system_network_internode_recv_bytes_total").increment(bytes);
     }
 
     pub fn record_outgoing_request(&self) {
         self.outgoing_requests_total.fetch_add(1, Ordering::Relaxed);
-        counter!("rustfs.internode.requests.outgoing.total").increment(1);
+        counter!("rustfs_system_network_internode_requests_outgoing_total").increment(1);
     }
 
     pub fn record_incoming_request(&self) {
         self.incoming_requests_total.fetch_add(1, Ordering::Relaxed);
-        counter!("rustfs.internode.requests.incoming.total").increment(1);
+        counter!("rustfs_system_network_internode_requests_incoming_total").increment(1);
     }
 
     pub fn record_error(&self) {
         self.errors_total.fetch_add(1, Ordering::Relaxed);
-        counter!("rustfs.internode.errors.total").increment(1);
+        counter!("rustfs_system_network_internode_errors_total").increment(1);
     }
 
     pub fn record_dial_result(&self, duration: Duration, success: bool) {
@@ -83,11 +83,11 @@ impl InternodeMetrics {
         self.dial_total_time_nanos.fetch_add(elapsed_nanos, Ordering::Relaxed);
         let samples = self.dial_samples_total.fetch_add(1, Ordering::Relaxed) + 1;
         let total = self.dial_total_time_nanos.load(Ordering::Relaxed);
-        gauge!("rustfs.internode.dial.avg_time.nanos").set(total as f64 / samples as f64);
+        gauge!("rustfs_system_network_internode_dial_avg_time_nanos").set(total as f64 / samples as f64);
 
         if !success {
             self.dial_errors_total.fetch_add(1, Ordering::Relaxed);
-            counter!("rustfs.internode.dial.errors.total").increment(1);
+            counter!("rustfs_system_network_internode_dial_errors_total").increment(1);
         }
 
         let now_ms = SystemTime::now()
@@ -101,11 +101,7 @@ impl InternodeMetrics {
     pub fn snapshot(&self) -> InternodeMetricsSnapshot {
         let dial_samples_total = self.dial_samples_total.load(Ordering::Relaxed);
         let dial_total_time_nanos = self.dial_total_time_nanos.load(Ordering::Relaxed);
-        let dial_avg_time_nanos = if dial_samples_total == 0 {
-            0
-        } else {
-            dial_total_time_nanos / dial_samples_total
-        };
+        let dial_avg_time_nanos = dial_total_time_nanos.checked_div(dial_samples_total).unwrap_or(0);
 
         InternodeMetricsSnapshot {
             sent_bytes_total: self.sent_bytes_total.load(Ordering::Relaxed),

crates/concurrency/Cargo.toml

@@ -26,7 +26,7 @@ thiserror = { workspace = true }
 tracing = { workspace = true }
 
 [dev-dependencies]
-tokio = { workspace = true, features = ["full"] }
+tokio = { workspace = true, features = ["test-util","macros","rt-multi-thread"] }
 
 [features]
 default = ["timeout", "lock", "deadlock", "backpressure", "scheduler"]

crates/config/README.md

@@ -41,6 +41,7 @@ Examples:
 - `RUSTFS_ADDRESS`
 - `RUSTFS_VOLUMES`
 - `RUSTFS_LICENSE`
+- `RUSTFS_LICENSE_PUBLIC_KEY`
 
 Current guidance:
 - Prefer module-specific names only when they are not top-level product configuration.
@@ -51,10 +52,36 @@ Current guidance:
   - `RUSTFS_ENABLE_HEAL` -> `RUSTFS_HEAL_ENABLED`
   - `RUSTFS_DATA_SCANNER_START_DELAY_SECS` -> `RUSTFS_SCANNER_START_DELAY_SECS`
 
+## License environment variables
+
+- `RUSTFS_LICENSE` contains the signed license token.
+- `RUSTFS_LICENSE_PUBLIC_KEY` contains the RSA public key used to verify signed license tokens.
+
+## CORS environment variables
+
+- `RUSTFS_CORS_ALLOWED_ORIGINS` defaults to empty, so the S3 endpoint emits no generic CORS headers unless configured. Set `*` for wildcard origins without credentials, or a comma-separated allow-list for credentialed explicit origins.
+- `RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS` defaults to `*` for the console service.
+
 ## Scanner environment aliases
 
+- `RUSTFS_SCANNER_SPEED` (canonical, also accepts `MINIO_SCANNER_SPEED`)
+- `RUSTFS_SCANNER_CYCLE` (canonical, also accepts `MINIO_SCANNER_CYCLE`)
 - `RUSTFS_SCANNER_START_DELAY_SECS` (canonical)
 - `RUSTFS_DATA_SCANNER_START_DELAY_SECS` (deprecated alias for compatibility)
+- `RUSTFS_SCANNER_IDLE_MODE` (canonical)
+- `RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS` (canonical)
+
+## Drive timeout environment variables
+
+- `RUSTFS_DRIVE_METADATA_TIMEOUT_SECS`
+- `RUSTFS_DRIVE_DISK_INFO_TIMEOUT_SECS`
+- `RUSTFS_DRIVE_LIST_DIR_TIMEOUT_SECS`
+- `RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS`
+- `RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS`
+
+Legacy compatibility fallback:
+- `RUSTFS_DRIVE_MAX_TIMEOUT_DURATION`
+  This legacy variable is treated as a deprecated fallback for the operation-specific drive timeout variables above when a canonical variable is unset.
 
 ## 📄 License
 

crates/config/src/audit/amqp.rs

@@ -0,0 +1,60 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const AUDIT_AMQP_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::AMQP_URL,
+    crate::AMQP_EXCHANGE,
+    crate::AMQP_ROUTING_KEY,
+    crate::AMQP_MANDATORY,
+    crate::AMQP_PERSISTENT,
+    crate::AMQP_USERNAME,
+    crate::AMQP_PASSWORD,
+    crate::AMQP_TLS_CA,
+    crate::AMQP_TLS_CLIENT_CERT,
+    crate::AMQP_TLS_CLIENT_KEY,
+    crate::AMQP_QUEUE_DIR,
+    crate::AMQP_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_AUDIT_AMQP_ENABLE: &str = "RUSTFS_AUDIT_AMQP_ENABLE";
+pub const ENV_AUDIT_AMQP_URL: &str = "RUSTFS_AUDIT_AMQP_URL";
+pub const ENV_AUDIT_AMQP_EXCHANGE: &str = "RUSTFS_AUDIT_AMQP_EXCHANGE";
+pub const ENV_AUDIT_AMQP_ROUTING_KEY: &str = "RUSTFS_AUDIT_AMQP_ROUTING_KEY";
+pub const ENV_AUDIT_AMQP_MANDATORY: &str = "RUSTFS_AUDIT_AMQP_MANDATORY";
+pub const ENV_AUDIT_AMQP_PERSISTENT: &str = "RUSTFS_AUDIT_AMQP_PERSISTENT";
+pub const ENV_AUDIT_AMQP_USERNAME: &str = "RUSTFS_AUDIT_AMQP_USERNAME";
+pub const ENV_AUDIT_AMQP_PASSWORD: &str = "RUSTFS_AUDIT_AMQP_PASSWORD";
+pub const ENV_AUDIT_AMQP_TLS_CA: &str = "RUSTFS_AUDIT_AMQP_TLS_CA";
+pub const ENV_AUDIT_AMQP_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_AMQP_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_AMQP_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_AMQP_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_AMQP_QUEUE_DIR: &str = "RUSTFS_AUDIT_AMQP_QUEUE_DIR";
+pub const ENV_AUDIT_AMQP_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_AMQP_QUEUE_LIMIT";
+
+pub const ENV_AUDIT_AMQP_KEYS: &[&str; 13] = &[
+    ENV_AUDIT_AMQP_ENABLE,
+    ENV_AUDIT_AMQP_URL,
+    ENV_AUDIT_AMQP_EXCHANGE,
+    ENV_AUDIT_AMQP_ROUTING_KEY,
+    ENV_AUDIT_AMQP_MANDATORY,
+    ENV_AUDIT_AMQP_PERSISTENT,
+    ENV_AUDIT_AMQP_USERNAME,
+    ENV_AUDIT_AMQP_PASSWORD,
+    ENV_AUDIT_AMQP_TLS_CA,
+    ENV_AUDIT_AMQP_TLS_CLIENT_CERT,
+    ENV_AUDIT_AMQP_TLS_CLIENT_KEY,
+    ENV_AUDIT_AMQP_QUEUE_DIR,
+    ENV_AUDIT_AMQP_QUEUE_LIMIT,
+];

crates/config/src/audit/kafka.rs

@@ -0,0 +1,53 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Kafka Environment Variables
+pub const ENV_AUDIT_KAFKA_ENABLE: &str = "RUSTFS_AUDIT_KAFKA_ENABLE";
+pub const ENV_AUDIT_KAFKA_BROKERS: &str = "RUSTFS_AUDIT_KAFKA_BROKERS";
+pub const ENV_AUDIT_KAFKA_TOPIC: &str = "RUSTFS_AUDIT_KAFKA_TOPIC";
+pub const ENV_AUDIT_KAFKA_ACKS: &str = "RUSTFS_AUDIT_KAFKA_ACKS";
+pub const ENV_AUDIT_KAFKA_TLS_ENABLE: &str = "RUSTFS_AUDIT_KAFKA_TLS_ENABLE";
+pub const ENV_AUDIT_KAFKA_TLS_CA: &str = "RUSTFS_AUDIT_KAFKA_TLS_CA";
+pub const ENV_AUDIT_KAFKA_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_KAFKA_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_KAFKA_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_KAFKA_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_KAFKA_QUEUE_DIR: &str = "RUSTFS_AUDIT_KAFKA_QUEUE_DIR";
+pub const ENV_AUDIT_KAFKA_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_KAFKA_QUEUE_LIMIT";
+
+pub const ENV_AUDIT_KAFKA_KEYS: &[&str; 10] = &[
+    ENV_AUDIT_KAFKA_ENABLE,
+    ENV_AUDIT_KAFKA_BROKERS,
+    ENV_AUDIT_KAFKA_TOPIC,
+    ENV_AUDIT_KAFKA_ACKS,
+    ENV_AUDIT_KAFKA_TLS_ENABLE,
+    ENV_AUDIT_KAFKA_TLS_CA,
+    ENV_AUDIT_KAFKA_TLS_CLIENT_CERT,
+    ENV_AUDIT_KAFKA_TLS_CLIENT_KEY,
+    ENV_AUDIT_KAFKA_QUEUE_DIR,
+    ENV_AUDIT_KAFKA_QUEUE_LIMIT,
+];
+
+/// A list of all valid configuration keys for a Kafka audit target.
+pub const AUDIT_KAFKA_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::KAFKA_BROKERS,
+    crate::KAFKA_TOPIC,
+    crate::KAFKA_ACKS,
+    crate::KAFKA_TLS_ENABLE,
+    crate::KAFKA_TLS_CA,
+    crate::KAFKA_TLS_CLIENT_CERT,
+    crate::KAFKA_TLS_CLIENT_KEY,
+    crate::KAFKA_QUEUE_DIR,
+    crate::KAFKA_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];

crates/config/src/audit/mod.rs

@@ -16,10 +16,24 @@
 //! This module defines the configuration for audit systems, including
 //! webhook and MQTT audit-related settings.
 
+mod amqp;
+mod kafka;
 mod mqtt;
+mod mysql;
+mod nats;
+mod postgres;
+mod pulsar;
+mod redis;
 mod webhook;
 
+pub use amqp::*;
+pub use kafka::*;
 pub use mqtt::*;
+pub use mysql::*;
+pub use nats::*;
+pub use postgres::*;
+pub use pulsar::*;
+pub use redis::*;
 pub use webhook::*;
 
 use crate::DEFAULT_DELIMITER;
@@ -29,8 +43,24 @@ pub const AUDIT_PREFIX: &str = "audit";
 pub const AUDIT_ROUTE_PREFIX: &str = const_str::concat!(AUDIT_PREFIX, DEFAULT_DELIMITER);
 
 pub const AUDIT_WEBHOOK_SUB_SYS: &str = "audit_webhook";
+pub const AUDIT_AMQP_SUB_SYS: &str = "audit_amqp";
+pub const AUDIT_KAFKA_SUB_SYS: &str = "audit_kafka";
 pub const AUDIT_MQTT_SUB_SYS: &str = "audit_mqtt";
-
+pub const AUDIT_MYSQL_SUB_SYS: &str = "audit_mysql";
+pub const AUDIT_NATS_SUB_SYS: &str = "audit_nats";
+pub const AUDIT_POSTGRES_SUB_SYS: &str = "audit_postgres";
+pub const AUDIT_PULSAR_SUB_SYS: &str = "audit_pulsar";
+pub const AUDIT_REDIS_SUB_SYS: &str = "audit_redis";
+pub const AUDIT_REDIS_DEFAULT_CHANNEL: &str = "rustfs_audit_channel";
 pub const AUDIT_STORE_EXTENSION: &str = ".audit";
-#[allow(dead_code)]
-pub const AUDIT_SUB_SYSTEMS: &[&str] = &[AUDIT_MQTT_SUB_SYS, AUDIT_WEBHOOK_SUB_SYS];
+pub const AUDIT_SUB_SYSTEMS: &[&str] = &[
+    AUDIT_AMQP_SUB_SYS,
+    AUDIT_KAFKA_SUB_SYS,
+    AUDIT_MQTT_SUB_SYS,
+    AUDIT_MYSQL_SUB_SYS,
+    AUDIT_NATS_SUB_SYS,
+    AUDIT_POSTGRES_SUB_SYS,
+    AUDIT_PULSAR_SUB_SYS,
+    AUDIT_REDIS_SUB_SYS,
+    AUDIT_WEBHOOK_SUB_SYS,
+];

crates/config/src/audit/mysql.rs

@@ -0,0 +1,53 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// A list of all valid configuration keys for an audit MySQL target.
+pub const AUDIT_MYSQL_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::MYSQL_DSN_STRING,
+    crate::MYSQL_TABLE,
+    crate::MYSQL_FORMAT,
+    crate::MYSQL_TLS_CA,
+    crate::MYSQL_TLS_CLIENT_CERT,
+    crate::MYSQL_TLS_CLIENT_KEY,
+    crate::MYSQL_QUEUE_DIR,
+    crate::MYSQL_QUEUE_LIMIT,
+    crate::MYSQL_MAX_OPEN_CONNECTIONS,
+    crate::COMMENT_KEY,
+];
+
+// MySQL environment variables for audit target.
+pub const ENV_AUDIT_MYSQL_ENABLE: &str = "RUSTFS_AUDIT_MYSQL_ENABLE";
+pub const ENV_AUDIT_MYSQL_DSN_STRING: &str = "RUSTFS_AUDIT_MYSQL_DSN_STRING";
+pub const ENV_AUDIT_MYSQL_TABLE: &str = "RUSTFS_AUDIT_MYSQL_TABLE";
+pub const ENV_AUDIT_MYSQL_FORMAT: &str = "RUSTFS_AUDIT_MYSQL_FORMAT";
+pub const ENV_AUDIT_MYSQL_TLS_CA: &str = "RUSTFS_AUDIT_MYSQL_TLS_CA";
+pub const ENV_AUDIT_MYSQL_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_MYSQL_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_MYSQL_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_MYSQL_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_MYSQL_QUEUE_DIR: &str = "RUSTFS_AUDIT_MYSQL_QUEUE_DIR";
+pub const ENV_AUDIT_MYSQL_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_MYSQL_QUEUE_LIMIT";
+pub const ENV_AUDIT_MYSQL_MAX_OPEN_CONNECTIONS: &str = "RUSTFS_AUDIT_MYSQL_MAX_OPEN_CONNECTIONS";
+
+pub const ENV_AUDIT_MYSQL_KEYS: &[&str; 10] = &[
+    ENV_AUDIT_MYSQL_ENABLE,
+    ENV_AUDIT_MYSQL_DSN_STRING,
+    ENV_AUDIT_MYSQL_TABLE,
+    ENV_AUDIT_MYSQL_FORMAT,
+    ENV_AUDIT_MYSQL_TLS_CA,
+    ENV_AUDIT_MYSQL_TLS_CLIENT_CERT,
+    ENV_AUDIT_MYSQL_TLS_CLIENT_KEY,
+    ENV_AUDIT_MYSQL_QUEUE_DIR,
+    ENV_AUDIT_MYSQL_QUEUE_LIMIT,
+    ENV_AUDIT_MYSQL_MAX_OPEN_CONNECTIONS,
+];

crates/config/src/audit/nats.rs

@@ -0,0 +1,60 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const ENV_AUDIT_NATS_ENABLE: &str = "RUSTFS_AUDIT_NATS_ENABLE";
+pub const ENV_AUDIT_NATS_ADDRESS: &str = "RUSTFS_AUDIT_NATS_ADDRESS";
+pub const ENV_AUDIT_NATS_SUBJECT: &str = "RUSTFS_AUDIT_NATS_SUBJECT";
+pub const ENV_AUDIT_NATS_USERNAME: &str = "RUSTFS_AUDIT_NATS_USERNAME";
+pub const ENV_AUDIT_NATS_PASSWORD: &str = "RUSTFS_AUDIT_NATS_PASSWORD";
+pub const ENV_AUDIT_NATS_TOKEN: &str = "RUSTFS_AUDIT_NATS_TOKEN";
+pub const ENV_AUDIT_NATS_CREDENTIALS_FILE: &str = "RUSTFS_AUDIT_NATS_CREDENTIALS_FILE";
+pub const ENV_AUDIT_NATS_TLS_CA: &str = "RUSTFS_AUDIT_NATS_TLS_CA";
+pub const ENV_AUDIT_NATS_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_NATS_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_NATS_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_NATS_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_NATS_TLS_REQUIRED: &str = "RUSTFS_AUDIT_NATS_TLS_REQUIRED";
+pub const ENV_AUDIT_NATS_QUEUE_DIR: &str = "RUSTFS_AUDIT_NATS_QUEUE_DIR";
+pub const ENV_AUDIT_NATS_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_NATS_QUEUE_LIMIT";
+
+pub const ENV_AUDIT_NATS_KEYS: &[&str; 13] = &[
+    ENV_AUDIT_NATS_ENABLE,
+    ENV_AUDIT_NATS_ADDRESS,
+    ENV_AUDIT_NATS_SUBJECT,
+    ENV_AUDIT_NATS_USERNAME,
+    ENV_AUDIT_NATS_PASSWORD,
+    ENV_AUDIT_NATS_TOKEN,
+    ENV_AUDIT_NATS_CREDENTIALS_FILE,
+    ENV_AUDIT_NATS_TLS_CA,
+    ENV_AUDIT_NATS_TLS_CLIENT_CERT,
+    ENV_AUDIT_NATS_TLS_CLIENT_KEY,
+    ENV_AUDIT_NATS_TLS_REQUIRED,
+    ENV_AUDIT_NATS_QUEUE_DIR,
+    ENV_AUDIT_NATS_QUEUE_LIMIT,
+];
+
+pub const AUDIT_NATS_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::NATS_ADDRESS,
+    crate::NATS_SUBJECT,
+    crate::NATS_USERNAME,
+    crate::NATS_PASSWORD,
+    crate::NATS_TOKEN,
+    crate::NATS_CREDENTIALS_FILE,
+    crate::NATS_TLS_CA,
+    crate::NATS_TLS_CLIENT_CERT,
+    crate::NATS_TLS_CLIENT_KEY,
+    crate::NATS_TLS_REQUIRED,
+    crate::NATS_QUEUE_DIR,
+    crate::NATS_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];

crates/config/src/audit/postgres.rs

@@ -0,0 +1,51 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const AUDIT_POSTGRES_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::POSTGRES_DSN_STRING,
+    crate::POSTGRES_TABLE,
+    crate::POSTGRES_FORMAT,
+    crate::POSTGRES_TLS_REQUIRED,
+    crate::POSTGRES_TLS_CA,
+    crate::POSTGRES_TLS_CLIENT_CERT,
+    crate::POSTGRES_TLS_CLIENT_KEY,
+    crate::POSTGRES_QUEUE_DIR,
+    crate::POSTGRES_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_AUDIT_POSTGRES_ENABLE: &str = "RUSTFS_AUDIT_POSTGRES_ENABLE";
+pub const ENV_AUDIT_POSTGRES_DSN_STRING: &str = "RUSTFS_AUDIT_POSTGRES_DSN_STRING";
+pub const ENV_AUDIT_POSTGRES_TABLE: &str = "RUSTFS_AUDIT_POSTGRES_TABLE";
+pub const ENV_AUDIT_POSTGRES_FORMAT: &str = "RUSTFS_AUDIT_POSTGRES_FORMAT";
+pub const ENV_AUDIT_POSTGRES_TLS_REQUIRED: &str = "RUSTFS_AUDIT_POSTGRES_TLS_REQUIRED";
+pub const ENV_AUDIT_POSTGRES_TLS_CA: &str = "RUSTFS_AUDIT_POSTGRES_TLS_CA";
+pub const ENV_AUDIT_POSTGRES_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_POSTGRES_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_POSTGRES_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_POSTGRES_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_POSTGRES_QUEUE_DIR: &str = "RUSTFS_AUDIT_POSTGRES_QUEUE_DIR";
+pub const ENV_AUDIT_POSTGRES_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_POSTGRES_QUEUE_LIMIT";
+
+pub const ENV_AUDIT_POSTGRES_KEYS: &[&str; 10] = &[
+    ENV_AUDIT_POSTGRES_ENABLE,
+    ENV_AUDIT_POSTGRES_DSN_STRING,
+    ENV_AUDIT_POSTGRES_TABLE,
+    ENV_AUDIT_POSTGRES_FORMAT,
+    ENV_AUDIT_POSTGRES_TLS_REQUIRED,
+    ENV_AUDIT_POSTGRES_TLS_CA,
+    ENV_AUDIT_POSTGRES_TLS_CLIENT_CERT,
+    ENV_AUDIT_POSTGRES_TLS_CLIENT_KEY,
+    ENV_AUDIT_POSTGRES_QUEUE_DIR,
+    ENV_AUDIT_POSTGRES_QUEUE_LIMIT,
+];

crates/config/src/audit/pulsar.rs

@@ -0,0 +1,54 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const ENV_AUDIT_PULSAR_ENABLE: &str = "RUSTFS_AUDIT_PULSAR_ENABLE";
+pub const ENV_AUDIT_PULSAR_BROKER: &str = "RUSTFS_AUDIT_PULSAR_BROKER";
+pub const ENV_AUDIT_PULSAR_TOPIC: &str = "RUSTFS_AUDIT_PULSAR_TOPIC";
+pub const ENV_AUDIT_PULSAR_AUTH_TOKEN: &str = "RUSTFS_AUDIT_PULSAR_AUTH_TOKEN";
+pub const ENV_AUDIT_PULSAR_USERNAME: &str = "RUSTFS_AUDIT_PULSAR_USERNAME";
+pub const ENV_AUDIT_PULSAR_PASSWORD: &str = "RUSTFS_AUDIT_PULSAR_PASSWORD";
+pub const ENV_AUDIT_PULSAR_TLS_CA: &str = "RUSTFS_AUDIT_PULSAR_TLS_CA";
+pub const ENV_AUDIT_PULSAR_TLS_ALLOW_INSECURE: &str = "RUSTFS_AUDIT_PULSAR_TLS_ALLOW_INSECURE";
+pub const ENV_AUDIT_PULSAR_TLS_HOSTNAME_VERIFICATION: &str = "RUSTFS_AUDIT_PULSAR_TLS_HOSTNAME_VERIFICATION";
+pub const ENV_AUDIT_PULSAR_QUEUE_DIR: &str = "RUSTFS_AUDIT_PULSAR_QUEUE_DIR";
+pub const ENV_AUDIT_PULSAR_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_PULSAR_QUEUE_LIMIT";
+
+pub const ENV_AUDIT_PULSAR_KEYS: &[&str; 11] = &[
+    ENV_AUDIT_PULSAR_ENABLE,
+    ENV_AUDIT_PULSAR_BROKER,
+    ENV_AUDIT_PULSAR_TOPIC,
+    ENV_AUDIT_PULSAR_AUTH_TOKEN,
+    ENV_AUDIT_PULSAR_USERNAME,
+    ENV_AUDIT_PULSAR_PASSWORD,
+    ENV_AUDIT_PULSAR_TLS_CA,
+    ENV_AUDIT_PULSAR_TLS_ALLOW_INSECURE,
+    ENV_AUDIT_PULSAR_TLS_HOSTNAME_VERIFICATION,
+    ENV_AUDIT_PULSAR_QUEUE_DIR,
+    ENV_AUDIT_PULSAR_QUEUE_LIMIT,
+];
+
+pub const AUDIT_PULSAR_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::PULSAR_BROKER,
+    crate::PULSAR_TOPIC,
+    crate::PULSAR_AUTH_TOKEN,
+    crate::PULSAR_USERNAME,
+    crate::PULSAR_PASSWORD,
+    crate::PULSAR_TLS_CA,
+    crate::PULSAR_TLS_ALLOW_INSECURE,
+    crate::PULSAR_TLS_HOSTNAME_VERIFICATION,
+    crate::PULSAR_QUEUE_DIR,
+    crate::PULSAR_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];

crates/config/src/audit/redis.rs

@@ -0,0 +1,81 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const ENV_AUDIT_REDIS_ENABLE: &str = "RUSTFS_AUDIT_REDIS_ENABLE";
+pub const ENV_AUDIT_REDIS_URL: &str = "RUSTFS_AUDIT_REDIS_URL";
+pub const ENV_AUDIT_REDIS_CHANNEL: &str = "RUSTFS_AUDIT_REDIS_CHANNEL";
+pub const ENV_AUDIT_REDIS_USERNAME: &str = "RUSTFS_AUDIT_REDIS_USERNAME";
+pub const ENV_AUDIT_REDIS_PASSWORD: &str = "RUSTFS_AUDIT_REDIS_PASSWORD";
+pub const ENV_AUDIT_REDIS_KEEP_ALIVE_INTERVAL: &str = "RUSTFS_AUDIT_REDIS_KEEP_ALIVE_INTERVAL";
+pub const ENV_AUDIT_REDIS_QUEUE_DIR: &str = "RUSTFS_AUDIT_REDIS_QUEUE_DIR";
+pub const ENV_AUDIT_REDIS_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_REDIS_QUEUE_LIMIT";
+pub const ENV_AUDIT_REDIS_MAX_RETRY_ATTEMPTS: &str = "RUSTFS_AUDIT_REDIS_MAX_RETRY_ATTEMPTS";
+pub const ENV_AUDIT_REDIS_RECONNECT_RETRY_ATTEMPTS: &str = "RUSTFS_AUDIT_REDIS_RECONNECT_RETRY_ATTEMPTS";
+pub const ENV_AUDIT_REDIS_MIN_RETRY_DELAY: &str = "RUSTFS_AUDIT_REDIS_MIN_RETRY_DELAY";
+pub const ENV_AUDIT_REDIS_MAX_RETRY_DELAY: &str = "RUSTFS_AUDIT_REDIS_MAX_RETRY_DELAY";
+pub const ENV_AUDIT_REDIS_CONNECTION_TIMEOUT: &str = "RUSTFS_AUDIT_REDIS_CONNECTION_TIMEOUT";
+pub const ENV_AUDIT_REDIS_RESPONSE_TIMEOUT: &str = "RUSTFS_AUDIT_REDIS_RESPONSE_TIMEOUT";
+pub const ENV_AUDIT_REDIS_PIPELINE_BUFFER_SIZE: &str = "RUSTFS_AUDIT_REDIS_PIPELINE_BUFFER_SIZE";
+pub const ENV_AUDIT_REDIS_TLS_POLICY: &str = "RUSTFS_AUDIT_REDIS_TLS_POLICY";
+pub const ENV_AUDIT_REDIS_TLS_CA: &str = "RUSTFS_AUDIT_REDIS_TLS_CA";
+pub const ENV_AUDIT_REDIS_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_REDIS_TLS_CLIENT_CERT";
+pub const ENV_AUDIT_REDIS_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_REDIS_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_REDIS_TLS_ALLOW_INSECURE: &str = "RUSTFS_AUDIT_REDIS_TLS_ALLOW_INSECURE";
+
+pub const ENV_AUDIT_REDIS_KEYS: &[&str; 20] = &[
+    ENV_AUDIT_REDIS_ENABLE,
+    ENV_AUDIT_REDIS_URL,
+    ENV_AUDIT_REDIS_CHANNEL,
+    ENV_AUDIT_REDIS_USERNAME,
+    ENV_AUDIT_REDIS_PASSWORD,
+    ENV_AUDIT_REDIS_KEEP_ALIVE_INTERVAL,
+    ENV_AUDIT_REDIS_QUEUE_DIR,
+    ENV_AUDIT_REDIS_QUEUE_LIMIT,
+    ENV_AUDIT_REDIS_MAX_RETRY_ATTEMPTS,
+    ENV_AUDIT_REDIS_RECONNECT_RETRY_ATTEMPTS,
+    ENV_AUDIT_REDIS_MIN_RETRY_DELAY,
+    ENV_AUDIT_REDIS_MAX_RETRY_DELAY,
+    ENV_AUDIT_REDIS_CONNECTION_TIMEOUT,
+    ENV_AUDIT_REDIS_RESPONSE_TIMEOUT,
+    ENV_AUDIT_REDIS_PIPELINE_BUFFER_SIZE,
+    ENV_AUDIT_REDIS_TLS_POLICY,
+    ENV_AUDIT_REDIS_TLS_CA,
+    ENV_AUDIT_REDIS_TLS_CLIENT_CERT,
+    ENV_AUDIT_REDIS_TLS_CLIENT_KEY,
+    ENV_AUDIT_REDIS_TLS_ALLOW_INSECURE,
+];
+
+pub const AUDIT_REDIS_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::REDIS_URL,
+    crate::REDIS_CHANNEL,
+    crate::REDIS_USERNAME,
+    crate::REDIS_PASSWORD,
+    crate::REDIS_KEEP_ALIVE_INTERVAL,
+    crate::REDIS_QUEUE_DIR,
+    crate::REDIS_QUEUE_LIMIT,
+    crate::REDIS_MAX_RETRY_ATTEMPTS,
+    crate::REDIS_RECONNECT_RETRY_ATTEMPTS,
+    crate::REDIS_MIN_RETRY_DELAY,
+    crate::REDIS_MAX_RETRY_DELAY,
+    crate::REDIS_CONNECTION_TIMEOUT,
+    crate::REDIS_RESPONSE_TIMEOUT,
+    crate::REDIS_PIPELINE_BUFFER_SIZE,
+    crate::REDIS_TLS_POLICY,
+    crate::REDIS_TLS_CA,
+    crate::REDIS_TLS_CLIENT_CERT,
+    crate::REDIS_TLS_CLIENT_KEY,
+    crate::REDIS_TLS_ALLOW_INSECURE,
+    crate::COMMENT_KEY,
+];

crates/config/src/constants/app.rs

@@ -131,24 +131,30 @@ pub const ENV_RUSTFS_ADDRESS: &str = "RUSTFS_ADDRESS";
 /// Environment variable for server volumes.
 pub const ENV_RUSTFS_VOLUMES: &str = "RUSTFS_VOLUMES";
 
+/// Environment variable to explicitly bypass local physical disk independence checks.
+pub const ENV_UNSAFE_BYPASS_DISK_CHECK: &str = "RUSTFS_UNSAFE_BYPASS_DISK_CHECK";
+
+/// Compatibility alias used by legacy MinIO CI pipelines.
+///
+/// RustFS keeps this alias for backward compatibility only. Prefer
+/// `ENV_UNSAFE_BYPASS_DISK_CHECK` for explicit bypass control.
+pub const ENV_MINIO_CI: &str = "MINIO_CI";
+
+/// Default flag value for bypassing local physical disk independence checks.
+pub const DEFAULT_UNSAFE_BYPASS_DISK_CHECK: bool = false;
+
 /// Environment variable for server access key.
 pub const ENV_RUSTFS_ACCESS_KEY: &str = "RUSTFS_ACCESS_KEY";
 
 /// Environment variable for server access key file.
 pub const ENV_RUSTFS_ACCESS_KEY_FILE: &str = "RUSTFS_ACCESS_KEY_FILE";
 
-/// Environment variable for server root user.
-pub const ENV_RUSTFS_ROOT_USER: &str = "RUSTFS_ROOT_USER";
-
 /// Environment variable for server secret key.
 pub const ENV_RUSTFS_SECRET_KEY: &str = "RUSTFS_SECRET_KEY";
 
 /// Environment variable for server secret key file.
 pub const ENV_RUSTFS_SECRET_KEY_FILE: &str = "RUSTFS_SECRET_KEY_FILE";
 
-/// Environment variable for server root password.
-pub const ENV_RUSTFS_ROOT_PASSWORD: &str = "RUSTFS_ROOT_PASSWORD";
-
 /// Environment variable for server OBS endpoint.
 pub const ENV_RUSTFS_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";
 
@@ -223,6 +229,9 @@ pub const ENV_RUSTFS_REGION: &str = "RUSTFS_REGION";
 /// Environment variable for server license.
 pub const ENV_RUSTFS_LICENSE: &str = "RUSTFS_LICENSE";
 
+/// Environment variable for the RSA public key used to verify server licenses.
+pub const ENV_RUSTFS_LICENSE_PUBLIC_KEY: &str = "RUSTFS_LICENSE_PUBLIC_KEY";
+
 /// Default log filename for rustfs
 /// This is the default log filename for rustfs.
 /// It is used to store the logs of the application.
@@ -284,9 +293,9 @@ pub const DEFAULT_OBS_LOGS_EXPORT_ENABLED: bool = true;
 
 /// Default profiling export enabled
 /// It is used to enable or disable exporting profiles
-/// Default value: true
+/// Default value: false
 /// Environment variable: RUSTFS_OBS_PROFILING_EXPORT_ENABLED
-pub const DEFAULT_OBS_PROFILING_EXPORT_ENABLED: bool = true;
+pub const DEFAULT_OBS_PROFILING_EXPORT_ENABLED: bool = false;
 
 /// Default log local logging enabled for rustfs
 /// This is the default log local logging enabled for rustfs.
@@ -336,6 +345,7 @@ mod tests {
     fn test_environment_constants() {
         // Test environment related constants
         assert_eq!(ENVIRONMENT, "production");
+        assert_eq!(ENV_RUSTFS_LICENSE_PUBLIC_KEY, "RUSTFS_LICENSE_PUBLIC_KEY");
         assert!(
             ["development", "staging", "production", "test"].contains(&ENVIRONMENT),
             "Environment should be a standard environment name"

crates/config/src/constants/capacity.rs

@@ -39,6 +39,9 @@ pub const ENV_CAPACITY_STAT_TIMEOUT: &str = "RUSTFS_CAPACITY_STAT_TIMEOUT";
 /// Environment variable for sample rate
 pub const ENV_CAPACITY_SAMPLE_RATE: &str = "RUSTFS_CAPACITY_SAMPLE_RATE";
 
+/// Environment variable for metrics logging interval
+pub const ENV_CAPACITY_METRICS_INTERVAL: &str = "RUSTFS_CAPACITY_METRICS_INTERVAL";
+
 /// Environment variable for following symbolic links during capacity calculation
 pub const ENV_CAPACITY_FOLLOW_SYMLINKS: &str = "RUSTFS_CAPACITY_FOLLOW_SYMLINKS";
 
@@ -89,6 +92,10 @@ pub const DEFAULT_STAT_TIMEOUT_SECS: u64 = 3;
 /// Default: 200
 pub const DEFAULT_SAMPLE_RATE: usize = 200;
 
+/// Capacity metrics logging interval in seconds
+/// Default: 600 seconds (10 minutes)
+pub const DEFAULT_CAPACITY_METRICS_INTERVAL_SECS: u64 = 600;
+
 /// Follow symbolic links during capacity calculation
 /// Default: false (disabled for safety)
 pub const DEFAULT_CAPACITY_FOLLOW_SYMLINKS: bool = false;
@@ -130,6 +137,7 @@ mod tests {
         assert_eq!(ENV_CAPACITY_MAX_FILES_THRESHOLD, "RUSTFS_CAPACITY_MAX_FILES_THRESHOLD");
         assert_eq!(ENV_CAPACITY_STAT_TIMEOUT, "RUSTFS_CAPACITY_STAT_TIMEOUT");
         assert_eq!(ENV_CAPACITY_SAMPLE_RATE, "RUSTFS_CAPACITY_SAMPLE_RATE");
+        assert_eq!(ENV_CAPACITY_METRICS_INTERVAL, "RUSTFS_CAPACITY_METRICS_INTERVAL");
         assert_eq!(ENV_CAPACITY_FOLLOW_SYMLINKS, "RUSTFS_CAPACITY_FOLLOW_SYMLINKS");
         assert_eq!(ENV_CAPACITY_MAX_SYMLINK_DEPTH, "RUSTFS_CAPACITY_MAX_SYMLINK_DEPTH");
         assert_eq!(ENV_CAPACITY_ENABLE_DYNAMIC_TIMEOUT, "RUSTFS_CAPACITY_ENABLE_DYNAMIC_TIMEOUT");
@@ -147,6 +155,7 @@ mod tests {
         assert_eq!(DEFAULT_MAX_FILES_THRESHOLD, 200_000);
         assert_eq!(DEFAULT_STAT_TIMEOUT_SECS, 3);
         assert_eq!(DEFAULT_SAMPLE_RATE, 200);
+        assert_eq!(DEFAULT_CAPACITY_METRICS_INTERVAL_SECS, 600);
         assert_eq!(DEFAULT_CAPACITY_MAX_SYMLINK_DEPTH, 3);
         assert_eq!(DEFAULT_CAPACITY_MIN_TIMEOUT_SECS, 2);
         assert_eq!(DEFAULT_CAPACITY_MAX_TIMEOUT_SECS, 15);

crates/config/src/constants/console.rs

@@ -17,16 +17,21 @@
 pub const ENV_CORS_ALLOWED_ORIGINS: &str = "RUSTFS_CORS_ALLOWED_ORIGINS";
 
 /// Default CORS allowed origins for the endpoint service
-/// Comes from the console service default
-/// See DEFAULT_CONSOLE_CORS_ALLOWED_ORIGINS
-pub const DEFAULT_CORS_ALLOWED_ORIGINS: &str = DEFAULT_CONSOLE_CORS_ALLOWED_ORIGINS;
+/// Empty means the S3 endpoint emits no generic CORS headers unless configured.
+pub const DEFAULT_CORS_ALLOWED_ORIGINS: &str = "";
 
 /// CORS allowed origins for the console service
 /// Comma-separated list of origins or "*" for all origins
 pub const ENV_CONSOLE_CORS_ALLOWED_ORIGINS: &str = "RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS";
 
-/// Default CORS allowed origins for the console service
-pub const DEFAULT_CONSOLE_CORS_ALLOWED_ORIGINS: &str = "*";
+/// Default CORS allowed origins for the console service.
+///
+/// Empty string means same-origin only — no `Access-Control-Allow-Origin`
+/// header is emitted, so browsers will not allow cross-origin reads of
+/// console responses by default. Operators that need cross-origin access set
+/// `RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS` to a comma-separated allow-list, or
+/// to `*` to keep the previous permissive behavior.
+pub const DEFAULT_CONSOLE_CORS_ALLOWED_ORIGINS: &str = "";
 
 /// Enable or disable the console service
 pub const ENV_CONSOLE_ENABLE: &str = "RUSTFS_CONSOLE_ENABLE";
@@ -89,3 +94,20 @@ pub const ENV_UPDATE_CHECK: &str = "RUSTFS_CHECK_UPDATE";
 
 /// Default value for update toggle
 pub const DEFAULT_UPDATE_CHECK: bool = true;
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn endpoint_cors_default_is_restrictive() {
+        assert_eq!(ENV_CORS_ALLOWED_ORIGINS, "RUSTFS_CORS_ALLOWED_ORIGINS");
+        assert_eq!(DEFAULT_CORS_ALLOWED_ORIGINS, "");
+    }
+
+    #[test]
+    fn console_cors_default_is_same_origin_only() {
+        assert_eq!(ENV_CONSOLE_CORS_ALLOWED_ORIGINS, "RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS");
+        assert_eq!(DEFAULT_CONSOLE_CORS_ALLOWED_ORIGINS, "");
+    }
+}

crates/config/src/constants/drive.rs

@@ -0,0 +1,68 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// Legacy global drive timeout fallback.
+/// Deprecated in favor of per-operation drive timeout knobs.
+pub const ENV_DRIVE_MAX_TIMEOUT_DURATION: &str = "RUSTFS_DRIVE_MAX_TIMEOUT_DURATION";
+
+/// Default timeout in seconds for the legacy global drive timeout fallback.
+pub const DEFAULT_DRIVE_MAX_TIMEOUT_DURATION_SECS: u64 = 30;
+
+/// Timeout for metadata-oriented drive operations such as `read_metadata`.
+pub const ENV_DRIVE_METADATA_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_METADATA_TIMEOUT_SECS";
+pub const DEFAULT_DRIVE_METADATA_TIMEOUT_SECS: u64 = 5;
+
+/// Timeout for `disk_info()` calls on local and remote drives.
+pub const ENV_DRIVE_DISK_INFO_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_DISK_INFO_TIMEOUT_SECS";
+pub const DEFAULT_DRIVE_DISK_INFO_TIMEOUT_SECS: u64 = 5;
+
+/// Timeout for `list_dir()` style metadata listing operations.
+pub const ENV_DRIVE_LIST_DIR_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_LIST_DIR_TIMEOUT_SECS";
+pub const DEFAULT_DRIVE_LIST_DIR_TIMEOUT_SECS: u64 = 5;
+
+/// Total timeout for `walk_dir()` operations.
+pub const ENV_DRIVE_WALKDIR_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS";
+pub const DEFAULT_DRIVE_WALKDIR_TIMEOUT_SECS: u64 = 5;
+
+/// Maximum time without forward progress while consuming a `walk_dir()` stream.
+pub const ENV_DRIVE_WALKDIR_STALL_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS";
+pub const DEFAULT_DRIVE_WALKDIR_STALL_TIMEOUT_SECS: u64 = 5;
+
+/// Interval in seconds between active health probes for local and remote drives.
+pub const ENV_DRIVE_ACTIVE_CHECK_INTERVAL_SECS: &str = "RUSTFS_DRIVE_ACTIVE_CHECK_INTERVAL_SECS";
+pub const DEFAULT_DRIVE_ACTIVE_CHECK_INTERVAL_SECS: u64 = 15;
+
+/// Timeout in seconds for a single active health probe.
+pub const ENV_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS";
+pub const DEFAULT_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS: u64 = 5;
+
+/// Number of consecutive failures before a suspect drive is classified as offline.
+pub const ENV_DRIVE_SUSPECT_FAILURE_THRESHOLD: &str = "RUSTFS_DRIVE_SUSPECT_FAILURE_THRESHOLD";
+pub const DEFAULT_DRIVE_SUSPECT_FAILURE_THRESHOLD: u64 = 2;
+
+/// Number of consecutive successful recovery probes before a returning drive is considered online again.
+pub const ENV_DRIVE_RETURNING_SUCCESS_THRESHOLD: &str = "RUSTFS_DRIVE_RETURNING_SUCCESS_THRESHOLD";
+pub const DEFAULT_DRIVE_RETURNING_SUCCESS_THRESHOLD: u64 = 3;
+
+/// Probe interval in seconds while a drive is in the recovery path.
+pub const ENV_DRIVE_RETURNING_PROBE_INTERVAL_SECS: &str = "RUSTFS_DRIVE_RETURNING_PROBE_INTERVAL_SECS";
+pub const DEFAULT_DRIVE_RETURNING_PROBE_INTERVAL_SECS: u64 = 2;
+
+/// Duration in seconds for classifying a recovered drive as a short offline event.
+pub const ENV_DRIVE_OFFLINE_GRACE_PERIOD_SECS: &str = "RUSTFS_DRIVE_OFFLINE_GRACE_PERIOD_SECS";
+pub const DEFAULT_DRIVE_OFFLINE_GRACE_PERIOD_SECS: u64 = 30;
+
+/// Duration in seconds after which a recovered drive is classified as long offline.
+pub const ENV_DRIVE_LONG_OFFLINE_THRESHOLD_SECS: &str = "RUSTFS_DRIVE_LONG_OFFLINE_THRESHOLD_SECS";
+pub const DEFAULT_DRIVE_LONG_OFFLINE_THRESHOLD_SECS: u64 = 172_800;

crates/config/src/constants/env.rs

@@ -26,6 +26,21 @@ pub const RUSTFS_WEBHOOK_SKIP_TLS_VERIFY_DEFAULT: bool = false;
 pub const ENABLE_KEY: &str = "enable";
 pub const COMMENT_KEY: &str = "comment";
 
+/// Global switch for enabling the audit module.
+pub const ENV_AUDIT_ENABLE: &str = "RUSTFS_AUDIT_ENABLE";
+/// Global switch for enabling the notify module.
+pub const ENV_NOTIFY_ENABLE: &str = "RUSTFS_NOTIFY_ENABLE";
+/// Default global audit switch (disabled by default).
+pub const DEFAULT_AUDIT_ENABLE: bool = false;
+/// Default global notify switch (disabled by default).
+pub const DEFAULT_NOTIFY_ENABLE: bool = false;
+/// Canonical ILM process boundary env key (seconds).
+pub const ENV_ILM_PROCESS_TIME: &str = "RUSTFS_ILM_PROCESS_TIME";
+/// Deprecated ILM process boundary env key kept for compatibility.
+pub const ENV_ILM_PROCESS_TIME_DEPRECATED: &str = "_RUSTFS_ILM_PROCESS_TIME";
+/// Default ILM process boundary in seconds (24h).
+pub const DEFAULT_ILM_PROCESS_TIME_SECS: i32 = 86400;
+
 /// Medium-drawn lines separator
 /// This is used to separate words in environment variable names.
 pub const ENV_WORD_DELIMITER_DASH: &str = "-";
@@ -290,4 +305,10 @@ mod tests {
             assert!(state.is_disabled());
         }
     }
+
+    #[test]
+    fn test_global_audit_notify_switch_constants() {
+        assert_eq!(ENV_AUDIT_ENABLE, "RUSTFS_AUDIT_ENABLE");
+        assert_eq!(ENV_NOTIFY_ENABLE, "RUSTFS_NOTIFY_ENABLE");
+    }
 }

crates/config/src/constants/heal.rs

@@ -56,6 +56,15 @@ pub const ENV_HEAL_TASK_TIMEOUT_SECS: &str = "RUSTFS_HEAL_TASK_TIMEOUT_SECS";
 /// - Note: A higher concurrency limit can speed up healing but may lead to resource contention.
 pub const ENV_HEAL_MAX_CONCURRENT_HEALS: &str = "RUSTFS_HEAL_MAX_CONCURRENT_HEALS";
 
+/// Environment variable name that specifies the maximum number of concurrent heal operations
+/// allowed for a single erasure set.
+///
+/// - Purpose: Prevent one degraded set from consuming all global heal slots.
+/// - Unit: number of operations (usize).
+/// - Valid values: any positive integer.
+/// - Example: `export RUSTFS_HEAL_MAX_CONCURRENT_PER_SET=1`
+pub const ENV_HEAL_MAX_CONCURRENT_PER_SET: &str = "RUSTFS_HEAL_MAX_CONCURRENT_PER_SET";
+
 /// Default value for enabling authentication for heal operations if not specified in the environment variable.
 /// - Value: true (authentication enabled).
 /// - Rationale: Enabling authentication by default enhances security for heal operations.
@@ -86,3 +95,46 @@ pub const DEFAULT_HEAL_TASK_TIMEOUT_SECS: u64 = 300; // 5 minutes
 /// - Rationale: This default concurrency limit helps balance healing speed with resource usage, preventing system overload.
 /// - Adjustments: Users may modify this value via the `RUSTFS_HEAL_MAX_CONCURRENT_HEALS` environment variable based on their system capacity and expected heal workload.
 pub const DEFAULT_HEAL_MAX_CONCURRENT_HEALS: usize = 4;
+
+/// Default maximum number of concurrent heal operations per erasure set.
+///
+/// - Value: 1 concurrent heal operation per set.
+/// - Rationale: Keeps a degraded set from monopolizing the global heal scheduler.
+pub const DEFAULT_HEAL_MAX_CONCURRENT_PER_SET: usize = 1;
+
+/// Environment variable that controls whether low-priority heal requests should merge into
+/// an existing queued request with the same deduplication key.
+pub const ENV_HEAL_LOW_PRIORITY_MERGE_ENABLE: &str = "RUSTFS_HEAL_LOW_PRIORITY_MERGE_ENABLE";
+
+/// Environment variable that allows low-priority heal requests to be dropped when the queue is full.
+pub const ENV_HEAL_LOW_PRIORITY_DROP_WHEN_FULL: &str = "RUSTFS_HEAL_LOW_PRIORITY_DROP_WHEN_FULL";
+
+/// Environment variable that controls concurrent object heals within a single erasure-set page.
+pub const ENV_HEAL_PAGE_OBJECT_CONCURRENCY: &str = "RUSTFS_HEAL_PAGE_OBJECT_CONCURRENCY";
+
+/// Environment variable that toggles notify-driven scheduler wakeups.
+pub const ENV_HEAL_EVENT_DRIVEN_SCHEDULER_ENABLE: &str = "RUSTFS_HEAL_EVENT_DRIVEN_SCHEDULER_ENABLE";
+
+/// Environment variable that toggles per-set bulkhead scheduling.
+pub const ENV_HEAL_SET_BULKHEAD_ENABLE: &str = "RUSTFS_HEAL_SET_BULKHEAD_ENABLE";
+
+/// Environment variable that toggles page-level parallel object healing for erasure-set repair.
+pub const ENV_HEAL_PAGE_PARALLEL_ENABLE: &str = "RUSTFS_HEAL_PAGE_PARALLEL_ENABLE";
+
+/// Default behavior is to merge duplicate low-priority requests.
+pub const DEFAULT_HEAL_LOW_PRIORITY_MERGE_ENABLE: bool = true;
+
+/// Default behavior is to drop low-priority requests instead of blocking when the queue is full.
+pub const DEFAULT_HEAL_LOW_PRIORITY_DROP_WHEN_FULL: bool = true;
+
+/// Default per-page object heal concurrency for erasure-set healing.
+pub const DEFAULT_HEAL_PAGE_OBJECT_CONCURRENCY: usize = 8;
+
+/// Default behavior is to keep notify-driven scheduler wakeups enabled.
+pub const DEFAULT_HEAL_EVENT_DRIVEN_SCHEDULER_ENABLE: bool = true;
+
+/// Default behavior is to keep per-set bulkhead scheduling enabled.
+pub const DEFAULT_HEAL_SET_BULKHEAD_ENABLE: bool = true;
+
+/// Default behavior is to keep erasure-set page parallelism enabled.
+pub const DEFAULT_HEAL_PAGE_PARALLEL_ENABLE: bool = true;

crates/config/src/constants/health.rs

@@ -0,0 +1,28 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// Enable or disable public `/health` and `/health/ready` endpoints.
+/// When disabled, the routes are not registered and return 404.
+pub const ENV_HEALTH_ENDPOINT_ENABLE: &str = "RUSTFS_HEALTH_ENDPOINT_ENABLE";
+pub const DEFAULT_HEALTH_ENDPOINT_ENABLE: bool = true;
+
+/// Cache TTL for storage readiness runtime-state evaluation (milliseconds).
+/// This reduces storage-layer pressure when probes are called at high frequency.
+pub const ENV_HEALTH_READINESS_CACHE_TTL_MS: &str = "RUSTFS_HEALTH_READINESS_CACHE_TTL_MS";
+pub const DEFAULT_HEALTH_READINESS_CACHE_TTL_MS: u64 = 1000;
+
+/// Enable minimal health payload mode for GET `/health*` responses.
+/// When enabled, only `status` and `ready` fields are returned.
+pub const ENV_HEALTH_MINIMAL_RESPONSE_ENABLE: &str = "RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE";
+pub const DEFAULT_HEALTH_MINIMAL_RESPONSE_ENABLE: bool = false;

crates/config/src/constants/internode.rs

@@ -0,0 +1,62 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// Timeout for establishing a new internode gRPC connection.
+pub const ENV_INTERNODE_CONNECT_TIMEOUT_SECS: &str = "RUSTFS_INTERNODE_CONNECT_TIMEOUT_SECS";
+pub const DEFAULT_INTERNODE_CONNECT_TIMEOUT_SECS: u64 = 3;
+
+/// TCP keepalive interval for internode gRPC channels.
+pub const ENV_INTERNODE_TCP_KEEPALIVE_SECS: &str = "RUSTFS_INTERNODE_TCP_KEEPALIVE_SECS";
+pub const DEFAULT_INTERNODE_TCP_KEEPALIVE_SECS: u64 = 10;
+
+/// HTTP/2 keepalive interval for internode gRPC channels.
+pub const ENV_INTERNODE_HTTP2_KEEPALIVE_INTERVAL_SECS: &str = "RUSTFS_INTERNODE_HTTP2_KEEPALIVE_INTERVAL_SECS";
+pub const DEFAULT_INTERNODE_HTTP2_KEEPALIVE_INTERVAL_SECS: u64 = 5;
+
+/// HTTP/2 keepalive timeout for internode gRPC channels.
+pub const ENV_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS: &str = "RUSTFS_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS";
+pub const DEFAULT_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS: u64 = 3;
+
+/// Overall timeout for a single internode gRPC request.
+pub const ENV_INTERNODE_RPC_TIMEOUT_SECS: &str = "RUSTFS_INTERNODE_RPC_TIMEOUT_SECS";
+pub const DEFAULT_INTERNODE_RPC_TIMEOUT_SECS: u64 = 10;
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn internode_timeout_defaults_stay_in_expected_bounds() {
+        assert_eq!(DEFAULT_INTERNODE_CONNECT_TIMEOUT_SECS, 3);
+        assert_eq!(DEFAULT_INTERNODE_TCP_KEEPALIVE_SECS, 10);
+        assert_eq!(DEFAULT_INTERNODE_HTTP2_KEEPALIVE_INTERVAL_SECS, 5);
+        assert_eq!(DEFAULT_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS, 3);
+        assert_eq!(DEFAULT_INTERNODE_RPC_TIMEOUT_SECS, 10);
+    }
+
+    #[test]
+    fn internode_timeout_env_names_are_stable() {
+        assert_eq!(ENV_INTERNODE_CONNECT_TIMEOUT_SECS, "RUSTFS_INTERNODE_CONNECT_TIMEOUT_SECS");
+        assert_eq!(ENV_INTERNODE_TCP_KEEPALIVE_SECS, "RUSTFS_INTERNODE_TCP_KEEPALIVE_SECS");
+        assert_eq!(
+            ENV_INTERNODE_HTTP2_KEEPALIVE_INTERVAL_SECS,
+            "RUSTFS_INTERNODE_HTTP2_KEEPALIVE_INTERVAL_SECS"
+        );
+        assert_eq!(
+            ENV_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS,
+            "RUSTFS_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS"
+        );
+        assert_eq!(ENV_INTERNODE_RPC_TIMEOUT_SECS, "RUSTFS_INTERNODE_RPC_TIMEOUT_SECS");
+    }
+}

crates/config/src/constants/mod.rs

@@ -17,8 +17,11 @@ pub(crate) mod body_limits;
 pub(crate) mod capacity;
 pub(crate) mod compress;
 pub(crate) mod console;
+pub(crate) mod drive;
 pub(crate) mod env;
 pub(crate) mod heal;
+pub(crate) mod health;
+pub(crate) mod internode;
 pub(crate) mod object;
 pub(crate) mod oidc;
 pub(crate) mod profiler;

crates/config/src/constants/oidc.rs

@@ -17,6 +17,7 @@ pub const OIDC_CONFIG_URL: &str = "config_url";
 pub const OIDC_CLIENT_ID: &str = "client_id";
 pub const OIDC_CLIENT_SECRET: &str = "client_secret";
 pub const OIDC_SCOPES: &str = "scopes";
+pub const OIDC_OTHER_AUDIENCES: &str = "other_audiences";
 pub const OIDC_REDIRECT_URI: &str = "redirect_uri";
 pub const OIDC_REDIRECT_URI_DYNAMIC: &str = "redirect_uri_dynamic";
 pub const OIDC_CLAIM_NAME: &str = "claim_name";
@@ -34,6 +35,7 @@ pub const ENV_IDENTITY_OPENID_CONFIG_URL: &str = "RUSTFS_IDENTITY_OPENID_CONFIG_
 pub const ENV_IDENTITY_OPENID_CLIENT_ID: &str = "RUSTFS_IDENTITY_OPENID_CLIENT_ID";
 pub const ENV_IDENTITY_OPENID_CLIENT_SECRET: &str = "RUSTFS_IDENTITY_OPENID_CLIENT_SECRET";
 pub const ENV_IDENTITY_OPENID_SCOPES: &str = "RUSTFS_IDENTITY_OPENID_SCOPES";
+pub const ENV_IDENTITY_OPENID_OTHER_AUDIENCES: &str = "RUSTFS_IDENTITY_OPENID_OTHER_AUDIENCES";
 pub const ENV_IDENTITY_OPENID_REDIRECT_URI: &str = "RUSTFS_IDENTITY_OPENID_REDIRECT_URI";
 pub const ENV_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC: &str = "RUSTFS_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC";
 pub const ENV_IDENTITY_OPENID_CLAIM_NAME: &str = "RUSTFS_IDENTITY_OPENID_CLAIM_NAME";
@@ -46,12 +48,13 @@ pub const ENV_IDENTITY_OPENID_EMAIL_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_EMAIL_
 pub const ENV_IDENTITY_OPENID_USERNAME_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_USERNAME_CLAIM";
 
 /// List of all environment variable keys for an OIDC provider.
-pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 15] = &[
+pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 16] = &[
     ENV_IDENTITY_OPENID_ENABLE,
     ENV_IDENTITY_OPENID_CONFIG_URL,
     ENV_IDENTITY_OPENID_CLIENT_ID,
     ENV_IDENTITY_OPENID_CLIENT_SECRET,
     ENV_IDENTITY_OPENID_SCOPES,
+    ENV_IDENTITY_OPENID_OTHER_AUDIENCES,
     ENV_IDENTITY_OPENID_REDIRECT_URI,
     ENV_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC,
     ENV_IDENTITY_OPENID_CLAIM_NAME,
@@ -71,6 +74,7 @@ pub const IDENTITY_OPENID_KEYS: &[&str] = &[
     OIDC_CLIENT_ID,
     OIDC_CLIENT_SECRET,
     OIDC_SCOPES,
+    OIDC_OTHER_AUDIENCES,
     OIDC_REDIRECT_URI,
     OIDC_REDIRECT_URI_DYNAMIC,
     OIDC_CLAIM_NAME,

crates/config/src/constants/protocols.rs

@@ -57,3 +57,114 @@ pub const ENV_WEBDAV_CERTS_DIR: &str = "RUSTFS_WEBDAV_CERTS_DIR";
 pub const ENV_WEBDAV_CA_FILE: &str = "RUSTFS_WEBDAV_CA_FILE";
 pub const ENV_WEBDAV_MAX_BODY_SIZE: &str = "RUSTFS_WEBDAV_MAX_BODY_SIZE";
 pub const ENV_WEBDAV_REQUEST_TIMEOUT: &str = "RUSTFS_WEBDAV_REQUEST_TIMEOUT";
+
+/// Default SFTP server bind address.
+pub const DEFAULT_SFTP_ADDRESS: &str = "0.0.0.0:2222";
+
+/// Default for SFTP host-key directory. None means no default. Operators
+/// must set RUSTFS_SFTP_HOST_KEY_DIR explicitly when SFTP is enabled.
+pub const DEFAULT_SFTP_HOST_KEY_DIR: Option<&str> = None;
+
+/// SFTP environment variable names.
+pub const ENV_SFTP_ENABLE: &str = "RUSTFS_SFTP_ENABLE";
+pub const ENV_SFTP_ADDRESS: &str = "RUSTFS_SFTP_ADDRESS";
+pub const ENV_SFTP_HOST_KEY_DIR: &str = "RUSTFS_SFTP_HOST_KEY_DIR";
+pub const ENV_SFTP_HOST_KEY_RELOAD_ENABLE: &str = "RUSTFS_SFTP_HOST_KEY_RELOAD_ENABLE";
+pub const ENV_SFTP_HOST_KEY_RELOAD_INTERVAL: &str = "RUSTFS_SFTP_HOST_KEY_RELOAD_INTERVAL";
+pub const ENV_SFTP_IDLE_TIMEOUT: &str = "RUSTFS_SFTP_IDLE_TIMEOUT";
+/// S3 multipart part size in bytes. Default DEFAULT_SFTP_PART_SIZE (16 MiB).
+/// Valid range 5 MiB to 5 GiB (S3 protocol bounds), enforced at startup.
+///
+/// The per-upload size ceiling is part_size * 10_000 (the S3 parts cap),
+/// so the default caps single uploads at 160 GiB. Deployments expecting
+/// larger single files must raise this: 64 MiB -> 640 GiB, 128 MiB ->
+/// 1.25 TiB, 512 MiB -> 5 TiB (S3 object max). Rename is not affected;
+/// multipart_copy scales the per-part size dynamically and handles up
+/// to the 5 TiB S3 object limit regardless of this setting.
+pub const ENV_SFTP_PART_SIZE: &str = "RUSTFS_SFTP_PART_SIZE";
+pub const ENV_SFTP_READ_ONLY: &str = "RUSTFS_SFTP_READ_ONLY";
+pub const ENV_SFTP_BANNER: &str = "RUSTFS_SFTP_BANNER";
+/// Optional environment variable. If RUSTFS_SFTP_HANDLES_PER_SESSION
+/// is not set in the process environment, the server uses the default
+/// of 64 handles per session and emits no warning. If set, the value
+/// must be in the inclusive range 8 to 1024. Out-of-range values fall
+/// back to the default of 64 with a warn-level log naming the
+/// requested value and the bounds.
+///
+/// Caps the maximum number of simultaneously-open SFTP handles per
+/// session. A handle is the server-side identifier returned by
+/// SSH_FXP_OPEN and SSH_FXP_OPENDIR. One client typically uses one
+/// handle per file in flight plus one per directory listing.
+/// Operators running clients with deep pipelining may raise this.
+pub const ENV_SFTP_HANDLES_PER_SESSION: &str = "RUSTFS_SFTP_HANDLES_PER_SESSION";
+
+/// Optional environment variable. If RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS
+/// is not set in the process environment, the server uses the default
+/// of 60 seconds and emits no warning. If set, the value must be in
+/// the inclusive range 5 to 600 seconds. Out-of-range values fall
+/// back to the default with a warn-level log naming the requested
+/// value and the bounds.
+///
+/// Bounds every storage backend call issued by the SFTP driver. A
+/// backend that does not respond within this many seconds returns
+/// Failure to the client and emits a warn log naming the backend
+/// method. This catches a backend that accepted the request and never
+/// returned a body, which the SSH keepalive cannot detect because the
+/// transport itself remains live.
+pub const ENV_SFTP_BACKEND_OP_TIMEOUT_SECS: &str = "RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS";
+
+/// Optional environment variable. If RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES
+/// is not set in the process environment, the server uses a 4 MiB
+/// default and emits no warning. If set, the value must be in the
+/// inclusive range MAX_READ_LEN (256 KiB) to 64 MiB. Out-of-range
+/// values fall back to the default with a warn-level log naming the
+/// requested value and the bounds.
+///
+/// Per-handle byte window the SFTP read path fetches in one backend
+/// call on a cache miss. Subsequent FXP_READs within that window are
+/// served from the buffer without a backend round trip. For
+/// sequential downloads the backend round-trip count drops by
+/// window_bytes / MAX_READ_LEN. Random-access workloads should set
+/// the window equal to MAX_READ_LEN to opt out of read-ahead.
+pub const ENV_SFTP_READ_CACHE_WINDOW_BYTES: &str = "RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES";
+
+/// Optional environment variable. If
+/// RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES is not set in the process
+/// environment, the server uses a 256 MiB default and emits no
+/// warning. If set, the value must be at least 16 MiB. Below-min
+/// values fall back to the default with a warn-level log naming the
+/// requested value and the bound.
+///
+/// Process-wide ceiling on cumulative read cache memory across every
+/// live SFTP handle. Once the accumulator plus a new window would
+/// exceed this value, the populate call on the per-handle cache is
+/// skipped. The read still completes from the freshly-fetched bytes
+/// without storing them in the cache, at the cost of one backend
+/// call per FXP_READ. High-concurrency deployments expecting many
+/// parallel downloads should raise this in step with the per-session
+/// handle cap.
+pub const ENV_SFTP_READ_CACHE_TOTAL_MEM_BYTES: &str = "RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES";
+
+/// Default idle session timeout in seconds.
+pub const DEFAULT_SFTP_IDLE_TIMEOUT: u64 = 600;
+
+/// Default SFTP host key hot reload enabled state.
+pub const DEFAULT_SFTP_HOST_KEY_RELOAD_ENABLE: bool = false;
+
+/// Default SFTP host key hot reload interval in seconds.
+pub const DEFAULT_SFTP_HOST_KEY_RELOAD_INTERVAL: u64 = 30;
+
+/// Default S3 multipart upload part size in bytes (16 MiB).
+///
+/// The per-upload size ceiling is part_size * 10_000 (the S3 parts cap),
+/// so the default gives a 160 GiB single-upload limit. Deployments that
+/// expect single files larger than this must raise part_size:
+/// 64 MiB -> 640 GiB, 128 MiB -> 1.25 TiB, 512 MiB -> 5 TiB (S3 max).
+/// The minimum is 5 MiB and the maximum is 5 GiB (S3 protocol bounds).
+pub const DEFAULT_SFTP_PART_SIZE: u64 = 16_777_216;
+
+/// Default read-only mode (disabled).
+pub const DEFAULT_SFTP_READ_ONLY: bool = false;
+
+/// Default SSH identification string (no version disclosure).
+pub const DEFAULT_SFTP_BANNER: &str = "SSH-2.0-RustFS";

crates/config/src/constants/proxy.rs

@@ -20,6 +20,11 @@ pub const ENV_TRUSTED_PROXY_ENABLED: &str = "RUSTFS_TRUSTED_PROXY_ENABLED";
 /// Trusted proxy middleware is enabled by default.
 pub const DEFAULT_TRUSTED_PROXY_ENABLED: bool = true;
 
+/// Environment variable to select the trusted proxy implementation.
+pub const ENV_TRUSTED_PROXY_IMPLEMENTATION: &str = "RUSTFS_TRUSTED_PROXY_IMPLEMENTATION";
+/// The simplified implementation is used by default.
+pub const DEFAULT_TRUSTED_PROXY_IMPLEMENTATION: &str = "simple";
+
 /// Environment variable for the proxy validation mode.
 pub const ENV_TRUSTED_PROXY_VALIDATION_MODE: &str = "RUSTFS_TRUSTED_PROXY_VALIDATION_MODE";
 /// Default validation mode is "hop_by_hop".

crates/config/src/constants/runtime.rs

@@ -59,9 +59,31 @@ pub const DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT: usize = 10;
 pub const DEFAULT_RUNTIME_DIAL9_SAMPLING_RATE: f64 = 1.0; // 100% sampling
 // Note: S3 bucket/prefix have no default; absence means upload is disabled (modeled as Option<String>)
 
-/// Threshold for small object seek support in megabytes.
+// Allocator reclaim configuration
+pub const ENV_ALLOCATOR_RECLAIM_ENABLED: &str = "RUSTFS_ALLOCATOR_RECLAIM_ENABLED";
+pub const ENV_ALLOCATOR_RECLAIM_INTERVAL_SECS: &str = "RUSTFS_ALLOCATOR_RECLAIM_INTERVAL_SECS";
+pub const ENV_ALLOCATOR_RECLAIM_FORCE: &str = "RUSTFS_ALLOCATOR_RECLAIM_FORCE";
+pub const ENV_ALLOCATOR_RECLAIM_IDLE_INTERVALS: &str = "RUSTFS_ALLOCATOR_RECLAIM_IDLE_INTERVALS";
+pub const DEFAULT_ALLOCATOR_RECLAIM_ENABLED: bool = false;
+pub const DEFAULT_ALLOCATOR_RECLAIM_INTERVAL_SECS: u64 = 30;
+pub const DEFAULT_ALLOCATOR_RECLAIM_FORCE: bool = true;
+pub const DEFAULT_ALLOCATOR_RECLAIM_IDLE_INTERVALS: u64 = 3;
+
+// File page-cache reclaim configuration
+pub const ENV_OBJECT_FILE_CACHE_RECLAIM_WRITE_ENABLE: &str = "RUSTFS_OBJECT_FILE_CACHE_RECLAIM_WRITE_ENABLE";
+pub const ENV_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE: &str = "RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE";
+pub const ENV_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD: &str = "RUSTFS_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD";
+pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_WRITE_ENABLE: bool = false;
+pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE: bool = false;
+pub const DEFAULT_OBJECT_FILE_CACHE_RECLAIM_THRESHOLD: usize = 4 * 1024 * 1024;
+
+/// Threshold for small object seek support in bytes.
 ///
-/// When an object is smaller than this size, rustfs will provide seek support.
+/// When an object response is smaller than this size, rustfs may provide
+/// in-memory seek support. Runtime GET logic also enforces a hard safety cap
+/// (`64 MiB`) to prevent large-download memory spikes even if this threshold
+/// is configured higher.
 ///
-/// Default is set to 10MB.
+/// Default is set to 10 MiB.
+pub const ENV_OBJECT_SEEK_SUPPORT_THRESHOLD: &str = "RUSTFS_OBJECT_SEEK_SUPPORT_THRESHOLD";
 pub const DEFAULT_OBJECT_SEEK_SUPPORT_THRESHOLD: usize = 10 * 1024 * 1024;

crates/config/src/constants/scanner.rs

@@ -25,6 +25,12 @@ pub const ENV_SCANNER_START_DELAY_SECS: &str = "RUSTFS_SCANNER_START_DELAY_SECS"
 #[deprecated(note = "Use RUSTFS_SCANNER_START_DELAY_SECS instead")]
 pub const ENV_DATA_SCANNER_START_DELAY_SECS: &str = "RUSTFS_DATA_SCANNER_START_DELAY_SECS";
 
+/// Environment variable that specifies the scanner cycle interval in seconds.
+/// If set, this overrides the cycle interval derived from `RUSTFS_SCANNER_SPEED`.
+/// - Unit: seconds (u64).
+/// - Example: `export RUSTFS_SCANNER_CYCLE=3600` (1 hour)
+pub const ENV_SCANNER_CYCLE: &str = "RUSTFS_SCANNER_CYCLE";
+
 /// Environment variable that selects the scanner speed preset.
 /// Valid values: `fastest`, `fast`, `default`, `slow`, `slowest`.
 /// Controls the sleep factor, maximum sleep duration, and cycle interval.
@@ -39,9 +45,24 @@ pub const DEFAULT_SCANNER_SPEED: &str = "default";
 /// - Example: `export RUSTFS_SCANNER_IDLE_MODE=false`
 pub const ENV_SCANNER_IDLE_MODE: &str = "RUSTFS_SCANNER_IDLE_MODE";
 
+/// Environment variable that controls scanner cache save timeout in seconds.
+/// The scanner enforces a minimum value of `1`.
+/// - Unit: seconds (u64).
+/// - Example: `export RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS=30`
+pub const ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS: &str = "RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS";
+
 /// Default scanner idle mode.
 pub const DEFAULT_SCANNER_IDLE_MODE: bool = true;
 
+/// Compatibility flag kept for Patch 3 rollback windows.
+///
+/// Inline scanner heal execution has been removed in favor of heal-candidate enqueue.
+/// When this flag is enabled, RustFS logs a warning and continues to use enqueue-based heal.
+pub const ENV_SCANNER_INLINE_HEAL_ENABLE: &str = "RUSTFS_SCANNER_INLINE_HEAL_ENABLE";
+
+/// Default inline scanner heal compatibility mode.
+pub const DEFAULT_SCANNER_INLINE_HEAL_ENABLE: bool = false;
+
 /// Scanner speed preset controlling throttling behavior.
 ///
 /// Each preset defines three parameters:
@@ -56,6 +77,8 @@ pub const DEFAULT_SCANNER_IDLE_MODE: bool = true;
 /// | `default` | 2x     | 1 second  | 1 minute       |
 /// | `slow`    | 10x    | 15 seconds| 1 minute       |
 /// | `slowest` | 100x   | 15 seconds| 30 minutes     |
+///
+/// The cycle interval can be overridden by `RUSTFS_SCANNER_CYCLE`.
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
 pub enum ScannerSpeed {
     Fastest,

crates/config/src/constants/targets.rs

@@ -40,6 +40,94 @@ pub const MQTT_TLS_CLIENT_CERT: &str = "tls_client_cert";
 pub const MQTT_TLS_CLIENT_KEY: &str = "tls_client_key";
 pub const MQTT_TLS_TRUST_LEAF_AS_CA: &str = "tls_trust_leaf_as_ca";
 pub const MQTT_WS_PATH_ALLOWLIST: &str = "ws_path_allowlist";
+pub const KAFKA_BROKERS: &str = "brokers";
+pub const KAFKA_TOPIC: &str = "topic";
+pub const KAFKA_ACKS: &str = "acks";
+pub const KAFKA_QUEUE_DIR: &str = "queue_dir";
+pub const KAFKA_QUEUE_LIMIT: &str = "queue_limit";
+pub const KAFKA_TLS_ENABLE: &str = "tls_enable";
+pub const KAFKA_TLS_CA: &str = "tls_ca";
+pub const KAFKA_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const KAFKA_TLS_CLIENT_KEY: &str = "tls_client_key";
+
+pub const AMQP_URL: &str = "url";
+pub const AMQP_EXCHANGE: &str = "exchange";
+pub const AMQP_ROUTING_KEY: &str = "routing_key";
+pub const AMQP_MANDATORY: &str = "mandatory";
+pub const AMQP_PERSISTENT: &str = "persistent";
+pub const AMQP_USERNAME: &str = "username";
+pub const AMQP_PASSWORD: &str = "password";
+pub const AMQP_TLS_CA: &str = "tls_ca";
+pub const AMQP_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const AMQP_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const AMQP_QUEUE_DIR: &str = "queue_dir";
+pub const AMQP_QUEUE_LIMIT: &str = "queue_limit";
+
+pub const NATS_ADDRESS: &str = "address";
+pub const NATS_SUBJECT: &str = "subject";
+pub const NATS_USERNAME: &str = "username";
+pub const NATS_PASSWORD: &str = "password";
+pub const NATS_TOKEN: &str = "token";
+pub const NATS_CREDENTIALS_FILE: &str = "credentials_file";
+pub const NATS_TLS_CA: &str = "tls_ca";
+pub const NATS_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const NATS_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const NATS_TLS_REQUIRED: &str = "tls_required";
+pub const NATS_QUEUE_DIR: &str = "queue_dir";
+pub const NATS_QUEUE_LIMIT: &str = "queue_limit";
+
+pub const PULSAR_BROKER: &str = "broker";
+pub const PULSAR_TOPIC: &str = "topic";
+pub const PULSAR_AUTH_TOKEN: &str = "auth_token";
+pub const PULSAR_USERNAME: &str = "username";
+pub const PULSAR_PASSWORD: &str = "password";
+pub const PULSAR_TLS_CA: &str = "tls_ca";
+pub const PULSAR_TLS_ALLOW_INSECURE: &str = "tls_allow_insecure";
+pub const PULSAR_TLS_HOSTNAME_VERIFICATION: &str = "tls_hostname_verification";
+pub const PULSAR_QUEUE_DIR: &str = "queue_dir";
+pub const PULSAR_QUEUE_LIMIT: &str = "queue_limit";
+
+pub const BASE_DSN_STRING: &str = "dsn_string";
+
+pub const MYSQL_DSN_STRING: &str = BASE_DSN_STRING;
+pub const MYSQL_TABLE: &str = "table";
+pub const MYSQL_FORMAT: &str = "format";
+pub const MYSQL_TLS_CA: &str = "tls_ca";
+pub const MYSQL_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const MYSQL_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const MYSQL_QUEUE_DIR: &str = "queue_dir";
+pub const MYSQL_QUEUE_LIMIT: &str = "queue_limit";
+pub const MYSQL_MAX_OPEN_CONNECTIONS: &str = "max_open_connections";
+
+pub const REDIS_URL: &str = "url";
+pub const REDIS_CHANNEL: &str = "channel";
+pub const REDIS_USERNAME: &str = "username";
+pub const REDIS_PASSWORD: &str = "password";
+pub const REDIS_KEEP_ALIVE_INTERVAL: &str = "keep_alive_interval";
+pub const REDIS_QUEUE_DIR: &str = "queue_dir";
+pub const REDIS_QUEUE_LIMIT: &str = "queue_limit";
+pub const REDIS_MAX_RETRY_ATTEMPTS: &str = "max_retry_attempts";
+pub const REDIS_RECONNECT_RETRY_ATTEMPTS: &str = "reconnect_retry_attempts";
+pub const REDIS_MIN_RETRY_DELAY: &str = "min_retry_delay";
+pub const REDIS_MAX_RETRY_DELAY: &str = "max_retry_delay";
+pub const REDIS_CONNECTION_TIMEOUT: &str = "connection_timeout";
+pub const REDIS_RESPONSE_TIMEOUT: &str = "response_timeout";
+pub const REDIS_PIPELINE_BUFFER_SIZE: &str = "pipeline_buffer_size";
+pub const REDIS_TLS_POLICY: &str = "tls_policy";
+pub const REDIS_TLS_CA: &str = "tls_ca";
+pub const REDIS_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const REDIS_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const REDIS_TLS_ALLOW_INSECURE: &str = "tls_allow_insecure";
+
+pub const POSTGRES_DSN_STRING: &str = BASE_DSN_STRING;
+pub const POSTGRES_TABLE: &str = "table";
+pub const POSTGRES_FORMAT: &str = "format";
+pub const POSTGRES_TLS_REQUIRED: &str = "tls_required";
+pub const POSTGRES_TLS_CA: &str = "tls_ca";
+pub const POSTGRES_TLS_CLIENT_CERT: &str = "tls_client_cert";
+pub const POSTGRES_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const POSTGRES_QUEUE_DIR: &str = "queue_dir";
+pub const POSTGRES_QUEUE_LIMIT: &str = "queue_limit";
 
 /// Environment variable controlling whether target queue files are Snappy-compressed.
 /// Applies to both notify and audit target queue stores.

crates/config/src/lib.rs

@@ -25,10 +25,16 @@ pub use constants::compress::*;
 #[cfg(feature = "constants")]
 pub use constants::console::*;
 #[cfg(feature = "constants")]
+pub use constants::drive::*;
+#[cfg(feature = "constants")]
 pub use constants::env::*;
 #[cfg(feature = "constants")]
 pub use constants::heal::*;
 #[cfg(feature = "constants")]
+pub use constants::health::*;
+#[cfg(feature = "constants")]
+pub use constants::internode::*;
+#[cfg(feature = "constants")]
 pub use constants::object::*;
 #[cfg(feature = "constants")]
 pub use constants::profiler::*;

crates/config/src/notify/amqp.rs

@@ -0,0 +1,60 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const NOTIFY_AMQP_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::AMQP_URL,
+    crate::AMQP_EXCHANGE,
+    crate::AMQP_ROUTING_KEY,
+    crate::AMQP_MANDATORY,
+    crate::AMQP_PERSISTENT,
+    crate::AMQP_USERNAME,
+    crate::AMQP_PASSWORD,
+    crate::AMQP_TLS_CA,
+    crate::AMQP_TLS_CLIENT_CERT,
+    crate::AMQP_TLS_CLIENT_KEY,
+    crate::AMQP_QUEUE_DIR,
+    crate::AMQP_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_NOTIFY_AMQP_ENABLE: &str = "RUSTFS_NOTIFY_AMQP_ENABLE";
+pub const ENV_NOTIFY_AMQP_URL: &str = "RUSTFS_NOTIFY_AMQP_URL";
+pub const ENV_NOTIFY_AMQP_EXCHANGE: &str = "RUSTFS_NOTIFY_AMQP_EXCHANGE";
+pub const ENV_NOTIFY_AMQP_ROUTING_KEY: &str = "RUSTFS_NOTIFY_AMQP_ROUTING_KEY";
+pub const ENV_NOTIFY_AMQP_MANDATORY: &str = "RUSTFS_NOTIFY_AMQP_MANDATORY";
+pub const ENV_NOTIFY_AMQP_PERSISTENT: &str = "RUSTFS_NOTIFY_AMQP_PERSISTENT";
+pub const ENV_NOTIFY_AMQP_USERNAME: &str = "RUSTFS_NOTIFY_AMQP_USERNAME";
+pub const ENV_NOTIFY_AMQP_PASSWORD: &str = "RUSTFS_NOTIFY_AMQP_PASSWORD";
+pub const ENV_NOTIFY_AMQP_TLS_CA: &str = "RUSTFS_NOTIFY_AMQP_TLS_CA";
+pub const ENV_NOTIFY_AMQP_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_AMQP_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_AMQP_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_AMQP_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_AMQP_QUEUE_DIR: &str = "RUSTFS_NOTIFY_AMQP_QUEUE_DIR";
+pub const ENV_NOTIFY_AMQP_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_AMQP_QUEUE_LIMIT";
+
+pub const ENV_NOTIFY_AMQP_KEYS: &[&str; 13] = &[
+    ENV_NOTIFY_AMQP_ENABLE,
+    ENV_NOTIFY_AMQP_URL,
+    ENV_NOTIFY_AMQP_EXCHANGE,
+    ENV_NOTIFY_AMQP_ROUTING_KEY,
+    ENV_NOTIFY_AMQP_MANDATORY,
+    ENV_NOTIFY_AMQP_PERSISTENT,
+    ENV_NOTIFY_AMQP_USERNAME,
+    ENV_NOTIFY_AMQP_PASSWORD,
+    ENV_NOTIFY_AMQP_TLS_CA,
+    ENV_NOTIFY_AMQP_TLS_CLIENT_CERT,
+    ENV_NOTIFY_AMQP_TLS_CLIENT_KEY,
+    ENV_NOTIFY_AMQP_QUEUE_DIR,
+    ENV_NOTIFY_AMQP_QUEUE_LIMIT,
+];

crates/config/src/notify/kafka.rs

@@ -0,0 +1,53 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// A list of all valid configuration keys for a Kafka target.
+pub const NOTIFY_KAFKA_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::KAFKA_BROKERS,
+    crate::KAFKA_TOPIC,
+    crate::KAFKA_ACKS,
+    crate::KAFKA_TLS_ENABLE,
+    crate::KAFKA_TLS_CA,
+    crate::KAFKA_TLS_CLIENT_CERT,
+    crate::KAFKA_TLS_CLIENT_KEY,
+    crate::KAFKA_QUEUE_DIR,
+    crate::KAFKA_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+// Kafka Environment Variables
+pub const ENV_NOTIFY_KAFKA_ENABLE: &str = "RUSTFS_NOTIFY_KAFKA_ENABLE";
+pub const ENV_NOTIFY_KAFKA_BROKERS: &str = "RUSTFS_NOTIFY_KAFKA_BROKERS";
+pub const ENV_NOTIFY_KAFKA_TOPIC: &str = "RUSTFS_NOTIFY_KAFKA_TOPIC";
+pub const ENV_NOTIFY_KAFKA_ACKS: &str = "RUSTFS_NOTIFY_KAFKA_ACKS";
+pub const ENV_NOTIFY_KAFKA_TLS_ENABLE: &str = "RUSTFS_NOTIFY_KAFKA_TLS_ENABLE";
+pub const ENV_NOTIFY_KAFKA_TLS_CA: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CA";
+pub const ENV_NOTIFY_KAFKA_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_KAFKA_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_KAFKA_QUEUE_DIR: &str = "RUSTFS_NOTIFY_KAFKA_QUEUE_DIR";
+pub const ENV_NOTIFY_KAFKA_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_KAFKA_QUEUE_LIMIT";
+
+pub const ENV_NOTIFY_KAFKA_KEYS: &[&str; 10] = &[
+    ENV_NOTIFY_KAFKA_ENABLE,
+    ENV_NOTIFY_KAFKA_BROKERS,
+    ENV_NOTIFY_KAFKA_TOPIC,
+    ENV_NOTIFY_KAFKA_ACKS,
+    ENV_NOTIFY_KAFKA_TLS_ENABLE,
+    ENV_NOTIFY_KAFKA_TLS_CA,
+    ENV_NOTIFY_KAFKA_TLS_CLIENT_CERT,
+    ENV_NOTIFY_KAFKA_TLS_CLIENT_KEY,
+    ENV_NOTIFY_KAFKA_QUEUE_DIR,
+    ENV_NOTIFY_KAFKA_QUEUE_LIMIT,
+];

crates/config/src/notify/mod.rs

@@ -12,13 +12,27 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+mod amqp;
 mod arn;
+mod kafka;
 mod mqtt;
+mod mysql;
+mod nats;
+mod postgres;
+mod pulsar;
+mod redis;
 mod store;
 mod webhook;
 
+pub use amqp::*;
 pub use arn::*;
+pub use kafka::*;
 pub use mqtt::*;
+pub use mysql::*;
+pub use nats::*;
+pub use postgres::*;
+pub use pulsar::*;
+pub use redis::*;
 pub use store::*;
 pub use webhook::*;
 
@@ -63,24 +77,31 @@ pub const ENV_NOTIFY_SEND_CONCURRENCY: &str = "RUSTFS_NOTIFY_SEND_CONCURRENCY";
 /// Adjust this value based on your system's capabilities and expected load.
 pub const DEFAULT_NOTIFY_SEND_CONCURRENCY: usize = 64;
 
-#[allow(dead_code)]
-pub const NOTIFY_SUB_SYSTEMS: &[&str] = &[NOTIFY_MQTT_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS];
+pub const NOTIFY_SUB_SYSTEMS: &[&str] = &[
+    NOTIFY_AMQP_SUB_SYS,
+    NOTIFY_KAFKA_SUB_SYS,
+    NOTIFY_MQTT_SUB_SYS,
+    NOTIFY_MYSQL_SUB_SYS,
+    NOTIFY_NATS_SUB_SYS,
+    NOTIFY_POSTGRES_SUB_SYS,
+    NOTIFY_PULSAR_SUB_SYS,
+    NOTIFY_REDIS_SUB_SYS,
+    NOTIFY_WEBHOOK_SUB_SYS,
+];
 
-#[allow(dead_code)]
 pub const NOTIFY_KAFKA_SUB_SYS: &str = "notify_kafka";
 pub const NOTIFY_MQTT_SUB_SYS: &str = "notify_mqtt";
-#[allow(dead_code)]
-pub const NOTIFY_MY_SQL_SUB_SYS: &str = "notify_mysql";
+pub const NOTIFY_MYSQL_SUB_SYS: &str = "notify_mysql";
 #[allow(dead_code)]
 pub const NOTIFY_NATS_SUB_SYS: &str = "notify_nats";
 #[allow(dead_code)]
 pub const NOTIFY_NSQ_SUB_SYS: &str = "notify_nsq";
 #[allow(dead_code)]
 pub const NOTIFY_ES_SUB_SYS: &str = "notify_elasticsearch";
-#[allow(dead_code)]
 pub const NOTIFY_AMQP_SUB_SYS: &str = "notify_amqp";
-#[allow(dead_code)]
 pub const NOTIFY_POSTGRES_SUB_SYS: &str = "notify_postgres";
 #[allow(dead_code)]
 pub const NOTIFY_REDIS_SUB_SYS: &str = "notify_redis";
+pub const NOTIFY_REDIS_DEFAULT_CHANNEL: &str = "rustfs_notify_channel";
+pub const NOTIFY_PULSAR_SUB_SYS: &str = "notify_pulsar";
 pub const NOTIFY_WEBHOOK_SUB_SYS: &str = "notify_webhook";

crates/config/src/notify/mysql.rs

@@ -0,0 +1,53 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// A list of all valid configuration keys for a MySQL target.
+pub const NOTIFY_MYSQL_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::MYSQL_DSN_STRING,
+    crate::MYSQL_TABLE,
+    crate::MYSQL_FORMAT,
+    crate::MYSQL_TLS_CA,
+    crate::MYSQL_TLS_CLIENT_CERT,
+    crate::MYSQL_TLS_CLIENT_KEY,
+    crate::MYSQL_QUEUE_DIR,
+    crate::MYSQL_QUEUE_LIMIT,
+    crate::MYSQL_MAX_OPEN_CONNECTIONS,
+    crate::COMMENT_KEY,
+];
+
+// MySQL Environment Variables
+pub const ENV_NOTIFY_MYSQL_ENABLE: &str = "RUSTFS_NOTIFY_MYSQL_ENABLE";
+pub const ENV_NOTIFY_MYSQL_DSN_STRING: &str = "RUSTFS_NOTIFY_MYSQL_DSN_STRING";
+pub const ENV_NOTIFY_MYSQL_TABLE: &str = "RUSTFS_NOTIFY_MYSQL_TABLE";
+pub const ENV_NOTIFY_MYSQL_FORMAT: &str = "RUSTFS_NOTIFY_MYSQL_FORMAT";
+pub const ENV_NOTIFY_MYSQL_TLS_CA: &str = "RUSTFS_NOTIFY_MYSQL_TLS_CA";
+pub const ENV_NOTIFY_MYSQL_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_MYSQL_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_MYSQL_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_MYSQL_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_MYSQL_QUEUE_DIR: &str = "RUSTFS_NOTIFY_MYSQL_QUEUE_DIR";
+pub const ENV_NOTIFY_MYSQL_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_MYSQL_QUEUE_LIMIT";
+pub const ENV_NOTIFY_MYSQL_MAX_OPEN_CONNECTIONS: &str = "RUSTFS_NOTIFY_MYSQL_MAX_OPEN_CONNECTIONS";
+
+pub const ENV_NOTIFY_MYSQL_KEYS: &[&str; 10] = &[
+    ENV_NOTIFY_MYSQL_ENABLE,
+    ENV_NOTIFY_MYSQL_DSN_STRING,
+    ENV_NOTIFY_MYSQL_TABLE,
+    ENV_NOTIFY_MYSQL_FORMAT,
+    ENV_NOTIFY_MYSQL_TLS_CA,
+    ENV_NOTIFY_MYSQL_TLS_CLIENT_CERT,
+    ENV_NOTIFY_MYSQL_TLS_CLIENT_KEY,
+    ENV_NOTIFY_MYSQL_QUEUE_DIR,
+    ENV_NOTIFY_MYSQL_QUEUE_LIMIT,
+    ENV_NOTIFY_MYSQL_MAX_OPEN_CONNECTIONS,
+];

crates/config/src/notify/nats.rs

@@ -0,0 +1,60 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const NOTIFY_NATS_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::NATS_ADDRESS,
+    crate::NATS_SUBJECT,
+    crate::NATS_USERNAME,
+    crate::NATS_PASSWORD,
+    crate::NATS_TOKEN,
+    crate::NATS_CREDENTIALS_FILE,
+    crate::NATS_TLS_CA,
+    crate::NATS_TLS_CLIENT_CERT,
+    crate::NATS_TLS_CLIENT_KEY,
+    crate::NATS_TLS_REQUIRED,
+    crate::NATS_QUEUE_DIR,
+    crate::NATS_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_NOTIFY_NATS_ENABLE: &str = "RUSTFS_NOTIFY_NATS_ENABLE";
+pub const ENV_NOTIFY_NATS_ADDRESS: &str = "RUSTFS_NOTIFY_NATS_ADDRESS";
+pub const ENV_NOTIFY_NATS_SUBJECT: &str = "RUSTFS_NOTIFY_NATS_SUBJECT";
+pub const ENV_NOTIFY_NATS_USERNAME: &str = "RUSTFS_NOTIFY_NATS_USERNAME";
+pub const ENV_NOTIFY_NATS_PASSWORD: &str = "RUSTFS_NOTIFY_NATS_PASSWORD";
+pub const ENV_NOTIFY_NATS_TOKEN: &str = "RUSTFS_NOTIFY_NATS_TOKEN";
+pub const ENV_NOTIFY_NATS_CREDENTIALS_FILE: &str = "RUSTFS_NOTIFY_NATS_CREDENTIALS_FILE";
+pub const ENV_NOTIFY_NATS_TLS_CA: &str = "RUSTFS_NOTIFY_NATS_TLS_CA";
+pub const ENV_NOTIFY_NATS_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_NATS_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_NATS_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_NATS_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_NATS_TLS_REQUIRED: &str = "RUSTFS_NOTIFY_NATS_TLS_REQUIRED";
+pub const ENV_NOTIFY_NATS_QUEUE_DIR: &str = "RUSTFS_NOTIFY_NATS_QUEUE_DIR";
+pub const ENV_NOTIFY_NATS_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_NATS_QUEUE_LIMIT";
+
+pub const ENV_NOTIFY_NATS_KEYS: &[&str; 13] = &[
+    ENV_NOTIFY_NATS_ENABLE,
+    ENV_NOTIFY_NATS_ADDRESS,
+    ENV_NOTIFY_NATS_SUBJECT,
+    ENV_NOTIFY_NATS_USERNAME,
+    ENV_NOTIFY_NATS_PASSWORD,
+    ENV_NOTIFY_NATS_TOKEN,
+    ENV_NOTIFY_NATS_CREDENTIALS_FILE,
+    ENV_NOTIFY_NATS_TLS_CA,
+    ENV_NOTIFY_NATS_TLS_CLIENT_CERT,
+    ENV_NOTIFY_NATS_TLS_CLIENT_KEY,
+    ENV_NOTIFY_NATS_TLS_REQUIRED,
+    ENV_NOTIFY_NATS_QUEUE_DIR,
+    ENV_NOTIFY_NATS_QUEUE_LIMIT,
+];

crates/config/src/notify/postgres.rs

@@ -0,0 +1,51 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const NOTIFY_POSTGRES_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::POSTGRES_DSN_STRING,
+    crate::POSTGRES_TABLE,
+    crate::POSTGRES_FORMAT,
+    crate::POSTGRES_TLS_REQUIRED,
+    crate::POSTGRES_TLS_CA,
+    crate::POSTGRES_TLS_CLIENT_CERT,
+    crate::POSTGRES_TLS_CLIENT_KEY,
+    crate::POSTGRES_QUEUE_DIR,
+    crate::POSTGRES_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_NOTIFY_POSTGRES_ENABLE: &str = "RUSTFS_NOTIFY_POSTGRES_ENABLE";
+pub const ENV_NOTIFY_POSTGRES_DSN_STRING: &str = "RUSTFS_NOTIFY_POSTGRES_DSN_STRING";
+pub const ENV_NOTIFY_POSTGRES_TABLE: &str = "RUSTFS_NOTIFY_POSTGRES_TABLE";
+pub const ENV_NOTIFY_POSTGRES_FORMAT: &str = "RUSTFS_NOTIFY_POSTGRES_FORMAT";
+pub const ENV_NOTIFY_POSTGRES_TLS_REQUIRED: &str = "RUSTFS_NOTIFY_POSTGRES_TLS_REQUIRED";
+pub const ENV_NOTIFY_POSTGRES_TLS_CA: &str = "RUSTFS_NOTIFY_POSTGRES_TLS_CA";
+pub const ENV_NOTIFY_POSTGRES_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_POSTGRES_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_POSTGRES_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_POSTGRES_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_POSTGRES_QUEUE_DIR: &str = "RUSTFS_NOTIFY_POSTGRES_QUEUE_DIR";
+pub const ENV_NOTIFY_POSTGRES_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_POSTGRES_QUEUE_LIMIT";
+
+pub const ENV_NOTIFY_POSTGRES_KEYS: &[&str; 10] = &[
+    ENV_NOTIFY_POSTGRES_ENABLE,
+    ENV_NOTIFY_POSTGRES_DSN_STRING,
+    ENV_NOTIFY_POSTGRES_TABLE,
+    ENV_NOTIFY_POSTGRES_FORMAT,
+    ENV_NOTIFY_POSTGRES_TLS_REQUIRED,
+    ENV_NOTIFY_POSTGRES_TLS_CA,
+    ENV_NOTIFY_POSTGRES_TLS_CLIENT_CERT,
+    ENV_NOTIFY_POSTGRES_TLS_CLIENT_KEY,
+    ENV_NOTIFY_POSTGRES_QUEUE_DIR,
+    ENV_NOTIFY_POSTGRES_QUEUE_LIMIT,
+];

crates/config/src/notify/pulsar.rs

@@ -0,0 +1,54 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub const NOTIFY_PULSAR_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::PULSAR_BROKER,
+    crate::PULSAR_TOPIC,
+    crate::PULSAR_AUTH_TOKEN,
+    crate::PULSAR_USERNAME,
+    crate::PULSAR_PASSWORD,
+    crate::PULSAR_TLS_CA,
+    crate::PULSAR_TLS_ALLOW_INSECURE,
+    crate::PULSAR_TLS_HOSTNAME_VERIFICATION,
+    crate::PULSAR_QUEUE_DIR,
+    crate::PULSAR_QUEUE_LIMIT,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_NOTIFY_PULSAR_ENABLE: &str = "RUSTFS_NOTIFY_PULSAR_ENABLE";
+pub const ENV_NOTIFY_PULSAR_BROKER: &str = "RUSTFS_NOTIFY_PULSAR_BROKER";
+pub const ENV_NOTIFY_PULSAR_TOPIC: &str = "RUSTFS_NOTIFY_PULSAR_TOPIC";
+pub const ENV_NOTIFY_PULSAR_AUTH_TOKEN: &str = "RUSTFS_NOTIFY_PULSAR_AUTH_TOKEN";
+pub const ENV_NOTIFY_PULSAR_USERNAME: &str = "RUSTFS_NOTIFY_PULSAR_USERNAME";
+pub const ENV_NOTIFY_PULSAR_PASSWORD: &str = "RUSTFS_NOTIFY_PULSAR_PASSWORD";
+pub const ENV_NOTIFY_PULSAR_TLS_CA: &str = "RUSTFS_NOTIFY_PULSAR_TLS_CA";
+pub const ENV_NOTIFY_PULSAR_TLS_ALLOW_INSECURE: &str = "RUSTFS_NOTIFY_PULSAR_TLS_ALLOW_INSECURE";
+pub const ENV_NOTIFY_PULSAR_TLS_HOSTNAME_VERIFICATION: &str = "RUSTFS_NOTIFY_PULSAR_TLS_HOSTNAME_VERIFICATION";
+pub const ENV_NOTIFY_PULSAR_QUEUE_DIR: &str = "RUSTFS_NOTIFY_PULSAR_QUEUE_DIR";
+pub const ENV_NOTIFY_PULSAR_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_PULSAR_QUEUE_LIMIT";
+
+pub const ENV_NOTIFY_PULSAR_KEYS: &[&str; 11] = &[
+    ENV_NOTIFY_PULSAR_ENABLE,
+    ENV_NOTIFY_PULSAR_BROKER,
+    ENV_NOTIFY_PULSAR_TOPIC,
+    ENV_NOTIFY_PULSAR_AUTH_TOKEN,
+    ENV_NOTIFY_PULSAR_USERNAME,
+    ENV_NOTIFY_PULSAR_PASSWORD,
+    ENV_NOTIFY_PULSAR_TLS_CA,
+    ENV_NOTIFY_PULSAR_TLS_ALLOW_INSECURE,
+    ENV_NOTIFY_PULSAR_TLS_HOSTNAME_VERIFICATION,
+    ENV_NOTIFY_PULSAR_QUEUE_DIR,
+    ENV_NOTIFY_PULSAR_QUEUE_LIMIT,
+];

crates/config/src/notify/redis.rs

@@ -0,0 +1,82 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// A list of all valid configuration keys for a Redis target.
+pub const NOTIFY_REDIS_KEYS: &[&str] = &[
+    crate::ENABLE_KEY,
+    crate::REDIS_URL,
+    crate::REDIS_CHANNEL,
+    crate::REDIS_USERNAME,
+    crate::REDIS_PASSWORD,
+    crate::REDIS_KEEP_ALIVE_INTERVAL,
+    crate::REDIS_QUEUE_DIR,
+    crate::REDIS_QUEUE_LIMIT,
+    crate::REDIS_MAX_RETRY_ATTEMPTS,
+    crate::REDIS_RECONNECT_RETRY_ATTEMPTS,
+    crate::REDIS_MIN_RETRY_DELAY,
+    crate::REDIS_MAX_RETRY_DELAY,
+    crate::REDIS_CONNECTION_TIMEOUT,
+    crate::REDIS_RESPONSE_TIMEOUT,
+    crate::REDIS_PIPELINE_BUFFER_SIZE,
+    crate::REDIS_TLS_POLICY,
+    crate::REDIS_TLS_CA,
+    crate::REDIS_TLS_CLIENT_CERT,
+    crate::REDIS_TLS_CLIENT_KEY,
+    crate::REDIS_TLS_ALLOW_INSECURE,
+    crate::COMMENT_KEY,
+];
+
+pub const ENV_NOTIFY_REDIS_ENABLE: &str = "RUSTFS_NOTIFY_REDIS_ENABLE";
+pub const ENV_NOTIFY_REDIS_URL: &str = "RUSTFS_NOTIFY_REDIS_URL";
+pub const ENV_NOTIFY_REDIS_CHANNEL: &str = "RUSTFS_NOTIFY_REDIS_CHANNEL";
+pub const ENV_NOTIFY_REDIS_USERNAME: &str = "RUSTFS_NOTIFY_REDIS_USERNAME";
+pub const ENV_NOTIFY_REDIS_PASSWORD: &str = "RUSTFS_NOTIFY_REDIS_PASSWORD";
+pub const ENV_NOTIFY_REDIS_KEEP_ALIVE_INTERVAL: &str = "RUSTFS_NOTIFY_REDIS_KEEP_ALIVE_INTERVAL";
+pub const ENV_NOTIFY_REDIS_QUEUE_DIR: &str = "RUSTFS_NOTIFY_REDIS_QUEUE_DIR";
+pub const ENV_NOTIFY_REDIS_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_REDIS_QUEUE_LIMIT";
+pub const ENV_NOTIFY_REDIS_MAX_RETRY_ATTEMPTS: &str = "RUSTFS_NOTIFY_REDIS_MAX_RETRY_ATTEMPTS";
+pub const ENV_NOTIFY_REDIS_RECONNECT_RETRY_ATTEMPTS: &str = "RUSTFS_NOTIFY_REDIS_RECONNECT_RETRY_ATTEMPTS";
+pub const ENV_NOTIFY_REDIS_MIN_RETRY_DELAY: &str = "RUSTFS_NOTIFY_REDIS_MIN_RETRY_DELAY";
+pub const ENV_NOTIFY_REDIS_MAX_RETRY_DELAY: &str = "RUSTFS_NOTIFY_REDIS_MAX_RETRY_DELAY";
+pub const ENV_NOTIFY_REDIS_CONNECTION_TIMEOUT: &str = "RUSTFS_NOTIFY_REDIS_CONNECTION_TIMEOUT";
+pub const ENV_NOTIFY_REDIS_RESPONSE_TIMEOUT: &str = "RUSTFS_NOTIFY_REDIS_RESPONSE_TIMEOUT";
+pub const ENV_NOTIFY_REDIS_PIPELINE_BUFFER_SIZE: &str = "RUSTFS_NOTIFY_REDIS_PIPELINE_BUFFER_SIZE";
+pub const ENV_NOTIFY_REDIS_TLS_POLICY: &str = "RUSTFS_NOTIFY_REDIS_TLS_POLICY";
+pub const ENV_NOTIFY_REDIS_TLS_CA: &str = "RUSTFS_NOTIFY_REDIS_TLS_CA";
+pub const ENV_NOTIFY_REDIS_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_REDIS_TLS_CLIENT_CERT";
+pub const ENV_NOTIFY_REDIS_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_REDIS_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_REDIS_TLS_ALLOW_INSECURE: &str = "RUSTFS_NOTIFY_REDIS_TLS_ALLOW_INSECURE";
+
+pub const ENV_NOTIFY_REDIS_KEYS: &[&str; 20] = &[
+    ENV_NOTIFY_REDIS_ENABLE,
+    ENV_NOTIFY_REDIS_URL,
+    ENV_NOTIFY_REDIS_CHANNEL,
+    ENV_NOTIFY_REDIS_USERNAME,
+    ENV_NOTIFY_REDIS_PASSWORD,
+    ENV_NOTIFY_REDIS_KEEP_ALIVE_INTERVAL,
+    ENV_NOTIFY_REDIS_QUEUE_DIR,
+    ENV_NOTIFY_REDIS_QUEUE_LIMIT,
+    ENV_NOTIFY_REDIS_MAX_RETRY_ATTEMPTS,
+    ENV_NOTIFY_REDIS_RECONNECT_RETRY_ATTEMPTS,
+    ENV_NOTIFY_REDIS_MIN_RETRY_DELAY,
+    ENV_NOTIFY_REDIS_MAX_RETRY_DELAY,
+    ENV_NOTIFY_REDIS_CONNECTION_TIMEOUT,
+    ENV_NOTIFY_REDIS_RESPONSE_TIMEOUT,
+    ENV_NOTIFY_REDIS_PIPELINE_BUFFER_SIZE,
+    ENV_NOTIFY_REDIS_TLS_POLICY,
+    ENV_NOTIFY_REDIS_TLS_CA,
+    ENV_NOTIFY_REDIS_TLS_CLIENT_CERT,
+    ENV_NOTIFY_REDIS_TLS_CLIENT_KEY,
+    ENV_NOTIFY_REDIS_TLS_ALLOW_INSECURE,
+];

crates/config/src/observability/mod.rs

@@ -22,6 +22,14 @@ pub const ENV_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";
 pub const ENV_OBS_TRACE_ENDPOINT: &str = "RUSTFS_OBS_TRACE_ENDPOINT";
 pub const ENV_OBS_METRIC_ENDPOINT: &str = "RUSTFS_OBS_METRIC_ENDPOINT";
 pub const ENV_OBS_LOG_ENDPOINT: &str = "RUSTFS_OBS_LOG_ENDPOINT";
+pub const ENV_OBS_ENDPOINT_HEADERS: &str = "RUSTFS_OBS_ENDPOINT_HEADERS";
+pub const ENV_OBS_ENDPOINT_TRACES_HEADERS: &str = "RUSTFS_OBS_ENDPOINT_TRACES_HEADERS";
+pub const ENV_OBS_ENDPOINT_METRICS_HEADERS: &str = "RUSTFS_OBS_ENDPOINT_METRICS_HEADERS";
+pub const ENV_OBS_ENDPOINT_LOGS_HEADERS: &str = "RUSTFS_OBS_ENDPOINT_LOGS_HEADERS";
+pub const ENV_OBS_ENDPOINT_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_TIMEOUT_MILLIS";
+pub const ENV_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS";
+pub const ENV_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS";
+pub const ENV_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS";
 pub const ENV_OBS_PROFILING_ENDPOINT: &str = "RUSTFS_OBS_PROFILING_ENDPOINT";
 pub const ENV_OBS_USE_STDOUT: &str = "RUSTFS_OBS_USE_STDOUT";
 pub const ENV_OBS_SAMPLE_RATIO: &str = "RUSTFS_OBS_SAMPLE_RATIO";
@@ -108,6 +116,14 @@ mod tests {
         assert_eq!(ENV_OBS_TRACE_ENDPOINT, "RUSTFS_OBS_TRACE_ENDPOINT");
         assert_eq!(ENV_OBS_METRIC_ENDPOINT, "RUSTFS_OBS_METRIC_ENDPOINT");
         assert_eq!(ENV_OBS_LOG_ENDPOINT, "RUSTFS_OBS_LOG_ENDPOINT");
+        assert_eq!(ENV_OBS_ENDPOINT_HEADERS, "RUSTFS_OBS_ENDPOINT_HEADERS");
+        assert_eq!(ENV_OBS_ENDPOINT_TRACES_HEADERS, "RUSTFS_OBS_ENDPOINT_TRACES_HEADERS");
+        assert_eq!(ENV_OBS_ENDPOINT_METRICS_HEADERS, "RUSTFS_OBS_ENDPOINT_METRICS_HEADERS");
+        assert_eq!(ENV_OBS_ENDPOINT_LOGS_HEADERS, "RUSTFS_OBS_ENDPOINT_LOGS_HEADERS");
+        assert_eq!(ENV_OBS_ENDPOINT_TIMEOUT_MILLIS, "RUSTFS_OBS_ENDPOINT_TIMEOUT_MILLIS");
+        assert_eq!(ENV_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS, "RUSTFS_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS");
+        assert_eq!(ENV_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS, "RUSTFS_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS");
+        assert_eq!(ENV_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS, "RUSTFS_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS");
         assert_eq!(ENV_OBS_PROFILING_ENDPOINT, "RUSTFS_OBS_PROFILING_ENDPOINT");
         assert_eq!(ENV_OBS_USE_STDOUT, "RUSTFS_OBS_USE_STDOUT");
         assert_eq!(ENV_OBS_SAMPLE_RATIO, "RUSTFS_OBS_SAMPLE_RATIO");

crates/credentials/src/constants.rs

@@ -66,7 +66,7 @@ mod tests {
 
         // In production environment, access key and secret key should be different
         // These are default values, so being the same is acceptable, but should be warned in documentation
-        println!("Warning: Default access key and secret key are the same. Change them in production!");
+        assert_eq!(DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY);
     }
 
     #[test]
@@ -74,10 +74,8 @@ mod tests {
         // Test security best practices
 
         // These are default values, should be changed in production environments
-        println!("Security Warning: Default credentials detected!");
-        println!("Access Key: {DEFAULT_ACCESS_KEY}");
-        println!("Secret Key: {DEFAULT_SECRET_KEY}");
-        println!("These should be changed in production environments!");
+        assert_eq!(DEFAULT_ACCESS_KEY, "rustfsadmin");
+        assert_eq!(DEFAULT_SECRET_KEY, "rustfsadmin");
 
         // Verify that key lengths meet minimum security requirements
         assert!(DEFAULT_ACCESS_KEY.len() >= 8, "Access key should be at least 8 characters");

crates/credentials/src/credentials.rs

@@ -20,7 +20,7 @@ use std::collections::HashMap;
 use std::env;
 use std::fmt;
 use std::io::Error;
-use std::sync::OnceLock;
+use std::sync::{LazyLock, OnceLock};
 use time::OffsetDateTime;
 
 /// Global active credentials
@@ -29,6 +29,12 @@ static GLOBAL_ACTIVE_CRED: OnceLock<Credentials> = OnceLock::new();
 /// Global RPC authentication token
 pub static GLOBAL_RUSTFS_RPC_SECRET: OnceLock<String> = OnceLock::new();
 
+/// Public error returned when RPC authentication is not safely configured.
+pub const RPC_SECRET_REQUIRED_MESSAGE: &str = "RPC authentication secret is not configured";
+
+/// Operator-facing guidance for configuring RPC authentication safely.
+pub const RPC_SECRET_REQUIRED_OPERATOR_MESSAGE: &str = "RUSTFS_RPC_SECRET must be set to a non-default value or RUSTFS_SECRET_KEY must be changed from the default for RPC authentication";
+
 /// Error type for credentials operations
 #[derive(Debug)]
 pub enum CredentialsError {
@@ -216,13 +222,39 @@ pub fn gen_secret_key(length: usize) -> std::io::Result<String> {
 /// # Returns
 /// * `String` - The RPC authentication token
 ///
+fn resolve_rpc_secret(env_secret: Option<&str>, global_secret: Option<&str>) -> Option<String> {
+    if let Some(secret) = env_secret.map(str::trim).filter(|secret| !secret.is_empty()) {
+        return (secret != DEFAULT_SECRET_KEY).then(|| secret.to_string());
+    }
+
+    global_secret
+        .map(str::trim)
+        .filter(|secret| !secret.is_empty() && *secret != DEFAULT_SECRET_KEY)
+        .map(ToOwned::to_owned)
+}
+
+pub fn try_get_rpc_token() -> std::io::Result<String> {
+    if let Some(secret) = GLOBAL_RUSTFS_RPC_SECRET.get() {
+        return resolve_rpc_secret(None, Some(secret)).ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE));
+    }
+
+    let env_secret = env::var(ENV_RPC_SECRET).ok();
+    let global_secret = get_global_secret_key_opt();
+    let secret = resolve_rpc_secret(env_secret.as_deref(), global_secret.as_deref())
+        .ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE))?;
+
+    match GLOBAL_RUSTFS_RPC_SECRET.set(secret.clone()) {
+        Ok(()) => Ok(secret),
+        Err(_) => GLOBAL_RUSTFS_RPC_SECRET
+            .get()
+            .and_then(|stored| resolve_rpc_secret(None, Some(stored)))
+            .ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE)),
+    }
+}
+
+#[deprecated(note = "use try_get_rpc_token to handle missing RPC secrets explicitly")]
 pub fn get_rpc_token() -> String {
-    GLOBAL_RUSTFS_RPC_SECRET
-        .get_or_init(|| {
-            env::var(ENV_RPC_SECRET)
-                .unwrap_or_else(|_| get_global_secret_key_opt().unwrap_or_else(|| DEFAULT_SECRET_KEY.to_string()))
-        })
-        .clone()
+    try_get_rpc_token().expect(RPC_SECRET_REQUIRED_MESSAGE)
 }
 
 /// A wrapper struct for masking sensitive strings in Debug implementations.
@@ -301,7 +333,7 @@ impl fmt::Debug for Credentials {
         f.debug_struct("Credentials")
             .field("access_key", &self.access_key)
             .field("secret_key", &Masked(Some(&self.secret_key)))
-            .field("session_token", &self.session_token)
+            .field("session_token", &Masked(Some(&self.session_token)))
             .field("expiration", &self.expiration)
             .field("status", &self.status)
             .field("parent_user", &self.parent_user)
@@ -314,6 +346,17 @@ impl fmt::Debug for Credentials {
 }
 
 impl Credentials {
+    /// Returns a reference to this credential's claims, or a shared empty map
+    /// when the credential has no claims attached. Avoids per-call allocation
+    /// at call sites that need an `&HashMap<String, Value>`.
+    pub fn claims_or_empty(&self) -> &HashMap<String, Value> {
+        static EMPTY: LazyLock<HashMap<String, Value>> = LazyLock::new(HashMap::new);
+        match &self.claims {
+            Some(c) => c,
+            None => &EMPTY,
+        }
+    }
+
     pub fn is_expired(&self) -> bool {
         if self.expiration.is_none() {
             return false;
@@ -484,6 +527,63 @@ mod tests {
         assert!(!key.contains('='));
     }
 
+    #[test]
+    fn test_resolve_rpc_secret_rejects_default_fallback() {
+        assert!(resolve_rpc_secret(None, None).is_none());
+        assert!(resolve_rpc_secret(None, Some(DEFAULT_SECRET_KEY)).is_none());
+        assert!(resolve_rpc_secret(Some(DEFAULT_SECRET_KEY), Some("custom-global-secret")).is_none());
+    }
+
+    #[test]
+    fn test_rpc_secret_public_error_omits_configuration_details() {
+        assert!(!RPC_SECRET_REQUIRED_MESSAGE.contains("RUSTFS_"));
+        assert!(!RPC_SECRET_REQUIRED_MESSAGE.contains(DEFAULT_SECRET_KEY));
+        assert!(RPC_SECRET_REQUIRED_OPERATOR_MESSAGE.contains("RUSTFS_RPC_SECRET"));
+    }
+
+    #[allow(deprecated)]
+    #[test]
+    fn test_get_rpc_token_preserves_string_return_type() {
+        fn assert_string_return(_: fn() -> String) {}
+
+        assert_string_return(get_rpc_token);
+    }
+
+    #[test]
+    fn test_resolve_rpc_secret_accepts_non_default_secret() {
+        assert_eq!(resolve_rpc_secret(Some("custom-rpc-secret"), None).as_deref(), Some("custom-rpc-secret"));
+        assert_eq!(
+            resolve_rpc_secret(None, Some("custom-global-secret")).as_deref(),
+            Some("custom-global-secret")
+        );
+    }
+
+    #[test]
+    fn test_resolve_rpc_secret_trims_and_falls_back_from_blank_env() {
+        assert_eq!(
+            resolve_rpc_secret(Some("  custom-rpc-secret  "), None).as_deref(),
+            Some("custom-rpc-secret")
+        );
+        assert_eq!(
+            resolve_rpc_secret(Some(""), Some("custom-global-secret")).as_deref(),
+            Some("custom-global-secret")
+        );
+        assert_eq!(
+            resolve_rpc_secret(Some("  "), Some("  custom-global-secret  ")).as_deref(),
+            Some("custom-global-secret")
+        );
+        assert_eq!(
+            resolve_rpc_secret(Some("  "), Some("custom-global-secret")).as_deref(),
+            Some("custom-global-secret")
+        );
+    }
+
+    #[test]
+    fn test_resolve_rpc_secret_returns_none_for_trimmed_default_secret() {
+        let padded_default_secret = format!("  {}  ", DEFAULT_SECRET_KEY);
+        assert!(resolve_rpc_secret(Some(padded_default_secret.as_str()), Some("custom-global-secret")).is_none());
+    }
+
     #[test]
     fn test_masked_debug() {
         // Test None
@@ -513,6 +613,24 @@ mod tests {
         assert_eq!(format!("{:?}", Masked(Some("中文测试"))), "中***试|4");
     }
 
+    #[test]
+    fn test_credentials_debug_masks_sensitive_fields() {
+        let cred = Credentials {
+            access_key: "debug-access-key".to_string(),
+            secret_key: "debug-secret-key".to_string(),
+            session_token: "debug-session-token".to_string(),
+            parent_user: "parent-user".to_string(),
+            ..Default::default()
+        };
+
+        let output = format!("{cred:?}");
+
+        assert!(output.contains("debug-access-key"));
+        assert!(output.contains("parent-user"));
+        assert!(!output.contains("debug-secret-key"));
+        assert!(!output.contains("debug-session-token"));
+    }
+
     #[test]
     fn test_credentials_expiration_serialize_as_rfc3339() {
         use time::OffsetDateTime;

crates/crypto/Cargo.toml

@@ -31,7 +31,6 @@ workspace = true
 [dependencies]
 aes-gcm = { workspace = true, optional = true }
 argon2 = { workspace = true, optional = true }
-cfg-if = { workspace = true }
 chacha20poly1305 = { workspace = true, optional = true }
 jsonwebtoken = { workspace = true }
 pbkdf2 = { workspace = true, optional = true }

crates/crypto/src/encdec/aes.rs

@@ -13,19 +13,23 @@
 // limitations under the License.
 
 pub fn native_aes() -> bool {
-    cfg_if::cfg_if! {
-        if #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] {
+    cfg_select! {
+        any(target_arch = "x86", target_arch = "x86_64") => {
             std::is_x86_feature_detected!("aes") && std::is_x86_feature_detected!("pclmulqdq")
-        } else if #[cfg(target_arch = "aarch64")] {
+        }
+        target_arch = "aarch64" => {
             std::arch::is_aarch64_feature_detected!("aes")
-        } else if #[cfg(target_arch = "powerpc64")] {
+        }
+        target_arch = "powerpc64" => {
             false
-        } else if #[cfg(target_arch = "s390x")] {
+        }
+        target_arch = "s390x" => {
             std::is_s390x_feature_detected!("aes")
                 && std::is_s390x_feature_detected!("aescbc")
                 && std::is_s390x_feature_detected!("aesctr")
                 && (std::is_s390x_feature_detected!("aesgcm") || std::is_s390x_feature_detected!("ghash"))
-        } else {
+        }
+        _ => {
             false
         }
     }

crates/e2e_test/Cargo.toml

@@ -26,8 +26,10 @@ workspace = true
 [features]
 default = []
 ftps = []
+sftp = []
 
 [dependencies]
+rustfs-config = { workspace = true, features = ["constants"] }
 rustfs-ecstore.workspace = true
 rustfs-common.workspace = true
 rustfs-rio.workspace = true
@@ -41,7 +43,6 @@ serde_json.workspace = true
 tonic = { workspace = true }
 tokio = { workspace = true }
 tokio-stream = { workspace = true }
-clap = { workspace = true }
 rustfs-madmin.workspace = true
 rustfs-filemeta.workspace = true
 bytes.workspace = true
@@ -53,8 +54,6 @@ async-compression = { workspace = true, features = ["tokio", "bzip2", "xz"] }
 async-trait = { workspace = true }
 flate2.workspace = true
 http.workspace = true
-http-body.workspace = true
-http-body-util.workspace = true
 reqwest = { workspace = true }
 rustfs-signer.workspace = true
 tracing = { workspace = true }
@@ -75,4 +74,7 @@ suppaftp = { workspace = true, features = ["tokio", "rustls-aws-lc-rs"] }
 rcgen.workspace = true
 anyhow.workspace = true
 rustls.workspace = true
+russh = { workspace = true }
+russh-sftp = { workspace = true }
 zip.workspace = true
+clap.workspace = true

crates/e2e_test/src/archive_download_integrity_test.rs

@@ -218,6 +218,34 @@ mod tests {
         Ok(builder.send().await?)
     }
 
+    async fn signed_get_request_with_headers(
+        url: &str,
+        access_key: &str,
+        secret_key: &str,
+        extra_headers: &[(&str, &str)],
+    ) -> Result<reqwest::Response, Box<dyn Error + Send + Sync>> {
+        let uri = url.parse::<http::Uri>()?;
+        let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
+        let mut request = http::Request::builder()
+            .method(http::Method::GET)
+            .uri(uri)
+            .header(HOST, authority)
+            .header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
+        for (name, value) in extra_headers {
+            request = request.header(*name, *value);
+        }
+
+        let signed = sign_v4(request.body(Body::empty())?, 0, access_key, secret_key, "", "us-east-1");
+
+        let client = local_http_client();
+        let mut builder = client.get(url);
+        for (name, value) in signed.headers() {
+            builder = builder.header(name, value);
+        }
+
+        Ok(builder.send().await?)
+    }
+
     async fn assert_archive_object_content_encoding(
         client: &S3Client,
         bucket: &str,
@@ -655,6 +683,42 @@ mod tests {
         Ok(())
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn test_multipart_get_ignores_empty_conditional_etag_headers() -> Result<(), Box<dyn Error + Send + Sync>> {
+        init_logging();
+        let mut env = RustFSTestEnvironment::new().await?;
+        env.start_rustfs_server(vec![]).await?;
+        env.create_test_bucket(MULTIPART_ARCHIVE_TEST_BUCKET).await?;
+
+        let client = env.create_s3_client();
+        let key = "multipart-empty-conditional-headers.zip";
+        let zip_bytes =
+            complete_archive_multipart_upload_with_content_encoding(&client, MULTIPART_ARCHIVE_TEST_BUCKET, key, None).await?;
+        let object_url = format!("{}/{}/{}", env.url, MULTIPART_ARCHIVE_TEST_BUCKET, key);
+
+        let response = signed_get_request_with_headers(
+            &object_url,
+            &env.access_key,
+            &env.secret_key,
+            &[("if-match", ""), ("if-none-match", "")],
+        )
+        .await?;
+        let status = response.status();
+        let body = response.bytes().await?;
+
+        assert_eq!(
+            status,
+            StatusCode::OK,
+            "unexpected multipart GET status {status}, body: {}",
+            String::from_utf8_lossy(body.as_ref())
+        );
+        assert_eq!(body.as_ref(), zip_bytes.as_slice());
+
+        env.stop_server();
+        Ok(())
+    }
+
     #[tokio::test]
     #[serial]
     async fn test_archive_multipart_with_aws_chunked_and_effective_encoding_roundtrips_by_default()

crates/e2e_test/src/bin/tls_gen.rs

@@ -0,0 +1,22 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use clap::Parser;
+use e2e_test::tls_gen::{Args, run};
+
+fn main() -> anyhow::Result<()> {
+    let out_dir = run(Args::parse())?;
+    println!("Generated RustFS TLS bundle in {}", out_dir.display());
+    Ok(())
+}