test(notify): cover prefix suffix event dispatch (#2960)

Co-authored-by: houseme <housemecn@gmail.com>

.agents/skills/code-change-verification/SKILL.md

@@ -0,0 +1,78 @@
+---
+name: code-change-verification
+description: Verify code changes by identifying correctness, regression, security, and performance risks from diffs or patches, then produce prioritized findings with file/line evidence and concrete fixes. Use when reviewing commits, PRs, and merged patches before/after release.
+---
+
+# Code Change Verification
+
+Use this skill to review code changes consistently before merge, before release, and during incident follow-up.
+
+## Quick Start
+
+1. Read the scope: commit, PR, patch, or file list.
+2. Map each changed area by risk and user impact.
+3. Inspect each risky change in context.
+4. Report findings first, ordered by severity.
+5. Close with residual risks and verification recommendations.
+
+## Core Workflow
+
+### 1) Scope and assumptions
+- Confirm change source (diff, commit, PR, files), target branch, language/runtime, and version.
+- If context is missing, state assumptions before deeper analysis.
+- Focus only on requested scope; avoid reviewing unrelated files.
+
+### 2) Risk map
+- Prioritize in this order:
+  - Data correctness and user-visible behavior
+  - API/contract compatibility
+  - Security and authz/authn boundaries
+  - Concurrency and lifecycle correctness
+  - Performance and resource usage
+- Give higher priority to stateful paths, migration logic, defaults, and error handling.
+
+### 3) Evidence-based inspection
+- Read each modified hunk with neighboring context.
+- Trace call paths and call-site expectations.
+- Check for:
+  - invariant breaks and missing guards
+  - unchecked assumptions and null/empty/error-path handling
+  - stale tests, fixtures, and configs
+  - hidden coupling to shared helpers/constants/features
+- If a point is uncertain, mark it as an open question instead of guessing.
+
+### 4) Findings-first output
+- Order findings by severity:
+  - P0: critical failure, security breach, or data loss risk
+  - P1: high-impact regression
+  - P2: medium risk correctness gap
+  - P3: low risk/quality debt
+- For each finding include:
+  - Severity
+  - `path:line` reference
+  - concise issue statement
+  - impact and likely failure mode
+  - specific fix or mitigation
+  - validation step to confirm
+- If no issues exist, explicitly state `No findings` and why.
+
+### 5) Close
+- Report assumptions and unknowns.
+- Suggest targeted checks (tests, canary checks, logs/metrics, migration validation).
+
+## Output Template
+
+1. Findings
+2. No findings (if applicable)
+3. Assumptions / Unknowns
+4. Recommended verification steps
+
+## Finding Template
+
+- `[P1] Missing timeout for downstream call`
+  - Location: `path/to/file.rs:123`
+  - Issue: ...
+  - Impact: ...
+  - Fix suggestion: ...
+  - Validation: ...
+

.agents/skills/code-change-verification/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Code Change Verification"
+  short_description: "Prioritize risks and verify code changes before merge."
+  default_prompt: "Inspect a patch or diff, identify correctness/security/regression risks, and return prioritized findings with file/line evidence and fixes."

.agents/skills/pr-creation-checker/SKILL.md

@@ -0,0 +1,91 @@
+---
+name: pr-creation-checker
+description: Prepare PR-ready diffs by validating scope, checking required verification steps, drafting a compliant English PR title/body, and surfacing blockers before opening or updating a pull request in RustFS.
+---
+
+# PR Creation Checker
+
+Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whether a branch is ready for PR.
+
+## Read sources of truth first
+
+- Read `AGENTS.md`.
+- Read `.github/pull_request_template.md`.
+- Use `Makefile` and `.config/make/` for local quality commands.
+- Use `.github/workflows/ci.yml` for CI expectations.
+- Do not restate long command matrices or template sections from memory when the files exist.
+
+## Workflow
+
+1. Collect PR context
+- Confirm base branch, current branch, change goal, and scope.
+- Confirm whether the task is: draft a new PR, update an existing PR, or preflight-check readiness.
+- Confirm whether the branch includes only intended changes.
+
+2. Inspect change scope
+- Review the diff and summarize what changed.
+- Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
+- Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
+- Scan the diff for newly added string literals and confirm whether they duplicate values already defined as constants/enums/typed wrappers in the same module or shared modules.
+- Treat introducing a new hardcoded literal where a project constant already exists as a likely regression risk; require either a refactor to reuse the constant or an explicit exception explanation in the PR body.
+
+3. Verify readiness requirements
+- Require `make pre-commit` before marking the PR ready.
+- If `make` is unavailable, use the equivalent commands from `.config/make/`.
+- Add scope-specific verification commands when the changed area needs more than the baseline.
+- If required checks fail, stop and return `BLOCKED`.
+
+4. Draft PR metadata
+- Write the PR title in English using Conventional Commits and keep it within 72 characters.
+- If a generic PR workflow suggests a different title format, ignore it and follow the repository rule instead.
+- In RustFS, do not use tool-specific prefixes such as `[codex]` when the repository requires Conventional Commits.
+- Keep the PR body in English.
+- Use the exact section headings from `.github/pull_request_template.md`.
+- Fill non-applicable sections with `N/A`.
+- Include verification commands in the PR description.
+- Do not include local filesystem paths in the PR body unless the user explicitly asks for them.
+- Prefer repo-relative paths, command names, and concise summaries over machine-specific paths such as `/Users/...`.
+
+5. Prepare reviewer context
+- Summarize why the change exists.
+- Summarize what was verified.
+- Call out risks, rollout notes, config impact, and rollback notes when applicable.
+- Mention assumptions or missing context instead of guessing.
+
+6. Prepare CLI-safe output
+- When proposing `gh pr create` or `gh pr edit`, use `--body-file`, never inline `--body` for multiline markdown.
+- Return a ready-to-save PR body plus a short title.
+- If not ready, return blockers first and list the minimum steps needed to unblock.
+
+## Output format
+
+### Status
+- `READY` or `BLOCKED`
+
+### Title
+- `<type>(<scope>): <summary>`
+
+### PR Body
+- Reproduce the repository template headings exactly.
+- Fill every section.
+- Omit local absolute paths unless explicitly required.
+
+### Verification
+- List each command run.
+- State pass/fail.
+
+### Risks
+- List breaking changes, config changes, migration impact, or `N/A`.
+
+## Blocker rules
+
+- Return `BLOCKED` if `make pre-commit` has not passed.
+- Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
+- Return `BLOCKED` if required template sections are missing.
+- Return `BLOCKED` if the title/body is not in English.
+- Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
+- Return `BLOCKED` if the diff introduces string literals that should use existing constants but did not.
+
+## Reference
+
+- Use [pr-readiness-checklist.md](references/pr-readiness-checklist.md) for a short final pass before opening or editing the PR.

.agents/skills/pr-creation-checker/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "PR Creation Checker"
+  short_description: "Draft RustFS-ready PRs with checks, template, and blockers."
+  default_prompt: "Inspect a branch or diff, verify required PR checks, and produce a compliant English PR title/body plus blockers or readiness status."

.agents/skills/pr-creation-checker/references/pr-readiness-checklist.md

@@ -0,0 +1,15 @@
+# PR Readiness Checklist
+
+- Confirm the branch is based on current `main`.
+- Confirm the diff matches the stated scope.
+- Confirm no secrets, logs, temp files, or unrelated refactors are included.
+- Confirm `make pre-commit` passed, or document why it could not run.
+- Confirm extra verification commands are listed for risky changes.
+- Confirm the PR title uses Conventional Commits and stays within 72 characters.
+- Confirm the PR title does not use tool-specific prefixes such as `[codex]`.
+- Confirm the PR body is in English.
+- Confirm the PR body keeps the exact headings from `.github/pull_request_template.md`.
+- Confirm non-applicable sections are filled with `N/A`.
+- Confirm the PR body does not include local absolute paths unless explicitly required.
+- Confirm multiline GitHub CLI commands use `--body-file`.
+- Confirm new hardcoded string literals were not introduced for values already represented by existing constants/enums (including protocol labels, error identifiers, headers, and metric names), or record a justified exception.

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: rustfs-release-version-bump
+description: Prepare a RustFS release branch like PR #2957 by bumping versioned files, aligning release assets, running required verification, and finishing with commit, push, and gh PR creation. Use when publishing a new RustFS alpha, beta, or stable release.
+---
+
+# RustFS Release Version Bump
+
+Use this skill when the task is to prepare a new RustFS version release branch following the pattern validated in PR `#2957`.
+
+## Read first
+
+- Read `AGENTS.md`.
+- Read `.github/pull_request_template.md`.
+- Inspect the current branch diff against `origin/main`.
+- Do not assume every release file should change in the same way every time.
+
+## Scope validated by PR #2957
+
+The `1.0.0-beta.3` release branch updated these files:
+
+- `Cargo.toml`
+- `Cargo.lock`
+- `README.md`
+- `README_ZH.md`
+- `flake.nix`
+- `helm/rustfs/Chart.yaml`
+- `rustfs.spec`
+
+Treat this file list as the default checklist for future release bumps. Only drop a file if the repository's current release pattern clearly says it is no longer part of the publish flow.
+
+## Workflow
+
+1. Confirm release intent
+- Identify the target version string exactly, for example `1.0.0-beta.4`.
+- Check whether the user wants only local commit preparation or the full `commit + push + PR` flow.
+- Compare the branch against `origin/main` and isolate only release-related edits.
+- If the target version is not explicit, stop and ask for it before editing.
+
+2. Update core Rust workspace versions
+- Bump the workspace version in `Cargo.toml`.
+- Bump all internal workspace crate versions in `Cargo.toml`.
+- Ensure `Cargo.lock` reflects the same release version for workspace packages.
+- Re-read the diff and verify there are no partial version leftovers.
+
+3. Align release assets
+- Update versioned Docker examples in `README.md` and `README_ZH.md` using the `<version>` tag form, for example `rustfs/rustfs:1.0.0-beta.4`.
+- Update `flake.nix` package version.
+- Update `helm/rustfs/Chart.yaml` `appVersion`.
+- Update `helm/rustfs/Chart.yaml` `version` using the release-chart rule:
+- `1.0.0-beta.3` -> `0.3.0`
+- `1.0.0-beta.4` -> `0.4.0`
+- Follow the same pattern for later beta releases unless the repository rule changes.
+- Update `rustfs.spec` release metadata and changelog entry.
+- Set `rustfs.spec` `Release` to the prerelease suffix without the base version, for example `beta.3` for `1.0.0-beta.3`.
+- Keep the release asset changes in a separate commit from the core Rust workspace version bump when the branch contains both.
+
+4. Stop and discuss before changing release policy
+- Ask before changing the established Docker tag style away from `<version>`.
+- Ask before changing the established Helm chart version mapping away from `beta.N -> 0.N.0`.
+- Ask before changing the established `rustfs.spec` `Release` rule away from the release suffix form such as `beta.N`.
+- Ask before widening the release scope beyond the files already validated in PR `#2957`.
+
+5. Verify before shipping
+- Run `make pre-commit`.
+- If verification succeeds, run `cargo clean` to remove generated build artifacts before wrapping up.
+- If `make pre-commit` fails, stop and return `BLOCKED`.
+
+6. Commit structure
+- Prefer two commits when the change naturally splits:
+- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`
+- `chore(release): align release assets for <version>` for docs and packaging metadata
+- If the user asks for a single commit, follow that request.
+
+7. Push and PR
+- Push the release branch with `git push -u origin <branch>` or `git push` if upstream already exists.
+- Create the PR with `gh pr create --base main --head <branch> --title ... --body-file ...`.
+- Keep the PR title and body in English.
+- Keep the `.github/pull_request_template.md` headings exactly.
+
+## Ready-to-check commands
+
+- `git diff --name-only origin/main...HEAD`
+- `git diff --stat origin/main...HEAD`
+- `make pre-commit`
+- `cargo clean`
+- `git status --short --branch`
+
+## Output expectations
+
+When using this skill, return:
+
+- The files changed for the release bump
+- Any uncertainty that needs user confirmation before editing
+- Verification status
+- Commit messages used
+- Push status and PR URL when the GitHub flow was requested
+
+## Established release policy
+
+- Docs use Docker tags in `<version>` form, not `v<version>`.
+- `helm/rustfs/Chart.yaml` `version` follows `beta.N -> 0.N.0` based on the current release policy.
+- `rustfs.spec` `Release` follows the release suffix form, for example `beta.3`.
+- If any of these rules need to change in the future, pause and confirm before editing.

.agents/skills/rustfs-release-version-bump/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "RustFS Release Bump"
+  short_description: "Prepare RustFS release branches like PR #2957."
+  default_prompt: "Use $rustfs-release-version-bump to prepare a RustFS release version, ask about any unclear version policy, and finish the commit/push/PR flow."

.agents/skills/security-advisory-lessons/SKILL.md

@@ -0,0 +1,119 @@
+---
+name: security-advisory-lessons
+description: Apply RustFS security lessons distilled from repository GitHub Security Advisories. Use when making or reviewing RustFS code changes, doing security checks, handling PR review for auth/authz, IAM, storage, RPC, logging, CORS, console/browser, encryption, policy, or endpoint changes, and when deciding which security regression tests are required.
+---
+
+# RustFS Security Advisory Lessons
+
+Use this skill as a RustFS-specific security lens before changing or approving code. For the distilled advisory lessons and review patterns, read [advisory-patterns.md](references/advisory-patterns.md).
+
+When currentness matters, fetch the live advisory inventory instead of relying on this skill as a status mirror:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories --paginate \
+  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
+```
+
+Fetch full advisory details only when the live summary suggests a new or changed lesson:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
+```
+
+For the full pattern map, read [advisory-patterns.md](references/advisory-patterns.md).
+
+## Workflow
+
+### 1. Scope the change
+- Identify touched routes, handlers, storage paths, credentials, logs, browser surfaces, CI/release code, and policy checks.
+- Treat these paths as security-sensitive by default: `rustfs/src/admin/`, `rustfs/src/storage/`, `rustfs/src/auth.rs`, `rustfs/src/server/layer.rs`, `crates/iam/`, `crates/policy/`, `crates/credentials/`, `crates/ecstore/src/rpc/`, `crates/rio/`, and console preview/auth code.
+
+### 2. Map to advisory classes
+- Read [advisory-patterns.md](references/advisory-patterns.md) for matching GHSA lessons.
+- Do not rely on advisory titles alone. Confirm whether the issue is authentication, authorization, input validation, storage invariant, browser isolation, logging, or operational hardening.
+
+### 3. Verify fail-closed behavior
+- Check that unauthenticated, wrong-permission, cross-user, cross-bucket, malformed-input, and default-config cases fail explicitly.
+- Prefer exact action/permission checks over broad helper calls or inferred ownership.
+- Confirm lower storage/RPC layers do not bypass checks done in upper layers.
+
+### 4. Require regression evidence
+- For behavior changes, add focused negative tests that reproduce the advisory class.
+- For sensitive fixes, include tests for the bypass form, not only the happy path.
+- If a test is impractical, explain the residual risk and provide a manual verification command.
+
+### 5. Report clearly
+- Lead with concrete findings and file/line evidence.
+- Separate proven vulnerabilities from hardening risks.
+- Avoid exaggerating unauthenticated impact when the code actually rejects unauthenticated requests but allows a low-privileged authenticated bypass.
+
+## Advisory-Derived Guardrails
+
+### Auth and admin authorization
+- Every admin or diagnostic route needs an explicit authn and authz story. Route registration, router whitelist, and handler-level authorization must agree.
+- Match the admin action to the operation exactly. Copy-paste action constants are a known RustFS vulnerability class.
+- Avoid authentication-only helpers for state-changing admin APIs; use `validate_admin_request` or the established equivalent with the right `AdminAction`.
+- Do not assume admin-action `Resource` scoping constrains blast radius unless the policy engine actually enforces resources for that action.
+
+### IAM and service accounts
+- Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
+- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims.
+- Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
+- Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
+
+### S3 copy, multipart, and presigned POST
+- Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
+- Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
+- Presigned POST policies are server-side contracts. Enforce `content-length-range`, key prefix, exact metadata/content-type, and all signed policy conditions.
+
+### Paths, object keys, and filesystem access
+- Never join untrusted bucket/object/RPC path strings onto filesystem roots without normalization and boundary checks.
+- Reject or safely handle `..`, absolute paths, URL-encoded traversal, platform separators, empty components, and paths that canonicalize outside the intended root.
+- Validate both S3 object-key paths and internode/RPC disk paths; storage helpers can bypass S3 authorization if they trust already-parsed paths.
+
+### Secrets, default credentials, and crypto
+- Do not ship hard-coded shared tokens, HMAC secrets, private keys, or production test keys.
+- Defaults for internode/RPC auth must fail closed for network-reachable deployments or require explicit opt-in with loud warnings.
+- License or token validation must use signatures with embedded public/verifying keys only; do not use private-key decryption as authenticity.
+- Plan key rotation and key IDs when removing exposed keys.
+
+### Logging and debug output
+- Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials.
+- Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces.
+- Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies.
+
+### RPC, parsing, and panic safety
+- Treat all RPC payload bytes as attacker-controlled. Replace `unwrap`, `expect`, and panic-prone deserialization with typed errors.
+- Malformed request tests should cover empty bytes, truncated MessagePack/protobuf, invalid enum values, stale timestamps, and invalid signatures.
+- RPC authentication must be independently strong; do not depend on S3 admin credentials unless the fallback is explicit and safe.
+
+### Browser, CORS, and console surfaces
+- Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
+- Do not render user-controlled object content in a same-origin iframe with console credentials available to JavaScript.
+- Prefer origin separation for object preview/download, `nosniff`, CSP, strict content-type handling, and avoiding durable credentials in `localStorage`.
+- License/version-like metadata endpoints should expose only coarse public data unless authenticated.
+
+### Profiling, debug, and health endpoints
+- Profiling and debug endpoints are not health checks. They require admin auth, opt-in enablement, rate limiting, and safe responses.
+- Do not return absolute filesystem paths or other deployment layout in unauthenticated or low-privilege responses.
+- Ensure health endpoint allowlists cannot accidentally include expensive diagnostics.
+
+### Trusted proxy and network identity
+- Only honor `X-Forwarded-For` or `X-Real-IP` when the request came from a configured trusted proxy.
+- Direct clients must use the socket peer address for `aws:SourceIp` and policy condition evaluation.
+- Add tests for direct spoofed headers and trusted-proxy headers.
+
+### SSE and storage invariants
+- Encryption metadata is not proof that bytes were encrypted on disk.
+- When touching reader/writer wrappers such as hashing, encryption, compression, or warp readers, verify wrapper order and inspect stored bytes in regression tests.
+- Avoid helper shortcuts that unwrap nested readers and accidentally bypass encryption or integrity layers.
+
+## Review Prompts
+
+Use these prompts while reviewing a diff:
+
+- Could a low-privileged authenticated user reach this path with the wrong action, parent, bucket, or source object?
+- Does a public/default/empty config change security behavior from fail-closed to fail-open?
+- Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
+- Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
+- Does the test prove the exploit form is denied, or only that the intended form still works?

.agents/skills/security-advisory-lessons/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Security Advisory Lessons"
+  short_description: "Apply advisory lessons in reviews."
+  default_prompt: "Review code changes against past RustFS security advisory lessons and report concrete risks, missing tests, and recommended fixes."

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -0,0 +1,93 @@
+# RustFS Advisory Pattern Map
+
+This file is a lesson map, not an advisory inventory mirror. It keeps durable security patterns distilled from RustFS GitHub Security Advisories.
+
+When current advisory state, severity, URLs, or full text matters, fetch it live:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories --paginate \
+  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
+gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
+```
+
+Update this file only when an advisory adds or changes a reusable lesson, affected surface, validation pattern, or regression-test expectation. Do not update it for state-only, URL-only, count-only, or timestamp-only changes.
+
+## Pattern Index
+
+### Admin authorization and route exposure
+
+- `GHSA-pfcq-4gjr-6gjm`: notification target endpoints accepted authenticated users but skipped admin authorization. Lesson: distinguish authn from authz; admin target CRUD must call the operation-specific admin authorization path.
+- `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
+- `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
+- `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
+- `GHSA-x5xv-223c-8vm7`: console license metadata endpoint was public. Lesson: public metadata endpoints should be coarse or authenticated.
+
+### IAM import, service accounts, and privilege boundaries
+
+- `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
+- `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
+- `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
+
+### S3 copy, multipart, and upload policy validation
+
+- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
+- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
+- `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
+
+### Filesystem paths and object key traversal
+
+- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
+- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
+
+### Secrets, defaults, and cryptographic misuse
+
+- `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
+- `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
+- `GHSA-923g-jp7v-f97f`: license verification embedded a production RSA private key and used private-key decryption as authenticity. Lesson: ship verifying/public keys only and use real signature verification.
+
+### Sensitive logging and debug output
+
+- `GHSA-r54g-49rx-98cr`: STS credentials were logged at info level. Lesson: generated credentials must never be logged in plaintext.
+- `GHSA-8cm2-h255-v749`: debug logs leaked session tokens, secret keys, JWT claims, and raw STS response bodies. Lesson: redaction must cover custom `Debug` implementations and dependency response-body logging.
+- `GHSA-333v-68xh-8mmq`: invalid RPC signature logging included the shared HMAC secret and expected signature. Lesson: error paths often leak secrets; never log raw secrets or derived authenticators.
+
+### RPC input validation and panic safety
+
+- `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
+- `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
+
+### Browser, CORS, and console isolation
+
+- `GHSA-v9fg-3cr2-277j`: object preview rendered attacker-controlled HTML in a same-origin iframe, exposing console credentials stored in `localStorage`. Lesson: user content must be origin-isolated from the console and protected with `nosniff`, CSP, and strict content-type handling.
+- `GHSA-x5xv-223c-8vm7`: default CORS reflected arbitrary origins with credentials. Lesson: never combine reflected origins with `Access-Control-Allow-Credentials: true`; default should be fail-closed.
+
+### Trusted proxy and source IP conditions
+
+- `GHSA-fc6g-2gcp-2qrq`: `aws:SourceIp` trusted client-supplied `X-Forwarded-For` or `X-Real-IP`. Lesson: forwarded IP headers are valid only behind configured trusted proxies; direct clients use socket peer IP.
+
+### SSE and on-disk storage invariants
+
+- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
+
+## Useful Search Seeds
+
+Use these targeted searches when a diff touches security-sensitive code:
+
+```bash
+rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
+rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
+rg -n "PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
+rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
+rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
+rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
+```
+
+## Minimum Regression Test Expectations
+
+- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
+- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
+- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
+- Path fixes: include encoded traversal, absolute path, nested traversal, valid object keys that resemble traversal text but should be rejected, and canonical boundary checks.
+- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
+- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
+- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.

.agents/skills/test-coverage-improver/SKILL.md

@@ -0,0 +1,66 @@
+---
+name: test-coverage-improver
+description: Run project coverage checks, rank high-risk gaps, and propose high-impact tests to improve regression confidence for changed and critical code paths before release.
+---
+
+# Test Coverage Improver
+
+Use this skill when you need a prioritized, risk-aware plan to improve tests from coverage results.
+
+## Usage assumptions
+- Focus scope is either changed lines/files, a module, or the whole repository.
+- Coverage artifact must be generated or provided in a supported format.
+- If required context is missing, call out assumptions explicitly before proposing work.
+
+## Workflow
+
+1. Define scope and baseline
+   - Confirm target language, framework, and branch.
+   - Confirm whether the scope is changed files only or full-repo.
+
+2. Produce coverage snapshot
+   - Rust: `cargo llvm-cov` (or `cargo tarpaulin`) with existing repo config.
+   - JavaScript/TypeScript: `npm test -- --coverage` and read `coverage/coverage-final.json`.
+   - Python: `pytest --cov=<pkg> --cov-report=json` and read `coverage.json`.
+   - Collect total, per-file, and changed-line coverage.
+
+3. Rank highest-risk gaps
+   - Prioritize changed code, branch coverage gaps, and low-confidence boundaries.
+   - Apply the risk rubric in [coverage-prioritization.md](references/coverage-prioritization.md).
+   - Keep shortlist to 5–8 gaps.
+   - For each gap, capture: file, lines, uncovered branches, and estimated risk score.
+
+4. Propose high-impact tests
+   - For each shortlisted gap, output:
+     - Intent and expected behavior.
+     - Normal, edge, and failure scenarios.
+     - Assertions and side effects to verify.
+     - Setup needs (fixtures, mocks, integration dependencies).
+     - Estimated effort (`S/M/L`).
+
+5. Close with validation plan
+   - State which gaps remain after proposals.
+   - Provide concrete verification command and acceptance threshold.
+   - List assumptions or blockers (environment, fixtures, flaky dependencies).
+
+## Output template
+
+### Coverage Snapshot
+- total / branch coverage
+- changed-file coverage
+- top missing regions by size
+
+### Top Gaps (ranked)
+- `path:line-range` | risk score | why critical
+
+### Test Proposals
+- `path:line-range`
+  - Test name
+  - scenarios
+  - assertions
+  - effort
+
+### Validation Plan
+- command
+- pass criteria
+- remaining risk

.agents/skills/test-coverage-improver/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Test Coverage Improver"
+  short_description: "Find top uncovered risk areas and propose high-impact tests."
+  default_prompt: "Run coverage checks, identify largest gaps, and recommend highest-impact test cases to improve risk coverage."

.agents/skills/test-coverage-improver/references/coverage-prioritization.md

@@ -0,0 +1,25 @@
+# Coverage Gap Prioritization Guide
+
+Use this rubric for each uncovered area.
+
+Score = (Criticality × 2) + CoverageDebt + (Volatility × 0.5)
+
+- Criticality:
+  - 5: authz/authn, data-loss, payment/consistency path
+  - 4: state mutation, cache invalidation, scheduling
+  - 3: error handling + fallbacks in user-visible flows
+  - 2: parsing/format conversion paths
+  - 1: logging-only or low-impact utilities
+
+- CoverageDebt:
+  - 0: 0–5 uncovered lines
+  - 1: 6–20 uncovered lines
+  - 2: 21–40 uncovered lines
+  - 3: 41+ uncovered lines
+
+- Volatility:
+  - 1: stable legacy code with few recent edits
+  - 2: changed in last 2 releases
+  - 3: touched in last 30 days or currently in active PR
+
+Sort by score descending, then by business impact.

.cargo/config.toml

@@ -0,0 +1,26 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# RustFS Cargo configuration
+
+# Enable tokio_unstable cfg for dial9-tokio-telemetry support
+# This allows dial9 to hook into Tokio's internal runtime events
+[build]
+# Enable Tokio unstable features required by dial9-tokio-telemetry for runtime tracing.
+# See: https://docs.rs/tokio/latest/tokio/#unstable-features
+rustflags = ["--cfg", "tokio_unstable"]
+
+# Enable frame pointers for CPU profiling (Linux only, optional but recommended)
+# Uncomment the following line for better CPU profiling data
+# rustflags = ["--cfg", "tokio_unstable", "-C", "force-frame-pointers=yes"]

.config/make/build-docker-buildx-dev.mak

@@ -0,0 +1,64 @@
+## —— Development/Source builds using direct buildx commands ---------------------------------------
+
+.PHONY: docker-dev
+docker-dev: ## Build dev multi-arch image (cannot load locally)
+	@echo "🏗️ Building multi-architecture development Docker images with buildx..."
+	@echo "💡 This builds from source code and is intended for local development and testing"
+	@echo "⚠️  Multi-arch images cannot be loaded locally, use docker-dev-push to push to registry"
+	$(DOCKER_CLI) buildx build \
+		--platform linux/amd64,linux/arm64 \
+		--file $(DOCKERFILE_SOURCE) \
+		--tag rustfs:source-latest \
+		--tag rustfs:dev-latest \
+		.
+
+.PHONY: docker-dev-local
+docker-dev-local: ## Build dev single-arch image (local load)
+	@echo "🏗️ Building single-architecture development Docker image for local use..."
+	@echo "💡 This builds from source code for the current platform and loads locally"
+	$(DOCKER_CLI) buildx build \
+		--file $(DOCKERFILE_SOURCE) \
+		--tag rustfs:source-latest \
+		--tag rustfs:dev-latest \
+		--load \
+		.
+
+.PHONY: docker-dev-push
+docker-dev-push: ## Build and push multi-arch development image # e.g (make docker-dev-push REGISTRY=xxx)
+	@if [ -z "$(REGISTRY)" ]; then \
+		echo "❌ Error: Please specify registry, example: make docker-dev-push REGISTRY=ghcr.io/username"; \
+		exit 1; \
+	fi
+	@echo "🚀 Building and pushing multi-architecture development Docker images..."
+	@echo "💡 Pushing to registry: $(REGISTRY)"
+	$(DOCKER_CLI) buildx build \
+		--platform linux/amd64,linux/arm64 \
+		--file $(DOCKERFILE_SOURCE) \
+		--tag $(REGISTRY)/rustfs:source-latest \
+		--tag $(REGISTRY)/rustfs:dev-latest \
+		--push \
+		.
+
+.PHONY: dev-env-start
+dev-env-start: ## Start development container environment
+	@echo "🚀 Starting development environment..."
+	$(DOCKER_CLI) buildx build \
+		--file $(DOCKERFILE_SOURCE) \
+		--tag rustfs:dev \
+		--load \
+		.
+	$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
+	$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
+	$(DOCKER_CLI) run -d --name $(CONTAINER_NAME) \
+		-p 9010:9010 -p 9000:9000 \
+		-v $(shell pwd):/workspace \
+		-it rustfs:dev
+
+.PHONY: dev-env-stop
+dev-env-stop: ## Stop development container environment
+	@echo "🛑 Stopping development environment..."
+	$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
+	$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
+
+.PHONY: dev-env-restart
+dev-env-restart: dev-env-stop dev-env-start ## Restart development container environment

.config/make/build-docker-buildx-production.mak

@@ -0,0 +1,41 @@
+## —— Production builds using docker buildx (for CI/CD and production) -----------------------------
+
+.PHONY: docker-buildx
+docker-buildx: ## Build production multi-arch image (no push)
+	@echo "🏗️ Building multi-architecture production Docker images with buildx..."
+	./docker-buildx.sh
+
+.PHONY: docker-buildx-push
+docker-buildx-push: ## Build and push production multi-arch image
+	@echo "🚀 Building and pushing multi-architecture production Docker images with buildx..."
+	./docker-buildx.sh --push
+
+.PHONY: docker-buildx-version
+docker-buildx-version: ## Build and version production multi-arch image # e.g (make docker-buildx-version VERSION=v1.0.0)
+	@if [ -z "$(VERSION)" ]; then \
+		echo "❌ Error: Please specify version, example: make docker-buildx-version VERSION=v1.0.0"; \
+		exit 1; \
+	fi
+	@echo "🏗️ Building multi-architecture production Docker images (version: $(VERSION))..."
+	./docker-buildx.sh --release $(VERSION)
+
+.PHONY: docker-buildx-push-version
+docker-buildx-push-version: ## Build and version and push production multi-arch image # e.g (make docker-buildx-push-version VERSION=v1.0.0)
+	@if [ -z "$(VERSION)" ]; then \
+		echo "❌ Error: Please specify version, example: make docker-buildx-push-version VERSION=v1.0.0"; \
+		exit 1; \
+	fi
+	@echo "🚀 Building and pushing multi-architecture production Docker images (version: $(VERSION))..."
+	./docker-buildx.sh --release $(VERSION) --push
+
+.PHONY: docker-buildx-production-local
+docker-buildx-production-local: ## Build production single-arch image locally
+	@echo "🏗️ Building single-architecture production Docker image locally..."
+	@echo "💡 Alternative to docker-buildx.sh for local testing"
+	$(DOCKER_CLI) buildx build \
+		--file $(DOCKERFILE_PRODUCTION) \
+		--tag rustfs:production-latest \
+		--tag rustfs:latest \
+		--load \
+		--build-arg RELEASE=latest \
+		.

.config/make/build-docker-production.mak

@@ -0,0 +1,16 @@
+## —— Single Architecture Docker Builds (Traditional) ----------------------------------------------
+
+.PHONY: docker-build-production
+docker-build-production: ## Build single-arch production image
+	@echo "🏗️ Building single-architecture production Docker image..."
+	@echo "💡 Consider using 'make docker-buildx-production-local' for multi-arch support"
+	$(DOCKER_CLI) build -f $(DOCKERFILE_PRODUCTION) -t rustfs:latest .
+
+.PHONY: docker-build-source
+docker-build-source: ## Build single-arch source image
+	@echo "🏗️ Building single-architecture source Docker image..."
+	@echo "💡 Consider using 'make docker-dev-local' for multi-arch support"
+	DOCKER_BUILDKIT=1 $(DOCKER_CLI) build \
+		--build-arg BUILDKIT_INLINE_CACHE=1 \
+		-f $(DOCKERFILE_SOURCE) -t rustfs:source .
+

.config/make/build-docker.mak

@@ -0,0 +1,22 @@
+## —— Docker-based build (alternative approach) ----------------------------------------------------
+
+# Usage: make BUILD_OS=ubuntu22.04 build-docker
+# Output: target/ubuntu22.04/release/rustfs
+
+.PHONY: build-docker
+build-docker: SOURCE_BUILD_IMAGE_NAME = rustfs-$(BUILD_OS):v1
+build-docker: SOURCE_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
+build-docker: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
+build-docker: ## Build using Docker container # e.g (make build-docker BUILD_OS=ubuntu22.04)
+	@echo "🐳 Building RustFS using Docker ($(BUILD_OS))..."
+	$(DOCKER_CLI) buildx build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) .
+	$(DOCKER_CLI) run --rm --name $(SOURCE_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(SOURCE_BUILD_IMAGE_NAME) $(BUILD_CMD)
+
+.PHONY: docker-inspect-multiarch
+docker-inspect-multiarch: ## Check image architecture support
+	@if [ -z "$(IMAGE)" ]; then \
+		echo "❌ Error: Please specify image, example: make docker-inspect-multiarch IMAGE=rustfs/rustfs:latest"; \
+		exit 1; \
+	fi
+	@echo "🔍 Inspecting multi-architecture image: $(IMAGE)"
+	docker buildx imagetools inspect $(IMAGE)

.config/make/build.mak

@@ -0,0 +1,55 @@
+## —— Local Native Build using build-rustfs.sh script (Recommended) --------------------------------
+
+.PHONY: build
+build: ## Build RustFS binary (includes console by default)
+	@echo "🔨 Building RustFS using build-rustfs.sh script..."
+	./build-rustfs.sh
+
+.PHONY: build-dev
+build-dev: ## Build RustFS in Development mode
+	@echo "🔨 Building RustFS in development mode..."
+	./build-rustfs.sh --dev
+
+.PHONY: build-musl
+build-musl: ## Build x86_64 musl version
+	@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
+	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
+	./build-rustfs.sh --platform x86_64-unknown-linux-musl
+
+.PHONY: build-gnu
+build-gnu: ## Build x86_64 GNU version
+	@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
+	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
+	./build-rustfs.sh --platform x86_64-unknown-linux-gnu
+
+.PHONY: build-musl-arm64
+build-musl-arm64: ## Build aarch64 musl version
+	@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
+	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
+	./build-rustfs.sh --platform aarch64-unknown-linux-musl
+
+.PHONY: build-gnu-arm64
+build-gnu-arm64: ## Build aarch64 GNU version
+	@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
+	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
+	./build-rustfs.sh --platform aarch64-unknown-linux-gnu
+
+
+.PHONY: build-cross-all
+build-cross-all: core-deps ## Build binaries for all architectures
+	@echo "🔧 Building all target architectures..."
+	@echo "💡 On macOS/Windows, use 'make docker-dev' for reliable multi-arch builds"
+	@echo "🔨 Generating protobuf code..."
+	cargo run --bin gproto || true
+
+	@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
+	./build-rustfs.sh --platform x86_64-unknown-linux-musl
+
+	@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
+	./build-rustfs.sh --platform x86_64-unknown-linux-gnu
+
+	@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
+	./build-rustfs.sh --platform aarch64-unknown-linux-musl
+
+	@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
+	./build-rustfs.sh --platform aarch64-unknown-linux-gnu

.config/make/check.mak

@@ -0,0 +1,24 @@
+## —— Check and Inform Dependencies ----------------------------------------------------------------
+
+# Fatal check
+# Checks all required dependencies and exits with error if not found
+# (e.g., cargo, rustfmt)
+check-%:
+	@command -v $* >/dev/null 2>&1 || { \
+		echo >&2 "❌ '$*' is not installed."; \
+		exit 1; \
+	}
+
+# Warning-only check
+# Checks for optional dependencies and issues a warning if not found
+# (e.g., cargo-nextest for enhanced testing)
+warn-%:
+	@command -v $* >/dev/null 2>&1 || { \
+		echo >&2 "⚠️ '$*' is not installed."; \
+	}
+
+# For checking dependencies use check-<dep-name> or warn-<dep-name>
+.PHONY: core-deps fmt-deps test-deps
+core-deps: check-cargo ## Check core dependencies
+fmt-deps: check-rustfmt ## Check lint and formatting dependencies
+test-deps: warn-cargo-nextest ## Check tests dependencies

.config/make/deploy.mak

@@ -0,0 +1,6 @@
+## —— Deploy using dev_deploy.sh script ------------------------------------------------------------
+
+.PHONY: deploy-dev
+deploy-dev: build-musl ## Deploy to dev server
+	@echo "🚀 Deploying to dev server: $${IP}"
+	./scripts/dev_deploy.sh $${IP}

.config/make/help.mak

@@ -0,0 +1,38 @@
+## —— Help, Help Build and Help Docker -------------------------------------------------------------
+
+
+.PHONY: help
+help: ## Shows This Help Menu
+	echo -e "$$HEADER"
+	grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} ; {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}' | sed -e 's/\[36m##/\n[32m##/'
+
+.PHONY: help-build
+help-build: ## Shows RustFS build help
+	@echo ""
+	@echo "💡 build-rustfs.sh script provides more options, smart detection and binary verification"
+	@echo ""
+	@echo "🔧 Direct usage of build-rustfs.sh script:"
+	@echo ""
+	@echo "	./build-rustfs.sh --help                                # View script help"
+	@echo "	./build-rustfs.sh --no-console                          # Build without console resources"
+	@echo "	./build-rustfs.sh --force-console-update                # Force update console resources"
+	@echo "	./build-rustfs.sh --dev                                 # Development mode build"
+	@echo "	./build-rustfs.sh --sign                                # Sign binary files"
+	@echo "	./build-rustfs.sh --platform x86_64-unknown-linux-gnu   # Specify target platform"
+	@echo "	./build-rustfs.sh --skip-verification                   # Skip binary verification"
+	@echo ""
+
+.PHONY: help-docker
+help-docker: ## Shows docker environment and suggestion help
+	@echo ""
+	@echo "📋 Environment Variables:"
+	@echo "	REGISTRY                 Image registry address (required for push)"
+	@echo "	DOCKERHUB_USERNAME       Docker Hub username"
+	@echo "	DOCKERHUB_TOKEN          Docker Hub access token"
+	@echo "	GITHUB_TOKEN             GitHub access token"
+	@echo ""
+	@echo "💡 Suggestions:"
+	@echo "	Production use:          Use docker-buildx* commands (based on precompiled binaries)"
+	@echo "	Local development:       Use docker-dev* commands (build from source)"
+	@echo "	Development environment: Use dev-env-* commands to manage dev containers"
+	@echo ""

.config/make/lint-fmt.mak

@@ -0,0 +1,27 @@
+## —— Code quality and Formatting ------------------------------------------------------------------
+
+.PHONY: fmt
+fmt: core-deps fmt-deps ## Format code
+	@echo "🔧 Formatting code..."
+	cargo fmt --all
+
+.PHONY: fmt-check
+fmt-check: core-deps fmt-deps ## Check code formatting
+	@echo "📝 Checking code formatting..."
+	cargo fmt --all --check
+
+.PHONY: clippy-check
+clippy-check: core-deps ## Run clippy checks
+	@echo "🔍 Running clippy checks..."
+	cargo clippy --fix --allow-dirty
+	cargo clippy --all-targets --all-features -- -D warnings
+
+.PHONY: unsafe-code-check
+unsafe-code-check: ## Check unsafe_code allowances have SAFETY comments
+	@echo "🔒 Checking unsafe_code allowances..."
+	./scripts/check_unsafe_code_allowances.sh
+
+.PHONY: compilation-check
+compilation-check: core-deps ## Run compilation check
+	@echo "🔨 Running compilation check..."
+	cargo check --all-targets

.config/make/pre-commit.mak

@@ -0,0 +1,11 @@
+## —— Pre Commit Checks ----------------------------------------------------------------------------
+
+.PHONY: setup-hooks
+setup-hooks: ## Set up git hooks
+	@echo "🔧 Setting up git hooks..."
+	chmod +x .git/hooks/pre-commit
+	@echo "✅ Git hooks setup complete!"
+
+.PHONY: pre-commit
+pre-commit: fmt unsafe-code-check clippy-check compilation-check test ## Run pre-commit checks
+	@echo "✅ All pre-commit checks passed!"

.config/make/tests.mak

@@ -0,0 +1,27 @@
+## —— Tests and e2e test ---------------------------------------------------------------------------
+
+TEST_THREADS ?= 1
+
+.PHONY: script-tests
+script-tests: ## Run shell script tests
+	@echo "Running script tests..."
+	./scripts/test_build_rustfs_options.sh
+
+.PHONY: test
+test: core-deps test-deps script-tests ## Run all tests
+	@echo "🧪 Running tests..."
+	@if command -v cargo-nextest >/dev/null 2>&1; then \
+		cargo nextest run --all --exclude e2e_test; \
+	else \
+		echo "ℹ️ cargo-nextest not found; falling back to 'cargo test'"; \
+		cargo test --workspace --exclude e2e_test -- --nocapture --test-threads="$(TEST_THREADS)"; \
+	fi
+	cargo test --all --doc
+
+.PHONY: e2e-server
+e2e-server: ## Run e2e-server tests
+	sh $(shell pwd)/scripts/run.sh
+
+.PHONY: probe-e2e
+probe-e2e: ## Probe e2e tests
+	sh $(shell pwd)/scripts/probe.sh

.docker/README.md

@@ -0,0 +1,131 @@
+# RustFS Docker Infrastructure
+
+This directory contains the complete Docker infrastructure for building, deploying, and monitoring RustFS. It provides ready-to-use configurations for development, testing, and production-grade observability.
+
+## 📂 Directory Structure
+
+| Directory | Description | Status |
+| :--- | :--- | :--- |
+| **[`observability/`](observability/README.md)** | **[RECOMMENDED]** Full-stack observability (Prometheus, Grafana, Tempo, Loki). | ✅ Production-Ready |
+| **[`compose/`](compose/README.md)** | Specialized setups (e.g., 4-node distributed cluster testing). | ⚠️ Testing Only |
+| **[`mqtt/`](mqtt/README.md)** | EMQX Broker configuration for MQTT integration testing. | 🧪 Development |
+| **[`openobserve-otel/`](openobserve-otel/README.md)** | Alternative lightweight observability stack using OpenObserve. | 🔄 Alternative |
+
+---
+
+## 📄 Root Directory Files
+
+The following files in the project root are essential for Docker operations:
+
+### Build Scripts & Dockerfiles
+
+| File | Description | Usage |
+| :--- | :--- | :--- |
+| **`docker-buildx.sh`** | **Multi-Arch Build Script**<br>Automates building and pushing Docker images for `amd64` and `arm64`. Supports release and dev channels. | `./docker-buildx.sh --push` |
+| **`Dockerfile`** | **Production Image (Alpine)**<br>Lightweight image using musl libc. Downloads pre-built binaries from GitHub Releases. | `docker build -t rustfs:latest .` |
+| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Useful if you need specific dynamic libraries. | `docker build -f Dockerfile.glibc .` |
+| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code. Includes build tools. Ideal for local development and CI. | `docker build -f Dockerfile.source .` |
+
+### Docker Compose Configurations
+
+| File | Description | Usage |
+| :--- | :--- | :--- |
+| **`docker-compose.yml`** | **Main Development Setup**<br>Comprehensive setup with profiles for development, observability, and proxying. | `docker compose up -d`<br>`docker compose --profile observability up -d` |
+| **`docker-compose-simple.yml`** | **Quick Start Setup**<br>Minimal configuration running a single RustFS instance with 4 volumes. Perfect for first-time users. | `docker compose -f docker-compose-simple.yml up -d` |
+
+---
+
+## 🌟 Observability Stack (Recommended)
+
+Located in: [`.docker/observability/`](observability/README.md)
+
+We provide a comprehensive, industry-standard observability stack designed for deep insights into RustFS performance. This is the recommended setup for both development and production monitoring.
+
+### Components
+- **Metrics**: Prometheus (Collection) + Grafana (Visualization)
+- **Traces**: Tempo (Storage) + Jaeger (UI)
+- **Logs**: Loki
+- **Ingestion**: OpenTelemetry Collector
+
+### Key Features
+- **Full Persistence**: All metrics, logs, and traces are saved to Docker volumes, ensuring no data loss on restarts.
+- **Correlation**: Seamlessly jump between Logs, Traces, and Metrics in Grafana.
+- **High Performance**: Optimized configurations for batching, compression, and memory management.
+
+### Quick Start
+```bash
+cd .docker/observability
+docker compose up -d
+```
+
+---
+
+## 🧪 Specialized Environments
+
+Located in: [`.docker/compose/`](compose/README.md)
+
+These configurations are tailored for specific testing scenarios that require complex topologies.
+
+### Distributed Cluster (4-Nodes)
+Simulates a real-world distributed environment with 4 RustFS nodes running locally.
+```bash
+docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
+```
+
+### Integrated Observability Test
+A self-contained environment running 4 RustFS nodes alongside the full observability stack. Useful for end-to-end testing of telemetry.
+```bash
+docker compose -f .docker/compose/docker-compose.observability.yaml up -d
+```
+
+---
+
+## 📡 MQTT Integration
+
+Located in: [`.docker/mqtt/`](mqtt/README.md)
+
+Provides an EMQX broker for testing RustFS MQTT features.
+
+### Quick Start
+```bash
+cd .docker/mqtt
+docker compose up -d
+```
+- **Dashboard**: [http://localhost:18083](http://localhost:18083) (Default: `admin` / `public`)
+- **MQTT Port**: `1883`
+
+---
+
+## 👁️ Alternative: OpenObserve
+
+Located in: [`.docker/openobserve-otel/`](openobserve-otel/README.md)
+
+For users preferring a lightweight, all-in-one solution, we support OpenObserve. It combines logs, metrics, and traces into a single binary and UI.
+
+### Quick Start
+```bash
+cd .docker/openobserve-otel
+docker compose up -d
+```
+
+---
+
+## 🔧 Common Operations
+
+### Cleaning Up
+To stop all containers and remove volumes (**WARNING**: deletes all persisted data):
+```bash
+docker compose down -v
+```
+
+### Viewing Logs
+To follow logs for a specific service:
+```bash
+docker compose logs -f [service_name]
+```
+
+### Checking Status
+To see the status of all running containers:
+```bash
+docker compose ps
+```

.docker/compose/README.md

@@ -0,0 +1,62 @@
+# Specialized Docker Compose Configurations
+
+This directory contains specialized Docker Compose configurations for specific testing scenarios.
+
+## ⚠️ Important Note
+
+**For Observability:**
+We **strongly recommend** using the new, fully integrated observability stack located in `../observability/`. It provides a production-ready setup with Prometheus, Grafana, Tempo, Loki, and OpenTelemetry Collector, all with persistent storage and optimized configurations.
+
+The `docker-compose.observability.yaml` in this directory is kept for legacy reference or specific minimal testing needs but is **not** the primary recommended setup.
+
+## 📁 Configuration Files
+
+### Cluster Testing
+
+- **`docker-compose.cluster.yaml`**
+  - **Purpose**: Simulates a 4-node RustFS distributed cluster.
+  - **Use Case**: Testing distributed storage logic, consensus, and failover.
+  - **Nodes**: 4 RustFS instances.
+  - **Storage**: Uses local HTTP endpoints.
+
+### Legacy / Minimal Observability
+
+- **`docker-compose.observability.yaml`**
+  - **Purpose**: A minimal observability setup.
+  - **Status**: **Deprecated**. Please use `../observability/docker-compose.yml` instead.
+
+## 🚀 Usage Examples
+
+### Cluster Testing
+
+To start a 4-node cluster for distributed testing:
+
+```bash
+# From project root
+docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
+```
+
+### Script-Based 4-Node Validation (Recommended)
+
+Use the local validation script when you need local-source image build, failover checks,
+and benchmark workflow in one command:
+
+```bash
+# Default mode: WAIT_PROBE_MODE=service
+# This avoids false negatives where /health/ready remains 503 locally
+# while the service path is already available.
+./scripts/run_four_node_cluster_failover_bench.sh
+```
+
+Strict mode is available when you explicitly want `/health/ready == 200` as the gate:
+
+```bash
+WAIT_PROBE_MODE=ready ./scripts/run_four_node_cluster_failover_bench.sh
+```
+
+### (Deprecated) Minimal Observability
+
+```bash
+# From project root
+docker compose -f .docker/compose/docker-compose.observability.yaml up -d
+```

.docker/compose/docker-compose.cluster.local-build.yml

@@ -0,0 +1,144 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  node1:
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node1
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node1_data_0:/data/rustfs0
+      - node1_data_1:/data/rustfs1
+      - node1_data_2:/data/rustfs2
+      - node1_data_3:/data/rustfs3
+    ports:
+      - "9000:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node2:
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node2
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node2_data_0:/data/rustfs0
+      - node2_data_1:/data/rustfs1
+      - node2_data_2:/data/rustfs2
+      - node2_data_3:/data/rustfs3
+    ports:
+      - "9001:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node3:
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node3
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node3_data_0:/data/rustfs0
+      - node3_data_1:/data/rustfs1
+      - node3_data_2:/data/rustfs2
+      - node3_data_3:/data/rustfs3
+    ports:
+      - "9002:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node4:
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node4
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node4_data_0:/data/rustfs0
+      - node4_data_1:/data/rustfs1
+      - node4_data_2:/data/rustfs2
+      - node4_data_3:/data/rustfs3
+    ports:
+      - "9003:9000"
+    networks:
+      - rustfs-cluster-net
+
+volumes:
+  node1_data_0:
+  node1_data_1:
+  node1_data_2:
+  node1_data_3:
+  node2_data_0:
+  node2_data_1:
+  node2_data_2:
+  node2_data_3:
+  node3_data_0:
+  node3_data_1:
+  node3_data_2:
+  node3_data_3:
+  node4_data_0:
+  node4_data_1:
+  node4_data_2:
+  node4_data_3:
+
+networks:
+  rustfs-cluster-net:
+    driver: bridge

.docker/compose/docker-compose.cluster.yaml

@@ -0,0 +1,82 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  node0:
+    image: rustfs/rustfs:latest # Replace with your image name and label
+    container_name: node0
+    hostname: node0
+    environment:
+      - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
+    platform: linux/amd64
+    ports:
+      - "9000:9000" # Map port 9001 of the host to port 9000 of the container
+    volumes:
+      - ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
+    command: "/app/rustfs"
+
+  node1:
+    image: rustfs/rustfs:latest
+    container_name: node1
+    hostname: node1
+    environment:
+      - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
+    platform: linux/amd64
+    ports:
+      - "9001:9000" # Map port 9002 of the host to port 9000 of the container
+    volumes:
+      - ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
+    command: "/app/rustfs"
+
+  node2:
+    image: rustfs/rustfs:latest
+    container_name: node2
+    hostname: node2
+    environment:
+      - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
+    platform: linux/amd64
+    ports:
+      - "9002:9000" # Map port 9003 of the host to port 9000 of the container
+    volumes:
+      - ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
+    command: "/app/rustfs"
+
+  node3:
+    image: rustfs/rustfs:latest
+    container_name: node3
+    hostname: node3
+    environment:
+      - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
+    platform: linux/amd64
+    ports:
+      - "9003:9000" # Map port 9004 of the host to port 9000 of the container
+    volumes:
+      - ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
+    command: "/app/rustfs"

.docker/compose/docker-compose.observability.yaml

@@ -0,0 +1,282 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  # --- Observability Stack ---
+
+  tempo-init:
+    image: busybox:latest
+    command: [ "sh", "-c", "chown -R 10001:10001 /var/tempo" ]
+    volumes:
+      - tempo-data:/var/tempo
+    user: root
+    networks:
+      - rustfs-network
+    restart: "no"
+
+  tempo:
+    image: grafana/tempo:2.10.5
+    user: "10001"
+    command: [ "-config.file=/etc/tempo.yaml" ]
+    volumes:
+      - ../../.docker/observability/tempo.yaml:/etc/tempo.yaml:ro
+      - tempo-data:/var/tempo
+    ports:
+      - "3200:3200" # tempo
+      - "4317"      # otlp grpc
+      - "4318"      # otlp http
+      - "7946"      # memberlist
+    restart: unless-stopped
+    networks:
+      - rustfs-network
+    depends_on:
+      tempo-init:
+        condition: service_completed_successfully
+    healthcheck:
+      test: [ "CMD", "/tempo", "-version" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
+
+  otel-collector:
+    image: otel/opentelemetry-collector-contrib:latest
+    environment:
+      - TZ=Asia/Shanghai
+    volumes:
+      - ../../.docker/observability/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
+    ports:
+      - "1888:1888"   # pprof
+      - "8888:8888"   # Prometheus metrics for Collector
+      - "8889:8889"   # Prometheus metrics for application indicators
+      - "13133:13133" # health check
+      - "4317:4317"   # OTLP gRPC
+      - "4318:4318"   # OTLP HTTP
+      - "55679:55679" # zpages
+    networks:
+      - rustfs-network
+    depends_on:
+      - tempo
+      - jaeger
+      - prometheus
+      - loki
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "/otelcol-contrib", "--version" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+  jaeger:
+    image: jaegertracing/jaeger:latest
+    environment:
+      - TZ=Asia/Shanghai
+      - SPAN_STORAGE_TYPE=badger
+      - BADGER_EPHEMERAL=false
+      - BADGER_DIRECTORY_VALUE=/badger/data
+      - BADGER_DIRECTORY_KEY=/badger/key
+      - COLLECTOR_OTLP_ENABLED=true
+    volumes:
+      - ../../.docker/observability/jaeger.yaml:/etc/jaeger/config.yml:ro
+      - jaeger-data:/badger
+    ports:
+      - "16686:16686" # Web UI
+      - "14269:14269" # Admin/Metrics
+      - "4317"        # otlp grpc
+      - "4318"        # otlp http
+    command: [ "--config", "/etc/jaeger/config.yml" ]
+    networks:
+      - rustfs-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
+
+  prometheus:
+    image: prom/prometheus:latest
+    environment:
+      - TZ=Asia/Shanghai
+    volumes:
+      - ../../.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - ../../.docker/observability/prometheus-rules:/etc/prometheus/rules:ro
+      - prometheus-data:/prometheus
+    ports:
+      - "9090:9090"
+    command:
+      - '--config.file=/etc/prometheus/prometheus.yml'
+      - '--web.enable-otlp-receiver'
+      - '--web.enable-remote-write-receiver'
+      - '--enable-feature=promql-experimental-functions'
+      - '--storage.tsdb.path=/prometheus'
+      - '--storage.tsdb.retention.time=30d'
+    networks:
+      - rustfs-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+  loki:
+    image: grafana/loki:latest
+    environment:
+      - TZ=Asia/Shanghai
+    volumes:
+      - ../../.docker/observability/loki.yaml:/etc/loki/loki.yaml:ro
+      - loki-data:/loki
+    ports:
+      - "3100:3100"
+    command: -config.file=/etc/loki/loki.yaml
+    networks:
+      - rustfs-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
+      interval: 15s
+      timeout: 10s
+      retries: 5
+      start_period: 60s
+
+  pyroscope:
+    image: grafana/pyroscope:latest
+    ports:
+      - "4040:4040"
+    command:
+      - -self-profiling.disable-push=true
+    networks:
+      - rustfs-network
+    restart: unless-stopped
+
+  grafana:
+    image: grafana/grafana:latest
+    ports:
+      - "3000:3000" # Web UI
+    environment:
+      - GF_SECURITY_ADMIN_PASSWORD=admin
+      - GF_SECURITY_ADMIN_USER=admin
+      - TZ=Asia/Shanghai
+      - GF_INSTALL_PLUGINS=grafana-pyroscope-datasource
+      - GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/home.json
+    networks:
+      - rustfs-network
+    volumes:
+      - ../../.docker/observability/grafana/provisioning:/etc/grafana/provisioning:ro
+      - ../../.docker/observability/grafana/dashboards:/var/lib/grafana/dashboards:ro
+      - grafana-data:/var/lib/grafana
+    depends_on:
+      - prometheus
+      - tempo
+      - loki
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+  # --- RustFS Cluster ---
+
+  node1:
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    container_name: node1
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_LOGGER_LEVEL=debug
+    platform: linux/amd64
+    ports:
+      - "9001:9000"
+    networks:
+      - rustfs-network
+    depends_on:
+      - otel-collector
+
+  node2:
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    container_name: node2
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_LOGGER_LEVEL=debug
+    platform: linux/amd64
+    ports:
+      - "9002:9000"
+    networks:
+      - rustfs-network
+    depends_on:
+      - otel-collector
+
+  node3:
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    container_name: node3
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_LOGGER_LEVEL=debug
+    platform: linux/amd64
+    ports:
+      - "9003:9000"
+    networks:
+      - rustfs-network
+    depends_on:
+      - otel-collector
+
+  node4:
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    container_name: node4
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_LOGGER_LEVEL=debug
+    platform: linux/amd64
+    ports:
+      - "9004:9000"
+    networks:
+      - rustfs-network
+    depends_on:
+      - otel-collector
+
+volumes:
+  prometheus-data:
+  tempo-data:
+  loki-data:
+  jaeger-data:
+  grafana-data:
+
+networks:
+  rustfs-network:
+    driver: bridge
+    name: "network_rustfs_config"
+    driver_opts:
+      com.docker.network.enable_ipv6: "true"

.docker/mqtt/README.md

@@ -0,0 +1,30 @@
+# MQTT Broker (EMQX)
+
+This directory contains the configuration for running an EMQX MQTT broker, which can be used for testing RustFS's MQTT integration.
+
+## 🚀 Quick Start
+
+To start the EMQX broker:
+
+```bash
+docker compose up -d
+```
+
+## 📊 Access
+
+- **Dashboard**: [http://localhost:18083](http://localhost:18083)
+- **Default Credentials**: `admin` / `public`
+- **MQTT Port**: `1883`
+- **WebSocket Port**: `8083`
+
+## 🛠️ Configuration
+
+The `docker-compose.yml` file sets up a single-node EMQX instance.
+
+- **Persistence**: Data is not persisted by default (for testing).
+- **Network**: Uses the default bridge network.
+
+## 📝 Notes
+
+- This setup is intended for development and testing purposes.
+- For production deployments, please refer to the official [EMQX Documentation](https://www.emqx.io/docs/en/latest/).

.docker/mqtt/config/emqx.conf

@@ -0,0 +1,37 @@
+# 节点配置
+node.name = "emqx@127.0.0.1"
+node.cookie = "aBcDeFgHiJkLmNoPqRsTuVwXyZ012345"
+node.data_dir = "/opt/emqx/data"
+
+# 日志配置
+log.console = {level = info, enable = true}
+log.file = {path = "/opt/emqx/log/emqx.log", enable = true, level = info}
+
+# MQTT TCP 监听器
+listeners.tcp.default = {bind = "0.0.0.0:1883", max_connections = 1000000, enable = true}
+
+# MQTT SSL 监听器
+listeners.ssl.default = {bind = "0.0.0.0:8883", enable = false}
+
+# MQTT WebSocket 监听器
+listeners.ws.default = {bind = "0.0.0.0:8083", enable = true}
+
+# MQTT WebSocket SSL 监听器
+listeners.wss.default = {bind = "0.0.0.0:8084", enable = false}
+
+# 管理控制台
+dashboard.listeners.http = {bind = "0.0.0.0:18083", enable = true}
+
+# HTTP API
+management.listeners.http = {bind = "0.0.0.0:8081", enable = true}
+
+# 认证配置
+authentication = [
+  {enable = true, mechanism = password_based, backend = built_in_database, user_id_type = username}
+]
+
+# 授权配置
+authorization.sources = [{type = built_in_database, enable = true}]
+
+# 持久化消息存储
+message.storage.backend = built_in_database

.docker/mqtt/config/vm.args

@@ -0,0 +1,9 @@
+-name emqx@127.0.0.1
+-setcookie aBcDeFgHiJkLmNoPqRsTuVwXyZ012345
++P 2097152
++t 1048576
++zdbbl 32768
+-kernel inet_dist_listen_min 6000
+-kernel inet_dist_listen_max 6100
+-smp enable
+-mnesia dir "/opt/emqx/data/mnesia"

.docker/mqtt/docker-compose-more.yml

@@ -0,0 +1,74 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  emqx:
+    image: emqx/emqx:latest
+    container_name: emqx
+    restart: unless-stopped
+    environment:
+      - EMQX_NODE__NAME=emqx@127.0.0.1
+      - EMQX_NODE__COOKIE=aBcDeFgHiJkLmNoPqRsTuVwXyZ012345
+      - EMQX_NODE__DATA_DIR=/opt/emqx/data
+      - EMQX_LOG__CONSOLE__LEVEL=info
+      - EMQX_LOG__CONSOLE__ENABLE=true
+      - EMQX_LOG__FILE__PATH=/opt/emqx/log/emqx.log
+      - EMQX_LOG__FILE__LEVEL=info
+      - EMQX_LOG__FILE__ENABLE=true
+      - EMQX_LISTENERS__TCP__DEFAULT__BIND=0.0.0.0:1883
+      - EMQX_LISTENERS__TCP__DEFAULT__MAX_CONNECTIONS=1000000
+      - EMQX_LISTENERS__TCP__DEFAULT__ENABLE=true
+      - EMQX_LISTENERS__SSL__DEFAULT__BIND=0.0.0.0:8883
+      - EMQX_LISTENERS__SSL__DEFAULT__ENABLE=false
+      - EMQX_LISTENERS__WS__DEFAULT__BIND=0.0.0.0:8083
+      - EMQX_LISTENERS__WS__DEFAULT__ENABLE=true
+      - EMQX_LISTENERS__WSS__DEFAULT__BIND=0.0.0.0:8084
+      - EMQX_LISTENERS__WSS__DEFAULT__ENABLE=false
+      - EMQX_DASHBOARD__LISTENERS__HTTP__BIND=0.0.0.0:18083
+      - EMQX_DASHBOARD__LISTENERS__HTTP__ENABLE=true
+      - EMQX_MANAGEMENT__LISTENERS__HTTP__BIND=0.0.0.0:8081
+      - EMQX_MANAGEMENT__LISTENERS__HTTP__ENABLE=true
+      - EMQX_AUTHENTICATION__1__ENABLE=true
+      - EMQX_AUTHENTICATION__1__MECHANISM=password_based
+      - EMQX_AUTHENTICATION__1__BACKEND=built_in_database
+      - EMQX_AUTHENTICATION__1__USER_ID_TYPE=username
+      - EMQX_AUTHORIZATION__SOURCES__1__TYPE=built_in_database
+      - EMQX_AUTHORIZATION__SOURCES__1__ENABLE=true
+    ports:
+      - "1883:1883"    # MQTT TCP
+      - "8883:8883"    # MQTT SSL
+      - "8083:8083"    # MQTT WebSocket
+      - "8084:8084"    # MQTT WebSocket SSL
+      - "18083:18083"  # Web 管理控制台
+      - "8081:8081"    # HTTP API
+    volumes:
+      - ./data:/opt/emqx/data
+      - ./log:/opt/emqx/log
+      - ./config:/opt/emqx/etc
+    networks:
+      - mqtt-net
+    healthcheck:
+      test: [ "CMD", "/opt/emqx/bin/emqx_ctl", "status" ]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+    logging:
+      driver: "json-file"
+      options:
+        max-size: "100m"
+        max-file: "3"
+
+networks:
+  mqtt-net:
+    driver: bridge

.docker/mqtt/docker-compose.yml

@@ -0,0 +1,29 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  emqx:
+    image: emqx/emqx:latest
+    container_name: emqx
+    ports:
+      - "1883:1883"
+      - "8083:8083"
+      - "8084:8084"
+      - "8883:8883"
+      - "18083:18083"
+    restart: unless-stopped
+
+networks:
+  default:
+    driver: bridge

.docker/nginx/nginx.conf

@@ -0,0 +1,93 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+worker_processes auto;
+pid /var/run/nginx.pid;
+
+events {
+    worker_connections 1024;
+}
+
+http {
+    include /etc/nginx/mime.types;
+    default_type application/octet-stream;
+
+    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
+                    '$status $body_bytes_sent "$http_referer" '
+                    '"$http_user_agent" "$http_x_forwarded_for"';
+
+    access_log /var/log/nginx/access.log main;
+    error_log /var/log/nginx/error.log warn;
+
+    sendfile on;
+    keepalive_timeout 65;
+
+    # RustFS Server Block
+    server {
+        listen 80;
+        server_name localhost;
+
+        # Redirect HTTP to HTTPS (optional, uncomment if SSL is configured)
+        # return 301 https://$host$request_uri;
+
+        location / {
+            proxy_pass http://rustfs:9000;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+
+            # S3 specific headers
+            proxy_set_header X-Amz-Date $http_x_amz_date;
+            proxy_set_header Authorization $http_authorization;
+
+            # Disable buffering for large uploads
+            proxy_request_buffering off;
+            client_max_body_size 0;
+        }
+
+        location /rustfs/console {
+            proxy_pass http://rustfs:9001;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+        }
+    }
+
+    # SSL Configuration (Example)
+    # server {
+    #     listen 443 ssl;
+    #     server_name localhost;
+    #
+    #     ssl_certificate /etc/nginx/ssl/server.crt;
+    #     ssl_certificate_key /etc/nginx/ssl/server.key;
+    #
+    #     # Restrict to modern TLS versions and ciphers. Operators copying this
+    #     # example must keep at least these directives — without them, nginx
+    #     # may negotiate older protocol versions that have known weaknesses.
+    #     ssl_protocols TLSv1.2 TLSv1.3;
+    #     ssl_prefer_server_ciphers on;
+    #     ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305';
+    #     ssl_session_timeout 1d;
+    #     ssl_session_cache shared:SSL:10m;
+    #     ssl_session_tickets off;
+    #     # add_header Strict-Transport-Security "max-age=63072000" always;
+    #
+    #     location / {
+    #         proxy_pass http://rustfs:9000;
+    #         ...
+    #     }
+    # }
+}

.docker/observability/.gitignore

@@ -0,0 +1,5 @@
+jaeger-data/*
+loki-data/*
+prometheus-data/*
+tempo-data/*
+grafana-data/*

.docker/observability/README.md

@@ -0,0 +1,97 @@
+# RustFS Observability Stack
+
+This directory contains the comprehensive observability stack for RustFS, designed to provide deep insights into application performance, logs, and traces.
+
+## Components
+
+The stack is composed of the following best-in-class open-source components:
+
+- **Prometheus** (v2.53.1): The industry standard for metric collection and alerting.
+- **Grafana** (v11.1.0): The leading platform for observability visualization.
+- **Loki** (v3.1.0): A horizontally-scalable, highly-available, multi-tenant log aggregation system.
+- **Tempo** (v2.5.0): A high-volume, minimal dependency distributed tracing backend.
+- **Jaeger** (v1.59.0): Distributed tracing system (configured as a secondary UI/storage).
+- **OpenTelemetry Collector** (v0.104.0): A vendor-agnostic implementation for receiving, processing, and exporting telemetry data.
+
+By default, this stack uses Tempo in single-binary mode and does not require Kafka/Redpanda.
+If you want the Kafka-backed HA Tempo path, use `docker-compose-example-for-rustfs.yml` together with `docker-compose-tempo-ha-override.yml`.
+
+## Architecture
+
+1. **Telemetry Collection**: Applications send OTLP (OpenTelemetry Protocol) data (Metrics, Logs, Traces) to the **OpenTelemetry Collector**.
+2. **Processing & Exporting**: The Collector processes the data (batching, memory limiting) and exports it to the respective backends:
+    - **Traces** -> **Tempo** (Primary) & **Jaeger** (Secondary/Optional)
+    - **Metrics** -> **Prometheus** (via scraping the Collector's exporter)
+    - **Logs** -> **Loki**
+3. **Visualization**: **Grafana** connects to all backends (Prometheus, Tempo, Loki, Jaeger) to provide a unified dashboard experience.
+
+## Features
+
+- **Full Persistence**: All data (Metrics, Logs, Traces) is persisted to Docker volumes, ensuring no data loss on restart.
+- **Correlation**: Seamless navigation between Metrics, Logs, and Traces in Grafana.
+  - Jump from a Metric spike to relevant Traces.
+  - Jump from a Trace to relevant Logs.
+- **High Performance**: Optimized configurations for batching, compression, and memory management.
+- **Standardized Protocols**: Built entirely on OpenTelemetry standards.
+
+## Quick Start
+
+### Prerequisites
+
+- Docker
+- Docker Compose
+
+### Deploy
+
+Run the following command to start the entire stack:
+
+```bash
+docker compose up -d
+```
+
+### High Availability Tempo
+
+The default `docker-compose.yml` is the single-node stack.
+If you need the Kafka-backed HA Tempo configuration, start it with:
+
+```bash
+docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
+```
+
+### Access Dashboards
+
+| Service        | URL                                              | Credentials       | Description                    |
+| :------------- | :----------------------------------------------- | :---------------- | :----------------------------- |
+| **Grafana**    | [http://localhost:3000](http://localhost:3000)   | `admin` / `admin` | Main visualization hub.        |
+| **Prometheus** | [http://localhost:9090](http://localhost:9090)   | -                 | Metric queries and status.     |
+| **Jaeger UI**  | [http://localhost:16686](http://localhost:16686) | -                 | Secondary trace visualization. |
+| **Tempo**      | [http://localhost:3200](http://localhost:3200)   | -                 | Tempo status/metrics.          |
+
+## Configuration
+
+### Data Persistence
+
+Data is stored in the following Docker volumes:
+
+- `prometheus-data`: Prometheus metrics
+- `tempo-data`: Tempo traces (WAL and Blocks)
+- `loki-data`: Loki logs (Chunks and Rules)
+- `jaeger-data`: Jaeger traces (Badger DB)
+
+To clear all data:
+
+```bash
+docker compose down -v
+```
+
+### Customization
+
+- **Prometheus**: Edit `prometheus.yml` to add scrape targets or alerting rules.
+- **Grafana**: Dashboards and datasources are provisioned from the `grafana/` directory.
+- **Collector**: Edit `otel-collector-config.yaml` to modify pipelines, processors, or exporters.
+
+## Troubleshooting
+
+- **Service Health**: Check the health of services using `docker compose ps`.
+- **Logs**: View logs for a specific service using `docker compose logs -f <service_name>`.
+- **Otel Collector**: Check `http://localhost:13133` for health status and `http://localhost:1888/debug/pprof/` for profiling.

.docker/observability/README_ZH.md

@@ -0,0 +1,97 @@
+# RustFS 可观测性技术栈
+
+本目录包含 RustFS 的全面可观测性技术栈，旨在提供对应用程序性能、日志和追踪的深入洞察。
+
+## 组件
+
+该技术栈由以下一流的开源组件组成：
+
+- **Prometheus** (v2.53.1): 行业标准的指标收集和告警工具。
+- **Grafana** (v11.1.0): 领先的可观测性可视化平台。
+- **Loki** (v3.1.0): 水平可扩展、高可用、多租户的日志聚合系统。
+- **Tempo** (v2.5.0): 高吞吐量、最小依赖的分布式追踪后端。
+- **Jaeger** (v1.59.0): 分布式追踪系统（配置为辅助 UI/存储）。
+- **OpenTelemetry Collector** (v0.104.0): 接收、处理和导出遥测数据的供应商无关实现。
+
+默认情况下，这套技术栈使用 Tempo 单二进制模式，不依赖 Kafka/Redpanda。
+如果需要基于 Kafka 的 HA Tempo 路径，请使用 `docker-compose-example-for-rustfs.yml` 配合 `docker-compose-tempo-ha-override.yml`。
+
+## 架构
+
+1.  **遥测收集**: 应用程序将 OTLP (OpenTelemetry Protocol) 数据（指标、日志、追踪）发送到 **OpenTelemetry Collector**。
+2.  **处理与导出**: Collector 处理数据（批处理、内存限制）并将其导出到相应的后端：
+    -   **追踪** -> **Tempo** (主要) & **Jaeger** (辅助/可选)
+    -   **指标** -> **Prometheus** (通过抓取 Collector 的导出器)
+    -   **日志** -> **Loki**
+3.  **可视化**: **Grafana** 连接到所有后端（Prometheus, Tempo, Loki, Jaeger），提供统一的仪表盘体验。
+
+## 特性
+
+-   **完全持久化**: 所有数据（指标、日志、追踪）都持久化到 Docker 卷，确保重启后无数据丢失。
+-   **关联性**: 在 Grafana 中实现指标、日志和追踪之间的无缝导航。
+    -   从指标峰值跳转到相关追踪。
+    -   从追踪跳转到相关日志。
+-   **高性能**: 针对批处理、压缩和内存管理进行了优化配置。
+-   **标准化协议**: 完全基于 OpenTelemetry 标准构建。
+
+## 快速开始
+
+### 前置条件
+
+-   Docker
+-   Docker Compose
+
+### 部署
+
+运行以下命令启动整个技术栈：
+
+```bash
+docker compose up -d
+```
+
+### Tempo 高可用模式
+
+默认的 `docker-compose.yml` 对应单机栈。
+如果需要基于 Kafka 的 HA Tempo 配置，请使用：
+
+```bash
+docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
+```
+
+### 访问仪表盘
+
+| 服务 | URL | 凭据 | 描述 |
+| :--- | :--- | :--- | :--- |
+| **Grafana** | [http://localhost:3000](http://localhost:3000) | `admin` / `admin` | 主要可视化中心。 |
+| **Prometheus** | [http://localhost:9090](http://localhost:9090) | - | 指标查询和状态。 |
+| **Jaeger UI** | [http://localhost:16686](http://localhost:16686) | - | 辅助追踪可视化。 |
+| **Tempo** | [http://localhost:3200](http://localhost:3200) | - | Tempo 状态/指标。 |
+
+## 配置
+
+### 数据持久化
+
+数据存储在以下 Docker 卷中：
+
+-   `prometheus-data`: Prometheus 指标
+-   `tempo-data`: Tempo 追踪 (WAL 和 Blocks)
+-   `loki-data`: Loki 日志 (Chunks 和 Rules)
+-   `jaeger-data`: Jaeger 追踪 (Badger DB)
+
+要清除所有数据：
+
+```bash
+docker compose down -v
+```
+
+### 自定义
+
+-   **Prometheus**: 编辑 `prometheus.yml` 以添加抓取目标或告警规则。
+-   **Grafana**: 仪表盘和数据源从 `grafana/` 目录预置。
+-   **Collector**: 编辑 `otel-collector-config.yaml` 以修改管道、处理器或导出器。
+
+## 故障排除
+
+-   **服务健康**: 使用 `docker compose ps` 检查服务健康状况。
+-   **日志**: 使用 `docker compose logs -f <service_name>` 查看特定服务的日志。
+-   **Otel Collector**: 检查 `http://localhost:13133` 获取健康状态，检查 `http://localhost:1888/debug/pprof/` 进行性能分析。

.docker/observability/docker-compose-example-for-rustfs.yml

@@ -0,0 +1,268 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  rustfs:
+    security_opt:
+      - "no-new-privileges:true"
+    image: rustfs/rustfs:latest
+    container_name: rustfs-server
+    ports:
+      - "9000:9000" # S3 API port
+      - "9001:9001" # Console port
+    environment:
+      - RUSTFS_VOLUMES=/data/rustfs
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
+      - RUSTFS_OBS_LOGGER_LEVEL=info
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
+    volumes:
+      - rustfs-data:/data/rustfs
+    networks:
+      - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test:
+        [
+          "CMD",
+          "sh",
+          "-c",
+          "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health",
+        ]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 40s
+    depends_on:
+      otel-collector:
+        condition: service_started
+
+  rustfs-init:
+    image: alpine
+    container_name: rustfs-init
+    volumes:
+      - rustfs-data:/data
+    networks:
+      - otel-network
+    command: >
+      sh -c "
+        chown -R 10001:10001 /data &&
+        echo 'Volume Permissions fixed' &&
+        exit 0
+      "
+    restart: no
+
+  # --- Tracing ---
+
+  tempo:
+    image: grafana/tempo:latest
+    container_name: tempo
+    command: [ "-config.file=/etc/tempo.yaml" ]
+    volumes:
+      - ./tempo.yaml:/etc/tempo.yaml:ro
+      - tempo-data:/var/tempo
+    ports:
+      - "3200:3200" # tempo
+      - "4317" # otlp grpc
+      - "4318" # otlp http
+    networks:
+      - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "/tempo", "-version" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
+
+  redpanda:
+    image: redpandadata/redpanda:latest # for tempo ingest
+    container_name: redpanda
+    ports:
+      - "9092:9092"
+    networks:
+      - otel-network
+    restart: unless-stopped
+    command: >
+      redpanda start --overprovisioned
+      --mode=dev-container
+      --kafka-addr=PLAINTEXT://0.0.0.0:9092
+      --advertise-kafka-addr=PLAINTEXT://redpanda:9092
+
+  jaeger:
+    image: jaegertracing/jaeger:latest
+    container_name: jaeger
+    environment:
+      - SPAN_STORAGE_TYPE=badger
+      - BADGER_EPHEMERAL=false
+      - BADGER_DIRECTORY_VALUE=/badger/data
+      - BADGER_DIRECTORY_KEY=/badger/key
+      - COLLECTOR_OTLP_ENABLED=true
+    volumes:
+      - ./jaeger.yaml:/etc/jaeger/config.yml
+      - jaeger-data:/badger
+    ports:
+      - "16686:16686" # Web UI
+      - "14269:14269" # Admin/Metrics
+      - "4317" # otlp grpc
+      - "4318" # otlp http
+    command: [ "--config", "/etc/jaeger/config.yml" ]
+    networks:
+      - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
+
+  # --- Metrics ---
+
+  prometheus:
+    image: prom/prometheus:latest
+    container_name: prometheus
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - prometheus-data:/prometheus
+    ports:
+      - "9090:9090"
+    command:
+      - "--config.file=/etc/prometheus/prometheus.yml"
+      - "--web.enable-otlp-receiver" # Enable OTLP
+      - "--web.enable-remote-write-receiver" # Enable remote write
+      - "--enable-feature=promql-experimental-functions" # Enable info()
+      - "--storage.tsdb.retention.time=30d"
+    restart: unless-stopped
+    networks:
+      - otel-network
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+  # --- Logging ---
+
+  loki:
+    image: grafana/loki:latest
+    container_name: loki
+    volumes:
+      - ./loki.yaml:/etc/loki/loki.yaml:ro
+      - loki-data:/loki
+    ports:
+      - "3100:3100"
+    command: -config.file=/etc/loki/loki.yaml
+    networks:
+      - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
+      interval: 15s
+      timeout: 10s
+      retries: 5
+      start_period: 60s
+
+  # --- Collection ---
+
+  otel-collector:
+    image: otel/opentelemetry-collector-contrib:latest
+    volumes:
+      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
+    ports:
+      - "1888:1888" # pprof
+      - "8888:8888" # Prometheus metrics for Collector
+      - "8889:8889" # Prometheus metrics for application indicators
+      - "13133:13133" # health check
+      - "4317:4317" # OTLP gRPC
+      - "4318:4318" # OTLP HTTP
+      - "55679:55679" # zpages
+    networks:
+      - otel-network
+    restart: unless-stopped
+    depends_on:
+      - tempo
+      - jaeger
+      - prometheus
+      - loki
+    healthcheck:
+      test: [ "CMD", "/otelcol-contrib", "--version" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+  # --- Profiles ---
+
+  pyroscope:
+    image: grafana/pyroscope:latest
+    container_name: pyroscope
+    ports:
+      - "4040:4040"
+    command:
+      - -self-profiling.disable-push=true
+    networks:
+      - otel-network
+    restart: unless-stopped
+
+  # --- Visualization ---
+
+  grafana:
+    image: grafana/grafana:latest
+    container_name: grafana
+    ports:
+      - "3000:3000"
+    environment:
+      - GF_SECURITY_ADMIN_PASSWORD=admin
+      - GF_SECURITY_ADMIN_USER=admin
+    volumes:
+      - ./grafana/provisioning:/etc/grafana/provisioning:ro
+      - ./grafana/dashboards:/etc/grafana/dashboards:ro
+      - grafana-data:/var/lib/grafana
+    networks:
+      - otel-network
+    restart: unless-stopped
+    depends_on:
+      - prometheus
+      - tempo
+      - loki
+    healthcheck:
+      test:
+        [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+volumes:
+  rustfs-data:
+  tempo-data:
+  jaeger-data:
+  prometheus-data:
+  loki-data:
+  grafana-data:
+
+networks:
+  otel-network:
+    driver: bridge
+    name: "network_otel"
+    ipam:
+      config:
+        - subnet: 172.28.0.0/16
+    driver_opts:
+      com.docker.network.enable_ipv6: "true"

.docker/observability/docker-compose-tempo-ha-override.yml

@@ -0,0 +1,62 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Docker Compose override file for High Availability Tempo setup
+#
+# Usage:
+# docker-compose -f docker-compose-example-for-rustfs.yml \
+#                -f docker-compose-tempo-ha-override.yml up
+
+services:
+  # Override Tempo to use high-availability configuration
+  tempo:
+    volumes:
+      - ./tempo-ha.yaml:/etc/tempo.yaml:ro
+      - tempo-data:/var/tempo
+    ports:
+      - "3200:3200"   # Tempo HTTP
+      - "4317:4317"   # OTLP gRPC
+      - "4318:4318"   # OTLP HTTP
+      - "7946:7946"   # Memberlist
+      - "14250:14250" # Jaeger gRPC
+      - "14268:14268" # Jaeger Thrift HTTP
+      - "9411:9411"   # Zipkin
+    environment:
+      - TEMPO_MEMBERLIST_BIND_PORT=7946
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 30s
+    depends_on:
+      - redpanda
+
+volumes:
+  tempo-data:
+    driver: local
+    driver_opts:
+      type: tmpfs
+      device: tmpfs
+      o: "size=4g"  # Allocate 4GB tmpfs for Tempo data (adjust based on your needs)
+
+# Network configuration remains the same
+# networks:
+#   otel-network:
+#     driver: bridge
+#     name: "network_otel"
+#     ipam:
+#       config:
+#         - subnet: 172.28.0.0/16
+

.docker/observability/docker-compose.yml

@@ -0,0 +1,211 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+
+  # --- Tracing ---
+  tempo:
+    image: grafana/tempo:2.10.5
+    container_name: tempo
+    command: [ "-config.file=/etc/tempo.yaml" ]
+    volumes:
+      - ./tempo.yaml:/etc/tempo.yaml:ro
+      - tempo-data:/var/tempo
+    ports:
+      - "3200:3200" # tempo
+      - "4317" # otlp grpc
+      - "4318" # otlp http
+      - "7946" # memberlist
+    networks:
+      - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "/tempo", "-version" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
+
+  vulture:
+    image: grafana/tempo-vulture:latest
+    restart: always
+    command:
+      [
+        "-prometheus-listen-address=:8080",
+        "-tempo-query-url=http://tempo:3200",
+        "-tempo-push-url=http://tempo:4317",
+      ]
+    depends_on:
+      - tempo
+
+  jaeger:
+    image: jaegertracing/jaeger:latest
+    container_name: jaeger
+    environment:
+      - SPAN_STORAGE_TYPE=badger
+      - BADGER_EPHEMERAL=false
+      - BADGER_DIRECTORY_VALUE=/badger/data
+      - BADGER_DIRECTORY_KEY=/badger/key
+      - COLLECTOR_OTLP_ENABLED=true
+    volumes:
+      - ./jaeger.yaml:/etc/jaeger/config.yml
+      - jaeger-data:/badger
+    ports:
+      - "16686:16686" # Web UI
+      - "14269:14269" # Admin/Metrics
+      - "4317" # otlp grpc
+      - "4318" # otlp http
+    command: [ "--config", "/etc/jaeger/config.yml" ]
+    networks:
+      - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s
+
+  # --- Metrics ---
+
+  prometheus:
+    image: prom/prometheus:latest
+    container_name: prometheus
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - ./prometheus-rules:/etc/prometheus/rules:ro
+      - prometheus-data:/prometheus
+    ports:
+      - "9090:9090"
+    command:
+      - "--config.file=/etc/prometheus/prometheus.yml"
+      - "--web.enable-otlp-receiver" # Enable OTLP
+      - "--web.enable-remote-write-receiver" # Enable remote write
+      - "--enable-feature=promql-experimental-functions" # Enable info()
+      - "--storage.tsdb.retention.time=30d"
+    restart: unless-stopped
+    networks:
+      - otel-network
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+  # --- Logging ---
+
+  loki:
+    image: grafana/loki:latest
+    container_name: loki
+    volumes:
+      - ./loki.yaml:/etc/loki/loki.yaml:ro
+      - loki-data:/loki
+    ports:
+      - "3100:3100"
+    command: -config.file=/etc/loki/loki.yaml
+    networks:
+      - otel-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
+      interval: 15s
+      timeout: 10s
+      retries: 5
+      start_period: 60s
+
+  # --- Collection ---
+
+  otel-collector:
+    image: otel/opentelemetry-collector-contrib:latest
+    volumes:
+      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
+    ports:
+      - "1888:1888" # pprof
+      - "8888:8888" # Prometheus metrics for Collector
+      - "8889:8889" # Prometheus metrics for application indicators
+      - "13133:13133" # health check
+      - "4317:4317" # OTLP gRPC
+      - "4318:4318" # OTLP HTTP
+      - "55679:55679" # zpages
+    networks:
+      - otel-network
+    restart: unless-stopped
+    depends_on:
+      - tempo
+      - jaeger
+      - prometheus
+      - loki
+    healthcheck:
+      test: [ "CMD", "/otelcol-contrib", "--version" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+  # --- Profiles ---
+
+  pyroscope:
+    image: grafana/pyroscope:latest
+    container_name: pyroscope
+    ports:
+      - "4040:4040"
+    command:
+      - -self-profiling.disable-push=true
+    networks:
+      - otel-network
+    restart: unless-stopped
+
+  # --- Visualization ---
+
+  grafana:
+    image: grafana/grafana:latest
+    container_name: grafana
+    ports:
+      - "3000:3000"
+    environment:
+      - GF_SECURITY_ADMIN_PASSWORD=admin
+      - GF_SECURITY_ADMIN_USER=admin
+    volumes:
+      - ./grafana/provisioning:/etc/grafana/provisioning:ro
+      - ./grafana/dashboards:/etc/grafana/dashboards:ro
+      - grafana-data:/var/lib/grafana
+    networks:
+      - otel-network
+    restart: unless-stopped
+    depends_on:
+      - prometheus
+      - tempo
+      - loki
+    healthcheck:
+      test:
+        [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+volumes:
+  tempo-data:
+  jaeger-data:
+  prometheus-data:
+  loki-data:
+  grafana-data:
+
+networks:
+  otel-network:
+    driver: bridge
+    name: "network_otel"
+    ipam:
+      config:
+        - subnet: 172.28.0.0/16
+    driver_opts:
+      com.docker.network.enable_ipv6: "true"

.docker/observability/grafana/provisioning/dashboards/dashboard.yml

@@ -0,0 +1,25 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: 1
+
+providers:
+  - name: "default"
+    orgId: 1
+    folder: ""
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 10
+    options:
+      path: /etc/grafana/dashboards

.docker/observability/grafana/provisioning/datasources/datasources.yaml

@@ -0,0 +1,98 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    type: prometheus
+    uid: prometheus
+    url: http://prometheus:9090
+    access: proxy
+    isDefault: true
+    editable: false
+    jsonData:
+      httpMethod: GET
+      exemplarTraceIdDestinations:
+        - name: trace_id
+          datasourceUid: tempo
+
+  - name: Tempo
+    type: tempo
+    uid: tempo
+    access: proxy
+    url: http://tempo:3200
+    isDefault: false
+    editable: false
+    jsonData:
+      httpMethod: GET
+      serviceMap:
+        datasourceUid: prometheus
+      tracesToLogs:
+        datasourceUid: loki
+        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
+        mappedTags: [ { key: 'service.name', value: 'app' } ]
+        spanStartTimeShift: '-1h'
+        spanEndTimeShift: '1h'
+        filterByTraceID: true
+        filterBySpanID: false
+      tracesToMetrics:
+        datasourceUid: prometheus
+        tags: [ { key: 'service.name' }, { key: 'job' } ]
+        queries:
+          - name: 'Service-Level Latency'
+            query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m])) by (le)'
+          - name: 'Service-Level Calls'
+            query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))'
+          - name: 'Service-Level Errors'
+            query: 'sum(rate(traces_spanmetrics_calls_total{status_code="ERROR", $$__tags}[5m]))'
+      nodeGraph:
+        enabled: true
+
+  - name: Loki
+    type: loki
+    uid: loki
+    url: http://loki:3100
+    basicAuth: false
+    isDefault: false
+    editable: false
+    jsonData:
+      derivedFields:
+        - datasourceUid: tempo
+          matcherRegex: 'trace_id=(\w+)'
+          name: 'TraceID'
+          url: '$${__value.raw}'
+
+  - name: Jaeger
+    type: jaeger
+    uid: jaeger
+    url: http://jaeger:16686
+    access: proxy
+    isDefault: false
+    editable: false
+    jsonData:
+      tracesToLogs:
+        datasourceUid: loki
+        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
+        mappedTags: [ { key: 'service.name', value: 'app' } ]
+        spanStartTimeShift: '1s'
+        spanEndTimeShift: '-1s'
+        filterByTraceID: true
+        filterBySpanID: false
+
+  - name: Pyroscope
+    type: grafana-pyroscope-datasource
+    url: http://pyroscope:4040
+    jsonData:
+      minStep: '15s'

.docker/observability/jaeger.yaml

@@ -0,0 +1,74 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+service:
+  extensions: [jaeger_storage, jaeger_query]
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [jaeger_storage_exporter, spanmetrics]
+    metrics/spanmetrics:
+      receivers: [spanmetrics]
+      exporters: [prometheus]
+  telemetry:
+    resource:
+      service.name: jaeger
+    metrics:
+      level: detailed
+      readers:
+        - pull:
+            exporter:
+              prometheus:
+                host: 0.0.0.0
+                port: 8888
+    logs:
+      level: DEBUG
+
+extensions:
+  jaeger_query:
+    storage:
+      traces: some_storage
+      metrics: some_metrics_storage
+  jaeger_storage:
+    backends:
+      some_storage:
+        memory:
+          max_traces: 100000
+    metric_backends:
+      some_metrics_storage:
+        prometheus:
+          endpoint: http://prometheus:9090
+          normalize_calls: true
+          normalize_duration: true
+
+connectors:
+  spanmetrics:
+
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: "0.0.0.0:4317"
+      http:
+        endpoint: "0.0.0.0:4318"
+
+processors:
+  batch:
+
+exporters:
+  jaeger_storage_exporter:
+    trace_storage: some_storage
+  prometheus:
+    endpoint: "0.0.0.0:8889"

.docker/observability/loki.yaml

@@ -0,0 +1,63 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+auth_enabled: false
+
+server:
+  http_listen_port: 3100
+  grpc_listen_port: 9095
+  log_level: info
+  grpc_server_max_concurrent_streams: 1000
+
+common:
+  instance_addr: 127.0.0.1
+  path_prefix: /loki
+  storage:
+    filesystem:
+      chunks_directory: /loki/chunks
+      rules_directory: /loki/rules
+  replication_factor: 1
+  ring:
+    kvstore:
+      store: inmemory
+
+query_range:
+  results_cache:
+    cache:
+      embedded_cache:
+        enabled: true
+        max_size_mb: 100
+
+schema_config:
+  configs:
+    - from: 2020-10-24
+      store: tsdb
+      object_store: filesystem
+      schema: v13
+      index:
+        prefix: index_
+        period: 24h
+
+limits_config:
+  reject_old_samples: true
+  reject_old_samples_max_age: 168h
+  allow_structured_metadata: true
+  max_line_size: 256KB
+
+pattern_ingester:
+  enabled: true
+  metric_aggregation:
+    loki_address: localhost:3100
+
+frontend:
+  encoding: protobuf

.docker/observability/otel-collector-config.yaml

@@ -0,0 +1,116 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 0.0.0.0:4317
+      http:
+        endpoint: 0.0.0.0:4318
+
+processors:
+  batch:
+    timeout: 1s
+    send_batch_size: 1024
+  memory_limiter:
+    check_interval: 1s
+    limit_mib: 1024
+    spike_limit_mib: 256
+  transform/logs:
+    log_statements:
+      - context: log
+        statements:
+          - set(attributes["message"], body.string)
+          - set(attributes["log.body"], body.string)
+
+exporters:
+  otlp/tempo:
+    endpoint: "tempo:4317"
+    tls:
+      insecure: true
+    compression: gzip
+    retry_on_failure:
+      enabled: true
+      initial_interval: 1s
+      max_interval: 30s
+      max_elapsed_time: 300s
+    sending_queue:
+      enabled: true
+      num_consumers: 10
+      queue_size: 5000
+
+  otlp/jaeger:
+    endpoint: "jaeger:4317"
+    tls:
+      insecure: true
+    compression: gzip
+    retry_on_failure:
+      enabled: true
+      initial_interval: 1s
+      max_interval: 30s
+      max_elapsed_time: 300s
+    sending_queue:
+      enabled: true
+      num_consumers: 10
+      queue_size: 5000
+
+  prometheus:
+    endpoint: "0.0.0.0:8889"
+    send_timestamps: true
+    metric_expiration: 5m
+    resource_to_telemetry_conversion:
+      enabled: true
+
+  otlphttp/loki:
+    endpoint: "http://loki:3100/otlp"
+    tls:
+      insecure: true
+    compression: gzip
+
+extensions:
+  health_check:
+    endpoint: 0.0.0.0:13133
+  pprof:
+    endpoint: 0.0.0.0:1888
+  zpages:
+    endpoint: 0.0.0.0:55679
+
+service:
+  extensions: [ health_check, pprof, zpages ]
+  pipelines:
+    traces:
+      receivers: [ otlp ]
+      processors: [ memory_limiter, batch ]
+      exporters: [ otlp/tempo, otlp/jaeger ]
+    metrics:
+      receivers: [ otlp ]
+      processors: [ batch ]
+      exporters: [ prometheus ]
+    logs:
+      receivers: [ otlp ]
+      processors: [ batch, transform/logs ]
+      exporters: [ otlphttp/loki ]
+  telemetry:
+    logs:
+      level: "info"
+      encoding: "json"
+    metrics:
+      level: "normal"
+      readers:
+        - pull:
+            exporter:
+              prometheus:
+                host: '0.0.0.0'
+                port: 8888

.docker/observability/prometheus-rules/rustfs-dashboard.yml

@@ -0,0 +1,40 @@
+groups:
+  - name: rustfs-dashboard
+    interval: 30s
+    rules:
+      - record: rustfs:http_server_requests:rate5m
+        expr: sum by (job) (rate(rustfs_http_server_requests_total[5m]))
+
+      - record: rustfs:http_server_request_duration_seconds:p50_5m
+        expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
+      - record: rustfs:http_server_request_duration_seconds:p95_5m
+        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
+      - record: rustfs:http_server_request_duration_seconds:p99_5m
+        expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
+
+      - record: rustfs:http_server_response_body_size_bytes:p50_5m
+        expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
+      - record: rustfs:http_server_response_body_size_bytes:p95_5m
+        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
+      - record: rustfs:http_server_response_body_size_bytes:p99_5m
+        expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
+
+      - record: rustfs:log_cleaner_runs:rate15m
+        expr: sum by (job) (rate(rustfs_log_cleaner_runs_total[15m]))
+      - record: rustfs:log_cleaner_failure_ratio:rate5m
+        expr: sum by (job) (rate(rustfs_log_cleaner_run_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_runs_total[5m])), 1e-9)
+      - record: rustfs:log_cleaner_rotation_failure_ratio:rate5m
+        expr: sum by (job) (rate(rustfs_log_cleaner_rotation_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_rotation_total[5m])), 1e-9)
+      - record: rustfs:log_cleaner_rotation_duration_seconds:p95_5m
+        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_rotation_duration_seconds_bucket[5m])))
+      - record: rustfs:log_cleaner_compress_duration_seconds:p95_5m
+        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_compress_duration_seconds_bucket[5m])))
+
+      - record: rustfs:scanner_objects_scanned:rate5m
+        expr: sum by (job) (rate(rustfs_scanner_objects_scanned_total[5m]))
+      - record: rustfs:scanner_directories_scanned:rate5m
+        expr: sum by (job) (rate(rustfs_scanner_directories_scanned_total[5m]))
+      - record: rustfs:scanner_buckets_scanned:rate5m
+        expr: sum by (job) (rate(rustfs_scanner_buckets_scanned_total[5m]))
+      - record: rustfs:scanner_cycles_success:rate5m
+        expr: sum by (job) (rate(rustfs_scanner_cycles_total{result="success"}[5m]))

.docker/observability/prometheus.yml

@@ -0,0 +1,84 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+global:
+  scrape_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
+  evaluation_interval: 15s
+  external_labels:
+    cluster: 'rustfs-dev'  # Label to identify the cluster
+    replica: '1'  # Replica identifier
+
+rule_files:
+  - /etc/prometheus/rules/*.yml
+
+scrape_configs:
+  - job_name: 'otel-collector'
+    static_configs:
+      - targets: [ 'otel-collector:8888' ]  # Scrape metrics from Collector
+    scrape_interval: 10s
+
+  - job_name: 'rustfs-app-metrics'
+    static_configs:
+      - targets: [ 'otel-collector:8889' ]  # Application indicators
+    scrape_interval: 15s
+    metric_relabel_configs:
+      - source_labels: [ __name__ ]
+        regex: 'go_.*'
+        action: drop  # Drop Go runtime metrics if not needed
+
+  - job_name: 'tempo'
+    static_configs:
+      - targets: [ 'tempo:3200' ]  # Scrape metrics from Tempo
+
+  - job_name: 'jaeger'
+    static_configs:
+      - targets: [ 'jaeger:14269' ]  # Jaeger admin port (14269 is standard for admin/metrics)
+
+  - job_name: 'loki'
+    static_configs:
+      - targets: [ 'loki:3100' ]
+
+  - job_name: 'prometheus'
+    static_configs:
+      - targets: [ 'localhost:9090' ]
+
+  - job_name: 'vulture'
+    static_configs:
+      - targets:
+          - 'vulture:8080'
+
+otlp:
+  promote_resource_attributes:
+    - service.instance.id
+    - service.name
+    - service.namespace
+    - cloud.availability_zone
+    - cloud.region
+    - container.name
+    - deployment.environment.name
+    - k8s.cluster.name
+    - k8s.container.name
+    - k8s.cronjob.name
+    - k8s.daemonset.name
+    - k8s.deployment.name
+    - k8s.job.name
+    - k8s.namespace.name
+    - k8s.pod.name
+    - k8s.replicaset.name
+    - k8s.statefulset.name
+  translation_strategy: NoUTF8EscapingWithSuffixes
+
+storage:
+  tsdb:
+    out_of_order_time_window: 30m

.docker/observability/redpanda-console.yaml

@@ -0,0 +1,3 @@
+kafka:
+  brokers:
+    - redpanda:9092

.docker/observability/tempo-ha.yaml

@@ -0,0 +1,286 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# High Availability Tempo Configuration for docker-compose-example-for-rustfs.yml
+# Features:
+# - Distributed architecture with multiple components
+# - Kafka-based ingestion for fault tolerance
+# - Replication factor of 3 for data resilience
+# - Query frontend for load balancing
+# - Metrics generation from traces
+# - WAL for durability
+
+partition_ring_live_store: true
+stream_over_http_enabled: true
+
+server:
+  http_listen_port: 3200
+  http_server_read_timeout: 30s
+  http_server_write_timeout: 30s
+  grpc_server_max_recv_msg_size: 4194304  # 4MB
+  grpc_server_max_send_msg_size: 4194304
+  log_level: info
+  log_format: json
+
+# Memberlist configuration for distributed mode
+memberlist:
+  node_name: tempo
+  bind_port: 7946
+  join_members:
+    - tempo:7946
+  retransmit_factor: 4
+  node_timeout: 15s
+  retransmit_interval: 300ms
+  dead_node_reclaim_time: 30s
+
+# Distributor configuration - receives traces and routes to ingesters
+distributor:
+  ingester_write_path_enabled: true
+  kafka_write_path_enabled: true
+  rate_limit_bytes: 10MB
+  rate_limit_enabled: true
+  receivers:
+    otlp:
+      protocols:
+        grpc:
+          endpoint: "0.0.0.0:4317"
+          max_concurrent_streams: 0
+          max_receive_message_size: 4194304
+        http:
+          endpoint: "0.0.0.0:4318"
+          cors:
+            allowed_origins:
+              - "*"
+            max_age: 86400
+    jaeger:
+      protocols:
+        grpc:
+          endpoint: "0.0.0.0:14250"
+        thrift_http:
+          endpoint: "0.0.0.0:14268"
+    zipkin:
+      endpoint: "0.0.0.0:9411"
+  ring:
+    kvstore:
+      store: memberlist
+    heartbeat_timeout: 5s
+    replication_factor: 3
+    heartbeat_interval: 5s
+
+# Ingester configuration - stores traces and querying
+ingester:
+  lifecycler:
+    address: tempo
+    ring:
+      kvstore:
+        store: memberlist
+      replication_factor: 3
+      max_cache_freshness_per_sec: 10s
+      heartbeat_interval: 5s
+      heartbeat_timeout: 5s
+    num_tokens: 128
+    tokens_file_path: /var/tempo/tokens.json
+    claim_on_rollout: true
+  trace_idle_period: 20s
+  max_block_bytes: 10_000_000
+  max_block_duration: 10m
+  chunk_size_bytes: 1_000_000
+  chunk_encoding: snappy
+  wal:
+    checkpoint_duration: 5s
+    max_wal_blocks: 4
+  metrics:
+    enabled: true
+    level: block
+    target_info_duration: 15m
+
+# WAL configuration for data durability
+wal:
+  checkpoint_duration: 5s
+  flush_on_shutdown: true
+  path: /var/tempo/wal
+
+# Kafka ingestion configuration - for high throughput scenarios
+ingest:
+  enabled: true
+  kafka:
+    brokers: [ redpanda:9092 ]
+    topic: tempo-ingest
+    encoding: protobuf
+    consumer_group: tempo-ingest-consumer
+    session_timeout: 10s
+    rebalance_timeout: 1m
+    partition: auto
+    verbosity: 2
+
+# Query frontend configuration - distributed querying
+query_frontend:
+  compression: gzip
+  downstream_url: http://localhost:3200
+  log_queries_longer_than: 5s
+  cache_uncompressed_bytes: 100MB
+  max_outstanding_requests_per_tenant: 100
+  max_query_length: 48h
+  max_query_lookback: 30d
+  default_result_cache_ttl: 1m
+  result_cache:
+    cache:
+      enable_fifocache: true
+      default_validity: 1m
+  rf1_after: "1999-01-01T00:00:00Z"
+  mcp_server:
+    enabled: true
+
+# Querier configuration - queries traces
+querier:
+  frontend_worker:
+    frontend_address: localhost:3200
+    grpc_client_config:
+      max_recv_msg_size: 104857600
+  max_concurrent_queries: 20
+  max_metric_bytes_per_trace: 1MB
+
+# Query scheduler configuration - for distributed querying
+query_scheduler:
+  use_scheduler_ring: false
+
+# Metrics generator configuration - generates metrics from traces
+metrics_generator:
+  enabled: true
+  registry:
+    enabled: true
+    external_labels:
+      source: tempo
+      cluster: rustfs-docker-ha
+      environment: production
+  storage:
+    path: /var/tempo/generator/wal
+    remote_write:
+      - url: http://prometheus:9090/api/v1/write
+        send_exemplars: true
+        resource_to_telemetry_conversion:
+          enabled: true
+  processor:
+    batch:
+      timeout: 10s
+      send_batch_size: 1024
+    memory_limiter:
+      check_interval: 5s
+      limit_mib: 512
+      spike_limit_mib: 128
+  processors:
+    - span-metrics
+    - local-blocks
+    - service-graphs
+  generate_native_histograms: both
+
+# Backend worker configuration
+backend_worker:
+  backend_scheduler_addr: localhost:3200
+  compaction:
+    block_retention: 24h
+    compacted_block_retention: 1h
+  ring:
+    kvstore:
+      store: memberlist
+
+# Backend scheduler configuration
+backend_scheduler:
+  enabled: true
+  provider:
+    compaction:
+      compaction:
+        block_retention: 24h
+        compacted_block_retention: 1h
+        concurrency: 25
+        v2_out_path: /var/tempo/blocks/compaction
+
+# Storage configuration - local backend with proper retention
+storage:
+  trace:
+    backend: local
+    wal:
+      path: /var/tempo/wal
+      checkpoint_duration: 5s
+      flush_on_shutdown: true
+    local:
+      path: /var/tempo/blocks
+      bloom_filter_false_positive: 0.05
+      bloom_shift: 4
+      index:
+        downsample_bytes: 1000000
+        page_size_bytes: 0
+        cache_size_bytes: 0
+    pool:
+      max_workers: 400
+      queue_depth: 10000
+
+# Compactor configuration - manages block compaction
+compactor:
+  compaction:
+    block_retention: 168h  # 7 days
+    compacted_block_retention: 1h
+    concurrency: 25
+    v2_out_path: /var/tempo/blocks/compaction
+    shard_count: 32
+    max_block_bytes: 107374182400  # 100GB
+    max_compaction_objects: 6000000
+    max_time_per_tenant: 5m
+    block_size_bytes: 107374182400
+  ring:
+    kvstore:
+      store: memberlist
+    heartbeat_interval: 5s
+    heartbeat_timeout: 5s
+
+# Limits configuration - rate limiting and quotas
+limits:
+  max_traces_per_user: 10000
+  max_bytes_per_trace: 10485760  # 10MB
+  max_search_bytes_per_trace: 0
+  forgiving_oversize_traces: true
+  rate_limit_bytes: 10MB
+  rate_limit_enabled: true
+  ingestion_burst_size_bytes: 20MB
+  ingestion_rate_limit_bytes: 10MB
+  max_bytes_per_second: 10485760
+  metrics_generator_max_active_series: 10000
+  metrics_generator_max_churned_series: 10000
+  metrics_generator_forta_out_of_order_ttl: 5m
+
+# Override configuration
+overrides:
+  defaults:
+    metrics_generator:
+      processors:
+        - span-metrics
+        - local-blocks
+        - service-graphs
+      generate_native_histograms: both
+      max_active_series: 10000
+      max_churned_series: 10000
+
+# Usage reporting configuration
+usage_report:
+  reporting_enabled: false
+
+# Tracing configuration for debugging
+tracing:
+  enabled: true
+  jaeger:
+    sampler:
+      name: probabilistic
+      param: 0.1
+    reporter_log_spans: false
+

.docker/observability/tempo.yaml

@@ -0,0 +1,72 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+stream_over_http_enabled: true
+
+server:
+  http_listen_port: 3200
+  log_level: info
+
+memberlist:
+  node_name: tempo
+  bind_port: 7946
+  join_members:
+    - tempo:7946
+
+distributor:
+  receivers:
+    otlp:
+      protocols:
+        grpc:
+          endpoint: "0.0.0.0:4317"
+        http:
+          endpoint: "0.0.0.0:4318"
+
+ingester:
+  max_block_duration: 5m
+
+metrics_generator:
+  registry:
+    external_labels:
+      source: tempo
+      cluster: docker-compose
+  traces_storage:
+    path: /var/tempo/generator/traces
+  storage:
+    path: /var/tempo/generator/wal
+    remote_write:
+      - url: http://prometheus:9090/api/v1/write
+        send_exemplars: true
+
+query_frontend:
+  rf1_after: "1999-01-01T00:00:00Z"
+  mcp_server:
+    enabled: true
+
+storage:
+  trace:
+    backend: local
+    wal:
+      path: /var/tempo/wal # where to store the wal locally
+    local:
+      path: /var/tempo/blocks # where to store the traces locally
+
+overrides:
+  defaults:
+    metrics_generator:
+      processors: [ "span-metrics", "service-graphs", "local-blocks" ]
+      generate_native_histograms: both
+
+usage_report:
+  reporting_enabled: false

.docker/openobserve-otel/README.md

@@ -0,0 +1,61 @@
+# OpenObserve + OpenTelemetry Collector
+
+[![OpenObserve](https://img.shields.io/badge/OpenObserve-OpenSource-blue.svg)](https://openobserve.org)
+[![OpenTelemetry](https://img.shields.io/badge/OpenTelemetry-Collector-green.svg)](https://opentelemetry.io/)
+
+English | [中文](README_ZH.md)
+
+This directory contains the configuration for an **alternative** observability stack using OpenObserve.
+
+## ⚠️ Note
+
+For the **recommended** observability stack (Prometheus, Grafana, Tempo, Loki), please see `../observability/`.
+
+## 🌟 Overview
+
+OpenObserve is a lightweight, all-in-one observability platform that handles logs, metrics, and traces in a single binary. This setup is ideal for:
+- Resource-constrained environments.
+- Quick setup and testing.
+- Users who prefer a unified UI.
+
+## 🚀 Quick Start
+
+### 1. Start Services
+
+```bash
+cd .docker/openobserve-otel
+docker compose up -d
+```
+
+### 2. Access Dashboard
+
+- **URL**: [http://localhost:5080](http://localhost:5080)
+- **Username**: `root@rustfs.com`
+- **Password**: `rustfs123`
+
+## 🛠️ Configuration
+
+### OpenObserve
+
+- **Persistence**: Data is persisted to a Docker volume.
+- **Ports**:
+    - `5080`: HTTP API and UI
+    - `5081`: OTLP gRPC
+
+### OpenTelemetry Collector
+
+- **Receivers**: OTLP (gRPC `4317`, HTTP `4318`)
+- **Exporters**: Sends data to OpenObserve.
+
+## 🔗 Integration
+
+Configure your application to send OTLP data to the collector:
+
+- **Endpoint**: `http://localhost:4318` (HTTP) or `localhost:4317` (gRPC)
+
+Example for RustFS:
+
+```bash
+export RUSTFS_OBS_ENDPOINT=http://localhost:4318
+export RUSTFS_OBS_SERVICE_NAME=rustfs-node-1
+```

.docker/openobserve-otel/README_ZH.md

@@ -0,0 +1,61 @@
+# OpenObserve + OpenTelemetry Collector
+
+[![OpenObserve](https://img.shields.io/badge/OpenObserve-OpenSource-blue.svg)](https://openobserve.org)
+[![OpenTelemetry](https://img.shields.io/badge/OpenTelemetry-Collector-green.svg)](https://opentelemetry.io/)
+
+[English](README.md) | 中文
+
+本目录包含使用 OpenObserve 的**替代**可观测性技术栈配置。
+
+## ⚠️ 注意
+
+对于**推荐**的可观测性技术栈（Prometheus, Grafana, Tempo, Loki），请参阅 `../observability/`。
+
+## 🌟 概览
+
+OpenObserve 是一个轻量级、一体化的可观测性平台，在一个二进制文件中处理日志、指标和追踪。此设置非常适合：
+- 资源受限的环境。
+- 快速设置和测试。
+- 喜欢统一 UI 的用户。
+
+## 🚀 快速开始
+
+### 1. 启动服务
+
+```bash
+cd .docker/openobserve-otel
+docker compose up -d
+```
+
+### 2. 访问仪表盘
+
+- **URL**: [http://localhost:5080](http://localhost:5080)
+- **用户名**: `root@rustfs.com`
+- **密码**: `rustfs123`
+
+## 🛠️ 配置
+
+### OpenObserve
+
+- **持久化**: 数据持久化到 Docker 卷。
+- **端口**:
+    - `5080`: HTTP API 和 UI
+    - `5081`: OTLP gRPC
+
+### OpenTelemetry Collector
+
+- **接收器**: OTLP (gRPC `4317`, HTTP `4318`)
+- **导出器**: 将数据发送到 OpenObserve。
+
+## 🔗 集成
+
+配置您的应用程序将 OTLP 数据发送到收集器：
+
+- **端点**: `http://localhost:4318` (HTTP) 或 `localhost:4317` (gRPC)
+
+RustFS 示例：
+
+```bash
+export RUSTFS_OBS_ENDPOINT=http://localhost:4318
+export RUSTFS_OBS_SERVICE_NAME=rustfs-node-1
+```

.docker/openobserve-otel/docker-compose.yml

@@ -0,0 +1,87 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  openobserve:
+    image: public.ecr.aws/zinclabs/openobserve:latest
+    restart: unless-stopped
+    environment:
+      ZO_ROOT_USER_EMAIL: "root@rustfs.com"
+      ZO_ROOT_USER_PASSWORD: "rustfs123"
+      ZO_TRACING_HEADER_KEY: "Authorization"
+      ZO_TRACING_HEADER_VALUE: "Basic cm9vdEBydXN0ZnMuY29tOmQ4SXlCSEJTUkk3RGVlcEQ="
+      ZO_DATA_DIR: "/data"
+      ZO_MEMORY_CACHE_ENABLED: "true"
+      ZO_MEMORY_CACHE_MAX_SIZE: "256"
+      RUST_LOG: "info"
+      TZ: Asia/Shanghai
+    ports:
+      - "5080:5080"
+      - "5081:5081"
+    volumes:
+      - ./data:/data
+    healthcheck:
+      test: [ "CMD", "curl", "-f", "http://localhost:5080/health" ]
+      start_period: 60s
+      interval: 10s
+      timeout: 5s
+      retries: 6
+    networks:
+      - otel-network
+    deploy:
+      resources:
+        limits:
+          memory: 1024M
+        reservations:
+          memory: 512M
+
+  otel-collector:
+    image: otel/opentelemetry-collector-contrib:latest
+    restart: unless-stopped
+    environment:
+      - TZ=Asia/Shanghai
+    volumes:
+      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
+    ports:
+      - "4317:4317" # OTLP gRPC
+      - "4318:4318" # OTLP HTTP
+      - "13133:13133" # Health check
+      - "1777:1777" # pprof
+      - "55679:55679" # zpages
+      - "1888:1888" # Metrics
+      - "8888:8888" # Prometheus metrics
+      - "8889:8889" # Additional metrics endpoint
+    depends_on:
+      - openobserve
+    networks:
+      - otel-network
+    deploy:
+      resources:
+        limits:
+          memory: 10240M
+        reservations:
+          memory: 512M
+
+networks:
+  otel-network:
+    driver: bridge
+    name: otel-network
+    ipam:
+      config:
+        - subnet: 172.28.0.0/16
+          gateway: 172.28.0.1
+    labels:
+      com.example.description: "Network for OpenObserve and OpenTelemetry Collector"
+volumes:
+  data:

.docker/openobserve-otel/otel-collector-config.yaml

@@ -0,0 +1,92 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 0.0.0.0:4317
+      http:
+        endpoint: 0.0.0.0:4318
+  filelog:
+    include: [ "/var/log/app/*.log" ]
+    start_at: end
+
+processors:
+  batch:
+    timeout: 1s
+    send_batch_size: 1024
+  memory_limiter:
+    check_interval: 1s
+    limit_mib: 400
+    spike_limit_mib: 100
+
+exporters:
+  otlphttp/openobserve:
+    endpoint: http://openobserve:5080/api/default # http://127.0.0.1:5080/api/default
+    headers:
+      Authorization: "Basic cm9vdEBydXN0ZnMuY29tOmQ4SXlCSEJTUkk3RGVlcEQ="
+      stream-name: default
+      organization: default
+    compression: gzip
+    retry_on_failure:
+      enabled: true
+      initial_interval: 5s
+      max_interval: 30s
+      max_elapsed_time: 300s
+    timeout: 10s
+  otlp/openobserve:
+    endpoint: openobserve:5081 # http://127.0.0.1:5080/api/default
+    headers:
+      Authorization: "Basic cm9vdEBydXN0ZnMuY29tOmQ4SXlCSEJTUkk3RGVlcEQ="
+      stream-name: default
+      organization: default
+    compression: gzip
+    retry_on_failure:
+      enabled: true
+      initial_interval: 5s
+      max_interval: 30s
+      max_elapsed_time: 300s
+    timeout: 10s
+    tls:
+      insecure: true
+
+extensions:
+  health_check:
+    endpoint: 0.0.0.0:13133
+  pprof:
+    endpoint: 0.0.0.0:1777
+  zpages:
+    endpoint: 0.0.0.0:55679
+
+service:
+  extensions: [ health_check, pprof, zpages ]
+  pipelines:
+    traces:
+      receivers: [ otlp ]
+      processors: [ memory_limiter, batch ]
+      exporters: [ otlp/openobserve ]
+    metrics:
+      receivers: [ otlp ]
+      processors: [ memory_limiter, batch ]
+      exporters: [ otlp/openobserve ]
+    logs:
+      receivers: [ otlp, filelog ]
+      processors: [ memory_limiter, batch ]
+      exporters: [ otlp/openobserve ]
+  telemetry:
+    logs:
+      level: "info"  # Collector 日志级别
+    metrics:
+      address: "0.0.0.0:8888"  # Collector 自身指标暴露

.docker/test/issue-2715/docker-compose-test.yml

@@ -0,0 +1,52 @@
+services:
+  rustfs:
+    image: rustfs/rustfs:1.0.0-alpha.99-glibc
+    container_name: rustfs-issue-2715-test
+    security_opt:
+      - "no-new-privileges:true"
+    ports:
+      - "19000:9000"
+      - "19001:9001"
+    environment:
+      - RUSTFS_VOLUMES=/data/rustfs{0...8}
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_ACCESS_KEY=admin
+      - RUSTFS_SECRET_KEY=admin
+      - RUSTFS_OBS_LOGGER_LEVEL=info
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
+      - RUSTFS_STORAGE_CLASS_STANDARD=EC:2
+      - RUSTFS_STORAGE_CLASS_RRS=EC:1
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true
+      - RUSTFS_OBS_LOG_DIRECTORY=/opt/rustfs/logs
+    extra_hosts:
+      - "otel-collector:host-gateway"
+      - "pyroscope:host-gateway"
+    volumes:
+      - ./deploy/data/issue-2715/rustfs0:/data/rustfs0
+      - ./deploy/data/issue-2715/rustfs1:/data/rustfs1
+      - ./deploy/data/issue-2715/rustfs2:/data/rustfs2
+      - ./deploy/data/issue-2715/rustfs3:/data/rustfs3
+      - ./deploy/data/issue-2715/rustfs4:/data/rustfs4
+      - ./deploy/data/issue-2715/rustfs5:/data/rustfs5
+      - ./deploy/data/issue-2715/rustfs6:/data/rustfs6
+      - ./deploy/data/issue-2715/rustfs7:/data/rustfs7
+      - ./deploy/data/issue-2715/rustfs8:/data/rustfs8
+      - ./deploy/logs/issue-2715:/opt/rustfs/logs
+    restart: unless-stopped
+    healthcheck:
+      test:
+        [
+          "CMD",
+          "sh",
+          "-c",
+          "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health"
+        ]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 40s

.docker/test/issues-2815/.gitignore

@@ -0,0 +1 @@
+data/

.docker/test/issues-2815/README.md

@@ -0,0 +1,106 @@
+# Issue 2815 Local Docker Verification
+
+## Purpose
+
+This directory contains the local distributed Docker verification assets used to validate issue `#2815` against the current source build.
+
+The target behavior is:
+
+- 4-node distributed cluster starts successfully
+- `/health/ready` becomes reachable on each node
+- logs no longer contain `storage_info failed: Io error: wrong msgpack marker FixArray(1)`
+- internode RPC authentication succeeds with an explicit non-default RPC secret
+
+## Files
+
+- `docker-compose.yml`: 4-node distributed cluster using a locally built image
+
+## Data Directories
+
+Create the bind-mount directories before `docker compose up`:
+
+```bash
+mkdir -p .docker/test/issues-2815/data/rustfs{1..4}-disk{0..3}
+```
+
+## Build
+
+Apple Silicon / arm64 host:
+
+```bash
+docker build --platform linux/arm64 -f Dockerfile.source -t rustfs-issue-2815-local .
+```
+
+If you intentionally want amd64 emulation:
+
+```bash
+docker build --platform linux/amd64 -f Dockerfile.source -t rustfs-issue-2815-local .
+```
+
+## Run
+
+```bash
+docker compose -f .docker/test/issues-2815/docker-compose.yml up -d
+```
+
+If the image platform is not `linux/arm64`, align compose explicitly:
+
+```bash
+RUSTFS_DOCKER_PLATFORM=linux/amd64 docker compose -f .docker/test/issues-2815/docker-compose.yml up -d
+```
+
+## Health Checks
+
+Container-level healthcheck is now included and probes:
+
+```bash
+curl -fsS http://127.0.0.1:9000/health
+```
+
+Manual checks:
+
+```bash
+curl -i http://127.0.0.1:9101/health/ready
+curl -i http://127.0.0.1:9102/health/ready
+curl -i http://127.0.0.1:9103/health/ready
+curl -i http://127.0.0.1:9104/health/ready
+```
+
+## RPC Secret Requirement
+
+The current source build no longer reproduces the original `FixArray(1)` decode error from issue `#2815`.
+
+Earlier local Docker attempts failed during erasure bootstrap with:
+
+```text
+No valid auth token
+store init failed to load formats after 10 retries: erasure read quorum
+```
+
+Root cause:
+
+- RPC authentication rejects the default secret `rustfsadmin`
+- distributed local Docker validation therefore needs an explicit non-default secret
+
+This compose now sets both:
+
+- `RUSTFS_SECRET_KEY=issue-2815-secret`
+- `RUSTFS_RPC_SECRET=issue-2815-rpc-secret`
+
+With those values in place, the current 4-node local Docker cluster reaches healthy state and `/health/ready` returns `200`.
+
+In other words:
+
+- `RUSTFS_ACCESS_KEY` may still be `rustfsadmin` for local service credentials if desired
+- `RUSTFS_SECRET_KEY` can still be used for service credentials
+- but RPC authentication must not resolve to the default secret value `rustfsadmin`
+- if `RUSTFS_RPC_SECRET` is unset, the code falls back to `RUSTFS_SECRET_KEY`
+- so at least one of them must provide a non-default shared secret for internode RPC signing
+
+## Suggested Debug Commands
+
+```bash
+docker compose -f .docker/test/issues-2815/docker-compose.yml ps
+docker compose -f .docker/test/issues-2815/docker-compose.yml logs --no-color --tail=200
+docker compose -f .docker/test/issues-2815/docker-compose.yml down -v
+```

.docker/test/issues-2815/docker-compose.yml

@@ -0,0 +1,120 @@
+services:
+  rustfs1:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs1
+    container_name: rustfs-issue-2815-rustfs1
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs1-disk0:/data/rustfs0
+      - ./data/rustfs1-disk1:/data/rustfs1
+      - ./data/rustfs1-disk2:/data/rustfs2
+      - ./data/rustfs1-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9101:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+  rustfs2:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs2
+    container_name: rustfs-issue-2815-rustfs2
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs2-disk0:/data/rustfs0
+      - ./data/rustfs2-disk1:/data/rustfs1
+      - ./data/rustfs2-disk2:/data/rustfs2
+      - ./data/rustfs2-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9102:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+  rustfs3:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs3
+    container_name: rustfs-issue-2815-rustfs3
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs3-disk0:/data/rustfs0
+      - ./data/rustfs3-disk1:/data/rustfs1
+      - ./data/rustfs3-disk2:/data/rustfs2
+      - ./data/rustfs3-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9103:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+  rustfs4:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs4
+    container_name: rustfs-issue-2815-rustfs4
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs4-disk0:/data/rustfs0
+      - ./data/rustfs4-disk1:/data/rustfs1
+      - ./data/rustfs4-disk2:/data/rustfs2
+      - ./data/rustfs4-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9104:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+networks:
+  rustfs-issue-2815-net:
+    name: rustfs-issue-2815-net

.dockerignore

@@ -0,0 +1 @@
+target

.envrc

@@ -0,0 +1 @@
+use flake

.github/AGENTS.md

@@ -0,0 +1,36 @@
+# GitHub Workflow Instructions
+
+Applies to `.github/` and repository pull-request operations.
+
+## Pull Requests
+
+- PR titles and descriptions must be in English.
+- Use `.github/pull_request_template.md` for every PR body.
+- Keep all template section headings.
+- Use `N/A` for non-applicable sections.
+- Include verification commands in the PR details.
+- For `gh pr create` and `gh pr edit`, always write markdown body to a file and pass `--body-file`.
+- Do not use multiline inline `--body`; backticks and shell expansion can corrupt content or trigger unintended commands.
+- Recommended pattern:
+  - `cat > /tmp/pr_body.md <<'EOF'`
+  - `...markdown...`
+  - `EOF`
+  - `gh pr create ... --body-file /tmp/pr_body.md`
+
+## CI Alignment
+
+When changing CI-sensitive behavior, keep local validation aligned with `.github/workflows/ci.yml`.
+
+Current `test-and-lint` gate includes:
+
+- `cargo nextest run --all --exclude e2e_test`
+- `cargo test --all --doc`
+- `cargo test -p rustfs get_object_chunk_fast_path`
+- `cargo test -p rustfs materialize_chunk_stream_before_commit`
+- `touch rustfs/build.rs`
+- `cargo build -p rustfs --bins --jobs 2`
+- `cargo test -p e2e_test archive_multipart_roundtrip_preserves_bytes`
+- `cargo test -p e2e_test presigned_get_and_reverse_proxy_preserve_multipart_bytes_with_fast_path`
+- `cargo fmt --all --check`
+- `cargo clippy --all-targets --all-features -- -D warnings`
+- `./scripts/check_layer_dependencies.sh`

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,38 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Browser [e.g. chrome, safari]
+ - Version [e.g. 22]
+
+**Smartphone (please complete the following information):**
+ - Device: [e.g. iPhone6]
+ - OS: [e.g. iOS8.1]
+ - Browser [e.g. stock browser, safari]
+ - Version [e.g. 22]
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/actions/setup/action.yml

@@ -0,0 +1,93 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: "Setup Rust Environment"
+description: "Setup Rust development environment with caching for RustFS"
+
+inputs:
+  rust-version:
+    description: "Rust version to install"
+    required: false
+    default: "stable"
+  cache-shared-key:
+    description: "Shared cache key for Rust dependencies"
+    required: false
+    default: "rustfs-deps"
+  cache-save-if:
+    description: "Condition for saving cache"
+    required: false
+    default: "true"
+  install-cross-tools:
+    description: "Install cross-compilation tools"
+    required: false
+    default: "false"
+  target:
+    description: "Target architecture to add"
+    required: false
+    default: ""
+  github-token:
+    description: "GitHub token for API access"
+    required: false
+    default: ""
+
+runs:
+  using: "composite"
+  steps:
+    - name: Install system dependencies (Ubuntu)
+      if: runner.os == 'Linux'
+      shell: bash
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y \
+          musl-tools \
+          build-essential \
+          pkg-config \
+          libssl-dev
+
+    - name: Install protoc
+      uses: arduino/setup-protoc@v3
+      with:
+        version: "33.1"
+        repo-token: ${{ inputs.github-token }}
+
+    - name: Install flatc
+      uses: Nugine/setup-flatc@v1.2.4
+      with:
+        version: "25.9.23"
+
+    - name: Install Rust toolchain
+      uses: dtolnay/rust-toolchain@stable
+      with:
+        toolchain: ${{ inputs.rust-version }}
+        targets: ${{ inputs.target }}
+        components: rustfmt, clippy
+
+    - name: Install Zig
+      if: inputs.install-cross-tools == 'true'
+      uses: mlugg/setup-zig@v2
+
+    - name: Install cargo-zigbuild
+      if: inputs.install-cross-tools == 'true'
+      uses: taiki-e/install-action@cargo-zigbuild
+
+    - name: Install cargo-nextest
+      uses: taiki-e/install-action@cargo-nextest
+
+    - name: Setup Rust cache
+      uses: Swatinem/rust-cache@v2
+      with:
+        cache-all-crates: true
+        cache-on-failure: true
+        shared-key: ${{ inputs.cache-shared-key }}
+        save-if: ${{ inputs.cache-save-if }}

.github/cla.yml

@@ -0,0 +1,17 @@
+enabled: true
+
+document:
+  version: v1
+  url: https://github.com/rustfs/cla/blob/main/cla/v1.md
+
+signing:
+  mode: comment
+  comment_pattern: I have read and agree to the CLA.
+
+registry:
+  type: json-repo
+  repository: rustfs/cla
+  path_prefix: signatures
+
+status:
+  check_name: CLA Check

.github/copilot-instructions.md

@@ -0,0 +1 @@
+../AGENTS.md

.github/dependabot.yml

@@ -0,0 +1,53 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://help.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
+
+version: 2
+updates:
+  - package-ecosystem: "cargo" # See documentation for possible values
+    directory: "/" # Location of package manifests
+    schedule:
+      interval: "weekly"
+      day: "monday"
+      timezone: "Asia/Shanghai"
+      time: "08:00"
+    assignees:
+      - "houseme"
+    reviewers:
+      - "overtrue"
+      - "majinghe"
+    ignore:
+      - dependency-name: "object_store"
+        versions: [ "0.13.x" ]
+      - dependency-name: "libunftp"
+        versions: [ "0.23.x" ]
+      - dependency-name: "ratelimit"
+        versions: [ "1.x" ]
+      - dependency-name: "ratelimit"
+        versions: [ "2.x" ]
+    groups:
+      s3s:
+        update-types:
+          - "minor"
+          - "patch"
+        patterns:
+          - "s3s"
+          - "s3s-*"
+      dependencies:
+        patterns:
+          - "*"

.github/pull_request_template.md

@@ -0,0 +1,36 @@
+<!--
+Pull Request Template for RustFS
+-->
+
+## Related Issues
+<!--
+List related issues, e.g. Fixes #123.
+Use N/A when there is no related issue.
+-->
+
+## Summary of Changes
+<!--
+Briefly explain what changed and why reviewers should accept it.
+Focus on behavior, compatibility, and review-relevant context.
+-->
+
+## Verification
+<!--
+List the commands or checks you ran, for example:
+- `make pre-commit`
+
+Use N/A only when verification is not applicable.
+-->
+
+## Impact
+<!--
+Describe user-facing, compatibility, API, deployment, configuration, or
+documentation impact. Use N/A when there is no expected impact.
+-->
+
+## Additional Notes
+<!-- Any extra information for reviewers, or N/A. -->
+
+---
+
+Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v1.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.

.github/s3tests/README.md

@@ -0,0 +1,103 @@
+# S3 Compatibility Tests Configuration
+
+This directory contains the configuration for running [Ceph S3 compatibility tests](https://github.com/ceph/s3-tests) against RustFS.
+
+## Configuration File
+
+The `s3tests.conf` file is based on the official `s3tests.conf.SAMPLE` from the ceph/s3-tests repository. It uses environment variable substitution via `envsubst` to configure the endpoint and credentials.
+
+### Key Configuration Points
+
+- **Host**: Set via `${S3_HOST}` environment variable (e.g., `rustfs-single` for single-node, `lb` for multi-node)
+- **Port**: 9000 (standard RustFS port)
+- **Credentials**: Uses `${S3_ACCESS_KEY}` and `${S3_SECRET_KEY}` from workflow environment
+- **TLS**: Disabled (`is_secure = False`)
+
+## Test Execution Strategy
+
+### Network Connectivity Fix
+
+Tests run inside a Docker container on the `rustfs-net` network, which allows them to resolve and connect to the RustFS container hostnames. This fixes the "Temporary failure in name resolution" error that occurred when tests ran on the GitHub runner host.
+
+### Performance Optimizations
+
+1. **Parallel Execution**: Uses `pytest-xdist` with `-n 4` to run tests in parallel across 4 workers
+2. **Load Distribution**: Uses `--dist=loadgroup` to distribute test groups across workers
+3. **Fail-Fast**: Uses `--maxfail=50` to stop after 50 failures, saving time on catastrophic failures
+
+### Feature Filtering
+
+Tests are filtered using pytest markers (`-m`) to skip features not yet supported by RustFS:
+
+- `lifecycle` - Bucket lifecycle policies
+- `versioning` - Object versioning
+- `s3website` - Static website hosting
+- `bucket_logging` - Bucket logging
+- `encryption` / `sse_s3` - Server-side encryption
+- `cloud_transition` / `cloud_restore` - Cloud storage transitions
+- `lifecycle_expiration` / `lifecycle_transition` - Lifecycle operations
+
+This filtering:
+1. Reduces test execution time significantly (from 1+ hour to ~10-15 minutes)
+2. Focuses on features RustFS currently supports
+3. Avoids hundreds of expected failures
+
+## Running Tests Locally
+
+### Single-Node Test
+
+```bash
+# Set credentials
+export S3_ACCESS_KEY=rustfsadmin
+export S3_SECRET_KEY=rustfsadmin
+
+# Start RustFS container
+docker run -d --name rustfs-single \
+  --network rustfs-net \
+  -e RUSTFS_ADDRESS=0.0.0.0:9000 \
+  -e RUSTFS_ACCESS_KEY=$S3_ACCESS_KEY \
+  -e RUSTFS_SECRET_KEY=$S3_SECRET_KEY \
+  -e RUSTFS_VOLUMES="/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" \
+  rustfs-ci
+
+# Generate config
+export S3_HOST=rustfs-single
+envsubst < .github/s3tests/s3tests.conf > /tmp/s3tests.conf
+
+# Run tests
+docker run --rm \
+  --network rustfs-net \
+  -v /tmp/s3tests.conf:/etc/s3tests.conf:ro \
+  python:3.12-slim \
+  bash -c '
+    apt-get update -qq && apt-get install -y -qq git
+    git clone --depth 1 https://github.com/ceph/s3-tests.git /s3-tests
+    cd /s3-tests
+    pip install -q -r requirements.txt pytest-xdist
+    S3TEST_CONF=/etc/s3tests.conf pytest -v -n 4 \
+      s3tests/functional/test_s3.py \
+      -m "not lifecycle and not versioning and not s3website and not bucket_logging and not encryption and not sse_s3"
+  '
+```
+
+## Test Results Interpretation
+
+- **PASSED**: Test succeeded, feature works correctly
+- **FAILED**: Test failed, indicates a potential bug or incompatibility
+- **ERROR**: Test setup failed (e.g., network issues, missing dependencies)
+- **SKIPPED**: Test skipped due to marker filtering
+
+## Adding New Feature Support
+
+When adding support for a new S3 feature to RustFS:
+
+1. Remove the corresponding marker from the filter in `.github/workflows/e2e-s3tests.yml`
+2. Run the tests to verify compatibility
+3. Fix any failing tests
+4. Update this README to reflect the newly supported feature
+
+## References
+
+- [Ceph S3 Tests Repository](https://github.com/ceph/s3-tests)
+- [S3 API Compatibility](https://docs.aws.amazon.com/AmazonS3/latest/API/)
+- [pytest-xdist Documentation](https://pytest-xdist.readthedocs.io/)

.github/s3tests/s3tests.conf

@@ -0,0 +1,193 @@
+# RustFS s3-tests configuration
+# Based on: https://github.com/ceph/s3-tests/blob/master/s3tests.conf.SAMPLE
+#
+# Usage:
+#   Single-node: S3_HOST=rustfs-single envsubst < s3tests.conf > /tmp/s3tests.conf
+#   Multi-node:  S3_HOST=lb envsubst < s3tests.conf > /tmp/s3tests.conf
+
+[DEFAULT]
+## this section is just used for host, port and bucket_prefix
+
+# host set for RustFS - will be substituted via envsubst
+host = ${S3_HOST}
+
+# port for RustFS
+port = 9000
+
+## say "False" to disable TLS
+is_secure = False
+
+## say "False" to disable SSL Verify
+ssl_verify = False
+
+[fixtures]
+## all the buckets created will start with this prefix;
+## {random} will be filled with random characters to pad
+## the prefix to 30 characters long, and avoid collisions
+bucket prefix = rustfs-{random}-
+
+# all the iam account resources (users, roles, etc) created
+# will start with this name prefix
+iam name prefix = s3-tests-
+
+# all the iam account resources (users, roles, etc) created
+# will start with this path prefix
+iam path prefix = /s3-tests/
+
+[s3 main]
+# main display_name
+display_name = RustFS Tester
+
+# main user_id
+user_id = rustfsadmin
+
+# main email
+email = tester@rustfs.local
+
+# zonegroup api_name for bucket location
+api_name = default
+
+## main AWS access key
+access_key = ${S3_ACCESS_KEY}
+
+## main AWS secret key
+secret_key = ${S3_SECRET_KEY}
+
+## replace with key id obtained when secret is created, or delete if KMS not tested
+#kms_keyid = 01234567-89ab-cdef-0123-456789abcdef
+
+## Storage classes
+#storage_classes = "LUKEWARM, FROZEN"
+
+## Lifecycle debug interval (default: 10)
+#lc_debug_interval = 20
+## Restore debug interval (default: 100)
+#rgw_restore_debug_interval = 60
+#rgw_restore_processor_period = 60
+
+[s3 alt]
+# alt display_name
+display_name = RustFS Alt Tester
+
+## alt email
+email = alt@rustfs.local
+
+# alt user_id
+user_id = rustfsalt
+
+# alt AWS access key (must be different from s3 main for many tests)
+access_key = ${S3_ALT_ACCESS_KEY}
+
+# alt AWS secret key
+secret_key = ${S3_ALT_SECRET_KEY}
+
+#[s3 cloud]
+## to run the testcases with "cloud_transition" for transition
+## and "cloud_restore" for restore attribute.
+## Note: the waiting time may have to tweaked depending on
+## the I/O latency to the cloud endpoint.
+
+## host set for cloud endpoint
+# host = localhost
+
+## port set for cloud endpoint
+# port = 8001
+
+## say "False" to disable TLS
+# is_secure = False
+
+## cloud endpoint credentials
+# access_key = 0555b35654ad1656d804
+# secret_key = h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==
+
+## storage class configured as cloud tier on local rgw server
+# cloud_storage_class = CLOUDTIER
+
+## Below are optional -
+
+## Above configured cloud storage class config options
+# retain_head_object = false
+# allow_read_through = false    # change it to enable read_through
+# read_through_restore_days = 2
+# target_storage_class = Target_SC
+# target_path = cloud-bucket
+
+## another regular storage class to test multiple transition rules,
+# storage_class = S1
+
+[s3 tenant]
+# tenant display_name
+display_name = RustFS Tenant Tester
+
+# tenant user_id
+# Note: Using same user_id as main to avoid teardown failures.
+# RustFS does not currently support multi-tenancy, so the tenant client
+# effectively operates as the main user. This ensures nuke_prefixed_buckets()
+# in s3-tests teardown can successfully clean up resources.
+user_id = rustfsadmin
+
+# tenant AWS access key
+access_key = ${S3_ACCESS_KEY}
+
+# tenant AWS secret key
+secret_key = ${S3_SECRET_KEY}
+
+# tenant email
+email = tenant@rustfs.local
+
+# tenant name
+# Note: Empty tenant name to avoid multi-tenant path issues during teardown.
+# When s3-tests calls get_tenant_client(), it uses this tenant value in requests.
+# An empty value makes the tenant client behave like the main client, preventing
+# "bucket not found" errors when teardown tries to clean up test buckets.
+tenant =
+
+#following section needs to be added for all sts-tests
+[iam]
+#used for iam operations in sts-tests
+#email
+email = s3@rustfs.local
+
+#user_id
+user_id = rustfsiam
+
+#access_key
+access_key = ${S3_ACCESS_KEY}
+
+#secret_key
+secret_key = ${S3_SECRET_KEY}
+
+#display_name
+display_name = RustFS IAM User
+
+# iam account root user for iam_account tests
+[iam root]
+access_key = ${S3_ACCESS_KEY}
+secret_key = ${S3_SECRET_KEY}
+user_id = RGW11111111111111111
+email = account1@rustfs.local
+
+# iam account root user in a different account than [iam root]
+[iam alt root]
+access_key = ${S3_ACCESS_KEY}
+secret_key = ${S3_SECRET_KEY}
+user_id = RGW22222222222222222
+email = account2@rustfs.local
+
+#following section needs to be added when you want to run Assume Role With Webidentity test
+[webidentity]
+#used for assume role with web identity test in sts-tests
+#all parameters will be obtained from ceph/qa/tasks/keycloak.py
+#token=<access_token>
+
+#aud=<obtained after introspecting token>
+
+#sub=<obtained after introspecting token>
+
+#azp=<obtained after introspecting token>
+
+#user_token=<access token for a user, with attribute Department=[Engineering, Marketing>]
+
+#thumbprint=<obtained from x509 certificate>
+
+#KC_REALM=<name of the realm>

.github/workflows/audit.yml

@@ -0,0 +1,81 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Security Audit
+
+on:
+  push:
+    branches: [ main ]
+    paths:
+      - '**/Cargo.toml'
+      - '**/Cargo.lock'
+      - '.github/workflows/audit.yml'
+  pull_request:
+    branches: [ main ]
+    paths:
+      - '**/Cargo.toml'
+      - '**/Cargo.lock'
+      - '.github/workflows/audit.yml'
+  schedule:
+    - cron: '0 0 * * 0' # Weekly on Sunday at midnight UTC
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+env:
+  CARGO_TERM_COLOR: always
+
+jobs:
+  security-audit:
+    name: Security Audit
+    runs-on: ubicloud-standard-2
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Install cargo-audit
+        uses: taiki-e/install-action@v2
+        with:
+          tool: cargo-audit
+
+      - name: Run security audit
+        run: |
+          cargo audit -D warnings --json | tee audit-results.json
+
+      - name: Upload audit results
+        if: always()
+        uses: actions/upload-artifact@v6
+        with:
+          name: security-audit-results-${{ github.run_number }}
+          path: audit-results.json
+          retention-days: 30
+
+  dependency-review:
+    name: Dependency Review
+    runs-on: ubicloud-standard-2
+    if: github.event_name == 'pull_request'
+    permissions:
+      contents: read
+      pull-requests: write
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Dependency Review
+        uses: actions/dependency-review-action@v4
+        with:
+          fail-on-severity: moderate
+          comment-summary-in-pr: true

.github/workflows/build.yml

@@ -0,0 +1,869 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Build and Release Workflow
+#
+# This workflow builds RustFS binaries and automatically triggers Docker image builds.
+#
+# Flow:
+# 1. Build binaries for multiple platforms
+# 2. Upload binaries to OSS storage
+# 3. Trigger docker.yml to build and push images using the uploaded binaries
+#
+# Manual Parameters:
+# - build_docker: Build and push Docker images (default: true)
+# - platforms: Comma-separated platform IDs or 'all' (default: all)
+
+name: Build and Release
+
+on:
+  push:
+    tags: [ "*.*.*" ]
+    branches: [ main ]
+    paths-ignore:
+      - "**.md"
+      - "**.txt"
+      - ".github/**"
+      - "docs/**"
+      - "deploy/**"
+      - "scripts/dev_*.sh"
+      - "LICENSE*"
+      - "README*"
+      - "**/*.png"
+      - "**/*.jpg"
+      - "**/*.svg"
+      - ".gitignore"
+      - ".dockerignore"
+  schedule:
+    - cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
+  workflow_dispatch:
+    inputs:
+      build_docker:
+        description: "Build and push Docker images after binary build"
+        required: false
+        default: true
+        type: boolean
+      platforms:
+        description: "Comma-separated targets or 'all' (e.g. linux-x86_64-musl,macos-aarch64)"
+        required: false
+        default: "all"
+        type: string
+
+permissions:
+  contents: read
+
+env:
+  CARGO_TERM_COLOR: always
+  RUST_BACKTRACE: 1
+  # Optimize build performance
+  CARGO_INCREMENTAL: 0
+
+jobs:
+  # Build strategy check - determine build type based on trigger
+  build-check:
+    name: Build Strategy Check
+    runs-on: ubicloud-standard-2
+    outputs:
+      should_build: ${{ steps.check.outputs.should_build }}
+      build_type: ${{ steps.check.outputs.build_type }}
+      version: ${{ steps.check.outputs.version }}
+      short_sha: ${{ steps.check.outputs.short_sha }}
+      is_prerelease: ${{ steps.check.outputs.is_prerelease }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - name: Determine build strategy
+        id: check
+        run: |
+          should_build=false
+          build_type="none"
+          version=""
+          short_sha=""
+          is_prerelease=false
+
+          # Get short SHA for all builds
+          short_sha=$(git rev-parse --short HEAD)
+
+          # Determine build type based on trigger
+          if [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
+            # Tag push - release or prerelease
+            should_build=true
+            tag_name="${GITHUB_REF#refs/tags/}"
+            version="${tag_name}"
+
+            # Check if this is a prerelease
+            if [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
+              build_type="prerelease"
+              is_prerelease=true
+              echo "🚀 Prerelease build detected: $tag_name"
+            else
+              build_type="release"
+              echo "📦 Release build detected: $tag_name"
+            fi
+          elif [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
+            # Main branch push - development build
+            should_build=true
+            build_type="development"
+            version="dev-${short_sha}"
+            echo "🛠️  Development build detected"
+          elif [[ "${{ github.event_name }}" == "schedule" ]] || \
+               [[ "${{ github.event_name }}" == "workflow_dispatch" ]] || \
+               [[ "${{ contains(github.event.head_commit.message, '--build') }}" == "true" ]]; then
+            # Scheduled or manual build
+            should_build=true
+            build_type="development"
+            version="dev-${short_sha}"
+            echo "⚡ Manual/scheduled build detected"
+          fi
+
+          echo "should_build=$should_build" >> $GITHUB_OUTPUT
+          echo "build_type=$build_type" >> $GITHUB_OUTPUT
+          echo "version=$version" >> $GITHUB_OUTPUT
+          echo "short_sha=$short_sha" >> $GITHUB_OUTPUT
+          echo "is_prerelease=$is_prerelease" >> $GITHUB_OUTPUT
+
+          echo "📊 Build Summary:"
+          echo "  - Should build: $should_build"
+          echo "  - Build type: $build_type"
+          echo "  - Version: $version"
+          echo "  - Short SHA: $short_sha"
+          echo "  - Is prerelease: $is_prerelease"
+
+  # Build RustFS binaries
+  prepare-platform-matrix:
+    name: Prepare Platform Matrix
+    runs-on: ubicloud-standard-2
+    outputs:
+      matrix: ${{ steps.select.outputs.matrix }}
+      selected: ${{ steps.select.outputs.selected }}
+    steps:
+      - name: Select target platforms
+        id: select
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          selected="${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}"
+          selected="$(echo "${selected}" | tr -d '[:space:]')"
+          if [[ -z "${selected}" ]]; then
+            selected="all"
+          fi
+
+          all='{"include":[
+            {"target_id":"linux-x86_64-musl","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":""},
+            {"target_id":"linux-aarch64-musl","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":""},
+            {"target_id":"linux-x86_64-gnu","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
+            {"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
+            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"macos-x86_64","os":"macos-latest","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
+          ]}'
+
+          if [[ "${selected}" == "all" ]]; then
+            matrix="$(jq -c . <<<"${all}")"
+          else
+            unknown="$(jq -rn --arg selected "${selected}" --argjson all "${all}" '
+              ($selected | split(",") | map(select(length > 0))) as $req
+              | ($all.include | map(.target_id)) as $known
+              | [$req[] | select(( $known | index(.) ) == null)]
+            ')"
+            if [[ "$(jq 'length' <<<"${unknown}")" -gt 0 ]]; then
+              echo "Unknown platforms: $(jq -r 'join(\",\")' <<<"${unknown}")" >&2
+              echo "Allowed: $(jq -r '.include[].target_id' <<<"${all}" | paste -sd ',' -)" >&2
+              exit 1
+            fi
+
+            matrix="$(jq -c --arg selected "${selected}" '
+              ($selected | split(",") | map(select(length > 0))) as $req
+              | .include |= map(select(.target_id as $id | ($req | index($id))))
+            ' <<<"${all}")"
+          fi
+
+          echo "selected=${selected}" >> "$GITHUB_OUTPUT"
+          echo "matrix=${matrix}" >> "$GITHUB_OUTPUT"
+          echo "Selected platforms: ${selected}"
+
+  build-rustfs:
+    name: Build RustFS
+    needs: [ build-check, prepare-platform-matrix ]
+    if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 60
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+      # Always enable Tokio unstable features (required by dial9-tokio-telemetry).
+      # The RUSTFLAGS env var takes precedence over .cargo/config.toml [build] rustflags,
+      # so we must include --cfg tokio_unstable here explicitly; otherwise an empty
+      # RUSTFLAGS value would shadow the config-file flag and silently break tracing.
+      RUSTFLAGS: "--cfg tokio_unstable ${{ matrix.rustflags }}"
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJson(needs.prepare-platform-matrix.outputs.matrix) }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          target: ${{ matrix.target }}
+          cache-shared-key: build-${{ matrix.target }}-${{ hashFiles('**/Cargo.lock') }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/') }}
+          install-cross-tools: ${{ matrix.cross }}
+
+      - name: Download static console assets
+        shell: bash
+        run: |
+          mkdir -p ./rustfs/static
+          if [[ "${{ matrix.platform }}" == "windows" ]]; then
+            curl.exe -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o console.zip --retry 3 --retry-delay 5 --max-time 300
+            if [[ $? -eq 0 ]]; then
+              unzip -o console.zip -d ./rustfs/static
+              rm console.zip
+            else
+              echo "Warning: Failed to download console assets, continuing without them"
+              echo "// Static assets not available" > ./rustfs/static/empty.txt
+            fi
+          else
+            chmod +w ./rustfs/static/LICENSE || true
+            curl -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" \
+              -o console.zip --retry 3 --retry-delay 5 --max-time 300
+            if [[ $? -eq 0 ]]; then
+              unzip -o console.zip -d ./rustfs/static
+              rm console.zip
+            else
+              echo "Warning: Failed to download console assets, continuing without them"
+              echo "// Static assets not available" > ./rustfs/static/empty.txt
+            fi
+          fi
+
+      - name: Build RustFS
+        shell: bash
+        run: |
+          # Force rebuild by touching build.rs
+          touch rustfs/build.rs
+
+          if [[ "${{ matrix.cross }}" == "true" ]]; then
+            if [[ "${{ matrix.platform }}" == "windows" ]]; then
+              # Use cross for Windows ARM64
+              cargo install cross --git https://github.com/cross-rs/cross
+              cross build --release --target ${{ matrix.target }} -p rustfs --bins
+            else
+              # Use zigbuild for other cross-compilation
+              cargo zigbuild --release --target ${{ matrix.target }} -p rustfs --bins
+            fi
+          else
+            cargo build --release --target ${{ matrix.target }} -p rustfs --bins
+          fi
+
+      - name: Create release package
+        id: package
+        shell: bash
+        run: |
+          BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
+          VERSION="${{ needs.build-check.outputs.version }}"
+          SHORT_SHA="${{ needs.build-check.outputs.short_sha }}"
+
+          # Extract platform and arch from target
+          TARGET="${{ matrix.target }}"
+          PLATFORM="${{ matrix.platform }}"
+          # Map target to architecture and variant
+          case "$TARGET" in
+            *x86_64*musl*)
+              ARCH="x86_64"
+              VARIANT="musl"
+              ;;
+            *x86_64*gnu*)
+              ARCH="x86_64"
+              VARIANT="gnu"
+              ;;
+            *x86_64*)
+              ARCH="x86_64"
+              VARIANT=""
+              ;;
+            *aarch64*musl*|*arm64*musl*)
+              ARCH="aarch64"
+              VARIANT="musl"
+              ;;
+            *aarch64*gnu*|*arm64*gnu*)
+              ARCH="aarch64"
+              VARIANT="gnu"
+              ;;
+            *aarch64*|*arm64*)
+              ARCH="aarch64"
+              VARIANT=""
+              ;;
+            *armv7*)
+              ARCH="armv7"
+              VARIANT=""
+              ;;
+            *)
+              ARCH="unknown"
+              VARIANT=""
+              ;;
+          esac
+
+          # Normalize version used for package filenames
+          PACKAGE_VERSION="${VERSION}"
+          if [[ "$PACKAGE_VERSION" == v* ]]; then
+            PACKAGE_VERSION="${PACKAGE_VERSION#v}"
+          fi
+
+          # Generate package name based on build type
+          if [[ -n "$VARIANT" ]]; then
+            ARCH_WITH_VARIANT="${ARCH}-${VARIANT}"
+          else
+            ARCH_WITH_VARIANT="${ARCH}"
+          fi
+          PACKAGE_BASENAME="rustfs-${PLATFORM}-${ARCH_WITH_VARIANT}"
+
+          if [[ "$BUILD_TYPE" == "development" ]]; then
+            # Development build: rustfs-${platform}-${arch}-${variant}-dev-${short_sha}.zip
+            PACKAGE_NAME="rustfs-${PLATFORM}-${ARCH_WITH_VARIANT}-dev-${SHORT_SHA}"
+          else
+            # Release/Prerelease build: rustfs-${platform}-${arch}-${variant}-v${version}.zip
+            PACKAGE_NAME="${PACKAGE_BASENAME}-v${PACKAGE_VERSION}"
+          fi
+
+          # Create zip packages for all platforms
+          # Ensure zip is available
+          if ! command -v zip &> /dev/null; then
+            if [[ "${{ matrix.os }}" == "ubuntu-latest" ]]; then
+              sudo apt-get update && sudo apt-get install -y zip
+            fi
+          fi
+
+          cd target/${{ matrix.target }}/release
+          # Determine the binary name based on platform
+          if [[ "${{ matrix.platform }}" == "windows" ]]; then
+            BINARY_NAME="rustfs.exe"
+          else
+            BINARY_NAME="rustfs"
+          fi
+
+          # Verify the binary exists before packaging
+          if [[ ! -f "$BINARY_NAME" ]]; then
+            echo "❌ Binary $BINARY_NAME not found in $(pwd)"
+            if [[ "${{ matrix.platform }}" == "windows" ]]; then
+              dir
+            else
+              ls -la
+            fi
+            exit 1
+          fi
+
+          # Universal packaging function
+          package_zip() {
+            local src=$1
+            local dst=$2
+            if [[ "${{ matrix.platform }}" == "windows" ]]; then
+              # Windows uses PowerShell Compress-Archive
+              powershell -Command "Compress-Archive -Path '$src' -DestinationPath '$dst' -Force"
+            elif command -v zip &> /dev/null; then
+              # Unix systems use zip command
+              zip "$dst" "$src"
+            else
+              echo "❌ No zip utility available"
+              exit 1
+            fi
+          }
+
+          # Create the zip package
+          echo "Start packaging: $BINARY_NAME -> ../../../${PACKAGE_NAME}.zip"
+          package_zip "$BINARY_NAME" "../../../${PACKAGE_NAME}.zip"
+
+          cd ../../..
+
+          # Verify the package was created
+          if [[ -f "${PACKAGE_NAME}.zip" ]]; then
+            echo "✅ Package created successfully: ${PACKAGE_NAME}.zip"
+            if [[ "${{ matrix.platform }}" == "windows" ]]; then
+              dir
+            else
+              ls -lh ${PACKAGE_NAME}.zip
+            fi
+          else
+            echo "❌ Failed to create package: ${PACKAGE_NAME}.zip"
+            exit 1
+          fi
+
+          # Create latest version files right after the main package
+          LATEST_FILES=""
+          if [[ "$BUILD_TYPE" == "release" ]] || [[ "$BUILD_TYPE" == "prerelease" ]]; then
+            # Create latest version filename
+            # Convert from rustfs-linux-x86_64-musl-v1.0.0 to rustfs-linux-x86_64-musl-latest
+            LATEST_FILE="${PACKAGE_BASENAME}-latest.zip"
+
+            echo "🔄 Creating latest version: ${PACKAGE_NAME}.zip -> $LATEST_FILE"
+            cp "${PACKAGE_NAME}.zip" "$LATEST_FILE"
+
+            if [[ -f "$LATEST_FILE" ]]; then
+              echo "✅ Latest version created: $LATEST_FILE"
+              LATEST_FILES="$LATEST_FILE"
+            fi
+          elif [[ "$BUILD_TYPE" == "development" ]]; then
+            # Development builds (only main branch triggers development builds)
+            # Create main-latest version filename
+            # Convert from rustfs-linux-x86_64-dev-abc123 to rustfs-linux-x86_64-main-latest
+            MAIN_LATEST_FILE="${PACKAGE_BASENAME}-main-latest.zip"
+
+            echo "🔄 Creating main-latest version: ${PACKAGE_NAME}.zip -> $MAIN_LATEST_FILE"
+            cp "${PACKAGE_NAME}.zip" "$MAIN_LATEST_FILE"
+
+            if [[ -f "$MAIN_LATEST_FILE" ]]; then
+              echo "✅ Main-latest version created: $MAIN_LATEST_FILE"
+              LATEST_FILES="$MAIN_LATEST_FILE"
+
+              # Also create a generic main-latest for Docker builds (Linux only)
+              if [[ "${{ matrix.platform }}" == "linux" ]]; then
+                DOCKER_MAIN_LATEST_FILE="rustfs-linux-${ARCH_WITH_VARIANT}-main-latest.zip"
+
+                echo "🔄 Creating Docker main-latest version: ${PACKAGE_NAME}.zip -> $DOCKER_MAIN_LATEST_FILE"
+                cp "${PACKAGE_NAME}.zip" "$DOCKER_MAIN_LATEST_FILE"
+
+                if [[ -f "$DOCKER_MAIN_LATEST_FILE" ]]; then
+                  echo "✅ Docker main-latest version created: $DOCKER_MAIN_LATEST_FILE"
+                  LATEST_FILES="$LATEST_FILES $DOCKER_MAIN_LATEST_FILE"
+                fi
+              fi
+            fi
+          fi
+
+          echo "package_name=${PACKAGE_NAME}" >> $GITHUB_OUTPUT
+          echo "package_file=${PACKAGE_NAME}.zip" >> $GITHUB_OUTPUT
+          echo "latest_files=${LATEST_FILES}" >> $GITHUB_OUTPUT
+          echo "build_type=${BUILD_TYPE}" >> $GITHUB_OUTPUT
+          echo "version=${VERSION}" >> $GITHUB_OUTPUT
+
+          echo "📦 Package created: ${PACKAGE_NAME}.zip"
+          if [[ -n "$LATEST_FILES" ]]; then
+            echo "📦 Latest files created: $LATEST_FILES"
+          fi
+          echo "🔧 Build type: ${BUILD_TYPE}"
+          echo "📊 Version: ${VERSION}"
+
+      - name: Upload to GitHub artifacts
+        uses: actions/upload-artifact@v6
+        with:
+          name: ${{ steps.package.outputs.package_name }}
+          path: "rustfs-*.zip"
+          retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 7 }}
+
+      - name: Upload to Aliyun OSS
+        if: env.OSS_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
+        env:
+          OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
+          OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
+          OSS_REGION: cn-beijing
+          OSS_ENDPOINT: https://oss-accelerate.aliyuncs.com
+        shell: bash
+        run: |
+          BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
+
+          # Install ossutil (platform-specific)
+          OSSUTIL_VERSION="2.1.1"
+          case "${{ matrix.platform }}" in
+            linux)
+              if [[ "$(uname -m)" == "arm64" ]]; then
+                ARCH="arm64"
+              else
+                ARCH="amd64"
+              fi
+              OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-${ARCH}.zip"
+              OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-linux-${ARCH}"
+
+              curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
+              unzip "$OSSUTIL_ZIP"
+              mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
+              rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
+              chmod +x /usr/local/bin/ossutil
+              OSSUTIL_BIN=ossutil
+              ;;
+            macos)
+              if [[ "$(uname -m)" == "arm64" ]]; then
+                ARCH="arm64"
+              else
+                ARCH="amd64"
+              fi
+              OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-mac-${ARCH}.zip"
+              OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-mac-${ARCH}"
+
+              curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
+              unzip "$OSSUTIL_ZIP"
+              mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
+              rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
+              chmod +x /usr/local/bin/ossutil
+              OSSUTIL_BIN=ossutil
+              ;;
+            windows)
+              OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-windows-amd64.zip"
+              OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-windows-amd64"
+
+              curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
+              unzip "$OSSUTIL_ZIP"
+              mv "${OSSUTIL_DIR}/ossutil.exe" ./ossutil.exe
+              rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
+              OSSUTIL_BIN=./ossutil.exe
+              ;;
+          esac
+
+          # Determine upload path based on build type
+          if [[ "$BUILD_TYPE" == "development" ]]; then
+            OSS_PATH="oss://rustfs-artifacts/artifacts/rustfs/dev/"
+            echo "📤 Uploading development build to OSS dev directory"
+          else
+            OSS_PATH="oss://rustfs-artifacts/artifacts/rustfs/release/"
+            echo "📤 Uploading release build to OSS release directory"
+          fi
+
+          # Upload all rustfs zip files to OSS using glob pattern
+          echo "📤 Uploading all rustfs-*.zip files to $OSS_PATH..."
+          for zip_file in rustfs-*.zip; do
+            if [[ -f "$zip_file" ]]; then
+              echo "Uploading: $zip_file to $OSS_PATH..."
+              $OSSUTIL_BIN cp "$zip_file" "$OSS_PATH" --force
+              echo "✅ Uploaded: $zip_file"
+            fi
+          done
+
+          echo "✅ Upload completed successfully"
+
+  # Build summary
+  build-summary:
+    name: Build Summary
+    needs: [ build-check, build-rustfs ]
+    if: always() && needs.build-check.outputs.should_build == 'true'
+    runs-on: ubicloud-standard-2
+    steps:
+      - name: Build completion summary
+        shell: bash
+        run: |
+          BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
+          VERSION="${{ needs.build-check.outputs.version }}"
+
+          echo "🎉 Build completed successfully!"
+          echo "📦 Build type: $BUILD_TYPE"
+          echo "🔢 Version: $VERSION"
+          echo ""
+
+          # Check build status
+          BUILD_STATUS="${{ needs.build-rustfs.result }}"
+
+          echo "📊 Build Results:"
+          echo "  📦 All platforms: $BUILD_STATUS"
+          echo ""
+
+          case "$BUILD_TYPE" in
+            "development")
+              echo "🛠️  Development build artifacts have been uploaded to OSS dev directory"
+              echo "⚠️  This is a development build - not suitable for production use"
+              ;;
+            "release")
+              echo "🚀 Release build artifacts have been uploaded to OSS release directory"
+              echo "✅ This build is ready for production use"
+              echo "🏷️  GitHub Release will be created in this workflow"
+              ;;
+            "prerelease")
+              echo "🧪 Prerelease build artifacts have been uploaded to OSS release directory"
+              echo "⚠️  This is a prerelease build - use with caution"
+              echo "🏷️  GitHub Release will be created in this workflow"
+              ;;
+          esac
+
+          echo ""
+          echo "🐳 Docker Images:"
+          if [[ "${{ github.event.inputs.build_docker }}" == "false" ]]; then
+            echo "⏭️  Docker image build was skipped (binary only build)"
+          elif [[ "$BUILD_STATUS" == "success" ]]; then
+            echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
+          else
+            echo "❌ Docker image build will be skipped due to build failure"
+          fi
+
+  # Create GitHub Release (only for tag pushes)
+  create-release:
+    name: Create GitHub Release
+    needs: [ build-check, build-rustfs ]
+    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
+    runs-on: ubicloud-standard-2
+    permissions:
+      contents: write
+    outputs:
+      release_id: ${{ steps.create.outputs.release_id }}
+      release_url: ${{ steps.create.outputs.release_url }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - name: Create GitHub Release
+        id: create
+        env:
+          GH_TOKEN: ${{ github.token }}
+        shell: bash
+        run: |
+          TAG="${{ needs.build-check.outputs.version }}"
+          VERSION="${{ needs.build-check.outputs.version }}"
+          IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
+          BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
+
+          # Determine release type for title
+          if [[ "$BUILD_TYPE" == "prerelease" ]]; then
+            if [[ "$TAG" == *"alpha"* ]]; then
+              RELEASE_TYPE="alpha"
+            elif [[ "$TAG" == *"beta"* ]]; then
+              RELEASE_TYPE="beta"
+            elif [[ "$TAG" == *"rc"* ]]; then
+              RELEASE_TYPE="rc"
+            else
+              RELEASE_TYPE="prerelease"
+            fi
+          else
+            RELEASE_TYPE="release"
+          fi
+
+          # Check if release already exists
+          if gh release view "$TAG" >/dev/null 2>&1; then
+            echo "Release $TAG already exists"
+            RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
+            RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
+          else
+            # Get release notes from tag message
+            RELEASE_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
+            if [[ -z "$RELEASE_NOTES" || "$RELEASE_NOTES" =~ ^[[:space:]]*$ ]]; then
+              if [[ "$IS_PRERELEASE" == "true" ]]; then
+                RELEASE_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
+              else
+                RELEASE_NOTES="Release ${VERSION}"
+              fi
+            fi
+
+            # Create release title
+            if [[ "$IS_PRERELEASE" == "true" ]]; then
+              TITLE="RustFS $VERSION (${RELEASE_TYPE})"
+            else
+              TITLE="RustFS $VERSION"
+            fi
+
+            # Create the release
+            PRERELEASE_FLAG=""
+            if [[ "$IS_PRERELEASE" == "true" ]]; then
+              PRERELEASE_FLAG="--prerelease"
+            fi
+
+            gh release create "$TAG" \
+              --title "$TITLE" \
+              --notes "$RELEASE_NOTES" \
+              $PRERELEASE_FLAG \
+              --draft
+
+            RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
+            RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
+          fi
+
+          echo "release_id=$RELEASE_ID" >> $GITHUB_OUTPUT
+          echo "release_url=$RELEASE_URL" >> $GITHUB_OUTPUT
+          echo "Created release: $RELEASE_URL"
+
+  # Prepare and upload release assets
+  upload-release-assets:
+    name: Upload Release Assets
+    needs: [ build-check, build-rustfs, create-release ]
+    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
+    runs-on: ubicloud-standard-2
+    permissions:
+      contents: write
+      actions: read
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Download all build artifacts
+        uses: actions/download-artifact@v7
+        with:
+          path: ./artifacts
+          pattern: rustfs-*
+          merge-multiple: true
+
+      - name: Prepare release assets
+        id: prepare
+        shell: bash
+        run: |
+          VERSION="${{ needs.build-check.outputs.version }}"
+          TAG="${{ needs.build-check.outputs.version }}"
+
+          mkdir -p ./release-assets
+
+          # Copy and verify artifacts (including latest files created during build)
+          ASSETS_COUNT=0
+          for file in ./artifacts/*.zip; do
+            if [[ -f "$file" ]]; then
+              cp "$file" ./release-assets/
+              ASSETS_COUNT=$((ASSETS_COUNT + 1))
+            fi
+          done
+
+          if [[ $ASSETS_COUNT -eq 0 ]]; then
+            echo "❌ No artifacts found!"
+            exit 1
+          fi
+
+          cd ./release-assets
+
+          # Generate checksums for all files (including latest versions)
+          if ls *.zip >/dev/null 2>&1; then
+            sha256sum *.zip > SHA256SUMS
+            sha512sum *.zip > SHA512SUMS
+          fi
+
+          # Create signature placeholder files
+          for file in *.zip; do
+            echo "# Signature for $file" > "${file}.asc"
+            echo "# GPG signature will be added in future versions" >> "${file}.asc"
+          done
+
+          echo "📦 Prepared assets:"
+          ls -la
+
+          echo "🔢 Total asset count: $ASSETS_COUNT"
+
+      - name: Upload to GitHub Release
+        env:
+          GH_TOKEN: ${{ github.token }}
+        shell: bash
+        run: |
+          TAG="${{ needs.build-check.outputs.version }}"
+
+          cd ./release-assets
+
+          # Upload all files
+          for file in *; do
+            if [[ -f "$file" ]]; then
+              echo "📤 Uploading $file..."
+              gh release upload "$TAG" "$file" --clobber
+            fi
+          done
+
+          echo "✅ All assets uploaded successfully"
+
+  # Update latest.json for stable releases only
+  update-latest-version:
+    name: Update Latest Version
+    needs: [ build-check, upload-release-assets ]
+    if: startsWith(github.ref, 'refs/tags/')
+    runs-on: ubicloud-standard-2
+    steps:
+      - name: Update latest.json
+        env:
+          OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
+          OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
+          OSS_REGION: cn-beijing
+          OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
+        shell: bash
+        run: |
+          if [[ -z "$OSS_ACCESS_KEY_ID" ]]; then
+            echo "⚠️ OSS credentials not available, skipping latest.json update"
+            exit 0
+          fi
+
+          VERSION="${{ needs.build-check.outputs.version }}"
+          TAG="${{ needs.build-check.outputs.version }}"
+
+          # Install ossutil
+          OSSUTIL_VERSION="2.1.1"
+          OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-amd64.zip"
+          OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-linux-amd64"
+
+          curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
+          unzip "$OSSUTIL_ZIP"
+          mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
+          rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
+          chmod +x /usr/local/bin/ossutil
+
+          # Create latest.json
+          cat > latest.json << EOF
+          {
+            "version": "${VERSION}",
+            "tag": "${TAG}",
+            "release_date": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
+            "release_type": "stable",
+            "download_url": "https://github.com/${{ github.repository }}/releases/tag/${TAG}"
+          }
+          EOF
+
+          # Upload to OSS
+          ossutil cp latest.json oss://rustfs-version/latest.json --force
+
+          echo "✅ Updated latest.json for stable release $VERSION"
+
+  # Publish release (remove draft status)
+  publish-release:
+    name: Publish Release
+    needs: [ build-check, create-release, upload-release-assets ]
+    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
+    runs-on: ubicloud-standard-2
+    permissions:
+      contents: write
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Update release notes and publish
+        env:
+          GH_TOKEN: ${{ github.token }}
+        shell: bash
+        run: |
+          TAG="${{ needs.build-check.outputs.version }}"
+          VERSION="${{ needs.build-check.outputs.version }}"
+          IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
+          BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
+
+          # Determine release type
+          if [[ "$BUILD_TYPE" == "prerelease" ]]; then
+            if [[ "$TAG" == *"alpha"* ]]; then
+              RELEASE_TYPE="alpha"
+            elif [[ "$TAG" == *"beta"* ]]; then
+              RELEASE_TYPE="beta"
+            elif [[ "$TAG" == *"rc"* ]]; then
+              RELEASE_TYPE="rc"
+            else
+              RELEASE_TYPE="prerelease"
+            fi
+          else
+            RELEASE_TYPE="release"
+          fi
+
+          # Get original release notes from tag
+          ORIGINAL_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
+          if [[ -z "$ORIGINAL_NOTES" || "$ORIGINAL_NOTES" =~ ^[[:space:]]*$ ]]; then
+            if [[ "$IS_PRERELEASE" == "true" ]]; then
+              ORIGINAL_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
+            else
+              ORIGINAL_NOTES="Release ${VERSION}"
+            fi
+          fi
+
+          # Publish the release (remove draft status)
+          gh release edit "$TAG" --draft=false
+
+          echo "🎉 Released $TAG successfully!"
+          echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"

.github/workflows/ci.yml

@@ -0,0 +1,256 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Continuous Integration
+
+on:
+  push:
+    branches: [ main ]
+    paths-ignore:
+      - "**.md"
+      - "docs/**"
+      - "deploy/**"
+      - "scripts/dev_*.sh"
+      - "scripts/probe.sh"
+      - "LICENSE*"
+      - ".gitignore"
+      - ".dockerignore"
+      - "README*"
+      - "**/*.png"
+      - "**/*.jpg"
+      - "**/*.svg"
+      - ".github/workflows/build.yml"
+      - ".github/workflows/docker.yml"
+      - ".github/workflows/audit.yml"
+      - ".github/workflows/performance.yml"
+  pull_request:
+    branches: [ main ]
+    paths-ignore:
+      - "**.md"
+      - "docs/**"
+      - "deploy/**"
+      - "scripts/dev_*.sh"
+      - "scripts/probe.sh"
+      - "LICENSE*"
+      - ".gitignore"
+      - ".dockerignore"
+      - "README*"
+      - "**/*.png"
+      - "**/*.jpg"
+      - "**/*.svg"
+      - ".github/workflows/build.yml"
+      - ".github/workflows/docker.yml"
+      - ".github/workflows/audit.yml"
+      - ".github/workflows/performance.yml"
+  merge_group:
+    types: [checks_requested]
+  schedule:
+    - cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  CARGO_TERM_COLOR: always
+  RUST_BACKTRACE: 1
+  CARGO_BUILD_JOBS: 2
+
+jobs:
+
+  skip-check:
+    name: Skip Duplicate Actions
+    permissions:
+      actions: write
+      contents: read
+    runs-on: ubicloud-standard-2
+    outputs:
+      should_skip: ${{ steps.skip_check.outputs.should_skip }}
+    steps:
+      - name: Skip duplicate actions
+        id: skip_check
+        uses: fkirc/skip-duplicate-actions@v5
+        with:
+          concurrent_skipping: "same_content_newer"
+          cancel_others: true
+          paths_ignore: '["*.md", "docs/**", "deploy/**"]'
+          do_not_skip: '["workflow_dispatch", "schedule", "merge_group", "release", "push"]'
+
+  typos:
+    name: Typos
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
+    runs-on: ubicloud-standard-2
+    steps:
+      - uses: actions/checkout@v6
+      - uses: dtolnay/rust-toolchain@stable
+      - name: Typos check with custom config file
+        uses: crate-ci/typos@master
+
+  test-and-lint:
+    name: Test and Lint
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
+    runs-on: ubicloud-standard-4
+    timeout-minutes: 60
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-test-${{ hashFiles('**/Cargo.lock') }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Run tests
+        run: |
+          cargo nextest run --all --exclude e2e_test
+          cargo test --all --doc
+
+      - name: Check code formatting
+        run: cargo fmt --all --check
+
+      - name: Check unsafe code allowances
+        run: ./scripts/check_unsafe_code_allowances.sh
+
+      - name: Run clippy lints
+        run: cargo clippy --all-targets --all-features -- -D warnings
+
+      - name: Check layered dependencies
+        run: ./scripts/check_layer_dependencies.sh
+
+  build-rustfs-debug-binary:
+    name: Build RustFS Debug Binary
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
+    runs-on: ubicloud-standard-4
+    timeout-minutes: 30
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-rustfs-debug-binary-${{ hashFiles('**/Cargo.lock') }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Build debug binary
+        run: |
+          touch rustfs/build.rs
+          cargo build -p rustfs --bins --jobs 2
+
+      - name: Upload debug binary
+        uses: actions/upload-artifact@v6
+        with:
+          name: rustfs-debug-binary
+          path: target/debug/rustfs
+          if-no-files-found: error
+          retention-days: 1
+
+  e2e-tests:
+    name: End-to-End Tests
+    needs: [ skip-check, build-rustfs-debug-binary ]
+    if: needs.skip-check.outputs.should_skip != 'true'
+    runs-on: ubicloud-standard-2
+    timeout-minutes: 30
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Clean up previous test run
+        run: |
+          rm -rf /tmp/rustfs
+          rm -f /tmp/rustfs.log
+
+      - name: Download debug binary
+        uses: actions/download-artifact@v7
+        with:
+          name: rustfs-debug-binary
+          path: target/debug
+
+      - name: Make binary executable
+        run: chmod +x ./target/debug/rustfs
+
+      - name: Setup Rust toolchain for s3s-e2e installation
+        uses: dtolnay/rust-toolchain@stable
+
+      - name: Install s3s-e2e test tool
+        uses: taiki-e/cache-cargo-install-action@v2
+        with:
+          tool: s3s-e2e
+          git: https://github.com/s3s-project/s3s.git
+          rev: 4a04a670cf41274d9be9ab65dc36f4aa3f92fbad
+
+      - name: Run end-to-end tests
+        run: |
+          s3s-e2e --version
+          ./scripts/e2e-run.sh ./target/debug/rustfs /tmp/rustfs
+
+      - name: Upload test logs
+        if: failure()
+        uses: actions/upload-artifact@v6
+        with:
+          name: e2e-test-logs-${{ github.run_number }}
+          path: /tmp/rustfs.log
+          retention-days: 3
+
+  s3-implemented-tests:
+    name: S3 Implemented Tests
+    needs: [ skip-check, build-rustfs-debug-binary ]
+    if: needs.skip-check.outputs.should_skip != 'true'
+    runs-on: ubicloud-standard-4
+    timeout-minutes: 60
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Download debug binary
+        uses: actions/download-artifact@v7
+        with:
+          name: rustfs-debug-binary
+          path: target/debug
+
+      - name: Make binary executable
+        run: chmod +x ./target/debug/rustfs
+
+      - name: Run implemented s3-tests
+        run: |
+          DEPLOY_MODE=binary \
+          RUSTFS_BINARY=./target/debug/rustfs \
+          TEST_MODE=single \
+          MAXFAIL=1 \
+          ./scripts/s3-tests/run.sh
+
+      - name: Upload s3 test artifacts
+        if: always()
+        uses: actions/upload-artifact@v6
+        with:
+          name: s3tests-implemented-${{ github.run_number }}
+          path: artifacts/s3tests-single/**
+          if-no-files-found: ignore
+          retention-days: 3

.github/workflows/cla.yml

@@ -0,0 +1,70 @@
+# Copyright 2026 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: CLA Check
+
+on:
+  pull_request_target:
+    types: [opened, synchronize, reopened]
+  merge_group:
+    types: [checks_requested]
+  issue_comment:
+    types: [created, edited]
+
+permissions:
+  contents: write
+  pull-requests: write
+  issues: write
+  checks: write
+
+jobs:
+  cla:
+    if: ${{ github.event_name != 'issue_comment' || github.event.issue.pull_request }}
+    runs-on: ubuntu-latest
+    steps:
+      - name: Report CLA result for merge queue
+        if: github.event_name == 'merge_group'
+        uses: actions/github-script@v8
+        with:
+          script: |
+            await github.rest.checks.create({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              name: 'CLA Check',
+              head_sha: context.sha,
+              status: 'completed',
+              conclusion: 'success',
+              output: {
+                title: 'CLA requirements satisfied for merge queue',
+                summary: 'Queued pull requests must satisfy the required CLA check before they enter the merge queue. This reports the existing CLA result on the merge-group SHA.'
+              }
+            });
+
+      - name: Create token for rustfs/cla
+        if: github.event_name != 'merge_group'
+        id: registry-token
+        uses: actions/create-github-app-token@v3
+        with:
+          app-id: ${{ vars.CLA_BOT_APP_ID }}
+          private-key: ${{ secrets.CLA_BOT_APP_PRIVATE_KEY }}
+          owner: ${{ github.repository_owner }}
+          repositories: cla
+          permission-contents: write
+
+      - name: Run CLA Bot
+        if: github.event_name != 'merge_group'
+        uses: overtrue/cla-bot@v0.0.9
+        with:
+          github-token: ${{ github.token }}
+          registry-token: ${{ steps.registry-token.outputs.token }}

.github/workflows/docker.yml

@@ -0,0 +1,459 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Docker Images Workflow
+#
+# This workflow builds Docker images using pre-built binaries from the build workflow.
+#
+# Trigger Types:
+# 1. workflow_run: Automatically triggered when "Build and Release" workflow completes
+# 2. workflow_dispatch: Manual trigger for standalone Docker builds
+#
+# Key Features:
+# - Only triggers when Linux builds (x86_64 + aarch64) are successful
+# - Independent of macOS/Windows build status
+# - Uses workflow_run event for precise control
+# - Only builds Docker images for releases and prereleases (development builds are skipped)
+
+name: Docker Images
+
+# Permissions needed for workflow_run event and Docker registry access
+permissions:
+  contents: read
+  packages: write
+
+on:
+  # Automatically triggered when build workflow completes
+  workflow_run:
+    workflows: [ "Build and Release" ]
+    types: [ completed ]
+  # Manual trigger with same parameters for consistency
+  workflow_dispatch:
+    inputs:
+      push_images:
+        description: "Push images to registries"
+        required: false
+        default: true
+        type: boolean
+      version:
+        description: "Version to build (latest for stable release, or specific version like v1.0.0, v1.0.0-alpha1)"
+        required: false
+        default: "latest"
+        type: string
+      force_rebuild:
+        description: "Force rebuild even if binary exists (useful for testing)"
+        required: false
+        default: false
+        type: boolean
+
+env:
+  CONCLUSION: ${{ github.event.workflow_run.conclusion }}
+  HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
+  HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
+  TRIGGERING_EVENT: ${{ github.event.workflow_run.event }}
+  DOCKERHUB_USERNAME: rustfs
+  CARGO_TERM_COLOR: always
+  REGISTRY_DOCKERHUB: rustfs/rustfs
+  REGISTRY_GHCR: ghcr.io/${{ github.repository }}
+  REGISTRY_QUAY: quay.io/${{ secrets.QUAY_USERNAME }}/rustfs
+  DOCKER_PLATFORMS: linux/amd64,linux/arm64
+
+jobs:
+  # Check if we should build Docker images
+  build-check:
+    name: Docker Build Check
+    runs-on: ubicloud-standard-2
+    outputs:
+      should_build: ${{ steps.check.outputs.should_build }}
+      should_push: ${{ steps.check.outputs.should_push }}
+      build_type: ${{ steps.check.outputs.build_type }}
+      version: ${{ steps.check.outputs.version }}
+      short_sha: ${{ steps.check.outputs.short_sha }}
+      is_prerelease: ${{ steps.check.outputs.is_prerelease }}
+      create_latest: ${{ steps.check.outputs.create_latest }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+          # For workflow_run events, checkout the specific commit that triggered the workflow
+          ref: ${{ github.event.workflow_run.head_sha || github.sha }}
+
+      - name: Check build conditions
+        id: check
+        run: |
+          should_build=false
+          should_push=false
+          build_type="none"
+          version=""
+          short_sha=""
+          is_prerelease=false
+          create_latest=false
+
+          if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
+            # Triggered by build workflow completion
+            echo "🔗 Triggered by build workflow completion"
+
+            # Check if the triggering workflow was successful
+            # If the workflow succeeded, it means ALL builds (including Linux x86_64 and aarch64) succeeded
+            if [[ "$CONCLUSION" == "success" ]]; then
+              echo "✅ Build workflow succeeded, all builds including Linux are successful"
+              should_build=true
+              should_push=true
+            else
+              echo "❌ Build workflow failed (conclusion: $CONCLUSION), skipping Docker build"
+              should_build=false
+            fi
+
+            # Extract version info from commit message or use commit SHA
+            # Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
+            short_sha=$(git rev-parse --short "$HEAD_SHA")
+
+            # Determine build type based on triggering workflow event and ref
+            triggering_event="$TRIGGERING_EVENT"
+            head_branch="$HEAD_BRANCH"
+
+            echo "🔍 Analyzing triggering workflow:"
+            echo "   📋 Event: $triggering_event"
+            echo "   🌿 Head branch: $head_branch"
+            echo "   📎 Head SHA: $HEAD_SHA"
+
+            # Check if this was triggered by a tag push
+            if [[ "$triggering_event" == "push" ]]; then
+              # For tag pushes, head_branch will be like "refs/tags/v1.0.0" or just "v1.0.0"
+              if [[ "$head_branch" == refs/tags/* ]]; then
+                # Extract tag name from refs/tags/TAG_NAME
+                tag_name="${head_branch#refs/tags/}"
+                version="$tag_name"
+              elif [[ "$head_branch" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+ ]]; then
+                # Direct tag name like "v1.0.0" or "1.0.0-alpha.1"
+                version="$head_branch"
+              elif [[ "$head_branch" == "main" ]]; then
+                # Regular branch push to main
+                build_type="development"
+                version="dev-${short_sha}"
+                should_build=false
+                echo "⏭️  Skipping Docker build for development version (main branch push)"
+              else
+                # Other branch push
+                build_type="development"
+                version="dev-${short_sha}"
+                should_build=false
+                echo "⏭️  Skipping Docker build for development version (branch: $head_branch)"
+              fi
+
+              # If we extracted a version (tag), determine release type
+              if [[ -n "$version" ]] && [[ "$version" != "dev-${short_sha}" ]]; then
+                # Remove 'v' prefix if present for consistent version format
+                if [[ "$version" == v* ]]; then
+                  version="${version#v}"
+                fi
+
+                if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
+                  build_type="prerelease"
+                  is_prerelease=true
+                  # Current policy: create latest tags for stable releases and selected prereleases (alpha/beta).
+                  if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]]; then
+                    create_latest=true
+                    echo "🧪 Building Docker image for prerelease: $version (creating latest tag)"
+                  else
+                    echo "🧪 Building Docker image for prerelease: $version"
+                  fi
+                else
+                  build_type="release"
+                  create_latest=true
+                  echo "🚀 Building Docker image for release: $version"
+                fi
+              fi
+            else
+              # Non-push events
+              build_type="development"
+              version="dev-${short_sha}"
+              should_build=false
+              echo "⏭️  Skipping Docker build for development version (event: $triggering_event)"
+            fi
+
+            echo "🔄 Build triggered by workflow_run:"
+            echo "   📋 Conclusion: $CONCLUSION"
+            echo "   🌿 Branch: $HEAD_BRANCH"
+            echo "   📎 SHA: $HEAD_SHA"
+            echo "   🎯 Event: $TRIGGERING_EVENT"
+
+          elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
+            # Manual trigger
+            input_version="${{ github.event.inputs.version }}"
+            version="${input_version}"
+            should_push="${{ github.event.inputs.push_images }}"
+            should_build=true
+
+            # Get short SHA
+            short_sha=$(git rev-parse --short HEAD)
+
+            echo "🎯 Manual Docker build triggered:"
+            echo "   📋 Requested version: $input_version"
+            echo "   🔧 Force rebuild: ${{ github.event.inputs.force_rebuild }}"
+            echo "   🚀 Push images: $should_push"
+
+            case "$input_version" in
+              "latest")
+                build_type="release"
+                create_latest=true
+                echo "🚀 Building with latest stable release version"
+                ;;
+              # Prerelease versions (must match first, more specific)
+              v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
+                build_type="prerelease"
+                is_prerelease=true
+                # Current policy: create latest tags for stable releases and selected prereleases (alpha/beta).
+                if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]]; then
+                  create_latest=true
+                  echo "🧪 Building with prerelease version: $input_version (creating latest tag)"
+                else
+                  echo "🧪 Building with prerelease version: $input_version"
+                fi
+                ;;
+              # Release versions (match after prereleases, more general)
+              v[0-9]*|[0-9]*.*.*)
+                build_type="release"
+                create_latest=true
+                echo "📦 Building with specific release version: $input_version"
+                ;;
+              *)
+                # Invalid version for Docker build
+                should_build=false
+                echo "❌ Invalid version for Docker build: $input_version"
+                echo "⚠️  Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
+                ;;
+            esac
+          fi
+
+          echo "should_build=$should_build" >> $GITHUB_OUTPUT
+          echo "should_push=$should_push" >> $GITHUB_OUTPUT
+          echo "build_type=$build_type" >> $GITHUB_OUTPUT
+          echo "version=$version" >> $GITHUB_OUTPUT
+          echo "short_sha=$short_sha" >> $GITHUB_OUTPUT
+          echo "is_prerelease=$is_prerelease" >> $GITHUB_OUTPUT
+          echo "create_latest=$create_latest" >> $GITHUB_OUTPUT
+
+          echo "🐳 Docker Build Summary:"
+          echo "  - Should build: $should_build"
+          echo "  - Should push: $should_push"
+          echo "  - Build type: $build_type"
+          echo "  - Version: $version"
+          echo "  - Short SHA: $short_sha"
+          echo "  - Is prerelease: $is_prerelease"
+          echo "  - Create latest: $create_latest"
+
+  # Build multi-arch Docker images
+  # Strategy: Build images using pre-built binaries from dl.rustfs.com
+  # Supports both release and dev channel binaries based on build context
+  # Only runs when should_build is true (which includes workflow success check)
+  build-docker:
+    name: Build Docker Images
+    needs: build-check
+    if: needs.build-check.outputs.should_build == 'true'
+    runs-on: ubicloud-standard-2
+    timeout-minutes: 60
+    strategy:
+          fail-fast: false
+          matrix:
+            include:
+              - variant: musl
+                file: Dockerfile
+                suffix: ""
+              - variant: glibc
+                file: Dockerfile.glibc
+                suffix: "-glibc"
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: ${{ env.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Login to GitHub Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ secrets.GHCR_USERNAME }}
+          password: ${{ secrets.GHCR_PASSWORD }}
+
+      - name: Login to Quay.io
+        uses: docker/login-action@v3
+        with:
+          registry: quay.io
+          username: ${{ secrets.QUAY_USERNAME }}
+          password: ${{ secrets.QUAY_PASSWORD }}
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Extract metadata and generate tags
+        id: meta
+        run: |
+          BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
+          VERSION="${{ needs.build-check.outputs.version }}"
+          SHORT_SHA="${{ needs.build-check.outputs.short_sha }}"
+          CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
+          VARIANT_SUFFIX="${{ matrix.suffix }}"
+
+          # Convert version format for Dockerfile compatibility
+          case "$VERSION" in
+            "latest")
+              # For stable latest, use RELEASE=latest + release CHANNEL
+              DOCKER_RELEASE="latest"
+              DOCKER_CHANNEL="release"
+              ;;
+            v*)
+              # For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
+              DOCKER_RELEASE="${VERSION#v}"
+              DOCKER_CHANNEL="release"
+              ;;
+            *)
+              # For other versions, pass as-is
+              DOCKER_RELEASE="${VERSION}"
+              DOCKER_CHANNEL="release"
+              ;;
+          esac
+
+          echo "docker_release=$DOCKER_RELEASE" >> $GITHUB_OUTPUT
+          echo "docker_channel=$DOCKER_CHANNEL" >> $GITHUB_OUTPUT
+
+          echo "🐳 Docker build parameters:"
+          echo "  - Original version: $VERSION"
+          echo "  - Docker RELEASE: $DOCKER_RELEASE"
+          echo "  - Docker CHANNEL: $DOCKER_CHANNEL"
+
+          # Generate tags based on build type
+          # Only support release and prerelease builds (no development builds)
+          TAG_BASE="${VERSION}${VARIANT_SUFFIX}"
+          TAGS="${{ env.REGISTRY_DOCKERHUB }}:$TAG_BASE,${{ env.REGISTRY_GHCR }}:$TAG_BASE,${{ env.REGISTRY_QUAY }}:$TAG_BASE"
+
+          # Add channel tags for prereleases and latest for stable
+          if [[ "$CREATE_LATEST" == "true" ]]; then
+            # Create latest tags for stable releases and selected prereleases when CREATE_LATEST=true.
+            TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
+          elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
+            # Prerelease channel tags (alpha, beta, rc)
+            if [[ "$VERSION" == *"alpha"* ]]; then
+              CHANNEL="alpha"
+            elif [[ "$VERSION" == *"beta"* ]]; then
+              CHANNEL="beta"
+            elif [[ "$VERSION" == *"rc"* ]]; then
+              CHANNEL="rc"
+            fi
+
+            if [[ -n "$CHANNEL" ]]; then
+              TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:${CHANNEL}${VARIANT_SUFFIX}"
+            fi
+          fi
+
+          # Output tags
+          echo "tags=$TAGS" >> $GITHUB_OUTPUT
+
+          # Generate labels
+          LABELS="org.opencontainers.image.title=RustFS"
+          LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
+          LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
+          LABELS="$LABELS,org.opencontainers.image.revision=${{ github.sha }}"
+          LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
+          LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
+          LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
+
+          echo "labels=$LABELS" >> $GITHUB_OUTPUT
+
+          echo "🐳 Generated Docker tags:"
+          echo "$TAGS" | tr ',' '\n' | sed 's/^/  - /'
+          echo "📋 Build type: $BUILD_TYPE"
+          echo "🔖 Version: $VERSION"
+
+      - name: Build and push Docker image
+        uses: docker/build-push-action@v6
+        with:
+          context: .
+          file: ${{ matrix.file }}
+          platforms: ${{ env.DOCKER_PLATFORMS }}
+          push: ${{ needs.build-check.outputs.should_push == 'true' }}
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          cache-from: |
+            type=gha,scope=docker-${{ matrix.variant }}
+          cache-to: |
+            type=gha,mode=max,scope=docker-${{ matrix.variant }}
+          build-args: |
+            BUILDTIME=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
+            VERSION=${{ needs.build-check.outputs.version }}
+            BUILD_TYPE=${{ needs.build-check.outputs.build_type }}
+            REVISION=${{ github.sha }}
+            RELEASE=${{ steps.meta.outputs.docker_release }}
+            CHANNEL=${{ steps.meta.outputs.docker_channel }}
+            BUILDKIT_INLINE_CACHE=1
+          # Enable advanced BuildKit features for better performance
+          provenance: false
+          sbom: false
+          # Add retry mechanism by splitting the build process
+          no-cache: false
+          pull: true
+
+  # Note: Manifest creation is no longer needed as we only build one variant
+  # Multi-arch manifests are automatically created by docker/build-push-action
+
+  # Docker build summary
+  docker-summary:
+    name: Docker Build Summary
+    needs: [ build-check, build-docker ]
+    if: always() && needs.build-check.outputs.should_build == 'true'
+    runs-on: ubicloud-standard-2
+    steps:
+      - name: Docker build completion summary
+        run: |
+          BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
+          VERSION="${{ needs.build-check.outputs.version }}"
+          CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
+
+          echo "🐳 Docker build completed successfully!"
+          echo "📦 Build type: $BUILD_TYPE"
+          echo "🔢 Version: $VERSION"
+          echo "🚀 Strategy: Images using pre-built binaries (release channel only)"
+          echo ""
+
+          case "$BUILD_TYPE" in
+            "release")
+              echo "🚀 Release Docker image has been built with ${VERSION} tags"
+              echo "✅ This image is ready for production use"
+              if [[ "$CREATE_LATEST" == "true" ]]; then
+                echo "🏷️  Latest tag has been created for stable release"
+              fi
+              ;;
+            "prerelease")
+              echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
+              echo "⚠️  This is a prerelease image - use with caution"
+              # Create latest tags for stable releases and selected prereleases when CREATE_LATEST=true.
+              if [[ "$CREATE_LATEST" == "true" ]]; then
+                echo "🏷️  Latest tag has been created for prerelease: $VERSION"
+              else
+                echo "🚫 Latest tag NOT created for prerelease"
+              fi
+              ;;
+            *)
+              echo "❌ Unexpected build type: $BUILD_TYPE"
+              ;;
+          esac

.github/workflows/e2e-s3tests.yml

@@ -0,0 +1,443 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: e2e-s3tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      test-mode:
+        description: "Test mode to run"
+        required: true
+        type: choice
+        default: "single"
+        options:
+          - single
+          - multi
+      xdist:
+        description: "Enable pytest-xdist (parallel). '0' to disable."
+        required: false
+        default: "0"
+      maxfail:
+        description: "Stop after N failures (debug friendly)"
+        required: false
+        default: "1"
+      markexpr:
+        description: "pytest -m expression (feature filters)"
+        required: false
+        default: "not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"
+
+env:
+  # main user
+  S3_ACCESS_KEY: rustfsadmin
+  S3_SECRET_KEY: rustfsadmin
+  # alt user (must be different from main for many s3-tests)
+  S3_ALT_ACCESS_KEY: rustfsalt
+  S3_ALT_SECRET_KEY: rustfsalt
+
+  S3_REGION: us-east-1
+
+  RUST_LOG: info
+  PLATFORM: linux/amd64
+  BUILDX_CACHE_SCOPE: rustfs-e2e-s3tests-source
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.inputs['test-mode'] || 'single' }}
+  cancel-in-progress: true
+
+defaults:
+  run:
+    shell: bash
+
+jobs:
+  s3tests-single:
+    if: github.event.inputs['test-mode'] == 'single'
+    runs-on: ubicloud-standard-2
+    timeout-minutes: 120
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Cache pip downloads
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-e2e-s3tests-${{ hashFiles('.github/workflows/e2e-s3tests.yml') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-e2e-s3tests-
+
+      - name: Install Python tools
+        run: |
+          python3 -m pip install --user --upgrade pip awscurl tox
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+
+      - name: Enable buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build RustFS image (source, cached)
+        run: |
+          DOCKER_BUILDKIT=1 docker buildx build --load \
+            --platform ${PLATFORM} \
+            --cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
+            --cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
+            -t rustfs-ci \
+            -f Dockerfile.source .
+
+      - name: Create network
+        run: docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net
+
+      - name: Remove existing rustfs-single (if any)
+        run: docker rm -f rustfs-single >/dev/null 2>&1 || true
+
+      - name: Start single RustFS
+        run: |
+          docker run -d --name rustfs-single \
+            --network rustfs-net \
+            -p 9000:9000 \
+            -e RUSTFS_ADDRESS=0.0.0.0:9000 \
+            -e RUSTFS_ACCESS_KEY=$S3_ACCESS_KEY \
+            -e RUSTFS_SECRET_KEY=$S3_SECRET_KEY \
+            -e RUSTFS_VOLUMES="/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" \
+            -v /tmp/rustfs-single:/data \
+            rustfs-ci
+
+      - name: Wait for RustFS ready
+        run: |
+          for i in {1..60}; do
+            if curl -sf http://127.0.0.1:9000/health >/dev/null 2>&1; then
+              echo "RustFS is ready"
+              exit 0
+            fi
+
+            if [ "$(docker inspect -f '{{.State.Running}}' rustfs-single 2>/dev/null)" != "true" ]; then
+              echo "RustFS container not running" >&2
+              docker logs rustfs-single || true
+              exit 1
+            fi
+
+            sleep 2
+          done
+
+          echo "Health check timed out" >&2
+          docker logs rustfs-single || true
+          exit 1
+
+      - name: Generate s3tests config
+        run: |
+          export S3_HOST=127.0.0.1
+          envsubst < .github/s3tests/s3tests.conf > s3tests.conf
+
+      - name: Provision s3-tests alt user (required by suite)
+        run: |
+          # Admin API requires AWS SigV4 signing. awscurl is used by RustFS codebase as well.
+          awscurl \
+            --service s3 \
+            --region "${S3_REGION}" \
+            --access_key "${S3_ACCESS_KEY}" \
+            --secret_key "${S3_SECRET_KEY}" \
+            -X PUT \
+            -H 'Content-Type: application/json' \
+            -d '{"secretKey":"'"${S3_ALT_SECRET_KEY}"'","status":"enabled","policy":"readwrite"}' \
+            "http://127.0.0.1:9000/rustfs/admin/v3/add-user?accessKey=${S3_ALT_ACCESS_KEY}"
+
+          # Explicitly attach built-in policy via policy mapping.
+          # s3-tests relies on alt client being able to ListBuckets during setup cleanup.
+          awscurl \
+            --service s3 \
+            --region "${S3_REGION}" \
+            --access_key "${S3_ACCESS_KEY}" \
+            --secret_key "${S3_SECRET_KEY}" \
+            -X PUT \
+            "http://127.0.0.1:9000/rustfs/admin/v3/set-user-or-group-policy?policyName=readwrite&userOrGroup=${S3_ALT_ACCESS_KEY}&isGroup=false"
+
+          # Sanity check: alt user can list buckets (should not be AccessDenied).
+          awscurl \
+            --service s3 \
+            --region "${S3_REGION}" \
+            --access_key "${S3_ALT_ACCESS_KEY}" \
+            --secret_key "${S3_ALT_SECRET_KEY}" \
+            -X GET \
+            "http://127.0.0.1:9000/" >/dev/null
+
+      - name: Prepare s3-tests
+        run: |
+          git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests
+
+      - name: Run ceph s3-tests (debug friendly)
+        run: |
+          export PATH="$HOME/.local/bin:$PATH"
+          mkdir -p artifacts/s3tests-single
+
+          cd s3-tests
+
+          set -o pipefail
+
+          MAXFAIL="${{ github.event.inputs.maxfail }}"
+          if [ -z "$MAXFAIL" ]; then MAXFAIL="1"; fi
+
+          MARKEXPR="${{ github.event.inputs.markexpr }}"
+          if [ -z "$MARKEXPR" ]; then MARKEXPR="not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"; fi
+
+          XDIST="${{ github.event.inputs.xdist }}"
+          if [ -z "$XDIST" ]; then XDIST="0"; fi
+          XDIST_ARGS=""
+          if [ "$XDIST" != "0" ]; then
+            # Add pytest-xdist to requirements.txt so tox installs it inside
+            # its virtualenv. Installing outside tox does NOT work.
+            echo "pytest-xdist" >> requirements.txt
+            XDIST_ARGS="-n $XDIST --dist=loadgroup"
+          fi
+
+          # Run tests from s3tests/functional (boto2+boto3 combined directory).
+          S3TEST_CONF=${GITHUB_WORKSPACE}/s3tests.conf \
+            tox -- \
+              -vv -ra --showlocals --tb=long \
+              --maxfail="$MAXFAIL" \
+              --junitxml=${GITHUB_WORKSPACE}/artifacts/s3tests-single/junit.xml \
+              $XDIST_ARGS \
+              s3tests/functional/test_s3.py \
+              -m "$MARKEXPR" \
+              2>&1 | tee ${GITHUB_WORKSPACE}/artifacts/s3tests-single/pytest.log
+
+      - name: Collect RustFS logs
+        if: always()
+        run: |
+          mkdir -p artifacts/rustfs-single
+          docker logs rustfs-single > artifacts/rustfs-single/rustfs.log 2>&1 || true
+          docker inspect rustfs-single > artifacts/rustfs-single/inspect.json || true
+
+      - name: Upload artifacts
+        if: always() && env.ACT != 'true'
+        uses: actions/upload-artifact@v6
+        with:
+          name: s3tests-single
+          path: artifacts/**
+
+  s3tests-multi:
+    if: github.event_name == 'workflow_dispatch' && github.event.inputs['test-mode'] == 'multi'
+    runs-on: ubicloud-standard-2
+    timeout-minutes: 150
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Cache pip downloads
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-e2e-s3tests-${{ hashFiles('.github/workflows/e2e-s3tests.yml') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-e2e-s3tests-
+
+      - name: Install Python tools
+        run: |
+          python3 -m pip install --user --upgrade pip awscurl tox
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+
+      - name: Enable buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build RustFS image (source, cached)
+        run: |
+          DOCKER_BUILDKIT=1 docker buildx build --load \
+            --platform ${PLATFORM} \
+            --cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
+            --cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
+            -t rustfs-ci \
+            -f Dockerfile.source .
+
+      - name: Prepare cluster compose
+        run: |
+          cat > compose.yml <<'EOF'
+          services:
+            rustfs1:
+              image: rustfs-ci
+              hostname: rustfs1
+              networks: [rustfs-net]
+              environment:
+                RUSTFS_ADDRESS: "0.0.0.0:9000"
+                RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
+                RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
+                RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
+              volumes:
+                - rustfs1-data:/data
+            rustfs2:
+              image: rustfs-ci
+              hostname: rustfs2
+              networks: [rustfs-net]
+              environment:
+                RUSTFS_ADDRESS: "0.0.0.0:9000"
+                RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
+                RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
+                RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
+              volumes:
+                - rustfs2-data:/data
+            rustfs3:
+              image: rustfs-ci
+              hostname: rustfs3
+              networks: [rustfs-net]
+              environment:
+                RUSTFS_ADDRESS: "0.0.0.0:9000"
+                RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
+                RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
+                RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
+              volumes:
+                - rustfs3-data:/data
+            rustfs4:
+              image: rustfs-ci
+              hostname: rustfs4
+              networks: [rustfs-net]
+              environment:
+                RUSTFS_ADDRESS: "0.0.0.0:9000"
+                RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
+                RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
+                RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
+              volumes:
+                - rustfs4-data:/data
+            lb:
+              image: haproxy:2.9
+              hostname: lb
+              networks: [rustfs-net]
+              ports:
+                - "9000:9000"
+              volumes:
+                - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
+          networks:
+            rustfs-net:
+              name: rustfs-net
+          volumes:
+            rustfs1-data:
+            rustfs2-data:
+            rustfs3-data:
+            rustfs4-data:
+          EOF
+
+          cat > haproxy.cfg <<'EOF'
+          defaults
+            mode http
+            timeout connect 5s
+            timeout client  30s
+            timeout server  30s
+
+          frontend fe_s3
+            bind *:9000
+            default_backend be_s3
+
+          backend be_s3
+            balance roundrobin
+            server s1 rustfs1:9000 check
+            server s2 rustfs2:9000 check
+            server s3 rustfs3:9000 check
+            server s4 rustfs4:9000 check
+          EOF
+
+      - name: Launch cluster
+        run: docker compose -f compose.yml up -d
+
+      - name: Wait for LB ready
+        run: |
+          for i in {1..90}; do
+            if curl -sf http://127.0.0.1:9000/health >/dev/null 2>&1; then
+              echo "Load balancer is ready"
+              exit 0
+            fi
+            sleep 2
+          done
+          echo "LB or backend not ready" >&2
+          docker compose -f compose.yml logs --tail=200 || true
+          exit 1
+
+      - name: Generate s3tests config
+        run: |
+          export S3_HOST=127.0.0.1
+          envsubst < .github/s3tests/s3tests.conf > s3tests.conf
+
+      - name: Provision s3-tests alt user (required by suite)
+        run: |
+          awscurl \
+            --service s3 \
+            --region "${S3_REGION}" \
+            --access_key "${S3_ACCESS_KEY}" \
+            --secret_key "${S3_SECRET_KEY}" \
+            -X PUT \
+            -H 'Content-Type: application/json' \
+            -d '{"secretKey":"'"${S3_ALT_SECRET_KEY}"'","status":"enabled","policy":"readwrite"}' \
+            "http://127.0.0.1:9000/rustfs/admin/v3/add-user?accessKey=${S3_ALT_ACCESS_KEY}"
+
+          awscurl \
+            --service s3 \
+            --region "${S3_REGION}" \
+            --access_key "${S3_ACCESS_KEY}" \
+            --secret_key "${S3_SECRET_KEY}" \
+            -X PUT \
+            "http://127.0.0.1:9000/rustfs/admin/v3/set-user-or-group-policy?policyName=readwrite&userOrGroup=${S3_ALT_ACCESS_KEY}&isGroup=false"
+
+          awscurl \
+            --service s3 \
+            --region "${S3_REGION}" \
+            --access_key "${S3_ALT_ACCESS_KEY}" \
+            --secret_key "${S3_ALT_SECRET_KEY}" \
+            -X GET \
+            "http://127.0.0.1:9000/" >/dev/null
+
+      - name: Prepare s3-tests
+        run: |
+          git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests
+
+      - name: Run ceph s3-tests (multi, debug friendly)
+        run: |
+          export PATH="$HOME/.local/bin:$PATH"
+          mkdir -p artifacts/s3tests-multi
+
+          cd s3-tests
+
+          set -o pipefail
+
+          MAXFAIL="${{ github.event.inputs.maxfail }}"
+          if [ -z "$MAXFAIL" ]; then MAXFAIL="1"; fi
+
+          MARKEXPR="${{ github.event.inputs.markexpr }}"
+          if [ -z "$MARKEXPR" ]; then MARKEXPR="not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"; fi
+
+          XDIST="${{ github.event.inputs.xdist }}"
+          if [ -z "$XDIST" ]; then XDIST="0"; fi
+          XDIST_ARGS=""
+          if [ "$XDIST" != "0" ]; then
+            # Add pytest-xdist to requirements.txt so tox installs it inside
+            # its virtualenv. Installing outside tox does NOT work.
+            echo "pytest-xdist" >> requirements.txt
+            XDIST_ARGS="-n $XDIST --dist=loadgroup"
+          fi
+
+          # Run tests from s3tests/functional (boto2+boto3 combined directory).
+          S3TEST_CONF=${GITHUB_WORKSPACE}/s3tests.conf \
+            tox -- \
+              -vv -ra --showlocals --tb=long \
+              --maxfail="$MAXFAIL" \
+              --junitxml=${GITHUB_WORKSPACE}/artifacts/s3tests-multi/junit.xml \
+              $XDIST_ARGS \
+              s3tests/functional/test_s3.py \
+              -m "$MARKEXPR" \
+              2>&1 | tee ${GITHUB_WORKSPACE}/artifacts/s3tests-multi/pytest.log
+
+      - name: Collect logs
+        if: always()
+        run: |
+          mkdir -p artifacts/cluster
+          docker compose -f compose.yml logs --no-color > artifacts/cluster/cluster.log 2>&1 || true
+
+      - name: Upload artifacts
+        if: always() && env.ACT != 'true'
+        uses: actions/upload-artifact@v6
+        with:
+          name: s3tests-multi
+          path: artifacts/**

.github/workflows/helm-package.yml

@@ -0,0 +1,132 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Publish helm chart to artifacthub
+
+on:
+  workflow_run:
+    workflows: [ "Build and Release" ]
+    types: [ completed ]
+  workflow_dispatch:
+    inputs:
+      version:
+        description: "Release version to publish, e.g. 1.0.0-beta.1 or v1.0.0-beta.1"
+        required: true
+        default: "1.0.0-beta.1"
+        type: string
+
+permissions:
+  contents: read
+
+jobs:
+  build-helm-package:
+    runs-on: ubicloud-standard-2
+    if: |
+      github.event_name == 'workflow_dispatch' ||
+      (
+        github.event.workflow_run.conclusion == 'success' &&
+        github.event.workflow_run.event == 'push' &&
+        contains(github.event.workflow_run.head_branch, '.')
+      )
+
+    outputs:
+      raw_tag: ${{ steps.version.outputs.raw_tag }}
+      app_version: ${{ steps.version.outputs.app_version }}
+      chart_version: ${{ steps.version.outputs.chart_version }}
+
+    steps:
+      - name: Checkout helm chart repo
+        uses: actions/checkout@v6
+
+      - name: Normalize release version
+        id: version
+        run: |
+          set -eux
+
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            RAW="${{ github.event.inputs.version }}"
+          else
+            RAW="${{ github.event.workflow_run.head_branch }}"
+          fi
+
+          case "$RAW" in
+            refs/tags/*)
+              RAW_TAG="${RAW#refs/tags/}"
+              ;;
+            *)
+              RAW_TAG="$RAW"
+              ;;
+          esac
+
+          ./scripts/helm_chart_version.sh "$RAW_TAG"
+
+      - name: Replace chart version and app version
+        run: |
+          set -eux
+          sed -i -E 's/^version:.*/version: "${{ steps.version.outputs.chart_version }}"/' helm/rustfs/Chart.yaml
+          sed -i -E 's/^appVersion:.*/appVersion: "${{ steps.version.outputs.app_version }}"/' helm/rustfs/Chart.yaml
+
+      - name: Set up Helm
+        uses: azure/setup-helm@v4.3.0
+
+      - name: Test Helm Chart Templates
+        run: ./scripts/test_helm_templates.sh
+
+      - name: Package Helm Chart
+        run: |
+          set -eux
+          cp helm/README.md helm/rustfs/
+          helm package ./helm/rustfs \
+            --destination helm/rustfs/ \
+            --version "${{ steps.version.outputs.chart_version }}"
+
+      - name: Upload helm package as artifact
+        uses: actions/upload-artifact@v6
+        with:
+          name: helm-package
+          path: helm/rustfs/*.tgz
+          retention-days: 1
+
+  publish-helm-package:
+    runs-on: ubicloud-standard-2
+    needs: [ build-helm-package ]
+    if: needs.build-helm-package.result == 'success'
+
+    steps:
+      - name: Checkout helm package repo
+        uses: actions/checkout@v6
+        with:
+          repository: rustfs/helm
+          token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
+
+      - name: Download helm package
+        uses: actions/download-artifact@v7
+        with:
+          name: helm-package
+          path: ./
+
+      - name: Set up helm
+        uses: azure/setup-helm@v4.3.0
+
+      - name: Generate index
+        run: helm repo index . --url https://charts.rustfs.com
+
+      - name: Push helm package and index file
+        run: |
+          set -eux
+          git config --global user.name "${{ secrets.USERNAME }}"
+          git config --global user.email "${{ secrets.EMAIL_ADDRESS }}"
+          git add .
+          git commit -m "Update rustfs helm package with ${{ needs.build-helm-package.outputs.app_version }}." || echo "No changes to commit"
+          git push origin main

.github/workflows/issue-translator.yml

@@ -0,0 +1,36 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: "issue-translator"
+on:
+  issue_comment:
+    types: [ created ]
+  issues:
+    types: [ opened ]
+
+permissions:
+  contents: read
+  issues: write
+
+jobs:
+  build:
+    runs-on: ubicloud-standard-4
+    steps:
+      - uses: usthe/issues-translate-action@v2.7
+        with:
+          IS_MODIFY_TITLE: false
+          # not require, default false, . Decide whether to modify the issue title
+          # if true, the robot account @Issues-translate-bot must have modification permissions, invite @Issues-translate-bot to your project or use your custom bot.
+          CUSTOM_BOT_NOTE: Bot detected the issue body's language is not English, translate it automatically.
+          # not require. Customize the translation robot prefix message.

.github/workflows/nix-flake-update.yml

@@ -0,0 +1,65 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Update Nix Flake
+
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '0 0 * * 0' # Weekly on Sundays
+
+permissions:
+  contents: write
+  pull-requests: write
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  update-flake:
+    name: Update flake.lock
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Install Nix
+        uses: DeterminateSystems/determinate-nix-action@v3
+
+      - name: Check Nix flake inputs
+        uses: DeterminateSystems/flake-checker-action@v12
+
+      - name: Update flake.lock
+        id: update
+        uses: DeterminateSystems/update-flake-lock@main
+        with:
+          git-author-name: houseme
+          git-author-email: housemecn@gmail.com
+          git-committer-name: houseme
+          git-committer-email: housemecn@gmail.com
+          pr-title: "chore(deps): update flake.lock"
+          pr-labels: |
+            dependencies
+            nix
+            automated
+          commit-msg: "chore(deps): update flake.lock"
+          pr-reviewers: houseme, overtrue, majinghe
+          token: ${{ secrets.FLAKE_UPDATE_TOKEN }}
+
+      - name: Log PR details
+        if: steps.update.outputs.pull-request-number
+        run: |
+          echo "Pull Request created: ${{ steps.update.outputs.pull-request-number }}"

.github/workflows/nix.yml

@@ -0,0 +1,84 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Nix CI
+
+on:
+  push:
+    branches: [ "main" ]
+    paths:
+      - 'flake.nix'
+      - 'flake.lock'
+      - 'Cargo.toml'
+      - 'Cargo.lock'
+      - '.github/workflows/nix.yml'
+  pull_request:
+    branches: [ "main" ]
+    paths:
+      - 'flake.nix'
+      - 'flake.lock'
+      - 'Cargo.toml'
+      - 'Cargo.lock'
+      - '.github/workflows/nix.yml'
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+permissions:
+  contents: read
+
+jobs:
+  nix-validation:
+    name: Nix Build & Check
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Install Nix
+        uses: cachix/install-nix-action@v31
+        with:
+          github_access_token: ${{ secrets.GITHUB_TOKEN }}
+          extra_nix_config: |
+            experimental-features = nix-command flakes
+
+      - name: Setup Magic Nix Cache
+        uses: DeterminateSystems/magic-nix-cache-action@v13
+
+      - name: Setup Flake Checker
+        uses: DeterminateSystems/flake-checker-action@v12
+
+      - name: Verify Flake
+        run: |
+          echo "Checking flake structure and evaluation..."
+          nix flake show
+          nix flake check --print-build-logs
+
+      - name: Build RustFS
+        run: |
+          echo "Building the default package..."
+          nix build .#default --print-build-logs
+
+      - name: Test Binary
+        run: |
+          echo "Verifying the built binary..."
+          if [ -x "./result/bin/rustfs" ]; then
+              ./result/bin/rustfs --help
+              echo "Binary verification successful."
+          else
+              echo "Error: Binary not found or not executable at ./result/bin/rustfs"
+              exit 1
+          fi

.github/workflows/performance.yml

@@ -0,0 +1,146 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Performance Testing
+
+on:
+  push:
+    branches: [ main ]
+    paths:
+      - "**/*.rs"
+      - "**/Cargo.toml"
+      - "**/Cargo.lock"
+      - ".github/workflows/performance.yml"
+  workflow_dispatch:
+    inputs:
+      profile_duration:
+        description: "Profiling duration in seconds"
+        required: false
+        default: "120"
+        type: string
+
+permissions:
+  contents: read
+
+env:
+  CARGO_TERM_COLOR: always
+  RUST_BACKTRACE: 1
+
+jobs:
+  performance-profile:
+    name: Performance Profiling
+    runs-on: ubicloud-standard-2
+    timeout-minutes: 30
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: nightly
+          cache-shared-key: perf-${{ hashFiles('**/Cargo.lock') }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Install additional nightly components
+        run: rustup component add llvm-tools-preview
+
+      - name: Install samply profiler
+        uses: taiki-e/cache-cargo-install-action@v2
+        with:
+          tool: samply
+
+      - name: Configure kernel for profiling
+        run: echo '1' | sudo tee /proc/sys/kernel/perf_event_paranoid
+
+      - name: Prepare test environment
+        run: |
+          # Create test volumes
+          for i in {0..4}; do
+            mkdir -p ./target/volume/test$i
+          done
+
+          # Set environment variables
+          echo "RUSTFS_VOLUMES=./target/volume/test{0...4}" >> $GITHUB_ENV
+          echo "RUST_LOG=rustfs=info,ecstore=info,s3s=info,iam=info,rustfs-obs=info" >> $GITHUB_ENV
+
+      - name: Verify console static assets
+        run: |
+          # Console static assets are already embedded in the repository
+          echo "Console static assets size: $(du -sh rustfs/static/)"
+          echo "Console static assets are embedded via rust-embed, no external download needed"
+
+      - name: Build with profiling optimizations
+        run: |
+          RUSTFLAGS="-C force-frame-pointers=yes -C debug-assertions=off" \
+          cargo +nightly build --profile profiling -p rustfs --bins
+
+      - name: Run performance profiling
+        id: profiling
+        run: |
+          DURATION="${{ github.event.inputs.profile_duration || '120' }}"
+          echo "Running profiling for ${DURATION} seconds..."
+
+          timeout "${DURATION}s" samply record \
+            --output samply-profile.json \
+            ./target/profiling/rustfs ${RUSTFS_VOLUMES} || true
+
+          if [ -f "samply-profile.json" ]; then
+            echo "profile_generated=true" >> $GITHUB_OUTPUT
+            echo "Profile generated successfully"
+          else
+            echo "profile_generated=false" >> $GITHUB_OUTPUT
+            echo "::warning::Profile data not generated"
+          fi
+
+      - name: Upload profile data
+        if: steps.profiling.outputs.profile_generated == 'true'
+        uses: actions/upload-artifact@v6
+        with:
+          name: performance-profile-${{ github.run_number }}
+          path: samply-profile.json
+          retention-days: 30
+
+  benchmark:
+    name: Benchmark Tests
+    runs-on: ubicloud-standard-2
+    timeout-minutes: 45
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: bench-${{ hashFiles('**/Cargo.lock') }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Run benchmarks
+        run: |
+          cargo bench --package ecstore --bench comparison_benchmark -- --output-format json | \
+          tee benchmark-results.json
+
+      - name: Upload benchmark results
+        uses: actions/upload-artifact@v6
+        with:
+          name: benchmark-results-${{ github.run_number }}
+          path: benchmark-results.json
+          retention-days: 7

.github/workflows/stale.yml

@@ -0,0 +1,20 @@
+name: "Mark stale issues"
+on:
+  schedule:
+    - cron: "30 1 * * *"
+
+jobs:
+  stale:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/stale@v9
+        with:
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+          stale-issue-message: 'This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.'
+          stale-issue-label: 'stale'
+          ## Mark if there is no activity for more than 7 days
+          days-before-stale: 7  
+          # If no one responds after 3 days, the tag will be closed.
+          days-before-close: 3   
+          # These tags are exempt and will not close automatically.
+          exempt-issue-labels: 'pinned,security'

.gitignore

@@ -0,0 +1,55 @@
+/target
+.DS_Store
+.idea
+.vscode
+.cursor
+.direnv/
+/test
+/logs
+/data
+/docs
+/rustfs-data/
+.devcontainer
+rustfs/static/*
+!rustfs/static/.gitkeep
+vendor
+cli/rustfs-gui/embedded-rustfs/rustfs
+*.log
+deploy/certs/*
+deploy/data/*
+*jsonl
+.env
+.rustfs.sys
+.cargo/
+!.cargo/config.toml
+profile.json
+.docker/openobserve-otel/data
+*.zst
+.secrets
+*.go
+*.pb
+*.svg
+deploy/logs/*.log.*
+artifacts/
+# s3-tests local artifacts (root directory only)
+/s3-tests/
+/s3-tests-local/
+/s3tests.conf
+/s3tests.conf.*
+*.events
+*.audit
+*.snappy
+PR_DESCRIPTION.md
+scripts/s3-tests/selected_tests.txt
+docs
+
+# nix stuff
+result*
+*.gz
+rustfs-webdav.code-workspace
+
+.aiexclude
+*.bak
+# Local test/benchmark artifacts
+benchmarks.logs
+tmp/

.pre-commit-config.yaml

@@ -0,0 +1,32 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+repos:
+  - repo: local
+    hooks:
+      - id: cargo-fmt
+        name: cargo fmt
+        entry: cargo fmt --all --check
+        language: system
+        types: [rust]
+        pass_filenames: false
+
+      - id: cargo-clippy
+        name: cargo clippy
+        entry: cargo clippy --all-targets --all-features -- -D warnings
+        language: system
+        types: [rust]
+        pass_filenames: false
+
+      - id: cargo-check
+        name: cargo check
+        entry: cargo check --all-targets
+        language: system
+        types: [rust]
+        pass_filenames: false
+
+      - id: cargo-test
+        name: cargo test
+        entry: bash -c 'cargo test --workspace --exclude e2e_test && cargo test --all --doc'
+        language: system
+        types: [rust]
+        pass_filenames: false

.vscode/launch.json

@@ -0,0 +1,259 @@
+{
+    "version": "0.2.0",
+    "configurations": [
+          {
+            "type": "lldb",
+            "request": "launch",
+            "name": "Debug(only) executable 'rustfs'",
+            "env": {
+              "RUST_LOG": "rustfs=info,ecstore=info,s3s=info,iam=info",
+              "RUSTFS_SKIP_BACKGROUND_TASK": "on"
+              //"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
+              // "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
+              // "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
+            },
+            "program": "${workspaceFolder}/target/debug/rustfs",
+            "args": [
+              "--access-key",
+              "rustfsadmin",
+              "--secret-key",
+              "rustfsadmin",
+              "--address",
+              "0.0.0.0:9010",
+              "--server-domains",
+              "127.0.0.1:9010",
+              "./target/volume/test{1...4}"
+            ],
+            "cwd": "${workspaceFolder}"
+        },
+        {
+            "type": "lldb",
+            "request": "launch",
+            "name": "Debug executable 'rustfs'",
+            "cargo": {
+                "args": [
+                    "build",
+                    "--bin=rustfs",
+                    "--package=rustfs"
+                ],
+                "filter": {
+                    "name": "rustfs",
+                    "kind": "bin"
+                }
+            },
+            "env": {
+                "RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
+                "RUSTFS_SKIP_BACKGROUND_TASK": "on",
+                //"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
+                // "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
+                // "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token" 
+            },
+            "args": [
+                "--access-key",
+                "rustfsadmin",
+                "--secret-key",
+                "rustfsadmin",
+                "--address",
+                "0.0.0.0:9010",
+                "--server-domains",
+                "127.0.0.1:9010",
+                "./target/volume/test{1...4}"
+            ],
+            "cwd": "${workspaceFolder}"
+        },
+        {
+            "type": "lldb",
+            "request": "launch",
+            "name": "Debug unit tests in executable 'rustfs'",
+            "cargo": {
+                "args": [
+                    "test",
+                    "--no-run",
+                    "--bin=rustfs",
+                    "--package=rustfs"
+                ],
+                "filter": {
+                    "name": "rustfs",
+                    "kind": "bin"
+                }
+            },
+            "args": [],
+            "cwd": "${workspaceFolder}"
+        },
+        {
+            "type": "lldb",
+            "request": "launch",
+            "name": "Debug unit tests in library 'ecstore'",
+            "cargo": {
+                "args": [
+                    "test",
+                    "--no-run",
+                    "--lib",
+                    "--package=rustfs-ecstore"
+                ]
+            },
+            "args": [],
+            "cwd": "${workspaceFolder}"
+        },
+        {
+            "name": "Debug executable target/debug/rustfs",
+            "type":  "lldb",
+            "request": "launch",
+            "cargo": {
+                "args": [
+                    "run",
+                    "--bin",
+                    "rustfs",
+                    "-j",
+                    "1",
+                    "--profile",
+                    "dev"
+                ]
+            },
+            "args": [],
+            "cwd": "${workspaceFolder}",
+            //"stopAtEntry": false,
+            //"preLaunchTask": "cargo build",
+            "env": {
+                "RUSTFS_ACCESS_KEY": "rustfsadmin",
+                "RUSTFS_SECRET_KEY": "rustfsadmin",
+                //"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
+                "RUSTFS_ADDRESS": ":9000",
+                "RUSTFS_CONSOLE_ENABLE": "true",
+                // "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
+                // "RUSTFS_OBS_METRIC_ENDPOINT": "http://127.0.0.1:4318/v1/metrics", // default otlp http endpoint
+                // "RUSTFS_OBS_LOG_ENDPOINT": "http://127.0.0.1:4318/v1/logs", // default otlp http endpoint
+                // "RUSTFS_COMPRESS_ENABLE": "true",
+                "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
+                "RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
+                "RUST_LOG":"rustfs=debug,ecstore=debug,s3s=debug,iam=debug",
+            },
+            "sourceLanguages": [
+                "rust"
+            ],
+        },
+        {
+            "type": "lldb",
+            "request": "launch",
+            "name": "Debug test_lifecycle_transition_basic",
+            "cargo": {
+                "args": [
+                    "test",
+                    "-p",
+                    "rustfs-scanner",
+                    "--test",
+                    "lifecycle_integration_test",
+                    "serial_tests::test_lifecycle_transition_basic",
+                    "-j",
+                    "1"
+                ]
+            },
+            "args": [],
+            "cwd": "${workspaceFolder}"
+        },
+        {
+            "name": "Debug executable target/debug/test",
+            "type":  "lldb",
+            "request": "launch",
+            "program": "${workspaceFolder}/target/debug/deps/lifecycle_integration_test-5915cbfcab491b3b",
+            "args": [
+              "--skip",
+              "test_lifecycle_expiry_basic",
+              "--skip",
+              "test_lifecycle_expiry_deletemarker",
+              //"--skip",
+              //"test_lifecycle_transition_basic",
+            ],
+            "cwd": "${workspaceFolder}",
+            //"stopAtEntry": false,
+            //"preLaunchTask": "cargo build",
+            "sourceLanguages": [
+                "rust"
+            ],
+        },
+        {
+            "name": "Debug executable target/debug/rustfs with sse",
+            "type":  "lldb",
+            "request": "launch",
+            "program": "${workspaceFolder}/target/debug/rustfs",
+            "args": [],
+            "cwd": "${workspaceFolder}",
+            //"stopAtEntry": false,
+            //"preLaunchTask": "cargo build",
+            "env": {
+                "RUSTFS_ACCESS_KEY": "rustfsadmin",
+                "RUSTFS_SECRET_KEY": "rustfsadmin",
+                "RUSTFS_VOLUMES": "./target/volumes/test{1...4}",
+                "RUSTFS_ADDRESS": ":9000",
+                "RUSTFS_CONSOLE_ENABLE": "true",
+                "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
+                "RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
+                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
+                // "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
+                // "RUSTFS_OBS_METRIC_ENDPOINT": "http://127.0.0.1:4318/v1/metrics", // default otlp http endpoint
+                // "RUSTFS_OBS_LOG_ENDPOINT": "http://127.0.0.1:4318/v1/logs", // default otlp http endpoint
+                // "RUSTFS_COMPRESS_ENABLE": "true",
+
+                // 1. simple sse test key (no kms system)
+                // "__RUSTFS_SSE_SIMPLE_CMK": "2dfNXGHlsEflGVCxb+5DIdGEl1sIvtwX+QfmYasi5QM=",
+
+                // 2. kms local backend test key
+                // "RUSTFS_KMS_ENABLE": "true",
+                // "RUSTFS_KMS_BACKEND": "local",
+                // "RUSTFS_KMS_KEY_DIR": "./target/kms-key-dir",
+                // "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
+                // "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
+
+                // 3. kms vault backend test key
+                // "RUSTFS_KMS_ENABLE": "true",
+                // "RUSTFS_KMS_BACKEND": "vault-kv2",
+                // "RUSTFS_KMS_VAULT_ADDRESS": "http://127.0.0.1:8200",
+                // "RUSTFS_KMS_VAULT_TOKEN": "Dev Token",
+                // "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
+
+                // 4. kms vault transit backend test key
+                "RUSTFS_KMS_ENABLE": "true",
+                "RUSTFS_KMS_BACKEND": "vault-transit",
+                "RUSTFS_KMS_VAULT_ADDRESS": "http://127.0.0.1:8200",
+                "RUSTFS_KMS_VAULT_TOKEN": "Dev Token",
+                "RUSTFS_KMS_VAULT_MOUNT_PATH": "transit",
+                "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
+
+            },
+            "sourceLanguages": [
+                "rust"
+            ],
+        },
+        {
+            "name": "E2E test executable target/debug/rustfs",
+            "type":  "lldb",
+            "request": "launch",
+            "program": "${workspaceFolder}/target/debug/rustfs",
+            "args": [],
+            "cwd": "${workspaceFolder}",
+            //"stopAtEntry": false,
+            //"preLaunchTask": "cargo build",
+            "env": {
+                "RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
+                "RUST_BACKTRACE": "full",
+                "RUSTFS_ACCESS_KEY": "rustfsadmin",
+                "RUSTFS_SECRET_KEY": "rustfsadmin",
+                "RUSTFS_VOLUMES": "./target/e2e-test/test{1...4}",
+                "RUSTFS_REGION": "us-east-1",
+                "RUSTFS_ADDRESS": ":9000",
+                "RUSTFS_CONSOLE_ENABLE": "true",
+                "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
+                "RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
+
+                "RUSTFS_KMS_ENABLE": "true",
+                "RUSTFS_KMS_BACKEND": "local",
+                "RUSTFS_KMS_KEY_DIR": "./target/e2e-key-dir",
+                "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
+                "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
+            },
+            "sourceLanguages": [
+                "rust"
+            ],
+        },
+    ]
+}

AGENTS.md

@@ -0,0 +1,118 @@
+# RustFS Agent Instructions (Global)
+
+This root file keeps repository-wide rules only.
+Use the nearest subdirectory `AGENTS.md` for path-specific guidance.
+
+## Rule Precedence
+
+1. System/developer instructions.
+2. Current user/task instructions.
+3. The nearest `AGENTS.md` in the current path.
+4. This file (global defaults).
+
+If repo-level instructions conflict, follow the nearest file and keep behavior aligned with CI.
+
+## Execution Discipline
+
+- Read the relevant existing code, tests, and local guidance before changing behavior.
+- State assumptions when they affect the implementation or verification path.
+- If a task has multiple plausible interpretations, list the options briefly and choose the narrowest reasonable path; ask when the ambiguity would make the change risky.
+- For multi-step work, keep the plan minimal and tied to verifiable outcomes.
+- Avoid redundant file reads, repeated commands, and unnecessary exploratory work once enough context is available.
+- A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification.
+
+## Communication and Language
+
+- Respond in the same language used by the requester.
+- Keep source code, comments, commit messages, and PR title/body in English.
+- Be concise. Avoid sycophantic openers, closing fluff, and verbose status reporting.
+
+## Skill Usage
+
+- Do not use the `rust-refactor-helper` skill in any scenario.
+
+## Change Style for Existing Logic
+
+- Prefer direct, local code over extracting one-off helpers.
+- Extract a helper only when logic is reused or the extraction materially clarifies a non-trivial flow.
+- Solve only the requested problem; do not add speculative features, configurability, or adjacent improvements.
+- Prefer editing existing code over rewriting files or reshaping unrelated logic.
+- Modify only what is required and remove only artifacts introduced by your own changes.
+- Preserve the existing control-flow and logic shape when fixing bugs or addressing review comments, especially in init, distributed coordination, locking, metadata, and concurrency paths.
+- Do not refactor existing code only to make it easier to unit test.
+- Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths.
+- Keep code elegant, concise, and direct. Prefer minimal, readable implementations over over-engineering and excessive abstraction. Use comments to clarify non-obvious intent and invariants, not to compensate for unclear code.
+- Mention unrelated issues when useful, but do not fix them as part of a narrow task.
+
+## Constant and String Usage
+
+- Before introducing new string literals, search for existing constants/enums that already represent the same semantic value.
+- Reuse existing constants for protocol labels, error identifiers, header keys, event names, metric names, command tags, and similar fixed tokens.
+- If a new string is truly unique, define a local constant near related logic and avoid scattering the literal across multiple sites.
+- When changing existing behavior, keep naming and format consistency by aligning with established project constants.
+
+## Sources of Truth
+
+- Workspace layout and crate membership: `Cargo.toml` (`[workspace].members`)
+- Local quality commands: `Makefile` and `.config/make/`
+- CI quality gates: `.github/workflows/ci.yml`
+- PR template: `.github/pull_request_template.md`
+
+Avoid duplicating long crate lists or command matrices in instruction files.
+Reference the source files above instead.
+
+## Verification Before PR
+
+Convert changes into independently verifiable outcomes. Prefer focused tests for behavior changes and run the relevant checks before declaring completion.
+
+For code changes, run and pass the following before opening a PR:
+
+```bash
+make pre-commit
+```
+
+Before pushing code changes, make sure formatting is clean:
+
+- Run `cargo fmt --all`.
+- Run `cargo fmt --all --check` and ensure no files are modified unexpectedly.
+
+If `make` is unavailable, run the equivalent checks defined under `.config/make/`.
+Documentation-only or instruction-only changes are exempt from the verification commands above (including the `.config/make/` equivalents), though any installed git pre-commit hooks (for example, from `make setup-hooks`) may still run on commit unless explicitly skipped.
+After build-based verification completes, clean generated build artifacts before wrapping up to avoid unnecessary disk usage.
+Do not open a PR with code changes when the required checks fail.
+
+## Git and PR Baseline
+
+- Use feature branches based on the latest `main`.
+- Follow Conventional Commits, with subject length <= 72 characters.
+- Keep PR title and description in English.
+- Use `.github/pull_request_template.md` and keep all section headings.
+- Use `N/A` for non-applicable template sections.
+- Include verification commands in the PR description.
+- When using `gh pr create`/`gh pr edit`, use `--body-file` instead of inline `--body` for multiline markdown.
+- Do not include the literal sequence `\n` in any GitHub issue, pull request, or discussion comment.
+- After fixing code review comments or CI findings, always mark corresponding review
+  comments/threads as resolved before returning to the user.
+- In handling review comments, confirm the underlying issue before changing code.
+  If a suggested change is not appropriate for behavior or risk, reply with a
+  concise rationale instead of blindly applying it.
+
+## Security Baseline
+
+- Never commit secrets, credentials, or key material.
+- Use environment variables or vault tooling for sensitive configuration.
+- For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
+
+## Scoped Guidance in This Repository
+
+- `.github/AGENTS.md`
+- `crates/AGENTS.md`
+- `crates/config/AGENTS.md`
+- `crates/ecstore/AGENTS.md`
+- `crates/e2e_test/AGENTS.md`
+- `crates/iam/AGENTS.md`
+- `crates/kms/AGENTS.md`
+- `crates/policy/AGENTS.md`
+- `crates/targets/AGENTS.md`
+- `rustfs/src/admin/AGENTS.md`
+- `rustfs/src/storage/AGENTS.md`

ARCHITECTURE.md

@@ -0,0 +1,376 @@
+# ARCHITECTURE.md
+
+> Last updated: 2026-04-13 · Revision: 1 (draft)
+>
+> This document describes the high-level architecture of RustFS.
+> If you want to familiarize yourself with the code base, you are in the right place!
+>
+> See also [CONTRIBUTING.md](CONTRIBUTING.md) for development workflow.
+
+## Bird's Eye View
+
+RustFS is a high-performance, S3-compatible distributed object storage system written
+in Rust. It uses erasure coding for data durability, supports multi-tenancy through
+IAM/STS, and provides a web-based admin console.
+
+A running RustFS node exposes:
+
+- **S3 API** (port 9000) — the primary data path for object CRUD
+- **Admin API** (port 9000, `/minio/` prefix) — cluster management, IAM, metrics
+- **Console** (port 9001) — web UI backed by the Admin API
+- **Inter-node RPC** (gRPC/tonic) — cluster communication for distributed mode
+
+The core data flow for a PUT request looks like:
+
+```
+HTTP request
+  → server (TLS, auth, routing, compression)
+    → app/object_usecase (validation, policy, lifecycle)
+      → storage/ecfs (erasure coding, encryption, checksums)
+        → ecstore (disk pool selection, data distribution)
+          → rio (reader pipeline: encrypt → compress → hash → write)
+            → io-core (zero-copy I/O, buffer pool, direct I/O)
+              → local disk / remote disk via RPC
+```
+
+## Code Map
+
+The repository is a Cargo workspace with a flat `crates/` layout:
+
+```
+rustfs/                      # Workspace root (virtual manifest)
+├── rustfs/                  # Main binary + library crate (75K lines)
+│   └── src/
+│       ├── main.rs          # Entry point, startup sequence
+│       ├── lib.rs           # Module tree root
+│       ├── server/          # HTTP server, TLS, routing, middleware
+│       ├── admin/           # Admin API handlers and console
+│       ├── app/             # Use-case layer (object, bucket, multipart)
+│       ├── storage/         # Storage engine interface and implementation
+│       ├── auth.rs          # S3 request authentication
+│       ├── config/          # CLI args, config parsing, workload profiles
+│       └── ...
+├── crates/                  # 39 library crates
+│   ├── ecstore/             # Erasure-coded storage engine (⚠️ 87K lines)
+│   ├── rio/                 # Reader I/O pipeline (encrypt, compress, hash)
+│   ├── io-core/             # Zero-copy I/O, scheduling, buffer pool
+│   ├── io-metrics/          # I/O metrics collection
+│   ├── common/              # Shared runtime state, globals, data usage types
+│   ├── config/              # Configuration types and parsing
+│   ├── utils/               # Pure utility functions
+│   ├── ...                  # (see "Crate Reference" below)
+│   └── e2e_test/            # End-to-end integration tests
+└── docs/                    # Design documents and analysis
+```
+
+### Main Crate Layers (`rustfs/src/`)
+
+The main crate is organized in layers, top to bottom:
+
+| Layer | Directory | Responsibility |
+|-------|-----------|----------------|
+| **Server** | `server/` | HTTP listener, TLS, CORS, compression, middleware, graceful shutdown |
+| **Admin** | `admin/` | Admin API routing, 30+ handler modules, web console |
+| **App** | `app/` | Use-case orchestration: object_usecase, bucket_usecase, multipart_usecase |
+| **Storage** | `storage/` | S3 API translation, erasure-coded FS, SSE encryption, RPC, concurrency |
+| **Auth** | `auth.rs` | S3 signature verification, credential validation |
+| **Config** | `config/` | CLI parsing, config struct, workload profiles |
+
+A request flows **downward** through the layers. No layer should reach upward
+(e.g., storage must not import from admin).
+
+### Crate Reference
+
+Crates are organized in a dependency DAG with 9 depth levels (0 = leaf, 8 = top):
+
+```
+Depth 0 — LEAF (no internal deps):
+  appauth, checksums, config, credentials, crypto, io-metrics,
+  madmin, s3-common, workers, zip
+
+Depth 1:
+  io-core (→ io-metrics)
+  policy (→ config, credentials, crypto)
+  utils (→ config)                        ⚠️ inverted: utils should be leaf
+
+Depth 2:
+  concurrency, filemeta, keystone, kms, lock, obs,
+  signer, targets, trusted-proxies
+
+Depth 3:
+  common (→ filemeta, madmin)             ⚠️ inverted: common should be leaf
+
+Depth 4:
+  object-capacity, protos, rio
+
+Depth 5 — CORE:
+  ecstore (16 internal deps, 11 dependents — the architectural heart)
+
+Depth 6:
+  audit, heal, iam, metrics, notify, s3select-api, scanner
+
+Depth 7:
+  object-io, protocols, s3select-query
+
+Depth 8 — TOP:
+  rustfs (35 internal deps — the binary, depends on almost everything)
+```
+
+#### By Domain
+
+**Core Infrastructure:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `config` | 3.3K | Configuration types and environment parsing |
+| `utils` | 8.7K | Pure utilities (paths, compression, network, retry) |
+| `common` | 4.4K | Shared runtime state, globals, data usage types, metrics |
+| `madmin` | 5.5K | Admin API request/response types |
+
+**I/O Pipeline:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `io-core` | 6.5K | Zero-copy I/O, buffer pool, direct I/O, scheduling, backpressure |
+| `io-metrics` | 4.5K | I/O operation metrics and counters |
+| `rio` | 6.9K | Composable reader chain (encrypt → compress → hash → limit) |
+| `object-io` | 2.4K | High-level object read/write using rio + ecstore |
+| `concurrency` | 1.8K | Concurrency control wrappers over io-core |
+
+**Storage Engine:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `ecstore` | 87K | ⚠️ Erasure-coded storage: disks, pools, buckets, replication, lifecycle |
+| `filemeta` | 10K | File/object metadata types and versioning |
+| `checksums` | 732 | Checksum computation |
+| `lock` | 7.1K | Distributed lock manager |
+| `heal` | 5.9K | Data healing / bitrot repair |
+| `scanner` | 5.4K | Background data usage scanner |
+| `object-capacity` | 2.5K | Capacity tracking and management |
+
+**Security & Auth:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `crypto` | 1.6K | Encryption primitives |
+| `credentials` | 713 | Credential types (access key / secret key) |
+| `signer` | 1.4K | S3 v4 request signing |
+| `iam` | 9.0K | Identity and access management |
+| `policy` | 8.8K | Policy engine (S3 bucket/IAM policies) |
+| `kms` | 8.1K | Key management service integration |
+| `keystone` | 1.9K | OpenStack Keystone auth |
+| `appauth` | 143 | Application-level auth tokens |
+
+**Protocol & API:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `protos` | 5.7K | Protobuf/gRPC definitions for inter-node RPC |
+| `protocols` | 18K | FTP/FTPS, WebDAV, Swift API support |
+| `s3-common` | 738 | Shared S3 types |
+| `s3select-api` | 1.9K | S3 Select interface |
+| `s3select-query` | 3.6K | S3 Select query engine |
+
+**Observability:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `metrics` | 8.4K | Prometheus metric collectors |
+| `io-metrics` | 4.5K | I/O-specific metrics |
+| `obs` | 5.6K | OpenTelemetry tracing and telemetry |
+| `audit` | 2.4K | Audit logging |
+
+**Events:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `notify` | 5.5K | Event notification system |
+| `targets` | 3.2K | Notification targets (Kafka, AMQP, webhook, etc.) |
+
+**Other:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `trusted-proxies` | 4.0K | Trusted proxy / IP forwarding |
+| `zip` | 986 | ZIP archive support for bulk downloads |
+| `workers` | 136 | Simple worker abstraction |
+
+## Architecture Invariants
+
+> These are rules that the codebase should follow. Some are currently violated
+> (marked with ⚠️). Documenting them here makes the violations explicit and
+> trackable.
+
+1. **Layers flow downward.** Server → Admin/App → Storage → ecstore → rio/io-core.
+   No upward imports.
+
+2. **Leaf crates have zero internal dependencies.** `config`, `credentials`, `crypto`,
+   `io-metrics`, `madmin`, `s3-common` should depend only on external crates.
+   - ⚠️ VIOLATED: `utils` depends on `config`, `common` depends on `filemeta` and `madmin`.
+
+3. **Each type has exactly one definition.** Types shared across crates must be defined
+   in one crate and re-exported or imported by others.
+   - ⚠️ VIOLATED: `ReplicationStats` (4 copies), `LastMinuteLatency` (3 copies),
+     `BackpressureConfig` (3 copies), `DataUsageInfo` (2 copies).
+
+4. **ecstore does not know about HTTP or S3 protocol details.** It operates on
+   storage-level abstractions (objects, buckets, disks, pools).
+
+5. **The `rustfs` binary crate is the only place that wires everything together.**
+   Individual crates should be testable in isolation.
+
+6. **Error types use `thiserror` with descriptive names** (e.g., `StorageError`,
+   not bare `Error`).
+   - ⚠️ VIOLATED: 6 crates use `pub enum Error`; 2 crates use `snafu`;
+      `heal` use `anyhow` in library code.
+
+## Known Structural Issues
+
+> This section documents known problems in the current architecture.
+> It exists so the team can track and address them deliberately.
+
+### Critical
+
+- **common/scanner code duplication (~3K lines).** `scanner` depends on `common`
+  but maintains its own copies of `DataUsageInfo`, `LastMinuteLatency`, and related
+  types instead of importing them.
+
+- **ecstore is a monolith (87K lines, 163 files).** It contains disk management,
+  bucket management, erasure coding, replication, lifecycle, RPC, and configuration
+  — all in one crate. It should be decomposed along its existing subdirectories.
+
+### High
+
+- **Dependency inversions.** `utils → config` and `common → filemeta/madmin` break
+  the layering model. These need to be untangled.
+
+- **Three-layer BackpressureConfig/DeadlockConfig duplication** across io-core,
+  concurrency, and rustfs/storage. Should be defined once with builder/composition.
+
+### Medium
+
+- **Inconsistent error handling.** Three strategies (thiserror/snafu/anyhow) and
+  mixed naming (bare `Error` vs descriptive names).
+
+- **Ambiguous common vs utils boundary.** Both described as "utilities and data
+  structures." Need clear ownership rules.
+
+## Cross-Cutting Concerns
+
+### Error Handling
+
+The project convention is `thiserror` for typed errors with descriptive names.
+See `AGENTS.md`: "Prefer thiserror for library-facing error types."
+
+```rust
+// GOOD
+#[derive(Debug, thiserror::Error)]
+pub enum StorageError {
+    #[error("disk not found: {0}")]
+    DiskNotFound(String),
+}
+
+// AVOID
+pub enum Error { ... }        // too generic
+anyhow::Result<T>             // in library code (OK in tests/CLI)
+```
+
+### Logging & Tracing
+
+- Use `tracing` crate (`info!`, `warn!`, `error!`, `debug!`, `trace!`)
+- Structured fields: `tracing::info!(bucket = %name, "created bucket")`
+- Spans for request-scoped context
+
+### Metrics
+
+- Prometheus-style metrics via `rustfs-obs` runtime and schema
+- I/O-specific counters via `rustfs-io-metrics`
+- Registration happens at crate level, collection/reporting in `rustfs-obs`
+
+### Testing
+
+- Unit tests: `#[cfg(test)] mod tests` in the same file
+- Integration tests: inside respective crates (not top-level `tests/`)
+- E2E tests: `crates/e2e_test/` — tests against a running server
+- Run all: `make test` or `cargo nextest run`
+
+## Startup Sequence
+
+The binary (`main.rs`) boots in this order:
+
+1. Environment variable compatibility (`MINIO_*` → `RUSTFS_*`)
+2. Tokio runtime construction
+3. CLI argument parsing
+4. License, observability, TLS, trusted proxies initialization
+5. Config parsing, server address resolution
+6. Credentials, endpoints, local disks, lock client initialization
+7. Capacity management initialization
+8. HTTP server start (S3 API + optional console)
+9. ECStore initialization (erasure coding storage engine)
+10. Global config, background replication, KMS
+11. Optional: FTP/FTPS/WebDAV servers
+12. Event notifier, audit system, deadlock detector
+13. Bucket metadata, IAM, Keystone, OIDC
+14. Scanner and heal manager
+15. Metrics system, mark `FullReady`
+16. Wait for shutdown signal → graceful shutdown
+
+## Dependency Diagram (Simplified)
+
+```
+                            ┌─────────┐
+                            │  rustfs │  (binary + lib, 75K lines)
+                            │  main   │
+                            └────┬────┘
+                                 │
+                 ┌───────────────┼───────────────┐
+                 │               │               │
+            ┌────▼────┐     ┌────▼────┐   ┌──────▼─────┐
+            │ server  │     │  admin  │   │    app     │
+            │ (HTTP)  │     │(console)│   │(use-cases) │
+            └────┬────┘     └────┬────┘   └──────┬─────┘
+                 │               │               │
+                 └───────────────┼───────────────┘
+                                 │
+                          ┌──────▼──────┐
+                          │   storage   │
+                          │ (ecfs, SSE, │
+                          │  RPC, ACL)  │
+                          └──────┬──────┘
+                                 │
+              ┌──────────────────┼──────────────────┐
+              │                  │                  │
+        ┌─────▼──────┐    ┌──────▼──────┐    ┌──────▼──────┐
+        │  ecstore   │    │     rio     │    │   io-core   │
+        │ (87K,core) │    │  (readers)  │    │ (zero-copy) │
+        └─────┬──────┘    └─────────────┘    └─────────────┘
+              │
+     ┌─────┬──┼──┬─────┬──────┐
+     │     │  │  │     │      │
+ common utils config policy filemeta ...
+```
+
+## How to Navigate
+
+- **"Where does S3 PutObject go?"**
+  `server/` routes → `app/object_usecase` validates → `storage/ecfs` encodes →
+  `ecstore` distributes → `rio` encrypts/compresses → `io-core` writes
+
+- **"Where are bucket policies enforced?"**
+  `app/bucket_usecase` calls into `crates/policy/`
+
+- **"Where is replication configured?"**
+  `admin/handlers/replication.rs` and `admin/handlers/site_replication.rs` for API,
+  `ecstore/src/bucket/replication/` for engine
+
+- **"Where do I add a new admin endpoint?"**
+  Add handler in `admin/handlers/`, register in `admin/router.rs`
+
+- **"Where do I add a new metric?"**
+  Define descriptor/collector in `crates/obs/src/metrics/`, expose via `/minio/v2/metrics`
+
+---
+
+*Inspired by [matklad's ARCHITECTURE.md](https://matklad.github.io/2021/02/06/ARCHITECTURE.md.html)
+and [rust-analyzer's architecture.md](https://github.com/rust-analyzer/rust-analyzer/blob/master/docs/book/src/contributing/architecture.md).*

CHANGELOG.md

@@ -0,0 +1,160 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Fixed
+- **Helm Ingress**: `customAnnotations` are now merged with class-specific annotations (nginx/traefik) instead of being ignored when `ingress.className` is set.
+
+### Added
+- **OpenStack Keystone Authentication Integration**: Full support for OpenStack Keystone authentication via X-Auth-Token headers
+  - Tower-based middleware (`KeystoneAuthLayer`) self-contained within `rustfs-keystone` crate
+  - Task-local storage for async-safe credential passing between middleware and auth handlers
+  - Automatic detection of Keystone credentials (access keys prefixed with `keystone:`)
+  - Role-based permission mapping (admin/reseller_admin roles grant owner permissions)
+  - Token caching for high-performance validation with configurable cache size and TTL
+  - Dual authentication support: Keystone and standard AWS Signature v4 work simultaneously
+  - Immediate 401 response for invalid tokens (no fallback to local auth)
+  - XML-formatted error responses compatible with S3 API
+  - Comprehensive integration documentation with manual testing guide
+  - **32 unit and integration tests** covering middleware, auth handlers, task-local storage, and role detection
+- **SFTPv3 Protocol Support**: SSH-hosted SFTPv3 subsystem that translates each file operation into S3 calls against the local object store. Authentication uses IAM credentials (SSH username = access key, SSH password = secret key).
+  - Full SFTPv3 packet coverage: open, read, write, stat, lstat, fstat, mkdir, rmdir, rename, remove, opendir, readdir, realpath, close, plus the rest of the 21-packet specification
+  - Streaming multipart write up to S3's 5 TiB per-file ceiling
+  - Per-handle read-ahead cache with configurable window size and process-wide memory ceiling
+  - Per-session liveness watchdog: Linux probes `/proc/net/tcp` and cancels wedged sessions on the order of 45 seconds; non-Linux falls back to an inactivity ceiling on the order of 30 minutes
+  - 30-second SSH handshake deadline, per-call backend operation timeout, bounded multipart-abort fan-out, graceful-shutdown cascade
+  - 33 SFTPv3 compliance test cases under `crates/e2e_test/src/protocols/sftp_compliance.rs` spread across three entry points: `test_sftp_compliance_suite` (shared session), `test_sftp_compliance_readonly` (read-only mode), and `test_sftp_compliance_standalone` (one rustfs spawn per case)
+  - Four-layer regression-prevention tests guard against silent feature deletion: compile-time module assertion, module-presence unit test, cross-module `Protocol` enum assertion, end-to-end SSH banner test against the running binary
+
+### Changed
+- **HTTP Server Stack**: Integrated `KeystoneAuthLayer` middleware from `rustfs-keystone` crate into service stack (positioned after ReadinessGateLayer)
+- **IAMAuth**: Enhanced `get_secret_key()` to return empty secret for Keystone credentials (bypasses signature validation)
+- **Auth Module**: Modified `check_key_valid()` to retrieve Keystone credentials from task-local storage and determine admin status
+- **`StorageBackend` trait**: extended with multipart upload methods (`create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`) plus `upload_part_copy`. Streaming-upload code path is now available to FTPS, WebDAV, and Swift drivers as well.
+- **`Protocol` enum**: new `Protocol::Sftp` variant with corresponding `S3Action` mappings. Every match arm on `Protocol` updated to handle the new variant exhaustively.
+
+### Technical Details
+- Middleware is self-contained in `rustfs-keystone` crate following the trusted-proxies pattern for integration-specific middleware
+- Uses `BoxBody` pattern for Hyper 1.x compatibility
+- Task-local storage provides request-scoped credential passing without modifying HTTP request/response types
+- Integration preserves existing S3 authentication flow while adding Keystone support
+- Zero breaking changes to existing functionality
+- No new top-level directories in main binary crate (middleware lives in integration crate)
+- SSH/SFTP wire handling via the `russh` and `russh-sftp` crates. SFTPv3 framing is implemented by `russh-sftp`; the rustfs-side `SftpDriver` implements `russh_sftp::server::Handler` and dispatches to the storage backend
+- Drop-time abort for in-flight multipart uploads honours IAM Deny on `AbortMultipartUpload`. `start_multipart_upload` caches the authorisation decision so the synchronous `Drop` path can honour Allow / Deny policies without re-querying IAM
+- Per-handle read cache uses an `Arc<AtomicU64>` shared across every `SftpDriver` instance to enforce a process-wide memory ceiling. On ceiling breach the populate is skipped and the read serves correctly via a single-call backend fetch
+- Per-session liveness watchdog runs as a tokio task per accepted connection. Reads `/proc/net/tcp` and `/proc/net/tcp6` to look up the (local, peer) tuple's TCP state and cancels via `tokio_util::sync::CancellationToken` when wedge conditions are confirmed across two consecutive ticks
+- Path canonicalisation rejects paths containing `\0`, `\r`, or `\n` and resolves traversal via `path::clean()` before any backend dispatch
+- Cipher / KEX / MAC / host-key algorithm allowlists are hardcoded with no environment override. Strict-KEX (CVE-2023-48795 / Terrapin) marker presence asserted by unit test
+- Per-session handle cap (default 64, configurable 8 to 1024) with UUID-generated handle ids
+- Crate-level `#![deny(unsafe_code)]` is in force across `crates/protocols`. Socket fd duplication for the watchdog uses the safe `AsFd::try_clone_to_owned` path (Linux/Unix); non-Unix falls back to the inactivity ceiling
+- `cfg(unix)` gating around platform-specific imports (`std::os::fd::AsFd`, `std::os::unix::fs::PermissionsExt`); non-Unix targets fail SFTP at config-load with `SftpInitError::UnsupportedPlatform`
+
+### Documentation
+- Updated `crates/keystone/README.md` with complete integration architecture and workflow
+- Added detailed manual testing guide with 10 test scenarios
+- Updated main `README.md` to list Keystone authentication as available feature
+- Added troubleshooting section for common integration issues
+- Module-level rustdoc on `crates/protocols/src/sftp/mod.rs` describing the public API surface, configuration contract, and the architecture of the read cache and the wedge watchdog
+
+### Configuration
+New environment variables:
+- `RUSTFS_KEYSTONE_ENABLE` - Enable/disable Keystone authentication (default: false)
+- `RUSTFS_KEYSTONE_AUTH_URL` - Keystone API endpoint URL
+- `RUSTFS_KEYSTONE_VERSION` - Keystone API version (v3)
+- `RUSTFS_KEYSTONE_ADMIN_USER` - Admin username for privileged operations
+- `RUSTFS_KEYSTONE_ADMIN_PASSWORD` - Admin password
+- `RUSTFS_KEYSTONE_ADMIN_PROJECT` - Admin project name
+- `RUSTFS_KEYSTONE_ADMIN_DOMAIN` - Admin domain name (default: Default)
+- `RUSTFS_KEYSTONE_CACHE_SIZE` - Token cache size (default: 10000)
+- `RUSTFS_KEYSTONE_CACHE_TTL` - Token cache TTL in seconds (default: 300)
+- `RUSTFS_KEYSTONE_VERIFY_SSL` - Verify SSL certificates (default: true)
+- `RUSTFS_SFTP_ENABLE` - Enable/disable SFTP (default: false)
+- `RUSTFS_SFTP_ADDRESS` - Listen address (default: 0.0.0.0:2222)
+- `RUSTFS_SFTP_HOST_KEY_DIR` - Directory containing host key files (must exist; each file must be 0o600 or 0o400)
+- `RUSTFS_SFTP_IDLE_TIMEOUT` - Session idle timeout in seconds (default: 600)
+- `RUSTFS_SFTP_PART_SIZE` - Multipart part size in bytes (default: 16 MiB)
+- `RUSTFS_SFTP_READ_ONLY` - Reject write packets at the protocol layer (default: false)
+- `RUSTFS_SFTP_BANNER` - Optional SSH banner text
+- `RUSTFS_SFTP_HANDLES_PER_SESSION` - Per-session open-handle cap, 8 to 1024 (default: 64)
+- `RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS` - Per-call backend deadline in seconds, 5 to 600 (default: 60)
+- `RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES` - Per-handle read-cache window in bytes, 256 KiB to 64 MiB or 0 to disable (default: 4 MiB)
+- `RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES` - Process-wide read-cache memory ceiling in bytes, 16 MiB minimum (default: 256 MiB)
+
+### Files Added
+- `crates/protocols/src/sftp/mod.rs` - SFTP module entry point, public API surface, crate-level rustdoc, regression-prevention test
+- `crates/protocols/src/sftp/config.rs` - `SftpConfig` and `SftpInitError` types, env-var resolvers, host-key directory loader with permission enforcement
+- `crates/protocols/src/sftp/constants.rs` - Named constants grouped by purpose: S3 error codes, HTTP error codes, POSIX mode bits, protocol identifiers, operational limits
+- `crates/protocols/src/sftp/server.rs` - `SftpServer` SSH server, russh handler, password authentication against IAM, accept loop, per-session task spawn
+- `crates/protocols/src/sftp/driver.rs` - `SftpDriver` per-session SFTPv3 handler dispatching each operation onto the `StorageBackend`
+- `crates/protocols/src/sftp/state.rs` - `HandleState` variants for read, write-buffering, write-streaming, write-failed handles
+- `crates/protocols/src/sftp/lifecycle.rs` - Per-session activity stamp, weak-ref registry, `/proc/net/tcp` probe for the wedge watchdog
+- `crates/protocols/src/sftp/wedge_watchdog.rs` - Per-session liveness watchdog cancelling sessions silent at the SFTP layer while the kernel reports CLOSE_WAIT
+- `crates/protocols/src/sftp/read_cache.rs` - Per-handle in-memory read-ahead cache with shared atomic accumulator for the process-wide memory ceiling
+- `crates/protocols/src/sftp/attrs.rs` - SFTPv3 `FileAttributes` mapping for objects and directories, longname formatting, mtime clamping
+- `crates/protocols/src/sftp/dir.rs` - OPENDIR / READDIR pagination, root-bucket listing, sub-directory listing under a prefix
+- `crates/protocols/src/sftp/errors.rs` - `SftpError` thiserror enum and S3-error classification into SFTPv3 status codes
+- `crates/protocols/src/sftp/paths.rs` - Path canonicalisation, traversal rejection, `\0` / `\r` / `\n` rejection, bucket+key decomposition
+- `crates/protocols/src/sftp/read.rs` - READ packet handler, EOF semantics, `MAX_READ_LEN` bound, integration with the read cache
+- `crates/protocols/src/sftp/write.rs` - WRITE packet handler, in-memory buffering up to part size, transition to streaming multipart, CLOSE finalisation
+- `crates/protocols/src/sftp/test_support.rs` - Test fixtures and helper builders for SFTP unit tests
+- `crates/protocols/src/common/dummy_storage.rs` - In-memory `StorageBackend` test backend covering every method, used by SFTP unit tests and the FTPS / Swift / WebDAV test suites
+- `crates/e2e_test/src/protocols/sftp_core.rs` - End-to-end regressions for the handshake deadline, idle-timeout disconnect, and the wedge watchdog
+- `crates/e2e_test/src/protocols/sftp_compliance.rs` - SFTPv3 compliance suite entry points (`test_sftp_compliance_suite`, `test_sftp_compliance_readonly`, `test_sftp_compliance_standalone`)
+- `crates/e2e_test/src/protocols/sftp_compliance_tests.rs` - Per-case test bodies (CMPTST-01..33), shared fixture helpers, lifecycle counters
+- `crates/e2e_test/src/protocols/sftp_helpers.rs` - SFTP-specific test helpers and fixture seeders
+
+### Files Modified
+- `crates/keystone/src/middleware.rs` - Created Keystone authentication middleware (self-contained in keystone crate)
+- `crates/keystone/src/lib.rs` - Exported middleware module and KEYSTONE_CREDENTIALS
+- `crates/keystone/Cargo.toml` - Added Tower/HTTP dependencies for middleware functionality
+- `rustfs/src/server/http.rs` - Integrated KeystoneAuthLayer from rustfs-keystone crate
+- `rustfs/src/auth.rs` - Enhanced IAMAuth and check_key_valid for Keystone support, imported KEYSTONE_CREDENTIALS from rustfs-keystone
+- `crates/keystone/README.md` - Comprehensive integration documentation
+- `README.md` - Added Keystone as available feature
+- `Cargo.toml` - Added the `sftp` feature alongside the existing protocol features
+- `Cargo.lock` - Updated to include the new `russh`, `russh-sftp`, `socket2`, `tokio-util`, `subtle`, `uuid` dependencies and their transitive crates
+- `crates/protocols/Cargo.toml` - Declared `russh`, `russh-sftp`, `socket2`, `tokio-util`, `subtle`, `uuid` under the `sftp` feature flag
+- `crates/protocols/src/lib.rs` - Added `pub mod sftp` behind `#[cfg(feature = "sftp")]` plus the crate-level `#![deny(unsafe_code)]` lint
+- `crates/protocols/src/common/client/s3.rs` - Extended the `StorageBackend` trait with `create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`, and `upload_part_copy`
+- `crates/protocols/src/common/session.rs` - Added the `Protocol::Sftp` variant and its `S3Action` mappings
+- `crates/protocols/src/common/gateway.rs` - Handles the new `Protocol::Sftp` variant exhaustively
+- `crates/protocols/src/common/mod.rs` - Exposed the new `dummy_storage` module
+- `crates/protocols/src/constants.rs` - Added shared POSIX mode-bit constants used by SFTP and other protocols
+- `crates/config/src/constants/protocols.rs` - `RUSTFS_SFTP_*` environment variable names and defaults
+- `crates/utils/src/retry.rs` - Added the generic exponential-backoff retry helper used by the SFTP write path
+- `crates/e2e_test/Cargo.toml` - Added the e2e test dependencies for SFTP (paramiko fixture, SSH keypair generation)
+- `crates/e2e_test/src/protocols/mod.rs` - Registered the new `sftp_core`, `sftp_compliance`, `sftp_compliance_tests`, and `sftp_helpers` modules
+- `crates/e2e_test/src/protocols/README.md` - Documented the SFTP test entry points and case index
+- `crates/e2e_test/src/protocols/test_env.rs` - Added SFTP host-key directory provisioning to the shared protocol test environment
+- `crates/e2e_test/src/protocols/test_runner.rs` - Wired the SFTP entry points into the runner
+- `rustfs/Cargo.toml` - Added the `sftp` feature flag
+- `rustfs/src/lib.rs` - One-line addition exporting the SFTP wiring
+- `rustfs/src/init.rs` - Build and start the `SftpServer` when `RUSTFS_SFTP_ENABLE` is true
+- `rustfs/src/main.rs` - Routed shutdown signals to the SFTP server alongside the other protocols
+- `rustfs/src/protocols/client.rs` - Client-builder support for the new `Protocol::Sftp` variant
+
+### Testing
+- 16 unit tests in rustfs-keystone crate (config, auth, middleware, identity)
+- 10 integration tests in rustfs-keystone crate (task-local storage, middleware layer, scope isolation)
+- 6 auth unit tests in rustfs crate (role detection, task-local storage, Keystone credential handling)
+- **Total: 32 tests** passing with zero compilation errors
+- Manual testing guide provided for end-to-end validation
+- All Keystone tests passing with `cargo test --all --exclude e2e_test`
+- 33 SFTPv3 compliance test cases (CMPTST-01..33) split across three entry points: `test_sftp_compliance_suite` (shared session, cases 01-14), `test_sftp_compliance_readonly` (read-only mode, cases 15-23), `test_sftp_compliance_standalone` (one rustfs spawn per case, cases 24-33)
+- Regression-prevention tests at four layers: compile-time module assertion in `crates/protocols/src/lib.rs`, module-presence unit test in `crates/protocols/src/sftp/mod.rs`, cross-module `Protocol` enum assertion, and end-to-end SSH banner test against the running binary
+- Standalone end-to-end regressions for the SSH handshake deadline, the idle-timeout disconnect path, and the wedge watchdog (Linux fast-kill and the cross-platform fallback path)
+- Inline unit tests in every SFTP source file covering pure helpers (path canonicalisation, attribute mapping, S3-error classification, env-var bound resolvers)
+- Strict-KEX (CVE-2023-48795) marker presence assertion as a unit test in `crates/protocols/src/sftp/server.rs`
+- All tests passing with `cargo test --all --features sftp` against a 64-bit Linux target
+
+---
+
+## Previous Releases
+
+See [GitHub Releases](https://github.com/rustfs/rustfs/releases) for previous version history.

CLA.md

@@ -0,0 +1,60 @@
+# RustFS Individual Contributor License Agreement
+
+Thank you for your interest in contributing to RustFS. In order to clarify the intellectual property license granted with Contributions from any person or entity, RustFS, Inc. ("RustFS") must have a Contributor License Agreement ("CLA") on file that has been signed by each Contributor, indicating agreement to the license terms below. This license is for your protection as a Contributor as well as the protection of RustFS and its users; it does not change your rights to use your own Contributions for any other purpose.
+
+You accept and agree to the following terms and conditions for Your present and future Contributions submitted to RustFS. Except for the license granted herein to RustFS and recipients of software distributed by RustFS, You reserve all right, title, and interest in and to Your Contributions.
+
+**You understand and agree that You will not receive any royalty or other compensation for Your Contributions.**
+
+## 1. Definitions
+
+*   **"You"** (or **"Your"**) shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with RustFS. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
+
+*   **"Contribution"** shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to RustFS for inclusion in, or documentation of, any of the products or projects owned or managed by RustFS (the "Work"). For the purposes of this definition, "submitted" means any form of electronic or written communication sent to RustFS or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, RustFS for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution."
+
+## 2. Grant of Copyright License
+
+Subject to the terms and conditions of this Agreement, You hereby grant to RustFS and to recipients of software distributed by RustFS a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works under any license, including proprietary or commercial licenses and open-source licenses, at RustFS's sole discretion.
+
+## 3. Grant of Patent License
+
+Subject to the terms and conditions of this Agreement, You hereby grant to RustFS and to recipients of software distributed by RustFS a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed.
+
+## 4. Representations
+
+You represent that you are legally entitled to grant the above license. If your employer(s) has rights to intellectual property that you create that includes your Contributions, you represent that you have received permission to make Contributions on behalf of that employer, that your employer has waived such rights for your Contributions to RustFS, or that your employer has executed a separate Corporate CLA with RustFS.
+
+You represent that each of Your Contributions is Your original creation. You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware and which are associated with any part of Your Contributions.
+
+## 5. Support and Warranty
+
+You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of **TITLE**, **NON-INFRINGEMENT**, **MERCHANTABILITY**, or **FITNESS FOR A PARTICULAR PURPOSE**.
+
+## 6. Third-Party Work
+
+Should You wish to submit work that is not Your original creation, You may submit it to RustFS separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [named here]".
+
+## 7. Governing Law and Jurisdiction
+
+This Agreement shall be governed by and construed in accordance with the laws of the State of Delaware, United States of America, without regard to its conflict of laws principles. The parties expressly agree that the United Nations Convention on Contracts for the International Sale of Goods will not apply. Any legal action or proceeding arising under or in connection with this Agreement shall be brought exclusively in the state or federal courts located in the State of Delaware, United States of America, and the parties hereby irrevocably consent to the personal jurisdiction and venue therein.
+
+## 8. Severability
+
+If any provision of this Agreement is found to be invalid or unenforceable, the remaining provisions will continue in full force and effect.
+
+## 9. Entire Agreement and Assignment
+
+This Agreement constitutes the entire agreement between the parties with respect to the subject matter hereof and supersedes all prior agreements, understandings, negotiations, and discussions, whether written or oral. RustFS may assign its rights and obligations under this Agreement to any third party. You may not assign Your rights or obligations under this Agreement without the prior written consent of RustFS.
+
+---
+
+**Please read the terms of this Agreement carefully. By submitting a Contribution to RustFS, You agree to be bound by the terms and conditions of this Agreement.**
+
+| | |
+|---|---|
+| **Full Name** | __________________________________ |
+| **GitHub Username** | __________________________________ |
+| **Email Address** | __________________________________ |
+| **Date** | __________________________________ |
+
+*(Electronic signature or acknowledgement via GitHub commit/Pull Request constitutes valid acceptance of this Agreement).*

CODE_OF_CONDUCT.md

@@ -0,0 +1,128 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity
+and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the
+  overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or
+  advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email
+  address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official e-mail address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+hello@rustfs.com.
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series
+of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or
+permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior,  harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within
+the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.0, available at
+https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct
+enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see the FAQ at
+https://www.contributor-covenant.org/faq. Translations are available at
+https://www.contributor-covenant.org/translations.

CONTRIBUTING.md

@@ -0,0 +1,224 @@
+# RustFS Development Guide
+
+## 📋 Code Quality Requirements
+
+For instructions on setting up and running the local development environment, please see [Development Guide](docs/DEVELOPMENT.md).
+
+### 🔧 Code Formatting Rules
+
+**MANDATORY**: All code must be properly formatted before committing. This project enforces strict formatting standards to maintain code consistency and readability.
+
+#### Pre-commit Requirements
+
+Before every commit, you **MUST**:
+
+1. **Format your code**:
+
+   ```bash
+   cargo fmt --all
+   ```
+
+2. **Verify formatting**:
+
+   ```bash
+   cargo fmt --all --check
+   ```
+
+3. **Pass clippy checks**:
+
+   ```bash
+   cargo clippy --all-targets --all-features -- -D warnings
+   ```
+
+4. **Ensure compilation**:
+
+   ```bash
+   cargo check --all-targets
+   ```
+
+#### Quick Commands
+
+We provide convenient Makefile targets for common tasks:
+
+```bash
+# Format all code
+make fmt
+
+# Check if code is properly formatted
+make fmt-check
+
+# Run clippy checks
+make clippy
+
+# Run compilation check
+make check
+
+# Run tests
+make test
+
+# Run all pre-commit checks (format + clippy + check + test)
+make pre-commit
+
+# Setup git hooks (one-time setup)
+make setup-hooks
+```
+
+### 🔒 Automated Pre-commit Hooks
+
+This project includes a pre-commit hook that automatically runs before each commit to ensure:
+
+- ✅ Code is properly formatted (`cargo fmt --all --check`)
+- ✅ No clippy warnings (`cargo clippy --all-targets --all-features -- -D warnings`)
+- ✅ Code compiles successfully (`cargo check --all-targets`)
+
+#### Setting Up Pre-commit Hooks
+
+Run this command once after cloning the repository:
+
+```bash
+make setup-hooks
+```
+
+Or manually:
+
+```bash
+chmod +x .git/hooks/pre-commit
+```
+
+### 📝 Formatting Configuration
+
+The project uses the following rustfmt configuration (defined in `rustfmt.toml`):
+
+```toml
+max_width = 130
+fn_call_width = 90
+single_line_let_else_max_width = 100
+```
+
+### 🚫 Commit Prevention
+
+If your code doesn't meet the formatting requirements, the pre-commit hook will:
+
+1. **Block the commit** and show clear error messages
+2. **Provide exact commands** to fix the issues
+3. **Guide you through** the resolution process
+
+Example output when formatting fails:
+
+```
+❌ Code formatting check failed!
+💡 Please run 'cargo fmt --all' to format your code before committing.
+
+🔧 Quick fix:
+   cargo fmt --all
+   git add .
+   git commit
+```
+
+### 🔄 Development Workflow
+
+1. **Make your changes**
+2. **Format your code**: `make fmt` or `cargo fmt --all`
+3. **Run pre-commit checks**: `make pre-commit`
+4. **Commit your changes**: `git commit -m "your message"`
+5. **Push to your branch**: `git push`
+
+### 🛠️ IDE Integration
+
+#### VS Code
+
+Install the `rust-analyzer` extension and add to your `settings.json`:
+
+```json
+{
+    "rust-analyzer.rustfmt.extraArgs": ["--config-path", "./rustfmt.toml"],
+    "editor.formatOnSave": true,
+    "[rust]": {
+        "editor.defaultFormatter": "rust-lang.rust-analyzer"
+    }
+}
+```
+
+#### Other IDEs
+
+Configure your IDE to:
+
+- Use the project's `rustfmt.toml` configuration
+- Format on save
+- Run clippy checks
+
+### ❗ Important Notes
+
+- **Never bypass formatting checks** - they are there for a reason
+- **All CI/CD pipelines** will also enforce these same checks
+- **Pull requests** will be automatically rejected if formatting checks fail
+- **Consistent formatting** improves code readability and reduces merge conflicts
+
+### 🆘 Troubleshooting
+
+#### Pre-commit hook not running?
+
+```bash
+# Check if hook is executable
+ls -la .git/hooks/pre-commit
+
+# Make it executable if needed
+chmod +x .git/hooks/pre-commit
+```
+
+#### Formatting issues?
+
+```bash
+# Format all code
+cargo fmt --all
+
+# Check specific issues
+cargo fmt --all --check --verbose
+```
+
+#### Clippy issues?
+
+```bash
+# See detailed clippy output
+cargo clippy --all-targets --all-features -- -D warnings
+
+# Fix automatically fixable issues
+cargo clippy --fix --all-targets --all-features
+```
+
+## 📝 Pull Request Guidelines
+
+### Language Requirements
+
+**All Pull Request titles and descriptions MUST be written in English.**
+
+This ensures:
+- Consistency across all contributions
+- Accessibility for international contributors
+- Better integration with automated tools and CI/CD systems
+- Clear communication in a globally understood language
+
+#### PR Description Requirements
+
+When creating a Pull Request, ensure:
+
+1. **Title**: Use English and follow Conventional Commits format (e.g., `fix: improve s3-tests readiness detection`)
+2. **Description**: Write in English, following the PR template format
+3. **Code Comments**: Must be in English (as per coding standards)
+4. **Commit Messages**: Must be in English (as per commit guidelines)
+
+#### PR Template
+
+Always use the PR template (`.github/pull_request_template.md`) and fill in all sections:
+- Type of Change
+- Related Issues
+- Summary of Changes
+- Checklist
+- Impact
+- Additional Notes
+
+**Note**: While you may communicate with reviewers in Chinese during discussions, the PR itself (title, description, and all formal documentation) must be in English.
+
+---
+
+Following these guidelines ensures high code quality and smooth collaboration across the RustFS project! 🚀

Cargo.toml

@@ -0,0 +1,348 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+[workspace]
+members = [
+    "rustfs", # Core file system implementation
+    "crates/appauth", # Application authentication and authorization
+    "crates/audit", # Audit target management system with multi-target fan-out
+    "crates/checksums", # client checksums
+    "crates/common", # Shared utilities and data structures
+    "crates/config", # Configuration management
+    "crates/credentials", # Credential management system
+    "crates/crypto", # Cryptography and security features
+    "crates/ecstore", # Erasure coding storage implementation
+    "crates/e2e_test", # End-to-end test suite
+    "crates/filemeta", # File metadata management
+    "crates/heal", # Erasure set and object healing
+    "crates/iam", # Identity and Access Management
+    "crates/keystone", # OpenStack Keystone integration
+    "crates/kms", # Key Management Service
+    "crates/lock", # Distributed locking implementation
+    "crates/madmin", # Management dashboard and admin API interface
+    "crates/notify", # Notification system for events
+    "crates/obs", # Observability utilities
+    "crates/object-capacity", # Capacity scan and refresh core
+    "crates/policy", # Policy management
+    "crates/protocols", # Protocol implementations (FTPS, SFTP, etc.)
+    "crates/protos", # Protocol buffer definitions
+    "crates/rio", # Rust I/O utilities and abstractions
+    "crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
+    "crates/s3-common", # Common utilities and data structures for S3 compatibility
+    "crates/s3select-api", # S3 Select API interface
+    "crates/s3select-query", # S3 Select query engine
+    "crates/scanner", # Scanner for data integrity checks and health monitoring
+    "crates/signer", # client signer
+    "crates/targets", # Target-specific configurations and utilities
+    "crates/trusted-proxies", # Trusted proxies management
+    "crates/utils", # Utility functions and helpers
+    "crates/workers", # Worker thread pools and task scheduling
+    "crates/io-metrics", # Zero-copy metrics collection for performance analysis
+    "crates/io-core", # Zero-copy core reader and writer implementations
+    "crates/zip", # ZIP file handling and compression
+]
+resolver = "3"
+
+[workspace.package]
+edition = "2024"
+license = "Apache-2.0"
+repository = "https://github.com/rustfs/rustfs"
+rust-version = "1.95.0"
+version = "1.0.0-beta.3"
+homepage = "https://rustfs.com"
+description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
+keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
+categories = ["web-programming", "development-tools", "filesystem", "network-programming"]
+authors = ["RustFS Team"]
+
+[workspace.lints.rust]
+unsafe_code = "deny"
+
+[workspace.lints.clippy]
+all = "warn"
+needless_collect = "warn"
+redundant_clone = "warn"
+
+[workspace.dependencies]
+# RustFS Internal Crates
+rustfs = { path = "./rustfs", version = "1.0.0-beta.3" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.3" }
+rustfs-appauth = { path = "crates/appauth", version = "1.0.0-beta.3" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.3" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.3" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.3" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.3" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.3" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.3" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.3" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.3" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.3" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.3" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.3" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.3" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.3" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.3" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.3" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.3" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.3" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.3" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.3" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.3" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.3" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.3" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.3" }
+rustfs-s3-common = { path = "crates/s3-common", version = "1.0.0-beta.3" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.3" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.3" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.3" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.3" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.3" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.3" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.3" }
+rustfs-workers = { path = "crates/workers", version = "1.0.0-beta.3" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.3" }
+
+# Async Runtime and Networking
+async-channel = "2.5.0"
+mysql_async = { version = "0.36.1", default-features = false, features = ["default-rustls", "tracing"], git = "https://github.com/blackbeam/mysql_async", rev = "98d3d8067efdf97d3e93cdca7b9231753c904aca" }
+async-compression = { version = "0.4.42" }
+async-recursion = "1.1.1"
+async-trait = "0.1.89"
+async-nats = "0.48.0"
+axum = "0.8.9"
+futures = "0.3.32"
+futures-core = "0.3.32"
+futures-util = "0.3.32"
+pollster = "0.4.0"
+pulsar = { version = "6.7.2", default-features = false, features = ["tokio-rustls-runtime"] }
+lapin = { version = "4.7.2", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
+hyper = { version = "1.9.0", features = ["http2", "http1", "server"] }
+hyper-rustls = { version = "0.27.9", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
+hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
+http = "1.4.0"
+http-body = "1.0.1"
+http-body-util = "0.1.3"
+reqwest = { version = "0.13.3", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
+rustfs-kafka-async = { version = "1.2.0" }
+socket2 = { version = "0.6.3", features = ["all"] }
+tokio = { version = "1.52.3", features = ["fs", "rt-multi-thread"] }
+tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
+tokio-stream = { version = "0.1.18" }
+tokio-test = "0.4.5"
+tokio-util = { version = "0.7.18", features = ["io", "compat"] }
+tonic = { version = "0.14.6", features = ["gzip"] }
+tonic-prost = { version = "0.14.6" }
+tonic-prost-build = { version = "0.14.6" }
+tower = { version = "0.5.3", features = ["timeout"] }
+tower-http = { version = "0.6.10", features = ["cors"] }
+
+# Serialization and Data Formats
+bytes = { version = "1.11.1", features = ["serde"] }
+bytesize = "2.3.1"
+byteorder = "1.5.0"
+flatbuffers = "25.12.19"
+form_urlencoded = "1.2.2"
+prost = "0.14.3"
+quick-xml = "0.39.4"
+rmp = { version = "0.8.15" }
+rmp-serde = { version = "1.3.1" }
+serde = { version = "1.0.228", features = ["derive"] }
+serde_json = { version = "1.0.149", features = ["raw_value"] }
+serde_urlencoded = "0.7.1"
+
+# Cryptography and Security
+aes-gcm = { version = "0.11.0-rc.3", features = ["rand_core"] }
+argon2 = { version = "0.6.0-rc.8" }
+blake2 = "0.11.0-rc.6"
+chacha20poly1305 = { version = "0.11.0-rc.3" }
+crc-fast = "1.9.0"
+hmac = { version = "0.13.0" }
+jsonwebtoken = { version = "10.3.0", features = ["aws_lc_rs"] }
+openidconnect = { version = "4.0", default-features = false }
+pbkdf2 = "0.13.0"
+rsa = { version = "0.10.0-rc.18" }
+rustls = { version = "0.23.40", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
+rustls-native-certs = "0.8"
+rustls-pki-types = "1.14.1"
+sha1 = "0.11.0"
+sha2 = "0.11.0"
+subtle = "2.6"
+zeroize = { version = "1.8.2", features = ["derive"] }
+
+# Time and Date
+chrono = { version = "0.4.44", features = ["serde"] }
+humantime = "2.3.0"
+jiff = { version = "0.2.24", features = ["serde"] }
+time = { version = "0.3.47", features = ["std", "parsing", "formatting", "macros", "serde"] }
+
+# Database
+deadpool-postgres = { version = "0.14", features = ["rt_tokio_1"] }
+tokio-postgres = { version = "0.7", default-features = false, features = ["runtime", "with-serde_json-1"] }
+tokio-postgres-rustls = "0.13"
+
+# Utilities and Tools
+anyhow = "1.0.102"
+arc-swap = "1.9.1"
+astral-tokio-tar = "0.6.1"
+atoi = "2.0.0"
+atomic_enum = "0.3.0"
+aws-config = { version = "1.8.16" }
+aws-credential-types = { version = "1.2.14" }
+aws-sdk-s3 = { version = "1.132.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+aws-smithy-http-client = { version = "1.1.12", default-features = false, features = ["default-client", "rustls-aws-lc"] }
+aws-smithy-types = { version = "1.4.7" }
+backtrace = "0.3.76"
+base64 = "0.22.1"
+base64-simd = "0.8.0"
+brotli = "8.0.2"
+clap = { version = "4.6.1", features = ["derive", "env"] }
+const-str = { version = "1.1.0", features = ["std", "proc"] }
+convert_case = "0.11.0"
+criterion = { version = "0.8", features = ["html_reports"] }
+crossbeam-queue = "0.3.12"
+crossbeam-channel = "0.5.15"
+crossbeam-deque = "0.8.6"
+crossbeam-utils = "0.8.21"
+datafusion = "53.1.0"
+derive_builder = "0.20.2"
+enumset = "1.1.12"
+faster-hex = "0.10.0"
+flate2 = "1.1.9"
+glob = "0.3.3"
+google-cloud-storage = "1.12.0"
+google-cloud-auth = "1.10.0"
+hashbrown = { version = "0.17.1", features = ["serde", "rayon"] }
+hex = "0.4.3"
+hex-simd = "0.8.0"
+highway = { version = "1.3.0" }
+ipnetwork = { version = "0.21.1", features = ["serde"] }
+lazy_static = "1.5.0"
+libc = "0.2.186"
+libsystemd = "0.7.2"
+local-ip-address = "0.6.12"
+memmap2 = "0.9.10"
+lz4 = "1.28.1"
+matchit = "0.9.2"
+md-5 = "0.11.0"
+md5 = "0.8.0"
+mime_guess = "2.0.5"
+moka = { version = "0.12.15", features = ["future"] }
+netif = "0.1.6"
+num_cpus = { version = "1.17.0" }
+nvml-wrapper = "0.12.1"
+object_store = "0.13.2"
+parking_lot = "0.12.5"
+path-absolutize = "3.1.1"
+path-clean = "1.0.1"
+percent-encoding = "2.3.2"
+pin-project-lite = "0.2.17"
+pretty_assertions = "1.4.1"
+rand = { version = "0.10.1", features = ["serde"] }
+ratelimit = "0.10.1"
+rayon = "1.12.0"
+reed-solomon-erasure = { version = "6.0", default-features = false, features = ["std", "simd-accel"] }
+reed-solomon-simd = "3.1.0"
+regex = { version = "1.12.3" }
+rumqttc = { package = "rumqttc-next", version = "0.33.0", features = ["websocket"] }
+redis = { version = "1.2.1", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
+rustix = { version = "1.1.4", features = ["fs"] }
+rust-embed = { version = "8.11.0" }
+rustc-hash = { version = "2.1.2" }
+s3s = { git = "https://github.com/rustfs/s3s", rev = "a3b16608df35aaeed8fff08b4988d03f4ca9445b", features = ["minio"] }
+serial_test = "3.4.0"
+shadow-rs = { version = "2.0.0", default-features = false }
+siphasher = "1.0.3"
+smallvec = { version = "1.15.1", features = ["serde"] }
+smartstring = "1.0.1"
+snafu = "0.9.0"
+snap = "1.1.1"
+starshard = { version = "1.1.0", features = ["rayon", "async", "serde"] }
+strum = { version = "0.28.0", features = ["derive"] }
+sysinfo = "0.39.0"
+temp-env = "0.3.6"
+tempfile = "3.27.0"
+test-case = "3.3.1"
+thiserror = "2.0.18"
+tracing = { version = "0.1.44" }
+tracing-appender = "0.2.5"
+tracing-error = "0.2.1"
+tracing-opentelemetry = "0.32.1"
+tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
+transform-stream = "0.3.1"
+url = "2.5.8"
+urlencoding = "2.1.3"
+uuid = { version = "1.23.1", features = ["v4", "fast-rng", "macro-diagnostics"] }
+vaultrs = { version = "0.8.0" }
+walkdir = "2.5.0"
+wildmatch = { version = "2.6.1", features = ["serde"] }
+windows = { version = "0.62.2" }
+xxhash-rust = { version = "0.8.15", features = ["xxh64", "xxh3"] }
+zip = "8.6.0"
+zstd = "0.13.3"
+
+# Observability and Metrics
+metrics = "0.24.5"
+dial9-tokio-telemetry = "0.3"
+opentelemetry = { version = "0.31.0" }
+opentelemetry-appender-tracing = { version = "0.31.1", features = ["experimental_use_tracing_span_context", "experimental_metadata_attributes", "spec_unstable_logs_enabled"] }
+opentelemetry-otlp = { version = "0.31.1", features = ["gzip-http", "reqwest-rustls"] }
+opentelemetry_sdk = { version = "0.31.0" }
+opentelemetry-semantic-conventions = { version = "0.31.0", features = ["semconv_experimental"] }
+opentelemetry-stdout = { version = "0.31.0" }
+pyroscope = { version = "2.0.3", features = ["backend-pprof-rs"] }
+
+# FTP and SFTP
+libunftp = { version = "0.23.0", features = ["experimental"] }
+unftp-core = "0.1.0"
+suppaftp = { version = "8.0.3", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
+rcgen = "0.14.7"
+russh = "0.60.2"
+russh-sftp = "2.1.2"
+
+# WebDAV
+dav-server = "0.11.0"
+
+# Performance Analysis and Memory Profiling
+mimalloc = "0.1"
+# Use tikv-jemallocator as memory allocator and enable performance analysis
+tikv-jemallocator = { version = "0.6", features = ["profiling", "stats", "unprefixed_malloc_on_supported_platforms", "background_threads"] }
+# Used to control and obtain statistics for jemalloc at runtime
+tikv-jemalloc-ctl = { version = "0.6", features = ["use_std", "stats", "profiling"] }
+# Used to generate pprof-compatible memory profiling data and support symbolization and flame graphs
+jemalloc_pprof = { version = "0.8.2", features = ["symbolize", "flamegraph"] }
+# Used to generate CPU performance analysis data and flame diagrams
+# pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
+# Pyroscope uses a patched pprof, until they merge back upstream, replace all references. Otherwise, two pprof libs with symbol collision.
+pprof = { package = "pprof-pyroscope-fork", version = "0.1500.3", features = ["flamegraph", "protobuf-codec"] }
+
+[workspace.metadata.cargo-shear]
+ignored = ["rustfs"]
+
+[patch.crates-io]
+# Pinned to the simon-escapecode/russh fork carrying the upstream fix at
+# https://github.com/Eugeny/russh/pull/702. Drops out when a russh release
+# containing the fix is published.
+russh = { git = "https://github.com/simon-escapecode/russh", rev = "5cac2ed84945f9b80a52b673e058f2032bbe98ec" }
+
+[profile.release]
+opt-level = 3
+
+[profile.production]
+inherits = "release"
+lto = "fat"
+codegen-units = 1
+
+[profile.profiling]
+inherits = "release"
+debug = true

Dockerfile

@@ -0,0 +1,116 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM alpine:3.23.4 AS build
+
+ARG TARGETARCH
+ARG RELEASE=latest
+
+RUN apk add --no-cache ca-certificates curl unzip
+WORKDIR /build
+
+RUN set -eux; \
+    case "$TARGETARCH" in \
+      amd64)  ARCH_SUBSTR="x86_64-musl"  ;; \
+      arm64)  ARCH_SUBSTR="aarch64-musl" ;; \
+      *) echo "Unsupported TARGETARCH=$TARGETARCH" >&2; exit 1 ;; \
+    esac; \
+    if [ "$RELEASE" = "latest" ]; then \
+      TAG="$(curl -fsSL https://api.github.com/repos/rustfs/rustfs/releases \
+              | grep -o '"tag_name": "[^"]*"' | cut -d'"' -f4 | head -n 1)"; \
+      RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG")"; \
+    else \
+      TAG="$RELEASE"; \
+      RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG" 2>/dev/null || true)"; \
+      if [ -z "$RELEASE_JSON" ]; then \
+        if [ "${TAG#v}" = "$TAG" ]; then \
+          ALT_TAG="v$TAG"; \
+        else \
+          ALT_TAG="${TAG#v}"; \
+        fi; \
+        echo "Primary tag lookup failed, retrying with alternate tag: $ALT_TAG"; \
+        RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$ALT_TAG")"; \
+        TAG="$ALT_TAG"; \
+      fi; \
+    fi; \
+    echo "Using tag: $TAG (arch pattern: $ARCH_SUBSTR)"; \
+    # Find download URL in assets list for this tag that contains arch substring and ends with .zip
+    URL="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG" \
+           | grep -o "\"browser_download_url\": \"[^\"]*${ARCH_SUBSTR}[^\"]*\\.zip\"" \
+           | cut -d'"' -f4 | head -n 1)"; \
+    if [ -z "$URL" ]; then echo "Failed to locate release asset for $ARCH_SUBSTR at tag $TAG" >&2; exit 1; fi; \
+    echo "Downloading: $URL"; \
+    curl -fL "$URL" -o rustfs.zip; \
+    unzip -q rustfs.zip -d /build; \
+    # If binary is not in root directory, try to locate and move from zip to /build/rustfs
+    if [ ! -x /build/rustfs ]; then \
+      BIN_PATH="$(unzip -Z -1 rustfs.zip | grep -E '(^|/)rustfs$' | head -n 1 || true)"; \
+      if [ -n "$BIN_PATH" ]; then \
+        mkdir -p /build/.tmp && unzip -q rustfs.zip "$BIN_PATH" -d /build/.tmp && \
+        mv "/build/.tmp/$BIN_PATH" /build/rustfs; \
+      fi; \
+    fi; \
+    [ -x /build/rustfs ] || { echo "rustfs binary not found in asset" >&2; exit 1; }; \
+    chmod +x /build/rustfs; \
+    rm -rf rustfs.zip /build/.tmp || true
+
+
+FROM alpine:3.23.4
+
+ARG RELEASE=latest
+ARG BUILD_DATE
+ARG VCS_REF
+
+LABEL name="RustFS" \
+      vendor="RustFS Team" \
+      maintainer="RustFS Team <dev@rustfs.com>" \
+      version="v${RELEASE#v}" \
+      release="${RELEASE}" \
+      build-date="${BUILD_DATE}" \
+      vcs-ref="${VCS_REF}" \
+      summary="High-performance distributed object storage system compatible with S3 API" \
+      description="RustFS is a distributed object storage system written in Rust, supporting erasure coding, multi-tenant management, and observability." \
+      url="https://rustfs.com" \
+      license="Apache-2.0"
+
+RUN apk update && \
+    apk add --no-cache ca-certificates coreutils curl
+
+COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
+COPY --from=build /build/rustfs /usr/bin/rustfs
+COPY entrypoint.sh /entrypoint.sh
+
+RUN chmod +x /usr/bin/rustfs /entrypoint.sh
+
+RUN addgroup -g 10001 -S rustfs && \
+    adduser -u 10001 -G rustfs -S rustfs -D && \
+    mkdir -p /data /logs && \
+    chown -R rustfs:rustfs /data /logs && \
+    chmod 0750 /data /logs
+
+ENV RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
+    RUSTFS_VOLUMES="/data" \
+    RUSTFS_OBS_LOGGER_LEVEL=warn \
+    RUSTFS_OBS_LOG_DIRECTORY=/logs \
+    RUSTFS_OBS_ENVIRONMENT=production
+    
+EXPOSE 9000 9001
+
+VOLUME ["/data"]
+
+USER rustfs
+
+ENTRYPOINT ["/entrypoint.sh"]
+
+CMD ["rustfs"]

Dockerfile.decommission-local

@@ -0,0 +1,65 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM rust:1.93-trixie
+
+RUN set -eux; \
+    export DEBIAN_FRONTEND=noninteractive; \
+    apt-get update; \
+    apt-get install -y --no-install-recommends \
+    build-essential \
+    ca-certificates \
+    curl \
+    git \
+    pkg-config \
+    libssl-dev \
+    lld \
+    protobuf-compiler \
+    flatbuffers-compiler; \
+    rm -rf /var/lib/apt/lists/*
+
+WORKDIR /usr/src/rustfs
+
+COPY . .
+
+RUN ./scripts/static.sh
+RUN cargo run --bin gproto
+RUN cargo build --release --locked --bin rustfs
+
+RUN set -eux; \
+    groupadd -g 10001 rustfs; \
+    useradd -u 10001 -g rustfs -M -s /usr/sbin/nologin rustfs
+
+WORKDIR /app
+
+RUN set -eux; \
+    mkdir -p /data /logs; \
+    chown -R rustfs:rustfs /data /logs /app; \
+    chmod 0750 /data /logs
+
+COPY entrypoint.sh /entrypoint.sh
+RUN install -m 0755 /usr/src/rustfs/target/release/rustfs /usr/bin/rustfs && chmod +x /entrypoint.sh
+
+ENV RUSTFS_VOLUMES="/data" \
+    RUST_LOG="warn" \
+    RUSTFS_OBS_LOG_DIRECTORY="/logs" \
+    RUSTFS_USERNAME="rustfs" \
+    RUSTFS_GROUPNAME="rustfs" \
+    RUSTFS_UID="10001" \
+    RUSTFS_GID="10001"
+
+EXPOSE 9000 9001
+
+ENTRYPOINT ["/entrypoint.sh"]
+CMD ["/usr/bin/rustfs"]

Dockerfile.glibc

@@ -0,0 +1,125 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM ubuntu:24.04 AS build
+
+ARG TARGETARCH
+ARG RELEASE=latest
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    ca-certificates \
+    curl \
+    unzip \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /build
+
+RUN set -eux; \
+    case "$TARGETARCH" in \
+      amd64)  ARCH_SUBSTR="x86_64-gnu"  ;; \
+      arm64)  ARCH_SUBSTR="aarch64-gnu" ;; \
+      *) echo "Unsupported TARGETARCH=$TARGETARCH" >&2; exit 1 ;; \
+    esac; \
+    if [ "$RELEASE" = "latest" ]; then \
+      TAG="$(curl -fsSL https://api.github.com/repos/rustfs/rustfs/releases \
+              | grep -o '"tag_name": "[^"]*"' | cut -d'"' -f4 | head -n 1)"; \
+      RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG")"; \
+    else \
+      TAG="$RELEASE"; \
+      RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG" 2>/dev/null || true)"; \
+      if [ -z "$RELEASE_JSON" ]; then \
+        if [ "${TAG#v}" = "$TAG" ]; then \
+          ALT_TAG="v$TAG"; \
+        else \
+          ALT_TAG="${TAG#v}"; \
+        fi; \
+        echo "Primary tag lookup failed, retrying with alternate tag: $ALT_TAG"; \
+        RELEASE_JSON="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$ALT_TAG")"; \
+        TAG="$ALT_TAG"; \
+      fi; \
+    fi; \
+    echo "Using tag: $TAG (arch pattern: $ARCH_SUBSTR)"; \
+    URL="$(curl -fsSL "https://api.github.com/repos/rustfs/rustfs/releases/tags/$TAG" \
+           | grep -o "\"browser_download_url\": \"[^\"]*${ARCH_SUBSTR}[^\"]*\\.zip\"" \
+           | cut -d'"' -f4 | head -n 1)"; \
+    \
+    if [ -z "$URL" ]; then echo "Failed to locate release asset for $ARCH_SUBSTR at tag $TAG" >&2; exit 1; fi; \
+    \
+    curl -fL "$URL" -o rustfs.zip; \
+    unzip -q rustfs.zip -d /build; \
+    \
+    if [ ! -x /build/rustfs ]; then \
+      BIN_PATH="$(unzip -Z -1 rustfs.zip | grep -E '(^|/)rustfs$' | head -n 1 || true)"; \
+      if [ -n "$BIN_PATH" ]; then \
+        mkdir -p /build/.tmp && unzip -q rustfs.zip "$BIN_PATH" -d /build/.tmp && \
+        mv "/build/.tmp/$BIN_PATH" /build/rustfs; \
+      fi; \
+    fi; \
+    [ -x /build/rustfs ] || { echo "rustfs binary not found in asset" >&2; exit 1; }; \
+    chmod +x /build/rustfs; \
+    rm -rf rustfs.zip /build/.tmp || true
+
+FROM ubuntu:24.04
+
+ARG RELEASE=latest
+ARG BUILD_DATE
+ARG VCS_REF
+
+LABEL name="RustFS" \
+      vendor="RustFS Team" \
+      maintainer="RustFS Team <dev@rustfs.com>" \
+      version="v${RELEASE#v}" \
+      release="${RELEASE}" \
+      build-date="${BUILD_DATE}" \
+      vcs-ref="${VCS_REF}" \
+      summary="High-performance distributed object storage system (glibc)" \
+      url="https://rustfs.com" \
+      license="Apache-2.0"
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    ca-certificates \
+    curl \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY --from=build /build/rustfs /usr/bin/rustfs
+COPY entrypoint.sh /entrypoint.sh
+
+RUN chmod +x /usr/bin/rustfs /entrypoint.sh
+
+RUN groupadd -g 10001 rustfs && \
+    useradd -u 10001 -g rustfs -m -s /sbin/nologin rustfs && \
+    mkdir -p /data /logs && \
+    chown -R rustfs:rustfs /data /logs && \
+    chmod 0750 /data /logs
+
+ENV RUSTFS_ADDRESS=":9000" \
+    RUSTFS_CONSOLE_ADDRESS=":9001" \
+    RUSTFS_ACCESS_KEY="rustfsadmin" \
+    RUSTFS_SECRET_KEY="rustfsadmin" \
+    RUSTFS_CONSOLE_ENABLE="true" \
+    RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
+    RUSTFS_VOLUMES="/data" \
+    RUSTFS_OBS_LOGGER_LEVEL=warn \
+    RUSTFS_OBS_LOG_DIRECTORY=/logs \
+    RUSTFS_OBS_ENVIRONMENT=production
+    
+EXPOSE 9000 9001
+
+VOLUME ["/data"]
+
+USER rustfs
+
+ENTRYPOINT ["/entrypoint.sh"]
+
+CMD ["rustfs"]